Project: VictoriaMetrics (Observability and Analysis / Observability)
File: relabeling.md

sort: 37 weight: 37 title: Relabeling cookbook menu: docs: parent: 'victoriametrics' weight: 37 aliases: /relabeling.html VictoriaMetrics and support with . The relabeling is mostly used for the following tasks: Dropping unneeded scrape targets during . See . Adding or updating static labels at scrape targets. See . Copying target labels from another labels. See . Modifying scrape urls for discovered targets. See . Modifying `instance` and `job` labels. See . Extracting label parts into another labels. See . Removing prefixes from target label names. See . Removing some labels from discovered targets. See . Dropping some metrics during scape. See . Renaming scraped metrics. See . Adding labels to scraped metrics. See . Changing label values in scraped metrics. See . Removing some labels from scraped metrics. See . Removing some labels from metrics matching some . See . See also . Sometimes it may be needed to remove labels from a subset of scraped metrics, while leaving these labels in the rest of scraped metrics. In this case the `if` can be applied to `action: labeldrop` or `action: labelkeep`. For example, the following config drops labels with names starting from `foo_` prefix from metrics matching `a{b="c"}` series selector: ```yaml scrape_configs: job_name: test static_configs: targets: [host123] metricrelabelconfigs: action: labeldrop if: 'a{b="c"}' regex: "foo_.*" ``` See also: Metric name is a regular label with special name - `name` (see ). So renaming of metric name is performed in the same way as changing label value. Let's look at a few examples. The following config renames `foo` metric to `bar` across all the scraped metrics, while leaving other metric names as is: ```yaml scrape_configs: job_name: test static_configs: targets: [host123] metricrelabelconfigs: if: 'foo' replacement: bar targetlabel: name_ ``` The following config renames metrics starting from `foo` to metrics starting from `bar` across all the scraped metrics. For example, `foocount` is renamed to `barcount`: ```yaml scrape_configs: job_name: test static_configs: targets: [host123] metricrelabelconfigs: sourcelabels: [name_] regex: 'foo_(.*)' replacement: bar_$1 targetlabel: name_ ``` The following config replaces all the `-` chars in metric names with `` chars across all the scraped metrics. For example, `foo-bar-baz` is renamed to `foobar_baz`: ```yaml scrape_configs: job_name: test static_configs: targets: [host123] metricrelabelconfigs: sourcelabels: [name_] action: replace_all regex: '-' replacement: '_' targetlabel: name_ ``` See also . The following config sets `foo="bar"` across all the scraped metrics: ```yaml scrape_configs: job_name: test static_configs: targets: [host123] metricrelabelconfigs: target_label: foo replacement: bar ``` The following config sets `foo="bar"` label only for metrics matching `{job=~"my-app-.*",env!="dev"}` : ```yaml scrape_configs: job_name: test static_configs: targets: [host123] metricrelabelconfigs: if: '{job=~"my-app-.*",env!="dev"}' target_label: foo replacement: bar ``` See also . The following config adds `foo_` prefix to all the values of `job` label across all the scraped metrics: ```yaml scrape_configs: job_name: test static_configs: targets: [host123] metricrelabelconfigs: source_labels: [job] target_label: job replacement: foo_$1 ``` The following config adds `foo_` prefix to `job` label values only for metrics matching `{job=~"my-app-.*",env!="dev"}` : ```yaml scrape_configs: job_name: test static_configs: targets: [host123] metricrelabelconfigs: if: '{job=~"my-app-.*",env!="dev"}' source_labels: [job] target_label: job replacement: foo_$1 ``` See also . Sometimes it may be needed to remove labels from scraped metrics. For example, if some labels lead to or issues, then it may be a good idea to drop these labels during

This can be done with `action: labeldrop` or `action: labelkeep` relabeling rules at `metricrelabelconfigs` section: `action: labeldrop` drops labels with names matching the given `regex` option `action: labelkeep` drops labels with names not matching the given `regex` option For example, the following config drops labels with names starting with `foo_` prefix from all the metrics scraped from the `http://host123/metrics`: ```yaml scrape_configs: job_name: test static_configs: targets: [host123] metricrelabelconfigs: action: labeldrop regex: "foo_.*" ``` The `regex` option can contain arbitrary regular expression in . The `regex` option is applied to every label name in the target. It is automatically anchored, so it must match the whole label name. The label name is left as is if the `regex` doesn't match it. Important notes: Labels with `` prefix are automatically removed after the relabeling, so there is no need in removing them with relabeling rules. Make sure that metrics exposed by the target can be uniquely identified by their names and the remaining labels after label removal. Otherwise, duplicate metrics with duplicate timestamps and different values will be pushed to the storage. This is an undesired issue in most cases. See also . Sometimes it is needed to drop some metrics during scrapes. For example, if some metrics result in or issues, then it may be a good idea to drop these metrics during scrapes. This can be done with the `action: drop` or `action: keep` relabeling rules at `metricrelabelconfigs` section: `action: drop` drops all the metrics, which match the `if` `action: keep` drops all the metrics, which don't match the `if` For example, the following config drops all the metrics obtained from `http://host123/metrics`, which start with `foo_` prefix: ```yaml scrape_configs: job_name: test static_configs: targets: [host123] metricrelabelconfigs: if: '{name=~"foo_.*"}' action: drop ``` Note that the relabeling config is specified under `metricrelabelconfigs` section instead of `relabel_configs` section: The `relabel_configs` is applied to the configured/discovered targets. The `metricrelabelconfigs` is applied to metrics scraped from the configured/discovered targets. See also . Sometimes it is needed to remove some labels from a subset of , while leaving these labels in the rest of discovered targets. In this case the `if` can be added to `action: labeldrop` or `action: labelkeep` relabeling rule. For example, the following config discovers pod targets in , into labels with `foo_` prefix and then drops all the labels with `foobar` prefix in their names for targets matching `{address=~"pod123.+"}` selector: ```yaml scrape_configs: job_name: k8s kubernetessdconfigs: role: pod relabel_configs: action: labelmap regex: "metakubernetespodlabel(.+)" replacement: "foo_$1" action: labeldrop if: '{address=~"pod123.+"}' regex: "foobar.*" ``` See also . Sometimes it is needed to remove some labels from . In this case the `action: labeldrop` and `action: labelkeep` relabeling options can be used: `action: labeldrop` drops all the labels with names matching the `regex` option `action: labelkeep` drops all the labels with names not matching the `regex` option For example, the following config discovers pod targets in , into labels with `foo_` prefix and then drops all the labels with `foobar` prefix in their names: ```yaml scrape_configs: job_name: k8s kubernetessdconfigs: role: pod relabel_configs: action: labelmap regex: "metakubernetespodlabel(.+)" replacement: "foo_$1" action: labeldrop regex: "foobar.*" ``` The `regex` option can contain arbitrary regular expression in . The `regex` option is applied to every label name in the target. It is automatically anchored, so it must match the whole label

The label name is left as is if the `regex` doesn't match it. Important notes: Labels with `` prefix are automatically removed after the relabeling, so there is no need in removing them with relabeling rules. Do not remove `instance` and `job` labels, since this may result in duplicate scrape targets with identical sets of labels. See also . Sometimes it is needed to remove `meta_*` prefixes from meta-labels of the . For example, adds `metakubernetespodlabel<labelname>` labels per each pod-level label. In this case it may be needed to leave only the `<labelname>` part of such label names, while removing the `metakubernetespodlabel` prefix. This can be done with `action: labelmap` relabeling option: ```yaml scrape_configs: job_name: k8s kubernetessdconfigs: role: pod relabel_configs: action: labelmap regex: "metakubernetespodlabel(.+)" replacement: "$1" ``` The `regex` option can contain arbitrary regular expression in . The `regex` option is applied to every label name in the target. It is automatically anchored, so it must match the whole label name. It can contain capture groups such as `(.+)` in the config above. These capture groups can be referenced then inside `replacement` option with the `$N` syntax, where `N` is the number of the capture group in `regex`. The first capture group has the `$1` reference. The label name is left as is if the `regex` doesn't match it. See also . Relabeling allows extracting parts from label values and storing them into arbitrary labels. This is performed with `regex` and `replacement` options in relabeling rules. For example, the following config discovers pod targets in , extracts `bar` part from `foo/bar` container name and stores it into the `xyz` label with `abc_` prefix: ```yaml scrape_configs: job_name: k8s kubernetessdconfigs: role: pod relabel_configs: sourcelabels: [metakubernetespodcontainer_name] regex: "[^/]+/(.+)" replacement: "abc_$1" target_label: xyz ``` The `regex` option can contain arbitrary regular expression in . The `regex` option is automatically anchored, so it must match the whole value from `source_labels`. It can contain capture groups such as `(.+)` in the config above. These capture groups can be referenced then inside `replacement` option with the `$N` syntax, where `N` is the number of the capture group in `regex`. The first capture group has the `$1` reference. It is possible to construct a label from multiple parts of different labels. In this case just specify the needed source labels inside `source_labels` list. The values of labels specified in `source_labels` list are joined with `;` separator by default before being matched against the `regex`. The separator can be overridden via `separator` option. If the `regex` doesn't match the value constructed from `source_labels`, then the relabeling rule is skipped and the remaining relabeling rules are executed. See also . Single-node VictoriaMetrics and automatically add `instance` and `job` labels per each discovered target: The `job` label is set to `job_name` value specified in the corresponding . The `instance` label is set to the `host:port` part of `address` label value after target-level relabeling. The `address` label value is automatically set to the most suitable value depending on the used . The `address` label can be overridden during relabeling - see . Both `instance` and `job` labels can be overridden during

For example, the following config discovers pod targets in and overrides `job` label from `k8s` to `foo`: ```yaml scrape_configs: job_name: k8s kubernetessdconfigs: role: pod relabel_configs: target_label: job replacement: foo ``` See also . URLs for scrape targets are composed of the following parts: Scheme (e.g. `http` or `https`). The scheme is available during target relabeling in a special label - `scheme`. By default, the scheme is set to `http`. It can be overridden either by specifying the `scheme` option at level or by updating the `scheme` label during relabeling. Host and port (e.g. `host12:3456`). This information is available during target relabeling in a special label - `address`. Every sets the `address` label to the most suitable value. Sometimes this value needs to be modified. In this case just update the `address` label during relabeling to the needed value. The port part is optional. If it is missing, then it is automatically set either to `80` or `443` depending on the used scheme (`http` or `https`). The `host:port` part from the final `address` label is automatically set to `instance` label unless the `instance` label is explicitly set during relabeling. The `address` label can contain the full scrape url, e.g. `http://host:port/metrics/path?query_args`. In this case the `scheme` and `metrics_path` labels are ignored. URL path (e.g. `/metrics`). This information is available during target relabeling in a special label - `metrics_path`. By default, the `metrics_path` is set to `/metrics`. It can be overridden either by specifying the `metrics_path` option at or by updating the `metrics_path` label during relabeling. Query args (e.g. `?foo=bar&baz=xyz`). This information is available during target relabeling in special labels with `param_` prefix. For example, `paramfoo` would have the `bar` value, while `parambaz` would have the `xyz` value for `?foo=bar&baz=xyz` query string. The query args can be specified either via `params` section at or by updating/setting the corresponding `param_*` labels during relabeling. The resulting scrape url looks like the following: ``` <scheme> + "://" + <address> + <metrics_path> + <"?" + queryargsfromparamlabels> ``` It is expected that the target exposes metrics in at the resulting scrape url. Given the scrape url construction rules above, the following config discovers pod targets in and constructs per-target scrape url as `https://<podname>/foo/bar?baz=<containername>`: ```yaml scrape_configs: job_name: k8s kubernetessdconfigs: role: pod metrics_path: /foo/bar relabel_configs: targetlabel: scheme_ replacement: https sourcelabels: [metakubernetespodname] targetlabel: address_ sourcelabels: [metakubernetespodcontainer_name] targetlabel: parambaz ``` See also . Labels can be copied by specifying the source labels via `source_labels` relabeling option and specifying the target label via `target_label` relabeling option. For example, the following config copies `metakubernetespod_name` label to `pod` label for all the discovered pods in : ```yaml scrape_configs: job_name: k8s kubernetessdconfigs: role: pod relabel_configs: sourcelabels: [metakubernetespodname] target_label: pod ``` Note that the `source_labels` option accepts a list of labels in square brackets. If multiple labels are specified in the `source_labels` list, then the specified label values are joined into a single string with `;` delimiter by default. The delimiter can be modified by specifying it via `separator` option. For example, the following config sets the `podname:containerport` value to the `host_port` label for all the discovered pod targets in : ```yaml scrape_configs: job_name: k8s kubernetessdconfigs: role: pod relabel_configs: sourcelabels: [metakubernetespodname, metakubernetespodcontainerport_number] separator: ":" targetlabel: hostport ``` See also . Additional labels can be added to scrape targets by specifying the label name in `target_label` relabeling option and by specifying the label value in `replacement` relabeling

The same approach can be used for updating already existing label values at target level. For example, the following config adds `{foo="bar"}` label to all the discovered pods in : ```yaml scrape_configs: job_name: k8s kubernetessdconfigs: role: pod relabel_configs: target_label: "foo" replacement: "bar" ``` The labels, which are added to the target, are automatically added to all the metrics scraped from the target. For example, if the target exposes the metric `metric{label="value"}`, then the metric is transformed into `metric{label="value",foo="bar"}` before being sent to the storage. If the metric exported by the target contains the same label as the target itself, then the `exported_` prefix is added to the exported label name. For example, if the target exposes the metric `metric{foo="baz"}`, then the metric is transformed into `metric{exported_foo="baz",foo="bar"}`. This behaviour can be changed by specifying `honor_labels: true` option at the given scrape config. In this case the exported label overrides the target's label. In this case the `metric{foo="baz"}` stays the same. Example config with `honor_labels: true`: ```yaml scrape_configs: job_name: k8s kubernetessdconfigs: role: pod honor_labels: true relabel_configs: target_label: "foo" replacement: "bar" ``` See also . If a particular discovered target shouldn't be scraped, then `action: keep` or `action: drop` relabeling rules must be used inside `relabel_configs` section. The `action: keep` keeps only scrape targets with labels matching the `if` , while dropping the rest of targets. For example, the following config discovers pod targets in and scrapes only pods with names starting with `foo` prefix: ```yaml scrape_configs: jobname: foopods kubernetessdconfigs: role: pod relabel_configs: if: '{metakubernetespod_name=~"foo.*"}' action: keep ``` The `action: drop` drops all the scrape targets with labels matching the `if` , while keeping the rest of targets. For example, the following config discovers pod targets in and scrapes only pods with names starting with prefixes other than `foo`: ```yaml scrape_configs: jobname: notfoo_pods kubernetessdconfigs: role: pod relabel_configs: if: '{metakubernetespod_name=~"foo.*"}' action: drop ``` See also . Target relabeling can be debugged by clicking the `debug` link for the needed target on the `http://vmagent:8429/target` or on the `http://vmagent:8429/service-discovery` pages. See . Every discovered target contains a set of meta-labels, which start with `meta_` prefix. The specific sets of labels per each supported service discovery option are listed . Every discovered target contains additional labels with `` prefix other than `meta_` labels. See for more details. All the labels, which start with `` prefix, are automatically removed from targets after the relabeling. So it is common practice to store temporary labels with names starting with `` during target relabeling. All the target-level labels are automatically added to all the metrics scraped from targets. The list of discovered scrape targets with all the discovered meta-labels is available at `http://vmagent:8429/service-discovery` page for `vmagent` and at `http://victoriametrics:8428/service-discovery` page for single-node VictoriaMetrics. The list of active targets with the final set of labels left after relabeling is available at `http://vmagent:8429/targets` page for `vmagent` and at `http://victoriametrics:8428/targets` page for single-node VictoriaMetrics. Metric relabeling can be debugged at `http://vmagent:8429/metric-relabel-debug` page. See . All the labels, which start with `` prefix, are automatically removed from metrics after the relabeling. So it is common practice to store temporary labels with names starting with `` during metrics relabeling. All the target-level labels are automatically added to all the metrics scraped from targets, so target-level labels are available during metrics relabeling.