Project: VictoriaMetrics (Observability and Analysis / Observability)
File: scrape_config_examples.md

sort: 200 weight: 200 title: Scrape config examples menu: docs: parent: 'victoriametrics' weight: 200 aliases: /scrapeconfigexamples.html Let's start from a simple case with scraping targets at pre-defined addresses. Create a `scrape.yaml` file with the following contents: ```yaml scrape_configs: job_name: node-exporter static_configs: targets: localhost:9100 ``` After you created the `scrape.yaml` file, download and unpack to the same directory: ``` wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.100.1/victoria-metrics-linux-amd64-v1.100.1.tar.gz tar xzf victoria-metrics-linux-amd64-v1.100.1.tar.gz ``` Then start VictoriaMetrics and instruct it to scrape targets defined in `scrape.yaml` and save scraped metrics to local storage according to : ``` ./victoria-metrics-prod -promscrape.config=scrape.yaml ``` Now open the `http://localhost:8428/targets` page in web browser in order to see the current status for scrape targets. The page must contain the information about the target at `http://localhost:9100/metrics` url. It is likely the target has `state: down` if you didn't start on `localhost`. Let's add a new scrape config to `scrape.yaml` for scraping : ```yaml scrape_configs: job_name: node-exporter static_configs: targets: localhost:9100 job_name: victoriametrics static_configs: targets: http://localhost:8428/metrics ``` Note that the last specified target contains the full url instead of host and port. This is an extension supported by VictoriaMetrics and - you can use both `host:port` and full urls in scrape target lists. Send `SIGHUP` signal `victoria-metrics-prod` process, so it : ``` kill -HUP `pidof victoria-metrics-prod` ``` Now the `http://localhost:8428/targets` page must contain two targets - `http://localhost:9100/metrics` and `http://localhost:8428/metrics`. The last one should have `state: up`, since this is VictoriaMetrics itself. Let's query the scraped metrics. Open `http://localhost:8428/vmui/` aka , enter `up` in the query input field and press `enter`. You'll see a graph for `up` metrics. It must contain two lines for the targets defined in `scrape.yaml` file above. See about `up` metric. You can explore other scraped metrics in `vmui` via . Let's look closely to the contents of the `scrape.yaml` file created above: ```yaml scrape_configs: job_name: node-exporter static_configs: targets: localhost:9100 job_name: victoriametrics static_configs: targets: http://localhost:8428/metrics ``` The section contains a list of scrape configs. Our `scrape.yaml` file contains two scrape configs - for `jobname: node-exporter` and for `jobname: victoriametrics`. and can efficiently process thousands of scrape configs in production. Every scrape config in the list must contain `job_name` field - its' value is used as label in all the metrics scraped from targets defined in this scrape config. Every scrape config must contain at least a single section from . Every scrape config may contain other options described . In our case only sections are used. These sections consist of a list of static configs according to . Every static config contains a list of `targets`, which need to be scraped. The target address is used as label in all the metrics scraped from the target. and can efficiently process tens of thousands of targets in production. If you need scraping more targets, then see . Targets are scraped at `http` or `https` urls, which are formed according to . It is possible to modify scrape urls via if needed. It may be not so convenient updating `scrape.yaml` file with every time new scrape target is added, changed or removed. In this case can come to rescue. It allows defining a list of scrape targets in `JSON` files, and automatically updating the list of scrape targets at or side when the corresponding `JSON` files are updated. Let's create `nodeexportertargets.json` file with the following conents: ```json [ { "targets": ["host1:9100", "host2:9100"] } ] ``` Then create `scrape.yaml` file with the following contents: ```yaml scrape_configs: job_name: node-exporter filesdconfigs: files: nodeexportertargets.json ``` Then start according to : ```yaml wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.100.1/victoria-metrics-linux-amd64-v1.100.1.tar.gz tar xzf victoria-metrics-linux-amd64-v1.100.1.tar.gz ./victoria-metrics-prod

``` Then open `http://localhost:8428/targets` page in web browser and see that it contains the two targets defined in `nodeexportertargets.json` above. Now let's add more targets to `nodeexportertargets.json`: ```json [ { "targets": ["host1:9100", "host2:9100", "http://host3:9100/metrics", "http://host4:9100/metrics"] } ] ``` Note that the added targets contains full urls instead of host and port. This is an extension supported by VictoriaMetrics and - you can use both `host:port` and full urls in scrape target lists. Save the updated `nodeexportertargets.json`, wait for 30 seconds and then refresh the `http://localhost:8428/targets` page. Now this page must contain all the targets defined in the updated `nodeexportertargets.json`. By default and check for updates in `files` specified at every 30 seconds. This interval can be changed via `-promscrape.fileSDCheckInterval` command-line flag. For example, the following command starts VictoriaMetrics, which checks for updates in `filesdconfigs` every 5 seconds: ``` ./victoria-metrics-prod -promscrape.config=scrape.yaml -promscrape.fileSDCheckInterval=5s ``` If the `files` contents is broken during the check, then the previous list of scrape targets is kept. It is possible specifying `http` and/or `https` urls in `files` list. For example, the following config instructs obtaining fresh list of targets at `http://central-config-server/targets?type=node-exporter` url additionally to `nodeexportertargets.json` local file: ```yaml scrape_configs: job_name: node-exporter filesdconfigs: files: nodeexportertargets.json 'http://central-config-server/targets?type=node-exporter' ``` It is possible to specify directories with `*` wildcards for distinct sets of targets at `filesdconfigs`. See for details. and can efficiently scrape tens of thousands of scrape targets. If you need scraping more targets, then see . Targets are scraped at `http` or `https` urls, which are formed according to . It is possible to modify scrape urls via if needed. It may not so convenient maintaining a list of local files for . In this case can help. They allow specifying a list of `http` or `https` urls, which return targets, which need to be scraped. For example, the following periodically fetches the list of targets from the specified url: ```yaml scrape_configs: job_name: node-exporter httpsdconfigs: url: "http://central-config-server/targets?type=node-exporter" ``` Kubernetes target discovery is non-trivial task in general. That's why it is recommended using either or for Kubernetes monitoring. If you feel brave, let's look at a few typical cases for Kubernetes monitoring. The following instructs discovering and scraping all the targets inside Kubernetes cluster: ```yaml scrape_configs: job_name: node-exporter kubernetessdconfigs: role: pod relabel_configs: sourcelabels: [metakubernetespodcontainer_name] regex: node-exporter action: keep sourcelabels: [metakubernetespodnode_name] target_label: node ``` See for more details. See for details on `relabel_configs`. is a special metrics exporter, which exposes `state` metrics for all the Kubernetes objects such as `container`, `pod`, `node`, etc. It already sets `namespace`, `container`, `pod` and `node` labels for every exposed metric, so these metrics shouldn't be set in . The following instructs discovering and scraping target inside Kubernetes cluster: ```yaml scrape_configs: job_name: kube-state-metrics kubernetessdconfigs: role: pod relabel_configs: sourcelabels: [metakubernetespodcontainer_name] regex: kube-state-metrics action: keep sourcelabels: [metakubernetespodcontainerportnumber] regex: "8080" action: keep ``` See for more details. See for details on `relabel_configs`. exposes resource usage metrics for every container in Kubernetes. The following can be used for collecting `cadvisor` metrics in Kubernetes: ```yaml scrape_configs: job_name: cadvisor kubernetessdconfigs: role: node bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt relabel_configs: sourcelabels: [metakubernetesnodename] targetlabel: address_ regex: '(.+)' replacement: https://kubernetes.default.svc/api/v1/nodes/$1/proxy/metrics/cadvisor sourcelabels: [metakubernetesnodename] target_label: instance ``` See for more details. See for details on `relabel_configs`. See for details on `bearertokenfile` and `tls_config` options. The following instructs discovering and scraping metrics for all the containers with the name `my-super-app`. It is expected that these containers expose only a single TCP port, which serves its metrics at `/metrics` page according to : ```yaml scrape_configs: job_name: my-super-app kubernetessdconfigs: role: pod relabel_configs: sourcelabels: [metakubernetespodcontainer_name] target_label: job action: keepequal sourcelabels: [metakubernetes_namespace] target_label: namespace sourcelabels: [metakubernetespodnode_name] target_label: node sourcelabels: [metakubernetespodname] target_label: pod sourcelabels: