Project: Grafana Loki (Observability and Analysis / Observability)
File: scraping.md

title: Configuring Promtail for service discovery menuTitle: Configure service discovery description: Configuring Promtail for service discovery aliases: ../../clients/promtail/scraping/ weight: 400 Promtail currently supports scraping from the following sources: Promtail supports reading messages from Azure Event Hubs. Targets can be configured using the `azureeventhubs` stanza: ```yaml jobname: azureevent_hubs azureeventhubs: group_id: "mygroup" fullyqualifiednamespace: my-namespace.servicebus.windows.net:9093 connection_string: "my-connection-string" event_hubs: event-hub-name labels: job: azureeventhub relabel_configs: action: replace source_labels: azureeventhubs_category target_label: category ``` Only `fullyqualifiednamespace`, `connectionstring` and `eventhubs` are required fields. Read the section for more information. Promtail supports pulling HTTP log messages from Cloudflare using the . The Cloudflare targets can be configured with a `cloudflare` block: ```yaml scrape_configs: job_name: cloudflare cloudflare: api_token: REDACTED zone_id: REDACTED fields_type: all labels: job: cloudflare-foo.com ``` Only `apitoken` and `zoneid` are required. Refer to the configuration section for details. Promtail discovers locations of log files and extract labels from them through the `scrape_configs` section in the config YAML. The syntax is identical to what . `scrape_configs` contains one or more entries which are executed for each discovered target (i.e., each container in each new pod running in the instance): ``` scrape_configs: job_name: local static_configs: ... job_name: kubernetes kubernetessdconfig: ... ``` If more than one scrape config section matches your logs, you will get duplicate entries as the logs are sent in different streams likely with slightly different labels. There are different types of labels present in Promtail: Labels starting with `` (two underscores) are internal labels. They usually come from dynamic sources like service discovery. Once relabeling is done, they are removed from the label set. To persist internal labels so they're sent to Grafana Loki, rename them so they don't start with ``. See for more information. Labels starting with `metakubernetespodlabel*` are "meta labels" which are generated based on your Kubernetes pod's labels. For example, if your Kubernetes pod has a label `name` set to `foobar`, then the `scrape_configs` section will receive an internal label `metakubernetespodlabelname` with a value set to `foobar`. Other labels starting with `metakubernetes*` exist based on other Kubernetes metadata, such as the namespace of the pod (`metakubernetesnamespace`) or the name of the container inside the pod (`metakubernetespodcontainername`). Refer to for the full list of Kubernetes meta labels. The `path` label is a special label which Promtail uses after discovery to figure out where the file to read is located. Wildcards are allowed, for example `/var/log/.log` to get all files with a `log` extension in the specified directory, and `/var/log//.log` for matching files and directories recursively. For a full list of options check out the docs for the Promtail uses. The `path_exclude` label is another special label Promtail uses after discovery, to exclude a subset of the files discovered using `path` from being read in the current scrape_config block. It uses the same to enable usage of wildcards and glob patterns. The label `filename` is added for every file found in `path` to ensure the uniqueness of the streams. It is set to the absolute path of the file the line was read from. To scrape a set of log files defined manually on a machine, one can use `staticconfigs` or `filesdconfigs`. Using staticconfigs, one must reload Promtail to apply

Using `filesdconfigs` that reload is not needed. Promtail reloads discovery files when they are updated. The below excerpt of Promtail's configuration show a filesdconfigs that is used to scrape `apt` and `dpkg`'s logs. ```yaml scrape_configs: job_name: apt-dpkg filesdconfigs: files: /etc/promtail/dpkg-apt.yaml refresh_interval: 5m ``` The targets to be scraped by Promtail are defined in `/etc/promtail/dpkg-apt.yaml`. In fact, Promtail read the target to scrape in the list of file provided under `files`. Below is the content of `/etc/promtail/dpkg-apt.yaml`. ```yaml targets: ["localhost"] labels: job: dpkg path: /var/log/dpkg.log targets: ["localhost"] labels: job: apt path: /var/log/apt/*.log ``` As one can realize, `/etc/promtail/dpkg-apt.yaml` contains the list of targets we would have defined under . It defines two targets. The first one with label job set to `dpkg` and `path` specifying dpkg's log file: `/var/log/dpkg.log`. The second has two labels: the label `job` and again `path` specifying the path to APT's log files. This `path` contains a glob. Every log file matching that regular expression will be scrapped under that target. To summarize, the above `/etc/promtail/dpkg-apt.yaml` showcase YAML format of filesdconfig discovery file. The JSON format can be seen , While Promtail can use the Kubernetes API to discover pods as targets, it can only read log files from pods that are running on the same node as the one Promtail is running on. Promtail looks for a `host` label on each target and validates that it is set to the same hostname as Promtail's (using either `$HOSTNAME` or the hostname reported by the kernel if the environment variable is not set). This means that any time Kubernetes service discovery is used, there must be a `relabelconfig` that creates the intermediate label `host_` from `metakubernetespodnodename`: ```yaml relabel_configs: sourcelabels: ['metakubernetespodnode_name'] targetlabel: 'host_' ``` See for more information. For more information on how to configure the service discovery see the . Promtail supports scraping cloud resource logs such as GCS bucket logs, load balancer logs, and Kubernetes cluster logs from GCP. Configuration is specified in the `gcplog` section, within `scrape_config`. There are two kind of scraping strategies: `pull` and `push`. ```yaml job_name: gcplog gcplog: subscriptiontype: "pull" # If the `subscriptiontype` field is empty, defaults to `pull` project_id: "my-gcp-project" subscription: "my-pubsub-subscription" useincomingtimestamp: false # default rewrite timestamps. usefullline: false # default use textPayload as log line. labels: job: "gcplog" relabel_configs: sourcelabels: ['gcpresource_type'] targetlabel: 'resourcetype' sourcelabels: ['gcpresourcelabelsproject_id'] target_label: 'project' ``` Here `project_id` and `subscription` are the only required fields. `project_id` is the GCP project id. `subscription` is the GCP pubsub subscription where Promtail can consume log entries from. Before using `gcplog` target, GCP should be with pubsub subscription to receive logs from. It also supports `relabeling` and `pipeline` stages just like other targets. When Promtail receives GCP logs, various internal labels are made available for : `gcp_logname` `gcp_severity` `gcpresourcetype` `gcpresourcelabels_<NAME>` `gcplabels<NAME>` In the example above, the `projectid` label from a GCP resource was transformed into a label called `project` through `relabelconfigs`. ```yaml job_name: gcplog gcplog: subscription_type: "push" useincomingtimestamp: false labels: job: "gcplog-push" server: httplistenaddress:

httplistenport: 8080 relabel_configs: sourcelabels: ['gcpmessage_id'] targetlabel: 'messageid' sourcelabels: ['gcpattributeslogginggoogleapiscomtimestamp'] targetlabel: 'incomingts' ``` When configuring the GCP Log push target, Promtail will start an HTTP server listening on port `8080`, as configured in the `server` section. This server exposes the single endpoint `POST /gcp/api/v1/push`, responsible for receiving logs from GCP. For Google's PubSub to be able to send logs, Promtail server must be publicly accessible, and support HTTPS. For that, Promtail can be deployed as part of a larger orchestration service like Kubernetes, which can handle HTTPS traffic through an ingress, or it can be hosted behind a proxy/gateway, offloading the HTTPS to that component and routing the request to Promtail. Once that's solved, GCP can be to send logs to Promtail. It also supports `relabeling` and `pipeline` stages. When Promtail receives GCP logs, various internal labels are made available for : `gcpmessageid` `gcpsubscriptionname` `gcpattributes<NAME>` `gcp_logname` `gcp_severity` `gcpresourcetype` `gcpresourcelabels_<NAME>` `gcplabels<NAME>` In the example above, the `gcp_message_id` and the `gcpattributeslogginggoogleapiscom_timestamp` labels are transformed to `messageid` and `incomingts` through `relabel_configs`. All other internal labels, for example some other attribute, will be dropped by the target if not transformed. Promtail supports listening message using the UDP protocol. The GELF targets can be configured using the `gelf` stanza: ```yaml scrape_configs: job_name: gelf gelf: listen_address: "0.0.0.0:12201" useincomingtimestamp: true labels: job: gelf relabel_configs: action: replace source_labels: gelfmessagehost target_label: host action: replace source_labels: gelfmessagelevel target_label: level action: replace source_labels: gelfmessagefacility target_label: facility ``` Promtail supports receiving logs from a Heroku application by using a . Configuration is specified in a`herokudrain` block within the Promtail `scrapeconfig` configuration. ```yaml jobname: herokudrain heroku_drain: server: httplistenaddress: 0.0.0.0 httplistenport: 8080 labels: job: herokudraindocs useincomingtimestamp: true relabel_configs: sourcelabels: ['herokudrain_host'] target_label: 'host' sourcelabels: ['herokudrain_app'] target_label: 'source' sourcelabels: ['herokudrain_proc'] target_label: 'proc' sourcelabels: ['herokudrainlogid'] targetlabel: 'logid' ``` Within the `scrapeconfigs` configuration for a Heroku Drain target, the `jobname` must be a Prometheus-compatible . The section configures the HTTP server created for receiving logs. `labels` defines a static set of label values added to each received log entry. `useincomingtimestamp` can be used to pass the timestamp received from Heroku. Before using a `heroku_drain` target, Heroku should be configured with the URL where the Promtail instance will be listening. Follow the steps in for using the Heroku CLI with a command like the following: ``` heroku drains:add [http|https]://HOSTNAME:8080/heroku/api/v1/drain -a HEROKUAPPNAME ``` Note that the `herokudrainapp` label will contain the source of the log line, either `app` or `heroku` and not the name of the heroku application. The easiest way to provide the actual application name is to include a query parameter when creating the heroku drain and then relabel that parameter in your scraping config, for example: ``` heroku drains:add [http|https]://HOSTNAME:8080/heroku/api/v1/drain?appname=HEROKUAPPNAME -a HEROKUAPP_NAME ``` And then in a relabel_config: ```yaml relabel_configs: sourcelabels: ['herokudrainparamapp_name'] target_label: 'app' ``` It also supports `relabeling` and `pipeline` stages just like other targets. When Promtail receives Heroku Drain logs, various internal labels are made available for : `herokudrainhost` `herokudrainapp` `herokudrainproc` `herokudrainlog_id` In the example above, the `projectid` label from a GCP resource was transformed into a label called `project` through `relabelconfigs`. Promtail uses the Prometheus HTTP client implementation for all calls to Loki. Therefore it can be configured using the `clients` stanza, where one or more connections to Loki can be established: ```yaml clients: [ <client_option> ] ``` Refer to from the Promtail Configuration reference for all available

On systems with `systemd`, Promtail also supports reading from the journal. Unlike file scraping which is defined in the `static_configs` stanza, journal scraping is defined in a `journal` stanza: ```yaml scrape_configs: job_name: journal journal: json: false max_age: 12h path: /var/log/journal matches: _TRANSPORT=kernel labels: job: systemd-journal relabel_configs: sourcelabels: ['journalsystemdunit'] target_label: 'unit' ``` All fields defined in the `journal` section are optional, and are just provided here for reference. The `max_age` field ensures that no older entry than the time specified will be sent to Loki; this circumvents "entry too old" errors. The `path` field tells Promtail where to read journal entries from. The labels map defines a constant list of labels to add to every journal entry that Promtail reads. The `matches` field adds journal filters. If multiple filters are specified matching different fields, the log entries are filtered by both, if two filters apply to the same field, then they are automatically matched as alternatives. When the `json` field is set to `true`, messages from the journal will be passed through the pipeline as JSON, keeping all of the original fields from the journal entry. This is useful when you don't want to index some fields but you still want to know what values they contained. By default, Promtail reads from the journal by looking in the `/var/log/journal` and `/run/log/journal` paths. If running Promtail inside of a Docker container, the path appropriate to your distribution should be bind mounted inside of Promtail along with binding `/etc/machine-id`. Bind mounting `/etc/machine-id` to the path of the same name is required for the journal reader to know which specific journal to read from. For example: ```bash docker run \ -v /var/log/journal/:/var/log/journal/ \ -v /run/log/journal/:/run/log/journal/ \ -v /etc/machine-id:/etc/machine-id \ grafana/promtail:latest \ -config.file=/path/to/config/file.yaml ``` When Promtail reads from the journal, it brings in all fields prefixed with `journal` as internal labels. Like in the example above, the `SYSTEMD_UNIT` field from the journal was transformed into a label called `unit` through `relabel_configs`. See for more information, also look at for a list of fields exposed by the journal. Here's an example where the `SYSTEMDUNIT`, `HOSTNAME`, and `SYSLOGIDENTIFIER` are relabeled for use in Loki. Keep in mind that labels prefixed with `` will be dropped, so relabeling is required to keep these labels. ```yaml job_name: systemd-journal journal: labels: cluster: ops-tools1 job: default/systemd-journal path: /var/log/journal relabel_configs: source_labels: journalsystemd_unit targetlabel: systemdunit source_labels: journalhostname target_label: nodename source_labels: journalsyslogidentifier targetlabel: syslogidentifier ``` Promtail supports reading message from Kafka using a consumer group. The Kafka targets can be configured using the `kafka` stanza: ```yaml scrape_configs: job_name: kafka kafka: brokers: my-kafka-0.org:50705 my-kafka-1.org:50705 topics: ^promtail.* somefixedtopic labels: job: kafka relabel_configs: action: replace source_labels: metakafkatopic target_label: topic action: replace source_labels: metakafkapartition target_label: partition action: replace source_labels: metakafkagroup_id target_label: group action: replace source_labels: metakafkamessage_key targetlabel: messagekey ``` Only the `brokers` and `topics` are required. Read the section for more information. Each `scrapeconfigs` entry can contain a `relabelconfigs` stanza. `relabel_configs` is a list of operations to transform the labels from discovery into another form. A single entry in `relabel_configs` can also reject targets by doing an `action: drop` if a label value matches a specified

When a target is dropped, the owning `scrape_config` will not process logs from that particular source. Other `scrape_configs` without the drop action reading from the same target may still use and forward logs from it to Loki. A common use case of `relabel_configs` is to transform an internal label such as `metakubernetes*` into an intermediate internal label such as `service`. The intermediate internal label may then be dropped based on value or transformed to a final external label, such as `job`. Drop the target if a label (`service` in the example) is empty: ```yaml action: drop regex: '' source_labels: service ``` Drop the target if any of the `source_labels` contain a value: ```yaml action: drop regex: .+ separator: '' source_labels: metakubernetespodlabelname metakubernetespodlabelapp ``` Persist an internal label by renaming it so it will be sent to Loki: ```yaml action: replace source_labels: metakubernetesnamespace target_label: namespace ``` Persist all Kubernetes pod labels by mapping them, like by mapping `meta_kubemetakubernetespodlabelfoo` to `foo`. ```yaml action: labelmap regex: metakubernetespodlabel(.+) ``` Additional reading: Promtail supports receiving messages from a TCP or UDP stream. Receiving syslog messages is defined in a `syslog` stanza: ```yaml scrape_configs: job_name: syslog syslog: listen_address: 0.0.0.0:1514 listen_protocol: tcp idle_timeout: 60s labelstructureddata: yes labels: job: "syslog" relabel_configs: sourcelabels: ['syslogmessage_hostname'] target_label: 'host' ``` The only required field in the syslog section is the `listen_address` field, where a valid network address must be provided. The default protocol for receiving messages is TCP. To change the protocol, the `listen_protocol` field can be changed to `udp`. Note, that UDP does not support TLS. The `idle_timeout` can help with cleaning up stale syslog connections. If `labelstructureddata` is set, in the syslog header will be translated to internal labels in the form of `syslogmessagesd<ID><KEY>`. The labels map defines a constant list of labels to add to every journal entry that Promtail reads. Note that it is recommended to deploy a dedicated syslog forwarder like syslog-ng or rsyslog in front of Promtail. The forwarder can take care of the various specifications and transports that exist (UDP, BSD syslog, ...). See recommended output configurations for and . When Promtail receives syslog messages, it brings in all header fields, parsed from the received message, prefixed with `syslog_` as internal labels. Like in the example above, the `syslogmessagehostname` field from the journal was transformed into a label called `host` through `relabel_configs`. See for more information. ``` destination d_loki { syslog("localhost" transport("tcp") port(<promtail_port>)); }; ``` For sending messages via TCP: ``` . action(type="omfwd" protocol="tcp" target="<promtailhost>" port="<promtailport>" Template="RSYSLOGSyslogProtocol23Format" TCPFraming="octet-counted" KeepAlive="on") ``` For sending messages via UDP: ``` . action(type="omfwd" protocol="udp" target="<promtailhost>" port="<promtailport>" Template="RSYSLOG_SyslogProtocol23Format") ``` On Windows Promtail supports reading from the event log. Windows event targets can be configured using the `windows_events` stanza: ```yaml scrape_configs: job_name: windows windows_events: useincomingtimestamp: false bookmark_path: "./bookmark.xml" eventlog_name: "Application" xpath_query: '*' labels: job: windows relabel_configs: source_labels: ['computer'] target_label: 'host' ``` When Promtail receives an event it will attach the `channel` and `computer` labels and serialize the event in json. You can relabel default labels via if required. Providing a path to a bookmark is mandatory, it will be used to persist the last event processed and allow resuming the target without skipping logs. Read the section for more information. See the stage for extracting data from the `message`.