Project: Grafana Loki (Observability and Analysis / Observability)
File: pipelines.md

title: Pipelines menuTitle: description: How to use Promtail pipelines to transform single log lines, labels, and timestamps. aliases: ../../clients/promtail/pipelines/ weight: 600 A detailed look at how to set up Promtail to process your log lines, including extracting metrics and labels. A pipeline is used to transform a single log line, its labels, and its timestamp. A pipeline is comprised of a set of stages. There are 4 types of stages: Parsing stages parse the current log line and extract data out of it. The extracted data is then available for use by other stages. Transform stages transform extracted data from previous stages. Action stages take extracted data from previous stages and do something with them. Actions can: Add or modify existing labels to the log line Change the timestamp of the log line Change the content of the log line Create a metric based on the extracted data Filtering stages optionally apply a subset of stages or drop entries based on some condition. Typical pipelines will start with a parsing stage (such as a or stage) to extract data from the log line. Then, a series of action stages will be present to do something with that extracted data. The most common action stage will be a stage to turn extracted data into a label. A common stage will also be the stage to selectively apply stages or drop entries based on a . Note that pipelines can not currently be used to deduplicate logs; Grafana Loki will receive the same log line multiple times if, for example: Two scrape configs read from the same file Duplicate log lines in a file are sent through a pipeline. Deduplication is not done. However, Loki will perform some deduplication at query time for logs that have the exact same nanosecond timestamp, labels, and log contents. This documented example gives a good glimpse of what you can achieve with a pipeline: ```yaml scrape_configs: job_name: kubernetes-pods-name kubernetessdconfigs: .... pipeline_stages: match: selector: '{name="promtail"}' stages: regex: expression: '.level=(?P<level>[a-zA-Z]+).ts=(?P<timestamp>[T\d-:.Z]).component=(?P<component>[a-zA-Z]+)' labels: level: component: timestamp: format: RFC3339Nano source: timestamp match: selector: '{name="nginx"} |= "GET"' stages: regex: expression:

output: source: output match: selector: '{name="jaeger-agent"}' stages: json: expressions: level: level labels: level: job_name: kubernetes-pods-app kubernetessdconfigs: .... pipeline_stages: match: selector: '{app=~"grafana|prometheus"}' stages: regex: expression: ".(lvl|level)=(?P<level>[a-zA-Z]+).(logger|component)=(?P<component>[a-zA-Z]+)" labels: level: component: match: selector: '{app="some-app"} != "info"' stages: regex: expression: ".(?P<panic>panic: .)" metrics: panic_total: type: Counter description: "total count of panic" source: panic config: action: inc ``` The following sections further describe the types that are accessible to each stage (although not all may be used): The current set of labels for the log line. Initialized to be the set of labels that were scraped along with the log line. The label set is only modified by an action stage, but filtering stages read from it. The final label set will be index by Loki and can be used for queries. A collection of key-value pairs extracted during a parsing stage. Subsequent stages operate on the extracted map, either transforming them or taking action with them. At the end of a pipeline, the extracted map is discarded; for a parsing stage to be useful, it must always be paired with at least one action stage. The extracted map is initialized with the same set of initial labels that were scraped along with the log line. This initial data allows for taking action on the values of labels inside pipeline stages that only manipulate the extracted map. For example, log entries tailed from files have the label `filename` whose value is the file path that was tailed. When a pipeline executes for that log entry, the initial extracted map would contain `filename` using the same value as the label. The current timestamp for the log line. Action stages can modify this value. If left unset, it defaults to the time when the log was scraped. The final value for the timestamp is sent to Loki. The current log line, represented as text. Initialized to be the text that Promtail scraped. Action stages can modify this value. The final value for the log line is sent to Loki as the text content for the given log entry. Refer to the for the schema on the various supported stages supported.