Project: Grafana Loki (Observability and Analysis / Observability)
File: replace.md

title: replace menuTitle: description: The 'replace' Promtail pipeline stage. aliases: ../../../clients/promtail/stages/replace/ weight: The `replace` stage is a parsing stage that parses a log line using a regular expression and replaces the log line. Named capture groups in the regex support adding data into the extracted map. ```yaml replace: expression: <string> [source: <string>] [replace: <string>] ``` `expression` needs to be a [Go RE2 regex string](https://github.com/google/re2/wiki/Syntax). Every named capture group `(?P<name>re)` will be set into the `extracted` map. The name of the capture group will be used as the key in the extracted map. Because of how YAML treats backslashes in double-quoted strings, note that all backslashes in a regex expression must be escaped when using double quotes. For example, all of these are valid: `expression: \w*` `expression: '\w*'` `expression: "\\w*"` But these are not: `expression: \\w*` (only escape backslashes when using double quotes) `expression: '\\w*'` (only escape backslashes when using double quotes) `expression: "\w*"` (backslash must be escaped) Given the pipeline: ```yaml replace: expression: "password (\\S+)" replace: "" ``` And the log line: ``` 2019-01-01T01:00:00.000000001Z stderr P i'm a log message who has sensitive information with password xyz! ``` The log line becomes ``` 2019-01-01T01:00:00.000000001Z stderr P i'm a log message who has sensitive information with password ! ``` Given the pipeline: ```yaml json: expressions: level: msg: replace: expression: "\\S+ - \"POST (\\S+) .*" source: "msg" replace: "/loki/api/v1/push" ``` And the log line: ``` {"time":"2019-01-01T01:00:00.000000001Z", "level": "info", "msg":"11.11.11.11 - "POST /loki/api/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"} ``` The first stage would add the following key-value pairs into the `extracted` map: `time`: `2019-01-01T01:00:00.000000001Z` `level`: `info` `msg`: `11.11.11.11 - "POST /loki/api/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"` While the replace stage would then parse the value for `msg` in the extracted map and replaces the `msg` value. `msg` in extracted will now become `msg`: `11.11.11.11 - "POST /loki/api/v1/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"` Given the pipeline: ```yaml replace: expression: "^(\\S+) (\\S+) (\\S+) \\[([\\w:/]+\\s[+\\-]\\d{4})\\] \"(\\S+)\\s?(\\S+)?\\s?(\\S+)?\" (\\d{3}|-) (\\d+|-)\\s?\"?([^\"])\"?\\s?\"?([^\"])?\"?$" replace: '{{ if eq .Value "200" }}{{ Replace .Value "200" "HttpStatusOk" -1 }}{{ else }}{{ .Value | ToUpper }}{{ end }}' ``` And the log line: ``` 11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6" ``` The replace stage parses the log line and if the captured group has a value `200` it replaces the value to `HttpStatusOk`. The log line would become ```

- frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" HttpStatusOk 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6" ``` To obfuscate sensitive data, you can combine the `replace` stage with the `Hash` template method. ```yaml replace: expression: '([0-9]{3}-[0-9]{2}-[0-9]{4})' replace: 'SSN{{ .Value | Hash "salt" }}*' replace: expression: '(\d{1,3}[.]\d{1,3}[.]\d{1,3}[.]\d{1,3})' replace: 'IP4{{ .Value | Hash "salt" }}*' replace: expression: '([\w\.=-]+@[\w\.-]+\.[\w]{2,64})' replace: 'email{{ .Value | Hash "salt" }}*' replace: expression: '((?:\d[ -]*?){13,16})' replace: 'creditcard{{ .Value | Hash "salt" }}*' ``` Given the pipeline: ```yaml replace: expression: "^(?P<ip>\\S+) (?P<identd>\\S+) (?P<user>\\S+) \\[(?P<timestamp>[\\w:/]+\\s[+\\-]\\d{4})\\] \"(?P<action>\\S+)\\s?(?P<path>\\S+)?\\s?(?P<protocol>\\S+)?\" (?P<status>\\d{3}|-) (?P<size>\\d+|-)\\s?\"?(?P<referer>[^\"])\"?\\s?\"?(?P<useragent>[^\"])?\"?$" replace: '{{ .Value | ToUpper }}' ``` And the log line: ``` 11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6" ``` The replace stage parses the log line and replaces the value of all named captured groups to upper case. The named captured groups will be extracted to `ip`: `11.11.11.11` `identd`: `-` `user`: `FRANK` `timestamp`: `25/JAN/2000:14:00:01 -0500` `action`: `GET` `path`: `/1986.JS` `protocol`: `HTTP/1.1` `status`: `200` `size`: `932` `referer`: `-` `useragent`: `MOZILLA/5.0 (WINDOWS; U; WINDOWS NT 5.1; DE; RV:1.9.1.7) GECKO/20091221 FIREFOX/3.5.7 GTB6"` The log line would become ``` 11.11.11.11 - FRANK [25/JAN/2000:14:00:01 -0500] "GET /1986.JS HTTP/1.1" 200 932 "-" "MOZILLA/5.0 (WINDOWS; U; WINDOWS NT 5.1; DE; RV:1.9.1.7) GECKO/20091221 FIREFOX/3.5.7 GTB6" ``` Given the pipeline: ```yaml replace: expression: "^(?P<ip>\\S+) (?P<identd>\\S+) (\\S+) \\[(?P<timestamp>[\\w:/]+\\s[+\\-]\\d{4})\\] \"(?P<action>\\S+)\\s?(?P<path>\\S+)?\\s?(?P<protocol>\\S+)?\" (?P<status>\\d{3}|-) (?P<size>\\d+|-)\\s?\"?(?P<referer>[^\"])\"?\\s?\"?(?P<useragent>[^\"])?\"?$" replace: '{{ .Value | ToUpper }}' ``` And the log line: ``` 11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6" ``` The replace stage parses the log line and replaces the value of all named captured groups to upper case. The named captured groups will be extracted to. Observe here that `user` is not extracted since it was just `(\\S+)` and not a named captured group like this `(?P<user>\\S+)` `ip`: `11.11.11.11` `identd`: `-` `timestamp`: `25/JAN/2000:14:00:01 -0500` `action`: `GET` `path`: `/1986.JS` `protocol`: `HTTP/1.1` `status`: `200` `size`: `932` `referer`: `-` `useragent`: `MOZILLA/5.0 (WINDOWS; U; WINDOWS NT 5.1; DE; RV:1.9.1.7) GECKO/20091221 FIREFOX/3.5.7 GTB6"` The log line would become ``` 11.11.11.11 - FRANK [25/JAN/2000:14:00:01 -0500] "GET /1986.JS HTTP/1.1" 200 932 "-" "MOZILLA/5.0 (WINDOWS; U; WINDOWS NT 5.1; DE; RV:1.9.1.7) GECKO/20091221 FIREFOX/3.5.7 GTB6" ``` Given the pipeline: ```yaml replace: expression: "11.11.11.11 - (\\S+\\s)" replace: "" ``` And the log line: ``` 11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6" ``` The log line becomes ``` 11.11.11.11 - [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6" ```