Project: Grafana Loki (Observability and Analysis / Observability)
File: multiline.md

title: multiline menuTitle: description: The 'multiline' Promtail pipeline stage. aliases: ../../../clients/promtail/stages/multiline/ weight: The `multiline` stage merges multiple lines into a multiline block before passing it on to the next stage in the pipeline. A new block is identified by the `firstline` regular expression. Any line that does not match the expression is considered to be part of the block of the previous match. ```yaml multiline: firstline: <string> maxwaittime: <duration> max_lines: <integer> ``` Consider these logs from a simple service. ``` [2020-12-03 11:36:20] "GET /hello HTTP/1.1" 200 - [2020-12-03 11:36:23] ERROR in app: Exception on /error [GET] Traceback (most recent call last): File "/home/pallets/.pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app response = self.fulldispatchrequest() File "/home/pallets/.pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1952, in fulldispatchrequest rv = self.handleuserexception(e) File "/home/pallets/.pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1821, in handleuserexception reraise(exctype, excvalue, tb) File "/home/pallets/.pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise raise value File "/home/pallets/.pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1950, in fulldispatchrequest rv = self.dispatch_request() File "/home/pallets/.pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functions File "/home/pallets/src/deployment_tools/hello.py", line 10, in error raise Exception("Sorry, this route always breaks") Exception: Sorry, this route always breaks [2020-12-03 11:36:23] "GET /error HTTP/1.1" 500 - [2020-12-03 11:36:26] "GET /hello HTTP/1.1" 200 - [2020-12-03 11:36:27] "GET /hello HTTP/1.1" 200 - ``` We would like to collapse all lines of the traceback into one multiline block. In this example, all blocks start with a timestamp in brackets. Thus, we configure a `multiline` stage with the `firstline` regular expression `^\[\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2}\]`. This will match the start of the traceback, but not the following lines until `Exception: Sorry, this route always breaks`. These will be part of a multiline block and one log entry in Loki. ```yaml multiline: firstline: '^\[\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2}\]' maxwaittime: 3s regex: expression: '^(?P<time>\[\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2}\]) (?P<message>(?s:.*))$' ``` The example assumed you had no control over the log format. Thus, it required a more elaborate regular expression to match the first line. If you can control the log format of the system under observation, we can simplify the first line matching. This time we are looking at the logs of a simple . ``` [2021-01-07 14:17:43,494] [DEBUG] [akka.io.TcpListener] [HelloAkkaHttpServer-akka.actor.default-dispatcher-26] [akka://HelloAkkaHttpServer/system/IO-TCP/selectors/$a/0] - New connection accepted [2021-01-07 14:17:43,499] [ERROR] [akka.actor.ActorSystemImpl] [HelloAkkaHttpServer-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(HelloAkkaHttpServer)] - Error during processing of request: 'oh no! oh is unknown'. Completing with 500 Internal Server Error response. To change default exception handling behavior, provide a custom ExceptionHandler. java.lang.Exception: oh no! oh is unknown at com.grafana.UserRoutes.$anonfun$userRoutes$6(UserRoutes.scala:28) at akka.http.scaladsl.server.Directive$.$anonfun$addByNameNullaryApply$2(Directive.scala:166) at akka.http.scaladsl.server.ConjunctionMagnet$$anon$2.$anonfun$apply$3(Directive.scala:234) at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:68) at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161) at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47) at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:40) ... ``` At first sight these seem be like the others. Let's look at the log format. ```xml <configuration> <appender name="FILE" class="ch.qos.logback.core.FileAppender"> <file>crasher.log</file> <append>true</append> <encoder> <pattern>&ZeroWidthSpace;[%date{ISO8601}] [%level] [%logger] [%thread] [%X{akkaSource}] - %msg%n</pattern> </encoder> </appender> <appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender"> <queueSize>1024</queueSize> <neverBlock>true</neverBlock> <appender-ref ref="STDOUT" /> </appender> <root level="DEBUG"> <appender-ref ref="ASYNC"/> </root> </configuration> ``` There is nothing special for a configuration except for `&ZeroWidthSpace;` at the beginning of each log line. This is the HTML-code for the character. It makes identifying first lines much simpler and is not visible. Thus it will not change the view of the log. The new first line matching regular expression is then `\x{200B}\[`. `200B` is the Unicode code point for the zero-width space character. ```yaml multiline: firstline: '^\x{200B}\[' maxwaittime: 3s ``` Zero-width space might not suite everyone. Any special character that is unlikely to be part of your regular logs should do just fine.