Project: VictoriaMetrics (Observability and Analysis / Observability)
File: LogsQL.md

sort: 5 weight: 5 title: LogsQL menu: docs: parent: "victorialogs" weight: 5 aliases: /VictoriaLogs/LogsQL.html LogsQL is a simple yet powerful query language for . It provides the following features: Full-text search across . See , and . Ability to combine filters into arbitrary complex . Ability to extract structured fields from unstructured logs at query time. See . Ability to calculate various stats over the selected log entries. See . If you aren't familiar with VictoriaLogs, then start with . Then follow these docs: . . . The simplest LogsQL query is just a , which must be found in the . For example, the following query finds all the logs with `error` word: ```logsql error ``` If the queried clashes with LogsQL keywords, then just wrap it into quotes. For example, the following query finds all the log messages with `and` : ```logsql "and" ``` It is OK to wrap any word into quotes. For example: ```logsql "error" ``` Moreover, it is possible to wrap phrases containing multiple words in quotes. For example, the following query finds log messages with the `error: cannot find file` phrase: ```logsql "error: cannot find file" ``` Queries above match logs with any , e.g. they may return logs from the previous year alongside recently ingested logs. Usually logs from the previous year aren't so interesting comparing to the recently ingested logs. So it is recommended adding to the query. For example, the following query returns logs with the `error` , which were ingested into VictoriaLogs during the last 5 minutes: ```logsql error AND _time:5m ``` This query consists of two joined with `AND` : The filter on the `error` . The filter on the . The `AND` operator means that the must match both filters in order to be selected. Typical LogsQL query constists of multiple joined with `AND` operator. It may be tiresome typing and then reading all these `AND` words. So LogsQL allows omitting `AND` words. For example, the following query is equivalent to the query above: ```logsql error _time:5m ``` The query returns all the by default. See . Suppose the query above selects too many rows because some buggy app pushes invalid error logs to VictoriaLogs. Suppose the app adds `buggy_app` to every log line. Then the following query removes all the logs from the buggy app, allowing us paying attention to the real errors: ```logsql time:5m error NOT buggyapp ``` This query uses `NOT` for removing log lines from the buggy app. The `NOT` operator is used frequently, so it can be substituted with `!` char. So the following query is equivalent to the previous one: ```logsql time:5m error !buggyapp ``` Suppose another buggy app starts pushing invalid error logs to VictoriaLogs - it adds `foobar` to every emitted log line. No problems - just add `!foobar` to the query in order to remove these buggy logs: ```logsql time:5m error !buggyapp !foobar ``` This query can be rewritten to more clear query with the `OR` inside parentheses: ```logsql time:5m error !(buggyapp OR foobar) ``` Note that the parentheses are required here, since otherwise the query won't return the expected results. The query `error !buggyapp OR foobar` is interpreted as `(error AND NOT buggyapp) OR foobar`. This query may return error logs from the buggy app if they contain `foobar` . This query also continues returning all the error logs from the second buggy app. This is because of different priorities for `NOT`, `AND` and `OR` operators. Read for more

There is no need in remembering all these priority rules - just wrap the needed query parts into explicit parentheses if you aren't sure in priority rules. As an additional bonus, explicit parentheses make queries easier to read and maintain. Queries above assume that the `error` is stored in the . This word can be stored in other such as `log.level`. How to select error logs in this case? Just add the `log.level:` prefix in front of the `error` word: ```logsq time:5m log.level:error !(buggyapp OR foobar) ``` The field name can be wrapped into quotes if it contains special chars or keywords, which may clash with LogsQL syntax. Any also can be wrapped into quotes. So the following query is equivalent to the previous one: ```logsql "time":"5m" "log.level":"error" !("buggyapp" OR "foobar") ``` What if the application identifier - such as `buggyapp` and `foobar` - is stored in the `app` field? Correct - just add `app:` prefix in front of `buggyapp` and `foobar`: ```logsql time:5m log.level:error !(app:buggyapp OR app:foobar) ``` The query can be simplified by moving the `app:` prefix outside the parentheses: ```logsql time:5m log.level:error !app:(buggyapp OR foobar) ``` The `app` field uniquely identifies the application instance if a single instance runs per each unique `app`. In this case it is recommended associating the `app` field with during . This usually improves both compression rate and query performance when querying the needed streams via . If the `app` field is associated with the log stream, then the query above can be rewritten to more performant one: ```logsql time:5m log.level:error stream:{app!~"buggy_app|foobar"} ``` This query completely skips scanning for logs from `buggy_app` and `foobar` apps, thus significantly reducing disk read IO and CPU time needed for performing the query. Finally, it is recommended reading . Now you are familiar with LogsQL basics. Read if you want to continue learning LogsQL. LogsQL splits all the into words delimited by non-word chars such as whitespace, parens, punctuation chars, etc. For example, the `foo: (bar,"")!` string is split into `foo`, `bar` and `` words. Words can contain arbitrary chars. These words are taken into account by full-text search filters such as , and . LogsQL query must contain for selecting the matching logs. At least a single filter is required. For example, the following query selects all the logs for the last 5 minutes by using : ```logsql _time:5m ``` Additionally to filters, LogQL query may contain arbitrary mix of optional actions for processing the selected logs. These actions are delimited by `|` and are known as `pipes`. For example, the following query uses for returning the number of with the `error` for the last 5 minutes: ```logsql _time:5m error | stats count() errors ``` See . LogsQL supports various filters for searching for log messages (see below). They can be combined into arbitrary complex queries via . Filters are applied to by default. If the filter must be applied to other , then its' name followed by the colon must be put in front of the filter. For example, if `error` must be applied to the `log.level` field, then use `log.level:error` query. Field names and filter args can be put into quotes if they contain special chars, which may clash with LogsQL syntax. LogsQL supports quoting via double quotes `"`, single quotes `'` and backticks: ```logsql "some 'field':123":i('some("value")') AND `other"value'` ``` If doubt, it is recommended quoting field names and filter

The list of LogsQL filters: - matches logs with in the given time range - matches logs, which belong to the given - matches logs with the given - matches logs with the given phrase - matches logs with the given word prefix or phrase prefix - matches logs without the given - matches logs with the given non-empty - matches logs with the exact value - matches logs starting with the given prefix - matches logs with one of the specified exact values - matches logs with the given case-insensitive word, phrase or prefix - matches logs with the given sequence of words or phrases - matches logs for the given regexp - matches logs with numeric in the given range - matches logs with ip address in the given range - matches logs with in the given string range - matches logs with of the given length range - allows combining other filters VictoriaLogs scans all the logs per each query if it doesn't contain the filter on . It uses various optimizations in order to accelerate full scan queries without the `_time` filter, but such queries can be slow if the storage contains large number of logs over long time range. The easiest way to optimize queries is to narrow down the search with the filter on . For example, the following query returns ingested into VictoriaLogs during the last hour, which contain the `error` : ```logsql _time:1h AND error ``` The following formats are supported for `_time` filter: `_time:duration` matches logs with timestamps on the time range `(now-duration, now]`. Examples: `_time:5m` - returns logs for the last 5 minutes `_time:2.5d15m42.345s` - returns logs for the last 2.5 days, 15 minutes and 42.345 seconds `_time:1y` - returns logs for the last year `time:YYYY-MM-DD` - matches all the logs for the particular day by UTC. For example, `time:2023-04-25` matches logs on April 25, 2023 by UTC. `time:YYYY-MM` - matches all the logs for the particular month by UTC. For example, `time:2023-02` matches logs on February, 2023 by UTC. `time:YYYY` - matches all the logs for the particular year by UTC. For example, `time:2023` matches logs on 2023 by UTC. `time:YYYY-MM-DDTHH` - matches all the logs for the particular hour by UTC. For example, `time:2023-04-25T22` matches logs on April 25, 2023 at 22 hour by UTC. `time:YYYY-MM-DDTHH:MM` - matches all the logs for the particular minute by UTC. For example, `time:2023-04-25T22:45` matches logs on April 25, 2023 at 22:45 by UTC. `time:YYYY-MM-DDTHH:MM:SS` - matches all the logs for the particular second by UTC. For example, `time:2023-04-25T22:45:59` matches logs on April 25, 2023 at 22:45:59 by UTC. `time:[mintime, maxtime]` - matches logs on the time range `[mintime, maxtime]`, including both `mintime` and `max_time`. The `mintime` and `maxtime` can contain any format specified . For example, `time:[2023-04-01, 2023-04-30]` matches logs for the whole April, 2023 by UTC, e.g. it is equivalent to `time:2023-04`. `time:[mintime, maxtime)` - matches logs on the time range `[mintime, maxtime)`, not including `maxtime`. The `mintime` and `maxtime` can contain any format specified . For example, `time:[2023-02-01, 2023-03-01)` matches logs for the whole February, 2023 by UTC, e.g. it is equivalent to `time:2023-02`. It is possible to specify time zone offset for all the absolute time formats by appending `+hh:mm` or `-hh:mm` suffix. For example, `_time:2023-04-25+05:30` matches all the logs on April 25, 2023 by India time zone, while `_time:2023-02-07:00` matches all the logs on February, 2023 by California time zone. It is possible to specify generic offset for the selected time range by appending `offset` after the `_time`

Examples: `_time:5m offset 1h` matches logs on the time range `(now-1h5m, now-1h]`. `_time:2023-07 offset 5h30m` matches logs on July, 2023 by UTC with offset 5h30m. `_time:[2023-02-01, 2023-03-01) offset 1w` matches logs the week before the time range `[2023-02-01, 2023-03-01)` by UTC. Performance tips: It is recommended specifying the smallest possible time range during the search, since it reduces the amounts of log entries, which need to be scanned during the query. For example, `time:1h` is usually faster than `time:5h`. While LogsQL supports arbitrary number of `_time:...` filters at any level of , it is recommended specifying a single `_time` filter at the top level of the query. See . See also: VictoriaLogs provides an optimized way to select log entries, which belong to particular . This can be done via `_stream:{...}` filter. The `{...}` may contain arbitrary over fields associated with . For example, the following query selects with `app` field equal to `nginx`: ```logsql _stream:{app="nginx"} ``` This query is equivalent to the following query, but the upper query usually works much faster: ```logsql app:exact("nginx") ``` Performance tips: It is recommended using the most specific `_stream:{...}` filter matching the smallest number of log streams, which needs to be scanned by the rest of filters in the query. While LogsQL supports arbitrary number of `_stream:{...}` filters at any level of , it is recommended specifying a single `_stream:...` filter at the top level of the query. See . See also: The simplest LogsQL query consists of a single to search in log messages. For example, the following query matches with `error` inside them: ```logsql error ``` This query matches the following : `error` `an error happened` `error: cannot open file` This query doesn't match the following log messages: `ERROR`, since the filter is case-sensitive by default. Use `i(error)` for this case. See for details. `multiple errors occurred`, since the `errors` word doesn't match `error` word. Use `error*` for this case. See for details. By default the given is searched in the . Specify the in front of the word and put a colon after it if it must be searched in the given field. For example, the following query returns log entries containing the `error` in the `log.level` field: ```logsql log.level:error ``` Both the field name and the word in the query can contain arbitrary -encoded chars. For example: ```logsql : ``` Both the field name and the word in the query can be put inside quotes if they contain special chars, which may clash with the query syntax. For example, the following query searches for the ip `1.2.3.45` in the field `ip:remote`: ```logsql "ip:remote":"1.2.3.45" ``` See also: Is you need to search for log messages with the specific phrase inside them, then just wrap the phrase in quotes. The phrase can contain any chars, including whitespace, punctuation, parens, etc. They are taken into account during the search. For example, the following query matches with `ssh: login fail` phrase inside them: ```logsql "ssh: login fail" ``` This query matches the following : `ERROR: ssh: login fail for user "foobar"` `ssh: login fail!` This query doesn't match the following log messages: `ssh login fail`, since the message misses `:` char just after the `ssh`. Use `seq("ssh", "login", "fail")` query if log messages with the sequence of these words must be found. See for details. `login fail: ssh error`, since the message doesn't contain the full phrase requested in the

If you need matching a message with all the listed in the query, then use `ssh AND login AND fail` query. See for details. `ssh: login failed`, since the message ends with `failed` instead of `fail` word. Use `"ssh: login fail"*` query for this case. See for details. `SSH: login fail`, since the `SSH` word is in capital letters. Use `i("ssh: login fail")` for case-insensitive search. See for details. By default the given phrase is searched in the . Specify the in front of the phrase and put a colon after it if it must be searched in the given field. For example, the following query returns log entries containing the `cannot open file` phrase in the `event.original` field: ```logsql event.original:"cannot open file" ``` Both the field name and the phrase can contain arbitrary -encoded chars. For example: ```logsql :" " ``` The field name can be put inside quotes if it contains special chars, which may clash with the query syntax. For example, the following query searches for the `cannot open file` phrase in the field `some:message`: ```logsql "some:message":"cannot open file" ``` See also: If you need to search for log messages with / phrases containing some prefix, then just add `*` char to the end of the / phrase in the query. For example, the following query returns , which contain with `err` prefix: ```logsql err* ``` This query matches the following : `err: foobar` `cannot open file: error occurred` This query doesn't match the following log messages: `Error: foobar`, since the `Error` starts with capital letter. Use `i(err*)` for this case. See for details. `fooerror`, since the `fooerror` doesn't start with `err`. Use `re("err")` for this case. See for details. Prefix filter can be applied to . For example, the following query matches containing phrases with `unexpected fail` prefix: ```logsql "unexpected fail"* ``` This query matches the following : `unexpected fail: IO error` `error:unexpected failure` This query doesn't match the following log messages: `unexpectedly failed`, since the `unexpectedly` doesn't match `unexpected` . Use `unexpected AND fail` for this case. See for details. `failed to open file: unexpected EOF`, since `failed` occurs before the `unexpected` word. Use `unexpected AND fail*` for this case. See for details. By default the prefix filter is applied to the . Specify the needed in front of the prefix filter in order to apply it to the given field. For example, the following query matches `log.level` field containing any word with the `err` prefix: ```logsql log.level:err* ``` If the field name contains special chars, which may clash with the query syntax, then it may be put into quotes in the query. For example, the following query matches `log:level` field containing any word with the `err` prefix. ```logsql "log:level":err* ``` Performance tips: Prefer using and combined via instead of prefix filter. Prefer moving and in front of prefix filter when using . See . See also: Sometimes it is needed to find log entries without the given . This can be performed with `log_field:""` syntax. For example, the following query matches log entries without `host.hostname` field: ```logsql host.hostname:"" ``` See also: Sometimes it is needed to find log entries containing any non-empty value for the given . This can be performed with `log_field:*` syntax. For example, the following query matches log entries with non-empty `host.hostname` field: ```logsql host.hostname:* ``` See also: The and return , which contain the given word or phrase inside them. The message may contain additional text other than the requested word or

If you need searching for log messages or with the exact value, then use the `exact(...)` filter. For example, the following query returns log messages wih the exact value `fatal error: cannot find /foo/bar`: ```logsql exact("fatal error: cannot find /foo/bar") ``` The query doesn't match the following log messages: `fatal error: cannot find /foo/bar/baz` or `some-text fatal error: cannot find /foo/bar`, since they contain an additional text other than the specified in the `exact()` filter. Use `"fatal error: cannot find /foo/bar"` query in this case. See for details. `FATAL ERROR: cannot find /foo/bar`, since the `exact()` filter is case-sensitive. Use `i("fatal error: cannot find /foo/bar")` in this case. See for details. By default the `exact()` filter is applied to the . Specify the in front of the `exact()` filter and put a colon after it if it must be searched in the given field. For example, the following query returns log entries with the exact `error` value at `log.level` field: ```logsql log.level:exact("error") ``` Both the field name and the phrase can contain arbitrary -encoded chars. For example: ```logsql log.:exact("") ``` The field name can be put inside quotes if it contains special chars, which may clash with the query syntax. For example, the following query matches the `error` value in the field `log:level`: ```logsql "log:level":exact("error") ``` See also: Sometimes it is needed to find log messages starting with some prefix. This can be done with the `exact("prefix"*)` filter. For example, the following query matches log messages, which start from `Processing request` prefix: ```logsql exact("Processing request"*) ``` This filter matches the following : `Processing request foobar` `Processing requests from ...` It doesn't match the following log messages: `processing request foobar`, since the log message starts with lowercase `p`. Use `exact("processing request") OR exact("Processing request")` query in this case. See for details. `start: Processing request`, since the log message doesn't start with `Processing request`. Use `"Processing request"` query in this case. See for details. By default the `exact()` filter is applied to the . Specify the in front of the `exact()` filter and put a colon after it if it must be searched in the given field. For example, the following query returns log entries with `log.level` field, which starts with `err` prefix: ```logsql log.level:exact("err"*) ``` Both the field name and the phrase can contain arbitrary -encoded chars. For example: ```logsql log.:exact(""*) ``` The field name can be put inside quotes if it contains special chars, which may clash with the query syntax. For example, the following query matches `log:level` values starting with `err` prefix: ```logsql "log:level":exact("err"*) ``` See also: Sometimes it is needed to locate log messages with a field containing one of the given values. This can be done with multiple combined into a single . For example, the following query matches log messages with `log.level` field containing either `error` or `fatal` exact values: ```logsql log.level:(exact("error") OR exact("fatal")) ``` While this solution works OK, LogsQL provides simpler and faster solution for this case - the `in()` filter. ```logsql log.level:in("error", "fatal") ``` It works very fast for long lists passed to `in()`. The future VictoriaLogs versions will allow passing arbitrary into `in()` filter. For example, the following query selects all the logs for the last hour for users, who visited pages with `admin` in the `path` during the last day: ```logsql time:1h AND userid:in(time:1d AND path:admin | fields userid) ``` See the for details. See also: Case-insensitive filter can be applied to any word, phrase or prefix by wrapping the corresponding , or into

For example, the following query returns log messages with `error` word in any case: ```logsql i(error) ``` The query matches the following : `unknown error happened` `ERROR: cannot read file` `Error: unknown arg` `An ErRoR occured` The query doesn't match the following log messages: `FooError`, since the `FooError` has superflouos prefix `Foo`. Use `re("(?i)error")` for this case. See for details. `too many Errors`, since the `Errors` has superflouos suffix `s`. Use `i(error*)` for this case. By default the `i()` filter is applied to the . Specify the needed in front of the filter in order to apply it to the given field. For example, the following query matches `log.level` field containing `error` in any case: ```logsql log.level:i(error) ``` If the field name contains special chars, which may clash with the query syntax, then it may be put into quotes in the query. For example, the following query matches `log:level` field containing `error` in any case. ```logsql "log:level":i("error") ``` Performance tips: Prefer using case-sensitive filter over case-insensitive filter. Prefer moving , and in front of case-sensitive filter when using . See . See also: Sometimes it is needed to find with or phrases in a particular order. For example, if log messages with `error` word followed by `open file` phrase must be found, then the following LogsQL query can be used: ```logsql seq("error", "open file") ``` This query matches `some error: cannot open file /foo/bar` message, since the `open file` phrase goes after the `error` . The query doesn't match the `cannot open file: error` message, since the `open file` phrase is located in front of the `error` . If you need matching log messages with both `error` word and `open file` phrase, then use `error AND "open file"` query. See for details. By default the `seq()` filter is applied to the . Specify the needed in front of the filter in order to apply it to the given field. For example, the following query matches `event.original` field containing `(error, "open file")` sequence: ```logsql event.original:seq(error, "open file") ``` If the field name contains special chars, which may clash with the query syntax, then it may be put into quotes in the query. For example, the following query matches `event:original` field containing `(error, "open file")` sequence: ```logsql "event:original":seq(error, "open file") ``` See also: LogsQL supports regular expression filter with via `re(...)` expression. For example, the following query returns all the log messages containing `err` or `warn` susbstrings: ```logsql re("err|warn") ``` The query matches the following , which contain either `err` or `warn` substrings: `error: cannot read data` `2 warnings have been raised` `data trasferring finished` The query doesn't match the following log messages: `ERROR: cannot open file`, since the `ERROR` word is in uppercase letters. Use `re("(?i)(err|warn)")` query for case-insensitive regexp search. See for details. See also . `it is warmer than usual`, since it doesn't contain neither `err` nor `warn` substrings. By default the `re()` filter is applied to the . Specify the needed in front of the filter in order to apply it to the given field. For example, the following query matches `event.original` field containing either `err` or `warn` substrings: ```logsql event.original:re("err|warn") ``` If the field name contains special chars, which may clash with the query syntax, then it may be put into quotes in the query. For example, the following query matches `event:original` field containing either `err` or `warn` substrings: ```logsql "event:original":re("err|warn") ``` Performance tips: Prefer combining simple with instead of using regexp

For example, the `re("error|warning")` query can be substituted with `error OR warning` query, which usually works much faster. Note that the `re("error|warning")` matches `errors` as well as `warnings` , while `error OR warning` matches only the specified . See also . Prefer moving the regexp filter to the end of the , so lightweighter filters are executed first. Prefer using `exact("some prefix"*)` instead of `re("^some prefix")`, since the works much faster than the `re()` filter. See . See also: If you need to filter log message by some field containing only numeric values, then the `range()` filter can be used. For example, if the `request.duration` field contains the request duration in seconds, then the following LogsQL query can be used for searching for log entries with request durations exceeding 4.2 seconds: ```logsql request.duration:range(4.2, Inf) ``` The lower and the upper bounds of the range are excluded by default. If they must be included, then substitute the corresponding parentheses with square brackets. For example: `range[1, 10)` includes `1` in the matching range `range(1, 10]` includes `10` in the matching range `range[1, 10]` includes `1` and `10` in the matching range The range boundaries can be expressed in the following forms: Hexadecimal form. For example, `range(0xff, 0xABCD)`. Binary form. Form example, `range(0b100110, 0b11111101)` Integer form with `` delimiters for better readability. For example, `range(1000, 2345678)`. Note that the `range()` filter doesn't match with non-numeric values alongside numeric values. For example, `range(1, 10)` doesn't match `the request took 4.2 seconds` , since the `4.2` number is surrounded by other text. Extract the numeric value from the message with `parse(msg, "the request took <requestduration> seconds")` and then apply the `range()` to the extracted `request_duration` field. Performance tips: It is better to query pure numeric instead of extracting numeric field from text field via at query time. See . See also: If you need to filter log message by some field containing only addresses such as `1.2.3.4`, then the `ipv4_range()` filter can be used. For example, the following query matches log entries with `user.ip` address in the range `[127.0.0.0 - 127.255.255.255]`: ```logsql user.ip:ipv4_range(127.0.0.0, 127.255.255.255) ``` The `ipv4_range()` accepts also IPv4 subnetworks in . For example, the following query is equivalent to the query above: ```logsql user.ip:ipv4_range("127.0.0.0/8") ``` If you need matching a single IPv4 address, then just put it inside `ipv4_range()`. For example, the following query matches `1.2.3.4` IP at `user.ip` : ```logsql user.ip:ipv4_range("1.2.3.4") ``` Note that the `ipv4range()` doesn't match a string with IPv4 address if this string contains other text. For example, `ipv4range("127.0.0.0/24")` doesn't match `request from 127.0.0.1: done` , since the `127.0.0.1` ip is surrounded by other text. Extract the IP from the message with `parse(_msg, "request from <ip>: done")` and then apply the `ipv4_range()` to the extracted `ip` field. Hints: If you need searching for containing the given `X.Y.Z.Q` IPv4 address, then `"X.Y.Z.Q"` query can be used. See for details. If you need searching for containing at least a single IPv4 address out of the given list, then `"ip1" OR "ip2" ... OR "ipN"` query can be used. See for details. If you need finding log entries with `ip` field in multiple ranges, then use `ip:(ipv4range(range1) OR ipv4range(range2) ... OR ipv4_range(rangeN))` query. See for details. Performance tips: It is better querying pure IPv4 instead of extracting IPv4 from text field via at query time. See . See also: If you need to filter log message by some field with string values in some range, then `string_range()` filter can be used. For example, the following LogsQL query matches log entries with

field starting from `A` and `B` chars: ```logsql user.name:string_range(A, C) ``` The `string_range()` includes the lower bound, while excluding the upper bound. This simplifies querying distinct sets of logs. For example, the `user.name:string_range(C, E)` would match `user.name` fields, which start from `C` and `D` chars. See also: If you need to filter log message by its length, then `len_range()` filter can be used. For example, the following LogsQL query matches with lengths in the range `[5, 10]` chars: ```logsql len_range(5, 10) ``` This query matches the following log messages, since their length is in the requested range: `foobar` `foo bar` This query doesn't match the following log messages: `foo`, since it is too short `foo bar baz abc`, sinc it is too long It is possible to use `inf` as the upper bound. For example, the following query matches with the length bigger or equal to 5 chars: ```logsql len_range(5, inf) ``` The range boundaries can be expressed in the following forms: Hexadecimal form. For example, `len_range(0xff, 0xABCD)`. Binary form. Form example, `len_range(0b100110, 0b11111101)` Integer form with `` delimiters for better readability. For example, `lenrange(1000, 2345_678)`. By default the `len_range()` is applied to the . Put the in front of the `len_range()` in order to apply the filter to the needed field. For example, the following query matches log entries with the `foo` field length in the range `[10, 20]` chars: ```logsql foo:len_range(10, 20) ``` See also: Simpler LogsQL can be combined into more complex filters with the following logical operations: `q1 AND q2` - matches common log entries returned by both `q1` and `q2`. Arbitrary number of can be combined with `AND` operation. For example, `error AND file AND app` matches , which simultaneously contain `error`, `file` and `app` . The `AND` operation is frequently used in LogsQL queries, so it is allowed to skip the `AND` word. For example, `error file app` is equivalent to `error AND file AND app`. `q1 OR q2` - merges log entries returned by both `q1` and `q2`. Aribtrary number of can be combined with `OR` operation. For example, `error OR warning OR info` matches , which contain at least one of `error`, `warning` or `info` . `NOT q` - returns all the log entries except of those which match `q`. For example, `NOT info` returns all the , which do not contain `info` . The `NOT` operation is frequently used in LogsQL queries, so it is allowed substituting `NOT` with `!` in queries. For example, `!info` is equivalent to `NOT info`. The `NOT` operation has the highest priority, `AND` has the middle priority and `OR` has the lowest priority. The priority order can be changed with parentheses. For example, `NOT info OR debug` is interpreted as `(NOT info) OR debug`, so it matches , which do not contain `info` , while it also matches messages with `debug` word (which may contain the `info` word). This is not what most users expect. In this case the query can be rewritten to `NOT (info OR debug)`, which correctly returns log messages without `info` and `debug` . LogsQL supports arbitrary complex logical queries with arbitrary mix of `AND`, `OR` and `NOT` operations and parentheses. By default logical filters apply to the unless the inner filters explicitly specify the needed via `field_name:filter` syntax. For example, `(error OR warn) AND host.hostname:host123` is interpreted as `(msg:error OR msg:warn) AND host.hostname:host123`. It is possible to specify a single for multiple filters with the following syntax: ```logsql field_name:(q1 OR q2 OR ... qN) ``` For example,

OR log.level:warning OR log.level:info` can be substituted with the shorter query: `log.level:(error OR warning OR info)`. Performance tips: VictoriaLogs executes logical operations from the left to the right, so it is recommended moving the most specific and the fastest filters (such as and ) to the left, while moving less specific and the slowest filters (such as and ) to the right. For example, if you need to find with the `error` word, which match some `/foo/(bar|baz)` regexp, it is better from performance PoV to use the query `error re("/foo/(bar|baz)")` instead of `re("/foo/(bar|baz)") error`. The most specific filter means that it matches the lowest number of log entries comparing to other filters. See . Additionally to , LogsQL query may contain arbitrary mix of '|'-delimited actions known as `pipes`. For example, the following query uses , and pipes for returning top 10 with the biggest number of logs during the last 5 minutes: ```logsql time:5m | stats by (stream) count() perstreamlogs | sort by (perstreamlogs desc) | limit 10 ``` LogsQL supports the following pipes: copies . deletes . selects the given set of . limits the number selected logs. skips the given number of selected logs. renames . sorts logs by the given . calculates various stats over the selected logs. returns unique log entires. If some must be copied, then `| copy src1 as dst1, ..., srcN as dstN` can be used. For example, the following query copies `host` field to `server` for logs over the last 5 minutes, so the output contains both `host` and `server` fields: ```logsq _time:5m | copy host as server ``` Multiple fields can be copied with a single `| copy ...` pipe. For example, the following query copies to `timestamp`, while is copied to `message`: ```logsql time:5m | copy time as timestmap, _msg as message ``` The `as` keyword is optional. `cp` keyword can be used instead of `copy` for convenience. For example, `time:5m | cp foo bar` is equivalent to `time:5m | copy foo as bar`. See also: If some must be deleted, then `| delete field1, ..., fieldN` can be used. For example, the following query deletes `host` and `app` fields from the logs over the last 5 minutes: ```logsql _time:5m | delete host, app ``` `del` and `rm` keywords can be used instead of `delete` for convenience. For example, `time:5m | del host` is equivalent to `time:5m | rm host` and `_time:5m | delete host`. See also: By default all the are returned in the response. It is possible to select the given set of log fields with `| fields field1, ..., fieldN` . For example, the following query selects only `host` and fields from logs for the last 5 minutes: ```logsq time:5m | fields host, msg ``` See also: If only a subset of selected logs must be processed, then `| limit N` can be used, where `N` can contain any . For example, the following query returns up to 100 logs over the last 5 minutes: ```logsql _time:5m | limit 100 ``` `head` keyword can be used instead of `limit` for convenience. For example, `time:5m | head 100` is equivalent to `time:5m | limit 100`. By default rows are selected in arbitrary order because of performance reasons, so the query above can return different sets of logs every time it is executed. can be used for making sure the logs are in the same order before applying `limit ...` to

See also: If some selected logs must be skipped after , then `| offset N` can be used, where `N` can contain any . For example, the following query skips the first 100 logs over the last 5 minutes after soring them by : ```logsql time:5m | sort by (time) | offset 100 ``` `skip` keyword can be used instead of `offset` keyword for convenience. For example, `time:5m | skip 10` is equivalent to `time:5m | offset 10`. Note that skipping rows without sorting has little sense, since they can be returned in arbitrary order because of performance reasons. Rows can be sorted with . See also: If some must be renamed, then `| rename src1 as dst1, ..., srcN as dstN` can be used. For example, the following query renames `host` field to `server` for logs over the last 5 minutes, so the output contains `server` field instead of `host` field: ```logsql _time:5m | rename host as server ``` Multiple fields can be renamed with a single `| rename ...` pipe. For example, the following query renames `host` to `instance` and `app` to `job`: ```logsql _time:5m | rename host as instance, app as job ``` The `as` keyword is optional. `mv` keyword can be used instead of `rename` keyword for convenience. For example, `time:5m | mv foo bar` is equivalent to `time:5m | rename foo as bar`. See also: By default logs are selected in arbitrary order because of performance reasons. If logs must be sorted, then `| sort by (field1, ..., fieldN)` can be used. The returned logs are sorted by the given using . For example, the following query returns logs for the last 5 minutes sorted by and then by : ```logsql time:5m | sort by (stream, _time) ``` Add `desc` after the given log field in order to sort in reverse order of this field. For example, the following query sorts log fields in reverse order of `requestdurationseconds` field: ```logsql time:5m | sort by (requestduration_seconds desc) ``` The reverse order can be applied globally via `desc` keyword after `by(...)` clause: ```logsql _time:5m | sort by (foo, bar) desc ``` Sorting of big number of logs can consume a lot of CPU time and memory. Sometimes it is enough to return the first `N` entries with the biggest or the smallest values. This can be done by adding `limit N` to the end of `sort ...` pipe. Such a query consumes lower amounts of memory when sorting big number of logs, since it keeps in memory only `N` log entries. For example, the following query returns top 10 log entries with the biggest values for the `request_duration` during the last hour: ```logsql time:1h | sort by (requestduration desc) limit 10 ``` If the first `N` sorted results must be skipped, then `offset N` can be added to `sort` pipe. For example, the following query skips the first 10 logs with the biggest `request_duration` , and then returns the next 20 sorted logs for the last 5 minutes: ```logsql time:1h | sort by (requestduration desc) offset 10 limit 20 ``` Note that sorting of big number of logs can be slow and can consume a lot of additional memory. It is recommended limiting the number of logs before sorting with the following approaches: Adding `limit N` to the end of `sort ...` pipe. Reducing the selected time range with . Using more specific , so they select less logs. Limiting the number of selected via . See also: `| uniq

pipe allows returning only unique results over the selected logs. For example, the following LogsQL query returns unique values for `ip` over logs for the last 5 minutes: ```logsql _time:5m | uniq by (ip) ``` It is possible to specify multiple fields inside `by(...)` clause. In this case all the unique sets for the given fields are returned. For example, the following query returns all the unique `(host, path)` pairs for the logs over the last 5 minutes: ```logsql _time:5m | uniq by (host, path) ``` Unique entries are stored in memory during query execution. Big number of unique selected entries may require a lot of memory. Sometimes it is enough to return up to `N` unique entries. This can be done by adding `limit N` after `by (...)` clause. This allows limiting memory usage. For example, the following query returns up to 100 unique `(host, path)` pairs for the logs over the last 5 minutes: ```logsql _time:5m | uniq by (host, path) limit 100 ``` See also: `| stats ...` pipe allows calculating various stats over the selected logs. For example, the following LogsQL query uses for calculating the number of logs for the last 5 minutes: ```logsql time:5m | stats count() logstotal ``` `| stats ...` pipe has the following basic format: ```logsql ... | stats statsfunc1(...) as resultname1, ... statsfuncN(...) as resultnameN ``` Where `statsfunc*` is any of the supported , while `resultname*` is the name of the log field to store the result of the corresponding stats function. The `as` keyword is optional. For example, the following query calculates the following stats for logs over the last 5 minutes: the number of logs with the help of ; the number of unique with the help of : ```logsql time:5m | stats count() logstotal, countuniq(stream) streams_total ``` See also: The following LogsQL syntax can be used for calculating independent stats per group of log fields: ```logsql ... | stats by (field1, ..., fieldM) statsfunc1(...) as resultname1, ... statsfuncN(...) as resultnameN ``` This calculates `stats_func*` per each `(field1, ..., fieldM)` group of . For example, the following query calculates the number of logs and unique ip addresses over the last 5 minutes, grouped by `(host, path)` fields: ```logsql time:5m | stats by (host, path) count() logstotal, countuniq(ip) ipstotal ``` The following syntax can be used for calculating stats grouped by time buckets: ```logsql ... | stats by (_time:step) statsfunc1(...) as resultname1, ... statsfuncN(...) as resultnameN ``` This calculates `stats_func*` per each `step` of field. The `step` can have any . For example, the following LogsQL query returns per-minute number of logs and unique ip addresses over the last 5 minutes: ``` time:5m | stats by (time:1m) count() logstotal, countuniq(ip) ips_total ``` Additionally, the following `step` values are supported: `nanosecond` - equals to `1ns` . `microsecond` - equals to `1s` . `millisecond` - equals to `1ms` . `second` - equals to `1s` . `minute` - equals to `1m` . `hour` - equalst to `1h` . `day` - equals to `1d` . `week` - equals to `1w` . `month` - equals to one month. It properly takes into account the number of days per each month. `year` - equals to one year. It properly takes into account the number of days per each year. VictoriaLogs stores values as in nanoseconds. This time corresponds to time zone. Sometimes it is needed calculating stats grouped by days or weeks at non-UTC timezone. This is possible with the following syntax: ```logsql

| stats by (time:step offset timezoneoffset) ... ``` For example, the following query calculates per-day number of logs over the last week, in `UTC+02:00` : ```logsql time:1w | stats by (time:1d offset 2h) count() logs_total ``` Every log field inside `| stats by (...)` can be bucketed in the same way at `_time` field in . Any can be used as `step` value for the bucket. For example, the following query calculates the number of requests for the last hour, bucketed by 10KB of `requestsizebytes` : ```logsql time:1h | stats by (requestsize_bytes:10KB) count() requests ``` Stats can be bucketed by containing via the `ipfieldname:/network_mask` syntax inside `by(...)` clause. For example, the following query returns the number of log entries per `/24` subnetwork extracted from the `ip` during the last 5 minutes: ```logsql time:5m | stats by (ip:/24) count() requestsper_subnet ``` LogsQL supports the following functions for : calculates the average value over the given numeric . calculates the number of log entries. calculates the number logs with empty . calculates the number of unique non-empty values for the given . calcualtes the maximum value over the given numeric . calcualtes the value over the given numeric . calculates the minumum value over the given numeric . calculates the given quantile for the given numeric . calculates the sum for the given numeric . calculates the sum of lengths for the given . returns unique non-empty values for the given . returns all the values for the given . `avg(field1, ..., fieldN)` calculates the average value across all the mentioned . Non-numeric values are ignored. For example, the following query returns the average value for the `duration` over logs for the last 5 minutes: ```logsql time:5m | stats avg(duration) avgduration ``` See also: `count()` calculates the number of selected logs. For example, the following query returns the number of logs over the last 5 minutes: ```logsql _time:5m | stats count() logs ``` It is possible calculating the number of logs with non-empty values for some with the `count(fieldName)` syntax. For example, the following query returns the number of logs with non-empty `username` field over the last 5 minutes: ```logsq time:5m | stats count(username) logswith_username ``` If multiple fields are enumerated inside `count()`, then it counts the number of logs with at least a single non-empty field mentioned inside `count()`. For example, the following query returns the number of logs with non-empty `username` or `password` over the last 5 minutes: ```logsql time:5m | stats count(username, password) logswithusernameor_password ``` See also: `count_empty(field1, ..., fieldN)` calculates the number of logs with empty `(field1, ..., fieldN)` tuples. For example, the following query calculates the number of logs with empty `username` during the last 5 minutes: ```logsql time:5m | stats countempty(username) logswithmissing_username ``` See also: `count_uniq(field1, ..., fieldN)` calculates the number of unique non-empty `(field1, ..., fieldN)` tuples. For example, the following query returns the number of unique non-empty values for `ip` over the last 5 minutes: ```logsql time:5m | stats countuniq(ip) ips ``` The following query returns the number of unique `(host, path)` pairs for the corresponding over the last 5 minutes: ```logsql time:5m | stats countuniq(host, path) uniquehostpath_pairs ``` Every unique value is stored in memory during query execution. Big number of unique values may require a lot of memory. Sometimes it is needed to know whether the number of unique values reaches some limit. In this case add `limit N` just after `count_uniq(...)` for limiting the number of counted unique values up to `N`, while limiting the maximum memory

For example, the following query counts up to `1000000` unique values for the `ip` field: ```logsql time:5m | stats countuniq(ip) limit 1000000 as ips1000_000 ``` See also: `max(field1, ..., fieldN)` calculates the maximum value across all the mentioned . Non-numeric values are ignored. For example, the following query returns the maximum value for the `duration` over logs for the last 5 minutes: ```logsql time:5m | stats max(duration) maxduration ``` See also: `median(field1, ..., fieldN)` calculates the value across the give numeric . For example, the following query return median for the `duration` over logs for the last 5 minutes: ```logsql time:5m | stats median(duration) medianduration ``` See also: `min(field1, ..., fieldN)` calculates the minimum value across all the mentioned . Non-numeric values are ignored. For example, the following query returns the minimum value for the `duration` over logs for the last 5 minutes: ```logsql time:5m | stats min(duration) minduration ``` See also: `quantile(phi, field1, ..., fieldN)` calculates `phi` over numeric values for the given . The `phi` must be in the range `0 ... 1`, where `0` means `0th` percentile, while `1` means `100th` percentile. For example, the following query calculates `50th`, `90th` and `99th` percentiles for the `requestdurationseconds` over logs for the last 5 minutes: ```logsql _time:5m | stats quantile(0.5, requestdurationseconds) p50, quantile(0.9, requestdurationseconds) p90, quantile(0.99, requestdurationseconds) p99 ``` See also: `sum(field1, ..., fieldN)` calculates the sum of numeric values across all the mentioned . For example, the following query returns the sum of numeric values for the `duration` over logs for the last 5 minutes: ```logsql time:5m | stats sum(duration) sumduration ``` See also: `sum_len(field1, ..., fieldN)` calculates the sum of lengths of all the values for the given . For example, the following query returns the sum of lengths of across all the logs for the last 5 minutes: ```logsql time:5m | stats sumlen(msg) messageslen ``` See also: `uniq_values(field1, ..., fieldN)` returns the unique non-empty values across the mentioned . The returned values are encoded in JSON array. The order of the returned values is arbitrary. For example, the following query returns unique non-empty values for the `ip` over logs for the last 5 minutes: ```logsql time:5m | stats uniqvalues(ip) unique_ips ``` Every unique value is stored in memory during query execution. Big number of unique values may require a lot of memory. Sometimes it is enough to return only a subset of unique values. In this case add `limit N` after `uniq_values(...)` in order to limit the number of returned unique values to `N`, while limiting the maximum memory usage. For example, the following query returns up to `100` unique values for the `ip` over the logs for the last 5 minutes. Note that arbitrary subset of unique `ip` values is returned every time: ```logsql time:5m | stats uniqvalues(ip) limit 100 as uniqueips100 ``` See also: `values(field1, ..., fieldN)` returns all the values (including empty values) for the mentioned . The returned values are encoded in JSON array. For example, the following query returns all the values for the `ip` over logs for the last 5 minutes: ```logsql _time:5m | stats values(ip) ips ``` See also: LogsQL will support the ability to select the given number of surrounding log lines for the selected log lines on a basis. See the for details. It is possible to perform various transformations on the at client side with `jq`, `awk`, `cut`, etc. Unix commands according to

LogsQL will support the following transformations for the log entries: Extracting the specified fields from text according to the provided pattern. Extracting the specified fields from JSON strings stored inside . Extracting the specified fields from strings stored inside . Creating a new field from existing according to the provided format. Creating a new field according to math calculations over existing . Parsing duration strings into floating-point seconds for further . Creating a boolean field with the result of arbitrary applied to the current fields. Creating an integer field with the length of the given field value. This can be useful for . See the for details. It is possible to perform post-filtering on the at client side with `grep` or similar Unix commands according to . LogsQL will support post-filtering on the original and fields created by various . The following post-filters will be supported: Full-text . . See the for details. Stats over the selected logs can be calculated via . LogsQL will support calculating the following additional stats based on the and fields created by : It will be possible specifying an optional condition when calculating the stats. For example, `sum(responsesize) if (isadmin:true)` calculates the total response size for admins only. It is possible to perform stats calculations on the at client side with `sort`, `uniq`, etc. Unix commands according to . By default VictoriaLogs doesn't sort the returned results because of performance reasons. Use for sorting the results. LogsQL provides the following for limiting the number of returned log entries: and pipes allow limiting the set of to return. allows limiting the number of log entries to return. Specific log fields can be queried via . LogsQL accepts numeric values in the following formats: regular integers like `12345` or `-12345` regular floating point numbers like `0.123` or `-12.34` LogsQL accepts integer and floating point values with the following suffixes: `K` and `KB` - the value is multiplied by `10^3` `M` and `MB` - the value is multiplied by `10^6` `G` and `GB` - the value is multiplied by `10^9` `T` and `TB` - the value is multiplied by `10^12` `Ki` and `KiB` - the value is multiplied by `2^10` `Mi` and `MiB` - the value is multiplied by `2^20` `Gi` and `GiB` - the value is multiplied by `2^30` `Ti` and `TiB` - the value is multiplied by `2^40` All the numbers may contain `` delimiters, which may improve readability of the query. For example, `1234_567` is equivalent to `1234567`, while `1.234_567` is equivalent to `1.234567`. LogsQL accepts duration values with the following suffixes at places where the duration is allowed: `ns` - nanoseconds. For example, `123ns`. `s` - microseconds. For example, `1.23s`. `ms` - milliseconds. For example, `1.23456ms` `s` - seconds. For example, `1.234s` `m` - minutes. For example, `1.5m` `h` - hours. For example, `1.5h` `d` - days. For example, `1.5d` `w` - weeks. For example, `1w` `y` - years as 365 days. For example, `1.5y` Multiple durations can be combined. For example, `1h33m55s`. Internally duration values are converted into nanoseconds. It is highly recommended specifying in order to narrow down the search to specific time range. It is highly recommended specifying in order to narrow down the search to specific . Move faster filters such as and to the beginning of the query. This rule doesn't apply to and , which can be put at any place of the query. Move more specific filters, which match lower number of log entries, to the beginning of the query. This rule doesn't apply to and , which can be put at any place of the