Project: Grafana Tempo (Observability and Analysis / Observability)
File: 2022-04 TraceQL Concepts.md

Authors: Yuri Shkuro (@yurishkuro), Tom Wilkie (@tomwilkie), Cyril Tovena (@cyriltovena), Martin Disibio (@mdisibio), Joe Elliott (@joe-elliott) Created: 2021 October - 2022 March Last updated: 2022-04-12 This design document describes a language for selecting traces that Tempo will implement. This language is currently only focused on trace selection. It will be extended to derive metrics from traces in the future. This document is not meant to be a complete language specification. Rather it is meant to invite the community to comment on the basic structure and concepts of the language. - - - - A TraceQL query can select traces based on span attributes, timing and duration structural relationships between spans aggregated data from the spans in a trace The TraceQL language is designed to use the same syntax and semantics as PromQL and LogQL where possible. The syntax and semantics differ due to the specialized needs of querying traces. A query is an expression that is evaluated on one trace at a time. The query is structured as a set of chained expressions (a pipeline). Each expression in the pipeline selects or discards spansets from being included in the results set. E.g. `{ .http.status = 200 } | by(.namespace) | count() > 3` Queries select sets of spans and filter them through a pipeline of aggregators and conditions. If a spanset is produced after evaluation on a trace then this spanset (and by extension the trace) is included in the resultset of the query. In TraceQL the curly brackets `{}` always select a set of spans from the current trace. They will commonly be paired with a condition to reduce the spans being passed in. `{ .http.status = 200 }` This simple query will be evaluated on every span of every trace, one at a time. If the trace being evaluated contains no spans with an attribute `http.status` with the value `200` then no spans will be selected and this trace will not appear in our resultset. If the trace does contain spans with an attribute `http.status` with the value `200` then only those spans will be returned. The trace is reduced to only the set of spans that match the condition inside the `{}`. The resultset will contain only this subset of spans matching the condition. Every span has certain intrinsic fields that can be referenced when selecting spans | field name | description | | - | - | | `duration` | end - start time of the span | | `name` | operation or span name | | `status` | status values are error, ok, or unset | | `parent` | the parent of this span | Examples Find traces that contain spans whose duration is greater than 2 seconds: `{ duration > 2s }` Find traces that contain a span named "HTTP POST": `{ name = "HTTP POST" }` We can refer to dynamic attributes (also known as tags) on the span or the span's resource. Examples Find traces with the GET http method: `{ .http.method = "GET" }` Find traces that passed through the prod namespace: `{ .namespace = "prod" }` Find traces that cross service boundaries: `{ parent.service.name != .service.name }` Attributes can be specifically scoped to either "span" or "resource". Specifying "span" or "resource" can result in significant performance benefits. `{ span.http.status = 200 }` `{ resource.namespace = "prod" }` Fields can also be combined in various expected ways. Examples Find traces where the difference of the duration of a set of parent/child spans exceeds 500ms: `{

- duration > 500ms }` Find traces with "success" http status codes: `{ .http.status >= 200 && .http.status < 300 }` Note that the second expression requires both conditions to be true on the same span. The entire expression inside of `{}` must be evaluated as true on a single span for it to be included in the resultset. Logical operators combine sets of spans. For example, if we wanted to find a trace that went through two specific regions: `{ .region = "eu-west-0" } && { .region = "eu-west-1" } ` Note the difference between the above and the following: `{ .region = "eu-west-0" && .region = "eu-west-1" } ` The second expression will return no traces because it's impossible for both conditions to be simultaneously true on the same span. Structural operators evaluate traces based on the span relationships within the span tree. `{ } >> { }` This is the descendant operator. The spans returned from this operator will match the right hand side conditions while also being descendants of spans that match the left hand side conditions. `{ } > { }` This is the child operator. The spans returned from this operator will match the right hand side conditions while also being children of spans that match the left hand side conditions. `{ } ~ { }` This is the sibling operator. The spans returned from this operator will match the right hand side conditions while also being siblings of spans that match the left hand side conditions. All of the above expressions involve asking questions about individual spans. However, sometimes we want to ask questions about a set of spans. For that we have aggregate functions. Examples Find traces where the total number of spans is greater than 10: `count() > 10` Find traces where the average duration is greater than 1s: `avg(duration) > 1s` Pipelining allows us to "pipe" a set of spans from one expression to the next. This is particularly useful if you want to perform an aggregate over a subset of a trace. Examples Find traces that have more than 3 spans with an attribute `http.status` with a value of `200`: `{ .http.status = 200 } | count() > 3` Grouping allows us to take a trace and break it down into sets of spans that are evaluated by future pipeline entries. Each set of spans created by the group is individually evaluated by downstream expressions. Examples Find traces that have more than 5 spans in any region: `by(.region) | count() > 5 ` Any span matches an attribute: `{ .namespace = "prod" }` Two attributes appear on the same span: `{ .namespace = "prod" && .http.status = 200 }` Two attributes appear anywhere within a trace: `{ .namespace = "prod" } && { .http.status = 200 }` Any span has a duration over one second: `{ duration > 1s }` The trace as a whole has a duration of over one second: `max(end) - min(start) > 1s` The average duration of spans exceeds 1s and any span has a specific attribute: `avg(duration) > 1s && { .namespace = "prod" }` A trace has over 5 spans with http.status = 200 in any given namespace: `{ .http.status = 200 } | by(.namespace) | count() > 5` A trace passed through two regions in a specific order: `{ .region = "eu-west-0" } >> { .region = "eu-west-1" }` A trace sees network latency > 1s when passing between any two services: `{ parent.service.name != .service.name } | max(parent.duration - duration) > 1s`