Project: Grafana Tempo (Observability and Analysis / Observability)
File: v2-2.md

title: Version 2.2 release notes menuTitle: V2.2 description: Release notes for Grafana Tempo 2.2 weight: 50 The Tempo team is pleased to announce the release of Tempo 2.2. This release gives you: Major additions to the TraceQL language: structural operators (descendant, child, sibling), results grouped by attribute (`by()`), a `select` operator, and 3 new intrinsic attributes. Faster TraceQL results, thanks to a streaming endpoint that returns partial results as the query executes as well as a multitude of performance-related improvements. An experimental metrics-summary API that returns RED metrics for recently received spans grouped by your attribute of choice. Tempo 2.2 makes vParquet2, a Parquet version designed to be more compatible with other Parquet implementations, the default block format. This block format is required for using structural operators and improves query performance relative to previous formats. Read the for more examples and details about these improvements. {{< admonition type ="note" >}} For a complete list of changes, enhancements, and bug fixes refer to the . {{% /admonition %}} Some of the most important features and enhancements in Tempo 2.2 are highlighted below. With this release, weve added to the . TraceQL now offers: Structural operators: descendant (>>), child (>), and sibling (~) (). Find relevant traces based on their structure and relationships among spans. ] A `select()` operation that allows you to specify arbitrary span attributes that you want included in the TraceQL response () ] A `by()` operation that groups span sets within a trace by an attribute of your choosing. This operation is not supported in the Grafana UI yet; you can only use `by()` when querying Tempos search API directly. ( ] New intrinsic attributes for use in TraceQL queries: `traceDuration`, `rootName`, and `rootServiceName` () ] Read the for examples of how to use these new language additions. To learn more about the TraceQL syntax, see the . For information on planned future extensions to the TraceQL language, see . Were always trying to reduce the time you spend waiting to get results to your TraceQL queries, and weve made some nice progress on this front with this release. Weve added a endpoint to Tempos query frontend that allows a client to stream search results from Tempo. The Tempo CLI has been updated to use this new streaming endpoint ] . As of version 10.1, Grafana supports it as well, though you must first enable the `traceQLStreaming` feature toggle

By streaming results to the client, you can start to look at traces matching your query before the entire query completes. This is particularly helpful for long-running queries; while the total time to complete the query is the same, you can start looking at your first matches before the full set of matched traces is returned. In addition to streaming partial results, weve merged a number of improvements to speed up TraceQL queries. Here are just a few: Add support for query batching between frontend and queriers to improve throughput ] Improve performance of TraceQL regex ] Fully skip over Parquet row groups with no matches in the column dictionaries ] New synchronous read mode for vParquet and vParquet2 , ] Improved TraceQL throughput by asynchronously creating jobs. ] Tempo has added an experimental API that returns RED metrics (span count, erroring span count, and latency information) for spans of `kind=server` sent in the last hour, grouped by an attribute of your choice. For example, you could use this API to compare error rates of spans with different values of the `namespace` attribute. From here, you might see that spans from `namespace=A` have a significantly higher error rate than those from `namespace=B`. As another example, you could use this API to compare latencies of your spans broken down by the `region` attribute. From here, you might notice that spans from `region=North-America` have higher latencies than those from `region=Asia-Pacific`. This API is meant to enable ad-hoc analysis of your incoming spans; by segmenting your spans by attribute and looking for differences in RED metrics, you can more quickly isolate where problems like elevated error rates or higher latencies are coming from. Unlike RED metrics computed by Tempos , the values returned by this API are not persisted as time series. This has the advantage that you do not need to provide your own time series databases for storing and querying these metrics. It also allows you to compute RED metrics broken down by high cardinality attributes that would be too expensive to store in a time series database. Use the metrics generator if you want to store and visualize RED metrics over multi-hour or multi-day time ranges, or you want to alert on these metrics. To learn more about this API, refer to the This work is represented in multiple PRs: , , , , , , , , and . Tempos and APIs now support filtering ]. This lets you retrieve all valid attribute values and names given certain

For example, you can get a list of values for the attribute `namespace` seen on spans with attribute `resource=A.` This feature is off by default; to enable, configure `autocompletefilteringenabled`. (). Grafanas autocomplete can make use of this filtering capability to provide better suggestions starting in v10.2 . Tempos metrics-generator now supports span filtering. Setting up filters allows you to compute metrics over the specific spans you care about, excluding others. It also can reduce the cardinality of generated metrics, and therefore the cost of storing those metrics in a Prometheus-compatible TSDB. () ] Tempos metrics-generator can now detect virtual nodes () ]. As a result, youll now see these virtual nodes represented in your service graph. For more information, refer to the . When to Tempo 2.2, be aware of these breaking changes: JSonnet users only: We've converted the metrics-generator component from a k8s deployment to a k8s statefulset. Refer to the PR for seamless migration instructions. , ] Removed or renamed configuration parameters (see section below) While not a breaking change, upgrading to Tempo 2.2 will by default change Tempos block format to vParquet2. To stay on a previous block format, read the . We strongly encourage upgrading to vParquet2 as soon as possible as this is required for using structural operators in your TraceQL queries and provides query performance improvements, in particular on queries using the `duration` intrinsic. The following fields were removed or renamed. <table> <tr> <td>Parameter </td> <td>Comments </td> </tr> <tr> <td> <p> <code> query_frontend:</code> <p> <code> toleratefailedblocks: &lt;int></code> </td> <td>Remove support for `tolerantfailedblocks` [PR <a href="https://github.com/grafana/tempo/pull/2416">2416</a>] </td> </tr> <tr> <p> <code>storage:</code> <p> <code> trace:</code> <p> <code> s3:</code> <p> <code> insecureskipverify: true // renamed to tlsinsecureskip_verify</code> <p> </td> <td>Renamed `insecureskipverify` to `tlsinsecureskip_verify` [PR <a href="https://github.com/grafana/tempo/pull/2407">2407</a>] </td> </tr> </table> For a complete list, refer to the . Updated Alpine image version to 3.18 to patch Bumped Jaeger query docker image to 1.50.0 Fixed S3 credentials providers configuration Fixed node role auth IDMSv1 Fixed incorrect metrics for index failures Fixed a panic in the metrics-generator when using multiple tenants with default overrides Restored `tenantheaderkey` removed in Disabled streaming over HTTP by default Fixed an issue in the metrics-generator that prevented scaling up parallelism when remote writing of metrics was lagging behind ] Fixed an issue where metrics-generator was setting wrong labels for `tracestargetinfo` ] Fixed an issue where matches and other spanset level attributes were not persisted to the TraceQL results. ] Fixed an issue where ingester search could occasionally fail with `file does not exist` error ]