Project: Grafana Tempo (Observability and Analysis / Observability)
File: v2-1.md

title: Version 2.1 release notes menuTitle: V2.1 description: Release notes for Grafana Tempo 2.1 weight: 55 The Tempo team is pleased to announce the release of Tempo 2.1. With this release, Tempo enhances TraceQL so you can select traces using features such as span kind, aribtrary arithmetic, and new aggregates min/max/sum. In addition, Tempo 2.1 introduces vParquet2, a new experimental Parquet version designed to be more compatible with other Parquet implementations. Note: For a complete list of changes and enhancements, refer to the . Some of the most important features and enhancements in Tempo 2.1 are highlighted below. Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces in Tempo. TraceQL makes it possible to find traces that otherwise would have been difficult or even impossible to identify with Tempos 1.x search. With this release, weve expanded the TraceQL language. In Tempo 2.0, TraceQL allowed users to select traces based upon resource attributes, time, duration, and basic aggregates. In Tempo 2.1, users can also select traces based on the following: Enable querying by `kind`, an intrinsic attribute (PR ) `{ kind = server }`, `{ kind = client }`, `{ kind = consumer }` `{ kind = server && duration > 1s }` Support arithmetic expressions (PR ) `{ span.bytesProcessed > 10 1024 * 1024 }` `{ span.bytesProcessed / 1024 / 1024 > 10 }` `{ span.jobs / span.bytes > 3 }` Enable min, max, and sum aggregate filters `{} | min(duration) > 1s` `{} | max(span.bytesProcessed) &lt; 1024 1024` `{} | sum(duration) > 1h` Support for more numeric types: Compare floating point numbers and integers, allow floating point numbers for durations, and support negative numbers when doing aggregate filters. (PR , , ) `{ 1 = 1.0 }` `{ duration > 1.5s }`, `{duration > 1s500ms }` `{} | count() > -1` This update also includes major performance improvements to TraceQL query performance. ] To learn more about the TraceQL syntax, see the . For information on planned future extensions to the TraceQL language, see . Starting in Grafana 9.5, users have access to a Tempo Search tab powered by TraceQL. To enable this feature, use Grafanas `traceqlSearch` feature flag. The editor can autocomplete queries and has a streamlined interface. Grafana will soon include a new Search screen that can build a TraceQL query based upon options selected in drop-downs. Contact Support to enable this feature in Grafana Cloud. For more information, refer to the . Tempo 2.1 offers an experimental update to the Parquet backend: `vParquet2`. Due to the dedicated span duration column, there is a performance gain for TraceQL queries containing duration. vParquet2 will also be a prerequisite for TraceQL's structural operators, which will be introduced in a future

] You can select the block format Tempo uses by changing the block format version option. To use the new format, set the block format to `vParquet2`. For more information, read the . The new encoding implements Parquet schema changes that are not compatible with `vParquet`: Renamed columns to make the schema more consistent and closer to names and terminologies used by OTEL Add a dedicated span column for duration Additional columns that will enable new TraceQL features, including hierarchical queries For a complete list, refer to the . PR Log when a trace is too large to compact. Traces are dropped when they exceed a certain limit set in the ingesters. Any dropped traces are now logged with the number of dropped spans and the trace ID. PR Add support for Azure Workload Identity authentication PR Add command to tempo-cli to migrate a tenant PR Add support for s3 session token in static configuration When to Tempo 2.1, be aware of these breaking changes. All Prometheus metrics exposed by Tempo on its `/metrics` endpoint that were previously prefixed with `cortex` have now been renamed to be prefixed with `tempo` instead. (PR ) Tempo now includes SLO metrics to count where queries are returned within a configurable time range. (PR ) The `queryfrontendresultmetricsinspectedbytes` metric was removed in favor of `queryfrontendbytesprocessedpersecond`. Users can no longer search blocks in v2 format. Only vParquet and vParquet2 formats support search. The following search configuration options were removed from the overrides section: ``` overrides: maxsearchbytespertrace: searchtagsallow_list: searchtagsdeny_list: ``` The following metrics configuration was also removed: ``` tempoingestertracesearchbytesdiscardedtotal ``` This change means that if you upgrade directly from 1.9 to 2.1, you will not be able to search your v2 blocks. One way to avoid this is to upgrade to 2.0 first, since 2.0 supports searching both v2 and vParquet blocks. You can let your old v2 blocks gradually age out while Tempo creates new vParquet blocks from incoming traces. Once all of your v2 blocks have been deleted and you only have vParquet format-blocks, you can upgrade to Tempo 2.1. All of your blocks will be searchable. Finally, parquet files are no longer cached when carrying out searches (). For a complete list, refer to the . Fixed issue where Tempo sometimes flips booleans from false->true at storage time. Correctly connect context during compaction Apply `rate()` to the bytes/s panel in the tenant's dashboard. Retry copy operations during compaction in GCS backend. Improve locking and search head block in SearchTagValuesV2 Fix not closing WAL block file before attempting to delete the folder. Stop searching for virtual tags if there are any hits. This prevents invalid values from showing up for intrinsics like `status`. Correctly return unique spans when &&ing and ||ing spansets.