Project: Grafana Tempo (Observability and Analysis / Observability)
File: v2-4.md

title: Version 2.4 release notes menuTitle: V2.4 description: Release notes for Grafana Tempo 2.4 weight: 40 The Tempo team is pleased to announce the release of Tempo 2.4. This release gives you: New features, including multi-tenant queries and experimental TraceQL metrics queries Performance enhancements, thanks to the addition of new caching tiers Cost savings, thanks to polling improvements that reduce calls to object storage As part of this release, vParquet3 has also been promoted to the new default storage format for traces. For more about why were so excited about vParquet3, refer to . Read the for more examples and details about these improvements. These release notes highlight the most important features and bugfixes. For a complete list, refer to the . {{< youtube id="EYUx2DkNRas" >}} The most important features and enhancements in Tempo 2.4 are highlighted below. Tempo now allows you to query multiple tenants at once. Weve made multi-tenant queries compatible with streaming () so you can get query results as fast as possible. To learn more, refer to and . , ] Were excited to announce the addition of metrics queries to the TraceQL language. Metric queries extend trace queries by applying a function to trace query results. This powerful feature creates metrics from traces, much in the same way that LogQL metric queries create metrics from logs. In this case, we are calculating the rate of the erroring spans coming from the service `foo`. Rate is a `spans/sec` quantity. ``` { resource.service.name = "foo" && status = error } | rate() ``` In addition, you can use Grafana Explore to with the Tempo data soruce in Grafana or Grafana Cloud. For more information, refer to the documentation. , ] To learn more about the TraceQL syntax, see the . For information on planned future extensions to the TraceQL language, refer to . We continue to make query performance improvements so you spend less time waiting on results to your TraceQL queries. Below are some notable PRs that made it into this release: Improve TraceQL regex performance in certain queries. ] Improve TraceQL performance in complex queries. ] TraceQL/Structural operators performance improvement. ] Tempo 2.4 makes the default storage format. Were excited about relative to prior formats because of its support for , which help speed up queries on your largest and most queried attributes. We've seen excellent performance improvements when running it ourselves, and by promoting it to the default, we're signaling that it is ready for broad

Dedicated attribute columns, available using vParquet3, improve query performance by storing the largest and most frequently used attributes in their own columns, rather than in the generic attribute key-value list. For more information, refer to . ] If you had manually configured vParquet3, we recommend removing it to move forward with Tempo defaults. To read more about the design of vParquet3, refer to . For general information, refer to . Tempo has added two new caches to improve TraceQL query performance. The frontend-search cache handles job search caching. The parquet-page cache handles page level caching. Refer to the of the Configuration documentation for how to configure these new caching layers. As part of adding these new caching layers, weve refactored our caching interface. This includes breaking changes described in Breaking Changes. , , ] Weve improved how Tempo polls object storage, ensuring that we reuse previous results. This has dramatically reduced the number of requests Tempo makes to the object store. Not only does this reduce the load on your object store, for many, it will save you money (since most hosted object storage solutions charge per request). Weve also added the `listblocksconcurrency` parameter to allow you to tune the number of list calls Tempo makes in parallel to object storage so you can select the value that works best for your environment. Weve set the default value to `3`, which should work well for the average Tempo cluster. ] In addition, the following improvements have been made in Tempo 2.4: Improved Tempo error handling on writes, so that one erroring trace doesn't result in an entire batch of traces being dropped. Added per-tenant compaction window. Added `--max-start-time` and `--min-start-time` flag to tempo-cli command `analyse blocks`. Added per-tenant configurable `remote_write` headers to metrics-generator. Added variable expansion support to overrides configuration. Added HTML pages `/status/overrides` and `/status/overrides/{tenant}`. Precalculate and reuse the vParquet3 schema before opening blocks. Made the trace ID label name configurable for remote written exemplars. Performance improvements in span filtering. Introduced localblocks process configuration option to select only server spans. When to Tempo 2.4, be aware of these considerations and breaking changes. vParquet3 format is now the default block format. It is production ready and we highly recommend switching to it for improved query performance and . Upgrading to Tempo 2.4 modifies the Parquet block format. Although you can use Tempo 2.3 with vParquet2 or vParquet3, you can only use Tempo 2.4 with vParquet3. With this release, the first version of our Parquet backend, vParquet, is being deprecated. Tempo 2.4 still reads vParquet1 blocks. However, Tempo will exit with error if they are manually

] For information on upgrading, refer to and . <table> <tr> <td>Parameter </td> <td>Comments </td> </tr> <tr> <td><code>autocompletefilteringenabled</code> </td> <td>Set to <code>true</code> by default [PR <a href="https://github.com/grafana/tempo/pull/3178">3178</a>] </td> </tr> <tr> <td><code>distributor.logreceivedtraces</code> </td> <td>Use the <code>distributor.logreceivedspans</code> configuration block instead. [PR <a href="https://github.com/grafana/tempo/pull/3008">#3008</a>] </td> </tr> <tr> <td><code>tempoqueryfrontendqueriestotal{op="searchtags|metrics"}</code> </td> <td>Removed deprecated frontend metrics configuration option </td> </tr> <tr> <td> ```yaml storage: trace: cache: search: cache_control: background_cache: memcached: ``` </td> <td>These fields have been removed in favor of the new cache configuration. Refer to Cache configuration refactored. </td> </tr> </table> The distributor now returns 200 for any batch containing only `tracetoolarge` and `maxlivetraces` errors. The number of discarded spans are still reflected in the `tempodiscardedspans_total metrics`. GPRC is now the supported method for streaming results. Websockets support for search streaming has been removed. Websocket support was initially added due to conflicts with GRPC, HTTP, and TLS. Those issues were corrected . The major cache refactor allows multiple role-based caches to be configured. ] This change resulted in the following fields being deprecated. These have all been migrated to a top level `cache:` field. For more information about the configuration, refer to the section. The old configuration block looked like this: ```yaml storage: trace: cache: search: cache_control: background_cache: memcached: redis: ``` With the new configuration, you create your list of caches,- with either `redis` or `memcached` cluster with your configuration, then define the types of data and roles. ```yaml cache: caches: memcached: host: <some memcached cluster> roles: bloom parquet-footer memcached: host: <some memcached cluster> roles: frontend-search ``` The following vulnerabilities have been addressed: Addressed . Updated the `memcached` default image in Jsonnet for multiple CVEs. Update golang.org/x/net package to 0.24.0 to fix For a complete list, refer to the . Update golang.org/x/net package to 0.24.0 to fix . Fixed compaction/retention in AWS S3 and GCS when a prefix is configured. Prevent building parquet iterators that would loop forever. Sanitize name in mapped dimensions in span-metrics processor. Fixed an issue where cached footers were requested then ignored. Fixed a panic in autocomplete when the query condition had the wrong type. Fixed TLS when GRPC is enabled on HTTP. Correctly return 400 when max limit is requested on search. Fixed autocomplete filters sometimes returning erroneous results. Fixed trace context propagation between query-frontend and querier. Fixed parsing of span.resource.xyz attributes in TraceQL. Changed exit code if config is successfully verified. The tempo-cli analyze blocks command no longer fails on compacted blocks. Moved waitgroup handling for poller error condition. Fixed head block excessive locking in ingester search. Fixed an issue with ingester failed to write traces to disk after a crash or unclean restart.