Project: Grafana Tempo (Observability and Analysis / Observability)
File: v2-0.md

title: Version 2.0 release notes menuTitle: V2.0 description: Release notes for Grafana Tempo 2.0 weight: 60 The Tempo team is pleased to announce the newest major release of Tempo: version 2.0! Tempo 2.0s headline feature is the introduction of TraceQL. TraceQL is a query language modeled on PromQL and LogQL that allows you to interactively extract insights from your trace data. It provides a powerful, flexible way to pinpoint the traces you need to answer questions about your systems and analyze traces based on their structure. Continuing in the tradition of Tempos close integration with Grafana as a visualization layer, weve also added a TraceQL query editor to Grafanas Tempo data source. The editor is available in Grafana Cloud and for open source users in Grafana 9.3.2. We encourage open source users to upgrade to Grafana 9.4 as soon as it is released to take advantage of some additional enhancements and bug fixes. In addition to TraceQL, Tempo 2.0 also marks the promotion of the Apache Parquet block format to the new default storage format. With this change, the Tempo maintainers are signaling that this new format has received enough testing to be ready for broad adoption. These two changes actually go hand-in-hand: without the increased search speed enabled by the Apache Parquet block format, it would be impossible to pull off the sophisticated queries made possible by TraceQL. Note: For a complete list of changes and enhancements, refer to the . Some of the most important features and enhancements in Tempo 2.0 are highlighted below. Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces in Tempo. TraceQL makes it possible to find traces that otherwise would have been difficult or even impossible to identify with Tempo's existing search. It also makes it possible for you to understand the broad context for an event sequence. To understand more about why we built TraceQL and the use cases we think it unlocks, see . With this release, weve implemented a subset of the full TraceQL language we envision. Users can select traces based on the following: Span and resource attributes, timing, and duration Basic aggregates: `count()` and `avg()` To learn more about the TraceQL syntax, see the . For information on how we plan to extend the TraceQL language, see . You can run a TraceQL query either by issuing it to Tempos `q` parameter of the , or, for those using Tempo in conjunction with Grafana, by using Grafanas TraceQL query editor, described below. TraceQL requires Tempos Parquet columnar format, which is the default block format in Tempo 2.0. For more information, refer to the Tempo documentation. To make it easier to run and visualize the results of TraceQL queries, weve added a TraceQL query editor to Grafanas Tempo data source. The editor is available in Grafanas Explore interface starting in Grafana version 9.3.2. <p align="center"><img src="../../traceql/assets/query-editor-results-span.png" alt="Query editor showing span results" /></p> The query editor helps you learn TraceQL by offering autocomplete. <p align="center"><img src="../../traceql/assets/query-editor-auto-complete.png" alt="Query editor showing the auto-complete feature" /></p> For more information, refer to the . The experimental Apache Parquet columnar block format introduced in Tempo v1.5 has been promoted to stable and is now the default trace storage for Tempo 2.0. We're excited to switch to it because of the increased speed and efficiency it provides for search. With our previous block format, we'd seen our ability to search trace data cap out at a rate of ~40-50 GB per

Since switching to Parquet, we're now hitting search speeds of 300 GB/s on common queries and doing so with less compute. If we define the bytes processed as the total trace bytes that are scanned, then this example query resulted in ~1.2 TB/s: ```yaml { resource.cluster ="foo" } ``` and this query resulted in ~550 GB/s: ```yaml { span.foo ="bar" } ``` Another benefit of the Parquet block format relative to our previous format is that users can now leverage the large ecosystem of existing Parquet tools and libraries to process and transform their trace data. We look forward to see what downstream applications people create with this ability. The Parquet block format can be used as a drop-in replacement for Tempos existing block format. No data conversion or upgrade process is necessary. As soon as the Parquet format is enabled, Tempo starts writing data in that format, leaving existing data as-is. Tempo search will return results from both the legacy and Parquet block formats. However, search on legacy blocks has been marked as deprecated and will be removed in Tempo 2.1. TraceQL queries will only search trace data stored in Parquet blocks. For more information, refer to the . Note: There is a potential issue loading Tempo 1.5's experimental Parquet storage blocks. You may see errors or even panics in the compactors. We have only been able to reproduce this with interim commits between 1.5 and 2.0, but if you experience any issues please so we can isolate and fix this issue. Search and are now enabled by default in Tempo 2.0, and their corresponding configuration parameters (`searchenabled` and `metricsgenerator_enabled`) have been removed. Weve been excited by the feedback weve gotten on both features and are satisfied with their stability so we want all newcomers to Tempo to have access to them from day 1. Although the metrics-generator component is now enabled by default, no actual metrics will be generated until certain tenant-specific configuration parameters have been set. For more information, refer to the . NOTE: Grafana Cloud customers need to contact Grafana Support to enable metrics generation. As part of promoting the metrics-generator to default on, we've made the following improvements to it: The metrics-generator now handles collisions between user-defined and default dimensions. User-defined dimensions colliding with default dimensions will be prefixed with ``. The metrics-generator now makes intrinsic dimensions configurable and disables `status_message` by default . The metrics-generator now computes a metric that tells you the size (in bytes) of spans received broken down by team, service, or other relevant labels. Since span volume roughly correlates with the cost to operate Tempo, this metric is helpful for cost attribution and chargeback purposes. . Older spans are now filtered out before metrics are aggregated. This is helpful since including these older span in metric computation can lead to misleading or incorrect results. . Label names and values that exceed a configurable length are truncated. In this way, the Tempo metrics-generator sanitizes metrics before sending them to a receiving time-series-database (which usually has its own limits around maximum label length). . Zone awareness is a feature that ensures data is replicated across failure domains (which we refer to as "zones") to provide greater reliability. A failure domain is whatever you define it to be, but commonly may be an availability zone, data center, or server rack. When zone awareness is enabled for ingesters, incoming trace data is guaranteed to be replicated to ingesters in different zones. This allows the system to withstand the loss of one or more zones (depending on the replication

Thank you to community member manohar-koukuntla for making this contribution! ```yaml multizoneingester_enabled: false, multizoneingestermigrationenabled: false, multizoneingester_replicas: 0, multizoneingestermaxunavailable: 25, ``` When to Tempo 2.0, be aware of these breaking changes. The following configuration parameters have been updated with new default values. These defaults were changed to work better with the new Parquet block format. If you are sticking with the `v2` format you may prefer the old values. ```yaml query_frontend: maxoustandingper_tenant: 2000 search: concurrent_jobs: 1000 targetbytesper_job: 104857600 max_duration: 168h queryingestersuntil: 30m tracebyid: query_shards: 50 querier: maxconcurrentqueries: 20 search: prefer_self: 10 ingester: concurrent_flushes: 4 maxblockduration: 30m maxblockbytes: 524288000 storage: trace: pool: max_workers: 400 queue_depth: 20000 search: readbuffercount: 32 readbuffersize_bytes: 1048576 ``` The following tables describe the parameters that have been removed or renamed. (PR#1978) | Parameter | Comments | | :-- | :-- | | <pre>queryfrontend:<br/>&nbsp;&nbsp;queryshards:</pre> | Replaced by `tracebyid.query_shards` | | <pre>querier:<br>&nbsp;&nbsp;querytimeout:</pre> | Replaced by two different settings: `search.querytimeout` and `tracebyid.query_timeout` | | <pre>ingester:<br>&nbsp;&nbsp;useflatbuffersearch:</pre> | Removed and automatically determined based on block format | | `search_enabled` | Removed. Now defaults to true. | | `metricsgeneratorenabled` | Removed. Now defaults to true. | | <pre>storage:<br>&nbsp;&nbsp;wal:<br>&nbsp;&nbsp;&nbsp;&nbsp;version:</pre> | Removed and pinned to `block.version` | | Parameter | Comments | | | | | <pre>queryfrontend:<br>&nbsp;&nbsp;queryshards:</pre> | Renamed `queryfrontend.tracebyid.queryshards` | | <pre>querier:<br>&nbsp;&nbsp;querytimeout:</pre> | Renamed `querier.tracebyid.querytimeout` | The following `compactor` configuration parameters were renamed. | Parameter | Comments | | | | | <pre>compaction:<br>&nbsp;&nbsp;chunksizebytes:</pre> | Renamed to `v2inbuffer_bytes` | | <pre>compaction:<br>&nbsp;&nbsp;flushsizebytes:</pre> | Renamed to `v2outbuffer_bytes` | | <pre>compaction:<br>&nbsp;&nbsp;iteratorbuffersize:</pre> | Renamed to `v2prefetchtraces_count` | The following `storage` configuration parameters were renamed. | Parameter | Comments | | | | | <pre>wal:<br>&nbsp;&nbsp;encoding:</pre> | Renamed to `v2_encoding` | | <pre>block:<br>&nbsp;&nbsp;indexdownsamplebytes:</pre> | Renamed to `v2indexdownsample_bytes` | | <pre>block:<br>&nbsp;&nbsp;indexpagesizebytes:</pre> | Renamed to `v2indexpagesize_bytes` | | <pre>block:<br>&nbsp;&nbsp;encoding:</pre> | Renamed to `v2_encoding` | | <pre>block:<br>&nbsp;&nbsp;rowgroupsizebytes:</pre> | Renamed to `parquetrowgroupsize_bytes` | . The Azure Storage configuration now uses snake case with underscores (`_`) instead of dashes (`-`)Use snake case on Azure Storage config. Here is an example of using snake case on Azure Storage config: ```yaml storage: trace: azure: storageaccountname: storageaccountkey: container_name: ``` . Parquet is the new default block version. To continue using the v2 block format, see the . . We've deleted the `TempoRequestErrors` alert from mixin. Any Jsonnet users relying on this alert should copy this into their own environment. . We've added TLS support to the vulture. Internal types are updated to use `scope` instead of `instrumentation_library`. This is a breaking change in "trace by ID" type queries if JSON is requested. No longer return `status.code` from `/api/search/tags` unless it is an attribute present in the data, Pad leading zeroes in span id to always be 16 characters. Suppress logspam in single binary mode when metrics generator is disabled. Error more gracefully while reading some blocks written by an interim commit between 1.5 and 2.0. Correctly coalesce trace level data when combining Parquet traces. Unescape query parameters in AWS Lambda to allow TraceQL queries to work. Version 2.0 includes the following fixes: Stop distributors on OTel receiver fatal error. New wal file separator `+` for the NTFS filesystem and backward compatibility with the old separator `:`. Honor caching and buffering settings when finding traces by ID. Correctly propagate errors from the iterator layer up through the queriers. Make multitenancy work with HTTP. Fix parquet search bug fix on `http.status_code` that may cause incorrect results to be returned. Fix failing `SearchTagValues` endpoint after startup. tempo-mixin: tweak dashboards to support metrics without cluster label present. Fix docker-compose examples not running on Apple M1 hardware. Fix TraceQL parsing of most binary operations to not require spacing. Don't persist tenants without blocks in the