Project: Grafana Tempo (Observability and Analysis / Observability)
File: v2-3.md

title: Version 2.3 release notes menuTitle: V2.3 description: Release notes for Grafana Tempo 2.3 weight: 45 The Tempo team is pleased to announce the release of Tempo 2.3. This release gives you: New in TraceQL: Ancestor (`<<`) Parent (`<`) Not-child (`!>`) (experimental) Not-descendant (`!>>`) (experimental) Not-ancestor (`!<<`) (experimental) Not-parent (`!<`) (experimental) Not-siblings (`!~`) (experimental) TraceQL support for (for example, `{ span."name with spaces" = "foo" }`). for improving query performance on your most frequently queried attributes Tempo 2.3 introduces , a Parquet version designed to be more compatible with other Parquet implementations, available as a production-ready option. This block format improves query performance relative to previous formats. Read the for more examples and details about these improvements. These release notes highlight the most important features and bugfixes. For a complete list, refer to the . {{< youtube id="2FWi9_dSBdM?rel=0" >}} The most important features and enhancements in Tempo 2.3 are highlighted below. , an updated Parquet block format introduced in Tempo 2.2, has been promoted to stable and is production ready. We're excited about vParquet3 relative to prior formats because of its support for dedicated attribute columns, which help speed up queries on your largest and most queried attributes. ] To read more about the design of vParquet3, refer to . For general information, refer to . Dedicated attribute columns improve query performance by storing the largest and most frequently used attributes in their own columns, rather than in the generic attribute key-value list. Introduced with vParquet3, dedicated attribute columns are only available when using this storage format. To learn more about this new feature, refer to the documentation. Unique to Tempo, TraceQL is a query language that lets you perform custom queries into your tracing data. To learn more about the TraceQL syntax, see the . For information on planned future extensions to the TraceQL language, refer to . Weve made the following improvements to TraceQL: Added two structural operators, ancestor (`<<`) and parent (`<`) () ] Added experimental not structural operators: `!>>`, `!>`, `!<<`, `!<`, and `!~` () ] Added quoted attribute syntax so you can search for attributes with spaces and other special characters in their names () ] Improved the performance of TraceQL . Metrics-summary now also correctly handles missing attributes. ] Added support for searching by OpenTelemetrys using `statusMessage` intrinsic attribute () ] Fixed cases where an empty filter (`{}`) didn't return expected results ] Weve made the following improvements to metrics-generator: Added a scope query parameter to `/api/overrides` so users can choose between fetching the overrides stored by the API and the merged overrides (those actually used by Tempo) , ] Added `TempoUserConfigurableOverridesReloadFailing` alert ] Added a metrics-generator configuration option to enable/disable X-Scope-OrgID headers on remote write to better support single-tenant

] Allowed metrics-generator ingestion slack to be configured on a per-tenant, rather than global, basis ] Added several metrics-generator fields to user-configurable overrides, including histogram buckets, collection-interval, and span metrics filter policies , , , ] When to Tempo 2.3, be aware of these considerations and breaking changes. Although the vParquet3 format isn't yet the default, it's production ready and we highly recommend switching to it for improved query performance and . Upgrading to Tempo 2.3 doesnt modify the Parquet block format. You can use Tempo 2.3 with vParquet2 or vParquet3. vParquet2 remains the default backend for Tempo 2.3; vParquet3 is available as a stable option. {{< admonition type="note" >}} Tempo 2.2 cant read data stored in vParquet3. {{% /admonition %}} For information on upgrading, refer to upgrade documentation. If you are using Azure storage, we recommend using the v2 SDK, . You can use the `usev2sdk` configure option for switching. For more information, refer to the . ] You can now enable a new configuration block for the distributor to produce debug metrics. These metrics can be particularly useful when tracking down a process that's creating an exceptional amount of spans. ] ```yaml distributor: metricreceivedspans: enabled: false root_only: false ``` Weve added a new `defaults` block to the overrides module for configuring global or per-tenant settings. The Overrides change to indented syntax. For more information, read the . You can also use the Tempo CLI to migrate configurations. Refer to the . ] The old configuration block looked like this: ```yaml overrides: ingestionratestrategy: local ingestionratelimit_bytes: 12345 ingestionburstsize_bytes: 67890 maxsearchduration: 17s forwarders: ['foo'] metricsgeneratorprocessors: [service-graphs, span-metrics] ``` Here is an example of the new configuration block: ```yaml overrides: defaults: ingestion: rate_strategy: local ratelimitbytes: 12345 burstsizebytes: 67890 read: maxsearchduration: 17s forwarders: ['foo'] metrics_generator: processors: [service-graphs, span-metrics] ``` <table> <tr> <td><strong>Parameter</strong> </td> <td><strong>Comments</strong> </td> </tr> <tr> <td><code>distributor.logreceivedtraces</code> </td> <td>Use the <code>distributor.logreceivedspans</code> configuration block instead. [PR <a href="https://github.com/grafana/tempo/pull/3008">#3008</a>] </td> </tr> <tr> <td><code>tempoqueryfrontendqueriestotal{op="searchtags|metrics"}</code> </td> <td>Removed deprecated frontend metrics configuration option </td> </tr> </table> The following vulnerabilities have been addressed: Updated Alpine image version to 3.18 to patch Corrected a HTTP rapid reset issue vulnerability () For a complete list, refer to the . Include statusMessage intrinsic attribute in tag search. Fix compactor ignore configured S3 headers. Readd session token to s3 credentials. Loaded defaults for the internal server Fixed pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions Fixed a panic in metrics-summary API PR Fixed a rare deadlock when uploading blocks to Azure Blob Storage Only search ingester blocks that fall within the request time range. Aligned `tempoqueryfrontendqueriestotal` and `tempoqueryfrontendquerieswithinslototal`. This query now correctly tells you `%age` of requests that are within SLO: ``` sum(rate(tempoqueryfrontendquerieswithinslototal{}[1m])) by (op) / sum(rate(tempoqueryfrontendqueriestotal{}[1m])) by (op) ``` Fixed support for blob storage in Azure Stack Hub as a backend. Respected spss on GRPC streaming. Moved empty root span substitution from `querier` to `query-frontend`. Ingester errors correctly propagate on the query path Fixed an issue where the ingester didnt stop a query after timeout Reordered the S3 credential chain and upgraded `minio-go`. `nativeawsauth_enabled` is deprecated