Project: Grafana Tempo (Observability and Analysis / Observability)
File: upgrade.md

title: Upgrade your Tempo installation menuTitle: Upgrade description: Upgrade your Grafana Tempo installation to the latest version. weight: 310 You can upgrade an existing Tempo installation to the next version. However, any new release has the potential to have breaking changes that should be tested in a non-production environment prior to rolling these changes to production. The upgrade process changes for each version, depending upon the changes made for the subsequent release. This upgrade guide applies to on-premise installations and not for Grafana Cloud. {{< admonition type="tip" >}} You can check your configuration options using the in your Tempo installation. {{% /admonition %}} Tempo 2.4 has several considerations for any upgrade: vParquet3 is now the default backend Caches configuration was refactored Updated, removed, and renamed configuration parameters For a complete list of changes, enhancements, and bug fixes, refer to the . vParquet3 format is now the default block format. It is production ready and we highly recommend switching to it for improved query performance and . Upgrading to Tempo 2.4 modifies the Parquet block format. Although you can use Tempo 2.3 with vParquet2 or vParquet3, you can only use Tempo 2.4 with vParquet3. With this release, the first version of our Parquet backend, vParquet, is being deprecated. Tempo 2.4 will still read vParquet1 blocks. However, Tempo will exit with error if they are manually configured. ] For information on changing the vParquet version, refer to . The major cache refactor to allow multiple role-based caches to be configured. ] This change resulted in the following fields being deprecated. These have all been migrated to a top level `cache:` field. For more information about the configuration, refer to the section. The old configuration block looked like this: ```yaml storage: trace: cache: search: cache_control: background_cache: memcached: redis: ``` With the new configuration, you create your list of caches, with either `redis` or `memcached` cluster with your configuration, and then define the types of data and roles. Simple configuration example: ```yaml cache: caches: memcached: host: <some memcached cluster> roles: bloom parquet-footer memcached: host: <some memcached cluster> roles: frontend-search ``` <table> <tr> <td>Parameter </td> <td>Comments </td> </tr> <tr> <td><code>autocompletefilteringenabled</code> </td> <td>Set to <code>true</code> by default [PR <a href="https://github.com/grafana/tempo/pull/3178">3178</a>] </td> </tr> <tr> <td><code>distributor.logreceivedtraces</code> </td> <td>Use the <code>distributor.logreceivedspans</code> configuration block instead. [PR <a href="https://github.com/grafana/tempo/pull/3008">#3008</a>] </td> </tr> <tr> <td><code>tempoqueryfrontendqueriestotal{op="searchtags|metrics"}</code> </td> <td>Removed deprecated frontend metrics configuration option </td> </tr> </table> The distributor now returns 200 for any batch containing only `tracetoolarge` and `maxlivetraces` errors. The number of discarded spans are still reflected in the `tempodiscardedspans_total metrics`. Tempo 2.3 has several considerations for any upgrade: vParquet3 is available as a stable, production-read block format Configuration option to use Azure SDK v2 New `defaults` block in Overrides module configuration Several configuration parameters have been renamed or removed. For a complete list of changes, enhancements, and bug fixes, refer to the . vParquet3 provides improved query performance and . This block format is required for using dedicated attribute columns. While vParquet2 remains the default backend for Tempo 2.3, vParquet3 is available as a stable option. Both work with Tempo 2.3. Upgrading to Tempo 2.3 doesnt modify the Parquet block format. {{< admonition type="note" >}} Tempo 2.2 cant read data stored in vParquet3. {{% /admonition %}} Recommended update process: Upgrade your Tempo installation to version 2.3, remaining on vParquet2. Verify the upgrade is stable and performs as expected. If you notice any issues, you can downgrade to version 2.2, and data remains readable.

If you notice any issues on step 3 using the new block format, you can downgrade to vParquet2. All your data remains readable in Tempo 2.3. However, if you have vParquet3 blocks and have to downgrade to Tempo 2.2, you will have data loss. If you are using Azure storage, we recommend using the v2 SDK, . You can use the `usev2sdk` configure option for switching. For more information, refer to the . The Overrides module has a new `defaults` block for configuring global or per-tenant settings. The Overrides format now includes changes to indented syntax. For more information, read the . You can also use the Tempo CLI to migrate configurations. Refer to the . The old configuration block looked like this: ```yaml overrides: ingestionratestrategy: local ingestionratelimit_bytes: 12345 ingestionburstsize_bytes: 67890 maxsearchduration: 17s forwarders: ['foo'] metricsgeneratorprocessors: [service-graphs, span-metrics] ``` The new configuration block looks like this: ```yaml overrides: defaults: ingestion: rate_strategy: local ratelimitbytes: 12345 burstsizebytes: 67890 read: maxsearchduration: 17s forwarders: ['foo'] metrics_generator: processors: [service-graphs, span-metrics] ``` <table> <tr> <td><strong>Parameter</strong> </td> <td><strong>Comments</strong> </td> </tr> <tr> <td><code>distributor.logreceivedtraces</code> </td> <td>Use the <code>distributor.logreceivedspans</code> configuration block instead. [PR <a href="https://github.com/grafana/tempo/pull/3008">3008</a>] </td> </tr> <tr> <td><code>tempoqueryfrontendqueriestotal{op="searchtags|metrics"}</code> </td> <td>Removed deprecated frontend metrics configuration option </td> </tr> </table> Tempo 2.2 has several considerations for any upgrade: vParquet2 is now the default block format Several configuration parameters have been renamed or removed. For a complete list of changes, enhancements, and bug fixes, refer to the . While not a breaking change, upgrading to Tempo 2.2 will by default change Tempos block format to vParquet2. To stay on a previous block format, read the . We strongly encourage upgrading to vParquet2 as soon as possible as this is required for using structural operators in your TraceQL queries and provides query performance improvements, in particular on queries using the `duration` intrinsic. Tempo 2.2 updates the `microservices` JSonnet to support a `statefulset` for the `metrics_generator` component. {{< admonition type="note" >}} This update is important if you use the experimental `local-blocks` processor. {{% /admonition %}} To support a new `processor`, the metrics-generator has been converted from a `deployment` into a `statefulset` with a PVC. This requires manual intervention to migrate successfully and avoid downtime. Note that currently both a `deployment` and a `statefulset` will be managed by the JSonnet for a period of time, after which we will delete the deployment from this repo and you will need to delete user-side references to the `tempometricsgenerator_deployment`, as well as delete the deployment itself. Refer to the PR for seamless migration instructions. , ] The following fields were removed or renamed. | Parameter | Comments | ||| |<pre>queryfrontend:<br>&nbsp;&nbsp;toleratefailedblocks: <int></pre> | Remove support for `tolerantfailed_blocks` ] | |<pre>storage:<br>&nbsp;&nbsp;trace:<br>&nbsp;&nbsp;s3:<br>&nbsp;&nbsp;insecureskipverify: true // renamed to tlsinsecureskipverify</pre> | Renamed `insecureskipverify` to `tlsinsecureskipverify` ] | Tempo 2.1 has two major considerations for any upgrade: Support for search on v2 block is removed Breaking changes to metric names For more information on other enhancements, read the . Users can no longer search blocks in v2 format. Only vParquet and vParquet2 formats support search. The following search configuration options were removed from the overrides section: ``` overrides: maxsearchbytespertrace: searchtagsallow_list: searchtagsdeny_list: ``` The following metrics configuration was also removed: ``` tempoingestertracesearchbytesdiscardedtotal ``` Removing support for search on v2 blocks means that if you upgrade directly from 1.9 to 2.1, you will not be able to search your v2 blocks. To avoid this, upgrade to 2.0 first, since 2.0 supports searching both v2 and vParquet blocks. You can let your old v2 blocks gradually age out while Tempo creates new vParquet blocks from incoming

Once all of your v2 blocks have been deleted and you only have vParquet format-blocks, you can upgrade to Tempo 2.1. All of your blocks will be searchable. Parquet files are no longer cached when carrying out searches. All Prometheus metrics exposed by Tempo on its `/metrics` endpoint that were previously prefixed with `cortex` have now been renamed to be prefixed with `tempo` instead. (PR ) Tempo now includes SLO metrics to count where queries are returned within a configurable time range. (PR ) The `queryfrontendresultmetricsinspectedbytes` metric was removed in favor of `queryfrontendbytesprocessedpersecond`. Tempo 2.0 marks a major milestone in Tempos development. When planning your upgrade, consider these factors: Breaking changes: Renamed, removed, and moved configurations are described in section below. The `TempoRequestErrors` alert was removed from mixin. Any Jsonnet users relying on this alert should copy this into their own environment. Advisory: Changed defaults Are these updates relevant for your installation? TraceQL editor needs to be enabled in Grafana to use the query editor. Resource requirements have changed for Tempo 2.0 with the default configuration. Once you upgrade to Tempo 2.0, there is no path to downgrade. {{< admonition type="note" >}} There is a potential issue loading Tempo 1.5's experimental Parquet storage blocks. You may see errors or even panics in the compactors. We have only been able to reproduce this with interim commits between 1.5 and 2.0, but if you experience any issues please so we can isolate and fix this issue. {{% /admonition %}} Parquet provides faster search and is required to enable TraceQL. However, the Tempo installation will require additional CPU and memory resources to use Parquet efficiently. Parquet is more costly due to the extra work of building the columnar blocks, and operators should expect at least 1.5x increase in required resources to run a Tempo 2.0 cluster. Most users will find these extra resources are negligible compared to the benefits that come from the additional features of TraceQL and from storing traces in an open format. You can can continue using the previous `v2` block format using the instructions provided in the . Tempo will continue to support trace by id lookup on the `v2` format for the foreseeable future. TraceQL is enabled by default in Tempo 2.0. The TraceQL query editor requires Grafana 9.3.2 and later. The TraceQL query editor is in beta in Grafana 9.3.2 and needs to be enabled with the `traceqlEditor` feature flag. The following tables describe the parameters that have been removed or renamed. | Parameter | Comments | | | | | <pre>queryfrontend:<br>&nbsp;&nbsp;queryshards:</pre> | Replaced by `tracebyid.query_shards`. | | <pre>querier:<br>&nbsp;&nbsp;querytimeout:</pre> | Replaced by two different settings: `search.querytimeout` and `tracebyid.query_timeout`. | | <pre>ingester:<br>&nbsp;&nbsp;useflatbuffersearch:</pre> | Removed and automatically determined based on block format. | | `search_enabled` | Removed. Now defaults to true. | | `metricsgeneratorenabled` | Removed. Now defaults to true. | The following `compactor` configuration parameters were renamed. | Parameter | Comments | | | | | <pre>compaction:<br>&nbsp;&nbsp;chunksizebytes:</pre> | Renamed to `v2inbuffer_bytes` | | <pre>compaction:<br>&nbsp;&nbsp;flushsizebytes:</pre> | Renamed to `v2outbuffer_bytes` | | <pre>compaction:<br>&nbsp;&nbsp;iteratorbuffersize:</pre> | Renamed to `v2prefetchtraces_count` | The following `storage` configuration parameters were renamed. | Parameter | Comments | | | | | <pre>wal:<br>&nbsp;&nbsp;encoding:</pre> | Renamed to `v2_encoding` | | <pre>block:<br>&nbsp;&nbsp;indexdownsamplebytes:</pre> | Renamed to `v2indexdownsample_bytes` | | <pre>block:<br>&nbsp;&nbsp;indexpagesizebytes:</pre> | Renamed to `v2indexpagesize_bytes` | | <pre>block:<br>&nbsp;&nbsp;encoding:</pre> | Renamed to `v2_encoding` | | <pre>block:<br>&nbsp;&nbsp;rowgroupsizebytes:</pre> | Renamed to `parquetrowgroupsize_bytes` | The Azure Storage configuration section now uses snake case with underscores (`_`) instead of dashes (`-`). Example of using snake case on Azure Storage config: ```yaml storage: trace: azure: storageaccountname: storageaccountkey: container_name: ```