Project: Grafana Tempo (Observability and Analysis / Observability)
File: runbook.md

This document should help with remediation of operational issues in Tempo. Aside from obvious errors in the logs the only real lever you can pull here is scaling. Use the Reads or Writes dashboard to identify the component that is struggling and scale it up. The Query path is instrumented with tracing (!) and this can be used to diagnose issues with higher latency. View the logs of the Query Frontend, where you can find an info level message for every request. Filter for requests with high latency and view traces. The Query Frontend allows for scaling the query path by sharding queries. There are a few knobs that can be tuned for optimum parallelism - Number of shards each query is split into, configured via ``` query_frontend: tracebyid: query_shards: 10 ``` Number of Queriers (each of these process the sharded queries in parallel). This can be changed by modifying the size of the Querier deployment. More Queriers -> faster processing of shards in parallel -> lower request latency. Querier parallelism, which is a combination of a few settings: ``` querier: maxconcurrentqueries: 10 frontend_worker: matchmaxconcurrent: true // true by default parallelism: 5 // parallelism per query-frontend. ignored if matchmaxconcurrent is set to true storage: trace: pool: max_workers: 100 ``` MaxConcurrentQueries defines the total number of shards each Querier processes at a given time. By default, this number will be split between the query frontends, so if there are N query frontends, the Querier will process (Max Concurrent Queries/ N) queries per query frontend. Another way to increase parallelism is by increasing the size of the worker pool that queries the cache & backend blocks. A theoretically ideal value for this config to avoid any queueing would be (Size of blocklist / Max Concurrent Queries). But also factor in the resources provided to the querier. If trace lookups are fail with the error: `error querying store in Querier.FindTraceByID: queue doesn't have room for <xyz> jobs`, this means that the number of blocks has exceeded the queriers' ability to check them all. This can be caused by an increase in the number of blocks, or if the number of queriers was reduced. Check the following metrics and data points: Metric: `tempodbblocklistlength` - Look for a recent increase in blocks The number of queriers The queue_depth setting in the queriers: ``` storage: trace: pool: queue_depth: xyz (default 10000) ``` The queue won't have room when the number of blocks per querier exceeds the queue_depth: (blocklistlength / number of queriers) > queuedepth Consider the following resolutions: Increase the number of queriers Increase the queue_depth size to do more work per querier Adjust compaction settings to reduce the number of blocks If the request latency issues are due to backend searches with serverless/external endpoints there may be additional configuration options that will help decrease latency. Before you get started know that serverless functionality only impacts the `/api/search` endpoint if a `start` and `end` parameter are passed and `external_endpoints` are configured on the querier. One way to determine if external endpoints are getting hit is to check the Reads dashboard and look for the "Querier External Endpoint" row. Tuning serverless search can be difficult. Our includes a solid guide on the various parameters with

The linked documentation is augmented with some suggestions here: Consider provisioning more serverless functions and adding them to the `querier.search.external_endpoints` array. This will increase your baseline latency and your total throughput. Decreasing `querier.search.hedgerequestsat` and increasing `querier.search.hedgerequestsup_to` will put more pressure on the serverless endpoints but will result in lower latency. Increasing `querier.search.prefer_self` and scaling up the queriers will cause more work to be performed by the queriers which will lower latencies. Increasing `queryfrontend.maxoustandingpertenant` and `queryfrontend.search.concurrentjobs` will increase the rate at which the query_frontend tries to feed jobs to the queriers and can decrease latency. If this occurs access the at `/compactor/ring`. Use the "Forget" button to drop any unhealthy compactors. An unhealthy compactor or two has no immediate impact. Long term, however, it will cause the blocklist to grow unnecessarily long. If this occurs access the at `/distributor/ring`. Use the "Forget" button to drop any unhealthy distributors. An unhealthy distributor or two has virtually no impact except to slightly increase the amount of memberlist traffic propagated by the cluster. Check to determine the cause for the failures. Intermittent failures require no immediate action, because the compactors will reattempt the compaction, and any partially written data is ignored and expected to be cleaned up automatically with bucket cleanup rules in GCS/S3/etc. The most common cause for a failing compaction is an OOM from compacting an extremely large trace. The memory limit should be increased until it is enough to get past the trace, and must remain increased until the trace goes out of retention and is deleted, or else there is the risk of the trace causing OOMs later. Ingester limits should be reviewed and possibly reduced. If a block continues to cause problems and cannot be resolved it can be deleted manually. There are several settings which can be tuned to reduce the amount of work done by compactors to help with stability or scaling: compaction_window - The length of time that will be compacted together by a single pod. Can be reduced to as little as 15 or 30 minutes. It could be reduced even further in extremely high volume situations. maxblockbytes - The maximum size of an output block, and controls which input blocks will be compacted. Can be reduced to as little as a few GB to prevent really large compactions. v2inbuffer_bytes - The amount of (compressed) data buffered from each input block. Can be reduced to a few megabytes to buffer less. Will increase the amount of reads from the backend. flushsizebytes - The amount of data buffered of the output block. Can be reduced to flush more frequently to the backend. There are platform-specific limits on how low this can go. AWS S3 cannot be set lower than 5MB, or cause more than 10K flushes per block. How it works: Tempo ingesters flush blocks that have been completed to the backend If flushing fails, the ingester will keep retrying until restarted Blocks that have been flushed successfully will be deleted from the ingester, by default after 15m Failed flushes could be caused by any number of different things: bad block, permissions issues, rate limiting, failing backend, etc. Tempo will continue to retry sending the blocks until it succeeds, but at some point your WAL files will start failing to write due to out of disk issues. Known issue: this can trigger during a rollout of the ingesters, see

How to investigate: To check which ingesters are failing to flush and look for a pattern, you can use: ``` sum(rate(tempoingesterfailedflushestotal{cluster="...", container="ingester"}[5m])) by (pod) ``` If retries are failing it means that blocks are being reattempted. If this is only occurring to one block it strongly suggests block corruption: ``` increase(tempoingesterflushfailedretries_total) > 0 ``` Check the logs for errors If a single block can not be flushed, this block might be corrupted. A corrupted or bad block might be missing some files or a file might be empty, compare this block with other blocks in the WAL to verify. After inspecting the block, consider moving this file out of the WAL or outright deleting it. Restart the ingester to stop the retry attempts. Removing blocks from a single ingester will not cause data loss if replication is used and the other ingesters are flushing their blocks successfully. By default, the WAL is at `/var/tempo/wal/blocks`. If multiple blocks can not be flushed, the local WAL disk of the ingester will be filling up. Consider increasing the amount of disk space available to the ingester. See below for general information. If polls are failing check the component that is raising this metric and look for any obvious logs that may indicate a quick fix. We have only seen polling failures occur due to intermittent backend issues. See below for general information. If the following is being logged then things are stable (due to polling fallback) and we just need to review the logs to determine why there is an issue with the index and correct. ``` failed to pull bucket index for tenant. falling back to polling ``` If the following (or other errors) are being logged repeatedly then the tenant index is not being updated and more direct action is necessary. If the core issue can not be resolved delete any tenant index that is not being updated. This will force the components to fallback to bucket scanning for the offending tenants. ``` failed to write tenant index ``` See below for general information. If a cluster has no tenant index builders for a given tenant then nothing is refreshing the per tenant index. This can be dangerous b/c other components will not be aware there is an issue as they repeatedly download a stale tenant index. In Tempo the compactors play the role of building the tenant index. Ways to address this issue in order of preference: Find and . Increase the number of compactors that attempt to build the index. ``` storage: trace: blocklistpolltenantindexbuilders: 2 # <- increase this value ``` Delete tenant index files that are not being updated to force other components to fallback to scanning for these tenants. They are located at `/<tenant>/index.json.gz` in the configured long-term/object-storage backend. See below for general information. If the tenant indexes are too old we need to review the compactor logs to determine why they are failing to update. Compactors with `tempodbblocklisttenantindexbuilder` for the offending tenant set to 1 are expected to be creating the indexes for that tenant and should be checked first. If no compactors are creating tenant indexes refer to above. Additionally the metric `tempodbblocklisttenantindexage_seconds` can be grouped by the `tenant` label. If only one (or few) indexes are lagging these can be deleted to force components to manually rescan just the offending

In the case of all polling issues intermittent issues are not concerning. Sustained polling issues need to be addressed. Failure to poll just means that the component is not aware of the current state of the backend but will continue working otherwise. Queriers, for instance, will start returning 404s as their internal representation of the backend grows stale. Compactors will attempt to compact blocks that don't exist. If persistent backend issues are preventing any fixes to polling then reads will start to fail, but writes will remain fine. Alert your users accordingly! Note that tenant indexes are built independently and an issue may only be impacting one or very few tenants. `tempodbblocklisttenantindexbuilder`, `tempodbblocklisttenantindexageseconds` and `tempodbblocklisttenantindexerrorstotal` are all per-tenant metrics. If you can isolate the impacted tenants, attempt to take targeted action instead of making sweeping changes. Your easiest lever to pull is to simply delete stale tenant indexes as all components will fallback to bucket listing. The tenant index is located at: ``` /<tenant>/index.json.gz ``` The block list needs to remain under control to keep query performance acceptable. If the block list is rising too quickly, this might indicate the compactors are under scaled. Add more compactors until the block list is back under control and holding mostly steady. Fix the overrides! Overrides are loaded by the distributors so hopefully there is some meaningful logging there. This alert fires if the average number of samples ingested / sec in ingesters is above our target. How to fix: Scale up ingesters To compute the desired number of ingesters to satisfy the average samples rate you can run the following query, replacing <namespace> with the namespace to analyse and <target> with the target number of samples/sec per ingester (check out the alert threshold to see the current target): ``` sum(rate(tempoingesterbytesreceivedtotal{namespace="<namespace>"}[$rate_interval])) / (<target> * 0.9) ``` This alert fires when there are too many blocks to be compacted for a long period of time. The alert does not require immediate action, but is a symptom that compaction is underscaled and could affect the read path in particular. How to fix: Compaction's bottleneck is most commonly CPU time, so adding more compactors is the most effective measure. After compaction has been scaled out, it'll take a time for compactors to catch up with their outstanding blocks. Take a look at `tempodbcompactionoutstanding_blocks` and check if blocks start going down. If not, further scaling may be necessary. Since the number of blocks is elevated, it may also be necessary to review the queue-related settings to prevent . This alert fires when an ingester has encountered an error while replaying a block on startup. How to fix: Check the ingester logs for errors to identify the culprit ingester, tenant, and block ID. If an ingester is restarted unexpectedly while writing a block to disk, the files might be corrupted. The error "Unexpected error reloading meta for local block. Ignoring and continuing." indicates there was an error parsing the meta.json. Repair the meta.json and then restart the ingester to successfully recover the block. Or if it is not able to be repaired then the block files can be simply deleted as the ingester has already started without it. As long as the replication factor is 2 or higher, then there will be no data loss as the same data was also written to another ingester.