Project: Grafana Tempo (Observability and Analysis / Observability)
File: metrics-generator.md

title: Troubleshoot metrics-generator menuTitle: Metrics-generator description: Gain an understanding of how to debug metrics quality issues. weight: 500 aliases: ../operations/troubleshooting/metrics-generator/ If you are concerned with data quality issues in the metrics-generator, we'd first recommend: Reviewing your telemetry pipeline to determine the number of dropped spans. We are only looking for major issues here. Reviewing the to understand how they are built. If everything seems ok from these two perspectives, consider the following topics to help resolve general issues with all metrics and span metrics specifically. The distributor has a queue of outgoing spans to the metrics-generators. If that queue is full then the distributor will drop spans before they reach the generator. Use the following metric to determine if that is happening: ``` sum(rate(tempodistributorqueuepushesfailures_total{}[1m])) ``` For any number of reasons, the distributor can fail a push to the generators. Use the following metric to determine if that is happening: ``` sum(rate(tempodistributormetricsgeneratorpushesfailurestotal{}[1m])) ``` Spans are rejected from being considered by the metrics-generator by a configurable slack time as well as due to user configurable filters. You can see the number of spans rejected by reason using this metric: ``` sum(rate(tempometricsgeneratorspansdiscarded_total{}[1m])) by (reason) ``` If a lot of spans are dropped in the metrics-generator due to your filters, you will need to adjust them. If spans are dropped due to the ingestion slack time, consider adjusting this setting: ``` metrics_generator: metricsingestiontimerangeslack: 30s ``` If spans are regularly exceeding this value you may want to consider reviewing your tracing pipeline to see if you have excessive buffering. Note that increasing this value allows the generator to consume more spans, but does reduce the accuracy of metrics because spans farther away from "now" are included. The generator protects itself and your remote-write target by having a maximum number of series the generator produces. Use the `sum` below to determine if series are being dropped due to this limit: ``` sum(rate(tempometricsgeneratorregistryserieslimitedtotal{}[1m])) ``` Use the following setting to update the limit: ```yaml overrides: defaults: metrics_generator: maxactiveseries: 0 ``` Note that this value is per metrics generator. The actual max series remote written will be `<# of metrics generators> * <metricsgenerator.maxactive_series>`. For any number of reasons, the generator may fail a write to the remote write target. Use the following metrics to determine if that is happening: ``` sum(rate(prometheusremotestoragesamplesfailed_total{}[1m])) sum(rate(prometheusremotestoragesamplesdropped_total{}[1m])) sum(rate(prometheusremotestorageexemplarsfailed_total{}[1m])) sum(rate(prometheusremotestorageexemplarsdropped_total{}[1m])) ``` Service graphs have additional configuration which can impact the quality of the output metrics. The following metrics can be used to determine how many edges are failing to find a match. Rate of edges that have expired without a match: ``` sum(rate(tempometricsgeneratorprocessorservicegraphsexpired_edges{}[1m])) ``` Rate of all edges: ``` sum(rate(tempometricsgeneratorprocessorservicegraphsedges{}[1m])) ``` If you are seeing a large number of edges expire without a match, consider adjusting the `wait` setting. This controls how long the metrics generator waits to find a match before it gives up. ``` metrics_generator: processor: service_graphs: wait: 10s ``` The service graph processor has a maximum number of edges it will track at once to limit the total amount of memory the processor uses. To determine if edges are being dropped due to this limit, check: ``` sum(rate(tempometricsgeneratorprocessorservicegraphsdropped_spans{}[1m])) ``` Use `max_items` to adjust the maximum amount of edges tracked: ``` metrics_generator: processor: service_graphs: max_items: 10000 ```