Repository: matrix-org/synapse

Issue:
`synapse_storage_transaction_time_bucket` prometheus metric has too high cardinality due to desc label
### Description

The `desc` label on `synapse_storage_transaction_time_bucket` results in a very high cardinality metric. For a single instance, there's 248 variants of `desc`, multiplied by 15 buckets. This results in over 3k series for a single host.

Though this might be acceptable if you're only ingesting metrics for a single instance, for Prometheus instances that might be scraping multiple Synapses this quickly becomes a problem.

On one of our internal clusters this is resulting in over a million time series, which is about 5x the amount of time series of the next most problematic timeseries: [`synapse_http_server_response_time_seconds_bucket`](https://github.com/matrix-org/synapse/issues/11082). Though this is not causing storage issues, it causes unnecessarily high CPU load and memory usage ballooning on the ingesters (so we have to run with much bigger instances) and querying these series become problematic.

### Steps to reproduce

- Run lots of Synapse instances
- Scrape them all with a single Prometheus instance
- Prometheus gets sad

### Version information

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**: any
- **Version**: any since this metric got introduced
- **Install method**: unrelated
- **Platform**: unrelated


Discussion:
For reference, based on `promtool tsdb analyze` on one of our big clusters:

```
Highest cardinality metric names:
1054110 synapse_storage_transaction_time_bucket
212280 synapse_http_server_response_time_seconds_bucket
78990 synapse_storage_query_time_bucket
75546 synapse_util_caches_cache_evicted_size
```
I'd be pretty happy to drop this metric altogether. It's not used on our default grafana dashboard.
While we're in the area, it might be worth noting that lots of prometheus metrics are duplicated: #11106.

It's not quite a problem on the same scale as `synapse_storage_transaction_time_bucket`, but those cache metrics are non-trivial.
> I'd be pretty happy to drop this metric altogether. It's not used on our default grafana dashboard.

Yup, I noticed that too. Talking to @erikjohnston he was happy for us to stop ingesting this metric altogether, which strongly suggests we can drop it from the code base.
@08d2 noticed this in #12263. Is there anything stopping us from dropping these metrics altogether?
see #11124: turns out that `synapse_storage_transaction_time_count` and `synapse_storage_transaction_time_sum` are useful, so we need to be slightly more intelligent than just ripping them out.
I agree there's no reason for this to be a histogram and can easily be a summary. Histograms really only make sense when you want to perform statistical aggregations over multiple instances of the same service, but I guess nobody really runs multiple Synapse servers.

But is there anything which actually, definitely, consumes this metric? Are there specific operational use cases which need this level of granularity? If those requirements can't be explicitly enumerated, then IMO the metric can safely be eliminated. "Nice to have" isn't by itself sufficient justification for a telemetry signal :) _edit_: I understand there's a dashboard which uses this data, but does it really need the level of granularity currently on offer, or can it provide similar utility with a less-precise label set?
we have found these graphs on our grafana dashboard. They are even discussed in https://matrix-org.github.io/synapse/latest/usage/administration/understanding_synapse_through_grafana_graphs.html#transaction-count-and-transaction-duration. I would not be in favour of removing them.
Sure, dashboards exist which consume them. But do those dashboards actually provide meaningful operational telemetry? The cardinality of this metric is extraordinarily high. It should deliver extraordinary value to justify that cost. Does it?
yes
sorry, I just realised I missed a word in my previous comment. It should read:

> we have found these graphs [very] **useful** on our grafana dashboard.
@richvdh Sweet! What problems have you been able to diagnose and fix? Which servlets have been optimized? Do you have links to issues? If there's data, I'm happy to withdraw my objections.
For my own edification:

For each `desc` we are currently produce the following timeseries:
- [`len(DEFAULT_BUCKETS)` = 15](https://github.com/prometheus/client_python/blob/748ffb00600dc25fbd22d37d549578e8e370d996/prometheus_client/metrics.py#L544) cumulative bucket counter series
- a sum counter, of all events seen.
- a count of all events seen.

So I count 17 time series per named database transaction.

If we just want to track the amount of time spent in a type of transaction (e.g. `get_auth_chain_ids`) _across multiple instances of that transaction without knowing how many_, then a single counter would suffice.

If we want to track the mean time a single transaction instance takes, we'd need at least two counters (one for tracking duration, and one for tracking how many txns occur).

Prometheus histograms classify events based on fixed bucket thresholds, whereas Prometheus track the values of a given set of fixed percentiles. (Histogram: "how many requests take < 50ms"? Summary: "How long does it take to serve our quickest 99% of requests?) Or at least, that's how I'm interpreting https://prometheus.io/docs/concepts/metric_types/#histogram.

A histogram that doesn't track any buckets and a summary that doesn't track any quartiles look the same to me.

Additionally, the python client [doesn't support tracking quantiles](https://github.com/prometheus/client_python#summary).
Action for this issue, then:
- use a summary instead of a histogram. This should track 8x (17/2) less data per `desc`.

If metrics are still eating up space in the future, we could
- Consider reducing the number of buckets on other histograms, or convert them to summaries
- Gather together related queries under the same `desc`.

Resolving patch:
diff --git a/synapse/storage/database.py b/synapse/storage/database.py
--- a/synapse/storage/database.py
+++ b/synapse/storage/database.py
@@ -39,7 +39,7 @@
 )
 
 import attr
-from prometheus_client import Histogram
+from prometheus_client import Counter, Histogram
 from typing_extensions import Concatenate, Literal, ParamSpec
 
 from twisted.enterprise import adbapi
@@ -76,7 +76,8 @@
 sql_scheduling_timer = Histogram("synapse_storage_schedule_time", "sec")
 
 sql_query_timer = Histogram("synapse_storage_query_time", "sec", ["verb"])
-sql_txn_timer = Histogram("synapse_storage_transaction_time", "sec", ["desc"])
+sql_txn_count = Counter("synapse_storage_transaction_time_count", "sec", ["desc"])
+sql_txn_duration = Counter("synapse_storage_transaction_time_sum", "sec", ["desc"])
 
 
 # Unique indexes which have been added in background updates. Maps from table name
@@ -795,7 +796,8 @@ def new_transaction(
 
             self._current_txn_total_time += duration
             self._txn_perf_counters.update(desc, duration)
-            sql_txn_timer.labels(desc).observe(duration)
+            sql_txn_count.labels(desc).inc(1)
+            sql_txn_duration.labels(desc).inc(duration)
 
     async def runInteraction(
         self,


Test patch: