Project: Grafana Tempo (Observability and Analysis / Observability)
File: cardinality.md

aliases: /docs/tempo/latest/metrics-generator/cardinality title: Cardinality menuTitle: Cardinality description: What is cardinality and how it is impacted by metrics generation? weight: 100 Cardinality refers to the total combination of key/value pairs, such as labels and label values for a given metric series or log stream, and how many unique combinations they generate. For more information on cardinality, see the blog post. Because writes to a time-series database (TSDB) database are in series, high cardinality does not make a big difference to performance at ingest. However, cardinality can have a major impact on querying where, the higher the cardinality, the more items are required to be iterated over. Tempos server-side metrics generation adds functionality to the collection of traces by creating Prometheus-based metrics that track a variety of metrics such as: Total span call counts Span latency histograms Total span size count The metrics-generator creates metrics which define the relationship between services via edges and nodes. Each of these metrics are queryable using a set of Prometheus labels (key/value pairs). Each new value for a label increases the number of active series associated with a metric. (To learn more about active series, read the documentation.) This is also known as an increase in cardinality, and the number of active series generated for a metric is directly proportional to the number of labels that exist for that metrics alongside the number of values each label has added. In a non-modified instance of the metrics generator, a small number of labels are added automatically. Because labels like `spankind` and `statuscode` only have a few valid values, the largest variable for the number of active series produced for each metric depends on the number of service names and span names associated with trace spans. The metrics-generator can also be configured to also add extra labels on metrics, using span attribute key/value pairs which are mapped directly to these labels see the . Be careful when configuring custom attributes: the greater the number of values seen in a specific attribute, the greater the number of active series will be produced. For more information about active series, refer to the Let's say that you are adding a custom attribute that includes unique customer IDs as a metrics label. If you have 100 customers, this could potentially multiply the number of active series generated by up to 100 (for example, going from 25,000 active series to 2.5M). Always consider which attributes will actually be useful as labels for querying metrics, as well as the cardinality that they will increase metrics by. An often most reliable solution is by running the metrics-generator in a dry-run mode. Using the dry-run mode generates metrics but does not collecting them, thus not writing them to a metrics storage. The override `metricsgenerator.disablecollection` is defined for this use-case. To get an estimate, run the metrics-generator normally and set the override to `true`. Then, check `tempometricsgeneratorregistryactive_series` to get an estimation of the active series for that set-up.