Project: Grafana Mimir (Observability and Analysis / Observability)
File: configure-custom-trackers.md

aliases: ../configuring/configuring-custom-trackers/ configuring-custom-trackers/ ../operators-guide/configure/configure-custom-trackers/ description: Use the custom tracker to count the number of active series on an ingester. menuTitle: Custom active series trackers title: Configure custom active series trackers weight: 100 You can use the custom tracker feature to count the number of active series on an ingester that match a particular label pattern. The label pattern to match against is specified using the `-ingester.active-series-custom-trackers` CLI flag (or its respective YAML configuration option). Each custom tracker is defined as a key-value pair, where the key is the name of the tracker and the value is the label matcher. Both the key and the value are type `<string>`. The following example configures a custom tracker to count the active series coming from `dev` and `prod` namespaces for each tenant. ```yaml activeseriescustom_trackers: dev: '{namespace=~"dev-.*"}' prod: '{namespace=~"prod-.*"}' ``` If you configure a custom tracker for an ingester, the ingester exposes a `cortexingesteractiveseriescustom_tracker` gauge metric on its . Each custom tracker counts the active series matching its label pattern on a per-tenant basis, which means that each custom tracker generates as many as `# of tenants` series with metric name `cortexingesteractiveseriescustom_tracker`. To reduce the cardinality of this metric, only custom trackers that have matched at least one series are exposed on the metric, and they are removed if they become `0`. Series with metric name `cortexingesteractiveseriescustom_tracker` have two labels applied: `name` and `user`. The value of the `name` label is the name of the custom tracker specified in the configuration. The value of the `user` label is the tenant-id for which the series count applies. To illustrate this, assume that two custom trackers are configured as in the preceding YAML snippet, and that your Grafana Mimir cluster has two tenants: `tenant1` and `tenantwithonlyprodmetrics`. Assume that `tenantwithonlyprodmetrics` has three series with labels that match the pattern `{namespace=~"prod-.*"}` and none that match the pattern `{namespace=~"dev-.*"}`. Also assume that `tenant1` has five series that match the pattern `{namespace=~"dev-."}` and 10 series that match the pattern `{namespace=~"prod-."}`. In this example, the following output appears when the `/metrics` endpoint for the ingester component is scraped: ``` cortexingesteractiveseriescustomtracker{name="dev", user="tenant1"} 5 cortexingesteractiveseriescustomtracker{name="prod", user="tenant1"} 10 cortexingesteractiveseriescustomtracker{name="prod", user="tenantwithonlyprod_metrics"} 3 ``` For specific tenants, you can override the default configuration as previously described. To do so, edit the . You can override the active series custom trackers configuration for the tenant `tenantwithonlyprodmetrics` to track two services instead of the default matchers. See the following example: ``` overrides: tenantwithonlyprodmetrics: activeseriescustom_trackers: service1: '{service="service1"}' service2: '{service="service2"}' ``` After adding this override, and assuming that there is one matching series for `service1` and two matching series for `service2`, the output at `/metrics` changes: ``` cortexingesteractiveseriescustomtracker{name="dev", user="tenant1"} 5 cortexingesteractiveseriescustomtracker{name="prod", user="tenant1"} 10 cortexingesteractiveseriescustomtracker{name="service1", user="tenantwithonlyprod_metrics"} 1 cortexingesteractiveseriescustomtracker{name="service2", user="tenantwithonlyprod_metrics"} 2 ``` To set up runtime overrides, refer to . {{< admonition type="note" >}} Each ingester exposes the custom active series trackers. To understand the count of active series matching a particular label pattern in your Grafana Mimir cluster at a global level, you must collect and sum this metric across all ingesters. If you're running Grafana Mimir with a `replication_factor` > 1, you must also adjust for the fact that Mimir replicates the same series across your ingesters replication factor times. {{< /admonition >}}