Project: Kuma (Orchestration & Management / Service Mesh)
File: 004-metric-filter.md

Status: accepted Technical Story: https://github.com/kumahq/kuma/issues/1755 Envoy by default produces a lot of metrics. You may be interested only in subset of them to not stress your metric infrastructure. We need to have a mechanism to filter metrics. Filter metrics dynamically on scrape request Filter metrics statically on Envoy bootstrap Customizable bootstrap Chosen option: "Filter metrics dynamically on scrape request", because it follows Kuma philosophy of being as dynamic as possible. Additionally, as a followup we should implement "Customizable bootstrap", because it both enables static metrics and other advanced use cases. This is a subject for another MADR that will take into account more use cases. UX: ```yaml apiVersion: kuma.io/v1alpha1 kind: Mesh metadata: name: default spec: metrics: enabledBackend: prometheus-1 backends: name: prometheus-1 type: prometheus conf: skipMTLS: false port: 5670 path: /metrics envoy: filterRegex: "envoy.cluster.*" usedOnly: false ``` Implementation: Envoy can receive one Note: filter works on statsd names, not prometheus names, when configuring filter you need to take that into account. You can only provide one filter, so `envoyFilterRegex` cannot be a list. Then, in prometheusendpointgenerator.go we can configure route to add this query param, so the `:5670` is still `/metrics`, but it adds the filter under the hood. Then, metrics hijacker receives a request with query param and there are two options: It passes it only to Envoy. It passes it to Envoy and all the apps specified in aggregate. I think the first option is safer. Used only is a feature of Envoy to only publish metrics . It's useful in the environments when you don't set reachable services, or you want to restrict drastically set of published metrics and you understand that you will be missing 0 values. However, this might be confusing when you start with Kuma and metrics, so I think it should be opt-in. It's dynamic. You can see the result right away. You can adjust this quicker. You may want to enable all the metrics for a minute or two and then switch it back. Complex filter is hard to write (you cannot have a list of filters). Even if you don't care about some metrics, they are still kept in Envoy memory. UX: ```yaml apiVersion: kuma.io/v1alpha1 kind: Mesh metadata: name: default spec: metrics: enabledBackend: prometheus-1 backends: name: prometheus-1 type: prometheus conf: skipMTLS: false port: 5670 path: /metrics tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS kuma.io/service: dataplane-metrics envoy: staticFilter: rejectAll: true|false inclusion: match: exact: ... prefix: ... suffix: ... contains: ... regex: ... exclusion: [] # same ^ aggregate: name: my-service # name of the metric, required to later disable/override with pod annotations path: "/metrics/prometheus" port: 8888 name: other-sidecar port: 8000 ``` After applying the setting, you need to restart the proxy for setting to be applied. Implementation: Envoy exposes a setting to . It cuts the metrics in Envoy itself reducing memory usage. It's easier to write inclusion or exclusion lists. You can by accident introduce undefined behaviour. From docs: Excluding stats may affect Envoys behavior in undocumented ways. It requires proxy restart Like second option, but more general. It would be similar to the ContainerPatch when you can manipulate bootstrap using json patch. It gives users freedom to manipulate bootstrap the way they want including metrics. All advantages from the second option Brings more functionality to users All disadvantages from the second option More complex to use, although it is a separate feature.