Repository: redpanda-data/connect

Issue #1066: Metrics changes in V4
Created: 2022-01-17T19:59:42Z
Metrics in V4
A Benthos instance has always offered an abundance of metrics, and since graphite (with the statsd exporter) was the first ever target for these metrics they were limited to the name of the metric being used as the distinction for a given series. Therefore, in order to have unique metrics for individual inputs, processors, outputs, etc the name of the metric paths for each component was generated based on its position within the config (not an exact path).
For example, the following config:
input:
  broker:
    inputs:
      - generate: {}
      - kafka: {}

pipeline:
  processors:
    - bloblang: 'root.foo = this.foo' 
    - catch:
      - bloblang: 'root.bar = content().uppercase()' 

output:
  switch:
    cases:
      - check: this.foo
        output:
          amqp_0_9: {}
      - output:
          nats: {}
You would produce the following count metric names:

input.count
input.broker.inputs.0.count
input.broker.inputs.1.count
pipeline.processor.0.count
pipeline.processor.1.count
output.count
output.switch.0.output.count
output.switch.1.output.count

These names become unreasonably long as pipeline configs become more and more complex. This was partially solved when the label field was introduced, as it meant that for components with labels the metric name would become prefixed with that label instead of the path (e.g. foo.count). However, this system still means that unlabelled components pollute your metrics output and force you to tidy them manually using a path_mapping.
Since this initial work was done new players in the metrics space (Prometheus, Datadog, etc) have leant into the concept of labels or tags as a way to distinguish series of data, and instead encourage the cardinality of metric names to be low. This system is much easier to work with in a path_mapping as you can filter on specific fields rather than using regular expressions on segments of the name.
For Benthos v4 I'm proposing we change the way in which metrics are created so that labels/tags are the default way that components are identified, and metric names become generic. We would have a component label that identifies component (by its config label), and also (maybe) a path label that outlines the position of the component in the config. This would result in the following config:
input:
  broker:
    inputs:
      - label: foo
        generate: {}
      - kafka: {}

pipeline:
  processors:
    - bloblang: 'root.foo = this.foo' 
    - label: bar
      catch:
        - bloblang: 'root.bar = content().uppercase()' 

output:
  label: baz
  switch:
    cases:
      - check: this.foo
        output:
          amqp_0_9: {}
      - output:
          label: quz
          nats: {}
Producing the following count metric names by default:

input_count{component="",path="input"}
input_count{component="foo",path="input.broker.inputs.0"}
input_count{component="",path="input.broker.inputs.1"}
processor_count{component="",path="pipeline.processors.0"}
processor_count{component="bar",path="pipeline.processors.1"}
processor_count{component="",path="pipeline.processors.1.0"}
output_count{component="baz",path="output"}
output_count{component="",path="output.switch.cases.0.output"}
output_count{component="quz",path="output.switch.cases.1.output"}

Note that I'm not fully convinced we really need a path label. I might omit it and maybe introduce it later if there's demand for it. With these metrics names you could easily ignore any metrics for components that aren't labelled with a path_mapping of root = if meta("component").catch("") == "" { deleted() }.
Removing Prefix
Right now almost all metrics types have a prefix field with benthos as the default. This is no longer necessary as you can insert any type of prefix that you like with the path_mapping field so I think it makes sense to remove it.
Defaulting to Prometheus
I think it seems reasonable to say at this stage that Prometheus is becoming the more common metrics destination, and since the existing default for Benthos is the (label hostile) JSON http endpoint we can probably swap to providing prometheus metrics as the default without causing much aggro. It will still be easy to quickly glance at metrics with /stats as the prometheus format is quite readable as it is.
Migration
Ideally we want a reasonable migration story so that users of v3 can fix up their dashboards. In cases where users already use the label field in components they wish to track they can mostly get away with mappings like the following:
metrics:
  prometheus:
    path_mapping: |
      root = if meta("component").catch("") == "" {
        deleted()
      } else {
        "benthos_" + this.re_replace("^(input|output|processor)", meta("component"))
      }
      meta path = deleted()
      meta component = deleted()
Which would result in the following metrics (if starting with the previous config):

benthos_foo_count
benthos_bar_count
benthos_baz_count
benthos_quz_count

However, ultimately there will need to be some work done by users in order to get their old dashboards working. Therefore I'm looking for feedback here as to whether this new system is better enough to be worth that effort. Keep in mind that there will definitely be some breaking changes with metrics as the current system is odd in many ways, but we can definitely tune how different it's going to be.

Duplicate issue #1060: Prometheus input latency as a histogram metric type
Created: 2022-01-13T23:51:05Z
The input latency metric for Prometheus is currently exposed as a summary. This makes it difficult for creating dashboards to view quantiles for the following reasons:

Summary quantiles cannot be aggregated
It can be difficult to deduce what timeframe the quantiles cover when using a summary
Summaries calculate quantiles on the client side

It seems like the consensus in the community is to use histograms rather than summaries for quantiles.
Here are some helpful resources to provide further context:

In general, summaries are not recommended for use cases where a histogram can be used instead. This is because quantiles cannot be aggregated, and it can be difficult to deduce what timeframe the quantiles cover. Note: this is defined by each client library independently (e.g. Prometheus Go client library uses 10 minutes by default).

- https://chronosphere.io/learn/an-introduction-to-the-four-primary-types-of-prometheus-metrics/

This is one of the big problems with calculating quantiles on the client side. If it's the quantile since the processes started, then the samples get less and less relevant to current conditions as time goes on. It's also not possible to do any math or aggregations once you have a quantile, so you can't stitch things back together given the history of the time series.

- https://www.robustperception.io/how-does-a-prometheus-summary-work

The essential difference between summaries and histograms is that summaries calculate streaming φ-quantiles on the client side and expose them directly, while histograms expose bucketed observation counts and the calculation of quantiles from the buckets of a histogram happens on the server side using the histogram_quantile() function.

- https://prometheus.io/docs/practices/histograms/#quantiles
^ This link is from the official Prometheus documentation and has a handy graph covering the primary differences between histograms and summaries.

Closing comment (2022-01-25T19:44:09Z): This seems related to #1066. I'm happy to close this ticket and move the discussion there.