Project: Numaflow (App Definition and Development / Streaming & Messaging)
File: metrics.md

Numaflow provides the following prometheus metrics which we can use to monitor our pipeline and setup any alerts if needed. These metrics in combination can be used to determine the overall health of your pipeline These metrics can be used to determine throughput of your pipeline. | Metric name | Metric type | Labels | Description | | -- | -- | - | | | `forwarderdatareadtotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertextype=<vertex-type>` <br> `replica=<replica-index>` <br> `partition_name=<partition-name>` | Provides the total number of messages read by a given Vertex from an Inter-Step Buffer Partition | | `forwarderreadbytestotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertextype=<vertex-type>` <br> `replica=<replica-index>` <br> `partition_name=<partition-name>` | Provides the total number of bytes read by a given Vertex from an Inter-Step Buffer Partition | | `forwarderwritetotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertextype=<vertex-type>` <br> `replica=<replica-index>` <br> `partitionname=<partition-name>` | Provides the total number of messages written to Inter-Step Buffer by a given Vertex | | `forwarderwritebytestotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertextype=<vertex-type>` <br> `replica=<replica-index>` <br> `partition_name=<partition-name>` | Provides the total number of bytes written to Inter-Step Buffer by a given Vertex | | `forwarderacktotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertextype=<vertex-type>` <br> `replica=<replica-index>` <br> `partitionname=<partition-name>` | Provides the total number of messages acknowledged by a given Vertex from an Inter-Step Buffer Partition | | `forwarderdroptotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertextype=<vertex-type>` <br> `replica=<replica-index>` <br> `partitionname=<partition-name>` | Provides the total number of messages dropped by a given Vertex due to a full Inter-Step Buffer Partition | | `forwarderdropbytestotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertextype=<vertex-type>` <br> `replica=<replica-index>` <br> `partition_name=<partition-name>` | Provides the total number of bytes dropped by a given Vertex due to a full Inter-Step Buffer Partition | | Metric name | Metric type | Labels | Description | | - | -- | | | | `kafkasourceread_total` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` | Provides the number of messages read by the Kafka Source Vertex/Processor. | | `kafkasourceack_total` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` | Provides the number of messages acknowledged by the Kafka Source Vertex/Processor | | Metric name | Metric type | Labels | Description | | | -- | | | | `tickgensourceread_total` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` | Provides the number of messages read by the Generator Source

| | Metric name | Metric type | Labels | Description | | | -- | | - | | `httpsourceread_total` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` | Provides the number of messages read by the HTTP Source Vertex/Processor. | | Metric name | Metric type | Labels | Description | | | -- | | -- | | `kafkasinkwrite_total` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` | Provides the number of messages written by the Kafka Sink Vertex/Processor | | Metric name | Metric type | Labels | Description | | - | -- | | | | `logsinkwrite_total` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` | Provides the number of messages written by the Log Sink Vertex/Processor | These metrics can be used to determine the latency of your pipeline. | Metric name | Metric type | Labels | Description | | - | -- | - || | `sourceforwardertransformerprocessingtime` | Histogram | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertextype=<vertex-type>` <br> `replica=<replica-index>` <br> `partitionname=<partition-name>` | Provides a histogram distribution of the processing times of User-defined Source Transformer | | `forwarderudfprocessingtime` | Histogram | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertextype=<vertex-type>` <br> `replica=<replica-index>` | Provides a histogram distribution of the processing times of User-defined

(UDF's) | | `forwarderforwardchunkprocessingtime` | Histogram | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertex_type=<vertex-type>` <br> `replica=<replica-index>` | Provides a histogram distribution of the processing times of the forwarder function as a whole | | `reducepnfprocess_time` | Histogram | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `replica=<replica-index>` | Provides a histogram distribution of the processing times of the reducer | | `reducepnfforward_time` | Histogram | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `replica=<replica-index>` | Provides a histogram distribution of the forwarding times of the reducer | These metrics can be used to determine if there are any errors in the pipeline | Metric name | Metric type | Labels | Description | | | -- | - | | | `forwarderplatformerrortotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertextype=<vertex-type>` <br> `replica=<replica-index>` | Indicates any internal errors which could stop pipeline processing | | `forwarderreaderrortotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertextype=<vertex-type>` <br> `replica=<replica-index>` <br> `partition_name=<partition-name>` | Indicates any errors while reading messages by the forwarder | | `forwarderwriteerrortotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` `vertextype=<vertex-type>` <br> <br> `replica=<replica-index>` <br> `partition_name=<partition-name>` | Indicates any errors while writing messages by the forwarder | | `forwarderackerrortotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` <br> `vertextype=<vertex-type>` <br> `replica=<replica-index>` <br> `partition_name=<partition-name>` | Indicates any errors while acknowledging messages by the forwarder | | `kafkasourceoffsetackerrors` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` | Indicates any kafka acknowledgement errors | | `kafkasinkwriteerrortotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` | Provides the number of errors while writing to the Kafka sink | | `kafkasinkwritetimeouttotal` | Counter | `pipeline=<pipeline-name>` <br> `vertex=<vertex-name>` | Provides the write timeouts while writing to the Kafka sink | | `isbjetstreamreaderrortotal` | Counter | `partition_name=<partition-name>` | Indicates any read errors with NATS Jetstream ISB | | `isbjetstreamwriteerrortotal` | Counter | `partition_name=<partition-name>` | Indicates any write errors with NATS Jetstream ISB | | `isbredisreaderrortotal` | Counter | `partition_name=<partition-name>` | Indicates any read errors with Redis ISB | | `isbrediswriteerrortotal` | Counter | `partition_name=<partition-name>` | Indicates any write errors with Redis ISB | | Metric name | Metric type | Labels | Description | | - | -- | - | -- | | `isbjetstreamisFull_total` | Counter | `buffer=<buffer-name>` | Indicates if the ISB is full. Continual increase of this counter metric indicates a potential backpressure that can be built on the pipeline | | `isbjetstreambuffersoftusage` | Gauge | `buffer=<buffer-name>` | Indicates the usage/utilization of a NATS Jetstream ISB | | `isbjetstreambuffersolidusage` | Gauge | `buffer=<buffer-name>` | Indicates the solid usage of a NATS Jetstream ISB | | `isbjetstreambuffer_pending` | Gauge | `buffer=<buffer-name>` | Indicate the number of pending messages at a given point in time. | | `isbjetstreambufferackpending` | Gauge | `buffer=<buffer-name>` | Indicates the number of messages pending acknowledge at a given point in time | | Metric name | Metric type | Labels | Description | | | -- | - | -- | | `isbredisisFull_total` | Counter | `buffer=<buffer-name>` | Indicates if the ISB is full. Continual increase of this counter metric indicates a potential backpressure that can be built on the pipeline | | `isbredisbuffer_usage` | Gauge | `buffer=<buffer-name>` | Indicates the usage/utilization of a Redis ISB | | `isbredisconsumer_lag` | Gauge | `buffer=<buffer-name>` | Indicates the the consumer lag of a Redis ISB | You can follow the setup guide if you would like to use prometheus operator configured in your cluster. You can also set up prometheus operator via . ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: app.kubernetes.io/part-of: numaflow name: numaflow-pipeline-metrics spec: endpoints: scheme: https port: metrics targetPort: 2469 tlsConfig: insecureSkipVerify: true selector: matchLabels: app.kubernetes.io/component: vertex app.kubernetes.io/managed-by: vertex-controller app.kubernetes.io/part-of: numaflow matchExpressions: key: numaflow.numaproj.io/pipeline-name operator: Exists key: numaflow.numaproj.io/vertex-name operator: Exists ```