Project: SkyWalking (Observability and Analysis / Observability)
File: backend-kafka-monitoring.md

SkyWalking leverages Prometheus JMX Exporter to collect metrics data from the Kafka and leverages OpenTelemetry Collector to transfer the metrics to and into the . Kafka entity as a `Service` in OAP and on the `Layer: KAFKA`. The `prometheusJMXExporter` collect metrics data from Kafka. Note: Running the exporter as a Java agent. OpenTelemetry Collector fetches metrics from `prometheusJMXExporter` via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter. The SkyWalking OAP Server parses the expression with to filter/calculate/aggregate and store the results. Setup . This is an example for JMX Exporter configuration . Set up . The example for OpenTelemetry Collector configuration, refer to . Config SkyWalking . Kafka monitoring provides multidimensional metrics monitoring of Kafka cluster as `Layer: KAFKA` `Service` in the OAP. In each cluster, the kafka brokers are represented as `Instance`. | Monitoring Panel | Metric Name | Description | Data Source | |-|-|--|| | Under-Replicated Partitions | meterkafkaunderreplicatedpartitions | Number of under-replicated partitions in the broker. A higher number is a sign of potential issues. | Prometheus JMX Exporter | | Offline Partitions Count | meterkafkaofflinepartitionscount | Number of partitions that are offline. Non-zero values indicate a problem. | Prometheus JMX Exporter | | Partition Count | meterkafkapartition_count | Total number of partitions on the broker. | Prometheus JMX Exporter | | Leader Count | meterkafkaleader_count | Number of leader partitions on this broker. | Prometheus JMX Exporter | | Active Controller Count | meterkafkaactivecontrollercount | The number of active controllers in the cluster. Typically should be 1. | Prometheus JMX Exporter | | Leader Election Rate | meterkafkaleaderelectionrate | The rate of leader elections per minute. High rate could be a sign of instability. | Prometheus JMX Exporter | | Unclean Leader Elections Per Second | meterkafkauncleanleaderelectionspersecond | The rate of unclean leader elections per second. Non-zero values indicate a serious problem. | Prometheus JMX Exporter | | Max Lag | meterkafkamax_lag | The maximum lag between the leader and followers in terms of messages still needed to be sent. Higher lag indicates

| Prometheus JMX Exporter | | Monitoring Panel | Unit | Metric Name | Description | Data Source | |--|--|-||-| | CPU Usage | % | meterkafkabrokercputime_total | CPU usage in percentage | Prometheus JMX Exporter | | Memory Usage | % | meterkafkabrokermemoryusage_percentage | JVM heap memory usage in percentage | Prometheus JMX Exporter | | Incoming Messages | Msg/sec | meterkafkabrokermessagesper_second | Rate of incoming messages | Prometheus JMX Exporter | | Bytes In | Bytes/sec | meterkafkabrokerbytesinpersecond | Rate of incoming bytes | Prometheus JMX Exporter | | Bytes Out | Bytes/sec | meterkafkabrokerbytesoutpersecond | Rate of outgoing bytes | Prometheus JMX Exporter | | Replication Bytes In | Bytes/sec | meterkafkabrokerreplicationbytesinper_second | Rate of incoming bytes for replication | Prometheus JMX Exporter | | Replication Bytes Out | Bytes/sec | meterkafkabrokerreplicationbytesoutper_second | Rate of outgoing bytes for replication | Prometheus JMX Exporter | | Under-Replicated Partitions | Count | meterkafkabrokerunderreplicated_partitions | Number of under-replicated partitions | Prometheus JMX Exporter | | Under Min ISR Partition Count | Count | meterkafkabrokerunderminisrpartition_count | Number of partitions below the minimum ISR (In-Sync Replicas) | Prometheus JMX Exporter | | Partition Count | Count | meterkafkabrokerpartitioncount | Total number of partitions | Prometheus JMX Exporter | | Leader Count | Count | meterkafkabrokerleadercount | Number of partitions for which this broker is the leader | Prometheus JMX Exporter | | ISR Shrinks | Count/sec | meterkafkabrokerisrshrinkspersecond | Rate of ISR (In-Sync Replicas) shrinking | Prometheus JMX Exporter | | ISR Expands | Count/sec | meterkafkabrokerisrexpandspersecond | Rate of ISR (In-Sync Replicas) expanding | Prometheus JMX Exporter | | Max Lag | Count | meterkafkabrokermaxlag | Maximum lag between the leader and follower for a partition | Prometheus JMX Exporter | | Purgatory Size | Count | meterkafkabrokerpurgatorysize | Size of purgatory for Produce and Fetch operations | Prometheus JMX Exporter | | Garbage Collector Count | Count/sec | meterkafkabrokergarbagecollector_count | Rate of garbage collection cycles | Prometheus JMX Exporter | | Requests Per Second | Req/sec | meterkafkabrokerrequestsper_second | Rate of requests to the broker | Prometheus JMX Exporter | | Request Queue Time | ms | meterkafkabrokerrequestqueuetimems | Average time a request spends in the request queue | Prometheus JMX Exporter | | Remote Time | ms | meterkafkabrokerremotetime_ms | Average time taken for a remote operation | Prometheus JMX Exporter | | Response Queue Time | ms | meterkafkabrokerresponsequeuetimems | Average time a response spends in the response queue | Prometheus JMX Exporter | | Response Send Time | ms | meterkafkabrokerresponsesendtimems | Average time taken to send a response | Prometheus JMX Exporter | | Network Processor Avg Idle | % | meterkafkabrokernetworkprocessoravgidle_percent | Percentage of idle time for the network processor | Prometheus JMX Exporter | | Topic Messages In Total | Count | meterkafkabrokertopicmessagesintotal | Total number of messages per topic | Prometheus JMX Exporter | | Topic Bytes Out Per Second | Bytes/sec | meterkafkabrokertopicbytesoutpersecond | Rate of outgoing bytes per topic | Prometheus JMX Exporter | | Topic Bytes In Per Second | Bytes/sec | meterkafkabrokertopicbytesinpersecond | Rate of incoming bytes per topic | Prometheus JMX Exporter | | Topic Fetch Requests Per Second | Req/sec | meterkafkabrokertopicfetchrequestsper_second | Rate of fetch requests per topic | Prometheus JMX Exporter | | Topic Produce Requests Per Second | Req/sec | meterkafkabrokertopicproducerequestsper_second | Rate of produce requests per topic | Prometheus JMX Exporter | You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in `/config/otel-rules/kafka/kafka-cluster.yaml, /config/otel-rules/kafka/kafka-node.yaml`. The Kafka dashboard panel configurations are found in `/config/ui-initialized-templates/kafka`. For more details on