Project: SkyWalking (Observability and Analysis / Observability)
File: backend-clickhouse-monitoring.md

SkyWalking leverages ClickHouse built-in metrics data since v20.1.2.4. It leverages OpenTelemetry Collector to transfer the metrics to and into the . Configure ClickHouse to expose metrics data for scraping from Prometheus. OpenTelemetry Collector fetches metrics from ClickeHouse server through Prometheus endpoint, and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter. The SkyWalking OAP Server parses the expression with to filter/calculate/aggregate and store the results. Set up . Set up . For details on Prometheus Receiver in OpenTelemetry Collector, refer to . Config SkyWalking . ClickHouse monitoring provides monitoring of the metrics events and asynchronous_metrics of the ClickHouse server. ClickHouse cluster is cataloged as a `Layer: CLICKHOUSE` `Service` in OAP. Each ClickHouse server is cataloged as an `Instance` in OAP. | Monitoring Panel | Unit | Metric Name | Description | Data Source | | - | - | | - | -- | | CpuUsage | count | meterclickhouseinstancecpuusage | CPU time spent seen by OS per second(according to ClickHouse.system.dashboard.CPU Usage (cores)). | ClickHouse | | MemoryUsage | percentage | meterclickhouseinstancememoryusage | Total amount of memory (bytes) allocated by the server/ total amount of OS memory. | ClickHouse | | MemoryAvailable | percentage | meterclickhouseinstancememoryavailable | Total amount of memory (bytes) available for program / total amount of OS memory. | ClickHouse | | Uptime | sec | meterclickhouseinstance_uptime | The server uptime in seconds. It includes the time spent for server initialization before accepting connections. | ClickHouse | | Version | string | meterclickhouseinstance_version | Version of the server in a single integer number in base-1000. | ClickHouse | | FileOpen | count | meterclickhouseinstancefileopen | Number of files opened. | ClickHouse | | Monitoring Panel | Unit | Metric Name | Description | Data Source | | - | -- | | - | -- | | TcpConnections | count | meterclickhouseinstancetcpconnections<br/>meterclickhousetcp_connections | Number of connections to TCP server. | ClickHouse | | MysqlConnections | count | meterclickhouseinstancemysqlconnections<br/>meterclickhousemysql_connections | Number of client connections using MySQL protocol. | ClickHouse | | HttpConnections | count | meterclickhouseinstancehttpconnections<br/>meterclickhousemysql_connections | Number of connections to HTTP server. | ClickHouse | | InterserverConnections | count | meterclickhouseinstanceinterserverconnections<br/>meterclickhouseinterserver_connections | Number of connections from other replicas to fetch parts. | ClickHouse | | PostgresqlConnections | count | meterclickhouseinstancepostgresqlconnections<br/>meterclickhousepostgresql_connections | Number of client connections using PostgreSQL protocol. | ClickHouse | | ReceiveBytes | bytes | meterclickhouseinstancenetworkreceivebytes<br/>meterclickhousenetworkreceive_bytes | Total number of bytes received from network. | ClickHouse | | SendBytes | bytes | meterclickhouseinstancenetworksendbytes<br/>meterclickhousenetworksend_bytes | Total number of bytes send to

| ClickHouse | | Monitoring Panel | Unit | Metric Name | Description | Data Source | | - | | -- | | -- | | QueryCount | count | meterclickhouseinstancequery<br/>meterclickhouse_query | Number of executing queries. | ClickHouse | | SelectQueryCount | count | meterclickhouseinstancequeryselect<br/>meterclickhousequery_select | Number of executing queries, but only for SELECT queries. | ClickHouse | | InsertQueryCount | count | meterclickhouseinstancequeryinsert<br/>meterclickhousequery_insert | Number of executing queries, but only for INSERT queries. | ClickHouse | | SelectQueryRate | count/sec | meterclickhouseinstancequeryselectrate<br/>meterclickhousequeryselect_rate | Number of SELECT queries per second. | ClickHouse | | InsertQueryRate | count/sec | meterclickhouseinstancequeryinsertrate<br/>meterclickhousequeryinsert_rate | Number of INSERT queries per second. | ClickHouse | | Querytime | microsec | meterclickhouseinstancequerytimemicroseconds<br/>meterclickhousequerytime_microseconds | Total time of all queries. | ClickHouse | | SelectQuerytime | microsec | meterclickhouseinstancequerytimeselectmicroseconds<br/>meterclickhousequerytimeselect_microseconds | Total time of SELECT queries. | ClickHouse | | InsertQuerytime | microsec | meterclickhouseinstancequerytimeinsertmicroseconds<br/>meterclickhousequerytimeinsert_microseconds | Total time of INSERT queries. | ClickHouse | | OtherQuerytime | microsec | meterclickhouseinstancequerytimeothermicroseconds<br/>meterclickhousequerytimeother_microseconds | Total time of queries that are not SELECT or INSERT. | ClickHouse | | QuerySlowCount | count | meterclickhouseinstancequeryslow<br/>meterclickhousequery_slow | Number of reads from a file that were slow. | ClickHouse | | Monitoring Panel | Unit | Metric Name | Description | Data Source | | | -- | - | - | -- | | InsertQueryCount | count | meterclickhouseinstancequeryinsert<br/>meterclickhousequery_insert | Number of executing queries, but only for INSERT queries. | ClickHouse | | InsertedRowCount | count | meterclickhouseinstanceinsertedrows<br/>meterclickhouseinserted_rows | Number of rows INSERTed to all tables. | ClickHouse | | InsertedBytes | bytes | meterclickhouseinstanceinsertedbytes<br/>meterclickhouseinserted_bytes | Number of bytes INSERTed to all tables. | ClickHouse | | DelayedInsertCount | count | meterclickhouseinstancedelayedinsert<br/>meterclickhousedelayed_insert | Number of times the INSERT of a block to a MergeTree table was throttled due to high number of active data parts for partition. | ClickHouse | | Monitoring Panel | Unit | Metric Name | Description | Data Source | | - | -- | - | | -- | | ReplicatedChecks | count | meterclickhouseinstancereplicatedchecks<br/>meterclickhousereplicated_checks | Number of data parts checking for consistency. | ClickHouse | | ReplicatedFetch | count | meterclickhouseinstancereplicatedfetch<br/>meterclickhousereplicated_fetch | Number of data parts being fetched from replica. | ClickHouse | | ReplicatedSend | count | meterclickhouseinstancereplicatedsend<br/>meterclickhousereplicated_send | Number of data parts being sent to replicas. | ClickHouse | | Monitoring Panel | Unit | Metric Name | Description | Data Source | | - | -- | | - | -- | | BackgroundMergeCount | count | meterclickhouseinstancebackgroundmerge<br/>meterclickhousebackground_merge | Number of executing background

| ClickHouse | | MergeRows | count | meterclickhouseinstancemergerows<br/>meterclickhousemerge_rows | Rows read for background merges. This is the number of rows before merge. | ClickHouse | | MergeUncompressedBytes | bytes | meterclickhouseinstancemergeuncompressedbytes<br/>meterclickhousemergeuncompressed_bytes | Uncompressed bytes (for columns as they stored in memory) that was read for background merges. This is the number before merge. | ClickHouse | | MoveCount | count | meterclickhouseinstancemove<br/>meterclickhouse_move | Number of currently executing moves. | ClickHouse | | PartsActive | Count | meterclickhouseinstancepartsactive<br/>meterclickhouseparts_active | Active data part, used by current and upcoming SELECTs. | ClickHouse | | MutationsCount | count | meterclickhouseinstancemutations<br/>meterclickhouse_mutations | Number of mutations (ALTER DELETE/UPDATE). | ClickHouse | When works with . Kafka lets you: Publish or subscribe to data flows. Organize fault-tolerant storage. Process streams as they become available. | Monitoring Panel | Unit | Metric Name | Description | Data Source | | -- | -- | -- | | -- | | KafkaMessagesRead | count | meterclickhouseinstancekafkamessagesread<br/>meterclickhousekafkamessages_read | Number of Kafka messages already processed by ClickHouse. | ClickHouse | | KafkaWrites | count | meterclickhouseinstancekafkawrites<br/>meterclickhousekafka_writes | Number of writes (inserts) to Kafka tables. | ClickHouse | | KafkaConsumers | count | meterclickhouseinstancekafkaconsumers<br/>meterclickhousekafka_consumers | Number of active Kafka consumers. | ClickHouse | | KafkProducers | count | meterclickhouseinstancekafkaproducers<br/>meterclickhousekafka_producers | Number of active Kafka producer created. | ClickHouse | ClickHouse uses ZooKeeper for storing metadata of replicas when using replicated tables. If replicated tables are not used, this section of parameters can be omitted. | Monitoring Panel | Unit | Metric Name | Description | Data Source | | | -- | - | | -- | | ZookeeperSession | count | meterclickhouseinstancezookeepersession<br/>meterclickhousezookeeper_session | Number of sessions (connections) to ZooKeeper. | ClickHouse | | ZookeeperWatch | count | meterclickhouseinstancezookeeperwatch<br/>meterclickhousezookeeper_watch | Number of watches (event subscriptions) in ZooKeeper. | ClickHouse | | ZookeeperBytesSent | bytes | meterclickhouseinstancezookeeperbytessent<br/>meterclickhousezookeeperbytes_sent | Number of bytes send over network while communicating with ZooKeeper. | ClickHouse | | ZookeeperBytesReceive | bytes | meterclickhouseinstancezookeeperbytesreceived<br/>meterclickhousezookeeperbytes_received | Number of bytes send over network while communicating with ZooKeeper. | ClickHouse | | Monitoring Panel | Unit | Metric Name | Description | Data Source | | | -- | -- | | -- | | KeeperAliveConnections | count | meterclickhouseinstancekeeperconnectionsalive<br/>meterclickhousekeeperconnections_alive | Number of alive connections for embedded ClickHouse Keeper. | ClickHouse | | KeeperOutstandingRequets | count | meterclickhouseinstancekeeperoutstandingrequests<br/>meterclickhousekeeperoutstanding_requests| Number of outstanding requests for embedded ClickHouse Keeper. | ClickHouse | You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in `/config/otel-rules/clickhouse`. The ClickHouse dashboard panel configurations are found in `/config/ui-initialized-templates/clickhouse`.