Prometheus Integration

# HELP ClickPipes_Errors_Total Total errors ingesting data.
# TYPE ClickPipes_Errors_Total counter
ClickPipes_Errors_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="ClickPipes demo instace",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="Confluent demo pipe",clickpipe_source="confluent"} 0

# HELP ClickPipes_SentBytes_Total Total uncompressed bytes sent to ClickHouse.
# TYPE ClickPipes_SentBytes_Total counter
ClickPipes_SentBytes_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="ClickPipes demo instace",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="Confluent demo pipe",clickpipe_source="confluent"} 477187967

# HELP ClickPipes_FetchedBytesCompressed_Total Total compressed bytes fetched from the source. If data is uncompressed at the source, this will equal ClickPipes_FetchedBytes_Total
# TYPE ClickPipes_FetchedBytesCompressed_Total counter
ClickPipes_FetchedBytesCompressed_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="ClickPipes demo instace",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="Confluent demo pipe",clickpipe_source="confluent"} 873286202

# HELP ClickPipes_FetchedEvents_Total Total number of records fetched from the source.
# TYPE ClickPipes_FetchedEvents_Total counter
ClickPipes_FetchedEvents_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="ClickPipes demo instace",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="Confluent demo pipe",clickpipe_source="confluent"} 5535376
```

### Metric labels[​](#metric-labels "Direct link to Metric labels")

All metrics have the following labels:

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Label Description|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | clickhouse\_org Organization ID|  |  |  |  | | --- | --- | --- | --- | | clickhouse\_service Service ID|  |  | | --- | --- | | clickhouse\_service\_name Service name | | | | | | | |

For ClickPipes, metrics will also have the following labels:

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Label Description|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | clickpipe\_id ClickPipe ID|  |  |  |  | | --- | --- | --- | --- | | clickpipe\_name ClickPipe Name|  |  | | --- | --- | | clickpipe\_source ClickPipe Source Type | | | | | | | |

### Information metrics[​](#information-metrics "Direct link to Information metrics")

ClickHouse Cloud provides a special metric `ClickHouse_ServiceInfo` which is a `gauge` that always has the value of `1`. This metric contains all the **Metric Labels** as well as the following labels:

|  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Label Description|  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | clickhouse\_cluster\_status Status of the service. Could be one of the following: [`awaking` | `running` | `degraded` | `idle` | `stopped`]| clickhouse\_version Version of the ClickHouse server that the service is running|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | scrape Indicates the status of the last scrape. Could be either `full` or `partial`| full Indicates that there were no errors during the last metrics scrape|  |  | | --- | --- | | partial Indicates that there were some errors during the last metrics scrape and only `ClickHouse_ServiceInfo` metric was returned. | | | | | | | | | | | |

Requests to retrieve metrics won't resume an idled service. In the case that a service is in the `idle` state, only the `ClickHouse_ServiceInfo` metric will be returned.

For ClickPipes, there's a similar `ClickPipes_Info` metric `gauge` that in addition of the **Metric Labels** contains the following labels:

|  |  |  |  |
| --- | --- | --- | --- |
| Label Description|  |  | | --- | --- | | clickpipe\_state The current state of the pipe | | | |

### Configuring Prometheus[​](#configuring-prometheus "Direct link to Configuring Prometheus")

The Prometheus server collects metrics from configured targets at the given intervals. Below is an example configuration for the Prometheus server to use the ClickHouse Cloud Prometheus Endpoint:

```
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
    - targets: ["localhost:9090"]
  - job_name: "clickhouse"
    static_configs:
      - targets: ["api.clickhouse.cloud"]
    scheme: https
    params:
      filtered_metrics: ["true"]
    metrics_path: "/v1/organizations/<ORG_ID>/prometheus"
    basic_auth:
      username: <KEY_ID>
      password: <KEY_SECRET>
    honor_labels: true
```

Note the `honor_labels` configuration parameter needs to be set to `true` for the instance label to be properly populated. Additionally, `filtered_metrics` is set to `true` in the above example, but should be configured based on user preference.

## Integrating with Grafana[​](#integrating-with-grafana "Direct link to Integrating with Grafana")

Users have two primary ways to integrate with Grafana:

- **Metrics Endpoint** – This approach has the advantage of not requiring any additional components or infrastructure. This offering is limited to Grafana Cloud and only requires the ClickHouse Cloud Prometheus Endpoint URL and credentials.
- **Grafana Alloy** - Grafana Alloy is a vendor-neutral distribution of the OpenTelemetry (OTel) Collector, replacing the Grafana Agent. This can be used as a scraper, is deployable in your own infrastructure, and is compatible with any Prometheus endpoint.

We provide instructions on using these options below, focusing on the details specific to the ClickHouse Cloud Prometheus Endpoint.

### Grafana Cloud with metrics endpoint[​](#grafana-cloud-with-metrics-endpoint "Direct link to Grafana Cloud with metrics endpoint")

- Login to your Grafana Cloud account
- Add a new connection by selecting the **Metrics Endpoint**
- Configure the Scrape URL to point to the Prometheus endpoint and use basic auth to configure your connection with the API key/secret
- Test the connection to ensure you're able to connect

  

Once configured, you should see the metrics in the drop-down that you can select to configure dashboards:

  

### Grafana Cloud with Alloy[​](#grafana-cloud-with-alloy "Direct link to Grafana Cloud with Alloy")

If you're using Grafana Cloud, Alloy can be installed by navigating to the Alloy menu in Grafana and following the onscreen instructions:

  

This should configure Alloy with a `prometheus.remote_write` component for sending data to a Grafana Cloud endpoint with an authentication token. Users then need to only modify the Alloy config (found in `/etc/alloy/config.alloy` for Linux) to include a scraper for the ClickHouse Cloud Prometheus Endpoint.

The following shows an example configuration for Alloy with a `prometheus.scrape` component for scraping metrics from the ClickHouse Cloud Endpoint, as well as the automatically configured `prometheus.remote_write` component. Note that the `basic_auth` configuration component contains our Cloud API key ID and secret as the username and password, respectively.

```
prometheus.scrape "clickhouse_cloud" {
  targets = [{
  __address__ = "api.clickhouse.cloud",
  }]

  scheme       = "https"
  metrics_path = "/v1/organizations/<clickhouse_org_id>/prometheus"

  params = {
  "filtered_metrics" = ["true"],
  }

  honor_labels    = true
  scrape_interval = "30s"
  scrape_timeout  = "25s"

  basic_auth {
  username = "<clickhouse_api_key_id>"
  password = "<clickhouse_api_key_secret>"
  }

  forward_to = [prometheus.remote_write.grafana_cloud.receiver]
}

  prometheus.remote_write "grafana_cloud" {
  endpoint {
  url = "https://<grafana_prometheus_url>/api/prom/push"