Monitoring Kubernetes

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/example-datasets/kubernetes.md)

This guide allows you to collect logs and metrics from your Kubernetes system, sending them to **ClickStack** for visualization and analysis. For demo data we use optionally use the ClickStack fork of the official OpenTelemetry demo.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

This guide requires you to have:

- A **Kubernetes cluster** (v1.20+ recommended) with at least 32 GiB of RAM and 100GB of disk space available on one node for ClickHouse.
- **[Helm](https://helm.sh/)** v3+
- **`kubectl`**, configured to interact with your cluster

## Deployment options[​](#deployment-options "Direct link to Deployment options")

You can follow this guide using either of the following deployment options:

- **Open Source ClickStack**: Deploy ClickStack entirely within your Kubernetes cluster, including:

  - ClickHouse
  - HyperDX
  - MongoDB (used for dashboard state and configuration)
- **Managed ClickStack**, with ClickHouse and the ClickStack UI (HyperDX) managed in ClickHouse Cloud. This eliminates the need to run ClickHouse or HyperDX inside your cluster.

To simulate application traffic, you can optionally deploy the ClickStack fork of the [**OpenTelemetry Demo Application**](https://github.com/ClickHouse/opentelemetry-demo). This generates telemetry data including logs, metrics, and traces. If you already have workloads running in your cluster, you can skip this step and monitor existing pods, nodes, and containers.

Install cert-manager (Optional)

### Install cert-manager (Optional)[​](#install-cert-manager "Direct link to Install cert-manager (Optional)")

If your setup needs TLS certificates, install [cert-manager](https://cert-manager.io/) using Helm:

```
# Add Cert manager repo 

helm repo add jetstack https://charts.jetstack.io 

helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set startupapicheck.timeout=5m --set installCRDs=true --set global.leaderElection.namespace=cert-manager
```

Deploy the OpenTelemetry Demo (Optional)

### Deploy the OpenTelemetry Demo (Optional)[​](#deploy-otel-demo "Direct link to Deploy the OpenTelemetry Demo (Optional)")

This **step is optional and intended if you have no existing pods to monitor**. Although users with existing services deployed in their Kubernetes environment can skip, this demo does include instrumented microservices which generate trace and session replay data - allowing users to explore all features of ClickStack.

The following deploys the ClickStack fork of the OpenTelemetry Demo application stack within a Kubernetes cluster, tailored for observability testing and showcasing instrumentation. It includes backend microservices, load generators, telemetry pipelines, supporting infrastructure (e.g., Kafka, Redis), and SDK integrations with ClickStack.

All services are deployed to the `otel-demo` namespace. Each deployment includes:

- Automatic instrumentation with OTel and ClickStack SDKS for traces, metrics, and logs.
- All services send their instrumentation to a `my-hyperdx-hdx-oss-v2-otel-collector` OpenTelemetry collector (not deployed)
- [Forwarding of resource tags](/docs/use-cases/observability/clickstack/integrations/kubernetes#forwarding-resouce-tags-to-pods) to correlate logs, metrics and traces via the environment variable `OTEL_RESOURCE_ATTRIBUTES`.

```
## download demo Kubernetes manifest file
curl -O https://raw.githubusercontent.com/ClickHouse/opentelemetry-demo/refs/heads/main/kubernetes/opentelemetry-demo.yaml
# wget alternative
# wget https://raw.githubusercontent.com/ClickHouse/opentelemetry-demo/refs/heads/main/kubernetes/opentelemetry-demo.yaml
kubectl apply --namespace otel-demo -f opentelemetry-demo.yaml
```

On deployment of the demo, confirm all pods have been successfully created and are in the `Running` state:

```
kubectl get pods -n=otel-demo

NAME                                 READY   STATUS    RESTARTS   AGE
accounting-fd44f4996-fcl4k           1/1     Running   0          13m
ad-769f968468-qq8mw                  1/1     Running   0          13m
artillery-loadgen-7bc4bdf47d-5sb96   1/1     Running   0          13m
cart-5b4c98bd8-xm7m2                 1/1     Running   0          13m
checkout-784f69b785-cnlpp            1/1     Running   0          13m
currency-fd7775b9c-rf6cr             1/1     Running   0          13m
email-5c54598f99-2td8s               1/1     Running   0          13m
flagd-5466775df7-zjb4x               2/2     Running   0          13m
fraud-detection-5769fdf75f-cjvgh     1/1     Running   0          13m
frontend-6dcb696646-fmcdz            1/1     Running   0          13m
frontend-proxy-7b8f6cd957-s25qj      1/1     Running   0          13m
image-provider-5fdb455756-fs4xv      1/1     Running   0          13m
kafka-7b6666866d-xfzn6               1/1     Running   0          13m
load-generator-57cbb7dfc9-ncxcf      1/1     Running   0          13m
payment-6d96f9bcbd-j8tj6             1/1     Running   0          13m
product-catalog-7fb77f9c78-49bhj     1/1     Running   0          13m
quote-576c557cdf-qn6pr               1/1     Running   0          13m
recommendation-546cc68fdf-8x5mm      1/1     Running   0          13m
shipping-7fc69f7fd7-zxrx6            1/1     Running   0          13m
valkey-cart-5f7b667bb7-gl5v4         1/1     Running   0          13m
```

### Demo Architecture[​](#demo-architecture "Direct link to Demo Architecture")

The demo is composed of microservices written in different programming languages that talk to each other over gRPC and HTTP and a load generator that uses Locust to fake user traffic. The original source code for this demo has been modified to use [ClickStack instrumentation](/docs/use-cases/observability/clickstack/sdks).

*Credit: <https://opentelemetry.io/docs/demo/architecture/>*

Further details on the demo can be found in:

- [OpenTelemetry documentation](https://opentelemetry.io/docs/demo/)
- [ClickStack maintained fork](https://github.com/ClickHouse/opentelemetry-demo)

Add the ClickStack Helm chart repository

### Add the ClickStack Helm chart repository[​](#add-helm-clickstack "Direct link to Add the ClickStack Helm chart repository")

To deploy ClickStack, we use the [official Helm chart](https://clickhouse.com/docs/use-cases/observability/clickstack/deployment/helm).

This requires us to add the HyperDX Helm repository:

```
helm repo add hyperdx https://hyperdxio.github.io/helm-charts
helm repo update
```

Deploy ClickStack

### Deploy ClickStack[​](#deploy-clickstack "Direct link to Deploy ClickStack")

With the Helm chart installed, you can deploy ClickStack to your cluster. You can either run all components, including ClickHouse and HyperDX, within your Kubernetes environment, or just deploy the collector and rely on Managed ClickStack for ClickHouse and the UI HyperDX.

  
ClickStack Open Source (self-managed)

The following command installs ClickStack to the `otel-demo` namespace. The helm chart deploys:

- A ClickHouse instance
- HyperDX
- The ClickStack distribution of the OTel collector
- MongoDB for storage of HyperDX application state

Note

You might need to adjust the `storageClassName` according to your Kubernetes cluster configuration.

Users not deploying the OTel demo can modify this, selecting an appropriate namespace.

```
helm install my-hyperdx hyperdx/hdx-oss-v2   --set clickhouse.persistence.dataSize=100Gi --set global.storageClassName="standard-rwo" -n otel-demo
```

ClickStack in production

This chart also installs ClickHouse and the OTel collector. For production, it is recommended that you use the clickhouse and OTel collector operators and/or use Managed ClickStack.

To disable clickhouse and OTel collector, set the following values:

```
helm install myrelease <chart-name-or-path> --set clickhouse.enabled=false --set clickhouse.persistence.enabled=false --set otel.enabled=false
```

Managed ClickStack