Monitoring Temporal Cloud metrics with ClickStack

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/integration-examples/temporal.md)

Warning

OpenMetrics support in the Temporal platform is available in [Public Preview](https://docs.temporal.io/evaluate/development-production-features/release-stages#public-preview). Refer to [their documentation](https://docs.temporal.io/cloud/metrics/openmetrics) for more information.

Temporal offers an abstraction for building simple, sophisticated, resilient applications.

TL;DR

Monitor Temporal Cloud metrics in ClickStack using the OTel Prometheus receiver. Includes a pre-built dashboard.

## Integration with existing Temporal Cloud[​](#existing-temporal "Direct link to Integration with existing Temporal Cloud")

This section covers configuring ClickStack by configuring the ClickStack OTel collector with the Prometheus receiver.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

- ClickStack instance running
- Existing Temporal Cloud account
- HTTP network access from ClickStack to your Temporal Cloud

Create Temporal Cloud key

#### Create Temporal Cloud key[​](#create-temporal-cloud-key "Direct link to Create Temporal Cloud key")

Ensure you have a Temporal Cloud API key. This can be created by following the [Authentication guide](https://docs.temporal.io/production-deployment/cloud/metrics/openmetrics/api-reference#authentication) in the Temporal documentation.

Key file

Ensure these credentials are stored in a file `temporal.key` in the same directory as the config file created below. This key should just be stored as text with no preceding or following spaces.

Create custom OTel collector configuration

#### Create custom OTel collector configuration[​](#custom-otel "Direct link to Create custom OTel collector configuration")

ClickStack allows you to extend the base OpenTelemetry collector configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP.

Create a file named `temporal-metrics.yaml` with the following configuration:

```
receivers:
  prometheus/temporal:
    config:
      scrape_configs:
      - job_name: 'temporal-cloud'
        scrape_interval: 60s
        scrape_timeout: 30s
        honor_timestamps: true
        scheme: https
        authorization:
          type: Bearer
          credentials_file: /etc/otelcol-contrib/temporal.key
        static_configs:
          - targets: ['metrics.temporal.io']
        metrics_path: '/v1/metrics'

processors:
  resource:
    attributes:
      - key: service.name
        value: "temporal"
        action: upsert

service:
  pipelines:
    metrics/temporal:
      receivers: [prometheus/temporal]
      processors:
        - resource
        - memory_limiter
        - batch
      exporters:
        - clickhouse
```

This configuration:

- Connects to Temporal Cloud at `metrics.temporal.io`
- Collects metrics every 60 seconds
- Collects [key performance metrics](https://docs.temporal.io/production-deployment/cloud/metrics/openmetrics/metrics-reference)
- **Sets the required `service.name` resource attribute** per [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/resource/#service)
- Routes metrics to the ClickHouse exporter via a dedicated pipeline

Note

- You only define new receivers, processors, and pipelines in the custom config
- The `memory_limiter` and `batch` processors and `clickhouse` exporter are already defined in the base ClickStack configuration - you just reference them by name
- The `resource` processor sets the required `service.name` attribute per OpenTelemetry semantic conventions
- For multiple Temporal cloud accounts, customize `service.name` to distinguish them (e.g., `"temporal-prod"`, `"temporal-dev"`)

Configure ClickStack to load custom configuration

#### Configure ClickStack to load custom configuration[​](#load-custom "Direct link to Configure ClickStack to load custom configuration")

To enable custom collector configuration in your existing ClickStack deployment, you must:

1. Mount the custom config file at `/etc/otelcol-contrib/custom.config.yaml`
2. Set the environment variable `CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml`
3. Mount the `temporal.key` file at `/etc/otelcol-contrib/temporal.key`
4. Ensure network connectivity between ClickStack and Temporal

All commands assume they're executed from the sample directory as where `temporal-metrics.yaml` and `temporal.key` are stored.

##### Option 1: Docker Compose[​](#docker-compose "Direct link to Option 1: Docker Compose")

Update your ClickStack deployment configuration:

```
services:
  clickstack:
    # ... existing configuration ...
    environment:
      - CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml
    volumes:
      - ./temporal-metrics.yaml:/etc/otelcol-contrib/custom.config.yaml:ro
      - ./temporal.key:/etc/otelcol-contrib/temporal.key:ro
      # ... other volumes ...
```

##### Option 2: Docker run (all-in-one image)[​](#all-in-one "Direct link to Option 2: Docker run (all-in-one image)")

If using the all-in-one image with `docker run`:

```
docker run --name clickstack \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
  -v "$(pwd)/temporal-metrics.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
  -v "$(pwd)/temporal.key:/etc/otelcol-contrib/temporal.key:ro" \
  clickhouse/clickstack-all-in-one:latest
```

Verify metrics in HyperDX

#### Verify metrics in HyperDX[​](#verifying-metrics "Direct link to Verify metrics in HyperDX")

Once configured, log into HyperDX and verify metrics are flowing:

1. Navigate to the Metrics explorer
2. Search for metrics starting with `temporal` (e.g., `temporal_cloud_v1_workflow_success_count`, `temporal_cloud_v1_poll_timeout_count`)
3. You should see metric data points appearing at your configured collection interval

## Dashboards and visualization[​](#dashboards "Direct link to Dashboards and visualization")

To help you get started monitoring Temporal Cloud with ClickStack, we provide some example visualizations for Temporal Metrics.

Download the dashboard configuration

#### [Download](/docs/examples/temporal-metrics-dashboard.json) the dashboard configuration[​](#download "Direct link to download")

Import the pre-built dashboard

#### Import the pre-built dashboard[​](#import-dashboard "Direct link to Import the pre-built dashboard")

1. Open HyperDX and navigate to the Dashboards section
2. Click **Import Dashboard** in the upper right corner under the ellipses

3. Upload the `temporal-metrics-dashboard.json` file and click **Finish Import**

View the dashboard

#### View the dashboard[​](#created-dashboard "Direct link to View the dashboard")

The dashboard will be created with all visualizations pre-configured:

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### Custom config not loading[​](#troubleshooting-not-loading "Direct link to Custom config not loading")

Verify the environment variable `CUSTOM_OTELCOL_CONFIG_FILE` is set correctly:

```
docker exec <container-name> printenv CUSTOM_OTELCOL_CONFIG_FILE
```

Check that the custom config file is mounted at `/etc/otelcol-contrib/custom.config.yaml`:

```
docker exec <container-name> ls -lh /etc/otelcol-contrib/custom.config.yaml
# usually, docker exec clickstack ls -lh /etc/otelcol-contrib/custom.config.yaml
```

View the custom config content to verify it's readable:

```
docker exec <container-name> cat /etc/otelcol-contrib/custom.config.yaml
# usually, docker exec clickstack cat /etc/otelcol-contrib/custom.config.yaml
```

Confirm the `temporal.key` is mounted into the container:

```
docker exec <container-name> cat /etc/otelcol-contrib/temporal.key
# usually, docker exec clickstack cat /etc/otelcol-contrib/temporal.key
# This should output your temporal.key
```

### No metrics appearing in HyperDX[​](#no-metrics "Direct link to No metrics appearing in HyperDX")