Monitoring Redis Logs with ClickStack

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/integration-examples/redis-logs.md)

TL;DR

Collect and visualize Redis server logs in ClickStack using the OTel `filelog` receiver. Includes a demo dataset and pre-built dashboard.

## Integration with existing Redis[​](#existing-redis "Direct link to Integration with existing Redis")

This section covers configuring your existing Redis installation to send logs to ClickStack by modifying the ClickStack OTel collector configuration.
If you would like to test the Redis integration before configuring your own existing setup, you can test with our preconfigured setup and sample data in the ["Demo dataset"](/docs/use-cases/observability/clickstack/integrations/redis#demo-dataset) section.

### Prerequisites[​](#prerequisites "Direct link to Prerequisites")

- ClickStack instance running
- Existing Redis installation (version 3.0 or newer)
- Access to Redis log files

Verify Redis logging configuration

#### Verify Redis logging configuration[​](#verify-redis "Direct link to Verify Redis logging configuration")

First, check your Redis logging configuration. Connect to Redis and check the log file location:

```
redis-cli CONFIG GET logfile
```

Common Redis log locations:

- **Linux (apt/yum)**: `/var/log/redis/redis-server.log`
- **macOS (Homebrew)**: `/usr/local/var/log/redis.log`
- **Docker**: Often logged to stdout, but can be configured to write to `/data/redis.log`

If Redis is logging to stdout, configure it to write to a file by updating `redis.conf`:

```
# Log to file instead of stdout
logfile /var/log/redis/redis-server.log

# Set log level (options: debug, verbose, notice, warning)
loglevel notice
```

After changing the configuration, restart Redis:

```
# For systemd
sudo systemctl restart redis

# For Docker
docker restart <redis-container>
```

Create custom OTel collector configuration

#### Create custom OTel collector configuration[​](#custom-otel "Direct link to Create custom OTel collector configuration")

ClickStack allows you to extend the base OpenTelemetry Collector configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP.

Create a file named `redis-monitoring.yaml` with the following configuration:

```
receivers:
  filelog/redis:
    include:
      - /var/log/redis/redis-server.log
    start_at: beginning
    operators:
      - type: regex_parser
        regex: '^(?P\d+):(?P\w+) (?P\d{2} \w+ \d{4} \d{2}:\d{2}:\d{2})\.\d+ (?P[.\-*#]) (?P.*)$'
        parse_from: body
        parse_to: attributes
      
      - type: time_parser
        parse_from: attributes.timestamp
        layout: '%d %b %Y %H:%M:%S'
      
      - type: add
        field: attributes.source
        value: "redis"
      
      - type: add
        field: resource["service.name"]
        value: "redis-production"

service:
  pipelines:
    logs/redis:
      receivers: [filelog/redis]
      processors:
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse
```

This configuration:

- Reads Redis Logs from their standard location
- Parses Redis's log format using regex to extract structured fields (`pid`, `role`, `timestamp`, `log_level`, `message`)
- Adds `source: redis` attribute for filtering in HyperDX
- Routes logs to the ClickHouse exporter via a dedicated pipeline

Note

- You only define new receivers and pipelines in the custom config
- The processors (`memory_limiter`, `transform`, `batch`) and exporters (`clickhouse`) are already defined in the base ClickStack configuration - you just reference them by name
- The `time_parser` operator extracts timestamps from Redis Logs to preserve original log timing
- This configuration uses `start_at: beginning` to read all existing logs when the collector starts, allowing you to see logs immediately. For production deployments where you want to avoid re-ingesting logs on collector restarts, change to `start_at: end`.

Configure ClickStack to load custom configuration

#### Configure ClickStack to load custom configuration[​](#load-custom "Direct link to Configure ClickStack to load custom configuration")

To enable custom collector configuration in your existing ClickStack deployment, you must:

1. Mount the custom config file at `/etc/otelcol-contrib/custom.config.yaml`
2. Set the environment variable `CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml`
3. Mount your Redis log directory so the collector can read them

##### Option 1: Docker Compose[​](#docker-compose "Direct link to Option 1: Docker Compose")

Update your ClickStack deployment configuration:

```
services:
  clickstack:
    # ... existing configuration ...
    environment:
      - CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml
      # ... other environment variables ...
    volumes:
      - ./redis-monitoring.yaml:/etc/otelcol-contrib/custom.config.yaml:ro
      - /var/log/redis:/var/log/redis:ro
      # ... other volumes ...
```

##### Option 2: Docker Run (All-in-One Image)[​](#all-in-one "Direct link to Option 2: Docker Run (All-in-One Image)")

If you're using the all-in-one image with docker, run:

```
docker run --name clickstack \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
  -v "$(pwd)/redis-monitoring.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
  -v /var/log/redis:/var/log/redis:ro \
  clickhouse/clickstack-all-in-one:latest
```

Note

Ensure the ClickStack collector has appropriate permissions to read the Redis log files. In production, use read-only mounts (`:ro`) and follow the principle of least privilege.

Verifying Logs in HyperDX

#### Verifying Logs in HyperDX[​](#verifying-logs "Direct link to Verifying Logs in HyperDX")

Once configured, log into HyperDX and verify that logs are flowing:

## Demo dataset[​](#demo-dataset "Direct link to Demo dataset")

For users who want to test the Redis integration before configuring their production systems, we provide a sample dataset of pre-generated Redis Logs with realistic patterns.

Download the sample dataset

#### Download the sample dataset[​](#download-sample "Direct link to Download the sample dataset")

Download the sample log file:

```
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/redis/redis-server.log
```

Create test collector configuration

#### Create test collector configuration[​](#test-config "Direct link to Create test collector configuration")

Create a file named `redis-demo.yaml` with the following configuration:

```
cat > redis-demo.yaml << 'EOF'
receivers:
  filelog/redis:
    include:
      - /tmp/redis-demo/redis-server.log
    start_at: beginning  # Read from beginning for demo data
    operators:
      - type: regex_parser
        regex: '^(?P<pid>\d+):(?P<role>\w+) (?P<timestamp>\d{2} \w+ \d{4} \d{2}:\d{2}:\d{2})\.\d+ (?P<log_level>[.\-*#]) (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
      
      - type: time_parser
        parse_from: attributes.timestamp
        layout: '%d %b %Y %H:%M:%S'
      
      - type: add
        field: attributes.source
        value: "redis-demo"
      
      - type: add
        field: resource["service.name"]
        value: "redis-demo"

service:
  pipelines:
    logs/redis-demo:
      receivers: [filelog/redis]
      processors:
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse
EOF
```

Run ClickStack with demo configuration

#### Run ClickStack with demo configuration[​](#run-demo "Direct link to Run ClickStack with demo configuration")

Run ClickStack with the demo logs and configuration: