Monitoring Nginx Logs with ClickStack

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/integration-examples/nginx-logs.md)

TL;DR

Collect and visualize Nginx access logs (JSON format) in ClickStack using the OTel `filelog` receiver. Includes a demo dataset and pre-built dashboard.

## Integration with existing Nginx[​](#existing-nginx "Direct link to Integration with existing Nginx")

This section covers configuring your existing Nginx installation to send logs to ClickStack by modifying the ClickStack OTel collector configuration.
If you would like to test the integration before configuring your own existing setup, you can test with our preconfigured setup and sample data in the [following section](/docs/use-cases/observability/clickstack/integrations/nginx#demo-dataset).

##### Prerequisites[​](#prerequisites "Direct link to Prerequisites")

- ClickStack instance running
- Existing Nginx installation
- Access to modify Nginx configuration files

Configure Nginx log format

#### Configure Nginx log format[​](#configure-nginx "Direct link to Configure Nginx log format")

First, configure Nginx to output logs in JSON format for easier parsing. Add this log format definition to your nginx.conf:

The `nginx.conf` file is typically located at:

- **Linux (apt/yum)**: `/etc/nginx/nginx.conf`
- **macOS (Homebrew)**: `/usr/local/etc/nginx/nginx.conf` or `/opt/homebrew/etc/nginx/nginx.conf`
- **Docker**: Configuration is usually mounted as a volume

Add this log format definition to the `http` block:

```
http {
    log_format json_combined escape=json
    '{'
      '"time_local":"$time_local",'
      '"remote_addr":"$remote_addr",'
      '"request_method":"$request_method",'
      '"request_uri":"$request_uri",'
      '"status":$status,'
      '"body_bytes_sent":$body_bytes_sent,'
      '"request_time":$request_time,'
      '"upstream_response_time":"$upstream_response_time",'
      '"http_referer":"$http_referer",'
      '"http_user_agent":"$http_user_agent"'
    '}';

    access_log /var/log/nginx/access.log json_combined;
    error_log /var/log/nginx/error.log warn;
}
```

After making this change, reload Nginx.

Create custom OTel collector configuration

#### Create custom OTel collector configuration[​](#custom-otel "Direct link to Create custom OTel collector configuration")

ClickStack allows you to extend the base OpenTelemetry Collector configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP.

Create a file named nginx-monitoring.yaml with the following configuration:

```
receivers:
  filelog:
    include:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
    start_at: end 
    operators:
      - type: json_parser
        parse_from: body
        parse_to: attributes
      - type: time_parser
        parse_from: attributes.time_local
        layout: '%d/%b/%Y:%H:%M:%S %z'
      - type: add
        field: attributes.source
        value: "nginx"

service:
  pipelines:
    logs/nginx:
      receivers: [filelog]
      processors:
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse
```

This configuration:

- Reads Nginx Logs from their standard locations
- Parses JSON log entries
- Extracts and preserves the original log timestamps
- Adds source: Nginx attribute for filtering in HyperDX
- Routes logs to the ClickHouse exporter via a dedicated pipeline

Note

- You only define new receivers and pipelines in the custom config
- The processors (memory\_limiter, transform, batch) and exporters (clickhouse) are already defined in the base ClickStack configuration - you just reference them by name
- The time\_parser operator extracts timestamps from Nginx's time\_local field to preserve original log timing
- The pipelines route data from your receivers to the ClickHouse exporter via the existing processors

Configure ClickStack to load custom configuration

#### Configure ClickStack to load custom configuration[​](#load-custom "Direct link to Configure ClickStack to load custom configuration")

To enable custom collector configuration in your existing ClickStack deployment, you must:

1. Mount the custom config file at /etc/otelcol-contrib/custom.config.yaml
2. Set the environment variable CUSTOM\_OTELCOL\_CONFIG\_FILE=/etc/otelcol-contrib/custom.config.yaml
3. Mount your Nginx log directories so the collector can read them

##### Option 1: Docker Compose[​](#docker-compose "Direct link to Option 1: Docker Compose")

Update your ClickStack deployment configuration:

```
services:
  clickstack:
    # ... existing configuration ...
    environment:
      - CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml
      # ... other environment variables ...
    volumes:
      - ./nginx-monitoring.yaml:/etc/otelcol-contrib/custom.config.yaml:ro
      - /var/log/nginx:/var/log/nginx:ro
      # ... other volumes ...
```

##### Option 2: Docker Run (All-in-One Image)[​](#all-in-one "Direct link to Option 2: Docker Run (All-in-One Image)")

If using the all-in-one image with docker run:

```
docker run --name clickstack \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
  -v "$(pwd)/nginx-monitoring.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
  -v /var/log/nginx:/var/log/nginx:ro \
  clickhouse/clickstack-all-in-one:latest
```

Note

Ensure the ClickStack collector has appropriate permissions to read the nginx log files. In production, use read-only mounts (:ro) and follow the principle of least privilege.

Verifying Logs in HyperDX

#### Verifying Logs in HyperDX[​](#verifying-logs "Direct link to Verifying Logs in HyperDX")

Once configured, log into HyperDX and verify logs are flowing:

1. Navigate to the search view
2. Set source to Logs, and verify you see log entries with fields like request, request\_time, upstream\_response\_time, etc.

This is an example of what you should see:

## Demo dataset[​](#demo-dataset "Direct link to Demo dataset")

For users who want to test the nginx integration before configuring their production systems, we provide a sample dataset of pre-generated nginx access logs with realistic traffic patterns.

Download the sample dataset

#### Download the sample dataset[​](#download-sample "Direct link to Download the sample dataset")

```
# Download the logs
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/access.log
```

The dataset includes:

- Log entries with realistic traffic patterns
- Various endpoints and HTTP methods
- Mix of successful requests and errors
- Realistic response times and byte counts

Create test collector configuration

#### Create test collector configuration[​](#test-config "Direct link to Create test collector configuration")

Create a file named `nginx-demo.yaml` with the following configuration:

```
cat > nginx-demo.yaml << 'EOF'
receivers:
  filelog:
    include:
      - /tmp/nginx-demo/access.log
    start_at: beginning  # Read from beginning for demo data
    operators:
      - type: json_parser
        parse_from: body
        parse_to: attributes
      - type: time_parser
        parse_from: attributes.time_local
        layout: '%d/%b/%Y:%H:%M:%S %z'
      - type: add
        field: attributes.source
        value: "nginx-demo"

service:
  pipelines:
    logs/nginx-demo:
      receivers: [filelog]
      processors:
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse
EOF
```

Run ClickStack with demo configuration

#### Run ClickStack with demo configuration[​](#run-demo "Direct link to Run ClickStack with demo configuration")

Run ClickStack with the demo logs and configuration: