Ingesting with Vector

```
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/access.log
```

Note

This data has been collected from an Nginx instance configured to output logs in JSON format for easier parsing. For the Nginx configuration for these logs, see ["Monitoring Nginx Logs with ClickStack"](/docs/use-cases/observability/clickstack/integrations/nginx#configure-nginx).

Create a database and table

### Create a database and table[​](#create-database-table-nginx-managed "Direct link to Create a database and table")

Vector requires a table and schema to be defined prior to data ingestion.

First create a database. This can be done via the [ClickHouse Cloud console](/docs/cloud/get-started/sql-console).

Create a database `logs`:

```
CREATE DATABASE IF NOT EXISTS logs
```

Create a table for your data.

```
CREATE TABLE logs.nginx_logs
(
    `time_local` DateTime,
    `remote_addr` IPv4,
    `remote_user` LowCardinality(String),
    `request` String,
    `status` UInt16,
    `body_bytes_sent` UInt64,
    `http_referer` String,
    `http_user_agent` String,
    `http_x_forwarded_for` LowCardinality(String),
    `request_time` Float32,
    `upstream_response_time` Float32,
    `http_host` String
)
ENGINE = MergeTree
ORDER BY (toStartOfMinute(time_local), status, remote_addr)
```

Nginx primary key

The primary key above assumes typical access patterns in the ClickStack UI for Nginx logs, but may need to be adjusted depending on your workload in production environments.

Copy Vector configuration

### Copy Vector configuration[​](#copy-vector-configuration "Direct link to Copy Vector configuration")

Copy the vector configuration and create a file `nginx.yaml`, setting the `CLICKHOUSE_ENDPOINT` and `CLICKHOUSE_PASSWORD`.

```
data_dir: ./.vector-data
sources:
  nginx_logs:
    type: file
    include:
      - access.log
    read_from: beginning

transforms:
  decode_json:
    type: remap
    inputs:
      - nginx_logs
    source: |
      . = parse_json!(to_string!(.message))
      ts = parse_timestamp!(.time_local, format: "%d/%b/%Y:%H:%M:%S %z")
      # ClickHouse-friendly DateTime format
      .time_local = format_timestamp!(ts, format: "%F %T")

sinks:
  clickhouse:
    type: clickhouse
    inputs:
      - decode_json
    endpoint: "<CLICKHOUSE_ENDPOINT>"
    database: logs
    format: json_each_row
    table: nginx_logs
    skip_unknown_fields: true
    auth:
      strategy: "basic"
      user: "default"
      password: "<CLICKHOUSE_PASSWORD>"
```

Note

The example above uses the default user for Managed ClickStack. For production deployments, we recommend [creating a dedicated ingestion user](/docs/use-cases/observability/clickstack/ingesting-data/otel-collector#creating-an-ingestion-user) with appropriate permissions and limits.

Start Vector

### Start Vector[​](#start-vector "Direct link to Start Vector")

Start Vector with the following command, creating the data directory first to record file offsets.

```
mkdir ./.vector-data
vector --config nginx.yaml
```

Navigate to the ClickStack UI

### Navigate to the ClickStack UI[​](#navigate-to-clickstack-ui-nginx-managed "Direct link to Navigate to the ClickStack UI")

Navigate to your Managed ClickStack service and select "ClickStack" from the left-hand menu. If you’ve already completed the onboarding, this will launch the ClickStack UI in a new tab, and you will be automatically authenticated. If not, you can proceed through the onboarding and select “Launch ClickStack” once you’ve selected Vector as your input source.

Create a datasource

### Create a datasource[​](#create-a-datasource-nginx-managed "Direct link to Create a datasource")

Create a logs data source. If no data sources exist, you will be prompted to create one on first login. Otherwise, navigate to Team Settings and add a new data source.

The configuration assumes the Nginx schema with a `time_local` column used as the timestamp. This is the timestamp column declared in the primary key. This column is mandatory.

We have also specified the default select to be `time_local, remote_addr, status, request`, which defines which columns are returned in the logs view.

In the example above, a `Body` column doesn't exist in the data. Instead, it is defined as the SQL expression:

```
concat(
  remote_addr, ' ',
  remote_user, ' ',
  '[', formatDateTime(time_local, '%d/%b/%Y:%H:%M:%S %z'), '] ',
  '"', request, '" ',
  toString(status), ' ',
  toString(body_bytes_sent), ' ',
  '"', http_referer, '" ',
  '"', http_user_agent, '" ',
  '"', http_x_forwarded_for, '" ',
  toString(request_time), ' ',
  toString(upstream_response_time), ' ',
  '"', http_host, '"'
)
```

This reconstructs the log line from the structured fields.

For other possible options, see the [configuration reference](/docs/use-cases/observability/clickstack/config).

Explore the data

### Explore the data[​](#explore-the-data-nginx-managed "Direct link to Explore the data")

Navigate to the search view for `October 20th, 2025` to explore the data and begin using ClickStack.

The following guide assumes you have set up ClickStack Open Source with the [Getting Started guide](/docs/use-cases/observability/clickstack/getting-started/oss).

Installing vector

### Installing vector[​](#installing-vector-oss "Direct link to Installing vector")

Before proceeding, ensure that **Vector is installed** on the system where you plan to run your ingestion pipeline. Follow the [official Vector installation guide](https://vector.dev/docs/setup/installation/) to install a prebuilt binary or package appropriate for your environment:

Once installed, verify that the `vector` binary is available on your path before continuing with the configuration steps below.

This can be installed on the same instance as your ClickStack OTel collector.

Follow best practices for architecture and security when [moving Vector to production](https://vector.dev/docs/setup/going-to-prod/).

Download the sample data

### Download the sample data[​](#download-the-sample-data-oss "Direct link to Download the sample data")

If you wish to experiment with a sample dataset, download the following example nginx sample.

```
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/access.log
```

Note

This data has been collected from an Nginx instance configured to output logs in JSON format for easier parsing. For the Nginx configuration for these logs, see ["Monitoring Nginx Logs with ClickStack"](/docs/use-cases/observability/clickstack/integrations/nginx#configure-nginx).

Create a database and table

### Create a database and table[​](#create-a-database-table-nginx-oss "Direct link to Create a database and table")

Vector requires a table and schema to be defined prior to data ingestion.

First create a database. This can be done via the [ClickHouse Web user interface](/docs/interfaces/http#web-ui) at <http://localhost:8123/play>. Use the default username and password `api:api`.

Create a database `logs`:

```
CREATE DATABASE IF NOT EXISTS logs
```

Create a table for your data.

```
CREATE TABLE logs.nginx_logs
(
    `time_local` DateTime,
    `remote_addr` IPv4,
    `remote_user` LowCardinality(String),
    `request` String,
    `status` UInt16,
    `body_bytes_sent` UInt64,
    `http_referer` String,
    `http_user_agent` String,
    `http_x_forwarded_for` LowCardinality(String),
    `request_time` Float32,
    `upstream_response_time` Float32,
    `http_host` String
)
ENGINE = MergeTree
ORDER BY (toStartOfMinute(time_local), status, remote_addr)
```

Nginx primary key

The primary key above assumes typical access patterns in the ClickStack UI for Nginx logs, but may need to be adjusted depending on your workload in production environments.

Copy Vector configuration

### Copy Vector configuration[​](#copy-vector-configuration-nginx-oss "Direct link to Copy Vector configuration")

Ingestion to ClickStack for Vector should occur directly to ClickHouse, bypassing the OTLP endpoint exposed by the collector.

Copy the vector configuration and create a file `nginx.yaml`.