Ingesting with Vector

For other possible options, see the [configuration reference](/docs/use-cases/observability/clickstack/config).

Explore the data

### Explore the data[​](#explore-the-data-managed "Direct link to Explore the data")

Navigate to the logs view to explore the data and begin using ClickStack.

Create a database and table

### Create a database and table[​](#create-a-database-table-oss "Direct link to Create a database and table")

Vector requires a table and schema to be defined prior to data ingestion.

First create a database. This can be done via the [ClickHouse Web user interface](/docs/interfaces/http#web-ui) at <http://localhost:8123/play>. Use the default username and password `api:api`.

In the example below, we use `logs`:

```
CREATE DATABASE IF NOT EXISTS logs
```

Create a table for your data. This should match the output schema of your data. The example below assumes a classic Nginx structure. Adjust accordingly to your data, adhering to [schema best practices](/docs/best-practices/select-data-types). We **strongly recommend** familiarizing yourself with the [concept of Primary keys](/docs/primary-indexes), selecting your primary key based on the guidelines outlined [here](/docs/use-cases/observability/clickstack/performance_tuning#choosing-a-primary-key).

```
CREATE TABLE logs.nginx_logs
(
    `time_local` DateTime,
    `remote_addr` IPv4,
    `remote_user` LowCardinality(String),
    `request` String,
    `status` UInt16,
    `body_bytes_sent` UInt64,
    `http_referer` String,
    `http_user_agent` String,
    `http_x_forwarded_for` LowCardinality(String),
    `request_time` Float32,
    `upstream_response_time` Float32,
    `http_host` String
)
ENGINE = MergeTree
ORDER BY (toStartOfMinute(time_local), status, remote_addr)
```

Nginx primary key

The primary key above assumes typical access patterns in the ClickStack UI for Nginx logs, but may need to be adjusted depending on your workload in production environments.

Add ClickHouse sink to vector configuration

### Add ClickHouse sink to vector configuration[​](#add-clickhouse-config-to-sink-oss "Direct link to Add ClickHouse sink to vector configuration")

Ingestion to ClickStack for Vector should occur directly to ClickHouse, bypassing the OTLP endpoint exposed by the collector.

Modify your Vector configuration to include the ClickHouse sink, updating the `inputs` field to receive events from your existing pipelines.

This configuration assumes that your upstream Vector pipeline has already **prepared the data to match the target ClickHouse schema**, meaning that fields are parsed, named correctly, and typed appropriately for insertion. See the [**Nginx example below**](#example-dataset-with-vector) for a complete illustration of parsing and normalizing raw log lines into a schema suitable for ClickStack.

```
sinks:
  clickhouse:
    type: clickhouse
    inputs:
      - your_input
    endpoint: "http://localhost:8123"
    database: logs
    format: json_each_row
    table: nginx_logs
    skip_unknown_fields: true
    auth:
      strategy: "basic"
      user: "api"
      password: "api"
```

By default, we recommend using the **`json_each_row`** format, which encodes each event as a single JSON object per row. This is the default and recommended format for ClickStack when ingesting JSON data, and should be preferred over alternative formats such as JSON objects encoded as strings.

The ClickHouse sink also supports **Arrow stream encoding** (currently in beta). This can offer higher throughput but comes with important constraints: the database and table must be static, as the schema is fetched once at startup, and dynamic routing isn't supported. For this reason, Arrow encoding is best suited for fixed, well-defined ingestion pipelines.

We recommend reviewing the available sink configuration options in the [Vector documentation](https://vector.dev/docs/reference/configuration/sinks/clickhouse):

Note

The example above uses the `api` user for ClickStack Open Source. For production deployments, we recommend [creating a dedicated ingestion user](/docs/use-cases/observability/clickstack/ingesting-data/otel-collector#creating-an-ingestion-user) with appropriate permissions and limits. The above configuration also assumes that Vector is running on the same host as ClickStack. In production deployments, this is likely to be different. We would recommend sending data over the secure HTTPS port 8443.

Navigate to the ClickStack UI

### Navigate to the ClickStack UI[​](#navigate-to-clickstack-ui-oss "Direct link to Navigate to the ClickStack UI")

Navigate to the ClickStack UI at <http://localhost:8080>. Create a user if you haven't completed the onboarding.

Create a datasource

### Create a datasource[​](#create-a-datasource-oss "Direct link to Create a datasource")

Navigate to Team Settings and add a new data source.

The configuration above assumes an Nginx-style schema with a `time_local` column used as the timestamp. This should be, where possible, the timestamp column declared in the primary key. This column is mandatory.

We also recommend updating the `Default SELECT` to explicitly define which columns are returned in the logs view. If additional fields are available, such as service name, log level, or a body column, these can also be configured. The timestamp display column can also be overridden if it differs from the column used in the table's primary key and configured above.

In the example above, a `Body` column doesn't exist in the data. Instead, it is defined using a SQL expression that reconstructs an Nginx log line from the available fields.

For other possible options, see the [configuration reference](/docs/use-cases/observability/clickstack/config).

Explore the data

### Explore the data[​](#explore-the-data-oss "Direct link to Explore the data")

Navigate to the logs view to explore the data and begin using ClickStack.

## Example dataset with Vector[​](#example-dataset-with-vector "Direct link to Example dataset with Vector")

For a more complete example, we use an **Nginx log file** below.

- Managed ClickStack- Open Source ClickStack

The following guide assumes you have already created a Managed ClickStack service and recorded your service credentials. If you haven't, follow the [Getting Started](/docs/use-cases/observability/clickstack/getting-started/managed) guide for Managed ClickStack until promoted to configure Vector.

Installing Vector

### Installing Vector[​](#installing-vector "Direct link to Installing Vector")

Before proceeding, ensure that **Vector is installed** on the system where you plan to run your ingestion pipeline. Follow the [official Vector installation guide](https://vector.dev/docs/setup/installation/) to install a prebuilt binary or package appropriate for your environment:

Once installed, verify that the `vector` binary is available on your path before continuing with the configuration steps below.

This can be installed on the same instance as your ClickStack OTel collector.

Follow best practices for architecture and security when [moving Vector to production](https://vector.dev/docs/setup/going-to-prod/).

Download the sample data

### Download the sample data[​](#download-the-sample-data "Direct link to Download the sample data")

If you wish to experiment with a sample dataset, download the following example nginx sample.

```
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/access.log
```

Note

This data has been collected from an Nginx instance configured to output logs in JSON format for easier parsing. For the Nginx configuration for these logs, see ["Monitoring Nginx Logs with ClickStack"](/docs/use-cases/observability/clickstack/integrations/nginx#configure-nginx).

Create a database and table

### Create a database and table[​](#create-database-table-nginx-managed "Direct link to Create a database and table")

Vector requires a table and schema to be defined prior to data ingestion.

First create a database. This can be done via the [ClickHouse Cloud console](/docs/cloud/get-started/sql-console).

Create a database `logs`: