Integrating Amazon Kinesis with ClickHouse Cloud

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/clickpipes/kinesis/01_overview.md)

## Prerequisite[​](#prerequisite "Direct link to Prerequisite")

You have familiarized yourself with the [ClickPipes intro](/docs/integrations/clickpipes) and setup [IAM credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) or an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html). Follow the [Kinesis Role-Based Access guide](/docs/integrations/clickpipes/kinesis/auth) for information on how to setup a role that works with ClickHouse Cloud.

## Creating your first ClickPipe[​](#creating-your-first-clickpipe "Direct link to Creating your first ClickPipe")

1. Access the SQL Console for your ClickHouse Cloud Service.

2. Select the `Data Sources` button on the left-side menu and click on "Set up a ClickPipe"

3. Select your data source.

4. Fill out the form by providing your ClickPipe with a name, a description (optional), your IAM role or credentials, and other connection details.

5. Select Kinesis Stream and starting offset. The UI will display a sample document from the selected source (Kafka topic, etc). You can also enable Enhanced Fan-out for Kinesis streams to improve the performance and stability of your ClickPipe (More information on Enhanced Fan-out can be found [here](https://aws.amazon.com/blogs/aws/kds-enhanced-fanout))

6. In the next step, you can select whether you want to ingest data into a new ClickHouse table or reuse an existing one. Follow the instructions in the screen to modify your table name, schema, and settings. You can see a real-time preview of your changes in the sample table at the top.

You can also customize the advanced settings using the controls provided

7. Alternatively, you can decide to ingest your data in an existing ClickHouse table. In that case, the UI will allow you to map fields from the source to the ClickHouse fields in the selected destination table.

8. Finally, you can configure permissions for the internal ClickPipes user.

**Permissions:** ClickPipes will create a dedicated user for writing data into a destination table. You can select a role for this internal user using a custom role or one of the predefined role:

- `Full access`: with the full access to the cluster. It might be useful if you use materialized view or Dictionary with the destination table.
- `Only destination table`: with the `INSERT` permissions to the destination table only.

9. By clicking on "Complete Setup", the system will register you ClickPipe, and you'll be able to see it listed in the summary table.

The summary table provides controls to display sample data from the source or the destination table in ClickHouse

As well as controls to remove the ClickPipe and display a summary of the ingest job.

10. **Congratulations!** you have successfully set up your first ClickPipe. If this is a streaming ClickPipe it will be continuously running, ingesting data in real-time from your remote data source. Otherwise it will ingest the batch and complete.

## Supported data formats[​](#supported-data-formats "Direct link to Supported data formats")

The supported formats are:

- [JSON](/docs/interfaces/formats/JSON)

## Compression[​](#compression "Direct link to Compression")

ClickPipes for Kinesis automatically detects and decompresses compressed records. Unlike Kafka, where the client library handles decompression transparently, Kinesis delivers raw bytes — ClickPipes handles this for you with no configuration required.

The following compression codecs are supported:

- **gzip**
- **zstd**
- **lz4**
- **snappy** (framed format)

Compression is detected automatically via magic bytes in each record. If no known compression signature is found, the record is treated as uncompressed. The detected compression type is also surfaced during schema inference, so the sample data preview in the UI will correctly show the decompressed data.

Note

Auto-detection is safe for text-based formats like JSON and CSV, as printable ASCII characters will never collide with compression magic bytes.

## Supported data types[​](#supported-data-types "Direct link to Supported data types")

### Standard types support[​](#standard-types-support "Direct link to Standard types support")

The following ClickHouse data types are currently supported in ClickPipes:

- Base numeric types - [U]Int8/16/32/64, Float32/64, and BFloat16
- Large integer types - [U]Int128/256
- Decimal Types
- Boolean
- String
- FixedString
- Date, Date32
- DateTime, DateTime64 (UTC timezones only)
- Enum8/Enum16
- UUID
- IPv4
- IPv6
- all ClickHouse LowCardinality types
- Map with keys and values using any of the above types (including Nullables)
- Tuple and Array with elements using any of the above types (including Nullables, one level depth only)
- SimpleAggregateFunction types (for AggregatingMergeTree or SummingMergeTree destinations)

### Variant type support[​](#variant-type-support "Direct link to Variant type support")

You can manually specify a Variant type (such as `Variant(String, Int64, DateTime)`) for any JSON field
in the source data stream. Because of the way ClickPipes determines the correct variant subtype to use, only one integer or datetime
type can be used in the Variant definition - for example, `Variant(Int64, UInt32)` isn't supported.

### JSON type support[​](#json-type-support "Direct link to JSON type support")

JSON fields that are always a JSON object can be assigned to a JSON destination column. You will have to manually change the destination
column to the desired JSON type, including any fixed or skipped paths.

## Kinesis virtual columns[​](#kinesis-virtual-columns "Direct link to Kinesis virtual columns")

The following virtual columns are supported for Kinesis stream. When creating a new destination table virtual columns can be added by using the `Add Column` button.

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Name Description Recommended Data Type|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | \_key Kinesis Partition Key String|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | \_timestamp Kinesis Approximate Arrival Timestamp (millisecond precision) DateTime64(3)|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | \_stream Kinesis Stream Name String|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | \_sequence\_number Kinesis Sequence Number String|  |  |  | | --- | --- | --- | | \_raw\_message Full Kinesis Message String | | | | | | | | | | | | | | | | | |

The \_raw\_message field can be used in cases where only full Kinesis JSON record is required (such as using ClickHouse [`JsonExtract*`](/docs/sql-reference/functions/json-functions#jsonextract-functions) functions to populate a downstream materialized
view). For such pipes, it may improve ClickPipes performance to delete all the "non-virtual" columns.

## Limitations[​](#limitations "Direct link to Limitations")

- [DEFAULT](/docs/sql-reference/statements/create/table#default) isn't supported.
- Individual messages are limited to 8MB (uncompressed) by default when running with the smallest (XS) replica size, and 16MB (uncompressed) with larger replicas. Messages that exceed this limit will be rejected with an error. If you have a need for larger messages, please contact support.

## Performance[​](#performance "Direct link to Performance")

### Batching[​](#batching "Direct link to Batching")

ClickPipes inserts data into ClickHouse in batches. This is to avoid creating too many parts in the database which can lead to performance issues in the cluster.

Batches are inserted when one of the following criteria has been met: