Integrating Amazon Kinesis with ClickHouse Cloud

## Limitations[​](#limitations "Direct link to Limitations")

- [DEFAULT](/docs/sql-reference/statements/create/table#default) isn't supported.
- Individual messages are limited to 8MB (uncompressed) by default when running with the smallest (XS) replica size, and 16MB (uncompressed) with larger replicas. Messages that exceed this limit will be rejected with an error. If you have a need for larger messages, please contact support.

## Performance[​](#performance "Direct link to Performance")

### Batching[​](#batching "Direct link to Batching")

ClickPipes inserts data into ClickHouse in batches. This is to avoid creating too many parts in the database which can lead to performance issues in the cluster.

Batches are inserted when one of the following criteria has been met:

- The batch size has reached the maximum size (100,000 rows or 32MB per 1GB of replica memory)
- The batch has been open for a maximum amount of time (5 seconds)

### Latency[​](#latency "Direct link to Latency")

Latency (defined as the time between the Kinesis message being sent to the stream and the message being available in ClickHouse) will be dependent on a number of factors (i.e. Kinesis latency, network latency, message size/format). The [batching](#batching) described in the section above will also impact latency. We always recommend testing your specific use case to understand the latency you can expect.

If you have specific low-latency requirements, please [contact us](https://clickhouse.com/company/contact?loc=clickpipes).

### Active Shards[​](#active-shards "Direct link to Active Shards")

We strongly recommend limiting the number concurrently active shards to match your throughput requirements. For an "On Demand" Kinesis stream, AWS will automatically assign a matching number of shards based on throughput,
but for "Provisioned" streams, provisioning too many shards can cause latency as described below, plus have increased costs because Kinesis pricing for such streams is on a "per shard" basis.

If your producer application writes continuously to a large number of active shards, this can cause latency if your pipe isn't scaled high enough to efficiently process those shards. Based on Kinesis throughput limits,
ClickPipes assigns a specific number of "workers" per replica to read shard data. For example, at the smallest size, a ClickPipes replica will have 4 of these worker threads. If the producer is writing
to more than 4 shards at the same time, data won't be processed from the "extra" shards until a worker thread is available. In particular, if the pipe is using "enhanced fanout", each worker thread will subscribe to a
single shard for 5 minutes, and is unavailable to read any other shard during that time. This can cause latency "spikes" of 5 minute multiples.

### Scaling[​](#scaling "Direct link to Scaling")

ClickPipes for Kinesis is designed to scale both horizontally and vertically. By default, we create a consumer group with one consumer. This can be configured during ClickPipe creation, or at any other point under **Settings** -> **Advanced Settings** -> **Scaling**.

ClickPipes provides high-availability with an availability zone distributed architecture.
This requires scaling to at least two consumers.

Regardless number of running consumers, fault tolerance is available by design.
If a consumer or its underlying infrastructure fails,
the ClickPipe will automatically restart the consumer and continue processing messages.

## Authentication[​](#authentication "Direct link to Authentication")

To access Amazon Kinesis streams, you can use [IAM credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) or an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html). For more details on how to setup an IAM role, you can [refer to this guide](/docs/integrations/clickpipes/kinesis/auth) for information on how to setup a role that works with ClickHouse Cloud

- [Prerequisite](#prerequisite)- [Creating your first ClickPipe](#creating-your-first-clickpipe)- [Supported data formats](#supported-data-formats)- [Compression](#compression)- [Supported data types](#supported-data-types)
          - [Standard types support](#standard-types-support)- [Variant type support](#variant-type-support)- [JSON type support](#json-type-support)- [Kinesis virtual columns](#kinesis-virtual-columns)- [Limitations](#limitations)- [Performance](#performance)
                - [Batching](#batching)- [Latency](#latency)- [Active Shards](#active-shards)- [Scaling](#scaling)- [Authentication](#authentication)

Was this page helpful?