Integrating with ClickHouse Cloud

### Streaming ClickPipes[​](#clickpipes-advanced-settings-streaming "Direct link to Streaming ClickPipes")

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Setting Default value Description|  |  |  | | --- | --- | --- | | `Streaming max insert wait time` 5 s Configures the maximum wait period before inserting data into the ClickHouse cluster. | | | | | |

## Error reporting[​](#error-reporting "Direct link to Error reporting")

ClickPipes will store errors in two separate tables depending on the type of error encountered during the ingestion process.

### Record errors[​](#record-errors "Direct link to Record errors")

ClickPipes will create a table next to your destination table with the postfix `<destination_table_name>_clickpipes_error`. This table will contain any errors from malformed data or mismatched schema and will include the entirety of the invalid message. This table has a [TTL](/docs/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.

### System errors[​](#system-errors "Direct link to System errors")

Errors related to the operation of the ClickPipe will be stored in the `system.clickpipes_log` table. This will store all other errors related to the operation of your ClickPipe (network, connectivity, etc.). This table has a [TTL](/docs/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-ttl) of 7 days.

If ClickPipes can't connect to a data source after 15 min or to a destination after 1 hr, the ClickPipes instance stops and stores an appropriate message in the system error table (provided the ClickHouse instance is available).

## FAQ[​](#faq "Direct link to FAQ")

- **What is ClickPipes?**

  ClickPipes is a ClickHouse Cloud feature that makes it easy for you to connect your ClickHouse services to external data sources, specifically Kafka. With ClickPipes for Kafka, you can easily continuously load data into ClickHouse, making it available for real-time analytics.
- **Does ClickPipes support data transformation?**

  Yes, ClickPipes supports basic data transformation by exposing the DDL creation. You can then apply more advanced transformations to the data as it is loaded into its destination table in a ClickHouse Cloud service leveraging ClickHouse's [materialized views feature](/docs/guides/developer/cascading-materialized-views).
- **Does using ClickPipes incur an additional cost?**

  ClickPipes is billed on two dimensions: Ingested Data and Compute. The full details of the pricing are available on [this page](/docs/cloud/reference/billing/clickpipes). Running ClickPipes might also generate an indirect compute and storage cost on the destination ClickHouse Cloud service similar to any ingest workload.
- **Is there a way to handle errors or failures when using ClickPipes for Kafka?**

  Yes, ClickPipes for Kafka will automatically retry in the event of failures when consuming data from Kafka for any operational issue including network issues, connectivity issues, etc. In the event of malformed data or invalid schema, ClickPipes will store the record in the record\_error table and continue processing.

- [Introduction](#introduction)- [Supported data sources](#supported-data-sources)- [List of Static IPs](#list-of-static-ips)- [Adjusting ClickHouse settings](#adjusting-clickhouse-settings)- [Adjusting ClickPipes advanced settings](#clickpipes-advanced-settings)
          - [Object storage ClickPipes](#clickpipes-advanced-settings-object-storage)- [Streaming ClickPipes](#clickpipes-advanced-settings-streaming)- [Error reporting](#error-reporting)
            - [Record errors](#record-errors)- [System errors](#system-errors)- [FAQ](#faq)

Was this page helpful?