ClickHouse/clickhouse-docs-embeddings huggingface dataset:
Integrating Amazon MSK with ClickHouse

## Notes on networking for MSK Connect[​](#notes-on-networking-for-msk-connect "Direct link to Notes on networking for MSK Connect")

In order for MSK Connect to connect to ClickHouse, we recommend your MSK cluster to be in a private subnet with a Private NAT connected for internet access. Instructions on how to set this up are provided below. Note that public subnets are supported but not recommended due to the need to constantly assign an Elastic IP address to your ENI, [AWS provides more details here](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-internet-access.html)

1. **Create a Private Subnet:** Create a new subnet within your VPC, designating it as a private subnet. This subnet shouldn't have direct access to the internet.
2. **Create a NAT Gateway:** Create a NAT gateway in a public subnet of your VPC. The NAT gateway enables instances in your private subnet to connect to the internet or other AWS services, but prevents the internet from initiating a connection with those instances.
3. **Update the Route Table:** Add a route that directs internet-bound traffic to the NAT gateway
4. **Ensure Security Groups and Network ACLs Configuration:** Configure your [security groups](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-security-groups.html) and [network ACLs (Access Control Lists)](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html) to allow relevant traffic.
   1. From MSK Connect worker ENIs to MSK brokers on TLS port (commonly 9094).
   2. From MSK Connect worker ENIs to ClickHouse endpoint: 9440 (native TLS) or 8443 (HTTPS).
   3. Allow inbound on broker SG from the MSK Connect worker SG.
   4. For self-hosted ClickHouse, open the port configured in your server (default 8123 for HTTP).
5. **Attach Security Groups to MSK:** Ensure that these security groups are attached to your MSK cluster and MSK Connect workers.
6. **Connectivity to ClickHouse Cloud:**
   1. Public endpoint + IP allowlist: requires NAT egress from private subnets.
   2. Private connectivity where available (e.g., VPC peering/PrivateLink/VPN). Ensure VPC DNS hostnames/resolution are enabled and DNS can resolve the private endpoint.
7. **Validate connectivity (quick checklist):**
   1. From the connector environment, resolve MSK bootstrap DNS and connect via TLS to broker port.
   2. Establish TLS connection to ClickHouse on port 9440 (or 8443 for HTTPS).
   3. If using AWS services (Glue/Secrets Manager), allow egress to those endpoints.

- [Prerequisites](#prerequisites)- [The official Kafka connector from ClickHouse with Amazon MSK](#the-official-kafka-connector-from-clickhouse-with-amazon-msk)
    - [Gather your connection details](#gather-your-connection-details)- [Steps](#steps)- [Recommended IAM permissions (least privilege)](#iam-least-privilege)- [Performance tuning](#performance-tuning)- [Notes on networking for MSK Connect](#notes-on-networking-for-msk-connect)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/kafka/confluent/index.md)

Confluent platform provides two options to integration with ClickHouse

- [ClickHouse Connect Sink on Confluent Cloud](/docs/integrations/kafka/cloud/confluent/sink-connector)
- [ClickHouse Connect Sink on Confluent Platform](/docs/integrations/kafka/cloud/confluent/custom-connector) using the custom connectors feature
- [HTTP Sink Connector for Confluent Platform](/docs/integrations/kafka/cloud/confluent/http) that integrates Apache Kafka with an API via HTTP or HTTPS

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/language-clients/index.md)

In this section of the documentation, you can learn more about the many language client integrations
that ClickHouse offers.

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Page Description|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [C++](/docs/interfaces/cpp) C++ Client Library and userver Asynchronous Framework|  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [C#](/docs/integrations/csharp) Learn how to connect your C# projects to ClickHouse.|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [Go](/docs/integrations/go) Learn how to connect your Go projects to ClickHouse.|  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [JavaScript](/docs/integrations/javascript) Learn how to connect your JS projects to ClickHouse with the official JS client.|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | [Java](/docs/integrations/java) Learn more about several integrations for Java and ClickHouse.|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | [Python](/docs/integrations/python) Learn how to connect your Python projects to ClickHouse.|  |  |  |  | | --- | --- | --- | --- | | [Rust](/docs/integrations/rust) Learn how to connect your Rust projects to ClickHouse.|  |  | | --- | --- | | [Third-party clients](/docs/interfaces/third-party/client-libraries) Learn more about client libraries from third party developers. | | | | | | | | | | | | | | | | | |

Was this page helpful?
Java Client

### Custom HTTP Header[​](#custom-http-header "Direct link to Custom HTTP Header")

Custom HTTP headers can be set for all operations (client level) or a single one (operation level).

```
QuerySettings settings = new QuerySettings()
    .httpHeader(HttpHeaders.REFERER, clientReferer)
    .setQueryId(qId);
```

When options are set via `setOption` method (either the `Client.Builder` or operation settings class) then custom header name should be prefixed with `http_header_`. Method `com.clickhouse.client.api.ClientConfigProperties#httpHeader()` may be handy in this case.

## Common Definitions[​](#common-definitions "Direct link to Common Definitions")

### ClickHouseFormat[​](#clickhouseformat "Direct link to ClickHouseFormat")

Enum of [supported formats](/docs/interfaces/formats). It includes all formats that ClickHouse supports.

- `raw` - user should transcode raw data
- `full` - the client can transcode data by itself and accepts a raw data stream
- `-` - operation not supported by ClickHouse for this format

This client version supports:
Java Client

### Adding custom http headers[​](#v1-adding-custom-http-headers "Direct link to Adding custom http headers")

Java client support HTTP/S transport layer in case we want to add custom HTTP headers to the request.
We should use the custom\_http\_headers property, and the headers need to be `,` separated. The header key/value should be divided using `=`

## Java Client support[​](#v1-java-client-support "Direct link to Java Client support")

```
options.put("custom_http_headers", "X-ClickHouse-Quota=test, X-ClickHouse-Test=test");
```

## JDBC Driver[​](#v1-jdbc-driver "Direct link to JDBC Driver")

```
properties.setProperty("custom_http_headers", "X-ClickHouse-Quota=test, X-ClickHouse-Test=test");
```

- [Setup](#setup)- [Initialization](#initialization)
    - [Authentication](#authentication)- [Configuration](#configuration)- [Client Configuration](#client-configuration)
        - [Client Identification](#client-identification)- [Operation Identification](#operation-identification)- [Server Settings](#server-settings)- [Custom HTTP Header](#custom-http-header)- [Common Definitions](#common-definitions)
          - [ClickHouseFormat](#clickhouseformat)- [Insert API](#insert-api)
            - [insert(String tableName, InputStream data, ClickHouseFormat format)](#insertstring-tablename-inputstream-data-clickhouseformat-format)- [insert(String tableName, List<?> data, InsertSettings settings)](#insertstring-tablename-listlt-data-insertsettings-settings)- [InsertSettings](#insertsettings)- [InsertResponse](#insertresponse)- [Query API](#query-api)
              - [query(String sqlQuery)](#querystring-sqlquery)- [query(String sqlQuery, Map<String, Object> queryParams, QuerySettings settings)](#querystring-sqlquery-mapltstring-object-queryparams-querysettings-settings)- [queryAll(String sqlQuery)](#queryallstring-sqlquery)- [QuerySettings](#querysettings)- [QueryResponse](#queryresponse)- [Examples](#examples)- [Common API](#common-api)
                - [getTableSchema(String table)](#gettableschemastring-table)- [getTableSchemaFromQuery(String sql)](#gettableschemafromquerystring-sql)- [TableSchema](#tableschema)- [register(Class<?> clazz, TableSchema schema)](#registerclasslt-clazz-tableschema-schema)- [Usage Examples](#usage-examples)- [Reading Data](#reading-data)
                    - [Reading Arrays](#reading-arrays)- [Reading Data](#reading-data)- [Setup](#v1-setup)- [Initialization](#v1-initialization)- [Query API](#v1-query-api)- [Streaming Query API](#v1-streaming-query-api)- [Insert API](#v1-insert-api)- [Features](#v1-features)
                                - [Compression](#v1-compression)- [Multiple queries](#v1-multiple-queries)- [Named Parameters](#v1-named-parameters)- [Node Discovery](#v1-node-discovery)
                                  - [Load Balancing](#v1-load-balancing)- [Failover and retry](#v1-failover-and-retry)- [Adding custom http headers](#v1-adding-custom-http-headers)- [Java Client support](#v1-java-client-support)- [JDBC Driver](#v1-jdbc-driver)

Was this page helpful?
Advanced Inserting

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/language-clients/python/advanced-inserting.md)

## Inserting data with ClickHouse Connect: Advanced usage[​](#inserting-data-with-clickhouse-connect--advanced-usage "Direct link to Inserting data with ClickHouse Connect: Advanced usage")

### InsertContexts[​](#insertcontexts "Direct link to InsertContexts")

ClickHouse Connect executes all inserts within an `InsertContext`. The `InsertContext` includes all the values sent as arguments to the client `insert` method. In addition, when an `InsertContext` is originally constructed, ClickHouse Connect retrieves the data types for the insert columns required for efficient Native format inserts. By reusing the `InsertContext` for multiple inserts, this "pre-query" is avoided and inserts are executed more quickly and efficiently.

An `InsertContext` can be acquired using the client `create_insert_context` method. The method takes the same arguments as the `insert` function. Note that only the `data` property of `InsertContext`s should be modified for reuse. This is consistent with its intended purpose of providing a reusable object for repeated inserts of new data to the same table.

```
test_data = [[1, 'v1', 'v2'], [2, 'v3', 'v4']]
ic = test_client.create_insert_context(table='test_table', data='test_data')
client.insert(context=ic)
assert client.command('SELECT count() FROM test_table') == 2
new_data = [[3, 'v5', 'v6'], [4, 'v7', 'v8']]
ic.data = new_data
client.insert(context=ic)
qr = test_client.query('SELECT * FROM test_table ORDER BY key DESC')
assert qr.row_count == 4
assert qr[0][0] == 4
```

`InsertContext`s include mutable state that is updated during the insert process, so they're not thread safe.

### Write formats[​](#write-formats "Direct link to Write formats")

Write formats are currently implemented for limited number of types. In most cases ClickHouse Connect will attempt to automatically determine the correct write format for a column by checking the type of the first (non-null) data value. For example, if inserting into a `DateTime` column, and the first insert value of the column is a Python integer, ClickHouse Connect will directly insert the integer value under the assumption that it's actually an epoch second.

In most cases, it is unnecessary to override the write format for a data type, but the associated methods in the `clickhouse_connect.datatypes.format` package can be used to do so at a global level.

#### Write format options[​](#write-format-options "Direct link to Write format options")
Lightdash

---

Build dashboards

### Build dashboards[​](#building-dashboards "Direct link to Build dashboards")

Once you’ve explored your data and saved visualizations, you can combine them into **dashboards** to share with your team.

Dashboards in Lightdash are fully interactive — you can apply filters, add tabs, and view charts powered by real-time ClickHouse queries.

You can also create new charts **directly from within a dashboard**, which helps keep your projects organized and clutter-free. Charts created this way are **exclusive to that dashboard** — they can’t be reused elsewhere in the project.

To create a dashboard-only chart:

1. Click **Add tile**
2. Select **New chart**
3. Build your visualization in the chart builder
4. Save it — it will appear at the bottom of your dashboard

Learn more about how to create and organize dashboards here:  
[Building dashboards → Lightdash Docs](https://docs.lightdash.com/get-started/exploring-data/dashboards?utm_source=clickhouse&utm_medium=partner&utm_campaign=integration_docs)

---

Ask AI: self-serve analytics powered by dbt

### Ask AI: self-serve analytics powered by dbt[​](#ask-ai "Direct link to Ask AI: self-serve analytics powered by dbt")

**AI Agents** in Lightdash make data exploration truly self-serve.  
Instead of writing queries, you can simply ask questions in plain language — like *“What was our monthly revenue growth?”* — and the AI Agent automatically generates the right visualization, referencing your dbt-defined metrics and models to ensure accuracy and consistency.

It’s powered by the same semantic layer you use in dbt, meaning every answer stays governed, explainable, and fast — all backed by ClickHouse.

Tip

Learn more about AI Agents here: [AI Agents → Lightdash Docs](https://docs.lightdash.com/guides/ai-agents?utm_source=clickhouse&utm_medium=partner&utm_campaign=integration_docs)

## Learn more[​](#learn-more "Direct link to Learn more")

To learn more about connecting dbt projects to Lightdash, visit the [Lightdash Docs → ClickHouse setup](https://docs.lightdash.com/get-started/setup-lightdash/connect-project#clickhouse?utm_source=clickhouse&utm_medium=partner&utm_campaign=integration_docs).

- [Build an interactive dashboard with Lightdash and ClickHouse](#build-an-interactive-dashboard)
  - [Gather connection data](#connection-data-required)- [Configure your dbt profile for ClickHouse](#configuring-your-dbt-profile-for-clickhouse)- [Create a Lightdash project connected to ClickHouse](#creating-a-lightdash-project-connected-to-clickhouse)- [Explore your ClickHouse data in Lightdash](#exploring-your-clickhouse-data-in-lightdash)- [Define metrics and dimensions in Lightdash](#defining-metrics-and-dimensions-in-lightdash)- [Query your data from tables](#querying-your-data-from-tables)- [Build dashboards](#building-dashboards)- [Ask AI: self-serve analytics powered by dbt](#ask-ai)- [Learn more](#learn-more)

Was this page helpful?
Looker

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-visualization/looker-and-clickhouse.md)

Partner Integration

Looker can connect to ClickHouse Cloud or on-premise deployment via the official ClickHouse data source.

## 1. Gather your connection details[​](#1-gather-your-connection-details "Direct link to 1. Gather your connection details")

To connect to ClickHouse with HTTP(S) you need this information:

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter(s) Description|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | `HOST` and `PORT` Typically, the port is 8443 when using TLS or 8123 when not using TLS.|  |  |  |  | | --- | --- | --- | --- | | `DATABASE NAME` Out of the box, there is a database named `default`, use the name of the database that you want to connect to.| `USERNAME` and `PASSWORD` Out of the box, the username is `default`. Use the username appropriate for your use case. | | | | | | | |

The details for your ClickHouse Cloud service are available in the ClickHouse Cloud console.
Select a service and click **Connect**:

Choose **HTTPS**. Connection details are displayed in an example `curl` command.

If you're using self-managed ClickHouse, the connection details are set by your ClickHouse administrator.

## 2. Create a ClickHouse data source[​](#2-create-a-clickhouse-data-source "Direct link to 2. Create a ClickHouse data source")

Navigate to Admin -> Database -> Connections and click the "Add Connection" button in the top right corner.

  

Choose a name for your data source, and select `ClickHouse` from the dialect drop-down. Enter your credentials in the form.

  

If you're using ClickHouse Cloud or your deployment requires SSL, make sure you have SSL turned on in the additional settings.

  

Test your connection first, and, once it is done, connect to your new ClickHouse data source.

  

Now you should be able to attach ClickHouse Datasource to your Looker project.

## 3. Known limitations[​](#3-known-limitations "Direct link to 3. Known limitations")

1. The following data types are handled as strings by default:
   - Array - serialization doesn't work as expected due to the JDBC driver limitations
   - Decimal\* - can be changed to number in the model
   - LowCardinality(...) - can be changed to a proper type in the model
   - Enum8, Enum16
   - UUID
   - Tuple
   - Map
   - JSON
   - Nested
   - FixedString
   - Geo types
     - MultiPolygon
     - Polygon
     - Point
     - Ring
2. [Symmetric aggregate feature](https://cloud.google.com/looker/docs/reference/param-explore-symmetric-aggregates) isn't supported
3. [Full outer join](https://cloud.google.com/looker/docs/reference/param-explore-join-type#full_outer) isn't yet implemented in the driver

- [1. Gather your connection details](#1-gather-your-connection-details)- [2. Create a ClickHouse data source](#2-create-a-clickhouse-data-source)- [3. Known limitations](#3-known-limitations)

Was this page helpful?
Looker Studio

```
mysql --protocol tcp -h localhost -u mysql_user -P 9004 --password=LZOQYnqQN4L/T6L0
```

```
mysql> show databases;
+--------------------+
| name               |
+--------------------+
| INFORMATION_SCHEMA |
| default            |
| information_schema |
| system             |
+--------------------+
4 rows in set (0.00 sec)
Read 4 rows, 603.00 B in 0.00156 sec., 2564 rows/sec., 377.48 KiB/sec.
```

Finally, configure the Clickhouse Server to listen on the desired IP address(es). For example, in `config.xml`, uncomment out the following to listen on all addresses:

```
<listen_host>::</listen_host>
```

## Connecting Looker Studio to ClickHouse[​](#connecting-looker-studio-to-clickhouse "Direct link to Connecting Looker Studio to ClickHouse")

First, login to <https://lookerstudio.google.com> using your Google account and create a new Data Source:

  

Search for the official MySQL connector provided by Google (named just **MySQL**):

  

Specify your connection details. Please note that MySQL interface port is 9004 by default,
and it might be different depending on your server configuration.

  

Now, you have two options on how to fetch the data from ClickHouse. First, you could use the Table Browser feature:

  

Alternatively, you could specify a custom query to fetch your data:

  

Finally, you should be able to see the introspected table structure and adjust the data types if necessary.

  

Now you can proceed with exploring your data or creating a new report!

## Using Looker Studio with ClickHouse Cloud[​](#using-looker-studio-with-clickhouse-cloud "Direct link to Using Looker Studio with ClickHouse Cloud")

When using ClickHouse Cloud, you need to enable MySQL interface first. You can do that in connection dialog, "MySQL" tab.

  

In the Looker Studio UI, choose the "Enable SSL" option. ClickHouse Cloud's SSL certificate is signed by [Let's Encrypt](https://letsencrypt.org/certificates/). You can download this root cert [here](https://letsencrypt.org/certs/isrgrootx1.pem).

  

The rest of the steps are the same as listed above in the previous section.

- [ClickHouse Cloud setup](#clickhouse-cloud-setup)- [Creating a readonly MySQL user in ClickHouse Cloud](#creating-multiple-mysql-users-in-clickhouse-cloud)
    - [Troubleshooting multiple MySQL users in ClickHouse Cloud](#troubleshooting-multiple-mysql-users-in-clickhouse-cloud)- [On-premise ClickHouse server setup](#on-premise-clickhouse-server-setup)- [Connecting Looker Studio to ClickHouse](#connecting-looker-studio-to-clickhouse)- [Using Looker Studio with ClickHouse Cloud](#using-looker-studio-with-clickhouse-cloud)

Was this page helpful?
Move data from cloud object storage to ClickHouse Cloud

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/03_object-storage-to-clickhouse.md)

If you use a Cloud Object Storage as a data lake and wish to import this data into ClickHouse Cloud,
or if your current database system is able to directly offload data into a Cloud Object Storage, then you can use one of the
table functions for migrating data stored in Cloud Object Storage into a ClickHouse Cloud table:

- [s3](/docs/sql-reference/table-functions/s3) or [s3Cluster](/docs/sql-reference/table-functions/s3Cluster)
- [gcs](/docs/sql-reference/table-functions/gcs)
- [azureBlobStorage](/docs/sql-reference/table-functions/azureBlobStorage)

If your current database system isn't able to directly offload data into a Cloud Object Storage, you could use a [third-party ETL/ELT tool](/docs/cloud/migration/etl-tool-to-clickhouse) or [clickhouse-local](/docs/cloud/migration/clickhouse-local) for moving data
from you current database system to Cloud Object Storage, in order to migrate that data in a second step into a ClickHouse Cloud table.

Although this is a two steps process (offload data into a Cloud Object Storage, then load into ClickHouse), the advantage is that this
scales to petabytes thanks to a [solid ClickHouse Cloud](https://clickhouse.com/blog/getting-data-into-clickhouse-part-3-s3) support of highly-parallel reads from Cloud Object Storage.
Also you can leverage sophisticated and compressed formats like [Parquet](/docs/interfaces/formats/Parquet).

There is a [blog article](https://clickhouse.com/blog/getting-data-into-clickhouse-part-3-s3) with concrete code examples showing how you can get data into ClickHouse Cloud using S3.

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/cloud/onboard/02_migrate/01_migration_guides/01_overview.md)

There are several options for migrating data into ClickHouse Cloud, depending on where your data resides now:

- [Self-managed to Cloud](/docs/cloud/migration/clickhouse-to-cloud): use the `remoteSecure` function to transfer data
- [Another DBMS](/docs/cloud/migration/clickhouse-local): use the [clickhouse-local] ETL tool along with the appropriate ClickHouse table function for your current DBMS
- [Anywhere!](/docs/cloud/migration/etl-tool-to-clickhouse): use one of the many popular ETL/ELT tools that connect to all kinds of different data sources
- [Object Storage](/docs/integrations/migration/object-storage-to-clickhouse): easily insert data from S3 into ClickHouse

In the example [Migrate from Redshift](/docs/migrations/redshift/migration-guide), we present three different ways to migrate data to ClickHouse.

Was this page helpful?