title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/etl-tools/airbyte-and-clickhouse.md)

Partner Integration

Note

Please note that the Airbyte source and destination for ClickHouse are currently in Alpha status and not suitable for moving large datasets (> 10 million rows)

[Airbyte](https://www.airbyte.com/) is an open-source data integration platform. It allows the creation of [ELT](https://airbyte.com/blog/why-the-future-of-etl-is-not-elt-but-el) data pipelines and is shipped with more than 140 out-of-the-box connectors. This step-by-step tutorial shows how to connect Airbyte to ClickHouse as a destination and load a sample dataset.

Download and run Airbyte

## Download and run Airbyte[​](#1-download-and-run-airbyte "Direct link to Download and run Airbyte")

1. Airbyte runs on Docker and uses `docker-compose`. Make sure to download and install the latest versions of Docker.
2. Deploy Airbyte by cloning the official Github repository and running `docker-compose up` in your favorite terminal:

   ```
   git clone https://github.com/airbytehq/airbyte.git --depth=1
   cd airbyte
   ./run-ab-platform.sh
   ```
3. Once you see the Airbyte banner in your terminal, you can connect to [localhost:8000](http://localhost:8000)

   Note

   Alternatively, you can signup and use [Airbyte Cloud](https://docs.airbyte.com/deploying-airbyte/on-cloud)

Add ClickHouse as a destination

## Add ClickHouse as a destination[​](#2-add-clickhouse-as-a-destination "Direct link to Add ClickHouse as a destination")

In this section, we will display how to add a ClickHouse instance as a destination.

1. Start your ClickHouse server (Airbyte is compatible with ClickHouse version `21.8.10.19` or above) or login to your ClickHouse cloud account:

   ```
   clickhouse-server start
   ```
2. Within Airbyte, select the "Destinations" page and add a new destination:
3. Select ClickHouse from the "Destination type" drop-down list, and Fill out the "Set up the destination" form by providing your ClickHouse hostname and ports, database name, username and password and select if it's an SSL connection (equivalent to the `--secure` flag in the `clickhouse-client`):
4. Congratulations! you have now added ClickHouse as a destination in Airbyte.

Note

In order to use ClickHouse as a destination, the user you'll use need to have the permissions to create databases, tables and insert rows. We recommend creating a dedicated user for Airbyte (eg. `my_airbyte_user`) with the following permissions:

```
CREATE USER 'my_airbyte_user'@'%' IDENTIFIED BY 'your_password_here';

GRANT CREATE ON * TO my_airbyte_user;
```

Add a dataset as a source

## Add a dataset as a source[​](#3-add-a-dataset-as-a-source "Direct link to Add a dataset as a source")

The example dataset we will use is the [New York City Taxi Data](https://clickhouse.com/docs/getting-started/example-datasets/nyc-taxi/) (on [Github](https://github.com/toddwschneider/nyc-taxi-data)). For this tutorial, we will use a subset of this dataset which corresponds to the month of Jan 2022.

1. Within Airbyte, select the "Sources" page and add a new source of type file.
2. Fill out the "Set up the source" form by naming the source and providing the URL of the NYC Taxi Jan 2022 file (see below). Make sure to pick `parquet` as file format, `HTTPS Public Web` as Storage Provider and `nyc_taxi_2022` as Dataset Name.

   ```
   https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2022-01.parquet
   ```
3. Congratulations! You have now added a source file in Airbyte.

Create a connection and load the dataset into ClickHouse

## Create a connection and load the dataset into ClickHouse[​](#4-create-a-connection-and-load-the-dataset-into-clickhouse "Direct link to Create a connection and load the dataset into ClickHouse")

1. Within Airbyte, select the "Connections" page and add a new connection

2. Select "Use existing source" and select the New York City Taxi Data, the select "Use existing destination" and select you ClickHouse instance.
3. Fill out the "Set up the connection" form by choosing a Replication Frequency (we will use `manual` for this tutorial) and select `nyc_taxi_2022` as the stream you want to sync. Make sure you pick `Normalized Tabular Data` as a Normalization.

4. Now that the connection is created, click on "Sync now" to trigger the data loading (since we picked `Manual` as a Replication Frequency)

5. Your data will start loading, you can expand the view to see Airbyte logs and progress. Once the operation finishes, you'll see a `Completed successfully` message in the logs:

6. Connect to your ClickHouse instance using your preferred SQL Client and check the resulting table:

   ```
   SELECT *
   FROM nyc_taxi_2022
   LIMIT 10
   ```

   The response should look like:

   ```
   Query id: 4f79c106-fe49-4145-8eba-15e1cb36d325