ClickHouse/clickhouse-docs-embeddings huggingface dataset:
Connect DBeaver to ClickHouse

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/sql-clients/dbeaver.md)

Partner Integration

DBeaver is available in multiple offerings. In this guide [DBeaver Community](https://dbeaver.io/) is used. See the various offerings and capabilities [here](https://dbeaver.com/edition/). DBeaver connects to ClickHouse using JDBC.

Note

Please use DBeaver version 23.1.0 or above for improved support of `Nullable` columns in ClickHouse.

## 1. Gather your ClickHouse details[​](#1-gather-your-clickhouse-details "Direct link to 1. Gather your ClickHouse details")

DBeaver uses JDBC over HTTP(S) to connect to ClickHouse; you need:

- endpoint
- port number
- username
- password

## 2. Download DBeaver[​](#2-download-dbeaver "Direct link to 2. Download DBeaver")

DBeaver is available at <https://dbeaver.io/download/>

## 3. Add a database[​](#3-add-a-database "Direct link to 3. Add a database")

- Either use the **Database > New Database Connection** menu or the **New Database Connection** icon in the **Database Navigator** to bring up the **Connect to a database** dialog:

- Select **Analytical** and then **ClickHouse**:
- Build the JDBC URL. On the **Main** tab set the Host, Port, Username, Password, and Database:

- By default the **SSL > Use SSL** property will be unset, if you're connecting to ClickHouse Cloud or a server that requires SSL on the HTTP port, then set **SSL > Use SSL** on:

- Test the connection:

If DBeaver detects that you don't have the ClickHouse driver installed it will offer to download them for you:

- After downloading the driver **Test** the connection again:

## 4. Query ClickHouse[​](#4-query-clickhouse "Direct link to 4. Query ClickHouse")

Open a query editor and run a query.

- Right click on your connection and choose **SQL Editor > Open SQL Script** to open a query editor:

- An example query against `system.query_log`:

## Next steps[​](#next-steps "Direct link to Next steps")

See the [DBeaver wiki](https://github.com/dbeaver/dbeaver/wiki) to learn about the capabilities of DBeaver, and the [ClickHouse documentation](https://clickhouse.com/docs) to learn about the capabilities of ClickHouse.

- [1. Gather your ClickHouse details](#1-gather-your-clickhouse-details)- [2. Download DBeaver](#2-download-dbeaver)- [3. Add a database](#3-add-a-database)- [4. Query ClickHouse](#4-query-clickhouse)- [Next steps](#next-steps)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/tools/data-integration/easypanel/index.md)

Community Maintained

[Easypanel](https://easypanel.io) it's a modern server control panel. You can use it to deploy ClickHouse on your own server.

[![Deploy to Easypanel](https://easypanel.io/img/deploy-on-easypanel-40.svg)](https://easypanel.io/docs/templates/clickhouse)

## Instructions[​](#instructions "Direct link to Instructions")

1. Create a VM that runs Ubuntu on your cloud provider.
2. Install Easypanel using the instructions from the website.
3. Create a new project.
4. Install ClickHouse using the dedicated template.

- [Instructions](#instructions)

Was this page helpful?
Connecting ClickHouse to Fabi.ai

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-visualization/community_integrations/fabi-and-clickhouse.md)

Community Maintained

[Fabi.ai](https://www.fabi.ai/) is an all-in-one collaborate data analysis platform. You can leverage SQL, Python, AI, and no-code to build dashboard and data workflows faster than ever before. Combined with the scale and power of ClickHouse, you can build and share your first highly performant dashboard on a massive dataset in minutes.

## Gather Your Connection Details[​](#gather-your-connection-details "Direct link to Gather Your Connection Details")

To connect to ClickHouse with HTTP(S) you need this information:

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter(s) Description|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | `HOST` and `PORT` Typically, the port is 8443 when using TLS or 8123 when not using TLS.|  |  |  |  | | --- | --- | --- | --- | | `DATABASE NAME` Out of the box, there is a database named `default`, use the name of the database that you want to connect to.| `USERNAME` and `PASSWORD` Out of the box, the username is `default`. Use the username appropriate for your use case. | | | | | | | |

The details for your ClickHouse Cloud service are available in the ClickHouse Cloud console.
Select a service and click **Connect**:

Choose **HTTPS**. Connection details are displayed in an example `curl` command.

If you're using self-managed ClickHouse, the connection details are set by your ClickHouse administrator.

## Create your Fabi.ai account and connect ClickHouse[​](#connect-to-clickhouse "Direct link to Create your Fabi.ai account and connect ClickHouse")

Log in or create your Fabi.ai account: <https://app.fabi.ai/>

1. You’ll be prompted to connect your database when you first create your account, or if you already have an account, click on the data source panel on the left of any Smartbook and select Add Data Source.
2. You’ll then be prompted to enter your connection details.
3. Congratulations! You have now integrated ClickHouse into Fabi.ai.

## Querying ClickHouse.[​](#querying-clickhouse "Direct link to Querying ClickHouse.")

Once you’ve connected Fabi.ai to ClickHouse, go to any [Smartbook](https://docs.fabi.ai/analysis_and_reporting/smartbooks) and create a SQL cell. If you only have one data source connected to your Fabi.ai instance, the SQL cell will automatically default to ClickHouse, otherwise you can choose the source to query from the source dropdown.

## Additional Resources[​](#additional-resources "Direct link to Additional Resources")

[Fabi.ai](https://www.fabi.ai) documentation: <https://docs.fabi.ai/introduction>

[Fabi.ai](https://www.fabi.ai) getting started tutorial videos: <https://www.youtube.com/playlist?list=PLjxPRVnyBCQXxxByw2CLC0q7c-Aw6t2nl>

- [Gather Your Connection Details](#gather-your-connection-details)- [Create your Fabi.ai account and connect ClickHouse](#connect-to-clickhouse)- [Querying ClickHouse.](#querying-clickhouse)- [Additional Resources](#additional-resources)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/google-dataflow/dataflow.md)

ClickHouse Supported

[Google Dataflow](https://cloud.google.com/dataflow) is a fully managed stream and batch data processing service. It supports pipelines written in Java or Python and is built on the Apache Beam SDK.

There are two main ways to use Google Dataflow with ClickHouse, both of which leverage [`ClickHouseIO Apache Beam connector`](/docs/integrations/apache-beam).
These are:

- [Java runner](#1-java-runner)
- [Predefined templates](#2-predefined-templates)

## Java runner[​](#1-java-runner "Direct link to Java runner")

The [Java runner](/docs/integrations/google-dataflow/java-runner) allows you to implement custom Dataflow pipelines using the Apache Beam SDK `ClickHouseIO` integration. This approach provides full flexibility and control over the pipeline logic, enabling you to tailor the ETL process to specific requirements.
However, this option requires knowledge of Java programming and familiarity with the Apache Beam framework.

### Key features[​](#key-features "Direct link to Key features")

- High degree of customization.
- Ideal for complex or advanced use cases.
- Requires coding and understanding of the Beam API.

## Predefined templates[​](#2-predefined-templates "Direct link to Predefined templates")

ClickHouse offers [predefined templates](/docs/integrations/google-dataflow/templates) designed for specific use cases, such as importing data from BigQuery into ClickHouse. These templates are ready-to-use and simplify the integration process, making them an excellent choice if you prefer a no-code solution.

### Key features[​](#key-features-1 "Direct link to Key features")

- No Beam coding required.
- Quick and easy setup for simple use cases.
- Also suitable if you have minimal programming expertise.

Both approaches are fully compatible with Google Cloud and the ClickHouse ecosystem, offering flexibility depending on your technical expertise and project requirements.

- [Java runner](#1-java-runner)
  - [Key features](#key-features)- [Predefined templates](#2-predefined-templates)
    - [Key features](#key-features-1)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/google-dataflow/templates.md)

ClickHouse Supported

Google Dataflow templates provide a convenient way to execute prebuilt, ready-to-use data pipelines without the need to write custom code. These templates are designed to simplify common data processing tasks and are built using [Apache Beam](https://beam.apache.org/), leveraging connectors like `ClickHouseIO` for seamless integration with ClickHouse databases. By running these templates on Google Dataflow, you can achieve highly scalable, distributed data processing with minimal effort.

## Why use Dataflow templates?[​](#why-use-dataflow-templates "Direct link to Why use Dataflow templates?")

- **Ease of Use**: Templates eliminate the need for coding by offering preconfigured pipelines tailored to specific use cases.
- **Scalability**: Dataflow ensures your pipeline scales efficiently, handling large volumes of data with distributed processing.
- **Cost Efficiency**: Pay only for the resources you consume, with the ability to optimize pipeline execution costs.

## How to run Dataflow templates[​](#how-to-run-dataflow-templates "Direct link to How to run Dataflow templates")

As of today, the ClickHouse official template is available via the Google Cloud Console, CLI or Dataflow REST API.
For detailed step-by-step instructions, refer to the [Google Dataflow Run Pipeline From a Template Guide](https://cloud.google.com/dataflow/docs/templates/provided-templates).

## List of ClickHouse Templates[​](#list-of-clickhouse-templates "Direct link to List of ClickHouse Templates")

- [BigQuery To ClickHouse](/docs/integrations/google-dataflow/templates/bigquery-to-clickhouse)
- [GCS To ClickHouse](https://github.com/ClickHouse/DataflowTemplates/issues/3) (coming soon!)
- [Pub Sub To ClickHouse](https://github.com/ClickHouse/DataflowTemplates/issues/4) (coming soon!)

- [Why use Dataflow templates?](#why-use-dataflow-templates)- [How to run Dataflow templates](#how-to-run-dataflow-templates)- [List of ClickHouse Templates](#list-of-clickhouse-templates)

Was this page helpful?
title_wip

### Monitor the job[​](#monitor-the-job "Direct link to Monitor the job")

Navigate to the [Dataflow Jobs tab](https://console.cloud.google.com/dataflow/jobs) in your Google Cloud Console to
monitor the status of the job. You'll find the job details, including progress and any errors:

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### Memory limit (total) exceeded error (code 241)[​](#code-241-dbexception-memory-limit-total-exceeded "Direct link to Memory limit (total) exceeded error (code 241)")

This error occurs when ClickHouse runs out of memory while processing large batches of data. To resolve this issue:

- Increase the instance resources: Upgrade your ClickHouse server to a larger instance with more memory to handle the data processing load.
- Decrease the batch size: Adjust the batch size in your Dataflow job configuration to send smaller chunks of data to ClickHouse, reducing memory consumption per batch. These changes can help balance resource usage during data ingestion.

## Template source code[​](#template-source-code "Direct link to Template source code")

The template's source code is available in ClickHouse's [DataflowTemplates](https://github.com/ClickHouse/DataflowTemplates) fork.

- [Pipeline requirements](#pipeline-requirements)- [Template parameters](#template-parameters)- [Source and target tables schema](#source-and-target-tables-schema)- [Data type mapping](#data-types-mapping)- [Running the Template](#running-the-template)
          - [Install & Configure `gcloud` CLI](#install--configure-gcloud-cli)- [Run command](#run-command)- [Command breakdown](#command-breakdown)- [Expected response](#expected-response)- [Monitor the job](#monitor-the-job)- [Troubleshooting](#troubleshooting)
            - [Memory limit (total) exceeded error (code 241)](#code-241-dbexception-memory-limit-total-exceeded)- [Template source code](#template-source-code)

Was this page helpful?
Java clients overview

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/language-clients/java/index.md)

- [Client 0.8+](/docs/integrations/language-clients/java/client)
- [JDBC 0.8+](/docs/integrations/language-clients/java/jdbc)
- [R2DBC Driver](/docs/integrations/java/r2dbc)

## ClickHouse client[​](#clickhouse-client "Direct link to ClickHouse client")

Java client is a library implementing own API that abstracts details of network communications with ClickHouse server. Currently HTTP Interface is supported only. The library provide utilities to work with different ClickHouse formats and other related functions.

Java Client was developed far back in 2015. Its codebase became very hard to maintain, API is confusing, it is hard to optimize it further. So we have refactored it in 2024 into a new component `client-v2`. It has clear API, lighter codebase and more performance improvements, better ClickHouse formats support (RowBinary & Native mainly). JDBC will use this client in near feature.

### Supported data types[​](#supported-data-types "Direct link to Supported data types")
Java clients overview

[ClickHouse Data Types](/docs/sql-reference/data-types)

Note

- AggregatedFunction - ⚠️ doesn't support `SELECT * FROM table ...`
- Decimal - `SET output_format_decimal_trailing_zeros=1` in 21.9+ for consistency
- Enum - can be treated as both string and integer
- UInt64 - mapped to `long` in client-v1

### Features[​](#features "Direct link to Features")

Table of features of the clients:
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/language-clients/java/r2dbc.md)

## R2DBC driver[​](#r2dbc-driver "Direct link to R2DBC driver")

[R2DBC](https://r2dbc.io/) wrapper of async Java client for ClickHouse.

### Environment requirements[​](#environment-requirements "Direct link to Environment requirements")

- [OpenJDK](https://openjdk.java.net) version >= 8

### Setup[​](#setup "Direct link to Setup")

```
<dependency>
    <groupId>com.clickhouse</groupId>
    <!-- change to clickhouse-r2dbc_0.9.1 for SPI 0.9.1.RELEASE -->
    <artifactId>clickhouse-r2dbc</artifactId>
    <version>0.7.1</version>
    <!-- use uber jar with all dependencies included, change classifier to http or grpc for smaller jar -->
    <classifier>all</classifier>
    <exclusions>
        <exclusion>
            <groupId>*</groupId>
            <artifactId>*</artifactId>
        </exclusion>
    </exclusions>
</dependency>
```

### Connect to ClickHouse[​](#connect-to-clickhouse "Direct link to Connect to ClickHouse")

```
ConnectionFactory connectionFactory = ConnectionFactories
    .get("r2dbc:clickhouse:http://{username}:{password}@{host}:{port}/{database}");

    Mono.from(connectionFactory.create())
        .flatMapMany(connection -> connection
```

### Query[​](#query "Direct link to Query")

```
connection
    .createStatement("select domain, path,  toDate(cdate) as d, count(1) as count from clickdb.clicks where domain = :domain group by domain, path, d")
    .bind("domain", domain)
    .execute()
    .flatMap(result -> result
    .map((row, rowMetadata) -> String.format("%s%s[%s]:%d", row.get("domain", String.class),
        row.get("path", String.class),
        row.get("d", LocalDate.class),
        row.get("count", Long.class)))
    )
    .doOnNext(System.out::println)
    .subscribe();
```

### Insert[​](#insert "Direct link to Insert")

```
connection
    .createStatement("insert into clickdb.clicks values (:domain, :path, :cdate, :count)")
    .bind("domain", click.getDomain())
    .bind("path", click.getPath())
    .bind("cdate", LocalDateTime.now())
    .bind("count", 1)
    .execute();
```

- [R2DBC driver](#r2dbc-driver)
  - [Environment requirements](#environment-requirements)- [Setup](#setup)- [Connect to ClickHouse](#connect-to-clickhouse)- [Query](#query)- [Insert](#insert)

Was this page helpful?
ClickHouse JS

For Parquet, the main use case for selects likely will be writing the resulting stream into a file. See [the example](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/select_parquet_as_file.ts) in the client repository.

`JSONEachRowWithProgress` is an output-only format that supports progress reporting in the stream. See [this example](https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/select_json_each_row_with_progress.ts) for more details.

The entire list of ClickHouse input and output formats is available
[here](/docs/interfaces/formats).

## Supported ClickHouse data types[​](#supported-clickhouse-data-types "Direct link to Supported ClickHouse data types")

Note

The related JS type is relevant for any `JSON*` formats except the ones that represent everything as a string (e.g. `JSONStringEachRow`)
title_wip

```
%%sql
SELECT
    pickup_date,
    pickup_ntaname,
    SUM(1) AS number_of_trips
FROM trips
GROUP BY pickup_date, pickup_ntaname
ORDER BY pickup_date ASC
limit 5;
```

- clickhouse://default:\*\*\*@localhost:8123/default
  Done.

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| pickup\_date pickup\_ntaname number\_of\_trips|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 2015-07-01 Bushwick North 2|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 2015-07-01 Brighton Beach 1|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 2015-07-01 Briarwood-Jamaica Hills 3|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | 2015-07-01 Williamsburg 1|  |  |  | | --- | --- | --- | | 2015-07-01 Queensbridge-Ravenswood-Long Island City 9 | | | | | | | | | | | | | | | | | |

```
# %sql DESCRIBE trips;
```

```
# %sql SELECT DISTINCT(trip_distance) FROM trips limit 50;
```

```
%%sql --save short-trips --no-execute
SELECT *
FROM trips
WHERE trip_distance < 6.3
```

- clickhouse://default:\*\*\*@localhost:8123/default
  Skipping execution...

```
%sqlplot histogram --table short-trips --column trip_distance --bins 10 --with short-trips
```

```
<AxesSubplot: title={'center': "'trip_distance' from 'short-trips'"}, xlabel='trip_distance', ylabel='Count'>
```

```
ax = %sqlplot histogram --table short-trips --column trip_distance --bins 50 --with short-trips
ax.grid()
ax.set_title("Trip distance from trips < 6.3")
_ = ax.set_xlabel("Trip distance")
```

Was this page helpful?
Integrating Kafka with ClickHouse

### Other options[​](#other-options "Direct link to Other options")

- [**Confluent Cloud**](/docs/integrations/kafka/cloud/confluent) - Confluent Platform provides an option to upload and [run ClickHouse Connector Sink on Confluent Cloud](/docs/integrations/kafka/cloud/confluent/custom-connector) or use [HTTP Sink Connector for Confluent Platform](/docs/integrations/kafka/cloud/confluent/http) that integrates Apache Kafka with an API via HTTP or HTTPS.
- [**Vector**](/docs/integrations/kafka/kafka-vector) - Vector is a vendor-agnostic data pipeline. With the ability to read from Kafka, and send events to ClickHouse, this represents a robust integration option.
- [**JDBC Connect Sink**](/docs/integrations/kafka/kafka-connect-jdbc) - The Kafka Connect JDBC Sink connector allows you to export data from Kafka topics to any relational database with a JDBC driver.
- **Custom code** - Custom code using Kafka and ClickHouse [client libraries](/docs/integrations/language-clients) may be appropriate in cases where custom processing of events is required.

- [Available options](#available-options)
  - [ClickPipes for Kafka](#clickpipes-for-kafka)- [Kafka Connect Sink](#kafka-connect-sink)- [Kafka table engine](#kafka-table-engine)- [Choosing an option](#choosing-an-option)- [Other options](#other-options)

Was this page helpful?
ClickHouse Kafka Connect Sink

### Target tables[​](#target-tables "Direct link to Target tables")

ClickHouse Connect Sink reads messages from Kafka topics and writes them to appropriate tables. ClickHouse Connect Sink writes data into existing tables. Please, make sure a target table with an appropriate schema was created in ClickHouse before starting to insert data into it.

Each topic requires a dedicated target table in ClickHouse. The target table name must match the source topic name.

### Pre-processing[​](#pre-processing "Direct link to Pre-processing")

If you need to transform outbound messages before they're sent to ClickHouse Kafka Connect
Sink, use [Kafka Connect Transformations](https://docs.confluent.io/platform/current/connect/transforms/overview.html).

### Supported data types[​](#supported-data-types "Direct link to Supported data types")

**With a schema declared:**
ClickHouse Kafka Connect Sink

Likely the fields in your data don't match the fields in the table - this is especially common with CDC (and the Debezium format).
One common solution is to add the flatten transformation to your connector configuration:

```
transforms=flatten
transforms.flatten.type=org.apache.kafka.connect.transforms.Flatten$Value
transforms.flatten.delimiter=_
```

This will transform your data from a nested JSON to a flattened JSON (using `_` as a delimiter). Fields in the table would then follow the "field1\_field2\_field3" format (i.e. "before\_id", "after\_id", etc.).

#### "I want to use my Kafka keys in ClickHouse"[​](#i-want-to-use-my-kafka-keys-in-clickhouse "Direct link to \"I want to use my Kafka keys in ClickHouse\"")

Kafka keys aren't stored in the value field by default, but you can use the `KeyToValue` transformation to move the key to the value field (under a new `_key` field name):

```
transforms=keyToValue
transforms.keyToValue.type=com.clickhouse.kafka.connect.transforms.KeyToValue
transforms.keyToValue.field=_key
```

- [License](#license)- [Requirements for the environment](#requirements-for-the-environment)- [Version compatibility matrix](#version-compatibility-matrix)- [Main features](#main-features)- [Installation instructions](#installation-instructions)- [Configuration options](#configuration-options)- [Target tables](#target-tables)- [Pre-processing](#pre-processing)- [Supported data types](#supported-data-types)- [Configuration recipes](#configuration-recipes)- [Internal buffering](#internal-buffering)- [Logging](#logging)- [Monitoring](#monitoring)- [Limitations](#limitations)- [Performance tuning and throughput optimization](#tuning-performance)- [Troubleshooting](#troubleshooting)

Was this page helpful?