ClickHouse/clickhouse-docs-embeddings huggingface dataset:
Dataflow Java runner

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/google-dataflow/java-runner.md)

ClickHouse Supported

The Dataflow Java runner lets you execute custom Apache Beam pipelines on Google Cloud's Dataflow service. This approach provides maximum flexibility and is well-suited for advanced ETL workflows.

## How it works[​](#how-it-works "Direct link to How it works")

1. **Pipeline Implementation**
   To use the Java runner, you need to implement your Beam pipeline using the `ClickHouseIO` - our official Apache Beam connector. For code examples and instructions on how to use the `ClickHouseIO`, please visit [ClickHouse Apache Beam](/docs/integrations/apache-beam).
2. **Deployment**
   Once your pipeline is implemented and configured, you can deploy it to Dataflow using Google Cloud's deployment tools. Comprehensive deployment instructions are provided in the [Google Cloud Dataflow documentation - Java Pipeline](https://cloud.google.com/dataflow/docs/quickstarts/create-pipeline-java).

**Note**: This approach assumes familiarity with the Beam framework and coding expertise. If you prefer a no-code solution, consider using [ClickHouse's predefined templates](/docs/integrations/google-dataflow/templates).

- [How it works](#how-it-works)

Was this page helpful?
Database/SQL API

[Full Example](https://github.com/ClickHouse/clickhouse-go/blob/main/examples/std/open_telemetry.go)

## Compression[​](#compression "Direct link to Compression")

The standard API supports the same compression algorithms as native [ClickHouse API](/docs/integrations/language-clients/go/configuration#compression) i.e. `lz4` and `zstd` compression at a block level. In addition, gzip, deflate and br compression are supported for HTTP connections. If any of these are enabled, compression is performed on blocks during insertion and for query responses. Other requests e.g. pings or query requests, will remain uncompressed. This is consistent with `lz4` and `zstd` options.

If using the `OpenDB` method to establish a connection, a Compression configuration can be passed. This includes the ability to specify the compression level (see below). If connecting via `sql.Open` with DSN, utilize the parameter `compress`. This can either be a specific compression algorithm i.e. `gzip`, `deflate`, `br`, `zstd` or `lz4` or a boolean flag. If set to true, `lz4` will be used. The default is `none` i.e. compression disabled.

```
conn := clickhouse.OpenDB(&clickhouse.Options{
    Addr: []string{fmt.Sprintf("%s:%d", env.Host, env.HttpPort)},
    Auth: clickhouse.Auth{
        Database: env.Database,
        Username: env.Username,
        Password: env.Password,
    },
    Compression: &clickhouse.Compression{
        Method: clickhouse.CompressionBrotli,
        Level:  5,
    },
    Protocol: clickhouse.HTTP,
})
```

[Full Example](https://github.com/ClickHouse/clickhouse-go/blob/main/examples/std/compression.go#L27-L76)

```
conn, err := sql.Open("clickhouse", fmt.Sprintf("http://%s:%d?username=%s&password=%s&compress=gzip&compress_level=5", env.Host, env.HttpPort, env.Username, env.Password))
```

[Full Example](https://github.com/ClickHouse/clickhouse-go/blob/main/examples/std/compression.go#L78-L115)

The level of applied compression can be controlled by the DSN parameter compress\_level or the Level field of the Compression option. This defaults to 0 but is algorithm specific:

- `gzip` - `-2` (Best Speed) to `9` (Best Compression)
- `deflate` - `-2` (Best Speed) to `9` (Best Compression)
- `br` - `0` (Best Speed) to `11` (Best Compression)
- `zstd`, `lz4` - ignored

- [Connecting](#connecting)
  - [Connection settings](#connection-settings)- [Connecting over HTTP](#connecting-over-http)- [Sessions](#sessions)- [Execution](#execution)- [Batch insert](#batch-insert)- [Querying rows](#querying-rows)- [Async insert](#async-insert)- [Parameter binding](#parameter-binding)- [Using context](#using-context)- [Dynamic scanning](#dynamic-scanning)- [External tables](#external-tables)- [Open telemetry](#open-telemetry)- [Compression](#compression)

Was this page helpful?
JDBC driver

```
conn.setClientInfo(com.clickhouse.jdbc.ClientInfoProperties.APPLICATION_NAME.getKey(), "my-app-01");
```

Both ways will result in the following `http_user_agent` value in the query log:

```
my-app-01/1.0 jdbc-v2/0.9.7 clickhouse-java-v2/0.9.6 (Linux; jvm:17.0.17) Apache-HttpClient/5.4.4
```

**Note:** We recommend using `app_name/version` format for `client_name` property because it helps to identify the application in the query log.

### Operation Identification[​](#operation-identification "Direct link to Operation Identification")

JDBC driver generates `query_id` for each operation (Currently it is included in server exceptions).

To set `log_comment` for an operation use `com.clickhouse.jdbc.StatementImpl#getLocalSettings` method. This requires
`Statement` or `PreparedStatement` to be cast to `com.clickhouse.jdbc.StatementImpl` first.

```
StatementImpl stmt = (StatementImpl) conn.createStatement();
stmt.getLocalSettings().logComment("some-comment");
```

**Note:** this approach work for single threaded uses of statement because `localSettings` is shared between threads.

## Supported data types[​](#supported-data-types "Direct link to Supported data types")

JDBC driver supports the same data formats as the underlying [java client](/docs/integrations/java#supported-data-types).

### JDBC Type Mapping[​](#jdbc-type-mapping "Direct link to JDBC Type Mapping")

Following mapping applies to:

- `ResultSet#getObject(columnIndex)` - method will return object of the corresponding Java class. (`Int8` -> `java.lang.Byte`, `Int16` -> `java.lang.Short`, etc.)
- `ResultSetMetaData#getColumnType(columnIndex)` - method will return the corresponding JDBC type. (`Int8` -> `java.lang.Byte`, `Int16` -> `java.lang.Short`, etc.)

There are few ways to change the mapping:

- `ResultSet#getObject(columnIndex, class)` - method will try to convert value to `class` type. There are some conversion limitations. See each section for details.

**Numeric Types**
Connecting Middleware to ClickHouse

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/tools/data-integration/middleware/index.md)

Partner Integration

[Middleware](https://middleware.io/) is a cloud observability platform for monitoring infrastructure, logs, and application performance.

You can connect ClickHouse with Middleware to collect and visualize database telemetry as part of your broader monitoring workflows.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

- A running ClickHouse service (Cloud or self-managed)
- Access to your ClickHouse host, port, username, and password
- A Middleware account

## Connect ClickHouse in Middleware[​](#connect-clickhouse-in-middleware "Direct link to Connect ClickHouse in Middleware")

1. Sign in to your Middleware account.
2. Go to **Integrations** and search for **ClickHouse**.
3. Choose the ClickHouse integration and enter your connection details:
   - Host
   - Port
   - Database
   - Username
   - Password
4. Save the integration and run a connection test.

## Verify data collection[​](#verify-data-collection "Direct link to Verify data collection")

After setup, confirm that ClickHouse metrics and/or logs are visible in your Middleware dashboards.

If connection validation fails, verify that:

- ClickHouse accepts inbound connections from Middleware
- SSL/TLS settings match your ClickHouse endpoint
- Credentials and database permissions are correct

## Additional resources[​](#additional-resources "Direct link to Additional resources")

- [Middleware website](https://middleware.io/)

- [Prerequisites](#prerequisites)- [Connect ClickHouse in Middleware](#connect-clickhouse-in-middleware)- [Verify data collection](#verify-data-collection)- [Additional resources](#additional-resources)

Was this page helpful?
Data Integrations

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/tools/data-integration/index.md)

|  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Page Description|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | [Easypanel](/docs/integrations/easypanel) Easypanel allows you to deploy ClickHouse on your own server|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | [Middleware](/docs/integrations/middleware) Connect ClickHouse with Middleware for observability and monitoring workflows.|  |  |  |  | | --- | --- | --- | --- | | [Retool](/docs/integrations/retool) Quickly build web and mobile apps with rich user interfaces, automate complex tasks, and integrate AI—all powered by your data.|  |  | | --- | --- | | [Splunk](/docs/integrations/audit-splunk) Store ClickHouse Cloud audit logs into Splunk. | | | | | | | | | |

Was this page helpful?
Arrow

## Example usage[​](#example-usage "Direct link to Example usage")

### Inserting data[​](#inserting-data "Direct link to Inserting data")

You can insert Arrow data from a file into ClickHouse table using the following command:

```
$ cat filename.arrow | clickhouse-client --query="INSERT INTO some_table FORMAT Arrow"
```

### Selecting data[​](#selecting-data "Direct link to Selecting data")

You can select data from a ClickHouse table and save it into some file in the Arrow format using the following command:

```
$ clickhouse-client --query="SELECT * FROM {some_table} FORMAT Arrow" > {filename.arrow}
```

## Format settings[​](#format-settings "Direct link to Format settings")

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Setting Description Default|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `input_format_arrow_allow_missing_columns` Allow missing columns while reading Arrow input formats `1`| `input_format_arrow_case_insensitive_column_matching` Ignore case when matching Arrow columns with CH columns. `0`| `input_format_arrow_import_nested` Obsolete setting, does nothing. `0`| `input_format_arrow_skip_columns_with_unsupported_types_in_schema_inference` Skip columns with unsupported types while schema inference for format Arrow `0`| `output_format_arrow_compression_method` Compression method for Arrow output format. Supported codecs: lz4\_frame, zstd, none (uncompressed) `lz4_frame`| `output_format_arrow_fixed_string_as_fixed_byte_array` Use Arrow FIXED\_SIZE\_BINARY type instead of Binary for FixedString columns. `1`| `output_format_arrow_low_cardinality_as_dictionary` Enable output LowCardinality type as Dictionary Arrow type `0`| `output_format_arrow_string_as_string` Use Arrow String type instead of Binary for String columns `1`| `output_format_arrow_use_64_bit_indexes_for_dictionary` Always use 64 bit integers for dictionary indexes in Arrow format `0`| `output_format_arrow_use_signed_indexes_for_dictionary` Use signed integers for dictionary indexes in Arrow format `1` | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

- [Description](#description)- [Data types matching](#data-types-matching)- [Example usage](#example-usage)
      - [Inserting data](#inserting-data)- [Selecting data](#selecting-data)- [Format settings](#format-settings)

Was this page helpful?
JSONAsObject

[Edit this page](https://github.com/ClickHouse/ClickHouse/tree/master/docs/en/interfaces/formats/JSON/JSONAsObject.md)

## Description[​](#description "Direct link to Description")

In this format, a single JSON object is interpreted as a single [JSON](/docs/sql-reference/data-types/newjson) value. If the input has several JSON objects (comma separated), they are interpreted as separate rows. If the input data is enclosed in `[]`, it is interpreted as an array of JSONs.

This format can only be parsed for a table with a single field of type [JSON](/docs/sql-reference/data-types/newjson). The remaining columns must be set to [`DEFAULT`](/docs/sql-reference/statements/create/table#default) or [`MATERIALIZED`](/docs/sql-reference/statements/create/view#materialized-view).

## Example usage[​](#example-usage "Direct link to Example usage")

### Basic example[​](#basic-example "Direct link to Basic example")

```
CREATE TABLE json_as_object (json JSON) ENGINE = Memory;
INSERT INTO json_as_object (json) FORMAT JSONAsObject {"foo":{"bar":{"x":"y"},"baz":1}},{},{"any json stucture":1}
SELECT * FROM json_as_object FORMAT JSONEachRow;
```

```
{"json":{"foo":{"bar":{"x":"y"},"baz":"1"}}}
{"json":{}}
{"json":{"any json stucture":"1"}}
```

### An array of JSON objects[​](#an-array-of-json-objects "Direct link to An array of JSON objects")

```
CREATE TABLE json_square_brackets (field JSON) ENGINE = Memory;
INSERT INTO json_square_brackets FORMAT JSONAsObject [{"id": 1, "name": "name1"}, {"id": 2, "name": "name2"}];
SELECT * FROM json_square_brackets FORMAT JSONEachRow;
```

```
{"field":{"id":"1","name":"name1"}}
{"field":{"id":"2","name":"name2"}}
```

### Columns with default values[​](#columns-with-default-values "Direct link to Columns with default values")

```
CREATE TABLE json_as_object (json JSON, time DateTime MATERIALIZED now()) ENGINE = Memory;
INSERT INTO json_as_object (json) FORMAT JSONAsObject {"foo":{"bar":{"x":"y"},"baz":1}};
INSERT INTO json_as_object (json) FORMAT JSONAsObject {};
INSERT INTO json_as_object (json) FORMAT JSONAsObject {"any json stucture":1}
SELECT time, json FROM json_as_object FORMAT JSONEachRow
```

```
{"time":"2024-09-16 12:18:10","json":{}}
{"time":"2024-09-16 12:18:13","json":{"any json stucture":"1"}}
{"time":"2024-09-16 12:18:08","json":{"foo":{"bar":{"x":"y"},"baz":"1"}}}
```

## Format settings[​](#format-settings "Direct link to Format settings")

- [Description](#description)- [Example usage](#example-usage)
    - [Basic example](#basic-example)- [An array of JSON objects](#an-array-of-json-objects)- [Columns with default values](#columns-with-default-values)- [Format settings](#format-settings)

Was this page helpful?
JSONAsString

[Edit this page](https://github.com/ClickHouse/ClickHouse/tree/master/docs/en/interfaces/formats/JSON/JSONAsString.md)

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Input Output Alias|  |  |  | | --- | --- | --- | | ✔ ✗  | | | | | |

## Description[​](#description "Direct link to Description")

In this format, a single JSON object is interpreted as a single value.
If the input has several JSON objects (which are comma separated), they are interpreted as separate rows.
If the input data is enclosed in `[]`, it is interpreted as an array of JSON objects.

Note

This format can only be parsed for a table with a single field of type [String](/docs/sql-reference/data-types/string).
The remaining columns must be set to either [`DEFAULT`](/docs/sql-reference/statements/create/table#default) or [`MATERIALIZED`](/docs/sql-reference/statements/create/view#materialized-view),
or be omitted.

Once you serialize the entire JSON object to a String you can use the [JSON functions](/docs/sql-reference/functions/json-functions) to process it.

## Example usage[​](#example-usage "Direct link to Example usage")

### Basic example[​](#basic-example "Direct link to Basic example")

```
DROP TABLE IF EXISTS json_as_string;
CREATE TABLE json_as_string (json String) ENGINE = Memory;
INSERT INTO json_as_string (json) FORMAT JSONAsString {"foo":{"bar":{"x":"y"},"baz":1}},{},{"any json stucture":1}
SELECT * FROM json_as_string;
```

```
┌─json──────────────────────────────┐
│ {"foo":{"bar":{"x":"y"},"baz":1}} │
│ {}                                │
│ {"any json stucture":1}           │
└───────────────────────────────────┘
```

### An array of JSON objects[​](#an-array-of-json-objects "Direct link to An array of JSON objects")

```
CREATE TABLE json_square_brackets (field String) ENGINE = Memory;
INSERT INTO json_square_brackets FORMAT JSONAsString [{"id": 1, "name": "name1"}, {"id": 2, "name": "name2"}];

SELECT * FROM json_square_brackets;
```

```
┌─field──────────────────────┐
│ {"id": 1, "name": "name1"} │
│ {"id": 2, "name": "name2"} │
└────────────────────────────┘
```

## Format settings[​](#format-settings "Direct link to Format settings")

- [Description](#description)- [Example usage](#example-usage)
    - [Basic example](#basic-example)- [An array of JSON objects](#an-array-of-json-objects)- [Format settings](#format-settings)

Was this page helpful?
Values

[Edit this page](https://github.com/ClickHouse/ClickHouse/tree/master/docs/en/interfaces/formats/Values.md)

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Input Output Alias|  |  |  | | --- | --- | --- | | ✔ ✔  | | | | | |

## Description[​](#description "Direct link to Description")

The `Values` format prints every row in brackets.

- Rows are separated by commas without a comma after the last row.
- The values inside the brackets are also comma-separated.
- Numbers are output in a decimal format without quotes.
- Arrays are output in `[]`.
- Strings, dates, and dates with times are output in quotes.
- Escaping rules and parsing are similar to the [TabSeparated](/docs/interfaces/formats/TabSeparated) format.

During formatting, extra spaces aren't inserted, but during parsing, they are allowed and skipped (except for spaces inside array values, which are not allowed).
[`NULL`](/docs/sql-reference/syntax) is represented as `NULL`.

The minimum set of characters that you need to escape when passing data in the `Values` format:

- single quotes
- backslashes

This is the format that is used in `INSERT INTO t VALUES ...`, but you can also use it for formatting query results.

## Example usage[​](#example-usage "Direct link to Example usage")

## Format settings[​](#format-settings "Direct link to Format settings")

|  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Setting Description Default|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [`input_format_values_interpret_expressions`](/docs/operations/settings/formats#input_format_values_interpret_expressions) if the field could not be parsed by streaming parser, run SQL parser and try to interpret it as SQL expression. `true`| [`input_format_values_deduce_templates_of_expressions`](/docs/operations/settings/formats#input_format_values_deduce_templates_of_expressions) if the field could not be parsed by streaming parser, run SQL parser, deduce template of the SQL expression, try to parse all rows using template and then interpret expression for all rows. `true`| [`input_format_values_accurate_types_of_literals`](/docs/operations/settings/formats#input_format_values_accurate_types_of_literals) when parsing and interpreting expressions using template, check actual type of literal to avoid possible overflow and precision issues. `true` | | | | | | | | | | | |

- [Description](#description)- [Example usage](#example-usage)- [Format settings](#format-settings)

Was this page helpful?
AWS PrivateLink

### Connecting to a remote database[​](#connecting-to-a-remote-database "Direct link to Connecting to a remote database")

Let's say you're trying to use [MySQL](/docs/sql-reference/table-functions/mysql) or [PostgreSQL](/docs/sql-reference/table-functions/postgresql) table functions in ClickHouse Cloud and connect to your database hosted in an Amazon Web Services (AWS) VPC. AWS PrivateLink can't be used to enable this connection securely. PrivateLink is a one-way, unidirectional connection. It allows your internal network or Amazon VPC to connect securely to ClickHouse Cloud, but it doesn't allow ClickHouse Cloud to connect to your internal network.

According to the [AWS PrivateLink documentation](https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/aws-privatelink.html):

> Use AWS PrivateLink when you have a client/server set up where you want to allow one or more consumer VPCs unidirectional access to a specific service or set of instances in the service provider VPC. Only the clients in the consumer VPC can initiate a connection to the service in the service provider VPC.

To do this, configure your AWS Security Groups to allow connections from ClickHouse Cloud to your internal/private database service. Check the [default egress IP addresses for ClickHouse Cloud regions](/docs/manage/data-sources/cloud-endpoints-api), along with the [available static IP addresses](https://api.clickhouse.cloud/static-ips.json).

- [Important considerations](#considerations)- [Prerequisites for this process](#prerequisites)- [Steps](#steps)
      - [Obtain endpoint "Service name"](#obtain-endpoint-service-info)- [Create AWS endpoint](#create-aws-endpoint)- [Add "Endpoint ID" to ClickHouse service allow list](#add-endpoint-id-to-services-allow-list)- [Accessing an instance using PrivateLink](#accessing-an-instance-using-privatelink)- [Troubleshooting](#troubleshooting)
        - [Multiple PrivateLinks in one region](#multiple-privatelinks-in-one-region)- [Connection to private endpoint timed out](#connection-to-private-endpoint-timed-out)- [Private Hostname: Not found address of host](#private-hostname-not-found-address-of-host)- [Connection reset by peer](#connection-reset-by-peer)- [Checking endpoint filters](#checking-endpoint-filters)- [Connecting to a remote database](#connecting-to-a-remote-database)

Was this page helpful?