ClickHouse/clickhouse-docs-embeddings huggingface dataset:
Advanced installation methods

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/getting-started/install/advanced.md)

## Compile from source[​](#compile-from-source "Direct link to Compile from source")

To manually compile ClickHouse, follow the instructions for [Linux](/docs/development/build) or [macOS](/docs/development/build-osx).

You can compile packages and install them or use programs without installing packages.

```
Client: <build_directory>/programs/clickhouse-client
Server: <build_directory>/programs/clickhouse-server
```

You'll need to create data and metadata folders manually and `chown` them for the desired user. Their paths can be changed in server config (src/programs/server/config.xml), by default they're:

```
/var/lib/clickhouse/data/default/
/var/lib/clickhouse/metadata/default/
```

On Gentoo, you can just use `emerge clickhouse` to install ClickHouse from sources.

## Install a CI-generated Binary[​](#install-a-ci-generated-binary "Direct link to Install a CI-generated Binary")

ClickHouse's continuous integration (CI) infrastructure produces specialized builds for each commit in the [ClickHouse
repository](https://github.com/clickhouse/clickhouse/), e.g. [sanitized](https://github.com/google/sanitizers) builds, unoptimized (Debug)
builds, cross-compiled builds etc. While such builds are normally only useful during development, they can in certain situations also be
interesting for you.

Note

Since ClickHouse's CI is evolving over time, the exact steps to download CI-generated builds may vary.
Also, CI may delete old build artifacts, making them unavailable for download.

For example, to download an aarch64 binary for ClickHouse v23.4, follow these steps:

- Find the GitHub pull request for release v23.4: [Release pull request for branch 23.4](https://github.com/ClickHouse/ClickHouse/pull/49238)
- Click "Commits", then click on a commit similar to "Update autogenerated version to 23.4.2.1 and contributors" for the particular version you'd like to install.
- Click the green check / yellow dot / red cross to open the list of CI checks.
- Click "Details" next to "Builds" in the list; it will open a page similar to [this page](https://s3.amazonaws.com/clickhouse-test-reports/46793/b460eb70bf29b19eadd19a1f959b15d186705394/clickhouse_build_check/report.html).
- Find the rows with compiler = "clang-\*-aarch64" — there are multiple rows.
- Download the artifacts for these builds.

- [Compile from source](#compile-from-source)- [Install a CI-generated Binary](#install-a-ci-generated-binary)

Was this page helpful?
Install ClickHouse via script using curl

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/getting-started/install/quick-install-curl.md)

If you don't need to install ClickHouse for production, the quickest way to get
set up is to run an install script using curl. The script will determine a suitable
binary for your OS.

Install ClickHouse using curl

## Install ClickHouse using curl[​](#install-clickhouse-using-curl "Direct link to Install ClickHouse using curl")

Run the following comand to download a single binary for your operating system.

```
curl https://clickhouse.com/ | sh
```

Note

For Mac users: If you're getting errors that the developer of the binary can't be verified, please see [here](/docs/knowledgebase/fix-developer-verification-error-in-macos).

Start clickhouse-local

## Start clickhouse-local[​](#start-clickhouse-local "Direct link to Start clickhouse-local")

`clickhouse-local` allows you to process local and remote files using ClickHouse's
powerful SQL syntax and without the need for configuration. Table data is stored
in a temporary location, meaning that after a restart of `clickhouse-local`
previously created tables are no longer available.

Run the following command to start [clickhouse-local](/docs/operations/utilities/clickhouse-local):

```
./clickhouse
```

Start clickhouse-server

## Start clickhouse-server[​](#start-clickhouse-server "Direct link to Start clickhouse-server")

Should you wish to persist data, you'll want to run `clickhouse-server`. You can
start the ClickHouse server using the following command:

```
./clickhouse server
```

Start clickhouse-client

## Start clickhouse-client[​](#start-clickhouse-client "Direct link to Start clickhouse-client")

With the server up and running, open a new terminal window and run the following command
to launch `clickhouse-client`:

```
./clickhouse client
```

You will see something like this:

```
./clickhouse client
ClickHouse client version 24.5.1.117 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 24.5.1.

local-host :)
```

Table data is stored in the current directory and will still be available after a restart
of the ClickHouse server. If necessary, you can pass
`-C config.xml` as an additional command line argument to `./clickhouse server`
and provide further configuration in a configuration
file. All available configuration settings are documented [here](/docs/operations/server-configuration-parameters/settings) and in the
[example configuration file
template](https://github.com/ClickHouse/ClickHouse/blob/master/programs/server/config.xml).

You're now ready to start sending SQL commands to ClickHouse!

Tip

The [Quick Start](/docs/get-started/quick-start) walks you through the steps for creating tables and inserting data.

- [Install ClickHouse using curl](#install-clickhouse-using-curl)- [Start clickhouse-local](#start-clickhouse-local)- [Start clickhouse-server](#start-clickhouse-server)- [Start clickhouse-client](#start-clickhouse-client)

Was this page helpful?
Integrations

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/index.mdx)

Connect ClickHouse with your favorite tools and services to build powerful data pipelines and analytics workflows.

ClickHouse integrations are organized by their support level:

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Tier Description|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | Core integrations Built or maintained by ClickHouse, they're supported by ClickHouse and live in the ClickHouse GitHub organization|  |  |  |  | | --- | --- | --- | --- | | Partner integrations Built or maintained, and supported by, third-party software vendors|  |  | | --- | --- | | Community integrations Built or maintained and supported by community members. No direct support is available besides the public GitHub repositories and community Slack channels | | | | | | | |

Loading integrations...

Notice

Third-party logos and trademarks belong to their respective owners and are shown only to indicate available integrations.
No endorsement is implied.
title_wip

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
title_wip

| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `TableSchema.TypeName.DATE` `Schema.TypeName#DATETIME` ✅ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `TableSchema.TypeName.DATETIME` `Schema.TypeName#DATETIME` ✅ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `TableSchema.TypeName.ARRAY` `Schema.TypeName#ARRAY` ✅ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `TableSchema.TypeName.ENUM8` `Schema.TypeName#STRING` ✅ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `TableSchema.TypeName.ENUM16` `Schema.TypeName#STRING` ✅ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `TableSchema.TypeName.BOOL` `Schema.TypeName#BOOLEAN` ✅ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `TableSchema.TypeName.TUPLE` `Schema.TypeName#ROW` ✅ |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `TableSchema.TypeName.FIXEDSTRING` `FixedBytes` ✅ `FixedBytes` is a `LogicalType` representing a fixed-length   byte array located at   `org.apache.beam.sdk.schemas.logicaltypes`| `Schema.TypeName#DECIMAL` ❌ |  |  |  |  | | --- | --- | --- | --- | | `Schema.TypeName#MAP` ❌  | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/apache-spark/index.md)

ClickHouse Supported

[Apache Spark](https://spark.apache.org/) is a multi-language engine for executing data engineering, data
science, and machine learning on single-node machines or clusters.

There are two main ways to connect Apache Spark and ClickHouse:

1. [Spark Connector](/docs/integrations/apache-spark/spark-native-connector) - The Spark connector implements the `DataSourceV2` and has its own Catalog
   management. As of today, this is the recommended way to integrate ClickHouse and Spark.
2. [Spark JDBC](/docs/integrations/apache-spark/spark-jdbc) - Integrate Spark and ClickHouse
   using a [JDBC data source](https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html).

  

Both solutions have been successfully tested and are fully compatible with various APIs, including Java, Scala, PySpark, and Spark SQL.

### Spark Runtime Environments[​](#spark-runtime-environment "Direct link to Spark Runtime Environments")

#### Standard Spark runtimes[​](#standard-spark-runtime "Direct link to Standard Spark runtimes")

The Spark Connector works out of the box on environments that closely follow the upstream Apache Spark runtime, such as Amazon EMR or a Kubernetes-based Spark deployments.

#### Managed Spark platforms[​](#managed-spark-platforms "Direct link to Managed Spark platforms")

Platforms such as [AWS Glue](/docs/integrations/glue) and [Databricks](/docs/integrations/data-ingestion/apache-spark/databricks) introduce additional abstractions and environment-specific behavior.
While the core integration remains the same, they may require dedicated configuration and setup steps. See the respective documentation pages for details.

- [Spark Runtime Environments](#spark-runtime-environment)

Was this page helpful?
Spark connector

## Supported data types[​](#supported-data-types "Direct link to Supported data types")

This section outlines the mapping of data types between Spark and ClickHouse. The tables below provide quick references
for converting data types when reading from ClickHouse into Spark and when inserting data from Spark into ClickHouse.

### Reading data from ClickHouse into Spark[​](#reading-data-from-clickhouse-into-spark "Direct link to Reading data from ClickHouse into Spark")
Spark connector

also converted|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `MapType` `Map` ✅ No Keys are limited to `StringType`| `StructType` `Tuple` ✅ No Converted to named Tuple with field names.|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `VariantType` `JSON` or `Variant` ✅ No Requires Spark 4.0+ and ClickHouse 25.3+. Defaults to `JSON` type. Use `clickhouse.column.<name>.variant_types` property to specify `Variant` with multiple types.| `Object` ❌ |  |  |  |  |  | | --- | --- | --- | --- | --- | | `Nested` ❌ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
Spark connector

## Contributing and support[​](#contributing-and-support "Direct link to Contributing and support")

If you'd like to contribute to the project or report any issues, we welcome your input!
Visit our [GitHub repository](https://github.com/ClickHouse/spark-clickhouse-connector) to open an issue, suggest
improvements, or submit a pull request.
Contributions are welcome! Please check the contribution guidelines in the repository before starting.
Thank you for helping improve our ClickHouse Spark connector!

- [Choosing Between Catalog API and TableProvider API](#choosing-between-apis)
  - [Catalog API vs TableProvider API](#catalog-vs-tableprovider-comparison)- [Requirements](#requirements)- [Compatibility matrix](#compatibility-matrix)- [Installation & setup](#installation--setup)
        - [Import as a Dependency](#import-as-a-dependency)- [Download the library](#download-the-library)- [Register the catalog (required)](#register-the-catalog-required)- [Using the TableProvider API (Format-based Access)](#using-the-tableprovider-api)
            - [Format-based Read Example](#format-based-read)- [Format-based Write Example](#format-based-write)- [TableProvider Features](#tableprovider-features)- [TableProvider Connection Options](#tableprovider-connection-options)- [Configuring ClickHouse Options](#configuring-clickhouse-options)
              - [Using TableProvider API](#using-tableprovider-api-options)- [Using Catalog API](#using-catalog-api-options)- [ClickHouse Cloud settings](#clickhouse-cloud-settings)- [Read data](#read-data)- [Write data](#write-data)- [DDL operations](#ddl-operations)- [Working with VariantType](#working-with-varianttype)
                        - [ClickHouse Type Mapping](#clickhouse-type-mapping)- [Reading VariantType Data](#reading-varianttype-data)- [Writing VariantType Data](#writing-varianttype-data)- [Creating VariantType Tables with Spark SQL](#creating-varianttype-tables-spark-sql)- [Configuring Variant Types](#configuring-variant-types)- [Supported Variant Types](#supported-variant-types)- [Read Format Configuration](#read-format-configuration)- [Write Format Support](#write-format-support)- [Best Practices](#varianttype-best-practices)- [Example: Complete Workflow](#varianttype-example-workflow)- [Configurations](#configurations)- [Supported data types](#supported-data-types)
                            - [Reading data from ClickHouse into Spark](#reading-data-from-clickhouse-into-spark)- [Inserting data from Spark into ClickHouse](#inserting-data-from-spark-into-clickhouse)- [Contributing and support](#contributing-and-support)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/tools/data-integration/splunk/index.md)

Partner Integration

[Splunk](https://www.splunk.com/) is a data analytics and monitoring platform.

This add-on allows you to store the [ClickHouse Cloud audit logs](/docs/cloud/security/audit-logging) into Splunk. It uses [ClickHouse Cloud API](/docs/cloud/manage/api/api-overview) to download the audit logs.

This add-on contains only a modular input, no additional UI are provided with this add-on.

# Installation

## For Splunk Enterprise[​](#for-splunk-enterprise "Direct link to For Splunk Enterprise")

Download the ClickHouse Cloud Audit Add-on for Splunk from [Splunkbase](https://splunkbase.splunk.com/app/7709).

In Splunk Enterprise, navigate to Apps -> Manage. Then click on Install app from file.

Select the archived file downloaded from Splunkbase and click on Upload.

If everything goes fine, you should now see the ClickHouse Audit logs application installed. If not, consult the Splunkd logs for any errors.

# Modular input configuration

To configure the modular input, you'll first need information from your ClickHouse Cloud deployment:

- The organization ID
- An admin [API Key](/docs/cloud/manage/openapi)

## Getting information from ClickHouse Cloud[​](#getting-information-from-clickhouse-cloud "Direct link to Getting information from ClickHouse Cloud")

Log in to the [ClickHouse Cloud console](https://console.clickhouse.cloud/).

Navigate to your Organization -> Organization details. There you can copy the Organization ID.

Then, navigate to API Keys from the left-end menu.

Create an API Key, give a meaningful name and select `Admin` privileges. Click on Generate API Key.

Save the API Key and secret in a safe place.

## Configure data input in Splunk[​](#configure-data-input-in-splunk "Direct link to Configure data input in Splunk")

Back in Splunk, navigate to Settings -> Data inputs.

Select the ClickHouse Cloud Audit Logs data input.

Click "New" to configure a new instance of the data input.

Once you have entered all the information, click Next.

The input is configured, you can start browsing the audit logs.

# Usage

The modular input stores data in Splunk. To view the data, you can use the general search view in Splunk.

- [For Splunk Enterprise](#for-splunk-enterprise)- [Getting information from ClickHouse Cloud](#getting-information-from-clickhouse-cloud)- [Configure data input in Splunk](#configure-data-input-in-splunk)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/azure-data-factory/index.md)

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Page Description|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | [Overview](/docs/integrations/azure-data-factory/overview) Overview of the two approaches used for bringing Azure Data into ClickHouse|  |  |  |  | | --- | --- | --- | --- | | [Using ClickHouse's azureBlobStorage table function](/docs/integrations/azure-data-factory/table-function) Option 1 - an efficient and straightforward way to copy data from Azure Blob Storage or Azure Data Lake Storage into ClickHouse using the `azureBlobStorage` table function| [Using ClickHouse's HTTP interface](/docs/integrations/azure-data-factory/http-interface) Option 2 - instead of having ClickHouse pull the data from Azure, have Azure Data Factory push the data to ClickHouse using it's HTTP interface | | | | | | | |

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/azure-data-factory/overview.md)

ClickHouse Supported

Microsoft Azure offers a wide range of tools to store, transform, and analyze
data. However, in many scenarios, ClickHouse can provide significantly better
performance for low-latency querying and processing of huge datasets. In
addition, ClickHouse's columnar storage and compression can greatly reduce the
cost of querying large volumes of analytical data compared to general-purpose
Azure databases.

In this section of the docs, we will explore two ways to ingest data from Microsoft Azure
into ClickHouse:

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Method Description|  |  |  |  | | --- | --- | --- | --- | | [Using the `azureBlobStorage` Table Function](/docs/integrations/azure-data-factory/table-function) Involves using ClickHouse's [`azureBlobStorage` Table Function](https://clickhouse.com/docs/sql-reference/table-functions/azureBlobStorage) to transfer data directly from Azure Blob Storage.| [Using the ClickHouse HTTP interface](/docs/integrations/azure-data-factory/http-interface) Uses the [ClickHouse HTTP interface](https://clickhouse.com/docs/interfaces/http) as a data source within Azure Data Factory, allowing you to copy data or use it in data flow activities as part of your pipelines. | | | | | |

Was this page helpful?
AWS PrivateLink for ClickPipes

Once the grace period elapses, the endpoint and all associated resources are automatically deleted.

To prevent automatic removal, resolve the underlying issue before the grace period expires.
For example, accept a pending connection request in your AWS console,
or recreate the endpoint if it has entered a failed state.

- [Supported ClickPipes data sources](#supported-sources)- [Supported AWS PrivateLink endpoint types](#aws-privatelink-endpoint-types)
    - [VPC resource](#vpc-resource)- [MSK multi-VPC connectivity](#msk-multi-vpc)- [VPC endpoint service](#vpc-endpoint-service)- [Creating a ClickPipe with reverse private endpoint](#creating-clickpipe)- [Managing existing reverse private endpoints](#managing-existing-endpoints)- [Supported AWS regions](#aws-privatelink-regions)- [Limitations](#limitations)
            - [Automatic cleanup of inactive endpoints](#automatic-cleanup)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/clickpipes/kafka/index.md)

|  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Page Description|  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [Creating your first Kafka ClickPipe](/docs/integrations/clickpipes/kafka/create-your-first-kafka-clickpipe) Step-by-step guide to creating your first Kafka ClickPipe.|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | [Schema registries for Kafka ClickPipe](/docs/integrations/clickpipes/kafka/schema-registries) How to integrate for ClickPipes with a schema registry for schema management|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | [Reference](/docs/integrations/clickpipes/kafka/reference) Details supported formats, sources, delivery semantics, authentication and experimental features supported by Kafka ClickPipes|  |  |  |  | | --- | --- | --- | --- | | [Best practices](/docs/integrations/clickpipes/kafka/best-practices) Details best practices to follow when working with Kafka ClickPipes|  |  | | --- | --- | | [Kafka ClickPipes FAQ](/docs/integrations/clickpipes/kafka/faq) Frequently asked questions about ClickPipes for Kafka | | | | | | | | | | | |

Was this page helpful?