Spark connector

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/apache-spark/spark-native-connector.md)

ClickHouse Supported

This connector leverages ClickHouse-specific optimizations, such as advanced partitioning and predicate pushdown, to
improve query performance and data handling.
The connector is based on [ClickHouse's official JDBC connector](https://github.com/ClickHouse/clickhouse-java), and
manages its own catalog.

Before Spark 3.0, Spark lacked a built-in catalog concept, so users typically relied on external catalog systems such as
Hive Metastore or AWS Glue.
With these external solutions, users had to register their data source tables manually before accessing them in Spark.
However, since Spark 3.0 introduced the catalog concept, Spark can now automatically discover tables by registering
catalog plugins.

Spark's default catalog is `spark_catalog`, and tables are identified by `{catalog name}.{database}.{table}`. With the new
catalog feature, it is now possible to add and work with multiple catalogs in a single Spark application.

- [Choosing Between Catalog API and TableProvider API](#choosing-between-apis)
  - [Catalog API vs TableProvider API](#catalog-vs-tableprovider-comparison)- [Requirements](#requirements)- [Compatibility matrix](#compatibility-matrix)- [Installation & setup](#installation--setup)
        - [Import as a Dependency](#import-as-a-dependency)- [Download the library](#download-the-library)- [Register the catalog (required)](#register-the-catalog-required)- [Using the TableProvider API (Format-based Access)](#using-the-tableprovider-api)
            - [Format-based Read Example](#format-based-read)- [Format-based Write Example](#format-based-write)- [TableProvider Features](#tableprovider-features)- [TableProvider Connection Options](#tableprovider-connection-options)- [Configuring ClickHouse Options](#configuring-clickhouse-options)
              - [Using TableProvider API](#using-tableprovider-api-options)- [Using Catalog API](#using-catalog-api-options)- [ClickHouse Cloud settings](#clickhouse-cloud-settings)- [Read data](#read-data)- [Write data](#write-data)- [DDL operations](#ddl-operations)- [Working with VariantType](#working-with-varianttype)
                        - [ClickHouse Type Mapping](#clickhouse-type-mapping)- [Reading VariantType Data](#reading-varianttype-data)- [Writing VariantType Data](#writing-varianttype-data)- [Creating VariantType Tables with Spark SQL](#creating-varianttype-tables-spark-sql)- [Configuring Variant Types](#configuring-variant-types)- [Supported Variant Types](#supported-variant-types)- [Read Format Configuration](#read-format-configuration)- [Write Format Support](#write-format-support)- [Best Practices](#varianttype-best-practices)- [Example: Complete Workflow](#varianttype-example-workflow)- [Configurations](#configurations)- [Supported data types](#supported-data-types)
                            - [Reading data from ClickHouse into Spark](#reading-data-from-clickhouse-into-spark)- [Inserting data from Spark into ClickHouse](#inserting-data-from-spark-into-clickhouse)- [Contributing and support](#contributing-and-support)

## Choosing Between Catalog API and TableProvider API[​](#choosing-between-apis "Direct link to Choosing Between Catalog API and TableProvider API")

The ClickHouse Spark connector supports two access patterns: the **Catalog API** and the **TableProvider API** (format-based access). Understanding the differences helps you choose the right approach for your use case.

### Catalog API vs TableProvider API[​](#catalog-vs-tableprovider-comparison "Direct link to Catalog API vs TableProvider API")

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Feature Catalog API TableProvider API|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Configuration** Centralized via Spark configuration Per-operation via options|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Table Discovery** Automatic via catalog Manual table specification|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **DDL Operations** Full support (CREATE, DROP, ALTER) Limited (automatic table creation only)|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | **Spark SQL Integration** Native (`clickhouse.database.table`) Requires format specification|  |  |  | | --- | --- | --- | | **Use Case** Long-term, stable connections with centralized config Ad-hoc, dynamic, or temporary access | | | | | | | | | | | | | | | | | |

## Requirements[​](#requirements "Direct link to Requirements")

- Java 8 or 17 (Java 17+ required for Spark 4.0)
- Scala 2.12 or 2.13 (Spark 4.0 only supports Scala 2.13)
- Apache Spark 3.3, 3.4, 3.5, or 4.0

## Compatibility matrix[​](#compatibility-matrix "Direct link to Compatibility matrix")

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Version Compatible Spark Versions ClickHouse JDBC version|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | main Spark 3.3, 3.4, 3.5, 4.0 0.9.4|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0.10.0 Spark 3.3, 3.4, 3.5, 4.0 0.9.5|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0.9.0 Spark 3.3, 3.4, 3.5, 4.0 0.9.4|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0.8.1 Spark 3.3, 3.4, 3.5 0.6.3|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0.7.3 Spark 3.3, 3.4 0.4.6|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0.6.0 Spark 3.3 0.3.2-patch11|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0.5.0 Spark 3.2, 3.3 0.3.2-patch11|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0.4.0 Spark 3.2, 3.3 Not depend on|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0.3.0 Spark 3.2, 3.3 Not depend on|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | 0.2.1 Spark 3.2 Not depend on|  |  |  | | --- | --- | --- | | 0.1.2 Spark 3.2 Not depend on | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

## Installation & setup[​](#installation--setup "Direct link to Installation & setup")