ClickHouse/clickhouse-docs-embeddings huggingface dataset:
Schema changes propagation support

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/clickpipes/mysql/schema-changes.md)

ClickPipes for MySQL can detect schema changes in the source tables and, in some cases, automatically propagate the changes to the destination tables. The way each DDL operation is handled is documented below:

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Schema Change Type Behaviour|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | Adding a new column (`ALTER TABLE ADD COLUMN ...`) Propagated automatically. The new columns will be populated for all rows replicated after the schema change|  |  |  |  | | --- | --- | --- | --- | | Adding a new column with a default value (`ALTER TABLE ADD COLUMN ... DEFAULT ...`) Propagated automatically. The new columns will be populated for all rows replicated after the schema change, but existing rows won't show the default value without a full table refresh|  |  | | --- | --- | | Dropping an existing column (`ALTER TABLE DROP COLUMN ...`) Detected, but **not** propagated. The dropped columns will be populated with `NULL` for all rows replicated after the schema change | | | | | | | |

Note

Column additions during snapshot are currently not supported. The suggested workaround is to perform snapshots before or after planned schema changes, or, if the ClickPipe is already failing, to manually add a column of the appropriate type to the destination table.

### MySQL 5.x limitations[​](#mysql-5-limitations "Direct link to MySQL 5.x limitations")

MySQL versions older than [8.0.1](https://dev.mysql.com/blog-archive/more-metadata-is-written-into-binary-log/) do not include full column metadata in the binlog (`binlog_row_metadata=FULL`), so ClickPipes tracks columns by ordinal position. This means:

- **Adding a column at the end** (`ALTER TABLE ADD COLUMN ...`) is supported.
- **Any DDL that shifts column positions** will cause the pipe to raise an error, because ordinal positions can no longer be reliably mapped. This includes:
  - `ALTER TABLE DROP COLUMN ...`
  - `ALTER TABLE ADD COLUMN ... AFTER ...` / `FIRST`
  - `ALTER TABLE MODIFY COLUMN ... AFTER ...` / `FIRST`
  - `ALTER TABLE CHANGE COLUMN ... AFTER ...` / `FIRST`

If you hit this error, you will need to resync the pipe.

- [MySQL 5.x limitations](#mysql-5-limitations)

Was this page helpful?
ClickHouse Connect driver API

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/language-clients/python/driver-api.md)

Note

Passing keyword arguments is recommended for most api methods given the number of possible arguments, most of which are optional.

*Methods not documented here aren't considered part of the API, and may be removed or changed.*

## Client Initialization[​](#client-initialization "Direct link to Client Initialization")

The `clickhouse_connect.driver.client` class provides the primary interface between a Python application and the ClickHouse database server. Use the `clickhouse_connect.get_client` function to obtain a Client instance, which accepts the following arguments:

### Connection arguments[​](#connection-arguments "Direct link to Connection arguments")
clickhouse-client

```
clickhousectl cloud activity list --from-date 2024-01-01 --to-date 2024-12-31
clickhousectl cloud activity get <activity-id>
```

### JSON output[​](#json-output "Direct link to JSON output")

Use the `--json` flag to print JSON-formatted responses.

```
clickhousectl cloud --json service list
clickhousectl cloud --json service get <service-id>
```

## Skills[​](#skills "Direct link to Skills")

Install the official ClickHouse Agent Skills from [ClickHouse/agent-skills](https://github.com/ClickHouse/agent-skills).

```
# Default: interactive mode for humans, choose scope, then choose agents
clickhousectl skills

# Non-interactive: install into every supported project-local agent folder
clickhousectl skills --all

# Non-interactive: install only into detected agents
clickhousectl skills --detected-only

# Non-interactive: install into every supported global agent folder
clickhousectl skills --global --all

# Non-interactive: install into specific project-local agents
clickhousectl skills --agent claude --agent codex
```

### Non-interactive flags[​](#non-interactive-flags "Direct link to Non-interactive flags")

|  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Flag Description|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | `--agent <name>` Install Skills for a specific agent (can be repeated)|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | `--global` Use global scope; if omitted, project scope is used|  |  |  |  | | --- | --- | --- | --- | | `--all` Install Skills for all supported agents|  |  | | --- | --- | | `--detected-only` Install Skills for supported agents that were detected on the system | | | | | | | | | |

- [Installation](#installation)
  - [Quick install](#quick-install)- [Requirements](#requirements)- [Local](#local)
      - [Installing and managing ClickHouse versions](#installing-versions)- [Initializing a project](#initializing-project)- [Running queries](#running-queries)- [Creating and managing ClickHouse servers](#managing-servers)- [Authentication](#authentication)
        - [OAuth login (recommended)](#oauth-login)- [API key/secret](#api-key)- [Auth status and logout](#auth-status)- [Cloud](#cloud)
          - [Organizations](#organizations)- [Services](#services)- [Backups](#backups)- [Members](#members)- [Invitations](#invitations)- [Keys](#keys)- [Activity](#activity)- [JSON output](#json-output)- [Skills](#skills)
            - [Non-interactive flags](#non-interactive-flags)

Was this page helpful?
clickhousectl

```
clickhousectl cloud activity list --from-date 2024-01-01 --to-date 2024-12-31
clickhousectl cloud activity get <activity-id>
```

### JSON output[​](#json-output "Direct link to JSON output")

Use the `--json` flag to print JSON-formatted responses.

```
clickhousectl cloud --json service list
clickhousectl cloud --json service get <service-id>
```

## Skills[​](#skills "Direct link to Skills")

Install the official ClickHouse Agent Skills from [ClickHouse/agent-skills](https://github.com/ClickHouse/agent-skills).

```
# Default: interactive mode for humans, choose scope, then choose agents
clickhousectl skills

# Non-interactive: install into every supported project-local agent folder
clickhousectl skills --all

# Non-interactive: install only into detected agents
clickhousectl skills --detected-only

# Non-interactive: install into every supported global agent folder
clickhousectl skills --global --all

# Non-interactive: install into specific project-local agents
clickhousectl skills --agent claude --agent codex
```

### Non-interactive flags[​](#non-interactive-flags "Direct link to Non-interactive flags")

|  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Flag Description|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | `--agent <name>` Install Skills for a specific agent (can be repeated)|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | `--global` Use global scope; if omitted, project scope is used|  |  |  |  | | --- | --- | --- | --- | | `--all` Install Skills for all supported agents|  |  | | --- | --- | | `--detected-only` Install Skills for supported agents that were detected on the system | | | | | | | | | |

- [Installation](#installation)
  - [Quick install](#quick-install)- [Requirements](#requirements)- [Local](#local)
      - [Installing and managing ClickHouse versions](#installing-versions)- [Initializing a project](#initializing-project)- [Running queries](#running-queries)- [Creating and managing ClickHouse servers](#managing-servers)- [Authentication](#authentication)
        - [OAuth login (recommended)](#oauth-login)- [API key/secret](#api-key)- [Auth status and logout](#auth-status)- [Cloud](#cloud)
          - [Organizations](#organizations)- [Services](#services)- [Backups](#backups)- [Members](#members)- [Invitations](#invitations)- [Keys](#keys)- [Activity](#activity)- [JSON output](#json-output)- [Skills](#skills)
            - [Non-interactive flags](#non-interactive-flags)

Was this page helpful?
Formats for input and output data

[Edit this page](https://github.com/ClickHouse/ClickHouse/tree/master/docs/en/interfaces/formats.md)

ClickHouse supports most of the known text and binary data formats. This allows easy integration into almost any working
data pipeline to leverage the benefits of ClickHouse.

## Input formats[​](#input-formats "Direct link to Input formats")

Input formats are used for:

- Parsing data provided to `INSERT` statements
- Performing `SELECT` queries from file-backed tables such as `File`, `URL`, or `HDFS`
- Reading dictionaries

Choosing the right input format is crucial for efficient data ingestion in ClickHouse. With over 70 supported formats,
selecting the most performant option can significantly impact insert speed, CPU and memory usage, and overall system
efficiency. To help navigate these choices, we benchmarked ingestion performance across formats, revealing key takeaways:

- **The [Native](/docs/interfaces/formats/Native) format is the most efficient input format**, offering the best compression, lowest
  resource usage, and minimal server-side processing overhead.
- **Compression is essential** - LZ4 reduces data size with minimal CPU cost, while ZSTD offers higher compression at the
  expense of additional CPU usage.
- **Pre-sorting has a moderate impact**, as ClickHouse already sorts efficiently.
- **Batching significantly improves efficiency** - larger batches reduce insert overhead and improve throughput.

For a deep dive into the results and best practices,
read the full [benchmark analysis](https://www.clickhouse.com/blog/clickhouse-input-format-matchup-which-is-fastest-most-efficient).
For the full test results, explore the [FastFormats](https://fastformats.clickhouse.com/) online dashboard.

## Output formats[​](#output-formats "Direct link to Output formats")

Formats supported for output are used for:

- Arranging the results of a `SELECT` query
- Performing `INSERT` operations into file-backed tables

## Formats overview[​](#formats-overview "Direct link to Formats overview")

The supported formats are:
Formats for input and output data

You can control some format processing parameters with the ClickHouse settings. For more information read the [Settings](/docs/operations/settings/formats) section.

## Format schema[​](#formatschema "Direct link to Format schema")

The file name containing the format schema is set by the setting `format_schema`.
It's required to set this setting when it is used one of the formats `Cap'n Proto` and `Protobuf`.
The format schema is a combination of a file name and the name of a message type in this file, delimited by a colon,
e.g. `schemafile.proto:MessageType`.
If the file has the standard extension for the format (for example, `.proto` for `Protobuf`),
it can be omitted and in this case, the format schema looks like `schemafile:MessageType`.

If you input or output data via the [client](/docs/interfaces/client) in interactive mode, the file name specified in the format schema
can contain an absolute path or a path relative to the current directory on the client.
If you use the client in the [batch mode](/docs/interfaces/client#batch-mode), the path to the schema must be relative due to security reasons.

If you input or output data via the [HTTP interface](/docs/interfaces/http) the file name specified in the format schema
should be located in the directory specified in [format\_schema\_path](/docs/operations/server-configuration-parameters/settings#format_schema_path)
in the server configuration.

## Skipping errors[​](#skippingerrors "Direct link to Skipping errors")

Some formats such as `CSV`, `TabSeparated`, `TSKV`, `JSONEachRow`, `Template`, `CustomSeparated` and `Protobuf` can skip broken row if parsing error occurred and continue parsing from the beginning of next row. See [input\_format\_allow\_errors\_num](/docs/operations/settings/formats#input_format_allow_errors_num) and
[input\_format\_allow\_errors\_ratio](/docs/operations/settings/formats#input_format_allow_errors_ratio) settings.
Limitations:

- In case of parsing error `JSONEachRow` skips all data until the new line (or EOF), so rows must be delimited by `\n` to count errors correctly.
- `Template` and `CustomSeparated` use delimiter after the last column and delimiter between rows to find the beginning of next row, so skipping errors works only if at least one of them is not empty.

- [Input formats](#input-formats)- [Output formats](#output-formats)- [Formats overview](#formats-overview)- [Format schema](#formatschema)- [Skipping errors](#skippingerrors)

Was this page helpful?
Native clients & interfaces

[Edit this page](https://github.com/ClickHouse/ClickHouse/tree/master/docs/en/interfaces/native-clients-interfaces-index.md)

ClickHouse provides a number of different native clients and interfaces that allow you to connect to ClickHouse.

For more information see the pages below:

|  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Section Summary|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | [clickhousectl](/docs/interfaces/cli) The CLI for ClickHouse: local and cloud.|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | [ClickHouse Client](/docs/interfaces/client) Native command-line client supporting command-line options and configuration files.|  |  |  |  | | --- | --- | --- | --- | | [Drivers & Interfaces](/docs/interfaces/overview) A number of network interfaces, libraries and visual interfaces.|  |  | | --- | --- | | [SQL Console](/docs/integrations/sql-clients/sql-console) A fast and easy way to interact with your data in ClickHouse Cloud. | | | | | | | | | |

Was this page helpful?
[Knowledge Base](/docs/knowledgebase)

# Knowledge Base

RecentGrouped by tags

This covers an example of how you can use terraform to create/delete clusters using the API

Apr 9, 2026 · 5 min read

This article shows you how you can opt out of crash report collection on ClickHouse Cloud

Feb 4, 2026 · 2 min read

Learn about the different part types (Wide vs Compact) and storage formats (Full vs Packed) in ClickHouse, and how they affect performance.

Feb 3, 2026 · 6 min read

This knowledgebase article shows you how to find part counts by the type of part - wide or compact.

Jan 12, 2026 · 3 min read

This guide shows you how to collect and draw query traces with self-managed ClickHouse using either built-in methods or using Grafana. This is particularly useful when you're working with complex queries and need to understand the internal execution mechanics beyond what EXPLAIN provides.

Dec 30, 2025 · 4 min read

This article explains how to recover data when using replicated tables in atomic databases in ClickHouse and disks/storage on one of the replica is lost/currupted.

Nov 19, 2025 · 2 min read

A HAR (HTTP Archive) file captures the network activity in your browser. It can help our support team diagnose slow page loads, failed requests, or other network issues.

Nov 11, 2025 · 2 min read

Troubleshooting memory limit exceeded errors for a query

Jul 25, 2025 · 3 min read

Discusses the differences between OPTIMIZE FINAL and FINAL, and when to use and avoid them.

Jul 20, 2025 · 2 min read

Learn how to set up a custom DNS alias for your instance using a reverse proxy

May 16, 2025 · 2 min read
Archive

### 2021[​](#2021 "Direct link to 2021")

- [September 1 - Who is using ClickHouse?](/docs/knowledgebase/who-is-using-clickhouse)- [September 1 - Can I use ClickHouse as a time-series database?](/docs/knowledgebase/time-series)- [September 1 - Which ClickHouse version to use in production?](/docs/knowledgebase/production)- [September 1 - What is OLAP?](/docs/knowledgebase/olap)- [September 1 - Recommended Maximum Databases, Tables, Partitions, and Parts in ClickHouse](/docs/knowledgebase/maximum_number_of_tables_and_databases)- [September 1 - Why not use something like MapReduce?](/docs/knowledgebase/mapreduce)- [September 1 - Can I use ClickHouse as a key-value storage?](/docs/knowledgebase/key-value)- [September 1 - JSON Extract example](/docs/knowledgebase/json_extract_example)- [September 1 - How do I contribute code to ClickHouse?](/docs/knowledgebase/how-do-i-contribute-code-to-clickhouse)- [September 1 - What does “ClickHouse” mean?](/docs/knowledgebase/dbms-naming)- [September 1 - What is a columnar database?](/docs/knowledgebase/columnar-database)
Knowledge Base

This covers an example of how you can use terraform to create/delete clusters using the API

Apr 9, 2026 · 5 min read

ODBC authentication failed error when using the Power BI ClickHouse connector

Jul 10, 2024 · 2 min read

How to set up ClickHouse on Docker with ODBC to connect to a Microsoft SQL Server (MSSQL) database

May 29, 2024 · 4 min read

This article explains how to change the prompt in your Clickhouse client and clickhouse-local terminal window from :) to a prefix followed by :)

Nov 16, 2023 · 3 min read

Learn how to write partitioned data by year and month to an S3 bucket in ClickHouse, using a custom path structure for organizing the data.

Mar 24, 2023 · One min read

An example using Python and requests module to write and read to ClickHouse

Jul 10, 2022 · 2 min read

If you use Oracle as a source of ClickHouse external dictionaries via Oracle ODBC driver, you need to set the correct value for the `NLS\_LANG` environment variable in `/etc/default/clickhouse`.

Apr 22, 2022 · One min read
ClickHouse Cloud billing compliance

### How to pay using a credit card[​](#add-credit-card "Direct link to How to pay using a credit card")

Go to the Billing section in the ClickHouse Cloud UI and click the 'Add Credit Card'
button (shown below) to complete the setup. If you have any questions, [contact support](/docs/about-us/support) for help.

## How to pay via marketplaces[​](#marketplace-payg "Direct link to How to pay via marketplaces")

If you want to pay through one of our supported marketplaces (AWS, GCP, or Azure),
you can follow the steps [here](/docs/cloud/marketplace/marketplace-billing) for help.
For any questions related specifically to cloud marketplace billing, contact the cloud service provider directly.

Helpful links for resolving issues with marketplace billing:

- [AWS Billing FAQs](https://aws.amazon.com/aws-cost-management/aws-billing/faqs/)
- [GCP Billing FAQs](https://cloud.google.com/compute/docs/billing-questions)
- [Azure Billing FAQs](https://learn.microsoft.com/en-us/azure/cost-management-billing/cost-management-billing-faq)

- [Billing compliance](#billing-compliance)
  - [Pay-as-you-go billing with a credit card](#direct-payg)- [Cloud marketplace pay-as-you-go billing](#cloud-marketplace-payg)- [Committed contract billing](#committed-spend-contract)- [How to pay using a credit card](#add-credit-card)- [How to pay via marketplaces](#marketplace-payg)

Was this page helpful?
Managing payment methods

After completing this flow, the selected organization’s billing will be linked to the new Marketplace subscription and the billing page in the ClickHouse Cloud console will reflect the update.

## Support[​](#support "Direct link to Support")

If you run into any issues, please do not hesitate to [contact our support team](https://clickhouse.com/support/program).

- [Prerequisites](#prerequisites)- [Add or update a credit card payment method](#add-update-cc-payment-method)
    - [Steps to add or update your credit card](#steps-add-update)- [Configure an organization's billing to an existing marketplace subscription](#configure-billing-to-existing-mp-sub)
      - [Steps to switch an organization from credit card billing to a marketplace subscription that's already active on one of your other organizations](#steps-switch-org-already-active)- [Steps to change an organization's current marketplace subscription to one that's used by a different organization](#steps-switch-org-different-org)- [Adding a backup credit card to a marketplace organization](#add-backup-cc-to-marketplace-org)
        - [Steps to add a backup credit card](#steps-add-backup-cc)- [Set up marketplace billing from your cloud provider](#set-up-marketplace-billing-from-cp)- [Support](#support)

Was this page helpful?
Refreshable materialized view

The view will execute immediately and every minute thereafter as configured to ensure updates to the source table are reflected. Our previous query to obtain a summary of actors becomes syntactically simpler and significantly faster!

```
SELECT *
FROM imdb.actor_summary
ORDER BY num_movies DESC
LIMIT 5
```

```
┌─────id─┬─name─────────┬─num_movies─┬──avg_rank─┬─unique_genres─┬─uniq_directors─┬──────────updated_at─┐
│  45332 │ Mel Blanc    │        909 │ 5.7884793 │            19 │            148 │ 2024-11-11 12:01:35 │
│ 621468 │ Bess Flowers │        672 │  5.540605 │            20 │            301 │ 2024-11-11 12:01:35 │
│ 283127 │ Tom London   │        549 │ 2.8057034 │            18 │            208 │ 2024-11-11 12:01:35 │
│ 356804 │ Bud Osborne  │        544 │ 1.9575342 │            16 │            157 │ 2024-11-11 12:01:35 │
│  41669 │ Adoor Bhasi  │        544 │         0 │             4 │            121 │ 2024-11-11 12:01:35 │
└────────┴──────────────┴────────────┴───────────┴───────────────┴────────────────┴─────────────────────┘

5 rows in set. Elapsed: 0.007 sec.
```

Suppose we add a new actor, "Clicky McClickHouse" to our source data who happens to have appeared in a lot of films!

```
INSERT INTO imdb.actors VALUES (845466, 'Clicky', 'McClickHouse', 'M');
INSERT INTO imdb.roles SELECT
        845466 AS actor_id,
        id AS movie_id,
        'Himself' AS role,
        now() AS created_at
FROM imdb.movies
LIMIT 10000, 910;
```

Less than 60 seconds later, our target table is updated to reflect the prolific nature of Clicky's acting:

```
SELECT *
FROM imdb.actor_summary
ORDER BY num_movies DESC
LIMIT 5;
```

```
┌─────id─┬─name────────────────┬─num_movies─┬──avg_rank─┬─unique_genres─┬─uniq_directors─┬──────────updated_at─┐
│ 845466 │ Clicky McClickHouse │        910 │ 1.4687939 │            21 │            662 │ 2024-11-11 12:53:51 │
│  45332 │ Mel Blanc           │        909 │ 5.7884793 │            19 │            148 │ 2024-11-11 12:01:35 │
│ 621468 │ Bess Flowers        │        672 │  5.540605 │            20 │            301 │ 2024-11-11 12:01:35 │
│ 283127 │ Tom London          │        549 │ 2.8057034 │            18 │            208 │ 2024-11-11 12:01:35 │
│  41669 │ Adoor Bhasi         │        544 │         0 │             4 │            121 │ 2024-11-11 12:01:35 │
└────────┴─────────────────────┴────────────┴───────────┴───────────────┴────────────────┴─────────────────────┘

5 rows in set. Elapsed: 0.006 sec.
```

- [When should refreshable materialized views be used?](#when-should-refreshable-materialized-views-be-used)- [How do you refresh a refreshable materialized view?](#how-do-you-refresh-a-refreshable-materialized-view)- [When was a refreshable materialized view last refreshed?](#when-was-a-refreshable-materialized-view-last-refreshed)- [How can I change the refresh rate?](#how-can-i-change-the-refresh-rate)- [Using `APPEND` to add new rows](#using-append-to-add-new-rows)- [Examples](#examples)
            - [Stack Overflow](#stack-overflow)- [IMDb](#imdb)

Was this page helpful?