External disks for storing data

Note

Google Cloud Storage (GCS) is also supported using the type `s3`. See [GCS backed MergeTree](/docs/integrations/gcs).

### Using Plain Storage[​](#plain-storage "Direct link to Using Plain Storage")

In `22.10` a new disk type `s3_plain` was introduced, which provides a write-once storage.
Configuration parameters for it are the same as for the `s3` disk type.
Unlike the `s3` disk type, it stores data as is. In other words,
instead of having randomly generated blob names, it uses normal file names
(the same way as ClickHouse stores files on local disk) and does not store any
metadata locally. For example, it is derived from data on `s3`.

This disk type allows keeping a static version of the table, as it does not
allow executing merges on the existing data and does not allow inserting of new
data. A use case for this disk type is to create backups on it, which can be done
via `BACKUP TABLE data TO Disk('plain_disk_name', 'backup_name')`. Afterward,
you can do `RESTORE TABLE data AS data_restored FROM Disk('plain_disk_name', 'backup_name')`
or use `ATTACH TABLE data (...) ENGINE = MergeTree() SETTINGS disk = 'plain_disk_name'`.

Configuration:

```
<s3_plain>
    <type>s3_plain</type>
    <endpoint>https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/</endpoint>
    <use_environment_credentials>1</use_environment_credentials>
</s3_plain>
```

Starting from `24.1` it is possible configure any object storage disk (`s3`, `azure`, `hdfs` (unsupported), `local`) using
the `plain` metadata type.

Configuration:

```
<s3_plain>
    <type>object_storage</type>
    <object_storage_type>azure</object_storage_type>
    <metadata_type>plain</metadata_type>
    <endpoint>https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/</endpoint>
    <use_environment_credentials>1</use_environment_credentials>
</s3_plain>
```

### Using S3 Plain Rewritable Storage[​](#s3-plain-rewritable-storage "Direct link to Using S3 Plain Rewritable Storage")

A new disk type `s3_plain_rewritable` was introduced in `24.4`.
Similar to the `s3_plain` disk type, it does not require additional storage for
metadata files. Instead, metadata is stored in S3.
Unlike the `s3_plain` disk type, `s3_plain_rewritable` allows executing merges
and supports `INSERT` operations.
[Mutations](/docs/sql-reference/statements/alter#mutations) and replication of tables are not supported.

A use case for this disk type is for non-replicated `MergeTree` tables. Although
the `s3` disk type is suitable for non-replicated `MergeTree` tables, you may opt
for the `s3_plain_rewritable` disk type if you do not require local metadata
for the table and are willing to accept a limited set of operations. This could
be useful, for example, for system tables.

Configuration:

```
<s3_plain_rewritable>
    <type>s3_plain_rewritable</type>
    <endpoint>https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/</endpoint>
    <use_environment_credentials>1</use_environment_credentials>
</s3_plain_rewritable>
```

is equal to

```
<s3_plain_rewritable>
    <type>object_storage</type>
    <object_storage_type>s3</object_storage_type>
    <metadata_type>plain_rewritable</metadata_type>
    <endpoint>https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/</endpoint>
    <use_environment_credentials>1</use_environment_credentials>
</s3_plain_rewritable>
```

Starting from `24.5` it is possible to configure any object storage disk
(`s3`, `azure`, `local`) using the `plain_rewritable` metadata type.

### Using Azure Blob Storage[​](#azure-blob-storage "Direct link to Using Azure Blob Storage")

`MergeTree` family table engines can store data to [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/)
using a disk with type `azure_blob_storage`.

Configuration markup:

```
<storage_configuration>
    ...
    <disks>
        <blob_storage_disk>
            <type>azure_blob_storage</type>
            <storage_account_url>http://account.blob.core.windows.net</storage_account_url>
            <container_name>container</container_name>
            <account_name>account</account_name>
            <account_key>pass123</account_key>
            <metadata_path>/var/lib/clickhouse/disks/blob_storage_disk/</metadata_path>
            <cache_path>/var/lib/clickhouse/disks/blob_storage_disk/cache/</cache_path>
            <skip_access_check>false</skip_access_check>
        </blob_storage_disk>
    </disks>
    ...
</storage_configuration>
```

#### Connection parameters[​](#azure-blob-storage-connection-parameters "Direct link to Connection parameters")

|  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter Description Default Value|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `storage_account_url` (Required) Azure Blob Storage account URL. Examples: `http://account.blob.core.windows.net` or `http://azurite1:10000/devstoreaccount1`. -|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | `container_name` Target container name. `default-container`| `container_already_exists` Controls container creation behavior:  - `false`: Creates a new container  - `true`: Connects directly to existing container  - Unset: Checks if container exists, creates if needed - | | | | | | | | | | | |

Authentication parameters (the disk will try all available methods **and** Managed Identity Credential):

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter Description|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | `connection_string` For authentication using a connection string.|  |  |  |  | | --- | --- | --- | --- | | `account_name` For authentication using Shared Key (used with `account_key`).| `account_key` For authentication using Shared Key (used with `account_name`). | | | | | | | |

#### Limit parameters[​](#azure-blob-storage-limit-parameters "Direct link to Limit parameters")

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter Description|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `s3_max_single_part_upload_size` Maximum size of a single block upload to Blob Storage.|  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `min_bytes_for_seek` Minimum size of a seekable region.|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | `max_single_read_retries` Maximum number of attempts to read a chunk of data from Blob Storage.|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | `max_single_download_retries` Maximum number of attempts to download a readable buffer from Blob Storage.|  |  |  |  | | --- | --- | --- | --- | | `thread_pool_size` Maximum number of threads for `IDiskRemote` instantiation.| `s3_max_inflight_parts_for_one_file` Maximum number of concurrent put requests for a single object. | | | | | | | | | | | | | |

#### Other parameters[​](#azure-blob-storage-other-parameters "Direct link to Other parameters")