External disks for storing data

#### Other parameters[​](#azure-blob-storage-other-parameters "Direct link to Other parameters")

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter Description Default Value|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `metadata_path` Local filesystem path to store metadata files for Blob Storage. `/var/lib/clickhouse/disks/<disk_name>/`| `skip_access_check` If `true`, skips disk access checks during startup. `false`| `read_resource` Resource name for [scheduling](/docs/operations/workload-scheduling) read requests. Empty string (disabled)|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | `write_resource` Resource name for [scheduling](/docs/operations/workload-scheduling) write requests. Empty string (disabled)|  |  |  | | --- | --- | --- | | `metadata_keep_free_space_bytes` Amount of free metadata disk space to reserve. - | | | | | | | | | | | | | | | | | |

Examples of working configurations can be found in integration tests directory (see e.g. [test\_merge\_tree\_azure\_blob\_storage](https://github.com/ClickHouse/ClickHouse/blob/master/tests/integration/test_merge_tree_azure_blob_storage/configs/config.d/storage_conf.xml) or [test\_azure\_blob\_storage\_zero\_copy\_replication](https://github.com/ClickHouse/ClickHouse/blob/master/tests/integration/test_azure_blob_storage_zero_copy_replication/configs/config.d/storage_conf.xml)).

Zero-copy replication is not ready for production

Zero-copy replication is disabled by default in ClickHouse version 22.8 and higher. This feature is not recommended for production use.

## Using HDFS storage (Unsupported)[​](#using-hdfs-storage-unsupported "Direct link to Using HDFS storage (Unsupported)")

In this sample configuration:

- the disk is of type `hdfs` (unsupported)
- the data is hosted at `hdfs://hdfs1:9000/clickhouse/`

By the way, HDFS is unsupported and therefore there might be issues when using it. Feel free to make a pull request with the fix if any issue arises.

```
<clickhouse>
    <storage_configuration>
        <disks>
            <hdfs>
                <type>hdfs</type>
                <endpoint>hdfs://hdfs1:9000/clickhouse/</endpoint>
                <skip_access_check>true</skip_access_check>
            </hdfs>
            <hdd>
                <type>local</type>
                <path>/</path>
            </hdd>
        </disks>
        <policies>
            <hdfs>
                <volumes>
                    <main>
                        <disk>hdfs</disk>
                    </main>
                    <external>
                        <disk>hdd</disk>
                    </external>
                </volumes>
            </hdfs>
        </policies>
    </storage_configuration>
</clickhouse>
```

Keep in mind that HDFS may not work in corner cases.

### Using Data Encryption[​](#encrypted-virtual-file-system "Direct link to Using Data Encryption")

You can encrypt the data stored on [S3](/docs/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-s3), or [HDFS](#using-hdfs-storage-unsupported) (unsupported) external disks, or on a local disk. To turn on the encryption mode, in the configuration file you must define a disk with the type `encrypted` and choose a disk on which the data will be saved. An `encrypted` disk ciphers all written files on the fly, and when you read files from an `encrypted` disk it deciphers them automatically. So you can work with an `encrypted` disk like with a normal one.

Example of disk configuration:

```
<disks>
  <disk1>
    <type>local</type>
    <path>/path1/</path>
  </disk1>
  <disk2>
    <type>encrypted</type>
    <disk>disk1</disk>
    <path>path2/</path>
    <key>_16_ascii_chars_</key>
  </disk2>
</disks>
```

For example, when ClickHouse writes data from some table to a file `store/all_1_1_0/data.bin` to `disk1`, then in fact this file will be written to the physical disk along the path `/path1/store/all_1_1_0/data.bin`.

When writing the same file to `disk2`, it will actually be written to the physical disk at the path `/path1/path2/store/all_1_1_0/data.bin` in encrypted mode.

### Required Parameters[​](#required-parameters-encrypted-disk "Direct link to Required Parameters")

|  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter Type Description|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `type` String Must be set to `encrypted` to create an encrypted disk.| `disk` String Type of disk to use for underlying storage.|  |  |  | | --- | --- | --- | | `key` Uint64 Key for encryption and decryption. Can be specified in hexadecimal using `key_hex`. Multiple keys can be specified using the `id` attribute. | | | | | | | | | | | |

### Optional Parameters[​](#optional-parameters-encrypted-disk "Direct link to Optional Parameters")

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter Type Default Description|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `path` String Root directory Location on the disk where data will be saved.|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | `current_key_id` String - The key ID used for encryption. All specified keys can be used for decryption.|  |  |  |  | | --- | --- | --- | --- | | `algorithm` Enum `AES_128_CTR` Encryption algorithm. Options:  - `AES_128_CTR` (16-byte key)  - `AES_192_CTR` (24-byte key)  - `AES_256_CTR` (32-byte key) | | | | | | | | | | | | | | | |

Example of disk configuration:

```
<clickhouse>
    <storage_configuration>
        <disks>
            <disk_s3>
                <type>s3</type>
                <endpoint>...
            </disk_s3>
            <disk_s3_encrypted>
                <type>encrypted</type>
                <disk>disk_s3</disk>
                <algorithm>AES_128_CTR</algorithm>
                <key_hex id="0">00112233445566778899aabbccddeeff</key_hex>
                <key_hex id="1">ffeeddccbbaa99887766554433221100</key_hex>
                <current_key_id>1</current_key_id>
            </disk_s3_encrypted>
        </disks>
    </storage_configuration>
</clickhouse>
```

### Using local cache[​](#using-local-cache "Direct link to Using local cache")

It is possible to configure local cache over disks in storage configuration starting from version 22.3.
For versions 22.3 - 22.7 cache is supported only for `s3` disk type. For versions >= 22.8 cache is supported for any disk type: S3, Azure, Local, Encrypted, etc.
For versions >= 23.5 cache is supported only for remote disk types: S3, Azure, HDFS (unsupported).
Cache uses `LRU` cache policy.

Example of configuration for versions later or equal to 22.8:

```
<clickhouse>
    <storage_configuration>
        <disks>
            <s3>
                <type>s3</type>
                <endpoint>...</endpoint>
                ... s3 configuration ...
            </s3>
            <cache>
                <type>cache</type>
                <disk>s3</disk>
                <path>/s3_cache/</path>
                <max_size>10Gi</max_size>
            </cache>
        </disks>
        <policies>
            <s3_cache>
                <volumes>
                    <main>
                        <disk>cache</disk>
                    </main>
                </volumes>
            </s3_cache>
        <policies>
    </storage_configuration>
```

Example of configuration for versions earlier than 22.8: