Iceberg table engine

1. **Direct Path Specification**:

- If you set `iceberg_metadata_file_path`, the system will use this exact path by combining it with the Iceberg table directory path.
- When this setting is provided, all other resolution settings are ignored.

2. **Table UUID Matching**:

- If `iceberg_metadata_table_uuid` is specified, the system will:
  - Look only at `.metadata.json` files in the `metadata` directory
  - Filter for files containing a `table-uuid` field matching your specified UUID (case-insensitive)

3. **Default Search**:

- If neither of the above settings are provided, all `.metadata.json` files in the `metadata` directory become candidates

### Selecting the most recent file[​](#most-recent-file "Direct link to Selecting the most recent file")

After identifying candidate files using the above rules, the system determines which one is the most recent:

- If `iceberg_recent_metadata_file_by_last_updated_ms_field` is enabled:

  - The file with the largest `last-updated-ms` value is selected
- Otherwise:

  - The file with the highest version number is selected
  - (Version appears as `V` in filenames formatted as `V.metadata.json` or `V-uuid.metadata.json`)

**Note**: All mentioned settings (unless explicitly specified otherwise) are engine-level settings and must be specified during table creation as shown below:

```
CREATE TABLE example_table ENGINE = Iceberg(
    's3://bucket/path/to/iceberg_table'
) SETTINGS iceberg_metadata_table_uuid = '6f6f6407-c6a5-465f-a808-ea8900e35a38';
```

**Note**: While Iceberg Catalogs typically handle metadata resolution, the `Iceberg` table engine in ClickHouse directly interprets files stored in S3 as Iceberg tables, which is why understanding these resolution rules is important.

## Data cache[​](#data-cache "Direct link to Data cache")

`Iceberg` table engine and table function support data caching same as `S3`, `AzureBlobStorage`, `HDFS` storages. See [here](/docs/engines/table-engines/integrations/s3#data-cache).

## Metadata cache[​](#metadata-cache "Direct link to Metadata cache")

`Iceberg` table engine and table function support metadata cache storing the information of manifest files, manifest list and metadata json. The cache is stored in memory. This feature is controlled by setting `use_iceberg_metadata_files_cache`, which is enabled by default.

## Asynchronous metadata prefetching[​](#async-metadata-prefetch "Direct link to Asynchronous metadata prefetching")

Asynchronous metadata prefetching can be enabled at `Iceberg` table creation by setting `iceberg_metadata_async_prefetch_period_ms`. If set to 0 (default) or if metadata caching is not enabled, the asynchronous prefetching is disabled.
In order to enable this feature, a non-zero value of milliseconds should be given. It represents interval between prefetching cycles.

If enabled, the server will run a recurring background operation to list the remote catalog and to detect new metadata version. It will then parse it and recursively walk the snapshot, fetching active manifest list files and manifest files.
The files already available at the metadata cache, won't be downloaded again. At the end of each prefetching cycle, the latest metadata snapshot is available at the metadata cache.

```
CREATE TABLE example_table ENGINE = Iceberg(
    's3://bucket/path/to/iceberg_table'
) SETTINGS
    iceberg_metadata_async_prefetch_period_ms = 60000;
```

In order to make the most of asynchronous metadata prefetching at read operations, `iceberg_metadata_staleness_ms` parameter should be specified as Query or Session parameter. By default (0 - not specified) in the context of each query, the server will fetch latest metadata from the remote catalog.
By specifying tolerance to metadata staleness, the server is allowed to use the cached version of metadata snapshot without calling the remote catalog. If there's metadata version in cache, and it has been downloaded within the given window of staleness, it will be used to process the query.
Otherwise the latest version will be fetched from the remote catalog.

```
SELECT count() FROM icebench_table WHERE ...
SETTINGS iceberg_metadata_staleness_ms=120000
```

**Note**: Asynchronous metadata prefetching runs at `ICEBERG_SCEDULE_POOL`, which is server-side threadpool for background operations on active `Iceberg` tables. The size of this threadpool is controlled by `iceberg_background_schedule_pool_size` server configuration parameter (default is 10).

**Note**: Current expectation is that metadata cache size is sufficient to hold the latest metadata snapshot in full for all active tables, if asynchronous prefetching is enabled.

## See also[​](#see-also "Direct link to See also")

- [iceberg table function](/docs/sql-reference/table-functions/iceberg)

- [Create table](#create-table)- [Engine arguments](#engine-arguments)
    - [Example](#example)- [Aliases](#aliases)- [Data types](#data-types)
        - [Primitive types](#primitive-types)- [Complex types](#complex-types)- [Schema evolution](#schema-evolution)- [Partition pruning](#partition-pruning)- [Time travel](#time-travel)- [Processing of tables with deleted rows](#deleted-rows)
                - [Basic usage](#basic-usage)- [Important considerations](#important-considerations)- [Example scenarios](#example-scenarios)- [Metadata file resolution](#metadata-file-resolution)
                  - [Candidates search](#candidate-search)- [Selecting the most recent file](#most-recent-file)- [Data cache](#data-cache)- [Metadata cache](#metadata-cache)- [Asynchronous metadata prefetching](#async-metadata-prefetch)- [See also](#see-also)

Was this page helpful?