S3Queue table engine

URI of S3 bucket to move successfully processed files to, if the destination is another S3 bucket.

Possible values:

- String.

Default value: empty string.

### `after_processing_tag_key`[​](#after_processing_tag_key "Direct link to after_processing_tag_key")

Tag key to put tagging on successfully processed files, if `after_processing='tag'`.

Possible values:

- String.

Default value: empty string.

### `after_processing_tag_value`[​](#after_processing_tag_value "Direct link to after_processing_tag_value")

Tag value to put tagging on successfully processed files, if `after_processing='tag'`.

Possible values:

- String.

Default value: empty string.

### `keeper_path`[​](#keeper_path "Direct link to keeper_path")

Path to the queue metadata in ZooKeeper. If not specified explicitly, ClickHouse builds the path from `s3queue_default_zookeeper_path`, the database UUID, and the table UUID. Absolute values (starting with `/`) are used as provided, while relative values are appended to the configured prefix. Macros such as `{database}` or `{uuid}` are expanded before the engine connects to ZooKeeper.

To target an auxiliary ZooKeeper cluster, prefix the value with the configured name, for example `analytics_keeper:/clickhouse/queue/orders`. The name must exist in `<auxiliary_zookeepers>`; otherwise the engine reports `Unknown auxiliary ZooKeeper name ...`. The full string (including the prefix) is preserved in `SHOW CREATE TABLE` so the statement can be replicated verbatim.

Possible values:

- String.

Default value: `/`.

### `loading_retries`[​](#loading_retries "Direct link to loading_retries")

Retry file loading up to specified number of times. By default, there are no retries.
Possible values:

- Positive integer.

Default value: `0`.

### `processing_threads_num`[​](#processing_threads_num "Direct link to processing_threads_num")

Number of threads to perform processing. Applies only for `Unordered` mode.

Default value: Number of CPUs or 16.

### `parallel_inserts`[​](#parallel_inserts "Direct link to parallel_inserts")

By default `processing_threads_num` will produce one `INSERT`, so it will only download files and parse in multiple threads.
But this limits the parallelism, so for better throughput use `parallel_inserts=true`, this will allow to insert data in parallel (but keep in mind that it will result in higher number of generated data parts for MergeTree family).

Note

`INSERT`s will be spawned with respect to `max_process*_before_commit` settings.

Default value: `false`.

### `enable_logging_to_s3queue_log`[​](#enable_logging_to_s3queue_log "Direct link to enable_logging_to_s3queue_log")

Enable logging to `system.s3queue_log`.

Default value: `0`.

### `polling_min_timeout_ms`[​](#polling_min_timeout_ms "Direct link to polling_min_timeout_ms")

Specifies the minimum time, in milliseconds, that ClickHouse waits before making the next polling attempt.

Possible values:

- Positive integer.

Default value: `1000`.

### `polling_max_timeout_ms`[​](#polling_max_timeout_ms "Direct link to polling_max_timeout_ms")

Defines the maximum time, in milliseconds, that ClickHouse waits before initiating the next polling attempt.

Possible values:

- Positive integer.

Default value: `10000`.

### `polling_backoff_ms`[​](#polling_backoff_ms "Direct link to polling_backoff_ms")

Determines the additional wait time added to the previous polling interval when no new files are found. The next poll occurs after the sum of the previous interval and this backoff value, or the maximum interval, whichever is lower.

Possible values:

- Positive integer.

Default value: `0`.

### `tracked_files_limit`[​](#tracked_files_limit "Direct link to tracked_files_limit")

Allows to limit the number of Zookeeper nodes if the 'unordered' mode is used, does nothing for 'ordered' mode.
If limit reached the oldest processed files will be deleted from ZooKeeper node and processed again.

Possible values:

- Positive integer.

Default value: `1000`.

### `tracked_file_ttl_sec`[​](#tracked_file_ttl_sec "Direct link to tracked_file_ttl_sec")

Maximum number of seconds to store processed files in ZooKeeper node (store forever by default) for 'unordered' mode, does nothing for 'ordered' mode.
After the specified number of seconds, the file will be re-imported.

Possible values:

- Positive integer.

Default value: `0`.

### `cleanup_interval_min_ms`[​](#cleanup_interval_min_ms "Direct link to cleanup_interval_min_ms")

For 'Ordered' mode. Defines a minimum boundary for reschedule interval for a background task, which is responsible for maintaining tracked file TTL and maximum tracked files set.

Default value: `10000`.

### `cleanup_interval_max_ms`[​](#cleanup_interval_max_ms "Direct link to cleanup_interval_max_ms")

For 'Ordered' mode. Defines a maximum boundary for reschedule interval for a background task, which is responsible for maintaining tracked file TTL and maximum tracked files set.

Default value: `30000`.

### `buckets`[​](#buckets "Direct link to buckets")

For 'Ordered' mode. Available since `24.6`. If there are several replicas of S3Queue table, each working with the same metadata directory in keeper, the value of `buckets` needs to be equal to at least the number of replicas. If `processing_threads` setting is used as well, it makes sense to increase the value of `buckets` setting even further, as it defines the actual parallelism of `S3Queue` processing.

### `use_persistent_processing_nodes`[​](#use_persistent_processing_nodes "Direct link to use_persistent_processing_nodes")

By default S3Queue table has always used ephemeral processing nodes, which could lead to duplicates in data in case zookeeper session expires before S3Queue commits processed files in zookeeper, but after it has started processing. This setting forces the server to eliminate possibility of duplicates in case of expired keeper session.

### `persistent_processing_nodes_ttl_seconds`[​](#persistent_processing_nodes_ttl_seconds "Direct link to persistent_processing_nodes_ttl_seconds")

In case of non-graceful server termination, it is possible that if `use_persistent_processing_nodes` is enabled, we can have not removed processing nodes. This setting defines a period of time when these processing nodes can safely be cleaned up.

Default value: `3600` (1 hour).

## S3-related settings[​](#s3-settings "Direct link to S3-related settings")

Engine supports all s3 related settings. For more information about S3 settings see [here](/docs/engines/table-engines/integrations/s3).

## S3 role-based access[​](#s3-role-based-access "Direct link to S3 role-based access")

Scale plan feature

S3 Role-Based Access is available in the Scale and Enterprise plans. To upgrade, visit the plans page in the cloud console.

The s3Queue table engine supports role-based access.
Refer to the documentation [here](/docs/cloud/data-sources/secure-s3) for steps to configure a role to access your bucket.

Once the role is configured, a `roleARN` can be passed via an `extra_credentials` parameter as shown below:

```
CREATE TABLE s3_table
(
    ts DateTime,
    value UInt64
)
ENGINE = S3Queue(
                'https://<your_bucket>/*.csv',
                extra_credentials(role_arn = 'arn:aws:iam::111111111111:role/<your_role>')
                ,'CSV')
SETTINGS
    ...
```

## S3Queue ordered mode[​](#ordered-mode "Direct link to S3Queue ordered mode")

`S3Queue` processing mode allows to store less metadata in ZooKeeper, but has a limitation that files, which added later by time, are required to have alphanumerically bigger names.

`S3Queue` `ordered` mode, as well as `unordered`, supports `(s3queue_)processing_threads_num` setting (`s3queue_` prefix is optional), which allows to control number of threads, which would do processing of `S3` files locally on the server.