S3Queue table engine

Example:

```
SELECT *
FROM system.s3queue_metadata_cache

Row 1:
──────
zookeeper_path:        /clickhouse/s3queue/25ea5621-ae8c-40c7-96d0-cec959c5ab88/3b3f66a1-9866-4c2e-ba78-b6bfa154207e
file_name:             wikistat/original/pageviews-20150501-030000.gz
rows_processed:        5068534
status:                Processed
processing_start_time: 2023-10-13 13:09:48
processing_end_time:   2023-10-13 13:10:31
ProfileEvents:         {'ZooKeeperTransactions':3,'ZooKeeperGet':2,'ZooKeeperMulti':1,'SelectedRows':5068534,'SelectedBytes':198132283,'ContextLock':1,'S3QueueSetFileProcessingMicroseconds':2480,'S3QueueSetFileProcessedMicroseconds':9985,'S3QueuePullMicroseconds':273776,'LogTest':17}
exception:
```

2. `system.s3queue_log`. Persistent table. Has the same information as `system.s3queue_metadata_cache`, but for `processed` and `failed` files.

The table has the following structure:

```
SHOW CREATE TABLE system.s3queue_log

Query id: 0ad619c3-0f2a-4ee4-8b40-c73d86e04314

┌─statement──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ CREATE TABLE system.s3queue_log
(
    `event_date` Date,
    `event_time` DateTime,
    `table_uuid` String,
    `file_name` String,
    `rows_processed` UInt64,
    `status` Enum8('Processed' = 0, 'Failed' = 1),
    `processing_start_time` Nullable(DateTime),
    `processing_end_time` Nullable(DateTime),
    `ProfileEvents` Map(String, UInt64),
    `exception` String
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(event_date)
ORDER BY (event_date, event_time) │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```

In order to use `system.s3queue_log` define its configuration in server config file:

```
    <s3queue_log>
        <database>system</database>
        <table>s3queue_log</table>
    </s3queue_log>
```

Example:

```
SELECT *
FROM system.s3queue_log

Row 1:
──────
event_date:            2023-10-13
event_time:            2023-10-13 13:10:12
table_uuid:
file_name:             wikistat/original/pageviews-20150501-020000.gz
rows_processed:        5112621
status:                Processed
processing_start_time: 2023-10-13 13:09:48
processing_end_time:   2023-10-13 13:10:12
ProfileEvents:         {'ZooKeeperTransactions':3,'ZooKeeperGet':2,'ZooKeeperMulti':1,'SelectedRows':5112621,'SelectedBytes':198577687,'ContextLock':1,'S3QueueSetFileProcessingMicroseconds':1934,'S3QueueSetFileProcessedMicroseconds':17063,'S3QueuePullMicroseconds':5841972,'LogTest':17}
exception:
```

- [Create table](#creating-a-table)- [Settings](#settings)
    - [Mode](#mode)- [`after_processing`](#after_processing)- [`after_processing_retries`](#after_processing_retries)- [`after_processing_move_access_key_id`](#after_processing_move_access_key_id)- [`after_processing_move_prefix`](#after_processing_move_prefix)- [`after_processing_move_secret_access_key`](#after_processing_move_secret_access_key)- [`after_processing_move_uri`](#after_processing_move_uri)- [`after_processing_tag_key`](#after_processing_tag_key)- [`after_processing_tag_value`](#after_processing_tag_value)- [`keeper_path`](#keeper_path)- [`loading_retries`](#loading_retries)- [`processing_threads_num`](#processing_threads_num)- [`parallel_inserts`](#parallel_inserts)- [`enable_logging_to_s3queue_log`](#enable_logging_to_s3queue_log)- [`polling_min_timeout_ms`](#polling_min_timeout_ms)- [`polling_max_timeout_ms`](#polling_max_timeout_ms)- [`polling_backoff_ms`](#polling_backoff_ms)- [`tracked_files_limit`](#tracked_files_limit)- [`tracked_file_ttl_sec`](#tracked_file_ttl_sec)- [`cleanup_interval_min_ms`](#cleanup_interval_min_ms)- [`cleanup_interval_max_ms`](#cleanup_interval_max_ms)- [`buckets`](#buckets)- [`use_persistent_processing_nodes`](#use_persistent_processing_nodes)- [`persistent_processing_nodes_ttl_seconds`](#persistent_processing_nodes_ttl_seconds)- [S3-related settings](#s3-settings)- [S3 role-based access](#s3-role-based-access)- [S3Queue ordered mode](#ordered-mode)- [SELECT from S3Queue table engine](#select)- [Description](#description)- [Virtual columns](#virtual-columns)- [Wildcards in path](#wildcards-in-path)- [Limitations](#limitations)- [Introspection](#introspection)

Was this page helpful?