title_wip

To enable query slot scheduling for workloads create QUERY resource and set a limit for the number of concurrent queries or queries per second:

```
CREATE RESOURCE query (QUERY)
CREATE WORKLOAD all SETTINGS max_concurrent_queries = 100, max_queries_per_second = 10, max_burst_queries = 20
```

Workload setting `max_concurrent_queries` limits the number of concurrent queries that could run simultaneously for a given workload. This is analog of query [`max_concurrent_queries_for_all_users`](/docs/operations/settings/settings#max_concurrent_queries_for_all_users) and server [max\_concurrent\_queries](/docs/operations/server-configuration-parameters/settings#max_concurrent_queries) settings. Async insert queries and some specific queries like KILL are not counted towards the limit.

Workload settings `max_queries_per_second` and `max_burst_queries` limit number of queries for the workload with a token bucket throttler. It guarantees that during any time interval `T` no more than `max_queries_per_second * T + max_burst_queries` new queries will start execution.

Workload setting `max_waiting_queries` limits number of waiting queries for the workload. When the limit is reached, the server returns an error `SERVER_OVERLOADED`.

Note

Blocked queries will wait indefinitely and not appear in `SHOW PROCESSLIST` until all constraints are satisfied.

## Workloads and resources storage[​](#workload_entity_storage "Direct link to Workloads and resources storage")

Definitions of all workloads and resources in the form of `CREATE WORKLOAD` and `CREATE RESOURCE` queries are stored persistently either on disk at `workload_path` or in ZooKeeper at `workload_zookeeper_path`. ZooKeeper storage is recommended to achieve consistency between nodes. Alternatively `ON CLUSTER` clause could be used along with disk storage.

## Configuration-based workloads and resources[​](#config_based_workloads "Direct link to Configuration-based workloads and resources")

In addition to SQL-based definitions, workloads and resources can be predefined in the server configuration file. This is useful in cloud environments where some limitations are dictated by infrastructure, while other limits could be changed by customers. Configuration-based entities have priority over SQL-defined ones and cannot be modified or deleted using SQL commands.

### Configuration format[​](#config_based_workloads_format "Direct link to Configuration format")

```
<clickhouse>
    <resources_and_workloads>
        CREATE RESOURCE s3disk_read (READ DISK s3);
        CREATE RESOURCE s3disk_write (WRITE DISK s3);
        CREATE WORKLOAD all SETTINGS max_io_requests = 500 FOR s3disk_read, max_io_requests = 1000 FOR s3disk_write, max_bytes_per_second = 1342177280 FOR s3disk_read, max_bytes_per_second = 3355443200 FOR s3disk_write;
        CREATE WORKLOAD production IN all SETTINGS weight = 3;
    </resources_and_workloads>
</clickhouse>
```

The configuration uses the same SQL syntax as `CREATE WORKLOAD` and `CREATE RESOURCE` statements. All queries must be valid.

### Usage recommendations[​](#config_based_workloads_usage_recommendations "Direct link to Usage recommendations")

For cloud environments, a typical setup might include:

1. Define root workload and network IO resources in configuration to set infrastructure limits
2. Set `throw_on_unknown_workload` to enforce these limits
3. Create a `CREATE WORKLOAD default IN all` to automatically apply limits to all queries (since the default value for `workload` query setting is 'default')
4. Allow users to create additional workloads within the configured hierarchy

This ensures that all background activities and queries respect the infrastructure limitations while still allowing flexibility for user-specific scheduling policies.

Another use case is different configuration for different nodes in a heterogeneous cluster.

## Strict resource access[​](#strict_resource_access "Direct link to Strict resource access")

To enforce all queries to follow resource scheduling policies there is a server setting `throw_on_unknown_workload`. If it is set to `true` then every query is required to use valid `workload` query setting, otherwise `RESOURCE_ACCESS_DENIED` exception is thrown. If it is set to `false` then such a query does not use resource scheduler, i.e. it will get unlimited access to any `RESOURCE`. Query setting 'use\_concurrency\_control = 0' allows query to avoid CPU scheduler and get unlimited access to CPU. To enforce CPU scheduling create a setting constraint to keep 'use\_concurrency\_control' read-only constant value.

Note

Do not set `throw_on_unknown_workload` to `true` unless `CREATE WORKLOAD default` is executed. It could lead to server startup issues if a query without explicit setting `workload` is executed during startup.

## See also[​](#see-also "Direct link to See also")

- [system.scheduler](/docs/operations/system-tables/scheduler)
- [system.workloads](/docs/operations/system-tables/workloads)
- [system.resources](/docs/operations/system-tables/resources)
- [merge\_workload](/docs/operations/settings/merge-tree-settings#merge_workload) merge tree setting
- [merge\_workload](/docs/operations/server-configuration-parameters/settings#merge_workload) global server setting
- [mutation\_workload](/docs/operations/settings/merge-tree-settings#mutation_workload) merge tree setting
- [mutation\_workload](/docs/operations/server-configuration-parameters/settings#mutation_workload) global server setting
- [workload\_path](/docs/operations/server-configuration-parameters/settings#workload_path) global server setting
- [workload\_zookeeper\_path](/docs/operations/server-configuration-parameters/settings#workload_zookeeper_path) global server setting
- [cpu\_slot\_preemption](/docs/operations/server-configuration-parameters/settings#cpu_slot_preemption) global server setting
- [cpu\_slot\_quantum\_ns](/docs/operations/server-configuration-parameters/settings#cpu_slot_quantum_ns) global server setting
- [cpu\_slot\_preemption\_timeout\_ms](/docs/operations/server-configuration-parameters/settings#cpu_slot_preemption_timeout_ms) global server setting

- [Disk configuration](#disk_config)- [Workload markup](#workload_markup)- [Resource scheduling hierarchy](#hierarchy)- [Workload classifiers](#workload_classifiers)- [Workload hierarchy](#workloads)- [CPU scheduling](#cpu_scheduling)- [Threads vs. CPUs](#threads_vs_cpus)- [Query slot scheduling](#query_scheduling)- [Workloads and resources storage](#workload_entity_storage)- [Configuration-based workloads and resources](#config_based_workloads)
                    - [Configuration format](#config_based_workloads_format)- [Usage recommendations](#config_based_workloads_usage_recommendations)- [Strict resource access](#strict_resource_access)- [See also](#see-also)

Was this page helpful?