OSS usage recommendations

[Edit this page](https://github.com/ClickHouse/ClickHouse/tree/master/docs/en/operations/tips.md)

Not supported in ClickHouse Cloud

Note

This page isn't applicable to [ClickHouse Cloud](https://clickhouse.com/cloud). The procedure documented here is automated in ClickHouse Cloud services.

## CPU Scaling Governor[​](#cpu-scaling-governor "Direct link to CPU Scaling Governor")

Always use the `performance` scaling governor. The `on-demand` scaling governor works much worse with constantly high demand.

```
$ echo 'performance' | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
```

## CPU Limitations[​](#cpu-limitations "Direct link to CPU Limitations")

Processors can overheat. Use `dmesg` to see if the CPU's clock rate was limited due to overheating.
The restriction can also be set externally at the datacenter level. You can use `turbostat` to monitor it under a load.

## RAM[​](#ram "Direct link to RAM")

For small amounts of data (up to ~200 GB compressed), it is best to use as much memory as the volume of data.
For large amounts of data and when processing interactive (online) queries, you should use a reasonable amount of RAM (128 GB or more) so the hot data subset will fit in the cache of pages.
Even for data volumes of ~50 TB per server, using 128 GB of RAM significantly improves query performance compared to 64 GB.

Do not disable overcommit. The value `cat /proc/sys/vm/overcommit_memory` should be 0 or 1. Run

```
$ echo 0 | sudo tee /proc/sys/vm/overcommit_memory
```

Use `perf top` to watch the time spent in the kernel for memory management.
Permanent huge pages also do not need to be allocated.

### Using less than 16GB of RAM[​](#using-less-than-16gb-of-ram "Direct link to Using less than 16GB of RAM")

The recommended amount of RAM is 32 GB or more.

If your system has less than 16 GB of RAM, you may experience various memory exceptions because default settings do not match this amount of memory. You can use ClickHouse in a system with a small amount of RAM (as low as 2 GB), but these setups require additional tuning and can only ingest at a low rate.

When using ClickHouse with less than 16GB of RAM, we recommend the following:

- Lower the size of the mark cache in the `config.xml`. It can be set as low as 500 MB, but it cannot be set to zero.
- Lower the number of query processing threads down to `1`.
- Lower the `max_block_size` to `8192`. Values as low as `1024` can still be practical.
- Lower `max_download_threads` to `1`.
- Set `input_format_parallel_parsing` and `output_format_parallel_formatting` to `0`.
- disable writing in log tables, as it keeps the background merge task reserving RAM to perform merges of log tables. Disable `asynchronous_metric_log`, `metric_log`, `text_log`, `trace_log`.

Additional notes:

- To flush the memory cached by the memory allocator, you can run the `SYSTEM JEMALLOC PURGE`
  command.
- We do not recommend using S3 or Kafka integrations on low-memory machines because they require significant memory for buffers.

## Storage Subsystem[​](#storage-subsystem "Direct link to Storage Subsystem")

If your budget allows you to use SSD, use SSD.
If not, use HDD. SATA HDDs 7200 RPM will do.

Give preference to a lot of servers with local hard drives over a smaller number of servers with attached disk shelves.
But for storing archives with rare queries, shelves will work.

## RAID[​](#raid "Direct link to RAID")

When using HDD, you can combine their RAID-10, RAID-5, RAID-6 or RAID-50.
For Linux, software RAID is better (with `mdadm`).
When creating RAID-10, select the `far` layout.
If your budget allows, choose RAID-10.

LVM by itself (without RAID or `mdadm`) is ok, but making RAID with it or combining it with `mdadm` is a less explored option, and there will be more chances for mistakes
(selecting wrong chunk size; misalignment of chunks; choosing a wrong raid type; forgetting to cleanup disks). If you are confident
in using LVM, there is nothing against using it.

If you have more than 4 disks, use RAID-6 (preferred) or RAID-50, instead of RAID-5.
When using RAID-5, RAID-6 or RAID-50, always increase stripe\_cache\_size, since the default value is usually not the best choice.

```
$ echo 4096 | sudo tee /sys/block/md2/md/stripe_cache_size
```

Calculate the exact number from the number of devices and the block size, using the formula: `2 * num_devices * chunk_size_in_bytes / 4096`.

A block size of 64 KB is sufficient for most RAID configurations. The average clickhouse-server write size is approximately 1 MB (1024 KB), and thus the recommended stripe size is also 1 MB. The block size can be optimized if needed when set to 1 MB divided by the number of non-parity disks in the RAID array, such that each write is parallelized across all available non-parity disks.
Never set the block size too small or too large.

You can use RAID-0 on SSD.
Regardless of RAID use, always use replication for data security.

Enable NCQ with a long queue. For HDD, choose the mq-deadline or CFQ scheduler, and for SSD, choose noop. Don't reduce the 'readahead' setting.
For HDD, enable the write cache.

Make sure that [`fstrim`](https://en.wikipedia.org/wiki/Trim_(computing)) is enabled for NVME and SSD disks in your OS (usually it's implemented using a cronjob or systemd service).

## File System[​](#file-system "Direct link to File System")

Ext4 is the most reliable option. Set the mount options `noatime`. XFS works well too.
Most other file systems should also work fine.

FAT-32 and exFAT are not supported due to lack of hard links.

Do not use compressed filesystems, because ClickHouse does compression on its own and better.
It's not recommended to use encrypted filesystems, because you can use builtin encryption in ClickHouse, which is better.

While ClickHouse can work over NFS, it is not the best idea.

## Linux Kernel[​](#linux-kernel "Direct link to Linux Kernel")

Don't use an outdated Linux kernel.

## Network[​](#network "Direct link to Network")

If you are using IPv6, increase the size of the route cache.
The Linux kernel prior to 3.2 had a multitude of problems with IPv6 implementation.

Use at least a 10 GB network, if possible. 1 Gb will also work, but it will be much worse for patching replicas with tens of terabytes of data, or for processing distributed queries with a large amount of intermediate data.

## Huge Pages[​](#huge-pages "Direct link to Huge Pages")

Always set transparent huge pages to `madvise`. On older kernels (before 5.9), THP set to `always` can cause significant performance degradation — the kernel spends excessive time on memory defragmentation, especially on systems with 64 GB+ of RAM. Kernel 5.9 introduced proactive compaction, which handles THP much better, but ClickHouse still warns at startup if THP is set to `always`, so `madvise` is the recommended setting regardless of kernel version.

```
$ echo 'madvise' | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
```

If you want to modify the transparent huge pages setting permanently, editing the `/etc/default/grub` to add the `transparent_hugepage=madvise` to the `GRUB_CMDLINE_LINUX_DEFAULT` option:

```
$ GRUB_CMDLINE_LINUX_DEFAULT="transparent_hugepage=madvise ..."
```

After that, run the `sudo update-grub` command then reboot to take effect.

## Hypervisor configuration[​](#hypervisor-configuration "Direct link to Hypervisor configuration")

If you are using OpenStack, set

```
cpu_mode=host-passthrough
```

in `nova.conf`.

If you are using libvirt, set

```
<cpu mode='host-passthrough'/>
```

in XML configuration.

This is important for ClickHouse to be able to get correct information with `cpuid` instruction.
Otherwise you may get `Illegal instruction` crashes when hypervisor is run on old CPU models.

## ClickHouse Keeper and ZooKeeper[​](#zookeeper "Direct link to ClickHouse Keeper and ZooKeeper")

ClickHouse Keeper is recommended to replace ZooKeeper for ClickHouse clusters. See the documentation for [ClickHouse Keeper](/docs/guides/sre/keeper/clickhouse-keeper)