Testing ClickHouse

Note that all clickhouse tools (server, client, etc) are just symlinks to a single binary named `clickhouse`.
You can find this binary at `programs/clickhouse`.
All tools can also be invoked as `clickhouse tool` instead of `clickhouse-tool`.

Alternatively you can install ClickHouse package: either stable release from ClickHouse repository or you can build package for yourself with `./release` in ClickHouse sources root.
Then start the server with `sudo clickhouse start` (or stop to stop the server).
Look for logs at `/etc/clickhouse-server/clickhouse-server.log`.

When ClickHouse is already installed on your system, you can build a new `clickhouse` binary and replace the existing binary:

```
$ sudo clickhouse stop
$ sudo cp ./clickhouse /usr/bin/
$ sudo clickhouse start
```

Also you can stop system clickhouse-server and run your own with the same configuration but with logging to terminal:

```
$ sudo clickhouse stop
$ sudo -u clickhouse /usr/bin/clickhouse server --config-file /etc/clickhouse-server/config.xml
```

Example with gdb:

```
$ sudo -u clickhouse gdb --args /usr/bin/clickhouse server --config-file /etc/clickhouse-server/config.xml
```

If the system clickhouse-server is already running and you do not want to stop it, you can change port numbers in your `config.xml` (or override them in a file in `config.d` directory), provide appropriate data path, and run it.

`clickhouse` binary has almost no dependencies and works across wide range of Linux distributions.
To quick and dirty test your changes on a server, you can simply `scp` your fresh built `clickhouse` binary to your server and then run it as in examples above.

## Build tests[​](#build-tests "Direct link to Build tests")

Build tests allow to check that build is not broken on various alternative configurations and on some foreign systems.
These tests are automated as well.

Examples:

- cross-compile for Darwin x86\_64 (macOS)
- cross-compile for FreeBSD x86\_64
- cross-compile for Linux AArch64
- build on Ubuntu with libraries from system packages (discouraged)
- build with shared linking of libraries (discouraged)

For example, build with system packages is bad practice, because we cannot guarantee what exact version of packages a system will have.
But this is really needed by Debian maintainers.
For this reason we at least have to support this variant of build.
Another example: shared linking is a common source of trouble, but it is needed for some enthusiasts.

Though we cannot run all tests on all variant of builds, we want to check at least that various build variants are not broken.
For this purpose we use build tests.

We also test that there are no translation units that are too long to compile or require too much RAM.

We also test that there are no too large stack frames.

## Testing for protocol compatibility[​](#testing-for-protocol-compatibility "Direct link to Testing for protocol compatibility")

When we extend ClickHouse network protocol, we test manually that old clickhouse-client works with new clickhouse-server and new clickhouse-client works with old clickhouse-server (simply by running binaries from corresponding packages).

We also test some cases automatically with integrational tests:

- if data written by old version of ClickHouse can be successfully read by the new version;
- do distributed queries work in a cluster with different ClickHouse versions.

## Help from the Compiler[​](#help-from-the-compiler "Direct link to Help from the Compiler")

Main ClickHouse code (that is located in `src` directory) is built with `-Wall -Wextra -Werror` and with some additional enabled warnings.
Although these options are not enabled for third-party libraries.

Clang has even more useful warnings - you can look for them with `-Weverything` and pick something to default build.

We always use clang to build ClickHouse, both for development and production.
You can build on your own machine with debug mode (to save battery of your laptop), but please note that compiler is able to generate more warnings with `-O3` due to better control flow and inter-procedure analysis.
When building with clang in debug mode, debug version of `libc++` is used that allows to catch more errors at runtime.

## Sanitizers[​](#sanitizers "Direct link to Sanitizers")

Note

If the process (ClickHouse server or client) crashes at startup when running it locally, you might need to disable address space layout randomization: `sudo sysctl kernel.randomize_va_space=0`

### Address sanitizer[​](#address-sanitizer "Direct link to Address sanitizer")

We run functional, integration, stress and unit tests under ASan on per-commit basis.

### Thread sanitizer[​](#thread-sanitizer "Direct link to Thread sanitizer")

We run functional, integration, stress and unit tests under TSan on per-commit basis.

### Memory sanitizer[​](#memory-sanitizer "Direct link to Memory sanitizer")

We run functional, integration, stress and unit tests under MSan on per-commit basis.

### Undefined behaviour sanitizer[​](#undefined-behaviour-sanitizer "Direct link to Undefined behaviour sanitizer")

We run functional, integration, stress and unit tests under UBSan on per-commit basis.
The code of some third-party libraries is not sanitized for UB.

### Valgrind (memcheck)[​](#valgrind-memcheck "Direct link to Valgrind (memcheck)")

We used to run functional tests under Valgrind overnight, but don't do it anymore.
It takes multiple hours.
Currently there is one known false positive in `re2` library, see [this article](https://research.swtch.com/sparse).

## Fuzzing[​](#fuzzing "Direct link to Fuzzing")

ClickHouse fuzzing is implemented both using [libFuzzer](https://llvm.org/docs/LibFuzzer.html) and random SQL queries.
All the fuzz testing should be performed with sanitizers (Address and Undefined).

LibFuzzer is used for isolated fuzz testing of library code.
Fuzzers are implemented as part of test code and have "\_fuzzer" name postfixes.
Fuzzer example can be found at `src/Parsers/fuzzers/lexer_fuzzer.cpp`.
LibFuzzer-specific configs, dictionaries and corpus are stored at `tests/fuzz`.
We encourage you to write fuzz tests for every functionality that handles user input.

Fuzzers are not built by default.
To build fuzzers both `-DENABLE_FUZZING=1` and `-DENABLE_TESTS=1` options should be set.
We recommend to disable Jemalloc while building fuzzers.
Configuration used to integrate ClickHouse fuzzing to
Google OSS-Fuzz can be found at `docker/fuzz`.

We also use simple fuzz test to generate random SQL queries and to check that the server does not die executing them.
You can find it in `00746_sql_fuzzy.pl`.
This test should be run continuously (overnight and longer).

We also use sophisticated AST-based query fuzzer that is able to find huge amount of corner cases.
It does random permutations and substitutions in queries AST.
It remembers AST nodes from previous tests to use them for fuzzing of subsequent tests while processing them in random order.
You can learn more about this fuzzer in [this blog article](https://clickhouse.com/blog/fuzzing-click-house).

## Stress test[​](#stress-test "Direct link to Stress test")

Stress tests are another case of fuzzing.
It runs all functional tests in parallel in random order with a single server.
Results of the tests are not checked.

It is checked that:

- server does not crash, no debug or sanitizer traps are triggered;
- there are no deadlocks;
- the database structure is consistent;
- server can successfully stop after the test and start again without exceptions.

There are five variants (Debug, ASan, TSan, MSan, UBSan).

## Thread fuzzer[​](#thread-fuzzer "Direct link to Thread fuzzer")

Thread Fuzzer (please don't mix up with Thread Sanitizer) is another kind of fuzzing that allows to randomize thread order of execution.
It helps to find even more special cases.

## Security audit[​](#security-audit "Direct link to Security audit")

Our Security Team did some basic overview of ClickHouse capabilities from the security standpoint.

## Static analyzers[​](#static-analyzers "Direct link to Static analyzers")