Testing ClickHouse

It is checked that:

- server does not crash, no debug or sanitizer traps are triggered;
- there are no deadlocks;
- the database structure is consistent;
- server can successfully stop after the test and start again without exceptions.

There are five variants (Debug, ASan, TSan, MSan, UBSan).

## Thread fuzzer[​](#thread-fuzzer "Direct link to Thread fuzzer")

Thread Fuzzer (please don't mix up with Thread Sanitizer) is another kind of fuzzing that allows to randomize thread order of execution.
It helps to find even more special cases.

## Security audit[​](#security-audit "Direct link to Security audit")

Our Security Team did some basic overview of ClickHouse capabilities from the security standpoint.

## Static analyzers[​](#static-analyzers "Direct link to Static analyzers")

We run `clang-tidy` on per-commit basis.
`clang-static-analyzer` checks are also enabled.
`clang-tidy` is also used for some style checks.

We have evaluated `clang-tidy`, `Coverity`, `cppcheck`, `PVS-Studio`, `tscancode`, `CodeQL`.
You will find instructions for usage in `tests/instructions/` directory.

If you use `CLion` as an IDE, you can leverage some `clang-tidy` checks out of the box.

We also use `shellcheck` for static analysis of shell scripts.

## Hardening[​](#hardening "Direct link to Hardening")

In debug build we are using custom allocator that does ASLR of user-level allocations.

We also manually protect memory regions that are expected to be readonly after allocation.

In debug build we also involve a customization of libc that ensures that no "harmful" (obsolete, insecure, not thread-safe) functions are called.

Debug assertions are used extensively.

In debug build, if exception with "logical error" code (implies a bug) is being thrown, the program is terminated prematurely.
It allows to use exceptions in release build but make it an assertion in debug build.

Debug version of jemalloc is used for debug builds.
Debug version of libc++ is used for debug builds.

## Runtime integrity checks[​](#runtime-integrity-checks "Direct link to Runtime integrity checks")

Data stored on disk is checksummed.
Data in MergeTree tables is checksummed in three ways simultaneously\* (compressed data blocks, uncompressed data blocks, the total checksum across blocks).
Data transferred over network between client and server or between servers is also checksummed.
Replication ensures bit-identical data on replicas.

It is required to protect from faulty hardware (bit rot on storage media, bit flips in RAM on server, bit flips in RAM of network controller, bit flips in RAM of network switch, bit flips in RAM of client, bit flips on the wire).
Note that bit flips are common and likely to occur even for ECC RAM and in presence of TCP checksums (if you manage to run thousands of servers processing petabytes of data each day).
[See the video (russian)](https://www.youtube.com/watch?v=ooBAQIe0KlQ).

ClickHouse provides diagnostics that will help ops engineers to find faulty hardware.

\* and it is not slow.

## Code style[​](#code-style "Direct link to Code style")

Code style rules are described [here](/docs/development/style).

To check for some common style violations, you can use `utils/check-style` script.

To force proper style of your code, you can use `clang-format`.
File `.clang-format` is located at the sources root.
It mostly corresponding with our actual code style.
But it's not recommended to apply `clang-format` to existing files because it makes formatting worse.
You can use `clang-format-diff` tool that you can find in clang source repository.

Alternatively you can try `uncrustify` tool to reformat your code.
Configuration is in `uncrustify.cfg` in the sources root.
It is less tested than `clang-format`.

`CLion` has its own code formatter that has to be tuned for our code style.

We also use `codespell` to find typos in code.
It is automated as well.

## Test coverage[​](#test-coverage "Direct link to Test coverage")

We also track test coverage but only for functional tests and only for clickhouse-server.
It is performed on daily basis.

## Tests for tests[​](#tests-for-tests "Direct link to Tests for tests")

There is automated check for flaky tests.
It runs all new tests 100 times (for functional tests) or 10 times (for integration tests).
If at least single time the test failed, it is considered flaky.

## Test automation[​](#test-automation "Direct link to Test automation")

We run tests with [GitHub Actions](https://github.com/features/actions).

Build jobs and tests are run in Sandbox on per commit basis.
Resulting packages and test results are published in GitHub and can be downloaded by direct links.
Artifacts are stored for several months.
When you send a pull request on GitHub, we tag it as "can be tested" and our CI system will build ClickHouse packages (release, debug, with address sanitizer, etc) for you.

- [Test types](#test-types)- [Functional tests](#functional-tests)
    - [Running a test locally](#running-a-test-locally)- [Running fast tests](#running-fast-tests)- [Running stateless tests](#running-stateless-tests)- [Adding a new test](#adding-a-new-test)- [Restricting test runs](#restricting-test-runs)- [Specifying limits for random settings](#specifying-limits-for-random-settings)- [Choosing the test name](#choosing-the-test-name)- [Checking for an error that must occur](#checking-for-an-error-that-must-occur)- [Testing a distributed query](#testing-a-distributed-query)- [Working with temporary files](#working-with-temporary-files)- [Known bugs](#known-bugs)- [Integration tests](#integration-tests)- [Unit tests](#unit-tests)- [Performance tests](#performance-tests)- [Test tools and scripts](#test-tools-and-scripts)- [Miscellaneous tests](#miscellaneous-tests)- [Manual Testing](#manual-testing)- [Build tests](#build-tests)- [Testing for protocol compatibility](#testing-for-protocol-compatibility)- [Help from the Compiler](#help-from-the-compiler)- [Sanitizers](#sanitizers)
                          - [Address sanitizer](#address-sanitizer)- [Thread sanitizer](#thread-sanitizer)- [Memory sanitizer](#memory-sanitizer)- [Undefined behaviour sanitizer](#undefined-behaviour-sanitizer)- [Valgrind (memcheck)](#valgrind-memcheck)- [Fuzzing](#fuzzing)- [Stress test](#stress-test)- [Thread fuzzer](#thread-fuzzer)- [Security audit](#security-audit)- [Static analyzers](#static-analyzers)- [Hardening](#hardening)- [Runtime integrity checks](#runtime-integrity-checks)- [Code style](#code-style)- [Test coverage](#test-coverage)- [Tests for tests](#tests-for-tests)- [Test automation](#test-automation)

Was this page helpful?