Project: ClickHouse (App Definition and Development / Database)
File: allocation-profiling.md

slug: /en/operations/allocation-profiling sidebar_label: "Allocation profiling" title: "Allocation profiling" import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ClickHouse uses as its global allocator that comes with some tools for allocation sampling and profiling. To make allocation profiling more convenient, `SYSTEM` commands are provided along 4LW commands in Keeper. If we want to sample and profile allocations in `jemalloc`, we need to start ClickHouse/Keeper with profiling enabled using environment variable `MALLOC_CONF`. ```sh MALLOCCONF=backgroundthread:true,prof:true ``` `jemalloc` will sample allocation and store the information internally. We can tell `jemalloc` to flush current profile by running: <Tabs groupId="binary"> <TabItem value="clickhouse" label="ClickHouse"> SYSTEM JEMALLOC FLUSH PROFILE </TabItem> <TabItem value="keeper" label="Keeper"> echo jmfp | nc localhost 9181 </TabItem> </Tabs> By default, heap profile file will be generated in `/tmp/jemallocclickhouse.pid.seqnum.heap` where `pid` is the PID of ClickHouse and `seqnum_` is the global sequence number for the current heap profile. For Keeper, the default file is `/tmp/jemallockeeper.pid.seqnum_.heap` following the same rules. A different location can be defined by appending the `MALLOCCONF` environment variable with `profprefix` option. For example, if we want to generate profiles in `/data` folder where the prefix for filename will be `mycurrentprofile` we can run ClickHouse/Keeper with following environment variable: ```sh MALLOCCONF=backgroundthread:true,prof:true,profprefix:/data/mycurrent_profile ``` Generated file will append to prefix PID and sequence number. After we generated heap profiles, we need to analyze them. For that, we need to use `jemalloc`'s tool called which can be installed in multiple ways: installing `jemalloc` using system's package manager cloning and running autogen.sh from the root folder that will provide you with `jeprof` script inside the `bin` folder :::note `jeprof` uses `addr2line` to generate stacktraces which can be really slow. If thats the case, we recommend installing an of the tool. ``` git clone https://github.com/gimli-rs/addr2line cd addr2line cargo b --examples -r cp ./target/release/examples/addr2line path/to/current/addr2line ``` ::: There are many different formats to generate from the heap profile using `jeprof`. We recommend to run `jeprof --help` to check usage and many different options the tool

In general, `jeprof` command will look like this: ```sh jeprof path/to/binary path/to/heap/profile --outputformat [ > outputfile] ``` If we want to compare which allocations happened between 2 profiles we can set the base argument: ```sh jeprof path/to/binary --base path/to/first/heap/profile path/to/second/heap/profile --outputformat [ > outputfile] ``` For example: if we want to generate a text file with each procedure written per line: ```sh jeprof path/to/binary path/to/heap/profile --text > result.txt ``` if we want to generate a PDF file with call-graph: ```sh jeprof path/to/binary path/to/heap/profile --pdf > result.pdf ``` `jeprof` allows us to generate collapsed stacks for building flame graphs. We need to use `--collapsed` argument: ```sh jeprof path/to/binary path/to/heap/profile --collapsed > result.collapsed ``` After that, we can use many different tools to visualize collapsed stacks. Most popular would be which contains a script called `flamegraph.pl`: ```sh cat result.collapsed | /path/to/FlameGraph/flamegraph.pl --color=mem --title="Allocation Flame Graph" --width 2400 > result.svg ``` Another interesting tool is that allows you to analyze collected stacks in a more interactive way. If ClickHouse/Keeper were started with enabled profiler, they support additional commands for disabling/enabling allocation profiling during runtime. Using those commands, it's easier to profile only specific intervals. Disable profiler: <Tabs groupId="binary"> <TabItem value="clickhouse" label="ClickHouse"> SYSTEM JEMALLOC DISABLE PROFILE </TabItem> <TabItem value="keeper" label="Keeper"> echo jmdp | nc localhost 9181 </TabItem> </Tabs> Enable profiler: <Tabs groupId="binary"> <TabItem value="clickhouse" label="ClickHouse"> SYSTEM JEMALLOC ENABLE PROFILE </TabItem> <TabItem value="keeper" label="Keeper"> echo jmep | nc localhost 9181 </TabItem> </Tabs> It's also possible to control the initial state of the profiler by setting `prof_active` option which is enabled by default. For example, if we don't want to sample allocations during startup but only after we enable the profiler, we can start ClickHouse/Keeper with following environment variable: ```sh MALLOCCONF=backgroundthread:true,prof:true,prof_active:false ``` and enable profiler at a later point. `jemalloc` has many different options available related to profiler which can be controlled by modifying `MALLOC_CONF` environment variable. For example, interval between allocation samples can be controlled with `lgprofsample`. If you want to dump heap profile every N bytes you can enable it using `lgprofinterval`. We recommend to check `jemalloc`s for such options. ClickHouse/Keeper expose `jemalloc` related metrics in many different ways. :::warning Warning It's important to be aware that none of these metrics are synchronized with each other and values may drift. ::: ```sql SELECT * FROM system.asynchronous_metrics WHERE metric ILIKE '%jemalloc%' FORMAT Vertical ``` Contains information about memory allocations done via jemalloc allocator in different size classes (bins) aggregated from all arenas. All `jemalloc` related metrics from `asynchronous_metrics` are also exposed using Prometheus endpoint in both ClickHouse and Keeper. Keeper supports `jmst` 4LW command which returns . Example: ```sh echo jmst | nc localhost 9181 ```