Allocation profiling

- `raw` — raw heap profile as produced by jemalloc.
- `symbolized` — jeprof-compatible format with embedded function symbols. Since symbols are already embedded, `jeprof` can analyze the output without requiring the ClickHouse binary.
- `collapsed` (default) — FlameGraph-compatible collapsed stacks, one stack per line with the byte count.

For example, to get the raw profile:

```
SELECT * FROM system.jemalloc_profile_text
SETTINGS jemalloc_profile_text_output_format = 'raw'
```

To get symbolized output:

```
SELECT * FROM system.jemalloc_profile_text
SETTINGS jemalloc_profile_text_output_format = 'symbolized'
```

### Additional settings[​](#fetching-heap-profiles-settings "Direct link to Additional settings")

- `jemalloc_profile_text_symbolize_with_inline` (Bool, default: `true`) — Whether to include inline frames when symbolizing. Disabling this speeds up symbolization significantly but loses precision as inlined function calls will not appear in the stacks. Only affects `symbolized` and `collapsed` formats.
- `jemalloc_profile_text_collapsed_use_count` (Bool, default: `false`) — When using the `collapsed` format, aggregate by allocation count instead of bytes.

### Example: generating a flame graph from SQL[​](#example-flamegraph-from-sql "Direct link to Example: generating a flame graph from SQL")

Since the default output format is `collapsed`, you can pipe the output directly to FlameGraph:

```
clickhouse-client -q "SELECT * FROM system.jemalloc_profile_text" | flamegraph.pl --color=mem --title="Allocation Flame Graph" --width 2400 > result.svg
```

To generate a flame graph by allocation count instead of bytes:

```
clickhouse-client -q "SELECT * FROM system.jemalloc_profile_text SETTINGS jemalloc_profile_text_collapsed_use_count = 1" | flamegraph.pl --color=mem --title="Allocation Count Flame Graph" --width 2400 > result.svg
```

## Flushing heap profiles to disk[​](#flushing-heap-profiles "Direct link to Flushing heap profiles to disk")

If you need to save heap profiles as files for offline analysis with `jeprof`, you can flush them to disk.

By default, the heap profile file will be generated in `/tmp/jemalloc_clickhouse._pid_._seqnum_.heap` where `_pid_` is the PID of ClickHouse and `_seqnum_` is the global sequence number for the current heap profile.
For Keeper, the default file is `/tmp/jemalloc_keeper._pid_._seqnum_.heap`, and follows the same rules.

To flush the current profile:

- ClickHouse- Keeper

```
SYSTEM JEMALLOC FLUSH PROFILE
```

It will return the location of the flushed profile.

```
echo jmfp | nc localhost 9181
```

A different location can be defined by appending the `MALLOC_CONF` environment variable with the `prof_prefix` option.
For example, if you want to generate profiles in the `/data` folder where the filename prefix will be `my_current_profile`, you can run ClickHouse/Keeper with the following environment variable:

```
MALLOC_CONF=prof_prefix:/data/my_current_profile
```

The generated file will be appended to the prefix PID and sequence number.

## Analyzing heap profile files with `jeprof`[​](#analyzing-heap-profile-files-with-jeprof "Direct link to analyzing-heap-profile-files-with-jeprof")

After flushing heap profiles to disk, they can be analyzed using `jemalloc`'s tool called [jeprof](https://github.com/jemalloc/jemalloc/blob/dev/bin/jeprof.in). It can be installed in multiple ways:

- Using the system's package manager
- Cloning the [jemalloc repo](https://github.com/jemalloc/jemalloc) and running `autogen.sh` from the root folder. This will provide you with the `jeprof` script inside the `bin` folder

There are many different output formats available. Run `jeprof --help` for the full list of options.

### Symbolized heap profiles[​](#symbolized-heap-profiles "Direct link to Symbolized heap profiles")

Starting from version 26.1+, ClickHouse automatically generates symbolized heap profiles when you flush using `SYSTEM JEMALLOC FLUSH PROFILE`.
The symbolized profile (with `.symbolized` extension) contains embedded function symbols and can be analyzed by `jeprof` without requiring the ClickHouse binary.

For example, when you run:

```
SYSTEM JEMALLOC FLUSH PROFILE
```

ClickHouse will return the path to the symbolized profile (e.g., `/tmp/jemalloc_clickhouse.12345.0.heap.symbolized`).

You can then analyze it directly with `jeprof`:

```
jeprof /tmp/jemalloc_clickhouse.12345.0.heap.symbolized --output_format [ > output_file]
```

Note

**No binary required**: When using symbolized profiles (`.symbolized` files), you don't need to provide the ClickHouse binary path to `jeprof`. This makes it much easier to analyze profiles on different machines or after the binary has been updated.

If you have an older non-symbolized heap profile and still have access to the ClickHouse binary, you can use the traditional approach:

```
jeprof path/to/clickhouse path/to/heap/profile --output_format [ > output_file]
```

Note

For non-symbolized profiles, `jeprof` uses `addr2line` to generate stacktraces which can be really slow.
If that's the case, it is recommended to install an [alternative implementation](https://github.com/gimli-rs/addr2line) of the tool.

```
git clone https://github.com/gimli-rs/addr2line.git --depth=1 --branch=0.23.0
cd addr2line
cargo build --features bin --release
cp ./target/release/addr2line path/to/current/addr2line
```

Alternatively, `llvm-addr2line` works equally well (But note, that `llvm-objdump` is not compatible with `jeprof`)

And later use it like this `jeprof --tools addr2line:/usr/bin/llvm-addr2line,nm:/usr/bin/llvm-nm,objdump:/usr/bin/objdump,c++filt:/usr/bin/llvm-cxxfilt`

When comparing two profiles, you can use the `--base` argument:

```
jeprof --base /path/to/first.heap.symbolized /path/to/second.heap.symbolized --output_format [ > output_file]
```

### Examples[​](#examples "Direct link to Examples")

Using symbolized profiles (recommended):

- Generate a text file with each procedure written per line:

```
jeprof /tmp/jemalloc_clickhouse.12345.0.heap.symbolized --text > result.txt
```

- Generate a PDF file with a call-graph:

```
jeprof /tmp/jemalloc_clickhouse.12345.0.heap.symbolized --pdf > result.pdf
```

Using non-symbolized profiles (requires binary):

- Generate a text file with each procedure written per line:

```
jeprof /path/to/clickhouse /tmp/jemalloc_clickhouse.12345.0.heap --text > result.txt
```

- Generate a PDF file with a call-graph:

```
jeprof /path/to/clickhouse /tmp/jemalloc_clickhouse.12345.0.heap --pdf > result.pdf
```

### Generating a flame graph[​](#generating-flame-graph "Direct link to Generating a flame graph")

`jeprof` allows you to generate collapsed stacks for building flame graphs.

You need to use the `--collapsed` argument:

```
jeprof /tmp/jemalloc_clickhouse.12345.0.heap.symbolized --collapsed > result.collapsed
```

Or with a non-symbolized profile:

```
jeprof /path/to/clickhouse /tmp/jemalloc_clickhouse.12345.0.heap --collapsed > result.collapsed
```

After that, you can use many different tools to visualize collapsed stacks.

The most popular is [FlameGraph](https://github.com/brendangregg/FlameGraph) which contains a script called `flamegraph.pl`:

```
cat result.collapsed | /path/to/FlameGraph/flamegraph.pl --color=mem --title="Allocation Flame Graph" --width 2400 > result.svg
```

Another interesting tool is [speedscope](https://www.speedscope.app/) that allows you to analyze collected stacks in a more interactive way.

## Additional options for the profiler[​](#additional-options-for-profiler "Direct link to Additional options for the profiler")

`jemalloc` has many different options available, which are related to the profiler. They can be controlled by modifying the `MALLOC_CONF` environment variable.
For example, the interval between allocation samples can be controlled with `lg_prof_sample`.
If you want to dump the heap profile every N bytes you can enable it using `lg_prof_interval`.

It is recommended to check `jemalloc`s [reference page](https://jemalloc.net/jemalloc.3.html) for a complete list of options.

## Other resources[​](#other-resources "Direct link to Other resources")