Project: ClickHouse (App Definition and Development / Database)
File: stack_trace.md

slug: /en/operations/system-tables/stack_trace Contains stack traces of all server threads. Allows developers to introspect the server state. To analyze stack frames, use the `addressToLine`, `addressToLineWithInlines`, `addressToSymbol` and `demangle` . Columns: `thread_name` () Thread name. `thread_id` () Thread identifier. `query_id` () Query identifier that can be used to get details about a query that was running from the system table. `trace` () A which represents a list of physical addresses where the called methods are stored. :::tip Check out the Knowledge Base for some handy queries, including and . ::: Example Enabling introspection functions: ``` sql SET allowintrospectionfunctions = 1; ``` Getting symbols from ClickHouse object files: ``` sql WITH arrayMap(x -> demangle(addressToSymbol(x)), trace) AS all SELECT threadname, threadid, queryid, arrayStringConcat(all, '\n') AS res FROM system.stacktrace LIMIT 1 \G; ``` ``` text Row 1: thread_name: QueryPipelineEx thread_id: 743490 query_id: dc55a564-febb-4e37-95bb-090ef182c6f1 res: memcpy large_ralloc arena_ralloc do_rallocx Allocator<true, true>::realloc(void*, unsigned long, unsigned long, unsigned long) HashTable<unsigned long, HashMapCell<unsigned long, char, HashCRC32<unsigned long>, HashTableNoState, PairNoInit<unsigned long, char>>, HashCRC32<unsigned long>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>::resize(unsigned long, unsigned long) void DB::Aggregator::executeImplBatch<false, false, true, DB::AggregationMethodOneNumber<unsigned long, HashMapTable<unsigned long, HashMapCell<unsigned long, char, HashCRC32<unsigned long>, HashTableNoState, PairNoInit<unsigned long, char>>, HashCRC32<unsigned long>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>, true, false>>(DB::AggregationMethodOneNumber<unsigned long, HashMapTable<unsigned long, HashMapCell<unsigned long, char, HashCRC32<unsigned long>, HashTableNoState, PairNoInit<unsigned long, char>>, HashCRC32<unsigned long>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>, true, false>&, DB::AggregationMethodOneNumber<unsigned long, HashMapTable<unsigned long, HashMapCell<unsigned long, char, HashCRC32<unsigned long>, HashTableNoState, PairNoInit<unsigned long, char>>, HashCRC32<unsigned long>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true>>, true, false>::State&, DB::Arena, unsigned long, unsigned long, DB::Aggregator::AggregateFunctionInstruction, bool, char*) const DB::Aggregator::executeImpl(DB::AggregatedDataVariants&, unsigned long, unsigned long, std::1::vector<DB::IColumn const*, std::1::allocator<DB::IColumn const>>&, DB::Aggregator::AggregateFunctionInstruction, bool, bool, char*) const DB::Aggregator::executeOnBlock(std::1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::1::allocator<COW<DB::IColumn>::immutableptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::1::vector<DB::IColumn const*, std::1::allocator<DB::IColumn const*>>&, std::1::vector<std::1::vector<DB::IColumn const*, std::1::allocator<DB::IColumn const*>>, std::1::allocator<std::1::vector<DB::IColumn const*, std::_1::allocator<DB::IColumn const*>>>>&, bool&) const DB::AggregatingTransform::work() DB::ExecutionThreadContext::executeTask() DB::PipelineExecutor::executeStepImpl(unsigned long, std::1::atomic<bool>*) void std::1::function::policy_invoker<void ()>::callimpl<std::1::function::defaultallocfunc<DB::PipelineExecutor::spawnThreads()::$0, void ()>>(std::1::function::policy_storage const*) ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::1::list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) void std::1::function::policy_invoker<void ()>::callimpl<std::1::function::defaultallocfunc<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::1::function<void ()>, Priority, std::1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::1::function::policystorage const*) void std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void) ``` Getting filenames and line numbers in ClickHouse source code: ``` sql WITH arrayMap(x -> addressToLine(x), trace) AS all, arrayFilter(x -> x LIKE '%/dbms/%', all) AS dbms SELECT threadname, threadid, queryid, arrayStringConcat(notEmpty(dbms) ? dbms : all, '\n') AS res FROM system.stacktrace LIMIT 1 \G; ``` ``` text Row 1: thread_name: clickhouse-serv thread_id: 686 query_id: cad353e7-1c29-4b2e-949f-93e597ab7a54 res: /lib/x86_64-linux-gnu/libc-2.27.so /build/obj-x86_64-linux-gnu/../src/Storages/System/StorageSystemStackTrace.cpp:182 /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/vector:656 /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:1338 /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:751 /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:224 /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectWithUnionQuery.cpp:192 /build/obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:384 /build/obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:643 /build/obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:251 /build/obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1197 /build/obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:57 /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:856 /build/obj-x8664-linux-gnu/../contrib/poco/Foundation/include/Poco/MutexPOSIX.h:59 /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/AutoPtr.h:223 /lib/x86_64-linux-gnu/libpthread-2.27.so /lib/x86_64-linux-gnu/libc-2.27.so ``` See Also Which introspection functions are available and how to use them. Contains stack traces collected by the sampling query profiler. Description and usage example of the `arrayMap` function. Description and usage example of the `arrayFilter` function.