brendangregg/FlameGraph — pull request #29 by calio (2014-03-21 00:13:21+00:00)
Title: Add support for stack-reversed FlameGraph

Some thoughts on the FlameGraph (Take nginx process with ngx_lua module as an example):

Flame Graph shows the bottom of the stack trace clearly, which are usually “_start”, “main”, but we seldom care about these functions. We care about how much time spent on the top of the stack, but these functions are hard to see because these functions may have some different calling path. And that causes the same function to be split in different part of the graph. The top of the stack trace is the part where CPU is spending time on, but sometimes it’s hard to see the top of the stack trace on FlameGraph, because the top is usually very narrow.

If we reverse the stack trace, it will be better to focus on the functions which are costing CPU time. And it will show how much time is spent on each function itself (not including the time it calls other functions), thus it’s very clear to see which function are costing more CPU time.

Compare there two Flame Graphs:

This is an origin Flame Graph https://www.evernote.com/shard/s46/sh/fd0b5012-cb87-41d3-97ed-6a328653628a/965c688b95924610d1992beffc662b62, we can see that deflating cost a lot of CPU time. But other hot spots are not clear to see.

This is an stack-reversed Flame Graph generated using the same stack trace as the previous one. https://www.evernote.com/shard/s46/sh/51604b00-1484-4f27-93f7-fb4726c61657/fab39d4b1453135639b2207f32a1e89e. Since it is stack reversed, so the top is caller, and the bottom is callee. We can see except system call “accept4”, “epoll_wait” and “writev”, “deflate_slow”, “longest_match”, “compress_block” and “build_tree” cost about 15% of CPU time. And they belongs to deflating. This hot spot is already shown on origin Flame Graph. And besides that, we can also see “lj_str_new” costs about 4.15% of CPU time, which is totally hidden in the origin Flame Graph. And some small hot spots like “ngx_http_email_parse_email” can also be found in stack-reversed Flame Graph.