Chapter 19. Profiling CPU usage in real time with perf top You can use the perf top command to measure CPU usage of different functions in real time. Prerequisites You have the perf user space tool installed as described in Installing perf . 19.1. The purpose of perf top The perf top command is used for real time system profiling and functions similarly to the top utility. However, where the top utility generally shows you how much CPU time a given process or thread is using, perf top shows you how much CPU time each specific function uses. In its default state, perf top tells you about functions being used across all CPUs in both the user-space and the kernel-space. To use perf top you need root access. 19.2. Profiling CPU usage with perf top This procedure activates perf top and profiles CPU usage in real time. Prerequisites You have the perf user space tool installed as described in Installing perf . You have root access Procedure Start the perf top monitoring interface: The monitoring interface looks similar to the following: In this example, the kernel function do_syscall_64 is using the most CPU time. Additional resources perf-top(1) man page on your system 19.3. Interpretation of perf top output The perf top monitoring interface displays the data in several columns: The "Overhead" column Displays the percent of CPU a given function is using. The "Shared Object" column Displays name of the program or library which is using the function. The "Symbol" column Displays the function name or symbol. Functions executed in the kernel-space are identified by [k] and functions executed in the user-space are identified by [.] . 19.4. Why perf displays some function names as raw function addresses For kernel functions, perf uses the information from the /proc/kallsyms file to map the samples to their respective function names or symbols. For functions executed in the user space, however, you might see raw function addresses because the binary is stripped. The debuginfo package of the executable must be installed or, if the executable is a locally developed application, the application must be compiled with debugging information turned on (the -g option in GCC) to display the function names or symbols in such a situation. Note It is not necessary to re-run the perf record command after installing the debuginfo associated with an executable. Simply re-run the perf report command. Additional Resources Enabling debugging with debugging information 19.5. Enabling debug and source repositories A standard installation of Red Hat Enterprise Linux does not enable the debug and source repositories. These repositories contain information needed to debug the system components and measure their performance. Procedure Enable the source and debug information package channels: The USD(uname -i) part is automatically replaced with a matching value for architecture of your system: Architecture name Value 64-bit Intel and AMD x86_64 64-bit ARM aarch64 IBM POWER ppc64le 64-bit IBM Z s390x 19.6. Getting debuginfo packages for an application or library using GDB Debugging information is required to debug code. For code that is installed from a package, the GNU Debugger (GDB) automatically recognizes missing debug information, resolves the package name and provides concrete advice on how to get the package. Prerequisites The application or library you want to debug must be installed on the system. GDB and the debuginfo-install tool must be installed on the system. For details, see Setting up to debug applications . Repositories providing debuginfo and debugsource packages must be configured and enabled on the system. For details, see Enabling debug and source repositories . Procedure Start GDB attached to the application or library you want to debug. GDB automatically recognizes missing debugging information and suggests a command to run. Exit GDB: type q and confirm with Enter . Run the command suggested by GDB to install the required debuginfo packages: The dnf package management tool provides a summary of the changes, asks for confirmation and once you confirm, downloads and installs all the necessary files. In case GDB is not able to suggest the debuginfo package, follow the procedure described in Getting debuginfo packages for an application or library manually . Additional resources How can I download or install debuginfo packages for RHEL systems? (Red Hat Knowledgebase)

Commands:
perf top
Samples: 8K of event 'cycles', 2000 Hz, Event count (approx.): 4579432780 lost: 0/0 drop: 0/0 Overhead Shared Object Symbol 2.20% [kernel] [k] do_syscall_64 2.17% [kernel] [k] module_get_kallsym 1.49% [kernel] [k] copy_user_enhanced_fast_string 1.37% libpthread-2.29.so [.] pthread_mutex_lock 1.31% [unknown] [.] 0000000000000000 1.07% [kernel] [k] psi_task_change 1.04% [kernel] [k] switch_mm_irqs_off 0.94% [kernel] [k] fget 0.74% [kernel] [k] entry_SYSCALL_64 0.69% [kernel] [k] syscall_return_via_sysret 0.69% libxul.so [.] 0x000000000113f9b0 0.67% [kernel] [k] kallsyms_expand_symbol.constprop.0 0.65% firefox [.] moz_xmalloc 0.65% libpthread-2.29.so [.] __pthread_mutex_unlock_usercnt 0.60% firefox [.] free 0.60% libxul.so [.] 0x000000000241d1cd 0.60% [kernel] [k] do_sys_poll 0.58% [kernel] [k] menu_select 0.56% [kernel] [k] _raw_spin_lock_irqsave 0.55% perf [.] 0x00000000002ae0f3
subscription-manager repos --enable rhel-8-for-USD(uname -i)-baseos-debug-rpms subscription-manager repos --enable rhel-8-for-USD(uname -i)-baseos-source-rpms subscription-manager repos --enable rhel-8-for-USD(uname -i)-appstream-debug-rpms subscription-manager repos --enable rhel-8-for-USD(uname -i)-appstream-source-rpms
gdb -q /bin/ls Reading symbols from /bin/ls...Reading symbols from .gnu_debugdata for /usr/bin/ls...(no debugging symbols found)...done. (no debugging symbols found)...done. Missing separate debuginfos, use: dnf debuginfo-install coreutils-8.30-6.el8.x86_64 (gdb)
(gdb) q
dnf debuginfo-install coreutils-8.30-6.el8.x86_64