Chapter 20. Debugging a Running Application This chapter will introduce the techniques for debugging an application which can be executed as many times as needed, on a machine directly accessible to the developer. 20.1. Enabling Debugging with Debugging Information To debug applications and libraries, debugging information is required. The following sections describe how to obtain this information. 20.1.1. Debugging Information While debugging any executable code, two kinds of information allow the tools and by extension the programmer to comprehend the binary code: The source code text A description of how the source code text relates to the binary code This is referred to as debugging information. Red Hat Enterprise Linux uses the ELF format for executable binaries, shared libraries, or debuginfo files. Within these ELF files, the DWARF format is used to hold the debug information. DWARF symbols are read by the readelf -w file command. Caution STABS is occasionally used with UNIX. STABS is an older, less capable format. Its use is discouraged by Red Hat. GCC and GDB support the STABS production and consumption on a best effort basis only. Some other tools such as Valgrind and elfutils do not support STABS at all. Additional Resources The DWARF Debugging Standard 20.1.2. Enabling Debugging of C and C++ Applications with GCC Because debugging information is large, it is not included in executable files by default. To enable debugging of your C and C++ applications with it, you must explicitly instruct the compiler to create debugging information. Enabling the Creation of Debugging Information with GCC To enable the creation of debugging information with GCC when compiling and linking code, use the -g option: Optimizations performed by the compiler and linker can result in executable code which is hard to relate to the original source code: variables may be optimized out, loops unrolled, operations merged into the surrounding ones etc. This affects debugging negatively. For an improved debugging experience, consider setting the optimization with the -Og option. However, changing the optimization level changes the executable code and may change the actual behaviour so as to remove some bugs. The -fcompare-debug GCC option tests code compiled by GCC with debug information and without debug information. The test passes if the resulting two binary files are identical. This test ensures that executable code is not affected by any debugging options, which further ensures that there are no hidden bugs in the debug code. Note that using the -fcompare-debug option significantly increases compilation time. See the GCC manual page for details about this option. Additional Resources Section 20.1, "Enabling Debugging with Debugging Information" Using the GNU Compiler Collection (GCC) - 3.10 Options for Debugging Your Program Debugging with GDB - 18.3 Debugging Information in Separate Files The GCC manual page: 20.1.3. Debuginfo Packages Debuginfo packages contain debugging information and debug source code for programs and libraries. Prerequisites Understanding of debugging information Debuginfo Packages For applications and libraries installed in packages from the Red Hat Enterprise Linux repositories, you can obtain the debugging information and debug source code as separate debuginfo packages available through another channel. The debuginfo packages contain .debug files, which contain DWARF debuginfo and the source files used for compiling the binary packages. Debuginfo package contents are installed to the /usr/lib/debug directory. A debuginfo package provides debugging information valid only for a binary package with the same name, version, release, and architecture: Binary package: packagename - version - release . architecture .rpm Debuginfo package: packagename -debuginfo- version - release . architecture .rpm 20.1.4. Getting debuginfo Packages for an Application or Library using GDB The GNU Debugger (GDB) automatically recognizes missing debug information and resolves the package name. Prerequisites The application or library you want to debug is installed on the system GDB is installed on the system The debuginfo-install tool is installed on the system Procedure Start GDB attached to the application or library you want to debug. GDB automatically recognizes missing debugging information and suggests a command to run. Exit GDB without proceeding further: type q and Enter . Run the command suggested by GDB to install the needed debuginfo packages: Installing a debuginfo package for an application or library installs debuginfo packages for all dependencies, too. In case GDB is not able to suggest the debuginfo package, follow the procedure in Section 20.1.5, "Getting debuginfo Packages for an Application or Library Manually" . Additional Resources Red Hat Developer Toolset User Guide - Installing Debugging Information Red Hat Knowledgebase solution - How can I download or install debuginfo packages for RHEL systems? 20.1.5. Getting debuginfo Packages for an Application or Library Manually To manually choose (which) debuginfo packages (to install) for installation, locate the executable file and find the package which installs it. Note The use of GDB to determine the packages for installation is preferable. Use this manual procedure only if GDB is not able to suggest the package to install. Prerequisites The application or library must be installed on the system The debuginfo -install tool must be available on the system Procedure Find the executable file of the application or library. Use the which command to find the application file. Use the locate command to find the library file. If the original reasons for debugging included error messages, pick the result where the library has the same additional numbers in its file name. If in doubt, try following the rest of the procedure with the result where the library file name includes no additional numbers. Note The locate command is provided by the mlocate package. To install it and enable its use: Using the file path, search for a package which provides that file. The output provides a list of packages in the format name - version . distribution . architecture . In this step, only the package name is important, because the version shown in yum output may not be the actual installed version. Important If this step does not produce any results, it is not possible to determine which package provided the binary file and this procedure fails. Use the rpm low-level package management tool to find what package version is installed on the system. Use the package name as an argument: The output provides details for the installed package in the format name - version . distribution . architecture . Install the debuginfo packages using the debuginfo-install utility. In the command, use the package name and other details you determined during the step: Installing a debuginfo package for an application or library installs debuginfo packages for all dependencies, too. Additional Resources Red Hat Developer Toolset User Guide - Installing Debugging Information Knowledgebase article - How can I download or install debuginfo packages for RHEL systems? 20.2. Inspecting the Application's Internal State with GDB To find why an application does not work properly, control its execution and examine its internal state with a debugger. This section describes how to use the GNU Debugger (GDB) for this task. 20.2.1. GNU Debugger (GDB) A debugger is a tool that enables control of code execution and inspection of the state of the code. This capability is used to investigate what is happening in a program and why. Red Hat Enterprise Linux contains the GNU debugger (GDB) which offers this functionality through a command line user interface. For a graphical frontend to GDB, install the Eclipse integrated development environment. See Using Eclipse . GDB Capabilities A single GDB session can debug: multithreaded and forking programs multiple programs at once programs on remote machines or in containers with the gdbserver utility connected over a TCP/IP network connection Debugging Requirements To debug any executable code, GDB requires the respective debugging information: For programs developed by you, you can create the debugging information while building the code. For system programs installed from packages, their respective debuginfo packages must be installed. 20.2.2. Attaching GDB to a Process In order to examine a process, GDB must be attached to the process. Prerequisites GDB must be installed on the system Starting a Program with GDB When the program is not running as a process, start it with GDB: Replace program with a file name or path to the program. GDB starts execution of the program. You can set up breakpoints and the gdb environment before beginning the execution of the process with the run command. Attaching GDB to an Already Running Process To attach GDB to a program already running as a process: Find the process id ( pid ) with the ps command: Replace program with a file name or path to the program. Attach GDB to this process: Replace program with a file name or path to the program, replace pid with an actual process id number from the ps output. Attaching an Already Running GDB to an Already Running Process To attach an already running GDB to an already running program: Use the shell GDB command to run the ps command and find the program's process id ( pid ): Replace program with a file name or path to the program. Use the attach command to attach GDB to the program: Replace pid by an actual process id number from the ps output. Note In some cases, GDB might not be able to find the respective executable file. Use the file command to specify the path: Additional Resources Debugging with GDB - 2.1 Invoking GDB Debugging with GDB - 4.7 Debugging an Already-running Process 20.2.3. Stepping through Program Code with GDB Once the GDB debugger is attached to a program, you can use a number of commands to control the execution of the program. Prerequisites GDB must be installed on the system You must have the required debugging information available: The program is compiled and built with debugging information, or The relevant debuginfo packages are installed GDB is attached to the program that is to be debugged GDB Commands to Step Through the Code r (run) Start the execution of the program. If run is executed with arguments, those arguments are passed on to the executable as if the program was started normally. Users normally issue this command after setting breakpoints. start Start the execution of the program and stop at the beginning of the main function. If start is executed with arguments, those arguments are passed on to the executable as if the program was started normally. c (continue) Continue the execution of the program from the current state. The execution of the program will continue until one of the following becomes true: A breakpoint is reached A specified condition is satisfied A signal is received by the program An error occurs The program terminates n () Another commonly known name of this command is step over . Continue the execution of the program from the current state, until the line of code in the current source file is reached. The execution of the program will continue until one of the following becomes true: A breakpoint is reached A specified condition is satisfied A signal is received by the program An error occurs The program terminates s (step) Another commonly known name of this command is step into . The step command halts execution at each sequential line of code in the current source file. However, if the execution is currently stopped at a source line containing a function call , GDB stops the execution after entering the function call (rather than executing it). until location Continue the execution until the code location specified by the location option is reached. fini (finish) Resume the execution of the program and halt when the execution returns from a function. The execution of the program will continue until one of the following becomes true: A breakpoint is reached A specified condition is satisfied A signal is received by the program An error occurs The program terminates q (quit) Terminate the execution and exit GDB. Additional Resources Section 20.2.5, "Using GDB Breakpoints to Stop Execution at Defined Code Locations" Debugging with GDB - 4.2 Starting your Program Debugging with GDB - 5.2 Continuing and Stepping 20.2.4. Showing Program Internal Values with GDB Displaying the values of the internal variables of a program is important for understanding what the program is doing. GDB offers multiple commands that you can use to inspect the internal variables. This section describes the most useful of these commands. Prerequisites Understanding of the GDB debugger GDB Commands to Display the Internal State of a Program p (print) Displays the value of the argument given. Usually, the argument is the name of a variable of any complexity, from a simple single value to a structure. An argument can also be an expression valid in the current language, including the use of program variables and library functions, or functions defined in the program being tested. It is possible to extend GDB with pretty-printer Python or Guile scripts for customized display of data structures (such as classes, structs) using the print command. bt (backtrace) Display the chain of function calls used to reach the current execution point, or the chain of functions used up until execution was signalled. This is useful for investigating serious bugs (such as segmentation faults) with elusive causes. Adding the full option to the backtrace command displays local variables, too. It is possible to extend GDB with frame filter Python scripts for customized display of data displayed using the bt and info frame commands. The term frame refers to the data associated with a single function call. info The info command is a generic command to provide information about various items. It takes an option specifying the item. The info args command displays arguments of the function call that is the currently selected frame. The info locals command displays local variables in the currently selected frame. For a list of the possible items, run the command help info in a GDB session: l (list) Show the line in the source code where the program stopped. This command is available only when the program execution is stopped. While not strictly a command to show the internal state, list helps the user understand what changes to the internal state will happen in the step of the program's execution. Additional Resources Red Hat Developer blog entry - The GDB Python API Debugging with GDB - 10.9 Pretty Printing 20.2.5. Using GDB Breakpoints to Stop Execution at Defined Code Locations In many cases, it is advantageous to let the program execute until a certain line of code is reached. Prerequisites Understanding of GDB Using Breakpoints in GDB Breakpoints are markers that tell GDB to stop the execution of a program. Breakpoints are most commonly associated with source code lines: Placing a breakpoint requires specifying the source file and line number. To place a breakpoint : Specify the name of the source code file and the line in that file: When file is not present, the name of the source file at the current point of execution is used: Alternatively, use a function name to place the breakpoint: A program might encounter an error after a certain number of iterations of a task. To specify an additional condition to halt execution: Replace condition with a condition in the C or C++ language. The meaning of file and line is the same as above. To inspect the status of all breakpoints and watchpoints: To remove a breakpoint by using its number as displayed in the output of info br : To remove a breakpoint at a given location: 20.2.6. Using GDB Watchpoints to Stop Execution on Data Access and Changes In many cases, it is advantageous to let the program execute until certain data changes or is accessed. This section lists the most common watchpoints. Prerequisites Understanding of GDB Using Watchpoints in GDB Watchpoints are markers which tell GDB to stop the execution of a program. Watchpoints are associated with data: Placing a watchpoint requires specifying an expression describing a variable, multiple variables, or a memory address. To place a watchpoint for data change (write): Replace expression with an expression that describes what you want to watch. For variables, expression is equal to the name of the variable. To place a watchpoint for data access (read): To place a watchpoint for any data access (both read and write): To inspect the status of all watchpoints and breakpoints: To remove a watchpoint: Replace the num option with the number reported by the info br command. 20.2.7. Debugging Forking or Threaded Programs with GDB Some programs use forking or threads to achieve parallel code execution. Debugging multiple simultaneous execution paths requires special considerations. Prerequisites Understanding of the GDB debugger Understanding of the concepts of process forking and threads Debugging Forked Programs with GDB Forking is a situation when a program ( parent ) creates an independent copy of itself ( child ). Use the following settings and commands to affect the reaction of GDB to an occuring fork: The follow-fork-mode setting controls whether GDB follows the parent or the child after the fork. set follow-fork-mode parent After forking, debug the parent process. This is the default. set follow-fork-mode child After forking, debug the child process. show follow-fork-mode Displays the current setting of the follow-fork-mode . The set detach-on-fork setting controls whether the GDB keeps control of the other (not followed) process or leaves it to run. set detach-on-fork on The process which is not followed (depending on the value of the follow-fork-mode ) is detached and runs independently. This is the default. set detach-on-fork off GDB keeps control of both processes. The process which is followed (depending on the value of follow-fork-mode ) is debugged as usual, while the other is suspended. show detach-on-fork Displays the current setting of detach-on-fork . Debugging Threaded Programs with GDB GDB has the ability to debug individual threads, and to manipulate and examine them independently. To make GDB stop only the thread that is examined, use the commands set non-stop on and set target-async on . You can add these commands to the .gdbinit file. After that functionality is turned on, GDB is ready to conduct thread debugging. GDB uses the concept of current thread . By default, commands apply to the current thread only. info threads Displays a list of threads with their id and gid numbers, indicating the current thread. thread id Sets the thread with the specified id as the current thread. thread apply ids command Applies the command command to all threads listed by ids . The ids option is a space-separated list of thread ids. The special value all applies the command to all threads. break location thread id if condition Sets a breakpoint at a certain location with a certain condition only for the thread number id . watch expression thread id Sets a watchpoint defined by expression only for the thread number id . command& Executes the command command and returns immediately to the GDB prompt (gdb) , continuing code execution in the background. interrupt Halts execution in the background. Additional Resources Debugging with GDB - 4.10 Debugging Programs with Multiple Threads Debugging with GDB - 4.11 Debugging Forks 20.3. Recording Application Interactions The executable code of applications interacts with the code of the operating system and shared libraries. Recording an activity log of these interactions can provide enough insight into the application's behavior without debugging the actual application code. Alternatively, analyzing an application's interactions can help to pinpoint the conditions in which a bug manifests. 20.3.1. Useful Tools for Recording Application Interactions Red Hat Enterprise Linux offers multiple tools for analyzing an application's interactions. strace The strace tool enables the tracing of (and tampering with) interactions between an application and the Linux kernel: system calls, signal deliveries, and changes of process state. The strace output is detailed and explains the calls well, because strace interprets parameters and results with knowledge of the underlying kernel code. Numbers are turned into the respective constant names, bitwise combined flags expanded to flag lists, pointers to character arrays dereferenced to provide the actual string, and more. Support for more recent kernel features may be lacking, however. The use of strace does not require any particular setup except for setting up the log filter. Tracing the application code with strace may result in significant slowdown of the application's execution. As a result, strace is not suitable for many production deployments. As an alternative, consider using SystemTap in such cases. You can limit the list of traced system calls and signals to reduce the amount of captured data. strace captures only kernel-userspace interactions and does not trace library calls, for example. Consider using ltrace for tracing library calls. ltrace The ltrace tool enables logging of an application's user space calls into shared objects (dynamic libraries). ltrace enables tracing calls to any library. You can filter the traced calls to reduce the amount of captured data. The use of ltrace does not require any particular setup except for setting up the log filter. ltrace is lightweight and fast, offering an alternative to strace : it is possible to trace the respective interfaces in libraries such as glibc with ltrace instead of tracing kernel functions with strace . Note however that ltrace may be less precise at syscall tracing. ltrace is able to decode parameters only for a limited set of library calls: the calls whose prototypes are defined in the relevant configuration files. As part of the ltrace package, prototypes for some libacl , libc , and libm calls and system calls are provided. The ltrace output mostly contains only raw numbers and pointers. The interpretation of ltrace output usually requires consulting the actual interface declarations of the libraries present in the output. SystemTap SystemTap is an instrumentation platform for probing running processes and kernel activity on the Linux system. SystemTap uses its own scripting language for programming custom event handlers. Compared to using strace and ltrace , scripting the logging means more work in the initial setup phase. However, the scripting capabilities extend SystemTap's usefulness beyond just producing logs. SystemTap works by creating and inserting a kernel module. The use of SystemTap is efficient and does not create a significant slowdown of the system or application execution on its own. SystemTap comes with a set of usage examples. GDB The GNU Debugger is primarily meant for debugging, not logging. However, some of its features make it useful even in the scenario where an application's interaction is the primary activity of interest. With GDB, it is possible to conveniently combine the capture of an interaction event with immediate debugging of the subsequent execution path. GDB is best suited for analyzing responses to infrequent or singular events, after the initial identification of problematic situations by other tools. Using GDB in any scenario with frequent events becomes inefficient or even impossible. Additional Resources Red Hat Enterprise Linux SystemTap Beginners Guide Red Hat Developer Toolset User Guide 20.3.2. Monitoring an Application's System Calls with strace The strace tool enables tracing of (and optional tampering with) interactions between an application and the Linux kernel: system calls, signal deliveries, and changes of process state. Prerequisites strace is installed on the system To install strace , run as root: Procedure Note that the tracing specification syntax of strace offers regular expressions and syscall classes to help with the identification of system calls. Run or attach to the process you wish to monitor. If the program you want to monitor is not running, start strace and specify the program : Options used in the example above are not mandatory. Use when needed: The -f option is an acronym for "follow forks". This option traces children created by the fork, vfork, and clone system calls. The -v or -e abbrev=none option disables abbreviation of output, omitting various structure fields. The -tt option is a variant of the -t option that prefixes each line with an absolute timestamp. With the -tt option, the time printed includes microseconds. The -T option prints the amount of time spent in each system call at the end of the line. The -yy option is a variant of the -y option that enables printing of paths associated with file descriptor numbers. The -yy option prints not only paths, but also protocol-specific information associated with socket file descriptors and block or character device number associated with device file descriptors. The -s option controls the maximum string size to be printed. Note that filenames are not considered strings and are always printed in full. -e trace controls the set of system calls to trace. Replace call with a comma-separated list of system calls to be displayed. If call is left, strace will display all system calls. Shorthands for some groups of system calls are provided in the strace(1) manual page. If the program is already running, find its process ID ( pid ) and attach strace to it: If you do not wish to trace any forked processes or threads, do not use the -f option. strace displays the system calls made by the application and their details. In most cases, an application and its libraries make a large number of calls and strace output appears immediately, if no filter for system calls is set. strace exits when all the traced processes exit. To terminate the monitoring before the traced program exits, press Ctrl+C . If strace started the program, it will send the terminating signal (SIGINT, in this case) to the program being started. Note, however, that program, in turn, may ignore that signal. If you attached strace to an already running program, the program terminates together with strace . Analyze the list of system calls done by the application. Problems with resource access or availability are present in the log as calls returning errors. Values passed to the system calls and patterns of call sequences provide insight into the causes of the application's behaviour. If the application crashes, the important information is probably at the end of the log. The output contains a lot of extra information. However, you can construct a more precise filter and repeat the procedure. Notes It is advantageous to both see the output and save it to a file. To do this, run the tee command: To see separate output that corresponds to different processes, run: Output for a process with the process ID ( pid ) will be stored in your_log_file.pid . Additional Resources The strace(1) manual page. Knowledgebase article - How do I use strace to trace system calls made by a command? Red Hat Developer Toolset User Guide - strace 20.3.3. Monitoring the Application's Library Function Calls with ltrace The ltrace tool enables monitoring of the calls done by an application to functions available in libraries (shared objects). Prerequisites ltrace is installed on the system Procedure Identify the libraries and functions of interest, if possible. If the program you want to monitor is not running, start ltrace and specify program : Use the options -e and -l to filter the output: Supply the function names to be displayed as function . The -e function option can be used multiple times. If left out, ltrace will display calls to all functions. Instead of specifying functions, you can specify whole libraries with the -l library option. This option behaves similarly to the -e function option. See the ltrace (1)_ manual page for more information. If the program is already running, find its process id ( pid ) and attach ltrace to it: If you do not wish to trace any forked processes or threads, leave out the -f option. ltrace displays the library calls made by the application. In most cases, an application will make a large number of calls and ltrace output will appear immediately if no filter is set. ltrace exits when the program exits. To terminate the monitoring before the traced program exits, press ctrl+C . If ltrace started the program, the program terminates together with ltrace . If you attached ltrace to an already running program, the program terminates together with ltrace . Analyze the list of library calls done by the application. If the application crashes, the important information is probably at the end of the log. The output contains a lot of unnecessary information. However, you can construct a more precise filter and repeat the procedure. Note It is advantageous to both see the output and save it to a file. Use the tee command to achieve this: Additional Resources The strace(1) manual page Red Hat Developer Toolset User Guide - ltrace 20.3.4. Monitoring the Application's System Calls with SystemTap The SystemTap tool enables registering custom event handlers for kernel events. In comparison with strace , it is harder to use, but SystemTap is more efficient and enables more complicated processing logic. Prerequisites SystemTap is installed on the system Procedure Create a file my_script.stp with the contents: Find the process ID ( pid ) of the process you wish to monitor: Run SystemTap with the script: The value of pid is the process id. The script is compiled to a kernel module which is then loaded. This introduces a slight delay between entering the command and getting the output. When the process performs a system call, the call name and its parameters are printed to the terminal. The script exits when the process terminates, or when you press Ctrl+C . Additional Resources SystemTap Beginners Guide SystemTap Tapset Reference A larger SystemTap script, which approximates strace functionality, is available as /usr/share/systemtap/examples/process/strace.stp . To run the script: # stap --example strace.stp -x pid or # stap --example strace.stp -c "cmd args ... " 20.3.5. Using GDB to Intercept Application System Calls GDB enables stopping the execution in various kinds of situations arising during the execution of a program. To stop the execution when the program performs a system call, use a GDB catchpoint . Prerequisites Understanding of GDB breakpoints GDB is attached to the program Stopping Program Execution on System Calls with GDB Set the catchpoint: The command catch syscall sets a special type of breakpoint that halts execution when a system call is performed by the program. The syscall-name option specifies the name of the call. You can specify multiple catchpoints for various system calls. Leaving out the syscall-name option causes GDB to stop on any system call. If the program has not started execution, start it: If the program execution is only halted, resume it: GDB halts the execution after any specified system call is performed by the program. Additional Resources Section 20.2.4, "Showing Program Internal Values with GDB" Section 20.2.3, "Stepping through Program Code with GDB" Debugging with GDB - 5.1.3 Setting Catchpoints 20.3.6. Using GDB to Intercept the Handling of Signals by Applications GDB enables stopping the execution in various kinds of situations arising during the execution of a program. To stop the execution when the program receives a signal from the operating system, use a GDB catchpoint . Prerequisites Understanding of GDB breakpoints GDB is attached to the program Stopping the Program Execution on Receiving a Signal with GDB Set the catchpoint: The command catch signal sets a special type of breakpoint that halts execution when a signal is received by the program. The signal-type option specifies the type of the signal. Use the special value 'all' to catch all the signals. If the program has not started execution, start it: If the program execution is only halted, resume it: GDB halts the execution after the program receives any specified signal. Additional Resources Section 20.2.4, "Showing Program Internal Values with GDB" Section 20.2.3, "Stepping through Program Code with GDB" Debugging With GDB - 5.3.1 Setting Catchpoints

Commands:
gcc ... -g
man gcc
gdb -q /bin/ls Reading symbols from /usr/bin/ls...Reading symbols from /usr/bin/ls...(no debugging symbols found)...done. (no debugging symbols found)...done. Missing separate debuginfos, use: debuginfo-install coreutils-8.22-21.el7.x86_64 (gdb)
(gdb) q
debuginfo-install coreutils-8.22-21.el7.x86_64
which nautilus /usr/bin/nautilus
locate libz | grep so /usr/lib64/libz.so /usr/lib64/libz.so.1 /usr/lib64/libz.so.1.2.7
yum install mlocate updatedb
yum provides /usr/lib64/libz.so.1.2.7 Loaded plugins: product-id, search-disabled-repos, subscription-manager zlib-1.2.7-17.el7.x86_64 : The compression and decompression library Repo : @anaconda/7.4 Matched from: Filename : /usr/lib64/libz.so.1.2.7
rpm -q zlib zlib-1.2.7-17.el7.x86_64
debuginfo-install zlib-1.2.7-17.el7.x86_64
gdb program
ps -C program -o pid h pid
gdb program -p pid
(gdb) shell ps -C program -o pid h pid
(gdb) attach pid
(gdb) file path/to/program
(gdb) help info
(gdb) br file:line
(gdb) br line
(gdb) br function_name
(gdb) br file:line if condition
(gdb) info br
(gdb) delete number
(gdb) clear file:line
(gdb) watch expression
(gdb) rwatch expression
(gdb) awatch expression
(gdb) info br
(gdb) delete num
yum install strace
strace -fvttTyy -s 256 -e trace= call program
ps -C program (...) strace -fvttTyy -s 256 -e trace= call -p pid
strace ...-o |tee your_log_file.log >&2
strace ... -ff -o your_log_file
ltrace -f -l library -e function program
ps -C program (...) ltrace ... -p pid
ltrace ... |& tee your_log_file.log
probe begin { printf("waiting for syscalls of process %d \n", target()) } probe syscall.* { if (pid() == target()) printf("%s(%s)\n", name, argstr) } probe process.end { if (pid() == target()) exit() }
ps -aux
stap my_script.stp -x pid
(gdb) catch syscall syscall-name
(gdb) r
(gdb) c
(gdb) catch signal signal-type
(gdb) r
(gdb) c