A stack trace is one of the most important pieces of information you can provide to help others debug an application crash.  This page details the importance of stack traces and outlines several methods for obtaining stack traces. 

A stack trace is one of the most important pieces of information you can provide to help others debug an application crash.  This page details the importance of stack traces and outlines several methods for obtaining stack traces. 

−

= What is a stack trace (backtrace)?  =

+

If you are experiencing a crash, the basic steps to generating a useful stack trace for common Gnome desktop applications are:

* Include the stack trace in your bug report (see [[Bugs and feature requests]] for full instructions). 

+

+

If ABRT does not start automatically, you will need to start the program in a special way, using a debugger (such as gdb).  See [[#gdb|gdb instructions below]]. 

+

+

Special instructions apply for [[JavaStackTraces|Java programs]] and [[#firefox|Firefox]]. 

+

+

Starting with Fedora 12, ABRT is replacing Bug Buddy.  See [[#ABRT|abrt]] for more info. 

+

+

== What is a stack trace (backtrace)?  ==

A stack trace (sometimes also called a backtrace) is a list of function calls that leads to some point in the program.  Debugging tools like gdb or bug-buddy can get stack traces from crashed applications so that developers can figure out what went wrong. 

A stack trace (sometimes also called a backtrace) is a list of function calls that leads to some point in the program.  Debugging tools like gdb or bug-buddy can get stack traces from crashed applications so that developers can figure out what went wrong. 

−

= What does a stack trace look like?  =

+

== What does a stack trace look like?  ==

A typical stack trace looks similar to the following:

A typical stack trace looks similar to the following:

Line 31:

Line 43:

Notice the filenames and line numbers of where the functions are called appearing. 

Notice the filenames and line numbers of where the functions are called appearing. 

−

= What are debugging symbols, and why are they important?  =

+

== What are debugging symbols, and why are they important?  ==

When a program is compiled with special switches to generate debugging symbols (the -g compiler switch) extra information is stored in the program file.  This information can be used to generate a stack trace that contains much more information, such as the exact line number of the source file where things went wrong.  Without this information it is very hard to figure out what went wrong by looking at the stack trace. 

When a program is compiled with special switches to generate debugging symbols (the -g compiler switch) extra information is stored in the program file.  This information can be used to generate a stack trace that contains much more information, such as the exact line number of the source file where things went wrong.  Without this information it is very hard to figure out what went wrong by looking at the stack trace. 

−

= What are debuginfo rpms, and how do i get them?  =

+

{{Anchor|debuginfo}}

+

== What are debuginfo rpms, and how do I get them?  ==

Fedora includes a special type of rpms called debuginfo rpms.  These automatically created rpms contains the debugging information from the program files, but moved into an external file.  All the tools that handle debugging information knows how to automatically look in these files.  This lets you easily install debugging information when you need it.  You must install the exact matching version and architecture of debuginfo as the application you are trying to debug. 

Fedora includes a special type of rpms called debuginfo rpms.  These automatically created rpms contains the debugging information from the program files, but moved into an external file.  All the tools that handle debugging information knows how to automatically look in these files.  This lets you easily install debugging information when you need it.  You must install the exact matching version and architecture of debuginfo as the application you are trying to debug. 

Line 49:

Line 62:

Each package with binaries in the distribution has a corresponding debuginfo package. 

Each package with binaries in the distribution has a corresponding debuginfo package. 

−

== Installing debuginfo RPMs using yum ==

+

=== Installing debuginfo RPMs using yum ===

debuginfo-install is a handy utility, part of <code>yum-utils</code> package that automatically enable the debuginfo repositories and download all the debuginfo packages needed.  You can do:

debuginfo-install is a handy utility, part of <code>yum-utils</code> package that automatically enable the debuginfo repositories and download all the debuginfo packages needed.  You can do:

Line 57:

Line 70:

to install all the debuginfo packages needed for package <code>foo</code>. 

to install all the debuginfo packages needed for package <code>foo</code>. 

−

== Installing debuginfo RPMs manually ==

+

To install only the minimal set of debuginfo packages use the output of <code>debuginfo-install</code> (without actually installing anything) to get the names of debuginfo packages and their respective repositories.  Then construct the install command according to the following example:

This is useful when you don't want to install debuginfo for all the dependencies of the debugged package as their debuginfo is often not required. 

+

+

=== Installing debuginfo RPMs manually ===

These packages can be downloaded from the normal fedora mirrors in the "debug" subdirectory of the architecture directory.  For example, the debuginfo packages for the latest development version is available from the official fedora ftp at [ftp://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/debug/ ftp://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/debug/] .  Please use the mirror closest to you when downloading. 

These packages can be downloaded from the normal fedora mirrors in the "debug" subdirectory of the architecture directory.  For example, the debuginfo packages for the latest development version is available from the official fedora ftp at [ftp://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/debug/ ftp://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/debug/] .  Please use the mirror closest to you when downloading. 

−

== Packagers ==

+

=== Packagers ===

If you are a packager looking information about debuginfo rpms, see [[Packaging/Debuginfo]] . 

If you are a packager looking information about debuginfo rpms, see [[Packaging/Debuginfo]] . 

−

= How do I generate a backtrace?  =

+

== How do I generate a backtrace?  ==

First make sure that you have installed the debuginfo packages for the application you are debugging and all related libraries.  A developer often tells you to install specific debuginfo packages because he can tell from a stack trace which libraries are involved in the crash.  See below for recommended packages for some common type of applications. 

First make sure that you have installed the debuginfo packages for the application you are debugging and all related libraries.  A developer often tells you to install specific debuginfo packages because he can tell from a stack trace which libraries are involved in the crash.  See below for recommended packages for some common type of applications. 

* Obtaining a stack trace from a Gnome application using both [[#bugbuddygdb| Bug Buddy and GDB]] . 

+

−

+

−

The versions using gdb are useful for all applications, while the Bug Buddy versions are very nice and simple for Gnome applications that can use it. 

+

−

= What debuginfo packages should I install?  =

+

== What debuginfo packages should I install?  ==

At the very least you will need to install the debuginfo package for the application that is crashing.  You can find out what package this application is in by typing <code>rpm -qf path-of-program</code>. 

At the very least you will need to install the debuginfo package for the application that is crashing.  You can find out what package this application is in by typing <code>rpm -qf path-of-program</code>. 

−

For certain types of programs it is also very useful to install a couple of default packages that are usefull for almost all stack traces:

+

For certain types of programs it is also very useful to install a couple of default packages that are useful for almost all stack traces:

'''NOTE:''' If you are running a Java program such as Eclipse or Tomcat, the situation is a bit more complicated - see JavaStackTraces for details. 

+

== Obtaining a stack trace using just GDB ==

+

+

'''NOTE:''' If you are running a Java program such as Eclipse or Tomcat, the situation is a bit more complicated - see [[JavaStackTraces]] for details. 

First, run the following command to start gdb:

First, run the following command to start gdb:

Line 109:

Line 127:

Once the program is running, reproduce the crash and go back to the terminal where you ran gdb.  The gdb prompt should be shown - if not, hit Control+C to break into the debugger.  At the gdb debugger prompt, type:

Once the program is running, reproduce the crash and go back to the terminal where you ran gdb.  The gdb prompt should be shown - if not, hit Control+C to break into the debugger.  At the gdb debugger prompt, type:

<pre>

<pre>

−

thread apply all bt

+

thread apply all bt full

</pre>

</pre>

Line 118:

Line 136:

{{Anchor|core}}

{{Anchor|core}}

−

= Obtaining a stack trace from a core dump =

+

== Obtaining a stack trace from a core dump ==

If the program that crashes leaves a core dump, you can use GDB to obtain a stack trace.  Core dumps are saved in a file on your hard disk and are usually named something along the lines of "core" or "core.3124".  To obtain a stack trace from one of these files, run the following command:

If the program that crashes leaves a core dump, you can use GDB to obtain a stack trace.  Core dumps are saved in a file on your hard disk and are usually named something along the lines of "core" or "core.3124".  To obtain a stack trace from one of these files, run the following command:

Line 139:

Line 157:

</pre>

</pre>

−

{{Anchor|core}}

+

{{Anchor|ABRT}}

+

== How to install ABRT ==

−

= Obtaining a stack trace from a Gnome application using Bug Buddy =

+

If you installed Fedora via a LiveCD image, ABRT should already be installed.  You should be able to start it in Applications -> System Tools -> Automatic Bug Reporting Tool.  If ABRT is not installed, for whatever reason, you can install it manually by doing the following on a command line:

−

Gnome applications use a program called bug-buddy to let you submit bug reports.  The bug reports submitted by it normally go to the upstream bugtracking system, but you can also use it to just extract a stack trace. 

+

<pre> su -c 'yum install abrt'</pre>

−

When bug-buddy is installed, and an application which uses it crashes you will get a dialog saying ''The Application "application" has quit unexpectedly''.  You can then press the ''Inform Developers'' button, which will bring up bug-buddy.  It will show a dialog where it says its collecting debug info, with a button that says ''Show Debugging Details''.  If you press this button it will show you the stack trace of the crash, and lets you copy or save the text so that you can submit it to a bug report. 

+

or go to ''System'' -> ''Administration' -> ''Add/Remove Software'' in Gnome, and type <code>abrt</code> in the search box and select ''Find''.  Select the abrt package and apply the changes. 

−

When you are done with bug-buddy you can just press the ''Cancel'' button to quit without reporting a bug. 

+

== Configuring ABRT for Bugzilla ==

−

{{Anchor|bugbuddy}}

+

Go to Application -> System Tools -> Automated Bug Reporting Tool and select it to start it manually.  Once the gui window appears, choose Edit -> Plugins and from the Settings window, scroll down, highlight Bugzilla and choose "Configure Plugin".  The Bugzilla URL should be [https://bugzilla.redhat.com https://bugzilla.redhat.com] and enter your own Bugzilla login and password in the proper boxes. ( NOTE: if you do not yet have a Bugzilla account, now is the time to get one, just go to the URL as displayed on that page and create a new account.) [ https://bugzilla.redhat.com/createaccount.cgi https://bugzilla.redhat.com/createaccount.cgi]

−

= Obtaining a stack trace from a Gnome application using both Bug Buddy and GDB =

+

Next time you have a program crash, and ABRT triggers, when you hit Report, ABRT will be able to automatically log in to Bugzilla and submit a Bug Report for you. 

−

This starts similarly to the [[#bugbuddy| normal bug-buddy approach]] , but when you get the initial crash dialog you don't do anything with that window.  Instead you open a terminal and start gdb by typing:

+

== Using ABRT ==

−

<pre>

+

−

gdb name-of-program process-number

+

(The following assumes Gnome as the desktop ... someone else will have to update for KDE/Others)

−

</pre>

+

+

If ABRT detects a crashed program, you will get an ABRT alert.  This will be visually indicated by a flashing red light in the system tray.  Left click on the alert light, and the Automatic Bug Reporting Tool should start, displaying all of the crashes it has registered.  To report the bug, right click on it and choose report.  ABRT will gather the logs it needs to submit with the bug and will then let you know it is going to submit a bug on your behalf.  If you have configured ABRT as in the previous section, it will then ask you to verify whether or not to include the various logs, then will automatically go out to Bugzilla and open a bug, attaching the logs into the bug.  It will then show you the bug number so that you can track the bug as it is worked on. 

+

+

== Configuring ABRT when missing Debuginfos ==

+

+

When you right click on a bug in ABRT and choose Report, ABRT will attempt to go out and retrieve the logs it will need to send as part of the bug report.  The developers have added code to detect whether or not symbolic traces are included within the backtrace, and if it detects that there are none, ABRT will alert you of this, and will show you the command to run.  This is the same as what is shown in the [[#debuginfo|debuginfo]] section. 

+

==Special cases==

+

{{Anchor|special}}

+

== Programs running as another user ==

+

+

If you do not get any output from gdb after typing <code>thread apply all bt</code> or <code>bt</code>, it may be because the program is run as root or as another user.  In GNOME for example, this is the case when running gnome-games.  In such cases, you will need to be root in order to capture a trace.  So, quit gdb, login as root, and then repeat the steps to obtain a stack trace. 

* Run "firefox -g" on the command line.  That will start firefox running inside of gdb debugger. 

+

* In gdb, you should see gdb prompt "(gdb)".  Issue the command "run -safe-mode".  A dialog window will pop up, disable all add-ons here and continue in safe mode. 

+

* Do whatever is necessary to make firefox crash and follow the instructions above for gdb usage. 

+

* When Firefox crashes, [[ StackTraces#gdb | obtain the backtrace ]] and attach it to bugzilla. 

+

For additional info see [[Debugging guidelines for Mozilla products]]

+

+

=== Thunderbird ===

−

Where ''name-of-program'' is the name of the program that crashed along with the full path to where it is located (for example: /usr/bin/gnome-panel ; if you don't know the full path you can use <code>which gnome-panel</code>--but be sure to include the backward quote marks), and ''process-number'' is the process number of the application that crashed, which can be found using the <code>ps</code> command. 

+

It's almost the same as for Firefox, just the debug info packages are different.  Install them by "debuginfo-install thunderbird" command as root on the console. 

−

You then continue like exactly like the [[#gdb| normal gdb approach]] after the crash. 

+

For additional info see [[Debugging guidelines for Mozilla products]]

−

Normally bug-buddy does a good job of collecting the stack trace, but in some cases the developer might want some extra information from gdb.  In that case he will give some extra commands to be typed into gdb.  In that case this method of starting gdb is very useful. 

+

===Java programs===

−

{{Anchor|bugbuddygdb}}

+

See [[JavaStackTraces]] for info on getting stack traces from programs running in Java. 

−

= Obtaining stack traces for a program started by a daemon, or a daemon =

+

=== Daemons and their spawn ===

You will need to gather the backtrace from the core file. 

You will need to gather the backtrace from the core file. 

Line 192:

Line 233:

Note: you can test whether dumping a core file would work by running <code>kill -SEGV 1234</code> where 1234 is the PID of the program you're testing. 

Note: you can test whether dumping a core file would work by running <code>kill -SEGV 1234</code> where 1234 is the PID of the program you're testing. 

−

{{Anchor|firefox}}

+

==Other tools==

−

= Obtaining stack traces from Firefox =

+

−

* Run "firefox -g" on the command line.  That will start firefox running inside of gdb debugger. 

+

=== Valgrind ===

−

* In gdb, issue the command "run". 

+

−

* Do whatever is necessary to make firefox crash and follow the instructions above for gdb usage. 

+

−

{{Anchor|special}}

+

The brilliant tool [http://valgrind.org/ valgrind] is often able to say more about what is going wrong; it can give a strack trace to the point where things start to go wrong, which might be long time before the program actually crashes.  Programs run through valgrind will run an order of magnitude slower and use more memory, but it will be buddyamazingly usable. 

−

= Obtaining stack traces under special circumstances =

+

With valgrind installed ("yum install valgrind") you can use it on a program:

−

+

−

If you do not get any output from gdb after typing <code>thread apply all bt</code> or <code>bt</code>, it may be because the program is run as root or as another user.  In GNOME for example, this is the case when running gnome-games.  In such cases, you will need to be root in order to capture a trace.  So, quit gdb, login as root, and then repeat the steps to obtain a stack trace. 

+

−

+

−

+

−

----

+

−

+

−

Note: much of the text from this page comes from [http://bugzilla.gnome.org/getting-traces.cgi the excellent Gnome bugsquad on getting stack traces] . 

+

−

+

−

= References =

+

−

+

−

*http://linux.bytesex.org/gdb.html

+

−

+

−

= Valgrind =

+

−

+

−

The brilliant tool [http://valgrind.org/ valgrind] is often able to say more about what is going wrong; it can give a strack trace to the point where things start to go wrong, which might be long time before the program actually crashes.  Programs run through valgrind will run an order of magnitude slower and use more memory, but it will be amazingly usable. 

+

−

+

−

With valgrind installed you can use it on a program:

+

<pre>

<pre>

valgrind name-of-program program-arguments

valgrind name-of-program program-arguments

Line 225:

Line 246:

With debuginfo installed stacktraces will use symbolic names.  See [http://valgrind.org/ valgrind.org] for more info and tips and tricks. 

With debuginfo installed stacktraces will use symbolic names.  See [http://valgrind.org/ valgrind.org] for more info and tips and tricks. 

+

=== strace ===

+

+

strace can track all system calls made by a program, which can also be helpful in debugging, though it cannot produce stack traces.  Install with "yum install strace", and see "man strace" for details. 

+

+

== References ==

+

* Much of the text from this page comes from the excellent [http://live.gnome.org/GettingTraces GNOME bugsquad on getting stack traces]. 

+

* http://linux.bytesex.org/gdb.html

−

----

[[Category:Bugs]] [[Category:BugTriage]] [[Category:Debugging]]

[[Category:Bugs]] [[Category:BugTriage]] [[Category:Debugging]]

Revision as of 22:02, 15 November 2012

A stack trace is one of the most important pieces of information you can provide to help others debug an application crash.  This page details the importance of stack traces and outlines several methods for obtaining stack traces. 

If you are experiencing a crash, the basic steps to generating a useful stack trace for common Gnome desktop applications are:

What is a stack trace (backtrace)? 

A stack trace (sometimes also called a backtrace) is a list of function calls that leads to some point in the program.  Debugging tools like gdb or bug-buddy can get stack traces from crashed applications so that developers can figure out what went wrong. 

Notice the filenames and line numbers of where the functions are called appearing. 

What are debugging symbols, and why are they important? 

When a program is compiled with special switches to generate debugging symbols (the -g compiler switch) extra information is stored in the program file.  This information can be used to generate a stack trace that contains much more information, such as the exact line number of the source file where things went wrong.  Without this information it is very hard to figure out what went wrong by looking at the stack trace. 

What are debuginfo rpms, and how do I get them? 

Fedora includes a special type of rpms called debuginfo rpms.  These automatically created rpms contains the debugging information from the program files, but moved into an external file.  All the tools that handle debugging information knows how to automatically look in these files.  This lets you easily install debugging information when you need it.  You must install the exact matching version and architecture of debuginfo as the application you are trying to debug. 

Each package with binaries in the distribution has a corresponding debuginfo package. 

Installing debuginfo RPMs using yum

debuginfo-install is a handy utility, part of yum-utils package that automatically enable the debuginfo repositories and download all the debuginfo packages needed.  You can do:

$ debuginfo-install foo

to install all the debuginfo packages needed for package foo. 

To install only the minimal set of debuginfo packages use the output of debuginfo-install (without actually installing anything) to get the names of debuginfo packages and their respective repositories.  Then construct the install command according to the following example:

Packagers

How do I generate a backtrace? 

First make sure that you have installed the debuginfo packages for the application you are debugging and all related libraries.  A developer often tells you to install specific debuginfo packages because he can tell from a stack trace which libraries are involved in the crash.  See below for recommended packages for some common type of applications. 

Obtaining a stack trace using just GDB

NOTE: If you are running a Java program such as Eclipse or Tomcat, the situation is a bit more complicated - see JavaStackTraces for details. 

First, run the following command to start gdb:

gdb name-of-program

Where name-of-program is the name of the program that crashed (for example: /usr/bin/gnome-panel). 

Then, at the gdb prompt, type:

run

If you need to give any arguments to the program, give them after the run command, like:

run --argument

Once the program is running, reproduce the crash and go back to the terminal where you ran gdb.  The gdb prompt should be shown - if not, hit Control+C to break into the debugger.  At the gdb debugger prompt, type:

thread apply all bt full

If that does not work (meaning that you don't get any output--this may be the case in programs which are not multi-threaded), type bt instead.  If you still do not have any output, read this note about obtaining a stack trace under special circumstances.  The output is the stack trace.  Cut and paste all of it into a text file. 

You can quit gdb by typing quit. 

Obtaining a stack trace from a core dump

If the program that crashes leaves a core dump, you can use GDB to obtain a stack trace.  Core dumps are saved in a file on your hard disk and are usually named something along the lines of "core" or "core.3124".  To obtain a stack trace from one of these files, run the following command:

gdb name-of-program core-file-name

Where name-of-program is the name of the program that crashed (for example: /usr/bin/gnome-panel), and core-file-name is the name of the core file that contains the core dump (for example: core.7812). 

Then, at the gdb prompt, type:

thread apply all bt full

If that does not work (meaning that you don't get any output--this may be the case in programs which are not multi-threaded), type bt instead.  If you still do not have any output, read this note about obtaining a stack trace under special circumstances.  The output is the stack trace.  Cut and paste all of it into a text file. 

You can quit gdb by typing quit. 

Note that creation of core files is disabled in Fedora Core by default (in /etc/profile).  To enable them for a shell session, type at the shell prompt:

ulimit -c unlimited

How to install ABRT

If you installed Fedora via a LiveCD image, ABRT should already be installed.  You should be able to start it in Applications -> System Tools -> Automatic Bug Reporting Tool.  If ABRT is not installed, for whatever reason, you can install it manually by doing the following on a command line:

su -c 'yum install abrt'

or go to System -> Administration' -> Add/Remove Software in Gnome, and type abrt in the search box and select Find.  Select the abrt package and apply the changes. 

Configuring ABRT for Bugzilla

Go to Application -> System Tools -> Automated Bug Reporting Tool and select it to start it manually.  Once the gui window appears, choose Edit -> Plugins and from the Settings window, scroll down, highlight Bugzilla and choose "Configure Plugin".  The Bugzilla URL should be https://bugzilla.redhat.com and enter your own Bugzilla login and password in the proper boxes. ( NOTE: if you do not yet have a Bugzilla account, now is the time to get one, just go to the URL as displayed on that page and create a new account.)  https://bugzilla.redhat.com/createaccount.cgi

Next time you have a program crash, and ABRT triggers, when you hit Report, ABRT will be able to automatically log in to Bugzilla and submit a Bug Report for you. 

Using ABRT

(The following assumes Gnome as the desktop ... someone else will have to update for KDE/Others)

If ABRT detects a crashed program, you will get an ABRT alert.  This will be visually indicated by a flashing red light in the system tray.  Left click on the alert light, and the Automatic Bug Reporting Tool should start, displaying all of the crashes it has registered.  To report the bug, right click on it and choose report.  ABRT will gather the logs it needs to submit with the bug and will then let you know it is going to submit a bug on your behalf.  If you have configured ABRT as in the previous section, it will then ask you to verify whether or not to include the various logs, then will automatically go out to Bugzilla and open a bug, attaching the logs into the bug.  It will then show you the bug number so that you can track the bug as it is worked on. 

Configuring ABRT when missing Debuginfos

When you right click on a bug in ABRT and choose Report, ABRT will attempt to go out and retrieve the logs it will need to send as part of the bug report.  The developers have added code to detect whether or not symbolic traces are included within the backtrace, and if it detects that there are none, ABRT will alert you of this, and will show you the command to run.  This is the same as what is shown in the debuginfo section. 

Special cases

Programs running as another user

If you do not get any output from gdb after typing thread apply all bt or bt, it may be because the program is run as root or as another user.  In GNOME for example, this is the case when running gnome-games.  In such cases, you will need to be root in order to capture a trace.  So, quit gdb, login as root, and then repeat the steps to obtain a stack trace. 

Java programs

See JavaStackTraces for info on getting stack traces from programs running in Java. 

Daemons and their spawn

You will need to gather the backtrace from the core file. 

Make sure your daemon's initscript isn't forbidding dumping core files to the disk.  Add the line DAEMON_COREFILE_LIMIT=unlimited to its configuration file in /etc/sysconfig.  For example, the Bluetooth daemon (hcid) uses /etc/sysconfig/bluetooth. 

Then setup the kernel so that the core dump is written to a known location such as /tmp/. As root, run:

Finally, after reproducing your problem, you can double-check which binary created the core file with file /tmp/core.1234.  Then run gdb on the file to create a post-mortem stack trace:

gdb /path/to/binary/file /tmp/core.1234

and follow the instructions above for gdb usage. 

Note: you can test whether dumping a core file would work by running kill -SEGV 1234 where 1234 is the PID of the program you're testing. 

Other tools

Valgrind

The brilliant tool valgrind is often able to say more about what is going wrong; it can give a strack trace to the point where things start to go wrong, which might be long time before the program actually crashes.  Programs run through valgrind will run an order of magnitude slower and use more memory, but it will be buddyamazingly usable. 

With valgrind installed ("yum install valgrind") you can use it on a program:

valgrind name-of-program program-arguments

With debuginfo installed stacktraces will use symbolic names.  See valgrind.org for more info and tips and tricks. 

strace

strace can track all system calls made by a program, which can also be helpful in debugging, though it cannot produce stack traces.  Install with "yum install strace", and see "man strace" for details.