GNU gprofng
The next generation profiling tool for Linuxversion 2.43 (last updated 20 July 2024)

Ruud van der Pas

This document is the manual for gprofng, last updated 20 July 2024.
Copyright c 2022-2024 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any
later version published by the Free Software Foundation; with no Invariant
Sections, with no Front-Cover texts, and with no Back-Cover Texts. A copyof the license is included in the section entitled “GNU Free Documentation
License.”i

Table of Contents
1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2

A Brief Overview of gprofng . . . . . . . . . . . . . . . . . . . 3
2.1
2.2
2.3

3

Main Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Sampling versus Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Steps Needed to Create a Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

A Mini Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1

Getting Started. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1.1 The Example Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1.2 A First Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1.3 The Source Code View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.1.4 The Disassembly View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.1.5 Display and Define the Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.6 Customization of the Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.7 Name the Experiment Directory . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.1.8 Control the Number of Lines in the Output . . . . . . . . . . . . . . . 16
3.1.9 Sorting the Performance Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.10 Scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.11 A More Elaborate Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.12 The Call Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.1.13 More Information on the Experiment . . . . . . . . . . . . . . . . . . . . 20
3.1.14 Control the Sampling Frequency . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.1.15 Information on Load Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.2 Support for Multithreading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2.1 Creating a Multithreading Experiment . . . . . . . . . . . . . . . . . . . . 24
3.2.2 Commands Specific to Multithreading . . . . . . . . . . . . . . . . . . . . . 25
3.3 View Multiple Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.3.1 Aggregation of Experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.3.2 Comparison of Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.4 Profile Hardware Event Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.4.1 Getting Information on the Counters Supported . . . . . . . . . . . 34
3.4.2 Examples Using Hardware Event Counters . . . . . . . . . . . . . . . . 37
3.5 Java Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4

The gprofng Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.1
4.2
4.3
4.4

Tools Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
The gprofng.rc file with default settings . . . . . . . . . . . . . . . . . . . . . . . . 48
Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Supported Environment Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

ii

5

Performance Data Collection. . . . . . . . . . . . . . . . . . 51
5.1

6

View the Performance Information . . . . . . . . . . . 53
6.1

7

The gprofng collect app Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

The gprofng display text Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.1.1 The gprofng display text Commands . . . . . . . . . . . . . . . . . . . 53
Commands that List Experiment Details . . . . . . . . . . . . . . . . . . . . . . 53
Commands that Affect Listings and Output . . . . . . . . . . . . . . . . . . . 55
Predefined Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Commands to Set and Change Search Paths . . . . . . . . . . . . . . . . . . 56

Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
7.1
7.2
7.3
7.4
7.5
7.6
7.7
7.8
7.9

8

The Program Counter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Inclusive and Exclusive Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Metric Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
The Viewmode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
The Selection List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Load Objects and Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
The Concept of a CPU in gprofng . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Hardware Event Counters Explained . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
What is <apath>? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Other Document Formats . . . . . . . . . . . . . . . . . . . . . 65

Appendix A
A.1
A.2
A.3
A.4
A.5
A.6

The gprofng Man Pages . . . . . . . . . . . 67

Man page for gprofng . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Man page for gprofng collect app . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Man page for gprofng display text . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Man page for gprofng display html . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Man page for gprofng display src . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Man page for gprofng archive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

1

1 Introduction
The gprofng tool is the next generation profiler for Linux. It consists ofvarious commands to generate and display profile information.
This manual starts with a tutorial how to create and interpret a
profile. This part is highly practical and has the goal to get users up to
speed as quickly as possible. As soon as possible, we would like to show you
how to get your first profile on your screen.
This is followed by more examples, covering many of the features. At
the end of this tutorial, you should feel confident enough to tackle the more
complex tasks.
In a future update a more formal reference manual will be included aswell. Since even in this tutorial we use certain terminology, we have included
a chapter with descriptions at the end. In case you encounter unfamiliar
wordings or terminology, please check this chapter.
One word of caution. In several cases we had to somewhat tweak
the screen output in order to make it fit. This is why the output may looksomewhat different when you try things yourself.
For now, we wish you a smooth profiling experience with gprofng andgood luck tackling performance bottlenecks.

3

2 A Brief Overview of gprofng
Before we cover this tool in quite some detail, we start with a brief overview
of what it is, and the main features. Since we know that many of you would
like to get started rightaway, already in this first chapter we explain thebasics of profiling with gprofng.

2.1 Main Features
These are the main features of the gprofng tool:• Profiling is supported for an application written in C, C++, Java, or
Scala.
• Shared libraries are supported. The information is presented at the
instruction level.
• The following multithreading programming models are supported:
Pthreads, OpenMP, and Java threads.
• This tool works with unmodified production level executables. Thereis no need to recompile the code, but if the ‘-g’ option has been used
when building the application, source line level information is available.
• The focus is on support for code generated with the gcc compiler, butthere is some limited support for the icc compiler as well. Future
improvements and enhancements will focus on gcc though.
• Processors from Intel, AMD, and Arm are supported, but the level of
support depends on the architectural details. In particular, hardware
event counters may not be supported. If this is the case, all views not
related to these counters still ought to work though.
• Several views into the data are supported. For example, a function
overview where the time is spent, but also a source line, disassembly,
call tree and a caller-callees overview are available.
• Through filters, the user can zoom in on an area of interest.
• Two or more profiles can be aggregated, or used in a comparison. This
comparison can be obtained at the function, source line, and disassemblylevel.
• Through a simple scripting language, and customization of the metrics
shown, the generation and creation of a profile can be fully automatedand provide tailored output.

2.2 Sampling versus Tracing
A key difference with some other profiling tools is that the main data collection command gprofng collect app mostly uses Program Counter (PC)sampling under the hood.
With sampling, the executable is interrupted at regular intervals.
Each time it is halted, key information is gathered and stored. This includes

4

GNU gprofng

the Program Counter that keeps track of where the execution is. Hence thename.
Together with operational data, this information is stored in the experiment directory and can be viewed in the second phase.
For example, the PC information is used to derive where the program
was when it was halted. Since the sampling interval is known, it is relatively
easy to derive how much time was spent in the various parts of the program.
The opposite technique is generally referred to as tracing. With tracing, the target is instrumented with specific calls that collect the requested
information.
These are some of the pros and cons of PC sampling verus tracing:• Since there is no need to recompile, existing executables can be used
and the profile measures the behaviour of exactly the same executablethat is used in production runs.
With sampling, one inherently profiles a different executable, because
the calls to the instrumentation library may affect the compiler optimizations and run time behaviour.
• With sampling, there are very few restrictions on what can be profiled
and even without access to the source code, a basic profile can be made.
• A downside of sampling is that, depending on the sampling frequency,
small functions may be missed or not captured accurately. Although
this is rare, this may happen and is the reason why the user has control
over the sampling rate.
• While tracing produces precise information, sampling is statistical in nature. As a result, small variations may occur across seemingly identicalruns. We have not observed more than a few percent deviation though.
Especially if the target job executed for a sufficiently long time.
• With sampling, it is not possible to get an accurate count how often
functions are called.

2.3 Steps Needed to Create a Profile
Creating a profile takes two steps. First the profile data needs to be generated. This is followed by a viewing step to create a report from the information that has been gathered.
Every gprofng command starts with gprofng, the name of the driver.
This is followed by a keyword to define the high level functionality. Depending on this keyword, a third qualifier may be needed to further narrow down
the request. This combination is then followed by options that are specificto the functionality desired.
The command to gather, or “collect”, the performance data is called
gprofng collect app. Aside from numerous options, this command takes
the name of the target executable as an input parameter.

Chapter 2: A Brief Overview of gprofng

5

Upon completion of the run, the performance data can be found in
the newly created experiment directory.
Unless explicitly specified otherwise, a default name for this directoryis chosen. The name is test.<n>.er where <n> is the first integer number
not in use yet for such a name.
For example, the first time gprofng collect app is invoked, an experiment directory with the name test.1.er is created. Upon a subsequent
invocation of gprofng collect app in the same directory, an experimentdirectory with the name test.2.er will be created, and so forth.
Note that gprofng collect app supports an option to explicitly
name the experiment directory. Aside from the restriction that the name ofthis directory has to end with ‘.er’, any valid directory name can be used
for this.
Now that we have the performance data, the next step is to displayit.
The most commonly used command to view the performance information is gprofng display text. This is a very extensive and customizable
tool that produces the information in ASCII format.
Another option is to use gprofng display html. This tool generates
a directory with files in html format. These can be viewed in a browser,
allowing for easy navigation through the profile data.

7

3 A Mini Tutorial
In this chapter we present and discuss the main functionality of gprofng.
This will be a practical approach, using an example code to generate profile
data and show how to get various performance reports.

3.1 Getting Started
The information presented here provides a good and common basis for many
profiling tasks, but there are more features that you may want to leverage.
These are covered in subsequent sections in this chapter.

3.1.1 The Example Program
Throughout this guide we use the same example C code that implements
the multiplication of a vector of length n by an m by n matrix. The result
is stored in a vector of length m. The algorithm has been parallelized using
Posix Threads, or Pthreads for short.
The code was built using the gcc compiler and the name of the executable is mxv-pthreads.
The matrix sizes can be set through the -m and -n options. The
number of threads is set with the -t option. These are additional threads
that are used in the multiplication. To increase the duration of the run, thecomputations are executed repeatedly.
This is an example that multiplies a 8000 by 4000 matrix with avector of length 4000. Although this is a multithreaded application, initially
we will be using 1 threads. Later on we will show examples using multiple
threads.
$ ./mxv-pthreads -m 8000 -n 4000 -t 1
mxv: error check passed - rows = 8000 columns = 4000 threads = 1$

The program performs an internal check to verify that the computed
results are correct. The result of this check is printed, as well as the matrix
sizes and the number of threads used.

3.1.2 A First Profile
The first step is to collect the performance data. It is important to rememberthat much more information is gathered than may be shown by default.
Often a single data collection run is sufficient to get a lot of insight.
The gprofng collect app command is used for the data collection.
Nothing needs to be changed in the way the application is executed. The
only difference is that it is now run under control of the tool, as shown below:$ gprofng collect app ./mxv-pthreads -m 8000 -n 4000 -t 1

8

GNU gprofng

This produces the following output:
Creating experiment directory test.1.er (Process ID: 2749878) ...
mxv: error check passed - rows = 8000 columns = 4000 threads = 1

We see a message that an experiment directory with the nametest.1.er has been created. The process id is also echoed. The application completes as usual and we have our first experiment directory thatcan be analyzed.
The tool we use for this is called gprofng display text. It takes thename of the experiment directory as an argument.
If invoked this way, the tool starts in the interactive interpreter mode.
While in this environment, commands can be given and the tool responds.
This is illustrated below:$ gprofng display text test.1.er
Warning: History and command editing is not supported on this system.
(gp-display-text) quit$

While useful in certain cases, we prefer to use this tool in commandline mode by specifying the commands to be issued when invoking the tool.
The way to do this is to prepend the command(s) with a hyphen (‘-’) if usedon the command line.
Since this makes the command appear to be an option, they are also
sometimes referred to as such, but technically they are commands. Thisis the terminology we will use in this user guide, but for convenience thecommands are also listed as options in the index.
For example, below we use the functions command to request a listof the functions that have been executed, plus their respective CPU times:$ gprofng display text -functions test.1.er$ gprofng display text -functions test.1.er
Functions sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU
sec.
%
9.367 100.00
8.926 95.30
0.210
2.24
0.080
0.85
0.070
0.75
0.060
0.64
0.010
0.11
0.010
0.11
0.
0.
0.
0.
0.
0.

Incl. Total
CPU
sec.
%
9.367 100.00
8.926 95.30
0.420
4.49
0.210
2.24
0.130
1.39
0.060
0.64
0.020
0.21
0.010
0.11
8.926 95.30
0.440
4.70
0.020
0.21

Name

<Total>mxv_core
init_data
drand48
erand48_r__drand48_iterate_int_malloc
sysmalloc<static>@0x47960 (<libgp-collector.so>)__libc_start_main
allocate_dataChapter 3: A Mini Tutorial
0.
0.
0.
0.

0.
0.
0.
0.

8.926
0.440
0.020
8.926

95.30
4.70
0.21
95.30

9
driver_mxv
mainmallocstart_thread

As easy and simple as these steps are, we do have a first profile of ourprogram!
There are five columns. The first four contain the ”Total CPU Time”,
which is the sum of the user and system time. See Section 7.2 [Inclusive andExclusive Metrics], page 59, for an explanation of “exclusive” and “inclusive”
times.
The first line echoes the metric that is used to sort the output. By
default, this is the exclusive CPU time, but through the sort command, thesort metric can be changed by the user.
Next, there are four columns with the exclusive and inclusive CPU
times and the respective percentages. This is followed by the name of thefunction.
The function with the name <Total> is not a user function. It isa pseudo function introduced by gprofng. It is used to display the accumulated measured metric values. In this example, we see that the total
CPU time of this job was 9.367 seconds and it is scaled to 100%. All otherpercentages in the same column are relative to this number.
With 8.926 seconds, function mxv_core takes 95.30% of the total timeand is by far the most time consuming function. The exclusive and inclusive
metrics are identical, which means that is a leaf function not calling anyother functions.
The next function in the list is init_data. Although with 4.49%, theCPU time spent in this part is modest, this is an interesting entry because
the inclusive CPU time of 0.420 seconds is twice the exclusive CPU timeof 0.210 seconds. Clearly this function is calling another function, or evenmore than one function and collectively this takes 0.210 seconds. Below weshow the call tree feature that provides more details on the call structure of
the application.
The function <static>@0x47960 (<libgp-collector.so>) does
odd and certainly not familiar. It is one of the internal functions used by
gprofng collect app and can be ignored. Also, while the inclusive time
is high, the exclusive time is zero. This means it doesn’t contribute to theperformance.
The question is how we know where this function originates from?
There are several commands to dig deeper an get more details on a function.
See Section 3.1.15 [Information on Load Objects], page 22.

3.1.3 The Source Code View
In general, the tuning efforts are best focused on the most time consumingpart(s) of an application. In this case that is easy, since over 95% of the total

10

GNU gprofng

CPU time is spent in function mxv_core. It is now time to dig deeper andlook at the metrics distribution at the source code level. Since we measured
CPU times, these are the metrics shown.
The source command is used to accomplish this. It takes the nameof the function, not the source filename, as an argument. This is demonstrated below, where the gprofng display text command is used to show
the annotated source listing of function mxv_core.
Be aware that when using the gcc compiler, the source code has tobe compiled with the -g option in order for the source code feature to work.
Otherwise the location(s) can not be determined. For other compilers werecommend to check the documentation for such an option.
Below the command to display the source code of a function is shown.
Since at this point we are primarily interested in the timings only, we use themetrics command to request the exclusive and inclusive total CPU timings
only. See Section 3.1.5 [Display and Define the Metrics], page 14, for more
information how to define the metrics to be displayed.
$ gprofng display text -metrics ei.totalcpu -source mxv_core test.1.er

The output is shown below. It has been somewhat modified to fit theformatting constraints and reduce the number of lines.
Current metrics: e.totalcpu:i.totalcpu:nameCurrent Sort Metric: Exclusive Total CPU Time ( e.totalcpu )Source file: <apath>/mxv.c
Object file: mxv-pthreads (found as test.1.er/archives/...)
Load Object: mxv-pthreads (found as test.1.er/archives/...)
Excl.
Total
CPU sec.

Incl.
Total
CPU sec.

<lines deleted>

0.
0.

0.
## 4.613

0.
0.

0.
4.613

<Function: mxv_core>
43. void __attribute__ ((noinline))
mxv_core (int64_t row_index_start,
44.
int64_t row_index_end,45.
int64_t m,
46.
int64_t n,47.
double **restrict A,
48.
double *restrict b,
49.
double *restrict c)
50. {
50. {
51.
for (int64_t i=row_index_start;i<=row_index_end; i++)
52.
{
53.
double row_sum = 0.0;54.
for (int64_t j=0; j<n; j++)Chapter 3: A Mini Tutorial## 4.313
0.
0.

4.313
0.
0.

11
55.
56.
57.
58. }

row_sum += A[i][j] * b[j];c[i] = row_sum;}

The first line echoes the metrics that have been selected. The secondline is not very meaningful when looking at the source code listing, but itshows the metric that is used to sort the data.
The next three lines provide information on the location of the source
file, the object file and the load object (See Section 7.6 [Load Objects andFunctions], page 62).
Function mxv_core is part of a source file that has other functions as
well. These functions will be shown with the values for the metrics, but forlay-out purposes they have been removed in the output shown above.
The header is followed by the annotated source code listing. The
selected metrics are shown first, followed by a source line number, and thesource code.
The most time consuming line(s) are marked with the ##symbol. In this way they are easier to identify and find with a search.
What we see is that all of the time is spent in lines 54-55.
A related command sometimes comes handy as well. It is called linesand displays a list of the source lines and their metrics, ordered according to
the current sort metric (See Section 3.1.9 [Sorting the Performance Data],
page 16).
Below the command and the output. For lay-out reasons, only the top
10 is shown here and the last part of the text on some lines has been replacedby dots. The full text is ‘instructions without line numbers’ and meansthat the line number information for that function was not found.
$ gprofng display text -lines test.1.er
Lines sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU
sec.
%
9.367 100.00
4.613 49.25
4.313 46.05
0.160
1.71
0.080
0.85
0.070
0.75
0.060
0.64
0.040
0.43
0.010
0.11
0.010
0.11

Incl. Total
CPU
sec.
%
9.367 100.00
4.613 49.25
4.313 46.05
0.370
3.95
0.210
2.24
0.130
1.39
0.060
0.64
0.040
0.43
0.020
0.21
0.010
0.11

Name

<Total>mxv_core, line 54 in "mxv.c"
mxv_core, line 55 in "mxv.c"
init_data, line 118 in "manage_data.c"<Function: drand48, instructions ...>
<Function: erand48_r, instructions ...>
<Function: __drand48_iterate, ...>
init_data, line 124 in "manage_data.c"<Function: _int_malloc, instructions ...>
<Function: sysmalloc, instructions ...>

What this overview immediately highlights is that the third most time
consuming source line takes 0.370 seconds only. This means that the inclu-12

GNU gprofng

sive time is only 3.95% and clearly this branch of the code hardly impacts
the performance.

3.1.4 The Disassembly View
The source view is very useful to obtain more insight where the time isspent, but sometimes this is not sufficient. The disassembly view provides
more details since it shows the metrics at the instruction level.
This view is displayed with the disasm command and as with thesource view, it displays an annotated listing. In this case it shows the instructions with the metrics, interleaved with the source lines. The instructions have a reference in square brackets ([ and ]) to the source line theycorrespond to.
We again focus on the tmings only and set the metrics accordingly:$ gprofng display text -metrics ei.totalcpu -disasm mxv_core test.1.er
Current metrics: e.totalcpu:i.totalcpu:nameCurrent Sort Metric: Exclusive Total CPU Time ( e.totalcpu )Source file: <apath>/src/mxv.c
Object file: mxv-pthreads (found as test.1.er/archives/...)
Load Object: mxv-pthreads (found as test.1.er/archives/...)
Excl.
Total
CPU sec.

Incl.
Total
CPU sec.

<lines deleted>

0.

0.

0.
0.
0.
0.

0.
0.
0.
0.

0.

0.

43. void __attribute__ ((noinline))
mxv_core (int64_t row_index_start,
44.
int64_t row_index_end,45.
int64_t m,
46.
int64_t n,47.
double **restrict A,
48.
double *restrict b,
49.
double *restrict c)50. {<Function: mxv_core>[50]
401d56: mov0x8(%rsp),%r1051.
for (int64_t i=row_index_start;i<=row_index_end; i++)[51]
401d5b: cmp%rsi,%rdi[51]
401d5e: jg
0x47
[51]
401d60: add$0x1,%rsi[51]
401d64: jmp
0x36
52.
{
53.
double row_sum = 0.0;54.
for (int64_t j=0; j<n; j++)55
row_sum += A[i][j] * b[j];[55]401d66: mov(%r8,%rdi,8),%rdx

Chapter 3: A Mini Tutorial
0.
0.
0.110
1.921
2.282
## 4.613
0.
0.

0.
0.
0.110
1.921
2.282
4.613
0.
0.

13
[54]
[53]
[55]
[55]
[55]
[54]
[54]
[54]

56.
0.
0.
0.
0.
0.
0.
0.
0.

0.
0.
0.
0.
0.
0.
0.
0.

0.

0.

[56]
[51]
[51]
[51]
[53]
[54]
[54]
[54]
57.
}
58. }
[58]

401d6a: mov
401d6f: pxor
401d73: movsd
401d78: mulsd
401d7e: addsd
401d82: add
401d86: cmp
401d89: jne
c[i] = row_sum;
401d8b: movsd
401d91: add
401d95: cmp
401d98: je
401d9a: pxor
401d9e: test
401da1: jg
401da3: jmp

401da5:$0x0,%eax%xmm1,%xmm1(%rdx,%rax,8),%xmm0(%r9,%rax,8),%xmm0%xmm0,%xmm1$0x1,%rax%rax,%rcx
0xffffffffffffffea%xmm1,(%r10,%rdi,8)$0x1,%rdi%rsi,%rdi
0xd%xmm1,%xmm1%rcx,%rcx
0xffffffffffffffc5
0xffffffffffffffe8

ret

For each instruction, the timing values are given and we can immediately identify the most expensive instructions. As with the source levelview, these are marked with the ## symbol.
It comes as no surprise that the time consuming instructions originate
from the source code at lines 54-55. One thing to note is that the sourceline numbers no longer appear in sequential order. This is because thecompiler has re-ordered the instructions as part of the code optimizations it
has performed.
As illustrated below and similar to the lines command, we can get
an overview of the instructions executed by using the pcs command.
Below the command and the output, which again has been restricted to 10
lines. As before, some lines have been shortened for lay-out purposes.
$ gprofng display text -pcs test.1.er
PCs sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU
sec.
%
9.367 100.00
4.613 49.25
2.282 24.36
1.921 20.51
0.150
1.60
0.110
1.18
0.040
0.43
0.040
0.43
0.030
0.32

Incl. Total
CPU
sec.
%
9.367 100.00
4.613 49.25
2.282 24.36
1.921 20.51
0.150
1.60
0.110
1.18
0.040
0.43
0.040
0.43
0.030
0.32

Name

<Total>mxv_core + 0x0000002C, line 54 in "mxv.c"
mxv_core + 0x00000028, line 55 in "mxv.c"
mxv_core + 0x00000022, line 55 in "mxv.c"
init_data + 0x000000AC, line 118 in ...
mxv_core + 0x0000001D, line 55 in "mxv.c"
drand48 + 0x00000022
init_data + 0x000000F1, line 124 in ...
__drand48_iterate + 0x0000001E

14

GNU gprofng
0.020

0.21

0.020

0.21

__drand48_iterate + 0x00000038

What we see is that the top three instructions take 94% of the total CPU
time and any optimizations should focus on this part of the code..

3.1.5 Display and Define the Metrics
The metrics shown by gprofng display text are useful, but there is morerecorded than displayed by default. We can customize the values shown by
defining the metrics ourselves.
There are two commands related to changing the metrics shown:
metric_list and metrics.
The first command shows the currently selected metrics, plus all themetrics that have been stored as part of the experiment. The second command may be used to define the metric list.
This is the way to get the information about the metrics:$ gprofng display text -metric_list test.1.er

This is the output:
Current metrics: e.%totalcpu:i.%totalcpu:nameCurrent Sort Metric: Exclusive Total CPU Time ( e.%totalcpu )Available metrics:
Exclusive Total CPU Time: e.%totalcpu
Inclusive Total CPU Time: i.%totalcpu
Size: size
PC Address: address
Name: name

This shows the metrics that are currently used, the metric that isused to sort the data and all the metrics that have been recorded, but are
not necessarily shown.
In this case, the current metrics are set to the exclusive and inclusive
total CPU times, the respective percentages, and the name of the function,or load object.
The metrics command is used to define the metrics that need to be
displayed.
For example, to swap the exclusive and inclusive metrics, use thefollowing metric definition: i.%totalcpu:e.%totalcpu.
Since the metrics can be tailored for different views, there is also a
way to reset them to the default. This is done through the special keyword
default for the metrics definition (-metrics default).

3.1.6 Customization of the Output
With the information just given, the function overview can be customized.
For sake of the example, we would like to display the name of the functionChapter 3: A Mini Tutorial

15

first, only followed by the exclusive CPU time, given as an absolute number
and a percentage.
Note that the commands are parsed in order of appearance. This is
why we need to define the metrics before requesting the function overview:$ gprofng display text -metrics name:e.%totalcpu -functions test.1.er
Current metrics: name:e.%totalcpu
Current Sort Metric: Exclusive Total CPU Time ( e.%totalcpu )Functions sorted by metric: Exclusive Total CPU Time
Name

Excl. Total
CPU
sec.
%
<Total>9.367 100.00
mxv_core
8.926 95.30
init_data
0.210
2.24
drand48
0.080
0.85
erand48_r
0.070
0.75
__drand48_iterate
0.060
0.64
_int_malloc
0.010
0.11
sysmalloc
0.010
0.11
<static>@0x47960 (<libgp-collector.so>) 0.
0.
__libc_start_main
0.
0.
allocate_data0.
0.
driver_mxv
0.
0.
main
0.
0.
malloc0.
0.
start_thread0.
0.

This was a first and simple example how to customize the output.
Note that we did not rerun our profiling job and merely modified the display
settings. Below we will show other and also more advanced examples ofcustomization.

3.1.7 Name the Experiment Directory
When using gprofng collect app, the default names for experiments work
fine, but they are quite generic. It is often more convenient to select amore descriptive name. For example, one that reflects conditions for theexperiment conducted, like the number of threads used.
For this, the mutually exclusive -o and -O options come in handy.
Both may be used to provide a name for the experiment directory, but the
behaviour of gprofng collect app is different.
With the ‘-o’ option, an existing experiment directory is not overwritten. Any directory with the same name either needs to be renamed,
moved, or removed, before the experiment can be conducted.
This is in contrast with the behaviour for the ‘-O’ option. Any
existing directory with the same name is silently overwritten.

16

GNU gprofng
Be aware that the name of the experiment directory has to end with.er.

3.1.8 Control the Number of Lines in the Output
The limit <n> command can be used to control the number of lines printed
in various views. For example it impacts the function view, but also takes
effect for other display commands, like lines.
The argument <n> should be a positive integer number. It sets thenumber of lines in the (function) view. A value of zero resets the limit to
the default.
Be aware that the pseudo-function <Total> counts as a regular function. For example limit 10 displays nine user level functions.

3.1.9 Sorting the Performance Data
The sort <key> command sets the key to be used when sorting the performance data.
The key is a valid metric definition, but the visibility field (See
Section 7.3 [Metric Definitions], page 59) in the metric definition is ignored,since this does not affect the outcome of the sorting operation. For example
if the sort key is set to e.totalcpu, the values will be sorted in descendingorder with respect to the exclusive total CPU time.
The data can be sorted in reverse order by prepending the metricdefinition with a minus (‘-’) sign. For example sort -e.totalcpu.
A default metric for the sort operation has been defined and since
this is a persistent command, this default can be restored with default as
the key (sort default).

3.1.10 Scripting
The list with commands for gprofng display text can be very long. Thisis tedious and also error prone. Luckily, there is an easier and elegant way
to control the output of this tool.
Through the script command, the name of a file with commands canbe passed in. These commands are parsed and executed as if they appearedon the command line in the same order as encountered in the file. The
commands in this script file can actually be mixed with commands on thecommand line and multiple script files may be used. The difference between
the commands in the script file and those used on the command line is thatthe latter require a leading dash (‘-’) symbol.
Comment lines in a script file are supported. They need to start withthe ‘#’ symbol.

3.1.11 A More Elaborate Example
With the information presented so far, we can customize our data gatheringand display commands.

Chapter 3: A Mini Tutorial

17

As an example, we would like to use mxv.1.thr.er as the name forthe experiment directory. In this way, the name of the algorithm and thenumber of threads that were used are included in the name. We also don’t
mind to overwrite an existing experiment directory with the same name.
All that needs to be done is to use the ‘-O’ option, followed by the
directory name of choice when running gprofng collect app:$ exe=mxv-pthreads$ m=8000$ n=4000$ gprofng collect app -O mxv.1.thr.er ./$exe -m $m -n $n -t 1

Since we want to customize the profile and prefer to keep the command line short, the commands to generate the profile are put into a filewith the name my-script:$ cat my-script# This is my first gprofng script# Set the metrics
metrics i.%totalcpu:e.%totalcpu:name# Use the exclusive time to sort
sort e.totalcpu# Limit the function list to 5 lines
limit 5
# Show the function list
functionsThis script file is specified as input to the gprofng display text
command that is used to display the performance information stored inexperiment directory mxv.1.thr.er:$ gprofng display text -script my-script mxv.1.thr.er

This command produces the following output:# This is my first gprofng script# Set the metrics
Current metrics: i.%totalcpu:e.%totalcpu:nameCurrent Sort Metric: Exclusive Total CPU Time ( e.%totalcpu )# Use the exclusive time to sort
Current Sort Metric: Exclusive Total CPU Time ( e.%totalcpu )# Limit the function list to 5 lines
Print limit set to 5
# Show the function list
Functions sorted by metric: Exclusive Total CPU Time
Incl. Total
CPU
sec.
%
9.703 100.00
9.226 95.09

Excl. Total
CPU
sec.
%
9.703 100.00
9.226 95.09

Name<Total>mxv_core18

GNU gprofng
0.455
0.169
0.244

4.69
1.75
2.52

0.210
0.123
0.075

2.17
1.26
0.77

init_data
erand48_r
drand48In the first part of the output the comment lines in the script fileare echoed. These are interleaved with an acknowledgement message for thecommands.
This is followed by a profile consisting of 5 lines only. For both metrics, the percentages plus the timings are given. The numbers are sortedwith respect to the exclusive total CPU time. Although this is the default,
for demonstration purposes we use the sort command to explicitly define
the metric for the sort.
While we executed the same job as before and only changed the nameof the experiment directory, the results are somewhat different. This issampling in action. The numbers are not all that different though. It isseen that function mxv_core is responsbile for 95% of the CPU time andinit_data takes 4.5% only.

3.1.12 The Call Tree
The call tree shows the dynamic structure of the application by displaying the functions executed and their parent. The CPU time attributed toeach function is shown as well. This view helps to find the most expensive
execution path in the program.
This feature is enabled through the calltree command. For example, this is how to get the call tree for our current experiment:$ gprofng display text -calltree mxv.1.thr.er

This displays the following structure:
Functions Call Tree. Metric: Attributed Total CPU Time
Attr. Total
CPU
sec.
%
9.703 100.00
9.226 95.09
9.226 95.09
9.226 95.09
9.226 95.09
0.477
4.91
0.477
4.91
0.455
4.69
0.244
2.52
0.169
1.75
0.047
0.48
0.021
0.22
0.021
0.22
0.021
0.22

Name

+-<Total>+-start_thread| +-<static>@0x47960 (<libgp-collector.so>)|+-driver_mxv|+-mxv_core+-__libc_start_main+-main+-init_data| +-drand48|+-erand48_r|+-__drand48_iterate+-allocate_data| +-malloc|+-_int_malloc

Chapter 3: A Mini Tutorial
0.006
0.003
0.003
0.003
0.001
0.001

0.06
0.03
0.03
0.03
0.01
0.01

19

|
+-sysmalloc|+-__default_morecore|+-sbrk|+-brk+-pthread_create+-__pthread_create_2_1

At first sight this may not be what is expected and some explanation
is in place.
The top function is the pseudo-function <Total> that we have seen
before. It is introduced and shown here to provide the total value of themetric(s).
We also see function <static>@0x47960 in the call tree and apparently it is from libgp-collector.so, a library that is internal to gprofng.
The <static> marker, followed by the program counter, is shown if the nameof the function cannot be found. This function is part of the implementationof the data collection process and should be hidden to the user. This is partof a planned future enhancement.
In general, if a view has a function that does not appear to be part of
the user code, or seems odd anyhow, the objects and fsingle commands
are very useful to find out more about load objects in general, but also tohelp identify an unknown entry in the function overview. See Section 7.6[Load Objects and Functions], page 62.
Another thing to note is that there are two main branches. The one
under <static>@0x47960 and the second one under __libc_start_main.
This reflects the fact that this is a multithreaded program and the threadedpart shows up as a separate branch in the call tree.
The way to interpret this structure is as follows. The program starts
under control of __libc_start_main. This executes the main program
called main, which at the top level executes functions init_data, allocate_data, and pthread_create. The latter function creates and executes the
additional thread(s).
For this multithreaded part of the code, we need to look at the branch
under function start_thread that calls the driver code for the matrix-vector
multiplication (driver_mxv), which executes the function that performs theactual multiplication (mxv_core).
There are two things worth noting for the call tree feature:• This is a dynamic tree and since sampling is used, it most likely looks
slighlty different across seemingly identical profile runs. In case the runtimes are short, it is worth considering to use a high resolution throughthe ‘-p’ option. For example use ‘-p hi’ to increase the sampling rate.
• In case hardware event counters have been enabled (See Section 3.4 [Profile Hardware Event Counters], page 34), these values are also displayedin the call tree view.

20

GNU gprofng

3.1.13 More Information on the Experiment
The experiment directory not only contains performance related data. Several system characteristics, the profiling command executed, plus someglobal performance statistics are stored and can be displayed.
The header command displays information about the experiment(s).
For example, this is command is used to extract this data from for our
experiment directory:$ gprofng display text -header mxv.1.thr.er

The above command prints the following information. Note that someof the lay-out and the information has been modified. Directory paths havebeen replaced <apath> for example. Textual changes are marked with the‘<’ and ‘>’ symbols.
Experiment: mxv.1.thr.er
No errors
No warnings
Archive command ‘ /usr/bin/gp-archive -n -a on --outfile<apath>/archive.log <apath>/mxv.1.thr.er’
Target command (64-bit): ’./mxv-pthreads -m 8000 -n 4000 -t 1’
Process pid 2750071, ppid 2750069, pgrp 2749860, sid 2742080
Current working directory: <apath>
Collector version: ‘2.40.00’; experiment version 12.4 (64-bit)
Host ‘<the-host-name>’, OS ‘Linux <version>’, page size 4096,
architecture ‘x86_64’
4 CPUs, clock speed 2294 MHz.
Memory: 3506491 pages @ 4096 = 13697 MB.
Data collection parameters:
Clock-profiling, interval = 997 microsecs.
Periodic sampling, 1 secs.
Follow descendant processes from: fork|exec|combo
Experiment started <date and time>
Experiment Ended: 9.801216173
Data Collection Duration: 9.801216173

The output above may assist in troubleshooting, or to verify someof the operational conditions and we recommend to include this command
when generating a profile.
Related to this command there is a useful option to record comment(s)in an experiment. To this end, use the ‘-C’ option on the gprofng collect
app tool to specify a comment string. Up to ten comment lines can be
included. These comments are displayed with the header command on the
gprofng display text tool.

Chapter 3: A Mini Tutorial

21

The overview command displays information on the experiment(s)and also shows a summary of the values for the metric(s) used. This is an
example how to use it on the newly created experiment directory:$ gprofng display text -overview mxv.1.thr.er
Experiment(s):Experiment
TargetHostStart Time
Duration:mxv.1.thr.er: ’./mxv-pthreads -m 8000 -n 4000 -t 1’: <hostname> (<ISA>, Linux <version>): <date and time>: 9.801 Seconds

Metrics:
Experiment Duration (Seconds): [9.801]
Clock Profiling[X]Total CPU Time - totalcpu (Seconds): [*9.703]
Notes: ’*’ indicates hot metrics, ’[X]’ indicates currently enabled
metrics.
The metrics command can be used to change selections. The
metric_list command lists all available metrics.

This command provides a dashboard overview that helps to easily
identify where the time is spent and in case hardware event counters areused, it shows their total values.

3.1.14 Control the Sampling Frequency
So far we did not go into details on the frequency of the sampling process,but in some cases it is useful to change the default of 10 milliseconds.
The advantage of increasing the sampling frequency is that functions
that do not take much time per invocation are more accurately captured.
The downside is that more data is gathered. This has an impact on theoverhead of the collection process and more disk space is required.
In general this is not an immediate concern, but with heavily threaded
applications that run for an extended period of time, increasing the frequency
may have a more noticeable impact.
The -p option on the gprofng collect app tool is used to enable ordisable clock based profiling, or to explicitly set the sampling rate. This
option takes one of the following keywords:
off
Disable clock based profiling.
onEnable clock based profiling with a per thread sampling intervalof 10 ms. This is the default.
loEnable clock based profiling with a per thread sampling intervalof 100 ms.

22

GNU gprofnghiEnable clock based profiling with a per thread sampling intervalof 1 ms.

value

Enable clock based profiling with a per thread sampling intervalof value.

It may seem unnecessary to have an option to disable clock based
profiling, but there is a good reason to support this. By default, clock
profiling is enabled when conducting hardware event counter experiments(See Section 3.4 [Profile Hardware Event Counters], page 34). With the -p
off option, this can be disabled.
If an explicit value is set for the sampling, the number can be an
integer or a floating-point number. A suffix of ‘u’ for microseconds, or ‘m’
for milliseconds is supported. If no suffix is used, the value is assumed to bein milliseconds.
For example, the following command sets the sampling rate to 5123.4
microseconds:

$ gprofng collect app -p 5123.4u ./mxv-pthreads -m 8000 -n 4000 -t 1

If the value is smaller than the clock profiling minimum, a warning
message is issued and it is set to the minimum. In case it is not a multipleof the clock profiling resolution, it is silently rounded down to the nearest
multiple of the clock resolution. If the value exceeds the clock profilingmaximum, is negative, or zero, an error is reported.
Note that the header command echoes the sampling rate used.

3.1.15 Information on Load Objects
It may happen that the function view shows a function that is not known tothe user. This can easily happen with library functions for example. Luckily
there are three commands that come in handy then.
These commands are objects, fsingle, and fsummary. They provide details on load objects (See Section 7.6 [Load Objects and Functions],
page 62).
The objects command lists all load objects that have been referenced
during the performance experiment. Below we show the command and theresult for our profile job. Like before, some path names in the output havebeen shortened and replaced by the <apath> symbol that represents anabsolute directory path.
$ gprofng display text -objects mxv.1.thr.er

The output includes the name and path of the target executable:<Unknown> (<Unknown>)<mxv-pthreads> (<apath>/mxv-pthreads)

Chapter 3: A Mini Tutorial

23<libdl-2.28.so> (/usr/lib64/libdl-2.28.so)<librt-2.28.so> (/usr/lib64/librt-2.28.so)<libc-2.28.so> (/usr/lib64/libc-2.28.so)<libpthread-2.28.so> (/usr/lib64/libpthread-2.28.so)<libm-2.28.so> (/usr/lib64/libm-2.28.so)<libgp-collector.so> (/usr/lib64/gprofng/libgp-collector.so)<ld-2.28.so> (/usr/lib64/ld-2.28.so)<DYNAMIC_FUNCTIONS> (DYNAMIC_FUNCTIONS)

The fsingle command may be used to get more details on a specificentry in the function view, say. For example, the command below provides
additional information on the pthread_create function shown in the function overview.

$ gprofng display text -fsingle pthread_create mxv.1.thr.er

Below the output from this command. It has been somewhat modified
to match the display requirements.
+ gprofng display text -fsingle pthread_create mxv.1.thr.er
pthread_create
Exclusive Total CPU Time: 0.
( 0. %)
Inclusive Total CPU Time: 0.001 ( 0.0%)
Size:
258
PC Address: 8:0x00049f60
Source File: (unknown)
Object File: (unknown)
Load Object: /usr/lib64/gprofng/libgp-collector.so
Mangled Name:
Aliases:In this table we not only see how much time was spent in this function,
we also see where it originates from. In addition to this, the size and start
address are given as well. If the source code location is known it is also
shown here.
The related fsummary command displays the same information as
fsingle, but for all functions in the function overview, including <Total>:$ gprofng display text -fsummary mxv.1.thr.er
Functions sorted by metric: Exclusive Total CPU Time
<Total>Exclusive Total CPU Time: 9.703 (100.0%)
Inclusive Total CPU Time: 9.703 (100.0%)
Size:
0PC Address: 1:0x00000000
Source File: (unknown)
Object File: (unknown)
Load Object: <Total>
Mangled Name:24

GNU gprofng
Aliases:mxv_core
Exclusive Total CPU Time: 9.226 ( 95.1%)
Inclusive Total CPU Time: 9.226 ( 95.1%)
Size:
80
PC Address: 2:0x00001d56
Source File: <apath>/src/mxv.c
Object File: mxv.1.thr.er/archives/mxv-pthreads_ss_pf53V__5
Load Object: <apath>/mxv-pthreads
Mangled Name:
Aliases:... etc ...

3.2 Support for Multithreading
In this chapter the support for multithreading is introduced and discussed.
As is shown below, nothing needs to be changed when collecting the performance data.
The difference is that additional commands are available to get more
information on the multithreading details, plus that several filters allow theuser to zoom in on specific threads.

3.2.1 Creating a Multithreading Experiment
We demonstrate the support for multithreading using the same code andsettings as before, but this time 2 threads are used:$ exe=mxv-pthreads$ m=8000$ n=4000$ gprofng collect app -O mxv.2.thr.er ./$exe -m $m -n $n -t 2

First of all, in as far as gprofng is concerned, no changes are needed.
Nothing special is needed to profile a multithreaded job when using gprofng.
The same is true when displaying the performance results. The samecommands that were used before work unmodified. For example, this is allthat is needed to get a function overview:$ gprofng display text -limit 5 -functions mxv.2.thr.er

This produces the following familiar looking output:
Print limit set to 5
Functions sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU

Incl. Total
CPU

Name



Chapter 3: A Mini Tutorialsec.
%
9.464 100.00
8.961 94.69
0.224
2.37
0.105
1.11
0.073
0.77

sec.
%
9.464 100.00
8.961 94.69
0.469
4.95
0.177
1.88
0.073
0.77

25

<Total>mxv_core
init_data
erand48_r__drand48_iterate

3.2.2 Commands Specific to Multithreading
The function overview shown above shows the results aggregated over all
the threads. The interesting new element is that we can also look at theperformance data for the individual threads.
The thread_list command displays how many threads have been
used:



$ gprofng display text -thread_list mxv.2.thr.er

This produces the following output, showing that three threads havebeen used:
Exp Sel Total=== === =====1 all
3

The output confirms there is one experiment and that by default all
threads are selected.
It may seem surprising to see three threads here, since we used the-t 2 option, but it is common for a Pthreads program to use one additional
thread. Typically, there is one main thread that runs from start to finish. It
handles the sequential portions of the code, as well as thread management
related tasks. It is no different in the example code. At some point, the mainthread creates and activates the two threads that perform the multiplicationof the matrix with the vector. Upon completion of this computation, themain thread continues.
The threads command is simple, yet very powerful. It shows thetotal value of the metrics for each thread.
$ gprofng display text -threads mxv.2.thr.er

The command above produces the following overview:
Objects sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU
sec.
%
9.464 100.00
4.547 48.05
4.414 46.64
0.502
5.31

Name

<Total>
Process 1, Thread 3
Process 1, Thread 2
Process 1, Thread 126

GNU gprofng

The first line gives the total CPU time accumulated over the threads
selected. This is followed by the metric value(s) for each thread.
From this it is clear that the main thread is responsible for a little
over 5% of the total CPU time, while the other two threads take 47-48%
each.
This view is ideally suited to verify if there are any load balancing
issues and also to find the most time consuming thread(s).
While useful, often more information than this is needed. This iswhere the thread selection filter comes in. Through the thread_select command, one or more threads may be selected. See Section 7.5 [The Selection
List], page 60, how to define the selection list.
Since it is most common to use this command in a script, we do so
as well here. Below the script we are using:# Define the metrics
metrics e.%totalcpu# Limit the output to 5 lines
limit 5
# Get the function overview for thread 1
thread_select 1functions# Get the function overview for thread 2
thread_select 2functions# Get the function overview for thread 3
thread_select 3functions

The definition of the metrics and the output limit have been shownand explained earlier. The new command to focus on is thread_select.
This command takes a list (See Section 7.5 [The Selection List],
page 60) to select specific threads. In this case, the individual thread numbers that were obtained earlier with the thread_list command are selected.
This restricts the output of the functions command to the thread
number(s) specified. This means that the script above shows which function(s) each thread executes and how much CPU time they consumed. Boththe exclusive timings and their percentages are given.
Note that technically this command is a filter and persistent. The
selection remains active until changed through another thread selection command, or when it is reset with the ‘all’ selection list.
This is the relevant part of the output for the first thread:
Exp Sel Total=== === =====
1 1
3
Functions sorted by metric: Exclusive Total CPU Time

Chapter 3: A Mini Tutorial
Excl. Total
CPU
sec.
%
0.502 100.00
0.224 44.64
0.105 20.83
0.073 14.48
0.067 13.29

27

Name

<Total>init_data
erand48_r__drand48_iterate
drand48

As usual, the comment lines are echoed. This is followed by a confirmation of the selection. The first table shows that one experiment is loaded
and that thread 1 out of the three threads has been selected. What is displayed next is the function overview for this particular thread. Due to thelimit 5 command, there are only five functions in this list.
Clearly, this thread handles the data initialization part and as weknow from the call tree output, function init_data executes the 3 otherfunctions shown in this profile.
Below are the overviews for threads 2 and 3 respectively. It is seenthat all of the CPU time is spent in function mxv_core and that this time
is approximately the same for both threads.
# Get the function overview for thread 2
Exp Sel Total=== === =====
1 2
3
Functions sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU
sec.
%
4.414 100.00
4.414 100.00
0.
0.
0.
0.
0.
0.

Name

<Total>mxv_core<static>@0x48630 (<libgp-collector.so>)
driver_mxv
start_thread# Get the function overview for thread 3
Exp Sel Total=== === =====
1 3
3
Functions sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU
sec.
%
4.547 100.00
4.547 100.00
0.
0.
0.
0.
0.
0.

Name

<Total>mxv_core<static>@0x48630 (<libgp-collector.so>)
driver_mxv
start_thread

28

GNU gprofng

When analyzing the performance of a multithreaded application, it
is sometimes useful to know whether threads have mostly executed on thesame core, say, or if they have wandered across multiple cores. This sort ofstickiness is usually referred to as thread affinity.
Similar to the commands for the threads, there are several commands
related to the usage of the cores, or CPUs as they are called in gprofng (See
Section 7.7 [The Concept of a CPU in gprofng], page 62).
Similar to the thread_list command, the cpu_list command displays how many CPUs have been used.
The equivalent of the threads
threads command, is the cpus command, which shows the numbers of theCPUs that were used and the metric values for each one of them. Bothcommands are demonstrated below.

$ gprofng display text -cpu_list -cpus mxv.2.thr.er

This command produces the following output:+ gprofng display text -cpu_list -cpus mxv.2.thr.er
Exp Sel Total=== === =====
1 all
4
Objects sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU
sec.
%
9.464 100.00
4.414 46.64
2.696 28.49
1.851 19.56
0.502
5.31

Name

<Total>
CPU 2
CPU 0
CPU 1
CPU 3

The first table shows that there is only one experiment and that allof the four CPUs have been selected. The second table shows the exclusive
metrics for each of the CPUs that have been used.
As also echoed in the output, the data is sorted with respect to the
exclusive CPU time, but it is very easy to sort the data by the CPU id byusing the sort command:$ gprofng display text -cpu_list -sort name -cpus mxv.2.thr.er

With the sort added, the output is as follows:
Exp Sel Total=== === =====
1 all
4
Current Sort Metric: Name ( name )Objects sorted by metric: Name



Chapter 3: A Mini Tutorial
Excl. Total
CPU
sec.
%
9.464 100.00
2.696 28.49
1.851 19.56
4.414 46.64
0.502
5.31

29

Name

<Total>
CPU 0
CPU 1
CPU 2
CPU 3

While the table with thread times shown earlier may point at a load
imbalance in the application, this overview has a different purpose.
For example, we see that 4 CPUs have been used, but we know thatthe application uses 3 threads only. We will now demonstrate how filters
can be used to help answer the question why 4 CPUs are used, while theapplication has 3 threads only. This means that at least one thread has
executed on more than one CPU.
Recall the thread level timings:
Excl. Total
CPU
sec.
%
9.464 100.00
4.547 48.05
4.414 46.64
0.502
5.31

Name

<Total>
Process 1, Thread 3
Process 1, Thread 2
Process 1, Thread 1

Compared to the CPU timings above, it seems very likely that thread
3 has used more than one CPU, because the thread and CPU timings are
the same for both other threads.
The command below selects thread number 3 and then requests the
CPU utilization for this thread:$ gprofng display text -thread_select 3 -sort name -cpus mxv.2.thr.er

The output shown below confirms that thread 3 is selected and thendisplays the CPU(s) that have been used by this thread:
Exp Sel Total=== === =====
1 3
3
Objects sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU
sec.
%
4.547 100.00
2.696 59.29
1.851 40.71

Name<Total>
CPU 0
CPU 1

The results show that this thread has used CPU 0 nearly 60% of thetime and CPU 1 for the remaining 40%.



30

GNU gprofng

To confirm that this is the only thread that has used more than one
CPU, the same approach can be used for threads 1 and 2:$ gprofng display text -thread_select 1 -cpus mxv.2.thr.er
Exp Sel Total=== === =====
1 1
3
Objects sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU
sec.
%
0.502 100.00
0.502 100.00

Name<Total>CPU 3$ gprofng display text -thread_select 2 -cpus mxv.2.thr.er
Exp Sel Total=== === =====
1 2
3
Objects sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU
sec.
%
4.414 100.00
4.414 100.00

Name<Total>CPU 2

The output above shows that indeed threads 1 and 2 each have used a single
CPU only.

3.3 View Multiple Experiments
One thing we did not cover sofar is that gprofng fully supports the analysis
of multiple experiments. The gprofng display text tool accepts a list ofexperiments. The data can either be aggregated across the experiments, orused in a comparison.
The default is to aggregate the metric values across the experimentsthat have been loaded. The compare command can be used to enable thecomparison of results.
In this section both modes are illustrated with an example.

3.3.1 Aggregation of Experiments
If the data for multiple experiments is aggregrated, the gprofng display
text tool shows the combined results. For example, below is the script
to show the function view for the data aggregated over two experiments,
drop the first experiment and then show the function view fo the second
experiment only. We will call it my-script-agg.

Chapter 3: A Mini Tutorial

31# Define the metrics
metrics e.%totalcpu# Limit the output to 5 lines
limit 5
# Get the list with experiments
experiment_list# Get the function overview for all
functions# Drop the first experimentdrop_exp mxv.2.thr.er# Get the function overview for exp #2
functionsWith the exception of the experiment_list command, all commands
used have been discussed earlier.
The experiment_list command provides a list of the experimentsthat have been loaded. This may be used to get the experiment IDs and toverify the correct experiments are loaded for the aggregation.
Below is an example that loads two experiments and uses the above script
to display different function views.
$ gprofng display text -script my-script-agg mxv.2.thr.er mxv.4.thr.er

This produces the following output:# Define the metrics
Current metrics: e.%totalcpu:nameCurrent Sort Metric: Exclusive Total CPU Time ( e.%totalcpu )# Limit the output to 5 lines
Print limit set to 5
# Get the list with experiments
ID SelPID Experiment== === ======= ============
1 yes 1339450 mxv.2.thr.er
2 yes 3579561 mxv.4.thr.er# Get the function overview for all
Functions sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU
sec.
%
20.567 100.00
19.553 95.07
0.474
2.30
0.198
0.96
0.149
0.72

Name

<Total>mxv_core
init_data
erand48_r
drand48# Drop the first experiment
Experiment mxv.2.thr.er has been dropped32

GNU gprofng# Get the function overview for exp #2
Functions sorted by metric: Exclusive Total CPU Time
Excl. Total
CPU
sec.
%
11.104 100.00
10.592 95.39
0.249
2.24
0.094
0.84
0.082
0.74

Name

<Total>mxv_core
init_data
erand48_r
drand48

The first five lines should look familiar. The five lines following echo
the comment line in the script and show the overview of the experiments.
This confirms two experiments have been loaded and that both are active.
This is followed by the function overview. The timings have been summed
up and the percentages are adjusted accordingly.

3.3.2 Comparison of Experiments
The support for multiple experiments really shines in comparison mode.
In comparison mode, the data for the various experiments is shown side byside, as illustrated below where we compare the results for the multithreaded
experiments using two and four threads respectively.
This feature is controlled through the compare command.
The comparison mode is enabled through compare on and withcompare off it is disabled again. In addition to ‘on’, or ‘off’, this command also supports the ‘delta’ and ‘ratio’ keywords.
This is the script that will be used in our example. It sets the comparison mode to ‘on’:# Define the metrics
metrics e.%totalcpu# Limit the output to 5 lines
limit 5
# Set the comparison mode to differencescompare on# Get the function overview
functions

Assuming this script file is called my-script-comp, this is how it isused to display the differences:$ gprofng display text -script my-script-comp mxv.2.thr.er mxv.4.thr.er

This produces the output shown below. The data for the first experiment isshown as absolute numbers. The timings for the other experiment are shown
as a delta relative to these reference numbers:
mxv.2.thr.er

mxv.4.thr.erChapter 3: A Mini Tutorial
Excl. Total
CPU
sec.
%
9.464 100.00
8.961 94.69
0.224
2.37
0.105
1.11
0.073
0.77

Excl. Total
CPU
sec.
%
11.104 100.00
10.592 95.39
0.249
2.24
0.094
0.84
0.060
0.54

33
Name

<Total>mxv_core
init_data
erand48_r__drand48_iterateThis table is already helpful to more easily compare (two) profiles,
but there is more that we can do here.
By default, in comparison mode, all measured values are shown. Often profiling is about comparing performance data. It is therefore sometimes
more useful to look at differences or ratios, using one experiment as a reference.
The values shown are relative to this difference. For example if a ratio
is below one, it means the reference value was higher.
In the example below, we use the same two experiments used in thecomparison above. The script is also nearly identical. The only change isthat we now use the ‘delta’ keyword.
As before, the number of lines is restricted to 5 and we focus on the
exclusive timings plus percentages. For the comparison part we are interested
in the differences.
This is the script that produces such an overview:# Define the metrics
metrics e.%totalcpu# Limit the output to 5 lines
limit 5
# Set the comparison mode to differencescompare delta# Get the function overview
functions

Assuming this script file is called my-script-comp2, this is how we
get the table displayed on our screen:$ gprofng display text -script my-script-comp2 mxv.2.thr.er mxv.4.thr.er

Leaving out some of the lines printed, but we have seen before, we
get the following table:
mxv.2.thr.er
Excl. Total
CPU
sec.
%
9.464 100.00
8.961 94.69
0.224
2.37
0.105
1.11
0.073
0.77

mxv.4.thr.er
Excl. Total
CPU
delta
%
+1.640 100.00+1.631 95.39+0.0252.24-0.0110.84-0.0130.54

Name<Total>mxv_core
init_data
erand48_r__drand48_iterate34

GNU gprofng

It is now easier to see that the CPU times for the most time consuming
functions in this code are practically the same.
It is also possible to show ratio’s through the compare ratio command. The first colum is used as a reference and the values for the othercolumns with metrics are derived by dividing the value by the reference. The
result for such a comparison is shown below:
mxv.2.thr.er
Excl. Total
CPU
sec.
%
9.464 100.00
8.961 94.69
0.224
2.37
0.105
1.11
0.073
0.77

mxv.4.thr.er
Excl. Total CPU

x
x
x
x
x

ratio
%
1.173 100.00
1.182 95.39
1.111
2.24
0.895
0.84
0.822
0.54

Name

<Total>mxv_core
init_data
erand48_r__drand48_iterate

Note that the comparison feature is supported at the function, source,and disassembly level. There is no practical limit on the number of experiments that can be used in a comparison.

3.4 Profile Hardware Event Counters
Many processors provide a set of hardware event counters and gprofng provides support for this feature. See Section 7.8 [Hardware Event Counters
Explained], page 63, for those readers that are not familiar with such counters and like to learn more.
In this section we explain how to get the details on the event counter
support for the processor used in the experiment(s), and show several examples.

3.4.1 Getting Information on the Counters Supported
The first step is to check if the processor used for the experiments is supported by gprofng. The -h option on gprofng collect app will show theevent counter information:$ gprofng collect app -h

In case the counters are supported, a list with the events is printed.
Otherwise, a warning message will be issued.
For example, below we show this command and the output on an Intel
Xeon Platinum 8167M (aka “Skylake”) processor. The output has been split
into several sections and each section is commented upon separately.
Run "gprofng collect app --help" for a usage message.
Specifying HW counters on ‘Intel Arch PerfMon v2 on Family 6 Model 85’(cpuver=2499):

Chapter 3: A Mini Tutorial35-h {auto|lo|on|hi}turn on default set of HW counters at the specified rate-h <ctr_def> [-h <ctr_def>]...
-h <ctr_def>[,<ctr_def>]...
specify HW counter profiling for up to 4 HW counters

The first line shows how to get a usage overview. This is followedby some information on the target processor. The next five lines explain inwhat ways the -h option can be used to define the events to be monitored.
The first version shown above enables a default set of counters. Thisdefault depends on the processor this command is executed on. The keyword
following the -h option defines the sampling rate:
auto

Match the sample rate of used by clock profiling. If the latter isdisabled, Use a per thread sampling rate of approximately 100
samples per second. This setting is the default and preferred.

on

Use a per thread sampling rate of approximately 100 samples
per second.

lo

Use a per thread sampling rate of approximately 10 samples per
second.

hi

Use a per thread sampling rate of approximately 1000 samples
per second.

The second and third variant define the events to be monitored. Notethat the number of simultaneous events supported is printed. In this case
we can monitor four events in a single profiling job.
It is a matter of preference whether you like to use the -h option foreach event, or use it once, followed by a comma separated list.
There is one slight catch though. The counter definition below has
mandatory comma (,) between the event and the rate. While a default canbe used for the rate, the comma cannot be omitted. This may result in asomewhat awkward counter definition in case the default sampling rate isused.
For example, the following two commands are equivalent. Note thedouble comma in the second command. This is not a typo.
$ gprofng collect app -h cycles -h insts ...
$ gprofng collect app -h cycles,,insts ...

In the first command this comma is not needed, because a comma(“,”) immediately followed by white space may be omitted.
This is why we prefer the this syntax and in the remainder will use
the first version of this command.
The counter definition takes an event name, plus optionally one ormore attributes, followed by a comma, and optionally the sampling rate.
The output section below shows the formal definition.

36

GNU gprofng<ctr_def> == <ctr>[[~<attr>=<val>]...],[<rate>]

The printed help then explains this syntax. Below we have summarized and expanded this output:
The counter name must be selected from the available counterslisted as part of the output printed with the -h option. On most
systems, if a counter is not listed, it may still be specified by its
numeric value.

<ctr>~<attr>=<val>This is an optional attribute that depends on the processor.
The list of supported attributes is printed in the output. Examples of attributes are “user”, or “system”. The value can given
in decimal or hexadecimal format. Multiple attributes may be
specified, and each must be preceded by a ~.
<rate>The sampling rate is one of the following:
autoThis is the default and matches the rate used byclock profiling. If clock profiling is disabled, use
‘on’.

on

Set the per thread maximum sampling rate to ~100samples/secondlo

Set the per thread maximum sampling rate to ~10
samples/secondhi

Set the per thread maximum sampling rate to ~1000
samples/second<interval>
Define the sampling interval. See Section 3.1.14[Control the Sampling Frequency], page 21, how todefine this.
After the section with the formal definition of events and counters,a processor specific list is displayed. This part starts with an overview ofthe default set of counters and the aliased names supported on this specificprocessor.
Default set of HW counters:-h cycles,,insts,,llm
Aliases for most useful HW counters:alias

raw name

cyclesunhalted-core-cycles

type units regs description
CPU-cycles 0123 CPU Cycles

Chapter 3: A Mini Tutorial
insts
llm
br_mspbr_ins37

instruction-retired
events 0123 Instructions Executedllc-misses
events 0123 Last-Level Cache Misses
branch-misses-retired
events 0123 Branch Mispredict
branch-instruction-retired events 0123 Branch Instructions

The definitions given above may or may not be available on other processors.
The table above shows the default set of counters defined for thisprocessor, and the aliases. For each alias the full “raw” name is given, plus
the unit of the number returned by the counter (CPU cycles, or a raw count),the hardware counter the event is allowed to be mapped onto, and a short
description.
The last part of the output contains all the events that can be monitored:
Raw HW counters:
nameunhalted-core-cycles
unhalted-reference-cycles
instruction-retired
llc-referencellc-misses
branch-instruction-retired
branch-misses-retired
ld_blocks.store_forward
ld_blocks.no_sr
ld_blocks_partial.address_alias
dtlb_load_misses.miss_causes_a_walk
dtlb_load_misses.walk_completed_4k

type

units regs description
CPU-cycles 0123
events 0123
events 0123
events 0123
events 0123
events 0123
events 0123
events 0123
events 0123
events 0123
events 0123
events 0123

<many lines deleted>l2_lines_out.silent
l2_lines_out.non_silent
l2_lines_out.useless_hwpf
sq_misc.split_lock

events 0123
events 0123
events 0123
events 0123

As can be seen, these names are not always easy to correlate to a
specific event of interest. The processor manual should provide more clarity
on this.

3.4.2 Examples Using Hardware Event Counters
The previous section may give the impression that these counters are hard
to use, but as we will show now, in practice it is quite simple.
With the information from the -h option, we can easily set up our
first event counter experiment.
We start by using the default set of counters defined for our processor
and we use 2 threads:

38

GNU gprofng$ exe=mxv-pthreads$ m=8000$ n=4000$ exp=mxv.hwc.def.2.thr.er$ gprofng collect app -O $exp -h auto ./$exe -m $m -n $n -t 2

The new option here is -h auto. The auto keyword enables hardware
event counter profiling and selects the default set of counters defined for thisprocessor.
As before, we can display the information, but there is one practicalhurdle to take. Unless we like to view all metrics recorded, we would need toknow the names of the events that have been enabled. This is tedious andalso not portable in case we would like to repeat this experiment on anotherprocessor.
This is where the special hwc metric comes very handy. It automatically expands to the active set of hardware event counters used in theexperiment(s).
With this, it is very easy to display the event counter values. Notethat although the regular clock based profiling was enabled, we only want
to see the counter values. We also request to see the percentages and limit
the output to the first 5 lines:$ exp=mxv.hwc.def.2.thr.er$ gprofng display text -metrics e.%hwc -limit 5 -functions $exp
Current metrics: e.%cycles:e+%insts:e+%llm:nameCurrent Sort Metric: Exclusive CPU Cycles ( e.%cycles )Print limit set to 5
Functions sorted by metric: Exclusive CPU Cycles
Excl. CPU
Cyclessec.
%
2.691 100.00
2.598 96.54
0.035
1.31
0.026
0.95
0.018
0.66

Excl. Instructions
Executed%
7906475309 100.00
7432724378 94.01
188860269
2.39
73623396
0.9376824434
0.97

Excl. Last-Level
Cache Misses%
122658983 100.00
121745696 99.26
70084
0.06
763116
0.62
40040
0.03

Name

<Total>mxv_core
erand48_r
init_data
drand48

As we have seen before, the first few lines echo the settings. This
includes a list with the hardware event counters used by default.
The table that follows makes it very easy to get an overview where
the time is spent and how many of the target events have occurred.
As before, we can drill down deeper and see the same metrics atthe source line and instruction level. Other than using hwc in the metrics
definitions, nothing has changed compared to the previous examples:Chapter 3: A Mini Tutorial39$ exp=mxv.hwc.def.2.thr.er$ gprofng display text -metrics e.hwc -source mxv_core $exp

This is the relevant part of the output. Since the lines get very long,
we have somewhat modified the lay-out:
Excl. CPU Excl.
Excl.
CyclesInstructions Last-Level
sec.
Executed
Cache Misses
0.
0.
0.
0.
## 1.872
0.725
0.
0.

0

0

0
0
0
7291879319
140845059
000088150571
33595125000<Function: mxv_core>
32. void __attribute__ ((noinline))
mxv_core(...)
33. {
34.
for (uint64_t i=...) {35.
double row_sum = 0.0;36.
for (int64_t j=0; j<n; j++)37.
row_sum += A[i][j]*b[j];38.
c[i] = row_sum;
39.
}
40. }

In a smiliar way we can display the event counter values at the instruction level. Again we have modified the lay-out due to page width limitations:$ exp=mxv.hwc.def.2.thr.er$ gprofng display text -metrics e.hwc -disasm mxv_core $exp
Excl. CPU Excl.
Excl.
CyclesInstructions Last-Level
sec.
Executed
Cache Misses
0.

0

0

0.
0.
0.

0
0
0

0
0
0

0
0
0
12804230
60819025
67221804
7279075109
128042100000321394
386667729407054
87989870
80351
0

0
0

0
0

0.
0.
0.
0.002
0.141
0.582
## 1.871
0.002
0.
0.
0.

<Function: mxv_core>[33] 4021ba: mov0x8(%rsp),%r1034.
for (uint64_t i=...) {[34] 4021bf: cmp%rsi,%rdi[34] 4021c2: jbe
0x37[34] 4021c4: ret
35.
double row_sum = 0.0;36.
for (int64_t j=0; j<n; j++)37.
row_sum += A[i][j]*b[j];[37] 4021c5: mov(%r8,%rdi,8),%rdx[36] 4021c9: mov$0x0,%eax[35] 4021ce: pxor %xmm1,%xmm1[37] 4021d2: movsd (%rdx,%rax,8),%xmm0[37] 4021d7: mulsd (%r9,%rax,8),%xmm0[37] 4021dd: addsd %xmm0,%xmm1[36] 4021e1: add$0x1,%rax[36] 4021e5: cmp%rax,%rcx[36] 4021e8: jne
0xffffffffffffffea
38.
c[i] = row_sum;[38] 4021ea: movsd %xmm1,(%r10,%rdi,8)[34] 4021f0: add$0x1,%rdi

40

GNU gprofng
0.
0.
0.
0.
0.
0.

0
0
0
0
0
0

0
0
0
0
80350
0

0.

0

0

[34] 4021f4: cmp%rdi,%rsi[34] 4021f7: jb
0xd[35] 4021f9: pxor %xmm1,%xmm1[36] 4021fd: test %rcx,%rcx[36] 402200: jne
0xffffffffffffffc5[36] 402202: jmp
0xffffffffffffffe8
39.
}
40. }
[40] 402204: ret

So far we have used the default settings for the event counters. It isquite straightforward to select specific counters. For sake of the example,
let’s assume we would like to count how many branch instructions and retired
memory load instructions that missed in the L1 cache have been executed.
We also want to count these events with a high resolution.
This is the command to do so:$ exe=mxv-pthreads$ m=8000$ n=4000$ exp=mxv.hwc.sel.2.thr.er$ hwc1=br_ins,hi$ hwc2=mem_load_retired.l1_miss,hi$ gprofng collect app -O $exp -h $hwc1 -h $hwc2 $exe -m $m -n $n -t 2

As before, we get a table with the event counts. Due to the very longname for the second counter, we have somewhat modified the output.

$ gprofng display text -limit 10 -functions mxv.hwc.sel.2.thr.er
Functions sorted by metric: Exclusive Total CPU Time
Excl.
Incl.
Excl. Branch Excl.
Name
Total
Total
Instructions mem_load_retired.l1_miss
CPU sec. CPU sec.
Events2.597
2.597
1305305319
4021340<Total>2.481
2.481
1233233242
3982327
mxv_core
0.040
0.107
19019012
9003
init_data
0.028
0.052
23023048
15006
erand48_r
0.024
0.024
19019008
9004__drand48_iterate
0.015
0.067
11011009
2998
drand48
0.008
0.010
0
3002
_int_malloc
0.001
0.001
0
0
brk
0.001
0.002
0
0
sysmalloc
0.
0.001
0
0
__default_morecore

When using event counters, the values could be very large and it is
not easy to compare the numbers. As we will show next, the ratio feature
is very useful when comparing such profiles.
To demonstrate this, we have set up another event counter experiment
where we would like to compare the number of last level cache miss and theChapter 3: A Mini Tutorial

41

number of branch instructions executed when using a single thread, or two
threads.
These are the commands used to generate the experiment directories:$ exe=./mxv-pthreads$ m=8000$ n=4000$ exp1=mxv.hwc.comp.1.thr.er$ exp2=mxv.hwc.comp.2.thr.er$ gprofng collect app -O $exp1 -h llm -h br_ins $exe -m $m -n $n -t 1
$ gprofng collect app -O $exp2 -h llm -h br_ins $exe -m $m -n $n -t 2

The following script has been used to get the tables. Due to lay-out
restrictions, we have to create two tables, one for each counter.

# Limit the output to 5 lines
limit 5
# Define the metrics
metrics name:e.llm# Set the comparison to ratio
compare ratio
functions# Define the metrics
metrics name:e.br_ins# Set the comparison to ratio
compare ratio
functions

Note that we print the name of the function first, followed by thecounter data. The new element is that we set the comparison mode toratio. This divides the data in a column by its counterpart in the reference
experiment.
This is the command using this script and the two experiment directories as input:

$ gprofng display text -script my-script-comp-counters \mxv.hwc.comp.1.thr.er \mxv.hwc.comp.2.thr.er

By design, we get two tables, one for each counter:
Functions sorted by metric: Exclusive Last-Level Cache Misses

Name

mxv.hwc.comp.1.thr.er
Excl. Last-Level
Cache Misses<Total>122709276

mxv.hwc.comp.2.thr.er
Excl. Last-Level
Cache Misses
ratio
x
0.788

42

GNU gprofng
mxv_core
init_data
erand48_r
drand48121796001
723064100111
60065

x
x
x
x

0.787
1.055
0.500
1.167

Functions sorted by metric: Exclusive Branch Instructions

Name

mxv.hwc.comp.1.thr.er
Excl. BranchInstructions<Total>mxv_core
erand48_r
drand48__drand48_iterate

1307307316
1235235239
23023033
20020009
17017028

mxv.hwc.comp.2.thr.er
Excl. BranchInstructions
ratio
x 0.997
x 0.997
x 0.957
x 0.600x 0.882

A ratio less than one in the second column, means that this countervalue was smaller than the value from the reference experiment shown in thefirst column.
This kind of presentation of the results makes it much easier to quickly
interpret the data.
We conclude this section with thread-level event counter overviews,
but before we go into this, there is an important metric we need to mention.
In case it is known how many instructions and CPU cycles have been
executed, the value for the IPC (“Instructions Per Clockycle”) can be computed. See Section 7.8 [Hardware Event Counters Explained], page 63. Thisis a derived metric that gives an indication how well the processor is utilized.
The inverse of the IPC is called CPI.
The gprofng display text command automatically computes the
IPC and CPI values if an experiment contains the event counter values forthe instructions and CPU cycles executed. These are part of the metric list
and can be displayed, just like any other metric.
This can be verified through the metric_list command. If we goback to our earlier experiment with the default event counters, we get thefollowing result.
$ gprofng display text -metric_list mxv.hwc.def.2.thr.er
Current metrics: e.totalcpu:i.totalcpu:e.cycles:e+insts:e+llm:nameCurrent Sort Metric: Exclusive Total CPU Time ( e.totalcpu )Available metrics:
Exclusive Total CPU Time: e.%totalcpu
Inclusive Total CPU Time: i.%totalcpu
Exclusive CPU Cycles: e.+%cycles
Inclusive CPU Cycles: i.+%cycles
Exclusive Instructions Executed: e+%insts
Inclusive Instructions Executed: i+%insts

Chapter 3: A Mini Tutorial

43

Exclusive Last-Level Cache Misses: e+%llm
Inclusive Last-Level Cache Misses: i+%llm
Exclusive Instructions Per Cycle: e+IPC
Inclusive Instructions Per Cycle: i+IPC
Exclusive Cycles Per Instruction: e+CPI
Inclusive Cycles Per Instruction: i+CPI
Size: size
PC Address: address
Name: nameAmong the other metrics, we see the new metrics for the IPC andCPI listed.
In the script below, we use this information and add the IPC andCPI to the metrics to be displayed. We also use a the thread filter to display
these values for the individual threads.
This is the complete script we have used. Other than a different
selection of the metrics, there are no new features.
# Define the metrics
metrics e.insts:e.%cycles:e.IPC:e.CPI# Sort with respect to cycles
sort e.cycles# Limit the output to 5 lines
limit 5
# Get the function overview for all threads
functions# Get the function overview for thread 1
thread_select 1functions# Get the function overview for thread 2
thread_select 2functions# Get the function overview for thread 3
thread_select 3functions

In the metrics definition on the second line, we explicitly request thecounter values for the instructions (e.insts) and CPU cycles (e.cycles)
executed. These names can be found in output from the metric_list
command above. In addition to these metrics, we also request the IPC andCPI to be shown.
As before, we used the limit command to control the number offunctions displayed. We then request an overview for all the threads, followedby three sets of two commands to select a thread and display the function
overview.
The script above is used as follows:44

GNU gprofng$ gprofng display text -script my-script-ipc mxv.hwc.def.2.thr.er

This script produces four tables. We list them separately below, and haveleft out the additional output.
The first table shows the accumulated values across the three threads thathave been active.
Functions sorted by metric: Exclusive CPU Cycles
Excl.
Instructions
Executed
7906475309
7432724378
188860269
73623396
76824434

Excl. CPU
Cyclessec.
%
2.691 100.00
2.598 96.54
0.035
1.31
0.026
0.95
0.018
0.66

Excl.
IPC

Excl.
CPI

Name

1.473
1.434
2.682
1.438
2.182

0.679
0.697
0.373
0.696
0.458

<Total>mxv_core
erand48_r
init_data
drand48

This shows that IPC of this program is completely dominated by function
mxv_core. It has a fairly low IPC value of 1.43.
The next table is for thread 1 and shows the values for the main thread.
Exp Sel Total=== === =====
1 1
3
Functions sorted by metric: Exclusive CPU Cycles
Excl.
Instructions
Executed
473750931
188860269
73623396
76824434
134442832

Excl. CPU
Cyclessec.
%
0.093 100.00
0.035 37.93
0.026 27.59
0.018 18.97
0.013 13.79

Excl.
IPC

Excl.
CPI

Name

2.552
2.682
1.438
2.182
5.250

0.392
0.373
0.696
0.458
0.190

<Total>erand48_r
init_data
drand48__drand48_iterate

Although this thread hardly uses any CPU cycles, the overall IPC of 2.55 is
not all that bad.
Last, we show the tables for threads 2 and 3:
Exp Sel Total=== === =====
1 2
3
Functions sorted by metric: Exclusive CPU Cycles
Excl.
Instructions
Executed
3716362189
3716362189
00

Excl. CPU
Cyclessec.
%
1.298 100.00
1.298 100.00
0.
0.
0.
0.

Excl.
IPC

Excl.
CPI

Name

1.435
1.435
0.
0.

0.697
0.697
0.
0.

<Total>mxv_corecollector_root
driver_mxv

Chapter 3: A Mini Tutorial

45

Exp Sel Total=== === =====
1 3
3
Functions sorted by metric: Exclusive CPU Cycles
Excl.
Instructions
Executed
3716362189
3716362189
00

Excl. CPU
Cyclessec.
%
1.300 100.00
1.300 100.00
0.
0.
0.
0.

Excl.
IPC

Excl.
CPI

Name

1.433
1.433
0.
0.

0.698
0.698
0.
0.

<Total>mxv_corecollector_root
driver_mxv

It is seen that both execute the same number of instructions and takeabout the same number of CPU cycles. As a result, the IPC is the same forboth threads.

3.5 Java Profiling
The gprofng collect app command supports Java profiling. The -j on
option can be used for this, but since this feature is enabled by default, there
is no need to set this explicitly. Java profiling may be disabled through the-j off option.
The program is compiled as usual and the experiment directory iscreated similar to what we have seen before. The only difference with a
C/C++ application is that the program has to be explicitly executed by java.
For example, this is how to generate the experiment data for a Java
program that has the source code stored in file Pi.java:$ javac Pi.java$ gprofng collect app -j on -O pi.demo.er java Pi < pi.in

Regarding which java is selected to generate the data, gprofng firstlooks for the JDK in the path set in either the JDK_HOME environment
variable, or in the JAVA_PATH environment variable. If neither of thesevariables is set, it checks for a JDK in the search path (set in the PATH
environment variable). If there is no JDK in this path, it checks for the java
executable in /usr/java/bin/java.
In case additional options need to be passed on to the JVM, the-J <string> option can be used. The string with the option(s) has to bedelimited by quotation marks in case there is more than one argument.
The gprofng display text command may be used to view the performance data. There is no need for any special options and the same commands as previously discussed are supported.
The viewmode command See Section 7.4 [The Viewmode], page 60,
is very useful to examine the call stacks.

46

GNU gprofng

For example, this is how one can see the native call stacks. For lay-out
purposes we have restricted the list to the first five entries:$ gprofng display text -limit 5 -viewmode machine -calltree pi.demo.er
Print limit set to 5
Viewmode set to machine
Functions Call Tree. Metric: Attributed Total CPU Time
Attr.
Total
CPU sec.
1.381
1.171
0.110
0.110
0.070

Name

+-<Total>+-Pi.calculatePi(double)+-collector_root| +-JavaMain|+-jni_CallStaticVoidMethod

Note that the selection of the viewmode is echoed in the output.

47

4 The gprofng Tools
Several tools are included in gprofng. In subsequent chapters these are discussed in detail. Below a brief description is given, followed by an overview
of the environment variables that are supported.

4.1 Tools Overview
The following tools are supported by gprofng:
gprofng collect app
Collects the performance data and stores the results in an experiment directory. There are many options on this tool, butquite often the defaults are sufficient. An experiment directory
is required for the subsequent analysis of the results.
gprofng display text
Generates performance reports in ASCII format. Commandline
options, and/or commands in a script file are used to control thecontents and lay-out of the generated report(s).
gprofng display html
Takes one or more experiment directories and generates a directory with HTML files. Starting from the index.html file, theperformance data may be examined in a browser.
gprofng display src
Displays the source code, interleaved with the disassembled instructions.
gprofng archive
Archives an experiment directory by (optionally) including
source code and object files, as well as the shared libraries thathave been used.
gprofng display guiThis is an optional component that can be installed in addition
to the command line gprofng tools listed above. It supports thegraphical analysis of one or more experiments that have beencreated using gprofng collect app.
The GUI part of gprofng is a GNU project. This is thelink to the gprofng GUI page (https://savannah.gnu.org/
projects/gprofng-gui). This page contains more information(e.g. how to clone the repo). There is also a tar file distribution
directory (https://ftp.gnu.org/gnu/gprofng-gui) with tar
files that include everything that is needed to build and install
the GUI code. Various versions are available here. Be awarethat in order to build and use the gprofng GUI, Java needs to
be installed first. The minimum Java version required is Java 8.

48

GNU gprofng

4.2 The gprofng.rc file with default settings
The gprofng.rc file is used to define default settings for the gprofng
display text, gprofng archive, and gprofng display src tools, but theuser can override these defaults through local configuration settings whenbuilding and installing from the source code..
There are three files that are checked when the tool starts up. The
first file has pre-defined settings and comes with the installation, but througha hidden file called .gprofng.rc, the user can (re)define the defaults.
These are the locations and files that are checked upon starting theabove mentioned tools:
1. The system-wide filename is called gprofng.rc and is located in the/etc subdirectory in case an RPM was used for the installation..
If gprofng has been built from the source, this file is in subdirectory etc
in the top level installation directory.
2. The user’s home directory may have a hidden file called .gprofng.rc.
3. The directory where gprofng display text (or gprofng display src)
is invoked from, may have a hidden file called .gprofng.rc.
The settings of each file override the settings of the file(s) read before
it. Defaults in the system-wide file are overruled by the file in the user home
directory (if any) and any settings in the .gprofng.rc file in the current
directory override those.
Note that the settings in these files only affect the defaults. Unlike
the commands used in a script file, they are not commands for the tools.
The .gprofng.rc configuration files can contain the addpath,compare, dthresh, name, pathmap, printmode, sthresh, and viewmodecommands as described in this user guide.
They can also contain the following commands, which cannot be usedon either the command line, or in a script file:dmetrics metric-spec
Specify the default metrics to be displayed or printed in thefunction list. The syntax and use of the metric list is describedin section Section 7.3 [Metric Definitions], page 59. The orderof the metric keywords in the list determines the order in which
the metrics are presented.
Default metrics for the callers-callees list are derived fromthe function list default metrics by adding the corresponding
attributed metric before the first occurrence of each metric namein the list.
dsort metric-spec
Specify the default metric by which the function list is sorted.
The sort metric is the first metric in this list that matches aChapter 4: The gprofng Tools

49

metric in any loaded experiment, subject to the following conditions:• If the entry in metric-spec has a visibility string of an exclamation point (‘!’), the first metric whose name matches
is used, regardless of whether it is visible.
• If the entry in metric-spec has any other visibility string,
the first visible metric whose name matches is used.
The syntax and use of the metric list is described in section
Section 7.3 [Metric Definitions], page 59. The default sort metric for the callers-callees list is the attributed metric corresponding to the default sort metric for the function list.
en_desc {on | off | =regex}
Set the mode for reading descendant experiments to ‘on’ (enable
all descendants) or ‘off’ to disable all descendants. If ‘=’regex
is used, enable data from those experiments whose executablename matches the regular expression.
The default setting is ‘on’ to follow all descendants. In reading experiments with descendants, any sub-experiments that
contain little or no performance data are ignored by gprofng
display text.

4.3 Filters
Various filter commands are supported by gprofng display text. Thanks
to the use of filters, the user can zoom in on a certain area of interest. Withfilters, it is possible to select one or more threads to focus on, define a window
in time, select specific call stacks, etc.
While already powerful by themselves, filters may be combined tofurther narrow down the view into the data.
It is important to note that filters are persistent. A filter is active
until it is reset. This means that successive filter commands increasingly
narrow down the view until one or more are reset.
An example is the following:$ gprofng display text -thread_select 1 -functions \-cpu_select 2 -functions ...

This command selects thread 1 and requests the function view for thisthread. The third (cpu_select 2) command adds the constraint that only
the events on CPU 2 are to be selected. This means that the next function
view selects events that were executed by thread 1 and have been runningon CPU 2.
In contrast with this single command line, the two commands below look
similar, but behave very differently:50

GNU gprofng$ gprofng display text -thread_select 1 -functions ...
$ gprofng display text -cpu_select 2 -functions ...

The first command displays the function view for thread 1. The
second command shows the function view for CPU 2 for all threads thathave been running on this CPU.
As the following example demonstrates, things get a little more tricky
in case a script file is used. Consider the following script file:
thread_select 1functions
cpu_select 2functionsThis script file displays the function view for thread 1 first. This isfollowed by those functions that were executed by thread 1 and have been
run on CPU 2.
If however, the script should behave like the two command line invocations shown above, the thread selection filter needs to be reset before
CPU 2 is selected:
thread_select 1functions# Reset the thread selection filter:
thread_select all
cpu_select 2functions

In general, filters behave differently than commands or options. In
particular there may be an interaction between different filter definitions.
For example, as explained above, in the first script file the thread_
select and cpu_select commands interact.
For a list of all the predefined filters see [Predefined Filters], page 55.

4.4 Supported Environment Variables
Various environment variables are supported. We refer to the man page for
gprofng(1) for an overview and description (See Section A.1 [Man page for
gprofng], page 67).

51

5 Performance Data Collection
The gprofng collect app command is used to gather the application performance data while the application executes. At regular intervals, program
execution is halted and the required data is recorded. An experiment directory is created when the tool starts. This directory is used to store therelevant information and forms the basis for a subsequent analysis with oneof the viewing tools.

5.1 The gprofng collect app Tool
This is the command to collect the performance information for the target
application. The usage is as follows:$ gprofng collect app [OPTION(S)] TARGET [TARGET_ARGUMENTS]

Options to the command are passed in first. This is followed by the nameof the target, which is typically a binary executable or a script, followed byany options that may be required by the target.



53

6 View the Performance Information
Various tools to view the performance data stored in one or more experimentdirectories are available. In this chapter, these will all be covered in detail.

6.1 The gprofng display text Tool
This tool displays the performance information in ASCII format. It supports
a variety of views into the data recorded. These views can be specified in
two ways and both may be used simultaneously:• Command line options start with a dash (‘-’) symbol and may take anargument.
• Options may also be included in a file, the “script file”. In this case, thedash symbol should not be included. Multiple script files can be usedon the same command line.
While they may appear as an option, they are really commands andthis is why they will be referred to as commands in the documentation.
As a general rule, the order of options matters and if the same option,or command, occurs multiple times, the rightmost setting is selected.

6.1.1 The gprofng display text Commands
The most commonly used commands are documented in the man page forthis tool (See Section A.3 [Man page for gprofng display text], page 74).
In this section we list and describe all other commands that are supported.

Commands that List Experiment Details
experiment_ids
For each experiment that has been loaded, show the totals of themetrics recorded, plus some other operational characteristics likethe name of the executable, PID, etc. The top line contains theaccumulated totals for the metrics.
experiment_list
Display the list of experiments that are loaded. Each experimentis listed with an index, which is used when selecting samples,
threads, or LWPs, and a process id (PID), which can be used
for advanced filtering.
cpu_list
Display the total number of CPUs that have been used during
the experiment(s).
cpusShow a list of CPUs that were used by the application, alongwith the metrics that have been recorded. The CPUs are rep-

54

GNU gprofng
resented by a CPU number and show the Total CPU time bydefault.
Note that since the data is sorted with respect to the default
metric, it may be useful to use the sort name command to show
the list sorted with respect to the CPU id.

GCEvents
This commands is for Java applications only. It shows any
Garbage Collection (GC) events that have occurred while theapplication was executing..
lwp_list
Displays the list of LWPs processed during the experiment(s).
processes
For each experiment that has been loaded, this command displays a list of processes that were created by the application,
along with their metrics. The processes are represented by process ID (PID) numbers and show the Total CPU time metricby default. If additional metrics are recorded in an experiment,these are shown as well.
samples
Display a list of sample points and their metrics, which reflect
the microstates recorded at each sample point in the loaded
experiment. The samples are represented by sample numbers
and show the Total CPU time by default. Other metrics mightalso be displayed if enabled.
sample_list
For each experiment loaded, display the list of samples currently
selected.
seconds
Show each second of the profiling run that was captured in theexperiment, along with the metrics collected in that second. The
seconds view differs from the samples view in that it shows periodic samples that occur every second beginning at 0 and theinterval cannot be changed.
The seconds view lists the seconds of execution with the Total
CPU time by default. Other metrics might also be displayed if
the metrics are present in the loaded experiments.
threads
Show a list of threads and their metrics. The threads are represented by a process and thread pair and show the Total CPU
time by default. Other metrics might also be displayed by default if the metrics are present in the loaded experiment.

Chapter 6: View the Performance Information55

thread_list
Display the list of threads currently selected for the analysis.
The commands below are for use in scripts and interactive mode only. They
are not allowed on the command line.
add_exp exp-name
Add the named experiment to the current session.
drop_exp exp-name
Drop the named experiment from the current session.
open_exp exp-name
Drop all loaded experiments from the session, and then load thenamed experiment.

Commands that Affect Listings and Output
dthresh value
Specify the threshold percentage for highlighting metrics in the
annotated disassembly code. If the value of any metric is equal
to or greater than value as a percentage of the maximum value ofthat metric for any instruction line in the file, the line on which
the metrics occur has a ‘##’ marker inserted at the beginning of
the line. The default is 75.
printmode {text | html | single-char}
Set the print mode. If the keyword is text, printing will be donein tabular form using plain text. In case the html keyword isselected, the output is formatted as an HTML table.
Alternatively, single-char may be used in a delimiter separated
list, with the single character single-char as the delimiter.
The printmode setting is used only for those commands thatgenerate tables, such as functions. The setting is ignored forother printing commands, including those showing source anddisassembly listings.
sthresh value
Specify the threshold percentage for highlighting metrics in the
annotated source code. If the value of any metric is equal to or
greater than value (as a percentage) of the maximum value ofthat metric for any source line in the file, the line on which themetrics occur has a ‘##’ marker inserted at the beginning of theline. The default is 75.

Predefined Filters
The filters below use a list, the selection list, to define a sequence of numbers.
See Section 7.5 [The Selection List], page 60. Note that this selection ispersistent, but the filter can be reset by using ‘all’ as the selection-list.

56

GNU gprofng

cpu_select selection-list
Select the CPU ids specified in the selection-list.
lwp_select selection-list
Select the LWPs specified in the selection-list.
sample_select selection-list
thread_select selection-list
Select a series of threads, or just one, to be used in subsequentviews. The selection-list consists of a sequence of comma separated numbers. This may include a range of the form ‘n-m’.

Commands to Set and Change Search Paths
addpath path-list
Append path-list to the current setpath settings. Note thatmultiple addpath commands can be used in .gprofng.rc files,
and will be concatenated.
pathmap old-prefix new-prefix
If a file cannot be found using the path list set by addpath, orthe setpath command, one or more path remappings may be
set with the pathmap command.
With path mapping, the user can specify how to replace theleading component in a full path by a different string.
With this command, any path name for a source file, object
file, or shared object that begins with the prefix specified with
old-prefix, the old prefix is replaced by the prefix specified with
new-prefix. The resulting path is used to find the file.
For example, if a source file located in directory /tmp is shown in
the gprofng display text output, but should instead be takenfrom /home/demo, the following pathmap command redefines thepath:$ gprofng diplay text -pathmap /tmp /home/demo -source ...

Note that multiple pathmap commands can be supplied, andeach is tried until the file is found.
setpath path-list
Set the path used to find source and object files. The path isdefined through the path-list keyword. It is a colon separated
list of directories, jar files, or zip files. If any directory has acolon character in it, escape it with a backslash (‘\’).
The special directory name $expts, refers to the set of current
experiments in the order in which they were loaded. You canabbreviate it with a single ‘$’ character.
The default path is ‘$expts:..’ which is the directories of theloaded experiments and the current working directory.

Chapter 6: View the Performance Information

57

Use setpath with no argument to display the current path.
Note that setpath commands are not allowed .gprofng.rc configuration files.

59

7 Terminology
Throughout this manual, certain terminology specific to profiling tools, or
gprofng, or even to this document only, is used. In this chapter this terminology is explained in detail.

7.1 The Program Counter
The Program Counter, or PC for short, keeps track where program execution
is. The address of the next instruction to be executed is stored in a special
purpose register in the processor, or core.
The PC is sometimes also referred to as the instruction pointer, but
we will use Program Counter or PC throughout this document.

7.2 Inclusive and Exclusive Metrics
In the remainder, these two concepts occur quite often and for lack of abetter place, they are explained here.
The inclusive value for a metric includes all values that are part ofthe dynamic extent of the target function. For example if function A calls
functions B and C, the inclusive CPU time for A includes the CPU time spentin B and C.
In contrast with this, the exclusive value for a metric is computed by
excluding the metric values used by other functions called. In our imaginary
example, the exclusive CPU time for function A is the time spent outside
calling functions B and C.
In case of a leaf function, the inclusive and exclusive values for the
metric are the same since by definition, it is not calling any other function(s).
Why do we use these two different values? The inclusive metric shows
the most expensive path, in terms of this metric, in the application. For
example, if the metric is cache misses, the function with the highest inclusive
metric tells you where most of the cache misses come from.
Within this branch of the application, the exclusive metric points tothe functions that contribute and help to identify which part(s) to consider
for further analysis.

7.3 Metric Definitions
The metrics displayed in the various views are highly customizable. In thissection it is explained how to construct the metrics definition(s).
The metrics command takes a colon (‘:’) separated list,
where each item in the list consists of the following three fields:<flavor><visibility><metric-name>.

60

GNU gprofng

The <flavor> field is either ‘e’ for “exclusive”, and/or ‘i’ for “inclusive”. The <metric-name> field is the name of the metric and the <visibility>
field consists of one ore more characters from the following table:.

Show the metric as time. This applies to timing metrics andhardware event counters that measure cycles. Interpret as ‘+’for other metrics.

%

Show the metric as a percentage of the total value for this metric.

+

Show the metric as an absolute value. For hardware event counters this is the event count. Interpret as ‘.’ for timing metrics.

!

Do not show any metric value. Cannot be used with other visibility characters. This visibility is meant to be used in a dmetrics
command to set default metrics that override the built-in visibility defaults for each type of metric.

Both the <flavor> and <visibility> strings may have morethan one character. If both strings have more than one character,the <flavor> string is expanded first.
For example, ie.%user isfirst expanded to i.%user:e.%user, which is then expanded into
i.user:i%user:e.user:e%user.

7.4 The Viewmode
There are different ways to view a call stack in Java. In gprofng, this iscalled the viewmode and the setting is controlled through a command withthe same name.
The viewmode command takes one of the following keywords:
userThis is the default and shows the Java call stacks for Java
threads. No call stacks for any housekeeping threads are shown.
The function list contains a function <JVM-System> that represents the aggregated time from non-Java threads. When the
JVM software does not report a Java call stack, time is reported
against the function <no Java callstack recorded>.

expertShow the Java call stacks for Java threads when the Java codefrom the user is executed and machine call stacks when JVM
code is executed, or when the JVM software does not report aJava call stack. Show the machine call stacks for housekeeping
threads.

machineShow the actual native call stacks for all threads.

7.5 The Selection List
Several commands allow the user to specify a sequence of numbers called theselection list. Such a list may for example be used to select specific threads
from all the threads that have been used when conducting the experiment(s).

Chapter 7: Terminology

61

A selection list (or “list” in the remainder of this section) can bea single number, a contiguous range of numbers with the start and end
numbers separated by a hyphen (‘-’), a comma-separated list of numbers
and ranges, or the all keyword that resets the filter.
Lists must notcontain spaces.
Each list can optionally be preceded by an experiment list with asimilar format, separated from the list by a colon (:). If no experiment list
is included, the list applies to all experiments.
Multiple lists can be concatenated by separating the individual lists
by a plus sign.
These are some examples of various filters using a list:
thread_select 1
Select thread 1 from all experiments.
thread_select all:1
Select thread 1 from all experiments.
thread_select 1:all
Select all the threads from the first experiment loaded.
thread_select 1:2+3:4
Select thread 2 from experiment 1 and thread 4 from experiment3.
cpu_select all:1,3,5
Selects cores 1, 3, and 5 from all experiments.
cpu_select 1,2:all
Select all cores from experiments 1 and 2.
Recall that there are several list commands that show the mapping
between the numbers and the targets.
For example, the experiment_list command shows the name(s) of
the experiment(s) loaded and the associated number. In this example it isused to get this information for a range of experiments:$ gprofng display text -experiment_list mxv.?.thr.er

This is the output, showing for each experiment the ID, the PID, and thename:
ID SelPID Experiment== === ======= ============
1 yes 2750071 mxv.1.thr.er
2 yes 1339450 mxv.2.thr.er
3 yes 3579561 mxv.4.thr.er62

GNU gprofng

7.6 Load Objects and Functions
An application consists of various components. The source code files arecompiled into object files. These are then glued together at link time toform the executable. During execution, the program may also dynamicallyload objects.
A load object is defined to be an executable, or shared object. A
shared library is an example of a load object in gprofng.
Each load object, contains a text section with the instructions generated by the compiler, a data section for data, and various symbol tables.
All load objects must contain an ELF symbol table, which gives the names
and addresses of all the globally known functions in that object.
Load objects compiled with the -g option contain additional symbolic
information that can augment the ELF symbol table and provide information about functions that are not global, additional information about object
modules from which the functions came, and line number information relating addresses to source lines.
The term function is used to describe a set of instructions that represent a high-level operation described in the source code. The term also
covers methods as used in C++ and in the Java programming language.
In the gprofng context, functions are provided in source code format. Normally their names appear in the symbol table representing a set ofaddresses. If the Program Counter (PC) is within that set, the program is
executing within that function.
In principle, any address within the text segment of a load object canbe mapped to a function. Exactly the same mapping is used for the leaf PC
and all the other PCs on the call stack.
Most of the functions correspond directly to the source model of the
program, but there are exceptions. This topic is however outside of the scopeof this guide.

7.7 The Concept of a CPU in gprofng
In gprofng, there is the concept of a CPU. Admittedly, this is not the bestword to describe what is meant here and may be replaced in the future.
The word CPU is used in many of the displays. In the context of
gprofng, it is meant to denote a part of the processor that is capable ofexecuting instructions and with its own state, like the program counter.
For example, on a contemporary processor, a CPU could be a core.
In case hardware threads are supported within a core, a CPU is one of those
hardware threads.
To see which CPUs have been used in the experiment, use the cpu
command in gprofng display text.

Chapter 7: Terminology

63

7.8 Hardware Event Counters Explained
For quite a number of years now, many microprocessors have supported
hardware event counters.
On the hardware side, this means that in the processor there are one
or more registers dedicated to count certain activities, or “events”. Examples
of such events are the number of instructions executed, or the number ofcache misses at level 2 in the memory hierarchy.
While there is a limited set of such registers, the user can map events
onto them. In case more than one register is available, this allows for thesimultaenous measurement of various events.
A simple, yet powerful, example is to simultaneously count the number of CPU cycles and the number of instructions excuted. These two numbers can then be used to compute the IPC value. IPC stands for “Instructions Per Clockcycle” and each processor has a maximum. For example, if
this maximum number is 2, it means the processor is capable of executing
two instructions every clock cycle.
Whether this is actually achieved, depends on several factors, including the instruction characteristics. However, in case the IPC value is well
below this maximum in a time critical part of the application and this cannot
be easily explained, further investigation is probably warranted.
A related metric is called CPI, or “Clockcycles Per Instruction”. It is
the inverse of the CPI and can be compared against the theoretical value(s) of
the target instruction(s). A significant difference may point at a bottleneck.
One thing to keep in mind is that the value returned by a countercan either be the number of times the event occured, or a CPU cycle count.
In case of the latter it is possible to convert this number to time.
This is often easier to interpret than a simple count, but there is one
caveat to keep in mind. The CPU frequency may not have been constant
while the experimen was recorded and this impacts the time reported.
These event counters, or “counters” for short, provide great insight
into what happens deep inside the processor. In case higher level information
does not provide the insight needed, the counters provide the information to
get to the bottom of a performance problem.
There are some things to consider though.
• The event definitions and names vary across processors and it may evenhappen that some events change with an update. Unfortunately andthis is luckily rare, there are sometimes bugs causing the wrong countto be returned.
In gprofng, some of the processor specific event names have an alias
name. For example insts measures the instructions executed. These
aliases not only makes it easier to identify the functionality, but also
provide portability of certain events across processors.

64

GNU gprofng• Another complexity is that there are typically many events one canmonitor. There may up to hundreds of events available and it could require several experiments to zoom in on the root cause of a performance
problem.
• There may be restrictions regarding the mapping of event(s) onto thecounters. For example, certain events may be restricted to specific counters only. As a result, one may have to conduct additional experiments
to cover all the events of interest.
• The names of the events may also not be easy to interpret. In suchcases, the description can be found in the architecture manual for theprocessor.
Despite these drawbacks, hardware event counters are extremely useful and may even turn out to be indispensable.

7.9 What is <apath>?
In most cases, gprofng shows the absolute pathnames of directories. These
tend to be rather long, causing display issues in this document.
Instead of wrapping these long pathnames over multiple lines, we
decided to represent them by the <apath> symbol, which stands for “anabsolute pathname”.
Note that different occurrences of <apath> may represent different
absolute pathnames.

65

8 Other Document Formats
This chapter is applicable when building gprofng from the binutils source.
This document is written in Texinfo and the source text is madeavailable as part of the binutils distribution. The file name is gprofng.texi
and can be found in subdirectory gprofng/doc of the top level binutils
directory.
The default installation procedure creates a file in the info format
and stores it in the documentation section of binutils. This source file canhowever also be used to generate the document in the html and pdf formats.
These may be easier to read and search.
To generate this documentation file in a different format, go to thedirectory that was used to build the tools. The make file to build the otherformats is in the gprofng/doc subdirectory.
For example, if you have set the build directory to be <my-build-dir>,go to subdirectory <my-build-dir>/gprofng/doc.
This subdirectory has a single filed called Makefile that can be used
to build the documentation in various formats. We recommend to use thesecommands.
There are four commands to generate the documentation in the html
or pdf format. It is assumed that you are in directory gprofng/doc underthe main directory <my-build-dir>.
make html Create the html file in the current directory.
make pdf

Create the pdf file in the current directory.

make install-html
Create and install the html file in the binutils documentation
directory.
make install-pdf
Creat and install the pdf file in the binutils documentation directory.
For example, to install this document in the binutils documentation
directory, the commands below may be executed. In this notation, <format>
is one of html, or pdf:
$ cd <my-build-dir>/gprofng/doc$ make install-<format>

The binutils installation directory is either the default /usr/local or
the one that has been set with the --prefix option as part of the configurecommand. In this example we symbolize this location with <install>.
The documentation directory is <install>/share/doc/gprofng incase html or pdf is selected and <install>/share/info for the file in theinfo format.
Some things to note:

66

GNU gprofng• For the pdf file to be generated, the texi2dvi tool is required. It is forexample available as part of the texinfo-tex package.
• Instead of generating a single file in the html format, it is also possible
to create a directory with individual files for the various chapters. To
do so, remove the use of --no-split in variable MAKEINFOHTML in themake file in the <my-build-dir/gprofng/doc directory.

67

Appendix A The gprofng Man Pages
In this appendix the man pages for the various gprofng tools are listed.

A.1 Man page for gprofng
NAME
gprofng - The driver for the gprofng application profiling tool
SYNOPSIS
gprofng [option(s)] action [qualifier] [option(s)] target [options]
DESCRIPTIONThis is the driver for the gprofng tools suite to gather and analyze
performance data.
The driver executes the action specified. An example of an action is‘collect’ to collect performance data. Depending on the action, a
qualifier may be needed to further define the command. The last itemis the target that the command applies to.
There are three places where options are supported. The driver supports
options. These can be found below. The action, possibly in combination
with the qualifier also supports options. A description of these can befound in the man page for the command. Any options needed to executethe target command should follow the target name.
For example, to collect performance data for an application called a.out
and store the results in experiment directory ‘mydata.er’, the following
command may be used:$ gprofng collect app -o mydata.er a.out -t 2

In this example, the action is ‘collect’, the qualifier is ‘app’, the single
argument to the command is -o mydata.er and the target is a.out.
The target command is invoked with the ‘-t 2’ option.
If gprofng is executed without any additional option, action, or target,a usage overview is printed.
OPTIONS--version
Print the version number and exit.
--helpPrint usage information and exit.

ENVIRONMENT
The following environment variables are supported:68

GNU gprofng‘GPROFNG_MAX_CALL_STACK_DEPTH’
Set the depth of the call stack (default is 256).
‘GPROFNG_USE_JAVA_OPTIONS’May be set when profiling a C/C++ application that uses
dlopen() to execute Java code.
‘GPROFNG_ALLOW_CORE_DUMP’
Set this variable to allow a core file to be generated; otherwise an error report is created on ‘/tmp’.
‘GPROFNG_ARCHIVE’
Use this variable to define the settings for automatic archiving upon experiment recording completion.
‘GPROFNG_ARCHIVE_COMMON_DIR’
Set this variable to the location of the common archive.
‘GPROFNG_JAVA_MAX_CALL_STACK_DEPTH’
Set the depth of the Java call stack; the default is 256; set
to 0 to disable capturing of call stacks.
‘GPROFNG_JAVA_NATIVE_MAX_CALL_STACK_DEPTH’
Set the depth of the Java native call stack; the default is
256; set to 0 to disable capturing of call stacks (JNI andassembly call stacks are not captured).
‘GPROFNG_SYSCONFDIR’
Set the path to the gprofng.rc configuration file. By default, this file is placed in the etc subdirectory of the binutils installation directory. In case an RPM has been used
for the installation, this file is in directory /etc.
When building and installing from the source, the user can
set the path to this configuration file to a non-default location. If this is the case, the user may set the GPROFNG_
SYSCONFDIR environment variable to point to this location.
Otherwise, the gp-display-text, gp-display-src, andgp-archive tools cannot find this file.

NOTES
The gprofng driver supports the following commands.
Collect performance data:
gprofng collect app
Collect application performance data.
Display the performance results:
gprofng display text
Display the performance data in ASCII format.

Appendix A: The gprofng Man Pages

69

gprofng display html
Generate an HTML file from one or more experiments.
gprofng display guiStart the GUI. Note that this tool is not available by default
and needs to be installed seperately.
Miscellaneous commands:
gprofng display src
Display source or disassembly with compiler annotations.
gprofng archive
Include binaries and source code in an experiment directory.
It is also possible to invoke the lower level commands directly, but sincethese are subject to change, in particular the options, we recommend touse the driver.
SEE ALSO
gp-archive(1), gp-collect-app(1), gp-display-gui(1), gp-display-html(1),gp-display-src(1), gp-display-text(1)Each gprofng command also supports the --help option. This lists theoptions and a short description for each option.
For example this displays the options supported on the gprofng
collect app command:$ gprofng collect app --help

The user guide for gprofng is maintained as a Texinfo manual. If theinfo and gprofng programs are correctly installed, the command info
gprofng should give access to this document.
COPYRIGHT
Copyright c 2022-2024 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation; with noInvariant Sections, with no Front-Cover Texts, and with no Back-CoverTexts. A copy of the license is included in the section entitled “GNUFree Documentation License”.

70

GNU gprofng

A.2 Man page for gprofng collect app
NAME
gp-collect-app - Collect performance data for the target program
SYNOPSIS
gprofng collect app [option(s)] target [target-option(s)]DESCRIPTION
Collect performance data on the target program. In addition to Program Counter (PC) sampling, hardware event counters and various tracing options are supported.
For example, this command collects performance data for an executablecalled ‘a.out’ and stores the data collected in an experiment directorywith the name ‘example.er’.
$ gprofng collect app -o example.er ./a.out

OPTIONS--version
Print the version number and exit.
--helpPrint usage information and exit.
-v, --verbose
By default, verbose mode is disabled. This option enablesit.
-p {off | on | lo[w] | hi[gh] | <value>}
Disable (‘off’) or enable (‘on’) clock profiling using a default sampling granularity, or enable clock profiling implicitly by setting the sampling granularity (‘lo[w]’, ‘hi[gh]’,or a specific value in ms). By default, clock profiling isenabled (‘-p on’).
-h <ctr_def>[,<ctr_def>]
Enable hardware event counter profiling and select one ormore counter(s). To see the supported counters on thissystem, use the ‘-h’ option without other arguments.
-o <exp_name>Specify the name for the experiment directory. The namehas to end with ‘.er’ and may contain an absolute path(e.g. /tmp/experiment.er). An existing experiment withthe same name will not be overwritten.

Appendix A: The gprofng Man Pages

71-O <exp_name>
This is the same as the ‘-o’ option, but unlike this option,
silently overwrites an existing experiment directory with thesame name.
-C <comment_string>
Add up to 10 comment strings to the experiment. These
comments appear in the notes section of the header andcan be retrieved with the gprofng display text commandusing the ‘-header’ option.
-j {on | off | <path>}
Controls Java profiling when the target is a JVM machine.
The allowed values for this option are:on

Record profiling data for the JVM machine,
and recognize methods compiled by the JavaHotSpot virtual machine. Also record Java callstacks.

off

Do not record Java profiling data. Profilingdata for native call stacks is still recorded.

<path>

Records profiling data for the JVM, and use the
JVM as installed in <path>.

The default is ‘-j on’.
-J <jvm-option(s)>Specifies one or more additional options to be passed to
the JVM used. The jvm-option(s) list must be enclosed inquotation marks if it contains more than one option. The
items in the list need to be separated by spaces or tabs.
Each item is passed as a separate option to the JVM. Notethat this option implies ‘-j on’.
-t <duration>[m|s]Collects data for the specified duration. The duration canbe a single number, optionally followed by either ‘m’ to specify minutes, or ‘s’ to specify seconds, which is the default.
The duration can also consists of two numbers separated
by a minus (−) sign. If a single number is given, data iscollected from the start of the run until the given time.
If two numbers are given, data is collected from the firsttime to the second. In case the second time is zero, data iscollected until the end of the run. If two non-zero numbers
are given, the first must be less than the second.
-n72

GNU gprofng
This is used for a dry run. Several run-time settings are displayed, but the target is not executed and no performancedata is collected.
-F {off|on|=regex}
Control whether descendant processes should have theirdata recorded.
To disable/enable this feature, use‘off’/‘on’. Use ‘=’regex to record data on those processes
whose executable name matches the regular expression.
Only the basename of the executable is used, not the full
path. If spaces or characters interpreted by the shell areused, enclose the regex in single quotes. The default is ‘-Fon’.
-a {off|on|ldobjects|src|usedldobjects|usedsrc}
Specify archiving of binaries and other files. In addition
to disable this feature (‘off’), or enable archiving off all
loadobjects and sources (‘on’), the other options support amore refined selection.
All of these options enable archiving, but the keyword controls what exactly is selected: all load objects (ldobjects),
all source files (src), the loadobjects asscoiated with a program counter (usedldobjects), or the source files associated with a program counter (usedsrc). The default is ‘-aldobjects’.
-S {off|on|<seconds>}
Disable (off), or enable (on) periodic sampling of processwide resource utilization. By default, sampling occurs every
second. Use the <seconds> option to change this. The default is ‘-S on’.
-y <signal>[,r]Controls recording of data with the signal named <signal>,referred to as the pause-resume signal. Whenever the given
signal is delivered to the process, switch between paused (nodata is recorded) and resumed (data is recorded) states.
By default, data collection begins in the paused state. If theoptional ‘r’ is given, data collection begins in the resumed
state and data collection begins immediately.
SIGUSR1 or SIGUSR2 are recommended for this use, but
any signal that is not used by the target can be used.
-l <signal>
Specify a signal that will trigger a sample of process-wide
resource utilization. When the named <signal> is delivered
to the process, a sample is recorded.

Appendix A: The gprofng Man Pages

73

The signal can be specified using the full name, without theinitial letters SIG, or the signal number. Note that the killcommand can be used to deliver a signal.
If both the ‘-l’ and ‘-y’ options are used, the signal must
be different.
-s <option>[,<API>]
Enable synchronization wait tracing, where <option> is used
to define the specifics of the tracing (on, off, <threshold>,or all). The API is selected through the setting for <API>:‘n’ selects native/Pthreads, ‘j’ selects Java, and ‘nj’ selects
both. The default is ‘-s off’.
-H {off|on}
Disable (off), or enable (on) heap tracing. The default is‘-H off’.
-i {off|on}
Disable (off), or enable (on) I/O tracing. The default is ‘-i
off’.
NOTES
Any executable in the ELF (Executable and Linkable Format) object
format can be used for profiling with gprofng. If debug information isavailable, gprofng can provide more details, but this is not a requirement.
SEE ALSO
gprofng(1), gp-archive(1), gp-display-gui(1), gp-display-html(1), gpdisplay-src(1), gp-display-text(1)The user guide for gprofng is maintained as a Texinfo manual. If theinfo and gprofng programs are correctly installed, the command info
gprofng should give access to this document.
COPYRIGHT
Copyright c 2022-2024 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation; with noInvariant Sections, with no Front-Cover Texts, and with no Back-CoverTexts. A copy of the license is included in the section entitled “GNUFree Documentation License”.

74

GNU gprofng

A.3 Man page for gprofng display text
NAME
gp-display-text - Display the performance data in plain text format
SYNOPSIS
gprofng display text [option(s)] [commands] [-script script-file] experiment(s)DESCRIPTION
Print a plain text version of the various displays supported by gprofng.
The input consists of one or more experiment directories. Throughcommands, the user controls the output.
There is a rich set of commands to control the display of the data. The‘NOTES’ section lists the most common ones. The gprofng user guidelists all the commands supported.
Commands specified on the command line need to be prepended withthe dash (’-’) symbol.
In this example, a function overview will be shown, followed by thesource code listing of function ‘my-func’, annotated with the performance metrics that have been recorded during the data collection andstored in experiment directory ‘my-exp.er’:$ gprofng display text -functions -source my-func my-exp.er

Instead of, or in addition to, specifying these commands on the command line, commands may also be included in a file called the script-file.
Note that the commands are processed and interpreted from left toright, so the order matters.
If this tool is invoked without options, commands, or a script file, it
starts in interpreter mode. The user can then issue the commands
interactively. The session is terminated with the exit command in theinterpreter.
OPTIONS--version
Print the version number and exit.
--helpPrint usage information and exit.
-script script-file
Execute the commands stored in the script file. This feature
may be combined with commands specified at the commandline.

Appendix A: The gprofng Man Pages

75

NOTES
Many commands are supported. Below, the more common ones arelisted in mostly alphabetical order, because sometimes it is more logicalto swap the order of two entries.
callers-callees
In a callers-callees panel, it is shown which function(s) callthe target function (the callers) and what functions it iscalling (the callees). This command prints the callerscallees panel for each of the functions, in the order specified
by the function sort metric.
calltree

Display the dynamic call graph from the experiment,
showing the hierarchical metrics at each level.

compare {on | off | delta | ratio}
By default, the results for multiple experiments are aggregated. This command changes this to enable the comparison of experiments for certain views (e.g. the functionview). The first experiment specified is defined to be thereference. The following options are supported:on

For each experiment specified on the command
line, print the values for the metrics that havebeen activated for the experiment.

off

Disable the comparison of experiments. This is
the default.

delta

Print the values for the reference experiment.
The results for the other experiments are shown
as a delta relative to the reference (currentreference).

ratio

Print the values for the reference experiment.
The results for the other experiments areshown as a ratio relative to the reference(current/reference).

disasm function-name
List the source code and instructions for the function specified. The instructions are annotated with the metrics used.
fsingle function-name [‘n’]Write a summary panel for the specified function. The optional parameter n is needed for those cases where several
functions have the same name.
fsummary

Write a summary panel for each function in the function
list.

76

GNU gprofng
functions
Display a list of all functions executed. For each function
the used metrics (e.g. the CPU time) are shown.
headerlimit nlines

Shows several operational characteristics of the experiment(s) specified on the command line.
Limit the output to n lines.
Write a list of source lines and their metrics, ordered by
the current sort metric.

metric_list
Display the currently selected metrics in the function view
and a list of all the metrics available for the target experiment(s).
metrics metric-spec
Define the metrics to be displayed in the function andcallers-callees overviews.
The metric-spec can either be the keyword ‘default’ torestore the default metrics selection, or a colon separated
list with metrics.
A special metric is hwc. It automatically expands to theactive set of hardware event counters used in the experiment(s).
If both instructions and clock cycles have been measured,
the CPI and IPC metrics can be used to see the Clockcycles Per Instruction and Instructions Per Clockcyle values,
respectively.
The gprofng user guide has more details how to define metrics.
name {short | long | mangled}[:{soname | nosoname}]
Specify whether to use the short, long, or mangled formof function names. Optionally, the load object that thefunction is part of can be included in the output by adding
the soname keyword. It can also be ommitted (nosoname),
which is the default.
Whether there is an actual difference between these typesof names depends on the language.
Note that there should be no (white)space to the left andright of the colon (‘:’).
This option should not be confused with the keyword ‘name’in a metric definition, which is used to specify that thenames of functions should be shown in the function overview.

Appendix A: The gprofng Man Pages

77

overview

Shows a summary of the recorded performance data forthe experiment(s) specified on the command line.

pcs

Write a list of program counters (PCs) and their metrics,
ordered by the current sort metric.

sort metric-spec
Sort the function list on the metric-spec given.
The data can be sorted in reverse order by prepending themetric definition with a minus (‘-’) sign.
For example sort -e.totalcpu.
A default metric for the sort operation has been definedand since this is a persistent command, this default can berestored with default as the key (sort default).
source function-name
List the source code for the function specified, annotatedwith the metrics used.
viewmode {user | expert | machine}
This command is only relevant for Java programs. For allother languages supported, the viewmode setting has noeffect.
The following options are supported:
user

Show the Java call stacks for Java threads,
but do not show housekeeping threads.
The function view includes a function
called ‘<JVM-System>’. This represents theaggregated time from non-Java threads. In
case the JVM software does not report a Javacall stack, time is reported against the function‘<no Java callstack recorded>’.

expertShow the Java call stacks for Java threads when
the user Java code is executed, and machine callstacks when JVM code is executed, or when the
JVM software does not report a Java call stack.
Show the machine call stacks for housekeeping
threads.

machineShow the actual native call stacks for all
threads. This is the view mode for C, C++,and Fortran.

SEE ALSO
gprofng(1),gp-archive(1),gp-collect-app(1),gp-display-html(1), gp-display-src(1)gp-display-gui(1),78

GNU gprofng
The user guide for gprofng is maintained as a Texinfo manual. If theinfo and gprofng programs are correctly installed, the command info
gprofng should give access to this document.

COPYRIGHT
Copyright c 2022-2024 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation; with noInvariant Sections, with no Front-Cover Texts, and with no Back-CoverTexts. A copy of the license is included in the section entitled “GNUFree Documentation License”.

Appendix A: The gprofng Man Pages

79

A.4 Man page for gprofng display html
NAME
gp-display-html - Generate an HTML based directory structure tobrowse the profiles
SYNOPSIS
gprofng display html [option(s)] experiment(s)DESCRIPTION
Process one or more experiments to generate a directory containing theindex.html file that may be used to browse the experiment data.
OPTIONS--version
Print the version number and exit.
--helpPrint usage information and exit.
--verbose
Enable verbose mode to show diagnostic messages about theprocessing of the data. By default verbose mode is disabled.
-d [db-vol-size], --debug[=db-vol-size]
Control the printing of run time debug information to assist
with the troubleshooting, or further development of thistool.
The db-vol-size parameter controls the output volume andis one from the list ‘s’, ‘S’, ‘m’, ‘M’, ‘l’, ‘L’, ‘xl’, or ‘XL’. If
db-vol-size is not set, a modest amount of information isprinted. This is equivalent to select ‘s’, or ‘S’. The volumeof data goes up as the size increases. Note that currently‘l/L’ is equivalent to ‘xl/XL’, but this is expected to change
in future updates. By default debug mode is disabled.
--highlight-percentage=value
Set a percentage value in the interval [0,100] to select andcolor code source lines, as well as instructions, that arewithin this percentage of the maximum metric value(s).
The default is 90 (%). A value of zero disables this feature.
-o dirname, --output=dirname
Use dirname as the directory name to store the resultsin.
In absence of this option, the default name is

80

GNU gprofng‘display.<n>.html’. This directory is created in thecurrent directory. The number <n> is the first positive
integer number not in use in this naming scheme. An
existing directory with the same name is not overwritten.
In case the directory exists already, an error message isprinted and the tool terminates.
-O dirname, --overwrite=dirname
Use dirname as the directory name to store the resultsin.
In absence of this option, the default name is‘display.<n>.html’. This directory is created in thecurrent directory. The number <n> is the first positive
integer number not in use in this naming scheme.
An existing directory with the same name is silently
overwritten.
-q, --quiet
Disable the display of all warning, debug, verbose and anyother messages. If enabled, the settings for verbose anddebug are accepted, but ignored. With this option, there isno screen output, other than errors. By default quiet mode
is disabled.
--nowarnings
Disable the printing of warning messages on stdout. By
default warning messages are printed.

NOTES− The options and values are case sensitive.
− In this release, the option syntax has changed to be more compliant
with other tools and commands.
The options that used to have an ‘on’ or ‘off’ value only, now act
as a switch. The option negates the default setting. For example,
by default, verbose mode is disabled. It is enabled by using the‘--verbose’ option.
The long options, those starting with --, that require a value, expect the = sign between the option and the value.
While the previous syntax and choices are accepted still, we
strongly recommend to change the usage of the options according
to the new syntax and values. At some point, these legacy settings
may no longer be accepted.
To assist with the transition, a warning message is shown if thelegacy syntax, or value, or both, are used.
− The ‘-hp’ option is still accepted, but it will be deprecated in a
future release. Use the ‘--highlight-percentage’ option instead.

Appendix A: The gprofng Man Pages

81− When setting a directory name for the HTML files to be stored in,make sure that umask is set to the correct access permissions.
− Regardless of the setting for the warning messages, if there are
warnings, they are accessible through the main index.html page.
− The tool tries to accumulate as many warnings and errors as possible, before taking action. In this way, it is easier to address multiple issues at once. As a result of this approach, it may be thatthe messages do not show immediately. In particular, warnings areshown towards the end of the execution, but one or more errors willterminate execution before the processing begins.
SEE ALSO
gprofng(1),gp-archive(1),gp-collect-app(1),gp-display-src(1), gp-display-text(1)gp-display-gui(1),The user guide for gprofng is maintained as a Texinfo manual. If theinfo and gprofng programs are correctly installed, the command info
gprofng should give access to this document.
COPYRIGHT
Copyright c 2022-2024 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation; with noInvariant Sections, with no Front-Cover Texts, and with no Back-CoverTexts. A copy of the license is included in the section entitled “GNUFree Documentation License”.

82

GNU gprofng

A.5 Man page for gprofng display src
NAME
gp-display-src - Display the source code, optionally interleaved with thedisassembly of the target object
SYNOPSIS
gprofng display src [option(s)] target-file
DESCRIPTION
Display the source code listing, or source code interleaved with disassembly code, as extracted from the target file (an executable, sharedobject, object file, or a Java .class file).
For example, this command displays the source code and disassembly
listing for a function called ‘mxv_core’ that is part of object file ‘mxv.o’:$ gprofng display src -disasm mxv_core mxv.o

To list the source code and disassembly for all the functions in this file,
use the following command:$ gprofng display src -disasm all -1 mxv.o

The target-file is the name of an executable, a shared object, an object
file (.o), or a Java .class file.
If no options are given, the source code listing of the target-file is shown.
This is equivalent to ‘-source all -1’. If this information is not available, a message to this extent is printed.
OPTIONS--version
Print the version number and exit.
--helpPrint usage information and exit.
-functions
List all the functions from the given object.
-source item tag
Show the source code for item in target-file. The tag is used
to differentiate in case there are multiple occurences withthe same name. See the ‘NOTES’ section for the definition ofitem and tag.
-disasm item tag
Include the disassembly in the source listing. The default
listing does not include the disassembly. If the source code

Appendix A: The gprofng Man Pages

83

is not available, show a listing of the disassembly only. See
the ‘NOTES’ section for the definition of item and tag.
-outfile filename
Write results to file filename. A dash (−) writes to stdout.
This is also the default. Note that this option only affects
those options included to the right of the option.
NOTES
Use item to specify the name of a function, or of a source or object filethat was used to build the executable, or shared object.
The tag is an index used to determine which item is being referred towhen multiple functions have the same name. It is required, but will beignored if not necessary to resolve the function.
The item may also be specified in the form ‘function‘file‘’, in whichcase the source or disassembly of the named function in the source
context of the named file will be used.
The special item and tag combination ‘all -1’, is used to indicate generating the source, or disassembly, for all functions in the target-file.
SEE ALSO
gprofng(1),gp-archive(1),gp-collect-app(1),gp-display-html(1), gp-display-text(1)gp-display-gui(1),The user guide for gprofng is maintained as a Texinfo manual. If theinfo and gprofng programs are correctly installed, the command info
gprofng should give access to this document.
COPYRIGHT
Copyright c 2022-2024 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation; with noInvariant Sections, with no Front-Cover Texts, and with no Back-CoverTexts. A copy of the license is included in the section entitled “GNUFree Documentation License”.

84

GNU gprofng

A.6 Man page for gprofng archiveNAME
gp-archive - Archive the associated application binaries and sources fora gprofng experimentSYNOPSIS
gprofng archive [option(s)] experiment
DESCRIPTION
Archive the associated application binaries and source files in a gprofng
experiment to make it self contained and portable.
By default, the binaries are archived as part of the data collection, but
the application source files are not archived. Use this tool to change
this and afterwards archive additional components.
This tool has to be executed on the same system where the profilingdata was recorded.
OPTIONS--version
Print the version number and exit.
--helpPrint usage information and exit.
-a {off | on | ldobjects | src | usedldobjects | used[src]}
Specify archiving of binaries and other files. In addition
to disable this feature (‘off’), or enable archiving of all
loadobjects and sources (‘on’), the other choices support amore refined selection.
All of these choices enable archiving, but the keyword
controls what exactly is selected:
all load objects(‘ldobjects’), all source files (‘src’), the loadobjects
associated with a program counter (‘usedldobjects’),or the source files associated with a program counter(‘used[src]’). The default is ‘-a ldobjects’.
-d pathThe path is the absolute path to a common archive, which
is a directory that contains archived files. If the directory
does not exist, then it will be created. Files are saved in thecommon archive directory, and a symbolic link is created in
the experiment archive.
-FAppendix A: The gprofng Man Pages

85

Force writing, or rewriting of .archive files. All archived
files will be removed and recreated, except if the ‘-n’ or ‘-m’
option is used, or if the experiment is a subexperiment.
-m regex
Archive only those source, object, and debug info files whose
full path name matches the given POSIX compliant regex
regular expression.
-nArchive the named experiment only, not any of its descendants.
-q
Do not write any warnings to stderr. Warnings are incorporated into the .archive file in the experiment directory.
They are shown in the output of the gprofng display text
command.
-r pathThis option specifies the location of a common archive. The
value is the relative path to a common archive, which is adirectory that contains archived files. If the directory does
not exist, then it will be created. Files are saved in thecommon archive directory, and a symbolic link is created in
the experiment archive.
-s selection
Specify archiving of source files. The allowed values forselection are:no
Do not archive any source files.
all
Archive all source and object files that can befound.
used[src]
Archive source and object files for functions
against which data was recorded in the experiment, and that can be found.
By default, application source files are not archived into
the experiment. If the ‘-s all’, or ‘-s used’ option is used,sources and object files are archived. These options also ensure that source files are available in the experiment, even
if the original source files have been modified, or are inaccessible afterwards.

86

GNU gprofng
In case archive files cannot be found, use the ‘addpath’, or‘pathmap’ command, or both, in an .er.rc file to specify
the location of the missing file(s).

NOTES− Archiving of application binaries - By default, binaries are archived
automatically when an experiment is created. However, archiving
does not occur in one or more of the following circumstances:• If the profiled application is terminated before it exits normally.
• If a running process is profiled.
• If archiving is explicitly disabled when profiling. For example
by using the ‘-a off’ option on gprofng collect app.
In these cases, gprofng archive must be run manually and on thesame machine where the profiling data was recorded.
Archiving of experiment data during the data collection process canbe quite expensive. Especially if the experiment has many descendant processes. In such cases, a more efficient strategy is to use
the ‘-a off’ option when collecting the data. Once the collection
has completed, the data can be archived using the ‘-s all’ option.
This saves all executables and source files in the experiment.
If during the archiving there is an error message that an executable,or source file cannot be found, the ‘addpath’ command to add thepath to the missing file(s) can be included in the .er.rc file. After
this command has been added, archive the experiment again. The
archiving archiving can be repeated as many times as necessary toarchive all files.
Archiving should be done on the same system as was used to collect the experiment. If some files cannot be accessed from thissystem (e.g. sources or object files), then additional archiving canbe done using another system that can access them. For example,
the system where the application was built.
Some Java applications store shared objects in jar files. By default,
such shared objects are not automatically archived. To archive
shared objects contained in jar files, make sure to include the‘addpath’ command in an .er.rc file. The ‘addpath’ commandshould give the path to the jar file, including the jar file itself. The.er.rc file should be saved in the user home directory, or experiment parent directory.
− Archiving of application sources - By default, application source
files are not archived in the experiment. Execute the gprofng
archive command with the ‘-s all’, or ‘-s used’ option on eachexperiment to store source files in the experiment.

Appendix A: The gprofng Man Pages

87− Automatic archiving of application sources - Environment variable‘GPROFNG_ARCHIVE’ may be set to automatically archive sources
when the experiment has completed. This environment variable cancontain ‘-s’ and ‘-m’ arguments, as pairs of argument and options,separated by one or more blanks.
If more than one ‘-s’ argument appears on the command line, thelast one prevails. If ‘-s’ is both passed on the command line, andset by the environment variable, the option from the environment
variable prevails.
Note that in case automatic source archiving during data collection
has been enabled using either the ‘GPROFNG_ARCHIVE’ variable, orthe ‘-a src’, or ‘-a usedsrc’ option, it is recommended to confirm that source files have been correctly resolved by executing thegprofng archive -s all, or gprofng archive -s used command.
− The ‘-d’ and ‘-r’ options are mutually exclusive.
− When using the ‘-d’ or ‘-r’ option, environment variable‘GPROFNG_ARCHIVE_COMMON_DIR’ can be used to specify the location of the common archive. This can be very convenient whenusing a script to profile applications.
− If more than one ‘-s’ option is given on the command line, or
specified in the environment variable, the specified option for all
must be the same. If not, gprofng archive exits with an error.
− This tool does not work on experiments recorded with earlier versions of the tools. If invoked on such experiments, a warning is
printed. Use the version of gprofng archive from the same release with which the experiment was recorded.
SEE ALSO
gprofng(1), gp-collect-app(1), gp-display-gui(1), gp-display-html(1), gpdisplay-src(1), gp-display-text(1)The user guide for gprofng is maintained as a Texinfo manual. If theinfo and gprofng programs are correctly installed, the command info
gprofng should give access to this document.
COPYRIGHT
Copyright c 2022-2024 Free Software Foundation, Inc.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation; with noInvariant Sections, with no Front-Cover Texts, and with no Back-CoverTexts. A copy of the license is included in the section entitled “GNUFree Documentation License”.

89

Index
C
Command line mode . . . . . . . . . . . . . . . . . . . . 8
Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Commands, -header. . . . . . . . . . . . . . . . . . . 71
Commands, add_exp. . . . . . . . . . . . . . . . . . . 55
Commands, addpath . . . . . . . . . . . . . . . 56, 86
Commands, callers-callees . . . . . . . . . 75
Commands, calltree . . . . . . . . . . . . . . 18, 75
Commands, compare . . . . 30, 32, 34, 40, 75
Commands, cpu_list . . . . . . . . . . . . . . 28, 53
Commands, cpu_select . . . . . . . . . . . . . . . 56
Commands, cpus . . . . . . . . . . . . . . . . . . 28, 53
Commands, disasm . . . . . . . . . . . . 12, 75, 82
Commands, dmetrics . . . . . . . . . . . . . . . . . 48
Commands, drop_exp . . . . . . . . . . . . . . . . . 55
Commands, dsort . . . . . . . . . . . . . . . . . . . . . 48
Commands, dthresh. . . . . . . . . . . . . . . . . . . 55
Commands, en_desc. . . . . . . . . . . . . . . . . . . 49
Commands, experiment_ids . . . . . . . . . . 53
Commands, experiment_list . . 31, 53, 61
Commands, fsingle . . . . . . . . 19, 22, 23, 75
Commands, fsummary . . . . . . . . . . 22, 23, 75
Commands, functions . . . . . . . . . . 8, 76, 82
Commands, GCEvents . . . . . . . . . . . . . . . . . 54
Commands, header . . . . . . . . . . . . 20, 22, 76
Commands, limit . . . . . . . . . . . . . . 16, 43, 76
Commands, lines . . . . . . . . . . . . . . . . . 11, 76
Commands, lwp_list . . . . . . . . . . . . . . . . . 54
Commands, lwp_select . . . . . . . . . . . . . . . 56
Commands, metric_list . . . 14, 42, 43, 76
Commands, metrics . . . . . . . . . . . 14, 59, 76
Commands, name . . . . . . . . . . . . . . . . . . . . . . 76
Commands, objects . . . . . . . . . . . . . . . 19, 22
Commands, open_exp . . . . . . . . . . . . . . . . . 55
Commands, outfile. . . . . . . . . . . . . . . . . . . 83
Commands, overview . . . . . . . . . . . . . . 20, 77
Commands, pathmap. . . . . . . . . . . . . . . . . . . 56
Commands, pcs. . . . . . . . . . . . . . . . . . . . 13, 77
Commands, printmode . . . . . . . . . . . . . . . . 55
Commands, processes . . . . . . . . . . . . . . . . 54
Commands, sample-select . . . . . . . . . . . . 56
Commands, sample_list . . . . . . . . . . . . . . 54
Commands, samples. . . . . . . . . . . . . . . . . . . 54
Commands, script . . . . . . . . . . . . . . . . 16, 74
Commands, seconds. . . . . . . . . . . . . . . . . . . 54
Commands, setpath. . . . . . . . . . . . . . . . . . . 56
Commands, sort . . . . . . . . . . . . . . . 16, 28, 77
Commands, source . . . . . . . . . . . . 10, 77, 82

Commands, sthresh. . . . . . . . . . . . . . . . . . . 55
Commands, thread_list . . . . . . . . . . 25, 55
Commands, thread_select . . . . . . . . 26, 56
Commands, threads . . . . . . . . . . . . . . . 25, 54
Commands, viewmode . . . . . . . . . . 45, 60, 77
Compare experiments. . . . . . . . . . . . . . . . . . 32
CPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
CPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

D
Default metrics . . . . . . . . . . . . . . . . . . . . . . . . 14

E
ELF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Environment variables . . . . . . . . . . . . . . . . . 68
Environment variables,
GPROFNG_ARCHIVE . . . . . . . . . . . . . . . . . . . 87
Environment variables,
GPROFNG_ARCHIVE_COMMON_DIR . . . . . . . 87
Exclusive metric . . . . . . . . . . . . . . . . . . . . . . . 59
Experiment directory . . . . . . . . . . . . . . . 5, 51

F
Filters, Intro . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Filters, Persistence . . . . . . . . . . . . . . . . . . . . 49
Filters, Reset to default. . . . . . . . . . . . . . . . 61
Filters, Thread selection . . . . . . . . . . . . . . . 26
Flavor field . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

G
gprofng, archive. . . . . . . . . . . . . . . . . . . . . . 47
gprofng, collect app . . . . . . . . . . . . . . . . . 47
gprofng, display gui . . . . . . . . . . . . . . . . . 47
gprofng, display html . . . . . . . . . . . . . . 5, 47
gprofng, display src . . . . . . . . . . . . . . . . . 47
gprofng, display text . . . . . . . . . . . . . . 5, 47
gprofng.rc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

90

GNU gprofng

H

M

Hardware event counters, alias name . . . 63
Hardware event counters,
auto option . . . . . . . . . . . . . . . . . . . . . . . . . 38
Hardware event counters,
counter definition . . . . . . . . . . . . . . . . . . . 35
Hardware event counters, CPI . . . . . . . . . 42
Hardware event counters, CPI metric . . . 76
Hardware event counters, description . . 63
Hardware event counters,
hwc metric . . . . . . . . . . . . . . . . . . . . . . . 38, 76
Hardware event counters, IPC . . . . . . . . . 42
Hardware event counters, IPC metric . . . 76
Hardware event counters, variable
CPU frequency . . . . . . . . . . . . . . . . . . . . . . 63

Metric name field . . . . . . . . . . . . . . . . . . . . . . 59
Metrics, Flavor field . . . . . . . . . . . . . . . . . . . 59
Metrics, Metric name field . . . . . . . . . . . . . 59
Metrics, Reset to default. . . . . . . . . . . . . . . 14
Metrics, Visibility field . . . . . . . . . . . . . 16, 59
Miscellaneous , ## . . . . . . . . . . . . . . . . . . . . . 11
Miscellaneous, <apath> . . . . . . . . . . . . . . . . 22
Miscellaneous, <Total> . . . . . . . . . . . . . . . . . 9
mxv-pthreads . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

IInclusive metric . . . . . . . . . . . . . . . . . . . . . . . 59
Instruction level metrics . . . . . . . . . . . . . . . 12
Instruction pointer . . . . . . . . . . . . . . . . . . . . 59
Interpreter mode . . . . . . . . . . . . . . . . . . . . . . . 8
IPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

JJava profiling, -j on/off . . . . . . . . . . . . . . 45
Java profiling, -J <string> . . . . . . . . . . . . 45
Java profiling, <JVM-System>. . . . . . . . . . . 60
Java profiling, <no Java
callstack recorded> . . . . . . . . . . . . . . . 60
Java profiling, different view modes . . . . 45
Java profiling, JAVA_PATH . . . . . . . . . . . . . . 45
Java profiling, JDK_HOME . . . . . . . . . . . . . . . 45

L
Leaf function . . . . . . . . . . . . . . . . . . . . . . . . . . 59
List specification . . . . . . . . . . . . . . . . . . . . . . 60
Load object . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Load objects . . . . . . . . . . . . . . . . . . . . . . . . . . 22

O
Options, --debug . . . . . . . . . . . . . . . . . . . . . . 79
Options, --help . . . . . 67, 70, 74, 79, 82, 84
Options, --highlight-percentage. . . . . 79
Options, --nowarnings . . . . . . . . . . . . . . . . 80
Options, --output . . . . . . . . . . . . . . . . . . . . . 79
Options, --overwrite . . . . . . . . . . . . . . . . . 80
Options, --quiet . . . . . . . . . . . . . . . . . . . . . . 80
Options, --verbose . . . . . . . . . . . . . . . . 70, 79
Options, --version . . 67, 70, 74, 79, 82, 84
Options, -a . . . . . . . . . . . . . . . . . 72, 84, 86, 87
Options, -addpath . . . . . . . . . . . . . . . . . . . . . 56
Options, -callers-callees . . . . . . . . . . . 75
Options, -calltree . . . . . . . . . . . . . . . . 18, 75
Options, -compare . . . . . . 30, 32, 34, 40, 75
Options, -cpu_list . . . . . . . . . . . . . . . . 28, 53
Options, -cpu_select . . . . . . . . . . . . . . . . . 56
Options, -cpus . . . . . . . . . . . . . . . . . . . . 28, 53
Options, -C . . . . . . . . . . . . . . . . . . . . . . . . 20, 71
Options, -d . . . . . . . . . . . . . . . . . . . . 79, 84, 87
Options, -disasm. . . . . . . . . . . . . . . 12, 75, 82
Options, -dthresh . . . . . . . . . . . . . . . . . . . . . 55
Options, -experiment_ids . . . . . . . . . . . . . 53
Options, -experiment_list . . . . 31, 53, 61
Options, -fsingle . . . . . . . . . . 19, 22, 23, 75
Options, -fsummary . . . . . . . . . . . . 22, 23, 75
Options, -functions . . . . . . . . . . . . 8, 76, 82
Options, -F . . . . . . . . . . . . . . . . . . . . . . . . 72, 84
Options, -GCEvents . . . . . . . . . . . . . . . . . . . 54
Options, -h . . . . . . . . . . . . . . . . . . . . 34, 38, 70
Options, -header . . . . . . . . . . . 20, 22, 71, 76
Options, -H. . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Options, -i. . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Options, -j . . . . . . . . . . . . . . . . . . . . . . . . 45, 71
Options, -J . . . . . . . . . . . . . . . . . . . . . . . . 45, 71
Options, -l. . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Options, -limit . . . . . . . . . . . . . . . . 16, 43, 76
Options, -lines . . . . . . . . . . . . . . . . . . . 11, 76
Options, -lwp_list . . . . . . . . . . . . . . . . . . . 54
Options, -lwp_select . . . . . . . . . . . . . . . . . 56

IndexOptions, -m . . . . . . . . . . . . . . . . . . . . . . . . 85, 87
Options, -metric_list . . . . . 14, 42, 43, 76
Options, -metrics . . . . . . . . . . . . . 14, 59, 76
Options, -n . . . . . . . . . . . . . . . . . . . . . . . . 71, 85
Options, -name . . . . . . . . . . . . . . . . . . . . . . . . 76
Options, -o . . . . . . . . . . . . . . . . . . . . 15, 70, 79
Options, -objects . . . . . . . . . . . . . . . . . 19, 22
Options, -outfile . . . . . . . . . . . . . . . . . . . . . 83
Options, -overview . . . . . . . . . . . . . . . . 20, 77
Options, -O . . . . . . . . . . . . . . . . . 15, 17, 71, 80
Options, -p . . . . . . . . . . . . . . . . . . . . 19, 21, 70
Options, -pathmap . . . . . . . . . . . . . . . . . . . . . 56
Options, -pcs . . . . . . . . . . . . . . . . . . . . . . 13, 77
Options, -printmode . . . . . . . . . . . . . . . . . . 55
Options, -processes . . . . . . . . . . . . . . . . . . 54
Options, -q . . . . . . . . . . . . . . . . . . . . . . . . 80, 85
Options, -r . . . . . . . . . . . . . . . . . . . . . . . . 85, 87
Options, -s . . . . . . . . . . . . . . . . . 73, 85, 86, 87
Options, -sample-select . . . . . . . . . . . . . . 56
Options, -sample_list . . . . . . . . . . . . . . . . 54
Options, -samples . . . . . . . . . . . . . . . . . . . . . 54
Options, -script . . . . . . . . . . . . . . . . . . 16, 74
Options, -seconds . . . . . . . . . . . . . . . . . . . . . 54
Options, -setpath . . . . . . . . . . . . . . . . . . . . . 56
Options, -sort . . . . . . . . . . . . . . . . . 16, 28, 77
Options, -source. . . . . . . . . . . . . . . 10, 77, 82
Options, -sthresh . . . . . . . . . . . . . . . . . . . . . 55
Options, -S. . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Options, -t. . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Options, -thread_list . . . . . . . . . . . . 25, 55
Options, -thread_select . . . . . . . . . . 26, 56
Options, -threads . . . . . . . . . . . . . . . . . 25, 54
Options, -v. . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

91
Options, -viewmode . . . . . . . . . . . . 45, 60, 77
Options, -y. . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

P
PC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59, 62
PC sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Posix Threads . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Program Counter . . . . . . . . . . . . . . . . . . 59, 62
Program Counter sampling . . . . . . . . . . . . . 3
Pthreads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

S
Sampling frequency . . . . . . . . . . . . . . . . . . . . 21
Sampling interval . . . . . . . . . . . . . . . . . . 21, 22
Script files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Selection list . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Sort, Reset to default . . . . . . . . . . . . . . 16, 77
Sort, Reverse order . . . . . . . . . . . . . . . . 16, 77
Source level metrics . . . . . . . . . . . . . . . . . . . 10

T
texi2dvi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Thread affinity . . . . . . . . . . . . . . . . . . . . . . . . 28
Total CPU time . . . . . . . . . . . . . . . . . . . . . . . . 9V
Viewmode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Visibility field . . . . . . . . . . . . . . . . . . . . . . . . . 59