# Presentation: 649223

## SvPablo and the PERC Performance Tool Suite

**SvPablo and the PERC Performance Tool Suite**

- Evan Welbourne, NERSC/LBNLNUG Meeting, June 4th, 2002

## Credit / References

|  |
| --- |

| Daniel A. Reed, Ruth A. Aydt, Luiz DeRose, Celso L. Mendes, Randy L. Ribler, Eric Shaffer, Huseyin Simitci, Jeffrey S. Vetter, Daniel R. Wells, Shannon Whitmore, and Ying Zhang, "Performance Analysis of Parallel Systems:  Approaches and Open Problems," Joint Symposium on Parallel Processing (JSPP), Nagoya, Japan, June 1998 (invited paper and keynote presentation), pp, 239-256. 

PERC consortium, The Original Proposal for the PERC Project.
Proposal in response to the DOE SciDAC Solicitation 01-07,Mar. 9, 2001

PERC consortium, The Performance Evaluation Research Center (PERC).
slides from the poster session at the SciDAC PI meeting,Jan. 15-16, 2002

PERC consortium, personal contact

Slides 13-38 adapted from:

Celso Mendes, Ying Zhang, Dan Reed, SvPablo: A Toolkit for Performance Analysis of Parallel Systems.slides from tutorial given at the PERC SciDAC meeting during SC2001,
Nov. 2002 |
| --- |

## The Changing face ofHigh Performance Computing

- The Changing face ofHigh Performance Computing

|  |
| --- |

|   As the science and technology of high performance computing advances, our applications and the environment in which they execute is constantly evolving.

  Larger systems:  HPC systems are scaling to a larger and larger   numbers of processors.

  Complex codes:  Codes are often multi-language, and written using   object-oriented, and data-parallel programming languages.

  Diverse environment:  With the advent of the computational grid, our   codes will no longer execute on homogeneous systems, but on a large   heterogeneous system.

 Application performance is a complex function of many variables; it’s    often counter-intuitive, and probably not predictable using first-    principles.  Our performance tools are not suited for the future of HPC.
 |
| --- |

## Requirements for the Success of a Modern Performance Toolkit

|  |
| --- |

|   To succeed in the rapidly evolving world of HPC, a performance toolkit should be:

 Scalable: Tools should scale for use on large systems.

 Portable: Tools should work on every system in the grid.

 Compatible: Tools should be work with most programming languages.

 Versatile: Tools should leverage a variety of HW and SW techniques.

 Experimental: Since analysis from first-principles is unlikely.

  As always, we would like our performance tools to be:

 Easy to use:  Simple and intuitive for users.

 Non-intrusive: Disruption of the normal execution and usage pattern for                           an application should be minimal.

 The PERC project aims to develop a toolkit with these qualities.
 |
| --- |

## The PERC SciDAC Project

|  |
| --- |

|   The Performance Evaluation Research Center (PERC) is an “Integrated Software Infrastructure Center” (ISIC) sponsored under DoE’s SciDAC program.

 Funding: approx. $2.4 million per year.

 Mission:  
Develop a science of performance.
Engineer tools for performance analysis and optimization

 Focus:
Large, grand-challenge calculations, especially SciDAC application projects.

PERC website:  http://perc.nersc.gov |
| --- |

## PERC Participants

|  |
| --- |

|   PERC is a collaboration among eight institutions: four DoE laboratories, and four Universities.

 The PERC project is led by David H. Bailey at NERSC/LBNL.
 PERC’s tool development effort is led by Dan Reed at UIUC. |
| --- |

| Lawrence Berkeley National Lab
 David Bailey Erich Strohmaier | University of Tennessee Jack Dongarra |
| --- | --- |
| Argonne National Lab Paul Hovland Boyana Norris | University of Illinois at Urbana-ChampaignDan Reed |
| Lawrence Livermore National Lab Dan Quinlan Bronis de Supinski Jeffery Vetter | University of Maryland at College ParkJeff Hollingsworth |
| Oak Ridge National Lab Pat Worley Tom Dunigan | San Diego Supercomputing CenterAllan Snavely |

## The PERC Performance Tool Suite

|  |
| --- |

|   The goal of PERC’s tool effort is to produce an interoperable suite of measurement, analysis, and tuning tools that are suited for use on current and future HPC systems.

This goal requires three tightly coupled research efforts:

  End-user tools that integrate various analysis and measurement   approaches, providing a common interface for comparing performance   measurements across platforms and executions and correlating this   data with benchmark and application source code.

  Flexible instrumentation systems for capturing hardware and software   interactions, instruction execution frequencies, memory reference   behavior, and execution overheads.

  Data management infrastructure for tracking performance experiments   and data across time and space.  |
| --- |

## PAPI – Performance API

|  |
| --- |

| 
                       An API for portable hardware measurement                
                       Provides the tool designer with a consistent interface and                     methodology for use of the performance counter hardware                     found in most major microprocessors.

PAPI is available for Linux/x86, Windows 2000, Linux/IA-64, Sun Solaris/Ultra 2.8, IBM AIX/Power, SGI IRIX/MIPS, Compaq Tru64/Alpha Ev6 & Ev67, and Cray T3E/Unicos.

To use PAPI on Seaborg, issue the command:

% module load papi

For more information on PAPI at NERSC, see the NERSC help page:http://hpcf.nersc.gov/software/tools/papi.html 
PAPI project homepage: http://icl.cs.utk.edu/projects/papi/ |
| --- |

## Dyninst API

|  |
| --- |

| 
  An API for dynamic instrumentation at runtime.

  Provides a machine independent interfaceto permit the creation of tools that use runtime code patching. 

  The interface is analogous to a machine independent intermediate representation of instrumentation as an abstract syntax tree. 

  The Dyninst API is available for MIPS (IRIX), Power/PowerPC (AIX), SPARC (Solaris), and x86 (Linux, Solaris and NT).

Dyninst project homepage: http://www.dyninst.org |
| --- |

## Performance Bounds

|  |
| --- |

| 
  A set of tools for establishing bounds on the performance of an application or program construct.

  Will use source code analysis to determine what sections of code are memory bandwidth limited, instruction scheduling limited, etc. on a given architecture.

  The tools will also utilize optional user annotations in order to provide more accurate bounds for performance-critical sections of code |
| --- |

## Sigma

|  |
| --- |

| 
  A tool for memory hierarchy measurement.

  Uses runtime instrumentation to extract a detailed representation of the memory reference pattern of an application.

  The memory reference pattern information will be the input to a collection of post-execution tools that provide insight into memory performance issues such as cache conflicts and memory bandwidth contention.

  The Sigma effort is a joint collaboration between IBM and the University of Maryland.

 |
| --- |

## SvPablo

|  |
| --- |

| 
                          An end-user tool that supports source instrumentation                         and browsing of runtime performance data with a                        graphical user interface.

  Incorporates the APIs and other performance tools to provide a front-end to the PERC tool suite.

SvPablo is available for: SunOS 5.7, SGI Irix 6.5,IBM SP2 AIX 4.3, RedHat Linux 6.1, 6.2, 7.1, Intel Itanium IA-64/RedHat 7.1, and Compaq Alpha OSF1 5.1 

Supports C, Fortran-77, Fortran-90(free and fixed form), HPF, and MPI

SvPablo project homepage: http://www-pablo.cs.uiuc.edu/Software/SvPablo/svPablo.htm |
| --- |

## GUI

- SvPablo Components

|  |
| --- |

|  |
| --- |

- Instrumented
- source code

- Compiler

- SvPablo
- data capture
- library

- Performance
- file

- Per-task
- performance
-  files

- Execution on
- parallel
- architecture

- Virtue time
- tunnel display

- AP sensor
- data
- collector

- Source Code
- Instrumentation

- Linker

- Performance data
- visualization

- SvPabloCombine

- Instrumented
- executable

- Instrumented
- object code

- Source Code

- PAPI Lib

- Autopilot
- Lib

## PROJECT

- SvPablo Model

|  |
| --- |

|  |
| --- |

- Performance contexts

- Source files

- Performance
- data

- Performance
- data

- . . .

- . . .

## Using SvPablo at NERSC

|  |
| --- |

| 
SvPablo v5.1 is installed on NERSC’s Seaborg machine.

To use SvPablo on Seaborg, issue the command:

% module load svpablo

SvPablo’s main window can then be launched with the command:

% runSvPablo &

The SvPabloCombine command may be issued on the command line as well:

% SvPabloCombine  <parameters> |
| --- |

## The SvPablo Main Window

|  |
| --- |

|  |
| --- |

## Project Menu

|  |
| --- |

|  |
| --- |

## New Project Dialog Box

|  |
| --- |

|  |
| --- |

## Performance Context Dialog Box

|  |
| --- |

|  |
| --- |

## Interactive Instrumentation and Analysis

- Interactive Instrumentation and Analysis

|  |
| --- |

| 
Typically done in three steps:

source code instrumentation
program compilation and execution
performance data visualization

Each cycle (1-2-3) corresponds to a Performance-Context

If desired, the cycle can be repeated (multiple performance-contexts)

Steps 1 and 3 are done in the GUI |
| --- |

## GUI

- Step 1: Source Code Instrumentation

|  |
| --- |

|  |
| --- |

- Instrumented
- source code

- Compiler

- SvPablo
- data capture
- library

- Performance
- file

- Per-task
- performance
-  files

- Execution on
- parallel
- architecture

- Virtue time
- tunnel display

- AP sensor
- data
- collector

**Source Code**
**Instrumentation**

- Linker

- Performance data
- visualization

- SvPabloCombine

- Instrumented
- executable

- Instrumented
- object code

- Source Code

- PAPI Lib

- Autopilot
- Lib

## Source Code Instrumentation

|  |
| --- |

| 
  Instrumentable constructs:
 procedure calls
 outer loops

  Basic metrics:
 counts 
 inclusive durations
 exclusive durations

  Optional metrics:
 any metric provided by PAPI
 |
| --- |

## Interactive Instrumentation

|  |
| --- |

|  |
| --- |

**instrumentable**
**constructs**
**( function calls**
**and outer loops )**

## Line by Line Instrumentation

|  |
| --- |

|  |
| --- |

**Instrumentable lines**

**Instrumented lines**

## Line Instrumentation – Multiple Events

|  |
| --- |

|  |
| --- |

## Line Instrumentation – Multiple Events

|  |
| --- |

|  |
| --- |

## Automatic Instrumentation

|  |
| --- |

|  |
| --- |

## GUI

- Step 2: Program Compilation and Execution

|  |
| --- |

|  |
| --- |

- Instrumented
- source code

- Compiler

- SvPablo
- data capture
- library

- Performance
- file

- Per-task
- performance
-  files

- Execution on
- parallel
- architecture

- Virtue time
- tunnel display

- AP sensor
- data
- collector

- Source Code
- Instrumentation

- Linker

- Performance data
- visualization

- SvPabloCombine

- Instrumented
- executable

- Instrumented
- object code

- Source Code

- PAPI Lib

- Autopilot
- Lib

## Changes from Non-Instrumented Case

|  |
| --- |

| 
  Adjust application’s Makefile(s)
- Replace source code filenames
   e.g. prog.c  prog.Context.inst.c
- Compile InstrumentationInit.c and link with it
- Must always instrument main program
- Link with $(SVPABLO) and $(PAPI)

  Execute instrumented executable

  Combine per-task performance files:

% SvPabloCombine –o PerfFile c_SDDF*.asc |
| --- |

## Interface to PAPI Hardware Performance Counters

|  |
| --- |

| 
  User-configured file is read at runtime
- desired PAPI counters are specified in file
- if file unavailable, default counter set is used
- SvPabloLibrary versions: with or without PAPI
- synthesized metrics: e.g. MFLOPS, % branches mispredicted

  8 hardware counters are available on Seaborg, to see the available HW    events, issue the command:

% /usr/common/usg/papi/2.1/src/tests/avail

See the SvPablo user’s guide for details:ftp://www-pablo.cs.uiuc.edu/pub/Pablo.Release.5/Documentation/SvPabloGuide.ps.gz |
| --- |

## GUI

- Step 3: Performance Data Visualization

|  |
| --- |

|  |
| --- |

- Instrumented
- source code

- Compiler

- SvPablo
- data capture
- library

- Performance
- file

- Per-task
- performance
-  files

- Execution on
- parallel
- architecture

- Virtue time
- tunnel display

- AP sensor
- data
- collector

- Source Code
- Instrumentation

- Linker

- Performance data
- visualization

- SvPabloCombine

- Instrumented
- executable

- Instrumented
- object code

- Source Code

- PAPI Lib

- Autopilot
- Lib

## Performance Data Visualization

|  |
| --- |

| 
  Color encoded GUI
- configurable by the user
- no change required to display PAPI data

  Aggregate displays
- mean and standard deviation values across processors
- maximum value and its processor number
- minimum value and its processor number

  Detailed displays
- individual metric values per processor
 |
| --- |

## Performance Visualization

|  |
| --- |

|  |
| --- |

**count,**
**exclusive **
**duration**

## Function Visualization

|  |
| --- |

|  |
| --- |

**selected**
**function**

## Performance Metric Selection

|  |
| --- |

|  |
| --- |

## Source Code Visualization

|  |
| --- |

|  |
| --- |

**metrics**

## Multiple Events Metric Box

|  |
| --- |

|  |
| --- |

**mark for multiple**
**events**

## Performance Statistics

|  |
| --- |

|  |
| --- |

## The Roadmap for the PERC Tool Suite

|  |
| --- |

| Currently:
  Tools are being ported to more HPC platforms.
  Additional functionality is being added to the tools.
  Application groups are using the tools and providing feedback.

Future: 
  Port tools to all major HPC platforms.
  Improve functionality with the help of feedback from application groups.
  Enhance the functionality of the tools and work on interoperation.
  Integrate performance models with the tools.

  The HPC community will be presented with a robust, versatile, and     portable suite of performance tools that is suited to a modern HPC     environment. |
| --- |

## The Roadmap for the PERC Tool Suite

|  |
| --- |

|     |
| --- |

| 
PAPI

   Multi-way multiplexing
   Faster substrates
 | 
Dyninst

  Implement full functionality on   all platforms
  Build infrastructure for use with   parallel applications |
| --- | --- |
| 
Sigma++

  Predict Performance using    trace data
  Compile-time instrumentation      for collection of data-   dependence information | 
Performance Bounds

  Build on infrastructure for   C/C++ and Fortran-77 source   code
  Automate the application of   performance bounding      techniques |

## The Roadmap for the PERC Tool Suite

|  |
| --- |

| 
SvPablo

  Improve infrastructure for supporting Fortran-90 codes

  Develop an infrastructure for supporting C++ codes

  Add support for OpenMP

Interoperate with the other tools and instrumentation systems:

Dyninst: Use the Dyninst API to allow dynamic instrumentation and              analysis at runtime

Performance Bounding: Use to display performance bounds alongside                                       measured performance

Sigma: Enable Sigma instrumentation and visualization of Sigma results |
| --- |