# Presentation: 413247

## The Rice  Performance Tools 

- Rob FowlerJohn Mellor-Crummey,
- Monika Mevenkamp, Nathan Tallent, Gabriel Marin, Xuesong Yuan, Jason Eckhard, Scott Warren, Lynn Wang
- Center for High Performance Software
- Rice University

## Outline

- Background:  Why we’re building performance tools.
- Goals
- Approach
- Success Stories, Status, and Future.
- Examples.

## What we (Rice Parallel Compilers Group) do –
Code optimization
Aggressive (mostly source-to-source) compilation.
Hand application of optimizing transformations.
Try out transformations by hand first.
Work on real codes with algorithm, library, and application developers.
We spend a lot of time (too much?) analyzing  executions.  Why?
Deeply pipelined, out of order, superscalar  processors with non-blocking caches and deep memory hierarchies.
Aggressive ( -O4) and/or idiosyncratic vendor compilers.
What we did –-
Built tools to meet our needs w.r.t.  the run/analyze/tune cycle.
They were so useful that we are now distributing (one-on-one) them
In use at DoD, DoE, and NSF supercomputer facilities
Why I’m here --
To present the tools
To encourage collaborative work.

- Code optimization
- Aggressive (mostly source-to-source) compilation.
- Hand application of optimizing transformations.
- Try out transformations by hand first.
- Work on real codes with algorithm, library, and application developers.
- We spend a lot of time (too much?) analyzing  executions.  Why?
- Deeply pipelined, out of order, superscalar  processors with non-blocking caches and deep memory hierarchies.
- Aggressive ( -O4) and/or idiosyncratic vendor compilers.
- What we did –-
- Built tools to meet our needs w.r.t.  the run/analyze/tune cycle.
- They were so useful that we are now distributing (one-on-one) them
- In use at DoD, DoE, and NSF supercomputer facilities
- Why I’m here --
- To present the tools
- To encourage collaborative work.

- Background

## Problems with Existing Performance Tools

- Most tools are hard to use on big, complex, modular applications. Why?
- Despite 25 years of experience, profilers, etc are dramatically underused.
- Tools (feel like they) are designed to evaluate architectural features, or operating systems, rather than for application tuning.
- Tools have insufficient analytic power.
- Any one performance metric produces myopic view.
- Some metrics are causes.  Some are effects. Comparisons required!
- Ex. – misses, FLOPS, loads, mispredicts, cycles, stall cycles.
- Instruction balances (loads vs FLOPS vs integer ops vs branches).
- Fusion of compiler analysis, simulation, and counter profiles.
- Labor intensive analytic methodology is difficult and/or tedious.
- (Re-)inserting explicit instrumentation and recompiling.
- Manual correlation of data from multiple sources with source code.
- Aggregating per line data into larger scopes.
- Computing synthetic measures, e.g.  loads/flop.
-  Tune/reanalyze cycle slowed or prevented by manual intervention and analysis.

## More Problems

- Language- or compiler-based tools restrict domain of applicability
- Multi-language, multi-module (library.so) applications?
- Cross-platform, cross-compiler comparisons?
- User Interface Issues
- GUI problems
- Vendor specific, both for target and analysis machines.
- Non-portable, non-collaborative visualization.
- Single-metric displays don’t capture underlying problems
- Need for block, loop-level, and user-defined scopes.
- Failure to make compelling cases.
- Hard to convince application developers and/or management with a fat stack of printouts and hand-generated notes.

## Goal:  Build Tools that We Will Use

- Intuitive, top-down user interface for performance analysis.
- Provide information needed for analysis and tuning.
- Platform and language (compiler) independence
- Eliminate manual labor from the analyze, tune, run cycle.

## Approach

- Intuitive, top-down user interface for performance analysis.
- Use familiar hypertext browsing models.
- Provide information needed for analysis and tuning.
- Automatically compute derived performance metrics.
- Assimilate and combine data from multiple sources.
- Examples: FLOPS/cycle, miss ratios, (actual cycles – ideal), static analyses.
- Filters and XMLwriter convert any “profile” into a standard format.
- Platform and language (compiler) independence
- Multiple data sources  Cross Platform Comparisons
- Extract hierarchical program structure from (-g3) binaries.
- Handle  multiple languages: F77, F9x, C, C++, ...
- Use compiler-generated symbol table and object  source maps.
- Permits analysis of libraries, including dynamically loaded.
- Eliminate manual labor from the analyze, tune, run cycle.
- Computation of derived metrics.
- Drive the process with scripts and configuration files.

## The components of the HPCView Toolbox

- hpcview  creates performance database from sources, profiles, 	structure information.
- Netscape, Internet Explorer  initial user interface (static data)
- hpcviewer  Java-based viewer
- bloop  extracts structure from binaries
- Based on EEL, distribution restricted.
- open_analysis  extracts structure from binaries
- Uses Open64 infrastructure, Rice analysis  GPLable
- ptran, ProfileWriter   convert/write data in standard format
- hpcprof  cprof for Linux that generates XML files directly
- Cprof output not intended for down stream analysis
- xprof  prof extended to handle code replication
- Converts DCPI/ProfileMe output
- Better attribution for templates, includes, compiler replicated code
- msgprof  Compatible profiling of library calls that can block, e.g. MPI.
- Scripts and makefiles  glue necessary for automation

## Status

- hpcview  v2.01 “in production”, v2.1 in alpha test
- Netscape, Internet Explorer  available everywhere
- hpcviewer v1.0 in alpha test, Java portability issues surprise us!
- bloop  distributed to Gov. Labs.
- open_analysis  first use this week
- ptran, ProfileWriter   production: ssrun, uprofile, static analyses
- hpcprof  alpha test on P2 and P3 boxes.  Depends on PAPI, etc.
- xprof  works for DCPI/ProfileMe, but consumer tools not started
- msgprof  being used in GrADS project.  Distributable some day.
- Scripts and makefiles  an expanding set

## Near Future

- hpcview  v2.01 “in production”, v2.1 in alpha test (release in weeks)
- Netscape, Internet Explorer  available everywhere
- hpcviewer v1.0 in alpha test, Java portability issues surprise us!
- 		(release in weeks to friendly users for use on Windows)
- bloop  distributed to Gov. Labs.  (also to EEL licence holders)
- open_analysis  first use this week (release in May)
- ptran, ProfileWriter   production: ssrun, uprofile, static analyses
- hpcprof  alpha test on P2 and P3 boxes.  Depends on PAPI, etc.
- 		(will release with v2.1 of hpcview)
- xprof  works for DCPI/ProfileMe, but consumer tools not started
- 		(looking for a student)
- Scripts and makefiles  an expanding set

## Successes and ???

- The tool suite is in daily use at Rice.
- NCOMMAS project, among others.
- Successful ongoing use at Sandia and LANL.
- These have been the results of “hands-on deployment”.
- ASCI codes and “compact applications”
- Proprietary codes.
- Science applications: Ocean Modelling, Quantum Chemistry.
- Version(s) have been made available at NCSA, but haven’t caught on.

## source pane

**source pane**

**Children (loops and stmts)**

**This Loop**

**Enclosing Loop**

**flat, per line view**

## Slide 13

## big parallel cosmology code.

- 1% spent in sock_msg_avail_on_fd
- No real hotspots.

## Comparison of  the number

- of graduated loads on a MIPS
- R10K vs a R12K on the same
- input and binary.

## Estimate stall cycles by 

- Estimate stall cycles by
- comparing  actual vs ideal
- cycles a la MTOOL

## How much better is an EV67?

- This question is ill formed for
- program units that are too
- small.  Compilation issues,
- hardware cost attribution, etc.

## Placeholder for  one or two HPCView Screenshots of ASCI Benchmarks and Apps.

- Indices reordered!

## Thought for the Day

- The Hitchiker's Guide to the Galaxy, in a moment of reasoned lucidity which is almost unique among its current tally of five million, nine hundred and seventy-three thousand, five hundred and nine pages, says of the Sirius Cybernetics Corporation products that “It is very easy to be blinded to the essential uselessness of them by the sense of achievement you get from getting them to work at all.  In other words -  and this is the rock-solid principle on which the whole of the Corporation's galaxywide success is founded -- their fundamental design flaws are completely hidden by their superficial design flaws.”
- (Douglas Adams, "So Long, and Thanks for all the Fish")

## Contacts

- NCSA activities:_ __[http://www.cs.rice.edu/~dsystem/ncsa/index.htm](http://www.cs.rice.edu/~dsystem/ncsa/index.htm)_
- HPCView tools:_ http://www.cs.rice.edu/~dsystem/hpcview/_
- Group: _[hpc](mailto:hpc@cs.rice.edu)__[@](mailto:hpc@cs.rice.edu)__[cs](mailto:hpc@cs.rice.edu)__[.rice.](mailto:hpc@cs.rice.edu)__[edu](mailto:hpc@cs.rice.edu)_
- Rob: _[rjf](mailto:rjf@rice.edu)__[@rice.](mailto:rjf@rice.edu)__[edu](mailto:rjf@rice.edu)_
- John: _[johnmc](mailto:johnmc@rice.edu)__[@rice.](mailto:johnmc@rice.edu)__[edu](mailto:johnmc@rice.edu)_
- David: whalley@cs.fsu.edu