# Presentation: 694086

##  The Performance Evaluation Research Center (PERC)Patrick H. Worley, ORNL

**Participating Institutions:**
- Argonne Natl. Lab.		Univ. of California, San Diego
- Lawrence Berkeley Natl. Lab.	Univ. of Maryland
- Lawrence Livermore Natl. Lab.	Univ. of North Carolina
- Oak Ridge Natl. Lab.		Univ. of Tennessee, Knoxville
**Website:  http://perc.nersc.gov**

## Fishing for Performance

- 	"Give a person a fish; you have fed them for today. Teach a person to fish; and you have fed them for a lifetime." (Chinese proverb)
- 	PERC builds fishing equipment (poles, nets, and dynamite) and develops new fishing techniques for performance analysis, optimization, and modeling. We also do a little teaching and a little fishing.
- 	_PERC Mission:_
- Develop a science of performance.
- Engineer tools for performance analysis and optimization.

## New Fishing Techniques

**Performance Monitoring**
- Flexible performance instrumentation systems
- Performance data management infrastructure
- Tools to tie performance data to user's source code
**Performance Modeling**
- Convolution schemes to generate models from independent application and system characterizations
- Statistical and data fitting models
- Hierarchical phase models
- Performance bounds
**Performance Optimization**
- Source code analysis
- Self-tuning software
- Runtime adaptability

## Fishing Equipment

- _Tools being developed within PERC_
**PAPI**: cross-platform interface to hardware performance counters
**MetaSim Tracer**: tool for acquiring operation counts and memory address stream information
**CONE**: call-graph profiler for MPI applications
**mpiP**: lightweight scalable profiling tool for MPI
**MPIDTRACE**:  a communications (MPI and I/O) tracer
**Performance Assertions**: performance expressions for triggering actions at runtime in application codes
**Dyninst**: framework for modifying programs as they run
**dsd**: memory access pattern identification tool
**EXPERT**: automatic analysis of performance problems in traces
**CUBE**: analysis framework for hierarchical performance data
**MetaSim Convolver**: performance prediction tool for computational phases

## Fishing Equipment

- _Tools being developed within PERC_
**Active Harmony**: software architecture for runtime tuning of distributed applications.
**SvPablo**: graphical environment for instrumenting source code and analyzing performance data
**ROSE**: compiler framework for recognizing and optimizing high-level abstractions
**PBT**: tool for generating performance bounds from source code
- _Tools being leveraged in PERC research_
**DIMEMAS**: network simulator used to examine impact of varying bandwidth, latency, topology  and computational performance
**TAU**: tool suite for automatic instrumentation, data gathering, experiment multiplexing, and performance data storage and analysis
**ATOM**: binary instrumentation package on Alpha systems
**PIN:**  binary instrumentation package on Intel systems

## Recent Research Results

- Selected recent PERC research highlights:
- Scalability analysis in SvPablo
- Identifying and exploiting regularity in memory access patterns using dsd
- Convolution-based performance modeling

## SvPablo Scalability Analysis

**Features**
- Automatic generation of scalability data based on SvPablo performance files
- Detection of bottleneck moves as the number of processors changes
- Scalability Analysis at source code level
- Easy-to-use graphical user interface correlating to the main GUI
**Example**
- Enhanced Virginia Hydrodynamics One (EVH1)
- all source code in F90
- intertask communication via MPI
- Execution Environment
- IBM SP at LBNL/NERSC
- Test runs on 16, 32, 64, 128, 256, 512 processors

## Scalability Analysis: EVH1

- Colored boxes
- corresponding
- to inefficiency

- Colored boxes
- corresponding
- to inefficiency
- Each column
- represents one
- test run

- Line graph showing
- scalability across
- the executions

## Scalability Analysis: EVH1

- Major routine
***sweepy*** scales
- very well across
- the executions

## Exploiting Memory Access Patterns

**dsd: Regularity Measurement Tool**
- Instrumentation
- PAPI for bottlenecks
- DynInst for transitory instrumentation
- user control of detection overhead
- binary instrumentation only (source code not needed)
- Applied to several codes
- Fortran and C
- SPEC, NAS benchmarks, and production codes
- Regularity data used to guide optimizations
- 4% improvement in gzip
- 5% FT improvement

## Metrics and Optimizations

## Optimization of gzip Based on Regularity

- Code restructuring
- Targets TLB misses
- Alternate array access order in fill_window
- Results to the right
- Prefetching (SGI)
- 14% improvement for fill_window
- Overall 4% improvement
- Sensitivity of regularity
- fill_window invariant
- Additional guidance

| Function | TLB Miss% Change | Memory Stall% Change |
| --- | --- | --- |
| fill_window | 34.2 | 6.9 |
| deflate | 7.2 | -3.4 |
| longest_match | 10.8 | 2.5 |
| compress_block | 3.2 | 0.1 |
| send_bits | 5.2 | 0.7 |
| updcrc | 4.5 | 0.1 |
| ct_tally | 4.8 | 1.0 |
| Total | 6.4 | 2.4 |

## Convolution-based Performance Modeling

- Follows an automated “recipe” to generate the model
- Application characteristics gathered with MetaSim Tracer
- Number and types of primitive instructions (flops, loads, stores)
- Memory access patterns and ranges (yielding Cache hit rates)
- Communications patterns (MPI primitives) and message sizes
- I/O patterns and sizes
- System characteristics measured with PMaC HPC Benchmark Suite probes
- EFF_BW
- I/O Bench
- MAPS
- MAPS_CG
- MAPS_Ping
- PEAK

**Performance  Predictions for **
**POP Ocean Code**

**Performance  Predictions for **
**POP Ocean Code**

- Seconds per simulation day

| Application Test Case | HPC System | Number of Processors | Predicted Timing (sec) | Actual Timing (sec) | Percent Error |
| --- | --- | --- | --- | --- | --- |
| HYCOM Standard | IBM P3 | 59 | 4,558 | 4,359 | 4.6% |
| HYCOM Standard | IBM P4 | 59 | 2,081 | 1,799 | 15.7% |
| HYCOM Standard | HP SC45 | 59 | 2,382 | 1,960 | 21.5% |
| HYCOM Standard | IBM P3 | 96 | 2,605 | 2,624 | -0.7% |
| HYCOM Standard | IBM P4 | 96 | 1,205 | 1,210 | -0.4% |
| HYCOM Standard | HP SC45 | 96 | 1,432 | 1,378 | 3.9% |
| HYCOM Large | IBM P3 | 234 | 5,612 | 5,311 | 5.7% |
| HYCOM Large | IBM P4 | 234 | 2,692 | 2,062 | 30.6% |
| HYCOM Large | HP SC45 | 234 | 3,043 | 3,300 | -7.8% |
|  |  |  |  |  |  |

**Performance Predictions for HYCOM Ocean Code**

- Results of “blind” predictions using convolution recipe
- (predictions made by PMaC, confirmed by 3rd party at DoD HPCMO)

- Errors in the model were reduced below 10% in all cases *post-mortem*.

## Fish Stories(Recent Application Impacts)

- PERC analysis and optimization techniques have recently led to
- the following (easily quantifiable) performance improvements:
- AGILE-BOLTZTRAN (TSI):  performance improved up to 55% on IBM by eliminating redundant computations
- ZEUS-MP (TSI): single processor performance improved by 90% on IBM by changing compiler options
- GS2 (PMP): performance improved by 300% on IBM by changing data decomposition
- POP (CCSM): performance improved over 300% on Cray by modifying communication algorithms and correcting OS performance problems
- CAM (CCSM): performance improved by 10-30% from new load balancing schemes (in addition to performance optimizations reported last year).

## SciDAC code developed to study low-frequency turbulence in magnetized plasma. Typical use
assess the microstability of plasmas
calculate key properties of the turbulence
simulate turbulence in plasmas which occur in nature
Initial analysis
IBM Power3 (NERSC Seaborg: 8 nodes, 16 processors/node)
Runtime adaptation using Active Harmony
Performance (execution time) improvement by changing layout
55.06s =>16.25s (without collisions) 
71.08s => 31.55s (with collisions)
Work in progress
Tuning time spent in communication

- assess the microstability of plasmas
- calculate key properties of the turbulence
- simulate turbulence in plasmas which occur in nature
- Initial analysis
- IBM Power3 (NERSC Seaborg: 8 nodes, 16 processors/node)
- Runtime adaptation using Active Harmony
- Performance (execution time) improvement by changing layout
- 55.06s =>16.25s (without collisions)
- 71.08s => 31.55s (with collisions)
- Work in progress
- Tuning time spent in communication

**Runtime Optimization of GS2**

## Evolution of Ocean Model Performance

- Performance analysis
- motivated code
- modifications and
- identified OS
- optimizations
- that increased
- performance by more
- than a factor of 3 from
- May 7 baseline.

## Evolution of Atmospheric Model Performance as of  April 2003

- Last year’s results
- for 64x128x26 grid
- and 3 advected fields.
- Since then, resolution
- has increased to
- 128x256x26, 11
- advected fields and
- parameterizations for
- many new processes.

## Evolution of Atmospheric Model Performance as of  March 2004

- New resolution decreased
- simulation rate significantly.
- Previous optimizations
- continue to be important,
- But new opportunities for
- optimization arise as
- science in code evolves
- and as code is ported to
- new platforms.

## Using Visualization to Identify Load Imbalance

- Graphs show processor
- utilization on the Cray X1
- before and after enabling load
- balancing in the Community
- Atmospheric Model. The
- visualization was used to
- motivate using a load
- balancing option that
- has not been useful on
- other platforms.

## Fishing Lessons

- The PERC tutorial was presented at ScicomP’03 and SC’03, and
- will be taught again at Sigmetrics/Performance at Columbia
- University, N.Y. during the week of June 12th. PERC will also
- be happy to present the tutorial at SciDAC project meetings. The
- tutorial can also be downloaded from the PERC website:
**	****http://perc.nersc.gov**

## Looking to the Future:The Massively Parallel Challenge

- Systems featuring 10,000+ CPUs, present daunting challenges for performance analysis and tools:
- What performance phenomena should we measure?
- How can we collect and manage performance data spewed out by tens of thousands of CPUs?
- How can we visualize performance phenomena on 10,000+ CPUs?
- How can we identify bottlenecks in these systems?
*Intelligent, highly automated performance tools, applicable over a*
*wide range of system sizes and architectures, are needed.*

## Looking to the Future:Benchmarking and Modeling

- How can a center meaningfully procure a system with 10,000+ CPUs, a system 10 to 100 times more powerful than any system currently in existence?
- How can we define a benchmark that provides meaningful results for systems spanning four orders of magnitude in size?
- Can we assess the programming effort required to achieve these extra-high levels of performance?
*Reliable performance modeling techniques, usable with modest *
*effort and expertise, offers the best hope for a solution here.*
*The DARPA High-Productivity Computing Systems (HPCS)*
*program is also examining benchmarks for future high-end*
*systems.*

## Looking to the Future: System Simulation

- “Computational scientists have become expert in simulating every phenomena except for the systems they run on.” -- Speaker at recent HPC workshop.
- System simulations heretofore have been used sparingly in system studies, because of the great cost and difficulty in parallelization of such simulations.
*Such simulations are now feasible, due to:*
*Availability of large-scale parallel systems*
*Developments in the parallel discrete event simulation field*

## Looking to the Future:User-Level Automatic Tuning

- Self-tuning software technology has already been demonstrated in a few large-scale libraries:
- FFTW  (MIT).
- LAPACK-ATLAS  (Univ. of Tennessee).
*Near term: Adapting these techniques to a wider group of widely used scientific libraries.*
*Mid term: Automatically incorporate simple performance models into user application codes.*
*Future goal: Automatically incorporate simple run-time tests, using compiler technology, into user application codes.*

## Working with PERC

**For further information:   **
**http://perc.nersc.gov**