# Presentation: 180572

## Performance EngineeringResearch Institute (PERI)

- Patrick H. Worley
**Computational Earth Sciences Group****Computer Science and Mathematics Division**

## Performance engineering:Enabling petascale science

- Petascale computing is about delivering performance to scientists

**Maximizing performance****is getting harder:**

**PERI addresses this challenge****in three ways:**

**Systems are more complicated**
**O (100 K) processors **
**Multi-core with SIMD extensions**
**Scientific software****is more complicated:**
**Multi-disciplinary and multi-scale**

**Model and predict****application****performance**
**Assist SciDAC****scientific code****projects with****performance****analysis and tuning**
**Investigate novel strategies for automatic performance tuning**

**Cray XT4****at ORNL**

**BeamBeam3D accelerator modeling**

**POP model****of El Nino**

**IBM BlueGene****at LLNL**

- Performance engineering:Enabling petascale science

## SciDAC-1Performance EvaluationResearch Center (PERC): 2001–2006

**In the last two years, added emphasis on optimizing performance of SciDAC applications, including**
**Community Climate****System Model**
**Plasma Microturbulence Project (GYRO, GS2)**
**Omega3P****accelerator model**

**Develop performance-related tools and methodologies for**
**Benchmarking**
**Analysis**
**Modeling**
**Optimization**

- Second phase:

- Initial goals:

## Performance portability is critical:
Codes outlive computing systems.
Scientists can’t publish that they ported and optimized code.
Most computational scientistsare not interested in performance tools:
They want performance experts to work with them.
Such experts are not “scalable,” i.e., they are a limited resource and introduce yet another bottleneck in optimizing code.

**Codes outlive computing systems.**
**Scientists can’t publish that they ported and optimized code.**
**Most computational scientists****are not interested in performance tools:**
**They want performance experts to work with them.**
**Such experts are not “scalable,” i.e., they are a ****limited resource and introduce yet another bottleneck ****in optimizing code.**

- Some lessons learned

## SciDAC-2Performance EngineeringResearch Institute (PERI)

**Providing****near-term****impact****on the performance optimization****of SciDAC applications**

**Providing guidance****in automatic tuning**

**Long-term research goal to improve performance portability**

**Evaluating architectures****and algorithms in****preparation for move****to petascale**

**Relieving the performance optimization burden****from scientific programmers**

**Informing long-term automated tuning efforts**

## Engaging SciDAC software developers

- Application engagement

- Application liaisons

- Tigerteams

**Work directly with****DOE computational scientists **
**Ensure successful performance porting****of scientific software**
**Focus PERI research****on real problems**

**Build long-term personal relationships between PERI researchers and scientific code teams**

**Focus on DOE’s highest priorities**
**SciDAC-2**
**INCITE**
**JOULE**

**Optimizing arithmetic kernels**

**45**

**15**

**30**

**0**

**4**

**16**

**64**

**256**

**1**

**Processors**

**Simulation years per day**

**Load bal., MPI/OpenMP, improved dyn., land, and physics**
**Load bal., MPI/OpenMP, improved dyn., and land**
**V2.0, load bal., MPI/OpenMP**
**V2.0, load balanced, MPI-only**
**V2.0, original (2001) settings**

**IBM p690 cluster, T42L26 benchmark**

**Maximizing scientific throughput**

**1**

**Community Atmosphere Model Performance Evolution**

## FY 2007 application engagement activities

- Application     survey

- Application liaisons

- Tigerteams

**Collect and maintain data on SciDAC-2 and DOE INCITE code characteristics and performance requirements **
**Use data to determine efficient allocation of PERI engagement resources and provide direction for PERI research**
**Provide DOE with data on SciDAC-2 code portfolio**
_***http://icl.cs.utk.edu/peri/***_

**Active engagement (identifying and addressing significant performance issues) with five SciDAC-2 and one INCITE projects, drawn from accelerator, fusion, materials, groundwater, and nanoscience**
**Passive engagement (tracking performance needs and providing advice as requested) with an additional eight SciDAC-2 projects**

**Working with S3D (combustion) and GTC (fusion) code teams to achieve 2007 JOULE report computer performance goals**
**Tiger Team members drawn from across PERI collaboration, currently involving six of the ten PERI institutions**

## Performance modeling

- Modeling is critical for automation of tuning:

- Recent improvements:

**Guidance to the developer:**
**New algorithms, systems, etc.**
**Need to know where****to focus effort:**
**Where are the bottlenecks? **
**Need to know****when we are done:**
**How fast should****we expect to go?**
**Predictions for new systems.**

**Reduced human/system cost.**
**Genetic Algorithms now “learn” application response to system parameters.**
**Application tracing sped up and storage requirements reduced by three orders of magnitude.**
**Greater accuracy**
**S3D (combustion), AVUS (CFD), Hycom (ocean), and Overflow (CFD) codes modeled within 10% average error.**

## Long-term goals for PERI
Obtain hand-tuned performance from automatically generated code for scientific applications
General loop nests
Key application kernels
Reduce the performance portability challenge facing computational scientists
Adapt quickly to new architectures 
Integrate compiler-based and empirical search tools into a framework accessible to application developers
Runtime adaptation of performance-critical parameters

**Obtain hand-tuned performance from automatically generated code for scientific applications**
**General loop nests**
**Key application kernels**
**Reduce the performance portability challenge facing computational scientists**
**Adapt quickly to new architectures **
**Integrate compiler-based and empirical search tools into a framework accessible to application developers**
**Runtime adaptation of performance-critical parameters**

- Automatic performance tuning of scientific code

## Automatic tuning flowchart

| 1: Triage | Where to focus effort |
| --- | --- |
| 2: Semantic analysis | Traditional compiler analysis |
| 3: Transformation | Code restructuring |
| 4: Code generation | Domain- specific code |
| 5: Code selection | Modeling and empirical search |
| 6: Assembly | Choosethe best components |
| 7: Training runs | Performance data for feedback |
| 8: Runtime 
    adaptation | Optimize long-running jobs |

**Persistent database**

**Training****runs**

**Production**
**execution**

**Runtime**
**adaptation**

- Guidance
- Measurements
- Models
- Hardware information
- Sample input
- Annotations
- Assertions

- Runtime performance data

**Domain-specific****code generation**

**External software**

## Model-guided empirical optimization

**transformation modules**

**code variant generation **

- phase 1

- set of parameterized code variants + constraints on unbound parameters

- optimized code +
- representative input data set

**search engine**

- phase 2

- optimized
- code

## The team

** ****David Bailey**
** ****Daniel Gunter**
**Katherine Yelick **

**Lawrence****Berkeley National Laboratory**

**Paul Hovland**
**Dinesh Kaushik**
**Boyana Norris **

**Argonne National Laboratory**

**Sadaf Alam**
**G. Mahinthakumar**
**Philip Roth**
**Jeffrey Vetter**
**Patrick Worley**

**Oak Ridge****National Laboratory**

**Bronis de Supinski**
**Daniel Quinlan **

**Lawrence Livermore National Laboratory**

**Allan Snavely **
**Laura Carrington**

**University of California -****San Diego**

**John****Mellor-Crummey **

**Rice University**

**Jacqueline Chame**
**Mary Hall**
** ****Robert Lucas (P.I.) **

**University of Southern California**

**Rob Fowler**
**Daniel Reed**
**Ying Zhang**

**University of****North Carolina**

**Jack Dongarra**
**Shirley Moore**
**Daniel Terpstra **

**University of Tennessee**

**Jeffrey Hollingsworth **

**University****of Maryland**

## Contacts

-    Worley_PERI_SC07

- Patrick H. Worley
- Computational Earth Sciences Group Computer Science and Mathematics Division
- (865) 574-3128
- worleyph@ornl.gov
- Fred Johnson
- DOE Program ManagerOffice of Advanced Scientific Computing ResearchDOE Office of Science