SUPER!

The SciDAC-3 Institute for Sustained
Performance, Energy, and Resilience(SUPER)David H Bailey, Lawrence Berkeley National Lab, USA
http://crd.lbl.gov/~dhbailey!

1Why is performance importantSUPER!

· Peta- and exa-scale computing has opened up new vistas for scientificcomputing.
· BUT: Achieved performance is often poor – typically only 1-5% of peak.
· Systems are more complicated:§ 100,000 or more multi-core CPUs.
§ GPU accelerators.

· Codes are more complicated:§ Multi-disciplinary.
§ Multi-scale.

· Low performance is unacceptable, not only because of the high purchasecost of state-of-the-art systems, but also because of the increasing cost of
maintenance and electrical power for these systems.

2Predecessor organizations
SUPER!

Performance Evaluation Research Center (PERC):· 8-institution SciDAC-1 consortium, in operation 2001-2006.
· Led by DHB of LBNL.
· Focus areas:
§
§
§

Benchmarking.
Modeling.
Understanding

Performance Engineering Research Institute (PERI):· 10-institution SciDAC-2 collaboration, in operation 2006-2011.
· Led by Robert F. Lucas of USC / ISI and David H. Bailey of LBNL.
· Focus areas:§§§Performance modeling.
Application engagement.
Semiautomatic tuning research.

3SUPER!

The Sustained Performance, Energyand Resilience (SUPER) Institute· Started September 2011, with funding from DOE SciDAC-3 program.
· Director: Robert F. Lucas of USC/ISI and David H. Bailey of LBNL.
· Focus areas:
§
§
§
§
§
§
§

Automatic performance tuning.
Energy efficient-computing.
Resilient computing.
Combined optimization of performance, energy efficiency and resilience.
Engagement with other institutes and scientific application projects.
Tool integration.
Outreach and tutorials.

4

The SUPER teamSUPER!

ANLLBNL

LLNL

OregonORNLPaul Hovland
Boyana Norris
Stefan Wild

David Bailey
Lenny Oliker
Sam WilliamsBronisde Supinski
Daniel Quinlan

Allen Malony
Sameer Shende

Gabriel Marin
Philip RothPatrick Worley

UCSD

UMD

UNC

USC

UTK

Utah

Laura Carrington

Jeffrey
Hollingsworth

Rob Fowler
Allan Porterfield

Jacque Chame
Robert Lucas (PI)

Shirley Moore
Dan Terpstra

Mary Hall
Chun Chen5

Energy minimization
SUPER!

· New activity, led by Laura Carrington of UC San Diego.
· Objective: Help users and centers to conserve energy while at the sametime achieve good performance.
· Approach:§§§§§§§

Develop new energy-aware application-programmer interfaces (APIs) forusers.
Obtain more precise data regarding energy consumption.
Extend PAPI to sample hardware energy monitors.
Build some prototype PowerMon devices.
Extend performance models to encompass energy consumption.
Transform codes to minimize energy consumption.
Inform systems to exploit dynamic voltage frequency scaling (DVFS).

6

ResilenceSUPER!

· New activity, led by Bronis de Supinski of LLNL.
· Objective: Automate vulnerability assessment for scientific computing.
· Approach:§§§§§§§

Build on success of PERI autotuning.
Conduct fault injection experiments.
Determine which code regions or data structures fail catastrophically.
Determine what transformations enable them to survive.
Extend ROSE compiler to implement the transformations.
Investigate directive-based API for users.
Augments empirically derived vulnerability assessment.

7Performance is our middle nameSUPER!

· Performance measurement:
§
§
§
§

Adopt University of Oregon’s Tau system.
Extend PerfDMF to enable online collection and analysis.
Collaborate with Rice and HPCToolkit.
Develop and utilize a performance database.

· Performance modeling:§§§

Refine ANL’s PBound and LBNL’s Roofline performance models.
Extend MIAMI to model impact of architectural variation.
Extend UCSD’s PSINS to model communication.

8

Automatic performance tuning
SUPER!

· Started in PERI project; continuing in SUPER, led by Mary Hall.
· Challenge: We have found that application scientists are reluctant tolearn and use performance tools in day-to-day research work.
§We called a party, but nobody came.

· Solution: Extend semi-automatic performance tuning methods, such as
those developed for FFTW (FFTs) and ATLAS (dense matrices), togeneral large-scale scientific codes.
· Approach:§§§§§§

Extend basic autotuning technology developed in PERI to a broader set ofapplications and architectures.
Use Tau-based front-end for “triage.”
Extend OpenMP-CHiLL framework to multicore systems.
Use Active Harmony as the search engine.
Targeted autotuning: users write simple code, leaving tuning to us.
Whole program autotuning: parameters, algorithm choice, libraries, etc.

9SUPER!

The SUPER autotuning framework

TAU (Oregon)!
HPC Toolkit (Rice)!
ROSE (LLNL)!
CHiLL (USC/ISI and Utah)!
ROSE (LLNL)!
Orio (Argonne)!

OSKI (LBNL)!

{Active Harmony (UMD)!
GCO (UTK)!

PerfTrack (LBNL, SDSC, RENCI)!

10

Applications of SUPER-PERI research
SUPER!

· LBHMD (lattice Boltzmann) and GTC (plasma toroidal):§§

LBMHD: Up to 3X speedup via autotuning.
GTC: Up to 1.77X speedup via autotuning.

· S3D (combustion):§§

12.7% overall performance improvement.
762,000 CPU-hours are potentially saved each year.

· PFLOTRAN (subsurface reactive flows):§§

2X speedup on two key PETSc routines via autotuning.
40X speedup in initialization; 4X improvement in I/O stage; overall 5X.

· SMG2000 (groundwater diffusion):§

2.37X speedup on one key kernel; overall 27% improvement.

· Nek5000 (turbulence):§

Up to 1.93X speedup.

· LS3DF (electronic structure):§§

Increased scalability from 1000-2000 to over 160,000 cores.
Achieved 442 Tflop/s on Jaguar.
11Autotuning LBMHD and GTC
SUPER!

· LBMHD (left): Implements a lattice Boltzmann method for magnetohydrodynamic
plasma turbulence simulation.
· GTC (right): A gyrokinetic toroidal code for plasma turbulence modeling.
threading of stream()threading of collision()
auto-tuned (ISA-specific)
auto-tuned (portable C)
reference

2.00

GFlop/s per Core

1.75
1.50
1.25
1.00
0.75
0.50
0.25

IntrepidFranklin

16GB

4GB

1GB

4GB

1GB

1GB

0.00

HopperCredit: Samuel Williams, LBNL
1235

Performance per Node relative to BGP

2.25

Optimized
Reference30
1.35x25
20

1.77x
15

1.02x

105
1.22x
0
BGP XE6Intel Fermi
Intrepid Hopper Cluster Cluster

S3DSUPER!

· Performs direct numerical simulation ofturbulent combusion.
· Developed at CRF/Sandia.
· Multiphysics (sprays, radiation, soot).
· Consumes 6,000,000 CPU-hours at NCCS.
· Tier 1 pioneering application for Jaguar.
· Compressible Navier-Stokes equations.
· High fidelity numerical methods:§§

8th order finite difference.
4th order explicit Runge-Kutta integrator.

· Hierarchy of molecular transport models.

!

Figures courtesy of S3D PI, Jacqueline H. Chen, SNL13

Performance tuning of S3DSUPER!

· Used performance models to identify improved compiler settings.
· Identified central importance of exp intrinsic:§§Developed custom exp function than ran faster than library exp.
Subsequently replaced with improved Cray library exp.

· Used tools from Rice University to improve performance:§§

Used HPCToolkit to identify and to assess bottlenecks.
LoopTool helped automate tedious code transformations.

Bottom line:· Achieved 12.7% overall improvement.
§§Node performance increased from 15% of peak to 17.4% of peak.
Estimated potential annual savings: 762,000 CPU-hours.

14

SMG2000
SUPER!

· SMG2000: Semicoarsening multigrid code, used for various applications,including modeling of groundwater diffusion.
· PERI researchers integrated several tools, then developed a “smart”
search technique to find an optimal tuning strategy among 581 million
different choices.
· Achieved 2.37X performance improvement on one key kernel.
· Achieved 27% overall performance improvement.

15

Autotuning the central SMG2000 kernelSUPER!

Outlined code (from ROSE outliner)!
for (si = 0; si < stencil_size; si++) !
for (kk = 0; kk < hypre__mz; kk++) !
for (jj = 0; jj < hypre__my; jj++) !
for (ii = 0; ii < hypre__mx; ii++) !
rp[((ri+ii)+(jj*hypre__sy3))+(kk*hypre__sz3)] -= !
((Ap_0[((ii+(jj*hypre__sy1))+ (kk*hypre__sz1))+!
(((A->data_indices)[i])[si])])* !
(xp_0[((ii+(jj*hypre__sy2))+(kk*hypre__sz2))+(( *dxp_s)[si])])); !
CHiLL transformation recipe !
permute([2,3,1,4])!
tile(0,4,TI)!
tile(0,3,TJ)!
tile(0,3,TK) !
unroll(0,6,US) !
unroll(0,7,UI)!
!
Credit: Mary Hall, Utah!

Constraints on search!
0 ≤ TI , TJ, TK ≤ 122 !
0 ≤ UI ≤ 16 !
0 ≤ US ≤ 10 !
compilers ∈ {gcc, icc} !
!
Search space: !
1223x16x10x2 = 581,071,360 points!
16

SUPER!

Search for optimal tuning parameters in
SMG200 kernel

Parallel heuristic search (using Active Harmony) evaluates 490 total points andconverges in 20 steps.

Selected parameters:
TI=122,TJ=106,TK=56,UI=8,US=3,Comp=gccPerformance gain on residual computation:
2.37X
Performance gain on full application:
27.23% improvement

Credit: Mary Hall, University of Utah

1717

SUPER!

Autotuning the triangular solve kernelof the Nek5000 turbulence code

Compiler

Original

Active Harmony

Exhaustive

Time

Time

(u1,u2)

SpeedupTime(u1,u2)

Speeduppathscale

0.58

0.32

(3,11)

1.81

0.30(3,15)

1.93

gnu

0.71

0.47

(5,13)

1.51

0.46

(5,7)

1.54

pgi

0.90

0.53

(5,3)

1.70

0.53

(5,3)

1.70

cray

1.13

0.70

(15,5)

1.61

0.69(15,15)

1.63

Credit: Jeff Hollingsworth, University of Maryland
18

LS3DFSUPER!

· LS3DF: “linearly scaling 3-dimensional fragment” code for electronic
structure calculation.
· Developed at LBNL by Lin-Wang Wang and several collaborators.
· Numerous applications in materials science and nanoscience.
· Employs a novel divide-and-conquer scheme including a new approachfor patching the fragments together.
· Achieves nearly linear scaling in computational cost versus size ofproblem, compared with n3 scaling in many other comparable codes.
· Potential for nearly linear scaling in performance versus number of cores.
Challenge:· Initial implementation of LS3DF had disappointingly low performance andparallel scalability.

19Performance analysis of LS3DFSUPER!

LBNL researchers (funded through PERI) applied performance monitoring
tools to analyze run-time performance of LS3DF. Key issues uncovered:· Limited concurrency in a key step, resulting in a significant load
imbalance between processors.
§Solution: Modify code for two-dimensional parallelism.

· Costly file I/O operations were used for data communication betweenprocessors.
§Solution: Replace all file I/O operations with MPI send-receive operations.

20

Resulting performance of tuned LS3DFSUPER!

· 135 Tflops/s on 36,864 cores of theCray XT4 Franklin system at LBNL.
§

40% efficiency on 36,864 cores.

· 224 Tflops/s on 163,840 processors of
the BlueGene/P Intrepid system at
Argonne Natl. Lab.
§

40% efficiency on 163,840 cores.

· 442 Tflops/s on 147,456 processors of
the Cray XT5 Jaguar system at OakRidge Natl. Lab.
§

33% efficiency on 147,456 cores.

2008 ACM Gordon Bell Prize in a
special category for “algorithminnovation.”21

Near-linear parallel scaling up to163,840 cores and up to 442 Tflop/s

SUPER!

LS3DF
500
450
400

TFlop/s.

350

JaguarIntrepidFranklin

300
250
200
150
100
5000

20,00040,00060,00080,000 100,000 120,000 140,000 160,000 180,000
Cores

22

SUPER!

Performance Tuning of Scientific
Applications

Editors: Bailey (LBNL), Lucas (USC/ISI), Williams (LBNL);
numerous individual authors of various chapters.
Publisher: Taylor and Francis / CRC, Jan 2011.
Contents:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.

Introduction
Parallel computer architecture
Software interfaces to hardware counters
Measurement and analysis of parallel program performance using TAUand HPCToolkit.
Trace-base tools
Large-scale numerical simulations on high-end computational platforms
Performance modeling: the convolution approach
Analytic modeling for memory access patterns based on Apex-MAP
The roofline model
End-to-end auto-tuning with active harmony
Languages and compilers for auto-tuning
Empirical performance tuning of dense linear algebra software
Auto-tuning memory-intensive kernels for multicore
Flexible tools supporting a scalable first-principles MD code
The community climate system model
Tuning an electronic structure code23

Disclosure: I receive amodest royalty (shared withother editors and LBNL)
from sales.!