スーパーコンピューティングニュース- 62 -

Vol. 14, No. 3

2012スーパーコンピューティングニュース- 63 -

Vol. 14, No. 3

2012スーパーコンピューティングニュース- 64 -

Vol. 14, No. 3

2012スーパーコンピューティングニュース- 65 -

Vol. 14, No. 3

2012スーパーコンピューティングニュース- 66 -

Vol. 14, No. 3

2012University of Tokyo
April 25, 2012

Dealing with Application Development
 Now and Henceforth Osni Marques

Lawrence Berkeley National Laboratory
OAMarques@lbl.gov(under the auspices of the Japan Society for the Promotion of Science)Contents‹ Applications and the software stack‹ The DOE ACTS Collection‹ Technology transition‹ Impact of hardware evolution on libraries

2

スーパーコンピューティングニュース- 67 -Vol. 14, No. 3

2012

HPC Applications‹‹‹‹‹‹

Accelerator ScienceEarth Sciences
Material Sciences
Biology
Chemistry
Astrophysics

Commonalities‹‹‹

Advancements in science and engineering
Increasing demands for computational power
Reliance on available computational systems,
languages, and software tools

3

Software Stack

APPLICATIONSGENERAL PURPOSE TOOLSSUPPORT TOOLS AND UTILITIESHARDWARE
Leading technology paths (swim lanes):

Risks in swim lane selection:‹ Multicore: maintain complex cores, and replicate (x86 andPower7, Blue Waters, NGSC)‹ Manycore/embedded: use many simpler, low power cores fromembedded systems (BlueGene, Dawning)‹ GPU/Accelerator: use highly specialized processors fromgaming/graphics market space (NVidia Tesla, Cell, Intel Knights
Corner/MIC)‹ Select too soon: users cannot follow‹ Select too late: fall behind performance curve‹ Select incorrectly: subject users to multiple
disruptive technology changes

4

スーパーコンピューティングニュース- 68 -

Vol. 14, No. 3

2012

The DOE ACTS Collection
Goal: The Advanced CompuTational Software Collection(ACTS) makes reliable and efficient software tools more
widely used, and more effective in solving the nationsengineering and scientific problems‹‹‹‹

Long term maintenanceIndependent test and evaluation
Outreach and dissemination
High level user support

5

The DOE ACTS Collection: Current Functionalities
CategoryTool

Functionalities

Trilinos

Algorithms
for the(Iterativeiterative solutionof large sparse linear systems
Linear Solvers
Methods):

Hypre

Algorithms for the iterative solution of large sparse linear systems (grid centricinterfaces)‹ CG
Eigenvalues

PETSc
SUNDIALS
Numerical

ScaLAPACKSLEPc
SuperLU
TAO

Code Development

Global Arrays
Overture

Code Execution

TAU

Library Development

ATLAS‹ GMRES
Singular
Value ofDecomposition
Tools
for the solution
PDEs (sparse linear and nonlinear systems of equations)Nonlinearequations(Newtonbased)‹ CGS
LinearLeast
SquaresSolvers
for thesolutionof systemsof ordinary
differential equations, nonlinear algebraic
Nonlinear
optimization‹Linear
BiCGSTAB
algebraic equations
equations,and differential
Solvers(Direct Methods):‹ Newton based
‹ QMR
‹ CG
‹‹ SYMMLQ
LU
Eigensolver package built on top of PETSc‹ Direct search‹ Cholesky
Generallibrary for the direct solution of large, sparse, nonsymmetric linear
TpurposeODEs‹‹ Various
preconditioners(tridiagonalmatrices)
LDL
systems
Sensitivity analysis‹‹‹ QR
distributed
dense arrays that can be accessed throughTools
for the solution of optimization problems (nonlinear least squares, unconstrained‹ Multigrid
like stylea sharedboundmemoryminimization,constrainedoptimization)‹ uses one sided communication model
Librarythat enables
a sharedmemoryview
for distributed memory computers‹‹ solutionofPDEs
onacomplex
geometry
supports the creation of a Partitioned
Global Address
Frameworksolving
partial differential
equations in complex geometries‹ movinggeometry
Space for(PGAS)
programming
modeloverlapping
grid MPI
Set
tools for analyzing
the performance of multi language programs‹‹ ofcompatible
withHigh performance dense linear algebra routines

Tools for the automatic generation of optimized numerical software (dense linearalgebra)

6

スーパーコンピューティングニュース- 69 -

Vol. 14, No. 3

2012

Software InterfacesÝßÔÔ ÞÔßÝÍÁÙÛÌø óïô ðô ×ÝÌÈÌ ÷ÝßÔÔ ÞÔßÝÍÁÙÎ×Ü×Ò×Ìø ×ÝÌÈÌô ùÎ±©ó³¿¶±®ùô ÒÐÎÑÉô ÒÐÝÑÔ ÷æÝßÔÔ ÞÔßÝÍÁÙÎ×Ü×ÒÚÑø ×ÝÌÈÌô ÒÐÎÑÉô ÒÐÝÑÔô ÓÇÎÑÉô ÓÇÝÑÔ ÷æÝßÔÔ ÐÜÙÛÍÊø Òô ÒÎØÍô ßô ×ßô Ößô ÜÛÍÝßô ×Ð×Êô Þô ×Þô ÖÞô ÜÛÍÝÞô ×ÒÚÑ ÷

½±³³¿²¼ ´·²»º«²½¬·±² ½¿´´
(ScaLAPACK)

ksp_type [cg,gmres,bcgs,tfqmr, ]pc_type [lu,ilu,jacobi,sor,asm, ]More advanced:(PETSc)Ô·²»¿® Í§­¬»³ ×²¬»®º¿½»­ksp_max_it <max_iters>
ksp_gmres_restart <restart>
pc_asm_overlap <overlap>
pc_asm_type[basic,restrict,interpolate,none]Ô·²»¿® Í±´ª»®­
ÙÓÙ

ÚßÝ

Ø§¾®·¼ô òòòßÓÙ»×ÔËô òòò°®±¾´»³ ¼±³¿·²(Hypre)Ü¿¬¿ Ô¿§±«¬­¬®«½¬«®»¼½±³°±­·¬»¾´±½µ­¬®½«²­¬®«½

ÝÍÎ

7Addressing Application Performance Issues‹‹‹‹‹‹‹‹‹‹How does performance vary with different compilers?
Is poor performance correlated with certain OS features?
Has a recent change caused unanticipated performance?
How does performance vary with MPI variants?
Why is one application version faster than another?
What is the reason for the observed scaling behavior?
Did two runs exhibit similar performance?
How are performance data related to application events?
Which machines will run my code the fastest and why?
Which benchmarks predict my code performance best?

(courtesy of Sameer Shende)

8スーパーコンピューティングニュース- 70 -

Vol. 14, No. 3

2012

TAU: Performance Analysis‹ Profiling: summary statistics of performance
metrics (# of times a routine was invoked exclusive
or inclusive time or hardware counts, calltrees andcallgraphs, memory and message sizes etc)
‹ Tracing: when and where events took place along a
global timeline (timestamped log of events,message communication events)‹ Automatic instrumentation of source code (PDT)‹ Runs on basically all HPC platforms‹ 3D profile browser (paraprof)‹ To use TAU, one only needs to set a couple ofenvironment variables and substitute the nameof the compiler with a TAU shell script‹ Ex. Flat profile of Miranda (LLNL; hydrodynamics / Fortran + MPI) on an BG/L;
node, context, thread(See http://tau.uoregon.edu/tau.ppt)
9Using TAU (basic level)‹ TAU supports several measurement options (profiling, tracing,profiling with hardware counters etc)
‹ Each measurement configuration of TAU corresponds to a uniquestub makefile and library that is generated when you configure it‹ To instrument source code using PDT, choose an appropriate TAUstub makefile in <arch>/lib:û ­»¬»²ª ÌßËÁÓßÕÛÚ×ÔÛ ñ«­®ñ´±½¿´ñ°¿½µ¿¹»­ñ¬¿«ñ·íèêÁ´·²«¨ñ´·¾ñÓ¿µ»º·´»ò¬¿«ó³°·ó°¼¬
û ­»¬»²ª ÌßËÁÑÐÌ×ÑÒÍ èó±°¬Ê»®¾±­» ›Ž ø­»» ç¬¿«Á½±³°·´»®ò­¸ Š¸»´°Œ÷‹ Use tau_f90.sh, tau_cxx.sh or tau_cc.sh as Fortran, C++ or Ccompilers:û ¬¿«Áºçðò­¸ º±±òºçðø·²­¬»¿¼ ±º ç³°·ºçð º±±òºçðŒ÷‹ Execute application and analyze performance data:û °°®±ºû °¿®¿°®±ºøº±® ¬»¨¬ ¾¿­»¼ °®±º·´» ¼·­°´¿§÷øº±® ÙË×÷10

スーパーコンピューティングニュース- 71 -Vol. 14, No. 3

2012

Software Tools for Application Development, Portability and Performance‹‹‹‹‹min[time_to_first_solution]min[time_to_solution]
min[software development cost]max[software_life]
max[resource_utilization](prototype)(production)‹ Outlive Complexity

Š Increasingly sophisticated models
Š Model couplingŠ Interdisciplinary

software evolution

Š Increasingly complex algorithms
Š Increasingly diverse architectures
Š Increasingly demanding applications

long term deliverables‹ Sustained Performance

11

Technology Transition… and impacts to a facility like NERSC
ïðéÛ¨¿­½¿´» õ áïðêÒÛÎÍÝóçï ÛÚ Ð»¿µÒÛÎÍÝóèïðð ÐÚ Ð»¿µïðëÒÛÎÍÝóéïð ÐÚ Ð»¿µïðìÙÐË ÝËÜßñÑ°»²ÝÔÑ® Ó¿²§½±®» ÞÙñÏô ÎØ±°°»® øÒê÷âï ÐÚ Ð»¿µïðíïðîïð

Ú®¿²µ´·² øÒë÷ïç ÌÚ Í«­¬¿·²»¼
ïðï ÌÚ Ð»¿µÚ®¿²µ´·² øÒë÷ õÏÝíê ÌÚ Í«­¬¿·²»¼
íëî ÌÚ Ð»¿µÝÑÌÍñÓÐÐ õ ÓÐ× øõ Ñ°»²ÓÐ÷ÝÑÌÍñÓÐÐ õ ÓÐ×îððê îððé îððè îððç îðïð îðïï îðïî îðïí îðïì îðïë îðïê îðïé îðïè îðïç îðîðSource: Horst Simon & Kathy Yelick
12

スーパーコンピューティングニュース- 72 -

Vol. 14, No. 3

2012

Parametric Research and Integration‹ Hand tuning algorithmic
parameters can becumbersome‹ Auto tuning produces asingle tuned library (maxcores per node)‹ Some applications wontscale

APPLICATIONS‹ Auto tuned algorithmicparameters‹ Auto tuned libraries throughsteering parameters(#cores/node)‹ Run time selection of tunedlibrary‹ Binary rewriting (?)

GENERAL PURPOSE TOOLSSUPPORT TOOLS AND UTILITIESHARDWARE

13

Hardware and Programming Impacts on Libraries‹ Workshop on Exascale Programming Challenges,
Marina del Rey, CA, July 27 29, 2011
‹ Workshop on Extreme Scale Solvers: Transition toFuture Architectures, Washington, DC, March 8 9,2012
‹ Focus on solvers

14

スーパーコンピューティングニュース- 73 -Vol. 14, No. 32012

Current PF Machines(Michael Feldman, http://www.hpcwire.com/hpcwire/2012 04 10/the_processors_of_petascale.html)
SystemCountry

Processor

Interconnect

Petaflops

K Computer

Japan

Fujitsu SPARC64

Tofu

11.28

Tianhe 1A

ChinaIntel x86/NVIDIA GPU

Galaxy4.70

Nebulae

ChinaIntel x86/NVIDIA GPU

InfiniBand

2.98

JaguarUS

AMD x86

Gemini

2.33

TSUBAME 2JapanIntel x86/NVIDIA GPU

InfiniBand

2.29

CURIE

FranceIntel x86/NVIDIA

InfiniBand

2.00

HeliosJapan

Intel x86

InfiniBand

1.50

RoadrunnerUS

AMD x86/PowerXCell

InfiniBand

1.37

Lomonosov

RussiaIntel x86/NVIDIA GPU

InfiniBand

1.37
1.36CieloUS

AMD x86

Gemini

Tianhe 1A Hunan

ChinaIntel x86/NVIDIA GPU

Galaxy1.34

PleiadesUSIntel x86

InfiniBand

1.32

Hopper

USAMD x86

Gemini

1.29

Tera 100

FranceIntel x86

InfiniBand

1.25

KrakenUS

AMD x86

SeaStar

1.17

Oakleaf FXJapanFujitsu SPARC64

Tofu

1.13

Sunway Blue Light

China

ShenWei SW1600

InfiniBand

1.07

Germany

AMD x86

Gemini

1.04

Mole 8.5

ChinaIntel x86/NVIDIA GPU

InfiniBand

1.01

JUGENE

Germany

PowerPC 450

Custom1.00

HERMIT

15

Challenges for Next Generation Solvers (100PF  and beyond)‹ Extreme levels of concurrency
Š millions of nodes with thousands of lightweight coresŠ hundreds of thousands of nodes with more aggressive cores‹ Resilience and non deterministic behavior
Š hard interrupts (failure of a device)Š soft errors (change of a data value due to faults in logic latches)‹ Reduced memory sizes per coreŠ more computation on local data, minimization of synchronization
Š shift the focus from the usual weak scaling to strong scaling

16

スーパーコンピューティングニュース- 74 -

Vol. 14, No. 3

2012Challenges for Next Generation Solvers (100PF  and beyond)‹ Data storage and movement
Š on a node, data movement will be much more costly, than otheroperations
Š data access will be much more sensitive to data layout‹ Deep memory hierarchiesŠ solvers may need to be hierarchical (e.g. cache oblivious)‹ Portability with performanceŠ current programming possibilities are not interoperable
Š abstractions

17

Next Generation Solvers Features‹ Communication/synchronization hiding algorithms
Š e.g. dot products interposed with matrix vector multiplies‹ Communication/synchronization reducing algorithms
Š e.g. s step Krylov methods‹ Mixed precision arithmetic algorithms
Š e.g. LU + triangular solves in s.p., iterative refinement in d.p.
Š reduction of memory usage‹ Fault tolerant and resilient algorithms
Š localized checkpoints and asynchronous recovery
Š checksum

18

スーパーコンピューティングニュース- 75 -

Vol. 14, No. 3

2012

Next Generation Solvers Features‹ Energy efficient algorithms‹ Stochastic algorithms
Š nondeterminism in data and operations at very large scales‹ Algorithms with reproducibilityŠ bit wise identical results from one run to another may be too
costly19

Transition to New Solvers‹ Evolutionary algorithmic researchŠ development and optimization on existing (heterogeneous)petascale architectures‹ Transition to new application library interfaces
Š departure from an MPI only programming model‹ Research community interaction (longer term)
Š complexity of issues that need to be addressed‹ Revolutionary algorithmic researchŠ rethink the solver process (algorithmic and programmingapproaches)Š revisit algorithms that may not perform well on current systems
Š co development of ideas in the computational science
community

20

スーパーコンピューティングニュース- 76 -

Vol. 14, No. 3

2012

The DOE ACTS Collection: Current Functionalities
Category

Numerical

Code DevelopmentToolFunctionalities

Trilinos

Algorithms for the iterative solution of large sparse linear systems

Hypre

Algorithms for the iterative solution of large sparse linear systems (grid centric interfaces)

PETSc

Tools for the solution of PDEs (sparse linear and nonlinear systems of equations)

SUNDIALS

Solvers for the solution of systems of ordinary differential equations, nonlinear algebraic
equations, and differential algebraic equations

ScaLAPACKHigh performance dense linear algebra routines

SLEPc

Eigensolver package built on top of PETSc

SuperLU

General purpose library for the direct solution of large, sparse, nonsymmetric linearsystems

TAO

Tools for the solution of optimization problems (nonlinear least squares, unconstrainedminimization, bound constrained optimization)

Global Arrays

Library that enables a shared memory view for distributed memory computers

Overture

Framework for solving partial differential equations in complex geometries

Code Execution

TAU

Set of tools for analyzing the performance of multi language programs

Library Development

ATLAS

Tools for the automatic generation of optimized numerical software (dense linear algebra)

http://nkl.cc.u tokyo.ac.jp/VECPAR2012
http://acts.nersc.gov/events/Workshop2012
21

Thank you!

スーパーコンピューティングニュース- 77 -

Vol. 14, No. 3

2012