# Presentation: 432440

## Evaluation of Ultra-Scale Applications on Leading Scalar and Vector Platforms

- Leonid Oliker
- Computational Research Division
- Lawrence Berkeley National Laboratory

## Overview

- Stagnating application performance is well-know problem in scientific computing
- By end of decade mission critical applications expected to have 100X computational demands of current levels
- Many HEC platforms are poorly balanced for demands of leading applications
- Memory-CPU gap, deep memory hierarchies, poor network-processor integration, low-degree network topology
- Traditional superscalar trends slowing down
- Mined most benefits of ILP and pipelining,Clock frequency limited by power concerns
- In order to continuously increase computing power _and_ reap its benefits: major strides necessary in architecture development, software infrastructure, and application development

## Application Evaluation

- Microbenchmarks, algorithmic kernels, performance modeling and prediction, are important components of understanding and improving architectural
- However full-scale application performance is final arbiter of system utility and necessary as baseline to support all complementary approaches
- Our evaluation work emphasizes full applications, with real input data, at the appropriate scale
- Requires coordination of computer scientists and application experts from highly diverse backgrounds
- Our initial efforts have focused on comparing performance between high-end vector and scalar platforms
- Effective code vectorization is an integral part of the process

## Benefits of Evaluation

- Full scale application evaluation lead to more efficient use of the community resources in both current installation and in future designs.
- Head-to-head comparisons on full applications:
- Help identifying the suitability of a particular architecture for a given service site or set of users,
- Give application scientists information about how well various numerical methods perform across systems
- Reveal performance-limiting system bottlenecks that can aid designers of the next generation systems.
- In-depth studies reveal limitation of compilers, operating systems, and hardware, since all of these components must work together at scale to achieve high performance.

## Application Overview

| NAME | Discipline | Problem/Method | Structure |
| --- | --- | --- | --- |
| MADCAP | Cosmology | CMB analysis | Dense Matrix |
| CACTUS | Astrophysics | Theory of GR | Grid |
| LBMHD | Plasma Physics | MHD  | Lattice |
| GTC | Magnetic Fusion | Vlasov-Poisson | Particle/Grid |
| PARATEC | Material Science | DFT | Fourier/Grid |
| FVCAM | Climate Modeling | AGCM  | Grid |

- Examining set of applications with potential to run at ultra-scale and
- _abundant_ data parallelism

## IPM Overview

**Integrated**
**Performance**
**Monitoring**
- portable, lightweight, scalable profiling
- fast hash method
- profiles MPI topology
- profiles code regions
- open source

- MPI_Pcontrol(1,”W”);
-  ...code...
- MPI_Pcontrol(-1,”W”);

**###########################################**
**# IPMv0.7 :: csnode041 256 tasks  ES/ESOS**
**# madbench.x (completed) 10/27/04/14:45:56**
**#**
**#       <mpi>      <user>      <wall> (sec)**
**#      171.67      352.16      393.80      **
**# ...**
**###############################################**
**# W**
**#       <mpi>      <user>      <wall> (sec)**
**#       36.40      198.00      198.36**
**#**
**# call            [time]      %mpi   %wall**
**# MPI_Reduce      2.395e+01   65.8     6.1**
**# MPI_Recv        9.625e+00   26.4     2.4**
**# MPI_Send        2.708e+00    7.4     0.7**
**# MPI_Testall     7.310e-02    0.2     0.0**
**# MPI_Isend       2.597e-02    0.1     0.0**
**###############################################**
**...**

## Plasma Physics: LBMHD

- LBMHD uses a Lattice Boltzmann method to model magneto-hydrodynamics (MHD)
- Performs 2D/3D simulation of high temperature plasma
- Evolves from initial conditions and decaying to form current sheets
- Spatial grid is coupled to octagonal streaming lattice
- Block distributed over processor grid

- Developed by George Vahala’s group College of William & Mary, ported Jonathan Carter

- Evolution of vorticity into turbulent structures

**Notes:**
Lattice boltzman methods popular approach for modeling the physics of fluid flows. Recently groups have applied the LBM to the problem of MHD.    

## LBMHD-3D: Performance

| Grid Size | P | NERSC (Power3) |  | Thunder (Itan2) |  | Phoenix (X1) |  | ES (SX6*) |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak |
| 2563 | 16 | 0.14 | 9% | 0.26 | 5% | 5.2 | 41% | 5.5 | 69% |
| 5123 | 64 | 0.15 | 9% | 0.35 | 6% | 5.2 | 41% | 5.3 | 66% |
| 10243 | 256 | 0.14 | 9% | 0.32 | 6% | 5.2 | 41% | 5.5 | 68% |
| 20483 | 512 | 0.14 | 9% | 0.35 | 6% |  |  | 5.2 | 65% |

- Not unusual to see vector achieve > 40% peak while superscalar architectures achieve < 10%
- There exists plenty of computation, however large working set causes register spilling in scalars
- Large vector register sets hide latency
- ES sustains 68% of peak up to 4800 processors: 26TFlops - the highest performance ever attained for this code by far!

## Astrophysics: CACTUS

- Numerical solution of Einstein’s equations from theory of general relativity
- Among most complex in physics: set of coupled nonlinear hyperbolic & elliptic systems with thousands of terms
- CACTUS evolves these equations to simulate high gravitational fluxes, such as collision of two black holes
- Evolves PDE’s on regular grid using finite differences

- Visualization of grazing collision of two black holes

- Developed at Max Planck Institute, vectorized by John Shalf

**Notes:**
One of the most challenging problems is astrophysics is the numberical solution of Einstein’s equation from the theory of general relatively..
CACTUS code developed to evolve this equations stably in 3D to simulate astrophyical phenomena with high gravitational fluxes, such as the collision of two black holes (shown) and the gravitational waves that radiate from that event.

## CACTUS: Performance

- ES achieves fastest performance to date: 45X faster than Power3!
- Vector performance related to x-dim (vector length)
- Excellent scaling on ES using fixed data size per proc (weak scaling)
- Opens possibility of computations at unprecedented scale
- X1 surprisingly poor (4X slower than  ES) - low ratio scalar:vector
- Unvectorized boundary, required 15% of runtime on ES and 30+% on X1
- < 5% for the scalar version: **unvectorized code can quickly dominate cost**
- Poor superscalar performance despite high computational intensity
- Register spilling due to large number of loop variables
- Prefetch engines inhibited due to multi-layer ghost zones calculations

| ProblemSize | P | NERSC (Power 3) |  | Thunder (Itan2) |  | Phoenix (X1) |  | ES (SX6*) |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak |
| 250x80x80perprocessor | 16 | 0.10 | 6% | 0.58 | 10% | 0.81 | 6% | 2.8 | 35% |
|  | 64 | 0.08 | 6% | 0.56 | 10% | 0.72 | 6% | 2.7 | 34% |
|  | 256 | 0.07 | 5% | 0.55 | 10% | 0.68 | 5% | 2.7 | 34% |

## Magnetic Fusion: GTC

- Gyrokinetic Toroidal Code: transport of thermal energy (plasma microturbulence)
- Goal magnetic fusion is burning plasma power plant producing cleaner energy
- GTC solves 3D gyroaveraged gyrokinetic system w/ _particle-in-cell approach_ (PIC)
- PIC scales *N *instead of *N**2* – particles interact w/ electromagnetic field on grid
- Allows solving equation of particle motion with ODEs (instead of nonlinear PDEs)

- Electrostatic potential in magnetic fusion device

- Developed at Princeton Plasma Physics Laboratory, vectorized by Stephane Ethier

**Notes:**
Performance of such device is determine by rate at which energy is transported out of hot core to the colder edge of plasma
PIC: Instead of interacting with one another, particles interact with electromagnetic field described on a grid

## GTC: Performance

- New particle decomposition method to efficiently utilize large numbers of processors (as opposed to 64 on ES)
- Breakthrough of Tflop barrier on ES: 3.7 Tflop/s on 2048 processors
- Opens possibility of new set of high-phase space-resolution simulations, that have not been possible to date
- X1 suffers from overhead of scalar code portions
- Scalar architectures suffer from low computational intensity, irregular data access, and register spilling

| Part/Cell | P | NERSC (Power3) |  | Thunder (Itan2) |  | Phoenix (X1) |  | ES (SX6*) |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak |
| 200 | 128 | 0.14 | 9% | 0.39 | 7% | 1.2 | 9% | 1.6 | 20% |
| 400 | 256 | 0.14 | 9% | 0.39 | 7% | 1.2 | 9% | 1.6 | 20% |
| 800 | 512 | 0.14 | 9% | 0.38 | 7% |  |  | 1.5 | 19% |
| 1600 | 1024 | 0.14 | 9% | 0.37 | 7% |  |  | 1.9 | 24% |
| 3200 | 2048 | 0.14 | 9% | 0.37 | 7% |  |  | 1.8 | 23% |

## Cosmology: MADCAP

- Anisotropy Dataset Computational Analysis Package
- Optimal general algorithm for extracting key cosmological data from Cosmic Microwave Background Radiation (CMB)
- Anisotropies in the CMB contains early history of the Universe
- Recasts problem in dense linear algebra: ScaLAPACK
- Out of core calculation: holds approx 3 of the 50 matrices in memory

- Temperature anisotropies in CMB (Boomerang)

- Developed by Julian Borrill, LBNL

**Notes:**
DFT has become standard technique in Mat Sci to calculate the structural and electronic properties of new materials with a full QUANTUM MECHANICAL treatment of the electrons.
Picture shows induced current and charge density in crystalized glcine.The simulations used to better undersand nuclear magnetic resonance.

## MADCAP: Performance

- Overall performance can be surprising low, for dense linear algebra code
- I/O takes a heavy toll on Phoenix and Columbia: I/O optimization currently in progress
- NERSC Power3 shows best system balance wrt to I/O
- ES lacks high-performance parallel I/O

| NumberPixels | P | NERSC (Power3) |  | Columbia (Itan2) |  | Phoenix (X1) |  | ES (SX6*) |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak |
| 10K | 64 | 0.73 | 49% | 1.2 | 20% | 2.2 | 17% | 2.9 | 37% |
| 20K | 256 | 0.76 | 51% | 1.1 | 19% | 0.6 | 5% | 4.0 | 50% |
| 40K | 1024 | 0.75 | 50% |  |  |  |  | 4.6 | 58% |

## Climate: FVCAM

- Atmospheric component of CCSM
- AGCM: consists of physics and dynamical core (DC)
- DC approximates Navier-Stokes eqn’s to describe dynamics of atmosphere
- Default approach uses spectral transform (1D decomp)
- Finite volume (FV) approach uses a 2D decomposition in latitude and level: allows higher concurrency
- Requires remapping between Lagrangian surfaces and Eulerian reference frame

- Experiments conducted by Michael Wehner, vectorized by Pat Worley, Art Mirin, Dave Parks

**Notes:**
DFT has become standard technique in Mat Sci to calculate the structural and electronic properties of new materials with a full QUANTUM MECHANICAL treatment of the electrons.
Picture shows induced current and charge density in crystalized glcine.The simulations used to better undersand nuclear magnetic resonance.

## FVCAM: Performance

- 2D approach allows both architectures to effectively use >2X as many procs
- At high concurrencies both platforms achieve low % peak (about 4%)
- ES suffers from short vector lengths for fixed problem size
- ES can achieve more than 1000 simul year / wall clock year (3200 on 896 processors), NERSC cannot exceed 600 regardless of concurrency
- Speed up of 1000x or more is necessary for reasonable turnaround time
- Preliminary results: CAM3.1 experiments currently underway on ES, X1, Thunder, Power3

- CAM3.0 results on ES and Power3, using *D *Mesh (0.5ox0.625o)

## Material Science: PARATEC

**Material Science: PARATEC**

- PARATEC performs first-principles quantum mechanical total energy calculation using pseudopotentials & plane wave basis set
- Density Functional Theory to calc structure & electronic properties of new materials
*DFT calc are one of the largest consumers of supercomputer cycles in the world*
- 33% 3D FFT,  33% BLAS3,  33% Hand coded F90
- Part of calculation in real space other in Fourier space
- Uses specialized 3D FFT to transform wavefunction

- Crystallized glycine induced current & charge

**Notes:**
DFT has become standard technique in Mat Sci to calculate the structural and electronic properties of new materials with a full QUANTUM MECHANICAL treatment of the electrons.
Picture shows induced current and charge density in crystalized glcine.The simulations used to better undersand nuclear magnetic resonance.

## PARATEC: Performance

- All architectures generally achieve high performance due to computational intensity of code (BLAS3, FFT)
- ES achieves fastest performance to date: 5.5Tflop/s on 2048 procs
- Main ES advantage for this code is fast interconnect
- Allows never before possible, high resolution simulations
- X1 shows lowest % of peak
- Non-vectorizable code much more expensive on X1 (32:1)
- Lower bisection bandwidth to computational ratio (2D Torus)

| Problem | P | NERSC (Power3) |  | Thunder (Itan2) |  | Phoenix (X1) |  | ES (SX6*) |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak |
| 488 AtomCdSeQuantumDot | 128 | 0.93 | 63% | 2.8 | 51% | 3.2 | 25% | 5.1 | 64% |
|  | 256 | 0.85 | 62% | 2.6 | 47% | 3.0 | 24% | 5.0 | 62% |
|  | 512 | 0.73 | 57% | 2.4 | 44% |  |  | 4.4 | 55% |
|  | 1024 | 0.60 | 49% | 1.8 | 32% |  |  | 3.6 | 46% |

- Developed by Andrew Canning with Louie and Cohen’s groups (UCB, LBNL)

## Overview

- Tremendous potential of vector architectures:  4 codes running faster than ever before
- Vector systems allows resolution not possible with scalar arch (regardless of # procs)
- Opportunity to perform scientific runs at unprecedented scale
- ES shows high raw and much higher sustained performance compared with X1
- Limited X1 specific optimization - optimal programming approach still unclear (CAF, etc)
*Non-vectorizable code segments become very expensive (8:1 or even 32:1 ratio)*
- Evaluation codes contain sufficient regularity in computation for high vector performance
- GTC example code at odds with data-parallelism
*Much more difficult to evaluate codes poorly suited for vectorization*
- Vectors potentially at odds w/ emerging techniques (irregular, multi-physics, multi-scale)
- Plan to expand scope of application domains/methods, and examine latest HPC architectures

| Code | (P=64)  % peak |  |  |  |  | (P=Max avail)   Speedup ES vs.  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Pwr3 | Pwr4 | Altix | ES | X1 | Pwr3 | Pwr4 | Altix | X1 |
| LBMHD3D |  |  |  |  |  |  |  |  |  |
| CACTUS |  |  |  |  |  |  |  |  |  |
| GTC |  |  |  |  |  |  |  |  |  |
| MADCAP |  |  |  |  |  |  |  |  |  |
| PARATEC |  |  |  |  |  |  |  |  |  |
| FVCAM |  |  |  |  |  |  |  |  |  |
| AVERAGE |  |  |  |  |  |  |  |  |  |

**Notes:**
With the emergence of the next generation of supercomputing systems - it is clear that no single architecture is best suited for the full scope of application domanins - it is therefore critical to understand which computational areas are best suited for a given architectural paradigm, to maximize scientific productivity.

## Collaborators

- Rupak Biswas, NASA Ames
- Andrew Canning LBNL
- Jonathan Carter, LBNL
- Stephane Ethier, PPPL
- Bala Govindasamy, LLNL
- Art Mirin, LLNL
- David Parks, NEC
- John Shalf, LBNL
- David Skinner, LBNL
- Yoshinori Tsunda, JAMSTEC
- Michael Wehner, LBNL
- Patrick Worley, ORNL