# Presentation: 669700

## Towards Petascale Computing for Science

** ****Horst Simon**
- Lawrence Berkeley National Laboratory
**ICCSE 2005 Istanbul**
**June 30, 2005**
**With contributions by Lenny Oliker, David Skinner, and Erich Strohmaier**

## National Energy Research Scientific Computing Center

**National Energy Research Scientific Computing Center**
**Berkeley, California**

**~2500 Users in **
**~250 projects**

**Serves the **_**entire**_** scientific community**

**Focus on large-scale computing**

**Notes:**
Computing from networking to ESnet to 2000 users
Opportunities for computing
It began that computing was to be pervasive in all science, we have achieved that
Now computing to drive modeling and scientific discovery

## Outline

- Science Driven Architecture
- Performance on today’s (2004 - 2005) platforms
- Challenges with scaling to the Petaflop/s level
- Two tools that can help: IPM and APEX/MAP

**Notes:**
latency?
particle methods?
fusion is MHD

## Scientific Applications and Underlying Algorithms Drive Architectural Design 

- 50 Tflop/s - 100 Tflop/s **sustained** performance on applications of national importance
- Process:
- identify applications
- identify computational methods used in these applications
- identify architectural features most important for performance of these computational methods

**Reference: **Creating Science-Driven Computer Architecture: A New Path to Scientific Leadership, (Horst D. Simon, C. William McCurdy, William T.C. Kramer, Rick Stevens, Mike McCoy, Mark Seager, Thomas Zacharia, Jeff Nichols, Ray Bair, Scott Studham, William Camp, Robert Leland, John Morrison, Bill Feiereisen), Report LBNL-52713, May 2003. (see www.nersc.gov/news/reports/HECRTF-V4-2003.pdf)

**Notes:**
Mature applications – not biology;
Its not that we picked a vector architecture and then found applications that worked well

## Capability Computing Applications in the Office of Science (US DOE)

**Capability Computing Applications in the Office of Science (US DOE)**

** ****Accelerator modeling**
** ****Astrophysics**
** ****Biology**
** ****Chemistry**
** ****Climate and Earth Science**
** ****Combustion**
** ****Materials and Nanoscience**
** ****Plasma Science/Fusion**
** ****QCD**
** ****Subsurface Transport**

**Notes:**
Discipline statement, not DOE focus

## Capability Computing Applications in the Office of Science (US DOE)

- These applications and their computing needs have been well-studied in the past years:
-  “A Science-Based Case for Large-scale Simulation”, David Keyes, Sept. 2004 (http://www.pnl.gov/scales).
-  “Validating DOE’s Office of Science “Capability” Computing Needs”, E. Barsis, P. Mattern, W. Camp, R. Leland, SAND2004-3244, July 2004.

## Science Breakthroughs Enabled by Petaflops Computing Capability 

| Science Areas | Goals | Computational Methods | Breakthrough Target (50-100 Tflop/s) |
| --- | --- | --- | --- |
| Nanoscience | Simulate the synthesis and predict the properties of multi-component nanosystems | Quantum molecular dynamicsQuantum Monte CarloIterative eigensolversDense linear algebraParallel 3D FFTs | Simulate nanostructures with hundreds to thousands of atoms as well as transport and optical properties and other parameters |
| Combustion | Predict combustion processes to provide efficient, clean and sustainable energy | Explicit finite differenceImplicit finite differenceZero-dimensional physicsAdaptive mesh refinementLagrangian particle methods | Simulate laboratory scale flames with high fidelity representations of governing physical processes |
| Fusion | Understand high-energy density plasmas and develop an integrated simulation of a fusion reactor | Multi-physics, multi-scaleParticle methodsRegular and irregular accessNonlinear solversAdaptive mesh refinement | Simulate the ITER reactor |
| Climate | Accurately detect and attribute climate change, predict future climate and engineer mitigation strategies | Finite difference methodsFFTsRegular and irregular accessSimulation ensembles | Perform a full ocean/atmosphere climate model with 0.125 degree spacing, with an ensemble of 8-10 runs |
| Astrophysics | Determine through simulations and analysis of observational data the origin, evolution and fate of the universe, the nature of matter and energy, galaxy and stellar evolutions | Multi-physics, multi-scaleDense linear algebraParallel 3D FFTs
Spherical transformsParticle methodsAdaptive mesh refinement | Simulate the explosion of a supernova with a full 3D model |

**Notes:**
Breakthrough applications; only place in the presentation:

80 mpg vehicle
Clean cheap source of energy
Fuel cells (nano) solar cells (Zunger)
Climate

Cleaner, cheaper, more sustainable, impact on climate, 

## Opinion Slide

- One reason why we have failed so far to make a good case for increased funding in supercomputing is that we have not yet made a compelling science case.

- A better example: “The Quantum Universe”
- “It describes a revolution in particle physics and a quantum leap in our understanding of the mystery and beauty of the universe.”
- http://interactions.org/quantumuniverse/

## How Science Drives Architecture

- State-of-the-art computational science requires increasingly diverse and complex algorithms
- Only balanced systems that can perform well on a variety of problems will meet future scientists’ needs!
- Data-parallel and scalar performance are both important

| Science Areas | Multi-Physics and Multi-Scale | Dense Linear Algebra | FFTs | Particle Methods | AMR | Data Parallelism | Irregular Control Flow |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Nanoscience | X | X | X | X |  | X | X |
| Combustion | X |  |  | X | X | X | X |
| Fusion | X | X |  | X | X | X | X |
| Climate | X |  | X |  | X | X | X |
| Astrophysics | X | X | X | X | X | X | X |

## Phil Colella’s “Seven Dwarfs”

- Algorithms that consume the bulk of the cycles of current high-end systems in DOE:
- Structured Grids
- Unstructured Grids
- Fast Fourier Transform
- Dense Linear Algebra
- Sparse Linear Algebra
- Particles
- Monte Carlo
- (Should also include optimization / solution of nonlinear systems, which at the high end is something one uses mainly in conjunction with the other seven)

## “Evaluation of Leading Superscalar and VectorArchitectures for Scientific Computations”

- Leonid Oliker, Andrew Canning, Jonathan CarterLBNL
- Stephane EthierPPPL
- (see SC04 paper at _[http://crd.lbl.gov/~oliker/](http://crd.lbl.gov/~oliker/)_ )

## Material Science: PARATEC

- PARATEC performs first-principles quantum mechanical total energy calculation using pseudopotentials & plane wave basis set
- Density Functional Theory to calculate structure & electronic properties of new materials
- DFT calc are one of the largest consumers of supercomputer cycles in the world

- PARATEC uses all-band CG approach to obtain wavefunction of electrons
- Part of calc. in real space other in Fourier space using specialized 3D FFT to transform wavefunction
- Generally obtains high percentage of peak on different platforms
- Developed by A. Canning (LBNL) with Louie and Cohen’s groups (UCB, LBNL), Raczkowski

**Notes:**
DFT has become standard technique in Mat Sci to calculate the structural and electronic properties of new materials with a full QUANTUM MECHANICAL treatment of the electrons.
Picture shows induced current and charge density in crystalized glcine.The simulations used to better undersand nuclear magnetic resonance.

## PARATEC: Code Details 

- Code written in F90 and MPI  (~50,000 lines)
- 33% 3D FFT,  33% BLAS3,  33% Hand coded F90
- Global Communications in 3D FFT  (Transpose)
- 3D FFT handwritten, minimize comms. reduce latency (written on top of vendor supplied 1D complex FFT )
- Code has setup phase then performs many (~50) CG steps to converge the charge density of the system (data on speed is for 5CG steps, does not include setup)

## PARATEC: 3D FFT

- 3D FFT done via 3 sets of 1D FFTs and 2 transposes
- Most communication in global transpose (b) to (c) little communication (d) to (e)
- Many FFTs done at the same time to avoid latency issues
- Only non-zero elements communicated/calculated
- Much faster than vendor supplied 3D-FFT

**PARATEC: 3D FFT**

- (a)

- (b)

- (e)

- (c)

- (f)

- (d)

**Source: Andrew Canning, LBNL**

## PARATEC: Performance

| DataSize | P | Power 3 |  | Power4 |  | Altix |  | ES |  | X1 |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak |
| 432Atom | 32 | 0.95 | 63% | 2.0 | 39% | 3.7 | 62% | 4.7 | 60% | 3.0 | 24% |
|  | 64 | 0.85 | 57% | 1.7 | 33% | 3.2 | 54% | 4.7 | 59% | 2.6 | 20% |
|  | 128 | 0.74 | 49% | 1.5 | 29% | --- | --- | 4.7 | 59% | 1.9 | 15% |
|  | 256 | 0.57 | 38% | 1.1 | 21% | --- | --- | 4.2 | 52% | --- | --- |
|  | 512 | 0.41 | 28% | --- | --- | --- | --- | 3.4 | 42% | --- | --- |
| 686Atom | 128 |  |  |  |  |  |  | 4.9 | 62% | 3.0 | 24% |
|  | 256 |  |  |  |  |  |  | 4.6 | 57% | 1.3 | 10% |

## Magnetic Fusion: GTC

- Gyrokinetic Toroidal Code: transport of thermal energy (plasma microturbulence)
- Goal magnetic fusion is burning plasma power plant producing cleaner energy
- GTC solves gyroaveraged gyrokinetic system w/ _particle-in-cell approach_ (PIC)
- PIC scales N instead of N2 – particles interact w/ electromag field on grid
- Allows solving equation of particle motion with ODEs (instead of nonlinear PDEs)

- Main computational tasks:
- _Scatter_: deposit particle charge to nearest grid points
- _Solve_ the Poisson eqn to get potential at each grid point
- _Gather_: Calc force on each particle based on neighbors potential
- _Move_ particles by solving eqn of motion along the characteristics
- _Find_ particles moved outside local domain and update
- Developed at Princeton Plasma Physics Laboratory, vectorized by Stephane Ethier

**Notes:**
Performance of such device is determine by rate at which energy is transported out of hot core to the colder edge of plasma
PIC: Instead of interacting with one another, particles interact with electromagnetic field described on a grid

## GTC: Performance

| Number Particles | P | Power 3 |  | Power4 |  | Altix |  | ES |  | X1 |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak | Gflops/P | %peak |
| 10/cell
20M | 32 | 0.13 | 9% | 0.29 | 5% | 0.29 | 5% | 1.15 | 14% | 1.00 | 8% |
|  | 64 | 0.13 | 9% | 0.32 | 5% | 0.26 | 4% | 1.00 | 13% | 0.80 | 6% |
| 100/cell
200M | 32 | 0.13 | 9% | 0.29 | 5% | 0.33 | 6% | 1.62 | 20% | 1.50 | 12% |
|  | 64 | 0.13 | 9% | 0.29 | 5% | 0.31 | 5% | 1.56 | 20% | 1.36 | 11% |
|  | 1024 | 0.06 | 4% |  |  |  |  |  |  |  |  |

- GTC is now scaling to 2048 processors on the ES for a total of 3.7 TFlops/s

**Notes:**
Scalar units compute indices for indirect addressing: vector memory fetch dependant on scalar calculations

## Application Status in 2005

- A few Teraflop/s sustained performance
- Scaled to 512 - 1024 processors

**Parallel job size at NERSC**

**Notes:**
latency?
particle methods?
fusion is MHD

## Applications on Petascale Systems will need to deal with

- (Assume nominal Petaflop/s system with 100,000 commodity processors of 10 Gflop/s each)
**Three major issues:**
**Scaling to 100,000 processors and multi-core processors**
**Topology sensitive interconnection network**
**Memory Wall**

**Notes:**
latency?
particle methods?
fusion is MHD

## Integrated Performance Monitoring (IPM)

- brings together multiple sources of performance metrics into a single profile that characterizes the overall performance and resource usage of the application
- maintains low overhead by using a unique hashing approach which allows a fixed memory footprint and minimal CPU usage
- open source, relies on portable software technologies and is scalable to thousands of tasks
- developed by David Skinner at NERSC (see _[http://www.nersc.gov/projects/ipm/](http://www.nersc.gov/projects/ipm/)_ )

**Notes:**
latency?
particle methods?
fusion is MHD

## Scaling Portability: Profoundly Interesting

- A high level description of the
- performance of cosmology code MADCAP on four well known architectures.

**Source: David Skinner, NERSC**

## 16 Way for 4 seconds

- (About 20 timestamps per second per task) *( 1...4 contextual variables)

## 64 way for 12 seconds

## Applications on Petascale Systems will need to deal with

- (Assume nominal Petaflop/s system with 100,000 commodity processors of 10 Gflop/s each)
**Three major issues:**
**Scaling to 100,000 processors and multi-core processors**
**Topology sensitive interconnection network**
**Memory Wall**

**Notes:**
latency?
particle methods?
fusion is MHD

## Even today’s machines are interconnect topology sensitive

**Four (16 processor)  IBM Power 3 nodes with Colony switch**

## Application Topology

- 1024 way MILC

- 1024 way MADCAP

- 336 way FVCAM

- If the interconnect is topology
- sensitive, mapping will become an issue (again)

**“****Characterizing Ultra-Scale Applications Communincations Requirements”, by John Shalf et al., submitted to SC05**

## Interconnect Topology BG/L

## Applications on Petascale Systems will need to deal with

- (Assume nominal Petaflop/s system with 100,000 commodity processors of 10 Gflop/s each)
**Three major issues:**
**Scaling to 100,000 processors and multi-core processors**
**Topology sensitive interconnection network**
**Memory Wall**

**Notes:**
latency?
particle methods?
fusion is MHD

## The Memory Wall

- Source: “Getting up to speed: The Future of Supercomputing”, NRC, 2004

## Characterizing Memory Access

_**Memory Access Patterns/Locality**_

**Source: David Koester, MITRE**

**APEX-Map:  A Synthetic Benchmark ****to Explore the Space of Application Performances**

**Erich Strohmaier, Hongzhang Shan****Future Technology Group, LBNL****EStrohmaier@lbl.gov**
**Co-sponsored by DOE/SC and NSA**

## Apex-MAP characterizes architectures through a synthetic benchmark

## Apex-Map Sequential

## Apex-Map Sequential

## Apex-Map Sequential

## Apex-Map Sequential

## Parallel APEX-Map

## Parallel APEX-Map

## Parallel APEX-Map

## Parallel APEX-Map

## Parallel APEX-Map

## Summary

- Applications will face (at least) three challenges in the next five years
- Scaling to 100,000s of processors
- Interconnect topology
- Memory access
- Three sets of tools (applications benchmarks, performance monitoring, quantitative architecture characterization) have been shown to provide critical insight into applications performance