Why we need Exascale and why we
won’t get there by 2020
Horst Simon
Lawrence Berkeley National Laboratory
August 27, 2013

Overview• Current state of HPC: petaflops firmly

established• Why we won’t get to exaflops (in 2020)• The case for exascale­ Science (old)­ Science (new)­ Technology and economic competitiveness­ Discovery

Thank you to the DOE Hexlab group, the TOP500 team, John Shalf, Kathy
Yelick and many others22Current State of HPC: petaflops firmly established33Performance Development
1 Eflop/s

224 PFlop/s

100 Pflop/s

33.9 PFlop/s

10 Pflop/s

1 Pflop/s
96.6 TFlop/s

SUM

100 Tflop/s
10 Tflop/s

1 Tflop/s 1.17 TFlop/
100 Gflop/s
10 Gflop/s

N=1s
59.7 GFlop/s

N=500

1 Gflop/s
100 Mflop/s 400 MFlop/s

Source: TOP500 June 2013

4

4

Top 10 Systems in June 2013#

Site

Manufacturer

1

National University of Defense
Technology

NUDT

2

Oak Ridge National Laboratory

Cray

3

Lawrence Livermore National
Laboratory

IBM

4

RIKEN Advanced Institute for
Computational Science

Fujitsu

5

Argonne National Laboratory

IBM

6

Texas Advanced Computing
Center/UT

Dell

7

Forschungszentrum Juelich
(FZJ)

IBM

8

Lawrence Livermore National
LaboratoryIBM

9

Leibniz Rechenzentrum

IBM

10

National SuperComputer
Center in Tianjin

NUDT

Computer
Tianhe-2
NUDT TH-IVB-FEP,
Xeon 12C 2.2GHz, IntelXeon Phi
Titan
Cray XK7, Opteron 16C 2.2GHz,
Gemini, NVIDIA K20x
Sequoia
BlueGene/Q,
Power BQC 16C 1.6GHz, Custom
K Computer
SPARC64 VIIIfx 2.0GHz,
Tofu Interconnect
Mira
BlueGene/Q,
Power BQC 16C 1.6GHz, Custom
Stampede
PowerEdge C8220,
Xeon E5 8C 2.7GHz, Intel Xeon Phi
JuQUEENBlueGene/Q,
Power BQC 16C 1.6GHz, Custom
VulcanBlueGene/Q,
Power BQC 16C 1.6GHz, Custom
SuperMUC
iDataPlex DX360M4,
Xeon E5 8C 2.7GHz, Infiniband FDR
Tianhe-1A
NUDT TH MPP,
Xeon 6C, NVidia, FT-1000 8C

Rmax Power[Pflops] [MW]Country

Cores

China

3,120,000

33.9

17.8

USA

560,640

17.6

8.21

USA

1,572,864

17.2

7.89

Japan

795,024

10.5

12.7

USA

786,432

8.59

3.95

USA

462,462

5.17

4.51

Germany

458,752

5.01

2.30

USA

393,216

4.29

1.97

Germany

147,456

2.90

3.52

China

186,368

2.57

4.04

Technology Paths to Exascale

Leading Technology Paths (Swim Lanes):

Multicore

Maintain complex cores, and replicate(x86, SPARC, Power7) [#4 and 9]

Manycore /EmbeddedUse many simpler, low power cores fromembedded (BlueGene) [ #3, 5, 7, and 8]

GPU /Accelerator

Use highly specialized processors fromgaming/graphics market space (NVidia
Fermi, Cell, Intel Phi (MIC) ) [# 1, 2, 6, and 8]

6

6

Tianhe-2 (TH-2) at NUDT, China

77

Sequoia at Lawrence Livermore National
Laboratory (Manycore/embedded)• Fully deployed in June 2012
• 16.32475 petaflops Rmax and 20.13266 petaflops Rpeak• 7.89 MW Power consumption• IBM Blue Gene/Q• 98,304 compute nodes• 1.6 million processor cores• 1.6 PB of memory• 8 or 16 core Power Architecture processors

built on a 45 nm fabrication method• Lustre Parallel File System

88

K Computer at RIKEN Advanced Institute for
Computational Science (Multicore)• #3 on the TOP500 list
• Distributed memory architecture• Manufactured by Fujitsu• Performance: 10.51 PFLOPS Rmax and 11.2804 PFLOPS Rpeak• 12.65989 MW Power Consumption, 824.6 GFLOPS/Kwatt• Annual running costs - $10 million• 864 cabinets:− 88,128 8-core SPARC64 VIIIfx processors@ 2.0 GHz− 705,024 cores− 96 computing nodes + 6 I/O nodes per
cabinet• Water cooling system minimizes
failure rate and power consumption

99

Choice of Swim Lanes• Swim Lane choices – currently all three are equally

represented• Risk in Swim Lane switch for large facilities­ Select too soon: Users cannot follow­ Select too late: Fall behind performance curve­ Select incorrectly: Subject users to multiple disruptive technology

change1010Events of 2011/2012 and the Big Questions
for the Next Three Years until 2015
Leading Technology Paths (Swim Lanes):
IBM cancels “Blue Waters” contract.
Is this an indication that multicore withcomplex cores is nearing the end of the line?
Blue Gene is the last of the line.
Will there be no more large scale
embedded multicore machines?
Intel bought Cray’s interconnect
technology and WhamCloud andintroduced Xeon-Phi.
Will Intel develop complete systems?

Multicore

Manycore /EmbeddedGPU /Accelerator

Prediction: All Top 10 systems in 2015
will be GPU/Accelerator based11
11

State of Supercomputing in 2013
• Pflops computing fully established with more than 20

machines• Three technology “swim lanes” are thriving• Interest in supercomputing is now worldwide, andgrowing in many new markets• Exascale projects in many countries and regions• Rapid growth predicted by IDC for the next three years

1212Why we won’t get to Exaflops by 2020

13

13

The Power and Clock Inflection Point in 2004
1.E+04

NuclearReactor

100

Clock(MHz)

1.E+03

HotPlate10

1.E+02

100Wlightbulb
1

HistoricalSingleCore

2020

2015

2010

2005

2000

1995

1990

1985

1980

1/1/16

1/1/12

1/1/08

1/1/04NodeProcessorClock(MHz)(H)
NodeProcessorClock(MHz)(L)NodeProcessorClock(MHz)(M)
AccleratorCoreClock(MHz)(H)
AccleratorCoreClock(MHz)(L)
AccleratorCoreClock(MHz)(M)HistoricalMultiͲCore

ITRSHiPerf

Source: Kogge and Shalf, IEEE CISE

1/1/000

1/1/96

1/1/92

1.E+01

1975

WattsperSquarecm

1,000

14

14

Towards Exascale• Exascale has been discussed in numerous workshops,

conferences, planning meetings for about six years• Exascale projects have been started in the U.S. andmany other countries and regions• Key challenges to exascale remain

15

15

Projected Performance Development

1 Eflop/s
100 Pflop/s
10 Pflop/s

1 Pflop/s

SUM

100 Tflop/s
10 Tflop/s

N=1

1 Tflop/s
100 Gflop/s

N=500

10 Gflop/s

1 Gflop/s
100 Mflop/s

Data from: TOP500 June 2013

16

16

The Exascale Challenge• There is a lot of vagueness in the exascale discussion
and what it means to reach exascale. Let me propose aconcrete and measurable goal.
• Build a system before 2020 that will be #1 on the TOP500
list with an Rmax > 1 exaflops• Personal bet (with Thomas Lippert, Jülich, Germany) that
we will not reach this goal by November 2019 (for $2,000or €2000)• Unfortunately, I think that I will win the bet. But I would
rather see an exaflops before the end of the decade andloose my bet.

17

17

At Exascale, even LINPACK is a Challenge

Source: Jack Dongarra

http://bit.ly/hpcg-benchmark
1818

At Exascale HPL will take 5.8 days

Source: Jack Dongarra, ISC’12

19

19

Total Power Levels (MW) for TOP500 Systems

Source: TOP500 June 201320

20

Power Consumption

5.04 x in 5 y

3.13 x in 5 y

3.25 x in 5 y

Source: TOP500 June 201321

21

Power Efficiency went up significantly in 2012

Data from: TOP500 June 2013

22

22

Most Power Efficient Architectures
Rmax/
Power

Computer
Adtech, ASUS, Xeon 8C 2.0GHz, Infiniband FDR, AMD FirePro

2,973

Appro GreenBlade, Xeon 8C 2.6GHz, Infiniband FDR, Intel Xeon Phi

2,450

BlueGene/Q, Power BQC 16C 1.60 GHz, Custom

2,300

Cray XK7, Opteron 16C 2.1GHz, Gemini, NVIDIA Kepler

2,243Eurotech Aurora HPC, Xeon 8C 3.1GHz, Infiniband QDR, NVIDIA K20

2,193

iDataPlex DX360M4, Xeon 8C 2.6GHz, Infiniband QDR, Intel Xeon Phi

1,935

Tianhe-2, NUDT, Intel Xeon 6C 2.2GHz, TH Express-2, Intel Xeon Phi

1,902

RSC Tornado, Xeon 8C 2.9GHz, Infiniband FDR, Intel Xeon Phi

1,687

SGI Rackable, Xeon 8C 2.6GHz, Infiniband FDR, Intel Xeon Phi

1,613

Chundoong Cluster, Xeon 8C 2GHz, Infiniband QDR, AMD Radeon HD

1,467[Mflops/Watt]
Data from: TOP500 June 201323

23

Power Efficiency over TimeAMD	
  FirePro	
  Intel	
  Mic	
  BlueGene/Q	
  

Cell	
  

Data from: TOP500 June 201324

24

Power Efficiency over Time

Linpack/Power	  	  	
  [Gﬂops/kW]	
  

3,000
2,500
2,000

One time
technology
improvement, nota change in trend
rate

1,500

TOP10

1,000

TOP50	
  500

TOP500	
  

0
2008

2009

Data from: TOP500 November 2012

2010

25

2011

2012

25

The Problem with Wires:
Energy to move data proportional to distance• Cost to move a bit on copper wire:• power= bitrate * Length / cross-section area

• Wire data capacity constant as feature size shrinks• Cost to move bit proportional to distance• ~1TByte/sec max feasible off-chip BW (10GHz/pin)• Photonics reduces distance-dependence of bandwidth
Photonics requires no redrive
and passive switch little power

Copper requires to signal amplificationeven for on-chip connections

26

Adapted from John Shalf

26

The Cost of Data Movement

pico joules / bit

10000	
  

SMP	
  

MPI	

1000	
  

100	
  
now	
  

10	
  

1	
  

27

Adapted from John Shalf

27

The Cost of Data Movement

pico joules / bit

10000

SMP	
  

MPI

1000

100

CMP

Cost of a FLOP
now

10

1

28

Adapted from John Shalf

28

The Cost of Data Movement in 2018
FLOPs will cost lessthan on-chip data
movement! (NUMA)

1000"

FLOPs

100"

now#
2018#

10"

#sy
st
em

#

None of the fast
forward projects
addresses thischallenge

Cr

os
s

co
nn
ec

t#

#
lo
ca
l#i
nt
er

hi
p/
DR
AM

#o
n2
ch
ip
#
m

5m

Oﬀ
2c

#o
n2
ch
ip
#
m

#

1m

gi
st
er
Re

#FL
OP

#

1"
DP

picoPicoJoules#
joules / bit

10000"

29

Adapted from John Shalf29

It’s the End of the World as We Know It!
2.25
2.00
1.75
1.50
1.25
1.00

Rmax(Gflop/s)

TotalCores

AveCycles/secpercore(Mhz)

Mem/Core(GB)Source: Kogge and Shalf, IEEE CISE

30

01/01/12

01/01/08

01/01/04

0.50

01/01/00

0.75

01/01/96

CompoundAnnualGrowthRate:CAGR

Summary Trends

30Why I believe that I will win my bet
1. HPL at Exaflop/s level will take about a week to run

challenge for center directors and new systems
2. We realized a one time gain in power efficiency byswitching to accelerator/manycore. This is not asustainable trend in the absence of other new
technology.
3. Data movement will cost more than flops (even on

chip)
4. Limited amount of memory, low memory/flop ratios(processing is free)31

31

The Logical Conclusion
If FLOPS are free, then why do we need an “exaflops”
initiative?

“Exaflops”“Exascale”“Exa”-anything has become a bad brand• Associated with buying big machines for the labs
• Associated with “old” HPC• Sets up the community for “failure” if “goal” can’t be
met
32

32

It is not just “exaflops” – we are changing thewhole computational model
Current programming systems have WRONG optimization targets
Old Constraints

New Constraints• Peak clock frequency as primary• Power is primary design constraint
limiter for performance improvement for future HPC system design• Cost: FLOPs are biggest cost forsystem: optimize for compute• Concurrency: Modest growth ofparallelism by adding nodes• Cost: Data movement dominates:
optimize to minimize data movement• Concurrency: Exponential growth ofparallelism within chips• Memory scaling: maintain byte per • Memory Scaling: Compute growing

flop capacity and bandwidth

2x faster than capacity or bandwidth• Locality: MPI+X model (uniform• Locality: must reason about data
costs within node & between nodes) locality and possibly topology• Uniformity: Assume uniform

system performance• Heterogeneity: Architectural andperformance non-uniformity increase!###"

On4chip++/+CMP+
communica2on+

!##"

21@"
A#!B"

!#"

"

*4
."
22

-"?.*
0

,2
.*
/4
1

>/
1-

"

"
):
;
9$
<1

4=
<"

"

23
45
,6

(8
34
5,6

23
45
,6

70
0
"1

!0
0
"1

)*
+,.*
/"

!"
$%
"&'
(%
"

problem• Reliability: Cannot count on
hardware protection alone

Picojoules*Per*Opera/on*• Reliability: It’s the hardware’s

Internode/MPI+
Communica2on+

Intranode/SMP+
Communica2on+!####"

Fundamentally breaks our current programming paradigm
and computing ecosystem
Adapted from John Shalf
33

33

The Science Case for Exascale (old)3434Climate change analysis

Simulations

Extreme data

• Cloud resolution, quantifying
uncertainty, understanding
tipping points, etc., will drive
climate to exascale platforms• “Reanalysis” projects need 100× morecomputing to analyze observations• Machine learning and other analytics
are needed today for petabyte data
• New math, models, and systems
sets
support will be needed• Combined simulation/observation will
empower policy makers and scientists
35

35

Qualitative Improvement of Simulation with
Higher Resolution (2011)

Michael Wehner, Prabhat, Chris Algieri, Fuyu Li, Bill Collins, Lawrence Berkeley National Laboratory; Kevin Reed, University ofMichigan; Andrew Gettelman, Julio Bacmeister, Richard Neale, National Center for Atmospheric Research

36

36The Science Case for Exascale (new)3737

Materials Genome
Computing 1000× today

Data services for industry and science• Key to DOE’s Energy Storage Hub• Results from tens of thousandsof simulations web-searchable• Tens of thousands of simulations
used to screen potential materials• Need more simulations and fidelity
for new classes of materials, studies
in extreme environments, etc.

• Materials Project launched in October
2012, now has >3,000 registered users• Increase U.S. competitiveness; cut in
half 18 year time from discovery tomarketToday’s batteries

Voltage limit

By 2018:•Increase energy density (70 miles → 350 miles)•Reduce battery cost per mile ($150 → $30)
38

Interestingmaterials!

38

DOE Systems Biology Knowledgebase: KBase• Integration and modeling for predictive biology• Knowledgebase enabling predictive systems biology­ Powerful modeling framework­ Community-driven, extensible and scalable open-source software andapplication system­ Infrastructure for integration and reconciliation of algorithms and data sources­ Framework for standardization, search, and association of data
­ Resource to enable experimental design and interpretation of results

Microbes

Communities
39

Plants39

Obama Announces BRAIN Initiative –Proposes $100 Million in his FY2014 Budget• Brain Research through Advancing

Innovative Neurotechnologies• Create real-time traffic maps toprovide new insights into brain
disorders• “There is this enormous mysterywaiting to be unlocked, and the
BRAIN Initiative will change that bygiving scientists the tools they needto get a dynamic picture of thebrain in action and better
understand how we think and how
we learn and how we remember,”
Barack Obama
See Alivisatos et al., Neuron 74, June 2012, pg 970
4040Modha Group at IBM Almaden

Mouse

Rat

Cat

MonkeyHuman

N: 16 x 106

56 x 106763 x 106

2 x 109

22 x 109

S: 128 x 109

448 x 109

6.1 x 1012

20 x 1012

220 x 1012

New results for SC12

AlmadenWatsonWatsonShaheen

LLNL Dawn

LLNL Sequoia

BG/L

BG/L

BG/P

BG/P

BG/Q

December, 2006

April, 2007

March, 2009

May, 2009

June, 2012

Latest simulations in 2012 achieve unprecedented scale of
65*109 neurons and 16*1012 synapses

Towards Exascale

1 Eflop/s
100 Pflop/s
10 Pflop/s

1 Pflop/s

SUM	
  100 Tflop/s
10 Tflop/s

N=1	
  

1 Tflop/s
100 Gflop/s

N=500	
  10 Gflop/s

1 Gflop/s
100 Mflop/s

Data: TOP500 November 201242

42

Towards Exascale – the Power Conundrum• Straight forward extrapolation results in a real time

human brain scale simulation at about 1 - 10 Exaflop/s
with 4 PB of memory• Current predictions envision Exascale computers in2020 with a power consumption of at best 20 - 30 MW• The human brain takes 20W• Even under best assumptions in 2020 our brain will still

be a million times more power efficient

43

43

Exascale is Critical to the Nation to Develop Future Technologies
and Maintain Economic Competitiveness

44

44

U.S. competitive advantage demands exascale
resources• Digital design and prototyping at exascale enable rapid deliveryof new products to market by minimizing the need for expensive,
dangerous, and/or inaccessible testing• Potential key differentiator for American competitiveness• Strategic partnerships between DOE labs and industrial partners

to develop and scale applications to exascale levels

45

45

Exascale computing is key for national security“I think that what we’ve seen is that they may be somewhat
further ahead in the development of that aircraft than our
intelligence had earlier predicted.”—Defense Secretary Robert M. Gates,(The New York Times, Jan. 9, 2011)

Potential adversaries are not unaware of this46

46

Exascale technologies are the foundation for
future leadership in computing
The U.S. is not the only country capable of achieving exascale computing.
The country that is first to exascale will have significant competitive
intellectual, technological, and economic advantages.
Achievement of the power efficiency and reliability goals needed for
exascale will have enormous positive impacts on consumer electronics
and business information technologies and facilities.

The gap in available supercomputing capacity between the United States and the rest of
the world has narrowed, with China gaining the most ground.
Data from TOP500.org; Figure from Science; Vol 335; 27 January 2012

47

47

Exascale is Discovery

4848

Admiral Zheng He’s Voyages in 1416

http://en.wikipedia.org/wiki/Zheng_He
49

49

The Rise of the West
“Like the Apollo moon missions,
Zheng He’s voyages had been aformidable demonstration of wealthand technological sophistication.
Landing … on the East African coastin 1416 was in many ways …
comparable to landing an American
astronaut on the moon in 1969. By
abruptly canceling oceanic
exploration, Yongle’s successors
ensured that the economic benefits
remained negligible.”

50

50

Exascale is the next step in a long voyage of
discovery, we should not give up now and retreat

51

51

Discussion

52

52

Exascale rationale• Exascale computing means performing computer simulations on a machine capable of 10 to the 18th

power floating point operations per second – ~1,000× the current state-of-the-art.
• The U.S. is capable of reaching exascale simulation capability within ten years with cost-effective,

efficient, computers. An accelerated program to achieve exascale is required to meet this goal, and doing
so will enable the U.S. to lead in key areas of science and engineering.
• The U.S. is not the only country capable of achieving exascale computing. The country that is first to

exascale will have significant competitive intellectual, technological, and economic advantages.
• Exascale systems and applications will improve predictive understanding in complex systems (e.g.

climate, energy, materials, nuclear stockpile, emerging national security threats, biology, …)• Tightly coupled exascale systems make possible fundamentally new approaches to uncertainty

quantification and data analytics – uncertainty quantification is the essence of predictive simulation.
• American industry is increasingly dependent on HPC for design and testing in a race to get products tomarket. Exascale computing and partnerships with DOE laboratories could be THE key differentiator, awinning U.S. strategy for advanced manufacturing and job creation.
• The scale of computing developed for modeling and simulation will have a revolutionary impact on data

analysis and data modeling. Much of the R&D required is common to big data and exascale computing.
• Exascale provides a ten year goal to drive modeling and simulation developments forward and to attract

young scientific talent to DOE in a variety of scientific and technological fields of critical importance to the
future of the U.S.
• Achievement of the power efficiency and reliability goals needed for exascale will have enormous positive

impacts on consumer electronics and business information technologies and facilities.
53

53

Summary• There is progress in Exascale with many projects now

focused and on their way, e.g. FastForward, Xstack, and CoDesign Centers in the U.S.
• HPC has moved to low power processing, and the processor

growth curves in energy-efficiency could get us in the rangeof exascale feasibility• Memory and data movement are still more open challenges• Programming model needs to address heterogeneous,massive parallel environment, as well as data locality• Exascale applications will be a challenge just because of their

sheer size and the memory limitations

54

54