





50

High Performance Computing and
Big Data:
Challenges for the FutureJack Dongarra
University of Tennessee
Oak Ridge National Laboratory
University of Manchester

7/10/191

Outline• Overview of High Performance Computing• Look at issues on convergence of HPC and Big Data
• Killer apps for HPC, BD, and Machine Learning.

2

HPCI Tier 2 Systems Roadmap (As of Nov. 2018
Fiscal Year

Hokkaido

2016

2017

2018

HITACHI
SR16000/M1172TF, 22TBHITACHI SR16000/M1172TF,
22TB
Cloud System BS2000 44TF, 14TB

Cloud System BS2000 44TF, 14TB 
Data Science Cloud / Storage HA8000 / WOS7000

10TF, 1.96PB

TohokuTsukuba

2020

PPX1(62TF)

HAPACS(1166TF)

2022

NagoyaKyoto
Osaka
KyushuJAMSTEC
ISM

Fujitsu FX100 (2.9PF, 81 TiB)
Fujitsu CX400 (774TF, 71TiB)

100~200 PF,
100-200 PB/s
(CFL-D/CFL-D) ~4 MW

PACS-XI 100PF (TPF)

1.4MW (total)

Cray XC40(5.5PF) + CS400(1.0PF)
1.33 MW(983TF)

Cray XC30 (584TF)

NEC SX-ACE NEC Express5800(423TF) (22.4TF)
HA8000 (712TF, 242 TB)
SR16000 (8.2TF, 6TB)

2.0MW

FX10 (272.4TF, 36 TB)
CX400 (966.2 TF, 183TB)

20+ PF (FAC/UCC + CFL-M)

100+ PF (FAC/UCC+CFL-M)

up to 3MW

up to 3MW

20-40+ PF(FAC/TPF + UCC) 1.5 MW

80-150+ PF(FAC/TPF + UCC) 2 MW

3.2PB/s,15~25Pflop/s, 1.0-1.5MW (CFL-M)

OCTPUS 1.463PF (UCC)

25.6 PB/s, 50-100Pflop/s(TPF)
1.5-2.0MW

100+ PF(FAC/TPF + UCC/TPF)

Fujitsu PRIMERGY CX subsystem A + B, 10.4 PF (UCC/TPF) 2.7MW
FX10 (90.8TFLOPS)

3MW

100PF, 3MW

SX-ACE(1.3PF, 320TiB) 3MW
UV2000 (98TF,
128TiB) 0.3MW

200+ PF(FAC)
6.58.0MW

TSUBAME 4.0 (~100 PF, ~10PB/s, ~2.0MW)

2MW in total
Cray:XE6 +GB8K XC30

2027

BDEC 60+ PF (FAC) 3.5-4.5MW

  5 PF(542TF, 71TiB)

2026

100+ PF 4.5-6.0MW(UCC + TPF)

Oakbridge-II 4+ PF 1.0MW

TSUBAME 3.0 (12.15 PF, 1.66 PB/s)
TSUBAME 2.6  (5.7 PF, 110+ TB, 1160 TB/s), 1.4MW

2025

35 PF UCC +CFL-M 2MW

Oakforest-PACS (OFP) 25 PF(UCC + TPF) 3.2 MW

Reedbush-U/H: 1.92 PF (FAC) 0.7MW (Reedbush-U20206)Reedbush-L1.4 PF (FAC) 0.2 MW

TSUBAME 2.5

2024~30PF, ~30PB/s Mem BW (CFL-D/CFL-M) ~3MW

Hitachi SR16K/M1 (54.9 TF, 10.9 TiB, 28.7 TB/s)

Tokyo Tech.

2023

Cygnus 3+PF (TPF) 0.4MW

PPX2 (62TF)

Fujitsu FX10 (Oakleaf/Oakbridge) (1.27PFlops, 168TiB, 460 TB/s),

2021

3.96 PF UCC + CFL/M 0.9 MW
0.16 PF Cloud 0.1MW

SX-ACE(707TF,160TB, 655TB/s)
LX406e(31TF), Storage(4PB), 3D Vis, 2MW

COMA (PACS-IX) (1001 TF)

Tokyo

2019

2PF, 0.3MW
Power is the maximum consumption including coolingSince 1993
H. Meuer, H. Simon, E. Strohmaier, & JD

- Listing of the 500 most powerful Computers in the World- Yardstick: Rmax from LINPACK MPP
Ax=b, dense problem- Updated twice a year
SC‘xy in the States in November
Meeting in Germany in JuneRate

TPP performance

Size- All data available from www.top500.org
4State of Supercomputing in 2019
• Pflops (> 1015 Flop/s) computing fully established with all 500
systems.
• Three technology architecture possibilities or “swim lanes” arethriving.
• Commodity (e.g. Intel)• Commodity + accelerator (e.g. GPUs) (133 systems)• Special purpose lightweight cores (e.g. IBM BG, Knights Landing, TaihuLight,
ARM (1 system))• Interest in supercomputing is now worldwide, and growing in many
new markets (~50% of Top500 computers are in industry).
• Intel processors largest share, 96% followed by AMD, .6%.
• Exascale (1018 Flop/s) projects exist in many countries and regions.

5June 2019: The TOP 10 Systems (1/3 of the Total Performance)
Rank

Site

ComputerCountry

Cores

Rmax
[Pflops]

% of
Peak

Power GFlops/
[MW]
Watt

1

DOE / OSOak Ridge Nat Lab

Summit, IBM Power 9 (22C, 3.0GHz),
Nvidia GV100 (80C), Mellonox EDR

USA

2,397,824

149.

74

11.1

14.7

2

DOE / NNSA
L Livermore Nat Lab

Sierra, IBM Power 9 (22C, 3.1GHz),
Nvidia GV100 (80C), Mellonox EDR

USA

1,572,480

94.6

75

7.44

12.7

3

National Super Computer
Center in Wuxi

Sunway TaihuLight, SW26010 (260C) + Custom

China

10,649,000

93.0

74

15.4

6.05

4

National Super Computer
Center in Guangzhou

Tianhe-2A NUDT,
Xeon (12C) + MATRIX-2000 + Custom

China

4,981,760

61.4

61

18.5

3.32

5

Texas Advanced Computing Frontera, Dell C6420, Xeon Platinum, 8280 28C 2.7
Center / U of Texas
GHz, Mellanox HDR

USA

448,448

23.5

61

6

Swiss CSCS

Piz Daint, Cray XC50, Xeon (12C) +Nvidia P100 (56C) + Custom

Swiss

387,872

21.2

78

2.38

8.90

7

DOE / NNSA
Los Alamos & Sandia

Trinity, Cray XC40,Xeon Phi (68C) + Custom

USA

979,968

20.2

49

7.58

2.66

8

Nat Inst of Advanced
Indust Sci & TechAI Bridging Cloud Infrast (ABCI) Fujitsu
Xeon (20C, 22.4GHz) Nvidia V100 (80C) IB-EDR

Japan

391,680

16.9

61

1.65

12.05

9

Leibniz Rechenzentrum

SuperMUC-NG, Lenovo,ThinkSystem SD530, Xeon
Platinum 8174 24C 3.1GHz, Intel Omni-Path

Germany

311,040

19.5

72

10

DOE / NNSA
L Livermore Nat Lab

Lassen, IBM Power System p9 22C 3.1 GHz,
Mellanox EDR, Nvdida V100 (80C)USA

288,288

18.2

79

PERFORMANCE DEVELOPMENT OF HPC OVER THE LAST 26
YEARS FROM THE TOP500
1E+10

1.56 EFlop/s

1 Eflop/s
1E+09

148 PFlop/s

100 Pflop/s
100000000
10 Pflop/s
10000000

SUM

11000000
Pflop/s

N=1

100
Tflop/s
1000001.02 PFlop/s

N=5001010000
Tflop/s

1 Tflop/s
1000
1.17 TFlop/s
100 Gflop/s
100

My Laptop: 166 Gflop/s

59.7 GFlop/s

10 Gflop/s
10 Laptop > Fujitsu VPP500 with 80 processors

1 Gflop/s
1

100 Mflop/s
0.1

400 MFlop/s
1994

1996

1998

2000

2002

2004

2006

2008

2010

2012

2014

2016

20182019

PERFORMANCE DEVELOPMENT
1E+10

SUM

1 Eflop/s

1E+09

100 Pflop/s

N=1

10 Pflop/s

N=10100000000

10000000

1 Pflop/s

N=100

1000000

100 Tflop/s

100000

10 Tflop/s

10000

1 Tflop/s

1000

100 Gflop/s

100

10 Gflop/s

10

1 Gflop/s

1

81994 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 2020
2022
Tflops (1012)
Achieved
ASCI Red
Sandia NL

Pflops (1015)
Achieved
RoadRunner
Los Alamos NL

Eflops (1018)
Achieved?
China says 2020
U.S. says 2021

COUNTRIES SHARE
Chart Title of Systems in Country
Count of Number

China
US
Japan
UK
FranceGermany
Canada
Italy
Poland050100China has 42% of the systems
US has 23% of the systems
Japan has 5.8% of the systems
In terms of performance:
US has 38%
China has 30%
Japan has 7.5%

150

200

250

219
116
29
18
19
14
8
5
1

Performance Distribution
Top50

Top500160000160000140000140000120000120000100000100000

80000

80000

60000

60000

40000

40000

20000

20000

0

0
0

50

100

150

200

250

300

350

For all 500 systems
Tpeak500 = 1.022 Pflop/s
10

7/10/19

400

450

500

0

5

10

15

20

25

30

35

40

45

50

For top 50 systems
Tpeak50 = 3.944 Pflop/s
54% of the total performance of Top500 in Top50

COUNTRIES (TOP50) / SYSTEM SHARE
United States
China

Others, 5, 10%

Japan
FranceItaly, 2, 4%

United Kingdom
United States, 19,
38%

Germany, 4, 8%

Germany
Italy
OthersUnited Kingdom, 3, 6%
France, 5,10%Japan, 9, 18%

China, 3, 6%

VENDORS (TOP50) / SYSTEM SHARE
Cray Inc.
HPE
IBM
Bull, 3, 6%

Others, 8, 16%

Fujitsu
LenovoCray Inc., 14, 28%

Lenovo, 3, 6%

Bull
OthersFujitsu, 5,10%IBM, 7, 14%

HPE, 10, 20%

24 #1 Systems for the Top500

13

Current #1 System Overview
System Performance
•

•

15

Peak performance of 200
Pflop/s for modeling &simulation
Peak performance of
3.3 Eflop/s for 16 bit
floating point used in fordata analytics, ML, andartificial intelligence

Each node has•

2 IBM POWER9 processors• Each w/22 cores• 2.3% performance of system•

6 NVIDIA Tesla V100 GPUs
• Each w/80 SMs• 97.7% performance ofsystem••608 GB of fast memory
1.6 TB of NVMe memory

The system includes• 4608 nodes• 27,648 GPUs
• Street value $15K each

• Dual-rail Mellanox EDR
InfiniBand network• 250 PB IBM Spectrum
Scale
file system transferring
data at 2.5 TB/s

Department of Energy (DOE) Roadmap to Exascale Systems
An impressive, productive lineup of accelerated node systems supporting DOE’s mission
Pre-Exascale Systems [Aggregate Linpack (Rmax) = 323 PF]
2012

2016

2018

Titan (9)IBM BG/Q

ORNL>1.5 Eflop/s

Theta (24)ANL

AMD/Cray

Cray/Intel KNL

NERSC-9
Perlmutter

LBNL

LBNL

Cray/Intel Xeon/KNL

LLNL
IBM BG/Q$1.8 B,
Just for 3 Hardware systems at Exascale

IBM/NVIDIA

Cori (12)Sequoia (10)2021-2023

ORNL

Cray/AMD/NVIDIA

ANL

2020

Summit (1)ORNL

Mira (21)First U.S. Exascale Systems

Trinity (6)LANL/SNL
Cray/Intel Xeon/KNL

Cray/AMD/NVIDIA

LLNL

ANL
Intel/Cray

LLNL

Sierra (2)IBM/NVIDIA

AuroraTBD

LANL/SNL
TBD

16

As#scien(ﬁc#research#increasingly#depends#on#both#high5speed##compu(ng#and#data#analy(cs,#the#poten(al#interoperability#and##scaling#convergence#of#these#two#ecosystems#is#crucial#to#the#future.##D. Reed & J. Dongarra, CACM 2015

Reed & Dongarra, CACM 2015

As#scien(ﬁc#research#increasingly#depends#on#both#high5speed##compu(ng#and#data#analy(cs,#the#poten(al#interoperability#and##scaling#convergence#of#these#two#ecosystems#is#crucial#to#the#future.##Its All About Convergence• High-end data analytics (big data) and HPC are bothessential elements of an integrated computing
research-and-development agenda; neither should besacrificed or minimized to advance the other.
• Programming models and tools are perhaps the biggest
point of divergence between the scientific-computing
and big-data ecosystems.

Why Converge? (or Mostly Converge)
Independent paths: More Cost, Less Science,• $ multiple hardware software infrastructures• $ developing software for two communities• $ learning two computing models• $ smaller discovery community, fewer ideas• Less science

Comparing Architecture• Scientific HP Computing• Significant Cost in memory
and interconnect bandwidth• Significant Cost in resiliencehardware to reduce wholesystem MTTI

• Scientific Big Data
• ? Cost in memory andinterconnect bandwidth• Little Cost for hardware to
support system-wideresilience

Comparing Operations• Scientific HP Computing• New tightly integrated systempurchased every 4 years• Users charged for CPU hours,
storage and networking is free
• Periodic access to computeresources via job submitted toscheduler and queue• Space-shared compute resources
for exclusive access during jobs

• Scientific Big Data
• New hardware capacity purchased
incrementally• Users charged for all resources(storage, cpu, networking)• Continuous access to long-lived“services” created by science
community• Time-shared access to elastic
resources

Comparing Data
• Scientific HP Computing• Inputs arrive infrequently,
buffering carefully managed• Data often reproducible (repeat
simulation)• Data generated from simulation(error: from simulation)• Data rate limited by platform• Scientific Big Data
• Inputs arrive continuously,
streaming workflows• Data is unrepeatable
snapshot in time
• Data generated by sensors(error: from measurement)• Data rate limited by sensors

Square Kilometre Array• Next Generation radio telescope – compared to best current
instruments it is ...
• ~100 times sensitivity• ~ 106 times faster imaging the sky• More than 1 square km of collecting area on
baseline 150km• Will address some of the key problems of astrophysics and cosmology(and physics)• Builds on techniques developed in Europe• It is an interferometer• Uses innovative technologies...
• Major data/compute project• Need performance at low unit cost24Science

Phase I : 2020

Exploring the Universe
with the World’s Largest Radio Telescope

50 MHz130,000 element
Low Frequency Aperture Array

Exoplanets

100 MHz

Cosmology &Galaxy Evolution

200 dishes

Pulsars

1 GHz

Cosmic Magnetism

10 GHzCradle of Life

Footer text

Standard Interferometers

Astronomical signal(EM wave)

B.s

V(B) = E1 E2*= I(s) exp( i w B.s/c )Detect &amplifyB

1

Digitise &
delay

2

• Resolution determined by
maximum baseline
qmax ~ l / Bmax

X

• Field of View (FoV) determined by
the size of each dishqdish ~ l / D

Correlate

X

X

X

X

Integrate
Process

Visibility:

Calibrate, grid, FFTX

SKY Image26

SKA: A Leading Big Data Challenge for 2020
Digital Signal Processing(DSP)
Antennas

Imaging (SDP) – HPC problem
2020: 100 PBytes/day
2028: 10,000 PBytes/day
Over 10’s to 1000’s kms

Transfer antennas to DSP
2020: 20,000 PBytes/day
2028: 200,000 PBytes/day
Over 10’s to 1000’s kms
130,000 antennas
Spread over 65 kms

High Performance Computing
Facility (HPC)
HPC Processing
2020: 300 Pflop/s
2028: 30 Eflop/s27

Science Instrument Continuum28

HPC Computing• The highest concentrations ofcomputing power and storage
are in the “center”.
• Much of the rapid increase indata volumes and the dramatic
proliferation of data generators
is occurring in the edge, where
the processing and storage
infrastructure needed to copewith this rising flood of data is
ad hoc and under provisioned
at best.

29

The Computing Continuum
Fog

IoT/Edge

HPC/Cloud/Instrument

Size

Nano

Micro

Milli

ServerFog

CampusFacility

ExampleAdafruit
TrinketParticle.io
Boron

Array of Things

Linux Box

Co-locatedBlades1000-nodecluster

Datacenter

Memory

0.5K

256K

8GB

32GB

256G

32TB

16PB

Network

BLE

WiFi/LTE

WiFi/LTE

1 GigE

10GigE

40GigE

N*100GigE

Cost

$5

$30

$600

$3K

$50K

$2M

$1000M

The number of network-connected devices—sensors, actuators, instruments, computers, and data stores—
Now substantially exceeds the number of humans on this planet
We lack a programming and execution model that is inclusive and capable of harnessing
the entire computing continuum to program our new intelligent world.

Machine Learning in Computational Science
Many fields are beginning to adopt machine learning to augment modeling and simulation
methods• Climate• Biology• Drug Design• Epidemology• Materials• Cosmology• High-Energy Physics

ML is changing ScienceØ HPC HW&SW must changeØ Data will “slosh” (model to/from training)Ø Scientists will share models

Some Deep Learning Applications

32

The Edge Computing Revolution
New Sensors
LIDAR

SoftwareDefinedRadiosPowerful
Parallel Edge
Computing

Edge computing anddeep learning withfeedback forcontinuous
improvement
HPC

Semantic
OutputCamerasHyperspectral
Imaging

Reduced, Compressed data

New inference (program code)
Microfluidic
Sensors

AI/ML Inference

Deep Learning
Training

The Take Away
• Three computer revolutions • Technical implications§ High performance computing§ Deep learning§ Edge & AI

• The very small (edge/fog
computing and sensors)• The very large (clouds,
exascale, and big data)§ Fluid end-to-end
cyberinfrastructure§ Interdisciplinary data andinfrastructure planning• Cultural implications§ Change management andstrategic planning§ Community collaboration

Harnessing the computing continuum will catalyze new consumer
services, business processes, social services, and scientific discovery.

34