Extreme Big Data (EBD)
Convergence of Extreme Computing
and Big Data Technologies	
Global  Scientiﬁc  Information  and  Computing  CenterTokyo  Institute  of  Technology
Hitoshi  SatoBig Data Examples
Rates and Volumes are extremely immense	
Social NW	

Social Simulation	

••Facebook–––••

500 million active users
340 million tweets per day

Target Area: Planet–7 billion people

Input Data––

300 million new websites per year
48 hours of video to YouTube per minute30,000 YouTube videos played per second

Road Network for Planet:
300GB (XML)Trip data for 7 billion people
10KB (1trip) x 7 billion = 70TB
Real-Time Streaming Data(e.g., Social sensor, physical data)•

Simulated Output for 1 Iteration–Genomics	     ––Internet–––Applications(Open Street Map)

Twitter––•〜1 billion users
Average 130 friends
30 billion pieces of content
shared per month

700TB

Weather	 Sequencing)data)(bp)/$))becomes)x4000)per)5)years)
c.f.,)HPC)x33)in)5)years

Phased	  Array	  Radar
1GB/30sec/2	   radars

Himawari
500MB/2.5min

A-­‐1.	  Quality	  ControlA-­‐2.	  Data	  ProcessingB-­‐1.	  Quality	  ControlB-­‐2.	  Data	  Processing①30-­‐sec	  Ensemble	  Forecast	  Simulations
2	  PFLOP②Ensemble
Data	  Assimilation
2	  PFLOPAnalysis	  Data
2GBEnsemble	  Forecasts
シミュレーションシミ
ュレーショ200GB ン
データ
データ

Ensemble	  Analyses
シミュレーション
シミュレーショ200GB ン
データ
データ

③30-­‐minForecast	  Simulation
1.2	  PFLOPRepeat	  every	  30	  sec.
    30-­‐min Forecast
2GBBig Data Examples
Rates and Volumes are extremely immense	
Social NW	

Social Simulation	

••Facebook–––Applications

Future “Extreme Big Data”〜1 billion users
Average 130 friends
30 billion pieces of content
shared per month–Target Area: Planet–7 billion people(Open Street Map)

• NOT mining Tbytes Silo Data•
Input Data•
Twitter– Road Network for Planet:• Peta~Zetabytes of Data
300GB (XML)– 500 million active users•
UltraHigh-BWData
Stream– Trip data for 7 billion people– 340 million tweets per day
10KB (1trip) x 7 billion = 70TB
•Internet• Highly Unstructured, Irregular– Real-Time Streaming Data
– 300 million new websites per year
(e.g., Social sensor, physical data)
Complexcorrelations
between •datafrommultiple sources– • 48 hoursof video to YouTube
per minute
Simulated Output for 1 Iteration– • 30,000Extreme
YouTube videos
played per second– 700TBAll Required
Capacity,Bandwidth, Compute
Genomics	
Weather	      Sequencing)data)(bp)/$))becomes)x4000)per)5)years)
c.f.,)HPC)x33)in)5)years

Phased	  Array	  Radar
1GB/30sec/2	   radars

Himawari
500MB/2.5min

A-­‐1.	  Quality	  ControlA-­‐2.	  Data	  Processing

B-­‐1.	  Quality	  Control
B-­‐2.	  Data	  Processing①30-­‐sec	  Ensemble	  Forecast	  Simulations
2	  PFLOP②Ensemble
Data	  Assimilation
2	  PFLOP

Analysis	  Data
2GBEnsemble	  Forecasts
シミ
ュレーショ
ン
シミ
ュレーショ
200GB ン
データ
データ

Ensemble	  Analyses
シミ
ュレーショ
ン
シミ
ュレーショ
200GB ン
データ
データ

③30-­‐minForecast	  Simulation
1.2	  PFLOPRepeat	  every	  30	  sec.
    30-­‐min Forecast
2GBGraph500	  “Big	  Data”	  Benchmark	
Kronecker graph BSP Problem

A: 0.57, B: 0.19
C: 0.19, D: 0.05

November	  15,	  2010	  Graph	  500	  Takes	  Aim	  at	  a	  New	  Kind	  of	  HPC	  Richard	  Murphy	  (Sandia	  NL	  =>	  Micron)	  “	  I	  expect	  that	  this	  ranking	  may	  at	  Imes	  look	  very	  diﬀerent	  from	  the	  TOP500	  list.	  Cloud	  architectures	  will	  almost	  certainly	  dominate	  a	  major	  chunk	  of	  part	  of	  the	  list.”	  The 8th Graph500 List (June2014): K Computer #1, TSUBAME2 #12
Koji Ueno, Tokyo Institute of Technology/RIKEN AICSRIKEN Advanced Institute for Computational
Science (AICS)’s K computer

Reality: Top500 Supercomputers Dominate
No Cloud IDCs at allis ranked

No.1

on the Graph500 Ranking of Supercomputers with17977.1 GE/s on Scale 40
on the 8th Graph500 list published at the International
Supercomputing Conference, June 22, 2014.

Congratulations from the Graph500 Executive Committee#1 K Computer	Global Scientific Information and Computing Center, Tokyo Instituteof Technology’s TSUBAME 2.5
is ranked

No.12
on the Graph500 Ranking of Supercomputers with1280.43 GE/s on Scale 36
on the 8th Graph500 list published at the International
Supercomputing Conference, June 22, 2014.
Congratulations from the Graph500 Executive Committee#12 TSUBAME2	

GPU-‐‑‒based  Heterogeneous  Supercomputer:  TSUABME2  (TokyoTech)

17PF(SFP)/5.7PF(DFP)  #13  on  Top500  (Jun  2014)
System  1408  nodes,  44  racks
CPU:  Intel  Xeon  X5670  (6  cores)  ×  3  GPU:  NVIDIA  Tesla  K20X  ×  3
MEM:  58  GB  DDR3  1333MHz  SSD:  60  GB  ×  2  Network  4×  QDR  Full  bisection  Optical  InﬁnibandStorage

SSD:  200TB
HDD:  7PB  (Lustre,  GPFS)
Tape:  4PB  (SorageTek  SL8500×  2)

TSUBAME2 System Overview

11PB (7PB HDD, 4PB Tape, 200TB SSD)

CompuIng	  Nodes： 17.1PFlops(SFP),	  5.76PFlops(DFP),	  224.69TFlops(CPU),	  ~100TB	  MEM,	  ~200TB	  SSD	  Thin	  nodes	 1408nodes	  	  	  (32nodes	  x44	  Racks)	  	  	  Medium	  nodes	

HP	  Proliant	  SL390s	  G7	  1408nodes
CPU:	  Intel	  Westmere-­‐EP	  	  2.93GHz	  	  	  	  	  	  	  	  	  	  	  6cores	  ×	  2	  =	  12cores/node	  GPU:	  NVIDIA	  Tesla	  K20X,	  3GPUs/node	  Mem:	  54GB	  (96GB)	  SSD:	  	  60GB	  x	  2	  =	  120GB	  (120GB	  x	  2	  =	  240GB) 	  ・・・・・・	

Fat	  nodes	

HP	  Proliant	  DL580	  G7	  24nodes	  	  CPU:	  Intel	  Nehalem-­‐EX	  2.0GHz	  	  	  	  	  	  	  	  	  	  8cores	  ×	  2	  =	  32cores/node	  GPU:	  NVIDIA	  	  Tesla	  S1070,	  	  	  	  	  	  	  	  	  	  	  	  NextIO	  vCORE	  Express	  2070	  Mem:128GB	  SSD:	  120GB	  x	  4	  =	  480GB	  		  	  Local SSDs	

HP	  Proliant	  DL580	  G7	  10nodes	  CPU:	  Intel	  Nehalem-­‐EX	  2.0GHz	  	  	  	  	  	  	  	  	  	  8cores	  ×	  2	  =	  32cores/node	  	  GPU:	  NVIDIA	  Tesla	  S1070	  Mem:	  256GB	  (512GB)	  SSD:	  120GB	  x	  4	  =	  480GB	  

	  
	  

Interconnets: Full-­‐bisecIon	  OpIcal	  QDR	  Inﬁniband	  Network	Core	  Switch	  Edge	  Switch	  	  	  Edge	  Switch	  (/w	  10GbE	  ports)	  179switches	  12switches	  	  	  Voltaire	  Grid	  Director	  4036	  ×179	  IB	  QDR	  :	  36	  ports	  Voltaire	  Grid	  Director	  4700	  	  ×12	  IB	  QDR:	  324	  ports	  Inﬁniband	  QDR	  Networks	

6switches	  	  	  Voltaire	  	  Grid	  Director	  4036E	  ×6	  IB	  QDR:34ports	  	  	  10GbE:	  	  2port	  QDR	  IB(×4)	  ×	  20	

QDR	  IB	  (×4)	  ×	  8	GPFS#1	

SFA10k	  #1	

SFA10k	  #2	GPFS+Tape	/data0	  	/data1	  	“Global	  Work	  Space”	  #1	2.4	  PB	  HDD	  +	  	  SFA10k	  #3	

Lustre	/work0	“Global	  Work	  Space”	  #2	SFA10k	  #5	

SFA10k	  #4	/work1	  	  	  	  	  /gscr	

“Global	  Work	  Space”	  #3	

3.6	  PB	Parallel	  File	  System	  Volumes	

GPFS#2	

GPFS#3	10GbE	  ×	  2	GPFS#4	

HOME	

HOME	

Home	System	  applicaQon	

SFA10k	  #6	“cNFS/Clusterd	  Samba	  w/	  GPFS”	  	  iSCSI	“NFS/CIFS/iSCSI	  by	  BlueARC”	  	  Home	  Volumes	

1.2PB	

TSUBAME2 System Overview

11PB (7PB HDD, 4PB Tape, 200TB SSD)

CompuIng	  Nodes： 17.1PFlops(SFP),	  5.76PFlops(DFP),	  224.69TFlops(CPU),	  ~100TB	  MEM,	  ~200TB	  SSD	  Thin	  nodes	 1408nodes	  	  	  (32nodes	  x44	  Racks)	  	  	  Medium	  nodes	

HP	  Proliant	  SL390s	  G7	  1408nodes
CPU:	  Intel	  Westmere-­‐EP	  	  2.93GHz	  	  	  	  	  	  	  	  	  	  	  6cores	  ×	  2	  =	  12cores/node	  GPU:	  NVIDIA	  Tesla	  K20X,	  3GPUs/node	  Mem:	  54GB	  (96GB)	  SSD:	  	  60GB	  x	  2	  =	  120GB	  (120GB	  x	  2	  =	  240GB) 	  ・・・・・・	

Fat	  nodes	

HP	  Proliant	  DL580	  G7	  24nodes	  	  CPU:	  Intel	  Nehalem-­‐EX	  2.0GHz	  	  	  	  	  	  	  	  	  	  8cores	  ×	  2	  =	  32cores/node	  GPU:	  NVIDIA	  	  Tesla	  S1070,	  	  	  	  	  	  	  	  	  	  	  	  NextIO	  vCORE	  Express	  2070	  Mem:128GB	  SSD:	  120GB	  x	  4	  =	  480GB	  		  	  HP	  Proliant	  DL580	  G7	  10nodes	  CPU:	  Intel	  Nehalem-­‐EX	  2.0GHz	  	  	  	  	  	  	  	  	  	  8cores	  ×	  2	  =	  32cores/node	  	  GPU:	  NVIDIA	  Tesla	  S1070	  Mem:	  256GB	  (512GB)	  SSD:	  120GB	  x	  4	  =	  480GB	  	  	  Fine-­‐grained	  R/W	  I/O	  (check	  point,	  temporal	  ﬁles)	

Local SSDs	

Interconnets: Full-­‐bisecIon	  OpIcal	  QDR	  Inﬁniband	  Network	Core	  Switch	  Edge	  Switch	  	  	  Edge	  Switch	  (/w	  10GbE	  ports)	  179switches	  12switches	  	  	  Voltaire	  Grid	  Director	  4036	  ×179	  IB	  QDR	  :	  36	  ports	  Voltaire	  Grid	  Director	  4700	  	  ×12	  IB	  QDR:	  324	  ports	  Inﬁniband	  QDR	  Networks	

Read	  mostly	  I/O	  	  QDR	  IB(×4)	  ×	  20	
(data-­‐intensive	  apps,	  parallel	  workﬂow,	  parameter	  survey)	
SFA10k	  #1	

SFA10k	  #2	GPFS+Tape	/data0	  	/data1	  	“Global	  Work	  Space”	  #1	Backup	2.4	  PB	  HDD	  +	  	  SFA10k	  #3		  Voltaire	  	  Grid	  Director	  4036E	  ×6	  IB	  QDR:34ports	  	  	  10GbE:	  	  2port	  QDR	  IB	  (×4)	  ×	  8	GPFS#1	

SFA10k	  #5	

SFA10k	  #4	

Lustre	/work0	6switches	  	  /work1	  	  	  	  	  /gscr	

Fine-­‐grained	  
R/W	  I/O	  “Global	  Work	  Space”	  #p2	“Global	  ﬁWles)	
ork	  Space”	  #3	
(check	  oint,	  temporal	  3.6	  PB	Parallel	  File	  System	  Volumes	

GPFS#2	

GPFS#3	

10GbE	  ×	  2	GPFS#4	

HOME	

HOME	

Home	System	  applicaQon	SFA10k	  #6	iSCSI	•	  Home	  storage	  for	  compuQng	  nodes	  “NFS/CIFS/iSCSI	  by	  BlueARC”	  	  •	  Cloud-­‐based	  campus	  storage	  services	“cNFS/Clusterd	  Samba	  w/	  GPFS”	  	  Home	  Volumes	

1.2PB	

TSUBAME2 Storage Usage Since Nov. 2010

TSUBAME2.5  as  a  BigData  Infrastructure• GPU-‐‑‒based  Many-‐‑‒core  Accelerator– NVIDIA  Tesla  K20X  4224  cards• Huge  Memory  Volumes  per  Node
– 58GB  per  node  ×  1408  nodes• Fat  Tree  Dual-‐‑‒Rail  QDR  InﬁniBand– 200Tbps  of  full  bisection  bandwidth• Local  SSD  devices  per  node
– 200TB  in  total• Large-‐‑‒scale  Storage  Systems– Lustre,  GPFS
– 7PB  of  HDDs,  4PB  of  Tapes

TSUBAME2.5  as  a  BigData  Infrastructure• GPU-‐‑‒based  Many-‐‑‒core  Accelerator– NVIDIA  Tesla  K20X  4224  cards• Huge  Memory  Volumes  per  Node
– 58GB  per  node  ×  1408  nodes• Fat  Tree  Dual-‐‑‒Rail  QDR  InﬁniBand– 200Tbps  of  full  bisection  bandwidth• Local  SSD  devices  per  node
– 200TB  in  total• Large-‐‑‒scale  Storage  Systems– Lustre,  GPFS
– 7PB  of  HDDs,  4PB  of  Tapes

Supercomputer Tokyo Tech.
Tsubame 2.0#4 Top500 (2010)

A Major Northern Japanese
Cloud Datacenter (2013)	
the	  Internet	

Advanced Silicon
Photonics 40G
single CMOS Die
1490nm DFB
100km Fiber

10GbE	>>

10GbE	
Juniper	  MX480	

Juniper	  MX480	

2	  zone	  switches	  (Virtual	  Chassis)	

Juniper	  EX8208	

Juniper	  EX8208	

x1000!	

~1500 nodes compute & storage
Full Bisection Multi-Rail
Optical Network
Injection 80GBps/Node
Bisection 220Terabps	

10GbE	

10GbE	LACP	

Juniper	  EX4200	

Juniper	  EX4200	

Zone	  (700	  nodes)	

Juniper	  EX4200	

Juniper	  EX4200	

Zone	  (700	  nodes)	

Juniper	  EX4200	

Juniper	  EX4200	

Zone	  (700	  nodes)	

8 zones, Total 5600 nodes,
Injection 1GBps/Node
Bisection 160Gigabps	

Towards Extreme-scale
Supercomputers and BigData Machines	• Computation– Increase in Parallelism, Heterogeneity, Density• Multi-core, Many-core processors• Heterogeneous processors• Hierarchial Memory/Storage Architecture– NVM (Non-Volatile Memory),
SCM (Storage Class Memory)• FLASH,  PCM,  STT-‐‑‒MRAM,  ReRAM,  HMC,  etc.

– Next-gen HDDs (SMR),Tapes (LTFS)
Problems		  Algorithm

Scalability
Heterogeneity

PowerFTNetwork

Productivity

Locality
Storage Hierarchy
I/OExtreme Big Data (EBD)
Next Generation Big Data
Infrastructure Technologies Towards
Yottabyte/Year 	Principal Investigator
Satoshi Matsuoka

Global Scientific Information andComputing CenterTokyo Institute of Technology	
2014/11/05 JST CREST Big Data SymposiumAleksandr Drozd, Naoya Maruyama, Satoshi Matsuoka A(TITECH)
Multi GPU Read Alignment Alg

In most living otheir developme
molecule called
DNA consists ocalled nucleotid
The four bases
A), cytosine (CIntroductionEBE Research Scheme

Problem Domain

Future Non-Silo Extreme Big Data Apps	
Large Scale
Metagenomics

Co-Design	
EBD Bag

Ultra Large Scale
Graphs and Social
Infrastructures

Massive Sensors andData Assimilation in
Weather Prediction

Co-Design	 Co-Design	
日本地図

EBD System Softwareincl. EBD Object System

GraphStore

CartesianPlane KV
S

KV
S
NVM/
NVM/
2Tbps HBM
NVM/
Flash	
Flash	
4~6HBM Channels NVM/
NVM/
NVM/
Flash	
Flash	
1.5TB/s DRAM & Flash	Flash	DRAM	DRAM	NVM BW
DRAM	
DRAM	
DRAM	
DRAM	
Low
Low
High Powered
30PB/sI/O BW Possible
Main CPU	PowerPower
1Yottabyte/Year
CPU	
CPU	
TSV Interposer	

Exascale Big Data HPC

13/06/06 22:36

KVS

EBD KVS

1000kmfile:///Users/shirahata/Pictures/日本地図.svg

1/1 ページ

PCB	Convergent Architecture (Phases 1~4)
Large Capacity NVM, High-Bisection NW

Cloud IDC
Very low BW & Efficiencty

Supercomputers
Compute&Batch-Oriented	

Extreme Big Data (EBD) Team
Co-Design EHPC and EBD Apps	
• Satoshi Matsuoka (PI), Toshio• Yutaka Akiyama, Ken
Endo, Hitoshi Sato (TokyoKurokawa (Tokyo Tech, 5-1)
Tech.) (Tasks 1, 3, 4, 6)• Osamu Tatebe (Univ.
Tsukuba) (Tasks 2, 3)• Toyotaro Suzumura(IBM Lab, 5-2)

• Michihiro Koibuchi (NII)(Tasks 1, 2)• Takemasa Miyoshi (RikenAICS, 5-3)