次世代MD専用計算機	  MDGRAPE-­‐4開発と今後の展開	
泰地 真弘人	  	  理化学研究所	  	  	  	  	  生命システム研究センター 副センター長	  	  	  	  	  主任研究員	  	  taiji@riken.jp	  Challenges	  in	  Molecular	  Dynamics	  simula@ons	  of	  biomolecules	
Times
Cell

ms

Large
Structural
Changesμs

Local
Structural
Changesns
psfs

K computer

Molec
ular
Vibrati
onnm

μmmm

Sizen Timestep	  	  〜fsec	  n Target	  @mescale	  	  μsec〜sec	  	  To	  cover	  109〜1015	  @mescale	  diﬀerence	  Scaling	  challenges	  in	  MD	n 〜50,000	  FLOP/par@cle/step	  n Typical	  system	  size	  :	  N=105	  n 5	  GFLOP/step	  n 5TFLOPS	  eﬀec@ve	  performance	  	  1msec/step	  =	  170nsec/day	  	  Rather	  Easy	  n 5PFLOPS	  eﬀec@ve	  performance	  	  1μsec/step	  =	  200μsec/day???	  	  Diﬃcult,	  but	  important	  	  	Scaling of MD on K Computer
Strong scaling〜50 atoms/core~3M atoms/Pflops

1,674,828
atoms	

4

What	  is	  GRAPE?	  
n GRAvity	  PipE	  n Special-­‐purpose	  accelerator	  for	  classical	  par@cle	  simula@ons	  ▷Astrophysical	  N-­‐body	  simula@ons	  ▷Molecular	  Dynamics	  Simula@ons	  	  J. Makino & M. Taiji, Scientific Simulations with Special-Purpose Computers,
John Wiley & Sons, 1997.

History	  of	  GRAPE	  computers	  Eight Gordon Bell Prizes`95, `96, `99, `00 (double), `01, `03, `06

GRAPE	  as	  Accelerator	  n Accelerator	  to	  calculate	  forces	  	  by	  dedicated	  pipelines	  	  Par>cle	  Data	  Host	  Computer	  GRAPE	  Results	  Most	  of	  Calcula>on 	  →	  GRAPE	  Others	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  →	  Host	  computer	  •Communica>on	  =	  O	  (N)	  <<	  Calcula>on	  =	  O	  (N2)	  •Easy	  to	  build,	  Easy	  to	  use	  •Cost	  Eﬀec>ve	  GRAPE	  in	  1990s	
n GRAPE-­‐4(1995):	  The	  ﬁrst	  Teraﬂops	  machine	
Host CPU~ 0.6 Gflops
Accelerator PU~ 0.6 Gflops
Host:
Single or SMP

GRAPE	  in	  2000s	  n MDGRAPE-­‐3:	  The	  ﬁrst	  petaﬂops	  machine	
Host	  CPU	  	  ~	  20	  Gﬂops	  	  Accelerator	  PU	  	  ~	  200	  Gﬂops	  	  Host:	  Cluster	  Scientists not shown but exist	

Sustained	  Performance	  of	  Parallel	  System	  (MDGRAPE-­‐3)	  n Gordon	  Bell	  2006	  Honorable	  Men@on,	  Peak	  Performance	  n Amyloid	  forming	  process	  of	  Yeast	  Sup	  35	  pep@des	  n Systems	  with	  17	  million	  atoms	  n Cutoﬀ	  simula@ons	  (Rcut	  =	  45	  Å)	  n 0.55sec/step	  n Sustained	  performance:	  	  	  185	  Tﬂops	  n Eﬃciency	  ~	  45	  %	  n ~million	  atoms	  necessary	  	  for	  petascale	  Problem	  in	  Heterogeneous	  System	  -­‐	  GRAPE/GPUs	n In	  small	  system	  ▷Good	  accelera@on,	  High	  performance/cost	  n In	  massively-­‐parallel	  system	  ▷Scaling	  is	  olen	  limited	  by	  host-­‐host	  network,	  host-­‐accelerator	  interface	Typical Accelerator System	
Accelerator

System-on-Chip

Accelerator

Low-Bandwidth
Host
Computer

SoC-based System	

HostComputer

Host Network

Accelerator

Accelerator

Generalpurpose core

Generalpurpose core

Embeddedmemories

Embeddedmemories

High-Latency

High-Bandwidth
Low-Latency

Host
Computer
Low-Bandwidth
High-Latency

Network
Low-Latency

Anton	n D.	  E.	  Shaw	  Research	  n Special-­‐purpose	  pipeline	  	  +	  General-­‐purpose	  CPU	  core	  	  	  +	  Specialized	  network	  n Anton	  showed	  the	  importance	  of	  the	  op@miza@on	  in	  communica@on	  system	

R. O. Dror et al., Proc. Supercomputing 2009, in USB memory.	

MDGRAPE-­‐4	n Special-­‐purpose	  computer	  for	  MD	  simula@on	  n Test	  planorm	  for	  special-­‐purpose	  machine	  n Target	  performance	  ▷20μsec/step	  for	  100K	  atom	  system	  ▷8.6μsec/day	  (2fsec/step)	  n Target	  applica@on	  :	  GROMACS	  n Comple@on:	  	  1Q	  2014	  n Enhancement	  from	  MDGRAPE-­‐3	  ▷130nm	  →	  40nm	  process	  ▷SoC	  Integra,on	  of	  Network	  /	  CPU	  ▷Keep	  system	  as	  simple	  as	  possible	  MDGRAPE-­‐4	  System	MDGRAPE-4
SoC
12 lane
6Gbps
Electric= 7.2GB/s(after 8B10Bencoding)
48 OpticalFibers	

12 lane
6Gbps
Optical	

Total 512 chips(8x8x8)	

Node(2U Box)Total 64 Nodes(4x4x4)=4 pedestals	

MDGRAPE-­‐4	  Node	MDGRAPE-­‐4	  System-­‐on-­‐Chip	n 40	  nm	  (Hitachi),	  ~	  230mm2	  n 64	  force	  calcula@on	  pipelines	  	  @	  0.8GHz	  n 64	  general-­‐purpose	  processors	  	  Tensilica	  Extensa	  LX4	  	  @0.6GHz	  n 72	  lane	  SERDES	  @6GHz	  SoC	  Block	  Diagram	
Instruction
Memory (CGP)
Pipeline Blocks

Control GP

Instruction
Memory (1)GP Blocks

8 Pipelines
Core

8 Pipelines

CoreInstruction
Memory (2)

IMem

IMem
DMem

CoreIMem
DMem

DMem
Core

Message
Queue

IMem
DMem

Bus Arbiter/DMAC

8 Pipelines

CoreIMemDMem

Bus Arbiter/DMAC

Global Memory

Network Unitxyz

6Gbps x 12 x 6

FPGA IF
100MHz x 128

SoC	  Physical	  Image	

FPGA Interface	

Pipelines	
GlobalMemory
GP
Cores	

ControlGP

GP
Cores	

Pipelines	
Inst.
Mem.

SerDes & Network IFs	

SerDes & Network IFs	

SerDes & Network IFs	

Inst.
Mem.

Embedded	  Global	  Memories	  in	  SoC	n ~1.8MB	  n 4	  Block	  n For	  Each	  Block	  2 Pipeline
Blocks	 Network	2 GPBlocks	

▷128bit	  X	  2	  for	  General-­‐purpose	  core	  ▷192bit	  X	  2	  for	  Pipeline	  ▷64	  bit	  X	  6	  for	  Network	  ▷256bit	  X	  2	  for	  Inter-­‐block	  	  	  	  GM4 Block
460KB	

GM4 Block
460KB	

GM4 Block
460KB	

GM4 Block
460KB	

Paralleliza@on	  of	  Force	  Calcula@on	
n in	  small	  number	  of	  par@cles	  n We	  have	  to	  explore	  many	  ways	  of	  paralleliza@on	  Fi = ∑ f (ri , rj )j

1. Index	  i 	 	 	 	 	 	GRAPE-­‐3	

2. Index	  j 	 	 	 	 	 	MDGRAPE-­‐3	

3. Calcula@ons	  of	  f	 	 	GRAPE-­‐1	  	  i-­‐Paralleliza@on	Fi = ∑ f (ri , rj )jn Parallel	  Calcula@on	  of	  Fi	  n O(N)	  Reads	  →	  O(NNc)	  calc	  →	  O(N)	  writes	  n Use	  of	  Ac@on-­‐Reac@on	  Symmetry:	  	  O(NNc)	  writes	  with	  possible	  conﬂicts.	  	  Most	  of	  GPU	  implementa@on	  avoid	  this.	  
n Nproc	  ≳	  N	  ~	  105-­‐6	  	  Not	  enough!	  	  Broadcast	  Paralleliza@on	  n In	  par@cle	  simula@ons	  	  n Two-­‐body	  forces	  n For	  parallel	  calcula@on	  of	  Fi,	  	  	  we	  can	  use	  the	  same	  	  n Broadcast	  Paralleliza@on	  	  -­‐	  relax	  Bandwidth	  Problem	  	  Pipeline 1

Pipeline 2

Pipeline ij-­‐Paralleliza@on	

j1(k )Fi = ∑ ∑ fij
k j= j 0(k )j1(k )(k )Fn Parallel	  Calcula@on	  of	  	   i = ∑ fij
j= j 0(k )n Reduc@on	  of	  Fi(k)	  is	  necessary	  ▷Fast	  reduc@on	  /	  synchroniza@on	  is	  required	  ▷This	  kind	  of	  paralleliza@on	  will	  be	  important	  	  	  	  Pipeline	  Block	  Diagram	
xjyj
zjxi

YiyiYizi

YiGroup-based
Coulomb Cutoff

log qjlog qi
1/Rc,Coulomb

Coulomb
cutoff function
-x1.5

Sqr

-x0.5

Sqr

logSqr
¡2

mici
cj

¡i

mij

van der Waals
coefficients¡ij

van der Waals
combination rulex0/x1

exp

Yx0/
x1

exp

-x6

Yx0/
x1

exp

Yx0/
x1

exp

Yx0/
x1

exp

Softcoremj
¡j

exp

-x3

Fused calculation of r2
Atom typex0/x1

Group-based
vdW Cutoff

Exclusion atoms

YijYij~28 stages	

Yj
YjYj

Pipeline	  Func@ons	n Nonbond	  forces	  	  n and	  poten@als	  n Gaussian	  charge	  assignment	  &	  back	  interpola@on	  n Sol-­‐core	  n ~50G	  interac@ons/sec/chip	  Pipeline	  speed	n Tenta@ve	  performance	  ▷8x8	  pipelines	  @0.8GHz(worst	  case)	  	  64x0.8=51.2	  G	  interac@ons/sec	  	  512	  chips	  =	  26	  T	  interac@ons/sec	  	  Lcell=Rc=12A,	  Half-­‐shell	  ..2400	  atoms	  	  105	  X	  2400/	  26T	  ~	  9.2	  μsec	  ▷Flops	  count	  ~50	  opera@ons	  /	  pipeline	  	  2.56	  Tﬂops/chip	  General-­‐Purpose	  Core	Core	n Tensilica	  LX	  @	  0.6	  GHz	  n 32bit	  integer	  /	  32bit	  Floa@ng	  n 4KB	  I-­‐cache	  /	  4KB	  D-­‐cache	  n 8KB	  Local	  Memory	  4KB	  Dcache	

8KB	  
D-­‐ram	

Core	  Integer		  	   Queue	
Floa>ng		  ▷DMA	  or	  PIF	  access	  4KB	  Icache	n 8KB	  Local	  Instruc@on	  Memory	  8KB	  I-­‐ram	▷DMA	  read	  from	  512KB	  Instruc@on	  memory	  GP Block	
Instruction
MemoryInst-­‐ruc>on	  DMAC	  Core	

Barrier	  Core	Core	Core	Core	DMAC	
PIF	Core	Core	Core	Queue	  IF	

GlobalMemory	Control
Processor	Solware	  /	  Programming	  Environment	n Por@ng	  GROMACS	  n C	  (C++)Language	  for	  GP	  core	  n Minimal	  library	  support	  ▷No	  external	  memory	  ▷No	  VM	  n Manual	  Load	  to	  local	  instruc@on	  memory	  ▷8kB	  (4kB+4kB	  double	  buﬀer)	  ▷	  for	  performance	Synchroniza@on	n 8-­‐core	  synchroniza@on	  unit	  n Tensilica	  Queue-­‐based	  synchroniza@on	  	  send	  messages	  	  ▷Pipeline	  →	  Control	  GP	  ▷Network	  IF	  →	  Control	  GP	  ▷GP	  Block	  →	  Control	  GP	  n Synchroniza@on	  at	  memory	  	  	  –	  accumula@on	  at	  memory	Power	  Dissipa@on	  (Tenta@ve)	
n Dynamic	  Power	  (Worst)	  <	  40W	  ▷Pipeline 	   	   	   	   	  ▷General-­‐purpose	  core 	  ▷Others 	   	   	   	   	   	  	  ~	  50%	  	  ~	  17%	  	  ~	  33%	  	  	  	  n Sta@c	  (Leakage)	  Power	  	  ▷Typical	  ~5W,	  Worst	  ~30W	  n ~	  50	  Gﬂops/W	  ▷Low	  precision	  ▷Highly-­‐parallel	  opera@on	  at	  modest	  speed	  ▷Energy	  for	  data	  movement	  is	  small	  in	  pipeline	  	  	  MD	  Simula@ons	  enabled	  by	  MDGRAPE-­‐4	Times
ms

MDGRAPE-4
Anton
Cell
Large
Structural
Changesμs

Local
Structural
Changes

nspsfsn Protein	  structure	  predic@on	  ~	  100	  residues	  n Structure	  op@miza@on	  ▷ﬂuctua@ons	  	  ▷muta@ons	

K computer

Molec
ularVibrati
on

nmμm

mm

Size

Enhances biologicalapplications of MD
simulations

Current	  status	  of	  MDGRAPE-­‐4	n Board	  evalua@on	  ▷Force	  calcula@on	  pipelines	  …	  mostly	  conﬁrmed	  ▷General-­‐purpose	  cores	  …	  mostly	  conﬁrmed	  ▷Network	  …	  under	  evalua@on	  n System	  comple@on	  ▷Within	  FY2013	  	Reﬂec@on	Though	  the	  design	  is	  not	  ﬁnished	  yet…	  n Latency	  in	  Memory	  Subsystem	  ▷More	  distribu@on	  inside	  SoC	  n Latency	  in	  Network	  ▷More	  intelligent	  Network	  controller	  n Pipeline	  /	  General-­‐purpose	  balance	  ▷Shil	  for	  general-­‐purpose?	  
▷#	  of	  Control	  GP	  Shared	  Use	  of	  MDGRAPE-­‐4	n 2014	  1Q	  :	  Hardware	  comple@on	  n 2014	  3Q	  :	  Solware	  ready	  n 2014	  3Q	  –	  2016	  1Q	  :	  	  Use	  under	  research	  collabora@on	  agreement	  n 2016	  2Q	  –	  	  Shared	  use	  n ~6	  research	  subject	  /	  year	  ▷3	  QBiC	  /	  Research	  collabora@on	  Future	  Perspec@ves	  	  	For	  Molecular	  Dynamics	  n Single-­‐chip	  system	  ▷>1/10	  of	  the	  MDGRAPE-­‐4	  system	  can	  be	  embedded	  with	  11nm	  process	  ▷For	  typical	  simula@on	  system	  it	  will	  be	  the	  most	  convenient	  ▷S@ll	  network	  is	  necessary	  inside	  SoC	  n For	  further	  performance	  improvement	  ▷#	  of	  opera@ons	  /	  step	  /	  20Katom	  ~	  109	  	  ▷#	  of	  arithme@c	  units	  in	  system	  ~	  106	  /Pﬂops	  	  Exascale	  means	  “Flash”	  (one-­‐path)	  calcula,on	  ▷More	  specializa@on	  is	  required	  What	  happens	  in	  future?	

TOP500	  Average	  Frequency	
CPU Frequency (MHz)	

3,000
2,500
2,000
1,500
1,000
500
-Date	n CPU	  frequency	  became	  constant	  aler	  2005	  n Performance	  increase	  relies	  on	  paralleliza@on	TOP500	  Average	  Power	  Eﬃciency	  	  (MFLOPS/W)	
1,000	  	  MFLOPS/W	Avg	  MF/W	  Top10	  MF/W	  Avg	  RMAX	  1000	  100	  	  100	  10	  	  Linpack GFLOPS	

10000	  Year	n Pro	  :	  Exponen@al	  Growth	  n Con:	  Growth	  speed	  slower	  than	  performance	  growth	Exascale	  Systems	n 1018	  FLOPS	  @	  2GHz	  n 2.5	  X	  108	  FP	  ALUs	  n ~105	  Processors	  →	  103	  ALUs	  /	  Processors	  n Possible	  direc@ons	  ▷SoC	  Integra@on	  ▷BlueGene/L	  type	  machines	  ▷Heterogeneous	  What	  Exaﬂops	  means	n 108	  arithme@c	  units	  in	  parallel	  n Diﬀerent	  from	  TFLOPS	  -­‐>	  PFLOPS	  n Which	  applica@on	  can	  use	  this???	  
▷It	  may	  be	  be}er	  to	  have	  a	  center	  with	  exascale	  performance	  ▷However,	  the	  real	  target	  performance	  for	  applica@ons	  will	  be	  ~10PFLOPS	  ▷Design	  should	  consider	  this	  point	  	  Future	  of	  high	  performance	  compu@ng	n End	  of	  Si	  cycle	  ~	  2020	  n Produc@on	  cost	  of	  semiconductor	  will	  con@nue	  to	  decrease	  for	  a	  while	  n Dedicated	  approach	  /	  semi-­‐dedicated	  approach	  will	  be	  an	  answer	  to	  improve	  performance	  aler	  the	  end	  of	  Si	  cycle	

Acknowledgements	n RIKEN	  Mr.	  I}a	  Ohmura	  Dr.	  Gentaro	  Morimoto	  Dr.	  Yousuke	  Ohno	  Mr.	  Aki	  Hasegawa	n Japan	  IBM	  Service	  Mr.	  Ken	  Namura	  Mr.	  Mitsuru	  Sugimoto	  Mr.	  Masaya	  Mori	  Mr.	  Tsubasa	  Saitoh	  n Hitachi	  Co.	  Ltd.	  	  Mr.	  Iwao	  Yamazaki	  	  	  Mr.	  Tetsuya	  Fukuoka	  	  Mr.	  Makio	  Uchida	  	  Mr.	  Toru	  Kobayashi	  	  and	  many	  other	  staﬀs	  n Hitachi	  JTE	  Co.	  Ltd.	  	  Mr.	  Satoru	  Inazawa	  	  Mr.	  Takeshi	  Ohminato