# Presentation: 808553

## Petaflops Special-Purpose Computer for Molecular Dynamics Simulations

- Makoto Taiji
- High-Performance Molecular Simulation Team
- Computational & Experimental Systems Biology Group
- Genomic Sciences Center, RIKEN
- (Next-Generation Supercomputer R&D Center, RIKEN)

## Acknowledgements

- For MDGRAPE-3 Project
- 	Dr. Tetsu Narumi
- 	Dr. Yousuke Ohno
- 	Dr. Atsushi Suenaga
- 	Dr. Noriaki Okimoto
- 	Dr. Noriyuki Futatsugi
- 	Ms. Ryoko Yanai
- Ministry of Education, Culture, Sports, Science & Technology
- Intel Corporation for early processor support
- Japan SGI for system integration

## Brief Introduction of RIKEN(Institute of Physical and Chemical Research)

- Only research institute covers whole range of natural science and technology in Japan
- ~3,000 staffs
- Budget: ~700 million dollars/year
- 7 bioscience centers:
- Genomic Sciences Center
- SNP Research Center
- Plant Science Center
- Center for Allergy and Immunology
- Brain Science Institute
- Center for developmental biology
- BioResource Center
- Next-Generation Supercomputer (10PFLOPS at FY2011)
- Genomic Science Center:
- The most important national center of genome/post-genome research
- National projects
- Protein 3000 Project
- ENU Mouse mutagenesis
- Genome Network Project

## What is GRAPE?

- GRAvity PipE
- Special-purpose accelerator for classical particle simulations
- Astrophysical *N*-body simulations
- Molecular Dynamics Simulations
- MDGRAPE-3 : Petaflops GRAPE for Molecular Dynamics simulations

- J. Makino & M. Taiji*, Scientific Simulations with Special-Purpose Computers,*
- John Wiley & Sons, 1997*.*

## MDGRAPE-3 (aka Protein Explorer)

- Petaflops special-purpose computer for molecular dynamics simulations
- Started at April 2002,
- 	Finished at June 2006
- Part of Protein 3000 project – a project to determine 3,000 protein structures

*M. Taiji et al, Proc. Supercomputing 2003, on CDROM.*
*M. Taiji, Proc. Hot Chips 16, on CDROM (2004).*

- EGFR

- TT RNA Polymerase

**Notes:**
Now, I will explain the main topic, MDGRAPE-3. We first call the system as Protein Explorer by political reason but we return to the name MDGRAPE-3, since the name Protein explorer has already been used the famous software. This is the project to build a petaflops computer system for molecular dynamics simulations. It started in year 2002 and will finish in 2006. For this system, we have developed the MDGRAPE-3 chip, which performs the force calculation, and the most important part of the system. The chip is fabricated in 130 nm process HDL4N by Hitachi. In the worst case condition it operate at 250 MHz, which is equivalent to 165 Gflops. We confirmed the successful operation of the sample at 350 MHz, 230 Gflops.

## Molecular Dynamics Simulations

- Folding of Chignolin,
- 10-residue β-hairpin
- design peptide
- (by Dr. A. Suenaga)

- Force calculation dominates
- computational time
- Require large computational power

**Notes:**
Our LSI is dedicated for molecular dynamics simulations, which is widely used in material science, biophysics, and biochemistry. In molecular dynamics simulations, we decompose molecule into atoms, and calculate classical forces between atoms. Then, orbits of atoms are integrated according to Newton’s law. Since we have to calculate all pairwise interactions between atoms,  most of computational time is spent for long- and middle-range interactions, namely, Coulomb force and van der Waals force. Therefore, if we build the hardware accelerator to calculate these forces, we can  accelerate whole simulations quite much.
This animation shows the simulation of alpha helix formation of peptides. Simulation starts with straight amino acid chain, then we can observe the process to form helix structure. Random brownian motions are due to thermal motion of surrounding water molecules, which are not displayed.

## Accelerator to calculate forces

- Accelerator to calculate forces

- Host
- Computer

- GRAPE

- Most of Calculation	→ GRAPE
- Others                   	→ Host computer

- Particle Data

- Results

- Communication = *O *(N) << Calculation = *O* (N2)
- Easy to build, Easy to use
- Cost Effective

## History of GRAPE computers

- Eight Gordon Bell Prizes
- `95, `96, `99, `00 (double), `01, `03, `06

**Notes:**
This shows the history of GRAPE systems. Red line shows the machines for molecular dynamics. MDM developed in RIKEN at 2000 has 78 Tflops performance and it still stands as the fastest  nominal peak-performance machine. Blue and magenta line denotes the machines for astrophysics, while magenta line is machines with low-precision arithmetics. Simulations with GRAPE systems won seven Gordon Bell prizes in total.

## Why we build special-purpose computers?

- Bottleneck of high-performance computing:
- Parallelization limit / Memory bandwidth
- Power Consumption = Heat Dissipation
- These problems will become more serious in future.
- Special-purpose approach:
- can solve parallelization limit for some applications
- relax power consumption
- ~100 times better cost-performance

**Notes:**
So, what are technical merits and why we build special-purpose computers? As all you know very well, current problems of high-performance LSI, or high-performance computing are memory bandwidth and heat dissipation.  Both factors limit the number of parallelization in a LSI, and situation becomes worse and worse as transistor density increases. On the other hand, by special-purpose approach, we can solve these problems. We can break the parallelization limit due to bandwidth thoroughly, and we can relax heat dissipation problem. As a result, a hundred times better cost-performance can be obtained.

## Broadcast Parallelization

- Molecular Dynamics Case
- Two-body forces
- For parallel calculation of ***F****i*,
- 	we can use the same
- 	- relax Bandwidth Problem

*Pipeline 1*

*Pipeline 2*

*Pipeline i*

**Notes:**
On the other hand, in special-purpose computers, we can use different approach. For example, in molecular dynamics simulations, the machine calculates two body forces for each particles, which are given by this formula. When we calculate forces on different particle – for different suffix I – in parallel, we can use the same coordinates x j. (x j is a coordinate of particles which exert forces on particle i). Thus, all the pipeline can share the same input x j, and x j is broadcasted to all pipelines. We call this architecture as broadcast memory architecture. Of course, this architecture is not valid for all kind of applications, but in some applications like molecular dynamics, it greatly enhance the LSI performance.

## Highly-Parallel Operations in Molecular Dynamics Processors

- For special-purpose computers
- Broadcast Memory Architecture
- Efficient : 720 operations/cycle/chip
- 				 in MDGRAPE-3 chip
- possible to increase according to Moore’s law
- In case of MD:

| MDGRAPE	 | 600 nm | 1 pipeline | 1Gflops |
| --- | --- | --- | --- |
| MDGRAPE-2	 | 250 nm | 4 pipelines | 16Gflops |
| MDGRAPE-3 | 130 nm | 20 pipelines | 180Gflops |

## Power Efficiency of Special-Purpose Computers

- If we compare at the same technology
- Pentium 4 (0.13 m, 3GHz, FSB800) 	...  14W/Gflops
- MDGRAPE-3 chip (0.13m) 	... 0.1W/Gflops
- Why ?
- Highly-parallel at low frequency
- 	MDGRAPE-3: 250MHz, 720-equivalent operations
- 	for example, single-precision multiplier has 3 pipeline stages
- Tuning accuracy
- 	Most of calculations are done in single precision
- Slow I/O
- 	84-bit wide input and output port at 125 MHz (GTL)

## Force Pipeline

- Calculate two-body central forces

- 8 multipliers, 9 adders, and 1 function evaluator
- 	= 33 equivalent operations for Coulomb force calculation
- A. H. Karp, *Scientific Programming*, **1**, pp133–141 (1992)
- Function Evaluator: approximate arbitrary functions by segmented fourth-order polynomials
- Multipliers : floating-point, single precision
- Adders: floating-point, single precision / fixed-point 40 or 80 bit

**Notes:**
This is the block diagram of force calculation pipeline. In the basic mode it calculates the simple equations shown here. the function g is arbitrary smooth function. The pipeline consists of 8 multipliers, 9 adders, and one function evaluator. The function evaluator approximates arbitrary smooth function by the fourth-order segmented polynomials. The pipeline calculates 33 equivalent operations in the case of Coulomb force calculations. All multiplier is single-precision accuracy, but we used different exponent length to avoid overflow. For adders we used floating-point single precision or 40-bit or 80-bit fixed-point format with floating point conversion.

## Block Diagram of MDGRAPE-3 chip

- Memory-in-a-chip Architecture
- 	Memory for 32,768 particles
- The same data is broadcasted to each pipeline

**Notes:**
This is the block diagram of the LSI. It has 20 force-calculation pipelines, which are completely same, and memory unit. Since in molecular dynamics simulations, we do not use so much memory and it is quite calculation intensive, The memory can be very small. Its capacity is 5.6 mega bits, which corresponds for 32,768 particles. As I already explained, the same 160-bits output is fed to all pipelines simultaneously. In addition, it has pipeline/memory control logic and I/O controllers. The chip has independent input and output to connect many chips in serial.

## MDGRAPE-3 chip

- 216 GFLOPS@300MHz
- 180 GFLOPS@250MHz
- 17W at 300 MHz
- Hitachi HDL4N
- 130 nm
- Vcore=+1.2V
- 15.7 mm X 15.7 mm
- 6.1 M random gates
-   + 9 Mbit memory
- 1444 pin FCBGA

**Notes:**
This is the photograph of the die. The size is 15.7mm square. There are 20 pipelines and main memories. Note that there are no communication between pipelines, therefore the layout is quite simple. (The simple floorplan I made with adobe illustrator worked perfectly).

## MDGRAPE-3 Board

- 12 Chips/Board
- 2 boards/2U subrack = 5 Tflops
- Connected to  PCI-X bus
- 	 via LVDS 10Gbit/s interface

**Notes:**
Next, we explain how the chips will be used in the system. Now we are developing a system board with 12 MDGRAPE-3 LSIs. Since the chip has the memory inside, all we need is the FPGA for interface control, MDGRAPE chips, and power supplies. Chips are daisy-chained by 81-bit GTL bus, with 1.27 Gbyte/sec peak performance. The system is connected to the host computer via 4 times 2.5Gbit/sec serial interface using Xilinx RocketIO. Currently we used PCI-X 133MHz add-in card for host I/O. Small PCI-X board with 2 MDGRAPE-3 LSIs on board has also developed. 

## MDGRAPE-3 system

- 4,778 dedicated LSI “MDGRAPE-3 chip”
- 300MHz(216Gflops)  3,890
- 250MHz(180Gflops)     888
- Nominal Peak Performance: 1 Petaflops
- Total 400 boards with 12(some 11) MDGRAPE-3 chips
- Host : Intel Xeon Cluster, 370 cores
- Dual-core Xeon 5150(Woodcrest 2.66GHz)  2way server  x 85 Nodes
- 		provided by Intel Corporation
- Xeon 3.2DGHz 2way server x 15 Nodes
- System Integration: Japan SGI
- Power Consumption : 200kW
- Size :  22 standard 19inch racks
- Cost : 8.6 M$ (including Labor)

## MDGRAPE-3 system

## Sustained Performance ofParallel System

- Gordon Bell 2006 Honorable Mention, Peak Performance
- Amyloid forming process of Yeast Sup 35 peptides
- Systems with 17 million atoms
- Cutoff simulations
- 	(Rcut = 45 Å)
- Nominal peak : 860 Tflops
- Running speed : 370 Tflops
- Sustained performance: 185 Tflops
- Efficiency ~ 45 %

## Applications suitable for broadcast memory architecture

- Multiple calculations using the same data
- Molecular dynamics / Astrophysical *N*-body simulations
- Dynamic programming for genome sequence analysis
- Boundary value problems
- Calculation of dense matrices(incl. Linpack)
- SIMD (vector) processor with broadcast memory architecture
- MACE (MAtrix Computing Engine)
- 		for dense matirix calculation
- 		3.5Gflops/chip, double precision, 180nm
- GRAPE-DR Project (2004-2009)

**Notes:**
Several applications are known suitable for our approach. There are two conditions required. Firstly, the application must require a large number of computation for small number of data. 
Second, the multiple calculations must be performed in parallel using the same data. There are several applications which satisfies these conditions in addition to molecular dynamics and astrophysical many-body simulations. For example, dynamic programming used in Genome sequence analysis is one, and several special-purpose computer has been built using FPGA in US, Israel, and Japan.
The another application is the boundary element method. While the finite-element method is not suitable for special-purpose approach, but finite-element method can be accelerated. The calculation of dense matrices like linpack benchmark can be also accelerated. 

## GRAPE-DR Project

- Greatly Reduced Array of Processor Elements with Data Reduction
- SIMD accelerator with broadcast memory architecture
- Full system: FY2008
- 0.5 TFLOPS / chip (single), 0.25 TFLOPS (double)
- 2 PFLOPS / system
- Prof. Kei Hiraki (U. Tokyo)
- 	Prof. J. Makino (National Astronomical Observatory)
- 	Dr. T. Ebisuzaki (RIKEN)

## SING (SING is not GRAPE) chip

- 512 Processor Elements, 500 MHz
- PE:
- FP Mul/Add
- Integer ALU
- 32-word Register File
- 256-word memory
- 0.5 TFLOPS, ~0.1W/GFLOPS(SP)
- 0.25TFLOPS, ~0.2W/GFLOPS(DP)

*J. Makino et al., *
- http://www.ccs.tsukuba.ac.jp/workshop/sympo-060404/pdf/3-7.pdf *(in japanese)*

## MDGRAPE-4: combination of dedicated and general-purpose units

- SIMD Accelerator with broadcast memory architecture
- 	Problem: too many parallelism
- 	~500/chip, 5M/system - Works with SIMD?
- What is good with dedicated pipelines
- Force calculation ~ 30 operations done by pipelined operations
- 	Systolic computing
- Can decrease parallelism
- VLIW-like (SIMD) processor with chained operation can mimic pipelined operations
- Allows to embed more dedicated units
- 	which can not be fully utilized by SIMD operations

## PE

- L2/
- Local
- Mem.

- L2/
- Local
- Mem.

- 128

- Additional
- Units

- Additional
- Units

- L3

- Additional Units can be:
- 	Lookup table
- 	(for polynomial interpolations or VdW coefficients),
- 	1/x, Function evaluator etc.
- Target: ~ 0.1W/GFLOPS (DP)

- Each PE: Simple in-order processor with L1$

## Summary

- MDGRAPE-3 achieved PetaFLOPS nominal peak for 200 kW
- Dedicated parallel pipelines at modest speed of ~250 MHz results high performance/power
- Generalized GRAPE approaches are being developed