Acceleration of large-scale MD simulation
for biological functions

Jaewoon Jung and Yuji Sugita
Computational Biophysics Research TeamRIKEN Center for Computational ScienceMD simulations for biomolecules
In MD, we solve classical equations ofmotions to examine protein structure,dynamics, and function relationship.

dri pidtmdpi Fi
dt
Equation ofmotion

piri (t  t )  ri (t )  tmpi (t  t )  pi (t )  Fi t
Integration

Nobel Prize in Chemistry 2013 was awarded to M.
Karplus, M. Levitt, and A. Warshel “for the developmentof multiscale models for complex chemical systems”.

Potential energy in MD
O(N)
O(N)
O(N)
O(N2)

Main bottleneck in MD

𝐤Real space, O(CN)

Reciprocal space, O(NlogN)
3

History of Biological MD simulations(1977 to current)

ps

ms

ns

ms

relaxation

vibration

BPTI(~6x102)

Membraneprotein

Cellularenvironment(# of atoms)

size

KNL

Reaction of protein

1977
AQP1
(~105)

HIV-1 capsid(~6.5x107)

Macromolecule
complextime

folding

Secondary structure change
proteins

Mycoplasma
genitalium(~1.0x108)
Chromatin10(~1.0x109)

HP36(~104)

2000

Ribosome(~2x106) 2013~2019

BPTI, Ubiquitin(~2x104)
ANTON(Gordon Bell Prize 2009)

HIV-1 capsid(~6.5x107)
Chromatophorevesicle(~1.4x108)

We need to extend
simulation time &size

GENESIS is highly parallelized MD software forbiomolecular simulations• We have recently developed new MD software, GENESIS(Generalized-Ensemble Simulation System).
– Jaewoon Jung, Takaharu Mori, et al. WIREs CMS, 5, 310-323 (2015)• The development of GENESIS was motivated to for large scale
MD simulation on K computer• We intend to solve the two problems in MD.
– Time-Scale Problem• The enhanced conformational sampling algorithms• Parallelization with a number of “Replicated” MD simulations.

– Size Problem• The domain decomposition methods for nonbonded interactions• Parallelization for a single MD simulation.

• Using Fugaku, we can more extend the time-scale
and target system size.

Parallelization of GENESIS (real-space)
Cell pair

Midpoint cell?

(1,1)

Yes

OpenMP
parallelization
thread18(2,2)

Yesthread2

9 1 2 1011 3 4 12…

…

(1,2)

Yes

thread4(1,3)

Yesthread1…

…

(1,5)

Yes

(1,6)

No

(1,7)

No

(1,8)

Yes

(1,9)

No

(1,10)

Yes

…

…

56 7

13 14 15 16

Unit cell
Subdomain of each MPI

thread2

thread3
thread4

J. Jung, T. Mori, and Y. Sugita, J. Comput. Chem. 35, 1064-1072 (2014).
J. Jung et al., Wiley Interdiscip. Rev. Mol. Sci. 5, 310-323 (2015).
C. Kobayashi et al., J. Comput. Chem. 38, 2193-2206 (2017).

Parallelization of GENESIS (reciprocal-space)
1. More frequent communications
than existing MD programs
2. MPI Alltoall communications
only in one-dimensional space(existing : communications intwo/three-dimensional space)
3. Reduce communicational cost forlarge number of processors by
reducing the number of processes
involved in communications
4. For Fugaku, GENESIS choose thebest FFT scheme automaticallyaccording to the process numbers.

Connection between real- and reciprocalspace parallelization in GENESISGENESIS

Real-space :Volumetric decomposition
Reciprocal-space:
Volumetric decomposition
No communication is needed.

Other MD programs

Real-space :Volumetric decomposition
Reciprocal-space:
Pencil or slab decomposition
Communication is necessary.
J. Jung, C. Kobayashi, T. Imamura, and Y. Sugita, Comp. Phys. Comm.200, 57-65 (2016).

Performance of GENESIS on K computer

The first 100 million atom
simulation of biomolecules

8.5 million atoms

GENESIS shows good parallel
efficiency on K computer, inparticular, for very large
biological systems.

28 million atoms

MD simulation example on K (cytoplasm)

GENESIS performance developments forvarious hardware architectures
GPU non-bonded scheme

KNL non-bonded scheme

J. jung, A. Naruse, C. Kobayashi and Y. Sugita, J. Chem. Theory Comput. 12, 4947-4958 (2016).
J. Jung et al. J. Comput. Chem. 40, 1919-1930 (2019)

Enhanced sampling scheme with replicas
1. From conventional MD without very long time-scale, simulation can betrapped at one of the local energy minimum.
2. Using the exchange of temperatures or other parameters between replicas,
we can sample a wider conformational space than the conventional MD.

Avoid trapping

Y. Sugita and Y. Okamoto, Chem. Phys. Lett. 314, 141-151 (1999).

Enhanced sampling scheme developments
in GENESIS
1. Temperature REMD
2. Surface-tension REMD
3. Generalized REST (replica exchange with solute tempering)
4. Multidimensional replica exchanges (REST/REUS, gREST/REUS and so on)
5. Reaction method with string method
gREST

Surface-tension REMD
Water
Surfacetension

𝐸𝛽𝐸𝛽+𝛽𝛽

𝐸+𝐸

Solvent-Solvent (vv)

Membrane
Solvent

Exchange surface tensions
in NPγT (surface tension constant)ensemble=Protein

Solute-Solvent (uv)
Solute-Solute (uu)LigandT. Mori, J. Jung, and Y. Sugita, J. Chem. Theory Comput. 9, 5620-5640 (2013).
Y. Matsunaga et al. J. Phys. Chem. Lett. 7, 1446-1451 (2016)
M. Kamiya and Y. Sugita, J. Chem. Phys. 149, 072304 (2018)

Sampling Hundred of Binding EventsMovies from arbitrary selected replicas•••GENESIS development on Fugaku
1. Optimization of GENESIS for Fugaku supercomputer: Continuous
effort from Fugaku co-design
1) New non-bonded interaction kernel with increased SIMD width.
2) Optimization of parallelization of Fugaku
FUGAKU Co-design
SystemApplication

Basic Arch. Collaboration
Start-up
DesignInvestigation
Manufacturing
Development

From K to Fugaku
GENESIS (2014)

GENESIS (2020)

x 125 (dt=2.5fs)

2. New functions/libraries
1) New parallel I/O to deal with inputs/outputs for very huge system.
2) Stable integration with increased time step.
3) Enhanced sampling schemes using replicas.

Summary
1. GENESIS is developed for high performance large-scale MD on Kcomputer.
2. With efficient parallelization, GENESIS can perform MD simulationsfor 100 million and 1 billion atom systems.
3. Various enhanced sampling schemes are also developed in GENESIS
to overcome current MD time-scale limitation.
4. With the development of GENESIS on Fugaku, we expect 125 times
performance improvement and more realistic MD simulations forhuge systems.