Joint Center for Advanced High Performance Computing

Applications with the Oakforest-PACS
SALMON: Scalable Ab-initio Light-Matter simulator
for Optics and Nanoscience
We are developing a computer code SALMON, Scalable Ab-initio Light-Matter simulator forOptics and Nanoscience (http://salmon-tddft.jp). It is based on first-principles timedependent density functional theory and describes electron dynamics in molecules,
nanostructures, and solids induced by optical electric fields by solving the time-dependent
Kohn-Sham equation in real time and real space. Recently, we have successfully achievedlarge-scale simulations for nano-optics phenomena solving a coupled equation of 3D
Maxwell for light electromagnetic fields and 3D time-dependent Kohn-Sham for lightinduced electron dynamics. It provides an accurate and precise platform of numericalexperiments that will be indispensable in forefront optical sciences.

Parallel Multigrid Methods on the Oakforest-PACS
System with IHK/McKernel
Parallel Multigrid Method• Although the parallel multigrid method is expected to be one of the most powerful
numerical algorithms in the exascale era due to its scalable features, there are manythings to be done.
• Target Application: pGW3D-FVM for 3D Groundwater Flow through HeterogeneousPorous Media– MGCG Solver with IC(0) Smoother, V-Cycle– Sliced ELL Format for Storage of Sparse Matrices• CGA (Coarse Grid Aggregation) [KN IEEE
Cluster 2012]– Switching to coarse-grid solver earlier to avoid
comm. Overhead– Significant overhead for coarse grid solver if
number of MPI proc. is O(104)• hCGA (Hierarchical CGA) [KN IEEE ICPADS
2014]

(a)

Groundwater flow through heterogeneous porous media.
(Left) Distribution of water conductivity; (Right) Streamlines– Hierarchical version of CGA (2-levels)
AM-hCGA: Adaptive Multilevel hCGA
– Number of MPI processes is reduced– Processes are repartitioned in an intermediate • If the number of MPI proc. is O(106-107),
level before the final coarse grid solver on anumber of proc. at the 2nd level of hCGA
single MPI process
could be O(104).
– Significant improvement of performance on– 2-Levels might not be enough for moreFujitsu PRIMEHPC FX10 with 4,096 nodes
processes(65,536 cores) for a problem with 17+B DOF:
– More levels are needed ? -> AM-hCGA
Weak Scaling-1.61x, Strong Scaling 6.27x(Adaptive Multilevel hCGA)(b)

IHK/McKernel(c)Macroscopic system (EM field)

Microscopic system (TDDFT)• Light Weight Multi Kernel OS for HPC by RIKEN [BG IPDPS 2016]– McKernel implements only a small set of performance sensitive system calls and the rest of the OS
services are delegated to Linux.
– Same binary on pure Linux can be used
– Lower Noise, Lower Communication Overhead– to be available on K, Fugaku (Post K)

laser field
I=1010W/cm2
ω=3.38eV

Summary of Preliminary Results• IHK/McKernel provided more efficiency and stablility– Reduction of Noise, Communication Overhead

I (W/cm2)

1091010

1011

1012

(a) Performance of SALMON using full nodes of Oakforest-PACS.
(b) Optical near-field generated in metal-organic framework, IRMOF10.
(c) Intense pulsed light irradiates on silicon nano-sphere of 250nm radius
by coupled Maxwell-TDDFT calculation.

• Improvement is more significant, if the prob. size/proc. is smaller– 4%: Medium, 17%: Small, 22%: Tiny

• Improvement by AM-hCGA = 10% for Tiny Case at 2,048 nodes with IHK/McKernel– Effect of AM-hCGA was not clear without IHK/McKernel, because fluctuation of computation time
was very large.
For more detail, please visit ITC /JCAHPC The University of Tokyo, #1949 .
Joint Center for Advanced HPC | http://jcahpc.jp/