27pA07
京における局所的ジャイロ運動論的シミュレーションコードGKVの並列化
Parallelization of the local gyrokinetic simulation code GKV on the K computer
前山伸也1，渡邉智彦2，井戸村泰宏1，仲田資季1，沼波政倫2，石澤明宏2
S. Maeyama1, T.-H. Watanabe2, Y. Idomura1, M. Nakata1, M. Nunami2, and A. Ishizawa2
原子力機構，2核融合研
1
JAEA, 2NIFSOne of the most important issues in magneticallyconfined plasma researches is turbulent transport.
Numerical simulations based on the gyrokinetic
theory have strongly promoted the understanding ofmicro-instabilities and associated plasma turbulence.
Most of gyrokinetic simulations have assumed scale
separations between ions and electrons to reduce
computational costs. Ion-scale and electron-scale
turbulence, however, are able to nonlinearly interact
via mesoscale structures. Therefore, plasma
turbulence is intrinsically multi-scale physics,
which includes temporal and spatial scales ofelectrons and ions simultaneously. Since
multi-scale turbulence simulations require hightemporal and perpendicular spatial resolutions, their
computational costs are estimated to be (mi/me)3/2
times higher than that of ion-scale turbulence. To
deal with this numerically challenging problem,
highly-scalable simulations on peta-scale supercomputers, such as the K computer, are necessary.
Our proposal for the general use of the K computerhas been accepted, and we have been developing
numerical schemes and parallelization methods for
multi-scale turbulence simulations.
We have extended the gyrokinetic Vlasov
simulation code GKV [Watanabe,NF2006] to run
efficiently on the K computer, where a key
numerical technique is the speed up of the parallel
two-dimensional (2D) Fast Fourier Transform(FFT)
algorithms.
MPI/OpenMP
hybridparallelization with 4D domain decomposition isnewly implemented to use a large number of cores.
Perpendicular dynamics is solved by using thespectral method with 2D FFTs, where the datatranspose often degrades scalability. The MPI
process mapping is optimized for the 3D torus
inter-node network on the K computer, so that thetranspose communications are performed among
the neighboring nodes located in a 3D box shape.
This maximizes a bi-section bandwidth available on
the torus network, and reduces communication
costs of data transpose. Additionally, overlaps of200
Performance [TFlops]

1w/o overlap
w overlap
150

10050004080120160

Number of cores [×103]

Fig. Strong scaling test of GKV on the K computer[where 1024 × 1024 × 64 × 64 × 32 grid points areemployed in five-dimensional phase space (x, y, z,v||, μ) and parallelized into 8 (and 16, 32, 64) × 8 ×
8 × 4 subdomains and 8 threads]. Square andtriangle dots plot the cases without/with overlaps on
parallel spectral calculations, respectively.
computations and communications are implementedby using OpenMP directives. Applying thedeveloped overlap method to parallel 2D FFTs, wemask communication costs of data transpose withFFTs and computations in the spectral method. TheFigure demonstrates that the overlaps of transposeand FFTs significantly improve strong scaling. Thecase with the overlaps achieves 168 TFlops (8.03%of the theoretical peak performance) at 131,072
cores, which is 29% higher performance than that in
the case without overlaps. Thanks to the developed
methods, GKV achieves almost linear speed-up
beyond 100k cores with high parallel efficiency~99.9998%.
We note that the newly developed overlap
methods of FFTs and transpose communications areapplicable not only for the kinetic simulations but
also for fluid simulations, if one finds independent
multi-dimensional FFTs.