30E20P
ペタスケールジャイロ運動論シミュレーションのための通信マスク手法開発	 
Development of communication overlap techniques for Peta-scale gyrokineticsimulations
井戸村	  泰宏1, 仲田	  資季1, 山田	  進1, 町田	  昌彦1, 今村	  俊幸2Y. Idomura1, M. Nakata1, S. Yamada1, M. Machida1, T. Imamura2
日本原子力研究開発機構1, 理化学研究所2Japan Atomic Energy Agency1, Riken2

In order to fit the code structure to hardware
architectures, multi-layer hybrid parallelization
models with multi-dimensional domain
decomposition are designed following thephysical symmetry properties of each operator
in basic equations. In this model, all thecommunications within and between thesolvers (operators) are implemented on a single
layer of the hierarchical network consisting ofmultiple MPI layers and a SMP layer. Thisavoids collective communications among all
the MPI processes, makes extensive use of thebi-section bandwidth, and suppresses the sizeof each MPI communicator below ∼100 up to∼106 MPI processes. In addition to this global
network design, novel communication overlap
techniques are developed to achieve theextreme concurrency. The techniques use either
non-blocking communications with additional
control sequences or communication threads

with heterogeneous OpenMP implementations.
Our communication overlap techniques arebased on conventional MPI libraries, and workon most of existing platforms based on
dedicated and commodity networks.
Performances of newly developed techniques
are tested on two Peta-scale platforms, theK-computer at Riken (K), and the Helios at theIFERC (Helios). A Torus network on K
naturally fit to the domain decomposition usedin GT5D, and therefore, basic communication
performances and their scaling against the
number of nodes are better than Helios. By
applyingthecommunication
overlaptechniques,communication
costsaredramatically reduced both on K and on Helios,
and the strong scaling on K achieved theparallel efficiency of ~99.9998% (See Fig.).
The parallel computing techniques accelarate
GT5D significantly, and enable us to accessnew physics regimes both in plasma sizes andin time scales.
Sustained*Performance*(TFlops)

Gyrokinetic simulations are essential andpowerful tools in studying turbulent transport
in fusion plasmas. In order to enhance thecapabilities of gyrokinetic simulations, furtherimprovements both in simulation models and incomputing power are needed. In particular, theplasma size scaling of turbulent transport is oneof critical issues, which require extreme scalecomputing. However, it is not so easy to extract
high computing power from modern Peta-scale
supercomputers because of 1) very complicatedhierarchy of core, CPU, node, and networkarchitectures, 2) extreme concurrency beyond
100k cores. In order to satisfy these technicalneeds, novel parallel computing techniques aredeveloped on a gyrokinetic toroidal 5D
Eulerian code GT5D [Idomura et al., Comput.
Phys. Commun. 2008; Nucl. Fusion 2009].

350"K(ITER)"300"

Helios(ITER)"250"

BX900(JT60)"

200"
150"
100"
50"
0"
0""

500""

1000"" 1500"" 2000"" 2500"" 3000"" 3500""
Peak*Performance*(TFlops)

Fig. The strong scaling of GT5D on BX900, Helios, and K.
Helios and K use ITER size parameters (NR, Nz, NZ, Nv//,Nm) = (768, 64, 768, 128, 32). BX900 uses JT-60U sizeparameters (NR, Nz, NZ, Nv//, Nm) = (240, 64, 240, 128, 32).