Implementation of AMR Technique into Parallelized Plasma
Codes and Its Effectiveness
Masanori Nunami 1, Hideaki Miura 1, Akihide Nagara 2, Masaharu Matsumoto 2, Hideyuki Usui 2, and Ryosuke Goto 3
121

National Institute for Fusion Science, 322-6 Oroshi-cho, Toki 509-5292, Japan
Graduate School of System Informatics, Kobe University, 1-1 Rokko-dai Nada-ku, Kobe 657-8501, Japan
3
Graduate University of Advanced Studies, 322-6 Oroshi-cho, Toki 509-5292, Japan

Introduction

To study a complex dynamical behaviour of plasmas, e.g.,space plasmas, laboratory plasmas and fusion plasmas,
numerical simulations with high-resolution in space and time are
prerequisite subjects. They are required not only to understanddetailed physics of small-scale events such as saturations ofshort wave instabilities, micro turbulence and magnetic
reconnections, but also to construct a macro-scale model topredict long-time behaviours of the plasmas because small-scale
events often affect the long-time behaviours through changing
local pressure gradients and/or magnetic configurations. In sucha situation with high-resolution of the calculation grids, we needa huge number of grid points to model the region to guarantee
the numerical stability. In the conventional codes which employ
uniform grid system, due to the recent accelerative gains ofmassive computational powers such as “Kei” computer, it haslearned to calculate the behaviours with a certain resolution
level. Nevertheless, in order to cover whole phenomena fromthe micro-scale events to the macro-scale behaviours of theplasmas, it is not efficient at all in terms of the usage of limited
computer resources such as the amount of memory andcalculation time.
For the purpose of carrying out the numerical simulations
with high resolution and a high efficiency in the possible
calculation time and computer resources, the Adaptive Mesh
Refinement (AMR) algorithm is one of the strongest candidates.
In the AMR simulation, grids with different spacing are
dynamically created in hierarchical structures according to thelocal conditions of phenomena. Fine grids suitable to the local
high density region are applied only there and other regions aresimulated by using moderate size grids. Therefore, increment of
the numerical cost due to the localized region is not serious byadopting the AMR technique. On the other hand, since the AMR
algorithm has to treat hierarchical grid system, the AMR codeshould be programmed with extremely different structure of thecomputation flows and data set from the conventional
simulation codes with uniform grid systems. In this paper, inorder to implement the technique into the conventional plasma
simulation codes, e.g., magneto hydrodynamic (MHD) code,
kinetic plasma code, and so on, we developed the AMR modulethat is based on block-structured AMR framework. The
effectiveness of the parallel computation in several computerarchitectures is also discussed.

2

Block-Structured AMR Framework

AMR techniques were implemented by several schemes, forexample, cell-based refinement structure, patch-based meshes,
block-structured framework, and so on. Furthermore, thetechniques were applied for many scientific fields. Cell-based
scheme is advantageous for saving the computer resources
compared with other schemes, since we can save the refinement

Fig.1 A schematic illustration of the data structure adopted
in the developed AMR module. Arrows indicate thepointers. Three types of pointers, neighbour, parent, andchild, are used to specify the relationship between twoBlocks. For simplicity, a case of two hierarchical levels isdepicted in the top panel. The bottom panel shows thedetail of the Block structure. In each block, self-similar
rectangular grid systems with the size (Nx,Ny,Nz), physicalvariables such as fields, particles, and so on, and some flags
for the refinement procedure are contained.
regions with minimum requirement resources. An AMR code,PARMER [1] is plasma particle code with AMR technique byusing cell-based refinement structure and it has good efficiency
in terms of saving the computer memories [2]. However, thecell-based implementation should uses quite different structureof data and grid systems from the conventional code that uses
adjustable arrays. Therefore, for more good portability ofimplementation of the AMR method, the block-based AMR
framework is more advantageous than the cell-based schemes.
The block-structured AMR method was originally developed byBerger and Oliger [3] for hyperbolic equations. In this approach,self-similar blocks divide whole space of the simulation system.
Since each block has self-similar grid systems with the size
(Nx,Ny,Nz), several conventional simulation codes based on
adjustable array systems are applicable in each block. As shownin Fig. 1, the developed module [4] employs the blockstructured AMR system and fully-tree-treaded (FTT) datastructure [4] with three types of pointers, ‘neighbour’ for thesurrounding grids, ‘parent’ for the original coarse grids in thelower hierarchy, and ‘child’ for the refined grids in the higher
hierarchy. These pointers are included into a structure ofFortran 90, and used to specify the relationship between twoblocks. By using the FTT data structure, it becomes easy tohandle the creation or removal of each hierarchical domain
during a simulation run.

3Parallelization and Effectiveness of the AMR
module in Plasma Simulations

In various plasma simulation studies with particle codes,
hydrodynamic codes, and so on, domain decomposition method
is one of most familiar parallelization schemes. For thedeveloped AMR module, we adopt the domain decomposition
scheme for the parallelization using MPI library. While theAMR module is based on the block-based approach and the selfsimilar refinement, it is designed to be applicable to theparallelization easily, because the communication between theblocks is quite similar to that between the decomposed
calculation domain nodes. Thus, in the module, thecommunications for parallel processing should be executed atboundaries of each block.
To demonstrate the effectiveness of the developed AMR
module in parallelized plasma simulations, we carry out MHD
simulations. The MHD equations are approximated either by theuse of the 4th order central finite difference scheme or by theuse of the 8th order compact finite difference scheme, whichhave been applied for the nonlinear MHD simulations of theLarge Helical Device [6,7]. The suitability of the compact finite
difference scheme to the AMR approach is also discussed. Since
the block structure variable can include particles easily in eachblock domain, the developed module is applicable to plasma
particle simulations. The portability of the module is anotherissue in the development. In this work, we demonstrate theplasma simulations with the AMR module in several computerarchitectures, e.g., HITACHI SR16000, Bull B510 and Cray
XE6.

4

Summary and Discussions

We have been developing an AMR module for parallelized
plasma simulation codes, and demonstrating the effectiveness ofthe module in the several computers. The developed module isdesigned for good portability into conventional plasma
simulation codes and any computer architectures. In this work,
we demonstrated the effectiveness of the module in different
computers and simulation codes, i.e., MHD code and particle
code in plasma physics.
The open difficulty in the development of the AMR moduleis the load balancing of the parallelization between processors,
because the number of spatial grids or particles belonging toeach sub-domain is not always constant due to the AMR
procedure. In each sub-domain, the hierarchical grid layers are
adaptively created or deleted. To achieve the load balancing
among sub-domains distributed to processors, we already obtainthe method based on decomposition by the Morton ordered
space-filling curve [8] in the development of PARMER code.
With the help of the technique of PARMER, we should number
all the blocks with the Morton ordered space-filling curve anddivide the order into the number of the processors in such amanner that each sub-domain has approximately the sameamount of the calculation costs, for example, numbers of blocks,particle calculation loops, and so on.

Acknowledgements
This work was supported in part by JST (Japan Science andTechnology Agency)/CREST (Core Research for Evolutional
Science and Technology), and in part by Japanese Ministry ofEducation, Culture, Sports, Science and Technology, Grant No.
23340182. The computations in the present study wereperformed with Plasma Simulator of NIFS, Helios of IFERCCSC, and KDK of RISH at Kyoto University.
References
[1]

[2]

[3]
[4]

[5]

[6]

[7]

[8]

H. Usui, M. Nunami, T. Moritaka, et al., A Multi-Scale
Electromagnetic Particle Code with Adaptive Mesh, Procedia
Computer Science 4, 2337 (2011).
H. Usui, Y. Yagi, T. Matsui, et al., Dynamic Domain
Decomposition for 3D PIC simulation with Adaptive Mesh
Refinement, 21st International Toki Conference, Toki, Japan, P179 (2011).
M. Berger, J. Oliger, Adaptive Mesh Refinement for Hyperbolic
Partial Differential Equations, J. Comp. Phys. 53, 484 (1984).
H. Miura, M. Nunami, H. Usui, et al., Development of a portable
AMR module applicable to various fluid/particle simulations, 21stInternational Toki Conference, Toki, Japan, P1-77 (2011).
A. M. Khokhlov, Fully Threaded Tree Algorithms for Adaptive
Refinement Fluid Dynamics Simulations, J. Comp. Phys. 143, 519(1998).
H. Miura, N. Nakajima, T. Hayashi et al., Nonlinear Evolution ofMagnetohydrodynamic Instability in LHD, Fusion Science andTechnology 51, 8 (2007).
H. Miura, N. Nakajima, Influences of ballooning modes withmoderate wave number on MHD equilibrium in LHD, Nuclear
Fusion 50, 054006 (2010).
M. S. Warren, J. K. Salmon, Proceedings of Supercomputing,IEEE 12, 21 (1993).