Effective Performance of Large-Scale MHD simulation forPlanetary Magnetosphere with Massively Parallel Computer
Keiichiro FUKAZAWA1, 2, and Takeshi NANRI1
12Introduction

The global configuration of magnetosphere is very large
while the scale of magnetospheric phenomena is varied as 108order. For example, the aurora is small scale phenomenon
however it is related to the global magnetospheric convection
through the magnetic field lines. The magnetosphere is formedby the interaction between the space plasma from the Sun andthe ionosphere, etc. and the planetary magnetic field. To
describe the behaviour of space plasma, the Vlasov-Maxwell
system equations are used for studies of electron-scale processes.
For larger-scale processes such as the magnetosphere, however,electron-scale processes can be sometimes neglected then theMagnetoHydroDynamic (MHD) equations are used. The MHD
equations are highly nonlinear and are very complex to solve byhand calculations. Thus in order to simulate the magnetosphere,
it is needed to use the supercomputer.
Recently the scalar-type supercomputers are dominant in theworld [1]. Thus it is necessary to optimize the simulation codes
to the scalar-type supercomputers. These supercomputers have alarge number of processors to achieve the high performance. To
use such many processors, simulation codes are required thehigh parallelism. In addition, the execution efficiency becomes
more important because the peak (ideal) performance greatlyincreases. Furthermore, post-processing including transferringand storing the simulation data is a serious problem.
The purpose of the present study is to effectively performthe MHD code for space plasma simulations on scalar-type
massively parallel supercomputer systems toward peta and exa
scale computing in a few years. In this paper some
improvements to our simulation code and sequences to performthe large scale and high resolution simulation on the massively
parallel computer are presented.
2Simulation Model

We use a three-dimensional MHD code, which has beenused for studies on global structures and dynamics of planetary
magnetospheres [2, 3, 4]. The MHD code uses the “ModifiedLeapfrog” method, in which partial difference equations aresolved by the two-step Lax-Wendroff method for one time stepand then by the Leapfrog scheme for 7 time steps and theprocedure is repeated. Thus the method has second-orderaccuracy in both space and time.
As a parallelization technique, the domain decomposition
method for dividing three-dimensional space is adopted. Usuallyin parallel computing on a distributed-memory computer, thedomain decomposition is an easy way to decompose threedimensional Eularian variables. In the case of a threedimensional model, the dimension of domain decomposition canbe chosen as one dimension, two dimensions, or threedimensions [5]. To use large number of cores for peta and exa
scale computing, we need to select the three-dimensional
decomposition method. For example if we calculate the3000×3000×3000 grids, we can use the 109 cores with threedimensional decomposition method while 106 cores with two-dimensional decomposition method and 103 cores with onedimensional decomposition.
Effective use of cache memory is important for obtaining
better performance on a scalar processor. In this study wemeasure the performance of normal three-dimensional
decomposition (Type A) and array replaced type threedimensional decomposition (Type B) to consider the cache hit
efficiency [5]. Here Type A array configuration is f(nx, ny, nz,m) and Type B is f(m, nx, ny, nz) where nx, ny and nz, are thenumber of x, y, and z directions, respectively. m is the variableof MHD (the plasma density, velocity vector, pressure, andmagnetic field vector).
3Performance Measurements

To understand the effect of cache, two supercomputer
systems are used in this study. The Fujitsu PRIMERGY RX200
S6 at Kyushu University is the PC-cluster type system consistsof Intel Xeon processor (Westmere) and the Fujitsu PRIMEHPCFX10 at the University of Tokyo is the successor computer
system of ‘K-computer’ which consists of SPARC64 IXfx
processor. We use the array of 64 MB/core for thecomputational domain and additionally 192 MB/core forworkspaces for computing the MHD equations with theModified Leapfrog method. To minimize the communication
time, we use a buffer array which stores all the boundary datafor inter-core and inter-node communications [5].
3.1

Fujitsu PRIMEGY RX200 S6Figure 1 shows the performance of MHD code with RX200
S6. Using 4704 cores, we achieved the computational peak
performance 17 TFlops and computational efficiency 32% withthree dimensional decomposition Type A. The efficiency ofType B (cache hit considered type) is about 10% less than TypeA (12 TFlops). Thus Type B is not suitable for this system. This
trend is the same as our previous results [5]. Here the differenceof 1% of computational efficiency becomes the difference of
500 GFlops in the performance so that we should add
performance tuning to simulation code using the massively
parallel computer.
20,000

Performance [GFlops]

1

Research Institute for Information Technology, Kyushu University, Japan
International Center for Space Weather Science and Education, Kyushu University, Japan

17,6273D decomposition Type A
3D decomposition Type B

16,00012,000

9,023

12,031

8,000
4,737
3,506

4,000
0

1,495
767
1,009

0 501

6,113

3,158
2,265

1,000

2,000

3,000

4,000

5,000

number of coresFig. 1. Performance of MHD code with RX200 S6.

Fujitsu PRIMEHPC FX10
Figure 2 shows the results of performance measurementswith FX10. In this case we use up to 1024 cores during a trial
run and we obtained 2.5 TFlops (16% of efficiency) in the TypeB case. This is adverse result of RX200 S6. The cache hit type issuitable for FX10. There is about 5% difference ofcomputational efficiency between the Type A and Type B.
From both results, it becomes clear that there are different
optimizations for scalar type computer and the optimization isvery important to perform the large scale computing.

Performance [Gflops3.2

number of coressystem at the University of Tokyo. For the MHD code, weevaluated two types of decomposition methods: regular threedimensional domain decompositions and a cache hit considered
type of three-dimensional decomposition. We found that theregular three-dimensional decomposition methods are suitablefor RX200 S6 and the cache hit considered type of threedimensional decomposition is suitable for FX10. As the results,
we understand the importance of optimization to perform thelarge scale computing.
To run the large scale MHD simulation, there are threeproblems which are data I/O, storage to store the data and postprocessing. Then we introduce the distributed I/O due to theshortage of memory to gather the divided data on each core to
one node. In addition we use the distributed storage system
Gfarm at NICT to store the sum of large scale time evolutiondata. This system has an advantage of data transfer. Finally wehave developed analysis and visualization programs to treat thelarge size of simulation results.
Thanks to these results we can simulate the planetary
magnetosphere with high resolution. For example we cancalculate the Saturn’s magnetosphere with good spatial
resolution and obtained the relation of small scale vortex andpatchy auroral emission [9].
Acknowledgements

Fig.2. Performance of MHD code with FX10.
4Data I/O and Post Processing

We start running the large scale MHD simulation of Saturn’smagnetosphere using the results of performance measurements
then overcome three problems. First problem is about data I/O. In
the simulation we write the large simulation data (about 400 GB
at maximum) with 864 cores. In the simulation code, thedistributed data divided by the number of rank due to the parallel
computing were gathered into one data in one rank and it waswritten as an output data. However, the speed of writing the large
size of data is very slow and the size of gathered data became
larger than the size of memory in one node (i.e. RX200 S6 has64GB memory per node) thus it is impossible to use the gather
technique in I/O part. Thus it is necessary to write the simulation
data in each rank for the massively parallel computing.
Then we have the problem of data storing. The output data
size is 400 GB/sampling time and 300 data are required to see
the time evolution of the magnetosphere thus the total size ofdata becomes over 120 TB. To store these data, the Gfarm [6]storage system operated by the National Institute of Information
and Communications Technology (NICT) [7] is used. TheGfarm is the distributed storage system using the gridcomputing and has the advantage of data mirroring and data
access speed. The Gfarm storage system of NICT consists ofover 50 storage servers at several locations in Japan and its size
reaches almost 3PB currently. The transfer speed of simulation
data from the supercomputer system at the University of Tokyo
to the NICT is 27 MB/s on average. In the exa scale computingera, due to the limitation of I/O bandwidth, the simulation dataon memory will be transferred to the memory on the storage
server directly.
Finally the post processing of distributed data is the problem.
It is impossible to analyse the large size data on one
computational node because there is not enough memory to treat
the data. Thus we develop the programs to analyse and visualize
the data [8]. These programs are based on CUI (Character UserInterface) so that the user cannot control the functions viscerally,
however, the user can use up to the limit of hardware (CPU andmemory), and parallelize programs easily.
5SummaryIn this study we have made performance measurements andtuning of MHD code for the massively parallel scalarsupercomputer which is the PRIMEGY RX200 S6 at theKyushu University and the PRIMEHPC FX10 supercomputerThis work was supported by CREST, the Japan Science andTechnology Agency (JST). The computational resource wasprovided by Information technology center in the University ofTokyo, Research Institute for Information Technology in theKyushu University and the OneSpaceNet in the NICT ScienceCloud. This work was also conducted as a Joint Usage /Research Center for Interdisciplinary Large-Scale Information
Infrastructures in Japan and Advanced Computational ScientificProgram 2011, Research Institute for Information Technology,Kyushu University.
References
[1]
[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

Top500 Supercomputing Sites. (http://www.top500.org/)
Ogino. T, R. J. Walker, M. Ashour-Abdalla, “A global
magnetohydrodynamic simulation of the magnetopause when theinterplanetary magnetic field is northward”, IEEE Trans. Plasma
Sci. vol. 20, 1992, 817-828.
Fukazawa, K., T. Ogino, and R.J. Walker, The Configuration andDynamics of the Jovian Magnetosphere, J. Geophys. Res., 111,A10207, doi:10.1029/2006JA011874, 2006.
Fukazawa, K., S. Ogi, T. Ogino, and R.J. Walker, MagnetosphericConvection at Saturn as a Function of IMF Bz, Geophys. Res. Lett.,34, L01105, doi:10.1029/2006GL028373, 2007.
Fukazawa, K., T. Umeda, T. Miyoshi, N. Terada, Y. Matsumoto,and T. Ogino, Performance measurement of magneto-hydrodynamic code for space plasma on the various scalar typesupercomputer systems, IEEE Transactions on Plasma Science,Vol. 38, No. 9, pp2254, 2010.
Tatebe, O., Y. Morita, S. Matsuoka, N. Soda, H. Sato, Y. Tanaka,S. Sekiguchi, Y. Watase, M. Imori, T. Kobayashi, "Grid Data
Farm for Petascale Data Intensive Computing", Technical Report,
Electrotechnical Laboratory, ETL-TR2001-4, 2001.
Ken T. Murata, S. Watari, T. Nagatsuma, M. Kunitake, H.
Watanabe, K. Yamamoto, Y. Kubota, H.Kato, T. Tsugawa, K.
Ukawa, K. Muranaga, E. Kimura, O. Tatebe, K. Fukazawa and Y.
Murayama, A Science Cloud for Data Intensive Sciences,CODATA Data Science Journal, submitted, 2012.
Three dimensional visualization codes with VRML.
(http://center.stelab.nagoyau.ac.jp/web1/simulation/jst2k/hpf02.html)
Fukazawa, K., T. Ogino, and R. J. Walker (2012), “A
Magnetohydrodynamic Simulation Study of Kronian FieldAligned Currents and Aurora”, J. Geophys. Res., 117, A02214,
doi:10.1029/2011JA016945.