Proceedings of ITC/ISHW2007

Competition and Synergy of
Macro- and Micro-scale Physics– Status and Prospects ofMacro-Micro Interlocked Simulation –Kanya KUSANO, Shigenobu HIROSE, Tooru SUGIYAMA,
Akio KAWANO, Shinichiro SHIMA, Hiroki HASEGAWA, Yuji KAWAMURAThe Earth Simulator Center,
Japan Agency for Marine-Earth Science and Technology, Yokohama 236-0001, Japan
Multi-scale phenomena are now crucial issue in the wide range of research ﬁelds, not only plasma
physics, but also chemistry, astrophysics, geo-science, bio-science and so on. For the last decade, aiming
to understand the interaction between macro-scale and micro-scale dynamics, several new methodologies
for numerical simulations have been quickly developed. Macro-Micro Interlocked (MMI) Simulation is
one of such new simulation frame-works. The MMI simulation is performed by the two-way connectionof diﬀerent numerical models, which may handle macroscopic and microscopic dynamics, respectively.
The application of the MMI simulation to diﬀerent multi-scale phenomena, for instance, cloud formation,gas detonation, and plasma dynamics, is investigated. The results of all the applications demonstrates
that the MMI simulation is a prospective methodology for multi-scale studies.
Keywords: multi-scale, simulation, plasma, cloud, detonation

1 Introduction
Multi-scale phenomena, in which the micro-scale elementary process and the macro-scale system dynamics
mutually inﬂuences each other, are now a crucial issue in vastly diﬀerent research ﬁelds, not only plasma
physics, but also chemistry, astrophysics, geo-science,
bio-science, material science, mechanical engineering,and so on. However, fundamental diﬃculty arises
for the research of the multi-scale processes, because
physical processes in diﬀerent scales are governed by
diﬀerent physical laws, for instance, the molecular dynamics in micro-scale and the continuum dynamics inmacro-scale.
Although computer simulation must be a powerful tool for the understanding and the predicting
the complex nonlinear phenomena, the applicabilityof simulation cannot get rid of restriction of the scale,because computer simulation is usually formulated on
the basis of the theoretical description like partial differential equations, which is valid only in some limited
scale. Therefore, the applicability of each simulation
model is also limited to the scale where the basic theory is valid.
To conquer this limitation, several new frameworks are proposed recently. So far, the multi-scale
simulations have been developed by two diﬀerent manners. The ﬁrst is the extension of macroscopic simulation, in which the microscopic eﬀects are included
author’s e-mail: kusano@jamstec.go.jp

as phenomenological parameters, just like the anomalous resistivity caused by micro-scale turbulence inhigh-temperature plasmas and the bulk cloud parameterization in atmospheric global circulation model.
However, since the phenomenological parameterization is not guaranteed to work well in unknown circumstances, the applicability of such a method must
be restricted by given knowledge.
Another approach is the large-scale microscopic
simulation, whereby the macroscopic phenomena areattempted to be built from the elementary block of
micro-scale processes. The large-scale molecular dynamics (MD) simulation is the typical example of that.
However, it is computationally so demanding. Since
the scale which we can directly calculate with the MD
model is limited to the order of micron meter, evenusing the world largest super-computer [4], we needthe 1024 (= 106·4 ) times faster computational capability than the current super computer. It would not be
feasible at least in near future.
The limitation of the computational feasibility
requires that a new type of software and mathematical frame-work should be developed to conquer
the diﬃculty of multi-scale simulation. In fact, several new numerical models have been proposed recently for this purpose. One of the most promising concept is given by the interconnection of diﬀerent numerical models. [1] developed a new way tomake the interconnection between the ﬁnite-element,
molecular dynamics and semi-empirical tight-bindingProceedings of ITC/ISHW2007
representations, and demonstrated that the connection model, called MAAD (macroscopic, atomistic,
ab initio dynamics), is eﬀective as the methodology
for the study of brittle crack propagation in silicon.
On the other hand, [13] proposed the concept ofheterogeneous multi-scale method (HMM) as a general methodology for multi-scale modeling. In the
HMM, scale separation is exploited so that coarsegrained variables can be evolved on macroscopic spatial/temporal scales using data that are predictedbased on the simulation of the microscopic process.
Similar model was proposed also for meteorologicalsimulation, which is called “super-parameterization,”by [3] and [7]. Super-parameterization is performedby the mutual connection between atmospheric global
circulation model (AGCM) and cloud system- resolving (CSR) model. In the super-parameterization, theCSR models are embedded inside each grid cell of
AGCM, and the results of them are used for the cloudparameterization of AGCM.
Macro-Micro Interlocked (MMI) Simulation was
proposed by [8] as a new methodology for the multiscale simulation. The fundamental principle of the
MMI simulation is in common with the heterogeneous multi-scale method (HMM) and the superparameterization. The MMI simulation consists ofthe macro- and micro-simulation models, which arecarried out simultaneously and interconnected to eachother. The micro-model is performed under the environmental condition calculated by the macro-model,
and the microscopic information is fed back to themacro-model [6]. Originally, the MMI simulation has
been proposed as a computational algorithm suitable
to the hardware of heterogeneous architecture, called
MMI simulator, in which diﬀerent types of architecture ﬁtting respectively to the numerical models ofmacro- and micro-scales are interconnected. In particular, heterogeneous vector/scalar architecture is themost promising design of the MMI simulator, and wehave developed several applications for that.
The objective of this paper is to present the brief
review of the application study of the MMI simulation.
In the following section, we will explain the principle
and the typical results of each MMI simulations forcumulus cloud formation, gas detonation, and plasma
dynamics. Finally, the prospects of the MMI simulation will be summarized in Sec.3.

2 Applications
2.1 Cloud Simulation
Although clouds play a crucial role in any meteorological phenomena like global warming, the numericalmodeling of cloud is not well established yet. The
reason of that is attributed to the fact that the forma-tion of clouds and the development of precipitation are
essentially governed by multiscale-multiphysics processes. The macro-scale processes such as the ﬂuid
motion of moist air associated with clouds is called”cloud dynamics”, and the micro-scale processes suchas the condensation and coalescence of water droplets
are called ”cloud microphysics”. These two processes
mutually aﬀect each other.
Although the cloud dynamics simulation has beenquickly developed with the achievement of large-scale
ﬂuid model, the ﬁrst-principle simulation of cloud microphysics is still diﬃcult. It is due to the fact thathuge number of tiny droplets are involved in cloud
processes. Therefore, some empirical approximation
called ’bulk-parameterization’ has been widely used
for cloud modeling. However, the applicability of any
empirical model is restricted as mentioned in Sec.1,
and we cannot use it for the prediction of unknown
phenomena like global warming.
Aiming to overcome this diﬃculty in the cloudmodeling, we recently developed a novel simulation
model of cloud microphysics, named Super-Droplet
Method (SDM) [10], which is a particle-based cloud
microphysics model. The super-droplet is deﬁned
as computational particle representing multiple real
droplets, which have common properties, e.g. position, velocity, cloud condensation nuclear (CCN), andelectric charge. The motion of each super-droplet is
calculated by the equation of motion or by the assumption that each droplet immediately attains terminal velocity. The condensation and the evaporation of droplets can be directly calculated based on
Köhler’s theory using the properties of super-droplet
and the state variable of atmospheric environment.
The coalescence of droplets is handled by the stochastic manner just like the Direct Simulation MonteCarlo (DSMC) method.
The cloud dynamics is simultaneously calculatedwith SDM using the non-hydrostatic model equations,DU,= −∇P − (ρ + ρw )g + λ∇2 U
Dt
P = ρRd T,
Dθ
L=−
Sv + κ∇2 θ,
Dt
cp Π
Dρ
= −ρ∇ · ρ,
Dt
Dqv
= Sv + κ∇2 qv ,
Dt

ρ

(1)
(2)
(3)
(4)
(5)

where ρ is the density of moist air, ρw the density of the wind velocity, P the pressure, λliquid water, U
the viscosity, Rd the gas constant for dry air, T thetemperature, θ the potential temperature, L the latentheat, qv the mixing ratio of vapor, Sv the source ofwater vapor, and Π is the Exner function, respectively.

Proceedings of ITC/ISHW2007

Fig. 1 The simulation results of cloud formation and precipitation using the super-droplet models. Horizontal line on the bottom indicates ground level, andgray-scale represents the cloud droplets.

The density of dropletsρw (x, t) =ξi mi (t)w(x, xi ),iand the conversion ratio between vapor and liquid
Sv (x, t) = −1 ∂ρw (x, t),ρ(x, t)∂t

are sent to the ﬂuid model, and used in the source
terms, where i is the index of super-droplet, ξi themultiplicity of super-droplet, mi the mass of droplet,
and w is the shape function of super droplet.
The algorithm of SDM, in which particles(droplets) and ﬂuid (moist air) interact with eachother, is basically same as particle-in-cell (PIC)
plasma simulation, in which particles for electron andions interact with electromagnetic ﬁelds assigned atcells. It is also similar to the plasma simulation thateach super-droplet represents multiple number of real
droplet.
In order to demonstrate the feasibility of SDM,
it has been applied to the simulation of cloud formation and precipitation in maritime cumulus. The
simulation system is given by 2-dimensional x − zdomain just for simplicity. The initial state consists of un-saturated stratiﬁed layer, which is slightly
unstable only under 2km in altitude, and numberof tiny droplets, which contain soluble substance as
Cloud Condensation Nuclear (CCN), are uniformly
distributed in the entire domain. As shown in Fig.1,
the results indicate that the particle-continuum coupled model may work to simulate the whole processfrom cloud formation to precipitation without introducing any empirical parameterizations. The newmodel may provide a powerful tool for the studyof cloud-related various problems, although the predictability of SDM should be evaluated more carefully.

2.2 Detonation Simulation
Combustion ﬂuid dynamics is also the typical subjectof multi-scale simulation, in which chemical reaction

is mutually interacted with the macroscopic ﬂow dynamics. In the conventional methods, the reaction istreated by the Arrhenius rate equation. However, thereaction rate of the equation must be derived from the
distribution function assuming local thermal equilibrium (LTE). Although the LTE assumption could besatisﬁed in normal condition of ﬂuids, when the local
Knudsen number, which is deﬁned as the ratio of mean
free path to characteristic length scale, is larger than0.01, the assumption of LTE may not be valid. In thecase, the ﬂow should be treated as rareﬁed gas andwe have to solve the Boltzmann equation. Especially,detonation, which is sustained by shock wave driven
by combustion wave, is the case, because the thickness of shock may be comparably thin as the mean
free path. It implies that the Arrhenius rate equation
may not be valid on the shock front.
So, we have developed a novel method for simulation of combustion by connecting a microscopic molecular model and a macroscopic continuum model, thoseare based on the Boltzmann equation and the NavierStokes equation, respectively. We adopted non-steady
DSMC method for the molecular model, and the continuum model is carried out by the HLLC method.
Our detonation model is an extension of Hybrid
Continuum-Atomistic Simulation, which has been developed by many authors [2, 14, 9]. The algorithm
to connect the molecular model and the continuum
model is summarized as follows: (1) Gradient of pressure is monitored during the simulation by the continuum model, and the molecular model is embedded in
the region where the steepness exceed some threshold
and the continuum model is failed. (2) Interlocking
layers are laid around the outer-most region of themolecular domains, where particles are generated to
let the distribution function be able to represent themacroscopic variables; density, velocity and pressure.
Numerical ﬂux on the outer-most boundary of the continuum domain is calculated from the particle motion.
Some result of the two-dimensional detonation
simulation is shown in Fig.2. It represents the distribution of pressure on the detonation front. The
domain between dotted lines corresponds to the region, where the particle-based model is adopted. The
smooth connection between ﬂuid and particle-basedmodels indicates that the continuum-atomistic simulation is applicable also to the combustion process.

2.3 Plasma Simulation
Plasma inherently forms a multi-scale system, inwhich diﬀerent characteristic scales related to electron
and ions mutually interact. The key issue in multiscale plasma processes is how macroscopic magnetohydrodynamics (MHD) is related to microscopic particle kinetics. For instance, in magnetic reconnectionProceedings of ITC/ISHW2007

B

t=0

t=12.7x

Fig. 2 The pressure distribution of the detonation simulation. The particle-based domain, which corresponds to the region bounded by dotted lines,
tracks the detonation front.

that is responsible for explosive energy release in hightemperature plasmas, the kinetics might be important
especially in the restricted diﬀusion region, which isformed when anti-parallel magnetic ﬂuxes collide eachother.
So, we have developed the new algorithm for sucha multi-scale plasma dynamics. Our model is constituted by the connection of the particle-in-cell (PIC)model and the magnetohydrodynamic (MHD) model,
the former of which is embedded in the MHD simulation domain[11]. This MHD-PIC interlocked modelis resembled to the continuum-atomistic simulation,
which was explained in the previous subsection. However, it should be mentioned that, in plasma simulation, several characteristics described by PIC model
are negated in continuum (MHD) model, in contrast
to the hydrodynamic simulations. So, in order to makethe smooth interlocking between MHD and PIC models, we have to introduce some special ﬁlter, which
passes only the MHD mode from the PIC domain to
the MHD domain. The ﬁltering process is actualized
by the sophisticated sub-cycle procedure. Refer to [11]for the detail.
Figure 3c shows the magnetic wave form in the
test-simulation for the one-dimensional Alfvén wave
propagation. In this simulation, PIC model is embedded only in a small part of the simulation box, and we
examined whether long wavelength Alfvén wave cansmoothly propagate in the interlocked system of PIC
and MHD. The ﬁgure shows that the wave form is
scarcely aﬀected by interlocking with the PIC model,
as high-frequency modes appear in the PIC domain.
It indicates that the particle-continuum connection
model is applicable also to plasma simulations.
Recently, [12] applies the interlocked techniqueto connect diﬀerent hybrid simulations for the study

Fig. 3 Magnetic field distribution in the test simulation
for Alfvén wave using the PIC-MHD interlocked
model. The region between two dotted line corresponds to the part where PIC model is embedded.

of particle acceleration on plasma collision-less shock.
In that study, the conventional hybrid simulation, in
which ions are calculated by particles, is used for the
calculation of particle acceleration in shock and isconnected to the energetic particle hybrid simulation,
which calculates only high energy ions as particles andhandles the thermal ion as ﬂuid. Since the computational cost of the energetic particle hybrid model ismuch cheaper than the ion particle hybrid model, we
can dramatically reduce the computational demandusing the new interlocked plasma simulation, and it
enables the macroscopic plasma simulation including
kinetic eﬀects.

3 summaryThree diﬀerent applications for the MMI simulation
are demonstrated. In any examples, the particlecontinuum connection technique plays a crucial role.
However, it is worthwhile to note that the particlecontinuum connection in the cloud simulation is notbased on the domain decomposition method, but theapplication of particle-in-cell technique, unlike theother two applications of plasma and detonation, because cloud is ubiquitous phenomena and we cannot
restrict the cloud domain into a limited region. It is a
new challenge to embed the cloud microphysics model
into the atmospheric global circulation model.
Although the MMI technique is much eﬀective forthe multi-scale simulation, in which the micro-scalemodel is necessary in a limited region, the gap of timescale between the macro-scale and micro-scale processes is still issue to be resolve, because the microscale model is usually time-consuming and it is difﬁcult to calculate the two diﬀerent models simulta-

Proceedings of ITC/ISHW2007
neously. Several new technique to fulﬁll the timescale gap is now proposed. For instance, equation-freemethod proposed by [5] tries to accelerate the time integration using extrapolation technique based on the
micro-scale calculation.
The MMI simulation is now opening a new worldof simulation science, because the new numericalmodel can cast oﬀ the restriction of theoretical description. The MMI simulation is applicable to vastly
wide range of ﬁelds, and some common technique canbe used for the model of even diﬀerent phenomena.
The particle-continuum connection and the particlein-cell technique are applicable to the simulation of
diﬀerent processes. It demonstrates that the MMI
simulation can also play an important role for interdisciplinary communication.
Acknowledge: The study is the results of the research program of “Development of Multi-scale Coupled Simulation Algorithm” in the Earth Simulator
Center. The works presented in this paper is partially
supported by the Grant-in-Aid for Scientiﬁc Researchfrom Japan Society for the Promotion of Science (Science Research (B) 19340180).
[1] F. F. Abraham, J. Q. Broughton, N. Bernstein and E.
Kaxiras, Europhys. Lett. 44 783-787 (1998).
[2] O. Aktas, N. R. Aluru, J. Comp. Phys. 178 2 (2002)
342–372.
[3] W. W. Grabowskii, & P. K. Smolarkiewicz, Physica D
133 171–178.
[4] K. Kadau, T. C. Germann, P. S. Lomdahl, Int. J. Modern Phys. C 12 (2006) 1755–1761.
[5] I. G. Kevrekidis, C. W. Gear, G. Hummer, Aiche J.
50 7 (2004) 1346-1355.
[6] K. Kusano, S. Hirose, T. Sugiyama, S. Shima, A.
Kawano, and H. Hasegawa, Computational Science–
ICCS 2007, Lecture Notes in Computer Science(Springer Berlin / Heidelberg) (2007) pp.0302-9743(Print) 1611-3349 (Online).
[7] D. Randall, M. Khairoutdinov, A. Arakawa, and W.
Grabowski Bulletin of the American MeteorologicalSociety 84, 11, 1547–1564 (2003)[8] T. Sato, Journal of Physics: Conference Series SciDAC
2005 Scientific Discovery through Advanced Computing, Vol.16, (2005) 310–316.
[9] T. E. Schwartzentruber, I. D. Boyd, J. Comp. Phys.
215 2 (2006) 402-416.
[10] S. Shima, K. Kusano, A. Kawano, T. Sugiyama, and
S. Kawahara, Super-Droplet Method for the simulation of Cloud and Precipitation; a Particle-Based
Microphysics Model Coupled with Non-hydrostatic
Model., to be submitted.
[11] T. Sugiyama, and K. Kusano, J. Comp. Phys. (inpress).
[12] T. Sugiyama, and K. Kusano, submitted to ASP
Conf. Ser.
[13] E. Weinan, B. Engquist, Z. Huang, Phys. Rev. B, 67
092101-1–4 (2003)[14] H. S. Wijesinghe, N. G. Hadjiconstantinou, Int. J.
Multiscale Computational Engineering 2 2 (2004) 189–
202.