Macro-Micro Interlocked Simulator

Tetsuya Sato

The Earth Simulator Center

Japan Agency for Marine-Earth Science and Technology

**Abstract**

Simulation Science is now standing on a turning point. After the
appearance of the Earth Simulator, HEC is struggling with several severe
difficulties due to the physical limit of LSI technologies and the
so-called latency problem. In this paper I would like to propose one
clever way to overcome these difficulties from the simulation algorithm
viewpoint. Nature and artificial products are usually organized with
several nearly autonomously working internal systems (organizations, or
layers). The Earth Simulator has gifted us with a really useful
scientific tool that can deal with the entire evolution of one internal
system with a sufficient soundness. In order to make a leap jump of
Simulation Science, therefore, it is desired to design an innovative
simulator that enables us to deal with simultaneously and as
consistently as possible a real system that evolves cooperatively with
several internal autonomous systems. Three years experience of the Earth
Simulator Project has stimulated to come up with one innovative
simulation algorithm to get rid of the technological barrier standing in
front of us, which I would like to call "Macro-Micro Interlocked
Algorithm", or "Macro-Micro Multiplying Algorithm", and present a couple
of such examples to validate the proposed algorithm. The first example
is an aurora-arc formation as a result of the mutual interaction between
the macroscopic magnetosphere-ionosphere system and the microscopic
field-aligned electron and ion system. The second example is the local
heavy rain fall resulting from the interaction between the global
climate evolution and the microscopic raindrop growth process. Based on
this innovative feasible algorithm, I came up with a Macro-Micro
Multiplying Simulator.

1.  **Introduction**

In the 60 years elapse of Computer Simulation since the birth of ENIAC,
three epoch-making computer technological innovations are to be noted.
The first one is the invention of the semi-conductor in 1948.
Undoubtedly, it was the semi-conductor invention that has brought
today's prosperity of simulation science. The second one, and the most
simulation oriented innovation, is the Seymour Cray's vector
architecture in 1976. The third one is the parallel architecture that
connects processors (or processor nodes) via an interconnection network.
The advantage of this parallel architecture is to increase the
theoretical peak flops by summing up the peak flops of all connected
processors.

The Earth Simulator (ES) is the successful product of taking advantage
of the vector and parallel architectures. Let me pick up a couple of
reasons for the great success of ES among many. The vector architecture
was employed and one-stage interconnection network system could succeed
in drastically reducing the so-called latency overhead because of the
smallness of the parallelism (640 processor nodes).

2.  **What the Earth Simulator has given us**

The goodness of the ES is certainly its excellence of sustained
performance for almost any scientific problems \[1\]. Besides, I would
like to point out a much more important impact the Earth Simulator has
brought us upon science. In our present Western science our attention
has been paid primarily to revealing the past history of how the present
Universe has been created and governed, and consequently, we have
discovered almost all fundamental (elementary) laws that are governing
the activity of the current universe, except the evolutionary equations
of life. Although we have devoted all our efforts to developing
mathematical tools to know and formulate the mechanism of revealed
elementary laws, we have unfortunately left behind us our enthusiasm of
invention of the tool to know the future evolution. This is because the
future is not that simple and is a complex manifestation resulting from
closely and highly tangled numerous elementary laws.

In the meantime, computer simulation was born to challenge to this
complex problem towards the end of the World War II. In this sense, it
would not be any exaggeration to say that simulation is the ultimate
tool to science.

I would say that it is the ES that has awakened us to realize "Time has
come". The ES has indeed recognized us that the future world has entered
into the realm of science. More specifically, the future becomes the
world of "Science Reality" which is so far treated as Science Fiction.
This implies that future evolution of nature such as prediction of
climate change and prediction of natural disasters are now in our hand,
and also that new materials and products can be developed without making
numerous, high-cost, time-consuming experimental tests as we usually do
so\[2\].

Since the products obtained by the ES to demonstrate the above-mentioned
statement are given elsewhere, I hope, interested readers should reach
there (Annual Reports and the Journal of the Earth Simulator published
by the Earth Simulator Center \[3, 4\]).

3.  **Beyond the Earth Simulator**

The real purpose of this paper is not confined within the territory of
the ES. Our precious time should be spent for our global design of the
21^st^ Century Simulation Science, and, more, the 21^st^ Century Science
itself.

As everyone is aware of it, we are now facing two essential computer
technological problems lying in front of us. The first one is the fact
that the LSI is approaching to the physical limit of its size
(\~10nanometer) as a semiconductor element. This indicates that an
avoidable physical limit is around the corner, no matter how
enthusiastic we are and that there is no way to sky-high enhancement of
chip performance as we used to. The smart way of getting out of this
difficulty was to connect the same processor in parallel as many as we
want, so that the face value of the performance can be increased
linearly with the increase of the number of connected processors.
However, as is well known, because of the increase of the communication
overhead in square of the number of processors, sustained performance is
drastically reduced, particularly when the parallelism exceeds ten
thousands (note that the ES connects 640 processor nodes). This becomes
more serious when the performance of a processor is more enhanced, like
one Telaflops.

Unless we develop a really innovative communication technology, it would
be a formidable task. Then, we have to patiently wait until such an
innovation would come up some day and until that day we are supposed to
do our gradual simulation research by using the existing computers day
by day. Certainly such steady work is necessary and important to weed
and plow the field, once it turns out to be able to grow crops. As far
as we rely on the existing machines, however, only crops of the similar
type and quality can be produced. No innovative and unpredictable crops
would no be discovered.

Now turn our eyes to the workings of nature. We can find that in any
system, all spatial and temporal scales are not fully occupied by energy
or information. Energy and information are always concentrated in
several discrete spatial and temporal scale domains in a system. This is
because if all spatial and temporal scale domains are fully occupied
with information or energy, then the system must be inactive and dead,
thus, uninteresting. That nature is active and meaningful is equivalent
to saying that nature is hierarchical, stratified, or, multiply
organized. This is also the case for any goods or products created by
humans. Information and energy are usually discretely localized in
spatial and temporal scale domains of any system, say, microscopic and
macroscopic.

Recall here that the ES has made it possible to deal with an entire
system. To be precise, the ES can make it possible to deal with an
entire system of only one internal system (organization) governed by the
same physical law. Therefore, the other organizations adjacent to this
organization must be plugged in as parameters and environmental
conditions. Usually all related organizations (layers) are closely
related to mutual evolution, hence, we call a system. Therefore, future
predictions made by the ES, for example, global warming, include
scientific uncertainties. There exists relatively large arbitrariness in
the choice of parameters and environmental conditions. This
parameterization procedure is unavoidable because even the ES is too
small to deal simultaneously and self-consistently with two mutually
cooperating adjacent organizations together. This is because to deal
with two cooperating organizations simultaneously and self-consistently
requires connection of some hundred millions of Earth Simulator scale
computers, namely, an ultra-massively connected simulators system.

Could you believe that such a giant computer system can be realized in
20 years from now on? The answer is absolutely negative.

In light of the fact that the ES has enabled us to deal with an entire
system of one organization as well as the fact that nature and man-made
products consist of discrete organizations, however, we can come up with
a very clever resolution to overcome the above seemingly formidable
difficulties.

4.  **Macro-Micro Interlocked Algorithm**

As is shown in Fig. 1, a system usually consists of plural, not more
than 5, organizations (layers) in spatial and temporal scale domains and
the intermediate (gap) region between neighboring organizations is a
void region where no active information and energy are in existence.
Thus, the void region may well be disregarded in the simulation of the
system, which drastically reduces the load of simulation. Hence, the
temporal and spatial scale domains could be drastically reduced, if
lower (microscopic) processes are dynamically solved at each, say, grid
point of the upper (macroscopic) spatial domain and at each macroscopic
time step. This reminds us of an idea that the macroscopic simulation
and microscopic simulation be done with their own simulators
independently and that the information exchanges be done only at each
macroscopic grid point with the connected microscopic processor
(simulator) (see Fig. 2).

Even if such an algorithm solving simultaneously macro and micro
processes be chosen, yet the situation is far beyond the horizon of the
technical feasibility of High-End-Computing. This can be easily
understood by seeing Fig. 2 where required are microscopic processors as
many as the number of macroscopic grid points, which is, say, of the
order of billions or so. Therefore, we are forced to give up this
scheme.

But, we do not have to be so pessimistic. Nature is not so ill-behaved.
Imagine, for example, the attack of earthquake. It is really a result of
microscopic crust destruction.

However, the crust destruction would not occur spontaneously without any
change in its environment. Prior to the fault breakup, energy
accumulation proceeds gradually there and the breakup is triggered when
it goes beyond the critical point.

Typhoon can be grown up when water vapors evaporated from the equatorial
hot sea surface ride updrafts at low pressure region. Thus, in order to
predict heavy rainfalls associated with the typhoon development,
required is that spontaneous microscopic raindrops formation including
rain condensation, fusion of raindrops by collision process and
gravitational fall are altogether solved in association with global
climate change simulation.

In many energy explosion phenomena that occur in space and universe such
as solar flares and galactic jets, environmental evolution of magnetized
plasmas forms locally a magnetic field configuration where electron and
ion microscopic interaction triggers collision free magnetic
reconnection. Once microscopic processes develop sufficiently that such
changes can change the ensuing macroscopic evolution, which can again
causes drastic changes in microscopic processes, maybe, in different
locations. Also, in full automobile design we are required to solve
self-consistently and simultaneously the mechanical dynamics including
transmission, wheel, handling, etc. and the engine combustion process
which are governed by different dynamical equations. In this example,
different processes exchange energy continuously through a fixed contact
point (boundary).

In most other fields like bio-science, nano-material, nuclear fusion,
etc., similar discrete macro-micro feedback processes are taking place.
This consideration has led me to come up with a really clever algorithm
of drastically reducing the number of macroscopic processors, because at
most macroscopic grid points where no drastic change in microscopic
state (phase transition) would occur we do not have to make microscopic
simulations. Only at specific macroscopic grid points where microscopic
instability, phase transition or catalytic action is about to occur,
microscopic simulation should be carried out in parallel. Consequently,
we can drastically reduce the number of microscopic processors, say,
from billions to one or few. This situation is depicted in Fig. 3.

5.  **Concrete Demonstrations of Macro-Micro Interlocked Simulations**

Let us show two examples. The first one is aurora formation resulting
from double layer acceleration at high altitude field-aligned regions
above the aurora region in self-consistent collaboration with the
magnetosphere --ionosphere Alfven wave feedback instability \[5, 6\].
The second example is heavy rainfalls resulting from microscopic
dynamical simulation that self-consistently solves the growth and fall
of rising water vapors based on condensation and collision processes
under the gravity force in self-consistent collaboration with the global
climate change simulation resulting in typhoon formation.

5.1 **Macro-Micro Interlocked Aurora Formation**

It is well known that the original energy source is the solar wind. The
solar wind interacts with the geomagnetic field resulting in the
formation of the magnetosphere that is roughly balanced between the
solar wind pressure and the geomagnetic pressure whereby twin vortices
are formed basically in the magnetospheric equatorial plane as is
schematically shown in Fig. 4. The clockwise and counter-clockwise
magnetospheric convections generate positive charges on the dawn side
and negative on the dusk side, which drive field-aligned currents
terminated by the ionospheric resistive current. This great current
system is unstable for a certain frequency disturbance. Thus, the
field-aligned current is locally enhanced. When it grows beyond a
certain threshold, the field-aligned current (primarily carried by
electrons) interacts with the background ions (protons) and an
electron-ion microscopic instability is excited to create double layers,
whereby downward electrons are strongly energized to produce excitation
of oxygen and nitrogen molecules at the ionosphere. The ionospheric
density is also strongly modulated by the local enhancement of
ionization, which in turn changes the evolution of the
magnetosphere-ionosphere current system (see, Fig, 5).

Really realistic aurora arcs are generated by this MMI algorithm. Two
snapshots (taken from animation) are shown in Fig. 6 \[7\].

2.  **Macro-Micro Interlocked Rainfall Formation**

In the present simulation of climate change, macroscopic atmospheric and
ocean circulations are solved based on Navier-Stokes equations. However,
cloud formation is not solved by means of first principles. They adopt
so-called parameterization, or at best microscopic physics-oriented
parameterization which adds more sophisticated parameters such as
several states of water (raindrops, snow flakes, etc.) and spatial
distribution within each macroscopic grid volume. It is a static model,
though, where neither raindrops nor snow flakes formation is dynamically
solved. No matter how nicely parameterized, therefore, dynamical
causality of raindrops formation is not involved. In order to make a
more realistic prediction of rainfalls at a local region, the
atmospheric dynamics and raindrops (snow flakes) formation are requested
to be solved simultaneously.

In this demonstration a global climate change simulation is done by a
newly developed non-hydrodynamic world-fastest global simulation code
with 5.5 km resolution \[8\]. A typhoon is generated and passing over
the Japan Island. On the other hand, developed is a new raindrop growth
simulation code that solves self-consistently the growth of water vapor
due to condensation and collision processes and gravitational fall
\[9\]. By rather arbitrarily choosing a low pressure position in the
typhoon rain band predicted by the global code, we give the
environmental condition obtained by the global code to the raindrop
formation code and obtain how heavy rainfall (see Fig. 7). One
preliminary example of rainfall associated with the typhoon at a rain
band position is shown in Fig. 8

6.  **Conclusion ― Proposal of Macro-Micro Interlocked Simulator**

The above two examples are just preliminary ones that are made to test
the validation of the proposed Macro-Micro Interlocked Algorithm.
Nevertheless, these tests clearly validate that the proposed algorithm
is eligible to be the next generation algorithm,

With this validation I come up with a good candidate for the next
generation simulator. Since the macroscopic simulation and the
microscopic simulation are carried out independently except for the
occasional necessary information exchanges, we provide two best
simulators that are large enough to make the macroscopic and microscopic
simulations in self-sufficient manner. Those simulators can not
necessarily be the same. One can be the vector-type and the other can be
the scalar type. Since both can be of different type, we provide a
simple data exchange device that can transfer necessary information on
both ways occasionally when the programs running on both simulators give
an order to do so. The required demand for this device is about the
latency. The latency must satisfy that the intervening time for data
exchange is at worst smaller than one macroscopic time step. Another
requirement is that the microscopic simulator should have a good
performance to ensure the completion of the microscopic simulation
within, hopefully, a couple of macroscopic time steps during which no
appreciable change occurs in macroscopic state so that we do not have to
worry about the loss of physical validation of microscopic simulation

Fig. 9 is a conceptual design of the Macro-Micro Interlocked Simulator
we propose for realizing the Macro-Micro Interlocked Algorithm. Suppose
that both macro and micro simulators have main memories that can deal
with 10**^4^** grid points in each direction, hence, the total grid
points of 10**^12^.** Then, the proposed system can do the job
equivalent with the job with 10**^4^** × 10**^4^ ＝** 10**^8^** grid
points in each direction, thus, 10**^24^** grid points in total (three
dimensions). In other words, the Macro-Micro Interlocked Simulator can
do an equivalent job to the simulator that could have the capability of
dealing with 10**^24^** grid points. As a matter of course, the
practical performance be requested to be increased at least
proportionally.

In conclusion, if the simulator system proposed in this paper is
developed, I am sure that real paradigm shift occurs in simulation and
cutting-edge realm of science is opened up, which can lead to a
revolution in academia and industry. Any cutting-edge science and
technology can gradually invading into the human society to cause a
revolution in life style and eventually end up with a paradigm shift in
humans' mind. I would call such a society "Simulation Culture".

**Acknowledgements**

First of all, I would like to dedicate this proposal to the late Hajime
Miyoshi without his enthusiasm the Earth Simulator would have not been
in existence, thus, the present excellent idea would have not seen the
light of day and have kept behind the scenes.

I would like to express my thanks to all members of the Earth Simulator
and JAMSTEC Headquarter, in particular, Dr. Watanabe, Dr. Kusano, Dr.
Kageyama, Dr. Sugiyama, Dr. Shima, Dr. Takahashi, Dr. Araki, Dr. Uehara,
Dr. Ohno, Dr. Furuichi, and Dr. Kawahara and my executive assistant Mr.
Hirano, and finally my secretary Ms. Horiki.

References

\[1\] Global Atmospheric Circulation Code (AFES) by Shingu was awarded
the Gordon Bell Award in the 2002 SC; Seismic Wave Simulation Code by
Tsuboi et al. was awarded in the 2003 SC; Dynamo and Mantle Dynamics
Simulation by Kageyama was awarded in the 2004 SC.

\[2\] Simulations for automobile crash done in collaboration with the
Automobile・・・

\[3\] Annual Reports of the Earth Simulator Center, 2002, 2003, and
2004.

\[4\] The Journal of the Earth Simulator, Vol. 1(2004), Vol.2(2005) and
Vol.3(2005).

\[5\] A Theory of Quiet Aurora Arcs, Journal of Geophysical Research,
・・・(1976??)

\[6\] Ion Acoustic Double Layer, Phy. Rev. Lett. ・・・

\[7\] In collaboration with Dr. Sugiyama and Dr. Watanabe.

\[8\] In collaboration with Dr. Takahashi.

\[9\] In collaboration with Dr. Sugiyama, Dr. Shima, and Dr. Kusano.

Figure Captions

Fig. 1 Any system consists of two or a few internal systems
(organizations or layers). Each layer is distinct in scale and
information (energy) in each layer is mostly confined within 10**^4^**
in scale. When the system consists of two layers, then the total scale
is 10**^8^** or larger sine there is a gap scale in between.

Fig. 2 A naïve conceptual design of two-layer (micro-macro) system

Fig. 3 A conceptual design of the macro-micro interlocked algorithm

Fig. 4 A conceptual picture of the solar wind-magnetosphere-ionosphere
system in which aurora is excited.

Fig. 5 Flow chart of the macro-micro interlocked simulation of aurora
formation

Fig. 6 A shot of aurora brightening seen from the ground (right) and a
shot looked down from a satellite (left) obtained by the MMI simulation

Fig. 7 Flow chart of the MMI simulation of raindrop formation

Fig. 8 Growing raindrop by condensation and collision processes in an
updraft and rainfall when the gravitational force overcomes the updraft
force (right) at a certain rain band position of the typhoon arm (left)
obtained by global climate simulation with the resolution of 5.5km

Fig. 9 Conceptual design of the MMI Simulator