HPCI magazine vol.7

High-Precision
Simulations of theEvolving Universe
to Weigh Cosmic
Neutrinos
In July 2021, a six-dimensional Vlasov simulation was executed using nearly all the nodes ofsupercomputer Fugaku. Researcher Kohji Yoshikawa became the first person in the world todevelop software that could successfully perform such a computation. His results showed, withmuch greater precision than previously possible, how the distribution of neutrinos throughout theuniverse has changed from the Big Bang up to the present day. How did he arrive at this achievement? And what is the scientific significance of these results?

Kohji Yoshikawa
Associate Professor
Center for Computational Sciences
University of Tsukuba

Neutrinos Influence theFormation of the Large-Scale
Structure of the Universe
The countless galaxies throughout the universe are known to not be distributed uniformly
but rather have a structure similar to stickingbubbles. Our current understanding is thatgalaxies are clustered along the surfaces ofmany enormous structures conceived as“ b u b b l e s ”, w i t h r e l a t i v e l y f e w g a l a x i e s
residing inside these bubbles. In the very
early days of our universe, matter was distributed mostly uniformly, albeit with someslight fluctuation variations in density. We
think the gravity of the slightly denser regions
attracted nearby matter, thereby causing
them to grow and attract ever more matter.

01

This difference in density gradually increased, creating the large-scale structureof the universe as it now exists.
“Dark matter” is believed to have played amajor role in the evolution of the universe.
Dark matter is an unidentified and invisible
component of the universe, only known
for exerting gravity on surrounding matter.
Although we can’t see dark matter, we
believe it makes up most of the mass inthe universe and that its distribution
throughout the universe corresponds tothe observed distribution of galaxies. To
better understand how the large-scale
structure of our universe came to be, researchers run “time evolution” simulations

Fig. 1and study how the distribution of dark
matter evolves over time. In such simulations, researchers have adopted awell-established method called “N-body”
simulation, in which the distribution ofmatter in the universe is represented by avery large number of particles mutuallyinteracting via gravity.
Around the year 2000, it was discoveredthat even tiny neutrinos have someamount of mass. Scientists then began toinclude neutrinos in their simulations of theevolving universe. “Neutrinos are extremely
light, but they’re quite abundant in theuniverse. They zip around at tremendous
speeds in the universe, which interferes

Vlasov Simulations and N-body Simulations of
Neutrino Distribution in the Universe

Simulations of the evolving universe compute the motion ofmatter in a virtual “phase space” in the form of 3D coordinates for both position and velocity. Real-world neutrinos
are smoothly distributed in the phase space (a). In N-bodysimulations (b), however, these distributions are represented
by discrete particles. In Vlasov simulations (c), the neutrino
distribution represented in the form of a distribution function
mapped onto the fine mesh grids is very smooth. So, theVlasov simulations produce less noisy, more accurate
results than the N-body simulations.

vol.7 - High-Precision Simulations of the Evolving Universe to Weigh Cosmic Neutrinos(a) Distribution of neutrinos in phase space(b) N-body simulations(c) Vlasov simulations

with the dark matter ’s ability to attract
mass gravitationally. So, scientists started
running N-body simulations that includen e u t r i n o s , b u t t h e y w e r e n ’t a c c u r a t eenough,” explains Yoshikawa.

Taking on an Equation
Once Thought Unsolvable
Rather than N-body simulations, Yoshikawa
opted for Vlasov simulations, which solve the“Vlasov equation”. Here’s how they differ.
Neutrinos hurtle through the universe atvarious speeds and directions. In computer
simulations, the positions and velocitiesof these neutrinos are represented bycoordinates of a virtual “phase space” which
consists of a 3D position space and a 3D
velocity space, meaning six dimensions in
total. This is the case in both of N-body andVlasov simulations (Fig. 1a). The differenceis that in N-body simulations, the virtual“particles” are placed in a neutrino-densearea and then their changes in position andvelocity over time are computed (Fig. 1b).
In Vlasov simulations, on the other hand, thephase space is divided into fine mesh grids
and the time evolution of the neutrino
distribution within those mesh grids iscomputed (Fig. 1c). Whereas real-world
neutrino distributions are quite smooth02

(Fig. 1a), the scattered particle images
produced by the N-body simulations containa significant amount of random noise. The
Vlasov simulations can properly computeeven low-density areas, allowing for moreaccurate results.

Yoshikawa began developing the numericalcode for these simulations in 2007, when
he started working at the University ofTsukuba. After several years of trial anderror, in 2013 he successfully demonstrated
that six-dimensional Vlasov equations
c o u l d b e s o l v e d o n t h e u n i v e r s i t y ’s
“T2K-Tsukuba” supercomputer. This was aworld’s first achievement. Next, at thes u g g e s t i o n o f N a o k i Yo s h i d a a t t h e
University of Tokyo, Yoshikawa decided touse his simulation code to tackle thereal-world problem of neutrino distributionin the universe over time.

Fig. 2T h e a c t u a l r e s u l t s o f Yo s h i k a w a ’ssimulations (Figure 2) show that Vlasov
simulations can reproduce the smooth
distribution of neutrinos. But, solving
six-dimensional Vlasov equations
requires a supercomputer with enormous
memory capacity and exceptional
performance, not to mention a numericalalgorithm that is both very efficient andv e r y a c c u r a t e . Yo s h i k a w a ’s r e s u l t s
overturned commonly-held long-standing
assumptions that such computations wereimpossible.

vol.7 - High-Precision Simulations of the Evolving Universe to Weigh Cosmic Neutrinos

However, developing a “numerical scheme”
to solve these 6D Vlasov equations for theexpanding universe with sufficient numerical accuracy was not easy.
Yoshikawa reflects “I thought about quitting
it many times.” Takashi Minoshima at theJapan Agency for Marine-Earth Scienceand Technology provided him with animportant clue to overcoming the difficulty.
The two promptly began working together.
By 2017, their numerical schemes had
been improved and become much moreaccurate. Yoshikawa performed numericalsimulations with the improved numerical
scheme on a variety of supercomputers,
including the K computer. Then in the fall ofComparing Vlasov Simulations with N-body Simulations

Comparison of neutrino distributions
obtained with a Vlasov simulation(a) and an N-body simulation with asimilar computational cost (b).
In the Vlasov simulation, the neutrino
distribution is smooth, whereas thecounterpart in the N-body simulation
appears somewhat pixelated, making
the subtleties of the structures harder
to be discerned.
(a) Vlasov simulation(b) N-body simulation03

2019, he was given permission to run hissimulations on an early access system ofFugaku.

All-node Calculations Get
Maximum Performance outof Fugaku
F u g a k u ’s h u g e m e m o r y s p a c e , a f a s tinterconnect and high parallelization
efficiency make it ideal for Vlasov simulations, in which calculations on a large
number of mesh grids are performed inparallel. However, since Fugaku has aSIMD (Single Instruction Multiple Data)
instruction set different from previoussupercomputers, the code has to be
refactored to match the Fugaku’s instruction
set for the maximum performance onFugaku. Yoshikawa devised an idea for
speeding up the calculations, andpostdoctoral researcher Satoshi Tanaka(currently a research fellow at the Yukawa
Institute for Theoretical Physics, KyotoUniversity) implemented it in the program.
In July 2021, a Vlasov simulation ofcosmic neutrinos adopting about 400
trillion mesh grids was successfully
performed using 147,456 nodes, or about
93% of the total 158,976 nodes on Fugaku(Fig. 3a).

vol.7 - High-Precision Simulations of the Evolving Universe to Weigh Cosmic Neutrinos

This was, of course, the largest Vlasov
simulation ever conducted in the world. Itwas also 10 times faster than a comparable
N-body simulation previously performed ona Chinese supercomputer.
The distributions of neutrinos and dark
matter are computed with Vlasov andN-body simulations, respectively (Fig 3b).
“We can constrain the possible range of
yet-to-be-known neutrino mass byanalyzing the simulated data andcomparing them with observational data.”,
explains Yoshikawa about the scientificsignificance of these simulations. Weighingth e n e u t rin o ma s s d ire c tly le a d s to th eunderstanding of the dynamical effect ofneutrinos on the large-scale structureformation in the universe. Furthermore, theVlasov simulations are expected to be
successfully applied to the numericalsimulations of magnetic plasma such as“accretion disks” formed around theastrophysical black holes and magnetic
storms in the inter-planetary space.

Fig. 3The Results of Simulations
Using 147,456 Nodes on Fugaku

The time evolution of neutrinos and dark matter wascomputed using a Vlasov simulation (a) and an
N-body simulation (b), respectively. The matter
distribution at the very beginning of the universe(about 200 million years after the Big Bang)obtained from actual observations was adopted as
the initial condition and evolved into the present-day
universe (after the 13.8 billion years after its birth).
A timestep of the simulation corresponds to about
2 million years. The size of the simulated volume is
5.6 billion light-years (1200 Mpc/h per) side, which
covers nearly all of the observable universe.
[Mpc/h is a unit of length corresponding to about 466
million light years. The h is the Hubble parameternormalized by 100km/s/Mpc and observationallyestimated to be around 0.7.]

Yo s h i k a w a r e f l e c t s , “ T h i s t i m e , t h esimulations used practically all the nodes ofFugaku, and also used up nearly all thememory space. There were various issues
both in the software and hardware
s i d e s . We b e l i e v e t h a t l o t s o f v a l u a b l eexperiences to overcome such issues will04

be reflected in the future operation ofFugaku and be useful to bring the best
performance it has.” This substantial
all-node run will be an encouraging impetus
to the future utilization of Fugaku.

About theResearcher“I’ll become a researcher because I can dowhat I love for a living.” Yoshikawa has always
thought so since he was a junior high school
student. In 1995, when he was a junior inundergraduate school, he started to study
astrophysics attracted by the emerging computer simulations which were not common yetat that time. There, he encountered the Vlasov

equation and decided to numerically solve it
himself one day. As a mentor, he encourages
his students to conduct research with theattitude of doing what no one else is doing. As
a native of Suzuka City, which is famous forholding Formula One Grand Prix in Japan, heloves watching motor sports, saying “I love thepursuit of pure speed.”

Associated Research Projects• “Toward a Unified View of the Universe: From Large-scale Structures to Planets” (hp200124/hp210164)
Principal Investigator: Junichiro Makino, Kobe University
2021 ACM Gordon Bell Prize finalist for Research Findings

vol.7 - High-Precision Simulations of the Evolving Universe to Weigh Cosmic Neutrinos

05

HPCI magazine FUGAKU HYAKKEI vol.7

Registered Institution for Facilities Use Promotion /The Representative for HPCI Operation

Research Organizationfor Information Science and Technology
1-5-2, Minatojimaminami-machi, Chuo-ku, Kobe, Hyogo, 650-0047 JAPAN
Tel: +81-78-599-9511

HPCI Portal Site

HPCI PR site

https://www.hpci-office.jp/folders/english

https://fugaku100kei.jp/e/Issued: February 2022
© 2022 Research Organization for Information Science and Technology