2023~2024

Security and Information Systems Department, Supercomputer DivisionJSS3 is a whole system of supercomputer systems supporting

Tsukuba Space Center【Computing Infrastructure】TOKI:TOkyo and ibaraKI

TOKI-TRURI Tsukuba General System

Chofu Aerospace Center

1 TOKI-SORA HPC System

TOKI-TLI Tsukuba Login System
TLI: Tsukuba LogIn system

TRURI: Tsukuba all-RoUnd Role Infrastructure

PRIMERGY RX2540 M5 x 2 nodes
5.98 TFLOPS = 2.99 TFLOPS/ node x 2 nodes

Peak Performance: 145 TFLOPS
Total Memory: 10.8 TiB
Total #Nodes: 49

SORA: Supercomputer for earth Observation, Rockets, and Aeronautics

FUJITSU PRIMEHPC FX1000
FUJITSU Processor A64FX

768 GiB = 384 GiB/ node x 32 node, Tesla V100 x 1

TOKI-TFS Tsukuba File System

TST PRIMERGY RX2540 M5

137 TFLOPS = 2.99 TFLOPS / node x 46 nodes
8.62 TiB =192 GiB / node x 46 nodes, Quadro x 1

TGP PRIMERGY CX2570 M5

Peak Performance: 19.4 PFLOPSThe HPC System (TOKI-SORA) is the core computer of
JSS3, which carries on the name "SORA" from JSS2.
It is a scalar-type parallel computer with the peak
performance of 19.4 PFLOPS.

the development of aerospace technology.

TFS: Tsukuba File System*

File System: FEFS
Total Memory: 0.4 PB

5.98 TFLOPS = 2.99 TFLOPS / node x 2 nodes
768 GiB = 384 GiB / node x 2 nodes, Tesla V100 x 4

Total Memory: 180 TiB(32GiB/ node)

TLM PRIMERGY RX2540 M5

2.99 TFLOPS = 2.99 TFLOPS / node x 1 node
1.50 TiB =1.50 TiB / node x 1 node, Quadro x 1

Total #Nodes: 5,760(15 racks)

Operation Management
SystemTsukuba Operation
Management System

InﬁniBand

High-speed Ethernet Backbone
InﬁniBand

2 TOKI-RURI

Water-cooling

General System

RURI: all-RoUnd Role Infrastructure

ST: STandard

PRIMERGY RX2540 M5
1,121 TFLOPS = 2.99 TFLOPS / node x 375 nodes
70 TiB =192 GiB / node x 375 nodes, Quadro x 1

GP: GPgpu

TOKI-SORA has adopted awater- cooling system that
efficiently removes the massive
heat generated by increasing
the computational performance.

【 Archiving Infrastructure】J-SPACE

5

J-SPACEJ-SPACE: Jaxa's Storage Platform for Archiving, Computing, and Exploring

Disk Cache: 3 PB
Tape: 70 PB*

PRIMERGY CX2570 M5
95.6 TFLOPS = 2.99 TFLOPS / node x 32 nodes
12.0 TiB = 384 GiB / node x 32 nodes, Tesla V100 x 4

The General System (TOKI-RURI) consists of ST node,
GP node, XM node and LM node in order to meet thevarious computing needs of users.

Peak Performance: 1.24 PFLOPS

Total Memory: 104 TiB

3 TOKI-LI Login System
LI: LogIn system

Powered by

XM: eXtra large Memory

PRIMERGY RX2540 M55.98 TFLOPS = 2.99 TFLOPS / node x 2 nodes
12.0 TiB = DCPMM 6.00 TiB / node x 2 nodes, Quadro x 1

LM: Large Memory

PRIMERGY RX2540 M5
20.9 TFLOPS = 2.99 TFLOPS / node x 7 nodes
10.5 TiB = DCPMM1.50 TiB / node x 7 nodes, Quadro x 1

4 TOKI-FS

File System

FS: File System

CMU（CPU Memory Unit)

1

2

3
5

4

PRIMERGY RX2540 M5 x 4 nodes

41.8 TFLOPS = 2.99 TFLOPS / node x 14 nodes

5.37 TiB = 384 GiB /node x 14 nodes, Quadro x1

2

File System: FEFS
All-Fash NVMe Storage: 10 PB
Hard Disk Drive Storage: 40 PB

Supercomputer Bldg. at the Chofu Aeropace Center

3

Start of JSS3 operation

JAXA is tackling technical issues in operation

JSS3 is the successor to JSS2.
Even if we install one of the most powerful supercomputer

High level of awareness of the technicalchallenges with supercomputers

Challenge

systems in the world today, its performance will be degraded

In order to solve the technical issues in the operation of

The circuit board in the supercomputer generates

in just a few years. As Japan's most advanced aerospaceresearch organization, JAXA always needs a supercomputer
system with the fastest computing power.

supercomputers, JAXA has cooperated with the
manufactures to apply technical improvements, andprovided the information necessary to users to run the

Supercomputer Division, Security and Information Systems

programs efficiently.

Cooling System

more heat as the density increases, but the conventional cooling system cannot handle the huge heat.
We have adopted water-cooling system with coldplates which removes more heat efficiently in JSS3.

Department has been operating JSS3 (JAXA Supercomputer
to JSS2, since December 2020. The name TOKI comes fromthe name of Japanese bird toki (crested ibis (Nipponia“solution” . TOKI also expresses “TOkyo and ibaraKI” , where
JSS3 is located.

HPC System (TOKI-SORA) is the mainframe of JSS3

Installation ofwater-cooled chiller

Power consumption per FLOPS has been drastically

Installation ofwater-cooled chiller

decreased with the progress of semiconductor
technology, but it is not enough to satisfy theever-increasing demand for processing power by thePrimary
cooling water

growing scale of the calculation. We have been

CDU

Rack

（Coolant Distribution Unit）

making every effort to reduce the power consumptionof the whole supercomputer system.

Supercomputer for further development of space and aircraft technology

Secondarycooling water

Power per 1 TFLOPS（Electricity per hour÷Peak Performance）

The HPC system (TOKI-SORA), the main computer of JSS3, is a computational system

JSS2

for large-scale numerical simulations, with 5,760 nodes of the PRIMEHPC FX1000
developed by Fujitsu. This supercomputer has a peak performance of 19.4 PFLOPS andJSS3

a total memory of 180 TiB.It will play a role as a high-performance computing (HPC)
platform to contribute to the enhancement of international competitiveness in theaerospace field.

▲Fujitsu Processer A64FX

(KW)
1000

0.34 KW
0.08 KW

Cold plate

Cool

The heat in CPU
is removed directlyHot
by circulatingcold-water in thecold plate.

3400 NWT（1996）
950KW/0.28TFLOPSCeNSS（2002）

General System (TOKI-RURI) with a choice of node types
A computer composed of four types of nodes
The programs that users calculate can be small or quite large,
depending on their research. The General System (TOKI-RURI)
consists of four different types of nodes, so the user can choose
the most efficient node for the program to be calculated.

Equipped with TOKI-RURI GP node for AI (artificial
intelligence) calculations

Various nodes of TOKI-RURI
ST nodeGP nodeLM nodethe Chofu Aerospace Center has 32 nodes, and TOKI-TRURI at
the Tsukuba Space Center has 2 nodes.
4

GPgpu

32 nodeseXtra large Memory1

JSS2（2016）
SORA-MA
1200KW/3.49PFLOPS

2000

2005

CMU（CPU Memory Unit）

JSS3（2021）
0.34

2010

2015

TOKI-SORA
1500KW/19.4PFLOPS

0.08
2020

▲Power per 1 TFLOPS of successive generationsof computers

Challenge

2 nodes

Large Memory

7 nodes

Program calculations with large memory

TOKI-RURIGPGPU is a technology that uses the GPU (graphics board) for

Each GPnode has four NVIDIA Tesla V100 GPUs. TOKI-RURI at

2030KW/135TFLOPSGPUGPnode

Swith

GPU
CPU

GPU
CPU

Chofu Aerospace Center

HPC System（TOKI-SORA）
General System（TOKI-RURI）
File System（TOKI-FS）

For researchers who use supercomputers forresearch throughout the year, the ideal system is
one that does not stop for maintenance during
the year.
JSS3 has the General System and the File

GPU

Swith

Uninterrupted Operation
Service

A system that can be used 365 days

Program calculations with extra large memory

GPGPU system that can be used for machine learning purposes.

the same calculation many times.

（2009）
15 JSS1

10

1995

Machine learning
XM node97 903KW/9.3TFLOPS

100

375 nodes

Normal program calculations

The General System of JSS3 is equipped with the GP node, anumerical calculations. It is suitable for calculations that repeat

Standard▲▲HPC System（TOKI-SORA)

nippon)), Japanese expression of “time and space” andPursuit for Efficiency

Challenge▲

System Generation 3), the successor supercomputer system

Either one isoperational.

System at each of its bases in Tokyo andTsukuba. Therefore, researchers can continue

Tsukuba Space Center

Tsukuba General System（TOKI-TRURI）
Tsukuba File System（TOKI-TFS）

their research even if one of the systems shuts
down its operation.

5

JSS3 has an advanced visualization system
Why “Visualization” is important
The data obtained by numerical simulations are processed bysupercomputer systems. These results are stored as numbers

Welcome to the JSS facility!

Supercomputer Observation Tours

HPC System（TOKI-SORA)

File System（TOKI-FS)

We have consistently offered Supercomputer Observation Tours to provide
an opportunity for individuals interested in understanding the supercomputer

Nice to meet you!

and they seem to us like a meaningless sequence of numbers.
In order to understand these data, they need to be shown on asystem to explore JSS3. Our tours feature a variety of exhibits, including
historical CPUs, posters showcasing numerical simulation results computed
using supercomputers, and various visualization items. However, for thosegraph, or displayed as a picture or moving image. We call thisprocess “visualization technology.” This technology is essential

residing outside of Japan or at a considerable distance, participating in these

Large Data

to pursue research using supercomputers.

TOKI-FS
20.8 Tbps

Necessity of a superior visualization systemdealing with bigger data

tours may not be feasible. As a result, we present a concise experience of

TOKI-SORA
12TbpsEnter the Supercomputer RoomInﬁniBandthe JSS3 tour for all of you.

Exhibition of the History of Supercomputers at JAXAInteresting!

TOKI-RURIAs the processing speed of supercomputers becomes faster,

45.7 Tbps

the output of calculation increases. The volume of output will
exceed the processing capacity of the existing visualization
system. JSS3 has been equipped with a special computer
system so as to perform large-scale calculations smoothly.

General System（TOKI-RURI)CPU Unit of JSS3

This is theSupercomputer
Room.

The system named “TOKI-RURI (General System),” which
is used for pre-processing for calculations and post-

Small Theater

processing for visualization of simulation results. Throughthe use of TOKI-RURI, the visualization procedure can be▲Wing tip vortex

Oh!

simplified. Namely, users can directly visualize the results
stored in a file system, TOKI-FS, without transferring the data

Visualizationfrom TOKI-FS to the users’ own computers.
JSS3 has adopted various applications for visualization such as

Numerical Simulation

Ensight, Fieldview and Paraview, and the visualization can be

TOKI-FS

performed under various conditions like remote desktop
function or client server.

▲Multiple turbines▲Launch pad

New technologies

Diversified 3D Visualization Devices
From 2D images to 3D images
Visualization of numerical simulation results was
mainly displayed on a 2D monitor, but with the▶4K autostereoscopic
displayevolution of 3D devices, calculation results can
now be confirmed in 3D.
The JSS3 Visualization Team is preparing new

Great system!
TOKI-SORA

TOKI-RURI◀MR device

Online Exhibition Hall

visualization technology using 4K auto-stereo-

For those eager to delve deeper into JSS3, we've curated

scopic display, MR device and 3D printer.

an online exhibition hall, offering a unique perspective on

See you!
▶3D models madeby using 3D printers

6various aspects of JSS3. Please access it using the linkprovided below.

https://www.jss.jaxa.jp/en/online-exhibit/

7

Development History of Computer System andNumerical Simulation in JAXAJAXA has a long history in research and development of computer

In 2003, JAXA was established by merging NAL, National Space

The installation of JSS3 as a new supercomputer system started insystem and numerical simulation. It began in 1960 with DATATRON

Development Agency (NASDA) and Institute of Space andmid-2020, and it has been in operation since December. The newmade in U.S. A., which the National Aerospace Laboratory (NAL), a

Astronautical Science (ISAS). Afterwards, the supercomputerssystem is called “TOKI”, and “TOKI-SORA”, which is configured

former organization of JAXA introduced as the first computer. Afterinstalled in each organization were put together into one system

as the main computer, has a theoretical computing performancean introduction of a domestic computer HITAC 5020, NAL andso as to enhance the research efficiency. In 2007, the integrated

of 19.4 PFLOPS and a total memory of 180 TiB. In addition,

Fujitsu started a joint development, and FACOM 230-75APU was

supercomputers started its operation as JAXA Supercomputer“TOKI-RURI” is available as a general-purpose system to enable

born as the first Japanese supercomputer in 1977.

System1 (JSS1).

researchers to use computing resources more efficiently.

In 1987, FACOM VP400, a special version of vector processing

The performance of JSS1 was updated in phases. In April 2009,

system for NAL, was developed for 3D CFD simulations based on

the peak performance of both M and P systems for the scalar partthe time-averaged Navier-Stokes equations. A full configuration

became 135 TFLOPS and the total memory size became 100 TB,

simulation around a whole aerospace vehicle was completed withinand since then JSS1 had been fully operational.

10-hour CPU time.

2021
2016
JSS2 SORA-MA

Subsequently, in October 2014, JAXA started to phase in a newsupercomputer system, JSS2, which consists of a main system

the same capability as the real wind tunnel experiment equipment. A“宙 (SORA)” and an archiver system, “J-SPACE.” In April 2016,

research and development project for NWT began in 1989 and it was

JSS2 was completed its performance update when “SORA-MA”

installed in February 1993. NWT behaved high performance and hadenhanced the capacity of the peak performance to 3.49 PFLOPS

served for national aerospace projects and fundamental research ofand the total memory size over 100 TiB.

1996
1993
NWT 140PE NWT 166PE 280 GFLOPS

fluid dynamics. From 1994 to 1996, NWT had won the Gordon Bell
Prize for its outstanding achievement for the third consecutive

3.49 PFLOPSIntel PARAGON 25.2 GFLOPS
1990
FACOM
VP2600
5 GFLOPS1987
FACOM VP400

years.
In 2002, NWT was replaced to a new system, Central Numerical
Simulation System (CeNSS), which had a Fujitsu-made

FACOM VP200 500 MIPS

performance in the development of next-stage numerical simulation
technologies.

1983
1977
FACOM230-75APU FACOM M380 FACOM M780 45 MIPS

1975
FACOM230-75

19.4 PFLOPSTOKI-RURI 1.24 PFLOPS

2009
JSS1 M System

120 TFLOPSSORA-PP 75.2 TFLOPS

V System 5.1 TFLOPS

CeViS SGI 0nyx 25 GFLOPS

1.14 GFLOPSPRIMEPOWER as a core machine. CeNSS also achieved impressive

2002
CeNSS 9.3 TFLOPS

JSS3 TOKI-SORA● 2003
CRAY Y-MP 334 MIPSFACOM VP2100 63.8 MIPS

20 MIPS

20 MFLOPS

1968
HITAC5020F 0.5 MIPS
1967
HITAC5020

1960

DATATRON205
1965

1970

1P

1T
Peocessor Speed

The next target was to develop Numerical Wind Tunnel (NWT) having

1960

1E

1G

1M

1K
1975

1980

1985

Pioneering

1990

1995

2000

Technology Development

2005

2010

Verification

2015

2020

2025

Practical use

ASKA

HOPE

SST

3-D Compressor

Combustion

Helicopter

Rocket Launch

Atomization

Rocket Engine

H3 Launch Vehicle

AircraftRocket Engine

Simulation of Full Configuration Experiment STOL
‘ASKA’

Themo-aerodynamic Designof Unmanned Reentry Vehicle
HOPE using GFD

CFD Analysis and Designof Next Generation SST

3-D Full Circle Compressor
Stage Simulation of Aerojet
EngineAnalysis of Turbulent
Combustion of Hydrogen

Numerial Simulation of
Helicopter Transient Flight

Sequential Photograhs ofPressure Wave Propagation
around LP1

Numerical Simulation of
Liquid Atomization
Processes

An End-to-end High Fidelity
Numerical Simulation of
the LE-X Engine

Arrangement of Rocket Engines
on Jet Flow inside Launch Padof H3 Launch Vehicle

Core Technology to Innovate
Aircraft Design and Operation

Large-eddy Simulation of a
Full-scale Liquid Rocket
Engine Combustor

1986

1993

1995

1995

2001

2003

2009

2011

2015

2018

2022

8

2006

9

Numerical Simulation of Aircraft
Research on Dynamic Stability Analysis

When obtaining a type certification for an aircraft, it is necessary to ensurethat the engine does not ingest a large amount of water spray generated
from the tires running on flooded runway. To predict the distribution of thewater spray, a particle simulation tool using MPS (Moving Particle
Simulation) method has been developed in JAXA. The tool is parallelized
via MPI technique, and the load balance is kept uniform using dynamic
domain decomposition method.

Aircraft dynamic stability is crucial for designing aircraft maneuver control systems andavoiding unstable flight motion, which prediction with textbook-based data or windtunnel tests sometimes prove challenging due to their limited information on detailed
unsteady flow fields. In this research, we have developed FaSTAR (Fast Aerodynamics Routines), a high-speed compressible flow solver that excels in providing both
highly accurate predictions of aircraft stability and substantial data productivity.

Experimental efforts to obtain in-house data are also being made tovalidate the simulation. The left figure is the photo of the quasi-full-scaletest, where the aircraft tire was attached to the end of the beams extending from a dolly running on a rail.

The figures presented here illustrate instances of unsteady flow over a Standard
Dynamics Model (SDM) undergoing oscillations in roll direction. To simulate theturbulent flow, we employed the Spalart-Allmaras-based Delayed Detached Eddy
Simulation (SA-DDES) method. The visualization of turbulent eddies allowed us tocapture the periodic separation occurring over the airfoil tip. We conducted quantitative
comparisons between the numerical and experimental results and confirmed that ournumerical method provides accurate predictions for aircraft's dynamic stability.

Iso-surface of Q criteria (color: Mach number)Research on Prediction Technologyof Water Spray Generated from
Aircraft Tire

The right figures are the simulation results of the test, where the metaball
and ray tracing techniques were utilized to render the result of the particle
simulation. The angle of the water spray viewed from the front side against
the ground was evaluated, and was confirmed to be consistent with thetest results.

Streamlines around the rotating aircraft

Research and Development of Core Technology
to Innovate Aircraft Design and Operation

Photo of the quasi-full-scale test of the water spray
Particle simulation result rendered withmeatball and ray tracing techniques

Aerodynamic Investigation of aMultiple-Rotor Drone in Ground
EffectLarger and heavier drones are being developed along with new trials to built
multiple-rotor type eVTOLs which can carry several people. However, the flowfields
around multiple rotors where the neighboring rotors are rotating in different directions
are very complex and not well understood. Espeicially when the multicopters arehovering near the ground, the so-called ground-effect is considered different with theconventional single rotor helicopters.

Computational grid around the aircraft
The development of new aircraft requires a high-speed and precise computational program which enables to reproduce the actual environment. To this end,
we have proceeded with the development of high-speed compressible flow
solver with geometric wall models and LES wall models based on thehigh-resolution hierarchical , orthogonal and equally spaced structured grids.

Computational model based on a prototype variable-pitch controlled quad-rotor droneis created. Flowfields and the drone performance are investigated for the drone
hovering at several different height from the ground. It is found that the flowfields forthe quad-rotor drone are much complex compared to those of a single rotor.

The computational grids near the aircraft body is finer than the outside. So we
can calculate with high accuracy around the high lift devices and landing gears.
The left images show the flow around the aircraft in flight at a seven-degree
attact angle|. We can see that the flow varies in complexities depending on the
shape of the aircraft.

10

Iso-surface of the vorticity (color: Velocity magnitude)Distribution of vortices around a multiple-rotor drone(volume rendering)11

Numerical Simulation of Spacecraft

Large-eddy simulation of afull-scale liquid rocket engine
combustor

Aiming to Develop Japanese New
Flagship Launch Vehicle, H3
Recently, many satellites that are closely related to our lives have beentransported to space, thus utilizing space has become part of our daily
lives. Under such a progressive society, H3 is aiming to become alaunch vehicle that attracts people’ s attention not only in Japan but
also globally as an easy-to-use space transportation system.
For H3 to succeed, JAXA will modernize the overall launch vehicle
based on our experience cultivated through the development andoperation of H-IIA. In that sense, we face technological challenges
including the development of a new large liquid engine (LE-9) and solid
rocket boosters (SRB-3). Technologies developed for H3 will beapplied to the Epsilon Launch Vehicle. JAXA and related companies
will make active use of Japanese technologies in various fields todevelop the new launch vehicle.

Aeroacoustic Simulationof H3 Launch Vehicle at Lift-off

In the development of a liquid rocket engine combustor, the risks of particularconcern are the oscillating combustion and erosion of the inner walls due toexcessive local heat load. These combustion anomalies are often observed
only after full-scale tests, and it is difficult to understand the phenomena
through measurement and visualization due to the high temperature andpressure environment. In order to reproduce and predict these unsteady
combustion phenomena, a compressible LES (Large Eddy Simulation) solver,
LS-FLOW-HO, has been developed.
To perform large-scale parallel computations on the order of 10 billion points,
LS-FLOW-HO employs the high-order Flux-Reconstruction method and the
low-cost Flamelet model. Compared to the conventional in-house solver, thecomputational cost of the LES of a combustor is dramatically reduced to lessthan 1/20. The overset grid approach is used to deal with more than 500
injectors on the full-scale combustor.

H3 Launch Vehicle Image

Liquid rocket engine combustor of LE-X engine

The analysis conditions are combustion pressure of 8.2 MPa and mixture
ratio of LOX to GH2 of 6.4 (O/F) for the LE-X engine : a technology demonstration engine for the development of the new LE-9. The total number ofcalculation points is about 2.6 billion. We used 960 nodes (48 cores/node) of
JSS3 TOKI-SORA (Fujitsu FX1000, 5760 nodes, 19.4 PFLOPS) . It took two
weeks to simulate 4.3 [msec] of physical time.
The instantaneous field during the transient (T~2.5 ms) of the flowfielddevelopment is shown in the figures. The pressure distribution is shown on
the injector and face-plate walls. Flame shapes are visualized by coloredisosurfaces of mixture fraction.
The coaxial flow of LOX and GH2 jetting from the injector forms turbulent
shear layer due to the large velocity ratio, and mixing and combustion occur.
The flame surface was observed to be shaken by the interference withneighboring jets and acoustic waves in the chamber.
Such thermoacoustic coupling between the acoustic modes specific to thefull-scale geometry and the turbulent diffusion flame formed by each injector,
is a phenomenon that can be reproduced only by full-scale simulation, and is
expected to be a means of predicting combustion oscillations.

Pressure fluctuations and flame deformation developed in the combustor
Enlarged image of flame near injector

12

Computational fluid dynamics (CFD) is applied to analyze generation andpropagation of acoustic wave generated from Japanese new flagship launchvehicle, H3, at lift-off. Exhaust jets of clustered liquid rocket engines andsolid boosters are visualized by volume rendering of the temperature field.
Acoustic field is shown by the pressure fluctuation, and it is found that the
acoustic wave returns to the launch vehicle.

13

Earth Observation Research Using Satellite Data
Contributing to Society Through Space-based Earth Observations
ALOS

Global Land-Cover Classification by
ALOS-2/PALSAR-2 etc.
■Purposes
Land-Use and Land-Cover (LULC) is one of the oldest remote sensing
applications and is essential as basic information for various applications.
There is an urgent need to develop algorithms that can cope with thediversity, high-resolution, and high-frequency of recent satellite data.
■Use Cases and SignificanceExamples of LULC use are summarized in the table on the bottom. Direct
and indirect contributions to many SDGs as fundamental information．

ALOS-2

ALOS-2(DAICHI-2): The Advanced Land Observing
Satellite-2 (ALOS-2, "DAICHI-2") is a follow-on mission
from the ALOS "Daichi". ALOS has contributed tocartography, regional observation, disaster monitoring,and resource surveys, since its launch in 2006. ALOS-2
succeeds this mission with enhanced capabilities.

■Analysis Methods and Results

2. JAXA satellite products such as ALOS-2/PALSAR-2,
GCOM-C/SGLI, GSMaP, and AW3D were used as input data.
Wide-area maps of LULC are developed by utilizing thecharacteristics of each satellite.

4. The final product was confirmed to have an overall accuracy of
86 % (Figure 1).

Figure 1: Final ProductOverall accuracy: 86%,
Number of categories: 16
Spatial resolution: 25 m
Time: June 2021 ~ May 2022

■Effects of JSS3 Utilization

Input data

2TB

Output data

100GB

Computation time

35.8 hours

Computing resources 10 nodes

JAXA LULC Map Application Examples
Radio propagation calculation

DisasterPrevention
Urban
Planning

Environment/
Biodiversity

Agriculture/
Public Health

Solar power plant siting location review
Simulation of river flooding
Survey of disaster damage and land-use change
Sediment runoff analysis
Urban planning status survey, flood control planningLand utilization, the current status of landscaping
Confirmation of embankment topography
Conservation of “SATOYAMA” forests
Environmental impact assessment survey
Understanding the location of bamboo groves
Extraction of forest change transition, logging sites, etc.
Landscape analysis and biodiversity analysis in ecology
Wildlife habitat mapping
Study of the relationship between bird communities
Habitat suitability assessment for endangered species
Prediction of crop pests and diseases
Creation of farmland maps
Risk mapping of sanitary pests

Climate Climate Change Impact Assessment
Change Input data for hydrologic and meteorological models
Exhibits at museums, photo galleries, etc.

Education Used for quiz shows, broadcast, and news coverage,
university lectures, etc.

14

Figure 1: Sea-surface temperature (Color) and horizontal velocity(vector) from LORA (LETKF-based Ocean Research Analysis) on
31st May 2023, showing southward large meandering south of Japanand northward Kuroshio overshooting along eastern coast of Japan.

1. Analysis ensemble mean and spread (3D)
2. All 128-member ensemble analyses at the sea surface (2D)
3. Analytical ocean mixed-layer heat and salinity budget terms (2D)

3. The output data consists of 16 categories, and we obtained atotal of 36,000 pieces of reference data.

Energy Stream flow calculation for hydropower dams

■Purposes
JAXA and RIKEN developed an ensemble
Kalman filter-based regional ocean data
assimilation system with satellite and in-situ
observations at a 1-day interval and createdensemble ocean analysis products in thewestern North Pacific and Maritime Continent
regions (LORA: LETKF-based Ocean Research
Analysis, Ohishi et al. 2022a, b, 2023). Here,
we used a high-performance computing
infrastructure of the JAXA Supercomputer
System 3 (JSS3) because the huge amount ofcomputation resource is required to integrate
the system.

This data assimilation system is unique in using frequent observations by Japanese geostationary satellites Himawari-8 and -9.
Compared with other existing datasets, LORA provides sufficiently accurate analyses for geoscience research and various
applications such as fisheries and marine transport with particular strength for Kuroshio and Kuroshio Extension in the
mid-latitude regions with large spatiotemporal variations. LORA is available online at JAXA-RIKEN Ocean Analysis webpage(https://www.eorc.jaxa.jp/ptree/LORA/index.html, Fig. 1) for the following variables (cf. Table 1):

1. We conducted a study to extend the CNN-based classification
method that has been developed for the Japanese region to a global
region using JSS. This time, we focused on the Asian region.

In this study, classification processing, which tookapproximately 215 hours on the existing Linux server,
was six times faster on JSS3 at 36 hours.
In the future, when expanding worldwide, JSS3 is expected
to be able to process data in about one month, whereas it isexpected to take about six months with the existing servers.

Development of an EnKF-based
ocean data assimilation system

Table: The computing resources.

The second can be used for atmospheric boundary
conditions and particle tracking, and the third is helpful forinvestigating spatiotemporal temperature and salinity
variations.
References:
Ohishi, Shun, Tsutomu Hihara, Hidenori Aiki, Joji Ishizaka, Yasumasa Miyazawa, Misako Kachi, andTakemasa Miyoshi, 2022: An ensemble Kalman filter system with the Stony Brook Parallel Ocean
Model v1.0, Geosci. Model Dev., 15, 8395–8410, doi:10.5194/gmd-15-8395-2022
Ohishi, Shun, Takemasa Miyoshi, and Misako Kachi, 2022: An ensemble Kalman filter-based ocean
data assimilation system improved by adaptive observation error inflation (AOEI), Geosci. Model Dev.,
15, 9057–9073, doi:10.5194/gmd-15-9057-2022
Ohishi, Shun, Takemasa Miyoshi, and Misako Kachi, 2023: LORA: A local ensemble transform Kalmanfilter-based ocean research analysis, Ocn. Dyn., 73, 117-143, doi:10.1007/s10236-023-01541-3

PeriodResolution
RegionOpen data

After August 2015 (About 8 years)
0.1°×50 σ-layers
western North Pacific [108°E-180°、12°-50°N]
Maritime Continent [95°-136°E、18°S-30°N]- Ensemble mean and spread- All 128-member analysis at the sea surface- Analytical heat and salinity budget terms

260 GB/year
Open data volume
Number of CPU cores 512
20 mins. for 1-day assimilation cycle
Computation time
Table 1: Overview of system setting, open data, and computation.

GOSAT 12 years observation data
has been processed by JSS3 every
time algorithm upgrade
GOSAT

GOSAT-2

The Greenhouse gases Observing SATellite "IBUKI"(GOSAT) is designed to measure the concentration of major
greenhouse gases from space.
GOSAT Project is a joint effort promoted by the Japan
Aerospace Exploration Agency (JAXA), the National Institutefor Environmental Studies (NIES) and the Ministry of theEnvironment (MOE).

Figure 1: Zonal mean CO2 density observed by GOSAT
between 2009 and 2019. A decade-long global GOSAT datashows annual increase of CO2 density that exceeded 400
ppm. Larger seasonal variations in the northern hemisphereindicate larger CO2 emission and stronger plant
photosynthesis in summer.

Input data

83TB (for 12 years)

Output data

35TB (3.5M products)

Computing resources 30 nodes (360cores)
Computing time

30 daysFigure 2: GOSAT Methane products
Partial column difference(lower - upper troposphere)(2019,California)

Figure 3: GOSAT CO2 products
Partial column difference(lower - upper troposphere)(2019,California)

Red and blue dots show positive and negative enhancements,
respectively.Positive enhancement suggests large local emissions
from the surface.
Table 1 : The computing resources.

15

More information about JSS3
You can get more information about JSS3
via this website.
We publish more information about JSS3. Please visit thewebsite to touch our supercomputer system!
You can see the configuration, history, outcomes, and somepublications about JSS. In addition to them, you can also
learn how to use JSS3.

https://www.jss.jaxa.jp/en/

Japan Aerospace Exploration Agency
Supercomputer Division

Chofu Aerospace Center

7-44-1 Jindaiji Higashi-machi, Chofu-shi,
Tokyo 182-8522

Oct. 2023