Big Data Assimilationin Weather Prediction:
From K to Fugaku
Takemasa Miyoshi

Ph.D. (Meteorology)
Data Assimilation Scientist

Data Assimilation Research TeamRIKEN

Only in 10 minutes!!

Kobe newspaper (https://www.kobe-np.co.jp/news/kobe/201807/0011486822.shtml)

1.34 min 10 minutes!!

5 people died in River Toga in Kobeon July 28, 2008

Phased Array Weather Radar (PAWR)(Courtesy of NICT)

100x

3-dim measurement using
a parabolic antenna (150 m,
15 EL angles in 5 min)data size

3-dim measurement using a phased array antenna(100 m, 100 EL angles in 30 sec)

3

New

radartechnology

Big Data Assimilation
Observations

100x Big Data
New sensors, IoT

Simulations

Big Data

100xPowerful supercomputer

Data Assimilation (DA)
Simulations

Observations

1

Data Assimilation+>2

1

Data Assimilation (DA)
Data-driven
Induction
Real world

Process-driven
Deduction
Cyber world

Observations

Simulations

1

Data Assimilation+>2

1

DA workflow(Best estimate)

Initial State

Process-driven

Simulation
Simulated StateDA

Sim-to-Obs
conversion

DA

Sim-minus-Obs

Data-driven
Observations

DA workflow(Best estimate)

Initial State
Simulation
Simulated StateDASim-to-Obs
conversion

DA

Sim-minus-Obs

Process-driven
Human knowledge
ScienceData-driven
Observations

DA workflow(Best estimate)

Initial State

Process-driven

Simulation
Simulated StateDA

Sim-to-Obs
conversion

Data-driven
Observations

Sim-minus-Obs
Broad-sense DA

DA = math of errors
forecast0obs
DAanalysis1

Merging 2 information (Bayesian estimation)p

Probability distribution
is essential.

Fcst
Merging
Fcst&Obs

Obs

9/11/2014, sudden local rain

Youtube: https://www.youtube.com/watch?v=42NZTGdp1Js

9/11/2014, sudden local rain

>41,000 views#6 of RIKEN channel

Youtube: https://www.youtube.com/watch?v=42NZTGdp1Js

Big Ensemble
DA

Miyoshi, Kondo, Terasaki(2014, Computer)doi:10.1109/MC.2015.332

Sample size = Resolution of probability
20

640

1.856N, 176.25E

40

1280

80

2560

160

5120

320

10240

Kondo&Miyoshi(2019, NPG)
doi:10.5194/npg-26-211-2019

20

Non-Gaussian metric (KLD)
640

40

1280

80

2560

160

5120

320

10240

Kondo&Miyoshi(2019, NPG)
doi:10.5194/npg-26-211-2019

2040

80

Non-Gaussian metric (KLD)
640

1280

Non-Guassian PDF captured with>1000
2560

1605120320

10240

Kondo&Miyoshi(2019, NPG)
doi:10.5194/npg-26-211-2019

Improving NWP
1-day fcst error = 5-day fcst error
80 members

10240 members

Kondo and Miyoshi (2016, MWR,
doi:10.1175/MWR-D-15-0388.1)

Pushing the limits
Big Data × Big Simulations
Big ensemble (10240 ensemble members)
Rapid update (30-second update)
High resolution (100-m mesh) Future Numerical Weather Prediction

Fugaku
Global 3.5-km mesh1000 samples

K
Global 112-km mesh
10240 samples

Fugaku : K = 100 : 1
Mesh size: 32x(Grid points: 1024x)Sample size: 0.1x

Toward Weather-Ready Society5.0 withsynchronize
predict & control

Cyberspace
Simulation

Big Data

Real world
SensingNatureBig Data Assimilation
110101111010111101011

Prepared for weather
IoT

AI x HPC

Human society and economy

Toward Weather-Ready Society5.0 withsynchronize
predict & control

Cyberspace
Simulation

Big Data

Real world
SensingNatureBig Ensemble to Warn
• early & accurately• without missAI x HPC
Effective
disaster prevention
Big Data Assimilation

110101111010111101011

Prepared for weather

IoT

Human society and economy

DA-AI Integration(Best estimate)Initial State
Simulation

DA algorithm
Simulated StateDA

Sim-to-Obs
conversion

Predict high-resolution
from low-resolution model
Predict model error
Surrogate model
Need to learn big
computation data on
HPC (cannot move)
Model-obs relationshipObservations
Quality control

Sim-minus-Obs
Broad-sense DAPAWR 3D Nowcasting
INPUT

Observation
Weather Simulation

OUTPUT

Parallel test of real-time
nowcasting w/ ConvLSTM
since June 6, 2019!!
3D Precip. Nowcasting
Improved forecast

No input of future dataat this moment

Conv-LSTM is effective.
(Work with Mr. Viet Phi Huynh and Prof. Pierre Tandeo)

2.5-min prediction

Future direction: Fusing ML+DA+Simulation
INPUT

Observation
Weather Simulation

OUTPUT

New 3D Precip. Nowcasting
Improved forecast

Input of future datafrom NWP!!

DA-AI fusion(Best estimate)

Predict high-resolution
from low-resolution model

Initial StateSimulation

DA algorithm

Predict model error
Surrogate model
Need to learn big
computation data on
HPC (cannot move)

Simulated State
Integrating
DA and AI

Model-obs relationship Pioneering new meteorology

DA

Sim-to-Obs
conversion

Observations
Quality control

Sim-minus-Obs
Broad-sense DA