Our model development activity
in RIKEN AICS
- introduction of SCALE and SCALE-LESS. Nishizawa, H. Yashiro, Y. Sato, T. Yamaura, S. Aadachi, R.
Yoshida, H. Tomita and Team SCALE
RIKEN Advanced Institute for Computational ScienceIntroduction• Meteorological simulation is not a first-principle simulation• based on many empirical rules.
• tones of tunable switches• Validity of simulations can be hardly confirmed especially• for paleo/future climate, or other planets• with higher resolution

Comparison plays important role for reliability of models.
Diversity of meteorological models is really important.
• Meteorological simulation models are getting complex more and more.
• Each process is more sophisticated• Number of included processes have become increasing• Programing code have become complex due to complex computer systems

Limitation of human and temporal resources is serious• As a possible solution of the problems, we have develop aframework (library and environment) to develop models.
• for model developers• Meteorological scientists could focus on physical performance• Computational issues are mainly handled by the framework• unify APIs
• switching schemes or exchange schemes between models could
become easier• comparison could become easier

SCALESCALE (Scalable Computing for Advanced Library and Environment)• from x86 PCs to next generation super computers• Collaborate with computer scientists• Open source with the 2 clause BSD license• http://scale.aics.riken.jp/• Components• model components (unified API)• dynamical cores: HE-VE, HE-VI, HI-VI• physics: microphysics, turbulence, radiation, surface flux
• misc: I/O, communication, logger, timer etc• documents• model description• knowledge repository (future work)• test cases• component unit test
• standard benchmarks• pre- and post-processing tools (future work)• parallel handling

We started to collaborate with other model developer groups(JMA/MRI, MIROC, NICAM, CReSS, AFES, MSSG, GAIA, MATSIRO,DENNOU)

SCALE-LES
A meteorological Large-Eddy simulation model using SCALE• An application of SCALE• To perform wide domain and high resolution simulation• O(10-100)km2 domain with O(10-100)m resolution; mesoscale
LESSCALE-LES model description• Prognostic variables:• density, momentums, mass-weighted potential temperature, massconcentration of tracers• Dynamical core:• governing equation: fully compressive equation• temporal integration:• full explicit (HE-VE) and implicit (HE-VI, HI-VI) schemes• 3 steps RK scheme• spatial difference:• 4th order central difference for advection• 2nd order central difference for terms related with acoustic wave

• grid : Arakawa-C grid
• topography: terrain following, thin wall (testing)• tracer advection: CWC + FCT• numerical filter: 4th or 8th order hyper diffusion• Physical processes• Microphysics•••••

single moment 3 category bulk (Kessler 1969)
single moment 6 category bulk (Tomita 2008)
double moment 6 category bulk (Seiki and Nakajima 2014)bin (Suzuki et al. 2010)super droplet (Shima 2009; testing)• Turbulence• Smagorinsky-Lilly type SGS turbulence• Mellar-Yamada Nakanishi-Nino level 2.5 (Nakanishi and Niino 2009)• Radiation• MstrnX (Sekiguchi and Nakajima 2008)• Surface flux
• Louise type (Uno et al. 1995)• Beljaars (1994)• Land model• multilayer becket model• Urban canopy• single layer canopy (Kusaka 2010)

computational performance• performance @ K computer• above 10% of peak performance (dynamical core)• 5~8% for whole simulation (including I/O)

• about 100% weak scaling to full system (663,552 cores)

4.0

8

3.5

7

3.0

6

2.5

5

2.0

4

1.5

31.0

20.5

10.0

Performance efficinecy [%]Elapsed time [sec/step]

690TFLOPS

016

2048

8192

32768

131072663552

Number of cores
elapse timeefficiency• For efficient development• Using git for version controlling• Using redmine for project management• Using CI for early finding bugs
• Run all the tests at each commit automaticallyDynamical core test
• Density current test case (Straka et al. 1993)
Straka et al. (1993)

SCALE-LES (100m resolution)

• density current without physical diffusion/viscosity• but with numerical diffusion• 51.2 km x 6.4 km 2-D domain• 1.5625 m resolution (134M grids)• 900 sec integration

The bigger spiral structure still remain
due to absent of the smaller scale instability.
It does not mixed well.

This implies thathigher resolution experiment is not always better than lower one
without appropriate treatment (parameterization).

Cloud microphysics and turbulence test
Temporal evolution

800

30600

2030

020050100150

1200

0

Heig nt [m ]H eig nt [m ]Heig nt [m ]

600

400

200

Total water
mixing ratio
4

5

60050

100

8

910600Total water
mixing ratio
Liquid water
mixing ratio

200
1203

4

5678

9100.05

0.1

0.15

0.2

200

0 200
00.25

0.3

0.35

01200 00.4

0.05

0.1

0.15

0.2

0.1

0.2

0.3

Mixing ratio of water contents.

11.2

1.4

1.6

1.82GCSS(mean)

GCSS(sigma)

SCALE(SS)0.35

29629830030230430

Heig ht [m ]

0.2

0.4

0.6

0.8

11.2

00.05

0.1

0.15

0.2

GCSS(sigma)

GCSS max min
GCSS std
GCSS mean
SCALE

600
800

400
600

0200
00.4

0.1

0.2

1.4

1.6

1.8

0.4
0.5
GCSS max min
GCSS std
GCSS mean
SCALE

0.25

0.3

0.35

0.4

SCALE(SS)

0-60

0.5

0.6

-0.1

-0.05

00.05

0.1

0.15

GCSS max min
Third moment of w’[m3/s3]
GCSS std
GCSS meanSCALEGCSS max min
GCSS std
GCSS mean
SCALE

800
1000

600
800

400600200
4000.05

0.1-500.15

0.2

0.25

0.30.35w'θl'0.40200-60-40-30-20-50-40-30-10w’theta’[W /m2]

010200-200-20-100

1020

w’theta’[W /m2]

w'q'sgs TKE [m2/s2]

w'θl'GCSS(mean)

0.4

1000
1200

SGS TKE
0200
00-0.1512000.6

400
600

20.3
Variance of w’[m/s]

200
400

sgs TKE [m2/s2]
GCSS(max-min)

GCSS(max-min)

0.3

600
800

00.8

294

Liquid water potential temperature [K]
GCSS max min
G CSS std
GCSS mean
SCALE

Third moment of w'

GCSS max min
GCSS std
GCSS mean
SCALE

8001000

SGS TKE

Grid resolved TKE [m2/s2]

0.25

1000
1200

Grid resolved TKE [m2/s2]

00.6

200

GCSS max min
GCSS std
GCSS mean
SCALE

200
400

Variance of w’[m/s]

GCSS max min
GCSS std
GCSS meanSCALE

GR TKE
0.4

150

800
1000

Liquid watermixing ratio

GR TKE
0.2

292

Liquid water mixing ratio [g/kg]

400 600

200 400

GCSS max min
G CSS std
GCSS mean
SCALE

Variance of w'

0200 011GCSSmax min
Liquid water mixing ratio[g/kg]
GCSS mean
GCSS mean
SCALE

40002901200

400600Heig nt [m ]Heig ht [m ]

600

100

1000
1200200
400600 800

Heig nt [m ]8000288200

GCSS max min
GCSS std
GCSS mean
SCALE

600
800

400

8001000

1000

150

Time [min]

800
1000

800

1200

GCSS max min
GCSS mean
GCSS mean
SCALE

50

Time [min]

1000
1200

GCSS max min
GCSS std
GCSS mean
SCALE

10001200

1200

0200

Hourly averaged profile (last 1 hour)
Variance of w'011Zi810

200

Time [min]Total water mixing ratio [g/kg]

7150810200GCSS max min
GCSS std
GCSS mean
SCALE

Total water mixing ratio [g/kg]

Heig ht [m ]1001200

8001000

400315010001200

600210050Zi

820Time [min]8001501200

1000002000200

GCSS max min
GCSS std
GCSS mean
SCALE

200

150

Tim e[m in]

Tim e[m in]

600
830

Heig ht [m ]

100

800
840

Heig ht [m ]

50400
850

0H eig nt [m ]

0850

830

100600860

820 400

LWP0870

840
200

400

1020

Liquid Water
Potential Temperature

8601000
Zi [m ]

401200870

GCSS max min
G CSS std
GCSS mean
SCALE880
800

Heig nt [m ]4050 1000

1000890

GCSS max min
G CSS std
GCSS mean
SCALE

880

Heig ht[m ]50

GCSS max min
GCSS std
GCSS m ean
SCALE

Vertically integrated
TKE

GCSS max min
GCSS std
GCSS m ean
SCALE

Vertically integrated
TKE

H eig nt [m ]

60

1200

60

Heig ht[m ]Liquid water path [g /m 2]

Liquid water path [g /m 2]

70

Time evolution

Time evolution

GCSS m ax min
GCSS std
GCSS mean
SCALE

890

1200

GCSS m ax min
GCSS std
GCSS mean
SCALE

70

Vertically integ ra ted TKE (SG S+G R )[m 2/s2]80

80

Verticallyra ted TKE (SG S+G R )[m 2/s2]
Zi [m integ]

DYCOMS-II RF01 case (Stratocumulus without rain)
Experimental setup is based on Stevens et al. (2005)• Domain size: 3.36km x 3.36km x 1.5 km(3D)• Resolution: dx=dy=35, dz=5m• Calculation time : 4 hour (dt=0.006s)• Cloud physics : 2-moment bulk [without rain andLWP
sedimentation]• Radiation : Parameterization of Stevens et al.
(2005)• Surface flux : Constant value

20406080

w’qt’[W /m2]

100120140160Current science targets with SCALE-LES• Shallow clouds• open/close stratocumulus• Very short range heavy rain forecast in Kansai (talk by Dr. Miyoshi)• big data assimilation• Future heavy rainfall in Kobe/Hyogo• Martian planetary boundary layer

Summary• Develop a framework for meteorological model development• Meteorologists could focus on physical performance againstcomplex future computer systems.
• We try to achieve well computational performance by R&D withcomputer scientists.
• Collaboration with other model developing groups would beenhanced.

• Wide-domain high-resolution simulations (mesoscale LES)• SCALE-LES is developed as an application of SCALE.
• Mesoscale LESs are now in progress.
• For future global LES (with icosahedral grid system)