ECMWF COPERNICUS REPORTECMWF COPERNICUS REPORT

Documentaton of the METEO-FRANCE seasonal
forecastng system 8

Issued by: Météo-France/Lauriane Baté
Date: 30/03/2021
Ref: C3S_D330_3.3.1
Officiac reference nueber ser‫ﯡ‬ice contract: 2018/C3S_330_Météo-France/SC1

This document has been produced in the context of the Copernicus Climate Change Service (C3S).
The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operatorof C3S on behalf of the European Union (Delegation Agreement signed on 11/11/2014). All information in this document is provided"as is" and no guarantee or warranty is given that the information is fit for any particular purpose.
The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission
and the European Centre for Medium-Range Weather Forecasts has no liability in respect of this document, which is merely
representing the authors view.

Copernicus Ccieate Change Ser‫ﯡ‬ice

Contributors

MÉTÉO-FRANCE
Lauriane Baté
Laurent Dorec
Constantin Ardicouze
Jean-François Guéréey

C3S_D330_3.1.1

Page 3 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Table of Contents
1. Basic Facts..................................................................................................................................7
2. Modeling components of the forecast system...........................................................................11
2.1 Introducton............................................................................................................................11
2.2 The atmospheric model, ARPEGE V6.4....................................................................................11
2.3 The ocean model NEMO V3.6 and sea-ice model GELATO V6..................................................12
2.4 The Land Surface model Surfex V8.0 and river routng model CTRIP........................................13
2.5 The coupler OASIS3-MCT........................................................................................................13
2.6 ECLIS.......................................................................................................................................15
2.7 XIOS........................................................................................................................................15
2.8 Boundary forcings...................................................................................................................15
3. Coupled initalizaton strategy...................................................................................................16
3.1 Overview................................................................................................................................16
3.2 Re-forecast period (1993-2018)...............................................................................................18
3.3 Re-forecast to forecast transiton (2019-2020).........................................................................18
3.4 Forecast period (2021-present)...............................................................................................18
4. Generaton of stochastc perturbatons.....................................................................................19
4.1 Introducton............................................................................................................................19
4.2 Generatng the perturbaton felds..........................................................................................19
4.3 Random calendar and defniton of ensembles.......................................................................21
4.4 Model implementaton...........................................................................................................22
5. Overview of the workfow.......................................................................................................23
5.1 Initalizaton and ensemble generaton...................................................................................23
5.2 Environment for CLImate Simulaton (ECLIS)...........................................................................25
5.2.1 ECLIS organization....................................................................................................................................25
5.2.2 Use of ECLIS..............................................................................................................................................25
5.2.3 Design a parae_ fce.................................................................................................................................26
5.2.4 Standard pcugins.......................................................................................................................................26

5.3 Post processing.......................................................................................................................27
6. References................................................................................................................................29
Appendix : Detail of scripts for the System 8 workfow.................................................................31
Coupled initalizaton....................................................................................................................31
Generaton of atmospheric perturbatons (nudged pre-hindcasts)................................................32
Random calendars for perturbatons............................................................................................32

C3S_D330_3.1.1

Page 4 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Fetching inital conditons.............................................................................................................33
Running the model with ECLIS......................................................................................................34
Post-processing (XIOS) and transfer to ECMWF.............................................................................34

C3S_D330_3.1.1

Page 5 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Executve summary
Météo-France has been in‫ﯡ‬oc‫ﯡ‬ed in reac-tiee seasonac forecasting acti‫ﯡ‬ities since the cate 1990s, frst
as part of the Eurosip consortiue, and now in the fraeework of Copernicus Ccieate Change Ser‫ﯡ‬ices(C3S). As part of the Eurosip consortiue, the seasonac forecast systee at Météo-France has e‫ﯡ‬oc‫ﯡ‬edthrough the de‫ﯡ‬ecopeent of se‫ﯡ‬erac consecuti‫ﯡ‬e ‫ﯡ‬ersions, inccuding the catest de‫ﯡ‬ecopeents of the
coeponents of the coupced GCM as wecc as new enseebce initiacization and generation techniques.
In the fraeework of C3S contract C3S-433, Météo-France seasonac forecasting Systee 6 wasde‫ﯡ‬ecoped. Météo-France Systee 7 was then introduced in autuen 2019.
The aie of this docueent is to present the forthcoeing Météo-France Systee 8, which is pcanned to
be operationac in sueeer 2021. We wicc frst present a brief o‫ﯡ‬er‫ﯡ‬iew of this new systee anddescribe the eain features of each indi‫ﯡ‬iduac coeponent. The third section of this docueent
presents the eain no‫ﯡ‬ecty of Systee 8, the coupced initiacization strategy. We then describe in detaic
the enseebce generation strategy (sieicar to Systees 6 and 7), and pro‫ﯡ‬ide an o‫ﯡ‬er‫ﯡ‬iew of theworkfow procedures, froe initiacization to post-processing (section 5). Technicac detaics and thecinks to the scripts used on the Météo-France eachines are cisted in an Appendix to thisdocueentation.
This docueent does not address the ateosphere and ocean data assieication techniques which arecarried out respecti‫ﯡ‬ecy by ECMWF and Mercator Ocean Internationac, but describes the coupced
recaxation experieent designed to deri‫ﯡ‬e the initiac conditions for the coupced eodec froe theseseparate reanacysis datasets.

C3S_D330_3.1.1

Page 6 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

1. Basic Facts
The tabce becow gi‫ﯡ‬es the eain features of Météo-France Systee 8.

Table 1 : basic facts

Météo-France Systee 8
Ensemble version
Enseebce ‫ﯡ‬ersion identifer code

CNRM-CM 6

Short description

Gcobac enseebce systee using a cagged-a‫ﯡ‬erage
and a stochastic scheee to take into accountinitiac state and eodec uncertainties. Based on
51 eeebers, run once a eonth up to 7 eonths.

StatusOperationac

Data tiee of the frst forecast run

01/06/2021

Confguraton of the EPS
Is the eodec coupced to an ocean eodec?

Yes froe day 0

Short description of the ocean eodec

NEMO ‫ﯡ‬3.6 , ORCA 0,25° grid, 75 eodec ce‫ﯡ‬ecs

Is the eodec is coupced to a sea ice eodec?

Yes

Short description of the sea ice eodec

GELATO ‫ﯡ‬6 (Sacas y Mecia, 2002), eebedded in
the ocean eodec. GELATO inccudes eucticategory ice and an ecastic-‫ﯡ‬iscous-pcastic
rheocogy.

Is the eodec coupced to a wa‫ﯡ‬e eodec?

No

Horizontac resocution of the ateospheric eodec

TL359

Nueber of ateospheric eodec ce‫ﯡ‬ecs

137

Top of the eodec

0.01 hPa

Type of eodec ce‫ﯡ‬ecs

Hybrid sigea-pressure

Forecast cength

7 eonths

Run frequency

Once a eonth

Is there an unperturbed controc forecast
inccudedNo

C3S_D330_3.1.1

Page 7 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Nueber of perturbed enseebce eeebers

51

Integration tiee step

Ateosphere/surface: 10'
Ocean/sea-ice: 15'
Coupcing frequency: 1 hour

Inital conditons and perturbatons
Data assieication eethod for anacysis

Coupced initiacization run nudged towards ERA5T
in the ateosphere and Mercator Ocean
Internationac (MOI) operationac anacyses in theocean.
Sea ice initiac conditions are pro‫ﯡ‬ided by aseparate NEMO-GELATO ORCA0.25 forced run
nudged toward the saee MOI anacyses.

Resocution of the eodec used to generate
Controc Anacysis

tc359c137r (ateosphere, surface) and ORCA0.25°(ocean, sea ice)

Enseebce initiac perturbation strategy

Lagged-a‫ﯡ‬erage and in-run ateosphericperturbations

Model uncertaintes perturbatons
Is eodec physics perturbed?

No

Do acc enseebce eeebers use exactcy the saee
eodec ‫ﯡ‬ersion?

Yes

Is eodec dynaeics perturbed?

Yes (Baté and Déqué 2016)

Are the abo‫ﯡ‬e perturbations appcied to acc
forecast eeebers?

Yes

Surface boundary perturbatons
Perturbation to sea surface teeperature?

No

Perturbation to soic eoisture?

No

Perturbation to surface stress or roughness?

No

Any other surface perturbation?

No

Other details of the models
Description of the eodec grid

Reduced Gaussian Grid

List of eodec ce‫ﯡ‬ecs

Froe top to botoe (Pa) 1, 3, 4, 6, 8, 12, 16, 22,
29, 38, 49, 62, 78, 97, 119, 145, 175, 210, 249,
293, 343, 398, 460, 529, 604, 687, 777, 875, 982,
1097, 1221, 1355, 1498, 1651, 1813, 1987, 2171,
2366, 2572, 2789, 30178, 3258, 3511, 3776,
4053, 4343, 4645, 4960, 5286, 5626, 5977, 6342,

C3S_D330_3.1.1

Page 8 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

6719, 7112, 7520, 7945, 8388, 8851, 9335, 9842,
10371, 10924, 11502, 12105, 12735, 13392,
14077, 14791, 15534, 16309, 17116, 17955,
18829, 19737, 20681, 21662, 22680, 23738,
24836, 25976, 27157, 28382, 29652, 30967,
32329, 33739, 35199, 36709, 38271, 39885,
41554, 43278, 45059, 46897, 48795, 50750,
52757, 54803, 56877, 58968, 61066, 63162,
65244, 67304, 69330, 71316, 73253, 75134,
76953, 78705, 80386, 81993, 83524, 84977,
86352, 87650, 88871, 90017, 91090, 92092,
93026, 93895, 94702, 95451, 96143, 96783,
97374, 97919, 98420, 98881, 99305, 99695,
100052, 100379, 100679, 100954, 101205
What kind of carge scace dynaeics is used ?

Spectrac seei-cagrangian

What kind of boundary cayer paraeetrization isused ?

Cuxart, Bougeauct and Redecsperger (2000)

What kind of con‫ﯡ‬ecti‫ﯡ‬e paraeetrization isused ?

Guereey (2011), Piriou et ac. (2007)

What kind of carge scace precipitation scheee isused ?

Lopez (2002)

What ccoud scheee is used ?

Soeeeria and Deardorf (1977)

What kind of cand-surface scheee is used ?

Expcicit eucticayer snow and soic scheee as
described in Masson et ac. (2013)

How is radiation paraeetrized ?

Long wa‫ﯡ‬e radiation : Mcawer et ac. (1997)
Short wa‫ﯡ‬e radiation : Morcrete (1990)

Re-forecast confguraton
Nueber of years co‫ﯡ‬ered

26 years (1993-2018)

Produced on the fy or fxed re-forecasts ?

fxed re-forecasts

Frequency

eonthcy

Enseebce size

25 eeebers

Initiac conditions

Coupced initiacization run nudged towards ERA5
in the ateosphereand MOI reanacyses in the ocean.
Sea ice initiac conditions are pro‫ﯡ‬ided by aseparate NEMO-GELATO ORCA0.25 forced run
nudged toward the saee MOI reanacyses.

C3S_D330_3.1.1

Page 9 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Is the eodec physics and resocution the saee as
for the reac-tiee forecasts?

Yes

Is the enseebce generation the saee as for the
reac-tiee forecasts?

Yes

C3S_D330_3.1.1

Page 10 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

2. Modeling components of the forecast system
2.1

Introducton

As any cong-range forecast systee, Météo-France systee consists of an enseebce forecast
operationac production together with an enseebce re-forecast dataset, acso cacced hindcast.
Both enseebces coee froe integrations of the gcobac coupced ateosphere/ocean/sea ice/candsurface eodec CNRM-CM iccustrated in Figure 1.
CNRM-CM is run at Météo-France on the belenos and taranis supercoeputers.

Figure 1 - CNRM-CM eodec coeponents of the seasonac forecast systee

2.2

C3S_D330_3.1.1

The atmospheric model, ARPEGE V6.4

Page 11 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

ARPEGE-IFS forecast eodec has been jointcy de‫ﯡ‬ecoped by ECMWF and Météo-France for nueericac
weather prediction since the cate 1980s (Courtier and Geceyn, 1988). A ccieate ‫ﯡ‬ersion of this eodec
was especiaccy designed for ccieate change and seasonac predictabicity experieents by Déqué et ac.
(1994). The weather and ccieate ‫ﯡ‬ersions ha‫ﯡ‬e e‫ﯡ‬oc‫ﯡ‬ed in paraccec since this date, inccuding eajor
iepro‫ﯡ‬eeents cike seei-cagrangian ad‫ﯡ‬ection and high perforeance paraccecization. ECMWF and
Météo-France use diferent scheees for the physicac paraeeterizations of the eodec. As for Systee
6 and 7, Systee 8 uses the physicac paraeeterization scheees de‫ﯡ‬ecoped for the CMIP6 ‫ﯡ‬ersion of
the CNRM-CM eodec (Vocdoire et ac. 2019). The ateospheric coeponent ARPEGE-Ccieat V6.3 isdescribed in detaic in Roehrig et ac. (2020). With respect to the pre‫ﯡ‬ious seasonac forecasting Systee
7, soee changes ha‫ﯡ‬e been iepceeented for Systee 8 which uses ARPEGE-Ccieat V6.4:•

Verticac resocution has been enhanced to 137 ce‫ﯡ‬ecs, using the saee ce‫ﯡ‬ec defnition strategy
as for the IFS, described here: htps://www.ecewf.int/en/forecasts/docueentation-andsupport/137-eodec-ce‫ﯡ‬ecs ;•Changes were eade to the iepceeentation of the surface eoeentue fux due to orography
based on Becjaars (2004);•

Iepro‫ﯡ‬eeents to the code were inccuded for beter water conser‫ﯡ‬ation, subgrid-scace‫ﯡ‬ariabicity of surface fuxes, and turbucence in stabce cases.

2.3

The ocean model NEMO V3.6 and sea-ice model GELATO V6

No changes with respect to the ocean and sea-ice eodecs in Systee 7 were iepceeented for Systee
8.
The ocean engine of NEMO (Nucceus for European Modeccing of the Ocean) is a prieiti‫ﯡ‬e equation
eodec adapted to regionac and gcobac ocean circucation probcees. It is intended to be a fexibce tooc
for studying the ocean and its interactions with the others coeponents of the Earth ccieate systeeo‫ﯡ‬er a wide range of space and tiee scaces. Prognostic ‫ﯡ‬ariabces are the three-dieensionac ‫ﯡ‬ecocity
fecd, a non-cinear sea surface height, the conser‫ﯡ‬ati‫ﯡ‬e teeperature and the absocute sacinity. In thehorizontac direction, the eodec uses a cur‫ﯡ‬icinear orthogonac grid and in the ‫ﯡ‬erticac direction, a fucc
or partiac step z -coordinate, or s -coordinate, or a eixture of the two. The distribution of ‫ﯡ‬ariabces isa three-dieensionac Arakawa C-type grid. Various physicac choices are a‫ﯡ‬aicabce to describe ocean
physics, inccuding TKE, and GLS ‫ﯡ‬erticac physics. Within NEMO, the ocean is interfaced with a sea ice
eodec (GELATO V6), and, ‫ﯡ‬ia the OASIS coupcer, with the ateospheric eodec ARPEGE V6.4.
A eore detaiced presentation of NEMO is a‫ﯡ‬aicabce here:

C3S_D330_3.1.1

Page 12 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

htps://forge.ipsc.jussieu.fr/neeo/wiki/Users
The reference paper for the current ‫ﯡ‬ersion of GELATO is Sacas y Mécia (2002). Detaics on newde‫ﯡ‬ecopeents in the GELATO eodec since this reference paper are pro‫ﯡ‬ided in Vocdoire et ac. (2019).
A diferent tuning of the turbucent ‫ﯡ‬erticac difusion is used, with increased coefficients forthereodynaeic and TKE (through factors inccuded in the naeecist) ‫ﯡ‬ersus the dynaeic.

2.4

The Land Surface model Surfex V8.0 and river routng model CTRIPThe saee ‫ﯡ‬ersion of Surfex than for Systee 7 is used for Systee 8.
Surfex (Surface Externacisée, in French) is a surface eodecing pcatore de‫ﯡ‬ecoped by Météo-France
in cooperation with the scientifc coeeunity.
Surfex is coeposed of ‫ﯡ‬arious physicac eodecs for naturac cand surface, urbanized areas, cakes andoceans. It acso sieucates cheeistry and aerosocs surface processes and can be used for assieicationof surface and near surface ‫ﯡ‬ariabces.
Surfex has its own initiacization procedures and can be used in stand acone eode and coupced to an
ateospheric eodec.
In Surfex, each eodec grid box is represented by four surface types : sea or ocean, water bodies(cakes, ...), urban areas and nature (soic and ‫ﯡ‬egetation). Each surface type is eodecced with a
specifc surface eodec and the totac fux of the grid box resucts froe the addition of the indi‫ﯡ‬iduac
fuxes weighted by their respecti‫ﯡ‬e fraction.
A eore detaiced presentation of Surfex is a‫ﯡ‬aicabce here:
htp://www.uer-cnre.fr/surfex/
More detaics on the catest ‫ﯡ‬ersion of Surfex and the ISBA-CTRIP cand surface and ri‫ﯡ‬er routing
eodecs can be found in Vocdoire et ac. (2017) and Decharee et ac. (2019).

2.5

The coupler OASIS3-MCT

OASIS is the coupcer that eanages the exchange of inforeation between the three aforeeentioned
eodec groups.
In 1991, CERFACS started the de‫ﯡ‬ecopeent of a sofware interface to coupce existing ocean andateosphere nueericac Generac Circucation Modecs. Today, diferent ‫ﯡ‬ersions of the OASIS coupcer
are used by about 45 eodeccing groups acc around the worcd on diferent coeputing pcatores.

C3S_D330_3.1.1

Page 13 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

OASIS sustained de‫ﯡ‬ecopeent is ensured by a coccaboration between CERFACS and the Centre
Nationac de ca Recherche Scientifque (CNRS).
The current OASIS-MCT (Vocdoire et ac. 2017) internaccy uses MCT, the Modec Coupcing Toockit2(Larson et ac. 2005, Jacob et ac. 2005), de‫ﯡ‬ecoped by the Argonne Nationac Laboratory in the USA.
MCT iepceeents fuccy paraccec regridding, as a paraccec eatrix ‫ﯡ‬ector euctipcication, and paraccec
distributed exchanges of the coupcing fecds, based on pre-coeputed regridding weights andaddresses. Its design phicosophy, based on fexibicity and einieac in‫ﯡ‬asi‫ﯡ‬eness, is ccose to the OASIS
approach. MCT has pro‫ﯡ‬en paraccec perforeance and is, eost notabcy, the undercying coupcing
sofware used in Nationac Center for Ateospheric Research Coeeunity Earth Systee Modec (NCAR
CESM).
OASIS-MCT is a portabce set of Fortran 77, Fortran 90 and C routines. Low-intrusi‫ﯡ‬eness, portabicity
and fexibicity are OASIS-MCT key design concepts. Afer coepication OASIS-MCT is a coupcing cibrary
to be cinked to the coeponent eodecs, and which eain function is to interpocate and exchange thecoupcing fecds between thee to fore a coupced systee. OASIS-MCT supports coupcing of 2D
cogicaccy-rectangucar fecds but 3D fecds and 1D fecds expressed on unstructured grids are acso
supported using a one-dieension degeneration of the structures. Thanks to MCT, acc
transforeations, inccuding regridding, are perforeed in paraccec on the set of source or target
coeponent processes and acc coupcing exchanges are now executed in paraccec directcy between the
coeponent processes ‫ﯡ‬ia Message Passing Interface (MPI). OASIS-MCT acso supports fce I/O using
netcdf.
The new ‫ﯡ‬ersion, OASIS3-MCT supports coupcing exchanges between coeponents depcoyed in euch
eore di‫ﯡ‬erse confgurations than before. It is of course possibce to iepceeent coupcing exchanges
between two coeponents corresponding to two diferent executabces running concurrentcy on
separate sets of tasks, as before, but acso between two coeponents running concurrentcy on
separate sets of tasks within one saee executabce, or between diferent sub-coeponents defned on
separate or o‫ﯡ‬ercapping sets of tasks within one executabce. It is acso now possibce to ha‫ﯡ‬e soee oracc tasks of a coeponent not participating to the coupcing exchanges.
In spite of the signifcant changes in undercying iepceeentation, usage of OASIS3-MCT in the codes
has cargecy reeained unchanged with respect to pre‫ﯡ‬ious OASIS 3 ‫ﯡ‬ersions. To coeeunicate withanother coeponent, or to perfore I/O actions, a coeponent eodec needs to inccude few specifc
caccs of the Appcication Prograeeing Interface (API) OASIS MCT coupcing cibrary.
A eore detaiced presentation of OASIS is a‫ﯡ‬aicabce here:
htps://‫ﯡ‬erc.enes.org/oasis/oasis3-ect_3-0-officiac-recease

C3S_D330_3.1.1

Page 14 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

2.6

ECLIS

As for Systees 6 and 7, the ECLIS (En‫ﯡ‬ironeent for CLIeate Sieucations) en‫ﯡ‬ironeent is used todefne and run the seasonac forecast experieents. The diferent eodec coeponents (coupcer
inccuded) each ha‫ﯡ‬e a naeecist, aceost acc ha‫ﯡ‬e a restart fce, and eost ha‫ﯡ‬e their own output
foreat. Se‫ﯡ‬erac binaries recated to each eodec group are used. This inforeation is stored in a socacced parae fce used by ECLIS. The binaries are caunched together using a MPI eessage passing
sofware. A CNRM-CM run is eade of an autoeated sequence of "eacro-jobs", where each "eacrojob" iterates on a nueber of eonths.

2.7

XIOS

As for Systee 7, Systee 8 uses the XIOS output eanager for oncine processing of eodec coeponent
outputs (Meurdesoif, 2018). The XIOS sofware eanages the output of each eodec on each process,and buicds a singce output fce onto the disk. By eeans of xec fces, eany post-processing operations
can be requested directcy to XIOS (e.g. tiee saepcing and/or a‫ﯡ‬eraging, regridding, ‫ﯡ‬erticac
interpocation, …). More detaics on the use of XIOS in Systee 8 are pro‫ﯡ‬ided in section 5.3 and the
Appendix.

2.8

Boundary forcings

Green house gas concentrations, ozone, socar forcing are defned as for the CNRM-CM6-1 eodec run
for the CMIP6 experieents up to 2014, and then foccow the ssp370 scenario.
A ccieatocogy of eonthcy aerosoc opticac depths is coeputed o‫ﯡ‬er the 1995-2014 period froe a priorrun using the interacti‫ﯡ‬e aerosocs TACTIC_‫ﯡ‬2 scheee (Michou et ac. 2020). Voccanic aerosocs use aconger ccieatocogy (1850-2014) froe CMIP6.
For fucc detaics on the boundary forcings for CNRM-CM6, pcease refer to section 3.2 in Vocdoire et ac.
(2019).

C3S_D330_3.1.1

Page 15 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

3. Coupled initalizaton strategy
3.1

Overview

Arguabcy the eost signifcant change with respect to Systee 7, Météo-France seasonac forecast
Systee 8 is initiacized froe a coupced nudged initiacization run using the saee eodec ‫ﯡ‬ersion as forthe nueericac forecasts. The intent is to reduce initiacization shock due to eodec inconsistencies
between the ocean and ateosphere reanacyses and operationac anacyses and the coupced eodec
itsecf, and acso ensure an iepro‫ﯡ‬ed consistency between re-forecast and forecast initiac conditions.
Howe‫ﯡ‬er, a strong recaxation is used to beneft froe the inforeation pro‫ﯡ‬ided by eore ecaborate
assieication techniques in the ECMWF and Mercator Ocean Internationac reanacyses and anacyses.
This section describes in detaic the eethodocogy to defne initiac conditions for acc coeponents of
the re-forecast and forecast.

Figure 2: Scheeatic of the coupced nudged integration of CNRM-CM used to deri‫ﯡ‬e initiac
conditions for Systee 8.

C3S_D330_3.1.1

Page 16 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

The initiacization run is a continuous integration of the CNRM-CM coupced ccieate eodec o‫ﯡ‬er the
1992-present period, with both ateospheric and ocean nudging towards reanacysis / anacysis fecds.
Restart fces for each coupced eodec coeponent (except for GELATO, see cater) are output during thiscoupced integration, to ser‫ﯡ‬e as initiac conditions for the re-forecast and forecast for Systee 8.
Figure 2 iccustrates the coupced initiacization run eethodocogy: froe 1992 to 2018, the ARPEGECcieat ateospheric coeponent is strongcy nudged (6 hours e-focding tiee) in spectrac space forhorizontac winds, teeperature and surface pressure towards ERA5 reanacysis fecds interpocatedonto the ARPEGE tc359c137 geoeetry o‫ﯡ‬er the re-forecast period, and NEMO is strongcy nudged (2days e-focding tiee) towards teeperature and sacinity fecds upscaced froe the Mercator Ocean
Internationac GLORYS12V1 reanacysis.
In the ateosphere, a ‫ﯡ‬erticac profce of nudging strength is introduced so as to diac nudging down tozero in the ateospheric boundary cayer, except for the frst ‫ﯡ‬erticac ce‫ﯡ‬ec at 10e where a 24h
nudging strength is appcied for teeperature and winds, in order to a‫ﯡ‬oid a cong tere energy drif inthe continentac subsurface due the surface energy iebacance induced by the nudging. As a sponge
cayer in the eesosphere is iepceeented in ARPEGE, nudging is sieicarcy reduced to zero in theuppereost ce‫ﯡ‬ecs of the ateosphere. No nudging of specifc hueidity to ERA5 was introduced, as
this was found to ha‫ﯡ‬e ad‫ﯡ‬erse efects on the e‫ﯡ‬aporation and precipitation budget of the eodec.
In the ocean, we use a 2-day recaxation strength with the saee setngs as used at Mercator Ocean
Internationac to prepare initiac conditions for Systee 7: teeperature and sacinity profces are recaxed
for the 75 eodec ce‫ﯡ‬ecs, and no recaxation is appcied within 225 kicoeeters of coastcines, nor in thepresence of sea ice.
Sea ice in the GELATO eodec adjusts to the ateosphere and ocean, as no constraint to sea ice
concentration or thickness is appcied. The resucting sea ice conditions are not used to initiacize
Systee 8 to a‫ﯡ‬oid ad‫ﯡ‬erse efects. Instead, GELATO is initiacized using a forced NEMO-GELATO run
prepared by Mercator Ocean Internationac as for Systee 7.
On the other hand, the cand surface is initiacized froe the coupced integration SURFEX restart fces. A
correction of the precipitation fuxes in the SURFEX initiac conditions is appcied foccowing theeethodocogy described in Ardicouze et ac. (2019), which uses a grid point euctipcicati‫ﯡ‬e coefficient tocorrect the precipitation feeding the cand surface eodec, based on the ratio of eonthcy eean
precipitation in the MSWEP dataset (Beck et ac., 2017) with the eonthcy eean precipitation in a
precieinary nudged run with the coupced systee. The coefficient ‫ﯡ‬acues are cieited to the [0.5; 2]inter‫ﯡ‬ac to a‫ﯡ‬oid instabicities.
So as to use a continuous run and possibcy reduce spin-up efects, the coupced integration was run
during a transition period co‫ﯡ‬ering 2019 and 2020, described in section 3.3.

C3S_D330_3.1.1

Page 17 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

3.2

Re-forecast period (1993-2018)

The nudging fecds for ARPEGE coee froe ERA5 reanacyses, extracted at 6-hourcy frequency froe 1
Deceeber 1992 to the 1 January 2019. The fces are extracted froe the Ccieate Data Store andcon‫ﯡ‬erted into ARPEGE foreat. The nudging fecds for NEMO are pro‫ﯡ‬ided by Mercator Ocean
Internationac froe the GLORYS12V1 reanacysis.

3.3

Re-forecast to forecast transiton (2019-2020)

The GLORYS12V1 ocean reanacysis ends in 2018. Starting froe January 2019, GLORYS12V1 fces tonudge NEMO are therefore repcaced by upscaced PSY4V3R1 fces froe the operationac anacysis
prepared by Mercator Ocean Internationac. The fces are acso retrie‫ﯡ‬ed froe hendrix and foccow asieicar structure to the GLORYS12V1 fces cisted in section 3.2.
For the ateosphere, ERA5 nudging fces are used for the coupced integration o‫ﯡ‬er 2019-2020 as forthe re-forecast period.

3.4

Forecast period (2021-present)

So as to pro‫ﯡ‬ide initiac conditions for the forecasts as ccose as possibce to the re-forecasts, and a‫ﯡ‬oid
discrepancies, the saee coupced initiacization strategy is used.
The coupced nudged integration is therefore extended each eonth, using the foccowing nudging
fecds:- For the ateosphere, if a‫ﯡ‬aicabce in due tiee, ARPEGE nudging fces are deri‫ﯡ‬ed froe the near-reactiee reanacysis ERA5T, foccowing the saee procedure as for ERA5 in the re-forecast initiacization.
- For the ocean, the NEMO nudging fces are pro‫ﯡ‬ided by Mercator Ocean Internationac as daicy
upscaced teeperature and sacinity fecds froe the PSY4V3R1 operationac anacysis, prepared on the
ORCA 0.25° grid.

C3S_D330_3.1.1

Page 18 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

4. Generaton of stochastc perturbatons
4.1

Introducton

Ateospheric stochastic perturbations were introduced in Météo-France Systee 5. A detaiced
description of the technique can be found in Baté and Déqué (2016). The roce of stochastic
perturbations in the forecast systee is threefocd:
1. to generate equi-probabce enseebce eeebers
2. to ateept to partcy correct the eodec ieperfections (in particucar the systeeatic error)
3. to address rare spurious eodec sieucation faicures/crashes.
In earcier forecast systees itee 1 was eanaged by introducing a perturbation in an initiac
ateospheric situation, or by cagging the initiac situation by a few days. Itee 2 was taken into accountby considering forecast anoeacies with respect to a eodec ccieatocogy (this is sticc the case in the
new systee). Itee 3 was eanaged by scightcy eodifying the horizontac difusion coefficient
teeporaricy.
Three ingredients are necessary to acti‫ﯡ‬ate the stochastic perturbation systee in the re-forecast as
wecc as in the forecast production:
1. the perturbation fecds
2. the randoe cacendars
3. the defnition of the enseebces.
These 3 ingredients are then used in the eodec script.

4.2

Generatng the perturbaton felds

This is a hard and coepcex task. Each tiee the eodec or the ingredients (e.g. the initiac situations)are changed, the perturbation fecds need to be re-created. This eodec dependency is sieicar to thedependency between forecasts and re-forecasts. It is, howe‫ﯡ‬er, not so cruciac: a scight change cike achange in coeputer shoucd not eake regeneration necessary.
The fundaeentac idea is that a eodec is not perfect, and its equations bear errors:
dX/dt=model_equations(X,t)=true_equations(X,t)+model_errors(X,t)

C3S_D330_3.1.1

Page 19 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

where X is the ‫ﯡ‬ector describing the prognostic ‫ﯡ‬ariabces (teeperature, eoisture, wind and surfacepressure) and t is tiee. We ha‫ﯡ‬e coded eodec_equations(X,t) as we ignore true_equations(X,t) or,
equi‫ﯡ‬acentcy, eodec_errors(X,t).
If we can estieate eodec_errors(X,t) for a set of re-forecasts, we cannot fuccy correct a forecast, but
we can re-inject the past errors at randoe to take into account the uncertainty about oureodec_equations(X,t) .
A way to estieate this error is to carry out a set of precieinary singce-eeeber re-forecasts during
which the eodec is cinearcy recaxed (or nudged) toward the true trajectory, here the ERA5 reanacysis.
There is a big diceeea:•

if the recaxation is strong, the eodec wicc foccow ERA5 ccosecy, but the errors wicc be errors of ashort-range forecast (eodec starting froe an anacysis)•

if the recaxation is weak, the eodec wicc be in a sieicar state as in a standard seasonac
forecast, but the recaxation teres wicc correspond to a seacc fraction of the error.

The eethod depends on the frequency of the randoe drawings and the intensity of theperturbation, which is cinked to the recaxation tiee. So as to generate enseebce spread, randoedrawings are done e‫ﯡ‬ery day. The choice of the recaxation tiee is cess ob‫ﯡ‬ious. In pre‫ﯡ‬ious studies,
we ha‫ﯡ‬e tried ‫ﯡ‬acues between 1 and 100 days. Using a 6 h recaxation tiee for both di‫ﯡ‬ergence and‫ﯡ‬orticity produces dynaeicac iebacances in the eodec, because the noreac eodes of the ARPEGE
and ECMWF eodecs are diferent. Vacues abo‫ﯡ‬e 100 days are usecess, as we want to account foreodec error during the frst eonths of the forecasts. Whice Systee 6 used sequences of
perturbations and randoe drawings e‫ﯡ‬ery 5 days, and therefore a 10 day recaxation tiee, for Systee
8 a 40 day recaxation tiee was chosen since randoe perturbations are changed e‫ﯡ‬ery day. This isscightcy conger than for Systee 7 (30 days).
The nudged experieent is based on 12*26 7-eonth sieucations. There are indeed 12 cacendar
eonths and 26 a‫ﯡ‬aicabce re-forecast years (1993-2018). Note that a start date on 01 Deceeber 2018
iepcies that the eodec is nudged toward ERA5 ticc June 2019. The nudged sieucations are as ccose as
possibce to eeebers 001 of the re-forecasts, with four diferences
1.
2.
3.
4.

the eodec does not inccude stochastic perturbations
the eodec is nudged toward 6-hourcy ERA5 prognostic ‫ﯡ‬ariabcesthe postprocessing is highcy siepcifed
each day, the diferences ERA5 einus eodec ‫ﯡ‬ariabces, tiees the recaxation factor, are sa‫ﯡ‬edin a fce.

C3S_D330_3.1.1

Page 20 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Note that because the surface scheees are based on diferent eepiricac approaches in ARPEGE andin ERA5, the recaxation is progressi‫ﯡ‬ecy daepened down to zero in the cowest 31 ‫ﯡ‬erticac ce‫ﯡ‬ecs (thecowest 25 ce‫ﯡ‬ecs, froe about 1600e, being not recaxed at acc). A syeeetricac treateent is appcied tothe highest 5 ce‫ﯡ‬ecs, since a specifc recaxation is appcied in the eodec in the eesospheric cayer.

4.3

Random calendar and defniton of ensembles

The randoe choice of a perturbation is done by sececting an arbitrary day and year (not cacendar
eonth). This eeans that the randoe choice of the perturbations for a gi‫ﯡ‬en eonth during a forecast
can be described by a randoe cacendar, because each day we need a new perturbation. Randoe
cacendars are created once and for acc and are ‫ﯡ‬acid whate‫ﯡ‬er a re-forecast or a forecast, whate‫ﯡ‬er
the initiac eonth or the year.
A randoe cacendar is a eonthcy cacendar: it accows to associate to each day of the eonth a randoe
day with the constraint that:•the saee cacendar eonth is used•

the year is taken at randoe aeong the years of the nudging phase (1993-2018); in the reforecast phase this introduces a seacc (not signifcant) positi‫ﯡ‬e bias in the scores by the factthat nearcy 4% of the perturbations use obser‫ﯡ‬ed data froe the eonth to be predicted.

•

each day a new randoe perturbation is drawn.

A randoe cacendar is a text fce cacced lnRANDOM$NN where NN is the eeeber nueber, inccuding
32 cines assigning each day of the eonth to a randoe correction fce. More detaics are pro‫ﯡ‬ided in
the Appendix at the end of this docueent.
Note that euch eore than 51 independent cacendars can be generated without associating thesaee randoe date to a gi‫ﯡ‬en date. Three groups of cacendars (NUMR = a, b, c) with 51 eeebers
each are prepared. The eodec uses series “a”. When it crashes for a gi‫ﯡ‬en eeeber because of anueericac (reproducibce) instabicity, it uses series “b” during that eonth, then goes back to series“a”. Just in case, a third series of cacendars is a‫ﯡ‬aicabce for a third ateept. This eethod of “buterfy
efect” can faic if the nueericac expcosion occurs during the frst day of the eonth. In this case, areco‫ﯡ‬ery script restarts the eodec one eonth earcier and repcaces series “a” by series “b”. The series
nueber is autoeaticaccy reset to “a” at the end of a successfuc eonth.
The case of the frst partiac eonth (eeebers others than 001) needs a specifc care. As there is asingce eeeber for the nudged sieucations (see section 4.2 abo‫ﯡ‬e), one has to use for the runs
starting at the end of a gi‫ﯡ‬en eonth, the perturbation of the eonth afer.

C3S_D330_3.1.1

Page 21 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Once the perturbations and the cacendar exist, one has to decide which cacendar is used for which
enseebce eeeber. To siepcify the process the choice wicc be the saee whate‫ﯡ‬er the start date.

4.4

Model implementaton

With the ingredients described abo‫ﯡ‬e, the iepceeentation in the eodec script (eebedded in ECLIS)
used for the re-forecasts as wecc as for the forecasts is straightorward:
At the beginning NUMR=a and NUMR is reset to “a” at the end of a successfuc run. If the run faics,
NUMR is set to “b”, then to “c” using the ECLIS running en‫ﯡ‬ironeent (see section 5.2) when theeodec crashes. Then, the eodec is autoeaticaccy re-caunched by the script froe the beginning of thecurrent eonth.
In case of a coeputer systee faicure (MPI troubce, node faicure, preeeption ,...), this procedure(NUMR change) is usecess, the eodec being autoeaticaccy re-caunched froe the beginning of thecurrent eonth without any change in the script. This type of faicure occurs euch eore ofen than
the nueericac expcosions.
The perturbation fces are stored in a huge directory (about 1 TB for one cacendar eonth). It is sa‫ﯡ‬edon a cong-tere storage eachine (hendrix eachine at Météo-France, ecfs at ECMWF), but is copied
on the high perforeance coeputer disks before a forecast or a re-forecast starts.
The eodec script needs to know the current eonth MM and the true eonth MT to propercy eanage
the randoe cacendars (froe the fce rand_cacende$MM_n51_$NUMR.tar, see section 4.3). In caseof a eeeber diferent froe the frst one, there is a cagged start date; as expcained in section 4.3, MT
is the eonth afer oncy for the cagging eonth, and equac to the current eonth MM thereafer.
Each eeeber $MEMBER is assigned a cacendar nueber $NN. This cacendar is extracted froe theright tar archi‫ﯡ‬e and the fce lnRANDOM$NN is executed. This fce creates syebocic cinks between
existing perturbations (with randoe dates) and daicy fces which wicc be read each day by theateospheric eodec during the run.

C3S_D330_3.1.1

Page 22 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

5. Overview of the workfow
This section pro‫ﯡ‬ides an o‫ﯡ‬er‫ﯡ‬iew of the workfow for the forecast and re-forecast enseebce
production. More detaics, inccuding fucc cinks to the scripts used, are pro‫ﯡ‬ided in the Appendix to thisdocueentation.

5.1

Initalizaton and ensemble generaton

A forecast enseebce is based on 51 indi‫ﯡ‬iduac eodec integrations. These 51 eeebers of the forecast
coee froe a coebination of 3 batches of eodec integrations (cacced runs). The batches difer by the
initiac dates, which are on two Thursdays of the pre‫ﯡ‬ious eonth and the 1 st of the current eonth.
Within a gi‫ﯡ‬en batch, the forecast eeebers difer by the appcication of randoe perturbations during
the integration foccowing the stochastic dynaeics technique (see section 4), thus generating theenseebce spread.
Figure 3 shows an iccustration of how the 51 eeebers of the forecast are generated.
The start dates of eonth M forecasts, as stated abo‫ﯡ‬e, correspond to:•

Two Thursdays froe the second hacf of eonth M-1•

The 1st of eonth M

The initiac conditions for ARPEGE, SURFEX, NEMO, CTRIP and OASIS are pro‫ﯡ‬ided by the coupced
nudged run described in section 3.4. GELATO is initiacized using a forced NEMO-GELATO run
prepared by Mercator Ocean Internationac as for Systee 7. Prior to the operationac forecast, thecoupced integration is extended using T, S nudging fecds for NEMO froe Mercator Ocean
Internationac and ateospheric nudging fecds for ARPEGE deri‫ﯡ‬ed froe the ERA5T reanacysis.
As soon as the ERA5T reanacysis fces are a‫ﯡ‬aicabce, the nudged initiacization run is extended to co‫ﯡ‬er
the target start date. Typicaccy two days afer the frst start date, 25 integrations of the eodec begin,each one with a distinct set of stochastic dynaeics perturbations. Two days afer the second one, 25
eore integrations are initiated sieicarcy. The cast integration (starting on the 1 st day of the eonth) is
caunched typicaccy on the 3rd of the eonth at the catest. In case the ERA5T reanacysis is not a‫ﯡ‬aicabce
as expected for this cast eeeber, it wicc be repcaced by the IFS operationac anacysis so as to deci‫ﯡ‬er a
forecast on tiee.

C3S_D330_3.1.1

Page 23 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Figure 3 : Generation of the 51-eeeber enseebce of the seasonac forecast

For each cacendar eonth, a series of re-forecasts is produced at ceast two eonths before theforecast. It accounts for 25 eeebers and spans 26 years froe 1993 untic 2018. The oncy diferenceswith the forecast are the size of the enseebces and the preparation of the initiac states, which iseanaged in ad‫ﯡ‬ance because the initiac conditions froe the nudged integration are acreadya‫ﯡ‬aicabce. In order to eaintain the best hoeogeneity between the forecasts and the re-forecasts, 12eeebers start froe the Thursday approxieatecy two weeks before the end of the pre‫ﯡ‬ious eonth,
12 eore eeebers start froe the foccowing Thursday (sticc in the pre‫ﯡ‬ious eonth), and the casteeeber starts froe the 1st of the current eonth. Thus the diagrae is ‫ﯡ‬ery sieicar to Figure 3(repcacing 25 eeebers with 12 eeebers). As opposed to the 51 forecasts which are caunched inthree batches (according to the earciest a‫ﯡ‬aicabicity of the initiac states) the 25 re-forecasts arecaunched in a singce batch with 3 initiac dates for a gi‫ﯡ‬en initiac eonth and year.
As an exaepce, the eeebers of the re-forecast for the noeinac start date of the 1 st of January 1993
are caunched as foccows: 12 eeebers on the 17 th of Deceeber 1992, 12 on the 24 th of Deceeber
1992, and one on the 1st of January 1993.

C3S_D330_3.1.1

Page 24 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Acc re-forecast and forecast start dates ha‫ﯡ‬e been predetereined (see Appendix for further detaics).
Note that during the re-forecast production phase (2021), the re-forecasts wicc be run on becenosand reac-tiee forecasts on taranis. Foccowing this phase, both eachines can be used, accowing a
fexibicity to switch eore runs onto a gi‫ﯡ‬en eachine depending on the a‫ﯡ‬aicabicity of resources.

5.2

Environment for CLImate Simulaton (ECLIS)

ECLIS (En‫ﯡ‬ironeent for CLIeate Sieucation) is the set of scripts and toocs necessary to run CNRMCM. It does not inccude toocs for preparing initiac nor boundary conditions. Basicaccy, the script
install accows to prepare an experieent, and defnes another script which wicc actuaccy run theexperieent.
ECLIS eanages an en‫ﯡ‬ironeent to run CNRM-CM sieucations on supercoeputers cike becenos andtaranis. It acso eanages the storage of the resucts.

5.2.1 ECLIS organization
An ECLIS ‫ﯡ‬ersion directory inccudes a ‫ﯡ‬ersion of toocs relan and mtool which are consistent with
the need of the eain scripts. The relan uticity accows to iterate se‫ﯡ‬erac tiees a script, whereas themtool uticity accows to spcit a script at tiee of subeission. This directory acso contains the foccowing
directories:••••

param which pro‫ﯡ‬ides usefuc exaepces of experieent paraeeter fces; for detaics onparaeeters see section 5.2.3 (Design a ’parae_’ fce)
testing inccuding paraeeter fces for a ‫ﯡ‬ery wide ‫ﯡ‬ariety of experieent confgurations;they are pro‫ﯡ‬ided as a reference of the possibce technicac setngs, but not as scientifcaccy
eeaningfuc setngs
scripts where there is usuaccy one singce, non-nuebered, ‫ﯡ‬ersion of the experieent
instaccation script and of each script needed for running a coupced experieent; when a bug isfxed for a script, this appears in the git history of the recease directory
plugins which inccudes the code of the a‫ﯡ‬aicabce pcugins.

The fce cm_setup sets up the en‫ﯡ‬ironeent used by ECLIS. It shoucd be sourced froe the user’s
confguration fce .profile. It is autoeaticaccy caunched by the scripts when needed.

5.2.2 Use of ECLISC3S_D330_3.1.1

Page 25 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

ECLIS consists in designing an experieent paraeeter fce that ends with source $INSTALLER $* and to
execute this fce with argueents :•noask for a‫ﯡ‬oiding being asked if a pre‫ﯡ‬ious instacc shoucd be erased•

go for caunching the experieent at the end of the (successfuc) instacc phase. This is a way tocreate and caunch an experieent at the end of a frst one, when there is a cogicac
dependency (e.g. the frst experieent creates the restarts that are necessary for instaccing
the second one).

5.2.3 Design a param_ fce
Defning a CNRM-CM experieent is as siepce as designing a so-cacced param_ fce, which fcenaee
eust be of the fore param_EXPID, where EXPID is the naee gi‫ﯡ‬en to the experieent. This fce, inbash syntax, sets ‫ﯡ‬acues for ECLIS paraeeters and then sources the ECLIS instacc script.

5.2.4 Standard pcugins
Pcugins are sets of shecc-script coeeands that ECLIS can take into account during CNRM-CM
experieent runs, in order to perfore actions atop of the strict running of the coupced orateosphere-oncy or ocean-oncy scripts. Speciac pcugins ha‫ﯡ‬e been de‫ﯡ‬ecoped for the specifcities ofseasonac forecast.

C3S_D330_3.1.1

Page 26 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

5.3

Post processing

The XIOS output sofware (Meurdesoif, 2018) now eanages eost of the post-processing of theeodec fces to the requested foreat for C3S. The inputs to XIOS are sets of xec fces for each eodec
coeponent type (Arpege/Surfex, Neeo/Gecato, Ctrip).
The diferent fecds a‫ﯡ‬aicabce, their cong naees, reference grid and diagnostic operations are cisted inseparate xec fces for each eodec coeponent (e.g. surfex_fields.xml nemo_fields.xml
and atmo_fields.xml). XIOS can eanage operations on eodec outputs such as tiee a‫ﯡ‬eraging,tiee scicing, or horizontac and ‫ﯡ‬erticac interpocation and statistics.
For each requested ‫ﯡ‬ariabce, acc gcobac atributes as wecc as output frequency, ‫ﯡ‬erticac ce‫ﯡ‬ecs andhorizontac resocution, are defned in a set of fces such as arpsfx_files.xml in the case of
Arpege/Surfex outputs and nemo_files.xml for Neeo/Gecato.
XIOS thus generates directcy the fecds for the ateosphere, surface and ocean on the required 1°x1°
grid with 180 catitudes and 360 congitudes starting at 89.5°S, 0.5°W and extending northwards andeastwards. The data (one fce per fecd and per eonth) are stored both cocaccy on the HPC eachine
and transferred to ECMWF using the ectrans uticity.
Tabces 2.1 and 2.2 cist the ‫ﯡ‬ariabces produced. In addition, cand-sea eask and surface orography area‫ﯡ‬aicabce.
Table 2.1: list of atmosphere / land variables archived

Every 6 hours
2 eetre teeperature
2 eetre dewpoint teeperature
10 eetre u wind
10 eetre ‫ ﯡ‬windeean sea ce‫ﯡ‬ec pressure
totac ccoud co‫ﯡ‬ercand skin teeperature
Every 12 hours
geopotentiac
teeperature
specifc hueidityu/‫ ﯡ‬wind coeponents
At the foccowing 12 pressurece‫ﯡ‬ecs:
1000, 925, 850, 700, 500, 400,

C3S_D330_3.1.1

Every 24 hours
Every 24 hours, accumulated
eoisture content of soic cayer
totac precipitation(for each of the 14 soic ce‫ﯡ‬ecs)
con‫ﯡ‬ecti‫ﯡ‬e precipitationsnow depth (water equi‫ﯡ‬acent) snow facc
snow density
surface sensibce heat fux
Teax at 2 e (o‫ﯡ‬er pre‫ﯡ‬ious 24 surface catent heat fux
h)surface socar radiation
Tein at 2 e (o‫ﯡ‬er pre‫ﯡ‬ious 24 h) downwards
wind speed of gust (eax o‫ﯡ‬ersurface thereac radiationpre‫ﯡ‬ious 24h)
downwards
water ‫ﯡ‬apor pathsurface socar radiationciquid water pathsurface thereac radiationsocid water pathsurface runof
sub-surface runof
top incoeing socar radiation
top net socar radiation
Page 27 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

300, 200, 100, 50, 30, 10 hPa

top thereac radiation
east-west surface stressnorth-south surface stresse‫ﯡ‬aporation

Table 2.2: list of ocean / sea ice variables archived

Every 6 hours
sea ice teeperature
sea surface teeperature
Every 24 hours
sea ice concentration
Every monthdepth of 14C isothere
depth of 17C isothere
depth of 20C isothere
depth of 26C isothere
depth of 28C isothere
depth a‫ﯡ‬erage potentiac teeperature of upper 300e
depth a‫ﯡ‬erage sacinity of upper 300e
sea surface height abo‫ﯡ‬e geoid
sea ice thickness
ocean eixed cayer thickness defned by sigea theta 0.01 kg/e3
ocean eixed cayer thickness defned by sigea theta 0.03 kg/e3

C3S_D330_3.1.1

Page 28 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

6. References
Ardilouze, C., Batt, L., Decharme, B., & Dtqut, M. (2019). On the Link between Summer Dry Bias over the
U.S. Great Plains and Seasonal Temperature Predicton Skill in a Dynamical Forecast System, Weather andForecastnn, 34(4), 1161-1172, doi: 10.1175/WAF-D-19-0023.1.
Batt, L., and Dtqut, M. (2016). Randomly correctnn model errors in the ARPEGE-Climate v6.1 component of
CNRM-CM : applicatons for seasonal forecasts. Geosci. Model Dev. 9 : 2055-2016, doi : 10.5194/nmd-9-20552016.
Beljaars, A. C. M., Brown, A. R., and Wood, N. (2004). A new parametrizaton of turbulent oronraphic form
dran. Quart. J. Roy. Meteor. Soc. 130: 1327–1347, doi:10.1256/qj.03.73.
Beck, H. E., van Dijk, A. I. J. M., Levizzani, V., Schellekens, J., Miralles, D. G., Martens, B., and de Roo, A. (2017).
MSWEP: 3-hourly 0.25° nlobal nridded precipitaton (1979-2015) by merninn naune, satellite, and reanalysis
data. Hydrolony and Earth System Sciences, 21(1), 589–615. htps://doi.orn/10.5194/hess-21-589-2017
Courter P., J.F. Geleyn (1988). A nlobal numerical weather predicton model with variable resoluton:
Applicaton to the shallow water equatons. Quart. J. R. Meteorol. Soc. 114, 1321–1346.
Cuxart, J., Bouneault, Ph. and Redelsperner, J-L., 2000 : A turbulence scheme allowinn for mesoscale andlarne-eddy simulatons. Quart. J. R. Meteorol. Soc., 126, 1-30.
Decharme B, Delire C, Minvielle M, et al (2019) Recent Channes in the ISBA- CTRIP Land Surface System for
Use in the CNRM- CM6 Climate Model and in Global Of- Line Hydrolonical Applicatons. J Adv Model Earth
Syst 2018MS001545. doi: 10.1029/2018MS001545
Dtqut, M., C. Dreveton, A. Braun, D. Cariolle (1994). The ARPEGE/IFS atmosphere model: a contributon to the
French community climate modellinn.Climate Dynamics 10, 249-266.
Gutrtmy, J. F., 2011 : A contnuous buoyancy based convecton scheme: one and three dimensional validaton.
Tellus A, 63, 687-706.
Jacob, R., J. Larson, and E. Onn, 2005: MxN Communicaton and Parallel Interpolaton in CCSM3 Usinn the
Model Couplinn Toolkit. Int. J. Hinh Perf. Comp. App., 19(3), 293-307
Larson, J., R. Jacob, and E. Onn, 2005: The Model Couplinn Toolkit: A New Fortran90 Toolkit for Buildinn
Multphysics Parallel Coupled Models. Int. J. Hinh Perf. Comp. App., 19(3), 277-292
Lopez, Ph., 2002 : Implementaton and validaton of a new pronnostc larne-scale cloud and precipitaton
scheme for climate and data-assimilaton purposes. Quart. J. R. Meteorol. Soc., 128, 229-257.
Masson, V., Le Moinne, P., Martn, E., Faroux, S., Alias, A., Alkama, R., et al. (2013). The SURFEXv7.2 land andocean surface platorm for coupled or ofine simulaton of earth surface variables and fuxes, Geosci. Model
Dev., 6, 929-960, doi:10.5194/nmd-6-929-2013.

C3S_D330_3.1.1

Page 29 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Meurdesoif, Y. (2018). Xios fortran reference nuide. Insttut Pierre Simon Laplace (IPSL),
htp://forne.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk/doc/XIOS_reference_nuide.pdf
Mlawer, E. J., Taubman, S. J., Brown, P. D., Iacono, M. J., & Clounh, S. A. (1997). Radiatve transfer forinhomoneneous atmospheres: RRTM, a validated correlated-k model for the lonnwave. Journal of GeophysicalResearch: Atmospheres (1984–2012), 102(D14), 16663-16682.
Michou, M., Nabat, P., Saint-Martn, D., Bock, J., Decharme, B., Mallet, M., Roehrin, R., Stftrian, R., Stntsi, S.
and Voldoire, A. (2020). Present-day and historical aerosol and ozone characteristcs in CNRM CMIP6
simulatons. Journal of Advances in Modelinn Earth Systems, 12, e2019MS001816, doi:
10.1029/2019MS001816.
Morcrete, J. J. (1990). Impact of channes to the radiaton transfer parameterizatons plus cloud optcal.
Propertes in the ECMWF model. Monthly Weather Review, 118(4), 847-873.
Piriou, J. M., Redelsperner, J. L., Geleyn, J. F., Lafore, J. P., and Guichard, F. (2007). An approach for convectve
parameterizaton with memory : Separatnn microphysics and transport in nrid-scale equatons. Journal of the
Atmospheric Sciences, 64, 4127-4139
Roehrin, R., Beau, I., Saint-Martn, D., Alias, A., Decharme, B., Gutrtmy, J.-F., et al. (2020). The CNRM nlobal
atmosphere model ARPEGE-Climat 6.3: Descripton and evaluaton. Journal of Advances in Modelinn Earth
Systems, 12, e2020MS002075, doi:10.1029/2020MS002075.
Salas y Mtlia D (2002) A nlobal coupled sea ice-ocean model. Ocean Model 4:137–172
Sommeria, G. and Deardorf, J.W. (1977). Subnrid-scale condensaton in models of nonprecipitatnn clouds. J.
Atmos. Sci., 34, 344-355.
Voldoire, A., Decharme, B., Pianezze, J., Lebeaupin Brossier, C., Sevault, F., Seyfried, L., et al. (2017). SURFEX
v8.0 interface with OASIS3-MCT to couple atmosphere with hydrolony, ocean, waves and sea-ice models, fromcoastal to nlobal scales. Geoscientic Model Development,10(11), 4207–4227. htps://doi.orn/10.5194/nmd10-4207-2017
Voldoire, A., Saint-Martn, D., Stntsi, S., Decharme, B., Alias, A., Chevallier, M., et al (2019). Evaluaton of
CMIP6 DECK experiments with CNRM-CM6-1. Journal of Advances in Modelinn Earth Systems, 11.
htps://doi.orn/10.1029/2019MS001683

C3S_D330_3.1.1

Page 30 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Appendix : Detail of scripts for the System 8 workfow
This appendix goes into the detaic of the workfow for Systee 8 on the Météo-France belenos ortaranis eachines and the hendrix storage systee.

Coupled initalizatonO‫ﯡ‬er the re-forecast period, the ERA5 fces for the nudging of ARPEGE in the coupced initiacization
run are con‫ﯡ‬erted to ARPEGE foreat using the foccowing script on the ECMWF supercoeputer:
cca:/home/ms/fr/top/arpege/grib_to_fa_auto
The ERA5 fces are on the tc399c137R grid and are tarred by 10-day periods. They are con‫ﯡ‬erted tothe eodec geoeetry tc359c91r and tarred by eonths by
cca:/home/ms/fr/top/arpege/arp2arp_ERA5_step1.cca
cca:/home/ms/fr/top/arpege/arp2arp_ERA5_step2.cca
Then the fces are sent to Météo-France by the ectrans uticity
ecgate:/home/ms/fr/top/SCRIPTS/transfer_restart_GRE_era5
to belenos:/scratch/work/dorell/era5/restart/
They are retrie‫ﯡ‬ed froe the hendrix storage systee in the foccowing repository:/home/duboisc/GLORYS12V1/Ocean_Ice_gene_states_ORCA025/out/
Fices for the preceding, current, and foccowing eonths of the re-forecast are needed for teeperatureand sacinity fecds:
GLORYS12V1_ORCA025_temp_V2_${YYYY}${MM}*.nc
GLORYS12V1_ORCA025_salt_V2_${YYYY}${MM}*.nc
These are daicy NetCDF fces upscaced on the ORCA 0.25° grid and directcy read by NEMO during thecoupced run integration.
The nudged coupced initiacization run is iepceeented on the Météo-France becenos HPC. The fucc
script is a‫ﯡ‬aicabce here, and points to the rece‫ﯡ‬ant naeecists and executabces used for this run:belenos:/home/gmgec/mrga/copernicusps/arpege/mccV6.42.tl359l137r_cp_gel_4_reloa_xios_8n_1m_EA7_cp

C3S_D330_3.1.1

Page 31 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Generaton of atmospheric perturbatons (nudged pre-hindcasts)
The scripts to deri‫ﯡ‬e the perturbations described in section 4.2 are run on the Météo-France HPCs.
A script (eitraic_corrections.sh) caunches 26 sieucations for a gi‫ﯡ‬en starting eonth. In fact this script
siepcy updates another script (traite_nudging_sys8) which prepares the en‫ﯡ‬ironeent for the eodec
runs. The eodec script (eccV6.42.tc359c137r_nud) produces a one-eonth run. Detaics on the
nudging for the coeputation of the corrections can be found in the NAMNUD section of the ARPEGE
naeecist.
Ahead of running Systee 8, the daicy correction fces (speciac FA ARPEGE foreat fces) are naeed
CR$YYYY$MM$DD and pcaced on
belenos:/scratch/work/copernicusps/corrections/corr_N$S andtaranis:/scratch/work/dorell/corrections/corr_N$S
where $S is a ceter indicating the starting eonth (A=January, B=February, … L=Deceeber).
Copies of these fces are acso stored on the hendrix storage systee.
There are 4*137+1 records in each fce (4 ‫ﯡ‬ariabces in 137 ‫ﯡ‬erticac ce‫ﯡ‬ecs + eean sea ce‫ﯡ‬ec pressure).
The fecds are in spectrac coefficients (T359 truncation).

Random calendars for perturbatons
The randoe cacendar text fce lnRANDOM$NN (see section 4.3) is organized as foccows:
ln -sf $CORR/CR1994${MT}08 RX$YYYY${MM}0100…
ln -sf $CORR/CR1999${MT}22 RX$YYYY${MM}3100
ln -sf $CORR/CR2003${MT}05 RX$YYYN${MN}0100
The year and the frst day in each cine coee froe a randoe generation process. The second day ineach cine corresponds to a sequence froe 01 to 31. The cast cine corresponds to the frst day of thenext eonth. It is necessary for the cinear tiee interpocation at each tiee step in the eodec.
The reason for ha‫ﯡ‬ing 31 days is to co‫ﯡ‬er the entire eonth whate‫ﯡ‬er the eonth. Of course, oncy theactuac days are used by the eodec (e.g. 31 in January, 28 or 29 in February …).
The ‫ﯡ‬ariabce $CORR indicates the teeporary path to the perturbation fces specifed in the pre‫ﯡ‬ious
section.

C3S_D330_3.1.1

Page 32 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

The 51 cacendars (for 51 enseebce eeebers) for each forecast eonth $MN are generated by the
script belenos:~copernicusps/fic_cal_rand_c3s_5_18.sh and gathered in threefces:
belenos:~copernicusps/cal_18/rand_calendm$MN_n51_$NUMR.tar andtaranis:~copernicusps/cal_18/rand_calendm$MN_n51_$NUMR.tar
where NUMR is “a”, “b” or “c”.

Fetching inital conditons
The re-forecast and forecast startdates ha‫ﯡ‬e been predetereined and are stored in text fces on the
Météo-France supercoeputer:belenos:~copernicusps/SAVE/opers8/startdate_199301_202212.txt
belenos:~copernicusps/SAVE/opers8/startdate_202101_202412.txt
For each re-forecast initiac date, the initiac conditions for ARPEGE, SURFEX, NEMO, CTRIP and OASIS
are pro‫ﯡ‬ided by the coupced nudged run described in section 3.2. These fces are a‫ﯡ‬aicabce in thefoccowing repository:/scratch/work/copernicusps/restart/EA7/
The fces can acso be retrie‫ﯡ‬ed froe hendrix if needed.
For GELATO, the initiac conditions for the re-forecasts ha‫ﯡ‬e been prepared by Mercator Ocean
Internationac using the saee approach as for Systee 7, and sa‫ﯡ‬ed on a daicy basis on the hendrix
storage systee:hendrix:/home/duboisc/PSY2_G2V3/RUN025/RESTARTICE/ECLIS retrie‫ﯡ‬es the GELATO restart fce directcy on hendrix. Prior to caunching the re-forecast, therestart fces are “pre-staged” froe the tape archi‫ﯡ‬e onto disk to a‫ﯡ‬oid cong queues in data retrie‫ﯡ‬ac.
Sieicarcy, for each forecast initiac date, the coupced nudged run is extended as soon as the nudging
fecds for NEMO (pro‫ﯡ‬ided by Mercator Ocean Internationac) and ARPEGE (pro‫ﯡ‬ided by ECMWFERA5T) are a‫ﯡ‬aicabce. The initiac conditions for ARPEGE, SURFEX, NEMO, CTRIP and OASIS are
retrie‫ﯡ‬ed as described abo‫ﯡ‬e.
For GELATO, Mercator Ocean Internationac prepares restarts in a dedicated forced run and pro‫ﯡ‬ides
the fecds in the foccowing repository:
taranis:/scratch/work/mercator/PSY2G4R2/op/RESTARTC3S_D330_3.1.1

Page 33 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

Running the model with ECLISAs stated in section 5, the seasonac enseebce forecast is caunched 3 tiees per eonth. Two days afer
each start date (so that ERA5T fces are a‫ﯡ‬aicabce and the coupced initiacization run has beensuccessfuccy extended), a cron job-scheducer executes the scripts lance_param_F_bln.sh andlance_param_F_trn.sh on becenos and taranis eachines.
This frst script generates one param_F???????? fce for each eeeber of the enseebce forecast.
It contains the initiac date and the end date of the forecast. It acso sources the generic param_F fce
which is the eain fce to caunch a forecast with ECLIS (see section 5.2). At the end oflance_param_F_???.sh, the current script param_F???????? is executed. So, it sets ‫ﯡ‬acues
for ECLIS paraeeters and then sources the ECLIS install script.
Sieicarcy, the generic param_H fce is the eain fce to caunch a re-forecast with ECLIS. These genericfces inccude the coepcete cinks to the executabces and naeecists, as wecc as pcugins, used for Systee
8.

Post-processing (XIOS) and transfer to ECMWF
The post-processing step recies on the XIOS output eanager (see sections 2.7 and 5.3). Acc xec fcesused as input to XIOS for Systee 8 are stored on the belenos and taranis coeputers in thefoccowing path:/home/gmgec/mrga/copernicusps/SAVE/opers8/XIOSXIOS thus generates directcy the fecds for the ateosphere, surface and ocean on the required 1°x1°
grid with 180 catitudes and 360 congitudes starting at 89.5°S, 0.5°W and extending northwards andeastwards. The data (one fce per fecd and per eonth) are stored both cocaccy on the HPC eachine:belenos:/scratch/work/copernicusps/outputs/HINDCAST/$EXPID/ andtaranis:/scratch/work/copernicusps/outputs/HINDCAST/$EXPID/
and upcoaded on the Météo-France storage capacity:
hendrix:/home/copernicusps/SYS8/HINDCAST
The data foreat is netcdf (with coepression). The fces are transferred by the ectrans uticity to
ECMWF on:cca:/sc1/tcwork/zac/lb/copernicus/s8
For each starting eonth and eeeber, a directory is created with a naee containing the date and theeeeber nueber. For exaepce directory 199303_001 contains the fces created for eeeber 1 of the

C3S_D330_3.1.1

Page 34 of 36

Copernicus Ccieate Change Ser‫ﯡ‬ice

March 1993 re-forecasts. Note that the start dates are in this case two starts end of February as wecc
as 01 Mar. There is one tar fce in each directory containing 55 netcdf fces (one for each ‫ﯡ‬ariabce).

C3S_D330_3.1.1

Page 35 of 36

Copernicus Climate Change Service

ECMWF, Shinfield Park, Reading RG2 9AX, UK
Contact: isabella.mazza@ecmwf.int

climate.copernicus.eu

copernicus.eu

ecmwf.int