https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/

Adam Phillips and Clara DeserJohn Fasullo, Isla Simpson, Dave Schneider
Climate Analysis Section, NCARCGD Seminar
17 November 2020

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/

Adam Phillips and Clara DeserJohn Fasullo, Isla Simpson, Dave Schneider
Climate Analysis Section, NCARCGD Seminar
17 November 2020

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/

An automated analysis tool anddata repository for exploring
forced and internal components
of climate variability and change.

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/

An automated analysis tool anddata repository for exploring
forced and internal components
of climate variability and change.

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/

An automated analysis tool anddata repository for exploring
forced and internal components
of climate variability and change.

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/•

How well does my model simulate: ENSO? AMV?
PDV? AMOC? NAO? Variability in general?

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/•

How well does my model simulate: ENSO? AMV?
PDV? AMOC? NAO? Variability in general?

•

How does my model compare with other models(and are there true structural differences)?

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/•

How well does my model simulate: ENSO? AMV?
PDV? AMOC? NAO? Variability in general?

•

How does my model compare with other models(and are there true structural differences)?

•

Does climate change affect internal variability?

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/•

How well does my model simulate: ENSO? AMV?
PDV? AMOC? NAO? Variability in general?

•

How does my model compare with other models(and are there true structural differences)?

•

Does climate change affect internal variability?

•

What are the relative contributions of internal variability
and forced climate change to long-term trends?

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/•

How well does my model simulate: ENSO? AMV?
PDV? AMOC? NAO? Variability in general?

•

How does my model compare with other models(and are there true structural differences)?

•

Does climate change affect internal variability?

•

What are the relative contributions of internal variability
and forced climate change to long-term trends?

Ø Initial-condition Large Ensembles

Initial-condition Large Ensembles

Initial-condition Large Ensembles
30 March 2020
Deser et al.

US CLIVAR Working Group on Large Ensembles

What are they? Why are they useful?
How large do they need to be?
How are they best designed?
Emerging applications and future directions?

The CESM1 Large Ensemble Project

Year 401

Historical & RCP8.5
radiative forcing
simulation #1

1850 control

2100

The CESM1 Large Ensemble Project

Year 401

1850 control

Historical & RCP8.5
radiative forcing
1920

40 simulations(parallel worlds)
2100

Different atmospheric initial states Perturbed by order 10-14 K
Same ocean, ice & land initial states (1 Jan 1920)

The CESM1 Large Ensemble ProjectInternal climate
variability+
Year 401

1850 control

Historical & RCP8.5
radiative forcing
1920

40 simulations(parallel worlds)
2100

Different atmospheric initial states Perturbed by order 10-14 K
Same ocean, ice & land initial states (1 Jan 1920)

The CESM1 Large Ensemble Project+
Year 401

1850 control

Historical & RCP8.5
radiative forcing
1920

Internal climate
variability

40 simulations(parallel worlds)

2100
spread is not predictable! (beyond ~ 10 years)

Different atmospheric initial states Perturbed by order 10-14 K
Same ocean, ice & land initial states (1 Jan 1920)

(MMLEA)
US CLIVAR Working Group on Large Ensembles(Flavio Lehner)

7 CMIP5 models(10 CMIP6 models)

https://www.cesm.ucar.edu/projects/community-projects/MMLEA/

Like the original CVDP•••

Computes modes of variability, trends, and climate indices.
All output saved to a data repository for later use.
User specifies the data sets (models & observations).

New for the CVDP-LE•••

Computes ensemble mean and ensemble spread.
Quantitative comparison to observations (rank metrics).
Comprehensive User’s Guide.

User’s Guide (35 pages)
•
•
•
•
•
•
•

Tutorial and teaching resource

Background on internal climate variability
Utility of Large Ensembles
Diagnostics and metrics (fully referenced)
Treatment of observational uncertainty
Two views: Ensemble Summary vs. Individual Members
Interpretation of plots and metrics
Best practices and tips for applying the package

User’s Guide (35 pages)
•
•
•
•
•
•
•

Tutorial and teaching resource

Background on internal climate variability
Utility of Large Ensembles
Diagnostics and metrics (fully referenced)
Treatment of observational uncertainty
Two views: Ensemble Summary vs. Individual Members
Interpretation of plots and metrics
Best practices and tips for applying the package

Diagnostics Overview

Diagnostics Overview

User-specified title

Diagnostics Overview
Link toUser’s Guide

User-specified title

Diagnostics Overview
Link toUser’s Guide

Two waysof viewing
the output

User-specified title

Diagnostics Overview
Link toUser’s Guide

User-specified title
Summary Metrics

Two waysof viewing
the output

Diagnostics Overview
Link toUser’s Guide

User-specified title
Summary Metrics

Two waysof viewing
the output

VariablesDiagnostics Overview
Link toUser’s Guide

User-specified title
Summary Metrics

Two waysof viewing
the output

Variables

SeasonsAdam
• Overview of the development process• Technical guidance• Webpage and data repository
Clara• Interpretation of plots and metrics• Application to the CMIP5 and CMIP6 archives

CVDP-LE Development Path
- Feb 2019-present, started w/CVDP codebase- Overriding development requirement:
Ease of use- Implemented feature requests (subset):•

Allow user to specify unlimited number of observations/simulations•

Allow different time periods for each input dataset•

Allow missing model data•

Compute pattern correlations•

Allow different units, grids and variable names•

Compute the % of time observations falls within model spread•

Allow no observations•

Allow different number of ensemble members•

Form differences, no matter the grid•Comment the code extensively•

Display number of valid members per ensemble•

Provide documentation and written directions on adding a metric•

Provide clear error messages•

Output graphics should be publication quality•

Package should avoid erroring out•

Add consistent titles and statistics to each plot•

Titles/statistics should rarely overlap one another•Calculate ensemble metrics, and show via graphics•

Modularize as much as it makes sense to do so•

Output all calculations (including ensemble means) to netCDF files

Insert Picture
Showing Long Path

Metrics & Coding details
Metrics- 28,000 lines of code, 26 scripts, ~30 new functions, ~1000 plots produced- New website, new User’s Guide, new readme file- 250 run MMLEA 1950-2018 comparison took 12 hours w/parallelization
384 run CMIP6 Historical 1900-2014 comparison took 18 hours
38 member CESM Control Comparison (100yr each) took 9 hours
Coded in NCL- Why on earth was NCL used; wasn’t NCL developed in the 90’s?
Highlighted Coding Improvements/Additions- Model file identification coding script completely rewritten.
- Ensemble graphics/metrics + biases from observations are calculated.
- Data only mode.
- Code is highly modularized; calculations separated from graphics.
Modularity makes it easier to strip code out for outside applications.
- Package will read in previously created metrics to save time.

Running the CVDP-LE• Similar to the CVDP, 3 text files need to be set up for CVDP-LE to run:driver.ncl script (sets options)
namelist (set paths of simulations)
namelist_obs (sets specific observations to be used)• Software requirements: NCL, python and Image Magick.
• Analyzes monthly CESM/CMIP timeseries files.

Running the CVDP-LE
driver.ncl

Running the CVDP-LE

new entry required: Identify the ensemble

namelist

There are no restrictions on the number of models listed.
Ensemble sizes can be different, as can the analyzed years.

Running the CVDP-LE
namelist

The same simulation can be specified multiple times.

Running the CVDP-LE
namelist_obs

There are no restrictions on the number of observations listed

Running the CVDP-LE• To submit the package: “ncl driver.ncl”• Can be run on any machine that has NCL, Image
Magick and python installed. Regularly run on
NCAR-CGD/CISL processing machines.

Using CVDP-LE Outputs
Even if you do not wish to
run the package, you candownload comparison
results via the website.

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/Using CVDP-LE Outputs
Even if you do not wish to
run the package, you candownload comparison
results via the website.
Every model run has its own
netCDF file containing all the
calculated metrics.

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/Selected Graphics
CMIP6 Historical Comparison, 1900-2014
Ensemble Metrics (subset)

Pattern Correlation with Reference Observations

Selected Graphics
CMIP6 Historical Comparison, 1900-2014
Individual Members View

Selected Graphics

CESM2/CESM1 piControl
100yr Slice Comparison
Individual Members View

Selected Graphics
CESM2/CESM1 piControl
100yr Slice Comparison
Ensemble Summary View

Available Now

CVDP-LE now available for use on NCAR CGD or CISL machines
CGD: /home/asphilli/CESM-diagnostics/CVDP-LE/Release/v1.0.0
CISL: /glade/u/home/asphilli/CESM-diagnostics/CVDP-LE/Release/v1.0.0
Code will be available on github by the end of this week.
https://github.com/NCAR/CVDP-LE

Diagnostics Overview

Individual
Members

Ensemble
Summary

Metrics

Diagnostics Overview

Individual
Members

Ensemble
Summary

Ensemble
Summary

Metrics

Diagnostics Overview

Individual
Members

Ensemble
Summary

Ensemble
Summary

Metrics

1) Spatial patterns2) Time series & derived quantities

CCMA (40 members)Model
1950-2018

Observations (ERA20C)

Observations45%
39/48/57%

DifferenceObs rank within ensemble

Difference

Rank

43%

% area withvalues of
10-90%

hPa

hPa
0

20 80

100 %

CCMA (40 members)Model
1950-2018

Observations (ERA20C)

Observations45%
39/48/57%

DifferenceObs rank within ensemble

Difference

Rank

43%

% area withvalues of
10-90%

hPa

hPa
0

20 80

100 %

CCMA (40 members)Model
1950-2018

Observations (ERA20C)

Observations45%
39/48/57%

DifferenceObs rank within ensemble

Difference

Rank

43%

% area withvalues of
10-90%hPa

hPa
0

NAO = EOF1 SLP (20-80N, 90W-40E) Hurrell and Deser (2009)
Calculated for each ensemble member, then averaged.
All graphics, data and metrics saved to a repository.

20 80

100 %

Distribution of % variance explained(10th/50th/90th percentiles)
CCMA (40 members)Model
1950-2018

Observations (ERA20C)

Observations45%
39/48/57%

DifferenceObs rank within ensemble

Difference

Rank

43%

% area withvalues of
10-90%hPa

hPa
0

NAO = EOF1 SLP (20-80N, 90W-40E) Hurrell and Deser (2009)
Calculated for each ensemble member, then averaged.
All graphics, data and metrics saved to a repository.

20 80

100 %

Distribution of % variance explained(10th/50th/90th percentiles)
CCMA (40 members)Model
1950-2018

Observations (ERA20C)

Observations45%
39/48/57%

DifferenceObs rank within ensemble

Difference

Rank

43%

% area withvalues of
10-90%hPa

hPa
0Pattern correlation
with observations.

NAO = EOF1 SLP (20-80N, 90W-40E) Hurrell and Deser (2009)
Calculated for each ensemble member, then averaged.
All graphics, data and metrics saved to a repository.

20 80

100 %

Distribution of % variance explained(10th/50th/90th percentiles)
CCMA (40 members)Model
1950-2018

Observations (ERA20C)

Observations45%
39/48/57%

DifferenceObs rank within ensemble

Difference

Rank

43%

% area withvalues of
10-90%hPa

hPa
0Pattern correlation
with observations.

NAO = EOF1 SLP (20-80N, 90W-40E) Hurrell and Deser (2009)
Calculated for each ensemble member, then averaged.
All graphics, data and metrics saved to a repository.

20 80

100 %

Distribution of % variance explained(10th/50th/90th percentiles)
CCMA (40 members)Model
1950-2018

Observations (ERA20C)

Observations45%
39/48/57%

DifferenceObs rank within ensemble

Difference

Rank

43%

% area withvalues of
10-90%hPa

hPa
0Pattern correlation
with observations.

NAO = EOF1 SLP (20-80N, 90W-40E) Hurrell and Deser (2009)
Calculated for each ensemble member, then averaged.
All graphics, data and metrics saved to a repository.

20 80

100 %

White areas: Observations
lie within 20-80th % ofmodel ensemble spread.
Obs < any member
Obs > any member

Distribution of % variance explained(10th/50th/90th percentiles)
CCMA (40 members)Model
1950-2018

Observations (ERA20C)

Observations45%
39/48/57%

DifferenceObs rank within ensemble

Difference

Rank

43%

% area withvalues of
10-90%hPa

hPa
0Pattern correlation
with observations.

NAO = EOF1 SLP (20-80N, 90W-40E) Hurrell and Deser (2009)
Calculated for each ensemble member, then averaged.
All graphics, data and metrics saved to a repository.

20 80

100 %

White areas: Observations
lie within 20-80th % ofmodel ensemble spread.
Obs < any member
Obs > any member

Distribution of % variance explained(10th/50th/90th percentiles)
CCMA (40 members)Model
1950-2018

Observations (ERA20C)

Observations45%
39/48/57%

DifferenceObs rank within ensemble

Difference

Rank

43%

% area withvalues of
10-90%hPa

hPa
0Pattern correlation
with observations.

NAO = EOF1 SLP (20-80N, 90W-40E) Hurrell and Deser (2009)
Calculated for each ensemble member, then averaged.
All graphics, data and metrics saved to a repository.

20 80

100 %

White areas: Observations
lie within 20-80th % ofmodel ensemble spread.
Obs < any member
Obs > any member

Distribution of % variance explained(10th/50th/90th percentiles)
CCMA (40 members)Model
1950-2018

Observations (ERA20C)

Observations45%
39/48/57%

Difference

10-90th %Fractional Area
Obs rank within ensemble

Difference

Rank

43%

% area withvalues of
10-90%hPa

hPa
0Pattern correlation
with observations.

NAO = EOF1 SLP (20-80N, 90W-40E) Hurrell and Deser (2009)
Calculated for each ensemble member, then averaged.
All graphics, data and metrics saved to a repository.

20 80

100 %

White areas: Observations
lie within 20-80th % ofmodel ensemble spread.
Obs < any member
Obs > any member

CCMA(40 members)

CSIRO(30 members)

GFDL-CM3(20 members)

GFDL-ESM2M(30 members)

MPI(70 members)

NCAR CESM1(40 members)

SMHI_KNMI(16 members)

CMIP5 Multi-Model
Large Ensemble ArchiveCCMA(40 members)

CSIRO(30 members)

GFDL-CM3(20 members)

GFDL-ESM2M(30 members)

MPI(70 members)

NCAR CESM1(40 members)

SMHI_KNMI(16 members)

CMIP5 Multi-Model
Large Ensemble ArchiveCCMA(40 members)

CSIRO(30 members)

GFDL-CM3(20 members)

GFDL-ESM2M(30 members)

MPI(70 members)

NCAR CESM1(40 members)

SMHI_KNMI(16 members)

CMIP5 Multi-Model
Large Ensemble ArchiveCCMA(40 members)

CSIRO(30 members)

GFDL-CM3(20 members)

GFDL-ESM2M(30 members)

MPI(70 members)

NCAR CESM1(40 members)

SMHI_KNMI(16 members)

CMIP5 Multi-Model
Large Ensemble ArchiveCCMA(40 members)

CMIP5 Multi-Model
Large Ensemble Archive

CSIRO(30 members)

GFDL-CM3(20 members)

GFDL-ESM2M(30 members)

MPI(70 members)

NCAR CESM1(40 members)

SMHI_KNMI(16 members)

MPI is the most realistic:largest (60%) areal coverageof observations falling within
10th – 90th % of model spread.

CCMA(40 members)

CMIP5 Multi-Model
Large Ensemble Archive

CSIRO(30 members)

GFDL-CM3(20 members)

GFDL-ESM2M(30 members)

MPI(70 members)

NCAR CESM1(40 members)

SMHI_KNMI(16 members)

MPI is the most realistic:largest (60%) areal coverageof observations falling within
10th – 90th % of model spread.
But need to check that
amplitude of internal variability
is realistic (compare s maps).

Diagnostics Overview

Individual
Members

Ensemble
Summary

Ensemble
Summary

Metrics

1) Spatial patterns2) Time series & derived quantities

CMIP5 Multi-Model
Large Ensemble Archive
Observations
Model
Ensemble Mean

Ensemble Spread
25-75%
10-90%

Global mean Air Temperature

CMIP5 Multi-Model
Large Ensemble Archive
Observations
Model
Ensemble Mean

Ensemble Spread
25-75%
10-90%

Global mean Air Temperature

CMIP5 Multi-Model
Large Ensemble Archive
Observations
Model
Ensemble Mean

Ensemble Spread
25-75%
10-90%

Global mean Air Temperature

Distribution of trend values(10th/50th/90th percentiles).

CMIP5 Multi-Model
Large Ensemble Archive
Percentageof time that
Observations
Obs falls Model
within 10-90%
of modelspread.
Ensemble
Mean

Ensemble Spread
25-75%
10-90%

Global mean Air Temperature

Distribution of trend values(10th/50th/90th percentiles).

CMIP5 Multi-Model
Large Ensemble Archive

Global mean Air Temperature

Distribution of trend values(10th/50th/90th percentiles).

Percentageof time that
Observations
Obs falls Model
within 10-90%
of modelspread.
Ensemble
Mean

Ensemble Spread
25-75%
10-90%

Color-codedby modelCMIP5 Multi-Model
Large Ensemble Archive

Global mean Air Temperature

Distribution of trend values(10th/50th/90th percentiles).

Percentageof time that
Observations
Obs falls Model
within 10-90%
of modelspread.
Ensemble
Mean

Ensemble Spread
25-75%
10-90%All graphics, data
and metrics saved
to a repository.

Color-codedby modelCMIP6Large Ensembles
Observations
Model
Ensemble Mean

Ensemble Spread
25-75%
10-90%All graphics, data
and metrics saved
to a repository.

Lag Autocorrelations(0 - 48 months)
Niño3.4 SST Index(detrended)CMIP6Large Ensembles
Observations
Model
Ensemble Mean

Ensemble Spread
25-75%
10-90%All graphics, data
and metrics saved
to a repository.

Lag Autocorrelations(0 - 48 months)
Niño3.4 SST Index(detrended)CMIP6Large Ensembles
Observations
Model
Ensemble Mean

Ensemble Spread
25-75%
10-90%All graphics, data
and metrics saved
to a repository.

Power Spectra
Niño3.4 SST Index(detrended)Diagnostics Overview
Individual
Members

Ensemble
Summary

Metrics

Diagnostics Overview
Individual
Members

Ensemble
Summary

Metrics

1) Graphics
2) Tables

CMIP5 Multi-Model Large Ensemble Archive
Winter NAO (Model vs. ERA-20C 1950-2018)
Pattern Correlations

Spatial RMS Differences

Longer bars: 10th / 50th / 90th percentiles
All graphics, data and metrics saved to a repository.

Observational
Uncertainty
CERA20C (1950-2018)
ERA-I (1979-2018)
MERRA2 (1980-2017)

CMIP5 Multi-Model Large Ensemble Archive
Winter NAO (Model vs. ERA-20C 1950-2018)
Pattern Correlations

Spatial RMS Differences

Longer bars: 10th / 50th / 90th percentiles
All graphics, data and metrics saved to a repository.

Observational
Uncertainty
CERA20C (1950-2018)
ERA-I (1979-2018)
MERRA2 (1980-2017)

CMIP5 Multi-Model Large Ensemble Archive
Winter NAO (Model vs. ERA-20C 1950-2018)
Pattern Correlations

Spatial RMS Differences

CSIRO 90th %< MPI 10th %Models are
structurally
different.
See also Fasullo
et al. (2020)

Longer bars: 10th / 50th / 90th percentiles
All graphics, data and metrics saved to a repository.

Observational
Uncertainty
CERA20C (1950-2018)
ERA-I (1979-2018)
MERRA2 (1980-2017)

CMIP5 Multi-Model Large Ensemble Archive
Winter NAO (Model vs. ERA-20C 1950-2018)
Pattern Correlations

Spatial RMS Differences

CSIRO 90th %< MPI 10th %CSIRO 10th %> MPI 90th %Models are
structurally
different.

Models are
structurally
different.

See also Fasullo
et al. (2020)

Longer bars: 10th / 50th / 90th percentiles
All graphics, data and metrics saved to a repository.

CMIP6 Large Ensembles
Winter NAO (Model vs. ERA-20C 1900-2014)

Observational
Uncertainty
CERA20C (1901-2014)
ERA-I (1979-2014)
MERRA2 (1980-2014)

CMIP6 Large Ensembles
Winter NAO (Model vs. ERA-20C 1900-2014)
Pattern Correlations

Spatial RMS Differences

Longer bars: 10th - 50th - 90th percentiles
All graphics, data and metrics saved to a repository.

Observational
Uncertainty
CERA20C (1901-2014)
ERA-I (1979-2014)
MERRA2 (1980-2014)

CMIP6 Large Ensembles
Winter NAO (Model vs. ERA-20C 1900-2014)
Pattern Correlations

Spatial RMS Differences

Lowestpatterncorrelation

Highestspatial rmse
Longer bars: 10th - 50th - 90th percentiles
All graphics, data and metrics saved to a repository.

11 Metrics(4 ENSO, AMV, PDV, NAO, PNA,
SAM, s SST, s Land Precip)

11 Metrics(4 ENSO, AMV, PDV, NAO, PNA,
SAM, s SST, s Land Precip)

Longer bars: 10th - 50th - 90th percentiles

11 Metrics(4 ENSO, AMV, PDV, NAO, PNA,
SAM, s SST, s Land Precip)

Compare distributions toassess whether models are
structurally different orwhether their differences
are within the noise ofinternal variability (andobservational uncertainty)
See Fasullo et al. (2020)

Longer bars: 10th - 50th - 90th percentiles

11 Metrics(4 ENSO, AMV, PDV, NAO, PNA,
SAM, s SST, s Land Precip)

Compare distributions toassess whether models are
structurally different orwhether their differences
are within the noise ofinternal variability (andobservational uncertainty)
See Fasullo et al. (2020)

Longer bars: 10th - 50th - 90th percentiles

Diagnostics Overview
Individual
Members

Ensemble
Summary

Metrics

1) Graphics
2) Tables

Metrics Tables
Pattern Correlations
and Spatial RMSE
10th / 50th / 90th percentiles
for each model.
Color coded for ease-of-use.

Metrics Tables
Pattern Correlations
and Spatial RMSE
10th / 50th / 90th percentiles
for each model.
Color coded for ease-of-use.

Sortable(here Mean Score is used).

Metrics Tables
Pattern Correlations
and Spatial RMSE
10th / 50th / 90th percentiles
for each model.
Color coded for ease-of-use.

Sortable(here Mean Score is used).
All graphics, data
and metrics saved
to a repository.

Metrics Tables
Pattern Correlations
and Spatial RMSE
10th / 50th / 90th percentiles
for each model.
Color coded for ease-of-use.

Sortable(here Mean Score is used).
All graphics, data
and metrics saved
to a repository.

Also available for every
member of every model.

Some Application Ideas(User’s Guide)

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/•Multiple time periods tosee if modes of variabilitychange with time.

•

Subsets of ensemble
members to assessrobustness.

•

Filter the data toinvestigate dependenceon time scale.

•

Use an “ensemble” ofshorter segments from acontrol simulation.

Some Application Ideas(User’s Guide)

https://www.cesm.ucar.edu/working_groups/CVC/cvdp-le/•Multiple time periods tosee if modes of variabilitychange with time.

•

Subsets of ensemble
members to assessrobustness.

Enjoy exploring! • Filter the data to
Suggestions welcome. investigate dependenceon time scale.

•

Use an “ensemble” ofshorter segments from acontrol simulation.