# Taking climate model evaluation to the next level

Veronika Eyring, Peter M. Cox, Gregory M. Flato, Peter J. Gleckler, Gab Abramowitz, Peter Caldwell, William D. Collins, Bettina K. Gier, Alex D. Hall, Forrest M. Hoffman, et al.

► To cite this version:

Veronika Eyring, Peter M. Cox, Gregory M. Flato, Peter J. Gleckler, Gab Abramowitz, et al. 10.1038/s41558-018-0355-y. insu-03727027

HAL Id: insu-03727027

https://insu.hal.science/insu-03727027v1

Submitted on 2 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Taking climate model evaluation to the next level

## Journal Article

### Author(s):

Eyring, Veronika; Cox, Peter M.; Flato, Gregory M.; Gleckler, Peter J.; Abramowitz, Gab; Caldwell, Peter; Collins, William D.; Gier, Bettina K.; Hall, Alex D.; Hoffman, Forrest M.; Hurtt, George C.; Jahn, Alexandra; Jones, Chris D.; Klein, Stephen A.; Krasting, John P.; Kwiatkowski, Lester; Lorenz, Ruth; Maloney, Eric; Meehl, Gerald A.; Pendergrass, Angeline G.; Pincus, Robert; Ruane, Alex C.; Russell, Joellen L.; Sanderson, Benjamin M.; Santer, Benjamin D.; Sherwood, Steven C.; Simpson, Isla R.; Stouffer, Ronald J.; Williamson, Mark S.

### Publication date:

2019-02

### Permanent link:

https://doi.org/10.3929/ethz-b-000323491

### Rights / license:

Creative Commons Attribution 3.0 Unported

### Originally published in:

Nature Climate Change 9(2), https://doi.org/10.1038/s41558-018-0355-y
# Taking climate model evaluation to the next level

Veronika Eyring<sup>1,2*</sup>, Peter M. Cox<sup>3</sup>, Gregory M. Flato<sup>4</sup>, Peter J. Gleckler<sup>5</sup>, Gab Abramowitz<sup>6</sup>, Peter Caldwell<sup>5</sup>, William D. Collins<sup>7,8</sup>, Bettina K. Gier<sup>1,2</sup>, Alex D. Hall<sup>9</sup>, Forrest M. Hoffman<sup>10,11</sup>, George C. Hurtt<sup>12</sup>, Alexandra Jahn<sup>13</sup>, Chris D. Jones<sup>14</sup>, Stephen A. Klein<sup>5</sup>, John P. Krasting<sup>15</sup>, Lester Kwiatkowski<sup>16</sup>, Ruth Lorenz<sup>17</sup>, Eric Maloney<sup>18</sup>, Gerald A. Meehl<sup>19</sup>, Angeline G. Pendergrass<sup>19</sup>, Robert Pincus<sup>18</sup>, Alex C. Ruane<sup>20</sup>, Joellen L. Russell<sup>21</sup>, Benjamin M. Sanderson<sup>19</sup>, Benjamin D. Santer<sup>5</sup>, Steven C. Sherwood<sup>6</sup>, Isla R. Simpson<sup>19</sup>, Ronald J. Stouffer<sup>21</sup> and Mark S. Williamson<sup>3</sup>

Earth system models are complex and represent a large number of processes, resulting in a persistent spread across climate projections for a given future scenario. Owing to different model performances against observations and the lack of independence among models, there is now evidence that giving equal weight to each available model projection is suboptimal. This Perspective discusses newly developed tools that facilitate a more rapid and comprehensive evaluation of model simulations with observations, process-based emergent constraints that are a promising way to focus evaluation on the observations most relevant to climate projections, and advanced methods for model weighting. These approaches are needed to distil the most credible information on regional climate changes, impacts, and risks for stakeholders and policy-makers.

The Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) concluded that the warming of the climate system is unequivocal and human influence on the climate system is clear<fnref n="1" />. Observed increases of greenhouse gases have contributed significantly to warming of the atmosphere and ocean, sea-ice decline and sea-level rise. The size and rapidity of these changes is concerning. Human-caused climate change is already affecting many aspects of societies and ecosystems. These impacts will become more visible and more serious in the twenty-first century. It should, therefore, be an international priority to improve our understanding of the climate system, and to reduce current uncertainties in projections of future change. This will rely on information from theory, observations, and Earth system model (ESM) simulations that are coordinated as part of the World Climate Research Programme (WCRP) Coupled Model Intercomparison Project (CMIP; refs. ²⁻⁵). CMIP is now in its sixth phase (CMIP6)⁵ and is confronted with a number of new challenges. Compared to CMIP5, an increased number of institutions participate in CMIP6, many with multiple model versions. The latest generation of climate models feature increases in spatial resolution, improvements in physical parameterizations (in the representation of clouds, for example) and inclusion of additional Earth system processes (such as nutrient limitations on the terrestrial carbon cycle)

and components (such as ice sheets). These additional processes are needed to represent key feedbacks that affect climate change, but are also likely to increase the spread of climate projections across the multimodel ensemble. This escalates the need for innovative and comprehensive model evaluation approaches.

CMIP provides the basis for multimodel evaluation and has, over the years, revealed a variety of systematic differences between models and observations, with many persisting from one model generation to the next⁶,⁷. An important issue that remains to be fully addressed is the extent to which model errors affect the quality of climate projections and subsequent impact assessments⁸. Traditionally, many climate projections are shown as multimodel averages in the peer-reviewed literature and IPCC reports, with the spread across models presented as a measure of projection uncertainty⁹. There is now emerging evidence that weighting based on model performance may improve projections for specific applications¹⁰⁻¹². A further complication in devising model weighting approaches is that many CMIP models share components, or are variants of another model in the ensemble, and hence are not truly independent¹²⁻¹⁶. This has the potential to bias the multimodel results in ways that are only beginning to be explored. The lack of independence challenges the notion of a 'model democracy', in which each model is weighted equally¹⁷.

<footnote n="1">¹Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Institut für Physik der Atmosphäre, Oberpfaffenhofen, Germany. ²University of Bremen, Institute of Environmental Physics (IUP), Bremen, Germany. ³College of Engineering, Mathematics and Physical Sciences, University of Exeter, Exeter, UK.</footnote>

<footnote n="4">⁴Canadian Centre for Climate Modelling and Analysis, Environment and Climate Change Canada, Victoria, British Columbia, Canada. ⁵Program for Climate Model Diagnosis and Intercomparison (PCMDI), Lawrence Livermore National Laboratory, Livermore, CA, USA. ⁶Climate Change Research Centre and ARC Centre of Excellence for Climate Extremes, University of New South Wales, Sydney, New South Wales, Australia. ⁷Climate and Ecosystem Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA. ⁸Department of Earth and Planetary Science, University of California, Berkeley, CA, USA. ⁹University of California, Los Angeles, CA, USA. ¹⁰Computational Earth Sciences Group and Climate Change Science Institute, Oak Ridge National Laboratory, Oak Ridge, TN, USA. ¹¹Department of Civil and Environmental Engineering, University of Tennessee, Knoxville, TN, USA. ¹²Department of Geographical Sciences, University of Maryland, College Park, MD, USA. ¹³Department of Atmospheric and Oceanic Sciences and Institute of Arctic and Alpine Research, University of Colorado, Boulder, CO, USA. ¹⁴Met Office Hadley Centre, Exeter, UK. ¹⁵Geophysical Fluid Dynamics Laboratory, NOAA, Princeton, NJ, USA. ¹⁶Laboratoire de Météorologie Dynamique (LMD), IPSL, Paris, France. ¹⁷Institute for Atmospheric and Climate Science, ETH Zurich, Zurich, Switzerland. ¹⁸Cooperative Institute for Research in Environmental Sciences, University of Colorado, Boulder, CO, USA. ¹⁹National Center for Atmospheric Research (NCAR), Boulder, CO, USA. ²⁰NASA Goddard Institute for Space Studies, New York, NY, USA. ²¹University of Arizona, Tucson, AZ, USA. *e-mail: veronika.eyring@dlr.de</footnote>
<figure><img src="image_2.png" /><figcaption>**Fig. 1** | Annual mean SST error from the CMIP5 multi-model ensemble. Systematic errors are particularly visible in the upwelling zones denoted by black rectangles off the west coasts of each continent. Figure reproduced from ref. ²³, Wiley.</figcaption></figure>

The growing number and complexity of models, the expanding suite of outputs they produce, the multitude of downstream applications and the growing availability of observational datasets drive a need for more routine and systematic evaluation, utilizing a comprehensive set of existing model performance metrics and diagnostics. Newly developed CMIP evaluation tools¹⁸,¹⁹ will ultimately enhance our ability to identify model errors, to investigate their causes and to quantify and potentially reduce projection uncertainties.

In this Perspective, we summarize key advances since AR5 and key scientific opportunities for improving climate model analyses that will be assessed in the AR6. Our focus is on gaps in the understanding of systematic errors, the development of CMIP model evaluation tools, emergent constraints and weighting methods. We also address the need for more user- and policy-oriented model evaluation at the regional scale required for impact studies. Finally, we discuss how the scientific community might provide more robust climate model information and more tightly constrained model projections.

## From model errors to understanding processes

Comparing model results to observations provides insight into the quality of model simulations and the way in which various processes are represented. Comparisons with observations can reveal shortcomings in individual models and systematic errors in a large multi-model ensemble⁷,²⁰. An example of a systematic error is the excessive simulated band of precipitation in the tropical Pacific south of the Equator, a feature not present in observations. Taken together with the usually correctly simulated climatological intertropical convergence zone (ITCZ) precipitation maximum that stretches across the tropical Pacific north of the Equator, this systematic splitting of tropical Pacific rainfall into two discrete branches is commonly referred to as the double ITCZ. Other examples of systematic errors include a dry Amazon bias, a warm bias in the eastern parts of tropical ocean basins, differences in the magnitude and frequency of El Niño and La Niña events, biases in sea surface temperatures (SSTs) in the Southern Ocean, a warm and dry bias of land surfaces during summer, and differences in the position of the Southern Hemisphere atmospheric jet⁷.

One major challenge is that it is often not possible to attribute a specific cause to a specific systematic error. For example, it has been suggested that the systematic warm bias in the upwelling zones off the west coasts of each continent (see Fig. 1) is associated with biases in the representation of stratocumulus clouds²¹ and boundary layer convection²² in these regions. However, other studies suggest that the root cause of this warm bias is the representation of ocean upwelling and its forcing from surface winds²³. An additional complication is that a regional difference between the simulation and observations may be a consequence of errors that occur far from the region in question, and are manifested via teleconnections. Certain regional SST biases, for example, are related to biases in other ocean basins and to aspects of the large-scale ocean overturning circulation²⁴. In some cases, although the link between a particular bias and some physical process may seem robust, the specific cause of the bias — as well as its remedy — may remain elusive.

But there are also compelling examples of how a multimodel analysis of a particular systematic bias can lead to a clearer understanding of underlying causes. One systematic bias revealed in the evaluation of CMIP5 models was the apparent difference between the observed and modelled global mean surface temperature increase in the early twenty-first century⁷. These differences motivated a range of targeted analyses exploring model performance, internal variability, external forcing and observational uncertainty²⁵. Although the magnitude of the slowdown differs slightly depending on which global observational dataset is analysed, this focused effort revealed that the observed slowdown was due to a combination of factors, chiefly involving internally generated decadal-tim-escale variability in the tropical Pacific²⁶,²⁷ and the missing effects of a series of moderate volcanic eruptions²⁸. Averaging the time series across a collection of coupled model simulations strongly reduces the effects of internally generated variability, more clearly revealing the underlying externally forced response. There is, therefore, a mismatch between the precise observed sequence of variability and the smooth evolution of temperature in the multimodel mean. Models initialized with observations from the years immediately before the early twenty-first century slowdown were able to capture aspects of the observed change in warming rate after 2000²⁹.
<figure><img src="image_3.png" /><figcaption>**Fig. 2** | Schematic diagram of the workflow for CMIP Evaluation Tools running alongside the ESGF. The tools will routinely produce a broad characterization of model performance for CMIP model output utilizing relevant observations and reanalyses and running alongside the ESGF infrastructure. MMM: multi-model mean; WGCM: Working Group on Coupled Modelling; WGNE: Working Group on Numerical Experimentation; ECVs: essential climate variables. Figure reproduced from ref. ³⁸, EGU, under a Creative Commons license.</figcaption></figure>

These results highlight the importance of using different simulation frameworks (for example, coupled simulations and decadal predictions initialized with observations) to understand the causes of differences between modelled and observed climate changes. Stronger observed warming since 2014, which is replicated in initialized model predictions for the period after 2014³⁰,³¹, adds to the evidence that the weaker warming before 2014 had a large contribution from internal climate variability.

A related question is the extent to which observational uncertainties and inhomogeneities³² are hampering model evaluation. Just as efforts to improve models continue, there is a parallel effort to improve observationally based datasets. Even for a very basic climate quantity such as temperature, this involves refined corrections for biases and incomplete global coverage in the raw surface observations³³ and corrections for biases in satellite retrievals³⁴. Observations are also critically important in model tuning³⁵,³⁶, which should be clearly documented and taken into account in model evaluation studies. One difficulty in comparing models against observations is posed by inconsistency in the sampling or definition of the quantities compared (for example, model data may be daily averages whereas satellite samples may be for a certain time of day). This inconsistency can be addressed by incorporating simulators of specific instruments into climate models³⁷.

## New CMIP model evaluation tools

The scope of model evaluation has expanded dramatically in recent years. Well-established aspects of model evaluation are now becoming more routine, results are available more rapidly than for CMIP5, enhancing their value for model analysts and developers³⁸. A key development for CMIP6 is the availability of the Earth System Model Evaluation Tool (ESMValTool¹⁸; https://www.esmvaltool.org/) and the Coordinated set of Model Evaluation Capabilities (CMEC; https://cmec.llnl.gov), which are both open-source capabilities. ESMValTool includes a large collection of diagnostics and

performance metrics for atmospheric, oceanic and terrestrial variables; not only for the mean state, but also for trends, variability, key physical processes and emergent constraints. ESMValTool also has the capability to reproduce figures from several chapters of AR5 and incorporates targeted analysis packages, such as the National Center for Atmospheric Research (NCAR) Climate Variability Diagnostics Package³⁹. CMEC comprises the PCMDI Metrics Package (PMP¹⁹), the International Land Modeling Benchmarking Project package (ILAMB⁴⁰) and the parallel toolkit for extreme climate analysis (TECA⁴¹). CMEC emphasizes a diverse suite of physical and biogeochemical summary statistics gauging the consistency between models and observations across a range of space and timescales.

Both ESMValTool and CMEC have undergone rapid development over the past few years, and are now mature, well-tested tools that provide end-to-end provenance tracking to ensure reproducibility. One goal is to routinely provide evaluation results through the Earth System Grid Federation (ESGF) shortly after new CMIP6 simulations are published. This workflow is depicted in Fig. 2: the tools are run at selected ESGF nodes, utilizing observations available in standard formats or provided by the user³⁸. The foundations for this significant undertaking are the community-based experimental protocols and conventions of CMIP, including their extension to observations (obs4MIPs⁴²) and reanalysis.

## Emergent constraints on Earth system sensitivities

One of the biggest challenges in ESM evaluation is to identify the performance metrics that are most relevant to climate projections⁷. The reliability of models can only be assessed with observations of the past and present. This means that models are assessed against criteria that are not necessarily informative in terms of the quality of model projections of future climate change. The emergent constraint approach attempts to address this problem by identifying robust, physically interpretable relationships between Earth system feedback behaviours on short, well-observed timescales and on
Box 1 | Emergent constraints

An emergent constraint on an Earth system sensitivity requires two key components. First, there needs to be an emergent relationship between the sensitivity (y axis of the figure below) and some measure of variation in the contemporary climate (x axis), which is evident across an ensemble of ESMs. Second, there needs to be an observation of the variation (x axis) in the real world, together with a measure of observational uncertainty. Ideally, the observation should have less uncertainty than the spread of the x-axis variable within the model ensemble. The observation of the x-axis variable and the model-derived emergent relationship between the y and x variables can then be combined to give a constraint on the Earth system sensitivity. The resulting emergent constraint is conditional on the model ensemble providing a realistic emergent relationship, and also on the availability of sufficiently accurate observations.

**Emergent constraints.** Schematic illustrating the concept of emergent constraints. Each blue dot represents (hypothetical) output from different ESMs. The comparison here involves a diagnostic based on a model's performance in a historical simulation (x axis) and in a projection of future climate change (y axis). The relationship between the past or present-day diagnostic and future projection illustrates an emergent relationship, which is normally quantified by a linear regression (red line). Once a physically plausible relationship is found, observations can be used to reduce the intermodel uncertainty in the long-term projection (compare the yellow and grey probability distribution functions). Uncertainty in the new projection (grey shading) arises from two sources: uncertainty in the observational constraint (blue shading) and in the linear regression (red shading).

<figure><img src="image_4.png" /></figure>

timescales that span the twenty-first century and beyond<sup>43,44</sup> (see the figure in Box 1). Emergent constraints use an ensemble of ESMs to define a relationship between a measured aspect of current or past climate and the strength of a simulated Earth system feedback in the future. It is the model ensemble behaviour (rather than the behaviour of a single model) that defines the emergent relationship between the observed variability and the projection of the future

climate. When combined with observational data and a measure of observational uncertainty, the model-derived emergent relationship can be converted into an emergent constraint on the Earth system sensitivity in the real world.

When AR5 was published, numerous emergent constraints had been identified. Examples include studies of snow–albedo feedback<sup>43</sup>, sea-ice<sup>45</sup>, tropical precipitation extremes<sup>46</sup>, carbon loss from tropical land under warming<sup>47</sup> and the future latitudinal shift of the Southern Hemisphere westerlies<sup>48</sup>. Such studies have proliferated since AR5, including constraints on cloud feedbacks and equilibrium climate sensitivity (ECS)<sup>49–57</sup>, strengthening of the hydrological cycle<sup>58,59</sup>, the temperature sensitivity of tropical land carbon storage<sup>60</sup>, CO₂ fertilization of plant photosynthesis<sup>61</sup>, future changes in ocean net primary production<sup>62</sup>, permafrost loss<sup>63</sup>, changes in natural sources and sinks<sup>64</sup> of CO₂ and mid-latitude daily heat extremes<sup>65</sup>. The proposed observable constraints involve historical trends<sup>43,45,64</sup>, interannual variability<sup>47,55,60</sup>, seasonal cycles<sup>43</sup>, trends in the seasonal cycle<sup>61</sup> and spatial variability<sup>63</sup>. Constraints have been tested against different ensembles and scenarios<sup>43,47,60,66</sup>. For example, a relationship between the ECS and the inferred strength of upward mixing in the tropical lower troposphere was used to discount ECS values below 3 °C, as all models with lower ECS had too little mixing (Fig. 3, left) and by implication a decreased positive cloud feedback at low levels<sup>53</sup>. This would narrow the range of ECS significantly compared to the 1.5–4.5 °C range assessed by AR5¹. However, other emergent constraint studies for ECS lead to different estimates of this uncertainty range<sup>51–57</sup>, pointing to the need for further research. For carbon cycle feedbacks, an emergent constraint on the impact of increased CO₂ on photosynthesis was found based on observed changes in the seasonal cycle of atmospheric CO₂, suggesting that doubling of the CO₂ concentration in the atmosphere will cause global plant photosynthesis to increase by approximately one third (Fig. 3, right)⁶¹.

Despite the attractiveness of emergent constraints, there are some well-justified concerns. Most importantly, the emergent relationship between the observable and the sensitivity to be constrained is derived from a model ensemble. The emergent constraint may be misleading if the model ensemble has a systematic error (such as the double ITCZ) that affects the emergent relationship, or reflects the simplicity of a parameterization common to many models rather than an intrinsic underlying process. Second, there is a danger of finding spurious relationships between observables and Earth system sensitivities if the high-dimensional outputs available from ESMs are simply data-mined for high correlations⁶⁷. The correlations found in a data-mining approach should be restricted to those that have a physical explanation. Finally, we should not expect short-term variability to yield constraints on slow feedbacks that have negligible effects on that variability. For example, interannual variations in sea level are unlikely to provide constraints on century-timescale sea-level rise due to ice-sheet melt. On the other hand, fast processes (such as water vapour and cloud feedbacks) are evident in short-term variability as well as trends, and are therefore much better candidates for emergent constraints that relate variability and sensitivity. Many of the most uncertain feedbacks are fast feedbacks such as these, which are more amenable to the emergent constraint technique.

Weighting multimodel climate projections

Traditionally, CMIP models were treated as independent, equally plausible estimates of future climate. Confidence in projections was inferred from model agreement on the sign and magnitude of future change⁹. In the context of multimodel ensemble projections, an increasing number of studies have weighted models that agree better with historical observations of that quantity or relationships between the projected quantity and observable metrics¹¹,⁶⁷⁻⁷⁰. However, the majority of weighting studies of certain climate properties such as
<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

**Fig. 3** | Examples of newly developed physical and biogeochemical emergent constraints since the AR5. **a**, Emergent constraint on ECS showing a correlation between ECS and a lower-tropospheric mixing index (LTMI) from 43 CMIP5 models. LTMI is calculated as the sum of an index for the small-scale component of mixing that is proportional to the differences of temperature and relative humidity between 700 hPa and 850 hPa and an index for the large-scale lower-tropospheric mixing. The error bars indicate the 2σ range of the direct radiosonde estimate. The linear correlation coefficient *r* and the two reanalyses ERA-I (square) and MERRA (diamond) are given in addition. **b**, Emergent constraint on the relative increase of large-scale gross primary production (GPP) for a doubling of CO₂ (2×CO₂), showing a correlation between the sensitivity of the CO₂ amplitude to annual mean CO₂ increases at Point Barrow (71.3° N, 156.6° W) and the high-latitude (60° N–90° N) CO₂ fertilization on GPP at 2×CO₂. The grey shading shows the range of the observed sensitivity. The red line shows the linear best fit across the CMIP5 ensemble together with the prediction error (orange), and error bars show the standard deviation for each data point. The black dashed line shows the central estimate of the emergent constraint on CO₂ fertilization. Figure reproduced from ref. ⁵³ (**a**) and ref. ⁶¹ (**b**), Springer Nature Limited.

sea-ice extent in AR5 include only a small set of metrics that are not always clearly related to the projected quantity in question.

An increase in weighted skill scores can be relatively simply achieved in-sample (that is, in the observational period and/or location used to derive weights). However, only a few studies have specifically focused on the likelihood of weighted results providing benefits for the intended application (that is, out-of-sample, typically twenty-first century projections)¹⁰⁻¹², ¹⁴, ¹⁵, ⁶⁸, ⁷¹, ⁷². Although we clearly have no observations of future climate, model-as-truth (also termed pseudo-reality in some studies¹¹,⁶⁸) and calibration-valida- tion exercises for different time periods of the observations yield valuable information on the potential benefits of different weight- ing approaches⁷³. In addition to testing whether projections of a specific variable and metric can be improved through weighting, thorough out-of-sample testing can help guard against other poten- tial issues with weighting. For example, there is no single metric that reliably captures all aspects of model performance for all purposes, even if interest is restricted to a very specific scientific question⁷⁴. Out-of-sample testing can tell us whether optimizing in one metric or variable transfers any benefits to other metrics or variables⁷¹. It can also indicate whether internal variability has played a role in any in-sample success of weighting, help to avoid the issue of the same datasets being used to calibrate and weight models, and reveal whether weighting has artificially reduced ensemble spread. A fur- ther problem is the risk of systematic errors in observational prod- ucts producing inappropriately weighted ensembles. Furthermore, weighting schemes have no capacity to account for errors that are shared across an ensemble — an issue that is particularly important in the case of small ensembles.

Another relevant issue is model interdependence. Some of the nominally different models in the CMIP archive share individual components or parameterizations, or represent key processes in the same way. This can lead to shared biases that have the potential to compromise the efficacy of performance-based weighting⁷² and to create artificially strong emergent constraints⁷⁵. Using model error correlation as a measure for interdependence, it was found that the

effective number of independent climate models was likely to be significantly lower than the total number of models in the CMIP ensemble⁷⁶,⁷⁷. Several studies have subsequently introduced alterna- tive ways of quantifying and accounting for interdependence¹³, ¹⁴, ⁷⁸, ⁷⁹. Recently, the US National Climate Assessment weighted each mem- ber of the CMIP5 archive using both a multivariate skill score for historical climatology and a measure of uniqueness in the archive¹². Figure 4 shows the resulting skill weight versus the indepen- dence weight for all CMIP5 models. Skill weights are calculated as multivariate root mean square errors over a North American domain, whereas independence weights are computed using model error bias correlation. No model receives high weights for both skill and independence (see the empty upper-right corner). This suggests that the ensemble has been unintentionally skill weighted by the inclusion of multiple versions of better-performing models. As in the case of efforts to define broadly applicable performance metrics, it is evident that there is no universally accepted definition of model dependence: accounting for model dependence is prob- lem-specific. Weighting exercises that accounted for model depen- dence were sensitive to almost all aspects of the problem, including the selected metric, variable, analysis period and constraining observational dataset.

## Regional model evaluation for impact and risk assessments

A primary goal of climate research is to identify how climate vari- ability and change affect society and to inform strategies for miti- gation and adaptation to climate change. Impact sectors include agriculture, forestry, water resources, infrastructure, energy pro- duction, land and marine ecosystems, and human health. To accu- rately capture many of the significant effects of climate change in sectoral impacts models, high levels of detail regarding the evolv- ing climate state are necessary. The impacts community generally needs rigorous regional-scale evaluation of the seasonal cycles of temperature, precipitation, humidity, wind speed and downwelling solar radiation. Although some sectors are affected by mean climate changes, the most acute impacts are related to extreme events.
<figure><img src="image_7.png" /><figcaption>**Fig. 4 | Model skill and independence weights for CMIP5 models evaluated over the contiguous United States/Canada domain.**</figcaption></figure>

Contours show the overall weighting, which is the product of the two individual weights. Figure reproduced from ref. ¹², GMD, under a Creative Commons license.

Annual mean temperature and precipitation, monsoon timing and intensity, and modes of variability that can alter the probability of extreme events have been evaluated in the CMIP5 ensemble⁷. The Expert Team on Climate Change Detection and Indices compiled a set of indices to quantify extreme events⁸⁰. Observational estimates of these indices have been used to evaluate CMIP5 models⁸¹, and are now incorporated into the ESMValTool for evaluation of CMIP6 models. The overall model performance was mixed in capturing the observed behaviour of these extreme event metrics. However, such comparisons remain difficult to interpret because of substantial uncertainties and data gaps in many of the observational datasets, and due to limited availability of CMIP5 model output at sub-monthly and daily frequencies.

Stakeholder-oriented applications have benefited from improved models, more user-relevant metrics, more robust observational systems, and longer observational records since AR5. Improvements in remote sensing products have enabled the evaluation of interannual and sub-seasonal events⁸². Models show continuing improvement in the representation of the diurnal cycle, storm tracks, the effects of

blocking on extreme events, the El Niño/Southern Oscillation, tropical cyclones and other circulation features. Awareness of multivariate extremes⁸³ has begun to emerge. The production and application of dynamically and empirically downscaled model information has advanced⁸⁴. Extreme event detection and attribution has made substantial strides, with real-time probabilistic event attribution⁸⁵ now feasible. Applications in different impact sectors have also been advanced by hybrid climate forcing datasets combining models and observations⁸⁶, and by sector-specific information such as the geographic distribution of nitrogen fertilizer or irrigation applications and land-use patterns for agricultural modelling⁸⁷.

Interactions between the climate modelling and applications communities are facilitated by the Vulnerability, Impacts, Adaptation, and Climate Services (VIACS) Advisory Board⁸⁸, a CMIP6-Endorsed MIP⁵. This Board is an effort to draw a broader array of climate experts, practitioners and information brokers into the CMIP process, and to leverage the community engagements organized under the Global Programme of Research on Climate Change Vulnerability, Impacts and Adaptation (PROVIA) and as part of efforts such as the Inter-Sectoral Impact Model Intercomparison Project (ISIMIP)⁸⁹. The VIACS Advisory Board has solicited sustained community engagement: (1) on priority experiments; (2) on output variables to inform CMIP6 data requests; and (3) to highlight evaluations that would help to establish model credibility with the VIACS community. Given that it is common practice to adjust model biases with monthly mean information from present-day fields, the evaluation of the seasonal evolution and distributions of monthly climate at regional scales is among the highest priorities for VIACS users⁹⁰. ESMs often have less variance than observations at hourly, daily and interannual timescales, which can lead to spurious effects when bias adjustment relies on standardized anomalies. Tropical rainfall biases are particularly problematic.

## Ways ahead

The CMIP6 experiment design provides an opportunity for sophisticated, consistent characterization of the ensemble and its predecessors⁵. Targeted MIPs associated with CMIP6 will accelerate efforts to disentangle internal climate variability from forced responses, and to evaluate which model processes are relevant to a wide range of climate characteristics. Insights into the underlying causes of systematic errors are likely to be gained from idealized experiments (such as aquaplanets⁹¹,⁹²), systematic assessment of the influence of horizontal resolution, analyses of forcing uncertainty and the evaluation of individual model components. The diverse numerical experiments proposed in CMIP6 may help the climate science community to gain a deeper understanding of model behaviour and processes than has been possible in the past. Further diagnostic benefits should accrue from the development of convectively resolving models, dynamic vegetation, three-dimensional ice-sheet models and refined physical parameterizations.

Model development, evaluation and weighting will be facilitated by the ongoing development and deployment of new climate observing systems with continuous quality assessment and independent verification. Rigorous quantification of observational uncertainties is now routine rather than exceptional. Examples include the availability of 'ensembles of observations' for a single observational product, which account for uncertainties associated with different subjective processing choices⁹³. Challenges remain in propagating these uncertainties to derived quantities such as trends or conditional averages. New measurements and measurements made at higher frequencies will also provide further insights into systematic errors. ARGO floats⁹⁴ and new satellite missions are prime examples.

An exciting opportunity is provided by the new CMIP evaluation tools ESMValTool¹⁸ and CMEC¹⁹. Both evaluation packages will be routinely executed whenever new model simulations are
contributed to the CMIP6 archive. This allows rapid, quantitative comparisons of model results to a wide range of climate observations³⁸. Such rapid and comprehensive feedback on model performance should help in addressing the causes of long-standing systematic errors and facilitate a shift towards more process-oriented diagnostics, while ensuring continuity with more 'traditional' diagnostics applied in previous CMIP phases. The hope is that ESMValTool and CMEC will be further enhanced by the CMIP6-Endorsed MIPs and other science teams, leading to widespread adoption by model development teams and the user community. Other promising diagnostic developments on the horizon that should be further advanced include studies that assess responses to perturbations rather than mean climate⁹⁵, and the application of innovative data science methods in Earth system science⁹⁶ such as neural networks⁹⁷, machine learning-based anomaly detection techniques⁹⁸, graphical models and causal discovery⁹⁹.

Physically robust emergent constraints are a promising concept for understanding and constraining Earth system feedbacks and narrowing uncertainty in future projections. They may ultimately influence model development and observational strategies. In addition to new research on consistency across different emergent constraints and across generations of model ensembles, we anticipate the use of more sophisticated statistical analyses, which have so far typically involved one-dimensional linear regressions. Higher-dimensional emergent relationships related to more than one observable should yield more robust conclusions and avoid the possibility of contradictory constraints derived from separate one-dimensional relationships. There is a new opportunity to test emergent constraints developed in previous model generations against the outputs from CMIP6 models. Finally, there needs to be a greater focus on developing emergent constraints for regional climate change that are more relevant to impacts than many of the large-scale metrics that are the current focus of emergent constraints⁸.

To guard against misleading emergent constraints arising from spurious correlations or from the dependence introduced by a parameterization common to many models — rather than from an intrinsic underlying process — we suggest that the development of emergent constraints should be treated as a form of hypothesis testing. For example, emergent relationships between variability and sensitivity could be derived on the basis of physical theory or simple underlying models. The predicted emergent relationship can then be tested against outputs from full-form ESMs. This approach could also yield an improved theoretical understanding of relationships between variability and sensitivity in the Earth system. Even where the outputs of one generation of models seem to be consistent with the hypothesized emergent relationship, the robustness of the relationship should be tested out of sample against models that were not used to define the relationship. The hypothesis testing approach that we propose would also protect from attempts to artificially tune a model to fit an observational constraint. Where this is carried out unphysically, the tuned model is likely to move away from the theoretical curve (that is, it will fit the x-axis observation but it will no longer be consistent with the y-axis sensitivity).

There is enough evidence now that the continued assumption of model democracy cannot be fully justified in future IPCC assessment reports. It is not yet clear, however, whether all variables of interest can be reliably constrained. Successful skill weighting has thus far been implemented for a limited number of specific applications. In these applications, the target property of interest is constrained by a small number of clearly relevant variables¹⁰⁻¹²,⁶⁷,⁶⁸. Future work for more complex chains of influence will need to consider orthogonal uncertainties and processes. For example, regional precipitation change may be influenced by global-scale warming, large-scale dynamics and microphysical parameterizations. For regional climate projections, a weighting that is based on processes

controlling the region of interest and biases in large-scale atmospheric circulation is advocated⁸.

In addition, it has been demonstrated that CMIP models are not independent. Most inferences in the literature about model interdependence are derived from error correlation¹³,⁷⁹. This cannot identify the specific model components that are interdependent. Identification of these common components is a difficult task due to the large number of models involved in CMIP and lack of detailed information regarding individual model versions. Further work is required to understand how interdependence can best be assessed. These efforts can proceed in tandem with research to better understand the effects of model construction and genealogy. Comprehensive databases of shared code, parameterizations, model development and tuning practices could help disentangle how models are related, and for what purposes they can be considered independent estimates of change. There is also the potential for better quantification of natural variability from palaeoclimate simulations¹⁰⁰ and enhanced collaboration with the detection and attribution community, whose statistical approaches provide information on whether model responses to changes in external forcings are consistent with observations. Simpler representations of the Earth system in a hierarchy of models will also be useful to improve more complex ESMs.

For improved assessments of regional impacts and risks, a key challenge and opportunity will be to derive collective understanding from global and regional climate models, as well as from regional-scale observations. To do so it will be important to bridge the gap between the climate model and impacts communities, and between the different scales at which these communities typically operate. CMIP6 will include weather-resolving global model resolutions (~25 km or finer) that need to be compared to regional model results and downscaled coarse-resolution simulations. Concerns about bias correction of climate change simulations have been raised, and ways to address these concerns have been proposed⁸. Many of the key systematic errors that hampered the reliable simulation of surface variables and extreme events will benefit from increasing spatial resolution¹⁰¹, variable-resolution grids, improved parameterizations, and advances in bias corrections and downscaling techniques⁸⁴. Curated archives and the CMIP evaluation tools will enable participation by a broader diagnostic community, many of whom are not presently capable of advanced interrogation of climate model simulations. The provision of useful climate information and messages for the impacts, risk and climate services communities requires a process that is rooted in sustained engagement with stakeholders that concentrates on areas of particular vulnerability or exposure. The projection of changing hazard metrics and the construction of driving scenarios for impact models across a range of local, regional and national scales should benefit from a process that distils information from many different sources. Such sources include multiple ESMs, statistically and dynamically downscaled models (through the Coordinated Regional Downscaling Experiment, CORDEX, now a CMIP6-Endorsed MIP⁸⁴) and bias adjusted models⁸. The publication of AR6 will enhance the focus on regional climate information through a regionally defined Atlas and new chapters on global-to-regional linkages, extreme events, and impact- and risk-relevant climate hazards.

Despite substantial progress in climate modelling over the past few decades, there remains a substantial spread in projections of future climate change. For example, the range of model estimates for ECS to a doubling of CO₂ concentrations has not decreased since the 1970s⁷, although understanding of the processes that are involved has certainly increased. The need to inform mitigation policy and adaptation remains¹⁰². We believe that there is now an unprecedented opportunity to constrain policy-relevant metrics such as cumulative CO₂ emissions consistent with specific temperature targets with observations, and to reduce uncertainties in
climate projections, both at global and regional scales. The challenge is to make intelligent use of the petabyte-scale output that will become available from the new CMIP6 project, along with modern observation systems, new model evaluation tools and novel data science techniques. A combination of different process-based emergent constraints together with model-weighting approaches that consider both model performance and interdependence have the potential to yield more robust multimodel information for a wide array of societally and environmentally critical applications.

Received: 25 February 2018; Accepted: 8 November 2018;
Published online: 7 January 2019

## References

1. IPCC *Climate Change 2013: The Physical Science Basis* (eds Stocker, T. F. et al.) (Cambridge Univ. Press, 2013).

2. Meehl, G. A., Boer, G. J., Covey, C., Latif, M. & Stouffer, R. J. The Coupled Model Intercomparison Project (CMIP). *Bull. Am. Meteorol. Soc.* **81**, 313–318 (2000).

3. Meehl, G. A. et al. THE WCRP CMIP3 multimodel dataset: a new era in climate change research. *Bull. Am. Meteorol. Soc.* **88**, 1383–1394 (2007).

4. Taylor, K. E., Stouffer, R. J. & Meehl, G. A. An overview of CMIP5 and the experiment design. *Bull. Am. Meteorol. Soc.* **93**, 485–498 (2012).

5. Eyring, V. et al. Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. *Geosci. Model Dev.* **9**, 1937–1958 (2016).

6. Stouffer, R. J. et al. CMIP5 scientific gaps and recommendations for CMIP6. *Bull. Am. Meteorol. Soc.* **98**, 95–105 (2017).

7. Flato, G. et al. in *Climate Change 2013: The Physical Science Basis* (eds Stocker, T. F. et al.) Ch. 9 (IPCC, Cambridge Univ. Press, 2013).

8. Maraun, D. et al. Towards process-informed bias correction of climate change simulations. *Nat. Clim. Change* **7**, 764–773 (2017).

9. Collins, M. et al. in *Climate Change 2013: The Physical Science Basis* (eds Stocker, T. F. et al.) Ch. 12 (IPCC, Cambridge Univ. Press, 2013).

10. Knutti, R. et al. A climate model projection weighting scheme accounting for performance and interdependence. *Geophys. Res. Lett.* **44**, 1909–1918 (2017).

11. Wenzel, S., Eyring, V., Gerber, E. P. & Karpechko, A. Y. Constraining future summer austral jet stream positions in the CMIP5 ensemble by process-oriented multiple diagnostic regression. *J. Clim.* **29**, 673–687 (2016).

12. Sanderson, B. M., Wehner, M. & Knutti, R. Skill and independence weighting for multi-model assessments. *Geosci. Model Dev.* **10**, 2379–2395 (2017).

13. Sanderson, B. M., Knutti, R. & Caldwell, P. Addressing interdependency in a multimodel ensemble by interpolation of model properties. *J. Clim.* **28**, 5150–5170 (2015).

14. Bishop, C. H. & Abramowitz, G. Climate model dependence and the replicate Earth paradigm. *Clim. Dynam.* **41**, 885–900 (2013).

15. Abramowitz, G. & Bishop, C. H. Climate model dependence and the ensemble dependence transformation of CMIP projections. *J. Clim.* **28**, 2332–2348 (2015).

16. Alexander, K. & Easterbrook, S. M. The software architecture of climate models: a graphical comparison of CMIP5 and EMICAR5 configurations. *Geosci. Model Dev.* **8**, 1221–1232 (2015).

17. Knutti, R. The end of model democracy? *Climatic Change* **102**, 395–404 (2010).

18. Eyring, V. et al. ESMValTool (v1.0) — a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP. *Geosci. Model Dev.* **9**, 1747–1802 (2016).

19. Gleckler, P. et al. A more powerful reality test for climate models. *Eos* https://doi.org/10.1029/2016eo051663 (2016).

20. Lauer, A. et al. Process-level improvements in CMIP5 models and their impact on tropical variability, the Southern Ocean, and monsoons. *Earth Syst. Dynam.* **9**, 33–67 (2018).

21. Ma, C.-C., Mechoso, C. R., Robertson, A. W. & Arakawa, A. Peruvian stratus clouds and the tropical pacific circulation: a coupled ocean-atmosphere GCM study. *J. Clim.* **9**, 1635–1645 (1996).

22. Hourdin, F. et al. Parameterization of convective transport in the boundary layer and its impact on the representation of the diurnal cycle of wind and dust emissions. *Atmos. Chem. Phys.* **15**, 6775–6788 (2015).

23. Richter, I. Climate model biases in the eastern tropical oceans: causes, impacts and ways forward. *WIREs Clim. Change* **6**, 345–358 (2015).

24. Wang, C. Z., Zhang, L. P., Lee, S. K., Wu, L. X. & Mechoso, C. R. A global perspective on CMIP5 climate model biases. *Nat. Clim. Change* **4**, 201–205 (2014).

25. Fyfe, J. C., Gillett, N. P. & Zwiers, F. W. Overestimated global warming over the past 20 years. *Nat. Clim. Change* **3**, 767–769 (2013).

26. Meehl, G. A., Hu, A., Arblaster, J. M., Fasullo, J. & Trenberth, K. E. Externally forced and internally generated decadal climate variability associated with the Interdecadal Pacific Oscillation. *J. Clim.* **26**, 7298–7310 (2013).

27. Fyfe, J. C. et al. Making sense of the early-2000s warming slowdown. *Nat. Clim. Change* **6**, 224–228 (2016).

28. Santer, B. D. et al. Volcanic contribution to decadal changes in tropospheric temperature. *Nat. Geosci.* **7**, 185–189 (2014).

29. Meehl, G. A., Teng, H. & Arblaster, J. M. Climate model simulations of the observed early-2000s hiatus of global warming. *Nat. Clim. Change* **4**, 898–902 (2014).

30. Thoma, M., Greatbatch, R. J., Kadow, C. & Gerdes, R. Decadal hindcasts initialized using observed surface wind stress: evaluation and prediction out to 2024. *Geophys. Res. Lett.* **42**, 6454–6461 (2015).

31. Meehl, G. A., Hu, A. & Teng, H. Initialized decadal prediction for transition to positive phase of the Interdecadal Pacific Oscillation. *Nat. Commun.* **7**, 11718 (2016).

32. Mears, C. A., Santer, B. D., Wentz, F. J., Taylor, K. E. & Wehner, M. F. Relationship between temperature and precipitable water changes over tropical oceans. *Geophys. Res. Lett.* **34**, L24709 (2007).

33. Karl, T. R. et al. Possible artifacts of data biases in the recent global surface warming hiatus. *Science* **348**, 1469–1472 (2015).

34. Mears, C. A. & Wentz, F. J. The effect of diurnal correction on satellite-derived lower tropospheric temperature. *Science* **309**, 1548–1551 (2005).

35. Mauritsen, T. et al. Tuning the climate of a global model. *J. Adv. Model. Earth Syst.* **4**, M00A01 (2012).

36. Hourdin, F. et al. The art and science of climate model tuning. *Bull. Am. Meteorol. Soc.* **98**, 589–602 (2017).

37. Bodas-Salcedo, A. et al. COSP Satellite simulation software for model assessment. *Bull. Am. Meteorol. Soc.* **92**, 1023–1043 (2011).

38. Eyring, V. et al. Towards improved and more routine Earth system model evaluation in CMIP. *Earth Syst. Dynam.* **7**, 813–830 (2016).

39. Phillips, A. S., Deser, C. & Fasullo, J. Evaluating modes of variability in climate models. *Eos* **95**, 453–455 (2014).

40. Luo, Y. Q. et al. A framework for benchmarking land models. *Biogeosciences* **9**, 3857–3874 (2012).

41. Prabhat, et al. TECA: a parallel toolkit for extreme climate analysis. *Proc. Comput. Sci.* **9**, 866–876 (2012).

42. Ferraro, R., Waliser, D. E., Gleckler, P., Taylor, K. E. & Eyring, V. Evolving Obs4MIPs to support phase 6 of the Coupled Model Intercomparison Project (CMIP6). *Bull. Am. Meteorol. Soc.* **96**, ES131–ES133 (2015).

43. Hall, A. & Qu, X. Using the current seasonal cycle to constrain snow albedo feedback in future climate change. *Geophys. Res. Lett.* **33**, L03502 (2006).

44. Allen, M. R. & Ingram, W. J. Constraints on future changes in climate and the hydrologic cycle. *Nature* **419**, 224–232 (2002).

45. Massonnet, F. et al. Constraining projections of summer Arctic sea ice. *Cryosphere* **6**, 1383–1394 (2012).

46. O'Gorman, P. A. Sensitivity of tropical precipitation extremes to climate change. *Nat. Geosci.* **5**, 697–700 (2012).

47. Cox, P. M. et al. Sensitivity of tropical carbon to climate change constrained by carbon dioxide variability. *Nature* **494**, 341–344 (2013).

48. Kidston, J. & Gerber, E. P. Intermodel variability of the poleward shift of the austral jet stream in the CMIP3 integrations linked to biases in 20th century climatology. *Geophys. Res. Lett.* **37**, L09708 (2010).

49. Tushima, Y. et al. Robustness, uncertainties, and emergent constraints in the radiative responses of stratocumulus cloud regimes to future warming. *Clim. Dynam.* **46**, 3025–3039 (2016).

50. Brient, F. & Bony, S. Interpretation of the positive low-cloud feedback predicted by a climate model under global warming. *Clim. Dynam.* **40**, 2415–2431 (2013).

51. Brient, F. & Schneider, T. Constraints on climate sensitivity from space-based measurements of low-cloud reflection. *J. Clim.* **29**, 5821–5835 (2016).

52. Lipat, B. R., Tselioudis, G., Grise, K. M. & Polvani, L. M. CMIP5 models' shortwave cloud radiative response and climate sensitivity linked to the climatological Hadley cell extent. *Geophys. Res. Lett.* **44**, 5739–5748 (2017).

53. Sherwood, S. C., Bony, S. & Dufresne, J. L. Spread in model climate sensitivity traced to atmospheric convective mixing. *Nature* **505**, 37–42 (2014).

54. Tian, B. Spread of model climate sensitivity linked to double-Intertropical Convergence Zone bias. *Geophys. Res. Lett.* **42**, 4133–4141 (2015).

55. Cox, P. M., Huntingford, C. & Williamson, M. S. Emergent constraint on equilibrium climate sensitivity from global temperature variability. *Nature* **553**, 319–322 (2018).

56. Klein, S. A. & Hall, A. Emergent constraints for cloud feedbacks. *Curr. Clim. Change Rep.* **1**, 276–287 (2015).

57. Dessler, A. E. & Forster, P. M. An estimate of equilibrium climate sensitivity from interannual variability. *J. Geophys. Res. Atmos.* **123**, 8634–8645 (2018).

58. DeAngelis, A. M., Qu, X., Zelinka, M. D. & Hall, A. An observational radiative constraint on hydrologic cycle intensification. *Nature* **528**, 249–253 (2015).
59. Li, G., Xie, S.-P., He, C. & Chen, Z. Western Pacific emergent constraint lowers projected increase in Indian summer monsoon rainfall. *Nat. Clim. Change* 7, 708–712 (2017).

60. Wenzel, S., Cox, P. M., Eyring, V. & Friedlingstein, P. Emergent constraints on climate-carbon cycle feedbacks in the CMIP5 Earth system models. *J. Geophys. Res. Biogeosci.* 119, 794–807 (2014).

61. Wenzel, S., Cox, P. M., Eyring, V. & Friedlingstein, P. Projected land photosynthesis constrained by changes in the seasonal cycle of atmospheric CO₂. *Nature* 538, 499–501 (2016).

62. Kwiatkowski, L. et al. Emergent constraints on projections of declining primary production in the tropical oceans. *Nat. Clim. Change* 7, 355–358 (2017).

63. Chadburn, S. E. et al. An observation-based constraint on permafrost loss as a function of global warming. *Nat. Clim. Change* 7, 340–344 (2017).

64. Hoffman, F. M. et al. Causes and implications of persistent atmospheric carbon dioxide biases in Earth system models. *J. Geophys. Res. Biogeosci.* 119, 141–162 (2014).

65. Donat, M. G., Pitman, A. J. & Angélil, O. Understanding and reducing future uncertainty in mid-latitude daily heat extremes via land surface feedback constraints. *Geophys. Res. Lett.* 45, 10627–10636 (2018).

66. Qu, X. & Hall, A. What controls the strength of snow-albedo feedback? *J. Clim.* 20, 3971–3981 (2007).

67. Waugh, D. W. & Eyring, V. Quantitative performance metrics for stratospheric-resolving chemistry-climate models. *Atmos. Chem. Phys.* 8, 5699–5713 (2008).

68. Karpechko, A. Y., Maraun, D. & Eyring, V. Improving Antarctic total ozone projections by a process-oriented multiple diagnostic ensemble regression. *J. Atmos. Sci.* 70, 3959–3976 (2013).

69. Räisänen, J., Ruokolainen, L. & Ylhäisi, J. Weighting of model results for improving best estimates of climate change. *Clim. Dynam.* 35, 407–422 (2010).

70. Lorenz, R. et al. Prospects and caveats of weighting climate models for summer maximum temperature projections over North America. *J. Geophys. Res. Atmos.* 123, 4509–4526 (2018).

71. Abramowitz, G. et al. Model dependence in multi-model climate ensembles: weighting, sub-selection and out-of-sample testing. *Earth Syst. Dynam. Discuss.* https://doi.org/10.5194/esd-2018-51 (2018).

72. Herger, N. et al. Selecting a climate model subset to optimise key ensemble properties. *Earth Syst. Dynam.* 9, 135–151 (2018).

73. Herger, N. et al. Calibrating climate model ensembles for assessing extremes in a changing climate. *J. Geophys. Res. Atmos.* 123, 5988–6004 (2018).

74. Santer, B. D. et al. Incorporating model quality information in climate change detection and attribution studies. *Proc. Natl Acad. Sci. USA* 106, 14778–14783 (2009).

75. Caldwell, P. M. et al. Statistical significance of climate sensitivity predictors obtained by data mining. *Geophys. Res. Lett.* 41, 1803–1808 (2014).

76. Masson, D. & Knutti, R. Climate model genealogy. *Geophys. Res. Lett.* 38, L08703 (2011).

77. Pennell, C. & Reichler, T. On the effective number of climate models. *J. Clim.* 24, 2358–2367 (2011).

78. Sunyer, M. A., Madsen, H., Rosbjerg, D. & Arnbjerg-Nielsen, K. A Bayesian approach for uncertainty quantification of extreme precipitation projections including climate model interdependency and nonstationary bias. *J. Clim.* 27, 7113–7132 (2014).

79. Sanderson, B. M., Knutti, R. & Caldwell, P. A representative democracy to reduce interdependency in a multimodel ensemble. *J. Clim.* 28, 5171–5194 (2015).

80. Zhang, X. et al. Indices for monitoring changes in extremes based on daily temperature and precipitation data. *WIREs Clim. Change* 2, 851–870 (2011).

81. Sillmann, J., Kharin, V. V., Zhang, X., Zwiers, F. W. & Bronaugh, D. Climate extremes indices in the CMIP5 multimodel ensemble. Part 1. Model evaluation in the present climate. *J. Geophys. Res. Atmos.* 118, 1716–1733 (2013).

82. Pendergrass, A. G. & Hartmann, D. L. The atmospheric energy constraint on global-mean precipitation change. *J. Clim.* 27, 757–768 (2014).

83. Zscheischler, J. et al. Future climate risk from compound events. *Nat. Clim. Change* 8, 469–477 (2018).

84. Gutowski, W. J. Jr et al. WCRP COordinated Regional Downscaling Experiment (CORDEX): a diagnostic MIP for CMIP6. *Geosci. Model Dev.* 9, 4087–4095 (2016).

85. van Oldenborgh, G. J., Otto, F. E. L., Haustein, K. & Cullen, H. Climate change increases the probability of heavy rains like those of storm Desmond in the UK—an event attribution study in near-real time. *Hydrol. Earth Syst. Sci. Discuss.* 12, 13197–13216 (2015).

86. Ruane, A. C., Goldberg, R. & Chryssanthacopoulos, J. Climate forcing datasets for agricultural modeling: merged products for gap-filling and historical climate series estimation. *Agric. For. Meteorol.* 200, 233–248 (2015).

87. Elliott, J. et al. The Global Gridded Crop Model Intercomparison: data and modeling protocols for Phase 1 (v1.0). *Geosci. Model Dev.* 8, 261–277 (2015).

88. Ruane, A. C. et al. The Vulnerability, Impacts, Adaptation and Climate Services Advisory Board (VIACS AB v1.0) contribution to CMIP6. *Geosci. Model Dev.* 9, 3493–3515 (2016).

89. Warszawski, L. et al. The Inter-Sectoral Impact Model Intercomparison Project (ISI–MIP): project framework. *Proc. Natl Acad. Sci. USA* 111, 3228–3232 (2014).

90. Ruane, A. C. & McDermid, S. P. Selection of a representative subset of global climate models that captures the profile of regional changes for integrated climate impacts assessment. *Earth Perspect.* 4, 1 (2017).

91. Stevens, B. & Bony, S. What are climate models missing? *Science* 340, 1053–1054 (2013).

92. Webb, M. J. et al. The Cloud Feedback Model Intercomparison Project (CFMIP) contribution to CMIP6. *Geosci. Model Dev.* 10, 359–384 (2017).

93. Mears, C. A., Wentz, F. J., Thorne, P. & Bernie, D. Assessing uncertainty in estimates of atmospheric temperature changes from MSU and AMSU using a Monte-Carlo estimation technique. *J. Geophys. Res. Atmos.* 116, D08112 (2011).

94. Argo Argo Float Data and Metadata from Global Data Assembly Centre (Argo GDAC) (SEANOE, 2000); https://doi.org/10.17882/42182

95. Malavelle, F. F. et al. Strong constraints on aerosol–cloud interactions from volcanic eruptions. *Nature* 546, 485–491 (2017).

96. Reichstein, M. et al. Deep learning and process understanding for data-driven Earth system science. *Nature* (in the press).

97. Fountalis, I., Bracco, A. & Dovrolis, C. ENSO in CMIP5 simulations: network connectivity from the recent past to the twenty-third century. *Clim. Dynam.* 45, 511–538 (2015).

98. Barz, B., Rodner, E., Guanche Garcia, Y. & Denzler, J. Detecting regions of maximal divergence for spatio-temporal anomaly detection. *IEEE Trans. Pattern Anal. Mach. Intell.* https://doi.org/10.1109/TPAMI.2018.2823766 (2018).

99. Runge, J. et al. Identifying causal gateways and mediators in complex spatio-temporal systems. *Nat. Commun.* 6, 8502 (2015).

100. Kageyama, M. et al. The PMIP4 contribution to CMIP6 – Part 1: Overview and over-arching analysis plan. *Geosci. Model Dev.* 11, 1033–1057 (2018).

101. Haarsma, R. J. et al. High Resolution Model Intercomparison Project (HighResMIP v1.0) for CMIP6. *Geosci. Model Dev.* 9, 4185–4208 (2016).

102. IPCC Special Report on Global Warming of 1.5 °C (Cambridge Univ. Press, 2018).

## Acknowledgements

The authors acknowledge the Aspen Global Change Institute (AGCI) for hosting a workshop on Earth System Model Evaluation to Improve Process Understanding in August 2017 as part of its traditionally landmark summer interdisciplinary sessions (http://www.agci.org/event/17s2). NASA, the Heising-Simons Foundation, Horizon 2020 European Union's Framework Programme for Research and Innovation under Grant Agreement No 641816, the Coordinated Research in Earth Systems and Climate: Experiments, kKnowledge, Dissemination and Outreach (CRESCENDO) project, the ESA Climate Change Initiative (CCI) Climate Model User Group (CMUG), WCRP and the Department of Energy (DOE) all provided support for the workshop. The viewpoint presented here substantially draws on conclusions from that workshop. Portions of this study were supported by the Regional and Global Model Analysis (RGMA) component of the Earth and Environmental System Modeling Program of the US DOE Office of Biological & Environmental Research (BER) Cooperative Agreement DE-FC02-97ER62402 and Contract No. DE-AC05-00OR22725 and the National Science Foundation. NCAR is sponsored by the National Science Foundation.

## Author contributions

V.E., P.M.C., G.M.F. and P.J.G. were the co-chairs of the AGCI workshop and led the writing of the paper. All authors participated in the AGCI workshop and contributed to discussions and writing of the text.

## Competing interests

The authors declare no competing interests.

## Additional information

Reprints and permissions information is available at www.nature.com/reprints.

Correspondence should be addressed to V.E.

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© Springer Nature Limited 2019