Temperature variability projections remain uncertain after constraining them to best performing Large Ensembles of individual Climate Models

Domain: earth_science
Published: 2025-12-13
Editorial outcome: minor_revision

Abstract
Changes in temperature variability affect the frequency and intensity of extreme events, as well as the regional range of temperatures that ecosystems and society need to adapt to. While accurate projections of temperature variability are vital for understanding climate change and its impacts, they remain highly uncertain. We use rank-frequency analysis to evaluate the performance of eleven single model initial-condition large ensembles (SMILEs) against observations in the historical period, and use those that best represent historical regional variability to constrain projections of future temperature variability. Constrained projections from the best-performing SMILEs still show large uncertainties in the intensity and the sign of the variability change for large areas of the globe. Our results highlight poorly modelled regions where observed variability is not well represented such as large parts of Australia, South America, and Africa, particularly in their local summer season, underscoring the need for further modelling improvements over crucial regions. In these regions, the constrained projected change is typically larger than in the unconstrained ensemble, suggesting that in these regions, multi-model mean projections may underestimate future variability change.

Introduction
To understand how global warming will affect the Earth’s climate, society and ecosystems, we must not only understand the changes in the mean climate, but also whether and how climate variability will change1. Projecting changes in temperature variability is crucial for understanding potential future changes in extreme events, the range of temperature conditions that any given region could experience1–4, and the impacts of these changes on ecosystem dynamics and habitats (e.g., refs. 5,6). This is because changes in extreme values in a distribution are affected by changes in the mean, its variance, and the skewness of the distribution7–9. For example, projected decreases in winter temperature variability in the northern hemisphere mid-latitudes result in less frequent cold air outbreaks than expected by warming alone10. Furthermore, increases in summer temperature variability in Europe are expected to exacerbate the increases in heat extremes projected from global warming2–4,11.

Although future changes in temperature variability can have important consequences, there are large uncertainties in the projections of how temperature variability will change under warming. The IPCC AR6 report flagged only medium confidence in changes in the extratropics and in the winter hemisphere mid-to-high latitudes, with low model agreement found everywhere else on the globe12. This uncertainty in the projected estimates of surface air temperature variability applies not only to the intensity of the changes, but also to their sign, with low model agreement on the direction of the variability change over large areas13.

While overall projections are uncertain, there is some consensus on how temperature variability will likely change over certain regions. Variability is projected to increase over the tropical land surface in CMIP5, single model initial-condition large ensembles (SMILEs), and 17-member Ensemble Simulations of Extreme Weather Events under Nonlinear Climate Change (ESSENCE) models due to drying soil in the southern hemisphere and increases in atmospheric variability in the northern hemisphere and thermal advection1,14,15. In Europe, projected reductions in winter variability are related to a lower land-sea temperature contrast16 and a decrease in horizontal temperature gradients and therefore advection17, while projected increases in summer variability2 relate to changes in soil drought frequency3,18, regions that lose summer snow cover19, and changes in the surface heat balance16. In the mid-latitudes, in CMIP5 and ESSENCE, variability is projected to decrease due to decreases in meridional temperature gradients and sea ice loss1,14, with 60N the latitude where the largest changes in fractional snow cover occur. The 60N latitude is also a region of decreasing variability, particularly in winter20. In the Arctic, a study using SMILEs shows that all models have increased temperature variability as long as sea ice exists, with variability decreasing once sea ice is lost15. This decrease can be constrained by choosing models that represent sea ice well, which causes the magnitude of the variability decrease to be larger than in unconstrained models21.

Daily temperature variability projections follow a similar pattern to interannual variability, with a projected reduction in variability in northern mid-latitudes and projected increases in low latitudes. In the northern hemisphere summer, this reduction in the high latitudes is only in North America, the high Arctic, and Africa with increases elsewhere4,22. Although in general there are large regions where evidence on a broad scale shows agreement on the projected sign of the change, models can still have large differences in the magnitude of the change, demonstrating high uncertainty, particularly on an annual scale23.

It remains, however, unclear whether climate models disagree on the direction and intensity of temperature variability changes because they do not adequately simulate present-day temperature variability, or whether they disagree due to intrinsic model differences in how the drivers of temperature variability change in a future climate. Therefore, a possible hypothesis is that the ability of models to project future temperature variability depends on their ability to simulate historical variability. If this were true, subsampling models based on their historical performance in capturing observed variability would yield reduced uncertainty in future projections. This means that by constraining projections to models that best match observations and agree on their historical variability representation, the range of future variability change projections across best-performing models would be reduced, and thus be less uncertain. In contrast, the alternative hypothesis is that different models’ projections of future temperature variability depend not simply on past temperature variability, but rather on how the Earth system as a whole evolves over different regions in each model. If this were true, subsampling models based on their historical performance would not necessarily reduce future uncertainty, as long as those historically best-performing models still exhibit diverging behaviors in their representation of future temperature variability changes. In this study, we address these questions by, first evaluating in a multi-model super ensemble of SMILEs which models can adequately represent observed regional variability under current climate conditions. Second, we subsample the super ensemble to those adequate models, per season and region, and compute constrained future projections for regional temperature variability. Third, we compare the full ensemble with this subset of the ensemble to determine if the uncertainty in our projections is lowered by constraining them to only models that adequately capture historical variability.

There has been a suite of work that aims to understand the best way to weight and/or subsample climate model output (e.g., refs. 24–27). These studies typically consider two important factors: model performance and model independence24. For individual applications, subsampling for models that adequately represent observations can lead to a reduction of errors28. However, when considering future change, model skill has been found to only relate weakly to the projected change, highlighting that past model performance does not guarantee better future projections29. Climate models are also known to not be independent of each other30,31, leading to multi-model means that are not created from independent samples. Studies such as 24 have shown that weighting by a combination of performance and independence can increase the skill of projections by up to 17%, highlighting the added value of using subsampling and constraining methods. The SMILEs used in our analysis are largely independent (as found by comparing the family trees in refs. 24,30) across CMIP5 and CMIP6 models (excluding GFDL-SPEAR-MED, which has not yet been included in previous work on this topic). Therefore, we limit the super-ensemble constraining to performance-only metrics. We note that some of the models used here reflect several generations of model development and are thus not independent, e.g., CanESM2 and CanESM5, CESM-LE and CESM2-LE, and MPI-GE5 and MPI-GE6, which likely share model features32. This study provides, to our knowledge, the first performance-constrained projections of temperature variability and its change globally.

Furthermore, previous work assessing variability changes typically confounds model differences and internal variability, by using a multi-model mean (e.g., ref. 1), or uses a single SMILE ignoring potential model differences3, or uses several SMILEs, but does not consider model performance when assessing model uncertainty (e.g., ref. 13). SMILEs themselves are an invaluable tool for assessing projections of temperature variability3,12,13,15,33. Without SMILEs, long time periods are needed to sufficiently sample temperature variability. These long time periods, by necessity, encompass a range of warming levels above pre-industrial conditions, thus potentially experiencing changes in the variability itself. SMILEs, however, allow for temperature variability to be composited from all ensemble members at a given year (or period of years), therefore yielding time-varying (or global-warming level varying) estimates of temperature variability that are not confounded by potential underlying changes in variability.

SMILEs also allow for a more robust evaluation of model performance and comparison against observations, determining performance by assessing whether observations fall within the range of possible climate outcomes, now better-sampled by large ensembles of climate model simulations33. Thus, SMILEs allow us to evaluate climate model estimates of internal variability in the historical period more robustly than ever before. In this study, we make use of these advantages and build on previous work by employing and expanding the rank-frequency model evaluation framework of ref. 33 to assess how well models simulate historical summer and winter temperature variability. This rank-frequency evaluation framework, which resembles probabilistic forecast verification techniques in the climate prediction literature34–36 is based on assessing whether observations occur across all ensemble ranks (i.e., the position an observation takes among the sorted ensemble members for a given time step) of an SMILE with comparable frequency. We then use the results of this region- and season-based evaluation to constrain projections of regional temperature variability for both seasons at five warming levels above pre-industrial conditions.

The aims of our study are threefold:
• Evaluate how well each SMILE captures historical temperature variability in both summer and winter and summer seasons over individual regions.
• Assess model agreement in the projections of temperature variability across all SMILEs for both the sign of the change and the spread of the ensemble.
• Constrain the projections using the SMILEs that perform well over the historical period, and assess whether this provides an increase in model agreement for future projections.

Results
Temperature variability evaluation
We begin by evaluating model performance in capturing temperature variability over the historical record. For this, we apply an expanded version of the rank-frequency model evaluation framework presented in ref. 33 to detrended and non-detrended monthly mean temperature anomalies in 11 SMILEs compared to historical observations, for 9 Ocean and 24 Land regions in both the December, January, February (DJF) and June, July, August (JJA) seasons. We expand the rank-frequency evaluation framework in ref. 33 by incorporating two formally defined evaluation criteria (e.g., Fig. 1). This rank-frequency evaluation framework assesses model performance based on a simple principle: whether observations occur uniformly across the whole ensemble spread, or in other words, across all ensemble ranks. Traditionally, this has been assessed in the ensemble forecasting literature by evaluating rank histogram flatness35,37. However, the fact that we are evaluating free-running uninitialised simulations against a relatively short observational record means that the internal variability in the climate system may be insufficiently sampled on these timescales, and thus rank histograms may appear to be not flat, even for perfectly performing models.

To overcome this, we assess the flatness of the rank histogram using a perfect-model rank range test, which does not make any assumptions about the shape of the underlying distribution and takes into account both ensemble size and serial correlation. This perfect-model range is constructed by treating each ensemble member in each SMILE as if it were observations, and calculating the resulting spread of model rank histograms. This provides a distribution of rank histograms that reflect perfect-model behavior for the record length considered here, yet may indeed not be perfectly flat. This perfect-model rank range gives us a baseline of possible deviations from flatness for the observations rank histogram that could occur due to insufficient internal variability sampling or other factors. Therefore, as long as the observations rank histogram is within the perfect-model rank histogram range (e.g., as for detrended and non-detrended L8 examples in Fig. 1a, e), the model captures the variability in regionally averaged temperatures adequately for the region and season considered (Criteria 1; see “Methods” for further details).

This first evaluation criteria based on spatially aggregated metrics (e.g., time series and histograms in Fig. 1a–h), is complemented by a second grid-cell based evaluation criteria (e.g., Fig. 1i, j), to account for biases at the grid-cell level that may be smoothed or compensated by spatial averages. This second test follows similar uniform rank frequency principles, but is simplified to allow for a computationally efficient grid-cell level performance assessment. Instead of assessing the full spectrum of ranks against a perfect-model range, we assess only the minimum and maximum ranks and central rank sections against fixed frequencies assumed to be a reflection of uniform rank frequency distribution. Thus, this test assesses whether anomalies at the grid-cell level show model biases by observations clustering around the center bounds of the simulated ensemble spread or falling outside of its limits with too high frequencies (Criteria 2; see “Methods” for further details).

These two types of biases would equate to non-flat rank histograms. For variability overestimation biases, when observations cluster in the center of the ensemble spread, rank histograms would exhibit a mound or convex shape (e.g., detrended L12 (g) or both detrended and non-detrended L20 examples (d, h) in Fig. 1). For variability underestimation biases, when observations fall too frequently outside of both ends of the ensemble spread, rank histograms would exhibit a ’U’ or concave shape (e.g., detrended L19 example in Fig. 1f). For observations falling too frequently outside of only one side of the ensemble spread, this could indicate an asymmetrical variability underestimation bias (isolated most clearly for detrended data) or a bias in the forced response (i.e., too strong or too weak warming signal in the model compared to observations, most clearly visible for non-detrended data). In this case, rank histograms would exhibit a slanted shape (e.g., non-detrended L19 (b) and L12 (c) examples in Fig. 1). Therefore, the two evaluation criteria in this rank-frequency evaluation framework are as follows:
• Evaluation Criteria 1. Perfect-model rank range regional level performance: the observational rank histogram for regionally averaged temperatures must lie within the perfect-model rank range across all ranks (with a maximum 10% deviation).
• Evaluation Criteria 2. Threshold-based grid-cell level performance: at least 50% of the region’s grid-cells must be unbiased, meaning, at the grid-cell level, (a) observations do not cluster excessively within the central percentiles of the ensemble (i.e., observations do not occur with more than 80% frequency within the central 75th percentile ensemble bounds, indicative of variability overestimation bias) and (b) fall outside the ensemble range too frequently (i.e., observations do not occur with more than 8% frequency outside of the ensemble spread, indicative of variability underestimation bias for detrended data).

Only when both criteria are fulfilled do we determine that the model in question offers an adequate performance for the region and season considered, and is thus selected to be part of the constrained ensemble for that particular region and season. To create our performance-constrained ensemble based on how well the models represent the variability in historical observations, and to minimize the effect of potential model biases in the representation of forced changes and warming trends on our evaluation, we base this constraint on an evaluation of detrended temperature data (e.g., Fig. 2a; right column in Fig. 1). However, for users interested in assessing model performance in capturing both temperature variability and forced changes, we also provide the full evaluation for non-detrended temperature data (e.g., Fig. 2b; left column in Fig. 1). We also provide the full results of this evaluation, including evaluation time series and rank histograms and results for Criteria 1 and the grid-cell level assessment maps for Criteria 2 for all 11 SMILEs separately, for the DJF and JJA seasons and for both detrended and non-detrended temperatures against GISTEMPv4 and ERSSTv5 observations, respectively for land and ocean regions, in the Supplementary Information Section 1.

The results summarized in the temperature variability Evaluation Matrices (Fig. 2) reflect both evaluation criteria. Numbers mark the percentage of grid cells in the region that exhibit an adequate, unbiased representation of observations. The color shading marks cases that fulfill Criteria 1 (with the observations rank histogram within the perfect-model range). For the regions and seasons that also fulfill Criteria 2 (at least 50% of the grid cells are non-biased) and are deemed to be adequately simulated by a given model both at regionally-aggregated and at the grid-cell level, the fields are highlighted in green and considered adequately captured and part of the constrained ensemble. When only Criteria 1 is fulfilled (the rank of the spatially aggregated metric fits the perfect model range, but less than 50% of the grid cells are unbiased) the fields are highlighted in gray and considered inadequate.

Our evaluation shows that some regions are systematically not well represented (Fig. 2a) by all or most models across both seasons: e.g., Arctic (O1) and Antarctic (O2) Oceans, for which Criteria 1 is fulfilled, indicating adequate performance for spatially averaged temperatures, but exhibit unbiased area fractions below 10% of the region’s grid cells for most SMILEs. Other regions where most SMILEs offer an inadequate representation of temperature variability are the Southern Ocean (O9), the Indian Peninsula (L20), and Northern Australia (L22), as well as the Amazon basin (L6), Northern (L12) and Eastern Africa (L14), and Southern Australia (L23), all specifically in the local summer season. Our results also show that certain areas of the world are systematically well captured by most models, especially in the local winter season, namely North America (L1–L4), Southern South America (L8), Europe and Northern Africa (L9–L11) and North and Central Asia (L16–L19). We also find that local winter seasons are generally simulated better by most models more often than local summer seasons. This maybe because models struggle to simulate summer convective processes or land processes and land-atmosphere interactions more than they struggle to simulate large-scale winter circulation. Lastly, it is also worth noting that some of the regions that our evaluation highlights as systematically misrepresented in most models are regions with known widespread observational data quality issues, such as the Southern Ocean (O9) or Northern and Eastern Africa (L12, L14)38–40. This lack of sufficient temporal and spatial observational coverage could yield a potentially unreliable model performance assessment over these areas.

Our assessment also reveals some biases common across most models (Supplementary Information). Over land, most models show a marked overestimation of temperature variability over the tropics, especially over the Amazon basin, South Asia and Australia, both in DJF and JJA, and present for both detrended and non-detrended data. GFDL-SPEAR-MED and CESM-LE produce a substantially less biased temperature representation over these regions. Variability underestimation biases are relatively rare over land for most models, potentially due to our relatively conservative threshold for this bias to account for varying ensemble sizes (see “Methods” for further details).

Variability underestimation biases are much more commonly found over the oceans, especially once the confounding effect of forced signal biases is removed by basing the evaluation on detrended data. These variability underestimation biases over the oceans occur most commonly over the Eastern Pacific and North Atlantic oceans and near sea-ice edges in both polar oceans. In contrast, variability overestimation biases are also commonly found over the ocean for most models, especially over the Central Pacific and Southern Oceans, and both Polar Oceans, for which all models perform poorly across both seasons.

Strikingly, the Top 3 best performing models in our framework simulate 47 to 40 regions adequately, from a total of 66 regions across the 9 ocean regions and 24 land regions for the two seasons. The Top 3 best performing models in our framework in term of highest number of adequately simulated regions (in brackets) are: CESM-LE (47), CESM2-LE (43), GFDL-SPEAR-MED (40). The Top 3 best performing models based on non-detrended data (Fig. 2b), meaning the models offering the best general model performance for both variability and forced changes simultaneously, are CESM-LE (33), CESM2-LE (28), and MPI-GE5 (26). SMILEs most affected by forced response biases, as in exhibiting the largest decreases in adequately simulated regions using non-detrended versus detrended data, are CSIRO-MK3.6 (with 33 adequately simulated regions with detrended data to 10 with non-detrended data), GFDL-SPEAR-MED (40 to 21 adequate regions), and ACCESS (28 to 11 adequate regions). This performance drop indicates that these models misrepresent forced changes over large areas; even though some, such as GFDL-SPEAR-MED, may capture the variability in historical temperatures adequately.

The evaluation of non-detrended data (Fig. 2b and Supplementary Information Section 1) reveals generally worsened model performance over most ocean regions except the Northern Atlantic Ocean (O3) and over most generally poorly-model land areas, such as Central and South American (L4, L6, L7) and most of Asia and Oceania (L17–L24). This indicates that models present ample biases in their representation of forced changes over these areas, in addition to potential variability biases. However, in some limited cases, there may appear to be higher agreement between non-detrended data and observations than for detrended data (e.g., for L12 in JJA, as in Fig. 1 for CanESM5). This improved agreement could arise from the fact that misrepresented forcing signals can counteract commonly-found variability overestimation biases, e.g., observations warming faster than the model ensemble and thus crossing the ensemble spread from its lower to upper sections in the historical period, yet remaining within the ensemble limits due to an overestimation of variability (as in L12 for Fig. 1 or in the idealized example in Fig. 1f in ref. 33). This is just an example of how forced response biases may confound variability assessments. To avoid these confounding effects, we base our constraint criteria exclusively on detrended data. Note, however, that, particularly for observations, detrending may not perfectly remove all forced changes (see “Methods” for further details) and may influence potential discrepancies between observed and simulated detrended data. Users that seek an assessment of which models best capture real world behavior as observed, including both variability and forced signal responses simultaneously, may again refer to this non-detrended evaluation (Supplementary Information Section 1 and Fig. 2b).

Constrained vs. unconstrained temperature variability projections
The projections of temperature variability in each SMILE for all 5 warming levels and each region for DJF (Fig. 3) and JJA (Fig. 4) show that, typically, the multi-ensemble mean (MEM) exhibits the same sign of the change with increasing warming level in the full unconstrained ensemble as in the constrained ensemble (same sign of change as compared to the first bar which is variability at 1 degree of warming). Beyond the sign of the change, the absolute magnitude of the MEM temperature variability is qualitatively lower or similar in the constrained ensemble as in the unconstrained ensemble (see difference between colored and black solid lines). The exceptions where variability is larger in the constrained ensemble are Central North America (L3), Northern Asia (L16), Central Asia + Tibetan Plateau (L18), Eastern Asia (L19), South Pacific (O7), Indian (O8) in DJF and North Atlantic (O3), South Pacific (O7) in JJA. These results indicate that the MEM is generally a good representation of the sign of the change, also in the full unconstrained ensemble, but that the magnitude of the variability itself tends to be overestimated in the full ensemble as compared to the constrained ensemble. Additionally, while the constraint does not change the sign of the projected change in temperature variability, it can help to reduce the uncertainty in these changes, reducing the potential range of magnitudes of the change, particularly in regions that are poorly represented in climate models.

How this multi-model spread or uncertainty in the magnitude of the change (error bars in Figs. 3 and 4) changes between the constrained and unconstrained ensembles is somewhat complex. The error bars themselves show the multi-model ensemble spread, and are thus determined by the end members of the multi-model ensemble. The interquartile range of the multi-model ensemble is shown in the wider bars. By design, the error bars only decrease when the end members of the ensemble are removed. Therefore, only if our evaluation proves the more extreme model members to not perform adequately, and thus removes them from the constrained ensemble, does the uncertainty in the projections decrease. This becomes more likely to happen by chance with the removal of more models, but is not a predetermined nor expected outcome. Generally, constraining the projections to best-performing models yields a decrease in model spread, thus an increase in model agreement and improved (as in less uncertain) projections. For regions where substantially fewer models are used in the constrained ensemble (6 or less), the model spread decreases substantially, as seen for example for the Amazon (L6) in both seasons or Southern Australia (L23) in DJF or the Tropical Pacific Ocean (O6) in JJA. However, this is not always the case, as seen in North Atlantic (O3) in DJF and Western North America and Central North America (L2 and L3) in JJA where the constrained ensemble consisting of 3 and 6 adequately performing models respectively has an almost identical spread as the unconstrained 11-model ensemble. We note that this exception is, albeit interesting, rare. Over several regions, the constrained projections indeed show an increase in model agreement, even for a large number of selected best-performing models. This can be seen for example in Northern Europe (L9) and Central North America (L3) in DJF with 8 and 9 models respectively included or in Northern Asia (L16) and Southern Australia (L23) in JJA both with 8 models included. These results indicate that performance constraining can indeed decrease the uncertainty in future variability projections, although this uncertainty is not eliminated completely, with decreased yet substantial remaining multi-model spreads in the constrained ensemble for most adequately modeled regions and uncertainties still remaining large over key, poorly-modeled regions.

The magnitude of the temperature variability at 1°C (Fig. 5; DJF & Fig. 6; JJA — left column panels) in both seasons is typically lower in the constrained compared to the full ensemble also at the grid-cell level, except for in the northern hemisphere extratropics in DJF. This highlights the improved representation of the present day variability, as the lower variability in the constrained ensemble over land is in better agreement with observations, except for again in the northern hemisphere extratropics in DJF (Fig. 7). We note that the variability over the ocean shows little improvement in the constrained ensemble, and tends to be overestimated in DJF and underestimated in JJA. While the variability itself is lower in the constrained ensemble, the change in variability from 1° to 3°C of global warming increase (Fig. 5; DJF & Fig. 6; JJA — second column panels) tends to be larger in the constrained ensemble compared to the unconstrained. In particular, the variability in the constrained ensemble is much larger in both seasons over Northern South America and is somewhat larger over central Africa, Australia, and the extratropical Indian Ocean in JJA. The change in variability itself in both ensembles is typically positive in sign except for most of the northern hemisphere extratropical and higher latitude land masses and parts of the ocean in DJF in agreement with previous work1,2,12,14–21.

Over three key areas that are typically poorly represented in many climate models, namely Australia & South-East Asia, South America, and Africa (Figs. 5 and 6), temperature variability is overestimated at the 1°C warming level as compared to observations (Fig. 7). This overestimation, while present in both seasons, is largest in the local summer: for Australia and central South America in the austral summer (DJF), and larger in Northern Africa in JJA. This overestimation, while still present, is smaller in the constrained ensemble (Fig. 7). While the mean estimate of temperature variability is overestimated, the projected variability change from 1–3°C is typically larger in the constrained ensemble over these regions. This overestimation of the variability itself at 1°C and underestimation of the change from 1°C to 3°C manifests as larger variability in the full ensemble at 3°C than the constrained ensemble in these regions (Fig. 7 left panels). This implies that the absolute value of the variability at 3°C is overestimated in the full ensemble, but the change in variability may be underestimated, meaning that the interpretation of results could be biased in either direction depending on whether one computes the actual value of future variability or the value of the future change in variability. This is particularly true over South America in both seasons and central and South Africa and Australia in JJA, emphasizing that our projections of increased temperature variability and consequently temperature extremes may be underestimated in these poorly represented regions.

While we cannot assess whether the future projections are more realistic in the constrained ensemble compared to the full ensemble, as we do not have observations of the future, we can compare the projections with those from the most realistic model over these poorly modeled regions in the historical period. CESM2 is one of the two models that best represent Australia & South-East Asia (L21–23), South America (L6–8), and Africa (L12–16; the other being CESM-LE), with 15 “good performances” out of 22 across both regions and seasons. By using this model as a proxy for possible future observations, we can compare the differences at 3°C of warming between the full and constrained ensemble, the constrained ensemble (excluding CESM2), and CESM2 (Fig. 8; see Supplementary Information Figs. 3.1 and 3.2 for a spatially aggregated comparison across all regions). We find that, except for South America (where CESM2-LE fails to capture historical variability according to our evaluation for L6 and L7) and Africa in DJF the constrained estimate is closer to CESM2 (our “good model”) than the full ensemble, tentatively suggesting that our future projections might be more realistic in the constrained ensemble compared to the full ensemble.

Finally, we assess model agreement in two ways. First, we consider agreement on the sign of the change. We find some patches of improved model agreement on the sign of the temperature variability change (Figs. 5 and 6, when panel i is blue) in India and Australia in DJF and small parts of South America, Africa and all of northern Australia in JJA. Next, we assess model agreement on the range of potential magnitude of the change. This is assessed by taking the standard deviation across the multi-model ensemble constituted by each model’s mean change in temperature variability. Regions with high uncertainty in the magnitude of projected temperature variability change (Figs. 5 and 6 j, k — darker green colors) in both seasons are the Tropical Pacific Ocean, Southern Ocean, Northern South America and the northern Hemisphere extratropical land surface. We find that the uncertainty in the magnitude of the change is similar or larger in the full ensemble compared to the constrained ensemble, except for in patches of the ocean. Regions that show substantially less uncertainty in the magnitude of the change in the constrained ensemble are South America, India, Australia & South-East Asia, the west Pacific Ocean and Southern Ocean in DJF and north South America, Africa, parts of Australia and the Southern Ocean in JJA. Typically, regions where model agreement on the sign of the change is substantially improved are regions where the constrained ensemble includes many fewer models than the full ensemble. Note, however, that this is not a predetermined result. We tested the change in the standard deviation (understood as the uncertainty in the magnitude of the change) in a constrained ensemble of either 4 or 6 randomly selected models by sampling 1000 times, and find the standard deviation decreased approximately 2/3 of the time and increased 1/3 of the time.

In summary, our results highlight that constraining by historical model performance in the past does not necessarily improve model agreement in the future. However, while the agreement on the sign of the change is not drastically improved by the constraint, the spread of projections and hence the uncertainty in the variability change can be limited using this method. This shows that selecting models that best simulate historical variability can still be useful, albeit not a silver bullet that can be used to rectify all model differences and uncertainty in future temperature variability change projections.

Methods
Climatemodelsimulationsandobservationaldata
Weinclude11SMILEs from abroad range ofclimatemodels across differentCMIPgenerations:ACCESS-ESM1.541,CanESM242,CanESM543,CESM-LE44,CESM2-LE45,CSIRO-MK3.646,GFDL-ESM2M47,GFDL-SPEAR-MED48,MIROC649,MPI-GE550andMPI-GE651.
EachSMILEconsistsofmanysimulationsforeachclimatemodel thatdifferonlyintheirinitialstate,andevolveunderthesamespecific forcing conditions. However, the SMILEs differ in the number of simulationsincluded(from30upto100members),intheirrateof warmingunderincreasinganthropogenicemissions(EquilibriumCli- mateSensitivity,ECS,values of2.4Ktomorethan5K),in the initi- alizationmethod(frommicroatmosphericperturbationstodifferent initialstatessampledfromthecontrolsimulation),inthegenerationof forcingscenariosused(CMIP5toCMIP6),andintheforcingscenario (historical simulations are extended with a high emissions scenario such as RCP8.5, SSP370 or SSP585). More details can be found in Table1,andinpreviousstudies33,50,52,53andreferencestherein.
Observed surface air temperature data from the GISSTEMPv439 datasetfortheperiodof1880-2024andseasurfacetemperaturedata from ERSSSTv554 from 1854–2024 are used for the evaluation and comparisontotheSMILEsimulations.Wedefinesurfaceairtempera- tures(TAS)asthenear-surface2mairtemperatureanomalyoverland grid cells, and sea surface temperatures (SST) as the surface tem- peratureoveroceangridcells.Forthepurposeofcomparingmodel simulationsandobservations,allsimulateddataareregriddedtothe coarserresolutionofGISSTEMPv4(180×90)andERSSTv4(180×89) observations,andsubsampledtogridboxeswhereobservationsare available. Both observed and simulated data are compared as anomalies calculated with respect to the climatological baseline defined by the period 1961–1990 in each ensemble member and observations.
Wedefine24landregionsand9oceanregions(Fig.1k,l)roughly followingtheIPCCSREXregiondefinition55.Someregionshavebeen mergedorsightlyreshapedtoavoidhavingregionsconsistingoftoo fewlandoroceangridcells,respectively.
As projections of temperature variability depend on both the greenhousegasemissionsandtheclimatesensitivityofthemodel22; webaseourmulti-modelcomparisononwarminglevelsratherthan selecting specific time periods. Thisallows fora comparison thatis independentofhowfastamodelwarms.Warminglevelsareshownin SupplementaryInformationTable1,anddefinedrelativetoabaseline periodof1850–1899ineachSMILE.Eachwarmingleveliscomputedas theyearwhentheglobalmeansurfacetemperature(annual)fromthe SMILEensemblemeancrossesanindividualwarminglevel.

Rank-frequencyvariabilityevaluationframework
The rank-frequency evaluation framework applied in this study determineshowwellmodelscapturetheinternalvariabilityinobser- vations based on a simple principle: whether the range of climate states, in this case defined by winter and summer monthly mean temperatureanomalies,assimulatedbyaclimatemodelagreeswell withtherangethatobservationscoverinthehistoricalperiod.This frameworkallowsustoassessamodel’sperformanceinsimulatingthe variabilityinobservationswithouttheneedtoparametrizeormake anyassumptionsregardingtheshapesoftheobservedorsimulated distributions.Thebasisofthisevaluationframework,whichresembles probabilisticforecastverificationtechniquesintheclimateprediction literature34–36,isdemonstratedforannualmeantemperaturesand10 SMILEsinref.33.ItwasfirstdevelopedtoevaluateEuropeansummer temperature and precipitation in MPI-GE5 in ref. 56; Supporting InformationFigs.S1–S3;andfurtherexpandedgloballyforMPI-GE5in ref. 50 and in ref. 11 for annual mean temperatures and summer maximum temperatures, respectively, as well as in ref. 57 for temperaturesoverNorthAmericainsixSMILEs.Forfurtherdetailson the existing framework and its theoretical justification and inter- pretationbasedonidealizedandspecificexamplesseeref.33.
Here,weexpandonthisexistingevaluationframeworkbyincor- poratingtwoformallydefinedevaluationcriteria,whichagainassess modelperformancebasedonwhetherobservationsoccuruniformly acrossallensembleranksofanSMILE(i.e.,thepositionanobservation takesamongthesortedensemblemembersforagiventimestep).The firstcriteria,basedonspatiallyaveragedmetrics,determineswhether observationsoccuracrossallrankwindowswithuniformfrequency. Traditionally,thishasbeenassessedintheensembleforecastinglit- eraturebyevaluating rankhistogramflatness35,37.However,therela- tivelyshortlengthoftheobservationalrecordmeansthattheinfluence ofinternalvariabilitymaynotberobustlysampledonthesetimescales. This potentially insufficient variability sampling, combined with the fact that we are comparing observations against uninitialised, free- running modelsimulationsmeansthat,withthesamplesizeavailable, rank histograms may not be perfectly flat, even for perfectly per- formingmodels.
Toovercomethis,weapplyaperfect-modelrankrangetest.This rangeisconstructedbytreatingeachensemblememberineachSMILE asifitwereobservations,andcalculatingtheresultingspreadofmodel rankhistograms.Thisapproachprovidesarelativelywidedistribution ofrankhistogramsthatreflectperfect-modelbehavior,yetmayindeed notbeperfectlyflat.Thesedeviationsfromflatnessinaperfect-model set up can occur either systematically because the simulated tem- peraturedistributionisskewedornon-normal,orbecausetherecord length is too short to sufficiently sample rank variability, or both. Similarly,theobservationalrankhistogrammayalsoappeartobenon flatforthesamereasons,withoutnecessarilyimplyingincorrectmodel performance.Sinceinaperfect-modelsetupeachmembercapturesits modelbehaviorperfectly,thisrankrangegivesusabaselineofpos- sibledeviationsthatcouldoccurduetoinsufficientinternalvariability samplingorotherfactors,evenformodelsthatcaptureobservations adequately.
Therefore,wedetermineadequatemodelperformanceforspa- tiallyaveragedmetricswhentheobservationsrankhistogramiswithin the perfect-model rank histogram range (Criteria1). In this test, we allow for a maximum deviation of 10% of the rank frequencyvalue beyond the perfect-model range bounds (so for an upper perfect- modelrankfrequencyboundof3%foranyparticularrank,themax- imum allowed rank frequency for observations would be 3.3%). To minimizetheeffectofvaryingensemblesizesandpotentiallyspurious effectsresultingfromtheinsufficientrankvariabilitysamplingdueto thefinitesamplesize,thistestisperformedoverwindowsofaggre- gatedranks.Thereforerankfrequenciesareassessedastherunning meanacrossindividualrankfrequenciesoverann/5rankwindow,with n being the ensemble size (for n=50 members, 10-rank running average)forranks1ton−1,andastheactualrankfrequencyforranks 0andn(e.g.,Fig.9).
Thisfirstevaluationcriteriabasedonspatiallyaggregatedmetrics is complemented by a second grid-cell level evaluation criteria, to account for biases at the grid-cell level that may be smoothed or compensatedinthisspatialaverage.Thissecondtestfollowssimilar uniform rank frequency principles, but is simplified to allow for a computationally efficient grid-cell level performance assessment. Insteadofassessingthefullspectrumofranksagainstaperfect-model range,weassessonlythetop,bottomandcentralranksectionsagainst fixed frequencies assumed to be a reflection of an adequate rank distribution. Therefore,this test assesses whether observed anomaliesatthegrid-celllevelclusteraroundthecenterboundsofthe simulatedensemblespreadorfalloutsideofitslimitswithtoohigh frequencies.
We determine when these grid-cell level variability biases are present(Criteria2),accordingtothefollowingprinciples:(a)Avaria- bility overestimation bias is detected when observed values cluster withinthecentral75thpercentileensemblebounds(12.5thto87.5th percentiles) more frequently than 80% of the time steps. This bias impliesthatthemodelsimulationsaresystematicallymoreextreme than observed values and the width of the distribution is over- estimated by the model,and would result ina mound shaped rank histogram. (b) A variability underestimation bias is detected when observed values exceed the ensemble minima or maxima, also understood as observations taking minimum or maximum ranks respectively,morethan8%ofthemonths.Thisbiasimpliesthatthe model fails to simulate extreme enough events with adequate fre- quency,whichmeansthatobservationsexhibitasystematicallywider distributionthanthemodel,andwouldresultinaconcaveshapedrank histogram.Toaccountfornon-symmetricalbehavioroverthewarm andcoldtailsofthetemperaturedistributions,weaccountforthisbias invariabilityunderestimationwhenobservationsfalltoooftenbelow the ensemble minima or above the ensemble maxima separately. Lastly,whenneitherbias(a)nor(b)arepresent,wedetermineagrid- celltobeunbiased.Thereforeforthesecondcriteria,weconsidera regionandseasontobeadequatelysimulatedbyagivenmodelatthe grid-cell level when at least 50% of the grid cells in the region are unbiased(Criteria2).
Notethatthevariabilityunderestimationthresholdof8%ischo- sentoberatherconservativetoaccountforthedifferentensemble sizesandrelativelyshortobservationalrecordconsideredinthiseva- luation.Fora50-memberperfect-modelensembleassessedoveran infinitely long observational record, observations would exceed the ensemblemaxima,whichmarksonaveragea1-in-50-yearsevent,2%of thetime,ortwiceacentury.Duetointernalvariability,thisfrequency may fluctuate in any given century, as seen in the minimum and maximumrankfrequenciesforperfectmodelranges(graycrossesin the rank frequency histograms in the Supplementary Information evaluation or in Fig. 9) oscillating between 0 to over 10% for 50- memberensembles.Additionally,theseperfectmodelrangesinthe minimumandmaximumrankfrequenciesalsofluctuateforregions thatexhibitnon-normalorskewedbehaviorintheirtemperaturedis- tributions. This means that in some particular cases this variability underestimationthresholdfrequencymaybelargerthanthetheore- ticalexpectations,eveninaperfectmodelsetup.Forthisreason,and intheinterestofsimplicityforourevaluationframework,wechoosea fixed frequency of 8%in ensemble limit exceedance as a variability underestimation bias threshold. For a more in-depth assessment of thesebiaseswerecommendtoaccountfortheeffectofensemblesize andassignavaryingthresholdforthiscriteriaforeachSMILEassessed, ideallydependingontheensemblesizeandfollowingperfect-model behaviorasguideline.Similarly,thevariabilityoverestimationthresh- oldfrequencydetermininghowoftenobservationsmayclusterwithin thecentral75thpercentileoftheensemblespreadissetat80%,and notat75%,toaccountforthiseffectofinternalvariability.
Therefore,thetwoevaluationcriteriausedinourframeworkare asfollows:
• Evaluation Criteria 1. Perfect-model rank range regional level performance: the observations rank histogram for regionally averaged temperatures must lie within the perfect-model rank rangeacrossallranks(withamaximum10%deviation).
• Evaluation Criteria 2. Threshold-based grid-cell level performance:atleast50%oftheregion’sgrid-cellsmustbeunbiased, meaning, at the grid-cell level, (a) observations do not cluster excessively within the central percentiles of the ensemble (i.e., observationsdonotoccurwithmorethan80%frequencywithin the central 75th percentile ensemble bounds, indicative of variabilityoverestimationbias)and(b)falloutsidetheensemble rangetoofrequently(i.e.,observationsdonotoccurwithmore than8%frequencyoutsideoftheensemblespread,indicativeof variabilityunderestimationbias).
Onlywhen both criteriaarefulfilled, we consider the model in questionoffersanadequateperformancefortheregionandseason considered,andisselectedtobepartoftheconstrainedensemblefor theparticularregionandseason.
These evaluation results are summarized in the temperature variability Evaluation Matrix for different ocean (O1–O9) and land regions(L1–L24)forDJFandJJAmonthsforthe11SMILESfordetren- deddatausedforourassessmentconstraint(Fig.2a)andadditionally fornon-detrendeddata(Fig.2b).Numbersmarkthepercentageofgrid cellsintheregionthatexhibitanadequate,unbiasedrepresentationof observations.ThecolorshadingmarkscasesthatfulfillCriteria1,with theobservationsrankhistogramwithintheperfect-modelrange.For theregionsandseasonsthatalsofulfillCriteria2andaredeemedtobe adequatelysimulatedbyagivenmodelatthegrid-cell(atleast50%of thegridcellsarenon-biased),thefieldsarehighlightedingreenand considered adequately captured. When only Criteria 1 is fulfilled, meaningtherankofthespatiallyaggregatedmetricfitstheperfect modelrangebut50%orlessofthegridcellsareunbiased,thefieldsare highlightedingrayandconsideredinadequate.
Wealsoshowthefullresultsofthisevaluation,includingevalua- tiontimeseriesandrankhistogramsandresultsforCriteria1andthe grid-celllevelassessmentmapsforCriteria2forall11SMILEssepa- rately,fortheDJFandJJAseasonsandforbothdetrendedandnon- detrended temperatures against GISTEMPv4 and ERSSTv5 observa- tions, respectively for land and ocean areas, in the Supplementary Information.
Forthepurposesofthisevaluation,weusemonthlymeantem- peratureanomaliesrelativetotheperiod1961–1990,andmodelout- putdataareregriddedtomatchthedifferentobservationalgridsfor land-andocean-basedevaluations.Allevaluationsareperformedover theperiodforwhicheachobservationalrecordisavailablestartingin 1900 until 2024, and restricted to the period when simulations are availableforthemodelsthatspanshorterperiodsthanobservations. Toavoidspuriouseffectsofin-homogeneousvariabilitybehaviorover theAmazonregioninsomemodelsoverthefirstyearsoftheirsimu- lations, Amazon land region (L6) evaluations start only from 1920 onward.Toisolatevariabilitybiasesfrompotentialbiasesintheforced warming rate, we base our main performance assessment and con- straint on detrended temperatures, and provide the analogous assessment on non-detrended temperatures for comparison.
Detrendingisdoneformodeldatabysubtractingfromeachmodel member the model’s ensemble mean, and by subtracting a least squaresquadratictrendfromobservations,bothatthegrid-celllevel. Notethat,incontrasttosubtractingtheensemblemean,subtractinga quadratictrendmaynotensureaperfectremovaloftheforcedsignal fromobservations,andremainingforcedeffectsinobservationsmay contribute to some of the discrepancies found in our evaluation. Lastly, to allow a more finely resolved temporal analysis and to increase sample size we perform this rank-frequency evaluation on monthlymeantemperatureanomaliesinsteadofseasonalaverages.

Variabilityestimates
Temperature variability is computed for each season at each of 5 warminglevelsasthestandarddeviationoftemperaturepooledfrom eachofthe3-monthsintheseasonfor11-yearscenteredontheyear definedforeachwarminglevel.Allothercalculationsusethemetho- dologyofref.23totakemulti-ensemblemeansandcalculatechanges in temperature variability.Calculations for the full ensemble useall datafromallSMILEs,whilecalculationsfortheconstrainedensemble usedatafromtheSMILEsdeemedasadequateinthevariabilityeva- luationframeworkinFig.2aforeachspecificregionandseason.The errorbarsinFigs.3and4aredefinedastheminimumandmaximum valuesacrosstheensemble.Modelagreementinthethirdcolumnof Figs.5and6isdefinedaswhengreaterthan50%(lightblueandred) and80%(darkblueandred)ofthemodelsintheensembleagreeon thesignofthechange.Uncertaintyinthemagnitudeofthechangein thefourthcolumnofFigs.5and6isshownasthestandarddeviation acrosseachmodel’sindividualestimateintheensemble(i.e.forthefull ensembleacross11estimates,onefromeachmodel).Resultsin Figs.3–8areshownasthemulti-ensemblemean(MEM)oftheindivi- dualSMILEresults.ResultsaretypicallyshownfortheMEMofthefull and the MEM of the constrained ensemble separately, with the dif- ferencealsoplotted.WenotethatFigs.3–6excluderegionsArctic(O1) andAntarctic(O2)asnomodelsperformadequatelyhere,howeverwe areconcernedsea-icemayinfluenceourresultsinthisregionwhen calculatingvariability.

Dataavailability
TheSMILEdatausedinthisstudycanbefoundonacommongridin theMultiModellarge ensemble archive version2 [MMLEAv253]and downloadedfromtheNSFNCARGeoscienceDataExchangehttps:// www.cesm.ucar.edu/community-projects/mmlea/v2.GISSTEMPv4can be downloaded from NASA-GISS at https://data.giss.nasa.gov/ gistemp39andERSSTv5canbedownloadedfromtheNOAAPhysical Sciences Website https://psl.noaa.gov/data/gridded/data.noaa.ersst. v5.html54.

Codeavailability
Allcodeusedinthisresearchcanbefoundathttps://doi.org/10.5281/ zenodo.17058694. The analysis and figures in this articlehave been created using Climate Data Operator (CDO) software, Python and NCARCommandLanguage(NCL,NCAR2019;Version6.6.2).

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "Summary and main comments\nThe authors evaluated historical simulations of seasonal (DJF and JJA) temperature variability in 11 large ensembles\ncompared to observations using rank-frequency analysis (known as rank histograms in the ensemble forecasting literature).\nModels whose historical performance was judged adequate (by region and season) were included in a performance-\nconstrained ensemble. Projections based on the full and performance-constrained ensemble were compared.\n1. The introduction provides few (one?) references to application of performance-based constraints/subsetting to projections.\nThere is an extensive literature on this topic, and I think it would be helpful to connect the current work to it. The other topics\n(importance of temperature variability and SMILES) are well introduced. A arbitrarily chosen example is\nBrunner, L., Pendergrass, A. G., Lehner, F., Merrifield, A. L., Lorenz, R., and Knutti, R.: Reduced global warming from CMIP6\nprojections when weighting models by performance and independence, Earth Syst. Dynam., 11, 995–1012,\nhttps://doi.org/10.5194/esd-11-995-2020, 2020.\nThis would also provide an opportunity to give the reader a clear idea of what has been done before in this direction and\nhow this work improves upon previous efforts. It may be the case that model weighting/subsetting has not been applied to\nquestions of variability changes, and if so, pointing this out would highlight the value of the work. Or perhaps other\napproaches suffer from limitations that are addressed here. At present, I found the limitations of past work/goal of present\nwork statement not very clear (line 43): \"However, it remains unclear whether climate models disagree in the direction and\nintensity of temperature variability changes because they do not adequately simulate temperature variability over certain\nregions, or because of other, more systematic model differences in future temperature variability behaviour. In this study, we\naddress this question by constraining projections of temperature variability in a multi-model super ensemble of single model\ninitial-condition large ensembles (SMILEs) to models that adequately represent observed regional variability under current\nclimate conditions.\"\n2. Rank-frequency analysis or more commonly rank histograms are a standard metric for assessing the reliability of\nensemble forecasts. Classical references are\nAnderson, J. L., 1996: A method for producing and evaluating probabilistic forecasts from ensemble model integrations. J.\nClimate,9, 1518–1530.\nHamill, T. M., and S. J. Colucci, 1997: Verification of Eta–RSM Short-Range Ensemble Forecasts. Mon. Wea. Rev., 125,\n1312–1327, https://doi.org/10.1175/1520-0493(1997)125<1312:VOERSR>2.0.CO;2.\nTalagrand, O., R. Vautard, and B. Strauss, 1997: Evaluation of probabilistic prediction systems. Proceedings, ECMWF\nWorkshop on Predictability, ECMWF, 1–25. [Available from ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, United\nKingdom.].\nI think that acknowledging the connection to forecast verification would provide additional support for using this approach.\nMore importantly, there are standard (perhaps many?) methods for assessing the flatness of the rank histograms. These\nmethods take into account the number of verifying observations and ensemble size. The present work does not use any\nstatistical methods to assess the flatness of the rank histogram and instead introduces admittedly ad hoc procedures to\ndetermine whether model performance is adequate. Using rigorous methods to assess the rank histograms would better\nquantify uncertainty and potentially strengthen and simplify the results. An important question which could be answered\n(currently unanswered) is whether *any* of the simulations have variability that is statistically consistent with observations. A\nclassic reference on assessment of rank histograms is\nJolliffe, I. T., and C. Primo, 2008: Evaluating Rank Histograms Using Decompositions of the Chi-Square Test Statistic. Mon.\nWea. Rev., 136, 2133–2139, https://doi.org/10.1175/2007MWR2219.1.\nThere may be better, newer methods, too.\n3. As noted in point 1, I did not find it completely clear what hypothesis is being tested in the current work or exactly what the\nconclusion is. Presumably, the overall goal is more accurate projections. This, of course, is impossible to assess since we\ndon't know the future. Moreover, there is the familiar issue that past performance does not guarantee future performance, a\npoint which I did not see mentioned. The present work emphasizes reductions (or lack of) in projection uncertainty in the\nperformance-constrained ensemble. However, as noted in the text, these reduction might simply be due to removing models.\nPerhaps this hypothesis could be tested. That is, are the changes in uncertainty greater than would be expected if models\nwere removed at random instead of based on historical performance?\nSince we don't have observations of the future, a standard method of evaluating model weighting/subsetting schemes is the\nso-called perfect model approach where one model at a time is treated as \"observations.\" By doing so, it can be checked\nwhether a subsetting strategy based on historical performance actually gives more accurate projections. This method is\nwidely used, including in the Brunner reference above and this one\nKnutti, R., J. Sedláček, B. M. Sanderson, R. Lorenz, E. M. Fischer, and V. Eyring (2017), A climate model projection\nweighting scheme accounting for performance and interdependence, Geophys. Res. Lett., 44, 1909–1918,\ndoi:10.1002/2016GL072012.\nNote that both works also consider model independence which might not be an issue here but could be considered and\nmentioned.\n4. Figures 2–5 show constrained vs. full ensemble results and differences without statistical significance statements. I find it\nhard to interpret whether the results are \"significant\" or just \"different.\" I think the sample size is fairly large (11 years x\nensemble size) but variance tends to be more subject to sampling variability than means. I'm not sure that the ensemble-\nbased error bars correctly capture the uncertainty since sampling variability goes up with reduced ensemble size (right?) and\nthe error bars would tend to shrink?\nOther comments.\nI would find the figures easier to read if the captions and plots used the term standard deviation instead of temperature\nvariability. Figures 2 and 3 do say \"temperature variability defined as the standard deviation\" but 4 and 5 don't. Seems like\nstandard deviation would be more precise and concise.\nLine 43. \"However, it remains unclear whether climate models disagree in the direction and intensity of temperature\nvariability changes because they do not adequately simulate temperature variability over certain regions, or because of\nother, more systematic model differences in future temperature variability behaviour. In this study, we address this question\nby constraining projections of temperature variability in a multi-model super ensemble of single model initial-condition large\nensembles (SMILEs) to models that adequately represent observed regional variability under current climate conditions.\"\nThis seems important but was not very clear to me how the question was being addressed. I think the idea (after reading the\npaper) is that if the performance-based ensemble doesn't have smaller uncertainty, then other differences (i.e., ones that do\nnot result in variability biases during the historical period) are present.\nFigure 1. What is the purpose of the time series shown? Are they mentioned in the text? I was unable to make out the very\nlight red and blue shading. The caption does not say what say what the Above, Below, and Center values are. I think the\ngray lines are a 90% confidence interval but I could not follow the description (below) in the caption. In particular, if the lines\nillustrate the slope, the slope should be zero but I don't see zero being plotted. Perhaps say what the lines represent and\ngive the details in the methods section instead of the caption. I didn't see any more details in the current methods section.\nFig. 1 caption: \"Lines illustrate the rank histogram’s slope, as the mean rank frequency over a centered 6-bin window for\nobservations (coloured lines), and the 5-95th percentile perfect model range (the slopes of all ensemble members treated as\nif they were observations; gray dashed lines).\"\n\"Crosses represent the frequency of minimum (0) and maximum (number of members) ranks for observations (colors), and\nfor the 5-95th percentile perfect model range (gray)\" Isn't the frequency of the observation rank already given by the bars?\nLine 113-116. \"we also include time series and rank frequency histograms for all regions and models where model\nperformance based on spatially aggregated data can be assessed. Furthermore, time series for this spatially aggregated\nevaluation also allow changes in potential biases over different periods to be assessed.\" I don't understand what it means to\ninclude time series or what the time series in question are. If this is a technical detail, perhaps keep move it to the methods\nsection where it can explained more completely. Also repeated later.\nTable 1 caption \"total amount\" -> \"number\"\nLine 132 \"struggling\" -> \"struggle\"\nFig. 2. Starting at zero seems to waste considerable space and reduce readability. There is no objective measure here of\nwhether projection values are different between the constrained and full ensembles and also whether uncertainty is reduced.\nThis issue is especially unclear when only a few models are retained.\n\"Horizontal lines are plotted at both 1 degree (solid line) and 4 degrees (dashed line) of warming.\" Why? Don't the bars\nshow the same information? Perhaps it would help to put the constrained and unconstrained values for the same warming\nlevel beside each other to aid comparison.\nLine 206 \"constraining the projections to best-performing models yields a decrease in model spread\" Would selecting any\nsubset of models (including randomly) yield a decrease in spread?\nLine 207. \"Less\" -> fewer\nLine 215. \"These results indicate the performance constraining can indeed decrease the uncertainty in future variability\nprojections\" It seems likely that many criteria for removing models reduce spread.\nLine 233. \"temperature variability is significantly overestimated at the 1oC warming level.\" Compared to observations? Or\nsomething else?\nFig. 4. Is it possible to indicate statistical significance of the differences?\nFig. 4 caption and elsewhere. Is that is a degree symbol?\nThe difference of STD in Fig. 4l looks small but I'm not sure what is considered small.\nLine 243. \"This implies that the absolute value of the variability at 3oC is overestimated in the full ensemble, but the change\nin variability is underestimated, meaning that the interpretation of results could be biased in either direction depending on\nwhich metric is used.\" I find this hard to follow. What does \"the absolute value of the variability\" mean? STD is always\npositive? Which \"interpretation of the results\" is being referred to here?\nLine 249 \"it's\" -> its\nFig. 5. Possible to add or comment on statistical significance? Especially for panel l.\nLine 257. \"Typically regions where the constrained ensemble includes many less models than the full ensemble are regions\nwhere model agreement is substantially changed.\" Is this obvious? If so, does it provide a meaningful conclusion? Also less\n-> fewer.\nLine 260. \"the spread of projections and hence the magnitude of the change can be limited using this method.\" Is this the\ngoal? Is it a meaningful metric of success? This is a place where the using one model as observations might permit stronger\nresults.\nIs Fig. 6 mentioned in the text? Does Fig. 6 provide anything that's not in Figs. 4 and 5? Is a t-test the right test for a\ndifference in variances? Or F-test? Though this is a difference in variances. Again, isn't comparing the full ensemble to one\nwith fewer models a bit of a straw man? Also I cannot visually tell the difference between small values (white) and\ninsignificant values (also white?). What does \"general region\" mean in the caption? Is there some smoothing?\nLine 267. \"This evaluation enables end-users of these ensembles to select the best fitting model for their region and time of\ninterest\" Is there evidence that such a strategy leads to more accurate projections in the perfect model (each model in turn as\nobservations) setting?\nLine 274. \"they may, however, underestimate the projected temperature variability change\" or they may not? Right?\nLine 277. \"providing an improved estimate\" In what sense is the estimate improved?\nLine 281. \"In these regions we find that temperature variability itself is typically overestimated in present day climate; while the\nconstrained ensemble provides a larger projected change in variability.\" It would be helpful to provide the reader with\npointer to where these results appear. Or perhaps some summary graphic showing this relation?\nLine 310. \"Our attempt to constrain projections of future temperature variability change is an example of change in the\nclimate system that cannot yet be foreseen with the best available knowledge.\" What does this mean?\nLine 365. \"date\" -> data\nLine 370. \"Lastly, to allow a more finely resolved temporal analysis and to increase sample size we perform this evaluation\non monthly mean temperature anomalies instead of seasonal averages.\" What evaluation? The rank-frequency evaluation\nor another diagnostic?\nLine 384 \"To illustrate how internal variability may affect rank frequencies\" isn't the rank histogram itself assessing\nvariability? Or do you mean finite sample size? Line 397 \"To quantify when variability biases are present, we select the\nfollowing thresholds.\" These seem arbitrary. It would be sensible to include some sort of significance testing of rank\nhistograms. Line 408. \"Note that the variability underestimation threshold of 8% is chosen to be rather conservative to\naccount for the different ensemble size\" Wouldn't it be more objective to use a standard significance test which accounts for\nensemble size? Perhaps these are all points where a method that takes into account sample and ensemble size would be\nbeneficial.\nLine 385 \"and\" -> an\nLine 428 \"Lastly, we determine, per grid-cell, if a model simulates observed temperatures adequately when none of these\nbiases are present.\" Is this saying when none of the biases in the previous paragraph are present, an additional procedure is\napplied to determine if a model simulation is adequate? If so what is that procedure?\nLines 430–433 are a repeat of lines 111–114? I still don't quite get what the time series refer to.\nLine 459. \"Significance for Figure 7 is calculated using a t-test approach for each domain.\" Fig. 7 is the domains?", "reviewer_2": "Review for “Temperature variability projections remain uncertain after constraining them to best performing SMILEs”\nI have read the manuscript and have several concerns, mainly about the scope of the study. The authors essentially use\noutput from large ensembles of climate model simulations, evaluate whether or not they have a good representation of\nhistorical temperature variability, then assemble a “constrained ensemble” based on admittedly subjective criteria to\nevaluate how much removing models with poor representations of historical variability changes future projections. In\ngeneral, they find that the answer is basically unchanged with or without their constraint.\nWhile this is a worthy effort, I do not believe this constitutes a novel study that adds to our understanding of temperature\nvariability in climate model simulations. The authors provide a detailed description of their results, and while I have no\nobjection to their findings or methodology (though I am slightly confused by the ranking figure) I do not believe this paper\nconstitutes a sufficient advance to be considered in Nature Communications.\nMajor Comments:\nFigure 1: The shading is extremely difficult to see, which makes this plot difficult to interpret. I found the discussion of rank\nhistograms slightly confusing and thought the authors should say more about what a “desirable” scenario would be for these\nhistograms. Presumably it would be that the observations sit somewhere in the middle of the ensemble, but that’s just my\nguess.\nThe discussion of detrended versus non-detrended data was interesting, but qualitative and speculative. I was also\nconfused by lines 172-175, where the authors say “The higher agreement between non-detrended data and observations\ncould arise from the fact that misrepresented forcing signals can counteract commonly-found variability overestimation\nbiases” Wouldn’t this be the opposite? I would think that higher agreement between non-detrended data and observations\nwould arise from accurate representation of the forcing signal, rather than misrepresentation.\nMinor Comments: Figure 6 never called or discussed?"}, "author_response": "Response to the Reviewers' comments on manuscript NCOMMS-25-1462\n“Temperature variability projections remain uncertain after constraining them\nto best performing SMILEs” by Suarez-Gutierrez & Maher.\nPlease find enclosed our response to the Reviewers' remarks on our manuscript. Following their\nsuggestions, we have revised our work by expanding the introduction and conclusions sections to clarify\nthe goals and implications of the study and to better place our contribution in the context of prior work\non performance- and independence-based constraining, highlighting the novelty of applying such\nmethods to temperature variability. We have also strengthened the evaluation framework by introducing\na new formally defined performance criteria based on whether observational rank histograms fall within\nperfect-model rank ranges, thereby providing a more rigorous and conceptually consistent evaluation\nbenchmark. We have also updated the main figures to better showcase how our variability estimates\ncompare to observations and to clarify the extent of the uncertainty reduction achieved by the\nperformance constraining, completely changing the concept of Figure 7 (previously Figure 6) to compare\nfuture projections to the projections by the ‘best performing model’. We thank the reviewers for these\nsuggestions and comments, which we believe have greatly improved our work and which we address\nindividually more in depth, along with all other remarks, below (in blue).\nReviewer #1 (Remarks to the Author):\nSummary and main comments\nThe authors evaluated historical simulations of seasonal (DJF and JJA) temperature variability in 11\nlarge ensembles compared to observations using rank-frequency analysis (known as rank histograms in\nthe ensemble forecasting literature). Models whose historical performance was judged adequate (by\nregion and season) were included in a performance-constrained ensemble. Projections based on the full\nand performance-constrained ensemble were compared.\n1. The introduction provides few (one?) references to application of performance-based\nconstraints/subsetting to projections. There is an extensive literature on this topic, and I think it would\nbe helpful to connect the current work to it. The other topics (importance of temperature variability and\nSMILES) are well introduced. A arbitrarily chosen example is\nBrunner, L., Pendergrass, A. G., Lehner, F., Merrifield, A. L., Lorenz, R., and Knutti, R.: Reduced\nglobal warming from CMIP6 projections when weighting models by performance and independence,\nEarth Syst. Dynam., 11, 995–1012, https://doi.org/10.5194/esd-11-995-2020, 2020.\nThis would also provide an opportunity to give the reader a clear idea of what has been done before\nin this direction and how this work improves upon previous efforts. It may be the case that model\nweighting/subsetting has not been applied to questions of variability changes, and if so, pointing this out\nwould highlight the value of the work. Or perhaps other approaches suffer from limitations that are\naddressed here. At present, I found the limitations of past work/goal of present work statement not very\nclear (line 43): \"However, it remains unclear whether climate models disagree in the direction and\nintensity of temperature variability changes because they do not adequately simulate temperature\nvariability over certain regions, or because of other, more systematic model differences in future\ntemperature variability behaviour. In this study, we address this question by constraining projections of\ntemperature variability in a multi-model super ensemble of single model initial-condition large\nensembles (SMILEs) to models that adequately represent observed regional variability under current\nclimate conditions.\"\nWe thank the reviewer for highlighting this. We have now expanded the introduction to expand on\nthe previous work on performance and independence weighting, adding a new paragraph on line 88. We\nhave also clarified that such performance subsampling approaches have not been applied to temperature\nvariability before and our study is the first to do so (Line 105).\nWe have also updated line 43, now line 69 for clarity as follows:\n“It remains however unclear whether climate models disagree on the direction and intensity of\ntemperature variability changes because they do not adequately simulate present-day temperature variability\ncorrectly, or whether they disagree due to intrinsic model differences in how the drivers of temperature\nvariability change in a future climate. Therefore, a possible hypothesis is that future temperature variability\ndepends on historical variability. If this were true, subsampling models based on their historical performance\nin capturing observed variability would yield reduced uncertainty in future projections. This means that by\nconstraining projections to models that best match observations and agree on their historical variability\nrepresentation, the range of future variability change projections across best-performing models would be\nreduced, and thus be less uncertain. In contrast, the alternative hypothesis is that future temperature\nvariability is intrinsic to how the Earth system evolves over different regions. If this were true, subsampling\nmodels based on their historical performance would not necessarily reduce future uncertainty, as long as those\nbest-performing models still exhibit diverging behaviours in their representation of future temperature\nvariability changes. In this study, we address these questions by, first evaluating in a multi-model super\nensemble of single model initial-condition large ensembles (SMILEs) which models can adequately represent\nobserved regional variability under current climate conditions. Second, we subsample the super ensemble to\nthose adequate models, per season and region, and compute constrained future projections for regional\ntemperature variability. Third, we compare the full ensemble with this subset of the ensemble to determine if\nthe uncertainty in our projections is lowered by constraining them to only models that adequately capture\nhistorical variability.”\n2. Rank-frequency analysis or more commonly rank histograms are a standard metric for assessing\nthe reliability of ensemble forecasts. Classical references are\nAnderson, J. L., 1996: A method for producing and evaluating probabilistic forecasts from ensemble\nmodel integrations. J. Climate,9, 1518–1530.\nHamill, T. M., and S. J. Colucci, 1997: Verification of Eta–RSM Short-Range Ensemble Forecasts. Mon.\nWea. Rev., 125, 1312–1327, https://doi.org/10.1175/1520-0493(1997)125<1312:VOERSR>2.0.CO;2.\nTalagrand, O., R. Vautard, and B. Strauss, 1997: Evaluation of probabilistic prediction systems.\nProceedings, ECMWF Workshop on Predictability, ECMWF, 1–25. [Available from ECMWF, Shinfield Park,\nReading, Berkshire RG2 9AX, United Kingdom.].\nI think that acknowledging the connection to forecast verification would provide additional support\nfor using this approach. More importantly, there are standard (perhaps many?) methods for assessing the\nflatness of the rank histograms. These methods take into account the number of verifying observations\nand ensemble size. The present work does not use any statistical methods to assess the flatness of the\nrank histogram and instead introduces admittedly ad hoc procedures to determine whether model\nperformance is adequate. Using rigorous methods to assess the rank histograms would better\nquantify uncertainty and potentially strengthen and simplify the results. An important question which could be\nanswered (currently unanswered) is whether *any* of the simulations have variability that is statistically\nconsistent with observations. A classic reference on assessment of rank histograms is\nJolliffe, I. T., and C. Primo, 2008: Evaluating Rank Histograms Using Decompositions of the\nChi-Square Test Statistic. Mon. Wea. Rev., 136, 2133–2139, https://doi.org/10.1175/2007MWR2219.1.\nThere may be better, newer methods, too.\nWe thank the reviewer for highlighting the connection to the forecast verification literature and for\nproviding relevant references, which have been added to the revised manuscript.\nRegarding the use of standard statistical methods to assess the flatness of rank histograms, or\nwhether any simulations exhibit variability that is statistically consistent with observations, we\nappreciate the reviewer’s suggestion. We agree with the reviewer that applying formal statistical tests\ncould, in principle, yield a more rigorous framework for evaluating histogram flatness. However, the\nrelatively short length of the observational record means that the influence of internal variability cannot\nbe robustly sampled in these timescales for the metrics considered here, therefore the observed\nvariability estimates derived from this record would not necessarily reflect the full range of real-world\nvariability, and may not be a suitable standard to evaluate the models. Therefore, tests assessing the\nstatistical similarity to observations or flatness of rank histograms directly may lead to misleading\nconclusions about model performance, in a similar way than assessing model performance based on only\none or a few realizations yields a less robust assessment than using a large ensemble of simulations.\nThis lack of robust variability sampling from the finite and relatively short observational record,\ncombined with the fact that we are comparing observations to uninitialized, free-running model\nsimulations means that, with the sample size available, we would for example not expect rank histograms\nto be perfectly flat, even for perfectly performing models. This is highlighted in our analysis by including\na perfect-model rank histogram range (e.g., in Fig. A below, now Fig. 9 in the revised manuscript). This\nrange is constructed by treating each ensemble member in each SMILE as if it were observations and\ncalculating the resulting spread of model rank histograms, which can be seen in the example figures\nbelow (gray dashed lines and crosses in left panels in example Fig. A). This approach provides a relatively\nwide distribution of rank histograms that reflect perfect-model behaviour, and are indeed not perfectly\nflat (such as almost twice as high perfect-model upper bound for 0 rank frequency than for n+1 rank\nfrequency for L7 region in Fig. A, for example). This can occur either systematically because the\nsimulated temperature distribution may be skewed or non-normal, or because the record length is too\nshort to sufficiently sample rank variability, or both.\nWe recognize however that, while an statistical assessment of observations rank histogram flatness\nmay not be the most adequate approach to assess model performance in this case, our evaluation\nframework could still be improved from our original approach to incorporate these principles. Therefore,\nwe have substantially expanded our evaluation framework to include a new formally defined evaluation\ncriteria: that the observations rank histogram fits within the perfect-model rank histogram range. The\nperfect-model rank range delineates a range of deviations from flatness that would be plausible for a\nmodel that perfectly captures observations for the metrics and record length considered here. Therefore,\nwe argue that as long as the observations rank histogram is within the perfect-model rank range, the\nmodel in question adequately simulates the variability in observations for that particular region, even if\nthe histogram of the observations itself is not flat. Considering the limitations of the observational\nrecord discussed above, we believe that using this perfect-model rank range provides a more appropriate\nbenchmark in this context and greatly improves our evaluation framework.\nWhile the perfect-model rank range was already included in our original analysis as a visual element,\nit is now also used as an additional evaluation criteria. This new evaluation criteria, now Criteria 1, is\nperformed over spatially averaged metrics, and is complemented by a slightly revised version of our\noriginal grid-cell level evaluation criteria, now Criteria 2, to account for biases at the grid-cell level that\nmay be smoothed or compensated in this spatial average. Therefore, the two evaluation criteria used in\nour framework are now as follows:\n1. Regional perfect-model rank range performance: observations rank histogram of the\nspatially averaged temperature for a given region must lie within the perfect-model rank range\nacross all ranks.\nThe number of total ranks depends on the ensemble size, ranging from 0 to n, with n being the\nnumber of ensemble members. Therefore, to minimize the effect of comparing ensembles of\ndifferent sizes and potentially spurious effects resulting from the insufficient rank variability\nsampling due to the finite sample size, both model and observations rank frequencies are shown\nas the running mean across individual rank frequencies over an n/5 rank window (for n=50\nmembers, 10-rank running average; dashed gray and solid color lines in rank histograms in Fig.\nA) for ranks 1 to n -1, and as the actual rank frequency for ranks 0 and n (gray and color crosses\nin Fig. A). We allow observations rank running-mean to exhibit a maximum deviation outside of\nthis perfect-model range of 10% (e.g., for an upper limit in the smoothed perfect-model rank\nrange of 3% for a particular rank window, a maximum observations rank frequency of 3.3% would\nbe accepted for adequate performance). The fulfilment of this criteria for all rank windows is\nmarked by a green star in the histogram panels (e.g., Fig. A, Fig.1 in the revised manuscript),\nwhile inadequate performance is marked by a red cross.\n2. Threshold-based grid-cell level performance: at least 50% of the region's grid-cells must be\nunbiased, meaning, at the grid-cell level, a.) observations do not cluster excessively within the\ncentral percentiles of the ensemble (observations occur with more than 80% frequency within\nthe central 75th percentile ensemble bounds, indicative of variability overestimation bias) and b.)\nfall outside the ensemble range too frequently (observations occur with more than 8% frequency\noutside of the ensemble spread, indicative of variability underestimation bias).\nCriteria 2. has been updated from our original framework to account for 50% instead of 60% of\nunbiased grid cells in a region. This slightly relaxed threshold ensures the condition is not overly\nrestrictive while still maintaining a robust standard for model adequacy. Only when both criteria\nare fulfilled, we consider the model in question offers an adequate performance and is selected to\nbe part of the constrained ensemble for the particular region and season.\nThe addition of Criteria 1. makes our evaluation framework more restrictive than our original\napproach, with a 22% reduction in total number of regions being adequately captured (calculated\nmaintaining the original threshold in Criteria 2 of 60% of unbiased grid-cells). Relaxing this threshold to\n50% unbiased grid-cells yields approximately the same number of total regions being adequately\ncaptured as in our original framework (i.e., 354 originally, 353 now). However, even though the total\nnumber of adequately captured regions remains somewhat stable, this additional criteria affects the\nnumber of adequate regions per model, in some cases substantially (e.g., drop from 37 to 28 adequate\nregions for ACCESS, increase from 21 to 33 regions for MPI-GE5). The full account of the updated\nevaluation results can be found summarized in Table 1 for detrended data, and additionally in Table 2 for\nnon-detrended data in the revised manuscript, as well as in the revised supplementary information files\nfor all SMILEs.\nThis revised evaluation framework improves on our original approach in two key ways. First, by\nexplicitly assessing the agreement between the observed rank histogram and the perfect-model rank\nrange, incorporating the key aspects of the probabilistic verification framework suggested by the reviewer\nin a suitable way, accounting for sample limitations and the potential effect of limited internal variability\nsampling in rank histogram flatness. Second, the revised framework improves our assessment by\nensuring that the full rank distribution and the effect of ensemble size is considered in our evaluation\n(Criteria 1.), while including the grid-cell level performance in a computationally efficient way (Criteria\n2.). Applying Criteria 1. at the grid-cell level (calculating the perfect-model rank range assessment for\neach member of each SMILE for each grid-cell) would potentially be the more rigorous alternative to\nCriteria 2. However, we believe this would entail excessive computational costs and limit the applicability\nof our framework. Therefore, together these two complementary criteria provide a more comprehensive\nand conceptually consistent assessment of model performance.\nWe have revised the manuscript to reflect this expansion of the evaluation framework to explicitly\nincorporate the agreement between the observed rank histogram and the perfect-model rank range as a\nperformance metric and the formally defined evaluation criteria, both for the introduction (lines\n126-132), results (lines 141-219) and methods sections (lines 514-625).\nFigure A: Example time series and rank histogram evaluation for MIROC6 against GISTEMPv4\nobservations. Time series and rank histograms show the evaluation of spatially averaged temperatures for\nselected regions (Criteria 1). Time series show the ensemble maximum and minimum (color lines) and central\n75th percentile ensemble spread (shading) against observations (dots). Rank histograms show observations\nrank frequency aggregated over a 3-rank window (bars), the running mean rank frequency over a centered n/5\nrank window (lines; for 1 to n-1 ranks) and the absolute frequencies of rank 0 and n (crosses), with n the\nnumber of ensemble members, for observations (color) and perfect model rank range (gray). If observations\nrank frequency is within perfect-model range for all rank windows, Criteria 1 is met and this is highlighted by a\ngreen star at the top right; if not, by a red cross. Percentages at the top left show the frequency of regionally\naveraged observations occurring above (red) or below (blue) ensemble limits, or clustering within the central\n75th percentile range (gray), analogous to the grid-cell evaluation in Criteria 2. Bottom maps show the\ngrid-cell evaluation of temperature variability globally (Criteria 2). Gray hatching (variability overestimation\nbias) shows observations clustering within the central 75th percentile with more than 80% frequency. Red and\nblue shading (variability underestimation bias) shows observations falling beyond the ensemble maximum or\nminimum, respectively, with more than 8% frequency. Dotted areas represent ocean grid cells or where\nobservations are missing and therefore excluded. Left column shows results for non-detrended data, right\ncolumn shows analogous results for detrended data. Adapted from Fig. 9 in the revised manuscript. See\nMethods for further details and Supporting Information for results for all 11 SMILEs.\n3. As noted in point 1, I did not find it completely clear what hypothesis is being tested in the current\nwork or exactly what the conclusion is. Presumably, the overall goal is more accurate projections. This, of\ncourse, is impossible to assess since we don't know the future. Moreover, there is the familiar issue that\npast performance does not guarantee future performance, a point which I did not see mentioned. The\npresent work emphasizes reductions (or lack of) in projection uncertainty in the\nperformance-constrained ensemble. However, as noted in the text, these reduction might simply be due\nto removing models. Perhaps this hypothesis could be tested. That is, are the changes in uncertainty\ngreater than would be expected if models were removed at random instead of based on historical\nperformance?\nWe thank the reviewer for bringing these points to our attention. Indeed, the goal of our study is to\nreduce uncertainty in projections of temperature variability and its change. To clarify this, we now clearly\nstate the specific aims of our work (lines 132-138 of the introduction), which are threefold :\n- (i) Evaluate how well each SMILE captures historical temperature variability in both summer and\nwinter seasons over individual regions.\n- (ii) Assess model agreement in the projections of temperature variability across all SMILEs for\nboth the sign of the change and the spread of the ensemble\n- (iii) Constrain the projections using the SMILEs that perform well over the historical period, and\nassess whether this provides an increase in model agreement for future projections.\nWe have also restructured the introduction and discussion sections to present the goal of our work more\nclearly (e.g., lines 69-87 and 123-131).\nFurthermore, we agree with the reviewer that adequate past performance does indeed not guarantee\nfuture performance, although studies show that in some cases it can, such as in Brunner et al 2020. We\nhave expanded the introduction to clearly reflect these issues in lines 88 and 92. We have also expanded\nour analysis and the accompanying discussion of the results where relevant to more clearly address to\nwhat extent this uncertainty reduction is achieved over the different metrics considered (e.g., paragraphs\non lines 297 and 365)\nRegarding the reviewer’s comment on whether the model spread and thus uncertainty in the\nprojections would decrease simply by the fact of removing models from the sample randomly, we thank\nthe reviewer for bringing this up. We have tested the hypothesis of randomly removing models. To do this\nwe repeated the analysis in Figures 2 and 3 by creating thousand 6-model constrained ensembles\nrandomly rather than by using our constraining method. We find that a decrease in the model spread is\nnot guaranteed by design in this case. A decrease in the spread occurs only if the models at the ends of\nthe spread are randomly removed, but not otherwise.\nWe note that the errorbars on Figures 2-3 show actually the end members of the model spread. We\nhave one estimate for each model of variability (STD over the ensemble) as such these errorbars only\ndecrease if the constraint removes the end member. This was not clear in the original text and this has\nbeen clarified in the figure caption:\n“Errorbars show the full model spread (i.e. the end members of each ensemble), with the fatter errorbars\nhighlighting the 25th and 75th percentiles.\nLastly, to further clarify to what extent our approach reduces uncertainty in model projections, we\nhave also expanded the results and discussion sections (e.g., paragraphs on lines 353, 365, 443) and\nadded a new section, Conclusions and Implications (lines 458-484), that clearly states the conclusions\nemerging from our work, which are as follows:\n“(i) We provide a comprehensive evaluation of the state-of-the-art SMILEs ability to represent the\nhistorical summer and winter temperature variability in observations. We identify CESM-LE and CESM2-LE as\nthe SMILEs that provide the best representation of isolated temperature variability as well as of both\ntemperature variability and forced change, with GFDL-SPEAR-MED and MPI-GE5 as close third in each\ncategory respectively. This multi-model evaluation across all available CMIP and CMIP6 SMILEs provides a\nbasis for model selection for the assessment of temperature variability and extremes, which cannot be correctly\nsimulated if the underlying variability is incorrect.\n(ii) Our evaluation also shows that some regions are systematically not well represented such as the\nSouthern Ocean (O9), the Indian Peninsula (L20), and Northern Australia (L22), as well as the Amazon basin\n(L6), Northern (L12) and Eastern Africa (L14), and Southern Australia (L23), all specifically in the local\nsummer season. We also find that certain areas of the world are systematically well captured by most models,\nespecially in the local winter season, namely North America (L1-L4), Southern South America (L8), Europe\nand Northern Africa (L9-L11) and North and Central Asia (L16-L19). Finally, we conclude that local winter\nseasons are generally simulated better by most models more often than local summer seasons.\n(iii) Similar to the IPCC, we find low model agreement on the sign of the change over large parts of the\nEarth’s surface and large model disagreement on the magnitude of the projected change under future warming,\nparticularly over the land surface and the tropical Pacific Ocean. This implies that we cannot afford to move\naway from multi-model ensembles that sufficiently capture the uncertainty in our future projections.\n(iv) The constrained ensemble decreases model spread over some regions and gives a lower range of\nprojected futures, providing a smaller range of potential future projections and greater model agreement,\nparticularly in South America, India, Australia & South-East Asia, the west Pacific Ocean and Southern\nOcean in DJF and north South America, Africa, parts of Australia and the Southern Ocean in JJA. However, the\nconstraint does not substantially increase model agreement on the sign of the projected change.”\nSince we don't have observations of the future, a standard method of evaluating model\nweighting/subsetting schemes is the so-called perfect model approach where one model at a time is\ntreated as \"observations.\" By doing so, it can be checked whether a subsetting strategy based on historical\nperformance actually gives more accurate projections. This method is widely used, including in the\nBrunner reference above and this one\nKnutti, R., J. Sedláček, B. M. Sanderson, R. Lorenz, E. M. Fischer, and V. Eyring (2017), A climate\nmodel projection weighting scheme accounting for performance and interdependence, Geophys. Res.\nLett., 44, 1909–1918, doi:10.1002/2016GL072012.\nNote that both works also consider model independence which might not be an issue here but could\nbe considered and mentioned.\nWe appreciate the reviewers idea to use a perfect model approach considering each individual\nensemble member as if it were observations. We agree that this would be a valuable exercise and well in\nline with the perfect model rank range analysis we introduce in the evaluation section of this manuscript.\nA test as proposed would answer the question of whether future variability depends on historical\nvariability, specifically for each model across those considered, which expands on our goal of reducing\nuncertainty across temperature variability projections. We agree that this would be an interesting\nexercise and an additional layer of rigour to the analysis proposed here. While we recognize the value of\nperforming this additional check, carrying out this additional perfect-model assessment treating each\nmember of each one of all 11 SMILEs as observations and redoing the evaluation and constrained\nvariability projections analysis excluding the respective SMILE would increase the length and complexity\nof manuscript substantially. To keep the analysis presented here as concise yet robust as possible, we\nhave included an adapted version of this perfect-model projections test suggested by the reviewer based\non CESM2-LE alone. Removing CESM2-LE, one of the models closest to observations particularly over\nthe most poorly represented regions, we recalculate the full and constrained ensemble for future\nprojections and assess how closely it resembles the CESM2-LE future projections focusing on key, poorly\ncaptured areas, and distinguishing between the cases where CESM2-LE adequately captures observations\nfrom those where it does not.\nWe explore this comparison of the full vs. constrained projections against the projections of the\n‘good performing model’ CESM2 below (Figs. B and C, replicating Figs 2 and 3 in the revised manuscript,\nrespectively, but now excluding CESM2). These results highlight how the new constrained (excluding\nCESM2) and the full ensemble (also excluding CESM2) compare with CESM2 (blue stars for regions and\nsessions when CESM2 is a good performer, red when it is not). These Figures and related discussion have\nbeen included to the paper (Figs. 10 and 11, lines 664-686)\nOur main conclusion is that sometimes the constrained ensemble of adequately performing models is\ncloser to CESM2 when it is a good performer, and sometimes it isn’t. Likely, this is because in some cases\nfuture variability is more closely linked to past variability performance, versus in other cases climatic and\nenvironmental changes intrinsic to each model may affect the temperature variability distribution\ndifferently across different models, even for those that show a common and adequate historical\nrepresentation. Disentangling the drivers of these diverging behaviours for future variability change\nacross historically adequate performing models would be an interesting potential follow up work. This\nwould entail a careful consideration of the relevant processes involved in temperature variability change\nacross different regions and rigorous process-based evaluation. However, albeit interesting, we believe\nthis is out of scope of this manuscript at this stage.\nFigure B. DJF Comparison of ‘new’ constrained ensemble without CESM2 against CESM2 projections\n(stars). Blue stars mark regions for which CESM2 offers an adequate representation of historical variability,\nred stars represent where it does not. Same as Fig. 10 in the revised manuscript.\nFigure C. JJA Comparison of ‘new’ constrained ensemble without CESM2 against CESM2 projections\n(stars). Blue stars mark regions for which CESM2 offers an adequate representation of historical variability,\nred stars represent where it does not. Same as Fig. 11 in the revised manuscript.\nFurthermore, we have also completely revised Figure 6 (now Figure 7) to expand this new assessment\nover key, poorly modeled areas, and expanded the related discussion on line 353 as follows:\n“While we cannot assess whether the future projections are more realistic in the constrained ensemble\ncompared to the full ensemble, as we do not have observations of the future, we can compare the projections\nwith those from the most realistic model over these poorly modelled regions in the historical period. CESM2 is\none of the two models that best represent Australia & South-East Asia (L21-23), South America (L6-8), and\nAfrica (L12-16; the other being CESM-LE), with 15 \"good performances\" out of 22 across both regions and\nseasons. By using this model as a proxy for possible future observations, we can compare the differences at 3˚C\nof warming between the full and constrained ensemble, the constrained ensemble (excluding CESM2), and\nCESM2 (Fig. 7, see Figs. 10 and 11 for a spatially aggregated comparison across all regions). We find that,\nexcept for South America (where CESM2-LE fails to capture historical variability according to our evaluation\nfor L6 and L7) and Africa in DJF the constrained estimate is closer to CESM2 (our \"good model\") than the full\nensemble, tentatively suggesting that our future projections might be more realistic in the constrained\nensemble compared to the full ensemble. ”\n4. Figures 2–5 show constrained vs. full ensemble results and differences without statistical\nsignificance statements. I find it hard to interpret whether the results are \"significant\" or just \"different.\" I\nthink the sample size is fairly large (11 years x ensemble size) but variance tends to be more subject to\nsampling variability than means. I'm not sure that the ensemble-based error bars correctly capture the\nuncertainty since sampling variability goes up with reduced ensemble size (right?) and the error bars\nwould tend to shrink?\nRegarding the remark on the statistical significance of these results, we thank the reviewer for the\nsuggestion, but we are not sure it is feasible to perform such tests considering our sample. For typically\nused tests, such as a t-test or a f-test, they are not appropriate to use on these data, as the full and\nconstrained ensembles are not independent samples, and would violate the assumptions of both\nstatistical tests. We have not found a test that would appropriately assess this statistical significance\nwithout violating the core assumptions of said test. Therefore, if the reviewer has any specific\nsuggestions as to how to appropriately calculate significance for this study we are open to including\nthem. To overcome these issues and the lack of suitable statistical significance testing, we include the\nbottom two panels in Figure 4 and 5s. The second last row shows the percentage of models that agree on\nthe sign of the change and the last row shows the standard deviation (STD) across models, effectively\nillustrating the spread of the model projections. We note that we changed the difference of this last row\nto be a percentage change, which is easier to interpret visually. We propose these two bottom rows of\nfigures to assess the significance of our results as an effective alternative to potentially flawed statistical\ntests that may lead to misleading conclusions on this significance of our results. While one might assume\nthat the STD would decrease as we remove models, we have tested this assumption and find that the\nmagnitude of the STD is dependent on which of the models are removed and it can increase or decrease.\nWe had added discussion on this point on line 378.\n“Typically, regions where model agreement on the sign of the change is substantially improved are regions\nwhere the constrained ensemble includes many fewer models than the full ensemble. Note, however, that this\nis not a predetermined result. We tested the change in the standard deviation (understood as the uncertainty\nin the magnitude of the change) in a constrained ensemble of either 4 or 6 randomly selected models by\nsampling 1000 times, and find the standard deviation decreased approximately 2/3 of the time and\nincreased 1/3 of the time. ”\nWe have also tested the hypothesis of randomly removing models just decreasing the model spread\nby design and find this is not predetermined. Text has been added on line 298:\n“The errorbars themselves show the multi-model ensemble spread, and are thus determined by the end\nmembers of the multi-model ensemble. The interquartile range of the multi-model ensemble is shown in the\nwider bars. By design, the errorbars only decrease when the end members of the ensemble are removed.\nTherefore, only if our evaluation proves the more extreme model members to not perform adequately, and thus\nremoves them from the constrained ensemble, does the uncertainty in the projections decrease. This becomes\nmore likely to happen by chance with the removal of more models, but is not a predetermined nor expected\noutcome.“\nOther comments.\nI would find the figures easier to read if the captions and plots used the term standard deviation\ninstead of temperature variability. Figures 2 and 3 do say \"temperature variability defined as the standard\ndeviation\" but 4 and 5 don't. Seems like standard deviation would be more precise and concise.\nWe have added this definition to the captions of Figures 4 and 5. We wish to keep temperature\nvariability defined as the standard deviation, as it is important that the reader knows that what we are\ndiscussing is this variability. Using only the standard division has the possibility of the link between\nstandard deviation and temperature variability being missed.\nLine 43. \"However, it remains unclear whether climate models disagree in the direction and intensity\nof temperature variability changes because they do not adequately simulate temperature variability over\ncertain regions, or because of other, more systematic model differences in future temperature variability\nbehaviour. In this study, we address this question by constraining projections of temperature variability\nin a multi-model super ensemble of single model initial-condition large ensembles (SMILEs) to models\nthat adequately represent observed regional variability under current climate conditions.\" This seems\nimportant but was not very clear to me how the question was being addressed. I think the idea (after\nreading the paper) is that if the performance-based ensemble doesn't have smaller uncertainty, then\nother differences (i.e., ones that do not result in variability biases during the historical period) are\npresent.\nYes, the reviewer's interpretation is correct. We have rewritten this paragraph to more clearly reflect\nthis reasoning (starting in Line 69), which now reads as:\n“It remains unclear whether climate models disagree on the direction and intensity of temperature\nvariability changes because they do not adequately simulate present-day temperature variability\ncorrectly, or whether they disagree due to intrinsic model differences in how the drivers of temperature\nvariability change in a future climate. Therefore, a possible hypothesis is that future temperature\nvariability depends on historical variability. If this were true, subsampling models based on their\nhistorical performance in capturing observed variability would yield reduced uncertainty in future\nprojections. This means, by constraining projections to models that best match observations and agree\non their historical variability representation, the range of future variability change projections across\nbest-performing models would be reduced, and thus be less uncertain. In contrast, the alternative\nhypothesis is that future temperature variability is intrinsic to how the Earth system evolves over\ndifferent regions. If this were true, subsampling models based on their historical performance would not\nnecessarily reduce future uncertainty, as long as those best-performing models still exhibit diverging\nbehaviours in their representation of future temperature variability changes. ”\nWe agree with the reviewer’s interpretation that, in cases where the performance-constrained\nensemble does not have reduced uncertainty, future differences could be driven by different evolutions of\nthe Earth system in these regions intrinsic to each model; while this intrinsic model behaviour may not\naffect historical performance as discussed above. As an hypothetical example, over areas such as the\nAmazon basin this could be caused by changes to the Amazon ecosystem not strictly driven by local\ntemperatures (e.g., from changes in precipitation patterns to drastic forest composition changes or forest\ndie-back), that could cause this ecosystem and its associated temperature variability to evolve in\ndifferent ways in different models, even though today’s climate may be similar and well captured in the\nmodels. Disentangling the drivers of diverging temperature variability behaviour across different,\npotentially adequately performing models over different regions would be an interesting yet complex\nexercise. This assessment would need to be carefully performed considering process-based evaluation\napproaches across relevant metrics guided by regional expert knowledge of both climatic and\nenvironmental change over key hotspot regions of variability change uncertainty. We believe this would\nmake for an interesting and extremely relevant follow-up paper, or several papers, but is at this stage\nbeyond the scope of our current manuscript.\nFigure 1. What is the purpose of the time series shown? Are they mentioned in the text? I was unable\nto make out the very light red and blue shading. The caption does not say what say what the Above,\nBelow, and Center values are. I think the gray lines are a 90% confidence interval but I could not follow\nthe description (below) in the caption. In particular, if the lines illustrate the slope, the slope should be\nzero but I don't see zero being plotted. Perhaps say what the lines represent and give the details in the\nmethods section instead of the caption. I didn't see any more details in the current methods section.\nFig. 1 caption: \"Lines illustrate the rank histogram’s slope, as the mean rank frequency over a centered 6-bin window for\nobservations (coloured lines), and the 5-95th percentile perfect model range (the slopes of all ensemble members treated as\nif they were observations; gray dashed lines).\"\n\"Crosses represent the frequency of minimum (0) and maximum (number of members) ranks for\nobservations (colors), and for the 5-95th percentile perfect model range (gray)\" Isn't the frequency of the\nobservation rank already given by the bars?\nWe thank the reviewer for highlighting these unclear points. The time series shown reflect the base\nfor the spatially aggregated rank-frequency evaluation, now Criteria 1. We believe including the time\nseries alongside the rank histograms is useful to help readers not familiar with rank histograms to\nunderstand how they are constructed and what different shapes translate to in terms of potential biases,\ntherefore we choose to keep both visual elements in the main manuscript. We have however expanded\nthe text clearly stating the meaning and relevance of these figures in the main results section (lines XX).\nFurthermore we have revised the format of the map figures to make the hatching more clearly readable.\nWe have also extended the figure captions for this type of figure to clearly describe all visual\nelements through the paper (i.e., Figs. 1, 9 and Supporting Information). To clarify the elements brought\nup in the remark above specifically:\n- The ‘Above, Below, Center’ frequencies mark the frequencies with which spatially averaged\nobservations fall within the rank windows assessed in the grid-cell based evaluation, now Criteria\n2 (namely above the ensemble maxima or rank n, with n being the number of ensemble\nmembers; below the ensemble minimum or rank 0; and within the central rank window marked\nby the central 75th percentile ensemble bounds).\n- The lines in the rank histograms mark the running mean of the rank frequencies averaged over a\nn/5 rank window. The color lines illustrate this running mean for observations (for which rank\nfrequencies are also shown in the bars accumulated over a 3-rank bin), while the gray lines mark\nthe perfect-model rank range in this metric as described above.\n- The crosses mark the absolute rank frequencies for rank 0 and rank n for observations (in color)\nand for the perfect-model range (in gray). We choose to highlight this individual frequencies for\nminimum and maximum rank since they reflect the variability underestimation bias highlighted\nin Criteria 2 (i.e., ‘Above, Below’ frequencies in rank histograms, Red and Blue hatching in maps).\nLastly, since we have incorporated the perfect-model rank range into our valuation framework explicitly,\nfor clarity we have modified the visual elements to illustrate the full range used and not the 5-95th\npercentile range as in the original manuscript.\nLine 113-116. \"we also include time series and rank frequency histograms for all regions and models\nwhere model performance based on spatially aggregated data can be assessed. Furthermore, time series\nfor this spatially aggregated evaluation also allow changes in potential biases over different periods to be\nassessed.\" I don't understand what it means to include time series or what the time series in question are.\nIf this is a technical detail, perhaps keep move it to the methods section where it can explained more\ncompletely. Also repeated later.\nWe thank the reviewer for highlighting this, which has now been clarified. The time series in question\nnow reflect Criteria 1 of our evaluation and are introduced with more detail in the results and methods\nsection, as well as in the supporting information where the time series and rank histograms that serve as\nbased for the spatially aggregate evaluation in Criteria 1, and the maps for the grid-cell evaluation that\nserves as based for Criteria 2 can be found for all 11 SMILEs across all regions and seasons considered.\nTable 1 caption \"total amount\" -> \"number\"\nThis has been changed as suggested.\nLine 132 \"struggling\" -> \"struggle\"\nThis has been changed as suggested.\nFig. 2. Starting at zero seems to waste considerable space and reduce readability. There is no\nobjective measure here of whether projection values are different between the constrained and full\nensembles and also whether uncertainty is reduced. This issue is especially unclear when only a few\nmodels are retained.\nThanks for the suggestion, we have changed the Figure as suggested to be centred around the full\nensemble at 1˚C of warming. We have also updated the text so the errorbars are more easily understood,\nsee caption:\n“Error bars show the full model spread (i.e. the end members of each ensemble), with the fatter error bars\nhighlighting the 25th and 75th percentiles.”\n\"Horizontal lines are plotted at both 1 degree (solid line) and 4 degrees (dashed line) of warming.\"\nWhy? Don't the bars show the same information? Perhaps it would help to put the constrained and\nunconstrained values for the same warming level beside each other to aid comparison.\nThanks for the suggestion, we have removed the horizontal lines and put the constrained and\nunconstrained next to each other to aid comparison.\nLine 206 \"constraining the projections to best-performing models yields a decrease in model spread\"\nWould selecting any subset of models (including randomly) yield a decrease in spread?\nAs discussed above, it would not. We have tested this and have elaborated on this in the paper:\nLine 298: “The errorbars themselves show the multi-model ensemble spread, and are thus determined by\nthe end members of the multi-model ensemble. The interquartile range of the multi-model ensemble is shown\nin the wider bars. By design, the errorbars only decrease when the end members of the ensemble are removed.\nTherefore, only if our evaluation proves the more extreme model members to not perform adequately, and thus\nremoves them from the constrained ensemble, does the uncertainty in the projections decrease. This becomes\nmore likely to happen by chance with the removal of more models, but is not a predetermined nor expected\noutcome.“\nLine 378: “Typically, regions where model agreement on the sign of the change is substantially improved\nare regions where the constrained ensemble includes many fewer models than the full ensemble. Note,\nhowever, that this is not a predetermined result. We tested the change in the standard deviation (understood\nas the uncertainty in the magnitude of the change) in a constrained ensemble of either 4 or 6 randomly selected\nmodels by sampling 1000 times, and find the standard deviation decreased approximately 2/3 of the time and\nincreased 1/3 of the time. ”\nLine 207. \"Less\" -> fewer\nThis has been changed as suggested.\nLine 215. \"These results indicate the performance constraining can indeed decrease the uncertainty\nin future variability projections\" It seems likely that many criteria for removing models reduce spread.\nAs discussed above, this is generally not the case - see added text and response above to the\ncomment on line 206.\nLine 233. \"temperature variability is significantly overestimated at the 1oC warming level.\" Compared\nto observations? Or something else?\nWe thank the reviewer for bringing this unclear statement to our attention. We have now added a\nnew Figure 6 that compares the model ensembles with observations. We have also included the following\ntext to explain this on line 323:\n“This highlights the improved representation of the present day variability, as the lower variability in the\nconstrained ensemble over land is in better agreement with observations, except for again in the northern\nhemisphere extratropics in DJF (Fig. 6). We note that the variability over the ocean shows little improvement in\nthe constrained ensemble, and tends to be overestimated in DJF and underestimated in JJA. ”\nWe have also updated line 233 (now line 339) to add “compared to observations”.\nFig. 4. Is it possible to indicate statistical significance of the differences?\nAs highlighted in the major comments section above, this is not possible due to the lack of\nindependence between the full and constrained ensemble samples that violate the assumptions in\nstandard statistical tests.\nFig. 4 caption and elsewhere. Is that is a degree symbol?\nWe thank the reviewer for catching this typo, which seems to have emerged from an incorrect\nexpression in LaTeX. We have modified this to show the correct degree symbol through the manuscript.\nThe difference of STD in Fig. 4l looks small but I'm not sure what is considered small.\nWe have changed this to be a percentage of the STD itself in the full ensemble for an easier visual\nassessment of how big the differences really are.\nLine 243. \"This implies that the absolute value of the variability at 3oC is overestimated in the full\nensemble, but the change in variability is underestimated, meaning that the interpretation of results\ncould be biased in either direction depending on which metric is used.\" I find this hard to follow. What\ndoes \"the absolute value of the variability\" mean? STD is always positive? Which \"interpretation of the\nresults\" is being referred to here?\nWe have rephrased this to say ‘This implies that the absolute value of the variability at 3oC is\noverestimated in the full ensemble, but the change in variability may be underestimated, meaning that\nthe interpretation of results could be biased in either direction depending on whether one computes the\nactual value of future variability or the value of the future change in variability.”\nLine 249 \"it's\" -> its\nThis has been changed as suggested.\nFig. 5. Possible to add or comment on statistical significance? Especially for panel l.\nAs highlighted in the major comments section, this is not possible due to the lack of independence\nbetween the full and constrained ensemble that violate the assumptions in standard statistical tests.\nLine 257. \"Typically regions where the constrained ensemble includes many less models than the full\nensemble are regions where model agreement is substantially changed.\" Is this obvious? If so, does it\nprovide a meaningful conclusion? Also less -> fewer.\nThis is not obvious, but we have now elaborated on the regions, which reflect regions already\ndiscussed as those that include many fewer models in the constrained ensemble. Less has been changed\nto fewer.\nLine 260. \"the spread of projections and hence the magnitude of the change can be limited using this\nmethod.\" Is this the goal? Is it a meaningful metric of success? This is a place where the using one model\nas observations might permit stronger results.\nWe thank the reviewer for highlighting this statement, which was somewhat imprecise. Reducing the\nmagnitude of the variability change is not the goal of our work. Instead, our goal is to reduce uncertainty\nacross the projections of this magnitude of the change across different models by using performance\nconstraining. We have modified the wording in text to reflect this more clearly (line 385).\nRegarding the suggestion of using one model as a proxy for future observations in a perfect-model\napproach, we believe a test as proposed would answer the question of whether future variability depends\non historical variability, specifically for each model across those considered. We agree that this would be\nan interesting exercise and an additional layer of rigour to the analysis proposed here. However, to\nanswer the question of whether future variability depends on historical variability across different\nmodels we believe is at this stage beyond the scope of our current manuscript. To keep the analysis\npresented here as concise yet robust as possible, we have included an adapted version of this\nperfect-model projections test suggested by the reviewer based on CESM2-LE alone, and find that indeed\nfor most regions the constrained ensemble is closer to the chosen “good model”, as discussed above in\nmore detail.\nIs Fig. 6 mentioned in the text? Does Fig. 6 provide anything that's not in Figs. 4 and 5? Is a t-test the\nright test for a difference in variances? Or F-test? Though this is a difference in variances. Again, isn't\ncomparing the full ensemble to one with fewer models a bit of a straw man? Also I cannot visually tell the\ndifference between small values (white) and insignificant values (also white?). What does \"general\nregion\" mean in the caption? Is there some smoothing?\nThanks for the comment - we have revisited and changed Figure 6 and this does not apply any more.\nLine 267. \"This evaluation enables end-users of these ensembles to select the best fitting model for\ntheir region and time of interest\" Is there evidence that such a strategy leads to more accurate\nprojections in the perfect model (each model in turn as observations) setting?\nThere is certainly evidence that this could allow studies in the historical period to be more accurate -\neven if evidence for future variability projections is more limited, as discussed above. Users could for\nexample use a large ensemble to get extreme statistics in a model with realistic temperature variability\nfor the historical period. This may not be possible in observations due to the low finite sample size.\nLine 274. \"they may, however, underestimate the projected temperature variability change\" or they\nmay not? Right?\nCorrect, this is why we used the word ‘may’ not do.\nLine 277. \"providing an improved estimate\" In what sense is the estimate improved?\nIn this stance, we are referring to a potential reduction in uncertainty in future projections that\nresults from constraining these projections to those models that exhibit adequate historical performance.\nWe agree with the reviewer that the original wording was somewhat imprecise, and have updated this\nline to ‘providing a performance-constrained estimate of both the projected change and the model\nuncertainty around it’ for rigour.\nLine 281. \"In these regions we find that temperature variability itself is typically overestimated in\npresent day climate; while the constrained ensemble provides a larger projected change in variability.\" It\nwould be helpful to provide the reader with pointer to where these results appear. Or perhaps some\nsummary graphic showing this relation?\nWe have added a reference to the figures that show these results in this sentence.\nLine 310. \"Our attempt to constrain projections of future temperature variability change is an\nexample of change in the climate system that cannot yet be foreseen with the best available knowledge.\"\nWhat does this mean?\nWe are referring here to our findings indicating that, while constraining projections with historical\nmodel performance can reduce model uncertainties in some cases, these uncertainties are not completely\nerased. Even when successfully identifying several models that perform adequately in capturing today's\nclimate in certain regions, model disagreement in temperature variability projections remains, in many\ncases, large. And thus, model differences across temperature variability change under future climatic\nconditions remain, over large areas, unreconciled. As we discuss above, remaining protection uncertainty\nacross constrained ensembles of models that capture historical variability correctly could be driven by\ndifferent climatic and environmental changes across different models. And these differences in future\nmodel behaviour in many cases (as the above mentioned example for the Amazon forest) cannot be\nreconciled using best available knowledge today, including evaluations such as those presented here. We\nhave expanded this paragraph to reflect these aspects more clearly.\nLine 365. \"date\" -> data\nThis has been changed as suggested.\nLine 370. \"Lastly, to allow a more finely resolved temporal analysis and to increase sample size we\nperform this evaluation on monthly mean temperature anomalies instead of seasonal averages.\" What\nevaluation? The rank-frequency evaluation or another diagnostic?\nThanks for bringing this to our attention. This statement refers only to the rank frequency\nevaluation. We have changed the line to read “Lastly, to allow a more finely resolved temporal analysis\nand to increase sample size we perform this rank-frequency evaluation on monthly mean temperature\nanomalies instead of seasonal averages.” for clarity.\nLine 384 \"To illustrate how internal variability may affect rank frequencies\" isn't the rank histogram\nitself assessing variability? Or do you mean finite sample size?\nYes, we are referring to the effect of insufficient variability sampling in the finite sample size\nprovided by the relatively short observational record. This effect may cause the rank variability to be\nundersampled as well, resulting in a non-flat rank histogram, either for adequately performing models or\neven in a perfect-model framework. We have expanded on these issues on point 2. of the response to the\nreviewers remark above, and have clarified this point in the revised manuscript where relevant. This\nparticular line has now been removed from the revised manuscript, but the wording has been updated to\nreflect this suggestion throughout the manuscript.\nLine 397 \"To quantify when variability biases are present, we select the following thresholds.\" These\nseem arbitrary. It would be sensible to include some sort of significance testing of rank histograms.\nWe thank the reviewer again for this suggestion. We have now expanded our evaluation framework to\ninclude an additional criteria reflecting the whole spectrum of ranks based on the perfect-model rank\nhistogram range, as discussed in the response to point 2. above. We agree that these thresholds and also\nthe percentile ranges are somewhat arbitrary, and that assessing rank frequencies across all ranks against\nthe perfect-model range is the superior evaluation benchmark. However, the complexity and\ncomputational costs of performing such an evaluation at the grid-cell level would be substantial, and\nwould complicate the applicability and reproducibility of this evaluation framework by potential\nend-users. For these reasons we have decided to include the superior evaluation of the full rank\nhistogram against the perfect model range for spatially averaged metrics as a primary criteria, and have\nmaintained a revised version of this threshold-based criteria as a secondary condition in our evaluation,\nto ensure that a grid-cell performance assessment that is computationally sensible is also included in our\nframework. Also, although we agree that the thresholds chosen are somewhat arbitrary, they agree well\nwith the ranges found in the perfect-model rank range evaluation for the assessed rank windows.\nLine 408. \"Note that the variability underestimation threshold of 8% is chosen to be rather\nconservative to account for the different ensemble size\" Wouldn't it be more objective to use a standard\nsignificance test which accounts for ensemble size? Perhaps these are all points where a method that\ntakes into account sample and ensemble size would be beneficial.\nWe again agree with the reviewer and have expanded our evaluation framework to reflect this, using\nthe previously described perfect-model rank range which indeed reflects ensemble size as well as other\nmodel- and region-specific characteristics, as discussed in our response to point 2. above. For this\nadditional Criteria 1., allowed frequencies for observations occurring outside of the ensemble spread as\nrank 0 or rank n (with n being the number of members) now emerge from the perfect-model frequency\nrange, which by design takes into consideration both the ensemble size as well as the simulated regional\ntemperature distributions. Upper bounds of this range go from around 2% (e.g., for CESM2-LE region\nL10) to above 10% (e.g., for GFDL-ES2M2 region L2; these values can be found in the Supporting\nInformation figures). As discussed above, we have decided to also keep the fixed threshold assessment as\nadditional criteria to incorporate a sensible and non-overtly complex grid-cell level assessment in our\nevaluation. We also maintain the discussion of the caveats of applying such fixed thresholds to evaluate\nSMILEs of different sizes included in our original manuscript.\nLine 385 \"and\" -> an\nThis has been changed as suggested.\nLine 428 \"Lastly, we determine, per grid-cell, if a model simulates observed temperatures adequately\nwhen none of these biases are present.\" Is this saying when none of the biases in the previous paragraph\nare present, an additional procedure is applied to determine if a model simulation is adequate? If so what\nis that procedure?\nWe thank the reviewer for bringing this unclear statement to our attention. Indeed we are referring to\nthe previously discussed biases, with no additional procedure incorporated in this stage. We have\nclarified this section to specifically name the two biases assessed, and this line to read ‘Lastly, when\nneither bias a) nor b) are present, we determine a grid-cell to be unbiased.’ for clarity\nLines 430–433 are a repeat of lines 111–114? I still don't quite get what the time series refer to.\nWe have added a more elaborated description of what the time series represent and how they\nillustrate the biases highlighted by our rank-frequency evaluation both in the results and methods\nsections, as described above.\nLine 459. \"Significance for Figure 7 is calculated using a t-test approach for each domain.\" Fig. 7 is\nthe domains?\nThis is no longer relevant as while delving into the reviewer's comments we have found the t-test\napproach was not appropriate for our analysis.\nReviewer #2 (Remarks to the Author):\nReview for “Temperature variability projections remain uncertain after constraining them to best\nperforming SMILEs”\nI have read the manuscript and have several concerns, mainly about the scope of the study. The\nauthors essentially use output from large ensembles of climate model simulations, evaluate whether or\nnot they have a good representation of historical temperature variability, then assemble a “constrained\nensemble” based on admittedly subjective criteria to evaluate how much removing models with poor\nrepresentations of historical variability changes future projections. In general, they find that the answer\nis basically unchanged with or without their constraint.\nWhile this is a worthy effort, I do not believe this constitutes a novel study that adds to our\nunderstanding of temperature variability in climate model simulations. The authors provide a detailed\ndescription of their results, and while I have no objection to their findings or methodology (though I am\nslightly confused by the ranking figure) I do not believe this paper constitutes a sufficient advance to be\nconsidered in Nature Communications.\nWe thank the reviewer for their time and comments on this paper. We disagree with the reviewer that\nthis is not a ‘novel study that adds to our understanding of temperature variability in climate model\nsimulations’. We believe this study is novel for the following reasons:\n1. Without evaluation studies such as this one we cannot know whether our models correctly\nsimulate the world we live in. We cannot improve our models, nor effectively analyse their results\nwithout this knowledge. Suarez-Gutierrez et al, 2021 was the first paper to be able to investigate\nmodel performance of temperature variability by utilizing SMILEs. This paper, published less\nthan 5 years ago, already has 84 citations, highlighting the value of such work. This work has\nbeen cited extensively both by users applying the novel model evaluation approaches introduced\nthere within the climate science community, as well as by users both within and outside this\ncommunity (e.g. climate model data end-users across risk assessment, economic, agricultural\nand environmental modelling communities, etc.) aiming to assess and compare the performance\nof their climate model(s) of choice without the need to carry out extensive analysis themselves.\nTherefore, we believe bringing accessible and robust multi-model evaluation results, particularly\nfor impact-relevant metrics such as temperature variability, is crucial and will be of great benefit\nboth for the climate modelling community and for its end-users, and therefore be of great\ninterest to the Nature Communications readership.\n2. This work expands on previous related work such as Surez-Gutierrez et al, 2021, first, by\nexpanding the evaluation framework used to incorporate more robust and formally defined\nevaluation criteria, and second, by considering and evaluating metrics more closely related to\nclimate-driven environmental and socioeconomic impacts. This work offers the first evaluation\nof all available CMIP5 and CMIP6 SMILEs for temperature variability in local summer and winter\nseasons. Evaluating such variability is crucial for the representation of climate extremes, as well\nas for example for ecosystem stability or agricultural productivity, in a way that most commonly\nassessed metrics such as mean temperature or annual temperature variability are not. As such,\nour present study builds on previous work and provides an essential evaluation of how well\nstate-of-the-art climate models represent temperature variability. Furthermore, following the\nsuggestions of Reviewer 1, we have also modified our evaluation framework to incorporate\nformally defined and more objective evaluation criteria.\n3. Building on points 1. and 2., this evaluation will provide a basis for model selection and\nassessments for the evaluation of temperature related extremes, which cannot be correctly\nsimulated if the underlying variability is incorrect. Therefore our manuscript will provide an\nessential tool and guideline for future research on these issues.\n4. The latest IPCC assessment report flagged future projections of temperature variability as\nuncertain, with large model disagreement. The goal of our manuscript is to determine whether\nthis is due to incorrect representation of temperature variability today, and whether uncertainty\ncan be reduced by introducing performance constraining. A possible null hypothesis to explore is\nthat future temperature variability depends on historical variability, and therefore, by selecting\nmodels that best match observations and agree on their current variability representation, future\nprojections across best-performing models will also agree more and thus be less uncertain. In\ncontrast, the alternative hypothesis is that future temperature variability is intrinsic to how the\nEarth system evolves over different regions, and therefore even constraining projections of future\ntemperature variability change to models that best capture historical variability will not\nnecessarily reduce future uncertainty if those models that capture historical variability adequate\nstill exhibit diverging behaviours in the future. To the best of our knowledge, our findings are the\nfirst to reveal that, for large areas of the world, the second hypothesis is true. This means that,\nfirst, uncertainty in temperature variability change cannot be fully constrained nor reduced\nglobally based on historical model performance. Second, the tendency in the climate modelling\ncommunity towards pooling resources and developing fewer, less independent models, may lead\nto an insufficient sampling of model uncertainty, at least for this particular metric of\ntemperature variability, which is an example of change in the climate system that cannot yet be\nforeseen with the best available knowledge.\n5. Lastly, while our results indicating that variability change projections remain largely unchanged\nby picking the models that perform well historically may seem uninteresting, to the climate\nmodelling and evaluation community this highlights a critical knowledge gap; an area of\nuncertainty in our future projections that cannot just be improved by picking ‘the good models’.\nThis underscores the need for further investigation, model development and improvement, and\nthe use of multiple models to robustly estimate the range of potential changes for this climate-\nand impact-relevant metric. Results such as those presented here provide strong arguments\nagainst moving towards one ‘super climate model’ and will hopefully guide future modelling\nefforts to be as efficient and rigorous as possible.\nTo highlight and reflect upon these issues in the manuscript, we extensively expanded on the\nintroduction and discussion sections, and have added the following conclusions and implications\n(starting in line 458):\n“(i)We provide a comprehensive evaluation of the state-of-the-art SMILEs ability to represent the\nhistorical summer and winter temperature variability in observations. We identify CESM-LE and CESM2-LE as\nthe SMILEs that provide the best representation of isolated temperature variability as well as of both\ntemperature variability and forced change, with GFDL-SPEAR-MED and MPI-GE5 as close third in each\ncategory respectively. This multi-model evaluation across all available CMIP and CMIP6 SMILEs provides a\nbasis for model selection for the assessment of temperature variability and extremes, which cannot be correctly\nsimulated if the underlying variability is incorrect.\n(ii) Our evaluation also shows that some regions are systematically not well represented such as the\nSouthern Ocean (O9), the Indian Peninsula (L20), and Northern Australia (L22), as well as the Amazon basin\n(L6), Northern (L12) and Eastern Africa (L14), and Southern Australia (L23), all specifically in the local\nsummer season. We also find that certain areas of the world are systematically well captured by most models,\nespecially in the local winter season, namely North America (L1-L4), Southern South America (L8), Europe\nand Northern Africa (L9-L11) and North and Central Asia (L16-L19). Finally, we conclude that local winter\nseasons are generally simulated better by most models more often than local summer seasons.\n(iii) Similar to the IPCC, we find low model agreement on the sign of the change over large parts of the\nEarth’s surface and large model disagreement on the magnitude of the projected change under future warming,\nparticularly over the land surface and the tropical Pacific Ocean. This implies that we cannot afford to move\naway from multi-model ensembles that sufficiently capture the uncertainty in our future projections.\n(iv) The constrained ensemble decreases model spread over some regions and gives a lower range of\nprojected futures, providing a smaller range of potential future projections and greater model agreement,\nparticularly in South America, India, Australia & South-East Asia, the west Pacific Ocean and Southern\nOcean in DJF and north South America, Africa, parts of Australia and the Southern Ocean in JJA. However, the\nconstraint does not substantially increase model agreement on the sign of the projected change.”\nMajor Comments:\nFigure 1: The shading is extremely difficult to see, which makes this plot difficult to interpret. I found\nthe discussion of rank histograms slightly confusing and thought the authors should say more about\nwhat a “desirable” scenario would be for these histograms. Presumably it would be that the observations\nsit somewhere in the middle of the ensemble, but that’s just my guess.\nWe thank the reviewer for bringing these issues to our attention. We have modified the format of the\nfigure to improve its readability, which should be corrected now in standard non-web based pdf readers.\nWe have also expanded the discussion along this figure (e.g., lines 114-197) as well as in the methods\nsection (lines 514-625) to clarify our evaluation framework, including what desirable performance\noutcomes may look like (lines 163-168), as well as what the different biases considered may look like for\nthe updated evaluation criteria used (lines 177-186).\nThe discussion of detrended versus non-detrended data was interesting, but qualitative and\nspeculative. I was also confused by lines 172-175, where the authors say “The higher agreement between\nnon-detrended data and observations could arise from the fact that misrepresented forcing signals can\ncounteract commonly-found variability overestimation biases” Wouldn’t this be the opposite? I would\nthink that higher agreement between non-detrended data and observations would arise from accurate\nrepresentation of the forcing signal, rather than misrepresentation.\nWe have expanded the results section regarding the comparison of detrended and non-detrended\nevaluation results, and we also include an example of these two versions of our evaluation in Figure 1 and\nrelated discussion. We provide a quantitative comparison on model performance based on regions that\nare captured adequately for detrended and non-detrended data as well as potential arguments exploring\nwhy performance may be affected by detrending in the results section (lines 177-186, 198-210, 220-280).\nIn regards to the comment on ‘higher agreement between non-detrended data and observations\ncould arise from the fact that misrepresented forcing signals can counteract commonly-found variability\noverestimation biases’, we thank the reviewer for highlighting this unclear point. In general, for\ndetrended data, good agreement between models and observations indicates that models capture\ntemperature variability correctly. When evaluating non-detrended data, good agreement indicates that\nmodels capture both variability and forced signals correctly. Therefore, since the second is a stricter test\nthat requires both the variability and the forced signal to be well captured, models would perform either\nequally well (for the cases where the variability was already well captured and the now non-detrended\nforced signal is also well captured) or worse (for the cases where the variability was already well captured\nand the now non-detrended forced signal is not well captured). Generally, the cases where the forced\nsignal is well captured but the variability is not would still yield a negative performance evaluation with\nnon-detrended data. Therefore, in theory, we do not expect improved performance by assessing\nnon-detrended data.\nHowever, we find limited examples of cases where our evaluation indeed yields improved agreement\nusing non-detrended data, as we noted in the line highlighted by the reviewer. What we mean in this case\nis, rather than revealing an improved agreement, the use of non-detrended data is confounding our\nevaluation and masking biases. For this, we refer to an example for this situation, which has also been\nadded to the revised manuscript (see Figure 1 L12 panel and Fig. D below).\nWe see from the non-detrended evaluation for region L12 simulated by CanESM5 (left panels in Fig.\nB), that observations occur slightly towards the upper portion of the ensemble in the early 20th century,\nbeing somewhat warmer than most simulations. However, towards the end of the record the forced\nwarming signal in the model appears to be stronger than observed, and observations clearly cluster in the\nbottom section of the ensemble, while in both cases remaining well within the ensemble limits. This\nresults in a relatively flat rank histogram (color line in left rank histogram in Fig. D) that is within the\nperfect-model rank histogram range (gray dashes lines in left rank histogram in Fig. D), and therefore\nclassified as an adequate representation in our non-detrended evaluation.\nIn contrast, the evaluation of detrended data removes the bias in the forced warming and reveals a\nvariability overestimation bias, with observations occurring too frequently along the central ranks of this\nensemble (right panels in Fig. B). This results in the observations' rank frequencies that are larger than\nthe perfect-model rank range for the central rank windows, and therefore classified and inadequately\nrepresented in our detrended evaluation.\nDue to this too large variability and too strong warming signal in the model, historical\nnon-detrended observations cross the ensemble without exceeding the ensemble limits with a frequency\nthat appears uniform when assessing the non-detrended rank histogram, but is revealed to be indeed\nbiased upon closer inspection for the detrended evaluation. Since we use exclusively detrended data for\nour performance constraint, we expect such confounding factors to not affect our evaluation. However,\nwe still want to bring these issues to the readers attention, since potential end-users may choose to\nselect models based on their non-detrended performance. We have clarified and expanded on these\naspects in the revised manuscript (lines 267-278).\nFigure D: Example of the rank-frequency evaluation framework for JJA temperatures in CanESM5\nagainst GISTEMPv4 observations. Time series and rank histograms show the evaluation of spatially\naveraged temperatures for region L12 (Evaluation Criteria 1). Time series show the ensemble maximum and\nminimum (color lines) and central 75th percentile ensemble spread (shading) against observations (dots).\nRank histograms show observations rank frequency aggregated over a 3-rank bin (bars), the running mean\nrank frequency over a centered n/5 rank window (lines; for 1 to n-1 ranks) and the absolute frequencies of rank\n0 and n (crosses), with n the number of ensemble members, for observations (color) and perfect model rank\nrange (gray). If observations rank frequency is within perfect-model range for all rank windows, Criteria 1 is\nmet and this is highlighted by a green star at the top right; if not, by a red cross. Percentages at the top left\nshow the frequency of regionally averaged observations occurring above (red) or below (blue) ensemble limits,\nor clustering within the central 75th percentile range (gray), analogous to the grid-cell evaluation in Criteria 2.\nMinor Comments: Figure 6 never called or discussed?\nWe agree, this figure was not particularly useful and has been removed and replaced with a new\nFigure 6 and 7. We now include Figure 6 that compares temperature variability in the models at 1degree\nof warming with the observational record. Figure 7 has been updated to consider temperature variability\nat 3 degrees over regions that are poorly represented in climate models. This new Figure compares the\nconstrained and full ensembles and the constrained ensemble with the results from CESM2, which we\nfound to be one of the best performing models over these regions. Below is the discussion on this new\nFigure found on line 353.\n“While we cannot assess whether the future projections are more realistic in the constrained ensemble\ncompared to the full ensemble, as we do not have observations of the future, we can compare the projections\nwith those from the most realistic model over these poorly modelled regions in the historical period. CESM2 is\none of the two models that best represent Australia & South-East Asia (L21-23), South America (L6-8), and\nAfrica (L12-16; the other being CESM-LE), with 15 \"good performances\" out of 22 across both regions and\nseasons. By using this model as a proxy for possible future observations, we can compare the differences at 3˚C\nof warming between the full and constrained ensemble, the constrained ensemble (excluding CESM2), and\nCESM2 (Fig. 7, see Figs. 10 and 11 for a spatially aggregated comparison across all regions). We find that,\nexcept for South America (where CESM2-LE fails to capture historical variability according to our evaluation\nfor L6 and L7) and Africa in DJF the constrained estimate is closer to CESM2 (our \"good model\") than the full\nensemble, tentatively suggesting that our future projections might be more realistic in the constrained\nensemble compared to the full ensemble. ”"}, {"round": 2, "reviewer_comments": {"reviewer_1": "The authors have responded carefully to all my comments. I have one comment about a possible significance testing approach and a few minor comments.\nThe authors state in their response \"Therefore, if the reviewer has any specific suggestions as to how to appropriately calculate significance for this study we are open to including them.\" I think that a permutation test is appropriate in this case, as it does not rely on distributional assumptions and is exact under the null hypothesis of exchangeability. Specifically, compute the difference in variances between the constrained and full ensembles. Then, repeatedly create random constrained ensembles (of the same size etc) by sampling without replacement from the full ensemble, and compute the same variance difference for each permutation. The p-value (two sided) is the fraction of permutations where the absolute value of the permuted variance difference exceeds that of the observed difference. I think that your calculation mentioned at line 380 is very close to this.\nMinor comments\nLine 46 define SMILE at first use in text (not at line 83)\nLine 56 “with 60N”?\nLine 57 \"This is also a region of decreasing variability\" what is \"this\"?\nLine 72 \"Therefore, a possible hypothesis is that future temperature variability depends on historical variability\" I don't know what this means. I think you want to mention models. Perhaps, \"Therefore, a possible hypothesis is that the ability of models to project future temperature variability depends on their ability to simulate historical variability\"\nLine 78 \"In contrast, the alternative hypothesis is that future temperature variability is intrinsic to how the Earth system evolves over different regions.\" I don't know what this means.\nLine 128 Delete comma\nLine 153 Readers will recognize \"perfectly flat\" as a bit of a straw man since no tests require perfect flatness, and the usual tests take sample size into account. The usual chi-squared test does (I think) assume independence. You could drop the sentence starting at line 150 and simply say at line 154 \"We tested the flatness of the rank histogram using a perfect-model rank range test which make no distributional assumptions and takes into account both ensemble size and serial correlation\"\nLine 166 and throughout \"Criteria\" is plural so Criterion 1?\nLine 254 space before GFDL\nLines 337 and 338 em dashes instead of hyphens", "reviewer_2": "In my original review, I expressed concerns about the novelty of this study and the importance of the results. The authors have done a better job at explaining why their study is different from the original 2021 study that used a similar approach to understand temperature variability in the CMIP5 models. I appreciate the new description of the two evaluation criterion, which helps clarifies what distinguishes this approach from the previous efforts to understand temperature variations in climate models.\nThe result that the constrained ensemble does not help projections of temperature variability converge is an interesting one, but in the authors’ response they write a hypothesis that “future temperature variability is intrinsic to how the Earth system evolves over different regions, and therefore even constraining projections of future temperature variability change to models that best capture historical variability will not necessarily reduce future uncertainty if those models that capture historical variability adequate[ly] still exhibit diverging behaviors in the future.” This is definitely supported by their results, but without an explanation of why the models disagree with respect to future climate change, it’s not really satisfying to me as a result. I completely understand, and agree with, the authors’ point that “uncertainty in our future projections that cannot just be improved by picking ‘the good models’” but I think this isn’t really a knowledge gap, but more of a philosophical problem in the climate modeling community.\nIn general the authors have addressed my comments thoroughly, which I appreciate given my somewhat critical review."}, "author_response": "Response to the Reviewers' comments on manuscript NCOMMS-25-1462A\n“Temperature variability projections remain uncertain after constraining them\nto best performing SMILEs” by Suarez-Gutierrez & Maher.\nPlease find enclosed our response to the Reviewers' remarks on our manuscript.\nWe thank the reviewers for their assessment and remaining comments, which we address individually\nmore in depth (in blue).\nReviewer #1 (Remarks to the Author):\nThe authors have responded carefully to all my comments. I have one comment about a possible\nsignificance testing approach and a few minor comments.\nThe authors state in their response \"Therefore, if the reviewer has any specific suggestions as to how to\nappropriately calculate significance for this study we are open to including them.\" I think that a\npermutation test is appropriate in this case, as it does not rely on distributional assumptions and is exact\nunder the null hypothesis of exchangeability. Specifically, compute the difference in variances between\nthe constrained and full ensembles. Then, repeatedly create random constrained ensembles (of the same\nsize etc) by sampling without replacement from the full ensemble, and compute the same variance\ndifference for each permutation. The p-value (two sided) is the fraction of permutations where the\nabsolute value of the permuted variance difference exceeds that of the observed difference. I think that\nyour calculation mentioned at line 380 is very close to this.\nWe had actually considered doing this bootstrap type in response to the last revision, but decided against\nit as we believe the test is not entirely appropriate. The suggested calculation would tell us whether the\nconstrained sample is different to a random subsample of the same number of members as we have\nconstrained to. However, the actual question we believe needs to be answered is whether the constrained\nestimate itself is statistically different from the full ensemble estimate. We believe this cannot be\nassessed with the suggested bootstrapping approach and, therefore, we argue against including it. We\nthank the reviewer again however for this suggestion, and in general for their dedicated response.\nMinor comments\nLine 46 define SMILE at first use in text (not at line 83)\nThanks for catching this - we have fixed the issue.\nLine 56 “with 60N”?\nWe have changed this to read “with 60N the latitude where the largest changes in fractional snow cover\noccur”.\nLine 57 \"This is also a region of decreasing variability\" what is \"this\"?\nWe have changed this to read “The 60N latitude is also a region…” to clarify this.\nLine 72 \"Therefore, a possible hypothesis is that future temperature variability depends on historical\nvariability\" I don't know what this means. I think you want to mention models. Perhaps, \"Therefore, a\npossible hypothesis is that the ability of models to project future temperature variability depends on\ntheir ability to simulate historical variability\"\nThanks for the suggestion - we have changed this as suggested.\nLine 78 \"In contrast, the alternative hypothesis is that future temperature variability is intrinsic to how\nthe Earth system evolves over different regions.\" I don't know what this means.\nWe have reformulated this sentence for clarity and now reads “In contrast, the alternative hypothesis is\nthat different models' projections of future temperature variability depend not simply on past\ntemperature variability, but rather on how the Earth system as a whole evolves over different regions in\neach model. If this were true, subsampling models based on their historical performance would not\nnecessarily reduce future uncertainty, as long as those historically best-performing models still exhibit\ndiverging behaviours in their representation of future temperature variability changes.”\nLine 128 Delete comma\nDone as suggested.\nLine 153 Readers will recognize \"perfectly flat\" as a bit of a straw man since no tests require perfect\nflatness, and the usual tests take sample size into account. The usual chi-squared test does (I think)\nassume independence. You could drop the sentence starting at line 150 and simply say at line 154 \"We\ntested the flatness of the rank histogram using a perfect-model rank range test which make no\ndistributional assumptions and takes into account both ensemble size and serial correlation\"\nWe thank the reviewer for bringing this to our attention. We have rephrased this section for clarity, which\nnow reads:\n\"However, the fact that we are evaluating free-running uninitialised simulations against a relatively short\nobservational record means that the internal variability in the climate system may be insufficiently\nsampled on these time scales, and thus rank histograms may appear to be not flat, even for perfectly\nperforming models.\nTo overcome this, we assess the flatness of the rank histogram using a perfect-model rank range test,\nwhich does not make any assumptions about the shape of the underplaying distribution and takes into\naccount both ensemble size and serial correlation.\"\nLine 166 and throughout \"Criteria\" is plural so Criterion 1?\nWe thank the reviewer for the suggestion, although we believe Criteria is also appropriate as it is very\ncommonly used in singular case, and we would prefer to keep it as is.\nLine 254 space before GFDL\nDone as suggested.\nLines 337 and 338 em dashes instead of hyphens\nThis has been changed to commas.\nReviewer #2 (Remarks to the Author):\nIn my original review, I expressed concerns about the novelty of this study and the importance of the\nresults. The authors have done a better job at explaining why their study is different from the original\n2021 study that used a similar approach to understand temperature variability in the CMIP5 models. I\nappreciate the new description of the two evaluation criterion, which helps clarifies what distinguishes\nthis approach from the previous efforts to understand temperature variations in climate models.\nThe result that the constrained ensemble does not help projections of temperature variability converge is\nan interesting one, but in the authors’ response they write a hypothesis that “future temperature\nvariability is intrinsic to how the Earth system evolves over different regions, and therefore even\nconstraining projections of future temperature variability change to models that best capture historical\nvariability will not necessarily reduce future uncertainty if those models that capture historical variability\nadequate[ly] still exhibit diverging behaviors in the future.” This is definitely supported by their results,\nbut without an explanation of why the models disagree with respect to future climate change, it’s not\nreally satisfying to me as a result. I completely understand, and agree with, the authors’ point that\n“uncertainty in our future projections that cannot just be improved by picking ‘the good models’” but I\nthink this isn’t really a knowledge gap, but more of a philosophical problem in the climate modeling\ncommunity.\nIn general the authors have addressed my comments thoroughly, which I appreciate given my somewhat\ncritical review.\nWe thank the reviewer for their insights. Regarding their point about reducing uncertainty in future\nprojections, we agree with part of the issue being a philosophical question, yet the question is indeed in\npractice often being answered by focusing on fewer, potentially ‘better’ or at least more complex models.\nTherefore, we believe presenting perspective as those in this paper to the modeling community is\nimperative to ensure that both reducible and irreducible projection uncertainties are addressed as\nrobustly as possible. We want to thank the reviewer again for their time and feedback."}]

Decision letter