Validating Climate Models with Spherical Convolutional Wasserstein Distance
Authors: Robert C. Garrett
Trevor Harris
Zhuo Wang
Bo Li
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Climate Models
Wasserstein Distance
Convolution
Functional Data
Primary area: machine_learning_for_physical_sciences
TL;DR: We create a Wasserstein distance variant based on spherical convolutions of functional data and apply the method to climate model validation.

Abstract: The validation of global climate models is crucial to ensure the accuracy and efficacy of model output. We introduce the spherical convolutional Wasserstein distance to more comprehensively measure differences between climate models and reanalysis data. This new similarity measure accounts for spatial variability using convolutional projections and quantifies local differences in the distribution of climate variables. We apply this method to evaluate the historical model outputs of the Coupled Model Intercomparison Project (CMIP) members by comparing them to observational and reanalysis data products. Additionally, we investigate the progression from CMIP phase 5 to phase 6 and find modest improvements in the phase 6 models regarding their ability to produce realistic climatologies.

Validating Climate Models with Spherical
Convolutional Wasserstein Distance
Robert C. Garrett' Trevor Harris? Zhuo Wang' Bo Li'
University of Illinois Urbana-Champaige Texas A&M University {rog4, zhuowang, Libo)eillinoii edu tharris@stat. tamu edu
Abstract
The validation of global climate models is crucial to ensure the accuracy and efficacy of model output. We introduce the spherical convolutional Wasserstein distance to more comprehensively measure differences between climate models and reanalysis data. This similarity measure accounts for spatial variability using convolutional projections and quantifies local differences in the distribution of climate variables. We apply this method evaluate the historical model outputs of the Coupled Model Intercomparison Project (CMIP) members by comparing them observational and reanalysis data products. Additionally, we investigate the progression from CMIP phase 5 to phase 6 and find modest improvements the phase 6 models regarding their ability produce realistic climatologies.
Introduction
Climate Model Validation General Circulation Models, or climate models, are mathematical representations of the climate system that describe interactions between matter and energy through the ocean, atmosphere, and land [Washington and Parkinson] 2005). Climate models are the primary tool for investigating the response of the climate system to changes in forcing, such as increases in CO2, and projecting future climate states [Flato et al.) 2014). To assess the plausibility of climate models, climate scientists compare output from model simulations against observational data [Rood] 2019 This comparison is the focus of climate model validation techniques for ensuring that climate models capture the dynamics of the climate system [Roca al. 202T
The Coupled Model Intercomparison Project (CMIP) was initiated 1995 as a comprehensive and systematic program for assessing climate models against each other and observational data (Eyring wide variety of experiments such as performing
et al. 2016 Each model in CMIP participates pre-industrial control simulation, and various simulations representing different scenarios for CO2 emissions [Eyring et al., 2016). Because historical simulations coincide with observational measurements, distribution of observational or quasi-observational data products [Raäisaänen] (2007). to assess their reconstructive skill. For complete spatial coverage compare against reanalysis data, blend of observations and short-range weather forecasts through data assimilation [Bengtsson et al. 2004 This has become one popular climate model validation method [Flato al. 2014).
historical simulation,
compare each model's 's synthetic climate distribution the
Previous approaches Many statistical and machine sarming-based methods have been applied to assess climate model output against reanalysis fields. The most common approach to compute the root mean square error (RMSE) between long-term means of the climate model output and the reanalysis field [Li et al. 2021 Zamani al. 2020 Karim et al. 2020 Ayugi et al., 202T]. RMSE provides a direct measure of the differences between two climate fields but does not take into account internal variability, so we should use with caution in evaluating climate models. Another approach,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

which is invariant to bias, is to compute measures of correlation between climate model output and
reanalysis fields (Zhao et al. 2021 Zamani et al. 2020 Karim et al. 2020 Ayugi al. 202T].
More comprehensive approaches employ techniques for random processes to compare two spatial fields. For example, [Shen et al.] (2002) and Cresssi et al. [2008] use the wavelet decomposition to compare the spatial-frequenge content of two fields. Hering and Genton| [2011] measures the loss differential between models of spatial processes, Lund and Li [2009) and Li and Smerdon 2012
compare the first and second moments of two random processes, and Yun et al. [022]] identifies
local differences in the mean and dependency structure between two spatiotemporal climate fields. Functional data analysis techniques have been introduced to compare spatial and spatiotemporal random fields, considering the random fields as continuous functions. Many of these approaches compare the underlying mean functions from two sets of functional data (Zhang and Chen 2007 Horváth et al., 2013 Staicu et al. 2014]. Other approaches include the second-order structure in the comparison [Zhang and Shao 2015 Li et al. 2016), or compare the distributions of two spatial
random processes [Harris et al. [021].
Since climate models aim to mimic the real climate which is the underlying pattern of weather, directly assessing the distributional differences between the modeled and observed data seems more thorough approach to evaluating climate models. [ et al. (2020) proposed to use the Wasserstein distance (WD), a popular metric for comparing probability distributions [Villani 2009]. J. for such purpose. The WD has also been considered for other use cases in climate science, such as data assimilation (Tamang et al. 2020 2021 2022]. The WD can be computationally expensive or even impossible to calculate between high- dimensinni distributions (Kolouri et al. [019]. so Vissio jet al. [2200] first converts each climate field to a single spatial mean and then only compares the distribution of spatial means. However, this dimension reduction puts their method at the risk of missing important spatial variability information, and consequently failing to accurately distinguish two climate fields that are different.
Recent contributions in the Machine Learning (ML) literature seek compare multivariate distri- butions using many features while leveraging the efficiency of the one mee-dimnniionl WD [Vallender 1974]. The sliced WD (Bonneel et al. 2015 compares random projections of distributions on R" and the generalized sliced WD Kolouri et al. 2019] extends this to broader class of projections. The convolutional sliced WD Nguyen and Ho| 2022 compares distributions of discrete square images in the space R accounting for the spatial structure using kernel convolutions. However, spatial fields from climate models are defined on a spherical domain, making them incompatible with the vectorized/square nature of these distances. The spherical sliced WD (Bonet et al.) and other spherical transport methods [Quellmalz et al.) [023 Cui et al. 2019) can compare distributions of spatial point processes over sphere, such as locations of natural disasters and extreme weather events, but cannot be used for smooth fields such as daily temperature and precipitation. Lastly, other non- non-WD approaches have been considered for multivariate distributional comparisons. For example, Mooers et al. (2023) compared distributions arising from global corm-resolling models using variational autoencoders.
Our proposal We propose the functional sliced WD as a generalization of the sliced WD to distributions of functional data. To create a tailored tool for climate model evaluation, we define the Spherical Convolutional Wasserstein distance (SCWD) as special case of the functional sliced WD for functions on the unit sphere $², manifold on which latitude-longiudde coordinates are defined, SCWD creates slices containing a small region of spatial fields to characterize local differences in the distribution of climate variables, which are further integrated into a single measure for global differences. Compared to the spatial mean- besand WD from Vissio et al. [2020]. SCWD accounts for spatial features while maintaining the modest computation for univariate WD when comparing climate models against reanalysis data, resulting in comprehensive and more robust evaluation. We
apply SCWD to rank climate models and assess the progression of the new CMIP era with respect to
daily average surface temperature and daily total precipitation.
2 Preliminaries
We consider the problem of comparing two probability distributions P and Q. Each distribution is a member of P(2), the set of Borel probability measures on some sample space, Ω. In our application, we treat climate fields as functional data (Wang et al., 2016) over a spatial domain S, thus we often assume Q is space of functions. Our sample space of interest is L (S), the set of square integrable
2

---

functions from S - IR where S is compact subset of R". CMIP model outputs are available at global scale, so we consider the spatial domain be the unit sphere S?, the space over which latitude and longitude coordinates are assigned to the Earth's 's surface. In fact, the space (3? has been
previously considered for modeling climate fields (Heaton et al.. 2014]. To compare two functional data distributions P and Q, we define distance function D(P. Q) to act as a similarity measure.
Comparing Distributions of Functions Comparisons for distributions of functions in (S) have been studied for specific cases of S (Hall and Van Keilegom 2007 Bugni and Horowitz 2021
Pomann et al. 2016 Harris et al. 2021], but none of these have focused on (,*( The mathematical
properties probability measures in (S)) have been studied before [Gijbels and Nagy 2017
Kim 2006), but it is challenging to calculate distances such as the WD in this space without additional assumptions [Li and Ma 2020]. To define a similarity measure for distributions in (S)) we build on the theory of the sliced WD and its various extensions, which have only been defined for distributions of miite-dimeesionall data.
Sliced WD Given a Borel function я : -2 + R. the pushforward of P under π is valid distribution in P(R) defined as T P(B) = P(T P(m-=(B)) (B)) for all Borel sets B in R. The r-th order sliced WD
[Bonneel et al., 2015) between P. Q E P(IR") is a metric defined as the mean of the ordinary WD
over univariate pushtorwards:
SW.(P.Q) = ( W. W, m)#
1/r
(I)
where (x) = 0 for x E Rn and 0 E S"-1, We call my the slicing function because it produces one-dimensional "slices" of the data using projection matrices. Because No# P and # are valid distributions in ((R)), the WD inside the integral can be calculated with the commonly used analytical form for univariate measures [Vallender 1974..
Generalized Sliced WD The generalized sliced WD (Kolouri et al. 2019 replaces with a more general class of slicing functions, denoted as 90- Because of this increased flexibility, the generalized sliced WD is a pseudometric rather than metric except for specific cases of go [Kolouri jet al. 2019]. For given use case, the utility and metric properties of the generalized sliced WD therefore determined by the choice of slicing function. Slicing functions can be chosen via optimization, estimated using neural networks, or specified by the researcher to isolate features of interest [Kolouri et al. 2019 The last option is desirable for our application, allowing the slices to be restricted to spatial features of interest to climate modelers. However, the generalized sliced WD is defined between distributions of data in R", a space that is not suitable for distributions of spatial fields. Climate fields could be coerced to vectors in IR" to be made compatible with the generalized sliced WD. However, this would result in loss of the inherent spatial structure, making it challenging to specify a slicing function that can handle considerations such as area weighting and spatial correlations.
Convolution Sliced WD The convolution sliced WD Nguyen and Ho 2022 represents images as matrices in the space R=*" The slicing function is replaced with kernel convolutions, or possibly a sequence of kernel convolutions, of d x pixels for d E N. The kernel aggregates nearby locations to isolate local features, which could provide useful information to climate modelers. However, when climate fields are represented on a rectangular grid, the geographic area represented by each grid cell varies drastically between latitudes due to the non non-Euclidean structure. Thus, a d x d pixel kernel will cover different sizes and shapes of geographic areas depending on location. For our application, the kernel radius should therefore be defined using geographic distance, not pixels.
3 Methods
We introduce the functional sliced WD framework which extends the flexible slicing process from the generalized sliced WD to the dimensioddmll case of functions in L LS))). We focus on the special case of (S2), which we call the spherical convolutional WD (SCWD), for our climate model validation application. SCWD adapts the kernel convolution- based slicing idea of the convolutional sliced WD to a continuous setting while accounting for the non-euclidean structure of climate fields realized over the Earth's 's surface.
3

---

ERA5 Reanalysis Data
Slicing Kernel
Time Series of Slices
Histogram of Slices
80
MMMM
1/2/177 1//1979
20
CMIP6 Model Output
1980
1996
-25
1//1979
200
1/21 1/31979
...
-10
50-
0-
1990
1982
1984
1996
1988
-10
CMPE Model
ERAS Reanalysis
CMP6 Model
ERAS Reanalysis
Figure I: Diagram representing the calculation of SCWD between distributions of daily mean surface temperature (in degrees Celsius) from ERA5 and a CMIP6 model. Each day, many projections are computed using kernel convolutions, represented here two locations. The resulting projections, called slices, summarize the local climate conditions in each dataset. The slices for each day are viewed sample from the marginal distribution each location, represented here histograms. SCWD is computed as global mean over the univariate WD between each pair of local distributions.
3.1 Functional Sliced Wasserstein Distance
In general it is not possible to analytically characterize distributions in P(L2(S)) and there are no closed form solutions for computing the WD. However, it is possible to slice elements of LS),,, meaning we can leverage the analytical form of the one-dimensional WD to define computable sliced WD. We extend the convolution slicer from Nguyen and Ho 2022] to the functional data case, allowing us to project functions in L (S) to values in R while preserving local spatial information.
Definition 3.1 (Convolution Slicer). Let be a compact subset of R", 8 E S, and k called the kernel function, be continuous function from S S [0, oe). We define the convolution slicer (f),, 2, a linear operator from f E (22(S) + R, as follows:
c,( = / (u)k(( u)))
To create valid functional sliced WD, we construct pushforward measures based the convolution slicer (f). To satisfy the definition of a pushfoward measure, we must show that ef( is Borel measurable function from 1.2 T2?(S) + R. By continuity of k, when location E S is fixed, k(s, u) is - R. Because is compact, k(ss, u) continuous function
a continuous function from on compact set and is thus bounded and L L'-integrable. It follows that the convolution slicer c, (f) an integral of the product of two functions f. E LS),)) so by Hölder's inequality, c2 (f) is bounded linear operator from (2( + R. Stein and Shakarchi [2011) states that bounded linear continuous linear operator and thus Borel measurable. So,
operators are also continuous, So cf((f)
for any measure P P(L (S)). the pushforward c.# P is valid measure in P(R). Therefore, can define a functional sliced WD between distributions ((L)(S)) as follows:
Definition 3.2 (Functional Sliced WD). Let S be a compact subset of IR", r 2 1, and P.Q E (S)) Let e, be operator satisfying Definition 3.1 We define the (r-th order) functional sliced WD between P and Q as follows:
FSW.(P,Q) -/W.ICCAPP..OHA
where W, is the Wasserstein metric on P(R).
Because C2 # and #Q are valid univariate probability measures, the analytical form of the univariate WD can be applied for efficient calculations. We introduce theoretical properties for the functional sliced WD in Theorem 3.3
Theorem 3.3. For all compuct subsets S C TR. FSW, is a pseudometric on P(Fs) and maintains the r-convexity property of the ordinary W,, metric.

---

A It is unknown if the final positivity property of a metric satisfied. Proof of this property would require an invertible Radon-like transformation be defined for probability measures in (S).. Therefore, as with the generalized sliced WD, it is up to the researcher to specify an appropriate kernel function over the domain of interest. We provide such a choice for our application and give theoretical justification in Section 3.2
Proof of Theorem 3.3 is provided in Appendix
3.2 Spherical Convolutional Wasserstein Distance
For our application climate model validation, specify the spatial domain S to be the unit sphere S², the space over which latitude-longitude coordinates are assigned to locations on the Earth. To preserve local spatial information, we specify the slicing function be radial kernel function. We introduce the spherical convolutional WD (SCWD) as specific case of the functional sliced WD:
Definition 3.4 (Spherical Convolutional WD). Let P. Q E (())(S?)) and r 2 We define the (r-th order) SCWD between P and Q as follows
SCW, = ( W. W.W.
1/r
w,( = | (()))(s,u)dd,
where W, is convolution slicer that satisfies Definition 3.1 with associated radial kernel ((s, u). Because Φ is a radial kernel function, W, aggregates local information and the resulting pushforward measures Wa# P and wa W+ represent the local distribution around each location s. SCWD is therefore calculated as the global mean of the WD between local distributions at each location. The local WD values can be recorded and later visualized as a map to pinpoint regions with higher or lower similarity. Figure deemonstrates the process for calculating SCWD between two distributions of surface temperature helds, and details on the implementation are provided in Appendix
Kernel In our analysis, we specify the kernel function to be o[|s ul; I), where |s is the chordal distance between s and u and & is the Wendland kernel function used in Nychka et al. 2015):
(d:: = (1 (35d + 18d + 3)/3
d 1
d d>l,
with range parameter 1 > 0 determining the radius over which the kernel is nonzero. The Wendland kernel meets the continuity assumption in Definition [.I and is also compact, which enables efficient sparse computations for our analysis.
Positive definite kernels, such the Wendland kemnel for 7 less than the diameter (Hubbert and Jäger 2023). allow us to retain full spatial information via the spectral density. This is because the convolution theorem on S2 DDriscoll and Healy 1994] gives an injective correspondence between the spectral density of a function / E (S ) and the spectral density of the convolution (ʃ (s) Jga [u))k(s, u)du when is positive definite. Note that as oc, the Wendland kernel converges to the flat kernel (dd; oo) case, the SCWD will be equal to the global mean-based WD from Vissio et al. 22020), leading to a complete loss of spatial variability information. In our analysis, we ensure positive definite kernel by specifying | to be less than the diameter of the Earth (about 12, 750 km). We study the sensitivity of our results this parameter in Section 4.4
resulting in a SCWD where every slice is the global mean. In this
Spatial Analysis Projection selection approaches such as the Max-Sliced WD (Deshpande et al. 2019) and Energy-Based Sliced WD [Nguyen and Ho 2024) have been introduced as alternatives to the sliced WD that give higher slicing weight to the directions of greatest variability between high-dimensional distributions. Each of our slices corresponds to a local mean around a specific location, so applying projection selection methods to SCWD would be equivalent to identifying the geographic regions which two distributions of spatial fields have the greatest differences. This is of keen interest when validating climate models, so we examine spatial maps of our slices in Section 4.2 to identify these regions. However, for a comprehensive evaluation of climate fields, we favor the geographically-babae balancee SCWD in Definition 3.4 when evaluating similarity to reference datasets.
4 Climate Model Validation
We consider climate model outputs from the Coupled Model Intercomparison Project (CMIP) histori- cal experiment phases 5 and 6. We focus on daily average near-surface (2m) temperature in degrees
5

---

Celsius and daily total precipitation in mm. The CMIP6 historical simulations are organized by ensembles, each of which is distinguished with an ripf identifier (rip for CMIP5), representing realization, initialization, physics, and forcings of the model, respectively [Eyring et al. [016]. We obtain 46 CMIP6 model outputs with the rii1p1f1 ID and 33 CMIP5 model outputs with the r1i1p1 ID. Output was obtained either at the daily frequency or aggregated from 3-hourly data. At the time of writing, two of the 79 total models did not have output available at a suitable frequency for surface temperature. All 79 models had output available for total precipitation.
To serve as references for climate model evaluation, we collect the European Centre for Medium- Range Weather Forecasts (ECMWF) Reanalysis 5th Generation (ERA5) [Hersbach et al. 2020]
as well as the Reanalysis-2 data from the National Centers for Environmental Protection (NCEP)
[Kanamitsu et al. [002) reanalysis datasets. Both datasets were obtained for surface temperature and total precipitation at daily frequency. Due to known issues with reanalysis data for precipitation [Tapiador et al. 2017]. we obtain ohservations from the National Centers for Environmental Informa- tion (NCHI) Global Precipitation Climatology Project (GPCP) Daily Precipitation Analysis Climate
Data Record [Huffman et al., 2001 Adler et al. 2020 an additional reference for precipitation.
The historical time periods for each climate variable were chosen maximize the available model outputs and reference datasets. For surface temperature, a common time period of January I, 1979 to November 30th, 2005 was collected for each CMIP output and reanalysis dataset. For total precipitation, we restrict the time period to October 1, 1996 to November 30th. 2005 to accommodate the first available day of observations in the GPCP dataset. Each data product represents the climate variables on different latitude-longitdde grid, which varies in size and structure. See Appendix C for full details on the spatial resolution and availability of temperature/preeiptation for each dataset.
4.1 Evaluation of CMIP6 Models
To evaluate the skill of CMIP6 models in characterizing historical climate distributions, we compute SCWD hetween each model output and the reference datasets. Models which excel at replicating the local climate distribution in many different areas of the Earth will have a low SCWD. Conversely, models which fail to capture features of the local climate distributions, such as the mean, variance, or extremes, will have higher SCWD. We do not expect perfect agreement between models and historical data, so, similar to Vissio et al. [2020). we additionally calculate SCWD between the reference datasets a baseline for comparison. All SCWD calculations in this section use the Wendland kernel with range parameter of 1,000km for slicing.
We select the ERAS Reanalysis as our reference dataset for (2m) surface temperature due to its high spatial resolution. We calculate distances from each surface temperature model output in our CMIP5 and CMIP6 ensembles ERA5. In addition, we calculate SCWD between ERAS and the NCEP Reanalysis to compare the variability between reanalysis datasets to the variability between models and reanalysis. For total precipitation, calculate distances from each total precipitation model output in our CMIP5 and CMIP6 ensembles GPCP. We include SCWD calculations from GPCP both ERA5 and NCEP for comparison, with the secondary goal of assessing the accuracy of each reanalysis in faithfully filling gaps in observed precipitation measurements. Full details the SCWD rankings can be seen in Tables 2 4 and in Appendix D Here, we focus on the results for CMIP6, and Figure provides the SCWD from each CMIP6 model output the ERA5 surface temperature field and GPCP total precipitation field.
select the GPCP observational data as reference. We
Surface Temperature For surface temperature, NCEP has a lower SCWD to ERAS than all CMIP models. This is not a surprise: both ERA5 and NCEP are based on observations, so we expect their temperature distributions to be similar most regions. Among the model outputs, many models have a SCWD to ERA5 similar to that of NCEP In particular, AWI-CM-1-I-MR from the Alfred Wegener Institute and PEPI-ESM--2-HHE from the Max Planck Institute have the lowest SCWD for surface temperature, with few models close behind.
Total Precipitation Compared to surface temperature, NCEP no longer has a lower SCWD to GPCP than the CMIP6 models. Instead, the ERA5 total precipitation field, which has the lowest SCWD to GPCP, serves as a better baseline for comparison. Deficiencies of precipitation from reanalysis have been reported in previous studies [Janowiak et al. 1998). In brevity, precipitation is sensitive to model physics and is not strongly constrained by observations via data assimilation. The low SCWD of FRAS can likely be attributed to the high model resolution and more advanced model physics and data assimilation system compared to NCEP, Among the CMIP6 models, the

---

Ranking CMIP6 Model Outputs by SCWD to ERA5 and GPCP
BCC-ESM1
Model Group
3.0
SCWD to GPCP Total Preciplation
ACCESS
IITM
AWI
INM
FGOALS-FE -3--L
BCC
IPSL
2.5
Can
KACE
NCEP
CESM
KIOST
CMCC
MIROC
G155-E2-2-6 /
E3SM
EC-Earth
MRI
IITM -ESM
FGOALS
NESM
AWI-CM-1-1-MR
GFDL
Nor
MPI-ESM1-2-HR
GISS
SAM
EC CC-aarhh3
ICON
Tal
NorE MorESM2-5 M2-MM
ERA5 Total recipitation
SCWD to ERA5 (2m) Surface Temperature
Figure Ranking CMIP6 model outputs using SCWD. Each model output is represented by point on the scatter plot and models from the same group share the color and shape. The x-axis and y-axis values represent each model's SCWD to the ERA5 surface temperature and GPCP total precipitation fields, respectively. The NCEP reanalysis is included as blank triangle with dashed lines representing the SCWD to ERA5 and GPCP. The SCWD from the ERA5 total precipitation field to GPCP is represented as solid line.
Norwegian Climate Centre NorESM2-MM model output stands out with the lowest SCWD to GPCP by relatively wide margin. The EC-Earth3 and CESM model outputs also form clusters with low SCWD values for precipitation.
No single model has the lowest SCWD value for both surface temperature and total precipitation, but NorESM2-MA seems to have the best balance of low distances for each variable and is not far off from the intersection of the lines for our reference datasets. Similarly, model has the highest SCWD value for both climate variables. The GISS-E2-2-66 model from NASA's Goddard Institute for Space Studies is distinct outlier with high surface temperature SCWD to ERAS, and the Beijing Climate Center BCC-ESMI model an outlier in terms of high SCWD to GPCP. We investigate
these high SCWD values in Section 4.2
4.2 Spatial Comparisons
Because SCWD is calculated as a global mean of local WD values, we can investigate the geographic sources of these outlying high SCWD values. Figure 3 provides a spatial breakdown the local WD values obtained when calculating SCWD for surface temperature between ERA5 and AWI-CM-I-1- MR well ERA5 and GISS-E2-2-G. the models with the lowest and highest SCWD ERAS, respectively. Overall, both maps seem smooth or continuous in space, with little variation between neighboring locations in most cases. For the map between AWI-CM-I-I-MR and ERAS, the local WD values are relatively low everywhere, with regions of slightly higher values near the poles and mountains. Compared to AW1-CM-I-I-MB the GISS-E2-2-G model has similarly low WD values in the tropics. However, closer to the poles, the local WD values begin to increase. In particular, the Arctic region has extremely high WD values relative the rest of the Earth. This is indicative the previously documented winter cool bias in the Artic region for GISS-E2-2-G [Kelley et al. 2020
7

---

SCWD Map from AWI-CM -1-1-MR to ERAS
SCWD Map from GISS -E2-2-G to ERA5
2m Surface Temperature
2m Surface Temperature
WD
15
SCWD Map from NorESM2-MM to GPCP Total iP Precipiation
SCWD Map from BCC-ESM1 M1Mu to GPCP
Total P Preciptation
WD
Figure 3: Top: Map local Wasserstein distances from ERA5 to two CMIP6 2m surface temperature outputs: AWI-CM-1-1-MR and GISS-E2-2-G. Bottom: Map of local Wasserstein distances from GPCP to two CMIP6 total precipitation outputs: NorESM2-MM and BCC-ESMI. Color fill at each location is determined by the WD between the local distributions obtained from the convolution slicer in Definition 3.4 The color scale shared for all maps and continental boundaries are included in black aid spatial comparisons.
Similar maps are provided for the NorESM2-MM and BCC-ESMI outputs compared to GPCP total precipitation. Overall, both maps are less smooth than the surface temperature maps, potentially due the more localized nature of precipitation. Looking at both models, the higher values of SCWD near the equator in the Pacific and Atlantic oceans may be related to the double-Inten nouble-Intertropccll Convergence Zone problem common in CMIP models, in which excessive precipitation produced in the southern tropics [Mchoso et al. 1995 This trend is much more pronounced for BCC-ESMI than for NorESM2-MM. Additionally, BCC-ESM1 has a region of particularly high WD values around eastern Indonesia, where wind-terrain interaction plays important role in the regional distribution of precipitation. This region has been previously highlighted in Zhang et al.) [2021] as an area where BCC-ESMI heavily overestimates annual mean precipitation. SCWD maps for both climate variables and all CMIP5/CMIP6 model outputs are provided the supplemental material.
4.3 Comparing CMIP5 and CMIP6
To assess the progression from CMIP5 to CMIP6, Figure 4 provides boxplots of the SCWD from each CMIP model the reference datasets. The left panel provides SCWD calculations from the surface temperature models and NCEP to the ERAS Reanalysis. The median SCWD value for CMIP6 models to ERA5 is lower than that of CMIP5. However, the two boxplots share similar range, so the difference between the CMIP5 and CMIP6 ensembles is subtle. Overall, we see a promising, albeit limited, decrease in SCWD for typical CMIP6 models compared to CMIP5. This indicates improved performance of CMIP6 when it comes to reconstructing realistic temperature distributions the local level. The right panel provides SCWD calculations from each precipitation model and ERAS/NCEP to the GPCP dataset. The median SCWD value for CMIP6 is again lower than that of CMIP5. Compared to surface temperature, the difference between CMIP5 and CMIP6 more distinct. This indicates that relative to surface temperature, precipitation modeling has seen greater gains with the transition from CMIP5 to CMIP6. The improvement of CMIP6 models compared to CMIP5 in precipitation representation has been reported in previous studies using different evaluation

---

methods (e.g., Chen et al. [202T]). It can be probably attributed to the more advanced model physics and overall higher model resolution in CMIP6.
SCWD from CMIP Models to ERA5 Surface Temperature
SCWD from CMIP Models GPCP
Total P Pecipitation
3.5
3.0
3.0
2.5
NCEP
SCWD
SCWD
2.5
2.0
8
2.0
1.5
NCEP
ERA5
1.0 CMIPS
CMIP5
CMIP6
CMIP6
Figure 4: Left: Boxplots of SCWD from the CMIP5 and CMIP6 model outputs to the ERA5 Reanalysis for 2m surface temperature. Right: Boxplots SCWD from the CMIP5 and CMIP6 model outputs to the GPCP observational dataset for total precipitation. Each plot contains points representing the SCWD from each CMIP model output to the reference dataset (ERA5 or GPCP), with CMIP5 and CMIP6 separated into two boxplots for comparison. Dotted and dashed lines are included to represent the SCWD from NCEP to ERAS/GPCP and ERA5 to GPCP, respectively.
4.4 Metric Comparison
We compare our method to baseline methods and evaluate the sensitivity of our results the choice of range parameter. We generate seven additional rankings for the CMIP models: SCWD with range of 500km and 2.500km respectively, the global mean-based WD (GMWD) from Vissio et al.) [2020), RMSE and MAE computed on long-term mean climatologies, and WD and Sliced WD on data regridded to an icosahedral grid to preserve area weighting. Technical details and rankings together with our original 1,000km SCWD results can be seen in Tables and in Appendix [
All SCWD rankings are nearly identical regardless of the range parameter, except that several precipitation models, such as GFDL-CM4 and E3SM-2-0, rank better with a larger range. From here forward. we focus only on the chosen 1, 1.000km range parameter for SCWD. For most baseline methods (RMSE, MAE, WD. and Sliced WD), their rankings are moderately similar to those of SCWD, indicating general agreements, though each metric still shows unique perspectives. However, there is a large discrepancy between the rankings of GMWD and all other rankings. For surface temperature, SCWD, RMSE, and WD all show that the distance from NCEP to ERA5 is lower than that of all CMIP6 models, and MAE and Sliced WD also rank NCEP favorably among the models. However, GMWD ranks NCEP only among the median CMIP6 models. For precipitation, we see similar result for ERA5 compared to GPCP: ERA5 has the best ranking when using SCWD, RMSE, MAE, and WD, but is third the rankings for Sliced WD and again around the median for GMWD.
To better understand the differences in rankings between SCWD and baseline model evaluation methods such as GMWD and RMSE/MAE, Appendix E provides an experiment and data example. The experiment shows a case where SCWD detects changes in both the climatological mean and the variance of the anomalies, while the other methods detect only one type of change. In particular, we show that the RMSE and MAE criteria are unable to detect isolated changes in the variance of the anomalies. The data example focuses on the AAMO--NICON surface temperature model. Despite the overall similarity in rankings for CMIP6 surface temperature. this model ranks significantly higher by RMSE/MAE than by SCWD. Further investigation shows that SAMO-UNICON exhibits significant differences in both the climatological mean and the variance of the anomalies. Because RMSE/MAF are unable to detect shifts in the variance of the anomalies they artificially inflate the ranking of this model compared to SCWD.

---

5
Discussion
Climate model validation critical for ensuring that climate models faithfully represent the Earth system. To this end, we developed new similarity measure, called spherical convolutional Wasser- stein distance (SCWD), which quantifies model performance in a way that properly accounts for spatial variability. SCWD builds on previous sliced WD methods to compare distributions of infinite- dimensional functional data, specifically surfaces on the sphere S Overall, our results indicate that incorporating local perspectives, rather than just the global mean, is essential when evaluating the similarities between climate models and observational data. In both theory and practice, a model that represents the global mean well may have large compensating errors at the regional scale. SCWD better represents the regional performance of climate models than the previously proposed WD- evaluation criteria in [Vissio al. 2020]. Furthermore, SCWD is not limited only comparing the long-term mean climate state as with RMSE and MAF, and our convolution slicing provides better spatial insights (in the form of maps) than WD and Sliced WD.
For surface temperature, we found evidence to suggest that SCWD is more accurate than the previous WD-based approach for evaluating climate models, such as the two reanalysis datasets being more similar compared to climate models and reanalysis. However, for precipitation, similar findings were made for only one of the reanalysis datasets. Given the limitations with reanalysis for precipitation discussed in Section 4.1 additional experiments from different perspectives may be needed to fully assess the accuracy of our method. Because we have only shown that SCWD is pseudometrie, rather than a metric, SCWD may not be able to fully distinguish between climate fields which have differences only in the joint spatial distribution, rather than at the local level. We acknowledge that determining the sample complexity of SCWD is an area that requires further investigation.
We hope our method will be tested under many different scenarios and variables in climate science. For example, calculating SCWD on monthly mean or even annual data could help to detect biases in terms of longer range temporal variability. Another idea is to remove long-term climatological means from the data before computing SCWD. This would provide additional insight into which features are being captured beyond the mean climate state. SCWD can also be applied to machine-assisted climate model tuning HHourdin et al., [201]]. a growing area of research that relies on quality model evaluation metrics. Some ideas for future work to further improve SCWD include learning an optimal range value for the Wendland kernel function or using a neural network to estimate the kernel function, similar to Kolouri al. 2019).
Outside of climate science, SCWD can be used as a loss function for training generative models for 360" images. This only requires straightforward extension of SCWD to allow for multiple convolution layers, similar to the rectangular case in Nguyen and Ho [0022]. Likewise, the more general functional sliced WD can be used as a loss function for generative models on any functional manifold. For example, we can apply the functional sliced WD to texture mapping or color transfer on the surface of 3D models, which are typically considered as non-euclidean manifolds. SCWD can be also adapted to compare distributions of spatiotemporal fields, rather than spatial fields as in this article, by including both time and space in the functional data domain. This would require specifying a space-time kernel function [Porcu et al. 2016). The more functional sliced WD framework has a variety of potential use cases for comparing distributions of functions on broad class of manifolds. Potential application areas where the data lie on non-trivial manifolds include facial recognition Li al. 2014], astronomy (Szapudi 2008 and ecology (Sutherland al. 2015)
Acknowledgments and Disclosure of Funding
We acknowledge the World Climate Research Programme, which, through its Working Group on Coupled Modelling, coordinated and promoted CMIP6. We thank the climate modeling groups for producing and making available their model output, the Earth System Grid Federation (ESGF) for archiving the data and providing access, and the multiple funding agencies who support CMIP6 and ESGF. NCEP/DOE Reanalysis II data provided by the NOAA PSL, Boulder, Colorado, USA, from their website at httts:: / /psl noaa. gov. This work is partially supported by the National Science Foundation grants NSF-DMS- TTH-MMS-1833311 1303112, KSF-DGE-1922758 and NSF-DMS-2124576 as well as the National Oceanic and Atmospheric Administration (NOAA) grant NA180AR4410071,. Lastly, we thank the referces and area chairs for their time and valuable feedback.
10

---

References
Robert Adler, Jian-Jian Wang, Mathew Sapiano, George Huffman, David Bolvin, Eric Nelkin, et al. Global Precipitation Climatology Project (GPCP) Climate Data Record (CDR), Version 1.3 (Daily). 2020.
Brian Ayugi, Jiang Zhihong, Huanbuan Zhu, Hamida Ngoma, Hassen Babaousmail, Karim Rizwan, and Victor Dike. Comparison of CMIP6 and CMIP5 models in simulating mean and extreme precipitation over East Africa. International Journal of Climatology, 41(15):6474-64%6 2021.
Lennart Bengtsson, Stefan Hagemann, and Kevin I Hodges. Can climate trends be calculated from reanalysis data? Journal of Geophysical Research: Ammospheres, 109(D11), 1), 2004.
Clément Bonet, Paul Berg, Nicolas Courty, François Septier, Lucas Drumetz, and Minh Tan Pham. Spherical Sliced-Wart Wasserstein. In The Eleventh International Conference Learning Representa-
fions.
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51:22 45, 2015.
Federico A Bugni and Joel L Horowitz. Permutation tests for equality of distributions of functional data. Journal of Applied Econometrics, 36(7):661-8)) 2021.
Chao-An Chen, Tluang-Hsiung Hsu, and Hsin-Chien Liang. Evaluation and comparison of CMIP6 and CMIP5 model performance in simulating the seasonal extreme precipitation in the Western North Pacific and East Asia. Weather and Climate Extremes, 31: 31:100303, 2021.
Noel Cressie, Martina Pavlicová, and Thomas J Santner. Detecting signals in FMRI data using powerful FDR procedures. Statistics and its interface, 1(1):23-32, 2008.
Li Cui, Xin Qi, Chengfeng Wen, Na Lei, Xinyuan Li, Min Zhang, and Xianfeng Gu. Spherical optimal transportation. Computer-Aided Design, 1111111,, 2019.
Ishan Deshpande, Yuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander G Schwing. Max-Sliced Wasserstein Distance and its use for GANs. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10648-10656, 2019.
James Driscoll and Dennis M Healy. Computing Fourier transforms and convolutions on the 2-sphere. Advances in applied mathematics, 15(2):202-250, 1994.
V. Eyring. S. Bony, G. A. Meehl, C. A. Senior, B. Stevens, R. Stouffer, and K. E. Taylor. Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. doi: 10.5194/gmd-99 1937-2016. URL
Geoscientific Model Development, 195) 2016.
https: //gdd. coperni cus
/ttp///999997//1/////
Gregory Flato, Jochem Marotzke, Babatunde Abiodun, Pascale Braconnot, Sin Chan Chou, William Collins, Peter Cox, Fatima Driouech, Seita Emori, Veronika Eyring, et al. Evaluation of climate models. In Climate change 2013: the physical science basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change, pages 741 166. Cambridge University Press, 2014.
Andrew R Friedman, Yen-Ting Hwang, John CH Chiang, and Dargan MW Frierson. Interhemispheric temperature asymmetry over the twentieth century and future projections. Journal of Climate, 26(15):5419-545 2013.
Irêne Gijbels and Stanislav Nagy. On general definition of depth for functional data. Statistical Science, pages 630-639, 2017.
Peter Gleckler, Charles Doutriaux, Paul Durack, Karl Taylor, Yuying Zhang, Dean Williams, Erik Mason, and Jérôme Servonnat. A more powerful reality test for climate models. Eos, 97, 2016.
Peter Hall and Ingrid Van Keilegom. Two-sample tests functional data analysis starting from discrete data. Statistica Sinica, pages 1511-1531, 2007.

---

James Hansen, Makiko Sato, and Reto Ruedy. Perception of climate change. Proceedings of the National Academy of Sciences, (2937))E2415-E2222 2012.
Trevor Harris, Bo Li, Nathan Steiger, Jason E Smerdon, Naveen Narisetty, and Derek Tucker. Evaluating proxy influence in assimilated paleoclimate recontrruttions-Tssinn the exchangeabil- ity of two ensembles of spatial processes. Journal of the American Statistical Association, 116 (535): 1055)1100--113, 2021.
Heaton, M Katzfuss, C Berrett, and DW Nychka, Constructing valid spatial processes on the sphere using kernel convolutions. Environmetries, 25(1):2-15, 2014.
Amanda Hering and Marc Genton. Comparing spatial predictions. Technometrics, 53(4):414 425, 2011.
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The ERAS global reanalysis. Quarterly Journal of the Royal Meteorological Society, (166(3)):1199-200
2020.
Lajos Horváth, Piotr Kokoszka, and Ron Reeder. Estimation of the mean of functional time series and two-sample problem. Journal of the Royal Statistical Society; Series B (Statistical Methodology), 15(1):103-122, 2013.
Frédéric Hourdin, Thorsten Mauritsen, Andrew Gettelman, Jean-Christophe Golaz, Venkatramani Balaji, Qingyun Duan, Doris Folini, Duoying Ji, Daniel Klocke, Yun Qian, et al. The art and science of climate model tuning. Bulletin of the American Meteorological Society, 98(3):589-660 2017.
Simon Hubbert and Janin Jäger. Generalised Wendland functions for the sphere. Advances in Computational Mathematics, 49(1):3, 2023.
George Huffman, Robert Adler, Mark M Morrissey, David T Bolvin, Scott Curtis, Robert Joyce, Brad McGavock, and Joel Susskind. Global precipitation at one-degree daily resolution from multisatellite observations. Journal of hydrometeorology, 2(1):36-50, 2001.
John E Janowiak, Arnold Gruber, CR Kondragunta, Robert E Livezey, and George Huffiman. A comparison the NCEP-NCAR reanalysis precipitation and the GPCP rain gauge-satellite combined dataset with observational error considerations. Journal of Climate, 11611)2990-2979, 1998.
Masao Kanamitsu, Wesley Ebisuzaki, Jack Woollen, Shi-Keng Yang, JJ Hnilo, M Fiorino, and GL Potter. NCEP-DOP AMIP-II Reanalysis (R-2). Bulletin of the American Meteorological Society, (3(11))16311166 2002.
Rizwan Karim, Guirong Tan, Brian Ayugi, Hassen Babaousmail, and Fei Liu. Evaluation of histor- ical CMIP6 model simulations seasonal mean temperature over Pakistan during 1970-2014. Atmosphere, 1005) 2020.
Maxwell Kelley, Gavin Schmidt, Larissa Nazarenko, Susanne E Bauer, Reto Ruedy, Gary Russell, Andrew S Ackerman, Igor Aleinov, Michael Bauer, Rainer Bleck, et al. GISS-E2. I: Configurations and climatology. Journal of Advances in Modeling Earth Systems, 12(8): c2019MS002025, 2020.
Jong Uhn Kim. Invariant measures for stochastic nonlinear Schrödinger equation. Indiana University mathematics journal, pages 687-717, 2006.
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde, Generalized sliced Wasserstein distances. Advances in Neural Information Processing Systems, 32, 2019.
Bo Li and Jason E Smerdon. Defining spatial comparison metrics for evaluation of paleoclimatic field reconstructions of the Common Era. Environmetrics, 23(5):394 406, 2012.
Li, Xianyang Zhang, and Jason E Smerdon. Comparison between spatio-temporal random processes and application to climate model data. Environmetrics, 2765):677-279 2016.
12

---

Jingjing Li, Ran Huo, Hua Chen, Ying Zhao, and Tianhui Zhao. Comparative assessment and future prediction ssing CMIP6 and CMIP5 for annual precipitation and extreme precipitation simulation. Frontiers in Earth Science, 9, 2021. ISSN 2296-6463. doi: 10.3389/feart. 2022 .687966. URL https: / www. wwontiersiin. org/ argg/rttice//10. 3389/feart. 687976
Ruonan Li, Pavan Turaga, Anuj Srivastava, and Rama Chellappa. Differential geometric repre- sentations and algorithms for some pattern recognition and computer vision problems. Puttern Recognition Letters, 43:3-16, 2014.
Tao Li and Jinwen Ma. Functional data clustering analysis via the learning of gaussian processes with wasserstein distance. In Neural Information Processing: 27th International Conference, ICONIP 2020, Bangkok, Thailand, November 23-27, 2020, Proceedings, Part II 27, pages 393-403. Springer, 2020.
Robert Lund and Bo Li. Revisiting climate region definitions via clustering. Journal of Climate, 22 (7):1787-1800, 2009.
Carlos R Mechoso, Andrew W Robertson, N Barth, MK Davey, P Delecluse, PR Gent, S Ineson, B Kirtman, Mojib Latif, H Le Treut, al. The seasonal cycle over the tropical Pacific in coupled ocean-atmosphere general circulation models. Monthly Weather Review, 123(9))2825-2836 1995.
Griffin Mooers, Mike Pritchard, Tom Beucler, Prakhar Srivastava, Harshini Mangipudi, Liran Peng, Pierre Gentine, and Stephan Mandt. Comparing storm resolving models and climates via unsuper- vised machine learning. Scientific Reports, 13(1):22365, 2023.
Khai Nguyen and Nhat Ho. Revisiting sliced Wasserstein on images: From vectorization to convolu- tion. Advances in Neural Information Processing Systems, 35:17788 1181788-17801, 2022.
Khai Nguyen and Nhat Ho. Energy-based sliced Wasserstein distance. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Douglas Nychka, Soutir Bandyopadhyay. Dorit Hammerling, Finn Lindgren, and Stephan Sain. A multiresolution Gaussian process model for the analysis of large spatial datasets. Journal of Computational and Graphical Statistics, 24(2):579-599, 2015.
Gina-Maria Pomann, Ana-Maria Staicu, and Sujit Ghosh. A two sample distribution-free for functional data with application to a diffusion tensor imaging study of multiple sclerosis. Journal of the Royal Statistical Society. Series C. Applied Statistics, 65(3):395, 2016.
Emilio Porcu, Moreno Bevilacqua, and Marc G Genton. Spatio-temporal covariance and cross- covariance functions of the great circle distance on a sphere. Journal of the American Statistical Association, 111(514):888 1188888888888888888 2016.
Michael Quelimalz, Robert Beinert, and Gabriele Steidl. Sliced optimal transport on the sphere. Inverse Problems, 39(10):055005, 2023.
Jouni Raäisaänen. How reliable are climate models? Tellus A: Dynamic Meteorology and Oceanog- raphy, 59(1):2-29, 2007.
Rémy Roca, Ziad S Haddad, Fumie F Akimoto, Lisa Alexander, Ali Behrangi, George Huffman, Seiji Kato, Chris Kidd, Rerre--Emanuus Kirstetter, Takuji Kubota, et al. The joint IPWG/GEWEX precipitation assessment, 2021.
Richard B. Rood. Validation of Climate Models: An Essential Practice, pages 737-762. Springer Inter- national Publishing, Cham, 2019. ISBN 978-3-319-7076622 doi: 11100779778-3311-7776---- URL /attps://do:: org/10
Xiaotong Shen, Hsin-Cheng Huang, and Noel Cressie. Nonparametric hypothesis testing for spatial signal. Journal of the American Statistical Association, 97(460): 122-1 2002.
Ana-Maria Staicu, Yingxing Li, Ciprian M Crainiceanu, and David Ruppert. Likelihood ratio tests for dependent data with applications to longitudinal and functional data analysis. Scandinavian Journal of Statistics, 41(4):932-96) 949229999.209 2014.
13

---

Elias M Stein and Rami Shakarchi. Functional analysis: introduction to further topics in analysis, volume 4. Princeton University Press, 2011.
Chris Sutherland, Angela K Fuller, and Andrew Royle. Modelling non-Euclidean movement and landscape connectivity in highly structured ecological networks. Methods in Ecology and Evolution, 112):169-177, 2015.
István Szapudi. Introduction to higher order spatial statistics in cosmology. In Data Analysis in Cosmology, pages 457-492. Springer, 2008
Sagar K Tamang, Ardeshir Ebtehaj, Dongmian Zou, and Gilad Lerman. Regularized variational data assimilation for bias treatment using the Wasserstein metric. Quarterly Journal of the Royal Meteorological Society, 146(730):23322234. 2020.
Sagar Tamang, Ardeshir Ebtehaj. Peter Van Leeuwen, Dongmian Zou, and Gilad Lerman. Ensemble Riemannian data assimilation over the Wasserstein space. Nonlinear Processes in Geophysics Discussions, 2021:1-22 2021.
Sagar K Tamang, Ardeshir Ebtchaj. Peter Jan Van Leeuwen, Gilad Lerman, and Efi Foufoula- Georgiou. Ensemble Riemannian data assimilation: towards large-scale dynamical systems. Nonlinear Processes in Geophysics, 29(1)77--9,, 2022.
FJ Tapiador, A Navarro, V Levizzani, García-Ortega, GJ Huffman, C Kidd, PA Kucera, CD Kum- merow, H Masunaga, WA Petersen, al. Global precipitation measurements for validating climate models. Atmospheric Research, 1-0,, 2017.
SS Vallender. Calculation of the Wasserstein distance between probability distributions on the line. Theory of Probability Its Applications, 18(4):784-786, 1974.
Cédric Villani. The wasserstein distances. Optimal Transport: Old and New, pages 93-111, 2009.
Gabriele Vissio, Valerio Lembo, Valerio Lucarini, and Michael Ghil. Evaluating the perfor- mance of climate models based Wasserstein distance. Geophysical Research Letters, 47 (21):e2000GL0089335, 2020.
Jane-Ling Wang, Jeng-Min Chiou, and Mtans-Geoy Müller. Functional data analysis. Annual Review of Statistics and its application, 3:257-295, 2016.
Warren M Washington and Claire Parkinson. Introduction to three-dimensional climate modeling. University science books, 2005.
Sooin Yun, Xianyang Zhang, and Bo Li. Detection of local differences in spatial characteristics between two spatiotemporal random fields. Journal of the American Statistical Association, 117 (537):291-306, 2022.
Yasin Zamani, Seyed Arman Hashemi Monfared, Mohsen Hamidianpour, al. A comparison of CMIP6 and CMIP5 projections for precipitation to observational data: the case of Northeastern Iran. Theoretical and Applied Climatoloyy, 142(3):1613-1622 2020.
Jie Zhang, Tongwen Wu, Fang Zhang, Kalli Furtado, Xiaoge Xin, Xueli Shi, Jianglong Li, Min Chu,
Li Zhang, Qianxia Liu, et al. BCC-ESMI model datasets for the CMIP6 Aerosol Chemistry Model Intercomparison Project (AerChemMIP). Advances in Amoopheric Sciences, 38:317-328, 2021.
Jin-Ting Zhang and Jianwei Chen. Statistical inferences for functional data. The Annals of Statistics, pages 1052-1079, 2007.
Xianyang Zhang and Xiaofeng Shao. Two sample inference for the second-order property of
temporally dependent functional data. Bernoulli, 21(2):909 929, 2015.
Shanshan Zhao, Wenping He, Tianyun Dong, Jie Zhou, Xiaoqiang Xie, Ying Mei, Shiquan Wan, and Yundi Jiang. Evaluation of the performance of CMIP5 models to simulate land surface air temperature based on long-tange correlation. Frontiers in Environmental Science, page 6, 2021.
14

---

A Proof of Theorem 3.3
Proof. Let 1, let & be compact subset of R", and let P.Q,U E (S)) We show the identity property holds:
FSW.(P,P) (1 W.lcHRRaPPyd 1/r
(18 O'ds)
1/r
The second line holds by the identity property of the ordinary WD. Next, the symmetry property:
(1 W, W.
1/r
FSW,(P.Q)+
( W, Q, Py ds
1/r
(Q, The second line holds by the symmetry property of the ordinary WD. Next, the triangle inequality:
FSW,(P,U) ( W.l.. 1/r
1/r
W. (W ds)
1/r Wr(co W.(csPP.cHH))d. +(
W 1/r
F.W,( + FSW,
The second line holds by the triangle inequality property of the ordinary WD and the third line holds by the Minkowski inequality. Lastly, we check the r-convexity property. Let À E [0, 1]. If we take P as the reference distribution, this property tells us what happens to the functional sliced WD as we interpolate between Q and U:
1/r
(P, + -A))) ( (. (1-)) x)
1/r
Wolc,#P,Ac, AUC ds
AW.(c.#P,c.#O)) + (1 P. 1/r
W. +
(1 X) / W.Ccc.P...HUU ds
1/r
1/r (VFS. (1-
The second line follows from properties of linear operators (in this case c.). The third line follows by the r-convexity of the ordinary WD, and the fourth line follows from the Minkowski inequality. We have shown all three pseuduometric properties and the c-coveexty property, so our proof is complete.
15

---

B SCWD Implementation
The following algorithm describes the process by which we calculated the SCWD values shown in All computations were performed using R version 4.3.2 on an Ubuntu operating system with an Intel i5-9600k processor (6 cores), 32GB RAM, and 3TB hard disk space. Total computation time for all experiments was under 72 hours. The calculation of each distance measure was fast, but loading the required datasets into memory for processing took the majority of the compute time,
Section g
Algorithm 1 Spherical Convolutional Wasserstein Distance Approximation
DATA
Reference dataset Xo()), E To: sample of spatial fields
Model outputs (1),, E Ti, (t), & E Tn n samples of spatial fields to be compared to
Xo
PARAMETERS AND APPROXIMATION GRIDS
Wasserstein order parameter 2
Range parameter 1,000 km kernel radius
Approximation quantiles Q: 200 evenly spaced quantiles ranging from to 1 with step size of 0.005
Grid G1: 60 120 regular latitude-longitude grid of center points for strided convolution
Grid G2: 361 x 720 regular latitude-longitude grid to provide discrete approximation of spherical
domain
STEP 1. PRECOMPUTE SLICING WEIGHTS
for each location E 11 do
Calculate vector of chordal distances from 8 to all locations in G2
Calculate Wendland function (2) with range / for all locations G2 using distance vector Apply area weighting to the Wendland kernel values using the area of each grid cell in G2 Normalize area- weighted kernel and store the results as a sparse vector W(s)
end for
STEP 2. COMPUTE SLICED QUANTILES
for E 0, J,.,,..., 72 do
Re-grid X, G2 without smoothing (one nearest neighbor upsampling) for each location E G1 do
Slice X; into one dimension using the dot product (s, = (W(s),
Calculate the sliced quantile function X (s,4), denoted as (x)
end for end for
STEP 3. CALCULATE APPROXIMATE SCWD
for E I,..., n do
for each location G1 do Calculate the local WD between Xo and X; centered around s as ds(s)) = Eyeq F (q) (q)l" end for
Calculate the approximate SCWD between Xo and X; SCWD(X,, X.) R (Exea, d((s)r
end for
The climate model outputs and reanalysis datasets (ERA, NCEP) used our analysis have no missing data. However, the GPCP observational dataset used as the reference for total precipitation did have missing data at some sites for a few days in the historical period. To handle missing data in the GPCP dataset, we modified the slicing process in Step 2. If locations corresponding to greater than 50% of the convolution weight were missing when calculating a slice value, an NA value was recorded. The local WD calculations in Step 3 were computed ignoring these NA values. In total, 17,943 slices were missing sufficient data when using the 50% threshold. However, total of 24, 105,600 slices were considered for the GPCP dataset (3,348 days in the analysis period with 7,200 slices per day in the strided convolution), so the missing slices constituted under 0. 1% of the final sliced GPCP data used in the analysis.
16

---

C Table of Data Details and Access Links
Rissnalanalssi Data
Longs
TAS
CMIP6 Models
Longs
NCTP Reanalysis
144
73
Yes
Yes
ACCESS CM2
192
144
Yes
Yes
FRAS Reanalysis
1440
721
Yes
Yes
ACCESS-ESMEL
192
145
Yes
Yes
GPCP Observations
360
Yos
AWI-CM-1-1-MRR
384
192
Yos
Yos
AWL-ESM-I-I-LE
192
96
Yes
Yes
BCC-ESMI
128
Yos
Yes
CMIP5 Models
Longs
Lars
TAS
PR
CESM2
288
192
Yos
Yes
ACCESSI.0
192
145
Yes
Yes
BESM2-FV2
144
96
Yes
Yes
ACCESS
192
145
Yes
Yes
CESM2. WACCM
288
192
Yes
Yes
CanCM4
128
64
No
Yes
DISM2-WACCM-FV2
144
96
Yes
Yes
CMCC-CESM
CMOC-CME-HRC
288
192
Yos
Yos
CMCC-CM
480
240
Yes
Yes
CMCC-CM2-SKE
288
192
Yes
Yes
CMCC-CMS
192
96
Yes
Yes
CMOC-ESM2
288
192
Yos
Yos
CNRM-CMM
256
128
Yes
Yes
CanESM5
128
64
Yes
Yes
CSIRO-MI3-600
192
96
Yes
Yes
E3SM 1-0
360
180
Yes
Yes
CanESM2
128
64
Yes
Yes
E35M-2-0
360
180
Yes
Yes
EC-EARTH
320
160
Yes
Yes
E:SM-2-0-NARMM
360
180
Yos
Yes
FGOALS g2
128
EC-Earthh3
Yos
Yos
FGOALS-22
128
108
Yes
Yes
ECC Earih3 AerChem
512
256
Yes
Yes
GFDL-CM3
144
90
Yes
Yes
EC- 3CC-Eathh--CC
512
256
Yos
Yes
OFDL-ESMER
144
Yes
Yes
EC Ear--Erthh-- Veg
512
256
Yes
Yes
GFDL-ESMON
144
90
Yes
Yes
E.C. Farth3 Veg-LR
320
160
Yes
Yes
HadCM3
Yes
Yes
FGOALS-F3-L
288
180
Yes
Yes
HAUGEM2-AO
192
145
Yes
Yes
PGOALS-g3
180
80
Yos
Yes
HadGEM2
GHDL-CM
Yes
Yes
HadGEM2-ES
192
145
Yes
Yes
GFDL
288
180
Yes
Yes
INMCM
180
120
Yes
Yes
GISS-E2-2-G
144
90
Yes
Yes
Yes
Yes
ICON ESM-LR*
N/A
N/A
Yes
Yes
IPSL-CMSA-MR
144
143
Yes
Yes
IITM-FSM
192
94
Yes
Yes
IPSL-CMSB-LR
Yes
Yes
INM-CM4-8
180
120
Yos
Yes
MIROC-EST ESM
128
64
Yes
Yes
INM-CMS-0
180
120
Yes
Yes
MIROC.ESM-CO
Yes
PPSL-CM5A2-IIH
Yes
Yes
MIROC4h
640
320
Yes
Yes
PPSI-CMMA-1A
144
143
Yes
Yes
MIROCS
256
128
Yes
Yes
LISL-CMMA-LR-INCA
144
143
No
Yes
MPLESM-LR
192
Yes
Yes
KACE I.0.G
192
144
Yes
Yes
MPI-ESM-MR
192
96
Yes
Yes
KIOST-ESM
192
96
Yes
Yes
MPL-ESM-P
MIROC6
256
128
Yos
Yos
MRI CGCM3
320
160
Yes
Yes
MPL-ESM-1-2-HAM
192
96
Yes
Yes
MRLESMI
Yes
ESMI
Yes
NorESMI-M -M
144
96
Yes
Yos
MPL-ESMI-2-LR
192
96
Yes
Yes
MRI-ESM2
320
160
Yes
Yes
NESMS
192
Yes
Yes
NorCPMi
144
96
Yes
Yes
NoFESME-LM
144
Yos
Yes
NorESM2-N MM
288
192
Yes
Yes
192
Yes
TaIES
288
192
Yes
Yes
Table 1: Details for each observed/reanalysis data product and CMIP model output. The columns
give the model name, longitude and latitude resolution, and availability of (2m) surface temperature (TAS) and total precipitation (PR) for each dataset. All datasets were obtained on rectangular grid except ICON-ESM-L.R which was obtained on an icosahedral grid with 10,242 total cells.
CMIP5 and CMIP6 outputs:
ERA5 hourly data on single levels from 1940 present: http:::/cd..cliaatee....
NCEP/DOE 9r/2pp/p//aaaass//m////caaaaaee2
II:
html
GPCP Degree Daily Precipitation https: www..nc.e..... nww.. http:p//wwww.nn...nnaaaaeeooooor
17

---

D Full SCWD Rankings and Metric Comparison
In the following pages, we provide tables detailing the climate model rankings generated by SCWD and baseline methods from both the climate science and ML literature. First, we provide details on the implementation of the baseline metrics. See Section 4.4 for a discussion of these results.
D.1 Baseline Climate Model Evaluation Metrics
Past metrics for climate model validation include the root mean square error (RMSE) and mean absolute error (MAE) applied the climatologies, or long term means of the data [Gleckler et al. 2016). To implement these approaches, we first require regridding the data to a common dimension. So, regrid the data common resolution of 120 60 using the remapnn function from the Climate Data Operator (CDO) command line tools. This resolution was chosen to match the grid of locations over which the slices were computed for SCWD. Afterwards, we compute long term means at each location, ignoring leap days, to produce 365 values representing the climatologies for each day of the year. RMSE and MAE are computed between the climatologies for the reference dataset and each climate model at each location. The final rankings are determined by the area-weighted global mean over RMSE/MAF values at each location. Models marked as "NA" operate on 360-day calendar, making it challenging to estimate 365-day climatology to match the reference datasets.
D.2 Baseline Wasserstein Distance Metrics
We compare our approach against existing Wasserstein- based metries. Although these metrics have never been used for model validation, they can still provide sense of what our method offers over the naive application of existing tools. We include ordinary Wasserstein distance (WD), Sliced Wasserstein Distance (SWD), and attempted to include Convolution SWD (CSWD). For these methods to work, we first have regrid each model to a common set of grid points. However, unlike the baseline RMSE and MAE-based approaches described in the previous paragraph, none of the proposed Wasserstein methods account for area weighting geographic data. So, we again regrid the climate models and reference datasets common resolution but now using Icosahedral grid of 642 locations, which provides near-uniform spacing on the sphere to alleviate concerns about area weighting. The reduced dimensionality is not ideal for evaluating climate models, but serves make the computation of the multivariate WD and sliced WD more feasible given the costs already incurred from regridding. Because are using Icosahedral grid to account for area weighting, unable to compute convolution sliced WD, which is designed for square grids.
18

---

CMIP6 Surface Temperature Rankings
SCMD (3000km)
SCWDD
WO
SWD
(500km)
(2500km)
(climeralooy
(clmetalocyl
1.197
0.923
ANI-CM-2-1--R-
1.382
1.277
1.044
0.308
1.518
1.309
86.069
1.214
HPI-E5M1-2--1R
1.404
1.294
1.039
0.389
1.582
1.368
85.374
1.322
CESM2-WACCK
1.419
1.311
1.074
0.277
1.495
1.288
85.684
1.276
KorESM2
1.363
1.262
86.715
1.476
1.373
1.149
0.449
1.515
1.307
85.850
1.321
GFOL-E5MC
1.544
1.443
1.196
0.377
1.496
1.263
87.304
1.289
MP -P--E5M2-2-LR
1,600
1,450
1.117
0.070
1.706
1.483
89.824
1.539
CNCC-CM2-1IR--
1.561
1,459
1.228
1.714
1.506
89.084
1.409
-2-1-18-
1.150
91.563
MRI-E 5M2-0-
1.581
1.467
1.198
0.161
1.613
1.374
92.149
1.510
CESM2-NACCH-FVV
1,679
1.526
1.246
0.452
1.713
1.489
91.093
1.641
CESM2-FV2-
1.734
1.584
1.296
0.331
1.729
1.501
92.046
1.658
NorESW2-NN
93,452
1,816
1.652
1.365
0.267
1,922
1,696
93.122
1.621
TDESS1
1.797
1,680
1.443
0.141
1.601
1.381
91.811
1.606
E35M-1-0-
1.797
1.689
1.456
0.182
1.773
1.522
93.169
1.558
CMCC-E5M2 1.302
1.694
1.455
0.592
1.645
1.396
90.681
1.548
1.458
94.189
GFD -CM4
1,806
1.725
1.552
1,008
1.659
1.430
89.509
1.379
INN CMS 1.937
1,736
1.364
0.435
1.977
1.724
98.334
1.832
IPSL PSL--CMAZ--NCA 1.895
1.752
1.450
0,446
1.992
1.718
94.499
1.676
KACE-1-0-G 1,896
1.791
1.559
0.202
1.721
1.463
91.996
1.677
EPI-ESM-1-2-HAN
1.949
1.814
1.521
0.174
1.897
1.668
93.734
1.661
ACCESS-CM2- 1,964
1.840
1.597
0.092
1.868
1.638
94.319
1.685
EC- 1.943
1.877
1.737
0,475
1.807
1.596
93.386
1.695
96.014
CanE
2.104
1.913
1.567
0.265
2,123
1.877
102.338
2.106
ECHEam3 AorCham
2.016
1.946
1.795
0.207
1.868
1.656
94, 194
1.738
Eartarths-ro LB
2.051
1.971
1,817
0.150
1.993
1.768
96.038
1.861
E35M-2-0- 2.072
1.973
1.775
0.541
1.999
1.748
96.460
1.620
E3SM-2-0-NARRN
2.087
1.997
1.815
0.658
1.990
1.755
96.134
1.740
FGOAL5-B-L-- 2.119
2.004
1.751
0.614
1.886
1.635
96.587
1.703
INM MMM-MM4-8 2.236
2.035
1.629
0.252
2,086
1.821
101 024
1.917
EC-Earth3 2.156
2.087
1.939
0.238
1.955
1.738
97,007
1.812
EC-Eant3-CC 2.254
2. 1.192
2.054
0.842
1.972
1.761
96.599
1.810
KIDST-E5N 2.430
2, 3.8
2.139
0.631
2.178
1.927
102 481
2.065
ICON ICON-ESM-LR- 2.519
2. 2.396
2.010
0.489
2.416
2.095
103 412
2,280
NortPM1 2.638
2.492
2.185
1.218
2.301
2.025
108 400
2.329
FGONLS-93-
2.738
2.603
2.325
0.695
2.180
1.918
108
2.479
MIRDC6 3.006
2.878
2.611
1,531
2.360
2.103
108 782
2.423
JIM--ESM 2.996
2.905
2.709
0.692
2.496
2.258
110 685
2.525
3.726
3.644
3.432
2,802
2.533
122 535
2,851
Table 2: CMIP6 model rankings for (2m) surface temperature based similarity the ERA5 Reanalysis. Distances are calculated using our proposed spherical convolutional WD (SCWD) as well the global mean-based WD (GMWD), RMSE and MAE (both computed on the climatologies), WD and Sliced WD (both computed on data regridded to an icosahedral grid). For the SCWD calculations, three different range parameters are chosen for the Wendland kernel: 500km, 1000km (our proposed choice), and 2500km. Color fill is unique to each column in the table, and is calculated using ranks.
19

---

Total Precipitation Rankings
SCWDD (500km)
SCWDD (1000km)
SCWD
RMSE (clmatolocy)
IcImataloqyi
licosahedrali
NorESN2-MM
1.335
0.762
0.167
2.579
1.845
182.057
0.766
EC-harthh
2.028
1.589
0.968
0.236
2.608
1.906
175
1.136
CESM2
2.425
1.597
0.891
0.263
2.678
1.930
184 416
0.945
EC-E BC-Eam3-CC-
2.043
1.613
0.998
0.275
2.660
1.945
177 141
1.159
2.399
1.640
0.889
0.246
2.648
1.909
182
0.903
EC-Larbh--eg-
2.083
1.646
1.009
0.249
2.656
1.936
176.401
1.183
EC-Earth3 Veg-LR
1.902
172.888
EC-Eardaa-Arc
2.592
1.683
0.946
2,840
2,0A8
362
1.294
NorESM2-LM-
2.137
1.686
0.972
0.182
2.674
1.916
180.
1.051
CESM2-FV2
2.134
1.690
1.014
0.265
2.648
1.910
176.198 164
1.129
TaE5M1
2.593
1.723
1.028
0.384
2.652
1.948
180. 100.010
0.995
W22--5
2.453
1.749
1.044
0.304
2.767
2,014
286.046
1.148
2.199
1.750
1.037
0.262
2.696
1.948
176.087
1.249
KAGE-1-0-6
ESM2
1.787
1.073
0,402
1,969
179
1.058
ACCESS-CM2-
2.816
1.796
1.062
0.447
3.059
2.179
209 684
2.051
E35M-1-0- 2.432
1,800
1.055
0.359
2,715
1.976
178 568
1.154
AWI-CN-2-1-HH
2.327
1.802
1.074
0.234
2.769
2.009
185 090
1.288
CMCC CN2-SR5
2.578
1.318
1.115
0.412
2.710
1.986
179 119.817
1.104
CMCC- CRCC-EM2-HR4
2.00
1.818
1.112
0.291
2.827
2.015
186
1.103
GFDI-CM4
199.477
MPI-LSM1-2-LLR
E35M-2-0
3.042
1.861
0.967
0.276
2.736
1.942
184
1.033
E35W-2-D-NARRM
3.120
1.873
0.953
0.276
2.732
1.943
187
1.030
(PSL -L--CNAA-LR
2.845
1.895
1.143
0.345
2,948
2.098
203 981
1.700
SLL-CHAA-LR---- -NCCA
2.926
1.952
1.163
0.349
2,963
2.101
204 900
1.800
JITM-ESM
2.616
1.964
1.177
0.240
2.931
2.115
188.449 449
1.209
NESM3 2.446
1.967
1.196
0.161
2.937
2.073
184.775
1.083
GEOL-E 5M44
2,040
ACCESS-ESS -E5M1-5
2.827
1.979
1.158
0.546
2.965
2,152
197.594
1.538
NorCPM1 2.469
1,960
1,233
0.060
2.701
1.935
172
1,530
SAMO UNICON 3.064
1.984
L,066
0.332
2.797
2,015
187 960
1.094
INH W4-0 2.658
2.027
1333
0.368
2,790
2,061
180 967
1.307
Can SMS 3,303
2,088
1.123
0.209
2.973
2.090
198 500
1.557
MPI-ESM-1 PPI-ESM-1-2HHAT
2.681
2.144
1,294
0.238
2.902
2.100
181 181.616 616
1.552
FGOALS
2.157
1.136
0.097
2,208
MMSTTEEMM
2.190
1.239
2.757
1.983
179
GISS 1E55-E2::0 -2222-G 944
2.201
1.366
0.152
2.857
2,092
180 662
1.519
NNI-ESH --1-1-LR- 2.745
2.214
1.349
0.160
2,800
2.035
176.
1.739
ICON-ESI
2.849
2.295
1.413
0.118
3.018
2.153
185 359
1.428
PSL-EMEAZ-NCA
2.996
2.321
1. 1,380
0.143
2.796
2,039
073.032
1.766
KCEP 3.738
2.490
1.289
0.361
2.758
1.943
618
2:533
-G0ALS-O-L- 4.614
2.752
1.264
0.191
3.155
2.199
226 689
2.839
BCC-ESMI
Table 3: CMIP6 model rankings for total precipitation based similarity the GPCP observations. Distances are calculated using our proposed spherical convolutional WD (SCWD) as well as the global mean-based WD (GMWD), RMSE and MAE (both computed on the climatologies), WD and Sliced WD (both computed on data regridded to an icosahedral grid). For the SCWD calculations, three different range parameters are chosen for the Wendland kernel: 500km, 1000km (our proposed choice), and 2500km. Color fill is unique to each column in the table, and is calculated using ranks.
20

---

CMIP5 Surface Temperature Rankings
SCWD (500km)
SCWD 11000km]
SCWD (2500km)
GMWD
RMSE (climeolooyy
Iclimatooogy/
MAE
WD (ccsahedral
licosahedral)
SWD
NCEP Reanalysk
1.412
1.197
0.923
0.308
1.456
1.341
63.307
1.455
WPI-ESM-P 1.491
1.347
1.039
0.157
1.671
1.440
88.946
1.505
MFI-ESM-NR-
MPI-ESM
1.530
1.390
1.089
0.132
1.687
1.458
89.557
1.502
MIROC4h-
1.620
1.516
1.292
0.648
1.638
1.408
85.834
1.417
1.807
1.612
1.238
0.209
1.826
1.593
93.936
1.735
NorESMI-N
1.816
1.634
1.310
0.485
1.855
1.623
94.197
1.775
ACCESSI-0-
1.848
1.683
1.385
0.055
1.780
1.531
94.300
1.633
1.833
1.719
1.448
0.307
1.869
1.637
91.688
1.643
GFDI-CM3-
1.877
1.745
1.446
0.371
1,884
1.634
92.071
1.626
HadGEM2-1C
1.977
1.815
1.516
0.305
NA
NA
96.722
1.692
1.921
1.841
1.650
0.862
1.994
1.777
93.500
1.720
HadGEM2-ES-
2.014
1.857
1.582
0.336
NA
NA
97.603
1.845
CMCC
1.865
1.999
1.767
97.743
1.813
VIROC-ESM-CHES -ESM-CHEN 2.118
1,870
1.528
0.322
2.162
1.902
98.743
2.045
CNRH HCR--CC5--
2.018
1.891
1.618
0.30.
1.986
1.720
97.237
1,628
MIROC-ESH EES
1.896
0.276
2.170
1.905
98.937
2,054
2.203
2.049
1.763
0.192
2.189
1.909
100.727
1.912
SM2 5M2- 2.267
2.057
1.703
0.265
2.188
1.914
104.647
2.213
IPSL-CNSA-LR
2.187
2.084
848
0.108
2.088
1.808
98.953
1.946
CNCC-CK- 2.197
2.100
866
0.273
1.981
1.746
99.122
1.942
MRI-COCM3-
DL- 2.309
2.155
1.868
0.348
2,273
1.987
101.759
1.975
HDGGEM2-EC- 2.350
2.205
1.950
0.628
NA
NA
102.304
2.205
MIRDES
0,855
96.780
CNCC-CESN- 2.614
2.389
2.088
0.147
2,549
2,280
109.000
2.346
HadCM3+
2.575
2.402
2.111
0.602
NA
NA
111.246
2.404
111.153
FEDALS -2-- 2.877
2.662
2.253
1.034
2.558
2.263
114.429
2.526
CSIRO MK3 6-0-
2.961
2.763
2.393
0.977
2.445
2.127
115.992
2.612
FGOALS-92-
2,864
IPSL-C53-LLR- 3.361
3,238
2.929
0.626
2.741
2.421
117 348
2,731
Table 4: CMIP5 model rankings for (2m) surface temperature based similarity to the ERAS Reanalysis. Distances are calculated using our proposed spherical convolutional WD (SCWD) as well the global mean-based WD (GMWD). RMSE and MAE (both computed the climatologies), WD and Sliced WD (both computed on data regridded to an icosahedral grid). For the SCWD calculations, three different range parameters chosen for the Wendland kernel: 500km, 1000km (our proposed choice), and 2500km. Color fill is unique to each column in the table, and is calculated using ranks.
21

---

CMIP5 Total Precipitation Rankings
SCWD
SCWD
RMSE
MAE
WD
SWD
ACCESS1-0-
2. 2.257
1.665
1.033
0.410
2,887
2.099
196.045
1.417
HadGEM2-ES-
2.274
1,668
1.039
0.394
NA
NA
196.479
1.339
1.274
MaOGEM2 AO
2.387
1.733
1.067
0.428
NA
NA
199.118
1.446
MIROC5
2.380
1.752
1.082
0.533
2.772
2.038
185.439
1.132
2,809
1.817
1.080
0.278
2,937
2.053
208.359
1.765
EC-EARTH
2.334
1.859
1.156
0.199
2.581
1.879
171.102
1.462
ACCESSI-3-
2.642
1.913
1.146
0.487
2.930
2.138
193.608
1.500
92-
2.774
1913
1.109
0.132
2,969
2.104
202 659
1.566
NoTESMI-N
2.435
1.937
1.212
0.147
2.725
1.959
172.984
1.463
ConE5M2-
2.767
1.989
1.185
0.107
2.719
1.961
179.564
1.251
2.498
2,002
0.251
2.893
192.831
1.194
MPI-ESM-LRR-
2.527
2.009
1.169
0.275
2,896
2.091
190.268
1.292
GFD- 2.695
2.771
2,006
177.580
1.441
HedCM3 2.550
2.027
1.266
0.234
NA
NA
176.560
1.744
CanCMe 2.855
2.028
1.188
0.111
2.713
1.954
179.501
1.257
CNCC-CN- 3.011
1.156
0.227
3.057
2.173
206.521
1.803
CNCC-CESN 2.685
2.129
1.319
0.204
2.798
2.045
176.287
1.653
MFI-LSM-NR-- 2.652
2.129
1.253
0.331
2,956
2.127
193.569
1.328
MPI-ESM-F-
190.158
IPSL-CHSA
2.914
2.189
1.315
0.144
2.787
2.030
177.914
1.576
MIROC-ESN- 2.727
2.211
1.399
0.134
2.718
1.965
170.350
1.896
MIROG-ESMAL MMOC-ESM-CHEM
170.061
IPSL-CNSA-LR- 2.882
2.230
1.333
0.080
2.709
1.968
169.559
1.867
GFDL-ESM2G- 2.971
2.242
1.353
0.309
2.765
2 2.009
176.612
1.568
NRI-COCMS
2.204
214.016
2,192
GFOL-ESM2N- 3.280
2.279
1.232
0.317
2.772
2.005
178.057
1.386
NRI-ESMI- 3.698
2.288
1.243
0.257
3.164
2,210
216.396
2.297
-11R0-W12--69-
182.964
FEDALS -22- 3.199
2.367
1.342
0.086
2,946
2.109
194.122
1.432
CNRM CMS- 3.764
2.412
1.218
0.384
2,828
2.039
189.153
1.254
Immomé 3.151
2.570
1.621
0.481
2.689
2.069
168.
2.351
DSL-CMMB-LR
4.504
2.630
1.168
0.145
3.228
2.196
219.657
2.561
Table 5: CMIP5 model rankings for total precipitation based similarity the GPCP observations. Distances are calculated using our proposed spherical convolutional WD (SCWD) as well as the global mean-based WD (GMWD), RMSE and MAE (both computed the climatologies), WD and Sliced WD (both computed on data regridded to an icosahedral grid). For the SCWD calculations, three different range parameters chosen for the Wendland kernel: 500km, 1000km (our proposed choice), and 2500km. Color fill is unique to each column in the table, and is calculated using ranks.
22

---

E
Investigation of Differences Between Rankings
The impacts of climate change on temperature are not limited to changes in the local means and may include changes in variance or other moments. This phenomenon is documented in Hansen et al. [2012), where extreme tail behavior in local temperature anomalies are shown occur alongside warming in the mean climate state. We construct the following synthetic experiment to assess the ability of each climate model evaluation method (SCWD, GMWD, RMSE, and MAE) to detect changes in regional means and variances simultaneously. Based on these findings, we re-examine the CMIP6 surface temperature rankings Table and show that RSME/MAE rank SAMO-UNICON artificially high due to their inability to detect variance changes in the temperature anomalies.
E.1 Synthetic Experiment
To generate realistic synthetic example, we start with the ERA5 data and linearly transform to modify its mean and variance structure. Specifically, let Y be the ERA5 reanalysis surface temperature data. can be separated into the climatology, C, which the temporal mean climate state for each location, and the anomalies A = C. We compare Y C + A to modified datasets of the form:
MM, M
which are alternate versions of ERA5 with mean shift of M and anomaly scale factor of s. We consider s 1.1, 8 1.3, and 1.5, which respectively represent 10%, 30%, and 50% increases in the variance of the anomalies. We also consider three cases of M which vary in the Northern and Southern Hemispheres (NH and SH), including (+0.5K, +0K) in NH/SH, (+1K, -0.5K) in NH/SH, and (+1.5,, - 1K) in NH/SH. These chosen provide the same global mean change with different interhemispheric temperature asymmetries, which are an important climate feature Friedman et al. 2013). For each value of M and we compare YM, to Y using all four metrics.
SOWD
GMWD
RMSE
MAE
Anomaly scale faclor (s)
1.296
1.476
1.783
0.255
0.255
0.255
0.25
0.75
1.25
0.25
0.75
1.25
1.3
0.828
1.088
1.478
0.252
0.252
0.252
0.25
0.75
1.25
0.25
0.75
1.25
1.1
0.433
0.829
1.299
0.25
0.25
0.25
0.25
0.75
1.25
0.25
0.75
1.25
+0.5 NH
NH
+1.5 NH
+0.5 NH
NH
+1.5 NH
-0.5 NH
+1 NH
+1.5 NH
+0.5 NH
+1 NH
+1.5 NH
-0SH
-0.5
-0.SH
-1.SH
Mean shift NH/SH (M)
Figure 5: Results from the simulation comparing the original ERA5 data synthetic modifications. Each panel gives the results for a different distance function. The y axis represents the anomaly scale parameter, 8, and the axis represents the M parameter which controls mean shift in the northern/southern hemispheres, For each method, the distance is provided from the original ERA5 data to the modified ERA5 data with each of the nine combinations of M and The color fill is determined by the rankings within each method, with light yellow representing low ranking and dark red representing high ranking.
The results are shown Figure Because each M has the same global mean, GMWD does not change with M, although it does (very slightly) increase for higher values of Conversely, scales only the variance of the anomalics and does not impact the climatology, so RMSE/MAE are only able to detect changes in M, not Only SCWD detects changes in both the local means and the variance of the anomalies in this example.
F.2 Investigating Differences in CMIP6 Rankings
The experiment demonstrates that SCWD is able to distinguish additional sources of variability beyond just differences in the climatological mean state. To understand how this ability impacts our climate model rankings, we investigate SAMO-UNICON, which ranks poorly using SCWD relative
23

---

to RMSE/MAE. Looking at the SCWD map in the supplemental material. SAMO-UNICON exhibits the highest local WD values in regions such the coast of Antarctica. We average over the region where the local WD is the highest and compare the climatologies and anomalies for FRAS and SAMO-UNICON F
Anomalies
Climatologies
0.15
Temperature (K)
-10
Donsity
0.10
ERA5 Reanalysis
SAMD-UNICON
0.05
20
0.00
-10
20
100
200
800
Temperature Anomalies (K)
Day of the year
SDIERA5 anomalles)) 3.52
SD(SAM) anomalles)
Figure 6: Comparison of anomalies and climatologies in the region where SAMO-UNICON has the largest local WD values (greater than 7.91). ERAS is shown in blue and SAMO-UNICON is shown in red. The left plot a density plot showing the distribution of the anomalies, and the right plot is time series plot showing the climatologies starting on January 1t and ending on December 31st.
Relative ERA5, we large difference in the climatology for SAMO-UNICON, which will impact both SCWD and RMSE/MAF. However, there is also a large increase in the variance of the anomalies in UNICON,, which only measured by SCWD. We have demonstrated that this region has both high local WD values and large differences that are undetected by typical comparisons of the climatologies. So, this region is likely responsible part for the harsher rankings of SAMO-UNICON by SCWD compared to RMSE/MAE.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract states that introduce new similarity measure and apply it to evaluate CMIP climate models. These claims are addressed in S.c
Abstract also states that found modest improvements from CMIP5 to CMIP6, which is addressed in Section 4.3 Introduction states that compared to a previous WD-based similarity measure for climate model data, our proposed method accounts for spatial features (addressed in Section 3.2 and provides a more robust evaluation (addressed in Section 4.4 from an applied and experimental perspective).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are addressed in the second paragraph of Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
25

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
Theoretical results are stated in Theorem 3.3 and proven in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification; Full theoretical details on our method are included in Section 3.2 the algorithm is provided in Appendix B and the code is provided in the supplement. All data are publicly available, with full details Appendix C For the synthetic experiment, all details on generating the data are included in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
26

---

(d)
The authors should provide scripts to reproduce all experimental results for the new reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Full theoretical details on our method are included in Section 3.2 the algorithm is provided in Appendix [ and the code to reproduce the SCWD calculations for CMIP5 and CMIP6 to the reference datasets is provided in the supplemental material. All data are publicly available, with full details in Appendix [ Note that the data cannot be directly included in the supplement due to their large size (around 2TB before preprocessing).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//mips.ccppubic//uui wwtp://guuiieeceeeo Sttggudddde/oeeemmiiiiieeee Sat////ooSSSSeiaeeeeee httpee////eeeeebbbssssooooollolo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: For the main analysis, specification of the kernel function is in Section 3.2 and specification the range parameter and reference datasets is in Section 4.1 For the additional synthetic experiment, all details are provided in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
27

---

Answer: [Yes]
Justification: For our initial CMIP6 climate model rankings, there are no error bars because we compute pairwise distances between whole climate model runs, of which we do not have replicates to measure the full variability the pairwise distances. Because our slicing mechanism is non-random, there is Monte Carlo enror to report. We do, however, include other appropriate information about the significance of these rankings. In particular, we study the sensitivity of our results to the kernel range parameter in Section 44 For the claims regarding the progression from CMIP5 to CMIP6, variability information is included in the form of boxplots in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.: Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: This is addressed in Appendix B The overall computing requirement low, with the caveat of requiring large amount of storage space for the climate model and reanalysis datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 'tttt make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:://aaaarp.c.cpppaaa//cccaaa
Answer: [Yes]
Justification: No human subjects were involved in this research and all analyses were based on publicly available datasets. All required details were documented and algorithms and code are provided.
28

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Climate models are the primary tool for climate prediction and projection. However, deficiencies persist among climate models despite decades improvement. Our paper offers succinct and informative proposal of a new method for evaluating climate models against observational and reanalysis data, both on the global and regional scales. provides valuable guidance for climate model improvement and assist in selecting more reliable models to reduce uncertainty in climate projections, which is crucial for climate mitigation and adaptation efforts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks. Our method is built to evaluate and explain discrepancies climate model distributions (in terms of specific regions), which could aid in responsible use of climate model projections.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
29

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We credit the creators of the climate model and reanalysis datasets in Appendix C Calong with citations throughout the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.ppaerssii paperswi thcode. com datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code is provided supplement, with additional theoretical documen- tation in Section and technical documentation in Section B
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing or human subjects were involved in this research.
Guidelines:
30

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing human subjects involved in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "801x3ouEgw", "reviewer": "Reviewer_V9JK", "summary": "Developing metrics for comparison between high dimensional, multivariate climate models is an important and open area of study . Vissio et al (2020) proposed the use of the Wasserstein distance to quantify the similarity between climate models. However this approach involves spatial averaging, and therefore significant information is lost. This work introduces functional sliced Wasserstein distance in spherical coordinates, which provides a computationally tractable Wasserstein metric without spatial averaging. The method is demonstrated by comparisons between CMIP model data and ERA reanalysis data.", "questions": "I am wondering whether the method could be demonstrated on a simpler toy problem where the ground truth is better established, and the complexity and high dimensionality of the system is retained, before application to a reanalysis-vs-model comparison. As discussed in section 4.1, reanalsyis comparisons are still subject to discrepencies from other factors such as model physics.", "limitations": "Limitations of both the method (lines 107-119 and 356-359) and the results (section 4.1) are discussed.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The work is timely and presents a strong contribution to the field of climate science. The presentation is excellent - motivation and connections to previous work are clearly established. The new method is clearly explained, and demonstrated in a sensible set of experiments. The capability of the slicing kernel to focus on specific local regions provides a tremendous amount of flexibility to the metric, which will have utility in a wide range of important applications. Comparisons to other standard metrics are also made, and in cases where there are discrepancies with baselines, these discrepancies are discussed. Finally the authors speculate on potential applications beyond climate science.", "weaknesses": "The paper has no obvious weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and suggestions! In response to your comment:\n\n>**I am wondering whether the method could be demonstrated on a simpler toy problem where the ground truth is better established, and the complexity and high dimensionality of the system is retained, before application to a reanalysis-vs-model comparison**\n\nWe shared your thoughts on this subject. While we did not have space for our full exploration in the main text, Appendix E.1 provides a synthetic data experiment that we believe addresses your concerns. We retain the high complexity of climate observations by making modifications to the mean trends and anomalies of the ERA5 data, and compare performance between metrics.\n\nIn that experiment, we found that SCWD could detect changes in both the climatological means as well as the variance of the anomalies. The global mean-based Wasserstein distance failed to capture differences when the climatological mean had compensating errors in space. Baseline metrics such as RMSE/MAE, which must be computed using long-term climatological means to avoid issues with time misalignment of climate models and observations, were unable to distinguish changes in the scale of the anomalies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vpYqCQ34Ib", "reviewer": "Reviewer_UJm5", "summary": "The paper introduces a new method for validating climate models by comparing their outputs to reanalysis data. The proposed method, Spherical Convolutional Wasserstein Distance (SCWD), accounts for spatial variability and local differences in the distribution of climate variables. The authors apply SCWD to evaluate historical model outputs from the Coupled Model Intercomparison Project (CMIP) phases 5 and 6, demonstrating modest improvements in the phase 6 models in producing realistic climatologies. The technical claims are well-supported by thorough theoretical and empirical analyses. The authors provide a robust mathematical foundation for SCWD and demonstrate its effectiveness in capturing spatial variability through extensive experiments. The paper is well-structured and written, with detailed explanations of the methodology and comprehensive evaluation results. However, some sections could benefit from additional clarity, particularly the mathematical derivations and kernel selection process.", "questions": "Could the authors provide more details on the selection process and theoretical justification for the specific kernel function used in SCWD?\nHave the authors considered applying SCWD to other climate variables or different temporal resolutions to evaluate its generalizability?", "limitations": "The authors adequately address the limitations of their work, including the need for device-aware optimizations and the challenges in parameter selection. They also acknowledge the potential for further improvements and generalization of SCWD, providing constructive suggestions for future research.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Originality: The introduction of SCWD as a new metric for climate model validation is innovative and addresses the limitations of existing methods.\nQuality: The methodology is rigorously developed and supported by extensive experimental validation using real-world climate data.\nClarity: The paper provides clear explanations of the SCWD methodology, supported by visualizations and detailed examples.\nSignificance: The proposed method has significant implications for improving climate model validation, which is crucial for accurate climate projections and policy-making.", "weaknesses": "Mathematical Derivations: Some mathematical derivations, particularly those related to the convolution slicer and kernel functions, could be explained more clearly to enhance understanding.\nGeneralization: While the method is well-validated on historical climate data, additional experiments on different climate variables and temporal resolutions would strengthen the generalizability of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions. In response to your comments on including additional climate variables/temporal resolutions, i.e.\n\n>**While the method is well-validated on historical climate data, additional experiments on different climate variables and temporal resolutions would strengthen the generalizability of the findings.**\n\nWe certainly hope our method will see wider use in the climate community with additional variables/resolutions. We have already seen evidence of wider interest in SCWD. For example, we were approached by a climate modeling center interested in SCWD and are working with them to apply this method to other datasets. The work is ongoing, but so far SCWD has proven useful for assessing long-term (1-10 year) daily climate forecasts and for identifying problematic regions in those forecasts. One key focus of that work is considering the performance over different time scales and different seasons.\n\nAdditionally, in regard to your comments on providing additional clarifications, i.e.\n\n>**Some mathematical derivations, particularly those related to the convolution slicer and kernel functions, could be explained more clearly to enhance understanding.**\n\nWe agree that some additional clarity on our theoretical details would help convey our message. We expanded the discussions of the convolution slicer and kernel selection as follows:\n\nFor the **convolution slicer**,  we edited the paragraph starting line 144 as follows: \n\n\"To create a valid functional sliced WD, we construct pushforward measures based on the convolution slicer $c_s(f)$. To satisfy the definition of a pushfoward measure, we must show that $c_s(f)$ is a Borel measurable function from $L^2(\\mathcal S) \\rightarrow \\mathbb{R}$. By continuity of $k$, when location $s\\in\\mathcal S$ is fixed, $k(s,u)$ is a continuous function from $u\\in\\mathcal{S}\\rightarrow\\mathbb{R}$. Because $\\mathcal{S}$ is compact, $k(s,u)$ is a continuous function on a compact set and is thus bounded and $L^2$-integrable. It follows that the convolution slicer $c_s(f)$ is an integral of the product of two functions $f,k \\in L^2(\\mathcal S)$, so by Hölder's inequality, $c_s(f)$ is a bounded linear operator from $L^2(\\mathcal{S})\\rightarrow\\mathbb{R}$. Stein et al. [2011] states that bounded linear operators are also continuous, so $c_s(f)$ is a continuous linear operator and thus Borel measurable. So, for any measure $P\\in\\mathcal{P}(L^2(\\mathcal{S}))$, the pushforward $c_s\\\\#P$ is a valid measure in $\\mathcal{P}(\\mathbb{R})$. Therefore, we can define a functional sliced WD between distributions in $\\mathcal{P}(L^2(\\mathcal{S}))$ as follows:\"\n\nWe hope this provides some more clarity on the formulation and properties of the convolution slicer.\n\nFor the **choice of kernel function**, we considered both the Kent distribution function and Wendland function because both are positive definite on the sphere and are popular choices in spatial statistics. Of the two, we prefer the Wendland function due to its compactness, which enables sparse matrix computations in our analysis. To guarantee positive definiteness, we need the kernel to be smaller than the radius of the sphere. Beyond that, the choice is left to the user, and we chose 1,000km to balance the need for fine-scale spatial perspectives and the available spatial resolution of the data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5GbqXQng3w", "reviewer": "Reviewer_qRTk", "summary": "The paper defines SCWD as a special case of their proposal for functional sliced WD which it uses to compare CMIP members against reanalysis data. Additionally, with this new distance, it analyses the effectiveness of CMIP phase 6 over phase 5", "questions": "- L258 suggests that the resolutions between multiple outputs isn't consistent throughout? Have they been readjusted to similar spatial dimensions?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents its ideas succinctly\n- Motivates the need to find a good distance measure for comparing distributions of functions defined over $L^2(S^2)$\n- Provides a smooth transition from sliced WD to its functional variant\n- Method seems robust to kernel parameter\n- Allows easy visualization of the differences and helps isolate regions where the fields diverge\n- Validation experiments were sufficiently extensive", "weaknesses": "- Would have been interesting to see VAEs as a baseline as proposed by [1]\n\n[1] Mooers, G., Pritchard, M., Beucler, T., Srivastava, P., Mangipudi, H., Peng, L., ... & Mandt, S. (2023). Comparing storm resolving models and climates via unsupervised machine learning. Scientific Reports, 13(1), 22365.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, we particularly appreciate your attention to our climate application!\n\nIn response to your comment:\n\n>**Would have been interesting to see VAEs as a baseline as proposed by** (Mooers et al.)\n\nWe’ve added this paper to our literature review in the introduction. We believe this could make for an interesting future comparison, as both methods can be used to compare climate models and learn patterns of spatial variability. In our analysis, we focus on a single layer of convolutions and our results highlight regions where the local climate distribution of each model disagrees with observations. On the other hand, the VAE method identifies broader spatial patterns of variability that are shared between clusters of models. Both types of differences are interesting to climate scientists. For our application, we are curious about the computational speed of the VAE method. One of the advantages of SCWD is the computational speed - for example, for a single range parameter value, SCWD calculations were performed for all of our models of interest in less than 24 hours on a personal computer (the full analysis took longer as we considered multiple range values and other metrics). We are also curious about the amount of training data required to train the VAE. We do not have time to add this to our paper, but we are also very interested in this comparison and will investigate further.\n\nIn regards to your comment: \n\n>**L258 suggests that the resolutions between multiple outputs isn't consistent throughout? Have they been readjusted to similar spatial dimensions?**\n\nIndeed, each model is obtained at a different spatial resolution. One of the challenges with implementing our method is that the underlying functional convolutions are continuous while the data are discrete. In our code implementation (detailed in Appendix B), we handle this by first performing a (computationally cheap) one nearest neighbor (1NN) regridding to a high resolution grid. Typical climate model validation analyses use more sophisticated (and expensive) regridding techniques to handle spatial interpolation, however, interpolation in SCWD is handled as a natural part of the metric via the kernel convolution slicing. So, the only reason we apply 1NN upsampling is to enable a high-resolution approximation of the underlying continuous kernel. The functional kernel represents a circular region on the Earth’s surface, so the high-resolution grid allows, for example, for partial weighting of the (rectangular) grid cells on the edge of the kernel radius. Intuitively, our convolution method is able to \"slice\" circular regions out of the original rectangular model grids.\n\nAlternatively, an exact approach with no regridding is possible by taking the integral of the kernel function over each grid cell to obtain weights for each pixel in the model output grid. The slices would be obtained by multiplying those weights by the temperature/precipitation values associated with each pixel and taking the sum. However, this would be much more computationally expensive because it would require spatial geometry operations to be performed for each different model output grid. Additionally, even though the results from the slicing process would be exact, the analysis would still be limited by the available spatial resolution of the model outputs!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1uPeGbZXvZ", "reviewer": "Reviewer_vRQU", "summary": "The paper proposes a new distance measure based on Wasserstein distance for data on a sphere. The work applies the methodology to climate model data, with primary focus on ranking climate models based on their agreement with reanalysis data.", "questions": "From what I understand, the data used in daily averages of temperature and precipitation for a historic period. Is the distance calculated on the daily basis? If so, how do you take into account the fact that climate models are known to be poorly temporally aligned with observation data? Would it make more sense to aggregate the data temporally? If so, do you have any thoughts on optimal ways to do that (i.e. how to pick the optimal window for aggregation)?\n\nYou mention that baseline methods (e.g. RMSE) are unable to detect the variance of the anomalies. Do you have any thoughts on how your proposed methodology performs in the tails of the distributions that you're comparing? In most situations, the only parts of the climate distributions that are of interest to the end users are the extremes.\n\nDo you have any thoughts on whether it makes sense to rank climate models in the first place? I assume the ranking can then be used to create a weighted ensemble of models.\n\nMinor:\nLine 52: for such a purpose\nFig. 2 y-axis  - preCipitation", "limitations": "Both, methodological and application specific limitations are discussed in Sec. 5.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper is well-written and easy to follow. It provides adequate background discussion on both, the methodology and the specific problem of climate model inter-comparison. The methodology is introduced rigorously, carefully defining the terms and the associated spaces. The experiment section includes a large number of models (which is not a small undertaking given the size of climate simulation data).", "weaknesses": "While the specific methodology is novel and may be of interest beyond climate modelling, it is a minor extension to the existing methods. Furthermore, the paper is heavily focused on a specific application. Consequently, a venue that is primarily focused on climate informatics would be more appropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide thoughtful feedback, we appreciate your attention to detail.\n\nFirst, in response to your “Contribution” score of 1 and comment:\n>**While the specific methodology is novel and may be of interest beyond climate modelling, it is a minor extension to the existing methods. Furthermore, the paper is heavily focused on a specific application. Consequently, a venue that is primarily focused on climate informatics would be more appropriate.**\n \nPlease see our above rebuttal to all reviewers for our response. We believe our work (both in methods and applications) provides significant contributions to ML and climate science. We hope the above discussion better demonstrates our work's impact and potential in addressing climate and traditional ML applications.\n\nIn response to your remaining comments:\n>**From what I understand, the data used in daily averages of temperature and precipitation for a historic period. Is the distance calculated on the daily basis? If so, how do you take into account the fact that climate models are known to be poorly temporally aligned with observation data?**\n\nWe would like to clarify the misunderstanding that the distance was calculated on a daily basis. As you mentioned, climate models are known to be poorly temporally aligned with observation data, so no method (including ours) calculates the distance on a daily basis. Many methods focus on temporally aggregated data, such as seasonal/annual means, but fail to assess extremes that happen on shorter time scales. In contrast, our method compares the *distributions* of daily model output to the observations. Our use of unaggregated daily data allows us to reasonably approximate the distribution generated by the models. The optimal transport approach matches similar observations from model outputs and observations to avoid the time alignment issue.\n>**Would it make more sense to aggregate the data temporally? If so, do you have any thoughts on optimal ways to do that (i.e. how to pick the optimal window for aggregation)?**\n\nMost conventional methods focus on temporally aggregated data, such as annual means or long-term climatologies, to avoid the issues you mentioned above. However, the impacts of climate change on, e.g., temperature are not limited to changes in the means and may include changes in variance or other moments. Changes in the distribution tails (extremes) can have high impacts on society, thus the distribution of daily data is important to climate model evaluation and provides vital information for climate change mitigation. For example, understanding extremes for daily precipitation would help us to understand the potential for floods/droughts in different regions, which is important for agriculture and water management. Model representation of daily temperature is closely related to the model’s capability in projecting heatwaves, an important issue for public health. \n>**You mention that baseline methods (e.g. RMSE) are unable to detect the variance of the anomalies. Do you have any thoughts on how your proposed methodology performs in the tails of the distributions that you're comparing? In most situations, the only parts of the climate distributions that are of interest to the end users are the extremes.**\n\nIn Appendix E.1, we provide an example where RMSE is not able to detect changes in the variance of the anomalies, but SCWD is. This is because increasing the variance creates compensating high and low extremes that cancel out when computing temporal averages. One of the strengths of SCWD is that the underlying Wasserstein distance considers all moments and quantiles of the distributions, including extremes. We demonstrate this for a specific climate model in Appendix E.2. The results suggest that differences in the rankings from SCWD and RMSE come from the fact that SCWD accounts for differences in the tail behavior of two distributions.\n\nClimate models can be evaluated from different perspectives. We focus on the entire distribution of the synthetic climate as all characteristics including mean, variance, and extremes are of interest to climate modelers. Of course, extremes are more connected to events such as natural disasters, which are of keen interest to end users. To focus only on the extreme climate, we can adapt our method to consider only certain quantiles of interest. This is similar to the trimmed Wasserstein distance discussed in [7], but we would focus solely on the tails. Our theoretical properties would be maintained in this case.\n>**Do you have any thoughts on whether it makes sense to rank climate models in the first place? I assume the ranking can then be used to create a weighted ensemble of models.**\n\nQuantitative evaluation of climate models often yields a ranking, i.e. [8] [9] [10]. Such rankings are dependent on the evaluation method and the variables being evaluated and need to be interpreted in a proper context. However, since quantitative evaluations provide users with important information about model performance, rankings can be used to create weighted multi-model ensembles as you pointed out or be used to curate a subset of high-performing models for specific applications. Model evaluation via quantitative metrics is essential for tuning climate model parameters, which was traditionally done manually. Beyond the rankings, we also showed our method's utility in producing maps to understand where models differ from historical data.\n\nLastly, we thank you for catching those two typos, they have been fixed for the latest version of the paper!\n\n[7] Manole, T et al. Minimax confidence intervals for the sliced Wasserstein distance. 2022\n\n[8] Gleckler, PJ et al. Performance metrics for climate models. 2008\n\n[9] Schaller, N et al. Analyzing precipitation projections: A comparison of different approaches to climate model evaluation. 2011 \n\n[10] Vissio, G et al. Evaluating the performance of climate models based on Wasserstein distance. 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you all for taking your time to provide a thorough review of our work. One shared concern from a few of the reviewers was the generalizability of our method to other tasks within climate science and ML. First, we provide some further insight on our contributions by responding to a comment from **Reviewer vRQU**, then we expand the discussion section in the manuscript to highlight future opportunities for our method to address other ML tasks.

## **1. In response to the following comment from Reviewer vRQU:**

>**While the specific methodology is novel and may be of interest beyond climate modelling, it is a minor extension to the existing methods. Furthermore, the paper is heavily focused on a specific application. Consequently, a venue that is primarily focused on climate informatics would be more appropriate.**
 
We disagree strongly that our method is a “minor” extension to existing methods. Sliced variants of the Wasserstein distance (WD) and other optimal transport methods are a popular subject in ML. Many such works, including Sliced WD (SWD) [1], Generalized SWD [2], Convolution SWD [3], Intrinsic SWD [4] and more, are published in prestigious ML conference proceedings, especially NeurIPS. The contributions in our own paper are analogous to the extension from CNNs to Spherical CNNs [5] published in ICLR.

Furthermore, NeurIPS embraces various topics as shown in the call for papers, specifically:

> (NeurIPS 2024) is an interdisciplinary conference that brings together researchers in machine learning, ..., **natural sciences**, social sciences, and other adjacent fields. We invite submissions presenting new and original research on topics including but not limited to the following … **Applications** (e.g., vision, language, speech and audio, Creative AI) … **Machine learning for sciences (e.g. climate...)**

Although our application may be suitable for a climate informatics conference, we believe our paper as a whole is a better fit for NeurIPS. In particular, our introduction of new methodology and supporting theoretical results will be much better received by the ML community than the climate science community. Also, NeurIPS has shown a clear commitment to addressing climate issues through hosting the “Tackling Climate Change with Machine Learning” workshop series the last couple years.

Evaluation of climate models is an important topic given the impacts of climate change on society and the fact that climate models are the primary tool for climate projection. This application is a long standing and important problem in climate science, so we expect our work will be highly impactful.

## **2. Generalizability of our method and future work in ML/climate**

Furthermore, we believe our work is generalizable to tackle other important topics in climate science and more broadly in ML. In our individual responses, we have addressed the following:

+ **Reviewer UJm5:** generalizations to other climate variables and temporal resolutions. We’ve started work on these topics in collaboration with researchers from a climate modeling center who reached out to us about SCWD.
+ **Reviewer vRQU:** extension of SCWD to focus only on climate extremes, which are important to end users of climate models. A straightforward adaptation of SCWD is possible to consider certain distributional quantiles of interest.

See our responses to those reviewers for details. Additionally, we consider the following tasks as opportunities for future work using SCWD for climate science:

1. Learn an optimal value for the range parameter of our chosen Wendland kernel. The challenge would be to determine a climatologically relevant loss/criteria to evaluate different range values. 
2. Similar to the neural-network based defining functions in [2], we could use estimate a neural network to use as our kernel function. Note that we chose the Wendland kernel in our current manuscript because of the importance of comparing local spatial features in climate models. However, neural network-based functions may work better for other applications. 
3. Machine-assisted climate model tuning [6]. Climate models are mathematical simulations that rely on parameters to control different aspects of the model behavior. Historically, these parameters were hand-tuned, but recent work has focused on incorporating machine learning. Due to the extreme computational costs in climate modeling, quality metrics are essential for this task. This is one of the motivations for establishing the validity and utility of SCWD.

SCWD and the more general functional SWD introduced in our paper have potential for broader applications in ML. Similar to [3] which was used to train generative models for rectangular images, SCWD can be used to train generative models for $360^o$  images. Likewise, the functional SWD can be used to train generative models for functions on any manifold. Lastly, the functional SWD could be applied to texture mapping and/or color transfer (common use cases for WD variants) on the surface of 3D models, which can be considered as non-euclidean manifolds. 

We’ve added much of the discussion here to Section 5 of the paper. We see our work as bridge to adapt cutting edge methods from the ML literature to address important topics in climate science and hope to make this more clear!

[1] Sliced and radon wasserstein barycenters of measures. Bonneel et al., 2015

[2] Generalized sliced Wasserstein distances. Kolouri et al., 2019 (NeurIPS)

[3] Revisiting Sliced Wasserstein on Images: From Vectorization to Convolution. Nguyen and Ho, 2022 (NeurIPS)

[4] Intrinsic Sliced Wasserstein Distances for Comparing Collections of Probability Distributions on Manifolds and Graphs. Rustamov and Majumdar, 2023 (ICML)

[5] Spherical CNNs. Cohen et al., 2018 (ICLR)

[6] Toward machine-assisted tuning avoiding the underestimation of uncertainty in climate change projections. Hourdin et al., 2023

Meta-review: The paper introduces a novel Wasserstein distance metric allowing for the comparison of climate model data, using it to compare CMIP6 models with reanalysis data, as well as the differences between CMIP6 and CMIP5. The reviewers agree that this is an excellent paper in terms of topic, results, and presentation. I accordingly strongly recommend acceptance.

A citation that may be useful to add is:

Mooers, G., Pritchard, M., Beucler, T., Srivastava, P., Mangipudi, H., Peng, L., Gentine, P. and Mandt, S., 2023. Comparing storm resolving models and climates via unsupervised machine learning. Scientific Reports, 13(1), p.22365.
Decision: Accept (spotlight)