RainShift: A Benchmark for Precipitation Downscaling Across Geographies
Paula Harder , Luca Schmidt , Francis Pelletier , Nicole Ludwig , Matthew Chantry , Christian Lessig , Alex Hernandez-Garcia , and David Rolnick
1 Mila Quebec AI Institute, Montreal, Canada 2 European Centre for Medium Range Weather Forecasts (ECMWF), Bonn, Germany 3 Cluster of Excellence Machine Learning, University of T¨ubingen, T¨ubingen, Germany 4 McGill University, Montreal, Canada - luca-marie.schmidt@mila.quebec + these authors contributed equally to this work
ABSTRACT
Earth System Models (ESM) are our main tool for projecting the impacts of climate change. However, running these models at sufficient resolution for local-scale risk-assessments is not computationally feasible. Deep learning-based super-resolution models offer a promising solution to downscale ESM outputs to higher resolutions by learning from data. Yet, due to regional variations in climatic processes, these models typically require retraining for each geographical area—demanding highresolution observational data, which is unevenly available across the globe. This highlights the need to assess how well these models generalize across geographic regions. To address this, we introduce RainShift, a dataset and benchmark for evaluating downscaling under geographic distribution shifts. We evaluate state-of-the-art downscaling approaches including GANs and diffusion models in generalizing across data gaps between the Global North and Global South. Our findings reveal substantial performance drops in out-of-distribution regions, depending on model and geographic area. While expanding the training domain generally improves generalization, it is insufficient to overcome shifts between geographically distinct regions.
We show that addressing these shifts through, for example, data alignment can improve spatial generalization. Our work advances the global applicability of downscaling methods and represents a step toward reducing inequities in access to high-resolution climate information. Introduction
High-resolution climate projections are crucial for planning effective responses to extreme weather and climate events. With ongoing climate change, extreme events such as floods, droughts, and heatwaves are expected to become more frequent and severe, threatening infrastructure, agriculture, energy systems, and public health . Among these, precipitation extremes are particularly destructive, causing floods, landslides, and soil erosion; and are projected to intensify at local scales[,] .
More recently, diffusion models have also shown strong performance in modeling complex, high-dimensional distributions[–] . Among all climate variables, precipitation is particularly challenging to model and forecast due to its stochastic, high-frequency, spatial and temporal variability. Capturing these fine-scale variations and inherent uncertainties would make generative approaches successful for precipitation downscaling. Unlike dynamical downscaling, statistical downscaling algorithms are not restricted to specific data sources or geographic regions. By construction, statistical downscaling relies on statistical relationships learned during the training phase, allowing it to be applied to new regions at inference time. However, this transferability hinges on the assumption that both predictor and predictand distributions, as well as their statistical relationships, remain stationary across space. In practice, this assumption does not hold due to substantial geographic variability in topography, climatic conditions and processes. For example, precipitation in equatorial regions is strongly driven by convection and, therefore, substantively different than in Europe and North America. Consequently, statistical downscaling models often require retraining for each target region, which relies on the availability of high-resolution observational data. Despite the large amount of weather and climate datasets, high-quality observations are unevenly distributed globally. Ground-based radar and gauge data, essential for training and validating downscaling models, are particularly sparse in many parts of the Global South. Yet, it is also these regions that are often the most exposed and vulnerable to climate change and extreme weather events like heavy rainfall and flooding . This global imbalance in data availability—coupled with heterogeneous regional climate processes—presents challenges in generalization of deep learning-based downscaling models. To address these challenges, we introduce RainShift, a large-scale global benchmark and dataset designed to evaluate the geographical generalization of deep learning-based downscaling. RainShift defines test scenarios where models are trained on subsets of data-rich regions and tested on regions with scarce availability of high-resolution observations. The dataset is built from ERA5 reanalysis and IMERG satellite precipitation data. We establish baseline results by evaluating state-of-the-art models for probabilistic precipitation downscaling, including GANs and diffusion-based architectures. RainShift is intended to support the development of approaches that generalize to low-data regions, particularly in underrepresented areas such as the Global South. Our main contributions are the following:
- We frame the task of downscaling across a geographically varying data distribution, based on a critical gap in Earth system modeling. - We introduce the RainShift benchmark dataset, along with tools for expanding it to new regions, data loaders, training pipelines and evaluation frameworks, within an accessible library to facilitate further research. - We evaluate a variety of state-of-the-art machine learning downscaling models on RainShift and find substantial variation in spatial generalization across models and regions; we show that data alignment techniques can improve performance in cases where generalization is limited by strong geographic distribution shifts. Results
We train a variety of downscaling models in scenarios representing data-abundant regions and then evaluate their performance in data-sparse regions, mainly in the Global South. The downscaling task consists of learning a mapping from coarser resolution reanalysis data to higher resolution satellite-based precipitation (IMERG), as illustrated in Figure 2. These globally consistent data sources enable a controlled benchmark setup. To evaluate spatial generalization, we define 12 training regions and 6 evaluation regions. We combine the training regions into four progressively larger training scenarios to simulate varying levels of observational coverage; their geographic distribution, along with that of the evaluation regions, is shown in Figure 4. The analysis includes several state-of-the-art downscaling approaches: ResNets (a deterministic approach), Wasserstein GAN with gradient penalty (a probabilistic approach), and a diffusion-based method (also probabilistic). We also include a simple bilinear interpolation as a baseline. We evaluate model performance with respect to spatial generalization using the continuous ranked probability score (CRPS), considering out-of-distribution performance both in absolute terms and relative to in-distribution evaluation. Our analysis investigates key factors affecting generalization, including the downscaling model, the geographic region, and size of the training domain. To address the observed performance drops in unseen regions, we propose a simple data alignment approach based on quantile mapping to improve the spatial generalization ability of the evaluated models.
Figure 1. Map of ground-based radar stations. The map shows the availability of precipitation data, with each blue dot representing a station. Coverage is relatively high in the Global North and comparatively low across the Global South. Image from the Tropical Globe radar database . Added value of downscaling models All learned models demonstrate some generalization to unseen target regions. As shown in Figure 5, they achieve consistent improvements (except for two outlier values) on the ERA5 interpolation, which serves as a baseline input compared to IMERG. This trend holds across geographical areas. The consistency of these improvements across diverse target areas shows the general validity and robustness of the downscaling approach, indicating that such models can be used to downscale coarse-resolution inputs even when applied to regions not seen during training. However, the magnitude of relative improvement varies greatly between target regions, with relative CRPS reductions ranging between approximately 30% and 50%. Probabilistic models outperform deterministic ones Across almost all evaluation regions, probabilistic generative approaches, both GAN and diffusion models, consistently outperform the deterministic ResNet model. In some cases, ResNet even shows unstable behavior, such as in region E5 (Tibetan Plateau) under configuration A1, where the CRPS value becomes unreliable due to numerical instabilities that occur exclusively during inference. The issue appears to be the result from a combination of limited training data and a pronounced distribution shift between training and target areas. Specifically, the Tibetan Plateau has much lower precipitation compared to the A1 (Western North America) training region. In preliminary experiments in Table 1, we show that aligning the data distributions between training and target areas effectively resolves this issue. Both GAN and diffusion show stable and consistent improvements over bilinear interpolation across regions and training configurations. In the largest training setup, GAN and diffusion achieve comparable absolute and relative improvements over the baseline. However, the two models show differences in how they respond to extending the training domain. The diffusion model already generalizes well when trained on a smaller training domain, whereas the GAN shows more pronounced improvements with increases in training data. This suggests that diffusion models are more robust to limited training data, while GANs benefit more from larger, more diverse training datasets. Expanding the training area is not always beneficial Overall, model performance generally improves when expanding the training areas from A1 to A4. In some regions—such as Cape Horn, Amazon Basin, West Africa and Melanesia—expanding the training domain leads to clear improvements in CRPS scores. However, this trend is less consistent in other regions. In the Horn of Africa and the Tibetan Plateau adding more training data does not necessarily lead to better generalization ability. Furthermore, the benefits of expanding training area tend to diminish with larger training domains. This suggests that increasing training data alone does not guarantee continued improvements in spatial generalization. In-distribution training does not always lead to the best performance For most regions and models, in-distribution training leads to the best CRPS scores. Consequently, we observe substantial relative performance drops in Figure 6 when models are trained out-of-distribution. For the smallest training setup sp
cp
cape
twc
tlwc
ERA5 (atmospheric reanalysis data)
... Probabilistic samples
~~Generate~~
200 × 200 px, 20 [◦] × 20 [◦]
Loss
CRPS
tisr u wind v wind
tp
80 × 80 px, 20 [◦] × 20 [◦]
Train: 2001 2020
Eval.: 2021 2022 Geographical features land sea mask
orography
200 × 200 px, 20 [◦] × 20 [◦]
Ti
Train Downscaling
Evaluate Model
E j
IMERG
(ground truth)
200 × 200 px, 20 [◦] × 20 [◦]
Figure 2. Graphical summary of RainShift setup. The inputs of the downscaling model are a combination of ERA5 time series data and geographical features. The downscaling model is then able to generate probabilistic samples. For training, we sample from geographic areas T 1 ,..., T 12 and years 2001–2020, and compare the generated samples with the ground truth target IMERG to compute the loss. For evaluation, we use areas E 1 ,..., E 6, and years 2021–2022.
Illustration of training configurations. The training configurations A 1 ,..., A 4 are composed of progressively larger subsets of the 12 selected training regions located in the Global North. The choice of regions is guided by availability of high-resolution observational data and inspired by existing works[,] . Figure 4. Illustration of location splits and training configurations. Patches T 1 ,..., T 12 represent training regions and patches E 1 ,..., E 6 correspond to evaluation areas that are used within 4 sub-tasks, simulating different scenarios that correspond to varying levels of data availability. distributional correction techniques such as quantile mapping used to align the feature distribution of training and target regions can greatly improve performance. This is achieved by matching the cumulative distribution functions of the precipitation inputs in the target region to that of the training region. Figure 8 illustrates the distributional discrepancies, which are particularly pronounced in regions like Tibetan Plateau and Melanesia. After applying quantile correction, the CDFs are much more closely aligned, leading to improved performance across nearly all regions. Discussion
This paper introduces the RainShift benchmark to evaluate the out-of-distribution generalization of deep learning models for precipitation downscaling across geographically distinct regions. Our dataset is constructed from paired reanalysis inputs and satellite-based precipitation targets. It covers twelve training areas with dense observational coverage and six evaluation areas located in the Global South, where high-quality observations are sparse. The benchmark task evaluates a model’s ability to learn from high-resolution training data and generalize to unseen regions, a crucial requirement to the real-world deployment of these models. Our results show that all evaluated models add substantial value over raw input precipitation, with probabilistic generative approaches (GAN and diffusion) showing better performance than deterministic ones, both within training domains and in generalizing to unseen geographies. However, performance still degrades when applied to unseen regions, highlighting current limitations in cross-regional generalization. Our results indicate that spatial generalization is more strongly limited by geographic variability and distribution shifts than by differences in model architecture or training data size, indicating that addressing these shifts is a critical path forward. Figure 5. Heatmap of % improvement relative to interpolation. Change in CRPS (lower better) in [%] for each model relative to bilinear interpolation. A 1 ,..., A 4 represent hierarchical training scenarios with progressively more high-resolution data, from training on a single region to using several training regions across the Global North. Positive values show improvements over the interpolation baseline. Model performance generally improves when expanding the training areas from A1 to A4, but this trend depends strongly on the geographical region. The N/A value indicates an instance where the CRPS value is not reliable due to numerical instabilities during inference, likely driven by large distributional differences between training and target regions. To advance progress in this area, we frame cross-regional generalization as a central challenge for current downscaling models and encourage methodological innovation to address it. Our results show that simple data alignment techniques, such as quantile-based input alignment, can enhance model generalization across regions. Beyond this, other promising directions in this regard could include, for example, the use of location-aware embeddings as auxiliary inputs, or unsupervised domain adaptation methods that align feature distributions between labeled and unlabeled target domains .
Further strategies such as meta-learning, which enables faster adaptation to new regions with limited data, and the integration of physical constraints to improve model robustness, also have potential to improve spatial generalization. Our choice of training and evaluation regions is guided by the availability of high-resolution observations. To facilitate development and deployment of new methods, we use a globally homogeneous dataset, IMERG, as the target. However, a promising avenue for future work is to expand the benchmark to incorporate additional, more localized sources, such as precipitation radar data from individual countries. Models trained on such data could then be applied to regions without high-quality observational data available, including the Global South. More broadly, the RainShift benchmark is designed to provide a framework for evaluating how well models generalize to regions most in need of accurate, high-resolution climate information—bridging the gap between highly localized model development and global applicability. Methods
We introduce the RainShift dataset, along with its preprocessing pipeline, baseline models, and evaluation framework. All components are fully reproducible and will be made publicly available upon acceptance. RainShift dataset
The RainShift dataset builds on three global data sources: atomospheric reanalysis, satellite-based precipitation estimates (IMERG), and two invariant geographical features—land-sea mask and orography. The use of globally consistent satellite and reanalysis data enables a controlled benchmark setup that is essential for evaluating how well downscaling models generalize spatially.
Using satellite-derived precipitation as the target allows for evaluation in data-sparse regions like the Global South, while limiting confounding differences due to differences between data products. Input data As low-resolution input data, we use ERA5, the fifth-generation atmospheric reanalysis product of the European Center for Medium-Range Weather Forecasts (ECMWF). Reanalysis data are the result of combining historical observations with Earth system models through data assimilation to obtain global estimates of the observed climate.
It provides hourly global data at 0 . 25 [◦] × 0 . 25 [◦] resolution on a regular latitude-longitude grid and spans years from 1950 to the present. For compatibility with IMERG, we use data from 2001 onward. Figure 6. Heatmap of % performance drop between in and out-of-distribution training. Change in CRPS (lower better) in [%] for each model relative to training the model directly on the target regions. A 1 ,..., A 4 represent hierarchical training scenarios with progressively more high-resolution data, from training on a single region to using several training regions across the Global North. Many regions show large negative values, indicating substantial drop in performance when evaluating on regions out-of-distribution, highlighting challenges in spatial generalization. The N/A value indicates an instance where the CRPS value is not reliable due to numerical instabilities during inference, likely driven by large distributional differences between training and target regions. We select nine input variables as shown in Table 2, based on meteorological relevance to predict subgrid rainfall variability guided by the ecPoint model and domain-specific knowledge discussed elsewhere . We additionally include geographic covariates at 0 . 1 [◦] resolution: (i) A land-sea mask indicating the land fraction per pixel, and (ii) an elevation map (geopotential height at surface). Target data The Integrated Multi-satellite Retrievals for GPM (IMERG) is a product of NASA’s Global Precipitation Measurement (GPM) mission and serves as high-resolution target data. IMERG provides precipitation estimates based on the GPM satellite constellation and additional observations such as gauge data. IMERG has full coverage between 60 [◦] N and 60 [◦] Sat 0 . 1 [◦]
resolution on a regular latitude-longitude grid. We use the IMERG V07 Final Run product, averaging its half-hourly data to hourly to match ERA5’s temporal resolution. Data was accessed via NASA Goddard’s GES DISC. Summary statistics Table 3 shows the average amount of precipitation in mm/h for each target region in low- and high-resolution datasets. We observe a strong correlation between the error and mean precipitation of a region, i.e. regions with higher precipitation tend to have higher prediction errors. Data processing The datasets are downloaded and preprocessed for consistency. Global data is divided into subregions and stored in the ML-friendly Zarr format . Zarr archives retain metadata (e.g., latitude, longitude, timestamps) and are chunked for efficient loading during training. Each chunk contains 200 timesteps, optimized for sampling speed: about 10 MB per chunk for ERA5 variables and 30 MB for IMERG precipitation. Precipitation estimates from model-based products such as ERA5 are generally considered less accurate than those from satellite-based products such as IMERG[,] . To mitigate known artifacts in ERA5 precipitation data—particulary the overestimation of weak precipitation, mis-detection of non-precipitation events, and unrealistically large values—we clip the precipitation variable using the lowest and highest values from IMERG as thresholds. Very small values are set to zero, while large values are adjusted downward to align with the IMERG data distribution.
Each variable is then standardized via Z-score normalization using statistics computed across all time steps, latitudes and longitudes, and aggregated across all training regions. The global mean is calculated as the average of region-wise means, and the overall variance is estimated using the pooled variance, i.e. 2
, �
˜ σ =
n
n ∑
i =1
� σ i [+] [ µ] i � −
1
n �
n ∑ µ ii =1 Target Areas Model QM E1 E2 E3 E4 E5 E6 Cape Horn Amazon Basin West Africa Horn of Africa Tibetan Pl. Melanesia
DM No 0.091 0.182 0.084 0.054 0.028 0.295
DM Yes 0.092 0.182 0.077 0.054 0.024 0.310
GAN No 0.075 0.179 0.093 0.055 0.027 0.326
GAN Yes 0.093 0.179 0.080 0.055 0.024 0.310
ResNet No 0.083 0.187 0.308 0.057 18113.558 0.389
ResNet Yes 0.093 0.181 0.079 0.055 0.025 0.310
Table 1. Impact of quantile-based domain alignment on model accuracy. CRPS (lower values are better) of models trained on scenario A1 (Western North America) and evaluated on the designated target regions for precipitation in [mm/h] over test
− years 2021 2022. Results show model accuracy with and without applying quantile mapping (QM) to align the input distributions between training and evaluation regions. All predictions are transformed back to the original target domain data range and re-normalized before computing the metrics. Applying quantile mapping equals or improves performance across most models and regions (except Cape Horn) demonstrating the potential of data alignment techniques to enhance spatial generalization under geographical distribution shifts. Variable Description
tp Total precipitation cp Convective precipitation cape Convective potential energy twc Total water content
tlwc Total liquid water content sp Surface pressure tisr Top-of-the-atmosphere incident solar radiation u Eastward wind velocity at 700 hPa v Northward wind velocity at 700 hPa
Table 2. List of atmospheric variables used as predictors. where n is the number of training regions, µ iis the mean, and σ i [the variance of region] [ T] [i] [.] Following standardization, the two precipitation variables tp (total precipitation) and cp (convective precipitation) as well as orography are log-transformed by ˜ x = log . The land-sea mask remains unchanged with values between 0 and 1 . For the diffusion model, we additionally perform bilinear interpolation to the ERA5 inputs to match the spatial resolution of IMERG for compatibility with the UNet architecture. Rather than directly predicting the high-resolution target, the UNet is trained to learn the residual between the fine-resolution target and the interpolated coarse-resolution input, following prior work . Temporal splits We treat every time step as an independent sample.
However, as the data is a time series, we split training and evaluation data temporally. The training data in areas T 1 ,..., T 12 covers years 2001–2020, the testing data in areas E 1 ,..., E 6 covers years
Target Areas E1 E2 E3 E4 E5 E6
Cape Amazon West Horn Tibetan Melanesia Horn Basin Africa of Africa Plateau
Mean precip. ERA5 0.11 0.18 0.07 0.06 0.04 0.33 Mean precip. IMERG 0.10 0.18 0.08 0.05 0.02 0.31
Table 3. Mean precipitation values. Values are in mm/h averaged over each evaluation region using years 2021-2022.
2021–2022. Location splits We create RainShift choosing 18 regions worldwide, covering all continents and climate zones, each spanning 20 [◦] × 20 [◦], as shown in Figure 4. These regions are divided into 12 training regions and six evaluation regions. The six evaluation regions are: Cape Horn, Amazon Basin, West Africa, Horn of Africa, Tibetan Plateau and Melanesia. Training regions are selected from areas with high observational data availability, while evaluation regions are located in data-scarce areas, primarily in the Global South. Downscaling task formulation The RainShift benchmark focuses on a probabilistic downscaling task, where the goal is to learn the conditional distribution, p ( y|x ), of high-resolution precipitation, y, given a low-resolution forecast and invariant features, x . A generative model, G, is trained to approximate the true distribution
G ( x ) ∼ p G ( ·|x ) such that p G ( ·|x ) ≈ p ( ·|x ) . Here, the high-resolution 2D target sample y ∈ R [h] [h] [×][w] [h] is a single-channel precipitation field from the IMERG satellite data. The input x consists of both low-resolution and high-resolution components: x = ( x ℓ , x h ), with x ℓ ∈ R [c][×][h] [ℓ] [×][w] [ℓ] representing a low-resolution c -channel forecast and x h ∈ R [d][×][h] [h] [×][w] [h] denoting d -channel high-resolution invariant features (land-sea mask and orography). With an upsampling factor N ∈ R [+], the high-resolution dimensions are given by h h = N · h ℓ and w h = N · w ℓ . In this benchmark, N = 2 . 5 and image patches are square: h ℓ = w ℓ = 80 and h h = w h = 200 . The channel dimensions are c = 9 and d = 2.
This downscaling task is illustrated in Figure 2. Geographical generalization Unlike existing benchmarks that evaluate models within the same geographic region, RainShift is designed to assess generalization across different geographies. We consider 12 different training regions and 6 regions for evaluation. Given a training area, A, the corresponding local data distribution p A and samples ( x A , y A ) ∼ p A, the goal is to learn the distribution p E ( ·|x E ) in a separate evaluation area E . Here, training and evaluation areas are disjoint, A ∩ E = /0 . The task is a zero-shot prediction, with no available labels in the evaluation set. A generative model G, is desired to approximate the target distribution
G ( x ) ∼ p G ( ·|x ) such that p G ( ·|x E ) ≈ p E ( ·|x E ) .
Training sub-tasks To simulate different scenarios corresponding to varying levels of data availability, we define four training sub-tasks, as illustrated in Figure 4. Each sub-task is associated with a subset of training areas A i ⊆ T = [�] j =1 [T] [j] [for] [ i] [ =] [,...,] [. The subsets] are hierarchical—that is, A i ⊆ A i +1 for i = 1 ,..., 3 reflecting varying levels of observational availability:
1. A 1 := T 1 : a scenario resembling the common task of just training in one geographic area. 2. A 2 := T 1 ∪ T 2 ∪ T 5 : inspired by existing works that use both North American and European datasets[,] . 3. A 3 := A 2 ∪ T 10 ∪ T 11 ∪ T 12 : an extension of A 2 adding three areas with high availability of observational data: two areas in Eastern Asia and one in Eastern Australia. 4.
A 4 := [�] i =1 [T] [i] [: a very optimistic scenario that assumes access to avast amount of high-resolution observations from a] variety of regions and sources, including places with limited data availability. Usage and contribution The RainShift benchmark dataset is hosted via Hugging Face at [https://huggingface.co/datasets/RainShift/](https://huggingface.co/datasets/RainShift/rainshift) [rainshift](https://huggingface.co/datasets/RainShift/rainshift) . The dataset consists of a zipped Zarr directory per region, resulting in 300 GB of overall data. The download, data-loading, and training instructions can be found in our repository, which will be made available upon acceptance. All baselines are made available via our repository. For the purpose of a unified benchmark task, we fix different sets of training and evaluation regions. However, we provide the tools to add new areas of interest to enable optimization or testing in any location worldwide. The instructions to create new regional subsets are provided in our repository, that will be made available upon acceptance. Baseline models
We evaluate a diverse set of baseline models to provide a comprehensive view of performance across different model classes. These include a deterministic ResNet model, as well as two successful probabilistic methods: generative adversarial networks (GANs) and a diffusion-based approach.
To contextualize these results, we also include bilinear interpolation as a simple baseline to establish a lower performance bound, and in-region training as an upper bound on expected performance. Interpolation baseline As precipitation is both an input and output variable, we may construct a simple baseline by bilinearly interpolating ERA5 total precipitation to approximate IMERG data. This helps assess whether poor model performance in specific regions stems from generalization issues or challenges inherent to the input data. ResNet Super-resolution CNNs, often ResNet variants, were the first DL tools applied to downscaling . In deterministic downscaling they still achieve state-of-the-art performance[,] . We include a fully-convolutional residual network, following methodology shown to perform highly for precipitation forecast downcaling . This serves as a deterministic baseline. Generative Adversarial Networks GANs are a popular approach for probabilistic downscaling of meteorological data . In this context, conditional GANs are typically used, where both the generator and the discriminator are conditioned on low-resolution input fields in addition to the generator’s noise input. We leverage the Wasserstein GAN (WGAN) with gradient penalty, which is less prone to training instabilities such as mode collapse. The model architecture is based on prior work[,], and represents a commonly used model in downscaling. Diffusion-based models Diffusion models (DMs), in particular denoising score-matching approaches are gaining traction in climate- and weathermodeling applications[,] . In this work, we follow the diffusion-based downscaling framework introduced in ClimateDiffuse, which combines several established components into an effective approach for conditional downscaling of climate fields. The model is a score-based model trained under the denoising score matching framework, in which a neural network is optimized to learn the score function. This score function is parameterized by a conditional U-Net, conditioned on low-resolution input fields, and the forward and reverse diffusion processes are defined via the stochastic differential equation formulation. Consistent with prior work, we also incorporate several key design choices, such as the use of improved preconditioning and a higher-order integration scheme for the differential equation solver. Training details The GANs and DMs are trained for 60-168 hours (depending on training area size) with an effective batch size of 128 on 4 NVIDIA A100 GPUs. The ResNets are trained for 45-280 hours on 2 NVIDIA RTX8000. The years 2019 and 2020 are used as validation data for hyperparameter tuning and choosing the best checkpoint. Evaluation
To evaluate spatial generalization ability of the models, we compare their performance across the different training scenarios using the continuous ranked probability score as a metric. Model performance is reported as absolute scores, relative improvements over interpolation baseline and relative performance compared to training directly on the target area. Quantitative evaluation
The overall most successful model for each subtask is determined by the point-wise continuous ranked probability score (CRPS), calculated using 8 samples at each of the six target locations. The CRPS is a metric used to evaluate the accuracy of probabilistic forecasts. For a given forecast probability distribution F and the observed outcome y, the CRPS metric is defined as follows:
∞ CRPS( F, y ) = � − ∞ [[] [F] [(] [z] [)] [−] [(] [z][ ≥] [y] [)]] [dz .]
Here, F ( z ) is the cumulative distribution function of the forecast distribution at point z and 1 ( · ) the indicator function. For a deterministic forecast, the CRPS reduces to the mean absolute error. In-area training The standard evaluation in deep learning-based downscaling consists of training and evaluating in the same area. To show that the geographical generalization is indeed a challenge we also report scores that use the target area labels. For this, we train the above mentioned models, ResNets, GANs and DMs on the respective target areas E 1 ,..., E 6 directly, while keeping the temporal train-test split: training on years 2001-2020 and testing on years 2021 and 2022. Qualitative evaluation
In addition to CRPS scores, we provide a qualitative comparison of downscaled precipitation fields for one sample as well as temporally averaged precipitation and CRPS values. Target Areas E1 E2 E3 E4 E5 E6
Model Training Cape Amazon West Horn Tibetan Melanesia Areas Horn Basin Africa of Africa Plateau
ERA5 Interp. - 0.120 0.258 0.113 0.084 0.050 0.434
GAN A 1 0.075 0.179 0.093 0.055 0.027 0.326 DM A 1 0.091 0.173 0.084 0.054 0.028 0.295 ResNet A 1 0.083 0.187 0.308 0.057 18113.558 0.389
GAN A 2 0.076 0.178 0.088 0.057 0.032 0.324 DM A 2 0.078 0.177 0.080 0.056 0.027 0.288 ResNet A 2 0.083 0.181 0.080 0.055 0.034 0.315
GAN A 3 0.072 0.169 0.080 0.053 0.024 0.288 DM A 3 0.074 0.171 0.079 0.054 0.025 0.290 ResNet A 3 0.081 0.181 0.080 0.055 0.025 0.306
GAN A 4 0.070 0.167 0.077 0.054 0.025 0.272 DM A 4 0.072 0.170 0.077 0.056 0.027 0.284 ResNet A 4 0.083 0.179 0.078 0.055 0.024 0.298
GAN E i 0.077 0.180 0.072 0.053 0.025 0.272 DM E i 0.070 0.171 0.075 0.054 0.023 0.275 ResNet E i 0.078 0.181 0.078 0.057 0.025 0.311
Table 4. Accuracy of models on spatial generalization tasks. Test CRPS (lower better) for precipitation in mm/h on the
− designated evaluation area and averaged over test years 2021 2022. Shown is the mean, pixel-wise CRPS of 8 ensemble members. For deterministic models, the MAE is shown. We report the mean precipitation amount in input and target data in mm/h. Best scores per subtask are in bold.
The last three rows with scores in blue are not contestants in the benchmark but show what is possible when training directly on the target. Background
Model generalization to new geographic regions is an active research area in applied machine learning, particularly in remote sensing and biodiversity modeling. In agricultural classification and segmentation, approaches such as task-informed metalearning, versions of model-agnostic meta-learning, and multi-source unsupervised domain adaptation have shown promise in adapting to new regions with minimal data. In biodiversity monitoring, previous work integrates remote sensing and citizen science data to improve generalization in data-sparse regions like Kenya, while spatial implicit neural representations have been leveraged for scalable global species range estimation using noisy, sparse data . In climate science, spatial generalization in deep learning-based downscaling remains underexplored. A few recent studies have tested model transferability between subregions . For instance, some works examine generalization between different areas on the US West Coast[,] . Others analyze performance across regions in the UK and the United States or evaluate generalization from the DACH region (Germany, Austria, and Switzerland) to North America .
While these efforts provide valuable insights, their geographic scope remains limited. Another important development has been the creation of benchmark datasets aimed at standardizing and accelerating machine learning methods for Earth System Modeling. Notable examples include WeatherBench, ClimateBench, ClimateLearn
and ClimateSet which have provided standardized datasets, tasks and evaluation frameworks. Several benchmarks specifically target precipitation forecasting and downscaling. RainBench introduces a global benchmark for precipitation forecasting based on IMERG data . RainNet focuses on precipitation super-resolution, targeting a region on the US East Coast with singlevariable input data in a deterministic setup . The ClimateLearn benchmark supports evaluation of downscaling techniques that map low-resolution CMIP6 model outputs to high-reslution ERA5 data . These efforts have made climate modeling more accessible to the broader machine learning community. Despite this, no such existing benchmark has yet specifically addressed the challenge of generalizing across geographies. In this paper, we present RainShift, seeking to fill this gap by introducing a large-scale global benchmark dataset specifically designed to evaluate and improve the geographical generalization of deep learning-based downscaling. Figure 7. Qualitative comparison of downscaled precipitation fields. This plot shows a sample, one time step from the evaluation set in the Cape Horn area in the first two columns and aggregated features in the last column. The random sample includes the matching input and target (IMERG) precipitation values in logarithmic scale and two samples each from the two generative approaches, GANs and DMs. The right column shows on top, the mean precipitation over the whole testing period as well as the spatial distribution of CRPS scores for GAN and DM. Quantile mapping for geographical generalization Geographic generalization is a central challenge in statistical downscaling, resulting from differences in climatic conditions and their underlying processes across geographical areas. Such differences can lead to substantial performance drops when models trained in one region are applied to new regions with distinct climatic characteristics. In our experiments, we observe large performance drops in target regions whose precipitation distributions differ markedly from those of the training regions. Therefore, a promising strategy to address these distributional shifts is to align the input data distributions of the training and target regions. To this end, we use a strategy based on quantile mapping. While the quantile mapping (QM) technique is traditionally used to correct systematic distributional biases in climate model simulations relative to observations (leveraging historical relationships between simulations and observations to adjust future simulations), we apply it to address mismatches between the input distributions of the training and target regions. By aligning the input distribution of the target region more closely with that of the training data, the model may be better able to generalize and generate more reliable predictions in unseen regions. Specifically, we learn a mapping between the cumulative distribution functions (CDFs) of the precipitation input data of the training regions, F train ,h, and that of the target regions, F target ,h, over a historical period h . This results in the following transfer function:
x ˆ target , f ( t ) = F train [−] ,h � F target ,h � x target , f ( t )��,
which adapts the precipitation value x target , f ( t ) from a target region within some future period f . For a non-negative variable such as precipitation we use the multiplicative variant of quantile mapping, where the values are lower bounded by zero. We perform quantile mapping to the low-resolution precipitation inputs using 1000 quantiles prior to data normalization during inference. Figure 8 illustrates the discrepancy between the CDFs of the training and target regions, which is pronounced for all target regions except Cape Horn. After correction, the target region CDFs align more closely with the training CDF. Consistent with this reduction in distributional mismatch, we observe in Table 1 that the ResNet model evaluated with quantile-based aligned inputs achieves improved performance in all target regions except Cape Horn, compared to using unaligned inputs. Figure 8. Cumulative distribution functions (CDFs) of precipitation inputs for training and target regions. For each target region, a mapping is constructed between the historic CDF of the training region and the historic precipitation inputs of the target region. This mapping is subsequently used to correct the future input data from the target region, aligning them more closely with the training data distribution. Since precipitation is highly right-skewed with many zero and near-zero values, which causes most values to fall within the first bin of the CDF, we show the logarithm of the CDF to better visualize the discrepancies in distributions. Data availability
The RainShift benchmark dataset is hosted via Hugging Face at [https://huggingface.co/datasets/RainShift/](https://huggingface.co/datasets/RainShift/rainshift) [rainshift](https://huggingface.co/datasets/RainShift/rainshift) . The dataset consist of a zipped Zarr directory per region, resulting in overall 300GB of data. The download, data-loading, and training instructions can be found in our repository, that will be made available upon acceptance.