Precipitation Downscaling with Spatiotemporal Video Diffusion
Authors: Prakhar Srivastava
Ruihan Yang
Gavin Kerrigan
Gideon Dresdner
Jeremy J McGibbon
Christopher S. Bretherton
Stephan Mandt
Venue: NeurIPS 2024 poster (2024)
Keywords: precipitation downscaling
diffusion models
Primary area: machine_learning_for_physical_sciences

Abstract: In climate science and meteorology, high-resolution local precipitation (rain and snowfall) predictions are limited by the computational costs of simulation-based methods. Statistical downscaling, or super-resolution, is a common workaround where a low-resolution prediction is improved using statistical approaches. Unlike traditional computer vision tasks, weather and climate applications require capturing the accurate conditional distribution of high-resolution given low-resolution patterns to assure reliable ensemble averages and unbiased estimates of extreme events, such as heavy rain. This work extends recent video diffusion models to precipitation super-resolution, employing a deterministic downscaler followed by a temporally-conditioned diffusion model to capture noise characteristics and high-frequency patterns. We test our approach on FV3GFS output, an established large-scale global atmosphere model, and compare it against six state-of-the-art baselines. Our analysis, capturing CRPS, MSE, precipitation distributions, and qualitative aspects using California and the Himalayas as examples, establishes our method as a new standard for data-driven precipitation downscaling.

Precipitation Downscaling with Spatiotemporal Video Diffusion
Prakhar Srivastava'
Ruíhan Yang'
Gavin Kerrigan'
Gideon Dresdner
Jeremy McGibbon Christopher Bretherton" Stephan Mandt
University of California, Irvine 2 Allenn Institute for AI, Seattle
(prakhs2, [prakhs2,ruihan. ruihan. yang, gavin. k, mandt]]uui..ede jgideond, jeremym, christopherb/ Gallenai. org
Abstract
In climate science and meteorology. high-resolution local precipitation (rain and snowfall) predictions are limited by the computational costs of simulation-based methods. Statistical downscaling, or super-resolution, a common workaround where low-resolution prediction is improved using statistical approaches. Unlike traditional computer vision tasks, weather and climate applications require captur- ing the accurate conditional distribution of high- resolutool given low-resolution patterns to assure reliable ensemble averages and unbiased estimates of extreme events, such heavy rain. This work extends recent video diffusion models precipitation super-resolution, employing a deterministic downscaler followed by temporally-com -onoiialyy-ooniiiooneddd diffusion model capture noise characteristics and high-frequency patterns. We test our approach on FV3GFS output, an established large-scale global atmosphere model, and compare it against six state-of-the-art baselines. Our analysis, capturing CRPS, MSE, precipitation distributions, and qualitative aspects using California and the Himalayas examples, establishes our method as new standard for data-drive precipitation downscaling.
Introduction
Coarse Grid Input
STVD Output
Figure I: Static snapshot from the Spatiotemporal Video Diffusion (STVD) model, illustrating input (left) and output (right) precipitation frames. The input panel displays simulated russuuoneoo precipitation (rain, snow) fields (Section 33, super-resolved into the high-resolution output shown in the right panel. Both frames use Robinson projection and cover six tiles of the cubed-sphere grid, providing detailed global view (optimal viewing with zoom). For dynamics, see Fig 3
Precipitation patterns central to human and natural life. In rapidly warming climate, reliable simulations of changing precipitation patterns can help adapt to climate change. However, these simulations challenging due to the multi-scale variability of weather systems and the influence of complex surface features (like mountains and coastlines) on precipitation trends and extremes
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

goT
00T
Loss
Downscaler
Diffusion
20.2
N
Our model's training and inference pipelines: Blue blocks apply to both phases, red blocks to training only, and green blocks to inference only. It deterministically downscales low- resolution precipitation sequence using spatio-temporal factorized attention and models residuals with conditional diffusion (with factorized attention). Here, T denotes sequence length and N denotes diffusion steps. The parameters (0 o, tb) are optimized jointly during training. See A. for details.
Figure 2:
For many purposes, such estimating flood hazards, precipitation must be estimated at spatial resolutions of only a few kilometers. Fluid-dynamical models of the global atmosphere are too expensive run routinely at such fine scales [65], so the climate adaptation community relies "downscaling""
58
o of coarse-grid simulations to finer grid. Traditional downscaling methods are either dynamical (running fine-grid fluid-dynamical model limited to the region of interest, which requires specialized knowledge and computational resources) or statistical (typically restricted to simple univariate methods) [67]. Our work builds on vision based super-resolution methods to improve statistical downscaling and is a natural follow-up to recent deep-learning haa--llaninn--beee based weather/climate prediction methods, which have revolutionized data-driven forecasting. These approaches boast improvements of orders of magnitude in runtime without sacrificing accuracy 46 29].
We address the downscaling problem for sequence. Our objective is transform sequence ("video") of low-resolution precipitation frames into a sequence of high-resolution frames. Despite differences from natural videos, precipitation's 's hourly temporal continuity allows us use video super-resolution techniques to leverage multiple context frames for stochastic downscaling [53 38
Recent efforts to enhance the resolution of climate states like precipitation have relied on deterministic regression methods using convolutions or transformers. However, repeeurrsooution is a ane-to-many mapping with continuum of "correct" answers. Supervised learning for these problems often leads to visual artifacts from mode averaging, where the network predicts an average of incompatible solutions, causing blurriness in visual data 30 76). Besides visual artifacts, mode averaging can have more dramatic implications in climate and weather modeling, such as the underestimation of extreme precipitation [44) which is mainly induced by regional weather patterns the unresolved scale. A natural alternative to supervised super-resolution methods [12 (2 775, I [23) to prevent mode averaging is conditional generative modeling, which captures multimodal conditional distributions.
To that end, recent works propose using generative adversarial networks (GANs) for precipitation downscaling. These methods often face challenges, tending to converge specific modes of the data distribution and occasionally fixating on isolated points in extreme cases. Despite their perceptual appeal, the scientific utility of super-resolution requires accurate modeling of the statistical distribution of high-resolution data given low-resolution input, which GANs typically fail to capture.
Video Diffusion (STVD for precipitation downscaling. We use deterministic regression model ("downscaler") for coarse prediction, refined by a conditional video diffusion model that captures the residual error for adding fine-grained details. Both modules rely on spatio-temporal factorized attention to process the input sequence. Diffusion models are well- suited for precipitation downscaling as they successfully capture high dimensional and multimodal distributions, alleviating a key drawback of GAN- methods for climate science applications.
We propose
'This the climate science terminology super-resolution. thub. www.e//nntttaa//eeeeee Codee: httppp:///iiibb..ooco/ooaadddodd

---

This study highlights the capability of conditional diffusion models to meet the specific needs of statistical precipitation downscaling, with our key contributions being:
1. We introduce novel framework for temporal precipitation downscaling using diffusion models. Our model combines a deterministic downscaling module with diffusion- residual module. It leverages spatio-temporal factorized attention to process information from multiple low-resolution frames.
2. Our model outperforms six strong super-resolution baselines across multiple criteria, includ- ing MSE and several distributional metrics. We compare against two image super-resolution models and four video super-resolution models using the FV3GFS global atmosphere simulation dataset 777 [0].
3. Our approach captures key characteristics of precipitation, including extreme precipitation probabilities and spatial patterns of annual precipitation in mountainous regions, which are crucial for domain science applications.
Our paper is structured as follows: we first describe our method (Sec. 2 followed by our experimental findings (Sec. B). Finally, we discuss relevant literature (Sec. ( and its connection to our work. Fig. shows global view of the input and predicted precipitation of our model. The code for our model is available at htttps///gtthiiae
com -aaaandt-lab/TV
Downscaling via Spatiotemporal Video Diffusion
Problem Statement At training time, we assume access to a collection of high-resolution precipi-
tation frame sequences y and their corresponding low-resolution precipitation frame sequences
= Such a low-resolution sequence can be obtained through area-weighted coarsening (39] of the corresponding high-resolution sequence. The dataset is discussed extensively in Sec. Frame
indices are represented by superscripts, where we assume that each sequence consists of T frames for simplicity. While may be possible to roll out predictions for multiple sequences autoregressively using techniques such as reconstruction guidance [21), we leave this exploration for future work. Our objective is train a model to effectively downscale, or super-resolve given sequence 00:7 with yu? serving as the target. We use "downscaling" and "super-resolution" interchangeably.
More formally, let x' RCx and y' B RIxslxxW represent individual low-resolution and high-resolution frames. Here, $ E N denotes the downscaling factor, is the number of channels (quantities used input to the model to characterize the atmospheric state in each low-rusolution grid cell so as to add skill to the precipitation prediction), and H, W indicate the height and width of the low-resolution frame. For our study, adopt a downscaling factor of 8 and have C 12 total low-resolution channels. In addition to the low-resolution precipitation state, we provide eleven channels of information to the model, such topography, wind velocity, and surface temperature; see A.2 for details.
Solution Sketch
Our approach treats the downscaling problem conditional generative modeling task. We devise a model to learn the conditional distribution of high-resolutin precipitation frames, incorporating contextual information from the fow-resolution precipitation frame sequence.
Our proposed solution, SpatioTeempors Video Diffusion (STVD) (Fig. 2 relies on two modules: a deterministic downscaler and a stochastic component based on conditional diffusion models 20 63] both using spatio-tempora factorized attention. The first module uses a UNet with factorized attention to integrate information from low-resolution frame sequence, resulting in initial prediction frame sequence y0, The second module is a conditional diffusion model that stochastically generates a sequence of additive residual frames D: which serves to add fine-grained details to the initial prediction. Together, these two modules produce a high-resolution frame sequence 50:7 5 + TO:T Both modules are trained end-to-end.
Decomposing the prediction into a deterministic mean and a stochastic residual is inspired by predictive-coding-bass video decompression. This approach aims to predict a sequence of video frames while compressing the sparse residuals 22 74] which are easier to model than dense frames.
Similarly, it is easier to generate residuals than dense images when using diffusion models 77 41.
3

---

(Ground Truth)
STUD (Ours)
PERT (40)
RVRT [36]
VRT (44
[411
VOM [21]
Swin-ft (35
Coarse Grid
Timastag
Timestap 2
Timattop
Timestop
Timastap 5
Figure 3: A qualitative comparison between our proposed model and top baseline for precipitation event associated with a cold front impinging on the Northern California coast and then the Sierra mountain range (coastline marked in hazy white). Fig. plots the regional topography. The time interval between adjacent frames 3 hours; the plotted region is 1000 1000 km. Our model resolves the fine-grid precipitation structure better than the considered baselines. See A.3 for full-page high quality samples from Himalayas and Sierra.
In what follows, we first describe our overall probabilistic framework for downscaling. Then, we discuss the deterministic module along with spatio-temporal attention, followed by the remaining residual prediction module based diffusion generative modeling. See A.T for architecture details.
2.1 Probabilistic Modeling of Downscaling
Given sequence of low-resolution frames 0: and the corresponding high-resolution frames O:T we aim to learn a parametric approximation pe the conditional distribution 0y7
x x s
Pe OyT: x0:T") Importantly, we do not assume independence across time; each generated frame y can depend on all other generated frames. The generated high-resolution frame sequence is conditioned on the entire low-resolution frame sequence, capturing long-range temporal correlations and enhancing the fidelity and cohesion of the high-resolution reconstruction.
As noted earlier, the likelihood Po (y-:7" xx)TT) is modeled using deterministic downscaler and residual diffusion model. We will discuss how the model parameters 0 (Ø, v) decompose into those for downscaler (ő) and a diffusion model (15).
2.1.1 Deterministic Downscaling
Our first module is a deterministic downscaler that predicts an initial high-resolution frame sequence 50. Ms (x"-T) where M4 is network generating a deterministic high-resolution prediction with parameters é. We perform bicubic interpolation each frame of x D: before passing the sequence through the network Pa. Since the diffusion network operates on high-resolution inputs (i.c. denoising the high-resolution residuals), this choice allows us use the same UNet [52] architecture (with different weights) for both the downscaling module As and the residual diffusion module. This enables to easily share features across the modules via concatenation. See for further details.
Importantly, A4 incorporates a temporal attention mechanism that allows any frame time I, or its corresponding feature map, attend to all context frames from 0 to T. This architecture enables the concurrent inference of all frames within the sequence O0T The attention weights differ for each frame, allowing for the flexible incorporation of information across time.

---

2.1.2 Stochastic Residual Modeling via Diffusion
After computing the initial prediction y0:T, finer details are modeled by residuals learned from conditional diffusion model. Our final stochastic high-resolution frame sequence y0:2 is generated by sampling an additive residual sequence r ro: from this model: yo y ro:f" Thus, we seek to model the residuals p0:2" = y - y0rr" Our diffusion model generates the entire residual sequence gof concurrently, with the generation of each residual r' dependent on the others. This is achieved via a UNet architecture with apatio-temporal attention, similar to the mechanism used for the deterministic downscaling module. See A. for further details.
To model the distribution of ro-7" we use DDPM 20 To that end, we introduce a collection of latent variables O... where the lower subscripts indicate the denoising diffusion step. In the forward process, the latent variable 1 is created from r via additive noise. In the reverse process for generation, denoising model (with parameters vb) is trained to predict 3-1 from rD:TT N denotes the total number of denoising steps. Note that r0: = r i.e. the first diffusion step corresponds to the true residual. Additionally, r0 implicitly depends on the downscaler parameters o, allowing us simultaneously optimize all model parameters
(Φ, () within the context of diffusion modeling.
As is standard diffusion models [20]. we parameterize the reverse process via a Gaussian distribu- tion with mean determined by neural network Mv.
O.T cc) = N N (1 (roT n,, ,n,c), 71); Pe (ro:T) (22-11
where Mp is a denoising network and is a hyperparameter for variance. The diffusion model directly accesses the context = (x":T 5 ), and is implicitly conditioned on x via concatenation of feature maps from the downscaler module. As the downscaler, bicubically upsample "h:
before channel-wise concatenation with yo: to match the dimensions when forming c.
2.1.3 Loss Function
To train our model, we use the angular parametrization suggested by [59]. Specifically, this results in the diffusion loss of the form
T Σ 1=0
L (v. Egar your m.e
l|
My (r n, 1)
(2)
where (0, I), n is sampled uniformly from {1, ..., N), and the sequences OOT yor sampled from the training distribution. Here, = (x":1" U:T yoT)) where yu: M (x":)) The scalars o are used to define On a, Training and inference
o Im= B:) and o are concurrent across multiple frames due to spatio-temporal attention. Alg. training and sampling strategy under the angular parametrization. We use DDIM sampling (62] to generate frame residuals with fewer diffusion steps.
the
2.1.4 Network Architecture
Both the downscaler and the conditional diffusion model employ a UNet backbone with similar architectures and key adaptations to the attention mechanism (see A. The downscaler takes the multi-channel input frames (xx:T,, ), yielding an initial estimate (y ). The diffusion UNet conditions on diffusion step n and concatenates feature maps from the downscaler with its own. The concatenated
input to the diffusion UNet 0.T D.T and r (.T), ) along with the conditioning variables (diffusion step and the feature maps from downscaler), yields the output
Computing full attention for temporal coherence across the entire video data cube is very expensive for processing long sequences or high-resolution inputs. To optimize efficiency, we decouple attention between spatial and temporal dimensions, use linear variant of self-attention [26] for non-bottleneck layers (where the effective number of "tokens" for attention is relatively large), focus spatial attention on localized patches (instead of the entire feature map, which could be wasteful), and calculate per-channel temporal attention in large spatial dimensions (namely, the ultimate and penultimate expansion and contraction layers of UNet). These modifications dramatically reduce the time complexity and memory footprint of these transformer blocks.
5

---

Reallsm- Reallem-Bisttrtien Distortion Tradeoff
Algorithm 1: Training STVD
Algorithm 2: Sampling STVD
while not converged do
Get an equally spaced increasing sub-sequence 7 of length K & N:
Sample x " and y uuT
~ (0, 1.2, N);
yo
pip :
(0, 1):
y yor = Mo (x 0:T)
c=
=x01 y DTT);;
ro y 0:T goT,
F for in reversed( do
DuT
NO
v= = ane On DTT
r 0.T an off
+ ONE
v = My (I u.
c):
c (x"i,, .U
:..T):
f ag ro on V;
Figure 4: Tradeoff between mean square error and percentile error (see Sec.B) Inference at Himalayan re-
My (Il.T, n, n,
é == (To - -));
L = 1|v - -V11
n- 18 On-1é;
D.T
(+),) (1), Vo L1
go. 50: r0
O.T
gion (see Figs. 6 and 12
3 Experiments
We conduct comprehensive evaluation of our proposed method, Spatio Temporall Video Diffusion (STVD), against six contemporary ofate-of-the-art baselines. The first two baselines are image super- resolution models based on the Swin Vision Transformer (Swin- -R) (35) and its residual diffusion variant (Swin-IR-Diff). The next two baselines are video super-resolution models grounded in vision transformer architecture (VRT) 34 and its recurrent variant (RVRT) 36. The latter incorporates guided deformable attention for clip alignment, enhancing its temporal modeling capabilities. We compare against another video-superr-rssoutiin baseline (PSRT) (60) which also relies on the transformer architecture but uses multi-frame attention groups Finally, we compare against a video diffusion baseline (VDM) 21. Fig. shows global view of the input and predicted precipitation.
We perform ablation studies in three configurations. In the first two, we experiment with the input sequence length. While our proposed model uses context length of 5 frames, we also conduct experiments with frames and frame (STVD-3 and STVD-1). Note that using a single context frame ablates for the temporal attention block as well. The third ablation (STVD-Single) involves removing the additional input channels (i.e. only providing the model with the low-resolution precipitation sequence) to assess their impact on performance metries. In summary, our experiments demonstrate that our method outperforms all baselines across all metrics considered. Additionally, our ablation studies highlight the importance of temporal context and additional climate inputs.
Dataset Our dataset derives from an I 11meemer initial condition ensemble of 13-month simulations using a global atmosphere model, FV3GFS, run 25 km resolution and forced by climatological sea surface temperatures and sea ice. The first month of each simulation is discarded to allow the simulations to spin up and meteorologically diverge, effectively providing 11 years of reference data (of which first 10 years are used for training and the last year for validation). FV3GFS, developed by the National Oceanic and Atmospheric Administration (NOAA), is version of NOAA's operational global weather forecast model (177) I0)..
Three- average data were saved from this entire simulation, which used 25 km horizontal "fine grid". We further coarsened the selected fields by a factor of 8 to create a 200 km horizontal "coarse grid", resulting in paired data (æ,, Mt), where Te the coarse-grid global state and yı the corresponding fine-grid global state. Our goal is to apply video downscaling to the coarse-grid precipitation field obtain temporally smooth fine-grid precipitation estimates that are statistically similar to the true data. This approach is attractive because many fine-gria precipitation features, such as cold fronts and tropical cyclones, are poorly resolved on the coarse grid but are temporally coherent across periods much longer than 3 hours. We use 12 coarse coarse-grid input fields, including precipitation, topography, and horizontal vector wind at various levels. See for the list of included atmospheric variables. FV3GFS uses a cubed-sphere grid, where the surface of the globe is divided into six tiles, each of which covered by an S array of points. Our data fields reflect this structure with S 48 for the 200 km coarse grid and S 384 for the 25 km fine grid.

---

104
Target
Table I: Quantitative comparison between our method and other competitive baselines. EMD represents the Earth-Mover Distance, PE denotes the 99.999th percentile error and SAE is the spatial autioollauttoorlaaiionn error. Overall, our proposed method (STVD) outperforms the baselines across all metrics. In our ablation study, the exclusion of additional side information STVVD-single or decrement in context length (STVD-3 and STVD- 1) appreciably degrades performance.
SIVD (Ours)
Swin-R oif (41)
10
[34]
[36]
PSAT [60]
Denzity
10
Swin-iR (35)
10
10-1
0.00
0.25
0.50
0.75
1.00
1.25
1,50
1.75
2,00
Precipitation
1a-2
CRPS
MSE
LMD
PE
SAE
Figure 5: Distributions of the fine-grid three- hourly average precipitation, for all gridpoints
(10 5)
[10
(10
" [10
(10 %
(ours)
0.59
PSRT CD
2.15
0.66
4.21
3.8
6.24
around the globe. The Swin-IR baseline over- estimates large precipitation events, whereas all other baselines underestimate key extreme and rare precipitation events. Our model aligns best with the fine-grid ground truth than any the other model. This is also evident with the the EMD
RVKT 50
3.35
1.73
4.33
3.6
7.39
VRT 84
3.58
1.74
4.81
4.0
7.39
Swi--R-Dir
2.29
1.94
7.7U
VDM 21
2.21
0.73
12. 70
6.4
8.84
Swin 1R 83
2.36
2.29
17.40
23.40
18.9
STVD-sigh STVD-3
1.81
0.62
4.64
2.3
6.09
1.96
0.68
4.04
2.6
4.09
and PE metries discussed in Tab. D and Sec.
STVD-1
2.05
The application presented here serves as a pilot for broader uses of our methodology. Fine-grid simulations are significantly more computationally expensive than coarse-grid simulations (an 8-fold reduction in grid spacing requires almost 1000x more computation), so a coarse-grid simulation with super-resolved details in desired regions could be highly cost-effective for many applications.
During training, our model randomly selects data from one of the six tiles. This strategy ensures that the model learns from the diverse spatial contexts and weather regimes that produce precipitation worldwide. fost-training, for localized analysis, we selectively sample super-resolved precipita- tion channels from regions with complex terrain, such as California (Fig. 6). These regions can systematically pattern the precipitation on fine scales. This analysis helps us to see how well the super-resolution can learn the time- mean spatial patterns (e.g. precipitation enhancement on the windward side of mountain ranges and lee rain shadows) the fine-grid reference data.
Training and Testing Details We downscale a sequence of precipitation frames from FV3GFS output by factor of 8. Our approach (STVD) trains on 5 consecutive frames that are downscaled jointly. We optimize our model end-to-end with single diffusion loss using Adam (2 with an initial learning rate of 1 x 10 10-4, decaying to x 10-? with cosine annealing during training, executed on an NVidia RTX A6000 GPU. The diffusion model is trained using v-parametrization [59], with fixed diffusion depth (N = 1400). Random tiles extracted from the cube-sphere representation of Earth, with dimensions 384 in high-resolution and 48 in low-resolution, are used during training. We train for one million steps, requiring approximately 7 days on a single node (slightly less for ablations). We use a batch size one, apply logarithmic transformation precipitation states, and normalize to the range 1], During testing, we employ DDIM sampling with 30 steps on an Exponential Moving Average (EMA) variant of our model (for full frame size), with a decay rate of 0.995.
Baseline Models We compare our generative setup against several recent nigh-performing transformer- video super-resolution models. These models are trained deterministically. The first, Video Restoration Transformer (VRT) [34], allows for parallel frame prediction and long-range temporal dependency modeling. The second, recurrent VRT (RVRT) [36], incorporates guided deformable attention for effective clip alignment, enhancing its temporal modeling capabilities. The third, PSRT 600, removes the aligmment module and modifies the attention window. We also compare against the recent Video Diffusion Model (VDM) [1 which employs global quadratic attention.
To assess the benefits of multi-frame downscaling, we compare with Swin-IR [35, super-resolution model that hamesses Swin Transformer blocks. However, Swin-IR is trained in a supervised fashion. Thus, as a generative baseline, we compare to Swin-IR-Diff. This model generates a deterministic prediction using SWin-IR (35), followed by modeling stochastic residual
popular image
7

---

Pine gold Prediitation
Preciphttion
IGroued Truth)
(0um)
(Grousd trush
2300
Moue
Mice
Mo00
5080
Figure 6: Precipitation over regions (left: Himalayas; right: Northern California coast, same region as Fig. B). averaged across a year, for our STVD model and the ground-truth For each half, the topography of the region shown the corresponding top-left whereas the predicted annual average is shown in the corresponding bottom-right. Annually-averaged precipitation is an important indicator of water availability in region. STVD successfully captures many details of the precipitation that are tied to local topography and are too fine to be resolved the coarse-grid data.
using diffusion. This baseline is inspired by concurrent work on single-image radar-reflectivity downscaling 41 where UNet is used instead of Swin-IR. See A.2 for details.
Evaluation Metrics We evaluate our model differently from standard vision tasks. In addition to Mean Square Error (MSE), which measures the average squared difference between predicted and actual values but lacks full distributional information, we use several distribution-level metrics for more meaningful comparison. One such metric is the Continuous Ranked Probability Score (CRPS) 8 (66), which assesses the discrepancy between the predicted cumulative distribution function and the observed data. We compute CRPS over 10 stochastic realizations of our predictions. Fig. visualizes several of these samples.
Furthermore, given the distinctive light-tailed exponential distribution of the precipitation climate state, it is crucial to ensure that downscaling does not significantly alter the distribution of pre- cipitation rates. This necessitates two additional metries. First, we compute the Earth Mover (or I-Wasserstein) Distance [54] to quantify the agreement between the target and predicted global precipitation distributions, which are strongly affected by high-resolution details. Second, we focus on tail events and extreme precipitation by considering the 99.999th percentile error (PE), providing a nuanced understanding of the model's performance on rare and extreme precipitation events.
To further assess the spatial fidelity of our downscaling approach, we use the Spatial Autocorrelation Error (SAE) [68]. This metric calculates the mean absolute error between the spatial autocorrelation of the predictions and ground truth. Low SAE ensures that the spatial patterns and the fine structure in precipitation data are preserved during downscaling, which is critical for accurate climate modeling.
Qualitative and Quantitative Analysis Tab. provides quantitative evaluation comparing method with state-of-the-art baselines and ablations. Our model (STVD) performs strongly across all metrics, outperforming all baselines. We highlight the distributional characteristics in Fig. [ Swin-IR overestimates precipitation, while all other baselines underestimate This discrepancy undesirable, as poor performance on rare and extreme precipitation events can negatively imapet disaster mitigation policies. In contrast, our method closely matches the precipitation distribution, as measured by PE and EMD.
Using only precipitation as an input (STVD- results in slightly worse performance across all metrics, indicating the predictive value of additional inputs. In contrast, our ablation model STVD-1, which lacks full sequence information, performs significantly worse, highlighting the importance of temporal attention in our approach (which decays function of time lag shown in Fig. 14).
Figs. 3 I and depict the performance of our model compared to other baselines on examples of precipitation feature interacting with mountainous terrain. Our model generates high quality results

---

(Ground Truth)
Sample
Sample
Sample 3
Sample 4
Sample
Sample
Varlance
Coarse Grid
Timastag
Timestap 2
Timattop
Timestog
Timastap 5
Figure 7: A visualization of the stochastic samples predicted by STVD for a given coarse-grid data. The precipitation event is the same as depicted in Fig- Additionally, we also plot variance map over the set of these samples to analyze the stochasticity better. Red regions correspond to high variance whereas blue regions correspond low variance. Model stochasticity seems to be meaningful since the variance is large where mean precipitation is large.
which preserves most patterns with high degree of similarity. PSRT and RVRT produce slightly more diffuse precipitation features, while Swin-IR produces slightly more pixelated features.
Fig. sshows annuall--aveageel precipitation from the patches in Figs. 3 and 12 Accurately capturing the fine-grid structure of time-mean precipitation crucial for assessing long-term water availability. Our method (which includes fine-grid topography as training input) effectively replicates the ground truth. This includes the strength and narrow spatial structure of high precipitation bands along the Northern California coastal mountains and the Sierras. These features are not resolved by the coarse-gre grid inputs to the super-resolution. 3A.3 for full-page high-resolution samples and spectral analysis. Fig. 13 reveals that the spectra for baselines decay more rapidly than for STVD.
Realism- Diitortiooto Tradeoff' Distortion metrics such as MSE often conflict with perceptual quality, where reducing distortion typically degrades perceptual realism 77. In our context, this tradeoff translates to balancing MSE and PE. While MSE captures the average accuracy of predictions, PE represents the model's ability to reproduce extreme events, thereby serving as proxy for realism. Realism in climate modeling refers to the accurate representation of extreme weather patterns, which are crucial for applications like flood forecasting and disaster mitigation. PE is distributional criterion that effectively captures these tail events, offering robust measure of realism. Fig. illustrates this tradeoff, with darker colors corresponding to fewer STVD sampling steps. As the number of diffusion sampling steps increases, MSE tends to rise slightly, but PE decreases significantly. Depending on the application, this tradeoff may potentially be exploited by practitioners. Essentially, the conditional mean minimizes MSE, any deviation from it increases MSE-ever even if the deviation appears more realistic. As for sampling steps, fewer steps correspond to larger time increments in the diffusion process. At one extreme, single step predicts the conditional mean, minimizing MSE. Conversely, more sampling steps more accurately simulate diffusion, generating diverse, realistic samples that increase MSE while reducing PE.
Related Work
Diffusion Models Diffusion models 61 20
40 are class of generative models based iterative denoising process. Closely related to work are diffusion models for video. Recent
64

---

models 73] generate deterministic next-frame predictions autoregressively with additional residuals generated by a diffusion model, or generate videos directly in pixel space [ 69 21 or in latent space 6 S While some works on video diffusion 6 21 employ video super-resolution as step
in the overall modeling process, our work focuses exclusively on the video super-resolution task, particularly within the context of precipitation downscaling.
Resol--esolutton Within the computer vision community, the paradigm for single image super- resolution has shifted from classical approaches [4]
13 to deep learning based methods [71]. Gen-
erative approaches, like cascaded diffusion [49 55 SR3 [56]. and DifiPIR (78 employ diffusion
models for image super-resolution. However, these are unable to leverage temporal context. On
the other hand, many approaches for video super-resolution have been proposed 9 14 22]. For a more comprehensive overview, see (38). Recent models of note include the transformer-based
models PSRT [60] and VRT 3. as well as the recurrent variant RVRT (37, which focuses on parallel decoding and guided clip aligmment. We emphasize that these state-of-the-an approaches are deterministic, where our approach is generative. This allows us to preventing mode averaging and to produce more realistic samples, which is particularly critical in the context of precipitation modeling.
Data-driven weather and climate modeling Recent years have seen advancements in data-driven climate and weather modeling [2), with models like GraphCast [29]. GenCast [48], and Four- CastNet [46] providing forecasts that are competitive with meteorological methods while being significantly faster. Rather than replacing numerical forecasting methods, our approach seeks to augment their capabilities by downscaling coarse-grid predictions.
While downscaling for climate and weather has been approached using techniques based on domain knowledge [25] [4], we focus here on data-driven approaches. [68] draw inspiration from FRVSR [57, adopting an iterative approach that uses the high-resolution frame estimated in the previous step as input for subsequent iterations. 772 employed Fourier neural operators for downscaling at arbitrary resolutions. [T6] generate physically consistent downscaled climate states, using a softmax layer to enforce conservation laws. These approaches, though, deterministic and trained by minimizing the MSE, thus lacking the realism and uncertainty quantification provided by a generative approach.
In terms of generative approaches, concurrent work [1] employs diffusion models for downscaling climate states. The use of GANs has also been pervasive in downscaling and precipitation prediction 32 47
50 15 70). However, these baseb approaches inherit the mode collapse and training difficulties present in all GAN- based models []. Here, we highlight that these approaches are applied at each frame and cannot incorporate temporal information is done in our model.
17
Beyond downscaling, ( demonstrate the diffusion model's 's efficacy in synthesizing full rain density from vorticity inputs. Additionally, [19] uses a diffusion model for downscaling solar irradiance. These models are also used for probabilistic weather forecasting and nowcasting [3]
5 Conclusion
We propose a video super-resolution method for probabilistic precipitation downscaling. Our model, STVD, deterministically super-resolves a given low-resolution frame sequence and then stochastically models the residual details via diffusion. Our model effectively resolves how fine-grid precipitation features, generated as weather systems, interact with complex topography based on temporally coherent coarse-grin -rii information. Our method outperforms several competitive baselines range of quantitative metrics. This is an important step towards designing effective statistical downscaling methods, providing highly localized information for planning against extreme weather events, such as floods or hurricanes in a warming climate, using tractable coarse-grid atmospheric models.
Limitations and Broader Impacts A limitation of our approach is the necessity of paired low- resolution and high-resolution images for training. While this can be done once prior training. designing methods that only require the low-resolution states is an interesting challenge. In terms of broader impacts, our approach could potentially have harmful consequences if adopted blindly to new dataset, where distribution shift could cause the model performance to degrade, potentially leading to underestimation of extreme weather risks such as droughts or floods. To mitigate these risks, the model should be re-trained and rigorously evaluated the dataset of interest.
10

---

Acknowledgments and Disclosure of Funding
We thank Kushagra Pandey and Yibo Yang for valuable feedback. Prakhar Srivastava was supported by the Allen Institute for AI summer internship for much of this work. Stephan Mandt acknowledges support from the National Science Foundation (NSF) under an NSF CAREER Award IIS-2047418 and IIS-2007719, the NSF LEAP Center, by the Department of Energy under grant DE-SC0022331, the IARPA WRIVA program, the Hasso Plattner Research Center at UCI, and by gifts from Qualcomm and Disney. Gavin Kerrigan is supported in part by the HPI Research Center in Machine Learning and Data Science at UC Irvine.
References
[1] Addison, H., Kendon, E., Ravuri, S., Aitchison, L., Watson, P.A.: Machine learning emulation of local-scale uk climate model. arXiv preprint arXiv:2211.1611 (2022)
[2] Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang. S.J., Toderici, G.: Scale-space flow for end-to-end optimized video compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8503-8512 (2020)
[3] Arora, S., Risteski, A., Zhang, Do gans learn the distribution? some theory and empirics. In: International Conference on Learning Representations (2018)
[4] Bascle, B., Blake, A., Zisserman, A.: Motion deblurring and super-resolution from an image sequence. In: Computer Aision-ECCv'OO 4th European Conference on Computer Vision Cambridge, UK, April 15-18, 1996 Proceedings Volume 4. pp. 571-582 Springer (1996)
[5] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M.. Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al.: Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15111 (2023)
[6] Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align your latents: High-resolution video synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22563-22575 (2023)
[7] Blau, Y., Michaeli, T.:
The perception-distortinn tradeoff. In: Proceedings of the IEEE confer- computer vision and pattern recognition. pp. 6228-6237 (2018)
[8] Brown, T.A.: Admissible scoring systems for continuous distributions. (1974)
[9] Chan, K.C., Wang. X., Yu, K., Dong, C., Loy, C.C.: Basievsr: The search for essential components in video super-resolution and beyond. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4947-4956 (2021)
[10] Community, U.: Ufs weather model (Jan 2021). doi (0000000888//zmodd44466099999 org/10. 5281/zenooo 4460292
https://doi
[11] Dai, T., Cai, J., Zhang, Y., Xia, S.T., Zhang, L.: Second-one Recond-order attention network for single image
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11065-11074 (2019)
[12] Dong, C., Loy, C.C., He, K., Tang. X.: Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(2), 295-307 (2015)
[13] Farsiu, S., Robinson, M.D., Elad, M., Milanfar, P. Fast and robust multiframe super resolution. IEEE transactions on image processing 13(10), 1327-1344 (2004)
[14] Fuoli, D., Gu, S., Timofte, R.: Efficient video super-resolution through recurrent latent space propagation. In: 2019 IEFF/CVF International Conference on Computer Vision Workshop (ICCVW). pp. 3476-3485. IEEE (2019)

---

[15] Gong, A., Li, R., Pan, B., Chen, H., Ni. G., Chen, M.: Enhancing spatial variability representa- tion of radar noweasting with generative adversarial networks. Remote Sensing 15(3)). 3306 (2023)
[16] Harder, P., Yang, Q., Ramesh, V., Sattigeri, P., Farmandez-Garcia, A., Watson, C., Szwarcman, D., Rolnick, D.: Generating falysicall-cooniiteen high-resolution climate data with hard- constrained neural networks. arXiv preprint urXiv:2208.0544 (2022)
[17] Harris, L., McRae, A.T., Chantry, M., Dueben, P.D., Palmer, T.N.: A generative deep learning approach to stochastic downscaling of precipitation forecasts. Journal of Advances in Modeling Earth Systems 14())), e2022MS003 120 (2022)
[18] Harvey, W., Naderiparizi, S., Masrani, V., Weilbach, C.D., Wood, E: Flexible diffusion modeling of long videos. In: Advances in Neural Information Processing Systems (2022)
[19] Hatanaka, Y., Glaser, Y., Galgon, G., Torri, G., Sadowski, P.: Diffusion models for high- resolution solar forecasts. arXiv preprint arXiv:2302.00110 (2023)
[20] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33, 6840-6851 (2020)
[21] Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, DJJ.: Video diffusion models. arXiv preprint arXiv:2204.03458 (2022)
[22] Huang, Y., Wang, W., Wang, L.: Bidirectional recurrent convolutional networks for nulti-frame
super-resolution. Advances in neural information processing systems 28 (2015)
[23] Kappeler, A., Yoo, S., Dai, Q., Katsaggelos, A.K.: Video super-resolution with convolutional neural networks. IEEE transactions on computational imaging 2(2), 109-122 (2016)
[24] Karger, D.N., Nobis, M.P., Normand, S., Graham, C.H., Zimmermann, N.E.: Chelsa-trace2 Ik- high-resolution (1 km) downscaled transient temperature and precipitation data since the last glacial maximum. Climate of the Past 19(2), 439-456 (2023)
[25] Karger, D.N., Wilson, A.M., Mahony, C., Zimmermann, N.E., Jetz, W.: Global daily km land surface precipitation based on cloud cover-informed downscaling. Scientific Data 8(1), 307 (2021)
[26] Katharopoulos, A., Vyas, A., Pappas, N., Fleuret, F.: Transformers are rnss: Fast autoregressive transformers with linear attention. In: International conference on machine learning. pp. 5156- 5165. PMLR (2020)
[27] Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1646-1654 (2016)
[28] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
[29] Lam, R., Sanzhez-Gonzalez, A., Willson, M., Wirnsberger, P., Fortunato, M., Alet, F., Ravuri, S., Ewalds, T., Eaton-Rosen, Z., Hu, W., et al.: Learning skillful medium-range global weather forecasting. Science 38276)77), 1416-1421 (2023)
[30] Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1681-4690 (2017)
[31] Leinonen, J., Hamann, U., Nerini, D.,, Germann, U., Franch, G.: Latent diffusion models for generative precipitation noweasting with accurate uncertainty quantification. arXiv preprint 1EXi9 (2023)
[32] Leinonen, J., Nerini, D., Berne, A.: Stochastic super-resolution for downscaling time-evolving atmospheric fields with generative adversarial network. IEEE Transactions on Geoscience and Remote Sensing 59(9), 7211-7223 (2020)
12

---

[33] Li. L., Carver, R., Lopez-Gomezz I., Sha, F., Anderson, J.: Seeds: Emulation of weather forecast ensembles with diffusion models. arXiv preprint arXiv:2306. 14066 (2023)
[34] Liang, J., Cao, J., Fan, Y., Zhang, K., Ranjan, R., Li, Y., Timofte, R., Van Gool, L.: Vrt: A video restoration transformer. arXiv preprint arXiv:2201.12288 (2022)
[35] Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L.,, Timofte, R.: Swinir: Image restoration using swin transformer. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1833-1844 (2021)
[36] Liang, Fan, Y., Xiang, X., Ranjan, R., llg, E., Green, S., Cao, J., Zhang, K., Timofte, R., Gool, L.V.: Recurrent video restoration transformer with guided deformable attention. Advances in Neural Information Processing Systems 35, 378-393 (2022)
[37] Liang, J., Fan, Y., Xiang, X.,
Ranjan, R., 1lg. E.., Green, S., Cao. J., Zhang, K., Timofte, R., Van Gool, L.: Recurrent video restoration transformer with guided deformable attention. In: Advances in Neural Information Processing Systems (2022)
[38] Liu, H., Ruan, Z., Zhao, P., Dong, C., Shang, F., Liu, Y., Yang, L., Timofte, R.: Video super- resolution based deep learning: comprehensive survey. Artificial Intelligence Review 55(8), 5981-6035 (2022)
[39] Mahecha, M.D., Gans, F., Brandt, G., Christiansen, R., Cornell, S.E., Fomferra, N., Kraemer, G., Peters, J.. Bodesheim, P., Camps- Valls, G., et al.: Earth system data cubes unravel global multivariate dynamics. Earth System Dynamics 11(1), 201-234 (2020)
[40] Manduchi, L., Pandey, K., Bamler, R., Cotterell, R., Däubener, S., Fellenz, S., Fischer, A., Gärtner, T., Kirchler, M.. Kloft, M., et al.: On the challenges and opportunities in generative ai. arXiv preprint arXiv:2403.000 0000500500505 (2024)
[41] Mardani, M., Brenowitz, N., Cohen, Y., Pathak, J., Chen, C.Y., Liu, C.C., Vahdat, A., Kashinath, K., Kautz, J., Pritchard, M.: Generative residual diffusion modeling for km-scale atmospheric downscaling. arXiv preprint arXiv:2309.12211 (2023)
[42] Mooers, G., Pritchard, M., Beucler, T., Srivastava, P., Mangipudi, H., Peng, L.., Gentine, P., Mandt, S.: Comparing storm resolving models and climates via unsupervised machine learning. Scientific Reports 13(1,, 22365 (2023)
[43] Pandey, K., Mandt, S.: A complete recipe for diffusion generative models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4261-4272 (2023)
[44] Pandey, K., Pathak, J., Xu, Y., Mandt, S., Pritchard, M., Vahdat, A., Mardani, M.: Heavy-tailed diffusion models. arXiv preprint arXiw:2410.14171 (2024)
[45] Pandey, K., Rudolph, M., Mandt, S.: Efficient integrators for diffusion generative models. In: International Conference on Learning Representations (2024)
[46] Pathak, J., Subramanian, S., Harrington, P., Raja, S., Chattoppadyayy, A., Mardani, M., Kurth, T. Hall, D., Li, Z.., Arizzadenesheli, K., et al.: Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.1 1 1214 (2022)
[47] Price, I., Rasp. S.: Increasing the accuracy and resolution of precipitation forecasts using deep generative models. In: International conference artificial intelligence and statistics. pp. 10555-10571. PMLR (2022)
[48] Price, I., Sanchez-Gonzalez, A., Alet, F., Ewalds, T., El-Kadi, A., Stott, J., Mohamed, S., Battaglia, P., Lam, R., Willson, M.: Gencast: Diffusion- based ensemble forecasting for medium- range weather. arXiv preprint arX1v:2312.15796 (2023)
[49] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hicrarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06122 (2022)
[50] Ravuri, S., Lene, K., Willson, M., Kangin, D., Lam, R., Mirowski, P., Fitzsimons, M., Athanassi- adou, M., Kashem, S., Madge, S., et al.: Skilful precipitation nowcasting using deep generative models of radar. Nature 59777878), 672-677 (2021)
13

---

[51] Reichstein, M., Camps-Vare Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., Prabhat, f.: Deep learning and process understanding for data-driven earth system science. Nature 566(7743), 195-204 (2019)
[52] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assister Intervention-MIC 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. pp. 234-241. Springer (2015)
[53] Rota, C., Buzzelli, M., Bianco, S., Schettini, R.: Video restoration based on deep learning: comprehensive survey. Artificial Intelligence Review pp. 1-48 (2022)
[54] Rubner, Y., Tomasi, C., Guibas, I.J.: metric for distributions with applications to image databases. In: Sixth international conference on computer vision (IEEE Cat. No. 98CH36271). pp. 59-66. IEEE (1998)
[55] Saharia, C., Chan, W., Saxena, S., Li, L.., Whang, J., Denton, E.,, Ghasemipour, S.K.S., Gontijo- Lopes, R., Ayan, B.K., Salimans, T., Ho, J., Fleet, D.J., Norouzi, M.: Photorealistic text-to- image diffusion models with deep language understanding. In: Advances in Neural Information Processing Systems (2022)
[56] Saharia, C., Ho, J., Chan, W. Salimans, T., Fleet, DJ., Norouzi, M.: Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence pp. 1-14 (2022)
[57] Sajjadi, M.S., Vemulapalli, R., Brown, M.: Frame-recurrent video super-resolution. In: Pro- ceedings of the IEEE conference computer vision and pattern recognition, pp. 6626-6634 (2018)
[58] Salathé, E.P., Steed, R., Mass, C.F., Zahn, P.H.:: A high-resolution climate model for the us pacific northwest: Mesoscale feedbacks and local responses climate change. Journal of climate 21(21), 5708-5726 (2008)
[59] Salimans, T., Ho, J.: Progressive distillation for fast sampling diffusion models. ArXiv abs/2202.00512 (2022)
[60] Shi, S., Gu, J., Xie, L.., Wang, X., Yang, Y., Dong, C.: Rethinking alignment in video super- resolution transformers. Advances in Neural Information Processing Systems 35, 36081-3600 (2022)
[61] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamies. In: International Conference on Machine Learning. pp. 2256-2265 (2015)
[62] Song, J., Meng, C., Ermon, Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
[63] Song. Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems 32 (2019)
[64] Song. Y. Sih--Dickttein, J., Kingma, D.P., Kumar, A., Emnon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. In: International Conference on Learning Representations (2021)
[65] Stevens, B., Satoh, M., Auger, L., Biercamp. J., Bretherton, C.S., Chen, X., Düben, P., Judt, F. Khairoutdinov, M., Klocke, D., et al.: Dyamond: the dynamics of the atmospheric general circulation modeled on non-hydrostatic domains. Progress in Earth and Planetary Science 6(1), 1-17 (2019)
[66] Taillardar, M., Fougères, A.L., Naveau, P., De Fondeville, R.: Evaluating probabilistic forecasts of extremes using continuous ranked probability score distributions. International Journal of Forecasting 39(3), 1448-1459 (2023)
14

---

[67] Tang, J., Niu, X., Wang, S., Gao, H., Wang, X., Wu, J.: Statistical downscaling and dynamical downscaling of regional climate in china: Present climate evaluations and future climate projections. Journal of Geophysical Research: Atmospheres 121( 2110-2129 (2016)
[68] Teufel, B., Carmo, F., Sushama, L., Sun, L., Khaliq, M., Bélair, S., Shamseldin, A., Kumar, D.N., Vaze, J.: Physics-informed deep learning framework to model intense precipitation events super resolution. Geoscience Letters 10(1), 19 (2023)
[69] Voleti, V., Jolicoeur-Martineau, A., Pal, C.: Mevd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in neural information processing systems 35, 23371-23335 (2022)
[70] Vosper, E., Watson, P., Harris, L., McRae, A., Santos-Rodriguez, R., Aitchison, L., Mitchell, D.: Deep learning for downscaling tropical cyclone rainfall to buzard-relevan spatial scales. Journal of Geophysical Research: Atmospheres p. e2022JD038163 (2023)
[71] Wang, Z.,, Chen, J., Hoi, S.C.: Deep learning for image super-resolution: survey. IEEE transactions on pattern analysis and machine intelligence 43(10), 3365-3387 (2020)
[72] Yang, Q. Hermandez-Gareia A., Harder, P., Ramesh, V., Sattegeri, P., Szwareman, D., Watson, C.D., Rolnick, D.: Fourier neural operators for arbitrary resolution climate data downscaling. arXiv preprint arXiv:2305.1 14452 (2023)
[73] Yang, R., Srivastava, P., Mandt, S.: Diffusion probabilistic modeling for video generation. Entropy 25(10), 1469 (2023)
[74] Yang, R., Yang. Y., Marino, J., Mandt, S.: Insights from generative modeling for neural video compression. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
[75] Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y:: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV), pp. 286-301 (2018)
[76] Zhao, S., Song, J., Ermon, S.: Towards deeper understanding of variational autoencoding models. arXiv preprint arXiv:1702.08655 (2017)
[77] Zhou, I.,, Lin, S.J., Chen, J.H., Harris, L.M., Chen, X., Rees, S.I.: Toward convective-scale prediction within the next generation global prediction system. Bulletin of the American Metcorological Society 100(7), 1225-1243 (2019)
[78] Zhu, Y., Zhang, K., Liang. J., Cao, J., Wen, B., Timofte, R., Van Gool, L.: Denoising diffusion models for plug-and-play image restoration. In: Proceedings of the IERF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1219-1229 (2023)
15

---

Precipitation Downscaling with Spatiotemporal Video Diffusion
Supplementary Materials
A.1 Model Architecture
Our architecture is conditional extension of the DDPM 20 and SR3 [56] models. As discussed in Sec. 2 Fig. [ presents the architecture of the proposed denoising and downscaling networks, while Fig. 9 provides detailed description of each component. Before elaborating on the specifics, we define the naming conventions for the parameter choices adopted in this section:
ChannelDim: ResBlock channel dimension for first contractive layer of UNet.
Channe Mhannellultippierss channel dimension multipliers for subsequent contractive layers (including the first layer) in both the downscaling and denoising modules. The expansive layer multipliers follow in reverse.
ResBlock: a standard ResBlock implementation consisting of two blocks, each with a weight-standardiser convolution using 3 x kernel, Group Normalization over groups of 8 and SiLU activation, followed by a channel-adjusting x 1 convolution.
Attention: a standard implementation of quadratic or linear attention, incorporating attention heads, each with a 32-dimensional representation. Using 3 x 3 convolutional kernel, query, key, and value feature maps are generated, resulting in feature maps of size [B x C. X, YJ. where B, T. C, X, and denote batch, time, channel, height, and width, respectively, with H representing the number of heads. These feature maps are rearranged [BATCH, HEAD, CHANNEL, TOKEN]), facilitating self-attention between TOKENS of CHANNEL dimensions. The rearrangement order determines whether spatial or temporal self-attention is performed. Subsequently, the feature maps revert to their original format, upon which separate convolution operation is conducted, followed by layer normalization to project the feature maps back to their original dimensions, akin to their state before the initial convolution within the attention block. Further elaboration on the rearrangement choices for the variants is discussed, adhering to einopy notation:
Q-Spatial: quadratic variant applied in the bottleneck layer; self attends between every pixel of the feature map with the following rearrangement [B T, H x C, X,
Y] [B X T, H, C, X X Y]
Q-Temporal: quadratic variant applied in the bottleneck layer; self attends between feature maps across time with the following rearrangement [B T, H X C, X, -}
-
[B, H. C x X x T]
L-Spatial: linear variant applied in expansive and contractive layers of UNet; self attends between every pixel within a patch of the feature map with the following rearrangement [B P is patch size, starting with 192, halving at each contractive layer and doubling at each expansive layer.
C,
P] [B x т.
Y, C, X P] where
L-meppor::: linear variant applied in expansive and contractive layers of UNet; self attends between feature maps across time in channel factorised manner with the following rearrangement [B X T. X C, X, Y] -> [B, II X X X Y, C, T]
MLP: conditioning on the denoising step n is achieved through this block, which uses 32- dimensional random Fourier features, followed by a linear layer, GELU activation and another linear layer to transform the noise step to a higher dimension.
Cov/TransCov: these are convolutional (3 x kernel) downsampling and upsampling blocks that change the spatial size by factor of 2.
Fig. g illustrates the interaction between the U-Net architecture of the denoising and downscaling networks. The top U-Net depicts the denoising network, while the bottom U-Net depicts the

---

HEP
uBu
UDU
U0U
UDU
UDU
UDU
V0U
U.U
ULU
Uuu
U.U
UJU
Figure 8: The figure depicts the overall model architecture where the top UNet performs diffusion the residual, conditioned on the noise step n, x0:2" you and the context of the bottom U-Net feature map (refer to Eq. (I) in Sec, 2.1p. The bottom UNet the deterministic downscaler. Details each block are shown in Figure
Rodttook
Sedtlot
Restlock
Redtbock
dinccdng
L-spatal Attenton
L-Spoial Attertice
L-ppsa attnnion
L-parial Attrrice
Uneer
GEU
Ul Attertce
Looetiy
Idorityy
L'Toggooa 402 rion
Cow
ToreCore 12
Cc 12
TaraCorv 12
Conmacader
Conmaamping Uni
Uni
Downssmplnng
Upsampiing (DUL)
(JDL)
(U03)
((OX.)
Figure 9: The details of the components of the modules shown in Figure modules correspond to the arrows with the same color in Figure
g the colored arrows in the
downscaling network. The denoising network is conditioned in three ways. First, the network is conditioned the bicubically downscaled low-resolution frames xO: concatenated with both the noisy residual O.:T the downscaler output and 50.7 along the channel dimension. Second, the network also conditioned on the feature maps generated by the downscaler network indicated by the green arrows connecting the downsampling units of both the networks. The L- (Spatial/Temporal Attention blocks from contractive layers of the downscaler network yield feature map that gets concatenated with the inputs of both ResBlocks of the contractive layer of the denoising network. Finally, each contractive and expansive layer of the diffusion UNet gets conditioned on the denoising step n. shown via the black arrows. This conditional embedding for the step is generated through MLP and received by both ResBlocks. The information flows from the noisy residual through the network shown by the blue arrows to predict the angular parameter Both U-Nets have skip connections, indicated by the red arrows, between both ResBlocks of contractive and expansive layers of the same UNet.
Table 2: FV3GFS uses a cubed-sphere grid, in which the surface of the globe divided into six tiles. Each high-resolution tile covers 25 km and is 384 384. Our UNet Encoder and Decoder have 6 layers with a base channel dimension of 64 and multipliers as stated above.
TileSize ChannelDim ChannelHultipliers
384 384
64
1,1,2,2,3,4

---

yd
p
Losa Furction
x
Swii
Diffusiong
FN
Figure 10: An illustration of the training and inference pipelines of Swin-IR-Diff: Similar to Fig. blue blocks represent operations common to both training and inference phases. Red blocks signily operations exclusive training, while green blocks indicate inference-only processes. However, in contrast, it is an image-only downscaler. This model takes in current low-resolution frame which is deterministically downscaled via Swin-IR, followed by modeling of the residual details via conditional diffusion. The model details remain similar to what is described A. with the absence of downssaler-conndiionnng and temporal attention in the diffusion model.
Table Additional Variables in FV3GFS dataset.
Short Name Long Name
Units
CPRATsfc
Surface convective precip. rate
kg/mi/s
DSWRFtoa
Top of atmos. down shortwave flux
W/m"
TMPsfe
Surface temperature
UGRDI0m
10-meter eastward wind
m/s
VGRDIOm
meter northward wind
m/s
ps
Surface pressure
u700
700-mb castward wind
m/s
northward wind
liq_
Vert. integral cloud water ratio
kg/kg kg/m² kg/kg kg/m²
sphum
Vert. integral of specific humidity
zsurf
Topography
A.2 On Swin-IR-Diff and Multiple Channels
Here, we discuss SwinlR-Diff, which expands one of our robust baselines. Sec. provides concise overview of Swin-IR-Diff. Shown as a sketch Fig. 10 this model opts to downscale each precipitation state individually, akin to an image super-resolution model. Resembling SR3 in its foundation of conditional diffusion model, Swin-IR-Diff adopts residual pipeline. involves a deterministic prediction corrected by residual generated from the conditional diffusion model, with the Swin-IR model serving as the deterministic downscaler this context.
We conducted an ablation, focusing on the incorporation of additional climate states as input to our precipitation downscaling model STVD. The rationale for including these states drawn from the insights of [17), who justified a similar selection for the task of precipitation forecast based on domain science. While Tab. 3 provides detailed information on the various states employed, the utility of these states is closely examined in Tab. [ with specific attention to STVD (multiple input states) and STVD-single (only precipitation state as input). Clearly, the introduction additional channels yields notable improvement in performance.
A.3 Additional Samples
In addition to re-illustrating precipitation downscaling in the Sierras and Central California from Fig
in Fig. IT), we present our model's output for another unique region- Himalayas. Fig. mirrors
Fig. 11 displaying outputs from different models. Note that for the same regional topography. Fig.
(left) compares the annual precipitation time average. We also provide video samples corresponding
to Fig. ( hcccalfforiaae.. and Fig. [2 (himalaya, gif) in the supplementary zip file.
3

---

Timestep
Timestep
Timestep
Timestep
Timestep
(Ground Truth)
Fine Grid
105
120
0.93
0.73
8.0
0.15
STVD (Ours)
0.90
0:75
0.63
8.45
1.35
PSRT [60]
0.90
0:75
0.03
0.30
3
1.35
RVRT [36]
1.05
1.20
0.75
0.02
0.45
0.33
0
1.35
1.20
VRT [34]
1.05
0.93
0.45
0.33
013
Swin-IR-On -R-fff [42]
1.20
1.05
0.00
0:75
0.30
0.15
VDM [21]
1.20
1.0
0.93
0.75
B.02
0.15
Swin-IR [35]
0.90
0.75
0.0
845
(Ground Truth)
1.35
Coarse Grid
0.73
0.60
0.5
0.35
015
Figure 11: Qualitative comparison between our proposed model and all baselines for specific precipitation event in the Sierra mountain range. This figure is repetition of Fig. Fig-3foo for better visual overview. The first row represents the ground truth fine-grid precipitation state sequence, and the last row represents the coarse-grid precipitation that is being downscaled. All other rows correspond to model and the baseline outputs. The time interval between adjacent frames is 3 hours; the plotted region 1000 x 1000 km.

---

Timestep
Timestep
Timestep
Timestep
Fine Grid Truth)
0.02
0.3
033
025
0:2
0.05
STVD (Ours)
0.35
0.33
0.21
0:25
0.05
0.30
0.5
0.25
0:30
0:2
0.20
0.5
012
0.45
0.25
VRT
0.2
0.5
010
0.45
Swin-IR Diff [41]
0
0.25
815
0.13
UUS
0.45
0.40
0.35
VDM
0.20
0.15
010
0.05
0.45
[35]
0.40
0:
0.33
0.5
010
0.05
Coarse Grid
UAS
0.80
0.3
015
0.10
0.5
Figure 12: Another qualitative comparison between our proposed model and baselines for specific precipitation event in the Himalayan mountain range. Fig. (left) plots the regional topography. Similar to Fig. the first row represents the ground truth fine-grid precipitation state sequence, and the last row represents the coarse-grid precipitation that being downscaled. All other rows correspond to model and the baseline outputs. The time interval between adjacent frames is hours; the plotted region is 1000 x 1000 km.

---

Pine Orid (Ground Truih)
STVD louns)
EVRT 1591
VON 1211
Swin-att [SS]
VRT (44)
PART (60)
Swin-it Dif [4.1)
Figure 13: RVRT, VRT, PSRT and Swin-IR-Dire show spectra which decays too rapidly, i.e. placing too little energy on the high-frequency components. The Swin-IR baseline exhibits artifacts the spectra. Overall, the spectra of samples from our method most closely match the ground-truth spectra.
A.3.1 Spectra
we plot (in log-scale) the squared-magnicee of the complex-valued FFT applied to image in our evaluation set. Overall, we see that the samples from STVD closely match the ground- truth high resolution spectrum. The baselines RVRT, VSRT and PSRT demonstrate spectrum which decays too rapidly, placing too little energy in the high-frequency components. Additionally, we see banding these spectra. This indicates that these baselines ground-truth and follow similar banding patterns of the corresponding low resolution image. For the Swin-IR baseline, we observed outliers of large magnitudes in the generated precipitation maps, which we hypothesize leads to the observed checkerboard pattern seen in the spectrum. Swin-IR-DD IR-Diff and VDM seem to decay the spectra more rapidly than STVD.
In Fig. 13
overly smooth compared the
A.3.2 Temporal Attention Behaviour
Average Temporal Attention
Figure 14: A visualization of the temporal attention weights averaged the entire validation and attention heads for the bottleneck layer of the deterministic downscaler. 71 - denotes the temporal sequence. The weights evidently decay as a function of temporal distance which makes physical sense. For example, feature map at position T2 attends the most towards itself along with immediate temporal neighbors T1 and T3. Lighter colors correspond larger weight.

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Sec. 3 outlines the experiments and Tab. clearly demonstrates that the
performance metrics for our model are better than the state-of-the-art baselines.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss some limitations of our approach in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
This paper doesn't 't carry any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The architecture details have been discussed Supplementary Materials A. and the experimental setup is discussed in Sec.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
8

---

Answer: [Yes]
Justification:
The code for our model
is
available
at
https://github hom/mandt--ll lab/ 1abDSTDDD STVD
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssiicyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details can be found in Sec. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Training the models takes long time and reporting meaningful error bars is computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details can be found in Sec. 3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://awwaa...cc/bbbb..ccccctaa
Answer: [Yes]
Justification: Yes, our paper adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts our work are discussed in Section 5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
10

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our proposed model poses no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the reported baselines have been cited in Tab. I and Sec.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not release any assets. The details of the data and architecture be found in Sec. 3 along with Supplementary Material A.1
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
12


Reviews:
[{"review_id": "AzxhZptdvH", "reviewer": "Reviewer_EgC8", "summary": "This paper extends video diffusion model to precipitation super-resolution, where a deterministic downscaler is used to produce initial results and a temporally-conditioned diffusion model is utilized to refine previous coarse results. By combing deterministic and statistical downscaling models, \"mode averaging\" problems are obviously alleviated. Experimental results demonstrates its effectiveness.", "questions": "Please refer to Weaknesses part.", "limitations": "The authors clearly point out the limitations of their work and the negative social impact after Conclusion section.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well organized, the motivations and method details are clearly described.\n2.  The explanation of professional terms is very good, so that researchers in other fields can easily read the paper.\n3. It is very reasonable to predict the low-frequency part with a deterministic model and then generate the high-frequency residual with a statistical model. \n4. The comparisons with other methods in experiment parts seems sufficient.", "weaknesses": "There are some statements, method and experiment details remain to be clear.\n1. Is there any design that guarantees that the output high-resolution frames are smooth over the time series?\n2. For the high-frequency prediction part, if given the same conditions but different sampling noise, will output completely different results?\n3. The authors state that the generative models can capture multimodal conditional distributions and alleviate underestimation of extreme precipitation. Could the authors show experimentally that their approach is better at modeling extreme precipitation than traditional supervised methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_Thank you for taking the time to read our work. We are pleased that you find the paper easy to read, appreciate the residual nature of our model, and find the experiments sufficient. We address your concerns as follows:_\n\n- ___“...any design that guarantees that the output high-resolution frames are smooth over time?”:___\n\n    To encourage the temporal smoothness of high-resolution frames, our model employs temporal attention mechanisms in both the mean downscaling and residual diffusion modules. This smoothness is not explicitly hard-coded, but the fact that the input is a frame sequence and processed with temporal cross-attention enables the model to generate a temporally-coherent output sequence. This approach aligns with other video diffusion works [1] that achieve temporal coherence without hard-coding. For reference, note that we have provided sample output __videos in the supplementary zip__ (california.gif, himalaya.gif), which are as temporally smooth as the ground truth.\n\n- ___“...same conditions but different sampling noise, will output completely different results?”:___\n\n    Yes, as a conditional generative model, our approach produces different, stochastic high-resolution output sequences under the same input conditions. These samples differ in their high-resolution details but are all consistent with the low-resolution input. This variability is crucial in climate modeling as it captures the inherent uncertainty and variability in weather patterns, essential for generating ensemble forecasts. This is why we include CRPS as an evaluation metric since it helps us compare a set of stochastic samples against a deterministic ground truth. For better visualization, we have included __Figure 1 in the attached file in the global response section__. The figure depicts five stochastic samples that our model (STVD) generated for the same precipitation event in the Sierra Nevada region, as shown in the main paper. Additionally, we provide a variance map that depicts variability across these samples. Interestingly, we find that high variance correlates with high precipitation regions, reflecting the fact that precipitation is a highly stochastic and hard-to-predict phenomenon.\n\n- ___“...show experimentally that their approach is better at modeling extreme precipitation than traditional supervised methods?”:___\n\n    We agree that demonstrating our model's effectiveness in capturing the precipitation distribution, particularly extreme precipitation, is essential. In our paper, we address this through two metrics, Earth Mover Distance (EMD) and the 99.999th percentile error (PE), as highlighted in Table 1. These methods refer to the annual precipitation distribution, shown in Figure 4. Table 1 indicates that our method results in a low PE and EMD, indicating that we capture the annual precipitation distribution well and better than other methods. Figure 4 visually reveals that our approach also captures the tail end of this distribution better than other methods. This is what we mean by the statement that our approach is “better at modeling extreme precipitation”. We will clarify this in the paper. \n\n_Thank you again for taking the time to review our work. We hope that our clarifications and additional figures on stochastic samples address your concerns. We will incorporate the additional content prompted by your feedback in the paper revision._\n\n_[1] Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., & Fleet, D. J. (2022). Video diffusion models. Advances in Neural Information Processing Systems, 35, 8633-8646._", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TfJQXpw0BP", "reviewer": "Reviewer_St3G", "summary": "The method proposes a diffusion model to statistical downscale precipitation. The model requires a combination of high resolution and low resolution video data for training (a common scenario in weather and climate modeling). The diffusion model takes in a video of low resolution atmospheric variables and outputs a sample video of high-resolution precipitation. \n\nIn experiments, the model is trained on an ensemble of year-long runs from the standard US forecasting system (FV3GFS). Comparisons are made to deterministic downscaling models. The models are evaluated using mean squared error (MSE), continuous rank probability score (CRPS), earth mover distance, 99.999th percentile error (PE), and spatial autocorrelation error (SAE). The results confirm the advantage of using a probabilistic method in estimating the risk of extreme events.", "questions": "- 280: For an annual average, the diffusion model isn't really necessary. Wouldn't any of the deterministic statistical downscaling models would perform just as well? This should be clarified.\n- 294: It was unclear from the text why additional sampling steps in the STVD increases MSE. I imagine this is because the deterministic downscaling model predicts the mean of the possible rainfall (low MSE) and sampling from the distribution of residuals with the diffusion model will almost certainly increase this. So more sampling steps means a better sample of the residual, which means a higher MSE?", "limitations": "- As the authors note, the model is demonstrated on simulation data where high-resolution video inputs are provided. In many applications of statistical downscaling, the input and output are from different sources (e.g. observed high-res vs. GFS model output or reanalysis), and there may be inconsistencies. So the experiments done here are on \"clean\" data, and further experiments are needed to evaluate the method on applications where it will be useful.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The use of machine learning to downscale atmospheric variables, particularly precipitation, is a hot topic in weather forecasting and climate modeling. This paper will be of interest to the community. \n- Nice ablation studies showing the importance of the temporal dimension and additional atmospheric variables.", "weaknesses": "- There have been a number of papers using diffusion models for downscaling (Hatanaka, et al. 2023 for solar irradiance) and nowcasting (Gao, et al. 2023 and Yu, et al. 2024 for precipitation). These are missing from the related work section. \n- The use of transformer-based models is not justified. It seems likely that vision transformer models have a poor inductive bias for this task, which probably has a lower degree of long-range spatiotemporal dependency than in natural images/video. The authors mention \"key adaptations to the attention mechanism\" but these appear to be simplifications that reduce computation and increase the locality bias --- have the authors tried removing attention all together?\n- Some typos that could be caught with a spell checker.\n- The baselines used for comparison are weak. They are all deterministic, so it is expected that they will fail to capture the extremes. I don't think it is necessary to have comparisons, but I think this point could be made more clearly in the text. \n- In experiments, only 10 samples are taken from the diffusion model. This seems small in the context of estimating the risk of extreme events. \n- Some clarifications would be helpful (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_Thank you for taking the time to read our manuscript. We are happy you see the utility of our work in climate modeling and appreciate the ablation study. We address your concerns as follows:_\n- ___Missing related work:___\n\n    We appreciate the reviewer's suggestion, and will include these references. Hatanaka et al. focus on solar irradiance, and Gao et al. and Yu et al. focus on nowcasting. Our work addresses the challenges posed by the temporal dynamics in precipitation downscaling by leveraging temporal attention mechanisms to encourage consistency over time. We emphasize that downscaling fundamentally differs from nowcasting, as we are provided the entire low-resolution video when doing downscaling and thus do not predict future frames.\n- ___“use of transformer-based models not justified… have the authors tried removing attention?”:___\n\n    Before addressing this, we would like to clarify that our model's backbone is not a Vision Transformer but a Conv UNet with Spatial and Temporal Attention blocks applied to the convolutional features. Removing attention would remove the temporal context, resulting in our “STVD-1” ablation, where the performance degrades appreciably. Although 3D convolutions are an alternative, they are very expensive. To visualize the inner workings of temporal attention, we have added Figure 2 in the attached file in the global response section. It visualizes the temporal attention map from the bottleneck layer of the downscaler. We have averaged this map over the whole validation data and multiple attention heads. As one may expect, we see that, on average, attention decays as a function of time lag, meaning that the model learns to assign more weight to features that are temporally closer.\n- ___Typos:___\n\n    Thank you for pointing this out. We will correct all typos in the final draft.\n\n- ___“comparisons are weak... all deterministic... expected that they will fail to capture the extremes…”:___\n\n     Note that not all baselines we used are deterministic. Swin-IR-Diff and VDM are diffusion-based baselines and thus stochastic. Despite being diffusion-based, our experiments reveal that these methods do not capture extreme events as effectively as our model. As shown in Table 1, our method outperforms these baselines in both the PE and EMD metrics. Furthermore, Figure 4 demonstrates that the precipitation distribution produced by our model closely matches the actual distribution, particularly in the extremes, better than the distributions generated by the baselines. Additionally, we have used more competitive vision VSR baselines, acknowledging that most state-of-the-art baselines for VSR remain deterministic.\n\n- ___“only 10 samples, seems small…”:___\n\n    We may have miscommunicated this. To clarify, the histogram in Figure 4 is calculated across *all grid points* over the entire one year of validation data (~$10^8$ points). The 10 samples referred to in the context are the 10 stochastic samples generated from the same input condition, which are used specifically for calculating CRPS.\n\n- ___“annual average, the diffusion model isn't necessary…”:___\n\n    We stress that predicting annual averages was not our prediction goal but rather serves as a *diagnostic* with high relevance to climate modeling practitioners. The primary use case of our model is to provide high-resolution instantaneous downscaling, which captures detailed local behavior and dynamic precipitation patterns. While deterministic methods may capture broad annual trends, our model simultaneously captures both local behavior and annual patterns, which are crucial for practical applications such as water availability and management.\n\n- ___“unclear why additional sampling steps in the STVD increases MSE…”:___\n\n    Yes, precisely, your interpretation is correct. We agree that this could have been explained more clearly in the submission, and we will include a more detailed discussion in the final draft. To elaborate, the conditional mean is the theoretical minimizer of the MSE, and any deviation from this conditional mean would result in a higher MSE -- even if the resulting deviation is more realistic. Regarding the relationship with sampling steps, fewer sampling steps correspond to taking larger time steps in the diffusion process. At one extreme, taking a single step would correspond to predicting the conditional mean (see [1] for a discussion), minimizing the MSE. Conversely, increasing the number of sampling steps results in a more accurate simulation of the diffusion process, producing more diverse and realistic samples, and thus increasing the MSE while decreasing the PE as shown in Figure 3.\n\n- ___“the experiments done here are on \"clean\" data, and further experiments are needed…”:___\n\n    As discussed in [2], fluid-dynamical emulators of the global atmosphere are too expensive to run routinely at such fine scales. So the climate adaptation community relies on “downscaling” of coarse-grid simulations to a finer grid. Our work builds on vision-based super-resolution methods to improve statistical downscaling by allowing a cheap run of fluid-dynamics emulators on a coarse grid followed by a downscaling using our model on the region of interest. Please also refer to our global response for a discussion of this point.\n\n_Thank you again for taking the time to read our work, as well as for the many suggestions and typo highlighting. We will incorporate these changes in the paper revision to improve readability, as well as the additional content prompted by your feedback._\n\n_[1] Karras, T., ... & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35, 26565-26577._\n\n_[2] Stevens, B., Satoh, M., Auger, L., Biercamp, J., Bretherton, C. S., Chen, X., ... & Zhou, L. (2019). DYAMOND: the DYnamics of the Atmospheric general circulation Modeled On Non-hydrostatic Domains. Progress in Earth and Planetary Science, 6(1), 1-17._", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xG5T4aQf7g", "reviewer": "Reviewer_giVy", "summary": "This paper presents a novel framework for spatio-temporal precipitation downscaling, comprising two modules: a deterministic downscaling module and a diffusion module. The model is able to outperform the SOTA models, especially in extreme events and in mountainous areas.", "questions": "1. In line 121, “bicubic interpolation” does not have a explanation. Can you explain it?\n2. In line 121, what is ”pixelated features” ?\n3. Is the input, additional climate states, the L1 data or L2 data?", "limitations": "Is this work of practical value? The authors acknowledge that switching to a different dataset requires retraining due to differences in data distribution. However, can the model be trained if high-resolution ground truth images are not available? If high-resolution images are available, does it make sense to reduce their resolution before training the model?", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Multiple losses, such as PE and EMD, are used to measure the effectiveness of the results. At the same time, the discussion of the trade-off between realism and bias is novel and informative, using MSE to represent the average accuracy of predictions and PE to represent the model's ability to reproduce extreme events.\n2. The model outperforms six strong super-resolution baselines and can be established as a new standard for data-driven precipitation downscaling.", "weaknesses": "Experiments are insufficient. Such as the effectiveness of the sharing features across the modules is not proven by ablation experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_Thank you for taking the time to read our work. We are happy you find novelty in the realism-distortion tradeoff and appreciate the PE and EMD metrics. We address your concerns as follows:_\n\n- ___“...the effectiveness of the sharing features across the modules is not proven by ablation experiments.”:___\n\n    We thank the reviewer for suggesting an additional ablation study. Due to time constraints, we are unable to run this ablation for the rebuttal, but we will include this ablation in an updated version of our paper. However, we feel that this design choice is a relatively minor part of our overall architecture, and our existing experiments demonstrate the strength of our proposed method (reviewer EgC8) and ablate the key components of our model (reviewer St3G). We would be glad to discuss any additional experiments the reviewer feels would further strengthen our submission.\n\n- ___Explanation of \"bicubic interpolation\":___\n\n    Bicubic interpolation is a resampling method that uses the values of the 16 nearest pixels (4x4 grid) to estimate the value of each pixel in the upsampled image. This method ensures a smoother and higher-quality image than simpler methods like nearest-neighbor or bilinear interpolation. In our context, since the input and output dimensions of our UNet are the same, we use bicubic interpolation to upsample the low-resolution input to match the high-resolution output dimensions (scale factor of 8). We will include this explanation in the revised manuscript for clarity.\n\n- ___Explanation of \"pixelated features\":___\n\n    By \"pixelated features,\" we refer to traditional pixelation artifacts or block artifacts that can occur in super-resolution tasks. These artifacts are characterized by a blocky appearance in the upscaled image, where individual pixels or groups of pixels become visibly distinct, leading to a loss of detail and smoothness. We will clarify this in the manuscript.\n\n- ___“Is the input, additional climate states, the L1 data or L2 data?”:___\n\n    We appreciate the reviewer's question about the input data and additional climate states. However, we are unsure about the specific reference to \"L1 data\" or \"L2 data\" in this context. Could you please clarify what you mean by these terms? In our study, the input includes both the primary low-resolution precipitation data and additional climate states. These additional covariates are provided at a low resolution. Refer to the STVD-single experiment to highlight the importance of this side data. Additionally, we clarify in the appendix how these side data were selected.\n\n- ___“Is this work of practical value?...”:___\n\n    As discussed in [1], fluid-dynamical emulators of the global atmosphere are too expensive to run routinely at such fine scales. So the climate adaptation community relies on “downscaling” of coarse-grid simulations to a finer grid. Our work builds on vision-based super-resolution methods to improve statistical downscaling by allowing a cheap run of fluid-dynamics emulators on a coarse grid followed by a downscaling using our model on the region of interest. Our model's practical value lies in its ability to provide quick and cheap high-resolution downscaling for regions of interest. Additionally, our model emulates long-term annual trends pretty well, which is critical for applications like water availability and management. The main motivation behind this project was to create computationally efficient and realistic proxies for climate emulators:\n    - Generate high-resolution training data for a limited time horizon (e.g., one year).\n    - Train the super-resolution model using this data.\n    - Generate 'cheap' climate predictions at low resolution and use the super-resolution model for focussed regional predictions.\n\n    Please also refer to our global response for a discussion of this point.\n\n    We will also add a discussion in the paper revision on the general issue of data distribution shifts and the potential for developing domain adaptation techniques for climate applications, highlighting it as an interesting future research direction. This context will underscore the practical value and broader applicability of our approach.\n\n_Again, we appreciate the time you took reading our work. We will integrate our responses, along with additional clarifications, into the paper. These enhancements will contribute to the overall readability, clarity and completeness._\n\n_[1] Stevens, B., Satoh, M., Auger, L., Biercamp, J., Bretherton, C. S., Chen, X., ... & Zhou, L. (2019). DYAMOND: the DYnamics of the Atmospheric general circulation Modeled On Non-hydrostatic Domains. Progress in Earth and Planetary Science, 6(1), 1-17._", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for taking the time to review our work and appreciate the detailed feedback and constructive comments. In this response, we address a common point about a claimed limitation of the paper. All other questions are addressed in reviewer-specific responses.

Some reviewers pointed out the use of “clean data” for training (e.g., the availability of paired high-resolution/low-resolution samples) as a potential practical limitation. We stress that this setup is an established way of performing precipitation downscaling, commonly called the 'perfect prediction' paradigm according to the taxonomy of [1]. In this approach, the coarse-grid features used for super-resolution are derived by coarse-graining the fine-grid data used for training. This most basic super-resolution task is an important first step toward a longer-term goal: training super-resolution models to enhance coarse-grid features from an inexact emulator of coarse-grid meteorology to super-resolve the fine-grid data. This process can be broken into two steps: (1) correcting biases in the emulator compared to the coarsened fine-grid data, and (2) performing the ‘perfect prediction’ task. The first step is emulator-specific, while the second is more generic and of broader interest.

In addition, some reviewer questions asked for additional investigations. We provide here a pdf containing the results of those investigations, containing

- Figure 1: Visualization of stochastic samples and variance map
- Figure 2: Visualization of temporal attention

Specific context and discussions are presented in the corresponding review rebuttals.

Many thanks,

The authors

_[1] Rampal, N., Hobeichi, S., Gibson, P. B., Baño-Medina, J., Abramowitz, G., Beucler, T., ... & Gutiérrez, J. M. (2024). Enhancing Regional Climate Downscaling through Advances in Machine Learning. Artificial Intelligence for the Earth Systems, 3(2), 230066._

Meta-review: This paper presents a novel framework for precipitation super-resolution using a video diffusion model. It combines a deterministic downscaling module for initial results with a temporally-conditioned diffusion model to refine these results. The authors claim that this hybrid approach alleviates "mode averaging" issues and demonstrates effectiveness through experimental validation. Based on the three reviews, there is a general inclination toward acceptance, with no major flaws identified. I recommend accepting this paper as a poster. However, the authors should make further revisions before publication to meet publication standards. Specifically, the related work section lacks important references, the experimental details are unclear, and additional ablation studies are needed.
Decision: Accept (poster)