Precipitation Downscaling with Spatiotemporal Video Diffusion

Authors: prakhar  srivastava
ruihan  yang
gavin  kerrigan
gideon  dresdner
jeremy  mcgibbon
christopher  bretherton
stephan  mandt
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 13:00:51

Abstract: In climate science and meteorology, high-resolution local precipitation (rain and snowfall) predictions are limited by the computational costs of simulation-based methods. Statistical downscaling, or super-resolution, is a common workaround where a low-resolution prediction is improved using statistical approaches. Unlike traditional computer vision tasks, weather and climate applications require capturing the accurate conditional distribution of high-resolution given low-resolution patterns to assure reliable ensemble averages and unbiased estimates of extreme events, such as heavy rain. This work extends recent video diffusion models to precipitation super-resolution, employing a deterministic downscaler followed by a temporally-conditioned diffusion model to capture noise characteristics and high-frequency patterns. We test our approach on FV3GFS output, an established large-scale global atmosphere model, and compare it against six state-of-the-art baselines. Our analysis, capturing CRPS, MSE, precipitation distributions, and qualitative aspects using California and the Himalayas as examples, establishes our method as a new standard for data-driven precipitation downscaling.

Main paper: 
 introduction: 
 Figure 1 : Static snapshot from the Spatiotemporal Video Diffusion (STVD) model, illustrating input (left) and output (right) precipitation frames. The input panel displays simulated coarse-resolution precipitation (rain, snow) fields (Section 3), super-resolved into the high-resolution output shown in the right panel. Both frames use Robinson projection and cover six tiles of the cubed-sphere grid, providing a detailed global view (optimal viewing with zoom). For dynamics, see Fig. 3 .Precipitation patterns are central to human and natural life. In a rapidly warming climate, reliable simulations of changing precipitation patterns can help adapt to climate change. However, these simulations are challenging due to the multi-scale variability of weather systems and the influence of complex surface features (like mountains and coastlines) on precipitation trends and extremes Figure 2 : Our model's training and inference pipelines: Blue blocks apply to both phases, red blocks to training only, and green blocks to inference only. It deterministically downscales a lowresolution precipitation sequence using spatio-temporal factorized attention and models residuals with conditional diffusion (with factorized attention). Here, T denotes sequence length and N denotes diffusion steps. The parameters (θ = ϕ, ψ) are optimized jointly during training. See A.1 for details. [58] . For many purposes, such as estimating flood hazards, precipitation must be estimated at spatial resolutions of only a few kilometers. Fluid-dynamical models of the global atmosphere are too expensive to run routinely at such fine scales [65] , so the climate adaptation community relies on "downscaling"1 of coarse-grid simulations to a finer grid. Traditional downscaling methods are either dynamical (running a fine-grid fluid-dynamical model limited to the region of interest, which requires specialized knowledge and computational resources) or statistical (typically restricted to simple univariate methods) [67] . Our work builds on vision based super-resolution methods to improve statistical downscaling and is a natural follow-up to recent deep-learning-based weather/climate prediction methods, which have revolutionized data-driven forecasting. These approaches boast improvements of orders of magnitude in runtime without sacrificing accuracy [46, 29] .We address the downscaling problem for a sequence. Our objective is to transform a sequence ("video") of low-resolution precipitation frames into a sequence of high-resolution frames. Despite differences from natural videos, precipitation's hourly temporal continuity allows us to use video super-resolution techniques to leverage multiple context frames for stochastic downscaling [53, 38] .This study highlights the capability of conditional diffusion models to meet the specific needs of statistical precipitation downscaling, with our key contributions being:1. We introduce a novel framework for temporal precipitation downscaling using diffusion models. Our model combines a deterministic downscaling module with a diffusion-based residual module. It leverages spatio-temporal factorized attention to process information from multiple low-resolution frames.2. Our model outperforms six strong super-resolution baselines across multiple criteria, including MSE and several distributional metrics. We compare against two image super-resolution models and four video super-resolution models using the FV3GFS global atmosphere simulation dataset [77, 10] .3. Our approach captures key characteristics of precipitation, including extreme precipitation probabilities and spatial patterns of annual precipitation in mountainous regions, which are crucial for domain science applications.Our paper is structured as follows: we first describe our method (Sec. 2), followed by our experimental findings (Sec. 3). Finally, we discuss relevant literature (Sec. 4) and its connection to our work. Fig. 1 shows a global view of the input and predicted precipitation of our model. The code for our model is available at https://github.com/mandt-lab/STVD. 
downscaling via spatiotemporal video diffusion: 
 Problem Statement At training time, we assume access to a collection of high-resolution precipitation frame sequences y 0:T and their corresponding low-resolution precipitation frame sequences x 0:T . Such a low-resolution sequence can be obtained through area-weighted coarsening [39] of the corresponding high-resolution sequence. The dataset is discussed extensively in Sec. 3 . Frame indices are represented by superscripts, where we assume that each sequence consists of T + 1 frames for simplicity. While it may be possible to roll out predictions for multiple sequences autoregressively using techniques such as reconstruction guidance [21] , we leave this exploration for future work. Our objective is to train a model to effectively downscale, or super-resolve a given sequence x 0:T with y 0:T serving as the target. We use "downscaling" and "super-resolution" interchangeably.More formally, let x t ∈ R C×H×W and y t ∈ R 1×sH×sW represent individual low-resolution and high-resolution frames. Here, s ∈ N denotes the downscaling factor, C is the number of channels (quantities used as input to the model to characterize the atmospheric state in each low-resolution grid cell so as to add skill to the precipitation prediction), and H, W indicate the height and width of the low-resolution frame. For our study, we adopt a downscaling factor of s = 8 and have C = 12 total low-resolution channels. In addition to the low-resolution precipitation state, we provide eleven channels of information to the model, such as topography, wind velocity, and surface temperature; see A.2 for details.Our approach treats the downscaling problem as a conditional generative modeling task. We devise a model to learn the conditional distribution of high-resolution precipitation frames, incorporating contextual information from the low-resolution precipitation frame sequence.Our proposed solution, SpatioTemporal Video Diffusion (STVD) (Fig. 2 ), relies on two modules: a deterministic downscaler and a stochastic component based on conditional diffusion models [20, 63] , both using spatio-temporal factorized attention. The first module uses a UNet with factorized attention to integrate information from a low-resolution frame sequence, resulting in an initial prediction frame sequence ȳ0:T . The second module is a conditional diffusion model that stochastically generates a sequence of additive residual frames r 0:T which serves to add fine-grained details to the initial prediction. Together, these two modules produce a high-resolution frame sequence ŷ0:T = ȳ0:T + r 0:T . Both modules are trained end-to-end.Decomposing the prediction into a deterministic mean and a stochastic residual is inspired by predictive-coding-based video decompression. This approach aims to predict a sequence of video frames while compressing the sparse residuals [2, 74] which are easier to model than dense frames. Similarly, it is easier to generate residuals than dense images when using diffusion models [73, 41] . In what follows, we first describe our overall probabilistic framework for downscaling. Then, we discuss the deterministic module along with spatio-temporal attention, followed by the remaining residual prediction module based on diffusion generative modeling. See A.1 for architecture details.Given a sequence of low-resolution frames x 0:T and the corresponding high-resolution frames y 0:T , we aim to learn a parametric approximation p θ of the conditional distribution p y 0:T | x 0:T ≈ p θ y 0:T | x 0:T . Importantly, we do not assume independence across time; each generated frame y t can depend on all other generated frames. The generated high-resolution frame sequence is conditioned on the entire low-resolution frame sequence, capturing long-range temporal correlations and enhancing the fidelity and cohesion of the high-resolution reconstruction.As noted earlier, the likelihood p θ y 0:T | x 0:T is modeled using a deterministic downscaler and a residual diffusion model. We will discuss how the model parameters θ = (ϕ, ψ) decompose into those for a downscaler (ϕ) and a diffusion model (ψ).Our first module is a deterministic downscaler that predicts an initial high-resolution frame sequence ȳ0:T = µ ϕ x 0:T where µ ϕ is a network generating a deterministic high-resolution prediction with parameters ϕ. We perform bicubic interpolation on each frame of x 0:T before passing the sequence through the network µ ϕ . Since the diffusion network operates on high-resolution inputs (i.e. denoising the high-resolution residuals), this choice allows us to use the same UNet [52] architecture (with different weights) for both the downscaling module µ ϕ and the residual diffusion module. This enables us to easily share features across the modules via concatenation. See A.1 for further details.Importantly, µ ϕ incorporates a temporal attention mechanism that allows any frame at time t, or its corresponding feature map, to attend to all context frames from 0 to T . This architecture enables the concurrent inference of all frames within the sequence ȳ0:T . The attention weights differ for each frame, allowing for the flexible incorporation of information across time.After computing the initial prediction ȳ0:T , finer details are modeled by residuals learned from a conditional diffusion model. Our final stochastic high-resolution frame sequence ŷ0:T is generated by sampling an additive residual sequence r 0:T from this model: ŷ0:T = ȳ0:T + r 0:T . Thus, we seek to model the residuals r 0:T = y 0:T -ȳ0:T . Our diffusion model generates the entire residual sequence r 0:T concurrently, with the generation of each residual r t dependent on the others. This is achieved via a UNet architecture with spatio-temporal attention, similar to the mechanism used for the deterministic downscaling module. See A.1 for further details.To model the distribution of r 0:T , we use DDPM [20] . To that end, we introduce a collection of latent variables r 0:T 0:N , where the lower subscripts indicate the denoising diffusion step. In the forward process, the latent variable r 0:T n is created from r 0:T n-1 via additive noise. In the reverse process for generation, a denoising model (with parameters ψ) is trained to predict r 0:T j-1 from r 0:T j . N denotes the total number of denoising steps. Note that r 0:T = r 0:T 0 , i.e. the first diffusion step corresponds to the true residual. Additionally, r 0:T 0 implicitly depends on the downscaler parameters ϕ, allowing us to simultaneously optimize all model parameters θ = (ϕ, ψ) within the context of diffusion modeling.As is standard in diffusion models [20] , we parameterize the reverse process via a Gaussian distribution with a mean determined by a neural network M ψ ,EQUATIONwhere M ψ is a denoising network and γ is a hyperparameter for variance. The diffusion model directly accesses the context c = (x 0:T , y 0:T ), and is implicitly conditioned on x 0:T via concatenation of feature maps from the downscaler module. As in the downscaler, we bicubically upsample x 0:T before channel-wise concatenation with y 0:T to match the dimensions when forming c.To train our model, we use the angular parametrization suggested by [59] . Specifically, this results in the diffusion loss of the formEQUATIONwhere ϵ ∼ N (0, I), n is sampled uniformly from {1, 2, . . . , N }, and the sequences x 0:T , y 0:T are sampled from the training distribution. Here, c = (x 0:T , y 0:T ) where ȳ0:T = µ ϕ x 0:T . The scalarsα 2 n = n i=1 (1 -β i ) and σ 2 n = 1 -α 2 n are used to define v ≡ α n ϵ -σ n r 0:T 0 .Training and inference are concurrent across multiple frames due to spatio-temporal attention. Alg. 1 and 2 demonstrate the training and sampling strategy under the angular parametrization. We use DDIM sampling [62] to generate frame residuals with fewer diffusion steps.Both the downscaler and the conditional diffusion model employ a UNet backbone with similar architectures and key adaptations to the attention mechanism (see A.1). The downscaler takes the multi-channel input frames (x 0:T ), yielding an initial estimate (ȳ 0:T ). The diffusion UNet conditions on diffusion step n and concatenates feature maps from the downscaler with its own. The concatenated input to the diffusion UNet (x 0:T , ȳ0:T , and r 0:T n ), along with the conditioning variables (diffusion step n and the feature maps from downscaler), yields the output v.Computing full attention for temporal coherence across the entire video data cube is very expensive for processing long sequences or high-resolution inputs. To optimize efficiency, we decouple attention between spatial and temporal dimensions, use a linear variant of self-attention [26] for non-bottleneck layers (where the effective number of "tokens" for attention is relatively large), focus spatial attention on localized patches (instead of the entire feature map, which could be wasteful), and calculate per-channel temporal attention in large spatial dimensions (namely, the ultimate and penultimate expansion and contraction layers of UNet). These modifications dramatically reduce the time complexity and memory footprint of these transformer blocks. 3 Experiments; v = M ψ r 0:T n , n, c ; L = ||v -v|| 2 ; (ψ, ϕ) = (ψ, ϕ) -∇ ψ,ϕ L;We conduct a comprehensive evaluation of our proposed method, SpatioTemporal Video Diffusion (STVD), against six contemporary state-of-the-art baselines. The first two baselines are image superresolution models based on the Swin Vision Transformer (Swin-IR) [35] and its residual diffusion variant (Swin-IR-Diff). The next two baselines are video super-resolution models grounded in vision transformer architecture (VRT) [34] and its recurrent variant (RVRT) [36] . The latter incorporates guided deformable attention for clip alignment, enhancing its temporal modeling capabilities. We compare against another video-super-resolution baseline (PSRT) [60] which also relies on the transformer architecture but uses multi-frame attention groups. Finally, we compare against a video diffusion baseline (VDM) [21] . Fig. 1 shows a global view of the input and predicted precipitation.We perform ablation studies in three configurations. In the first two, we experiment with the input sequence length. While our proposed model uses a context length of 5 frames, we also conduct experiments with 3 frames and 1 frame (STVD-3 and STVD-1). Note that using a single context frame ablates for the temporal attention block as well. The third ablation (STVD-Single) involves removing the additional input channels (i.e. only providing the model with the low-resolution precipitation sequence) to assess their impact on performance metrics. In summary, our experiments demonstrate that our method outperforms all baselines across all metrics considered. Additionally, our ablation studies highlight the importance of temporal context and additional climate inputs.Dataset Our dataset derives from an 11-member initial condition ensemble of 13-month simulations using a global atmosphere model, FV3GFS, run at 25 km resolution and forced by climatological sea surface temperatures and sea ice. The first month of each simulation is discarded to allow the simulations to spin up and meteorologically diverge, effectively providing 11 years of reference data (of which first 10 years are used for training and the last year for validation). FV3GFS, developed by the National Oceanic and Atmospheric Administration (NOAA), is a version of NOAA's operational global weather forecast model ( [77, 10] ).Three-hourly average data were saved from this entire simulation, which used a 25 km horizontal "fine grid". We further coarsened the selected fields by a factor of 8 to create a 200 km horizontal "coarse grid", resulting in paired data (x t , y t ), where x t is the coarse-grid global state and y t is the corresponding fine-grid global state. Our goal is to apply video downscaling to the coarse-grid precipitation field to obtain temporally smooth fine-grid precipitation estimates that are statistically similar to the true data. This approach is attractive because many fine-grid precipitation features, such as cold fronts and tropical cyclones, are poorly resolved on the coarse grid but are temporally coherent across periods much longer than 3 hours. We use 12 coarse-grid input fields, including precipitation, topography, and horizontal vector wind at various levels. See A.2 for the list of included atmospheric variables. FV3GFS uses a cubed-sphere grid, where the surface of the globe is divided into six tiles, each of which is covered by an S × S array of points. Our data fields reflect this structure with S = 48 for the 200 km coarse grid and S = 384 for the 25 km fine grid. The application presented here serves as a pilot for broader uses of our methodology. Fine-grid simulations are significantly more computationally expensive than coarse-grid simulations (an 8-fold reduction in grid spacing requires almost 1000x more computation), so a coarse-grid simulation with super-resolved details in desired regions could be highly cost-effective for many applications.During training, our model randomly selects data from one of the six tiles. This strategy ensures that the model learns from the diverse spatial contexts and weather regimes that produce precipitation worldwide. Post-training, for localized analysis, we selectively sample super-resolved precipitation channels from regions with complex terrain, such as California (Fig. 6 ). These regions can systematically pattern the precipitation on fine scales. This analysis helps us to see how well the super-resolution can learn the time-mean spatial patterns (e.g. precipitation enhancement on the windward side of mountain ranges and lee rain shadows) in the fine-grid reference data.Training and Testing Details We downscale a sequence of precipitation frames from FV3GFS output by a factor of 8. Our approach (STVD) trains on 5 consecutive frames that are downscaled jointly. We optimize our model end-to-end with a single diffusion loss using Adam [28] with an initial learning rate of 1 × 10 -4 , decaying to 5 × 10 -7 with cosine annealing during training, executed on an NVidia RTX A6000 GPU. The diffusion model is trained using v-parametrization [59] , with a fixed diffusion depth (N = 1400). Random tiles extracted from the cube-sphere representation of Earth, with dimensions 384 in high-resolution and 48 in low-resolution, are used during training. We train for one million steps, requiring approximately 7 days on a single node (slightly less for ablations). We use a batch size of one, apply a logarithmic transformation to precipitation states, and normalize to the range [-1, 1]. During testing, we employ DDIM sampling with 30 steps on an Exponential Moving Average (EMA) variant of our model (for full frame size), with a decay rate of 0.995.Baseline Models We compare our generative setup against several recent high-performing transformer-based video super-resolution models. These models are trained deterministically. The first, Video Restoration Transformer (VRT) [34] , allows for parallel frame prediction and long-range temporal dependency modeling. The second, recurrent VRT (RVRT) [36] , incorporates guided deformable attention for effective clip alignment, enhancing its temporal modeling capabilities. The third, PSRT [60] , removes the alignment module and modifies the attention window. We also compare against the recent Video Diffusion Model (VDM) [21] which employs global quadratic attention.To assess the benefits of multi-frame downscaling, we compare with Swin-IR [35] , a popular image super-resolution model that harnesses Swin Transformer blocks. However, Swin-IR is trained in a supervised fashion. Thus, as a generative baseline, we compare to Swin-IR-Diff. This model generates a deterministic prediction using Swin-IR [35] , followed by modeling a stochastic residual Figure 6 : Precipitation over two regions (left: Himalayas; right: Northern California coast, same region as Fig. 3 ), averaged across a year, for our STVD model and the ground-truth. For each half, the topography of the region is shown in the corresponding top-left whereas the predicted annual average is shown in the corresponding bottom-right. Annually-averaged precipitation is an important indicator of water availability in a region. STVD successfully captures many details of the precipitation that are tied to local topography and are too fine to be resolved the coarse-grid data.using diffusion. This baseline is inspired by concurrent work on single-image radar-reflectivity downscaling [41] , where a UNet is used instead of Swin-IR. See A.2 for details.Evaluation Metrics We evaluate our model differently from standard vision tasks. In addition to the Mean Square Error (MSE), which measures the average squared difference between predicted and actual values but lacks full distributional information, we use several distribution-level metrics for a more meaningful comparison. One such metric is the Continuous Ranked Probability Score (CRPS) [8, 66] , which assesses the discrepancy between the predicted cumulative distribution function and the observed data. We compute CRPS over 10 stochastic realizations of our predictions. Fig. 7 visualizes several of these samples.Furthermore, given the distinctive light-tailed exponential distribution of the precipitation climate state, it is crucial to ensure that downscaling does not significantly alter the distribution of precipitation rates. This necessitates two additional metrics. First, we compute the Earth Mover (or 1-Wasserstein) Distance [54] to quantify the agreement between the target and predicted global precipitation distributions, which are strongly affected by high-resolution details. Second, we focus on tail events and extreme precipitation by considering the 99.999th percentile error (PE), providing a nuanced understanding of the model's performance on rare and extreme precipitation events.To further assess the spatial fidelity of our downscaling approach, we use the Spatial Autocorrelation Error (SAE) [68] . This metric calculates the mean absolute error between the spatial autocorrelation of the predictions and ground truth. Low SAE ensures that the spatial patterns and the fine structure in precipitation data are preserved during downscaling, which is critical for accurate climate modeling.Qualitative and Quantitative Analysis Tab. 1 provides a quantitative evaluation comparing our method with state-of-the-art baselines and ablations. Our model (STVD) performs strongly across all metrics, outperforming all baselines. We highlight the distributional characteristics in Fig. 5 . Swin-IR overestimates precipitation, while all other baselines underestimate it. This discrepancy is undesirable, as poor performance on rare and extreme precipitation events can negatively imapct disaster mitigation policies. In contrast, our method closely matches the precipitation distribution, as measured by PE and EMD.Using only precipitation as an input (STVD-single) results in slightly worse performance across all metrics, indicating the predictive value of additional inputs. In contrast, our ablation model STVD-1, which lacks full sequence information, performs significantly worse, highlighting the importance of temporal attention in our approach (which decays as a function of time lag as shown in Fig. 14 ). The precipitation event is the same as depicted in Fig. 3 . Additionally, we also plot a variance map over the set of these samples to analyze the stochasticity better. Red regions correspond to high variance whereas blue regions correspond to low variance. Model stochasticity seems to be meaningful since the variance is large where mean precipitation is large.which preserves most patterns with a high degree of similarity. PSRT and RVRT produce slightly more diffuse precipitation features, while Swin-IR produces slightly more pixelated features.Fig. 6 shows annually-averaged precipitation from the patches in Figs. 3 and 12 . Accurately capturing the fine-grid structure of time-mean precipitation is crucial for assessing long-term water availability.Our method (which includes fine-grid topography as a training input) effectively replicates the ground truth. This includes the strength and narrow spatial structure of high precipitation bands along the Northern California coastal mountains and the Sierras. These features are not resolved by the coarse-grid inputs to the super-resolution. See A.3 for full-page high-resolution samples and spectral analysis. Fig. 13 reveals that the spectra for baselines decay more rapidly than for STVD.Realism-Distortion Tradeoff Distortion metrics such as MSE often conflict with perceptual quality, where reducing distortion typically degrades perceptual realism [7] . In our context, this tradeoff translates to balancing MSE and PE. While MSE captures the average accuracy of predictions, PE represents the model's ability to reproduce extreme events, thereby serving as a proxy for realism. Realism in climate modeling refers to the accurate representation of extreme weather patterns, which are crucial for applications like flood forecasting and disaster mitigation. PE is a distributional criterion that effectively captures these tail events, offering a robust measure of realism. Fig. 4 illustrates this tradeoff, with darker colors corresponding to fewer STVD sampling steps.As the number of diffusion sampling steps increases, MSE tends to rise slightly, but PE decreases significantly. Depending on the application, this tradeoff may potentially be exploited by practitioners. Essentially, the conditional mean minimizes MSE, so any deviation from it increases MSE-even if the deviation appears more realistic. As for sampling steps, fewer steps correspond to larger time increments in the diffusion process. At one extreme, a single step predicts the conditional mean, minimizing MSE. Conversely, more sampling steps more accurately simulate diffusion, generating diverse, realistic samples that increase MSE while reducing PE.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss some limitations of our approach in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: Yes, our paper adheres to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
related work: 
 Diffusion Models Diffusion models [61, 20, 64, 43, 45, 40] are a class of generative models based on an iterative denoising process. Closely related to our work are diffusion models for video. Recent models [73] generate deterministic next-frame predictions autoregressively with additional residuals generated by a diffusion model, or generate videos directly in pixel space [18, 69, 21] or in a latent space [6, 5] . While some works on video diffusion [6, 21] employ video super-resolution as a step in the overall modeling process, our work focuses exclusively on the video super-resolution task, particularly within the context of precipitation downscaling.Super-Resolution Within the computer vision community, the paradigm for single image superresolution has shifted from classical approaches [4, 13] to deep learning based methods [71] . Generative approaches, like cascaded diffusion [49, 55] , SR3 [56] , and DiffPIR [78] employ diffusion models for image super-resolution. However, these are unable to leverage temporal context. On the other hand, many approaches for video super-resolution have been proposed [9, 14, 22] . For a more comprehensive overview, see [38] . Recent models of note include the transformer-based models PSRT [60] and VRT [34] , as well as the recurrent variant RVRT [37] , which focuses on parallel decoding and guided clip alignment. We emphasize that these state-of-the-art approaches are deterministic, where our approach is generative. This allows us to preventing mode averaging and to produce more realistic samples, which is particularly critical in the context of precipitation modeling.Data-driven weather and climate modeling Recent years have seen advancements in data-driven climate and weather modeling [51, 42] , with models like GraphCast [29] , GenCast [48] , and Four-CastNet [46] providing forecasts that are competitive with meteorological methods while being significantly faster. Rather than replacing numerical forecasting methods, our approach seeks to augment their capabilities by downscaling coarse-grid predictions.While downscaling for climate and weather has been approached using techniques based on domain knowledge [25, 24] , we focus here on data-driven approaches. [68] draw inspiration from FRVSR [57] , adopting an iterative approach that uses the high-resolution frame estimated in the previous step as input for subsequent iterations. [72] employed Fourier neural operators for downscaling at arbitrary resolutions. [16] generate physically consistent downscaled climate states, using a softmax layer to enforce conservation laws. These approaches, though, are deterministic and trained by minimizing the MSE, thus lacking the realism and uncertainty quantification provided by a generative approach.In terms of generative approaches, concurrent work [41] employs diffusion models for downscaling climate states. The use of GANs has also been pervasive in downscaling and precipitation prediction [32, 47, 17, 50, 15, 70] . However, these GAN-based approaches inherit the mode collapse and training difficulties present in all GAN-based models [3] . Here, we highlight that these approaches are applied at each frame and cannot incorporate temporal information as is done in our model.Beyond downscaling, [1] demonstrate the diffusion model's efficacy in synthesizing full rain density from vorticity inputs. Additionally, [19] uses a diffusion model for downscaling solar irradiance. These models are also used for probabilistic weather forecasting and nowcasting [31, 33] . 
conclusion: 
 We propose a video super-resolution method for probabilistic precipitation downscaling. Our model, STVD, deterministically super-resolves a given low-resolution frame sequence and then stochastically models the residual details via diffusion. Our model effectively resolves how fine-grid precipitation features, generated as weather systems, interact with complex topography based on temporally coherent coarse-grid information. Our method outperforms several competitive baselines on a range of quantitative metrics. This is an important step towards designing effective statistical downscaling methods, providing highly localized information for planning against extreme weather events, such as floods or hurricanes in a warming climate, using tractable coarse-grid atmospheric models.Limitations and Broader Impacts A limitation of our approach is the necessity of paired lowresolution and high-resolution images for training. While this can be done once prior to training, designing methods that only require the low-resolution states is an interesting challenge. In terms of broader impacts, our approach could potentially have harmful consequences if adopted blindly to a new dataset, where distribution shift could cause the model performance to degrade, potentially leading to underestimation of extreme weather risks such as droughts or floods. To mitigate these risks, the model should be re-trained and rigorously evaluated on the dataset of interest. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The broader impacts our work are discussed in Section 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our proposed model poses no such risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All the reported baselines have been cited in Tab. 1 and Sec. 3.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Our architecture is a conditional extension of the DDPM [20] and SR3 [56] models. As discussed in Sec. 2, Fig. 8 presents the architecture of the proposed denoising and downscaling networks, while Fig. 9 provides detailed description of each component. Before elaborating on the specifics, we define the naming conventions for the parameter choices adopted in this section:• ChannelDim: ResBlock channel dimension for first contractive layer of UNet.• ChannelMultipliers: channel dimension multipliers for subsequent contractive layers (including the first layer) in both the downscaling and denoising modules. The expansive layer multipliers follow in reverse.• ResBlock: a standard ResBlock implementation consisting of two blocks, each with a weight-standardised convolution using a 3 × 3 kernel, Group Normalization over groups of 8 and SiLU activation, followed by a channel-adjusting 1 × 1 convolution.• Attention: a standard implementation of quadratic or linear attention, incorporating 4 attention heads, each with a 32-dimensional representation. Using a 3 × 3 convolutional kernel, query, key, and value feature maps are generated, resulting in feature maps of size [B × T, H × C, X, Y], where B, T, C, X, and Y denote batch, time, channel, height, and width, respectively, with H representing the number of heads. These feature maps are rearranged to [BATCH, HEAD, CHANNEL, TOKEN]), facilitating self-attention between TOKENs of CHANNEL dimensions. The rearrangement order determines whether spatial or temporal self-attention is performed. Subsequently, the feature maps revert to their original format, upon which a separate convolution operation is conducted, followed by layer normalization to project the feature maps back to their original dimensions, akin to their state before the initial convolution within the attention block. Further elaboration on the rearrangement choices for the variants is discussed, adhering to einops3 notation:-Q-Spatial: quadratic variant applied in the bottleneck layer; self attends between every pixel of the feature map with the following rearrangement : [B × T, H × C, X, Y] -> [B × T, H, C, X × Y] -Q-Temporal: quadratic variant[B × T, H × C, X, Y] -> [B, H × X × Y, C, T]• MLP: conditioning on the denoising step n is achieved through this block, which uses 32dimensional random Fourier features, followed by a linear layer, GELU activation and another linear layer to transform the noise step to a higher dimension.• Cov/TransCov: these are convolutional (3 × 3 kernel) downsampling and upsampling blocks that change the spatial size by a factor of 2.Fig. 8 illustrates the interaction between the U-Net architecture of the denoising and downscaling networks. The top U-Net depicts the denoising network, while the bottom U-Net depicts the downscaling network. The denoising network is conditioned in three ways. First, the network is conditioned on the bicubically downscaled low-resolution frames x 0:T , concatenated with both the noisy residual r 0:T n the downscaler output and ȳ0:T along the channel dimension. Second, the network is also conditioned on the feature maps generated by the downscaler network as indicated by the green arrows connecting the downsampling units of both the networks. The L-(Spatial/Temporal) Attention blocks from contractive layers of the downscaler network yield a feature map that gets concatenated with the inputs of both ResBlocks of the contractive layer of the denoising network. Finally, each contractive and expansive layer of the diffusion UNet gets conditioned on the denoising step n, shown via the black arrows. This conditional embedding for the step is generated through MLP and received by both ResBlocks. The information flows from the noisy residual through the network as shown by the blue arrows to predict the angular parameter v. Both U-Nets have skip connections, indicated by the red arrows, between both ResBlocks of contractive and expansive layers of the same UNet.Table 2 : FV3GFS uses a cubed-sphere grid, in which the surface of the globe is divided into six tiles. Each high-resolution tile covers 25 km and is 384 × 384. Our UNet Encoder and Decoder have 6 layers with a base channel dimension of 64 and multipliers as stated above. Figure 12 : Another qualitative comparison between our proposed model and baselines for a specific precipitation event in the Himalayan mountain range. Fig. 6 (left) plots the regional topography. Similar to Fig. 3 , the first row represents the ground truth fine-grid precipitation state sequence, and the last row represents the coarse-grid precipitation that is being downscaled. All other rows correspond to our model and the baseline outputs. The time interval between adjacent frames is 3 hours; the plotted region is 1000 × 1000 km.EQUATIONFigure 13 : RVRT, VRT, PSRT and Swin-IR-Diff show a spectra which decays too rapidly, i.e. placing too little energy on the high-frequency components. The Swin-IR baseline exhibits artifacts in the spectra. Overall, the spectra of samples from our method most closely match the ground-truth spectra.In Fig. 13 , we plot (in log-scale) the squared-magnitude of the complex-valued FFT applied to an image in our evaluation set. Overall, we see that the samples from STVD closely match the groundtruth high resolution spectrum. The baselines RVRT, VSRT and PSRT demonstrate a spectrum which decays too rapidly, placing too little energy in the high-frequency components. Additionally, we see a banding in these spectra. This indicates that these baselines are overly smooth compared to the ground-truth and follow similar banding patterns of the corresponding low resolution image. For the Swin-IR baseline, we observed outliers of large magnitudes in the generated precipitation maps, which we hypothesize leads to the observed checkerboard pattern seen in the spectrum. Swin-IR-Diff and VDM seem to decay the spectra more rapidly than STVD. NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.38th Conference on Neural Information Processing Systems (NeurIPS 2024).This is the climate science terminology for super-resolution.Code : https://github.com/mandt-lab/STVDhttps://einops.rocks/1-einops-basics 


Decision: This paper presents a novel framework for precipitation super-resolution using a video diffusion model. It combines a deterministic downscaling module for initial results with a temporally-conditioned diffusion model to refine these results. The authors claim that this hybrid approach alleviates "mode averaging" issues and demonstrates effectiveness through experimental validation. Based on the three reviews, there is a general inclination toward acceptance, with no major flaws identified. I recommend accepting this paper as a poster. However, the authors should make further revisions before publication to meet publication standards. Specifically, the related work section lacks important references, the experimental details are unclear, and additional ablation studies are needed.

Reviews: review: paper_summary: paper_summary: This paper extends video diffusion model to precipitation super-resolution, where a deterministic downscaler is used to produce initial results and a temporally-conditioned diffusion model is utilized to refine previous coarse results. By combing deterministic and statistical downscaling models, "mode averaging" problems are obviously alleviated. Experimental results demonstrates its effectiveness.
strength_weakness: strength_weakness: There are some statements, method and experiment details remain to be clear.
1. Is there any design that guarantees that the output high-resolution frames are smooth over the time series?
2. For the high-frequency prediction part, if given the same conditions but different sampling noise, will output completely different results?
3. The authors state that the generative models can capture multimodal conditional distributions and alleviate underestimation of extreme precipitation. Could the authors show experimentally that their approach is better at modeling extreme precipitation than traditional supervised methods?
questions: questions: Please refer to Weaknesses part.
limitations: limitations: The authors clearly point out the limitations of their work and the negative social impact after Conclusion section.
paper_summary: paper_summary: The method proposes a diffusion model to statistical downscale precipitation. The model requires a combination of high resolution and low resolution video data for training (a common scenario in weather and climate modeling). The diffusion model takes in a video of low resolution atmospheric variables and outputs a sample video of high-resolution precipitation. 

In experiments, the model is trained on an ensemble of year-long runs from the standard US forecasting system (FV3GFS). Comparisons are made to deterministic downscaling models. The models are evaluated using mean squared error (MSE), continuous rank probability score (CRPS), earth mover distance, 99.999th percentile error (PE), and spatial autocorrelation error (SAE). The results confirm the advantage of using a probabilistic method in estimating the risk of extreme events.
strength_weakness: strength_weakness: - There have been a number of papers using diffusion models for downscaling (Hatanaka, et al. 2023 for solar irradiance) and nowcasting (Gao, et al. 2023 and Yu, et al. 2024 for precipitation). These are missing from the related work section. 
- The use of transformer-based models is not justified. It seems likely that vision transformer models have a poor inductive bias for this task, which probably has a lower degree of long-range spatiotemporal dependency than in natural images/video. The authors mention "key adaptations to the attention mechanism" but these appear to be simplifications that reduce computation and increase the locality bias --- have the authors tried removing attention all together?
- Some typos that could be caught with a spell checker.
- The baselines used for comparison are weak. They are all deterministic, so it is expected that they will fail to capture the extremes. I don't think it is necessary to have comparisons, but I think this point could be made more clearly in the text. 
- In experiments, only 10 samples are taken from the diffusion model. This seems small in the context of estimating the risk of extreme events. 
- Some clarifications would be helpful (see questions below).
questions: questions: - 280: For an annual average, the diffusion model isn't really necessary. Wouldn't any of the deterministic statistical downscaling models would perform just as well? This should be clarified.
- 294: It was unclear from the text why additional sampling steps in the STVD increases MSE. I imagine this is because the deterministic downscaling model predicts the mean of the possible rainfall (low MSE) and sampling from the distribution of residuals with the diffusion model will almost certainly increase this. So more sampling steps means a better sample of the residual, which means a higher MSE?
limitations: limitations: - As the authors note, the model is demonstrated on simulation data where high-resolution video inputs are provided. In many applications of statistical downscaling, the input and output are from different sources (e.g. observed high-res vs. GFS model output or reanalysis), and there may be inconsistencies. So the experiments done here are on "clean" data, and further experiments are needed to evaluate the method on applications where it will be useful.
paper_summary: paper_summary: This paper presents a novel framework for spatio-temporal precipitation downscaling, comprising two modules: a deterministic downscaling module and a diffusion module. The model is able to outperform the SOTA models, especially in extreme events and in mountainous areas.
strength_weakness: strength_weakness: Experiments are insufficient. Such as the effectiveness of the sharing features across the modules is not proven by ablation experiments.
questions: questions: 1. In line 121, “bicubic interpolation” does not have a explanation. Can you explain it?
2. In line 121, what is ”pixelated features” ?
3. Is the input, additional climate states, the L1 data or L2 data?
limitations: limitations: Is this work of practical value? The authors acknowledge that switching to a different dataset requires retraining due to differences in data distribution. However, can the model be trained if high-resolution ground truth images are not available? If high-resolution images are available, does it make sense to reduce their resolution before training the model?

Comments: comment: Hi, we wanted to follow up on our recent rebuttal submission to ensure that our responses adequately addressed your concerns. If there are any remaining issues or if you have any further questions, we would be grateful for your feedback.

Thank you for your time and consideration.
Hi, we wanted to follow up on our recent rebuttal submission to ensure that our responses adequately addressed your concerns. If there are any remaining issues or if you have any further questions, we would be grateful for your feedback.

Thank you for your time and consideration.
Thanks to the authors for the detailed response. I acknowledge I have read the rebuttal.