Title: PreDiff: Precipitation Nowcasting with Latent Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Machine Learning for Earth Science

Spatiotemporal Forecasting

Generative Models

Diffusion Models
Abstract: Earth system forecasting has traditionally relied on complex physical models that are computationally expensive and require significant domain expertise.
In the past decade, the unprecedented increase in spatiotemporal Earth observation data has enabled data-driven forecasting models using deep learning techniques.
These models have shown promise for diverse Earth system forecasting tasks but either struggle with handling uncertainty or neglect domain-specific prior knowledge, resulting in averaging possible futures to blurred forecasts or generating physically implausible predictions.
To address these limitations, we propose a two-stage pipeline for probabilistic spatiotemporal forecasting: 1) We develop *PreDiff*, a conditional latent diffusion model capable of probabilistic forecasts. 2) We incorporate an explicit knowledge alignment mechanism to align forecasts with domain-specific physical constraints. 
This is achieved by estimating the deviation from imposed constraints at each denoising step and adjusting the transition distribution accordingly.
We conduct empirical studies on two datasets: N-body MNIST, a synthetic dataset with chaotic behavior, and SEVIR, a real-world precipitation nowcasting dataset. 
Specifically, we impose the law of conservation of energy in N-body MNIST and anticipated precipitation intensity in SEVIR. 
Experiments demonstrate the effectiveness of PreDiff in handling uncertainty, incorporating domain-specific prior knowledge, and generating forecasts that exhibit high operational utility.

PreDiff: Precipitation Nowcasting with Latent Diffusion Models

Abstract

Earth system forecasting has traditionally relied on complex physical models that are computationally expensive and require significant domain expertise. In the past decade, the unprecedented increase in spatiotemporal Earth observation data has enabled data-driven forecasting models using deep learning techniques. These models have shown promise for diverse Earth system forecasting tasks. However, they either struggle with handling uncertainty or neglect domain-specific prior knowledge; as a result, they tend to suffer from averaging possible futures to blurred forecasts or generating physically implausible predictions. To address these limitations, we propose a two-stage pipeline for probabilistic spatiotemporal forecasting: 1) We develop PreDiff, a conditional latent diffusion model capable of probabilistic forecasts. 2) We incorporate an explicit knowledge alignment mechanism to align forecasts with domain-specific physical constraints. This is achieved by estimating the deviation from imposed constraints at each denoising step and adjusting the transition distribution accordingly. We conduct empirical studies on two datasets: N -body MNIST, a synthetic dataset with chaotic behavior, and SEVIR, a real-world precipitation nowcasting dataset. Specifically, we impose the law of conservation of energy in N -body MNIST and anticipated precipitation intensity in SEVIR. Experiments demonstrate the effectiveness of PreDiff in handling uncertainty, incorporating domain-specific prior knowledge, and generating forecasts that exhibit high operational utility.

1 Introduction

Earth's intricate climate system significantly influences daily life. Precipitation nowcasting, tasked with delivering accurate rainfall forecasts for the near future (e.g., 0-6 hours), is vital for decisionmaking across numerous industries and services. Recent advancements in data-driven deep learning (DL) techniques have demonstrated promising potential in this field, rivaling conventional numerical methods [8, 5] with their advantages of being more skillful [5], efficient [37], and scalable [3]. However, accurately predicting the future rainfall remains challenging for data-driven algorithms.

Xingjian Shi †

Boson AI

Xiaoyong Jin

The state-of-the-art Earth system forecasting algorithms [47, 61, 41, 37, 8, 69, 2, 29, 3] typically generates blurry predictions. This is caused by the high variability and complexity inherent to Earth's climatic system. Even minor differences in initial conditions can lead to vastly divergent outcomes that are difficult to predict. Most methods adopt a point estimation of the future rainfall and are trained by minimizing pixel-wise loss functions (e.g., mean-squared error). These methods lack the capability of capturing multiple plausible futures and will generate blurry forecasts which lose important operational details. Therefore, what are needed instead are probabilistic models that can represent the uncertainty inherent in stochastic systems. The probabilistic models can capture multiple plausible futures, generating diverse high-quality predictions that better align with real-world data.

The emergence of diffusion models (DMs) [22] has enabled powerful probabilistic frameworks for generative modeling. DMs have shown remarkable capabilities in generating high-quality images [40, 45, 43] and videos [15, 23]. As a likelihood-based model, DMs do not exhibit mode collapse or training instabilities like GANs [10]. Compared to autoregressive (AR) models [53, 46, 63, 39, 65] that generate images pixel-by-pixel, DMs can produce higher resolution images faster and with higher quality. They are also better at handling uncertainty [62, 34, 57-59] without drawbacks like exposure bias [13] in AR models. Latent diffusion models (LDMs) [42, 52] further improve on DMs by separating the model into two phases, only applying the costly diffusion in a compressed latent space. This alleviates the computational costs of DMs without significantly impairing performance.

Despite DMs' success in image and video generation [42, 15, 66, 36, 32, 56], its application to precipitation nowcasting and Earth system forecasting is in early stages [16]. One of the major concerns is that this purely data-centric approach lacks constraints and controls from prior knowledge about the dynamic system. Some spatiotemporal forecasting approaches have incorporated domain knowledge by modifying the model architecture or adding extra training losses [11, 1, 37]. This enables them to be aware of prior knowledge and generate physically plausible forecasts. However, these approaches still face challenges, such as requiring to design new model architectures or retrain the entire model from scratch when constraints change. More detailed discussions on related works are provided in Appendix A.

Inspired by recent success in controllable generative models [68, 24, 4, 33, 6], we propose a general two-stage pipeline for training data-driven Earth system forecasting model. 1) In the first stage, we focus on capturing the intrinsic semantics in the data by training an LDM. To capture Earth's long-term and complex changes, we instantiate the LDM's core neural network as a UNet-style architecture based on Earthformer [8]. 2) In the second stage, we inject prior knowledge of the Earth system by training a knowledge alignment network that guides the sampling process of the LDM. Specifically the alignment network parameterizes an energy function that adjusts the transition probabilities during each denoising step. This encourages the generation of physically plausible intermediate latent states while suppressing those likely to violate the given domain knowledge. We summarize our main contributions as follows:

We introduce a novel LDM based model PreDiff for precipitation nowcasting.

We propose a general two-stage pipeline for training data-driven Earth system forecasting models. Specifically, we develop knowledge alignment mechanism to guide the sampling process of PreDiff. This mechanism ensures that the generated predictions align with domain-specific prior knowledge better, thereby enhancing the reliability of the forecasts, without requiring any modifications to the trained PreDiff model.

Our method achieves state-of-the-art performance on the N -body MNIST [8] dataset and attains state-of-the-art perceptual quality on the SEVIR [55] dataset.

2 Method

We follow [47, 48, 55, 1, 8] to formulate precipitation nowcasting as a spatiotemporal forecasting problem. The L in-step observation is represented as a spatiotemporal sequence y = [ y j ] L in j =1 ∈ R L in × H × W × C , where H and W denote the spatial resolution, and C denotes the number of measurements at each space-time coordinate. Probabilistic forecasting aims to model the conditional probabilistic distribution p ( x | y ) of the L out-step-ahead future x = [ x j ] L out j =1 ∈ R L out × H × W × C , given the observation y . In what follows, we will present the parameterization of p ( x | y ) by a controllable LDM.

⋯

⋯

𝑦

𝑧

!"#$

, 𝑧

%

𝑧

!"#$

, 𝑧

&'(

𝑧

!"#$

, 𝑧

&

𝑧

!"#$

, 𝑧

)

𝑥 #

𝑝

!,#

(𝑧

$

|𝑧

$%&

, 𝑦, ℱ

'

)

𝑝

!

𝑧

$

𝑧

$%&

, 𝑧

()*+

Earthformer-UNet

Knowledge Alignment

𝑒

,-

ℱ

.

"

/

#

,$,	1 ,ℱ

$

(1)

×

=

×𝑇

denoise

D

E

2.1 Preliminary: Diffusion Models

Diffusion models (DMs) learn the data distribution p ( x ) by training a model to reverse a predefined noising process that progressively corrupts the data. Specifically, the noising process is defined as q ( x t | x t -1 ) = N ( x t ; √ α t x t -1 , (1 -α t ) I ) , 1 ≤ t ≤ T , where x 0 ∼ p ( x ) is the true data, and x T ∼ N (0 , I ) is random noise. The coefficients α t follow a fixed schedule over the timesteps t . DMs factorize and parameterize the joint distribution over the data x 0 and noisy latents x i as p θ ( x 0: T ) = p ( x T ) ∏ T t =1 p θ ( x t -1 | x t ) , where each step of the reverse denoising process is a Gaussian distribution p θ ( x t -1 | x t ) = N ( µ θ ( x t , t ) , Σ θ ( x t , t )) , which is trained to recover x t -1 from x t .

To apply DMs for spatiotemporal forecasting, p ( x | y ) is factorized and parameterized as p θ ( x | y ) = ∫ p θ ( x 0: T | y ) dx 1: T = ∫ p ( x T ) ∏ T t =1 p θ ( x t -1 | x t , y ) dx 1: T , where p θ ( x t -1 | x t , y ) represents the conditional denoising transition with the condition y .

2.2 Conditional Diffusion in Latent Space

To improve the computational efficiency of DM training and inference, our PreDiff follows LDM to adopt a two-phase training that leverages the benefits of lower-dimensional latent representations. The two sequential phases of the PreDiff training are: 1) Training a frame-wise variational autoencoder (VAE) [28] that encodes pixel space into a lower-dimensional latent space, and 2) Training a conditional DM that generates predictions in this acquired latent space.

Frame-wise autoencoder. We follow [7] to train a frame autoencoder using a combination of the pixel-wise loss (e.g. L2 loss) and an adversarial loss. Different from [7], we exclude the perceptual loss since there are no standard pretrained models for perception on Earth observation data. Specifically, the encoder E is trained to encode a data frame x j ∈ R H × W × C to a latent representation z j = E ( x j ) ∈ R H z × W z × C z . The decoder D learns to reconstruct the data frame ̂ x j = D ( z j ) from

the encoded latent. We denote z ∼ p E ( z | x ) ∈ R L × H z × W z × C z as equivalent to z = [ z j ] = [ E ( x j )] , representing encoding a sequence of frames in pixel space into a latent spatiotemporal sequence. And x ∼ p D ( x | z ) denotes decoding a latent spatiotemporal sequence.

Latent diffusion. With the context y being encoded by the frame-wise encoder E into the learned latent space as z cond ∈ R L in × H z × W z × C z as (1). The conditional distribution p θ ( z 0: T | z cond ) of the latent future z i ∈ R L out × H z × W z × C z given z cond is factorized and parameterized as (2):

where z T ∼ p ( z T ) = N (0 , I ) . As proposed by [22, 45],an equivalent parameterization is to have the DMs learn to match the transition noise ϵ θ ( z t , t ) of step t instead of directly predicting z t -1 . The training objective of PreDiff is simplified as shown in (3):

where ( x, y ) is a sampled context sequence and target sequence data pair, and given that, z t ∼ q ( z t | z 0 ) p E ( z 0 | x ) and z cond ∼ p E ( z cond | y ) .

Instantiating p θ ( z t -1 | z t , z cond ) . Compared to images, modeling spatiotemporal observation data in precipitation nowcasting poses greater challenges due to their higher dimensionality. We propose replacing the UNet backbone in LDM [42] with Earthformer-UNet , derived from Earthformer's encoder [8], which is known for its ability to model intricate and extensive spatiotemporal dependencies in the Earth system.

Earthformer-UNet adopts a hierarchical UNet architecture with self cuboid attention [8] as the building blocks, excluding the bridging cross-attention in the encoder-decoder architecture of Earthformer. More details of the architecture design of Earthformer-UNet are provide in Appendix B.1. We find Earthformer-UNet to be more stable and effective at modeling the transition distribution p θ ( z t -1 | z t , z cond ) . It takes the concatenation of the encoded latent context z cond and the noisy latent future z t along the temporal dimension as input, and predicts the one-step-ahead noisy latent future z t -1 (in practice, the transition noise ϵ from z t to z t -1 is predicted as shown in (3)).

2.3 Incorporating Knowledge Alignment

Though DMs hold great promise for diverse and realistic generation, the generated predictions may violate physical constraints, or disregard domain-specific prior knowledge, thereby fail to give plausible and non-trivial results [14, 44]. One possible reason for this is that DMs are not necessarily trained on data full compliant with domain knowledge. When trained on such data, there is no guarantee that the generations sampled from the learned distribution will re-

Algorithm 1 One training step of the knowledge alignment network U ϕ

1:

sampled from data

2:

3:

4:

(

x, y

t

∼

)

Uniform

z

t

∼

L

U

(0

, T

)

q

(

z

t

|

z

0

)

←∥

U

ϕ

(

p

E

z

t

(

z

0

, t, y

|

)

x

)

-F

(

x, y

)

∥

main physically realizable. The causes may also stem from the stochastic nature of chaotic systems, the approximation error in denoising steps, etc.

To address this issue, we propose knowledge alignment to incorporate auxiliary prior knowledge:

into the diffusion generation process. The knowledge alignment imposes a constraint F on the forecast ̂ x , optionally with the observation y , based on domain expertise. E.g., for an isolated physical system, the knowledge E ( ̂ x, · ) = E 0 ( y L in ) ∈ R imposes the conservation of energy by enforcing the generation ̂ x to keep the total energy E ( ̂ x, · ) the same as the last observation E 0 ( y L in ) . The violation ∥F ( ̂ x, y ) -F 0 ( y ) ∥ quantifies the deviation of a prediction ̂ x from prior knowledge. The larger violation indicates ̂ x diverges further from the constraints. Knowledge alignment hence aims to suppress the probability of generating predictions with large violation. Notice that even the target

̸

̸

futures x from training data may violate the knowledge, i.e. F ( x, y ) = F 0 ( y ) , due to noise in data collection or simulation.

Inspired by classifier guidance [4], we achieve knowledge alignment by training a knowledge alignment network U ϕ ( z t , t, y ) to estimate F ( ̂ x, y ) from the intermediate latent z t at noising step t . The key idea is to adjust the transition probability distribution p θ ( z t -1 | z t , z cond ) in (2) during each latent denoising step to reduce the likelihood of sampling z t values expected to violate the constraints:

where λ F is a guidance scale factor. The knowledge alignment network is trained by optimizing the objective L U in Alg. 1. According to [4], (5) can be approximated by shifting the predicted mean of the denoising transition µ θ ( z t +1 , t, z cond ) by -λ F Σ θ ∇ z t ∥ U ϕ ( z t , t, y ) -F 0 ( y ) ∥ , where Σ θ is the variance of the original transition distribution p θ ( z t | z t +1 , z cond ) = N ( µ θ ( z t +1 , t, z cond ) , Σ θ ( z t +1 , t, z cond )) . Detailed derivation is provided in Appendix C.

The training procedure of knowledge alignment is outlined in Alg. 1. The noisy latent z t for training the knowledge alignment network U ϕ is sampled by encoding the target x using the frame-wise encoder E and the forward noising process q ( z t | z 0 ) , eliminating the need for an inference sampling process. This makes the training of the knowledge alignment network U ϕ independent of the LDM training. At inference time, the knowledge alignment mechanism is applied as a plug-in, without impacting the trained V AE and the LDM. This modular approach allows training lightweight knowledge alignment networks U ϕ to flexibly explore various constraints and domain knowledge, without the need for retraining the entire model. This stands as a key advantage over incorporating constraints into model architectures or training losses.

3 Experiments

We conduct empirical studies and compare PreDiff with other state-of-the-art spatiotemporal forecasting models on a synthetic dataset N -body MNIST [8] and a real-world precipitation nowcasting benchmark SEVIR 2 [55] to verify the effectiveness of PreDiff in handling the dynamics and uncertainty in complex spatiotemporal systems and generating high quality, accurate forecasts. We impose data-specific knowledge alignment: energy conservation on N -body MNIST and anticipated precipitation intensity on SEVIR. Experiments demonstrate that PreDiff under the guidance of knowledge alignment (PreDiff-KA) is able to generate predictions that comply with domain expertise much better, without severely sacrificing fidelity.

3.1 N -body MNIST Digits Motion Forecasting

Dataset. The Earth is a chaotic system with complex dynamics. The real-world Earth observation data, such as radar echo maps and satellite imagery, are usually not physically complete. We are unable to directly verify whether certain domain knowledge, like conservation laws of energy and momentum, is satisfied or not. This makes it difficult to verify if a method is really capable of modeling certain dynamics and adhering to the corresponding constraints. To address this, we follow [8] to generate a synthetic dataset named N -body MNIST 3 , which is an extension of MovingMNIST [50]. The dataset contains sequences of digits moving subject to the gravitational force from other digits. The governing equation for the motion is d 2 x i dt 2 = -∑ j = i Gm j ( x i -x j ) ( | x i -x j | + d soft ) r , where x i is the spatial coordinates of the i -th digit, G is the gravitational constant, m j is the mass of the j -th digit, r is a constant representing the power scale in the gravitational law, and d soft is a small softening distance that ensures numerical stability. The motion occurs within a 64 × 64 frame. When a digit hits the boundaries of the frame, it bounces back by elastic collision. We use N = 3 for chaotic 3 -body motion [35]. The forecasting task is to predict 10 -step ahead future frames x ∈ R 10 × 64 × 64 × 1 given the length10 context y ∈ R 10 × 64 × 64 × 1 . We generate 20,000 sequences for training and 1,000 sequences for testing. Empirical studies on such a synthetic dataset with known dynamics helps provide useful insights for model development and evaluation.

Target:

𝑥

E.MSE=0.0261

ConvLSTM

E.MSE=0.0329

Earthformer

E.MSE=0.0296

VideoGPT

E.MSE=0.0253

LDM

E.MSE=0.0354

PreDiff

E.MSE=0.0277

PreDiff-KA

E.MSE=0.0086

step 1

step 2

step 3

step 4

step 5

step 6

step 7

step 8

step 9

step 10

Context:

𝑦

6

6

6

又

6

6

6

#

又

6

飞

?6

双

6

6

6

，

R

?

6

6

2

6

##

##

6

6

3

2

吃

。

#

a6

6

6

6

2

6

6

26

6

26

飞

又

6

6

26

6

26

文

26

飞

6

Evaluation. In addition to standard metrics MSE, MAE and SSIM, we also report the scores of Fréchet Video Distance (FVD) [51], a metric for evaluating the visual quality of generated videos. Similar to Fréchet Inception Distance (FID) [20] for evaluating image generation, FVD estimates the distance between the learned distribution and the true data distribution by comparing the statistics of feature vectors extracted from the generations and the real data. The inception network used in FVD for feature extraction is pre-trained on video classification and is not specifically adapted for processing "unnatural videos" such as spatiotemporal observation data in Earth systems. Consequently, the FVD scores on the N -body MNIST and SEVIR datasets cannot be directly compared with those on natural video datasets. Nevertheless, the relative ranking of the FVD scores remains a meaningful indicator of model ability to achieve high visual quality, as FVD has shown consistency with expert evaluations across various domains beyond natural images [38, 26]. Scores for all involved metrics are calculated using an ensemble of eight samples from each model.

3.1.1 Comparison with the State of the Art

We evaluate seven deterministic spatiotemporal forecasting models: UNet [55], ConvLSTM [47], PredRNN [61], PhyDNet [11], E3D-LSTM [60], Rainformer [1] and Earthformer [8], as well as two probabilistic spatiotemporal forecasting models: VideoGPT [65] and LDM [42]. All baselines are trained following the default configurations in their officially released code. More implementation details of baselines are provided in Appendix B.2. Results in Table 1 show that PreDiff outperforms these baselines by a large margin in both conventional video prediction metrics (i.e., MSE, MAE, SSIM), and a perceptual quality metric, FVD. The example predictions in Fig. 2 demonstrate that PreDiff generate predictions with sharp and clear digits in accurate positions. In contrast, deterministic baselines resort to generating blurry predictions to accommodate uncertainty. Probabilistic baselines, though producing sharp strokes, either predict incorrect positions or fail to reconstruct the digits. The performance gap between LDM [42] and PreDiff serves as an ablation study that highlights the importance of the latent backbone's spatiotemporal modeling capacity. Specifically, the EarthformerUNet utilized in PreDiff demonstrates superior performance compared to the UNet in LDM [42].

3.1.2 Knowledge Alignment: Energy Conservation

In the N -body MNIST simulation, digits move based on Newton's law of gravity, and interact with the boundaries through elastic collisions. Consequently, this system obeys the law of conservation of

energy. The total energy of the whole system E ( x j ) at any future time step j during evolution should equal the total energy at the last observation time step E ( y L in ) .

We impose the law of conservation of energy for the knowledge alignment on N -body MNIST in the form of (4) :

The ground-truth values of the total energy E ( y L in ) and E ( x j ) are directly accessible since N -body MNIST is a synthetic dataset from simulation. The total energy can be derived from the velocities (kinetic energy) and positions (potential energy) of the moving digits. A knowledge alignment network U ϕ is trained following Alg. 1 to guide the PreDiff to generate forecasts ̂ x that conserve the same energy as the initial step E ( y L in ) .

To verify the effectiveness of the knowledge alignment on guiding the generations to comply with the law of conservation of energy, we train an energy detector E det ( ̂ x ) 4 that detects the total energy of the forecasts ̂ x . We evaluate the energy error between the forecasts and the initial energy using E.MSE ( ̂ x, y ) ≡ MSE ( E det ( ̂ x ) , E ( y L in )) and E.MAE ( ̂ x, y ) ≡ MAE ( E det ( ̂ x ) , E ( y L in )) . In this evaluation, we exclude the methods that generate blurred predictions with ambiguous digit positions. We only focus on the methods that are capable of producing clear digits in precise positions.

As illustrated in Table 1, PreDiff-KA substantially outperforms all baseline methods and PreDiff without knowledge alignment in E.MSE and E.MAE. This demonstrates that the forecasts of PreDiffKA comply much better with the law of conservation of energy, while still maintaining high visual quality with an FVD score of 4 . 063 .

Furthermore, we detect energy errors in the target data sequences. The first row of Table 1 indicates that even the target from the training data may not strictly adhere to the prior knowledge. This could be due to discretization errors in the simulation. Table 1 shows that all baseline methods and PreDiff have larger energy errors than the target, meaning purely data-oriented approaches cannot eliminate the impact of noise in the training data. In contrast, PreDiff-KA, guided by the law of conservation of energy, overcomes the intrinsic defects in the training data, achieving even lower energy errors compared to the target.

A typical example shown in Fig. 2 demonstrates that while PreDiff precisely reproduces the groundtruth position of digit '2' in the last frame (aligned to the red dashed line), resulting in nearly the same energy error (E.MSE = 0 . 0277 ) as the ground-truth's (E.MSE = 0 . 0261 ), PreDiff-KA successfully

Context:

𝑦

Target:

𝑥

ConvLSTM

CSI = 0.339

CSI-p16 = 0.471

Earthformer

CSI = 0.339

CSI-p16 = 0.454

VideoGPT

CSI = 0.309

CSI-p16 = 0.566

LDM

CSI = 0.266

CSI-p16 = 0.568

PreDiff

CSI = 0.329

CSI-p16 = 0.619

+10 Min

+20 Min

+30 Min

+40 Min

+50 Min

+60 Min

-50 Min

-40 Min

-30 Min

-20 Min

-10 Min

0 Min

corrects the motion of digit '2', providing it with physically plausible velocity and position (slightly off the red dashed line). The knowledge alignment ensures that the generation complies better with the law of conservation of energy, resulting in a much lower E.MSE = 0 . 0086 . On the contrary, none of the evaluated baselines can overcome the intrinsic noise from the data, resulting in energy errors comparable to or larger than that of the ground-truth.

Notice that the pixel-wise scores MSE, MAE and SSIM are less meaningful for evaluating PreDiffKA, since correcting the noise of the energy results in changing the velocities and positions of the digits. A minor change in the position of a digit can cause a large pixel-wise error, even though the digit is still generated sharply and in high quality as shown in Fig. 2.

3.2 SEVIR Precipitation Nowcasting

Dataset. The Storm EVent ImageRy (SEVIR) [55] is a spatiotemporal Earth observation dataset which consists of 384 km × 384 km image sequences spanning over 4 hours. Images in SEVIR are sampled and aligned across five different data types: three channels (C02, C09, C13) from the GOES-16 advanced baseline imager, NEXRAD Vertically Integrated Liquid (VIL) mosaics, and GOES-16 Geostationary Lightning Mapper (GLM) flashes. The SEVIR benchmark supports scientific research on multiple meteorological applications including precipitation nowcasting, synthetic radar generation, front detection, etc. Due to computational resource limitations, we adopt a downsampled version of SEVIR for benchmarking precipitation nowcasting. The task is to predict the future VIL up to 60 minutes (6 frames) given 70 minutes of context VIL (7 frames) at a spatial resolution of 128 × 128 , i.e. x ∈ R 6 × 128 × 128 × 1 , y ∈ R 7 × 128 × 128 × 1 .

Evaluation. Following [55, 8], we adopt the Critical Success Index ( CSI ) for evaluation, which is commonly used in precipitation nowcasting and is defined as CSI = # Hits # Hits +# Misses +# F . Alarms . To count the # Hits (truth=1, pred=1), # Misses (truth=1, pred=0) and # F . Alarms (truth=0, pred=1), the prediction and the ground-truth are rescaled to the range 0 -255 and binarized at thresholds [16 , 74 , 133 , 160 , 181 , 219] . We also follow [41] to report the CSI at pooling scale 4 × 4 and 16 × 16 , which evaluate the performance on neighborhood aggregations at multiple spatial scales. These pooled CSI metrics assess the models' ability to capture local pattern distributions. Additionally, we incorporate FVD [51] and continuous ranked probability score

Context:

𝑦

Target:

𝑥

AvgInt = 0.137

+10 Min

+20 Min

+30 Min

+40 Min

+50 Min

+60 Min

-50 Min

-40 Min

-30 Min

-20 Min

-10 Min

0 Min

!

AvgInt = 0.201

+2𝜎

!

AvgInt = 0.192

PreDiff

AvgInt = 0.126

-2𝜎

!

AvgInt = 0.104

-4𝜎

!

AvgInt = 0.067

(CRPS) [9] for assessing the visual quality and uncertainty modeling capabilities of the investigated methods. CRPS measures the discrepancy between the predicted distribution and the true distribution. When the predicted distribution collapses into a single value, as in deterministic models, CRPS reduces to Mean Absolute Error (MAE). A lower CRPS value indicates higher forecast accuracy. Scores for all involved metrics are calculated using an ensemble of eight samples from each model.

3.2.1 Comparison to the State of the Art

We adjust the configurations of involved baselines accordingly and tune some of the hyperparameters for adaptation on the SEVIR dataset. More implementation details of baselines are provided in Appendix B.2. The experiment results listed in Table 2 show that probabilistic spatiotemporal forecasting methods are not good at achieving high CSI scores. However, they are more powerful at capturing the patterns and the true distribution of the data, hence achieving much better FVD scores and CSI -pool 16 . Qualitative results shown in Fig. 3 demonstrate that CSI is not aligned with human perceptual judgement. For such a complex system, deterministic methods give up capturing the real patterns and resort to averaging the possible futures, i.e. blurry predictions, to keep the scores from appearing too inaccurate. Probabilistic approaches, of which PreDiff is the best, though are not favored by per-pixel metrics, perform better at capturing the data distribution within a local area, resulting in higher CSI -pool 16 , lower CRPS, and succeed in keeping the correct local patterns, which can be crucial for recognizing weather events. More detailed quantitative results on SEVIR are provided in Appendix D.

3.2.2 Knowledge Alignment: Anticipated Average Intensity

Earth system observation data, such as the Vertically Integrated Liquid (VIL) data in SEVIR, are usually not physically complete, posing challenges for directly incorporating physical laws for guidance. However, with highly flexible knowledge alignment mechanism, we can still utilize auxiliary prior knowledge to guide the forecasting effectively. Specifically for precipitation nowcasting on SEVIR, we use anticipated precipitation intensity to align the generations to simulate possible extreme weather events. We denote the average intensity of a data sequence as I ( x ) ∈ R + . In order to estimate the conditional quantiles of future intensity, we train a simple probabilistic time series forecasting model with a parametric (Gaussian) distribution p τ ( I ( x ) | [ I ( y j )]) = N ( µ τ ([ I ( y j )]) , σ τ ([ I ( y j )])) that predict

the distribution of the average future intensity I ( x ) given the average intensity of each context frame [ I ( y j )] L in j =1 (abbreviated as [ I ( y j )] ). By incorporating F ( ̂ x, y ) ≡ I ( ̂ x ) and F 0 ( y ) ≡ µ τ + nσ τ for knowledge alignment, PreDiff-KA gains the capability of generating forecasts for potential extreme cases, e.g., where I ( ̂ x ) falls outside the typical range of µ τ ± σ τ .

Fig. 4 shows a set of generations from PreDiff and PreDiff-KA with anticipated future intensity µ τ + nσ τ , n ∈ {-4 , -2 , 2 , 4 } . This qualitative example demonstrates that PreDiff is not only capable of capturing the distribution of the future, but also flexible at highlighting possible extreme cases like rainstorms and droughts with the knowledge alignment mechanism, which is crucial for decision-making and precaution.

According to Table 2, the FVD score of PreDiff-KA ( 34 . 18 ) is only slightly worse than the FVD score of PreDiff ( 33 . 05 ). This indicates that knowledge alignment effectively aligns the generations with prior knowledge while maintaining fidelity and adherence to the true data distribution.

4 Conclusions and Broader Impacts

In this paper, we propose PreDiff, a novel latent diffusion model for precipitation nowcasting. We also introduce a general two-stage pipeline for training DL models for Earth system forecasting. Specifically, we develop knowledge alignment mechanism that is capable of guiding PreDiff to generate forecasts in compliance with domain-specific prior knowledge. Experiments demonstrate that our method achieves state-of-the-art performance on N -body MNIST and SEVIR datasets.

Our work has certain limitations: 1) Benchmark datasets and evaluation metrics for precipitation nowcasting and Earth system forecasting are still maturing compared to the computer vision domain. While we utilize conventional precipitation forecasting metrics and visual quality evaluation, aligning these assessments with expert judgement remains an open challenge. 2) Effective integration of physical principles and domain knowledge into DL models for precipitation nowcasting remains an active research area. Close collaboration between DL researchers and domain experts in meteorology and climatology will be key to developing hybrid models that effectively leverage both data-driven learning and scientific theory. 3) While Earth system observation data have grown substantially in recent years, high-quality data remain scarce in many domains. This scarcity can limit PreDiff's ability to accurately capture the true distribution, occasionally resulting in unrealistic forecast hallucinations under the guidance of prior knowledge as it attempts to circumvent the knowledge alignment mechanism. Further research on enhancing the sample efficiency of PreDiff and the knowledge alignment mechanism is needed.

In conclusion, PreDiff represents a promising advance in knowledge-aligned DL for Earth system forecasting, but work remains to improve benchmarking, incorporate scientific knowledge, and boost model robustness through collaborative research between AI and domain experts.

References

Cong Bai, Feng Sun, Jinglin Zhang, Yi Song, and Shengyong Chen. Rainformer: Features extraction balanced network for radar-based precipitation nowcasting. IEEE Geoscience and Remote Sensing Letters , 19:1-5, 2022.

Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nature , pages 1-6, 2023.

Kang Chen, Tao Han, Junchao Gong, Lei Bai, Fenghua Ling, Jing-Jia Luo, Xi Chen, Leiming Ma, Tianning Zhang, Rui Su, et al. Fengwu: Pushing the skillful global medium-range weather forecast beyond 10 days lead. arXiv preprint arXiv:2304.02948 , 2023.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Lasse Espeholt, Shreya Agrawal, Casper Sønderby, Manoj Kumar, Jonathan Heek, Carla Bromberg, Cenk Gazen, Jason Hickey, Aaron Bell, and Nal Kalchbrenner. Skillful twelve hour precipitation forecasts using large context neural networks. arXiv preprint arXiv:2111.07470 , 2021.

Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011 , 2023.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873-12883, 2021.

Zhihan Gao, Xingjian Shi, Hao Wang, Yi Zhu, Yuyang Wang, Mu Li, and Dit-Yan Yeung. Earthformer: Exploring space-time transformers for earth system forecasting. In NeurIPS , 2022.

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association , 102(477):359-378, 2007.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11474-11484, 2020.

John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587 , 2021.

Shantanu Gupta, Hao Wang, Zachary Lipton, and Yuyang Wang. Correcting exposure bias for link recommendation. In ICML , 2021.

Derek Hansen, Danielle C. Maddix, Shima Alizadeh, Gaurav Gupta, and Michael W. Mahoney. Learning physical models that can respect conservation laws. In Proceedings of the 40 th of International Conference on Machine Learning , volume 202, 2023.

William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. arXiv preprint arXiv:2205.11495 , 2022.

Yusuke Hatanaka, Yannik Glaser, Geoff Galgon, Giuseppe Torri, and Peter Sadowski. Diffusion models for high-resolution solar forecasts. arXiv preprint arXiv:2302.00170 , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín MuñozSabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society , 146(730):1999-2049, 2020.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 , 2012.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 , 2022.

Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778 , 2023.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr \ 'echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466 , 2018.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Alexander Pritzel, Suman Ravuri, Timo Ewalds, Ferran Alet, Zach Eaton-Rosen, et al. Graphcast: Learning skillful medium-range global weather forecasting. arXiv preprint arXiv:2212.12794 , 2022.

Jussi Leinonen, Ulrich Hamann, Daniele Nerini, Urs Germann, and Gabriele Franch. Latent diffusion models for generative precipitation nowcasting with accurate uncertainty quantification. arXiv preprint arXiv:2304.12891 , 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv e-prints , pages arXiv-2303, 2023.

Fran c cois Mazé and Faez Ahmed. Topodiff: A performance and constraint-guided diffusion model for topology optimization. arXiv preprint arXiv:2208.09591 , 2022.

Lu Mi, Hao Wang, Yonglong Tian, and Nir Shavit. Training-free uncertainty estimation for neural networks. In AAAI , 2022.

Valtonen MJ, Mauri Valtonen, and Hannu Karttunen. The three-body problem . Cambridge University Press, 2006.

Haomiao Ni, Changhao Shi, Kai Li, Sharon X. Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models, 2023.

Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. FourCastNet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.11214 , 2022.

Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Gunter Klambauer. Fréchet chemnet distance: a metric for generative models for molecules in drug discovery. Journal of chemical information and modeling , 58(9):1736-1741, 2018.

Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprint arXiv:2006.10704 , 2020.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Suman Ravuri, Karel Lenc, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, et al. Skilful precipitation nowcasting using deep generative models of radar. Nature , 597(7878):672-677, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22500-22510, 2023.

Nadim Saad, Gaurav Gupta, Shima Alizadeh, and Danielle C. Maddix. Guiding continuous operator learning through physics-based boundary constraints. In Proceedings of the 11 th International Conference on Learning Representations , 2023.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 , 2022.

Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 , 2017.

Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In NeurIPS , volume 28, 2015.

Xingjian Shi, Zhihan Gao, Leonard Lausen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun Woo. Deep learning for precipitation nowcasting: A benchmark and a new model. In NeurIPS , volume 30, 2017.

Casper Kaae Sønderby, Lasse Espeholt, Jonathan Heek, Mostafa Dehghani, Avital Oliver, Tim Salimans, Shreya Agrawal, Jason Hickey, and Nal Kalchbrenner. Metnet: A neural weather model for precipitation forecasting. arXiv preprint arXiv:2003.12140 , 2020.

Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using LSTMs. In ICML , pages 843-852. PMLR, 2015.

Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. In DGS@ICLR , 2019.

Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Neural Information Processing Systems (NeurIPS) , 2021.

Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. Advances in neural information processing systems , 29, 2016.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , volume 30, 2017.

Mark Veillette, Siddharth Samsi, and Chris Mattioli. SEVIR: A storm event imagery dataset for deep learning applications in radar and satellite meteorology. Advances in Neural Information Processing Systems , 33:22009-22019, 2020.

Vikram Voleti, Alexia Jolicoeur-Martineau, and Christopher Pal. Masked conditional video diffusion for prediction, generation, and interpolation. arXiv preprint arXiv:2205.09853 , 2022.

Hao Wang, SHI Xingjian, and Dit-Yan Yeung. Natural-parameter networks: A class of probabilistic neural networks. In NIPS , pages 118-126, 2016.

Hao Wang and Dit-Yan Yeung. Towards bayesian deep learning: A framework and some existing methods. TDKE , 28(12):3395-3408, 2016.

Hao Wang and Dit-Yan Yeung. A survey on bayesian deep learning. CSUR , 53(5):1-37, 2020.

Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3D LSTM: A model for video prediction and beyond. In International conference on learning representations , 2018.

Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip Yu, and Mingsheng Long. PredRNN: A recurrent neural network for spatiotemporal predictive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Ziyan Wang and Hao Wang. Variational imbalanced regression: Fair uncertainty quantification via probabilistic smoothing. In NeurIPS , 2023.

Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In International Conference on Learning Representations , 2019.

Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV) , pages 3-19, 2018.

Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157 , 2021.

Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. arXiv preprint arXiv:2212.02500 , 2022.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023.

Lu Zhou and Rong-Hua Zhang. A self-attention-based neural network for three-dimensional multivariate modeling and its skillful enso predictions. Science Advances , 9(10):eadf2827, 2023.

A Related Work

Deep learning for precipitation nowcasting In recent years, the field of DL has experienced remarkable advancements, revolutionizing various domains of study, including Earth science. One area where DL has particularly made significant strides is in the field of Earth system forecasting, especially precipitation nowcasting. Precipitation nowcasting benefits from the success of DL architectures such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformers, which have demonstrated their effectiveness in handling spatiotemporal tensors, the typical formulation for Earth system observation data. ConvLSTM [47], a pioneering approach in DL for precipitation nowcasting, combines the strengths of CNNs and LSTMs processing spatial and temporal data. PredRNN [61] builds upon ConvLSTM by incorporating a spatiotemporal memory flow structure. E3D-LSTM [60] integrates 3D CNN to LSTM to enhance long-term high-level relation modeling. PhyDNet [11] incorporated partial differential equation (PDE) constraints in the latent space. MetNet [49] and its successor, MetNet-2 [5], propose architectures based on ConvLSTM and dilated CNN, enabling skillful precipitation forecasts up to twelve hours ahead. DGMR [41] takes an adversarial training approach to generate sharp and accurate nowcasts, addressing the issue of blurry predictions.

In addition to precipitation nowcasting, there has been a surge in the modeling of global weather and medium-range weather forecasting due to the availability of extensive Earth observation data, such as the European Centre for Medium-Range Weather Forecasts (ECMWF)'s ERA5 [19] dataset. Several DL-based models have emerged in this area. FourCastNet [37] proposes an architecture with Adaptive Fourier Neural Operators (AFNO) [12] as building blocks for autoregressive weather forecasting. FengWu [3] introduces a multi-model Transformer-based global medium-range weather forecast model that achieves skillful forecasts up to ten days ahead. GraphCast [29] combines graph neural networks with convolutional LSTMs to tackle sub-seasonal forecasting tasks, representing weather phenomena as spatiotemporal graphs. Pangu-Weather [2] proposes a 3D Transformer model with Earth-specific priors and a hierarchical temporal aggregation strategy for medium-range global weather forecasting. While recent years have seen remarkable progress in DL for precipitation nowcasting, existing methods still face some limitations. Some methods are deterministic, failing to capture uncertainty and resulting in blurry generation. Others lack the capability of incorporating prior knowledge, which is crucial for machine learning for science. In contrast, PreDiff captures the uncertainty in the underlying data distribution via diffusion models, avoiding simply averaging all possibilities into blurry forecasts. Our knowledge alignment mechanism facilitates post-training alignment with physical principles and domain-specific prior knowledge.

Diffusion models Diffusion models (DMs) [22] are a class of generative models that have become increasingly popular in recent years. DMs learn the data distribution by constructing a forward process that adds noise to the data, and then approximating the reverse process to remove the noise. Latent diffusion models (LDMs) [42] are a variant of DMs that are trained on latent vector outputs from a variational autoencoder. LDMs have been shown to be more efficient in both training and inference compared to original DMs. Building on the success of DMs in image generation, DMs have also been adopted for video generation. MCVD [56] trains a DM by randomly masking past and/or future frames in blocks and conditioning on the remaining frames. It generates long videos by autoregressively sampling blocks of frames in a sliding window manner. PVDM [66] projects videos into low-dimensional latent space as 2D vectors, and presents a joint training of unconditional and frame conditional video generations. LFDM [36] employs a flow predictor to estimate latent flows between video frames and learns a DM for temporal latent flow generation. VideoFusion [32] decomposes the transition noise in DMs into per-frame noise and the noise along time axis, and trains two networks jointly to match the noise decomposition. While DMs have demonstrated impressive performance in video synthesis, its applications to precipitation nowcasting and other Earth science tasks have not been well explored. Hatanaka et al. [16] uses DMs to super-resolve coarse numerical predictions for solar forecast. Concurrent to our work, LDCast [30] applies LDMs for precipitation nowcasting. However, LDCast has not studied how to integrate prior knowledge to the DM, which is a unique advantage and novelty of PreDiff.

Conditional controls on diffusion models Another key advantage of DMs is the ability to condition generation on text, class labels, and other modalities for controllable and diverse output. For instance, ControlNet [68] enables fine-tuning a pretrained DM by freezing the base model and training a copy

end-to-end with conditional inputs. Composer [24] decomposes images into representative factors used as conditions to guide the generation. Beyond text and class labels, conditions in other modalities, including physical constraints, can also be leveraged to provide valuable guidance. TopDiff [33] constrains topology optimization using load, boundary conditions, and volume fraction. Physdiff [67] trains a physics-based motion projection module with reinforcement learning to project denoised motions in diffusion steps into physically plausible ones. Nonetheless, while conditional control has proven to be a powerful technique in various domains, its application in DL for precipitation nowcasting remains an unexplored area.

B Implementation Details

All experiments are conducted on machines with NVIDIA A10G GPUs (24GB memoery). All models, including PreDiff, knowledge alignment networks and the baselines, can fit in a single GPU without the need for gradient checkpointing or model parallelization.

B.1 PreDiff

Frame-wise autoencoder We follow [7, 42] to build frame-wise VAEs (not VQVAEs) and train them adversarially from scratch on N -body MNIST and SEVIR frames. As shown in Sec. 2.2, on N -body MNIST dataset, the spatial downsampling ratio is 4 × 4 . A frame x j ∈ R 64 × 64 × 1 is encoded to z j ∈ R 16 × 16 × 3 by parameterizing p E ( z j | x j ) = N ( µ E ( x j ) | σ E ( x j )) . On SEVIR dataset, the spatial downsampling ratio is 8 × 8 . A frame x j ∈ R 128 × 128 × 1 is encoded to z j ∈ R 16 × 16 × 4 similarly.

The detailed configurations of the encoder and decoder of the VAE on N -body MNIST are shown in Table 3 and Table 4. The detailed configurations of the encoder and decoder of the V AE on SEVIR are shown in Table 5 and Table 6. The discriminators for adversarial training on N -body MNIST and SEVIR datasets share the same configurations, which are shown in Table 7.

Latent diffusion model that instantiates p θ ( z t -1 | z t , z cond ) Stemming from Earthformer [8], we build Earthformer-UNet , which is a hierarchical UNet with self cuboid attention [8] layers as basic building blocks, as shown in Fig. 5.

On N -body MNIST, it takes the concatenation along the temporal dimension (the sequence length axis) of z cond ∈ R 10 × 16 × 16 × 3 and z t ∈ R 10 × 16 × 16 × 3 as input, and outputs z t -1 ∈ R 10 × 16 × 16 × 3 . On SEVIR, it takes the concatenation along the temporal dimension (the sequence length axis) of z cond ∈ R 7 × 16 × 16 × 4 and z t ∈ R 6 × 16 × 16 × 4 as input, and outputs z t -1 ∈ R 6 × 16 × 16 × 4 . Besides, we add the embedding of the denoising step t to the state in front of each cuboid attention block via an embeding layer TEmbed , following [22]. The detailed configurations of the Earthformer-UNet is described in Table 8.

Knowledge alignment networks A knowledge alignment network parameterizes U ϕ ( z t , t, y ) to predict F ( ̂ x, y ) using the noisy latent z t . In practice, we build an Earthformer encoder [8] with a final pooling block as the knowledge alignment network to parameterize U ϕ ( z t , t, z cond ) , which takes t , and the concatenation of z cond and z t , instead of t , y and z t as the inputs. We find this implementation accurate enough when t is small. The detailed configurations of the knowledge alignment network is described in Table 9

2D-CNN + Downsample

𝑧

!"#$

𝑧

%&'

Cuboid Attention Block

×𝐷

Downsample

Cuboid Attention Block

×𝐷

Upsample

Cuboid Attention Block

×𝐷

×𝑀

×𝑀

2D-CNN + Upsample

𝑧

%

res connect

not used

Optimization We train the frame-wise VAEs using the Adam optimizer [27] following [7]. We train the latent Earthformer-UNet and the knowledge alignment network using the AdamW optimizer [31] following [8]. Detailed configurations are shown in Table 10, Table 11 and Table 12 for the frame-wise VAE, the latent Earthformer-UNet and the knowledge alignment network, respectively. We adopt data parallel and gradient accumulation to use a larger total batch size while the GPU can only afford a smaller micro batch size.

B.2 Baselines

We train baseline algorithms following their officially released configurations and tune the learning rate, learning rate scheduler, working resolution, etc., to optimize their performance on each dataset. We list the modifications we applied to the baselines for each dataset in Table 13.

C Derivation of the Approximation to Knowledge Alignment Guidance

We derive the approximation to the knowledge alignment guided denoising transition (5) following [4]. We rewrite (5) to (8) using a normalization constant Z that normalizes Z ∫ e -λ F ∥ U ϕ ( z t ,t,y ) -F 0 ( y ) ∥ dz t = 1 :

In what follows, we abbreviate µ θ ( z t +1 , t, z cond ) as µ θ , and Σ θ ( z t +1 , t, z cond ) as Σ θ for brevity. We use C i , i = { 1 , . . . , 7 } to denote constants.

By assuming that log Ze -λ F ∥ U ϕ ( z t ,t,y ) -F 0 ( y ) ∥ has low curvature compared to Σ -1 θ , which is reasonable in the limit of infinite diffusion steps ( ∥ Σ θ ∥ → 0 ), we can approximate it by a Taylor expansion at z t = µ θ

where g = -λ F ∇ z t ∥ U ϕ ( z t , t, y ) -F 0 ( y ) ∥| z t = µ θ . By taking the log of (8) and applying the results from (9) and (10), we get

Therefore, the transition distribution under the guidance of knowledge alignment shown in (5) can be approximated by a Gaussian similar to the transition without knowledge guidance, but with its mean shifted by Σ θ .

D More Quantitative Results on SEVIR

D.1 Quantitative Analysis of BIAS on SEVIR

Similar to Critical Success Index ( CSI ) introduced in Sec. 3.2, BIAS = # Hits +# F . Alarms # Hits +# Misses is calculated by counting the # Hits (truth=1, pred=1), # Misses (truth=1, pred=0) and # F . Alarms (truth=0, pred=1) of the predictions binarized at thresholds [16 , 74 , 133 , 160 , 181 , 219] . This measurement assesses the model's inclination towards either F . Alarms or Misses .

The results from Table 14 demonstrate that deterministic spatiotemporal forecasting models, such as UNet [55], ConvLSTM [47], PredRNN [61], PhyDNet [11], E3D-LSTM [60], and Earthformer [8], tend to produce predictions with lower intensity. These models prioritize avoiding high-intensity predictions that have a higher chance of being incorrect due to their limited ability to handle such uncertainty effectively. On the other hand, probabilistic spatiotemporal forecasting baselines, including DGMR [41], VideoGPT [65] and LDM [42], demonstrate a more daring approach by predicting possible high-intensity signals, even if it results in lower CSI scores, as depicted in Table 2. Among these baselines, PreDiff achieves the best performance in BIAS . It consistently achieves BIAS scores closest to 1 , irrespective of the chosen threshold. These results demonstrate that PreDiff has effectively learned to unbiasedly capture the distribution of intensity.

D.2 CSI at Varying Thresholds on SEVIR

We include representative deterministic methods ConvLSTM and Earthformer, and all studied probabilistic methods to compare CSI , CSI , CSI -pool 4 and CSI -pool 16 at varying thresholds. It is important to note that CSI tends to favor conservative predictions, especially in situations with high levels of uncertainty. To ensure a fair comparison, we calculated the CSI scores by averaging the samples for each model, while scores in other metrics are averaged over the scores of each sample. The results presented in Table 15, 16, 17 demonstrate that our PreDiff achieves competitive CSI scores and outperforms baselines in CSI scores at pooling scale 4 × 4 and 16 × 16 , particularly at higher thresholds.

E More Qualitative Results on N -body MNIST

Fig. 6 to Fig. 13 show several sets of example predictions on the N -body MNIST test set. In each figure, visualizations from top to bottom are context sequence y , target sequence x , predictions by ConvLSTM [47], Earthformer [8], VideoGPT [65], LDM [42], PreDiff, PreDiff-KA. E.MSE denotes the average error between the total energy (the sum of kinetic energy and potential energy) of the predictions E ( ̂ x j ) and the total energy of the last step context E ( y L in ) .

D

5

5

5

5

5

5

5

94

5

5

5

5

5

5

5

54

90

0

0

r

00

5

5

9

5

5

5

5

rg

A

A/

C

78

15

5

5

5

5

5

5

5

5

20

5

5

5

5

5

5

5

5

54

5

4

A/

5

5

5

5

5

5A

19

step 1

step 2

step 3

step 4

step 5

step 6

step 7

step 8

step 9

step 10

5

2

05

05

05

052

052

25

2

50

50

50

5

205

L5

45

98

50

50

50

C5

05

er

41

2券

y

济

了

06

2

2

50

2

50

2

50

50

5

72

50

50

50

5

205

5

56

2

50

50

21

step 1

step 2

step 3

step 4

step 5

g das

step 7

step 8

step 9

step 10

D

4

28

0

1

1

06

0

0

r

89

60

Ch

Q4

09

了

01

40

4

4

22

0

D

。

08

0

ho

C

step 1

step 2

step 3

step 4

step 5

step 6

step 7

step 8

step 9

step 10

79

95

52

品

63

er

43

古

9

6

02

G

？

B

自

7

76

39

#

9

6

54

6

step 2

step 3

step 4

step 5

step 6

step 7

step 9

step 1

step 8

step 10

4

h

4

4

y

3

3

3

3

3

5

3

53

54

3ys

5

5

5

60

3y

5

5

3

5

4

3

5

4

3

5

4

3

5

y

3

5

3

5

4

3

3

47

G

3

3

3

3

3

3

er

3

34

5

3

5

3

3

5

3

5

3

5

3

5

3

5

3

57

4

y

h

4

h

y

h

i4

86

3

3

3

3

3

3

3

3

5

3

f

3

3y

5

3

5

3

5

3

5

3

5

3

5

3

5

3

5

3

5

74

4

4

4

y

3

5

34

5

3

5

3

5

3

5

3

5

3

S

3

5

3

5

3

5

54

4

h

4

y

4

4

34

3

5

34

5

3

4

5

3

5

h

3

5

h

3

5

3

5

3

5

3

5

3

5

step 1

step 2

step 3

step 4

step 5

step 6

step 7

step 8

step 9

step 10

3

3

3

4

4

3

4

3

4

31

4

31

4

13

187

4

4

3

力

143

43

T3

4

45

r

04

4

14

43

4B

3

力

3

05

多

43

34

3

98

13

3

3

02

4

13

3

4

4

3

43

143

B

13

4

143

4B

12

13

3

step 2

step 6

step 9

step 1

step 3

step 4

step 5

step 7

step 8

step 10

5

9

P

7°

2°

10

10

1

。

10

70

70

5

6

中

9

01

。

84

70

5

5

5

5

b

51

70

er

99

70

5

9

01

LO

o

9

07

97

70

5

o

07

07

0

62

70

S

5

5

5

中

9

3

01

LO

01

70

89

5

5

中

01

07

01

40

70

5

step 1

step 2

step 3

step 4

step 5

step 6

step 7

step 8

step 9

step 10

2多

AC

22

29

2

12

23

er

25

27

7

47

46

26

2

22

05

37

step 1

step 7

step 8

step 2

step 3

step 4

step 5

step 6

step 9

step 10

F More Qualitative Results on SEVIR

Fig. 14 to Fig. 19 show several sets of example predictions on the SEVIR test set. In subfigure (a) of each figure, visualizations from top to bottom are context sequence y , target sequence x , predictions by ConvLSTM [47], Earthformer [8], VideoGPT [65], LDM [42], PreDiff, PreDiff-KA. In subfigure (b) of each figure, visualizations from top to bottom are context sequence y , target sequence x , predictions by PreDiff-KA with anticipated average future intensity µ τ + nσ τ , n = 4 , 2 , 0 -2 , -4 .

(a)

(b)

-50 Minutes

-40 Minutes

-30 Minutes

-20 Minutes

-10 Minutes

0 Minutes

x

et

TM

968

0.698

421

0.647

PT

354

0.655

299

0.638

ff

386

0.708

10 Minutes

20 Minutes

30 Minutes

40 Minutes

50 Minutes

60 Minutes

-50 Minutes

-40 Minutes

-30 Minutes

-20 Minutes

-10 Minutes

0Minutes

xt

et

0.071

ma

0.098

ma

0.097

iff

0.075

na

0.060

na

0.048

10 Minutes

20 Minutes

30 Minutes

40 Minutes

50 Minutes

60 Minutes

(a)

(b)

-50Minutes

40 Minutes

-30Minutes

-20 Minutes

-10 Minutes

0Minutes

xt

et

Avg

TM

376

0.669

Avg

367

0.563

Avg

SPT

293

0.592

Avg

298

0.578

Avg

iff

363

0.703

Avg

10 Minutes

20 Minutes

30 Minutes

40 Minutes

50 Minutes

60 Minutes

-50Minutes

-40 Minutes

-30Minutes

-20 Minutes

-10 Minutes

0 Minutes

xt

ot

0.105

0.161

S

0.153

ff

0.108

na

0.081

9.

na

0.060

10 Minutes

20 Minutes

30 Minutes

40 Minutes

50 Minutes

60 Minutes

(a)

(b)

-50 Minutes

-40 Minutes

-30 Minutes

-20 Minutes

-10 Minutes

0Minutes

xt

AV

TM

452

0.662

Avg

464

0.614

Avg

PT

428

0.712

Avg

399

0.654

Avg

iff

444

0.763

Avg

10 Minutes

20 Minutes

30 Minutes

40 Minutes

50 Minutes

60 Minutes

-50 Minutes

-40 Minutes

-30 Minutes

-20 Minutes

-10 Minutes

0Minutes

xt

et

0.205

0.353

ff

0.213

na

0.144

na

10 Minutes

20 Minutes

30 Minutes

40 Minutes

50 Minutes

60 Minutes

(a)

(b)

-50 Minutes

-40 Minutes

-30 Minutes

-20 Minutes

-10 Minutes

0 Minutes

xt

et

Avg

TM

316

0.462

Avg

352

0.436

Avg

SPT

265

0.456

Avg

232

0.441

Avg

iff

292

0.518

Avg

10 Minutes

20 Minutes

40 Minutes

50 Minutes

60 Minutes

-50 Minutes

-40 Minutes

-30 Minutes

-20 Minutes

-10 Minutes

0 Minutes

xt

et

0.079

0.103

na

0.098

ff

0.077

na

0.062

na

0.049

10 Minutes

20 Minutes

30 Minutes

40 Minutes

50 Minutes

60 Minutes

(a)

(b)

-50 Minutes

-40 Minutes

-30 Minutes

-20 Minutes

-10 Minutes

0Minutes

xt

Avg

TM

469

0.605

Avg

515

0.563

Avg

PT

419

0.599

Avg

345

0.601

Avg

451

0.669

Avg

10 Minutes

20 Minutes

30 Minutes

40 Minutes

50 Minutes

60 Minutes

-50 Minutes

-40 Minutes

-30Minutes

-20 Minutes

-10 Minutes

0Minutes

xt

t

0.137

0.211

0.197

f

0.147

na

0.117

na

0.088

10 Minutes

20 Minutes

30 Minutes

40 Minutes

50 Minutes

60 Minutes

(a)

(b)

-50 Minutes

-40 Minutes

-30 Minutes

-20 Minutes

-10 Minutes

0 Minutes

xt

Avg

TM

245

0.335

Avg

243

0.312

Avg

SPT

219

0.380

Avg

208

0.389

Avg

iff

223

0.404

Avg

10 Minutes

20 Minutes

30 Minutes

40 Minutes

50 Minutes

60 Minutes

-50 Minutes

-40 Minutes

-30 Minutes

-20 Minutes

-10 Minutes

0 Minutes

xt

et

0.105

0.161

0.149

ff

0.104

na

0.078

na

0.057

10 Minutes

20 Minutes

30 Minutes

40 Minutes

50 Minutes

60 Minutes

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the PreDiff for precipitation nowcasting. PreDiff adopts the latent diffusion for prediction. Besides, a special knowledge control design is used in a classifier guidance way to incorporate auxiliary prior knowledge, which can guide the model to satisfy conservation constrain. In both synthetic and real-world benchmarks, PreDiff achieves a good performance.\n\nSTRENGTHS:\n1. This paper presents a practical method for precipitation nowcasting. Using diffusion model can benefit the prediction skill.\n\n2. The proposed knowledge control is reasonable and effective.\n\n3. This paper is well-organized.\n\nWEAKNESSES:\n1. The major concern is about novelty.\n\nThis paper introduces the famous diffusion model techniques into precipitation nowcasting task, including the condition diffusion, latent diffusion model and classifier guidance. \n\nThe key design is the knowledge control. But accoding to the paper, the knowledge control requires maual designed for every specific task. And there does not exist evidence that anticipated precipitation intensity is a right prior knowledge for precipitation nowcasting. More explanation are expected.\n\n2. More comparing baselines are insufficient.\n\nThere are some advanced deep models for precipitation nowcasting that the authors should take into consideration, especially the second baseline that is based on the GAN.\n\n[1] MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions, CVPR 2021\n\n[2] Skilful precipitation nowcasting using deep generative models of radar, Nature 2021\n\n3. Model efficiency comparisons are also required, including the GPU memory and running time.\n\nQUESTIONS:\n(1) In precipitation nowcasting, how to decide the hyper-parameter n?\n\n(2) It is interesting to see the results in Figure 3 (b). I want to know the CSI, Bias and FVD metrics for the $4\\sigma$ and $-4\\sigma$ models.\n\nLIMITATIONS:\nThey have discussed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper applies latent diffusion models to the problem of precipitation nowcasting, developing a modular and flexible method for enforcing external constraints during the generating process. Constraints are general in that they can be specified in terms of any function F from the data to the reals, F:(x,y)->R. In the proposed \"Knowledge Control\" (KC) method, a separate NN is trained to predict F from the noised latent data. Conveniently this can be trained before training the diffusion model; it is then used while training the diffusion model as a regularizer, punishing the diffusion model for generating samples that violate the constraints. \n\nExperiments are performed on a simulation dataset (MovingMNIST) to demonstrate the advantage of the modeling choices and the KC method, and compare diffusion models with other deep learning models. The KC method is used to enforce conservation of energy, and the experiments demonstrate that KC is effective at constraining the generated samples without interfering with the other performance metrics.\n\nThe model is then applied to the SEVIR precipitation dataset and compared against a number of other deep learning models. Experiments demonstrate the use of the KC method in controlling the overall amount of rainfall.\n\nSTRENGTHS:\n- The knowledge control method will be of high interest to those applying diffusion models to scientific applications, where there are often physical constraints.\n- Experiments are well done and will be valuable to researchers in this area. Experiments are done on both simulated and real data, with multiple performance metrics, and comparisons to strong baselines. \n- The paper is well-written.\n\nWEAKNESSES:\nNone.\n\nQUESTIONS:\n- In experiments, deterministic models are compared against probabilistic models. For the probabilistic models, a single sample was generated from the probabilistic distribution in each forecast. The probabilistic models perform worse than the deterministic models on some metrics. Isn't it likely that the performance of the probabilistic models would be improved if multiple samples were drawn from them?\n\nLIMITATIONS:\nThe authors concisely and fairly address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new method for doing precipitation nowcasting. The method is based on building a conditional diffusion model in a global latent space, which is found separately in an autoencoding fashion. Additionally, a “knowledge control” mechanism is introduced to help ensure that the generated samples satisfy known physical constraints. Numerical validations are conducted on a synthetic and a real precipitation dataset.\n\nSTRENGTHS:\n* The problem of precipitation nowcasting is highly significant and has a lot of potential for real-life impact.\n* The paper is generally well written. I find it easy to understand. Tables and figures are informative.\n* Lots of details for reproducibility.\n\nWEAKNESSES:\n* The model itself is not the most novel. Both latent diffusion and knowledge control (similar to classifier guidance) are not new. This is somewhat compensated by the fact that the model is being applied to a specific and impactful domain. \n* Despite the seemingly rich set of baselines being compared, some of the most recent SOTA models developed specifically for the precipitation nowcasting problem are missing (despite being mentioned in the paper): Metnet2 (ref 5), DGMR (ref 28) - these models were applied to real weather data, but one should be able to adapt them to the 2nd benchmark dataset considered here. It is important to know where the model stands compared to these, especially considering that the methodology alone does not qualify for accept due to lack of novelty.\n* The use of the FVD metric is questionable (see questions below).\n* Value of knowledge control on the precipitation dataset is unclear.\n* A few other places that need further clarifications (see questions section).\n\nI feel in its current state these outweigh the strengths so I vote for borderline reject.\n\nQUESTIONS:\n* Ln 47 - “AR models” refer to autoregressive models? I didn’t see this acronym defined \n* Equation (1) - what is $p_\\mathcal{E}$ exactly? I presume $y$ gets encoded through $\\mathcal{E}$ and how do you draw $z_\\text{cond}$?\n* Ln 196: “.. by comparing the statistics of feature vectors extracted from real data and generations” - what is this real data? The references cited on ln 201 (in audio and chemical applications) both use embeddings trained on their respective domain-specific datasets. It is meaningless to compare against references not related to the specific tasks since the feature distributions will be completely different.\n*  Ln 209-210: “.. generate predictions with sharp and clear digits” - this is too handwavy to me. Although the statement looks qualitatively true, a quantitative measure would make it much more rigorous. A potential option is to look at Fourier spectrum  -‘blurry’ images tend to show underestimated energy in high frequency modes.\n* Ln 299-300: “.. highlighting possible extreme cases like rainstorms and droughts” - this is not easy to verify by looking at Figure 3 for someone without domain knowledge. Additional labels or captions would help.\n* I do not fully understand the purpose of knowledge control discussed in section 3.2.2. Is it only meant to encourage the generation of extreme scenarios? Do you have ways to evaluate them besides FVD?\n\nLIMITATIONS:\nThe authors mention limitations related to potential misalignment between visual quality and scores, as well as dependence on data quality and quantity.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This work proposes a latent diffusion model called PreDiff for spatiotemporal forecasting. The main contributions of this paper are two-fold: 1) they use a conditional latent diffusion model to handle the uncertainty of spatiotemporal forecasting. 2) They introduce an explicit knowledge control mechanism to align the model with domain-specific prior knowledge.

STRENGTHS:
1. This paper is well-written and easy to follow.
2. The design of the knowledge control mechanism is interesting. It is decoupled with the latent diffusion model, thus providing more flexibility to the whole framework.

WEAKNESSES:
1. My main concern is the experiments. This paper does not conduct enough ablation studies, and the effectiveness of the diffusion model cannot be supported by experiments shown in Tables 1 & 2.
2. Table 2 only reports the mean of CSI at different thresholds, which is insufficient to prove the superiority of the proposed method. CSI with a high threshold correlates with strong weather, thus, it is important in this work to prove that PreDiff can generate better forecasts, which is also one of the main claims of this paper. I suggest authors list CSI at different thresholds in Table 2 to make a comprehensive comparison.
3. The parameter number of PreDiff is significantly higher than that of EarthFormer (about 15 times). Thus the comparison in Tables 1 & 2 is unfair. So is it possible to reduce the number of parameters to make a fair comparison with other methods?

QUESTIONS:
Please see the weakness.

LIMITATIONS:
Please see the weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-08