STLDM: Spatio-Temporal Latent Diffusion Model for Precipitation Nowcasting
Authors: Shi Quan Foo
Chi-Ho Wong
Zhihan Gao
Dit-Yan Yeung
Ka-Hing Wong
Wai-Kin Wong
Venue: Accepted by TMLR ()

Abstract: Precipitation nowcasting is a critical spatio-temporal prediction task for society to prevent severe damage owing to extreme weather events. Despite the advances in this field, the complex and stochastic nature of this task still poses challenges to existing approaches. Specifically, deterministic models tend to produce blurry predictions while generative models often struggle with poor accuracy. In this paper, we present a simple yet effective model architecture termed STLDM, a diffusion-based model that learns the latent representation from end to end alongside both the Variational Autoencoder and the conditioning network. STLDM decomposes this task into two stages: a deterministic forecasting stage handled by the conditioning network, and an enhancement stage performed by the latent diffusion model. Experimental results on multiple radar datasets demonstrate that STLDM achieves superior performance compared to the state of the art, while also improving inference efficiency. The code is available in https://github.com/sqfoo/stldm_official.

Published in Transactions on Machine Learning Research (12/2025)
STLDM: Spatio-Temporal Latent Diffusion Model for Pre- cipitation Nowcasting
Shi Quan Foo
sgfoo @coomeet..uu......
The Hong Kong University of Science and Technology
Chi-Ho Wong
chuonggee@torrneec.uuu..
The Hong Kong Universiby of Science and Technology
Zhihan Gao
shihan. @connect.ust..
The Hong Kong University of Science and Technology
Dan Yan Yeung
dyyeung@cse.ust. ust.kk
The Hong Kong University of Science and Technology
Ka-Hing Wong
khwong@hko.gon
Hong Kong Observatory
Wai-Kin Wong
Setwon@@hko.go..h
Hong Kong Obsereatory
Reviewed on OpenReview: https:// o/teervve/pnrevveeu. net/foruse id- f40Juxn3gg
Abstract
Precipitation nowcasting critical spatio-temporal prediction task for society to prevent severe damage owing extreme weather events. Despite the advances in this field, the complex and stochastic nature of this task still poses challenges to existing approaches. Specifically, deterministic models tend to produce blurry predictions while generative mod- els often struggle with poor accuracy. In this paper, we present simple yet effective model architecture termed STLDM, a diffusion- model that learns the latent representation from end to end alongside both the Variational Autoencoder and the conditioning network. STLDM decomposes this task into two stages: deterministic forecasting stage handled by the conditioning network, and an enhancement stage performed by the latent diffusion model. Experimental results on multiple radar dalasets demonstrate that STLDM achieves superior performance compared to the state of the art, while also improving inference effi- httt:::/ggthubb..mm/ssfff//// cttp:/ffo////oooooooo///ooooooco sttss__mdffffieeeeeeeeeeeeeteeet
ciency. The code is available in https://gitthnn
1 Introduction
Precipitation nowcasting short-term prediction task for precipitation events over a specific region, based on weather data such as radar and satellite observations. Accurate and timely nowcasting is crucial sociely, it enables impacts due to extreme weather. Traditionally, meteorologists utilized algorithmic methods such as optical- flow methods (Pulkkinen et al. 2019) and the guidance from numerical weather prediction (NWP) models on this nowcasting task.
lake preventive actions for mitigating potential economic loss and other adverse
With the emergence of deep learning, data-driven models have been extensively explored for modeling the spatio-temporal patterns of precipitation events. Despite their lack of interpretability, these doep learning
approaches often outperform traditional methods in terms of accuracy and efficiency. Broadly, these ap-
proaches be categorized into two main research categories: video prediction (Shi et al. 2015
2017
Gao

---

Published in Transactions on Machine Learning Research (12/2025)
al. 2022b), which models the 4D spatio-temporal trend with ground truth observation for performance
evaluation; and video generation (Zhang et al. 2023 Leinonen et al. 2023 Gao et al. 2023 which adopts
generative models to synthesize the target data distributions with less consideration to the alignment to the ground truth and more emphasis on the visual fidelity.
Recent works highlight the challenges posed by the stochastic nature of precipitation nowcasting due to the inherent unpredictability of open systems. De- terministic models, in particular, tend to capture the global motion trend well with the missing of details at the micro-level, resulting in blurry pre- dictions over longer lead times. This leads to the difficulty practical forecasting operations (Ravuri et al. 2021). On the other hand, generative mod- els capable of modeling micro-level weather phe- nomena through simulating the data distribution, which tolcrates the nature of stochasticity, thereby producing realistic and sharp forecasts (Zhang et al. 2023 Leinonen et al. 2023 Gao et al. 2023). How- ever, they often suffer from low accuracy in predict- ing large-scale weather events.
Grvund Trush
Deterninistic
Generativve
Ours
30 maiz
00
summary, both these approaches could only achieve either high ac- curacy (deterministic models) high visual quality (generative models) as shown in Figure 1
Figure This demonstrates that deterministic models result in blurry predictions while generative models suffer from the issuc of inaccurate predictions. Our proposed STLDM is capable of forecasting accurate predictions while maintaining nice appearance.
In this paper, we first re- re-formulate the precipitation nowcasting task into two sequential subtasks: Forecasting and Enhancement based on the observations above. First, a Translator with the same archilecture as deterministic models is implemented to perform the forecasting task, i.e. roughly forecast the upcoming precipitation events, Y. The objective here is capture the approximate global motion trend of the future. Then, diffusion model used to refine the first estimation by the Translator, Y, by introducing it as the conditional variable such that the global motion trend of generated samples is constrained well. To accelerate the inference process, employ this sampling process in the latent space.
Therefore, we propose and present a novel and simple for these re-formulated objectives, effectively handling the stochasticity in precipitation nowcasting. STLDM consists of three modules: a Variational AutoEn,,der, a Translator (aka Conditioning Network), and Latent Denoising Network. Furthermore, train STLDM in manner from end to end to encourage cooperation among all modules. The experimental results show that STLDM is capable of achieving the state-of-the-arr performance on both pixel accuracy and visual fidelity, outperforming most diffusion-based models. The contributions of this work are summarized as follows:
Latent Diffusion Model - STLDM
We re-formulate the precipitation nowcasting task into: Forecasting and Enhancement sequence. Based on this, we propose STLDM, simple yet efficient model utilizing the Latent Diffusion Model. To the best of our knowledge, this is the first. work that trains a Latent Denoising Network alongside Variational AutoEncoder and Conditioning Network for the precipitation nowcasting task.
Our STLDM achieves state-of-the-ate performance multiple real-life radar echo datasets across most ovaluation metrics while offering faster sampling speeds compared to other diffusion- based models.
2

---

Published in Transactions on Machine Learning Research (12/2025)
2 Related Works
2.1 Precipitation Nowcasting as a Spatio-Tena Temporal Task
Precipitation nowcasting is commonly interpreted as spatio-temporal predictive task to predict the next N output frames, YI:N, given M imput frames, XI:M- It is formulated as:
arg max (( I..,. YN X1,.., XM)
(1)
E1.....N
Based on this formulation, various deep learning models that leverage spatio-temporal features have been proposed. ConvLSTM (Shi et al. 2015), which integrales convolution layers into LSTM cells in an encoder- forecaster architecture, is the first method proposed for this task. Later, PredRNN (Wang et al., 2017) introduced a novel structure, the ST-LSTM unit, to extract spatio-temporal features and model futuure frames in zigzag memory flow. Building on this foundation, several modifications have been proposed, including Memory In Memory (MIM) (Wang et al. 2019), the gradient highway (Wang et. al. 2018 and reversed scheduled sampling [Wang et al. 2023).
Following the success of Transformers and their attention mechanisms in natural language processing (NLP) tasks (Vaswani et al. 2017) and vision processing tasks (Dosovitskiy et al., 2021b, several Transformer-based models have been deployed to capture the long-term spatio-temporal features of this nowcasting task. For instance, Cuboid attention in Earthformer (Gao et al. 2022b) and Feature Extraction Balance Module in Rainformer (Bai et al.
2022 are proposed to extract and model both the global and local rainfall features.
In paraliel, CNN-based models have also been explored for spatio-temporal modeling, like SimVP (Gao et al. 2022a) and TAU (Tan et al.j 2023a). Both works adopt. a U-Net-like structure, i.e., Encoder-Translaar Decoder with Spatial Encoder and Decoder. SimVP and TAU introduced Inception modules and Temporal Attention Unit as translators, respectively, to learn the temporal evolution. These works are primarily composed of convolution operations, achieving efficiency and remarkable performance.
However, neither of these works can produce sharp predictions for long lead times due to the stochastic nature of this task. To address this issue, several loss functions were proposed alternatives to the conventional L2 loss, such as SSL (Chen et al., 2020) and FACL (Yan et al. 2024). Meanwhile, probabilistic models have been employed to capture the spatio-temporal features by estimating the conditional distribution of the future frames, thereby enabling ensemble predictions with high visual fidelity, such as GANs (Ravuri ct al. 2021 Chang al.] 2022; Zhang al., (203)) and variational autoencoders (VAEs) (Denton & Fergus, 2018
Franceschi al.] 2020). However, these models are often criticized for their training instability and
potential for mode collapse.
2.2
Diffusion-basser Models
Diffusion models (DMs) (Ho al. 2020) have achieved high visual fidelity in image generation (Saharia
et al. 2022 Ramesh et al. 2022) and video generation (Ho et al. 2022 Yang et al. 2023 Voleti et al. 2022
and are increasingly being applied in atmospheric forecasting Price et al. 2023 and downscaling Mardani
et al. 2023 Wan et al.) 2024). Unlike GANs, DMs, which are optimized with a likelihood objective (Kingma & Gao 2023), do not suffer from mode collapse. However, they are computationally expensive and have
longer inference times. Several techniques have been proposed to accelerate the sampling process, such as
DDIM (Song et al. 2021 and progressive distillation (Salimans & Ho. 2022
Applying the denoising process in the latent space, Latent Diffussin Model (LDM), also reduces the heavy demand of computational resources. LSGM (Vahdat et al. 2021), the first LDM with end-to-end training scheme for both VAE and LDM, achieved state-of-thc-art performance in image generation tasks and provided a solid theoretical analysis. Later, follow-up works (Rombach et al.] 2022) decomposed this framework into a two-stage process: pre-training a VAE followed by the training of LDM, showcasing its effectiveness in conditional generation tasks with lower computational demands.
LDCast (Liinonen et al. 2023) employs Adaptive Fourier Neural Operators (AFNOs) in conditional LDM to predict. the evolution of radar echo via the denoising process. Similarly, PreDiff (Goo et al. 2023
3

---

Published in Transactions on Machine Learning Research (12/2025)
incorporates prior knowledge through knovledge-control network to align the output with prior knowledge, while replacing the U-Net-style architecture with Farthformer to capture complex spatio-temporal features. These works adopt the two-stage training framework described above.
Although these diffusion- based models could produce predictions with high visual quality, they often suffer from the issue of low accuracy. To address this, recent works have attempted achieve both high visual quality and high accuracy via the cooperation between deterministic models and DMs. DiffCast (Yu et al, 2024) treats this nowcasting task as combination of trend prediction (handled by deterministic models) and local stochastic variation (handled by DMs); while CasCast (Gong et al. 2024 introduces cascaded modeling approach that combines deterministic models with Casformer.
In summary, Dif[Cast runs the denoising process in the pixel space, which leads to a longer inference time. Conversely, LDCast, PreDiff, and CasCast deploy DMs in the latent space with a pre-trained VAE. This approach is proven be computationally efficient, but the independence to train every module limits the model'ss generative capability, as every module trained with the corresponding objectives. End-lo-end training of all components may further improve performance.
3 Methodology
In this section, we first provide the background of diffusion models. Then, we revisit and reformulate the objective of the precipitation nowcasting task. Lastly, propose and present a Spatio-Temppral Latent Dilfusion Model - STLDM in detail, including its training loss function and each component in STLDM.
3.1
Preliminary
3.1.1
Diffusion Models
Diffusson Models (DMs) Ho et al. 2020g learn the data distribution, p(z), by modeling the reverse diffusion process from Gaussian noise, x² using corrupted samples, x², with their corresponding diffusion step, t. T, is defined as:
The forward diffusion process, which gradually adds noise from
|xr') - - /1
(2)
where 3 E (0, increases monotonically over
The reverse diffusion procoss iteratively removes noise, starting from pure Gaussian noise, x², and is formu- lated as:
pa pa(z-~1ax") = NN(===t Ho(o,, t), of
(3)
with the paramcterized postcrior mean function, μ4 defined as:
Br co t)), at
pola" .) Vas (x"
(4)
-I-
where = 1 - 3,, a, = -T a, and fo(x", t) is trainable denoising function that estimates the noise at the diffuuson step, 4. The network parameters, 0, are optimized by minimizing the diffusion loss, Ldiffusion: as formulated as:
(((t)lc- colVāuo + VI oic,t)]1?
Ldiffusion
(5)
where 7t) is the weighting function.
3.1.2 Classifier-Fres Free Guidance
To balance the trade-off between sample quality and diversity during generation, guidance introduced during sampling to ensure that the sample generated by the diffusion model is constrained with the given
4

---

Published in Transactions on Machine Learning Research (12/2025)
conditional variables, C, such class labels and text prompts. Inspired by GANs, Classifier Guidance (Dhariwal & Nichol 2021] is initially implemented by incorporating the gradient of trained classifier, & into the diffusion score during sampling as follows:
Fg.0(1 c) = co( + wv ār a,V,,,o((aa))
(6)
which modifies the sampling distribution to: p(xx(c) lc)(ccaa)) with the guidance strength, w. According to the Bayes' Theorem, we could rewrite p(cla.) x p(z,lc)/p(zz) thereby the sampling distribution With this intuition, Classifier-Free Guid- ance (CFG) (Ho & Salimans| 2022 was proposed as alternative without an external classifier. Tnstead, 'implicit classifier' is achieved by jointly training conditional diffusion model, p(zale), and uncondi- tional diffusion model, p(z,). CFG is achieved by modifying the diffusion score, Eø(x", c), for every sampling timestep, t, as:
could be further modified as: p(r+(c) x p(z,. (c) [=(p(=)p((aacc)))
Ep EØ c) = es (a(" - wlea(x', 3) ca()"
where the null sign indicates the unconditional case.
3.2 Proposed Approach and Details
In this part, first reformulate the Precipitation Nowcasting task defined in Equation 1 and derive the corresponding loss function for the proposed STLDM. Later, describe each component of the proposed STLDM in detail.
3.2.1 Task Reformulation
With introducing the intermediate variables, VI,N, the conditional probability, p( P(Pi: (X1:), in Equation could be rewritten
/ p( P(
i,,,,,,N,,,,,. XuMM)d I:N;
(8)
where Vi.N and Y I:N represent the decoded radar frames predicted by the Latent Denoising Network and Translator, respectively, with the given input radar frames, XI:M- The details can be found in Appendix B
The first term in Equation corresponds to the Forecasting task of the first estimation, YIN, with the given input radar frames, X XI:M; while the second term represents the Visual Enhancement task conditioned on both the first estimation, VI.N, and input radar frames, XI:M- This motivates us to reformulate the precipitation nowcasting task into two sequential sub-tasks: Forecasting and Enhancement.
3.2.2 Derivation of Loss Function
Here, we derive the corresponding loss function of the proposed Soatio-Tempory Latent Diffusion Model (STLDM) as illustrated in Figure 2 Briefly speaking, the input radar frames, XIM, is first encoded by Spatial Encoder, E, then Translator, Ve does the first estimation, YIN,, and the Latent Denoising Network, Do, further enhance its visual quality and obtain the final prediction, YENN
We start with the conditional hierarchical VAE loss function over L variational layers (Vahdat & Kautz b020
=enn((/)/,,==))llallll LELBO = [-tt)t)]- log + DRr. (c(xl))))))) 1=2
(9)
We interpret STLDM as a 3-layer VAE and derive its corresponding loss function, LELBO, by substituting:
21 < Zz: 22 f 21,N, , 23 + 2T and 24 t 21,4, where 2g is the latent representation of the radar frames and -T'
-1:N is the pure Gaussian Noise with mean of 0 and standard deviation of 1.
5

---

Published in Transactions on Machine Learning Research (12/2025)
3
4
D
X1:M
Y
Estimation, Ay
E: Encoder
Decoder
Dg
D
Va: Translator
Dg: LDM
2T.
Shared parameter
Conditioning
11:N
Reckook
Redilocs
Recklock
Allent
Alsaniinn
Aller Lue
Temsoral Anient
Temporri leventio
Temperal Atte ritention
zi.v
Dg
Di Dowsompping
Upcarrpiing (ickk(2))
10 Miiliiii (499))
Blok
Figure 2: Top: Model architecture of the proposed STLDM, consisting of a Variational Autoencoder, ([E,D)), a Conditioning Network (aka Translator), Ve, and a Temiorll Latent Denoising Network, Dg. Input radar frames are denoted as XI.M; while the decoded outputs of both the final prediction after denoising from pure Gaussian Noise, 2T and the first estimation, 81N,, are denoted as YL,N and Y respectively. Bottom: Overall architecture of Do (Yellow Box) and the details of its sub-modules (Green Box). 'L-Spatial Attention' stands for Linearized Spatial Attention.
LELBO (ogp(al)))))(((..)))
(A)
+ (K((((Z))))ll(=z))
(B)
+
(22)
(C)
+ P ZLw Fnw) (lp() FI:N))
(D)
DEL ( Fi.n *I (lw SI,N, =T.N)
(E)
Term A corresponds to a typical reconstruction loss term, MMEE to minimize the difference between the reconstructed radar frames and the ground truth. As mentioned above, the global motion of the prediction from the deterministic model has a good aligmment with the ground truth. To ensure that the final prediction, YI,,, has the same global motion alignment, we propose constraint loss term, Lc, for regulating the first estimation, 3i,, which also acts the conditioning vector the denoising network, Dø- Hence, this loss term consists of two terms: typical Mean Squared Error loss for VAE and constraint loss term Translator (aka Conditioning Network), Vo.
E Eg [x) log p(x[2z)) = || -
+|| - 11.
(10)
EMSE
Cc
where X1:M4N is the concatenation of both the input radar frames and the ground truth, and Y:N is the ground truth itself, while any notations with" mean the prediction from STLDM.

---

Published in Transactions on Machine Learning Research (12/2025)
Term B represents regular KL-divergence loss term for constraining the distribution of the encoded latent
representation by the encoder, &, (Noo, oa); towards a Standard Gaussian distribution, NN,,, 1):
1 1 + + log ae]: =
(11)
while Term C is also a representation, ( has similar pattern as a Standard Gaussian distribution, N/ 1):
loss term for regularizing the distribution of the first estimated latent
(1 = - Jooi + +
Logōn..]. (12)
The loss term presented in Term D is represented as Prior Loss that ensures the disrupted latent represen- tation, =i.N, follows Equation 2 In both the formulation of DDPM and LDM with pre-trained VAE, this loss term would be dropped as it is irrelevant to the denoising network itself. However, this matters for our proposed STLDM with the end-to-end tuning framework, and it defined as:
DKl(a(=f.N xr 2z, & [Ip(=f/NN ZnN)) = DKL(W (1 (o-r)))))
(13)
where R is the encoded latent representation of the ground truth radar frames, YI-N-
Following the previous work (Kingma & Gao 2023), the last loss term stated in Equation 5 could be further defined and expressed as general diffusion loss function for the latent denoising network:
DKL(g(Z1:N J,, 2z, SI,, Ilp)) &i., &.,,, = x(t]le ca co(VaizLN + age t)l
(14)
In summary, the derived loss function for our proposed STLDM consists of VAE reconstruction loss, KL- divergence regularization loss, Conditioning regularization loss, Prior loss, and a diffusion loss.
3.2.3 Components of STLDM
Spatio Temporal Latent Diffusion Model (STLDM) consists of three main components: a Variational Au- toEncoder, D), a Conditioning Network (aka Translator), Ve, and a Latent Denoising Network, Da. Its details are illustrated in Figure - Model performance is further improved with the technique of CFG. Since CFG involves both the conditional case (e = f and the unconditional case (c =c= Φ) during the inference, we jointly train the latent denoising network, Do, on both cases with the provided algorithm in (Ho & Salimans 2022..
Variational AutoEncoder, {E, D), learns the spatial mappings between the radar images, X,, and their corresponding latent variables, Rt. Specifically, the encoder, E, transforms the input radar image, X,, from pixel space to the latent space; while the decoder, D, reconstructs the predicted latent variables back from the encoded latent space to the pixel space. We denote the encoded radar image in the latent space as 24 ~ E(X,) and its reconstructed radar image be X, ~ D(z))
Va, learns the temporal evolution from the input encoded input frames, XI:M, the target output frames, YI.N, in the latent space. Following the success of SimVP-V2, we employ its Gated Spatio-Temporey Attention (gSTA) module TTan et al. 2023b), which relies solely convolution operations V model the underlying relation between XI:M and YINN in the latent space. We denote this prediction a first estimation, ΣLM, and its decoded output as the first estimation, Y YIN--
Conditioning Network/Translatee
Latent Denoising Network, Do, is a conditional latent diffusion model that generates probabilistic predictions with conditioning on the latent prediction by Vg. To effectively capture the spatio-temporal features, we decouple the apatio-temporal attention mechanism in Dg into spatial attention and temporal attention modules. To optimize the computation, a lincar variant of spatial attention (Katharopoulos et al. 2020) which reduces the complexity from O(N2d) to O(Nd) is implemented for every Downsampling and T Upsampling blocks, where N and are the number of sequences and their projected dimension respectively.
7

---

Published in Transactions on Machine Learning Research (12/2025)
Briefly speaking, the difference between this linearized variant and standard attention is the operation order of the query, Q, key, K; and value, V, as shown below:
Astandard - o(V); ALinear - =(0)(((()(()))
(15)
STLDM trained from end to end with the objective defined in Scction 3.2.2
4 Experiments
4.1
Experimental Settings
We evaluate the performance and effectiveness of our proposed STLDM with several deterministic models serving as baselines, together with various diffusion- based models designed for precipitation nowcasting: LD- Cast (Leinonen et al.] 2023), PreDiff (Gao al., 2023), and DiffCast (Yu et al. 2024), on three real-life radar datasets: SEVIR (Veillette et al. 2020), HKO-7 (Shi et al. 2015), and MeteoNet (Larvor et al. 2020)
To alleviate the computation cost, downscale the spatial resolution to 128 x 128 while preserving their temporal dimension.
4.1.1 Dataset
SEVIR. (Veillette et al., 2020) is a curated and spatial-temporally aligned dataset that captures the weather events consisting of five different modalities in the US from 2017 to 2019. Each event consists of an image sequence spanning four hours with time interval of 5 minutes, covering the region with the geographical size of 384km x 384km in the US. The data range of the frames is set to [0 255]. Following previous work (Gao et al. 2022b), we specifically select the Vertically Integrated Liquid (VIL) channel and formulate the lask as predicting the next 12 frames (60 minutes) given 13 input frames (65 minutes). We span the data collected from June to December 2019 as the test set, while the remaining is the training set.
HKO-7 SShi el al. 2015 is a meteorological dataset containing sequences of observed Constant Altitude Plan Position Indicator (CAPPI) radar reflectivity an altitude of 2km, covering the region with radius of 256km centered at Hong Kong. The data are collected from 2009 to 2015 with time interval of 6 minutes.
The data range of the frames is set to [0 255]. Following previous work (Yan et al. 2024], this task as predicting the future radar echoes up to 2 hours (20 frames) based on the past 30 minutes (5 frames). We sample the collected data from 2009 to 2014 as the training set, while the rest is allocated to the test set.
formulate
an open-source meteorological dataset that consists of both satellite and radar observations with a 5-minute interval in France. The data covers geographical areas: the Northwestern and Southeast quarters of France, with the observation size of 550km 550km from 2016 to 2018. The data range of the frames is set to j0 - 70] Like the HKO-7 dataset, formulate this task to forecast the next 20 frames (100 minutes) of radar echoes based on the provided 5 frames (25 minutes) of radar echoes. Following previous work Yu et al. 2024h, we select the data specifically from Northwestern France and filter out the noisy precipitation events. The data collected from June to December 2018 serves as the test set, while the rest are used as the training set.
MeteoNet (arvoor et al. 2020
4.1.2 Evaluation
Following previous works (Gao al. 2023 Yu et al. 2024 Yan et al. 2024 various commonly used forecasting skill scores, such as the Critical Success Score (CSI) and the Heidke Skill Score (HSS), are reported (IoU), measures how
to evaluate the performance of the models. CSI, also known as Intersectiion-veer-UUn accurate the model prediction is after binarizing pixels of both prediction and observation with a specific threshold. The reported CSI is averaged over multiple selected thresholds, i.c., (16, 74, 133, 160, 181, 219) for SEVIR, (84, 117, 140, 158, 185) for HKO-7, and {12, 18, 24, 32) for MeteoNet. With the toleration of spatial deviations, averaged CSTs with pooling sizes of 4 and 16, which correspond to medium and large spatial

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: Performance comparison on multiple precipitation nowcasting benchmarks: SEVIR, HKO-7, and MeteoNet. The best score among all models is highlighted in bold, while the best score among the proba- bilistic models, including ours, is underlined. The inference time, sample (in seconds), single RTX3090 GPU and the sampling steps, N, of those diffusion models are reported as well.
Metrics
Dataset
Model
sammple
N
SSIM
IPPPS4
CSI-mt
CST4-m
CSh1--m
HSS1
ConvLSTM
0.7216
0.3025
0.3458
0.3411
0.3607
0.4467
0.03
PredRNN
0.7238
0.2708
0.3553
0.3702
0.4153
0.4621
0.15
SimVP
0.7209
0.2793
0.3788
0.3803
0.4160
0.4920
0.01
SEVIR
Earthformer
0.7102
0.3254
0.3556
0.3533
0.3838
0.4611
0.02
LDCast
0.5772
0.2906
0.2193
0.2898
0.4598
0.2995
4.26
50
PreDiff
0.6279
0,2217
0.3276
0.4271
0.6096
0.4498
73.09
1000
DiffCast
0.6979
0.1948
0.3580
0.4555
0.6281
0.4751
5.20
250
STLDM
0.7183
0.1929
0.3804
0.4662
0.6178
0.5024
0.51
20
COWVLSTM
0.5987
0.3184
0.2905
0.2628
0.2774
0.4076
0.03
PredRNN
0.5785
0.3131
0.2857
0.2872
0.3263
0.4026
0.15
SimVP
0.6039
0.3596
0.3020
0.2852
0.3115
0.4236
0.02
HKO-7
Earthformer
0.5864
0.3373
0.2817
0.2532
0.2704
0.3939
0.02
LDCast
0.6003
0.2322
0.2145
0.3122
0.5345
0.3165
4.75
PreDiff
0.5922
0.2391
0.2799
0.3787
0.5081
0.3973
70.80
1000
DifICast
0.6198
0.1949
0.3013
0.4084
0.6084
0.4240
20.50
250
STLDM
0.6433
0.1943
0.3191
0.4413
0.6511
0.4447
0.55
20
ConvLSTM
0.7938
0.2203
0.3619
0.3687
0.4130
0.5056
0.02
PredRNN
0.8158
0.1419
0.3455
0.4904
0.5837
0.4810
0.16
MeteoNet
SimVP
0.8134
0.1734
0.3858
0.4467
0.5746
0.5358
0.02
Earthformer
0.7806
0.2739
0.3401
0.3244
0.3488
0.4786
0.02
LDCast
0.7654
0.1691
0.2620
0.3658
0.5685
0.3904
4.76
50
PreDiff
0.7059
0.1543
0.2657
0.3854
0.5692
0.3782
70.80
1000
DiffCast
0.8167
0.1280
0.3831
0.4771
0.6335
0.5328
21.30
250
STLDM 0.8053
0.1275
0.3748
0.4921
0.6575
0.5233
0.51
20
tolerance, measured well. Besides that, HSS is skill score that assesses the model's ability to predict precipitation events after considering the actual distribution of corresponding thresholds mentioned above.
In addition, we also report two perceptiual-related metrics: Structural Similarity Index Measure (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS). SSIM is to evaluate the model's prediction in terms of structural similarity at the pixel level, while LPIPS is assess the visual quality of predictions by measuring the distance between each pair of prediction and observation encoded by a pre-trained model.
To judge the model efficiency during the inference, we report the prediction time per sample, Tsample single RTX3090 GPU. We also report the required sampling steps (aka denoising steps), N for those diffusion- based models, i.e,, LDCast, PreDiff, DiffCast, and STLDM.
4.2 Compared the State-of-the-Art
To evaluate the performance of our STLDM, we report three diffusion-based probabilistic models: LD- Cast (Leinonen et al., 2023), PreDiff (Gao al. 2023), and DiffCast (Yu et. al. 2024) as baselines. Both LDCast and PreDiff perform the diffusion process in the latent space, while DiffCast is composed of de- terministic model and a diffusion model running in the pixel space. For fairness, the performance of these probabilistic models, including STLDM, is evaluated among 10 ensemble predictions. Additionally, various deterministic models: ConvLSTM (Shi et al.) 2015), PredRNN (Wang et. al. 2017) SimVP (Gao el al. 2022a) and Earthformer (Gao et al. 2022b are included as references.
In Table g it is noted that STLDM is capable of achieving the best performance for most evaluation metrics, especially the HKO-7 dataset. Our STLDM achieves the best. performance in terms of TPIPS, which
9

---

Published in Transactions on Machine Learning Research (12/2025)
Impul
Ground
Sin VinVP
LDCas
PreDiE
DiC
SELLES
Figure 3: A set of sample predictions on the SEVIR test set. From top to bottom: Input, Ground truth, SimVP, PreDiff, DiffCast, and STLDM. The red region of the last prediction frame is zoomed in for a clearer comparison.
is deep-learning- based perceptual metric. This is supported by the visualization shown in Figure STLDM has the prediction that is the perceptually closest to the ground truth. However, STLDM does not consistently outperform all baselines in every metric across the other two benchmarks. On the SEVIR dataset, STLLDM is 1.64% worse on CSI16-m compared to DiffCast, while outperforming it on other metrics. On the MeteoNet. benchmark, STLDM shows performance drop of 1.40% to 2.16% SSIM, CSI-m, and HSS, but gains 0.39% 3.79% improvements on the remaining metrics. Detailed visualizations different datasets are shown in Figure
X that
and 15 respectively.
Since adopted the latent-diffusion design well as the model architecture, STLDM requires fewer sam- pling steps, enabling it to be 10X faster than DiffCast on the SEVIR datasct and 40X faster on both the HKO-7 and MeteoNet datasets. Notably, for models like Dif[Cast, the inference time scales exponentially with the image resolution. Hence, this efficiency gap between DiffCast and STLDM is even larger for data with higher resolution, as reported in Table D Morcover, those latent-diffusion related works: LDCast,, PreDiff, and our STLDM have relatively consistent sampling time across benchmarks, and STLDM is still the most efficient. among them.
10

---

Published in Transactions on Machine Learning Research (12/2025)
4.3 Analysis and Ablation Study
To identify which component contributes the most to STLDM's performance, we conducted several ablation studies, including the significance of the proposed loss term, Lc. and different training strategies every component of STLDM. Furthermore, we explore an alternative setup that treats the visual enhancement for every frame independently, rather than our current setting Spalio-Temporal Visual Enhancement Task. Same as the evaluation settings above, all ablation studies are conducted here with ten ensemble predictions.
4.3.1
Significance of Proposed Constraint Loss Term, Lo
Table 2:
Ablation study of the impact of Lc on the performace of STLDM with the SEVIR datasct. A better score highlighted in bold.
Metrics
Existence of Lc
SSIM
LPIPSI 0.2046
CSI-mf
CSI.-mf
CSh 0.6030
HSS1
0.7244 0.7183
0.3569
0.4533
0.4659
0.1929
0.3804
0.4662
0.6178
0.5024
As discussed in Section 3.2.2 Le is to constrain the final prediction, YI.,, such that it has the global motion trend same as the first estimation, YI.N- With the absence of Lc, the conditioning network, Vo. is trained only with the KL-divergence and diffusion loss in Terms C and respectively. Both terms implicitly constrain the prediction of Ve, resulting in a not well-constrained Y INN as shown in Figure 7 i.e., a noisy prediction. This noisy 1:NN does not help much the final prediction, Yi.N, by the latent denoising network, Do, ultimately causing STLDM to fail to prodict the precipitation events accurately.
The claim above is supported by the comparison of the model performance with and without Lo reported in Table 2 From Table 2 these two cases have similar performance in both SSIM and LPIPS, corresponding to the visual assessment of prediction. The existence of Le during training improves all forecasting skill scores: both CSI and HSS notably. This observation is confirmed by Figure 7 that YI-N tends to over-predict due to its noisy first estimation, VI.N from V in the absence of Lo during training. Tn contrast, Le regularizes the first estimation, Y INN, thereby enabling more accurate final predictions. This indicates that Le plays a crucial role in improving STLDM's accuracy via constraining the first estimation of Va.
Consequently, explicitly constraining the conditioning network, Vo, with the proposed constraint loss, Lo: consistently improves the performance of STLDM as it ensures the correctness of its prediction.
4.3.2 Different Training Strategies
Table 3: Ablation study on different training strategies: Which model components are trained along with the denoising network on the SEVIR dataset. better score highlighted in bold.
Components Trained Together
Metrics
Strategy
{E, D}
V
Da
SSIM
LPIPSJ
CSI-m
CSL-mf
CSh
HSS1
A
x
X
/
0.7086
0.2121
0.3809
0.4676
0.6209
0.5028
B
x
/
0.7173
0.1955
0.3822
0.4680
0.6209
0.5043
C
1
0.7183
0.1929
0.3804
0.4662
0.6178
0.5024
In this part, we investigate the impact of different training strategics on the performance of STLDM. Be- yond our current end-to-end buning strategy (Strategy C), we also report two allernatives: (i) Train every component individually (Strategy A), and (ii) Jointly train both the Latent Denoising Network, De, and Conditioning Network, Vo, without further tuning the pre-trained Variational Autoencoder (Strategy B).
These two approaches share a common feature: they do not require further tuning of pre-trained VAE during the training of the denoising network, Dg, resulting in a lower GPU memory demand for model training. Hence, these strategics are widely implemented because of their training efficiency. Additionally, using a pre-trained Conditioning Network further decreases the demand for GPU memory and also benefits from exposure to broader pre-training dataset, potentially improving generalivation.
11

---

Published in Transactions on Machine Learning Research (12/2025)
From Table all these three approaches have comparable performance in all forecasting skill scores, with the case that Strategy B: Tuning demoising network, Do, along with the Conditioning Network, Vo, has a slightly better performance. Furthermore, it is noted that our current training strategy, i.e. Strategy C: End-to-End Tuning, yields the best performance in terms of perceptual metrics, spocifically SSIM and LPIPS, while Strategy Tuning every component individually achieves the worst performance in these metrics.
Moreover, we study the training convergence of these approaches with the validation MSE and LPIPS reported in Figure G From the plots in Figure [6] Strategy A with both pre-trained VAE and v has the fastest convergence speed but fails to achieve those low metrics the other approaches could. Our current training setting, i.c., Strategy C, cxhibits slightly better validation performance than Strategy B after training. This reveals that the collaboration among all components in STLDM is important in resulting in better model performance, especially the cooperation between the Conditioning Network, Ve. and the Latent Denoising Network, Dg.
4.3.3 Can We Enhance Every Frames Independently?
Ablation study different kinds of visual enhancement tasks the HKO-7 dataset. A better score is highlighted in bold.
Table 4:
Types of Enhancement
LPIPSI
FVD
Tsample
Spatial
0.1878
241.17
0.4157
Spatio-Tempore
0.1943
87.14
0.5533
As mentioned in Section 3.2.1 we reformulate the nowcasting task as two sub-tasks: Forecasting and En- hancement sequence. In our current setting, we treat this visual enhancement task as a spatio-temporal task with a Spatial Temporal Latent Denoising Network, Dg. Here, we explore an alternative formulation that treats enhancement as a purely spatial (ask by removing all Temporal Atlention in Dp-
In addition to the required inference time, Teamples LPIPS (Zhang et al. 2018) and FVD (Unterthiner et al.) 2019) for assessing the perceptual scores in spatial and spatio-temporal, respectively, on the HKO-7 dataset in Table 3 as well. From Table 4 we observe that treating this visual enhancement task frame-wisely has slightly better LPIPS score and shorter inference time, Tsample- However, STLDM's spatio-temporal enhancement task achieves a significantly better FVD score, indicating superior temporal consistency. This finding is further supported by the visualizations in Figure| and 10 that the cloud motion in red bounding boxes in the settings of the Spatial Visual Enhancement task has drastic change, resulting in the temporal inconsistency; while our current setting and the ground truth share similar cloud movement, i.e., steady and slow. These results highlight the importance of treating this enhancement task as spatiotemporal task to achieve better temporal coherence.
single RTX3090 GPU, report two metrics;
4.3.4 Application in Meteorological Operation
Despite their fast inference time, low resource requirement, and reasonable accuracy, the radar nowcasts predicted by deterministic deep learning models are inadequate for supporting the prediction of convective weather systems, as they commonly struggle with blurriness issues, as shown in Figure [ In assessing the evolution of significant convective weather systems, it is critical to preserve both the spatial structure of high-intensity reflectivity pixels and their temporal consistency. STLDM is capable of improving the prediction of small-scale and complex structures of significant convective systems over longer time range, resulting in sharp and realistic nowcasts. This enables the weal forecasters to issue more timely alerts and warnings to the public. Furthermore, STLDM's fast inference time relative to other diffusion models enables probabilistic nowcasting with a large ensemble size real-time forccasting operations. This facilitates the assessment of the likelihood and alternative scenarios regarding the movement and severity of high-impact rainstorm events, thereby mitigating potential societal losses of life and property.
12

---

Published in Transactions on Machine Learning Research (12/2025)
5 Conclusion and Future Work
In this work, we propose simple yet effective Spatio-Temporal Latent Diffusion Model (STLDM) for precip- itation nowcasting, based on the idea of reformulating this lask into two sub-tasks in sequence: Forecasting and Enhancement. STLDM is composed of three modules: Variational AutoEncoder, Conditioning and a Latent Denoising Network. Extensive experimental results across multiple radar datasets demonstrate that STLDM outperforms existing techniques, validating its effectiveness. Further- more, we argue that introducing conditional regularization on the Translator during training consistently improves the modcl's performance. Morcover, we also reveal that training the Latent Denoising Network along with other components yields better results than training each component individually. Lastly, we also emphasize that modeling the visual enhancement task spatio-temporal task ensures better temporal consistency.
Limitations and future work. Despite STLDM's competilive performance, it still struggles to accurately forecast precipitation events that originate outside the observation region or are driven by unknown factors, as illustrated in Figure [6] A possible solution to introduce a multimodal model capable of capturing precipitation events beyond the observation window. Additionally, STLDM must be retrained for a new dataset in different region. Although the precipitation events in different. regions exhibit unique character- istics, they also share underlying common features, such the governing physical principles. Therefore, future direction is to develop a unified and physics-informed multimodal model across multiple benchmarks, offering not only broader generalization but also improved physical interpretability.
Acknowledgments
This work has been made possible by a Research Impact Fund project (RIF R6003-21) and a General Research Fund project (GRF 16203224) funded by the Research Grants Council (RGC) of the Hong Kong Government.
References
Cong Bai, Feng Sun, Jinglin Zhang, Yi Song, and Shengyong Chen. Rainformer: Features extraction balanced network for radar-bacee based precipitation nowcasting. IEEE Geoscience and Remote Sensing Letters, 19:1 5, 2022. doi: 10.1109/ 10119///RR.... /LG/L STS.S.222...1222..
Zheng Chang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. Strpm: A spatiotemporal residual predictive model for high-resolution video prediction. In CVPR, 2022.
Lei Chen, Yuan Cao, Leiming Ma, and Junping Zhang. A deep learning- based methodology for precipitation nowcasting with radar. Earth und Syace Science, 7(2), 2020.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In JCML, 2018.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. In NewrIPS, 2021.
Alexcy Dosovitskiy, Lueas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Jean-Yves Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. In ICML, 2020.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z. Li. SimVP: Simpler yet better video prediction. In CVPR, 2022a.
Zhihan Gao, Xingjian Shi, Hao Wang, Yi Zhu, Yuyang Wang, Mu Li, and Dit- Yan Yeung. Earthformer: Exploring space-time transformers for earth system forecasting. In NewrIPS, 2022b.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Zhihan Gao, Xingjian Shi, Boran Han, Hao Wang, Xiaoyong Jin, Danielle Maddix, Yi Zhu, Mu Li, and Bernie Wang. Prediff: Precipitation nowcasting with latent diffusion models. In NewrIPS 2023 AT for Science Workshop, 2023.
Junchao Gong, Lei Bai, Peng Ye, Wanghan Xu, Na Liu, Jianhua Dai, Xiaokang Yang, and Wanli Ouyang. Cascast: Skillful high-resolution precipitation nowcasting via cascaded modelling. In JCML, 2024.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint Xiv:22207. 1209007..255. 12598, 2022. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NearIPS, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusson models. In NewrIPS, 2022.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Francois Fleuret. Transformers are rmns: Fast autoregressive transformers with línear attention. In TCML, 2020.
Diederik P. Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data aug- mentation. In NewrIPS, 2023.
Gwennaëlle Larvor, Léa Berthomie, Vincent Chabot, Brice Le Pape, Bruno Pradel, and Lior Perez. Meteonet: An open reference weather dataset for ai by météo-france. Technical report, Metco France, 2020.
Jussi Leinonen, Ulrich Hamann, Daniele Nerini, Urs Germann, and Gabriele Franch. Latent diffusion models for generative precipitation nowcasting with accurate uncertainty quantification. CoRR, abs/2304.12891, 2023. doi: (15500/rXivv.20/..
Morteza Mardani, Noah Brenowitz, Yair Cohen, Jaideep Pathak, Chich-Yu Chen, Cheng-Chie Chig- Liu, Arash Vahdat, Mohammad Amin Nabian, Tao Ge, Akshay Subramaniam, Karthik Kashinath, Jan Kautz, and Mike Pritchard. Residual corrective diffusion modeling for km-scale atmospheric downscaling. arXiv preprinl ar Xrv:22300 15214, 2023.
llan Price, Alvaro Sanchez-Gonzalle Ferran Alet, Tom R. Andersson, Andrew El-Kadi, Dominic Masters, Timo Ewalds, Jacklynn Stout, Shakir Mohamed, Peter Battaglia, Remi Lamn, and Matthew Willson. Gen- cast: Diffusion- based ensemble forecasting for medium-range weather. arXic preprint arXiv:2312.15796, 2023.
Seppo Pulkkinen, Daniele Nerini, Andrés A. Pérez Hortal, Carlos Velasco-Forero, Alan Seed, Urs Germann, and Loris Foresti. Pysteps: an open-source python library for probabilistic procipitation nowcasting (v1.0). Geoscienlific Model Development, pp. 4185-4219, 2019. doi:
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint ar arxiv22204.000 0et:2204406125 2022.
Suman Ravuri, Karel Lenc, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsi- mons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, Rachel Prudden, Amol Mandhane, Aidan Clark, Andrew Brock, Karen Simonyan, Raia Hadsell, Niall Robinson, Ellen Clancy, Alberto Arribas, and Shakir Mohamed. Skilful precipitation nowcasting using deep generative models of radar. Nature, 597: 672-677, 2021. doi:
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High-resohttion image synthesis with latent diffusion models. In CVPR, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Raphael Gontijo-Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yan Young, Wai kin Wong, and Wang chun Woo. Convolu- tional LSTM Network: A machine learning approach for precipitation nowcasting. In NewrIPS, 2015.
Xingjian Shi, Zhihan Gao, Leonard Lausen, Hao Wang, Dit- Yan Yeung, Wai kin Wong, and Wang chun Woo. Deep learning for precipitation nowcasting: A benchmark and a new model. In NewrIPS, 2017.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Prakhar Srivastava, Ruihan Yang, Gavin Kerrigan, Gideon Dresdner, Jeremy McGibbon, Christopher Bretherton, and Stephan Mandt. Precipitation downscaling with spatiotemporal video diffusion. NeurIPS, 2024.
Cheng Tan, Zhangyang Gao, Lirong Wu, Yongjic Xu, Jun Xia, Siyuan Li, and Stan Z Li. Temporal attention unit: Towards elficient spaliotemporal predictive learning. In CVPR, pp. 18770 18782, 2023a.
Cheng Tan, Siyuan Li, Zhangyang Gao, Wenfei Guan, Zedong Wang, Zicheng Liu, Lirong Wu, and Stan Z Li. OpenSTT: A comprehensive benchmark of spatio-temporal predictive learning. In Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2023b.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Syl- vain Gelly, FVD: A new metric for video generation, 2019. URL https://opearrriew.ntttt wot.:oorii rylgEULtdN
Arash Vahdat and Jan Kautz. Nvac: A deep hierarchical variational autoencoder. In NeurIPS, 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz, Score-based generative modeling in latent space. In NeurIPS, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
Marc Veillette, Siddharth Samsi, and Christopher J. Mattioli. SEVIR: A storm event imagery dataset for deep learning applications in radar and satellite metcorology. In NewrIPS, 2020.
Vikram Voleti, Alexia Jolicoeur-Marti for prediction, generation, and interpolation. In NeurIPS, 2022.
and Christopher Pal. MCVD: Masked conditional video diffusion
Zhong Yi Wan, Ignacio Lopez-Gomez, Robert Carver, Tapio Schneider, John Anderson, Fei Sha, and Leonardo Zepeda-Núñcz. Regional climate risk assessment from climate models using probabilistic machine learning. arXiv preprint 1rXiv:2412.0880 2024.
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S. Yu. PredRNN: Recurrent neural
networks for predictive learning using spatiotemporal Istms. In NeurIPS, 2017.
Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and Philip S Yu. PredRNN++: Towards a resolution of the deep-in- time dilemma in spatiotemporal predictive learning. In ICML, 2018.
Yunbo Wang, Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, and Philip S Yu. Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. In CVPR, 2019.
Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip S. Yu, and Mingsheng Long, PredRNN: A recurrent neural network for spatiotemporal predictive learning. IFEE Trans. Pollern Anal. Mach, Intell., 45(2):2208 2225, 2023. doi: 10. (0.100//TAAM..20231155555
Chiu-Was Yan, Shi Quan Foo, Van Hoan Trinh, Dit-Yan Young, Ka-Hing Wong, and Wai-Kin Wong. Fourier amplitude and correlation loss: Beyond using 12 loss for skillful precipitation nowcasting. In NeurIPS, 2024.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video genera- tion. Entropy. 25(10), 2023. ISSN 1099-4300. doi: 10.3390/c25101449. URL https::/ww..mmg:: com/
1099-4300/5510/1469
Demin Yu, Xutao Li, Yunming Ye, Baoquan Zhang, Chuyao Luo, Kuai Dai, Rui Wang, and Xunlai Chen. Diffcast: A unified framework via residual diffusion for precipitation nowcasting. In CVPR, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
Yuchen Zhang, Mingsheng Longl, Kaiyuan Chen, Lanxiang Xing, Ronghua Jin, Michael 1. Jordan, and Jianmin Wang- Skilful nowcasting of extreme precipitation with nowcaslnet. Nolure, 619:526 532, 2023. doi: 10.1038/s41586- 123-06184-4.
16

---

Published in Transactions on Machine Learning Research (12/2025)
A Model Architecture of STLDM
In this section, we specify the details of every component of STLDM: a Variational autoencoder, a condition- ing network, and a latent denoising network. Additionally, STLDM's hyperparameters for both the training and inference processes are reported in this section.
A.1 Variational AutoEncoder, {E, D}
Channel Conversion
Channel Conversion
8STA Module
gSTA Module
DownSample
UpSample
Channel Conversion
Channel Conversion
gSTA Module
DownSample
UpSample
eSTA Module
Conditioning Network, 4e Figure 4: Illustration of the implemented encoder, &, decoder, 'D and conditioning network, V-
Encoder,
Decoder,
We follow these previous works (Gao et 2022a Tan al. 2023b) to build a Variational autoencoder, {E,D), D), shown in Figure 4 that solely relies convolutional operations. The only change we made is the removal of the skip connection between the encoder and the decoder. Specifically, the input radar frames at time 4, IR 121x128 128xx128 are encoded to the corresponding latent variables. E 323322x32 by the encoder while the decoder decodes the predicted latent variables back into the predicted frames.
The components inside the encoder and decoder are described here:
Channel Conversion: 3 x convolution layer, Group Normalization groups of 2 followed by a leaky ReLU activation layer.
Down/ Upsample: Transposed convolutional operations that downsample or upsample by a spatial
factor of
A.2 Conditioning Setwork/Translator V
(b,txe,h,w)
Attention (gSTA) modules (Tan et al. 2023b) as the conditioning network, V,, for modeling the underlying relation between the encoded imput frames and the tar- get output frames as illustrated in Figure [ The gSTA module is also solely composed convolutional operations for modeling spatio- temporal fcatures, imitating the spatio-temporal attention mechanism with large kernel convolution. Every gSTA module consists of depth-wise convolution, depth-wise dilation convolution, and channel-wise convolution as illustrated in Figure 5
We stack several Gated
Depth-wise Convolution
Depth-wise Dilation Convolution
Channel-wise Convolution
For constraining the conditional vector on Uhw Latent Denoising Nei- work ka. the prediction from Vo), we regularize the decoded predic- tion from Ve, YI,, with the ground truth as stated in the loss term, Le, included in Equation 10
(w,
Figure 5: Gated Spatio-Tempont Attention (gSTA) module
A.3 Latent Denoising Network, Do
The architecture of the latent denoising network, Dg, included our proposod STLDM shown in Figure In this section, elaborate different. components inside Dg:
17

---

Published in Transactions on Machine Learning Research (12/2025)
ResBlock: It is composed of two subblocks and followed by 1 x convolution for channel conver- sion. Fach subblock inside contains 3 x convolution, group normalization over groups of 8, and a SiLU activation.
Down/ /apsamplppmp::: These are the convolutional operations (with a kernel size of 4, padding of 1, and stride of 2) that downsamples or upsamplee by spatial factor of 2.
Spatial Attention: A self-attention between each pixel in every layer after interpreting the in- put batches of independent. frames (by shifting the temporal axis into batch dimension); i.e., [,,t, h, w| + [b t,c,, c, h, w].
Lincarized Spatial Attention (L-Spatial): This lincarized attention is self-attention between each pixel within patch with patch size, This is achieved by considering every independent patch as an attention head as well, after patching the query, key, and value. Patch size, p, halved for each Downsampling Block while p is doubled for each Upsampling Block. The order of computing the self-attention follows Equation 15
Temporal Attention: This is self-atlention between each frame in every layer along the lemporal dimension with the following reshape: [6, t, c, h w] + [6 x h w, c, t].
where b, c, h, and are denoted batch, time, channel, height, and width, respectively.
A.4 Hyper-Pane Parameters of Training and Inference
We trained the models for 200k training steps total on all benchmarks with batch size of The learning rate scheduled with 2k steps warm-up period, followed by a Cosine Annealing Scheduler decaying from the peak learning rate of le-4. Besides that, we set the total sampling steps of STLDM to 50.
During the inference process, we employ the DDIM technique (Song et al. 2021 of 20 sampling steps and the Classifier-Free Guidance (Ho & Salimans 2022 with the strength of 1.0.
B Details about Task Reformulation
In this section, we provide the details of the task reformulation process. Recall that, the objective of this nowcasting task is to find a predicted sequences, VELN, which has the highest conditional probability, p(Yn.) (( with the given input radar sequences, X1:ar.
First, the conditional probability of both events A and B happening given that event C occurred could be rewritten as the product of two conditional probabilities:
P(A, B = P,A,B,C)
P(C)
P(A C)P(
P(C)
= =P(AB.C)P(BIC)
Motivated by the idea above, introduce an intermediate variable (aka the first estimation), Y L,,, then could rewrite the conditional probability of this task objective follows:
p(PIN(XIM)) = / p(PLN, XL.
f p( (Y 1.N X1.M )dY I:
18

---

Published in Transactions on Machine Learning Research (12/2025)
Hence, we introduce additional task objective Forecasting that constraining the first estimation, 11:N to this nowcasting task:
(NN(XEM) (Y (Y (.. (
V Vologp(VI.N FiN|XXiMM
V PY 1.1 (
Ve log I: l i:N, X1:M):
Forerasting
Visual Enhancement
where 0 refers to the model parameters. The first term is obligated for the forecasting objective, while the latter term is responsible for the visual enhancement goal.
To fulfill this, we reformulate the objective of the nowcasting task into two sequential sub-tasks: Forecasting and Enhancement, with proposing a constraint loss, Lc, mentioned Equation 10
C Validation MSE and LPIPS of Different Training Strategies
14
Stratgyy Stratgyy
Stratgy A
Strategy
0.45
Strategy
0.40
MSE (10 3)
LPIPS
0.30
0.20
10
20
30
40
50
0
10
20
30
40
50
Training Epoch
Training Epoch
Figure 6: Validation MSE and LPIPS of different training strategies of STLDM during the training process.
Evaluation on a High-Resolution Dataset
To evaluate the feasibility of STLDM at different spatial resolutions, we also report its quantitative results on the HKO-7 dataset with spatial size of 256 x 256 with two basclines: LDCast and DiffCast in Table The forecasting task remains the same predicting the next 20 frames given input frames.
From Table we observe that STLDM achieves the best performance across all metrics. This result is consistent with its performance on the downscaled HKO-7 dataset reported in Table [ This makes conclude that STLDM consistently outperforms the baselines on the HKO-7 dataset regardless of spatial resolution. A detailed visualization is shown in Figure 11
Table 5: Performance comparison of our proposed STLDM with two baselines: LDCast and DiffCast on the HKO-7 dataset with the spatial size of 256. The best score among all models is highlighted in bold.
Metrics
Model
SSTM
T.PISS
CSI-mt
CSL
CST
HSS1
LDCast
0.6776
0.2476
0.1558
0.2096
0.3675
0.2375
DiffCast
0.6986
0.1854
0.3111
0.3799
0.5303
0.4372
STLDM
0.7097
0.1840
0.3778
0.4137
0.5665
0.4671
19

---

Published in Transactions on Machine Learning Research (12/2025)
E Running Time of STLDM and Baselines
We have demonstrated that STLDM is the most effective among diffusion-based models, achieving state- of-the-art performance across multiple real-life radar echo datasets. In this section, analyze STLDM's efficiency from two perspectives: inference time and training cost.
To compare inference elficiency with other baselines, we report their inference times on the HKO-7 benchmark at three spatial resolutions: 128, 256, and 512 in Table G From Table G we conchuded that STLDM consistently achieves the fastest sampling speed at every spatial scale, resulting the shortest inference time. Moreover, the performance gap between STLDM and the baselines is increasing with the spatial scale.
In addition, also reported the average training time of these models on the HKO-7 benchmark with spatial size of 128 and batch size of over 10k training steps in Table 7 From Table / despite being trained from end to end, STLDM still requires the least training time, leading to the lowest training cost. Table and 7 highlight that STLDM is the most efficient model, achieving both the shortest inference time and the lowest training cost.
Table 6: Inference time (in seconds) of each model on the HKO-7 dataset with different spatial sizes, using single RTX3090 GPU.
Inference Time (s)
Model Sampling Steps
128
256
512
LDCast
50
4.76
12.17
46.69
PreDiff
1000
70.80
222.73
900.15
DiffCastt
250
20.50
25.06
99.90
STLDM
20
0.55
0.66
2.41
Table 7: Average training time for each model on the HKO-7 benchmark with batch of over 10k training steps, using a single RTX3090 GPU.
Model LDCast with pre-trained VAE
Training Time
3 hours 48 minutes
DiffCast
4 hours 12 minutes
STLDM from end to end
2 hours 23 minutes
F The Implementation of Classifier-Free Guidance and Its Significance
As discussed in Section 3.1.2 the technique of Classifier-Free Guidance (CFG) Ho Salimans (2022 is ensure that the generated sample constrained with the given conditional variable, c, which is the first estimation, 21,N, in the proposed STLDM. In this section, we describe how CFG is implemented in STLDM and study the impact of CFG on the performance of STLDM.
As mentioned in Section 3.1.2 CFG involves two cases: both the conditional and unconditional cases. Following prior works, we adopt a joint training strategy where the conditioning signal is randomly dropped with the probability 15% (i.e, 85% conditional and 15% unconditional training). During inference, at each denoising timestop, we compute both the conditional and unconditional noise predictions, which is two forward passes per step. We then combine these predictions and modify the denoising score according to Equation with guidance strength, w 1.0. This ensures that CFG is applied throughout the sampling process.
Specifically, to apply CFG in our STLDM, we concatenate the latent output from the Translator module, E1N,, with the noisy latent input, zi.,, along the channel dimension Srivastava et al. (2024) at the first ResBlock of cach Downsampling Block, as illustrated in Figure [ This corresponds to the conditional case. For the unconditional case, we replace E with a zero tensor of the same size, which serves as the mull embedding here.
20

---

Published in Transactions on Machine Learning Research (12/2025)
To validate the effectiveness of CFG, conducted ablation study the SEVIR dataset here. As reported in Table applying the CFG technique consistently improves the performance of STLDM across all evaluation metrics. This improvement is validated by the visualization shown in Figure 12 These results demonstrate that incorporating CFG provides consistent performance improvements, indicating its effectiveness.
Table & Ablation study of the impact of CFG the performance of STLDM with the SEVIR dataset. A better score is highlighted in bold.
Metries
Existence of CFG
SSIM
LPIPS4
CSI-mi
CSI4-m²
CSh
HSS1
X
0.7041
0.2129 0.1929
0.3719
0.4376
0.5614 0.6178
0.4898
0.7183
0.3804
0.4662
0.5024
G
More Visualizations
6-6
(-5
3
Inpott
GT
Flrst Estination (Wihout Co)
Final Prediction [W5thous Eo)
First Estimation (With Co)
Final Prodiction (Wita Ci)
Figure 7: A set of first estimation and final prediction from STLDM trained in both cases, that with and without the proposed Constraint Loss, Lo, as mentioned in Term A during training the SEVIR test set.
21

---

Published in Transactions on Machine Learning Research (12/2025)
Inpot
Ground Truth
Strategg A
151
Strategg 13
Strategy C
2+3
f.5
2+1
Figure 8: A set of sample predictions from STLDM trained with different training strategies mentioned in Section 4.3.2 on the SEVIR test set.
Ground Frudh
Spabiel
Spielio fee--eemppee orae
Figure 9: A set of sample predictions from STLDM with two different kinds of enhancement: Spatial and Spatio Temporal on the HKQ-7 test set. The red region of the first five frames is zoomed in for a clearer comparison.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Sapus
Goomad Troih
Spartal
Spento- Terppoo
1+3
1+5
147
1.99
2=21
38
17
1+119
Figure 10: A set of sample predictions from STLDM with two different kinds of visual enhancement settings: Spatial and Spatio- Temporal, as mentioned in Section 4.3.3 the HKO-7 test set.
Input
Ground Thum
DINCORt
STL.DM
Figure 11: set of sample predictions on the HKO-7 test set with a spatial size of 256. From top to bottom:
Input, Ground truth, LDCast, DiffCast, and STLDM.
23

---

Published in Transactions on Machine Learning Research (12/2025)
Iuput.
Ground Truth
Without CFC
With CFG
1+1
1+3
645
C+T
1-
t+ 11
Figure 12: A set of sample predictions on the SEVIR test set with the absence and existence of Classifier FFee Guidance (CFG). The red region of the last prediction frame is zoomed in for a clearer comparison.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Iapal
Ground Treth
CoavLSTM
PreiRNN
Stav
255
Barthforeee
LDDCat
PreDiE
DIZCOse
STLDM
1+1
C+3
L+5
L+T
L+9
1+11
Figure 13: A set of sample predictions on the SEVTR test set. From top bottom: Input, Ground truth, ComvLSTM, PredRNN, SimVP, Earthformer, LDCast, PreDiff, DiffCast, and STLDM.
25

---

Published in Transactions on Machine Learning Research (12/2025)
Eupel
Glouall Tovis
Staff
BartiSmus
1200a
Figure 14: A set of sample predictions on the HKO-7 test set. From top to bottom: Input, Ground truth, ConvLSTM, PredRNN, SimVP, Earthformer, LDCast, PreDiff, DiffCast, and STLDM.
26

---

Published in Transactions on Machine Learning Research (12/2025)
Glourd Irats
Stal
Earliforun
STL.M
Figure 15: A set of sample predictions on the MeteoNet test set. From top to bottom: Input, Ground truth, ConvLSTM, PredRNN, SimVP, Earthformer, LDCast, DiffCast, and STLDM.
Sapus
Goouall Feoih
STTLEME
[+]
1+3
1+5
1+7
1+9
1-31
F+15
1+17
1+19
Figure 16: A set of sample predictions that both DiffCast and STLDM failed to predict the precipitation events spawning on the left due to the limited observation region on the HKO-7 test set.
27


Reviews:
[{"review_id": "LwcP0UuTrx", "reviewer": "Reviewer_4Mk9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper applies standard latent diffusion model on Nowcasting problem. The latent diffusion consists of a VAE and a latent denoising network. It claims the state of the art results on several nowcasting dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper claims the proposed method is more effective and superior to the state of the art. According to Table 1, the propsed method (STLDM) has better accuracy compared to previous methods such as DiffCast. However, several previous works have explored generative models in related tasks such as CorrDiff, GenCast, GenFocal.", "audience_yn": "Yes", "audience_explanation": "While is latent diffusion model is standard, the application to nowcasting dataset could be interesting to some audience in the venue.", "requested_changes": "Diffusion models have been broadly applied in weather forecast, nowcasting, and downscaling. The author should discuss these previous work in the field:\n- Corrdiff: Mardani, Morteza, et al. \"Residual diffusion modeling for km-scale atmospheric downscaling.\" (2023).\n- GenCast: Price, Ilan, et al. \"Gencast: Diffusion-based ensemble forecasting for medium-range weather.\" arXiv preprint arXiv:2312.15796 (2023).\n- GenFocal: Wen et al. \"Regional climate risk assessment from climate models using probabilistic machine learning.\"", "broader_impact_concerns": "No ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SlyVFisrd5", "reviewer": "Reviewer_WV7f", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper splits the precipitation nowcasting task into two parts, first predicting 'global' structure in the image and then conditioning on that to predict the 'local' structure. This address problems in existing approaches which tend to either get reasonable global structure (while producing blurry local structure) or get plausible local structure (while predicting poorly due to missed global structure). The paper proposes a latent diffusion model, trained end-to-end, to jointly solve the parts of the task. The method achieves state of the art performance compared to a reasonable selection of other methods and is evaluated across a number of real datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Claims of performance and good runtime are supported by ample experimental results on real data, compared to a decent selection of existing methods, and through ablation studies.", "audience_yn": "Yes", "audience_explanation": "Some members of the TMLR audience work on this exact problem (nowcasting precipitation), and there are even more working on related topics (general forecasting and learning from time-series data) who might find the paper interesting.", "requested_changes": "I have one critical request:\n1. Are all the values reported in the tables the result of one run? It would improve credibility of the analysis to instead perform multiple runs and report the mean/median and SD/CI. Or is this not computationally feasible?\n\nAnd here's a list of suggestions that I think would improve the presentation:\n1. elaborate upon the difference between \"more effective\" and \"superior to\" in the last sentence of the abstract (or remove the redundancy if they mean the same thing)\n2. writing \"ie.\" isn't standard, i.e., it's usually rather written in the preceding manner, with two commas and two periods\n3. when introducing notation, X, it helps to have a comma on both sides\n4. in Section 2.2: \n    1. singular \"...a likelihood...\" and then plural conjugation \"...which do not suffer...\"\n    2. \"AFNO\" is undefined\n    3. \"stochastic\" is an adjective, and its noun seems to be missing\n5. page 6: I don't understand the sentence \"Therefore, to enable...\"; maybe it could be structured better to improve clarity?\n6. page 7: \"while without\" could be changed to just \"without\"\n7. the Limitation and future work section could use some polishing:\n    1. maybe the plural \"Limitations...\" is more suitable\n    2. having both \"Although\" and \"however\" in the first sentence is a bit redundant and confusing\n    3. \"multi-modal\" is an adjective, and the noun its describing seems be missing (maybe \"model\"?)\n    4. \"even though\" and \"but\" in the second-to-last sentence is redundant/confusing; maybe \"that\" missing between \"events\" and \"happen\"? \n8. I found some of the word choices throughout the paper to a bit confusing/not quite right:\n    1. page 2: mitigate, incorporation\n    4. page 3: demonstrating, panacea, prominent, demonstrate", "broader_impact_concerns": "N/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JnhINE3MDN", "reviewer": "Reviewer_LXNm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Contributions of the manuscript: \n\n- Reformulates precipitation nowcasting into two sequential subtasks: Forecasting (global motion trend) and Enhancement (fine-grained details). Proposes STLDM (Spatio-Temporal Latent Diffusion Model) combining a VAE, Conditioning Network, and Latent Denoising Network, trained end-to-end.\n- Introduces a constraint loss (LC) to align global motion and enhance final predictions.\n- Demonstrates state-of-the-art performance on multiple radar datasets (SEVIR, HKO-7, MeteoNet) across both accuracy and perceptual metrics.\n- Achieves substantially faster inference than prior diffusion-based models due to latent-space design.\n\nStrengths of the manuscript:\n- Balanced trade-off: Improves both accuracy (like deterministic models) and sharpness/realism (like generative models).\n- Efficiency: 10–40× faster inference than pixel-space diffusion models.\n- Robustness: Strong performance across diverse datasets and metrics.\n- Novelty: First to jointly train VAE, conditioning, and diffusion modules for nowcasting.\n- Careful ablation studies support design choices (e.g., effectiveness of constraint loss).\n\nWeaknesses of the manuscript:\n- Inconsistent performance: Does not dominate across all datasets and metrics (e.g., CSI/SSIM on SEVIR and MeteoNet).\n- Complex architecture: Although efficient in inference, training involves multiple interacting components (VAE + diffusion + translator), which may increase training cost and tuning difficulty.\n- Limited interpretability: Like most deep generative models, lacks transparency for decision-making in meteorological operations.\n- Evaluation scope: Tested mainly on downscaled radar datasets; scalability to higher resolution or operational deployment is not fully explored.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the claims are generally supported by accurate and convincing evidence through extensive experiments on three benchmark datasets, use of multiple established metrics, and clear ablation studies that validate the proposed components. The efficiency and balanced performance claims are particularly well substantiated, with quantitative results and visualizations illustrating improvements. However, some claims of superiority are overstated, as the model does not consistently outperform baselines across all datasets and metrics, and evidence for operational scalability remains limited.", "audience_yn": "Yes", "audience_explanation": "The TMLR audience includes researchers in machine learning and applications such as spatiotemporal modeling, generative models, and weather forecasting, all of which are directly addressed by this paper.", "requested_changes": "- Tone down claims of universal superiority; highlight mixed results across datasets/metrics.\n- Expand discussion of limitations, especially scalability to higher-resolution or operational settings.\n- Clarify training complexity and computational costs relative to baselines.\n- Improve justification of how the constraint loss contributes to performance.\n- Provide more discussion on interpretability and potential use in real-world meteorological operations, e.g. well defined road for real practice.", "broader_impact_concerns": "A reasonable concern here is that the Broader Impact Statement may not fully cover the societal and ethical implications of this work. While precipitation nowcasting is beneficial for disaster preparedness and resource planning, deploying a generative model for weather prediction also carries risks: inaccurate or overconfident forecasts could mislead policymakers or the public, with potential consequences for safety and resource allocation. Additionally, the complexity and lack of interpretability of diffusion models may reduce trust and hinder accountability in operational meteorology. The paper would benefit from explicitly acknowledging these issues, discussing the importance of uncertainty communication, and clarifying that such models should complement, not replace, established forecasting systems.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)