PredLDM: Spatiotemporal Sequence Prediction with Latent Diffusion Models
Authors: Yechao Xu
Zhengxing Sun
Qian Li
Jiao Qu
Venue: Accepted by TMLR ()

Abstract: Predicting the accurate and realistic future is an attractive landmark in spatiotemporal sequence prediction. Despite recent progress in spatiotemporal predictive models, explorations in this field are challenging due to difficulties in intricate global coherence and comprehensive history understanding. In this study, we introduce latent diffusion models (LDMs) into spatiotemporal sequence prediction (PredLDM) with a two-stage training paradigm. (i) To compress intricate global coherent spatiotemporal content into latent space, we propose the masked-attention transformer-based variational autoencoder (MT-VAE) by exploiting transformers with masked self-attention layers. (ii) Different from LDMs in generation-related fields where the condition in our problem settings is historical observations instead of texts, the condition-aware LDM (CA-LDM) is provided for comprehensive understanding of historical sequences. Our denoising diffusion process learns the distribution of both conditional generation and condition-aware reconstruction. Results on KittiCaltech, KTH and SEVIR datasets show that our PredLDM provides promising performance and realistic predictions in multiple scenarios including car driving, humans and weather evolutions. (https://github.com/MaoWuToday/PredLDM.git)

Qian Li Published in Transactions on Machine Learning Research (01/2026)
PredLDM: Spatiotemporal Sequence Prediction with Latent Diffusion Models
Yechao Xu
zhigo.orishi@gg com
State Key Loboratory for Novel Software Technology, Nanjing University
Zhengxing Sun'
cenunjju.edu.cr
State Key Laboratory for Novel Software Technology, Nanjing University
public ligion@166 (1..mm1163.omm com
College of Meteorology and Oceanogrophy, National University of Defense Technology
Jiao Qu'
gujiao Cujjaaorjj..co edu.
State Key Laboratory of Phornaccutical Biotochnology and Nanjing Drum Tower Hospital the Affiliated Hospital of Nanjing University Medical School, School of Life Sciences, Nanjóng University
Reviewed on OpenReview: https: openreview. net/forunt forum? id= TWmmOUscCo
Abstract
Predicting the accurate and realistic future is an attractive landmark in spatiotemporal se- quence prediction. Despite recent progress in spatiotemporal predictive models, explorations in this field are challenging due to difficulties intricate global coherence and comprehen- sive history understanding. In this study, we introduce latent diffusion models (LDMss into spatiotemporal sequence prediction (PredLDM) with two-stage training paradigm. (i) To compress intricate global coherent spatiotemporal content into latent space, propose the masked-attention transformer-basee variational autoencoder (MT-VAE) by exploiting transformers with masked self-attention layers. (ii) Different from LDMs in generation- related fields where the condition in our problem settings is historical observations instead of texts, the condition-aware LDM (CA-LDM) is provided for comprehensive understand- ing of historical sequences. Our denoising diffusion process learns the distribution of both conditional generation and condition-aware reconstruction. Results KittiCaltech, KTH and SEVIR datasets show that our PredIDM provides promising performance and realis- tic predictions in multiple scenarios including car driving, humans and weather evolutions. [tttpp://gggaaat..cca//ooooooorr com./Maaokaaoddy////edd..... git)
(https://githe) com
1 Introduction
Spatiotemporal sequence prediction is a fundamental task in computer vision that given a sequence of images, neural nerwsrks predict the subsequent image sequence to describe what will happen in the future (Oprea et al., 2020; Shi al., 2015). Different from video generation (Ho et al., 2022b) predicting from text prompts or unconditionally, this task is conditioned on historical observations of dynamic scenes (Oprea et al., 2020). By learning underlying spatiotemporal patterns from successive data with unsupervised manners, model is to predict accurate dynamics with realistic visual appearance (Tee et al., 2018). Tt can serve various disciplines, such as autonomous driving (Kwon & Park, 2019), robotics planning (Finn et al., 2016), traffic management (Liu et al., 2024b) and weather forecasting (Zhang el al., 2023b).
ideal
For producing future frames, classical predictive models are mostly optimized by minimizing mean error between predictions and ground truth across spatial and temporal dimensions (Oprea et al., 2020). Shi
*Corresponding authors

---

Published in Transactions on Machine Learning Research (01/2026)
et al. (2015) introduce ConvLSTM networks, which is a milestone at grasping spatiotemporal aspects with convolutional recurrent architectures. Inspired by this, advanced recurrent models (Wang et al., 2017; 2018; Wu et al., 2021b; Sun et al., 2023; Villegas et al., 2017; Oliu et al., 2018) and recurrent-free ones (Gao et al., 2022; Tan et al., 2023) emerge out. However, the mean crror-related objective leads to generating blur for uncertain future outcomes (Oprea et al., 2020; Lee et al., 2018). To improve visual quality, although generative models like variational autoencoders (VAEs) (Villegas et al., 2019; Wu et al., 2021a; Babacizadeh et al., 2021), generative adversarial networks (GANs) (Clark et al., 2019; Tulyakov et al., 2018) and flow-based models (Dorkenwald et al., 2021) are alternatives, they are easy for mode collapase and the performance is not satisfactory. As LDMs reveal promising performance with high-fidelity appearance especially in T2T (Nichol et al., 2021; Rombach et al., 2022) and T2V (Ho et al., 2022b; He et al., 2022) through learning joint distributions with conditions in latent space by iterative denoising diffusion processes (Rombach et al., 2022), we introduce LDMs into spatiotemporal sequence prediction, under consideration of intricate global coherence and comprchensive history understanding.
In the light of intricate global coherence, existing predietive models are restricted by finite-scale temporal variations within training samples, whereas temporal transformations are complex and nearly infinite in nature. For simulating intricate temporal patterns, a solution is to model as many diverse variations as possible in the pretraining stage (Devlin, 2018; He et al., 2022) by masked modeling (Xie et al., 2022; Cheng et al., 2022). This can serve perceptual compression in LDMs (Singer et al., 2022). Meanwhile, existing LDMs exploit 3D convolutions (Ho et al., 2022b; He et al., 2022) and convolutional temporal layers (Singer et al., 2022) to extend T2I models to T2V applications. This leads to global dependencies being neglected, limited by compression of the local receptive field of convolutions (Li et al., 2023). For modeling global reliance, transformers are natural altcrnatives. To solve intricate global coherence, we expoct to propose transformer-baset VAE with masked modeling. In another light of comprehensive history understanding, different from generation pnterattioouupppoeeee models conditioned by text prompts, the condition in this problem setting is historical image sequences. Compared to text prompts describing scenes with highly dimensional symbols, conditions of spatiotemporal sequences are more difficult for machines to understand as raw pixels are low- level and diverse. It expected to leave conditions comprehensively understood latent space during denoising diffusion processes.
With respect to these problems, we propose spatiotemporal predictive model called PredLDM. (1) To com- press intricate global coherent spatiotemporal content into latent space, we propose exploiting transformer- based VAE with masked attention to capture complex and global coherence in MT-VAE. (ii) To comprehen- sively understand the historical observations, condition-aware latent diffusion is performed. The denoising diffusion process of CA-LDM learns the distribution for both conditional generation and condition-aware roconstruction.
Extensive experiments are conducted on KittiCaltech (Geiger et al., 2013), KTH (Schuldt et al., 2004) and SEVIR (Veillette et al., 2020) datasets. Results show accurate performance and realistic visual appearance of trained PredLDM, indicating the promising future of this study. Contributions can be summarized as:
To predict the accurate and realistic future image sequences, we propose a spatiotemporal predictive model called PredLDM, by introdueing LMDs into this field under consideration of intricate global coherent modeling and comprehensive history understanding.
For intricate global coherence, MT- is proposed by transformers with masked attention varia- lionally compressing complex temporal patterns and global reliance.
Considering comprehensive history understanding, CA-LDM is performed by learning distributions of both conditional generation and condition-aware reconstruction.
Experiments on several datasets including KittiCaltech, KTH and SEVIR show superior performance of PredLDM with realistic appearance, revealing potential for continuous research and applications.
2

---

Published in Transactions on Machine Learning Research (01/2026)
2
Related Works
2.1 Spatiotemporal Sequence Prediction
Spatiotemporal sequence prediction produces the future sequence of images given by historical observations to describe what is going to happen (Oprea et al., 2020). This rescarch direction originates from predictive coding (Huang & Rao, 2011; Rao Ballard, 1999) which reveals the human behavior predicting visual signals through both space and time dimensions. Initial attempis from Ranzato et al. (2014) and Srivastava et al. (2015) introduce recurrent language baselines to model natural spatiotemporal signals. For explicitly modeling spatial information, Shi et al. (2015) propose using convolutions to replace fully connected layers in recurrent units. This attempt greatly inspires the progress on the recurrent predictive architectures (Wang et al., 2017; 2018; Wu et al., 2021b; Sun et al., 2023; Villegas et al., 2017; Oliu et al., 2018) on this task. Besides fashion (Gao et al., 2022; Tan et al., 2023) is employed with
recurrently modeling, the sequence-to-seqquens efficient U-Net structures to predict with a simplified configuration of convolutions. When minimizing mean error of predictions and uncertain future outcomes, these models usually generate blur appearance (Oprea et al., 2020). A straightforward solution is exploit probabilistic models, like VAEs (Villegas et al., 2019; Wu et al., 2021a; Babaeizadeh el al., 2021), GANs (Clark el al., 2019; Tulyakov et al., 2018) and flow-based ones (Dorkenwald et al., 2021). However, they are highly likely to lead mode collapse and hard to fit (Oprea et al., 2020). As LDMs are dominant in generation-relatet works (Rombach et al., 2022; Ho et al., 2022b), we introduce LDMs in spatiotemporal sequence prediction for polential explorations in this field.
2.2 Latent Diffusion Models
Diffusion models (DMs) are one of likelihood-basee generative models, revealing first remarkable results in image generation communities (Ho et al., 2020; Nichol et al., 2021) by progressively reversing a. Markov chain which iteratively adds noise to target distributions (Romneberger et al., 2015). Benefiting from lower computational requirement and better expressivity than DMs (Song et al., 2020; Karras et al., 2022), recently LDMs (He el al., 2022; Rombach et al., 2022) have made tremendous breakthroughs on various lasks, including T2I generation (Nichol et al., 2021; Rombach et al., 2022; Balaji et al., 2022; Saharia et al., 2022; OpenAI, 2023; Midjourney, 2023; Peebles & Xie, 2023; Podell et al., 2023), T2V generation (Shi et al., 2015; He et al., 2022; Yan et al., 2021; Singer et al., 2022; Voleti al., 2022: Ho al., 2022a; Blattmann et al., 2023b; Zhou et al., 2022; Wang et al., 2023; Midjourney, 2023), text-to-audio generation (Liu et al., 2023a), 3D shape generation (Vahdat et al., 2022), video editing (Liew et al., 2023), tabular data generation (Zhang et al., 2023a), video frame interpolation (Danier et al., 2024), etc. Most related directions are T21 and T2V models. In T21 generation (Nichol et al., 2021; Rombach et al., 2022; Balaji et al., 2022), novel images are generated with textual descriptions given as conditions, where represontatives contain Dalle-2 (OpenAI, 2023), Midjourney (Midjourney, 2023), DiT (Peebles & Xie, 2023) and Stable Diffusion (Podell et al., 2023). T2V models are mostly inspired from T2T (Singer et al., 2022). VDM (Shi et al., 2015) reports first results by modifying 2D U-Net to factorized 3D network achieve video synthesis. Recent works include Imagen Video (Ho et al., 2022a), SORA (OpenAI, 2024), Make-A- Make-A-Vidoo (Singer et al., 2022). VidcoGPT (Yan et al., 2021), Magic (Zhou et al., 2022), Latte (Ma et al., 2024), StoryDiffusion (Zhou et al., 2024) and Cog VideoX (Yang et al., 2024), extending existing image- models to the video domain. They are usually built with two stages (Rombach et al., 2022). In the first stage, VQ-VAEs (Van Den Oord et al., 2017; Razavi et al., 2019) or VQ-GANs (Esser et al., 2021) are used as autoencoders to learn an expressive prior over discretized latent space. This autocncoder-diffusion paradigm has also been effectively adapted for sequential data synthesis, as demonstrated in TitoDut (Suh et al., 2024). In the second stage, a denoising network commonly implemented by U-Net (Ho et al., 2020; Ronneberger et al., 2015; Dhariwal & Nichol, 2021) is trained to predict the less noisy video samples progressively, reversing the diffusion process where Gaussian noisc is itcratively added onto the raw data with predefined timesteps. Concurrently, efficient temporal modeling techniques like those in Latent-Shift (An et al., 2023) have advanced video generation by introducing specialized operations in the latent space. Beyond generation tasks, PriSTI (Liu et al., 2023b) addresses missing data challenges through conditional diffusion frameworks. Similarly, Stochastic Diffusion (Liu al., 2024a) extends this capability to uncertainty-aware time scries forecasting, highlighting the versatility of diffusion models in handling complex temporal dependencies. Different form T2T and T2V
3

---

Published in Transactions on Machine Learning Research (01/2026)
Perceptual compresiio with MT-VAE
(R, E,02,))
Datel
Spatial encoeer_
Spacial decder
Staplee Ginssian noise
Dense
Spationamppoal sequence
Reconstructed sequeneo
Conditional sequence
Rat
((2g))
Lae arg Annd ((R,)) D(2.)) 12
Spucial Self-Attemion
Truining stage 2: Condition- latent diffesion
E. D.
MI-VAE
encoder
Diffusion prosess
Demoting concoallo Trane Romeerrerr
Tampural Self-Attamtion
Ground wuth
E.F.D.p
XT timestops
Racoo
Dereising ciscidad transfomerss
Reiti
2u00
Conditional generation
Faoi Preward
Cros: Aremiio
Conditional Cureo-ncntien
For leaming latent distibution p(Ral(
Conditonal
Spatill
Rapp
prodicts
Feed Forward
for Ryra based on historical observations
Prodicted
Rutti
(f(a(Ran))- t-U((17)
Ayon Feater ann [k ceccll)
cundition intogration
Predicted noisy samples after 7 timestepe
Exp()
MI-VAL
Diffision process
Pretrained decader in MT-VAE
E.F.D.p
X timesopps
Conditional soquenic
Ryecy
Denccising csscudad transforeets to
Fatt71
Condition-aman aww recomstruction for learning latert disembution p(C,))
Feed Pooern Foowaad
Tauponal
Se 6-iieertoo
Sputal Sdl-kmentt
<o predicts reversed
only for reconstrnctoon of conditions
Predicted sequence (
Prodicted
6o (o (f) u(L
Figure The pipeline PredLDM. The training is comprised two stages. the first stage, MT-VAE is exploited to compress the spatiotemporal sequences into latent space. In the second stage, CA-LDM contains the learning of both conditional generation and condition-aware reconstruction. For inferencing, the Gaussian noise is sampled and less noisy latent vectors predicted conditioned on the latent historical observations. The sampled latent vectors are then fed into the decoder for output.
models, the condition our PredLDM historical observations. This difference makes the condition is not as easy as text prompts used to be, as the spatiotemporal content is low dimensional and highly complex.
2.3 Compression of Spatiotemporal Sequences
In order to compress spatiotemporal data, 3D convolutions are straightforward solutions (Ho et al., 2022b; Tran et al., 2015). Given multiple frames, VDM (Ho et al., 2022b) and LVDM (He et al., 2022) exploit 3D U- Net convolutions by replacing each 2D convolutions image models with space-only 3D convolutions. Instead of 3D convolutions expensive fitting hard to train, 1D convolutional temporal layers are attractive combined with 2D convolutions (Tran et al., 2018). Make-A-Video (Singer al., 2022) initializes the spatial convolutional layers with pretrained T21 weights and adds temporal convolutions to correlate spatial features across time dimensions, similar as Imagen Vidco (Ho et al., 2022a), ModelScope (Wang et al., 2023), MagicVider (Zhou el al., 2022) and Stable Video Diffusion (Blattmann et. al., 2023a). There are also works combining convolutional temporal layers with 3D U-Net (Blattmann et al., 2023b). Although convolutions are effective in image modeling, diverse and global relations time dimensions are too complex limited for their local receptive field (Li et al., 2023). To additionally capture intricate global dependencies, we make attempts by proposing MT- VAF.

---

Published in Transactions on Machine Learning Research (01/2026)
3 Methods
The training of PredLDM includes two stages. In the first stage, PredLDM learns to compress spatiotemporal sequences into latent vectors by MT-VAE to model intricate global coherence. In the second stage, CA-LDM is designed for comprehensive understanding of historical spatiotemporal content in conditions. To inference, latent noise randomly sampled and denoised with trained LDMs conditioned on historical embeddings. Predicted latent vectors are finally fed the decoder for future content.
3.1 Background on Latent Diffusion Models
Diffusion formulation. Denoising diffusion probabilistic models (Ho et al., 2020) simulate a data distri- bution x ~ (a)tt( (x) by corrupting data with progressively added Gaussian noise and learning to reverse this process. The diffusion process leads corrupted data resembling pure noise by gradually adding Gaussian noise in serious of timesteps, along with the sampled noisy at timestep t,
(xal(r-1) = N (1:5 1,11
(1)
where { are set of linearly increasing hyperparameters with the prodefined variance schodule (He et al., 2022), T denotes the number of diffusion steps and W refers to the normal distribution. The denoising process reverses the above diffusion process to predict less noisy x2-1 iteratively,
(o N (10 t), Eo(xe, f)),
(2)
where Mg and Eg are accomplished by a parameterized denoising model Eq with learnable parameters 0. Specifically, (z t) trained to prodict the noisc at each step of the diffusion process by minimizing the difference between the actual noise and predicted ones,
LDM Bro,
(o(x))()1))
(3)
where U(1, T) refers uniformly sampling from (1,...7))
Latent diffusion models. LDMs (Rombach et al., 2022) are efficient variants of DMs by operating in latent space. This process begins with a pretrained variational encoder E: x - 2, compressing the input image x ~ Pdata (1) into latent representations F(x)). Similar as Equation and Equation 2 with 2,
LLDM BEL (11
3.2 Perceptual Compression with MT-VAE
In the first training stage, we compress spatiotemporal sequences with MT-VAE. The model structure of MT-VAE is provided as Figure 1. Given a sequence Por (zo); To XxXxXXX where L, H, W and C are the temporal length, height, width and channel number respectively, the spatial convolutional encoder E' encodes zo into latent vectors 20 E(xo). 2g is taken by masked temporal self-attention modules F, extracting intricate coherent temporal reliance,
2, Est = mAtt (Norm(z.)) + 2z, 2, = (E(xo) + U),
(5)
2st FeedForward (Norm(zzt) + Est>
(6)
where U is the positional embedding obtained by convolutions of input. Insipred by scaled dot-product attention (Vaswani, 2017) and masked modeling (Xie et al., 2022; Cheng et al., 2022), mAtt(-) is to capture complex global reliance after the layer normalization. Multi-head mechanism (Vaswani, 2017) is used to project representations inlo subspaces calculated by different attention heads, the number of heads is denoted by H. The process of mAtt(-) defined as below, assuming that normalized features of z., Norm(2.) is Zin.
mAtt(Zin) M,tt(Q,,,VVA.
(Q(0), K(O,, v(o) = Zin (WiQ..)) W(K,0) w(K))
(8)
5

---

Published in Transactions on Machine Learning Research (01/2026)
@QOKOO + M) rl.
( Font
softmax
(9)
vdx
0,
if md(i)=1.
where Mj
(10)
-0,
otherwise,
(,, Concaat (=oud>
Aout wo:
(H)
(11)
where Q, K and V are queries, keys and values of vectors for dot production (Vaswani, 2017). W(Q.),
W(Ka) and W(V.)) are the paramcters of lincar operations for i-th head to control the weights of Q, and V respectively, = 1,... H. Here, the masking operated on the attention matrix via Equation by M, where we can see this masking modulating the scaled dot production between queries and keys, md(j) E {0, 132 is the random binarized output with the masking ratio T of the same size as temporal length. M, indicates all zero all negative infinite matrices corresponding to time location j. The results from different attention heads are concatenated and projected back into representation space through the weight matrix WO, Here the compressed feature 2st is accessed. The mean vectors (p((xe) and variance vectors (xstt) are predicted with learnable parameters 9, which are implemented by two dense layers. The sampled latent features from the Gaussian distribution Eet ~ N (Hppz))) Ex((sr)) are the final compressed features used for the decoder to reconstruct Fg. D accomplished by the cascaded convolutional layers and D(ag)) is expected to minimize the difference between the predicted distributions and the real data Pdtta (zr)) The training objective is the reconstruction loss with pixel-level mean-squared error (MSE) and perceptual loss (Johnson et al., 2016; Ni et al., 2023). For compression, varying spatiotemporal patterns can be accessed by our MT-VAE, as the masking operation creates more attention matrix-bascd samples along time dimensions. The attention matrix in our work is accessed from dense convolutions from previous latent input samples. By creating more intermediate attention feature samples, complex fractions of the same piece of spatiotemporal sequences can be linked tightly.
3.3 Condition-aware Latent Diffusion
In the second stage, it is expected to train denoising network to predict less noisy samples from latent noise conditioned on historical embeddings (Sohn et al., 2015; Rombach et al., 2022). Different from existing video generation rescarch, the condition of this task is raw historical spatiotemporal sequences rather than high dimensional text as conditions, while spatiotemporal sequences are full of structural details, more difficult to comprehend. Our diffusion part consists of both the commonly used conditional generation objection and importantly the probabilistic modeling of the condition itself, maximizing the mutual information beyond joint distributions of conditional generation only. Given a sequence of future observations Puraaaaa and its corresponding historical observations as conditions yo-PPatt the trained MT-VAE compresses them into the latent features respectively as Eat and Est, Eat ~ ' (Hup(2s)) Exp(s)) 2st F(E(aa)) and Ea ~ N (Mu(cax) (c,()))) Cat F(E(vo)) Our CA-LDM is trained to simultaneously learn the denoising diffusion process of (i) conditional generation on the distributions p(āselãz)) and (ii) reconstruction of the conditions on the distribution p(č,r), as in Figure 1. For pseudo-code, please refer to Algorithm 1.
Conditional generation. For learning (st); the diffusion process progressively adds Gaussian noise onto det until it resembles pure noise along with the sampled noisy Exe(t) at timestep t. The denoising process reverses the diffusson process iteratively to approach the original latent samples žzeto>- Instead of using time-condittonal U-Net (Ronneberger et al., 2015), our denoising network eo is inspired by DiTs (Ma et al., 2024). It is consisted of the spatiotemporal self-attentions 6, and cross-attentions 6,, where and T are learnable parameters. This denoising neural network is trained by minimizing the difference between the actual noise and predicted ones,
LLDM: = Esssenee [||c- (o(cu(
(12)
where t U(1 T) and e IO, The condition latent features Eet is integrated the intermediate features of eo by the cross-attention layers 67.
6

---

Published in Transactions on Machine Learning Research (01/2026)
Condition-awary reconstruction. For learning p(Cst), the diffusion process is additionally conducted onto Eat with the sampled noisy Cat(1) at timestep l. The denoising phase is operated to reverse the noisy samples to the original distribution p(Cat). 'hee training objective of this branch is to progressively reconstruct the conditions by minimizing the difference between the actual noise and predicted oncs from the denoising network with Siamese spatiotemporal self-altentions ew in eo yet without cross-allentions,
Baaee [|ee [ (o((o(((())))))))
Lcx =
(13)
The overall learning objective of CA-LDM is the combination of LLDSS and LCA,
LCA-LDM = -LDM + CCA.
(14)
Algorithm Condition-aware Latent Diffusion (CA-LDM) Training
1: Input:
2: Future observations: T0 (Tn) historical observations (condition): yo ~ Pdata (UO);
3: Pretrained network:
Pre-trained MT-VAE encoder E, transformer F, and parameter yp;
Hyperparameters:
Number of diffuuson steps T; variance schedule (8.)2=1
Initialization:
8: Initialized denoising network with parameters 0 = TW
9: procedure CA-LDM TRAINING
10: Compress observations into latent space:
11: Est = =((((=)) Cat = F(E())));
12: Sample latent features:
13:
Est (=sp)(=ss) Exp(=x)); Est ~ (N(p((Cs)) Exp(saa);;
14:
for = 1 to 7' do
15:
Sample random noise: e ~ (0, t U(1,T))
16:
Add noise to latent features:
Eat(2) = 2 + VI- 5ee; Ese(e) = Varcst + VI- ãc;; where o = II (1 -3,);
17:
18:
Compute conditional generation loss:
19:
CLDM [||e cø (E 2.
20:
Compute condition-aware reconstruction loss:
21:
LCA = E [l|c- cE(t)(
22:
Update parameters:
23:
0 + 0 +
24:
end for
25: end procedure
26: Output: Trained denoising network co
3.4 Inference
As in Figure 1, to inference the expected spatiotemporal sequence zo with the condition yo, Gaussian noise is sampled in the latent space and the condition is compressed by MT-VAE as č.t. The denoising network eg in CA-LDM is used lo predict the less noisy samples 70.7-1 within the predefined T timesteps, while the la- tent condition vectors čæt are fused across cross-aa attentttn layers into the intermediate features of eæ to accom- plish conditional controlling. The less noisy sample can be predicted by 2t-1 N (2:-1:19(2) t), Eg (z,, t)). The latent vectors 2g can be approached after T timesteps denoising. Finally, the decoder in MT-VAE decodes 2g back to the pixel space, resulting predictions close possible to ground-truth Zip

---

Published in Transactions on Machine Learning Research (01/2020)
KiniCaltech
KintCattech
KintCaltech
KiniCattech
0.2
0.15
PSN0
LP1PS
0,10
20
0.05
18-
0,5
0.00
167
0.41
0.5
10
10
10
18
Time
Time
Tiax:
T1me
MSE on ICTI
PSNR on KTII
SSIM on KTII
LPIPS on KTH
0.05
1.00
0.109
0.5-
0.3
5504
12105
6.55
0.150-
78
0.51
0/154
26-
SSTM
1.0g
10
1.7
8.9
0.25
8.0
6.5-
0.8-
6.4-
0.7
6.4
0.10
0.59
0.05
4.5-
0.5
18
Thme
Thee
U-Net
PredRNK
PredRNN'+
ConvLSTM
MetionRNN
SinVP
SinVPP2
MCVD
PVDM
Ours
Figure 2: Temporal analysis of predictive models. Rows from KittiCaltech, KTH and SEVTR respectively.
4 Experiments
4.1 Experimental Setup
Datasets and metrics. Datasets in this study include KittiCaltech (Geiger et al., 2013), KTH (Schuldt et al., 2004) and SEVIR (Veillette et al., 2020). (i) KittiCaltech is driving-scene dataset, comprising a curated collection of high-quality images. The ability to predict the future dynamics of this scenario is paramount for the advancement of autonomous driving technology and dynamic comprehension, containing 127,20 271 frames in total, with 74, 833 frames for training and 52, 438 frames for testing. benchmark the field of human action recognition and prediction. depicting a variety of human activities. This dataset comprised of 51, 360 frames with 20, 420 frames for training and 30, 940 frames for testing. (iii) SEVIR has been curated in the realms of weather sensing and short-term forecasting, comprising thousands of weather events in multipe sensor modalities. We use vertically integrated liquid (VIL) data with 5-minute interval, and 1 km spatial resolutions. They are stored as integers ranging from 0 to 254, with a value of 255 indicating missing data. In our experiments, all frames are processed as 128 x 128 resolutions. The temporal length of input is uniformly 10 frames and output length is also 10 frames. For evaluating on KittiCaltech and KTH, the metries contain MSE, PSNR, SSIM (Jin et al., 2020; Wang al., 2004) and LPIPS (Zhang al., 2018). For evaluating weather patterns on SEVIR, event-level short-term prediction metrics and the image quality assessment metric, including
(ii) KTH stands as a encompasses diverse image sequences

---

Published in Transactions on Machine Learning Research (01/2026)
KittiCaltech
KTH
SEVIR
1.5
2.5
Lower is better
Lower better
SimVP
Lower better
MotionRNN
SimVP
SimVPv2
2.0
SimVP SimVPv2
PredRNN
1.0
1.5
PredRNN+ MotionRNN
KVD
U-Net
KVD
PredRNN+
U-Net
SimVP
MCVD
0.5-
MCVD
ConVLSTM
PredRNN
PredRNN+
U-Net
ConvLSTI
MotionRNN
PredRNN
1.0
ConvLSTM
Ours
PVDM
2-
MCVD
PVDM
0.5
PVDM
Oues
Lower is better
Ours
Lower is better
Lower is better
0.0
0.0
800
1000
1200
1400
1600
1800
1000
2000
3000
4000
1000
2000
3000
FVD
FVD
Figure 3: Plots of KVD/FVD distance scores from existing models and ours three datasets. Distance scores between distributions of ground-truth data and predictions from predictive models are visualized. Both metrics are the lower the better.
Comparison between existing models and PredLDM on KittiCaltech, KTH as well as SEVIR datasots. indicates the higher the better, whereas 4 is the opposite. The best results are marked as bold and the second best ones are marked as underline.
Table 1:
KittiCaltech
KTH
SEVIR
Models
PSNR f
SSIM
LPIPS
PSNR
SSIM 1
LPIPS 4
POD 1
FAR
CSI
U-Net
19.34
0.591
0.232
26.75
0.813
0.079
0.691
0.185
0.612
PredRNN
19.19
0.616
0.190
28.33
0,860
0.057
0.704
0.174
0.629
PredRNN +++
18.99
0.571
0.244
28.33
0.861
0.058
0.714
0.171
0.638
ConvLSTM
20.46
0.652
0.154
27.87
0.859
0.052
0.754
0.182
0.660
MotionRNNN
19.79
0.621
0.170
28.94
0.868
0.051
0.742
0.177
0.654
SimVP
19.19
0.614
0.239
28.47
0.838
0.060
0.752
0.184
0.657
SimVPv2
19.29
0.620
0.234
28.64
0.847
0.061
0.713
0.163
0.641
MCVD
19.41
0.607
0.177
25.78
0.770
0.094
0.659
0.197
0.583
PVDM
19.94
0.631
0.174
26.25
0.781
0.086
0.681
0.173
0.610
PredLDM 19.86
0.653
0.107
28.94
0.871
0.045
0.760
0.148
0.672
POD (Veillette et al., 2020), FAR (Veillette et al., 2020), CSI (Schaefer, 1990) and SSIM. More details can be accessed as in Supplementary Section A.1.
Implementation details. PredLDM is trained in stages. In the first stage, spatiotemporal sequences are autoencoded by MT-VAE. The masking ratio r is set 0.6. In the second stage, trained parameters in MT-VAE including F, F and are used to project data into latent space. The denoising model is trained to predict less noisy samples by the linear combination of LLDM and LCA. More details and hyper parameters implementing our PredLDM are available as in Supplementary Section A.1.
4.2 Results with Comparison to Existing Models
Basclines in our experiments include classical encoder-forecaster architectures U-Net (Ronneberger et al., 2015), PredRNN (Wang et al., 2017), PredRNN+ (Wang et al., 2018), ConvLSTM (Shi et. al., 2015), MotionRNN (Wu et al., 2021b), SimVP (Gao et al., 2022), SimVPv2 (Tan et al., 2023) as well as diffusion- basod probabilistic generation architectures MCVD (Voleti et al., 2022) and PVDM (Yu al., 2023). As in Table PredLDM almost achieves best scores in all metrics on KittiCaltech and KTH, except in PSNR. The most shining metric is LPIPS which resembles the perceptual evaluating abilities similar to human.

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Influence of different settings of autoencoders. 1 indicates the higher the better, whereas |. is the opposite. The best results marked as bold.
KittiCaltech
KTH
Autoencoders
SSIM
LPIPS
SSIM
LPIPS
3D VAEs
0.630
0.157
0.861
0.060
2D VAEs ID Convs
0.647
0.121
0.869
0.045
MT-VAE (Ours):
0.653
0.107
0.871
0.045
Table 3: Influence of condition-aware latent diffusion. t indicates the higher the better, whereas J is the opposite. The best results are marked as bold.
KittiCaltech
KTH
LDMs
SSIM f
LPIPS 4
SSIM f
LPIPS 4
LDM LLDM
0.622
0.184
0.858
0.067
CA-LDM (Ours):
LDM LLDM + LCA
0.653
0.107
0.871
0.045
where 30.5% improvement is achieved by PredLDM on KittiCaltech and 11.8% improvement is made on KTH. Results on weather nowcasting show the large margin in scores of PredLDM over other models.
4.3 Temporal Analysis
For observing temporal detailed performance, results are compiled by performance each time point as Figure 2. It is evident that the descending curve decreases slowly for PredLDM, while the degradation of other models is quite fast. This implies that PredLDM is capable of dealing with complex global temporal variations. This phenomenon is more prominent in LPIPS on KittiCaltech and KTH.
4.4 Experiments on KVD/FVD Comparison
For further analyzing the temporal retention ability of models, we calculate Fréchet Video Distance (FVD) and Kernel Video Distance (KVD) (Unterthiner et al., 2018) between predictions and groundtruth on Kit- tiCaltech, KTH and SEVIR datasets. Different from frame-level metrics, these two metrics measure the similarity of sequential distributions. As in Figure 3, results indicate better retention abilities of visual quality in temporal dimensions.
4.5 Predictive Error Analysis
For analyzing the predictive error between existing competitive implementations and ours, select chal- lenging sample when
is driving through the corner. As in Figure 4, the visual appearance from PredLDM shares quite similar details as the ground truth while others show evident visual difference. For visualizing the accumulated error over time on these models, we count the cumulative values of all absolute errors along the x-axis direction through time. The plot shows that the accumulated error over time is smaller for predictions from PredLDM than others. For the error map calculated from the last predicted frame, the error of the results very limited from ours. It can be observed that the visual quality and accumulative error evidently improved by our model.
10

---

Published in Transactions on Machine Learning Research (01/2026)
liput Eares
Groudd puth & predietions KrrsCaltei
b)
For 18
Goond truth KiniCatech
Groud tral at KinCaatec
120
Refone
Hosnetal diniluio
100
CondaaT
Preduciona prodaced by Profl
Predicions preduced CMmLNTM
MakkeRAN
Matuoltoi
Pri
Pred
40
Brne rup of praditions from Prodl.7M
Enor rup of predictions from Corvi STM
StaVP
S
MYN
AVDM
Figure 4: Predictive error analysis. a), A challenging case KittiClatech is visualized. b), Analysis error of predictions accumulated with time. We calculate the accumulated error along with x-axis, the distribution should be all blue if there is no error accumulated. c), Error maps of the last frame. The highlighted value indicates the largest. error.
a)
VAES
VAEs
Coms
MT-VAE (Ouns)
0.3
02
0.1
b)
1.5
Latent diffission model
Mean square arror
Conditional generation
1.0
Conditieli aware latent diffusson model
( 10
Timesteps 200
Timesteps = 200
KitiCallech
with LDM with CA-LDM
Condiiion
0.5
reconstruction Conditional generation
0.0
Denoising timesteps
GT al 10 KittChltech
Timesteps = 1R with L.DM
Timesteps 1K with CA-LDM
Figure 5: Ablation study. a), Heatmaps of correlation between synthetic data v.s. real data for MT-VAE. b), Influence of condition-aware latent diffusion. Performance in denoising processes is plotted on the left and the decoded predictions are on the right.
4.6 Ablation Study
For organizing the ablation study, we firstly compile the performance on different settings of autoencoders including 3D VAEs, 2D VAEs ID convolutions and ours MT-VAE, as in Table 2. can be seen that the setting of 2D VAEs with 1D convolutional attention is more competitive than 3D VAEs, consistent with existing research (Tran et al., 2018). Our setting of MT-VAE is better than this competitive one. Then
11

---

Published in Transactions on Machine Learning Research (01/2026)
KittiCaltech
KTH
SEVIR
0.85
0.87
SSIM
0.64
SSIM
0.86
SSIM
0.6
0.63
0.85
t Larger is better
Larger better
Larger Is better
0.62
0.84
0.5
Masking ratio r
Masking ratio
Masking ratio
Figure 6: Influence of masking ratios on forccasting performance. of PredLDM KittCalltech, KTH and SEVIR reported.
SSIM scores with different masking ratios
Insi aequenees
Grondtrch prodicton in KittCahech
Groundinat prolistion KTB
Grounduui prodictions SEVIR
T-2
C- 10
2
T-6
C-8
c-
C-2
10
c-2
c-f
T-6
t-8
r-
10
r-2
c- 10
:-2
t-
10
E-6
t-
t-
10
Oary
Oure
Gee
CONLSTTM
Metiond
Pudithyy
Peet
PreRRAM
ProttW
sio
Seo SeoVph2
BleV7222
U.No
DoNe
MCVD
MCVD
MCVD
PVTM
PVDM
PVDM
Figure 7: Challenging cases on three datascts. Visualized challenging cases on KittiCaltech, KTH and SEVIR are presented. The first row refers to input and ground-truth spatiotemporal sequences, following rows indicate predictions.
we report the influence of condition-aware latent diffusion in Table 3, showing that the condition-aware reconstruction-bant based loss is beneficial.
Further study is conducted as Figure 5. Correlations are analyzed from settings of autocncoders in Table 2. It shows that our MT-VAE is better for generating realistic data, where the correlation- distributions of ours indicate the most similar behavior as realistic structures. The mean square error of sampled results during the denoising process shows that the addition of condition-aware constraint brings lower error. From the decoded visual appearance, CA-LDM can produce more realistic visual quality. More analysis is available in Supplementary Section A.2.
12

---

Published in Transactions on Machine Learning Research (01/2026)
4.7 Influence on Masking Ratios
For investigating the influence of the masking ratio r on forecasting performance on three datasets, different settings of masking ratios are experimented as in Figure 6. More results are provided as in Supplementary Section A.3. Firstly, when the masking ratio is 0, the selting refers to the masking mechanism does not work in MT-VAE and instead the non-masked transformer-based VAE compresses spatiotemporal data. Comparing against the non-masked settings, it can be witnessed that the masked attention in MT-VAE is effective in most aspects on three datasets. Mcanwhile, these results also indicate that the setting of masking ratio is preferred as 0.6 in our experiments.
4.8 Case Study
Predictions of existing models and ours PredLDM on challenging cases of three datasets can be accessed in Figure 7. It can be seen that the predictions form PredLDM not only show the realistic visual appearance, but also share the most similar movement as the ground-truth sequences. More examples are available in Supplementary Section A.10.
4.9 Discussion and Limitation
Distortion-perception trade-off. We have observed phenomenon from our experimental results especially in Table that some classical predictive models such as ConvLSTM and MotionRNN behave excellent perfor- mance in structured- based metrics PSNR and SSIM, while not good at perceptual metric LPIPS. This states the trade-off in this context. Distortion (measured by PSNR and SSIM) and perceptual quality (LPIPS) are al odds with each other. Through our experiments, it can be found out that classical deterministic models excel in PSNR/SSIM because they are optimized for pixel-level fidelity, while probabilistic ones excel in perceptual metrics because they are optimized for statistical realism and diversity of outputs. This reveals the selection preference for specific downstream usage. For example, in some accuracy required occasions for example scientific measurement, the easy-implement and accurate classical models are preferred, while for some realistic necessary usages, probabilistic modeling may be an excellent choice.
Time-evolving performance. For autoregressive models like ConvLSTM, PredRNN and MotionRNN, pro- cess generating frames tokens one at time. For sequencc-to-ssqquence models like U-Net and SimVP, the whole spatiotemporal sequence is predicted by deterministic optimization. For diffusion-based models, the denoising process runs for predefined number of steps to predict results from noise. From results in Figure 2, Figure 4 and Figure 5, it can be observed that the diffusion- based modeling is beneficial to reduce time iterative error. Besides, the proposed PredLDM can serve as a competitive bascline in this field.
Masked operations attention matrix are beneficial to temporal modeling. DiT (Ma el al., 2024) proposes two kinds of masking strategies, including simple masking from one-hot notes on time tokens and frame-level masking, which is operated on the direct pixel of frames. SVD (Blattmann et al., 2023a) additionally exploits binary mask where indicates the presence of conditioning frame and 0 that of a mask embedding the input of U-Net denoising network. We can see their masking is on the input of denoising network by concatenating a masking conditioning frame-size embedding. Make-A-Videe A-Video (Singer et al., 2022) uses the image-based masking trained for frame interpolation, still limited on data-level. MCVD (Voleti et al., 2022) masks frames two ways on both past and future frames. Different from these methods, our MT-VAE is operated on the attention matrix, espocially on the scaled dot production matrix from query and key transformers of VAE. From results in Figure 6 and Figure 10, it can be observed that our masked modeling is also beneficial against the version without masking (when the masking ratio is 0). For future research, more deep research on attention matrix-based masked modeling may be of much value.
Real-time performance. While our model demonstrates competitive performance, it is important to acknowl- edge its computational limitations. Although the model scale and computational cost are not excessive com- pared to the diffusion bascline (MCVD) and our inference speed of 1.33 seconds per sequence faster, the current setting not ideal yet for real-world fast applications satisfying real-tine sal-time usage such as autonomous driving. Specifically, generating 10 frames takes 1.331 seconds, even with the compression provided by the
13

---

Published in Transactions on Machine Learning Research (01/2026)
MT-VAE. Therefore, for practical real-time deployment, further acceleration techniques would be necessary. For offline usages like urban supervision, analyzing or weather forecasting, it may remain capable of serving.
5 Conclusion
In this study, propose spatiotemporal predictive model with LDMs called PredLDM, towards predicting the accurate and realistic future. Under consideraton of intricate global coherence and comprchensive his- tory understanding, corresponding designs are made. (i) MT-VAE is proposed to compress intricate global coherent spatiotemporal latent representations with the combination of transformers with masked attention and convolutional VAEs. (ii) CA-LDM is proposed by learning distributions of both conditional generation and condition- aware reconstruction, comprehensively understand conditions which are spatiotemporal se- quences with diverse and complex context. Through extensive experiments multiple scenarios, PMedLDM shows accurate performance and realistic appearance in predictions, revealing promising potential in future rescarch and applications.
Acknowledgments
Being supported by The National Natural Science Foundation of China (Nos. 42075139, 42077232, 42075139, 82404639); The Science and technology program of Jiangsu Province (No. BE2020082, BE2022063); Natural Science Foundation of Jiangsu Province (BK20241249); National Model 'Digital Intangible Cultural Heritage' Immersive Training Base for VET.
References
Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia Bin Huang, Jiebo Luo, and Xiaoyue Yin. Latent- shift: Latent diffusion with temporal shift for efficient text-to-video generation. ArXiv, abs/ 288/2700.08477, 2023.
Mohammad Babacizadeh,, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint arXiv:2106. 13195, 2021. 2,
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, al. ediff-i: -onttto--mmeeeee diffusion models with an ensemble of expert denoiscrs. ar preprint arXiv:2211.00 1.01124, 2022. 3
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nature, 619:533 538, 2023. 26
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Macicj Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. orXiv preprint arXiv:2311.15197, 2023a. 4, 13, 27
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolutin video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563 22575, 2023b. 3, 4
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-atleettio mask transformer for universal image segmentation. In Proceedings of the TEEE/CVF conference computer vision and pattern recognition, pp. 1290 1299, 2022. 2, 5
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint ar xrXic:1907.06571, 2019. 2,
Duolikun Danier, Fan Zhang, and David Bull. Ldmvfi: Vidco frame interpolation with latent diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1480- 2024.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding.
arXiv
preprint Xrx: 04805, 2018. 2
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780- 8794, 2021.
Michael Dorkenwald, Timo Milbich, Andreas Blattmann, Robin Rombach, Konstantinos G Derpanis, and Bjorn Ommer. Stochastic image-to-video synthesis using cinns. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 3742-3753, 2021. 2, 3
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference compuler vision and pallern recognition, pp. 12873-12883 2021. 3
Chelsea Finn, lan Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016. 1
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet better video prediction. In Proceedings of the IEEE/CVE /CV conference on computer vision and pattern recognition, pp. 3170-3180, 2022.
2, 3, 9
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The killi dataset. The International Journal of Robotics Research, 32(11):1231-1237, 2013. 2, 8, 19
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In TEEE International Conference on Computer Vision, ICCV 2017, Venice Italy, October 22-29, 2017, pp. 5843-5851. IEEE Computer Society, 2017. doi: 10.1109/ICCV. 2017.622. URL https://doi http::://ooooo///////// 10. 1109/1109CCC 2017.62222 26
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arxiv:221 13221,, 2022. 2, 3, 4, 5
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural
information processing systems, 33:6840-688 2020. 3, 5
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poolc, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition vidco generation with diffusion models. arXiv preprint orXiv:2210.02300 2022a. 3,
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633 8646, 2022b. 1, 2, 3,
Yanping Huang and Rajesh PN Rao. Predictive coding. Wiley Interdisciplinary Reviews: Cognitive Science, 2(5):580-599) 593880553,,100 2011.
Beibei Jin, Yu Hu, Qiankun Tang, Jingyu Niu, Zhiping Shi, Yinhe Han, and Xiaowei Li. Exploring spatial- temporal multi- cuiti-frrqueccy analysis for high-fidelity and ings of the IEFF/CV Conference on Compuler Vision and Pallern Recognition, pp. 4554-4563, 2020. 8
video prediction. In Proceed-
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 694-711. Springer, 2016. 6
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basee generative models. Advances in neural information processing systems, 35:26565-26577 2022. 3
15

---

Published in Transactions on Machine Learning Research (01/2026)
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXi::1412.6980 2014. 20
Yong- Hoon Kwon and Min-Gyu Park. Predicting future frames using retrospective cycle gan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1811-1820, 2019.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXi::1804.01528, 2018. 1,
Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12581
2023. 2,
Jun Hao Liew, Hanshu Yan, Jianfeng Zhang, Zhongcong Xu, and Jiashi Feng. Magicedit: High-fidelity and temporally coherent video editing. arXiv preprint arXi::2308.144 2023. 3
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumb- ley. Audioldm: Text-to-andio generation with latent diffusion models. arXiv preprint vrXi:23311195003 2023a. 3
Mingzhe Liu, Han Huang, Hao Feng, Leilei Sun, Bowen Du, and Yanjie Fu. Pristi: A conditional diffu- sion framework for spatiotemporal imputation. 2023 IEEE 39th International Conference on Data Engineering (ICDE), pp. 1927-1939. IEEE, 2023b. 3
Yuansan Liu, Sudanthi Wijewickrema, Dongting Hu, Christofer Bester, Stephen O'Leary, and James Bailey, Stochastic diffusion: A diffusion probabilistic model for stochastic time series forecasting, arXiv preprinl arXiv:2406.0m 02827, 2024a. 3
Yutian Liu, Soora Rasouli, Melvin Wong, Tao Feng, and Tianjin Huang. Rt-gen: Gaussian-based spatiotem- poral graph convolutional network for robust traffic prediction. Information Fusion, 102:102078, 2024b.
1
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint (01:0:8/01.03048, 2024. 3, 6, 13, 27
Midjourney. Midjourney. https://www http://wwwmmmdjurro..o midjournoeyyy co, 2023. 3
Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. In Proceedings of the IEEE/CVF conference on compuler vision and pattern recognition, pp. 18444- 18455,, 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Tlya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiw:2112.100 10744, 2021. 2,
Mare Oliu, Javier Selva, and Sergio Escalera. Folded recurrent neural networks for future video prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 716 731, 2018.
Op Dalle-2. httpss//oppnat..cmm//llll- 2023. 3
OpenAl. Sora. httpss////peeai...c///a/e///es/s 2024. 3
Sergiu Oprea, Pablo Martinez- Alberto Garcia- Garcia-Garcia, John Alejandro Castro-Vargas, Sergio Orts- Escolano, Jose Carcia-Rodrigues, and Antonis Argyros. A review on deep learning techniques for video prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6):2806-255 -2266, 2020. 1, 2,
16

---

Published in Transactions on Machine Learning Research (01/2026)
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Avizwadeeesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators, 2022. URL https: wttpaiiwwi......aaaaoooo..o hwtggaab//2202.. 11214. 26
William Peebles and Saining Xic. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 4195 4205, 2023. 3, 20
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXie:2307.01955 2023. 3
MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: baseline for generative models of natural videos. arXiv preprint 1XXi:1412.6604, 2014. 3
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79-87, 1999. 3
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019. 3
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on compuler vision and pattern recognition, pp. 1069,, 2022. 2, 3, 5, 6, 20
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and compuler-assisted Citereeniion-MMCCCA. 2015: 18th inter- national conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234 241. Springer, 2015. 3, 6, 9
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffuusin models with deep language understanding. Advances in neural information processing systems, 35:36479 36494, 2022. 3
Joseph T Schaefer. The critical success index an indicator of warning skill. Weather and forecasting, 5 (4):570-575, 1990. 9
Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: local sym approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3, pp. 32-36. IEEE, 2004. 2, 8, 19
Xingjian Shi, Zhourong Chen, Hao Wang, Dan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convo- lutional Istm network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems, 28, 2015. 1, 9
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-vides generation without text-video data, arXiv preprint ar Xiv:22000. 1922 2022. 2, 3, 4, 13, 27
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep condi- tional generative models. Advances in neural information processing systems, 28, 2015. 6
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXio preprint arXiv:2011.18.4 184456 2020. 3
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representa- tions using lstms. In International conference on machine learning, pp. 843 852. PML.R 2015. 3
17

---

Published in Transactions on Machine Learning Research (01/2026)
Namjoon Suh, Yuning Yang, Din-Yin Hsich, Qitong Luan, Shirong Xu, Shixiang Zhu, and Guang Cheng, Timeautodiff: Combining autoencoder and diffusion model for time series tabular data synthesizing. arXiv preprint Aiv:2240 16028, 2024. 3
Mingzhen Sun, Weining Wang, Xinxin Zhu, and Jing Liu. Moso: Decomposing motion, scene and object for video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 18727-18733 2023. 2, 3
Cheng Tan, Siyuan Li, Zhangyang Gao, Wenfei Guan, Zedong Wang, Zicheng Liu, Lirong Wu, and Stan Z Li. Openstl: A comprchensive benchmark of spatio-temporal predictive learning. Advances in Neural Information Processing Systems, 36:69819-69831, 2023. 2, 3,9 9
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEFE international conference on computer vision, pp. 4489 4497, 2015.
Du Tran, Heng Wang, Lorenzo Torresani, Jamic Ray, Yann LcCun, and Manohar Paluri. A closer look spatiolemporal convolutions for action recognition. In Proceedings of the IEEE conference Compuler Vision and Pattern Recognition, pp. 6450-6459, 6459, 2018. 4, 11
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1526-1535, 2018. 2,
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Syl- vain Gelly. Towards accurate generative models of video: A new metric & challenges, orXiv preprint arXiv:1812.01111 2018. 10
Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, Karsten Krcis, et al. Lion: Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems, 35: 10021-10039, 2022.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. 3
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 5, 6
Mark Veillette, Siddharth Samsi, and Chris Mattioli. Sevir: A storm event imagery datasct for deep learning applications radar and satellite meteorology. Advances in Neural Information Processing Systems, 33: 22009 22019, 2020. 2, 8, 9, 19
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lec. Decomposing motion and content for natural video sequence prediction. arXiv preprint ar Xiv:: 706. 706.08033, 2017. 2,
Ruben Villegas, Arkanath Pathak, Harini Kamnan, Dumitru Erhan, Quoc V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. Advances in Neural Information Processing Systems, 32, 2019. 2, 3
Vikram Voleti, Alexia Jolicoeur-Martinene Martineau, and Chris Pal. fcvd-maaked conditional video diffusion for prediction, generation, and interpolation. Advances in neural information processing systems, 35:23371- 23385, 2022. 3, 9, 13, 27
Jiuniu Wang, Hangjee Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXio preprint orXiv:2308.06371 2023. 3,
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal Istms. Advances in neural information processing systems, 30, 2017. 2, 3,
18

---

Published in Transactions on Machine Learning Research (01/2026)
Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and S Yu Philip. Predrmn++: Towards resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In International conference on machine learning, pp. 5123 5132. PMLR, 2018. 2, 3,
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, (3(4):00 612, 2004.
Bohan Wu, Suraj Nair, Roberto Martin-Martin, Li Fei-Fei, and Chelsea Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2318 2328, 2021a. 2,
Haixu Wu, Zhiyu Yao, Jianmin Wang, and Mingsheng Long. Motionrnn: A flexible model for video prediction with spacetime-varying motions. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 15435 15444, 2021b. 2, 3,9
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the TEEE/CVT conference on compuler vision and pattern recognition, pp. 9653 9663, 2022. 2, 5
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104, 10157, 2021.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.0000 2024. 3
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18456 18466, 2023. 9
Hengrui Zhang, Jiani Zhang, Balasubramaniam Srinivasan, Zhengyuan Shen, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis. Mixed-type tabular data synthesis with score-basod diffusion latent space. arXiv preprint arXiv:2310.0900 09666, 2023a. 3
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586 595, 2018.
Yuchen Zhang, Mingsheng Long, Kaiyuan Chen, Lanxiang Xing, Ronghua Jin, Michael I Jordan, and Jianmin Wang. Skilful nowcasting of extreme precipitation with nowcastnet. Nature, 619(7970):5266 532, 2023b. 1
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizh Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint orXiv:2211.1101 2022. 3, 4
Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffusion: Consistent self-attention for Long-range image and video generation. arxiv preprint arXio:2405.01433 2024. 3
A Appendix
A.1 More Details on Data and Implementation
Data. Datasets in this study include driving sccne-related dataset KittiCaltech (Geiger et al., 2013), human action-related dataset KTH (Schuldt el al., 2004) and weather pattern-related dataset SEVIR (Veillette et al., 2020). (i) KittiCaltech dataset a cornerstone in the domain of computer vision, serving as an essential resource for autonomous driving research. It compriscs a curated collection of high-quality images that are vital for the understanding of driving scenarios. The ability to predict the future dynamics of these scenarios is paramount. for the advancement of autonomous driving technology, rendering this dataset
19

---

Published in Transactions on Machine Learning Research (01/2026)
exceptionally valuable for research in vision field, particularly in the areas of future scenario prediction and dynamic comprehension. This dataset is meticulously organized, consisting of total of 127, 271 frames. Within this collection, 74, 833 frames are allocated for training purposes, while 52, 438 frames are reserved for testing. This structured distribution ensures a comprehensive framework for both the development and validation of autonomous driving algorithms. (ii) KTH dataset stands as a benchmark in the field of human action recognition and prediction. It encompasses a diverse array of image sequences depicting a varicty of human activities, such as walking, jogging, running, boxing, waving, and clapping, totaling six distinct categories, capturing the intricacies of different individuals performing various actions. Comprising total of 51, 360 frames, the KTH dataset is segmented into 20, 420 frames for training and 30, 940 frames for testing. To maintain uniformity in evaluation, all frames are centrally cropped and resized to a consistent dimension of 128 x 128 pixels. The dataset's 's processing protocol specifies that the input consists of 10 frames, with the output also comprising 10 frames, ensuring a standardized framework for analysis and comparison. (iii) SEVIR datasct has been curatod to accelcrate research in the realms of weather sensing, avoidance and short-term forecasting. This comprehensive collection comprises thousands of weather events with each represented 4-hour sequence. Researchers are empowered to synthesize and harmonize diverse weather sensor data into unified dataset through SEVIR. The dataset encompasses variety of sensor modalities, including IR069 (infrared satellite imagery at 6.9 m), IR107 (infrared satellite imagery at 10.7 m), VIL (vertically integrated liquid), and L.GTT (iig) In this study, we use VIL modality. The VIL data is derived from NEXRAD radar mosaics, featuring a 384 x 384 pixel resolution, a 5-minute interval, and a 1 km spatial resolution. The geographically and chronologically aligned imagery, depicting a spectrum of weather events including high winds, tornadoes, and hail, is captured by GEOS-16 satellites and NEXRAD weather radars. This data is publicly available in HDF files, we convert them into recordings of images. The pixel values within these spatial grids correspond to processed statistics derived from the actual sensor readings. VIL images are stored as integers ranging from 0 to 254, with a value of 255 indicating missing data. All frames have been reprocessed to a 128 x 128 resolution. The temporal length of input is uniformly 10 frames and output length is also 10 frames.
Implementation Details. The training of PredLDM is comprised of two stages. In the first stage, spa- tiotemporal sequences are autoencoded by our MT-VAE with the loss CAE. The ADAM optimizer (Kingma, 2014) with constant learning rate of le - 4 is used. The batch size for training is set to be 4 and the number of total epochs is 100. The pretrained weights from inage-based 2D VA (Rombach et al., 2022) used for image synthesis are employed for the initialization of the convolutional encoder and decoder, where the loaded weights of E, D stay fixed during training. The parameters of the transformer F' and dense layers & are updated in this stage. No weight decaying schedule and data augmentation is used. For the designed temporal self-attention architecture, the number M of stacked attention layers is set to be 8 and the hidden dimension is 128 in this study, The masking ratio r is set as 0.6. In the second stage, the trained parameters in MT-VAE including E, F and yp are used to project data into latent space. In latent space, the denoising model is trained to predict less noisy samples by the 1:1 linear combination of LLDM and LCA. The diffusion transformer (DiT) structure (Peebles & Xic, 2023) is used for constructing the denoising model with cascaded transformers. The ADAM optimizer with the same learning rate of le -44 4 is used. The batch size is set to be 4 and the total epochs are 100. The parameters of the denoising model are the only learnable parameters in this stage. There are also no special weight decaying and augmentation schedules usod. The number N of cascaded DiTs is set to be 32 and the hidden size is 1152. These two stages are both performed on NVIDIA GeForce RTX 4090 with 24 GB x4. To inference with our PredLDM, the process is conducted and the timesteps T for denoising is 1000.
A.2 Additional Analysis on Critical Designs
For additional study of critical designs on KTH and SEVIR datasets, different settings of autoencoders including 3D VAEs, 2D VAEs + 1D convolutions and our MT-VAE are evaluated by pair-wise column correlation between synthetic data and real data. The value in heatmaps of correlation indicates the absolute divergence, where the more red area means the better correlation to real distributions. Meanwhile the performance calculated from predicted less noisy samples and ground truth during denoising processes with the trained denoising network with or without aadition-awarre reconstruction based constraints is plotted. The decoded predictions at different timesteps on these two datasets are also provided in this section.
20

---

Published in Transactions on Machine Learning Research (01/2026)
3D VAFs
VAFs ID Convs
MT-VAE (Ours)
0.6
0.4
0.2
0
b)
0.08
Latent diffusion model
Mean square error
Conditional generation
0.06
Conditton- latent diffusion model
Timesteps 200 GT at t 10
Timesteps 200
0.04
on KTH
with L.DM
with CA-LDM
-onddiiio--aaaee
reconstruction + Conditional generation
0.02
200
400
600
800
1000
Denoising timesteps
GT at t 10 Timesteps 1K with LDM
Timesteps = 1K with CA-LDM
on KTH
Figure Additional ablation study on KTH. a), Hcatmaps of correlation between synthetic data v.s. real data for MT-VAF. b), Influence of condition-aware latent diffusion. Performance in denoising processes is plotted the left and the decoded predictions the right.
For analysis on KTH dataset, the results can be seen as in Figure 8. As the directions of human movement in this dataset are always vertical or horizontal and close to the center of images, the heatmaps of correlation reflect this charactcristic. Results show that the synthetic distributions MT-VAE are the closest to the real dist ribution. The setting of 2D VAFs + convolutional temporal layers follows our setting and the setting of 3D VAEs behaves not competitive. This phenomenon shows that our MT-VAB is better compressing spatiotemporal content into latent space with realistic distributions and our setting is effective. From the mean square error of sampled results during the denoising process, it is evident that the condition-awaad reconstruction of CA-LDM is beneficial as the error of predictions from CA-LDM is much lower than the trained denoising model with only the original conditional generation related constraint. For the decoded visual results from timesteps at 200 and 1, 100, the decoded visual quality from CA-LDM better than the LDM setting and more timesteps are more conducive to producing detailed realistic visual appearance.
For analysis on SEVIR dataset, results are available as in Figure 9. The evolution of observations related to vertical liquid precipitation captured by the weather radar is distributed diversely in terms of physical geographic space, resulting the predicted dynamics difficult to be similar ground truth, so heatmaps of correlation here are distributed in disorderly manner. Results show that our setting of MT-VAE is still the best choice compared to other two seltings, with highest relation to the distribution of real data. This again reveals that our MT- VAE better at handling perceptual autoencoding. From the mean square error of sampled results during denoising processes, it can be seen that the false alarm rate of our predictions is much lower than predictions from the denoising model without condition-awan aware constraints. For the decoded visual results from different timesteps, the decoded predictions in this dataset reveal more accurate dynamics
21

---

Published in Transactions on Machine Learning Research (01/2026)
a)
3D VAEs
2D VAEs ID Convs
MT-VAE (Ours)
0.5
0.4
0.3
0.2
best Bast
0.1
b)
0.5
Latent diffusion model
False alarm rate
Conditional generation
0.4
Condition-wwaee latent diffusion model
GT at t 10 on SEVIR
Timesteps 200 with L.DM
Timesteps 200 with CA-LDM
0.3-
-onndiiion-wwaee
reconstruction + Conditional generation
0.2
200
400
600
800
1000
Denoising timesteps
GT at t 10 on SEVIR
Timesteps 1K with LDM
Timesteps = 1K with CA-LDM
Figure 9: Additional ablation study on SEVIR. a), Heatmaps of correlation between synthetic data v.s. real data for MT-VAF. b), Influence of condition-aware latent diffusion. Performance in denoising processes is plotted on the left and the decoded prodictions are on the right.
of heavy precipitation and realistic distributions of CA-LDM. Besides, realistic visual appearance with more details can be accessed by more denoising timesteps.
Table 4: Sensitivity analysis on the weighting scheme between conditional generation and condition-awand reconstruction on KittiCaltech. indicates the higher the better, whereas is the opposite. The best results are marked as bold.
LLDM
Lea
SSIM
LPIPS 4.
SSIM 1
LPIPS J
0.10
0.90
0.169
19.18
0.606
0.252
0.25
0.75
0.132
19.75
0.649
0.120
0.50
0.50
0.092
19.86
0.653
0.107
0.75
0.25
0.138
19.68
0.641
0.176
1.00
0.00
0.155
19.65
0.622
0.184
A.3 More Results on Influence of Masking Ratios
To supplement the results in checking influence of masked modeling, we provide more results as in Figure 10. These results show the similar phenomenon as in experiments the manuscript that the masked attention is necessary our pipcline and the preferred setting 0.6, although more settings like 0.65 are not tested, meaning more performance gain may lie in this ratio. Besides, for recommendations on tuning for new datasets, suggest directly using medium ratio the first choice, evidenced by good performance in
22

---

Published in Transactions on Machine Learning Research (01/2020)
KittiCaltech
KittiCaltech
KittiCaltech
0.115
MSE
PSNR
19.5
LPIPS
0.110
0.105
Lower is better
Larger is better
Lower Is botter
19.0
0.100
Masking ratio r
Masking ratio r
Masking ratio r
KTH
KTH
KTH
0.020
29.5
0.055
0.015
29.0
0.050
MSE
0.010
PSNR
LPIPS
28.5
0.045
0.005
Lower better
Larger Is better
Lower is better
0.000
28.0
0.040
0.6
0.8
0.3
0.6
0
0.3
0.8
Masking ratio
Masking ratio
Masking ratio
SEVIR
SEVIR
SEVIR
0.80
0.20
0.70
0.78
0.68
0.76
0.66
POD
FAR
0.15
0.74
0.64
Larger
Lower better
Larger
0.60
0.3
0.6
0.8
0.3
0.6
0.8
0
0.3
0.6
0.8
Masking ratio
Masking ratio
Masking ratio
Figure 10: More results on influence of masking ratios. Remaining metrics on three datasets are presented.
Table 5: Computational cost on FLOPS, model parameters, inference time, training time and GPU memory.
FLOPs (G)
Params (M)
Inference time (s/seq)
Training time (GPU hours)
GPU memory (GB)
ConvLSTM
129.64
2.02
0.052
9.2 (1) / 2.3 (4)
2.1
PNedRNN
243.06
7.25
0.495
84.1 (1) 21.0 (4)
4.3
MCVD
5,434288
367.60
1.620
646.1 (1) 161.5 (4)
8.9
Ours
1,566.57
421.26
1.331
530.4 (1) 132.6 (4)
14.8
experiments three different-occasiion datasets. The alternative choice may be larger one like 0.8, because for some simple occasions like KTH datasets large ratio is still beneficial.
A.4 Sensitivity Analysis on Weighting Scheme of Diffusion-based Loss
To investigate the scnsitivity on the weighting scheme of diffusion-based loss functions, we conduct an additional experiment on weights of tradit diffusion-based conditional generation loss and condition- aware reconstruction-based loss. Results are reported in the Table 4. It needs to mentioned that the setting for the conditional generation must not be 0, as this loss critical the task future prediction, so the minimum setting of this loss is 0.1. From experimental results on KittiCaltech, it appears that the setting of condition-aware loss is necessary. When the weight of Le becomes larger, scores KittCClatech
23

---

Published in Transactions on Machine Learning Research (01/2026)
framel -framee1
frame11 11 frame20
Input
Ground truth & Our predictions
framel
frame10
/rame15
/rane20
frame300 frume22
frame35
frume40
Input frames
Predictor
Pred.
Pred. frames
Prodictor
Pred. frames
Our predictions
Trame21-framee
PVDM
Prod.
Predictor
frumes
frumes
SimVPv2
feame21-rrame0
frame31-freen 31-fmmm60000
c)
d)
Degradation on MSE
Degradation PSNR
MSE Sthv2 Longer Horizons
PSNR Sthv2 Longer Horizons
0.3-
Ours
PVDM SimMPP22
NS0.
PONR
MSE
Ours PVDM SiIVPy2
20m
0.0
15
Lengn-2 Langth-30
Length+30
Time
Time
Degradation SSIM
Degradation LPIPS
Sthv2 Longer Horizons
LPIPS by Longer Horizons
1.0
Our
PVDM
0.2
SSIM
0.8
SiiVPy2
0.18-
6.7-
0.0-
Ours
0.6-
0.05
0.4
0.0
0.5
0.08
Time 30
0
10
15
20
Lenghwt0 Langth+20 Lengthw30
Langh-10 Langthe20 Lengthe50
Time
Figure 11: Additional analysis of our model at longer horizons and larger resulutions. Evaluation of our model on Sthv2 dataset is conducted the inference manner in a), iteratively predicting future frames. b), Predictions of our model over longer time visualized. Degration of preformance for model is plotted as in c), where MSE, PSNR, SSIM and LPIPS evaluated. d), Detailed comparison between competitive baselines and ours by longer horizons.
are more competitive. Besides, the setting of LLDM should be very small as this the basic objective in spatiotemporal predictive tasks. It indicates that the linear combination of these loss functions satisfied with the performance gain for common occasions.
A.5 Analysis on Computational Cost
We report scores of ours and some baselines for better showing the cost as in below Table 5. Results show that the model scale and computational cost of classical predictive methods like ConvLSTM and PredRNN are attractive. But these two cost-related scores of ours are not very expensive compared the diffusion baseline. Our inference speed is faster than MCVD, processing one sequence in 1.33 seconds. For training architecture, firstly the MT-VAE is trained from scratch with around 39.3 GPU hours, around 9 hours by 4 Nvidia GeForce 4090 GPUs. For the diffusion process, i.c., the second training stage may cost 530.4 GPU hours, around 5.5 days with 4 Nvidia GeForce 4090 GPUs. When inferencing one sequence, 14.8 GB GPU memory needed, can be easily deployed on Nvidia GeForce 4090. Considering the compiled data here, we think the current sctting not quite real-time, where PredLDM produces 10 frames 1.331 seconds, even with the compression ability from MT-VAE. If considering real-time application, acceleration skills are still needed.
24

---

Published in Transactions on Machine Learning Research (01/2026)
Input frames
Ground truth predictions on KTH
1 ==
10
t=1
r
==3 3
t=4 4
t
5
t=6 6
t
7
8
9
10
Input Noise Perturbations @ 0-0.00
Ours
PVDM
VFv2
Input Noise Perturbations @ a 0-0.5
Ours
PVDM
VPv2
Input Noise Perturbations @ 0=0.10
Ours
PVDM
Sim VPv2
Input Noise Perturbations @ a-0.20
Oun
PVDM
SimVPv2
Figure 12: Robustness analysis by adding noise input frames. Noise with different levels of standard deviation is added onto the input sequences. Predictive models including PVDM, SimVPv2 and ours are used to produce future sequences.
25

---

Published in Transactions on Machine Learning Research (01/2026)
Table 6: MSE/MAE comparison between existing models and PredLDM on KittiCaltech, KTH as well
SEVIR datascts. indicates the higher the better, whereas & is the opposite. The best results are markod
bold and the second best ones marked underline.
KittiCaltech
KTH
SEVIR
Models
MSE +
MAE
MSE 4
MAE
MSE 4
MAE
U-Net
0.158
1.631
0.023
0.214
0.055
0.026
PredRNN
0.166
1.574
0.018
0.188
0.0488
0.021
PredRNN+
0.166
1.843
0.018
0.199
0.045
0.020
LSnVLSTM
0.150
1.482
0.023
0.211
0.053
0.026
MotionRNN
0.151
1.605
0.015
0.168
0.042
0.023
SinVP
0.160
1.861
0.016
0.141
0.040
0.017
SimVPv2
0.159
1.435
0.015
0.178
0.041
0.020
MCVD
0.125
1.277
0.027
0.277
0.068
0.036
PVDM
0.114
1.027
0.023
0.223
0.051
0.024
PredLDM
0.092
1.477
0.013
0.147
0.034
0.019
Ours (N=100)
0.16
PVDM (N=100)
MCVD (N- 100)
0.14
Uncertainty score
0.12
0.10
0.08
Resid Vals flog)
0.06
0.04
-0.5
0.0
0.5
1.p
20
40
60
100
Sample size
Figure 13: Uncertainty quality is quantified by standard deviation of ensemble predictions. 100 predictions are made by experimental models.
A.6 Inference by Higher Resolutions and Longer Horizons
To evaluate by higher resolutions and longer horizons, adopt iterative inference strategy (Pathak et al., 2022; Bi et al., 2023). Given by 10 input frames, the model predicts the next 10 frames, which are then recursively fed back as input for subsequent predictions, as illustrated in Figure 11. Experinental results on the Sthv2 (Goyal et al., 2017) dataset where frames are processed at 256 x 256, revealing an acceptable decline in visual quality over extended time horizons, which is corroborated by a gradual degradation in quantitative metrics, including MSE, PSNR, SSIM, and LPIPS. From detailed temporal results compared by the competitive diffusion baseline PVDM and the deterministic baseline SimVPv2, our PredLDM shows better performance towards this experimental setting.
26

---

Published in Transactions on Machine Learning Research (01/2026)
A.7 More Discussion MT-VAE and CA-LDM
Firstly, we discuss more the masked attention in MT-VAF,, related with masking methods compared to DIT (Ma et al., 2024), SVD (Blattmann et al., 2023a), Mlake-A-Videe (Singer et al., 2022) and MCVD (Voleti et al., 2022). DiT (Ma et al., 2024) proposes two kinds of masking strategy noted by Only Video and Temporal Mask. For the first masking, they use simple masking from O or 1 on time tokens. For the second masking, they use frame-level one. Masking modeling is operated on the data space, on temporal tokens or the direct pixel of frames. SVD (Blattmann et al., 2023a) additionally exploits a binary mask where 1 indicates the presence of a conditioning frame and refers to a mask embedding on the input of U-Net denoising network. We can see their masking is on the input of denoising network by concatenating masking conditioning framo-size embedding. Make-A-Vided (Singer et al., 2022) uses the imago- masking trained for framn interpolation, their masking is on data-level. MCVD (Voleti el al., 2022) masks frames in two ways including masking future/pass frames, masking both past and future frames. It directly masks historical frames or future frames in data space. Different from these methods, our MT-VAE is operated on the attention matrix, especially on the scaled dot production matrix from query and key in transformers of VAF. Secondly, more discussion conducted on CA-LDM. From many classical generation or prediction methods (Ma et al., 2024; Blattmann et al., 2023a; Singer et al., 2022; Voleti et al., 2022), they can all refer to the main architectures DDPM or DDIM. For ours, architectural novelty may be the CA-LDM by additional condition-aware reconstruction of historical observations. Another diffusion denoising network is also used to probabilistically modeling only on the condition, i.c., historical spatiotemporal sequences. Our diffusion part design consists both the commonly used conditional generation objection and importantly the probabilistic modeling of the condition itself. The reason we design this process in the denoising diffusion process is that different. from existing T2V research, the condition of this task raw historical spatiotemporal soquenees while T2V methods use text as conditions, where text is high dimensional and our condition is more low-level and with plentiful structural details, more difficult to comprehend. So an additional diffusion denoising process is designed in our PredLDM.
A.8 Robustness Analysis by Adding Noise
To substantiate the model's reliability real-world conditions where input sequences may be noisy, we perform the robustness analysis. Our evaluation introduces additive Gaussian noise with zero mean and controlled standard deviation at 0.00, 0.05, 0.10 and 0.20. To assess generalization under distribution shift, we conduct on Sthv2 by PVDM, SimVPv2 and our PredLDM. As illustrated in Figure 12, our model exhibits markedly superior output consistency compared to baselines across all noisy levels. These results provide superior empirical robustness for our model.
A.9 Additional Results on MSE/MAE and Quantified Uncertainty
To enhance comparability, we provide more results on experimental analysis in MSE/MAE as Table 6. Fur- thermore, for our probabilistic model PredLDM,, we provide a quantitative uncertainty analysis implemented by predictive standard deviation of ensemble predictions to rigorously evaluate its trustworthiness in crit- ical applications on SEVIR in Figure 13. Results show the lower ensemble standard deviation primarily indicating higher predictability and lower intrinsic uncertainty for specific forecast cases.
A.10 More Challenging Cases
Additional challenging cases visualized with predictions by predictive models KittiCaltech, KTH and SEVIR datasets. As in Figure 14 and Figure 15, the challenging cases KittiCaltech are presented. The predictions show the accurate modeling capability in spatiotemporal variations on car driving scenes, with realistic visual appearance compared to other models where our predictions are of high fidelity and results of other models are in more blur over time. As in Figure 16 and Figure 17, the challenging cases on KTH dataset are available. In these challenging cases, we can witness that many models are in trouble of handling these cases, where small parts of image sequences appear significant variations of motion. Predictions from existing models are highly likely to generate blur, while the predictions of PredLDM are quite attractive
27

---

Published in Transactions on Machine Learning Research (01/2026)
Input frames
Ground truth & predictions on KittiCaltech
t= t= 10
t=1
= 2
t= 3
t=4 4
t=
t=6
t t=
t=8
t= 9
t= 10
Oury
ConvLSTM
MotionRNN
PredRNN
PredRNN++
Sim VP
SimVPv2
U-Net
MCVD
PVDM
Figure 14: Challenging cases on KittiCaltech dataset. The first row refers to input spatiotemporal sequences and ground truth. The second row indicates sequences predicted by our PredLDM. Following rows are predictions from other models.
that they still appear realistic even to the last predicted frame. Besides, the better accuracy of predicted movement in these cases is evident for our model. As in Figure 18 and Figure 19, the challenging cases on SEVIR dataset are given. In weather pattern-related examples, existing predictive models are still likely to produce precipitation values with rough geophysical details, where the locations with high probabilities of rainfall are easy omit, leading high risk for social failure preventing natural disasters. However, results from PredLDM, predicted locations of hcavy precipitation more accurate and the overall distribution is more alike to the ground truth.
28

---

Published in Transactions on Machine Learning Research (01/2026)
Input frames
Ground truth & predictions on KittiCaltech
t= t=4
5
==
t=
10
Ours
ConvLSTM
MotionRNN
PredRNN
PredRNN++
Sim VP
Sim VPv2
U-Net
MCVD
PVDM
Figure 15: Challenging cases on KittiCaltech dataset. The first row refers input spatiotemporal sequences and ground truth. The second row indicates sequences predicted by PredLDM. Following rows are predictions from other models.
29

---

Published in Transactions on Machine Learning Research (01/2026)
Input frames
Ground truth & predictions on KTH
[=1 t=
t=1
t= 2
== 3
t= 4
t=
t ==6
t=
t= 8
t= 9
t= 10
Ous
ConVLSTM
MotionRNN
PredRNN
PredRNN++
Sim Vp
VPv2
U-Net
MCVD
PVDM
Figure 16: Challenging cases KTH dataset. The first row refers input spatiotemporal sequences and ground truth. The second row indicates sequences predicted by our PredLDM. Following rows are predictions from other models.
30

---

Published in Transactions on Machine Learning Research (01/2026)
Input frames
Ground truth & predictions on KTHI
|
t=1
t= 10
t= t= 2
t= 3
t= 4
t= 5
t = 6
t= 7
t=
t= 9
t= 10
d
A
1
1
Ours
{
ConvLSTM
MotionRNN
A
A
A
A
A
4
4
PredRNN
d
d
PredRNN++
d
SimVP
A
A
Sim VPv2
U-Net
MCVD
i
PVDM
Å
Å
Figure 17: Challenging cases on KTT dataset. The first row refers to input spatiotemporal sequences and ground truth. The second row indicates sequences predicted by our PredLDM. Following rows are predictions from other models.
31

---

Published in Transactions on Machine Learning Research (01/2026)
Input frames
Ground & predictions on SEVIR
+
t=1
t=
10
t=1
[ 2
t=
t=
t=
= 9
t= 10
Ours
ConVLSTM
MotionRNN
PredRNN
PredRNN++
SimVP
Sim
U-Net
MCVD
PVDM
Figure 18: Challenging cases on SEVIR dataset. The first row refers to input spatiotemporal sequences and ground truth. The second row indicates sequences predicted by our PredLDM. Following rows are predictions from other models.
32

---

Published in Transactions on Machine Learning Research (01/2026)
Input frames
Ground truth & predictions SEVIR
*
t=1
t=
10
==1
t= 2
= 3
t= 4
t=
5
t=6
t=
=
t= 9
[= 10
Ours
ConVLSTM
MotionRNN
PredRNN
PredRNN++
SimVp
Sim VPv2
U-Net
MCVD
PVDM
Figure 19: Challenging cases on SEVIR dataset. The first row refers to input spatiotemporal sequences and ground truth. The second row indicates sequences predicted by our PredLDM. Following rows are predictions from other models.
33


Reviews:
[{"review_id": "Xx8gBW03xU", "reviewer": "Reviewer_Wx32", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents PredLDM, a video prediction model based on latent diffusion models (LDMs). Specifically, they improve the current practices of the two-stage framework---(1) VAE training and (2) diffusion model training as follows. For (1), the authors propose MT-VAE, which uses random latents masking during training, enabling the latent space to better capture temporal dynamics compared with naive VAE training. They also add transformer blocks between the CNN encoder and decoder to achieve it. For (2), the authors propose to learn both the conditional generation and condition-aware reconstruction, which also models p(c) as well as the conditional distribution p(z|c). The authors argue that this strategy enables the model to comprehensively understand the temporal dynamics in the condition.", "claims_and_evidence_yn": "Yes", "claims_explanation": "(Strengths)\n- Overall, the paper is generally well-written and easy to follow.\n- The proposed components (e.g., masking and condition reconstruction) are validated through ablation studies.\n- The paper tries to conduct the claim on various datasets, e.g., KittiCaltech, KTH, and SEVIR.\n- The paper tries to include many baselines.\n\n(Weaknesses)\n- The paper only deals with fine-grained datasets and does not include results on more complex datasets such as UCF-101 and Kinetics, which have been commonly used datasets in video prediction literature. I think adding these results is important to show the scalability of the proposed method with respect to the data complexity.\n - The paper deals with relatively short videos and low resolution: 128x128, and the target is to predict 10 frames. It would be great if the proposed method could be scalable to longer and higher-resolution video sequences.\n- By the definition of a model, the model should be able to predict longer videos by conditioning the model on previously predicted video frames.  I believe trying this would greatly improve the quality of videos.", "audience_yn": "Yes", "audience_explanation": "- I think jointly modeling p(c) along with p(z|c) would be something new, and some readers might be interested.\n- It might be somehow related to recent diffusion-forcing and similar approaches.", "requested_changes": "I would request the authors to add the following changes: \n- Results on more complex datasets such as UCF-101 and Kinetics.\n - Higher resolution (at least 256x256) and longer video sequences (at least predicting 16 frames).\n- The qualitative/quantitative results of the model predicting longer sequences with rollout\n- Figure 2: I recommend that the authors add a legend globally, not just in each plot.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AzInLPEgw5", "reviewer": "Reviewer_54ps", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces PredLDM, a latent diffusion model approach for the spatiotemporal sequence prediction tasks. The core idea is a two-stage framework: first, a masked-attention transformer VAE (MT-VAE) compresses input video frames into a latent representation, capturing complex global spatiotemporal patterns; second, a condition-aware latent diffusion model (CA-LDM) generates future frames in latent space while explicitly modeling the given historical frames for better context understanding. In other words, the method leverages transformers with masked modeling to encode global coherence in the latent space and employs a diffusion process conditioned on past frames to ensure a comprehensive understanding of the history. The authors evaluate PredLDM on three diverse datasets (driving scenes from Kitti-Caltech, human actions from KTH, and weather radar data from SEVIR), demonstrating that it produces more realistic and accurate future frame predictions than a range of baseline models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "There are the following strengths of this manuscript. \n\n- **(S1) Interesting and practical design:** The proposed PredLDM sensibly combines modern generative modeling techniques. Using a transformer-based VAE with masked self-attention allows the model to learn a rich latent space encoding of video frames, overcoming the locality of pure convolutional encoders. The condition-aware diffusion process is cleverly formulated: during diffusion, the model not only learns to predict future frames from noise but also to reconstruct the conditioning frames in latent space.\n\n- **(S2) Competitive comparison performances:** Firstly, the paper’s experimental section is comprehensive and well-designed to validate the claims. PredLDM demonstrates SOTA or competitive results on all evaluated datasets, indicating the method’s effectiveness. It consistently outperforms classical video prediction models and even recent diffusion-based models (MCVD, PVDM) in most metrics. Meanwhile, the model excels in the perceptual quality of predictions. Another impressive aspect is the temporal consistency of PredLDM’s outputs: the authors show that as one predicts further into the future, the degradation in PredLDM’s prediction quality is more gradual than for other models. Moreover, the paper includes extensive ablation studies and analyses.\n\n- **(S3) Presentation:** The manuscript is generally well-written and organized, which is easy to follow. The inclusion of a pipeline diagram (Figure 1) and other visualizations (e.g., qualitative examples, error heatmaps) assists in understanding the model architecture and results. Also, the paper adheres to a professional structure and uses clear notation. Apart from a few minor language issues (addressed below), the writing quality is sufficient for conveying the complex methodology.", "audience_yn": "Yes", "audience_explanation": "The researchers in spatiotemporal prediction and video generative modeling will value a clear latent‑diffusion baseline with masked‑attention compression. The ablations and the condition‑aware objective provide actionable insights for building conditional diffusion predictors in practice.", "requested_changes": "There are the following weaknesses of this manuscript, which could be clarified or improved during the rebuttal period.\n\n* **(W1) Practical efficiency is under‑analyzed for a forecasting method.** Training uses frozen 2D‑VAE enc/dec, learned temporal transformer, a large DiT denoiser (N=32, hidden size 1152), and T=1000 sampling steps on 4 GPUs; yet there is no report of runtime, throughput, memory, or step ablations, which limits deployment insights.\n\n* **(W2) Innovation is largely compositional rather than conceptual.** The approach combines masked modeling + transformer compression with latent diffusion and cross‑attention conditioning—all well‑established pieces. The “condition‑aware reconstruction” is a neat regularizer, but the paper does not position it against closely related diffusion‑for‑prediction works (e.g., MCVD, PVDM) beyond numeric gains, leaving the core novelty somewhat incremental.\n\n* **(W3) Limited scale (resolution/horizon) and missing robustness.** All experiments run at 128×128 with 10‑step horizons; there is no test at higher resolutions or longer horizons, and no analysis of robustness (e.g., noise). This reduces evidence for applicability in realistic settings.\n\n* **(W4)** Minor issues of writing. The manuscript is generally clear but has colloquial phrasing and minor typos; a careful pass is needed, e.g., wording around LPIPS highlights. Explanations of “condition‑aware reconstruction” could be tightened with a small schematic or pseudo‑code in the main text.\n\n* **(W5) Additional issue after rebuttal.** I suggest the authors provide the classical metrics (i.e., MSE and MAE) used in previous video prediction works (e.g., OpenSTL benchmark). The current manuscript evaluates the proposed KVD and FVD with PSNR, SSIM, and LPIPS. These metrics are comprehensive but might not be friendly to previous video prediction methods like PredRNN and SimVP variants. Meanwhile, the authors could also add probabilistic evaluation and calibration analysis, especially for risk-sensitive domains. Given that diffusion models are inherently probabilistic and some target domains (e.g., weather nowcasting, autonomous driving) are safety-critical, it would be informative to quantify uncertainty quality.", "broader_impact_concerns": "No serious ethical or broader-impact concerns are apparent for this work. The technique is applied to relatively benign domains (driving scenes, human activities, weather patterns) and is aimed at improving predictive accuracy and realism, which can be beneficial in applications. One point to consider is that if such predictive models are used in safety-critical settings, reliability and uncertainty estimation become important. The authors might consider adding a discussion on uncertainty or failure cases in a camera-ready version to ensure users of PredLDM are aware of its limitations.", "additional_comments": "Additionally, I have some questions that could be clarified or further explored.\n\n- **(Q1) Inference speed and efficiency.** Can the authors provide more details on PredLDM’s inference process and speed? Specifically, how many diffusion time steps are used to generate the future frames, and what is the typical runtime for predicting? This information is crucial for understanding the practicality of the model. Any quantitative comparison of inference time against a baseline would be very informative.\n\n- **(Q2) Balancing conditional reconstruction vs. generation.** In the CA-LDM stage, the model optimizes a combination of two objectives – the standard diffusion loss for predicting future latent frames and the loss for reconstructing the condition frames. How are these two losses weighted or balanced during training? The paper mentions a “linear combination” but does not detail whether a specific coefficient or schedule is used. Moreover, did the authors observe any trade-off when adding the condition reconstruction loss?\n\n- **(Q3) Scalability to higher resolution or longer sequences.** The experiments in the paper are conducted on sequences of relatively low resolution (e.g., 128$^2$ pixels) and predict a fairly short horizon (e.g., 10 frames). How well would PredLDM scale if we require higher-resolution predictions or a longer sequence of future frames?", "extra_scores": {}, "extra_text": {}}, {"review_id": "1szkGE3J8e", "reviewer": "Reviewer_aNY2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## (A) Brief Summary: \n\nThis submission introduces a spatiotemporal sequence prediction model PredLDM based on latent diffusion models (LDMs). It aims to address two key challenges: (1) capturing intricate global spatiotemporal coherence and (2) achieving a comprehensive understanding of historical image sequences, which serve as the condition. \n\nThis work proposes a two-stage framework to address this. The first stage uses a Masked-Attention Transformer-based VAE (MT-VAE) for perceptual compression. It employs a transformer-based masked self-attention to model global dependencies. The second stage introduces Condition-aware LDM (CA-LDM) to generate future latent sequences, conditioning on historical embeddings to bridge the gap between complex historical observation and future frame generation. \n\nPredLDM is evaluated on KittiCaltech, KTH, and SEVIR datasets and show competitive results compared to baselines, especially in perceptual and distributional metrics. It contains quantitative benchmarks, thorough ablations, and qualitative analyses, suggesting improvements in visual fidelity and temporal consistency for challenging sequence prediction scenarios.\n\n---\n## (B) Strengths: \n\n**(S1)** Clear problem formulation and motivation. Spatiotemporal sequence forecasting is important for real-world applications such as autonomous driving, robotics, and weather forecasting, each of which is included in this work’s experiments. It targets two key challenges in this field: global temporal coherence and history (condition) understanding, which are inadequately met by existing latent diffusion models like T2I and T2V. \n\n**(S2)** Reasonable two-stage method design. To address the two identified issues, this paper directly maps its two main contributions (MT-VAE and CA-LDM) to them. It leverages both a transformer-based VAE for perceptual compression and a condition-aware latent diffusion model for spatiotemporal prediction. This is a reasonable combination that reflects existing research trends in high-fidelity generative modeling, as shown in Fig. 1.\n\n**(S3)** Thorough quantitative and qualitative analysis. The experiments to support the claims are appreciated. It includes three diverse datasets (driving, human action, weather) with not only commonly-used frame/sequence-level metrics (PSNR, SSIM, LPIPS, POD, FAR, CSI) but also temporal dynamics evaluation (like those in Fig. 2 for time-evolving metrics) and distributional alignment (e.g., KVD/FVD in Fig 3). In addition, ablative studies in Tab 2–3 and Fig 5 show that effectiveness of individual components (MT-VAE, CA-LDM) in PredLDM.\n\n**(S4)** Competitive results. PredLDM yields strong and consistent performance on KittiCaltech for driving, KTH for human activity, and SEVIR for weather. Notably. it achieves substantial gains in perceptual metrics like LPIPS (e.g., 30.5% on KittiCaltech) and relative error accumulation, as shown in Tab 1 and Fig 4. This provides strong quantitative evidence for the claim of generating realistic sequences.\n\n**(S5) Visualization and error analysis.** Fig 4, 7, ablation heatmaps, and masking ratio studies in Fig 5, 6, 8, 9, 10 show careful error localization and condition-effect interpretation to support the claims of improved fidelity and robustness of PredLDM.\n\n---\n## (C) Weaknesses: \n\n**(W1)** Ambiguous method illustration. It is vague on the precise method mechanics. For example, the \"masked self-attention\" in the MT-VAE is not clearly defined (e.g., is it input masking or an attention mask?) nor is its benefit for compression (rather than pre-training) explained. Similarly, the technical reason why the $\\mathcal{L}_{CA}$ loss helps understanding is only described at a high level without precise justification.\n\n**(W2)** Results discussion. For example, Tab 1 presents aggregate results, but there is little discussion of why some models like ConvLSTM outperform PredLDM in PSNR for certain datasets. Many similar points remain inadequately discussed, which I find regrettable given the extensive experiments. I recommend that the authors thoroughly discuss and derive insights based on existing experiment results, incorporating different methods’ designs. I believe this would be helpful for future research in spatiotemporal sequence prediction. \n\n**(W3)** Some notation issues reduce readability. Some operators and function names are typeset in italics, which are commonly noted as upright (\\text{} or \\operatorname{}) to distinguish them from variables. Specifically, (i) “mAtt” in Eq. (5), (ii) “FeedForward” in Eq. (6), (iii) “softmax” in Eq. (9), (iv) “where” & “otherwise” in Eq. (10), (v) “Concat” in Eq. (11), and more. I recommend the authors thoroughly check the entire manuscript and revise all similar matters.\n\n**(W4)** Incomplete literature review. There are several diffusion-based models specifically for spatiotemporal and time-series prediction, including Latent-Shift [1], PriSTI [2], Stochastic Diffusion [3], and TimeAutoDiff [4]. They are not discussed or compared against in the manuscript. While some of these are preprints on arXiv, they have accumulated a significant number of citations, indicating considerable influence in the field. Considering the method relevance, I recommend the authors incorporate related discussion in the revised manuscript.\n\n**(W5)** Minor issues about ablation settings. MT-VAE is not compared against a non-masked transformer VAE. The CA loss is introduced with a 1:1 weighting to LDM loss without sensitivity analysis. I recommend the authors add related experiments to improve the robustness and academic rigor of this paper.\n\n**(W6)** Figure quality and formatting. The presentation quality of figures is low. Many figures like Fig. 1, 3, 4, 6, 7  appear to be low-resolution bitmaps with noticeable zigzag artifacts, especially as the figures are mostly large in the manuscript. I strongly recommend the authors thoroughly check all the figures in both main text and appendix, replace them with high-quality vector graphics (by exporting SVGs directly to PDF) for better presentation. It would be with much lower file size than screenshot but with much better quality. Furthermore, for figure sub-labels (e.g., in Fig. 4), using bracketed labels like \"a),\" \"b),\" and \"c)\" is often clearer for identification than \"a,\" \"b,\" \"c,\". Disregard this suggestion if this is a recommended format from the TMLR official template.\n\n**(W7)** Computational cost. This is a practical concern. The heavy two-stage model (Transformer VAE + 32-layer DiT) requires 1000 denoising steps for inference. There is no discussion of model parameters, training costs, or inference latency. IMHO, this is a very important aspect that is missing, especially in applications like autonomous driving where latency is crucial.\n\n---\n\n## Reference\n\n[1] Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation, arXiv 2023.\n\n- It introduces efficient temporal operations in latent diffusion for video/spatiotemporal generation, which is similar to this submission’s domain and design.\n\n[2] PriSTI: A Conditional Diffusion Framework for Spatiotemporal Imputation, ICDE 2023.\n\n- It proposes diffusion for spatiotemporal data imputation, underlying similar challenges in temporal modeling.\n\n[3] Stochastic Diffusion: A Diffusion Probabilistic Model for Stochastic Time Series Forecasting, SIGKDD 2025.\n\n- It is relevant as a model for uncertainty in spatiotemporal sequence prediction\n\n[4] TimeAutoDiff: Combining Autoencoder and Diffusion Model for Time Series Tabular Data Synthesizing, arXiv 2024.\n\n- It combines autoencoder and diffusion models for sequential data, which is relevant to this submission in architectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims made in this submission is quite reserved. It stated that PredLDM produces more \"realistic predictions\" than existing model. This is well-supported. First, it provides convincing evidence through competitive empirical results on the LPIPS, FVD, and KVD metrics, which are more suitable for measuring realism than traditional MSE/PSNR. Second, the qualitative results in Fig. 4, 7, and 11-16 further substantiate this, which shows a clear visual improvement in sharpness and detail retention over time.\n\nHowever, the claim that MT-VAE successfully compresses \"intricate global coherent spatiotemporal content\" is supported weakly. Tab. 2 ablation shows it outperforms 3D VAEs and 2D VAEs + 1D Convs, but there is no ablation against a standard, non-masked transformer VAE. Thus, it is unclear if the \"masked\" part of the contribution is meaningful.", "audience_yn": "Yes", "audience_explanation": "This paper targets spatiotemporal sequence prediction, which is a fundamental task with broad real-world applications. The adaptation of diffusion models to this conditional generation problem is a very active and important research field. IMHO, the CA-LDM with $\\mathcal{L}_{CA}$ loss in this work for improving conditionality is a design that researchers in generative modelling and video prediction would be interested in. The strong empirical results on perceptual metrics make it practical as a strong new baseline.", "requested_changes": "Most of my concerns and corresponding requested changes and suggestions have been stated in the Weaknesses section. Overall, I have a positive view on this submission. I encourage the authors to focus their efforts on addressing those points, especially **(W1)** **(W3)** **(W5)** **(W6)**, as they are critical for securing recommendation for acceptance.\n\nThe following are more specific and open questions to help the authors think more deeply about certain design choices and experiment setups, which I hope could be helpful for this and future work:\n\n**(Q1)** What is the impact of OOD (out-of-distribution) or distribution-shift scenarios (e.g., unusual traffic/weather patterns) on PredLDM’s predictive quality and calibration? Would the model’s performance degrade smoothly, or does it show catastrophic failures? Some discussion or targeted experiments would be valuable.\n\n**(Q2)** Could the authors clarify design choices regarding masking ratio in Fig 6, 10, and provide practical heuristics for tuning in new domains/datasets? It would also be helpful for the community to use PredLDM in other scenarios.\n\n**(Q3)** How does the length of the input and output sequences affect the model's performance? Can PredLDM easily adjust to different forecasting horizons? Are there any modifications necessary for very long or short sequence prediction tasks?", "broader_impact_concerns": "This work addresses generative video, but its application is constrained to short-term future prediction based on a historical context. It seems to pose no obvious ethical risks, and I have no broader impact concerns.", "additional_comments": "I hope my comments help fellow reviewers and editors understand the basis of my recommendation. I am open to follow-up discussions with the authors to help further strengthen this work and reach a consensus for the final decision.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)