Video Prediction Transformers without Recurrence or Convolution
Authors: Yujin Tang
Lu Qi
Xiangtai Li
Chao Ma
Ming-Hsuan Yang
Venue: Accepted by TMLR ()

Abstract: Video prediction has witnessed the emergence of RNN-based models led by ConvLSTM, and CNN-based models led by SimVP. Following the significant success of ViT, recent works have integrated ViT into both RNN and CNN frameworks, achieving improved performance. While we appreciate these prior approaches, we raise a fundamental question: Is there a simpler yet more effective solution that can eliminate the high computational cost of RNNs while addressing the limited receptive fields and poor generalization of CNNs? How far can it go with a simple pure transformer model for video prediction? In this paper, we propose PredFormer, a framework entirely based on Gated Transformers. We provide a comprehensive analysis of 3D Attention in the context of video prediction. Extensive experiments demonstrate that PredFormer delivers state-of-the-art performance across four standard benchmarks. The significant improvements in both accuracy and efficiency highlight the potential of PredFormer as a strong baseline for real-world video prediction applications. The source code and trained models will be released to the public.

Published in Transactions on Machine Learning Research (01/2026)
Video Prediction Transformers without Recurrence or Con volution
Yujin Tang"
tangynjjn0275@@maall com
Shanghai Jiao Tong University University of California, Merced
Lu Qi
qglu199200mmiil.com com
Wuhan University
Xiangtai
ziongtai94 @gmaill com
Nanyang Technological University
Chao Ma
chaoma990gmaill.com
Shanghai Jiao Tong Universily
Ming-Hsuan Yang
minghssnanyong@@@aal.cco
University of California, Merced
Reviewed on OpenReview: https: oppp://veeeee..ee.e.mmmmmm. net/ forum? id= Afuuu
Abstract
Video prediction has witnessed the emergence of RNN-based models led by ConvLSTM, and CNN-based models led by SimVP. Following the significant success ViT, recent works have integrated ViT into both RNN and CNN frameworks, achieving improved performance. While we appreciate these prior approaches, we raise a fundamental question: Is there a sim- pler yet more effective solution that can eliminate the high computational cost of RNNs while addressing the limited receptive fields and poor generalization of CNNs? How far can it go with simple pure transformer model for video prediction? In this paper, propose Pred- Former, framework entirely based on Gated Transformers. We provide a comprehensive analysis of 3D Attention in the context of vidco prediction. Extensive exporiments demon- strate that PredFormer delivers state-of-the-art performance across four standard bench- marks. The significant improvements both accuracy and efficiency highlight the potential of PredFormer as strong baseline for real-world video prediction applications. The source code and trained models are released at https: pettps://githut httpchhww..ooyyyyylyy.. http://www..oooooyyyyya/aaaa/rrr
1 Introduction
Video Prediction Wang et al. 2018c; Chang et al. 2021 Gao et al. 2022a), also named as involves learning spatial and temporal
predictive learning (Wang et al. 2017 2018b Tan et al. 2023a b) patterns by predicting future frames based on past observations. This capability is essential for various
applications, including weather forecasting (Rasp al. 2020 Pathak et al., 2022 Bi et al. 2023), traffic
flow prediction (Fang et al.) 2019 Wang et 2019) precipitation nowcasting (Shi et al. 2015 Gao al.
2022b) and human motion forecasting (Zhang et al. 2017b Wang et al. 2018a).
Despite the success of various video prediction methods, they often struggle to balance computational cost
and performance. On the one hand, high-powered recurrent-based methods (Shi et al. 2015 Wang et al.
Project Page: httpp://yyyujintest pttpghtt.../ppprtoooeerrrrrieeee Work done while visiting University of California. Merced. 1 Corresponding author.

---

Published in Transactions on Machine Learning Research (01/2026)
Recurrent-free
Recurrent-based
CNN-based
ViT-based +
Lt+
t+2
Lt+2 I ++3
Bt+z I t+3
f
RNN
RNN
CNN Decoder
Patch Recover
t
CNN/ MetaFormer
RNN
RNN
Transformer
RNN
RNN
CNN Encoder
Patch Embed t
t
Lt
+++1
It Et+1 (b) CNN-CNN-CNN
It It+1
(a) Stacked RNN
(c) Stacked Transformer
Figure 1: Main categories of video prediction framework. (a) Recurrent-baser Framework (b) CNN Encoder- beseder-based Recurrent-free Framework.
(c) Pure transformer-based Recurrent-free Framework.
2017 2019 Chang et al. 2021 Yu et al. 2019 Tang et al. 2023 2024 rely heavily on autoregressive RNN franeworks, which face significant limitations parallelizatio and computational efficiency. On the other hand, efficient recurrent-free methods (Gao el. al. 2022a Tan el. al., 2023a), such as those based the SimVP framework, use CNNs in an eecoder-decoder architecture but constrained by local receptive fields, limiting their scalability and generalization. The ensuing question is Can develop framework that autonomously learns spatiotemporal dependencies without relying on inductive bias?
An intuitive solution directly adopts a pure transformer (Vswaani et al. 2017) structure, as is an efficient alternative to RNNs and has better scalability than CNNs. Transformers have demonstrated remarkable success in visual tasks (Dosovitskiy et al. 2020 Liu et al., 2021; Bertasius et al., 2021] Arnab al.] 2021 Tarasiou et al. 2023 Previous video prediction methods try to combine Swin Transformer (Liu et al.
2021 in excurrent-basee frameworks such as SwinLSTM (Tang et al. 2023 and integrate MotaFormer Yuu et al. 2022)
temporal translator in recurrent-free CNN-based encoder-decoder frameworks such as Open- STL Tan et al.) 2023b). Despite these advances, pure causfformr-bbased architecture remains underexplored mainly due to the challenges of capturing spatial and temporal relationships within a unified framework. While merging spatial and temporal dimensions and applying full attention is conceptually straightforward, it is computationally expensive because of the quadratic scaling of attention with sequence length. Several recent methods (Bertasius et al., 2021 Arnab et al.) 2021 Tarasiou et al. 2023) decouple full attention and show that spatial and temporal relations can be treated separately in factorized interleaved manner to reduce complexity.
In this work, we propose PredFormer, a pure transformer- architecture for video prediction. PredFFormer dives into the decomposition of spatial and temporal transformers, integrating self-attention with gated linear units (Dauphin et al. 2017) more effectively capture complex spatiotemporal dynamics. addition retaining spatial-temporal full attention encoder and factorized encoder strategics for both spatial-fisst and temporal-first configurations, we introduce six novel interleaved spatiotemporal transformer architectures, resulting in nine configurations. We explore how far this simple framework can go with different strategies of 3D Attention. This comprehensive investigation pushes the boundaries of current models and sets valuable benchmarks for spatial-temporal modeling.
Notably, PredFormer achieves state-of-the--e performance across four benchmark data sets, including syn- thetic prediction of moving objects, real-world human motion prediction, traffic flow prodiction and weather forecasting, outperforming previous methods by a substantial margin without relying on complex model architectures specialized loss functions.
2

---

Published in Transactions on Machine Learning Research (01/2026)
The main contributions can be summarized as follows:
We propose PredFormer, pure gated transformer-base framework for video prediction. By clim- inating the inductive biases inherent in CNNs, PredFormer harnesses the scalability and general- ization capabilitics of the transformers, achieving significantly enhanced performance ceilings with efficiency.
We perform an in-depth analysis of spatial-temporal transformer factorization, exploring full- attention encoders and factorized encoders along with interleaved spatiotemporal transformer ar- chitectures, resulting in nine PredFormer variants. These variants address the differing spatial and temporal resolutions across tasks and datasets for optimal performance.
We conduct comprehensive study training VIT from scratch small datasets, exploring regu- larization and position encoding techniques.
Extensive experiments demonstrate the state-of-the-art performance of PredFormer. It outperforms SimVP by 48% while achicving 1.5x faster inference speed Moving MNIST. Besides, Premerrmer surpasses SimVP with 8x, 5x, and 3x inference speed on TaxiBJ, WeatherBench, and Human3.6m while achieving higher accuracy.
2 Related Work
Recurrent-based video prediction. Recent advancements in recurrent-based video prediction models have integrated CNNs, ViTs, and Vision Mamba (Liu et al. (024) into RNNs, employing various strategies capture spatiotemporal relationships. ConvLSTM (Shi al.) 2015), evolving from FC-LSTM (Srivastava 2015), innovatively integrates convolutional operations into the LSTM framework. PredNet (Lotter et al. 2017) leverages deep recurrent convolutional neural networks with bottom-up and top-down connec- tions to predict future video frames. PredRNN (Wang et al.] 2017) introduces the Spatiotemporal LSTM (ST-LSTM) unit, which effectively captures and memorizes spatial and temporal representation ions by propa- gating hidden states horizontally and vertically. ProdRNN+4 + (Wang et al. 2018b) incorporates a gradient highway unit and Causal LSTM to address the vanishing gradient problem and adaptively capture temporal dependencies. E3D-LSTM (Wang et al.) 2018c) extends the memory capabilities of ST-LSTM by integrating 3D convolutions. The MIM model (Wang al. 2019) further refines the ST-LSTM by reimagining the forget gate with dual recurrent units and utilizing differential information between hidden states. CrevNet (Yu) et al., 2019) employs CNN-based reversible architecture lo decode complex spaliolemporal patterns. Pre- dRNNv2 (aagg et al.) 2022) enhances PredRNN by introducing a memory decoupling loss and a curriculum lcarning strategy. MAU (Chang et al.) 2021) adds a motion-aware unit to capture dynamic motion infor- mation. SWinLSTM (Tang et al.] 2023) integrates the Swin Transformer (Liu el al. 2021 module into the LSTM architecture, while VMRNN (Tang et al.] 2024p extends this by incorporating the Vision Mamba module. Unlike these approaches, PredFormer is a recurrent-free method that offers superior efficiency.
et al.
Recurrent-free video prediction. Recent recurrent-fic models, e.g., SimVP (Gao et al. 2022a). are developed based on a CNN-based encoder-decoder with temporal translator. TAU (Tan et al. 2023a builds upon this by separating temporal attention into static intra-frame and dynamic inter-frame compo- nents, introducing a differential divergence loss to supervise inter-frame variations. OpenSTL (Tan et al. 2023b integrates a MetaFormer model as the temporal translator. Additionally, PhyDNet (Guen & Thome 2020) incorporates physical principles into CNN architectures, while DMVEN (Hu al.) 2023) introduces a dynamic multi-scale voxel flow network to enhance video prediction performance. EarthFormer (Gao et al. 2022b) presents 2D CNN encoder-decoder architecture with cuboid attention. WAST (Nie et al. 2024 proposes a wavelet-based method, coupled with wavelet-domain High-Frequency Focal Loss. In contrast to prior methods, PredFormer advances video prediction with its recurrent-free, pure transformer- ar- chitecture, leveraging a global receptive field to achieve superior performance, outperforming prior models without relying on complex architecture designs or specialized loss.
Recurrent-based approaches struggle with parallelization and performance, while CNN-based recurrent-free methods often sacrifice scalability and generalization despite their strong inductive biases. Tn contrast
3

---

Published in Transactions on Machine Learning Research (01/2026)
Spatio- Temporal Sequence
XxX
h R
Patch Split
8xTxN*D
Linear: Patch Embeddiig
Positon Encoding
Spacial View
BXTxN*D
PredFormer Encoder
FR
Fi
PC
PC
P1
PC
PK
Temporal View
2
Layer
BxTxCжH*W
Linear: Patch Recovery
(a) PredFFormer Model
(b) Sequence Facterization
Figure 2: Overview of the PredFormer framework.
to prior methods, PredFormer advances video prediction with its recurrent-free, pure transformer-based architecture, leveraging global receptive field to achieve superior performance, outperforming prior models without relying on complex model designs specialized loss designs.
Vision Transformer (ViT). VIT Dosovitskiy et al. 2020) has demonstrated exceptional performance on various vision tasks. In video processing, TimeSformer [Bertasius et al. 2021 investigates the factorization of spatial and temporal self-attention and proposes that divided attention where temporal and spatial attention are applied separately yields the best accuracy. ViVIT (Arnab attention, and dot product mechanisms, concluding that factorized encoder with spatial attention applied first performs better. On the other hand, TSVIT (Tarasiou et al. 2023) finds that factorized encoder prioritizing lemporal allention achieves superior results. Latte Ma et. al. 2024) investigales factorized encoders and factorized self-attention mechanisms, incorporating both spatial-first and spatial-temporal parallel designs, within the context of latent diffusion transformers for vidco generation. Despite these advancements, most existing models focus primarily on video classification, with limited research on applying ViTs to spatio-temporal predictive learning. Moving beyond earlier methods that focus on factorizing self- attention, PredFormer explores the decomposition of spatial and temporal transformers at deeper level by integrating self-attention with gated linear units and introducing innovative interleaved designs, allowing for more robust capture of complex spatiotemporal dynamics.
2021 explores factorized encoders, self-
3 Method
To systematically analyze the transformer structure of the network model in spatial-temporal predictive learning, we propose the PredFormer general model design, shown in Fig 2 In the following sections, introduce the pure transformer-based architecture in Sec 3.1 Next, we describe the Gated Transformer Block (GTB) in Sec 3.2 Finally, we present how to use GTB to build PredFormer layer and archilecture variants in Sec 3.3
3.1 Pure Transformer Based Architecture
Patch Embedding. Follow the ViT design, PredFormer splits sequence of frames N' into sequence of equally sized, non-overlapping patches of size each of which is flattened into a 1D Lokens.
N
4

---

Published in Transactions on Machine Learning Research (01/2026)
Temporal
Spatial
Dinpost
GLU
Temporal
Spatial
Lincer
M
Sputial
N Temporal
SiLUI
Self-Alleniinn
Spatial
Temponal
Liteer
Lineer
Norm
(b.1) Pull Amertio
Fac-T-8
(be3)
(6) Guted Trandfemm Black
(b) Full-Aitearion Eracderr and Factoriaad Encodars
Gated Transformer Block
PredFormer Layer
Spatial
Temperal
N
N
x
Spatial
Temporal
Temporal
Spatial
Spatial
Temporal
Temperal
Spatial
Temporal
Spatial
Temporul
Spatial
Spatial
Temporal
Spatial
Temporal
HlinaryST ST
Biinry-1S
Iriplee-S15
(c-4) Tripki-TST
Quddrppee-ST7S
(c.6) Quodruplet USST
(cl Interieaved Encoders
Figure 3: (a) Gated Transformer Block (b) Full Attention Encoder and Factorized Encoders (c) Interleaved Encoders with Binary, Triplet, and Quadrupled design
These tokens then linearly projected into hidden dimensions D and processed by a layer normalization (LN) layer, resulting in tensor X" BxXxxXx
Position Encoding. Unlike typical ViT approach, which employs learnable position embeddings, we incor- porale a spatiolemporal position encoding (PE) generated by sinusoidal functions with absolute coordinates for each patch.
PredFormer Encoder. The 1D tokens are then processed by a PredFormer Encoder for feature extraction. PredFormer Encoder stacked by Gated Transformer Blocks in various manners.
Patch Recovery. Since encoder is based pure gated transformer, without convolution resolution reduction, global context is modeled at every layer. This allows it to be paired with simple decoder, forming a powerful prediction model. After the encoder, a linear layer serves as the decoder, projecting the hidden dimensions back to recover the 1D) tokens to 2D patches.
3.2 Gated Transformer Block
The Standard Transformer model (Vaswani et al. 2017) alternates between Multi-Head Attention (MSA) and Food-Forward Networks (FFN). The attention mechanism for cach head defined as:
QK
Attention(Q.1) V) = Softmax
V.
(1)
Vdx
where in self-attention the queries Q, keys K, and values V are linear projections of the input X, represented as Q XW,,, g. K = XW and V - XW., with X, Q, K. V e Nxd, The FFN then processos each position in the sequence by applying two linear transformations.
Gated Linear Units (GLUs) (Dauphin et al. 2017), often used in place of simple linear transformations, involve the clement-wise product of two lincar projections, with one projection passing through a sigmoid function. Various GLU variants control the flow of information by substituting the sigmoid with other non-linear functions. For instance, SwiGLU (Shazeer 2020) replaces the sigmoid with the Swish activation
5

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Benchhmark datasets used in our experiments. "Interval" denotes the temporal gap between two consecutive frames in the sequence (e.g., framo-level sampling, 30 minutes, or hour).
Dataset
Training size
Testing size
Channel
Height
Width
Input T
Output T
Interval
Moving MNIST
10,000
10,000
64
64
10
Human3. 6m
73,404
8,582
3
256
256
4
frame
WentherBench--8
52,559
17,495
TaxiBJ
20,461
500
2
32
32
4
4
1 hour
function (SiLUU
(Hendrycks & Gimpel 2016); as shown in Eq
Swisha(x) x8xx)
SwiGLU(r, W,, V,4, (3 = Swish + b) 8 (xV + +c)
(2)
SwiGLU has been demonstrated to outperform Multi-layer Perceptrons (MLPs) in various natural language processing tasks(Shazeer) 2020). Inspired by the SwiGLU's success in these tasks, our Gated Transformer Block (GTB), incorporates MSA followed by a SwiGLU-based FFN, as illustrated in Fig aa. GTB is defined as:
Y = MSA(IN(Z))) + Z
z z1+1 SWI) + Y
(3)
3.3 Variants of PredFormer
Modeling spatiotemporal dependencies in video prediction is challenging, as the balance between spatial and temporal information differs significantly across tasks and datascts. Developing flexible and adaptive models that can accommodate varying dependencies and scales is thus critical. To address these, we explore both full-attention encoders and factorized encoders with spatial-firtt (Fac-S-T) and temporal-fistt (Fac-T-S) configurations, shown in Fig 3 b). In addition, we introduce six interleaved models based PredFormer layer, enabling dynamic interaction across multiple scales.
A PredFormer layer is a module capable of simultaneously processing spatial and temporal information. Building on this design principle, we propose three interleaved spatiotemporal paradigms, Binary, Triplet, and Quadruplet, which sequentially model the spatial and temporal views. Ultimately, they yield six distinct architectural configurations. A detailed illustration of these nine variants is provided in Fig
For full attention layers, given input X E BXXxxXXx attention is computed over the sequence of length T x N. As illustrated in Fig 3 (b.1), we merge and flatten the spatial and temporal tokens to compute attention through several stacked GTB..-
For Binary layers, cach GTB block procosses temporal or spatial sequence independently, which we denote as a binary-TS or binary-ST layer. The inpul. first reshaped, and processed through GTB,, where attention is applied over the temporal sequence. The tensor is then reshaped back to restore the temporal order. Subsequently, spatial attention is applied using another GTB,, where the tensor is flattened along the temporal dimension and processed.
For Triplet and Quadruplet layers, additional blocks are stacked on top of the Binary structure. The Quadruplet layer combincs two Binary layers in differont orders.
4 Experiments
We present exlensive evaluations of PredFormer and shale-of-the-art methods. We conduct. experiments across synthetic and real-world scenarios, including long-term inndiction(moovnng object trajectory prediction and weather forecasting), and short-term prediction (traffic flow prediction and human motion prodiction). The statistics of the data set are presented in the tab These datasets have different spatial resolutions, temporal frames, and intervals, which determine their different spatiotemporal dependencies.

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Quantitative comparison on Moving MNIST. Each model observes frames and predicts the subsequent 10 frames. We highlight the best experimental results bold red and the second-best bebtt in blue.
Method
(aras(M) Flops(G) FPS Memory (MB)
MSE
MAE
SSTM 1
ConVLSTM
15.0
56.8
113
68.8
103.3
182.9
0.707
PredRNN
23.8
116.0
54
107 6
56.8
126.
0.867
PredRNN++
MIM
38.0
179.2
37
160.6
44.2
101.1
0.910
E3D-LSTM
51.0
298.9
270.1
41.3
86.4
0.910
PNyDNet
3.1
15.3
182
149.5
21.1
70.3
0.917
MAU
4.3
17.8
201
35.4
27.6
86.3
0.9:7
ProdRNNv2
24.6
708.0
24
109.0
48.4
129.8
0.891
SwinLSTM
20.2
69.9
62
96.4
17.7
0.962
SimVP
58.0
19.4
209
284.7
23.8
68.9
0.948
TAU
14.7
16.0
283
322
19.8
60.3
0.957
OpenSTL VIT
46.1
16.9
290
331.8
19.0
60.8
0.955
OpenSTI. Swin
46.1
16.9
290
33 1.9
18.3
59.0
0.960
PredFormer
Attention
21.2
17.3
Fac-S-T
26.3
16.5
368
117.4
20.6
63.5
0.955
Fac-T-S
25.3
16.5
370
117.
16.9
53.8
0.963
Binery-TS
25.3
16.5
301
117.4
12.8
46.1
0.972
Binary-ST
25.3
16.5
316
117.4
13.4
17.1
0.971
Triplet-'TST
Triplct-STS
25.3
16.5
321
118.0
13.1
46.7
0.972
Quadruplet-
25.3
16.5
302
12.4
44.6
0.973
Quadruplet-STTS
25.3
16.4
322
118.0
12.4
44.9
0.973
Implementation Details Our method is implemented in PyTorch. The experiments conducted single 24GB NVIDIA RTX 3090. PredFormer is optimized using the AdamW (Loshchilov & Hutter 2019) optimizer with an L2 loss, weight decay of 1e-2, and a learning rate selected from {5e-4, 1e-3) for best performance. OneCyele scheduler is used for Moving MNIST and TaxiBJ, while the Cosine scheduler is applied for Human3.66 6.m and WeatherBench. Dropout (Hinton 2012) and stochastic depth (Huang et al. 2016) regularization prevent overfitting. Further hyperparameter details are provided the Appendix. For different PredFormer variants, maintain constant number of GTB blocks to ensure comparable parameters. In cases where the Triplet model cannot be evenly divided, use the number of GTB blocks closest to the others.
Evaluation Metrics We assess model performance using a suite of metrics. (1) Pixel-wise error is measured using Mean Squared Error (MSE), Mean Absolute Error (MAF),, and Root Mean Squared Error (RMSE). (2) Predicted frame quality evaluated using the structural similarity index measure (SSIM) metric WWagg et al.) 2004). Lower MSE, MAE, and RMSE values, combined with higher SSIM, signify better predictions. (3) Computational efficiency assessed by the number of parameters, floating-point operations (FLOPs), and inference speed in frames per second (FPS) on NVIDIA A6000 GPU. This evaluation framework comprehensively evaluates accuracy and efficiency.
4.1 Synthetic Moving Object Prediction
Moving MNIST. The moving MNIST dataset Srivastava et al. 2015) serves as benchmark synthetic dataset for evaluating sequence reconstruction models. We follow (Srivastava et al. 2015 to generate Moving MNIST sequences with 20 frames, using the initial frames for input and the subsequent 10 frames as the prediction target. We adopt 10000 sequences for training, and for fair comparisons, we use the pre-generated 10000 sequences (Gao et al.
2022a) for validation.
On the Moving MNIST dataset, following prior work (Gao et al. 2022a Tan et al. 2023a), we train our models for 2000 epochs and report our results in Tab 2 We train OpenS TL methods with VIT and Swin Transformer as temporal translators for 2000 epochs as recurrent-free baselines. We cite other results from each original paper for fair comparison.
Compared to SimVP, PredFormer achieves substantial performance gains while maintaining lightweight structure. Specifically, reduces MSF by 48% (from 23.8 to 12.4), significantly improving prediction accu-

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Quantitative comparison on Human3. Each model observes 4 frames and predicts the subse-
quent frames.
Method
Paras M) Flops(G) FPS Memory(M
MSS 1 MAE
SSIM
PSNN.
LPIPS
CoM
125.5
1566.7
0.9813
33.40
0.03537
PredNet
12.5
13.7
176
120.8
261.9
1625.3
0.9786
31.76
0.08264
PNecRNN
24.6
704.0
327.2
113.2
1458.3
0.9831
33.
0.05245
PredRNN
39.3
1033.0
110.0
1452.2
0.9832
0.03196
MIM
47.6
1061 0
17
434.8
112.1
1467.1
0.9829
33 95.97
0.03335
ESD BaD-LSTM
60.9
542.0
5418.9
143.3
1442.5
0.9803
32.52
0.04133
PhyDNet
125.7
1614.7
0.9804
33,05
0.03709
MAL
20.2
105.0
371.2
127.3
1677.0
0.9812
33.33
0.03561
PNe2
24.6
708.0
$50.5
114.9
1484.7
0.9827
33
0.05331
SinVP
41.2
197.0
26
556.4
115.8
1511.5
0,9822
33.73
0.03467
TAU
37.6
162.0
551.8
113.3
1390.7
0.9839
34.03
0.02783
OpenSTL
11.0
1170.00
136.3
1003.5
0.9796
33.10
0.03729
OpenSTL_ Swin
38.8
188.0
28
5622.3
133.2
1599.7
0.9799
33.16
0.03766
PredFormer
Full Atfention
12.7
155.0
16
1120.7
113.9
1412.4
0,9833
33 98
0.08279
Fac-S-T
12.7
65.0
352.7
153.4
1630.7
0.9784
32 32.30
0.04676
Far-T-8
12.7
65.0
336.7
118.4
1504.7
0.9820
33.67
0.03284
Binary-TS
12.7
65.0
75
352.7
111.2
1380.4 4
0. 9898
31.13
0.08006
Binary-ST
12.7
65.0
348.7
112.7
1336.3
0.9836
34.07
0.05017
Triples- TST
12.7
352.7
112.4
1406.2
0.9834
34.05
0.02748
"Tippes-8TSs*
12.7
69.3
64
356.7
111.8
1410.3
0.9834
34.07
0. 0.02933
Quadruplet- TSST
12.7
85.0
352.7
110.9
1380.3
0.9839
34.14
0.05089
Quadropkt-STTS
12.7
65.0
113.4
1405.7
0.9835
34.04
0.02918
models, add skip connection each PredFormer Layer stable training.
racy. Meanwhile, PredFormer requires far fewer parameters (25.3M vs. 58.0M in SimVP) and operates with lower FLOPs (16.5G vs. 19.4G), showcasing its superior efficiency.
Notably, even when SimVP incorporates ViT and Swin Transformer as the temporal translator, its perfor- mance remains far below that of PredFormer. This is because, while SimVP benefits from the inductive bias of using CNNs as the encoder and decoder, this design inherently limits the model's performance ceiling. In contrast, PredFormer effectively models global spatiotemporal dependencies, allowing to surpass these constraints and achiove superior predictive accuracy.
Compared to SWinLSTM, PredFormer achieves higher accuracy. Although SwinLSTM outperforms SimVP in terms of MSE (17.7 vs. 23.8), its reliance an RNN-based structure results in significantly higher computational cost. SwinLSTM exhibits high FLOPs (69.9G) and lower FPS, making it less efficient for large-scale deployment. This highlights the limitations of recurrent structures in video prediction, whereas PredFormer, with its recurrence- free framework, achieves both higher accuracy and superior efficiency.
Among the PredFormer variants, Quadruplet-TSSS achieves the best MSE of 12.4, followed closely by Quadruple--STTS. These results highlight. PredFormer's ability to fully leverage global information fully, further validating its effectiveness in video prediction.
4.2 Real-world Human Motion Prediction
Human3.6m The Human3.66 6.6 dataset (Ionescu et al., 2014 comprises 3.6 million unique human poses with their corresponding images, serving as a benchmark for motion prediction tasks. Human motion prediction is particularly challenging due to high resolution input and complex human movement dynamics. Following OpenSTT. (Tan al. (023b), we downsample the dataset from 1000x 1000x3 to 256x256x33 We use four observations to predict the next four frames.
PredFormer achieves SOTA on Human3.6MM Compared to SimVP, PredFormer Quadruplet-TSSS TSST reduces MSE from 115.8 to 110.9, significantly improving human motion prediction. At the same time, PredFormer requires only 12.7M parameters, much fewer than SimVP's 41.2M. Furthermore, its computational cost is only 65G FLOPs, less than one third of SimVP's 197G, while maintaining a higher inference speed.
PredFormer also substantially reduces the computational cost compared to PredRNN++. While Pre- dRNN++ requires 1033G FLOPs, ProdFormer Quadruplet-Tre achieves comparative MSE and superior MAF and SSTM using only one-tenth of the computation.
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: Quantitative comparison on TaxiBJ. Each model observes 4 frames and predicts the subsequent
frames.
Method
Paras(M) (G) FP$ Memory(Mi)
MSE J. MAE J. SSTM 1
ConvLSTM
15.0
20.7
815
67.0
0.485
17.7
0.978
PredRNN
23.7
12.1
116
105.5
0.461
16.9
0.977
+++
162.8
MIM
37.9
64.1
275
158.4
0.429
16.6
0.971
E3D-LSTM
51.0
240.5
0.432
0.979
PhyDNet
3.1
5.6
982
149.3
0.362
15.5
0.983
PredRNNv2
23.7
42.6
378
106.8
0,383
15.5
0.983
SwinLSTM
2.9
1.3
1425
22.0
0.303
15.0
0.984
ShnVP
13.8
0.982
TAU
9.6
2.5
1268
175.0
0.344
15.6
0.983
OpenSTL VIT
9.7
2.8
1301
174.8
0.317
15.2
0.984
OpenSTL. Swin
9.7
1506
274.3
0.313
15.1
0.985
PredFormer
Full Attention
8.4
2.4
2438
42.3
0.316
14.6
0.985
Fac-T-S
8.4
2.2
3224
42.4
0.283
14.4
0.985
Binary-TS
8.4
2.2
3192
12.4
0,286
11.6
0.985
Binary-ST
8.4
3172
42.4
0.277
14.3
0.986
Triplet-TST
6.3
1.6
1318
3144
0.293
11.7
0.985
Triplet-STSS
Quadruplet/TSSe TSST
8.4
2.2
3230
42.4
0.284
14.4
0.986
Quadruplet-STTS
3259
42.4
0.293
0.985
Compared to OpenSTI-Vil and OpenSTL-Swin Transformer, which rely on ViT-based architectures but struggle with prediction accuracy, PredFormer utilizes the Transformer structures more effectively for video prediction. OpenSTL-ViT and OpenSTL-Swin both perform worse than SimVP, indicating that simply applying Transformers does not guarantee strong results. In contrast, PredFormer outperforms them while maintaining an efficient design, demonstrating its capability in spatiotemporal modeling.
4.3 Traffic Flow Prediction
TaxiBJ. TaxiBJ (Zhang et al. 2017a) includes GPS data from taxis and meteorological data in Beijing, Each data frame is visualized
32 x 32 2 heatmap, where the third dimension encapsulates the inflow and outflow of traffic within designated area. Following previous work (Zhang el. al. 2017a). allocate the final four weeks' data for testing, utilizing the preceding data for training. Our prediction model uses four sequential observations to forecast the subsequent four frames.
PremFormer achieves SOTA on TaxiBJ, Compared to SimVP, PredFormer significantly improves the accuracy of the prediction, reducing the MSE from 0.414 0.277 (33%) while using fewer parameters (8.4M vs 13.8M) and a lower computational cost (2.2G FLOPs vs. 3.6G FLOPs). Despite this efficiency, PredFormer also dramatically increases inference speed, with FPS rising from 533 in SimVP to 4249.
Furthermore, OpenSTI-ViT and OpenSTI-Swin, which adopt ViT-based architectures as temporal trans- lators, achieve MSEs of 0.317 and 0.313, both worse than PredFormer's best results. This suggests that using CNNs for the encoder and decoder provides a strong inductive bias but inherently limits the model's performance. Among the variants of PredFormer, Binary-ST and Triplet-STS achieve the best MSE of 0.277.
4.4 Weather Forecasting
WeacherBench. Climate prediction is critical challenge in spatiotemporal predictive learning, The WeatherBench (Rasp et al. 2020) datasct provides a comprehensive global weather forccasting resource, covering various climatic factors. In our experiments, we utilize WeatherBench-S, a single-variable setup where each climatic factor trained independently. We focus on temperature prediction at a 5.625 resolution (32 64 grid points). The model is trained on data spanning 2010-2015, validated on data from 2016, and tested on data from 2017-2018, all with one-hour temporal interval. We input the first 12 frames and predict the subsequent 12 frames in this setting.

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Quantitative comparison on (eatherBenchh(2m) Each model observes 12 frames and predicts the subsequent frames.
Method
Paras(M) Plops(G) FPS Memory(Mi)
MSE 4 MAE 1 SSTM 1
ConvLSTM
14.9
136.0
46
69.8
1.521
0.7949
1.233
PredRNN
23.6
278.0
22
108.2
1.331
0.7216
1.154
+++
165.5
0.7883
MIM
37.8
109.0
126
275.9
1.784
0.8716
1,336
PhyDNet
3.1
150.6
285.9
8.7370
16.91
MAU
5.5
39.6
237
56.2
1.251
0.7036
1.119
PredRNNv2
23.6
279.0
110.8
1.345
0.7986
1:243
SimVP
14.8
8.0
196
191.7
1.238
0.7037
1.113
TAUU
1.078
OpenSTL VIT
12.4
8.0
432
19419
1.146
0.6712
1,070
OpenSTL Swin
12.4
6.9
581
195.1
1.143
0.6735
1.069
Premerrmer
Full Attention
185.4
0.6540
1.061
Fac-S-T
5.3
8.5
888
53.9
1.783
0.8688
1.335
Binary-TS
5.3
8.6
837
53.9
1.115
0.6508
1.056
Binary-ST
5.3
8.6
8/17
51.9
1.140
0.6571
1.068
Triplet-TST
4.0
1064
48.9
1.108
0.6492
L.033
Triplet-STS
-.0
6.5
1001
19.9
1.149
0.6658
1.072
Quadruplet-
Quadruplet-STTS
5.3
8.6
858
54.9
1.118
0.6507
1.057
Table 6: Ablation on PredFormer Layer Number on Moving MNIST.
TSST Layer
Paras(M)
Flops(G)
FPS
MSE 1 MAE
SSIM 1
2
8.5
3.5
887
20.1
65.2 2
0.955
3
12.7
8.3
621
16.2
55.1
0.963
4
16.9
11.0
457
13.5
47.9
0.970
5
21.1
13.7
376
12.7
15.5
0.972
6
7
29.5
19.2
277
12.3
44.2
0.973
33.7
22.0
240
11.7
41.8
0.975
On WeatherBench (T2m), RNN-based models like ConvLSTM and PredRNN have high computational costs but poor performance. PredRNN and PredRNNv2 reach 278.0G and 279.0G FLOPs, yet their MSE remains at 1.331 and 1.545, respectively, highlighting the inefficiency of RNN structures for this task.
SimVP reduces parameter count to half of PredRNN, lowers FLOPs to 8.0G, and achieves an improved MSE of 1.238, making it more efficient than RNN models. PredFormer further improves performance, using only half the parameters of SimVP while maintaining similar lower FLOPs, achieving the best MSE of 1.100. The Fac-T-S and Triplet-TST variants deliver the top results. It also demonstrates significant FPS advantage, with Binary-TS and Triplet-TST) achieving 837 and 1064 FPS, respectively, compared to SimVP's 196, highlighting the model's superior efficiency in both computation and prediction speed.
4.5 Ablation Study and Discussion
PredFormer Layer Number. We conduct an ablation study on the number of TSST layers in PredFormer evaluate its scalability and potential for performance improvement, as shown Tab G The results show that as the number of layers incrcasos, PredFormer continues to achieve better results, surpassing the layer TSST configuration reported in Tab With TSST layers, PredFormer already outperforms SimVP, achieving lower MSE of 20.1 while maintaining high efficiency. When increasing to 3 layers, PredFormer surpasses TAU, OpenSTI.-VIT, and OpenSTL-Swin, achieving a lower MSE of 16.2 while requiring only half the FLOPs of these models and delivering twice their FPS. With 8 layers, PredFormer achieves an MSE of 11.7, which represents a 51% MSE reduction compared to SimVP. This substantial improvement demonstrates the scalability of PredFormer.
We conduct ablation studies on PredFormer model design and summarize the results in Tab and Tab We choose the best Triplet-STS model TaxiBJ, and the best Fac-T-S model WeatherBench baselines.
10

---

Published in Transactions on Machine Learning Research (01/2026)
Table 7: Ablation on Gate Linear Unit and Position Encoding.
BeatherBench (T2m)
TaxBB
Model
MSE MAE RMSE
MSE 4 MAE J
PredFormer
1.100
0.6489
1.049
0.277
14.3
SwiGLU MLP
1.171
0.6707
1.082
0.306 0.288
15.1 14.6
PE: Abs - Learnable 1.164
0.6771
1.079
Table 8: Ablation on Dropout and Stochastic Depth.
WeatherBench (T2m)
TaxiBJ
Model
MSE 1 MAE RMSE
MSE J. MAE
DP Uni SD
1.100
0.6489
1.049
0.277
14.3
W/o Reg
0.7U57
1.115
DP
1.210
0.6887
1.100
0.283
14.5
1.075
DP + Linear SD
1.138
0,6033
1.067
0.299
14.8
Gate Linear Unit. Replacing SwiGLU with standard MLP results in notable performance degra- dation. On TaxiBJ, the MSE rises from 0.277 to 0.306, and on WeatherBench from 1.100 to 1.171. This consistent performance degradation highlights the critical role of the gating mechanism in modeling complex spatiotemporal dynamics.
Position Encoding. Additionally, the performance deteriorates when we replace the absolute positional encoding in our model with the learnable spatiotemporal encoding commonly used VIT. On Moving TaxiBJ, the MSE rises from 0.277 0.288, and on WeatherBench from 1.100 1.164. These ablation experiments consistently reveal similar trends across all three datasets, emphasizing the robustness of our Position Encoding designs.
Model Regularization. Pure transformer architectures like ViT generally require large datasets for cf- fective training, and overfitting can become challenging when applied to smaller datasets. In our experi- ments, overfitting is noticeable on WeatherBench and TaxiBJ.. We experiment with different regularization techniques Tab and find that both dropout(DP) and stochastic depth (SD) individually improve perfor- mance compared no regularization. However, the combination of the two provides the best results. Unlike conventional ViT practices, which use linearly scaled drop path rate across different. depths, uniform drop path rate performs significantly better for tasks.
1-1
t-2
1-3
1-4
1-5
1-6
t-7
1-8
t-
t- 10
Input
7
7
2
77
2
2
7
7
7
2
7
7
2 7
27
19
-
7
3
7
7
7 2
7 2
7
7
7
Target
2
2
2
2
PredFormer Prediction
-
3
3
7
7
7
7
2
7
2
7
7
7
2
PredFormer Emor
TAU Foror
Figure 4: Visualizations on Moving MNIST. Error comparison.
Prediction Target/. We amplify the error for better
11

---

Published in Transactions on Machine Learning Research (01/2020)
10
lepul
Prezation
hethina
thor
TAU
Ease
Figure 5: Visualizations on WeatherBench for global temperature lorecasting.
Visualization. Fig and provide a visual comparison of PredFormer's prediction results and prediction errors with Ground Truth. For Moving MNIST, our model accurately captures digit trajectories, with significantly lower accumulated error compared to TAU. On TaxiBJ, PredFormer effectively reconstructs the intricate spatial structures traffic patterns, reducing figh-freeuency noise present in TAU's predictions. On WeatherBench, PredFormer achioves sharper and more precise temperature forocasts, with error heatmaps showing lower doniations in critical regions. Lastly, for Human3. PredFormer consistently preserves fine-grained motion details, demonstrating superior temporal coherence in video prediction. Additional visualizations are provided in the supplementary material.
Input
Imput
Targat
PredFormer Prodiction
PredFormer Ersor
PredFonner
Ence
Erior
TaxtD)
(b) lluman3.óm
Figure 6: Visualizations TaxiBJ and Human3.6m 6mm
Discussion for PredFormer Variants. Despite our in-depth analysis of the spatiotemporal decompo- sition, the optimal model is not definite due to the different spatiotemporal dependent properties of the datasets. We recommend starting with the Quadruplet-TSSS which consistently performs well across datasets and configurations. Use M Qnadruplet-TSST layers and experiment with models having total of 4M GTBs to identify the optimal configuration. Then, explore Triplei-TST and Triplet-STS with M layers to find spatial and temporal dependencies. Unlike the SimVP framework, which adjusts hidden dimensions and block numbers separately for spatial encoder-decode and temporal translator, PredFormer uses fixed hyperparameters for spatial and temporal GTBs, leveraging the scalability of the Transformer architecture. By simply adjusting the number of PredFormer layers, optimal results be achieved with minimal tuning. We provide further theoretical analysis in the Appendix.
model for diverse video prediction tasks,
5 Conclusion
In this paper, we introduce PredFormer, a pure Transformer-based framework for vidco prediction, and systenatically study how different spatio-temporal factorization patterns affect efficiency and accuracy. Across four standard benchmarks, PredFormer achieves strong performance while maintaining favorable
12

---

Published in Transactions on Machine Learning Research (01/2026)
FLOPs, FPS, and memory usage compared to prior CNN- and RNN-based models, providing competitive recurrent-free and convolution-free alternative. On the four benchmarks study, our experiments suggest several dataset-dependent tendencies: interleaved spatio-temporal architectures generally offer the best bal- ance between cost and accuracy; temporal-firs factorizations often work particularly well for long-horizon lorecasting, while other interleaved patterns can be preferable when the spatial structure richer or the hori- zon is shorter; combining dropout with uniform stochastic depth is especially effective on overfitting- prone datasets; and absolute positional encoding consistently more robust than learnable alternatives.
References
Anurag Arnab, Mostala Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video under- standing? In ICML, 2021.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forccasting with 3d neural networks. In Nature, 2023.
Zheng Chang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, Yan Ye, Xiang Xinguang, and Wen Gao. Mau: A motion-aware unit for video prediction and beyond. In NewrIPS, 2021.
Yann Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolu- tional networks. In ICML, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. In ICLR, 2020.
Shen Fang, Qi Zhang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Gstnet: Global spatial-temporal network for traffic flow prediction. In IJCAI, 2019.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet. better video prediction. In CVPR, 2022a.
Zhihan Gao, Xingjian Shi, Hao Wang, Yi Zhu, Yuyang Bernie Wang, Mu Li, and Din Yeung. Earth- former: Exploring space-time transformers for carth system forecasting. In NewrIPS, 2022b.
Vincent Le Guen and Nicolas Thome. Discntangling physical dynamics from unknown factors for unsuper- vised video prediction. In CVPR, 2020.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). In Arziv, 2016.
GE Hinton. Improving neural networks by preventing co-adaptation of feature detectors. In Arzio, 2012.
Xiaotao Hu, Zhowei Huang, Ailin Huang, Jun Xu, and Shuchang Zhou. A dynamic multi-scale voxel flow network for video prediction. In CVPR, 2023.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
Catalin Tonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisesou. Human3 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. In TPAMI, 2014.
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowci Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. In Arzio, 2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for vidco prediction and unsupervised learning. In ICLR, 2017.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. In Arziu, 2024.
Xuesong Nie, Yunfeng Yan, Siyuan Li, Cheng Tan, Xi Chen, Haoyuan Jin, Zhihang Zhu, Stan Z Li, and Donglian Qi. Wavelet-driven spatiotemporal predictive learning: Bridging frequency and time variations. In AAAI, 2024.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. Fourcastnct: A global data-driven high-resolution weather model using adaplive fourier neural operators. In Arzio, 2022.
Stephan Rasp, Peter D Ducben, Sebastian Scher, Jonathan A Weyn, Sonkayna Mouatadid, and Nils Thucrey. Weatherbench: a benchmark data set for dala-driven weather forecasting. In Journal of Advances in Modeling Earth Systems, 2020.
Noam Shazeer. Glu variants improve transformer. In Arzie, 2020.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wii-Kin Wong, and Wang-chun Woo. Convolu- tional Istm network: A machine learning approach for precipitation nowcasting. In NeurIPS, 2015.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representa- tions using lstms. ICML, 2015.
Cheng Tan, Zhangyang Gao, Lirong Wu, Yongjie Xu, Jun Xia, Siyuan Li, and Stan Z Li. Temporal attention unit: Towards efficient spatiotemporal predictive learning. In CVPR, 2023a.
Cheng Tan, Siyuan Li, Zhangyang Gao, Wenfei Guan, Zedong Wang, Zicheng Liu, Lirong Wu, and Stan Z Li. Openstl: A comprehensive benchmark of spatio-temporal predictive learning. In NeurIPS, 2023b.
Song Tang, Chuang Li, Pu Zhang, and RongNian Tang. Swinlstm: Improving spatiotemporal prediction accuracy using swin transformer and lstm. In ICCV, 2023.
Yujin Tang, Peijie Dong, Zhenheng Tang, Xiaowen Chu, and Junwei Liang. Vmrnn: Integrating vision mamba and lstm for efficient and accurate spatiotemporal forecasting. In CVPRW, 2024.
Michail Tarasion, Erik Chavez, and Stefanos Zafeiriou. Vits for sits: Vision transformers for satellite image time series. In CVPR, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
Pichao Wang, Wanqing Li, Philip Ogunbona, Jun Wan, and Sergio Escalera. Rgb-d-based human motion recognition with deep learning: survey. In Computer Vision and Image Understanding, 2018a.
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhiseng Gao, and Philip Yu. Predrnn: Recurrent neural
networks for predictive learning using spatiotemporal Istms. In NeurIPS, 2017.
Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and S Yu Philip. Predrnn+#: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In ICML, 2018b.
Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d Istm: A model for video prodiction and beyond. In ICLR, 2018c.
Yunbo Wang, Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, and Philip S Yu. Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. In CVPR, 2019.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, S Yu Philip, and Mingsheng Long.
Predran: A recurrent neural network for spatiotemporal predictive learning. In TPAMT, 2022.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. In TIP, 2004.
Wei Yu, Yichao Lu, Steve Fasterbrook, and Sanja Fidler. Efficient and information-prrservinn future frame prediction and beyond. In ICLR, 2019.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhon, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR, 2022.
Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. In AAAI, 2017a.
Liang Zhang, Guangming Zhu, Peiyi Shen, Juan Song, Syed Afaq Shah, and Mohammed Bennamoun. Learning spatiotemporal features using 3denn and convolutional Istm for gesture recognition. In ICCVW, 2017b.
15

---

Published in Transactions on Machine Learning Research (01/2026)
A Appendix
A.1 Problem Definition
Vidco prodiction is to learn spatial and temporal patterns by predicting future frames based on past obser- valions. Given a sequence of frames X {x*)) which encapsulates the last T frames leading up to time t, the goal is to forecast the following T" frames y {x> [2')++++2 starting from time t The imput and the output sequence are represented as tensors x E T and y++L E xCxxx where C, H, and W denote channel, height, and width of frames, respectively. The T and T" are the input and output frame numbers. For brevity, we use X' and y to denote X and y in the following sections.
Generally, we adopt deep model equipped with learnable paramcters Fe for future frame prodiction. The optimal set of parameters O* is obtained by solving the optimization problem:
e min ((Fe()))))
(4)
where L the loss function measuring the difference between the prediction and the ground truth.
A.2 Data Transform
We provide a detailed description of the data transformation with PredFormer Binary-ST Layer in Eq The data transformations for other variants follow a similar process.
B, T, N, + [B T,N, Dj, (t flattenn, 1))
[ T. N, DJ + [B, D res T, N, (),,()
[. N. T, DJ - [ N. DJ, GTT 1))
[ N,T, [B, D,, Pus.reh (),T(1,2
A.3 Theoretical Complexity Analysis
Setup and notation. We provide a formal complexity analysis of the nine PredFormer variants in terms of the temporal length T, the number of spatial patches N, and the hidden dimension D. For a fair comparison, all variants are instantiated under the same overall attention budget: we fix the depth of the network (the number of spalio-temppral blocks) and only change how the self-attention operations inside each block are allocated along the temporal and spatial dimensions. In other words, the total mumber of self-attention blocks is kept comparable across variants, and the differences complexity come from the factorization pattern, rather than from simply increasing the number of layers.
Temporal vs. spatial self-attention. Let temporal self-attention block operate N independent sequences of length 7' (one for each spatial patch), and spatial self-attention block operate on T independent sequences of length N (one for each time step). Ignoring constant factors and linear projections, the dominant terms are:
Temporal self-attention over sequences of length T:
Compute O(T2ND),
Memory O).
Spatial self-attention over patchos:
Compute ~ O(TNDD),
Memory ~ O(TN*).
Full spatio-temporal self-attention over all TN tokens:
Compute O(TN))D))
Memory O(TN))).
16

---

Published in Transactions on Machine Learning Research (01/2026)
Table 9: Theoretical per-umit compute and memory complexity of the nine PredFormer variants and repre- sentative CNN/RNN baselines. For ProdFormer, T is the temporal length, N the number of spatial patch tokens (i.e., N = H for patch size P), and D the hidden dimension. For convolutional and recurrent baselines, instead use the spatial grid size H W (pixels grid cells) to denote the number of spatial locations. and C, denote the number of temporal and spatial self-attention blocks in one spatio-lemporal attention blocks. We report only the leading terms
unit and Cj denotes the number of joint and omit constant factors and lower-order terms.
Variant / Baseline
(c,, Ca. Cy)
Compute Complexity
Memory Complexity
ProdFormer variants (patch tokens N)
Full Attention
(0, 0, 1)
((TN)3D)
((TN))
Fac-S-T
(1,1,0)
(NN2D+74ND
O + +T2N
Fac-T-S
(1,1,0)
0 (TN2D+T'ND
O N
Binary-TS
(1,1,0)
OTN2D+77ND
O(TN2+72N NT
Binary-ST
(1,1,0)
D(TN2D+72M)
D(TN2+72N)
Triplet-TST
(2,1,0)
+TN3))
D(2T"N+TN'
Triplet-STS
(1,2,0)
((TTN++TN3))
O
Quadruplet-TSST
(2,2,0)
(12T2N+27H)
Quadruplet-STTS
(2,2,0)
0 ((2T2N + +TTN2))
0(2T2N + 22TN2
Recurrent-based baselines (spatial grid H -W)
ConvLSTM/ PredRNN-family
O (((HW))
(((H))D)
CNN-based baselines (spatial grid И WW O(T(HW)D*)
SimVP-famill
O(T(W))D)
Let C2 and C denote the number of temporal and spatial self-attention blocks in one spatio-temporal unit (macro-block) respectively. For the Full Attention baseline, we denote by çj the number of joint spatio- temporal attention blocks. The per-unit complexity of a factorized variant is then given by:
Compute
O( +c,IN))D),
Nog1NN+c.20 Memory O +c,TN)?
while for the Full Attention bascline it is
Compute (c,(TN))
Memory (cG(TN))).
Since the total depth is fixed, the overall complexity of the full network is lincar in the number of units, and we omit this multiplicative factor for clarily. The resulting per-unit complexities for all nine PredFormer variants are summarized Table
Normalization of attention blocks. In our default configuration, we keep the number of spalio-temporal units the same for all variants, and each unit follows a specific ordering of temporal (T) and spatial (S) self- attention. For example, Human3.66 6m, the Quadruplet-TSST variant is instantiated as TSSTx3, which uses the same number of unils as the Full Allention baseline; the only difference is that each unit in Quadruplet- TSST splits the attention into separate temporal and spatial operations, whereas the Full Attention baseline applies a joint self-attention over all TN tokens. This design ensures that our factorized variants do not gain an unfair advantage by simply reducing depth; instead, the observed differences in FLOPs, memory, and empirical performance can be attributed the factorization pattern itself.
A.4 Experiment Setting
We provide our hyperparameter setting in Tab 10 For Moving MNIST, we use 24 GTB blocks for all PredFormer variants, which means Quadruplet-TSST layers, 8 Triplct-TST layers, and 12 Binary-TS layers, respectively. For the TaxiBJ and WeatherBench datasets, we use 6 GTB blocks for the Triplet variants and 8 GTB blocks for the other variants.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Table 10: Hyperparameter Setting.
Hyperparameter
Moving MNIST TaxiBJ WeatherBench Human3.6m
Training Hyperparameter
Batch Size
16
16
16
8
Learning Rate
1e-3
5e-1
(de-1, 1e-3)
Learning Scheduler
Onecycle Adamw
Onecycle Adarow
Cosine
Cosine
Optimizer
Adamow
Adamm
Weight Decny
le-2
lo-2
le-2
le-2
Training Epochs
2000
200
50
50
Model Hyperparameter
Patch Size
8
GTB Blocks
24
(6.8)
{6.8)
12
GTB Dim
206
256
256
256
GTB Heads
8
SwiGLU Hidden Dim
1024
1024
512
1024
Attention Dropout
0.0
SwiGLU Dropout
0.0
0.1
0.1
0.1
Drop Path Rate
0.0
0,25
A.5 More Experiments
A.5.1
Comparison with Recent Recurrent Architectures
Tables [1 and 12 compare PredFormer with two recent recurrent architectures, SwinLSTM and VMRNN, the Moving MNIST and TaxiBJ datasets, respectively. Across both benchmarks, PredFormer achieves lower MSE and higher SSIM while using comparable or fewer parameters and FLOPs. At the same time, PredFormer provides substantially faster training (shorter epoch time) and higher inference throughput (FPS), highlighting a favorable trade-off between accuracy and efficiency compared to these recurrent designs.
Table 11: Comparisons of PredFormer, SwinLSTM, and VMRNN on the Moving MNIST dataset.
Method
Paras (M)
Flops (G)
Epoch Time
MSE
SSIM
SWinLSTM
20.2
09.9
9min
17.7
0.962
VMRNN
18min
16.5
0.965
ProdPormer 3TSST Layer
1.5min
ProdFormer 6TSST Layer
25.3
16.5
3.5min
12.5
0.973
Table 12: Comparison of PredFormer, SwinLSTM, and VMRNN on the TaxiB.J dataset.
Method
Paras (M)
Flops (G)
Epoch Time
PPS
MSE
MAE
SSIM
SWinLSTM
2.9
1.3
1425
0.303
15.0
0.9843
VMRNN
2.6
0.9
Smin 1min
526
0.289
14.7
0.9858
ProdFormer
6.3
1.8
2354
0.277
14.3
0.9864
A.5.2 Ablation on Patch Size
To further analyze the effect of patch size, fix the architecture to the Triplet-TST variant of PredFormer and vary the patch size from 8 to 4 on Moving MNIST. As shown in Table 13 using smaller patch size (4 x 4) increases the number of spatial tokens from N = 64 to N 256, which leads to higher computational cost (FLOPs 67.6G vs. 16.4G) and lower throughput (FPS 110 vs. 165). This finer granularity yields only a modest improvement prediction accuracy (MSE 11.9 vs. 13.4, MAE 42.0 vs. 47.2, SSIM 0.974 vs. 0.971). Since the Triplet-TST variant with patch size already outperforms all recurrent and convolutional basclines by a clear margin, we adopt patch size the default setting in the main experiments to achieve better balance between accuracy and efficiency.
A.6 More Visualizations
Fig 7(a) and (b) depict the inflow and outflow at the same time step. this case, the fourth frame shows significantly less traffic flow than the previous frames. Constrained by the inductive bias of CNNs, TAU continues to predict high traffic levels while PredFormer demonstrates superior generalization by accurately
18

---

Published in Transactions on Machine Learning Research (01/2026)
Table 13: Patch size ablation of PredFormer (Triplet-TST) on Moving MNIST after training for 2000 epochs. Each model observes 10 frames and predicts the subsequent 10 frames with an input resolution of 64 x 64.
Patch Sixe Resolution Frames (in-tout) # #Patchhs N Variant
Paras(M) Flops(G)
PTS
MSE 4. MAE + SSIM
64 64
10
10
256
Triplet-TST Triplct-TST
25.3
67.6
110
11.9
42.0
0.974
64 64
10
- 10
64
25.3
16.4
165
13.4
47.2
0.971
Table 14: Dataset characteristics and optimal PredFormer variants. P denotes patch size, T is the input temporal length, N is the number of spatial tokens after patch embedding, and N/T measures the spatial- temporal token ratio. The "Best Variant(s)" and "Best MSE" are taken from the main results tables under the default hyperparameters.
Dataset
Resolution (Hf W) Patch Size P Patches N
N/T Ratio
Best Variant(s)
Rest MSR
Moving MNIST
6
Quadruplet- TSST, Quadruplet-STTS
Human3.6m
256 x 256
8
32 x 32
1024
4
256
Quadruplet-TSSS
110.9
TaxiBJ
32 x 32
8 X 64
16
Binary-ST, Triplet-STS
0.277
WentherBench
32 64
8 16
128
Far-T-S 11
1.100
capturing this abrupt change. This capability highlights PredFormer's potential to handle extreme cases, which could be particularly valuable in applications like traffic flow prediction and weather forecasting.
1-4
t-)
1-2
t-3
t-
hpptt
Irpux
L-5
1-6
1-7
1-8
t-5
1-6
1-7
1-8
Taeget
Tirget
PredFonmer Prodiction
PredFormer
Prodiction
TAU Prodiction
TudiB Inflow
(b) TauiBi OuElow
Figure Visualizations on TaxiBJ InFlow and OutFlow. We amplify the error for better comparison.
A.7 Empirical Discussion on Optimal PredFormer Variants Across Tasks
To better understand why different ProdFormer variants become optimal on different benchmarks, we sum- marize in Table 14 the key spatio-temporal characteristics of each dataset together with the best-performing PredFormer variant(s). As shown in Table 14 all four benchmarks favor factorized variants of PredFormer rather than the full-attention baseline, but the specific optimal pattern is dataset-dependentt:
On WeatherBench, the temporal horizon is the longest (T = 12) while the spatial fields are relatively smooth. In this setting, the Fac-T-S variant, which applies temporal attention before spatial attention, performs best, suggesting that emphasizing temporal modeling early is beneficial for long-range geophysical forecasting,
On Human3.66 the temporal window is short (T 4), but the spatial resolution very high (256 256 with N/T 256), and human motion tightly couples spatial joints and temporal evolution. Here, the Quadruplet-
19

---

Published in Transactions on Machine Learning Research (01/2026)
TSST variant, which interleaves two temporal and two spatial attention stages, works best, indicating that this dataset benefits from balanced treatment of temporal and spatial dependencies.
On TaxiBJ, the temporal horizon is also short 4), but the spatial grid is relatively low-resolution (32x x3,, N/T = 16) and dominated by structured traffic-flow patterns. In this regime, variants that allocate more capacity to spatial modeling at the end of the block (Binary-ST and Triplet-STS) perform best, suggesting that refining spatial correlations is especially important.
On Moving MNIST, both quadruplet variants (TSST and STTS) achieve the best results under the default patch size P = 8, indicating that a balanced four-stage factorization is robust on this synthetic but relatively long-horizon (T = 10) benchmark.
Overall, these results indicate that different spatio-temporal regimes (temporal horizon, spatial resolution, and the N/T ratio) naturally favor different. factorization patterns within PredFormer, which explains why the optimal variant is not universal but datasct-specific.
A.8 Theoretical Analysis on PredFormer Variants' Performance Differences
We consistently observe that TSST outperforms TS, which in turn outperforms TST on datasets such as Moving MNIST, TaxiBJ, and Human3.61M An exception occurs in WeatherBench, where this trend diverges due to severe overfitting. To analyze this phenomenon, we examine the representational capacity of temporal- first interleaved models. Unlike prior work that performs spatial-temporal attention factorization with shared MLP, our approach allocates dedicated SwiGLU FFN to each spatial and temporal attention block, enhancing the model's learning capacity and expressiveness. Then, the PredFormer encoder can be viewed as spatial-temporal transformer sequence (e.g., TSSTTSST). We propose that a key factor influencing model performance is the number of unique spatial-temporal subsequence (e.g., TS, TST, TSST) partitions enabled by a given sequence. Sequences with richer and more diverse partition patterns are better able to capture complex spatio-temporal dependencies. We formalize this intuition through a unique partition counting algorithm, as described in Fig [ and report the corresponding statistics in Tab 15 Notably, the number of unique partitions correlates well with empirical performance across configurations, offering plausible explanation for the effectiveness of TSST.
Step1: Enumerate all substrings of
I'TSST, TSS, 'SSTS, 'ST, 'TSSS, 'STTT. 'STST, TSTS, 'SSTT, 'TTS,, 'TS,, 'SS, 'STT, 'ST,, 'STSS, TSTT, TTST, 'STS, TTIS' 'SSST, 'TSSS, 'SST)
length 2-4 that contain buth S and T form complete of valid primitives.
egg. TSSTTSST
Input Sequence
Total unique partitions:
ITTS' 'ST, 'SS, IST), ITSS, 'ST, TSSTJ, I'TS, 'STT, 'SST], ITTS, STTSS ITSS,,
TTS. 'ST), ('TSST, TSST]]
Step2: Recursively find all possible partitions of the input sequence using only valid primitives.
c.g. TSTSTSTS
Total unique partitions:
[TTS. TS. 'SS, TSJ, I'TS, 'TS. TSTS' ITS., 'ST, 'STS', I'TST, 'ST, 'STS) ITSTS, TSTS|]
e.g. TSTIST
Step3: Normalize each partition as multisct and retain only scmantically unique ones.
Total unique partitions: 3
[[TS,, TTST), I'SST, (TSTT, 'ST]]
Figure 8: Temporal GTB Unique Partition Algorithm.
20

---

Published in Transactions on Machine Learning Research (01/2026)
Table 15: Analysis of Unique Partition Numbers.
Moving MNIST
Human Num
TaxiBJI Num f
Seq
Num
MSE
Seq
MSE
Seq
MSE
TSST*6
160
12.4
TSST"3
16
110.9
TSST*2
0.284
TS*12
116
12.8
TS*6
13
111.2
TS*4
0.283
TST*8
13.4
TST
112.4
TST*2
0.293
21


Reviews:
[{"review_id": "1PHZgPHjmV", "reviewer": "Reviewer_yZJo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes PredFormer, a pure transformer-based framework for video prediction that entirely eschews the use of recurrence and convolution, which are mainstays of prior work. The core idea is to leverage a simple encoder-decoder architecture where the encoder is composed of stacked \"Gated Transformer Blocks\" that process flattened video patches.\n\nContributions:\n- introduction of PredFormer\n- an exploration of nine different ways to model spatiotemporal dependencies using transformer blocks\n- demonstration of state-of-the-art performance across four diverse and standard benchmarks (Moving MNIST, Human3.6m, TaxiBJ, WeatherBench). The proposed model not only achieves higher accuracy but does so with significantly improved computational efficiency (fewer parameters, lower FLOPs, and faster inference) compared to leading RNN- and CNN-based methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Key Strengths:\n\n- Simplicity and Effectiveness: The paper argues for a simpler architectural paradigm for video prediction and backs it up with empirical results. The elegance of the pure transformer design is a major strength.\n- Well-Written and Clear: The paper is easy to read and well-structured. The figures effectively illustrate the different architectures.\n\nPotential Weaknesses:\n- Incomplete Efficiency Analysis: The paper reports FLOPs and FPS but omits a discussion on memory requirements (e.g., peak GPU memory usage), I notably assume that KV cache has a huge memory footprint on deployment.\n- Limited Theoretical Complexity Analysis: The computational complexity is primarily discussed through empirical metrics. A formal theoretical analysis of how the different variants scale with input dimensions would strengthen the paper's claims about efficiency.", "audience_yn": "Yes", "audience_explanation": "The claims of achieving state-of-the-art performance and superior efficiency seems well-supported, however I must disclose that I am not sure all the relevant baselines are present in the comparison as it's not directly my field.", "requested_changes": "1. Memory Usage Analysis (Necessary)\nThe current efficiency analysis focusing on FLOPs and FPS is good, however to provide a more complete picture, I suggest adding the peak GPU memory consumption during inference to the main results tables. This is a critical metric for practitioners evaluating models for real-world deployment, as I expect that the KV cache isn't light compared to RNN.\n\n2. Theoretical Complexity Analysis (Necessary)\nThe paper would be significantly strengthened by a more formal, theoretical complexity analysis to complement the empirical results. Please consider adding a section or a table that provides the Big-O complexity of the main architectural variants for memory and compute. The analysis should be in terms of key parameters like sequence length (T), number of patches (N), and hidden dimension (D). This would provide clearer insight into how each variant scales and formalize the efficiency claims.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tkN5wO4q1D", "reviewer": "Reviewer_cjnx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes PredFormer for video prediction. PredFormer utilizes a Gated Transformer Block to balance computational cost and performance for video prediction. Extensive experiments on four diverse benchmarks verify that PredFormer shows significant improvements in both accuracy and efficiency over strong baselines like SimVP and SwinLSTM.\n\nStrengths:\n1.The architecture of PredFormer is carefully designed. The paper provides a clear and systematic analysis of spatiotemporal factorization in transformers. The authors compare 9 distinct encoder architectures, including a baseline Full Attention, two standard factorized encoders (Fac-S-T, Fac-T-S), and six novel interleaved variants (Binary, Triplet, Quadruplet).\n\n2.The empirical results are promising. The paper demonstrates state-of-the-art performance across four video prediction benchmarks. Moreover, the qualitative visualizations show better predictions and noticeably lower error residuals compared to the TAU baseline.\n\nWeakness:\n1.The goal of motivation and the method are mismatched. The introduction emphasizes that prior models “struggle to balance computational cost and performance” and that Transformers suffer from “quadratic scaling with sequence length.” However, the methods do not analyze how PredFormer’s architecture achieves computational efficiency. Moreover, the author does not discuss how the interleaved attention variants reduce cost. \n\n2.The core component of PredFormer lacks novelty. The author presents GTB as a core component of PredFormer, which is widely used in former works [1-2]. However, the author does not discuss why this specific gating mechanism benefits spatiotemporal modeling in video prediction, which makes this component feel like a simple application rather than a novel contribution.\n\n3.The spatial–temporal separation design is not a new concept, which is similar to recent model design [3] or time-series application [4-5]. I recommend that the author clarify what distinctive advantage this design offers in video prediction.\n\n4.The claim of model variants lacks support. The author claims that “Triplet-TST captures more temporal dependencies, while Triplet-STS focuses more on spatial dependencies”. However, the paper does not provide any analytical or empirical evidence to support the claim.\n\n5.The analysis on the compared baseline is incomplete and overstated.  The paper asserts that “using CNNs … limits the model’s performance ceiling,” but the experiments only compare with older CNN-based methods (e.g., SimVP, TAU, PredRNN). The author does not compare PredFormer with recent recurrent architectures such as RWKV-TS [6] or CNNs with large receptive fields (e.g., Wavelet Convolutions [7]).\n \n6.The optimal variant on different tasks lacks explanation. Quadruplet-TSST is best for Moving MNIST, Triplet-STS is best for TaxiBJ, and Fac-T-S is best for WeatherBench. However, the paper gives no theoretical or empirical analysis of why this result occurs.\n\n7.Lack of ablation on patch size. The author mentions splitting videos into non-overlapping 3D patches in Sec. 3.1. However, the author does not provide any ablation study to justify their choice or analyze its impact on performance and efficiency.\n\n[1] Liu H, Dai Z, So D, et al. Pay attention to mlps[J]. Advances in neural information processing systems, 2021, 34: 9204-9215.\n\n[2] Shazeer N. Glu variants improve transformer[J]. arXiv preprint arXiv:2002.05202, 2020.\n\n[3] Tolstikhin I O, Houlsby N, Kolesnikov A, et al. Mlp-mixer: An all-mlp architecture for vision[J]. Advances in neural information processing systems, 2021, 34: 24261-24272.\n\n[4] Ekambaram V, Jati A, Nguyen N, et al. Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting[C]//Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining. 2023: 459-469.\n\n[5] Zhong Z, Yu Z, Yang Y, et al. PatchAD: A lightweight patch-based MLP-mixer for time series anomaly detection[J]. IEEE Transactions on Big Data, 2025.\n\n[6] Hou H, Yu F R. Rwkv-ts: Beyond traditional recurrent neural network for time series tasks[J]. arXiv preprint arXiv:2401.09093, 2024.\n\n[7] Finder S E, Amoyal R, Treister E, et al. Wavelet convolutions for large receptive fields[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 363-380.", "claims_and_evidence_yn": "No", "claims_explanation": "1. The claim of model variants lacks support. The author claims that “Triplet-TST captures more temporal dependencies, while Triplet-STS focuses more on spatial dependencies”. However, the paper does not provide any analytical or empirical evidence to support the claim.\n\n2. The analysis on the compared baseline is incomplete and overstated.  The paper asserts that “using CNNs … limits the model’s performance ceiling,” but the experiments only compare with older CNN-based methods (e.g., SimVP, TAU, PredRNN). The author does not compare PredFormer with recent recurrent architectures such as RWKV-TS [6] or CNNs with large receptive fields (e.g., Wavelet Convolutions [7]).", "audience_yn": "Yes", "audience_explanation": "1.The architecture of PredFormer is carefully designed. The paper provides a clear and systematic analysis of spatiotemporal factorization in transformers. The authors compare 9 distinct encoder architectures, including a baseline Full Attention, two standard factorized encoders (Fac-S-T, Fac-T-S), and six novel interleaved variants (Binary, Triplet, Quadruplet).\n\n2.The empirical results are promising. The paper demonstrates state-of-the-art performance across four video prediction benchmarks. Moreover, the qualitative visualizations show better predictions and noticeably lower error residuals compared to the TAU baseline.", "requested_changes": "1. clarification of the motivation and method.\n\n2. clarify the distinctive advantage of the method design in video prediction.\n\n3. provide empirical evidence to suuport the claim \"Triplet-TST captures more temporal dependencies, while Triplet-STS focuses more on spatial dependencies\".\n\n4. compare PredFormer with recent recurrent architectures such as RWKV-TS [6] or CNNs with large receptive fields (e.g., Wavelet Convolutions [7])\n\n5. add explanation for the optimal variant on different tasks\n\n6. add an ablation on patch size.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rN0klkrnjv", "reviewer": "Reviewer_Vzfr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a spatio-temporal factorization variant of transformer architecture, called PredFormer, designed specifically for video prediction. The objective is to overcome the limitations of ViT (Vision Transformer)-integrated RNN and CNN-based approaches for spatio-temporal prediction. The results demonstrate both better efficiency than RNN-based approaches, and better prediction performance than CNN based approaches for 4 different datasets. The study also provides a technical report ablating some of the key design choices such as Gated Transformer block and use of learnt vs pre-defined positional embeddings. The main takeaway for me seems to be the idea that the temporal and spatial blocks should be factorized and processed separately by attention variant, and the interleaving between these blocks depends on the dataset properties. Though the novelty of this work is limited, specifically no new novel architecture has been proposed, the paper evaluates a simple idea and culminates in a nice technical report which might be of interest to some practitioners.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides various efficiency and fidelity quantitative results on four different datasets, comparing against various benchmarks validating their claims. Moreover, ablations are provided for various design decisions.", "audience_yn": "Yes", "audience_explanation": "As reflected in the summary, though the novelty of this work is limited, the paper evaluates the simple idea of spatio-temporal factorization in transformer attention and culminates in a nice technical report which might be of interest to some practitioners.", "requested_changes": "Overall there are a few weaknesses of this work. The choice of datasets, though diverse, is limited and has relatively small scale to make broader claims with respect to the benefits achievable by such architecture changes. Moreover, all chosen datasets are of extremely small resolution and temporal horizons. I would tone down conclusion statement 2 as some of these relative conclusions between the PredFormer variants may just be an artifact of the chosen datasets. I do not expect authors to scale up their experiments for securing my recommendation but a clarification on the dataset sizes should be included in the conclusion given the broader claim is on the architecture. \n\nSecond, I feel a qualitative analysis on why certain variants led to better performance on specific datasets could have been nice addition given the emphasis over variants in the paper. This can also clarify the benefits achieved from the spatio-temporal factorization of the architecture. Authors do take a half-pass attempt in Appendix A.5 to theoretically understand this but this section is clearly not a full-finished work and can be more fleshed out. \n\nA few minor clarifications:\n- Figure 4 has a typo: “(b) TaxiBJ”. It doesn’t show results for this data.\n- Table 1: T, T’, interval definitions were unclear", "broader_impact_concerns": "Not applicable.", "additional_comments": "Not applicable.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)