VideoTitans: Scalable Video Prediction with Integrated Short- and Long-term Memory
Authors: Young-Jae Park
Minseok Seo
Hae-Gon Jeon
Venue: NeurIPS 2025 poster ()
Keywords: Video Prediction
Memory-based architecture
Primary area: applications

Abstract: Accurate video forecasting enables autonomous vehicles to anticipate hazards, robotics and surveillance systems to predict human intent, and environmental models to issue timely warnings for extreme weather events. However, existing methods remain limited: transformers rely on global attention with quadratic complexity, making them impractical for high-resolution, long-horizon video prediction, while convolutional and recurrent networks suffer from short-range receptive fields and vanishing gradients, losing key information over extended sequences. To overcome these challenges, we introduce VideoTitans, the first architecture to adapt the gradient-driven Titans memory—originally designed for language modelling to video prediction. VideoTitans integrates three core ideas: (i) a sliding-window attention core that scales linearly with sequence length and spatial resolution, (ii) an episodic memory that dynamically retains only informative tokens based on a gradient-based surprise signal, and (iii) a small set of persistent tokens encoding task-specific priors that stabilize training and enhance generalization. Extensive experiments on Moving-MNIST, Human3.6M, TrafficBJ and WeatherBench benchmarks show that VideoTitans consistently reduces computation (FLOPs) and achieves competitive visual fidelity compared to state-of-the-art recurrent, convolutional, and efficient-transformer methods. Comprehensive ablations confirm that each proposed component contributes significantly.

TideoTitame Scalable Video Prediction with Integrated Short- and Long-term Memory
Young-Jay Park
Minseok Seo School of Electrical Engineering KAIST
Department of Al Convergence GIST
youngjae.. parkeg.. aast..cc.ta
Hae-Gon Jeon'
Department of Artificial Intelligence
Yonsei University earbol1@yonsei.co ac. kr
Abstract
Accurate video forecasting enables autonomous vehicles to anticipate hazards, robotics and surveillance systems to predict human intent, and environmental mod- els issue timely warnings for extreme weather events. However, existing methods remain limited: transformers rely on global attention with quadratic complexity, making them impractical for high-resolution, long-horizon video prediction, while convolutional and recurrent networks suffer from short-range receptive fields and vanishing gradients, losing key information over extended sequences. To over- come these challenges, we introduce VideoTitans, the first architecture to adapt the gradient-driven Titans eeemory-oriiinae designed for language modelling to video prediction. Video'Titan integrates three core ideas: (i) a sliding moeding-window window attention core that scales linearly with sequence length and spatial resolution, (ii) episodic memory that dynamically retains only informative tokens based on a gradient-based surprise signal, and (iii) a small set persistent tokens encod- ing task-specific priors that stabilize training and enhance generalization. Exten- sive experiments on Moving-MNISS MNIST, Human3.6M, M.6,, TrafficBJ and WeatherBench benchmarks show that Video Titans consistently reduces computation (FLOPs) and achieves competitive visual fidelity compared to state-of-the-a recurrent, convolu- tional, and efficient-transforme methods. Comprehensive ablations confirm that each proposed component contributes significantly.
Introduction
Accurate video forecasting enables proactive decision-making in critical real-world systems such
as autonomous driving 22
24
731, city-scale surveillance [15] 40 41], robotics [29 25| 52].
and weather forecasting 18
45
Predicting future video frames demands a delicate balance:
the model must precisely capture rapid, subtle changes between frames [6], yet retain memory of important events and scene dynamics
extended time horizons. Traditional approaches rely heavily
on convolutional 28 61 54 69] or recurrent architectures [66] 71), which handle local dynamics
effectively but face challenges due to limited receptive fields 7 47 and vanishing gradients (44 33
severely restricting their performance on long sequences.
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent transformer-base architectures address these limitations by employing global self-attention []] 32 9 21 capture long-range dependencies. However, this comes at the prohibitive cost of quadratic computational complexity 776 3, making them infeasible for realistic, high-resolution, long-sequenee applications. Attempts to circumvent this computational bottleneck such as hierarchical window attention [8), low-rank approximation [64], external memory introouruee rigid architectural constraints and task-specific heuristics, limiting their generalizability and flexibility across diverse forecasting scenarios.
An alternative approach emerges from recent advances in natural- -anguagg processing. Titans 21, gradient- episodic memory module, selectively commits information to memory only when its loss gradient signals substantial "surprise" -a a mechanism motivated by the way humans tend to remember unexpected or novel events [35]. This memory mechanism naturally aligns with video prediction tasks, where redundant frame sequences dominate, punctuated by critical rare events such as sudden object movements or abrupt camera motions. However, adapting Titans directly to video forecasting non-trivial: handling high-resolution frames substantially inflates memory complexity, standard transformer attention remains computational bottleneck, and visual forecasting benefits significantly from learned, static priors which episodic memory alone cannot provide.
In this paper, we introduce VideoTitans, the first architecture to successfully adapt the Titans gradient- driven memory the dense video forecasting domain. VideoTitam uniquely integrates three core components into a unified, computationally efficient framework: (i) lightweight sliding-window attention core whose complexity grows linearly with sequence length and spatial resolution, (ii) gradient-based episodic memory that selectively encodes surprising patch tokens, and (iii) small set of persistent tokens that inject input-agnostic, reusable priors into the prediction pipeline. The interplay of these modules is seamlessly coordinated by a single gating mechanism, ensuring the predictor remains end-to-end differentiable without reliance on manually tuned heuristics.
We conduct extensive evaluations across diverse and challenging Human3.6M 3.6M 226. TraffieBJ (74), and WeatherBench 45 200-mmmostttaiing that Video'Titan consis- tently reduces computational load while delivering superior visual fidelity in long- long-range forecasts compared sfate-of-the-art recurrent, convolutional, and aeficient-transfone prehensive ablation studies further verify that each proposed component is critical to achieving this performance. To facilitate further research and ensure full reproducibility, we will publicly release our source code, trained checkpoints, and demonstration videos.
(53]
methods. Our com-
Contributions
We demonstrate that gradient-driven Titans memory can be applied long, high-resolution video sequences without incurring quadratic computational growth, providing the first cross-domain evidence of its effectiveness beyond language.
architecture that balances efficiency and temporal We present a unified bidding-window attention, episodic memory and persistent priors coverage by combining behind a single gating mechanism.
Extensive experiments on Moving-MNNI -NIST, Human3.6M, TraffieBJ and WeatherBench show consistent reductions in computation and improvements in long-range visual fidelity over state-of-the- recurrent, convolutional, and trfeiennttransffomer baselines, while ablation studies confirm that every component of VideoTitaas is indispensable.
2
Related Works
2.1 Memory in RNNs and Transformers
Recurrent neural networks (RNNs) 77 36 53
and their variants, such as Long Short-Term Memory (LSTM) 223, 48 77 and Gated Recurrent Units (GRU) [10]
13 have been widely used for 16 7 These architectures utilize internal memory to retain past information, enabling them to capture rangrange dependencies 31) However, they suffer from vanishing gradients and sequential processing constraints, limiting their scalability to long video sequences [44] While various enhancements have been proposed to improve memory retention, RNN-based approaches remain computationally inefficient for high-dimensional spatio-temppral modeling [27 5).
11
modeling sequential dependencies video prediction [7
39

---

0.990
Smii
Videa Iaars
0.960
S5W
SSW
0.90
0.986
LITy
0.94
6.930
CHNLSTM
0.954
rangon
6.92
10t
101
102
Moving MNST
Toffic)
Human 3.6
GFLOPS Boy Scalet
GELOPs BLog Scale)
Datasets
(a) MMNIST: GFLOPsSSSIM
(b) TraffieBI: GFLOPs-SSIM
(c) SSIM Comparison
Figure 1: Performance and efficiency comparisons among video prediction models. Compared to other video prediction models on benchmark datasets, VideoTitanss achieves lower FLOPs(G) while delivering superior comparable performance.
Transformer-baet based architectures have gained prominence for their ability to model long-term depen- dencies through self-attention mechanisms, enabling parallelized sequence processing [62 60 20]
However, standard attention scales quadratically with sequence length, making it impractical for long video sequences. To address the issue, memory-aaggeentn transformers incorporate external memory to store and retrieve key representations, reducing computational overhead while preserving global contextual information [72 30). Despite these improvements, challenges remain in retrieval effi- ciency and adaptability to dynamic dependencies [59] [3 Our work builds upon these advancements by introducing a neural long-term memory module that selectively retains critical past information, improving both efficiency and predictive robustness for video forecasting.
2.2
Video Prediction
Video prediction involves forecasting future frames based on past observed frames by modeling intricate spatio-tempore dependencies. ConvLSTM [51] introduced convolutional recurrent units to jointly capture spatial and temporal contexts but struggled with long-tern stability. PredRNN [64] and its variants [65] 68) significantly improved temporal modeling by incorporating additional spatio- temporal memory units but came with considerable computational overhead. E3D-LSTM (66) further enhanced performance by integrating 3D convolutions, yet remained computationally demanding. PhyDNet [9 leveraged physical constraints better represent motion dynamics but limited in modeling highly complex scenarios.
Si 17 significantly simplified the prediction model by employing spatial-tempona separable architectures, balancing performance with computational efficiency. Building upon this, SimVP. meta (58) extended SP by integrating recurrent, convolutional, and transformer-ban tures into a unified meta-model framework, greatly advancing the field of video prediction. Following this work, the autoregressive-basel directions.
architec-
[50) model has further enriched the literature with promising
Inspired by these developments, paper introduces, for the first time, novel Titans-based 2 Siterrrreeeeeeeeee-------eeeeeee
designed enhance both long-term and short-term memory capabilities, thereby addressing critical challenges in video prediction tasks.
3 Preliminaries
Memory and Sequence Modeling. Sequential modeling tasks, such as video prediction, typically involve handling long-term temporal dependencies. Recurrent neural networks (RNNs) encode these dependencies into a compressed hidden state but often lose essential information over longer sequences. On the other hand, Transformers explicitly model dependencies using attention, but this comes with quadratic complexity, limiting their applicability to very long sequences such as videos.
Neural Memory and Adaptive Forgetting. Recent architectures introduce explicit neural memory modules that dynamically store historical context beyond the immediate attention window. These modules update memory state M, through recursive formulations such as:
M, f(,,,1,x),
(I)

---

Nanurall Long wer.e--mmmiii Edeeneery Alistiul
000000
Fogurr ng
Output
VideoTitans
Sharl-lerm Memory Mediale
Spatial
0.0 O C0g00
Positin Enoolinn
Patch Embedding
Persistent
00000000
Figure Overview of the VideoTitaas framework, which integrates neural long-term memory, sliding window attention, and persistent memory enhance video prediction. The model dynamically adapts to both short-term and long-term dependencies through a gradient- surprise mechanism. The final prediction is obtained by combining short-term attention and memory states via gating mechanism.
where f represents the memory update function that selectively encodes relevant information while employing adaptive forgetting mechanisms to discard redundant details. Such adaptive mechanisms are crucial to managing long-term dependencies without memory overflow.
Titans Framework. Titans proposes neural memory module specifically designed to dynamically learn, memorize, and retrieve crucial information. Titans define a gradient-based surprise mechanism determine the importance of events:
M, M + S,, S. - - V.V((M,11:)))
(2)
where S, captures both historical (past) and immediate (momentary) surprise. This allows Titans to dynamically adapt and selectively encode important events, balancing short-term attention and long-term memorization effectively.
Inspired by this approach, introduce Videolitans, adapting the Titans memory framework to video forecasting tasks, allowing efficient modeling of both local and global temporal dependencies inherent in video data.
4 Methodology
4.1 Problem Definition.
Given input video sequence X BBXTXXxHXX consisting of T observed frames, the goal of video forecasting accurately predict subsequent future video frames Y Bxxxxx xxW Here, B represents the batch size, denotes the number of observed frames, 7 is the number of future frames to predict, C' corresponds to the number of channels, and H, W indicate the spatial dimensions (height and width) of each frame. Formally, the task can defined as learning a function:
Y = F(X:),
where the model F,, parameterized by 6, aims to learn complex past video frames and leverage them generate precise, high-fidelity predictions for future frames. The challenge is that video data inherently contains both short-term dynamics (local correlations between consecutive frames) and long-term dependencies (slowly evolving periodic patterns across multiple frames), making the accurate modeling of both short-term and long-term temporal relationships critical for reliable predictions.
dependencies from
4.2 VideoTitans
Input Embedding and Positional Encoding. The input video sequence (B, T, C. H, W) is re-
shaped into (B x 7, C, H, W). Each frame is embedded into spatial patches with positional encoding (PE):
(B x C, W) + (B x T,, embed dim, H/16, W/16). then permute the embeddings into form suitable for temporal modeling: (B x T, embed dim, H/16, W/16) (B, H/16 W/16, T embed embed_dim).

---

MovingMNNSST
TrafheliJ
Human3.6
Method
MSR())
MAR(()
SSIM
(LOP%(()
MSE(()
MAR(U)
SSIM(T)
(L))
MSE(
MAR(]}
SSIM(
COST
29,7990
50.6396
0.9258
56.5
0.3358
15.1175
0.6836
20.74
125.220
1566.7080
0:9513
347.0
LSTM 0
0.9642
143
1442.4492
0:9925
542.0
22.5508
60.9673
0.9488
0.824
111
1463,4142
0.9630
PhyDNei
PicdRNN
15,3077
059338
0.9611
704.0
PeeIRNN22 [X
26.1136
73.7253
1166
0.3834
0.9876
42.63
114 8799
8729
0.9827
708.0
MAU
26.8364
78.2156
0.9396
17.3
0.3288
15,2582
0.5634
0.02
127.3176
1577,0112
0:9812
105.0
TAU
24,0229
71 9258
0.9454
16.0
0.1108
149941
(19649
2.49
113 3487
139906997
0.9630
182.0
SinVP-IncapU
[k]
32.1478
29.08998
0.9268
19.4
0.3282
15.4554
015635
3.61
115.8776
1511.4755
0.9822
197.0
SinVP-uSTA
26.00 6926
77.1883
0.9412
16.5
0.3247
15,0200
0.9644
2.62
108.0713
1444.5731
0.9633
74,6
SinVP-Swin 5H
29.0991
$4.11507
0.9331
15.4
0.3129
15,0699
0.58447
2.55
233.2033
1599.7281
0:9799
188.0
SinVR-UUndoreerr
30.3827
85.5 65.8719
0,9308
16.5
0.3368
15.1653
0.5844
2.71
116 3079
1497 66663
0:9634
211.0
SimVP-V VT
35,1473
95. 8649
0.9140
15.9
0.3171
15.1332
0.9841
2,80
136 3321
1603 1603.5026
0.9796
238,0
SinVP-PPollan Poollleoe
3949
118.4458
1484.1710
0.9827
VilkoTians
0.9898
3456
Table I: A performance comparison of VideoTitans with other approaches is conducted on three stan- dard benchmark datasets for future frame prediction. VideoTitans consistently achieves competitive results across Moving MNIST, TrafficJJ, and Human 3.6, which differ in characteristics.
Neural Long-term Memory Module. We introduce an adaptive neural long-term memory module based on gradient-ber based surprise mechanism. The memory state update rule at time is:
M2 = (1 - M +
where the surprise score S, computed by:
S. - V((M,
(5)
Here, parameters og Tr. and 0g control adaptive forgetting, surprise decay, and momentary surprise integration, respectively, enabling the selective memorization of crucial historical information.
Sliding Window Attention. To precisely model short-term temporal dependencies, sliding window attention applied to embedded input sequences:
Ys BeedinggiinoAAtteeenne (Xemb);
(6)
where Xemb denotes the reshaped spatial embeddings.
Persistent Memory. To encode task-specific and context-independen information, we incorporate persistent memory parameters P. These parameters are concatenated to the embedded input as follows:
X
P1. PN,,Il|Xx Xemb-
Decoding and Frame Reconstruction. The final prediction is obtained by decoding the combined representations of short-term attention and neural long-term memory through gating mechanism:
Y Decoder (o X M.)).
(8)
The decoded output Y reshaped back to the original video dimensions (B,T.C,H, C.H.W).
Experiments
In this section, we present extensive evaluations of our proposed VideoTitans architecture on widely adopted benchmarks for future frame pre- diction. Additionally, we analyze the effective- ness of each component of NiteoTitans through comprehensive ablation studies.
Dataset
Nuds
Neg.
[C.J,W1]
Moving MNIST 63
10,000
10,000
(1, 64, 64)
10
10
Traficck [74]
500
32)
Human3.6 [X]
73,404
8,582
(3, 128, 128)
4
4
Weatherhench
1,6
706
(1/2, 64)
Table 2: Summary of datasets used. Numin/ Niett
are sample counts. (C, H, W): input shape. T input/predicted frames.
Datasets We evaluate VideoTitans on four widely-used datasets for future frame prediction, summarized in Table 2) Moving MNIST (MMNIST) 53 TrafficBJ 74 Human 3.6 26), and
WeatherBench 445 Moving MNIST consists of synthetically generated video sequences depicting
5

---

two digits moving randomly within a constrained grid, making it ideal for evaluating models on simple nonlinear temporal dynamics. TrafficBJ contains real-world traffic flow data collected in Beijing, capturing complex urban spatio-temporal dynamics. Human 3.6M comprises high-resolution motion capture data of human activities, challenging the model to accurately capture subtle and intricate human motions. WeatherBench includes global meteorological variables such as temperature (t2m), wind fields (uv10), and total cloud cover (tcc), testing the model's capacity to handle complex global-scale spatio-temporal interactions.
Evaluation Metric We use widely adopted metrics to assess prediction quality and evaluate four different datasets. Specifically, MMNIST, TrafficBJ, and 6uman3.6 are evaluated using the Mean Square Error (MSE), Mean Absolute Error (MAE), and Structural Similarity Index Measure (SSIM). The weatherbench is evaluated with MSE, MAE, and Root Mean Square Error (RMSE).
Implementation Details Following 558 we optimize VideoTitans using the Adam optimizer and train with the Mean Squared Error (MSE) loss. We set the batch size to 8 for all experiments. The learning rate is adaptively adjusted using the epochs. Initial learning rates are selected from the set {10-2 5, x 10-3, 10,000550 5,5 10 10-4,10-11 10- and the pest-perrorming value used for each dataset. The total number of training epochs varies depending the dataset complexity and size. All experiments are implemented using Py PyTorch and conducted on 8 NVIDIA A100 GPUs. More details can be found in the supplementary material and the code.
scheduler with patience of 10
5.1 Quantitative results
The Moving MNIST dataset is characterized by simple yet highly nonlinear dy- namics involving continuous movements and interactions of digit shapes. Our VideoTitans effectively captures these nonlinear temporal dynamics, demonstrating state-of-the-art performance in long-term forecasting by adaptively preventing error accumulation. This indicates the robustness of our model in handling synthetic nonlinear trajectories. Detailed quantitative results are provided in the left column of Table
Moving MNIST
The TrafficJJ dataset contains real-world urban traffic sequences exhibiting complex spatio-temporal patterns and periodic fluetuations. VideoTitans successfully captures both short-term local variations and long-term global trends, achieving superior forecasting accuracy compared to transformer- and recurrent models. This highlights the applicability of VideoTitans dynamic urban traffic scenarios. Comprehensive performance comparisons are provided in the middle column of Table 1
TrafficBJ
Human 3.6M The Human 3.6M dataset includes sequences of articulated human motions captured under controlled conditions, demanding precise modeling of intricate spatio-temporal interactions. Video'iitans demonstrates robust performance by effectively modeling subtle short-term movements while maintaining long-term coherence in human motion sequences. These results underscore its strength in detailed human motion prediction. Full comparisons available in the right column of Table I
We further include results on the
[14] in the Supplementary Material.
WeatherBench The WeatherBench dataset in- volves forecasting long-range meteorological variables such as temperature (t2m), wind fields (uv10), and total cloud cover (tcc), character- ized by complex ppatio-temport dependencies and nonlinear interactions on global scale. Video Titans demonstrates robust performance on these variables, as detailed in Table achiev- ing competitive performance compared to pre- vious methods with lower FLOPs. This per- formance suggests that Viteo'iitans effectively captures large-scale dependencies inherent in meteorological data, making the model potentially suitable for weather forecasting tasks. Nonetheless, since VideoTitans does not explicitly model
GT
PredRNNv2
(e)
(d) Videa Vites
Figure 3: Qualitative comparison of predicted fu- ture frames on the Human3.6 6 dataset. Our model makes more accurate predictions, particularly ticeable the person's 's arm

---

12m
uv20
Ice
Methed
MSE()
MAE(U)
()
(
MSE(U
MAE()
RMSECD
FLOIS(G)
MSE(U)
MAE())
RMSE() FLOPs
1.3547
171
2,3940
0.9837
1.5473
3.3708
1.2363
18330
0.078
PNeDNet
250,9133
8.8492
17.0496
36.8
16,7869
29188
4,0972
36.8
0.0997
0.2261
0.3157
36.8
PreiRNK
1.7250
0.7967
1.1134
276
2,6378
10804
1.6441
270
0.0799
0.1503
0.2810
278
ProdkNN++
14575
0.7676
.2073
41.
2,5876
10348
1.5961
414
0.0997
0.1954
0.2824
413
PieIRNN22
1,7826
0.8074
1.3351
279
2,8991
1.3033
1.409)
280
0.0028
0.1874
0.2875
279
MAL
1.2413
0.6977
L1141
39.6
2,1530
0.9594
1.4673
39.6
0.0707
0.1715
0.2660
39,6
0.7056
1.3058
0.961
IncepU
1.3378
8.04
SigTA
1,0738
7.01
1.7272
0.8812
1.3142
7.00
0.1474
0.2166
SimVP-Swin
1,2235
0.6665
1.1061
6.88
1.5209
0.8688
1,2533
6.39
0.0899
0.1567
0.2425
6.88
Sim-P-Uniifre formerr
11948
0.6697
19930
7.45
1.4781
0,8099
1.2158
7.46
0.0561
0.1553
0.2368
7.45
SimVP VIT
1.2954
0.6873
L.1382
1.99
1.693
0.8512
1.2997
8.0
0.0615
0.1596
0.2480
7.99
SimVP- SinVV--Poollomeer
1.2525
0.6711
11111
3.61
1.6678
0.8427
1.2914
5.62
0.0562
0.1530
0.2571
5.61
Tinss
181822
0.0038
0.7984
1,1850
4.92
010544
4,92
Table 3: Performance comparison of VidaoTitans and state-of-the-art methods on the WeatherBench dataset for predicting temperature (t22m,, wind velocity (uv10), and total cloud cover (tcc). VideoTi- lans demonstrates competitive predictive accuracy across all variables, effectively capturing complex global spatio-temporal patterns inherent in weather forecasting tasks.
T-S
T
T-9
T=11
Inpan
T=15
T=17
T-19
T-21
T-21
GT
PredRNNv2
VIT
Video
Figure 4: Qualitative comparison of predicted total cloud cover (tcc) frames the WeatherBench dataset. Predictions from VideoTitans are compared against PredRNNv2 securrent-basede and ViT (transformer-based) Red boxes highlight regions where VideoTitans better preserves cloud patterns compared to other methods, indicating its capability to effectively model global spatio-temporal interactions in meteorological data.
the continuous fine-grained dynamics common atmospheric processes, additional architectural improvements could further enhance its predictive capabilities for subtle climate interactions.
5.2 Qualitative results
Human3.6 Figure 3 presents qualitative comparisons between VideoTitans and baseline methods on the Human 3.6 dataset. The superiority of VideoTitans is clearly evident, as it generates sharper and more accurate predictions for subtle and articulated human movements compared to recurrent (PredRNNv2) and transformer-based (ViT) methods. This highlights VideoTitans' capability to effectively capture detailed motion dynamics and maintain prediction quality.
WeatherBench We evaluate VideoTitans on the WeatherBench dataset using the standard protocol from OpenSTL. predicting weather variables at 1-hour intervals up to 12 hours into the future. The qualitative results in Figure visualize the predicted total cloud cover (tcc) at the reduced resolution following the OpenSTL standard experimental setup. Since our primary objective is general video
7

---

32
30.79
25
27
-0
26
23:51
converge
PLOPS (G)
23
24
22
21
21.33
21
20
10
Depth
Nemory Layers
Parsisent Nemory
(a) Depth vs. MSF & FLOPs
(b) Memory Layers vs. MSE
(c) Persistent Memory vs. MSE
Figure 5: Ablation analysis of ViteoTitans the Moving MNIST dataset. (a) illustrates how model memory depth affects both FLOPs and MSE, showing that moderate depth offers the best trade-off between efficiency and accuracy. (b) presents the effect of the number of neural memory layers, where performance peaks at two layers, while deeper configurations encounter training instability, (c) shows the impact of persistent memory parameters P, indicating that selecting the optimal value key to achieving the best performance.
prediction, we opt for this protocol; however, future work will extend our evaluation to the higher- resolution WeatherBench2 [46] dataset at its full spatial resolution (0.25-degree). This will allow us to better examine VideoTitans' capability in capturing finer-grained metcorological dynamics. Additional qualitative results across datasets are provided the Supplementary Material.
5.3 Ablation Study
Effect of Memory Depth We analyze the effect of varying the depth (number of attention blocks) of the VideoTittans on the Moving MNIST dataset in figure (a). Increasing the depth initially improves prediction accuracy, reaching the lowest MSE at a depth of 6. However, beyond this depth, we observe diminishing returns, as the MSE slightly increases depth suggesting trade-off between compu- tational complexity and prediction accuracy. These results indicate that depth of 6 achieves the opti- mal balance between model complexity (measured in FLOPs) and forecasting performance.
Hyperparameter
Value
Neural Memory Depth
Neural Demory Dim
512
Head
Momentum Order
Max Gradient Norm
1.0
Persistent Mem Tokens
4
Chunk Size
256
Segment Len
256
Long-term Mem Tokens
16
Table 4: Detailed hyperparameter configura- tion used for VideoTitans training.
Design of Neural memory We investigate the ef- fect of neural memory depth on the prediction performance of VideoTitans by varying the number of layers within the memory module from 1 to 4 in figure (b). The model achieves the best perfor- mance (MSE: 21.3265 with memory depth of 2. Despite extensive experimentation -including careful tuning of hyperparameters, gradient norm clipping, learning rate adjustments, and various initialization strategies- deeper memory modules (3 and layers) consistently faced severe training instabilities and failed to converge. This highlights critical trade-off between memory depth and training stability, indicating that very deep neural memory structures may require extensive and precise hyperparameter tuning or architectural modifications to ensure convergence and maintain stability.
Influence of Persistent Memory Parameter (P) Figure 5 (c) examines how varying the Persistent Memory Parameter (P) influences the prediction performance. The optimal performance (MSE: 21 21.3265) is achieved at P = 3. Smaller values (P = O or 1) and larger values (P P==5 5 or 10) degrade performance, suggesting that moderate value of balances model complexity and memory capacity for the best forecasting outcomes.
Effect of Persistent Memory Table evaluaaes the impact of including Persistent Memory in VideoTitan on the Moving MNIST dataset. The presence of Persistent Memory significantly reduces MSE from 23.5125 to 21.3265, indicating that Persistent Memory effectively helps the model retain critical temporal context, leading to improved prediction accuracy.

---

Attention Type
MSE
Method
MSE
Persistent Memory
MSE
Sliding Window
21.3
Memory Context
66.5201
With
21.3
Global Attention
24.4
Memory Gate
21.3265
Without
23.5
No Attention
30.7
Memory as Layer
24.4822
Table 5: Persistent Memory ab-
Table 6: Attention mechanism ablation on Moving MNIST.
Table 7: Comparison of VideoTi- tans performance with different memory integration strategies.
lation on Moving MNIST.
Impact of Attention Mechanism Table 6 compares the effect of different attention strate- gies-Sliding Window Attention, Global Attention, and no attention. Sliding Window Attention achieves the lowest MSE (21.3265), demonstrating its superior capability to effectively focus local temporal dynamics compared to Global Attention (MSE: 24.4822) or removing attention entirely (MSE: 30.79).
Memory Integration Strategies 7 the performance of VideoTitans on MMNIST using three different memory integration strategies of Titans 2 Memory as Context (MAC), Memory as Gate (MAG), and Memory as a Layer (MAL). MAG achieves significantly better performance (lowest MSE), clearly outperforming both MAC and MAL. This demonstrates that incorporating memory using gating mechanism is particularly effective at capturing and selectively integrating critical historical information, leading to superior video prediction results.
Limitation and Future Work
In this work, we extensively evaluate VideoTitans on the video prediction task. Although our proposed model demonstrates strong generalization across multiple diverse datasets, further studies should investigate the effectiveness of our method in broader vision tasks such as action recognition, video segmentation, and anomaly detection.
While diffusion-based models have recently shown strong performance in video generation, our focus is on video cnddiction-forecasting future frames based on observed inputs, rather than generating from noise or prompts. Diffusion models involve iterative denoising processes with significant computational cost, prioritizing high-quality synthesis. In contrast, video prediction demands temporal consistency and real-time efficiency. Therefore, we compare against models specifically designed for future frame prediction. Still, leveraging the high fidelity of diffusion models alongside the real-time efficiency of predictive models offers promising direction for long-range video forecasting.
Finally, we observe that despite the compelling strengths of the Titans concept, the choice of hyperparameters significantly impacts performance stability. Particularly, the neural memory module depth and gradient constraints crucial; exceeding two layers frequently causes numerical instability during training. One important contribution of this work is the identification and documentation of these critical hyperparameters (Table 4 enabling stable and reproducible implementations of VideoTitans in future research.
7 Conclusion
In this work, we propose VideoTittas, a neural architecture designed for spatio-temporal video prediction, effectively capturing both local motion dynamics and long-term dependencies. By in- tegrating three key components-Short Term Memory with attention- processing for recent frames, Long-Term Memory for selectively encoding and retrieving historical contexts, and Persistent Memory for mask-specific knowledge Viie--Viiiiiaaaaaaa efficiently models complex video sequences while maintaining computational scalability. Extensive experimental evaluations demonstrate that VideoTitans consistently outperforms CNN-based, Transformer-basee and recurrent models, achiev- ing superior predictive accuracy while significantly improving efficiency for long-term forecasting. These results underscore the effectiveness of Video Mitans as robust and scalable solution for video- based predictive modeling, paving the way for advancements in sal-world applications such as autonomous systems, surveillance, and robotics.

---

Acknowledgement. This work was partially supported by the National Research Foundation of Korea(NRE)) grant funded by the Korea govemmen(MSTT)RS-DD5) (S-200444444244 and Institute for Information & Communications Technology Planning & Evaluation (IIPP) grant funded by the Korea governmen((MSIT) NN..RS-2025-2544116 Development of a human foundation model for human-centric universal artificial intelligence and training of personnel).
References
[1] Anurag Amab, Mostafa Dehghami, Georg Heigold, Chen Sun. Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[2] Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663, 2024.
[3] Aydar Bulatov, Yuri Kuratov, Yermek Kapushev, and Mikhail Burtsev. Beyond attention: breaking the limits transformer context length with recurrent memory. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024.
[4]
Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv preprint arXiv:2006. / 1527, 2020.
[5] Wonmin Byeon, Thomas M Breuel, Federico Raue, and Marcus Liwicki. Scene labeling with Istm recurrent neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[6] Wonmin Byeon, Qin Wang. Rupesh Kumar Srivastava, and Petros Koumoutsakos. Contextvp: Fully context-aware video prediction. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[7] Lucy Chai, David Bau, Ser-Nam Lim, and Phillip Isola. What makes fake images detectable? understanding properties that generalize. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[8] Zheng Chang, Xinfeng Zhang. Shanshe Wang, Siwei Ma, Yan Ye, Xiang Xinguang, and Wen Gao. Mau: A aware unit for video prediction and beyond. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2021.
[9] Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vrsion-friendly transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[10] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NeurIPS Workshop on Deep Learning, 2014.
[11] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural networks, In Proceedings of the International Conference on Machine Learning (ICML), 2015.
[12] Hanming Deng, Yang Hua, Tao Song, Zongpu Zhang, Zhengui Xue, Ruhui Ma, Neil Robertson, and Haibing Guan. Object guided external memory network for video object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[13] Rahul Dey and Fathi M Salem. Gate-variants of gated recurrent unit (gru) neural networks. In International Midwest Symposium on Circuits and Systems (MWSCAS), 2017.
[14] Piotr Dollár, Christian Wojek, Bernt Schicle, and Pietro Perona. Pedestrian detection: A benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[15] Duarte Duque Henrique Santos, and Paulo Cortez. Prediction of abnormal behaviors for intelligent video surveillance systems. In 2007 IEEE Symposium on Computational Intelligence and Data Mining, pages 362-367. IEEE, 2007.
10

---

[16] Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. Proceedings of the International Conference Machine Learning (ICML), 2020.
[17] Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet better video prediction. In Proceedings of the IEEF/CVE Conference on Computer Vision and Puttern Recognition (CVPR). 2022.
[18] Zhihan Gao, Xingjian Shi, Boran Han, Hao Wang, Xiaoyong Jin, Danielle Maddix, Yi Zhu, Mu Li, and Yuyang Bernie Wang. Prediff: Precipitation nowcasting with latent diffusion models. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2023.
[19] Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[20] Xudong Guo, Xun Guo, and Yan Lu. Ssan: Separable self-attention network for video represen- tation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021.
[21] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 45(1):87-110, 2022.
[22] Simon Hecker, Dengxin Dai, and Luc Van Gool. Failure prediction for autonomous driving. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1792-1799. IEEE, 2018.
[23] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-17880 1780, 1997.
[24] Anthony Hu, Fergal Cotter, Nikhil Mohan, Corina Gurau, and Alex Kendall. Probabilistic future prediction for video scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[25] Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412. 14803, 2024.
[26] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 36(7):3325-13)3 13399, 2013.
[27] Nal Kalchbrenner, Ivo Danihelka, and Alex Graves. Grid long short-term memory. arXiv preprint arXiv: 150.0015 2015.
(28) Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[29] Jasmeen Kaur and Sukhendu Das. Future frame prediction a video sequence. arXiv preprint arXiv:2009.006 2020.
[30] Dongkyu Lee, Chandana Satya Prakash, Jack FitzGerald, and Jens Lehmann. Matter: Memory- augmented transformer using heterogeneous knowledge sources. In Findings of the Association for Computational Linguistics (ACL). 2024.
[31] Sunghoon Lim, Sun Jun Kim, YoungJae Park, and Nahyun Kwon. A deep learning- based time series model with missing value handling techniques to predict various types of liquid cargo traffic. Expert Systems with Applications, 184:1 15532, 2021.
[32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
11

---

[33] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. Proceedings of the International Conference on Learning Representations (ICLR), 2017.
[34] Ioannis Mademlis, Anastasios Tefas, Nikos Nikolaidis, and loannis Pitas. Summarization of human activity videos via low-rank approximation. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
[35] George Mandler. The structure of value: Accounting for taste. In Affect and cognition, pages 3-36. Psychology Press, 2014.
[36] Tomas Mikolov, Martin Karafíát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Recur- rent neural network based language model. In International Speech Communication Association (Interspeech), 2010.
[37] Marc Oliu, Javier Selva, and Sergio Escalera. Folded recurrent neural networks for future video prediction. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[38] Xuran Pan, Tianzuu Ye, Zhuofan Xia, Shiji Song, and Gao Huang. Slide-transformer: Hierar- chical vision transformer with local self-attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[39] Bo Pang. Kaiwen Zha, Hanwen Cao, Chen Shi, and Cewu Lu. Deep mnn framework for visual sequential applications. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2019.
[40] Jin-Hwi Park, Young-Jae Park, Junoh Lee, and Hae-Gon Jeon. Deviancenet: Learning to predict deviance from a large-scale geo-tagged dataset. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), 2022.
[41] Jin-Hwi Park, Young-Jae Park, Ilyung Cheong, Junoh Lee, Young Fun Huh, and Hae-Gon Jeon. What makes deviant places? IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 46(11))7405-7443 2024.
[42] Young-Jae Park, Minseok Seo, Doyi Kim, Hyeri Kim, Sanghoon Choi, Beomkyu Choi, Jeong- won Ryu, Sohce Son, Hac-Gon Jeon, and Yeji Choi. Long-term typhoon trajectory prediction: physics-conditioned approach without reanalysic sis data. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[43] Young-Jae Park, Doyi Kim, Minseok Sco, Hae- Gon Jeon, and Yeji Choi. Data-driven precipita- tion nowcasting using satellite imagery. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2025.
[44] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the International Conference on Machine Learning (ICML), 2013.
[45] Stephan Rasp. Peter D Dueben, Sebastian Scher, Jonathan A Weyn, Soukayna Mouatadid, and Nils Thuerey. Weatherhench: a benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems, 12(1) 1 1)2200000002200. 2020.
[46] Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russell, Alvaro Gonzaez- Vivian Yang. Rob Carver, Shreya Agrawal, et al. Weatherbench 2: benchmark for the next generation of data-driven global weather models. Journal of Advances in Modeling Earth Systems, 1660)e2033M5040019 2024.
[47] Mats I. Richter, Julius Schöning, Anna Wiedenroth, and Ulf Krumnack. Should you go deeper? optimizing convolutional neural network architectures without training. In IEEE International Conference on Machine Learning and Applications (ICMLA), 2021.
[48] Haşim Sak, Andrew Senior, and Françoise Beaufays. Long short-term memory based recur- rent neural network architectures for large vocabulary speech recognition. arXiv preprint arXiv:1402.1128, 2014.
12

---

[49] Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: local svm approach. In Proceedings of the IEEE International Conference on Pattern Recognition (ICPR).
IEEE, 2004.
[50] Minseok Seo, Hakjin Lee, Doyi Kim, and Junghoon Seo. Implicit stacked autoregressive model for video prediction. arXiv preprint arXiv:2303. 07849, 2023.
[51] Xingjian Shi, Zhourong Chen, Hao Wang. Dit- Yeung, Wii-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation noweasting. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2015.
[52] Gaurav Shrivastava and Abhinav Shrivastava. Video prediction by modeling videos as continu- ous multi-dimensional processes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[53] Nitish Srivastava, Elman Mansimov. and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In Proceedings of the International Conference on Machine Learning (ICML), 2015.
[54] Lin Sun, Kui Jia. Yan Yeung, and Bertram E Shi. Human action recognition using factor- ized spatio-temporal convolutional networks. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2015.
[55] Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In Proceedings of the International Conference on Machine Learning (ICML), 2011.
[56] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2014.
[57] Cheng Tan, Zhangyang Gao, Lirong Wu, Yongjie Xu, Jun Xia, Siyuan Li, and Stan Z Li. Temporal attention unit: Towards efficient spatiotemporal predictive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[58] Cheng Tan, Siyuan Li, Zhangyang Gao, Wenfei Guan, Zedong Wang, Zicheng Liu, Lirong Wu, and Stan Z Li. Openstl: A comprehensive benchmark of spatio-temporal predictive learning. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2023.
[59] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[60] Yi Tay. Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention for transformer models. In Proceedings of the International Conference on Machine Learning (ICMI). 2021.
[61] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2015.
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2017.
[63] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, arXiv preprint arXiv:2006.0477 2020.
[64] Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S Yu. Predrın: Recurrent neural networks for predictive learning using spatiotemporal Istms. Proceedings of the Neural Information Processing Systems (NeurIPS), 2017.
[65] Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and S Yu Philip. Predrn+++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
13

---

[66] Yunbo Wang, Lu Jiang, Ming-Hsuan Yang. Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d Istm: A model for video prediction and beyond. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[67] Yunbo Wang. Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, and Philip S Yu. Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[68] Yunbo Wang. Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang. S Yu Philip, and Ming- sheng Long. Predmn: A recurrent neural network for spatiotemporal predictive learning. IEEE Trunsactions on Puttern Analysis and Machine Intelligence (TPAMI), 45(2):2208- 2225, 2022.
[69] Philippe Weinzaepfel, Zaid Harchaoui, and Cordelia Schmid. Learning to track for spatio- temporal action localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2015.
[70] Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 12):2700-280 1989.
[71] Haixu Wu, Zhiyu Yao, Jianmin Wang, and Mingsheng Long. Motionrnn: flexible model for video prediction with spacetime-varying motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[72] Y Wu, Y Zhao, B Hu, P Minervini, P Stenetorp. and S Riedel. An efficient memory-augmented transformer for snowledge-intensive nlp tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
[73] Jiazhi Yang. Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, et al. Generalized predictive model for autonomous driving. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[74] Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-tempot residual networks for citywide crowd flows prediction. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2017.
[75] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[76] Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long- short transformer: Efficient transformers for language and vision. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2021.
[77] Xiaodan Zhu, Parinaz Sobihani, and Hongyu Guo. Long short-term memory over recursive structures. In Proceedings of the International Conference on Machine Learning (ICML), 2015.
14

---

A
Additional Dataset
A.1
Caltech-Pedestrian Dataset
The Caltech-Pedestricg [14] dataset presents challenging real-world urban scenarios involving diverse pedestrian movements, occlusions, and complex dynamics. It evaluated using metrics such as Mean Square Error (MSE), Structural Similarity Index Measure (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Learned Perceptual Image Patch Similarity (LPIPS) testing the robustmess and ac- curacy of predictive models. As shown in Table8 VideoTitans demonstrates competitive performance across these metrics, effectively capturing intricate pedestrian trajectories and spatial relationships. This highlights its practical applicability in dynamic environments, combining predictive accuracy with computational efficiency.
Method
MSE (L)
SSIM (T)
PSNR (*)
LPIPS (1)
FLOPS(G) (J)
ConVLSTM
139.6588
0.9345
27.4644
0.0857
595.0
E3D-LSTM
199.1374
0.9047
25.4612
0,1261
1004.0
MIM
123.9034
0.9410
28.1148
0.0642
1858.0
PhyDNet
310.6844
0.8615
23.2723
0.3218
40.4
PredRNN
129.3306
0.9375
27.8074
0.0745
1216.0
PredRNNv2
143.4366
0,9334
27.1864
0.0895
1223.0
MAU
177.4630
0.9174
26.1504
0.0969
172.0
TAU
128.9193
0.9458
27.8465
0.0551
80.0
SinCP-IncepU
160.2191
0.9338
26.8093
0.0675
60.6
VimVP-gSTA
127.7992
0.9456
27.9191
0.0577
96.3
SimVP-Swin
155.2470
0.9300
27.2542
0.0811
95.2
135.9496
0.9393
27.6607
0.0687
104.0
SimVP-Vi VP-ViT
146.3816
0.9380
27.4267
0.0666
155.0
PanoPPPoloorrer
153.3675
0.9334
27.3807
0,0700
79.8
VideoTitam
130.4290
0.9448
28,8861
0.0512
9.9
Table 8: Performance comparison on Caltech Pedestrian dataset.
A.2 KTH Dataset
The KTH [49 dataset is characterized by structured human actions and stable motion patterns which test model's ability to capture temporal dynamics and spatial coherence in controlled settings. As shown in Table 9
VideoTitans achieves state-of-the-art performance across all evaluation metrics including MSE, Mean Absolute Error (MAE), PSNR, and SSIM. combines high predictive accuracy with low computational complexity which confirms its practical effectiveness and shows its strength in modeling regular motion sequences with precision and efficiency.
Method
MSE (L)
MAE
PSNR (t)
SSIM (t)
FLOPs(G) (L)
ConvLSTM
47.65
445.5
26.99
0.8977
1368.0
F3D-LSTM
136.40
892.7
21.78
0.8153
217.0
MIM
40.73
380.8
27.78
0.9025
1099.0
PhyDNet
91.12
765.6
23.41
0.8322
93.6
PredRNN
41.07
380.6
27.95
0.9097
2800.0
PredRNNv2
39.57
368.8
28.01
0.9099
2815.0
MAU
$1.02
471.2
26.73
0.8945
399.0
TAU
45.32
421.7
27.10
0.9086
73.8
VinCPP-IncePU
41.11
397.1
27.46
0.9065
62.8
SimVP-gSTA
45.02
417.8
27.04
0.9049
76.8
SimVV-Swin
45.72
405.7
27.01
0,9039
75.9
SimVP-Uniformer
44.71
404.6
27.16
0.9058
78.3
VIMVP-VIT
56.57
459.3
26.19
0.8947
112.0
45.55
400.9
27.22
0.9065
63.6
VideoTitans
34.27
320.8
29.31
0.9197
50.9
Table Performance comparison on KTH dataset.
15

---

B
Qualitative Results
T-0
T-2
T-1
T-6
T-8
Figure shows qualitative comparisons be- tween Viteo'itaass recurrent (PredRNNv2), and transformer-based (ViT) methods the Moving MNIST dataset. Due to the dataset's 's relatively simple dynamics, models perform similarly well, making it challenging guish significant differences among predictions. Empirically, observe that differences primar- ily lie convergence speed rather than final per- formance, extending training epochs tends to improve accuracy for models. Nevertheless, VideoTitans consistently provides slightly more stable and accurate results. We also present qual- itative results for t2m and uv10 variables from the WeatherBench dataset. Further qualitative results of ViteoTitans also available as GIF animations for better visualization.
2
7
7
2
7
2
7
T-10
T-12
T-14
T-16
T-18
7
3
7
7
visually distin-
7
2
7
PeodRNN222
-
3
7
7
7
2
2
3
7
7
VII
2
2
VileeTTanx
-
3
7 7
7
2
2
Figure Qualitative comparison of predicted frames on the Moving MNIST dataset.
T-22
T
T-6
T-8
T-10
T-12
T=18
T=20
T-22
ProdRNNN22 E3DLSTM Cons STM
Unitormtn
VIT
Videoo
Figure 7: A qualitative comparison of predicted 2m temperature (t2m) frames on the WeatherBench dataset, comparing ViteoTitans" predictions with those of other video prediction models.
16

---

T-2
T-4
T-6
T-8
T-10
T-18
T-20
T-22
PredRNNW2 E3DLSTM Conv
Unifomee
Vidco
Figure 8: A qualitative comparison of predicted wind field (uv10) frames on the WeatherBench dataset, where VideoTitans' predictions are compared with those of other video prediction models.
17

---

C Implementation Details
C.1 Model Architecture
The architecture of VideoTitaas consists three main components: an encoder for spatial feature extraction, Titans-based temporal modeling module, and decoder for frame reconstruction.
The encoder captures spatial and low-level visual features from input frames. Given an input tensor of shape (B, T, C, H, W), each frame is independently processed by convolution-base patch embedding. Specifically, we employ a convolutional layer with kernel size 16 x 16 and stride 16, converting the input
Encoder.
(B H,C,H,W) (B T, embed_ dim,
H/16, W/16).
Afterward, spatial positional encodings are added to preserve positional information. The tensor is then reshaped for temporal processing:
(B x T, embed dim, H/16, W/16) + (B, x W/16, x embed embed_dim).
Titans-based Temporal Modeling. The temporal modeling is based on the Titans architecture, utilizing neural long-term memory that adaptively updates weights via a gradient-basee surprise metric, efficiently capturing essential temporal patterns. Key hyperparameters, such as memory depth, memory dimension, persistent memory tokens, and maximum gradient norm, are critical for stable training. particular, setting the maximum gradient norm to 1.0 prevents training instabilities such gradient explosions.
The Titans module processes embeddings in segments, employing sliding window attention model both local and global temporal dependencies. Persistent memory tokens encode context-independent knowledge enhance generalization across datasets.
Decoder. The decoder reconstructs predicted frames from the temporal features. Mirroring the encoder structure, it utilizes transpose convolutional layers (kernel size 16 16, stride 16) restore spatial dimensions:
(B, H/16 x W/16, T embed dim) - (B x T, C. H.
The decoded frames are reshaped to the original dimensions ground-truth.
(B,T, W) for comparison with
C.2 Training Procedure
We implement VideoTitans in PyTorch, using the Adam optimizer and Mean Squared Error (MSE) loss function. Key training parameters are summarized below:
Optimizer: Adam optimizer (3) 0.9, 83 = 0.999).
Learning Rate Scheduler: Cedtucel.ROPlatteed (patience=10 epochs), initial learning rate chosen from {10-2,5 5 x 10-3, 10 10-3,155000 5 x 10-4,10-4)
Batch Size: for all experiments.
Training Epochs: MMNIST (200 epochs), Caltech Pedestrian (100 epochs), Human3.6. TrafficBJ, WeatherBench (50 epochs each).
Additionally, apply the Exponential Moving Average (EMA) with decay of 0.995 during training to enhance model stability and generalization.
C.3 Hyperparameter Sensitivity
A key contribution of our study includes identifying sensitive hyperparameters essential for VideoTi- tans' stable training. Notably, removing gradient norm constraints (e.g., setting max gradient norm) caused training instabilities, and overly deep neural memory layers (depth > 2) frequently result in numerical instability. Careful hyperparameter tuning thus essential for robust training and optimal performance.
18

---

C.4 Memory Integration Strategies
There are three types of memory integration strategies in Titans: Memory Gate (MAG), Memory as Context (MAC), and Memory as a Layer (MAL). MAG uses a gating mechanism to dynamically combine short-term attention and long-term memory, allowing the model integrate previous knowl- edge adaptively. MAC retrieves past information from memory and appends it to the input sequence before processing with attention, enabling selective use historical data. MAL incorporates mem- ory as an independent processing layer before the attention, similar to traditional hybrid recurrent models. Among these approaches, MAG achieves the best performance by effectively balancing short-term precision with long-term recall, leading to its selection the baseline model for our work.
19

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The main contributions and scope of our work are explicitly described in the Introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
20

---

Justification:
The limitations of our work are addressed in the "Limitation and Future Work" section of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not present any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide both detailed descriptions and the source code to ensure the reprodueibility of our results.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The implementation details and code are provided to ensure reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
wtt.ssubmsiiioooooooioooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https c/nips.cc/publi///uuu
CodeeSumiiii wwabbbmssssiinii ssionpolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
22

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
The experimental settings are detailed in the main paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars or statistical significance tests are not reported due to the high computational cost associated with repeated runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the computational resources used in our experiments in the Imple- mentation Details section.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
This work conforms the NeurIPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The proposed model can benefit applications in autonomous systems and forecasting, but also acknowledge potential risks in misuse for surveillance.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
functioning correctly, harms that could arise when the
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [No]

---

Justification: We did not implement specific safeguards as our model is not designed for high-risk applications such as language generation facial synthesis. The model is intended for general-purposs video prediction tasks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite all datasets and bascline models used in our experiments, and ensure that their licenses and terms use are respected.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. con/dataees has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We provide code for our proposed model.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their
submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
25

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve research with human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This study does not involve human subjects or any form of crowdsourced data collection.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [No]
Justification: LLMs were used only for writing and editing purposes, not for methodological components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htcppp/s/aarrppsssc/orofeeeeeeee for what should or should be described.
26


Reviews:
[{"review_id": "UnfbvHvBqe", "reviewer": "Reviewer_nvj6", "summary": "The authors propose VideoTitans, an architecture for future frame prediction in videos, introducing three key components: (1) a Sliding-Window Attention Core that enables linear scaling in both temporal length and spatial resolution; (2) a Gradient-Driven Episodic Memory mechanism that stores only gradient-selected “surprising” patch tokens to support long-term memory; and (3) Persistent Task-Specific Priors that encode reusable knowledge to stabilize training. Extensive experiments across four small benchmarks (Moving-MNIST, TrafficBJ, Human3.6M, and WeatherBench) show that VideoTitans achieves competitive performance.", "questions": "1. Could the authors provide results on tasks or datasets with longer sequences to better demonstrate the claimed long-range modeling capabilities?\n2. How does VideoTitans perform when T > 1000 ? Any insights on memory usage in such settings?\n\nThese would directly address my main concern in Weaknesses.", "limitations": "yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Adapting the Titans gradient-based surprise memory to video is interesting.\n2. The technical details are described clearly.\n3. Detailed studies of memory depth, attention types, and integration strategies clarify design choices and trade-offs.\n\nWeaknesses:\nThe manuscript repeatedly emphasizes VideoTitans’ strength in modeling long-range dependencies over long sequences, yet all evaluation datasets use very short sequence lengths (T < 10), and the chosen frame-prediction tasks do not adequately demonstrate its long-range prediction capabilities.", "was_revised": true, "final_justification": "The evidence for long-range modeling in this work is unconvincing. In Moving-MNIST, a 10-frame input at 10 fps spans only 1 second—by any reasonable definition, this is not “long-term.” The spatial resolution is just 64×64, further limiting the complexity. For experiments on Human3.6M, the input length is only 4 frames, which likewise also cannot demonstrate long-range capability. In the discussion phase, the authors report a 100-frame test with MSE = 289.This is a decisive underperformance, and NeurIPS policy prevents updating visualizations to substantiate qualitative claims. Given these issues, I think this manuscript is not yet ready for publication.", "rebuttal": "We thank the reviewer for the helpful comments on long-range prediction and scalability. Based on comments, we address each of your questions with corresponding answers.\n\n**[Weakness1]** The manuscript repeatedly emphasizes VideoTitans’ strength in modeling long-range dependencies over long sequences, yet all evaluation datasets use very short sequence lengths (T < 10), and the chosen frame-prediction tasks do not adequately demonstrate its long-range prediction capabilities.\n\n**[Question1]** Could the authors provide results on tasks or datasets with longer sequences to better demonstrate the claimed long-range modeling capabilities?\n\n**[Answer for Weakness1 & Question1]** We agree and added a Moving‑MNIST‑100 benchmark (10 input + 100 prediction steps, image size 64 × 64), where longer-term predictions are obtained by sequentially applying our 10-step predictor iteratively 10 times. Results averaged over 3 random seeds are shown below (@ frame 10 and @ frame 100 denote the 10th and 100th predicted frames, respectively):\n\nModel              | MSE ↓ @ frame 10 | MSE ↓ @ frame 100\n------------------|------------------|-------------------\nSimVP‑Swin        | 35.2             | 480.3\nVideoTitans (ours)| **28.7**             |**289.6**\n\nVideoTitans maintains an error rate consistently less than 60% of SimVP‑Swin's error even after 100 steps, confirming stable long-range forecasting. The results of the benchmark are also used to address Question2 below.\n\n**[Question2]** How does VideoTitans perform when T > 1000 ? Any insights on memory usage in such settings?\n\n**[Answer for Question2]** Running video prediction models with sequence lengths T > 1000 poses significant challenges due to rapidly increasing computational complexity.  For example, processing 1,000-frame Moving-MNIST videos with VideoTitans (shape 1000×1×64×64) quickly exceeds manageable FLOPs limits even before accounting for gradients. The computational cost in a standard non-autoregressive setup rises from 173.553 GFLOPs at T = 100 to 1,490.638 GFLOPs at T = 1000.\n\nThe effective solution is to perform autoregressive rollouts using shorter predictions (e.g., 10-step segments). By doing so, long-range video predictions beyond conventional benchmarks become feasible, with each 10-frame rollout requiring only about 13 GFLOPs. VideoTitans’ neural long-term memory module retains only a small set of memory tokens between steps, preventing linear growth in computational cost and enabling stable, coherent predictions over extended horizons. Although autoregressive rollouts can suffer from error accumulation over longer prediction horizons, our results on the Moving‑MNIST‑100 benchmark (autoregressive) show improved accuracy and scalability compared to baseline methods, highlighting the architecture’s potential for long-term video forecasting."}, {"review_id": "uNGTrhsYVl", "reviewer": "Reviewer_yF1Q", "summary": "The paper introduces VideoTitans, a framework for scalable spatio-temporal video prediction leveraging a combination of sliding-window attention, episodic memory based on a gradient-driven surprise signal, and persistent tokens to integrate short- and long-term memory. While the architecture is clearly presented and the experimental evaluation covers multiple standard benchmarks, the memory mechanisms used are relatively basic and do not represent a significant leap over existing methods.", "questions": "Can the authors clarify whether VideoTitans can be easily extended to incorporate more advanced memory architectures or different encoder types?\n\nAre there scenarios where the proposed approach underperforms compared to recent transformer-based methods?\n\nHow sensitive is the performance to the specific design of the surprise signal and memory modules?", "limitations": "The authors include a limitations and future work section, but a more detailed discussion on the framework’s extensibility and potential bottlenecks would be valuable.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nThe integration of short-term and long-term memory with a surprise-driven mechanism is clearly explained.\n\nFLOPs reduction and computational efficiency are demonstrated across diverse benchmarks.\n\nThe paper allocates a clear section for limitations and future work, showing awareness of open issues.\n\nWeaknesses and Suggestions\n\nThe proposed memory integration methods are based on rather standard algorithms, and the paper does not present a substantial methodological advance or novel architecture over prior work.\n\nThe range of practical applications and depth of experimental analysis remain limited; results would be more convincing with comparison to the latest, most advanced video prediction models.\n\nThe exploration of alternative or more sophisticated memory mechanisms, or deployment with more complex encoders, is mentioned only as future work and not demonstrated.\n\nThe analysis of failure cases, or the framework's ability to generalize to more challenging scenarios, is not sufficiently addressed.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's insightful feedback. We have carefully addressed the points raised.\n\n**[Weakness1]** The proposed memory integration methods are based on rather standard algorithms, and the paper does not present a substantial methodological advance or novel architecture over prior work.\n\n**[Answer for Weakness1]** Prior work in video prediction has often leveraged Transformer-based architectures to leverage their expressive power, but the quadratic cost of attention with sequence length makes them inefficient for long videos. CNN- and ConvLSTM-based models, while efficient, suffer from limited receptive fields and fail to capture long-range temporal structure, leading to compounding errors—especially in scenarios with complex motion or long-term dynamics. To overcome these issues, we draw inspiration from the Titans framework, originally developed for language modeling.\n\nTitans was designed to preserve long-range dependencies through dynamic memory access, demonstrating strong performance in tasks like long-form text generation. However, video data poses distinct challenges due to its spatio-temporal nature and high dimensionality. VideoTitans addresses this by replacing discrete tokens with spatio-temporal patches—compact units encoding spatial and temporal structure—enabling richer context and broader receptive fields. To ensure scalability over long sequences, we pair localized attention with a memory mechanism adapted from Titans. While Titans uses a gating mechanism to combine sliding-window attention and long-term memory, we adapt this structure through patch-aware gating and attention mechanisms tailored to video dynamics. Unlike prior patch-based video models that depend on full space-time attention, our architecture maintains scalability and temporal coherence without processing the full sequence. These adaptations yield a video prediction framework that directly addresses the limitations of prior methods in receptive field coverage and long-horizon modeling.\n\n**[Weakness2]** The range of practical applications and depth of experimental analysis remain limited; results would be more convincing with comparison to the latest, most advanced video prediction models.\n\n**[Answer for Weakness2.]** We appreciate the reviewer’s suggestion to benchmark against more recent video prediction models. To that end, we conducted experiments on MovingMNIST, and trained VideoTitans for 2,000 epochs to follow the implementation and protocols of the respective baseline papers. Below, we provide results along with a brief description of each compared method.\n\nAdditional Comparison Methods:\n- VPTR [1]: An early transformer-based model for video prediction using a spatio-temporal transformer with attention. \n- MIMO-VP [2]: A video prediction model that adopts a Multi-In-Multi-Out (MIMO) architecture. It is based on a Transformer backbone augmented with convolutional modules such as 3D spatio-temporal blocks and a multi-output decoder.\n- PredFormer [3]: A recent state-of-the-art method using a pure transformer-based architecture, without any recurrence or convolution. It leverages global attention and achieves strong results across multiple datasets.\n\nModel                | FLOPs(G) ↓ | MAE ↓  | MSE ↓  | SSIM ↑\n---------------------|------------|--------|--------|--------\nVPTR (ICPR'22)       | 103.17     |   -    | 63.6   | 0.882\nMIMO-VP (AAAI'23)    | 66.5       | 51.6   | 17.7   | 0.964\nPredFormer (Arxiv'24)| 16.4       | 44.6   | 12.4   | 0.973\nVideoTitans (ours)      | **13.3**   | **42.7** | **11.9** | **0.976**\n\nWe note that the above results of comparison methods are directly referred from their original papers. These results demonstrate that VideoTitans achieves the lowest error (MSE/MAE) and the highest perceptual quality (SSIM), even with the lowest computational cost (FLOPs) among all compared models. Despite the strong performance of PredFormer, our method yields better accuracy and efficiency.\nThis confirms that VideoTitans is highly competitive not only with classical architectures but also with the latest transformer-based video prediction models, addressing concerns regarding relevance and comparison to state-of-the-art methods.\n\n[1] Ye, Xi, and Guillaume-Alexandre Bilodeau. \"Vptr: Efficient transformers for video prediction.\" 2022 26th International conference on pattern recognition (ICPR). IEEE, 2022.\n\n[2] Ning, Shuliang, et al. \"MIMO is all you need: A strong multi-in-multi-out baseline for video prediction.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 2. 2023.\n\n[3] Tang, Yujin, et al. \"Video Prediction Transformers without Recurrence or Convolution.\" arXiv preprint arXiv:2410.04733 (2024).\n\n**[Weakness3]** The exploration of alternative or more sophisticated memory mechanisms, or deployment with more complex encoders, is mentioned only as future work and not demonstrated.\n\n**[Question1]** Can the authors clarify whether VideoTitans can be easily extended to incorporate more advanced memory architectures or different encoder types?\n\n**[Answer for Weakness3 & Question1]** We acknowledge the reviewer’s valuable suggestions regarding the exploration of alternative memory mechanisms and encoder architectures. To address these concerns, we conducted additional experiments on MovingMNIST to demonstrate that VideoTitans is not only modular and extensible in principle but also in practice.\n\n**(1) Memory architectures:**\n We compared our NeuralMemory module with a Transformer-based memory module.\nMemory               | MSE (↓) | MAE (↓) | SSIM (↑) | FLOPs (G)\n---------------------|---------|---------|----------|-----------\nNeuralMemory (ours)  | 21.33   | 65.21   | 0.9463   | **13.33**\nTransformer Memory   | **21.28**   | **64.89**   | **0.9465**   | 19.85\n\nThe Transformer memory showed comparable performance to NeuralMemory (slight improvement in MSE and MAE, negligible SSIM change), but required significantly higher computational resources (\\~49% increase in FLOPs). Thus, while VideoTitans supports advanced memory structures, our choice balances accuracy and computational efficiency.\n\n**(2) Encoder architectures:**\nEncoder               | MSE (↓) | MAE (↓) | SSIM (↑) | FLOPs (G)\n----------------------|---------|---------|----------|-----------\nSimple Conv (ours)    | 21.33   | 65.21   | 0.9463   | **13.33**\nTransformer Encoder   | 20.89   | 63.88   | 0.9487   | 25.72\nConvNeXt Encoder      | **20.75**   | **63.27**   | **0.9494**   | 18.90\n\nTransformer encoder improved performance but nearly doubled computational cost (\\~93% increase in FLOPs). ConvNeXt encoder also significantly improved performance across all metrics, with a modest increase in computational cost (\\~42% higher FLOPs). These results demonstrate VideoTitans’ flexibility, allowing easy integration of diverse encoders to balance performance and computational complexity.\n\n**[Weakness4]** The analysis of failure cases, or the framework's ability to generalize to more challenging scenarios, is not sufficiently addressed.\n\n**[Answer for Weakness4]** We appreciate the reviewer’s comment regarding the analysis of failure cases. As with many video prediction models, a common failure case in the video prediction task is the gradual blurring of frames over time due to accumulated prediction errors, especially in the latter part of the sequence.\nTo address this, we provide evaluation results for two examples from the KTH dataset in the supplementary material. Despite this known limitation, VideoTitans consistently outperforms SimVP-gSTA, a strong baseline.\n\nIn particular, in the last predicted frame:\n- **kth_sample1:**\n  - SimVP-gSTA — PSNR: 27.12, SSIM: 0.634, MSE: 126.10  \n  - **VideoTitans — PSNR: 29.29, SSIM: 0.653, MSE: 76.56**\n- **kth_sample2:**\n  - SimVP-gSTA — PSNR: 23.59, SSIM: 0.641, MSE: 284.45  \n  - **VideoTitans — PSNR: 28.72, SSIM: 0.712, MSE: 87.23**\n\nThese results show that even in the most challenging failure cases, such as the last frame, VideoTitans delivers superior reconstruction quality, demonstrating better robustness over long prediction horizons.\n\n**[Question2]** Are there scenarios where the proposed approach underperforms compared to recent transformer-based methods?\n\n**[Answer for Question2]** We did not observe any clear scenarios in our experiments where VideoTitans significantly underperformed compared to recent transformer-based methods. However, we observed that both VideoTitans and the ViT-based baseline achieved identical MSE errors of 8.8 at the initial prediction frame (t=1) on the Moving-MNIST benchmark. This implies that for very short-term predictions, transformer-based methods—which utilize global self-attention—can match our performance, as global attention can effectively capture immediate spatial relationships in early frames. Nonetheless, VideoTitans demonstrates clear advantages at longer prediction horizons due to its specialized memory integration and selective storage mechanisms.\n\n**[Question3]**. How sensitive is the performance to the specific design of the surprise signal and memory modules?\n\n**[Answer for Question3]** Performance is indeed quite sensitive to the specific design choices of the surprise signal and memory modules. As shown in Figure 5(b) in the main paper, varying the number of memory layers significantly impacts convergence, with deeper memory configurations (e.g., >2 layers) often failing to converge. Additionally, we observed that carefully tuning hyperparameters—such as the gradient clipping threshold, learning rates, and memory dimensions—was essential for achieving stable training and optimal performance. This sensitivity highlights the importance of careful architectural and hyperparameter selection when using Titans memory in video prediction tasks. We will mention it as our limitation and future work."}, {"review_id": "5KlZhORnWI", "reviewer": "Reviewer_akfP", "summary": "This paper presents VideoTitans, a video prediction model that adapts the Titans memory framework—originally proposed for language modeling—to the domain of video prediction. VideoTitans is an direct use of Titans for video prediction. Authors directly suit Titans with video input and adopt the adaptive long-term neural memory, sliding window attention and persisitent memory to achieve this goal. The paper evaluates VideoTitans across four+two video datasets, achieving competitive or superior results compared to recurrent, convolutional, and transformer-based baselines.", "questions": "I list my questions in Weaknesses.", "limitations": "While the proposed VideoTitans framework offers a scalable approach to video prediction, it suffers from several key limitations. Most notably, the model directly adopts the Titans memory architecture from natural language processing without any adaptation to the spatio-temporal structure of video data. This raises concerns about its ability to capture spatial locality and motion hierarchies effectively.  Additionally, although the paper emphasizes computational efficiency, it lacks a resolution sensitivity study and omits important runtime metrics such as model size, training time, and frames per second (FPS).  \n\nI understand that adapting the Titans architecture from NLP to video prediction is an interesting and potentially impactful idea. However, the current implementation—VideoTitans—does not demonstrate a significant performance advantage over existing video prediction methods. Moreover, the paper does not report critical efficiency metrics, such as the number of model parameters or inference speed (e.g., frames per second), which are essential to substantiate claims of scalability and practicality. As a result, the current version appears promising in concept but leaves substantial room for improvement.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The idea of leveraging gradient-based memory originally developed for NLP (Titans) in video prediction is potentially impactful, particularly in managing long-range dependencies efficiently.\n\n2. VideoTitans demonstrates reduced FLOPs across all evaluated datasets compared to strong baselines, which is significant for scalability to videos.\n\nWeaknesses:\n\n1. A major limitation is that the paper directly ports the Titans memory module from NLP to video prediction with minimal adaptation. All key components—adaptive long-term memory, sliding window attention, and persistent memory—are used as-is. There is no domain-specific modification to account for the spatio-temporal nature of video data. This raises concerns: (1)How effectively can Titans model spatial locality or hierarchical motion patterns? and (2)  Why does this direct transplant work for videos without domain-specific tuning? \n\n2. Although the authors highlight linear scaling and computational savings, there is no systematic study of resolution sensitivity. \n\n3. Beyond FLOPs, the paper does not report other important efficiency metrics such as model parameters, training time, or inference speed (e.g., FPS). These are necessary for a more complete assessment of efficiency.\n\n4. There are a lot of minor errors or inconsistencies. For example, line-150: Neural Long-term Memory Module or long-term neural memory module? Line-171: five widely-used datasets but only four in Table 2. In Table A (Suppl.), the second best result of LPIPS should be 0.0551 not 0.0577.", "was_revised": true, "final_justification": "Thanks for the detailed response. Across the two rounds of review and rebuttal, the authors have adequately addressed my concerns, especially regarding the adaptation of NLP Titans to videos. I also agree with reviewer nvj6 that the long videos used in the experiments may not be sufficiently long to convincingly demonstrate the model’s long-range modeling capability. Additional experiments are needed to provide stronger evidence in this regard. Based on the current submission, my overall recommendation remains borderline.", "rebuttal": "We thank the reviewer for careful evaluation and helpful suggestions. All of your feedback has been thoroughly reviewed.\n\n**[Weakness1]** A major limitation is that the paper directly ports the Titans memory module from NLP to video prediction with minimal adaptation. All key components—adaptive long-term memory, sliding window attention, and persistent memory—are used as-is. There is no domain-specific modification to account for the spatio-temporal nature of video data. This raises concerns: (1)How effectively can Titans model spatial locality or hierarchical motion patterns? and (2) Why does this direct transplant work for videos without domain-specific tuning?\n\n**[Answer for Weakness1]** We would like to respectfully clarify that the adaptation of Titans memory to the video domain involved substantial domain-specific modifications beyond its original NLP formulation. In particular, several changes were essential to successfully tailoring the memory architecture for video prediction:\n\n- **Complex patch embedding and reshaping**:  \n  Unlike NLP’s straightforward 1D tokens, video data required converting high-dimensional tensors (frames × channels × height × width) into spatial patches, followed by meticulous reshaping into meaningful 1D embeddings suitable for attention modules. Without these structural modifications, applying Titans directly would lead to prohibitive memory usage and unstable convergence.\n\n- **Patch-level gradient-based memory commitment**:  \n  The gradient-based surprise mechanism, originally designed for NLP’s token-level representation, was structurally modified to operate at the video-specific patch level. Without this adaptation, the method would fail to capture the spatial-temporal redundancy and complexity inherent in video data, causing inefficient memory usage and suboptimal predictive performance\n\nThese domain-specific structural adaptations were key reasons VideoTitans successfully modeled spatial locality and hierarchical motion patterns, clearly going beyond a direct transplant of Titans memory. We will explicitly emphasize these structural distinctions in our revised manuscript.\n\n\n**[Weakness2]** Although the authors highlight linear scaling and computational savings, there is no systematic study of resolution sensitivity.\n\n**[Answer for Weakness2]** Thank you for this valuable suggestion. To address your concern, we conducted an additional resolution sensitivity analysis, systematically varying input resolutions and measuring computational complexity. Our experiments clearly show that VideoTitans consistently achieves lower FLOPs and better scalability compared to SimVP across various input resolutions:\n\nInput Shape           | VideoTitans FLOPs (G) ↓ | SimVP FLOPs (G) ↓\n----------------------|--------------------------|--------------------\n(1, 10, 1, 64, 64)     | 13.33                   | 16.54\n(1, 10, 1, 128, 128)   | 60.81                   | 167.38\n(1, 10, 1, 256, 256)   | 376.13                  | 669.52\n\nThese results confirm the linear computational scaling and significant efficiency advantage of VideoTitans over SimVP. Importantly, achieving substantially lower computational complexity compared to SimVP—despite SimVP’s inherently simpler CNN-based architecture—highlights the exceptional efficiency and scalability of our proposed memory integration strategy. We will include this systematic comparison and discussion in the revised manuscript.\n\n**[Weakness3]** Beyond FLOPs, the paper does not report other important efficiency metrics such as model parameters, training time, or inference speed (e.g., FPS). These are necessary for a more complete assessment of efficiency.\n\n**[Answer for Weakness3]** We appreciate this suggestion. To address your concern, we expanded our efficiency analysis by reporting inference speed (FPS), the number of parameters, training time, as well as FLOPs. As shown clearly in the table below, VideoTitans achieves strong efficiency metrics compared to baseline methods, providing significantly faster inference speed, lower parameters, and fewer FLOPs, while maintaining better predictive accuracy. These results clearly demonstrate the efficiency advantages of our proposed model on the MovingMNIST dataset. \n\n| Model             | FPS ↑ | Params ↓ | Training Time ↓ | FLOPs (G) ↓ | MSE ↓     |\n|-------------------|--------|-----------|------------------|--------------|-----------|\n| ConvLSTM          | 113    | 15.0M     | 1d 2h 50m        | 56.8         | 29.7990    |\n| E3D-LSTM          | 18     | 51.0M     | 8d 8h 47m        | 298.9        | 38.5383   |\n| MIM               | 37     | 38.0M     | 1d 16h 3m        | 179.2        | 22.5508   |\n| PhyDNet           | 182    | **3.1M**      | 9d 11h 19m       | 15.3         | 28.1955   |\n| PredRNN           | 54     | 23.8M     | 1d 5h 27m        | 116.0        | 23.9667   |\n| PredRNNv2         | 52     | 23.9M     | 1d 23h 46m       | 116.6        | 24.1136   |\n| MAU               | 201    | 4.5M      | 16d 13h 4m       | 17.8         | 26.8564   |\n| TAU               | 283    | 44.7M     | 5d 6h 34m        | 16.0         | 24.6029   |\n| SimVP-IncepU      | 209    | 58.0M     | 1d 8h 48m        | 19.4         | 32.1478   |\n| SimVP-gSTA        | 282    | 46.8M     | 9h 47m           | 16.5         | 26.6926   |\n| SimVP-Swin        | 294    | 46.1M     | **6h 54m**           | 16.4         | 29.6991   |\n| SimVP-Uniformer   | 296    | 44.8M     | 7h 9m            | 16.5         | 30.3827   |\n| SimVP-ViT         | 290    | 46.1M     | 1d 1h 46m        | 16.9         | 35.1473   |\n| SimVP-Poolformer  | 341    | 37.1M     | 1d 2h 28m        | 14.1         | 31.7882   |\n| VideoTitans (ours)   | **367**| 6.0M  |9h 14m       | **13.33**    | **21.3265** |\n\n**[Weakness4]** There are a lot of minor errors or inconsistencies. For example, line-150: Neural Long-term Memory Module or long-term neural memory module? Line-171: five widely-used datasets but only four in Table 2. In Table A (Suppl.), the second best result of LPIPS should be 0.0551 not 0.0577.\n\n**[Answer for Weakness4]** We thank the reviewer for pointing out the minor inconsistencies.\n- Line 150: The correct term is \"Neural Long-term Memory Module\", and we will ensure consistent usage throughout the paper.\n- Line 171: The mention of \"five widely-used datasets\" was incorrect — the correct number is four, as reflected in Table 2.\n- Table A (Supplementary): We confirm that the second-best LPIPS score should be 0.0551, not 0.0577. This will be corrected in the revised version.\n\nWe will carefully revise the manuscript and supplementary material to address these issues and ensure consistency."}]

Author rebuttal:

Meta-review: ## Summary

This paper proposes VideoTitans, an approach for video prediction (predicting future frames conditioned on past frames), the main idea is to adapt the gradient-based memory originally developed for NLP (Titans) into video prediction. Experiments are conducted on various benchmarks to demonstrate the efficiency of the proposed approach in both accuracy metrics (MSE, MAE) and FLOPs.

## Ratings
After rebuttal and discussion, 1 reviewer recommends a rejection and two other reviewers recommend borderline acceptance.

## Reasons to accept the paper
* The idea of adapting Titans architecture from NLP into video prediction is interesting and has potential impact.
* Detailed ablation studies of memory depth, attention types, and integration strategies clarify design choices and trade-offs.
* The FLOPs reduction and computational efficiency are demonstrated across diverse benchmarks.

## Reasons to reject the paper
* The experimental setup for long-range video prediction is unconvincing.
* A limitation is that the paper directly ports the Titans memory module from NLP to video prediction with minimal adaptation. There is no domain-specific modification to account for the spatiotemporal nature of video data.
* Missing comparison with the latest, most advanced video prediction models.


## Discussions
- During author-reviewers, reviewers-AC discussion, the author(s) had addressed most of the concerns including comparing with latest methods, explaining the adopting / changing from NLP Titans.
- AC finds the author(s) response with respect to the long-range video prediction is reasonable as it follows the standard setup used in previous work.
- The author(s) response in explaining the technical contributions in adopting NLP Titans to video prediction is still borderline, there are some domain specific adaptations.

## Decision
AC reads all reviews and joins in the discussions with reviewers. AC believes that the potential and solid performance can outweigh the limitation of novelty. The paper is further discussed with SAC and SAC also agrees to accept the paper. Both SAC and AC recommend to accept the paper. 

To the author(s): please incorporate all feedback from reviewers in your final camera-ready version.
Decision: Accept (poster)