PredFormer: Transformers Are Effective Spatial-Temporal Predictive Learners

Authors: Yujin Tang, Lu Qi, Fei Xie, Xiangtai Li, Chao Ma, Ming-Hsuan Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video Prediction, Spatio-temporal Prective Learning, Spatio-temporal Forecasting

Abstract:
Spatiotemporal predictive learning methods generally fall into two categories: recurrent-based approaches, which face challenges in parallelization and performance, and recurrent-free methods, which employ convolutional neural networks (CNNs) as encoder-decoder architectures. These methods benefit from strong inductive biases but often at the expense of scalability and generalization. This paper proposes PredFormer, a pure transformer-based framework for spatiotemporal predictive learning. Motivated by the Vision Transformers (ViT) design, PredFormer leverages carefully designed Gated Transformer blocks, following a comprehensive analysis of 3D attention mechanisms, including full-, factorized-, and interleaved- spatial-temporal attention. With its recurrent-free, transformer-based design, PredFormer is both simple and efficient, significantly outperforming previous methods by large margins. Extensive experiments on synthetic and real-world datasets demonstrate that PredFormer achieves state-of-the-art performance. On Moving MNIST, PredFormer achieves a 51.3% reduction in MSE relative to SimVP. For TaxiBJ, the model decreases MSE by 33.1% and boosts FPS from 533 to 2364. Additionally, on WeatherBench, it reduces MSE by 11.1% while enhancing FPS from 196 to 404. These performance gains in both accuracy and efficiency demonstrate PredFormer's potential for real-world applications. The source code and trained models will be made available to the public.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730091589551

thread_type:
official_review

<thread_object>
creator: reviewer_se64

<content>
summary:
This paper present a pure transformer-based architecture for spatiotemporal predictive learning. The authors conduct a detailed exploration of various design choices and training techniques. Their model demonstrates significant performance improvements over established baselines.

strengths:
- The manuscript is well-organized and clearly written.
- The paper offers a comprehensive analysis of architectural design choices and establishes strong baselines in the field of spatiotemporal predictive learning.
- Extensive experiments, including ablation studies, are performed, which highlight the advantages of the proposed PredFormer model.

weaknesses:
The technical novelty of the paper appears limited. For instance, the SwiGLU activation function, a core component of the model, is not new and has been explored extensively in other contexts. Additionally, factorized spatial-temporal attention is a well-known approach in video prediction, limiting the originality of the contribution.

questions:
- Are there any quantitative results regarding the efficiency of the proposed model compared to state-of-the-art recurrent-based methods (e.g., models incorporating Mamba-based architectures)?
- Could video generation models like Diffusion, GANs, or MagViT be adapted for spatiotemporal predictive tasks? How does PredFormer compare to these models in terms of performance and efficiency advantages?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648613133

thread_type:
official_review

<thread_object>
creator: reviewer_e793

<content>
summary:
The paper introduces PredFormer, a novel framework for spatiotemporal predictive learning that is based purely on transformers. Unlike traditional recurrent-based approaches that struggle with parallelization and performance, and recurrent-free methods like CNNs that can lack in scalability and generalization, PredFormer utilizes Gated Transformer blocks inspired by the design of Vision Transformers (ViT). The architecture is informed by a thorough examination of various 3D attention mechanisms—full, factorized, and interleaved spatial-temporal attention—which allows PredFormer to efficiently handle complex spatiotemporal data.

Key contributions of this work include:
- **Innovative Architecture**: A pure transformer-based model that is both simple and highly efficient.
- **Superior Performance**: Significant improvements over existing methods in terms of accuracy and efficiency, as evidenced by substantial reductions in Mean Squared Error (MSE) and increases in Frames Per Second (FPS) across multiple datasets.
- **Versatility Across Datasets**: Demonstrates state-of-the-art performance on diverse datasets, including synthetic (Moving MNIST) and real-world (TaxiBJ, WeatherBench) scenarios.
- **Public Availability**: Plans to release the source code and trained models to facilitate further research and practical applications.

These advancements highlight PredFormer's potential for broad application in areas requiring accurate and efficient spatiotemporal predictions.

strengths:
The strength of this paper lies in its novel introduction of PredFormer, a pure transformer-based model for spatiotemporal predictive learning. PredFormer innovatively integrates Gated Linear Units (GLUs) with self-attention mechanisms, effectively capturing complex spatiotemporal dynamics. The paper conducts a comprehensive analysis of different 3D attention mechanisms, leading to nine optimized configurations that address varying spatial and temporal resolutions. Extensive experiments on benchmark datasets (Moving MNIST, TaxiBJ, WeatherBench) demonstrate state-of-the-art performance, with significant improvements in accuracy and efficiency. Notably, PredFormer achieves a 51.3% reduction in MSE on Moving MNIST, a 33.1% reduction in MSE on TaxiBJ, and an 11.1% reduction in MSE on WeatherBench, while also offering faster inference speeds. The model's efficiency, broad applicability, and planned public availability of source code and trained models further enhance its impact and potential for real-world applications.

weaknesses:
- Firstly, PredFormer has not been evaluated on more complex and challenging datasets, such as Traffic4Cast for traffic flow prediction and Human3.6M for human motion forecasting. This limitation restricts the breadth of the model's validation and leaves questions about its performance on more intricate real-world scenarios. 

- Secondly, the experiments primarily use relatively small frame sizes, which makes it difficult to assess the model's effectiveness and scalability when applied to high-resolution data. These limitations suggest that further research is needed to fully demonstrate PredFormer's capabilities in handling complex and high-resolution spatiotemporal data.

- Additionally, the paper does not compare PredFormer with state-of-the-art transformer-based predictive learning methods, such as Earthformer. This omission limits the comprehensiveness of the performance evaluation and makes it harder to establish PredFormer's relative advantages and disadvantages in the context of the latest research. 

- As shown in Table 2 and Table 3, PredFormer does not demonstrate a significant advantage in running efficiency compared to recurrent-free methods, and its performance in terms of MSE is similar to these methods. This suggests that while PredFormer offers a novel and theoretically sound approach, its practical benefits in terms of efficiency and performance are not as pronounced as claimed.

questions:
- Why were more complex and challenging datasets, such as Traffic4Cast for traffic flow prediction and Human3.6M for human motion forecasting, not included in the evaluation? Could you provide insights into how PredFormer might perform on these datasets?

- The experiments primarily use relatively small frame sizes. I suggest that the paper should evaluate PredFormer's effectiveness and scalability when applied to high-resolution data.

- The paper does not compare PredFormer with state-of-the-art transformer-based predictive learning methods, such as Earthformer. Could you provide a comparison with these methods to better understand PredFormer's relative advantages and disadvantages?

- As shown in Table 2 and Table 3, PredFormer does not demonstrate a significant advantage in running efficiency compared to recurrent-free methods, and its performance in terms of MSE is similar. Could you explain why PredFormer does not show a more pronounced improvement in efficiency and performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692959864

thread_type:
official_review

<thread_object>
creator: reviewer_t3do

<content>
summary:
This paper proposes PredFormer, a transformer-based model for spatiotemporal predictive learning that replaces conventional recurrent and convolutional architectures with gated transformer blocks. PredFormer employs several encoder configurations—full attention, factorized, and interleaved models—to capture complex spatial and temporal relationships. The model reportedly achieves state-of-the-art results in accuracy and efficiency on multiple datasets, including Moving MNIST, WeatherBench, and TaxiBJ.

strengths:
1.	The paper is well-written and easy to follow.
2.	The model is evaluated on multiple synthetic and real-world datasets, which highlights its adaptability across different spatiotemporal tasks.
3.	PredFormer demonstrates a reduction in parameters and improved inference speed on specific configurations compared to previous methods, making it potentially applicable for real-time tasks.

weaknesses:
1.	The model’s design is primarily an adaptation of established concepts from Vision Transformers (ViTs) and gated transformers, without substantial innovation in model architecture or theoretical insight. This limits the paper’s contribution beyond incremental improvements.

2.	The authors should select a single architecture to compare against prior state-of-the-art methods for a fair assessment, as previous works typically present only one variation. Comparisons of different PredFormer architectures should instead be placed in the ablation study section.

3.	The best results for both recurrent-based and recurrent-free methods should be highlighted in bold. Some results from prior methods appear to outperform PredFormer, and this should be clearly indicated.

4.	I noticed that STCF [1] shows superior performance on the TaxiBJ dataset. Why was it not considered in the comparisons?

[1] STCF: Spatial-Temporal Contrasting for Fine-Grained Urban Flow Inference

questions:
Hope the author can answer the questions in the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712215357

thread_type:
official_review

<thread_object>
creator: reviewer_vtp1

<content>
summary:
The paper presents PredFormer, a pure transformer-based model for spatiotemporal predictive learning, designed to address challenges in balancing computation costs with predictive accuracy. PredFormer leverages Gated Transformer Blocks (GTBs) and a comprehensive exploration of attention mechanisms to surpass previous recurrent-based and convolutional encoder-decoder methods.

strengths:
- It boasts significant improvements in benchmark tasks involving datasets like Moving MNIST, TaxiBJ, and WeatherBench, showcasing both high accuracy and efficiency.
- The figures in the paper are clean and aesthetically pleasing, which enhances readability.

weaknesses:
- Emphasizing "the first pure transformer model in xxx domain" as a main contribution feels minor, given that ViT was introduced in ICLR 2021. Highlighting this three years later may not meet ICLR's innovation standards.
  - Noting that SwinLSTM[1] and OpenSTL[2] have already shown the capability of Transformer in this domain, which will further reduce the novelty of this work.
  - Especially the OpenSTL, which already incorporates and evaluates commonly used transformer or CNN blocks, i.e., ViT, Swin Transformer, Uniformer, MLP-Mixer, ConvMixer, Poolformer, ConvNeXt, VAN, HorNet, and MogaNet.

- The study of spatiotemporal block combinations has already been extensively explored in `various domains` by `numerous Transformer-based works`, such as **Latte** [3]. Therefore, the contribution in this aspect may appear somewhat weak or incremental. 
  - Due to the long sequence length problem of the Spatial-Temporal modeling task, most work delved into the architecture design of spatial block and temporal block, i.e., Latte also proposes 4 different spatial-temporal block variants, which are similar to this paper.

- The regularization and position encoding techniques used in this paper are well discovered in many language and vision papers. Utilizing them directly provides limited contribution.


[1] SwinLSTM: Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTM

[2] OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning

[3] Latte: Latent diffusion transformer for video generation

questions:
Please refer to the weakness part. I will raise my scores if the authors can resolve my concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731839429253

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Response

comment:
## **Overall Response:**

We thank the reviewers for acknowledging the strengths of our paper:

- Well-written content, clean figures, and easy-to-follow explanations.
- Significant performance improvements with high efficiency across multiple datasets.

We appreciate the constructive feedback from all reviewers and add additional experiments and more precise explanations of our contributions in the revised paper.

First, we address common questions and provide detailed responses to the questions posed by each reviewer.

### Novelty and Contribution (R#t3Do, R#sE64)

Compared to previous works based on RNN and CNN architectures, PredFormer introduces a simple yet effective pure-transformer framework for spatial-temporal predictive learning. While the transformer-based designs have been explored in various domains, we present the **meticulously designed gated transformer blocks that accommodate full-, factorized-, and interleaved- spatial-temporal attention**, that helps **achieving state-of-the-art results with high efficiency**, even in challenging real-world scenarios such as high-resolution datasets. Furthermore, our work provides comprehensive insights into spatiotemporal factorization, gated transformer block, positional encoding, and regularization techniques, offering advancements beyond prior Video ViT methods. Our work can serve as a strong baseline to promote spatial-temporal predictive learning. We would like to remind reviewers that this work is developed based on the  "*simplicity is the ultimate sophistication*" principle, and achieves state-of-the-art performance. 

### Task Difference between Video Prediction and Generation (R#vtP1, R#sE64)
- Video prediction and video generation differ in their goals and data characteristics. Video prediction emphasizes pixel-level accuracy, aiming to capture spatiotemporal relationships precisely and adhere to physical-world rules. In contrast, video generation focuses on unseen objects and scenes and the results are typically evaluated based on perceptual quality and consistency using metrics such as FVD. Additionally, videos typically have shorter temporal intervals and longer sequences, while real-world spatiotemporal data, such as traffic or weather datasets, often feature longer intervals and shorter temporal lengths. These distinctions result in different preferences for factorization approaches in the respective domains.
- Compared with Latte (a recent latent diffusion transformer for video generation), PredFormer introduces an effective spatiotemporal factorization method that not only differs from but also extends beyond prior Video ViT designs. Furthermore, the goals and findings of this work are significantly different from Latte, due to the specific characteristics of spatiotemporal predictive learning tasks, which differ from video generation. For example, we find that Fac-ST (spatial-first) performs worse than Fac-TS (temporal-first) for spatiotemporal predictive learning. 
- PredFormer, along with SimVP and PredRNN, belongs to the deterministic model category, which directly predicts pixel values. Deterministic models aim for precision and efficiency in spatiotemporal predictions. On the other hand, probabilistic models, such as GANs, VAEs, and diffusion models, operate in latent spaces and are designed to generate diverse outputs by sampling from probability distributions. These models often trade off inference speed for greater flexibility and diversity in generated results.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces PredFormer, a transformer-based model for spatiotemporal predictive learning, aiming to replace traditional recurrent and convolutional architectures with Gated Transformer Blocks (GTBs). The model leverages various attention mechanisms to capture complex spatial and temporal relationships, achieving strong results on datasets like Moving MNIST, TaxiBJ, and WeatherBench. 

The strengths of the paper include performance improvements in accuracy and efficiency, as highlighted by Reviewers e793 and t3Do. The model is evaluated on multiple datasets, demonstrating adaptability across different spatiotemporal tasks, as noted by Reviewer t3Do. The paper is well-written and organized, making it easy to follow, as mentioned by Reviewer sE64. Additionally, the extensive experiments and ablation studies highlight the advantages of the proposed model, and there are plans to release the source code and trained models, facilitating further research and practical applications.

However, the paper has several weaknesses. Reviewer vtP1 points out the limited novelty, as the model primarily adapts existing transformer concepts without substantial innovation. Reviewer e793 notes that the model has not been evaluated on more complex datasets like Traffic4Cast or Human3.6M, limiting its validation scope. Reviewer t3Do highlights the lack of comparison with some state-of-the-art methods, such as Earthformer, which limits the comprehensiveness of the evaluation. Additionally, PredFormer does not demonstrate a significant advantage in running efficiency compared to recurrent-free methods, and its performance in terms of MSE is similar, as noted by Reviewer e793. The contribution is seen as incremental, with the "first pure transformer model" claim feeling minor, as mentioned by Reviewer vtP1.

The AC aligns with the reject recommendation of all reviewers. Some concerns raised by the reviewers include that the contributions are primarily incremental, with limited innovation in model architecture or theoretical insights. The evaluation is insufficient, lacking complex datasets and comprehensive comparisons with state-of-the-art methods. The claimed improvements in efficiency and performance are not compelling enough, and the overall impact does not meet ICLR's acceptance threshold. The AC highly recommends the authors to address the concerns of the reviewers and take into account their suggestions of improvement when preparing a revised version. Especially the extra experiments promised in the rebuttal are crucial for justifying the method and could be included before resubmitting the work.