OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning
Authors: Cheng Tan
Siyuan Li
Zhangyang Gao
Wenfei Guan
Zedong Wang
Zicheng Liu
Lirong Wu
Stan Z. Li
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: spatio-temporal predictive learning
self-supervised learning
computer vision
TL;DR: This paper proposes OpenSTL, a comprehensive benchmark and open-source toolbox for spatio-temporal predictive learning that supports 14 state-of-the-art recurrent-based and recurrent-free models.

Abstract: Spatio-temporal predictive learning is a learning paradigm that enables models to learn spatial and temporal patterns by predicting future frames from given past frames in an unsupervised manner. Despite remarkable progress in recent years, a lack of systematic understanding persists due to the diverse settings, complex implementation, and difficult reproducibility. Without standardization, comparisons can be unfair and insights inconclusive. To address this dilemma, we propose OpenSTL, a comprehensive benchmark for spatio-temporal predictive learning that categorizes prevalent approaches into recurrent-based and recurrent-free models. OpenSTL provides a modular and extensible framework implementing various state-of-the-art methods. We conduct standard evaluations on datasets across various domains, including synthetic moving object trajectory, human motion, driving scenes, traffic flow, and weather forecasting. Based on our observations, we provide a detailed analysis of how model architecture and dataset properties affect spatio-temporal predictive learning performance. Surprisingly, we find that recurrent-free models achieve a good balance between efficiency and performance than recurrent models. Thus, we further extend the common MetaFormers to boost recurrent-free spatial-temporal predictive learning. We open-source the code and models at https://github.com/chengtan9907/OpenSTL.

OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning
Cheng Tan 1.2- Siyuan Li Zhangyang Gao 1.2 Wenfei Guan
Zedong Wang"
Zicheng Liu Lirong Wu Stan Z. Li
'Zhejiang University; XXidian University;
AA Lab, Research Center for Industries of the Future, Westlake University (tancheng; lisiyuan; gaozhangyang; guanwenfei; wangzedong; liuzicheng; wulirong; stan. wi..siioesttllke........ edu.
Abstract
Spatio-temporal predictive learning a learning paradigm that enables models to learn spatial and temporal patterns by predicting future frames from given past frames in an unsupervised manner. Despite remarkable progress recent years, lack of systematic understanding persists due the diverse settings, complex implementation, and difficult reproducibility. Without standardization, comparisons be unfair and insights inconclusive. To address this dilemma, propose OpenSTL, comprehensive benchmark for spatio-temporal predictive learning that categorizes prevalent approaches into recurrent-based and recurrent-free models. OpenSTL provides a modular and extensible framework implementing various state-of-the-art methods. We conduct standard evaluations on datasets across various domains, including synthetic moving object trajectory. human motion, driving scenes, traffic flow, and weather forecasting. Based on our observations, we provide detailed analysis of how model architecture and dataset properties affect spatio-temporal predictive learning performance. Surprisingly, we find that recurrent-free models achieve good balance between efficiency and performance than recurrent models. Thus, we further extend the common MetaFormers to boost recurrent-fiee spatial-temporal predictive learning. We open-source the code and models at http:///gtthbb.comm///uuuuuu/ppp
1 Introduction
Recent years have witnessed rapid and remarkable progress in spatio-tempora predictive learning [38, 29, 10, 41]. This burgeoning field aims learn latent spatial and temporal patterns through the challenging task of forecasting future frames based solely on given past frames in an unsupervised manner [40, 55, 56, 54]. By ingesting raw sequential data, these self-supervised models [4, 14, 24] can uncover intricate spatial and temporal interdependencies without the need for tedious manual annotation, enabling them to extrapolate coherently into the future in a realistic fashion [29, 12]. Spatio-tempores predictive learning benefits wide range of applications with its ability to anticipate future from the past data-driven way, including modeling the devastating impacts of climate change [38, 35], predicting human movement [61, 45], forecasting traffic flow in transportation systems [8, 51], and learning expressive representations from video [32, 19]. By learning to predict the future without supervision from massive datasets, these techniques have the potential to transform domains where anticipation and planning are crucial but limited labeled data exists [9, 2, 44, 31].
"Equal contribution. "Corresponding author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Rees
Sp-2
R,12
9,
ConvLSTM
CanVLSTM
ST-LSTM
ST-LSTM
112
G2
HF
ConvLSTM
ConVLSTM
ST-LSTM
ST-LSTM
MetaFommer
IncepUNett
142
GF.
ConVLSTM
ConVLSTM
ST-LSTM
ST-LSTM
MetaFoormer
IncepUNet
ConvLSTM
ConvISTM
ST-LSTM
ST-LSTM
Conv
Conv
Ppo1
Xy-1
Kee3
Xee1
(0) ConVLSTM
(b) PredRNN
(c) MetaV
(d) SimVP
Recurrent-based
Recurrent-free
Two typical sptaio-tempora predictive learning models. As illustrated by the left two instances (a)(b), the first type requires several recurrent modules to predict the next frame according to the previous frames in auto-regressive manner, dubbed recurrent- -aaedd models. As for the right two instances (d), the second type predicts all future frames based on all given frames at once, which inferences in parallel and is called the recurrent-free model.
Figure 1:
Despite the significance of spatio-temporal predictive learning and the development of various approaches, there remains a conspicuous lack of a comprehensive benchmark for this field covering various synthetic and practical application scenarios. We believe that comprehensive benchmark is essential for advancing the field and facilitating meaningful comparisons between different methods. In particular, there exists perennial question that has not yet been conclusively answered: is necessary to employ recurrent neural network architectures to capture temporal dependencies? In other words, recurrent free models achieve performance comparable to recurrent based models without explicil lemporal modeling?
Since the seminal work ConvLSTM M [38] was proposed, which ingeniously integrates convolutional networks and long-short term memory (LSTM) networks [15] to separately capture spatial and tem- poral correlations, researchers have vacillated between utilizing eschewing recurrent architectures. As shown in Figure 1, (a) ConvLSTM a prototypical recurrent-based model that infuses recurrent structure into convolutional networks. (b) PredRNN [49] represents series of recurrent models that revise the flow of information to enhance performance. (c) MetaVP is the recurrent-fres model that abstracted from Si by substituting its IncepU [10] modules with MetaFormers [59]. (d) Sim [10, 40] a typical recurrent- free model that achieves performance comparable to previous state-of-the-art models without explicitly modeling temporal dependencies.
In this study, we illuminate the long-standing question of whether explicit temporal modeling with recurrent neural networks is requisite for teapio-temporal predictive learning. To achieve this, we present a comprehensive benchmark, Open Pentio-Temmpral predictive Learning, dubbed OpenSTL. We revisit the approaches that represent the foremost strands within modular and extensive framework to ensure fair comparisons. We summarize our main contributions as follows:
We build OpenSTL, a comprehensive benchmark for spatio-temporal predictive learning that includes 14 representative algorithms and 24 models. OpenSTL covers wide range of methods and classifies them into two categories: recurrent-based and recurrent-free methods.
We conduct extensive experiments on a diversity of tasks ranging from synthetic moving object trajectories to real-world human motion, driving scenes, traffic flow, and weather forecasting. The datasets span synthetic to real-world data and micro-to-macro scales.
While recurrent-bas based models have been well developed, we rethink the potential recurrent-free models based on insights from OpenSTL. We propose generalizing MetaFormer-like architectures [59] to boost recurrent-free spatio-temporal predictive learn- ing. Recurrent-free models can thus reformulate the problem as downstream task of designing vision backbones for general applications.

---

2 Background and Related work
2.1 Problem definition
We propose the formal definition for the spatio-temporal predictive learning problem as follows. Given a sequence of video frames X², = {x')] I--+11 up to time spanning the past T frames, the objective is predict the subsequent 7 frames y {x*)(]++** from time t onwards, where each frame z; RCCx** typically comprises C' channels, with height H and width W pixels. In practice, we represent the input sequence of observed frames and output sequence of predicted frames respectively as tensors X E RTxxCxHxW and y E ERT xC
The model with learnable parameters @ learns a mapping Fe X'',"' + y by leveraging both spatial and temporal dependencies. In case, the mapping Fe corresponds neural network trained to minimize the discrepancy between the predicted future frames and the ground-tru future frames. The optimal parameters O are given by:
O arg min ((Fe(A putI."" )"
where L denotes loss function that quantifies such discrepancy.
In this study, we categorize prevalent spatio-temporal predictive learning methods into two classes: recurrent-based and recurrent-free models. For recurrent-based models, the mapping Fe comprises several recurrent interactions:
Fe: fo(x'' T+1 h²-T+1) fola', ( o fg -n -1),
where h represents the memory state encompassing historical information and fo denotes the mapping between each pair of adjacent frames. The parameters 0 are shared across each state. Therefore, the prediction process can be expressed as follows:
+ fo(ø", h'), vi ( 1......,,+++";
(3)
For recurrent-free models, the prediction process directly feeds the whole sequence of observed frames into the model and outputs the complete predicted frames at once.
2.2 Recurrent- models
Since the pioneering work ConvLSTM [38] was proposed. recurrent-based models [29, 30, 16, 12, 58, 31] have been extensively studied. PredRNN [49] adopts vanilla ConvI modules to build a Spatio-temporal LSTM (ST-LSTM) unit that models spatial and temporal variations simultaneously. PredRNN+ [47] proposes gradient highway unit to mitigate the gradient vanishing and CaSual-LSTM module to cascadely connect spatial and temporal memories. PredRNNv2 [50] further proposes curriculum learning strategy and memory decoupling loss to boost performance. MIM [51] introduces high-order non-stationarity learning in designing LSTM modules. PhyDNet [12] explicitly disentangles PDE dynamics from unknown complementary information with recurrent physical unit. E3DLSTM [48] integrates 3D convolutions into recurrent networks. MAU [3] proposes motion-aware unit that captures motion information. Although various recurrent- based models have been developed, the reasons behind their strong performance remain not fully understood.
2.3 Recurrent-free models
Compared to recurrent-based models, recurrent- free models have received less attention. Previous studies tend to use 3D convolutional networks to model temporal dependencies [28, 1]. PredCNN [57] and TrajectoryCNN [25] use 2D convolutional networks for efficiency. However, early recurrent- free models were doubted due to their poor performance. Recently, SimVP [10, 40, 41] provided simple but effective recurrent-freg baseline with competitive performance. PastNet [53] and IAM4VP [37] are two recent recurrent securren--free models that perform strong performance, In this study, we implemented representative recurrent-ban and recurrent-fret models under unified framework to systematically investigate their intrinsic properties. Moreover, we further explored the potential of recurrent-free models by reformulating the spatio-temporal predictive learning problem and extending MetaFormers [59] to bridge the gap between the visual backbone and apatio-tempor learning.
3

---

OpenSTL
3.1 Supported Methods
3.1.1 Overview
OpenSTI. has implemented 14 representative spatio-temporal predictive learning methods under unified framework, including 11 recurrent-based methods and 3 recurrent-free methods. We summarize these methods in Table where we also provide the corresponding conference/journal and the types of their spatial-temporal modeling components. The spatial modeling of these methods is fundamentally consistent. Most methods apply two-dimensional convolutional networks (Conv2D) to model spatial dependencies, while E3D-LSTM and CrevNet harness three-dimensional convolutional networks (Conv3D) instead.
The primary distinction between these methods lies in how they model temporal dependencies using their proposed modules. The ST-LSTM module, proposed in PredRNN [49], the most widely used module. CrevNet has similar modeling approach as PredRNN, but it incorporates an information-pe preseervng mechanism into the model. Analogously, Casual-LSTTM [47], MIM Block [51], E3D-LSTM [48], PhyCell [12], and MAU [3] represent variants of ConvLSTM proposed with miscellaneous motivations. MVFB built as a multi-scale voxel flow block that diverges from ConvLSTM However, DMVEN [17] predicts future frames frame-by-frane br-fmame which still qualifies as recurrent-basee basee model. IncepU [10] constitutes an Unet-like module that also exploits the multi-scale feature from the InceptionNet-IIl Nee-like architecture. gSTA [40] and TAU [41] extend the IncepU module to simpler and more efficient architectures without InceptionNet or Unet-like architectures. In this work, we further extend the temporal modeling of recurrent- free models by introducing MetaFormers [59] to boost recurrent-free spatio-temporal predictive learning.
Table 1: Categorizations of the supported spatial-temporal predictive learning methods in OpenSTL.
Category
Method
ConferenceeJournal
Spatial modeling Temporal modeling
ConvLSTM [38]
NeurIPS 2015
Conv2D
Conv-LSTM
PredNet [29]
ICLR 2017
Conv2D
ST-LSTM
PredRNN [49]
NeurIPS 2017
Conv2D
ST-LSTM
PredRNN++ [47]
ICMI. 2018
Conv2D
Casual-LSTMM
Recurrent-based
MIM [51]
CVPR 2019
Conv2D
MIM Block
E3D-LSTM [48]
ICLR 2019
Conv3D
E3D-LSTM
CrevNet [58]
ICLR 2020
Conv3D
ST-LSTM
PhyDNet [12]
CVPR 2020
Conv2D
Cony MonWLSMM+PyCCll
MAU [3]
NeurIPS 2021
Conv2D
MAU
PredRNNv2 [50]
TPAMI 2022
Conv2D
ST-LSTM
DMVEN [17]
CVPR 2023
Conv2D
MVFB
Sim [10]
CVPR 2022
Conv2D
IncepU
Recurrent-free
TAU [41]
CVPR 2023
Conv2D
TAU
Sim VPv2 [40]
arXiv
Conv2D
gSTA
3.1.2 Rethink the recurrent-free models
Although less studied, recurrent-free spatio-temporal predictive learning models share similar architecture, as illustrated in Figure 2. The encoder comprises several 2D convolutional networks, which project high- sigh--imensional input data into a low-dimensional latent space. When given batch of input observed frames E BBxx*CxXXXXX HxW the encoder focuses solely on intra-frame spatial correlations, ignoring temporal modeling. Subsequently, the middle temporal module stacks the low-dimensional representations along the temporal dimension to ascertain temporal dependencies. Finally, the decoder comprises several 2D convolutional upsampling networks, which reconstruct subsequent frames from the learned latent representations.
The encoder and decoder enable efficient temporal learning by modeling temporal dependencies in a low-dimensional latent space. The core component of recurrent-free models the temporal module. Previous studies have proposed temporal modules such as IncepU [10], TAU [41], and

---

gSTA [40] that have proved beneficial. However, we argue that the competence stems primarily from the general recurrent-free architecture instead of the specific temporal modules. Thus, we employ MetaFormers [59] the temporal module by changing the input channels from the original C' to inter-frame channels T C. By extending the recurrent-free architecture, we leverage the advantages of MetaFormers to enhance the recurrent-free model. In this work, we implement ViT [7], Swin Transformer [26], Uniformer [21]. MLP-Mixer [42], ConvMixer [43], Poolformer [59], ConvNeXt [27], VAN [13], HorNet [33], and MogaNet [22] for the SectaFormer-bbsce recurrent-free model, substituting the intermediate temporal module the original recurrent- architecture.
BxTxCxHxW
Input
Input
Input
Input
(BxT)xCXHXW
Encoder
Encoder
Encoder
Encoder
Bx(*xC)XXX
MetaFormer
IncepU
TAU
gSTA
(BXT)xCXXXW
Decoder
Decoder
Decoder
Decoder
BXTXCXlIXW
Output
Output
Output
Output
Recurrent-free architecture
SImVP
TAU
SimVPy2
Figure 2: The general architecture of recurrent-free models with three instances.
3.2
Supported Tasks
We have curated five diverse tasks in our OpenSTL benchmark, which cover a wide range of scenarios from synthetic simulations to real-world situations various scales. The tasks include synthetic moving object trajectories, real-world human motion capture, driving scenes, traffic flow, and weather forecasting. The datasets used in our benchmark range from synthetic real-world, and from micro to macro scales. We have provided a summary of the dataset statistics in Table 2.
Table 2: The detailed dataset statistics of the supported tasks in OpenSTL.
Dataset
Training size
Testing size
Channel
Height
Width
T
T"
Moving MNIST variants
10,000
10,000
1/3
64
64
10
10
KTH Action
4,940
3,030
128
20/40
Human3.6M
73,404
8,582
3
128
Kitti&Caltech
3,160
3,095
3
128
160
10
1
TaxiBJ
20,461
500
2
32
32
4
4
WeatherBench-S
2,167
706
32/128
64/256
12
12
WeatherBench-M 54,019
2,883
4
Synthetic moving object trajectory prediction Moving MNIST [39] is one of the seminal bench- mark datasets that has been extensively utilized. Each video sequence comprises two moving digits confined within a 64 x 64 frame. Each digit was assigned velocity whose direction was randomly chosen from a unit circle and whose magnitude was also arbitrarily selected from fixed range. Apart from the original Moving MNIST dataset, we provide two variants with more complicated objects (Moving FushionMNIST) that replace the digits with fashion objects and more complex scenes (Moving MNIST-CIFA that employ images from the CIFAR-10 dataset [20] as the background. Moreover, we provide three settings of Moving MNIST for robustness evaluations, including missing frames, dynamic noise, and perceptual occlusions.
5

---

Human motion capture Predicting human motion is challenging due to the complexity of human movements, which vary greatly among individuals and actions. We utilized the KTH dataset [36].
which includes six types of human actions: walking, jogging, running, boxing, hand waving, and hand clapping. We furnish two settings, predicting the next 20 and 40 frames respectively. Human3.66 3.M [18] is an intricate human pose dataset containing high-resolution RGB videos. Analogous to preceding studies [12, 51], we predict the next four frames by the observed four frames.
Driving scene prediction Predicting the future dynamics of driving scenarios is crucial for au- tonomous driving. Compared other tasks, this undertaking involves con-staionnary and diverse scenes. To address this issue, we follow the conventional approach [29] and train the model on the Kitri [1]] dataset. We then evaluate the performance on the Caltech Pedestrian [6] dataset. To ensure consistency, we center-cropped and downsized all frames to 128 x 160 pixels.
Traffic flow prediction Forecasting the dynamics of crowds is crucial for traffic management and public safery. To evaluate spatio-temporal predictive learning approaches for traffic flow prediction, use the TaxiB/ [60] dataset. This dataset includes GPS data from taxis and meteorological data in Beijing. The dataset contains two types of crowd flows, representing inflow and outflow. The temporal interval is 30 minutes, and the spatial resolution is 32 32.
Weather forecasting Global weather pattern prediction an essential natural predicament. The WeatherBench [34] dataset is large-scale weather forecasting dataset encompassing various types of climatic factors. The raw data is re-grid to 5.625 resolution (32 x 64 grid points) and 1.40625e (128 x 256 grid points). We consider two setups: First, WeatherBench-S is single-variable setup in which each climatic factor is trained independently. The model is trained on data from 2010-2015, validated data from 2016, and tested on data from 2017-2018, with one-hour temporal interval. Second, WeatherBench-M a marti-variable setup that mimics real-world weather forecasting more closely. All climatic factors trained simultaneously. The model is trained data from 1979 to 2015, using the same validation and testing data as WeatherBench-S. The temporal interval is extended to six hours, capturing broader range of temporal dependencies.
3.3 Evaluation Metrics
We evaluate the performance of supported models the aforementioned tasks using various metrics in thorough and rigorous manner. We use them for specific tasks according to their characteristics.
Error metrics We utilize the mean squared error (MSE) and mean absolute error (MAE) to evaluate the difference between the predicted results and the true targets. Root mean squared error (RMSE) also used in weather forecasting as it is more common in this domain.
Similarity metries We utilize the structural similarity index measure (SSIM) [52] and peak signal-to- noise ratio (PSNR) to evaluate the similarity between the predicted results and the true targets. Such metries are extensively used image processing and computer vision.
Perceptual metrics LPIPS [62] is implemented evaluate the perceptual difference between the predicted results and the true targets in the human visual system. I.IPS provides a perceptually- aligned evaluation for vision tasks. We utilize this metric in real-world video prediction tasks.
Computational metrics We utilize the number of parameters and the number of floating-poin operations (FLOPs) to evaluate the computational complexity of the models. We also report the frames per second (FPS) single NVIDIA V100 GPU to evaluate the inference speed.
3.4 Codebase Structure
While existing open- spen-sourced provides a modular and extensible framework that adheres to the design principles of OpenMMLab [5] and assimilates code elements from Open [23] and USB [46]. OpenSTL. excels in user- friendliness, organization, and comprehensiveness, surpassing the usability of existing open-source STL codebases. A detailed description of the codebase structure can be found in Appendix B.
predictive learning codebases are independent, OpenSTL

---

Experiment and Analysis
We conducted comprehensive experiments the mentioned tasks to assess the performance of the supported methods in OpenSTLL. Detailed analysis of the results is presented to gain insights into spatio-temporal predictive learning. Implementation details can be found in Appendix C.
4.1 Synthetic Moving Object Trajectory Prediction
We conduct experiments the synthetic moving object trajectory prediction task, utilizing three datasets: Moving MNIST, Moving FashionMNIST, and Moving MNIST-CIFAR, The performance of the evaluated models on the Moving MNIST dataset is reported in Table 3. The detailed results for the other two synthetic datasets are in Appendix D.1.
It can be observed that recurrent-basee models yield varied results that do not consistently outperform recurrent- free models, while recurrent-base models always exhibit slower inference speeds than their recurrent-fin free counterparts. Although PredRNN, PredRNN++, MIM, and PredRNNv2 achieve lower MSE and MAE values compared to recurrent- free models, their FLOPs are nearly five times higher, and their FPS are approximately seven times slower than all recurrent-fres models. Furthermore, there are minimal disparities in the performance of recurrent-free models as opposed to recurrent- models, highlighting the robustness of the proposed general recurrent-free architecture. The remaining two synthetic datasets, consisting of more intricate moving objects (Moving FashionMNIST) and complex scenes (Moving MNIST-CIFAR), reinforce the experimental findings that recurrent- models deliver comparable performance with significantly higher efficiency. In these toy datasets characterized by high frequency but low resolution, recurrent-base models excel in capturing temporal dependencies but are susceptible to high computational complexity.
Table 3: The performance on the Moving MNIST dataset.
Method
Params (M)
FLOPs
FPS
MSE J
MAE
SSIM f
PSNR f
ConvLSTM
15.0
56.8
113
29.80
90.64
0.9288
22.10
PredNet
12.5
8.4
659
161.38
201.16
0.7783
14.67
PredRNN
23.8
116.0
54
23.97
72.82
0.9462
23.28
PredRNN+
171.7
38
22.06
69.58
MIM
38.0
179.2
37
22.55
69.97
0.9498
23.56
Recurrent-hased
E3D-LSTM
$1.0
298.9
18
35.97
78.28
0.9320
21.11
CrevNet
270.7
30.15
86.28
0.9350
22.15 15
PhyDNet
3.1
15.3
182
28.19
78.64
0.9374
22.62
MAU
17.8
201
26.86
78.22
0.9398
PredRNNv2
23.9
116.6
52
24.13
73.73
0.9453
23.21
DMVEN
3.5
0.2
1145
123.67
179.96
0.8140
16.15
SimVP
58.0
19.4
209
32.15
89.05
0.9268
21.84
TAU
44.7
16.0
283
24.60
71.93
0.9454
23.19
SimVPv2
16.5
282
26.69
77.19
0.9402
22.78
VIT
46.1
16.9
290
35.15
95.87
0.9139
21.67
Swin Transformer
46.1
16.4
294
29.70
84.05
0.9331
22.22
Uniformer
44.8
16.5
296
30.38
85,87
0.9308
22.13
Recurrent-free
MI.e-Mixer
38.2
14.7
334
29.52
83.36
0.9338
22.22
ConyMixer
3,9
5.5
658
32.09
88,93
0.9259
21.93
Poolforner
37.1
14.1
341
31.79
88.48
0.9271
22.03
ConvNext
37.3
14.1
344
26.94
77.23
0.9397
22.74
VAN
44.5
16.0
288
26.10
76.11
0.9417
22.89
HorNet
45.7
16.3
287
29.64
83,26
0.9331
22.26
MogaNet
46.8
16.5
255
25.57
75.19
0.9429
22,99

---

4.2 Real-world Video Prediction
We perform experiments on real-world video predictions, specifically focusing on human motion capturing using the KTH and Human3. 6M datasets, as well as driving scene prediction using the Kitti&Caltech dataset. Due space constraints, we present the results for the Kitti&Caltech dataset in Table 4, while the detailed results for the other datasets can be found in Appendix D.2. We observed that as the resolution increases, the computational complexity of recurrent-basee models dramatically increases. In contrast, recurrent-free models achieve commendable balance between efficiency and performance. Notably, although some recurrent-baad based models achieve lower MSE and MAE values, their FLOPs are nearly 20 times higher compared to their recurrent-free counterparts. This highlights the efficiency advantage of recurrent-free models, especially in high-resolution scenarios.
Table 4: The performance on the Kitti&Calteci dataset.
Method
Parums (M) FLOPs
FPS
MSE
MAE
SSIM
PSNR
LPIPS
ComvL.STM
595.0
8.58
PredNet
42.8
1568.9
0.9286
27.21
11.29
1216.0
1525.5
7.40
PredRNN+
38.5
1803.0
125.5
1453.2
0.9433
28.02
13.21
Recurrent-
MIM
49.2
1858.0
125.1
1464.0
0.9409
28.10
6.35
E3D-LSTM
54,9
1004.0
10
200.6
1946.2
0.9047
25.45
12.60
PhyDNet
3.10
40.4
117
312.2
2754.8
0,8615
23.26
32,19
MAU
24.3
172.0
16
177.8
1800.4
0.9176
26.14
9.67
PredRNNv2
23.8
1223.0
16
147.8
1610.5
0,9330
27.12
8.92
DMVEN
3.6
1.2
557
183.9
1531.1
0,9314
26.78
4.94
SimVP
8.6
60.6
57
160.2
1690.8
0.9338
26.81
6.76
TAU
15.0
92.5
55
131.1
1507.8 8
0,9456
27.83
5.49
SimVPv2
15.6
96.3
40
129.7
1507.7
0,9454
27.89
5.57
VIT
12.7
155.0
25
146.4
1615.8
0,9379
27.43
6.66
Swin Transformer
15.3
95.2
155.2
1588.9
0,9299
27.25
8.11
Uniformer
11.8
104.0
28
135.9
1534.2
0,9393
27.66
6.87
Recurrent-free
MLP-Mixer
22.2
83.5
207.9
1835.9
0,9133
26.29
7.75
ConvMixer
23.1
129
174.7
1854.3
0,9232
26.23
7.76
Poolformer
79.8
153.4
1613.5
0.9334
27.38
7.00
ConvNext
80.2
92.5
127.5
1476.5
0,9462
27.98
5,50
MogaNet
96.2
131.4
1512.1
0,9442
27.79
5.39
4.3 Traffic and Weather Forecasting
Traffic flow prediction and weather forecasting are two critical tasks that have significant implications for public safety and scientific research. While these tasks operate at macro level, they exhibit lower frequencies compared to the tasks mentioned above, and the states along the timeline tend be more stable. Capturing subtle changes in such tasks poses significant challenge. In order to assess the performance of the supported models in OpenSTL, we conduct experiments on the TaxiBJ and WeatherBench datasets. It is worth noting that weather forecasting encompasses various settings, and we provide detailed results of them Appendix D.3.
Here, we present comparison of the MAE and RMSE metrics for representative approaches in single-variable weather factor forecasting at low resolution. Figure displays the results for four climatic factors, i.e., temperature, humidity, wind component, and cloud cover. Notably, recurrent- free models consistently outperform recurrent-base their potential to apply spatio-temporal predictive learning to macro-scale tasks instead of relying solely on recurrent-basee models. These findings underscore the promising nature of recurrent-fre free models and suggest that they can be viable alternative to the prevailing recurrent-bas based models in
models across all weather factors, indicating

---

Temperature
Humidity
Component
0.7
0.8
3.5
4,0
4.5
0.8
0.9
0.14
0.16
ConVLSTM
PredRNN
PredRNN++
PredRNNv2
MAU
SimVP
TAU
SimVPv2
Swin-T
MogaNet
(8) MAE
1.2
1.2
1.4
0.20
0.24
ConVLSTM
PredRNN
ProdRNNV2
MAU
SimVP
TAU
SimVPv2
MogaNet
(b) RMSE
Figure 3: The (a) MAE and (b) RMSE metrics of the representative approaches on the four weather forecasting tasks in WeatherBench.
the context of weather forecasting. Furthermore, in the Appendix, we provide additional insights into high-resolution and multi-vane vartilva weather forecasting, where similar trends are observed.
4.4 Robustness Analysis
To further understand the differences in robustness between the recurrent- recurrent-based and recurrent- spatiotemporal predictive learning methods, construeted three experimental setups: (i) Moving MNIST Missing, which deals with input frames with missing frames, where we set the probability of random missing frame 20%; (ii) Moving MNIST Dynamic, where added random Gaussian noise to the speed of each digit, making their movement speeds irregular; (iii) Moving MNIST Perceptual, where we randomly occluded the input frames, using a black 24 x4 patch for occlusion. We choose three representative recurrent-base -based and three recurrent-free methods for evaluation. The experimental results for these three setups are presented in Table 7, respectively,
can be observed that the recurrent-free methods exhibit remarkable robustness under both the missing and perceptual noise scenarios. Even when compared to situations without noise, there is little performance degradation due to their focus on global information. Conversely, recurrent- based methods encounter substantial performance drops. They overly focus the relationships individual frames can inadvertently lead to overfitting. In the case of dynamic noise, all methods faced significant performance setbacks, because the speed of the digits became irregular and harder predict.
Table 5: The performance the Moving MNIST Missing dataset.
Method
Params (M)
FLOPs (G)
FPS
MSE 1
MAE +
SSIM 1
PSNR
ConvLSTM
15.0
32.73
0.9201
Recurrent-basee
PredRNN
23.8
116.0
54
46.05
117.21
0.8800
20.35
PredRNN+
171.7
$3.89
118.45
0.8907
19.71
Si
58.0
19.4
209
34.92
95.23
0.9194
21.44
Recurrent-free
TAU
16.0
26.77
77.50
0.9400
22.74
Sim VPv2
46.8
16.5
282
28.63
81.79
0.9352
22.39

---

Table 6: The performance on the Moving MNIST Dynamic dataset.
Method
Params (M)
FLOPs (G)
FPS
MSF 4
MAR 4
SSIM f
PSNR
ConvLSTM
15.0
56.8
113
49.03
135.49
0.8683
19.73
Recurrent-basee
PredRNN
23.8
116.0
54
59.18
157.47
0.8220
19.09
PredRNN++
38.6
171.7
40.85
109.32
0.9030
20.65
SimVP
58.0
19.4
209
48.41
130.83
0.8725
19.91
Recurrent-free
TAU
44.7
16.0
283
43.37
121.31
0.8853
20.41
Sim VPv2
46.8
16.5
282
44.74
123.70
0.8823
20.28
Table 7:
The performance on the Moving MNIST Perceptual dataset.
Method
Params (M)
FLOPs (G)
FPS
MSE 4
MAE +
SSIM 1
PSNR
ConvLSTM
15.0
56.8
113
31.34
95.39
0.9227
21.85
Recurrent-basee
PredRNN
23.8
116.0
54
46.04
122.40
0.8792
20.28
PredRNN+
38.6
171.7
38
51.76
127.12
0.8722
19.85
SimVP
58.0
19.4
209
34.73
95.23
0.9196
21.44
Recurrent-free
TAU
44.7
16.0
283
26.87
78.08
0.9393
22.69
VPv2
46.8
16.5
282
28.83
82.65
0.9343
22.36
5 Conclusion and Discussion
This paper introduces OpenSTL, a comprehensive benchmark for spatio-temporal predictive learning with diverse set of 14 representative methods and 24 models, addressing wide range of challenging tasks. OpenSTL categorizes existing approaches into recurrent- -based and recurrent-free models. To unlock the potential of recurrent- ceeureet-ffree models, propose general recurrent- frreett--eeeeeeeee architecture and introduce MetaFormers for temporal modeling. Extensive experiments are conducted to sys- tematically evaluate the performance of the supported models across various tasks. In synthetic datasets, recurrent- based models excel at capturing temporal dependencies, while recurrent models achieve comparable performance with significantly higher efficiency. In real-world video prediction tasks, recurrent-free models strike a commendable halance between efficiency and perfor- their counterparts
mance. Additionally, recurrent-free models demonstrate significant superiority in weather forecasting, highlighting their potential for scientific applications at a macro-scale level.
Moreover, we observed that recurrent architectures are beneficial in capturing temporal dependencies, but they are not always necessary, especially for computationally expensive tasks. Recurrent free mod- els can be viable alternative that provides good balance between efficiency and performance. The effectiveness of recurrent- can be attributed to their sequential tracking of frame-by-free by-freme changes, providing local temporal inductive bias. On the other hand, recurrent-free models combine multiple frames together, exhibiting global temporal inductive bias that is suitable for ww-frrquency spatio-temporal dependencies. We hope that our work provides valuable insights and serves as reference for future research.
models in capturing high- figh-frequecc spatio-temporal dependencies
While our primary focus lies in general spatio-temporal predictive learning, there are still several open problems that require further investigation. One particular challenge is finding ways to effectively leverage the strengths of both securren-bbssed and recurrent-free models to enhance the modeling of dependencies. While there is a correspondence between the spatial encoding and temporal modeling Me and the token mixing and channel mixing in MetaFormer, it raises the question whether we can improve recurrent- free models by extending the existing MetaFormers.
Acknowledgments and Disclosure of Funding
This work was supported by the National Key R&D Program of China (2022 15100), the National Natural Science Foundation of China A20427,, the Competitive Research Fund (WU2022A009) from the Westlake Center for Synthetic Biology and Integrated Bioengineering.
10

---

References
[1] S. Aigner and M. Körner. Futuregan: Anticipating the future frames of video sequences using spatio-temporal 3d convolutions in progressively growing gans. arXiv preprins
arXiv:1810.01325, 2018.
[2] L. Castrejon, N. Ballas, and A. Courville. Improved conditional vmnss for video prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7608-7617, 2019.
[3] Z. Chang, X. Zhang, S. Wang, S. Ma, Ye, X. Xinguang, and W. Gao. Mau: A motion-aware unit for video prediction and beyond. Advances in Neural Information Processing Systems, 34, 2021.
[4] Chen, S. Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning. 2020.
[5] M. Contributors. MMCV: OpenMMLab computer vision foundation. https://github... comb open- mmlab/nmcv, 2018.
[6] Dollár, C. Wojek, Schiele, and Perona. Pedestrian detection: A benchmark. In CVPR,
June 2009.
[7] A. Desovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations.
[8] Fang. Q. Zhang. Meng. S. Xiang, and C. Pan. Gstnet: Global
network for
traffic flow prediction. In IJCAI, pages 2286-28862222222 2293, 2019.
[9] C. Finn, Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. Advances in Neural Information Processing Systems, 29, 2016.
[10] Z. Gao, C. Tan, and S. Z. Li. Simvp: Simpler yet better video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3170-3180, 2022.
[11] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research,
2013.
[12] V. L. Guen and N. Thome. Disentangling physical dynamics from unknown factors for unsuper- vised video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (1474-11484 2020.
[13] M.-H. Guo, C.-Z. Lu, Z.-N. Liu, M.-M. Cheng, and S.-M. Hu. Visual attention network. arXiv preprint arXiv:2202.0911 09741, 2022.
[14] K. He, Chen, S. Xie, Y. Li, Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[15] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 1735- 1780, 1997.
[16] J.-T. Hsich, Liu, D.-A. Huang. F. Fei-Fei, and J. C. Niebles. Learning decompose and disentangle representations for video prediction. Advances in Neural Information Processing Systems, 31, 2018.
[17] X. Hu, Z. Huang, A. Huang. J. Xu, and S. Zhou. A dynamic multi-scale voxel flow network for video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[18] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325-133) 2013.
[19] S. Jenni, G. Meishvili, and P. Favaro. Video representation learning by recognizing temporal transformations. In European Conference on Computer Vision, pages 425-442. Springer, 2020.
[20] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[21] K. Li, Wang, J. Zhang, P. Gao, Song, Y. Liu, H. Li, and Y. Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. arXiv preprint arXiv:2201.099 2022.
11

---

[22] S. Li, Z. Wang, Z. Liu, C. Tan, H. Lin, D. Wu, Z. Chen, J. Zheng, and S. Z. Li. Efficient multi-order gated aggregation network. arXiv preprint arXiv:2211.032295, 2022.
[23] S. Li, Z. Wang. Z. Liu, D. Wu, C. Tan, and S. Z. Li. Openmixup: Open mixup toolbox and benchmark for visual representation learning. ArXiv, abs/2209.0485. 2022.
[24] Li, D. WU, F. Wu, Z. Zang. K. Wang, L. Shang, B. Sun, H. Li, and Stan.Z.Li. Architecture- agnostic masked image modeling from vit back to enn. In International Conference on Machine Learning (ICML), 2023.
[25] X. Liu, J. Yin, J. Liu, P. Ding, J. Liu, and H. Liu. Trajectoryenn: new spatio-temporal feature learning network for human motion prediction. IEEE Transactions on Circuits and Systems for Video Technology, 31 (1()):1133 2146, 2020.
[26] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[27] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. arXiv preprint arXiv:2201.0388
2022.
[28] Z. Liu, R. A. Yeh, X. Tang, Y. Liu, and A. Agarwala. Video frame synthesis using deep voxel flow. In Proceedings of the IEEE International Conference on Computer Vision, pages
4463-4471, 2017.
[29] W. Lotter, G. Kreiman, and D. Cox. Deep predictive coding networks for video prediction and unsupervised learning. In International Conference on Learning Representations, 2017.
[30] M. Oliu, J. Selva, and S. Escalera. Folded recurrent neural networks for future video prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 716-731, 2018.
[31] S. Oprea, Marzinez-Gonzalez,, A. Garcia-Garcia, J. A. Castro- -argga, S. Orts-Escolano, J. Garciia-Rddriguez, and A. Argyros. A review on deep learning techniques for video prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[32] R. Qian, T. Meng. B. Gong, M.-H. Yang. H. Wang, S. Belongie, and Y. Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6964 2021.
[33] Y. Rao, W. Zhao, Y. Tang, J. Zhou, S. N. Lim, and J. Lu. Hornet: Efficient high-order spatial interactions with recursive gated convolutions. Advances in Neural Information Processing Systems, 35:103533 10366, 2022.
[34] S. Rasp. P. D. Dueben, S. Scher, J. A. Weyn, S. Mouatadid, and N. Thuerey. Weatherbench: a benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems, 12(1 2201)x22020S000220,, 2020.
[35] M. Reichstein, G. Camps- B. Stevens, M. Jung, J. Denzler, N. Carvalhais, et al. Deep learning and process understanding for data-driven earth system science. Nature, 56677733:-- 204, 2019.
[36] C. Schuldt, I. Laptev, and B. Caputo. Recognizing human actions: local svm approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3, pages 32-36. IEFE, 2004.
[37] M. seok Seo, H. Lee, D.-Y. Kim, and J. Seo. Implicit stacked autoregressive model for video prediction, ArXiv, abs/2303.008499 2023.
[38] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung. W.-K. Wong, and W.-c. Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Advances in Neural Information Processing Systems, 28, 2015.
[39] N. Srivastava, F. Mansimov, and R. Salakhudinov. Unsupervised learning of video representa- tions using Istms. In International conference on machine learning, pages 843-852. PMIR, 2015.
[40] C. Tan, Z. Gao, S. Li, and S. Z. Li. Simvp: Towards simple yet powerful spatiotemporal predictive learning. arXiv preprint arXiv:2211.12509 2022.
[41] C. Tan, Z. Gao, I.. Wu, Y. Xu, J. Xia, S. Li, and S. Z. Li. Temporal attention unit: Towards efficient spatiotemporal predictive learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2023.
12

---

[42] I. O. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner,
D. Keysers. Uszkoreit, al. Mip-mixer: An all-mip architecture for vision. Advances in neural information processing systems, 34:24261 222222, 2021.
[43] A. Trockman and Z. Kolter. Patches are all you need? arXiv preprint arXiv:2201.09792
2022.
[44] R. Villegas, D. Erhan, H. Lee, et al. Hierarchical long-term terg video prediction without supervision.
In International Conference on Machine Learning, pages 6038-6046. PMLR, 2018.
[45] P. Wang, W. Li, P. Ogunbona, J. Wan, and S. Escalera. Rgb-d-based human motion recognition with deep learning: A survey. Computer Vision and Image Understanding, 171:1 18 139, 2018.
[46] Y. Wang, H. Chen, Y. Fan, W. Sun, R. Tao, W. Hou, R. Wang, L. Yang, Z. Zhou, L.-Z. Guo, et al. Usb: unified semi-supervised learning benchmark for classification. Advances in Neural Information Processing Systems, 35:3938-399
2022.
[47] Y. Wang, Z. Gao, M. Long, J. Wang, and S. Y. Philip. Predmn+++ Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In International Conference on Machine Learning, pages 5123-5132. PMLR, 2018.
[48] Y. Wang, L. Jiang, M.-H. Yang, L.-. Li, M. Long, and L. Fei-Fei. Eidetic 3d Istm: A model for video prediction and beyond. In International conference on learning representations, 2018.
[49] Y. Wang, M. Long, J. Wang, Z. Gao, and P. S. Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal Istms. Advances in Neural Information Processing Systems, 30, 2017.
[50] Y. Wang. H. Wu, J. Zhang, Z. Gao, J. Wang. P. S. Yu, and M. Long. Predran: A recurrent neural network for spatiotemporal predictive learning. arXiv preprint arXiv:2103. 0.5004 2021.
[51] Y. Wang. J. Zhang, H. Zhu, M. Long, J. Wang, and P. S. Yu. Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9154-9162, 2019.
[52] Z. Wang. A. C. Bovik, H. R. Sheikh, and F. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
[53] H. Wu, W. Xion, F. Xu, X. Luo, C. Chen, X. Hua, and H. Wang. Pastnet: Introducing physical inductive biases for spatio-temporal video prediction. ArXiv, abs/2305. 11422, 2023.
[54] L. Wu, H. Lin, Z. Gao, C. Tan, and S. Z. Li. Graphmixup: Improving class-imbale imsalancbdd node classification on graphs by self-supervised context prediction. European Conference on Machine Learning, 2022.
[55] L. Wu, H. Lin. Y. Huang, and S. Z. Li. Knowledge distillation improves graph structure augmentation for graph neural networks. Advances in Neural Information Processing Systems, 2022.
[56] L. Wu, H. Lin, C. Tan, Z. Gao, and S. Z. Li. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Trunsactions on Knowledge and Data Engineering, 2021.
[57] Z. Xu, Y. Wang, M. Long, J. Wang, and M. KLiss. Predenn: Predictive learning with cascade convolutions. In IJCAL, pages 2940-2947, 2018
[58] W. Yu, Y. Lu, S. Easterbrook, and S. Fidler. Efficient and informatioon-rresrving future frame prediction and beyond. In International Conference on Learning Representations, 2019.
[59] W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang, Feng, and S. Yan. Metaformer actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10819-108299 2022.
[60] J. Zhang, Y. Zheng, and D. Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. In Thirty-first AAAI conference on artificial intelligence, 2017.
[61] L. Zhang, G. Zhu, P. Shen, J. Song, S. Afaq Shah, and M. Bennamoun. Learning spatiotemporal features using 3dcnn and convolutional Istm for gesture recognition. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 3120-3128, 2017.
[62] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
13


Reviews:
[{"review_id": "mldWFizc9e", "reviewer": "Reviewer_rqPr", "summary": "", "questions": "", "limitations": "The same as above.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This work constructs a unified framework for spatial-temporal predictive learning. Various models and modules can be quickly evaluated across datasets and metrics from the well-maintained GitHub repository. It enables the community to evolve their algorithms with minimal effort within such a framework.\n\n2. It conducts a comprehensive benchmarking study on 14 algorithms and 3 representative datasets with 4 metrics. It provides an overview of the recent progress of predictive learning.\n\n3. It provides an insight that recurrent-free models are much more efficient without sacrificing performance, which is very practical for real-world employment. It also finds that meta-former-based models can capture global temporal information, which sheds light on the low-frequency temporal change scenarios.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "E9WnHhgaKb", "reviewer": "Reviewer_HExp", "summary": "", "questions": "", "limitations": "Refer to *Opportunities For Improvement*.", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The author has conducted comprehensive experiments, providing valuable references for analyzing existing methods in this field.\n- The author has performed a thorough analysis of the results and obtained insightful conclusions.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "umbXuW4dSb", "reviewer": "Reviewer_jUek", "summary": "", "questions": "", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Spatio-temporal predictive learning is an important problem.\n- The source code is released. \n- This paper is well organized and clearly written. The technical details are also easy to follow.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JGvGZWGlXr", "reviewer": "Reviewer_sUSg", "summary": "", "questions": "", "limitations": "The paper only briefly talks about the future directions of predictive spatio-temporal learning in the last paragraph of the conclusion section. I'd encourage the authors to discuss more the limitations of the current benchmark.\n\nI don't see any big potential negative societal impact of the paper.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The task of predictive spatio-temporal learning is important, and the community lacks a unified benchmark. Therefore, OpenSTL can facilitate the reproducibility of the field. This will benefit the researchers in the STL community\n- The code has been released with a clear structure and detailed documents. I appreciate the authors' efforts in incorporating so many algorithms and datasets into one codebase\n- The observation on recurrent-free methods is interesting, especially on its inference speed and computational requirements", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper presents a new and comprehensive spatio-temporal learning in two categories - recurrent-based and recurrent-free, and tested on diverse datasets including driving scenes, traffic flow, and weather data. 
The authors provide a modular and extensible framework, easily customisable.
We ask the authors to add the robustness aspect of the evaluation requested by by reviewer rqPr in the final camera ready of the paper.
Decision: Accept (Poster)