SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cheng Tan $^{1,2*}$  Siyuan Li $^{1,2*}$  Zhangyang Gao $^{1,2}$  Wenfei Guan $^{3}$  Zedong Wang $^{2}$
Zicheng Liu $^{1,2}$  Lirong Wu $^{1,2}$  Stan Z. Li $^{2\dagger}$
$^{1}$ Zhejiang University;  $^{3}$ Xidian University;
$^{2}$ AI Lab, Research Center for Industries of the Future, Westlake University
{tancheng; lisiyuan; gaozhangyang; guanwenfei; wangzedong; liuzicheng;
wulirong; stan.zq.li}  $@$  westlake.edu.cn;
Spatio-temporal predictive learning is a learning paradigm that enables models to learn spatial and temporal patterns by predicting future frames from given past frames in an unsupervised manner. Despite remarkable progress in recent years, a lack of systematic understanding persists due to the diverse settings, complex implementation, and difficult reproducibility. Without standardization, comparisons can be unfair and insights inconclusive. To address this dilemma, we propose OpenSTL, a comprehensive benchmark for spatio-temporal predictive learning that categorizes prevalent approaches into recurrent-based and recurrent-free models. OpenSTL provides a modular and extensible framework implementing various state-of-the-art methods. We conduct standard evaluations on datasets across various domains, including synthetic moving object trajectory, human motion, driving scenes, traffic flow, and weather forecasting. Based on our observations, we provide a detailed analysis of how model architecture and dataset properties affect spatio-temporal predictive learning performance. Surprisingly, we find that recurrent-free models achieve a good balance between efficiency and performance than recurrent models. Thus, we further extend the common MetaFormers to boost recurrent-free spatial-temporal predictive learning. We open-source the code and models at https://github.com/chengtan9907/OpenSTL.
Recent years have witnessed rapid and remarkable progress in spatio-temporal predictive learning [38, 29, 10, 41]. This burgeoning field aims to learn latent spatial and temporal patterns through the challenging task of forecasting future frames based solely on given past frames in an unsupervised manner [40, 55, 56, 54]. By ingesting raw sequential data, these self-supervised models [4, 14, 24] can uncover intricate spatial and temporal interdependencies without the need for tedious manual annotation, enabling them to extrapolate coherently into the future in a realistic fashion [29, 12]. Spatio-temporal predictive learning benefits a wide range of applications with its ability to anticipate the future from the past in a data-driven way, including modeling the devastating impacts of climate change [38, 35], predicting human movement [61, 45], forecasting traffic flow in transportation systems [8, 51], and learning expressive representations from video [32, 19]. By learning to predict the future without supervision from massive datasets, these techniques have the potential to transform domains where anticipation and planning are crucial but limited labeled data exists [9, 2, 44, 31].
(a) ConvLSTM
Recurrent-based
(b)PredRNN
(c) MetaVP
Recurrent-free
Figure 1: Two typical sptaio-temporal predictive learning models. As illustrated by the left two instances (a)(b), the first type requires several recurrent modules to predict the next frame according to the previous frames in an auto-regressive manner, dubbed recurrent-based models. As for the right two instances (c)(d), the second type predicts all future frames based on all given frames at once, which inferences in parallel and is called the recurrent-free model.
(d) SimVP
Despite the significance of spatio-temporal predictive learning and the development of various approaches, there remains a conspicuous lack of a comprehensive benchmark for this field covering various synthetic and practical application scenarios. We believe that a comprehensive benchmark is essential for advancing the field and facilitating meaningful comparisons between different methods. In particular, there exists a perennial question that has not yet been conclusively answered: is it necessary to employ recurrent neural network architectures to capture temporal dependencies? In other words, can recurrent-free models achieve performance comparable to recurrent-based models without explicit temporal modeling?
Since the seminal work ConvLSTM [38] was proposed, which ingeniously integrates convolutional networks and long-short term memory (LSTM) networks [15] to separately capture spatial and temporal correlations, researchers have vacillated between utilizing or eschewing recurrent architectures. As shown in Figure 1, (a) ConvLSTM is a prototypical recurrent-based model that infuses a recurrent structure into convolutional networks. (b) PredRNN [49] represents a series of recurrent models that revise the flow of information to enhance performance. (c) MetaVP is the recurrent-free model that abstracted from SimVP by substituting its IncepU [10] modules with MetaFormers [59]. (d) SimVP [10, 40] is a typical recurrent-free model that achieves performance comparable to previous state-of-the-art models without explicitly modeling temporal dependencies.
In this study, we illuminate the long-standing question of whether explicit temporal modeling with recurrent neural networks is requisite for spatio-temporal predictive learning. To achieve this, we present a comprehensive benchmark, Open Spatio-Temporal predictive Learning, dubbed OpenSTL. We revisit the approaches that represent the foremost strands within a modular and extensive framework to ensure fair comparisons. We summarize our main contributions as follows:
- We build OpenSTL, a comprehensive benchmark for spatio-temporal predictive learning that includes 14 representative algorithms and 24 models. OpenSTL covers a wide range of methods and classifies them into two categories: recurrent-based and recurrent-free methods.
- We conduct extensive experiments on a diversity of tasks ranging from synthetic moving object trajectories to real-world human motion, driving scenes, traffic flow, and weather forecasting. The datasets span synthetic to real-world data and micro-to-macro scales.
- While recurrent-based models have been well developed, we rethink the potential of recurrent-free models based on insights from OpenSTL. We propose generalizing MetaFormer-like architectures [59] to boost recurrent-free spatio-temporal predictive learning. Recurrent-free models can thus reformulate the problem as a downstream task of designing vision backbones for general applications.
We propose the formal definition for the spatio-temporal predictive learning problem as follows. Given a sequence of video frames  $\mathcal{X}^{t,T} = \{\pmb{x}^i\}_{t - T + 1}^t$  up to time  $t$  spanning the past  $T$  frames, the objective is to predict the subsequent  $T'$  frames  $\mathcal{Y}^{t + 1,T'} = \{\pmb{x}^i\}_{t + 1}^{t + 1 + T'}$  from time  $t + 1$  onwards, where each frame  $\pmb{x}_i\in \mathbb{R}^{C\times H\times W}$  typically comprises  $C$  channels, with height  $H$  and width  $W$  pixels. In practice, we represent the input sequence of observed frames and output sequence of predicted frames respectively as tensors  $\mathcal{X}^{t,T}\in \mathbb{R}^{T\times C\times H\times W}$  and  $\mathcal{Y}^{t + 1,T'}\in \mathbb{R}^{T' \times C\times H\times W}$ .
The model with learnable parameters  $\Theta$  learns a mapping  $\mathcal{F}_{\Theta}:\mathcal{X}^{t,T}\mapsto \mathcal{Y}^{t + 1,T^{\prime}}$  by leveraging both spatial and temporal dependencies. In our case, the mapping  $\mathcal{F}_{\Theta}$  corresponds to a neural network trained to minimize the discrepancy between the predicted future frames and the ground-truth future frames. The optimal parameters  $\Theta^{*}$  are given by:
$$
\Theta^ {*} = \arg \min  _ {\Theta} \mathcal {L} \left(\mathcal {F} _ {\Theta} \left(\mathcal {X} ^ {t, T}\right), \mathcal {Y} ^ {t + 1, T ^ {\prime}}\right), \tag {1}
$$
where  $\mathcal{L}$  denotes a loss function that quantifies such discrepancy.
In this study, we categorize prevalent spatio-temporal predictive learning methods into two classes: recurrent-based and recurrent-free models. For recurrent-based models, the mapping  $\mathcal{F}_{\Theta}$  comprises several recurrent interactions:
$$
\mathcal {F} _ {\Theta}: f _ {\theta} \left(\boldsymbol {x} ^ {t - T + 1}, \boldsymbol {h} ^ {t - T + 1}\right) \circ \dots \circ f _ {\theta} \left(\boldsymbol {x} ^ {t}, \boldsymbol {h} ^ {t}\right) \circ \dots \circ f _ {\theta} \left(\boldsymbol {x} ^ {t + T ^ {\prime} - 1}, \boldsymbol {h} ^ {t + T ^ {\prime} - 1}\right), \tag {2}
$$
where  $h^i$  represents the memory state encompassing historical information and  $f_{\theta}$  denotes the mapping between each pair of adjacent frames. The parameters  $\theta$  are shared across each state. Therefore, the prediction process can be expressed as follows:
$$
\boldsymbol {x} ^ {t + 1} = f _ {\theta} \left(\boldsymbol {x} ^ {i}, \boldsymbol {h} ^ {i}\right), \forall i \in \{t + 1, \dots , t + T ^ {\prime} \}, \tag {3}
$$
For recurrent-free models, the prediction process directly feeds the whole sequence of observed frames into the model and outputs the complete predicted frames at once.
2.2 Recurrent-based models
Since the pioneering work ConvLSTM [38] was proposed, recurrent-based models [29, 30, 16, 12, 58, 31] have been extensively studied. PredRNN [49] adopts vanilla ConvLSTM modules to build a Spatio-temporal LSTM (ST-LSTM) unit that models spatial and temporal variations simultaneously. PredRNN++ [47] proposes a gradient highway unit to mitigate the gradient vanishing and a Casual-LSTM module to cascadelly connect spatial and temporal memories. PredRNNv2 [50] further proposes a curriculum learning strategy and a memory decoupling loss to boost performance. MIM [51] introduces high-order non-stationarity learning in designing LSTM modules. PhyDNet [12] explicitly disentangles PDE dynamics from unknown complementary information with a recurrent physical unit. E3DLSTM [48] integrates 3D convolutions into recurrent networks. MAU [3] proposes a motion-aware unit that captures motion information. Although various recurrent-based models have been developed, the reasons behind their strong performance remain not fully understood.
2.3 Recurrent-free models
Compared to recurrent-based models, recurrent-free models have received less attention. Previous studies tend to use 3D convolutional networks to model temporal dependencies [28, 1]. PredCNN [57] and TrajectoryCNN [25] use 2D convolutional networks for efficiency. However, early recurrent-free models were doubted due to their poor performance. Recently, SimVP [10, 40, 41] provided a simple but effective recurrent-free baseline with competitive performance. PastNet [53] and IAM4VP [37] are two recent recurrent-free models that perform strong performance. In this study, we implemented representative recurrent-based and recurrent-free models under a unified framework to systematically investigate their intrinsic properties. Moreover, we further explored the potential of recurrent-free models by reformulating the spatio-temporal predictive learning problem and extending MetaFormers [59] to bridge the gap between the visual backbone and spatio-temporal learning.
3 OpenSTL
3.1 Supported Methods
3.1.1 Overview
OpenSTL has implemented 14 representative spatio-temporal predictive learning methods under a unified framework, including 11 recurrent-based methods and 3 recurrent-free methods. We summarize these methods in Table 1, where we also provide the corresponding conference/journal and the types of their spatial-temporal modeling components. The spatial modeling of these methods is fundamentally consistent. Most methods apply two-dimensional convolutional networks (Conv2D) to model spatial dependencies, while E3D-LSTM and CrevNet harness three-dimensional convolutional networks (Conv3D) instead.
The primary distinction between these methods lies in how they model temporal dependencies using their proposed modules. The ST-LSTM module, proposed in PredRNN [49], is the most widely used module. CrevNet has a similar modeling approach as PredRNN, but it incorporates an information-preserving mechanism into the model. Analogously, Casual-LSTM [47], MIM Block [51], E3D-LSTM [48], PhyCell [12], and MAU [3] represent variants of ConvLSTM proposed with miscellaneous motivations. MVFB is built as a multi-scale voxel flow block that diverges from ConvLSTM. However, DMVFN [17] predicts future frames frame-by-frame which still qualifies as a recurrent-based model. IncepU [10] constitutes an Unet-like module that also exploits the multi-scale feature from the InceptionNet-like architecture. gSTA [40] and TAU [41] extend the IncepU module to simpler and more efficient architectures without InceptionNet or Unet-like architectures. In this work, we further extend the temporal modeling of recurrent-free models by introducing MetaFormers [59] to boost recurrent-free spatio-temporal predictive learning.
Table 1: Categorizations of the supported spatial-temporal predictive learning methods in OpenSTL.
Category	Method	Conference/Journal	Spatial modeling	Temporal modeling
Recurrent-based	ConvLSTM [38]	NeurIPS 2015	Conv2D	Conv-LSTM
PredNet [29]	ICLR 2017	Conv2D	ST-LSTM
PredRNN [49]	NeurIPS 2017	Conv2D	ST-LSTM
PredRNN++ [47]	ICML 2018	Conv2D	Casual-LSTM
MIM [51]	CVPR 2019	Conv2D	MIM Block
E3D-LSTM [48]	ICLR 2019	Conv3D	E3D-LSTM
CrevNet [58]	ICLR 2020	Conv3D	ST-LSTM
PhyDNet [12]	CVPR 2020	Conv2D	ConvLSTM+PhyCell
MAU [3]	NeurIPS 2021	Conv2D	MAU
PredRNNv2 [50]	TPAMI 2022	Conv2D	ST-LSTM
DMVFN [17]	CVPR 2023	Conv2D	MVFB
Recurrent-free	SimVP [10]	CVPR 2022	Conv2D	IncepU
TAU [41]	CVPR 2023	Conv2D	TAU
SimVPv2 [40]	arXiv	Conv2D	gSTA
3.1.2 Rethink the recurrent-free models
Although less studied, recurrent-free spatio-temporal predictive learning models share a similar architecture, as illustrated in Figure 2. The encoder comprises several 2D convolutional networks, which project high-dimensional input data into a low-dimensional latent space. When given a batch of input observed frames  $\mathcal{B} \in \mathbb{R}^{B \times T \times C \times H \times W}$ , the encoder focuses solely on intra-frame spatial correlations, ignoring temporal modeling. Subsequently, the middle temporal module stacks the low-dimensional representations along the temporal dimension to ascertain temporal dependencies. Finally, the decoder comprises several 2D convolutional upsampling networks, which reconstruct subsequent frames from the learned latent representations.
The encoder and decoder enable efficient temporal learning by modeling temporal dependencies in a low-dimensional latent space. The core component of recurrent-free models is the temporal module. Previous studies have proposed temporal modules such as IncepU [10], TAU [41], and
gSTA [40] that have proved beneficial. However, we argue that the competence stems primarily from the general recurrent-free architecture instead of the specific temporal modules. Thus, we employ MetaFormers [59] as the temporal module by changing the input channels from the original  $C$  to inter-frame channels  $T \times C$ . By extending the recurrent-free architecture, we leverage the advantages of MetaFormers to enhance the recurrent-free model. In this work, we implement ViT [7], Swin Transformer [26], Uniformer [21], MLP-Mixer [42], ConvMixer [43], Poolformer [59], ConvNeXt [27], VAN [13], HorNet [33], and MogaNet [22] for the MetaFormers-based recurrent-free model, substituting the intermediate temporal module in the original recurrent-free architecture.
Figure 2: The general architecture of recurrent-free models with three instances.
3.2 Supported Tasks
We have curated five diverse tasks in our OpenSTL benchmark, which cover a wide range of scenarios from synthetic simulations to real-world situations at various scales. The tasks include synthetic moving object trajectories, real-world human motion capture, driving scenes, traffic flow, and weather forecasting. The datasets used in our benchmark range from synthetic to real-world, and from micro to macro scales. We have provided a summary of the dataset statistics in Table 2.
Table 2: The detailed dataset statistics of the supported tasks in OpenSTL.
Dataset	Training size	Testing size	Channel	Height	Width	T	T'
Moving MNIST variants	10,000	10,000	1/3	64	64	10	10
KTH Action	4,940	3,030	1	128	128	10	20/40
Human3.6M	73,404	8,582	3	128	128	4	4
Kitti&Caltech	3,160	3,095	3	128	160	10	1
TaxiBJ	20,461	500	2	32	32	4	4
WeatherBench-S	2,167	706	1	32/128	64/256	12	12
WeatherBench-M	54,019	2,883	4	32	64	4	4
Synthetic moving object trajectory prediction Moving MNIST [39] is one of the seminal benchmark datasets that has been extensively utilized. Each video sequence comprises two moving digits confined within a  $64 \times 64$  frame. Each digit was assigned a velocity whose direction was randomly chosen from a unit circle and whose magnitude was also arbitrarily selected from a fixed range. Apart from the original Moving MNIST dataset, we provide two variants with more complicated objects (Moving FashionMNIST) that replace the digits with fashion objects and more complex scenes (Moving MNIST-CIFAR) that employ images from the CIFAR-10 dataset [20] as the background. Moreover, we provide three settings of Moving MNIST for robustness evaluations, including missing frames, dynamic noise, and perceptual occlusions.
Human motion capture Predicting human motion is challenging due to the complexity of human movements, which vary greatly among individuals and actions. We utilized the KTH dataset [36], which includes six types of human actions: walking, jogging, running, boxing, hand waving, and hand clapping. We furnish two settings, predicting the next 20 and 40 frames respectively. Human3.6M [18] is an intricate human pose dataset containing high-resolution RGB videos. Analogous to preceding studies [12, 51], we predict the next four frames by the observed four frames.
Driving scene prediction Predicting the future dynamics of driving scenarios is crucial for autonomous driving. Compared to other tasks, this undertaking involves non-stationary and diverse scenes. To address this issue, we follow the conventional approach [29] and train the model on the Kitti [11] dataset. We then evaluate the performance on the Caltech Pedestrian [6] dataset. To ensure consistency, we center-cropped and downsized all frames to  $128 \times 160$  pixels.
Traffic flow prediction Forecasting the dynamics of crowds is crucial for traffic management and public safety. To evaluate spatio-temporal predictive learning approaches for traffic flow prediction, we use the TaxiBJ [60] dataset. This dataset includes GPS data from taxis and meteorological data in Beijing. The dataset contains two types of crowd flows, representing inflow and outflow. The temporal interval is 30 minutes, and the spatial resolution is  $32 \times 32$ .
Weather forecasting Global weather pattern prediction is an essential natural predicament. The WeatherBench [34] dataset is a large-scale weather forecasting dataset encompassing various types of climatic factors. The raw data is re-grid to  $5.625^{\circ}$  resolution ( $32 \times 64$  grid points) and  $1.40625^{\circ}$  ( $128 \times 256$  grid points). We consider two setups: First, WeatherBench-S is a single-variable setup in which each climatic factor is trained independently. The model is trained on data from 2010-2015, validated on data from 2016, and tested on data from 2017-2018, with a one-hour temporal interval. Second, WeatherBench-M is a multi-variable setup that mimics real-world weather forecasting more closely. All climatic factors are trained simultaneously. The model is trained on data from 1979 to 2015, using the same validation and testing data as WeatherBench-S. The temporal interval is extended to six hours, capturing a broader range of temporal dependencies.
We evaluate the performance of supported models on the aforementioned tasks using various metrics in a thorough and rigorous manner. We use them for specific tasks according to their characteristics.
Error metrics We utilize the mean squared error (MSE) and mean absolute error (MAE) to evaluate the difference between the predicted results and the true targets. Root mean squared error (RMSE) is also used in weather forecasting as it is more common in this domain.
Similarity metrics We utilize the structural similarity index measure (SSIM) [52] and peak signal-to-noise ratio (PSNR) to evaluate the similarity between the predicted results and the true targets. Such metrics are extensively used in image processing and computer vision.
Perceptual metrics LPIPS [62] is implemented to evaluate the perceptual difference between the predicted results and the true targets in the human visual system. LPIPS provides a perceptually-aligned evaluation for vision tasks. We utilize this metric in real-world video prediction tasks.
Computational metrics We utilize the number of parameters and the number of floating-point operations (FLOPs) to evaluate the computational complexity of the models. We also report the frames per second (FPS) on a single NVIDIA V100 GPU to evaluate the inference speed.
3.4 Codebase Structure
While existing open-sourced spatio-temporal predictive learning codebases are independent, OpenSTL provides a modular and extensible framework that adheres to the design principles of OpenMMLab [5] and assimilates code elements from OpenMixup [23] and USB [46]. OpenSTL excels in user-friendliness, organization, and comprehensiveness, surpassing the usability of existing open-source STL codebases. A detailed description of the codebase structure can be found in Appendix B.
We conducted comprehensive experiments on the mentioned tasks to assess the performance of the supported methods in OpenSTL. Detailed analysis of the results is presented to gain insights into spatio-temporal predictive learning. Implementation details can be found in Appendix C.
4.1 Synthetic Moving Object Trajectory Prediction
We conduct experiments on the synthetic moving object trajectory prediction task, utilizing three datasets: Moving MNIST, Moving FashionMNIST, and Moving MNIST-CIFAR. The performance of the evaluated models on the Moving MNIST dataset is reported in Table 3. The detailed results for the other two synthetic datasets are in Appendix D.1.
It can be observed that recurrent-based models yield varied results that do not consistently outperform recurrent-free models, while recurrent-based models always exhibit slower inference speeds than their recurrent-free counterparts. Although PredRNN, PredRNN++, MIM, and PredRNNv2 achieve lower MSE and MAE values compared to recurrent-free models, their FLOPs are nearly five times higher, and their FPS are approximately seven times slower than all recurrent-free models. Furthermore, there are minimal disparities in the performance of recurrent-free models as opposed to recurrent-based models, highlighting the robustness of the proposed general recurrent-free architecture. The remaining two synthetic datasets, consisting of more intricate moving objects (Moving FashionMNIST) and complex scenes (Moving MNIST-CIFAR), reinforce the experimental findings that recurrent-free models deliver comparable performance with significantly higher efficiency. In these toy datasets characterized by high frequency but low resolution, recurrent-based models excel in capturing temporal dependencies but are susceptible to high computational complexity.