SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: The performance on the Moving MNIST dataset.
Method	Params (M)	FLOPs (G)	FPS	MSE ↓	MAE ↓	SSIM ↑	PSNR ↑
Recurrent-based	ConvLSTM	15.0	56.8	113	29.80	90.64	0.9288	22.10
PredNet	12.5	8.4	659	161.38	201.16	0.7783	14.67
PredRNN	23.8	116.0	54	23.97	72.82	0.9462	23.28
PredRNN++	38.6	171.7	38	22.06	69.58	0.9509	23.65
MIM	38.0	179.2	37	22.55	69.97	0.9498	23.56
E3D-LSTM	51.0	298.9	18	35.97	78.28	0.9320	21.11
CrevNet	5.0	270.7	10	30.15	86.28	0.9350	22.15
PhyDNet	3.1	15.3	182	28.19	78.64	0.9374	22.62
MAU	4.5	17.8	201	26.86	78.22	0.9398	22.57
PredRNNv2	23.9	116.6	52	24.13	73.73	0.9453	23.21
DMVFN	3.5	0.2	1145	123.67	179.96	0.8140	16.15
Recurrent-free	SimVP	58.0	19.4	209	32.15	89.05	0.9268	21.84
TAU	44.7	16.0	283	24.60	71.93	0.9454	23.19
SimVPv2	46.8	16.5	282	26.69	77.19	0.9402	22.78
ViT	46.1	16.9	290	35.15	95.87	0.9139	21.67
Swin Transformer	46.1	16.4	294	29.70	84.05	0.9331	22.22
Uniformer	44.8	16.5	296	30.38	85.87	0.9308	22.13
MLP-Mixer	38.2	14.7	334	29.52	83.36	0.9338	22.22
ConvMixer	3.9	5.5	658	32.09	88.93	0.9259	21.93
Poolformer	37.1	14.1	341	31.79	88.48	0.9271	22.03
ConvNext	37.3	14.1	344	26.94	77.23	0.9397	22.74
VAN	44.5	16.0	288	26.10	76.11	0.9417	22.89
HorNet	45.7	16.3	287	29.64	83.26	0.9331	22.26
MogaNet	46.8	16.5	255	25.57	75.19	0.9429	22.99
4.2 Real-world Video Prediction
We perform experiments on real-world video predictions, specifically focusing on human motion capturing using the KTH and Human3.6M datasets, as well as driving scene prediction using the Kitti&Caltech dataset. Due to space constraints, we present the results for the Kitti&Caltech dataset in Table 4, while the detailed results for the other datasets can be found in Appendix D.2. We observed that as the resolution increases, the computational complexity of recurrent-based models dramatically increases. In contrast, recurrent-free models achieve a commendable balance between efficiency and performance. Notably, although some recurrent-based models achieve lower MSE and MAE values, their FLOPs are nearly 20 times higher compared to their recurrent-free counterparts. This highlights the efficiency advantage of recurrent-free models, especially in high-resolution scenarios.
Table 4: The performance on the Kitti&Caltech dataset.
Method	Params (M)	FLOPs (G)	FPS	MSE ↓	MAE ↓	SSIM ↑	PSNR ↑	LPIPS ↓
Recurrent-based	ConvLSTM	15.0	595.0	33	139.6	1583.3	0.9345	27.46	8.58
PredNet	12.5	42.8	94	159.8	1568.9	0.9286	27.21	11.29
PredRNN	23.7	1216.0	17	130.4	1525.5	0.9374	27.81	7.40
PredRNN++	38.5	1803.0	12	125.5	1453.2	0.9433	28.02	13.21
MIM	49.2	1858.0	39	125.1	1464.0	0.9409	28.10	6.35
E3D-LSTM	54.9	1004.0	10	200.6	1946.2	0.9047	25.45	12.60
PhyDNet	3.10	40.4	117	312.2	2754.8	0.8615	23.26	32.19
MAU	24.3	172.0	16	177.8	1800.4	0.9176	26.14	9.67
PredRNNv2	23.8	1223.0	16	147.8	1610.5	0.9330	27.12	8.92
DMVFN	3.6	1.2	557	183.9	1531.1	0.9314	26.78	4.94
Recurrent-free	SimVP	8.6	60.6	57	160.2	1690.8	0.9338	26.81	6.76
TAU	15.0	92.5	55	131.1	1507.8	0.9456	27.83	5.49
SimVPv2	15.6	96.3	40	129.7	1507.7	0.9454	27.89	5.57
ViT	12.7	155.0	25	146.4	1615.8	0.9379	27.43	6.66
Swin Transformer	15.3	95.2	49	155.2	1588.9	0.9299	27.25	8.11
Uniformer	11.8	104.0	28	135.9	1534.2	0.9393	27.66	6.87
MLP-Mixer	22.2	83.5	60	207.9	1835.9	0.9133	26.29	7.75
ConvMixer	1.5	23.1	129	174.7	1854.3	0.9232	26.23	7.76
Poolformer	12.4	79.8	51	153.4	1613.5	0.9334	27.38	7.00
ConvNext	12.5	80.2	54	146.8	1630.0	0.9336	27.19	6.99
VAN	14.9	92.5	41	127.5	1476.5	0.9462	27.98	5.50
HorNet	15.3	94.4	43	152.8	1637.9	0.9365	27.09	6.00
MogaNet	15.6	96.2	36	131.4	1512.1	0.9442	27.79	5.39
4.3 Traffic and Weather Forecasting
Traffic flow prediction and weather forecasting are two critical tasks that have significant implications for public safety and scientific research. While these tasks operate at a macro level, they exhibit lower frequencies compared to the tasks mentioned above, and the states along the timeline tend to be more stable. Capturing subtle changes in such tasks poses a significant challenge. In order to assess the performance of the supported models in OpenSTL, we conduct experiments on the TaxiBJ and WeatherBench datasets. It is worth noting that weather forecasting encompasses various settings, and we provide detailed results of them in Appendix D.3.
Here, we present a comparison of the MAE and RMSE metrics for representative approaches in single-variable weather factor forecasting at low resolution. Figure 3 displays the results for four climatic factors, i.e., temperature, humidity, wind component, and cloud cover. Notably, recurrent-free models consistently outperform recurrent-based models across all weather factors, indicating their potential to apply spatio-temporal predictive learning to macro-scale tasks instead of relying solely on recurrent-based models. These findings underscore the promising nature of recurrent-free models and suggest that they can be a viable alternative to the prevailing recurrent-based models in
Figure 3: The (a) MAE and (b) RMSE metrics of the representative approaches on the four weather forecasting tasks in WeatherBench.
the context of weather forecasting. Furthermore, in the Appendix, we provide additional insights into high-resolution and multi-variable weather forecasting, where similar trends are observed.
4.4 Robustness Analysis
To further understand the differences in robustness between the recurrent-based and recurrent-free spatiotemporal predictive learning methods, we constructed three experimental setups: (i) Moving MNIST - Missing, which deals with input frames with missing frames, where we set the probability of random missing frame to  $20\%$ ; (ii) Moving MNIST - Dynamic, where we added random Gaussian noise to the speed of each digit, making their movement speeds irregular; (iii) Moving MNIST - Perceptual, where we randomly occluded the input frames, using a black  $24 \times 24$  patch for occlusion. We choose three representative recurrent-based and three recurrent-free methods for evaluation. The experimental results for these three setups are presented in Table 5, 6, 7, respectively.
It can be observed that the recurrent-free methods exhibit remarkable robustness under both the missing and perceptual noise scenarios. Even when compared to situations without noise, there is little performance degradation due to their focus on global information. Conversely, recurrent-based methods encounter substantial performance drops. They overly focus on the relationships individual frames can inadvertently lead to overfitting. In the case of dynamic noise, all methods faced significant performance setbacks, because the speed of the digits became irregular and harder to predict.
Table 5: The performance on the Moving MNIST - Missing dataset.
Method	Params (M)	FLOPs (G)	FPS	MSE ↓	MAE ↓	SSIM ↑	PSNR ↑
Recurrent-based	ConvLSTM	15.0	56.8	113	32.73	96.95	0.9201	21.65
PredRNN	23.8	116.0	54	46.05	117.21	0.8800	20.35
PredRNN++	38.6	171.7	38	53.89	118.45	0.8907	19.71
Recurrent-free	SimVP	58.0	19.4	209	34.92	95.23	0.9194	21.44
TAU	44.7	16.0	283	26.77	77.50	0.9400	22.74
SimVPv2	46.8	16.5	282	28.63	81.79	0.9352	22.39
Table 6: The performance on the Moving MNIST - Dynamic dataset.
Method	Params (M)	FLOPs (G)	FPS	MSE ↓	MAE ↓	SSIM ↑	PSNR ↑
Recurrent-based	ConvLSTM	15.0	56.8	113	49.03	135.49	0.8683	19.73
PredRNN	23.8	116.0	54	59.18	157.47	0.8220	19.09
PredRNN++	38.6	171.7	38	40.85	109.32	0.9030	20.65
Recurrent-free	SimVP	58.0	19.4	209	48.41	130.83	0.8725	19.91
TAU	44.7	16.0	283	43.37	121.31	0.8853	20.41
SimVPv2	46.8	16.5	282	44.74	123.70	0.8823	20.28
Table 7: The performance on the Moving MNIST - Perceptual dataset.
Method	Params (M)	FLOPs (G)	FPS	MSE ↓	MAE ↓	SSIM ↑	PSNR ↑
Recurrent-based	ConvLSTM	15.0	56.8	113	31.34	95.39	0.9227	21.85
PredRNN	23.8	116.0	54	46.04	122.40	0.8792	20.28
PredRNN++	38.6	171.7	38	51.76	127.12	0.8722	19.85
Recurrent-free	SimVP	58.0	19.4	209	34.73	95.23	0.9196	21.44
TAU	44.7	16.0	283	26.87	78.08	0.9393	22.69
SimVPv2	46.8	16.5	282	28.83	82.65	0.9343	22.36
This paper introduces OpenSTL, a comprehensive benchmark for spatio-temporal predictive learning with a diverse set of 14 representative methods and 24 models, addressing a wide range of challenging tasks. OpenSTL categorizes existing approaches into recurrent-based and recurrent-free models. To unlock the potential of recurrent-free models, we propose a general recurrent-free architecture and introduce MetaFormers for temporal modeling. Extensive experiments are conducted to systematically evaluate the performance of the supported models across various tasks. In synthetic datasets, recurrent-based models excel at capturing temporal dependencies, while recurrent-free models achieve comparable performance with significantly higher efficiency. In real-world video prediction tasks, recurrent-free models strike a commendable balance between efficiency and performance. Additionally, recurrent-free models demonstrate significant superiority over their counterparts in weather forecasting, highlighting their potential for scientific applications at a macro-scale level.
Moreover, we observed that recurrent architectures are beneficial in capturing temporal dependencies, but they are not always necessary, especially for computationally expensive tasks. Recurrent-free models can be a viable alternative that provides a good balance between efficiency and performance. The effectiveness of recurrent-based models in capturing high-frequency spatio-temporal dependencies can be attributed to their sequential tracking of frame-by-frame changes, providing a local temporal inductive bias. On the other hand, recurrent-free models combine multiple frames together, exhibiting a global temporal inductive bias that is suitable for low-frequency spatio-temporal dependencies. We hope that our work provides valuable insights and serves as a reference for future research.
While our primary focus lies in general spatio-temporal predictive learning, there are still several open problems that require further investigation. One particular challenge is finding ways to effectively leverage the strengths of both recurrent-based and recurrent-free models to enhance the modeling of spatial-temporal dependencies. While there is a correspondence between the spatial encoding and temporal modeling in MetaVP and the token mixing and channel mixing in MetaFormer, it raises the question of whether we can improve recurrent-free models by extending the existing MetaFormers.
This work was supported by the National Key R&D Program of China (2022ZD0115100), the National Natural Science Foundation of China (U21A20427), the Competitive Research Fund (WU2022A009) from the Westlake Center for Synthetic Biology and Integrated Bioengineering.