PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction

Authors: hao  wu
fan  xu
chong  chen
xian-sheng  hua
xiao  luo
haixin  wang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 10:39:46

Abstract: In this paper, we investigate the challenge of spatio-temporal video prediction task, which involves generating future video frames based on historical spatio-temporal observation streams. Existing approaches typically utilize external information such as semantic maps to improve video prediction accuracy, which often neglect the inherent physical knowledge embedded within videos. Worse still, their high computational costs could impede their applications for high-resolution videos. To address these constraints, we introduce a novel framework called \underline{P}hysics-\underline{a}ssisted \underline{S}patio-\underline{t}emporal \underline{Net}work (PastNet) for high-quality video prediction. The core of PastNet lies in incorporating a spectral convolution operator in the Fourier domain, which efficiently introduces inductive biases from the underlying physical laws. Additionally, we employ a memory bank with the estimated intrinsic dimensionality to discretize local features during the processing of complex spatio-temporal signals, thereby reducing computational costs and facilitating efficient high-resolution video prediction. Extensive experiments on various widely-used spatio-temporal video benchmarks demonstrate the effectiveness and efficiency of the proposed PastNet compared with a range of state-of-the-art methods, particularly in high-resolution scenarios.

Main paper: 
 introduction: 
 Spatial-temporal forecasting has emerged as a significant area of interest within the multimedia research community [8, 26, 53, 58] . Among numerous practical problems, the objective of video prediction is to generate future video frames by leveraging historical * Equal contribution.† Corresponding authors. frames [7, 49] . This problem bears considerable relevance to an array of applications, including human motion prediction [1] , climate change analysis [39] , and traffic flow forecasting [48] .In literature, a multitude of methods have been devised for efficacious video prediction, integrating deep neural networks to capture complex correlations within spatio-temporal signals [5] .Early approaches [29, 39, 44, 51] amalgamate convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to extract features from RGB frames and predict future trends, respectively. Several techniques also employ deep stochastic models to generate video prediction while taking into account diverse potential results [1, 13, 43, 52, 53] . More recently, a range of algorithms has sought to enhance video prediction by incorporating external information such as optical flow, semantic maps, and human posture data [22, 27, 32, 45] . For example, SADM [2] combines semantic maps with flow fields to supply contextual information exhibiting superior compatibility. Nevertheless, these external inputs could not be readily available in practical situations [20, 32, 50] . In light of this consideration, a recent study [14] demonstrates that a basic CNN-based model can achieve state-of-the-art performance through end-to-end optimization.Despite their remarkable achievements, the performance of existing approaches remains far from satisfactory for the following reasons: (1) Neglect of Underlying Physical Principles. Current methods typically employ deep neural networks to extract information from spatial space and the associated visual domains [14, 22, 27, 45] . However, video frames could be governed by underlying physical principles, such as partial differential equations (PDEs) [16, 28, 34] . For instance, climate videos are typically dominated by high-order equations. As a consequence, it is anticipated to explore these physical principles for effective video prediction.(2) Low Efficiency. As a dense prediction problem [36] , the scalability of neural network models is critical for high-resolution video prediction [5] . Regrettably, the majority of existing models rely on complex neural networks, such as deep CNNs and Vision Transformers [37, 56] , which entail significant computational costs and render them unsuitable for large-scale high-resolution videos.To address these concerns, this paper introduces a novel approach called Physics-assisted Spatio-temporal Network (PastNet) for high-quality video prediction. The croe of our PastNet is to introduce inductive physical bias using the data itself, which holds the potential to solve underlying PDEs. Specifically, we introduce a convolution operator in the spectral space that initially transfers video frames into the Fourier domain, followed by efficient parallelizable channel fusion [17] . Subsequently, we employ an inverse Fourier transform to generate the outputs. Furthermore, to enhance efficiency for high-resolution video implementation, our PastNet introduces a discrete spatio-temporal module, which not only estimates intrinsic dimensionality but also introduces memory banks to discretize local features during the processing of complex spatio-temporal signals, replacing local features with their nearest queries from the memory bank. Finally, a deconvolution decoder is incorporated to output the predictions, which are combined with outputs from the spectral space. Comprehensive experiments on various benchmark datasets substantiate the effectiveness and efficiency of our proposed PastNet. A glimpse of the compared results by various approaches is provided in Figure 1 and we can observe the huge superiority of our PastNet on MovingMNIST. Our main contributions can be summarized as follows:• New Perspective. We open up a new perspective to connect spatio-temporal video prediction with physical inductive biases, thereby enhancing the model with data itself. • Novel Methodology. Our PastNet not only employs a convolution operator in the spectral space to incorporate physical prior but also discretizes local features using a memory bank with the estimated intrinsic dimensionality to boost efficiency for highresolution video prediction. • High Performance and Efficiency. Comprehensive experiments on a variety of datasets demonstrate that the PastNet exhibits competitive performance in terms of both effectiveness and efficiency. 
related work 2.1 spatio-temporal video prediction: 
 Video prediction has emerged as an essential topic within the multimedia research community, and numerous methods have been proposed to address this challenge. Initial studies frequently examine spatio-temporal signals extracted from RGB frames [29, 39, 44, 47, 51] . For instance, ConvLSTM [39] employs convolutional neural networks (CNNs) to encode spatial data, which is subsequently integrated with an LSTM model to capture temporal dependencies. PredNet [29] draws inspiration from neuroscience, enabling each layer to make local predictions for video sequences. MCnet [44] introduces multiple pathways to encode motion and content independently, which are combined into an end-to-end framework.Various approaches strive to merge video prediction with external information from optical flow, semantic maps and human posture data [22, 27, 32, 45] . As a representative unsupervised method, DVF [27] predicts missing frames using masked ones. HVP [22] treats this problem as video-to-video translation from semantic structures, inspired by hierarchical models. SADM [2] combines semantic maps and flow fields to provide more compatible contextual information. However, this external information could be inaccessible in real-world applications [20, 32, 50] . Furthermore, the efficiency and effectiveness of current solutions remain suboptimal for high-resolution videos. To surmount these obstacles, we propose a novel method that incorporates both physical inductive biases and quantization operations for high-quality video prediction.Various machine learning problems can benefit from the incorporation of physical knowledge [21] . Modern physics-informed machine learning approaches can leverage knowledge from three aspects, i.e., observational biases, inductive biases, and learning biases. Observational biases primarily arise from the data itself [24, 30, 55] , offering a range of data augmentation strategies to expand datasets. Inductive biases guide the specific design of neural networks such as graph neural networks [3] and equivariant networks [9] , which possess properties of respecting additional symmetry groups [3] . Learning biases pertain to the process of imposing distinct constraints by incorporating loss objectives during optimization [15] , adhering to the principles of multi-task learning. For instance, physics-inspired neural networks [35] (PINNs) typically include constraints related to derivatives from PDEs, resulting in superior performance in modeling dynamical systems and predicting molecular properties. To bolster predictive performance, our PastNet introduces inductive biases from the underlying PDEs of video frames by integrating learnable neural networks in the Fourier domain.Compressing large-scale data is essential for enhancing the efficiency of both training and inference processes [31] . Learning to hash is a widely employed technique that accomplishes this by mapping continuous vectors to compact binary codes while preserving similarity relationships, which achieves extensive progress in approximate nearest neighbor search [4] . Another line to address this challenge is neural quantization. For instance, multi-codebook quantization [59] is akin to the process of k-means clustering, which stores centroids and assignments in the codebooks. Recently, VQ-VAE [41] has integrated neural quantization with auto-encoders, using discrete codes to reconstruct input images and leading to efficient models for large-scale applications. VQ-VAE has been successfully applied in various scenarios, including video generation [54] , image inpainting [33] , and semantic communication [19] . In this paper, our DST module is inspired by VQ-VAE, discretizing local features to improve efficiency in high-resolution video prediction. 
methodology 3.1 overview: 
 This paper studies the problem of spatio-temporal video prediction and existing solutions usually neglect underlying physical principles and suffer from larger computational space. To tackle this, our proposed PastNet introduces both Fourier-based physicsguided (FPG) module and discrete spatio-temporal (DST) module for effectively and efficiently video prediction as in Figure 2 . In particular, our FPG module first divides frame into non-overlapping patches and then introduce a Fourier-based priori spectral filters with the introduction of physical inductive biases. Then, our DST module not only estimates intrinsic dimensionality but also introduces a discrete memory bank to effectively and efficiently capture spatio-temporal signals. Following is the problem definition and the detailed description of these key components of our proposed PastNet. Problem Definition. To enhance clarity, we offer a comprehensive explanation of the relevant concepts. Assume a video trajectory represents a dynamic physical system in the temporal domain, consisting of 𝑇 time steps, denoted as𝑽 1:𝑇 = {𝑽 1 , • • • , 𝑽 𝑇 }.Each snapshot captures 𝐶 color space measurements over time at all locations within a spatial region, represented by an 𝐻 ×𝑊 grid. From a spatial viewpoint, the observation of these 𝐶 measurements at any specific time step 𝑖 can be depicted as a tensor, 𝑽 𝑖 ∈ R 𝐶 ×𝐻 ×𝑊 . Our objective is to leverage spatio-temporal data to deduce underlying physical priors and integrate feature representation learning in both spatial and temporal dimensions and predict the most probable future sequence of length 𝑇 𝑓 , denoted as 𝑽𝑓 𝑢𝑡 𝑇 +1:𝑇 +𝑇 𝑓 = {𝑽 𝑇 +1 , • • • , 𝑽 𝑇 +𝑇 𝑓 }.Our primary insight focuses on utilizing prior physical signals to achieve effective spatio-temporal video prediction. In fact, spatiotemporal data is often subject to complex, high-dimensional nonlinear physical equations (e.g., Navier-Stokes equation), which are challenging to capture. Inspired by previous studies [17, 24] , we employ spectral methods and develop an algorithm that seamlessly combines trainable neural networks with Fourier-based a priori spectral filters. It has been shown that the features transformed through Fourier transformation in the frequency domain correspond precisely with the coefficients of the underlying physical partial differential equation [18, 24] . As a result, we can utilize neural networks to approximate the analytical solution of the latent PDE. To be specific, we initially divide video frames into non-overlapping patches with initialized embeddings and subsequently transform them into a spectral space. The features within the frequency domain are fused, followed by an inverse transformation returning them to the spatial domain. This innovation supports a physical inductive bias derived from data, demonstrating significant potential for solving PDEs. Then, we introduce our FPG module in detail. Embedding Initialization. Given the input video 𝑽 ∈ R 𝑇 ×𝐶 ×𝐻 ×𝑊 , we extract the high-level learnable representations following ViT [11] . In particular, we divide the frame into non-overlapping 𝑁 = 𝐻𝑊 /ℎ𝑤 patches of size ℎ × 𝑤 and then project them into patch embeddings 𝑬 𝑝𝑎𝑡 ∈ R 𝑇 ×ℎ×𝑤×𝑑 , where 𝑑 denotes the embedding dimension. Position embeddings 𝑬 𝑝𝑜𝑠 ∈ R ℎ×𝑤×𝑑 are also applied to get the initial token representation matrix V ∈ R 𝑇 ×ℎ×𝑤×𝑑 . In formulation,EQUATIONwhere V𝑡 = V [𝑡, :, :] ∈ R ℎ×𝑤×𝑑 makes up the matrix V .Fourier-based Spectral Filter. We apply V as input to 𝐿 layers of filters, each layer containing three essential components, i.e., Fourier transform, separate mixing and inverse Fourier transform. Firstly, a 2D fast Fourier transform (FFT) is leveraged to generate the frequency domain token K 𝑡 ∈ R ℎ×𝑤×𝑑 at time step 𝑡:EQUATIONwhere 𝑖 is the imaginary unit, and 𝑢 and 𝑣 are the indices of rows and columns in the frequency domain, respectively. Secondly, the complex-valued token K = K 1:𝑇 is then split into its real and imaginary parts and concatenated along the channel dimension. To enhance the integration of feature information, we utilize token mixing across different channels, which allows for richer Fourier mode representations to emerge through greater fusion of channel-wise signals. It is implemented with separate MLPs for the real and imaginary parts separately as follows:EQUATIONwhere ℜ and ℑ denotes the operator to obtain the real part and imaginary part, respectively. Performing token mixing mixes different modes across the Fourier domain. As the Fourier domain possesses global attributes, it further explore long-range relationships for underlying physical features.Lastly, the mixed tokens are then transformed back to the spatial domain using the 2D inverse Fourier transform to obtain the output of spectral filter layers 𝒀 ∈ R 𝑇 ×ℎ×𝑤×𝑑 as follows:EQUATIONwhere 𝒀 𝑡 = 𝒀 [𝑡, :, :] ∈ R 𝑁 ×𝑀 makes up the matrix 𝒀 . Spatial Extraction. To better extract the latent spatial information, we introduce classic convolutional neural networks as a supplement of the priori spectral filters, which can be formulated as follows:EQUATIONThe convolutional layer is known for its ability to extract features in the spatial domain through its local receptive fields. By leveraging these learned features as filters in the frequency domain, the convolutional layer can effectively mine potential physical information from the input data. The extracted physical information can significantly enhance the performance of spatiotemporal prediction.Overall, the Fourier-based physics-guided module transforms the spatial domain of latent physical information into the frequency domain using Fourier transforms, and learns the analytical solutions of PDEs using neural networks. This approach allows PastNet to handle complex geometries and high-dimensional problems.Our discrete spatio-temporal (DST) module aims to explore spatiotemporal signals in video frames in an efficient manner. To achieve this, we not only estimate the intrinsic dimensionality for the hidden space, but also introduces a memory bank to vector quantization. In particular, it consists of four different modules, i.e., encoder, intrinsic dimensionality estimation, discrete quantization, dynamic propagation and decoder. Then, we introduce them in detail. Encoder. The encoder contains 𝐾 𝑒 ConvNormReLU blocks to capture spatial signals. Given 𝒁 (0) = 𝑽 and an activation function 𝜎, we have:EQUATIONwhere 𝒁 (𝑖 -1) and 𝒁 (𝑖 ) denote the input and output of the 𝑖-th block with the shapes (𝑇 , 𝐶, 𝐻,𝑊 ) and (𝑇 , Ĉ, Ĥ, Ŵ ), respectively. Intrinsic Dimensionality Estimation. How to decide the dimensionality of the hidden space remains a challenging problem [6] .In particular, too large dimensionality would bring in redundant computational time and potential overfitting while too small one would underfit data. Here, we turn to Levina-Bickel algorithm [23] to acquire intrinsic dimensionality.In particular, we start with a large dimensionality followed by mapping 𝒁 (𝐾 𝑒 ) back to the input using a decoder and minimize the reconstruction loss objective as 𝐿 𝑟𝑒𝑐 = ||𝑽 -V || where V is the reconstructed frame. Then, we identify the 𝑅 nearest neighbours for each vector 𝒉 𝑗 ∈ 𝒁 (𝐾 𝑒 ) , i.e., {𝒉 𝑗 1 , • • • , 𝒉 𝑗 𝑅 } and calculate the local estimator for the vector as:EQUATIONwhere 𝑑 (•, •) denotes the cosine distance between two vectors. Finally, we take the average all local estimators to generate the final estimator:EQUATION)where 𝐽 is the number of vectors in 𝒁 (𝐾 𝑒 ) and • denotes the ceiling function. After generating final estimated optimal dimension, we utilize 𝐷 as the hidden embedding instead. Discrete Quantization. Previous methods usually process video features directly using spatio-temporal convolution modules. However, directly feeding video features into these modules would bring in huge computational cost. Therefore, we introduce a discrete memory bank to discretize feature vectors, which are constructed by an variational autoencoder [41, 54] . In this way, computational costs can be largely reduced to fit for large-scale video prediction.In detail, we initialize the memory bank with variational autoencoder. Here, each embedding vector 𝒛 from 𝒁 (𝐾 𝑒 ) from the output of the encoder is mapped to the nearest point in the memory bank. The number of embeddings in the memory is set to 𝐷 2 empircally. Given the memory bank with 𝐷 2 embedding vectors, {𝒎 1 , • • • , 𝒎 𝐷 2 }, we construct a mapping 𝑉 𝑄:𝑉 𝑄 (𝒛) = 𝒎 k , where k = argmin 𝑘 ∥𝒛(𝒙) -𝒎 𝑘 ∥ 2 , (9)where each embedding 𝒛 is concatenated to generate matrix Z = 𝑉 𝑄 (𝒁 (𝐾 𝑒 ) ). The mapping connects continuous vectors with given vectors in the memory bank to save the computational cost. Then, to minimize the information loss, we map the concatenated matrix back to the input using a new decoder, i.e., Ṽ = 𝐷𝑒𝑐 ( Z ). The whole framework is optimized using the following objective as:EQUATIONwhere 𝛽 denotes a parameter to balance these objective and sg(•) is the stopgradient operator to cut off the gradient computation during back propagation. Here, the first term denotes the reconstruction loss and the last two terms minimize the quantization loss between continuous embedding vectors and their neighbours in the memory bank. Dynamic Propagation. After training the variational autoencoder, we remove the decoder, and then feed the quantized vector into temporal convolution on 𝑇 × 𝐷 channels. In particular, each temporal convolution block involves a bottleneck followed by group convolution operator:EQUATIONwhere Bottleneck denotes the 2D convolutional layer with 1 × 1 kernel and 𝐾 𝑡 is the number of blocks. The shape of input 𝒁 (𝑖 -1) and output 𝒁 (𝑖 ) are (𝑇 , 𝐷, Ĥ, Ŵ ) and ( T , 𝐷, Ĥ, Ŵ ), respectively. Decoder. Finally, our decoder contains 𝐾 𝑑 unConvNormReLU blocks to output the final predictions Ŷ 𝐷𝑆𝑇 = 𝒁 (𝐾 𝑒 +𝐾 𝑡 +𝐾 𝑑 ) . In formulation, we have: 1) ,𝑍 (𝑖 ) =𝜎 LayerNorm unConv2d 𝑍 (𝑖 -EQUATIONwhere unConv2d is implemented using ConvTranspose2d [12] . The shape of input 𝒁 (𝑖 -1) and output 𝒁 (𝑖 ) are ( T , 𝐷, Ĥ, Ŵ ) and (𝑇 , 𝐶, 𝐻,𝑊 ), respectively.Finally, we combine the output of both FPG and DST modules, which result in the final prediction:EQUATIONwhere ⊕ represents element-wise addition. The whole framework would be optimizing by minimizing the vanilla MSE loss between the predictions and the target frame. 
experiment 4.1 experimental setups: 
 Datasets. In this paper, the datasets studied can be classified into two categories from the perspective of PDE modeling or physics We conduct a thorough evaluation of PastNet by comparing it with several baseline models on both non-natural and natural phenomena datasets. This includes competitive RNN architectures such as ConvLSTM [39] , PredRNN-V1-2 [47] , E3D LSTM [46] , SA-ConvLSTM [25] , PhyDnet [16] , and MIM [48] . We also evaluate state-of-the-art CNN architecture SimVP [14] for non-natural phenomena datasets. For natural phenomenon datasets, we evaluate models that incorporate physical information, such as DLP [10] , which uses an advection-diffusion flow model and achieves stateof-the-art performance on the SST dataset, it is commonly used for generic physical processes. We also evaluate NLDM [6] , which combines manifold learning theory and autoencoder to discover fundamental variables hidden in physical experimental data for spatiotemporal prediction, as well as PhyDnet and SimVP. Our evaluation is meticulous to ensure the validity of the results. Table 2 demonstrates that PastNet outperforms other models on non-natural phenomena datasets. Specifically, PastNet achieves the best MSE and MAE metrics on MovingMNIST, with values that are 20% and 10% lower than SimVP, respectively. Moreover, PastNet also achieves higher MS-SSIM and PSNR metrics than other models, indicating better prediction ability for dynamic changes in videos. On TrafficBJ, while PhyDnet has the best MSE and PSNR performance, PastNet comes in second place and achieves top spot for MAE and MS-SSIM. On KTH, PastNet achieves the best MSE and MAE performance, with values that are 33.6% and 35.8% lower than the second-place MIM model, respectively. Overall, PastNet shows significant advantages over other baseline methods in all evaluation metrics.Results in Table 3 show that PastNet achieves the best performance in most evaluation metrics for all natural phenomena datasets. On the SEVIR dataset, PastNet achieves the lowest MSE and MAE scores, significantly better than other methods such as DLP, PhyDnet, and NLDM. On the RDS dataset, PastNet achieves the lowest MSE x 100 score, significantly better than other methods. On the EDPS dataset, PastNet achieves the highest MS-SSIM score, significantly better than SimVP. Overall, these results demonstrate that PastNet is highly effective in accurately predicting physical quantities and preserving structural information in various physical datasets, outperforming other state-of-the-art methods such as DLP, PhyDnet, NLDM, and SimVP.We present the qualitative prediction results of PastNet for various datasets, highlighting its capability to accurately predict future images. Our findings demonstrate that PastNet reliably predicts traffic flow changes in the TrafficBJ dataset. Furthermore, PastNet performs well in the SEVIR weather dataset, as shown in Figure 4 , where it accurately predicts high-resolution satellite images and infers future weather changes.In this section, we demonstrate PastNet's competitive performance on a wide range of datasets through a series of ablation studies. Table 4 and Figure 7 As shown in Table 5 , we select the latest baseline models and compare their efficiency on TrafficBJ and NS equations. PastNet has significantly lower FLOPs, parameter count, and inference time than SimVP and TAU, while also achieving better MSE and SSIM. In the first experiment, PastNet's FLOPs are 816.32M, parameters are 1.20M, inference time is 0.92s, MSE is 42.93, and SSIM is 0.9876. In the second experiment, FLOPs are 3269.70M, parameters are 1.20M, inference time is 1.02s, MSE is 30.21, and SSIM is 0.9655. Overall, PastNet excels in computational efficiency and prediction accuracy.To investigate the potential of PastNet for solving physical problems, we consider the Navier-Stokes equations, representing viscous incompressible fluids as vorticity on the unit torus, and the Shallow-Water equations, derived from the general Navier-Stokes equations. We focus on their 2D forms.We use PastNet to solve the Navier-Stokes equations with viscosity 𝜈 = 10 -3 and a resolution of 64 × 64 for training and testing. We The results in Table 6 show that PastNet has potential for solving PDE equations, especially for the SWE equation. It achieves lower MSE and MAE values for the SWE equation than for the NSE equation, while the time per epoch is similar for both equations. 
conclusion: 
 In this paper, we investigate the problem of spatio-temporal video prediction and propose a novel method named PastNet to tackle the problem. The key insight of our PastNet is to incorporate a spectral convolution operator in the Fourier domain, which effectively introduces inductive biases from the underlying physical laws. Moreover, we introduce both local feature discretization and intrinsic dimensionality estimation to reduce the computational costs with accuracy retained. Extensive experiments on a range of popular datasets show that the proposed PastNet is more effective and efficient than state-of-the-art techniques. In future works, we would develop more effective video prediction techniques by introducing high-level physical domain knowledge in various fields. 
 
 Appendix: MM'24, October 28 -November 1, 2024, Melbourne, Australia Hao Wu, Fan Xu, Chong Chen, Xian-Sheng Hua, Xiao Luo, and Haixin Wang 


Decision: The key novelty of the paper is introducing physical inductive biases for spatiotemporal video prediction. This, coupled with intrinsic dimensionality estimation with a memory bank, reduces computational time. The paper received 3 BAs and 1 BR after rebuttal. The remaining concerns are persuasion of visualization (Reviewer cQR6), more evaluation metrics (Reviewer LhCq), and addressing the connection and difference between the studied problem and spatio-temporal data prediction (Reviewer LhCq). The reviewers all liked the paper's new insight. The AC likes the paper's contribution and thinks the remaining concerns are addressable in a minor revision and, hence, recommends acceptance.

Reviews: review: paper_summary: paper_summary: The paper investigates the challenges of spatio-temporal video prediction, which involves generating future videos based on historical data streams. The paper proposes the novel approach of  inherent physical knowledge embedded within videos, and  introduce a novel approach called Physics-assisted Spatio-temporal Network (PastNet) for generating high-quality video prediction.
strength_weakness: strength_weakness: The paper opens a new perspective to link spatio-temporal video prediction with physical inductive bias so as to augment the model with the data itself.The proposed PastNet not only uses convolution operators in the spectral space to incorporate the physical prior, but also discretises the local features using memories with estimated intrinsic dimensions to improve the efficiency of high-resolution video prediction.The paper declares that PastNet has high performance and efficiency.
limitations: limitations: 1,the paper used for comparison should be more recent, e.g., Modular Action Concept Grounding in Semantic Video Prediction, 2022. comparing Correspondences: Video Prediction with Correspondence-wise Losses, 2022. MMVP: Motion-Matrix-Based Video Prediction 2023.
2,In the DST module mentioned in the paper, discrete libraries are built to save computational costs. Generally libraries are created to make better use of historical information, the article is how to achieve cost reduction by creating discrete libraries.
paper_summary: paper_summary: The proposed PastNet model introduces both Fourier-based Physics-Guided (FPG) and Discrete Spatio-Temporal (DST) modules, which effectively improve spatio-temporal video prediction by incorporating physical principles and intrinsic dimensionality estimation.
strength_weakness: strength_weakness: * The paper is well-structured and well-written.
* The method and experiments are persuasive.
limitations: limitations: **Weakness**
1. The claimed efficiency is limited if it is only reflected in training time. SimVP seems simpler and more efficient.
2. The Fourier-based token mixer is not novel.
3. The comparison experiments are unfair.

**Questions**
1. Compared to other algorithms, what are the parameter count and FLOPs of PastNet? What is its inference speed on real devices?
2. The experiments do not compare against the latest algorithms. It seems the most advanced work cited, SimAP, is from 2022. However, to my knowledge, there have been some recent works in 2023 and very recently [1].
3. The experimental dataset settings are inconsistent with those of the compared algorithms. Why?
4. In Table 2, citations and publication years for the compared algorithms can be added to help readers locate them.

[1] Tan, Cheng, et al. "Openstl: A comprehensive benchmark of spatio-temporal predictive learning." Advances in Neural Information Processing Systems 36 (2023): 69819-69831.
paper_summary: paper_summary: This paper studies an important problem of spatio-temporal data prediction. The authors propose a physics-assisted spatio-temporal network called PastNet to perform high-quality spatio-temporal data synthesis. The proposed PastNet considers the inherent physical knowledge embedded features and reduces the computational overhead. The extensive experiments show the superiority of the proposed method.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow. The organization of the paper is clear.

2. The proposed PastNet incorporates a spectral convolution operator in the Fourier domain to learn the inherent physical knowledge, which provides explainability.

3. The authors design a memory bank with the estimated intrinsic dimensionality to reduce the computational cost.

4. Extensive experiments offer the evidence of the effectiveness of the proposed method.
limitations: limitations: 1. It is suggested to illustrate the relationship between the proposed method (or the studied problem) and multimedia, which seems to be a little bit unclear.

2. It would be better to provide a discussion to show how the proposed FPG module is explainable. In addition, how does the proposed FPG module introduce physical inductive biases for the studied problem?

3. Compared to Simvp, the improvement of PastNet is limited in terms of computational overhead and accuracy, especially on EDPS. It is suggested to provide the advantages of PastNet. In addition, it is suggested to discuss the computational complexities (space and time complexities) of PastNet and baselines, to demonstrate the efficiency of PastNet theoretically. 

4. The evaluation metrics are not specified for spatio-temporal data prediction, which may neglect the specific and important spatio-temporal characteristics. More evaluation metrics are suggested to be included. Please refer to the following paper.

    [1] Sai et al., TIME WEAVER: A Conditional Time Series Generation Model, ICML 2024.

6. It is suggested to investigate more related work as follows.

    [2] Lin et al., Self-Attention ConvLSTM for Spatiotemporal Prediction, AAAI 2020.

    [3] Liang et al., GeoMAN: Multi-level Attention Networks for Geo-sensory Time Series Prediction, IJCAI 2018.

    [3] Wang et al., Multi-task adversarial spatial-temporal networks for crowd flow prediction, CIKM 2020.

    [4] Zhang et al., Predicting citywide crowd flows using deep spatio-temporal residual networks, Artificial Intelligence 2018. 

    [5] Xu et al., PIGAT: Physics-Informed Graph Attention Transformer for Air Traffic State Prediction, TITS 2024.

7. The studied problem seems similar to spatio-temporal data prediction, especially physics-informed spatio-temporal data prediction. It would be better to show the connection and difference between the studied problem and spatio-temporal data prediction. There are some physics-informed spatio-temporal data prediction methods [5, 6], especially in terms of traffic prediction, which are suggested to be compared for fairness.

    [6] Ji et al., STDEN: Towards physics-guided neural networks for traffic flow prediction, AAAI 2022.
paper_summary: paper_summary: This paper investigate spatio-temporal video prediction task, generating future video frames based on existing observation streams. Authors focus on physical knowledge embedded within videos and computational costs, proposing the PastNet. Extensive experiments on  spatio-temporal video benchmarks demonstrate the effectiveness.
strength_weakness: strength_weakness: + Spatio-temporal video prediction is In line with the theme of the ACM MM.
+ Paper is well-organized and the layout is orderly.
+ Authors initiate a new insight on spatio-temporal video prediction with physical inductive biases.
+ The experiments are abundant and the metrics are comprehensive
limitations: limitations: If authors can solve all issues I concerned, $\textbf{I am willing to raise my score according to rebuttal.}$
- Authors declare that their method has a better performance and show their prediction results on the SEVIR dataset in Figure 5. However, the comparison with prediction results of other SOTA is absent. It's not convincing. Authors are suggested to reveal their results like Figure 6. The same applies to Figure 3, 4 and Figure 1, 2 (In appendix).
- Computational costs are emphasized by authors. I can see authors show training in Table 4 while complexity is not discussed. What about the parameter number and Flops of the PastNet?
- According to the results in Table 1, 2. The improvement of PastNet is not clear when compared with SimVP.
- Authors should compare their method with recent method such as:
    - Ning et al. 2023. MIMO is all you need: A strong multi-in-multi-out baseline for video prediction. In AAAI.
    - Tan et al. 2023. Openstl: A comprehensive benchmark of spatio-temporal predictive learning. In NeurIPS.
- Some long and difficult sentences make it difficult to understand and writing mistakes including: 
    - expression of disunity “spatiotemporal prediction” or “spatio-temporal prediction” in the main paper. 
    - Subsection ?? at the end of the appendix.