MAU: A Motion-Aware Unit for Video Prediction and Beyond
Year: 2022

Abstract:
Accurately predicting inter-frame motion information plays a key role in video prediction tasks. In this paper, we propose a Motion-Aware Unit (MAU) to capture reliable inter-frame motion information by broadening the temporal receptive field of the predictive units. The MAU consists of two modules, the attention module and the fusion module. The attention module aims to learn an attention map based on the correlations between the current spatial state and the historical spatial states. Based on the learned attention map, the historical temporal states are aggregated to an augmented motion information (AMI). In this way, the predictive unit can perceive more temporal dynamics from a wider receptive field. Then, the fusion module is utilized to further aggregate the augmented motion information (AMI) and current appearance information (current spatial state) to the final predicted frame. The computation load of MAU is relatively low, and the proposed unit can be easily applied to other predictive models. Moreover, an information recalling scheme is employed into the encoders and decoders to help preserve the visual details of the predictions. We evaluate the MAU on both video prediction and early action recognition tasks. Experimental results show that the MAU outperforms the state-of-the-art methods on both tasks. ∗Corresponding author: Shanshe Wang. 35th Conference on Neural Information Processing Systems (NeurIPS 2021).

text: Accurately predicting inter-frame motion information plays a key role in video prediction tasks. In this paper, we propose a Motion-Aware Unit (MAU) to capture reliable inter-frame motion information by broadening the temporal receptive field of the predictive units. The MAU consists of two modules, the attention module and the fusion module. The attention module aims to learn an attention map based on the correlations between the current spatial state and the historical spatial states. Based on the learned attention map, the historical temporal states are aggregated to an augmented motion information (AMI). In this way, the predictive unit can perceive more temporal dynamics from a wider receptive field. Then, the fusion module is utilized to further aggregate the augmented motion information (AMI) and current appearance information (current spatial state) to the final predicted frame. The computation load of MAU is relatively low, and the proposed unit can be easily applied to other predictive models. Moreover, an information recalling scheme is employed into the encoders and decoders to help preserve the visual details of the predictions. We evaluate the MAU on both video prediction and early action recognition tasks. Experimental results show that the MAU outperforms the state-of-the-art methods on both tasks.
∗Corresponding author: Shanshe Wang.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

heading: 1 Introduction
text: Video prediction is a representative task in video predictive learning area, which aims to predict the unknown future on the basis of the limited knowledge and has been applied in a wide range of research areas, such as robotic control [1], video interpolation [2], autonomous driving [3], motion planning [4] and so on. However, compared with images, videos are more complex due to the time-varying motion information and predicting reliable motion information has always been a significant but challenging problem for video prediction tasks. Fortunately, deep learning technologies have shown their great power in learning meaningful features for multimedia data and have achieved great success in computer vision and natural language processing tasks. Motivated by this, learning-based methods have been applied for video prediction in recent years.
Recurrent neural networks (RNNs) are first applied to learn video representations due to their unique advantages in modeling sequential data [5]. Then the Long Short-Term Memory (LSTM) [6] and Gated Recurrent Unit (GRU) [7] are integrated into RNNs to help capture more reliable inter-frame temporal dependency [1, 8, 9, 10, 11, 12, 13, 14, 15]. In general, to save the computation resources and help predictive units to better perceive visual information, the fully connected layers in the predictive memories are replaced by convolutional layers in the above methods. Although the spatial receptive field of the unit has been improved by the integrated convolutional layers, the temporal receptive field is still narrow, and it is difficult for the unit at current time step to perceive what has happened in a longer past, which severely restrict the model expressivity to inter-frame motion information and the performance in predicting videos with complex scenarios and high resolutions is far from satisfactory.
Some works have attempted to broaden the temporal receptive field for the predictive units using 3D convolutional layers [16, 17]. However, the temporal receptive field is mainly determined by the kernel size of the integrated convolutional operators and the temporal dimension still needs to be set to a small value to meet the computation load requirement. Since then, the explorations for broadening the temporal receptive field for predictive models have been shelved and a variety of works begin to explore other ways to improve the expressivity of the model on videos with complex scenarios, which can be roughly categorized into two types, the structure-oriented methods and the loss-oriented methods. The structure-oriented methods utilized deep stochastic models to predict different futures for different samples based on their latent variables [18, 19, 20, 21]. However, the computation load of these methods is typically high, preventing their practicability in real world. And the loss-oriented methods aim to improve the traditional mean square error (MSE) based loss functions to generate more naturalistic results. Generative adversarial networks (GANs) [22, 23, 24, 25], perceptual loss [26] and so on have been utilized to generate results with higher perceptual quality. In spite of the explorations made by the above methods, only limited model performance improvements have been achieved and the unsatisfactory temporal receptive field still restricts the model performance in capturing reliable motion information between frames.
To solve the above problem, we propose the Motion-Aware Unit (MAU) to improve the model expressivity in capturing motion information by efficiently broadening the temporal receptive field. In particular, for each MAU, two modules are designed, the attention module and the fusion module. The attention module is designed for efficient attention and the fusion module is designed for efficient fusion. In particular, the attention module aims to help the unit to pay different levels of attention to the temporal states in the broadened temporal receptive field based on the corresponding spatial correlation scores. Using the attention scores, the temporal states can be aggregated to a more reliable augmented motion information (AMI) with a low computation load. The fusion module aims to further aggregate the augmented motion information (AMI) and the appearance information (the spatial state from current time step) using only two update gates. Moreover, an information recalling scheme is applied to further preserve the visual details of the predictions. Experimental results show that the proposed MAU can outperform other state-of-the-art methods on both video prediction and early action recognition tasks.

heading: 2 Related Work
text: In this section, we introduce the learning-based video prediction methods in detail. Due to the unique power in modeling sequential data, RNNs are first utilized to model videos. Ranzato et al. [5] utilized RNNs to propose a baseline model for unsupervised feature learning on video data. Srivastava et al.
[8] further integrated Long Short-Term Memories (LSTMs) [6] into RNN-based models to capture the temporal long-short term dependencies for videos, which is denoted as FC-LSTM. However, the fully connected layers in FC-LSTM are computation-expensive, which limits its practicability in real world. To further reduce the computation load and increase local perceptions to visual data for LSTMs, Shi et al. [9] proposed the ConvLSTM by replacing the fully connected layers with convolutional layers. Ballas et al. [10] further integrated convolutional layers into Recurrent Gated Unit (GRU) [7], denoted as ConvGRU, which has achieved a similar performance but with lower computation load compared with ConvLSTM. Motivated by the success achieved from the above video prediction methods, RNNs have been further explored to predict videos with higher visual quality and video prediction models are beginning to be applied into more research areas, such as robotics [1], precipitation nowcasting [11] and so on. However, the above works merely focus on exploring the temporal dependency but ignore the spatial features for videos. Wand et al. proposed PredRNN [13] to solve this problem by adding a spatial information processing module for ConvLSTMs. And to solve the gradient propagation difficulties in PredRNNs, Wang et al. further designed a Gradient Highway Unit [14] and the new model was denoted as PredRNN++.
Although the above RNN-based models have achieved some satisfactory results, the datasets utilized are either with simple scenarios or low resolutions and the model performance on videos with complex scenarios and high resolutions is still far from satisfactory. One main reason for this is that the predicted frames at current time step still mainly depend on the inputs from current time step and the temporal receptive field of the predictive model is narrow, which is not enough to predict reliable motion information for the next time step. To solve this problem, Wang et al. employed 3D convolutional layers into PredRNNs and utilized multi-term temporal states to help increase temporal receptive field for the predictive unit, which is denoted as E3D-LSTM [16]. However, the computation load of E3D-LSTM is extremely high due to the integrated 3D convolutional operators and the recall gate. To further improve the model expressivity for the predictive models, many other methods have also been proposed, which can be summarized into four types. The first type aims to protect the visual details for the predictions [12, 17, 27, 28]. The second type aims to predict different futures rather than an averaged future for each sample based on their latent variables [18, 19, 21]. The third type aims to refer to deeper models to increase the model expressivity [20, 29, 30]. And the last type aims to improve the loss functions to generate more naturalistic results [22, 23, 24, 25, 26]. However, in the above methods, the temporal receptive field problem was still not fully explored. In this paper, we propose the motion-aware unit (MAU), which can efficiently broaden the temporal receptive field to predict more reliable motion information for videos and the computation load is relatively low. In addition, MAU can be easily employed in other predictive models.

heading: 3 Method

heading: 3.1 Problem formulation
text: A video prediction model typically takes a video clip {v1, ..., vi} as the inputs and outputs the future video clip {v̂i+1, ..., v̂T }. And we want to optimize the following problem,
min T∑ t=i+1 [D(v̂t, vt)], (1)
where v̂t denotes the predicted frame at time step t, D denotes the loss function, such as the L1, L2 loss functions and so on.
To optimize the above problem, the error between the predicted frame v̂t and the ground truth vt is expected to be as small as possible. For most of the RNN-based video prediction methods, the frames are usually progressively predicted where the proposed model predicts one frame for each time step. In particular, the predicted frame v̂t depends on the spatial information, i.e. the frame input at the previous time step vt−1 and the temporal information, i.e. the transited motion information Tt−1. However, as the time steps move on, the predicted error in Eq. 1 dramatically accelerates due to the increasing uncertainty of the transited temporal information Tt−1. To solve this problem, the predictive unit needs to search for more useful temporal information from a wider range of time steps, i.e. the temporal receptive field is needed to be broadened. For time step t, the temporal receptive field is probably to be set as t− 1. However, it may be not efficient and necessary to utilize all the previous frames, thus the receptive field is more likely to be set to a fixed value τ . Although recent
work E3D-LSTM [16] has attempted to address this problem and the temporal receptive field is broadened, the computation load is extremely high with only a limited performance improvement (mainly achieved by the integrated 3D convolutional layers), which indicates the broadened temporal receptive field may not been fully used.
To ensure that the broadened temporal receptive field can be fully utilized, two problems are needed to be solved,
• The temporal states in current receptive field should be aggregated according to their importance.
• The motion information from the aggregated temporal state and the appearance information from the spatial state should be fused reasonably.
To solve the above two problems, we propose the Motion-Aware Unit (MAU) and the predicted frame v̂t can be represented as follows,
v̂t = Dec[MAU(Enc(vt−τ :t−1))], (2)
where Enc(·) denotes the encoder which is utilized to extract deep features from the video input and Dec(·) denotes the decoder which is utilized to map the predicted features to the frames. MAU(·) denotes the proposed motion-aware unit, which will be detailedly introduced in Section 3.2.

heading: 3.2 The proposed Motion-Aware Unit: MAU
text: In this section, we introduce the structure details of MAU, as shown in Fig. 1 (right). Each MAU consists of two modules: the attention module and the fusion module. Typically, to improve the model expressivity, multiple MAUs are stacked, as shown in Fig. 1 (left). In particular, at time step t in layer k, two inputs will be fed into the MAU: the temporal states set T kt−τ :t−1 from previous τ time steps and the spatial states set Sk−1t−τ :t from previous τ + 1 time steps.
To solve the first problem in Section 3.1, we design the attention module, which aims to help the predictive unit to pay different levels of attention to different historical temporal states. The expected situation is that the predictive can always pay the highest level of attention to the most correlated states. Now the problem comes to how to quantify the correlations between different temporal states. Considering the fact that the visual quality of each frame can be the most factors to evaluate a video prediction model, the correlations between the corresponding spatial states can be an optimal choice.
Based on the analysis, the attention score αj for the temporal state T kt−j can be denoted as follows,
αj = eqj∑τ i=1 e qi ,
qi = SUM(S k−1 t−i ⊙ S ′), i = 1, ..., τ, S′ = Ws ∗ Sk−1t , (3)
where ⊙, ∗ denote the Hadamard product and the convolutional operator, respectively. Using the computed attention score, the temporal state set can be aggregated as follows,
Tatt = τ∑ j=1 αj · T kt−j . (4)
Tatt can be treated as the long-term motion information. However, besides the long-term motion information Tatt, the short-term motion information T kt−1 is also needed to be utilized to strengthen the final motion information. We define a fusion gate Uf to control the fusion process, which can be shown as follows,
Uf = σ(Wf ∗ T kt−1), TAMI = Uf ⊙ T kt−1 + (1− Uf )⊙ Tatt, (5)
where σ denotes the sigmoid function, TAMI denotes the augmented motion information.
To solve the second problem in Section 3.1, we design the fusion module to aggregate the motion information in the augmented motion information TAMI with the appearance information in the current input Sk−1t . To control the fusion ratios for both temporal and spatial information, two update gates are denoted as follows,
Ut = σ(Wtu ∗ TAMI), Us = σ(Wsu ∗ Sk−1t ), (6)
where Ut denotes the temporal update gate and Us denotes the spatial update gate. Using both gates, the aggregating process can be conducted as follows,
T kt = Ut ⊙ (Wtt ∗ TAMI) + (1− Ut)⊙ (Wst ∗ Sk−1t ), Skt = Us ⊙ (Wss ∗ Sk−1t ) + (1− Us)⊙ (Wts ∗ TAMI) + γ · Sk−1t , (7)
where the residual-like term γ · Sk−1t is utilized to stabilize the training process. In particular, as shown in Fig. 1 (left), the frame input vt is encoded to deep features by the encoder and the predicted spatial state is decoded back to the frame by the decoder, represented as follows,
St0 = Enc(vt),
v̂t+1 = Dec(S N t ), (8)
where N denotes the total number of the employed MAUs.

heading: 3.3 Information recalling scheme
text: Considering the information loss problem during encoding, an information recalling scheme between encoders and decoders have been employed, which is defined as the information recalling scheme and can be represented as follows,
Dl = Decl(Dl−1 + E−l), l = 1, ..., N, (9)
where Dl, E−l denote decoded features from the lth layer of the decoder and the encoded features from the lth from the last layer of the encoder. Decl denotes the lth layer of the decoder. On the basis of the above information recalling scheme, the decoders can recall multi-level encoded information back and the visual quality of the predictions is better.

heading: 4 Experiments

heading: 4.1 Implementations
text: We evaluate the proposed MAU on five datasets, the Moving MNIST dataset [8], the KITTI dataset [31], the Caltech Pedestrian dataset [32], the TownCentreXVID dataset [33] and the SomethingSomething V2 dataset [34]. The number of the hidden state channels of MAUs are set to 64 and the integrated convolutional operators are set with a kernel size 5× 5 and stride 1. All experiments are optimized with the Adam optimizer. To stabilize the training process, we employ layer normalization operators after each integrated convolutional layer in MAUs.

heading: 4.2 Video Prediction
text: We conduct video prediction experiments on the the Moving MNIST dataset, the KITTI dataset, the Caltech Pedestrian dataset, and the TownCentreXVID dataset. The detailed experimental settings are summarized in Table 1. All models are optimized with the MSE loss function.

heading: 4.2.1 Moving MNIST
text: The Moving MNIST dataset can be the most widely-used dataset in video prediction, where each sequence consists of 20 successive frames with 2 digits randomly placed. Each frame is with a size of 64× 64. In our experiments, sequences are generated from the training set of the standard MNIST dataset [35] and we utilize the test set collected by Srivastava et al. [8] to evaluate the proposed model. The Mean Square Error (MSE), the Structural Similarity Index (SSIM) are employed to indicate the visual quality of the predictions.
Fig. 2 shows the visual results predicted from different methods, where the proposed MAU significantly outperforms other methods, especially for the prediction in the last two time steps. Table 2 summarizes the quantitative scores of the predictions from different methods. The proposed MAU has achieved the best scores compared with other state-of-the-art methods. And the employed recalling scheme can help further improve the model performance.
In addition, the parameters and inference time are also recorded in Table 3. For a fair comparison, all models are implemented with the same encoder and decoder with the same number of predictive units. In particular, for the proposed MAU, the recalling scheme is disabled and all models are trained
using Adam optimizer based on the MSE loss. We record the parameters for a single unit and the inference time is recorded over 800 samples. The summarized results show that the MAU can achieve the best scores with the fewest parameters and a relatively low computation load.
Table 4 shows the model performance with different temporal receptive field τ (w/o recalling). Although the quality of the predictions becomes better as the temporal receptive field increases, the computation load also dynamically increases. Thus, τ is typically set to an appropriate value to achieve a satisfactory trade-off between the visual quality and the computation load. In particular, we set τ = 5 in this paper.

heading: 4.2.2 KITTI and Caltech Pedestrian datasets
text: We use two car-mounted camera video datasets to evaluate the performance of MAU in real scenarios. KITTI and Caltech Pedestrian datasets are very similar to the real-world scenarios, which are collected to train the autonomous vehicles.
We follow the experimental settings in [37], where all frames are cropped and resized to 128× 160. The proposed model is trained on the KITTI dataset and tested on the Caltech Pedestrian dataset. In
particular, the frame rate of the Caltech Pedestrian dataset is adjusted to the same as KITTI (10 fps). A total of 32373 sequences are for training and 7725 sequences for testing. In addition, the proposed model is trained to predict the next frame with the first 10 frames as the inputs. While testing, the temporal period of the predictions is extended to 10 frames.
Fig. 3 shows the generated examples from the proposed MAU and two latest state-of-the-art methods. From the visual results, the proposed MAU can predict more clear traffic signs (t = 16), which indicates the MAU can capture more reliable temporal dynamics for videos. Table 5 shows the quantitative results from different methods, where the proposed MAU outperforms other methods in all scores. In particular, the recalling scheme can further improve the model performance.

heading: 4.2.3 TownCentreXVID
text: In this section, we evaluate the proposed model on a surveillance dataset, TownCentreXVID, which is more close to the real scenarios and with a resolution of 1920× 1080. TownCentreXVID dataset contains a total of 7500 frames. To further evaluate the model performance in predicting highresolution videos, all frames are not resized. The first 4500 frames are for training and the last 3500 frames are for testing. Fig. 4 shows the visual results from different methods, where the proposed method can generate much better visual details compared with other state-of-the-art methods.
We further conduct object detection tasks on the predictions from different methods using the pretrained Yolov5s model [41] and the results are shown in Fig. 5. More persons have been detected by the pre-trained Yolo model from the predictions generated from MAU and the confidence is higher than others.
In addition, Table 6 summarized the detailed quantitative results from MAU and other methods, where MAU has achieved the best objective (PSNR, SSIM) and perceptual (LPIPS) scores.

heading: 4.3 Early action recognition: Something-Something V2
text: The Something-SomethingV2 dataset is a large collection of labeled video clips that show humans performing pre-defined basic actions with everyday objects. The whole dataset consists of 174 categories of videos. The training set contains 168,913 videos and the validation set consists of 24,777 videos. The resolution of each video is 240 × 427. To further evaluate the performance in modeling high-level spatiotemporal representations for videos, the early action recognition task is conducted, which aims to categorize the whole video after observing only the front part of the videos. To accurately predict an activity category for current video, models need to extract useful spatiotemporal representations from merely the front part of the whole frames to predict a reliable future. In particular, we utilize the front 25% and 50% frames of each video to conduct this task, respectively. For each video, a total of 20 frames are sampled which can cover the whole temporal period. Each frame is resized to 128× 128. For 25% early action recognition task, 5 frames are as
the inputs to predict the next 15 frames. For 50% early action recognition task, 10 frames are as the inputs to predict the next 10 frames.
All models are first pre-trained to perform video prediction task with the front 25% or 50% frames as the inputs to predict the remaining 75% or 50% frames. After convergence, a total of 19 hidden states extracted from the pre-trained models are concatenated to train the classifier, which consists of 3 3D convolutional layers. In particular, the concatenated hidden states are transformed from C ×T ×H ×W : 64× 19× 16× 16 to 174× 1. To evaluate the performance of different predictive units in modeling spatiotemporal representations for videos, we utilize various predictive units as the backbone of the early action recognition model. For fair comparison, the encoder, decoder and classifier are set with the same structure for all models. A total of 16 predictive units are stacked for all models. The experimental results are summarized in Table 7. On the one hand, the predictive model with MAU has achieved the best PSNR score of the predictions. On the other hand, based on the learned spatiotemporal representations from the pre-trained models, the proposed MAU can also obtain the highest Top1 and Top5 accuracies.

heading: 5 Conclusion
text: We proposed the Motion-aware Unit (MAU) for video prediction and beyond. The motion-aware unit can take advantage in the broadened temporal receptive field, where more temporal states can be simultaneously perceived. In particular, the proposed unit are constructed based on the attention mechanism, which consists of two modules, the attention module and the fusion module. In particular, the attention module can extract attention weights from multi-term spatial states for multi-term temporal states. And the augmented motion information (AMI) can be aggregated from the multi-term temporal states. Then the fusion module is utilized to further aggregate the AMI and the spatial information to predict the final video frame. Although more states have been aggregated, the computation load is still relatively low because of the efficient structure in MAU. Moreover, an information recalling scheme was employed to help the decoders recall more multi-level encoded information. The proposed model was evaluated on two tasks, the video prediction task and the video classification task. Experimental results showed the proposed MAU can outperform other state-of-the-art methods on both tasks.

heading: 6 Limitations and Future Work
text: There are lots of issues that urgently need to be solved in the video prediction area and this work is also far from completely solving all the problems, such as high-resolution video prediction, low accuracy in action recognition, not practical enough to be applied into other downstream tasks and so on. All these problems need to be carefully taken into consideration in our future work. In addition, videos are still predicted in a deterministic way rather than a stochastic way, which may not practical enough for the complex real-world scenarios. Thus, in our future work, we will explore more to combine the spatiotemporal dynamics in videos into stochastic models, which are more similar to real-world scenarios.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China (62072008, 62025101), PKU-Baidu Fund (2019BD003) and High-performance Computing Platform of Peking University, which are gratefully acknowledged.

Review summaries:
This paper proposes a method for video prediction that relies on a newly proposed Motion-Aware Unit (MAU). MAU acts as a temporal receptive field based on an attention mechanism that is used to aggregate features from previous frames in order to predict future frames. The aggregate features are combined with “content” features extracted from the last observed frame and the decoder is connected to the encoder via an “Information recalling scheme” that directly passes the last observed frame features from each layer of the encoder to each layer of the decoder. In experiments, the authors show superior performance against the baselines pixel based metrics and a perceptual metric.

The paper proposes the Motion-Aware Unit (MAU) for video prediction. This module attempts to model the temporal information in videos by enlarging the temporal receptive field and aggregating it through an attention mechanism. The proposed module is employed in the bottleneck part of an autoencoder where the spatial and temporal information are combined at different MAU layers. The proposed method is compared to previous works on several benchmarks for next frames prediction and early action recognition showing state-of-the-art results.

In this work, the authors introduce a Motion-Aware Unit (MAU) for video prediction. Basically, they discover the previous motion and aggregate it via attention module and fusion module. In the attention module, they compute correlation between spatial states and use it as attention for aggregation the previous temporal states. In the fusion module, they design spatial and temporal update gates for fusing attentive temporal state and current spatial state.

This paper proposes a Motion-Aware Unit (MAU) for video prediction tasks. The MAU includes two modules, the attention module and the fusion module. The attention module aims to learn the correlation between the current spatial state and previous spatial states. The learnt attention map can be used for augmenting the motion information (AMI). The fusion module is used for aggregating the augmented motion information (AMI) and current spatial state. The authors have demonstrated superior performance compared to state-of-the-arts.