MAU: A Motion-Aware Unit for Video Prediction and Beyond

Accurately predicting inter-frame motion information plays a key role in video prediction tasks . In this paper , we propose a Motion-Aware Unit ( MAU ) to capture reliable inter-frame motion information by broadening the temporal receptive field of the predictive units . The MAU consists of two modules , the attention module and the fusion module . The attention module aims to learn an attention map based on the correlations between the current spatial state and the historical spatial states . Based on the learned attention map , the historical temporal states are aggregated to an augmented motion information ( AMI ) . In this way , the predictive unit can perceive more temporal dynamics from a wider receptive field . Then , the fusion module is utilized to further aggregate the augmented motion information ( AMI ) and current appearance information ( current spatial state ) to the final predicted frame . The computation load of MAU is relatively low , and the proposed unit can be easily applied to other predictive models . Moreover , an information recalling scheme is employed into the encoders and decoders to help preserve the visual details of the predictions . We evaluate the MAU on both video prediction and early action recognition tasks . Experimental results show that the MAU outperforms the state-of-the-art methods on both tasks . ∗Corresponding author : Shanshe Wang . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Introduction . Video prediction is a representative task in video predictive learning area , which aims to predict the unknown future on the basis of the limited knowledge and has been applied in a wide range of research areas , such as robotic control [ 1 ] , video interpolation [ 2 ] , autonomous driving [ 3 ] , motion planning [ 4 ] and so on . However , compared with images , videos are more complex due to the time-varying motion information and predicting reliable motion information has always been a significant but challenging problem for video prediction tasks . Fortunately , deep learning technologies have shown their great power in learning meaningful features for multimedia data and have achieved great success in computer vision and natural language processing tasks . Motivated by this , learning-based methods have been applied for video prediction in recent years . Recurrent neural networks ( RNNs ) are first applied to learn video representations due to their unique advantages in modeling sequential data [ 5 ] . Then the Long Short-Term Memory ( LSTM ) [ 6 ] and Gated Recurrent Unit ( GRU ) [ 7 ] are integrated into RNNs to help capture more reliable inter-frame temporal dependency [ 1 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 ] . In general , to save the computation resources and help predictive units to better perceive visual information , the fully connected layers in the predictive memories are replaced by convolutional layers in the above methods . Although the spatial receptive field of the unit has been improved by the integrated convolutional layers , the temporal receptive field is still narrow , and it is difficult for the unit at current time step to perceive what has happened in a longer past , which severely restrict the model expressivity to inter-frame motion information and the performance in predicting videos with complex scenarios and high resolutions is far from satisfactory . Some works have attempted to broaden the temporal receptive field for the predictive units using 3D convolutional layers [ 16 , 17 ] . However , the temporal receptive field is mainly determined by the kernel size of the integrated convolutional operators and the temporal dimension still needs to be set to a small value to meet the computation load requirement . Since then , the explorations for broadening the temporal receptive field for predictive models have been shelved and a variety of works begin to explore other ways to improve the expressivity of the model on videos with complex scenarios , which can be roughly categorized into two types , the structure-oriented methods and the loss-oriented methods . The structure-oriented methods utilized deep stochastic models to predict different futures for different samples based on their latent variables [ 18 , 19 , 20 , 21 ] . However , the computation load of these methods is typically high , preventing their practicability in real world . And the loss-oriented methods aim to improve the traditional mean square error ( MSE ) based loss functions to generate more naturalistic results . Generative adversarial networks ( GANs ) [ 22 , 23 , 24 , 25 ] , perceptual loss [ 26 ] and so on have been utilized to generate results with higher perceptual quality . In spite of the explorations made by the above methods , only limited model performance improvements have been achieved and the unsatisfactory temporal receptive field still restricts the model performance in capturing reliable motion information between frames . To solve the above problem , we propose the Motion-Aware Unit ( MAU ) to improve the model expressivity in capturing motion information by efficiently broadening the temporal receptive field . In particular , for each MAU , two modules are designed , the attention module and the fusion module . The attention module is designed for efficient attention and the fusion module is designed for efficient fusion . In particular , the attention module aims to help the unit to pay different levels of attention to the temporal states in the broadened temporal receptive field based on the corresponding spatial correlation scores . Using the attention scores , the temporal states can be aggregated to a more reliable augmented motion information ( AMI ) with a low computation load . The fusion module aims to further aggregate the augmented motion information ( AMI ) and the appearance information ( the spatial state from current time step ) using only two update gates . Moreover , an information recalling scheme is applied to further preserve the visual details of the predictions . Experimental results show that the proposed MAU can outperform other state-of-the-art methods on both video prediction and early action recognition tasks . 2 Related Work . In this section , we introduce the learning-based video prediction methods in detail . Due to the unique power in modeling sequential data , RNNs are first utilized to model videos . Ranzato et al . [ 5 ] utilized RNNs to propose a baseline model for unsupervised feature learning on video data . Srivastava et al . [ 8 ] further integrated Long Short-Term Memories ( LSTMs ) [ 6 ] into RNN-based models to capture the temporal long-short term dependencies for videos , which is denoted as FC-LSTM . However , the fully connected layers in FC-LSTM are computation-expensive , which limits its practicability in real world . To further reduce the computation load and increase local perceptions to visual data for LSTMs , Shi et al . [ 9 ] proposed the ConvLSTM by replacing the fully connected layers with convolutional layers . Ballas et al . [ 10 ] further integrated convolutional layers into Recurrent Gated Unit ( GRU ) [ 7 ] , denoted as ConvGRU , which has achieved a similar performance but with lower computation load compared with ConvLSTM . Motivated by the success achieved from the above video prediction methods , RNNs have been further explored to predict videos with higher visual quality and video prediction models are beginning to be applied into more research areas , such as robotics [ 1 ] , precipitation nowcasting [ 11 ] and so on . However , the above works merely focus on exploring the temporal dependency but ignore the spatial features for videos . Wand et al . proposed PredRNN [ 13 ] to solve this problem by adding a spatial information processing module for ConvLSTMs . And to solve the gradient propagation difficulties in PredRNNs , Wang et al . further designed a Gradient Highway Unit [ 14 ] and the new model was denoted as PredRNN++ . Although the above RNN-based models have achieved some satisfactory results , the datasets utilized are either with simple scenarios or low resolutions and the model performance on videos with complex scenarios and high resolutions is still far from satisfactory . One main reason for this is that the predicted frames at current time step still mainly depend on the inputs from current time step and the temporal receptive field of the predictive model is narrow , which is not enough to predict reliable motion information for the next time step . To solve this problem , Wang et al . employed 3D convolutional layers into PredRNNs and utilized multi-term temporal states to help increase temporal receptive field for the predictive unit , which is denoted as E3D-LSTM [ 16 ] . However , the computation load of E3D-LSTM is extremely high due to the integrated 3D convolutional operators and the recall gate . To further improve the model expressivity for the predictive models , many other methods have also been proposed , which can be summarized into four types . The first type aims to protect the visual details for the predictions [ 12 , 17 , 27 , 28 ] . The second type aims to predict different futures rather than an averaged future for each sample based on their latent variables [ 18 , 19 , 21 ] . The third type aims to refer to deeper models to increase the model expressivity [ 20 , 29 , 30 ] . And the last type aims to improve the loss functions to generate more naturalistic results [ 22 , 23 , 24 , 25 , 26 ] . However , in the above methods , the temporal receptive field problem was still not fully explored . In this paper , we propose the motion-aware unit ( MAU ) , which can efficiently broaden the temporal receptive field to predict more reliable motion information for videos and the computation load is relatively low . In addition , MAU can be easily employed in other predictive models . 3 Method . 3.1 Problem formulation . A video prediction model typically takes a video clip { v1 , ... , vi } as the inputs and outputs the future video clip { v̂i+1 , ... , v̂T } . And we want to optimize the following problem , min T∑ t=i+1 [ D ( v̂t , vt ) ] , ( 1 ) where v̂t denotes the predicted frame at time step t , D denotes the loss function , such as the L1 , L2 loss functions and so on . To optimize the above problem , the error between the predicted frame v̂t and the ground truth vt is expected to be as small as possible . For most of the RNN-based video prediction methods , the frames are usually progressively predicted where the proposed model predicts one frame for each time step . In particular , the predicted frame v̂t depends on the spatial information , i.e . the frame input at the previous time step vt−1 and the temporal information , i.e . the transited motion information Tt−1 . However , as the time steps move on , the predicted error in Eq . 1 dramatically accelerates due to the increasing uncertainty of the transited temporal information Tt−1 . To solve this problem , the predictive unit needs to search for more useful temporal information from a wider range of time steps , i.e . the temporal receptive field is needed to be broadened . For time step t , the temporal receptive field is probably to be set as t− 1 . However , it may be not efficient and necessary to utilize all the previous frames , thus the receptive field is more likely to be set to a fixed value τ . Although recent work E3D-LSTM [ 16 ] has attempted to address this problem and the temporal receptive field is broadened , the computation load is extremely high with only a limited performance improvement ( mainly achieved by the integrated 3D convolutional layers ) , which indicates the broadened temporal receptive field may not been fully used . To ensure that the broadened temporal receptive field can be fully utilized , two problems are needed to be solved , • The temporal states in current receptive field should be aggregated according to their importance . • The motion information from the aggregated temporal state and the appearance information from the spatial state should be fused reasonably . To solve the above two problems , we propose the Motion-Aware Unit ( MAU ) and the predicted frame v̂t can be represented as follows , v̂t = Dec [ MAU ( Enc ( vt−τ : t−1 ) ) ] , ( 2 ) where Enc ( · ) denotes the encoder which is utilized to extract deep features from the video input and Dec ( · ) denotes the decoder which is utilized to map the predicted features to the frames . MAU ( · ) denotes the proposed motion-aware unit , which will be detailedly introduced in Section 3.2 .

Summary: This paper proposes a method for video prediction that relies on a newly proposed Motion-Aware Unit (MAU). MAU acts as a temporal receptive field based on an attention mechanism that is used to aggregate features from previous frames in order to predict future frames. The aggregate features are combined with “content” features extracted from the last observed frame and the decoder is connected to the encoder via an “Information recalling scheme” that directly passes the last observed frame features from each layer of the encoder to each layer of the decoder. In experiments, the authors show superior performance against the baselines pixel based metrics and a perceptual metric.