MAU: A Motion-Aware Unit for Video Prediction and Beyond
Authors: Zheng Chang
Xinfeng Zhang
Shanshe Wang
Siwei Ma
Yan Ye
Xinguang Xiang
Wen Gao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Temporal receptive field
motion-aware
attention mechanism
video prediction

Abstract: Accurately predicting inter-frame motion information plays a key role in video prediction tasks. In this paper, we propose a Motion-Aware Unit (MAU) to capture reliable inter-frame motion information by broadening the temporal receptive field of the predictive units. The MAU consists of two modules, the attention module and the fusion module. The attention module aims to learn an attention map based on the correlations between the current spatial state and the historical spatial states. Based on the learned attention map, the historical temporal states are aggregated to an augmented motion information (AMI). In this way, the predictive unit can perceive more temporal dynamics from a wider receptive field. Then, the fusion module is utilized to further aggregate the augmented motion information (AMI) and current appearance information (current spatial state) to the final predicted frame. The computation load of MAU is relatively low and the proposed unit can be easily applied to other predictive models. Moreover, an information recalling scheme is employed into the encoders and decoders to help preserve the visual details of the predictions. We evaluate the MAU on both video prediction and early action recognition tasks. Experimental results show that the MAU outperforms the state-of-the-art methods on both tasks.

MAU: A Motion-Aware Unit for Video Prediction and Beyond
Zheng Chang
University of Chinese Academy of Sciences Institute of Computing Technology, Chinese Academy of Sciences changzhengl 80ma i ils. ucas.ac.cn
Xinfeng Zhang
Shanshe Wang Institute of Digital Media, Peking University sswangopku. edu cn
School of Computer Science and Technology, University of Chinese Academy of Sciences xfzhangQucas, ac cn
Siwei Ma
Yan Ye Alibaba Group yan wyweaalbaaa-iinccoommmm com
Institute of Digital Media, Information Technology R&D Innovation Center, Peking University swma@pku. edu. cn
Xinguang Xiang
Wen Gao Institute of Digital Media, Peking University
School of Computer Science and Engineering. Nanjing University of Science and Technology xgxiangenjust en
University of Chinese Academy of Sciences wgao@pku. edu. cn
Abstract
Accurately predicting inter-frame motion information plays key role in video prediction tasks. In this paper, we propose a Motion Aware Unit (MAU) to capture reliable inter-frame motion information by broadening the temporal receptive field of the predictive units. The MAU consists of two modules, the attention module and the fusion module. The attention module aims to learn an attention map based on the correlations between the current spatial state and the historical spatial states. Based on the learned attention map, the historical temporal states are aggregated to an augmented motion information (AMI). In this way, the predictive unit can perceive more temporal dynamics from a wider receptive field. Then, the fusion module is utilized to further aggregate the augmented motion information (AMI) and current appearance information (current spatial state) to the final predicted frame. The computation load of MAU is relatively low, and the proposed unit easily applied to other predictive models. Moreover, an information recalling scheme is employed into the encoders and decoders to help preserve the visual details of the predictions. We evaluate the MAU on both video prediction and early action recognition tasks. Experimental results show that the MAU outperforms the state-of- -feth--rrt methods on both tasks.
"Corresponding author: Shanshe Wang.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Introduction
Video prediction is a representative task in video predictive learning area, which aims to predict the unknown future on the basis of the limited knowledge and has been applied in wide range of research areas, such as robotic control M1. video interpolation 22, autonomous driving [3 motion planning 4 and so on. However, compared with images, videos are more complex due to the time-varying motion information and predicting reliable motion information has always been a significant but challenging problem for video prediction tasks. Fortunately, deep learning technologies have shown their great power in learning meaningful features for multimedia data and have achieved great success in computer vision and natural language processing tasks. Motivated by this, dearning-basee methods have been applied for video prediction in recent years.
Recurrent neural networks (RNNs) are first applied to learn video representations due to their unique advantages in modeling sequential data [5]. Then the Long Short-Term Memory (LSTM) 66] and Gated Recurrent Unit (GRU) 7 are integrated into RNNs to help capture more reliable inter-frame temporal dependency о& 8
13 15). In general, to save the computation resources and help predictive units to better perceive visual information, the fully connected layers in the predictive memories are replaced by convolutional layers in the above methods. Although the spatial receptive field of the unit has been improved by the integrated convolutional layers, the temporal receptive field is still narrow, and it is difficult for the unit at current time step to perceive what has happened in longer past, which severely restrict the model expressivity to inter-frame motion information and the performance in predicting videos with complex scenarios and high resolutions is far from satisfactory.
Some works have attempted to broaden the temporal receptive field for the predictive units using 3D convolutional layers [16 [7]. However, the temporal receptive field is mainly determined by the kernel size of the integrated convolutional operators and the temporal dimension still needs to be set to small value to meet the computation load requirement. Since then, the explorations for broadening the temporal receptive field for predictive models have been shelved and a variety of works begin to explore other ways to improve the expressivity of the model on videos with complex scenarios, which can be roughly categorized into two types, the structure-orient orrieted methods and the loss-oriented methods. The prructure-oriented methods utilized deep stochastic models to predict different futures for different samples based on their latent variables [18] 19 20 21]. However, the computation load of these methods is typically high, preventing their practicability in real world. And the loss-oriented methods aim to improve the traditional mean square error (MSE) based loss functions to generate
more naturalistic results. Generative adversarial networks (GANs) 22 23 24 25], perceptual loss (26 and so on have been utilized to generate results with higher perceptual quality. In spite of the explorations made by the above methods, only limited model performance improvements have been achieved and the unsatisfactory temporal receptive field still restricts the model performance in capturing reliable motion information between frames.
To solve the above problem, we propose the Motion-Aware Unit (MAU) to improve the model expressivity in capturing motion information by efficiently broadening the temporal receptive field. In particular, for each MAU, two modules are designed, the attention module and the fusion module. The attention module is designed for efficient attention and the fusion module is designed for efficient fusion. In particular, the attention module aims to help the unit to pay different levels of attention to the temporal states in the broadened temporal receptive field based on the corresponding spatial correlation scores. Using the attention scores, the temporal states can be aggregated to a more reliable augmented motion information (AMI) with a low computation load, The fusion module aims to further aggregate the augmented motion information (AMI) and the appearance information (the spatial state from current time step) using only two update gates. Moreover, an information recalling scheme is applied to further preserve the visual details of the predictions. Experimental results show that the proposed MAU can outperform other state-of of-the-ar methods on both video prediction and early action recognition tasks.
2 Related Work
In this section, we introduce the learning-based video prediction methods in detail. Due to the unique power in modeling sequential data, RNNs are first utilized to model videos. Ranzato al. (5) utilized RNNs to propose bascline model for unsupervised feature learning on video data. Srivastava et al.

---

8 further integrated Long Short- Memories (LSTMs) [6] into RNN-based models to capture the temporal long-short term dependencies for videos, which is denoted as FC-LSTM. However, the fully connected layers in FC-LSTMM are computation-expensiii
which limits its practicability in real world. To further reduce the computation load and increase local perceptions to visual data for LSTTss, Shi et al. [9] proposed the ConvLSTMM by replacing the fully connected layers with convolutional layers. Ballas et al. [10] further integrated convolutional layers into Recurrent Gated Unit (GRU) 7 denoted as ConvGRU, which has achieved similar performance but with lower computation load compared with ConvLSTM. Motivated by the success achieved from the above video prediction methods, RNNs have been further explored to predict videos with higher visual quality and video prediction models are beginning to be applied into more research areas, such as robotics []]. precipitation nowcasting and on. However. the above works merely focus exploring the temporal dependency but ignore the spatial features for videos. Wand el al. proposed PredRNN [13] to solve this problem by adding spatial information processing module for ConvLSSTMs. And to solve the gradient propagation difficulties in PredRNNs, Wang et al. further designed a Gradient Highway Unit [14
and the new model was denoted NredRNN++.
Although the above RNN-based models have achieved some satisfactory results, the datasets utilized are either with simple scenarios or low resolutions and the model performance on videos with complex scenarios and high resolutions is still far from satisfactory. One main reason for this is that the predicted frames current time step still mainly depend the inputs from current time step and the temporal receptive field of the predictive model is narrow, which is not enough to predict reliable motion information for the next time step. To solve this problem, Wang et al. employed 3D convolutional layers into PredRNNs and utilized multi-term temporal states to help increase temporal receptive field for the predictive unit, which is denoted E3D-LSTM [16]. However, the computation load of E3D-LSTM is extremely high due to the integrated 3D convolutional operators and the recall gate. To further improve the model expressivity for the predictive models, many other methods have also been proposed, which can be summarized into four types. The first type aims to The second type aims to predict different
protect the visual details for the predictions 12 17 27 28).
futures rather than an averaged future for each sample based on their latent variables [88 19, 211.
The third type aims to refer to deeper models to increase the model expressivity (20 29 30 And the last type aims to improve the loss functions to generate more naturalistic results 22 23 24 25 26].
However, in the above methods, the temporal receptive field problem was still not fully explored. In this paper, we propose the motion-aware unit (MAU), which can efficiently broaden the temporal receptive field to predict more reliable motion information for videos and the computation load is relatively low. In addition, MAU can be easily employed in other predictive models.
3 Method
3.1 Problem formulation
A video prediction model typically takes a video clip (ui,...,;; 0/} } as the inputs and outputs the future video clip (ūz-1,..., }. And we want to optimize the following problem.
min ((
(I)
where úe denotes the predicted frame at time step I, D denotes the loss function, such as the Li, L2 loss functions and so on.
To optimize the above problem, the error between the predicted frame én and the ground truth oo expected be as small as possible. For most of the RNN-based video prediction methods, the frames are usually progressively predicted where the proposed model predicts one frame for each time step. In particular, the predicted frame ú, depends on the spatial information, i.c. the frame input at the previous time step 2t-1 and the temporal information, i.e. the transited motion information T- However, as the time steps move on, the predicted error in Eq. [ dramatically accelerates due to the increasing uncertainty of the transited temporal information T--1- To solve this problem, the predictive unit needs to search for more useful temporal information from a wider range of time steps, iee the temporal receptive field is needed to be broadened. For time step I, the temporal receptive field is probably to be set as t-- --1 1. However, may be not efficient and necessary to utilize all the previous frames, thus the receptive field is more likely to be set to fixed value 7. Although recent
3

---

Motion-Aware Unit
Recall
Decoder
Sk-1
S
Hadamard Product
[-7d-1
ATT
Product
C 0.0.0e
Átiention Mop
MAU
Softmax
MUL
T
SUM
MAU
T
St
MAU
T
U
7
1-T8-1
AMI
Eneoddr
Attention Module
Pusion Module
Figure I: Left: The structure of the predictive network with stacked MAUs. Right: The structure of the proposed motion-aware unit: MAU. T denotes the temporal state at time step from layer k. Stt denotes the spatial state at time step from layer k. SUM and MUL represent summation and multiplication.
work E3D-LSTM [6] has attempted to address this problem and the temporal receptive field is broadened, the computation load is extremely high with only a limited performance improvement (mainly achieved by the integrated 3D convolutional layers), which indicates the broadened temporal receptive field may not been fully used.
To ensure that the broadened temporal receptive field can be fully utilized, two problems needed to be solved,
The temporal states in current receptive field should be aggregated according to their importance.
The motion information from the aggregated temporal state and the appearance information from the spatial state should be fused reasonably.
To solve the above two problems, we propose the Motion-Aware Unit (MAU) and the predicted frame d can be represented as follows.
i = AU (En(((r---r
(2)
where Enc(-) denotes the encoder which is utilized to extract deep features from the video input and Dec(-) denotes the decoder which is utilized to map the predicted features to the frames. M AU (-) denotes the proposed motion-aware unit, which will be detailedly introduced in Section 3.2
3.2 The proposed Motion-Aware Unit: MAU
In this section, introduce the structure details of MAU, shown in Fig. | (right). Each MAU consists of two modules: the attention module and the fusion module. Typically, to improve the model expressivity, multiple MAUs stacked, shown in Fig. [ (left). In particular, at time step l in layer k, two inputs will be fed into the MAU: the temporal states set T from previous
time steps and the spatial states set St-7ct from previous
time steps.
To solve the first problem in Section 3.1 we design the attention module, which aims help the predictive unit to pay different levels of attention to different historical temporal states. The expected situation is that the predictive can always pay the highest level of attention to the most correlated states. Now the problem comes to how to quantify the correlations between different temporal states. Considering the fact that the visual quality of each frame can be the most factors to evaluate a video prediction model, the correlations between the corresponding spatial states can be an optimal choice.

---

Based on the analysis, the attention score 0j for the temporal state Ty can be denoted as follows,
e
jj Ei E0
= SUM O i= 1=,,..,
S' W. St-
where O * denote the Hadamard product and the convolutional operator, respectively. Using the computed attention score, the temporal state set can be aggregated as follows,
Tatt
aj T
Torr can be treated as the long-term motion information. However, besides the long-term motion information Tau, the short-term motion information T is also needed be utilized to strengthen the final motion information. We define a fusion gate U to control the fusion process, which can be shown as follows,
= [W T TAMI U, o T + +11 (1-Ug) O Tatts
(5)
where a denotes the sigmoid function, TAMI denotes the augmented motion information.
To solve the second problem in Section 3.1| design the fusion module aggregate the motion information in the augmented motion information TAME with the appearance information in the current input 5 5k-1 To control the fusion ratios for both temporal and spatial information, two update gates denoted as follows,
U, (Wra TAMST).
U, = a (W... * S. -k-1)
(6)
where Ug denotes the temporal update gate and U, denotes the spatial update gate. Using both gates, the aggregating process can be conducted as follows,
The U, @ (Wa TAMT) + (1 - U.) @ (W St-2),
S U. o (W.. .. St )) + U,) ) (Wi. TAMI) + Sk-1
where the residual-like term y s is utilized to stabilize the training process.
In particular, as shown Fig. (eft), the frame input 2t is encoded to deep features by the encoder and the predicted spatial state decoded back to the frame by the decoder, represented follows,
S Ene(v:),
Ū Dec(S,)
(8)
where denotes the total number of the employed MAUs.
3.3 Information recalling scheme
Considering the information loss problem during encoding, an information recalling scheme between encoders and decoders have been employed, which is defined as the information recalling scheme and can be represented as follows,
Dj Deca(D: + E-1). 1= 1,
where Dr, E-z denote decoded features from the 7th layer of the decoder and the encoded features from the th from the last layer of the encoder. Dea denotes the 1th layer of the decoder. On the basis of the above information recalling scheme, the decoders can recall multi-level encoded information back and the visual quality of the predictions is better.

---

Inputs
3
3
3
3
13
3
3
3
73
73
72
7
7
7
7
Ground Truth
3
3
3
7
7
73
73
MAU
3T
3 T
3
Y
3
3
7
7
7
73
73
E3D-LSTM
3/r
3
7
3
3
7
7
7
70
73
73
+redRNN+
3+
3
7
7
70
70
73
PredRNN
3/7
3 4
Y
Y
3
7
7
70
7
73
73
73
ConvLSSTM
3x
3
3
3
2,
4
4
7
5
2"
7
Figure 2: The qualitative results from different methods on the Moving MNIST dataset.
4
Experiments
4.1
Implementations
We evaluate the proposed MAU on five datasets, the Moving MNIST dataset [8], the KITTI dataset 31. the Caltech Pedestrian dataset [32, the TownCentreXVID dataset [33] and the Something- Something V2 dataset 34). The number of the hidden state channels of MAUs set to 64 and the integrated convolutional operators are set with a kernel size 5 x and stride 1. All experiments are optimized with the Adam optimizer. To stabilize the training process, employ layer normalization operators after each integrated convolutional layer in MAUs.
4.2
Video Prediction
We conduct video prediction experiments the the Moving MNIST dataset, the KITTI dataset, the Caltech Pedestrian dataset, and the TownCentreXVID dataset. The detailed experimental settings are summarized in Table D All models optimized with the MSE loss function.
Table 1: Experimental settings. MAUs denotes the number of the stacked MAUs. Train and Test denotes the number of frames as the inputs and the outputs while training and testing.
Dataset
Resolution
MAUs Hidden channels Kernel
Train
Test
T Y
Moving MNIST
1 x 64 x 64
4
64
5 5 5
10
+ 10
10 + 10
5 0,0
KITTI & Caltech
3 128 160
16
64
5x 5
10
10
10
10
1.0
TownCentreXVID x 1088 X 1920
16
64
5x5
4+1
4
4
4
1.0
4.2.1 Moving MNIST
The Moving MNIST dataset can the most widely-used dataset in video prediction, where each sequence consists of 20 successive frames with 2 digits randomly placed. Each frame is with size of 64 x 64. In our experiments, sequences are generated from the training set of the standard MNIST dataset (35) and utilize the test set collected by Srivastava al. g evaluate the proposed model. The Mean Square Error (MSH), the Structural Similarity Index (SSIM) are employed to indicate the visual quality of the predictions.
Fig. shows the visual results predicted from different methods, where the proposed MAU signifi- cantly outperforms other methods, especially for the prediction in the last two time steps. Table summarizes the quantitative scores of the predictions from different methods. The proposed MAU has achieved the best scores compared with other state-of-the-ar methods. And the employed recalling scheme can help further improve the model performance.
In addition, the parameters and inference time are also recorded in Table 3 For fair comparison, all models are implemented with the same encoder and decoder with the same number predictive units. In particular, for the proposed MAU, the recalling scheme is disabled and all models are trained

---

Table 2: Quantitative results of different methods on the Moving MNIST dataset (10 frames + 10 frames). Lower MSE and higher SSIM scores indicate better visual quality. The results of the compared methods are reported in [36].
Moving MNIST
Method
SSIM/frame
MSE/frame.
ConvLSTM (NeurIPS2015)
0.707
103.3
FRNN (ECCV2018) [12
0.819
68.4
VPN (ICML2017) [2
0.870
70.0
PredRNN (NcurIPS2017)
0.869
56.8
PredRNN+ (ICML,208))
14
0.898
46.5
MIM (CVPR2019) [13]
0.910
44.2
E3D-LSTM (ICLR2019)
16
0.910
41.3
CrevNet (ICLR2020)
[17
0.928
38.5
MAU (w/o recalling)
0.931
29.5
MAU
0.937
27.6
Table 3: Ablation study on the Moving MNIST dataset (10 frames - 10 frames). For fair comparison, the encoders and decoders with the same structure for all models and All models are trained using Adam optimizer based on the MSE loss.
Method
Backbone
MSEL
SSIM
Parameters
Inference time
ConvLSTM (NcurIPS2015)
9
4xConvLSTMs
102.1
0.747
0.98M
16.47s
ST-LSTM (NeurIPS2017)
[3
4xST-LSTMs
54.5
0.839
1.57M
17.74s
Casual-LSTM (ICML.201)
[4
4xCasual-LSTMM
46.3
0.899
1.80M
21.25s
MIM (CVPR2019)
15
4xMIMs
44.1
0.910
3.03M
45.13s
E3D-LSTM (ICLR2019)
16
4X3S-LSSMM
40.1
0.912
4.70M
57.21s
RPM (ICLR2020)
[7
4xRPMs
42.0
0.922
1.77M
18.01s
MotionGRU (CVPR2021)
28
4xMotionGRUs
34.3
0.928
1.16M
17.58s
MAU
4xMAUs
29.5
0.931
0.78M
17.34s
Table 4: Model performance MAU with different temporal receptive field т. In particular, 0, X = 0. The percentage values are calculated based the MAU with 1.
3
5
10
MSE/frame 33.4
32.2 (13.6%%
29.5 (111.7%)
29.3 (41123%) 20.23s (135.8%)
Inference time
14.90s
15.85s (16.4%)
17.36s (11555%
using Adam optimizer based on the MSE loss. We record the parameters for a single unit and the inference time is recorded over 800 samples. The summarized results show that the MAU can achieve the best scores with the fewest parameters and relatively low computation load.
Table shows the model performance with different temporal receptive field (w/o recalling). Although the quality of the predictions becomes better as the temporal receptive field increases, computation load also dynamically increases. Thus, is typically set to an appropriate value to achieve satisfactory trade-off between the visual quality and the computation load. In particular, we set 5 in this paper.
4.2.2 KITTI and Caltech Pedestrian datasets
We car-mounted camera video datasets to evaluate the performance of MAU in real scenarios. KITTI and Caltech Pedestrian datasets are very similar to the real-world scenarios, which are collected train the autonomous vehicles.
We follow the experimental settings in 37 where all frames are cropped and resized 128 160. The proposed model is trained on the KITTI dataset and tested on the Caltech Pedestrian dataset. In

---

particular, the frame rate of the Caltech Pedestrian dataset is adjusted to the same as KITTI (I0 fps). total of 32373 sequences are for training and 7725 sequences for testing. In addition, the proposed model is trained to predict the next frame with the first 10 frames as the inputs. While testing, the temporal period of the predictions is extended frames.
Inputs t=
Predictions
1=10
t=11
t=12
1=13
t=14
1=
t=
t=17
t=18
t=19
t=
STOP
STAP
MAU
STUP
CTTP
Jin et al. (CVPR2020)
CrevNet (ICLR2020)
STnC
Figure 3: The qualitative results from different methods the Caltech Pedestrian dataset.
Table 5: Quantitative results of different methods on the Caltech Pedestrian dataset. Lower MSE, LPIPS scores and higher SSIM, PSNR scores indicate better frame-level visual quality (10 frames + 1 frame). Lower FVD score indicates better sequence-level visual quality (10 frames - 10 frames). The results of the compared methods are partially reported in [36].
Caltech Pedestrian
Method
MSE(10 O)
SSIM PSNRT LPIPS(10 3) FVD/10 frames.
BeyondMSE (ICLR2016) [26]
3.42
0.847
MCnet (ICLR2017) 38
2.50
0.879
CtrlGen (CVPR2018)
0.900
26.5
PredNet (ICLR2017)
37
2.42
0.905
27.6
9,89
2860.8
ContextVP (ECCV2018)
40
1.94
0.921
28.7
9.53
2451.6
E3D-LSTM (ICLR2019)
16
2.12
0.914
28.1
10.02
2311.2
Kwon al. (CVPR2019)
24
1.61
0.919
29.2
8.03
1663.2
CrevNet (ICLR2020) 17
1.55
0.925
29.3
9.11
1709.6
Jin al. (CVPR2020) 27
1.59
0.927
29.1
8.99
1441.1
MAU (w/o recalling)
1.34
0.939
29.4
8.51
1269.9
MAU
1.24
0.943
30.1
8.04
1204.0
Fig.
shows the generated examples from the proposed MAU and two latest state-of-the-art methods. From the visual results, the proposed MAU can predict more clear traffic signs (t = 16), which indicates the MAU can capture quantitative results from different methods, where the proposed MAU outperforms other methods in all scores. In particular, the recalling scheme can further improve the model performance.
reliable temporal dynamics for videos. Table 5 shows the
4.2.3 TownCentreXVID
In this section, we evaluate the proposed model on a surveillance dataset, TownCentreXVID, which more close the real scenarios and with resolution of 1920 1080. TownCentreXVII dataset contains a total 7500 frames. To further evaluate the model performance in predicting high- resized. The first 4500 frames for training and the last 3500
resolution videos, all frames frames are for testing. Fig. showw the visual results from different methods, where the proposed method can generate much better visual details compared with other ofte-oof-the-r methods.
We further conduct object detection tasks on the predictions from different methods using the pre- trained Yolov5s model [1 and the results are shown in Fig. 5 More persons have been detected by the tranted Yolo model from the predictions generated from MAU and the confidence higher than others.

---

E3D-LSTM (ICLR2019)
CrevNet (ICLR2020)
MAU
Ground Truth
Figure 4: Qualitative results from different methods on the TownCentreXVID dataset (4 frames frame).
0.87
0.85
0.83
0.96
0.86
0.80
0 87
0.9
E3D-LSTM (ICLR2019)
CrevNet (ICLR2020)
MAU
Ground Truth
Figure 5: Object detection experiments the predictions (4 frames 1 frame) from different methods using Yolov5s pre-trained model [1 Confidence threshold set to 0.8.
Table 6: Quantitative results different methods on the TownCentreXVID dataset (4 frames > 4 frames). Higher SSIM and PSNR scores indicate better objective quality. Lower LPIPS score indicates better perceptual quality.
XownCentreXVID
Method
t 5
t 8
PSNR
SSIM
LPIPS(10 3)
PSNR1
SSIM
LPIPS(10 2)
ConvLSTM (NcurIPS2015)
9
27.22
0.894
39.90
23.29
0.876
46.12
PredRNN (NeurIPS2017)
[3
28.95
0.921
32.48
23.82
0.885
37.85
PredRNN+ (ICML2018)
[4
29.50
0.926
30.59
24.37
0.894
39.54
E3D-LSTM (ICLR2019)
[6
29.70
0.929
29.47
24.34
0.901
36.82
CrevNet (ICLR2020)
177
30.12
0.933
27.87
24.62
0.910
33.70
MAU (w/o recalling)
30.61
0.937
25.87
25.52
0.913
32.42
MAU
31.87
0.969
8.28
27.14
0.942
12.89
In addition, Table 6 summarized the detailed quantitative results from MAU and other methods, where MAU has achieved the best objective (PSNR, SSIM) and perceptual (LPIPS) scores.
4.3 Early action recognition: Something-Somethhng V2
The
dataset is a large collection of labeled video clips that show humans performing pre-defined basic actions with everyday objects. The whole dataset consists of 174 categories of videos. The training set contains 168,913 videos and the validation set consists of 24,777 videos. The resolution of each video is 240 427. To further evaluate the performance in modeling high-level spatiotemporal representations for videos, the early action recognition task conducted, which aims to categorize the whole video after observing only the front part of the videos. To accurately predict an activity category for current video, models need to extract useful spatiotemporal representations from merely front part of the whole frames to predict a reliable future. In particular, we utilize the front 25% and 50% frames of each video conduct this task, respectively. For each video, total of 20 frames are sampled which can cover the whole temporal period. Each frame is resized to 128 128. For 25% carly action recognition task, 5 frames are as

---

the inputs to predict the next 15 frames. For 50% early action recognition task, 10 frames are as the inputs predict the next 10 frames.
All models are first pre-trained to perform video prediction task with the front 25% or 50% frames the inputs to predict the remaining 75% 50% frames. After convergence, total of 19 hidden states extracted from the pre-trained models are concatenated to train the classifier, which consists of 3 3D convolutional layers. In particular, the concatenated hidden states are transformed from X H x W 64 X 19 x 16 x 16 to 174 x 1. To evaluate the performance of different predictive units in modeling spatiotemporal representations for videos, we utillze various predictive units as the backbone of the early action recognition model. For fair comparison, the encoder, decoder and classifier are set with the same structure for all models. A total of 16 predictive units are stacked for all models. The experimental results are summarized in Table 7 On the one hand, the predictive model with MAU has achieved the best PSNR score the predictions. On the other hand, based on the learned spatiotemporal representations from the pre-trained models, the proposed MAU can also obtain the highest Topl and Top5 accuracies.
Table 7: The results of the early action recognition experiment of different methods on the Something- Something V2 dataset.
SemettinggSommthhinggW V2
Front 25%
Front 50%
Method
PSNR
top-11 11
top-5t
PSNR
top-11
top-5t
ST-LSTM (NeurIPS2017)
[3
14.87
3.77
14.17
15.78
8.91
19.18
Casual-LSTM (ICML2018)
14
15.34
4.14
14.67
16.51
9.57
22.57
E3D-LSTM (ICLR2019) [16]
16.21
4.76
14.98
17.05
9.85
24.24
RPM (ICLR2020) 17
16.53
4.98
15.07
17.57
10.01
24.51
MotionGRU (CVPR2021)
28
17.01
5.11
15.16
17.86
10.22
27.65
MAU
17.36
5.40
19.00
18.47
10.60
30.90
Conclusion
proposed the Motion-aware Unit (MAU) for video prediction and beyond. The motion-aware unit can take advantage in the broadened temporal receptive field, where more temporal states can be simultaneously perceived. In particular, the proposed unit are constructed based on the attention mechanism, which consists of two modules, the attention module and the fusion module. In particular, the attention module can extract attention weights from multi-term spatial states for multi-tern temporal states. And the augmented motion information (AMI) can be aggregated from the multi-term temporal states. Then the fusion module utilized to further aggregate the AMI and the spatial information predict the final video frame. Although more states have been aggregated, the computation load is still relatively low because of the efficient structure in MAU. Moreover, an information recalling scheme employed to help the decoders recall more multi-level encoded information. The proposed model was evaluated on two tasks, the video prediction task and the video classification task. Experimental results showed the proposed MAU outperform other state-of-the-at methods on both tasks.
Limitations and Future Work
There are lots of issues that urgently need to be solved in the video prediction area and this work is also far from completely solving all the problems, such as high-resolution video prediction, low accuracy action recognition, not practical enough to be applied into other downstream tasks and so on. All these problems need to be carefully taken into consideration in our future work. In addition, videos are still predicted deterministic way rather than a stochastic way, which may not practical enough for the complex real-world scenarios. Thus, in our future work, we will explore more to combine the spatiotemporal dynamics videos into stochastic models, which are more similar to real-world scenarios.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China (62072008. 62025101), PKU-Baidu Fund (2019BD003) and High-performance Computing Platform of Peking University, which are gratefully acknowledged.
References
[1] Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In International Conference on Neural Information Processing Systems, pages 64-72, 2016.
[2] Ziwei Liu. Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. In International Conference Computer Vision, pages 4463 4471, 2017.
[3] Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Long-term on-board prediction of people in traffic scenes under uncertainty. In IEEE Conference on Computer Vision and Patter Recognition, pages 4202 2018.
[4] Hema S Koppula and Ashutosh Saxena. Anticipating human activities using object affordances for reactive robotic response. IEEE transactions on pattern analysis and machine intelligence, 38(1):14-29, 2015.
[5] MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathicu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint 14Xiv:1412.000 2014.
[6] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98)):1735-1780, 1997.
[7] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing, 2014.
[8] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representa- tions using Istms. In International conference on machine learning, pages 843-852, 2015.
[9] Xingjian Shi. Zhourong Chen, Hao Wang, Dit Yan Yeung. Wai Kin Wong, and Wangchun Woo. Convo- lutional 1stm network: A machine learning approach for precipitation nowcasting. Advances in Neural Information Processing Systems, 2015.
[10] Nicolas Ballas, Li Yao, Chris Pal, and Aaron C Courville. Delving deeper into convolutional networks for learning video representations. In International Conference on Learning Representations, 2016.
[11] Xingjian Shi, Zhihan Gao, Leonard Lausen, Hao Wang, Dan Yeung, Wai-Kin Wong, and Wang-chun Woo, Deep learning for precipitation nowcasting: A benchmark and a new model. In International Conference on Neurul Information Processing Systems, 2017.
[12] Mare Oliu, Javier Selva, and Sergio Escalera. Folded recurrent neural networks for future video prediction. In European Conference Computer Vision, pages 716-731, 2018.
[13] Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S Yu. Predrnn: Recurrent neural networks for predictive lcarning using spatiotemporal Istms. In Advances in Neural Information Processing Systems, pages 879-888, 2017.
[14] Yunbo Wang. Zhifeng Gao, Mingsheng Long, Jianmin Wang, and S Yu Philip. Predrau++: Towards resolution of the deep-in-time dilemma spatiotemporal predictive learning. International Conference on Machine Learning, pages 5123-5132, 2018.
[15] Yunbo Wang, Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, and Philip Yu. Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. In IEEE Conference on Computer Vision and Pattern Recognition, pages 9154-9162, 2019.
[16] Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d Istm: A model for video prediction and beyond. In International conference on learning representations, 2019.
[17] Wei Yu, Yichao Lu, Steve Easterbrook, and Sanja Fidler. Efficient and information preserviineegggg future frame prediction and beyond. In International Conference on Learning Representations, 2020.
11

---

[18] Mohammad Babaeizadeh, Chelsea Finn. Dumitru Erhan, Roy H Campbell. and Sergey Levine. Stochastic variational video prediction. In International Conference on Learning Representations, 2018.
[19] Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In International Conference on Machine Learning, pages 1174-1183, 2018.
[20] Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoe V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In Advances in Neural Information Processing Systems, 2019.
[21] aves Franceschi. Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochas- tic latent residual video prediction. International Conference on Machine Learning, pages 3233-3246. PMLR, 2020.
[22] Alex X Lee, Richard Zhang, Frederik Ehert, Pieter Abbeel, Chelsea Finn, and Sergey Levine, Stochastic adversarial video prediction. arXiv preprint (rX.vv880400152: 2018.
[23] Zhihang Hu and Jason Wang. novel adversarial inference framework for video prediction with action control. In IEEE International Conference on Computer Vision Workshops, pages 0-0, 2019.
[24] Yong-Hoon Kwon and Min-Gyu Park. Predicting future frames using retrospective eycle gan. In IEFE Conference on Computer Vision and Pattern Recognition, pages 181 1811-1820, 2019.
[25] Xinyuan Chen, Chang Xu, Xiaokang Yang, and Dacheng Tao. Long-term video prediction via criticization and retrospection. IEEE Transactions on Image Processing, 29:7090-7103, 2020.
[26] Michael Mathicu, Camille Couprie, and Yann LeCun, Deep multi-scale video prediction beyond mean square crror. In International Conference on Learning Representations, 2016.
[27] Beihei Jin, Hu, Qiankun Tang, Jingyu Niu, Zhiping Shi, Yinhe Han, and Xiaowei Li. Exploring spatial-temporal multi-frequency analysis for high-fidelity and temporal-consistenge video prediction. In IEEE Conference on Computer Vision and Pattern Recognàtion, pages 4554-4563, 2020.
[28] Haixu Wu, Zhiyu Yao, Mingsheng Long, and Jianmin Wan. Motionmn: A flexible model for vidco prediction with spacetime-varying motions. arXiv preprint arX0v:2103.02243 2021.
[29] Nal Kalchhrenner, Aäron Oord, Karen Simonyan, Tvo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In International Conference on Machine Learning, pages 1771-1779, 2017.
[30] Bohan Wu, Suraj Nair, Roherto Martin-Martin Li Fei-Fei, and Chelsea Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. arXiv preprint 10X.v:2103.04: 041144 2021.
[31] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):131-1133 2013.
[32] Piotr Dollar, Christian Wojek. Bernt Schiele, and Pietro Perona. Pedestrian detection: An evaluation of the state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(4):743-761, 2011.
[33] Ben Benfold and lan Reid. Stable multi- target tracking in real-time surveillance video. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3457-3464, 2011.
[34] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Hacnel, Ingo Fruend, Peter Yianilos, Moritz Mucller-Freitag, al. The" something something" video database for learning and evaluating visual common sense. International Conference on Computer Vision, pages 1842-5850, 2017.
[35] Yann LeCun. The mnist database of handwritten digits. http://yam. lecun. com/exdb/nmist/ 1998.
Alberto Garcia-Garch John Alejandro Castro- Varga,, Sergio [36] Sergiu Oprea, Pablo Rodriguez, and Antonis Argyros. A review on deep learning techniques for EscoEssoonn, Jose Garcia-Rodrighe video prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[37] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised lcarning. 2017.
[38] Ruben Villegas, Jimei Yang. Seunghoon Hong. Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In International Conference on Learning Representations. 2017.
12

---

[39] Zekun Hao, Xun Huang, and Serge Belongie. Controllable video generation with sparse trajectories. In IEEE Conference Computer Vision and Potiern Recognition, pages 7854-7863, 2018.
[40] Wonmin Byeon, Qin Wang, Rupesh Kumar Srivastava, and Petros Koumoutsakos. Contextvp: Fully context-aware video prediction. In European Conference Computer Vision (ECCV), pages 753-769, 2018.
[41] Glenn Jocher, Alex Stoken, Jirka Borovec, NanoCode012, Ayush Chaurasia, TaoXie, Liu Changyu, Abhiram Laughing. tkianai, yxNONG, Adam Hogan, lorenzomammana, AlexWang 1900, Jan Hajek, Laurentiu Diaconu, Mare, Yonghye Kwon, oleg, wanghaoyang0106, Yann Defretin, Aditya Lohia, ml5ah, Ben Milanko, Benjamin Fineran, Daniel Khromov, Ding Yiwei, Doug. Durgesh, and Francisco Ingham. ultralyties/yolov5: v5.0 YOLOv5-P6 1280 models, AWS, Supervise.ly and YuuTube integrations, April 2021.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes) See Section [
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
3
(b) Did you include complete proofs of all theoretical results? [Yes]] See Section
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes] See the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Ys]] See section 4
(c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]] See section
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're re us-
ing/eurating? [N/A]
(e) Did discuss whether the data you using/curating contains personally identifiable informa- tion offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
[N/A]
(b) Did you descrihe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "wa3tdWik88A", "reviewer": "Reviewer_LrcY", "summary": "The paper proposes the Motion-Aware Unit (MAU) for video prediction. This module attempts to model the temporal information in videos by enlarging the temporal receptive field and aggregating it through an attention mechanism. The proposed module is employed in the bottleneck part of an autoencoder where the spatial and temporal information are combined at different MAU layers. The proposed method is compared to previous works on several benchmarks for next frames prediction and early action recognition showing state-of-the-art results.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method shows interesting results on both the video prediction and early action recognition tasks. Although the idea of enlarging the temporal receptive field when dealing with sequence forecasting is not new [1], in the video prediction task, this idea is not already investigated. However, the method introduced is incremental as the main contribution stands on the aggregation of spatial and temporal features using an attention-based mechanism.\n\nFigure 1 on the right is barely comprehensible; I suggest making it clearer. In line 110, the L1 and L2 terms are not introduced before; it would be better to state what they are clearly.\n\nMany important details regarding the model and the experiments are missing.\n- Which encoder and decoder backbones do you use for the experiments?\n- Are the encoder and the decoder comparable with respect to previous works in the results reported in the Caltech Pedestrian, KITTI, and TownCentreXVID datasets?\n- What is the dimensionality of the internal features of the model?\n- Does such dimensionality change in the different benchmarks you used?\n- How does such dimensionality impact the efficiency of the model?\n- Is your architecture fully convolutional?\n\nLots of details for reproducing the results are also missing (training parameters, ...).\n\nIt would be useful to visualize the attention scores of the MAU module in order to better understand the behavior of the proposed model when it generates new frames.\n\nReferences:\n\n[1] Attention is All you Need, Vaswani et al, NIPS 2017.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not highlight any limitations of the proposed method. Moreover, I do not find any potential negative societal impact regarding this work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "wPpLwikO94q", "reviewer": "Reviewer_Bgyd", "summary": "In this work, the authors introduce a Motion-Aware Unit (MAU) for video prediction. Basically, they discover the previous motion and aggregate it via attention module and fusion module. In the attention module, they compute correlation between spatial states and use it as attention for aggregation the previous temporal states. In the fusion module, they design spatial and temporal update gates for fusing attentive temporal state and current spatial state.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality: The novelty is relatively limited. Attentively discovering historical motion has been widely investigated in online action recognition [Skeleton-Based Online Action Prediction Using Scale Selection Network, TPAMI2020] and detection [Learning to Discriminate Information for Online Action Detection, CVPR2020]\n2. Quality: The motivation, model design and experiments basically make sense.\n3. Clarity: It is an OK paper for me. \n4. Significance: It is probably useful for online action recognition and detection, etc. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to Main Review for the detailed comments.\n1 Novelty is limited. The design is not quite new, based on the fact that attention for motion learning has been widely used in video understanding.\n2 By the way, temporal shift module [TSM: Temporal Shift Module for Efficient Video Understanding, ICCV2019] is a popular mechanism for early action recognition. It would be interesting to see how it works in Table 7.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "PtHp6LL9Nwo", "reviewer": "Reviewer_k1Ye", "summary": "This paper proposes a Motion-Aware Unit (MAU) for video prediction tasks. The MAU includes two modules, the attention module and the fusion module. The attention module aims to learn the correlation between the current spatial state and previous spatial states. The learnt attention map can be used for augmenting the motion information (AMI). The fusion module is used for aggregating the augmented motion information (AMI) and current spatial state. The authors have demonstrated superior performance compared to state-of-the-arts.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Apart from the good results presented in the paper, I have some questions about the paper as below:\n1) P2 L67, the authors mentioned the good performance on the early action recognition task, but I have not found early action related experiment presented in the paper.\n2) P4 L147, the authors clarify that they aim to quantify the correlation between different temporal states. But for **2nd line in eq.3**, how the Hadamard Product operation between two spatial states is able to quantify the correlation between matrices? \n3) Do SUM and MUL in Figure 1 represent summation and multiplication? Needs an explanation here in the caption and main text.\n4)  **2nd and 3rd lines in eq.3**, I suppose the objective of the 2nd line is to calculate the correlation between one state $t$ and all previous temporal states in window $\\tau$ based on this state. If so, the subscription of $q_i$ is confused, it should be $q_t$.\n5) In the experiment part, are backbones in encoder and decoder the same as a backbone in previous approaches? This needs description in the experiment part e.g., what backbones are used in encoder and decoder as well as for other approaches.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not stated negative parts. Current video prediction tasks are still low resolution (max $128\\times 128$), which might not be suitable for real-life applications. Future work can be extended based on high resolution.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "230IkHwcs3x", "reviewer": "Reviewer_tpAZ", "summary": "This paper proposes a method for video prediction that relies on a newly proposed Motion-Aware Unit (MAU). MAU acts as a temporal receptive field based on an attention mechanism that is used to aggregate features from previous frames in order to predict future frames. The aggregate features are combined with “content” features extracted from the last observed frame and the decoder is connected to the encoder via an “Information recalling scheme” that directly passes the last observed frame features from each layer of the encoder to each layer of the decoder. In experiments, the authors show superior performance against the baselines pixel based metrics and a perceptual metric.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ A Motion-Aware Unit (MAU) that enables direct retrieval of previous frames extracted features using attention.\n+ Combination layers that aggregate motion and content information by choosing important information from each information stream.\n+ Outperforms baselines\n\nWeaknesses:\n\n- Evaluation (Table5 and Table 6):\nAs far as I can tell, the experimental setup is described in Table 1. However, when I read the captions on top of Table 5 and 6, I see that the authors describe something different. They indicate that for the Caltech and TownCentreXVID experiments their test setups are (10->1) and (4->1), respectively. This is completely different to what Table1 says. The test setups in Table 1 are (10 -> 10) and (10->10). If we assume that they mean train setup, it is the same story. The train setups in Table 1 are (10->1) and (10->10). Can the authors clarify this? I would also suggest the authors include a sequence level evaluation metric such as Fretchet Video Distance. Frame level evaluations do not usually tell the full story video prediction since we need to also measure if the dynamics are also plausible. FVD repo: https://github.com/google-research/google-research/blob/master/frechet_video_distance/frechet_video_distance.py. Finally, the use of evaluation metrics are not consistent throughout the experiments. I understand that perceptual evaluations such as LPIPs may not make sense for the Bouncing MNIST experiments, but they are applicable to both Caltech and TownCentreXVID experiments. However, I only see perceptual evaluations on the latter. Can the authors explain why this is the case?\n\n- Ablations:\nThe authors have provided ablations of \\tau for the number of frames considered in the temporal receptive field. However, there are other pieces in the architecture/formulation that would be good to see how much they contribute to performance. For example, what happens in Equation 5 if we simply let T_{AMI} = T_{attn} or T_{AMI} = T^k_{t-1}? T_{attn}. Another place where an ablation is necessary is  Equation 7. What if we let S^k_t = W_{ts} * T_{AMI} + \\gamma S^k-1_t? Or can we visualize U_s to see if the network simply decided to set U_s=0 during training? This would be informative to decide whether the current complexity is necessary or not.\n\n- TownCentreXVID experiments:\nTable 6 shows that the proposed method outperforms the baselines. However, it is very difficult to tell from Figure 4 whether the proposed method is simply copying the last observed frame or making an actual prediction of the future. From very closely inspecting MAU against the groundtruth, it looks like what MAU outputs is what seems to be the frame before the Grouthtruth. It has what seems to be a temporal progression. However, as I said before, it’s hard to tell without previous image context. In addition, the folder in the supplementary material where visual results should be for this dataset is empty. The author’s should have provided a baseline where the last observed frame is copied and evaluate it to check the numbers against the proposed method and make sure it’s not doing the same thing.\n\n- Video results:\nPapers on video prediction usually provide video results since it’s very difficult to tell the quality of the prediction from frames on a pdf. This would have clarified a lot of the concerns mentioned above.\n\n\n- Limitations of this work?\nThe authors have not mentioned any limitations of this work, and also marked N/A in the limitations question of the checklist. Why are there no limitations? Did the proposed method completely solve the problem of video prediction? Does the method learn perfect features for the action recognition/anticipation task or other sequence downstream tasks? From reading the paper, I see there are a lot of limitations, some of which we should be able to see if we have videos to observe. Also the action recognition/anticipation experiments (quality of learned features verification) accuracy is really low. In the future, the authors should have a paragraph or two describing what is failing in their method and what needs to be done next so that other researchers can pick up from there or get inspired by the work. That’s what academia is all about. \n\nSuggestions:\n- Action recognition experiments against recognition methods: \nThe provided action recognition experiments are great to show that the proposed method learns something useful from predicting future frames. I also suggest the authors train a network to do action recognition from scratch and compare against it. It would give readers a good idea about the gap between self-supervised and fully-supervised feature learning for this task.\n\n- Action recognition without “recalling” in the prediction training:\nThe “recalling” connections from encoder to decoder are basically skip connections which have been used by previous works to get better frame prediction and make learning easier. However, these skip connections could have a negative impact on feature learning since the network can just copy a large portion of the pixels in the input image. I wonder if the action recognition performance improves if the recalling connections are removed because the features have to represent the entire image during prediction.\n\nConclusion:\nI really like the formulation of this method as it is simple and seems that it should work well. However, the experimental section has a lot of issues that I cannot look past because I like the method. Because of this, I am at the borderline/leaning towards rejection of this paper. I am looking forward to the author's response to hopefully address my doubts/concerns.\n\n\n#####################################\n##### Post rebuttal comments ############\n#####################################\n\nThe authors addressed all my concerns. Therefore, I have decided to increase my score to accept.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They have not addressed their limitations or societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was a robust discussion between the reviewers on the merits of this work. The author response helped clarify many of the things that the reviewers asked for. All in all, I feel this work should be accepted at NeurIPS. While the reviewers as a whole felt that there is some lack of novelty in the work, and had some minor constructive feedback (e.g. more justification for not performing multi-step prediction in their TownCentreXVID experiments), the conclusion is that the contributions are sufficient and interesting enough to warrant acceptance. 
Decision: Accept (Poster)