Efficient and Information-Preserving Future Frame Prediction and Beyond
Authors: Wei Yu
Yichao Lu
Steve Easterbrook
Sanja Fidler
Venue:  (2020)
Keywords: self-supervised learning
generative pre-training
video prediction
reversible architecture

Abstract: Applying resolution-preserving blocks is a common practice to maximize information preservation in video prediction, yet their high memory consumption greatly limits their application scenarios. We propose CrevNet, a Conditionally Reversible Network that uses reversible architectures to build a bijective two-way autoencoder and its complementary recurrent predictor. Our model enjoys the theoretically guaranteed property of no information loss during the feature extraction, much lower memory consumption and computational efficiency. The lightweight nature of our model enables us to incorporate 3D convolutions without concern of memory bottleneck, enhancing the model's ability to capture both short-term and long-term temporal dependencies. Our proposed approach achieves state-of-the-art results on Moving MNIST, Traffic4cast and KITTI datasets. We further demonstrate the transferability of our self-supervised learning method by exploiting its learnt features for object detection on KITTI. Our competitive results indicate the potential of using CrevNet as a generative pre-training strategy to guide downstream tasks.

Published as a conference paper at ICLR 2020
EFFICIENT AND INFORMATION-PRRSSERVVNN FUTURE FRAME PREDICTION AND BEYOND
Wei Yu', Yichao Lu', Steve Easterbrook! Sanja Fidler
DDppartment of Computer Science, University of Toronto Vector Institute, Canada
NVIIDA
(gnosis, yi yichao, sme, fidler www.stooooto.eoe
ABSTRACT
Applying resolution- pereriovingee blocks is common practice to maximize informa- tion preservation in video prediction, yet their high memory consumption greatly limits their application scenarios. We propose CrevNet, a Conditionally Reversible Network that uses reversible architectures to build bijective two-way autoencoder and its complementary recurrent predictor. Our model enjoys the theoretically guaranteed property of no information loss during the feature extraction, much lower memory consumption and computational efficiency. The lightweight nature of our model enables to incorporate 3D convolutions without concern of memory bottleneck, enhancing the model's ability to capture both short-term and long-term temporal dependencies. Our proposed approach achieves state-of-the-arr results on Moving MNIST, Traffic4cast and KITTI datasets. We further demonstrate the transferability of our self-supervised learning method by exploiting its learnt fea- tures for object detection on KITTI. Our competitive results indicate the potential of using CrevNet as a generative pre-training strategy to guide downstream tasks.
INTRODUCTION
Deep learning has enjoyed tremendous success in recent years due its ability to capture complex dependencies and non-lincarities in large datasets (Krizhevsky et al. (2012): He et al. 20 /omez al. (2017). ) Excellent performance has been achieved on a wide range of supervised machine learning tasks, ranging from image classification (He et al. (2016)) and object detection (Ren et al. (2015) to speech recognition (Amodei et al.] (2016)). Despite the significant breakthrough in supervised learning, the potential of applying deep architectures to unsupervised learning problems remains largely unexplored. Lately there has been surge of interest the task of video prediction, ie.. predict future frames of video sequence ( (Wang et al. 2017 2018): ; Denton et al. (0177); Denton & Fergus 2018); Villegas et al. (2017); Lee al. (2018). The significance of video prediction primarily lies in its potential of discovering dynamics in the physical world. The self-supervised nature of video prediction aligns well with how humans learn, without requiring large amounts of labeled data. addition, videos can provide an abundant and virtually unlimited source of visual information. This allows video prediction models to serve as a generative pre-training strategy of feature representation learning for a variety of downstream supervised tasks.
To date, most of the existing models for video prediction employ a hybrid of convolutional and recurrent layers as the underlying architecture (Wang et al. (2017); (Shi al. (2015); Lotter al.) (2016)). Such architectural design enables the model to simultancously exploit the ability of convo- lutional units to model spatial relationships and the potential of recurrent units to capture temporal dependencies. Despite their prevalence in the literature, classical video prediction architectures suffer from two major limitations. Firstly, dense prediction tasks such as video prediction, models required to make pixel-wise predictions, which emphasizes the demand for the preservation of information through layers. Prior works attempt to address such demand through the extensive use of resolution- eesolution-preserving preserving blocks (Wang et al. (2017) 2018); Kalchbrenner et al. (2016)). Nevertheless, these resolution resolution-preserving blocks are not guaranteed to preserve all the relevant information, and they greatly increase the memory consumption and computational cost of the models. The second

---

Published as a conference paper at ICLR 2020
Input/Output
Two-way autoencoder
Predictor
Encoding
Decoding
Recurrent connection
Warmup
Next-frame Generation
Multi-frame Prediction
The pipeline of our proposed CrevNet where a single two-way autoencoder both encoder and decoder. CrevNet first observes a warm-up video sequence and then starts multi- fallt--rrame video prediction without refeeding its own predictions.
Figure I:
drawback of existing video prediction models is that they cannot efficiently take advantage of 3D convolutions, as that would make these already cumbersome architectures even larger. 3D convolu- tions have been shown to be very effective alternative RNNs capture temporal relations in
variety of video tasks (Liu et al. (2018): Carreira & Zisserman (2017), and thus desirable to exploit.
Recently, reversible architectures (Dinh al. 2014); Gomez al. (2077): Jacobsen al. (2018)
have attracted attention due to their light memory demand and their information preserving property by design. However, the effectiveness of reversible models remains greatly unexplored in the video literature. In this paper, we introduce novel, conditionally reversible video prediction model, CrevNet, in the sense that when conditioned on previous hidden states, can exactly reconstruct the input from its predictions. The contribution of this work can be summarized as follows:
We introduce two-way autoencoder that uses the forward and backward passes of invert- ible network as encoder and decoder (Fig [. The volume-preserrvng two-way autoencoder not only greatly reduces the memory demand and computational cost, but also enjoys the theoretically guaranteed property of no information loss. The lightweight nature of our model enables us incorporate 3D convolutions without concern of memory bottleneck.
We propose the reversible predictive module (RPM), as illustrated in Fig 2b, which extends the reversibility from spatial temporal domain. RPM, together with two-way autoen- coder, provides a conditionally reversible architecture (CrevNet) for spatiotemporal learning. CrevNet achieves the statc-of-the-art results on Moving MNIST, Traffic4cast and KITTI.
We evaluate the effectiveness of features learnt from self seff-supervision by adapting CrevNet for object detection on KITTI. Our competitive results indicate the potential of using CrevNet generative pre-training strategy guide downstream CV tasks.
2
APPROACH
We first outline the general pipeline our method, Our CrevNet consists of two subnetworks, an autonencoder network with an encoder &, decoder D and recurrent predictor P bridging encoder and decoder. Let 2, E "xx represent the fnh frame in video æ, where h, and denote its width, height, and the number of channels. Given F0:-11, the model predicts the next frame â as follows:
ft- D(P((((:-1)))o==--)
(1)
In the case of 3D convolution, 24 E xxxwxxxx denotes the short video clip from to tot + & - instead of single frame at timestep t, where & is the temporal dimension of input output. During the multi-frant generation process without access to the ground truth frames, the model uses its previous predictions instead.
2.1 THE INVERTIBLE TWO-WAY AUTOENCODER
We propose a bijective two-way autoencoder based on the additive coupling layer introduced in NICE (Dinh al. 2014)). We begin with describing the building block of the two-way autoencoder
2

---

Published as a conference paper at ICLR 2020
Two-way autoencoder
c. Inverse computation
Feature map
f CNN
ReshupetSplit
Weighted sum
Add
Minus
Forward
Input
ConYRNN
Pixel shuffle
Attention
b. Reversible predictive module
Figure 2: The network architecture of CrevNet (Better viewed in color). The input video frames are first reshaped and split channelwise into two groups. These groups passed to the two-way autoencoder (a) for feature extraction, and then to the predictor made up of multiple reversible predictive modules (b). The transformed high-level features produced by predictor then passed back through the decoding pass of (a), shown here as representative block (c) to yield its prediction.
Fig 1). Formally, the input is first reshaped and split channelwise into two groups, denoted as x! and T During the forward pass of each building block, one group, e.g. x', passes through several convolutions and activations and then added to another group, x², like a residual block:
2² 2 (i
a -'= F2
(2)
where F is composite non-linear operator consisting of convolutions and activations, and f and d² are the updated x and x²2 Note that x and x² can be simply recovered from x² and 4 by the
inverse computation (Fig 2 follows:
x² - -'
x 2-
(3)
Multiple building blocks stacked in alternating fashion between x and x²² to construct two-way autoencoder, as shown in Fig A series of the forward and inverse computations builds one-to-one and onto, i.c. bijective mapping between the input and features. Such invertibility ensures that there no information loss during the feature extraction, which is presumably more favorable for video prediction since the model is expected to restore the future frames with fine- grained details. To enable the invertibility of the entire autoencoder, our two-way autoencoder uses a bijective downsampling, pixel shuffle layer (Shi et al. (2016), that changes the shape of feature from (w, h, c) to (w/n, h /,, x n²). The resulting volume- preserving architecture can greatly reduce its memory consumption compared with the existing resolution resolution-preserving methods.
We further argue that for generative tasks, e.g. video prediction, we can effectively utilize a single two-way autoencoder, and to use its forward and backward pass as the encoder and the decoder, respectively. The predicted frame i, is thus given by
i = E E (P(((()))))
(4)
where E-1 is the backward pass of E. Our rationale is that, such setting would not only reduce the number of parameters in the model, but also encourage the model to explore the shared feature space between the inputs and the targets. As result, our method does not require any form of information sharing, e.g. skip connection, between the encoder and decoder. In addition, our two-way autoencoder can enjoy lower computational cost at the multi-fram prediction phase where the encoding pass is no longer needed and the predictor directly takes the output from previous timestep as input, as shown in Fig 1 since E(E-1) an identity mapping.
2.2 REVERSIBLE PREDICTIVE MODULE
In this section, describe the second part of our video prediction model, the predictor P, which computes dependencies along both the space and time dimensions. Although the traditional stacked- ConvRNN layers architecture the most straightforward choice of predictor, find that it fails to
3

---

Published as a conference paper at ICLR 2020
establish consistent temporal dependency when equipped with our two-way autoencoder through experiments. Therefore, we propose a novel reversible predictive module (RPM), which can be regarded recurrent extension of the two-way autoencoder. In the RPM, we substitute all standard convolutions with layers from the Conv family (e.g. ConvL.TVV or spatiotemporal LSTM) and introduce soft attention (weighting gates) mechanism to form weighted sum of the two groups instead of the direct addition. The main operations of RPM used in this paper are given as follows:
h = ConvRNN 11-1)) = W ReLU(W, * 11 bi) + b2)
ComRNN
Attention module
27 = (1 - gt) o of + g: 3 h
Weighted sum
where z and z² denote two groups of features at timestep t. hy denote the hidden states of ConvRNN layer, ó is sigmoid activation, the standard convolution operator and is the Hadamard product. The architecture of reversible predictive module is also shown in Fig 2p. RPM adopts similar architectural design as the wooway autoencoder to ensure pixel-wise alignment between the input and the output, i.e. each position of features can be traced back to certain pixel, and thus make it compatible with our two-way autoencoder. It also mitigates the vanishing gradient issues across stacked layers since the coupling layer provides nice property w.r.t. the Jacobian (Dinh et (2014). In addition, the attention mechanism in the RPM enables the model to focus on objects in motion instead of background, which further improves the video prediction quality. Similarly, multiple RPMs alternate between the two groups to form a predictor. We call this predictor conditionally reversible since, given h4-1, are able to reconstruct - from d; if there are numerical errors:
---1 = E E (E (((1))
where P-1 is the inverse computation of the predictor P. We name the video prediction model using two-way autoencoder as its backbone and RPMs as its predictor CrevNet. Another key factor of RPM is the choice of ConvRNN. In this paper, we mainly employ ConvLSTM (Shi et al. (2015) and spatiotemporal LSTM (ST-LSTM, Wang et al. (217 to enable a fair comparison with baselines.
2.3 3D CONVOLUTIONS
3D convolutions are proposed to address the shortcomings of standard 2D convolutions. The major difference between 2D-CNNs and 3D-CNNs is that at each time step 2D-CNNs take as input one video frame, while 3D-CNNs read in and output short video clip containing & continuous video frames. By applying convolutions on the temporal dimension along with the spatial dimension, models equipped with 3D convolution filters can not only extract representative spatiotemporal features, but also learn to produce consistent video clip at each generation, which further improve the quality of long-term prediction. In some cases, c.g. sequences are too short, we will use consecutive frames stacked in the channel dimension instead as input at each timestep to assemble valid warm-up up sequence for ConyRNN.
3 EXPERIMENTS
3.1 LONG-TERM PREDICTION MOVING MNIST
Moving MNIST (Srivastava et al. (213)) is synthetically generated dataset that contains an infinite number of sequences of length 20. Each sequence shows how 2 digits move at constant speed and bounce inside 64 64 frame, where each handwritten digit randomly sampled from the MNIST dataset. By assigning different initial locations and velocities to each digit, it is possible to generate an unlimited number of sequences, thus enabling us to accurately evaluate the performance of each model without the concern of data insufficiency issues. In the default setting. models are trained to predict 10 future frames after observing 10 prior frames in the sequence. Although the dynamics of Moving MNIST seems to be simple at first glance. It is quite hard to generate consistent future frames in the task of long-term prediction as digits can bounce or occlude each other frequently.
Datasets and Setup: The general architecture of CrevNet used Moving MNIST is composed of 36-layer two-way autoencoder and 8 RPMs. All variants of CrevNet are trained by using the Adam optimizer with starting learning rate of 5 x 10-1 minimize MSE The training process is stopped
4

---

Published as a conference paper at ICLR 2020
after 300, 000 iterations with the batch size of 16 and evaluated with fixed test set containing 5, 000 sequences. To ensure that all samples in the test set are unseen by the model, digits in the training set and the testing set are separately sampled from two mutually exclusive subsets of MNIST.
MNIST-2
MNIST-3
Memory (1 sample)
FLOPS (1 frame)
Model
SSIM
MSE
Human
SSIM
MSE
ConvLSTM (Shi al.
(2015))
0.707
103.3
0.923
0.695
127.3
1043 MB
107.4 G
FRNN (Oliu al. 201
0.819
68.4
0.848
0.791
90.4
717 MB
80.1 G
VPN (Kalchbrenner el al.
2016))
0.870
70.0
0.831
0.820
85.6
5206 MB
309.6 G
PredRNN Wang et al. (017
0.869
56.8
0.837
0.822
83.1
1666 MB
192.9 G
PredRNN+ Wang et al.
(018))
0.898
46.5
0.781
0.864
68.4
2017 MB
106.8 G
E3D-LSTM Waag
2019
0.910
41.3
0.706
0.870
62.4
2695 MB
381.3 G
CrevNet + ConvL.SII
0.928
38.5
0.602
0.886
57.2
133 MB
0.919 G
CrevNet + ST-LSTM
0.949
22.3
0.558
0.916
40.6
195 MB
1.618 G
Table 1: Quantitative evaluation of different methods Moving MNIST. All metrics are averaged over the 10 predictions. Lower MSE and higher SSIM indicates better prediction accuracy.
We compare CrevNet to six popular benchmark models from the literature: (i) ConvI. (Shi et al. (2015). ). (ii) FRNN (Oliu et al. (0118). (iii) VPN (Kalchbrenner et (2016). (iv) PredRNN (Wang) al. 2017 (v) PredRNN++ (Wang al. (2018)) and (vi) E3D-LSTM (Wang et al. (2019) ).. All baselines are implemented and optimized by following their corresponding protocols. To test our model in a more challenging setting, we also extend Moving MNIST a 3-digit version where digits are more likely to occlude each other.
The performance of each model in terms of per-frame MSE and the Structural Similarity Index Measure (SSIM) (Wang et al. (2004) is presented in Table CrevNet outperforms all previous methods by a wide margin on both metrics while memory consumption of all Crev Net variants is significantly lower than that of other baselines. In particular, CrevNet with ConvLSTM only uses 130 MB memory per sample and is still capable of yielding results better than any baselines.
Results:
To analyze the contribution of each module, we conduct an ablation study on both ConvLSTM and ST-LSTM with respect to 3D convolution, two-way autoencoder and RPM and summarize the results in Table 2 Note that do not include the quantitative results of the combination of two-way autoencoder and Barck-CowyRNN predictor because it fails to produce consistent long-term generations and choose UNet (Ronneberger et al. (2015) as an alternative to two-way autoencoder. We can observe a significant improvement over ConvLSTMM after we embed into our CrevNet framework, indicating the effectiveness of reversible architectures. Also, integrating 3D convolution can consistently enhance the performance of all architectures. To further show the superior performance of CrevNet, we evaluate it on a harder 3-digit setting. Results are shown in the right column of Table Compared with the 2-digit setting, all models suffer a deterioration in quantitative performance due to the more frequent occurrence of overlapping digits. Nevertheless, our CrevNet still achieves the best resuit.
Input
3
3
3
3
3
>
>
ConvLSTM MSE
ST-LSTM
Model
SSIM
MSE
SSIM
3
1
33
>>
23
23
3/
2
Stacked RNN
103.3
0.707
56.8
0.869
Creviet
3
2
>>
2>
23
23
23
23
3D Stacked-RNM
85,8
0.785
46.2
0.878
15p-
>
2
2D
40
43
40
UNet+RPM
UNet+Stac
83,5
63.4
0.793
0.855
58.8
50.2
0.865
0.896
Pred
3
$
i
1
23
26
20
2
20
CrevNet w/o 3D
50.2
0.888
40.4
0.916
3
3
>
2>
23
20
2a
2
CrevNet + 3D
38.5
0.928
22.3
0.949
Figure 3: An extremely hard sequence of Mov- ing MNIST where two digits are continuously overlapped during the warm-up phase.
Table 2: An ablation study w.r.t. 3D convolution, RPM and two-way autoencoder. All metrics are averaged over the 10 predictions.
In Fig our qualitative analysis shows how each model performs on an extremely hard case of Moving MNIST where two digits are continuously overlapped during the warm-up phase.
5

---

Published as a conference paper at ICLR 2020
Input
Ground Truth
CrevNet
UNet ST-LSTM
10
11
12
10
11
12
10
11
12
Figure 4: The visual comparison of 4rasf The red boxes track some dynamics successfully captured by CrevNet. Better viewed large.
As we can see, our model is the only model that can differentiate the overlapping digits. The information-preservine property of the two-way autoencoder enables our method to reconstruct every fine detail moving digits after occlusion while baselines typically only restore the basic shape of these numbers. In fact, our CrevNet works almost perfectly on Moving MNIST, with most of its generations being visually indistinguishable from groundtruth.
We perform a human study to assess the fidelity of the video clips generated by different models. We presented pairs of video clips to human judges, where each pair consists of video clip from the test set together with the prediction generated by the model. The judges were asked to decide which of the two video clips is more likely be the groundtruth. To make each trail blind, the judges were not informed which model is used for generation and two sequences were randomly displayed on either side of the screen. We totally collected 2439 responses made by 58 human subjects and then calculated the probability that human judges answered correctly. The results are reported in Table [ The accuracy of 55.8 suggests that subjects could hardly detect the difference and their decisions were very close to random guesses.
3.2 SHORT-TERM PREDICTION TRAFFIC FLOW FORECASTING
Next, we evaluate our model on more complicated real-world dataset, Traffic4cast (IARAI (201)) which collects the traffic statuses of 3 big cities over year at S-minute interval. Traffic forecasting can be straightforwardly defined as video prediction task by its spatiotemppral nature. However, this dataset is quite challenging for the following reasons. (1). High resolution: The frame resolution of Traffic4cast is 495 x 436, which is the highest among all datasets. Existing resolution prrtioiinneee methods can hardly be adapted to this dataset since they all require extremely large memory and computation. Even if these models can be fitted in GPUs, they still do not have large enough receptive fields capture the meaningful dynamics as vehicles can move up 100 pixels between consecutive frames. (2). Complicated nonlinear dynamics: Valid data points only reside on the hidden roadmap of each city, which not explicitly provided in this dataset. Moving vehicles on these curved roads along with tangled road conditions will produce very complex nonlinear behaviours. It also involves many unobservable conditions or random events like weather and car accidents.
Each frame in Traffic4cast dataset is 495 x 436 x 3 heatmap, where the last dimension records 3 traffic statuses representing volume, mean speed and major direction at given location. The architecture of CrevNet is the same as the one we used on Moving MNIST. As we mentioned before, the existing resolution-prest ppreserinp methods cannot handle such high resolution input. Thus, to make the comparison possible, we add U-Net encoder-decoder to the baseline models including ConvLSTM and ST-LSTM. We train each model to predict next 3 frames (the next 15 minutes) from 9 observations and evaluate prediction with MSE criterion.
Datasets and Setup:
Model
CrevNet
UNeU+ST-LSTM
UNet+ComyLSTM
Best Team
2nd Best Team
MSE
9.340 x 10-3
9.725 x 10-3
9.846 x 10-3
9.559 x 10-3
9.717 x 1
Table 3: Quantitative evaluation on Traffic4cast. Lower MSE indicates better prediction accuracy.

---

Published as a conference paper at ICLR 2020
Input
Predictions (T 11)
T
Groundtruth
CrovNet
ContextVP
PredNet
Figure 5: The visual comparison of next-frame predictions on Caltech Pedestrian.
The quantitative comparison including the best two results on the leaderboard before the submission of this paper is provided in Table Unlike all previous state-of-the-art methods, CrevNet does not suffer from high memory consumption so that we were able to train our model in a single V100 GPU. The invertibility of two-way autoencoder preserves all necessary information for spatio- temporal learning and allows model to generate sharp and reasonable predictions. As illustrated in Fig[4, our model can identify and remember the hidden roadmap of each city through the learning of complicated nonlinear dynamics and accurately predict how traffic system will evolve.
Results:
3.3 NEXT-FRAME PREDICTION AND BEYOND- CAR-MOUNTED CAMERA VIDEO
The real-world videos are usually long-term unpredictable because of the intrinsic randomness and the lack of necessary information. Thus, the common practice for datasets like KITTI (Geiger et al.) (2012)), ) car-mounted camera video dataset, is to perform next-frame prediction. In this section, we further demonstrate the superior performance of our CrevNet by conducting experiments on KITTI and Caltech Pedestrian (Dollár et al. (2009). Compared with the previous two settings, car- mounted camera videos dataset presents another level difficulty for video prediction as it describes various nonlinear three-dimensional dynamics of multiple moving objects including backgrounds. Furthermore, as our well- trained model is capable of generating authentic future frames, it should spontaneously learn at least the shape and location of all moving objects, which indicates that the learnt features are very informative for downstream tasks. For example, in the case of object detection, these features can be incorporated estimate accurate locations and sizes of bounding boxes. Therefore, we also explore the effectiveness our self-supervised learning method on the 2D object detection KITTI.
3.3. VIDEO PREDICTION
Datasets and Setup: We follow the same protocol used in PredNet (Lotter et al. (2016)) for prepro-
cessing and evaluation. We first center-crop all video frames and resize them into 128 x 160. We compare our proposed method with state-of-the-art benchmark models. Models are trained on KITTI dataset to predict the next frame after 10-frame warm-up and are evaluated on Caltech Pedes- trian. The architecture of CrevNet used on KITTI is composed of 48- 48-layer two-way autoencoder
7

---

Published as a conference paper at ICLR 2020
T=1
=
T=5
=
9=
T=
PredNet
CycleGAN
CrevNet
Ground Truth
Figure 6: The visual comparison of 12-frame prediction Caltech Pedestrian. Notice how well CrevNet captures the detail and geometry of the buildings in the background, and the overall shading.
and 40 RPMs. Note that it the memory efficiency of our method that allows us to deploy such deep model. Since model also possess good capability of long-term prediction. We add 12 frame prediction comparison with CycleGAN (Kwon & Park (2019) and PredNet (Lotter et al. (016)). )
Model
Fram-Frame
3rd
6th
9th
12th
Average
PSNR
SSIM
SSIM
Copy-Last-Frame
23.3
0.779
Dual Motion GAN Liang et al.
2017)
0.899
ContextVP (Byeon et al. 2018
28.7
0.921
PredNet (Lotter et al. (2016)
27.6
0.905
0.72
0.66
0.61
0.58
0.701
CycleGAN (wwon & Park 2019)
29.2
0.919
0.83
0.73
0.67
0.63
0.752
CrevNet
29.3
0.925
0.84
0.76
0.70
0.65
0.776
Table 4: Quantitative evaluation of different methods on the Caltech Pedestrian dataset. Higher PSNR
or SSIM means better prediction accuracy.
Results: Performance different models in terms of PSNR and SSIM is displayed in Table CrevNet outperforms all baselines in both next-frame and multi-frame prediction regimes. Visual comparisons are provided in Fig ald Fig 6 Especially, in the case of 12-frame generation, we can observe that compared with our method, PredNet suffers severely from the famous eтrоr propagation of RNN issue while CycleGAN produces realistic yet physically inconsistent predictions.
3.3.2 2D OBJECT DETECTION
Datasets and Setup: KITTI provides three prior frames of unlabeled data for each labeled image. This allows us to run our CrevNet to extract useful spatiotemporal features for object detection. All video sequences were recorded at 10 Hz with resolution of 1242x375. We first resize each frame to 416x 128 and finetune our best model on the video prediction task solely. The combinations of features extracted by two-way autoencoder and attention masks of the target frame are then fed into the detection head for the further training. Note that we do not update the weights CrevNet at this stage purely demonstrate the power of self-supervised learning. Two image-based detection
8

---

Published as a conference paper at ICLR 2020
models, SqueezeDet and RRC, are compared as baselines. We also add experiment on transfer learning of features learnt by PredNet on KITTI as comparison. To be consistent with related work, we use SSD (Liu et al. (2016) as detection head.
Pedestrain M
Cyclist M
Buckhome
Add ons
mAP
E
M
II
B
E
H
SqueezeDet
Wa
2017
90.4
87.1
78.9
81.4
71.3
68.5
87.6
80.3
78.1
80.4
RRC
Ken et al
2
90.61
90.23
87.44
84.16
75.33
70.39
84,96
76.49
65.46
80.56
Best Results on the Leaderboard
91.96
91.97
84.57
88.27
81.73
75.29
84.28
79.24
71.22
83.17
PredNet
Lotter al. (016))
59.05
41.61
37.53
50.88
47.51
43,44
46.25
43.79
38.66
45.41
attention mask
91.53
90.95
85.71
89.31
82.55
75.21
85:51
80.41
71.52
83.63
CrovNet
+ extracted features
91.94
91.84
85.97
89.66
83.17
75.80
87.33
80.91
72.21
84.31
Table 5: Quantitative evaluation of different methods on 2D KITTI detection in term of Average Precision (AP). The numbers in the "Best Results on the Leaderboard are achieved by different best models on each class according to the default ranking mechanism on the Leaderboard.
The results of all experiments and baselines can be found in Table 5 Surprisingly, our CrevNet even outperforms the combination of the best model on each class in term of mAP. Since our model capable of capturing the motion information, it is sensitive to the small (hard) moving objects. However, the motion information alone is not sufficient for object detection due to the appearance of relatively static objects. Therefore, we can observe a performance boost after we incorporate the features extracted by our two-way autoencoder. Another advantage of our method is that it can provide a better localization of bounding box since the learnt features of CrevNet remain the pixel-wa wise alignment with the imput and output frame. Finally, thanks to the lightweight nature of our CrevNet, our best detection model can run at 6.8 FPS at the testing time.
Results:
RELATED WORK
Deep Learning in Video Prediction: Mainstream video prediction models can mostly be categorized into two frameworks, stacked ConvRNNs and mncoder-predictor-deeeee
models. The former framework attempts to design a new spatiotemporal module and then stacks multiple such modules to form the final model, while the latter usually utilizes an autoencoder project video frames into their latent representations and then employs a recurrent neural network to model the temporal transformations. PredNet (Lotter et al. (2016) is a good representative of stacked ConvRNNs framework. In PredNet, each ConvLSTM layer produces a layer-specific prediction at every time step to transmit an error term to the next layer. This model works well for predicting the next frame, but fails to maintain its performance in long-term setting. To tackle long-term predictions, PredRNN (Wang et al. (2017)) proposed a new spatiotemporal LSTM, which allows memory to flow both vertically and horizontally. PredRNN+ (Wang al. (2018) further improved the results by rearranging spatial and temporal memory in a cascaded mechanism, and by using a gradient highway architecture ease the optimization. F3D-LSSTM (Wang al. (2019) effectively recalled the previous memory states and also proposed to include 3D convolutions to enhance its performance. ContextVP (Bycon et al. (2018) introduced fully context-aware architecture consisting of parallel multi-dimensional LSTM units and blending units. Methods from stacked ConvRNNs family usually yield more accurate deterministic predictions but they consume considerable GPU memory and computational power as they abandon downsampling to prevent information loss.
The preddrr-rrdditto-ddcddeee framework, on the other hand, provides more flexibility than its counterpart. MCNET (Villegas et al.] (2017)) and DrNet (Denton et al.] (2017) decompose the content and motion in videos by building their corresponding encoders and then integrate this disentangled information to yield the next frame. Retrospective CycleGAN (Kwon & Park (2019)) combines sequential adversarial loss with frame adversarial loss, which encourages the model to generate frames that are visually similar to authentic images. In terms of modeling stochasticity. SVG (Denton) & Fergus (018)) and SAVG (Lee et al. (2018) utilize a prior inference network to mimic the uncertainty in the environment, and then embed it into deterministic generative model to produce stochastic video frames. VPN (Kalchbrenner et al, (2016)) estimates the discrete joint distribution of the raw pixel values in video using the well-established PixelCNNs. It is worth noticing that VPN

---

Published as a conference paper at ICLR 2020
employs resolution preseevvng encoder to circumvent the information loss, showing the need for an efficient information ieformation-preseriing encoder in the community.
Comparison with Related Works: To the best of our knowledge, CrevNet is the first condition- ally reversible model in the video prediction literature. There are three prior arts, E3D-LSTM (Wang) al. (2019)) FRNN (Oliu al. (2018) and VideoFlow (Kumar al. 2019)) having some similari- ties with our CrevNet. While E3D-LSTM also employs 3D convolutions, their implementation is essentially equivalent to applying two 2D convolutional operations, there is no shared filter on the temporal dimension. Similar to CrevNet, FRNN reduces its computational cost by eliminating the need to re-encode the output of decoder. However, FRNN has substantially different architecture compared to CrevNet. While the encoder and decoder in our model do not need information sharing at all, FRNN relies heavily on the sharing of the hidden states between them. Although VideoFlow also utilizes invertible transformation. This approach is very different from ours because: (1). VideoFlow built upon Glow, a very memory-consuminng architecture. Such memory limits preclude the use of 3D convolutions, even from training the model with Adam. (2). They use ANN to model temporal relationship. As such, VideoFlow cannot capture complex dynamics. (3). So far, VideoFlow has only been applied to stochastic video generation instead of deterministic video prediction.
The Reversible and Invertible Architectures:
The idea of the coupling layer was initially intro- to make the computation of the determinant of the Jacobian
duced in NICE (Dinh et al. (2014) and inverse Jacobian trival. Inspired by additive coupling layer, RevNet (Gomez et al.] (217)) introduced reversible block that allowed the reconstruction of activations of each layer from that of the next layer, thus eliminating the need to store activations between downsampling and significantly reducing its memory consumption. The follow-up work by (Jacobsen et al.) (2018)) further proposed an invertible extension, i-RevNet, which enabled the model to preserve all information of input
through layers while still being capable of extracting a useful representation for classification.
CONCLUSION
We described a novel conditionally reversible network, CrevNet, for pixel-level prediction of future frames in videos. The originality of our model lies in our use of the reversible two-way autoencoder and the accompanying reversible predictive module. Such architectural design enables the model to preserve fine-grained information without significant memory and computation overhead. CrevNet achieves state-of-the-an results on both synthetic and real-world datasets. The subsequent detection experiments demonstrate the potential of CrevNet to be continuous sule-sspe learning system to enhance downstream CV tasks, as shown in the case of BERT (Devlin et al. (2018) ) for NLP tasks.
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, al. Deep speech 2: End- to-end speech recognition in english and mandarin. In International Conference on Machine Learning, pp. 173-182, 2016.
Wonmin Byeon, Qin Wang, Rupesh Kumar Srivastava, and Petros Koumoutsakos. Contextvp: Fully context-aware video prediction. In Proceedingy of the European Conference on Computer Vision (ECCV), pp. 753-769, 2018
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2299-6308. 2017.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 1174-1183, Stockholmsmässann Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http:httt/////ceeeeteeesssss ngs. ml press/
v800/dentoo18 html
Emily L Denton et al. Unsupervised learning of disentangled representttins from video. In Advances in Neural Information Processing Systems, pp. 4417-4426. 2017.
10

---

Published as a conference paper at ICLR 2020
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components
estimation. arXiv preprint arXiv:1410.855 1410.8516, 2014.
Piotr Dollár, Christian Wojek, Bernt Schiele, and Pietro Perona. Pedestrian detection: A benchmark. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pp. 3354-3354330 3361. IEEE, 2012.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In Advances in Neural Information Processing Systems, pp. 221 2211-2 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
IARAL. Traffic4cast: Traffic map movie forecasting.
https://www http:///ww.a......... www.iaara.........
traffic4cast] 2019. Accessed: 2019-09-25.
Jörn-Henrik Jacobsen, Arnold Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. arXiv preprint arXiv:1802.07088, 2018
Nal Kalchbrenner, Aaron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. arXiv preprint 1610v0161. 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Manoj Kumar, Mohammad Babacizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. arXiv preprint 1903.01444 2019.
Hoog Kwon and Min-Gyu Park. Predicting future frames using retrospective cycle gan. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 181 1811-1820, 2019.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial vidco prediction. arXiv preprint arXiv:1804.01523, 2018.
Xiaodan Liang, Lisa Lee, Wei Dai, and Eric P Xing. Dual motion gan for future- fflore-flowwoooo embedded video prediction. In Proceedings of the IEFE International Conference on Computer Vision, pp. 1744-1752 2017.
Kun Liu, Wu Liu, Chuang Gan, Mingkui Tan, and Huadong Ma. T-c3d: Temporal convolutional 3d network for real-time action recognition. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng Yang Fu, and Alexander Berg. Ssd: Single shot multibox detector. European conference on computer vision, pp. 21-37. Springer, 2016.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv: 001048 2016.
Mare Oliu, Javier Selva, and Sergio Escalera. Folded recurrent neural networks for future video prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 716-731, 2018.
11

---

Published as a conference paper at ICLR 2020
Jimmy Ren, Xiaohao Chen, Jianbo Liu, Wenxiu Sun, Jiahao Pang, Qiong Yan, Yu-Wing lu-Wing Tai, and Xu. Accurate single stage detector using recurrent rolling convolution. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 5420-5428, 2017.
Shaoqing Ren, Kaiming He, Ross Girprerhick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, pp. 1874-1883, 2016.
Xingjian Shi, Zhourong Chen, Hao Wang. Dit- Din Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting, In Advances in neural information processing systems, pp. 802-810, 2015.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using lstms. In International conference machine learning, pp. 843-852, 2015.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. arXiv preprint arXiv:1706.0800 080333 2017.
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and S Yu Philip. Predrnn: Recur- rent neural networks for predictive learning using spatiotemporal Istms. In Advances in Neural Information Processing Systems, pp. 879-888, 2017.
Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and Philip S Yu. PredRNN++: Towards a resolution of the deep-in- time dilemma spatiotemporal predictive learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5123-5132, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://proceedding. mlr. press/ v800 1lb. htm.1
Yunbo Wang, Lu Jiang, Ming-Hsuan Yang. Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d LSTM: model for video prediction and beyond. In International Conference on Learning Representations, 2019,
pectpps:/openreviiee
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 1344)6600-612, 2004.
Bichen Wu, Forrest landola, Peter Jin, and Kurt Keutzer. Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 129-137, 2017.
12

---

Published as a conference paper at ICLR 2020
A CONVLSTM AND ST-LSTM
The key equations of CSnvLSTMM are shown as belows.
i o 0t 24 + Whs Hi-1 + b,)
f. a (W,, zf X Why H + by)
C/ f. C + 4 tanh ( sc 2₁ + Whe H4-1 + be)
0 = a (Wzo X. + Who * H6 Weo o C₁ + bo)
H = tanh(C.)
where denotes the convolution operator and o denotes the Hadamard product. Based on ConvLSTM, spatiotemporal L.TTM (ST-LSTM) in PredRNN adds another vertical memory flow enhance the long-term temporal dependency as follow.
4 aW ni X, + Wh H + b;)
fr a aW., a. X4 Why H4- + b,)
C ft - 4 tanh We 24 + Whe H4 + bc)
& aWW, * 24 + mm M +b)
fr a oW,, X, + W -M + b'g)
M = H M + 1/ tanh( sm X, + W mm * M bm)
0t = a (W.. X4 + Who * Hi-1 + We o C. + Wmo M + ba)
Hi = tanh (1 IC,, M)
where blue part overlaps ConvLSTMM. Note that M usually receives information from the previous layer instead of the previous state and the special case is that M₁ receives M to constitute zigzag information flow. As we can see, ST-LSTM basically doubles the size of feature map and the number of parameters compared with ComvLSTM.
B CONDITIONAL REVERSIBILITY
As we mentioned in Section 2.2, conditional reversibility is an interesting property of our CrevNet. In this section, we will provide more details about Given &2, x! and h 1. the reversible predictive module can recover af as follow
h ConvRNN(x!..oe) (111)
9: <(W * h + +b)
x ( 2 01111/19
Ideally, if there is no numerical error during the calculation, we can get the perfect reconstruction of input by applying this inverse operation repeatedly. In practice, while the most of reverse generations are successful, the inevitable numerical error will still result in some failing cases, especially in the case of a very deep architecture because will be amplified layer by layer.
C OBJECT DETECTION
13

---

Published as a conference paper at ICLR 2020
0
100
Gaf
Fatarar
e can rian
car
200
300
200
400
600
800
1000
1200
(a)
100
car
car
200
300
200
400
600
800
1000
1200
(b)
100
car
200
300
200
400
600
800
1000
1200
0
100
200
300
200
400
600
800
1000
1200
(d)
Figure 7: Visualization of 2D Object Detection on KITTI.
14


Reviews:
[{"review_id": "BkxpyxE79H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed CrevNet, a conditionally reversible network, that performs video prediction efficiently and effectively.\nThe proposed CrevNet used the bijective two-way autoencoder to make sure the frames are inherently invertible from the latent space.\n\nWhile the idea of the paper is good, I am not convinced of some points raised in the paper. And I hope to get response from the authors.\n(1) The latent space of the proposed method is not very small, it should be at least the same dimension as the input given invertibility, so it is w*h*c. What are the dimensions of resolution-preserving methods?\n(2) According to previous studies, although i-RevNet preserves all information in the latent features, it is not make robust mappings: a small perturbation on the features will ruin the inverted image. Is it still the case in CrevNet? Can we some how visualize what is happening/changing in the latent space? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xCq-c6tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces Conditionally Reversible Network (CrevNet) that consists of the invertible autoencoder and a reversible predictive module (RPM). The two-way autoencoder is an invertible network that preserves the volume with no information loss while reducing memory consumption by using bijective downsampling. The RPM is a recurrent extension of two-way autoencoder that provides the reversiblity in temporal domain. The experiments on Moving MNIST, Traffic4cast, KITTI, and 2D object detection on KITTI show the improvement compare to other state-of-the-art models. \n\nThe paper is well-written and the contributions are clear. The experiments on diverse tasks and datasets are provided. Especially, the use of pretrained generative model for the object detection task shows the benefit of the model.   \n\nHowever, it is not clear the necessity of the proposed model and how the memory demand and computation cost are reduced. \n\nTo have a better support for the contributions, I have following suggestions. \n\n- The benefit or necessity of each module is not clear. The comparisons with 1) only the invertible two-way autoencoder (without Section 2.2 and 2.3)\", 2) only RPM (without Section 2.1 and/or 2.3), and 3)two-way autoencoder + reversible predictive model without 3D convolutions (without Section 2.3) are required. \n\n- Model size/memory comparison is provided only for the MNIST dataset. Since the main benefit of the proposed model is low memory demand and computation cost, I suggest to provide the the comparisons of the memory demand with other models as well.  \n1) Other models on on Caltech Pedestrian dataset, Especially, the performance gap on Caltech Pedestrian dataset between the proposed method and CycleGAN is very small. \n2) Some of the compared models on 2D KITTI detection are also lightweighted e.g., SqeeqeDet. The comparison of the model size/memery/test time with these models are necessary. \n\n---- \n<After rebuttal>\nAuthors have mostly addressed my concerns, and the contributions are clearer. I adjusted my rating accordingly. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1x9JLh6Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new method of self-supervised feature learning from videos based on learning future frame prediction. The idea is similar as BERT like NLP tasks, but for videos, the computational cost and memory cost could be very large. To solve this problem efficiently, the authors adopt several existing techniques such as pixel shuffle layer, 3D-CNN,  ConvRNN and Attention module to efficiently and effectively capture video information. Experiments on several datasets are conducted to show the effectiveness of the proposed method.\n\nThe idea of self-supervised feature learning from videos are not novel. The key is how to learn good features that can generalize very well. The authors show that the learned features in this paper can be used on other tasks, such as object detection. And state-of-the-art results on KITTI dataset could be achieved based on the learned features with fixed backbone parameters. Although the provided results may not be state-of-the-art (for car, it seems that KITTI best results are 97%+ for easy, 95%+ for medium, 90%+ for hard, instead of 92%, 92%, and 85% provided in the paper), the generalization ability on object detection looks very interesting.\n\nIs the generalization ability of the proposed method better that existing methods such as CycleGAN, PredNet and ContextVP? More experiments about the quality of the learned features of the proposed method are recommended to improve the importance of this paper. The advantage of the proposed method looks weak without these comparisons.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new approach that consists of the invertible autoencoder and a reversible predictive module (RPM) for video future-frame prediction.

Reviewers agree that the paper is well-written and the contributions are clear. It achieves new state-of-the-art results on a diverse set of video prediction datasets and with techniques that enable more efficient computation and memory footprint. Also, the video representation learned in a self-supervised way by the approach can have good generalization ability on downstream tasks such as object detection. The concerns of the paper were relatively minor, and were successfully addressed in the rebuttal.

AC feels that this work makes a solid contribution with well-designed model and strong empirical performance, which will attain wide interests in the area of video future-frame prediction and self-supervised video representation learning.

Hence, I recommend accepting this paper.
Decision: Accept (Poster)