Efficient and Information-Preserving Future Frame Prediction and Beyond

1 INTRODUCTION . Deep learning has enjoyed tremendous success in recent years due to its ability to capture complex dependencies and non-linearities in large datasets ( Krizhevsky et al . ( 2012 ) ; He et al . ( 2016 ) ; Gomez et al . ( 2017 ) ) . Excellent performance has been achieved on a wide range of supervised machine learning tasks , ranging from image classification ( He et al . ( 2016 ) ) and object detection ( Ren et al . ( 2015 ) ) to speech recognition ( Amodei et al . ( 2016 ) ) . Despite the significant breakthrough in supervised learning , the potential of applying deep architectures to unsupervised learning problems remains largely unexplored . Lately there has been a surge of interest in the task of video prediction , i.e. , to predict future frames of a video sequence ( Wang et al . ( 2017 ; 2018 ) ; Denton et al . ( 2017 ) ; Denton & Fergus ( 2018 ) ; Villegas et al . ( 2017 ) ; Lee et al . ( 2018 ) ) . The significance of video prediction primarily lies in its potential of discovering dynamics in the physical world . The self-supervised nature of video prediction aligns well with how humans learn , without requiring large amounts of labeled data . In addition , videos can provide an abundant and virtually unlimited source of visual information . This allows video prediction models to serve as a generative pre-training strategy of feature representation learning for a variety of downstream supervised tasks . To date , most of the existing models for video prediction employ a hybrid of convolutional and recurrent layers as the underlying architecture ( Wang et al . ( 2017 ) ; Shi et al . ( 2015 ) ; Lotter et al . ( 2016 ) ) . Such architectural design enables the model to simultaneously exploit the ability of convolutional units to model spatial relationships and the potential of recurrent units to capture temporal dependencies . Despite their prevalence in the literature , classical video prediction architectures suffer from two major limitations . Firstly , in dense prediction tasks such as video prediction , models are required to make pixel-wise predictions , which emphasizes the demand for the preservation of information through layers . Prior works attempt to address such demand through the extensive use of resolution-preserving blocks ( Wang et al . ( 2017 ; 2018 ) ; Kalchbrenner et al . ( 2016 ) ) . Nevertheless , these resolution-preserving blocks are not guaranteed to preserve all the relevant information , and they greatly increase the memory consumption and computational cost of the models . The second drawback of existing video prediction models is that they can not efficiently take advantage of 3D convolutions , as that would make these already cumbersome architectures even larger . 3D convolutions have been shown to be a very effective alternative to RNNs to capture temporal relations in a variety of video tasks ( Liu et al . ( 2018 ) ; Carreira & Zisserman ( 2017 ) ) , and thus desirable to exploit . Recently , reversible architectures ( Dinh et al . ( 2014 ) ; Gomez et al . ( 2017 ) ; Jacobsen et al . ( 2018 ) ) have attracted attention due to their light memory demand and their information preserving property by design . However , the effectiveness of reversible models remains greatly unexplored in the video literature . In this paper , we introduce a novel , conditionally reversible video prediction model , CrevNet , in the sense that when conditioned on previous hidden states , it can exactly reconstruct the input from its predictions . The contribution of this work can be summarized as follows : • We introduce a two-way autoencoder that uses the forward and backward passes of an invertible network as encoder and decoder ( Fig 1 ) . The volume-preserving two-way autoencoder not only greatly reduces the memory demand and computational cost , but also enjoys the theoretically guaranteed property of no information loss . The lightweight nature of our model enables us to incorporate 3D convolutions without concern of memory bottleneck . • We propose the reversible predictive module ( RPM ) , as illustrated in Fig 2b , which extends the reversibility from spatial to temporal domain . RPM , together with the two-way autoencoder , provides a conditionally reversible architecture ( CrevNet ) for spatiotemporal learning . CrevNet achieves the state-of-the-art results on Moving MNIST , Traffic4cast and KITTI . • We evaluate the effectiveness of features learnt from self-supervision by adapting our CrevNet for object detection on KITTI . Our competitive results indicate the potential of using CrevNet as a generative pre-training strategy to guide downstream CV tasks . 2 APPROACH . We first outline the general pipeline of our method . Our CrevNet consists of two subnetworks , an autonencoder network with an encoder E , decoder D and a recurrent predictor P bridging encoder and decoder . Let xt ∈ Rw×h×c represent the tth frame in video x , where w , h , and c denote its width , height , and the number of channels . Given x0 : t−1 , the model predicts the next frame x̂t as follows : x̂t = D ( P ( E ( xt−1 ) |x0 : t−2 ) ) ( 1 ) In the case of 3D convolution , xt ∈ Rk×w×h×c denotes the short video clip from t to t + k − 1 instead of a single frame at timestep t , where k is the temporal dimension of input or output . During the multi-frame generation process without access to the ground truth frames , the model uses its previous predictions instead . 2.1 THE INVERTIBLE TWO-WAY AUTOENCODER . We propose a bijective two-way autoencoder based on the additive coupling layer introduced in NICE ( Dinh et al . ( 2014 ) ) . We begin with describing the building block of the two-way autoencoder ( Fig 2a ) . Formally , the input x is first reshaped and split channelwise into two groups , denoted as x1 and x2 . During the forward pass of each building block , one group , e.g . x1 , passes through several convolutions and activations and is then added to another group , x2 , like a residual block : x̂2 = x2 + F1 ( x1 ) x̂1 = x1 + F2 ( x̂2 ) ( 2 ) where F is a composite non-linear operator consisting of convolutions and activations , and x̂1 and x̂2 are the updated x1 and x2 . Note that x1 and x2 can be simply recovered from x̂2 and x̂1 by the inverse computation ( Fig 2c ) as follows : x1 = x̂1 −F2 ( x̂2 ) x2 = x̂2 −F1 ( x1 ) ( 3 ) Multiple building blocks are stacked in an alternating fashion between x1 and x2 to construct a two-way autoencoder , as shown in Fig 2a . A series of the forward and inverse computations builds a one-to-one and onto , i.e . bijective , mapping between the input and features . Such invertibility ensures that there is no information loss during the feature extraction , which is presumably more favorable for video prediction since the model is expected to restore the future frames with finegrained details . To enable the invertibility of the entire autoencoder , our two-way autoencoder uses a bijective downsampling , pixel shuffle layer ( Shi et al . ( 2016 ) ) , that changes the shape of feature from ( w , h , c ) to ( w/n , h/n , c× n2 ) . The resulting volume-preserving architecture can greatly reduce its memory consumption compared with the existing resolution-preserving methods . We further argue that for generative tasks , e.g . video prediction , we can effectively utilize a single two-way autoencoder , and to use its forward and backward pass as the encoder and the decoder , respectively . The predicted frame x̂t is thus given by x̂t = E−1 ( P ( E ( xt−1 ) |x0 : t−2 ) ) ( 4 ) where E−1 is the backward pass of E . Our rationale is that , such setting would not only reduce the number of parameters in the model , but also encourage the model to explore the shared feature space between the inputs and the targets . As a result , our method does not require any form of information sharing , e.g . skip connection , between the encoder and decoder . In addition , our two-way autoencoder can enjoy a lower computational cost at the multi-frame prediction phase where the encoding pass is no longer needed and the predictor directly takes the output from previous timestep as input , as shown in Fig 1 , since E ( E−1 ) is an identity mapping . 2.2 REVERSIBLE PREDICTIVE MODULE . In this section , we describe the second part of our video prediction model , the predictor P , which computes dependencies along both the space and time dimensions . Although the traditional stackedConvRNN layers architecture is the most straightforward choice of predictor , we find that it fails to establish a consistent temporal dependency when equipped with our two-way autoencoder through experiments . Therefore , we propose a novel reversible predictive module ( RPM ) , which can be regarded as a recurrent extension of the two-way autoencoder . In the RPM , we substitute all standard convolutions with layers from the ConvRNN family ( e.g . ConvLSTM or spatiotemporal LSTM ) and introduce a soft attention ( weighting gates ) mechanism to form a weighted sum of the two groups instead of the direct addition . The main operations of RPM used in this paper are given as follows : h1t = ConvRNN ( x 1 t , h 1 t−1 ) ConvRNN gt = φ ( W2 ∗ ReLU ( W1 ∗ h1t + b1 ) + b2 ) Attention module x̂2t = ( 1− gt ) x2t + gt h1t Weighted sum where x1t and x 2 t denote two groups of features at timestep t , h 1 t denote the hidden states of ConvRNN layer , φ is sigmoid activation , ∗ is the standard convolution operator and is the Hadamard product . The architecture of reversible predictive module is also shown in Fig 2b . RPM adopts a similar architectural design as the two-way autoencoder to ensure a pixel-wise alignment between the input and the output , i.e . each position of features can be traced back to certain pixel , and thus make it compatible with our two-way autoencoder . It also mitigates the vanishing gradient issues across stacked layers since the coupling layer provides a nice property w.r.t . the Jacobian ( Dinh et al . ( 2014 ) ) . In addition , the attention mechanism in the RPM enables the model to focus on objects in motion instead of background , which further improves the video prediction quality . Similarly , multiple RPMs alternate between the two groups to form a predictor . We call this predictor conditionally reversible since , given ht−1 , we are able to reconstruct xt−1 from x̂t if there are no numerical errors : xt−1 = E−1 ( P−1 ( E ( x̂t ) |ht−1 ) ) ( 5 ) where P−1 is the inverse computation of the predictor P . We name the video prediction model using two-way autoencoder as its backbone and RPMs as its predictor CrevNet . Another key factor of RPM is the choice of ConvRNN . In this paper , we mainly employ ConvLSTM ( Shi et al . ( 2015 ) ) and spatiotemporal LSTM ( ST-LSTM , Wang et al . ( 2017 ) ) to enable a fair comparison with baselines .

Summary: In this paper, the authors propose a new method of self-supervised feature learning from videos based on learning future frame prediction. The idea is similar as BERT like NLP tasks, but for videos, the computational cost and memory cost could be very large. To solve this problem efficiently, the authors adopt several existing techniques such as pixel shuffle layer, 3D-CNN,  ConvRNN and Attention module to efficiently and effectively capture video information. Experiments on several datasets are conducted to show the effectiveness of the proposed method.