Efficient and Information-Preserving Future Frame Prediction and Beyond

1 INTRODUCTION . Deep learning has enjoyed tremendous success in recent years due to its ability to capture complex dependencies and non-linearities in large datasets ( Krizhevsky et al . ( 2012 ) ; He et al . ( 2016 ) ; Gomez et al . ( 2017 ) ) . Excellent performance has been achieved on a wide range of supervised machine learning tasks , ranging from image classification ( He et al . ( 2016 ) ) and object detection ( Ren et al . ( 2015 ) ) to speech recognition ( Amodei et al . ( 2016 ) ) . Despite the significant breakthrough in supervised learning , the potential of applying deep architectures to unsupervised learning problems remains largely unexplored . Lately there has been a surge of interest in the task of video prediction , i.e. , to predict future frames of a video sequence ( Wang et al . ( 2017 ; 2018 ) ; Denton et al . ( 2017 ) ; Denton & Fergus ( 2018 ) ; Villegas et al . ( 2017 ) ; Lee et al . ( 2018 ) ) . The significance of video prediction primarily lies in its potential of discovering dynamics in the physical world . The self-supervised nature of video prediction aligns well with how humans learn , without requiring large amounts of labeled data . In addition , videos can provide an abundant and virtually unlimited source of visual information . This allows video prediction models to serve as a generative pre-training strategy of feature representation learning for a variety of downstream supervised tasks . To date , most of the existing models for video prediction employ a hybrid of convolutional and recurrent layers as the underlying architecture ( Wang et al . ( 2017 ) ; Shi et al . ( 2015 ) ; Lotter et al . ( 2016 ) ) . Such architectural design enables the model to simultaneously exploit the ability of convolutional units to model spatial relationships and the potential of recurrent units to capture temporal dependencies . Despite their prevalence in the literature , classical video prediction architectures suffer from two major limitations . Firstly , in dense prediction tasks such as video prediction , models are required to make pixel-wise predictions , which emphasizes the demand for the preservation of information through layers . Prior works attempt to address such demand through the extensive use of resolution-preserving blocks ( Wang et al . ( 2017 ; 2018 ) ; Kalchbrenner et al . ( 2016 ) ) . Nevertheless , these resolution-preserving blocks are not guaranteed to preserve all the relevant information , and they greatly increase the memory consumption and computational cost of the models . The second drawback of existing video prediction models is that they can not efficiently take advantage of 3D convolutions , as that would make these already cumbersome architectures even larger . 3D convolutions have been shown to be a very effective alternative to RNNs to capture temporal relations in a variety of video tasks ( Liu et al . ( 2018 ) ; Carreira & Zisserman ( 2017 ) ) , and thus desirable to exploit . Recently , reversible architectures ( Dinh et al . ( 2014 ) ; Gomez et al . ( 2017 ) ; Jacobsen et al . ( 2018 ) ) have attracted attention due to their light memory demand and their information preserving property by design . However , the effectiveness of reversible models remains greatly unexplored in the video literature . In this paper , we introduce a novel , conditionally reversible video prediction model , CrevNet , in the sense that when conditioned on previous hidden states , it can exactly reconstruct the input from its predictions . The contribution of this work can be summarized as follows : • We introduce a two-way autoencoder that uses the forward and backward passes of an invertible network as encoder and decoder ( Fig 1 ) . The volume-preserving two-way autoencoder not only greatly reduces the memory demand and computational cost , but also enjoys the theoretically guaranteed property of no information loss . The lightweight nature of our model enables us to incorporate 3D convolutions without concern of memory bottleneck . • We propose the reversible predictive module ( RPM ) , as illustrated in Fig 2b , which extends the reversibility from spatial to temporal domain . RPM , together with the two-way autoencoder , provides a conditionally reversible architecture ( CrevNet ) for spatiotemporal learning . CrevNet achieves the state-of-the-art results on Moving MNIST , Traffic4cast and KITTI . • We evaluate the effectiveness of features learnt from self-supervision by adapting our CrevNet for object detection on KITTI . Our competitive results indicate the potential of using CrevNet as a generative pre-training strategy to guide downstream CV tasks . 2 APPROACH . We first outline the general pipeline of our method . Our CrevNet consists of two subnetworks , an autonencoder network with an encoder E , decoder D and a recurrent predictor P bridging encoder and decoder . Let xt ∈ Rw×h×c represent the tth frame in video x , where w , h , and c denote its width , height , and the number of channels . Given x0 : t−1 , the model predicts the next frame x̂t as follows : x̂t = D ( P ( E ( xt−1 ) |x0 : t−2 ) ) ( 1 ) In the case of 3D convolution , xt ∈ Rk×w×h×c denotes the short video clip from t to t + k − 1 instead of a single frame at timestep t , where k is the temporal dimension of input or output . During the multi-frame generation process without access to the ground truth frames , the model uses its previous predictions instead . 2.1 THE INVERTIBLE TWO-WAY AUTOENCODER . We propose a bijective two-way autoencoder based on the additive coupling layer introduced in NICE ( Dinh et al . ( 2014 ) ) . We begin with describing the building block of the two-way autoencoder ( Fig 2a ) . Formally , the input x is first reshaped and split channelwise into two groups , denoted as x1 and x2 . During the forward pass of each building block , one group , e.g . x1 , passes through several convolutions and activations and is then added to another group , x2 , like a residual block : x̂2 = x2 + F1 ( x1 ) x̂1 = x1 + F2 ( x̂2 ) ( 2 ) where F is a composite non-linear operator consisting of convolutions and activations , and x̂1 and x̂2 are the updated x1 and x2 . Note that x1 and x2 can be simply recovered from x̂2 and x̂1 by the inverse computation ( Fig 2c ) as follows : x1 = x̂1 −F2 ( x̂2 ) x2 = x̂2 −F1 ( x1 ) ( 3 ) Multiple building blocks are stacked in an alternating fashion between x1 and x2 to construct a two-way autoencoder , as shown in Fig 2a . A series of the forward and inverse computations builds a one-to-one and onto , i.e . bijective , mapping between the input and features . Such invertibility ensures that there is no information loss during the feature extraction , which is presumably more favorable for video prediction since the model is expected to restore the future frames with finegrained details . To enable the invertibility of the entire autoencoder , our two-way autoencoder uses a bijective downsampling , pixel shuffle layer ( Shi et al . ( 2016 ) ) , that changes the shape of feature from ( w , h , c ) to ( w/n , h/n , c× n2 ) . The resulting volume-preserving architecture can greatly reduce its memory consumption compared with the existing resolution-preserving methods . We further argue that for generative tasks , e.g . video prediction , we can effectively utilize a single two-way autoencoder , and to use its forward and backward pass as the encoder and the decoder , respectively . The predicted frame x̂t is thus given by x̂t = E−1 ( P ( E ( xt−1 ) |x0 : t−2 ) ) ( 4 ) where E−1 is the backward pass of E . Our rationale is that , such setting would not only reduce the number of parameters in the model , but also encourage the model to explore the shared feature space between the inputs and the targets . As a result , our method does not require any form of information sharing , e.g . skip connection , between the encoder and decoder . In addition , our two-way autoencoder can enjoy a lower computational cost at the multi-frame prediction phase where the encoding pass is no longer needed and the predictor directly takes the output from previous timestep as input , as shown in Fig 1 , since E ( E−1 ) is an identity mapping . 2.2 REVERSIBLE PREDICTIVE MODULE . In this section , we describe the second part of our video prediction model , the predictor P , which computes dependencies along both the space and time dimensions . Although the traditional stackedConvRNN layers architecture is the most straightforward choice of predictor , we find that it fails to establish a consistent temporal dependency when equipped with our two-way autoencoder through experiments . Therefore , we propose a novel reversible predictive module ( RPM ) , which can be regarded as a recurrent extension of the two-way autoencoder . In the RPM , we substitute all standard convolutions with layers from the ConvRNN family ( e.g . ConvLSTM or spatiotemporal LSTM ) and introduce a soft attention ( weighting gates ) mechanism to form a weighted sum of the two groups instead of the direct addition . The main operations of RPM used in this paper are given as follows : h1t = ConvRNN ( x 1 t , h 1 t−1 ) ConvRNN gt = φ ( W2 ∗ ReLU ( W1 ∗ h1t + b1 ) + b2 ) Attention module x̂2t = ( 1− gt ) x2t + gt h1t Weighted sum where x1t and x 2 t denote two groups of features at timestep t , h 1 t denote the hidden states of ConvRNN layer , φ is sigmoid activation , ∗ is the standard convolution operator and is the Hadamard product . The architecture of reversible predictive module is also shown in Fig 2b . RPM adopts a similar architectural design as the two-way autoencoder to ensure a pixel-wise alignment between the input and the output , i.e . each position of features can be traced back to certain pixel , and thus make it compatible with our two-way autoencoder . It also mitigates the vanishing gradient issues across stacked layers since the coupling layer provides a nice property w.r.t . the Jacobian ( Dinh et al . ( 2014 ) ) . In addition , the attention mechanism in the RPM enables the model to focus on objects in motion instead of background , which further improves the video prediction quality . Similarly , multiple RPMs alternate between the two groups to form a predictor . We call this predictor conditionally reversible since , given ht−1 , we are able to reconstruct xt−1 from x̂t if there are no numerical errors : xt−1 = E−1 ( P−1 ( E ( x̂t ) |ht−1 ) ) ( 5 ) where P−1 is the inverse computation of the predictor P . We name the video prediction model using two-way autoencoder as its backbone and RPMs as its predictor CrevNet . Another key factor of RPM is the choice of ConvRNN . In this paper , we mainly employ ConvLSTM ( Shi et al . ( 2015 ) ) and spatiotemporal LSTM ( ST-LSTM , Wang et al . ( 2017 ) ) to enable a fair comparison with baselines .

Summary: This paper introduces Conditionally Reversible Network (CrevNet) that consists of the invertible autoencoder and a reversible predictive module (RPM). The two-way autoencoder is an invertible network that preserves the volume with no information loss while reducing memory consumption by using bijective downsampling. The RPM is a recurrent extension of two-way autoencoder that provides the reversiblity in temporal domain. The experiments on Moving MNIST, Traffic4cast, KITTI, and 2D object detection on KITTI show the improvement compare to other state-of-the-art models.