Unsupervised Video Decomposition using Spatio-temporal Iterative Inference

1 INTRODUCTION . Unsupervised representation learning , which has a long history dating back to Boltzman Machines ( Hinton & Sejnowski , 1986 ) and original works of Marr ( 1970 ) , has recently emerged as one of the important directions of research , carrying the newfound promise of alleviating the need for excessively large and fully labeled datasets . More traditional representation learning approaches focus on unsupervised ( e.g. , autoencoder-based ( Pathak et al. , 2016 ; Vincent et al. , 2008 ) ) or selfsupervised ( Noroozi & Favaro , 2016 ; Vondrick et al. , 2016 ; Zhang et al. , 2016 ) learning of holistic representations that , for example , are tasked with producing ( spatial ( Noroozi & Favaro , 2016 ) , temporal ( Vondrick et al. , 2016 ) , or color ( Zhang et al. , 2016 ) ) encodings of images or patches . The latest and most successful methods along these lines include ViLBERT ( Lu et al. , 2019 ) and others ( Sun et al. , 2019 ; Tan & Bansal , 2019 ) that utilize powerful transformer architectures ( Vaswani et al. , 2017 ) coupled with proxy multi-modal tasks ( e.g. , masked token prediction or visua-lingual alignment ) . Learning of good disentangled , spatially granular , representations that are , for example , able to decouple object appearance and shape in complex visual scenes consisting of multiple moving objects remains elusive . Recent works that attempt to address this challenge can be characterized as : ( i ) attention-based methods ( Crawford & Pineau , 2019b ; Eslami et al. , 2016 ) , which infer latent representations for each object in a scene , and ( ii ) iterative refinement models ( Greff et al. , 2019 ; 2017 ) , which decompose a scene into a collection of components by grouping pixels . Importantly , the former have been limited to latent representations at object- or image patch-levels , while the latter class of models have illustrated the ability for more granular latent representations at the pixel ( segmentation ) -level . Specifically , most refinement models learn pixel-level generative models driven by spatial mixtures ( Greff et al. , 2017 ) and utilize amortized iterative refinements ( Marino et al. , 2018 ) for inference of disentangled latent representations within the VAE framework ( Kingma & Welling , 2014 ) ; a prime example is IODINE ( Greff et al. , 2019 ) . However , while providing a powerful model and abstraction which is able to segment and disentangle complex scenes , IODINE ( Greff et al. , 2019 ) and other similar architectures are fundamentally limited by the fact that they only consider images . Even when applied for inference in video , they process one frame at a time . This makes it excessively challenging to discover and represent individual instances of objects that may share properties such as appearance and shape but differ in dynamics . In computer vision , it has been a long-held belief that motion carries important information for segmenting objects ( Jepson et al. , 2002 ; Weiss & Adelson , 1996 ) . Armed with this intuition , we propose a spatio-temporal amortized inference model capable of not only unsupervised multi-object scene decomposition , but also of learning and leveraging the implicit probabilistic dynamics of each object from perspective raw video alone . This is achieved by introducing temporal dependencies between the latent variables across time . As such , IODINE ( Greff et al. , 2019 ) could be considered a special ( spatial ) case of our spatio-temporal formulation . Modeling temporal dependencies among video frames also allows us to make use of conditional priors ( Chung et al. , 2015 ) for variational inference , leading to more accurate and efficient inference results . The resulting model , illustrated in Fig . 1 , achieves superior performance on complex multi-object benchmark datasets ( Bouncing Balls and CLEVRER ) with respect to state-of-the-art models , including R-NEM ( Van Steenkiste et al. , 2018 ) and IODINE ( Greff et al. , 2019 ) in terms of segmentation , prediction , and generalization . Our model has a number of appealing properties , including temporal extrapolation , computational efficiency , and the ability to work with complex data exhibiting non-linear dynamics , colors , and changing number of objects within the same video sequence . In addition , we introduce an entropy prior to improve our model ’ s performance in scenarios where object appearance alone is not sufficiently distinctive ( e.g. , greyscale data ) . 2 RELATED WORK . Unsupervised Scene Representation Learning . Unsupervised scene representation learning can generally be divided into two groups : attention-based methods , which infer latent representations for each object in a scene , and more complex and powerful iterative refinement models , which often make use of spatial mixtures and can decompose a scene into a collection of estimated components by grouping pixels together . Attention-based methods , such as AIR ( Eslami et al. , 2016 ) ( Xu et al. , 2019 ) and SPAIR ( Crawford & Pineau , 2019b ) , decompose scenes into latent variables representing the appearance , position , and size of the underlying objects . However , both methods can only infer the objects ’ bounding boxes and have not been shown to work on non-trivial 3D scenes with perspective distortions and occlusions . MoNet ( Burgess et al. , 2019 ) is the first model in this family tackling more complex data and inferring representations that can be used for instance segmentation of objects . On the other hand , it is not a probabilistic generative model and thus not suitable for density estimation . GENESIS ( Engelcke et al. , 2020 ) extends it and alleviates some of its limitations by introducing a probabilistic framework and allowing for spatial relations between the objects . DDPAE ( Hsieh et al. , 2018 ) is a framework that uses structured probabilistic models to decompose a video into low-dimensional temporal dynamics with the sole purpose of prediction . It is shown to operate on binary scenes with no perspective distortion and is not capable of generating per-object segmentation masks . Iterative refinement models started with Tagger ( Greff et al. , 2016 ) that reasons about the segmentation of its inputs . However , it does not allow explicit latent representations and can not be scaled to more complex images . NEM ( Greff et al. , 2017 ) , as an extension of Tagger , uses a spatial mixture model inside an expectation maximization framework , but is limited to binary data . Finally , IODINE ( Greff et al. , 2019 ) is a notable example of a model employing iterative amortized inference w.r.t . a spatial mixture formulation and achieves state-of-the-art performance in scene decomposition and segmentation . Unsupervised Video Tracking and Object Detection . SQAIR ( Kosiorek et al. , 2018 ) , SILOT ( Crawford & Pineau , 2019a ) and SCALOR ( Jiang et al. , 2020 ) are temporal extensions of the static attention-based models that are tailored to tracking and object detection tasks . SQAIR is restricted to binary data and operates at the level of bounding boxes . SILOT and SCALOR are more expressive and can cope with cluttered scenes , a larger numbers of objects , and dynamic backgrounds , but do not work on colored perspective1 data ; accurate segmentation remains a challenge . Unsupervised Video Decomposition and Segmentation . Models employing spatial mixtures and iterative inference in a temporal setting are closest to our method from a technical perspective . Notably , there are only few models falling into this line of work : RTagger ( Prémont-Schwarz et al. , 2017 ) is a recurrent extension of Tagger and has same limitations as its predecessor . R-NEM ( Van Steenkiste et al. , 2018 ) effectively learns the objects ’ dynamics and interactions through a relational module and can produce segmentations but is limited to orthographic binary data . Methods without Latent Modeling . GAN-based ReDO ( Chen et al. , 2019 ) uses a model built around the assumption that object regions are independent , guiding the generator by drawing objects ’ pixel regions separately and composing them after segmentation . Another model ( Arandjelović & Zisserman , 2019 ) employs the same principles but guide the generator by copying a region of an image into another one . Both architectures are shown to operate on static images only and do not have a clearly interpretable latent space or prediction capabilities . Our method allows an effective use of temporal information in object-centric decompositions of colored video data . This places our approach between methods like R-NEM , which strictly operates on binary data , and IODINE , whose usage of temporal information is ad-hoc and produces results of limited quality ( Table 1 ) . In practice , we leverage a 2D-LSTM and employ an implicit modeling of dynamics by incorporating the hidden states into a conditional prior in the efficient runtime manner . 3 DYNAMIC VIDEO DECOMPOSITION . We now introduce our dynamic model for unsupervised video decomposition . Our approach builds upon a generative model of multi-object representations and leverages elements of iterative amortized inference . We briefly review both concepts ( §3.1 ) and then introduce our model ( §3.2 ) . 3.1 BACKGROUND . Multi-Object Representations . The multi-object framework introduced in Greff et al . ( 2019 ) decomposes a static image x = ( xi ) i ∈ RD into K objects ( including background ) . Each object is represented by a latent vector z ( k ) ∈ RM capturing the object ’ s unique appearance and can be thought of as an encoding of common visual properties , such as color , shape , position , and size . For each z ( k ) independently , a broadcast decoder ( Watters et al. , 2019 ) generates pixelwise pairs ( m ( k ) i , µ ( k ) i ) describing the assignment probability and appearance of pixel i for object k. Together , they induce the generative image formation model p ( x|z ) = D∏ i=1 K∑ k=1 m ( k ) i N ( xi ; µ ( k ) i , σ 2 ) , ( 1 ) where z = ( z ( k ) ) k , ∑K k=1m ( k ) i = 1 and σ is the same and fixed for all i and k. The original image pixels can be reconstructed from this probabilistic representation as x̃i = ∑K k=1m ( k ) i µ ( k ) i . Iterative Amortized Inference . Our approach leverages the iterative amortized inference framework ( Marino et al. , 2018 ) , which uses the learning to learn principle ( Andrychowicz et al. , 2016 ) to close the amortization gap ( Cremer et al. , 2017 ) typically observed in traditional variational inference . The need for such an iterative process arises due to the multi-modality of Eq . ( 1 ) , which results in an order invariance and assignment ambiguity in the approximate posterior that standard variational inference can not overcome ( Greff et al. , 2019 ) . The idea of amortized iterative inference is to start with randomly guessed parameters λ ( k ) 1 for the approximate posterior qλ ( z ( k ) 1 |x ) and update this initial estimate through a series of R refinement steps . Each refinement step r ∈ { 1 , . . . , R } first samples a latent representation from qλ to evaluate 1Perspective videos are more complex as objects can occlude one another and change in size over time . the ELBO L and then uses the approximate posterior gradients ∇λL to compute an additive update fφ , producing a new parameter estimate λ ( k ) r+1 : z ( k ) r k∼ qλ ( z ( k ) r |x ) , λ ( k ) r+1 k←− λ ( k ) r + fφ ( a ( k ) , h ( k ) r−1 ) , ( 2 ) where a ( k ) is a function of z ( k ) r , x , ∇λL , and additional inputs ( mirrors definition in Greff et al . ( 2019 ) ) . The function fφ consists of a sequence of convolutional layers and an LSTM . The memory unit takes as input a hidden state h ( k ) r−1 from the previous refinement step .

Summary: The authors extend previous work of Greff et al. on unsupervised, multi-object scene decomposition to incorporate temporal information.  In particular, they apply the LSTM defined for each candidate object not only over inference steps, but also over time. This allows the model to capture temporal cues, such as object motion, to better decompose the scene into objects. In addition, this allows to speed up the inference, since they perform fewer inference steps at each consecutive frame, capitalizing on temporal consistency in videos (LSTM state can be largely refused between consecutive frames, since the appearance doesn't change a lot). Experimentally demonstrate that their approach indeed outperforms prior work on two toy datasets (bouncing balls and CLEVERER), while being more computationally efficient.