Benchmarking Unsupervised Object Representations for Video Sequences

1 INTRODUCTION . Humans understand the world in terms of objects . Being able to decompose our environment into independent objects that can interact with each other is an important prerequisite for reasoning and scene understanding . Similarly , an artificial intelligence system would benefit from the ability to both extract objects and their interactions from video streams , and keep track of them over time . Recently , there has been an increased interest in unsupervised learning of object-centric representations . The key insight of these methods is that the compositionality of visual scenes can be used to both discover ( Eslami et al. , 2016 ; Greff et al. , 2019 ; Burgess et al. , 2019 ) and track objects in videos ( Greff et al. , 2017 ; van Steenkiste et al. , 2018 ; Veerapaneni et al. , 2019 ) without supervision . However , it is currently not well understood how the learned visual representations of different models compare to each other quantitatively , since the models have been developed with different downstream tasks in mind and have not been evaluated using a common protocol . Hence , in this work , we propose a benchmark based on procedurally generated video sequences to test basic perceptual abilities of object-centric video models under various challenging tracking scenarios . An unsupervised object-based video representation should ( 1 ) effectively identify objects as they enter a scene , ( 2 ) accurately segment objects , as well as ( 3 ) maintain a consistent representation for each individual object in a scene over time . These perceptual abilities can be evaluated quantitatively in the established multi-object tracking framework ( Bernardin & Stiefelhagen , 2008 ; Milan et al. , 2016 ) . We propose to utilize this protocol for analyzing the strengths and weaknesses of different object-centric representation learning methods , independent of any specific downstream task , in order to uncover the different inductive biases hidden in their choice of architecture and loss formulation . We therefore compiled a benchmark consisting of three procedurally generated video datasets of varying levels of visual complexity and two generalization tests . Using this benchmark , we quantitatively compared three classes of object-centric models , leading to the following insights : • All of the models have shortcomings handling occlusion , albeit to different extents . • OP3 ( Veerapaneni et al. , 2019 ) performs strongest in terms of quantitative metrics , but exhibits a surprisingly strong dependency on color to separate objects and accumulates false positives when fewer objects than slots are present . • Spatial transformer models , TBA ( He et al. , 2019 ) and SCALOR ( Jiang et al. , 2020 ) , train most efficiently and feature explicit depth reasoning in combination with amodal masks , but are nevertheless outperformed by the simpler model , VIMON , lacking a depth or interaction model , suggesting that the proposed mechanisms may not yet work as intended . We will make our code , data , as well as a public leaderboard of results available . 2 RELATED WORK . Several recent lines of work propose to learn object-centric representations from visual inputs for static and dynamic scenes without explicit supervision . Though their results are promising , methods are currently restricted to handling synthetic datasets and as of yet are unable to scale to complex natural scenes . Furthermore , a systematic quantitative comparison of methods is lacking . Selecting and processing parts of an image via spatial attention has been one prominent approach for this task ( Mnih et al. , 2014 ; Eslami et al. , 2016 ; Kosiorek et al. , 2018 ; Burgess et al. , 2019 ; Yuan et al. , 2019 ; Crawford & Pineau , 2019 ; Locatello et al. , 2020 ) . As an alternative , spatial mixture models decompose scenes by performing image-space clustering of pixels that belong to individual objects ( Greff et al. , 2016 ; 2017 ; 2019 ; van Steenkiste et al. , 2018 ) . While some approaches aim at learning a suitable representation for downstream tasks ( Watters et al. , 2019a ; Veerapaneni et al. , 2019 ) , others target scene generation ( Engelcke et al. , 2019 ; von Kügelgen et al. , 2020 ) . We analyze three classes of models for processing videos , covering three models based on spatial attention and one based on spatial mixture modeling . Spatial attention models with unconstrained latent representations use per-object variational autoencoders , as introduced by Burgess et al . ( 2019 ) . von Kügelgen et al . ( 2020 ) adapts this approach for scene generation . So far , such methods have been designed for static images , but not for videos . We therefore extend MONET ( Burgess et al. , 2019 ) to be able to accumulate evidence over time for tracking , enabling us to include this class of approaches in our evaluation . Recent concurrent work on AlignNet ( Creswell et al. , 2020 ) applies MONET frame-by-frame and tracks objects by subsequently ordering the extracted objects consistently . Spatial attention models with factored latents use an explicit factorization of the latent representation into properties such as position , scale and appearance ( Eslami et al. , 2016 ; Crawford & Pineau , 2019 ) . These methods use spatial transformer networks ( Jaderberg et al. , 2015 ) to render per-object reconstructions from the factored latents ( Kosiorek et al. , 2018 ; He et al. , 2019 ; Jiang et al. , 2020 ) . SQAIR ( Kosiorek et al. , 2018 ) does not perform segmentation , identifying objects only at the bounding-box level . We select Tracking-by-Animation ( TBA ) ( He et al. , 2019 ) and SCALOR ( Jiang et al. , 2020 ) for analyzing spatial transformer methods in our experiments , which explicitly disentangle object shape and appearance , providing access to object masks . Spatial mixture models cluster pixels using a deep neural network trained with expectation maximization ( Greff et al. , 2017 ; van Steenkiste et al. , 2018 ) . IODINE ( Greff et al. , 2019 ) extends these methods with an iterative amortised variational inference procedure ( Marino et al. , 2018 ) , improving segmentation quality . SPACE ( Lin et al. , 2020 ) combines mixture models with spatial attention to improve scalability . To work with video sequences , OP3 ( Veerapaneni et al. , 2019 ) extends IODINE by modeling individual objects ’ dynamics as well as pairwise interactions . We therefore include OP3 in our analysis as a representative spatial mixture model . 3 OBJECT-CENTRIC REPRESENTATION BENCHMARK . To compare the different object-centric representation learning models on their basic perceptual abilities , we use the well-established multi-object tracking ( MOT ) protocol ( Bernardin & Stiefelhagen , 2008 ) . In this section , we describe the datasets and metrics considered in our benchmark , followed by a brief description of the models evaluated . 3.1 DATASETS . Current object-centric models are not capable of modeling complex natural scenes ( Burgess et al. , 2019 ; Greff et al. , 2019 ; Lin et al. , 2020 ) . Hence , we focus on synthetic datasets that resemble those which state-of-the-art models were designed for . Specifically , we evaluate on three synthetic datasets1 ( see Table 1 ) , which cover multiple levels of visual and motion complexity . Synthetic stimuli enable us to precisely generate challenging scenarios in a controllable manner in order to disentangle sources of difficulty and glean insights on what models specifically struggle with . We design different scenarios that test complexities that would occur in natural videos such as partial or complete occlusion as well as similar object appearances . Sprites-MOT ( SpMOT , Table 1 left ) , as proposed by He et al . ( 2019 ) , features simple 2D sprites moving linearly on a black background with objects moving in and out of frame during the sequence . Video-Multi-dSprites ( VMDS , Table 1 right ) is a video dataset we generated based on a colored , multi-object version of the dSprites dataset ( Matthey et al. , 2017 ) . Each video contains one to four sprites that move non-linearly and independently of each other with the possibility of partial or full occlusion . Besides the i.i.d . sampled training , validation and test sets of VMDS , we generate seven additional challenge sets that we use to study specific test situations we observed to be challenging , such as guaranteed occlusion , specific object properties , or out-of-distribution appearance variations . Video Objects Room ( VOR , Table 1 middle ) is a video dataset we generated based on the static Objects Room dataset ( Greff et al. , 2019 ) , which features static objects in a 3D room with a moving camera . For full details on the datasets and their generation , see Appendix B . 3.2 METRICS . Our evaluation protocol follows the multi-object tracking ( MOT ) challenge , a standard and widelyused benchmark for supervised object tracking ( Milan et al. , 2016 ) . The MOT challenge uses the CLEAR MOT metrics ( Bernardin & Stiefelhagen , 2008 ) , which quantitatively evaluate different performance aspects of object detection , tracking and segmentation . To compute these metrics , predictions have to be matched to ground truth . Unlike Bernardin & Stiefelhagen ( 2008 ) and Milan et al . ( 2016 ) , we use binary segmentation masks for this matching instead of bounding boxes , which helps us better understand the models ’ segmentation capabilities . We consider an intersection over union ( IoU ) greater than 0.5 as a match ( Voigtlaender et al. , 2019 ) . The error metrics used are the fraction of Misses ( Miss ) , ID switches ( ID S. ) and False Positives ( FPs ) relative to the number of ground truth masks . In addition , we report the Mean Squared Error ( MSE ) of the reconstructed image outputs summed over image channels and pixels . To quantify the overall number of failures , we use the MOT Accuracy ( MOTA ) , which measures the fraction of all failure cases compared to the total number of objects present in all frames . A model with 100 % MOTA perfectly tracks all objects without any misses , ID switches or false positives . To quantify the segmentation quality , we define MOT Precision ( MOTP ) as the average IoU of segmentation masks of all matches . A model with 100 % MOTP perfectly segments all tracked objects , but does not necessarily track all objects . Further , to quantify detection and tracking performance 1Datasets are available at this https URL . independent of false positives , we measure the Mostly Detected ( MD ) and Mostly Tracked ( MT ) metrics , the fraction of ground truth objects that have been detected and tracked for at least 80 % of their lifespan , respectively . If an ID switch occurs , an object is considered detected but not tracked . For full details regarding the matching process and the evaluation metrics , refer to Appendix A . 3.3 MODELS . We consider three classes of unsupervised object-centric representation learning models : ( 1 ) a spatial attention model with unconstrained latents , VIMON , which is our video extension of MONET ( Burgess et al. , 2019 ) ; ( 2 ) spatial transformer-based attention models , TBA ( He et al. , 2019 ) and SCALOR ( Jiang et al. , 2020 ) ; ( 3 ) a scene mixture model , OP3 ( Veerapaneni et al. , 2019 ) . At a high-level , these methods share a common structure which is illustrated in Fig . 1a . They decompose an image into a fixed number of slots ( Burgess et al. , 2019 ) , each of which contains an embedding zt , k and a mask mt , k of ( ideally ) a single object . These slots are then combined in a decoding step to reconstruct the image . Below , we briefly describe each method . Appendix C provides a detailed explanation in a unified mathematical framework . Video MONet ( VIMON ) is our video extension of MONET ( Burgess et al. , 2019 ) . MONET recurrently decomposes a static scene into slots , using an attention network to sequentially extract attention masks mk ∈ [ 0 , 1 ] H×W of individual objects k. A Variational Autoencoder ( VAE ) ( Kingma & Welling , 2014 ) encodes each slot into a latent representation zk ∈ RL of the corresponding object . We use MONET as a simple frame-by-frame baseline for detection and segmentation that does not employ temporal information . VIMON accumulates evidence about the objects over time to maintain a consistent object-slot assignment throughout the video . This is achieved by ( 1 ) seeding the attention network the predicted mask m̂t , k ∈ [ 0 , 1 ] H×W from the previous time step and ( 2 ) introducing a gated recurrent unit ( GRU ) ( Cho et al. , 2014 ) , which aggregates information over time for each slot separately , enabling it to encode motion information . For full details on MONET and VIMON , as well as ablations to provide context for the design decisions , refer to Appendix C.1 , C.2 and E.3 . Tracking-by-Animation ( TBA ) ( He et al. , 2019 ) is a spatial transformer-based attention model . Frames are encoded by a convolutional feature extractor f before being passed to a recurrent block g called Reprioritized Attentive Tracking ( RAT ) . RAT re-weights slot input features based on their cosine similarity with the slots from the previous time step and outputs latent representations for all K slots in parallel . Each slot latent is further decoded into a mid-level representation yt , k consisting of pose and depth parameters , as well as object appearance and shape templates ( see Fig . 1c ) . For rendering , a Spatial Transformer Network ( STN ) ( Jaderberg et al. , 2015 ) is used with an additional occlusion check based on the depth estimate . TBA is trained on frame reconstruction with an additional penalty for large object sizes to encourage compact bounding boxes . TBA can only process scenes with static backgrounds , as it preprocesses sequences using background subtraction ( Bloisi & Iocchi , 2012 ) . For full details on TBA , refer to Appendix C.3 . Object-centric Perception , Prediction , and Planning ( OP3 ) ( Veerapaneni et al. , 2019 ) extends IODINE ( Greff et al. , 2019 ) to operate on videos . IODINE decomposes an image into objects and represents them independently by starting from an initial guess of the segmentation of the entire frame , and subsequently iteratively refines it using the refinement network f ( Marino et al. , 2018 ) . In each refinement step m , the image is represented by K slots with latent representations zm , k . OP3 applies IODINE to each frame xt to extract latent representations zt , m , k , which are subsequently processed by a dynamics network d ( see Fig . 1e ) , which models both the individual dynamics of each slot k as well as the pairwise interaction between all combinations of slots , aggregating them into a prediction of the posterior parameters for the next time step t+ 1 for each slot k. For full details on IODINE and OP3 , refer to Appendix C.4 and C.5 , respectively . SCALable Object-oriented Representation ( SCALOR ) ( Jiang et al. , 2020 ) is a spatial transformer-based model that factors scenes into background and multiple foreground objects , which are tracked throughout the sequence . Frames are encoded using a convolutional LSTM f . In the proposal-rejection phase , the current frame t is divided into H ×W grid cells . For each grid cell a object latent variable zt , h , w is proposed , that is factored into existence , pose , depth and appearance parameters . Subsequently , proposed objects that significantly overlap with a propagated object are rejected . In the propagation phase , per object GRUs are updated for all objects present in the scene . Additionally , SCALOR has a background module to encode the background and its dynamics . Frame reconstructions are rendered using a background decoder and foreground STNs for object masks and appearance . For full details on SCALOR , refer to Appendix C.6 .

Summary: The paper presents an empirical evaluation of a number of recent models for unsupervised object-based video modelling. Five different models are evaluated on three (partially novel) benchmarks, providing a unifying perspective on the relative performance of these models. Several common issues are identified and highlighted using challenge datasets: The reliance on color as a cue for object segmentation, occlusion, object size, and change in object appearance. The paper concludes with several ideas for alleviating these issues.