Benchmarking Unsupervised Object Representations for Video Sequences

1 INTRODUCTION . Humans understand the world in terms of objects . Being able to decompose our environment into independent objects that can interact with each other is an important prerequisite for reasoning and scene understanding . Similarly , an artificial intelligence system would benefit from the ability to both extract objects and their interactions from video streams , and keep track of them over time . Recently , there has been an increased interest in unsupervised learning of object-centric representations . The key insight of these methods is that the compositionality of visual scenes can be used to both discover ( Eslami et al. , 2016 ; Greff et al. , 2019 ; Burgess et al. , 2019 ) and track objects in videos ( Greff et al. , 2017 ; van Steenkiste et al. , 2018 ; Veerapaneni et al. , 2019 ) without supervision . However , it is currently not well understood how the learned visual representations of different models compare to each other quantitatively , since the models have been developed with different downstream tasks in mind and have not been evaluated using a common protocol . Hence , in this work , we propose a benchmark based on procedurally generated video sequences to test basic perceptual abilities of object-centric video models under various challenging tracking scenarios . An unsupervised object-based video representation should ( 1 ) effectively identify objects as they enter a scene , ( 2 ) accurately segment objects , as well as ( 3 ) maintain a consistent representation for each individual object in a scene over time . These perceptual abilities can be evaluated quantitatively in the established multi-object tracking framework ( Bernardin & Stiefelhagen , 2008 ; Milan et al. , 2016 ) . We propose to utilize this protocol for analyzing the strengths and weaknesses of different object-centric representation learning methods , independent of any specific downstream task , in order to uncover the different inductive biases hidden in their choice of architecture and loss formulation . We therefore compiled a benchmark consisting of three procedurally generated video datasets of varying levels of visual complexity and two generalization tests . Using this benchmark , we quantitatively compared three classes of object-centric models , leading to the following insights : • All of the models have shortcomings handling occlusion , albeit to different extents . • OP3 ( Veerapaneni et al. , 2019 ) performs strongest in terms of quantitative metrics , but exhibits a surprisingly strong dependency on color to separate objects and accumulates false positives when fewer objects than slots are present . • Spatial transformer models , TBA ( He et al. , 2019 ) and SCALOR ( Jiang et al. , 2020 ) , train most efficiently and feature explicit depth reasoning in combination with amodal masks , but are nevertheless outperformed by the simpler model , VIMON , lacking a depth or interaction model , suggesting that the proposed mechanisms may not yet work as intended . We will make our code , data , as well as a public leaderboard of results available . 2 RELATED WORK . Several recent lines of work propose to learn object-centric representations from visual inputs for static and dynamic scenes without explicit supervision . Though their results are promising , methods are currently restricted to handling synthetic datasets and as of yet are unable to scale to complex natural scenes . Furthermore , a systematic quantitative comparison of methods is lacking . Selecting and processing parts of an image via spatial attention has been one prominent approach for this task ( Mnih et al. , 2014 ; Eslami et al. , 2016 ; Kosiorek et al. , 2018 ; Burgess et al. , 2019 ; Yuan et al. , 2019 ; Crawford & Pineau , 2019 ; Locatello et al. , 2020 ) . As an alternative , spatial mixture models decompose scenes by performing image-space clustering of pixels that belong to individual objects ( Greff et al. , 2016 ; 2017 ; 2019 ; van Steenkiste et al. , 2018 ) . While some approaches aim at learning a suitable representation for downstream tasks ( Watters et al. , 2019a ; Veerapaneni et al. , 2019 ) , others target scene generation ( Engelcke et al. , 2019 ; von Kügelgen et al. , 2020 ) . We analyze three classes of models for processing videos , covering three models based on spatial attention and one based on spatial mixture modeling . Spatial attention models with unconstrained latent representations use per-object variational autoencoders , as introduced by Burgess et al . ( 2019 ) . von Kügelgen et al . ( 2020 ) adapts this approach for scene generation . So far , such methods have been designed for static images , but not for videos . We therefore extend MONET ( Burgess et al. , 2019 ) to be able to accumulate evidence over time for tracking , enabling us to include this class of approaches in our evaluation . Recent concurrent work on AlignNet ( Creswell et al. , 2020 ) applies MONET frame-by-frame and tracks objects by subsequently ordering the extracted objects consistently . Spatial attention models with factored latents use an explicit factorization of the latent representation into properties such as position , scale and appearance ( Eslami et al. , 2016 ; Crawford & Pineau , 2019 ) . These methods use spatial transformer networks ( Jaderberg et al. , 2015 ) to render per-object reconstructions from the factored latents ( Kosiorek et al. , 2018 ; He et al. , 2019 ; Jiang et al. , 2020 ) . SQAIR ( Kosiorek et al. , 2018 ) does not perform segmentation , identifying objects only at the bounding-box level . We select Tracking-by-Animation ( TBA ) ( He et al. , 2019 ) and SCALOR ( Jiang et al. , 2020 ) for analyzing spatial transformer methods in our experiments , which explicitly disentangle object shape and appearance , providing access to object masks . Spatial mixture models cluster pixels using a deep neural network trained with expectation maximization ( Greff et al. , 2017 ; van Steenkiste et al. , 2018 ) . IODINE ( Greff et al. , 2019 ) extends these methods with an iterative amortised variational inference procedure ( Marino et al. , 2018 ) , improving segmentation quality . SPACE ( Lin et al. , 2020 ) combines mixture models with spatial attention to improve scalability . To work with video sequences , OP3 ( Veerapaneni et al. , 2019 ) extends IODINE by modeling individual objects ’ dynamics as well as pairwise interactions . We therefore include OP3 in our analysis as a representative spatial mixture model . 3 OBJECT-CENTRIC REPRESENTATION BENCHMARK . To compare the different object-centric representation learning models on their basic perceptual abilities , we use the well-established multi-object tracking ( MOT ) protocol ( Bernardin & Stiefelhagen , 2008 ) . In this section , we describe the datasets and metrics considered in our benchmark , followed by a brief description of the models evaluated . 3.1 DATASETS . Current object-centric models are not capable of modeling complex natural scenes ( Burgess et al. , 2019 ; Greff et al. , 2019 ; Lin et al. , 2020 ) . Hence , we focus on synthetic datasets that resemble those which state-of-the-art models were designed for . Specifically , we evaluate on three synthetic datasets1 ( see Table 1 ) , which cover multiple levels of visual and motion complexity . Synthetic stimuli enable us to precisely generate challenging scenarios in a controllable manner in order to disentangle sources of difficulty and glean insights on what models specifically struggle with . We design different scenarios that test complexities that would occur in natural videos such as partial or complete occlusion as well as similar object appearances . Sprites-MOT ( SpMOT , Table 1 left ) , as proposed by He et al . ( 2019 ) , features simple 2D sprites moving linearly on a black background with objects moving in and out of frame during the sequence . Video-Multi-dSprites ( VMDS , Table 1 right ) is a video dataset we generated based on a colored , multi-object version of the dSprites dataset ( Matthey et al. , 2017 ) . Each video contains one to four sprites that move non-linearly and independently of each other with the possibility of partial or full occlusion . Besides the i.i.d . sampled training , validation and test sets of VMDS , we generate seven additional challenge sets that we use to study specific test situations we observed to be challenging , such as guaranteed occlusion , specific object properties , or out-of-distribution appearance variations . Video Objects Room ( VOR , Table 1 middle ) is a video dataset we generated based on the static Objects Room dataset ( Greff et al. , 2019 ) , which features static objects in a 3D room with a moving camera . For full details on the datasets and their generation , see Appendix B . 3.2 METRICS . Our evaluation protocol follows the multi-object tracking ( MOT ) challenge , a standard and widelyused benchmark for supervised object tracking ( Milan et al. , 2016 ) . The MOT challenge uses the CLEAR MOT metrics ( Bernardin & Stiefelhagen , 2008 ) , which quantitatively evaluate different performance aspects of object detection , tracking and segmentation . To compute these metrics , predictions have to be matched to ground truth . Unlike Bernardin & Stiefelhagen ( 2008 ) and Milan et al . ( 2016 ) , we use binary segmentation masks for this matching instead of bounding boxes , which helps us better understand the models ’ segmentation capabilities . We consider an intersection over union ( IoU ) greater than 0.5 as a match ( Voigtlaender et al. , 2019 ) . The error metrics used are the fraction of Misses ( Miss ) , ID switches ( ID S. ) and False Positives ( FPs ) relative to the number of ground truth masks . In addition , we report the Mean Squared Error ( MSE ) of the reconstructed image outputs summed over image channels and pixels . To quantify the overall number of failures , we use the MOT Accuracy ( MOTA ) , which measures the fraction of all failure cases compared to the total number of objects present in all frames . A model with 100 % MOTA perfectly tracks all objects without any misses , ID switches or false positives . To quantify the segmentation quality , we define MOT Precision ( MOTP ) as the average IoU of segmentation masks of all matches . A model with 100 % MOTP perfectly segments all tracked objects , but does not necessarily track all objects . Further , to quantify detection and tracking performance 1Datasets are available at this https URL . independent of false positives , we measure the Mostly Detected ( MD ) and Mostly Tracked ( MT ) metrics , the fraction of ground truth objects that have been detected and tracked for at least 80 % of their lifespan , respectively . If an ID switch occurs , an object is considered detected but not tracked . For full details regarding the matching process and the evaluation metrics , refer to Appendix A . 3.3 MODELS . We consider three classes of unsupervised object-centric representation learning models : ( 1 ) a spatial attention model with unconstrained latents , VIMON , which is our video extension of MONET ( Burgess et al. , 2019 ) ; ( 2 ) spatial transformer-based attention models , TBA ( He et al. , 2019 ) and SCALOR ( Jiang et al. , 2020 ) ; ( 3 ) a scene mixture model , OP3 ( Veerapaneni et al. , 2019 ) . At a high-level , these methods share a common structure which is illustrated in Fig . 1a . They decompose an image into a fixed number of slots ( Burgess et al. , 2019 ) , each of which contains an embedding zt , k and a mask mt , k of ( ideally ) a single object . These slots are then combined in a decoding step to reconstruct the image . Below , we briefly describe each method . Appendix C provides a detailed explanation in a unified mathematical framework . Video MONet ( VIMON ) is our video extension of MONET ( Burgess et al. , 2019 ) . MONET recurrently decomposes a static scene into slots , using an attention network to sequentially extract attention masks mk ∈ [ 0 , 1 ] H×W of individual objects k. A Variational Autoencoder ( VAE ) ( Kingma & Welling , 2014 ) encodes each slot into a latent representation zk ∈ RL of the corresponding object . We use MONET as a simple frame-by-frame baseline for detection and segmentation that does not employ temporal information . VIMON accumulates evidence about the objects over time to maintain a consistent object-slot assignment throughout the video . This is achieved by ( 1 ) seeding the attention network the predicted mask m̂t , k ∈ [ 0 , 1 ] H×W from the previous time step and ( 2 ) introducing a gated recurrent unit ( GRU ) ( Cho et al. , 2014 ) , which aggregates information over time for each slot separately , enabling it to encode motion information . For full details on MONET and VIMON , as well as ablations to provide context for the design decisions , refer to Appendix C.1 , C.2 and E.3 . Tracking-by-Animation ( TBA ) ( He et al. , 2019 ) is a spatial transformer-based attention model . Frames are encoded by a convolutional feature extractor f before being passed to a recurrent block g called Reprioritized Attentive Tracking ( RAT ) . RAT re-weights slot input features based on their cosine similarity with the slots from the previous time step and outputs latent representations for all K slots in parallel . Each slot latent is further decoded into a mid-level representation yt , k consisting of pose and depth parameters , as well as object appearance and shape templates ( see Fig . 1c ) . For rendering , a Spatial Transformer Network ( STN ) ( Jaderberg et al. , 2015 ) is used with an additional occlusion check based on the depth estimate . TBA is trained on frame reconstruction with an additional penalty for large object sizes to encourage compact bounding boxes . TBA can only process scenes with static backgrounds , as it preprocesses sequences using background subtraction ( Bloisi & Iocchi , 2012 ) . For full details on TBA , refer to Appendix C.3 . Object-centric Perception , Prediction , and Planning ( OP3 ) ( Veerapaneni et al. , 2019 ) extends IODINE ( Greff et al. , 2019 ) to operate on videos . IODINE decomposes an image into objects and represents them independently by starting from an initial guess of the segmentation of the entire frame , and subsequently iteratively refines it using the refinement network f ( Marino et al. , 2018 ) . In each refinement step m , the image is represented by K slots with latent representations zm , k . OP3 applies IODINE to each frame xt to extract latent representations zt , m , k , which are subsequently processed by a dynamics network d ( see Fig . 1e ) , which models both the individual dynamics of each slot k as well as the pairwise interaction between all combinations of slots , aggregating them into a prediction of the posterior parameters for the next time step t+ 1 for each slot k. For full details on IODINE and OP3 , refer to Appendix C.4 and C.5 , respectively . SCALable Object-oriented Representation ( SCALOR ) ( Jiang et al. , 2020 ) is a spatial transformer-based model that factors scenes into background and multiple foreground objects , which are tracked throughout the sequence . Frames are encoded using a convolutional LSTM f . In the proposal-rejection phase , the current frame t is divided into H ×W grid cells . For each grid cell a object latent variable zt , h , w is proposed , that is factored into existence , pose , depth and appearance parameters . Subsequently , proposed objects that significantly overlap with a propagated object are rejected . In the propagation phase , per object GRUs are updated for all objects present in the scene . Additionally , SCALOR has a background module to encode the background and its dynamics . Frame reconstructions are rendered using a background decoder and foreground STNs for object masks and appearance . For full details on SCALOR , refer to Appendix C.6 .

Summary: The paper proposes a benchmark for the evaluation of unsupervised learning of object-centric representation. The benchmark consists of three datasets, multi-object tracking metrics and of the evaluation of four methods. The proposed dataset consists of three sets of video sequences, procedurally generated, which are either generated from slight variations of existing works (Sprites-MOT) or on the basis of existing datasets (dSpirites, Video Object Room). For evaluation, authors propose to use a slight variation of the protocol of the MOT challenge for evaluation (with the addition of a Mostly Detected measure which does not penalize ID switches). As part of the  paper, they also evaluate and discuss the performances of four object-centric representation models, one of them (Video MONet) being an extension of an existing approach, proposed as part of this paper, and the remaining being state of the art approaches for the task.