Unsupervised Object Learning via Common Fate

1 INTRODUCTION . Machine learning excels if sufficient training data is available that is representative of the task at hand . In recent years , this i.i.d . data paradigm has been shown not only to apply for pattern recognition problems , but also for generative modeling ( Goodfellow et al. , 2014 ) . In practice , the amount of data required to reach a given level of performance will depend on the dimensionality of the data . The generation of high-dimensional images thus either requires huge amounts of data ( Karras et al. , 2020 ) or clever methods that exploit prior information , for instance on multi-scale structure or compositionality ( Razavi et al. , 2019 ) . Imagine we would like to automatically generate realistic images of yearbook group photos . A “ brute force ” approach would be to collect a massive dataset and train a large GAN ( Goodfellow et al. , 2014 ) , hoping that the model will not only learn typical backgrounds , but also the shape of individual humans ( or human faces ) , and arrange them into a group . A more modular approach , in contrast , would be to learn object models ( e.g. , for faces , or humans ) , and learn in which positions and arrangements they appear , as well as typical backgrounds . This approach would be more dataefficient : each training image would contain multiple humans , and we would thus effectively have more data for the object learning task . In addition , the sub-task would be lower-dimensional than the original task . Finally , if we leave the i.i.d . setting ( by , say , having a second task with different group sizes ) , the modular approach would lend itself more readily to knowledge transfer . Object-centric approaches aim to capture this compositionality and have been considerably improved over the past few years ( e.g. , Locatello et al . 2020 ; Engelcke et al . 2021 ) . However , these models tend to be difficult to train and do not yet scale well to visually more complex scenes . In addition , the commonly employed end-to-end learning approaches make it difficult to dissect the causes of these difficulties and what principles may be crucial to facilitate unsupervised object learning . In human vision , the Principle of Common Fate of Gestalt Psychology ( Wertheimer , 2012 ) has been shown to play an important role for object learning ( Spelke , 1990 ) . It posits that elements that are moving together tend to be perceived as one—a perceptual bias that may have evolved to be able to recognize camouflaged predators ( Troscianko et al. , 2009 ) . In our work , we show that this principle can be successfully used also for machine vision by using it in a multi-stage object learning approach ( Fig . 1 ) : First , we use unsupervised motion segmentation to obtain a candidate segmentation of a video frame . Second , we train generative object and background models on this segmentation . While the regions obtained by the motion segmentation are caused by objects moving in 3D , only visible parts can be segmented . To learn the actual objects ( i.e. , the causes ) , a crucial task for the object model is learning to generalize beyond the occlusions present in its input data . To measure success , we provide a dataset including object ground truth . As the last stage , we show that the learned object and background models can be combined into a flexible scene model that allows sampling manipulated novel scenes . Thus , in contrast to existing object-centric models trained end-to-end , our work aims at decomposing object learning into evaluable subproblems and testing the potential of exploiting object motions for building scalable object-centric models that allow for causally meaningful interventions in generation . Summing up , the present work makes the following contributions : • We provide the novel FISHBOWL dataset , positioned between simplistic toy scenarios and real world data , providing ground truth information for evaluating causal scene models . • We show that the Common Fate Principle can be succesfully used for object learning by proposing a multi-stage object learning approach based on this principle . • We demonstrate that the generative object and background models learned in this way can be combined into flexible scene models allowing for controlled out-of-distribution sampling . The dataset with rendering code and models including training code will be made publicly available . 2 RELATED WORK . Modular scene modeling . The idea to individually represent objects in a scene is not new . One approach , motivated by the analysis-by-synthesis paradigm from cognitive science ( Bever & Poeppel , 2010 ) , assumes a detailed specification of the generative process and infers a scene representation by trying to invert this process ( Kulkarni et al. , 2015 ; Wu et al. , 2017 ; Jampani et al. , 2015 ) . Many methods instead aim to also learn the generative process in an unsupervised way , see Greff et al . ( 2020 ) for a recent survey . Several models use a recurrent approach to sequentially decompose a given scene into objects ( Eslami et al. , 2016 ; Stelzner et al. , 2019 ; Kosiorek et al. , 2018 ; Gregor et al. , 2015 ; Mnih et al. , 2014 ; Yuan et al. , 2019 ; Engelcke et al. , 2020 ; Weis et al. , 2020 ; von Kügelgen et al. , 2020 ; Burgess et al. , 2019 ) , or directly learn a partial ordering ( Heess et al. , 2011 ; Le Roux et al. , 2011 ) . This sequential approach has also been extended with spatially-parallel components in ( Dittadi & Winther , 2019 ; Jiang et al. , 2019 ; Lin et al. , 2020b ; Chen et al. , 2020 ) . Other methods infer all object representations in parallel ( Greff et al. , 2017 ; van Steenkiste et al. , 2018 ) , with subsequent iterative refinement ( Greff et al. , 2019 ; Veerapaneni et al. , 2019 ; Locatello et al. , 2020 ; Nanbo et al. , 2020 ) . Whereas most of the above models are trained using a reconstruction objective—usually in a variational framework ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) —several works have also extended GANs ( Goodfellow et al. , 2014 ) to generate scenes in a modular way ( Yang et al. , 2017 ; Turkoglu et al. , 2019 ; Nguyen-Phuoc et al. , 2020 ; Ehrhardt et al. , 2020 ; Niemeyer & Geiger , 2021 ) . Those approaches typically use additional supervision such as ground-truth segmentation or additional views , with Ehrhardt et al . ( 2020 ) ; Niemeyer & Geiger ( 2021 ) being notable exceptions . While most methods can decompose a given scene into its constituent objects , only few are fully-generative in the sense that they can generate novel scenes ( Lin et al. , 2020a ; Ehrhardt et al. , 2020 ; Engelcke et al. , 2020 ; von Kügelgen et al. , 2020 ; Engelcke et al. , 2021 ; Niemeyer & Geiger , 2021 ; Dittadi & Winther , 2019 ) . Our approach differs from previous works in the following three key aspects . First , previous approaches typically train a full scene model in an end-to-end fashion and include architectural biases that lead to the models decomposing scenes into objects . While elegant in principle , those methods have not be shown to scale to more realistic datasets yet . Using a multi-stage approach , as in the present work , enables re-use of existing computer vision methods ( such as unsupervised motion segmentation ) for well-studied sub-tasks and therefore scales more easily to visually complex scenes . Second , while some existing methods make use of temporal information from videos ( Lin et al. , 2020a ; Crawford & Pineau , 2020 ; Kosiorek et al. , 2018 ; Weis et al. , 2020 ) , they do not explicitely use motion signals to discover ( i.e. , segment ) objects . Inspired by the development of the human visual system ( Spelke , 1990 ) , we instead explicitly include this segmentation cue in our approach . Third , most existing fully-generative approaches use a spatial mixture model to compose objects into a scene ( Ehrhardt et al. , 2020 ; Engelcke et al. , 2020 ; 2021 ) . While this simplifies training , it clearly does not match the true , underlying scene generation process . In this work , we instead follow the dead leaves model-approach of von Kügelgen et al . ( 2020 ) and scale it to more complex scenes . Motion Segmentation . We require an unsupervised motion segmentation method that is able to segment multiple object instances . For this , we build on a line of work that tracks points with optical flow , and then performs clustering in the space of the resulting point trajectories ( Brox & Malik , 2010 ; Ochs et al. , 2014 ; Ochs & Brox , 2012 ; Keuper et al. , 2015 ) . Motion segmentation methods that require supervision ( Dave et al. , 2019 ; Xie et al. , 2019 ; Tokmakov et al. , 2017a ; b ) or only perform binary motion segmentation ( Yang et al. , 2021 ; 2019 ; Ranjan et al. , 2019 ) are not applicable in our unsupervised setting . Learning from motion . In the present work , we propose exploiting motion information to decompose a scene into objects , and to learn generative object and scene models . Motion information is believed to be an important cue for the development of the human visual system ( Spelke , 1990 ) and has also been used as a training signal for computer vision ( Pathak et al. , 2017 ; Dorfman et al. , 2013 ; Mahendran et al. , 2018a ; b ) . While similar in spirit , these works however do not address learning of generative object and scene models . 3 THE FISHBOWL DATASET . Several video datasets have been used for object-centric represention learning before ( Weis et al. , 2020 ; Yi et al. , 2019 ; Ehrhardt et al. , 2020 ; Kosiorek et al. , 2018 ) . The ground truth object masks and appearances provided by those datasets , however , only cover the visible parts of the scene . In order to evaluate the capabilities of the object model to infer and represent the full objects even in the presence of occlusions , we propose the novel FISHBOWL dataset positioned between complex real world data and simplistic toy datasets . This dataset consist of 20,000 training and 1,000 validation and test videos recorded from a publicly available WebGL demo of an aquarium,1 each with a resolution of 480×320px and 128 frames . We adapted the rendering to obtain ground truth segmentations of the scene and the ground truth unoccluded background and objects ( Fig . 2 ) . More details regarding the recording setup can be found in the supplement . 4 A MULTI-STAGE APPROACH FOR UNSUPERVISED SCENE MODELLING . We model an image x as a composition of a background ( m0 = 1 , x0 ) and an ordered list of objects ( mi , xi ) , each represented as a binary mask mi and appearance xi . The background and objects are composed into a scene using a simple “ dead leaves ” model , i.e. , the value of each pixel ( u , v ) is determined by the foremost object covering that pixel . We propose a multi-stage approach ( Fig . 1 ) for learning generative models representing objects , backgrounds and scenes in this fashion . 1http : //webglsamples.org/aquarium/aquarium.html , 3-clause BSD license STAGE 1 : MOTION SEGMENTATION—OBTAINING CANDIDATE OBJECTS FROM VIDEOS As a first step , we use unsupervised motion segmentation to obtain candidate segmentations of the input videos . We build on the minimum cost multicut method by Keuper et al . ( 2015 ) , which tracks a subset of the pixels through the video using optical flow and then , inspired by the Common Fate Principle mentioned earlier , clusters the trajectories based on pairwise motion affinities . We use the original implementation of the authors , but replace the postprocessing required to obtain a dense segmentation with a simpler but faster non-parametric watershed algorithm ( Beucher & Meyer , 1993 ) followed by computing spatiotemporal connected components ( Silversmith , 2021 ) . The quality of the motion segmentation critically depends on the quality on the optical flow estimation , so we explore different models for that step . ARFlow ( Liu et al. , 2020 ) is the current state of the art self-supervised optical flow method that combines a common warping based objective with self-supervision using various augmentations . We use the published pretrained models as well as a variant trained on the Fishbowl dataset ( see supplement for details ) . Similar augmentations as used by ARFlow can alternatively be used to synthesize training data for supervised methods , as done for generating the FlyingChairs and FlyingThings datasets ( Dosovitskiy et al. , 2015 ; Mayer et al. , 2016 ) . We experiment with FlowNet 2.0 ( Ilg et al. , 2017 ) and the more recent RAFT ( Teed & Deng , 2020 ) trained on those two datasets . To obtain background masks for training the background model , it is not necessary to differentiate between multiple object instances . We aim for a low rate of foreground pixels being mistaken for background pixels , while background pixels mistaken for foreground are of less concern . Hence , we use an ensemble of different background-foreground segmentation models from the bgslibrary ( Sobral , 2013 ) . Based on early experiments , we used the PAWKS ( St-Charles et al. , 2016 ) , LOBSTER ( St-Charles & Bilodeau , 2014 ) , Σ−∆ estimation ( Manzanera & Richefeu , 2007 ) and static frame differences and label every pixel detected as foreground by either of the methods as a foreground pixel . We found that this rather simple model can faithfully remove the foreground objects in most of the cases . We provide additional details in the appendix . STAGE 2A : OBJECT MODEL—LEARNING TO GENERATE UNOCCLUDED , MASKED OBJECTS Object extraction . We use the bounding boxes of the candidate segmentation to extract object crops from the original videos and rescale them to a common size of 128× 64px . We filter out degenerate masks by ignoring all masks with an area smaller than 64 pixels and only considering bounding boxes with a minimum distance of 16px to the frame boundary . Accordingly , we extract the candidate segmentation masks m0 , . . . , mK for each crop . For notational convenience , we take m0 and m1 to correspond to the background and the object of interest ( i.e. , that whose bounding box was used to create the crop ) , respectively , so that mk with k≥ 2 correspond to the masks of other objects . Task . We use the segmented object crops for training a β -VAE-based generative object model ( Higgins et al. , 2017 ) . Input to the model is the object crop without the segmentation , output is the reconstructed object appearance including the binary object mask . We train the model with the standard β -VAE loss with an adapted reconstruction term including both the appearance and the mask . For an input batch , let c and m0 : K be the ground truth crops with candidate segmentations , and ĉ and m̂ the reconstructed object appearances ( RGB values for each pixel ) and shapes ( foreground probabiliy for each each pixel ) . The reconstruction loss LR for these objects is then the weighted sum of the pixel-wise MSE for the appearance and the pixel-wise binary cross entropy for the mask : LR , appear . = ∑ i ( ∑ u , v m ( i ) 1 ( u , v ) ∥∥∥c ( i ) ( u , v ) − ĉ ( i ) ( u , v ) ∥∥∥2 2 /∑ u , v m ( i ) 1 ( u , v ) ) , LR , mask = ∑ i ( ∑ u , v [ m ( i ) 0 +m ( i ) 1 ] ( u , v ) ·BCE [ m ( i ) 1 ( u , v ) , m̂ ( i ) ( u , v ) ] /∑ u , v [ m ( i ) 0 +m ( i ) 1 ] ( u , v ) ) . As the task for the object model is to only represent the central object in each crop , we restrict the appearance loss to the candidate mask of the object ( m1 ) and the mask loss to the union of the candidates masks of the object and the background ( m0 +m1 ) . Importantly , the reconstruction loss is not evaluated for pixels belonging to other objects according to the candidate masks . Therefore , the object model is not penalized for completing object parts that are occluded by another object . Learning object completion via artificial occlusions . To encourage the model to correctly complete partial objects , we use artificial occlusions as an augmentation during training . Similar to a denoising autoencoder ( Vincent et al. , 2008 ) , we compute the reconstruction loss using the unaugmented object crop . We consider two types of artificial occlusions : first , we use a cutout augmentation ( DeVries & Taylor , 2017 ) placing a variable number of grey rectangles on the input image . As an alternative , we use the candidate segmentation to place another , randomly shifted object from the same input batch onto each crop . Model . We use a β -VAE with 128 latent dimensions . The encoder is a ten layer CNN , the appearance decoder is a corresponding CNN using transposed convolutions ( Dumoulin & Visin , 2018 ) and one additional convolutional decoding layer . We use a second decoder with the same architecture but only a single output channel do decode the object masks . During each epoch , we use crops from two random frames from every object . We train our model for 60 epochs using Adam ( Kingma & Ba , 2015 ) with a learning rate of 10−4 , which we decrease by a factor of 10 after 40 epochs . We chose the optimal hyperparameters for this architecture using grid searches . More details regarding the model architecture and the hyperparameters are provided in the supplement . STAGE 2B : BACKGROUND MODEL—LEARNING TO GENERATE UNOCCLUDED BACKGROUNDS Task . We use an ensemble of background extraction techniques outlined above to estimate background scenes for each frame . We train a β -VAE on these backgrounds using the appearance loss LR , appear . with the inferred background mask , without any additional cutout or object augmentation . Architecture . The β -VAE has the same architecture as the object model , but only uses a single decoder for the background appearance . We do not focus on a detailed reconstruction of background samples and limit the resolution to 96×64px . When sampling scenes , the outputs are upsampled to the original resolution of 480×320px using bilinear interpolation . STAGE 3 : SCENE MODEL—LEARNING TO GENERATE COHERENT SCENES In the final stage , we combine the object and background model into a scene model that allows sampling novel scenes . As the scene model can reuse the decoders from the previous stages , its main task is to model the parameters defining the scene composition such as object counts , locations and dependencies between the background and the object latents . Compared to an end-to-end approach , the complexity of the learning problem is greatly reduced in this setting . It is straightforward to generalize the scene model beyond the training distribution : E.g. , it is easy to sample more objects than observed in the input scenes . We use a scene model following the causal graph depicted in Fig . 3 : First , we sample a background latent zbg which describes global properties of the scene such as its composition and illumination ; zbg is then decoded by the background model into a background image x0 . Conditioned on the background latent , we sequentially sample K tuples ( zappk , z pos k , z scale k ) of latents encoding appearance , position , and scale of object k , respectively ; the number of objects K is sampled conditional on zbg as well . Each appearance latent zappk is decoded by the object model into a masked object ok = ( mi , xi ) , which is subsequently re-scaled by zscalek and placed in the scene at position z pos k according to a dead-leaves model ( i.e. , occluding previously visible pixels at the same location ) . Due to the formulation of the model , we are flexible in specifying the conditional and prior distributions needed to generate samples . A particular simple special case is to sample all latents ( indicated as circles in Fig . 3 ) independently . This can be done by informed prior distributions , or by leveraging the training dataset . In the former case , we sample zbg and all zappk from the standard normal prior of the β -VAE , but reject objects for which the binary entropy of the mask ( averaged across all pixels ) exceeds a threshold ( for figures in the main paper , 100 bits ) . We found that empirically , this entropy threshold can be used to trade diversity of samples for higher-quality samples ( cf . supplement ) . For the coordinates , a uniform prior within the image yields reasonable samples , and scales can be sampled from a uniform distribution between 64× 32px and 192× 96px and at fixed 2 : 1 ratio . Alternatively , all distributions can be fit based on values obtained from the motion segmentation ( object and background latents , distribution of sizes , distribution of coordinates ) . We provide a detailed analysis in the supplement .

Summary: The paper introduces an object-centric generative model for visual scenes. The model decouples the problem into three tasks: 1) modelling the  2D appearance and shape of individual objects with a variational auto encoder; 2) same thing for background; 3) sampling the position, size and appearance of individual objects (i.e. scene composition) conditioned on the background. To my understanding, these three components are trained independently.  Contributions: 1) Decoupling these three tasks allows for a more interpretable representation compared to existing end-to-end methods, which allows for better interactions with users (e.g. change the number of objects or their position)  2) This decoupling potentially makes the third task easier, because it only needs to learn relationships between positions and size given a latent representation (as opposed to learning everything jointly). 3) Training the tasks independently requires a way to obtain "ground-truth" object segmentations - the paper achieves this automatically from videos using motion segmentation with good results