Unsupervised Object Learning via Common Fate

1 INTRODUCTION . Machine learning excels if sufficient training data is available that is representative of the task at hand . In recent years , this i.i.d . data paradigm has been shown not only to apply for pattern recognition problems , but also for generative modeling ( Goodfellow et al. , 2014 ) . In practice , the amount of data required to reach a given level of performance will depend on the dimensionality of the data . The generation of high-dimensional images thus either requires huge amounts of data ( Karras et al. , 2020 ) or clever methods that exploit prior information , for instance on multi-scale structure or compositionality ( Razavi et al. , 2019 ) . Imagine we would like to automatically generate realistic images of yearbook group photos . A “ brute force ” approach would be to collect a massive dataset and train a large GAN ( Goodfellow et al. , 2014 ) , hoping that the model will not only learn typical backgrounds , but also the shape of individual humans ( or human faces ) , and arrange them into a group . A more modular approach , in contrast , would be to learn object models ( e.g. , for faces , or humans ) , and learn in which positions and arrangements they appear , as well as typical backgrounds . This approach would be more dataefficient : each training image would contain multiple humans , and we would thus effectively have more data for the object learning task . In addition , the sub-task would be lower-dimensional than the original task . Finally , if we leave the i.i.d . setting ( by , say , having a second task with different group sizes ) , the modular approach would lend itself more readily to knowledge transfer . Object-centric approaches aim to capture this compositionality and have been considerably improved over the past few years ( e.g. , Locatello et al . 2020 ; Engelcke et al . 2021 ) . However , these models tend to be difficult to train and do not yet scale well to visually more complex scenes . In addition , the commonly employed end-to-end learning approaches make it difficult to dissect the causes of these difficulties and what principles may be crucial to facilitate unsupervised object learning . In human vision , the Principle of Common Fate of Gestalt Psychology ( Wertheimer , 2012 ) has been shown to play an important role for object learning ( Spelke , 1990 ) . It posits that elements that are moving together tend to be perceived as one—a perceptual bias that may have evolved to be able to recognize camouflaged predators ( Troscianko et al. , 2009 ) . In our work , we show that this principle can be successfully used also for machine vision by using it in a multi-stage object learning approach ( Fig . 1 ) : First , we use unsupervised motion segmentation to obtain a candidate segmentation of a video frame . Second , we train generative object and background models on this segmentation . While the regions obtained by the motion segmentation are caused by objects moving in 3D , only visible parts can be segmented . To learn the actual objects ( i.e. , the causes ) , a crucial task for the object model is learning to generalize beyond the occlusions present in its input data . To measure success , we provide a dataset including object ground truth . As the last stage , we show that the learned object and background models can be combined into a flexible scene model that allows sampling manipulated novel scenes . Thus , in contrast to existing object-centric models trained end-to-end , our work aims at decomposing object learning into evaluable subproblems and testing the potential of exploiting object motions for building scalable object-centric models that allow for causally meaningful interventions in generation . Summing up , the present work makes the following contributions : • We provide the novel FISHBOWL dataset , positioned between simplistic toy scenarios and real world data , providing ground truth information for evaluating causal scene models . • We show that the Common Fate Principle can be succesfully used for object learning by proposing a multi-stage object learning approach based on this principle . • We demonstrate that the generative object and background models learned in this way can be combined into flexible scene models allowing for controlled out-of-distribution sampling . The dataset with rendering code and models including training code will be made publicly available . 2 RELATED WORK . Modular scene modeling . The idea to individually represent objects in a scene is not new . One approach , motivated by the analysis-by-synthesis paradigm from cognitive science ( Bever & Poeppel , 2010 ) , assumes a detailed specification of the generative process and infers a scene representation by trying to invert this process ( Kulkarni et al. , 2015 ; Wu et al. , 2017 ; Jampani et al. , 2015 ) . Many methods instead aim to also learn the generative process in an unsupervised way , see Greff et al . ( 2020 ) for a recent survey . Several models use a recurrent approach to sequentially decompose a given scene into objects ( Eslami et al. , 2016 ; Stelzner et al. , 2019 ; Kosiorek et al. , 2018 ; Gregor et al. , 2015 ; Mnih et al. , 2014 ; Yuan et al. , 2019 ; Engelcke et al. , 2020 ; Weis et al. , 2020 ; von Kügelgen et al. , 2020 ; Burgess et al. , 2019 ) , or directly learn a partial ordering ( Heess et al. , 2011 ; Le Roux et al. , 2011 ) . This sequential approach has also been extended with spatially-parallel components in ( Dittadi & Winther , 2019 ; Jiang et al. , 2019 ; Lin et al. , 2020b ; Chen et al. , 2020 ) . Other methods infer all object representations in parallel ( Greff et al. , 2017 ; van Steenkiste et al. , 2018 ) , with subsequent iterative refinement ( Greff et al. , 2019 ; Veerapaneni et al. , 2019 ; Locatello et al. , 2020 ; Nanbo et al. , 2020 ) . Whereas most of the above models are trained using a reconstruction objective—usually in a variational framework ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) —several works have also extended GANs ( Goodfellow et al. , 2014 ) to generate scenes in a modular way ( Yang et al. , 2017 ; Turkoglu et al. , 2019 ; Nguyen-Phuoc et al. , 2020 ; Ehrhardt et al. , 2020 ; Niemeyer & Geiger , 2021 ) . Those approaches typically use additional supervision such as ground-truth segmentation or additional views , with Ehrhardt et al . ( 2020 ) ; Niemeyer & Geiger ( 2021 ) being notable exceptions . While most methods can decompose a given scene into its constituent objects , only few are fully-generative in the sense that they can generate novel scenes ( Lin et al. , 2020a ; Ehrhardt et al. , 2020 ; Engelcke et al. , 2020 ; von Kügelgen et al. , 2020 ; Engelcke et al. , 2021 ; Niemeyer & Geiger , 2021 ; Dittadi & Winther , 2019 ) . Our approach differs from previous works in the following three key aspects . First , previous approaches typically train a full scene model in an end-to-end fashion and include architectural biases that lead to the models decomposing scenes into objects . While elegant in principle , those methods have not be shown to scale to more realistic datasets yet . Using a multi-stage approach , as in the present work , enables re-use of existing computer vision methods ( such as unsupervised motion segmentation ) for well-studied sub-tasks and therefore scales more easily to visually complex scenes . Second , while some existing methods make use of temporal information from videos ( Lin et al. , 2020a ; Crawford & Pineau , 2020 ; Kosiorek et al. , 2018 ; Weis et al. , 2020 ) , they do not explicitely use motion signals to discover ( i.e. , segment ) objects . Inspired by the development of the human visual system ( Spelke , 1990 ) , we instead explicitly include this segmentation cue in our approach . Third , most existing fully-generative approaches use a spatial mixture model to compose objects into a scene ( Ehrhardt et al. , 2020 ; Engelcke et al. , 2020 ; 2021 ) . While this simplifies training , it clearly does not match the true , underlying scene generation process . In this work , we instead follow the dead leaves model-approach of von Kügelgen et al . ( 2020 ) and scale it to more complex scenes . Motion Segmentation . We require an unsupervised motion segmentation method that is able to segment multiple object instances . For this , we build on a line of work that tracks points with optical flow , and then performs clustering in the space of the resulting point trajectories ( Brox & Malik , 2010 ; Ochs et al. , 2014 ; Ochs & Brox , 2012 ; Keuper et al. , 2015 ) . Motion segmentation methods that require supervision ( Dave et al. , 2019 ; Xie et al. , 2019 ; Tokmakov et al. , 2017a ; b ) or only perform binary motion segmentation ( Yang et al. , 2021 ; 2019 ; Ranjan et al. , 2019 ) are not applicable in our unsupervised setting . Learning from motion . In the present work , we propose exploiting motion information to decompose a scene into objects , and to learn generative object and scene models . Motion information is believed to be an important cue for the development of the human visual system ( Spelke , 1990 ) and has also been used as a training signal for computer vision ( Pathak et al. , 2017 ; Dorfman et al. , 2013 ; Mahendran et al. , 2018a ; b ) . While similar in spirit , these works however do not address learning of generative object and scene models . 3 THE FISHBOWL DATASET . Several video datasets have been used for object-centric represention learning before ( Weis et al. , 2020 ; Yi et al. , 2019 ; Ehrhardt et al. , 2020 ; Kosiorek et al. , 2018 ) . The ground truth object masks and appearances provided by those datasets , however , only cover the visible parts of the scene . In order to evaluate the capabilities of the object model to infer and represent the full objects even in the presence of occlusions , we propose the novel FISHBOWL dataset positioned between complex real world data and simplistic toy datasets . This dataset consist of 20,000 training and 1,000 validation and test videos recorded from a publicly available WebGL demo of an aquarium,1 each with a resolution of 480×320px and 128 frames . We adapted the rendering to obtain ground truth segmentations of the scene and the ground truth unoccluded background and objects ( Fig . 2 ) . More details regarding the recording setup can be found in the supplement . 4 A MULTI-STAGE APPROACH FOR UNSUPERVISED SCENE MODELLING . We model an image x as a composition of a background ( m0 = 1 , x0 ) and an ordered list of objects ( mi , xi ) , each represented as a binary mask mi and appearance xi . The background and objects are composed into a scene using a simple “ dead leaves ” model , i.e. , the value of each pixel ( u , v ) is determined by the foremost object covering that pixel . We propose a multi-stage approach ( Fig . 1 ) for learning generative models representing objects , backgrounds and scenes in this fashion . 1http : //webglsamples.org/aquarium/aquarium.html , 3-clause BSD license STAGE 1 : MOTION SEGMENTATION—OBTAINING CANDIDATE OBJECTS FROM VIDEOS As a first step , we use unsupervised motion segmentation to obtain candidate segmentations of the input videos . We build on the minimum cost multicut method by Keuper et al . ( 2015 ) , which tracks a subset of the pixels through the video using optical flow and then , inspired by the Common Fate Principle mentioned earlier , clusters the trajectories based on pairwise motion affinities . We use the original implementation of the authors , but replace the postprocessing required to obtain a dense segmentation with a simpler but faster non-parametric watershed algorithm ( Beucher & Meyer , 1993 ) followed by computing spatiotemporal connected components ( Silversmith , 2021 ) . The quality of the motion segmentation critically depends on the quality on the optical flow estimation , so we explore different models for that step . ARFlow ( Liu et al. , 2020 ) is the current state of the art self-supervised optical flow method that combines a common warping based objective with self-supervision using various augmentations . We use the published pretrained models as well as a variant trained on the Fishbowl dataset ( see supplement for details ) . Similar augmentations as used by ARFlow can alternatively be used to synthesize training data for supervised methods , as done for generating the FlyingChairs and FlyingThings datasets ( Dosovitskiy et al. , 2015 ; Mayer et al. , 2016 ) . We experiment with FlowNet 2.0 ( Ilg et al. , 2017 ) and the more recent RAFT ( Teed & Deng , 2020 ) trained on those two datasets . To obtain background masks for training the background model , it is not necessary to differentiate between multiple object instances . We aim for a low rate of foreground pixels being mistaken for background pixels , while background pixels mistaken for foreground are of less concern . Hence , we use an ensemble of different background-foreground segmentation models from the bgslibrary ( Sobral , 2013 ) . Based on early experiments , we used the PAWKS ( St-Charles et al. , 2016 ) , LOBSTER ( St-Charles & Bilodeau , 2014 ) , Σ−∆ estimation ( Manzanera & Richefeu , 2007 ) and static frame differences and label every pixel detected as foreground by either of the methods as a foreground pixel . We found that this rather simple model can faithfully remove the foreground objects in most of the cases . We provide additional details in the appendix . STAGE 2A : OBJECT MODEL—LEARNING TO GENERATE UNOCCLUDED , MASKED OBJECTS Object extraction . We use the bounding boxes of the candidate segmentation to extract object crops from the original videos and rescale them to a common size of 128× 64px . We filter out degenerate masks by ignoring all masks with an area smaller than 64 pixels and only considering bounding boxes with a minimum distance of 16px to the frame boundary . Accordingly , we extract the candidate segmentation masks m0 , . . . , mK for each crop . For notational convenience , we take m0 and m1 to correspond to the background and the object of interest ( i.e. , that whose bounding box was used to create the crop ) , respectively , so that mk with k≥ 2 correspond to the masks of other objects . Task . We use the segmented object crops for training a β -VAE-based generative object model ( Higgins et al. , 2017 ) . Input to the model is the object crop without the segmentation , output is the reconstructed object appearance including the binary object mask . We train the model with the standard β -VAE loss with an adapted reconstruction term including both the appearance and the mask . For an input batch , let c and m0 : K be the ground truth crops with candidate segmentations , and ĉ and m̂ the reconstructed object appearances ( RGB values for each pixel ) and shapes ( foreground probabiliy for each each pixel ) . The reconstruction loss LR for these objects is then the weighted sum of the pixel-wise MSE for the appearance and the pixel-wise binary cross entropy for the mask : LR , appear . = ∑ i ( ∑ u , v m ( i ) 1 ( u , v ) ∥∥∥c ( i ) ( u , v ) − ĉ ( i ) ( u , v ) ∥∥∥2 2 /∑ u , v m ( i ) 1 ( u , v ) ) , LR , mask = ∑ i ( ∑ u , v [ m ( i ) 0 +m ( i ) 1 ] ( u , v ) ·BCE [ m ( i ) 1 ( u , v ) , m̂ ( i ) ( u , v ) ] /∑ u , v [ m ( i ) 0 +m ( i ) 1 ] ( u , v ) ) . As the task for the object model is to only represent the central object in each crop , we restrict the appearance loss to the candidate mask of the object ( m1 ) and the mask loss to the union of the candidates masks of the object and the background ( m0 +m1 ) . Importantly , the reconstruction loss is not evaluated for pixels belonging to other objects according to the candidate masks . Therefore , the object model is not penalized for completing object parts that are occluded by another object . Learning object completion via artificial occlusions . To encourage the model to correctly complete partial objects , we use artificial occlusions as an augmentation during training . Similar to a denoising autoencoder ( Vincent et al. , 2008 ) , we compute the reconstruction loss using the unaugmented object crop . We consider two types of artificial occlusions : first , we use a cutout augmentation ( DeVries & Taylor , 2017 ) placing a variable number of grey rectangles on the input image . As an alternative , we use the candidate segmentation to place another , randomly shifted object from the same input batch onto each crop . Model . We use a β -VAE with 128 latent dimensions . The encoder is a ten layer CNN , the appearance decoder is a corresponding CNN using transposed convolutions ( Dumoulin & Visin , 2018 ) and one additional convolutional decoding layer . We use a second decoder with the same architecture but only a single output channel do decode the object masks . During each epoch , we use crops from two random frames from every object . We train our model for 60 epochs using Adam ( Kingma & Ba , 2015 ) with a learning rate of 10−4 , which we decrease by a factor of 10 after 40 epochs . We chose the optimal hyperparameters for this architecture using grid searches . More details regarding the model architecture and the hyperparameters are provided in the supplement . STAGE 2B : BACKGROUND MODEL—LEARNING TO GENERATE UNOCCLUDED BACKGROUNDS Task . We use an ensemble of background extraction techniques outlined above to estimate background scenes for each frame . We train a β -VAE on these backgrounds using the appearance loss LR , appear . with the inferred background mask , without any additional cutout or object augmentation . Architecture . The β -VAE has the same architecture as the object model , but only uses a single decoder for the background appearance . We do not focus on a detailed reconstruction of background samples and limit the resolution to 96×64px . When sampling scenes , the outputs are upsampled to the original resolution of 480×320px using bilinear interpolation . STAGE 3 : SCENE MODEL—LEARNING TO GENERATE COHERENT SCENES In the final stage , we combine the object and background model into a scene model that allows sampling novel scenes . As the scene model can reuse the decoders from the previous stages , its main task is to model the parameters defining the scene composition such as object counts , locations and dependencies between the background and the object latents . Compared to an end-to-end approach , the complexity of the learning problem is greatly reduced in this setting . It is straightforward to generalize the scene model beyond the training distribution : E.g. , it is easy to sample more objects than observed in the input scenes . We use a scene model following the causal graph depicted in Fig . 3 : First , we sample a background latent zbg which describes global properties of the scene such as its composition and illumination ; zbg is then decoded by the background model into a background image x0 . Conditioned on the background latent , we sequentially sample K tuples ( zappk , z pos k , z scale k ) of latents encoding appearance , position , and scale of object k , respectively ; the number of objects K is sampled conditional on zbg as well . Each appearance latent zappk is decoded by the object model into a masked object ok = ( mi , xi ) , which is subsequently re-scaled by zscalek and placed in the scene at position z pos k according to a dead-leaves model ( i.e. , occluding previously visible pixels at the same location ) . Due to the formulation of the model , we are flexible in specifying the conditional and prior distributions needed to generate samples . A particular simple special case is to sample all latents ( indicated as circles in Fig . 3 ) independently . This can be done by informed prior distributions , or by leveraging the training dataset . In the former case , we sample zbg and all zappk from the standard normal prior of the β -VAE , but reject objects for which the binary entropy of the mask ( averaged across all pixels ) exceeds a threshold ( for figures in the main paper , 100 bits ) . We found that empirically , this entropy threshold can be used to trade diversity of samples for higher-quality samples ( cf . supplement ) . For the coordinates , a uniform prior within the image yields reasonable samples , and scales can be sampled from a uniform distribution between 64× 32px and 192× 96px and at fixed 2 : 1 ratio . Alternatively , all distributions can be fit based on values obtained from the motion segmentation ( object and background latents , distribution of sizes , distribution of coordinates ) . We provide a detailed analysis in the supplement .

Summary: This paper provides a multi-stage solution to unsupervised, frame-wise segmentation in videos. The common fate heuristic is used to provide initial object detections and segmentations. Then VAE based model is used to refine the initial results. A new simulated dataset is proposed.  Results show that the proposed method successfully segments out moving objects and is highly scalable.