Unsupervised Object Learning via Common Fate
Authors: Matthias Tangemann
Steffen Schneider
Julius Von Kügelgen
Francesco Locatello
Peter Vincent Gehler
Thomas Brox
Matthias Kuemmerer
Matthias Bethge
Bernhard Schölkopf
Venue: ICLR 2022 Submitted (2022)
Keywords: object learning
scene modeling
scene generation
causal modeling
causal representation learning
generative modeling
common fate

Abstract: Learning generative object models from unlabelled videos is a long standing problem and is required for causal scene modeling. We decompose this problem into three easier subtasks, and provide candidate solutions for each of them. Inspired by the Common Fate Principle of Gestalt Psychology, we first extract (noisy) masks of moving objects via unsupervised motion segmentation. Second, generative models are trained on the masks of the background and the moving objects, respectively. Third, background and foreground models are combined in a conditional ``dead leaves scene model to sample novel scene configurations where occlusions and depth layering  arise naturally. To evaluate the individual stages, we introduce the Fishbowl dataset positioned between complex real-world scenes and common object-centric benchmarks of simplistic objects. We show that our approach allows learning generative models that generalize beyond the occlusions present in the input videos, and represent scenes in a modular fashion that allows sampling plausible scenes outside the training distribution by permitting, for instance, object numbers or densities not observed in the training set.

Under review as a conference paper at ICLR 2022
UNSUPERVISED OBJECT LEARNING VIA COMMON FATE
Anonymous authors
Paper under double-blind review
ABSTRACT
Learning generative object models from unlabelled videos is long standing problem and required for causal scene modeling. We decompose this problem into three easier subtasks, and provide candidate solutions for each of them. Inspired by the Common Fate Principle of Gestalt Psychology, we first extract (noisy) masks of moving objects via unsupervised motion segmentation. Second, gener- ative models are trained on the masks of the background and the moving objects, respectively. Third, background and foreground models are combined ditional "dead leaves" scene model to sample novel scene configurations where occlusions and depth layering arise naturally. To evaluate the individual stages, we introduce the FISHBOWI. dataset positioned between complex real-world scenes and common object-centric benchmarks of simplistic objects. We show that approach allows learning generative models that generalize beyond the occlusions present in the input vidcos, and represent scenes in modular fashion that allows sampling plausible scenes outside the training distribution by permitting, for instance, object numbers or densities not observed in the training set.
con-
INTRODUCTION
Machine learning excels if sufficient training data is available that is representative of the task at hand. In recent years, this i.i.d. data paradigm has been shown not only apply for pattern recog- nition problems, but also for generative modeling (Goodfellow et al., 2014). In practice, the amount of data required to reach given level of performance will depend on the dimensionality of the data. The generation of high-dimensional images thus either requires huge amounts of data (Karras al., 2020) or clever methods that exploit prior information, for instance multi-scale structure or compositionality (Razavi et al., 2019).
Imagine would like to automatically generate realistic images of yearbook group photos. A "brute force" approach would be to collect massive dataset and train a large GAN (Goodfellow et al., 2014), hoping that the model will not only learn typical backgrounds, but also the shape of individual humans (or human faces), and arrange them into a group. A more modular approach, in contrast, would be to learn object models (e.g., for faces, or humans), and learn in which positions and arrangements they appear, as well as typical backgrounds. This approach would be more data- efficient: each training image would contain multiple humans, and we would thus effectively have more data for the object learning task. In addition, the sub-task would be lower-dimensional than the original task. Finally, if we leave the i.i.d. setting (by, say. having a second task with different group sizes), the modular approach would lend itself more readily knowledge transfer.
Object-centric approaches aim to capture this compositionality and have been considerably im- proved over the past few years (e.g.. Locatello al. 2020; Engelcke al. 2021). However, these models tend to be difficult to train and do not yet scale well to visually more complex scenes. In addi- tion, the commonly employed end-to-end learning approaches make it difficult to dissect the causes of these difficulties and what principles may be crucial to facilitate unsupervised object learning.
In human vision, the Principle of Common Fate of Gestalt Psychology (Wertheimer, 2012) has been shown to play an important role for object learning (Spelke, 1990). It posits that elements that are moving together tend to be perceived as one- a perceptual bias that may have evolved to be able to recognize camouflaged predators (Troscianko et al., 2009).
In our work, we show that this principle can be successfully used also for machine vision by using it in multi-stage object learning approach (Fig. 1): First, we use unsupervised motion segmenta- tion to obtain a candidate segmentation of a video frame. Second, we train generative object and background models this segmentation. While the regions obtained by the motion segmentation

---

Under review as a conference paper at ICLR 2022
Motion Segmentation
Object Model
3| Scane Model
Extracted objecta
Original dallsse
Crppeed furne and masio
Pmort
2B Backkrrondd Model
Compoition Intuicc mode
Extractedd back backkgrouddh
backgrandss
Figure I: We propose a multi-stage approach for learning object-centric generative scene models: (1) Motion segmentation detects moving objects in the input videos. The predicted (noisy) segmentation masks are used to (2A) extract object crops for training a generative object model and (2B) extract backgrounds for training a generative background model. (3) A scene model combines the object and background models to sample novel scenes, permitting interventions variables such as the background and the number and locations of fish.
are caused by objects moving in 3D, only visible parts can be segmented. To learn the actual objects (i.e.. the causes), a crucial task for the object model is learning to generalize beyond the occlusions present in its input data. To measure success, we provide dataset including object ground truth. As the last stage, we show that the learned object and background models can be combined into flexible scene model that allows sampling manipulated novel scenes. Thus, in contrast to exist- ing object-centric models trained end-to-end, our work aims at decomposing object learning into evaluable subproblems and testing the potential of exploiting object motions for building scalable object-centric models that allow for causally meaningful interventions in generation.
Summing up. the present work makes the following contributions:
We provide the novel FISHBOWL. dataset, positioned between simplistic toy scenarios and real world data, providing ground truth information for evaluating causal scene models.
We show that the Common Fate Principle can be succesfully used for object learning by proposing a multi-stage object learning approach based on this principle.
We demonstrate that the generative object and background models learned in this way can be combined into flexible scene models allowing for controlled outtrf-distriuution sampling.
The dataset with rendering code and models including training code will be made publicly available.
2 RELATED WORK
Modular scene modeling. The idea to individually represent objects in scene is not new. One approach, motivated by the analysis-by-synteeiis paradigm from cognitive science (Bever & Poeppel, 2010), assumes a detailed specification of the generative process and infers a scene representation by trying to invert this process (Kulkarni et al., 2015; Wu al., 2017: Jampani et al., 2015). Many methods instead aim to also learn the generative process in an unsupervised way, see Greff et al. (2020) for a recent survey. Several models use recurrent approach to sequentially decompose a given scene into objects (Eslami et al., 2016; Stelzner et al., 2019; Kosiorek al., 2018; Gregor et al., 2015; Mnih et al., 2014; Yuan et al., 2019; Engelcke et al., 2020: Weis et al., 2020; von Kügelgen et al., 2020; Burgess et al., 2019), or directly learn partial ordering (Heess et al., 2011; Le Roux et al., 2011). This sequential approach has also been extended with apatially-paralle components in (Dittadi& & Winther, 2019; Jiang et al., 2019; Lin et al., 2020b; Chen et al.. 2020). Other methods infer all object representations in parallel (Greff et al., 2017; van Steenkiste et al., 2018), with subsequent iterative refinement (Greff et al., 2019; Veerapaneni et al., 2019; Locatello et al., 2020; Nanbo et al., 2020). Whereas most of the above models are trained using a reconstruction objective-usually in a variational framework (Kingma & Welling, 2014; Rezende et al., 2014)-severtt several works have also extended GANs (Goodfellow et al., 2014) to generate scenes in a modular way (Yang et al., 2017: Turkoglu et al., 2019: Nguyen- Phuoc et al., 2020; Ehrhardt et al., 2020; Niemeyer & Geiger, 2021). Those approaches typically use additional supervision such as ground-truth segmentation or additional views. with Ehrhardt et al. (2020); Niemeyer & Geiger (2021) being notable exceptions. While most methods can decompose a given scene into its constituent objects, only few are fully-generative in the sense that they can generate novel scenes (Lin et al., 2020a; Ehrhardt et al., 2020; Engelcke et al., 2020; von Kügelgenn et al., 2020; Engelcke al., 2021; Niemeyer & Geiger, 2021; Dittadi & Winther, 2019).
2

---

Under review as a conference paper at ICLR 2022
Frame
Segmentation
Ooject with musk
Object N with mask
Background
Figure 2: The FISHBOWW dataset: each video contains 128 frames with ground truth segmentation; renderings and masks (without occlusion) of every fish and background are provided in the validation and test sets.
Our approach differs from previous works in the following three key aspects. First, previous ap- proaches typically train a full scene model in an end-to-end fashion and include architectural biases that lead to the models decomposing scenes into objects. While elegant in principle, those methods have not be shown to scale to more realistic datasets yet. Using multi-stage approach, as in the present work, enables re-use of existing computer vision methods (such as unsupervised motion seg- mentation) for well-studied sub-tasks and therefore scales more easily to visually complex scenes. Second, while some existing methods make use of temporal information from videos (Lin et al., 2020a; Crawford & Pineau, 2020; Kosiorek al., 2018; Weis et al., 2020), they do not explicitely use motion signals to discover (i.e., segment) objects. Inspired by the development of the human visual system (Spelke, 1990), instead explicitly include this segmentation cue in our approach. Third, most existing fully-gened fully-generative approaches use a spatial mixture model to compose objects into scene (Ehrhardt et al., 2020; Engelcke et al., 2020; 2021). While this simplifies training, it clearly does not match the true, underlying scene generation process. this work, we instead follow the dead leaves model-approach of von Küggllgen et al. (2020) and scale it to more complex scenes.
Motion Segmentation. We require an unsupervised motion segmentation method that is able to segment multiple object instances. For this, we build on a line of work that tracks points with optical flow, and then performs clustering in the space of the resulting point trajectories (Brox & Malik, 2010; Ochs et al., 2014; Ochs & Brox, 2012; Keuper et al., 2015). Motion segmentation methods that require supervision (Dave et al., 2019; Xie et al., 2019; Tokmakov al., 201 7a;b) or only perform binary motion segmentation (Yang et al., 2021; 2019; Ranjan et al., 2019) are not applicable in our unsupervised setting.
Learning from motion. In the present work, we propose exploiting motion information to decom- pose scene into objects, and to learn generative object and scene models. Motion information is believed to be an important cue for the development of the human visual system (Spelke, 1990) and has also been used as a training signal for computer vision (Pathak et al., 2017; Dorfman et al., 2013; Mahendran et al., 2018a; While similar in spirit, these works however not address learning of generative object and scene models.
THE FISHBOWI DATASET
Several video datasets have been used for object-centric represention learning before (Weis et al., 2020: Yi et al., 2019; Ehrhardt et al., 2020; Kosiorek et al., 2018). The ground truth object masks and appearances provided by those datasets, however, only cover the visible parts of the scene. In order to evaluate the capabilities of the object model to infer and represent the full objects even in the presence of occlusions, we propose the novel FISHBOWL dataset positioned between com- plex real world data and simplistic toy datasets. This dataset consist of 20, 000 training and 1,000 validation and test videos recorded from a publicly available WebGL demo of an aquarium," cach with resolution of 480 x 320px and 128 frames. We adapted the rendering to obtain ground truth segmentations of the scene and the ground truth unoccluded background and objects (Fig. 2). More details regarding the recording setup can be found in the supplement.
4 A MULTI-STAGE APPROACH FOR UNSUPERVISER SCENE MODELLING
We model an image x as a composition of a background (my- = 1, X²) and an ordered list of objects xm), each represented as binary mask m, and appearance x, The background and objects are composed into scene using a simple "dead leaves" model, i.c., the value of each pixel v) is determined by the foremost object covering that pixel. We propose a multi-stage approach (Fig, 1) for learning generative models representing objects, backgrounds and scenes in this fashion.
http://wwctgkaaakpsoooo/aaarrnru 3-clause BSD license
3

---

Under review as a conference paper at ICLR 2022
STAGE 1: MOTION SEGMENTATION OBBAINING CANDIDATE OBJECTS FROM VIDEOS
As a first step, we use unsupervised motion segmentation to obtain candidate segmentations of the input videos. We build on the minimum cost multicut method by Keuper et al. (2015), which tracks a subset of the pixels through the video using optical flow and then, inspired by the Common Fate Principle mentioned earlier, clusters the trajectories based on pairwise motion affinities. We use the original implementation of the authors, but replace the posiprocessing required to obtain dense segmentation with simpler but faster non-parametric watershed algorithm (Beucher & Meyer, 1993) followed by computing spatiotemporal connected components (Silversmith, 2021).
The quality of the motion segmentation critically depends on the quality on the optical flow estima- tion, so we explore different models for that step. ARFlow (Liu ct al., 2020) is the current state of the art self-supervised optical flow method that combines common warping based objective with self-supervision using various augmentations. We use the published pretrained models as well as a variant trained on the Fishbowl dataset (see supplement for details). Similar augmentations as used by ARFlow can alternatively be used to synthesize training data for supervised methods, as done for generating the FlyingChairs and PlyingThings datasets (Dosovitskiy et al., 2015; Mayer et al., 2016). We experiment with FlowNet 2.0 (Il et al., 2017) and the more recent RAFT (Teed & Deng, 2020) trained on those two datasets.
To obtain background masks for training the background model, it is not necessary differentiate between multiple object instances. We aim for a low rate of foreground pixels being mistaken for background pixels, while background pixels mistaken for foreground are of less concern. Hence, we use an ensemble of different bral, 2013). Based on early experiments, we used the PAWKS (St-Charles et al., 2016), LOBSTER (St-Charles & Bilodeau, 2014), A estimation (Manzanera & Richefeu, 2007) and static frame differences and label every pixel detected as foreground by either of the methods foreground pixel. We found that this rather simple model can faithfully remove the foreground objects in most of the cases. We provide additional details in the appendix.
segmentation models from the bgslibrary (So-
STAGE 2A: OBJECT HEDDE--LARNING TO GENERATE UNOCCLUDED, MASKED OBJECTS
Object extraction. We use the bounding boxes of the candidate segmentation to extract object crops from the original videos and rescale them a common size of 128 64px. We filter out degenerate masks by ignoring all masks with an area smaller than 64 pixels and only considering bounding boxes with minimum distance of 16px to the frame boundary. Accordingly, we extract the candidate segmentation masks mo...., mg for each crop. For notational convenience, we take mo and mj to correspond to the background and the object of interest (i.e., that whose bounding box was used to create the crop), respectively, so that m with k 2 2 correspond to the masks of other objects.
Task. We use the segmented object crops for training B-VAF-based generative object model (Higgins et al., 2017). Input to the model the object crop without the segmentation, output is the reconstructed object appearance including the binary object mask. We train the model with the standard B VAV loss with an adapted reconstruction term including both the appearance and the mask. For an input batch, let and mo.x be the ground truth crops with candidate segmentations, and è and i the reconstructed object appearances (RGB values for each pixel) and shapes (Foreground probabiliy for each each pixel). The reconstruction loss Lx for these objects is then the weighted sum of the pixel-wise MSE for the appearance and the pixel-wise binary cross entropy for the mask:
LRaappear. =
Σ
Σ
m (u,
v)
|c (u.v) (u.
Σ mmma (mf((..v))
3 + -m mm(,(,))). Σ Σ m + -m (H, BCE [m] (u, v), ( (u, v)/ Σ mm
LR.musk =
As the task for the object model is to only represent the central object in each crop, we restrict the appearance loss to the candidate mask of the object (m;) and the mask loss to the union of the candidates masks of the object and the background (m, + +m;). Importantly, the reconstruction loss is not evaluated for pixels belonging to other objects according to the candidate masks. Therefore, the object model is not penalized for completing object parts that are occluded by another object.
4

---

Under review as a conference paper at ICLR 2022
Table
Segmentation performance of the unsupervised motion segmentation (Keuper et al., 2015) for different optical flow estimators.
X
Optical flow
IoU
Recall
Estimator
Training data
Background
Objects
@00.0
@0.5
ARFlow
KITTI
0.874
0.246
0.828
0.199
ARFlow
Sintel
0.890
0.243
0.809
0.213
ARFlow
Fishbowl
0.873
0.248
0.842
0.204
Causal graph for our scene model; circles and diamonds denote random and deterministic quantities.
Figure 3:
RAFT
Flying
0.930
0.318
0.663
0.351
FlowNet 2
Flying Things
0.934
0.365
0.674
0.416
Learning object completion via artificial occlusions. To encourage the model to correctly complete partial objects, use artificial occlusions
augmentation during training. Similar to a denoising autoencoder (Vincent et al., 2008), we compute the reconstruction loss using the unaugmented object crop, We consider two types of artificial occlusions: first, we use cutour augmentation (DeVries & Taylor, 2017) placing a variable number of grey rectangles on the input image. As an alternative, object from the same input batch onto each crop.
use the candidate segmentation to place another, randomly shifted
Model. We use a B- B-VAE with 128 latent dimensions. The encoder is a ten layer CNN. the corresponding CNN using transposed convolutions (Dumoulin & Visin,
appearance decoder 2018) and one additional convolutional decoding layer. We use second decoder with the same architecture but only a single output channel do decode the object masks. During each epoch, we use crops from two random frames from every object. We train our model for 60 epochs using Adam (Kingma Ba, 2015) with learning rate of 10 which we decrease by a factor of 10 after 40 epochs. We chose the optimal hyperparameters for this architecture using grid searches. More details regarding the model architecture and the hyperparameters are provided in the supplement.
STAGE 2B: BACKGROUND MODEI-LEARRING TO GENERATE UNOCCLUDED BACKGROUNDS
Task. We use an ensemble of background extraction techniques outlined above to estimate back- ground scenes for each frame. We train a B- VAE on these backgrounds using the appearance loss LR.uppaur. with the inferred background mask, without any additional cutout or object augmentation.
Architecture. The B-VAR has the same architecture as the object model, but only uses a single decoder for the background appearance. We do not focus on detailed reconstruction of background samples and limit the resolution 96 64px. When sampling scenes, the outputs upsampled to the original resolution of 480 x 320px using bilinear interpolation.
STAGE 3: SCENE MODEL-LLARNING TO GENERATE COHERENT SCENES
In the final stage, we combine the object and background model into scene model that allows sampling scenes. task to model the parameters defining the scene composition such object counts, locations and dependencies between the background and the object latents. Compared to an end-to-end approach, the complexity of the learning problem is greatly reduced in this setting. It is straightforward to generalize the scene model beyond the training distribution: E.g., it is easy to sample more objects than observed in the input scenes.
scenes. As the scene model can reuse the decoders from the previous stages, its main
We use a scene model following the causal graph depicted in Fig. 3: First, we sample a background latent zh which describes global properties of the scene such as its composition and illumination; h is then decoded by the background model into background image Xo. Conditioned the background latent, we sequentially sample K tuples (z _ *x position, and scale of object k, respectively; the number of objects K is sampled conditional on zr8 as well. Each appearance latent aPP is decoded by the object model into a masked object 02 = (m, x,), which is subsequently re-scaled by zicale and placed in the scene at position z!!" according to a dead-leaves model (i.c., occluding previously visible pixels the same location).
pos scale) of latents encoding appearance,
Due to the formulation of the model, we are flexible in specifying the conditional and prior distribu- tions needed to generate samples. A particular simple special case is sample all latents (indicated as circles in Fig. 3) independently. This can be done by informed prior distributions, or by lever- aging the training dataset. the former case, we sample zº and all ?PP from the standard normal prior of the B-VAE, but reject objects for which the binary entropy of the mask (averaged across all
5

---

Under review as a conference paper at ICLR 2022
pixels) exceeds threshold (for figures in the main paper, 100 bits). We found that empirically, this entropy threshold can be used to trade diversity of samples for higher-quality samples (cf. supple- ment). For the coordinates, uniform prior within the image yields reasonable samples, and scales can be sampled from a uniform distribution between 64 32px and 192 x 96px and at fixed ratio. Alternatively, all distributions can be fit based on values obtained from the motion segmenta- tion (object and background latents, distribution of sizes, distribution of coordinates). We provide a detailed analysis in the supplement.
5 EXPERIMENTS
Motion Segmentation. We quantify the quality of the motion segmentation approach to understand which and how many enrors are propagated to later stages. Motion segmentation was scored as follows: For every frame, we matched the predicted and ground truth masks using the Hungarian algorithm (Kuhn, 1955) with the pairwise IoU (Intersection over Union) as matching cost. The frame-wise IoU scores were then aggregated as in the DAVIS-2017 benchmark (Pont-Tuset et al., 2017) by first averaging IoUs over frames for each ground truth object, and then averaging over objects. In Tab. I we report the segmentation performance separately for the background and the foreground objects.
The best performance is reached when using optical flow predicted by the FlowNet 2 model, thus we're using this model for all subsequent experiments. Our setting seems be especially difficult for the ARFlow model. Different from the other networks, this model does not easily transfer from unrelated training settings and doesn't 't improve much when training the Fishbowl data directly using the default hyperparameters. It might however be possible to improve the model by adapting the training setup to our dataset more closely.
Overall, the evaluation reveals two types of errors: (1) even the best model variant only detected 67. 47.4% of all objects, i.e., one third of the objects are missed and get included in the background masks. (2) The recall decreases when increasing the IoU threshold to 0.5, i.e., the predicted masks are often not precise. While the motion segmentation therefore induces a sort of label noise, it has to be considered that not all errors are problematic for the motion segmentation. In particular, this holds for all objects that have not been detected at all and unsystematic errors.
Object model. To evaluate the capability of the object model reconstruct unoccluded objects from occluded inputs, we extract object crops from the validation set using the bounding boxes of the ground truth unoccluded masks. For those crops, compare the masks and appearances reconstructed by the model to the unoccluded ground truth (Fig. 2) using IoU and mean average error (MAE), respectively. As we are only interested in the reconstruction error within the object masks, we evaluate the MAE only on the intersection of the predicted and ground truth mask. We consider the ground truth unoccluded segmentation masks as our baseline: they correspond to a model that perfectly segments all visible object parts, but does not complete partial objects. We train variants of the object model using all augmentation strategies described above. For comparison, we also train each model variant using the ground truth occluded segmentation masks so that errors propagated from the motion segmentation can be differentiated from errors inherent to the object model.
As the results in Tab. show, using another object from the input batch as artificial occlusions results in the best IoU of 0.827. When considering only those objects that are occluded at least by 50%, the IoU drops to 0.705. This performance is substantially greater than the baseline (0.27), which indi- cates that the model learns to complete partial objects even for those hard cases. In terms of the ap- pearance error, using the cutout augmentation performs best. Interestingly though, for both the mask and appearance error, training without any augmentation at all reduces the performance only slightly.
When training on the ground truth segmentation instead of the motion segmentation, the error in terms of the loU reduces by roughly one third. Performance gains can therefore be expected from improving the generative object model and, to a lesser extent, the motion segmentation. Overall, the object model seems to be quite robust to the label noise induced by the motion segmentation.
We visualize reconstructions and samples from the object model in Fig. 4. In agreement with the quantitative results before, there is a visible quality difference between the model trained on motion segmentation and ground truth segmentation, respectively. In some cases, parts of the fish such as the tail fin is missing and in other cases, the mask covers areas not properly covered by the appearance. In particular this holds for the hard cases with substantial occlusions. Moreover, the samples from the model trained motion segmentation masks seems to contain fewer sharks (a rare class). The

---

Under review as a conference paper at ICLR 2022
Table 2: Comparison of the reconstructions from the object model with the ground truth unoceluded objects. The reconstructed masks are evaluated using the intersection over union (IoU), the appearance evaluated using mean average error (MAE) of the RGB values (0-255) on the intersection of the ground truth and predicted masks. Additionally, we report those metrics for hard samples which most half of object visible (IoU@0.5 and MAE@0.5). We consider model variants that use different artificial occlusions as augmentations during training. Furthermore, we compare the performance of each model variant when trained on the motion segmentation and the ground truth masks, respectively. All results are averaged over independent runs with 3 different seeds and reported with the standard error of the mean. As baseline we report the IoU of the ground truth occluded masks with the ground truth unoccluded masks.
Training data
Augmentation
IoU1 1
MAE 4
loU@0.5
MAE@0.5
None
0.820:100002
13.0=0.047
0.669=0.002
24.3110.337
Motion Segmentation
Cutout
0.822:4:0001
13.0=0.032
0.677=0.002
24,0=10.017
Other object
0.827:10.001
14.8=0.095
0.705=10001
23.41000499
None
0.885-4-0001
12.4=0.144
0.738=0.002
17,5++02100
Ground Truth
Cutout
0.887:10.001
1..3=00035
0.743=00003
17.3:00087
Segmentution
Other object
0.883++0.001
14.1+0.250
0.002
17.7+0.197
Baseline
0.271
Mation Segmentation
Graund Truth Segmentation
Samples Reconstruction
Samples Reconstrruction
Figure 4: Qualitative results from the object model using another object as augmentation. Left: Object model trained on the masks predicted by motion segmentation. Right: Object model trained using ground truth seg- mentations. Top: Reconstructions ements that are occluded by 0.5 0.05. Bottom: objects sampled from the respective model. The shown reconstructions and samples are not cherrypicked and do not use entropy filtering as the scene model.
validation set elements. Middle: Reconstructions
validation set el-
majority of the reconstructions and samples however looks convincing, with the model capturing the relevant properties of the object masks and appearances.
Background model. We train two variants of the background model using the foreground/back- ground segmentation and the ground truth segmentation masks, respectively. We show samples from the former variants as part of the scene model in Fig. 5, and provide additional samples in the supplement. We deliberately chose to not exhaustively tune the background model and defer improvements (which are conceivable given the recent advances in high resolution image modeling with VAEs) to future work. Within the scope of this work, limit ourselves to the constrained resolution resulting in blurry samples. They still reflect the main variations in the dataset, like the overall illumination and background color.
Scene model. In Fig. 5 we visualize samples from the scene model following the scene statis- tics from the training dataset. As before, we consider two variants of the scene model using the background and object models trained on the motion segmentation and ground truth segmentation, respectively. The samples from both models nicely resemble the scenes from the training dataset. However, in both cases the errors from the object and the background model, as discussed before, are also observable in the generated scenes.
A particular strength of our modular approach is that it allows to separately intervene on causally independent mechanisms (Pearl, 2009; Schölkopf et al., 2021) of the scene generation. In Fig. 6 we show such interventions their positions and sizes are all represented explicitly by the scene model so that we can manipulate those scene properties separately. Moreover, the background and the objects are represented
sample from the unsupervised scene model. The number of objects,
7

---

Under review as a conference paper at ICLR 2022
Original Image
Scene re-sempling with matching latents
Scene re- se--alinngg with new latents
MoSeg
Figure 5: Samples from training dataset comparison the scene model using the background object models trained on the motion segmentation and ground truth segmentation, respectively.
Fish of
Beckground
Locatin
Figure 6: Interventions the scene properties of sample from the unsupervised scene model. Top bottom: (i) Varying number objects, (ii) varying object identities, (iii) varying the scales individual objects, (iv) changing backgrounds while keeping objects constant, (v) re-sampling locations objects.
separately by their respective latent codes so that scene components individually.
exchange and manipulate each of those
Modularity. A downside of end-to-end models are non-trivial dependencies between individual components, making harder to train and debug such models. In contrast, our modular approach allows to exchange individual components in the object learning pipeline. In Figure 7, we show proof-of-concept for using a GAN (Mescheder et al., 2018) as part of hybrid object model (image modeling: GAN, mask modeling: VAE) within our framework.
Comparison to prior work. In the appendix. end-to-end object-centric models (Lin et al., 2020b; Engelcke et al., 2021). ). Both models struggle to scale to the FISHBOWL dataset, highlighting the advantage of multi-stage approach compared to end-to-end scene models. Moreover, we apply our method to the RealTraffic dataset of real world videos captured by traffic camera, showing that approach transfers well to other datasets.
further compare our approach with two recent
DISCUSSION
The fields of machine learning and causal modeling have recently begun to cross- cross-pollinate. Causal- ity offers principled framework to think about classes of distributions related to each other by domain shifts and interventions, and independent/modular components (e.g.. objects) that are in- variant across such changes. Vice versa, machine learning has something to offer to causality when it comes to learning causal variables and representations trriitaiiiiiistdddiiinnllllll research in causal discovery and inference built on the assumption that those variables be given a priori. Much like the shift from classic AI towards machine learning included shift from processing human-defined symbols towards automatically learning features and statistical representations, causal representation learn- ing aims to learn interventional representations from raw data (Schölkopf et al., 2021). It is in this
8

---

Under review as a conference paper at ICLR 2022
Object Samples
Scene Samples
Traversals / Interventions
The modularity of our multi-stage approach allows to easily exchange individual compo- nents. Here, we replace the image-mone modeling part of the object model with a GAN instead of the previously used VAE. As in our main model, this allows meaningful traversals/mteervnttonss since objects and their positions/sizes are presented as individual entities.
Figure 7:
sense that we like to think of our scene model as a causal generative model, even though it is fairly straightforward from the causal point of view.
We propose multi-stage approach for learning generative object models from unlabelled videos. Inspired by the Gestalt Principle of Common Fate, we use candidate segmentations obtained from unsupervised optic flow to weakly supervise generative object and background models, and combine them into a scene model. The scene model causal/mechanistic in the sense that it (1) learns complete objects (although occlusions are abundant in the training data)² and (2) properly handles partial depth orderings in the scene, the latter through the dead leaves approach- as opposed to the computationally simpler additive mixture approach which is common place in end-to-end scene models, but contradicts the physics of scene generation. Moreover, while many existing models only decompose and represent scenes, (3) our model can generate plausible novel and out-oo-distributioo scenes (c.g., higher fish density, different sizes). This showcases the challenges faced by end-to-end learning when not endowed with sufficient prior structure and inductive biases.
While learning object models part of an end-to-end trainable object centric scene model might be more elegant in principle, it has not been shown to scale to more realistic data yet. We instead chose a multi- stage approach that allows developing and analyzing the modeling stages individually. This approach allowed us conclude that exploiting motion information for object discovery, as done in the motion segmentation stage, is valuable cue that greatly simplifies object learning. Also, since the scene model uses the component models as modelling "atoms", this naturally permits intervening on the parameters defining the scene composition without side-effects, as stipulated by the principle of Independent Causal Mechanisms (Parascandolo al., 2018; Peters et al., 2017).
We evaluated the modelling stages on the novel FISHBOWL. dataset, consisting of short videos recorded from synthetic 3D aquarium. Notably, we found that the generative object model proves to be fairly robust to errors in the motion segmentation. We expect further performance gains to be achievable by improving the visual fidelity of the generative object and background models (Razavi et al., 2019; Vahdat & Kautz, 2020); however, this was not the motivation for the present work.
We see several ways extend our model in the future, beyond improving the implementations of the individual stages by taking advantage progress in computer vision. One limitation of using mo- tion segmentation is that we can only decompose scenes of moving objects. This could be addressed by integrating object models trained using our approach into an end-to-end scene model, e.g., the work of Eslami et al. (2016) and its successors. When using a learning based motion segmenta- tion approach, there could be a feedback circle with both models improving each other as training progresses and this way closing the observed gap to using ground truth segmentations. Finally, our object model does currently not capture object motions. Given that the motion segmentation yields temporally consistent segmentation masks, this could, in principle, well be included in our approach.
since inpainting takes place the level of masks/shapes, rather than only for pixels as in most prior work.

---

Under review as a conference paper at ICLR 2022
Ethics statement. While there is some way go when it comes to the generation of photorealistic scenes, we note that being able to automatically generate coherent scenes, including the interven- tional control that our approach pioneers, could be misused to synthesize or modify scenes, rein- foreing the need develop methods that allow to reliably detect manipulated content. On the other hand, the interpretability that comes with the modularity of our method can yield insights into scene perception, and could therefore facilitate better explanations for model decisions.
Reproducibility statement. We took several steps to make our work reproducible: For the dataset generation, we modified the Aquarium simulation on which our dataset based, in order to make the rendering fully deterministic. The configuration files describing the recording settings for all samples will be published with the generation code along the dataset, which allows re-recording all videos. Moreover, the code for training our models is deterministic and will be made publicly available.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. arXiv: 1607.0640, July 2016.
David Bau, Jan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Seeing what gan cannot generate. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4502-4511, 2019.
S. Beucher and F. Meyer. The Morphological Approach to Segmentation: The Watershed Transfor- mation. In E. R. Dougherty (cd.), Mathematical Morphology in Image Processing, pp. 433-481. 1. Marcel Dekker, New York, 1993.
Thomas G. Bever and David Poeppel. Analysis by synthesis: A (re-Jemerging program of research for language and vision. Biolinguistics, 4(2):174-200 2010.
Thomas Brox and Jitendra Malik. Object Segmentation by Long Term Analysis of Point Trajecto- ries. In Kostas Daniilidis, Petros Maragos, and Nikos Paragios (eds.). Computer Vision ECCV 2010, Lecture Notes in Computer Science, pp. 282-295,, Berlin, Heidelberg, September 2010. Springer.
Christopher P. Burgess, Loic Matthey Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised Scene Decomposition and Repre- sentation. arXiv::901.1 11390, January 2019.
Chang Chen, Fei Deng, and Sungjin Ahn. Learning to infer 3d object models from images. arXiv preprint arXiv:2006.061130 2020.
Eric Crawford and Joelle Pineau. Spatiaal invariant unsupervised object detection with convolutional neural networks. In Thirty-Third AAAI Conference on Artificial Intelligence, 2019.
Eric Crawford and Joelle Pineau. Exploiting Spatial Invariance for Scalable Unsupervised Object Tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34(04), pp- 3684-3692, April 2020.
Achal Dave, Pavel Tokmakov, and Deva Ramanan. Towards Segmenting Anything That Moves. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, October 2019.
Terrance Dries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 2017.
Andrea Dittadi and Ole Winther. arXiv:1909. 1813, September 2019.
LAVAE: Disentangling Location and Appearance.
Nimrod Dorfman, Daniel Harari, and Shimon Ullman. Learning to Perceive Coherent Objects. In Markus Knauff, Michael Pauen, Natalie Sebanz, and lpke Wachsmuth (eds.), Proceedings of the Annual Meeting of the Cognitive Science Society, volume 35(35), pp. 394-399, Austin, TX, August 2013, Cognitive Science Society.
10

---

Under review as a conference paper at ICLR 2022
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg. Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. FlowNet: Learning Optical Flow With Convolutional Networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2758-2766, December 2015.
Vincent Dumoulin and Francesco Visin. guide to convolution arithmetic for deep learning. arXiv:1603.07285, January 2018.
Sebastien Ehrhardt, Oliver Groth, Aron Monszpart, Martin Engelcke, Ingmar Posner, Niloy Mi- tra, and Andrea Vedaldi. RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces. In Advances in Neural Information Processing Systems, volume 33 of Advances in Neural Information Processing, December 2020.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Genera- tive Scene Inference and Sampling with Object-Centric Latent Representations. In International Conference on Learning Representations (ICLR), April 2020.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. GENESIS-VE V2: Inferring unordered object representations without iterative refinement. wrXiv:2104.00958 2021.
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey Hinton. Attend, Infer, Repeat: Fast Scene Understanding with Generative Models. In D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, volume 29 of Advances in Neural Infor- mation Processing, pp. 3233-3233,
Curran Associates, Inc., December 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems 27, pp. 6672-26800 Curran Associates, Inc., 2014.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp- 2017.
Klaus Greff, Raphaël Lopez. Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi Object Representation Learning with Iterative Variational Inference. In Kamalika Chaudhuri and Ruslan Salakhudinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2424-2433, Long Beach, California, June 2019. PMLR.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the Binding Problem in Artificial Neural Networks, arXiv:2012.05208, December 2020.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In International Conference on Machine Learning. pp- 1462-1471, 2015.
Nicolas Heess, Nicolas Le Roux, and John Winn. Weakly supervised learning of foreground- background segmentation using masked rbms. In International Conference on Artificial Neural Networks, pp. 9-16. Springer, 2011.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-VAE: Learning Basic Visual Concepts with Constrained Variational Framework. In ICLR 2017, April 2017.
Eddy Ilg. Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. FlowNet 2.0: Evolution of Optical Flow Estimation With Deep Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2462-2470, July 2017.
Varun Jampani, Sebastian Nowozin, Matthew Loper, and Peter V. Gehler. The informed sampler: A discriminative approach to bayesian inference in generative computer vision models. Computer Vision and Image Understanding, 136:32 44, 2015.

---

Under review as a conference paper at ICLR 2022
Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjin Ahn. SCALOR: Generative World Models with Scalable Object Representations. WxXiv:1910.0238/ October 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Im- proved Quality, Stability, and Variation. In International Conference on Learning Representations (ICLR), February 2018.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and Improving the Image Quality of StyleGAN. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattem Recognition (CVPR), pp. 8110-811111 June 2020.
Margret Keuper, Bjoern Andres, and Thomas Brox. Motion Trajectory Segmentation via Minimum Cost Multicuts. In Proceedings of the IEEE International Conference on Computer Vision, pp- 3271-3279, December 2015.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR), May 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Adam Kosiorek, Hyunjik Kim, Yee Whye Tch, and Ingmar Posner. Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31 of Advances in Neural Information Processing. Curran Associates, Inc., December 2018.
Harold W Kuhn. The Hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
T. D. Kulkarni, P. Kohli, J. B. Tenenbaum, and V. Mansinghka. Picture: A probabilistic program- ming language for scene perception. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 4390-4399, June 2015.
Nicolas Le Roux, Nicolas Heess, Jamie Shotton, and John Winn. Learning generative model of images by factoring appearance and shape. Neural Computation, 23(3):593-650, 2011.
Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving Gen- erative Imagination in Object-Centric World Models. In Hal Daumé III and Singh Aarti (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 6140-6149, PMLR, July 2020a.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition. In International Conference on Learning Representations (ICLR), April 2020b.
Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang, Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li,, and Feiyue Huang. Learning by Analogy: Reliable Supervision From Transfor- mations for Unsupervised Optical Flow Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6489-6498, June 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-Centric Learning with Slot At- tention. In Advances in Neural Information Processing Systems, volume 33, December 2020.
Aravindh Mahendran, James Thewlis, and Andrea Vedaldi. Cross Pixel Optical Flow Similarity for Self-Supervised Learning, arXiv: July 2018a.
Aravindh Mahendran, James Thewlis, and Andrea Vedaldi. Self-Supervised Segmentation by Grouping Optical-Flow. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, September 2018b.
12

---

Under review as a conference paper at ICLR 2022
Antoine Manzanera and Julien C Richefeu. A new motion detection algorithm based on o back- ground estimation. Pattern Recognition Letters, 28(3):320-328, 2007.
Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4040-4048, June 2016.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which Training Methods for GANs do actually Converge? In International Conference on Machine Learning, pp. 3481-3490. PMLR, July 2018.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pp. 2204-2212, 2014.
Li Nanbo, Cian Eastwood, and Robert Fisher. Learning object-centric representations of multi- object scenes from multiple views. Advances in Neural Information Processing Systems, 33, 2020.
Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra. Block- gan: Learning 3d object-aware scene representations from unlabelled images. arXiv preprint arXiv:2002. 08988, 2020.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In Pruc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
Peter Ochs and Thomas Brox. Higher order motion models and spectral clustering. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 614-621, June 2012.
Peter Ochs, Jitendra Malik, and Thomas Brox. Segmentation of Moving Objects by Long Term Video Analysis. IEEE Transactions on Patten Analysis and Machine Intelligence, 36(6); 1200, June 2014. ISSN 0162-8828, 2160-9292.
G. Parascandolo, N. Kilbertus, M. Rojus-Carulla, and B. Schölkopf. Learning independent causal mechanisms. In Proceedings of the 35th International Conference Machine Learning (ICML), pp. 1033-4041, 2018.
Deepak Pathak, Ross Girshick, Piotr Dollar, Trevor Darrell, and Bharath Hariharan. Learning Fea- tures by Watching Objects Move. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2701-2710, July 2017.
Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, USA, second edition, November 2009. ISBN 978-0-511-80316-1.
Peters, Janzing, and Schölkopf. Elements of Causal Inference Foundatiinss and Learning Algorithms. MIT Press, Cambridge, MA, USA, 2017.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 DAVIS challenge on video object segmentation. arXiv: 00077700.007,,.. 2017.
Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J. Black. Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12240-12249, June 2019.
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating Diverse High-Fidelity Images with VQ-VAE-2. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, volume 32 of Advances in Neural Information Processing, pp. 14866-14876. Curran Associates, Inc., December 2019.
Danilo Jimenez Rezende and Fabio Viola. Taming VAEs, arXiv:1810.005997 October 2018
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International Conference on Machine Leam- ing, pp. 1278- 1286, 2014.
13

---

Under review as a conference paper at ICLR 2022
Bernhard Schölkopf. Francesco Locatello, Stefan Bauer, Nan Rosemary Ke. Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):112-634, 2021.
William Silversmith. cc3d: Connected Components on Multilabel 3D Images. GPL-3.0 License, January 2021.
Andrews Sobral. BGSLibrary: An openev C++ background subtraction library. In IX Workshop de Visão Computacional (WVC'2013), Rio de Janeiro, Brazil, Jun 2013.
MIT License.
Elizabeth S. Spelke, Principles of object perception. Cognitive Science, 14(1):29-56, 1990. ISSN
0364-0213.
Fierre-Lue St-Charles and Balillummm--lxaaddrr Bilodeau. Improving background subtraction using local binary similarity patterns. IEEE winter conference on applications of computer vision, pp. 509-515, IEEE, 2014.
Lue St-Charles, Guillaume- Alexxanrr Bilodeau, and Robert Bergevin. Universal background subtraction using word consensus models. IEEE Transactions on Image Processing, 25(10)44768-
2711,2016
Karl Stelzner, Robert Peharz, and Kristian Kersting. Faster Attend-InferRepeat with Tractable Probabilistic Models. In Kamalika Chaudhuri and Ruslan Salakhudinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5966-5975, Long Beach, California, June 2019. PMLR.
A.K Subramanian. vat
2020.
Zachary Teed and Jia Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In The European Conference Computer Vision (ECCV), August 2020.
Pavel Tokmakov, Karteek Alahari, and Cordelia Schmid. Learning Video Object Segmentation With Visual Memory. Proceedings of the IEEE International Conference on Computer Vision, pp. 4481-4490, October 2017a.
Pavel Tokmakov, Karteek Alahari, and Cordelia Schmid. Learning Motion Patterns in Videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3386- 3394, July 2017b.
Tom Troscianko, Christopher P Benton, George Lovell, David J Tolhurst, and Zygmunt Pizlo. Camouflage and visual perception. Philosophical transactions of the Royal Society of London. Series Biological sciences, 364(1516):449441,, 02 2009.
Mehmet Ozgur Turkoglu, William Thong, Luuk Spreeuwers. and Berkay Kicanaoglu. A layer-based sequential framework for scene generation with gans. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8901-8908, 2019.
Arash Vahdat and Jan Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. Advances in Neural Information Processing Systems, 33: 19667-9967,, 2020.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In Interna- tional Conference on Learning Representations, 2018.
Rishi Veerapaneni, John D. Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua B. Tenenbaum, and Sergey Levine. Entity Abstraction in Visual Model-Based Reinforce- ment Learning. In 3rd Conference on Robot Learning (CoRL 2019), October 2019.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
Julius von Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. In ICLR 2020 Workshops, April 2020.
14

---

Under review as a conference paper at ICLR 2022
Marissa A Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander Bcker. Unmasking the inductive biases of unsupervised object representations for video sequences. arXiv preprini ur06:2200..07034, 2020.
Max Wertheimer. On Perceived Motion and Figural Organization. MIT Press, Cambridge, MA, 2012.
Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 699-707, 2017.
Christopher Xie, Yu Xiang, Zaid Harchaoui, and Dieter Fox. Object Discovery in Videos Fore- ground Motion Clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 9994-10003, June 2019.
Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised Video Object Segmentation by Motion Grouping. arXiv:2104.07658, April 2021.
Jianwei Yang, Anitha Kannan, Dhruv Batra, and Devi Parikh. Lr-gan: Layered recursive generative adversarial networks for image generation. In International Conference on Learning Representa- tions, 2017.
Yanchao Yang. Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised Mov- ing Object Detection via Contextual Information Separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 879-888, June 2019.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations, 2019.
Jinyang Yuan, Bin Li, and Xiangyang Xue. Generative modeling of infinite occluded objects for compositional scene representation. In International Conference Machine Learning, pp. 7222- 7231, 2019.
15

---

Under review as a conference paper at ICLR 2022
A ADDITIONAL DETAILS ABOUT THE METHOD
A.1 TRAINING ARFLOW ON FISHBOWL
We build on the official implementation of ARFlow provided in https:/github.com//ii wttpg/hhbb..om/iii///RRRoooo To train the model on the Fishbowl dataset, we make the following adaptations to the original con- figuration used for training on Sintel:
The internal resolution of the model is set to 320x448 pixels.
For training, pairs which is substantially more than the Sintel dataset (pretraining: 14,570, main training: 1041), which is the largest dataset which ARFlow was originally trained. Initial experiments using the first 1000 videos did not lead to an improvement in the training objective compared to only using 200 videos.
the first 200 videos of the Fishhowl dataset. This amounts 25.4K frame
We train the model using batch size of 24 and use 300 random batches per epoch. We per- form both the pretraining and main training stage, but using the same data for both stages. The pretraining stage is shortened to 100 epochs, after which the training loss did not improve further.
We selected above parameters by pilot experiments using the final training loss as criterion. All other hyperparameters, in particular regarding the losses, the augmentations and the model architecture, are used unchanged. We remark that the hyperparameters are chosen differently for the two datasets used in the original paper, when closely adapting the training scheme to our setting.
conjecture that the performance on this model most likely improves
A.2 OBJECT MODEL
We loosely build on the B-VAE implementation by Subramanian (2020). We reimplemented the training script and modified architecture details like the latent dimensionality due differences in the image size.
We use CNN with 10 layers as an encoder. Each layer consists of a 3 x 3 convolution, followed by layer normalization (Ba et al., 2016) and a leaky ReLU nonlinarity with a negative slope of 0.01. The decoders are built symmetrically by using the reversed list of layer parameters and transposed convolutions. The decoders both use additional convolutional decoding layer, without normaliza- tion and nonlinearity. The detailed specification of the default hyperparameters used by the object model is given in the following table:
Table 3: Default hyperparameters used for the object model.
Parameter
Value
sample size hidden layers: channels
128 x 64
32, 32, 64, 64, 128, 128, 256, 256, 512, 512 2, 2, 1,2, 1, 1,
hidden layers: strides
latent dimensions
prior loss weight (B)
0.0001
mask loss weight (y)
0.1
learning rate, epoch 1-40
0.0001
learning rate, epoch 41-60
0.00001
We chose the parameters regarding the architecture based on early experiments by qualitatively evaluating the sample quality. The learning rate and mask loss weight y where determined using grid searches with the IoU of the mask as selection criterion. However, of consistency between the rankings in terms of the mask IoU and appearance MAE. The best reconstruction quality was obtained when not regularizing the prior (i.e.. B 0). We chose the final value of B = 0.0001 as compromise between reconsutruction and sampling capabilites based on visual inspection of the results.
noticed high degree
For the mask and appearance losses defined in the main paper. we use an implementation based on the following like pseudo code:
def pojjec__modell loss (image, mask 2g, mask bg, gamma 1., beta 0.001): image (b, G, h, w), mask_fg (b, h, w), rask_bfg (b, w)
16

---

Under review as a conference paper at ICLR 2022
latents encode (image) mask_pred; Lng_preed
(latents), img_decoden (latents)
T_img = (mask_fg * mse (img_pred, image) . sum ()
L mask = (mask_ + mask_bg) * bee (mask_ mask_ig)) sum
l_reg - <i_divergence (1atentss)
2 img, z mask mask fg. (), (mask fg mask lg) sum
11
return Lig 2 imi 12
gamma L_mask / 2_mask beta L_reg
A.3 BACKGROUND MODEL.
Training details We use similar B-VAE and training objective for the background model as for the object model. Different to the object model, we do not predict the mask and in- stead only reconstruct non-occluded background sample. We sweep over learning rates in {1-10-3 5. 10 10 and B in {10 110-2,10 10 1000,,00 10-33 and select the model with B = 10 and learning rate 10-4 which obtained lowest reconstruction and prior loss. A higher value for B caused the training to collapse (low prior loss, but high reconstruction loss). As noted in the main text, the background model performance (and resolution) models, more hyperparameter tuning and better decoders suited to the high resolution. As noted in the main paper, omit these optimizations within the scope of this work and rather focus the object and scene models.
2.00
be very likely improved by using larger
Implementation The background model loss is similar to the reconstruction loss of the foreground model. We omit reconstructing the mask. In the input image for the background model, all fore- ground pixels are replaced by the average background RGB value to avoid distorting the input color distribution, refer to the example images in Fig. 11. The loss be implemented as follows:
def background model loss (image, mask bg, beta = 0.001): # image (b, c, w), mask_fg w), mask_bfg (b,h,
latents encode (image)
decoder (latents)
I_img (mask_ba mse (img_pred) image) ) sum
r_reg k1_divergence (latents)
2 img mask mask_bg.. sum ()
10
return L img 11
Z img beta L reg
A.4 SCENE MODEL.
Mask temperature By default, the mask is computed by passing the logits obtained from the object mask decoder through sigmoid non-linearity for obtaining probabilities. During scene gen- eration, we added sharpness to the samples by varying the temperature t, yielding an object mask
+ exp(-x/t)
where is the logit output by the mask decoder and the exponential is applied element-wiie "Cool- ing" the model to values around r = 0.1 yields sharper masks and better perceived sample quality. Note that the entropy threshold introduced in the next section depends on the value of t.
Entropy filtering When sampling objects during scene generation, we filter the samples according to the model "confidence". A simple form of rejection sampling is used by computing the mask for given object latent, and then computing the entropy of that mask. Given mask m r), adapt the sampling process considering the average entropy across all pixels in the mask,
1
W H m(u, m( 2=
(m(m)
Σ
m(u, v)) logz(1 m(u.v)
(2)
WH
17

---

Under review as a conference paper at ICLR 2022
and reject samples where Hm exceeds threshold. Reasonable values for mask temperature of t 0.1 are around 100 to 200 bits. possible to trade sample quality and sharpness for an increased variability in the samples by increasing the threshold.
18

---

Under review as a conference paper at ICLR 2022
B ADDITIONAL DETAILS ABOUT THE DATASET GENERATION
A demo of the original aquarium WebGl. demo used for our experiments is available at webglsam- alessor//quuriuu//auurru html. The original source code released under a BSD 3-clause license (Google Inc.) and available at github.com gjthub.co/WeebGSSamge WebGLSamplee//Weleee WebGLSample Samples.gitthu.bio
We apply the following modifications to the original demo for our recording setup:
We implemented client-server recording setup: The client pulls recording parameters from the server and sends back the recorded frames with ground truth segmentation to the server. The server maintains the overall list of randomly sampled aquarium configurations and post-processes the recordings into their final form. We run this setup parallel on multiple nodes, using the headless Chrome browser".
All samples recorded for 128 frames at 30Hz. We modified the aquarium only advance to the next frame once the current frame is captured and sent to the server.
For recording the masks, we modified the texture shaders to use unique color for rendering every object. Furthermore, we disabled all rendering steps modifying those colors (anti-aliasing, alpha compositing).
To make the recording fully deterministic, we disabled the bubbles in the aquarium.
To increase the diversity of the fish in the aquarium, we applied random color shifts to all fish textures except sharks, for which those color shifts look very unnatural. We shifted each color by rotating each color value in the IQ plane of the YIQ color space. For each fish, we independently and uniformly sampled one of discrete color shifts regularly spaced [0 330%)
During post-processing, we used ffmpeg* to rescale and centrally crop 480 x 320px from every frame and encode all frames as video. We cropped and rescaled the masks same way, but exported them as json using the run length encoding provided by the COCO API".
For recording the unoccluded ground truth, we used above setup record and post-process frames and masks of the validation and test samples, but only render the respective fish of interest.
The code used to generate the dataset including all parameters will made publicly available.
hhan::///hiib.cooo/ppppeetpppept #haps://ffipeg..rg
19

---

Under review as a conference paper at ICLR 2022
C ADDITIONAL RESULTS
C.I OBJECTS EXTRACTED BY THE MOTION SEGMENTATION
The following figure shows examples objects extracted from the original videos using motion segmentation and the ground truth occluded masks, respectively. The figure reveals typical failure modes, such as multiple fish that are segmented jointly, and parts the background contained the object mask. Most of the fish however, are segmented very accurately.
Figure 8: Objects extracted from the training videos that are used for training the object model. Left: Objects extracted using the motion segmentation. Right: Objects extracted using the ground truth occluded segmenta- tion masks.
20

---

Under review as a conference paper at ICLR 2022
C.2 OBJECT MODEL: ADDITIONAL RECONSTRUCTIONS
In Fig. we show additional reconstruction from the object model trained on motion segmentation and ground truth occluded masks, respectively. Moderate occlusion levels up to 30% are handled well by both variants. higher noise levels however, only the variant of the object model trained on the ground truth masks is able to correctly complete the partial objects.
Trained motion segmentation
Occlusion
Trained ground truth masks
0.0
0.1
0.2
0.3 0.4
0.4-0.5
0.6
0.8
Figure 9: Reconstructions from the object madel for input crops with different occlusion levels (0.0 occlusion, 1.0 fully occluded). For each occlusion level, the input images and the respective reconstructions are shown. Both model variants are trained using another fish artificial occlusion during training. Left: Object model trained using motion segmentation masks. Right: Object model trained using the ground truth unoccluded masks.
21

---

Under review as a conference paper at ICLR 2022
C.3 OBJECT MODEL: ADDITIONAL SAMPLES
The following figure shows additional samples from the object models trained using another object artificial occlusion (the same models used for Fig. 4 in the main paper).
Figure 10: Samples from the object model using another input object as augmentation during training. These are the same models as used for Fig. in the main paper. Left: Object model trained on the motion segmentation. Right: Object model trained on the ground truth occluded masks.
C.4 BACKGROUND MODEL: INPUTS
The background model trained on images pre-processed by an ensembling of foreground- background algorithms from Sobral (2013). We use Σ A (Manzanera & Richefeu, 2007), static frame differences. LOBSTER (St-Charles Bilodeau, 2014) and PAWKS (St-Charles et al., 2016) with standard hyperparameters set in Sobral (2013). The goal behind this choice is to detect as many objects as possible and remove the amount of erroneously included foreground pixels might be possible to even further improve this pre-processing step with different algorithms and hyperparameter tuning.
22

---

Under review as a conference paper at ICLR 2022
We give a qualitative impression of the background input samples in Fig. 11. Note that all foreground pixels were replaced by the average color value obtained by averaging over background pixels to avoid inputting unrealistic colors into the B-VAE.
Figure 11: Input samples to the background V-VAE. Objects were removed by applying an ensemble of foreground- foreground-bcckground background segmentation algorithms. Images are resized to 96 64px to trade-off training speed for sample quality.
C.5 ADDITIONAL SAMPLES FROM THE SCENE MODEL
Moving single objects
Changing background
Figure 12: Two object "reconstructions". Object latents are obtained from a reference sample. Our modular scene model makes it straightforward to vary locations of single objects, exchanging single objects, changing the background without affecting the output sample (top to bottom).
23

---

Under review as a conference paper at ICLR 2022
Figure 13: Additional samples from the scene model. Depicted samples different (reconstructed) backgrounds, samples from the object model are obtained from the standard normal prior and tered with 150 bit entropy threshold at = 0.2, sample sizes are constrained on a reference training sample, object positions sampled independently from uniform prior. Samples are cherryp- icked.
24

---

Under review as a conference paper at ICLR 2022
C.6 CONDITIONAL SAMPLING FROM THE SCENE MODEL
We present conditional samples from the scene model. As a simple baseline, we use a k-nearest neighbour approach for conditionally sampling object latents based on background latents. First, we extract paired dataset of background and foreground latents from the training dataset. Second, we sample background latents from the standard Normal distribution (prior of the background model). Third, we compute the 2% nearest neighbouring videos based on the 1.² distance from the back- ground latent. Fourth, we randomly sample a subset of foreground latents extracted by the motion segmentation. Finally, we reconstruct the scene using the background latents along with the cho- sen subset of foreground latents. We reject foreground latents with an entropy larger than 150 bit. Samples (non-cherrypicked) are depicted in Fig. 14
Figure 14: Conditional samples from the model. Fish appearances (qualitatively, mainly the bright- ness) now vary according to the background sample.
25

---

Under review as a conference paper at ICLR 2022
Original Image
Scene re-sampling with matching latents
Scenee re-sempling with new latents
MaSeg
Figure 15: Conditional sampling of object locations based on reference scene. For matching latents sampling. we extract the object latents and positions from the reference scene using the motion segmentation model. Samples obtained by matched sampling are more similar to the reference scene than samples obtained by unconstrained sampling from the model priors.
Similarly, we can constrain other latents like the and y positions to a particular background. In Fig. 15 we show examples of constraining object locations based on a reference sample for both the ground truth and motion segmentation object model.
26

---

Under review as a conference paper at ICLR 2022
C.7
ENTROPY SAMPLING
A1 Mask Entropy distribution
A2 Predicting Entropy from latents
n=25,000
1000
Count
Train ~0.46 (n=5,0 Test +0.45 (n=20,000)
500
Model: Linear Regression, 128d
500
1000
1500
Entropy [bits]
Top 64/1000 semples in terms Mask Entropy
C Worst 64/1000 samples in terms of Mask Entropy
Figure 16: Details of entropy filtering when using the object model. (A1) Distribution of mask entropies for given foreground model, estimated over 25,000 randomly sampled objects. The distribution typically peaks around 100 to 200 bits, makeing this reasonable range for picking the cut-off. (A2) The 128d latent vector of objects reasonably correlated with the entropy (R² 0.45), making it possible to alter the prior to sample from to encourage low entropy samples. (B) 64 samples with the lowest entropy after drawing 1000 random samples from the object models and sorting according to entropy. While this strategy encourages some non- plausible, low entropy objects (row 4 and 6), it generally filters the dataset for samples with sharp boundaries and good visual quality. (C) 64 samples with the highest entropy after drawing 1000 random samples from the object models and sorting according to entropy. With a few exceptions of plausible samples (c.g. in row 2), most of the samples should be rejected in the scene model.
27

---

Under review as a conference paper at ICLR 2022
D COMPARISON TO RELATED METHODS
D.1 GAN BASELINE
As baseline method for generating novel scenes for the FISHBOWL, dataset, we consider the GAN used by Mescheder et al. (2018) for the CelehA-HQ dataset (Karras et al., 2018). We used the official and only changed the resolution of the generated images to 192 x 128px. For training, we used every 16th frame from each input video in the training set rescaled to the resolution of the GAN, resulting in 160k training images. We trained the model for 90 epochs.
implementation available at https://githuh. httptthhh..oommMeeeheeeeeeeeeeee MeweeeeeeeeeeGGANAAaaaaaaa
Fig. 17 shows samples from the model comparison to training frames. Overall, the GAN is able to generate images of convincing quality resembling the training images well. A particular strength of the GAN comparison to our method, it's ability to generate backgrounds with many details- which we explicitly left for future work. Several fish generated by the GAN however look distorted, in agreement with previous works concluding that GANs are good at generating "stuff". but have difficulties generating objects Bau et al. (2019). This becomes especially apparent when visualizing interpolations in the latent space between samples, as done in Fig. 18.
Many differences between the samples generated by the GAN and our method, respectively, stem from the GAN being able to learn the overall statistics of the scenes well, but not learning notion of objects. Compared to the GAN baseline, our object-centric approach offers several conceptual represented individually by method, which makes it straightforward to intervene on the generated samples in a meaningful way (Fig. 6). While directions in the latent space of the GAN that correspond to semantically meaningful changes in the image might well exist, the GAN framework does not offer a principled way to find those directions without supervision. (2) While the GAN is able to generate novel scenes, it cannot used to infer the composition of input scenes. (3) An optimally trained GAN perfectly captures the statistics of the input scene scene-making it impossible training distribution. As our model explicitly represents scene parameters such as the number and position of fish, our model can be used for controlled out-of out-o--distribution sampling (Fig. 6).
advantages: (1) The background and the objects in each
generate samples beyond the
Training images
Samples
Figure 17: Samples from baseline GAN (Mescheder al., 2018) trained frames from the training set of
the FISHBOWL dataset, Top: Input frames from the dataset. Bottom: Samples generated by the GAN.
28

---

Under review as a conference paper at ICLR 2022
Figure 18: Samples from the bascline GAN, obtained by interpolating the latent vectors between random samples.
D.2 SPACE (LIN ET AL., 2020B)
SPACE (Lin et al., 2020b) is an object-centric representation learning method following the Attend- Infer-Repeal (AIR) framework (Eslami et al., 2016; Crawford Pineau, 2019). Different from the AIR model, the detection of objects and inference of their latent representation is fully parallelized to make the approach faster and more scalable. We trained SPACE on the Fishbowl dataset us- ing the implementation provided by the authors the default hyperparameters and trained respectively. As for the GAN baseline in appendix D, we used every 16th frame for training this model keep the training time reasonable (160k frames in total). Despite the subsampling, this is still substantially more data than the model was trained on in the original paper (60k images). SPACE expects the input images have equal width and height, therefore we used central square crop from every frame.
http://gituub..om//iiuuaaa -innannn-iSSSCCE). We used variants using 4x4 and 8x8 grid of object detectors,
Training the model 18h for the 8x8 grid), confirming the performance advantage the parallel object detectors. As the results in the Fig. 19 show, the object reconstructions from the model overall look reasonable. Most structure in the background missed by the model, however we conjecture that this might be solvable by adapting the capacity of the background network. The visualization of the object detections however reveals a more fundamental failure mode due to the grid structure used by the object detector in SPACE: Larger fish are split across several cells, even when using only 4x4 cells. As each cell can only handle most one object, decreasing the cell count further is not expected to yield sensible results, as this limits the number of objects too much. Lin et al. (2020b) mentioned this problem and introduced boundary loss to address however on the Fishbowl dataset this does not resolve the problem. We hypothesize that an approach based on a fixed grid, while adequate in some cases and having substantial performance advantages, doesn't work well with objects showing large scale variations as present in our newly proposed dataset. We believe this a limitation of SPACE that cannot be resolved with further model tuning.
single Nvidia RTX 2080ti GPU is relatively fast (14h for the 4x4 grid and
29

---

Under review as a conference paper at ICLR 2022
SPACE 4x4
SPACE 8x8
Reconstruction
Reconstructed
Object Detections
Input
Reconstruction
Reconstructed Objects
Object Detections
Objects
Figure 19: Scenes from the Fishbowl dataset reconstructed by SPACE (Lin et al., 2020b). The model is trained using the official implementation provided by the authors with the default parameters. The two variants shown use grid sizes of 4x4 and 8x8, respectively.
We performed an additional experiment that integrates our approach with SPACE: As a first step, we trained a variant of object model that uses the glimpse encoder and decoder from SPACE. We increased the learning rate 0.0005 following an initial grid search, but otherwise left hyperpa- rameters unchanged. As the second step, we included the pretrained glimpse encoder and decoder into the SPACE model and trained the full model end-to-end as described above (using grid size of 4x4). When we continued to train the glimpse encoder and decoder in the end-to-end setting, we noticed similar failures as described above, Therefore we froze the respective weights after the pretraining step.
Figure 20 shows results from this model variant. Early during the training, the model is in many cases able to detect the objects in the input scenes. This especially holds for larger objects that have been shown to be problematic for the end-to-end model before. Later during training SPACE however increasingly uses the background model to also explain foreground objects. In addition, evaluated both the pretrained and end-to-end trained object model from SPACE using the ground truth unoccluded objects. As the quantitative results Table Tab. show, the reconstruction of the appearance works better with the SPACE object model than with our architecture", but has substantial gap in the segmentation performance. More importantly, the results clearly show an improvement of the motion segmentation based training over the original end-to-end approach, con- firming the qualitative results.
Overall those results show that motion segmentation based approach for object-centric model- ing can also be used for scaling existing architectures to more complex settings. The off-the-shelf SPACE model however still falls short of our model when trained this way. In the future see great potential in combining our object learning approach with state-of-the-arr object-centric scene models.
Too some degree this is confounded with the worse segmentation performance, the appearance error only evaluated the intersection of the ground truth and predicted mask.
30

---

Under review as a conference paper at ICLR 2022
Table 4: Quantitative evaluation of the SPACE object model trained end-to-end within SPACE and pretrained based motion segmentations using the loss object model. The pretrained SPACE object model ours use cutout augmentation. metrics are the same as in Tab.
Object model
IoU f
MAE
IoU@0.5 f
MAE@0.5
SPACE (end-to-end)
0.533
12.3
0,425
16.4
SPACE (pretrained)
0.734
7.60
0.605
12.4
Ours
0.822
13.0
0.677
24.0
Baseline
0.915
0.271
Early training
Late training
Input
Recons.
Recons.
Object
Recons.
Input
Recons.
Recons.
Object
Recons.
Foreground
Detections
Background
Foreground
Detections
Background
Figure 20: Scenes from the Fishbowl dataset reconstructed by SPACE (Lin et al., 2020b). The glimpse encoder and decoder where trained on the candidate objects given by the motion segmentation using the object model loss proposed in this work. Afterwards, the remaining components where trained end-to-end using the official implementation provided by the authors with the default parameters.
D.3 GENESIS-V2 (ENGELCKE ET AL., 2021)
We trained GENESIS-VE v2 on the Fishbowl dataset using the official implementation by the authors (https://github.. wttppau..o//apppp...a... -iaeaaiie////gssss with default hyperparameters except for the image reso- lution, number of object slots and batch size. We modified the model code work for rectangular images and used resolution of 128x 192 pixels. We trained GENESIS-v2 having 5 object slots on Nvidia RTX 2080Ti GPUs using a batch size of 64. Initial experiments with 10 object slots lead to the background being split up into multiple slots. As for the GAN baseline before, we used every 16th frame for training this model (160k frames in total).
In Fig. 21 we show qualitative results from GENESIS-v2 on the Fishbowl dataset. The reconstruc- tions of the model look somewhat blurry but capture the major structure in the input images well.
31

---

Under review as a conference paper at ICLR 2022
Importantly the visualization of the segmentation map and the individual slots reveal that the model succeeds in learning to decompose the scenes into background and objects. Sampling objects and scenes however fails with this model configuration. Most likely this happens due the GECO ob- jective (Rezende
Viola, 2018), that decreases the weight of the KL. term in the loss function as long as the log likelihood of the input sample is below the target value. Training GENESIS-v2 with the original VAE objective instead of GECO leads to better scene samples, the decomposition of the scene however fails with this objective Fig. 22.
As comparison the end-to-end training within GENESIS-v2, we trained variant of our object model using the architecture of the GENESIS-v2 component decoder. As encoder, we use a CNN constructed symmetrically
the decoder, using regular convolutions instead of the transposed con- volutions. We trained the model with the Cutout augmentation and using the same loss and training schedule as for the object model described in the main paper.
The results in Tab. and Fig. 23 show that the model generally performs well, but worse than our original object model. This most likely be explained by the larger capacity of our object model (10 vs layers, 128 vs latent dimensions) which seems be necessary to model the visually more complex objects in our setting. Also when trained separately, the object model has difficulties with sampling novel fish, which can be addressed at the cost of worse reconstructions 24.
Overall we conclude that our modular object learning approach scales much better to visually more complex scenes than end-to-end training as done by GENESIS-v2. Even when using the GENESIS- v2 component decoder within our framework, the necessary trade-off between reconstruction and generation capabilities seems to be more favorable when using our modular approach as opposed to the end-to-end training. We remark that this comparison comes with a grain of salt: We neither adapted the hyperparameters of GENESIS-v2 GENESIS-VE for the background, too. The strong qualitative difference in sample quality however makes it unlikely that this explains all of the difference. Moreover, learning generative object models whereas GENESIS-v2 v2 is also capable to infer the decomposition of static input scenes. For the future, we therefore see much potential in combining the respective strengths of both methods.
of our method and the same decoder is used within
approach only addresses
32

---

Under review as a conference paper at ICLR 2022
Input
Reconstructed Scene
Reconstructed Segmentation
Sampled Objects
Sampled Scenes
Qualitative results of GENESIS-v2 applied on the FISHBOWL dataset. The reconstruction in the second row look somewhat blurry, but capture all major structure in the input images shown in the first row. The visualization of the reconstructed segmentation shows that the model succeeds in decomposing the input image into the background and the different objects. Sampling from the model however fails in this setting.
Figure 21:
33

---

Under review as a conference paper at ICLR 2022
Input
Reconstruction
Segmentation
Samples
Figure 22: Qualitative results of GENESIS-VE v2 trained on the Fishbowl dataset using the default VAE objective instead of the GECO objective. Sampling from the model works much better in this sctting; the model fails however to segment the scene into the indiviual objects.
Table 5: Comparison reconstructions from object model based on the GENESIS-v2 component decoder with our original object model using the same metrics as Tab. 2. Both models are trained using cutout augmentation.
Training data
Architecture
IoU f
MAE +
IoU@0.5 f
MAE@0.5 +
Motion
GENESIS-v2
0.779-10.002
15.3.100063
0.66110.003
25.710.114
Segmentation
Ours
0.822-10.001
13.0:10.032
0.67710.002
24.011.0117
Ground Truth
GENESIS-v2
0.844-10.0M
13.7110109
0.722-10.001
19.8:10002
Segmentation
Ours
0.887-10.001
12.3:10.035
0.743-10.003
17.3:10.087
Motion Segmentation
Ground Truth Segmentation
Reconstruction
Samples Reconstruction
Samples
Figure 23: Qualitative results when using the GENESIS-<2 object decoder object model within modular training approach using the same loss and training schedule. Reconstructions and samples look worse than with the original object model, hinting at larger capacity of our object model being necessary for dataset.
34

---

the boundary to 8px Under review as a conference paper at ICLR 2022
Motion Segmentation
Ground Segmentation
Samples Reconstruction
Samples Reconstruction
Figure 24: Qualitative results when using the GENESIS-V2 object decoder as object model within our modular training approach using larger weight KL divergence the VAE training loss. At the prize worse reconstructions, samples from the model substantially improved this way.
D.4 COMPARISON ON THE REALTRAFFIC DATASET
To evaluate how well our method transfers to other settings, we trained our model on the RealTraffic dataset (Ehrhardt et al., 2020). As the resolution of the images is smaller in the RealTrafffic dataset, we reduced the object size threshold to 64px and the minimal distance for the object extraction. We trained the object model with the same architecture as used for the Fishbowl dataset. Due the smaller dataset size, we trained the model for 600 epochs and reduced the learning rate only after 400 epochs. As the videos in the dataset were all recorded from the same stationary traffic camera, background directly. In contrast to the aquarium dataset, the object positions and sizes are not distributed uniformly for this dataset. For the scene model, therefore sample directly from the empirical joint distribution object positions and sizes (extracted from the object masks obtained by the motion segmentation).
did not train a background model but used the mean-filterrs
35

---

Under review as a conference paper at ICLR 2022
Object Samples
Ours
Samples
Ours
GENESIS [15]
RELATE [13]
BlockGAN [83]
Reconstruction
SPACE [37]
Figure 25: Results from model trained the RealTraffiic dataset compared results from other models trained this datasct.
Fig. 25, we compare samples from our model to other models for which results on the RealTraffic dataset have been reported in (Ehrhardt et al., 2020). Overall, our model transfers well to this real world setting, given that our model is used largely unchanged. In comparison to the GAN-based RELATE and BlockGAN, the samples from our model look slightly more blurry. However the results from our method clearly improve over the VAE-based GENESIS model.
The adaptation of the scene statistics as described above works well for the object position, as the cars correctly positioned the street only. We consider the possibility of this straight-forward adaptation to novel scene statistics to be a nice advantage of our object-centric modeling approach. In the future, this could be improved further by, e.g., learning to sample latent variables conditioned on the object positions.
36


Reviews:
[{"review_id": "njK-2VFVbLN", "reviewer": "Reviewer_GUfQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. I appreciate the effort of building a new dataset.\n2. Compared with other methods, the scalability of the proposed method is indeed strong.\n3. I am convinced that the direct application of \"the common fate heuristic\" is novel and is of great potential.\n4. While most related work can only generate masks for occluded (partial) objects, the proposed method can generate masks for full objects occluded or not.\n5. The ability to sample novel scenes is impressive.\n\nWeakness:\nI apologize if some of my concerns are already detailed in the paper and I fail to pick them out.\n1. Discussion of related works.\n    While the author did a comprehensive literature review, I am not convinced by some of the claims.\n    1): I am not sure why \"the spatial mixture model does not match the true, underlying scene generation process\".\n         For example, SPACE employs a depth prediction to predict the relative depth between objects.\n         Then the depth information is integrated into the mask value.\n         SPACE effectively transforms a 2D segmentation task into 2.5D.\n    2): \"End-to-End object-centric approaches are difficult to train.\"\n         Without further explanation, I am not sure what \"difficult to train\" means under the context.\n    3): \"Full scene models trained in an end-to-end fashion are not shown to scale to a more realistic dataset yet.\"\n         As demonstrated by the paper SCALOR: GENERATIVE WORLD MODELS WITH SCALABLE OBJECT REPRESENTATIONS (which is already cited by the author), such method can be applied to challenging cases.\n\n2. Dataset\nThe dataset proposed is indeed interesting and challenging.\nMy only question is that is it true that all foreground objects are moving all the time and the backgrounds do not have any moving parts?\n\n3. Baseline\nIt is surprising that no baseline model is evaluated on the same dataset.\nI understand that there are not many works targeting exactly the same setup, but works like SCALOR or SPACE can also be used as a baseline for numerical results.\nAfter all, being able to segment without \"the common fate\" heuristic and a rule based object proposal stage seems like an advanrages to me.\nTo show that the loss function proposed in this work is better, the author can modify SPACE to take bounding boxes proposed by the rule based motion segmentation pipeline. This should be an easy modification.\nIf under this modification the performance of SPACE is better than before but still behind this paper, then the method propsed in this work is truely great.\nSimilar modification can be done to SCALOR as well I think.\n", "summary_of_the_paper": "This paper provides a multi-stage solution to unsupervised, frame-wise segmentation in videos.\nThe common fate heuristic is used to provide initial object detections and segmentations.\nThen VAE based model is used to refine the initial results.\nA new simulated dataset is proposed.\n\nResults show that the proposed method successfully segments out moving objects and is highly scalable.", "summary_of_the_review": "The results provided in the paper is impressive.\nBut the authors argue multiple times that the method proposed in this work is better than end-to-end generative model without enough justifiation.\nThus I cannot say that this paper is above accept threshold at current stage.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WzAUlNtab_n", "reviewer": "Reviewer_dAqW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Object-centric generative models have gained relatively less attentions compared to the normal generation models. The authors have generated a new dataset for this research area. However, I have many concerns that make me hesitant to accept this paper.\n\nFIrst of all, the dataset is far from the realistic setting. It is related to missing literature search. There are many datasets for amodal segmentation. For example, https://openaccess.thecvf.com/content_CVPR_2019/papers/Hu_SAIL-VOS_Semantic_Amodal_Instance_Level_Video_Object_Segmentation_-_A_CVPR_2019_paper.pdf <- this paper basically proposes a more realistic dataset, which provides the same information (amodal masks). I assume the authors have not looked into this literature. There are many other amodal datasets, including KINS and COCO-A.\n\nMy second point is that the proposed model has not been evaluated well. I have gone through the paper as well as the supplementary, but there was no appropriate comparison to the previous works. It's hard to believe all the conventional algorithms fail at generating object-centric images.", "summary_of_the_paper": "This work proposes an object-centric generative model. It consists of motion segmentation, object model, background model, and scene model. The object model is trained to reconstruct an object as if it is not occluded. The authors have introduced a new dataset, called Fishbowl, which provides inmodal and amodal segmentation masks of objects. ", "summary_of_the_review": "Because of the above two major weaknesses, I'm skeptical of accepting this paper. I am open to change my rating depends on the rebuttal. For now, my initial rating is reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AfAXr7JHtVl", "reviewer": "Reviewer_3iwv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the paper quite useful and interesting. I find it useful because the decomposition allows for better ablative analysis of the model performance compared to previous end-to-end approaches, and because it opens up additional practical applications, such as given user direct and interpretable control on the output generated by the model (e.g. manually input number of objects or an individual object position). I find the results interesting, particularly the thorough ablative analysis: comparing training on ground truth segmentation vs training on motion segmentation.\n\nThe paper has some weaknesses, e..g. some claims not thoroughly backed up, lack of clarity on some details. I list them here in detail, from more important to less important: \n1) The paper strongly couples two concepts: a) the choice of using a multi-stage learning approach rather than end-to-end; and b) the use of motion segmentation to enable training this model. In practice, it seems to me that these two things are not really that coupled, but rather motion segmentation is just a way to automatically generate the training data that the model needs, because unlike end-to-end method, this submission needs explicit object and segmentation masks (if I understand correctly). One thing I would like the authors to highlight more  is that, due to this, a limitation of this method is that it can be trained only on video data and not on static images, unless some form of segmentation ground truth is available or there is a way to automatically generate it -  this is mentioned in the conclusions, but I encourage making it clear earlier in the paper\n2) It is unclear if the parameters of the scene models learnt entirely independently from the foreground and background model, i.e. by learning the distribution of the latent vectors produced by the VAEs conditioned on the background? Or is there some gradient propagating between these steps\n3) von kugelgen et al (2020) also provide a controllable way to change the way the model generates the scenes (e.g. number and type of objects). Can you highlight the main differences? Is it mostly that in this submissions there is also control on scale and position?\n4) Results on the fishbowl dataset look quite impressive, less on the car dataset (I suggest adding some visuals from this dataset and of visual comparisons to existing methods in the main body of the paper). It would be interesting to see this method run on other datasets like the Atari dataset or the 3D room dataset from the SPACE paper, to further test generalisation of the multi-stage approach in other domains - do ground truth segmentations exist to test there? \n5) Understanding the performance of this methods on generating new scenes compared to existing work is quite challenging because it is done mostly qualitatively, although I recognise that is a challenge of the domain and existing methods also seems to suffer from this", "summary_of_the_paper": "The paper introduces an object-centric generative model for visual scenes. The model decouples the problem into three tasks: 1) modelling the  2D appearance and shape of individual objects with a variational auto encoder; 2) same thing for background; 3) sampling the position, size and appearance of individual objects (i.e. scene composition) conditioned on the background. To my understanding, these three components are trained independently.\n\nContributions:\n1) Decoupling these three tasks allows for a more interpretable representation compared to existing end-to-end methods, which allows for better interactions with users (e.g. change the number of objects or their position) \n2) This decoupling potentially makes the third task easier, because it only needs to learn relationships between positions and size given a latent representation (as opposed to learning everything jointly).\n3) Training the tasks independently requires a way to obtain \"ground-truth\" object segmentations - the paper achieves this automatically from videos using motion segmentation with good results", "summary_of_the_review": "While I do not see a significant amount of novel technical contributions, the proposed paradigm of breaking down scene generative models into multiple stages of learning that allows encoding prior structure and human-interpretable elements is valuable. It can help generate theoretical insights, and the ablative analysis provided here is a promising first step. It also has the potential to be practically useful, by allowing users more fine-grained controls on the parameters of the generated scenes.\n\nThere are some claims with minor issues, and some lack of clarity in some sections (see weaknesses in the review section above). All in all, I think the paper is above the acceptance threshold, and addressing these weaknesses would make me lean towards a clearer accept\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the challenging problem of object-centric generation of visual scenes. While the paper has some novel ideas that make it interesting, its (quantitative and qualitative) comparison with existing methods is currently premature to allow drawing conclusions with sufficient evidence.

Instead of claiming that existing models cannot do well for the more realistic datasets mentioned by reviewer dAqW, it would be more convincing to conduct a comprehensive experimental study by comparing the proposed method with existing methods on a range of datasets, from simple ones to more realistic ones. The synthetic Fishbowl dataset introduced in this paper can be one of them.

Moreover, the clarity of the paper could be improved to make it appeal better to the readers.

All three reviewers engaged actively in discussions (both including and not including the authors). Although one reviewer recommends 6 (weak accept), the reviewer also shares some of the concerns of the other reviewers. As it stands, the paper is not ready for acceptance. If the comments and suggestions are incorporated to revise the paper, it will have potential to be a good paper for future submission.
Decision: Reject