Unsupervised Foreground Extraction via Deep Region Competition
Year: 2021

Abstract:
We present Deep Region Competition (DRC), an algorithm designed to extract foreground objects from images in a fully unsupervised manner. Foreground extraction can be viewed as a special case of generic image segmentation that focuses on identifying and disentangling objects from the background. In this work, we rethink the foreground extraction by reconciling energy-based prior with generative image modeling in the form of Mixture of Experts (MoE), where we further introduce the learned pixel re-assignment as the essential inductive bias to capture the regularities of background regions. With this modeling, the foregroundbackground partition can be naturally found through Expectation-Maximization (EM). We show that the proposed method effectively exploits the interaction between the mixture components during the partitioning process, which closely connects to region competition [1], a seminal approach for generic image segmentation. Experiments demonstrate that DRC exhibits more competitive performances on complex real-world data and challenging multi-object scenes compared with prior methods. Moreover, we show empirically that DRC can potentially generalize to novel foreground objects even from categories unseen during training.1

heading: 1 Introduction
text: Foreground extraction, being a special case of generic image segmentation, aims for a binary partition of the given image with specific semantic meaning, i.e., a foreground that typically contains identifiable objects and the possibly less structural remaining regions as the background. There is a rich literature on explicitly modeling and representing a given image as foreground and background (or more general visual regions), such that a generic inference algorithm can produce plausible segmentations ideally for any images without or with little supervision [1–8]. However, such methods essentially rely on low-level visual features (e.g., edges, color, and texture), and some further require human intervention at initialization [4, 5], which largely limits their practical performance on modern datasets of complex natural images with rich semantic meanings [9, 10]. These datasets typically come with fine-grained semantic annotations, exploited by supervised methods that learn representation and inference algorithm as one monolithic network [11–16]. Despite the success of densely supervised learning, the unsupervised counterpart is still favored due to its resemblance to how humans perceive the world [17, 18].
1Code and data available at https://github.com/yuPeiyu98/DRC
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Attempting to combine unsupervised or weakly supervised learning with modern neural networks, three lines of work surge recently for foreground extraction: (1) deep networks as feature extractors for canonical segmentation algorithms, (2) GAN-based foreground-background disentanglement, and (3) compositional latent variable models with slot-based object modeling. Despite great successes of these methods, the challenge of unsupervised foreground extraction remains largely open.
Specifically, the first line of work trains designated deep feature extractors for canonical segmentation algorithms or metric networks as learned partitioning criteria [19–21]. These methods (e.g., W-Net [19]) define foreground objects’ properties using learned features or criteria and are thus generally bottle-necked by the selected post-processing segmentation algorithm [22, 23]. As a branch of pioneering work that moves beyond these limitations, Yang et al. [24, 25] have recently proposed a general contextual information separation principle and an efficient adversarial learning method that is generally applicable to unsupervised segmentation, separation and detection. GAN-based models [26–31] capture the foreground objectness with oversimplified assumptions or require additional supervision to achieve foreground-background disentanglement. For example, the segmentation model in ReDO [28] is trained by redrawing detected objects, which potentially limits its application to datasets with diverse object shapes. OneGAN [31] and its predecessors [29, 30], though producing impressive results on foreground extraction, require a set of background images without foreground objects as additional inputs. Lastly, compositional latent variable models [32–40] include the background as a “virtual object” and induce the independence of object representations using an identical generator for all object slots. Although these methods exhibit strong performance on synthetic multi-object datasets with simple backgrounds and foreground shapes, they may fail on complex real-world data or even synthetic datasets with more challenging backgrounds [37, 38]. In addition, few unsupervised learning methods have provided explicit identification of foreground objects and background regions. While they can generate valid segmentation masks, most of these methods do not specify which output corresponds to the foreground objects. These deficiencies necessitate rethinking the problem of unsupervised foreground extraction. We propose to confront the challenges in formulating (1) a generic inductive bias for modeling foreground and background regions that can be baked into neural generators, and (2) an effective inference algorithm based on a principled criterion for foreground-background partition.
Inspired by Region Competition [1], a seminal approach that combines optimization-based inference [41–43] and probabilistic visual modeling [44, 45] by minimizing a generalized Bayes criterion [46], we propose to solve the foreground extraction problem by reconciling energy-based prior [47] with generative image modeling in the form of Mixture of Experts (MoE) [48, 49]. To generically describe background regions, we further introduce the learned pixel re-assignment as the essential inductive bias to capture their regularities. Fueled by our modeling, we propose to find the foreground-background partition through Expectation-Maximization (EM). Our algorithm effectively exploits the interaction between the mixture components during the partitioning process, echoing the intuition described in Region Competition [1]. We therefore coin our method Deep Region Competition (DRC). We summarize our contributions as follows:
1. We provide probabilistic foreground-background modeling by reconciling energy-based prior with generative image modeling in the form of MoE. With this modeling, the foregroundbackground partition can be naturally produced through EM. We further introduce an inductive bias, pixel re-assignment, to facilitate foreground-background disentanglement.
2. In experiments, we demonstrate that DRC exhibits more competitive performances on complex real-world data and challenging multi-object scenes compared with prior methods. Furthermore, we empirically show that using learned pixel re-assignment as the inductive bias helps to provide explicit identification for foreground and background regions.
3. We find that DRC can potentially generalize to novel foreground objects even from categories unseen during training, which may provide some inspiration for the study of out-of-distribution (OOD) generalization in more general unsupervised disentanglement.

heading: 2 Related Work
text: A typical line of methods frames unsupervised or weakly supervised foreground segmentation within a generative modeling context. Several methods build upon generative adversarial networks (GAN) [26] to perform foreground segmentation. LR-GAN [27] learns to generate background re-
gions and foreground objects separately and recursively, which simultaneously produces the foreground objects mask. ReDO (ReDrawing of Objects) [28] proposes a GAN-based object segmentation model, based on the assumption that replacing the foreground object in the image with a generated one does not alter the distribution of the training data, given that the foreground object is correctly discovered. Similarly, SEIGAN [29] learns to extract foreground objects by recombining the foreground objects with the generated background regions. FineGAN [30] hierarchically generates images (i.e., first specifying the object shape and then the object texture) to disentangle the background and foreground object. Benny and Wolf [31] further hypothesize that a method solving an ensemble of unsupervised tasks altogether improves the model performance compared with the one that solves each individually. Therefore, they train a complex GAN-based model (OneGAN) to solve several tasks simultaneously, including foreground segmentation. Although LR-GAN and FineGAN do produce masks as part of their generative process, they cannot segment a given image. Despite SEIGAN and OneGAN achieving decent performance on foreground-background segmentation, these methods require a set of clean background images as additional inputs for weak supervision. ReDO captures the foreground objectness with possibly oversimplified assumptions, limiting its application to datasets with diverse object shapes.
On another front, compositional generative scene models [32–40], sharing the idea of scene decomposition stemming from DRAW [50], learn to represent foreground objects and background regions in terms of a collection of latent variables with the same representational format. These methods typically exploit the spatial mixture model for generative modeling. Specifically, IODINE [37] proposes a slot-based object representation method and models the latent space using iterative amortized inference [51]. Slot-Attention [38], as a step forward, effectively incorporates the attention mechanism into the slot-based object representation for flexible foreground object binding. Both methods use fully shared parameters among individual mixture components to entail permutation invariance of the learned multi-object representation. Alternative models such as MONet [36] and GENESIS [39] use multiple encode-decode steps for scene decomposition and foreground object extraction. Although these methods exhibit strong performance on synthetic multi-object datasets with simple background and foreground shapes, they may fail when dealing with complex real-world data or even synthetic datasets with more challenging background [37, 38].
More closely related to the classical methods, another line of work focuses on utilizing image features extracted by deep neural networks or designing energy functions based on data-driven methods to define the desired property of foreground objects. Pham et al. [52] and Silberman et al. [53] obtain impressive results when depth images are accessible in addition to conventional RGB images, while such methods are not directly applicable for data with RGB images alone. W-Net [19] extracts image features via a deep auto-encoder jointly trained by minimizing reconstruction error and normalized cut. The learned features are further processed by CRF smoothing to perform hierarchical segmentation. Kanezaki [20] proposes to employ a neural network as part of the partitioning criterion (inspired by Ulyanov et al. [54]) to minimize the chosen intra-region pixel distance for segmentation directly. Ji et al. [21] propose to use Invariant Information Clustering as the objective for segmentation, where the network is trained to be part of the learned distance. As an interesting extension, one may also consider adapting methods that automatically discover object structures [55] to foreground extraction. Though being pioneering work in image segmentation, the aforementioned methods are generally bottle-necked by the selected post-processing segmentation algorithm or require extra transformations to produce meaningful foreground segmentation masks. Yang et al. [24, 25] in their seminal work propose an information-theoretical principle and adversarial contextual model for unsupervised segmentation and detection by partitioning images into maximally independent sets, with the objective of minimizing the predictability of one set by the other sets. Additional efforts have also been devoted to weakly supervised foreground segmentation using image classification labels [56–58], bounding boxes [59, 60], or saliency maps [61–63].

heading: 3 Methodology
text: Foreground extraction performs a binary partition for the image I to extract the foreground region. Without explicit supervision, we propose to use learned pixel re-assignment as a generic inductive bias for background modeling, upon which we derive an EM-like partitioning algorithm. Compared with prior methods, our algorithm can handle images with more complex foreground shapes and background patterns, while providing explicit identification of foreground and background regions.

heading: 3.1 Preliminaries
text: Adopting the language of EM algorithm, we assume that for the observed sample x P RD, there exists z P Rd as its latent variables. The complete-data distribution is
pθpz,xq “ pαpzqpβpx|zq, (1)
where pαpzq is the prior model with parameters α, pβpx|zq is the top-down generative model with parameters β, and θ “ pα, βq. The prior model pαpzq can be formulated as an energy-based model, which we refer to as the Latentspace Energy-Based Model (LEBM) [47] throughout the paper:
pαpzq “ 1
Zα exp pfαpzqq p0pzq, (2)
where fαpzq can be parameterized by a neural network, Zα is the partition function, and p0pzq is a reference distribution, assumed to be isotropic Gaussian prior commonly used for the generative model. The prior model in Eq. (2) can be interpreted as an energy-based correction or exponential tilting of the original prior distribution p0.
The LEBM can be learned by Maximum Likelihood Estimation (MLE). Given a training sample x, the learning gradient for α is derived as shown by Pang et al. [47],
δαpxq “ Epθpz|xq r∇αfαpzqs ´ Epαpzq r∇αfαpzqs . (3)
In practice, the above expectations can be approximated by Monte-Carlo average, which requires sampling from pθpz|xq and pαpzq. This step can be done with stochastic gradient-based methods, such as Langevin dynamics [64] or Hamiltonian Monte Carlo [65].
An extension to LEBM is to further couple the vector representation z with a symbolic representation y [66]. Formally, y is a K-dimensional one-hot vector, where K is the number of possible z categories. Such symbol-vector duality can provide extra entries for auxiliary supervision; we will detail it in Section 3.4.

heading: 3.2 Generative Image Modeling
text: Mixture of Experts (MoE) for Image Generation Inspired by the regional homogenity assumption proposed by Zhu and Yuille [1], we use separate priors and generative models for foreground and background regions, indexed as αk and βk, k “ 1, 2, respectively; see Fig. 1. This design leads to the form of MoE [48, 49] for image modeling, as shown below.
Let us start by considering only the i-th pixel of the observed image x, denoted as xi. We use a binary one-hot random variable wi to indicate whether the i-th pixel belongs to the foreground region. Formally, we have wi “ rwi1, wi2s, wik P t0, 1u and ř2 k“1 wik “ 1. Let wi1 “ 1 indicate that the i-th pixel xi belongs to the foreground, and wi2 “ 1 indicate the opposite. We assume that the distribution of wi is prior-dependent. Specifically, the mixture parameter πik, k “ 1, 2, is defined as the output of a gating function πik “ pβpwik “ 1|zq “ Softmaxplikq; lik “ hβkpzkq, k “ 1, 2 are the logit scores given by the foreground and background generative models respectively; β “ tβ1, β2u, z “ tz1, z2u. Taken together, the joint distribution of wi is
pβpwi|zq “ 2 ź
k“1 πwikik . (4)
The learned distribution of foreground and background contents are
pβpxi|wik “ 1, zkq “ pβkpxi|zkq „ Npgβkpzkq, σ2Iq, k “ 1, 2 (5)
where we assume that the generative model for region content, pβkpxi|zkq, k “ 1, 2, follows a Gaussian distribution parameterized by the generator network gβk . As in VAE, σ takes an assumed value. We follow the common practice and use a shared generator for parameterizing πik and pβkpxi|zkq. We use separate branches only at the output layer to generate logits and contents.
Generating xi based on wi’s distribution involves two steps: (1) sample wi from the distribution pβpwi|zq, and (2) choose either the foreground model (i.e., pβ1pxi|z1q) or the background model (i.e., pβ2pxi|z2q) to generate xi based on the sampled wi. As such, this distribution of xi is a MoE,
pβpxi|zq “ 2 ÿ
k“1 pβpwik “ 1|zqpβpxi|wik “ 1, zkq “
2 ÿ k“1 πikpβkpxi|zkq, (6)
wherein the posterior responsibility of wik is
γik “ ppwik “ 1|xi, zq “ πikpβkpxi|zkq
ř2 m“1 πimpβmpxi|zmq
, k “ 1, 2. (7)
Using a fully-factorized joint distribution of x, we have pβpx|zq “ śD i“1 ř2 k“1 πikpβkpxi|zkq as the generative modeling of x P RD.
Learning Pixel Re-assignment for Background Modeling We use pixel re-assignment in the background generative model as the essential inductive bias for modeling the background region. This is partially inspired by the concepts of “texture” and “texton” by Julez [45, 67], where the textural part of an image may contain fewer structural elements in preattentive vision, which coincides with our intuitive observation of the background regions.
We use a separate pair of energy-based prior model αpix and generative model βpix to learn the re-assignment. For simplicity, we absorb αpix and βpix in the models for background modeling, i.e., α2 and β2, respectively. In practice, the re-assignment follows the output of βpix, a shuffling grid with the same size of the image x. Its values indicate the re-assigned pixel coordinates; see Fig. 2. We find that shuffling the background pixels using the learned re-assignment facilitates the model to capture the regularities of the background regions. Specifically, the proposed model with this essential in-
ductive bias learns to constantly give the correct mask assignment, whereas most previous fully unsupervised methods do not provide explicit identification of the foreground and background regions; see discussion in Section 4.1 for more details.

heading: 3.3 Deep Region Competition: from Generative Modeling to Foreground Extraction
text: The complete-data distribution from the image modeling is
pθpx, z,wq “ pβpx|w, zqpβpw|zqpαpzq
“ ˜ D ź
i“1
2 ź k“1 pβkpxi|zkqwik
¸ ˜
D ź
i“1
2 ź k“1 πwikik
¸
pαpzq
“ pαpzq D ź
i“1
2 ź k“1 pπikpβkpxi|zkqq wik ,
(8)
where pαpzq “ pα1pz1qpα2pz2q is the prior model given by LEBMs. α “ tα1, α2u, and θ “ tα, βu. w is the vector of pwiq, i “ 1, ...D, whose joint distribution is assumed to be fully-factorized. Next, we derive the complete-data log-likelihood as our learning objective:
Lpθq “ log pθpx, z,wq “ log pαpzq ` D ÿ
i“1
2 ÿ k“1 wik plog πik ` log pβkpxi|zkqq . (9)
Of note, w and z are unobserved variables in the modeling, which makes it impossible to learn the model directly through MLE. To calculate the gradients of θ, we instead optimize Ez„ppz|xq,w„ppw|x,zqrLpθqs based on the fact that underlies the EM algorithm:
∇θ log pθpxq “ ż
z
pθpz|xqdz ż
w
pθpw|z,xq∇θ log pθpx, z,wqdw
“ Ez„pθpz|xq,w„pθpw|x,zqr∇θ log pθpx, z,wqs. (10)
Therefore, the derived surrogate learning objective becomes
max θ
Ez„pθpz|xq rJ pθqs , s.t. @i, 2 ÿ
k“1 πik “ 1, (11)
J pθq “ log pαpzq looomooon
objective for LEBM
` D ÿ
i“1
2 ÿ k“1 γik log πik looooooooomooooooooon
foreground-background partitioning
` D ÿ
i“1
2 ÿ k“1 γik log pθkpxi|zkq loooooooooooooomoooooooooooooon
objective for image generation
, (12)
where J pθq “ Ew„pθpw|x,zq rLpθqs is the conditional expectation of w, which can be calculated in closed form; see the supplementary material for additional details.
Eq. (11) has an intuitive interpretation. We can decompose the learning objective into three components as in Eq. (12). In particular, the second term
řD i“1 ř2 k“1 γik log πik has a similar form to the
cross-entropy loss commonly used for supervised segmentation task, where the posterior responsibility γik serves as the target distribution. It is as if the foreground and background generative models compete with each other to fit the distribution of each pixel xi. If the pixel value at xi fits better to the distribution of foreground, pβ1pxi|z1q, than to that of background, pβ2pxi|z2q, the model tends to assign that pixel to the foreground region (see Eq. (7)), and vice versa. This mechanism is similar to the process derived in Zhu and Yuille [1], which is the reason why we coin our method Deep Region Competition (DRC).
Prior to our proposal, several methods [1, 37, 38] also employ mixture models and competition among the components to perform unsupervised foreground or image segmentation. The original Region Competition [1] combines several families of image modeling with Bayesian inference but is limited by the expressiveness of the pre-specified probability distributions. More recent methods, including IODINE [37] and Slot-attention [38], learn amortized inference networks for latent variables and induce the independence of foreground and background representations using an identical generator. Our method combines the best of the two worlds, reconciling the expressiveness of learned generators with the regularity of generic texture modeling under the framework of LEBM.
To optimize the learning objective in Eq. (11), we approximate the expectation by sampling from the prior pαpzq and posterior model pθpz|xq9pαpzqpβpx|zq, followed by calculating the Monte Carlo average. We use Langevin dynamics [64] to draw persistent MCMC samples, which iterates
zt`1 “ zt ` s∇z logQpztq ` ? 2sϵt, (13)
where t is the Langevin dynamics’s time step, s the step size, and ϵt the Gaussian noise. Qpzq is the target distribution, being either pαpzq or pθpz|xq. ∇z logQpztq is efficiently computed via automatic differentiation in modern learning libraries [68]. We summarize the above process in Algorithm 1.
Algorithm 1: Learning models of DRC via EM. Input: Learning iterations T , initial parameters for LEBMs αp0q “ tαp0q1 , α p0q 2 u and generators
βp0q “ tβp0q1 , β p0q 2 u, θp0q “ tαp0q, βp0qu, learning rate ηα for LEBMs, ηβ for foreground and background generators, observed examples txpiquNi“1, batch size M , and initial latent variables tzpiq´ “ tz piq 1´, z piq 2´u „ p0pzquNi“1 and tz piq ` “ tz piq 1`, z piq 2`u „ p0pzquNi“1.
Output: θpT q “ tαpT q1 , β pT q 1 , α pT q 2 , β pT q 2 u.
1 for t “ 0 : T ´ 1 do 2 Sample a minibatch of data txpiquMi“1; 3 Prior sampling for learning LEBMs: For each xpiq, update zpiq´ using Eq. (13), with target distribution πpzq “ pαptq pzq; 4 Posterior sampling for foreground and background generation: For each xpiq, update zpiq` using Eq. (13), with target distribution Qpzq “ pθptq pz|xq; 5 Update LEBMs: αpt`1q “ αptq ` ηα 1m řm i“1r∇αfαptq pz piq ` q ´ ∇αfαptq pz piq ´ qs; 6 Update foreground and background generators: βpt`1q “ βptq ` ηβ 1m řm i“1 ∇β log pβptq px piq|zpiq` q;
During inference, we initialize the latent variables z for MCMC sampling from Gaussian white noise and run only the posterior sampling step to obtain z`. The inferred mask and region images are then given by the outputs of generative models pβkpw|z`q and pβkpx|z`q, k “ 1, 2, respectively.

heading: 3.4 Technical Details
text: Pseudo label for additional regularization Although the proposed DRC explicitly models the interaction between the regions, it is still possible that the model converges to a trivial extractor, which treats the entire image as the foreground or background region, leaving the other region null. We exploit the symbolic vector y emitted by the LEBM (see Section 3.1) for additional regularization. The strategy is similar to the mutual information maximization used in InfoGAN [69]. Specifically, we use the symbolic vector y inferred from z as the pseudo-class label for z and train an auxiliary classifier jointly with the above models; it ensures that the generated regions xk contain similar symbolic information for zk. Intuitively, this loss prevents the regions from converging to null since the symbolic representation yk would never be well retrieved if that did happen.
Implementation We adopt a similar architecture for the generator as in DCGAN [70] throughout the experiments and only change the dimension of the latent variables z for different datasets. The generator consists of a fully connected layer followed by five stacked upsample-conv-norm layers. We replace the batch-norm layers [71] with instance-norm [72] in the architecture. The energy-term in LEBM is parameterized by a 3-layered MLP. We adopt orthogonal initialization [73] commonly used in generative models to initialize the networks and orthogonal regularization [74] to facilitate training. In addition, we observe performance improvement when adding Total-Variation norm [75] for the background generative model. More details, along with specifics of the implementation used in our experiments, are provided in the supplementary material.

heading: 4 Experiments
text: We design experiments to answer three questions: (1) How does the proposed method compare to previous state-of-the-art competitors? (2) How do the proposed components contribute to the model performance? (3) Does the proposed method exhibit generalization on images containing unseen instances (i.e., same category but not the same instance) and even objects from novel categories?
To answer these questions, we evaluate our method on five challenging datasets in two groups: (1) Caltech-UCSD Birds-200-2011 (Birds) [76], Stanford Dogs (Dogs) [77], and Stanford Cars (Cars) [78] datasets; (2) CLEVR6 [79] and Textured Multi-dSprites (TM-dSprites) [80] datasets. The first group of datasets covers complex real-world domains, whereas the second group features environments of the multi-object foreground with challenging spatial configurations or confounding backgrounds. As to be shown, the proposed method is generic to various kinds of input and produces more competitive foreground-background partition results than prior methods.

heading: 4.1 Results on Foreground Extraction
text: Single object in the wild In the first group of datasets, there is typically a single object in the foreground, varying in shapes, texture, and lighting conditions. Unsupervised foreground extraction on these datasets requires much more sophisticated visual cues than colors and shapes. Birds dataset consists of 11,788 images of 200 classes of birds annotated with high-quality segmentation masks, Dogs dataset consists of 20,580 images of 120 classes annotated with bounding boxes, and Cars dataset consists of 16,185 images of 196 classes. The latter two datasets are primarily made for fine-grained categorization. To evaluate foreground extraction, we follow the practice in Benny and Wolf [31], and approximate ground-truth masks for the images with Mask R-CNN [16], pre-trained on the MS COCO [9] dataset with a ResNet-101 [81] backend. The pre-trained model is acquired from the detectron2 [82] toolkit. This results in 5,024 dog images and 12,322 car images with a clear foreground-background setup and corresponding masks.
On datasets featuring a single foreground object, we use the 2-slot version of IODINE and Slotattention. Since ReDO, IODINE, and Slot-Attention do not distinguish foreground and background in output regions, we choose the best-matching scores from the permutation of foreground and background masks as in [28]. We observe that the proposed method and Grabcut are the only two methods that provide explicit identification of foreground objects and background regions. While the Grabcut algorithm actually requires a predefined bounding box as input that specifies the foreground region, our method, thanks to the learned pixel re-assignment (see Section 3.2), can achieve this in a fully unsupervised manner. Results in Table 1 show that our method outperforms all the unsupervised baselines by a large margin, exhibiting comparable performance even to the weakly supervised baseline that requires additional background information as inputs [31]. We provide samples of foreground extraction results as well as generated background and foreground regions in Fig. 3. Note that our final goal is not to synthesize appealing images but to learn foreground extractors in a fully unsupervised manner. As the limitation of our method, DRC generates foreground and background regions less realistic than those generated by state-of-the-art GANs, which hints a possible direction for future work. More detailed discussions of the limitation can be found in supplementary material.
Multi-object scenes The second group of datasets contains images with possibly simpler foreground objects but more challenging scene configurations or background parts. Visual scenes in the CLEVR6 dataset contain various objects and often with partial occlusions and truncations. Following the evaluation protocol in IODINE and Slot-attention, we use the first 70K samples from CLEVR [79] and filter the samples for scenes with at most 6 objects for training and evaluation, i.e., CLEVR6. The TM-dSprites dataset is a variant of Multi-dSprites [80] but has strongly confounding backgrounds borrowed from Textured MNIST [32]. We generate 20K samples for the experiments. Similar to Greff et al. [37] and Locatello et al. [38], we evaluate on a subset containing 1K samples for testing. Note that IODINE and Slot-attention are designed for segmenting complex multi-object scenes using slot-based object representations. Ideally, the output of these models consists of masks for each individual object, while the background is viewed as a virtual “object” as well. In practice, however, it is possible that the model distributes the background over all the slots as mentioned in Locatello et al. [38]. We therefore propose two corresponding approaches (see the supplementary material for more details) to convert the output object masks into a foreground-background partition and report the best results of these two options for IODINE and Slot-attention in Table 1.
On the CLEVR6 dataset, we use the publicly available pretrained model for IODINE, which achieves a reasonable ARI (excluding background pixels) of 94.4 on the testing data, close to the testing results in Greff et al. [37]. We observe that IODINE distributes the background over all the slots for some of the testing samples, resulting in much lower IoU and Dice scores. We re-train the Slot-attention model using the official implementation on CLEVR6, as no pretrained model is publicly available. The re-trained model achieves a foreground ARI of 98.0 on the 1K testing samples, which we consider as a sign of valid re-implementation. Results in Table 1 demonstrate that the proposed method can effectively process images of challenging multi-object scenes. To be specific, our method demonstrates competitive performance on the CLEVR6 dataset compared with the SOTA object discovery method. Moreover, as shown empirically in Fig. 3, the proposed method can handle the strongly confounding background introduced in Greff et al. [32], whereas previous methods are distracted by the background and mostly fail to capture the foreground objects.
4.2 Ablation Study Model IoU Dice
amortized inference˚ - - w/o pix. re-assign. 21.8 35.3 w/o pseudo label 48.7 64.2 w/o TV-norm reg. 53.0 68.1 w/o ortho. reg. 54.3 69.2 short-run chain: 52.5 67.7 Full model 56.4 70.9
Table 2: Ablation study on Birds. *We replace the LEBM with encoders to perform amortized inference for the latent variables z within a variational framework as in VAE [83]. †We explore the possibility of using short-run MCMC [84] instead of persistent chain sampling.
We provide ablation studies on the Birds dataset to inspect the contribution of each proposed component in our model. As shown in Table 2, we observe that replacing the LEBMs in the foreground and background models with amortized inference networks significantly harms the performance of the proposed method. In particular, the modified model fails to generate any meaningful results (marked as - in Table 2). We conjecture that LEBM benefits from the low-dimensionality of the latent space [47] and therefore enjoys more efficient learning. However, the inference networks need to learn an extra mapping from the high-dimensional image space to the latent space and require more elaborate architecture and tuning for convergence. Furthermore, we observe that the
model that does not learn pixel re-assignment for background can still generate meaningful images but only vaguely captures masks for foreground extraction.

heading: 4.3 Generalizable Foreground Extraction
text: Extracting novel foreground objects from training categories We show results on generalizing to novel objects from the training classes. To evaluate our method, we split the Birds dataset following Chen et al. [28], resulting in 10K training images and 1K testing images. On Dogs and Cars datasets, we split the dataset based on the original train-test split [77, 78]. This split gives 3,286 dog images and 6,218 car images for training, and 1,738 dog images and 6,104 car images for testing, respectively. As summarized in Table 3, our method shows superior performances compared with baselines; the performance gap between training and testing is constantly small over all datasets.
Extracting novel foreground objects from unseen categories To investigate how well our method generalizes to categories unseen during training, we evaluate the models trained in real-world single object datasets on the held-out testing data from different categories. We use the same training and testing splits on these datasets as in the previous experiments. Table 4 shows that our method outperforms the baselines on the Birds dataset when the model has trained on Dogs or Cars dataset, which have quite different distributions in foreground object shapes. Competitors like ReDO also exhibit such out-of-distribution generalization but only to a limited extent. Similar results are observed when using Dogs or Cars as the testing set. We can see that when the model is trained on Dogs and evaluated on Cars or vice versa, it still maintains comparable performances w.r.t. those are trained&tested on the same class. We hypothesize that these two datasets have similar distributions in fore-
ground objects and background regions. In the light of this, we can further entail that the distribution of Dogs is most similar to that of Cars and less similar to that of Birds according to the results, which is consistent with our intuitive observation of the data. We provide a preliminary analysis of the statistics of these datasets in the supplementary material.

heading: 5 Conclusion
text: We have presented the Deep Region Competition, an EM-based fully unsupervised foreground extraction algorithm fueled by energy-based prior and generative image modeling. We propose learned pixel re-assignment as an inductive bias to capture the background regularities. Experiments demonstrate that DRC exhibits more competitive performances on complex real-world data and challenging multi-object scenes. We show empirically that learned pixel re-assignment helps to provide explicit identification for foreground and background regions. Moreover, we find that DRC can potentially generalize to novel foreground objects even from categories unseen during training. We hope our work will inspire future research along this challenging but rewarding research direction.

heading: Acknowledgements
text: The work was supported by NSF DMS-2015577, ONR MURI project N00014-16-1-2007, and DARPA XAI project N66001-17-2-4029. We would like to thank Bo Pang from the UCLA Statistics Department for his insights on the latent-space energy-based model and four anonymous reviewers (especially Reviewer RcgA) for their constructive comments.

Review summaries:
The paper tackles the problem of unsupervised foreground segmentation with a combination of energy-based and deep-learning-based models. It models the foreground and background as two components that can be selected by a latent variable. The background has an inductive bias that pixels can be re-assigned to a different location. The whole model is generative. During inference, EM is needed to figure out the foreground and background separation by maximizing probability to generate the image and set the right latent states.

In this work, the authors propose an unsupervised, generative model for foreground-background segmentation of RGB images which is based on the previously proposed "Latent Space Engergy-Based Prior Model" (LEBM). Similar to a VAE, a latent vector sampled from a prior distribution is transformed into an image by a decoder network. Differently from VAEs however, the LEBM uses a more complex, learnable prior distribution and sampling from the prior and posterior distributions is realized by differentiable MCMC sampling using Langevin dynamics. An image is modelled using a spatial mixture of foreground and background images decoded by separate decoder networks. To foster the segregation into foreground and background, the backround model includes an additional, spatial resampling step targeted at the assumed background statistics. While the reconstructed images tend to be blurry, the reconstructed foreground-background masks have been shown to be more accurate than previous methods on several datasets.

This work proposes a novel unsupervised foreground segmentation technique based on Mixture of Experts energy based modelling with a generative neural network. They introduce an interesting inductive bias, pixel reassignment, which works as the main regularizer and the secret sauce that enables the fg/bg separation. They essentially reshuffle the pixels for the background generation based on the learned index matrix and ablations shows that's the most significant regularizer.  They compare their results on 6 datasets, including birds, dogs, and clevr6 to both object centric approaches such as slot attention and gan based methods. They consistently outperform the previous methods with a significant margin.

This paper proposes deep region competition to extract foreground objects from images in an unsupervised manner. The proposed method leverages energy-based prior with generative image modeling in the form of Mixture of Experts. In addition, a pixel re-assignment scheme is further learned to capture the regularities of background regions. Experiments demonstrate that the proposed method can achieve promising results on real-world images.