Unsupervised Foreground Extraction via Deep Region Competition
Authors: Peiyu Yu
Sirui Xie
Xiaojian Ma
Yixin Zhu
Ying Nian Wu
Song-Chun Zhu
Venue: NeurIPS 2021 Poster (2021)
Keywords: unsupervised learning
foreground extraction
generative model
generalization

Abstract: We present Deep Region Competition (DRC), an algorithm designed to extract foreground objects from images in a fully unsupervised manner. Foreground extraction can be viewed as a special case of generic image segmentation that focuses on identifying and disentangling objects from the background. In this work, we rethink the foreground extraction by reconciling energy-based prior with generative image modeling in the form of Mixture of Experts (MoE), where we further introduce the learned pixel re-assignment as the essential inductive bias to capture the regularities of background regions. With this modeling, the foreground-background partition can be naturally found through Expectation-Maximization (EM). We show that the proposed method effectively exploits the interaction between the mixture components during the partitioning process, which closely connects to region competition, a seminal approach for generic image segmentation. Experiments demonstrate that DRC exhibits more competitive performances on complex real-world data and challenging multi-object scenes compared with prior methods. Moreover, we show empirically that DRC can potentially generalize to novel foreground objects even from categories unseen during training.

Unsupervised Foreground Extraction via Deep Region Competition
Peiyu Yu yupeiyu980cs. ucla. edu
Sirui Xie srxieducla.edu
Xiaojian Ma' xiaojian www.mla edu
Yixin Zhu y@bigai ai
Ving Nian Wu² ywußstat. ucla.clloooooooo edu
Song-Chun Zhu 1.2.3 sczhu@stat. ucla. edu
UCLA Department of Computer Science *Beijing Institute for General Artificial Intelligence (BIGAI)
UCCA Department of Statistics
Abstract
We present Deep Region Competition (DRC), algorithm designed to extract foreground objects from images in a fully unsupervised manner. Foreground ex- traction can be viewed special case of generic image segmentation that focuses on identifying and disentangling objects from the background. this work, we rethink the foreground extraction by reconciling energy-based prior with genera- tive image modeling in the form of Mixture of Experts (MoE), where we further the essential inductive bias to cap-
introduce the learned pixel re-assignment ture the regularities of background regions. With this modeling, the foreground- background partition can be naturally found through popectaaioonMxximizattti (EM). We show that the proposed method effectively exploits the interaction be- tween the mixture components during the partitioning process, which closely con- nects to region competition [I] seminal approach for generic image segmenta- tion. Experiments demonstrate that DRC exhibits more competitive performances complex real-world data and challenging multi-object scenes compared with prior methods. Moreover, we show empirically that DRC can potentially general- ize to novel foreground objects even from categories unseen during training
Introduction
Foreground extraction, being special case of generic image segmentation, aims for binary par- tition the given image with specific semantic meaning, i.e., a foreground that typically contains identifiable objects and the possibly less structural remaining regions as the background. There is rich literature on explicitly modeling and representing given image as foreground and background (or more general visual regions), such that generic inference algorithm mentations ideally for any images without or with little supervision (T-). However, such methods essentially rely low-level visual features (e.g., edges, color, and texture), and some further quire human intervention at initialization 4 (). which largely limits their practical performance on modern datasets of complex natural images with rich semantic meanings 10) These datasets typically come with fine-gra mee-graieed semantic annotations, exploited by supervised methods that learn representation and inference algorithm as one monolithic network (1166).. Despite the success of densely supervised learning, the unsupervised counterpart still favored due to its resemblance to how humans perceive the world [7 [8]
produce plausible seg-
Code and data available at https://ggitbb.ooomyyuulyyyysggr
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Attempting to combine unsupervised or weakly supervised learning with modern neural networks, three lines of work surge recently for foreground extraction: (1) deep networks as feature extractors for canonical segmentation algorithms, (2) GAN-based
disentanglement, and (3) compositional latent variable models with slot-based object modeling. Despite great suc- cesses of these methods, the challenge of unsupervised foreground extraction remains largely open.
Specifically, the first line of work trains designated deep feature extractors for canonical segmen- lation algorithms or metric networks as learned partitioning criteria [1921]] These methods (e.g., W-Net (19) define foreground objects' properties using learned features or criteria and are thus gen- erally neckle by the selected post-processing segmentation algorithm 222 23 As a branch of pioneering work that moves beyond these limitations, Yang et al. [24 have recently proposed a general contextual information separation principle and an efficient adversarial learning method that generally applicable to unsupervised segmentation, separation and detection. GAN-based models [26/31 capture the foreground objectness with oversimplified assumptions or require addi- tional supervision to achieve foreground-backgrrundd disentanglement. For example, the segmenta- tion model in ReDO [28] is trained by redrawing detected objects, which potentially limits its ap- plication to datasets with diverse object shapes. OneGAN [31] and its predecessors [29] 30), though producing impressive results on foreground extraction, require set of background images with- out foreground objects as additional inputs. Lastly, compositional latent variable models [32 include the background as a "virtual object" and induce the independence of object representations using an identical generator for all object slots. Although these methods exhibit strong performance on synthetic multi-object datasets with simple backgrounds and foreground shapes, they may fail on complex real-world data or even synthetic datasets with more challenging backgrounds (37 38
In addition, few unsupervised learning methods have provided explicit identification of foreground objects and background regions. While they can generate valid segmentation masks, most of these methods do not specify which output corresponds to the foreground objects. These deficiencies ne- cessitate rethinking the problem of unsupervised foreground extraction. We propose to confront the challenges in formulating (1) generic inductive bias for modeling foreground and background re- gions that can be baked into neural generators, and (2) an effective inference algorithm based on a partition.
principled criterion for breground-backgrreg
Inspired by Region Competition [.. a seminal approach that combines optimization- infer- ence 43 and probabilistic visual modeling 44 45 by minimizing a generalized Bayes cri-
terion 46), we propose to solve the foreground extraction problem by reconciling energy-based
prior 47 with generative image modeling in the form of Mixture of Experts (MoE) 48 49]. To
generically describe background regions, we further introduce the learned pixel re-assignment as the essential inductive bias to capture their regularities. Fueled by our modeling, we propose find partition through Expectation- Expectation-Maxxiizaatee Maximizaaionnn (EM). Our algorithm ef- fectively exploits the interaction between the mixture components during the partitioning process, echoing the intuition described in Region Competition [.. We therefore coin our method Deep Region Competition (DRC). We summarize our contributions as follows:
the foreground-backgrownn
1. We provide probabilistic srngrouudd-bccgground modeling by reconciling energy-based prior with generative image modeling in the form of MoE. With this modeling, the foreground- background partition can be naturally produced through EM. We further introduce an inductive bias, pixel re-assigmment, to facilitate srngrounnd-ackgrooun disentanglement.
2. In experiments, we demonstrate that DRC exhibits more competitive performances on complex real-world data and challenging multi-object scenes compared with prior methods. Furthermore, we empirically show that using learned pixel re-assignment as the inductive bias helps to provide explicit identification for foreground and background regions.
3. We find that DRC can potentially generalize to novel foreground objects even from categories unseen during training, which may provide some inspiration for the study of out-of-distribbtion (OOD) generalization in more general unsupervised disentanglement.
2 Related Work
A typical line of methods frames unsupervised or weakly supervised foreground segmentation within a generative modeling context. Several methods build upon generative adversarial networks (GAN) (26) to perform foreground segmentation. LR-GAN [27] learns to generate background re-

---

gions and foreground objects separately and recursively, which simultaneously produces the fore- ground objects mask. ReDO (ReDrawing of Objects) [28] proposes GAN-based object segmen- tation model, based on the assumption that replacing the foreground object in the image with generated one does not alter the distribution of the training data, given that the foreground object is correctly discovered. Similarly, SEIGAN (29] learns to extract foreground objects by recombining the foreground objects with the generated background regions. FineGAN (30 hierarchically gen- erates images (i.e., first specifying the object shape and then the object texture) to disentangle the background and foreground object. Benny and Wolf [3]] further hypothesize that a method solving an ensemble of unsupervised tasks altogether improves the model performance compared with the one that solves each individually. Therefore, they train a complex GAN-based model (OneGAN) to solve several tasks simultancously, including foreground segmentation. Although LR-GAN and FineGAN do produce masks as part of their generative process, they cannot segment given image. Despite SEIGAN and OneGAN achieving decent performance on foreground backundudd segmen- tation, these methods require a set of clean background images as additional inputs for weak super- vision. ReDO captures the foreground objectness with possibly oversimplified assumptions, limiting its application to datasets with diverse object shapes.
On another front, compositional generative scene models
sharing the idea of scene decom- position stemming from DRAW [50], learn to represent foreground objects and background regions in terms of collection of latent variables with the same representational format. These methods typ- ically exploit the spatial mixture model for generative modeling. Specifically, IODINE 337 proposes slot-based object representation method and models the latent space using iterative amortized in- ference [51] Slot-Attention [38], as step forward, effectively incorporates the attention mechanism into the slot-based object representation for flexible foreground object binding. Both methods use fully shared parameters among individual mixture components to entail permutation invariance of the learned multi-object representation. Alternative models such as MONet (36) and GENESIS [39] use multiple encode-decode steps for scene decomposition and foreground object extraction. Al- though these methods exhibit strong performance on synthetic multi-object datasets with simple background and foreground shapes. they may fail when dealing with complex real-world data or even synthetic datasets with more challenging background [37 38).
More closely related to the classical methods, another line of work focuses on utilizing image fea- tures extracted by deep neural networks or designing energy functions based on data-driven methods to define the desired property of foreground objects. Pham et al. (52) and Silberman et al. [53 ob- tain impressive results when depth images accessible in addition conventional RGB images, while such methods are not directly applicable for data with RGB images alone. W-Net [9 extracts image features via a deep auto-encoder jointly trained by minimizing reconstruction error and nor- malized cut. The learned features are further processed by CRF smoothing to perform hierarchical segmentation. Kanezaki [20] proposes to employ a neural network as part of the partitioning criterion (inspired by Ulyanov et al. (54)) to minimize the chosen intra-region pixel distance for segmentation directly. Ji et al. (21) propose to use Invariant Information Clustering as the objective for segmen- tation, where the network is trained to be part of the learned distance. As an interesting extension, one may also consider adapting methods that automatically discover object structures (55) to fore- ground extraction. Though being pioneering work in image segmentation, the aforementioned meth- ods are generally bottle-necked by the selected post-proct pestppoccssing segmentation algorithm or require extra transformations to produce meaningful foreground segmentation masks. Yang et al. [24]] 25 in their seminal work propose an information-theoree
principle and adversarial contextual model for unsupervised segmentation and detection by partitioning images into maximally independent sets, with the objective of minimizing the predictability of one set by the other sets. Additional efforts have also been devoted to weakly supervised foreground segmentation using image classification labels (86)58). bounding boxes [59 60), or saliency maps 66]633..
3 Methodology
Foreground extraction performs a binary partition for the image I to extract the foreground region. Without explicit supervision, we propose to use learned pixel re-assignment as generic inductive bias for background modeling, upon which we derive an EM-like partitioning algorithm. Compared with prior methods, our algorithm can handle images with more complex foreground shapes and background patterns, while providing explicit identification of foreground and background regions.
3

---

3.1 Preliminaries
Adopting the language of EM algorithm, we assume that for the observed sample x E RD, there exists E Rd as its latent variables. The complete-data distribution
po(z, x) Po (x)
(1)
where Pa (z) is the prior model with parameters a, (x(x is the top-down generative model with parameters 9, and 0 (ax,
The prior model (z) can be formulated as an energy-based model, which we refer to as the Latent- space Energy-Based Model (LEBM) 47 throughout the paper:
1 Za
(z(
exp (f)) pu(z),
(2)
where (z) can be parameterized by a neural network, Za is the partition function, and (z()) is reference distribution, assumed to be isotropic Gaussian prior commonly used for the generative model. The prior model in Eq. (2) can be interpreted as an energy- based correction or exponential tilting of the original prior distribution Po-
The LEBM can be learned by Maximum Likelihood Estimation (MLE). Given training sample x, the learning gradient for is derived as shown by Pang et al.
47
8a (x) (x(x))
[a)a(zz]
px)(2) V [z)].
In practice, the above expectations can be approximated by Monte-Carke average, which requires sampling from pa(z|x) x) and P. (z). This step can be done with stochastic gradient-based methods,
such Langevin dynamics [64] or Hamiltonian Monte Carlo [65]
An extension to LEBM is to further couple the vector representation z with a symbolic representa- tion y [66]. Formally, y is K-dimensional one-hot vector, where K is the number of possible categories. Such symbol- vector duality can provide extra entries for auxiliary supervision; we will detail it in Section 3.4
3.2 Generative Image Modeling
Mixture of Experts (MoE) for Image Generation Inspired by the regional homogenity assump- tion proposed by Zhu and Yuille [1] we use separate priors and generative models for foreground and background regions, indexed as og and 3r, 1, respectively; see Fig. [ This design leads to the form of MoE 48 49 for image modeling, as shown below.
Let us start by considering only the i-th pixel of the observed image x, denoted as x,, We use a binary one-hot random variable w; to indicate whether the i-th pixel belongs to the foreground region. Formally, we have w, = [wa, W122. Wa {0, 1} and Σ Wie = 1. Let wa1 = indicate that the i-th pixel x; belongs to the foreground, and W,2 = 1 indicate the opposite.
We assume that the distribution of w, is prior-dependent. Specifically, the mixture parameter Tik, A = 2, is defined as the output of gating function тi p,5(wa 1 Softmax (la);; Li ha, [zx), k = 1,2 are the logit scores given by the foreground and background generative models respectively; 3 {81, 32), z (z1, z2). Taken together, the joint distribution of w; is
Pg (3 (z)
T
(4)
The learned distribution of foreground and background contents are
( X Ww:k 1, 2x) = Ps. (x, (2.) ~ Nígs, (zt), o0I), h: = 1,2
(5)
where we assume that the generative model for region content, P9, [x; (ZE), k 1, 2, follows Gaus- sian distribution parameterized by the generator network ga,- As in VAE, takes an assumed value. We follow the common practice and use a shared generator for parameterizing wa and p3, (x, (2)) We use separate branches only at the output layer to generate logits and contents.

---

Iteration 0
Iteration
Foreground
update Z, 2,
softmax
Prn
Poz
LEBM
Pos
Pog
LEBM
Background
Z,
Appaant w...C... Normn
Piss Generator
Pás Generator
2
Pay (ua
Vlegpsdo k2)
(Polil Vieensfeakk)
Pa.
Pd
Mixture Regions
(a)
(b)
X2
Figure 1: Overview of DRC. (a) The model generates foreground and background regions using sampled latent variables {z1, m2). ; k 1,2 represents the generator for each region. Of note, the pixel assignment function absorbed
background generator; see Section [3.]] for details. (b) DRC samples the latent variables z in an iterative manner. Let x denote the observed image; we use &t, represent the image generated by pa (x(2) at the t-th sampling step. DRC has a two-step workflow for learning unsupervised foreground extractors that resembles the F- and M-step in the classic EM algorithm. In the E-step, it employs gradient-based MCMC sampling
O, 1,... to
infer the latent variables latent variables are updated this step. In the M-step, the sampled latent variables z are fed into model for image generation as shown in (a), where the generators are updated to minimize the reconstruction error.
shown in (b). Of note, only the
Generating x; based w;'s 's distribution involves two steps: (1) sample w, from the distribution pa( wilz), and (2) choose either the foreground model (i.e., Ps; (x; (3/2))) )) or the background model (i.e., Pag (x: (22)) to generate x; based the sampled w,. As such, this distribution of is MoF,
px(x112)=
Σ ps( (1z)ps(>;/10s) 1, 2k)
TaPPp (x, 2k);
(6)
wherein the posterior responsibility of wie is
TekPo (x, (k)
Yik p( wis 1|x;, z)
k 1,2.
2m== TimPam
(x; Zmm
Using a fully-factorized joint distribution of x, we have p& (x(z) the generative modeling of R.D
2 1= TiPP (x; (2x)) as
Learning Pixel Re-assignment for Background Modeling We pixel re-assignment in the back- ground generative model as the essential inductive bias for modeling the background region. This partially inspired by the concepts of "texture" and "texton" by Julez [45] 677, where the textural part of an image may contain fewer structural elements in preattentive vision, which coincides with our intuitive observation of the background regions.
Pixel coordinates
Background imagex pss (xlx)) before pixel re-asaignment
Apply the learned re-assignment
Grid sampling
(x. (on
We a separate pair of energy-based prior model Opix and generative model Spex to learn the re-assignment. For simplicity, we absorb Opix and 3ptx in the mod- els for background modeling, ie., 02 and 82. respec- tively. In practice, the re-assignment follows the output of Spix. shuffling grid with the same size of the image x. Its values indicate the re-assigned pixel coordinates; see Fig. 2 We find that shuffling the background pix- els using the learned re-assignment facilitates the model to capture the regularities of the background regions. Specifically, the proposed model with this essential in- ductive bias learns constantly give the correct mask assignment, whereas most previous fully unsupervised methods do not provide explicit identification of the foreground and background re- gions; see discussion in Section 4.1 for more details.
Final background Image
Figure Pixel re-assignment. The output of 3, can be viewed as a learned re-assignment of the original background pixels that follows the mapped grid P Pap.,s, (z, C). Note that the assignment function Pa Pa,.,, (z.-)) might not be injective. final background image is gen- erated via grid sampling.

---

3.3 Deep Region Competition: from Generative Modeling to Foreground Extraction
The complete-data distribution from the image modeling is
po (x, &, w) = (x/ z)pp((/z)pp. (w/z)pa(z))
D
(n
D
2
=
(x:
wis
Pa(z)
(8)
= (z)
(TAPA, (x.(z)))
where (z) Pa. (Z1) (Z2) is the prior model given by LEBMs. a = {a1, 02), and 0 = {a, 83.
w is the vector of (w:). 1, D. whose joint distribution assumed be fully-factorized.
Next, we derive the complete-dar log-likelihooe likkliiikoodd as our learning objective:
D Σ Σ
((0) = log (x 2, w) log (z) +
Wik (log Ti + log (x,(2)))
(9)
Of note, w and z are unobserved variables in the modeling, which makes it impossible to learn the model directly through MLE. To calculate the gradients of 0, we instead optimize Eg Du=>p(z)x), wwwpwwwwwwee (x.))//ex))(((() based on the fact that underlies the EM algorithm:
[[
Vo logg pox)) Po (z(x))) po(wl2, x log po 2, w)dw
(10)
[x=pp(z)x) ww-p Vo (x w)].
Therefore, the derived surrogate learning objective becomes
max E, Ex-po(alx) [J(0]] s.t. Vi,
Tik = 1,
(11)
I(0)-
log Pa
Tik log Tiik
Tik log (x, 22))
(12)
objective LEBM
foreground-hackgruund partitioning
objective for image generation
where J(6) Ew xw) [C(0)] is the conditional expectation of w, which can be calculated in closed form; see the supplementary material for additional details.
Eq. 11 has intuitive interpretation. We decompose the learning objective into three compo- nents as in Eq. (12). In particular, the second term > Tik log Ta& has similar form the cross-entropy loss commonly used for supervised segmentation task, where the posterior responsi- bility 7/k serves the target distribution. is as if the foreground and background generative models compete with each other fit the distribution of each pixel x,. If the pixel value at fits better to distribution foreground, ps,, (x,(2)), than to that of background, PR, (x, (22), the model tends
to assign that pixel to the foreground region (see Eq. (7)) and vice versa. This mechanism is similar to the process derived in Zhu and Yuille [I which the reason why we coin our method Deep Region Competition (DRC).
also employ mixture models and competition among the components to perform unsupervised foreground or image segmentation. The original Region Competition combines several families of image modeling with Bayesian inference but is limited by the expressiveness of the pre-specified probability distributions. More recent meth- ods, including IODINE [37 and Slot-attention [38], learn amortized inference networks for latent variables and induce the independence of foreground and background representations using iden- tical generator. Our method combines the best the two worlds, reconeiling the expressiveness of leamned generators with the regularity of generic texture modeling under the framework of LEBM.
Prior our proposal, several methods

---

To optimize the learning objective in Eq. I1 we approximate the expectation by sampling from the prior Pa (z) and posterior model x (x)),( followed by calculating the Monte Carlo average. We use Langevin dynamics 64 to draw persistent MCMC samples, which iterates
Z Ze
log ((z+) /2ser,
(13)
where f is the Langevin dynamics's ' time step, the step size, and e, the Gaussian noise. Q(z) is the target distribution, being either pa (z) or pe(z) (x) V, log Q(z+) is efficiently computed via automatic differentiation in modern learning libraries [68] We summarize the above process in Algorithm
Algorithm 1: Learning models of DRC via EM.
Input: Learning iterations T, initial parameters for LEBMs a (0) (0) (0) } and generators
(22
g = { .32 _ 00000 = "a learning rate for LEBMs, me for foreground and
background generators, observed examples {x(") 12=1, batch size M, and initial latent variables
[x = [z] ( po[z)] and {z {2 22.) } ~ po(z));-1-
Output: ((3) {a] 1 (T) ((
for 0: do
Sample minibatch of data (x MM
Prior sampling for learning LEBMs: For each x²)) update z using Eq. 13 with target distribution (2) Po,ce) (x);
Posterior sampling for foreground and background generation: For each (9) update x( using Eq. (3), with target distribution (() = Pate) (z x);
Update LEBMs: (t+1) (t) + 7a - - [Va.aaaaa (2(2) Vafaco (2
Update foreground and background generators:
g((+1) g0()) n) Σn V (o (2))
During inference, initialize the latent variables for MCMC sampling from Gaussian white noise and run only the posterior sampling step to obtain z.. The inferred mask and region images are then given by the outputs of generative models pa, (w (z)) and pa, x(zz),, k 1, 1, respectively.
3.4 Technical Details
Pseudo label for additional regularization Although the proposed DRC explicitly models the in- teraction between the regions, is still possible that the model converges a trivial extractor, which treats the entire image as the foreground or background region, leaving the other region null. We exploit the symbolic vector y emitted by the LEBB (see Section 3.T) for additional regularization.
The strategy is similar to the mutual information maximization used in InfoGAN (69) Specifically,
we use the symbolic vector y inferred from z as the pseudo-class label for z and train an auxiliary
classifier jointly with the above models; it ensures that the generated regions x contain similar symbolic information for zs. Intuitively, this loss prevents the regions from converging to null since the symbolic representation y& would never be well retrieved if that did happen.
Implementation We adopt similar architecture for the generator as in DCGAN [70] throughout the experiments and only change the dimension of the latent variables z for different datasets. The generator consists of a fully connected layer followed by five stacked upsample-conv-non We replace the batch-norm layers [71] with instance-nom 722 in the architecture. The energy-term in LEBM is parameterized by a 3-layee MLP. We adopt orthogonal initialization 73 commonly
layers.
used in generative models to initialize the networks and orthogonal regularization 74 to facilitate
training. In addition, we observe performance improvement when adding Variatvai norm [75) for the background generative model. More details, along with specifics of the implementation used in our experiments, are provided in the supplementary material.
4 Experiments
We design experiments to answer three questions: (1) How does the proposed method compare to previous state-of-the-ard competitors? (2) How do the proposed components contribute to the model performance? (3) Does the proposed method exhibit generalization on images containing unseen instances (i.e., same category but not the same instance) and even objects from novel categories?
7

---

To answer these questions, we evaluate our method on five challenging datasets in two groups: (I) CClD Birds-200-2011 (Birds) [76]. Stanford Dogs (Dogs) 777. and Stanford Cars (Cars) 78 datasets; (2) CLEVR6 79 and Textured Multi-dSprites (TM-dSprites) [80] datasets.
The first group of datasets covers complex real-world domains, whereas the second group features environments of the multi-object foreground with challenging spatial configurations or confounding backgrounds. As to be shown, the proposed method is generic to various kinds of input and produces
more competitive foreground-backgroun partition results than prior methods.
4.1 Results on Foreground Extraction
Single Object
Multi-Object
Model
Birds
Dogs
Cars
CLEVR6
TM-dSprites
IoU
Dice
IoU
Dice
IoU
Dice
IoU
Dice
loU
Dice
W-Net*
24.8
38.9
47.7
62.1
52.8
67.6
GrabCut
30.2
42.7
58.3
70.9
61.3
73.1
19.0
30.5
61.9
71.0
ReDO
46.5
60.2
55.7
70.3
52.5
68.6
18.6
31.0
9.4
17.2
OneGAN*
55.5
69.2
71.0
81.7
71.2
82.6
IODINE
30.9
44.6
54.4
67.0
51.7
67.3
19.9
32.4
7.3
12.8
Slot-Attn." 8
35.6
51.5
38.6
55.3
41.3
58.3
83.6
90.7
7.3
13.5
Ours
56.4
70.9
71.7
83.2
72.4
83.7
84.7
91.5
78.8
87.5
Table I: Foreground extraction results on training data measured IoU and Dice. Higher is better all scores. "Results of W-Net and OneGAN are provided by Benny and Wolf [31]. Of note, results of these two models on Dogs and Cars datasets may not be directly comparable to other listed methods, as the data used for training and evaluation could be different. We include these results as rough reference since no official implementation or pretrained model are publicly available. & indicates unfair baseline results obtained using extra ground truth information, i.e., we choose the best-matching scores from the permutation foreground and background masks. *OneGAN is a strong weakly supervised baseline, which requires clean background images to provide additional supervision. We include this model as potential upper bound of the fully unsu- pervised methods.
Single object in the wild In the first group of datasets, there is typically single object in the foreground, varying in shapes, texture, and lighting conditions. Unsupervised foreground extraction on these datasets requires much more sophisticated visual cues than colors and shapes. Birds dataset consists of 11,788 images of 200 classes of birds annotated with high-quality segmentation masks, Dogs dataset consists of 20,580 images of 120 classes annotated with bounding boxes, and Cars dataset consists of 16,185 images of 196 classes. The latter two datasets are primarily made for fine-grained categorization. To evaluate foreground extraction, we follow the practice in Benny and Wolf [3I] and approximate ground-truth masks for the images with Mask R- R-CNN [16), pre-trained on the MS COCO [9] dataset with a ResNet-101 [81 backend. The pre-trained model is acquired from the detectron2 (82 toolkit. This results in 5,024 dog images and 12,322 car images with a clear backgruudd setup and corresponding masks.
On datasets featuring a single foreground object, we use the 2-slot version of IODINE and Slot- attention. Since ReDO, IODINE, and Slot-Attentio do not distinguish foreground and background in output regions, we choose the best-matching scores from the permutation of foreground and back- ground masks as in 23 We observe that the proposed method and Grabeut the only two methods that provide explicit identification of foreground objects and background regions. While the Grab- cut algorithm actually requires a predefined bounding box as input that specifies the foreground region, our method, thanks to the learned pixel re-assignment (see Section (.2). can achieve this in a fully unsupervised manner. Results in Table T vised baselines by a large margin, exhibiting comparable performance even to the weakly supervised baseline that requires additional background information as inputs [3I] We provide samples of fore- ground extraction results as well as generated background and foreground regions in Fig. 3 Note that our final goal is not to synthesize appealing images but to learn foreground extractors in a fully unsupervised manner. As the limitation of our method, DRC generates foreground and background regions less realistic than those generated by state-of-the-aa GANs, which hints a possible direction for future work. More detailed discussions of the limitation can be found in supplementary material.
show that our method outperforms all the unsuper-
8

---

Figure 3: Foreground extraction results for each dataset; zoom in for better visibility. From top bottom: (i) observed images, (ii) generated images, (iii) masked generated foregrounds, (iv) generated backgrounds, (v) ground-truth foreground masks, and (vi) inferred foreground masks. More samples and results of baselines can be found in the supplementary material.
Multi-object scenes The second group of datasets contains images with possibly simpler fore- ground objects but more challenging scene configurations or background parts. Visual scenes in the CLEVR6 dataset contain various objects and often with partial occlusions and truncations. Fol- lowing the evaluation protocol in IODINE and Slot-attention, we use the first 70K samples from CLEVR [7] and filter the samples for scenes with at most 6 objects for training and evaluation, i.e. CLEVR6. The TM-dSprites dataset is variant of Multi-dSprites [80] but has strongly confounding backgrounds borrowed from Textured MNIST
32). We generate 20K samples for the experiments. Similar to Greff et al. (37 and Locatello et al. 38 evaluate on subset containing 1K samples for testing. Note that IODINE and Slot-attention are designed for segmenting complex multi-object scenes using slot-based object representations. Ideally, the output of these models consists of masks for each individual object, while the background is viewed as a virtual "object" as well. In practice, however, it is possible that the model distributes the background over all the slots as mentioned in Locatello et al. (38) We therefore propose two corresponding approaches (see the supplementary material for more details) to convert the output object masks into a foreground-bacckroundd partition
and report the best results of these two options for IODINE and Slot-attention in Table D
On the CLEVR6 dataset, use the publicly available pretrained model for IODINE, which achieves a reasonable ARI (excluding background pixels) of 94.4 on the testing data, close to the testing re- sults in Greff et al. [37]. We observe that IODINE distributes the background over all the slots for some of the testing samples, resulting in much lower IoU and Dice scores. We re-train the Slot-attention model using the official implementation on CLEVR6, no pretrained model is pub- licly available. The re-trained model achieves a foreground ARI of 98.0 on the IK testing samples, which we consider sign of valid re-implementation. Results in Table I demonstrate that the pro- posed method can effectively process images of challenging multi-object scenes. To be specific, our method demonstrates competitive performance on the CLEVR6 dataset compared with the SOTA object discovery method. Moreover, as shown empirically in Fig. 3 the proposed method can handle the strongly confounding background introduced Greff al. (32), whereas previous methods are distracted by the background and mostly fail to capture the foreground objects.
4.2 Ablation Study
Model
InU
Dice
We provide ablation studies on the Birds dataset to in- spect the contribution of each proposed component in our model. As shown in Table [ observe that re- placing the LEBMS in the foreground and background models with amortized inference networks significantly harms the performance of the proposed method. In par- ticular, the modified model fails generate any mean- ingful results (marked as in Table We conjecture that LEBM benefits from the diweenenniionlttyy of the latent space [7] and therefore enjoys more efficient learning. However, the inference networks need to learn an extra mapping from the high-dimensional image space to the latent space and require more elaborate architecture and tuning for convergence. Furthermore, we observe that the
amortized inference* w/o pix. re-assign.
21.8
35.3
w/o pseudo label
48.7
64.2
w/o T'-norm reg.
53.0
68.1
w/o ortho. reg.
54.3
69.2
short-rur run chain! Full model
52.5
67.7
56.4
70.9
Table 2: Ablation study on Birds. *We replace the L.BM with encoders per- form amortized inference for the latent vari- ables z within a variational framework as in VAE [83] +We explore the possibility of us- ing short-run MCMC (4] instead of persis- tent chain sampling.

---

model that does not learn pixel re-assignment for background can still generate meaningful images but only vaguely captures masks for foreground extraction.
4.3 Generalizable Foreground Extraction
Extracting novel foreground objects from training categories We show results on generalizing to novel objects from the training classes. To evaluate our method, we split the Birds dataset fol- lowing Chen et al. (28), resulting in 10K training images and IK testing images. On Dogs and Cars datasets, we split the dataset based on the original train-test split [77 78). This split gives 3,286 dog images and 6,218 18 car images for training, and 1,738 dog images and 6,104 car images for testing, respectively. As summarized in Table 3] our method shows superior performances compared with baselines; the performance gap between training and testing is constantly small over all datasets.
Birds
Dogs
Cars
Model
IoU
Dice
IoU
Dice
IoU
Dice
Tr./T,.
Tr.ITe.
Tr.IT.
Tr./Te.
Tr./Te.
Tr./Te.
GrabCut
30.2130.3
42.7142.8
58.3157.9
70.8170.5
60.9161.6
72.7173.5
ReDO
46,8147.1
61.4161.7
54.3152.8
69.2167.9
52.6152.5
68.7168.6 83.3182.5
Ours
54.8154.6
69.5169.4
71.6172.3
83.2183.6
71.9170.8
Table 3: Performance of DRC on training and held-out testing data. *Note that GrabCut is a deterministic method that docs not require training. We report the results of GrabCut [5] on these splits only for reference. Tr. indicates the performance training data, and Te. indicates the performance on testing data.
Extracting novel foreground objects from unseen categories To investigate how well our method gen- eralizes to categories unseen during training, we eval- uate the models trained in real-world single object datasets on the held-out testing data from different cat- egories. We use the same training and testing splits on these datasets as in the previous experiments. Table 4 shows that our method outperforms the baselines on the Birds dataset when the model has trained on Dogs or Cars dataset, which have quite different distribu- tions in foreground object shapes. Competitors like ReDO also exhibit such out-of-distribution general- ization but only to limited extent. Similar results are observed when using Dogs or Cars as the test- ing set. We can see that when the model is trained on Dogs and evaluated on Cars or vice versa, it still maintains comparable performances w.r.t. those trained&tested on the same class. We hypothesize that these two datasets have similar distributions in fore- ground objects and background regions. In the light of this, we can further entail that the distribu- tion of Dogs most similar that of Cars and less similar that of Birds according the results, which is consistent with our intuitive observation of the data. We provide a preliminary analysis of the statistics of these datasets in the supplementary material.
GrabCut
ReDO
Ours
Test
Train
IoU Dice IoU Dice IoU Dice
Birds*
47.1 61.7 54.6 69.4
Birds
Dogs 30.3 42.8 22.2 35.3 41.3 57.4
Cars
16.4 27.7 39,2 55,3
Dogs*
52.8 67.9 72.3 83.6
Dogs
Cars 57.9 70.5 44.5 61.2 67.8 80.4
Birds
44.0 60.3 53.6 69.1
Cars*
52.5 68.6 70.8 82.5
Cars
Dogs
61.6 73.5 51.6 67.1 68.6 81.0
Birds
41.8 58.6 45.1 61.7
Table 4: Performance of DRC unseen test- ing categories. #We include the testing results of models trained with data from the same cate- gories for reference.
5 Conclusion
We have presented the Deep Region Competition, an EM-based fully unsupervised foreground ex- traction algorithm fueled by energy-based prior and generative image modeling. We propose learned pixel re-assignment as an inductive bias to capture the background regularities. Experiments demon- strate that DRC exhibits more competitive performances on complex real-world data and challenging multi-object scenes. We show empirically that learned pixel re-assignment helps to provide explicit identification for foreground and background regions. Moreover, we find that DRC can potentially generalize to novel foreground objects even from categories unseen during training. We hope our work will inspire future research along this challenging but rewarding research direction.
10

---

Acknowledgements
The work was supported by NSF DMS-2015577, ONR MURI project N00014-16-1-2007, and DARPA XAI project N66001-17-2-4029 We would like to thank Bo Pang from the UCLA Statistics Department for his insights on the latent-space energy-ban based model and four anonymous reviewers (especially Reviewer RegA) for their constructive comments.
References
[1] Song-Chun Zhu and Alan Yuille. Region competition: Unifying snakes, region growing, and bayes/mdl for multiband image segmentation. IEFE Transactions on Pattern Analysis and Machine Inteliiencee (TPAMI), (89)0884-900, 1996.
[2] Jianbo Shi and Jitendra Malik. Normalized cuts image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 22(8):888-905, 2000.
Zhuowen Tu and Song-Chun Zhu. Image segmentation by data-driven markov chain monte carlo. JEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 24(5):657-617 673, 2002.
[4] Yuri Y Boykov and M-P Jolly, Interactive graph cuts optimal boundary & region segmentation of objects in nd images. Proceedings of International Conference on Computer Vision (ICCV), 2001.
[5] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "rrbcut" interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics (TOG), 23(3):009 314, 2004.
[6] Ming-Ming Cheng, Niloy Mitra, Xiaolei Huang, Philip Torr, and Shi-Min Hu. Global contrast based salient region detection. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 37 (3):569-582, 2014.
[7] Huaizu Jiang. Jingdong Wang. Zejian Yuan, Yang Wu, Nanning Zheng, and Shipeng Li. Salient object detection: A discriminative regional feature integration approach. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2013.
[8] Wangjiang Zhu, Shuang Liang, Yichen Wei, and Jian Sun. Saliency optimization from robust background detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[9] Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of European Conference on Computer Vision (ECCV), 2014.
[10] Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision (ICV)), 88(2): 303-338, 2010.
(11] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid seene parsing network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[12] Jonathan Long. Evan Shelhamer. and Trevor Darrell. Fully convolutional networks for semantic segmen- tation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2015.
[13] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), (9(12):48812499 2017.
[14] Liang-Chise Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 40(4):834 848, 2017.
[15] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assister Intervention, 2015.
[16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of Interna- tional Conference Computer Vision (ICCV), 2017.

---

[17] Nick Chater, Joshua B Tenenbaum, and Alan Yuille. Probabilistic models of cognition: Conceptual foun- dations. Trends in Cognitive Sciences, 10(7):287-211, 2006.
[18] Thomas F Shipley and Philip Kellman. fragments to objects: Segmentation and grouping in vision. Elsevier, 2001.
[19] Xide Xia and Brian Kulis. W-net: A deep model for fully unsupervised image segmentation. arXiv preprint arXiv:7711.08506, 2017.
[20] Asako Kanezaki. Unsupervised image segmentation by backpropagation. In International Conference on Acoustics, Speeck and Signal Processing (ICASSP), 2018.
[21] Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of International Conference on Computer Vision (ICCV), 2019.
[22] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 33(5): 898 916, 2010.
[23] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 34(1 (4(11))2274 2202422222222 2012.
[24] Yanchao Yang. Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervisedd moving ohject detection via contextual information separation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (C (CVPR), 2019.
[25] Yanchao Yang, Brian Lai, and Stefano Soatto. Dystab: Unsupervised object segmentation via dynamic- static bootstrapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2021.
[26] lan Goodfellow, Jean Pouget-Abaase Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of Advances in Neural Information Processing Systems (NeurlPS), 2014.
[27] Jianwei Yang, Anitha Kannan, Dhruv Batra, and Devi Parikh. Lr-gan: Layered recursive generative adver- sarial networks for image generation. In International Conference on Learning Reprexentations (ICLR), 2017.
[28] Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised object segmentation by redrawing. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2019.
[29] Pavel Ostyakov, Roman Suvorov, Elizaveta Logacheva, Oleg Khomenko, and Sergey I Nikolenko, Seigan: Towards compositional image generation by simultancously learning to segment, enhance, and inpaint. arXiv preprint 181iv: 0766000000200 2018
[30] Krishna Kumar Singh, Utkarsh Ojha, and Yong Jac Lec. Finegan: Unsupervised hicrarchical disentan- glement for fine-grained object generation and discovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[31] Yaniv Benny and Lior Wolf. Onegan: Simultaneous unsupervised learning of conditional image gener- ation, foreground segmentation, and fine-grained clustering. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
[32] Klaus Greff, Antti Rasmus, Mathias Berglund. Tele Hotloo Hao, Jürgen Schmidhuber, and Harri Valpola. Tagger: Deep unsupervised perceptual grouping. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2016.
[33] SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2016.
[34] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Pro- ceedings of Advances in Neural Information Processing Systems (NeurIPS). 2017.
[35] Sjoerd van Steenkiste, Michael Chang. Klaus Greff, and Jürgen Schmidhuber. Relational neural expecta- tion maximization: Unsupervised discovery of objects and their interactions. In International Conference Learning Representations (ICLR), 2018.
12

---

[36] Christopher P Burgess, Loic Matthey, Nicholas Watters. Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 10wv:/901.119000 2019.
[37] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Mathhey Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. Proceedings of International Conference on Machine Learning (ICML). 2019.
[38] Francesco Locatello, Dirk Weissenbom, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
[39] Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene in- ference and sampling with object-centric latent representations. International Conference on Learning Representations (ICLR), 2020.
[40] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations (ICLR), 2020.
[41] Michael Kass, Andrew Witkin, and Demetri Terzopoulos. Snakes: Active contour models. International Journal of Computer Vision (IJCV), 1(4):321-33)) 1988.
[42] Laurent D Cohen. On active contour models and balloons. CVGIP: Image understandingg 53(2):211-215 1991.
[43] Rolf Adams and Leanne Bischof. Seeded region growing. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 16(6):641-647, 1994.
[44] Song-Chun Zhu, Yingnian Wu, and David Mumford. Filters, random fields and maximum entropy (frame): Towards a unified theory for texture modeling. International Journal of Computer Vision (IJCV), 27(2):107-126, 1998.
[45] Cheng-en Guo, Song-Chun Zhu, and Ying Nian Wu. Primal sketch: Integrating structure and texture. Computer Vision and Image Understanding (CVIU), 106(1):5-19, 2007.
[46] Yvan G Leclerc. Constructing simple stable descriptions for image partitioning, International Journal of Computer Vision (IJCV), 102, 1989.
[47] Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy- based prior model. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
[48] Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991.
[49] Michael Jordan and Robert Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural Computation, 6(2):181-2 1994.
[50] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In Proceedings of International Conference on Machine Learning (ICML), 2015.
[51] Joe Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. In Proceedings of Interna- tional Conference on Machine Learning (ICML), 2018
[52] Trung T Pham, Thanh-Toan Do, Niko Sünderhauf, and Ian Reid. Scenecut: Joint geometric and object segmentation for indoor scenes. In Proceedings of International Conference on Robotics and Automation (ICRA), 2018
[53] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbel images. In Proceedings of European Conference on Computer Vision (ECCV), 2012.
[54] D Ulyanov, A Vedaldi, and V Lempitsky. Deep image prior. International Journal of Computer Vision (IJC)), 128(7), 2020.
[55] Dominik Lorenz, Leonard Bereska, Timo Milbich, and Bjom Ommer. Unsupervised part-basee based disentan- gling of object shape and appearance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
13

---

[56] George Papandreou. Liang- Ching Chen, Kevin P Murphy, and Alan L Yuille. Weakly-an semi-supervised learning of deep convolutional network for semantic image segmentation. In Proceedings of Interna- tional Conference on Computer Vision (ICCV), 2015.
[57] Deepak Pathak, Philipp Krahenbuhl, and Trevor Darrell. Constrained convolutional neural networks for weakly supervised segmentation. Proceedings of International Conference on Computer Vision (ICCV), 2015.
[58] Zilong Huang, Xinggang Wang, Jiasi Wang. Wenyu Liu, and Jingdong Wang. Seakly-supervisee seman- tic segmentation network with deep seeded region growing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[59] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
[60] Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[61] Seong Joon Oh, Rodrigo Benenson, Anna Khoreva, Zeynep Akata, Mario Fritz, and Bernt Schiele Ex- ploiting saliency for object segmentation from image level labels. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[62] Yu Zeng, Yunzhi Zhuge, Huchuan Lu, and I Lihe 7hang. Joint learning of saliency detection and weakly su- pervised semantic segmentation. In Proceedings of International Conference on Computer Vision (ICCV), 2019.
[63] Andrey Voynov, Stanislav Morozov, and Artem Babenko. Object segmentation without labels with large- scale generative models. In Proceedings of International Conference on Machine Learning (ICML), 2021.
[64] Max Welling and Yee Teh, Bayesian learning via stochastic gradient langevin dynamics. In Proceed- ings of International Conference on Machine Learning (ICML), 2011.
[65] Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-La -iao-LiiMe Meng. Handbook of markov chain monte carlo. CRC press, 2011.
[66] Bo Pang and Ying Nian Wu. Latent space energy-based model of symbol-vector coupling for text gen- eration and classification. In Proceedings of International Conference on Machine Learning (ICML), 2021.
[67] Bela Julesz. Textons, the elements of texture perception, and their interactions. Nature, 290(5802)91 97, 1981.
[68] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2019.
[69] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbcel. Infogan: inter- pretable representation learning by information maximizing generative adversarial nets. In Proceedings of Advances in Neural Information Processing Syxtems (NeurIPS), 2016.
[70] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convo- lutional generative adversarial networks. arXiv preprint arXiv: 06444,, 2015.
[71] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of International Conference on Machine Learning (ICML), 2015.
[72] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arv: 1607.08022. 2016.
[73] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations (ICLR), 2014.
[74] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspec- tive adversarial networks. arXiv preprint arXiv: 1609.07093, 2016.
14

---

[75] Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algo- rithms. Physica D: nonlinear phenomena, 60( 10:2-4)299-26 1992.
[76] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds California Institute of Technology. 2010.
200. Technical Report
[77] Aditya Khosla, Nitananna Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine- grained image categorization: Stanford dogs. In CVPR Workshop Fime-Gruined Visual Categorization (FGVC), 2011.
[78] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fine-grained categorization, In ICCV workshops, 2013.
[79] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[80] Matthey, Irina Higgins. Demis Hassabis, Alexander Lerchner. dsprites: Disentanglement testing sprites dataset.
2017.
[81] Kaiming He, Xiangyu Zhang. Shaoqing Ren, Jian Sun. Deep residual learning image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2016.
2016.
[82] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick, Detectron2. https: 2019.
[83] Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114, 2013.
[84] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ving Nian Wu. Learning non-convergent non-persisten short-run meme toward energy-based model. In Proceedings of Advances in Neural Information Process- ing Systems (NeurIPS), 2019.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contri-
butions and scope? (ess] See Section 4.1] Section 4.2 and Section
(b) Did you describe the limitations your work? [Yes] Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code. data, and instructions needed reproduce the main experimental results (either in supplemental material as a URL)? [Yes] We provide pytorch-style codes and detailed instructions of how to reproduce the main results in the supplemental material.
(b) Did you specify sen)? [Yes] See Section Section 4.2 Section 4.3 and supplemental material.
training details (e.g., data splits, hyperparameters, how they were cho-
(c) Did you report error bars (c.g.. with respect to the random seed after running experiments mul- tiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplemental material.
If you are using existing assets (e.g., code, data, models) or curating/releasngg new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A] and and Section 4.3
(d) Did you discuss whether and how consent was obtained from people whose data you's 're us-
ing/curating? [Yes] See Section [.11 Section 4.2
(c) Did you discuss whether the data you are using/curating contains personally identifiable infor- mation or offensive content? [N/A]
If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "wKDMRSG12Bc", "reviewer": "Reviewer_GuBW", "summary": "This work proposes a novel unsupervised foreground segmentation technique based on Mixture of Experts energy based modelling with a generative neural network. They introduce an interesting inductive bias, pixel reassignment, which works as the main regularizer and the secret sauce that enables the fg/bg separation. They essentially reshuffle the pixels for the background generation based on the learned index matrix and ablations shows that's the most significant regularizer. \nThey compare their results on 6 datasets, including birds, dogs, and clevr6 to both object centric approaches such as slot attention and gan based methods. They consistently outperform the previous methods with a significant margin.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work formulates the foreground/background separation as a 3 part task. The latent space prior which they use an energy based formulation LEBM. The pixel assignment classification. The generative model based on the assignment and the latent. They use a langevin dynamics mcmc to iteratively refine their image separation. They also add the ability to reshuffle the pixels for background generation, intuitively make it more coherent. Moreover, they augment the latent z in their LEBM formulation with a one hot vector y which later on they can use as pseudo labels for enforcing the bipartition. Based on the ablation studies the pseudo label and the pixel reassignment are the two most crucial manipulations for this method to work.\n\nApart from impressive results on 6 datasets and outperforming the prior work, they also evaluate their model on generalizing to unseen categories and novel objects. The results suggests that their model learns a consistent model of the background pixels which is to some degree robust to new objects. (Table 4 results should be bold for best in the middle row of each category).\n\nThe writeup is quite dense with too many notations but the additional info in the appendix provide enough context. The pixel reassignment is not clear when is applied. Maybe adding a line to the algorithm or fig.1 would help clarifying this. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The failure cases are mentioned in the appendix briefly. One question is the computation cost and speed for training and inference.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "dXSyzBD8g5L", "reviewer": "Reviewer_RcgA", "summary": "In this work, the authors propose an unsupervised, generative model for foreground-background segmentation of RGB images which is based on the previously proposed \"Latent Space Engergy-Based Prior Model\" (LEBM). Similar to a VAE, a latent vector sampled from a prior distribution is transformed into an image by a decoder network. Differently from VAEs however, the LEBM uses a more complex, learnable prior distribution and sampling from the prior and posterior distributions is realized by differentiable MCMC sampling using Langevin dynamics. An image is modelled using a spatial mixture of foreground and background images decoded by separate decoder networks. To foster the segregation into foreground and background, the backround model includes an additional, spatial resampling step targeted at the assumed background statistics. While the reconstructed images tend to be blurry, the reconstructed foreground-background masks have been shown to be more accurate than previous methods on several datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\n- This work is an application of an existing modeling technique (LEBM) to an existing task (unsupervised foreground segmentation). To my knowledge, the combination however is novel.\n- It is made clear in the paper, to which degree the approach is adapted from previous work and which components are contributed with this work.\n- The paper does not contain a related work section. Instead, some relations to previous work are discussed in the introduction, and some in the method section (L184-191). In my view, discussing previous works to provide context for the current work is an integral part of every scientific article and deserves to be discussed in a separate section with greater detail. I am happy to improve my rating of the paper if this is addressed. (**Update:** This has been well addressed by the authors.)\n\n### Quality\n- The derivation of the model and its training procedure is described in great detail, and technically sound to me. The information given in the main paper is sufficient to comprehend the model and experiments, further details about the datasets, hyperparameters and model training are provided in-depth in the supplemental information.\n- The model is compared with previous approaches on five datasets, including datasets with natural images. An ablation study is performed to judge the contribution of every modeling component. In the main results on the segmentation performance (Table 1), the models are however evaluated on the training set. Only in Table 3 when evaluating the generalizability, a subset of the models is evaluated on a separate test set for a subset of the datasets. In my view, testing on held out samples should also be the norm for unsupervised methods, since overfitting might also happen in this case. Ideally, the same information would be given in Table 1, but using separate test sets. If evaluating on the training set is required for comparing to previous methods, this should be made explicit and featured less prominently.\n  (**Update**: This has been addressed by the authors\n- The paper only shows and evaluates reconstructions (including the segmentation mask) of images, but no samples from the model. While not strictly necessary for the target task of foreground-background segmentation, in my view this is still necessary to fully understand the modelling approach in this setting. The authors propose a generative model, which is derived with the goal of fitting the data distribution, but do not directly evaluate to which degree the model succeeds in that respect. Evaluating and/or showing samples from the model would be a means to obtain more insights into this.\n\n### Clarity\n- The paper and supplement are very well written. The derivation of the model and the experiments are well described, and the necessary concepts from previous works briefly introduced. My main criticism is about the missing related work section (see above).\n\n### Significance\n- Foreground-background segmentation is a classical task in computer vision with several applications, and using unsupervised models has many practical advantages. This model is therefore relevant to both researchers and practitioners in that field. Compared to more general segmentation approaches as, e.g. semantic or instance segmentation, the applicability of the model is limited in practice. From the perspective of applying this method, the paper would profit from a discussion of the real world relevance of unsupervised foreground-background segmentation in the introduction.\n- Interest in unsupervised methods in computer vision has increased in recent years. Structured latent-space models are one particular area with much recent interest, often under the umbrella term \"object-centric representation learning\". Mostly this is approached by using GAN or VAE-based models. The sucessful demonstration of a different generative method for foreground-background modelling is very interesting and might well inspire further research in that field. From the perspective of this field, the present work however has two important limitations: (1) As shown in the supplemental material, the modelling approach in its current form is not able to model multiple objects well, which is one of the core challenges in that field. (2) The model is not able to handle appearances well, as opposed to the masks.\n\n\n### Update\nOverall my concerns have been addressed, especially regarding the related work. Therefore I improve my rating of this paper accordingly. In summary I think this is a good paper, however I still see some minor points where this paper could be improved further.\n\n**Testing on training data.** This decision is justified to be able to compare to previous methods, and the proposed model is additionally tested on held-out data. In my view the paper could however be improved further by restructuring the result presentation to push more towards proper evaluation.\n\n**Testing generative modeling capabilities.** I still think it would have been interesting to go more into depth regarding the generative modeling capabilities of the model. Since the main focus of this work is foreground-background segmentation, leaving this for future work is however justified.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The authors do not discuss the limitations of the proposed model, e.g. the necessary preprocessing of the images or conditions in which the background prior might fail.\n- There is no discussion of the potential societal impact included in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "cj-IFhDRAB-", "reviewer": "Reviewer_QLw6", "summary": "The paper tackles the problem of unsupervised foreground segmentation with a combination of energy-based and deep-learning-based models. It models the foreground and background as two components that can be selected by a latent variable. The background has an inductive bias that pixels can be re-assigned to a different location. The whole model is generative. During inference, EM is needed to figure out the foreground and background separation by maximizing probability to generate the image and set the right latent states.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n1.\tThe method is built on decent math. With the factorized probabilistic formulation, the optimization follows naturally by applying established techniques. \n2.\tThe idea of using pixel reassignment as an inductive bias is interesting though used in the previous non-deep-learning method. \n3.\tThe experimental results demonstrate significant performance improvement over a few previous methods for foreground-background separation \n\nCons:\n1.\tIt is unclear if the proposed method is scalable. What are the sizes of the images used in experiments? Can the method achieve similar relative performance gain over other methods on higher resolution images? What’s the method’s efficiency compared to other methods?\n2.\tThere is a line of work to automatically discover object structures, such as landmarks and object parts. For example, a CVPR’19 paper, “Unsupervised Part-Based Disentangling of Object Shape and Appearance,” (not cited. Just taking this as an example. There are more papers) can discover object parts, which can be adapted to foreground detection. Compared to this line of work, what is the advantage of the proposed method? And is there technical connections?\n\n---\n\nThanks for the authors' response.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is a general purpose method. Not sure if there is any specific societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "8aOyBrLuiVk", "reviewer": "Reviewer_rkT2", "summary": "This paper proposes deep region competition to extract foreground objects from images in an unsupervised manner. The proposed method leverages energy-based prior with generative image modeling in the form of Mixture of Experts. In addition, a pixel re-assignment scheme is further learned to capture the regularities of background regions. Experiments demonstrate that the proposed method can achieve promising results on real-world images.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the proposed method is of some novelty and may be insightful for the community. It has also shown superior performance on 5 challenging datasets. My major concern on this paper is clarity. It is hard for me to fully understand the description. Instead of directly going into the technical details, I suggest the authors first provide a high-level overview of the task as well as the proposed method, and explain what are the important components of this method, what are their functions, how are they incorporated and learned? By doing so, the authors can further claim which part of the method is their unique design, and what are the advantages or differences to prior methods.\n\nIn the methodology sections, I do not understand how to achieve pixel re-assignment, and what is its relationship to the proposed method. In the experiment section, it is not clear what is the input resolution and how long does it take to train/test the proposed method. \n\nThe authors claim to provide limitations in section 4.1. I cannot find detailed discussions. Please provide in-depth discussions on the limitation of this work and potential solutions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It seems that the proposed method can only work under single objects or multi-objects with clear backgrounds. It is not clear the performance of the proposed method on images in the wild. I suggest the authors to compare their methods with unsupervised salient object detection approaches to further verify their merits.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers rate this work as interesting and unanimously recommend acceptance of the paper but still see room for improvement.
Decision: Accept (Poster)