Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper tackles the problem of unsupervised foreground segmentation with a combination of energy-based and deep-learning-based models. It models the foreground and background as two components that can be selected by a latent variable. The background has an inductive bias that pixels can be re-assigned to a different location. The whole model is generative. During inference, EM is needed to figure out the foreground and background separation by maximizing probability to generate the image and set the right latent states.
main_review: Pros: 
1.	The method is built on decent math. With the factorized probabilistic formulation, the optimization follows naturally by applying established techniques. 
2.	The idea of using pixel reassignment as an inductive bias is interesting though used in the previous non-deep-learning method. 
3.	The experimental results demonstrate significant performance improvement over a few previous methods for foreground-background separation 

Cons:
1.	It is unclear if the proposed method is scalable. What are the sizes of the images used in experiments? Can the method achieve similar relative performance gain over other methods on higher resolution images? What’s the method’s efficiency compared to other methods?
2.	There is a line of work to automatically discover object structures, such as landmarks and object parts. For example, a CVPR’19 paper, “Unsupervised Part-Based Disentangling of Object Shape and Appearance,” (not cited. Just taking this as an example. There are more papers) can discover object parts, which can be adapted to foreground detection. Compared to this line of work, what is the advantage of the proposed method? And is there technical connections?

---

Thanks for the authors' response.
limitations_and_societal_impact: It is a general purpose method. Not sure if there is any specific societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a novel unsupervised foreground segmentation technique based on Mixture of Experts energy based modelling with a generative neural network. They introduce an interesting inductive bias, pixel reassignment, which works as the main regularizer and the secret sauce that enables the fg/bg separation. They essentially reshuffle the pixels for the background generation based on the learned index matrix and ablations shows that's the most significant regularizer. 
They compare their results on 6 datasets, including birds, dogs, and clevr6 to both object centric approaches such as slot attention and gan based methods. They consistently outperform the previous methods with a significant margin.
main_review: This work formulates the foreground/background separation as a 3 part task. The latent space prior which they use an energy based formulation LEBM. The pixel assignment classification. The generative model based on the assignment and the latent. They use a langevin dynamics mcmc to iteratively refine their image separation. They also add the ability to reshuffle the pixels for background generation, intuitively make it more coherent. Moreover, they augment the latent z in their LEBM formulation with a one hot vector y which later on they can use as pseudo labels for enforcing the bipartition. Based on the ablation studies the pseudo label and the pixel reassignment are the two most crucial manipulations for this method to work.

Apart from impressive results on 6 datasets and outperforming the prior work, they also evaluate their model on generalizing to unseen categories and novel objects. The results suggests that their model learns a consistent model of the background pixels which is to some degree robust to new objects. (Table 4 results should be bold for best in the middle row of each category).

The writeup is quite dense with too many notations but the additional info in the appendix provide enough context. The pixel reassignment is not clear when is applied. Maybe adding a line to the algorithm or fig.1 would help clarifying this. 
limitations_and_societal_impact: The failure cases are mentioned in the appendix briefly. One question is the computation cost and speed for training and inference.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this work, the authors propose an unsupervised, generative model for foreground-background segmentation of RGB images which is based on the previously proposed "Latent Space Engergy-Based Prior Model" (LEBM). Similar to a VAE, a latent vector sampled from a prior distribution is transformed into an image by a decoder network. Differently from VAEs however, the LEBM uses a more complex, learnable prior distribution and sampling from the prior and posterior distributions is realized by differentiable MCMC sampling using Langevin dynamics. An image is modelled using a spatial mixture of foreground and background images decoded by separate decoder networks. To foster the segregation into foreground and background, the backround model includes an additional, spatial resampling step targeted at the assumed background statistics. While the reconstructed images tend to be blurry, the reconstructed foreground-background masks have been shown to be more accurate than previous methods on several datasets.
main_review: ### Originality
- This work is an application of an existing modeling technique (LEBM) to an existing task (unsupervised foreground segmentation). To my knowledge, the combination however is novel.
- It is made clear in the paper, to which degree the approach is adapted from previous work and which components are contributed with this work.
- The paper does not contain a related work section. Instead, some relations to previous work are discussed in the introduction, and some in the method section (L184-191). In my view, discussing previous works to provide context for the current work is an integral part of every scientific article and deserves to be discussed in a separate section with greater detail. I am happy to improve my rating of the paper if this is addressed. (**Update:** This has been well addressed by the authors.)

### Quality
- The derivation of the model and its training procedure is described in great detail, and technically sound to me. The information given in the main paper is sufficient to comprehend the model and experiments, further details about the datasets, hyperparameters and model training are provided in-depth in the supplemental information.
- The model is compared with previous approaches on five datasets, including datasets with natural images. An ablation study is performed to judge the contribution of every modeling component. In the main results on the segmentation performance (Table 1), the models are however evaluated on the training set. Only in Table 3 when evaluating the generalizability, a subset of the models is evaluated on a separate test set for a subset of the datasets. In my view, testing on held out samples should also be the norm for unsupervised methods, since overfitting might also happen in this case. Ideally, the same information would be given in Table 1, but using separate test sets. If evaluating on the training set is required for comparing to previous methods, this should be made explicit and featured less prominently.
  (**Update**: This has been addressed by the authors
- The paper only shows and evaluates reconstructions (including the segmentation mask) of images, but no samples from the model. While not strictly necessary for the target task of foreground-background segmentation, in my view this is still necessary to fully understand the modelling approach in this setting. The authors propose a generative model, which is derived with the goal of fitting the data distribution, but do not directly evaluate to which degree the model succeeds in that respect. Evaluating and/or showing samples from the model would be a means to obtain more insights into this.

### Clarity
- The paper and supplement are very well written. The derivation of the model and the experiments are well described, and the necessary concepts from previous works briefly introduced. My main criticism is about the missing related work section (see above).

### Significance
- Foreground-background segmentation is a classical task in computer vision with several applications, and using unsupervised models has many practical advantages. This model is therefore relevant to both researchers and practitioners in that field. Compared to more general segmentation approaches as, e.g. semantic or instance segmentation, the applicability of the model is limited in practice. From the perspective of applying this method, the paper would profit from a discussion of the real world relevance of unsupervised foreground-background segmentation in the introduction.
- Interest in unsupervised methods in computer vision has increased in recent years. Structured latent-space models are one particular area with much recent interest, often under the umbrella term "object-centric representation learning". Mostly this is approached by using GAN or VAE-based models. The sucessful demonstration of a different generative method for foreground-background modelling is very interesting and might well inspire further research in that field. From the perspective of this field, the present work however has two important limitations: (1) As shown in the supplemental material, the modelling approach in its current form is not able to model multiple objects well, which is one of the core challenges in that field. (2) The model is not able to handle appearances well, as opposed to the masks.


### Update
Overall my concerns have been addressed, especially regarding the related work. Therefore I improve my rating of this paper accordingly. In summary I think this is a good paper, however I still see some minor points where this paper could be improved further.

**Testing on training data.** This decision is justified to be able to compare to previous methods, and the proposed model is additionally tested on held-out data. In my view the paper could however be improved further by restructuring the result presentation to push more towards proper evaluation.

**Testing generative modeling capabilities.** I still think it would have been interesting to go more into depth regarding the generative modeling capabilities of the model. Since the main focus of this work is foreground-background segmentation, leaving this for future work is however justified.
limitations_and_societal_impact: - The authors do not discuss the limitations of the proposed model, e.g. the necessary preprocessing of the images or conditions in which the background prior might fail.
- There is no discussion of the potential societal impact included in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes deep region competition to extract foreground objects from images in an unsupervised manner. The proposed method leverages energy-based prior with generative image modeling in the form of Mixture of Experts. In addition, a pixel re-assignment scheme is further learned to capture the regularities of background regions. Experiments demonstrate that the proposed method can achieve promising results on real-world images.
main_review: I think the proposed method is of some novelty and may be insightful for the community. It has also shown superior performance on 5 challenging datasets. My major concern on this paper is clarity. It is hard for me to fully understand the description. Instead of directly going into the technical details, I suggest the authors first provide a high-level overview of the task as well as the proposed method, and explain what are the important components of this method, what are their functions, how are they incorporated and learned? By doing so, the authors can further claim which part of the method is their unique design, and what are the advantages or differences to prior methods.

In the methodology sections, I do not understand how to achieve pixel re-assignment, and what is its relationship to the proposed method. In the experiment section, it is not clear what is the input resolution and how long does it take to train/test the proposed method. 

The authors claim to provide limitations in section 4.1. I cannot find detailed discussions. Please provide in-depth discussions on the limitation of this work and potential solutions.
limitations_and_societal_impact: It seems that the proposed method can only work under single objects or multi-objects with clear backgrounds. It is not clear the performance of the proposed method on images in the wild. I suggest the authors to compare their methods with unsupervised salient object detection approaches to further verify their merits.
review_title: null

Meta-review: All reviewers rate this work as interesting and unanimously recommend acceptance of the paper but still see room for improvement.
Decision: Accept (Poster)