Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper makes a solid contribution toward unsupervised multi-object segmentation in images. The COMUS method leverages unsupervised saliency detectors to initially estimate object proposal masks (for accurate object localization). It then uses self-supervised feature representation networks for feature extraction from the designated region. These representations are clustered into different categories for different object discovery. The cluster IDs are combined with the saliency masks to form an initial set of pseudo-labels used for object segmentation, not just category discovery. 
strength_and_weaknesses: Strengths:
* The tackled problem is challenging and of great interest to the research community.
* State-of-the-art performance (by a large margin compared to very recent published work) on a well-known benchmark PASCAL VOC.
* The first to report results on a more challenging (more object categories) dataset - MS COCO.
* The paper presentation and experimental analysis is well thought out and executed - if it proves the method's efficiency.
* Good ablation studies.
* The dedicated section with the author's discussion regarding the method's limitations is a plus. 

Weaknesses: 
* The design decision of using another saliency model BasNet on the saliency masks from DeepUSPS is not argued. What are the implications of not doing this step and just using the initial saliency maps provided by DeepUSPS - why is this distillation procedure needed? 

Minor comments:
* Algorithm 1 is not referenced in the paper.
* Table 1 caption - MaskContrast is misspelled.
* Figure 4 is first referenced on Page 4, but you need to scroll down to Page 9 to actually see it.

Question: How can this work be extended (if) to semantic segmentation not just object-centric, to be comparable to methods such as STEGO (referenced in the paper)? 
clarity,_quality,_novelty_and_reproducibility: Clarity/Quality - The paper is well-written, well-structured and clear. Contributions are clearly stated. The whole story of the method and its development is sound and makes it quite an interesting read. Discussion and references to relevant work are on point.

Novelty - The method presents some novel ideas - especially the approach in which the initial pseudo-labels are extracted using the object-centric bias and the spectral clustering procedure and the attribution of classes to each mask without any annotations - afterward I do not consider them as novel ideas since the teacher-student paradigm has been intensively studied before and same goes for the iterative refinement procedure of pseudo labels - definitely useful but not novel. Nonetheless, this is a good example of a simple but effective procedure. 

Reproducibility - The authors offered all the necessary details (pseudocode and implementation details). They also mentioned code release upon acceptance.
summary_of_the_review: Overall the paper is good and the contribution is solid. The experiment that was particularly interesting was COMUS under a distribution shift - how well it performs out-of-distribution and the implications of starting with easier datasets and gradually increasing the difficulty. The reviewer has no grounds for rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
This paper addresses the problem of unsupervised semantic segmentation with self-supervised object-centric representations. The authors use object-centric datasets on which localization and categorization priors are learned in a self-supervised way, Combining these priors with an iterative self-training procedure allows to obtain highly performing semantic segmentation framework.
The method has been validated on 2 public datasets (MS COCO and PASCAL VOC). The abblation study shows that the method is able to identify individual components of the unsupervised learning process for semantic segmentation. Final results on PASCAL VOC achieved 47,3 of mIoU and on MS COCO - 40,7. 
strength_and_weaknesses: The strengths are as follows:
1. The paper is addressing a very important problem - unsupervised semantic segmentation.
2. The pipeline is composed of straighforward techniques with solid backgrdound.
3. The final pipeline is very well justified.
4. State of the art is updated and complete.
5.  Validation is showing the high performance of the method.
6. Public datasets are used to validate the method.
7. Abblation study is presented.
7. Appendices additionally  give insight on the performance of the method.

Weaknesses:
1 Although the pipeline is composed of technically sound techniques, in its essence the novelty is limited to how to combine these techniques. Still, this weakness is considered not so substantial given the good justification of the pipeline and the final segmentation results.
2. My main concern is that the method is not compared to any state of the art unsupervised semantic segmentation technique. In the paper it is said that the performance of the proposed method is comparable to the supervised semantic segmentation but the paper does not report any state of the art (superivesd neither unsupervised segmentation) method performance.
3. It is highly recommendable to apply the same methodology (datasets, statistical measures, etc.) of other state of the art methods for unsupervised semantic segmentation and compare them. 

clarity,_quality,_novelty_and_reproducibility: The paper is well structured and clearly written. Novelty is due to the proposed pipeline composed of different steps: self-supervised representation learning, self-supervised saliency detection, unsupervised object discovery and iterative self-training.
Reproducibility is not very clear since the authors do not publish their code neither plan to do it and since the method is composed by many steps, it will be difficult to assure that all parameters are clear in order to reproduce the process.
summary_of_the_review: This paper addresses an important problem - unsupervised semantic segmentation. The paper proposes a well justified and technically sound pipeline composed of straightforward steps. Validation is extensive and final results are looking very promissing. However, unfortunately the authors do not compare their method with any state of the art supervised and unsupervised semantic segmentation method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work develops a pipeline for unsupervised semantic segmentation. The distinguishing feature of this approach is its generalization from training on binary classification examples to multiple classes. This is achieved by clustering feature embeddings extracted with the help of unsupervised saliency detection methods and a self-training formulation of multi-class model training.
strength_and_weaknesses: I like this work: it is clearly presented, the pipeline is carefully designed in that it leverages some of the latest advances in unsupervised learning, and achieves a substantial improvement of the segmentation accuracy over previous methods in this problem domain.
The experiments also extend to COCO and include some analysis of different design choices for the feature extractor and the saliency detection method. This is very nice.

On the downside, this multi-stage pipeline may look a bit involved, as it requires pre-training of feature embeddings and of a saliency detection network, clustering and multiple rounds of self-training. It would be great if the paper provided more detail on the computational requirements of these individual steps, since such considerations (model complexity) are often part of the game.

Table 2,4 lack some reference. Why not evaluate MaskContrast or DeepSpectral on VOC/COCO for a comparison? Pre-trained models are either publicly available or could be surely requested from the respective authors.

What is the strategy for model selection? (validation data and metric)
What is the effect of more self-training rounds? Would test-time augmentation help (or already used)?

Some analysis of failure modes would be helpful both for understanding the inherent biases in the approach and as inspiration for future work.

(Apologies if I overlooked any answers to these questions — I would be grateful for the pointers.)
clarity,_quality,_novelty_and_reproducibility: This work is of high quality and reads well.
The technical novelty of individual steps of the pipeline is not great, but their composition as a whole is.
Although the pipeline is rather complex, there is sufficient detail in the paper on the implementation and the training methodology.
summary_of_the_review: Unsupervised semantic segmentation is an emerging problem domain within the realm of unsupervised learning. This paper makes a notable leap and I can see its potential in inspiring a large following.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem of unsupervised semantic segmentation from images, which is relatively new in the literature. The methodology is based on saliency detection, clustering of self-supervised features (e.g., DINO) within the salient regions to obtain pseudo-masks, followed by a few iterations of self-training. This method achieves state-of-the-art results on unsupervised semantic segmentation on PASCAL-VOC and MS COCO.  
strength_and_weaknesses: 
**Strengths**

The proposed method is simple, yet effective as it achieves over 10 points mIoU above prior work. Results are also shown for the first time on the challenging MS COCO dataset (80 classes).   

The method directly builds on pre-trained representations, with optional self-training. As such, and in contrast to some other approaches, it does not require costly self-supervised pre-training at object level (i.e., dense representations) to achieve a pixel-wise task, such as semantic segmentation.  


**Weaknesses**

(1) One major issue with the paper is its close resemblance to DeepSpectral (Melas-Kyriazi et al., 2022), although the authors do not discuss this explicitly. DeepSpectral uses a similar pipeline for (unsupervised) semantic segmentation: unsupervised extraction of image regions, clustering for categorizing these regions, and self-training. The most notable difference appears to be the choice of method for the extraction of relevant image regions, that is DeepUSPS for this paper and spectral clustering for DeepSpectral. The rest of the pipeline seems to be quite similar with minor changes (spectral vs k-means clustering, or the number of iterations during self-training). This limits the technical contribution of the paper, despite its strong performance. Therefore, the authors should provide an extensive discussion of the differences to DeepSpectral and highlight their significance. 

(2) The approach relies on a saliency detection method (DeepUSPS/BASNet) to get an initial set of object masks. Despite common belief, I would argue that DeepUSPS may not be considered *truly* unsupervised since:  
 - it distills an ensemble of *handcrafted* priors
 - it uses *supervised* pretraining (e.g., on ImageNet or Cityscapes according to (Nguyen et al., 2019)) 

This fact gives an inherent advantage to DeepUSPS vs fully unsupervised approaches for saliency detection. This is also clear from Table 7, where DeepUSPS performs significantly better than approaches such as LOST or DINOSeg which only rely on self-supervised features to estimate saliency. This in turn gives a clear advantage to the object proposal masks used by the rest of the method. 

Notably, DeepSpectral also relies on self-supervised features (e.g., DINO) for object proposal masks. Therefore, the question that arises is whether the biggest part of the performance boost of this paper could be simply attributed to the advantage of DeepUSPS over DINO-based saliency. The authors could investigate this further, e.g., by swapping DeepUSPS for the object proposal part of DeepSpectral (namely, spectral decomposition). 

(3) In most figures, it is notable that this method still struggles with images containing more than one semantic category. This is due to the initial assumption that there is only a single object per image (i.e., assigning a single cluster ID to the entire salient region in an image). Despite self-training, the model does not seem to fully recover from this assumption, especially in PASCAL VOC where the majority of images do indeed contain a single object, so there is little training signal. This likely can be measured, e.g., by measuring the performance and predicted object count on images containing more than one object 

(4) The authors could also consider running experiments on COCO-Stuff, to facilitate comparisons with other state-of-the-art methods, such as STEGO (Hamilton et al., 2022) and PiCIE (Cho et al., 2021).  

(5) The authors should also consider citing, discussing, and (where possible) comparing to the following related methods:

[1] Ziegler and Asano, “Self-Supervised Learning of Object Parts for Semantic Segmentation.” (CVPR 2022)

[2] Ke et al. "Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformers." (CVPR 2022)

[3] Henaff et al., “Object discovery and representation networks.” (ECCV 2022)

clarity,_quality,_novelty_and_reproducibility: Regarding clarity, the paper is well-structured, straightforward, and easy to follow. Enough details are provided that make the paper mostly reproducible. However, due to the reasons outlined above, the proposed idea is not very original and thus the paper lacks in terms of novelty. 
summary_of_the_review: Although this paper achieves state-of-the-art results on the task of unsupervised semantic segmentation by a large margin, these results may be (to a large degree) due to the superior saliency estimation method that is used in this paper in contrast to prior work. The rest of the pipeline is very similar to that of (Melas-Kyriazi et al., 2022). As a result, I find that the novelty of this paper is not significant enough to warrant acceptance to ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper deals with challenging problem of unsupervised multi-object segmentation in images. A distinguishing aspect of the proposed method is its generalization from training on binary classification examples to multiple classes, by clustering feature embeddings extracted with the help of unsupervised saliency detection methods and a self-training formulation of multi-class model training. The paper reported substantial improvement of the segmentation accuracy over SOTA methods, especially the first to report results on a more challenging (more object categories) MS COCO dataset.
justification_for_why_not_higher_score: it is a good paper but not strong enough in terms of scores
justification_for_why_not_lower_score: good paper solving important problem (unsupervised segmentation) with significant results
Decision: Accept: notable-top-25%