Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper advances a self-supervised representation learning strategy, dubbed SlotCon, that can learn from scene-centric data by reasoning and processing visual information at the pixel level. This is in contrast with the large majority of the self-supervised methods designed for object-centric data (e.g., ImageNet) where image-level reasoning is sufficient. 
Recent methods have addressed scene-centric data, however the authors argue that such approaches are limited by the hand-crafted priors (e.g., objects as superpixels) or too specialized pretext tasks as they can limit their generalization.
SlotCon builds upon the SlotAttention method and uses slots as learnable prototypes for pixels that are thus grouped via the assignments to their corresponding slots. SlotCon is composed of a teacher and student network. The pixel grouping supervision resembles both SWaV (aligning the pixel assignments between teacher and student spatial-aligned pixels from different views) and DINO (student + EMA teacher, centering of teacher logits, different temperatures on student and teacher). This loss encourages grouping of pixels into object-like structures via the learned slots.
In order to discriminate slots that carry the same visual or semantic information across views from other non-informative or redundant slots, SlotCon has also a contrastive objective (InfoNCE loss) at the slot level that encourages similarity between different views of the same slot and discourages similarity of slots from different views and slots from other images.

SlotCon is evaluated on a number of pre-training settings (ImageNet, COCO, COCO+) and downstream tasks (object detection, instance segmentation, semantic segmentation, unsupervised segmentation) and datasets (COCO, Cityscapes, Pascal VOC, ADE20k), with nice performance and results.
strengths_and_weaknesses: ### Post-rebuttal update

The detailed rebuttal addresses all my questions and offers convincing responses and additional insisghts (across the responses to all reviewers).
I don't have any other questions for the authors and I confirm my positive recommendation for this submission.

==========================================


### Recommendation
This paper advance a nice and effective idea for self-supervised learning from scene-centric images. The approach looks sound and the results are encouraging. I'm overall positive about this work and leaning towards recommending it for acceptance.

### Paper strengths

- _Clarity_: I find this paper is mostly well written and argued. The intuition, reasoning and formalism of the method are rather well explained and overview figure 1 is very nice. Then the authors conduct a number of ablation and sensitivity studies (number of prototypes, loss balancing weights, temperature of the teacher) for better understanding. There are also some interesting qualitative results (Figure 2) visualizing pixels grouped by slots and their nearest neighbors. The approach is convincing.

- _Significance_: The paper addresses a challenging problem related to self-supervised representation learning: how to learn from scene-centric datasets that are not as well balanced as ImageNet, more cluttered and usually with fewer images to learn from. The results reached by SlotCon are good.


- _Quality_: The approach seems technically sound. There are several experiments on different settings and different relevant baselines considered. The authors do not provide the code in the supplementary, but provide rich information about the implementation -- it would be better to release the code though. The supplementary is abundant in additional experiments, ablations and implementaiton details. 

- _Originality_: This work combines various good practices from the recent literature (SWAV, DINO, Slot Attention, see my comments from the summary), however the output method does still have originality and is effective. The masking of uninformatives slots (eq. 5) is an interesting idea.


- _Misc_: I appreciate that the authors stick to the lower epoch regime (100-200 epochs) which is more reasonable and allows comparison with a larger spectrum of methods


### Paper weaknesses

Mostly minor concerns:

#### Originality:
- as mentioned above, to me SlotCon is a fairly original work aggregating in an effective manner various good practices and approaches from the literature.
- from the text it can be understood that using SlotAttention for unsupervised pixel-level representation learning is a finding of the authors. However other prior methods have used it for weakly-supervised or  self-supervised learning [a], [b]. Those are different approaches and I don't think a discussion is necessary, however the text should be adjust to better delimitate the contributions and prior works
- the approach is similar in spirit with Odin [31] as both aim to learn representations without human object priors. The qualitative examples in both papers are similar. I think it would be good to acknowledge the relatedness of these concurrent works


#### Scope of experiments:
- COCO is indeed a challenging scene-centric dataset and it's great to see pre-training and evaluation done on it, moving beyond ImageNet
- I think this work would benefit from showing results on more complex settings, like autonomous driving data where the amount of objects per scene can be larger (BDD100K, Cityscapes). Some options could be:
    + self-supervised pretraining on an autonomous driving dataset, e.g., BDD100K[56]
    + self-supervised pretraining on smaller autonomous driving datasets, e.g., Cityscapes, and evaluation on unsupervised semantic segmentation on these datasets, see PiCIE, STEGO[c]

#### Computational cost:
- the authors discuss the computational cost in appendix D and give exact number of FLOPS for forward operation compared to a standard ResNet-50 backbone. It appears SlotCon is 12.3% more expensive in terms of FLOPS.
- However this does not tell us much on how fast are forwards and backward passes and how does SlotCon fare against other methods. It's nice that SlotCon can be trained in fewer epochs, but it would be great to be able to compare also the training times
- To get an idea of more straightforward and easy to visualize and understand format I recommend the time and memory cost studies from MoCov2, OBoW[d] and iBOT[e]


#### Baseline results and implementations:
- In Table 3 (transfer results with ImageNet-1k pre-training), it's not clear where to the scores from DetCon (200 epochs) come from. In the original paper there are some result on a few plots and maybe the authors eye-balled them (e.g., COCO instance segmentation), but not all results can be found in the original paper. The authors don't mention another reference for them. Are these scores reproduced by the authors, taken from a different paper (if so, please cite the corresponding source).
- PixPro results in Table 3 seem lower than originally reported, about 0.8-1.0 for COCO. Do the authors know why?

- The common protocol for COCO downstream as proposed in MoCO uses the default `ROI_BOX_HEAD` configuration with 2 FC layers. PixPro seems to have modified the default configuration of 2FC to 4 convolution layers + 1 FC layer that seems to improve scores. Was this setting used for PixPro or SlotCon?


#### Related work:
- this work mentions a whole lot of relevant works from the literature
- however for the unsupervised semantic segmentation section there are several key methods in addition to the now usual PiCIE and IIC and the very recent ones mentioned by the authors (SegDiscover).
- here are a few suggestions for this area of the literature:[f],[g],[h],[c]


**References:**


[a] T. Kipf et al., Conditional Object-Centric Learning from Video, ICLR 2022

[b] Z. Bao et al., Discovering Objects That Can Move, CVPR 2022

[c] M. Hamilton et al., Unsupervised Semantic Segmentation by Distilling Feature Correspondences, ICLR 2022

[d] S. Gidaris et al., Online Bag-of-Visual-Words Generation for Unsupervised Representation Learning, CVPR 2021

[e] J. Zhou et al., iBOT: Image BERT Pre-Training with Online Tokenizer, ICLR 2022

[f] J. Hwang et al., Segsort:Segmentation by discriminative sorting of segments, ICCV 2019

[g] M. Chen et al., Unsupervised object segmentation by redrawing, NeurIPS 2019

[h] X. Wang et al., FreeSOLO: Learning to Segment Objects without Annotations, CVPR 2022

questions: 
Here are a few questions and suggestions to help improve the paper, that could be potentially addressed in the rebuttal


1) Conduct a comparison of the computational cost for training SlotCon compared to related methods, MOCOv2, PixPro, etc.

2) Clarify results and implementations for baseline methods

3) (Optional) How is SlotCon performing on more complex scene-centric datasets, e.g., autonomous driving datasets?

limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a self-supervised learning framework, SlotCon, from unlabeled scene-centric data. The method adopts joint semantic grouping -- softly assigning pixels to learnable prototypes shared by the datasets, and contrastive learning -- first perform attentive pooling on the prototypes to from slots, and then conduct contrastive learning on slots from two different views. In experiments, the authors evaluated on COCO, ImageNet-1K and COCO+ dataset. On transfer learning tasks (COCO detection, COCO segmentation, cityscape & Pascal VOC & ADE20k semantic segmentation), the proposed method achieves performance on par or surpassing previous methods. It also achieves better mIoU on unsupervised semantic segmentation results compared with previous methods.
strengths_and_weaknesses: Strengths:
- The method is purely data-driven without the need of hand-crafted priors or specialized pretext tasks.
- The proposed method perform self-supervised learning on pixel level, which is shown to perform better on dense prediction downstream tasks compared with image-level self-supervised learning methods.
- Extensive experiments are conducted to show the effectiveness of the proposed methods.
- The paper is relatively easy to follow.

Weaknesses:
- Limited novelty and lack of citation on related works. Pixel-level clustering with contrastive learning are already studied in previous works, e.g., [1]. Prototype-based semantic segmentation are also already studied, e.g., in [2]. But many related works are not discussed. 

- The representation in the method section is not very clear. e.g., is $A^l_{\theta}$ (Eq. 4) identical to $P^l_{\theta}$ (Eq. 1)? Eq.6 is unclear, since $q_\theta$ is not introduced. Whether it's contrastive learning among two views of a single image, or among a batch is not clear from the equation.
Some symbols in the Method Section are not explained.

- The experiments only show performance on ResNet50, whether the proposed method scales to larger backbones is not clear.

- The COCO detection task is only trained and compared with on 1x schedule, while recent study show that 1x schedule is far from convergence. Hence the results are not very conclusive, it's likely that the proposed method only converges faster, but after training longer, the performance will become similar with comparison methods.

- Type: L211 "solarization".

[1] Ke, Tsung-Wei, Jyh-Jing Hwang, and Stella X. Yu. "Universal weakly supervised segmentation by pixel-to-segment contrastive learning." ICLR 2021.
[2] Zhou, Tianfei, et al. "Rethinking Semantic Segmentation: A Prototype View." CVPR 2022.
questions: - How does the proposed method perform on image classification tasks? e.g., ImageNet.

limitations: The authors addressed the limitations and negative societal impact in the supp.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method for learning objectness from visual data by combining deep clustering in feature space and a contrastive objective that is made invariant to geometric augmentations.

The method involves two losses:
- $L_\text{group}$ which operates per-pixel and that encourages each pixel to be clustered according to the clusters produced by an EMA-updated teacher,
- $L_\text{slot}$ which operates at the slot level and ensures consistency between slot representations in different augmentations of the same image through a contrastive objective.

After self-supervised pre-training, the encoder is used as the backbone for several vision tasks, namely object detection, semantic segmentation and instance segmentation. In all cases, fine-tuning the pre-trained model yields better results than starting from a random initialization, provided that the number of fine-tuning epochs is equal for the pre-trained and from-scratch models.
strengths_and_weaknesses: The paper is well written and easy to follow, and I am overall positive about it.
The structure of the text introduces the method with gradual complexity, which helps navigating the multitude of symbols and variables.
Although figure 1 appears confusing at first because it has "too much going on", it becomes progressively clearer with the contents of section 3.1 and 3.2.
Most details are not given in the main text, but the supplementary material goes into great detail about datasets and training schedules.

In terms of originality, the paper fits into the popular category of deep clustering methods.
This category has recently seen a shift from image-centric clustering to a finer level of detail.
The strength of this method is to explicitly optimize for desirable properties of clusters, rather than trying to extract clusters from models trained for another objective.

Indeed, the formulation of $L_\text{group}$ resembles a per-pixel extension of DINO because of the prototype-based unsupervised clustering.
Thus, I am happy that the authors anticipated this discussion at the end of Section 3.1.
I agree with the distinction that DINO uses a large number of prototypes while this method uses a smaller number of per-object prototypes.
My intuition is that DINO prototypes capture scene semantics that can be more complex than patch semantics due to compositionality, hence the need for more prototypes.
I am not sure, however, how the prototypes of this method can be "adaptive to each image" (L164).
Eq. 1 clearly shows that the assignments for $L_\text{group}$ are obtained by computing a similarity with learned dataset-wide prototypes, same as DINO. It's only in eq. 4 that slots are extracted by pooling image features using prototype-based attention, these slots can be said to be "adaptive". It would be good to clarify the terms "prototypes", "assignments" and "slots".

Regarding the experiments, I appreciate their exhaustiveness and the choice of datasets.
My main complaint is directed at the lack of confidence intervals for the reported metrics.
Considering how much results can vary due to randomness it is important to report (at least) the mean and std of 3 runs to be sure that improvement is due to the proposed method.
If the number of pages is a constraint at least provide extended versions of tab. 2-6 in the appendix.

Last, a critique for the choice of ablation studies.
Of course, it is interesting to study how the method behaves with a different number of prototypes or different weights for the losses.
However, I do not think the chosen ablations target the most important points of the method.
It would be more interesting to ask: are geometric augmentations (and the subsequent inversion) necessary to learn object-centric representations? Would the model learn something different if only color augmentations were employed on two identical crops of each image? What happens when the binary indicator is not used and all slots contribute to the loss? Would an image-level contrastive objective improve or hinder learning object-level prototypes?

Minor:
- L132 revise "two-layer multilayer perceptron"
- L162 "much less prototypes" -> "fewer"
questions: It would be interesting to discuss statistics about the binary indicator in eq. 5 and 6:
- How many slots are active on average for each image? This is mentioned in the supplementary for estimating FLOPS but not discussed from the perspective of representation learning.
- How often is one slot active over the whole dataset?
- How many terms are excluded in eq. 6 because either $1_\text{teacher}$ or $1_\text{student}$ are 0?
- Is there any regularization to ensure that all slots are used evenly? Or to ensure a min/max number of slots per image?

L226: Why the number of prototypes is 256 for COCO and 2048 for ImageNet? Intuitively COCO contains more diverse objects than ImageNet and should use more prototypes.

L281: the text mentions that the method "successfully localizes small objects" w.r.t. column 5 of the figure in table 5. However, that colum depicts a laptop on a bed that are recognized as a small piece of "sky" on the "floor". How come the method has learned to distinguish small portions of pixels but hasn't learned much about semantics? Shouldn't the two losses in eq. 8 optimize for semantic consistency?

Is there anything in the method that explicitly biases the optimization towards learning object-centric representations as opposed to parts?
If not explicitly, why do you think the model learns to group together entire objects rather than parts? Alternatively, do you have evidence that no preference is made between objects and parts?
limitations: Yes, but only in the supplementary material.
If possible, I would appreciate a mention of limitations in the main text with a link to the corresponding section in the appendix.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors design a method to learn visual representations in a self-supervised fashion from scene-centric data. The work is built upon the choice of exploiting intrinsic scene-centric characteristics of the data, and contrasting with existing literature that exploits object-related priors to guide the learning process. 

The method includes a pixel-level semantic grouping learning mechanism based on learning a set of prototypes to which the pixels are assigned. The set of prototypes can adapt to each sample by an adaptive pooling mechanism.
strengths_and_weaknesses: *Strenghts*
- novelty: the work approaches the self-supervised learning strategy in a novel way favoring pixel-level semantic grouping rather than image-level or object-level representation learning pre-text tasks. The design of the learning procedure has its merit, and the authors state clearly similarity, inspirations from and differences with other methods (e.g. DINO)
- technical soundness: the description of the method is clear and well-argumented, From my perspective, it has a good balance of mathematical formulation, motivations and textual explanation of the several concepts proposed.
- experiments on benchmark datasets: the experimental analysis is done on ImageNet1K and COCO(+), which guarantees that the results can be analysed and commented in full
- the paper is very well-written, clear and concise

*Weaknesses*
- results: results are comparable with those of existing approaches, which have different working principles. Most of the time the slight improvements are marginal (or in some cases the results are also marginally lower than others). This does not allow to fully appreciate the quality and usefulness of the method. 
- motivation/background: the main difference with other works is stated to be the fact that no priors related to instance discrimination are used in the pre-text task for self-sup training as they will limit the learning potential. This moves the authors to focus on only exploiting the data intrinsic properties and characteristics. I find this a stretched motivation, as priors related to human knowledge about any problem have been demonstrated to steer the learning process in favor of good performance and efficient use of the data (avoiding to re-learn that prior knowledge somehow from the data).  The fact that this argumentation is weak, is reflected also in the results, which do not show that using only knowledge from the data would contribute to overall better performance (see point above).
questions: I would revisit the statements about the use of prior knowledge as a guidance to the training process, and relate this work better with existing streams of work that show successful use of priors both for performance improvement and data-efficiency (e.g. in the VIPriors workshop series). 

The results to not clearly show superiority of the proposed approach, lacking a full support of some statements in the introduction. Neither the results show complementarity, in the way they are presented. An error and overlap prediction analysis wrt other methods would contribute to a better understanding of the cases in which one method is better than another. Also, marginal improvements might be checked with statistical tests of significance. If not done, these marginal improvements are bound to be interpreted as a result of randomness.
limitations: The authors do not state limitations explicitly. They could discuss more extensively in which cases this approach would be better than others and in which cases not. From the results it is not clear why and when this method would be better to be used.
I do not find direct implications as potentially negative societal impact. It is a general approach to learn CV models, and its uses are very broad.
review_title: null

Meta-review: metareview: This paper proposes an object-centric representation learning based on a data-driven semantic slots from scene-centric data. In specific, the proposed SlotCon simultnesobly performs semantic grouping and contrastive representation learning over groups (slots), which naturally leads to obtaining object-level representations without any prior knowledge. The proposed algorithm is technically sound and novel. It is clearly distinct from the previous dense contrastive learning in that it jointly learns the target grouping of pixels. Most of all, it first shows encouraging performances by object-centric representation learning on natural image datasets. Even though the performance improvements seem to be somewhat marginal in comparison to the previous SOTA algorithms, the proposed method fairly demonstrates the effectiveness and feasibility in the use of object-centric representation learning for scene-centric data and the corresponding downstream tasks. In addition, the authors properly addressed almost all concerns and questions raised by the reviewers. In conclusion, I would like to recommend to accept this paper.
confidence: Certain
Decision: Accept