Kati8/ICLR_2023_2025 huggingface dataset:
authors: Tim Lebailly
authors: Thomas Stegmüller
authors: Behzad Bozorgtabar
authors: Jean-Philippe Thiran
authors: Tinne Tuytelaars
reviews: 
Comments: This paper works on self-supervised learning from scene-centric images, and locates in between the use of object-level contrastive learning and cross-image nearest neighbours. Compared with prior related work Hummingbird, it proceeds on object regions discovered by the network online instead of pixels, enabling strong representations for both in-context learning evaluation and traditional evaluations (linear probing & fine-tuning). A novel cycle-consistency strategy, which requires NN's another view to be of consistent semantic with another view's NN, creates positive pairs with reasonable variation, and helps improve representation. Experiment results are supportive.
Strengths: *Originality*: The core idea of this paper is novel and original. Though the overall framework still fits in common practices of object-level contrastive learning, the integration of nearest neighbours and the design of cycle consistency are elegant and useful.

*Quality*: The proposed method is well-motivated and extensively evaluated. Its relationship to prior works on different topics is also clearly discussed. I also like the discussions on over-clustering and how it helps scene-centric learning (compositionality), which provided some new insight.

*Clarity*: The delivery is very clear and easy to understand, I did not find issues in understanding.

*Significance*: This work is inspiring and could be helpful for both in-context scene understanding and representation learning.
Weaknesses: - (minor) It would be better if a comparison in GPU memory / time cost in both pre-training and in-context inference stages. 
- (minor) The comparison of scene-centric learning is limited to those with ViT-based architectures. If possible, other closely related works on this topic should also be discussed and compared (eg, ORL, SlotCon, COMUS). For instance, I find from SlotCon similar visualizations as fig.3, and also similar model-discovered compositional concepts (eg, parts). Is it possible to apply in-context learning evaluation to these prior works?

References:
- ORL: Unsupervised object-level representation learning from scene images, NeurIPS'21
- SlotCon: Self-supervised visual representation learning with semantic grouping, NeurIPS'22
- COMUS: Unsupervised semantic segmentation with self-supervised object-centric representation, ICLR'23
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The authors propose a shift in paradigm for self-supervised learning (SSL) methods; most SSL methods consider global (image-level) representations and are trained on object-centric datasets but the majority existing real-world images are instead composed of multiple objects (scene-centric). Their proposal is to add to the traditional cross-view consistency with cross-view object-level consistency and cross-image object-level consistency, approaching every image as a collection of objects that may also be present in other images. They show state-of-the-art performance on dense nearest neighbors retrieval and segmentation with a linear head and have SoTA-comparable performance when fine-tuned for segmentation.
Strengths: - The manuscript is easy to follow and compelling to read. Preliminaries are helpful and the organization of the design into three steps is easy to follow.
- The authors present a direct improvement on the method of Hummingbird with a more general framework for scene-centric SSL
- The method is the SoTA on dense nearest neighbors retrieval without any fine-tuning steps
- Empirical evidence shows that each addition was beneficial to the final performance; I have found the B.2 and B.4 experiments to be particularly beneficial to support the implementation of the cycle consistency criterium, one of the "riskier"/less intuitive ideas from the paper.
Weaknesses: - While most of the paper is easy to follow, the section on cycle-consistency matching (3.2.2) is particularly convoluted; on a similar note, Figure 3 is not very intuitive on a first glance and it is hard to determine "who is being compared to whom" to determine consistency.
- Experiments on linear segmentation fail to comment on direct backbone comparisons; this is important since it seems that the model is only better than TimeT (Salehi et al., 2023) -- a ViT-S/16 model when using a larger ViT-B/16 backbone; this is probably due to the larger training set used but still deserves a mention.
- Opinion: I disagree that the fine-tuning regime is suboptimal for comparing SSL models since this is the regime that is going to be used by partitioners when enough training data is available; I also do not think lower/comparable performance in one of the downstream tasks detracts from the manuscript. 

Extra notes:
- Table 6 has swapped headers (dataset and epochs)
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent