Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper aims to improve contrastive learning for multi-object scenes. In addition to making single-object representations close when objects are similar, the paper seeks to encourage multi-object scenes that share similar objects to also be close in the representation space. However, a given number of objects can be composed into exponentially many possible scenes, making it challenging to learn scene representations in the Euclidean space. Therefore, the proposed method maps the scene representations to a hyperbolic space, which is better at handling combinatorial explosion, and proposes a hyperbolic loss. Experiments show that the hyperbolic loss leads to 1~2% improvement in downstream classification, detection, and segmentation tasks. It is also shown that the learned scene representation can be used to quantify label uncertainty and detect out-of-context objects without further training.
strengths_and_weaknesses: - Strengths
    - SSL works well on images containing one dominant object, but has limitations on multi-object images. The paper tackles this limitation and is relevant to the community.
    - The proposed hyperbolic loss is quite novel and well motivated. It also leads to an interesting property of the representation that a larger norm indicates more object types in the image.
    - The paper is well written. It is easy for me to understand the high-level idea behind using the hyperbolic space although I am unfamiliar with the exact mathematical definitions.
- Weaknesses
    - Empirical evaluation seems insufficient. As mentioned in the Related Work section, quite a few models have been proposed to improve SSL on multi-object datasets, but none of them are included as a baseline. As an example, ORL[1] achieves 59% top-1 accuracy on ImageNet-1k when pretrained on COCO, while the proposed method only achieves ~55%.
    - Comparison to MoCo-v2 seems unfair, because MoCo-v2 only receives the full image as input while the proposed method takes object regions.
    - The definitions in L105-111 do not seem to form a tree. Consider an image $s$ with three objects $a$, $b$, and $c$, and a region $r$ covering objects $a$ and $b$. Then $a$, $r$, and $s$ form a cycle, so it is not a tree.

[1] Unsupervised Object-Level Representation Learning from Scene Images. (https://arxiv.org/pdf/2106.11952.pdf)
questions: Please see my main concerns listed in Weaknesses. Additionally, I have the following two minor questions:
- I did not understand why the objects are the root nodes (rather than the scenes).
- Is there an intuitive explanation why the norm of representation will be large when there are multiple objects?
limitations: Limitations and potential negative societal impact have been addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a hyperbolic contrastive objective function (HCL) to formulate the object-centric scene hierarchy for self-supervised learning on scene images. The main motivation of the paper is that the hyperbolic space is more suitable to embed the hierarchical structure than
 Euclidean space. Using the ground truth or Selective search to obtain objects, this paper extends another branch on MoCo to constrain the similarity of scene crops and object crops in the hyperbolic space. The paper shows superior performance on downstream visual tasks when compared with its baseline method.  HCL also shows interesting properties on label uncertainty quantification and out-of-context object detection.
strengths_and_weaknesses: Strengths:
1. The paper utilizes the hyperbolic space to formulate the scene-object relationship, which builds up another way to represent scene images (small norms represent objects and large norms represent scenes).
2. The paper achieves better downstream tasks than its baseline.

Weakness:
1. The experimental results only contain the comparison with baseline. Several relevant papers [1,2], which focus on the problem of self-supervised learning on scene images and also use Selective Search to acquire objects, are not mentioned in the experimental section. 
2. The paper only trained HCL for 200 epochs, it is not clear whether the performance gain can be maintained when training for more epochs (800ep). 


[1] Xie, Jiahao, et al. "Unsupervised object-level representation learning from scene images." Advances in Neural Information Processing Systems 34 (2021): 28864-28876.
[2] Li, Zhaowen, et al. "UniVIP: A Unified Framework for Self-Supervised Visual Pre-training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
questions: 1. Train HCL for 800 epochs and compare HCL with ORL and other Selective Search-based methods.  This result will be more important to support the claim of hyperbolic loss.
2. The paper claims in Line 213 that "HCL would improve more on the dataset with more objects per image", which is not sound enough. On OpenImages, the paper only compares HCL w/ and w/o hyperbolic loss when using the ground truth bounding boxes (which can not be acquired during self-supervised learning). The authors should provide the performance of HCL/${L_{hyp}}$ with SS to support the claim.
3. Is the additional hyperbolic loss time-consuming?
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors address the problem of contrastive representation learning of objects and scenes simultaneously. To this end, they employ the Poincaré embedding to model a hierarchy from simple (single object) towards more complex (multi object, scenes) images.
In the experiment it is shown that HCL improves over MoCo-V2 [1]. The method is trained on COCO (118,000 images) or OpenImages (212,000 images) for 200 epochs.
strengths_and_weaknesses: Strengths:
- The idea of using hyperbolic geometry in conjunction with contrastive learning is interesting and promising.
- Results indicate the proposed method works: Quantitative improvements as well as representation norms correlating with number of labels.

Weaknesses:
- Performance gains of using hyperbolic loss are small and improvements might be due to using additional information, i.e. detections, instead of only the images.
- Relevant baselines are missing from Table 1. For detection for example [3, 4] achieve similar performance without relying on detection labels for pre-training.
- The claimed performance improvement is only shown over a weak baseline (MoCo-V2 on a small dataset). Performance in general does not seem competitive with current SSL models. For example, MoCo-V2 [1] achieves an accuracy of 67.5% when trained on ImageNet-1K for 200 epochs (71.1% after 800 epochs), the more recent DINO [2] even 75.3%. HCL achieves only an accuracy of 58.5% despite using the additional information of where objects are in an image. (This is of course difficult to compare due to using a different pre-training dataset, but a proper comparison to other SSL models would be needed here).
- The used datasets are small for SSL. To show that this method is useful in practice, I'd like to see some scores on a more competitive benchmarks. E.g. by using a dataset of similar size to ImageNet.
- Compared to other contrastive methods, HCL requires supervision. While selective search is proposed as a way to avoid reliance on ground truth, it decreases performance. 

Minor remarks:
- "...treated using the same lens": This contradicts the claims of the next sentences
- Fig. 4 should indicate standard deviations.
- The precise task on the VOC dataset is unclear since there are multiple. I assume multi-label classification on Pascal VOC 2011 is meant but this should be explicitly stated.
- L189: MoCO-V2 should be trained using the hyperparameters from [1] and not the ones that work best for HCL (i.e. learning rate etc). MoCo-V2 is trained using a cosine scheduler which was apparently not being used here.


The paper presents an interesting idea of applying contrastive learning on multi-image scenes using poincare embeddings and promising first results. However, the experiments fail to convince me. 
There is a large gap between this method's and state-of-the-art performance in self-supervised representation learning, which raises concerns regarding the utility of the method.

[1] Chen et al. Improved Baselines with Momentum Contrastive Learning
[2] Caron et. al. Emerging Properties in Self-Supervised Vision Transformers
[3] Wang et al. Dense Contrastive Learning for Self-Supervised Visual Pre-Training 
[4] Xiao et al. Region Similarity Representation Learning 
questions: A more extensive comparison to other SSL methods would be necessary, especially to other methods that focus on object-level contrastive learning and/or that use object detection as additional information such as HCL.
limitations: Limitations are briefly addressed.
review_title: null

Meta-review: metareview: Overall, reviewers found that the method is sound but the results are marginal.

There are numerous frameworks for self-supervised learning today. The one introduced here underperforms compared to others, like ORL and Dense-CL, as pointed out by the reviewers. The authors in their response then combined their method with ORL and Dense-CL. This resulted in 1.1% improvement over ORL. This is marginal. I understand that the authors intended to reposition their work as a general-purpose add-on that increases performance. But, far more analysis is required to establish that this 1.1% improvement is real and that is meaningful. There are many tricks for SSL that improve performance by 1% or so, often these are not used because the slowdown they incur is not worth the effort. And the increase in performance is not noticeable.

At least, if the authors wish the pivot in this way, then the manuscript requires a rewrite to read as an add-on to many methods and to properly evaluate this. As it stands, by not comparing against ORL and other methods in the main manuscript, the submission cannot be accepted as is.

I encourage the authors to submit to a computer vision venue where such results may be appreciated more; where they may be given more room to thrive into something bigger. And to fully incorporate methods like ORL while demonstrating that their method really does produce a meaningful improvement.

confidence: Certain
Decision: Reject