Hyperbolic Contrastive Learning for Visual Representations beyond Objects
Authors: Songwei Ge
Shlok Kumar Mishra
Simon Kornblith
Chun-Liang Li
David Jacobs
Venue: NeurIPS 2022 Submitted (2022)
Keywords: contrastive learning
self-supervised learning
Riemannian geometry
representation learning

Abstract: Despite the rapid progress in visual representation learning driven by self-/un-supervised methods, both objects and scenes have been primarily treated using the same lens. In this paper, we focus on learning representations for objects and scenes explicitly in the same space. Motivated by the observation that visually similar objects are close in the representation space, we argue that the scenes and objects should further follow a hierarchical structure based on their compositionality. To exploit such a structure, we propose a contrastive learning framework where a Euclidean loss is used to learn object representations and a hyperbolic loss is used to regularize scene representations according to the hierarchy. This novel hyperbolic objective encourages the scene-object hypernymy among the representations by optimizing the magnitude of their norms. We show that when pretraining on the COCO and OpenImages datasets, the hyperbolic loss improves downstream performance across multiple datasets and tasks, including image classification, object detection, and semantic segmentation. We also show that the properties of the learned representations allow us to solve various vision tasks that involve the interaction between scenes and objects in a zero-shot way.

Hyperbolic Contrastive Learning for Visual Representations beyond Objects
Anonymous Author(s)
Affiliation Address email
Abstract
Despite the rapid progress in visual representation learning driven by self-/un- supervised methods, both objects and scenes have been primarily treated using the same lens. In this paper, focus learning representations for objects and scenes explicitly in the same space. Motivated by the observation that visually similar objects are close in the representation space, we argue that the scenes and objects should further follow a hierarchical structure based on their compositionality. To exploit such structure, we propose contrastive learning framework where Fuclidean loss is used to learn object representations and a hyperbolic loss is used to regularize scene representations according the hierarchy. This novel hyperbolic objective encourages the scene-object hypernymy among the representations by optimizing the magnitude of their norms. We show that when pretraining on the COCO and OpenImages datasets, the hyperbolic loss improves downstream performance across multiple datasets and tasks, including image classification, object detection, and semantic segmentation. We also show that the properties of the learned representations allow us to solve various vision tasks that involve the interaction between scenes and objects in zero-shot way.
2
4
6
8
10
11
12
13
14
15
Introduction
18
Our visual world is diverse and structured. Imagine taking close-up of box of cereal in the morning. we zoom out slightly, we may see different nearby objects such as bowl of milk, cup of hot coffee, today's 's newspaper, reading glasses. Zooming out further, we will probably recognize that these items are placed on dining table with the kitchen background rather than inside bathroom. Such scene-object structure is diverse, not completely random. In this paper, we aim at learning
19
20
21
23 visual representations of both the cereal box (objects) and the entire dining table (scenes) in the same
24 space while preserving such hierarchical structures.
25 Un-/self-sppervise learning has become standard method learn visual representations [27, 12, 25, 13, 6, 7, 49]. Although these methods attain superior performance over the supervised pretraining 28
27 on object-centric datasets such as ImageNet [25. 6]. inferior results are observed on images depicting
28 multiple objects such as OpenImages or COCO [67]. Several methods have been proposed to mitigate
29 this issue [67, 68, 37, 1], but all focus on learning improved object representations or dense pixel representations, instead of explicitly modeling the representations for scene images. The object representations learned by these methods present a natural topology [66]. That is, the objects from visually similar classes lie close to each other in the representation space. However, it is not clear
30
31
33 how the representations of scene images should fit into that topology. Naively applying existing contrastive learning results in sub-optimal topology of scenes and objects as well as unsatisfactory 35 performance as we will show in the experiment. To this end, we argue that a hierarchical structure can be naturally adopted. Considering scenes the composition of different kinds of objects,
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

37 can construct a forest structure to describe such relationships, where the root nodes are the visually similar objects, and the scene images consisting of them are placed as the descendants. We call this structure the object-centric scene hierarchy.
38
The intermediate modeling difficulty induced by this structure is the combinatorial explosion. A finite number of objects can lead exponentially many kinds of scenes due to the composition. Hy- perbolic space is known for its provably better capacity in modeling infinite trees compared with Euclidean space [21, 26, 34]. Therefore, we pro- pose to employ a hyperbolic objective to regularize the scene representations. Our framework builds upon MoCo [27], which has been shown to learn good object representations. To learn representa- sample the co-occurring scene-
41
42
43
44
45
46
47
48
49
50
tions of scenes, object pairs as the positive pairs, and objects that not part of that scene the negative samples, 54 and use these pairs to compute an auxiliary hyper- bolic contrastive objective. Our model is trained 56 to reduce the distance between positive pairs and push away the negative pairs in hyperbolic space.
52
ORDER
58
Contrastive learning models generally compute their objectives on hypersphere [27, 12]. By discarding the norm information, these models effectively circumvent the shorteut of minimizing objectives by tuning the norms and obtain better downstream performance. At the same time, they also lose control of the representative power in the magnitude the norm and leave the images disorganized. However, in hyperbolic space, is the magnitude of the norm that is used to model the hypernymy of the hierarchical structure [43, 58, 51]. When projecting the representations to the hyperbolic space, the norm information preserved and used determine the Riemannian distance, which eventually affects our loss. Since the hyperholic space is diffeomorphic and conformal to the Euclidean, our hyperbolic contrastive loss completely differentiable and complementary to the original contrastive objective.
Figure 1: Illustration of the representation space learned by models. Object images of the same class tend to gather near the center around similar directions, while the scene images are far away in these directions with larger norms.
59
60
62
64
86
68
70
When training simultaneously with the original contrastive objective for objects and our proposed hyperbolic contrastive objective for scenes, the resulting representation space exhibits the desired hierarchical structure while keeping the object clustering topology intact as shown in Figure 1. We demonstrate the effectiveness of the learned representations several downstream tasks, from image classification to object detection. We also show that the properties possessed by the representations allow us to perform various vision tasks in zero-shot way, from label uncertainty quantification to out-of-context object detection. Our contributions are summarized below:
72
74
76
77
1. We propose to learn representations for both object and scene images simultaneously using -tffsself-uupervisee methods. We identify object-centeric scene hierarchy that the representations are expected to follow.
79
80
2. We propose a framework with novel hyperbolic contrastive loss to regularize the scene representations with positive and negative pairs sampled from the hierarchy.
81
82
3. We show that the magnitude of representation norms effectively reflect the scene-objective hypernymy, and such representations transfer better to multiple downstream tasks.
83
2
Method
85
In this section, we elaborate our approach to learn visual representations of object and scene images. We start with describing the hicrarchical structure between objects and scenes.

---

87
2.1
Object-Centric Scene Hierarchy
88
From simple object co-oceurrence statistics [20, 39] to finer object relationships [29, 31J, using hierarchical relationships between objects and scenes to understand images is not new. Previous studies primarily work on an instance-level hicrarchy by dividing an image into its lower-level clements recursively scene contains multiple objects, object has different parts, and each part may consist of even lower-level features [47, 46, 15J. While this is intuitive, it describes a hicrarchical structure contained in the individual images. In our task, would like work the structure from the view of the entire dataset to learn a representation space shared by objects and scenes. To this end, argue that it is more natural to consider an object-centric hierarchy.
90
91
92
94
95
96 is known that when training an image classifier, though not being optimized directly, the objects from visually similar classes often lie close to each other in the representation space [66], which has become the cornerstone of contrastive learning [27, 12]. Motivated by this observation, we believe that the representation of each scene image should also be close to the object clusters it consists of. However, they require a much larger volume due to the exponential number of possible compositions. Another way to think about the object-centric hierarchy is through the generality and specificity as often discussed in the language literature [40, 43J. An object concept is general when standing alone in the visual world, and it will become specific when certain context is given. For example, "a desk" is thought to be a more general concept than "a desk in a classroom with a boy sitting on it".
97
96
100
102
103
104
105
Therefore, we propose to study object-centric hierarchy across the entire dataset. Formally, given a set of images S = {si, 82, ,8n), O {ot o7, ...... , } are the object bounding ,r("") to be
106
107
boxes contained in the image 8;. We define the regions of scene R. =
partial areas of the image s; that contain multiple objects such that r! Uko where of
108
109
Oj and object k is in the region j. We define the object-centric forest T = (V, E) to be that V
110
(u, v) is an edge of T if C v or C u. Note that the natural scene images S are always put as the leaf nodes.
SUOUR, R. where R R2 U..U..RR R,, and O O₁ UO...U, For u, u V,
111
112
2.2
Representation Learning beyond Objects
113
To describe our proposed model that is built on this hierarchy, we begin with a brief review of the hyperbolic space and its several properties that will be used in our model. For comprehensive introductions to the Riemannian geometry and hyperbolic space, we refer the readers to [32, 17].
114
115
116
2.2.1
Hyperbolic Space
117
A hyperbolic space (HI" g) is a complete, connected Riemannian manifold with constant negative sectional curvature. These special manifolds are all isometric to each other with the isometries defined as O+ (m, 1). Among these isometries, there are five common models that previous studies
118
119
often work on [5]. In this paper, we choose the Poincaré ball DD == (p E RR ||p||² < 22) as our
120
121
basic model [43, 58, 22], where
is the radius of the ball. The Poincaré ball is coupled with
122
a Riemannian metric (p(p) (1-1eePP/ay 9E, where E D and Øa is the canonical metric of the
123
Euclidean space. For p, q E DD, the Riemannian distance the Poincaré ball induced by its metric gn
124
is defined as follows:
(I-P- 91l
dp q) 2r tarh-1
125
where @ is the Möbius addition and is clearly differentiable. In addition, the Poincaré ball can be natural counterpart of the hypersphere it allows all directions, unlike the other models such as the halfspace or hemisphere models that have constraints on the directions. The hyperbolic space is globally differomorphic to the Euclidean space, which is stated in the theorem below:
126
viewed
127
128
129
Theorem 1. (Cartan-Hadamardd) For every point p E T the exponential map exp, TT TH a
130
R"
H is a smooth covering map. Since H is simply connected, it is diffeomorphic to TR...
131
Specifically, for p E D and v E T2DD xe R", the exponential map of the Poincaré ball exp,
T,D" + 1 is defined as
r|le||
r
exp,(v) = p @ ttanh 22
(2)
12
||
3

---

Encoder Backhone
Normalize
Zeuc
Cosine
Similarity
Momentum Encoder Backbone
Normalize
2 Zeuc
Exponential al Map
Encoder Backhone
2 kyp
Objoct
Hyperbolic Distance
Scene
Momentum
Exponential al Map
Encoder Backhone
hyp
Figure 2: Our Hyperbolic Contrastive Learning (HCL) framework has two branches: given scene image, two object regions are cropped to learn the object representations with a loss defined in the Euclidean space focusing on the representation directions. A scene region as well as a contained object region are used to learn the scene representations with a loss defined the hyperbolic space that affects the representation norms.
133
The exponential map gives us way map the output of network, which in the Euclidean space, to the Poincaré ball. In practice, to avoid numerical issues, we clip the maximal norm of with r before the projection, where
134
135
0. During the backpropagation, we perform RSGD [4] by scaling the gradients with gD (p)_1. Intuitively, this forces the optimizer to take a smaller step when is closer to the boundary. The scaling factor is lower bounded by (c
136
137
138
The immediate consequence of the negative curvature is that for any point E H"", there are no conjugate points along any geodesic starting from p. Therefore, the volume grows exponentially faster in hyperbolic space than in Euclidean space. Such property makes it suitable to embed the hierarchical structure that has constant branching factors and exponential number of nodes. This is
139
140
141
142
formally stated in the theorem below:
Theorem 2. 121, 26] Given Poincaré ball DD' with an arbitrary dimension n 2 and any set of points Pı,' .Pm E DD, there exists finite weighted tree (T, dr) and an embedding J - D such that for all j.
dr (z), dp (2,, 2,,I Olog(1 + 22) log(m))
143
Intuitively, the theorem states that any tree can be embedded into a Poincaré disk (n = 2) with low distortion. On the contrary, it is known that the Euclidean space with unbounded number of dimensions is not able to achieve such low distortion [34]. One useful intuition [51] to help
145
understand the advantage of the hyperbolic space is given two points p. D s.t. ||||| = ||gl|.
dp q) + do (p. 0) + dp q), as ||l|| = ||l|| +
147
This property basically reflects the fact that the shortest path in tree is the path through the earliest common ancestor, and it is reproduced in the Poincaré when points are both close the boundary.
148
149
2.2.2
Hyperbolic Contrastive Learning
150
With the theoretical benefits of the hyperbolic space stated above, we propose a contrastive learning framework as shown in Figure We adopt two losses learn the object and scene representations. First, as shown in the top branch of Figure 2, we crop two views of jittered and slightly expanded object region as the positive pairs and feed into the base and momentum encoders to calculate the
151
152
153
154
object representations. We denote the output after the normalization to be z and 2 Considering the computational cost of large batch sizes, we follow MoCo [27, 14] leverage a memory bank to
155
store the negative representations zn which are the features 2 from the previous batches. The
156
157
Euclidean loss for this image is then calculated as:
exp (Z)nc 2 /T)
Leu:
log cxp (Z 2 (T) + En, cxp (Z Zire Ziie/T)

---

158
where 7 is a temperature parameter.
159
While this loss aims at learning object representations, objective to learn the representations for scene images. We sample the positive region pairs u and e from object-centric scene hierarchy T such that (u,v) v) E F. In other words, as shown in the bottom branch of Figure 2, the objects contained in one region are required to be a subset of the objects in the other. We sample the negative samples of u to be N₁ {u] v) & J. However, building and sampling from the entire hierarchy explicitly is slow and memory consuming. Instead, according to the assumption that there are exponentially more scenes than object classes in practice, given a scene image s, we always sample u E R U { to be a scene region, V E O to be an object that occurs in u, and N₂ to be the other objects that are not in u.
also design hyperbolic contrastive
160
182
163
184
165
186
167
168
The pair of scene and object images are fed into the base and momentum encoders that share the weights with the Fuclidean branch. However, instead of normalizing the output of the encoders,
169
170
use the exponential map defined in the equation (2) to project these features in the Euclidean space to
171
the Poincaré ball, which are denoted zyy and zhyp. Further, we replace the inner product in the
172
cross-entrope loss with the negative hyperbolic distance as defined in equation (1). We calculate the
173
hyperbolic contrastive loss as follows:
exp ( do (Z
hyyp Pugp)
Luyp - log
(-do (Zhyp: )/T + En exp -dd p/pp)/T)
exp
When minimizing the distances of all the positive pairs, With the intuition from Equation (3), it would 175 be beneficial to put the nodes near the root close to the center to achieve a overall lower loss. The
174
178 overall loss function of our model is as follows:
L Leue + Xhhyps
177
where À is an scaling parameter control the trade-off between hyperbolic and Euclidean losses.
Experiments
178
179
3.1
Implementation Details
180
Pre-training phase. We pre-train our method on two datasets: COCO [33] and subset of Open-
181
Images [41]. Both of these datasets are multi-object datasets; OpenImages [41] (~ 212k images) contains 12 objects on average per image and COCO (~ 118k) contains 6 objects on average. We experiment with both the ground truth bounding box (GT) and using selective search [60] following the previous method [67] (SS) to acquire objects. For the optimizer setups and augmentation recipes, we follow the standard protocol described in MoCo-v2 [14] unless denoted otherwise. We find that a base learning rate of 0.3 works better for us as compared to 0.03. We adopt the linear learning rate scaling receipt that Ir = 0.3 x BatchSize/ /56 [24] and batch size of 128 by default on NVIDIA also pre-train the baselines with learning rate of 0.3. We train our models on both datasets for 200 epochs. For the hyperparameters of our hyperbolic objective, we use = 4.5, À = 0. and E = le More details on the OpenImages dataset as well as training setups can be found in Appendix A.
182
183
185
186
187
p6000 gpus. To ensure fair comparison,
189
191
192
Downstream tasks. We evaluate our pre-trained models on image classification, object-detection and semantic segmentation. For classification, we show linear evaluation (lineval) accuracy, i.c we freeze the backbone and only train the final fe layer. We test on VOC [19], ImageNet- 100 [57] and ImageNet-1k [16] datasets. To test the discriminative capacity of the representations on both objects and scenes, we create dataset by mixing the ImageNet- 100 and subset of Place-205 [70] datasets, which we refer to as the INPMix dataset. More details of this dataset can be found in Appendix A.
193
194
195
196
197
For object detection and semantic segmentation, we show results on the COCO and Pascal VOC trainval2017 datasets. For VOC object detection, COCO object detection and COCO semantic segmentation, we closely follow the common protocols listed in Detectron2 [65].
199
5

---

Detection
Dataset
AP
APso
APıs
Pre-train datasct
Bbox type
VOC
IN-100
INPMix IN-Ik
MoCo-v2
CoCO
34.6
53.5
37.0
HCL/Lmp
COCO
36.1
55.2
37.9
MoCo-v2
COCO
64.79
64.84
41.83
51.17
HCL
COCO
37.0
56.1
39.8
HCL/E
COCO
SS
73.13
73.84
51.28
54.21
HCL/Lhyp
COCO
GT
75.55
76.22
$1.25
54.52
MoCo-v2
VOC
51.5
79.4
56.1
HCL Lupp
VOC
53.7
80.5
59.4
HCL
COCO
74.19
75.16
51.35
55.03
HCL
VOC
54.4
81.4
60.2
HCL
COCO
GT
76.51
76.74
51.63
55.63
Segmentation Dataset
AP,
AP:
APm
MoCo-v2
OpenImages
69.95
72.80
49.59
54.12
HCI//Cup
OpenImages
GT
73.79
77.36
52.96
57.57
MoCo-v2
COCO 30.4
50.1
32.3
HCL.
OpenImages
SS
74.31
78.14
53.21
58.12
HCI//opp
COCO
31.5
52.0
33.8
HCL
Openlmages
GT
75.40
79.08
53.82
58.51
HCL
COCO
32.5 52.9
34.6
Table 1: Classification results with linear evaluation. Our model improves scene-level classification the VOC [19] and INPMix [70] datasets, and object-level classification on
Table 2: Object detection and Semantic Segmentation results. Our model im- proves on both tasks on COCO [33] and VOC [19] datasets.
ImageNet-100 100 [57] and ImageNet-1k [16] datasets.
3.2
Main Results
202
This section discusses our main results on the downstream image classification, object detection, and semantic segmentation tasks. As the goal of this paper is not present another state-of-the-an self-super learning method, we primarily compare with the backbone model MoCo-v2 [27]. Another important baseline consider is model without the hyperbolic loss Lhyps therefore only the object representations are learned, which we denote as HCL/Lhyp-
204
206
Image classification. As shown in Table 1, HCL improves image classification on both scene-level datasets (VOC and INPMix) and object-level datasets (ImageNet). When pretraining on OpenImages, HCL improves ImageNet lineval accuracy by 0.94% and VOC lineval classification accuracy by 1.61 mAP. We observe similar improvements when pretraining on COCO. HCL improves accuracy whether we ground truth object bounding boxes or boxes generated by selective search. In general, we observe a larger improvement using HCL on Openlmages than COCO, which supports our observation that HCL would improve more the dataset with more objects per images.
208
210
212
Object detection and semantic segmentation. Table 2 reports the object detection and semantic segmentation results using Mask R-CNN, following [14]. It shows consistent improvements over the baselines on VOC object detection, COCO object detection, and COCO semantic segmentation.
215
216
3.3
Properties of Models Trained with HCL
218
The visual representations learned by HCL have several useful properties. this section, we evaluate the representation norm as measure of the label uncertainty for image classification datasets, and evaluate the object-scene similarity in terms of out-of-context detection.
220
3.3.1
Label Uncertainty Quantification
Representation Norm
0.40
Datasets
Method
Indicator
IN-Real
COCO
0.35
MoCo
Entropy
0.633
0.791
Supervised
Entropy
0.671
0.793
0.30
HCL
Norm
0.655
0.839
Ensemble
Entropy+Norm 0.717
0.823
Number of Labels Figure Average representation
Table 3: NDCG scores of the image rankings based on the different indicators and models, and evaluated by the the number of labels per image.
norms of images with different num- [3].
ber of labels in ImageNet-
ImageNet [16] is an image classification dataset consisting of object-centered images, each of which has single label. As performance on this dataset gradually saturated, the original labels have been scrutinized more carefully [50, 59, 54, 3, 61]. Prevailing labeling issues in the validation set
223

---

Smallest norms (objects)
Largest norms (scenes)
Figure 3: Images from ImageNet training set. The 5 images on the left have the smallest representation norms among all the images from the same class, and the on the right have the largest norms.
have been recently identified [59, 54, 3]. including labeling errors, multi-label images with only single label provided, and so on. Although Beyer et al. [3] provide reassessed labels for the entire validation set, relabeling the entire training set can be infeasible.
226
228
Our learned representations provide a potential automatic way to identify images with multiple labels from datasets like ImageNet. Specifically, we first show in Figure 4 that there is a strong correlation between the representation norms and the number of labels per image according to the reassessed labels. For each class of the ImageNet training set, we rank the images according to their norms. The extreme images some classes are shown in Figure 3 and also Appendix. Images with smaller norms tend to capture a single object, while those with larger norms are likely to depict scene.
230
232
234
To quantitatively evaluate this property, we report the NDCG metric on the ranked images as shown in Table 3. NDCG assesses how often the scene images are ranked at the top. As a baseline, we rank the images based on the entropy of the class probability predicted by a classifier, which a widely adopted label uncertainty indicator [1 45]. We use both MoCo-v2 and supervised ResNet-50 as the classifier. As shown in Table 3, using norms with HCL. achieves similar rank quality as using entropy with the supervised ResNet-50 on the ImageNet-Real. dataset. In addition, when combining two ranks using simple ensemble methods such as Borda count, the score further improved to 0.717. This shows that the entropy and the norm might look different aspects of the multi-label issue. For example, the entropy indicator can be affected by the bias of the model and the norm indicator can be wrong on the images with multiple objects from the same class. In addition, our method is dataset agnostic and does not need further training. To demonstrate this benefit, we report the same metric on the COCO validation, where also have the number of labels for each image. Our method achieves much better NDCG scores than the supervised ResNet-50 as shown in Table 3. This finding can be potentially useful to guide label reassessment, or provide an extra signal for model training.
235
236
237
238
240
242
244
246
248
3.3.2
Out-of-Context Detection
249
Our hyperbolic loss Lnyp essentially encourages the model to capture the similarity between the object and scene. We further investigate this property on detecting the out-of-context objects, which can be useful in designing data augmentation for object detection [18]. We are especially interested the out-of-context images with conflicting backgrounds. To this end, we use the out-of-context images proposed in the SUN09 dataset [15]. We first compute the representation of each object as well as the entire scene image with that object masked out. We then calculate the hyperbolic distance between the representations mapped to the Poincaré ball. Some example images from this dataset as
250
251
252
253
254
255
256
well as the distance of each contained object are shown in Figure 5. We find that the out-of-context objects generally have a large distance, i.e. smaller similarity, to the overall scene image. To quantify
258
this finding, we compute the MAP of the object ranking on each image and obtain 0.61 for HCI, As
comparison, the MoCo similarity gives mAP- 0.52 and the random ranking gives mAp 0.44.
7

---

0.4547
0.5695
0.4949
0.5010
0.4206
0,4322
0
D 8025
0.4053
0.5393
0.1698
Figure 5: Out-of-context images from the SUNO9 dataset [15]. The bounding box of each object, as well as its hyperbolic distance to the scene are displayed. The regular objects are blue and the out-of-context objects are in purple. Note that the out-of-context objects tend to have large distances.
Main Ablation Studies
260
In this section, report the results of several important ablation studies with respect to HCL. All the models are trained on the subset of the Openlmages dataset and linearly evaluated on the
282
283
ImageNet-100 100 and INPMix datasets. The top-1 accuracy reported.
Dist.
Center
IN-100
IPS
IN-100
IPS
Optim.
IN-100
IPS
77.36
52.96
0.01
77.70
53.43
RSGD
0.1
79.08
53.82
Hyp.
Scene
79.08
53.82
0.1
79.08
53.82
RSGD
0.5
0
0
Hyp. Object
76.96
52.74
0.2
78.64
53.84
SGD
0.1
70.16
48.47
Euc.
Scene
76.68
52.58
0.5
0
0
SGD
0.5
74.18
42.75
Table 4: Ablation on the similarity Table 5: Ablation on the Table 6: Ablation on the RSGD ver- losses trade-off.
measure and hierarchy center.
sus SGD optimizers.
Similarity measure and the center of the scene-object hierarchy. We propose to use the negative hyperbolic distance as the similarity measure of the scene-object pairs. As an alternative, one can cosine similarity the hypersphere the measure just like the original contrastive objective. However, this basically minimizing the similarity between a single object and multiple objects. These objects are probably from different classes and hence conflict with the original objective. As shown in Table 4, replacing the negative hyperbolic distance with the Euclidean similarity impairs downstream performance. The resulting accuracy is even worse than the model without any loss function on the scene-object pairs. In terms of the hierarchy. we also test the assumption of scene- centric hierarchy [46, 47] by sampling the negative pairs the objects and unpaired scenes. However, we notice a significant decrease in the downstream accuracy with this modification in Table 4.
265
267
289
271
273
274
Trade-off between the Euclidean and hyperbolic losses. We adopt the Euclidean loss to learn object-ohject similarity and the hyperbolic loss to learn object-scene similarity. hyperparameter used to control the trade-off between them. As shown in Table 4, we find that a smaller X = 0.01 leads to marginal improvement. However, we also observe that larger As can lead to unstable and even stalled training. With careful inspection, we find that in the early stage of the training, the gradient provided by the hyperbolic loss can be inaccurate but strong, which pushes the representations to be close to the boundary. As a result, the Riemannian SGD causes the gradient to be small and the training is consequently stuck at some the early point.
275
277
279
281
282
Optimizer. With the observation above, we ask whether RSGD is still necessary for practical usage. We replace the RSGD optimizer with SGD. To avoid the numerical issue when the representations are too close to the boundary, we increase from le to 1e We first notice that this allows larger be used as opposed to the RSGD. However, SGD always yields inferior performance RSGD. Therefore, it shows that the accurate gradient provided by RSGD is still necessary.
283
284
285
Related Work
287
288
Representation Learning with Hyperbolic Space. Representations are typically learned in Fu- clidean space. Hyperbolic space has been adopted for its expressiveness in modeling tree-like
8

---

290
structures existing in various domains such as language [58, 21, 51, 43, 44], graphs [2, 8, 9, 48], and vision [30, 10, 56]. The corresponding deep neural network modules have been designed to boost the progress of such applications [9. 22, 35, 55]. The hierarchical structure presented in the datasets can come from multiple factors, motivating the use of hyperbolic space. 1) Generality: the hypernym- hyponym property is natural feature of words (e.g. WordNet [40]) and the hyperbolic space is
291
292
293
294
295
extensively exploited to learn word embeddings that preserve that property [58, 21, 51, 43, 44].
Some image datasets also adopt the classes from WordNet for labeling, e.g. ImageNet [16], and consequently inherits the hierarchy in its labeling system. [36, 69, 38] take advantage of hyperbolic
297
space to capture such information in the visual embeddings. 2) Uncertainty: Several studies have found that applying hyperbolic neural network modules to different tasks leads to a natural modeling of the uncertainty [23, 30, 56]. 3) Compositionality: The compositionality of different basic elements can form a natural hierarchy. We focus on learning the representations that capture the hierarchy
299
301
302
between the objects and scenes. The hierarchical representations learned in the hyperbolic space have been applied to various tasks with the aforementioned motivations such as image classification [30]
303
304
or segmentation [64, 23], zero-/few-shot learning [38, 36), action recognition [38], and video pre-
305
diction [56]. In this paper, we aim at learning image representations for general purposes that can transfer various downstream tasks.
308
307
Self Supervisd Learning on Scenes. Supe-Super Learning (SSL. has made great strides in closing the performance with supervised methods [12, 14] when pretrained on the object-centric datasets like ImageNet. However, recent works have shown that SSL are limited on the multi- object datasets like COCO [52, 63] and Openlmages [41]. Several works have tried to address this issue by proposing different techniques. Dense-CL [63] works on pre-average pool features and uses dense features on pixel level to show improved performance dense tasks such semantic segmentation. DetCon [28] uses unsupervised semantic segmentation masks to generate features for the corresponding objects in the two views. CAST [53] uses GradCAM [52] to figure out same objects across views and applies contrastive loss on these features. PixContrast [68] uses pixel-to- texel-to-proppe propagation consistency pretext task to build features for both dense downstream tasks and diseriminative downstream tasks. Pixel-to-Pixee Contrast [62] uses pixel-level contrastive learning to build better features for semantic segmentation. Self-EMD [37] uses earth mover distance with YOL [25] for pretraining on the COCO dataset. ORL [67] uses selective search to generate object proposals, then applies object-level contrastive loss to enforce object-level consistency. ContraCAM
309
311
313
314
315
316
317
318
319
320
321
[42] removes the scene bias issue by doing self-supervised object localization and performing contrastive loss on them. One of the reasons below-par performance of SSL methods can be attributed
322
323
to treating scenes and objects using similar techniques, which often results in similar representations.
In our work, instead of treating them in the same functionality, we use a hyperbolic loss, which builds representation that disambiguates scenes and objects based on the norm of the embeddings. Our method not only separates scenes and objects, but also helps us in improving downstream tasks such as image classification.
324
325
326
327
6
Closing Remarks
329
Conclusion We present HCL., contrastive learning framework that learns visual representation for both objects and scenes in the same representation space. The major novelty of our method is a hyperbolic contrastive objective built on an object-centric scene hierarchy. We show the effectiveness of HCL on several benchmarks including image classification, object detection, and semantic seg- mentation. We also demonstrate the useful properties of the representations under several zero-shot settings from detecting out-of-context objects to quantifying the label uncertainty in the datasets like ImageNet. More generally, we hope this paper can encourage studies towards building more holistic visual representation space and draw attention to the -on-Euclidean representation learning.
331
333
335
337
Limitations Our model is shown to improve the classification performance on the ImageNet dataset, but not much on the more fine-graines classification tasks as shown in Appendix B.2. We conjecture that the largest improvement brought by our model to the object representations are modeling the context information, while most of these datasets share general class whose contexts are more or
338
339
340
341
less similar. In addition, although we provide some insights about the Riemannian optimization, its underlying mechanism in the visual representation learning is still not fully understood. We conduct
343
more experiments on training hyperbolic linear classifiers in Appendix C. However, more efforts needed to fully unleash the potential of non -un-ldciid representation learning.

---

345 7
Societal Impact
Our work is technical contribution and much of societal impact depends upon the models used in work. We hope that our work will be used for betterment of the society and doesn't 't have any negative impact.
348
References
319
350
[I] Y. Bai, X. Chen, A. Kirillov, A. Yuille, and A. C. Berg. Point-level region contrast for object detection pre-training. CVPR, 2022.
351
352
[2] I. Balazevic, C. Allen, and T. Hospedales. Multi-relational poincaré graph embeddings. NeurIPS, 2019.
353
[3] L. Beyer, J. Hénaff, A. Kolesnikov, X. Zhai, and A. van den Oord. Are we done with imagenet?, 2020.
S. Bonnabel. Stochastic gradient descent riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2217-2229, 2013.
355
358
[5] J. W. Cannon, W. J. Floyd, R. Kenyon, W. R. Parry, et al. Hyperbolic geometry. Flavors of geometry. 31(59-115):2, 1997.
358
[6] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurlPS, 2020.
359
380
[7] Caron, H. Touvron, Misra, H. Jégou, Mairal, Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. ICCV, 2021.
361
[8] I. Chami, A. Wolf, D.-C. Juan, F. Sala, S. Ravi, and C. Ré. Low-dimensional hyperbolic knowledge graph embeddings. In ACL, 2020.
383
364
[9]
I. Chami, Z. Ying, C. Ré, and J. Leskovec. Hyperbolic graph convolutional neural networks. NewrlPS, 2019.
365
366
[10] Chen, J. Qin, Y. Shen, I., Liu, F. Zhu, and Shao. Learning attentive and hierarchical representations for 3d shape recognition. In ECCV, 2020.
387
[11] P. Chen, B. B. Liao, G. Chen, and S. Zhang. Understanding and utilizing deep neural networks trained with noisy labels. JCML, 2019.
369
370
[12] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
372
[13] T. Chen, S. Komblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big seff-supervised models are strong semi semi-supervised learners. NeurlPS, 2020.
373
374
[14] X. Chen, H. Fan, Girshick, and K. He. Improved baselines with momentum contrastive leamingg arXiv preprint XXXx:2003304297. 2020.
375
[15] M. J. Choi, J. J. Lim, A. Torralba, and A. S. Willsky, Exploiting hicrarchical context on large datahase of object categories. In CVPR, 2010.
377
378
[16] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image
379
database. In CVPR, 2009.
380
[17] P. Carmo and J. Flaherty Francis. Riemannian geometry, volume 6. Springer. 1992.
[18] N. Dvorikk, J. Mairal, and C. Schmid. On the importance of visual context for data augmentation in scene understanding. PAMI, 43(6):2014-2022 2019.
383
[19] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. IJC,, 88(2):303-333 338, 2010.
384
385
[20] C. Galleguillos, A. Rabinovich, and S. Belongie. Object categorization using co-occurrence, location and appearance. In CVPR, 2008
386
[21] O. Ganea, G. Bécigneul, and T. Hofmann. Hyperbolic entailment cones for learning hicrarchical embed- dings. In ICML, 2018
389
[22] O. Ganea, G. Bécigneul, and T. Hofmann. Hyperbolic neural networks. NewrlPS, 2018.
380
[23] M. GhadimiAtigh, Schoep, F. Acar, N. van Noord, and P. Mettes. Hyperbolic image segmentation. arXiv preprint arXiv:2203.00 05898, 2022.
391
382
[24] P. Goyal, P. Dollár. R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint 1rXiv:1706.02677 2017.
394
[25] J. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent new approach to self-supervised learning. In NeurIPS, 2020.
385
10

---

397
[26]
M. Gromov. Hyperbolic groups, In Essays in group theory, pages 75-263. Springer, 1987.
396
[27] K. He, Fan, Wu, S. Xie, and Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
399
[28] O. Hénaff, S. Koppula, J.-B. Alayrac, van den Oord, O. Vinyals, and J. Carreira. Efficient visual pretraining with contrastive detection. In ICCV, pages 10086-10096, 2021.
401
[29] J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bersstein, and L. Fei-Fei. Image retrieval using scene graphs. In CVPR, 2015.
403
404
[30] V. Khrulkov, L. Mirvakhabova, E. Ustinova, I. Oseledets, and V. Lempitsky. Hyperbolic image embeddings.
In CVPR, 2020.
406
[31] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJV, 2017.
407
409
[32]
J. M. Loc. Introduction to Riemannian manifolds. Springer, 2018
[33] T.-Y. Lin, M. Maire, S. Belongie, Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft Common objects context. In ECCV, 2014.
411
coco:
412
[34] N. Linial, E. London. and Y. Rabinovich. The geometry of graphs and some of its algorithmic applications.
Combinatorica, 15(2):215-245, 1995.
414
[35] Q. Liu, M. Nickel, and D. Kiela. Hyperbolic graph neural networks. NeurlPS, 2019.
[36] S. Liu, J. Chen, Pan, C.-W. Ngo, T.-S. Chua, and Y.-G. Jiang. Hyperbolic visual embedding learning for zero-shot recognition. CVPR, 2020.
416
[37] S. Liu, 7. Li, and J. Sun. Self-emd: Self-supervised object detection without imagenet, 2021.
418
[38]
T. Long, P. Mettes, H. T. Shen, and G. M. Snock. Searching for actions on the hyperbole. In CVPR, 2020.
419
420
[39]
T. Mensink, E. Gavves, and C. G. Snock. Costa: Co-oceurrence statistics for zero-shot classification. In CVPR, 2014.
421
[40] G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross, and K. J. Miller. Introduction wordnet: An on-line
423
lexical database, International journal of lexicography. 3(4):335-24 2244, 1990.
424
[41] S. K. Mishra, A. B. Shah, A. Bansal, A. N. Jagannatha, A. Sharma, D. Jacobs, and D. Krishnan. Object- aware cropping for self- celf-super learning. ArXiv, 19.031, 2021.
426
[42] S. Mo, H. Kang, K. Sohn, C.-L. Li, and J. Shin. Object-aware contrastive learning for debiased scene
427
representation. In NeurlPS, 2021.
428
[43] Nickel and Kiela. Poincaré embeddings for learning hierarchical representations. NeurlPS, 2017.
[44] M. Nickel and D. Kiela. Learning continuous hicrarchies in the lorentz model of hyperbolic geometry. In ICML, 2018.
431
[45] C. Northcutt, L. Jing, and I. Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411, 2021.
432
433
[46]
D. Parikh and Chen, Hierarchical semantics of objects (hsos). In ICCV, 2007.
[47]
D. Parikh, C. L. Ziticck, and T. Chen. Unsupervised learning of hierarchical spatial structures in images.
435
In CVPR, 2009.
436
[48] J. Park, J. Cho, H. J. Chang, and J. Y. Choi. Unsupervised hyperbolic representation learning via message
passing auto-eneoders. In CVPR, 2021.
438
[49] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
439
J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
440
[50] B. Recht, R. Roelofs, Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
441
[51] F. Sala, C. De Sa, A. Gu, and C. Ré. Representation tradeoff's for hyperbolic embeddings. In JCML, 2018.
443
[52] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via bradient-based localization. In ICCV, 2017.
444
445
|53| R. R. Selvaraju, K. Desai, J. Johnson, and N. Naik. Casting your model: Learning to localize improves self-supervised representations. In CVPR, 2021.
446
447
[54] V. Shankar, Roclofs, Mania, A. Fang, B. Recht, and Schmidt. Evaluating machine accuracy on
448
imagenet. JCMI. 2020.
[55] R. Shimizu, Y. Mukuta, and T. Harada. Hyperbolic neural networks++. In ICLR, 2021.
11

---

450
[56] D. Surís, R. Liu, and C. Vondrick. Learning the predictability of the future. In CVPR, 2021.
451
[S7]
Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. ECCV, 2020.
[58] A. Tifrea, G. Bécigneul, and O.-E. Ganea. Poincaré glove: Hyperbolic word embeddings. In ICLR.
OpenReview, 2018.
454
[59]
D. Tsipras. S. Santurkar, L. Engstrom, A. Ilyas, and A. Madry. From imagenet to image classification: Contextualizing progress on benchmarks. In JCML, 2020.
455
456
600]
Uijlings, E. Van Sande, T. Gevers, and W. Smeulders. Selective search object recognition.
457
ICV, 104(2):154 1112),144-11.,22 2013.
[61] V. Vasudevan, B. Caine. R. Gontijo-Lpes, Fridovich-Kete and R. Roelofs. When does dough become bagel? analyzing the remaining mistakes on imagenet. arXiv preprint arXiv:2205. 04496, 2022.
459
480
[62] W. Wang, T. Zhou, F. Yu, J. Dai, E. Konukoglu, and L. Van Gool, Exploring cross-image pixel contrast for semantic segmentation. In ICCV,
462
[63] X. Wang. R. Zhang. C. Shen, T. Kong, and L. Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021.
463
[64] Z. Weng, M. G. Ogut, Limonchik, and S. Yeung. Unsupervised discovery of the long-tail in instance segmentation using hicrarchical self-supervision. In CVPR, 2021.
465
665 Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick. Detectron2. https://gi com/
fa hatbooorressarcchdeettttroo 2019.
468
[66]
7. Wu, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
470
677
Xie, Zhan, Z. Liu, Y. S. Ong, and Loy. Unsupervised object-level representation learning from scene images. In NearlPS, 2021.
471
[68] Z. Xie, Y. Lin, Z. Zhang, Y. Cao, S. Lin, and H. Hu. Propagate yourself: Exploring pixel-Ievel consistency
473
unsupervised visual representation learning. CVPR, 2021.
474
[69]
J. Yan, L. Luo. C. Deng, and H. Huang. Unsupervised hyperbolic metric learning. In CVPR, 2021.
475
[70] Zhou, A. Lapedriza, Xiao, A. Torralba, A. Oliva, Learning deep features scene recognition
476
using places database. NeurlPS, 2014.
12

---

Checklist
477
478
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
480
contributions and scope? [Yes]
481
(b) Did you descrihe the limitations of your work? [Yes]
482
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
484
485
2. If you are including theoretical results...
486
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
487
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
490
481
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
482
493
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
494
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
496
497
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
499
(b) Did you mention the license of the assets? [N/A]
500
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
502
(d) Did you discuss whether and how consent was obtained from people whose data you're ro using/curating? [N/A]
503
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
505
508
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
508
509
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
510
511
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "Xv5xBVStvD", "reviewer": "Reviewer_SzXC", "summary": "The paper aims to improve contrastive learning for multi-object scenes. In addition to making single-object representations close when objects are similar, the paper seeks to encourage multi-object scenes that share similar objects to also be close in the representation space. However, a given number of objects can be composed into exponentially many possible scenes, making it challenging to learn scene representations in the Euclidean space. Therefore, the proposed method maps the scene representations to a hyperbolic space, which is better at handling combinatorial explosion, and proposes a hyperbolic loss. Experiments show that the hyperbolic loss leads to 1~2% improvement in downstream classification, detection, and segmentation tasks. It is also shown that the learned scene representation can be used to quantify label uncertainty and detect out-of-context objects without further training.", "questions": "Please see my main concerns listed in Weaknesses. Additionally, I have the following two minor questions:\n- I did not understand why the objects are the root nodes (rather than the scenes).\n- Is there an intuitive explanation why the norm of representation will be large when there are multiple objects?", "limitations": "Limitations and potential negative societal impact have been addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n    - SSL works well on images containing one dominant object, but has limitations on multi-object images. The paper tackles this limitation and is relevant to the community.\n    - The proposed hyperbolic loss is quite novel and well motivated. It also leads to an interesting property of the representation that a larger norm indicates more object types in the image.\n    - The paper is well written. It is easy for me to understand the high-level idea behind using the hyperbolic space although I am unfamiliar with the exact mathematical definitions.\n- Weaknesses\n    - Empirical evaluation seems insufficient. As mentioned in the Related Work section, quite a few models have been proposed to improve SSL on multi-object datasets, but none of them are included as a baseline. As an example, ORL[1] achieves 59% top-1 accuracy on ImageNet-1k when pretrained on COCO, while the proposed method only achieves ~55%.\n    - Comparison to MoCo-v2 seems unfair, because MoCo-v2 only receives the full image as input while the proposed method takes object regions.\n    - The definitions in L105-111 do not seem to form a tree. Consider an image $s$ with three objects $a$, $b$, and $c$, and a region $r$ covering objects $a$ and $b$. Then $a$, $r$, and $s$ form a cycle, so it is not a tree.\n\n[1] Unsupervised Object-Level Representation Learning from Scene Images. (https://arxiv.org/pdf/2106.11952.pdf)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TfOaOCTVgWM", "reviewer": "Reviewer_78go", "summary": "This paper proposed a hyperbolic contrastive objective function (HCL) to formulate the object-centric scene hierarchy for self-supervised learning on scene images. The main motivation of the paper is that the hyperbolic space is more suitable to embed the hierarchical structure than\n Euclidean space. Using the ground truth or Selective search to obtain objects, this paper extends another branch on MoCo to constrain the similarity of scene crops and object crops in the hyperbolic space. The paper shows superior performance on downstream visual tasks when compared with its baseline method.  HCL also shows interesting properties on label uncertainty quantification and out-of-context object detection.", "questions": "1. Train HCL for 800 epochs and compare HCL with ORL and other Selective Search-based methods.  This result will be more important to support the claim of hyperbolic loss.\n2. The paper claims in Line 213 that \"HCL would improve more on the dataset with more objects per image\", which is not sound enough. On OpenImages, the paper only compares HCL w/ and w/o hyperbolic loss when using the ground truth bounding boxes (which can not be acquired during self-supervised learning). The authors should provide the performance of HCL/${L_{hyp}}$ with SS to support the claim.\n3. Is the additional hyperbolic loss time-consuming?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper utilizes the hyperbolic space to formulate the scene-object relationship, which builds up another way to represent scene images (small norms represent objects and large norms represent scenes).\n2. The paper achieves better downstream tasks than its baseline.\n\nWeakness:\n1. The experimental results only contain the comparison with baseline. Several relevant papers [1,2], which focus on the problem of self-supervised learning on scene images and also use Selective Search to acquire objects, are not mentioned in the experimental section. \n2. The paper only trained HCL for 200 epochs, it is not clear whether the performance gain can be maintained when training for more epochs (800ep). \n\n\n[1] Xie, Jiahao, et al. \"Unsupervised object-level representation learning from scene images.\" Advances in Neural Information Processing Systems 34 (2021): 28864-28876.\n[2] Li, Zhaowen, et al. \"UniVIP: A Unified Framework for Self-Supervised Visual Pre-training.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "SwXPFbHFFd", "reviewer": "Reviewer_L2gv", "summary": "The authors address the problem of contrastive representation learning of objects and scenes simultaneously. To this end, they employ the Poincaré embedding to model a hierarchy from simple (single object) towards more complex (multi object, scenes) images.\nIn the experiment it is shown that HCL improves over MoCo-V2 [1]. The method is trained on COCO (118,000 images) or OpenImages (212,000 images) for 200 epochs.", "questions": "A more extensive comparison to other SSL methods would be necessary, especially to other methods that focus on object-level contrastive learning and/or that use object detection as additional information such as HCL.", "limitations": "Limitations are briefly addressed. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea of using hyperbolic geometry in conjunction with contrastive learning is interesting and promising.\n- Results indicate the proposed method works: Quantitative improvements as well as representation norms correlating with number of labels.\n\nWeaknesses:\n- Performance gains of using hyperbolic loss are small and improvements might be due to using additional information, i.e. detections, instead of only the images.\n- Relevant baselines are missing from Table 1. For detection for example [3, 4] achieve similar performance without relying on detection labels for pre-training.\n- The claimed performance improvement is only shown over a weak baseline (MoCo-V2 on a small dataset). Performance in general does not seem competitive with current SSL models. For example, MoCo-V2 [1] achieves an accuracy of 67.5% when trained on ImageNet-1K for 200 epochs (71.1% after 800 epochs), the more recent DINO [2] even 75.3%. HCL achieves only an accuracy of 58.5% despite using the additional information of where objects are in an image. (This is of course difficult to compare due to using a different pre-training dataset, but a proper comparison to other SSL models would be needed here).\n- The used datasets are small for SSL. To show that this method is useful in practice, I'd like to see some scores on a more competitive benchmarks. E.g. by using a dataset of similar size to ImageNet.\n- Compared to other contrastive methods, HCL requires supervision. While selective search is proposed as a way to avoid reliance on ground truth, it decreases performance. \n\nMinor remarks:\n- \"...treated using the same lens\": This contradicts the claims of the next sentences\n- Fig. 4 should indicate standard deviations.\n- The precise task on the VOC dataset is unclear since there are multiple. I assume multi-label classification on Pascal VOC 2011 is meant but this should be explicitly stated.\n- L189: MoCO-V2 should be trained using the hyperparameters from [1] and not the ones that work best for HCL (i.e. learning rate etc). MoCo-V2 is trained using a cosine scheduler which was apparently not being used here.\n\n\nThe paper presents an interesting idea of applying contrastive learning on multi-image scenes using poincare embeddings and promising first results. However, the experiments fail to convince me. \nThere is a large gap between this method's and state-of-the-art performance in self-supervised representation learning, which raises concerns regarding the utility of the method.\n\n[1] Chen et al. Improved Baselines with Momentum Contrastive Learning\n[2] Caron et. al. Emerging Properties in Self-Supervised Vision Transformers\n[3] Wang et al. Dense Contrastive Learning for Self-Supervised Visual Pre-Training \n[4] Xiao et al. Region Similarity Representation Learning ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject