Compositional Entailment Learning for Hyperbolic Vision-Language Models
Authors: Avik Pal
Max van Spengler
Guido Maria D'Amely di Melendugno
Alessandro Flaborea
Fabio Galasso
Pascal Mettes
Venue: ICLR 2025 Oral (2025)
Keywords: Vision-Language Models
Hyperbolic Geometry
Representation Learning
CLIP
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We explore the benefits brought in when using visual-semantic compositional hierarchies for learning hyperbolic representations through unsupervised contrastive training.

Abstract: Image-text representation learning forms a cornerstone in vision-language models, where pairs of images and textual descriptions are contrastively aligned in a shared embedding space. Since visual and textual concepts are naturally hierarchical, recent work has shown that hyperbolic space can serve as a high-potential manifold to learn vision-language representation with strong downstream performance. In this work, for the first time we show how to fully leverage the innate hierarchical nature of hyperbolic embeddings by looking beyond individual image-text pairs. We propose Compositional Entailment Learning for hyperbolic vision-language models. The idea is that an image is not only described by a sentence but is itself a composition of multiple object boxes, each with their own textual description. Such information can be obtained freely by extracting nouns from sentences and using openly available localized grounding models. We show how to hierarchically organize images, image boxes, and their textual descriptions through contrastive and entailment-based objectives. Empirical evaluation on a hyperbolic vision-language model trained with millions of image-text pairs shows that the proposed compositional learning approach outperforms conventional Euclidean CLIP learning, as well as recent hyperbolic alternatives, with better zero-shot and retrieval generalization and clearly stronger hierarchical performance.

Published as a conference paper at ICLR 2025
COMPOSITIONAL ENTAILMENT LEARNING FOR HYPERBOLIC VISION-LANGUAGE MODELS
Avik Pal'
Max van Spengler'
Guido Maria D' D'Amely di Melendugno
Alessandro Flaborea" 234
Fabio Galasso²
Pascal Mettes
University of Amsterdam
Sapienza University of Rome
3 ItalAl
Procederaii
ABSTRACT
Image-text representation learning forms cornerstone in vision-language models, where pairs of images and textual descriptions are contrastively aligned in a shared embedding space. Since visual and textual concepts are naturally hierarchical, recent work has shown that hyperbolic space can serve as settenatttiaal manifold to learn vision-language
representation with strong downstream performance, In this work, for the first time we show how to fully leverage the innate hierarchical nature of hyperbolic embeddings by looking beyond individual image-tex pairs. We propose Compositional Entailment Learning for hyperbolic vision- langungg models. The idea is that image not only described by a sentence but is itself a composition of multiple object boxes, each with their own textual description. Such information can be obtained freely by extracting nouns from sentences and using openly available localized grounding models. We show how to hierarchically organize images, image boxes, and their textual descriptions through contrastive and entailment- -bsed objectives. Empirical evaluation on hyperbolic vision- language model trained with millions of image-text pairs shows that the proposed compositional learning approach outperforms conventional Euclidean CLIP Iearn- ing, as well as recent hyperbolic alternatives, with better zero-shot and retrieval generalization and clearly stronger hierarchical performance. Code available at attps://githut. com / http//aalliikyyccliiiii
INTRODUCTION
modeling has witnessed rapid progress in recent years with innovative approaches such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) using extensive vision- laniuae data to train encoders for understanding visual and textual content simultaneously. Such encoders align visual scenes with textual descriptions in a shared high-dimensional Euclidean space, facilitating semantic understanding (Radford al., 2021). While effective, conventional vision-language models only take a holistic approach to image-text representation learning, neglecting the intrinsic hicrarchy and composition of elements within images. Indeed, visual scene is commonly composed of multiple objects interacting with one another to form precise context. See for example Fig. 1b with description: "Mineral water with fresh herbs in a glass carafe on a garden table". Individually, these objects provide limited semantic meaning. Only through the interactions between these do understand the specific context of both the scene and its parts, characterizing the single entities (cf. Fig. la). This object-scene hierarchy is analogous to parent-child connection in discrete tree where broader concepts are closer to the root while specific concepts reside deeper in the tree. These tree-like structures cannot be well represented in Fuclidean space due its polynomial volume growth (Matoušek, 1999), whereas hyperbolic geometry does accommodate the exponential growth of trees (Gromov, 1987), making it more suitable for representing hierarchies.
Recently, Desai et al. (2023) introduced MERU, a hyperbolic contrastive languaee model. MERU projects Euclidean embeddings from image and text encoders onto hyperbolic space and enforces inter-modal (text to image) partial ordering (Vendrov et al., 2016) using an entailment loss (Ganea al., 201 8a; Le et al., 2019) when optimizing encoder weights. Such hyperbolic image-text text alignment has demonstrated strong quantitative performance on zero-shot downstream tasks, well increased interpretability of the shared emhedding space. They, however, ignore the

---

Published as a conference paper at ICLR 2025
Entoilment
Image,
Apple cider cocatail
T
Imagge Local box, P
T
ootter with hoctir
honio cook
hur,
(b)
Figure 1: Compositional Entailment Learning for hyperbolic vision-language models. (a) same object appearing in different vision-language contexts (b) Visual-semantic ordering: T (whole image) and 7 (full caption) provide context to the more general Tooz (image local box) and T (text local box). (c) This specifie- specifie-general general ordering between (1, (1 ,,boz), (1, bbz), (I, Troo) enforced in hyperbolic space using entailment cones. The external angle of specific concept (?") is pushed be within the aperture threshold you of the general concept (Tboz).
intra-modal hierarchical compositions of image-text pairs. Indeed, there is hicrarchical semantics in language (Everaert et al., 2015), which has been leveraged to embed textual data in hyperbolic space (Dhingra al., 2018). In the vision domain, work by Ge et al. (2023) uses object-centric scene hierarchies to learn a hyperbolic space where visually similar objects are clustered near the origin and scenes consisting of them are descendants. Zhong et al. (2022) propose RegionCLIP that only learns regional representations using contrastive learning and knowledge distillation. These prior works beg the question of what strategy can be adapted compound the individual benefits of the modal hierarchy and the two intra-modal hierarchies to encompass scene and region level understanding.
To this end, introduce Hyperbolic Compositional CLIP (HyCoCLIP), contrastive learning method that accounts for compositional orders in both inter-modal and intra-moda settings in hyperbolic space. We approach the problem by using explicit hierarchies while training the encoders. This hierarchy is constituted of object crops (image boxes) within an image and corresponding nouns/phrases (text boxes) within the text as broader concepts of the whole image-text concept. We outline robust hierarchical learning objective by using both entire images and image boxes, as well as complete captions and text boxes. This strategy involves both inter-modal hierarchies, where text generally provides broader context than images, and intra-modi hierarchies, where consider the "boxes" more general than the complete image. In the hierarchical spatial representation, broader concepts embedded near the origin of the metric space, while more fine-grained concepts positioned towards the border, akin to tree graphs, see Fig. 1c.
We show that HyCoCLIP outperforms CLIP and MERU on zero-shot image classification and is competitive on zero-shot retrieval and object detection when trained on 20M pre-training dataset. Additionally, show that HyCoCLIP improves the baselines and that its representation space is more interpretable and hierarchically aligned. Our contributions are summarized follows: (1) We introduce HyCoCLIP for learning vision-language representations in shared hyperbolic space using scene compositions that are semantically and hierarchically aligned. (2) We propose Compositional Entailment Learning, where image-text compositions are optimized through hyperbolic contrastive and entailment cone losses. (3) We demonstrate empirically that HyCoCLIP more hierarchically aware and is highly competitive to existing vasion-language models.
hierarchical classification tasks compared to
2 HYPERBOLIC COMPOSITIONAL. CLIP HYCOCLIP
We propose compositional learning scheme that enforees the semantic alignment of latent represen- tations in the hyperbolic space, explicitly modeling intra- and inter-modal relationships of visual and language data by leveraging their joint hierarchical nature. Here, first provide short background
2

---

Published as a conference paper at ICLR 2025
of Eoperooke
Taxk Bax Reshebiing
(12))
Tead box ()
COCeon and
SpaCy
V7T-5-16
GLIP
Fall Thes Enthelling
Miterral withgrreh bertalin glass caradi surden tabl Tost (T)
BCC kannny
(..))
Figure 2: An overview of HyCoCLIP. Text and image boxes are extracted offline from image-text datasets (sides). Next, HyCoCLIP's encoder modules embed the images and texts, projecting the representations in the hyperbolic latent space. HyCoCLIP preserves the inter-modal and intra-modal relationships by accommodating broader/finer concepts close the center/border and by using entailment cones to give an interpretable structure to the learned latent space (cf. Fig. lc).
with the required hyperbolic functions to make such compositional learning possible. Afterward, we outline compositional encoding of image-text pairs.
2.1 BACKGROUND
Hyperbolic geometry is a non-Euclidean geometry characterized by a constant negative curvature. The resulting space has the desirable property that volumes of subsets can grow exponentially as a function of their radius, making it an ideal choice for learning representations of data with an inherent hierarchical tree-like structure (Sarkar, 2011; Nickel Kiela, 2017; Krioukov al., 2010). While several isometric models are used in literature for modeling hyperbolic space, we limit our background discussion to the Lorentz (or hyperboloid) model used in this work and refer to Cannon et al. (1997); Peng et al. (2022) for detailed information the other models.
The Lorentz model, denoted by TL", an n-dimensional manifold represented as the upper sheet of two-sheeted hyperboloid in (n 1)-dimensional Minkowski spacetime. For each vector p R=+1, the first dimension is taken as the time-axis, denoted Po. and the remaining n dimensions as the spatial-coordinates, denoted p RR, This model is described as
ppe
B=+1 (p.P)L = - - K
Po = 1/ >00
(1)
where -K E TR is the curvature the space and (1.-)1. is the Lorentzian inner product defined for p.q E _h
(p, q) = Pogo + (p. g).
(2)
with (., .E denoting the Euclidean inner product. The Lorentzian distance between two points in L the length of the shortest path (geodesic) connecting them, which can be computed as
di (p. = v1/K- cosh -1 ( x<p. q)c), P"
This metric induces the Lorentzian norm ||P|L| = (p. P)L. The tangent space TpL" is well-defined for all the points p L", and the exponential map represents the projecting map from the tangent space to the hyperboloid. Given a point v E I. the exponential map can be computed as
expr(v) = cosh ( sinh( (Msllvllll))
Vollv v.
(4)
Such a map can be used to move from Euclidean space to hyperbolic space by considering Euclidean vectors to be tangent vectors at the origin 0 = ( (/I/K, ........................ 0)" of the hyperbolic space and using expğ to project these onto the hyperholoid (Khrulkov et al., 2020).

---

Published as a conference paper at ICLR 2025
2.2 COMPOSITIONAL ENTAILMENT LEARNING
We strive to learn the hierarchical compositional relations of images, boxes, and textual descriptions. Our idea is based on the following observation: the content inside box of an image is hierarchically more general than the entire image. While counter-intuitive at first glance, Fig. 1b shows why this the case: the box shows an object and the entire image additionally shows the context in which the object occurs, making it a semantically more specific scenario. From a hyperbolic perspective. semantically general/broad concepts are embedded closer to the origin, while more fine-grained concepts are positioned towards the border, akin to tree graphs (cf. Fig. Ic).
In this work, we are given a dataset D [Ix, T&)] of K image-text pairs. Our goal is to train image and text encoders with shared embedding space to align the visual and semantic inputs. The method is summarized in Fig. 2. Let (/ T image-text pair obtained following the automated procedure detailed in Appendix A. We propose a Compositional Entailment Learning objective in hyperbolic space to optimize the hicrarchical compositions. Our approach consists of two parts, namely a compositional contrastive loss and compositional entailment loss which we discuss next.
) be the local box with a short description from an
Hierarchical Compositional Contrastive (hCC) learning. Image-text models commonly rely on contrastive objectives to align and distribute the multi-modal data. In our approach, we rely on hyperbolic embeddings to align visuals and text. Let fr(.) and fr()) denote arbitrary encoders for the image and text inputs respectively. And, let gr (T&) = expo (Tx( and gr (7x) ) expo (Tx))) denote the hyperbolic representation of image 45 and textual description Ti respectively. To compute the contrastive loss over image-text pairs in batch B, we take the negative Lorentzian distance as our similarity metric and formulate with the softmax, using temperature т, for batch of size (B) containing images (I) and text (T) as
-10g log exp(di(or(Ti) (T.)))))rT(TT)//
Lonn
(5)
Σ 2k=1k exp(dd (I)) 9r (Tk))/T)*
where negatives for an image are picked from the texts in the batch. Similarly, we can define the loss when picking negatives for a text from images in the batch as I. (T, 7). To extend such contrastive setup with our image-text compositions, we have to consider that due to the generalized information in a box, different images in a batch can have similar box-level information. To avoid unwanted negatives in a batch, we only contrast the box image with other entire images, and vice versa which have specifie information. This avoids negative alignment between image-box pairs and boxes from different images. The final hierarchical Compositional Contrastive (hCC) loss is formulated
AC boz 1 L0o
1 4
Léont (1,T) + L'ont (T, + Leone (1 (1 ,T) + Licne (h 1))
spocific-info contrast
gencral-info contrast
Hierarchical Compositional Entailment (hCE) learning. Ganea et al. (2018a) introduced hyper- bolic entailment cones that generalize the idea of partial order embeddings (Vendrov et al., 2016) by using the inherent hierarchical structure of the hyperbolic space. Entailment cones define region Ry for every possible point in the space such that all points p E R. are semantically linked to q as its child concepts. As such, points in R, are expected to contain specific information for the general concept Considering the Lorentz model L", the half-aperture of these conical regions (R,) formulated by Le al. (2019); Desai et al. (2023) as
2K
w(q) sin
where -k is the curvature of the space and constant K = 0.1 is set to limit values near the origin (see Ganea et al. (2018a)). The aperture inversely depends on the norm ||ğ]]. Inferring from this, a general concept with wider aperture would lie closer the origin. A specific concept would have narrower aperture and lie further from the origin in the hyperbolic space.
To learn partial orders in this space, specific concepts p must be pushed be within the aperture w(q).
This is done by penalizing encoders with the angular residual of outward point having an exterior angle d(p. q) as shown in Fig. Ic. This is formulated by Le et al. (2019); Desai al. (2023) as
4

---

Published as a conference paper at ICLR 2025
Aperture threshold
Lent (pm q) = max(0, @(p, (P..)) w())), where the exterior angle is given by,
(8)
Thox box
Po + (p. (s q)))
o(p q) cos-1
77
However, intuitively observing the entailment loss presented in Eq. 8 shows that this loss would push any outward point only towards the Rg region's ss border. Hence, we add a threshold to the half-aperture ((q) effectively making it flexible to accommodate p various spatial distances from q, see Fig. 3. We reformulate Eq. 8 with half-aperture threshold as
Figure 3: Aperture threshold scaling the aperture w to in- crease or decrease the width of the entailment cone.
eet (p, q) = max(0, o(p. q) mg)()))
(10)
Entailment cones enable us to enforce the hierarchical image-text relations given by the compositions. We formulate the Hicrarchical Compositional Entailment (hCE) loss by considering that images and textual descriptions are not identical, but that text precedes image, akin to Desai et al. (2023). We additionally consider the relation whole = box for both images and texts. Hence, the hCE loss would comprise both image-text inter-modality entailments and text-text, image-image intra-modality entailments
ACE(II I Ibozy Leme(Ibor Tboo) + Lene (I, + [I, (ooz) + Lene (T,T
inter-modality entailment
intra-modality entailment
(11)
In Fig. Ic visualize how the image-text compositions should be organized in hyperbolic composi- tional entailment.
Hierarchical Compositional (hC) learning. We aggregate the losses to form the overall hierarchi- cal Compositional (hC) loss for HyCoCLIP by taking a weighted sum of the two loss components:
hc hCC + y.CE.
(12)
In Appendix B, we detail all hyperparameters, thresholds, and further implementation details.
Computational complexity. Our approach enables to double the amount of visual and textual data to learn from. The training time scales linearly with the increase in training volume; for ViT-B/16. HyCoCLIP requires 73 hours of training. compared to 46 hours for MERU and 45 hours for CLIP. We note that our method inference maintains the same efficiency as CLIP and MERU and allows for scalable deployment.
EXPERIMENTS
3.1 BENCHMARK
Datasets We develop our models using grounded vision-language pairs. This could be human- annotated such as the Localized narratives subset of Open Images (Pont-Tuset et al., 2020) or the Flickr30K Entities dataset (Plummer et al., 2015). However, the sizes of these datasets are fairly limited considering the intensive efforts of manual labelling. Hence, we depend on auto- matic grounded information generated by pre-trained phrase grounding models. Several large-scale grounded inguage-vision datasets are publicly available by Li et al. (2023) and Peng et al. (2023). We train our models using the large-scale training corpus Grounded Image-Tex! Pairs (GRIT) dataset (Peng et al., 2023) containing 20.5 million grounded vision-language pairs which are processed from the even larger COYO-700X (Byeon et al., 2022) dataset. Information on the grounding procedure is added in Appendix A. We similarly use the grounding procedure the RedCaps dataset (Desai et al., 2021) originally used to train MERU. Additionally, we use the smaller-scale grounded Conceptual Captions 3M (CC3M) (Li al., 2023; Sharma et al., 201 8) dataset for hyperparameter search.
5

---

Published as a conference paper at ICLR 2025
Zero-shot image classification evaluation. denotes reproduced results from MERU. When using boxes during pre-training, numbers in squared brackets represent the additional box-pairs counts. For RedCaps, we find results for CLIP and MERU consistent with Desai et (2023) even when trained with a smaller batch size. Bold-face numbers are the best overall performances, while underlined figures are the best within the models sharing the same ViT backbone. Our method outperforms baselines on 15 out of the 16 evaluation datasets.
Table I:
General dansets
Flne grined dinsets
Misc
InageNNe
CIBAR
CIAAK
Calech 101
STL-10
Food 101
ExeSSAT
RESISCK
Countyyl1
Pess
DTD
w/ boses
samples (M)
RedCaps
CLIP
32.5
66.7
661
89.8
72.5
72.5
16.4
30.1
5.0
11,4/6.3
90.3
76.5
42.4
46
89.3
17.0
20.9
4.7
MERU
[6.4
29.9
76.4
39.9
26.6
62.3
89.5
25.4
R.9
13.0
30.5
27.6
4.3
HyCOCLIP
31.9
27.4
37.7
27.6
64.5
90.9
71.1
28.8
9.7
1.1
70.5
41.4
13.4
22.7
30.2
4.4
GRIT
CLIP
20.5
36.7
70.2
42.6
49.5
75.6
89.7
44.7
9.8
6.9
2.0
44.6
14.8
23.3
40.7
40.1
5.1
5/16
WI
MERU
MERU
CLIP
20.5 20.5
(355()
20.5
35.4
36.2
35.0
84.2
71.2
85.0
51.8
54.0
42.0
44.6
46.1
48.6
73.9
73.0
74.1
91.6
91.6
89.8
43.2
48.8
11.9
10.9
10.1
6.0
6.5
5.6
2.3
2.5
43.3
45.0
42.7
15.9
18.1
17.3
24.0
18.6
24.55
39.1
19.3
32.4
35.5
31.5
58.9
48
4.7
5.3
HyCsCL
20.5 (35.9)
41.7
85.0
53.6
52.5
75.
92.5
50.2
14.7
8.1
4.2
$3.0
20.5
23.3
33.8
45.7
5.
CLIP
78.9
92.4
23.4
MERI
78.6
31.7
5.6
1lyCoCLIP
(35.9]
35.8
55.6
Baseline Comparisons We compare HyCoCLIP against CLIP and MERU. We reproduce the CLIP and MERU models by training on the RedCaps dataset, reducing the batch size 768 fit on our available compute. We further retrain CLIP and MERU from scratch on the GRIT dataset. To fairly evaluate the impact of including image-text boxes, we also retrain CLIP and MERU when image-text boxes are included as additional pre-training samples.
3.2 DOWNSTREAM TASKS
To assess performance, we evaluate HyCoCLIP on several down- stream tasks. For zero-shot image classification, the label set is fitted to multiple prompts which are embedded using the text encoder and then averaged to obtain single embedding per label. The closest text embedding is picked from the collection as the predicted class for an image. We report the model's accuracy datasets. Similarly, we assess our method on zero-shot retrieval tasks determine if complex concepts, like scenes and captions, are accu- rately preserved in the representation space. Further, we evaluate the models on object detection task to analyze the regional understanding of HyCoCLIP. We also evaluate the hierarchical nature of HyCo- CLIP using multiple hierarchical metrics. Additionally, we assess the scene understanding capability of HyCoCLIP on two compositional benchmarks VL-Checklist (Zhao al., 2022) and VG Attribution (Yüksekgönül et al., 2023).
GRIT
RedCaps
% of camplees
16 image classification
0.00
0.25
0.50
0.75
1.00
sices boxfimage
Figure 4: Histogram of ra- tios of box area wrl the full image for GRIT and Red- Caps. The latter reports gen- erally larger crops, indicating lower precision in grounding concepts.
Zero-shot image classification From Table find repro- duced results for CLIP and MERU are fairly consistent with Desai al. (2023) even when trained with smaller batch size on RedCaps. On grounding RedCaps and filtering noise, we notice only 5,8 million image-text pairs are retained containing 6.3 million boxes. Training the baselines with these additional boxes also demonstrates reduced performance. Alternatively, the quantity of data is significantly higher for GRIT with 20.5 million image-text pairs and 35.9 million boxes in total. To differentiate between the datasets, we compare the ratio of the box area with the image area of all data points and plot histogram in Fig. 4. lower ratio signifies that phrases have more localized information in the image and constitute a better semantic parent for the whole image. This is evident for GRIT while boxes generated for RedCaps do not seem to localize well. We, therefore, recommend GRIT over RedCaps for grounded pre-training and thus report the results of models pre-trained on GRIT, We find that HyCoCIIIP

---

Published as a conference paper at ICLR 2025
Table 2: Zero-shot retrieval, detection, and hierarchical classification. HyCoCLIP performs best in image retrieval, and hierarchical classification and is competitive in text retrieval. Bold figures indicate the best results for each ViT backbone.
Text rerriecal
Imiage retrieval
Hierarchical metries
Vision enooder
COCO
Flickr
CDCD
Flickr
WordNet
Model
botes
R685
Rd910
R:655
R:6110
R:955
R:6110
R
R:6:10
T(R())
[.CA()
J(1)
Pu((1)
Ru(()
69.3
0.76
VIT S/16
MERU
CLIP
X
68.8
60.7
71.8
78.8
89.4
94.8
91.3
47.1
53.6
65.3
80.4
87.5
82.1
4.08
4.0%
2.39
0.26
0.76
0.83
0.83
0.83
0.83
MERU
72.7
$1.9
83.5
90.1
46.6
58.3
60.0
71.7
4.08
2.39
0.75
0.83
0.83
HyCoCLIP
69.5
79.5
89.1
93.9
55,2
66.6
81.5
88.1
3.55
2.17
0.79
0.86
0.85
VIT B/16
CLIP MERU
X
71.4
72.3
82.0
93.5
96.9
96.2
57.4
57.4
68.6
81.0
89.9
90.0
3,60
3,63
2.22
0.79
0.78
0.85
0.85
0.85
0.85
HyCoCLIP
72.0
82.0
92.6
95.4
58.4
69.3
84.9
90.3
3.17
2,05
0.81
0.87
0.87
performs best across wide range of datasets and settings when pre-training is done GRIT. We especially note the performance on ImageNet, where we obtain an accuracy of 45.8% compared to 40.1% % (MERU) and 40.6% (CLIP). Interestingly, adding image-text boxes CLIP and MERU training does not improve performance, despite nearly doubling the training samples.
Zero-shot retrieval For the retrieval task, the top-k image/text embeddings are picked from a collection for input text/image embedding based on the distance score (Eq. 3). We perform this task zero-shot on the COCO validation set (Lin et al., 2014) and the Flickr30K test set (Young et al., 2014; Karpathy & Fei-Fei, 2015). We show the retrieval results in Table 2. We find that our method performs slightly worse on Flickr text retrieval while demonstrating increased performance on image retrieval over CLIP and MERU. We also note a significant decrease in the performance of CLIP and MERU when adding local information. These results further highlight the need for our approach. Naively adding these boxes as additional samples is not effective because the boxes are often without broader context, and the text is highly generic compared to the whole images. Only by optimizing for their hicrarchical compositional nature as done in our approach is it possible to get better performance. Our method aims to obtain hierarchically aligned representation space, but this is not necessarily beneficial for the task of retrieval, where proximity of text and image embeddings is key. Regardless, our approach remains highly competitive.
Hierarchical Classification A characteristic feature of hyperbolic spaces is their ability to represent hierarchical structures present in data. We evaluate our models for this property on several hierarchical classification metrics (Kosmopoulos al., 2015) described in Appendix C. We the WordNet hierarchy (Miller. 1994) of the ImageNet class labels (Deng et al., 2009; Russakovsky et al., 2015) for the hierarchical classification task. The image classification setup is kept similar and the final scores are averaged over the validation set. Table 2 reports the results of HyCoCLIP and other baselines on these metrics. We observe a consistent improvement, confirming that the hierarchy of the class labels is better represented in its embedding space.
Zero-shot object detection We utilize pre-trained vision- language models to recognize proposed object regions. Specifically, evaluate the scenario where ground-truth bounding boxes from the COCO detection dataset are used as region proposals and predict the correct categories with a setup similar to image classification. We compare our method with RegionCLiP (Zhong et al., 2022) whose vision encoder (ResNet50x4) was trained on CC3M with a frozen text encoder (originally trained CLIP400M). We report the average precision (AP) on the 17 novel categories data split (Bansal al.. 2018). As shown in Table 3, HyCoCLIP outperforms the baselines, surpassing RegionCLIP on the novel categories. We believe this highlights a key advantage of our approach- its ability to leverage inherent hierarchies for more effective semantic concept alignment.
Table 3: Zero-shot object detec- tion with ground-truth boxes evalu- ated on COCO 17 novel categories split (Bansal et al., 2018). HyCoCLIP shows the best average precision (AP).
Model
AP
CLIP
51.2
MERU
55.8
RegionCLIP
65.2
HyCoCLIP
68.5
7

---

Published as a conference paper at ICLR 2025
Table 4: Ablation study on loss terms during pre-training of JyyCoLLI-ViiT-S//e on grounded CC3M evaluated for image classification and Flickr image/text retrieval. Lower accuracy/R@* @ indicates more influential loss term.
Table 5: Scene understanding evaluation. HyCoCLIP show better performance on both benchmarks indicating good object compre- hension of the visual scene.
Model
VL-CO
VG-A
classification
retrieval
CLIP
49.3
63.3
ImageN
Food- 101
Meant (6)
MERU HyCoCLIP
Image
59.8
68.4
Text
Pre-training losses
HyCoCLIP
16.7
10.6
22.3
56.2
46.4
Table 6: Ablation study on batch size shows saturation after 768.
bCC loss
Lume (1, T'
16.0
9.4
22.5
55.2
46.2
Lione! (T,
16.1
10.2
22.2
55.2
45.4
Licoel T)
13.8
8.7
19.3
49.1
42,9
Hatch size
ImageNet
LicoeeThoon 1
15.2
7.6
20.4
55.9
44.5
512
11.1
BCB
640
11.3
L.
14.9
9.9
21.8
54.3
46.1
768
12.2
Ligne (1box Traz)
16.1
9.3
22.3
54.8
45,4
896
12.2
Lieu (I, 1b0m)
16.1
10.1
21.5
55.0
45.6
1024
12,1
(I, (90))
16.3
9.2
22.3
55.6
45.1
1536
12.1
Scene Understanding Hyperbolic embeddings have previously demonstrated enhanced spatial awareness (lbrahimi et al., 2024). We expect HyCoCLIP to be provisioned with further localized object/noun information in both vision and language and improve upon such aspects. The setup for these benchmarks is the same, for given image the model has to pick between the correct caption and a hard negative caption. For more information on the methods used to generate the hard negative captions, we refer to Appendix E. From Table 5, performance for the VL-Checklist-Objee (VL-CO) (Zhao et al., 2022) benchmark in which object information in the captions is perturbed. HyCoCLIP improves considerably reaching 60% accuracy indicating good object comprehension of the visual scene. HyCoCLIP also performs well on VG-Attribution (VG-A) (Yüksekgönül al., 2023) reporting mean accuracy of 68.4% surpassing other methods. We refer to Appendix E. for further analysis.
that CLIP and MERU give near-random
these experiments
3.3 ABLATION STUDY
Pre-training loss terms We examine the impact of the terms in hCC (Eq. 6) and hCE (Eq. 11) losses by pre-training the model several times, each time turning off single loss term. We use the grounded CC3M dataset and train for 40k steps. Table 4 shows the results of this experiment. A lower accuracy and recall image classification and retrieval respectively, indicate higher influence of corresponding loss term. For hCC loss, we find that our hypothesis of contrasting the generalized
Images
2.0
mages
Image
Texts
Imago baxes
Textt
Texts
Text bosss
toxt boves
Textt boxes
2.0
00
312
055
(a)
(b)
(c)
Figure 5: Visualizing the learned hyperbolic space of HyCoCLIP lower dimensions using samples from GRIT. (a) distribution of embedding distances from the origin, HyCoCLIP embeds text data closer to the origin wrt the images and boxes samples with a smaller radius wrt full images/captions. On the right, (b) HoroPCA and (c) CO-SNE visualizations of the latent space in L.?
8

---

Published as a conference paper at ICLR 2025
information in boxes against entire images and text is indeed beneficial. For hCE loss, see that the terms entailing the image (I) are most influential.
Scaling w.r.t batch size We train our models using batch size 768 according available compute (Appendix B). To study the influence of this hyperparameter, we train our primary baseline MERU- ViT-S using CC3M for various batch sizes and report their zero-shot performance on ImageNet classification. Table 6 indicates
empirical benefits when working with larger batch sizes. In the contrastive setting, the number of positives grows linearly, and the number of negatives grows quadratically in batch. When using softmax, the ratio of positives negatives affects loss functions differently depending on the type of similarity metric that is being used. This can explain the difference in batch size behavior of our approach. The saturation of softmax loss with increasing batch size has been previously discussed by Zhai et al. (2023), and the entailment loss may also contribute this early saturation.
SOURCE IMAGE
ANALYZING THE HYPERBOLIC SPACE
Visualizing the learned hyperbolic space We visualize the learned hyperbolic space in lower dimensions to see if the image, text, and corresponding box embeddings are dis- tributed in proper semantic hierarchy. To this end, plot the distribution of the spatial norms of 128k random sam- ples of training data in histogram. Furthermore, HoroPCA (Chami et al., 2021) for reducing the dimension- ality for 200 image-text pairs along with their boxes. Lastly, we extract 50 principal components to suppress noise and use CO-SNE (Guo et al., 2022) to bring the embeddings the low-dimensional space.
Fig. 5a shows that the embedding distributions of texts and their corresponding boxes are well separated, while images and their box representations display similar norms. This spatial contraction in image embeddings arises from the con- vergence of contrastive loss within confined entailment cone, as noted by Ramasinghe et al. (2024). Furthermore, many image boxes are almost identical to the full image (cf. Fig. 4), making it challenging for the network to differentiate between them. Nonetheless, the bottom plot in Fig. 5a shows that the box embeddings distribute closer to the origin, thus displaying hierarchical ordering. From Fig. 5b and 5c, we observe the semantic separation in the two principal compo- nents of HoroPCA and in the 2D space formed with CO-SNE, indicating an apparent hierarchy between the components.
-a -eildgo Wf and r90
-a eaphrrggpph ofan inslan
1th
ORIGIN
Interpolating between points in hyperbolic space We in- terpolate the geodesic connecting an image (source) with an- other image (target) and also with the origin similar Desai et al. (2023), which have been visualized on the bottom-right of Fig. 6. This intuitively represents traversing between nodes in discrete tree. This is useful in visualizing the ancestors of any given image and qualitatively verifying the hierarchi- cal properties of the learned hyperbolic space. We do the shortest path traversal in the tangent space details of which are in Appendix G. We use grounded Flickr30K (Li et al., 2023) to generate the collection of representations of images, texts, and corresponding boxes. Fig. 6 shows the result of 100 points being interpolated between two randomly selected images from pexels s com as well to the origin. We observe that HyCoCLIP can fetch representations of both data modes in very rational hicrarchy. More interpolation examples are added in Appendix I where we also compare interpolation in MERU and CL.P representation space.
TARGET
Figure Interpolation between points. Multimodal retrieval results when moving from (top) an image to (leff) another image (right) the ori- gin, as depicted in the (bottom-righl) circle.

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Vision-language models Currently expanding at a rapid pace, this topic has been in focus for multi- ple decades with initial works in image retrieval (Mori et al., 1999), semantic segmentation (Barnard et al., 2003), and object classification (Wang et al., 2009) leveraging natural language descriptors for computer vision. Later works (He & Peng, 2017) utilized more expressive representations from multi-modal neural network encoders. The advent of transformers (Vaswani et al., 2017) and vision transformers (Dosovitskiy et al., 2021) helped construct a highly semantic embedding space for texts and images, respectively. Recent works (Gan et al., 2022) have explored creating a shared embedding space by leveraging various pre-training strategies to integrate text and image information. Many approaches use contrastive learning as a core method, like CLIP (Radford et al., 202 and ALIGN (Jia et al., 2021). More recently, MERU (Desai et al., 2023) combines entailment learn- ing (Ganea al., 2018a; Le et al., 2019) with the CLIP approach to learn embeddings in hyperbolic space capturing latent visual-semantic hierarchies. We extend this to include image patches and caption parts, enforcing an ordering that reflects the hierarchy shared by both modalities.
Learning in hyperbolic space Hyperbolic space for representation learning has desirable properties for data with an inherent hierarchical tree-like structure (Nickel & Kicla, 2017: Chamberlain et al., 2017). When generating embeddings in hyperbolic space from such data, its innate hierarchical structure can be retained with minimal distortion. As a result, hyperbolic deep learning has rapidly gained traction (Peng et al., 2022; Mettes et al., 2023). Recent works have developed methods for building neural networks that operate in hyperbolic space (Ganea et al., 201 8b; Shimizu et al., 2021) and corresponding optimization algorithms (Bécigneul & Ganea, 2019; Bonnabel, 2013).
This led to the use of hyperbolic models in many different modalities such as graphs (Liu et al., 2019; Franco et al., 2023; Flaborea et al., 2024), text (Dhingra et al., 2018; Tifrea et al., 2019), images (van Spengler et al., 2023; Atigh et al., 2022; Franco et al., 2024), videos (Long et al., 2020), time series (Flaborea al., 2023), etc. Other recent work has focused on combining embedding spaces of different modalities (Liu et al., 2020; Desai et al., 2023). Mandica et al. (2024) also explore scaling hyperbolic embeddings in vision-language models billions of parameters while maintaining stability and meaningful uncertainty estimates. Our work similarly learns multimodal representations in hyperbolic space benefit from its inductive hierarchical bias.
Hierarchies in vision and language Vendrov et al. (2016) use visual-semantic hierarchy over words, sentences, and images to learn representations in a supervised fashion. They consider hypernym-hyponym relations in language construct hierarchy. This concept has been used in hypernymy detection tasks (Nguyen et al., 2017; Vulic & Mrksic, 2018). Hierarchies formed by constituency-bass parse trees have been used to learn embeddings in hyperbolic space by Dhingra et al. (2018). In vision, several works sought to connect scenes to objects and parts of objects within the scene. Early works have used such information for pose estimation, image segmentation, and object and contour detection (Bourdev & Malik, 2009; Arbeláez et al., 2011). Recently, un-/self- supervised methods have been used for representation learning leveraging hierarchical segmentation of an image by Zhang & Maire (2020) and object-scene hypernymy by Xie et al. (2021); Ge et al. (2023). We combine hypernymy relations of vision and language.
CONCLUSION
The idea of this work is to use object compositions within scene and its description, along with the visual-semantic ordering between image and text to learn hyperbolic representations that are semantically and hierarchically aligned. Our proposed HyCoCLIP improves over standard CLIP and its recent hyperbolic extension MERU in zero-shot classification. Moreover, our approach has increased scene understanding and better hierarchical structuring. Further, we qualitatively analyze the space by visualizing representations and through point-to-point interpolation which substantiates HyCoCLIP's ability to embed multi-modal hierarchies in a shared space. The method has certain limitations, with key challenge being the need to generate bounding box information from image- caption pairs during training. This increases the volume of visual and textual data processed by HyCoCLIP, though it still preserves scalability during inference. Additionally, while our hierarchical training strategy improves interpretability by separating images and texts into distinct regions in the embedding space, it may not be optimal for tasks like large-scale retrieval.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We sincerely acknowledge the financial support given to Avik Pal by the Amsterdam ELLIS unit for the research and presentation of this paper the conference. Max van Spengler acknowledges the University of Amsterdam Data Science Centre for financial support. We also extend our gratitude to SURF (www surf, n1) for granting compute resources from the National Supercomputer Snellius. We are grateful to Panasonic for partially supporting this work. We acknowledge the financial support from the PNRR MUR project PE0000013-FAS and from Sapienza grant RG1231 18833FFFAAS (CENTS). Thanks CINECA and the ISCRA initiative for nigh-performance computing resources and support.
REFERENCES
Morris Alper and Hadar Averbuch-Elor. Emergent visual-semantic hierarchies in image-text representations. CoRR, abs/2407.08521, 2024. doi: 10.48500ARRXV.2400.05855 URL https://dii.or///12 wtt..////....../.ra........o www.7.i..40..... 2407.0852..... 08521.
Pablo Arbeláez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5):898 996, 2011. doi: 10. 109/TPAMM.2010.161.
Mina Ghadimi Atigh, Julian Schoep, Erman Acar, Nanne van Noord, and Pascal Mettes. Hyperbolic image segmentation. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 4443-4452, IEEE, 2022. doi: 10. 1109/CVPR52688.22222 202.2088..022004444... URL https: /tt.doo...gg/o//ooooooooooooo./. 1109/110000 CVPR52 688 2022. 00441.
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8- 14, 2018, Proceedings, Part volume 1 11205 of Lecture Notes Computer Science, pp. Springer, 2018 doi: 978- 3- 030 0- 5_224.......
24. URL https: http://i.oo....0000./.......1
Kobus Barnard, Pinar Duygulu, David A. Forsyth, Nando de Freitas, David M. Blei, and Michael I. Jordan. Matching words and pictures. J. Mach. Learn. Res., 3:1107-1135, 2003. URL http:
/tjmmr/org//////eetreee/.. ptsp/s/pperss/// /ba (barnard03a html.
Gary Bécigneul and Octavian-Eugen Ganea. Riemannian adaptive optimization methods. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https::/ /owpr/vieeeeeeeeeeeeemme net forum?id= rleiqi09K: 09K7.
Silvère Bonnabel. Stochastic gradient descent riemannian manifolds. IEEE Trans. Autom, Control., 58(9):2217-222, 2013. doi: 10. N0.1109/TAC.201322556111 URL htt.s://// /ooi org/ 10. 1109/ TAC 2013.225455
Lubomir Bourdev and Jitendra Malik. Poselets: Body part detectors trained using 3d human pose annotations. In 2009 IEEE 12th International Conference on Computer Vision, pp. 3865-1372. 2009. doi: 10. 111100//CCC.22009555559555
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset.
coyo-dataset, 2022.
James W Cannon, William Floyd, Richard Kenyon, Walter R Parry, et al. Hyperbolic geometry. Flavors of geometry, 31(59-115)22, 1997.
Benjamin Paul Chamberlain, James R. Clough, and Mare Peter Deisenroth. Neural embeddings of graphs in hyperbolic space. CoRR, abs/1705. 10359, 2017. URL httpp:/a //tp://axxivvva//ii////i/iii/aia org abs/ 1705. 1 9.359.
11

---

Published as a conference paper at ICLR 2025
Ines Chami, Albert Gu, Dat Nguyen, and Christopher Ré. Horopea: Hyperbolic dimensionality reduction via horospherical projections. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 1419-1429. PMIR, 2021. URL hutttp:/pproee
ngs ml r. www.prrsssllll /vsss//vvv33// chami i 21a html.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision ICCV 2021, Mon- treal, QC, Canada, October 10-17, 2021, pp. 9620 9629. IEEE, 2021. doi: 10.1 101109//C248922.. 2021.00950. URL https: doi. org/ 10 1109 ICCV4
00950.
Ofer Dekel, Joseph Keshet, and Yoram Singer. Large margin hicrarchical classification. In Carla Brodley (ed.), Machine Learning. Proceedings of the Twenty-first International Conference (ICML 2004), Banff. Alberta, Canada, July 4-8, 2004, volume 69 of ACM International Conference Proceeding Series. ACM, 2004, doi: 01114//1015330.11537777 URL https: /tt///ooooo//// org/ 10. 01114//111330.1100 10113333.0.117.4.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255 IEEE URL /itttp::/doi.oogg
Computer Society, 2009. doi: 10. 10.1 (000//CVP.220995206600 1109/ 2009. 5206848.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redeaps: Web-curated image- text data created by the people, for the people. In Joaquin Vanschoren and Sai-Kit Yeung (eds.). Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https: http::::///atttttsssseeeee hash/e00daa33665.aaeommmm org.aaac.....aacccoooooooooaa... att.:oaaar.2..ee--asaaa-------e- roundl. html.
Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakr- ishna Vedantam. Hyperbolic image-text representations. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaäi, USA, vol- ume 202 of Proceedings of Machine Learning Research, pp. https: www.eeeinggg....oooo...p........ mlr. wol..ppres//eee v2222/ / desai 23a html.
PMLR, 2023. URL
Bhuwan Dhingra, Christopher J. Shallue, Mohammad Norouzi, Andrew M. Dai, and George E. Dahl. Embedding text in hyperbolic spaces. In Goran Glavas, Swapna Somasundaran, Martin Riedl, and Eduard H. Hovy (eds.). Proceedings of the Twelfth Workshop Gruph-Based Methods for Natural Language Processing, TexIGruphs @ NAACL-HHLT 2018, New Orleans, Louisiana, USA, June 2018. pp. 59-69. Association for Computational Linguistics, 2018. doi: N..11865////11 /t////11-777. 1708. URL https:://///.. / doii wtt..o/o.......//11111.1...
10.
-1008
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:/ https::/ /wwp///veeeeveieeeeeeeeee net / forum?id Yicb YicbFdNTTy.
Martin B.H. Everaert, Marinus A.C. Huybregts. Noam Chomsky, Robert C. Berwick, and Johan J. Bolhuis. Structures, not strings: Linguistics as part of the cognitive sciences. Trends in Cog- nitive Sciences, (9(12):729-743 2015. ISSN 1364-6613. doi: https://doi.or//1..0111//
2015.09.008 URL battps://wwe www. www.ssccieeeeeeeeeee com/ http:/ciencc/aaiice/ppll $136466131500
Alessandro Flaborea, Bardh Prenkaj, Bharti Munjal, Marco Aurelio Sterpa, Dario Aragona, Luca Podo, and Fabio Galasso. Are certain it's anomalous? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2897-2907, June 2023.
12

---

Published as a conference paper at ICLR 2025
Alessandro Flaborea, Guido Maria D' A'Amly di Melendugno, Stefano D' D'Arigo, Marco Aurelio Sterpa, Alessio Sampieri, and Fabio Galasso. Contracting skeletal kinematics for human-related video anomaly detection. Pattem Recognition, 156:1108/ 2024. ISSN 0031-3203. doi:
https///oi..rr//10. pttp:///oo....////oo..oooo.ooo 10817. URL https::///// www.www............ sc encedi rect com/ sccience/artticlleel////////
Luca Franco, Paolo Mandica, Bharti Munjal, and Fabio Galasso. Hyperbolic self-paced learning for sulf-super skeleton-basee action representations. In The Eleventh International Confer- ence on Learning Representations, 2023. URL httpt:t/////... http://pppneeviiie net/ forum?id- - 3B165RPKS3J.
Luca Franco, Paolo Mandica, Konstantinos Kallidromitis, Devin Guillory, Yu-Teng Li. Trevor Darrell, and Fabio Galasso. Hyperbolic active learning for semantic segmentation under domain shift. In Proceedings of the 4 International Conference on Machine Learning, ICML'24. JMgR.org. 2024.
Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, and Jianfeng Gao. Vision-language pre-training: Basics, recent advances, and future trends. Found. Trends Comput. Graph. Vis., doi: 10.1561/0000001105. URL https: http:/oo..oo/ wtg.o..../...1... 1561/
14(3-4):133-352 2022. 0600000105.
Octavian- Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm. Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1632- 1641. PMLR, 2018a. URL http: http:::/ppocceeeee pttp/edecceeenssseeeeeeep mlr.. war..presss... / vv00 ggggaaaaalllaa 8a html.
Octavian-Eugee Eugen Ganea, Gary Béeigneul, and Thomas Hofmann. Hyperbolic neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp- 5350-5360, 2018b. URL https: : htttps:/prrceedi ner.ri newrips.............. wtcppp.ccc.........o..oee. sscppppr//// 2018 1/////haahhh hww.ahhh/eeee dbb.badcabaaaaabbbo 9d0 9 Obea 1 42 Abstract.htmi
songwei Ge, Shlok Mishra, Simon Kornblith, Chun-Liang Li, and David Jacobs. Hyperbolic contrastive learning for visual representations beyond objects. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 1840-6849, IEEE, 2023. doi: 10.1 1099//VPR52799.223.06666 URL /tattss://doi.. http://oo.oooo orr/ 10. 19..1109/00 /Bt//VVR552222 2023 60661.
Mikhael Gromov. Hyperbolic groups. In Essays in group theory, pp. 75-263. Springer, 1987.
Yunhui Guo, Haoran Guo, and Stella X. Yu. CO-SNE: dimensionality reduction and visualization for hyperbolic data. In JEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 11-20. IEEE, 2022. doi: 10.1 109CC 2022.00011. URL https:::p..... / /oi. org / 10.
688 2022. 00011.
Xiangteng He and Yuxin Peng. Fine- Fine-grained image classification via combining vision and language. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 7332-7340 IEEE Computer Society, 2017. doi: 10.1 10.1109/CVPR. 2017.775. URL https::/p///......... do .wt.ooo....../o.oooo.oooooooo.o. 1 2111110///PPP 2017. 7 775.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industrial- strength natural language processing in python. 2020. doi: 10.5281/7enodee 1212303.
Sarah Ibrahimi, Mina Ghadimi Atigh, Nanne van Noord, Pascal Mettes, and Marcel Worring. Intriguing properties of hyperbolic embeddings in lanion-language models. Trans. Mach. Learn. Res., 2024, 2024. URL https: openreview. net for d f 14g..
Chao Jia Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham. Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- Language representation learning with noisy text supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th
13

---

Published as a conference paper at ICLR 2025
International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 4904 4916. PMLR, 2021. URL attp::/prrocedliin
mlr. wwwppppesssssseeee http//iaalllblhhheea
Andrej Karpathy and Li Fei-Fei, Deep visual-semantic alignments for generating image descriptions. In IEEE Conference on Computer Vision and Patlern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 3128-3137. IEEE Computer Society, 2015. doi: 10. 10.1...0..... 7298932. URL https::ps/.......... /oi. org 10 www.9/1119/.... 72989922999..
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 6418-6428, 2020.
Aris Kosmopoulos, Ioannis Partalas, Éric Gaussier, Georgios Paliouras, and Ion Androutsopoulos. Evaluation measures for hierarchical classification: unified view and novel approaches. Data Min. Knowl. Discow, 29(3):820-865, 2015. doi: 10. 1007//11618-014400
URL https:
Dmitri Krioukov, Fragkiskos Papadopoulos, Maksim Kitsak, Amin Vahdat, and Marián Boguñá. Hyperbolic geometry of complex networks. Phys. Rev. E, 82:036106, Sep 2010. doi: 10.1103/ PhysRcvE.82.0366106 URL https://tinns aps. org. /tlddooi/111111 1103 101111///Pyyee Phw.//hyyyseee..eeee 82. 036106.
Matt Le, Stephen Roller, Laetitia Papaxanthos, Douwe Kiela, and Maximilian Nickel. Inferring concept hierarchies from text corpora via hyperbolic embeddings. In Anna Korhonen, David R. Traum, and Lluis Marquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019. Volume 1: Long Pupers, pp. 3231-3241. Association for Computational Linguistics, 2019. doi: 10.18653/V1/ P19-1313. URL https:ttpt..... : doii. org/ 1 . 113653/vl/// 1- 1313.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Veng-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded sunguage-image image pre-training. In JEEE/CVF Conference on Computer Vision and Pattern Recog- nition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10955- 10965. IEEE, 2022. doi: 10.1 109/CV 202220869222220069 URL https: /tle://ddooe dooi org 10. 11 00111199//VVV2250 2022. 01069.
Yuheng Li. Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: open-set grounded text-to-image generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 22511 2251112 IEEE, 2023, doi: 10.1 https: doi. . .or.orgggoo /10. 10. 1109 CVWR527222222 2023 02156.
URL
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr
Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pp. 740-750755 Springer, 2014. doi: 10.1007/ 978-3-319-1066 1',00 18. URL https:/ //oii/o..///// org. /10 1007 978 31 10602-1 1_ 48.
Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. In Hanna M. Wallach. Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buec Buc, Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neurul Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8228-8239, 2019. URL https:tt////// /ttpc//rcceeeeeeee .gw..ng....r.e... newrips.e..... cce..cc/ppe ccpppppr/////pcc/pcccccccp...epp 2019 2019/hashl 9/hash///eeeeeaaaa 131003333dd3337
77 6a 3badaee Abst html.
Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong Maong-WWhh Ngo, Tat-Seng Chua, and Yu-Gang Jiang. Hyperbolic visual embedding learning for zero-shot recognition. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9270-9278. Computer Vision Foundation/ / IEFE, 2020. doi: 10.1 w0w.///V22200000000.....
14

---

Published as a conference paper at ICLR 2025
URL https://openaceeee htt.s//etteeceeeeotettttttttteee htt.o//c.../ccooeot....///////// syperbolic_Visuall Leeuuali_beddiinggggggaa Lembeedinn_Laainnnnnnnnnoo for Zero- Shot wtwhhhoteetotiiiiioooooo CVPR_ Satt0_aper.hheee
Teng Long, Pascal Mettes, Heng Tao Shen, and Cees G. M. Snoek. Searching for actions the hyperbole. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CriiionnnC (CVPR), June 2020.
Ily Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: oppnrevie.. net/ met/fose
9Scxx.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https::/ wwtp:/eeeeeeteteeeeeeeeeeeteetet wtt.rrrrmmmiiiigo.iicco....
Paolo Mandica, Luca Franco, Konstantinos Kallidromitis, Suzanne Petryk, and Fabio Galasso. Hyperbolic learning with multimodal large language models. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2024.
Jiří Matoušek. On embedding trees into uniformly convex banach spaces. Israel Journal of Mathe- matics, 11((1):221-377, 1999.
Pascal Mettes, Mina Ghadimi Atigh, Martin Keller-Ressel, Jeffrey Gu, and Serena Yeung. Hyperbolic deep learning in computer vision: survey. CoRR, abs/2305.0661 1, 2023. doi: A0.48550/ARXIV. 2305.06611, 1. URL https::///// /ttp:s:///oiii/ /tw.///ii.//////////// wrg/ 48550/4555isi 2305.06611.
George A. Miller. WordNet: A lexical database for English. In Human Language Technology: Proceedings of Workshop held al Plainsboro, New Jersey, March 8-11, 1994, 1994. URL
https: hatp:s//aallatthlogyy..
oww.////rg///9991100 H94-1194111111
Yasuhide Mori, Hironobu Takahashi, and Ryu ichi Oka. Image-to-worte transformation based on divid- ing and vector quantizing images with words. 1999. URL htt.h:tps::/aaa semant iescholl
ID: 18574318.
Kim Anh Nguyen, Maximilian Köper, Sabine Schulte im Walde, and Ngoc Thang Vu. Hierarchical embeddings for hypernymy detection and directionality. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.). Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 233- 243. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D11--0022 URL
Maximilian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. (338-6347, 2017. URL wtw.t:tppt///tc http:::://////ceeeeeeeeeeee neuri nwwrips.ccmmmmm ps. cc/ html.
Wei Peng, Tuomas Varanka, AbdeIrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyperbolic deep neural networks: survey, JEEE Trans. Pattern Anal, Mach. Intell., 44(12):10023-100444 2022. doi: 10.1 10.1199/PAMM22201.1333977 URL https: 1ttp://////.i...o0110100///1//// 2021. 3136921.
Zhiliang Peng, Wenhui Wang, Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. CoRR, abs/2306. 14824, 2023. doi: 1044850/ARXIV.230611 14824. URL https: /ww..iiooooo 0000///..455//axi... 2306 14824.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting segion-to-phrase correspondences for richer image-to-sentence models. In 2015 IEEE International Conference on Computer Vision, ICCV
15

---

Published as a conference paper at ICLR 2025
2015, Santiago, Chile, December 7-13, 2015, pp. 2641-2649. IEEE Computer Society, 2015. doi: N0.1109/CCCV220153303. URL https: www.tttpss/pooo/toeteeeettteeeee 11t.:/..1111/C//11 2015. 303.
Jordi Pont-Tuset, Jasper R. R. Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V. volume 12350 of Lecture Notes in Computer Science, pp. 647-664. Springer, 2020. doi: 10. URL https: /ttp::://do..e org 10 1007/ 9776//9/788 8- 5850-5555887_330
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-87888 PMLR, 2021. URL http: / http:://prooeeee pttpppeeeceeggneee/ee .mr.. hil...pres//v3/ radford21a html.
Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, and Ajanthan Thalaiyasingam. Accept the modality gap: An exploration in the hyperbolic space, Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 27263-27272 June 2024.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bemstein, Alexander C. Berg, and Li Fei-Fei. Ima- genet large scale visual recognition challenge. Ini. J. Comput. Vis., 1 115(3))21 2015. doi: 10. 1007/S1 263-015--8888
URL https: : //tp:///..o/////oo http://oi..oog///// www.org/1100 007 12
Rik Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane. Mare J. van Kreveld and Bettina Speckmann (eds.), Graph Drawing 19th International Symposium, GD 2011, Eind- hoven, The Netherlands, September 21-23, 201 Revised Selected Papers, volume 7034 of Lecture Notes in Computer Science, pp. 366-366. Springer, 2011. doi: 10. 1010000997---722-220 URL https:/ /to..ioiooooooo org 1007 / 8 000-6662222-------
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao (eds.). Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 2556-2565, Association for Computational Linguistics, 2018. doi: 10.1863/V1/P18--222 URL. https: / //t/aaaaantttt wwwhhhhooyyy.oooooooo wtw/oorg/.... 1100//111111111111
Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. Hyperbolic neural networks+++ In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: : wpwpe/iiee.eeeeeeee/ooee net /ast/foo for//oormmm///
Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL 1eee:t/ooomnddd.aa0000011 denniidd-EESbbUbbb net/forund d=55ke5r3AqR7.
Alexandru Tifrea, Gary Bécigneul, and Octavian-Euun Eugin Ganea. Poincare glove: Hyperbolic word embeddings. In 7th International Conference Learning Representations, ICLR 2019, New
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 10347-10357. PMLR, 2021. URL http: httt:::/proceedings mml.. press/v139/tow//o. htmml
Max van Spengler, Erwin Berkhout, and Pascal Mettes. Poincaré resnet. In /EEE/CVE International Conference on Computer Viston. ICCV 2023, Paris, France, October 1-6, 2023, pp. 5396-5405 IEEE, 2023. doi: 10.110000CCV5110000222.00 TCCV51076 2023. 00499.
URL https: pestts::/dol..
16

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwamathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. URL https: /ttpc//eecopppcpceccccccccccccee neurips pttppps.ppp.pppppppeee//e//e 2017/hash/ hash/ 3f5ee2435 2esee223347deeee 3ee.555ee99ee.. htt.eeeeeeedoo51114e44433.58
Abstract html.
Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: //ap::///axiii org/ abs C6361.
Ivan Vulic and Nikola Mrksic. Specialising word vectors for lexical entailment. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-
HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Long Papers), pp. 1134- 1145. Association for Computational Linguistics, 2018. doi: (0.18653/V1/118-11000 URL
/attps://dole org /10. www.1116661/// /tt////////n///////
Josiah Wang. Katja Markert, and Mark Everingham. Learning models for object recognition from natural language descriptions. In Andrea Cavallaro, Simon Prince, and Daniel C. Alexander (eds.), British Machine Vision Conference, BMVC 2009, London, UK, September 7-10, 2009. Proceedings, pp. 1-11. British Machine Vision Association, 2009. doi: 10.5244/C.23.2. URL https://doi. ottp:00.rgg11... 5244/C.23.2
Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Unsu- pervised object-level representation learning from scene images. In Marc' Aurelio Ran- zato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.). Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 28864-28876, 2021. URL https: /ptpceeeooocceeeeeeeeeeee neurips www.ripp.ccpppprr//222. cctppaper/222//aae.ee.eee E1b6f2857 fww.54ff6664444 4224444dd33c700 10a.1 Abst html.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Trans, Assoc. Comput. Linguistics, 2:67-78, 2014. doi:
(002/TACLAA))001 URL https:/ //doii.
org/10. /tclell/accaa///e/ a 00166.
Mert Yükkekgönüül, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: http//ppnrrvvveeeeeeeeeeeeeeeeee wwt.:ntttt/dddddd.ttttt
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 11941-11952. IEEE, 2023, doi: 10. 11111//CCCV11100020.000001 10. URL https: /ttp::////..ooo.........////1111 200.551110..222222....222 2023. 01100.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Uni- fying localization and vasion-language understanding. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems 35: Annual Conference
Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: http://aapppp.ee/ee/ nips. cc pwpprrapppp_p/eee wiless/aper//22// hhahh/ 0437041976004221ee2e22333 ottp://2e2e22eeaaaaaaaate----aae
html.
Xiao Zhang and Michael Maire. Self Self-super visual representation learning from hierarchical grouping. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 62,
17

---

Published as a conference paper at ICLR 2025
2020, virtual, 2020. URL / www.:us..eeniiii..ccccc.c
http:////...ceeeeoo..oooooo///// cctppp//220c//pp////20////0 / 4bb -ttprr22448.22eebbbaaaat.ttttte. moostract.htm..
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations, CoRR, abs/2207.0022 2022. doi: 10.48550/ARXV..227..000 URL https:
//doi. .0001.02200 org/ 48550/arXiv 2207 00221.
Yiwu Zhong. Jianwei Yang. Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionclip: Region-based language-image pretraining. In JEEE/CVE Conference on Computer Vision and Pattern Recog- nition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 16772-16782. IEEE, 2022. doi: LLL.L1/CCVPR228822222200000 2.022.0622
URL https: : / /ooi org.orrg/iee 1100//.//5////
18

---

Published as a conference paper at ICLR 2025
A GENERATING BOX INFORMATION
We derive box information using an image grounding pipeline similar to Peng et al. (2023). Given an image-caption pair, noun entities are initially extracted from the caption into a list using spaCy (Honnibal et al., 2020). To minimize noise, we remove abstract nouns such as {life, humor, love, ... from the list. We then predict the bounding boxes of the extracted entities within the image using the pre-trained grounding model GLIP (Li et al., 2022; Zhang et al., 2022). We exclude boxes with sizes lower than x 32. We also threshold the predictions at least 0.65 CLIP confidence score for the generated bounding box with the corresponding noun entity. Image-caption pairs for which no boxes could be generated or retained while filtering, are dropped. Further, referring expressions for noun chunks taken from the dependency tree of the caption using spaCy, also included text boxes. This increases the robustness the stem towards linguistic complexities. A few samples from the GRIT dataset visualized in Fig.
B IMPLEMENTATION DETAILS
Model architecture We use similar setup Desai et al. (2023), where the language encoder is the same one used by the original CLIP (Radford al., 2021) consisting of a 12-layer Transformer architecture (Vaswani et al., 2017) with a width of 512 dimensions. The maximum input token size set to 77 with vocab size of 49408. For the vision encoder, we use the small and base Vision Transformer (Dosovitskiy et al., 2021; Chen et al., 2021: Touvron al., 2021) backbone using a patch size of 16. The images are resized using border padding and random cropping (with scale [0.5, I.0) to 224 224, which results in an input sequence size of 196. A fixed set of 2- sine-cosine position embeddings included in the input sequence to instill positional inductive bias.
Table Training HyCo- CLIP various settings of k on grounded CC3M dataset. dne denotes did not converge.
K
ImageNet
Fixed param.
1.0
15.2
0.8
dnc
0.6
14.7
0.3
15.1
0.1
16,0
Learnable puram.
Initializing Lorentz model and Loss We train HyCOCLIP with a fixed curvature value of the Lorentz model on the grounded CC3M dataset for 40k steps. To find the optimal setting, we additionally
1.0
16.7
People siting by the pool for a Virginia wedding recaption
Tela Golden retriever against deocr abed Chrostmes free in the room
Qucumbar a glass of water
For Cate (Wff), her aeperience 2t Rio her Theartrokkon.
Young dactor with patient proporing on xrxxaayy ray
Aibertaa Cornen stogrert
Walreas
banquette: dishes the table
put
pets and their people
cranes
Gemman shophord sitting. isclated
against an Isolated background
Figure 7: Samples from the GRIT dataset. The GRIT dataset (Peng et al., 2023) contains 20.5 million grounded vision-language pairs of which 10 random samples visualized in this illustration. The grounded information includes noun terms or their referring expressions (highlighted in blue) and the corresponding bounding boxes (in red within the image).
19

---

Published as a conference paper at ICLR 2025
train by keeping the curvature learnable parameter with an initial value of 1.0 and clamped within [0.1, 10.0). As shown in Table 7, keeping the curvature a learnable parameter yields the best performance. Similar to Desai et al. (2023), scale batch of vectors before projecting it to the hyperboloid using learnable sealars Cing and Ctzt, respectively, in both image and text modes. These scalars are initialized with a value of Cimg = Cext = 1/1512. The adaptive softmax temperature of the contrastive loss initialized with 0.07 and clipped 0.01. All of these scalar values are learned in the logarithmic space.
In the hCE loss (Equations 10,11), we set separate values of the parameter for mnter-modality entailments Tinttr 0.7 and intra- modality entailments Mintro = 1.2 through hyperparameter search while pre-training on the CC3M dataset for 75k steps and eval- uating on ImageNet zero-shot image classification (cf. Table 8). Intuitively, this is because embeddings of images and text exist in different regions of the space, making it easier for text to entail the corresponding image texts nearer the origin and have a wider aperture w (cf. Eq. 7). Hence, we make the loss stricter by reducing the aperture of text embeddings. Similarly, the intra-modal box rep- resentations are closer in their corresponding spaces, Accordingly, we increase the aperture of the box regions relax the entailment loss. In the final hC loss, we set the weight for hCE loss 7 0.1.
Table 8: Hyperparameter search for Vinter performed on grounded CC3M dataset.
Tjinter
ImageNet
1.0
12.5
0,9
12.6
0,8
13.1
0.7
13.4
0,6
13.3
0.5
12.8
Optimizer and Hardware We train our models on 4 A100 GPUs for 500k steps using batch size of 768 on an internal cluster. Similar to Desai et al. (2023), we use the AdamW optimizer (Loshchilov & Hutter, 2019) with hyperparameters 31 = 0.9, 32 0.98 and weight decay 0.2 which is disabled for the learnable scalars. We use a cosine learning rate scheduler (Loshchilov & Hutter, 2017) with a maximum learning rate of 10-4 and a linear rate for the initial 4k steps.
C METRICS FOR HIERARCHICAL CLASSIFICATION
This section provides more details on the metries used for our hierarchical classification experiment. For pair of predicted and true class (g. y). the Tree Induced Error (TIE) (Dekel al., 2004) the distance between ý and in the graph (cf. Fig. 8a). This is defined as Σes We, where E(i, j) is the set of edges with weights we along the path connecting nodes and j. For the WordNet graph, we set w. Similarly, the Lowest Common Ancestor (L.CA) error the distance to the deepest common node in the graph which is shared between the ancestors of classes and y.
For set-based measures, we define Yanc and Yane as the set of ancestor nodes of classes á and respectively (cf. Fig. 8b). Other relevant set-based hierarchical metrics such as, Jaccard Similarity J, and Hierarchical precision (Pμ) and recall (RH) (Kosmopoulos et al., 2015) are then given by
J = Yanc O Yanel
Yanc n Yanel
Yaac O Yanel Yanc
Pи
RH
(13)
Panc U Yane!
Yanel
D REGIONCLIP ON OTHER TASKS
In addition to object detection (cf. Sec. 3.2, Zero-shot object detection), we evaluate RegionCLIP on the other downstream tasks described in Sec. 3.2. We compare RegionCLIP, which uses a ResNet-50 backbone (~25M parameters), to HyCoCLIP, which employs VIT-S/16 backbone with ~22M parameters. As shown in Table 9, our method significantly outperforms RegionCLIP across all these tasks. This is expected, as RegionCLIP is primarily optimized for object detection. By contrast, HyCoCLIP reaches the level of performance of RegionCLIP on object detection despite being designed to learn hierarchical representation spaces.
E SCENE UNDERSTANDING BENCHMARKS
In this section, we describe in detail the experiments of the compositional reasoning benchmarks used evaluate our models in Sec. 3.2.
20

---

Published as a conference paper at ICLR 2025
Yasc
LCA
=
(b)
Figure 8: Hierarchical classification metrics when labels have tree-like structure. Predicted label in red and true label in green. (a) Tree Induced Error (TIE) the graph distance till the correct label while Least Common Ancestor (LCA) is the distance till the first common ancestor label as shown in blue. (b) shows ancestor label set of the predicted label (Y (Yanc) and the true label (Yanc).
Table 9: RegionCLIP performance downstream tasks. The pre-trane pre-training of RegionCLIP using for object detection. Thus displays poorer performance in comparison
boxes is primarily optimized HyCoCLIP which is designed to learn hierarchical representations from the boxes. denotes scene understanding.
classification
retrieval
hierarchical metrics
CIFAR-ION 100
ImageNet
Food- 101
Mean( 6)
SUN397
Text
Image
TIE (L)
LCA
<t>
Pk kt)
VL-CO
VG-A
Model
RegionCLIP
40.6
23.2
43.4
41.3
36.4
38.5
31.5
3.76
2.29
0.77
0.84
0.84
52.5
59.7
HyCoCLIP
41.7
53.6
52.5
50.2
41.1
69.5
55.2
3.55
2.17
0.79
0.86
0.85
59.8
68.4
E. BENCHMARKS
VL-Checklist-Obiee (VL-CO) This benchmark (Zhao et al., 2022) modifies the caption in several aspects. An object term the caption replaced with random noun phrase. The model results are categorized for different sizes and locations of the object within the image to check for invariance which are summarized as follows.
O-Small: The object covers small within the image. Following Zhao al. (2022), the threshold of the object area is set to below 32 x 32.
O-Medium: The object covers a moderate area within the image. The threshold of
object area is set between 32 x 32 and 96 96.
O-Large: The object covers a large area within the image. Any object with an area greater than 96 96 fits this category.
O-Center: The object center lies within the center region of the image. If is the half-length diagonal, and y is the distance between the center of the object and the center of the image, the object considered lie in the center region if y 3/-
O-Margin: The object center lies at the margin of the image. This when 3
O-Mid: The object center lies in between the center and margin region which is when
VG-Attribution (VG-A) This benchmark (Yüksekgönül et al., 2023) tests the capability of the model to correctly identify the attribute word associated with object in sentence in the context of an image. For instance, the model has to pick between "the crouched cat and the open door" and "the open cat and the crouched "oor".
21

---

Published as a conference paper at ICLR 2025
Large
MERU
0.06
HyCaCLIP
black white
brown
white
wood
HyCoCLIP
blue
white
blacck
wood
0.60
G-Mediur
0.57
Margin
blue
0,78
white
0.54
brown
0.85
green
0.51
0.63
9:308
blue
white
green
0.55
g aay
6.50
white
rowwn
browb
hite
0-5malh
G Mid
blue
blue
green
black
brow white
green
gray
green brown
white
O-Center
white
(a)
(b)
Figure 9: Performance on scene understanding benchmarks (a) model performance on VL- Checklist-Objec (VL-CO) understanding experiments. HyCoCLIP performs best object under- standing tasks. (b) model performance on Visual Genome (VG) Attribution benchmark, accuracy values plotted for the 19 most frequent attribute pairs in the experiment. HyCoCLIP gives the best results on most of these attribute pairs.
Images
Images
Image
Image boxes
of cample
bowes Bodt
Text
Texts
Texts
Text boxes
Best lowes
.
1805
(a)
(b)
(c)
Figure 10: Visualizing the learned hyperbolic space of MERU in lower dimensions using samples from GRIT. (a) distribution of embedding distances from the origin, MERU embeds text data closer to the origin wri the images but box samples don't 't show smaller radius wrl full images/captions. On the right, (b) HoroPCA does not show local ordering and (c) CO-SNE visualizations of the latent space in 2 are quite distorted.
E.2 PERFORMANCE
We reported the mean performance for scene understanding benchmarks in Table 5. In addition to this, here we provide the results of HyCoCLIP compared against MERU on the individual categories of VL-CO and the top 19 most frequently occurring attribute pairs in the VG-A evaluation set. Fig. 9 highlights the significant improvements achieved by our method indicating better semantic scene understanding.
22

---

Published as a conference paper at ICLR 2025
F VISUALIZING REPRESENTATION SPACE -EEUU
Following Sec. 4, we additionally plot the learned hyperbolic space of MERU in lower dimensions in Fig. 10. We observe that the distributions text embeddings and image embeddings are overlapped with corresponding box distributions from Fig. 10a. This is also apparent in the 2D plot when using HoroPCA in Fig. 10b. The embeddings for modes seem to collapse small region with CO-SNE as seen in Fig. 10c.
G INTERPOLATION DETAILS
The logarithmic map is the inverse of the exponential map and, for p.q L", given by
(g) cosh ' (-n qL)) ((x(p. -1 (q + (tP)
(14)
Here, we will use interpolate between points in hyperbolic space (Desai al., 2023).
For hyperbolic representations (gr(Is), (r(T)) of source image Is and target image IT. we take the logarithmic maps (gr((gr(s)) and logo (gr(Ir)) and obtain a set N equally spaced representations on the line joining these vectors given by,
SS = (p. ETLL" p3 = (1 - 44) logo (gr (Ts)) + logo(ar(I))), 4 4= N i (1 (E......,
N) (15)
These are then mapped back to the hyperholoid using exponential mapping given by,
S. SW = f E L q. expo(pi), P SEJ.
(16)
The closest representations are retrieved for all points the S. using Lorentzian distance as the similarity metric from collection of representations of images and texts. Further, we drop any duplicate representations retrieved. For hyperbolic models, the origin (0) of the space taken as the root node, whereas for Euclidean CLIP, the root node is taken the centroid of all the training samples.
H HIERARCHICAL IMAGE-TEXT MATCHING BENCHMARK
In this section, evaluate model and baselines on the hierarchical image-text matching benchmark using the HierarCaps (Alper & Averbuch-Elor, 2024) test set. Similar to the setup of Alper & Averbuch- Elor (2024) and Appendix G, we take 50 equally- spaced points between the root node and the nearest text embedding to the image embedding and calcu- late the precision (P) and recall (R) relative to the four hierarchically relevant ground-truth captions. In addition, we also report the order-aware metric Tas which shows if the captions are ordered correctly in the embedding space. From the results in Table 10, find HyCoCLIP outperforms CLIP and MERU, further demonstrating the enhanced hierarchical understanding achieved with our method. Further, from Fig. 11 we empirically find HyCoCLIP performs better hierarchical image-text matching HierarCaps.
Table Hierarchical text-image matching performed on HierarCaps benchmark.
Model
P
R
Ta
CLIP
0.13
0.29
0.83
MERU
0.12
0.39
0.84
HyCoCLIP
0.12
0.46
0.88
23

---

Published as a conference paper at ICLR 2025
ROOT
IMAGE
HyCoCLIP
MERU
CLIP
HyCOCLIP
MERU
CLIP
out
picture
buhind
simething
looking
opening
semsshing
looking
wit
photeproph
good
pleture of food.
littl
people
stufted
swiking
eokerrycic food
bod
viutled
baking food
tuddy
vegutablie in
vegetable
leguse
Junk
sluffid antnal is foed
sone regetablee
plani vogetable
Lay room
affection
teddy beary
poold hanana
green vapple
that looks aliny Af
way
<dible vogetable
Bied filed
stuffid snimale.
edible vegetable
levcodli
stutfee bear
natural liber
coople boown hears
a vegetable
6led vegetablos
plate of brosolli
stuffed hears
Indly
the tobly Icing dowra on quilt
Hinder aume
place bencoali
vogriables in
etutiid teday bear sitting
a little stufled besz
brigiing beers blue pillow
saladd
putr teddy
sulted tubdy bear
huldiing lesecdi
vegetatle
hoare
stuffed teddy heors
The beccooll plattes in the
row sntything growing
s.stufled bear others
stufted bears
resling nest other
6 pair of taddy bears
warn dit top piecn
Twn Reggss bobling
stuffed
fnggr holding benooll
bear
dbuffed bear others
difiseent souggling
tagrshee.
STUPERD BEAR AND
beare laylng
bed undee
couple leoson Soiliy
taddy beare are lacing sode
quih.
toddy boars roating next
are and ose then is weening
Thrye tedily hears nach tagetlier.
Two stutledd animala
HyCoCLIP
MERU
CLIP
HyCoCLIP
MERU
CLIP
sometiing
looking
semothing
living thing
logume
let
picture
win
legune
pppctues of food.
curmodgoss
placal
serving
vanury bood
fsed
Itabee
dumeetic unitaal
Jurry
with aeversd didlerint
cooking
iagrediente.
meal
boolo
oaf
slerping
dah
x diub made from food on
plate with cuoked Bood
bing
kitty
cutte
Bied
plate of foed.
meal
en plate.
Ferry
kittes
dips that snv aborping in
peto
twiled Bood
Uhe
Vegriables and chicken
filins
cab
pota slouping ia the bef
plato of food is
leying
aleeping cata
bot
kitty
petoilo lof
rate
Chirkes meal
plato with cooked bood
eat lylng dowra
slopping eat
cate sloeping with
rerll plate
plato
Seed with and and
sleeping cat
There down us smnrliigg
the cala pest on
msal with sons
parsea disb with dijiaa
rals veeting
kitty sleeping
sleeping oats
The nats rest on each
plato with makked Soad
A plato of vegeaaions, rine. A plate of chirkin and bovi rics.
douplng
the cals are lying down
Two case lay tegother on
plate with cosked fond
Tun ooto Laying tegicher
solasto
A couplo of lyyling
segrisbles and chinken
Chere cate that are
the csnalutor
dinser that while sestard with shicken.
Twu together a Minnott.
Two cate by tegeire on
Mankee
two lylng two ressitee purple
Acopple of cata lafing
ablckon, and while sion.
yorinng on with
esuplo nate lopiin on
Roed dosk plane
taee nots lsing sner
wetting on the takis.
that
sheels
Figure II: Hierarchical image-text matching qualitative results on HierarCaps test set. The color gradient reflects the concept's radius from the root of the space.
24

---

Published as a conference paper at ICLR 2025
1 MORE INTERPOLATION EXAMPLES
SOURCE IMAGE
IMAE
IAAEE
ROOT
BUAG
HyCoCLIP
MERU
CLIP
HyCoCLIP
MERU
CLIP
bridge ski
Yext City
photopraph ofan Indtin
caylie
the skyiks of Youk
Kigh
the
photo
York City
ROOT
TARGET IMAGE
SOURCE IMAGE
IMAG
IMADE
ROOT
IAAE
HyCoCLIP
MERU
CLIP
HyCoCLIP
MERU
kil lo being Bown sust And the bost and the peraoo Rying Als
harbor
asspansiie arddpe
stoperaken undggs persan Sylin sin are
perzon tying the iixx
TWo sallooss and lake
the Narbor
with Nol, Brossol
AGNITC Vier
calory in
1h' phobo
Fat
"Just
ROOT
TARGET IMAGE
Figure 12: Interpolation between points. Multimodal retrieval results when moving from (top) an image to (left) another image or (right) the root. For HyCoCLIP and MERU, the root is the origin of the space, whereas it is the centroid of training sample representations for CLIP.
25

---

Published as a conference paper at ICLR 2025
SOURCE IMAGE
BOURDE
SOUNCE ROOT
TARDET
HyCoCLIP
MERU
CLIP
HyCoCLIP
MERU
CLIP
of Ankka
Meric
snage of colsered
monamentt bushing
stuckare
ROOT
the roachayy and catheched beckgroundd
the witeur Niww
Couste
Aptited bway alver
TARGET IMAGE
Figure 13: Interpolation between points. Multimodal retrieval results when moving from (top) an image to (left) another image (right) the root. For HyCoCLIP and MERU, the root is the origin of the space, whereas it is the centroid of training sample representations for CLIP.
26

---

Published as a conference paper at ICLR 2025
SOURCE IMAGE
SOURCE
SOURCE HADE
IMAE
ROOT
TARGET
HyCoCLIP
MERU
CLIP
HyCoCLIP
MERU
20TH goods
ALTWOAA feeu
other acedes Namu
waisus suggiesas
propt
bunch of aunglss
binn
And part
crefl work
erystaa
buan recorree pur
Who
ROOT
TARGET IMAGE
SOURCE IMAGE
IMAG
IMAG
ROOT
BAGG
HyCoCLIP
CLIP
HyCoCLIP
eaayy rapove
adult viding
eapeur maix the eky
pUPt
Aftorte arnong Aille
reacy
elssvilly
berch with manyy vtibly
arivors
WW pistue
ROOT
TARGET IMAGE
Figure 14: Interpolation between points. Multimodal retrieval results when moving from (top) image to (leff) another image or (right) the root. For HyCoCLIP and MERU, the root is the origin of the space, whereas it is the centroid of training sample representations for CLIP.
27

---

Published as a conference paper at ICLR 2025
SOURCE IMAGE
SOURCE
SOURCE HADE
IMAE
ROOT
TARGET
HyCoCLIP
MERU
CLIP
HyCoCLIP
MERU
CLIP
A Araas oheref buyy
parted cara
A vite of street coree
jeralial parted war
ahllgg and onfooherr enjoy the
wwwrral other phaws
A grous boltesseer
ROOT
TARGET IMAGE
Figure 15: Interpolation between points. Multimodal retrieval results when moving from (top) an image to (left) another image or (right the root. For HyCoCLIP and MERU, the root is the origin of the space, whereas it is the centroid of training sample representations for CLIP.
28

---

Published as a conference paper at ICLR 2025
SOURCE IMAGE
SOURCE
SOURCE HADE
IMAE
ROOT
TARGET
HyCoCLIP
MERU
CLIP
HyCoCLIP
MERU
CLIP
brightty paczage
the bain backa
dreaaed whitd capas nanning
nel bucks
Rold
Dall Are
staiie
Me
shanng rovvmmants Var other fores saferee
ROOT
TARGET IMAGE
Figure 16: Interpolation between points. Multimodal retrieval results when moving from (top) an image to (leff) another image or (right) the root. For HyCoCLIP and MERU, the root is the origin of the space, whereas it is the centroid of training sample representations for CLIP.
29


Reviews:
[{"review_id": "YoCYif11t2", "reviewer": "Reviewer_SpcR", "summary": "This work proposes a novel learning method for training vision-language models. Specifically, the method involves pretraining such models with 2 losses --- hierarchical compositional contrastive and entailment losses. The hierarchical concepts correspond to image boxes and the corresponding text boxes. The experiments are conducted on large scale dataset (GRIT) consisting of 20.5M image-text pairs. In Appendix A, the authors describe an automatic procedure to obtain the text boxes (noun entities in this case) and their corresponding bounding boxes in the images. The paper details empirical results on a variety of tasks including zero-shot image classification, retrieval, object detection and scene understanding.", "questions": "Can the authors share the results of HyCoCLIP on RedCaps dataset?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The proposed method is simple and elegant and can be easily applied to large scale pretraining of vision-language models. The procedure to automatically generate paired image and text boxes is also relatively straightforward.\n* The empirical results show improvement across several tasks which demonstrates the improved representation learning - classification, retrieval, detection and understanding.\n* Table 1 results show that CLIP trained on additional image-text boxes doesn't improve the performance. However, training on the same data but with the proposed hierarchical compositional learning losses shows significant improvement in performance. This further demonstrates the effectiveness of the proposed technique.", "weaknesses": "When training CLIP on additional image-text boxes shows no improvement (Table 1), it could be because there is limited new information in such examples (as original image-text pairs are already present in the training data). For a better understanding of this, an experiment such as this might help: split the GRIT dataset into 2 random subsets of 10M each. Then compare the results on the following settings:\n\n[1] CLIP trained on 10M image-text pairs\n\n[2] CLIP trained on 10M image-text pairs + additional image-text boxes\n\n[3] HyCoCLIP trained on 10M image-text pairs + additional image-text boxes\n\n[4] CLIP trained on 20M image-text pairs\n\nThe paper presents the comparison of [1] vs [2] vs [3] (but on all 20M image-text pairs) in Table 1 but comparing [3] vs [4] will help answer the above question. It is worth noting that even if the comparison shows similar results, [3] might still be slightly favored since it can be applied on top of any existing large dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kNemB2selH", "reviewer": "Reviewer_cUjb", "summary": "This paper introduces a novel approach named HyCoCLIP to vision-language modeling that leverages the hierarchical nature of hyperbolic space to better align visual and textual data. It organizes image and text data as hierarchical compositions, where objects within an image and their corresponding text descriptions are represented at different levels of abstraction in hyperbolic space.The experiments demonstrate that HyCoCLIP achieves significant performance improvements across multiple tasks.", "questions": "Could you please provide more details on the choice of hyperbolic space parameters?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-organized. The motivation is easy to follow, and the method is easy-to-understand.\n2. The proposed HyCoCLIP is novel and effective. It organizes data at multiple abstraction levels, providing an inspiring approach to multi-modal learning.\n3. The authors performs exhaustive experiments to show that the effectiveness of HyCoCLIP. It outperforms baselines on general and fine-grained image classification tasks.", "weaknesses": "1. While the paper compare with CLIP and MERU, it should also compare some recently proposed VLMs.\n2. The paper should explore how sensitive the model is to the choice of hyperbolic space parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Go6M1iG7Sj", "reviewer": "Reviewer_UEf2", "summary": "The authors proposed to incorporate hierarchical pretraining for hyperbolic vision language models and the resulting model Hyperbolic Compositional CLIP (HyCoCLIP). The core idea is to construct object regions (image boxes) and corresponding text phrases (text boxes) to build a multi-layered, compositional hierarchy within the shared hyperbolic embedding space. The HyCoCLIP shows competitive performance in zero-shot classification and retrievals. The author also conducted experiments to show how HyCOCLIP can outperform CLIP and the hyperbolic contrastive model MERU in zero-shot hierarchical classification and scene understanding tasks.", "questions": "In Table 1/2, the authors bold the best performance overall across different model backbones. Wouldn’t it be more informative and fair to bold the best performance within each backbone group (e.g., ViT-S/16, ViT-B/16) to allow for a clearer comparison of HyCoCLIP’s performance relative to baselines on similar architectures?\n\nRegarding the choice of batch size, the authors used a batch size of 768 due to memory limitations. Did the authors consider implementing techniques like gradient accumulation to effectively simulate a larger batch size? This could provide further insights into how batch size impacts model performance, especially since batch size has been shown to affect contrastive learning tasks significantly.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think this paper is very well written and I find it easy to follow. Overall the idea behind HyCoCLIp is well motivated and I believe the authors have conducted sufficient experiments to empirically demonstrate the proposed method and model’s efficacy. The empirical performance of HyCoCLIP is very strong and to the best of my knowledge, the proposed HyCoCLIP achieved the state-of-results on many of the reported zero-shot tasks from a contrastive-pretrained model.", "weaknesses": "One major concern is the incremental nature of this work. Hyperbolic embeddings for representing hierarchical relationships have been explored in previous models, and this paper primarily builds upon these established ideas. However, the specific contributions of HyCoCLIP, particularly in enhancing hierarchical and scene understanding tasks, offer sufficient merit to make this work valuable to the broader community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lMOZfWSoFx", "reviewer": "Reviewer_5zgK", "summary": "This paper proposes the novel Compositional Entailment Learning framework to train VLMs, by using as supervision the hierarchical relations between images, captions, and constituent nouns and their bounding boxes. Their results show that this outperforms standard CLIP and the hyperbolic CLIP variant MERU on both standard multimodal and hierarchical benchmarks. This is supported by qualitative results illustrating the learned hierarchical semantics of the learned space.", "questions": "Could the use of objects as supervision bias the model towards nouns and concrete concepts, possibly at the expense of attributes, dynamic actions (verbs), etc.?\n\nSome details that are unclear from Supp. A: How were abstract nouns filtered? Are the nouns that can be grounded open-vocabulary (not limited to a fixed list)? How accurate is the GLIP-based grounding procedure?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The central idea is clever and novel – utilizing the hierarchical nature of nouns mentioned in image captions as supervision for a hyperbolic model. The exposition is clear and concepts are well-illustrated. The quantitative experiments are extensive and overall convincing.", "weaknesses": "Qualitative results (Sec 4, Supp 8) are fairly limited. In particular, it is missing a qualitative comparison to existing models (CLIP, MERU) to illustrate whether HyCoCLIP’s embedding space represents hierarchies in a more qualitatively satisfying way.\n\nWhile a comparison to CLIP trained from scratch is provided, recent work has found pretrained foundation VLMs to represent hierarchies in Euclidean space [1]. It would be useful to compare to such results to understand whether HyCoCLIP trained from scratch is competitive with such models.\n\n[1] Alper and Averbuch-Elor. Emergent Visual-Semantic Hierarchies in Image-Text Representations. ECCV 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)