Intriguing Properties of Hyperbolic Embeddings in Vision-Language Models
Authors: Sarah Ibrahimi
Mina Ghadimi Atigh
Nanne Van Noord
Pascal Mettes
Marcel Worring
Venue: Accepted by TMLR ()

Abstract: Vision-language models have in short time been established as powerful networks, demonstrating strong performance on a wide range of downstream tasks. A key factor behind their success is the learning of a joint embedding space where pairs of images and textual descriptions are contrastively aligned. Recent work has explored the geometry of the joint embedding space, finding that hyperbolic embeddings provide a compelling alternative to the commonly used Euclidean embeddings. Specifically, hyperbolic embeddings yield improved zero-shot generalization, better visual recognition, and more consistent semantic interpretations. In this paper, we conduct a deeper study into the hyperbolic embeddings and find that they open new doors for vision-language models. In particular, we find that hyperbolic vision-language models provide spatial awareness that Euclidean vision-language models lack, are better capable of dealing with ambiguity, and effectively discriminate between distributions. Our findings shed light on the greater potential of hyperbolic embeddings in large-scale settings, reaching beyond conventional down-stream tasks. Our code is available at https://github.com/saibr/hypvl

Published in Transactions on Machine Learning Research (07/2024)
Intriguing Properties of Hyperbolic Embeddings in Vision-Language Models
Sarah Ibrahimi University of Amsterdam
s.brrahims@uvo.nl ml
Mina Ghadimi Atigh University of Amsteriam
m.ghkadiminkkghhhnw....d
Nanne van Noord University of Amstendam
n.j.c.vannom@@@uvo..l
Pascal Mettes University of Amsterdam
p.s.m.mettes@wv..ll
Marcel Worring
m.worringGuuv..t ml
University of Amsteniam
Reviewed on OpenReview: /ttps: openreveew. net/ forum? id- P602gf
Abstract
Vision-language models have in short time been established powerful networks, demon- strating strong performance
wide range of downstream tasks. A key factor behind their success is the learning of joint embedding space where pairs of images and textual descriptions are contrastively aligned. Recent work has explored the geometry of the joint embedding space, finding that hyperbolic embeddings provide a compelling alternative to the commonly used Euclidean embeddings. Specifically, hyperbolic embeddings yield improved zero-shot generalization, better visual recognition, and more consistent semantic interpro- tations. In this paper, we conduct deeper study into the hyperbolic embeddings and find that they open new doors for vision-language models. In particular, we find that hyperbolic vision-language models provide spatial awareness that Euclidean vision- vision-language models lack, are better capable of dealing with ambiguity, and effectively discriminate between dis- tributions. Our findings shed light on the greater potential of hyperbolic embeddings in large-scale settings, reaching beyond conventional down-stream tasks. Our code is available at httpss://iithub com/ www.ccoccssssseece saibb//bybb/ll
1 Introduction
Continued advances large-scale contrastive learning between visual data and rich semantic descriptions have produced influential vision-language models, such as Flamingo (Alayrac et al., 2022), FLAVA (Singh et al.
2022 and CLIP (Radford et al. 2021). These models not only allow for multimodal retrieval, their original training objective, but also show to be powerful networks for down-stream zero-shot (Ge et al.
2023b Menon & Vondrick 2023 Pratt et al.) 2023) and fine-tuning tasks (Luo et al. 2022 Goyal et al.
2023 Paiss et al. 2023). Though commonly optimized in Euclidean space, Desai el. al. (2023 recently proposed using hyperbolic embeddings instead, yielding further improvements on retrieval and zero-shot gencralization. Since hyperbolic geometry differs vastly from Euclidean geometry, embedding imago-text pairs in this alternative space leads inherently different representational properties. The goal of our work is uncover the new capabilities that emerge from using hyperbolic geometry for vision-language models.

---

Published in Transactions on Machine Learning Research (07/2024)
The potential of hyperbolic geometry has recently been shown for singlo-modality analysis in computer
vision (Atigh et al. 2021: Ermolov et al.] 2022 Franco et al. 2023: Ge et al. 2023b Liu et al. 2020a:
Khrulkov et al. 2020) and language understanding (Tifrea et al. 2019 Zhu et al. 2020; Dhingra et al, 2018). A key advantage of the hyperbolic space is its inherent ability to represent hierarchical structures
and complex tree structures with minimal distortion (Ganea et al.; 2018a: Nickel & Kiela, 2017 Sala et al,
2018) which not directly feasible in the Euclidean space (Nickel & Kiela 2017). Building on those insights,
Desai et al. (2023) integrated vision-language models in the hyperbolic space. Their results demonstrate that the benefits generalize to common tasks in the multimodal domain, with competitive zero-shot image classification and improved zero-shot retrieval over Euclidean models, in particular with lower dimensional embeddings. However, an open question remains: are hyperbolic vision-language models useful beyond the common zero-shot classification ion and image-lext retrieval tasks and do hyperbolic embeddings bring new capabilities to vision-language models?
Hyperbolic emboddings have shown advantages over Euclidean embeddings in unimodal settings for tasks including out-of-distribution generalization, uncertainty quantification, and hierarchical representation learn- ing (Mettes et al.) 2024). In multimodal scenarios, hyperbolic embeddings might show similar benefits. In this work, we systematically compare Euclidean and hyperbolic image-text embeddings and find three prop- erties where hyperbolic embeddings shine. These properties provide insights into why hyperbolic embeddings achieve strong performance and also demonstrate their potential for real-world impact.
Spatial Awareness. Our analysis reveals that hyperbolic vision-language models have better spatial awareness compared to Euclidean models. While recent. benchmarks have been proposed for evaluating spatial reasoning (Zhao et al. 2022 Yüksekgönüi et al.) 2023 Lewis et al.) 2023). large vision-language models still struggle on such tasks, potentially due to contrastive training encouraging shorteut behaviors YYüksekgönül et al. 2023). We demonstrate that the hyperbolic vision-language model outperforms its Euclidean counterpart on three spatial awareness benchmarks. Notably, the Euclidean model performs no better than random chance on two cases, whereas the hyperbolic model succeeds. To our knowledge, spatial awareness remains unexplored for hyperbolic vision-language works and our findings reveal new potential for hyperbolic embeddings in vision- language learning in overcoming its limitation spatial reasoning.
Ambiguity Resolution. We find that hyperbolic vision-language models are better in handling ambiguity. While many multimodal tasks involve literal appearances, real-world data contains sub- stantial ambiguity; from words with context-dependent meanings to memes with vague image-text relationships. Ambiguity in a machine learning context can be related to uncertainty and to seman- tics. In this work, we refer to a specific type of semantic ambiguity. The interpretation of memes and other multimodal polysemic challenges involves a grounding problem where the semantics of the text data can only be resolved by grounding it to the image or video, therefore we will refer to visually-grounnee language ambiguity' in the remainder of this paper whenever we use the term ambiguity. We demonstrate that the hyperbolic vision- languaee model outperforms the Euclidean model on three tasks involving multimodal ambiguity: propaganda classification (Dimitrov et al. 2021), visual word sense disambiguation (Raganato ct al. 2023) and text-to-GIF retrieval (Song & Soleymani 2019). A potential explanation is that the hyperbolic space better preserves sernantic hierarchies in text embeddings. Through experiments mapping hierarchical relationships, we con- firm lower embedding distortion with the hyperbolic model compared to the Euclidcan model. We suggest that the improved performance on anbiguity lasks likely comes from implicit hierarchical representations in the hyperbolic embedding space.
Out-of-distribution Discrimination. We find that hyperbolic vision-language models effectively discriminate between in-distribution and out-of oifiiiiiiiitttioonnnnoooo samples. For deployable real-world sys- lems, simply classifying Lest samples is insufficient and models must detect when inputs differ seman- tically from the training distribution (Yang et al. 2022). Evaluating across three evaluation metrics, three out-of-distribution detection methods, and five distinct datasots, we consistently observe sub- stantial gains in out-of-distribution detection for the hyperbolic vision-language model compared to the Fuclidean model.
2

---

Published in Transactions on Machine Learning Research (07/2024)
With these findings, we shed new light on the importance of the embedding space and the potential of hyperbolic geometry in vision-language models.
2
Related Work
2.1 Image- Text Representation Learning
Large-scale vision-language models have demonstrated strong performance on wide variety of downstream tasks (Alayrac et al. 2022; Singh et al. 2022 Radford et al., 2021j. The motivation for training such models has been the large availability of images with raw web-text annotations that can be used in training as a way of free-form language supervision. These models have been applied to classification, retrieval and detection across both unimodal and multimodal contexts in zcro-shot (Roth et al., [023] Pratt ct al. 2023
Menon& & Vondrick 2 few-shot (Zhang et. al. 2022 P et al. 2023). and full fine-tuning scenarios [Goya]
et al. 2023 Ibrahimi et al. 2023 Paiss et al. 2023). Beyond individual tasks, research has also explored
robustness to natural distribution shifts (Manli et al. 2022 interpretability (Materzynska et al. 2022 and
embedding space properties (Liang et al. 2022). Another emerging area evaluates reasoning about objects,
attributes, and relations (Zhao et. al. 2022 Yükkekgönül et al. 2023 Lewis et al. 2023). Moreover, recent work from Fang et al. (2022) studied the robustness of CLIP and showed that this is mostly related to the data distribution of the training set, while factors such as the language supervision play a minor part. The dominant performance of vision-language models across plethora of down-stream tasks has made them staple technique in the field.
2.2 Hyperbolic Learning
Recent years have witnessed rapid progress hyperbolic learning, as surveyed in Mettes al. (0024) Peng
et al. 20022). and Yang et al. (2023). Specifically in the visual domain, hyperbolic learning has shown to be
effective for hierarchical learning (Dhall ct al. 2020
Atigh et al. 2021 Long et al. 2020 Liu et al. 2020a
few-shot learning (Gao et al. 2021 Ma et al. 2022
Khrulkov et al. 2020 robust learning 4 @aan Spengler
et al. 2023 Jie Hong 2023). and low-dimensional learning (Chami et al. 2020 Nagano et al.) 20199] Bose
et al.
2020 Beyond these general areas, promising results have been demonstrated for specific vision tasks
such as object detection [Lang et al. 2022 Go et al. 2023a), image segmentation (Atigh ct al., 2022). action
recognition (Gulshad et al.] 20023 Franco el al. 2023), and deep metric learning (Ermolov el al. 2022). For
language specific tasks, hyperbolic learning has improved word embeddings (Tifrea et al. 2019
Zhu et al.,
2020 Dhingra ct al.] [00_]2011) graph representations (Chami et al. 2020 Liu ct al.) 2019) and recommender
systems (Mirvakhabova et al. 2020 Wang et al. 2021). The effectiveness of hyperbolic spaces across diverse
learning paradigms and tasks emphasizes their potential as a fundamental technique in machine learning.
Desai et al. (2023) introduced the first large-scale hyperbolic vision-language model, demonstrating the emergence of hierarchical structure at text level representations when trained with a loss function enforcing entailment cones (Ganea et al.) 2018ah. In follow-up work, Shen et al. (2023) evaluated the hierarchical properties of this hyperbolic model on superclass- based image classification and subclass clustering of image embeddings. Their experiments showed comparable performance to Euclidean baseline, indicating limi- tations in capturing hierarchical semantics in the image embedding space. Shen et al. (2023) highlighted that hyperbolic vision-language models are not only effective for standard evaluation tasks, but also for interprelability. However, Desai el. al.] (2023) showed that the hierarchical properties are enforced due to the entailment loss and are therefore most likely present in the text embedding space. In this work, we confirm the hicrarchical propertics being present in the text embedding space and we show that hyperbolic embeddings in vision- language models indeed open new doors, namely for spatial reasoning, ambiguity, and out-of-distribution discrinination, all out-of-the-box.
3

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Euclidean
maazonin
resscclla
(bear in kloss" bboor out of glass
Ambiguuly
CLP image encoder
CLIP lexi ancaddr
Amtiguly
(Tis a propegards about srwwers on This propegerale
datuiti solection
Patel deteccioon
I' a proto dli bapel you aia
Hyparbolic
COO
Figure 1: Hyperbolic and Euclidean Geometry for Vision-Language Representation Learning. An image and text encoder are used to map images to embeddings in the embedding space. To map the embeddings into the hyperbolic space, an additional exponential mapping function is required, defined by Equation g Examples of the visual and textual input for the three properties are presented on the left and right. The color tones indicate whether the embeddings Euclidean (blue) hyperbolic (red) and the shapes represent visual embedding (circle) or text emmbedding (square). The Euclidean space brings correspouding image and text embeddings closer logether and pushes dissimilar embeddings apart. In the hyperbolic space, the concept of imaginary entailment cones is visualized with a text embedding at the start of the cone and an image embedding within its cone.
3
Insights into the Hyperbolic Geometry for Vision- Models
3.1 Background
Image- Text Representation Learning
Typical large-scale vision-language models consist of an image encoder and a text encoder I' that encode the two modalities separately and use a lincar projection layer to retrieve the respective embedding. The resulting embeddings F, and ET are then L2 normalized and aligned through either (1) loss function operating on the encoder outputs directly or (2) an additional multimodal encoder. Due to their overwhelming popularity, we focus on (1) with two unimodal encoders and a shared embedding space. Given the image embedding and text embedding, their similarity is measured by the cosine similarity. During training, this similarity is maximized for matching image-text, pairs and minimized for dissimilar pairs. The cross-entropy loss is applied over the similarity scores to achieve this. In this paper, we focus on the ubiquitous CLIP model [Radford et al., ) which consists of a Vision Transformer (ViT) (Dosovitskiy et al. 2021 as the image encoder and a regular Transformer (Vaswani et al. 2017) as the text encoder following Radford et al. (2019) As CLIP is trained on imago-sentence pairs, text prompts are typically formatted into full sentences at inference time to match the training distribution. For the visual recognition task, the common approach is to transform class labels into descriptions akin to "this is a photo of a [class name]" (Menon & Vondrick
2023
Pratt et al. 2023 Radford et al. 2021 This space is visualized in the upper half of Figure
Hyperbolic Geometry for Visual and Textual Representation Learning
Hyperbolic geometry possesses constant negative curvature, contrasting the zero curvature of the flat Eu- clidean space. This enables unique properties relevant for representation learning, including an inherent hicrarchical structure and exponential expansion, mcaning that the volume of ball in hyperbolic space grows exponentially as funetion of its diameter, rather than polynomially as in Euclidean geometry. Sem- inal work hyperbolic learning leverages these traits for hierarchical embeddings with minimal distortion, vastly outperforming Euclidean embeddings (Nickel & Kiela 2017). This finding has resulted in a rapid up- take in machine learning, with the development of hyperbolic network layers (Ganea et al.) 2018b
Shimizu
et al. 2021 van Spengler et al., 2023) and corresponding optimization ion methods (Lou et al. et al.
2020 Weber
2020
More generally, hyperbolic space can be represented by multiple models (Cannon al. Poincaré ball and the Lorentzian hyperboloid common choices in vision (Mettes al. 2024 and language
1997 with the
4

---

Published in Transactions on Machine Learning Research (07/2024)
Peng et al. 2022 representation learning. Each formulation uses specific distance metric well exponential maps, which project points from the tangent (Euclidean) space to hyperbolic space.
Following Desai et al. 2023), we adopt the Lorentzian hyperboloid to avoid numerical instabilities associated with the Poincaré distance metric (Nickel & Kiela, 2018). For completeness, we repeat the definitions for the Lorentzian hyperboloid as presented by Desai et al. (2023) consisting of the Lorentzian inner product, the Lorentzian norm, geodesics, the tangent space, the exponential map and the logarithmie map. Furthermore, provide the key equations for projecting Euclidean embeddings and measuring similarity in this space, leveraged subsequently in our analysis. The model represents n-dimensional hyperbolic space the upper half of two-shected hyperboloid in R Aligning with special relativity theory, Desai et al. (2023) delineate the axis of symmetry as the lime dimension, with the remaining axes termed space dimensioons. A vector x E R=+1, written as [spacce Zismel: comprises space dimensions Xspace E R" and a time dimension Prime E R
Lorentzian inner product. Let (5)) be the Euclidean inner product and (,)) the Lorentzian inner product, produced by the Riemannian metric of the Lorentz model. For two vectors x, E R=+1, the Lorentzian inner product is defined as:
(x, x) = (xx==(xppoot (Xepace: Napacee Sspace)
Prime Utime
(1)
Lorentzian norm. This defined by ||x||c vl(x,x) The Lorentz model having a constant curvature -c is defined the set of vectors:
C E
1/c c 0
(2)
All vectors in this set satisfy the following constraint:
Itime 3/0 + ||xspcce ||
(3)
Geodesics. A geodesic are is the distance minimizing curve between two points on the manifold. Geodesics in the Lorentz model are intersections of the hyperboloid with hyperplanes passing through the origin of ga+1, The Lorentzian distance between two points x, y E L is:
dc(x,y) Teee cosh (
(4)
Tangent space. The tangent space point z E L is a Euclidean space of vectors that are orthogonal to according the Lorentzian inner product:
TaL" = {VER**I v) =
(5)
An orthogonal projection can project a vector in ambient space 1=++1 to the langent space TaL".
v proja (u) = u cz (z, u)c
(6)
Exponential map. This maps vector v from tangent spaces onto x on the manifold. For point the hyperboloid, it is defined as expm, T + d with the expression:
sinh (é l|v|le)
== expm, 2 (v) = cosh( (Vvlc) 2
Ve lvllc
Logarithmic map. This is the reverse of the exponential map (logm, L + Tz,C), that maps x from the hyperboloid back to v in the tangent space:
== loogn (x) cosh (1cc (2,x)
projz (x)
(8)
V( x))
5

---

Published in Transactions on Machine Learning Research (07/2024)
We follow Desai al. (0033 and only consider the maps where is the origin of the hyperboloid (0 [0, v
Projection of embeddings. We project the Euclidean image and text embeddings Ej and ET into the Lorentzian manifold via the exponential mapping. Since Ej, Er E R", a transformation needs to be applied to make sure that the projected embeddings lie on the Lorentz hyperboloid f in Ra+1, This can be done by ET Er, 0] E |g++1 and ET [E,, 0] E I Embeddings belong to the tangent space at the hyperboloid origin O and therefore only the space dimensions used in this projection, which results in E E Expce and Er Eppace Now Equation 7 can be simplified to the following equation for k {I, T):
sinh( Ve
[ Kaapaeo
E
Ekspace!
E (I,
Kapace
Ve E Exspaceel
We measure the similarity between Ej and ET using the Lorentzian inner product from Equation calculation, we need the time components that can be calculated using Equation for E The inner product results in:
1 For this Lepacn and E
Tappce"
Ei. E =Ei
ET appccc
Ei,ame
(10)
Desai et al. (2023) train a hyperbolic CLIP model using a combination of contrastive and entailment losses. The contrastive component mirrors the loss from Radford al. (2021), substituting the Euclidean similarity function with the negative Lorentzian distance function. The entailment loss is based on work by Gaanea ct al. 2018a) and Le et al. (2019) and aims to enforce hicrarchical structure. Specifically, image embeddings Ej are pushed into imaginary cones projected by the corresponding text embeddings Er, positioning visual concepts towards the edges of the hyperbolic space. This space visualized in the lower half of Figure [ Since fine tuning and implementation details for these losses are out of scope of this work, we refer readers to Desai et al. (2023) for specifics.
Next, we discuss three assumptions the potential benefits of hyperbolic geometry for vision-language modeling.
3.2 Assumption 1: Hyperbolic vision-language models exhibit the potential for spatial awareness.
Recent work has analyzed the spatial reasoning capacities of pre-trained Euclidean vision-language models by studying compositional relationships, regarding objects, attributes and relations (Yüksekgönül et al.
2023 Zhao et al. 2022 Lewis et al., 2023). Results show poor zero-shot performance, with models failing capture semantic relationships between objects, attributes, and relations (Yüksekgönül et al. 2023 Zhao 2023). The models exhibit "bag-of-wordss behavior indicated by Yüksekgönül et al. and Zhao et al.
et al. 2022 Lewis et al.
20033)
2022), with performance unrelated to scale or training data size. Yüksekgönül et al. 20233 hypothesize that contrastive pre-training enables shortcuts that inhibit learning true compositionality.
Ullike Euclidean alternatives, hyperbolic CLIP uses an additional entailment loss that may prevent shortcuts
and enforce latent hierarchies during training. We expect this geometric structure and optimization offers improved spatial awareness lacking conventional vision-language models.
3.3 Assumption 2: Hyperbolic vision-language models are better in dealing with ambiguity.
Ambiguity is pervasive in language (Piantadosi et al. 2012 More important, ambiguity is omnipresent in real-world multimodal data, making it an important aspect to consider for deployment. As Chen et al. (2022) observed, ambiguity adds representational granularity by presenting multiple perspectives on identical concepts. We expect that hyperbolic vision Iision-language models are better equipped to tackle ambiguity given their latent hicrarchical organization. We investigate this assumption by first quantitatively comparing Euclidean and hyperbolic models on tasks with ambiguity, after which we dive deeper into the hierarchical nature of both models.
6

---

Published in Transactions on Machine Learning Research (07/2024)
3.4 Assumption 3: Hyperbolic vision-language models are good oisiiitiiituiion discriminators.
Hyperbolic learning has demonstrated polential for robustness lasks and uncertainly quantification in uni- modal scenarios (Mettes et al,, 2024). A critical challenge in this domain is out-of-distribution (OOD) detection, aiming to identify inputs that deviate from the distribution of training or fino-tuning data. OOD detection links to several active research areas like open-set recognition, anomaly detection, novelty detec- tion, and selective prediction (Vang al., 2022). We expect that the geometric properties of hyperbolic embeddings can enhance distributional awareness in vision-language models. Specifically, the concentration of semantic concepts combined with expansion towards the borders could sharpen discrimination of in- versus out-of-distribution data. We test this via OOD detection lasks comparing hyperbolic and Euclidean CLIP.
3.5 Setup
Throughout this paper, we compare two models, namely the Euclidean CLIP model and the hyperbolic CLIP model presented by Desai et al. (2023). In their work, they trained ViT-S/16 /6, ViT-B/ 16 and ViT- L/16 backbones and they publicly released their model chockpoints, providing one chockpoint per model. Both Euclidean and hyperbolic models share the same encoder architecture and were trained on 12M image- text pairs from RedCaps (Desai et al. 2021
In our study, evaluate the Euclidean and hyperbolic ViT-L/16 backbones on all tasks and for cach property, add comparison study including all backbone sizes. For fair assessment, we use the provided checkpoints from Desaī et al. (2023) in a zero-shot evaluation mode. Thus, we do not train nor finetune any model ourselves. All models output 512-dim embeddings, we will use these embeddings directly for all our experiments and do not add any additional linear layers to the model.
Since we have only one chockpoint for each model, we are unable to average the results over multiple checkpoints. However, we provide statistical significance tests on the results by using a bootstrapping technique on the test set. More specifically, we follow (Sanchez-Lenger Lengeling et al. (019)) by using a resampling method with replacement on the test set for 500 times, due to common practice. This results in a distribution of 500 data points for which we report the mean and standard deviation for all experiments. We perform one-sided t-tests on the data distributions of the Euclidean and hyperbolic models with the null hypothesis stating that the data distribution of the hyperbolic model has a higher mean value than the Euclidean model with a significance level of 2.5%.
Tasks use the original similarity score from CLIP Radford al. 2021 which is the cosine similarity, and the Lorentzian inner product for hyperbolic CLIP, unless otherwise specified. An overview of the pipeline is presented in Figure]
4 Results
4.1 Hyperbolic vision-language models exhibit spatial awareness
Sctup. To study the first property, we take three benchmarks that have been recently formulated to study spatial awareness: VL- CL-Chhcklist (Zhao al.] 2002),, VG-Relations (Yüksekgönül et al. 2023) and CLIPbind-
[Lwwis et al. 2023). In all datasets, the setup is relation R between two nouns a and b, with notation altb.
VL-Checklist: a spatial reasoning benchmark dataset that uses 30k Visual Genome (Krshhna al. 2017) images with two different descriptions, a correct description d4 with relation R and hard- negative description d_ with a different relation S (aSb). The model is given image with two sen- tences as input and returns similarity scores between the image and both sentences, the Lorentzian inner products (ET ETa, )e and (E₁, Era )c for the hyperbolic model and cosine-similarity be- tween E1 and Era, and between E₁ and Era for the Euclidean model. When (E7, Ei )2 (Ej, Eia >c, the prediction is considered as correct for the hyperbolic model. The same holds for the cosine similarity scores. Accuracy is the percentage of images correctly scored higher for the true relation. As this is binary task, random chance is 50%.
7

---

Published in Transactions on Machine Learning Research (07/2024)
Table I: Hyperbolic vision-language models exhibit the potential for spatial awareness. The Euclidean CLIP exhibits little no spatial awareness for VL-Checklist, even leading to results worse than random guessing for some spatial lasks (CLIPbind-r). Hyperbolic CLIP performs clearly better across all three challenging spatial tasks, highlighting that these models do not shorteut spatial relations between objects and altributes and provide a richer embedding space, even though they are trained on the exact same data. L,, B, and S refer to the ViT-L/16, ViT-B/16 and V/T-8/16 models and for all results indicated with 3 the hyperbolic model is statistically significant better than the Euclidean model with p < 0.025.
Model
VL-Checkllst
VG-Relations
CLIPbind-r
8
B
L
S
B
S
B
Random guess
50.0
50.0
50.0
50.0
50.0
50.0
20.0
20.0
20.0
Enclidean CLIP
55.3 + 1.6
52.1
1.5
52.3
1.6
58.5 + 1.7
57.9 + 1.6
56.3 + 1.7
16.6 0.2
4.6
0.1
12.8 + 0.2
Hyperbolic CLIP
54.8 + 1.6
57.6 + 1.5
50.9 + 1.5
61.5
1.8
59.2
1.5
58.9
1.7
12.6
0.2
17.0 0.2
25.2 + 0.2
VG-Relations: a spatial reasoning benchmark dataset that consists of 48 relations with nearly 24k
test. cases, with images from Visual Genome (Krishna et al. 2017), containing both spatial relations such as "in front of and "below" as well as actions such as "cating" and "standing on". Each image is paired with two descriptions, one being correct and one incorrect, similar to VT--Checklis. However, the main difference is that instead of a change in relation, the order of the two nouns is reversed, which results in aRb and bRa. The score calculations are the same as for VL-Checklist.
CLIPbind-r: a synthetic CLEVR-inspired benchmark dataset (Johnson et al. 2017) that is created by tool for 3D modelling and rendering (Lewis et al. 2023) with 10k images for validation. Each image has two objects, chosen out of a set of three, and one relation, out of four possibilities, which results in 24 combinations of spatial relations. Next to the pair aRb, four hard negatives are created with a different relation S, a different object e and reverse ordering, namely d_ = (b,
Similar to the other tasks, this task looks at the highest similarity scores between the image and the spatial relation compositions, except that there are four distractors. With five candidates per image, random performance is 20%.
Results. Quantitative results in Table | show clear advantages for the hyperbolic model on the three benchmarks for the best-performing model size. On VT-CCecklist, the large model achieves the highest per- formance and for this model size the Euclidean CLIP scores near random chance (50%), while the hyperbolic architecture achieves 59.9% accuracy which is siguificantly higher. For CLIPbind-r the Euclidean model scores lower than random for all model sizes, which is similar to the behavior of some models evaluated in Lowis et al. (2023), but we notice that the hyperbolic model with the large size performs better than random. In fact, the CLIP trained by OpenAl (Radford et al., 2021), which has been pre-trained on 400M image-text pairs, has an accuracy of 26.8%, which is only slightly higher than the hyperbolic CLIPP trained on 12M image-text pairs. The gap reduces on VG-Relations but remains substantial at over 3% absolute improvement. For VL-Checklist and CLIPbind-r, the large hyperbolic model achieves the highest scores, where for VG-Relations the small model performs best. We can conclude that overall the large hyperbolic model provides the most consistent results, since it does perform better than random on two tasks.
Analysis. To gain further insights into which spatial properties are better preserved in hyperbolic vision- language models, we have performed additional qualitative and quantitative analyses. In Figure 2 we show a few samples from the VL.-Checkistt datasct, indicating the correct/ /incorrect relation by color. Euclidean CLIP struggles with more complex relations such as "on") "on"/""undrr and in "between") ""nearr"""erry"" compared to the hyperbolic alternative. For spatial relations that are contextual, such as "near" 'ffrr from", both models still struggle, as such cases require explicit knowledge about the real-world size of the corresponding objects.
Since the VG- TG-Relations benchmark uses spatial prepositions and action verbs, we analyze performance per relation in Figure 3 focusing on frequently occurring spatial terms. For this set, we use the relations and actions with spatial relations with at least 20 occurrences in the datasct. For a few relations such as "at", "above", "sitting at", and "lying on", Euclidean CLIP is able to compete with hyperbolic CLIP. For many other relations however, large differences occur, especially for "standing on", "sitting on", and "looking at..
8

---

looking at Published in Transactions on Machine Learning Research (07 (07/202) 2024)
lemon on under food (a)
road in between nearby cone
grass near far from runway
(b)
(c)
Figure Success and failure cases hyperbolic spatial reasoning. We show three examples from the VL- Ch-Checklist spatial split, each with image and two relational descriptions: correct (green) and incorrect (red), e.g., lemon on food (correct) and lemon under food (incorrect) in example (a). For samples (a) and (b) the hyperbolic model assigns higher similarity to the correct relations, while the Euclidean embeddings score the incorrect relation higher. However, in (c) both models fail because of incorrectly preferring "grass far from runway" over the true "grass runway" description, likely due to the high contextual nature of the relation il requires explicit knowledge about the size of the object and understanding about the distance between the photographer and the objects in the
above
under
around
right
Eacidean CUP
to the left
behind
standing
below
standing
beneath
16
sitting on
in
sitting
in frant of
sitting
on top of on the side of on
lying
next
near
Figure 3: Hyperbolic CLIP for spatial relations with more than 20 samples. Hyperbolic and Euclidean CLIP are competitive for number of spatial relations, but hyperbolic CLIP performs much better for others, highlighting more robust and diverse spatial awareness in hyperbolic CLIP.
spatial relations. In this plot, the difference in performance is shown
Overall, hyperbolic CLIP is more robust to the different spatial relations and provides better option across all settings.
Models that perform well on spatial awareness tasks are useful for many applications, since it requires an understanding of order and compositionality. For real-world examples, this could be e.g., systems assisting visually impaired people robots in their interactions with people or in industrial environments.

---

Published in Transactions on Machine Learning Research (07/2024)
4.2 Hyperbolic vision-language models are better in dealing with ambiguity
Setup. Where deep networks excel when given clearly defined semantic labels, they struggle to deal with ambiguity (Kim et al.] 2023). Ambiguity can take many forms, from disambiguating words to dealing with data that is ambiguous and playful by design, such as memes. Here we investigate which embodding space is more suited for dealing with ambiguity. We study three benchmarks:
Propaganda Memes: This benchmark dataset contains 950 ambiguous image-text memes span- ning 22 propaganda techniques, including whataboutism, obfuscation, and glittering generalities (Dimitrov et al. 2021). Each meme comes with an extracted textual description and one or more cat- egory labels. The task is multi-label classification with models ranking similarity between prompted label embeddings of the form "this is propaganda about [class label]" and the meme embedding. For the hyperbolic model, we compute the similarity score by the Lorentzian inner product given by Equation ! between the meme embedding and the text embedding of the prompted class names (E memer ET) with three variants for the meme embedding:
1. text-only E E E memetext
2. image-only Emeem E Emmemiiagee
3. averaged embeddings E Ememe = (E + E mmmmmemngge
A fourth setup is analyzed where the scores of the text-only and image-only embeddings are averaged:
4. averaged similarity scores ((Einemiimage E (Ememetext c)/
The Euclidean model uses the cosine similarity on the same variations. We evaluate with the weighted mcan Average Precision described in (Wang et al. 2023).
My Reaction When: This benchmark dataset has 50K video-sentence pairs from social media depicting physical/ /mootooal reactions to textual captions inchuding a high ambiguity (Song & So- leymani, 2019). Videos are GIFs representing responses to the paired sentences. We evaluate on video-text and text-video retrieval using widely used mean pooling to aggregate per-frame CLIP embeddings into single video representation (Luo et al. 2022 For hyperbolic CLIP, video-text similarity is the Lorentzian inner product between mean pooled video and sentence embeddings, (E : e. Euclidean CLIP uses cosine similarity. Evaluation is via recall@k (top-k accuracy), checking if ground truth pairs are retrieved in closest neighborhoods. This tests semantic matching between ambignous pairs of reactions and descriptive captions.
Visual Word Sense Disambiguation: This benchmark evaluates associating ambiguous words like "mouse", with intended meanings from contextual images (Raganato et al.) 2023). Each case has a target word and candidate images, one of which matches its sense. Words are provided alone or with 1-2 word phrases for minimal context, e.g. "andromeda" alone vs "andromeda tree". Models rank image similarity to the word/phrase to identify the intended visual depiction among distractors. We compute hyperbolic CLIP's Lorentzian inner product and Euclidean CLIP's cosine similarity, following the setup in (Raganato et al. 2023). We evaluate the model on the full dataset that consists of 13332 samples. Evaluation uses Mean Reciprocal Rank at 5 and 10 (MRR&5, MRR@ 10) plus Hit Rate at 1 (HIT@1) to measure disambiguation capabilities given varying levels of contextual grounding.
The Propaganda Memes and My Reaction When benchmark datasets involve ambiguity in aligning textual captions with corresponding visuals. Memes combine images and text for rhetorical effect, with complex connotations that obscure intended meanings. Likewise, associating social media posts to emotive animated GIFs relies on interpreting muanced contextual implications. On the contrary, the Visual Word Sense Dis- ambiguation benchmark directly evaluates resolution of ambignous words through selection among depictive image candidates. Rather than complex compositional understanding between modalities, the core challenge is isolating the intended sense of polysemous concepts given limited grounding context.
While ambiguity types vary, all tasks require models to handle uncertainty in cross-modal correlations: so- ciopolitical, affective, or lexical. They measure how well languaee models leverage context to illuminate intended meanings amidst ambiguity spanning text, static visuals, and video.
10

---

Published in Transactions on Machine Learning Research (07 (224))
Table 2: Hyperbolic vision-language models can deal better with ambiguity (1), as demonstrated on the Propaganda memes benchmark. Regardless of whether we investigate text only, image only, averaging image- text features or predictions, hyperbolic embeddings provide vastly better results. The results are for the ViT-L/16 backbone and for all settings the improvement by the hyperbolic model is statistically significant.
Model Euclidean CLIP
Image only
Text only
Avg image-text feats
Avg image- preds 33.9 1 0.8
36.3 1 0.9
33.5 1 0.8
33.9 1 0.8
Hyperbolic CLIP
37.7 0.9t
34.2 = 0.8
35.9 + 0.9
36.0 + 0.9
Ground Truth:
Ground Truth: Nama calling og Smaars: Transfor
Appeal (earirojudios,
Mon Essentis!
Block and who
DiieetrDen BuuggealionMMiinniaa
ALONE
CONTROL
Language
Hyperbolic CLIP 'Smwann, "Treslerr, Doube"
GLISSAITE.
Hyperbolic CUP:
NOW SHUT THE FUCK MP OR ITLL
AAppoal feoninojudioee Repallort, Doute"
Exaggoral on saloon,
BESTER
Euclidean
FRY YOUR
Transler,
ASSES,
Euclidean CUP:: Waepetton Doubf; Mppooal to authorty.. Ilandwagon"
Bandwagorf
(a)
(b)
Figure 4: Propaganda Memes samples. We show examples from the Propaganda Memes benchmark dataset with their ground truth labels and the predictions of both models where the meme embedding is the average of the image and text embedding. In example (a), the hyperbolic CLIP model is able to predict two out of four ground truth labels correctly whereas the Euclidean model does not predict any of those. In example (b), the hyperbolic is able predict two out of three categories correctly, including smears, where its Euclidean counterpart has only one correct prediction.
Results. In Table we show the results on the Propaganda Memes benchmark. We see that the hyperbolic embeddings perform significantly better than their Euclidean counterpart. Dimitrov et al. (2021) have indicated that smears, loaded language and name propaganda memes, with smears covering more than half of the dataset, where they argue that this dataset is good representation of memes that can be found on social media. They define smears as an effort to damage call into question somcone's reputation, by propounding negative propaganda. Examples of memes are given in Figure A
are the most common meme types in
Table 3 shows text-video retrieval results on the My Reaction When validation (1K) and test (5K) sets, Scaling up the dataset increases the retrieval difficulty, as observed from the decrcased performance on the test set. Still, hyperbolic CLIP consistently surpasses Euclidean variant, particularly for top few (small k) retrieved results.
Table 4 shows Visual Word Sense Disambiguation results, with hyperbolic CLIP outperforming Euclidean both phrase and word inputs. Notably, gains persist even when only the ambiguous word is provided without any disambiguating context. This suggests inherent benefits in reasoning about polysemous lexical ambiguity. However, strong overall performance remains elusive - vision-language models struggle to resolve fine-grained distinctions absent in pre-training data like RedCaps. In summary, while hyperbolic space better handles ambiguity, truly contending with granular real-world ambiguity requires acquiring specialist visual knowledge of distinctions like mouso-animal versus mouse-computer. Nevertheloss, the consistent improvements validate geometric advantages in multinodal ambiguity.
Analysis. One potential explanation for the superior performance of hyperbolic CLIP on ambiguity tasks is its inherent hicrarchical embedding space. Ambiguity introduces additional granularity in recognition tasks by adding an instance-level layer (Chen et al. 2022). However, hyperbolic vision-language models are trained on the same image-text pairs as Euclidean models rather than explicit hierarchies. Recent work showed limitations visual hicrarchy encoding (Shen et al.] 2023), but the textual encoder capturing task semantics more relevant here. Before relating ambiguity and hierarchy, we first verify hyperbolic text embeddings exhibit. stronger latent hierarchical properties.
11

---

Published in Transactions on Machine Learning Research (07/2024)
Table 3: Hyperbolic vision-language models can deal better with ambiguity (II), as demonstrated on the My Reaction When benchmark. Also when dealing with video inputs and emotional ambiguity, hyperbolic embeddings are preferred. These results are for the ViT-L/16 backbone. Most of the results are slatistically significaut except for the r01 for text-video retrieval on the validation set and r01 and r@5 for video-text retrieval on the test set. Note that the results are reported in one decimal due to space constraints of the table. The r@1 for text-video retrieval on the test-set does not have the exact same decimals but is in fact 0.100 + 0.044 for the Euclidean model and 0.120 + 0.049 for the hyperbolic model and with that statistically significant with < 0.025, indicated with 4.
(1k-
Model
Text- Video Retrieval
Video Text Retrieval
rSal
1935
19:20
rW1
1835
r5110
rS320
Euclidean CLIP
0.2 + 0.1
0.8 + 0.3
1.7 + 0.4
3.3 = 0.5
0.2 0.1
0,6 = 0.2
1.6 : 0.4
2.9 = 0.5
Hyperbolic CLIP
0.2 + 0.1
1.3 0.4
2.4 0.5
4.2
0.6
0.5 + 0.2
1.2 + 0.4
2.4 + 0.5)
4.2 0.6
Test-sel (5k)
r991
r95
ri110
r0:0
rwl
r&55
r010
r$020
Enclidean CLIP 0.1 = 0.0
0.3 # 0.1
0.5 = 0.1
0.9 = 0.1
0.1 # 0.1
0.5 = 0.1
0.7 = 0.1
1.2 = 0.1
Hyperbolic CLIP 0.1 1 0.0
0.5 0.1*
1.0
0.1*
1.4 1 0.2
0.1 1 0.1
0.5 0.1
0.9 1 0.1
1.5 1 0.2
Table 4: Hyperbolic vision-language models can deal better with ambiguity (III), as demonstrated on the Visual Word Sense Disambiguation benchmark. The hyperbolic CLIP outperforms its Euclidean variant in ranking images similar the ambiguous word or phrase. L, B, and S refer to the ViT-L/16, ViT-B/16 and ViT-S/16 models and for all settings the improvement by the hyperbolic model is slatistically significant, indicated with t. The large backbone performs best for all setups and metrics.
Model
Target
MRRAS
MRRA10
B
S
I
Euclideso CLIP
phrase
47.7 + 0.4
19.2 +
0.5
49.1 + 0.5
61.2 + 0.3
62.4
+ 0.4
62.3 : 0.3
63.3 :: 0.3
61.4 # 0.3
61.3 1 0.3
Hyperbolic CLIP
phrase
0.5
1 0.4
49.4
61.6 0.4
62.3
0.3
62.6
0.3
63.7
613 0.31
0.3
B
8
B
S
B
L
Euclidesn CLIP
word
39.8 + 0.4
40.8 +
0.4
40.9 + 0.4
53.4 # 0.4
51.1 # 0.3
51.0 # 0.3
56.3 + 0.3
67.0 # 0.3
37.0 0.3
Hyperbolic CLIP
word
41.1 0.4
1 0.4
41.9 0.4
511
0.4
519 0.4
55.2
0.4
57.1 0.3
57.7 0.3
58.0 0.3
To quantify textual hierarchy encoding, we leverage label trees from six vision datasets: CIFAR-100, Animals with Attributes 2 (AWA2) (Xian et al. 2019), , PASCAL-VOC (Everingham et al. 2010), UCF (Soomro al. 2012), Kinetics (Kay et al.) 2017), and ActivityNet (Heilbron et al. 2015). The first three are provided by the creators of the datasets, while the latter three are taken from Long et al. (0020) and Gulshad et al. (2023). Leaf nodes are encoded as "this is a photo of a [class name]" sentences using Euclidean and hyperbolic CLIP text encoders. We evaluate whether hierarchy distance corresponds with embedding distance, where hicrarchy distance between classes is defined by tree levels to the common ancestor. We use the Pearson correlation coefficient to measure whether a linearity between the embedding distances and hierarchy distances is present.
Table 5 shows Pearson correlation between embedding and hicrarchy distances. Hyperbolic CLIP exhibits higher correlation than Fuclidean variants on five of six datasets, demonstrating improved latent encoding of hierarchical relationships. To validate that these gains span beyond the lowest hierarchy level, we filter leaf nodes from the NetivityNet taxonomy and remeasure correlations. Even within higher layers, hyperbolic embeddings show greater alignment with tree distances (Euclidean: 0.2434, Hyperbolic: 0.2823).
Models that are able to deal with unelear and subjective situations are required in practice. It requires models to learn how to incorporate contexts into the decision-making process. A practical application for this task could be monitoring of posts of users on online platforms. A model should be able to interpret all sorts of memes and other posts based on the visible information in the post itself and well-known contextual information. This task is still even difficult for humans and therefore far from solvable for models, but these benchnarks are a required initial step.
In summary, hyperbolic text encoders intrinsically structure semantic concepts more hicrarchically, which is an advantage originating from the underlying geometry. We hypothesize these latent textual hicrarchies aid in resolving ambiguity by providing framework to structurally represent uncertain, fine-grained distinctions.
12

---

Published in Transactions on Machine Learning Research (07/2024)
Table 5: Hyperbolic vision-language models are more hierarchically structured. We embed six hierarchies in Euclidean and hyperbolic embeddingg through prompting and find that hyperbolic embeddingg betler correlate with the underlying structure of the hierarchies. While trained on exactly the same data, the hierarchical nature of hyperbolic geomelry allows for better lalent hierarchical structuring of semantics.
Model
PASCAL-VOC
AWA2
UCF101
CIFAR-100
Kinctics
ActivityNet
Hierarchy levels
4
6
3
3
3
3
Number of concepts
55
77
125
127
239
244
Euclidean CLIP
0.3689
0.2313
0.1200
0.2683
0.1675
0.1450
Hyperbolic CLIP
0.4562
0.2861
0.1179
0.3646
0.2190
0.1830
However, directly confirming the connection between hierarchy and improved ambiguity handling remains an open question for future work and would require extensive further analysis into geometric mechanisms granting these advantages.
4.3 Hyperbolic vision-language models are powerful out-of-distribution discriminators
Lastly, we evaluate the out-of-distribution discrimination potential of hyperbolic imago-text embed- dings. The ability to discriminate between in- and out-of-distribution samples is a requirement for real-world deployment. Here, we leverage the comprehensive out-of-distributee detection benchmark of (Yang et al.) h We evaluate three common out-of-distribution detection methods that can be directly applied as a post- prosespocesiing technique on pre-trained models:
Sctup.
MLS: The Maximum Logit Score (MLS) (Hendrycks et al. 2022 uses the maximum logit. Operating on logits provides compatibility across manifolds and robustinss to overconfidence in extreme multi- class scenarios. For hyperbolic CLIP, logits are computed via the Lorentzian inner product (Equation 1 between image embeddings and embeddings of prompted class names: "this is photo of a [class name]". Then the prediction and confidence score are calculated with max (EE, ET ,T,,) c) For the Euclidean CLIP, we get the maximum over the cosine similarity scores.
EBO: Energy-Bascd OOD (EBO) scoring applics an energy function to the logit outputs Liu et al. 2020b). Energy scores are theoretically aligned with the probability density of the in- puts and mitigate overconfidence. Like ML.S, this score is manifold agnostic and the logits are calculated in the same manner as for MLS. For the hyperbolic CLIP, the predictions are de- termined by max; (softmax( (Ej, ET E))) The confidence score is determined with the energy score and since this is bon-probabilistic, it be calculated with the logsumexp operator by T log Σx exp((E; ET c/T)), where the temperature hyperparameter. For the Euclidean CLIIP, get the same expressions by using the cosine similarity score.
The Mahalanobis Distance Score (MDS) fits the class-conditionas Gaussian distribution the penultimate layer features of the classifier and computes OOD scores using the Mahalanobis distance metric, which is only defined for Euclidean spaces (Lee el al. 2018). For valid comparison, hyperbolic embeddings are first projected back into the Euclidean geometry via the logarithmic map (Equation
MDS:
Following Lee et al. (2018). we compute the empirical class mean Pee and covariance E of the training samples. Then the confidence score (x) is defined by using the Mahalanobis distance between the image embedding Ej and the closest class-con condiitoniii Gaussian distribution, i.c.,
M(E,) = max - EE, P.) $ (Eạ -(E--Pa) -pa))
(11)
Since MDS operates on the penultimate layer features, similar to the unimodal scenario, we use the image embeddings as an imput for the confidence scoring function. For the predictions, we calculate the maximum over the logits which are calculated by the cosine similarity score. Since we chose a specific point of projection from the hyperboloid back to the tangent space, all computations happen in the Euclidean space and well-defined for the MDS calculation.
13

---

Published in Transactions on Machine Learning Research (07/2024)
Table 6: Out-of-distribution detection with the Im0geNet220,, and ImageNet-Ik by using the AUROC score. For all methods, the improvement by the hyperbolic model is statistically significant with < 0.025, indicated with f.
techniques MLS, EBO, and MDS on CIFAR-10.
CIFAR-10
InaggeNal-200
InageNNt-11K
InuggeNet-200 Booo
InuggeNet-1K Bood
Method
Model
Near OOD
Tar OOD
Noar OOD
Far OOD
Near OOD
Far OOD
Near OOD
Far OOD
Noar OOD
OOD
MISS
Eudlidan CLIP
84.6 + 0.2
902 + 0.1
79.3 02
87.7 + 0.2
63.7 0.2
745 + 02
68.6 + 0.2
79.3 + 0.1
002 0.2
71.6 + 0.2
Hy pestllili OLIP
86.4 = 0.24
98.7 = 0.1
80.3 0.21
88.8 = 0.2
60.0 = 0.2
76.6 # 0.2
09.7 + 0.2
80.7 = 0.1-
63.0 # 0.2
73.2 * 0.21
Fuchidoan CTFF
12.5
132
Hypettollic CLI
75.1 0.24
75.1 L 0.1
00.3 0.3
51.3 0,21
61.2 0.2
50.0 0.2
15.6 1 0.2
66.8 0.2
09.4
1 0.2
58.5 0.0
MIS
Euchddon CLI
81.4 0.2
0044
76.3100 02
861 1.11
06.8
0.2
SL.4 01
18.4 1.2
102
a 0.2
50d - 02
06.0 1 0.1.1
Hypetbolli CLIP
0,24
98.2 0.1
82.2 0.2
0.1
72.0 0.2"
0.1"
45.7 0.2
67.0 0.2
53.5 1 0.2
74.9 0.11
For these three out-of-distribution techniques, we use the following settings for evaluation:
Out-of-distribution type: we consider two out-of-distribe distribbinn detection scenarios. 1) Standard out-of-distribution detection considers distribution shifts in semantic content. 2) Full Spectrum out-of-distribution detoction additionally takes the covariate shift into account, which in this sctup specifically addresses image corruptions, style change, and resampling bias.
Metrics: We use four metries: 1) false positive rate at 95% true negative rate (FPR@95). 2) the area under the receiver operating characteristic (AUROC) which indicates the probability that the detector correctly soparates in- and out-of-distribution samples, and 3) area under the Precision- Recall curve (AUPR) where either ID images are specified as posilive (AUPR-in) or 4) out-of- distribution images are seen as positive (AUPR-out) Yuu & Aizawa 2019
Datasets: As In-Distribution (ID) datasets, we use CIFAR-10, ImageNet200,, and ImageNet- Ik. The out-of-distribution datasets are divided into two categories: near-OOD (hard-OOD) and far- OOD (easy-OOD), based on image semantics and empirical difficulty. For CIFAR-10, we use CIFAR- 100 and TinyImageNet. (Le & Yang) as the near-OOD datasets and MNIST (LeCun et al., 1998). SVHN (Netzer et al.) 2011), Textures (Cimpoi et al. 2014), and Places365 (Zhou et al. 2017 as far-OOD datasets. For ImageNet200 and ImageNet we use the following datasets. For near- OOD, we use SSB-hard (Vaze et al. 2022), a dataset composed of images from ImageNet-21K and NINCO by Bitterwolf ct al. (2023). For far-OOD, we iNaturalist (Horn et al. 2018), Textures (Cimpoi et al. 2014) and Openlmage-O (Wang et al. 2022).
For more details regarding the evaluation setup, we refer to Zhang et al. 2023
Results. Table G shows AUROC scores for the three OOD methods on near- and far-OOD data. Across all techniques, hyperbolic embeddings cousistently improve OOD delection over Euclidean counterparts in both standard and full spectrum settings. The Mahalanobis Distance Score (MDS) performs best overall, achieving a mean score of 72.0% and 89.6% AUROC for near- and far-OOD with hyperbolic embeddings for ImageNet-1K. These gains are significant compared to 66.8% and 81.4% for Euclidean MDS. Interestingly, this improvement comes despite projecting hyperbolic embeddings back into the Euclidean space required for computing Mahalanobis distance. The robustness to this transformation indicates the hyperbolic geometry provides intrinsic distributionndiisciiinaall
properties beyond the distance metric alone.
Table breaks down ImageNet results by dataset for Mahalanobis scoring, providing per-distribution FPR95, AUROC AUPR-in, and AUPR-out. Significant OOD detection improvements from hyperbolic over Euclidean are observed consistently. These granular outcomes support the aggregate near/lar-OOD con- clusions from Table G hyperbolic geometry is better in oft-of-distribution discrimination. The substantial gains demonstrate real-world viability for deploying hyperbolic vision-language models in open-domain ap- plications.
Table presents a comparison of the large, base and small backbones on AUROC scores on ImageNet- 1k for all 5 OOD datascts. Where the hyperbolic CLIP outperforms the Euclidean CLIP on the base and large backbones, the Euclidean CLIP performs better for the small backbone. Overall the hyperbolic CLIP base model performs best out of all models.
14

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Table 7: Out-of-distribution detection with the MDS method on ImageNet-Ik 1k with SSB-Hard and NINCO out-of-distribution datasets and INaturalist, Textures and Openimage-O far out-of-distribution datasets. Across different near and lar and out-of-distribution datasets, hyperbolic CLIP is more robust than its Euclidean counterpart. For all OOD-dalasets the improvement by the hyperbolic model is statistically significant with 0.025, indicated with І.
ImageNet-Ik
Dataset
Model
FPR95
AUROC1 f
AUPR-in 1
AUPR-out
Euclidean CLIP
85.5 + 0.3
62.6 = 0.2
61.4 + 0.3
61.4 + 0.2
SSB-hard
Hyperbolic CLIP
80.8 + 0.3
66.8 0.2
66.0 + 0.2
66.1 + 0.2
Euclidean CLIP
75.5 0.7
71.0 =
94.8 0.1
21.3 0.4
NINCO
Hyperbolic CLIP
70.2 + 1.11
77.3 0..3
96.0 + 0.11
30.4 = 0.61
Enclidean CLIP
62.8 + 0.5
71.5 = 0.2
92.7 0.1
27.6 + 0.3
iNaturalist
Hyperbolic CLIP
45.6 + 0.6
84.4 = 0.2
96.2 + 0.1*
46.2 + 0.5+
Euclidean CLIP
41.7 + 1.1
90.3 = 0.2
98.7 + 0.0
60.1 + 0.7
Textures
Hlyperbolic CLIP
26.9 + 0.91
94.4 0.2
99.3 + 0.0
73.7 + 0.6
Enclidean CLIP
52.7 + 0.6
82.4 = 0.2
93.2 = 0.1
58.1 = 0.4
Openimago-O
Hyperbolic CLIP
38.5 + 0.61
89.9 0.1
96.2 + 0.1
74.5 + 0.41
Table 8: Out-of-distribution detection ImageNet-1k with SSB-Hard and NINCO as near out-of- dist ristribution datasets and iNaturalist, Textures and Openimage-O as far out-of-distribttion datasets. Results are reported on small, base and large VIT backbones. Across different near and far and out-of-distributin datasets, hyperbolic ViTs are more robust than their Euclidean counterparts for the base and large models. These results are statistically significant with < 0.025, indicated with t.
ImageNNtt1lk
Dataset
Model
AUROC f
S
B
L
Euclideau CLIP
64.7 1 0.2
65.2 1 0.2
62.6 0.2
SSB-hard
Hyperbolic CLIP
62.3 +
68.0
0.2
66.8 + 0.2
Euclidean CLIP
75.1 1 0.4
73.9 1 0.3
71.0 0.3
NINCO
Hyperbolic CLIP
72.7 + 0.4
78.0
0.3
77.3 + 0.3
Euclidean CLIP
88.8 + 0.2
83.3 + 0.2
71.5 + 0.2
iNaturalist
Hyperbolic CLIP
83.7 + 0.2
90.2 + 0.1
84.4 + 0.2
Euclidran CLIP
94.5 4 0.1
95.0 + 0.1
90.3 0.2
Textures
Hyperbolic CLIP
91.8 +
97.3
0.1
94.4 # 0.2
Euclideau CLIP
89.4 1
86.9 1
0.2
Openimage-0 0
Hyperbolic CLIP
85.8 + 0.2
92.2 0.1
89.9 # 0.1
Analysis. To investigate the OOD gains, Figure plots distribution shifts for Mahalanobis scoring on ncar-OOD NINCO (left) and far-OOD iNaturalist (right). ID scores (orange) separate further from OOD (blue) with hyperbolic embeddings. The peaks flatten and variance increases, reducing overlap.
Figure 6 shows per-class FPR95 difference between hyperbolic and Euclidean CLIP embeddings on the NINCO near-OOD dataset. Images in NINCo's 64 fine-grained categories are visually similar to some ImageNet classes but belong to different distributions. According to Bitterwolf et al. (022)), images from the NINCO dataset from for instance the categories chicken quesadilla, spaghetti bolognese, and donut are most confusing to ImageNet classes burrito, carbonara, and bagel respectively. Remarkably, hyperbolic embeddings improve OOD detection over Euclidean on 54 out of 64 NINCO categories. Over half exhibit substantial gains above 10% FPR95. This granular analysis confirms and provides insight into the aggregated near-OOD results. There are 9 categories for which the Euclidean CLIP performs 1-6% better than the hyperbolic CLIP, however the absolute performance of the models for these calegories varies heavily. 7 out of of these categories are classes from the Species dataset. We did not find a strong correlation between specific charactoristics in properties of these classes and their images. Looking at real-world applications, OOD detection is useful for many such as finding unidentified objects in forensics data or finding anomalies in medical imaging data.
15

---

Published in Transactions on Machine Learning Research (07 (07/2024) 2024)
0.005
0.004
0.003
Frequency
0.003
0.002
0.002
0.001
0.001
0.000
0.000
1400
-1200
800
-600
-400
-200
1400
-1200
-1000
-800
-400
-200
Mahalanobis
Mahalanools
(a) Near-OOD: Ninco(Blug ve [orrange) for Euclidran CLIP
(h) Far-OOD: (Natrralist(Bluc) (( for Euclidran CLIP
0.007
0.006
0.004
0.005
Frequency
0.003
0.004
0.003
0.002
0.002
0.001
0.000
1000
-800
500
-400
-200
0.000
-900
-800
-700
-600
500
400
-300
-200
Mahalanobis score
Mahalancois
(c) Near-OOD: Ninco(Blue) va DD(orange) for hyperbolic CLIP (d) Far-ODD: (Naturalist(Blue) va ID(orauge) for hyperbolic CLIP
Figure Out-of-distributton detection. This distribution plot presents the Mahalanobis
(x-axis) and
frequency (y-axis) for Euclidean CLIP (top) and hyperbolic CLIP (bottom). The ID and OOD samples show less overlap the plots for hyperbolic CLIP, which indicates better OOD detection performance.
a FPR95
fine e ceceeeee
andor
ail oi on
seee hon conernane in aomein the diimen
te auscots eritari maimar porati gart of mit sopie dnfiicinale cremm gralee deea
palt
gii
sel
scissors
gonus
melisr high heels
aedlaa no coageeee
french fries pencartiin
mbira
pllosus
takes
secoon rubresti cup to eet
dood
a
aoooooooo maydeds
srrgin cecteeie (i suuig sndosoo
ila sout
sor vous uouuozzeeede
tumarn cotteee
arana
B smeudd
Si Ponnnng srao
sdoiinn
zagiraa gra
edieg
playcoss
PPOM Apduaa
sumppzen
NINCO Category
Figure 6: Out-of-distribution detection results NINCO per-class: FPR95 difference between hyperbolic Euclidean CLIP embeddings. A positive means that hyperbolic embeddings are better for the considered category.
5 Conclusion
Vision-language models have gained lot of traction due to their broad applicability and strong performance established down-stream tasks. In vision-language models, the de facto shared embedding space is Eu- clidean. Recent work shows that this default choice is not the only option and hyperbolic geometry provides a viable alternative with strong performance on often studied benchmarks such as retrieval and zero-shot recognition. this work, we broaden the scope of the potential of hyperbolic vision-language models and demonstrate their power on their important but under-studied goals.
We find that hyperbolic languael models obtain strong spatial reasoning results compared their Fuclidean counterpart, confirming higher intrinsic awareness of visual compositionality. The consistent im-
16

---

Published in Transactions on Machine Learning Research (07/2024)
provements over the Euclidean bascline indicates room for further advances, for instance, by developing specialized fine-tuning strategies targeting compositional tasks. Beyond explicit spatial relationships, analo- gous ideas may generalize to other latent forms of visual and linguistic compositionality. An investigation of the inductive biases introduced by the hyperbolic geometry remains an intriguing direction for future work. Additionally, hyperbolic models exhibit enhanced reasoning amid ambiguity in text, images, and video. We argued a possible connection between improved ambiguity handling and the hierarchical structure of the em- bedding space, but further in-depth analysis is required to confirm this. Lastly, hyperbolic vision-language models outperform Euclidean vision-language models in tuttof-ddisributioo detection, which complements existing evidence of this advantage in unimodal settings. However majority of the OOD detection techniques require specific fine-tuning and this remains an open challenge for future work.
While hyperbolic vision- lasguae models provide advantages, they also come with challenges. First, hyper- bolic operations increase the computational cost of a neural network due to additional nonlinear operations. Compared to the bascline Euclidean model, hyperbolic image-text models include an additional exponen- tial map, while distance calculations are performed with a Lorentzian inner product instead of Euclidean distance. This causes, however, minimal increase for the running time of our experiments (evaluated on VL-Checklist taking 114.8 seconds for hyperbolic CLIP evaluation and 113.6 seconds for Euclidean CLIP). This is because the main computational load is given by the image and text encoders, which are identical for both models. While a minor increase for evaluation, some additional run time is to be expected during pre-training or fine-uning due to the use of an additional entailment loss. Current models focus on hyper- bolic embeddings only. If transformer layers would be replaced with hyperbolic alternatives as well, could increase the computational cost drastically and different optimization techniques should be performed to overcome this issue. This is active area of rescarch studiod by Lou et al. (2020), [Shimizu et al. (2021) and van Spengler et al. (2023). Another challenge is the numerical stability in the training process that comes from exponential volume growth (Mishne et al., (023). Where the Poincaré model faces difficulties when dealing with small numbers, the Lorentzian model suffers from dealing with large numbers. These challenges have to be taken into account when developing new models and training them end-to-end.
Overall, to make more progress in this field, fine-tuning hyperbolic rision-language models on more tasks and exposing such models to more data for pro-training are promising next steps. This could be done by exploring new constraints for the fine-tuning process, for example through new loss functions to complement the contrastive and entailment losses, or even different architectural modifications such as the use of hyperbolic layers. A deeper study into the behavior of different model capacity sizes would also be an interesting direction for future rescarch. While the large model gives the most stable performances across all tasks, an in-depth analysis of the structures of the embedding spaces of the different models might provide more insights in their bohavior. Due to its potential, we hope that this work can serve as a starting point for more hyperbolic rision-language related research in both the hyperbolic learning and vision-language communities.
References
Jcan-Baptiste Alayrac, Jeff Donahuc, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for few-shot learning. In NewrIPS, 2022.
Mina Ghadimi Atigh, Martin Ressel, and Pascal Mettes. Hyperbolic busemann learning with ideal prototypes. Tn NewrIPS, 2021.
Mina Ghadimi Atigh, Julian Schoop, Erman Acar, Nane van Noord, and Pascal Mettes. Hyperbolic image segmentation. In CVPR, 2022.
Julian Bitterwolf, Maximilian Mueleer, and Matthias Hein. In or out? fixing imagenet ofitt--disttibutiinn detection evaluation. In TCML, 2023.
17

---

Published in Transactions on Machine Learning Research (07/2024)
Avishek Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and William L. Hamilton. Latent variable modelling with hyperbolic normalizing flows. In ICML, 2020.
James W Cannon, William J Floyd, Richard Kenyon, Walter R Parry, et al. Hyperbolic geometry. Flavors of geometry, (1(59-115):2, 1997.
Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. Low-dimensional hyperbolic knowledge graph embeddings. In ACL, 2020.
Jingzhou Chen, Peng Wang, Jian Liu, and Yuntao Qian. Label relation graphs enhanced hierarchical residual
network for hicrarchical multi-gramularity classification. In CVPR, 2022.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people. In NeurIPS, Datasets and Benchmarks, 2021.
Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedan- tam. Hyperbolic image-lext represenlations. In ICML, 2023.
Ankit Dhall, Anastasia Makarova, Oclavian Ganea, Dario Pavllo, Michael Greeff, and Andreas Krause. Hierarchical image classification using entailment cone embeddings. In CVPR, 2020.
Bhuwan Dhingra, Christopher J. Shallue, Mohammad Norouzi, Andrew M. Dai, and George E. Dahl. Em- bedding text in hyperbolic spaces. In TertCraph 2018.
Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. Detecting propaganda techniques in memes. In ACL/IJCNLP, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan V. Oseledets. Hyperbolic vision transformers: Combining improvements in metric learning. In CVPR, 2022.
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. IJCV, 88(2), 2010.
Alex Fang, Gabriel Tharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). In ICML, 2022.
Luca Franco, Paolo Mandica, Bharti Munjal, and Fabio Galasso. Hyperbolic self-paced learning for self- supervised skeleton-based action representations. In ICLR, 2023.
Octavian-Eugen Ganca, Gary Bécigneul, and Thomas Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In ICML, 2018a.
Octavian-Eugen Ganea, Gary Béeigneul, and Thomas Hofmann. Hyperbolic neural networks. In NewrIPS, 2018b.
Zhi Gao, Yuwei Wu, Yunde Jia, and Mehrtash Harandi. Curvature generation in curved spaces for sew-shot learning. In ICCV, 2021.
Songwei Ge, Shlok Mishra, Simon Kornblith, Chun-Liang Li, and David Jacobs. Hyperbolic contrastive learning for visual representations beyond objects. Tn CVPR, 2023a.
18

---

Published in Transactions on Machine Learning Research (07/2024)
Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming- Hing-Hsuan Yang, Hartwig Adam, Laurent Itti, Balaji Takshminarayanan, and Jiaping Zhao. Improving zero-shot generalization and robustness of multi- modal models. In CVPR, 2023b.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finctuning of zero-shot vision models. In CVPR, 2023.
Sadaf Gulshad, Teng Long, and Nanne van Noord. Hierarchical explanations for video action recognition. In CVPR Workshops, 2023.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large- scale video benchmark for human activity understanding. In CVPR, 2015.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. In ICML, 2022.
Grant. Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
Sarah Ibrahimi, Xiaohang Sun, Pichao Wang, Amanmect Garg, Ashutosh Sanan, and Mohamed Omar. Audio-enhanced texi-to-video retrieval using lexi-conditioned feature alignment. In ICCV, 2023.
Weihao Junlin Han Lars Petersson Mchrtash Harandi Jie Hong, Pengfei Fang. Curved geometric networks for visual anomaly recognition. Tn TNNLS, 2023.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint crXic:1705.0690 2017.
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan V. Oseledets, and Victor S. Lempitsky. Hyperbolic image embeddings. In CVPR, 2020.
Dongwon Kim, Namyup Kim, and Suha Kwak. Improving cross-modal retrieval with set of diverse embed- dings. In CVPR, 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123(1), 2017.
Christopher Lang, Alexander Braun, Lars Schilliggman, and Abhinav Valada. On hyperbolic embeddings in object detection. In Björn Andres, Florian Bernard, Daniel Cremers, Simone Frintrop, Bastian Goldlücke, and Tvo Thrke (eds.), German Conference on Pattern Recognition, 2022.
Matt Le, Stephen Roller, Laetitia Papaxanthos, Douwe Kiela, and Maximilian Nickel. Inferring concept hicrarchies from text corpora via hyperbolic embeddings. In ACI, 2019.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-barr learning applied to docu- ment recognition. Proc. IEEE, 86(11), 1998.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of- distribution samples and adversarial attacks. In NewrIPS, 2018.
19

---

Published in Transactions on Machine Learning Research (07/2024)
Martha Lewis, Qinan Yu, Jack Merullo, and Ellie Pavlick. Does CLIP bind concepts? probing composition- ality in large image models. aixiv preprint orXin:2222. 10537, 2023.
Weixin Liang, Yuhui Zhang, Yougchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS, 2022.
Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. In NewrIPS, 2019.
Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong- Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. Hy- perbolic visual embedding learning for zero-shot recognition. In CVPR, 2020a.
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurIPS, 2020b.
Teng Long, Pascal Mettes, Heng Tao Shen, and Cees G. M. Snoek. Searching for actions on the hyperbole. In CVPR, 2020.
Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge J. Belongie, Ser-Nam Lim, and Christopher De Sa. Differentiating through the fréchet mean. In ICML, 2020.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning. Neurocomputing, 508, 2022.
Rongkai Ma, Pengfei Fang, Tom Drummond, and Mehrtash Harandi. Adaptive poincaré point to set distance for few-shot classification. In A AAAT, 2022.
Shu Manli, Nie Weili, Huang De-An, Yu Zhiding, Goldstein Tom, Anandkumar Anima, and Xiao Chaowei. Test-time prompt tuning for zero-shot generalization in vision-language models. In NewrIPS, 2022.
Joanna Materzynska, Antonio Torralba, and David Bau, Disentangling visual and written concepts in CLIP, In CVPR, 2022.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023.
Pascal Mettes, Mina Ghadimi Atigh, Martin Keller Keller-Ressel, Jeffrey Gu, and Serena Yeung. Hyperbolic deep learning computer vision: A survey. IJCV, 2024.
Leyla Mirvakhabova, Evgeny Frolov, Valentin Khrulkov, Tvan V. Oseledets, and Alexander Tuzhilin. Perfor- mance hyperbolic geometry models on top-n recommendation tasks. In RecSys, 2020.
Gal Mishne, Zhengehao Wan, Yusu Wang, and Sheng Yang. The numerical stability of hyperbolic represen- tation learning. In ICML, 2023.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped normal distri- bution on hyperbolic space for gradient-based learning. In ICML, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading dig- its in natural images with unsupervised feature learning. In NewrIPS Workshop on Deep Learning and Unsupervised Fealure Learning 2011, 2011.
Maximilian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. In NewrIPS, 2017.
Maximilian Nickel and Douwe Kiela. Learning contimuous hicrarchics in the lorentz model of hyperbolic geometry. In TCML, 2018.
Jishnu Jaykumar P, Kamalesh Palanisamy, Yu-Wei Chao, Xinya Du, and Yu Xiang. Proto-clip: Vision- language prototypical network for few-shot learning. arXiv preprint arXiv:2307.088 0300079. 2023.
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In ICCV, 2023.
20

---

Published in Transactions on Machine Learning Research (07/2024)
Wei Peng, Thomas Varanka, AbdeIrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyperbolic deep neural networks: A survey. TPAMI, 44(12), 2022.
Steven T. Piantadosi, Harry Tily, and Edward Gibson. The communicative function of ambiguity language. Cognition, 122(3), 2012.
Sarah Pratt, lan Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? customized prompts for zero-shot image classification. In ICCV, 2023.
generating
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Πlya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramosh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πya Sutskever. Learning transferable visual models from natural language supervision. In JCML, 2021.
Alessandro Raganato, lacer Calixto, Asahi Ushio, Jose Camacho- Camacho-Collados. and Mohammad Taher Pilehvar. SemEval-2023 task 1: Visual word sense disambiguation. In Atul Kr. Ojha, A. Seza Doğruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, and Elisa Sartori (eds.), SemBual, 2023.
Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Wallling around for performance: Visual classification with random words and broad concepts. In ICCV, 2023.
Frederic Sala, Christopher De Sa, Albert Gu, and Christopher Ré. Representation tradeoffs for hyperbolic embeddings. In ICML, 2018.
Benjamin Sanchez-Tennellngg Jennifer N. Wei, Brian K. Lee, Richard C. Gerkin, Alán Aspuru-Guzik, and Alexander B. Wiltschko. Machine learning for scent: Learning generalizable perceptual representations of small molecules. ar prepriat 191iv:1910. 10685, 2019.
Ethan Shen, Ali Farhadi, and Adilya Kusupati. Are "hierarchical" visual representations hierarchical? In NewrIPS, Symmetry and Geometry Workshop in Neural Representations, 2023.
Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. Hyperbolic neural networks In ICLR, 2021.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In CVPR, 2022.
Yale Song and Mohammad Soleymani. Polysemous visual-semantic embedding for cross-modal retrieval. In CVPR, 2019.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0/ 0400000, 2012.
Alexandru Tifrea, Gary Bécigneul, and Dctavian-Eugen Ganea. Poincare glove: Hyperbolic word embed- dings. In JCLR, 2019.
Max Spengler, Erwin Berkhout, and Pascal Mettes. Poincare resnet.. In ICCV, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In ICLR, 2022.
Fengjun Wang, Sarai Mizrachi, Moran Beladev, Guy Nadav, Gil Amsalem, Karen Lastmann Assaraf, and Hadas Harush Boker. Mumic multimodal embedding for multi-label image classification with tempered sigmoid. In AAAT, 2023.
21

---

Published in Transactions on Machine Learning Research (07/2024)
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In CVPR, 2022.
Liping Wang, Fenyu Hu, Shu Wu, and Liang Wang. Fully hyperbolic graph convolution network for recom- mendation. In CIKM, 2021.
Melanie Weber, Manzil Zaheer, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Robust large-margin learning in hyperbolic space, In NewrIPS, 2020.
Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning A comprehen- sive evaluation of the good, the bad and the ugly. TPAMI, (1))), 2019.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. Openood: Benchmarking generalized out-of-distribution detection. In NewrIPS, 2022.
Menglin Yang, Min Zhou, Zhihao Li, Jiahong Liu, Lujia Pan, Hui Xiong, and Irwin King. Hyperbolic graph neural networks: A review of methods and applications. arXiv preprint orXio:2202.138.0 2023.
Qing Yu and Kiyoharu Aizawa. Unsupervised ont-of-distribution detection by maximum classifier discrep- ancy. In ICCV, 2019.
Mert Yükkekgönüü, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In ICLR, 2023.
Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. Openood v1.5: Enhanced benchmark for out-of-distribution detection. arXiv preprint arXiv:2306.0988 09301, 2023.
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-fif free adaption of CLIP for few-shot classification. In ECCV, 2022.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint xiv::2207.0022 00221, 2022.
Bolci Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. TPAMI, 2017.
Yudong Zhu, Di Zhou, Jinghui Xiao, Xin Jiang, Xiao Chen, and Qun Liu. Hypertext: Endowing fasttext with hyperbolic geometry. In EMNLP, 2020.
22


Reviews:
[{"review_id": "Ec6V5rCu6h", "reviewer": "Reviewer_Grex", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Novelty: The exploration of spatial awareness, ambiguity handling, and distribution discrimination specifically within hyperbolic VL models is a significant new contribution to the field.\n* Experimental Rigor: The experiments seem well-designed, and the results support the authors' claims about the advantages of hyperbolic embeddings.\n* Clarity and Organization: The paper is clearly written, flows logically, and makes effective use of examples and figures to illustrate complex concepts.\n\nWeaknesses\n* Narrower Dataset Focus: The authors could broaden the types of datasets used to further validate the universality of their findings.\n* Unexplored Limitations: A deeper discussion of potential shortcomings of hyperbolic embeddings (e.g., computational costs or challenges in fine-tuning) would strengthen the work.\n* Real-World Use Cases: Providing more concrete examples of real-world scenarios where the highlighted properties of hyperbolic VL models translate into tangible benefits would bolster the paper's impact.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper offers a compelling analysis of the advantages of using hyperbolic embeddings within the context of vision-language (VL) models. While researchers have touched on the benefits of hyperbolic embeddings, this work goes a step further by focusing specifically on large-scale VL settings.  The authors uncover several key properties that showcase the unique advantages of hyperbolic embeddings.\n\nFirstly, the paper demonstrates that VL models using hyperbolic embeddings develop an intrinsic spatial awareness that models based on traditional Euclidean embeddings do not possess. This translates to a marked improvement in tasks that demand a nuanced understanding of spatial relations between objects or concepts. Secondly, compared to their Euclidean counterparts, hyperbolic VL models are more adept at handling ambiguity present in both language and visual modalities. This likely stems from the inherent ability of hyperbolic space to accommodate hierarchical structures and represent multiple interpretations simultaneously.  Lastly, the authors show that hyperbolic VL models excel at distinguishing between differing distributions of data, a characteristic that could pave the way for greater robustness in out-of-distribution generalization and the detection of anomalies in real-world applications.\n\nCollectively, these findings shed new light on the reasons behind the performance gains observed when hyperbolic embeddings are employed in VL models, extending the discussion beyond simple metrics and towards a broader potential for impact.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Computational Cost Analysis: Providing an analysis of the computational complexity of the proposed hyperbolic VL models in comparison to their Euclidean equivalents is essential. If a significant increase in cost exists, suggesting potential optimization techniques or outlining research directions to address this would be crucial for the practical adoption of these methods.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZuLziFJgA6", "reviewer": "Reviewer_KqHF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe work sets out to measure quantities of broad interest to the community: what are the benefits of geometric spaces other than Euclidean for zero-shot evaluation, and how can we understand any observed performance improvement?\n\nThe work is very clearly laid out. The two models chosen for the comparison are as similar as possible, controlling for as many confounders as is realistic.\n\nThe conclusions are of high utility, with the performance difference between the two embedding methods being consistent across almost every single evaluation of every task investigated.\n\n# Weaknesses\n\nThe work hypothesizes the relation between of latent structure and related performance under ambiguity stemming from the entailment loss of the MERU model. The MERU paper performed ablations investigating the impact of the presence or not of the entailment loss in the pretraining objective. The work would be more informative for the research community if comparisons using a non-entailment pretrained model were included, such that the posed hypothesis is rejected or accepted.\n\nThe work only studies models at a single scale (the ViT-L/16 model). Models of medium and small size also available at https://github.com/facebookresearch/meru. It would be more useful to understand also if the trends observed for the large model holds for smaller models.\n\nIn result reporting throughout, it is not clear when results are statistically significantly different, as opposed to e.g. the accuracies reported just being larger in one case than the other. This is particularly problematic in Table 3, where many of the numbers from Euclidean and Hyperbolic CLIP are similar (or even the same at this level of numerical significance), and only the Hyperbolic CLIP result are boldened.\n\nThere are cases where Hyperbolic CLIP fails compared to Euclidean CLIP (e.g. see Figure 6), the authors do not explain these/provide any error analysis.\n\nThe presentation of some of the figures would be improved, e.g. figures should be vector graphics where possible. Figure 5 has no x/y axis levels and the text is small, Figure 6 the text is small.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work investigates the importance of embedding space geometry on zero-shot performance of vision-language models.\n\nSpecifically, the work compares:\n\n* A Euclidean CLIP Model\n* A Hyperbolic CLIP Model\n\nUsing text prompts in the corresponding text encoder to perform zero-shot classification. The Hyperbolic CLIP Model is the MERU type from https://arxiv.org/abs/2304.09172, using a Lorentzian hyperboloid. Both models are taken from https://arxiv.org/abs/2304.09172, are equally sized (ViT-L/16 Image encoder), were trained on the same data (12M image-text pairs from RedCaps https://arxiv.org/abs/2111.11431), and are not further trained in this work.\n\nThe work compares the impact of embedding geometry upon model behavior for three types of task through zero shot evaluation:\n\n* Capturing spatial awareness\n* Dealing with ambiguity\n* Identifying OOD samples\n\n\nThe authors find that:\n\n* Hyperbolic embedding geometry gives consistent performance improvements in all three task types.\n* Hyperbolic embedding geometry produces hierarchical embeddings in vision-language models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Required (critical)\n\n* Check statistical significance of results in Table 3. (e.g. bootstrap CI/permutation test)\n\n\n# Recommended for strengthening but not critical \n\n* Add statistical significance to results throughout.\n* Include results with hyperbolic CLIP models pretrained without entailment loss.\n* Include results for small medium and models.\n* Use vector graphics for all figures where possible.\n* (page 1, paragraph 1, line 8) Remove “vastly” (avoid subjective), potentially qualify instead with the nature of the difference.\n* (page 1, paragraph 2, final line) Define “common tasks”.\n* (page 2, Ambiguity Resolution) Define “distortion”.\n* (page 2, Image-Text Representation Leaning) add discussion regarding the origin of vision-language models as potentially from data, not specifically the vision-language aspect (https://arxiv.org/abs/2205.01397), as far as I understand, the origin of CLIP’s robustness being due to multi-modality is now contested\n* (page 3, Hyperbolic Learning) Clarify paragraph regarding the limitations of Shen that vision-only hyperbolic models do not perform well, but vision-text hyperbolic models do.\n* (page 6, table 1) Explain why worse than random is OK for Euclidean CLIP. I know you mention it in text that this is consistent with findings of other work, it would be useful for readers to see these claims in context here,\n* (page 7, propaganda Memes) the description of points 1. - 4. for how embeddings are produced is confusing as the fourth point returns the score itself, not an embedding. Maybe separate point 4. from 1.-3. or, turn 1.-3. into score produces by prepending the inner product with the text embed of prompt class.\n* (page 8, My Reaction When) define “common”\n* (page 11) Provide equations for MLS and EBO.\n* (page 11, MDS) Explain that because we have chosen a specific point of projection from the hyperboloid back to the tangent space, that the inner products are all defined for the MDS calculation (otherwise a reader may think the MDS is doing inner products between vectors existing in different tangent spaces).\n* (page 13, Fig 5) Add x and y axes labels, make text larger.\n* (page 13, Fig 6) Add error analysis/discussion for elements with negative delta.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fQXJwGcMdR", "reviewer": "Reviewer_fEQu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The paper is clear and presents hyperbolic VLM representations well. It presents the 3 explorations well enough and the Setup-Results-Analysis construction of each section was very constructive and helpful.\n   1. However, a lot of the theory and supporting knowledge has to be gleaned from reading the original papers, the summary on page 4 is quite terse.\n2. I found it hard to clearly understand that the authors did not train any models of their own, and instead just used publicly released models and checkpoints.\n   1. This is mentioned in Section 3.5, but I feel like this should be made more prominent.\n   2. It is hard to know if the results would have followed when using a reproduced Euclidian CLIP? All numbers are very close to each other, which might just indicate that these particular checkpoints aren’t particularly strong or that the RedCaps dataset isn’t sufficiently complex.\n   3. This isn’t discussed in the paper, and the models are taken as is and compared directly. Unless these are standard models that have been reused by the community, this feels like a source of uncertainty about the reproducibility of these observations.\n3. Exploration 1 on spatial abilities is well executed. Datasets are chosen appropriately, they target known issues with CLIP and they assess the correct metric.\n   1. Figure 3 is interesting, and provides good evidence for the benefit of hyperbolic representations.\n   2. However I feel like the results are only slightly better than Euclidean CLIP, and hence the claim of superiority is a bit stretched.\n4. Exploration 3 on OOD is also well executed, and shows the strongest and clearest results of the paper. I am not an expert in this domain so would leave others to comment on this, but this section is strong enough for me.\n5. Exploration 2 on ambiguity is not as well-defined as the other two, and I am not convinced it provides evidence for the claims made.\n   1. It is hard to appropriately quantify how well models handle “ambiguity”, especially given these are contrastive models with no intrinsic measure of uncertainty. I find it strange that the authors consider that doing better on “ambiguous vision-language matching datasets” would imply “handle semantic ambiguity better”.\n   2. There are too many alternative failure modes possible that could explain these results without ambiguity or hyperbolic representations providing a clear explanation for the change in metrics.\n   3. Table 2 also shows extremely close results for the two models, I would probably not make strong claims based on these.\n   4. Only the Visual Word Sense Disambiguation dataset feels like it enables some sort of “hierarchy of semantic specificity”, which I see how this would be enabled by hyperbolic VLMs, but again scores in Table 4 are very close together.\n6. Mostly a nitpick, but I find the phrasing of the “hypotheses” to be too opinionated and not actually taking the form of an hypothesis.\n   1. For example, hypothesis 3: “hyperbolic VLM are good OOD discriminators”, this is an observation/result, not a hypothesis. The hypothesis ought to be the specific reason/mechanism for why that would hold, which is mentioned in the paragraph below.\n   2. Similarly, “Hyperbolic VLMs exhibit spatial awareness” is a strong claim, I do not believe the results warrant such a claim, they are slightly better than Euclidian CLIP at best", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper compares CLIP embeddings to Hyperbolic CLIP embeddings on 3 different axes: spatial awareness, ambiguity resolution and out-of-distribution discrimination.\n\nThis is a strictly evaluative paper, which reuses models trained and released by Desai et al 2023 (if I understood correctly, see weakness and requested change below).\n\nOverall I find that the first and third explorations are well executed and show interesting observations that would interest some members of the TMLR community (albeit a small minority). However, the second exploration is more problematic to me, as I find it does not provide sufficient evidence to the claim of “hyperbolic vision-language models are better in handling ambiguity”, in particular due to how they chose to assess ambiguity with the datasets selected.\n\nAs it stands, I’m leaning toward rejection, given this issue and the limited scope (even given TMLR’s broad acceptance guidelines).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical:\n1. Clarify model checkpoint usage and on strength of the base CLIP model\n2. Comment on the closeness of the scores throughout the paper, how much better would one need to be to be confident about one model performing significantly better than the other?\n3. Comment on Exploration 2 and how they provide enough evidence for the current claim.\n   1. Alternatively, reduce the claim to a lower level or remove this section.\n\nStrengthen:\n1. Comment on other issues raised above.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Seems like good empirical work, of interest to a wide range of folks!
Decision: Accept (poster)