The Double-Ellipsoid Geometry of CLIP
Authors: Meir Yossef Levi
Guy Gilboa
Venue: ICML 2025 poster (2025)
Keywords: CLIP embedding
modality gap
narrow cone effect
Primary area: deep_learning->foundation_models
TL;DR: We reveal that the embedding space of CLIP consist of two ellipsoids, one per modality

Abstract: Contrastive Language-Image Pre-Training (CLIP) is highly instrumental in machine learning applications within a large variety of domains. 
We investigate the geometry of this embedding, which is still not well understood, and show that text and image reside on linearly separable ellipsoid shells, not centered at the origin. We explain the benefits of having this structure, allowing to better embed instances according to their uncertainty during contrastive training.
Frequent concepts in the dataset yield more false negatives, inducing  greater uncertainty.
A new notion of conformity is introduced, which measures the average cosine similarity of an instance to any other instance within a representative data set. We show this measure can be accurately estimated by simply computing the cosine similarity to the modality mean vector. Furthermore, we find that CLIP's modality gap optimizes the matching of the conformity distributions of image and text.

The Double-Ellipsoid Geometry of CLIP
Meir Yossef Levi'
Guy Gilboa'
Abstract
Common themes
Contrastive Language-Image Pre-Training (CLIP) is highly instrumental in machine learning appli- cations within a large variety of domains. We in- vestigate the geometry of this embedding, which is still not well understood, and show that text and image reside linearly separable ellipsoid shells, not centered at the origin. We explain the benefits of having this structure, allowing to bet- ter embed instances according to their uncertainty during contrastive training. Frequent concepts in the dataset yield more false negatives, inducing greater uncertainty. A new notion of conformity is introduced, which measures the average cosine similarity of an instance any other instance within a representative data set. We prove this measure can be accurately estimated by simply computing the cosine similarity to the modality mean vector. Furthermore, we find that CLIP's modality gap optimizes the matching of the con- formity distributions of image and text.
Rare themes
Text
Origin
Image
CLIP space
Figure I. Sketch of CLIP general geometry: image and text are embedded linearly separable ellipsoid shells, not centered the origin. This allows to control uncertainty in contrastive learning, where themes hecome more (lower uncertainty) they reside farther from the mean modality vector.
2022).
Despite these advances, the structure of CLIP's latent space remains poorly understood. Existing studies focus proper- ties like alignment, uniformity, and the modality gap (Liang et al., 2022) but overlook the geometry underlying this multi- modal space. The 1.2-normalization phase, which is integral when performing cosine similarity, practically reducing the data to the unit hypersphere. Since normalization is an information-redducing process, understanding the primary embeddings prior to normalization can reveal deeper in- sights into the latent space geometry.
1. Introduction
Multi-modal approaches, particularly Contrastive Language- Image Pre-Training (CLIP) (Radford al., 2021), have revolutionized computer vision tasks, enabling applications such high-quality image generation (Ramesh et al., 2022; Nichol et al., 2021), open-vocabulary classification (He et al., 2023), segmentation (Liang et al., 2023; Yu et al., 2024), detection (Wu et al., 2023), captioning (Mokady et al., 2021: Cho et al., 2022), and semantic editing (Kim al., 2022; Kawar et al., 2023). Beyond images, CLIP's success extends to 3D (Hegde et al., 2023; Chen et al., 2023; Zhang et al., 2022), video (Tang et al., 2021; Luo et al., 2022), and audio domains (Wu et al., 2022; Guzhov et al.,
In this paper, we propose analyzing the pre-normalized CLIP primary embedding for three key reasons: (1) Enhane- ing downstream tasks. While L."-normalization is integral to the cosine similarity used during training, the primary em- bedding directly employed in critical downstream tasks, including image generation and semantic editing. Analysis of the latent geometry can enhance the performance of these tasks. (2) Semantic significance of magnitude. Despite the cosine similarity is agnostic to the norm, we observe that magnitude still plays a significant and meaningful role. Notably, the largest embeddings MS-COCO correspond
'Viterbi Faculty of Electrical and Computer Engineering, Tech- nion- Israel Institute of Technology, Haifa, Israel. Correspon- dence to: Meir Yossef Levi levil campus. campus.tcchnion.ac. ii.,, Guy Gilboa Eguy.gilbos ec teclnion.ac.il>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Double-Ellipsoid Geometry of CLIP
to unusual or exotic captions (e.g.. "I am not sure what this image is", see full histogram and examples in Figure 14 in the Appendix). (3) Deeper understanding of contrastive learning. CLIP is an exceptional semantic encoder achieved through a rather generic contrastive loss and huge training data. Investigating the solutions found by CLIP allows deeper insights on contrastive learning, possible approaches to tackle false negatives and may shed light on unresolved phenomena, such as the modality gap and the narrow cone effect (Liang et al., 2022).
strong correlation with similarity to the mean vector, offering insights into semantic organization.
5. We highlight the role of conformity in explaining the modality gap and propose its use in ranking text and image generators.
6. We introduce vertical SLERP (vSLERP), an interpola- tion method leveraging the geometry of CLIP's latent space.
Our analysis reveals that CLIP's primary latent space ex- hibits a double-ellipsoid geometry, with one ellipsoid for images and another for text. Both are shifted from the ori- gin (see Fig. 1), in line with the narrow cone effect and the modality gap (Liang et al., 2022: Fahim al., 2024; Schrodi et al., 2024). Using the MS-COCO validation set (Lin et al., 2014), we show that both modalities exhibit the thin-shell phenomenon (Klartag, 2023; Klartag & Lehec, 2022), where most of the mass concentrates within a specific range from the mean.
2. Related Work
Contrastive representation learning is powerful learning scheme, where models are trained to associate positive pairs (e.g., different views of the same image (Chen et al., 2020)) closely in the embedding space while pushing negative pairs (e.g., different images) apart. This simple yet effective approach has led to significant advances across a wide range of applications, i.e. image classification (Chen et al., 2020; He et al., 2020), natural language processing (Gao et al., 2021; Kim et al., 2021), 3D analysis (Afham et al., 2022; Xie et al., 2020) and more.
This geometry affords several advantages. The offset from the origin allows CLIP control the sharpness of its re- sponse in contrastive learning, mitigating false negatives (Byun et al., 2022; Li et al., 2022; Yang et al., 202 stances that are conceptually similar but incorrectly treated as negatives. Frequent concepts with higher uncertainty are embedded closer to the mean vector, phenomenon we term semantic blurring, reducing loss and improving perfor- mance. Our experiments confirm that frequent concepts are better aligned to the mean vector of the ellipsoid, achieving excellent agreement with our hypothesis.
The latent space induced by contrastive learning has been widely explored (Arora et al., 2019; Ji al., 2023; Wang et al., 2022: Wang & Isola, 2020), often conceptualized as a normalized hypersphere (Wang & Isola, 2020; Liang et al., 2022). Alignment and uniformity (Wang & Isola, 2020) are key properties of the Normalized Temperature-scaled Cross-Entropy (NT-Xent) loss (Chen et al., 2020). Opti- mizing alignment and uniformity was shown to be crucial for preserving rich semantic structures in the latent space, leading to improvements in downstream performance across multiple domains (Fahim et al., 2024).
Leveraging this deeper understanding, we introduce new definition of concept conformity, quantifying how close a sample resides with respect to all others. We prove that there a proportion between conformity and cosine similarity to the mean vector (See proof in Supp. CI, and empirically with Pearson correlation: 0.9998 for MS-COCO). Further- more, we show that the distribution of conformity differs between modalities, with CLIP's s ellipsoid alignment offer- ing a plausible explanation for the modality gap.
With the rise of cross- contrastive models, such as CLIP (Radford et al., 2021), which align images and text in a shared embedding space, new challenges in latent space geometry have emerged. A notable issue is the modality gap (Liang et al., 2022), where embeddings from different modalities, such as images and text, are separated in the shared latent space. Moreover, the narrow cone effect was observed (Liang et al., 2022; Schrodi et al., 2024), where features occupy only a limited portion of the angular space.
Our contributions are as follows:
1. We reveal that CLIP embeddings form separable ellip- soid shells for each modality, shifted from the origin.
One of the main challenges in multimodal contrastive learn- ing is of obtaining high-quality pairs. Web-scale datasets may include mismatched positive pairs (Chun et al., 2022: Gadre et al., 2024; Maini et al., 2023; Wang et al., 2023) or mislabeled negative pairs that are actually positive, referred to false negatives (Byun et al., 2022; Li et al., 2022; Yang et al., 2022). Numerous approaches have emerged to address this challenge, such as by identifying and introducing hard negative examples (Byun et al., 2024; Chuang et al., 2020; Robinson al., 2020; Kalantidis et al., 2020). Our obser-
2. We analyze the benefits of this structure, including its role in controlling sharpness in contrastive learning.
3. We show that frequent concepts benefit most from this geometry, optimizing the contrastive loss near the el- lipsoid offsets for MS-COCO.
4. We define concept conformity and demonstrate its
2

---

The Double-Ellipsoid Geometry of CLIP
vations are that false negatives appear to play a significant role in forming the geometry of CLIP's latent space.
Recent studies have shown this bound for a 4 (Klartag & Lehec, 2022), a 2.23 (Jambulapati et al., 2022) and most recently for _ (Klartag, 2023). See more details in the above papers and the references therein. Essentially, this means the mass of the distribution is concentrated around a shell of radius vT.
3. Random vectors in high dimensions
3.1. Notations
We investigate CLIP space induced by ViT-B/32 encoders of n = 512 dimensions, X = R Let X. C X be the image subspace and X2 C X be the text (captions) subspace. We will reaffirm that they are different and in fact linearly separable (Schrodi et al., 2024; Liang et al., 2022). Let E X be vector in this space. We denote by E X; vectors of images and by Us E Xr vectors of text. The symbol E stands for the expected value. The respective modality mean of image and text are m, = Ex, [ui] and me = Ex, [v.]. Let ū be the vector after subtraction of the respective modality mean. That is, for images, ün 0 m; e Mr and for text ü₁ = Ut my i2 E X4-
Let us farther examine this for the more general anisotropic case. Let variables of different distributions (not iid), each of mean zero. Let the norm of x, which is a random variable, be defined by |||||| = Unorm + y, where Unorm == [|||=||] and y is a random variable of zero mean. We examine the term [||| = tr(C), where tr is the trace and C is the covariance matrix of
(z([),.... z(n)) be vector of random
E |= 12] BBly + 2nnorm!! + +0] Unorm + var(y).
E[ + y)
(3)
Therefore, for Unorm > var(y) we can approximate
Our statistical analysis and many experimental results are based on MS-COCO (Lin et al., 2014) validation set, a common standard image-text dataset.
||| = Pnorm
E| = / tC(CC.
(4)
Here the squared expected Euclidean norm and the trace of the covariance matrix approximately coincide, We can thus view (t))) as a rescale of the coordinate system in dimension k with respect to a unit sphere.
3.2. High dimensional geometry of random vectors
It is often challenging to obtain good intuition on the prob- ability manifold and its geometry in high dimensions. We outline below some fundamental concepts.
festure
Fanuure 334
Image
Text
3.2.1. THIN SHELL THEORY
1nd.
Foetarr
There is intensive research related to the thin shell phe- nomenon (Kannan et al., 1995; Paouris, 2006; Klartag & Lehec, 2022; Jambulapati et al., 2022; Klartag, 2023). Defi- nitions of log concave distributions and isotropic random vectors appear in the Appendix. Since isotropic random vectors have unit second moment for any x<k),, A = I,,,, we get that the expected value of the squared Euclidean norm is
Figure 2. Normalized histograms of certain CLIP features. Image and text are clearly drawn from different statistics. On the right it is shown that even features are sufficient to obtain full linear separability. The results of linear SVM classifier are shown (blue dashed line, with 100% accuracy on MS- MS-COCO).
4. Geometric Analysis
|| = E x(k) = Σ B[x(k))1 = n.
(L)
We begin by examining the statistics of image and text in the CLIP embedding space X. This part is completely data- driven without any prior assumptions related to the training process. We focus on the primary CLIP embedding. which is the output of the encoder before L2 normalization, i.c. before projection onto the unit hypersphere. This projec- tion loses important information. basically "flattens" the original geometry artificially, in a manner which hard to analyze. More details and statistical data are provided in the Appendix. Let us first examine the known modality gap phenomenon (Liang et al., 2022) in the primary embedding. In Fig. 2, normalized histograms are shown for features 93, 134 and 494 of the CLIP latent vector. We get a bimodal distribution where image and text are clearly not drawn from the same distribution. For feature 93, for instance,
As shown for example in (Paouris, 2006), ||||=||] ~
E² [|]]]]] the expected norm of x can be approximated by
|[||=|| ~ Vn..
(2)
For isotropic log-concave distributions we have the thin shell property:
Theorem 3.1 (Thin shell). Let the thin shell parameter be defined by
O sup vn)²,
where the supremum is over isotropic, log-cow cog-concave random vectors in R Then on I ollogn),, where c is a universal constant.
3

---

The Double-Ellipsoid Geometry of CLIP
distribution appears smooth, unimodal, with peak around zero. The norm ||ö|| however, is distributed within small range (thin shell) such that there is no mass near zero.
Manuumi heve distict
Irsgee
Tec
230
500
Incanfo(( (f))
Figure 3. Scparability of features (left) and 10 most significant features & for image and text, with high absolute mean, compared the feature's 's standard deviation.
Figure 5. Normalized histograms of feature variance (left) show a long tail, indicating an ellipsoid rather than a hypersphere. Off- diagonal dominance (Eq. 6) suggests strong feature correlations, implying a tilted ellipsoid.
We can further check the validity of Eq. 3, we examine images here. In the case of MS-COCO statistics we have: Anorm = 7.5873, var(y) = 0.1914, yielding P2 57.5671 >> var(y), where the approximation /E]| 7.6007 is just with 0.18% relative error. We can therefore conclude:
Norm of images
Norm of text
0.15
Thin Shell:
02
Thinsshell:
Shell:
0.15
0.1
Mcak-7759
Mean=5.59
St 0.44
St d. = 0.35
0.05
Property 2: The mass of each modality is concen- trated within thin shell, with zero mass near the mean of the distribution.
0.08
Figure Statistics of image and text features after mean subtrac- tion. Top: The first 10 features for image (top) and text (hottom). Bottom: Histograms ||ق|| for images and text, showing thin- shell phenomenon with no volume below threshold, typical for high dimensions.
Let us now investigate the geometry of each shell. We examine the variance each feature é. In a uniform hy- persphere embedding we expect to have similar variance for all dimensions. We observe in Fig. (left part) this is not the case, with long tail distribution, where some fea- tures exhibit considerably larger variance, hence ellipsoid structure:
the KL.-divergence between the distributions is x 301 (a value above implies considerable deviation between the distributions). It was previously shown in (Shi et al., 2023; Fahim et al., 2024; Schrodi et al., 2024) that image and text can be separated linearly. We find there are actually 9 features which serve as sort of "tags" for image and text. More formally, we can define the measure of separability of a feature e by
Property 3: The embedding of both text and image is of an ellipsoid shell.
We now examine inter-comreatiooss between features. Let us define off- di-ddiagonal dominance of a row i in the covariance matrix C by
(m,(1) - (8) Sep(E) V vvrr(e(()) + (ar((,()))
ODD(() = Exee [Cekl
(5)
(6)
Cue
plot of the features sorted by separability is shown in Fig. 3 (left). Fig. 2 (right) shows that the modalities are linearly separable (with 100% accuracy) using only two such tag features (93 and 134), based on linear SVM classifier (decision boundary shown in blue). We can thus state the following property (which holds exactly for MS-COCO):
Diagonally dominant matrices have ODD(E) < 1, V en- suring a non-singular matrix. We observe (see Fig. 5 two right plots) that the off diagonals are significant, implying non-negligible correlation between features, thus:
Property 4: The ellipsoids of both modalities are tilted.
Property Image and text reside on separate sub- spaces, X n 24
Finally, we check the location of each ellipsoid, with re- spect to the origin. We recall my, m, E R are the mean value vectors of image and text. Let di d, R" be the standard deviation vectors of image and text, respectively. We have |mm]]| ||]]||
In Fig. 4, we show some statistics the features of é; and é; (where the mean is subtracted). To get impression, the first 10 features in each vector are shown for both modalities. The
0.94 and || 1.03. Viewing ||o|| as ||all|

---

The Double-Ellipsoid Geometry of CLIP
mean vector magnitude of the ellipsoid shell, the means significantly shifted from the origin, compared to the size of the ellipsoid. This is caused by few features, with strong deviation from the origin (compared to the respective feature's s standard deviation), as shown in Fig. 3 (middle and right). Thus we can state:
36
Unif Uniforiity
3.69
43
30
75
Loss
allgnment
25
20
-1
-0.5
0
0.5
Property 5: The ellipsoids are not centered near the origin.
Misclassif ted
LO8%
lest
5. Loss behavior on a double-ellipsoid
Both
Correctly Classified
aridin
In this section, we validate that non-origin-centered doublc-ellipsoid structure achieves optimality in terms of the CLIP contrastive learning loss.
-0.5
0.5
Figure 6. Loss vs. embedding center position. The parameter a controls the embedding center (Eq. 9, with a = 0 as the current non-origin--entered CLIP position). (Tep). The unified loss balances uniformity and alignment optimally for non-origin- centered positions. (Bottom). The loss increases for misclassified instances and decreases for well-classifeed ones, with balanced accuracy at a R O.
For a batch containing M image-text pairs, we denote by of of = [o and w/ "g [| the normalized image and text features of the j-th pair in the batch respectively. The multi-modal learning loss used in CLIP the normalized cemperature-scaled cross entropy loss (NT-Xent), varia- tion of InfoNCE (Oord et al., 2018) loss:
log Σj cl 0//7 + log e ofttr
ct of/7
1
leipp ==
2
E J.KM
linear interpolation and extrapolation of the mean relative to the origin, using single scalar parameter a. This measure is conducted on a grid of a values from -1 to 1, with the loss calculated on image features as follows:
(7)
As observed by (Wang& Isola, 2020), the loss can be de- composed into two terms: (1) Alignment, which encourages high cosine similarity for positive pairs, and (2) Uniformity, encourages low cosine similarity among negative ones.
of = a.mm Vj E M.
(9)
The values of o remain unchanged. Unlike the Embedding Shift Experiment in (Liang et al., 2022), here, the modalities are not directly shifted to each other, but to the origin.
alignment
Lotip
J0 +
JEM
uniformity
The results show that the loss for correctly classified samples decreases monotonically with the shift toward the origin (i.e. that for perfect alignment as assumed for example by (Liang et al., 2022), shifting to the origin would be preferable). Conversely, the loss for misclassified samples increases. The overall loss balances alignment and uniformity for both correctly and misclassified samples, reaching an optimal near zero. This aligns with the current CLIP embedding, though some deviation is expected, as the MS-COCO valida- tion set is only an approximation of the full training set. For completeness, the Appendix includes the same experiment with the text ellipsoid shifted instead of the image, showing consistent behavior. To conclude:
M
E AEM
1 log
T o
/T
+
1
log
e
ET
T//T
2
2
j=1
To empirically analyze the uniformity and alignment terms in Eq. 8 alongside the overall loss in Eq. 7, we use the MS-COCO validation set. Fig. 6 shows the overall loss (bottom) and its breakdown into uniformity and alignment losses (top). We treat the entire validation set (5k samples) as single batch. The overall loss is further separated into correctly classified, misclassified, and combined cases; the union of the correct and misclassified is equivalent to both, and they are mutually exclusive.
Property 6: CLIP's loss is optimized for non-origin- centered ellipsoids, balancing alignment and unifor- mity for both correct and misclassified instances.
In this experiment, we examine different values of the mean value of the image embedding. For simplicity, we apply
5

---

The Double-Ellipsoid Geometry of CLIP
BROW
C:
C.0.9
C.
Centered near origin
Centered far from origin
02
0.25
Figure 8. High low conformity MS-COCO. Low-
Extreme vector
Extreme vector
Mean vector
conformity images often depict unique, distinguishable individu- or objects, whereas high-conformity images capture common seenes that could be found anywhere.
0.15
0.15
0.1
Mean vector
0.1
0.00
the modality gap.
02
0
0.2
0.4
0.5
0.5
-0.
0.6
0.8
Blur through a non-origin centered sphere. Fig. 7 (bot- tom) illustrates the difference between origin-centered and non-origin-centered spheres through an experiment. We draw 1000 random vectors v E R2522 where each element follows an independent Gaussian distribution with unit stan- dard deviation. In the first experiment (Fig. 7, bottom left), the sphere is centered at the origin with an empirical mean m close zero. The blue histogram shows cos(m, v²) for 1, ..... 1000. We then identify the furthest vector from m, y arg min cos m, )²), and plot the histogram of cos fa v²) (orange), excluding ffr In the second ex- periment (Fig. 7, bottom right), the sphere is centered at (10, 5, 5, 0, .,..... modeling three dominant features with mean distinctly far from zero. The same trial is repeated. The results highlight a significant difference: for an origin- centered sphere, the distributions of cosine similarity for the mean and the extreme vector similar. In contrast, for a cen--origin-ceneee sphere, the mean vector exhibits much higher average similarity. This allows the network to embed vectors with uncertainty closer to the mean, enabling semantic blurereduced contrast the response. This anal- ysis, supporting a non-zero mean, leads the following prediction:
Figure 7. Top: Example of segmentation score blur (right), com- mon in semantic segmentation, increases. Bottom: Similarity histograms of normally distributed samples for the mean vector (blue) and the furthest vector from the mean (" ("extreme", orange). Results are shown for sphere centered near the origin (left) and one centered far from the origin (right). In contrastive learning, blur can be controlled by adjusting the sphere's offset. Embedding vectors closer to center induces blur, while positioning them farther away sharpens the response.
memeerrhem uncertainty
6. False negatives and conformity
We demonstrate how the embedding geometry discussed earlier provides advantages in handling false negatives. Ad- ditionally, we introduce the concept of conformity, which plays major role in forming the latent space distribution. A well-known challenge in contrastive learning the presence of false negatives-pairs with similar meanings that not dedicated pairs. Such samples should not be embedded far apart, as they fail to represent true negatives effectively. This issue arises in both single- and multi-modality settings and has been addressed by proposing new training proce- dures or alternative contrastive losses (Byun et al., 2024; Chuang et al., 2020). In CLIP, training uses a contrastive loss that does not explicitly address false negatives. How- ever, argue that this issue is partially mitigated by the embedding geometry. In classification and segmentation tasks, uncertainty typically results in softer predictions that reflect lower class membership probabilities, For example, Fig. 7 (top) illustrates a segmentation score where reduced confidence blurs the sheep's boundary, a phenomenon we term semantic blur. For contrastive networks, when false negatives are present, we expect lower confidence and blurred response. On high-dimentn digeesiinnsi sphere centered at the origin, such blurring is challenging, small perturba- tions lead to large changes in cosine distance. We show that shifting the sphere away from the origin can effectively mit- igate this issue. Concurrently, and closely related, Schrodi al. (2024) discuss the relationship between entropy and
Common themes, which occur more frequently in the training set, expected to be em- bedded in closer proximity to the mean vector.
Prediction 1:
6.1. Conformity
To validate Prediction 1, we first formalize the term common themes, by defining new notion, termed conformity.
Delinition 1 (Conformity). Conformity of a vector & within set S measures the expected value of the cosine similarity to this vector:
C(r
E [oos(v" (
(10)
1 ES
k
where for given fintt set S, the empirical mean is taken.
6

---

The Double-Ellipson Geometry of CLIP
0.9
picture of some people by the street.
30
KL Devergeene
0.8
A picture of some food on plate.
0.7
Pearson 0.9998
20
Pearson 9998
0.5
Images Captions
10
0.5
0.4
A Vintage yellow refrigerator surrounded by wood Cabinetry.
-0.5
0.5
0.3
A Calico cat drinking from sink Faucet.
0.7
0.2
0.25
0.3
0.35
0.4
C
0.45
0.5
0.55
0.8
0.65
Figure 11. Modality Gap matches conformity distributions. The parameter controls the embedding offset from the origin (as shown Fig. 6). When a O, i.e., the trained setting, image and text conformity distributions align well, with K Lo=o & 0.14 indicating good distribution matching.
Figure 9. Conformity. Estimated conformity Õ, Eq. 11, against conformity C, Eq. 13, on MS-COCO (Lin et al., 2014). The corre- lation is almost perfect. We can thus the proposed estimated conformity reliably to quantify how common sample is. More exotic captions have lower conformity (all examples are of cight words).
6.2. Modality gap assists in distribution matching
Brotp
4 woman wearing a blue e-shirt white rooking her cell phone and sieeing on bench naxe to brighe pink wall.
360
soldiers seanding neve 00 each other. 0.98
We now aim to provide reason that can justify the pres- ence of the well known modality gap (Liang et al., 2022). Our rationale for that phenomenon is as follows. The same incentive of having mean not centered at the origin applics for both image and text modalities. However, in a single amage-pairr instance the uncertainty for each modality may differ (see Fig. 10). The same arguments before pro- mote uncertain instances to be near the mean and certain ones to be far from it. If both image and text of a pair are embedded at the same location we may get contradicting requirements. Having separate embeddings for text and im- age allows to control the uncertainty of each instance for each modality. More generally, we would like to match the distribution of the conformity of both modalities. In Fig. 11 show the KL-divergence of the conformity distribution as function of the mean from the origin, as in Eq. 9, see illustration in Fig. 6. We show that the best distribution match is near a = 0, i.e., the current embedding of CLIP.
300
G. 0.31
250
200
150
0..36
100
0.72
0
-0.1
C Ce
Figure 10. Conformity Differences. The conformity distributions of text and image modalities differ, as a common image may be described by unique caption, and vice versa.
To provide more intuition, we present examples of high and low conformity from MS-COCO in Fig. 8, well on ImageNet-a (Hendrycks et al., 2021b) and ImageNet-R (Hendrycks et al., 2021a) in the Appendix. Following our prediction above, we propose a surrogate measure of con- formity (which is much faster to compute). The estimation uses the following definition.
parameter controlling the distance of
7. Applications
Definition 2 (Estimated conformity). In contrastive learn- ing embedding, for given set of vectors S with mean m = Eves(v), the estimated conformity of E S is:
7.1. Conformity as measure of expressiveness
We propose using conformity as metric assess gener- ative method diversity. We measure conformity in images generated from MS-COCO captions by unCLIP (Ramesh et al., 2022) and Glide (Nichol et al., 2021), I), as shown in Fig. 13. Glide-generated images exhibit high conformity, indicating low detail and diversity, while unCLIP images are more varied and detailed. Both models, however, lack the diversity seen in real images. Similarly, captioning methods by measuring conformity in captions generated by ClipCap (Mokady et al., 2021) and Caption Reward (Cho et al., 2022). ClipCap produces common cap- tions, while Caption Reward generates diverse captions that even surpass human annotations.
Ō(O)) = a cos(m, r²)
(11)
where and b are scalars determined by the embedding.
In Appendix C1 we prove this correlation under the thin- shell assumption, and in Fig. C versus Ö are plot- ted for the entire MS-COCO set, for both image and text embeddings. A close to perfect correlation is obtained, with Pearson correlation of 0.9998 for both image and text where a = 1.411, b = -0.008 for text and a = 1.461, b = -0.002 for images, validating with close to perfect alignment with the rigorous mathematic derivation.
evaluate

---

The Double-Ellipsee Geometry of CLIP
Nogative
Source
Positive
Target
Increasing
Figure 12. Vertical SLERP (vSIERP) enables optimization-free semantic editing. Interpolated images preserve the object with pose variations roughly maintain backgrounds, with interpolation magnitude controlled by a.
Example
Captioning
traint On the jine.
In Fig. 12, we show images generated by an extension of SLERP, which we term as vertical SLERP (YSLERP):
stall train
CiloCap
raltroad tracks.
McCOCO
Capolon Reward
Ressainger train str vinv drsun stae arie tulffi the baniind
VSLERP (v), , 00,00.00 a) SLERP (O'RPPu a -am, 0g) +a
(12)
For brevity, m. and 8; are referred to as m and v. With a fixed = O, adjusting a allows controlled manipulation of the same instance. This approach parallels real-image editing techniques; however, unlike methods relying on text inversion (Han et al., 2024; Gal al., 2022; Mokady et al., 2023) test-time optimization (Kawar et al., 2023), which are computationally heavy, vSLERP requires no training or optimization, thus, highly efficient.
0.5
0.4
U.5 CCnnROVniiyy
0.7
0.8
0.9
0 browh white and black laving gray couch
Image Synthesis
UnCili
Mscoco
Glide
0:4
0.5
Cofformity 0.7 0.5
0.9
8. Discussion and Conclusion
Figure 13. Conformity analysis of captioning and image synthe- sis. Image Synthesis (top): Glide generates more common images detailed images
The paper examines the primary CLIP embedding, prior to projection onto the unit sphere, revealing that each modality forms distinct, shifted ellipsoid with unique centers and radii. This geometry is the source of the modality gap and narrow cone phenomena (Liang et al., 2022; Schrodi et al., 2024; Afham et al., 2022), previously observed on the unit sphere embedding. We introduced conformity, a measure of similarity of an instance with an entire representative set. Our analysis shows that each modality exhibits unique conformity distribution, with optimal alignment achieved when the ellipsoids are shifted from the origin. This pro- vides a useful tool for assessing the diversity of captioning and image synthesis methods. Finally, we propose vertical SLERP (vSLERP), a training free interpolation technique for specific object interpolation.
with less fine detail, while unCLIP creates closer to natural distributions. Captioning (bottom): ClipCap pro- duces more common captions, while Caption Reward generates more unique captions, even surpassing human annotations.
7.2. Unguided, training-free semantic generation
The unCLIP framework (Ramesh al., 2022) introduces image interpolation technique using spherical linear inter- polation (SLERP) to transform source image into target image gradually. While this method produces visually ap- pealing results, it often fails to preserve the same instance along interpolation, instead generating random instances.
8

---

The Double-Ellipsoid Geometry of CLIP
Acknowledgements
Chun, S., Kim, W., Park, S., Chang, M., and Oh, S. Ecev caption: Correcting false negatives by collecting machine- and-human-verified image-caption associations for ms- coco. In European Conference on Computer Vision, pp. 1-19. Springer, 2022.
We would like acknowledge support by the Israel Science Foundation (Grant 1472/23) and by the Ministry of Science and Technology (Grant No. 5074/22).
Impact Statement
Fahim, A., Murphy, A., and Fyshe, A. Its not modality gap: Characterizing and addressing the contrastive gap. arXiv preprint arXiv:2405. 18570, 2024.
Our work advances machine learning by improving the geo- metrical understanding of CLIP's latent space. The findings may influence downstream tasks, though specific societal consequences do not need further emphasis here.
Gadre, Y., Iharrco, G., Fang. A., Hayase, J., Smyrnis, G.,
Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
References
Afham, M., Dissanayake, I., Dissanayake, D., Dharmasiri, A., Thilakarathna, K., and Rodrigo, R. Crosspoint: Self- supervised cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9902-9912, 2022.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 2022.
Gao, T., Yao, X., and Chen, Simese: Simple con- trastive learning of sentence embeddings. arXiv preprint arXiv:2104.0888 100.088211, 2021.
Arora, S., Khandeparkar, H., Khodak, M., Plevrakis, O., and Saunshi, N. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:/902.0922 092299 2019.
Guzhov, A., Raue, F., Hees, J., and Dengel, A. Audioclip: Extending clip to image, text and audio. In ICASSP 2022- 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980. IEFF, 2022.
Byun, J., Hwang. T., Fu, J., and Moon, T. Grit-vlp: Grouped sampling for efficient vision and language pre- training. In European Conference on Computer Vision, pp. 395-412. Springer, 2022.
Han, L., Wen, S., Chen, Q. Zhang, Z., Song, K., Ren, M., Gao, R., Stathopoulos, A., He, X., Chen, Y., al. Proxedit: Improving tuning-free free real image editing with proximal guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4291-4301, 2024.
Byun, J., Kim, D., and Moon, T. Mafa: Managing false neg- atives for vision-language pre- pre-training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 27314-27324, 2024.
Chen, R., Liu, Y., Kong, L., Zhu, X., Ma, Y., Li, Y., Hou, Y., Qiao, Y., and Wang, W. Clip2scene: Towards label- efficient 3d scene understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7020-7030, 2023.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
He, S., Guo, T., Dai. T., Qiao, R., Shu, X., Ren, B., and Xia, S.-T. Open-vocabulan multi-label classification via multi-modal knowledge transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 808-816, 2023.
Cho, J., Yoon, S., Kale, A., Dernoncourt F., Bui, T., and Bansal, M. Fine Fine-grained image captioning with clip re- ward, arXiv preprint arXiv:2205.1 13, 2022.
Hegde, D., Valanarasu, J. M. J., and Patel, V. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 2028-2038, 2023.
Chuang, C.-Y., Robinson, J., Lin, Y--C., Torralba, A., and Jegelka, S. Debiased contrastive learning. Advances in neural information processing systems, 33:8765-8775 2020.

---

The Double-Ellipsoid Geometry of CLIP
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F,, Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. $340-8349, 2021a.
Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., and Marculescu, D. Open- vocabulary semantic segmentation with mask-adapted clip. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Puttern Recognition, pp. 7061-7070, 2023.
Liang, V. W. Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Y. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Ad- vances Neural Information Processing Systems, 35: 17612-17625, 2022.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 15262-15271, 2021b.
Jambulapati, A., Lee, Y. T., and Vempala, S. S. A slightly improved bound for the kls constant. arXiv preprint arXiv:2208. 1644, 2022.
Lin, T.-Y, Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland. September 6-12. 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Ji, W. Deng, Z., Nakada, R., Zou, J., and Zhang, L. The power of contrast for feature learning: A theoretical anal- ysis. Journal of Machine Learning Research, 24(330): 1-78, 2023.
Luo, H., Ji, L.., Zhong, M., Chen, Y., Lei, W., Duan, N., and Li, T. Clip4clip: An empirical study of clip for end end video clip retrieval and captioning. Neurocomputing. 508:293-304. 2022.
Kalantidis, Y. Sariyildiz, M. B., Pion, N., Weinzaepfel, P. and Larlus, D. Hard negative mixing for contrastive learning. Advances in neural information processing systems, 33:21798-21800 2020.
Maini. P., Goyal, S., Lipton, Z. C., Kolter, J. Z., and Raghu- nathan, A. T-mars: Improving visual representations by circumventing text feature learning. arXiv preprint arXiv:2307.03113, 2023.
Kannan, R., Lovász, L.. and Simonovits, M. Isoperimetric problems for convex bodies and localization lemma. Discrete & Computational Geometry. 13:541-559, 1995.
Mokady, R., Hertz, A., and Bermano, A. H. Clip- cap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09334 2021.
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Mokady, R., Hertz, A., Aberman, K.,, Pritch, Y., and Cohen- Or, D. Null-text inversion for editing real images us- ing guided diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Kim, G., Kwon, T., and Ye, J. C. Diffusionclip: Text- guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2426-2435, 2022.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXhv:2112.10741, 2021.
Kim, T., Yoo, K. M., and Lee, S.-g. Self-guided contrastive learning for bert sentence representations. arXiv preprint (xXiv:2106.073455 2021.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv: 1807.03748. 2018.
Klartag, B. Logarithmic bounds for isoperimetry and slices of convex sets. Ars Inveniendi Analytica, 4, 2023.
Klartag, and Lehec, J. Bourgain's 's slicing problem and kls isoperimetry up to polylog. Geometric and functional analysis, 32(5): (155):11344111 1111114-11151 1159, 2022.
Paouris, G. Concentration of mass on convex bodies. Geo- metric & Functional Analysis GAFA, 16(5):1021-1044, 2006.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping eegguage-image pre-training for unified laniunee understanding and generation. In International confer- ence on machine learning. pp. 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8763 PMLR, 2021.
PMLR,
10

---

The Double-Ellipsoid Geometry of CLIP
Ramesh, A., Dhariwal, Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125) 1(2)3, 2022.
Yang, J., Duan, J., Tran, S., Xu, Y., Chanda, S., Chen, L., Zeng, B., Chilimbi, T., and Huang, J. Vision-language pre-training with triple contrastive learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15671-15680, 2022.
Robinson, J., Chuang, C.-Y., Sra, S., and Jegelka, S. Con- trastive learning with hard negative samples. arXiv preprint N0w:2000004592. 2020.
Yu, Q. He, J., Deng, X., Shen, X., and Chen, L.-C. Con- volutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36, 2024.
Schrodi, S., Hoffmann, D. T., Argus, M., Fischer, V., and Brox, T. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language representation learning. arXiv preprint arXiv:2404.07983, 2024.
Zhang, R., Guo, Z., Zhang, W., Li, K., Miao, X., Cui, B., Qiao, Y., Gao, P., and Li, H. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3552-8562, 2022.
Shi, P., Welle, M. C., Björkman, M., and Kragic, D. Towards understanding the modality gap in clip. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023.
Tang, M., Wang, Z., Liu, Z., Rao, F., Li, D., and Li, X. ClipAcaption: Clip for video caption. Proceedings of the 29th ACM International Conference on Multimedia, pp. 4858 4862, 2021.
Wang, J., Lin, K. Q., Zhang, J. Lei, S. W., and Shou, M. Z. Too large; data reduction for vision-language pre- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 3147-3157, 2023.
Wang, T. and Isola, Understanding contrastive represen- tation learning through alignment and uniformity the hypersphere. In International conference muchine learning, pp. 9929-9939. PMLR, 2020.
Wang. Y., Zhang, Q., Wang, Y., Yang, J., and Lin, Chaos is a ladder: new theoretical understanding of con- trastive learning via augmentation overlap. arXiv preprint arXiv:2203.134577 2022.
Wu, H.-H., Scetharaman, P., Kumar, K., and Bello, J. P. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4563-4567. IEEE, 2022.
Wu, X., Zhu, F., Zhao, R., and Li, H. Cora: Adapting clip for open- opea-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7031-7040, 2023.
Xie, S., Gu, J., Guo, D., Qi, C. R., Guibas, L., and Litany, O. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 574-591. Springer, 2020.

---

The Double-Ellipsoid Geometry of CLIP
A. Enlraged Visualizations
In Figure 15 and Figure 16, we provide the same visualizations as the main paper, but enlraged, to enhance visibility. CLIP of higher dimension. We also show some results for CLIP with ViT-L/14 encoders, n 768. In Figure 17 we show the distinct different statisties of image and text, mostly appearing in several pronounced features. Here as well, linear separation (100% classification accuracy) be reached with only two features. In Figure 18 we show that the embedding can also be modeled as two separate thin shell ellipsoids for image and text.
B. Statistical Analysis
We provide here the definitions of log concave distributions and isotropic random vectors, notions which are used in Section 4 of the main paper.
Definition 3 (Log concave distribution). A log concave distribution R has density p which admits, Vx, y E R [0, 1].
p(Ax + (1 \)y) 2 p(x)
The above definition is equivalent to stating that the logarithm of the density function is concave p(A + (1- x)y) > log p(x) + (1 - A) logr(y)) p(y). Many well-known distributions admit this property, such as normal and multivariate normal distributions, exponential, Laplace, chi, Dirichlet, gamma and more.
Definition 4 (Isotropic random vector). random vector x E R is isotropic if [[x] = and = I, where E is the covariance matrix of a and is the identity matrix.
1500
There is no image here to provide caption for.
am not sure what this image is. unable to the itnage above.
There no image describe for this question. That looks like may be hiding under something.
Histogram
1000
An individual is taken in this very picture. unable to see this image in this particular hit
500
0
6
8
10 Norm
12
Figure 14. Norm distribution. While norm magnitudes are disregarded during training due to the normalization inherent in cosine similarity, they still capture meaningful semantic information.
We give below additional analysis related applying linear transformation that turns each ellipsoid into sphere. This process is termed sphering or whitening. For lack of space, this part did not get into the main paper. However, we believe this analysis is of sufficient merit to be presented here.
C. Additional Experiments and Visualizations
C.1. Close relations between conformity and surrogate-conformity
We show below the validity of our conformity approximation under the thin-shell assumption.
Proposition Let S {or, ,o,,...,,, NN be set of vectors in FRF exhibiting the thin-shell phenomenon, Le.,
|| 5 R for all i,
12

---

The Double-Ellipsoid Geometry of CLIP
where ū - + Σ vs is the sample mean and use the Euclidean norm ||o|| =1f==E(0ee Ex(e))²
Then, for any of S, the
following approximation holds:
Beies|cos(y's v²)] so A cos(v', 8),
(13)
Hoom Pnorm |ō|| and the symbol z2 represents the shell approximation (which becomes more accurate as the width of the shell decreases) and approximate orthogonality between a random vector and the mean vector.
where A Rd
Proof. We start by expanding the left-hand side:
N
1
E, Est [²)]
N
2
|
Writing explicitly the inner-product we have:
N
|
Now, consider the right-hand side of Equation (13):
F
N
N
F
cos(v', 0))
|
||0||
/morm
Uk
N
N.
||
Mnoem
k=1
Observe that the only difference between the two expressions lies in the difference between Mnorm and |/ ||- that under the thin-shell assumption ||od ~ R2 + Whomm
We show below
Let us define by 2 the difference vector between a vector vy and the mean vector G, that is = 8. Then,
||02|| ||اا + ō|| || + 2n e + ||
In high dimensions, the inner product 21 small due to approximate orthogonality, so:
||e2 ze || + p ~e R² + P
Taking square roots:
|| 2 v R + 12om
Thus, the scalar factor A in Equation (13) is given by:
A Pparm
Pnom
R
Empirically we know for Vit-B/32 that Иnorm = 7.587 and R ~ 7.59, thus the mathematical derivation state that
A-1 /7.69277.587 1.414 For images and A +6.09"
1.4, very close the empirical observations (note
7.587
5.75
that the correlation reversed in the main paper).
C.2. Conformity
High- and Low-Conformity Images. We provide additional visualizations of high- and low-conformity images across various datasets. Figure 19 illustrates examples of sketches from ImageNet-R, while Figure 20 showcases examples from ImageNet-A. A. Both datasets contain out-of-distribure ImageNet-R features renditions of objects, such as origami or sketches.
examples: ImageNet-A emphasizes natural adversarial images, while
From these visualizations, we observe that high-conformity images tend to contain less information. Sketches are simpler, and natural images often feature large uniform backgrounds or repetitive structures. In contrast, low-conformity images frequently include substantial text, while natural images exhibit collages of objects with unique or diverse colors.
13

---

The Double-Ellipsoid Geometry of CLIP
Feature 93
Feature 134
Feature 494
0.4
0.5
Image
Text
0.4
0.3
Image
Text
Image
0.08
0.3
0.06
0.2
0.2
0.04
0.1
Text
0.1
0.02
-10
-5
0
2
4
6
-1
2
8
Linear decision
Text
boundary
Feature 134
6
4
Image
0
-10
-5
0
Feature 93
features have distinct
Separability
separability above
2
1
0
0
200
400
600
Sorted features
Figure 15. Enlarged plots from Section 4.
14

---

The Double-Ellipsoid Geometry of CLIP
Incon(reCCO (l((F((())
Image
Text
330
313
313
-- 134
494
377
[mean(v,()))
Imean (v, (r))
0.00
0.00
0.02
0.08
0.00
Norm of images
Norm of text
0.2
0.25
Thin Shell:
0.2
Thin Shell:
0.15
0.1
Mean=7.59
Meaw=5.59
St d = 0.44
0.1
d 0.35
0.05
0.05
Image
Text
0.6
0.4
0.2
0.5
Feature varlance
Feature variance
Image
Text
0.25
0.1
0.08
0.15
0.06
0.1
0.04
0.02
100
150
50
Off diagonal dominance
Off diagonal dominance
Figure 16. Enlarged plots from Section 4.
15

---

The Double-Ellipsoid Geometry of CLIP
Feature 414
Feature 441
Feature 645
0.4
0.4
0.4
0.3
Text
0.3
Image
0.3
Text
0.2
Image
0.2
0.2
Image
0.1
0.1
Text
0.1
0
-10
2
6
0
5
10
15
10
Linear decision
Feature 441
boundary
0
Image
Text
-5
10
15
Feature 645
6
Seperablity
3
2
100
200
300
400
500
600
700
800
Sorted features
Figure 17. Enlarged plots for CLIP embedding of n = 768. There are dominant features with clearly different distribution between image and text. Both modalities can separated (with perfect accuracy) by a linear SVM classifier based on only features. With respect to separability (bottom), there are 20 features with value above 1.
16

---

The Double-Ellipsoid Geometry of CLIP
(mmon(r(())
[y(] (t
Image
Text
317
414
317
Imean (v)(
10
2 (r,an(v,())
0.12
0.12
p.0c
0.02
0.02
0.02
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.04
0.04
0.00
0.00
0.04
0.00
0.00
0.00
0.02
0.02
0.02
0.02
0.02
Norm of images
Norm of text
0.25
0.25
Thin Shell:
0.2
Thin Shell:
0.15
Meaw=11.15
Mean 10.6-10.64
St d 0.11
0.1
St d = 0.59
0.05
10
Image
Text
0.4
04-
0.2
0
0
15
7
25
3
Feature variance
Feature variance
Image
Text
0.2
0.
0.15
0.1
0.1
0.05
0.05
50
100
150
20
40
80
80
100
120
Off diagonal dominance
Off diagonal dominance
Figure 18. CLIP 768, thin shell phenomenon. We observe similar geometry in the of 512) of two tilted ellipsoids, one for each modality, not centered at the origin.
17

---

The Double-Ellipsoid Geometry of CLIP
C.3. Reaffirming loss and conformity matching experiments
We revisit the loss experiment presented in Fig. 6 of the main paper and the conformity matching experiment shown in Fig. 11. To further validate our findings, conduct these experiments under two alternative settings.
First, we shift the text ellipsoid instead of the image ellipsoid, applying the following transformation:
= 2 a my Vj E M,
(14)
where the values of o remain unchanged. The results this experiment are presented Figure 21. In the second setting, we align both the image and text ellipsoids at the origin by applying the following transformations:
v
mi: of = aa.ma E M.
(15)
Here, for 0, the ellipsoids remain in their optimal positions after training, while for 1, both ellipsoids are shifted to the origin as in Figure 22.
Both experiments reaffirm that the current positioning of the ellipsoids yields optimal results in terms of loss and conformity matching. These findings further support our claims across different alignment scenarios.
C.4. vSLERP
Here, we provide additional examples of vSLERP, shown in Figure 23 and Figure 24. As discussed in the main paper, the standard SLERP process typically generates interpolated images representing different objects or individuals. contrast, our proposed vSLERP method produces diverse variations of the same object.
C: Q 88
C: 0.89
C: 0.88
Ć: 0. 2242
teeo Mf Owmin Gueen
C: O. 0.46
how Crow King Cobra
Bald Eagle
C: 0.39
Now:Drowwkiinsk..comm .com
Figure 19. High and low conformity of sketches from ImageNet-R. Images with high conformity tend to be simpler and cleaner, while low-conformity images often feature complex details covered by large portions of text deseriptions.
18

---

The Double-Ellipsoid Geometry of CLIP
(1 Th ofher ae De por condic
C:0.32
Č:O. 79
Queen
D DINK
$0
C:0.34
C.0.81
0.9.9
Č:0.82
1B
0.39
Soutweel
1000 Thanks!
82
40
C: 0.0.3
RoiPay Stooking Buotte
Figure Conformity on ImageNet-a. is possible that high conformity images are with more unique colors, perhaps contains people text, whereas low conformity images tends to contain low amount of information.
19

---

The Double-Ellipsoid Geometry of CLIP
Misciosrified
KL. Divergencee
Loss
arspin
0.5
Cornactly classified
-.4
0.0
-2.5
-.5
-04
-2.
00
6.2
0.
68
08
Figure 21. Shifting text ellipsoid only. Conformity distribution matching and loss experiments when shifting text ellipsoid only as Equation (14)
0.20
Divergenc gence
text
0.2
Loss
Misclassifted
Image
orighn
0.14
Both
d=1
Correctty dggssifted
-05
-05
-0.4
-0.2
0.2
0.4
24
08
0.5
α
Figure 22. Shifting both ellipsoids. Conformity distribution matching and loss experiments when shifting both text and image ellipsoids Equation (15).
Negative
Positive
Source Increasing e
Target
Figure 23. vSLERP lamp to vase.
20

---

The Double-Ellipsoi Geometry of CLIP
Negative
Source
Positive
Target
Increasing
KSPEST
Figure 24. vSLERP Kevin Durant to Lebron James.
21


Reviews:
[{"review_id": "kPIR4PuBRB", "reviewer": "Reviewer_MmZu", "summary": "This paper investigates the geometry of the pre-normalized CLIP embedding space. The main finding is that image and text embeddings reside on linearly separable ellipsoid shells, which are not centered at the origin. This non-origin-centered, double-ellipsoid structure is proposed as a key factor in controlling uncertainty during contrastive learning, where more frequent concepts with higher uncertainty are embedded closer to the modality mean vector, a phenomenon the authors term \"semantic blurring.\"\n\nThe paper introduces the concept of \"conformity,\" defined as the expected cosine similarity of an instance to all other instances in a representative dataset. A significant result is the strong correlation (Pearson correlation: ~0.9998 on MS-COCO) between this conformity measure and the cosine similarity of an instance to the modality mean vector. Furthermore, the paper demonstrates that the modality gap observed in CLIP can be explained by the need to align the different conformity distributions of image and text, and that the current non-zero offset of the ellipsoids optimizes this alignment.\n\nThe paper claims to contribute the following conceptual ideas and findings: (1) revealing the double-ellipsoid geometry of CLIP embeddings, shifted from the origin; (2) analyzing the benefits of this geometry in controlling sharpness in contrastive learning and mitigating false negatives; (3) showing that frequent concepts benefit most from this geometry; (4) defining concept conformity and demonstrating its strong correlation with similarity to the mean vector; (5) highlighting the role of conformity in explaining the modality gap; and (6) proposing a new interpolation method, vertical SLERP (vSLERP), that leverages the identified geometric properties for improved semantic editing.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback, intriguing comments and thought-provoking questions.\n\n- The geometry explains the modality gap and the narrow cone effect:\nWe agree that the geometry does not fully explain the reasons to all observed phenomena. We have a strong evidence that the offset of the ellipsoids is alleviating the impact of false negatives. There have been several proposed efforts in the literature to mitigate the false negatives in contrastive learning. However, to the best of our knowledge, we are the first to make the link between false negatives and the geometry CLIP converged to.  Part of our message is that the geometry analysis should be performed in the native raw embedding, not on the unit sphere, which reduces information. Hence, “modality gap” is actually “linearly separable ellipsoids” and “narrow cone effect” is better understood as “non-origin-centered”. We believe our findings can facilitate future research on contrastive learning and are thus of interest to the community. \n\n- Optimality: All of the references to “optimality” are related to the losses mentioned in Figs 6 and 11. By “optimal” we mean that the value of alpha is attained at the loss minimum. We agree that it holds only under restricted cases, as explained in the paper, where not all possible combinations of geometry are examined (hence we cannot claim global optimality, with respect to the tested variables). This should indeed be clarified and toned down, as we intend to do in the final version.\n\n- Backbone/training procedure: we validate our findings on ViT-L as well (on top of ViT-B in the main paper), thus we feel confident that architecture size (number of layers/heads or even feature size) is probably not playing a significant role in forming the latent geometry. Further examinations on additional architectures or larger datasets are important, we plan this for future study. Since CLIP is a fundamental backbone of many vision and text algorithms today, we believe knowledge on its geometry is nevertheless of high significance.\n\nQuestions: \n\nLeveraging geometric knowledge to enhance CLIP training efficiency and/or quality: possible direction could be to force geometric constraints during training, possibly by centralizing both ellipsoids as you suggested. In Fig. 22 (Supp.) we show the loss values when both ellipsoids are shifted to the origin (alpha=1). As can be seen, although correctly classified instances yield lower loss, the combination with the misclassified increase the loss. The reason is implicitly shown in Fig. 7. The average cosine similarity for non-centered ellipsoid is approx. 0.2 (highly likely that misclassified samples will lay there), whereas for centered ellipsoid it is around zero. This affects both the classified and misclassified, increasing the loss for the latter. Alternatively, one may propose an alignment which does not necessarily shift both cases to the origin. By doing so we can decouple the modality gap from the narrow cone effect, hopefully mitigating at least one of them. We are eager to further study this direction from a geometric perspective. We hope our clarifications answer your concerns and if so we would be delighted if  the final paper rank can be upgraded.", "claims_and_evidence": "*   **Image and text reside on linearly separable ellipsoid shells, not centered at the origin:** The paper mentions statistical analysis of the MS-COCO validation set and Figure 1 as a sketch illustrating this geometry. Figures 4 and 5 are also referenced in relation to the thin-shell phenomenon and the non-uniform variance of features (suggesting an ellipsoid rather than a hypersphere). Figure 2 shows linear separability.\n*   **Offset from the origin helps mitigate false negatives and control sharpness:** This is discussed in Section 4, and Figure 7 illustrates the concept of blur control through sphere offset.\n*   **Frequent concepts are embedded closer to the mean vector (semantic blurring):** This is hypothesized in Section 4 and linked to the non-origin-centered geometry. The experiments confirm the better alignment of frequent concepts to the mean vector.\n*   **Strong correlation between conformity and cosine similarity to the mean vector:** The paper explicitly states a Pearson correlation of 0.9998 on MS-COCO in Section 4, supported by Figure 9.\n*   **Modality gap helps in aligning conformity distributions:** This is argued in Section 6.2 and visually supported by Figure 11, showing the KL-divergence of conformity distributions as a function of the mean offset.\n*   **vSLERP leverages CLIP's latent space geometry for semantic editing:** Figure 12, Figure 23, and Figure 24 provide visual examples of the vSLERP method in action.\n*   **Conformity as a measure of expressiveness:** Section 7.1 proposes this, supported by conformity measurements on generated images (Figure 13). Lower conformity means diversity\n\nNeed more evidence:\n\n*   **The extent to which the embedding geometry \"explains\" the modality gap and narrow cone effect:** While the paper links these phenomena to the identified geometry, the depth of this explanation and whether it's fully convincing might require more detailed theoretical grounding.\n*   **The \"optimality\" claimed for various aspects of CLIP's geometry:** The term \"optimal\" can be strong. The evidence would need to clearly demonstrate that the observed structure indeed leads to the best possible performance in relevant aspects (e.g., loss, alignment, uniformity, conformity matching) compared to alternative geometries.", "methods_and_evaluation_criteria": "Methods:\n\nThe paper is theoretical analysis for the CLIP geometry. The major proposed method is Conformity and Estimated Conformity.\nConformity defines a quantitative measure for how common or unique an embedding is.\nEstimated Conformity is based on cosine similarity to the mean vector proposed a computationally efficient surrogate for conformity. It makes practical sense for large datasets and real-world applications. The strong correlation demonstrated with the actual conformity measure supports its validity.\n\n\nEvaluation: \n\n- Using a standard and widely used image-text dataset like MS-COCO for statistical analysis of CLIP embeddings is appropriate for understanding general properties.\n- Using KL-divergence to quantify the difference between the conformity distributions of image and text modalities is a standard and meaningful metric for comparing probability distributions. This makes sense in the context of analyzing the modality gap.\n- The visual examples provided for vSLERP (Figure 12, 23, 24) are impressive to demonstrate its potential for preserving object identity during interpolation, which is a key challenge in semantic editing.\n- Using the introduced \"conformity\" metric to assess the diversity of generated images and diversity of the generated captions makes more sense", "theoretical_claims": "The mathematical definitions are sound. theoretical claims were derived on the basis of analysis and observation. Thus no proofs to check.", "experimental_designs_or_analyses": "Sound:\n\n- Analyzing Feature Statistics (Figure 2, 4, 5, 15, 16, 17, 18): check\n\n- Linear Separability Analysis (Figure 2, 15, 17): check\n\n- Thin Shell Phenomenon Analysis (Figure 4, 16, 18): check\n\n- Conformity Analysis (Figure 9, 10): check\n\n- Conformity as a Measure of Expressiveness (Figure 13): check\n\n- Vertical SLERP (vSLERP) Demonstration (Figure 12, 23, 24): check\n\nMinor Questionable:\n\n- Though I believe there may not be much difference in conclusion, I am still wondering how MS-COCO training data differs. \n- Will different image/text backbone matter? Or only the contrastive training nature matters?", "relation_to_broader_scientific_literature": "- The modality gap, where embeddings from different modalities (image and text) are separated, and the narrow cone effect, where features occupy a limited angular space, have been identified and studied in CLIP in previous works. The paper offers a geometric explanation for these phenomena based on the identified non-origin-centered ellipsoid shells for each modality.\n\n- The introduction of \"conformity\" as a measure of expressiveness and diversity in generative models (like unCLIP and Glide) is a novel contribution. This helps to evaluate the diversity of generated content which is crucial for generative models.\n\n- The issue of false negatives (semantically similar pairs treated as negatives) is a recognized challenge in contrastive learning. The paper suggests that the non-origin-centered ellipsoid geometry inherently helps in handling false negatives by allowing for \"semantic blurring\", where uncertain instances are embedded closer to the mean. This offers a different perspective – that the embedding structure itself plays a role in managing this issue, rather than solely relying on training procedures or loss modifications.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Originality: \n- Modality gap is already discovered, but this paper added a geometric explanation\n- The introduction of \"conformity\" as a measure of how common or unique an embedding is within a dataset is an original concept.\n- Proposed applications: 1. measure of diversity 2. vSLERP are very impressive to make CLIP a better tool for wider community.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Open questions:\n\n- How can the analysis in this paper help with \n1. a more efficient CLIP training?\n2. a higher quality CLIP training?\n\n- Can we potentially offset the ellipsoid to mitigate the modality gap to make the similarity between different modalitiy-pairs comparable?\ne.g. \nImage(Dog A), Image(Dog B) has cosine distance a\nImage(Dog A), Text(Dog B) has cosine distance b\nDue to the modality gap, cosine distance a !=b. and it is meaningless to compare a and b due to different modality. What if I want to make them comparable?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kd6Szi5UeL", "reviewer": "Reviewer_UMeX", "summary": "This paper investigates the geometric properties of the CLIP embedding space, proposing that image and text modalities form independent double-ellipsoid structures displaced from the origin. The authors argue that this structure improves the performance of contrastive learning and provides explanations for previously known phenomena such as the modality gap and narrow cone effect.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- Limited novelty: as other reviewers pointed out, we study well known phenomena using a different lens, i.e. a geometric perspective. As far as we know, we are the first to analyze the raw features prior to the normalization phase. The normalized features are forming a unit hypersphere by definition, overshadowing the complicated double-ellipsoid structure lost in the projection stage. Thus, since no other paper investigate the raw features, we are not familiar with thin shell explanation prior to us (including in \"Mind the Gap\" where the words \"thin\" as well as \"shell\" are absent in the entire paper). Moreover, the work shows novel geometric findings which capture the popularity of concepts, referred by us as “conformity”.\n\n- Conformity essentially yield a scalar per feature correlated with an image/caption measuring its popularity (its average similarity with other concepts drawn at random). Unlike cosine similarity, which yields a similarity measure between two instances (image and/or text) as input, conformity has a single input. We show conformity is almost perfectly aligned with cosine similarity to the mean of the ellipsoid of the respective modality. This finding has two significant consequences: From an applied perspective, it is fast and easy to compute (only one cosine distance computation with a given vector). From a contrastive-learning perspective, we highlight an interesting novel phenomenon: frequent concepts are embedded closer to the modality mean. We also explain the rationale, by reducing the loss of false negatives, which are expected to occur more for frequent concepts.\n\n- Our primary goal is to better understand the existing embedding space of CLIP, therefore we refrained from training or changing it. The natural follow-up study could be to leverage this new knowledge to retrain CLIP in a better fashion in some sense. We share directions for better training with reviewer MmZu. Regarding vSLERP, the main contribution is the preservation of the object appearance, which poses a challenge in real image editing.\n\n- Supp. are provided. According to the ICML policy, it appears following the main paper and includes additional clarifications and visualizations. We kindly encourage the reviewer to examine it.\n\n\n- \"Mind the Gap\" (Liang 2022) is indeed a pioneering paper in this field, consequently it is cited 12 times! (on pages 1,2,3,4,7 and 8, in some pages multiple times). We believe we carefully gave it the appropriate credit.", "claims_and_evidence": "The main claim of the paper is that the CLIP embedding space exhibits a double-ellipsoid geometry rather than the simpler hyperspherical structure typically assumed in prior literature. However, this claim suffers from significant shortcomings:\n- The key finding of the double-ellipsoid geometry largely reiterates existing results from prior studies, notably \"Mind the Gap\" (Liang et al., 2022), which has already thoroughly explored the modality gap and thin-shell phenomena. Thus, the contribution of this work does not clearly differentiate itself from earlier research, casting substantial doubt on the novelty and significance of the proposed ellipsoidal geometry.\n- The \"Conformity\" concept introduced by the authors shows an extremely high Pearson correlation (0.9998) with the cosine similarity to the mean vector, questioning its practical novelty. This implies that the Conformity metric essentially duplicates existing cosine similarity measures and lacks a compelling justification for its introduction as a distinct concept.", "methods_and_evaluation_criteria": "- The experimental analyses conducted in this study are limited exclusively to a few samples in the MS-COCO dataset without any training or new empirical results.\n- The proposed vertical SLERP (vSLERP) method lacks sufficient empirical evaluation. The authors fail to provide clear quantitative evidence of improvement over the traditional SLERP method or adequately explain theoretically why vSLERP better leverages the geometric properties of CLIP embeddings.", "theoretical_claims": "I have no issues here.", "experimental_designs_or_analyses": "- The claim that the double-ellipsoid structure alleviates the False Negative problem in contrastive learning is not convincingly supported by empirical evidence. The authors do not disentangle the independent effects of the ellipsoidal structure itself from the impact of displacement from the origin, leaving the exact reason for any observed improvements unclear.", "relation_to_broader_scientific_literature": "- Efforts to understand CLIP embeddings were widely explored before, however, I do not agree that this paper brings new things.", "essential_references_not_discussed": "\"Mind the Gap\" (Liang et al., 2022) would be one of the essential references. This paper already provides a comprehensive visual and analytical examination of the modality gap phenomenon in CLIP, making it directly relevant and necessary to cite given the main claims of this study.", "other_strengths_and_weaknesses": "see the above", "other_comments_or_suggestions": "see the above", "questions_for_authors": "see the above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S6WupYoJI4", "reviewer": "Reviewer_qxx2", "summary": "The work analyses the geometric properties of CLIP embeddings.  It builds upon previous work that studied the modality gap in embeddings.  The main finding is that image and text embeddings both live within separate ellipsoid thin shells in high dimensional embedding space.  The authors demonstrate that this particular geometric configuration arises from the contrastive loss function and noise in the training dataset, i.e. patches with similar meaning that are not dedicated pairs (and thus used as \"false\" negatives in the loss).  The paper introduces a measure of \"conformity\", indicating that embeddings of common themes (in image or text space) should be close to the mean embedding vector.  Based on this definition, the authors provide a geometric justification for the modality gap, i.e. the present non-overlapping distributions for text and image embeddings minimize the KL divergence of the conformity distributions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback. The applications section is indeed small since we devoted our paper to the geometric and statistical analysis, which we found the most important to delve in, explain and support experimentally. See additional comments embedded in our answers to the other two reviewers, which we do not want to repeat here.\n\nThe applicative examples are preliminary, still we found them exciting and useful. Studying these applications more thoroughly and suggesting additional ones would require an in-depth analysis which we believe would better fit a dedicated paper. Thanks again for your feedback.", "claims_and_evidence": "Numerous claims are made in the manuscript, which are all supported well.  I particularly enjoyed the simple but effective geometric interpretation of the results, e.g. Fig 6 or 11.", "methods_and_evaluation_criteria": "The geometric analysis is based on the theory of random vectors in high dimensional spaces and thus well grounded.  Experiments are conducted on the MS-COCO dataset which has 328k images accompanied by natural language descriptions and thus seems adequate for the task.", "theoretical_claims": "Claims are argued for by geometric interpretations and experiments, which appear correct.  The definition of conformity Eq. 10 is plausible The validity of its approximation Eq. 11 demonstrated by data, Fig. 9", "experimental_designs_or_analyses": "See other comments. The paper itself is a detailed problem analysis.", "relation_to_broader_scientific_literature": "Yes.  The paper presents interesting findings, which are relevant for anyone using a contrastive loss across modalities in ML.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "+ Well written paper. The analysis is easy to follow.\n\n- The applications of Sec. 7 are only exemplary and further usage of the introduced conformity measure remains unclear.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper analyzes the geometry of CLIP embeddings, showing that image and text embeddings lie on linearly separable ellipsoid shells. This paper introduces conformity to measure embedding similarity and explains how this structure manages uncertainty and aligns modality distributions. This paper offers an analysis of the modality gap and contrastive learning dynamics.

This paper has mixed opinions (2 Accept and 1 Reject). The AC and the reviewers had discussions, but their opinions didn't reach a consensus. More specifically, Reviewer UMeX pointed out three issues:

1. The conformity metric and analyzing raw features are not novel
2. Experiments are limited to the small subset of MS-COCO
3. The visual evidence is insufficient to demonstrate a clear practical advantage of the proposed method.

In terms of novelty, I think that the result of this paper is sufficiently novel compared to "Mind the Gap" (Liang et al., 2022). Although the "thin-shell phenomena" itself was revealed in Mind the Gap paper, I believe that the theoretical approach of this paper is different from the Mind the Gap paper (despite that the claim itself is not novel). I asked Reviewer UMeX to provide more evidence to support the "lack of novelty" comment, but I couldn't get a response from the reviewer. As far as the AC knows, the theoretical analysis of this paper (specifically, linking the false negative problem and conformity) is a sufficiently novel result.

Reviewer MmZu also has the same perspective on point 2 but stands an opposite site to point 3. -- Reviewer MmZu clarified that the experimental results are "good to have" but not critical. Reviewer qxx2 also clarified that the current experimental validation is limited, but the reviewer clarified that they think this work as a detailed problem analysis rather than the presentation of a solution to a certain problem, hence the evaluation is not a critical issue. I slightly lean to the other reviewers' opinions, but lack of the empirical validation could be a valid argument.

For the third issue, I think Reviewer UMeX's opinion is sound.

Overall, I think this paper is a borderline paper where the advantages of this paper (providing a new lens of VL embeddings) slightly outweigh its disadvantages (insufficient empirical results). I recommend "Weak Accept" for this paper; I don't think this paper is strong enough to be accepted, but if there is a room in the program, I am willing to recommend Accept.
Decision: Accept (poster)