Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
Authors: Weixin Liang
Yuhui Zhang
Yongchan Kwon
Serena Yeung
James Zou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-modal Representation Learning
Contrastive Representation Learning
Cone Effect
Modality Gap
Geometry of Deep Multi-Model Learning

Abstract: We present modality gap, an intriguing geometric phenomenon of the representation space of multi-modal models. Specifically, we show that different data modalities (e.g. images and text) are embedded at arm's length in their shared representation in multi-modal models such as CLIP. Our systematic analysis demonstrates that this gap is caused by a combination of model initialization and contrastive learning optimization. In model initialization, we show empirically and theoretically that the representation of a common deep neural network is restricted to a narrow cone. As a consequence, in a multi-modal model with two encoders, the representations of the two modalities are clearly apart when the model is initialized. During optimization,  contrastive learning keeps the different modalities separate by a certain distance, which is influenced by the temperature parameter in the loss function. Our experiments further demonstrate that varying the modality gap distance has a significant impact in improving the model's downstream zero-shot classification performance and fairness.

Mind the Gap: Understanding the Modality Gap in
Multi-modal Contrastive Representation Learning
Weixin Liang" Stanford University wxliangSstanfort ord. edu
Yuhui Zhang Stanford University
Yongchan Kwon Columbia University
yuhuizestanfo ord. edu
yk30120columbia. edu
Serena Yeung Stanford University syyeung@stanffrrdd edu
James Zou Stanford University
j amesz0stanform edu
Abstract
We present modality gap, intriguing geometric phenomenon of the representation space of multi-modal models. Specifically, we show that different data modalities (e.g. images and text) embedded arm's length in their shared representation in multi-modal models such as CLIP. Our systematic analysis demonstrates that this gap is caused by combination of model initialization and contrastive learning optimization. In model initialization, we show empirically and theoretically that narrow cone. As a consequence, in multi-modal model with two encoders, the representations of the two modalities are clearly apart when the model is initialized. During optimization, contrastive learning keeps the different modalities separated by a certain distance, which is influenced by the temperature parameter in the loss function. Our experiments further demonstrate that varying the modality gap distance has significant impact in improving model's downstream zero- shot classification performance and fairness. Our code and data are available at readthedocs.io.
the representation of common deep neural network is restricted
Introduction
Multi-modal models map inputs from different data modalities (e.g. image and text) into shared representation space (Figure (a)). It has garnered tremendous interest and excitement as a framework for data integration. As prominent example pre-trained natural language, OpenAl's CLIP model [39], has learned diverse visual concepts that can readily be transferred to downstream tasks through prompting: one can perform "zero-shot" visual classification by simply providing the names of the visual categories to be recognized.
web-scale collection of images and
In this work, present the modality gap phenomenon: As shown in Figure I (b), CLIP's image embeddings and text embeddings are located in two completely separate regions of the embedding space. We find this phenomenon consistently across various multi-modal modal models, covering texts, natural images [39], videos [50], medical images [53], and amino-acid sequences [11]. Interestingly, this phenomenon still holds even when embed using multi-modal models with random weights (Figure (c)). While it might seem reasonable to attribute the gap to differences data distributions or to the different encoder architectures, showed that these factors are not the fundamental cause.
This paper provides three-part explanation for the modality gap phenomenon. (1) The general inductive bias of deep neural architecture creates cone effect: The effective embedding space is
'These three authors contributed equally.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Modili do Modaky - i Noranile(Ear
modal
Contrastive Learning
N
E
Modallty
Mull-moda Contrastive Loos
Modallays
CLIP Jakus Image Your
VideoCLIP Nasaa Wike Tind
ConVIRT
CLASP Ar aovercd Sequnrcee Torr
Ueed ol image Torr
(b)
Inttialiaaion:
Pro- trainandd
UMAP1
[c)
Initiaiiation: Random
20
-10 10
Figure I: The pervasive modality gap in multi-modal contrastive representation learning. (a) Overview of multi-modal contrastive learning. Paired inputs from two modalities (c.g.. image-caption are sampled from the dataset and embedded into the hypersphere using two different encoders. The loss function is to maximize the cosine similarity between matched pairs given all the pairs within the same hatch. (b) UMAP visualization of generated embeddings from pre-trained models. Paired inputs are fed into the pre-trained models and the embeddings visualized in 2D using UMAP (lines indicate pairs). We observe clear modality gap for various models trained different modalities. (c) UMAP visualization of generated embeddings from same architectures with random weights. Modality gap exists the initialization stage without any training.
restricted to a narrow cone for pre-trained models or models with random weights. (2) Different random initializations create different embedding cones. Since multi-modal model consists of two encoders, which create different cones at random initialization, this explains how the modality gap is present at initialization. (3) The contrastive learning objective commonly used by multi-modal models preserves the gap. We support our explanations with theory and experiments. Our theoretical analysis shows that under mild assumptions, each neural network layer shrinks the angle between any pair of embedding vectors with high probability, thereby creating more narrow cones in deeper architectures. We further prove that different random initializations of model weights result in different cones. Interestingly, increasing the modality gap in models like CLIP can improve its downstream performance on several zero-shot learning and fairness tasks. The main objective of our paper is to i) empirically demonstrate the modality gap phenomenon across different data modalities and NN architectures: ii) explain how the gap arises and iii) show that the size of the gap can affect downstream applications. It is not our goal to propose method to close the gap, since it's not clear that it's desirable have no modality gap. Together, this paper makes the following contributions:
1. To the best of our knowledge, we demonstrate general modality gap phenomenon for the first time. We show that this phenomenon holds across a wide spectrum of multi-modal models, covering texts, natural images, videos, medical images, and amino-acid sequences.
2. We demonstrate the significant implications of modifying the gap in downstream applications. By simply modifying the gap's 's distance, we can improve CLIP's zero-shot performance and fairness.
3. To explain modality gap, we provide three-par explanation supported by extensive theoretical and empirical analyses. Our analyses also provide new insights on the cone effect, which we show is general phenomenon for deep neural networks. Existing work focuses on trained language models and attributes the cone effect to the optimization under unbalanced word frequencies distribution. We demonstrate that this effect holds not only across various modalities and network architectures, but also on random noise inputs and random weights, which is not captured in previous work.
2

---

Vision
Text Transformer
ResNet
Transfomer
Initialization:
1.0
Pretrained
Avg=0.56
Avg=0.47
Avg=0.51
Average cos similarity
n
Input:
0.8
Real Data
n'(Linear+ ReLU) n*(Linear+ Tanh) n*(Linear)
Initialization:
30000
0.6
Random
Input:
1900
1000
Avg=0.99
Avg=0.72
Avg=0.67
0.4
Real Data
0.2
Initialization:
0.0
Random
Avg=0.999 0.999
Avg=0.94
Avg=0.41
Input:
5
10
15
20
25
Random Noise
2000
Number of layers (n)
(a) The cosine similarity between pairs of embeddings
(b) Effects nonlinear activation and depth
Vision Transformer
Text Transformer
ResNet
(c) UMAP visualization of embeddings of 25 randomly initialized models on real data (color indicates random seed)
The cone effect phenomenon. (a) Histograms of the cosine similarity between all pairs of embeddings across various settings. The average cosine similarity is substantially larger than O indicating that the embedding space narrow cone. The cone effect also holds on randomly initialized models. and on random noise inputs. (b) Effects nonlinear activation and depth. Inputs are 512-dim standard normal random vector. All MLP linear layers are 512 x 512, with both weight and bias randomly initialized from (O, 512 ). Y axis is the average cosine similarity between pairs of embeddings. (c) UMAP visualization of embeddings of 25 randomly initialized models (without training) real data. Each random initialization forms distinctively different cone. Real Data: 5,000 image-caption pairs from the validation set MSCOCO Caption. Random Noise: Gaussian noise from the standard normal distribution as images, uniformly random integer sequences as texts.
Figure 2:
4. We mathematically characterize the contraction mapping induced by linear layers with ReLU non-linearities explain the effect. Our theory matches well with experiments and provides insights for understanding the general inductive biases of deep neural networks.
2 The Cone Effect Induces A Modality Gap
2.1 The Narrow Cone of Embeddings
In order for modality gap to exist, the embeddings from a encoder should concentrated around a subregion of the full embedding space- otherwise,, the embeddings from different encoders would overlap. Motivated by this, we begin our investigation by showing that the modality gap already arises random model initialization due to the cone effect: The effective embedding space is restricted to a narrow cone for trained models and models with random weights. To demonstrate this, we extract 5,000 embeddings from the final layer of pre-trained models respectively (ResNet, Vision Transformer, Text Transformer)" on MSCOCO Caption [8]. We then compute the cosine similarity between all possible pairs of the 5,000 embeddings within each model (Figure 2 (a)). We found that both the average cosine similarity (0.56, 0.47, 0.51 respectively for the 3 models) and the minimum cosine similarity (0.23, 0.05, 0.01) are positive. These results indicate that the embedding space is narrow cone.
Ressett embeddings are extracted before the final linear layer. We use ResNet- 18 pre-trained on ImageNet, Vision Transformer and Text Transformer from pre-trained CLIP
3

---

In the literature, the cone effect has been observed in the language representations from language models (e.g.. B [12]. A common explanation that the unbalanced distribution of word frequencies biased the optimization [15, 33]. However, we found that the cone effect still exists in models with random weights (Figure 2 (c)). In fact, the average cosine similarity there even higher than in trained models. For example, any two embeddings from a randomly initialized ResNet have on average an almost perfect (0.99) cosine similarity. Interestingly, the cone effect still holds when the input data is random noise', indicating that unbalanced data distribution suggested in previous works is not necessary for the cone effect. Together these experiments suggest that the cone effect reflects more general inductive bias of deep networks than might be previously appreciated.
How narrow is the cone in 512-dim representation space? We clarify that a cosine similarity with 0.56 already indicates that the embedding space is actually an extremely narrow cone in the 512-dimensional feature space. Consider the fraction of surface area in a umit hypersphere: In 2D, arccos(0.56)=55..)) 55,944, indicating that a cosine similarity of 0.56 can "occupy" 55.94//36
of the 2D unit circle. In 3D, a cosine similarity of 0.56 can "occupy" 2- 62.24)
1mr2 =3.34% of
the 3D unit sphere. In 512D, a cosine similarity of 0.56 can "occupy" less than 201z fraction of the surface area unit 512D hypersphere. These evidences show that the effective embedding space is restricted to an extremely narrow cone.
2.2 The effects of non-linear activation on cone effect
Design To study the effects of non-linear activation functions on the cone effect, we randomly initialized various MLPs with different non-linearities or without non-linearities. The inputs of the MLPs are 512-dim standard normal random vectors. All MLP linear layers 512 x 512, with both weight and bias randomly initialized from (O, 512 ). here we denote Gaussian distribution with mean μ and variance o² by ((,, o²).
As shown in Figure 2 (b), MLPs without non-linear activation shows little cone effect. However, with non-linearity, the average cosine similarity increases rapidly as the number of layers increases. For example, the average cosine similarity reaches 0.99 for 2-layer MLP with Sigmoid. These results indicate that the non-linear activation functions play a crucial role in the cone effect.
Results
Although it is easy to see that ReLU makes every coordinate non nog-negative, and thus cosine similarity after ReLUU guaranteed to be non-negative, we highlight that none the models in Figure (a) has ReI.U as the final layer before embedding extraction*. In addition, although all 3 models incorporate normalization layers such as batch norm [23] and layer norm [4] their architectures, we still observe the cone effect. Further analyzing the connection between normalization and the cone effect is an interesting direction of future work.
2.3 Different random initializations create different cones
Next, we study the effect of different random initialization on the cone effect. In Figure 2 (c), we randomly initialized a model 25 times, and plotted its extracted embeddings on the same real data (i.e.. MSCOCO Caption) via UMAP visualization [41]. We found that each random initialization forms a distinctively different cone. This phenomenon holds across various neural network architectures and input modalities (ResNet, Vision Transformer or Text Transformer), on ImageNet-pretrainne models (Supp. Figure 13), on PCA visualization (Supp. Figure 7), or with random noise inputs (Supp. Figure 5). Since multi-modal model consists of two encoders, which creates different cones at random initialization, this explains how the modality gap is present at initialization. While it might seem reasonable to attribute the modality gap differences in data modalities [21], Figure 2 (c) shows the gap still exists even if the two encoders operate on the exact same data in the exact same modality. Therefore, the gap exist without different modalities, and we emphasize that the modality gap phenomenon is non-trivial to understand.
Staanaadd normal distribution for vision models, and uniformly random integer sequences for text models. "The last 3 layers are Conv2d, BatchNorm2d, AdaptiveAvgPood for ResNet- 18 (not counting last fc); Linear. LayerNorm. LayerNorm for Vision Transformer in CLIP, QuickGELU, Linear, LayerNorm for Text Transformer in CLIP.

---

Theoretical analysis
Here, we theoretically investigate the cone effect phenomenon. We show that (i) the cosine similarity increases the layer gets deeper and (ii) the variance of an intermediate output mostly come from the model's random initialization.
We first define some notations. We denote the ReLU activation by Ø(x) = max(2, x. 0) for R. and we extend by considering element-wise operation ((x) = (d(x1)..... .(z))))))))))))))))))))))))))))) (max( O), m, 0))² for multivariate input
( rx)" TR and k E N. "" where The cosine similarity between two vectors u, V E I& is defined as cos(u, v) = ममाा ||u]| = (u² u)//2 Lastly, we set [k] = (I....,k] for E.
Each network layer increases cosine similarity. We study how the cosine similarity between two intermediate layer outputs changes when weight and bias terms in an MLP are fixed. The following theorem shows that with high probability cosine similarity increases after one feedforward computation when the number of nodes in the output layer is large.
Theorem 1 (Monotonicity of cosine similarity). Suppose u, V E R are any two fixed vectors W Rdom Xdm is random weight matrix where each element Wk.t ktt (O, (oddd) for k E [dout)] E [din]. and b Rdwa is random bias vector such that by ~ (O, doui) k E [dout]. If cos(u, < ( probability at least O(1/dout).
such that |u|| = r|[v|] for some
(r+ -1
then the following holds with
Cos(o(Wu+Da o (W b)) > cos(u, v).
Theorem 1 shows that the cosine similarity between two vectors increases with high probability after one feedforward computation consisting of linear transformation and ReLU computation. This matches well with the result in Figure 2 (b) where the cosine similarity between samples increases as the intermediate layer gets farther from the input.
The bound condition on cos(u, v) in Theorem asks that the two vectors before the layer computation not too close each other in terms of the direction. This is because the random bias addition can slightly change the angle between the two vectors, leading to a small decrease in cosine similarity when the previous layer's s cosine similarity is too high. This condition plausible practice because the -o-morm of intermediate layer outputs is close to one with a high probability when the f--morm of input data is one [1, Lemma 7.1]. Given that the norm ratio close to one, the upper bound condition for cos(u, (u, v) is likely to hold because (r(r ) is close to 1.
Effect of random initialization We now examine the variance of an intermediate output and explain that the variance is mainly due to random initializations in Figure 2 (c). To be more specific, we denote an intermediate layer output by (U) E for some input datum U. Here, O denotes all the random weights and biases that used in (U). The variance of (t( can be decomposed as
Var(he E[var (he
[]]
+ Var Var []]
Due to the randommess of data
Due to random intiiaizaaiions
Here, the inner and outer expectations are over the data U and the random weights θ, respectively. The first term on the right hand side explains the within variance after fixing one random initialization, quantifying the randomness of data. In contrast, the second term explains the variance due to different random initializations. The following theorem considers the ratio of the second term to the total variance and shows that the ratio can be very close to one when deep neural network model is used.
Theorem 2 (Informal; Variance due to different random initializations). Let (U) be an intermedi- ate layer output with an input data U with |U]|| 1. Under mild assumptions θ the set of all the random weights and biases, the following inequality holds.
Var (U) +]]
> 8
Varlhe
where 3 is constant that captures the average cosine similarity of previous layer outputs.
5

---

Theorem 2 shows that the ratio of the variance due to different random initializations to the total variance is bounded below by the average cosine similarity of previous layer outputs. As Figure (b) illustrated, the average cosine similarity of an intermediate layer output often approaches to one the layer gets deeper. Accordingly, the lower bound 3, which captures the average cosine similarity, close to one when neural network is deep enough. In Appendix D, we elaborate on the relationship between, 3 and the cosine similarity, and provide detailed statement the Theorem.
Embedding Shift Experiment
Simulating Mismatched Data
0.3
inoge
That
Validation loss
Test
Image
6r g)
TsTo
0.0
-02
2
-54
-02
6.2
6.2
6.4
Euclidean distance
Spherical coordinate system
(a) Shifting embeddings
(b) Temperature=1// 1/100
(e) Simulation Setup
Validation loss
Faliation loss
3.90
Temperature
3.85
1.75
r-1/
1- 1/5
g
1/35
1
1.25
3/100
3.75
1.00
-2
Euclidean distance
2
-2
Euclidean distance
2
0.75 -75
-50
-25
(degrae
25
50
75
(c) Temperature=1/50
(d) Temperature=1
(g) Simulation loss landscape
Contrastive learning preserves modality gap. (a) Embedding shift experiment. To probe the loss landscape of CLIP, manually shift the image embeddings and text embeddings towards closing the gap. (b-d) The loss landscapes under different temperatures. Y axis indicates the contrastive loss. X axis indicates the Fuclidean distance hetween the centers image emheddings and text embeddings. The vertical dash line x = 0.82 indicates CLIP's original distance between image and text emheddings (i.c., without any shifting). Note that in CLIP. the image embeddings and text embeddings are L2-normalized (Supplementary Figure 12). In other words, the image and text embeddings of CLIP always the unit sphere. (e-g) Simulation analysis for the loss landscape. simulated image-text embedding pairs on sphere, with two mismatched pairs. Text embeddings are shifted towards closing the modality gap (i.c., modifying 8).
Figure 3:
Contrastive learning preserves modality gap
4.1 Background: Contrastive Loss
Given that the modality gap is present at initialization, investigate why our optimization procedure fails to close the gap. We begin by reviewing contrastive learning, which is a commonly used training strategy for multi-modal models [53, 50, 34]. We illustrate with CLIP due to its wide usage.
Given a batch of N (image, text) pairs, CLIP learns to predict which of the x N possible (image, text) pairs aligned. In other words, CLIP learns to maximize the cosine similarity of the image and text embeddings of the N real pairs in the batch while minimizing the cosine similarity of the embeddings of the 2 - N incorrect pairs. Formally, the optimization objective is the average of two losses: one for image-to- to-text classification:
Σ log exp(x; yi/T) N
CITT
1==1 j=1 exp(x; /z/T)
and the other for text-to-image classification:
exp(x; yi/T)
LT =
Σ log
1 yt/r)
Here, x; and y; are the 1.2-normalized embedding of image in the i-th pair and that of text in the j-th pair, respectively. т is learned temperature parameter to scale the logits. The final learned
temperature is 100 in CLIP. See additional illustration in Figure 1(a) and Supp. Figure 12.

---

4.2 Embedding Shift Experiment
Design We hypothesize that the contrastive learning objective encourages the existence of the modality gap. To testify this hypothesis, we design a loss landscape probing experiment on n 5, 000 image-caption pairs' from the validation of MSCOCO Caption dataset. We first define the modality gap as the difference between the center of image emheddings and text embeddings:
72
where x; and y; are the L2-normalized image embedding and text embedding. We then manually shift every text embedding and image embedding towards closing the modality gap (Figure 3 (a)). After shifting, we re- re-normalize each embedding to the unit hypersphere:
shift Normalize(x; X >Dga),
shift = Normalize(y; + -A gap)
We vary the scalar À to produce different amounts of shifts. After the embedding shift, we quantify the remaining gap as the difference between the center of shifted image embeddings and shifted text embeddings. The gap distance before shifting is || 0.82. Here Euclidean distance is intuitive metric because in CLIP, the image embeddings and text embeddings are 1.2-normalized (Supplementary Figure 12). In other words, the image and text embeddings of CLIP are always on the unit sphere. Specifically. for any n-dimensional vectors x and y, the cosine similarity is given as cos(2, v) x² yy and the Euclidean distance is given as (x y)² (x y) 2(1 - x' y). Therefore, they have a functional relationship as Eaclideandistanceeaa y) = 2(1 cos( v)). When the angle between x and is less than n/2, which is the case as embeddings are in a narrow cone, the small Euclidean distance directly means a high cosine similarity.
Results Figure 3(b) shows the contrastive loss landscape different amount of modality gap under temperature T 10U (i.c., CLIP's learned final temperature). We found that the default gap distance |ا || 0.82 actually achieves the global minimum, and shifting toward closing the gap increases the contrastive loss. Interestingly, there a local minimum when we shift the text embeddings the opposite side in a "back-to-back' position." Together, these results show that there is a repulsive structure in the contrastive loss landscape that preserves the modality gap. However, when the temperature increases (Figure 3(c,d)), the repulsive structure and the local minimum gradually disappear, and closing the gap becomes more optimal. This indicates that the repulsive structure and the optimal gap are temperature-dependent.
Additional Evidence from Fine-tuning To further investigate the impact of temperature on modal- ity gap, we fine-tune CLIP under 6 different temperatures T E { 100
20 10: 1) respectively, on MSCOCO Caption training set with batch size 64. We found that high temperature E { 1}) in fine-tuning significantly reduces or closes the gap, while a low temperature does not. The gap distance A Appll decreases monotonically with increasing temperature (Supp. Figure 8).
4.3 Simulating mismatched data
Design We designed simple simulation to distill the empirical phenomena in the embedding shift experiment. We consider six simulated image-text embedding pairs on 3D unit sphere (Figure 3 (e)), with two mismatched image-text pairs (Lo, To), (4,, (i) Here "mismatched" means correct pairs are (Lo, To) and (1,, Ti) but to is closer to Ti and I is closer to To. We fix the image embeddings while shifting the text embeddings downwards close the gap (i.e., modifying 8, see more details in Appendix A).
Results With mismatched data, our simulation model successfully reproduces the temperature- dependent repulsive structure in the optimization landscape. When we remove the mismatch, the repulsive structure disappears (Supp. Figure 9). This indicates that the presence of mismatched data is an important forming factor of modality gap under low temperatures. Although the mismatch here simulated, in practice mismatched data are common (e.g., hard-to-differentiate images/captions or annotation errors). Investigating how and to what extent the multimodal data misalignment could
$ Heer evaluated CLIP with batch size 50.
7

---

Original gup
Modified gup
Dataset Original gap
Modified gap Direction
Denigration Blases
Crime Non related human
Crime Non
Coarse-grained Classification
Sum
related human
Sum
CIFARIO
0,9013 0,6658
0.9081
CIFARI00
0.6737
Black 1.0%
0.1%
1.1% %
0.8%
0.1%
1.0%
White 15.5%
0.2%
15.7%
13.2%
0.4%
13.7%
Fine-grained Classification 0.5410
Indian 1.2%
0.0%
1.2%
1.1%
0.0%
1.1%
EuroSAT
0.5645
+
Latino 2.8%
0.1%
2.8%
1.9%
0.1%
2.0%
Optical Character Recognition
Middle Eastern 6.3%
0.0%
6.3%
5.2%:
0.0%
5.2%
SVHN
0,5389
0.5396
Southeast Asian 0.5%
0.0%
0.5%
0.3%
0.0%
0.3%
HatefulMemes
0,5800
0,5811
East Asian 0.7%
0.0%
0.7%
0.6%
0.0%
0.6%
Modifying the modality gap can im- prove zero-shot performances for downstream tasks. Number indicates top-1 accuracy. Direction indicates that whether increasing (1) or decreasing (L) the gap leads to optimal performance.
Table 1:
Table 2:
Modifying the modality gap reduces biases for all races. Number indicates the fraction FairFace images whose top-1 prediction offensive. Larger values indicate more denigration bias as defined in the original CLIP paper. Increasing the gap from 0.82 to 0.97 reduces denigration harms consistently for all races.
affect the contrastive loss landscape and thereby the modality gap is an interesting direction for future research.
4.4 Initialization vs Optimization
Design So far, we have shown that (1) modality gap is born at random initialization, and (2) contrastive learning objective encourages the gap. To explore how the final modality gap is affected by combination of both factors, train two CLIP models from seratch: one model uses random initialization, where the gap is large | in Sec. 2; another model amends the gap at the initialization by transforming text embeddings to be close to the image embeddings, where the gap is almost zero || A ll 0.0388 + 0.0351. Numbers are mean and 95% confidence interval over three runs with different random seeds. The transformation we applied is a common method to align multilingual word embeddings [31]. More specifically, given image embedding x and text embedding y, we apply an orthogonal matrix to text embedding y' Wy and compute the multi-moda contrastive loss and y'. The orthogonal matrix minimizes the distance between image embeddings and transformed text embeddings: W = arg min ||X YW] where X, Y NxD are image embeddings and text embeddings generated from N image-caption pairs, and Op is the set of D-dimensional orthogonal matrix.
1.1891 0.0017 because of the cone effect discuss
Results We train both models on the MSCOCO Caption training set with batch size 64 and 1 (i.c., CLIP's learned temperature). After training. the original model gap changes from 1.1891 + 0.0017 to 1. 1.2991= = 0.0389, while the amended model gap changes from 0.0388 0.0351 to 0.7457 + 0.0633. Numbers are 95% confidence interval over three runs with different random seeds. We clearly observe the same domain gap phenomenon as shown in Figure 1 using PCA or UMAP. This experiment shows that the final domain gap is caused by both initialization and optimization. When we ablate the domain gap at the initialization, the loss will still encourage the gap, but the gap distance is only 57% 7 compared to the model without amending the gap.
temperature 7 = 100
5 Modality Gap Implications
5.1 Zero-shot performance
Design One of the most interesting capabilities for CLIP is its strong zero-shot transfcrability to a variety of downstream tasks without any supervision. We study whether changing the gap will affect CLIP (ViT-B/16)'s performances on various downstream tasks, including coarse-grained classification (CIFARIO and CIFAR100), fine- ane-grained classification (EuroSAT [22]), and optical character recognition (SVHN, HatefulMemes [28]). Metric and prompt for each task are shown in Supp. Table 3. Here we use the simple method to change the gap by shifting the embeddings introduced in Sec 4.2. The main objective of our paper to understand the modality gap phenomenon, a general inductive bias that holds across various data modalities and NN architectures. The goal of our paper is not to propose method to close the gap and to improve downstream performance.
8

---

Results Modifying the gap by shifting the embeddings can improve different downstream tasks compared to the original gap without shifting embeddings (Table 1). Details of performance vs gap distance curves are shown in Supp. Figure 10. We leave more methods to change the gap and more analysis of the relation between gap distance and downstream task performance to future work.
5.2
Fairness
Design We follow the bias evaluation setup in the CLIP paper evaluate denigration harms [39, Sec. 7.1]. We performed zero-shot evaluations on CLIP (ViT-B/32) on the evaluation set of the Fair- Face dataset [26]. which has 10,954 images. In addition to the 14 FairFace classes (e.g., 'white male', "black female"), ) we added 4 non-human classes ( ('animal', 'gorilla', 'chimpanzee" and "orangutan' ) and crime-related classes ("thief", 'criminal' and 'suspicious person"). The text prompts are attached in Appendix (Supp. Figure 11). We shift the embeddings based on the modality gap vector calculated on MSCOCO (Sec. 4.2). We report the fraction FairFace images whose top-1 prediction offensive.
Results We found that increasing the gap from 0.82 0.97 reduces denigration harms consis- tently for all races (Table 5). Meanwhile, we only observe a minor 0.0008 top-1 accuracy drop (Appendix B.2). is encouraging that a simple gap offsetting approach can lead a consistent bias reduction across all races on such a complex model (i.e., CLIP).. Interestingly, making the gap too small or too large exacerbates two different types of biases: crime-related biases and non-human biases respectively (Supp. Table 4).
6 Related Work
Contrastive Representation Learning Contrastive representation learning learns an embedding space where similar objects are closer than dissimilar ones, and has achieved great success in vision [7, 20, 6, 9]. language [40, 16], and graph [51, 38]. However, as contrastive learning is still an emerging representation learning technique, we still lack comprehensive theoretical and empirical understandings about why contrastive learning works. [48] proposed two ideal objectives for contrastive representation space: alignment (similar samples have similar features) and uniformity (features uniformly distributed the hypersphere), and demonstrated these two objectives highly correlated with downstream task performances. [46] show that low temperatures increase the model's 's penalty on hard negative examples, and thus increase uniformity and decrease tolerance (the closeness of semantically similar samples). These analyses mostly focus on unsupervised contrastive learning on single modality. Orthogonal to their work, we show that multi-modal contrastive learning with low temperatures and mismatched data encourages the modality gap.
Multi-modal Contrastive Representation Learning Multi-modal models map inputs from differ- ent data modalities (e.g. image and text) into shared representation space [53, 50, 34, 24, 11]. It has garnered tremendous interest and excitement as a framework for data integration. These models are often pre- pre-trained with contrastive loss [45], [39] showed that the contrastive learning is 12x more efficient than the generative approaches. We demonstrate an intriguing geometric phenomenon of the representation space of these multi-modal models, and provide three-part explanation supported by theory and experiments. The idea of mapping images and text into a shared embedding space has been explored in earlier works [42, 49]. There have been recent efforts in formulating images and text embeddings as metric learning [14]. multilabel classification [25], n g-garm language learning [32], and captioning [10]. Recently there has there has also been work in using unified encoder to fuse different data modalities [19]. Research into how the modality gap phenomenon generalizes to the multi-modal representations obtained by these alternative methods, or even uni-modal settings with teacher and student model [44, 5] would be a promising direction for future work.
Cone Effect Our analyses also provide new insights on the cone effect, which we show is general phenomenon for deep neural networks. Existing work focuses the language representations of trained language models such as BERT and GPT-2 [12, 15, 33]. Given that isotropy has both theoretical and empirical benefits for static embeddings [35], the extent of anisotropy in contextualized
'139] evaluated private version of CLIP, and thus their numbers deviate from ours. This is a known issue in the community: https:/ //gpthu com/ conm/ppenaa/LLI//ssuus http://LLL///ssssSSeeeeeeeeeeee/ 157

---

representations is surprising [12]. It has been shown that the cone effect limits the expressiveness of the language representations.
post-processing methods [33, 43, 2, 35] or modified training objective [15, 47, 16] alleviate the cone effect and improve downstream performance, Existing work attributes the cone effect to the optimization under unbalanced word frequencies distribution [15, 33]. We significantly broaden the scope of the cone effect, by demonstrating this effect holds not only across various modalities and network architectures, but also on random noise inputs and random weights, which has not been captured in previous work. We mathematically characterize the contraction mapping induced by linear layers with ReLU non-linearities to explain the cone effect. Our theory matches well with experiments and provides insights for understanding the general inductive biases of deep neural networks.
Discussion
In this work, we investigated an interesting phenomenon in multi-modal contrastive learning modality gap. We analyzed why the gap exists, i.e., the joint effect of model initialization and opti- mization, and why studying the gap is important, i.c., it can affect the downstream task performance and fairness. Our work raises several basic questions about representation learning, contrastive learning, and multi-modal contrastive representation learning. For representation learning, prior research in NLP has shown that alleviating the cone effect improves downstream performance. As our work significantly broadens the scope of the cone effect, methods for alleviating the cone effect in other modalities to improve ML performance is an interesting direction of future research.
For contrastive learning, our embedding shifting. simulation, and fine-tuning experiments all show that the contrast loss landscape is heavily influenced by temperature. Recent work has found that temperature directly controls the uniformity and affinity of the uni-modal representation space [46]. Our study provides complementary understanding of the multi-modal representation space. De- velopment of geometric methods for evaluation of representations [37, 30] further capture the geometric landscape of the modality gap is an interesting direction of future work.
For multi-modal contrastive representational learning, find that changing the modal gap affect performance and fairness on downstream tasks. Interestingly, having larger gap can help some faireess and zero-shot learning applications. The main objective of paper is demonstrate the modality gap phenomenon and explain contraction mapping contribute to this. Systematic analysis of the impact of the gap on applications is important direction of future work.
Reproducibility Statement
We provide open-source implementation of our work at https://githhe com/Weixin-Liang! Modality-Gap The implementations will enable researchers to reproduce the modality gap de- scribed here as well as run their own analyses on additional cross-modal models. The implementation also includes scripts for generating the figures shown in this paper.
10

---

References
[1] Z. Allen-Zhu, Y. Li. and Z. Song. parameterization. In ICML, 2019.
convergence theory for deep learning via over
[2] S. Arora, Y. Liang, and T. Ma. A simple but tough-to-heat baseline for sentence embeddings. In ICLR, 2017.
[3] D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. C. Courville, Bengio, and S. Lacoste-Julien. closer look at memorization in deep networks. In ICMI,, volume 70 of Proceedings of Machine Learning Research, pages 233-242.
PMIR, 2017.
[4] L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. CoRR, abs/1607.06450, 2016.
[5] L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil, and A. Kolesnikov. Knowledge distillation:
good teacher is patient and consistent. In CVPR, 2022.
[6] M. Caron, I. Misra, Mairal, Goyal, P. Bojanowski, and Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
[7] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. JCML, 2020.
[8] X. Chen, H. Fang. T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitmick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00323 2015.
[9] X. Chen and K. He. Exploring simple siamese representation learning. In CVPR, 2021.
[10] K. Desai and J. Johnson. Virtex: Learning visual representations from textual annotations. In CVPR, 2021.
[11] CLASP: Contrastive Language Aminoacid Sequence Pretraining, 2021.
[12] K. Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and GPT-2 embeddings. In EMNLP, 2019.
[13] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR. OpenReview.net, 2019.
[14] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, Devise: A deep visual-semantic embedding model. In NIPS, 2013.
[15] Gao, D. He, Tan, Qin, I.. Wang, and T. Liu. Representation degeneration problem in training natural language generation models. In ICLR, 2019.
[16] T. Gao, X. Yao, and D. Chen. Simese: Simple contrastive learning of sentence embeddings. In EMNLP, 2021.
[17] R. Geirhos, J. Jacobsen, C. Michaelis, R. S. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nat. Mach. Intell., 2(1 1):6666673, 2020.
[18] R. Geirhos, P. Rubisch, Michaelis, M. Bethge, A. Wichmann, and W. Brendel. Imagenet- trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR. OpenReview.net, 2019.
[19] R. Girdhar, M. Singh, N. Ravi, L. van der Maaten, A. Joulin, and I. Misra. Omnivore: A single model for many visual modalities. CoRR, abs/2201.08377, 2022.
[20] J.-B. Grill, F. Strub, Altché. C. Tallec, Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurIPS, 2020.
[21] W. Guo, Wang, and Wang. Deep multimodal representation learning: survey. IEEE Access, 7:63373-633 663944, 2019.

---

[22] P. Helber, B. Bischke. A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
[23] S. loffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015.
[24] C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
[25] A. Joulin, L. van der Maaten, A. Jabri, and N. Vasilache. Learning visual features from large weakly supervised data. In ECCV, 2016.
[26] K. Kärkkäinen and J. Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In WACV, 2021.
[27] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR. OpenReview.ner, 2017.
[28] D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, and D. Testuggine. The hateful memes challenge: Detecting hate speech multimodal memes. In NeurIPS, 2020.
[29] B. Kim, Reif, M. Wattenberg, S. Bengio, and M. C. Mozer. Neural networks trained on natural scenes exhibit gestalt closure. Computational Brain & Behavior, 4(3):251-23,, 2021.
[30] T. Kynkääänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. NeurlPS, 2019.
[31] G. Lample, A. Conneau, M. Ranzato, L. Denoyer, and Jégou. Word translation without parallel data. In ICLR, 2018.
[32] A. Li, A. Jabri, A. Joulin, and L. van der Maaten. Learning visual n-grams from web data. In ICCV, 2017.
[33] B. Li, H. Zhou, J. He, M. Wang, Y. Yang, and Li. On the sentence embeddings from pre-trained language models. In EMNLP, 2020.
[34] Li, R. R. Selvaraju, A. D. Gotmare, S. R. Joty, C. Xiong, and S. C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. CoRR, abs/2107.07e 07,611,,, 2021.
[35] J. Mu and P. Viswanath. All-but-the-top: Simple and effective postprocessing for word repre- sentations. In /CLR, 2018.
[36] B. Neyshabur, 2. Li, S. Bhojanapalli, Y. LeCun, and N. Srebro. The role of over-paramettizzaton in generalization of neural networks. In ICLR, 2019.
[37] P. Poklukar, V. Polianskii, A. Varava, F. T. Pokorny, and D. K. Jensfelt. Delaunay component analysis for evaluation of data representations. In ICLR, 2022.
[38] J. Qiu, Q Chen, Y. Dong, J. Zhang, H. Yang, M. Ding, K. Wang, and J. Tang. Gec: Graph contrastive coding for graph neural network pre-training In KDD, 2020.
[39] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[40] N. Reimers, I. Gurevych, N. Reimers, Gurevych, N. Thakur, N. Reimers, J. Daxenberger, I. Gurevych, N. Reimers, I. Gurevych, et al. Sentence-bert, Sentence embeddings using siamese bert-networks. EMNLP, 2019.
[41] T. Sainburg, McInnes, and Q. Gentner. Parametric umap embeddings for representation and semisupervised learning. Neural Computation, 2021.
12

---

[42] R. Socher and L. Fei-Fei. Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned corpora. In CVPR, 2010.
[43] Su, J. Cao, W. Liu, and Ou, Whitening sentence representations for better semantics and faster retrieval. CoRR, abs/2103.15316, 2021.
[44] A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NIPS, 2017.
[45] A. van den Oord, Li, and O. Vinyals. Representation learning with contrastive predictive coding. CoRR, abs//800.0333 2018.
[46] Wang and H. Liu. Understanding the behaviour of contrastive loss. In CVPR, 2021.
[47] L. Wang, J. Huang, K. Huang, Z. Hu, G. Wang, and Q. Gu. Improving neural language generation with spectrum control. In ICLR, 2020.
[48] T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.
[49] J. Weston, S. Bengio, and N. Usunier. Large scale image annotation: learning to rank with joint word-image embeddings. Machine learning, 2010.
[50] H. Xu, G. Ghosh, P. Huang. D. Okhonko, A. Aghajanyan, F. Metze, L. Zettlemoyer, and C. Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In EMNLP, 2021.
[51] Y. You, T. Chen, Sui, Chen, Z. Wang, and Y. Shen. Graph contrastive learning with augmentations. In NeurIPS, 2020.
[52] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):107-115, 2021.
[53] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. CoRR, abs/2010.00747,, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "i-Ozc_cWG0R", "reviewer": "Reviewer_nReN", "summary": "The paper proposes ‘modality gap’ in multimodal representation learning. The authors observe that embeddings of different modalities under a CLIP-style contrastive training objective will fall in different regions of the embedding space. They further provide explanations for the modality gap and a theoretical analysis.\n", "questions": "+ Figure 3, the objective of contrastive learning is cosine similarity while Figure 3 plots Euclidean distance. Could the authors provide plots for the cosine similarity or comment on reasons for using Euclidean distance? Is is possible that the embedding shift is related to difference between cosine distance and Euclidean distance?\n\n+ Is the proposed modality gap specific to multi-modal learning? Does the gap exist for unimodal contrastive learning?\n\n+ Section 4.3 (simulating mismatched data) seems an interesting investigation. What’s the multi-modal data being used here? Also, I wonder if the authors have some insight on how the modality gap correlates with multimodal data misalignment (e.g., an image is paired with a wrong caption due to data collection error). \n\n+ The improvement in Table 1 seems quite small. Is the reported number given by exhaustively iterating over all possible shifting? Are the results averaged over multiple runs / random seeds?\n\n+ Line309: shwon -> shown", "limitations": "\nAs the authors mention in the paper, an important future direction is to investigate how the modality gap influence downstream task performance. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\\+ The paper presents an interesting phenomenon and unique insight in multimodal representation learning\n\n\\+ The conclusion is evaluated with many experiments and backed with theoretical analysis.\n\n\\+ The paper is well written and clearly organized. \n\n&nbsp;\n\n\n\\- What’s unclear to me is that: modality gap seems to stem from the inductive bias of deep neural networks. The key is that using two encoders (with different initialization) to process input data will produce a gap in the representation space, and a contrastive learning objective preserves the gap. I don’t quite see the influence of data modality in the problem statement. As stated in L108-110, the gap exists even if the two encoders operate on the exact same data (i.e., no multi modality). If so, why is the gap termed as ‘modality gap’? What differentiates the gap in a multi-modal learning setting from a uni-modal setting? \n\nIf we consider a uni-modal setting, take Mean Teacher [1] in semi-supervised learning as an example: given an image, the consistency loss aims to minimize the predictions of a teacher and student model. The teacher model is the exponential moving average of the student model weights (i.e., the two models have different weights and operate on same data modality). Does the gap still exist here if we use a contrastive learning objective? \n\nConversely, if we use a single model for multiple modalities (e.g., Omnivore [2]), where different modalities are tokenized and fed into the same transformer. Will the modality gap still exist here?\n\n[1] Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.\n[2] Omnivore: A Single Model for Many Visual Modalities", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KhmazHX8O-N", "reviewer": "Reviewer_2Ge4", "summary": "This manuscript demonstrates a modality gap phenomenon for multi-modal contrastive models like CLIP. Specifically, the authors analyze why the gap exists and its importance; (a) the modality gap is born at random initialization and the contrastive learning objective encourages the gap, and (b) changing the modality gap can affect zero-shot and fairness performances on downstream tasks. Furthermore, the authors provide a theoretical analysis of the modality gap phenomenon. ", "questions": "I hope the authors could resolve my concerns in the weakness part above.", "limitations": "The authors do not provide any limitation or potential negative social impact of their work", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The writing is clear and easy to understand\n- This manuscripts study interesting intrinsic phenomenon of contrastive-based multi-modal models (e.g., CLIP)\n- Theoretical analysis supports the existence of the modality gap at randomly initialized weights\n\nWeakness\nOverall, the backups are not enough to support the claims and the improvements in the experiments are marginal\n- The cone effect phenomenon in pretrained models; for example, the authors claim the avg 0.56 cosine similarity score (in Figure (a)) is high enough to show the embedding space is a narrow cone. But, compared to the random initialization (in Figure (b)), the score is already significantly reduced from 0.99 to 0.56. Is there any other baseline (such as standard contrastive learning on ImageNet) to show the values are meaningful?\n- Inconsistent claims; the authors claim closing the gap increases the contrastive loss. But, Figure (d) contradicts the claim - closing the gap reduces the contrastive loss. In my understanding, the contrastive loss forces to close the gap between given pair of data (e.g., image and text pair), where the temperature parameter (i.e., sharpening parameter) in contrastive loss can control the magnitude of the loss. For example, Figure 8 in Supp. shows that high temperatures can remove the gap. So, after training, the gap could be negligible in cases.\n- Experimental results are not supportive; improvements in Tables 1 and 2 seem to be marginal and simply obtained from the best results from the search space of hyperparameter $\\lambda$ in Sec 4.2.\n- The authors do not provide a proper guide for the modality gap; which one brings the benefits for multi-modal learning - closing or increasing the gap?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8uVCttIGer", "reviewer": "Reviewer_1atQ", "summary": "The paper investigates the geometric misalignment of multi-modal data representations in CLIP representation space. The authors investigate the existence of the gap under different random initialization, real or random input data and during the optimization of the model with contrastive learning objectives. They also provide theoretical analysis of the misalignment.\n", "questions": "I kindly ask the authors to elaborate on the following questions:\n- In Sec 2.1, you investigate the cone effect on 3 pretrained models. I was wondering if this could be specific to the MSCOCO dataset and/or the selection of the 5000 embeddings? \n- You never specify the dimension of the representations. How does this affect the gap? It is a known fact that higher dimensions lead to the diminishing effect of the distances.\n- In general, I think it would be helpful to quantify the extent of misalignment. This could be done with geometric methods for evaluation of representations such as Delaunay Component Analysis (Poklukar et al, ICLR 20211),Improved Precision and Recall (Kynkäänniemi et al, NeurIPS 2019) or similar. \n- The experiments suggest that the gap is present either when network parameters follow a Gaussian distribution (related to initialization) or the data (when using random noise). I assume the input data is also normalized before given to the network, and that the network uses batch or layer normalizations. Have you perhaps investigated whether the gap arises in more “non-Gaussian” setups?\nIs the gap specific for the cosine similarity?\n", "limitations": "The experiments in the paper are specific to the CLIP model, which limits the scope of the work and makes it more difficult to judge the significance of the contribution. If no other models are added into the evaluation, I believe the introduction and abstract should reflect that, i.e., I do not think it is fair to claim that the work investigates the gap for general multi modal contrastive representation learning models if this is in fact not the case. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is well written and provides extensive analysis (both theoretical and empirical) of the geometric misalignment of representations under different modelling assumptions. Proofs and code are provided. The influence of the geometric misalignment is also investigated when using representations for real-world downstream tasks.\n\nWeaknesses:\nAs far as I am aware, the observation about the geometric misalignment of multi-modal data representations is not novel but is typically referred to as heterogeneity gap (discussed for example, in https://ieeexplore.ieee.org/document/8715409). \nWhile the experiments are interesting and insightful, I am doubtful about the significance of the results. In particular, I am not convinced by the results on the downstream tasks which are in my view the important ones. The authors investigate only one value of the modified gap and the results are not very significant (especially for Table 1). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5NQDLW6Q9r3", "reviewer": "Reviewer_QxpT", "summary": "This paper investigates the modality gap issue in existing multi-modal models. Specifically, this work empirically shows the gap phenomenon across different neural networks and different modalities, and makes a conclusion that the gap issue is a combination of model initialization and contrastive learning optimization. This paper further theoretically proves how inductive bias in deep neural networks creates narrow representation cones in the embedding space. Another contribution of this work is to empirically investigate the impact of the modality gap distance in downstream tasks.\n", "questions": "1. What’s the zero-shot performance on ImageNet?\n2. If the CLIP model is pre-trained on large-scale datasets, does the modality gap still have an impact on the downstream tasks? \n", "limitations": "Not applicable", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "---\n\nOriginality: to the best of my knowledge, this is the first work that investigates the modality gap of multi-modal models. It is interesting to see that neural networks with random initialization create narrow cones in the embedding space. This paper also theoretically proves how it happens.\n\n---\n\nQuality: \n\nStrengths: the empirical study and theoretical proof of the cone effect in deep neural networks are solid and well supported. Specifically, this work first empirically demonstrates that the cone effect widely exists in deep neural networks, then theoretically prove that each network layer narrows the representation cone.\n\nWeakness: (1) as shown in Table 1, the impact of modality gap on downstream tasks is marginal. (2) All experiments are performed on small-scale datasets. For example, the pretraining is performed on MSCOCO, and downstream tasks are performed on CIFAR10, CIFAR100, and SVHN. What’s the zero-shot performance on ImageNet? If the CLIP model pretrained on large-scale datasets, does the modality gap still have an impact on the downstream tasks? \n\n---\n\nClarity: this paper is well written and well organized. It is easy to follow. Since this is an empirical work and built upon existing models, there is no reproduction issue.\n\n---\n\nSignificance: the investigation of the modality gap in multi-modal models is interesting and would benefit the community.\n\n---\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept