Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data
Authors: Yuhui Zhang
Elaine Sui
Serena Yeung
Venue: ICLR 2024 poster (2024)
Keywords: multi-modal contrastive learning
captioning
text-to-image generation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Our work explains the geometry of multi-modal contrastive representation space and introduces a three-step method to bridge the modality gap, achieving state-of-the-art results in zero-shot captioning and text-to-image generation.

Abstract: Building cross-modal applications is challenging due to limited paired multi-modal data. Recent works have shown that leveraging a pre-trained multi-modal contrastive representation space enables cross-modal tasks to be learned from uni-modal data. This is based on the assumption that contrastive optimization makes embeddings from different modalities interchangeable. However, this assumption is under-explored due to the poorly understood geometry of the multi-modal contrastive space, where a modality gap exists. In our study, we provide a theoretical explanation of this space's geometry and introduce a three-step method, $C^3$ (Connect, Collapse, Corrupt), to bridge the modality gap, enhancing the interchangeability of embeddings. Our $C^3$ method significantly improves cross-modal learning from uni-modal data, achieving state-of-the-art results on zero-shot image / audio / video captioning and text-to-image generation.

Published as a conference paper at ICLR 2024
Connect, Collapse, Corrupt: LEARNING CROSS-MODAL TASKS WITH UNI-MODAL DATA
Yuhui Zhang', Elaine Sui", Serena Yeung-Levy Stanford University
(yuhuiz, esui, syyeung)ecs stanford.edi syyeuggBesssstaffrrd..dd
ABSTRACT
Building cross-modal applications is challenging due to limited paired multi- modal data. Recent works have shown that leveraging pre-trained multi-modal contrastive representation space enables cross-modal tasks to be learned from uni- modal data. This is based on the assumption that contrastive optimization makes embeddings from different modalities interchangeable. However, this assump- tion is under-explored due to the poorly understood geometry of the multi-modal contrastive space, where a modality gap exists. In our study, we provide a the- oretical explanation of this space's geometry and introduce a three-step method, C (Connect, Collapse, Corrupt), to bridge the modality gap, enhancing the inter- changeability of embeddings. Our C method significantly improves cross-modal learning from uni-modal data, achieving state-of-the-ar results on zero-shot im- age audio video captioning and text-to-image generation.
INTRODUCTION
Building cross-modal applications often face a critical challenge: the scarcity of paired multi-modal data. Large- scale multi-modal contrastive learning has emerged as promising approach to address this gap. Pioneering works like CLIP (Radford et al., 2021) and ImageBind (Girdhar jet al. 2023) offer publicly available representation spaces learned from contrasting millions of web-based images and texts. These spaces that align embeddings of con- cepts across different modalities pave the way for lever- aging abundant uni-modal data in cross-modal tasks. For instance, instead of training image captioning system on image embeddings, one could train it on caption embed- dings. Likewise, text-to-image generation models can uti- lize image embeddings as opposed to text. During cross- modal inference, embeddings from the other modality are simply input into the model (Figure [ Notably, this ap- proach has achieved impressive results in image captioning (Li ct al. 2023 Tam et al. 2023; Nukrai et al. 2022) and text-to-image generation (Zhou al. 2022c a b), climinating the need for multi- modal paired data.
for
ex
cuto
oute
fy
ey
Uni-modal Training
Cross-modal Evaluation
Interchangeable of em- beddings enables learning cross-modal tasks with uni-modal data.
Figure 1:
This aforementioned process is based the hypothesis that image embeddings and text embed- dings can be used interchangeably in the multi-modal representation space. However, the validity of this assumption is not well understood. Recent works reveal that the resulting geometry from multi-modal contrastive learning is nontrivial - corresponding image and text embeddings do not necessarily collapse to the same points the space. Instead, there significant gap between embeddings from different modalities, potentially hindering the direct interchangeable use of image and text embeddings (Liang al., 2022 Zhang et al., 2023). The lack of comprehension of the re- sulting geometry from contrastive learning makes it challenging to design methods to leverage this representation space and learn cross- cros-modaa tasks from uni-modal data.
"Equal contribution. Project page available at httpt:t///////.... //oe: zh15. github. 1o/C3- wwt.c//eb//ttttttee website/

---

Published as a conference paper at ICLR 2024
In this study, conclude that the geometry of multi- modal contrastive representation space as (Figure 2
Modality Embedding
Cn
Exg
Ex - By = C1 +e
cute cat
where e. and Ey denote the embeddings of paired inputs from different modalities, CL is constant vector repre- senting the modality gap. which is orthogonal to the em- bedding span of ez and ey. and is random vector representing the alignment noise, which can be approx- imated by Gaussian distribution.
C1 c
Modality Embedding
%
smail
Figure 2: Geometry of the multi-modal contrastive representation space.
We provide a theoretical explanation of the above geometry. Specifically, the modality gap emerges during initialization as certain dimensions of image and text embeddings remain approximately constant in the embedding space, and the constants are distinct for images and texts due to separate initializations. During optimization, these constant dimensions lack a gradient that pushes them to align to the same value, leading to the preservation and orthogonality of the modality gap. Mean- while, the alignment noise arises from the stable region produced by the contrastive loss, where points within certain range result in a loss near zero, causing the optimization to halt.
Based on this understanding of the geometry, we propose simple method, Cs, to improve the interchangeability of embeddings from different modalities, thereby enabling the learning of cross- modal tasks with uni-modal data. C consists three steps:
1. Connect: Related concepts from different modalities connected via multi-modal contrastive learning, resulting in a shared representation space that can allow the interchangeability of em- beddings from the different modalities. However, modality gap and alignment noise exist.
2. Collapse: Based on its geometry, the modality gap can be closed by subtracting the embedding mean from each modality, eliminating the most dominant distributional difference between them. Additional noise is introduced as regularization during training to improve the perfor- mance and robustness of learning cross-modal tasks from uni-modal data, given the alignment noise from multi-modal contrastive learning.
3. Corrupt:
We demonstrate the effectiveness and broad generalization of C3 on four tasks: image, audio, video captioning and text-to-image generation, and achieve state-of-the-art performance on zero-shot eval- uation settings when trained solely on uni-modal data. We also provide a detailed analysis of each component that contributes to performance improvements. Our method sheds new light on the possi- bility of achieving cross-modal tasks under a low-data regime, and our theoretical analysis provides insight into understanding multi-moda contrastive learning.
Our contributions are three-fold:
1. We provide a theoretical explanation of the representation space geometry resulting from multi- modal contrastive learning.
Based on this geometry, we propose simple three-step solution to enhance the interchangeability of embeddings from different modalities, improving cross-modal learning with uni-modal data.
3. We show the effectiveness of our method on image / audio video captioning and text-to-image generation, achieving state-of-the-art results.
LEARNING CROSS-MODAL TASKS WITH UNI-MODAL DATA
In this section, we present the general notion of cross-modal task and how we can leverage uni- modal data to learn such tasks.
2.1 CROSS-MODAL TASK FORMULATION
Cross-modal tasks aim to learn model that maps inputs from one modality X' (e.g., images) to
another modality y (e.g., texts). Given paired multi-modal dataset D {(x.8) y) E x )} (e.g., an image-caption dataset), the task can be achieved by minimizing the empirical risk La between
the predicted target ò = ((fx(=)) and the true target y over the dataset D, where fx : X + Rd is
encoder that maps inputs from X' d-dimensional representation space, and g : Rd - y is
2

---

Published as a conference paper at ICLR 2024
decoder that maps outputs from the encoder to
min a.fx
C)lo.v), (E.N)ED
Σ
i - )-((fx(x)
D
La measures the discrepancy between the predicted and true targets, which can be mean squared error (MSE) for images and cross- cross-entropy loss for texts.
2.2 ENABLING CROSS-MODAL TASKS WITH UNI-MODAL DATA
The need for multi- modtll paired dataset D to learn cross-modal tasks is suboptimal, as collecting such datasets can be expensive and time-consuming. However, if we have encoder fy y Rdd that maps inputs from y to the same representation space as the encoder fx. i.e., Va, y
D, fx()) (V), can train the modal task using uni-modal dataset D = {y E 37:
min
Σ La(ū.v), WED!
ÿ= )=g(v(y))
D
Note that fy should be frozen during training maintain its embedding alignment with fx. When evaluating in a cross-modal setting, we can replace fx with fy, thus enabling cross-modal tasks with only uni-modal training data.
2.3 ESTABLISHING SHARED REPRESENTATION SPACE
Recent advances in multi-modal contrastive learning has enabled for encoders that map similar in- puts from different modalities to shared representation space. Specifically, given large multi- modal datase(" paired inputs are randomly sampled during each iteration and the following ob- jective is optimized (Radford et al., (022):
2n Σ6 log exp(er, Cy./T) + log EJ=1 explee, ey,/T)
1
exp(ex, Cyi/T) EJ= exp(ez, ey./T)
min L fx.fy
where ex [x(x). By fy(v), and is the temperature hyperparameter. This loss function encourages high similarity between the embeddings of the paired image-texts relative to the similar- ities between unpaired ones. Intuitively, after optimizing the loss, paired image and text embeddings should collapse to the same point. However, empirical results have shown significant gap between paired embeddings (Liang al., 2022), which prevents the direct interchangeable use of image and text embeddings. In the next section, we provide a detailed analysis of the geometry of the multi-modal contrastive representation space.
MULTI-MODAL CONTRASTIVE REPRESENTATION SPACE GEOMETRY
We begin by providing the following proposition that describes the geometry of the multi-modal contrastive representation space. These findings inform our general method of how to adapt uni- modal data for cross-modal learning.
Proposition 1. (Multi-modal Contrastive Representation Space Geometry)
Given paired image x and text y, the relationship between the (z-normalized image embedding ez and text embedding ey obtained from mulli-modal contrustive learning can be described as:
e₂ - Cy = c +
where 1 a constant vector representing the modality gap and is orthogonal to the image and text embedding span, i.e., Yea, Exy, C.1 ((e, Cn,) 0; (0,, o2 I) is random Gaussian vector representing the alignment noise.
In the following subsections, we will first explain why the modality gap exists before and after model optimization. Then, we will introduce the alignment noise and relation to the temperature parameter in contrastive loss.
AAquuirigg domain- domain-speciffc paired mulli-modaa datasets can be challenging. However, several works have gathered large-scale noisy image-caption pairs from the weh and made pre-trained encoders fx and fy avail- able for direct use.
3

---

Published as a conference paper at ICLR 2024
3.1 MODALITY GAP
The presence of modality gap and its orthogonality to the image and text embedding span are due to the joint effect of initialization and optimization.
The modality gap exists when randomly initializing multi-modal models, which can be explained by the dimensional collapse (Jing al. 2022 of the representation space defined below.
Initialization.
Definition 1, (Dimensional Collapse of the Representation Space)
Given a d-dimensional representation space Rd we define its effective dimension de as:
2i
de arg min d'
2i=1o
where the o; 's are the singular values of the representation covariance matrix in decreasing order, and thresholds the minimum variance explained by the de dimensions. Dimensional collapse occurs when de <
To demonstrate the dimensional collapse phe- nomenon, we took a randomly initialized CLIP with a d 512 representation space and fed MS-COCO images and captions as input. We
Loy of Singula Volue
Image Embeddings (Ra (Ra-stom ELI on COCO)
Text Embeddiings (Random CL on COCO!
obtained the corresponding image features and text features and performed SVD on the im- age feature and text feature covariance matri- ces. The distribution of the singular values is shown in Figure 3 where that the effective dimension of both the image and text features is small. Specifically, when setting y = 0.99, the effective dimension of image embeddings de.z
-15
400
Singular Value
clearly see
Figure 3: Dimensional collapse of the CLIP representation space. Singular values obtained from SVD reveal that the effective dimension of the image and text representation space is much smaller than the total number of dimensions.
and that of text embeddings is de,y = 230. Therefore, the ef- fective dimension of the shared representation
space de < de + de.y = 255. The equality holds only when all the effective dimensions of the image and text are orthogonal.
Dimensional collapse indicates that only small number of dimensions contribute significantly to the variance of the representation space, while the remaining dimensions can be viewed as constant.
Suppose the shared representation space has maximum effective dimension de 255. This indi- cates that the image and text embeddings will remain constant in the de = d - de = 257 ineffective dimensions. As result, modality gap exists at the beginning of model optimization, ineffective dimensions will be inherently different for images and texts, given random initialization.
these d,
To verify this, we synthesize n 1, 0000 image and text embeddings in d 512 space. For each image embedding, we initialize the first de., dimensions, and for each text embedding, the -e. to de.. dimensions. by randomly sampling a standard Gaussian distribution. (O 1). All the other dimensions are set to constant value drawn from a (O, 1). where the constant for the image and text embeddings are different. Figure 4 (left) illustrates this setup by showing the variance of each dimension. This setup mimics our findings of SVD analysis CLIP, where we set image em- beddings to have de, effective dimensions and text embeddings to have de,y, effective dimensions, and assume these effective dimensions are fully orthogonal. We normalize the embeddings to unit length before performing our analysis.
We observe clear modality gap at the beginning: the fy-distance between the mean of the image embeddings and the mean of the text embeddings is 1.21. When we only consider the last de ineffecttive dimensions, the average distance is 0.99. More discussion is in Appendix!!
Optimization. The analysis above reveals that the modality gap exists at initialization, and we further analyze why optimizing for the multi-modal contrastive loss fails to close the gap. The following lemma reveals that there is no gradient in the modality gap direction, therefore the gap and its orthogonality will be preserved.
4

---

Published as a conference paper at ICLR 2024
Log aaraaacc
-10
Synthet Image Fmmbeddings Synthecc Rat Enbecdings
Log of Norfance
Synthetic Emboddings
Embbedding
-12.5
-20
-15.0
-17.5
-25
-20.0
300 200 300
400
300
100
200
300
400
500
Dimmnsion Indix
Dimensioon Endkex
Figure 4: Variance of each dimension before (left) and after (right) multi- contrastive optimization. Our analysis reveals that gradients will only be propagated to effective dimensions and no gradient will be propagated to ineffective dimensions. Therefore, the effective dimensions are aligned while ineffective dimensions remain constant after optimization.
Lemma 1. (Gradients in Contrastive Optimization) (Proof in Appendix B
With the mild assumption of equal presence of n. images and texts with p(x:) = p(y:) =
1 1/n, optimizing the multi-modal contrustive loss = 2n Ei=1 (log ΣE exples, (ey/T) +
expler, (8,,
explex, ex/t)) log En explea (mp(ea)Bu/17)) (//lt)) yields the following gradients:
n Ve,, L
Vea, L -=
>= ys (eys - -yy),
Σ j=1
az (ezz Ezi)
/=1
where x = 1/(2nT), az, = p(az + p( ay, p(uslzs) + p(zalB5). p(z.lu)) =
explex; /T, Eest explext -ep; /T)) p(yi(or))
exp(e ex,/T)
= E.., exp(evs /7) and is temperature.
Lemma [ highlights that the gradients of image embeddings during contrastive optimization are fully determined by the text embedding span, and vice versa. Due to the dimensional collapse of the image and text embedding span, the contrastive optimization process fails to propagate gradients in the direction of the ineffective dimensions, resulting in gap preservation and orthogonality to the image and text embedding span after optimization. Lemma I also implies that the effective dimensionality of the joint representation space remains unchanged after optimization.
To empirically verify this, we optimize the multi-modal contrastive loss L on the n = 1, 000 pre- viously synthesized image and text embeddings. We optimize for 200K steps with learning rate 0.1, and CLIP's initial temperature of т = 0.07. We compute the variance of each dimension pre- and post-contrastive optimization and plot the results in Figure 4 (right). From the figure, we can see that the first de = 255 effective dimensions are aligned after optimization while the last de = 257 ineffective ones remain unchanged. This verifies that no gradient is propagated to the ineffective dimensions, as the variance for these dimensions remains zero after optimization. The modality gap becomes slightly smaller (0.82 compared to 0.99 before optimization) mainly due to fz-regularization, where changes in the effective dimensions affect changes in the ineffective ones.
In summary, due to dimensional collapse at model initialization, there are ineffective dimensions where image and text embeddings can be viewed as different constants, resulting in modality gap at initialization. During optimization, these ineffective dimensions have no gradient update, and thus, the modality gap and its orthogonality preserved after optimization.
3.2 ALIGNMENT NOISE
In this section, we explain alignment noise after multi-modal contrastive learning. This noise re- sults from the stable region of contrastive loss demonstrated in the following lemma, where we can consider a single term in the loss given the symmetry of the added terms.
Lemma 2. (Stable Region Controlled by Temperature) ( (Proof in Appendix
B
explez, -v, We consider a single term in the multi-modal contrastive loss L, = - log Σ exples, exples,eev//7 /v, We define the margin r = ex; Bey - max jti ez; By, as the measure of the similarity difference between the matched pair and the hardest negative pair. When r exceeds threshold given below, L, falls below a small pre-set value 8, where we assume optimization ends:
o(T)
log
exp(6) 1'
5

---

Published as a conference paper at ICLR 2024
where o(T) is monotonically increasing function of temperature that satisfies < o(T) <
Therefore, the required margin r is monotonically increasing with T.
Lemma suggests that there is stable gion of the contrastive loss. This region can be viewed function of temperature where region size increases as the temperature de- creases, with the required margin becoming smaller. Within this stable region, the loss falls below the small value 8, indicating that opti- mization has ended. In the extreme case where + -0+, if r 0, the loss will be less than (o + 0+ for this case). This means that given all the y/j 2i can end up within a region instead of fixed point, resulting in the same zero loss. Figure iillsstras the stable region defined by the margin with regard to the temperature. Therefore, there may be mismatch between the matched pairs in the representation space, resulting in alignment noise.
Stable region 7,77 maxa, tlagofri/expii0
Low
Temperature
High
Figure 5: Stable region (green area) of contrastive learning controlled by temperature. Within the stable region, the loss falls below a small preset value, indicating that optimization has ended. The region increases as the temperature decreases.
3.3 SUMMARY
Statistic
Mean
Std
In Section 3.1 and 3.2
explain the modality gap e and alignment noise in Proposition respectively. Combining them together, we explain the geometric relation of paired embeddings ex Ey C1 t
||
0.83 0.01
cos(d (a
0.99 0.00
cost (1
0.00 0.06
Ele cos
0.00 0.00
We verify this geometric relation using CLIP on the MS-COCO image- caption dataset. We randomly group each 100 images into group i, and define individual gap ( ex, Cg
0.00 0.10
(f) group gap d = EE (1) image difference (1) = e Cag- alignment noise of = d d where
Table 1: Statistics that reveals representation space geometry.
j, k are image or text indices. These statistics are computed in Table where the first three statistics show that the modality gap e approximates constant vector orthogonal to the image and text em- bedding span, and the last two statistics show that the alignment noise can be viewed as Gaussian noise. We provide detailed explanation of how interpret these statistics in Appendix C
This geometric analysis in this section serves as the foundation of the approach we introduce in the next section, where we develop a simple method align the shared representation space and enable learning cross modal tasks with uni-modal data.
4 CONNECT, COLLAPSE, CORRUPT
reveals the geometry of the multi-modal contrastive representation space. Based this, we propose three steps, connect, collapse, corrupt (C9), to align the repre- sentation space, making it possible for embeddings from different modalities to interchangeably consumed by the decoder and thus enabling learning cross-modal tasks from uni-modal data.
Proposition from Section 3
Stage 1: Connect. This stage establishes connections between similar concepts across different modalities. We leverage recent advances in multi-modal contrastive learning (Radford et 2021) and use encoders trained with this strategy to build cross-modal models. However, a modality gap and alignment noise exists after multi-modal contrastive learning, as shown Proposition I
Stage 2: Collapse When directly using embeddings of different modalities as input, there is drastic degradation in performance due to the modality gap, which causes input distributions the decoder to differ. To address this issue, we adopt simple approach proposed by (Zhang et al.) 2023) that effectively removes the modality gap. Specifically, during training, place of e,, we feed in e' e, [e,, to the decoder, and during inference with another modality, in place of ey, we feed in e', = e, Ey [ey]. This approach collapses the modality gap, eliminating the input distribution mismatch between the two modalities:
e/2 ey =eex eyy (Ey(er) Ryleyl)

---

Published as a conference paper at ICLR 2024
Stage 3: Corrupt After removing the modality gap, there is still alignment noise which can be approximated by a (N I)). During unsupervised training, instead of directly decoding y from e we add explicit Gaussian noise to the imput and decode from en = e, + following (Nukrai eet al. 2022 Zhou et al. 2022c). By introducing this noise, the uni-modal and multi-modal train ing processes become similar, and the learned decoder is more robust and invariant to the small perturbation (00 I)) leading to improved performance.
Appendix Algorithm summarizes the entire procedure of our proposed method, C that enables learning cross-modal tasks with uni-modal data.
RESULTS
In this section, we verify the effectiveness of our proposed method, C on four tasks: image cap- tioning, audio captioning, video captioning, and text-to-image generation. We show that our method achieves state-of-the-ar performances, generalizes to different modalities and contrastive embed- ding spaces, and especially useful when multi-modal data are limited.
5.1 IMAGE CAPTIONING
We use the ClipCap model (Mokady et al. 2021), pairing frozen CLIP ViT-B/32 image en- coder (Radford et al. 202T) with a GPT-2 decoder (Rafford et al. 2019). A lightweight MLP mapping network bridges the dimensional gap between CLIP (512-d) and GPT-2 (768-d) embed- dings and also produces a prefix for GPT-2 caption generation. We train and evaluate on the MS- COCO dataset (Lin et al., 2014) using the standard split (Karpathy& & Fei-Fei 2015), comprising 113K training images and 5K each for validation and testing, with each image having 5 captions. We utilize metries such as BLEU (Papineni et al. 2002) and ROUGE (Lin 2004) evaluate lexical and semantic similarity between generated and human captions.
We first train our model for text reconstruction using the MS-COCO captions only. Following C we extract the text embedding from the frozen CLIP text encoder and apply the collapse operation (remove pre-computed mean) and corrupt operation (add Gaussian noise). After training, we eval- uate our model in the cross-modal setting, replacing the text encoder with the CLIP image encoder and decoding captions from image embeddings. We refer to this evaluation setting as image-free zero-shot evaluation, as images are not seen during training. Additionally, we fine-tune the pre- trained model on different amounts of image-caption pairs and evaluate its performance. We refer to this evaluation setting semi-supervised evaluation. More details can be found in Appendix E We show image-free zero-shot captioning results in Table 2 and semi-supervised captioning results in Figure 6
C achieves state-of-the-art image-free zero-shot captioning results. As shown in Table 2 proposed method, C3, outperforms previous state-of-the-arr methods in image-free captioning. De- tails of the other methods be found in Appendix Section A Our ablation analysis demonstrates that both the collapse and corrupt components are crucial for improving cross-modal evaluation per- formance, as they eliminate the differences between embeddings from different modalities. Notably, the most competitive baseline, CapDec (Nukrai et al.) (022). can be viewed as an ablated version of C3, but without an analysis of why the corruption works. Our proposed method, however, pro- vides a clear explanation based on a geometric analysis of the multi-modal embedding space, and we further improve performance by introducing a collapse step. Overall, Ca represents potential standard approach for future works that use a multi-modal contrastive embedding space.
C is particularly useful in low-data regimes. In scenarios where multi-modal data is limited, our method remains highly effective. To demonstrate this, we fine-tuned our pre-trained model on 1%, 5%, 25%, and 100% of the MS-COCO training image-text pairs, and compared our perfor- mance with a fully supervised baseline (ClipCap) and ablated models (see Figure (). The results clearly show that 3 outperforms the fully supervised baseline across all metrics, with the most sig- nificant improvements seen in low-data regimes where multi-modal paired data are limited. Thus, method represents promising solution for achieving cross-modal tasks in such scenarios.
7

---

Published as a conference paper at ICLR 2024
Method
Conn. Coll. Corr. BLEU-I, BLEU4. METEOR, ROUGE-L, CIDER, SPICE;
Baselines
ZeroCap 2022
x
49.8
7.0
15.4
31.8
34.5
MAGIC 2022
X
56.8
12.9
17.4
39.9
49.3
11.3
ESPER 2022
X
21.9
21.9
78.2
CLIPRe
2023
J
4.6
13.3
25.6
9.2
DeCap [023
8.9
17.5
50.6
13.1
WS-ClipCap (2023)
50.3
9.6
15.2
37.5
33.7
8.6
WS-ClipCap-Nee 2023
65.5
22.1
22.2
48.0
74.6
14.9
CapDec (2022)
69.2
26.4
25.1
51.8
91.8
Ours
O
28.1
2.4
12.2
25.4
13.0
O7
44.4
6.1
33.6
25.2
69.0
25.5
24.3
50.8
87.6
17.6
71.0
27.7=001
25.0+00
52.0
93.3
18.3
Table 2: Image-free image-to-texx captioning results. We achieve state-of-the-art zero-shot image captioning and our ablation shows the effectiveness of each component in our method. Baseline results are from the original paper, where some metrics were not reported. Std with 3 runs reported.
ClipCap
(Connect)
C? (Connect Collapse)
Cj [Connect& & Corrupt)
32.5
110
30.0
27.5
CIDEF
100
SPICE
90
22.5
Fine Fine-tunin
% buning
tuning
Figure 6: Image-to-ter te-text captioning results in the low data regime. When paired multi-men -ooal data are limited, our approach that leverages uni-modal data for pre-training leads to substantial improve- ments compared to the purely supervised method (ClipCap).
Qualitative analysis of collapse and corrupt. Both the collapse and corrupt components of our method show consistent improvements, but is not immediately clear how they do so. To address this, we provide qualitative results in Appendix [ We can see that after collapsing, the generated captions are much more natural and fluent, it removes the most significant distributional difference between image and text embeddings. After corrupting, the model generates more accurate and faithful text descriptions of the image. We hypothesize that adding noise makes the decoder robust to small variations in the embedding space. Therefore when evaluating in cross- cross-moda modal settings, the alignment noise will not affect the prediction and thus reduces hallucination the generated caption.
5.2 TEXT-TO-IMAGE GENERATION
We further apply our method, C3, to text-to-image generation, the reverse of image captioning. We uti- lize LAFITE (Zhou et al. 2022c). which integrates frozen CLIP V/T- Radford al., (021) as the text encoder and modified StyleGAN2's gen- erator (Karras al. 2020) the trainable decoder.
Method
Conn. Coll. Corr. FID, IS;
Baselines
DALL-E 2021
X
x
27.5
17.9
Cie
ZOZT
X
x
27.1
18.2
LAFITE
20Z2c
20.9
24.9
We use the MS-COCO dataset (Lin et al. 2014) with LAFITE's official split, including 82K training and 40K validation images, with 5 captions each. We use the standard metrics such as FID (Fréchet Inception Distance) (Heusel et al. 2017) and IS (In- ception Score) (Salimans et al. 2016) assess the realism of generated images. Similar to the im- age captioning setting, we first train the model for image reconstruction using images only, and then evaluate the model in the cross-modal setting by language-free zero-shot image generation results in
Ours
C
/
x
x
29.8
22.4
Ca
/
21.7
24.4
19.8
25.5
19.6
26.0
Table 3: Language-free text-to-image generation results. Our method C consis- tently outperforms the baselines.
generating images from text. We show the Table 3 Similar to image captioning, find
8

---

Published as a conference paper at ICLR 2024
Image Captioning (MS-COCO 2014
Audio Captioning (Clotho 2020
Video Captioning (MSR-VT 2016
BLEU-1, METEOR- ROUGE-L,
BLEU-1,
METEOR- ROUGE-L,
BLEU-I, METEOR; ROUGE-L;
C
33.5
12.9
25.8
21.8
17.3
18.1
16.7
12.6
15.2
Ci
53.8
17.4
38.6
26.7
20.0
21.4
25.1
17.8
23.1
64.4
22.2
45.8
27.6
20.0
20.6
25.2
18.1
23.8
74.0
26.6
54.0
29.5
20.1
23.0
31.4
20.0
26.9
Table 4: Generalization of C other modalities, datasets, and contrastive embedding spaces.
that our method C consistently outperforms the baselines in terms of FID and IS. Our ablation study further reveals that each component of C is useful for improving the performance. More detailed setups and qualitative comparisons can be found in Appendix
5.3 GENERALIZATION TO OTHER MODALITIES AND EMBEDDING SPACES
To verify the generalization of our method to other modalities, datasets, and embedding spaces, further conduct experiments on zero-shot captioning from image, audio and video using ImageBind (Girdhar et al. 2023) embeddings. We use the same settings as image captioning with CLIP. Results are shown in Table We find that C consistently improves baselines in all the achidves achieves further improvements in image captioning
settings, and using ImageBind embeddings compared CLIP embeddings.
6 RELATED WORKS (FULL VERSION IN APPENDIX A
Multi-modal contrastive learning and resulting geometry. Multi-modal contrastive learning aims to bridge representations from different modalities, drawing similar concepts closer and dis- tancing dissimilar ones (Radford et al., 2021). CLIP (Radford et al. 2021), ImageBind (Girdhar al., 2023). and similar models have leveraged extensive multi-modal data construct such rep- resentation spaces, which have been demonstrated to effectively support a range of uni-modal and multi-modal applications (Wortsman al., 2022 Ramesh et al.] [222]] [hen al.] (022). How- ever, the resulting geometry in the shared representation space, particularly the "modality gap". where embeddings from different modalities are clearly separate the shared representation space, remains under-explored (Liang et al., 2022] /Zhng et al. (203)). In our work, we unify the obser- vations from Liang et al. (2022) and Zhang et al. (2023). and contribute a formal formulation and theoretical explanation of the unique geometry resulting from multi-modal contrastive learning.
Learning cross-modal tasks with uni-modal data. Given the expense of multi-modal data col- lection, there growing interest in learning cross-modal tasks using uni-modal data. Based on the assumption that contrastive optimization makes representations from different modalities inter- changeable, recent works have leveraged these representation spaces and shown great success in building image captioning models with text data only (Tam et al.] 2023] Li et al. 2023 Nukrai (ct al., 2022) and text-to-image generation models with image data only (Zhou et al. 2022c b). De- spite these advancements, these methods have noted the intriguing "modality gap" phenomenon and proposed different empirical methods to address this gap. such as (Tam et al. 2023) 's paraphrased decoding, (Li et al. 2023] Zhou et al. 2022a)'s memory retrieval, (Nukrai al. [022]] Zhou al. 2022ch's noise addition, and (Zhou et al. (022b) 's prior network. In our work, we first provide a theoretical analysis of the multi-modal representation space geometry. Based the geometry, propose simple method that addresses the "modality gap" in principled manner and ultimately improves performance on cross-mes modal tasks and outperforms these strategies.
CONCLUSION
In this work, we provide a theoretical explanation of the unique geometry that arises from multi- modal contrastive learning. Building upon this, present straightforward technique, CS, which enhances the interchangeability
embeddings between modalities, enabling the creation cross- modal applications using only uni-modal data. We demonstrate the effectiveness of our approach image, audio, video captioning and rext-to-image generation, achieving state-of-the oof-the-art art performance zero-shot evaluation settings when trained solely uni-modal data.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank all the reviewers for their constructive feedback. Serena Yeung-Levy is a Chan Zuckerberg Biohub San Francisco Investigator.
ETHICS STATEMENT
Drawing from a deep understanding the multi-modal contrastive representation space, our method enables the effortless creation of multi-modal content from uni-modal data. While this has the poten- tial to revolutionize content generation, the risks associated with generation, such as creating harmful content, persists. We strongly emphasize the importance of ethically and responsibly employing any advancements built upon our approach, ensuring that their impact enhances the betterment of our digital ecosystem.
REPRODUCIBILITY STATEMENT
We provide open-source implementation of our work https://github. com/ The implementations will enable researchers to reproduce all the exper- iments described here and run their own analyses on additional multi-modal models and datasets.
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: tional image caption evaluation. In European Conference Computer Vision (ECCV), 2016.
Semantic proposi-
Satanjeev Banerjee and Alon Lavie. Meteor:
An automatic metric for mt evaluation with im- proved correlation with human judgments. In Workshop of Annual Meeting of the Association for Computational Linguistics (ACL Workshop), 2005.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, al. Cogview: Mastering text-to-image generation transformers. Conference on Neural Information Processing Systems (NeurIPS), 2021.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2020.
EleutherAL. Clasp: Contrastive language aminoacid sequence pretraining, 2021. URL https:
//g //github. wtt.//.iii/////caaaeeeeeeeeeaeee
Sabri Eyuboglu, Maya Varma, Khaled Kamal Saab, Jean-Benoit Delbrouck, Christopher Lee- Messer, Jared Dunnmon, James Zou, and Christopher Re. Domino: Discovering systematic errors with cross-modall embeddings. In International Conference on Learning Representations (ICLR), 2022.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations (ICLR), 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter: Gans trained by two time-scale update rule converge to local nash equilibrium. Conference Neural Information Processing Systems (NeurIPS), 2017.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations (ICLR), 2022.
10

---

Published as a conference paper at ICLR 2024
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Ana- lyzing and improving the image quality of StyleGAN. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. DECAP: Decoding CLIP latents for zero-shot captioning. In International Conference on Learning Representations (ICLR), 2023.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Un- derstanding the modality gap in multi-modal contrastive representation learning. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries, In Workshop of Annual Meeting of the Association for Computational Linguistics (ACL Workshop). 2004.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr
Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
Ron Mokady, Amir Hertz, and Amit Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:211 1tt.:21111..999 1.09734, 2021.
David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise-injected CLIP. In Findings of Conference on Empirical Methods in Natural Language Processing (EMNLP Findings), 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguisties (ACL), 2002.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning (ICML), 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Conference on Neural Information Processing Systems (NeurIPS), 2016.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hyperymed, image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguisties (ACL), 2018.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can CLIP benefit and-l--add- tasks? In International Conference Learning Representations (ICLR), 2022.
Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. Language models can see: plugging visual controls in text generation. arxiv preprint arXiv:2205.0265. 2022.
11

---

Published as a conference paper at ICLR 2024
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Derek Tam, Colin Raffel, and Mohit Bansal. Simple weakly-supervised image captioning via CLIP's multimodal embeddings. In Workshop of AAAI Conference on Artificial Intelligence (AAAI Workshop), 2023.
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-text genera- tion for visual-semantic arithmetic. In Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Conference Computer Vision and Pattern Recognition (CVPR), 2015.
Joshua Vendrow, Saachi Jain, Logan Engstrom, and Aleksander Madry. Dataset interfaces: Diagnos- ing model failures using controllable counterfactual generation. arXiv preprint arXiv:2302.07865 2023.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongscok Namkoong, et al. Robust fine-tuning of zero-shot models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zetlleoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, JaeSung Park, Ximing Lu, Prithviraj Am- manabrolu, Rowan Zellers, Ronan Le Bras, Gunhee Kim, et al. Multimodal knowledge alignment with reinforcement learning. arXiv preprint 1.Xi::2205.12630 2022.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao. Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432. 2021.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Con- trastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare (MIHC), 2022.
Yuhui Zhang, Jeff Z HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Diagnosing and rectifying vision models using language. In International Conference on Learning Representations (ICLR), 2023.
Yufan Zhou, Chunyuan Li, Changyou Chen, Jianfeng Gao, and Jinhui Xu. Lafite2: Few-shot text- to-image generation. arXiv preprint arXiv:2210.14 14124, 2022a.
Yufan Zhou, Bingchen Liu, Yizhe Zhu, Xiao Yang. Changyou Chen, and Jinhui Xu. Shifted diffusion for text-to-image generation. arXiv preprint arXiv:2211.15338 2022b.
Yufan Zhou, Ruiyi Zhang. Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation. In Conference on Computer Vision and Pattern Recognition (CVPR), June 2022c.
12

---

Published as a conference paper at ICLR 2024
OVERVIEW OF APPENDIX
In this appendix, we supplement related works and additional details of theory and experiments.
In Appendix we provide detailed related works to contextualize work in existing works of multi-modal contrastive learning and learning cross-modal tasks with uni-modal data.
In Appendix we provide proofs of the two lemmas used in the main paper that reveal important properties of multi-modal contrastive learning.
In Appendix C
we employ statistical methods to validate the proposed geometric structure on large pre-trainee trained contrastive models.
In Appendix D
we summarize the C method into algorithm.
In Appendix we provide additional experimental details and qualitative results of image
captioning.
In Appendix
we provide additional experimental details and qualitative results of text-
to-image generation.
In Appendix G
we explain the importance of aligning embeddings from different modali-
ties.
In Appendix H
we offer further discussions about the effectiveness of collapse vs corrupt.
In Appendix we offer further insights into dimensional collapse and connect it to the cone effect identified by Liang et al. (2022).
A RELATED WORKS
Multi-modal contrastive learning and resulting geometry. Multi-modal contrastive learning aims to create a shared representation for different modalities by attracting similar while repelling dissimilar concepts from different modalities during the optimization process (Radford et al. 2021
Zhang et al., 2022; Xu et al. 2021 EleutherAl 2021] Yuan et al. 2021 Girdhar et al.) 2023)
Recent works such CLIP (Radford al., 2021) have leveraged large-scale image-text data dur- ing pre-training. resulting in models that can build strong uni-modal and cross-modal applications. Connecting different modalities can be advantageous given the complementarity of different modal- ities. For example, connecting vision and language enables zero-shot categorization of visual ob- jects (Radford et al. 2021: Yuan et al. 2021). explanation of model prediction errors or internal representations (Eyuboglu et al. 2022; Hernandez et al. 2022), diagnosis and rectification of vision models using language by composing different concepts (Zhang et al. 2023 Vendrow et al. 2023 and learning cross-modal tasks with uni-modal data.
However, the geometry resulting from multi-modal contrastive learning has received limited study. Recent work (Liang et al. 2022) found that there is clear distinction between embeddings from dif- ferent modalities in the shared representation space, which is referred to as the modality gap. Liang al. 2022) correctly attributed the gap the joint effect of model initialization and optimization. However, they did not study the geometric property of the gap and their theory cannot explain the geometry as well. Their theory can only show there is distributional difference between image and text embeddings. Subsequently, Zhang et al. (2023) studied the geometric properties of the modality gap and found that the gap can be empirically well approximated by constant vector orthogonal to the image or text embedding subspace. However, this work did not provide an explanation for how this unique geometry arises.
In our work, we unify the observations from Liang et al. 2022 and Zhang et al. (2023) and con- tribute a formal formulation and theoretical explanation of the unique geometry resulting from multi- modal contrastive learning. Specifically, we show two important factors of the geometry of the multi-modal representation space: modality gap and alignment noise, where the modality gap is a constant vector orthogonal to the image and text embedding span, and the alignment noise can be approximated by a Gaussian distribution. The modality gap arises due to the interplay between the dimensional collapse in model initialization and the resulting collapsed gradient during optimiza- tion, whereas the alignment noise is due to the stable region of contrastive loss. These explana- tions provide deeper understanding of multi-modal contrastive learning and resulting geometry.
13

---

Published as a conference paper at ICLR 2024
Moreover, this non-trivial geometry has important implications for building applications top of multi-modal representation space.
Learning cross-modal tasks with uni-modal data. Multi-modal data are less abundant and more expensive to collect than uni-modal data, making it ideal to learn cross-modal tasks with uni-modal data. The recent rise of multi-modal contrastive learning provides this possibility, as similar con- cepts from different modalities establish close connections during the optimization process. Many recent works have leveraged CLIP, which aligns images and text to achieve image-to-tex captioning with text-only data and text-to-image generation with image-only data. These methods assume and explore how to interchangeably use image embeddings and text embeddings resulting from multi- modal contrastive learning. They achieve better performance than prior methods that did not lever- age multi-modal representation space, outperforming ZeroCap (Tewel et al., al.] [2022), MAGIC (Su al. 2022). and ESPER (Yu et al. 2022) on captioning, and DALL-E (Ramesh et al. (021) and Cog View (Ding et al. 2021) on image generation. However, due to the peculiar geometry that arises from multi-modal contrastive learning, paired image embeddings and text embeddings are not col- lapsed to the same point, making it non-trivial to substitute one for the other. Therefore, researchers are proposing different methods to tackle this problem.
For image-to-text captioning, WS-ClipCap (Li al., 2023) was the first work to leverage CLIP's text embeddings to decode text during training, finding that training to decode a paraphrased version of the text from the corresponding CLIP's text embedding leads to significantly better performance That is, for datasets where a single image is paired with multiple captions, is better to decode caption by feeding the text embedding obtained from one of the other captions corresponding to the same image. Despite proposing the method, WS-Clippap failed to explain why it works and that on high level, this paraphrased decoding can be viewed adding noise. Decap (Li et al. 2023) maintains a memory of image embeddings and converts image embeddings to text embeddings by using a weighted average of the most similar image embeddings. They then feed the decoder with the converted embedding to decode the same text. While their method outperforms the baseline CLIPRR,, which directly retrieves the most similar captions based on image embeddings, it underperforms WS- WS-ClipCap- Mulli, despite proposing more complex method to tackle the modality gap. CapDec (Nukrai et al. 2022) found that directly adding Gaussian noise to the embedding and then feeding it into the decoder leads to comparably better performance. This method corresponds exactly to the corrupt stage in our method. CapDec attributes its success to the hypothesis that adding Gaussian noise closes the modality gap, however, this intuition is inaccurate.
(WS-CH) C.lipCap-MMuiit
For text-to-image generation, LAFITE (Zhou et al. 2022c) was the first work to use CLIP with uni-modal images only. Their approach be add Gaussian noise to the image embedding and decode the same image. They also incorrectly suggest that adding Gaussian noise closes the modality gap. In contrast, DALLE-2 (Ramesh al.) 2022) uses a complex and heavily trained prior network to convert CLIP text embeddings to CLIP image embeddings but did not provide clear evidence of why the prior network necessary and its effectiveness. Corgi (Zhou et al. (2022b) modifies the prior network by restricting its starting point, resulting in improved performance.
the inverse version of CapDec, where they
Despite the plethora of complex tricks proposed, there is no consensus on the best approach to inter- changeably use image and text embeddings, and none of them reveal fundamental reasons or perform systematic ablations. Our work unifies previous approaches by first analyzing the multi-modal rep- resentation space and how arises. Based on the geometric relation, we provide a straightforward method to tackle embedding interchangeability and demonstrate that our method achieves state-of- the-art results on cross modal tasks when only training on mni-modal data.
B THEORY PROOF
In this section, we provide proofs of the two lemmas used in the main paper that reveal important properties of multi-modal contrastive learning.
14

---

Published as a conference paper at ICLR 2024
B.1 PROOF OF LEMMA
Lemma 3. (Gradients in Contrastive Optimization)
With the mild assumption of equal presence of n. images and texts with p(zi) p(ys)
1 1/n, optimizing the multi-modal contrastive loss C = =- 2n Ei=1 (log Eز exples; (x, +
"uple,,ee,111
exp(ex; (r,/T))
log Ex explex, ) yields the following gradients:
Ve, L - 1
Og (ey, - Cy.)
es
j=l
Ven C = x
Ous (ez, exi)
where 1/(2nT), az, p(zz)1b)) + p(y1(zz)) ays p(ys(z,) + p(zalus). p(z.lys) expley /T) and T temperature. /7) p(yilx))) = Ei= mxpleyv (az/7])
exples; (v,/
Eiz: expletse ( (/)/77)
Proof of Lemma 7
p(=x)()) = using Bayes' theorem: p(ax)))) Σ p(uila))((H)) p(u)
We first prove wk, I
"
Σ
(1/n)
Then, we can prove the lemma using chain rule:
V e.,
exp(ez; /T) + log
exp(ex, eg./T)
llog
Gz
2n.
j== exp(ez, e,,/T) En= expleaş cy./T)
i=1
n
V
Veey
20x; ey./T log(
explex; (uy/T)) Σ log
explexx ey//T))]
2n
1=1
n
[2eyz/
Vern log TT
explesx eey
log
exples, E /T)]
2n
Esp
j=1
j=1
exp(er, eT//T)
exp(er, ep./T)
12m
ey./T
E j=1 exp(ez, By/TT eye/T)
2n
j= explez eyy
1
[2eyn
Σ p(ui lxxeey Σ
p(arklu)eew
2nt
1
(( eve) +
Σ plaxlv))(e emd) -01--1
(1 Σ p(ax (u:)
2nT
(p(un(on) + p(anl)))) ey,)
2nt
im1
ag (ey. eya)
Similarly, get Veg, L= -= (er ), which finishes the proof.
B.2 PROOF OF LEMMA
Lemma 4. (Stable Region Controlled by Temperature)
exp(ex, /7,
consider a single term the multi-modal contrastive loss L, - log y exple.; C /7y/7)
15

---

Published as a conference paper at ICLR 2024
define the margin r Cu;; -yy mmx Gz; 'C; the measure of the similarity difference between the matched pair and the hardest negative pair. When r exceeds threshold given below, L; falls below small pre-set value ò, where we assume optimization ends:
o(T)
2 T exp(o) -1
where o(T) is a monotonically increasing function of temperature T that satisfies < o(T) < n. Therefore, the required margin is monotonically increasing with
Proof of Lemma 2
We first prove E.
exp(t;/T) S o(T) exp(max, tz/T), where o(T) is monoton-
ically increasing function of that satisfies < <(r) < n. Let denote m arg max; 4 (no tie),
we have:
Σ e = exp(tm + Σ
exp((t; - tmm//T)
Let us denote d(r) + E m exp((t: fm )TT, since vi + m,t tm 0, we have 0 < exp((4: lm)/T) < 1, so 0 Enpm exp((l: - (mm n 1, therefore 1 d(()) < n.
Moreover, exp((t; fm T/) monotonically increases with т, therefore o'(T) is a monotonically increasing function of 7. We can denote o(T) = [o'(r)].
Based on this, we have:
C
exp(ea; eg./T)
= log
Ej=1 exp(Cr; Cy,/T)
1
=-log
1 Eii exp((ex, Ey, - Cx, ey./T)
llog11 + Σ exp((ex, Cy ez eg)/TT)
< log(1 + o(T) max exp((xx Ey; - -e, ey,)/T))
log(1 + ((p())xxp((/1))
Suppose L. < log(1 (xrr)x/p(r/m) S 8, have 2 log o(r) proof.
exp(8)-1' which finishes the
C EMPIRICAL VERIFICATION OF GEOMETRY
In Section established theoretical framework for the multi-modal contrastive representation space geometry. We prove that after contrastive learning, we have e, Ey = CT + where e, and ey are fz-normalized embeddings of paired image æ and text y, C1 is constant vector represent- ing the modality gap and is orthogonal to the image and text embedding span, and e ~ A/(0, o² o² is random Gaussian vector representing the alignment noise. Here we employ statistical meth- ods to validate the proposed geometric structure large pre-trained contrastive models. Figure visualizes all the definitions introduced in this section.
To analyze the modality gap CL, we need to first find way to isolate this vector, because it is entangled with alignment noise é in e, eg. Since e is Gaussian, averaging multiple instances of e Cy should neutralize this noise, and thus isolating c1. Therefore, we randomly group every 100 image-text pairs into group i. We define embedding difference for pair j in group i exj eg. This difference includes both the modality gap and alignment noise. By computing the expected value, i [E [d, we effectively eliminate the noise component and leave the modality gap.
as d
To demonstrate that the modality gap a constant vector, we analyze its magnitude and direction across different groups. The distribution of || (Figure 8 (Gap Length)) shows that the gap has
16

---

Published as a conference paper at ICLR 2024
near constant length of 0.83, while the 0.99 mean of cos (os , (Figure 8 confirms that the gap has the same direction. These findings collectively validate that the modality gap is constant vector.
(Gap Direction))
Next, we establish the orthogonality of the modality gap to the embedding spans. By demonstrat- ing its orthogonality the image embedding span, we implicitly confirm its orthogonality to the text embedding span, given that the modality gap is a constant vector. We define the embedding
difference of image j and l j.k ex; -
Exs -eey We observe the distribution of cos cos(d (d) j,k) shown in Figure 8 (Gap Orthogonality) has zero mean with a small standard deviation 0.06, which demonstrates the gap's 's orthogonality to embedding spans.
(i) (1)
Finally, we address the alignment noise. We define c,") = d d which eliminates the gap
and only leaves the noise. We first demonstrate its zero-mean nature Esyle" distribution of each dimension's 's mean. We see the mean of each dimension bounded between -e-- and 1e-8 (Figure (Noise Direction) that the distribution of cos(E) cx has zero mean and small standard deviation 0010, indicating that noise has random directions as every two noises are likely to be orthogonal. These findings affirm that the alignment noise can be approximated by a Gaussian noise (0, r²
(t) 0 through the of the noise vectors is
(Noise Mean)). Furthermore, we show in Figure (Noise
In summary, our empirical analyses support the proposed geometric model of the multi-modaa con- trastive representation space, with a constant modality gap vector orthogonal to embedding spans and an alignment noise characterizable as zero-mean Gaussian.
In Figure D we show that our analyses above fully apply to other modalities, datasets, and contrastive embedding spaces, such as image-captio (MS-COCO), audio-caption (Clotho), and video-caption (MSR-VTT) on ImageBind embeddings.
Group
Group
Group
Image Modalty
4
Text Modality y/
embedding of image; in group
e = F
modality gap group
embadding of text group
embeddiing difference between image] J and imaga in group
embedding difference between image j and text group
alignment between image and text, in group!
Figure 7: Visualization of the multi molti- contrastive representation space and various definitions
introduced in Appendix C
CUP Embeddings on MS-COCO Decasat
Niurc: SIz 0,21
Mear: D.MM, U00
Mearc -d.pp stal 8:06
Moam U.U, Sidt
Mearc U.00, u
53
200
Daroit
Dendly
50
Gap Langth
Gap Deociion
Gup OKT oogooraallkyy
Musi Moan
3--8
Soose Dvocton
Figure 8: Empirical verification of the multi-modal contrastive representation space geometry. The modality gap approximates a constant vector, indicated by the gap length and direction distributions. The modality gap is orthogonal to the span of embeddings from two modalities, indicated by the gap orthogonality distributions. The alignment noise can be approximated by zero-mean Gaussian, indicated by the noise mean and direction distributions.
17

---

Published as a conference paper at ICLR 2024
Imagetlnd Embeddingg MS-COCO Dataset
lea
340
310
100
Dearky
60
7%
20
0:975
-0.2
Gap
Direston
Orthoggnallly
Notre Mean
No
ImageBind Embeddingg on Clotho Dataset
Meen CI Srd 0.01
Mean 0.97 SM 0.00
70
Nean :0.00. Std: 0.03
Neew: 0.00, Skd: 0.00
Neen: 0.00, 9td 0:11
Derohy
30
1.1
22
1.3
10
5.00
2:92
666
6.96
6:97
-02
3.0
87
-0.5
0.86
025
635
035
Goo Length
Orthogonality
Imagetlind Embeddingg MSR-VIT Uataset
led
200
375
350
375
Denttyy
2.03
0.70
2.900
3.975
0.2
Gap
Orthogonallty
Note Mean
8
No Direction
Figure 9: Generalization of Figure 8 other modalities and multi-modal contrastive embedding spaces, including image-caption (MS-COCO), audio-caption (Clotho), and video-caption VSS- on ImageBind embeddings.
18

---

Published as a conference paper at ICLR 2024
D C ALGORITHM
In this section, we summarize the C algorithm as follows:
Algorithm C algorithm
Require: umpaired uni modal dataset X, y. fixed encoders fx X + 124 fy : Rd obtained
from multi-mon mollal contrastive learning, trainable decoder IR + noise level o
1: Cxx ExEx P fx(x)
2: By EveN (7))
3:
4: function TRAIN(U, fv: 9)
5: Cy fy(U)
6: 6 <+N(0,
7:
- g(e, Ey + E)
8: return L(ū. y)
9:
end function
10:
11: function TEST(r, fx. 9)
e₂ + fx(xx)
13:
ÿ g Ex)
return !
15: end function
E IMAGE CAPTIONING
In this section, provide additional experimental details and qualitative results of image caption- ing.
E. EXPERIMENTAL SETUP
Model. We employ the ClipCap model architecture (Mokady et al. [021), which utilizes the CLIP ViT-B/32 (Radford et al. (2221) as the image encoder /x, and a mapping network with a pre-trained GPT-2 (Radford et al. 2019) as the decoder g. The mapping network designed handle the difference in embedding dimensions between CLIP (512-d) and GPT-2 (768-d) and to generate "prefix" as input to GPT-2. It is implemented as lightweight MLP with single hidden layer, which transforms the CLIP embedding into prefix embeddings for GPT-2 to generate captions. During training. we fix the CLIP encoder to maintain the connection between image and text embeddings and only update the decoder, which includes the mapping network and GPT-2.
Data. To train and evaluate our model, we use the MS-COCO image-caption dataset (Lin et al, 2014). We adopt the widely-used data split (Karpathy & Fei-Feil 2015). which consists of training set of approximately 113K images, and validation and test sets of 5K images each, where each image has ground truth captions.
Evaluation. We evaluate our model using various commonly-used image captioning metrics, in-
cluding BLEU (Papineni et al. 2002), METEOR (Banerjee & Lavie 2005). ROUGE (Lin) 2004),
CIDEr /Vedantam et al. 2015), and SPICE (Anderson et al. (016)) These metrics measure the lexical and semantic similanties between the generated captions and the ground-truth captions.
Setup. We train our model for text reconstruction using the MS-COCO captions only. Following the C algorithm, we extract the text embedding from the CLIP text encoder fy and apply the col- lapse operation (removing the pre-computed mean) and corrupt operation (adding Gaussian noise). After pre-training, we evaluate our model in the cross- modsl setting. We replace the encoder with the CLIP image encoder and decode captions from image embeddings. Since image is seen dur- ing pre-training we refer to this evaluation setting as image- free zero-shot evaluation. Additionally, fine-tune our model different amounts of image-caption pairs and evaluate its performance.
19

---

Published as a conference paper at ICLR 2024
We refer to this evaluation setting semi-supervised evaluation. During both the pre-training and fine-tuning stages, we train the model for 10 epochs with batch size of 40, a learning rate of 2e-5, and AdamW (Loshchilov & Hutter 2019) optimizer with a linear warmup of 5K steps. We use early stopping on the validation set and report the test set performance.
E.2 QUALITATIVE EXAMPLES
We provide qualitative results for image captioning in Appendix Figure [0] which helps better understand the improvements of each component of C We observe that:
C generates captions that are highly repetitive and/or nonsensical.
Cf and C2 generates captions that are more fluent, but contain some hallucinations.
C generates captions that correct and concise with extraneous details.
E.3 QUANTITATIVE RESULTS
Method
Counect
Collapse
Corrupted
BLEU-1
BLEU-4
METEOR
ROUGE-L
CIDEr
1% Fine-tuning
ClipCap
64.5
21.2
47.6
71.1
66.5
22.9
22.6
48.7
78.5
15.9
X
66.8
23.0
22.6
48.7
78.5
15.9
71.7
28.8
25.3
52.7
96.3
18.3
(Ours)
5% Fine Luming
ClipCap
70.1
26.4
24.0
51.3
89.4
70.6
27.0
24.5
51.6
91.6
17.8
x
70.7
27.1
24.6
51.7
92.1
17.9
25.6
53.1
98.2
(Ours)
25% Fino-tuning
ClipCap
73.0
30.0
26.0
53.7
101.7
19.1
1
73.2
30.4
26.1
53.9
102.8
19.4
73.1
30.2
26.1
53.8
102.4
19.3
103.1
(Ours)
73.4
26.3
54.3
103.6
19.4
ID0S: Fine-tuning
ClipCap
74.0
31.5
26.8
54.7
106.6
19.9
74.6
32.4
27.2
55.2
109.7
20.3
74.6
32.3
27.1
55.2
109.2
20.1
108.8
(Ours)
74.0
32.2
27.1
55.2
108.9
20.2
Table 5: Image-to- captioning results in the low data regime. When paired multi-modaa data are limited, our approach that leverages uni-modal data for pre-training leads to substantial im- provements compared the purely supervised method (ClipCap). This is the table version used to reproduce Figure 5 in the main paper. Results are averaged over three random seeds for 1-25% fine-tuning reduce the effect of randomness.
We include Appendix Table 5, which is the table version to produce Figure 5 in the main text. In this table, we compare the fully supervised ClipCap (Mokady et al. 202T) an ablation of components in our method 3 and demonstrate the effectiveness of learning cross-modal tasks with uni-modal data. Results are averaged over three random seeds for 1-25% fine-tuning reduce the effect of randomness.
F TEXT-TO-IMAGE GENERATION
In this section, we provide additional experimental details and qualitative results of extt-t--image generation.
20

---

Published as a conference paper at ICLR 2024
c Two sport shiers in look Into iallo
C group of people in cance the water.
C group of people
people riding
Ch
driving by van photo photographor's grandmother's van.
of very strange horee with red wig is showing van fleid zombiss.
c horse standing field with man.
c+ horse is stonding in field next to a house.
Ground truth: woman is fooding honse at ranch.
Ch road sign that reads '"Bus Stop, Los Angelas, Callforna, USA!"
C:: road sign closed while walking and warking.
END ROAD WORK
C. street sign
sign on
sign saying
ch A bus stops the entrance Europeen Union bus.
C: train parked alongside passenger's luggage.
c: man in sult and bee is stending next to train.
C²: man walting af a train station while train passes him.
Ground truth: mon standing in front of passing train
Ch mitle
catching bail during
basebell the Bellimore Orioles.
Ca kid that hitting ball baseball game that the field.
Ca bay getting rady swing baseboll.
C² boy is getting ready e baseball while catcher umpire watch.
playor playing with umpire
C Dog sniffing the back of the bleck and while dog the cobblestone.
C Dog bahind a man's car the street.
Ch street sign is shown with red arrow.
C² dag looking at black cat on the street.
Ground truth: black cat scared by forge
Ch A red and white bus is route the Interational Space Station.
C+ PERSON is DRIVING THE SEA WAVE
C% group of people
large
background.
group of peopie
the
people
Ct.: person luggage, resting courtyard University Southem observing
Edffel Tower
C3: A person on a bench the midst of the street to take walk with their Abrarr.
C man aitling bench with book.
Ch man and women sitting a park bench.
Ground truth: man and woman sitting on a park bench under a huge tree
Figure 10: Qualitative examples of image-to-tex captioning on the MS-COCO dataset, C gener- ates captions that are highly repetitive and/or nonsensical. CF and C generates captions that more fluent, but contain some hallucinations. 3 generates captions that are more correct and con- cise with no extraneous details.
21

---

Published as a conference paper at ICLR 2024
F.1 EXPERIMENTAL SETUP
Model. We use the LAFITE (Zhou et al. 2022c) model for image generation, which uses the CLIP ViT-B/32 (Radford et al. 2021) the text encoder fx, and adapted version of the uncon- ditional generator from StyleGAN2 (Karras et al. 2020) as the decoder g. LAFITE's generator is adversarially trained alongside a discriminator with an additional contrastive objective to align the generator's s representation space to that of CLIP's. As with image captioning, during training. we fix the CLIP encoder to maintain the connection between image and text embeddings, and only update the decoder, which in this case, involves updating both the generator and discriminator.
Data. Same as image-to-text captioning, we train and evaluate the model on the MS-COCO dataset (Lin al. 2014). We use the same pre-processed data and data split provided in the LAFITE official code repository, comprised of 82K training images and 40K validation images with 5 cap- tions per image.
Evaluation. We evaluate our model using the widely-used image generation metric Frechet In- ception Distance (FID) (Heusel al. (017). This metric measure the realism of generated images by computing feature similarity between those of generated images and those of ground-truth im- ages, where the features are derived from the pre-trained Inception-v3 model (Szegedy al. 2016). We also report Inception Score (IS) (Salimans et al., 2016), which is similar to FID. Similarly to LAFITE, FID/IS scores are computed based on 50K generated images, using captions that are ran- domly sampled from the validation set.
Setup. We train the model for at most 750,000 steps with a batch size of 16, learning rate of 2.5e-3, and Adam optimizer. We initialize model with the official mre-traieed weights from LAFITE (Zhou et al. (022c) that was trained on CC3M (Sharma et al., 2018). We report the validation set performance with the lowest FID score.
F.2 QUALITATIVE EXAMPLES
We provide qualitative results for image generation in Figure 11 which helps us better understand the improvements of each component of C3, We observe that:
C² generates the basic scene that pertains the text description, but are less photo-realistic in terms of contrast and color.
Cf and C2 add more fine-grained detail despite still generating some artifacts.
C generates sharper images that are more detailed with fewer artifacts.
G WHY ALIGN EMBEDDING FROM DIFFERENT MODALITIES?
embeddings from different modalities are aligned, we can train model on one modality and then infer on another modality, enabling us to build cross-modal applications with only uni-modal data. This an emerging field that lacks principled approaches to easily applied without requiring more empirical tuning.
We added an experiment to explain this further. We train a text generator (image captioner) over CLIP's text embeddings z. During inference, manually shift all the to to simulate the modality gap (a constant vector orthogonal to original spans). We report the captioning performance in terms of gap distance ||e|| in Table 6 We observe substantial performance drops when the gap grows, showing the need to align embeddings.
H EFFECTIVENESS OF COLLAPSE VS CORRUPT
In Table we observe that adding noise (i.e., corrupt, C is more effective than closing the modality gap (i.e., collapse, C₁). We hypothesize that the greater effectiveness of C2 is because C2 has two effects: 1) injecting noise in the span to mitigate alignment noise; 2) injecting noise in the modality gap direction mitigate the model's sensitivity to this gap.
22

---

Published as a conference paper at ICLR 2024
traweing though candicaap
LAFITR;
Grawd Tub
LAPTER
amall town strect with old bulldnns
Goond Tush
LAFITR:
streef scone with a double dbcker bus on the road
hard shoop grazing in an opon flald.
Goand
Lorge bus transporitg open city
Qualitative examples of text-to-image generation the MS-COCO dataset, C² gen- erates the basic scene that pertains the text description, but are less photo-realistic in terms of contrast and color. C7 and C2 add more fine-grained detail despite still generating some artifacts. C generates sharper images that are more detailed with fewer artifacts.
Figure 11:
23

---

Published as a conference paper at ICLR 2024
Gap Distance ||c|
ROUGE-1
ROUGE-L
METEOR
0.0
85.5
81.2
83.1
0.2
76.3
71.5
73.8
0,4
55.8
50.6
52.5
0,6
40.6
35.9
36,6
0.8
30.5
26.7
26.5
1.0
24.1
21.0
20.3
16.6
14.4
13.8
2.0
13.8
12.1
11.6
Table 6: Image captioning performance when trained embedding and tested This shows the necessity to align embeddings when training a model on one modality and then inferring another modality.
We have added an experiment to verify this hypothesis. When adding noise sampled from Gaus- sian distributions, we remove its component the modality gap direction. Specifically, given (O00 I), we compute its projection on the gap direction cg = c.g 9 |ol l|al| where g is ||
the modality gap direction, then remove this projection get new noise this noise during training and name this experiment C2 (span noise only).
Eg. We add
From Table 7 we see that adding noise only in the span (i.e., C3 (span noise only)) makes its performance much worse than adding noise to all the directions (i.e., C2), and its performance is similar to removing the modality gap (i.e., Cf). Therefore, adding noise (i.e., corrupt) actually leads similar improvement removing the modality gap (i.e., collapse). The reason for the greater effectiveness of corrupt than collapse is that injecting Gaussian noise not only adds noise in the span but also the modality gap direction.
Given the substantial size of the modality gap, adding noise is not enough to fully diminish the gap. Therefore, adding noise and removing the gap (i.e., C3) still enhance the overall performance.
Method
Conn. Coll. Corr.
BLEU-1, BLEU-4 METEOR, ROUGE-L, CIDEr: SPICE-
x
28.1
2.4
12.2
25.4
13.0
6.8
CR
/
X
44.4
6.1
15.5
33.6
25.2
9.2
(span noise only)
41.2
6.2
14.9
33.6
22.8
8.3
X
69.0
25.5
24.3
50.8
87.6
17.6
71.0
27.7
25.0
52.0
93.3
18.3
Table 7: Collapse is actually effective corrupt. The reason for the greater effectiveness of corrupt than collapse is that injecting Gaussian noise not only adds noise in the span but also to the modality gap direction.
I DIMENSIONAL COLLAPSE
Echoing our main paper, the phenomenon of dimensional collapse (Jing et al. 2022) in randomly initialized image and text encoders creates modality gap prior to optimization that also persists after optimization. In this section, we offer further insights into dimensional collapse and demonstrate that 1) it is an inherent characteristic of deep neural networks and 2) deeper networks experience more pronounced dimensional collapse. Additionally, collapse and the cone effect identified by Liang et al. (2022).
establish connection between dimensional
I.1 REAL NETWORKS
In Figure we find that the vision encoder (vision transformer) and text encoder (transformer) of the CLIP have the dimensional collapse phenomenon initialization. Specifically, the effective dimension of the embeddings generated by the vision and text encoder is both much smaller than the full dimension, which induces modality gap. Since the gradients of multi-modal contrastive
24

---

Published as a conference paper at ICLR 2024
learning will only be propagated in the effective dimensions, the modality gap will be preserved after optimization.
Imagee tmoeddingg Oiandom tur or COCO)
Log of Singular VV0lue
Text Embeddingg iRandom CLIP COCO)
of Singutar
-10
-15
Image Embedtinng (fre-ramee CLP CCCD)
-20
Text Embeddinng (Pre traineeded CUP on COCO)
100
300 Singuier Value Index
200
500
300
200
300
400
500
Singuler Value Index
Figure 12: Dimensional collapse of the randomly initialized (left) and pre-trained (right) CLIP representation space. Singular values obtained from SVD reveal that the effective dimension of the image and text representation space is much smaller than the total number of dimensions.
1.2 SIMULATION
To investigate the dimensional collapse phenomenon, we initialize a simple Multi-Layer Perceptron (MLP) with blocks, where each block contains linear layer and Rectified Linear Unit (ReLU) activation. We set the dimensionality of the input space and hidden states to 512, and initialize the weights with Xavier uniform distribution and biases to zero. We initialize N 1000 input embeddings, where each dimension is sampled from standard normal distribution (00, 1). We feed these embeddings into the MLP and extract the features after every layers. We perform Singular Value Decomposition (SVD) on the feature covariance matrix to quantify the degree of dimensional collapse.
As shown in Figure our experiments reveal two key insights:
13
Dimensional collapse an inherent characteristic of deep neural networks and even a sim- ple MIP exhibits this behavior.
2. Deeper networks are more prone to dimensional collapse, resulting in a smaller effective dimension of the feature space.
Layerd
Layer'
Layer10
Layer15
-10
Layer35
Log of Singuler Value
Layerdo
Layer45
-20
-40
1.0
200
300
Figure 13: Simulation of dimensional collapse on a MLP (n (ReLU((inea)))) network. The a- coordinates the black dots indicate the effective dimensions for the embeddings from each of the layers, respectively, which quantifies the extent of collapse.
25

---

Published as a conference paper at ICLR 2024
1.3 CONNECTION TO CONE EFFECT
The dimensional collapse phenomenon can provide an explanation for the cone effect observed by Liang al. (2022). They found that the cosine similarities between any two embeddings out- putted by deep neural network were significantly higher than zero. Due to the dimensional col- lapse, all embeddings share similar values in the ineffective dimensions, leading to high cosine sim- ilarities. The stronger the dimensional collapse, the greater the number of ineffective dimensions, and hence, a stronger cone effect. This explanation illustrated in Figure 14 where collapsing the z-axis of a 3D representation space induces a cone, leading to significantly higher cosine similarities between any two embeddings.
Figure 14: Dimensional collapse explains the cone effect of deep neural networks. When the z-axis of a 3D representation space is collapsed, it results in a cone shape, where the cosine similarities between any two embeddings are significantly higher than zero
26


Reviews:
[{"review_id": "kgztcwgP27", "reviewer": "Reviewer_bNKP", "summary": "This paper explores the modality gap phenomenon in multimodal learning. Specifically, the authors claim that the modality gap emerges and is preserved due to a) dimensional collapse during initialization and training, and b) alignment noise controlled by temperature. To overcome the modality gap, this paper proposes the C^3 paradigm by subtracting mean of features and add Gaussian noise before decoding the features. Experiments on four tasks involving three modalities show the efficacy of C^3.", "questions": "1.\tFrom the experiment results, $C_2^2$ seems to be much more effective than $C_1^2$. Why?\n2.\tSection 3.2 mentions the effect of temperature, but no discussion is given in experiments. What are the effects of modifying temperature in stage 1 and std of Gaussian in stage 3?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well written and easy to follow. The empirical analysis well presents and supports the claims on modality gap, a significant problem in multimodal learning.\n2.\tExperiments are extensive. The authors experiment on image, text and audio modalities, and the results prove the method is applicable across various tasks.", "weaknesses": "1.\tMy major concern is about novelty. [1] has pointed out that random initialization and contrastive learning causes and preserves modality gap, and [2] has modeled modality gap as a constant orthogonal to image and text span. The proposed C^3 is also an ensemble of existing methods [2][3][4]. Especially, the cross-modal transferability in [2] seems quite similar to the ``interchangeability of embeddings from different modalities’’ in this paper. Please justify.\n2.\tThis paper proposes to align representations from different modalities but without convincing justification. In fact, it remains uncertain what effects are relevant to aligning modalities. [1] reports that making the modality gap too small or too large harms performance. [5] proves that strictly aligning modality representations is suboptimal. Therefore, I suggest adding reasons for aligning modalities.\n3.\tDespite that the authors have conducted experiments on various tasks, the comparison with existing methods is limited. Most comparisons in this paper are ablating over different components in C^3. Tab.2 shows marginal improvement over CapDec without reporting std over independent runs, which is not convincing. Tab.3, Tab.4 and Tab.5 report few comparisons with SOTA methods.\n\n[1] Liang, Victor Weixin, et al. \"Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.\" \n[2] Zhang, Yuhui, et al. \"Diagnosing and rectifying vision models using language.\" \n[3] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" \n[4] Zhou, Yufan, et al. \"Towards language-free training for text-to-image generation.\" \n[5] Jiang, Qian, et al. \"Understanding and constructing latent modality structures in multi-modal representation learning.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AlGdVejbNQ", "reviewer": "Reviewer_Vv1o", "summary": "The paper investigates the geometry of embedding spaces obtained through multi-modal contrastive learning (e.g. CLIP), collecting interesting insights and using these to motivate a simple-yet-effective 3-step approach to improve the performance of cross-modal tasks learned using uni-modal data.  In particular, the study suggests, both empirically and theoretically, that the difference between embeddings from different modalities originates from two components: i) a modality gap caused by ineffective dimensions being initialized differently in the two modalities and remaining constant during optimization, and ii) alignment noise that can be approximated as gaussian. The paper then suggests reducing this gap by centering the embeddings at both training time and inference time and adding noise during training. Experiments finally show that the suggested modifications result in state-of-the-art results across a wide set of cross-modal tasks.", "questions": "- Since the modality gap is due to the dimensional collapse, would reducing the dimensionality to the effective one help overcoming the issue?\n- Is there any relation between the decomposition of the modality gap with the content-style-modality specific decomposition assumed e.g. in [1]? Briefly, each latent vector in a multi-modal contrastive learning space is there assumed to have a part that is shared across modalities, i.e. the content, one that is shared but with some distortion, i.e. the style, and one that is not shared at all, i.e. the modality-specific component. Is it possible that the modality specific component in [1] is just the constant component caused by the different initializations seen in this work?\n- The solution to the modality gap is to center the embeddings, implying the modality gap is just a shift. Isn’t it possible that the difference in modality may also result in different scales?\n\n[1] Daunhawer, I., Bizeul, A., Palumbo, E., Marx, A., & Vogt, J. E. (2022, September). Identifiability Results for Multimodal Contrastive Learning. In The Eleventh International Conference on Learning Representations.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Originality\n\n- The paper studies the poorly-understood geometry of latent spaces obtained through multi-modal contrastive learning, building on top of existing works and integrating new insights into an overall recipe to improve cross-modal tasks using such spaces.\n- While two out of 3 steps in the CCC method are not novel, the motivation behind the overall framework is a valid contribution, as well as the need of collapsing the ineffective dimensions by subtracting the mean to the embeddings.\n- The finding of the modality gap being orthogonal to the text and image spaces is interesting and well motivated.\n\n### Clarity\n\n- The paper is well written and pleasing to read.\n- The concepts are explained both rigorously and more colloquially.\n- The experiments are well motivated, and their results properly discussed.\n\n### Significance\n\n- The theoretical framework looks solid: the difference of initialization and the lack of gradients for the ineffective dimensions convincingly explains the modality gap.\n- The empirical analyses make intuitive sense.\n- The framework results in improvements over a wide set of tasks (image/audio/video captioning and text-to-image generation), proving its general applicability.\n- The codebase looks carefully developed and seems free from glaring bugs.\n\nOverall, the paper tackles an extremely interesting question that many practitioners share: “how does, and possibly how should, a multi-modal space look like?” and attempt to characterize its geometry with simple yet convincing tools. Both the theoretical and empirical analyses make intuitive sense, and the empirical results on the considered tasks confirm the utility of its findings.", "weaknesses": "- The discussion on the alignment noise could be improved: in particular, the results in Table 1 are left for the reader to infer. The same statistics could also be easily computed upon any other modality combination in the appendix, it would be useful to see if it still applies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MSiRxwKJXd", "reviewer": "Reviewer_CpsN", "summary": "The paper provides a theoretical understanding of the geometry of the multi-modal contrastive representation space, which is related to the modality gap and alignment noise. Based on this, it presents a new 3-stage framework, called C3 (Connect, Collapse, Corrupt), for solving cross-modal tasks using single-modal data. C3 can effectively bridge the modality gap and enhance the interchangeability of embeddings in the shared representation space. The paper demonstrates the empirical effectiveness of C3 by showing that it achieves state-of-the-art results in various cross-modal tasks.", "questions": "1. How is the \"Collapse\" step implemented (i.e., computing e_x' and e_y')? Is it the same as batchnorm?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Theoretical Insight: The paper's theoretical understanding of the geometry of the multi-modal contrastive representation space is a significant contribution. It also helps shed light on the challenges related to the modality gap (by showing that it is attributed to the joint effect of initialization and optimization), which is the key issue in multi-modal/cross-modal learning.\n\n2. C3 Algorithm: The rationale behind the proposed C3 method is sound and well-explained. Even though each individual step has been explored by previous work, the combination of them leads to very competitive performance on a variety of tasks compared with recent strong baselines (as shown in Table 2-3). The paper also provides comprehensive ablation studies and qualitative examples to understand the effect of each component.\n\n3. Presentation: The presentation is clear, and the ideas are easy to follow. The visuals also help illustrate the effectiveness of the proposed method. The current submission does not include code, hopefully the authors can release them later to facilitate future research.", "weaknesses": "The proposed C3 algorithm has limited novelty on its own given that each step has been studied in previous work. However, the combination of these steps is new and well-motivated by the theoretical framework developed in this paper, which mitigates the lack-of-novelty issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AbwZbxUEc0", "reviewer": "Reviewer_XDw2", "summary": "The paper studies the space's geometry of the features learned by contrastive learning and finds there is a modality gap in such feature space. To bridge the modality gap, based on the analysis of the geometry, they propose a three-step method called C^3 (Connect, Collapse, Corrupt). They conduct experiments on zero-shot image/audio/video captioning and text-to-image generation.", "questions": "1. In the analysis of the space geometry of contrastive features, the paper proposes that the modality gap is a constant vector and orthogonal to each modality. But it can not be derived easily that why the modality gap is the constant vector from the dimension collapse as mentioned in section 3.1. Are there any formal propositions for the constant modality gap vector in the initialization stage?\n\n2. The paper aims to mitigate the modality gap between feature space of different modalities, but in the generation task or encoder/decoder-based architecture, is that the closeness of the features from different modalities indicate a better generation performance? I think it would be better to refer to previous works or conduct this kind of experiment to show the assumption of this paper is true in real applications.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The discussion about the gradient direction in contrastive optimization in Lemma 1 sheds light on how the features learned by contrastive learning are processed.\n2. The theoretical analysis and experiment results of the alignment noise are reasonable and effective.", "weaknesses": "1. The abstract and introduction emphasize the importance of the modality gap of the feature space, \nhowever, the results in the experiments seem show that the key component is the alignment noise. \nFor instance, in Table 2, C_2^2 (only use connect and corrupt) can achieve a similar performance as the C^3.\nCompared to C^1, only adding collapse boost 16.3 but 41 when only adding alignment noise.\n\n2. In section 3.3, it provides some statistic values to show the space geometry. \nI think such statistics may not actually reveal the space geometry since it averages all possible values.\nIs that better to provide a histogram of such statistics to demonstrate the space geometry?\nFurthermore, based on the constant modality gap analysis, is that the E_{i,j}[cos(d_i,d_j)] should have a value close to 1 since they should be parallel except for the noise effect.\n\n3. The experiments are conducted on generation tasks, the quantitative performance is similar for the method and baseline,\nwhile in the qualitative examples, I can not tell which method is better based on three examples. It will be better to provide more qualitative examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)