Title: Geodesic Multi-Modal Mixup for Robust Fine-Tuning
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-modal learning

robustness

fine-tuning

contrastive learning

CLIP

Mixup
Abstract: Pre-trained multi-modal models, such as CLIP, provide transferable embeddings and show promising results in diverse applications. However, the analysis of learned multi-modal embeddings is relatively unexplored, and the embedding transferability can be improved. In this work, we observe that CLIP holds separated embedding subspaces for two different modalities, and then we investigate it through the lens of \textit{uniformity-alignment} to measure the quality of learned representation. Both theoretically and empirically, we show that CLIP retains poor uniformity and alignment even after fine-tuning. Such a lack of alignment and uniformity might restrict the transferability and robustness of embeddings. To this end, we devise a new fine-tuning method for robust representation equipping better alignment and uniformity. First, we propose a \textit{Geodesic Multi-Modal Mixup} that mixes the embeddings of image and text to generate hard negative samples on the hypersphere. Then, we fine-tune the model on hard negatives as well as original negatives and positives with contrastive loss. Based on the theoretical analysis about hardness guarantee and limiting behavior, we justify the use of our method. Extensive experiments on retrieval, calibration, few- or zero-shot classification (under distribution shift), embedding arithmetic, and image captioning further show that our method provides transferable representations, enabling robust model adaptation on diverse tasks.

Geodesic Multi-Modal Mixup for Robust Fine-Tuning

Abstract

Pre-trained multi-modal models, such as CLIP, provide transferable embeddings and show promising results in diverse applications. However, the analysis of learned multi-modal embeddings is relatively unexplored, and the embedding transferability can be improved. In this work, we observe that CLIP holds separated embedding subspaces for two different modalities, and then we investigate it through the lens of uniformity-alignment to measure the quality of learned representation. Both theoretically and empirically, we show that CLIP retains poor uniformity and alignment even after fine-tuning. Such a lack of alignment and uniformity might restrict the transferability and robustness of embeddings. To this end, we devise a new finetuning method for robust representation equipping better alignment and uniformity. First, we propose a Geodesic Multi-Modal Mixup that mixes the embeddings of image and text to generate hard negative samples on the hypersphere. Then, we fine-tune the model on hard negatives as well as original negatives and positives with contrastive loss. Based on the theoretical analysis about hardness guarantee and limiting behavior, we justify the use of our method. Extensive experiments on retrieval, calibration, few- or zero-shot classification (under distribution shift), embedding arithmetic, and image captioning further show that our method provides transferable representations, enabling robust model adaptation on diverse tasks. Code: https://github.com/changdaeoh/multimodal-mixup

1 Introduction

Witnessing the remarkable success of large-scale pre-training approaches, there have been numerous attempts to build a single general-purpose model (so-called foundation model [1]) rather than casting multiple task-specific models from scratch. Once built, it can be adapted (e.g., fine-tuned) to a wide variety of downstream tasks by leveraging its transferable representation. To construct such a general-purpose model, besides architectural design [2, 3, 4] and datasets [5, 6, 7], learning methods [8, 9, 10, 11, 12] have been known to be crucial for inducing transferable representation and for adapting the model robustly.

Contrastive Learning (CL) [13, 14, 15] is one of the most popular learning methods that constructs a discriminative embedding space by minimizing distances between positive pairs while maximizing them between negative pairs. Based on its versatility, CL has been widely adopted to (pre-) train models on various domains [15, 16, 17, 18]. Beyond CL on a single modality, CLIP [19] popularized

Pre-trainedCLIPembeddingspace

ContrastiveFine-tuningwithGeodesicMulti-ModalMixup

Uniformlydistributedalignedembeddings

time spent togethef

Two friends enjoy

A dog is sitting

on the grass.

on α large nig

Three men on therood.

A caris driving

Woman having

afancy dinner

A little girl climbing

on red roping.

入0

:text

: image

on a large rig

Three men

Two friends enjoy

time spent

together.

A caris driving

on the rood.

:text+image

Woman having

α fancy dinner

—:positive

(1-A)θ

- - : hard-negative

:geodesicmixup

A little girl climbing

=geodesicmixup（，)

on red roping.

A dog is sitting

on the grass.

multi-modal CL , which aims to produce close embeddings for paired image-text instances and dissimilar ones for non-paired instances. Due to its generalization capability and transferability, pretrained CLIP and its embeddings have been employed on various downstream tasks [20, 21, 22, 23].

However, we observed an unexpected phenomenon: while CLIP's learning objective is designed to align the image and text embeddings explicitly, it has two separate subspaces for each modality, and there are large unexplored interspaces between them as in Fig. 1 (illustration) and Fig. 2 (DOSNES [24] visualization). We further analyzed this through the lens of uniformity-alignment [25, 26], which is well-studied under uni-modal CL settings but unexplored on multi-modal CLs, and found that CLIP has a poor uniformity-alignment (Fig. 2 middle) due to its bipartite embedding structure. Theoretically and empirically, we confirmed that this property is unchanged even after fine-tuning. As discussed by Wang et al. [25], low uniformity-alignment may limit the embedding transferability. Liang et al. [27], concurrently (in terms of ArXiv preprints) took a similar observation, modality gap , which incurs a bunch of following works, and they also found that increasing temperature ( τ ) in contrastive loss could somewhat reduce the embedding separation. However, varying the temperature requires manual engineering for each downstream task, and it incurs an inevitable trade-off between uniformity-alignment [26]. It raises an important question motivating this work: "How can we obtain a multi-modal representation dealing better with the uniformity-alignment for robust transfer?"

To answer this, we propose a fundamental learning method, geodesic multi-modal Mixup ( m 2 -Mix). As shown in Fig. 1, m 2 -Mix blends the embeddings of different domains, e.g., image and text, and utilizes the mixtures as new entrees of contrastive loss. Our m 2 -Mix contributes to robust representation learning in three perspectives. First, it generates hard negative samples which have high similarity with positives (incentivizing alignment), and they are known to be crucial for robust CL [28, 29, 26]. We provide a theoretical guarantee of the hardness of m 2 -Mixed samples with empirical results. Second, m 2 -Mix interpolates samples from heterogeneous domains to expand the effective embedding space (increasing uniformity) untapped by pre-trained CLIP, and this is further supported by the limiting behavior of m 2 -Mix (see Prop. 4.2). Third, m 2 -Mix produces virtual augmented samples in embedding space, so it endows a better perception on the out-of-distribution samples as well as in-distribution samples than standard CL.

Contributions: (1) We found that CLIP has a bipartite embedding space with poor uniformityalignment, which may limit the embedding transferability for downstream tasks. (2) To address this, we propose a new fine-tuning method based on geodesic multi-modal Mixup, m 2 -Mix. To our knowledge, this is the first work that adopts direct mixtures between heterogeneous embeddings. (3) We devised m 2 -Mix as a geodesic Mixup to ensure the mixed embeddings are on a hypersphere, and this is beneficial for stable learning with L 2 -normalized embeddings. (4) We validate our method by theoretical analyses and extensive experiments on retrieval, calibration, full-/few-shot classification (under distribution shift), embedding arithmetic, and image captioning.

2 Related Works

Contrastive Representation Learning Contrastive Learning (CL) has been widely utilized for representation learning on various domains [15, 17, 30, 19, 31]. The goal of CL is to learn an embedding function that maps data into an embedding space so that semantically similar data

have close embeddings. Most of CLs adopt L 2 -normalized embedding [13, 14, 15, 19] for stable learning [32], and it makes the embeddings lie on a unit hypersphere. There are key properties to analyze the hyperspherical embeddings, so-called Uniformity and Alignment [25]. Better uniformity and alignment can be regarded as a higher embedding transferability, so it is related to performance on downstream tasks. However, uniformity-alignment analysis on multi-modal settings has not been sufficiently explored yet [33], and we found that CLIP has poor uniformity-alignment before and even after fine-tuning. Meanwhile, it is known that hard negatives for constructing contrastive pairs are necessary to increase the robustness of contrastive representation learning [34, 35, 26, 36]. To this end, we devise a new approach for multi-modal CL, which generates hard negatives and achieves better uniformity-alignment for robust and transferable representation.

Mixup There have been numerous papers that claim Mixup [37] is helpful for robust representation learning and alleviates the overconfident problems and failure under distribution shift as well as the in-distribution accuracy [38, 39, 40]. Based on such success of Mixup, many works are adopting it as a component of learning frameworks (on vision [41, 42, 43, 44], language [45, 46, 47] and graph [48, 49]). Besides, CL with Mixup to help representation learning has also been studied. While traditional CL annotates as 1 for positive pairs and 0 for negative pairs, i -Mix [50] and UnMix [51] interpolate the images with ratio λ , and adopt contrastive loss with pseudo labels according to the mixing ratio λ . However, there are few works on Mixup for multi-modal learning [52, 53]. STEMM [52] mixes the speech and text features to align them in shared embedding space, but it cannot be widely used for multi-modal learning because of its architecture-specific design. Therefore, we propose m 2 -Mix, that can be broadly adopted for robust multi-modal representation learning.

Multi-modal Learning To build a universal intelligence system that simultaneously processes multimodal data streams, some focus on developing unified frameworks for multi-modal perception [2, 3, 54, 4, 55], others aim at representation learning under multi-modal correspondence [19, 31, 56, 57]. This work focuses on CLIP [19], the representative multi-modal representation learner. Based on the generalizable embedding space, CLIP has been utilized on numerous tasks [58, 59, 60, 61], and there are many attempts to fine-tune it efficiently [62, 63, 64, 65]. Moreover, robust fine-tuning methods have also been actively studied for generalization on both in- and out-of-distribution data. Wortsman et al. [10] propose a weight-ensemble method that interpolates the pre-trained and fine-tuned weights, Kumar et al. [11] adopt a two-stage learning scheme that combines linear probing and fine-tuning, and Goyal et al. [66] adopt contrastive loss during fine-tuning on image classification. However, previous works have only focused on uni-modal settings and downstream tasks are restricted to the image classification. This paper provides robust fine-tuning methods for multi-modal settings.

3 Observations and Problem Define

For a given batch D = { x i , y i } i M =1 of M instances where ( x i , y i ) denotes the i -th image-text pair, the goal of multi-modal learning is to learn the pair-wise relations between image and text. To do this, CLIP [19] learns image encoder f ( · ; θ 1 ) and text encoder g ( · ; θ 2 ) , so that embedding pairs { I i , T i } i M =1 = { f ( x i ; θ 1 ) , g ( y i ; θ 2 ) } i M =1 get closer to each other. Note that I i and T i are L 2 -normalized unit vectors in most vision-language models, and they lie on the hypersphere. For simplicity, we omit the learning parameters θ 1 and θ 2 from all the following equations. CLIP adopts InfoNCE-style [13] loss C ( · , · ) to enforce the similarity between positive pairs ( x i , y i ) and distance among all remain negative pairs ( x i , y j ) . This is formulated as below (Eq. 1):

Like many other CL approaches, CLIP uses a dot product as a similarity calculation sim ( · , · ) between two vectors and governs τ as a learnable temperature that controls the scale of measured similarity. Now, we analyze the multi-modal embedding in terms of uniformity and alignment, well-known properties in uni-modal CL literature [25, 26] but unexplored in multi-modal settings. Alignment 1 (Eq. 2) evaluates the difference between distances (or similarities) of positive pairs compared with

3.5

ZS

FT

Ours

0.150

fromZS

40

35

FT

+5.1%

+2.6%

10

2.0

0.075

scoret

25

30

+8.9%

+0.3

+2%

0

-5

10

△

20

-30%

15

15

ZS

FT

Ours

0.0

0.000

Averaged on Flickr30k and MS COCO

Uniformity

Alignment

R1(↑)

SIMAT(↑)

B@4(↑)

ID AcC(↑)

OODAcC(↑)

ECE(↓

the hardest negative pairs, while uniformity (Eq. 3) indicates how uniformly the data is distributed. The greater alignment and uniformity denote the more transferable representation [25, 26].

̸

Fig. 2 shows (Left) DOSNES [24] and (Middle) uniformity-alignment of CLIP's embedding space on image-caption datasets after pre-trained (ZS), naively fine-tuned (FT), and fine-tuned with our method (Ours). Pre-trained CLIP embeddings are separated by two subspaces for image and text with wide interspace between them, and even after fine-tuning, this structure remains unchanged. As a result, CLIP and its fine-tuned embedding have poor uniformity and alignment, and this might limit the transferability and robustness of embeddings to downstream data. From this observation, we aim to learn a more transferable and robust multi-modal representation with multi-modal mixup . To validate the transferability and robustness of embeddings, besides evaluating uniformity-alignment, we perform diverse downstream tasks: retrieval, few-/zero-shot classification on in-distribution (ID), out-of-distribution (OOD), classification under modality missing, embedding arithmetic, and image captioning in Section 5. Here, our method produces well-aligned and uniformly distributed embeddings resulting in consistent downstream performance gains (Fig. 2 Right 2 ).

4 Methodology

4.1 Understanding Geodesic Multi-Modal Mixup

From our finding that CLIP embedding is not robust enough, we improve it by fine-tuning CLIP via contrastive loss with virtual hard negative samples. First, we generate hard negatives by mixing the image and text embeddings via multi-modal Mixup , m 2 -Mix. Note that CLIP's L 2 -normalized embeddings lie on a hypersphere, and the mixed ones are also desirable to lie on that hypersphere. However, original Mixup [37, 41] does not guarantee that mixed data lie on the hypersphere. Therefore, we devise a new type of Mixup, geodesic Mixup (defined as Eq. 4). Geodesic Mixup interpolates two data points on geodesic path, so it ensures that mixed samples lie on the hypersphere.

Where λ is handled by a hyperparameter α in Beta distribution. It is well-known that L 2 -normalized embeddings are crucial for metric learning [68, 69] thus adopted by most of the modern contrastive learning methods [14, 15, 19]. Therefore, it is necessary that mixture samples lie on the unit sphere, which is guaranteed by our geodesic Mixup. Comparison between geodesic Mixup and standard Mixup following manual L 2 -normalization in Tab. 6. Then, we utilize the generated hard negatives for contrastive loss by replacing the original negatives (See Eq. 5). Here, we only change the denominator term and retain the numerator that compares the similarity between the positive pairs.

̸

CLIP

V-Mix

VL-Mix

V-Mix

VL-Mix

Txt

enc

入1-入

Txt

enc

入1-入

Txt

enc

+

1-入

入

Txt

enc

+

1-入

入

入

1-入

1-7

1

入

1-入

1

1

Img

1

Img

入

1-A

Img

1

Img

1

1-2

1-

enc

1

enc

1-A

入

enc

1

enc

1

入

1-A

入

1

1

Overwriteoriginalneg.logit

Overwrite originalneg.logit

Overwrite originalpos.logit

1-入

Embedding

Image

Text

Label

matrix

Pos. pair

Neg. pair .......

Geodesic

Embedding

O△

Because CLIP has two-sided polarized embeddings, the similarity between the original image (or text) embedding and mixed embedding is larger than that between the original image and text embeddings. Therefore, m 2 -Mix generates harder negative samples compared with original negatives. The existence of hard negative samples is known to be important in uni-modal CLs [25, 26], and we validate it under multi-modal CL settings in Section 5. Besides, we provide a theoretical result on hardness guarantee and limiting behavior of L m 2 -Mix in the following paragraph.

Theoretical Analysis In Section 3, we observed that naive fine-tuning with standard contrastive loss could not reduce the embedding separation. We speculate this limitation is derived from (1) lack of hard negative samples and (2) vanished learnable τ (0.01) in L CLIP. As shown by Wang et al. [26], when τ approaches zero, the contrastive loss behaves like triplet loss with zero-margin. This means that if the similarity between positive samples is greater than that of the nearest negative, there are no incentives to further pull or push the embeddings of positive and negative samples. Moreover, due to CLIP's bipartite embedding space, it might lack sufficient hard negative samples to incentivize the models to pursue more alignment and uniformity. Therefore, we argue that hard negatives are necessary for multi-modal CL when there is an embedding space modality gap [27].

Theorem 4.1 (Hardness of m 2 -Mixed samples) . Let's assume that two random variables x 1 and x 2 follow the M d ( µ 1 , κ ) and M d ( µ 2 , κ ) , von Mises-Fisher distribution with mean direction µ 1 , µ 2 and concentration parameter κ in R d , respectively. Let ˜ x = x 1 + x 2 and d = 2 . Then, D KL ( p ( x 1 ) || p ( ˜ x )) ≤ D KL ( p ( x 1 ) || p ( x 2 )) for sufficiently large κ .

Theorem 4.1 shows that KL-divergence between the pair of an original sample and a mixed sample is less (more confused with positive) than that of another original sample (proof in Supp. D). Meanwhile, because the converged τ of CLIP is significantly small (i.e., 0.01), it will be reasonable to consider an extreme case: when τ → 0 + . Based on Proposition 4.2, we argue that ones can explicitly enforce uniformity-alignment in multi-modal contrastive learning by equipping m 2 -Mix with L CLIP.

Proposition 4.2 (Limiting behavior of L CLIP with L m 2 -Mix ) . For sufficiently large M , as the temperature of contrastive loss τ → 0 + , the L CLIP and L m 2 -Mix converges to the triplet loss with zero-margin (i.e., corresponding to negative Alignment) and negative Uniformity, respectively. That is: lim τ → 0 + L CLIP + L m 2 -Mix ≃ -( Alignment + Uniformity )

m 2 -Mix brings two advantages on multi-modal CL with theoretical grounds. Firstly, by generating sufficiently hard negatives, it incentivizes the model to enforce alignment more strongly whether there exists a modality gap or not. Besides, the uniformity is also explicitly increased as L m 2 -Mix asymptotically converges to negative uniformity. Thus, L CLIP with m 2 -Mix induces well-aligned and uniformly distributed embeddings, so it makes the model robustly works on diverse tasks.

4.2 Uni-Modal Mixup for CLIP

m 2 -Mix generates hard negatives for CL to align and distribute embeddings uniformly. Meanwhile, Mixup is known to alleviate the overconfidence problem in uni-modal setups [38]. So, we further propose three uni-modal Mixups, Vision-Mix ( V -Mix), Language-Mix ( L -Mix), and Vision-Language Mix ( V L -Mix) to enhance multi-modal CL. Fig. 3 shows the overall structures.

uni -Mix V -Mix and L -Mix interpolate the image and text embedding, respectively. To be specific, V -Mix mixes the embeddings of images in batch and a flipped (reversed) batch with a ratio λ . Then,

m λ ( I i , I i ′ ) has information from i and i ′ = M -i indexed samples with λ and 1 -λ fraction, respectively. Thus, pseudo label for ( m λ ( I i , I i ′ ) , T i ) pair is λ while, that for ( m λ ( I i , I i ′ ) , T i ′ ) is 1λ .

L -Mix has the same formula with V -Mix, except that it is applied to text-side. The combination loss term of V - and L -Mix is defined as L uni -Mix = 1 2 ( C V ( I, T ) + C V ( T, I )) + 1 2 ( C L ( I, T ) + C L ( T, I ))

V L -Mix For pair-wise similarity contrast, V -Mix and L -Mix only mix the image and text embedding, respectively. We additionally propose V L -Mix that mixes the image and text embedding simultaneously. Note that m 2 -Mix mixes embeddings of image and text, while V L -Mix independently mixes them. Both m λ ( I i , I i ′ ) and m λ ( T i , T i ′ ) has i -th component and i ′ -th component with fraction λ and 1 -λ respectively, so the pseudo label for ( m λ ( I i , I i ′ ) , m λ ( T i , T i ′ )) is 1. Here, similarities between negative pairs are retain with that of original negatives likewise uni -Mix.

m 3 -Mix We name the combination of m 2 -Mix with uni-Mix and V L -mix as m 3 -Mix, multiple multi-modal Mixup. Complete objective function is denoted as (weights for each term are omitted):

5 Results

Settings Unless otherwise stated, we adopt CLIP ViT-B/32 as our backbone model. We consider the following methods as our baselines in retrieval and embedding arithmetic tasks: zero-shot inference (ZS) of CLIP, embedding shift (ES) [27], naive fine-tuning (FT), and its increased τ variants, recent Mixup-augmented uni-modal CL methods i -Mix [50] and Un-Mix [51]. We put task-specific setups on each section and Sec. A of Supplementary Material (SM). Further details, hyperparameters selection, pseudo code, and additional results are put in Sec. A, B, and C of SM, respectively.

5.1 Generated Samples by m 2 -Mix

(a) Retrieved images

(b) Initial epoch

(c) Last epoch

Queryimage-caption

Retrievedimagesfromthequeryimageembedding

Text→Image orMixed

Text→Image or Mixed

Original

Original

Mixed (Ours)

Mixed (Ours)

Image→TextorMixed

Image→TextorMixed

Retrievedimagesfromthemixedembeddingbym²-Mix

Ayounggirlpracticingrock

climbing on anindoor wall.

0.2

0.3

0.4

Similarity

0.5

0.6

0.7

0.8

0.2

0.3

0.4

Similarity

0.5

0.6

0.7

8°0

6°0

in the image embedding. (b) and (c) denote cosine similarities between given test instances and its top-1 nearest negative embedding during training epochs. m 2 similar to given instances.

To understand the properties of generated embedding by m 2 -Mix, we explore the mixed embedding from m 2 -Mix. Due to the non-trivial visualization of the embedding itself, we retrieve images that have the most similar embedding with a mixed one. In Fig. 4 (a), the embedding generated by m 2 -Mix has both features from image and text that lack in original image embedding, e.g., the second and third images from m 2 -Mix have rock climbing and indoor wall represented in the text. Besides, similarity histograms in Fig. 4 (b) and (c), show that m 2 -Mix consistently produces harder negatives than the original non-mixed counterparts from the initial to last training epochs.

5.2 Cross-Modal Retrieval with CLIP

First, we validate our method on image-text retrieval, a representative vision-language task, on Flickr30k [67] and MS COCO [70]. All methods are trained over 9 epochs with Adam optimizer (details in SM). Tab. 1 denotes top-1/-5 recall of retrieval. Our m 3 -Mix increases overall performance, while the standard fine-tuning approaches and Mixup-baselines [50, 51] have limited performance gain. Corresponding to previous works [26, 27], we also found that properly increasing temperature ( τ ) in contrastive loss is quite beneficial at improving performance for both FT and m 3 -Mix.

Besides, we observed the improved uniformity and alignment by m 3 -Mix (Fig. 2) not only enhances the Recall of retrievals but also contributes to the calibration [71]. The left side of Fig. 5 denotes the reliability diagrams with calibration errors of the text-to-image retrieval R1 score on Flickr30K. While the naively fine-tuned CLIP has poor

Table 2: Retrieval with disjoint unimodal models. ZS is a naive combination of two models without joint-tuning.

calibration, fine-tuning with m 3 -Mix alleviates the overconfidence issue somewhat and results in a better calibration. This is further confirmed by Tab. 3, in which our m 3 -Mix significantly improves the Expected Calibration Error (ECE) of CLIP. Meanwhile, it is known that the ECE value can be improved by adjusting the temperature τ , i.e., temperature scaling [71]. Therefore, we provide the sensitive analysis on varying τ . In Fig. 5 right side, our method shows relatively robust ECE under varying τ , implying that our multi-modal Mixup-based CL induces the well-calibrated multi-modal model, which is crucial for reliable AI applications.

FT

m3-Mix

FT

m3-Mix

1.0

Desiderata

1.0

Desiderata

10

の

12TR1 ECE

10

9

8

T21 R1 ECE

8

7

7

ECE(%)

6

ECE(%)

6

0.5

0.5

5

5

4

4

3

3

12T R1 ECE

2

2

T21 R1 ECE

0.0

0.0

Confidence

0.5

1.0

0.0

0.0

Confidence

0.5

1.0

1

0.0

0.5

1.0

1.52.02.5

temperature

3.0

3.5

4.0

1

0.0

0.5

1.0

1.5

temperature

2.02.53.0

3.5

4.0

5.3 Cross-Modal Retrieval with Uni-Modal Pre-Trained Models

Sometimes, the high-quality annotations for multi-modal datasets are expensive, and there are cases when plenty of paired multi-modal data is unavailable. Then, it is crucial to exploit the uni-modal pre-trained models for learning multi-modal embedding space [72]. To this end, we validate our m 3 -Mix on the fine-tuning of disjointly pre-trained uni-modal models. Specifically, we jointly fine-tune the pre-trained BERT [73] and ResNet-50 [74] with a contrastive loss on Flickr30k (in Table 2). Among candidates, m 3 -Mix with higher τ consistently achieves the highest performance so that it can be adopted as an effective joint tuning method for independently pre-trained uni-modal models.

5.4 Few-Shot Adaptation and Robustness on Distribution Shift

Next, we evaluate our methods on few-shot image classification under general (in Tab. 4) and distribution shift settings (in Tab. 5 and 6). We consider OxfordPets [75], SVHN [76], and CLEVR [77] for the general setting 3 and ImageNet-1k, ImageNetV2 [78], ImageNet-A [79], ImageNet-R [80], and ImageNet-Sketch [81] for distribution shift setting. Unlike MS COCO and Flickr30K, These datasets provide class name labels only and do not have captions corresponding to each image. To make CL methods amendable for this setting, we adopt a common prompt 'a photo of classname' that wraps the class name with a short context and use this as captions of images. Following [82, 64], we perform the tasks under a few-shot evaluation protocol: 16-shot training samples per class and inference on the entire test set. As baselines, we first consider zero-shot CLIP (ZS) and construct the contrastive loss adoption of vanilla fine-tuning (FT). Then, we showcase our methods with exhaustive ablation ( V -, L -, V L -, and m 2 -Mix) as well as our complete objective m 3 -Mix with its high-temperature variant. To further compare our approach with state-of-the-art (SOTA) fine-tuning methods, we consider MaPLe [64] that optimizes the continuous prompts inside the text and image encoders of CLIP, and the contrastive loss extended version of uni-modal fine-tune methods: LP-FT [11] and WiSE-FT [10].

In both general and distribution shift settings, m 2 -Mix and unimodal Mixups ( V -, L -, V L -) contribute to boost the few-/zeroshot classification performance. After integrating them, m 3 -Mix and its high-temperature variant give significant performance improvement, implying m 3 -Mix is an effective fine-tuning method that covers challenge generalization setup. Moreover, when m 3 -Mix combined with SOTA fine-tuning methods [10, 11, 64],

it consistently brings performance. Therefore, m 3 -Mix is a flexible plug-in method that can be collaborated with many algorithms. Besides, in Tab. 6 (mean Acc. of ImageNet variants), we show that geodesic Mixup achieves superior results than linear Mixup with manual L 2 -normalization. Thus, based on its analytic property, geodesic Mixup is more suitable than linear Mixup under frameworks that learn representation on a hypersphere, as in modern CLs.

5.5 Robustness on Modality Missing

In this section, we study whether m 2 -Mix can help the multi-modal representation learning for video recognition (CMU-MOSEI [83]) under modality missing. Recently, Geometric Multi-modal Contrastive Learning (GMC) [84] achieved competitive results on CMU-MOSEI.

However, GMC only considers the uni-to-joint modal relationship [84], while our m 2 -Mix explicitly generates the mixture of bi-modal semantics so that it can additionally consider the bi-to-joint modal relation. From this, we hypothesize that m 2 -Mix can further improve the robustness and informativeness of the representation. For evaluation, we add the m 2 -Mix on top of GMC.

Different from the CLIP fine-tuning cases, we use the multimodal mixed representation as both positive and negative pairs with the target joint representation because our goal in this task is to align the embedding between the joint and other modalities. As shown in Tab. 7, m 2 -Mix coherently improves the performance of GMC in terms of accuracy, alignment, and uniformity. While GMC strongly aligns the embedding of partial and joint modality, the alignment is further enhanced by the aid of m 2 -Mix (also confirmed in Fig. 6), which results in superior performance when only partial information is given in test-time (modality missing).

Modality

AL

Modalty

Pent

GMC

These results justify the use of m 2 -Mix for robust learning under modality-incomplete scenarios.

5.6 Multi-Modal Embedding Arithmetic

We expect that well-learned multi-modal embedding represents the structural relationship between instances like word vectors [87]. For validation, we evaluate the learned embeddings on SIMAT [88]. SIMAT evaluates text-driven image representation by retrieving a new image with the highest similarity to the latent vector x , which is transformed by text delta vectors when we change the word in an original text, i.e., formulated as: x = I original + λ · ( T new -T original ) . Here, I and T are image and text embedding vectors, and λ is a hyper-parameter about the strength of transformation. Table 8 presents the quantitative scores across methods after fine-tuning on Flickr30k and MS COCO, and evaluated on SIMAT. Learned representation from m 3 -Mix shows stable scores on both a multi-modal model and jointly fine-tuned uni-modal models, which is further confirmed by qualitative experiments (Fig. 7). These support that m 3 -Mix can be adopted as a delicate fine-tuner when embedding geometric structure and arithmetic property should be considered, e.g., controllable generation [89].

man sitting ona beach

horselayingingrass

peopleplayingwithsand

bearlayingingrass

horselayingingrass

bear

peoplelayingin sand

horse

man laying inbeach

laying in

horselaying in field

5.7 Multi-Modal Mixup on a State-of-the-Art Vision-Language Model

In this section, we investigate whether our multi-modal Mixup can also be beneficial for improving other recent large-scale vision-language models beyond CLIP. For this, we adopt Contrastive Captioner (CoCa) [90] ViT-L/14 configuration that pre-trained on LAION-2B [5] from OpenCLIP library as our target backbone model. We consider three learning objectives for CoCa fine-tuning: (1) autoregressive captioning loss (Cap), (2) contrastive loss and captioning loss (CL + Cap), and (3) contrastive loss, L m 2 -Mix, and captioning loss (CL w/ L m 2 -Mix + Cap). For all three methods, we train the model on MS COCO over one epoch with OpenCLIP -provided hyperparameter configuration. After that, we evaluate each model for image captioning (Tab. 9) on MS COCO and zero-shot and fine-tuned cross-modal retrieval (Tab. 10) on Flickr30K and MS COCO.

In Tab. 9, while a combination of vanilla contrastive loss with captioning loss underperforms the captioning-loss-only training, m 2 -Mix-assisted contrastive learning further increases the performance on image captioning in terms of five conventional metrics. Besides, Tab. 10 shows that m 2 -Mix generally improves the retrieval recalls of CoCa on zero-shot and fine-tuned settings. This implies that representation learning with our multi-modal Mixup is beneficial to generative tasks as well as discriminative tasks. The consistent improvement shown in the image captioning task is accorded with observations from other recent works [52, 53] that reveal the effectiveness of cross-modal Mixup on generative tasks by increasing cross-modal alignment.

6 Conclusion

This paper analyzes the representation learned by a multi-modal contrastive learner. We found that CLIP has separated text-versus-image embedding space with poor uniformity-alignment. These polarized embeddings with huge unexploited space may limit the transferability and robustness of representation on downstream tasks. From our findings, we propose Geodesic Multi-Modal Mixup that generates hard negatives for robust contrastive learning by mixing two heterogeneous embeddings. Theoretically, we validate that our method produces hardness-guaranteed samples and has desirable asymptotic behavior to induce better generalizable representation. Empirically, the proposed method effectively improves performances on diverse tasks and perspectives: retrieval, calibration, few-shot classification under distribution shift, embedding arithmetic, and image captioning.

Though the increased uniformity and alignment of multi-modal representation largely empowers the model to adapt robustly to a diverse range of tasks, we found that reckless uplift of them is harmful in some cases of retrieval (modest increment of uniformity-alignment was better than huge increment of them). Thus, more research on the reliable evaluation of multi-modal representation should be pursued in the era of foundation models.

Acknowledgement This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2021R1F1A1060117 and No.2022R1A4A3033874), and also supported by a grant (22183MFDS431) from Ministry of Food and Drug Safety in 2023.

References

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning , pages 4651-4664. PMLR, 2021.

Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver IO: A general architecture for structured inputs & outputs. In International Conference on Learning Representations , 2022.

Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uniperceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1680416815, 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.14108 , 2023.

Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations , 2022.

Jeff Z. HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. Beyond separability: Analyzing the linear transferability of contrastive representations to related subpopulations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. arXiv preprint arXiv:2109.01903 , 2021.

Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054 , 2022.

Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. In The Eleventh International Conference on Learning Representations , 2023.

Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints , pages arXiv-1807, 2018.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722 , 2019.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 5812-5823. Curran Associates, Inc., 2020.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Gowthami Somepalli, Avi Schwarzschild, Micah Goldblum, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. In NeurIPS 2022 First Table Representation Workshop , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Zhecan Wang, Noel Codella, Yen-Chun Chen, Luowei Zhou, Jianwei Yang, Xiyang Dai, Bin Xiao, Haoxuan You, Shih-Fu Chang, and Lu Yuan. Clip-td: Clip targeted distillation for vision-language tasks. arXiv preprint arXiv:2201.05729 , 2022.

Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. arXiv preprint arXiv:2303.15786 , 2023.

Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 15691-15701, October 2023.

Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11175-11185, 2023.

Yao Lu, Jukka Corander, and Zhirong Yang. Doubly stochastic neighbor embedding on spheres. Pattern Recognition Letters , 128:100-106, 2019.

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning , pages 9929-9939. PMLR, 2020.

Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2495-2504, 2021.

Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Joshua David Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations , 2021.

Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems , 33:2179821809, 2020.

Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. Graph matching networks for learning the similarity of graph structured objects. In International conference on machine learning , pages 3835-3845. PMLR, 2019.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021.

Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 4503-4513, 2018.

Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Information Processing Systems , 35:6704-6719, 2022.

Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 21798-21809. Curran Associates, Inc., 2020.

Yifan Zhang, Bryan Hooi, Dapeng Hu, Jian Liang, and Jiashi Feng. Unleashing the power of contrastive self-supervised visual models via contrast-regularized fine-tuning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learning by visualizing feature transformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10306-10315, 2021.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations , 2018.

Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems , 32, 2019.

Zongbo Han, Zhipeng Liang, Fan Yang, Liu Liu, Lanqing Li, Yatao Bian, Peilin Zhao, Bingzhe Wu, Changqing Zhang, and Jianhua Yao. UMIX: Improving importance weighting for subpopulation shift via uncertainty-aware mixup. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Zongbo Han, Zhipeng Liang, Fan Yang, Liu Liu, Lanqing Li, Yatao Bian, Peilin Zhao, Qinghua Hu, Bingzhe Wu, Changqing Zhang, et al. Reweighted mixup for subpopulation shift. arXiv preprint arXiv:2304.04148 , 2023.

Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning , pages 6438-6447. PMLR, 2019.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6023-6032, 2019.

Yoon-Yeong Kim, Kyungwoo Song, JoonHo Jang, and Il-chul Moon. Lada: Look-ahead data acquisition via augmentation for deep active learning. Advances in Neural Information Processing Systems , 34, 2021.

Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning , pages 5275-5285. PMLR, 2020.

Hongyu Guo. Nonlinear mixup: Out-of-manifold data augmentation for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4044-4051, 2020.

Soyoung Yoon, Gyuwan Kim, and Kyumin Park. Ssmix: Saliency-based span mixup for text classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 3225-3234, 2021.

Huiyun Yang, Huadong Chen, Hao Zhou, and Lei Li. Enhancing cross-lingual transfer by manifold mixup. arXiv preprint arXiv:2205.04182 , 2022.

Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, and Bryan Hooi. Mixup for node and graph classification. In Proceedings of the Web Conference 2021 , pages 3663-3674, 2021.

Vikas Verma, Meng Qu, Kenji Kawaguchi, Alex Lamb, Yoshua Bengio, Juho Kannala, and Jian Tang. Graphmix: Improved training of gnns for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. i-mix: A domainagnostic strategy for contrastive representation learning. In ICLR , 2021.

Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing. Un-mix: Rethinking image mixtures for unsupervised visual representation learning. 2022.

Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. Stemm: Self-learning with speech-text manifold mixup for speech translation. arXiv preprint arXiv:2203.10426 , 2022.

Xize Cheng, Tao Jin, Rongjie Huang, Linjun Li, Wang Lin, Zehan Wang, Ye Wang, Huadai Liu, Aoxiong Yin, and Zhou Zhao. Mixspeech: Cross-modality self-learning with audio-visual stream mixup for visual speech translation and recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15735-15745, 2023.

Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555 , 2022.

Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xiaogang Wang, Wenhai Wang, et al. Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2691-2700, 2023.

Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15671-15680, 2022.

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15638-15650, 2022.

Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2104.08860 , 2021.

Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097 , 2021.

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Zero-shot detection via vision and language knowledge distillation. CoRR , abs/2104.13921, 2021.

Gwanghyun Kim and Jong Chul Ye. Diffusionclip: Text-guided image manipulation using diffusion models. CoRR , abs/2110.02711, 2021.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930 , 2021.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. arXiv preprint arXiv:2210.03117 , 2022.

Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. Black box few-shot adaptation for vision-language models. arXiv preprint arXiv:2304.01752 , 2023.

Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. 2023.

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision , pages 2641-2649, 2015.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 815-823, 2015.

Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 212-220, 2017.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning , pages 1321-1330. PMLR, 2017.

Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18123-18133, June 2022.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT , pages 4171-4186, 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition , pages 3498-3505, 2012.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 , 2011.

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2901-2910, 2017.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning , pages 5389-5400. PMLR, 2019.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15262-15271, 2021.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8340-8349, 2021.

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems , pages 10506-10518, 2019.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16816-16825, 2022.

AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2236-2246, Melbourne, Australia, July 2018. Association for Computational Linguistics.

Petra Poklukar, Miguel Vasco, Hang Yin, Francisco S Melo, Ana Paiva, and Danica Kragic. Gmcgeometric multimodal contrastive representation learning. arXiv preprint arXiv:2202.03390 , 2022.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting , volume 2019, page 6558. NIH Public Access, 2019.

Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Towards understanding linear word analogies. arXiv preprint arXiv:1810.04882 , 2018.

Guillaume Couairon, Matthieu Cord, Matthijs Douze, and Holger Schwenk. Embedding arithmetic for text-driven image transformation. arXiv preprint arXiv:2112.03162 , 2021.

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. arXiv preprint arXiv:2301.07093 , 2023.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022.

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16 , pages 121-137. Springer, 2020.

Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems , 31(6):82-88, 2016.

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , ACL '02, page 311-318, USA, 2002. Association for Computational Linguistics.

Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.

Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4566-4575, 2015.

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 , pages 382-398. Springer, 2016.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217-2226, 2019.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops , pages 554-561, 2013.

Ivan Markovi´ c and Ivan Petrovi´ c. Bearing-only tracking with a mixture of von mises distributions. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 707-712. IEEE, 2012.

S Rao Jammalamadaka, Brian Wainwright, and Qianyu Jin. Functional clustering on a circle using von mises mixtures. Journal of Statistical Theory and Practice , 15(2):1-17, 2021.

P Martin, J Olivares, and A Sotomayor. Precise analytic approximation for the modified bessel function i 1 (x). Revista mexicana de física , 63(2):130-133, 2017.

J Olivares, P Martin, and E Valero. A simple approximation for the modified bessel function of zero order i0 (x). In Journal of Physics: Conference Series , volume 1043, page 012003. IOP Publishing, 2018.

Peer reviews:
comments: SUMMARY:
The paper proposes Geodesic Multi-Modal Mixup, which mixes heterogeneous modality embeddings on the hypersphere to generate harder negative samples for the contrastive loss of multi-modal models. Such training scheme improves modality alignment and uniformity. The method was evaluated on various tasks including retrieval, calibration, few-shot classification, and embedding arithmetic.

STRENGTHS:
- The multi-modal mixup approach is proposed opportunely.
- The method is comprehensive and straightforward.
- The idea of mixing different modalities is novel and interesting.
- Extensive experiments are impressive with significant gains in several settings.

WEAKNESSES:
1. I believe the key hyperparameter of the proposed method is lambda, as this should control the "hardness" of the mixed samples.  Please provide an analyses (at least a sensitivity test) on this component.
2. In some cases, the authors only report m^2-Mix, while others report only m^3-Mix. It would be great if the authors could provide full tables with both mixup methods, and discuss reasons if there are some differences in performance between the two.
3. Minor presentation issues:
- Some figures have too small a font size; e.g. Figure 1,6.
- Eq. (5) and L202-203 are confusing at first glance. Please use separate lines or a comma in between.
- Figure 6 right's title should be "GMC w/ M^2-Mix"?

QUESTIONS:
Overall, the paper was very well-written with plenty of applications. The mentioned weaknesses are relatively minor, and no critical issue was found on my side.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes three mix-up inspired regularizers for finetuning CLIP as a way to mitigate the visual-text feature gap in the CLIP feature space. The paper proposes some theory to backup why alignment of text feature space and image feature space might be a good idea. Results in cross-modal retrieval and zero-shot classification justify the method.\n\nSTRENGTHS:\nThe results seem reasonable at first glance, although they are significantly below state-of-the-art.\n\nThe method has some novelty.\n\nWEAKNESSES:\n(1) The modality gap is a well-known phenomenon [Liang22] [CyCLIP]. I think the authors misrepresent their contributions by implying that that they discovered this phenomenon by repeated stating in the introduction that \"we found .. \" that the image and text embeddings occupy separate subspaces in the hypersphere (without citing the prior work) . This phenomenon was studied in  [Liang22] and  [CyCLIP] .\n\n - Additionally, the findings of [Liang22] are somewhat misrepresented. In Table 1 of [Liang22], those authors find that regularizing the modality gap can increase or decrease the zero-shot performance on downstream tasks. In their discussion, Liang et al. state that a larger gap \"may help some fairness and zero-shot learning applications\". This is not discussed in the present work.\n\n(2) Theoretical analysis is weak:\n\n - Theorem 4.1 only applies to $d=2$\n\n- Proposition applies as $\\tau \\rightarrow 0^+$, and in this limit, the sum of CLIP loss and $m^2$-mix regularizer reduces to the negative sum of alignment and uniformity. However, the authors already claimed that vanishing small $\\tau$ is a bad thing, since \"We speculate this limitation is derived from (1) ... (2) vanished learnable $\\tau$ (0.01) in $\\mathcal{L}_{CLIP}$.\" Why then would you prove something about a vanishingly small $\\tau$?\n\n(3) It is unclear to me why $m^2$-mix + CLIP is not sufficient? Why do we need the additional regularization? Does  $m^2$-mix by itself lead to consistent gains in performance?\n\n(4) Concerns about experiments: \n\n - Why do you use ViT-B/32 instead of ViT-B/16? ViT-B/16 is more common [BLIP][Maple] and many papers don't even present ViT-B/32 results. This makes it hard for readers to compare the proposed modality mix-up to current state-of-the-art. The results in Table 1, 2, and 5 are significantly worse than the results in [BLIP] and [Maple].\n\n- Table 4: Why are there only results for Pets, SVHN, and CLEVR? It is hard to determine whether these results would generalize, perhaps you could run results on more datasets, e.g. the 10 datasets used in [Maple]?\n\n- In Table 4, $m^2$-mix leads to about 0.7 % gain over FT, while a much larger 5.2 % gain comes from $m^3$-mix and tuning $\\tau$. Table 5 shows that $m^2$-mix is worse than ZS by 3 %. This severely undermines the claims in this paper, which is that cross-modality mixup (i.e. $m^2$-mix) leads to gains in generalization performance. Where there are gains, most gains seem to come from the intra-modal mix-up regularization. \n\n[Liang22] Liang, Victor Weixin, et al. \"Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 17612-17625.\n\n[CyCLIP]  Goel, Shashank, et al. \"Cyclip: Cyclic contrastive language-image pretraining.\" Advances in Neural Information Processing Systems 35 (2022): 6704-6719.\n\n[BLIP] Li, Junnan, et al. \"Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.\" International Conference on Machine Learning. PMLR, 2022.\n\n[Maple]  Khattak, Muhammad Uzair, et al. \"Maple: Multi-modal prompt learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nMinor:\n\nIn my opinion, Figure 2 is misleading. This figure displays the text and image embeddings on opposite sides of a sphere. In reality, the modality gap is not so large that text and image embeddings corresponding to the same image-caption pair fall on opposite sides of the hypersphere. So I don't think Figure 2 is a good illustration of what's really going on. \n\nin theorem 4.1: Why is $\\tilde{x} = x_1 + x_2$? shouldn't $\\tilde{x}$ by the geodesic mean of the two so that $\\tilde{x}$ still lies on the sphere?\n\nSince this is a paper about mixup, the reader might expect FT+mixup to be a reasonable baseline (not feature mixup, but mixup in image input space).\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a data augmentation method for effectively finetuning multi-modal model (i.e., CLIP). To address poor alignment of image and language space, the authors mix the embeddings of image and text while considering the geometry of the hypersphere. The authors give theoretical analysis as well as extensive experiments on several tasks including retrieval, calibration, few-/zero-shot classification.\n\nSTRENGTHS:\n(1) As a data augmentation strategy for contrastive learning, I think that the proposed method has novelty to some extent, different from previous arts such as i-Mix and Un-Mix. Specifically, the proposed data augmentation is performed in the embedding spaces while considering the geometry of the hyperspheres, in contrast with i-Mix and Un-Mix which mix raw images. \n\n\n(2) The experiments are diverse and extensive and can validate the effectiveness of the proposed method. In particular, the performance on cross-modal retrieval performs significantly better than the competing methods.\n\nWEAKNESSES:\n(1) The first contribution regarding poor alignment of the two modal space of CLIP seems to be overclaimed. \n\nAs far as I know, Liang et al. [24] has clearly disclosed the modality gap in multi-modal contrastive representation learning, clarifying the two modalities in CLIP (and other models) are separately clustered (restricted to narrow cones) and have poor alignment and uniformity. What is difference between the authors finding and [24]? I expect an in-depth discussion in this respect and relax the claim if appropriate. \n\n(2) Mixup-like data augmentation in feature space has been proposed in previous works including [Ref01] and [Ref02]. I suggest authors explicitly discuss the connection and difference from them.  \n\n[Ref01] Terrance DeVries and Graham W Taylor. Dataset augmentation in feature space. arXiv preprint arXiv:1702.05538, 2017a.\n[Ref02] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.\n\nQUESTIONS:\nPlease see “Weaknesses” section.\n\nLIMITATIONS:\nThe authors have adequately discussed the limitations of the proposed method in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for robust fine-tuning called Geodesic Multi-Modal Mixup. The method improves the uniformity and alignment in multi-modal learning, thereby enhancing the performance of downstream tasks. Previous research has shown promising performance of large-scale pre-trained models on various downstream tasks, but the analysis of embedded representations for multi-modal tasks is still insufficient, and the transferability of cross-modal tasks needs to be improved. The motivation of this paper is to address these limitations by introducing the Geodesic Multi-Modal Mixup method, which demonstrates improved performance on various downstream tasks through experiments.\n\nSTRENGTHS:\n1. The paper provides a new perspective for understanding multi-modal embeddings in terms of uniformity and alignment.\n\n2. The paper proposes a new end-to-end fine-tuning method for improving the robustness and transferability of multi-modal representations for downstream tasks.\n\n3. The proposed method, Geodesic Multi-Modal Mixup, is shown to learn a more robust representation and provide improved performance on various downstream tasks such as retrieval, classification, and structure-awareness.\n\nWEAKNESSES:\n1. The paper highlights that the analysis of learned embeddings and transferability for cross-modal tasks has not been explored well, i.e., image captioning, visual questions answering tasks, and so on.\n\n2. The multiple multi-modal mixup seems too complex, would it cast shadow on the learning and generalization ability? More multi-modal backbones like the variants of CLIP or other vision-language models like DeCLIP, FILIP, CLOOB, CyCLIP. Since the approach is agnostic to the architecture and models, I would encourage the authors to perform additional experiments to demonstrate the effectiveness of the proposed approach on other vision-language models.\n\nQUESTIONS:\nsee weakness above\n\nLIMITATIONS:\nsee weakness above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper addresses the problem of improving vision-language representation learning using feature-space augmentation. The authors claim that approaches such as CLIP have led to poor alignment between text features and image features, and the space between them lacks uniformity. They propose an approach called Geodesic multimodal mix-up to mix vision and language embeddings in order to generate hard negative samples. These hard negative samples, along with the original negative and positive samples, are then used in existing contrastive algorithms. The authors provide a theoretical guarantee of the difficulty of these examples. They evaluate their approach on several common benchmarks such as MS-COCO/Flickr30K (retrieval) and Pets/SVHN/CLEVR (classification). Compared to standard mix-up techniques such as linear combination and re-normalization, their approach consistently shows improvements across different settings.

STRENGTHS:
The paper is clearly written and easy to follow. The approach is simple but intuitively makes sense, i.e., mixing up embeddings while still residing on the normalized surface should be better than linear combination and re-normalization. The authors also provided a sound theoretical justification for their approach. The gains on the chosen benchmark over the previous approaches are decent.

WEAKNESSES:
The improvement was shown in vanilla settings, for example, with basic CLIP and small datasets. It is not clear if such a mix-up scheme holds up in other large-scale settings where the data distribution might be different. Also, reported numbers are far from SOTA.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06