Geodesic Multi-Modal Mixup for Robust Fine-Tuning
Authors: Changdae Oh
Junhyuk So
Hoyoon Byun
YongTaek Lim
Minchul Shin
Jong-June Jeon
Kyungwoo Song
Venue: NeurIPS 2023 poster (2023)
Keywords: multi-modal learning
robustness
fine-tuning
contrastive learning
CLIP
Mixup

Abstract: Pre-trained multi-modal models, such as CLIP, provide transferable embeddings and show promising results in diverse applications. However, the analysis of learned multi-modal embeddings is relatively unexplored, and the embedding transferability can be improved. In this work, we observe that CLIP holds separated embedding subspaces for two different modalities, and then we investigate it through the lens of \textit{uniformity-alignment} to measure the quality of learned representation. Both theoretically and empirically, we show that CLIP retains poor uniformity and alignment even after fine-tuning. Such a lack of alignment and uniformity might restrict the transferability and robustness of embeddings. To this end, we devise a new fine-tuning method for robust representation equipping better alignment and uniformity. First, we propose a \textit{Geodesic Multi-Modal Mixup} that mixes the embeddings of image and text to generate hard negative samples on the hypersphere. Then, we fine-tune the model on hard negatives as well as original negatives and positives with contrastive loss. Based on the theoretical analysis about hardness guarantee and limiting behavior, we justify the use of our method. Extensive experiments on retrieval, calibration, few- or zero-shot classification (under distribution shift), embedding arithmetic, and image captioning further show that our method provides transferable representations, enabling robust model adaptation on diverse tasks.

Geodesic Multi-Modal Mixup for Robust Fine- Tuning
Changdae Oh University of Seoul
Junhyuk So* POSTECH
Hoyoon Byun University of Scoul
YongTack Taek Lim University of Scoul
Minchul Shin KAIST
Jong-June Jeon University of Scoul
Kyungwoo Song* Yonsei University
Abstract
Pre-trained multi-modal models, such as CLIP, provide transferable embeddings and show promising results in diverse applications. However, the analysis of learned multi-modal embeddings is relatively unexplored, and the embedding transferabil- ity can be improved. In this work, we observe that CLIP holds separated embedding subspaces for two different modalities, and then we investigate it through the lens of mifformity-aligmmeen to measure the quality of learned representation. Both theoretically and empirically, we show that CLIP retains poor uniformity and align- ment even after fine-tuning. Such a lack of alignment and uniformity might restrict the transferability and robustness of embeddings. To this end, we devise new fine- tuning method for robust representation equipping better alignment and uniformity. First, we propose Geodesic Multi-Modal Mixup that mixes the embeddings of image and text to generate hard negative samples on the hypersphere. Then, we fine-tune the model on hard negatives well original negatives and positives with contrastive loss. Based on the theoretical analysis about hardness guarantee and limiting behavior, we justify the use of our method. Extensive experiments on retrieval, calibration, few- or zero-shot classification (under distribution shift), embedding arithmetic, and image captioning further show that our method provides transferable representations, enabling robust model adaptation on diverse tasks. Code: https: /gi thub. com/ wwwngdaeohnnnnnnneee imwimmmmmmaa..
Introduction
Witnessing the remarkable success of large-scale pre-training approaches, there have been numerous
attempts to build single general- purposs model (so-called foundation model [1]) rather than casting multiple task-specific models from scratch. Once built, it can be adapted (e.g.. fine-tuned) to wide variety of downstream tasks by leveraging its transferable representation. To construct such a general-purpose model, besides architectural design [2, 3, 4] and datasets [5, 6, 7], learning methods [8, 9, 10, 11, 12] have been known to be crucial for inducing transferable representation and for adapting the model robustly.
Contrastive Learning (CL) [13, 14, 15] is one of the most popular learning methods that constructs discriminative embedding space by minimizing distances between positive pairs while maximizing them between negative pairs. Based its versatility, CL has been widely adopted to (pre-) train models on various domains [15, 16, 17, 18]. Beyond CL on a single modality, CLIP [19] popularized
'Equal contribution (changdae.cg oh @ uos.ac.krr kr. uunhyukss@postech.hcco
*Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

embedding space
Contrastive tuning with Geodesic Multi-Moda Misup
Uniformly distributed aligoed embeldings
AB
tost
Systin
tood
-posiive hand regallive gpeckheic mitupp
1)0
geodesic mixup(
Adg AAl
Figure 1: (Left) pre-trained CLIP has two separated clusters for image and text, with a large unexplored interspace. As result, it has poor uniformity and alignment that might limit the embedding transferability. (Middle) For robust representation, our geodesic mudti-modal Mixup (m² Mix) explores the unexploited interspace by mixing the heterogeneous embeddings on hypersphere. Generated samples by m2-Mix are regarded as hard negatives for contrastive loss. (Right) As result, fine-tuning with m²-Mi -MMx induces robust representation with better transferabilityy
multi-modal CL. which aims to produce close embeddings for paired image-text instances and dissimilar ones for non-paired instances. Due to its generalization capability and transferability, pre- trained CLIP and its embeddings have been employed on various downstream tasks [20, 21, 22, 23).
However, we observed an unexpected phenomenon: while CLIP's learning objective is designed to align the image and text embeddings explicitly, it has two separate subspaces for each modality, and there are large unexplored interspaces between them in Fig. 1 (illustration) and Fig. [25,
(DOSNES [24] visualization). We further analyzed this through the lens of aniformity-alignnenn 26]. which is well-studied under uni-modal CL settings but unexplored on multi-moda CLs, and found (Fig. 2 middle) due to its bipartite embedding structure. confirmed that this property is unchanged even after fine-tuning. As discussed by Wang et al. [25], low muiformity-alignned
that CLIP has a poor miformity-alignmee
Theoretically and empirically,
may limit the embedding transferability. Liang al. [27], concurrently (in terms of ArXiv preprints) took a similar observation, modality gap, which incurs a bunch of following works, and they also found that increasing temperature (T) in contrastive loss could somewhat reduce the embedding separation. However, varying the temperature requires manual engineering for each downstream task, and it incurs an inevitable trade-off between uniformity-alignmee [26]. raises an important question motivating this work: "How can we obtain a multi-modal representation dealing better with the uniformity mifformity-alinmment for robust transfer?"
To answer this, we propose fundamental learning method, geodesic multi-modal Mixup (m2 (-xx).... As shown in Fig. 1, 22-Mix blends the embeddings of different domains, e.g., image and text, and utilizes the mixtures as new entrees of contrastive loss. Our m²-Mix contributes to robust representation learning in three perspectives. First, it generates hard negative samples which have high similarity with positives (incentivizing alignment), and they are known be crucial for robust CL [28, 29, 26]. We provide a theoretical guarantee of the hardness of m 2-MMixed samples with empirical results. Second, 2 interpolates samples from heterogeneous domains expand the effective embedding space (increasing uniformity) untapped by pre-trained CLIP, and this is further supported by the limiting behavior of m--Mix (see Prop. 4.2). Third, m2-Mix produces virtual augmented samples in embedding space, so it endows better perception on the -itt----iitiittiion samples as well as in-distribution samples than standard CL.
Contributions: (1) We found that CLIP has a bipartite embedding space with poor uniformity-
alignment, which may limit the embedding transferability for downstream tasks. (2) To address this, we propose a new fine-tuning method based on geodesic multi-modal Mixup, 2-xxix. To our knowledge, this is the first work that adopts direct mixtures between heterogeneous embeddings. (3) We devised m2-Mix as a geodesic Mixup to ensure the mixed embeddings are on a hypersphere, and this is beneficial for stable learning with L2 normaazle embeddings. (4) We validate our method by theoretical analyses and extensive experiments on retrieval, calibration, full-/few-sha classification (under distribution shift), embedding arithmetic, and image captioning.
2 Related Works
Contrastive Representation Learning Contrastive Learning (CL) has been widely utilized for representation learning on various domains [15, 17, 30, 19, 31]. The goal of CL is to learn an embedding function that maps data into embedding space so that semantically similar data
2

---

have close embeddings. Most of CLs adopt L2 normalized embedding [13, 14, 15, 19] for stable learning [32]. and it makes the embeddings lie on unit hypersphere. There are key properties to analyze the hyperspherical embeddings, so-called Uniformity and Alignment [25]. Better uniformity and alignment can be regarded as a higher emhedding transferability, so it is related to performance on downstream tasks. However, uniformity-all allgnnntnnnnnnnnntt. analysis on multi-modal settings has not been before and
sufficiently explored yet [33], and we found that CLIP has poor serfformity-alignmen even after fine-tuning. Meanwhile, it is known that hard negatives for constructing contrastive pairs are necessary to increase the robustness of contrastive representation learning [34, 35, 26, 36]. To this end, we devise new approach for multi-modas CL, which generates hard negatives and achieves better cefffrmitt--lignmmenn for robust and transferable representation.
Mixup There have been numerous papers that claim Mixup [37] is helpful for robust representation learning and alleviates the overconfident problems and failure under distribution shift as well as the in-distribution accuracy [38, 39, 40J. Based such success of Mixup, many works are adopting as a component learning frameworks (on vision [41, 42, 43, 44J, language [45, 46, 47] and graph [48, 49]). Besides, CL with Mixup to help representation learning has also been studied. While traditional CL annotates as 1 for positive pairs and O for negative pairs, é-Mix [50] and Un- Mix [51] interpolate the images with ratio A, and adopt contrastive loss with pseudo labels according to the mixing ratio X. However, there are few works on Mixup for multi-modal learning [52, 53 STEMM [52] mixes the speech and text features to align them in shared embedding space, but it cannot be widely used for multi-modal learning because of its architecture-specffe design. Therefore, we propose m² m--Mix, that can be broadly adopted for robust multi-moda representation learning.
Multi-modal Learning To build a universal intelligence system that simultaneously processes multi- modal data streams, some focus on developing unified frameworks for multi-modal perception [2, 3, 54, 55), others aim at representation learning under multi-modal correspondence [19, 31, 56, 57J. This work focuses on CLIP [19], the representative multi-modal representation learner. Based on the generalizable embedding space, CLIP has been utilized on numerous tasks [58, 59, 60, 61J, and there are many attempts fine-tune it efficiently [62, 63, 64, 65]. Moreover, robust fine-tuning methods have also been actively studied for generalization on both in- and -istt-iiittiittion data. Wortsman al. [10] propose a weight-ensemble method that interpolates the pre-trained and fine-tuned weights, Kumar et al. [11] adopt two-stage learning scheme that combines linear probing and fine-tuning, and Goyal et al. [66] adopt contrastive loss during fine-tuning on image classification. However, previous works have only focused on uni- modal settings and downstream tasks are restricted to the image classification. This paper provides robust fine-tuning methods for multi-modal settings.
3 Observations and Problem Define
For a given batch D (ai, ue J.M of M instances where (1;, denotes the i-th image-text pair, the goal of multi-modal learning is to learn the pair-wise relations between image and text. To do this, CLIP [19] learns image encoder 0i):;;; and text encoder g( 02), so that embedding pairs [T. M = (z, (.0.)..gl)) g(y (2)), get closer to each other. Note that 7, and T, are To- normalized unit vectors in most vision- langual models, and they lie on the hypersphere. For simplicity, we omit the learning parameters 0 and 0g from all the following equations. CLIP adopts InfoNCE-style [13] loss C(.,) .) to enforce the similarity between positive pairs (xt, y1) and distance among all remain negative pairs (z), This is formulated as below (Eq. l):
M
exp(sim L, TT)/T)
C(I,T) = M log
Σ 1=1
CCuP = (CU,T)+((T,,) (I)
Σ/ exp (I, TT)/r)
Like many other CL approaches, CLIP uses a dot product as a similarity calculation sim(;.) between two vectors and governs Now, we analyze the multi-modal embedding in terms of uniformity and alignment, well-known properties in uni-modal CL literature [25, 26] but unexplored in multi-modal settings. Alignment' (Eq. 2) evaluates the difference between distances (or similarities) of positive pairs compared with
learnable temperature that controls the scale of measured similarity.
((3)) which ignores the similarity among negative pairs. We found that it is not directly related to downstream performances, so we modified the alignment Stative-alignment that handles the similarities of both positive and negative pairs.
Original formulation of alignment is > [Ilf(
3

---

0153
Score from 28
FT
+5.
+2.6%
Oup
+0.3
-5%
+8 01
X
30%
zs rr Ous
Jigened
aiF
SIAAT
Apr(1)
Acr(
ocp Acri 1)
B23 +
Figure 2: (Left) DOSNES [24] visualization of CLIP's embedding space Flickr 30k [67]. Greens and oranges denote the image and text embedding, respectively. Embeddings of pre-trained (ZS) and naively fine-tuned (FT) ones have separate clusters with low uniformity and alignment (Middle), which may limit embedding transferabilityy Multi-modal mixup induces more aligned and uniformly distributed embeddings, (Right) which largely improves downstream performance, including retrieval (RI), embedding arithmetic (SIMAT), image captioning (BLUE@4) 4), and classification (ID Acc. and OOD Acc.), and uncertainty calibration (ECE).
the hardest negative pairs, while uniformity (Eq. 3) indicates how uniformly the data is distributed. The greater alignment and uniformity denote the more transferable representation [25, 26].
Alignment = [1(()))) s(u Ill min [lf((;;) s(x))
(2)
=z1
Uniformity := log (gp [exp (-2 [f(a,) - )1.3)
(3)
Fig. shows (Left) DOSNES [24] and (Middle) uniformity-alignment of CLIP's embedding space on image-caption datasets after pre-trained (ZS), naively fine-tuned (FT), and fine-tuned with our method (Ours). Pre-trained CLIP embeddings are separated by two subspaces for image and text with wide interspace between them, and even after fine-tuning, this structure remains unchanged. As a result, CLIP and its fine-tuned embedding have poor uniformity and alignment, and this might limit the transferability and robustness of embeddings to downstream data. From this observation, we aim to learn a more transferable and robust multi-modal representation with multi-modal mixup. To validate the transferability and robustness of embeddings, besides evaluating muiformity-alignmen we perform diverse downstream tasks: retrieval, few-/zero-shot classification on in-distribution (ID), out-of-distribution (OOD), classification under modality missing, embedding arithmetic, and image captioning in Section 5. Here, our method produces well-aligned and uniformly distributed embeddings resulting in consistent downstream performance gains (Fig. 2 Right²).
4 Methodology
4.1
Understanding Geodesic Multi-Modal Mixup
From our finding that CLIP embedding is not robust enough, we improve it by fine-tuning CLIP via contrastive loss with virtual hard negative samples. First, we generate hard negatives by mixing the image and text embeddings via multi-modal Mixup, m--Mix. Note that CLIP's s Ln-nomallized embeddings lic on a hypersphere, and the mixed ones are also desirable to lie on that hypersphere. However, original Mixup [37, 41] does not guarantee that mixed data lie on the hypersphere. There- fore, we devise new type of Mixup, geodesic Mixup (defined as Eq. 4). Geodesic Mixup interpolates two data points on geodesic path, so it ensures that mixed samples lie on the hypersphere.
mx b) sin() =sin((1 X)0)
where 0 = cos (ā b) and À ~ Beta(o, o:)
(4)
sin(A)
sin(B)
Where À is handled by hyperparameter or in Beta distribution. It is well-known that Læ-normalized embeddings are crucial for metric learning [68, 69] thus adopted by most of the modern contrastive learning methods [14, 15, 19]. Therefore, it is necessary that mixture samples lie on the unit sphere, which is guaranteed by our geodesic Mixup. Comparison between geodesic Mixup and standard Mixup following manual Izznormalization in Tab. 6. Then, we utilize the generated hard negatives for contrastive loss by replacing the original negatives (See Eq. 5). Here, we only change the denominator term and retain the numerator that compares the similarity between the positive pairs.
M
exp(I: T2/T) exp(1, Ti/T) + Enhr exp(1s (3)/7
(I,)(IT)) M
log
(5)
L 1 (C.. + C.. (T.1))
'For each method, FT and Ours, we compute scores for each bar by averaging all values of FT and m -Mx in corresponding Tables, except OOD Acc computed by averaging values of WiSE-FT, LP-FT, and MaPLe.

---

CLIP
V-Mix
VL-Mix
m 22-Mix
V-Mix
VL-Mix
m2-Mix
Ts
Tu
SA
Img
THE
Ing
Overattr origlinll neg Angl
rit arigenel nep. lagtt Owmrit orighall pas Angh
Embbolding
Prs patr
galy
CAD
Figure 3: Comparison among contrastive losses. CLIP enforces the pair-wise similarity between matched image-text embedding pairs. m--Mix generates hard negative samples via mixing two heterogeneous embeddings. We additionally propose uni-modal Mixups, V-Mix (and L-Mixx and V L-Mix, that augment the homogenous embeddings in multi-modal contrastive loss.
Because CLIP has two-sided polarized embeddings, the similarity between the original image (or text) embedding and mixed embedding larger than that between the original image and text embeddings. Therefore, - generates harder negative samples compared with original negatives. The existence of hard negative samples is known to be important in uni moi-l CLs [25, 26], and we validate under multi-modal CL settings Section Besides, we provide a theoretical result on hardness guarantee and limiting behavior of 2 in the following paragraph.
Theoretical Analysis In Section 3, we observed that naive fine-tuning with standard contrastive loss could not reduce the embedding separation. We speculate this limitation derived from (1) lack of hard negative samples and (2) vanished learnable r (0.01) in CCUP. As shown by Wang et al. [26], when J approaches zero, the contrastive loss behaves like triplet loss with zero-margin. This means that if the similarity between positive samples is greater than that of the nearest negative, there are no incentives further pull or push the embeddings of positive and negative samples. Moreover, due to CLIP's bipartite embedding space, it might lack sufficient hard negative samples to incentivize models to pursue more alignment and uniformity. Therefore, we argue that hard negatives are necessary for multi-modal CL when there is an embedding space modality gap [27].
Theorem 4.1 (Hardness of m2-Mixed samples). Let's assume that two random variables z1 and 22 follow the Ma(H1, k) and Ma(p2, ni), von Mises-Fisher distribution with mean direction 11, 12 and concentration parameter in Rdd, respectively. Let T 31 + D2 and d Then, DKL (pL()(())lp(E)) S (xz(p(z))) (())))) for sufficiently large k.
Theorem 4.1 shows that KL--divergence between the pair of an original sample and mixed sample less (more confused with positive) than that of another original sample (proof in Supp. D). Meanwhile, because the converged т of CLIP is significantly small (i.e., 0.01), it will be reasonable to consider an extreme case: when + 0+. Based on Proposition 4.2, we argue that ones can explicitly enforce uniformity-alignment in multi-modal contrastive learning by equipping m²2Mix with LCUP.
Proposition 4.2 (Limiting behavior of LCLPP with L Mix). For sufficiently large M, as the tem- perature of contrastive loss + 0", the LeuP and Lm converges to the triplet loss with zero-margin (i.e., corresponding to negative Alignment) and negative Uniformity, respectively. That is: lim Lcur + fm²-Mix (Alignment + Uniformity)
m 2-M brings two advantages on modal CL with theoretical grounds. Firstly, by generating sufficiently hard negatives, it incentivizes the model enforce alignment more strongly whether there exists a modality gap or not. Besides, the uniformity is also explicitly increased as = asymptotically converges negative uniformity. Thus, LCUP with m²-Mix induces well-aligned and uniformly distributed embeddings, so makes the model robustly works on diverse tasks.
4.2 Uni-Modal Mixup for CLIP
m 2-M generates hard negatives for CL to align and distribute embeddings uniformly. Mcanwhile, Mixup is known to alleviate the overconfidence problem in uni-modal setups [38]. So, we further pro- pose three uni-modal Mixups, Vision-Mix (V-Mix), Language-Mix (L-Mix), and Vision-Language Mix (VL-Mix) to enhance multi-modal CL. Fig. 3 shows the overall structures.
wni-Mix
V-Mix and L-Mix interpolate the image and text embedding, respectively. To be specific, V-Mix mixes the embeddings of images in batch and a flipped (reversed) batch with ratio X. Then,
5

---

mx(4;.1)) Ip) has information from i and i' = M - indexed samples with À and 1 - À fraction, respectively. Thus, pseudo label for (mx (I, 1e), ), T:) pair is while, that for (m Ip): Tx) 1-X.
1 Cv(1, = M - -ooog
exp(mx [,) Ti/T) Σ₁ exp(T. Tz/r)
- -1 log exp(ma (T,, I2) Te/T))
Ex=rexp(1 TTyT)
L-Mix has the same formula with V-Mix, except that it is applied text-side. The combination loss term of V- and M.-Mix is defined as Luni-Mix = (CC(1,, CT, T)) + ((CL(T, CL,
V--Mix For pair-wise similarity contrast, V-Mix and L-Mix only mix the image and text em- bedding, respectively. We additionally propose V L-Mix that mixes the image and text embedding simultaneously. Note that m - mixes embeddings of image and text, while V L-MMix indepen- dently mixes them. Both m (I,, Ip) and m Tp) has i-th component and i'-th component with fraction À and À respectively, similarities between negative pairs are retain with that of original negatives likewise uni-Mix.
the pseudo label for (mx (I,, Ip). my (T, Tv) is 1. Here,
exp(ma 1x) (w)T.,TV)TT) LVLM ((rr Cv)(T,I) log ":=1exp(2 T₃/T)
Cvi(I,T) M
Σ
m--Mix We name the combination -Mix with uni-Mix and L-mix as m m--Mix, multiple multi-modal Mixup. Complete objective function is denoted as (weights for each term are omitted):
3 = LCLIP + 2 Mix + Luni Mix + LV
Results
Settings Unless otherwise stated, adopt CLIP ViT-B/32 as our backbone model. We consider the following methods as our baselines in retrieval and embedding arithmetic tasks: zero-shot inference (ZS) of CLIP, embedding shift (ES) [27]. naive fine-tuning (FT), and its increased variants, recent Mixup-augmented uni-modal CL methods i-Mix [50] and Un-Mix [51]. We put task-specific setups on each section and Sec. A of Supplementary Material (SM). Further details, hyperparameters selection, pseudo code, and additional results are put in Sec. A, B, and C of SM, respectively.
5.1 Generated Samples by m2-Mix
Query saptin
Retricued Imgacs query Image embcocdigg
Text Image or Hoad
Orgine
Text maga Moed
Origiall
Fiad
Mood
Trat Hord
yooung girl practicing rock dimbing on an indoar wall.
34
(a) Retrieved images
(b) Initial epoch
(c) Last epoch
Figure 4: On Flickr30k, (a) top-3 retrieved images by image embedding (Top) and m 2-xMieed (Bottom) from source instance (Left), m2-Mix generates an embedding that contains features from both modalities (young girl, climbing, and indoor wall), partly lacking
in the image embedding. (b) and (c) denote cosine similarities between given test instances and its top-1 nearest negative embedding during training epochs. m--Mix makes negatives that are highly similar to given instances.
To understand the properties of generated embedding by m²-Mix, we explore the mixed embedding from m²-Mix Due the non-trivial visualization of the embedding itself, we retrieve images that have the most similar embedding with mixed one. In Fig. 4 (a), the embedding generated by -ix has both features from image and text that lack original image embedding, e.g., the second and third images from m² m--Mix have rock climbing and indoor wall represented in the text. Besides, similarity histograms Fig. (b) and (c), show that 2-x consistently produces harder negatives than the original non-mixed counterparts from the initial to last training epochs.

---

5.2 Cross-Modal Retrieval with CLIP
First, we validate our method on image-text retrieval, representative vision-lang lision-language task, on Flickr30k [67] and MS COCO [70]. All methods are trained over 9 epochs with Adam optimizer (details SM). Tab. denotes top-1/-5 recall retrieval, Our m increases overall performance, while the standard fine-tuning approaches and Mixup-baselines [50, 51J have limited performance gain. Corresponding to previous works [26, 27], we also found that properly increasing temperature (T) in contrastive loss is quite beneficial at improving performance for both FT and m-ix..
Table 1: Image text (i-+) and text image retrieval (t-+) retrieval results (top-1/-5 Recall:R1, R5). ZS and FT denote pre-trained and fine-tuned CLIP, respectively.
Table 2:
Retrieval with disjoint uni- modal models. ZS is a naive combina- tion of two models without joint-tuning.
Flick330k
MS COCD
FlickrOnk
1-92
[-01
1-91
[-01
[->
R5
R1
R5
R1
R5
RI
R5
25 18127
71.1
90.1
68.5
88.9
31.9
56.9
28.5
53.1
Z5
0.1
0.4
0.1
0.2
71.8
90.0
31.9
56.9
28.7
53.0
ES [2]
0.2
81.2
95.4
63.6
36.9
26.7
(T 0.05)
K2.4
95.1
K2.1
95.7
40.2
68.2
41.6
69.9
(T 0.05)
31.5
61.2
29:2
61.4
(T = 0.10)
75.7
93.9
78.0
92.9
34.3
62.7
36.7
64.2
(T 0.10)
30.0
62.7
30.1
60.6
CMixx [50]
72.3
91.7
69.0
91.1
34.0
63.0
34.6
62.2
<-Mis [50]
27.6
60.3
27.1
60.7
Un-Mix [51]
78.5
95.4
74.1
91.8
38.8
66.2
33.4
61.0
Un [51]
31.5
61.3
29.2
61.2
R2.3
05.9
82.7
41.0
68.3
30.9
3. Mix
62.6
30.3
P.MMxx (v 0.05)
82.7
95.7
82.8
95.5
40.4
67.9
42.0
68.8
m Mixx (r 0.06)
32.5
64.7
30.4
63.4
Besides, we observed the improved uniformity and align- ment by mi-Mix (Fig. 2) not only enhances the Recall of retrievals but also contributes the calibration [71]. The left side of Fig. denotes the reliability diagrams with calibration errors of the text-to-image retrieval RI score on Flickr30K. While the naively fine-tuned CLIP has poor calibration, fine-tuning with m -Mix alleviates the overconfidence issue somewhat and results in a better calibration. This is further confirmed by Tab. 3, in which our ma-Mix significantly improves the Expected Calibration Error (ECE) of CLIP. Meanwhile, it is known that the ECE value can be improved by adjusting the temperature T, i.e., temperature scaling [71]. Therefore, we provide the sensitive analysis on varying r. In Fig. right side, our method shows relatively robust ECE under varying т, implying that our multi-moda model, which is crucial for reliable AI applications.
Table 3: Calibration on Flickr30k.
Metric
Task
ZS
FT
ECE
1-m
1.38
1.50
200
2,36
1.58
1.54
based CI. induces the well-calibrated multi-modal
FT
Mix
FT
3--iix
23
Des
15
30
Ule
Chahieence
Figure 5: Reliability diagram (left) and ECE under varying temperature 7 (right) on Flickr30K image-to-text retrieval. Our method shows (1) better reliability diagram (close y z), (2) achieves a lower minimum ECE value, and (3) more stable across varying 7 than naive tneettnnng. Thus, representation learning by 3-MMix robustly induces a well-calibrated multi-modal model.
5.3 Cross-Modal Retrieval with Uni-Modal Pre-Trained Models
Sometimes, the high-quality annotations for multi-modal datasets are expensive, and there are cases when plenty of paired multi-modal data is unavailable. Then, is crucial to exploit the uni-modal pre-trained models for learning multi-mon modal embedding space [72]. To this end, we validate our m on the fine-tuning of disjointly pre-trained uni-modal models. Specifically, we jointly fine-tune the pre-trained BERT [73] and ResNet-50 [74] with a contrastive loss on Flickr30k (in Table 2). Among candidates, m--Mix with higher consistently achieves the highest performance so that it can be adopted as effective joint tuning method for independently pre- pre-trained uni-modal models.
7

---

5.4 Few-Shot Adaptation and Robustness on Distribution Shift
Table 4: Few-shot adaptation under
Table 5: Few-shot (ImageNet; IN) and zero-shot evalu-
general setting.
ation under distribution shift (-V2, -A, -R, -S).
Matied
SVIJN
Dutase
CLEVR
Avg.
Method
IN
Damaset
Arg.
13.63
20.70
62.05
50.67
89.3
45.00
53.49
65.44
55.35
20:07
34.50
46.70
FT w/ V-Mix
$9.45
41.51
53.90
62.66
FT V-Mix
66.00
56.19
20.85
60,50
31,97
$7.70
w/ L-Mix
99:43
48.42
53.91
63.92
w/ L-Mix
65.96
55,95
20.57
60.54
35:25
47.65
L-MMix
45.22
VI.Mis
66.24
FT w/ m Mix
90.05
46.24
$3.60
63.29
IT m Mix
67.04
57.39
20.05
39.28
35.31
47.81
Mix
90.16
54.84
P Mix
20.80
35.86
45.45
m Mix (T = 0.05)
90:49
60.90
53.95
68.45
m. Mix 0.05)
68.40
58,51
22.17
62.28
37,62
49.50
WISE-FT [10]
91 80
35.04
41.93
56.25
WiSE-FT
69.00
59.66
28.01
64.84
41.05
$2.51
WiSE-FT m Mix
WISE-FT m Mix
29,16
LP-FT[UI]] [I]]
89.92
44.91
53.60
62.82
LP-FT [1]]
68.22
38,40
23.57
63.36
38.04
50.72
LP-FT w/ m3 Mix
91.03
64.24
55,30
70.16
LP-FT w/ 2 Mix
68.62
59.17
25.65
65.14
36.78
51.51
MaPLe [64]
90.87
47.62
MaPLe [64]
65.59
32.49
MaPLe w/ m Mix
91.14
52.72
45.20
63.02
MaPLe w/ mr Mix
65.76
58.16
32.52
68.20
42.67
53.46
Next, we evaluate our methods on few-shot image classification under general (in Tab. 4) and distri- bution shift settings (in Tab. and 6). We consider OxfordPets [75], SVHN [76], and CLEVR [77] for the general setting' and ImageNet-lk, ImageNetV2 [78]. ImageNet-A [79], ImageNet-R [80], and ImageNet-Skette [81] for distribution shift setting. Unlike MS COCO and Flickr30K, These datasets provide class name labels only and do not have captions corresponding to each image. To make CL methods amendable for this setting, we adopt common prompt photo of classname that wraps the class name with a short context and use this as captions of images. Following [82, 64], we perform the tasks under few-shot evaluation protocol: 16-shot training samples per class and inference on the entire test set. As baselines, we first consider zero-shot CLIP (ZS) and construct the contrastive loss adoption of vanilla fine-tuning (FT). Then, we showcase our methods with exhaustive ablation (V-, L-,, VL-, and m-ix as well as our complete objective m--MMix with its tegperrrmee variant. To further compare our approach with state-of-the-ar (SOTA) fine-tuning methods, we consider MaPLe [64] that optimizes the continuous prompts inside the text and im- age encoders of CLIP, and the contrastive loss extended version of uni-modal fine-tune methods: LP-FT [11] and WiSE-FT [10].
In both general and distribution shift settings, m--Miix and uni- modal Mixups (V-, L-, VL-) contribute to boost the few-/zero- shot classification performance. After integrating them, m m-x and its high-temperaturg variant give significant performance im- provement, implying -Mix is an effective fine-tuning method that covers challenge generalization setup. Moreover, when 3-Mix combined with SOTA fine-tuning methods [10, 11, 64),
Table 6: Ablation study on Mixup.
m Mix type geodesic
Temperature (r)
linear
0.01
48.36
48.45
0.05
48.48
49.80
0.10
45.20
46.41
it consistently brings performance. Therefore, m 3 -Mixx is flexible plug-in method that can be collaborated with many algorithms. Besides, in Tab. (mean Acc, of ImageNet variants), we show that geodesic Mixup achieves superior results than linear Mixup with manual L2 Lo-oommalizatio. Thus, based on its analytic property, geodesic Mixup is more suitable than linear Mixup under frameworks that learn representation on a hypersphere, in modern CLs.
5.5 Robustness on Modality Missing
In this section, we study whether m--Mix can help the multi-modal representation learning for video recognition (CMU-MOSEI [83]) under modality missing. Recently, Geometric Multi-modal Contrastive Learning (GMC) [84] achieved competitive results on CMU-MOSEL
However, GMC only considers the uni-to-joint modal relationship [84], while our m explicitly generates the mixture of bi-modal semantics so that it can additionally consider the bi-to-joint modal relation. From this, we hypothesize that m 2 can further improve the robustness and informativeness of the representation. For evaluation, we add the 2-MMix on top of GMC.
Results of CLIP VIT-B/16 on other transfer learning benchmark datasets are provided in Sec. C of SM.
8

---

Different from the CLIP fine-tum tinettuning cases, we use the multi- modal mixed representation as both positive and negative pairs with the target joint representation because our goal in this task is to align the embedding between the joint and other modalities. As shown in Tab. 7, m2-Mix coherently improves the performance of GMC in terms of accuracy, alignment, and uniformity. While GMC strongly aligns the embedding of partial and joint modality, the alignment is further enhanced by the aid of m- (also confirmed in Fig. 6), which results in superior performance when only partial information is given in test- assttimme (modality missing). These results justify the use of m² -Mi for robust learning under
GMC
Figure 6: t-SNE [85] on CMU-
MOSEI with partial information. Each color denotes embeddings of partial and joint modality.
scenarios.
Table Accuracy (ace.) (t), alignment (align.((t). and uniformity (unif.)(t) of multi-modal learning methods on CMU-MOSEI under complete and partial modalities. Averaged performance of five runs.
Test time Observed Modalinees
Fall(T+V+a))
B0C.
acc.
align.
acc.
align.
USiT.
BCC.
allgs.
uniE
acc.
allgu.
unif.
acc.
Boc.
allgn.
78.5
5.6 Multi-Modal Embedding Arithmetic
We expect that well-learned multi-modal embedding represents the structural relationship between instances like word vectors [87]. For validation, we evaluate the learned embeddings on SIMAT [88]. SIMAT evaluates text-driven image representation by retrieving new image with the highest similarity to the latent vector x, which is transformed by text delta vectors when we change the word an original text, i.c., formulated as: x Loriginal + X- (Tnew Toriginat). Here. and I' are image and text embedding vectors, and À is hyper-parameter about the strength of transformation. Table 8 presents the quantitative scores across methods after fine-tuning on Flickr30k and MS COCO, and evaluated on SIMAT. Learned representation from m 8-Mix shows stable scores on both multi-modal model and jointly fine-tuned uni-modal models, which further confirmed by qualitative experiments (Fig. 7). These support that m--Mix can be adopted as delicate fine-tuner when embedding geometric structure and arithmetic property should be considered, e.g., controllable generation [89].
Table 8: 3--Mix shows the stable SIMAT Score (1) CLIP and joint fine-tuning of uni- modal pre-trained models.
Joree laying grass Joear laying grace Pirrre leytnng grawe
people playing to sannyl
geople laping
CLIP COCO)
CLIP (Flickr30k)
BERT + RN50
Jean lavine beach
(Plick
2S
34.3
34.4
61
ES
34.6
34.5
6.2
FT
42.3
40.8
15.4
6-Mi [50]
37.3
40.0
12.9
Figure 7: Embeddings of 3-x on SIMAT source-target texts (reds) and images (source and top-1 retrieved). Black texts correspond to top-3 retrieved images from source image-caption.
Un-Mix [S1]
42.9
38.5
1588
38.9
5.7 Multi-Modal Mixup on a State-of-the-Art Vision-Language Model
In this section, we investigate whether our multi-modal Mixup can also be beneficial for improv- ing other recent large-scale vision-language models beyond CLIP. For this, we adopt Contrastive
Captioner (CoCa) [90] ViT-L/14 configuration that pre-trained on LAION-2B [5] from OpenCLIP
library as our target backbone model. We consider three learning objectives for CoCa fine-tuning: (1) autoregressive captioning loss (Cap), (2) contrastive loss and captioning loss (CL + Cap), and (3) contrastive loss, -- and captioning loss (CL w/ Ln=*Mxx+ Cap). For three methods, we train the model on MS COCO over one epoch with DpenCLIP-providee hyperparameter configuration. After that, we evaluate each model for image captioning (Tab. 9) on MS COCO and zero-shot and fine-tuned cross modal retrieval (Tab. 10) Flickr30K and MS COCO.

---

Table Image captioning results on MS COCO with CoCa ViT-L/14 model. We fine-tune the CoCa on MS COCO for epoch with three different learning objectives and evaluate them in terms of five conventional metrics. m² achieves performance gain on all the five metrics.
Method
Metrics
BLEU@4
METEOR
ROUGE-L
CIDEr
SPICE
7.2
12.4
26.3
35.2
9.3
Cap
36.0
29.4
57.3
125.1
23.1
CL. + Cap
35.7
29.3
57.1
124.9
23.0
CL. w/ fmm2MMs. + Cap
36.3
29.5
57.5
125.6
23.2
In Tab. 9, while combination of vanilla contrastive loss with captioning loss underperforms the captioning-los losiinng-oos-olly training. on image captioning in terms of five conventional metrics. Besides, Tab. 10 shows that m--Mix generally improves the retrieval recalls of CoCa on zero-shot and fine-tuned settings. This implies that representation learning with multi-modal Mixup is beneficial generative tasks well as discriminative tasks. The consistent improvement shown in the image captioning task accorded with observations from other recent works [52, 53] that reveal the effectiveness of cross-modal Mixup on generative tasks by increasing cross-modal alignment.
contrastive learning further increases the performance
Table 10: Cross-modal retrieval top- 1/5 recalls on MS COCO (fine-tuned) and Flickr30k (zero-shot transfer). m n--Mix generally enhances image-text retrieval of SOTA vision-language model, CoCa.
Method
(Zero-shat) Flickr30k i-5(R5)
i - (R1)
( (RI)
( i (R5)
Cap CL Cap
90.4
98.5
78.5
94.4
92.4
99.1
79.2
94.9
CL 2 Mix Cap
92.8
99.0
79.5
95.1
Method
(Fine-tuned) MS COCO
i i- (RI)
i- (RS)
(-> (RI)
(-> (R5)
Cap
68.5
87.9
53.2
77.8
Cap
73.9
91.2
56.3
CL w/ Lm Cap
73.9
91.0
56.5
80.6
Conclusion
This paper analyzes the representation learned by a multi-modal contrastive learner. We found that CLIP has separated cext-versus-imag embedding space with poor mniformity-allgnmennt These polarized embeddings with huge unexploited space may limit the transferability and robustness of representation downstream tasks. From our findings, we propose Geodesic Multi-Modal Mixup that generates hard negatives for robust contrastive learning by mixing two heterogeneous embeddings. Theoretically, we validate that our method produces hardness -attiness-ggratteeed samples and has desirable asymptotic behavior to induce better generalizable representation. Empirically, the proposed method effectively improves performances on diverse tasks and perspectives: retrieval, calibration, few-shot classification under distribution shift, embedding arithmetic, and image captioning.
Though the increased uniformity and alignment of multi-mode representation largely empowers the model to adapt robustly to a diverse range of tasks, we found that reckless uplift of them is harmful in some cases of retrieval (modest increment of uniformity-aligmmn of them). Thus, more research on the reliable evaluation of multi-modal representation should be pursued in the era of foundation models.
better than huge increment
Acknowledgement This work was supported by the National Research Foundation of Ko- rea (NRF) grant funded by the Korea government (MSIT) (No.2021RI) 11RIR 17 and No. 222222221444333388 and also supported by grant (22183MFDS431) from Ministry of Food and Drug Safety in 2023.
10

---

References
[1] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx. Michael S Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. 072588 2021.
[2] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Ioao Carreira. Perceiver: General perception with iterative attention. In International conference machine learning. pages 4651-4664, PMLR, 2021.
[3] Andrew Jacgle, Schastian Borgeaud, Jean- nan-Baptiste Alayrac, Carl Doersch, Catalin lonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Henaff, Matthew Botvinick, Andrew Zisserman. Oriol Vinyals, and Joao Carreira. Perceiver IO: A general architecture for structured imputs & oulputs. In International Conference on Leurning Representations, 2022.
[4] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni- perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16804- 16815, 2022.
[5] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[6] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang. De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internctsscale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[7] Samir Yitzhak Gadre, Gabriel Ilhareo, Alex Fang, Jonathan Hayase, Georgios Smymis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: search of the next generation of multimodal datasets. arXiv preprint arXiv:2304. 14108, 2023.
[8] Hong Liu, Jeff 7. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In International Conference Learning Representations, 2022.
[9] Jeff Z. HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. Beyond separability: Analyzing the linear transfcrability of contrastive representations related subpopulations. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[10] Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi. Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. arXiv preprint 10X1v:2109001903, 2021.
[11] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform
arXiv preprint arXiv:2202.10055: 2022.
[12] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for ro- bustness to spurious correlations. In The Eleventh International Conference on Learning Representations. 2023.
[13] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pages arXiv-1807, 2018.
[14] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv: 19X.::1111.007222 1.5722, 2019.
[15] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1697-1607. PMLR, 2020.
[16] Yuning You. Tianlong Chen, Yongduo Sui. Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5812-5823. Curran Associates, Inc., 2020.

---

[17] Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
[18] Gowthami Somepalli, Avi Schwarzschild, Micah Goldblum, C Improved neural networks for tabular data via row attention and contrastive pre-training. In NeurlPS 2022 First Table Representation Workshop, 2022.
Bayan Bruss, and Tom Goldstein. Saint:
[19] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8773. PMLR, 2021.
[20] Zhecan Wang, Noel Codella, Yen-Chun Chen. Luowei Zhou, Jianwei Yang, Xiyang Dai, Bin Xiao, Haoxuan You, Shih-Fu Chang, and Lu Yuan. Clip-td: Clip targeted distillation for vision-language tasks. arXiv preprint arXiv:2201.05721 2022.
[21] Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision- langual models. arXiv preprint arXiv:2303.15788 2023.
[22] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15691-15701, October 2023.
[23] Ziqin Zhou, Yinjie Lei, Bowen Zhang. Linggiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot semantic segmentation. In Proceddings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11 11175-111888 1185, 2023.
[24] Yao Lu, Jukka Corander, and Zhirong Yang. Doubly stochastic neighbor embedding spheres. Pattern Recognition Letters, 128:100- 106, 2019.
[25] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference Machine Learning, pages 9929-9939. PMLR, 2020.
[26] Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2495-2504, 2021.
[27] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understand- ing the modality gap in multi-modal contrastive representation learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[28] Joshua David Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021.
[29] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances Neural Information Processing Systems, 33:21798- 21809, 2020.
[30] Yujia Li, Chenjie Gu Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. Graph matching networks for learning the similarity of graph structured objects. In International conference on machine learning, pages 3835-3845. PMLR, 2019.
[31] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li. and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904 4916. PMLR, 2021.
[32] Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4503-4513, 2018.
[33] Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Information Processing Systems, 35:6704-671 6619, 2022.
[34] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors. Advances in Neural Information Processing Systems, volume 33, pages 21798-2 218899189 Curran Associates, Inc., 2020.
12

---

[35] Yifan Zhang, Bryan Hooi, Dapeng Hu, Jian Liang, and Jiashi Feng. Unleashing the power of contrastive self- self-supervised visual models via contrast-regularized fine-tuning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[36] Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learning by visualizing feature transformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10306-10315, 2021.
[37] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
[38] Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Rhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[39] Zongbo Han, Zhipeng Liang, Fan Yang. Liu Liu, Lanqing Li, Yatao Bian, Peilin Zhao, Bingzhe Wu, Changqing Zhang. and Jianhua Yao. UMIX: Improving importance weighting for subpopulation shift via uncertainty-awarc mixup. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[40] Zongbo Han, Zhipeng Liang, Fan Yang, Liu Liu, Lanqing Li, Yatao Bian, Peilin Zhao, Qinghua Hu, Bingzhe Wu, Changqing Zhang, et al. Reweighted mixup for subpopulation shift. arXiv preprint arXiv:2304.041 2023.
[41] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Joannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438 6447. PMLR, 2019.
[42] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023 6032, 2019.
[43] Yoon- Yeong Kim, Kyungwoo Song. JoonHo Jang, and I1-chul Moon. Lada: Look-ahead data acquisition via augmentation for deep active learning. Advances in Neural Information Processing Systems, 34, 1021.
[44] MhygH Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pages 5275 5285, PMLR, 2020.
[45] Hongyu Guo. Nonlinear mixup: Out-of-manifold data augmentation for text classification. In Proceedings the AAAI Conference on Artificial Intelligence, volume 34, pages
2020.
[46] Soyoung Yoon, Gyuwan Kim, and Kyumin Park. Ssmix: Saliency- based span mixup for text classification. In Findings of the Association for Computational Linguisties: ACI-IJCNIPP 2021, pages 3225-3234,
2021.
[47] Huiyun Yang, Huadong Chen, Hao Zhou, and Lei Li. Enhancing cross-lingual transfer by manifold mixup, arXiv preprini arXiv:2205.04182 2022.
[48] Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, and Bryan Hooi Mixup for node and graph classification. In Proceedings of the Web Conference 2021, pages 3663-3674, 2021.
[49] Vikas Verma, Meng Qu, Kenji Kawaguchi, Alex Lamb, Yoshua Bengio, Juho Kannala, and Jian Tang. Graphmix: Improved training of gnns for semi-supervised learning. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, 2021.
[50] Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-I Ling Li, Jinwoo Shin, and Honglak Lee. i-mix: A domain- agnostic strategy for contrastive representation lcarning. In ICLR, 2021.
[51] Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing. Un-mix: Rethinking image mixtures for unsupervised visual representation learning. 2022.
[52] Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. Stemm: Self-leaming with speech-text manifold mixup for speech translation. arXiv preprint arXiv:2203. 10426, 2022.
[53] Xize Cheng, Tao Jin, Rongjie Huang, Linjun Li. Wang Lin, Zehan Wang, Ye Wang. Huadai Liu, Aoxiong Yin, and Zhou Zhao. Mixspeech: Cross Cross-todality self- with audio-visual stream mixup for visual speech translation and recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15735-15744 2023.
13

---

[54] Alexei Baevski. Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv::200.003555, 2022.
[55] Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang. Yu Qiao, Xiangang Wang, Wenhai Wang, et al. Une-peeerreer v2: generalist model for large-scale vision and vision-language tasks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2691-2700, 2023.
[56] Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre pre-training with triple contrastive learning In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1567 1-671-15680, 2022.
[57] Amaupreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. Proceedings of the IEEE/CVF Conference on Compuder Vision and Paltem Recognition, pages 15638-15650, 2022.
[58] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. ClipAclip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2 104.8800 2021.
[59] Han Fang, Pengfei Xiong. Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106. 1097, 2021.
[60] Xiuye Gu, Tsung-Yr Yi Lin, Weicheng Kuo, and Yin Cui. Zero-shot detection via vision and language knowledge distillation. CoRR, abs/2104. 13921, 2021.
[61] Gwanghyun Kim and Jong Chul Ye. Diffusionclip: Text-guided image manipulation using diffusion models. CoRR, abs/21 10.02711, 2021.
[62] Kaiyang 7hou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13000):3337-2388, 2022.
[63] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930, 2021.
[64] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. arXiv preprint arXiv:22 2022.
[65] Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. Black box few-shot adaptation for vision vision-language language models. arXiv preprint arXiv:2304.01752, 2023.
[66] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zaco Kolter, and Aditi Raghumathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. 2023.
[67] Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2644 16449, 2015.
correspondences for richer tone-tt--eentene
[68] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.
[69] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 212-220, 2017.
[70] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In European conference on computer vision, pages 740 4407555 Springer, 2014.
[71] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
[72] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked- text tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 18133, June 2022.
14

---

[73] Jacob Devlin Ming-Wel Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of NAACL-HLL, pages 4171-4186, 2019.
[74] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[75] Omkar M Parkhi, Andrea Vodaldi, Andrew Zisserman, and C. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1498-3505, 2012.
[76] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
[77] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2901-2910, 2017.
[78] Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[79] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15277 2021.
[80] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distributio generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340 8349, 2021.
[81] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506-10518, 2019.
[82] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Procedingss of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[83] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philipee Morency, Muttimodal language analysis in the wild: CMU-MOSE: datasct and interpretable dynamic fusion graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 2236-2246, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[84] Petra Poklukar, Miguel Vasco, Hang Yin, Francisco S Melo, Ana Paiva, and Danica Kragic. Gme- geometric multimodal contrastive representation learning. arXiv preprint 22X.::2202.0399 2022.
[85] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
[86] Yao- Hung Hubert Tsai, Shaojic Bai, Paul Pu Liang, Zico Kolter, Louis-Philipes Moreney, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequenees. In Proceedings the conference. Association for Computational Linguistics. Meeting, volume 2019, page 6558. NIH Public Access, 2019.
[87] Kawin Ethayarajh. David Duvenaud, and Graeme Hirst. Towards understanding linear word analogies. arXiv preprint arXiv: 1810.04882, 2018.
[88] Guillaume Couairon, Matthieu Cord, Matthijs Douze, and Holger Schwenk. Embedding arithmetic for text driven image transformation. arXiv preprint arXiv:21 2021.
[89] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chumyuan Li, and Yong Jac Lec. Gligen: Open-set grounded text-to-image generation. arXiv preprint arXiv:2301.07093 2023.
[90] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu, Coca: Contrastive captioners image-text foundation models. arXiv preprint XxXiv22055009117 2022.
15

---

[91] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang. Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, al. Oscar: Object-semantics aligned pre-training for vision-language tasks. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020.
[92] Amir Zadeh, Rowan Zellers, Pincus, and Louis-Philippe Morency, Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31(6):82-B8) 088,22888888 2016.
[93] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.. 1078, 2014.
[94] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguisties, ACL '02, page
USA, 2002. Association for Computational Linguistics.
[95] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACI. Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Sunmarizaaton,, pages 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.
[96] Thin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguisties.
[97] Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basse based image description evaluation, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[98] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould, Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCC 2016: 14th Eumpean Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382-398. Springer, 2016.
[99] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth, Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2266, 2019.
[100] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1 2013.
[101] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Uef101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint /212.002 2012.
[102] Jonathan Krause, Michacl Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
[103] Ivan Marković and Ivan Petrovié. Bearing-only tracking with mixture of von mises distributions. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 707-712. IEEE, 2012.
[104] S Rao Jammalamaduka, Brian Wainwright, and Qianyu Jin. Functional clustering circle using von mises mixtures. Journal of Statistical Theory and Practice, 15(2):1-17, 2021.
[105] Martin, Olivares, and Sotomayor. Precise analytic approximation for the modified bessel function i 1 (x). Revista mexicana de fisica, 6302)3130-133, 2017.
[106] J Olivares, P Martin, and E Valero. A simple approximation for the modified hessel function of zero order (x). In Journal of Physics: Conference Series, volume 1043, page 012003. IOP Publishing, 2018
16


Reviews:
[{"review_id": "NZltfNq3bp", "reviewer": "Reviewer_uSAw", "summary": "This paper addresses the problem of improving vision-language representation learning using feature-space augmentation. The authors claim that approaches such as CLIP have led to poor alignment between text features and image features, and the space between them lacks uniformity. They propose an approach called Geodesic multimodal mix-up to mix vision and language embeddings in order to generate hard negative samples. These hard negative samples, along with the original negative and positive samples, are then used in existing contrastive algorithms. The authors provide a theoretical guarantee of the difficulty of these examples. They evaluate their approach on several common benchmarks such as MS-COCO/Flickr30K (retrieval) and Pets/SVHN/CLEVR (classification). Compared to standard mix-up techniques such as linear combination and re-normalization, their approach consistently shows improvements across different settings.\n\n\n\n", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\nThe paper is clearly written and easy to follow. The approach is simple but intuitively makes sense, i.e., mixing up embeddings while still residing on the normalized surface should be better than linear combination and re-normalization. The authors also provided a sound theoretical justification for their approach. The gains on the chosen benchmark over the previous approaches are decent.", "weaknesses": "The improvement was shown in vanilla settings, for example, with basic CLIP and small datasets. It is not clear if such a mix-up scheme holds up in other large-scale settings where the data distribution might be different. Also, reported numbers are far from SOTA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate you for your attention and valuable comment. We agree with your concerns and have conducted additional experiments to address them.\n\n---\n> **Q)** The improvement was shown in vanilla settings, for example, with basic CLIP and small datasets. It is not clear if such a mix-up scheme holds up in other large-scale settings where the data distribution might be different. Also, reported numbers are far from SOTA.\n\n**A)**\n\nIn this work, we focus on devising a robust fine-tuning method for pre-trained VL models. Here, we believe that the method (as a ‘robust’ fine-tuning approach) should have the following desirable properties: 1. Achieving stable performance on the data-scarce regime (Sec 5.4); 2. Robustness under distribution shift (Sec 5.4); 3. Robustness under modality missing (Sec 5.5); 4. calibration (Sec 5.2), and 5) preserving semantic embedding structures  (Sec 5.6). Therefore, we spared much of our attention to these components rather than the scale of experiments or SOTA performance, and we demonstrated that our method achieve all the above properties from each section.\n\nNonetheless, as you pointed out, it is crucial to determine whether our method is effective for large-scale SOTA VL backbones beyond the basic CLIP. To this end, we evaluate our method on the one of SOTA VL models, CoCa [Yu et al. 2022], with CoCa-Large configuration governing 787M parameters which are 9 times larger than the CLIP-ViT-Base setting in our manuscript. We fine-tuned the pre-trained CoCa model on the MS COCO dataset via CoCa’s pre-train objective with $m^2$-Mix-based contrastive loss. In Table 2. and Table 3. of the global response, our $m^2$-Mix consistently outperforms compare to the original CoCa fine-tuning baselines with regard to captioning evaluation metrics and recalls of retrieval. These results verify that our method can give favorable advantages on large-model fine-tuning in terms of both generative and discriminative tasks.\n\nMoreover, we compare our method with SOTA parameter-efficient fine-tuning approaches such as MaPLe [Khattak et al. 2023] both in our manuscript (Sec 5.4) and in this rebuttal (Table 1.). In these experiments, our method constantly improves performance, so it demonstrates its effectiveness and versatility as a plug-and-play module. \n\n---\n\n[Yu et al. 2022] Jiahui Yu, et al. \"CoCa: Contrastive Captioners are Image-Text Foundation Models\". Transactions on Machine Learning Research. (2022).\n\n[Khattak et al. 2023] Khattak, Muhammad Uzair, et al. \"Maple: Multi-modal prompt learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n"}, {"review_id": "9GnbmITwom", "reviewer": "Reviewer_yWq1", "summary": "This paper proposes a new method for robust fine-tuning called Geodesic Multi-Modal Mixup. The method improves the uniformity and alignment in multi-modal learning, thereby enhancing the performance of downstream tasks. Previous research has shown promising performance of large-scale pre-trained models on various downstream tasks, but the analysis of embedded representations for multi-modal tasks is still insufficient, and the transferability of cross-modal tasks needs to be improved. The motivation of this paper is to address these limitations by introducing the Geodesic Multi-Modal Mixup method, which demonstrates improved performance on various downstream tasks through experiments.", "questions": "see weakness above", "limitations": "see weakness above", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a new perspective for understanding multi-modal embeddings in terms of uniformity and alignment.\n\n2. The paper proposes a new end-to-end fine-tuning method for improving the robustness and transferability of multi-modal representations for downstream tasks.\n\n3. The proposed method, Geodesic Multi-Modal Mixup, is shown to learn a more robust representation and provide improved performance on various downstream tasks such as retrieval, classification, and structure-awareness.", "weaknesses": "1. The paper highlights that the analysis of learned embeddings and transferability for cross-modal tasks has not been explored well, i.e., image captioning, visual questions answering tasks, and so on.\n\n2. The multiple multi-modal mixup seems too complex, would it cast shadow on the learning and generalization ability? More multi-modal backbones like the variants of CLIP or other vision-language models like DeCLIP, FILIP, CLOOB, CyCLIP. Since the approach is agnostic to the architecture and models, I would encourage the authors to perform additional experiments to demonstrate the effectiveness of the proposed approach on other vision-language models. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your valuable feedback and that you find value in our work. We agree with your concerns and have conducted additional experiments based on your feedback.\n\n---\n> **Q1)** The paper highlights that the analysis of learned embeddings and transferability for cross-modal tasks has not been explored well, i.e., image captioning, visual questions answering tasks, and so on.\n> **Q2)** The multiple multi-modal mixup seems too complex, would it cast shadow on the learning and generalization ability? More multi-modal backbones like the variants of CLIP or other vision-language models like DeCLIP, FILIP, CLOOB, CyCLIP. Since the approach is agnostic to the architecture and models, I would encourage the authors to perform additional experiments to demonstrate the effectiveness of the proposed approach on other vision-language models.\n\n**A2)** \nDespite its effectiveness, as you noticed, $m^3$-Mix has multiple components contributing to learning and generalization, so it seems hard to clear out the core factor of learning and generalization. To clearly determine the effect of our main contribution ‘multi-modal mixup’, we perform all additional experiments in this rebuttal with $m^2$-Mix.\n\n**A1, 2)** \nIn our manuscript, we performed the following tasks: (uni-modal) few-shot image classification under general and distribution shift settings; (multi-modal) retrieval, sentiment classification, and embedding arithmetic. To further validate our method, we evaluate our method on the image captioning task with CoCa [Yu et al. 2022] model. We fine-tune CoCa on MS COCO dataset via CoCa objective (contrastive and captioning loss), and showcase the results of image captioning and retrieval in Table 2. and Table 3. of the global response, respectively. Our $m^2$-Mix consistently outperforms the standard fine-tuning method with regard to almost all metrics of interest. From this, we confirm that our $m^2$-Mix not only benefits the discriminative tasks but also the generative task.\n\n**A2)** \nBesides, in Table 1., we evaluate our method on CyCLIP [Goel et al. 2022] for few-shot image classification. Here, $m^2$-Mix also successfully improves performance across all considered datasets. This further demonstrates that our $m^2$-Mix is a versatile fine-tuning approach that can be seamlessly integrated across VL models (CLIP, CyCLIP, CoCa, and so on).\n\n---\n\n[Yu et al. 2022] Jiahui Yu, et al. \"CoCa: Contrastive Captioners are Image-Text Foundation Models\". Transactions on Machine Learning Research. (2022).\n\n[Goel et al. 2022] Goel, Shashank, et al. \"Cyclip: Cyclic contrastive language-image pretraining.\" Advances in Neural Information Processing Systems 35 (2022): 6704-6719."}, {"review_id": "vw8dmt8KxM", "reviewer": "Reviewer_ECSD", "summary": "\nThis paper studies a data augmentation method for effectively finetuning multi-modal model (i.e., CLIP). To address poor alignment of image and language space, the authors mix the embeddings of image and text while considering the geometry of the hypersphere. The authors give theoretical analysis as well as extensive experiments on several tasks including retrieval, calibration, few-/zero-shot classification.  \n", "questions": "Please see “Weaknesses” section.", "limitations": "The authors have adequately discussed the limitations of the proposed method in the supplementary material. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1) As a data augmentation strategy for contrastive learning, I think that the proposed method has novelty to some extent, different from previous arts such as i-Mix and Un-Mix. Specifically, the proposed data augmentation is performed in the embedding spaces while considering the geometry of the hyperspheres, in contrast with i-Mix and Un-Mix which mix raw images. \n\n\n(2) The experiments are diverse and extensive and can validate the effectiveness of the proposed method. In particular, the performance on cross-modal retrieval performs significantly better than the competing methods. \n", "weaknesses": "(1) The first contribution regarding poor alignment of the two modal space of CLIP seems to be overclaimed. \n\nAs far as I know, Liang et al. [24] has clearly disclosed the modality gap in multi-modal contrastive representation learning, clarifying the two modalities in CLIP (and other models) are separately clustered (restricted to narrow cones) and have poor alignment and uniformity. What is difference between the authors finding and [24]? I expect an in-depth discussion in this respect and relax the claim if appropriate. \n\n(2) Mixup-like data augmentation in feature space has been proposed in previous works including [Ref01] and [Ref02]. I suggest authors explicitly discuss the connection and difference from them.  \n\n[Ref01] Terrance DeVries and Graham W Taylor. Dataset augmentation in feature space. arXiv preprint arXiv:1702.05538, 2017a.\n[Ref02] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your productive feedback on our paper and your remark on the strength of our methodology and experiments.\n\n> **Q1)** The first contribution regarding poor alignment of the two modal space of CLIP seems to be overclaimed. As far as I know, Liang et al. [24] has clearly disclosed the modality gap in multi-modal contrastive representation learning, clarifying the two modalities in CLIP (and other models) are separately clustered (restricted to narrow cones) and have poor alignment and uniformity. What is difference between the authors finding and [24]? I expect an in-depth discussion in this respect and relax the claim if appropriate.\n\n**A1)** As you pointed out, it is already well clarified by [Liang et al. 2022] that pre-trained vision-language models have separated embeddings for each modality. While their finding is mainly based on the pair-wise cosine similarity and theoretical analysis, we weigh our attention on quantitative measurements of uniformity and alignment (Fig 2. of our manuscript), which are lacking in the previous work. While the author of CyCLIP [Goel et al. 2022] analyzed the uniformity and alignment of CLIP embeddings, they only considered the zero-shot embedding of CLIP. Meanwhile, our observation includes the fine-tuned CLIP (which is lacking in the CyCLIP paper) as well as pre-trained CLIP. We will be sure to detail these differences more directly in our paper, and soften our claim about the finding on separated embeddings in pre-trained CLIP. Thanks for the great feedback!\n\n> **Q2)** Mixup-like data augmentation in feature space has been proposed in previous works including [Ref01] and [Ref02]. I suggest authors explicitly discuss the connection and difference from them. [Ref01] Terrance DeVries and Graham W Taylor. Dataset augmentation in feature space. arXiv preprint arXiv:1702.05538, 2017a. [Ref02] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.\n\n**A2)** Thank you for suggesting valuable related works. Our methods and the related works, [Ref01] [Ref02], exploit the data augmentation in feature space, not input space. However, there are two contributions to our methods. First, we propose a hidden feature mixup method on heterogeneous modality. When there are text and image feature spaces, our $m^{2}$-Mix explores inter-relationship by mixing image and text features, while the previous methods explore intra-relationship by mixing image features themselves. Second, our methods interpolate between data points on the hypersphere, not Euclidean space. As shown in Equation (4) of the main paper, we interpolate the data points by geodesic. As a result, the augmented features by our methods lie on the hypersphere. Self-supervised methods project the embedding on the hypersphere; therefore, it is important to guarantee that the augmented features are on the hypersphere. Table 6 indicates the importance of geodesic interpolation.\n\nIn other words, our method is a more generalized Mixup that handles the multimodal and hypersphere spaces. From Equation (4) of the main paper, our method degenerates to the traditional Mixup or manifold Mixup if we change a and b as the same domain (text or image) and its coefficient as just $\\lambda$ instead of the sinusoidal function with $\\lambda$ and $\\theta$.\n\n---\n\n[Liang et al. 2022] Liang, Victor Weixin, et al. \"Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 17612-17625.\n\n[Goel et al. 2022] Goel, Shashank, et al. \"Cyclip: Cyclic contrastive language-image pretraining.\" Advances in Neural Information Processing Systems 35 (2022): 6704-6719."}, {"review_id": "jMRsxvLUUA", "reviewer": "Reviewer_9Koy", "summary": "The paper proposes three mix-up inspired regularizers for finetuning CLIP as a way to mitigate the visual-text feature gap in the CLIP feature space. The paper proposes some theory to backup why alignment of text feature space and image feature space might be a good idea. Results in cross-modal retrieval and zero-shot classification justify the method.", "questions": "N/A", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The results seem reasonable at first glance, although they are significantly below state-of-the-art.\n\nThe method has some novelty.", "weaknesses": "(1) The modality gap is a well-known phenomenon [Liang22] [CyCLIP]. I think the authors misrepresent their contributions by implying that that they discovered this phenomenon by repeated stating in the introduction that \"we found .. \" that the image and text embeddings occupy separate subspaces in the hypersphere (without citing the prior work) . This phenomenon was studied in  [Liang22] and  [CyCLIP] .\n\n - Additionally, the findings of [Liang22] are somewhat misrepresented. In Table 1 of [Liang22], those authors find that regularizing the modality gap can increase or decrease the zero-shot performance on downstream tasks. In their discussion, Liang et al. state that a larger gap \"may help some fairness and zero-shot learning applications\". This is not discussed in the present work.\n\n(2) Theoretical analysis is weak:\n\n - Theorem 4.1 only applies to $d=2$\n\n- Proposition applies as $\\tau \\rightarrow 0^+$, and in this limit, the sum of CLIP loss and $m^2$-mix regularizer reduces to the negative sum of alignment and uniformity. However, the authors already claimed that vanishing small $\\tau$ is a bad thing, since \"We speculate this limitation is derived from (1) ... (2) vanished learnable $\\tau$ (0.01) in $\\mathcal{L}_{CLIP}$.\" Why then would you prove something about a vanishingly small $\\tau$?\n\n(3) It is unclear to me why $m^2$-mix + CLIP is not sufficient? Why do we need the additional regularization? Does  $m^2$-mix by itself lead to consistent gains in performance?\n\n(4) Concerns about experiments: \n\n - Why do you use ViT-B/32 instead of ViT-B/16? ViT-B/16 is more common [BLIP][Maple] and many papers don't even present ViT-B/32 results. This makes it hard for readers to compare the proposed modality mix-up to current state-of-the-art. The results in Table 1, 2, and 5 are significantly worse than the results in [BLIP] and [Maple].\n\n- Table 4: Why are there only results for Pets, SVHN, and CLEVR? It is hard to determine whether these results would generalize, perhaps you could run results on more datasets, e.g. the 10 datasets used in [Maple]?\n\n- In Table 4, $m^2$-mix leads to about 0.7 % gain over FT, while a much larger 5.2 % gain comes from $m^3$-mix and tuning $\\tau$. Table 5 shows that $m^2$-mix is worse than ZS by 3 %. This severely undermines the claims in this paper, which is that cross-modality mixup (i.e. $m^2$-mix) leads to gains in generalization performance. Where there are gains, most gains seem to come from the intra-modal mix-up regularization. \n\n[Liang22] Liang, Victor Weixin, et al. \"Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 17612-17625.\n\n[CyCLIP]  Goel, Shashank, et al. \"Cyclip: Cyclic contrastive language-image pretraining.\" Advances in Neural Information Processing Systems 35 (2022): 6704-6719.\n\n[BLIP] Li, Junnan, et al. \"Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.\" International Conference on Machine Learning. PMLR, 2022.\n\n[Maple]  Khattak, Muhammad Uzair, et al. \"Maple: Multi-modal prompt learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nMinor:\n\nIn my opinion, Figure 2 is misleading. This figure displays the text and image embeddings on opposite sides of a sphere. In reality, the modality gap is not so large that text and image embeddings corresponding to the same image-caption pair fall on opposite sides of the hypersphere. So I don't think Figure 2 is a good illustration of what's really going on. \n\nin theorem 4.1: Why is $\\tilde{x} = x_1 + x_2$? shouldn't $\\tilde{x}$ by the geodesic mean of the two so that $\\tilde{x}$ still lies on the sphere?\n\nSince this is a paper about mixup, the reader might expect FT+mixup to be a reasonable baseline (not feature mixup, but mixup in image input space).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1)** (1) Modality gap is a well-known phenomenon [Liang22] [CyCLIP]. The authors misrepresent their contributions by implying that they discovered this. Additionally, in Tab1 of [Liang22], authors find that regularizing gap can increase or decrease the zero-shot performances and fairness. This is not discussed in the present work.\n\n**A1)** We'd like to thank you for your valuable comments. We already noted the papers and cited them in our paper.  As you pointed out, it is also investigated by [Liang22] and others that pre-trained VLMs have separated embeddings. However, we put more of our weight on quantitative uniformity and alignment perspective to analyze the VLM’s embedding space, and we propose a principle solution, $m^2$-mix. Instead, [Liang22] mainly analyzes it via pair-wise cosine similarity and theoretical evidence. While [Goel22] conducted the experiments on uniformity and alignment, they only provided the results on the zero-shot embeddings of pre-trained CLIP and lack the fine-tuned CLIP results which are observed by us. We'll be sure to detail these differences more directly in our paper, and relax our claim about the contribution.\n\nBased on your feedback, we have designed the experiment for a fairness application, and we will attach it to the final version of our work. In terms of zero-shot learning, we confirmed that reducing the modality gap by our $m^2$/$m^3$-Mix has advantages over some cases: zero-shot transfer under distribution shift in Sec 5.4, SIMAT in Sec 5.6, and retrieval in Tab. 3 of global response. In addition, recent work [Ouali23] shows that reducing gap helps to improve base-to-new generalization and zero-shot transfer under distribution shift. These results are somewhat at odds with the finding of [Liang22]. We believe that \"relationship between modality gap and downstream performance\" is a controversial topic and deserves further research. However, through our experiments, we conjecture that aligning embeddings and simultaneously increasing uniformity is beneficial for a variety of downstream tasks. Therefore, we argue that fine-tuning of VL models needs to consider both alignment and uniformity, and our methods $m^{2}$-Mix achieves high alignment and uniformity with downstream task performance improvements.\n\n> **Q2)** 2-1) Thm 4.1 only applies to $d=2$, 2-2) Prop 4.2 the authors already claimed that vanishing small $tau$ is a bad thing, why then would you prove something about a vanishingly small $\\tau$?\n \n4.1. The convolution of two independent von Mises distribution is also von Mises distribution approximately [Marković12], [Mardia00]. But, it has been known that the convolution of two independent von Mises-Fisher distribution does not have an exact form. Therefore, our theoretical analysis focuses on the von Mises distribution settings. We'll include the empirical evidence for von Mises-Fisher distribution as well as von Mises distribution in our final paper\n \n4.2. To conduct theoretical analysis on m2mix based contrastive loss, we assumed the vanishing $\\tau$. This is from the consideration of the reality situation because the vanishing $\\tau$ is actually the case during fine-tuning of CLIP. Specifically, the small $\\tau$ (0.01) of the pre-train CLIP remains small throughout the entire fine-tuning process, unless we manually increase it with a fixed constant value. Through Prop 4.2, we wanted to claim that robust representation is learned by improving alignment and uniformity with m2mix, even in this tau-vanishing situation.\n\n> **Q3)** (3) It is unclear to me why $m^2$-mix+CLIP is not sufficient? Does $m^2$-mix by itself lead to consistent gains in performance?\n\nWhile the $m^2$-Mix+CLIP induces performance improvements over baseline methods in few-shot adaptation (Tab. 4) and multi-modal classification (Tab.7) in draft, thanks to your suggestion, we felt that more extensive experiments are required to validate the  $m^2$-Mix. To this end, we conducted all the additional experiments with $m^2$-Mix, not with $m^3$-Mix. On few-shot classification (Tab.1), image captioning (Tab.2), and retrieval (Tab.3) with 3 different models, our $m^2$-Mix shows consistent improvement over baselines. These results augment the empirical evidence that supports the effectiveness of $m^2$-Mix.\n\n> **Q4)** concern about experiments:\n* ViT-B/32: Because our primary goal of this study is aiming at devising a robust fine-tuning method, we conducted the experiments based on ViT-B/32 in our manuscript for fast validation of the proposed method. However, we agree with you that validations in a standardized setting (ViT-B/16) and applicability to SOTA vision-language models are needed. To address this, we accomplished additional experiments on the global response with ViT-B/16 (Tab. 1) and CoCa-ViT-L/14 [Yu22] (Tab. 2 and 3). In Tab. 1, 2, and 3, our m2-mix consistently outperforms compare to the baselines with regard to few-shot classification, captioning, and retrieval. These results verify that our method can also give favorable advantages to fine-tuning SOTA as well as a standard model backbone\n* Datasets: Thanks for pointing out where we need to improve. In Tab1. of global response, we additionally provide the results on four more transfer learning benchmark datasets and demonstrate the effectiveness of m2-mix on these datasets. We will cover all remaining datasets in the final version.\n* Effect of m2-mix: The amount of improvement depends on the datasets and tasks. m2-mix achieved 1.2% and 1.6% improvements over FT on SVHN and ImageNet in Sec 5.4, and in Tab. 1 of the general response, m2-mix outperforms FT by 8.5%, 1.8%, 4.7%, and 5.2% on CLIP-Air, CLIP-Cars, CyCLIP-Air, and CyCLIP-Cars. Importantly, our m2-mix (without intra-modal mixups) shows consistent improvements on diverse setups.\n\n*due to space limit, we'll address your minor remarks and attach reference via official comments from tomorrow.*"}, {"review_id": "W8WbHCBytx", "reviewer": "Reviewer_iPuA", "summary": "The paper proposes Geodesic Multi-Modal Mixup, which mixes heterogeneous modality embeddings on the hypersphere to generate harder negative samples for the contrastive loss of multi-modal models. Such training scheme improves modality alignment and uniformity. The method was evaluated on various tasks including retrieval, calibration, few-shot classification, and embedding arithmetic.", "questions": "Overall, the paper was very well-written with plenty of applications. The mentioned weaknesses are relatively minor, and no critical issue was found on my side.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The multi-modal mixup approach is proposed opportunely.\n- The method is comprehensive and straightforward.\n- The idea of mixing different modalities is novel and interesting.\n- Extensive experiments are impressive with significant gains in several settings.", "weaknesses": "1. I believe the key hyperparameter of the proposed method is lambda, as this should control the \"hardness\" of the mixed samples.  Please provide an analyses (at least a sensitivity test) on this component.\n2. In some cases, the authors only report m^2-Mix, while others report only m^3-Mix. It would be great if the authors could provide full tables with both mixup methods, and discuss reasons if there are some differences in performance between the two.\n3. Minor presentation issues:\n- Some figures have too small a font size; e.g. Figure 1,6.\n- Eq. (5) and L202-203 are confusing at first glance. Please use separate lines or a comma in between.\n- Figure 6 right's title should be \"GMC w/ M^2-Mix\"?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions, and they are exceedingly helpful for us to improve our paper. We are grateful that you find strength in our idea, actual method, and experiments!\n\n> **Comment1)** I believe the key hyperparameter of the proposed method is lambda, as this should control the \"hardness\" of the mixed samples. Please provide analyses (at least a sensitivity test) on this component.\n\nTo address your suggestion, we present sensitivity analyses (Fig. 1 and Fig. 2 in the global response) on parameters of Beta distribution that contribute to the mixing rate between two embeddings. From the experiments, we would like to remark follows:\n* $m^2$-mix generally shows stronger performances when the parameter $\\alpha$ lay between 0.1 and 0.5 (U-shaped) rather than larger values (uniform or reversed U-shaped). This is consistent with the well-worked interval that was proposed in the original Mixup [Zhang et al. 2018] paper but different from the selected value by Manifold Mixup (alpha=2.0) [Verma et al. 2019], and implies that the many-to-small strategy is better than the half-to-half mixing strategy in our geodesic multi-modal mixup.\n* The linear scheduling variants of Beta parameters (0.1 -> 1.0, 0.1 -> 2.0, and their opposite direction counterparts) achieve promising results in some cases, e.g., 1.0 -> 0.1 and 2.0 -> 0.1 in Stanford Cars. For scheduling variants, it is important to ensure that the shape of Beta distribution approaches to U-shape at the endpoints of learning.\n* While scheduling-based variants have shown promising results in some cases, they are not significantly better than the fixed-parameter strategy, so for the sake of simplicity, we maintain the fixed-parameter strategy for the multi-modal mixup.\n\n\n> **Comment2)** In some cases, the authors only report $m^2$-Mix, while others report only $m^3$-Mix. It would be great if the authors could provide full tables with both mixup methods, and discuss reasons if there are some differences in performance between the two.\n\nWhile we conducted the experiments (in the global response) only with $m^2$-Mix for this rebuttal due to limited time and resources, we agree with your suggestion and have planned to put both $m^2$-Mix and $m^3$-Mix in all tables consistently on the final version of the paper and provide detailed discussions on differences in performance.\n\n\n> **Comment3)** Minor presentation issues\n\nThank you for taking the time to review the details as well as primary messages in our paper, we will reflect on all your remarks to revise our paper.\n\n\n---\n\n[Zhang et al. 2018] Zhang, Hongyi, et al. \"mixup: Beyond Empirical Risk Minimization.\" International Conference on Learning Representations. 2018.\n[Verma et al. 2019] Verma, Vikas, et al. \"Manifold mixup: Better representations by interpolating hidden states.\" International conference on machine learning. PMLR, 2019.\n"}]

Author rebuttal:
# Global Response
We thank the reviewers for taking the time to review our paper and for your valuable feedback. We have carefully considered your comments and reflected in our response. In this global response, 1) we first provide a brief review of our draft and then 2) provide a summary of additional statements that we’ve made to address your feedback and suggestions during the rebuttal period. Finally, 3) we elaborate on settings of additional experiments(**Tables and Figures are attached to pdf**).
- - -
## 1. Review of research highlights
* We observed that both the pre-trained CLIP and its naively fine-tuned counterpart  have separated embedding space for each modality and show quantitatively poor alignment and uniformity scores, which may limit the transferability of learned embedding.
* From our findings, we devised a new fundamental approach ‘geodesic multi-modal mixup ($m^2$-Mix)’-based contrastive learning for robust fine-tuning to enhance the alignment and uniformity of embeddings.
* We provide two theoretical analyses of the proposed method to support its validity
  * hardness-guarantee of the generated sample by $m^2$-Mix, which is crucial for the success of contrastive learning.
  * asymptotic behavior of standard CLIP loss with $m^2$-Mix that maximizes the alignment and uniformity with few assumptions.
* Through extensive experiments on retrieval, calibration, few-shot adaptation, zero-shot transfer under distribution shift, multi-modal classification under modality missing, and embedding arithmetic, we demonstrate that the proposed method effectively improves performance over baseline methods.

## 2. Additional remarks from the rebuttal period
During the rebuttal period, we conducted several additional experiments to address the reviewers' concerns and suggestions. Here, we provide our new findings from the results of extra experiments in summary:
* (9Koy) Without the aid of intra-modal mixup regularization such as V/L/VL-Mix, our $m^2$-Mix independently yields consistent performance gains on a variety of tasks, and in some cases the gains are significant (+5~8% Acc).
* (9Koy) We evaluate our $m^2$-Mix with ViT-B/16 (not B/32 of draft) on four additional transfer learning benchmarks, and $m^2$-Mix achieves consistent performance improvements.
* (yWq1) Beyond the basic CLIP, we demonstrated that our $m^2$-Mix is helpful to other recent vision-language models (VLM) such as CyCLIP [Goel et al. 2022] and CoCa [Yu et al. 2022].
* (yWq1) Our method gives an advantage not only for retrieval, multi-modal classification, and embedding arithmetic, but also for generative tasks such as image captioning.
* (uSAw, 9Koy) $m^2$-Mix brings benefits to large-scale state-of-the-art VLM (e.g., CoCa-Large that has 9 times more parameters compared to CLIP-Base) as well as standard CLIP.
* (iPuA) We tweaked the parameters of the Beta distribution and found that our $m^2$-Mix performs optimally between 0.1 and 0.5, just like the standard Mixup. This suggests that when choosing a mixed sample, mixing one side more than the other, rather than half-and-half, leads to more stable learning, which contributes to better generalization performance.

## 3. Experimental setup
We evaluated our method on three tasks (few-shot classification, image captioning, and retrieval), and the experimental setup for each task is as follows:

### 3.1. few-shot classification
We consider four standard benchmark datasets (EuroSAT, FGVC Aircraft, UCF101, and Stanford Cars) and two VLMs with different backbone and pre-trained weights (CLIP-ViT-B/32 and CyCLIP-ResNet50 with official pre-trained checkpoints) that are not included in our manuscript.
We adopt the same setting with Sec 5.4 of our draft to implement the contrastive learning-based fine-tuning on these image-label paired datasets, i.e., promptizing a label with a common context “a photo of {classname}” and regard this as a caption per images. As baselines, we consider zero-shot inference, vanilla fine-tuning with traditional image space mixups (Mixup [Zhang et al. 2018] and Cutmix [Yun et al. 2019]), robust fine-tuning method (WiSE-FT [Wortsman et al. 2022]), and a state-of-the-art (SOTA) parameter-efficient fine-tuning method (MaPLe [Khattak et al. 2023]). All experiments are done under a 16-shot training set and full test set, and top-1 accuracy is reported.
###  3.2. image captioning and retrieval
We adopt CoCa-ViT-L/14 with pre-trained checkpoint `laion2b_s13b_b90k` from the OpenCLIP library as our VLM to demonstrate the effectiveness of our method on a large-scale SOTA model. We fine-tune CoCa on MS COCO training set with 1) only captioning loss (this is a strategy that CoCa’s authors adopted), 2) contrastive loss + captioning loss, and 3) contrastive loss + $m^2$-Mix + captioning loss. After training, we evaluate the models in terms of captioning metrics such as BLEU4, CIDEr, etc  (on COCO) and retrieval recalls  (on COCO and Flickr30k).
- - - 
[Goel et al. 2022] Goel, Shashank, et al. "Cyclip: Cyclic contrastive language-image pretraining." Advances in Neural Information Processing Systems 35 (2022)

[Yu et al. 2022] Jiahui Yu, et al. "CoCa: Contrastive Captioners are Image-Text Foundation Models". Transactions on Machine Learning Research. (2022).

[Zhang et al. 2018] Zhang, Hongyi, et al. "mixup: Beyond Empirical Risk Minimization." International Conference on Learning Representations. 2018.

[Yun et al. 2019] Yun, Sangdoo, et al. "Cutmix: Regularization strategy to train strong classifiers with localizable features." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[Wortsman et al. 2022] Wortsman, Mitchell, et al. "Robust fine-tuning of zero-shot models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[Khattak et al. 2023] Khattak, Muhammad Uzair, et al. "Maple: Multi-modal prompt learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.


Meta-review: This submission has received five reviews with mixed ratings ranging from 3 to 7. The reviews were answered with a rebuttal and then discussed between the authors and reviewers. This paper was then discussed in depth by the area and senior are chair. 

The reviewers list as positive points that the approach presents a novel perspective and contains some novelty. One reviewer asks for experiments with a larger backbone that is answered to in the rebuttal but did not find influence on the respective review score. There are other minor technical questions all of which have been answered in the rebuttal as well. 

The main remaining discussion point is the claim of novelty with respect to Liang et al. 2022. We understand that this may have been concurrent work at arXiv, however that is more than one and a half year ago, so while novelty claims in the arXiv version may be justified in a conference version those passages in the text need to be adjusted. It is important to note that this concerns the text of the paper, we acknowledge that in the experiments a comparison to [24] has been included, so this is not a case where prior work has been ignored. Therefore in the discussion we concluded that the reject recommendation of reviewer 9KoY that is based mainly on this point is too strong. We believe based on the rebuttal that the authors are willing to reword carefully their discoveries with respect to Liang, [24] and their own arXiv work.

Besides this dispute most other points are answered to and the paper presents a new perspective on a relevant topic. Thus we decided to down-weight  the most negative score and recommend acceptance of the paper. The authors are expected to work the discussion and the rebuttal into the final camera ready submission.
Decision: Accept (poster)