SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Changdae Oh*
Junhyuk So*
POSTECH
Hoyoon Byun
YongTaek Lim
Minchul Shin
Jong-June Jeon
Kyungwoo Song+
Pre-trained multi-modal models, such as CLIP, provide transferable embeddings and show promising results in diverse applications. However, the analysis of learned multi-modal embeddings is relatively unexplored, and the embedding transferability can be improved. In this work, we observe that CLIP holds separated embedding subspaces for two different modalities, and then we investigate it through the lens of uniformity-alignment to measure the quality of learned representation. Both theoretically and empirically, we show that CLIP retains poor uniformity and alignment even after fine-tuning. Such a lack of alignment and uniformity might restrict the transferability and robustness of embeddings. To this end, we devise a new fine-tuning method for robust representation equipping better alignment and uniformity. First, we propose a Geodesic Multi-Modal Mixup that mixes the embeddings of image and text to generate hard negative samples on the hypersphere. Then, we fine-tune the model on hard negatives as well as original negatives and positives with contrastive loss. Based on the theoretical analysis about hardness guarantee and limiting behavior, we justify the use of our method. Extensive experiments on retrieval, calibration, few- or zero-shot classification (under distribution shift), embedding arithmetic, and image captioning further show that our method provides transferable representations, enabling robust model adaptation on diverse tasks.
Code: https://github.com/changdaeoh/multimodal-mixup
Witnessing the remarkable success of large-scale pre-training approaches, there have been numerous attempts to build a single general-purpose model (so-called foundation model [1]) rather than casting multiple task-specific models from scratch. Once built, it can be adapted (e.g., fine-tuned) to a wide variety of downstream tasks by leveraging its transferable representation. To construct such a general-purpose model, besides architectural design [2, 3, 4] and datasets [5, 6, 7], learning methods [8, 9, 10, 11, 12] have been known to be crucial for inducing transferable representation and for adapting the model robustly.
Contrastive Learning (CL) [13, 14, 15] is one of the most popular learning methods that constructs a discriminative embedding space by minimizing distances between positive pairs while maximizing them between negative pairs. Based on its versatility, CL has been widely adopted to (pre-) train models on various domains [15, 16, 17, 18]. Beyond CL on a single modality, CLIP [19] popularized
* Equal contribution (changdae.oh@uos.ac.kr; junhyukso@postech.ac.kr), † Corresponding author
Figure 1: (Left) pre-trained CLIP has two separated clusters for image and text, with a large unexplored interspace. As a result, it has poor uniformity and alignment that might limit the embedding transferability. (Middle) For robust representation, our geodesic multi-modal Mixup ( $m^2$ -Mix) explores the unexploited interspace by mixing the heterogeneous embeddings on hypersphere. Generated samples by  $m^2$ -Mix are regarded as hard negatives for contrastive loss. (Right) As a result, fine-tuning with  $m^2$ -Mix induces robust representation with better transferability.
multi-modal CL, which aims to produce close embeddings for paired image-text instances and dissimilar ones for non-paired instances. Due to its generalization capability and transferability, pretrained CLIP and its embeddings have been employed on various downstream tasks [20, 21, 22, 23].
However, we observed an unexpected phenomenon: while CLIP's learning objective is designed to align the image and text embeddings explicitly, it has two separate subspaces for each modality, and there are large unexplored interspaces between them as in Fig. 1 (illustration) and Fig. 2 (DOSNES [24] visualization). We further analyzed this through the lens of uniformity-alignment [25, 26], which is well-studied under uni-modal CL settings but unexplored on multi-modal CLs, and found that CLIP has a poor uniformity-alignment (Fig. 2 middle) due to its bipartite embedding structure. Theoretically and empirically, we confirmed that this property is unchanged even after fine-tuning. As discussed by Wang et al. [25], low uniformity-alignment may limit the embedding transferability. Liang et al. [27], concurrently (in terms of ArXiv preprints) took a similar observation, modality gap, which incurs a bunch of following works, and they also found that increasing temperature  $(\tau)$  in contrastive loss could somewhat reduce the embedding separation. However, varying the temperature requires manual engineering for each downstream task, and it incurs an inevitable trade-off between uniformity-alignment [26]. It raises an important question motivating this work: "How can we obtain a multi-modal representation dealing better with the uniformity-alignment for robust transfer?"
To answer this, we propose a fundamental learning method, geodesic multi-modal Mixup ( $m^2$ -Mix). As shown in Fig. 1,  $m^2$ -Mix blends the embeddings of different domains, e.g., image and text, and utilizes the mixtures as new entrees of contrastive loss. Our  $m^2$ -Mix contributes to robust representation learning in three perspectives. First, it generates hard negative samples which have high similarity with positives (incentivizing alignment), and they are known to be crucial for robust CL [28, 29, 26]. We provide a theoretical guarantee of the hardness of  $m^2$ -Mixed samples with empirical results. Second,  $m^2$ -Mix interpolates samples from heterogeneous domains to expand the effective embedding space (increasing uniformity) untapped by pre-trained CLIP, and this is further supported by the limiting behavior of  $m^2$ -Mix (see Prop. 4.2). Third,  $m^2$ -Mix produces virtual augmented samples in embedding space, so it endows a better perception on the out-of-distribution samples as well as in-distribution samples than standard CL.
Contributions: (1) We found that CLIP has a bipartite embedding space with poor uniformity-alignment, which may limit the embedding transferability for downstream tasks. (2) To address this, we propose a new fine-tuning method based on geodesic multi-modal Mixup,  $m^2$ -Mix. To our knowledge, this is the first work that adopts direct mixtures between heterogeneous embeddings. (3) We devised  $m^2$ -Mix as a geodesic Mixup to ensure the mixed embeddings are on a hypersphere, and this is beneficial for stable learning with  $L_2$ -normalized embeddings. (4) We validate our method by theoretical analyses and extensive experiments on retrieval, calibration, full-/few-shot classification (under distribution shift), embedding arithmetic, and image captioning.
Contrastive Representation Learning Contrastive Learning (CL) has been widely utilized for representation learning on various domains [15, 17, 30, 19, 31]. The goal of CL is to learn an embedding function that maps data into an embedding space so that semantically similar data
have close embeddings. Most of CLs adopt  $L_{2}$ -normalized embedding [13, 14, 15, 19] for stable learning [32], and it makes the embeddings lie on a unit hypersphere. There are key properties to analyze the hyperspherical embeddings, so-called Uniformity and Alignment [25]. Better uniformity and alignment can be regarded as a higher embedding transferability, so it is related to performance on downstream tasks. However, uniformity-alignment analysis on multi-modal settings has not been sufficiently explored yet [33], and we found that CLIP has poor uniformity-alignment before and even after fine-tuning. Meanwhile, it is known that hard negatives for constructing contrastive pairs are necessary to increase the robustness of contrastive representation learning [34, 35, 26, 36]. To this end, we devise a new approach for multi-modal CL, which generates hard negatives and achieves better uniformity-alignment for robust and transferable representation.
Mixup There have been numerous papers that claim Mixup [37] is helpful for robust representation learning and alleviates the overconfident problems and failure under distribution shift as well as the in-distribution accuracy [38, 39, 40]. Based on such success of Mixup, many works are adopting it as a component of learning frameworks (on vision [41, 42, 43, 44], language [45, 46, 47] and graph [48, 49]). Besides, CL with Mixup to help representation learning has also been studied. While traditional CL annotates as 1 for positive pairs and 0 for negative pairs,  $i$ -Mix [50] and UnMix [51] interpolate the images with ratio  $\lambda$ , and adopt contrastive loss with pseudo labels according to the mixing ratio  $\lambda$ . However, there are few works on Mixup for multi-modal learning [52, 53]. STEMM [52] mixes the speech and text features to align them in shared embedding space, but it cannot be widely used for multi-modal learning because of its architecture-specific design. Therefore, we propose  $m^2$ -Mix, that can be broadly adopted for robust multi-modal representation learning.
Multi-modal Learning To build a universal intelligence system that simultaneously processes multi-modal data streams, some focus on developing unified frameworks for multi-modal perception [2, 3, 54, 4, 55], others aim at representation learning under multi-modal correspondence [19, 31, 56, 57]. This work focuses on CLIP [19], the representative multi-modal representation learner. Based on the generalizable embedding space, CLIP has been utilized on numerous tasks [58, 59, 60, 61], and there are many attempts to fine-tune it efficiently [62, 63, 64, 65]. Moreover, robust fine-tuning methods have also been actively studied for generalization on both in- and out-of-distribution data. Wortsman et al. [10] propose a weight-ensemble method that interpolates the pre-trained and fine-tuned weights, Kumar et al. [11] adopt a two-stage learning scheme that combines linear probing and fine-tuning, and Goyal et al. [66] adopt contrastive loss during fine-tuning on image classification. However, previous works have only focused on uni-modal settings and downstream tasks are restricted to the image classification. This paper provides robust fine-tuning methods for multi-modal settings.
3 Observations and Problem Define
For a given batch  $D = \{x_{i},y_{i}\}_{i = 1}^{M}$  of  $M$  instances where  $(x_{i},y_{i})$  denotes the  $i$ -th image-text pair, the goal of multi-modal learning is to learn the pair-wise relations between image and text. To do this, CLIP [19] learns image encoder  $f(\cdot ;\theta_1)$  and text encoder  $g(\cdot ;\theta_2)$ , so that embedding pairs  $\{I_i,T_i\}_{i = 1}^M = \{f(x_i;\theta_1),g(y_i;\theta_2)\}_{i = 1}^M$  get closer to each other. Note that  $I_{i}$  and  $T_{i}$  are  $L_{2}$ -normalized unit vectors in most vision-language models, and they lie on the hypersphere. For simplicity, we omit the learning parameters  $\theta_{1}$  and  $\theta_{2}$  from all the following equations. CLIP adopts InfoNCE-style [13] loss  $C(\cdot ,\cdot)$  to enforce the similarity between positive pairs  $(x_{i},y_{i})$  and distance among all remain negative pairs  $(x_{i},y_{j})$ . This is formulated as below (Eq. 1):
$$
C (I, T) = \frac {1}{M} \sum_ {i = 1} ^ {M} - \log \frac {\exp \left(\operatorname {s i m} \left(I _ {i} , T _ {i}\right) / \tau\right)}{\sum_ {j = 1} ^ {M} \exp \left(\operatorname {s i m} \left(I _ {i} , T _ {j}\right) / \tau\right)} \quad \mathcal {L} _ {\mathrm {C L I P}} = \frac {1}{2} (C (I, T) + C (T, I)) \tag {1}
$$
Like many other CL approaches, CLIP uses a dot product as a similarity calculation  $sim(\cdot, \cdot)$  between two vectors and governs  $\tau$  as a learnable temperature that controls the scale of measured similarity. Now, we analyze the multi-modal embedding in terms of uniformity and alignment, well-known properties in uni-modal CL literature [25, 26] but unexplored in multi-modal settings. Alignment $^1$  (Eq. 2) evaluates the difference between distances (or similarities) of positive pairs compared with
<sup>1</sup>Original formulation of alignment is  $\mathbb{E}_{(x_i,y_i)}[||f(x_i) - g(y_i)||_2^2]$ , which ignores the similarity among negative pairs. We found that it is not directly related to downstream performances, so we modified the alignment to relative-alignment that handles the similarities of both positive and negative pairs.
Figure 2: (Left) DOSNES [24] visualization of CLIP's embedding space on Flickr 30k [67]. Greens and oranges denote the image and text embedding, respectively. Embeddings of pre-trained (ZS) and naively fine-tuned (FT) ones have two separate clusters with low uniformity and alignment (Middle), which may limit embedding transferability. Multi-modal mixup induces more aligned and uniformly distributed embeddings, (Right) which largely improves downstream performance, including retrieval (R1), embedding arithmetic (SIMAT), image captioning (BLUE@4), and classification (ID Acc. and OOD Acc.), and uncertainty calibration (ECE).
the hardest negative pairs, while uniformity (Eq. 3) indicates how uniformly the data is distributed. The greater alignment and uniformity denote the more transferable representation [25, 26].
$$
\text {A l g i n m e n t} := - \mathbb {E} _ {\left(x _ {i}, y _ {i}\right)} \left[ \| f \left(x _ {i}\right) - g \left(y _ {i}\right) \| _ {2} ^ {2} - \min  _ {k \neq i} \| f \left(x _ {i}\right) - g \left(y _ {k}\right) \| _ {2} ^ {2} \right] \tag {2}
$$
$$
\text {U n i f o r m i t y} := - \log \mathbb {E} _ {\left(x _ {i}, y _ {j}\right)} \left[ \exp \left(- 2 \| f \left(x _ {i}\right) - g \left(y _ {j}\right) \| _ {2} ^ {2}\right) \right] \tag {3}
$$
Fig. 2 shows (Left) DOSNES [24] and (Middle) uniformity-alignment of CLIP's embedding space on image-caption datasets after pre-trained (ZS), naively fine-tuned (FT), and fine-tuned with our method (Ours). Pre-trained CLIP embeddings are separated by two subspaces for image and text with wide interspace between them, and even after fine-tuning, this structure remains unchanged. As a result, CLIP and its fine-tuned embedding have poor uniformity and alignment, and this might limit the transferability and robustness of embeddings to downstream data. From this observation, we aim to learn a more transferable and robust multi-modal representation with multi-modal mixup. To validate the transferability and robustness of embeddings, besides evaluating uniformity-alignment, we perform diverse downstream tasks: retrieval, few-/zero-shot classification on in-distribution (ID), out-of-distribution (OOD), classification under modality missing, embedding arithmetic, and image captioning in Section 5. Here, our method produces well-aligned and uniformly distributed embeddings resulting in consistent downstream performance gains (Fig. 2 Right $^2$ ).
4.1 Understanding Geodesic Multi-Modal Mixup
From our finding that CLIP embedding is not robust enough, we improve it by fine-tuning CLIP via contrastive loss with virtual hard negative samples. First, we generate hard negatives by mixing the image and text embeddings via multi-modal Mixup,  $m^2$ -Mix. Note that CLIP's  $L_2$ -normalized embeddings lie on a hypersphere, and the mixed ones are also desirable to lie on that hypersphere. However, original Mixup [37, 41] does not guarantee that mixed data lie on the hypersphere. Therefore, we devise a new type of Mixup, geodesic Mixup (defined as Eq. 4). Geodesic Mixup interpolates two data points on geodesic path, so it ensures that mixed samples lie on the hypersphere.
$$
m _ {\lambda} (\vec {a}, \vec {b}) = \vec {a} \frac {\sin (\lambda \theta)}{\sin (\theta)} + \vec {b} \frac {\sin ((1 - \lambda) \theta)}{\sin (\theta)}, \quad \text {w h e r e} \theta = \cos^ {- 1} (\vec {a} \cdot \vec {b}) \text {a n d} \lambda \sim \operatorname {B e t a} (\alpha , \alpha) \tag {4}
$$
Where  $\lambda$  is handled by a hyperparameter  $\alpha$  in Beta distribution. It is well-known that  $L_{2}$ -normalized embeddings are crucial for metric learning [68, 69] thus adopted by most of the modern contrastive learning methods [14, 15, 19]. Therefore, it is necessary that mixture samples lie on the unit sphere, which is guaranteed by our geodesic Mixup. Comparison between geodesic Mixup and standard Mixup following manual  $L_{2}$ -normalization in Tab. 6. Then, we utilize the generated hard negatives for contrastive loss by replacing the original negatives (See Eq. 5). Here, we only change the denominator term and retain the numerator that compares the similarity between the positive pairs.
$$
C _ {m ^ {2}} (I, T) = \frac {1}{M} \sum_ {i = 1} ^ {M} - \log \frac {\exp \left(I _ {i} \cdot T _ {i} / \tau\right)}{\exp \left(I _ {i} \cdot T _ {i} / \tau\right) + \sum_ {j \neq i} \exp \left(I _ {i} \cdot m _ {\lambda} \left(I _ {j} , T _ {j}\right) / \tau\right)} \tag {5}
$$
$$
\mathcal {L} _ {m ^ {2} \text {- M i x}} = \frac {1}{2} \left(C _ {m ^ {2}} (I, T) + C _ {m ^ {2}} (T, I)\right)
$$
For each method, FT and Ours, we compute scores for each bar by averaging all values of FT and  $m^3$ -Mix in corresponding Tables, except OOD Acc computed by averaging values of WiSE-FT, LP-FT, and MaPLe.
Figure 3: Comparison among contrastive losses. CLIP enforces the pair-wise similarity between matched image-text embedding pairs.  $m^2$ -Mix generates hard negative samples via mixing two heterogeneous embeddings. We additionally propose uni-modal Mixups,  $V$ -Mix (and  $L$ -Mix) and  $VL$ -Mix, that augment the homogenous embeddings in multi-modal contrastive loss.
Because CLIP has two-sided polarized embeddings, the similarity between the original image (or text) embedding and mixed embedding is larger than that between the original image and text embeddings. Therefore,  $m^2$ -Mix generates harder negative samples compared with original negatives. The existence of hard negative samples is known to be important in uni-modal CLs [25, 26], and we validate it under multi-modal CL settings in Section 5. Besides, we provide a theoretical result on hardness guarantee and limiting behavior of  $\mathcal{L}_{m^2\text{-Mix}}$  in the following paragraph.
Theoretical Analysis In Section 3, we observed that naive fine-tuning with standard contrastive loss could not reduce the embedding separation. We speculate this limitation is derived from (1) lack of hard negative samples and (2) vanished learnable  $\tau$  (0.01) in  $\mathcal{L}_{\mathrm{CLIP}}$ . As shown by Wang et al. [26], when  $\tau$  approaches zero, the contrastive loss behaves like triplet loss with zero-margin. This means that if the similarity between positive samples is greater than that of the nearest negative, there are no incentives to further pull or push the embeddings of positive and negative samples. Moreover, due to CLIP's bipartite embedding space, it might lack sufficient hard negative samples to incentivize the models to pursue more alignment and uniformity. Therefore, we argue that hard negatives are necessary for multi-modal CL when there is an embedding space modality gap [27].
Theorem 4.1 (Hardness of  $m^2$ -Mixed samples). Let's assume that two random variables  $x_1$  and  $x_2$  follow the  $M_d(\mu_1, \kappa)$  and  $M_d(\mu_2, \kappa)$ , von Mises-Fisher distribution with mean direction  $\mu_1, \mu_2$  and concentration parameter  $\kappa$  in  $\mathbb{R}^d$ , respectively. Let  $\widetilde{x} = x_1 + x_2$  and  $d = 2$ . Then,  $D_{KL}(p(x_1) || p(\widetilde{x})) \leq D_{KL}(p(x_1) || p(x_2))$  for sufficiently large  $\kappa$ .
Theorem 4.1 shows that KL-divergence between the pair of an original sample and a mixed sample is less (more confused with positive) than that of another original sample (proof in Supp. D). Meanwhile, because the converged  $\tau$  of CLIP is significantly small (i.e., 0.01), it will be reasonable to consider an extreme case: when  $\tau \rightarrow 0^{+}$ . Based on Proposition 4.2, we argue that ones can explicitly enforce uniformity-alignment in multi-modal contrastive learning by equipping  $m^2$ -Mix with  $\mathcal{L}_{\mathrm{CLIP}}$ .
Proposition 4.2 (Limiting behavior of  $\mathcal{L}_{\mathrm{CLIP}}$  with  $\mathcal{L}_{m^2 -\mathrm{Mix}}$ ). For sufficiently large  $M$ , as the temperature of contrastive loss  $\tau \rightarrow 0^{+}$ , the  $\mathcal{L}_{\mathrm{CLIP}}$  and  $\mathcal{L}_{m^2 -\mathrm{Mix}}$  converge to the triplet loss with zero-margin (i.e., corresponding to negative Alignment) and negative Uniformity, respectively. That is:  $\lim_{\tau \to 0^{+}}\mathcal{L}_{\mathrm{CLIP}} + \mathcal{L}_{m^2 -\mathrm{Mix}}\simeq -(Alignment + Uniformity)$
$m^2$ -Mix brings two advantages on multi-modal CL with theoretical grounds. Firstly, by generating sufficiently hard negatives, it incentivizes the model to enforce alignment more strongly whether there exists a modality gap or not. Besides, the uniformity is also explicitly increased as  $\mathcal{L}_{m^2\text{-Mix}}$  asymptotically converges to negative uniformity. Thus,  $\mathcal{L}_{\mathrm{CLIP}}$  with  $m^2$ -Mix induces well-aligned and uniformly distributed embeddings, so it makes the model robustly works on diverse tasks.
4.2 Uni-Modal Mixup for CLIP
$m^2$ -Mix generates hard negatives for CL to align and distribute embeddings uniformly. Meanwhile, Mixup is known to alleviate the overconfidence problem in uni-modal setups [38]. So, we further propose three uni-modal Mixups, Vision-Mix (V-Mix), Language-Mix (L-Mix), and Vision-Language Mix (VL-Mix) to enhance multi-modal CL. Fig. 3 shows the overall structures.
uni-Mix  $V$ -Mix and  $L$ -Mix interpolate the image and text embedding, respectively. To be specific,  $V$ -Mix mixes the embeddings of images in batch and a flipped (reversed) batch with a ratio  $\lambda$ . Then,
$m_{\lambda}(I_i, I_{i'})$  has information from  $i$  and  $i' = M - i$  indexed samples with  $\lambda$  and  $1 - \lambda$  fraction, respectively. Thus, pseudo label for  $(m_{\lambda}(I_i, I_{i'}), T_i)$  pair is  $\lambda$  while, that for  $(m_{\lambda}(I_i, I_{i'}, T_{i'})$  is  $1 - \lambda$ .
$$
C _ {V} (I, T) = \frac {1}{M} \sum_ {i = 1} ^ {M} - \lambda \log \frac {\exp (m _ {\lambda} (I _ {i} , I _ {i ^ {\prime}}) \cdot T _ {i} / \tau)}{\sum_ {j = 1} ^ {M} \exp (I _ {i} \cdot T _ {j} / \tau)} - (1 - \lambda) \log \frac {\exp (m _ {\lambda} (I _ {i} , I _ {i ^ {\prime}}) \cdot T _ {i ^ {\prime}} / \tau)}{\sum_ {j = 1} ^ {M} \exp (I _ {i} \cdot T _ {j} / \tau)}
$$
$L$ -Mix has the same formula with  $V$ -Mix, except that it is applied to text-side. The combination loss term of  $V$ - and  $L$ -Mix is defined as  $\mathcal{L}_{uni-Mix} = \frac{1}{2}(C_V(I,T) + C_V(T,I)) + \frac{1}{2}(C_L(I,T) + C_L(T,I))$
$VL$ -Mix For pair-wise similarity contrast,  $V$ -Mix and  $L$ -Mix only mix the image and text embedding, respectively. We additionally propose  $VL$ -Mix that mixes the image and text embedding simultaneously. Note that  $m^2$ -Mix mixes embeddings of image and text, while  $VL$ -Mix independently mixes them. Both  $m_{\lambda}(I_i, I_{i'})$  and  $m_{\lambda}(T_i, T_{i'})$  has  $i$ -th component and  $i'$ -th component with fraction  $\lambda$  and  $1 - \lambda$  respectively, so the pseudo label for  $(m_{\lambda}(I_i, I_{i'}), m_{\lambda}(T_i, T_{i'}))$  is 1. Here, similarities between negative pairs are retain with that of original negatives likewise uni-Mix.
$$
C _ {V L} (I, T) = \frac {1}{M} \sum_ {i = 1} ^ {M} - \log \frac {\exp (m _ {\lambda} (I _ {i} , I _ {i ^ {\prime}}) \cdot m _ {\lambda} (T _ {i} , T _ {i ^ {\prime}}) / \tau)}{\sum_ {j = 1} ^ {M} \exp (I _ {i} \cdot T _ {j} / \tau)} \mathcal {L} _ {V L \mathrm {- M i x}} = \frac {1}{2} (C _ {V L} (I, T) + C _ {V L} (T, I))
$$
$m^3$ -Mix We name the combination of  $m^2$ -Mix with uni-Mix and  $VL$ -mix as  $m^3$ -Mix, multiple multi-modal Mixup. Complete objective function is denoted as (weights for each term are omitted):
$$
\mathcal {L} _ {m ^ {3} \text {- M i x}} = \mathcal {L} _ {C L I P} + \mathcal {L} _ {m ^ {2} \text {- M i x}} + \mathcal {L} _ {u n i \text {- M i x}} + \mathcal {L} _ {V L \text {- M i x}}
$$