SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Settings Unless otherwise stated, we adopt CLIP ViT-B/32 as our backbone model. We consider the following methods as our baselines in retrieval and embedding arithmetic tasks: zero-shot inference (ZS) of CLIP, embedding shift (ES) [27], naive fine-tuning (FT), and its increased  $\tau$  variants, recent Mixup-augmented uni-modal CL methods  $i$ -Mix [50] and Un-Mix [51]. We put task-specific setups on each section and Sec. A of Supplementary Material (SM). Further details, hyperparameters selection, pseudo code, and additional results are put in Sec. A, B, and C of SM, respectively.
5.1 Generated Samples by  $m^2$ -Mix
(a) Retrieved images
(b) Initial epoch
Figure 4: On Flickr30k, (a) top-3 retrieved images by image embedding (Top) and  $m^2$ -Mixed (Bottom) from a source instance (Left).  $m^2$ -Mix generates an embedding that contains features from both modalities (young girl, climbing, and indoor wall), partly lacking
(c) Last epoch
in the image embedding. (b) and (c) denote cosine similarities between given test instances and its top-1 nearest negative embedding during training epochs.  $m^2$ -Mix makes negatives that are highly similar to given instances.
To understand the properties of generated embedding by  $m^2$ -Mix, we explore the mixed embedding from  $m^2$ -Mix. Due to the non-trivial visualization of the embedding itself, we retrieve images that have the most similar embedding with a mixed one. In Fig. 4 (a), the embedding generated by  $m^2$ -Mix has both features from image and text that lack in original image embedding, e.g., the second and third images from  $m^2$ -Mix have rock climbing and indoor wall represented in the text. Besides, similarity histograms in Fig. 4 (b) and (c), show that  $m^2$ -Mix consistently produces harder negatives than the original non-mixed counterparts from the initial to last training epochs.
5.2 Cross-Modal Retrieval with CLIP
First, we validate our method on image-text retrieval, a representative vision-language task, on Flickr30k [67] and MS COCO [70]. All methods are trained over 9 epochs with Adam optimizer (details in SM). Tab. 1 denotes top-1/-5 recall of retrieval. Our  $m^3$ -Mix increases overall performance, while the standard fine-tuning approaches and Mixup-baselines [50, 51] have limited performance gain. Corresponding to previous works [26, 27], we also found that properly increasing temperature ( $\tau$ ) in contrastive loss is quite beneficial at improving performance for both FT and  $m^3$ -Mix.
Table 1: Image to text (i→t) and text to image retrieval (t→i) retrieval results (top-1/-5 Recall;R1, R5). ZS and FT denote pre-trained and fine-tuned CLIP, respectively.
Flickr30k	MS COCO
i→t	t→i	i→t	t→i
R1	R5	R1	R5	R1	R5	R1	R5
ZS	71.1	90.4	68.5	88.9	31.9	56.9	28.5	53.1
ES [27]	71.8	90.0	68.5	88.9	31.9	56.9	28.7	53.0
FT	81.2	95.4	80.7	95.8	36.7	63.6	36.9	63.9
FT (τ = 0.05)	82.4	95.1	82.1	95.7	40.2	68.2	41.6	69.9
FT (τ = 0.10)	75.7	93.9	78.0	92.9	34.2	62.7	36.7	64.2
i-Mix [50]	72.3	91.7	69.0	91.1	34.0	63.0	34.6	62.2
Un-Mix [51]	78.5	95.4	74.1	91.8	38.8	66.2	33.4	61.0
m3-Mix	82.3	95.9	82.7	96.0	41.0	68.3	39.9	67.9
m3-Mix (τ = 0.05)	82.7	95.7	82.8	95.5	40.4	67.9	42.0	68.8
Table 2: Retrieval with disjoint unimodal models. ZS is a naive combination of two models without joint-tuning.
Flickr30k
i→t	t→i
R1	R5	R1	R5
ZS	0.1	0.4	0.1	0.2
ES [27]	0.1	0.5	0.2	0.2
FT	28.7	61.7	26.7	59.4
FT (τ = 0.05)	31.5	64.2	29.2	61.4
FT (τ = 0.10)	30.0	62.7	30.1	60.6
i-Mix [50]	27.6	60.3	27.1	60.7
Un-Mix [51]	31.5	64.3	29.2	61.2
m3-Mix	31.9	62.6	30.3	61.0
m3-Mix (τ = 0.05)	32.5	64.7	30.4	63.4
Besides, we observed the improved uniformity and alignment by  $m^3$ -Mix (Fig. 2) not only enhances the Recall of retrievals but also contributes to the calibration [71]. The left side of Fig. 5 denotes the reliability diagrams with calibration errors of the text-to-image retrieval R1 score on Flickr30K. While the naively fine-tuned CLIP has poor
Table 3: Calibration on Flickr30k.
Metric	Task	ZS	FT	m3-Mix
ECE (↓)	i → t	1.90	2.26	1.54
t → i	1.88	2.00	1.58
calibration, fine-tuning with  $m^3$ -Mix alleviates the overconfidence issue somewhat and results in a better calibration. This is further confirmed by Tab. 3, in which our  $m^3$ -Mix significantly improves the Expected Calibration Error (ECE) of CLIP. Meanwhile, it is known that the ECE value can be improved by adjusting the temperature  $\tau$ , i.e., temperature scaling [71]. Therefore, we provide the sensitive analysis on varying  $\tau$ . In Fig. 5 right side, our method shows relatively robust ECE under varying  $\tau$ , implying that our multi-modal Mixup-based CL induces the well-calibrated multi-modal model, which is crucial for reliable AI applications.
Figure 5: Reliability diagram (left) and ECE under varying temperature  $\tau$  (right) on Flickr30K image-to-text retrieval. Our method shows (1) a better reliability diagram (close to  $y = x$ ), (2) achieves a lower minimum ECE value, and (3) more stable across varying  $\tau$  than naive fine-tuning. Thus, representation learning by  $m^3$ -Mix robustly induces a well-calibrated multi-modal model.
5.3 Cross-Modal Retrieval with Uni-Modal Pre-Trained Models
Sometimes, the high-quality annotations for multi-modal datasets are expensive, and there are cases when plenty of paired multi-modal data is unavailable. Then, it is crucial to exploit the uni-modal pre-trained models for learning multi-modal embedding space [72]. To this end, we validate our  $m^3$ -Mix on the fine-tuning of disjointly pre-trained uni-modal models. Specifically, we jointly fine-tune the pre-trained BERT [73] and ResNet-50 [74] with a contrastive loss on Flickr30k (in Table 2). Among candidates,  $m^3$ -Mix with higher  $\tau$  consistently achieves the highest performance so that it can be adopted as an effective joint tuning method for independently pre-trained uni-modal models.
5.4 Few-Shot Adaptation and Robustness on Distribution Shift
Table 4: Few-shot adaptation under general setting.
Method	Dataset
Pets	SVHN	CLEVR	Avg.
ZS	87.49	13.63	20.70	40.61
FT	89.37	45.00	53.49	62.62
FT w/ V-Mix	89.45	44.61	53.93	62.66
FT w/ L-Mix	89.43	48.42	53.91	63.92
FT w/ VL-Mix	89.56	45.22	53.75	62.84
FT w/ m2-Mix	90.05	46.24	53.60	63.29
m3-Mix	90.16	54.84	53.85	66.28
m3-Mix (τ = 0.05)	90.49	60.90	53.95	68.45
WiSE-FT [10]	91.80	35.04	41.93	56.25
WiSE-FT w/ m3-Mix	92.51	58.55	47.11	66.06
LP-FT [11]	89.92	44.91	53.62	62.82
LP-FT w/ m3-Mix	91.03	64.24	55.20	70.16
MaPLe [64]	90.87	47.62	43.05	60.51
MaPLe w/ m3-Mix	91.14	52.72	45.20	63.02
Table 5: Few-shot (ImageNet; IN) and zero-shot evaluation under distribution shift (-V2, -A, -R, -S).
Method	Dataset
IN	IN-V2	IN-A	IN-R	IN-S	Avg.
ZS	62.06	54.80	29.63	66.02	40.82	50.67
FT	65.44	55.35	20.07	58.16	34.50	46.70
FT w/ V-Mix	66.00	56.19	20.85	60.50	34.97	47.70
FT w/ L-Mix	65.96	55.95	20.57	60.54	35.25	47.65
FT w/ VL-Mix	66.24	56.70	21.36	61.07	35.11	48.10
FT w/ m2-Mix	67.04	57.39	20.05	59.28	35.31	47.81
m3-Mix	67.08	57.55	20.80	60.96	35.86	48.45
m3-Mix (τ = 0.05)	68.40	58.51	22.17	62.28	37.62	49.80
WiSE-FT [10]	69.00	59.66	28.01	64.84	41.05	52.51
WiSE-FT w/ m3-Mix	69.65	60.71	29.16	66.75	42.19	53.69
LP-FT [11]	68.22	58.40	25.57	63.36	38.04	50.72
LP-FT w/ m3-Mix	68.62	59.17	25.85	65.14	38.78	51.51
MaPLe [64]	65.59	58.44	32.49	68.13	42.53	53.44
MaPLe w/ m3-Mix	65.76	58.16	32.52	68.20	42.67	53.46
Next, we evaluate our methods on few-shot image classification under general (in Tab. 4) and distribution shift settings (in Tab. 5 and 6). We consider OxfordPets [75], SVHN [76], and CLEVR [77] for the general setting $^3$  and ImageNet-1k, ImageNetV2 [78], ImageNet-A [79], ImageNet-R [80], and ImageNet-Sketch [81] for distribution shift setting. Unlike MS COCO and Flickr30K, These datasets provide class name labels only and do not have captions corresponding to each image. To make CL methods amendable for this setting, we adopt a common prompt 'a photo ofclassname' that wraps the class name with a short context and use this as captions of images. Following [82, 64], we perform the tasks under a few-shot evaluation protocol: 16-shot training samples per class and inference on the entire test set. As baselines, we first consider zero-shot CLIP (ZS) and construct the contrastive loss adoption of vanilla fine-tuning (FT). Then, we showcase our methods with exhaustive ablation  $(V-,-,L-,-,VL-,-,$  and  $m^2$ -Mix) as well as our complete objective  $m^3$ -Mix with its high-temperature variant. To further compare our approach with state-of-the-art (SOTA) fine-tuning methods, we consider MaPLe [64] that optimizes the continuous prompts inside the text and image encoders of CLIP, and the contrastive loss extended version of uni-modal fine-tune methods: LP-FT [11] and WiSE-FT [10].
In both general and distribution shift settings,  $m^2$ -Mix and unimodal Mixups ( $V-$ ,  $L-$ ,  $VL-$ ) contribute to boost the few-/zero-shot classification performance. After integrating them,  $m^3$ -Mix and its high-temperature variant give significant performance improvement, implying  $m^3$ -Mix is an effective fine-tuning method that covers challenge generalization setup. Moreover, when  $m^3$ -Mix combined with SOTA fine-tuning methods [10, 11, 64],
it consistently brings performance. Therefore,  $m^3$ -Mix is a flexible plug-in method that can be collaborated with many algorithms. Besides, in Tab. 6 (mean Acc. of ImageNet variants), we show that geodesic Mixup achieves superior results than linear Mixup with manual  $L_2$ -normalization. Thus, based on its analytic property, geodesic Mixup is more suitable than linear Mixup under frameworks that learn representation on a hypersphere, as in modern CLs.
Table 6: Ablation study on Mixup.
Temperature (τ)	m3-Mix type
linear	geodesic
0.01	48.36	48.45
0.05	48.48	49.80
0.10	45.20	46.41
5.5 Robustness on Modality Missing
In this section, we study whether  $m^2$ -Mix can help the multi-modal representation learning for video recognition (CMU-MOSEI [83]) under modality missing. Recently, Geometric Multi-modal Contrastive Learning (GMC) [84] achieved competitive results on CMU-MOSEI.
However, GMC only considers the uni-to-joint modal relationship [84], while our  $m^2$ -Mix explicitly generates the mixture of bi-modal semantics so that it can additionally consider the bi-to-joint modal relation. From this, we hypothesize that  $m^2$ -Mix can further improve the robustness and informativeness of the representation. For evaluation, we add the  $m^2$ -Mix on top of GMC.
3Results of CLIP ViT-B/16 on other transfer learning benchmark datasets are provided in Sec. C of SM.
Different from the CLIP fine-tuning cases, we use the multimodal mixed representation as both positive and negative pairs with the target joint representation because our goal in this task is to align the embedding between the joint and other modalities. As shown in Tab. 7,  $m^2$ -Mix coherently improves the performance of GMC in terms of accuracy, alignment, and uniformity. While GMC strongly aligns the embedding of partial and joint modality, the alignment is further enhanced by the aid of  $m^2$ -Mix (also confirmed in Fig. 6), which results in superior performance when only partial information is given in test-time (modality missing).
These results justify the use of  $m^2$ -Mix for robust learning under modality-incomplete scenarios.
Figure 6: t-SNE [85] on CMUMOSEI with partial information. Each color denotes embeddings of partial and joint modality.
Table 7: Accuracy (acc.)  $(\uparrow)$ , alignment (align.)  $(\uparrow)$ , and uniformity (unif.)  $(\uparrow)$  of multi-modal learning methods on CMU-MOSEI under complete and partial modalities. Averaged performance of five runs.
Test-time Observed Modalities
Full (T+V+A)	T	V	A	T+V	T+A	V+A
acc.	unif.	acc.	align.	unif.	acc.	align.	unif.	acc.	align.	unif.	acc.	align.	unif.	acc.	align.	unif.	acc.	align.	unif.
MulT [86]	80.5	0.99	60.0	-	1.03	53.9	-	2.07	52.7	-	0.62	57.8	-	1.27	58.8	-	0.77	54.6	-	1.36
GMC [84]	80.1	3.06	78.5	0.20	3.03	64.7	0.17	3.01	66.0	0.09	3.03	77.0	0.07	2.94	77.4	0.08	3.00	67.3	0.05	2.98
GMC+m2-Mix	80.5	3.18	78.9	0.23	3.17	64.2	0.19	3.15	66.2	0.12	3.15	77.8	0.08	3.08	77.9	0.09	3.08	67.4	0.06	3.10
5.6 Multi-Modal Embedding Arithmetic
We expect that well-learned multi-modal embedding represents the structural relationship between instances like word vectors [87]. For validation, we evaluate the learned embeddings on SIMAT [88]. SIMAT evaluates text-driven image representation by retrieving a new image with the highest similarity to the latent vector  $x$ , which is transformed by text delta vectors when we change the word in an original text, i.e., formulated as:  $x = I_{\text{original}} + \lambda \cdot (T_{\text{new}} - T_{\text{original}})$ . Here,  $I$  and  $T$  are image and text embedding vectors, and  $\lambda$  is a hyper-parameter about the strength of transformation. Table 8 presents the quantitative scores across methods after fine-tuning on Flickr30k and MS COCO, and evaluated on SIMAT. Learned representation from  $m^3$ -Mix shows stable scores on both a multi-modal model and jointly fine-tuned uni-modal models, which is further confirmed by qualitative experiments (Fig. 7). These support that  $m^3$ -Mix can be adopted as a delicate fine-tuner when embedding geometric structure and arithmetic property should be considered, e.g., controllable generation [89].
Table 8:  ${m}^{3}$  -Mix shows the stable SIMAT Score (↑) on CLIP and joint fine-tuning of unimodal pre-trained models.
CLIP (MS COCO)	CLIP (Flickr30k)	BERT + RN50 (Flickr30k)
ZS	34.5	34.4	6.1
ES [27]	34.6	34.5	6.2
FT	42.3	40.8	15.4
i-Mix [50]	37.3	40.0	12.9
Un-Mix [51]	42.9	38.5	15.8
m3-Mix	44.4	38.9	19.0
Figure 7: Embeddings of  $m^3$ -Mix on SIMAT source-target texts (reds) and images (source and top-1 retrieved). Black texts correspond to top-3 retrieved images from a source image-caption.
5.7 Multi-Modal Mixup on a State-of-the-Art Vision-Language Model
In this section, we investigate whether our multi-modal Mixup can also be beneficial for improving other recent large-scale vision-language models beyond CLIP. For this, we adopt Contrastive Captioner (CoCa) [90] ViT-L/14 configuration that pre-trained on LAION-2B [5] from 0penCLIP library as our target backbone model. We consider three learning objectives for CoCa fine-tuning: (1) autoregressive captioning loss (Cap), (2) contrastive loss and captioning loss  $(\mathrm{CL} + \mathrm{Cap})$ , and (3) contrastive loss,  $\mathcal{L}_{m^2 -\mathrm{Mix}}$ , and captioning loss (CL w/  $\mathcal{L}_{m^2 -\mathrm{Mix}} + \mathrm{Cap}$ ). For all three methods, we train the model on MS COCO over one epoch with 0penCLIP-provided hyperparameter configuration. After that, we evaluate each model for image captioning (Tab. 9) on MS COCO and zero-shot and fine-tuned cross-modal retrieval (Tab. 10) on Flickr30K and MS COCO.
Table 9: Image captioning results on MS COCO with CoCa ViT-L/14 model. We fine-tune the CoCa on MS COCO for 1 epoch with three different learning objectives and evaluate them in terms of five conventional metrics.  $m^2$ -Mix achieves performance gain on all the five metrics.
Method	Metrics
BLEU@4	METEOR	ROUGE-L	CIDEr	SPICE
ZS	7.2	12.4	26.3	35.2	9.3
Cap	36.0	29.4	57.3	125.1	23.1
CL + Cap	35.7	29.3	57.1	124.9	23.0
CL w/ Lm2-Mix + Cap	36.3	29.5	57.5	125.6	23.2
In Tab. 9, while a combination of vanilla contrastive loss with captioning loss underperforms the captioning-loss-only training,  $m^2$ -Mix-assisted contrastive learning further increases the performance on image captioning in terms of five conventional metrics. Besides, Tab. 10 shows that  $m^2$ -Mix generally improves the retrieval recalls of CoCa on zero-shot and fine-tuned settings. This implies that representation learning with our multi-modal Mixup is beneficial to generative tasks as well as discriminative tasks. The consistent improvement shown in the image captioning task is accorded with observations from other recent works [52, 53] that reveal the effectiveness of cross-modal Mixup on generative tasks by increasing cross-modal alignment.
Table 10: Cross-modal retrieval top-1/5 recalls on MS COCO (fine-tuned) and Flickr30k (zero-shot transfer).  $m^2$ -Mix generally enhances image-text retrieval of a SOTA vision-language model, CoCa.
Method	(Zero-shot) Flickr30k
i → t (R1)	i → t (R5)	t → i (R1)	t → i (R5)
Cap	90.4	98.5	78.5	94.4
CL + Cap	92.4	99.1	79.2	94.9
CL w/ Lm2-Mix + Cap	92.8	99.0	79.5	95.1
Method	(Fine-tuned)
i → t (R1)	i → t (R5)	t → i (R1)	t → i (R5)
Cap	68.5	87.9	53.2	77.8
CL + Cap	73.9	91.2	56.3	80.4
CL w/ Lm2-Mix + Cap	73.9	91.0	56.5	80.6
This paper analyzes the representation learned by a multi-modal contrastive learner. We found that CLIP has separated text-versus-image embedding space with poor uniformity-alignment. These polarized embeddings with huge unexploited space may limit the transferability and robustness of representation on downstream tasks. From our findings, we propose Geodesic Multi-Modal Mixup that generates hard negatives for robust contrastive learning by mixing two heterogeneous embeddings. Theoretically, we validate that our method produces hardness-guaranteed samples and has desirable asymptotic behavior to induce better generalizable representation. Empirically, the proposed method effectively improves performances on diverse tasks and perspectives: retrieval, calibration, few-shot classification under distribution shift, embedding arithmetic, and image captioning.
Though the increased uniformity and alignment of multi-modal representation largely empowers the model to adapt robustly to a diverse range of tasks, we found that reckless uplift of them is harmful in some cases of retrieval (modest increment of uniformity-alignment was better than huge increment of them). Thus, more research on the reliable evaluation of multi-modal representation should be pursued in the era of foundation models.
Acknowledgement This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2021R1F1A1060117 and No.2022R1A4A3033874), and also supported by a grant (22183MFDS431) from Ministry of Food and Drug Safety in 2023.