SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zehan Wang $^{1}$  Yang Zhao $^{2}$  Xize Cheng $^{1}$  Haifeng Huang $^{1}$  Jiageng Liu $^{1}$  Li Tang $^{1}$  Linjun Li $^{1}$  Yongqi Wang $^{1}$  Aoxiong Yin $^{1}$  Ziang Zhang $^{1}$  Zhou Zhao $^{1,3*}$
<sup>1</sup>Zhejiang University <sup>2</sup>ByteDance <sup>3</sup>Shanghai AI Laboratory {wangzehan01}@zju.edu.cn
Multi-modal Contrastive Representation (MCR) learning aims to encode different modalities into a semantically aligned shared space. This paradigm shows remarkable generalization ability on numerous downstream tasks across various modalities. However, the reliance on massive high-quality data pairs limits its further development on more modalities. This paper proposes a novel training-efficient method for learning MCR without paired data called Connecting Multi-modal Contrastive Representations (C-MCR). Specifically, given two existing MCRs pre-trained on  $(\mathcal{A},\mathcal{B})$  and  $(\mathcal{B},\mathcal{C})$  modality pairs, we project them to a new space and use the data from the overlapping modality  $\mathcal{B}$  to aligning the two MCRs in the new space. Meanwhile, since the modality pairs  $(\mathcal{A},\mathcal{B})$  and  $(\mathcal{B},\mathcal{C})$  are already aligned within each MCR, the connection learned by overlapping modality can also be transferred to non-overlapping modality pair  $(\mathcal{A},\mathcal{C})$ . To unleash the potential of C-MCR, we further introduce a semantic-enhanced inter- and intra-MCR connection method. We first enhance the semantic consistency and completion of embeddings across different modalities for more robust alignment. Then we utilize the inter-MCR alignment to establish the connection, and employ the intra-MCR alignment to better maintain the connection for inputs from non-overlapping modalities. To demonstrate the effectiveness of C-MCR, we take the field of audio-visual and 3D-language learning as examples. Specifically, we connect CLIP and CLAP via texts to derive audio-visual representations, and integrate CLIP and ULIP via images for 3D-language representations. Remarkably, without using any paired data, C-MCR for audio-visual achieves state-of-the-art performance on audio-image retrieval, audio-visual source localization, and counterfactual audio-image recognition tasks. Furthermore, C-MCR for 3D-language also attains advanced zero-shot 3D point cloud classification accuracy on ModelNet40. Our project page is available at https://c-mcr.github.io/C-MCR/
Multi-modal Contrastive Representation (MCR) learning aims to map inputs from different modalities to a shared representation space. With the impressive generalization performance of vision-language contrastive pre-training models [1, 2, 3, 4] demonstrated on various downstream tasks [5, 6, 7, 8, 9, 10], learning MCR spaces between multiple modalities has become a promising area of research, attracting increasing attention [11, 12, 13, 14, 15].
However, the generalization ability of MCR primarily benefits from the accessibility of massive data pairs from the web. For modalities where obtaining semantically matching data pairs is significantly more costly, the representations directly learned from limited data pairs are unreliable. On the other
hand, these modality pairs with little direct paired data often have a large number of paired data with the same intermediate modality. For example, although audio-visual data are often vague, paired data of audio-language and language-image are sufficient and semantically explicit. Similarly, while 3D point-language pairs are rare, 3D point-image and image-language data are extensive.
Consider that there are already many MCRs between modalities with sufficient paired data. In this paper, we propose Connecting Multi-modal Contrastive Representations (C-MCR), a novel training-efficient and paired-data-free MCR learning method that extends the learned alignment knowledge in existing MCRs to more modalities. With regard to the overlapping modality, its representations in two MCRs are just different data views sharing the same inherent semantics. So we can take them as positive pairs to connect different MCRs. As modalities within each MCR are semantically aligned, the connections built from overlapping modalities can also be applied to non-overlapping modalities. The advantages of our C-MCR are two-fold: (1) Flexible. C-MCR enables MCR learning on modalities with limited paired data. More importantly, C-MCR treats each learned MCR space as a node and the overlapping modalities between different MCRs as links. Connecting the various isolated MCRs greatly extends the obtained multi-modal alignment knowledge, and discovers generalized contrastive representations of broader modalities. (2) Training-Efficient. Since C-MCR simply reprojects the learned representations into a new space, only two simple projectors are learnable during training. The training parameters and costs for connecting existing MCRs are very small.
However, two factors impede the acquisition of a robust and transferable connection: Firstly, embeddings in MCR spaces are incapable of comprehensively reflecting all the semantic information of the input, and this loss of meaning would be inherited and amplified, thereby compromising the robustness of the connection. Secondly, as discussed in [16], MCR spaces exhibit a modality gap phenomenon, i.e., the embeddings of different modalities are located in two completely separate regions in each MCR space. This poses a challenge for maintaining the connection based on overlapping modality while facing inputs from non-overlapping modalities.
Considering the above challenges, we propose a semantic-enhanced inter- and intra-MCR connection method. During training, the copious amounts of easily accessible unpaired unimodal data are first encoded into embeddings in two MCR spaces. We inject Gaussian noise into all the embeddings to mitigate the semantic bias, enhance the semantic completeness, and improve robustness. For directly quantifying the modality gap and the relationship between non-overlapping modalities, we exploit the inherent multi-modal alignment in MCR spaces to cluster semantic consistent embeddings and bridge different modalities. With the above strategies, we align the semantic-enhanced embeddings across different MCR spaces in a contrastive manner to establish the connection. To preserve the connection for inputs from non-overlap modalities, we realign the semantic similar embeddings across modalities within each MCR space to alleviate the modality gap.
(1) We propose Connecting Multi-modal Contrastive Representations (C-MCR), a novel paired-data-free and training-efficient method for MCR learning. By connecting existing MCR spaces with simple projectors, we can mine the multi-modal alignment knowledge in existing MCR space, and extend MCRs on more modalities that lack large-scale high-quality data pairs.
(2) We further propose a semantic-enhanced inter- and intra-MCR connection method to unleash our C-MCR. This approach establishes a transferable connection between two MCR spaces via overlapping modality and maintains it for non-overlapping modalities.
(3) To demonstrate the effectiveness of C-MCR, we connect the CLIP and CLAP through texts to acquire audio-visual representations, and interage CLIP and 3D-image MCR space (ULIP) via images for 3D-language representations. Remarkably, without requiring any pair data or fine-tuning, C-MCR for audio-visual achieves state-of-the-art performance on six datasets across three downstream audio-visual tasks. Furthermore, C-MCR for 3D-language also attains advanced zero-shot 3D point cloud classification accuracy on ModelNet40.
Multi-modal Contrastive Representation learning. Multi-modal contrastive representation focuses on learning separate unimodal encoders for different modalities, which can map inputs from different modalities into a shared representation space. These models are pre-trained on large-scale paired data
Figure 1: The pipeline of connecting CLIP and CLAP using our C-MCR. During training, we take text as input and encode it with frozen CLAP and CLIP text encoders, respectively. Audio(image) memory is generated by encoding lots of unimodal audio(image) data by pre-trained audio(image) encoder. Semantic enhancement enriches the semantic consistency and completion of embeddings. Then two projectors map embeddings to a new shared representation space where inter- and intra-MCR alignment establishes and maintains a stable connection between CLAP and CLIP. During inference, the audio and image are inputted to the corresponding encoder and projector.
using a contrastive loss. Recent vision-language contrastive pre-training models, such as CLIP [1] and ALIGN [2], demonstrate impressive zero-shot retrieval and classification performance and remarkable generalization capability across diverse downstream tasks [5, 6, 7, 8, 9, 10]. Inspired by the success of vision-language models, contrastive representation learning across more modalities has garnered increasing attention. CLAP [12, 11] construct a contrastive language-audio pre-training model by collecting large-scale audio-text pairs from diverse data sources. ULIP [17, 18] collect and generate 3D-image-text triplet data via 3D rendering and image captioning, and learn an extra 3D encoder for existing vision-language space. AudioCLIP [19] and WAV2CLIP [20] leverage the pre-trained CLIP image encoder and acquire audio-visual representations by training on audio-image pairs from AudioSet [21] and VGGSound [22], respectively. For certain modality pairs, such as audio-visual and 3D-language, the pre-training model's generalization capability is restricted by ambiguous or limited paired data. Our proposed method introduces a novel method for better contrastive representation learning on these modalities.
Audio-Visual Learning. Audio-visual learning [23] aims to exploit the relationship between audio and visual modalities, which is an essential part of intelligent multi-modal perception research. Previous methods primarily focus on learning specific audio-visual downstream tasks (such as retrieval [24, 25, 26], localization [27, 28, 29, 30, 31, 32, 33, 34, 35, 36], or generation [37, 38, 39, 40, 41, 42, 43, 44]) within limited domains, based on the manually cleaned small-scale datasets. Recently, several large-scale audio-image datasets [21, 22] collected from the web are proposed. However, these datasets contain many noisy image-audio pairs due to the ambiguous nature of both images and audio and the presence of non-visible sounds and silent objects in videos. Consequently, the generalization ability of audio-visual contrastive representation [19, 20] learned from these datasets is limited. Our C-MCR reduces the need for larger-scale high-quality data pairs. By extending the knowledge in CLIP and CLAP models, we acquire powerful audio-visual contrastive representations that exhibit powerful generalization capabilities across various downstream tasks.
3D-language Learning. 3D vision is an important way for robots to perceive the rich semantic and spatial information of the real world. The 3D-language learning including recognition [45, 46, 47, 48], localization [49, 50, 51, 52, 53], question-answer [54, 55] and general conversation [56, 57], have attracted increasing attentions. 3D-language contrastive representations are vital for the further development of 3D-language learning. However, due to the scarcity of 3D-language paired data, the development of 3D-language representation is limited. Recent ULIP [17, 18] focus on generating 3D-image-text triplet data, but they are still limited by the relatively low quality of training datasets. C-MCR gets rid of the dependence on 3D-language pairing data, and instead connects the reliable 3D-visual representation of ULIP and the visual-language representation of CLIP via images to obtain a more robust 3D-language contrastive representation.
In this section, we take connecting CLIP and CLAP for audio-visual, as an example to introduce C-MCR. As depicted in Figure 1 (a), we utilize two projectors to connect CLIP and CLAP through texts. Before delving into our method, we first introduce the mathematical formulations and revisit the multi-modal contrastive learning in Section 3.1. Then we discuss our semantic enhancement approach for robust representation alignment in Section 3.2. This is followed by the inter-MCR alignment to establish the connection between CLIP and CLAP in Section 3.3, and the intra-MCR alignment to ensure the connection can be maintained for image-audio inputs in Section 3.4.
Problem formulation. For text inputs, the embeddings obtained by CLIP and CLAP encoder can be denoted as  $\mathbf{t}^I\in \mathbb{R}^c$  and  $\mathbf{t}^A\in \mathbb{R}^d$  respectively. Our C-MCR method aims to leverage the inherent consistency between  $\mathbf{t}^I$  and  $\mathbf{t}^A$  and the multi-modal alignment in MCR to learn two projectors  $f_{1}(\cdot)$  and  $f_{2}(\cdot)$  that map the representations from CLIP and CLAP to a new shared representation space. The connection between CLIP and CLAP, learned from texts, can effectively accommodate audio and image inputs.
Multi-modal Contrastive Learning. Given  $N$  paired instances from two different modalities, we map the  $i$ -th pair to L2-normalized embeddings  $\mathbf{x}_i$  and  $\mathbf{z}_i$  via two encoders. Multi-modal contrastive learning aims to maximize the cosine similarity between  $\mathbf{x}_i$  and  $\mathbf{z}_i$  and minimize the cosine similarity between  $\mathbf{x}_i$  and  $\mathbf{z}_j$  where  $i \neq j$ . The contrastive loss can be formulated as:
$$
L _ {c o n s} = - \frac {1}{2} \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ \log \frac {\exp (\sin (\mathbf {x} _ {i} , \mathbf {z} _ {i}) / \tau)}{\sum_ {j = 1} ^ {N} \exp (\sin (\mathbf {x} _ {i} , \mathbf {z} _ {j}) / \tau)} + \log \frac {\exp (\sin (\mathbf {z} _ {i} , \mathbf {x} _ {i}) / \tau)}{\sum_ {j = 1} ^ {N} \exp (\sin (\mathbf {z} _ {i} , \mathbf {x} _ {j}) / \tau)} \right] (1)
$$
where  $\tau$  is the temperature parameter and the  $\mathrm{sim}(\cdot ,\cdot)$  is the operator for cosine distance. The contrastive loss is based on multi-modal data pairs, and the generalization of the learned representation relies on the scale and quality of the data pairs. To extend contrastive representation learning on more modalities, we propose using overlapping modalities to connect two learned MCR spaces and extend the learned multi-modal alignment knowledge to non-overlapping modalities.
3.2 Semantic Enhancement
To achieve more robust and comprehensive alignment, we enhance the semantics from two perspectives: inter-modality semantic consistency and intra-modality semantic completion.
Inter-modality Semantic Consistency. CLIP and CLAP have already learned shared image-text and audio-text representations. To better quantify the modality gap in the MCR space and directly explore the correlation between audio and image, we first utilize the inherent modality alignment properties of CLIP and CLAP to generate semantically consistent embeddings across modalities. Specifically, we encode massive unpaired images and audio using the CLIP image encoder and CLAP audio encoder, respectively. All the obtained image embeddings are served as image memory  $\mathbf{V} = \{\mathbf{v}_1,\mathbf{v}_2,\dots,\mathbf{v}_N\}$  and the audio embeddings are audio memory  $\mathbf{A} = \{\mathbf{a}_1,\mathbf{a}_2,\dots,\mathbf{a}_M\}$ , where  $N,M$  indicate the number of images and audios. Considering  $i$ -th text embeddings  $\mathbf{t}_i^I$  and  $\mathbf{t}_i^A$ , we can generate image embedding  $\mathbf{v}_i^I$  and audio embedding  $\mathbf{a}_i^A$  that are semantically similar to  $i$ -th text.
$$
\mathbf {v} _ {i} ^ {I} = \sum_ {k = 1} ^ {N} \frac {\exp (\sin (\mathbf {t} _ {i} ^ {I} , \mathbf {v} _ {k}) / \tau_ {1})}{\sum_ {j = 1} ^ {N} \exp (\sin (\mathbf {t} _ {i} ^ {I} , \mathbf {v} _ {j}) / \tau_ {1})} * \mathbf {v} _ {k}; \quad \mathbf {a} _ {i} ^ {A} = \sum_ {k = 1} ^ {M} \frac {\exp (\sin (\mathbf {t} _ {i} ^ {A} , \mathbf {a} _ {k}) / \tau_ {1})}{\sum_ {j = 1} ^ {M} \exp (\sin (\mathbf {t} _ {i} ^ {A} , \mathbf {a} _ {j}) / \tau_ {1})} * \mathbf {a} _ {k} \quad (2)
$$
The  $\tau_{1}$  is the temperature hyperparameter. By dynamically absorbing information from memories based on semantic similarity to the text embeddings  $\mathbf{t}_i^I$  and  $\mathbf{t}_i^A$ , we can generate more diverse and accurate semantically-consistent embeddings  $\mathbf{v}_i^I$  and  $\mathbf{a}_i^A$ .
Intra-modality Semantic Completion. The semantics in the original input data are often complex, and some information is inevitably lost when encoding it into the MCR space. When connecting and aligning existing representation spaces, this loss and bias of meaning will be inherited and amplified, affecting the robustness of alignment. To enhance the semantic completeness of each embedding,
we propose to serve Gaussian noise as an information augmentation method. Specifically, we add zero-mean Gaussian noises into the embeddings and re-normalize them to the unit hypersphere:
$$
\begin{array}{l} \tilde {\mathbf {t}} ^ {I} = \text {N o r m a l i z e} (\mathbf {t} ^ {I} + \boldsymbol {\theta} _ {1}); \\ \tilde {\mathbf {t}} ^ {4} = 3 2, \quad \mathbf {t} ^ {5} = (4, 0) \end{array} \quad \begin{array}{l} \tilde {\mathbf {v}} ^ {I} = \text {N o r m a l i z e} (\mathbf {v} ^ {I} + \boldsymbol {\theta} _ {2}) \\ \tilde {\mathbf {t}} ^ {4} = 3 2, \quad \mathbf {t} ^ {5} = (4, 0) \end{array} \tag {3}
$$
$$
\tilde {\mathbf {t}} ^ {A} = \operatorname {N o r m a l i z e} \left(\mathbf {t} ^ {A} + \boldsymbol {\theta} _ {3}\right); \quad \tilde {\mathbf {a}} ^ {A} = \operatorname {N o r m a l i z e} \left(\mathbf {a} ^ {A} + \boldsymbol {\theta} _ {4}\right)
$$
where noise items  $\pmb{\theta}_1, \pmb{\theta}_2 \in \mathbb{R}^c$  and  $\pmb{\theta}_3, \pmb{\theta}_4 \in \mathbb{R}^d$  are sampled from zero-mean gaussian distribution with variance  $\sigma^2$ , and they are not learnable.
Since the MCRs are L2 normalized, all embeddings are distributed on a unit sphere. As illustrated in Figure 1 (c), each embedding can be viewed as a point on the unit sphere's surface. The addition of Gaussian noise can transform the point into a small sphere, and re-normalizing projects the small sphere onto a circle on the surface of the unit sphere. Hence, aligning two embeddings with noise forces the model to acquire the ability to align all the embeddings within the two circles. In the MCR space, the closer two embeddings are to each other, the more similar their semantics are. Embeddings within the same circle share similar general semantics, and the semantics represented by the circle are more comprehensive and robust than the original embedding.
3.3 Inter-MCR Alignment
To establish the connection between two MCRs, we project the semantic-enhanced embeddings from CLIP and CLAP space to a new shared space via two learnable projectors  $f_{1}(\cdot)$  and  $f_{2}(\cdot)$ , respectively.
$$
\hat {\mathbf {t}} ^ {I} = f _ {1} \left(\tilde {\mathbf {t}} ^ {I}\right); \quad \hat {\mathbf {v}} ^ {I} = f _ {1} \left(\tilde {\mathbf {v}} ^ {I}\right); \quad \hat {\mathbf {t}} ^ {A} = f _ {2} \left(\tilde {\mathbf {t}} ^ {A}\right); \quad \hat {\mathbf {a}} ^ {A} = f _ {2} \left(\tilde {\mathbf {a}} ^ {A}\right) \tag {4}
$$
In the newly projected space, our objective is to ensure that embeddings with similar semantics from different MCR spaces are in close proximity to each other. The  $(\mathbf{t}_i^I,\mathbf{t}_i^A)$  from the same text is naturally semantic consistent, and it can be considered as a ground-truth pair label. Besides, there is pseudo consistency in  $(\mathbf{v}_i^I,\mathbf{t}_i^I)$  and  $(\mathbf{a}_i^A,\mathbf{t}_i^A)$  due to the multi-modal alignment properties in CLIP and CLAP. Thus the  $(\hat{\mathbf{v}}^I,\hat{\mathbf{a}}^A)$  derived from  $(\mathbf{t}_i^I,\mathbf{t}_i^A)$  can be viewed as a pseudo pair label. For a robust and stable connection of the two MCR, we propose to align both  $(\hat{\mathbf{t}}^I,\hat{\mathbf{t}}^A)$  and  $(\hat{\mathbf{v}}^I,\hat{\mathbf{a}}^A)$ . The text-text contrastive loss  $L_{ttc}$  and audio-visual contrastive loss  $L_{avg}$  are defined as:
$$
L _ {t t c} = - \frac {1}{2} \frac {1}{B} \sum_ {i = 1} ^ {B} \left[ \log \frac {\exp \left(\sin \left(\hat {\mathbf {t}} _ {i} ^ {I} , \hat {\mathbf {t}} _ {i} ^ {A}\right) / \tau_ {2}\right)}{\sum_ {j = 1} ^ {B} \exp \left(\sin \left(\hat {\mathbf {t}} _ {i} ^ {I} , \hat {\mathbf {t}} _ {j} ^ {A}\right) / \tau_ {2}\right)} + \log \frac {\exp \left(\sin \left(\hat {\mathbf {t}} _ {i} ^ {A} , \hat {\mathbf {t}} _ {i} ^ {I}\right) / \tau_ {2}\right)}{\sum_ {j = 1} ^ {B} \exp \left(\sin \left(\hat {\mathbf {t}} _ {i} ^ {A} , \hat {\mathbf {t}} _ {j} ^ {I}\right) / \tau_ {2}\right)} \right] \tag {5}
$$
$$
L _ {a v c} = - \frac {1}{2} \frac {1}{B} \sum_ {i = 1} ^ {B} \left[ \log \frac {\exp (\sin (\hat {\mathbf {v}} _ {i} ^ {I} , \hat {\mathbf {a}} _ {i} ^ {A}) / \tau_ {3})}{\sum_ {j = 1} ^ {B} \exp (\sin (\hat {\mathbf {v}} _ {i} ^ {I} , \hat {\mathbf {a}} _ {j} ^ {A}) / \tau_ {3})} + \log \frac {\exp (\sin (\hat {\mathbf {a}} _ {i} ^ {A} , \hat {\mathbf {v}} _ {i} ^ {I}) / \tau_ {3})}{\sum_ {j = 1} ^ {B} \exp (\sin (\hat {\mathbf {a}} _ {i} ^ {A} , \hat {\mathbf {v}} _ {j} ^ {I}) / \tau_ {3})} \right] \tag {6}
$$
$B$  corresponds to the batch, and  $\tau_{2},\tau_{3}$  are the temperature hyperparameters. The inter-MCR alignment loss  $L_{inter}$  is the combination of the two contrastive losses:
$$
L _ {i n t e r} = L _ {t t c} + L _ {a v c} \tag {7}
$$
The  $L_{ttc}$  and  $L_{avc}$  are complementary to each other. The semantics between  $(\mathbf{t}^I,\mathbf{t}^A)$  are highly consistent, thus the connection learned from them is much more robust, but their alignment is indirect for audio-visual representation. On the other hand,  $(\mathbf{v}^I,\mathbf{a}^A)$  pairs are directly beneficial to audio-visual representation learning, but their semantic coherence is less reliable. Note that since the semantic consistency in  $(\mathbf{v}^I,\mathbf{a}^A)$  is derived from  $(\mathbf{t}^I,\mathbf{t}^A)$ , the connection learned from pseudo pair  $(\mathbf{v}^I,\mathbf{a}^A)$  can still be considered as being established via overlapping modalities.
3.4 Intra-MCR Alignment
As discussed in [16], there exists a phenomenon known as the modality gap in MCR spaces. Although the embeddings from different modalities are semantically aligned in an MCR space, they are distributed in entirely distinct regions of the representation space. This implies that the more stable connection learned from  $(\mathbf{t}_i^I,\mathbf{t}_i^A)$  may not accommodate the inputs from audio and image.
To better maintain the connection, we propose closing the modality gap and guaranteeing that embeddings from different modalities with similar semantics are distributed in the same region of the representation space. The analysis in [16] suggests that the repulsive structure in contrastive loss
preserves the modality gap. Inspired by this observation, we derive the intra-MCR alignment loss by removing the repulsive structure in the contrastive loss. As introduced in 3.1, a typical contrastive item can be formulated as:
$$
- \log \frac {\exp \left(\operatorname {s i m} \left(\mathbf {x} _ {i} , \mathbf {z} _ {i}\right) / \tau\right)}{\sum_ {j = 1} ^ {N} \exp \left(\operatorname {s i m} \left(\mathbf {x} _ {i} , \mathbf {z} _ {j}\right) / \tau\right)} = \underbrace {- \operatorname {s i m} \left(\mathbf {x} _ {i} , \mathbf {z} _ {i}\right) / \tau} _ {\text {p u l l p o s i t i v e c l o s e}} + \underbrace {\log \sum_ {j = 1} ^ {N} \exp \left(\operatorname {s i m} \left(\mathbf {x} _ {i} , \mathbf {z} _ {j}\right) / \tau\right)} _ {\text {p u s h n e g a t i v e a w a y}} \tag {8}
$$
We only retain the mechanism of pulling samples closer together and remove the repulsive effect between negative pairs, which helps to close the modality gap in the newly learned MCR space. In the L2-normalized MCR space, there are  $(\mathbf{x}_i - \mathbf{y}_i)^T (\mathbf{x}_i - \mathbf{y}_i) = 2(1 - \mathbf{x}_i^T\mathbf{y}_i)$ . After removing the gradient-irrelevant constant terms, our intra-MCR alignment loss  $L_{intra}$  can be expressed as: