SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
L _ {i n t r a} = \frac {1}{2} \frac {1}{B} \sum_ {i = 1} ^ {B} \left(\| \hat {\mathbf {t}} _ {i} ^ {I} - \hat {\mathbf {v}} _ {i} ^ {I} \| _ {2} + \| \hat {\mathbf {t}} _ {i} ^ {A} - \hat {\mathbf {a}} _ {i} ^ {A} \| _ {2}\right) \tag {9}
$$
By realigning text-guided cross-modal semantically consistent embeddings in each MCR space, i.e., aligning  $(\hat{\mathbf{t}}_i^I,\hat{\mathbf{v}}_i^I)$  for CLIP and  $(\hat{\mathbf{t}}_i^A,\hat{\mathbf{a}}_i^A)$  for CLAP, the modality gap between embeddings from same MCR can be effectively alleviated in the new space. As a result, the more stable connection provided by Equation 5 can be maintained for audio-visual inputs.
During training, all pre-trained encoders in CLIP and CLAP are frozen to preserve the semantic correspondences between image-text and audio-text, and only the two projectors are learnable. To make the training more efficient, we pre-extract the text embeddings  $\mathbf{t}_i^I$  and  $\mathbf{t}_i^A$ . Since the semantic enhancements are training-free, the inter-modality semantic consistency strategy can also be precomputed, and the semantically consistent image  $\mathbf{v}_i^I$  and audio embedding  $\mathbf{a}_i^A$  are stored offline.
We apply a combination of inter- and intra-MCR alignment loss to optimize the two projectors for establishing a stable connection between CLIP and CLAP representation spaces, formulated as:
$$
L = L _ {i n t e r} + \lambda L _ {i n t r a} \tag {10}
$$
$\lambda$  is the hyper-parameter to balance the two terms.
During inference, as shown in Figure 1 (b), the image embedding in CLIP and the audio embedding in CLAP can be mapped into a shared space through corresponding projectors. The cosine scores in this space reflect the semantic similarity between images and audio.
4.1 Details of Connecting CLAP and CLIP
Text Datasets. We collected texts from three sources: image-text datasets (COCO [58] and CC3M [59]), video-text datasets (MSRVTT [60], MAD [61]), and audio-text datasets (AudioCap [62], Clotho [63]), to ensure that the texts contain sufficient visual, action, and audio information. To avoid overfitting visual information, we randomly selected one million descriptions from CC3M. In summary, the texts from image-text, video-text, and audio-text are 1.66M, 0.58M, and 77K, respectively, and there are 2.33M texts in total.
Audio/Image Memory. AudioSet [21] provides a vast collection of audio snippets from YouTube videos. All 1.8M audio data in the training set are encoded by the CLAP audio encoder to serve as the audio memory. ImageNet1K [64] is a large-scale image recognition dataset. We encoded all the 1.3M images in the train set of ImageNet1K using the CLIP image encoder to construct the image memory. It is worth noting that no annotations related to the audio and images are used.
Implementation Details. We employ a frozen pre-trained CLIP ViT-B/32 model [1] and CLAP model [13]. We adopt simple multi-layer perceptrons as our projectors  $f_{1}(\cdot)$  and  $f_{2}(\cdot)$ . The  $\tau_{1}, \tau_{2}$  and  $\tau_{3}$  in Equation 2, 5 and 6 are all set to 1/100. The variance  $\sigma^2$  of the noises in Equation 3 is set as 0.004. The hyper-parameter  $\lambda$  in Equation 10 is set to 0.1. We train our projectors for 36 epochs using a batch size of 10240. We use the AdamW optimizer with the initial learning rate  $1e - 3$  and the cosine learning rate decay strategy.
4.2 Details of Connecting ULIP and CLIP
Image Datasets. The image dataset used for connecting ULIP and CLIP is ImageNet1K [64], total 1.3M images without any annotations.
Text/3D Memory. We use the same 2.33M text dataset as described in Section 4.1, to construct the corresponding text memory. The 3D object point clouds from the training set of Objaverse [48] are utilized to construct 3D memory, 0.8M samples in total.
Implementation Details. We employ a frozen pre-trained CLIP ViT-B/32 model [1], and ULIP-2 PointBERT model [65, 13] pre-trained on ULIP-Objaverse triplets. The structure of the projector and the temperature parameters remain the same in Section 4.1. The variance  $\sigma^2$  of the noises in Equation 3 is set as 0.002. The hyper-parameter  $\lambda$  in Equation 10 is set to 0.4. We train our projectors for 24 epochs using a batch size of 8192. We also use the AdamW optimizer with the initial learning rate  $5e - 3$  and the cosine learning rate decay strategy.
4.3 Evaluation of Audio-Visual Representations
4.3.1 Downstream Audio-Visual Tasks
We assess the quality of audio-visual representations on three downstream audio-visual tasks in a zero-shot manner. More details about the datasets and implementation details are in Appendix.
Audio-Image Retrieval. It contains two subtasks: image-to-audio retrieval (I2A) and audio-to-image retrieval (A2I). We assess the zero-shot image-audio retrieval on the AVE [66] and Flickr-SoundNet [32]. Due to the small size of the test sets in both datasets, we utilized all available data in the train, eval, and test sets for evaluation, resulting in 4095 samples for AVE and 5000 samples for Flickr-SoundNet. For zero-shot inference, we encode all audio and images into our newly learned audio-visual MCR space and computed the cosine similarity for all audio-image pairs. The mAP, Top-1, and Top-5 metrics are used to evaluate retrieval accuracy.
Audio-Visual Source Localization. Audio-visual source localization aims to localize the visual sound sources in an image. The test sets of widely-used VGGSS [67] and MUSIC [68] benchmarks are employed for evaluation. To enable zero-shot inference, we first use a pre-trained object detector [3] to extract object proposals from the images and calculate the cosine similarity between each proposal and audio in our representations space. The proposal with the highest similarity score is token as the final prediction. We adopt Consensus Intersection over Union (cIoU) and Area Under Curve(AUC) metrics following [28, 69].
Counterfactual Audio-Image Recognition. For the non-visible sounds and images with silent objects, this task requires a model to distinguish the semantically unpaired audio-image from audio-image pairs. During the zero-shot inference phase, we employ an object detector [3] to extract object proposals from the image. Subsequently, for each image, the proposal with the highest matching score is considered the predicted object, and the matching score is regarded as the confidence score for this prediction. Experiments are conducted on the Extended VGGSS (Ex-VGGSS) [69] and Extended Flickr-SoundNet (Ex-FlickrNet) [69], and the comparison is based on the Average Precision (AP) and maximum F1 (Max-F1) metrics following [69].
In summary, these three tasks can evaluate a group of audio-visual contrastive representations from various perspectives. Audio-image retrieval is employed to assess the ability to match coarse-grained images and audio, audio-visual source localization is used to evaluate the ability to match fine-grained objects and audio, and counterfactual audio-image recognition is used to evaluate the understanding and reasoning ability of audio and visual inputs.
4.3.2 Analysis on Zero-shot Image-Audio Retrieval
We compared our model with AudioCLIP [19] and WAV2CLIP [20] which are contrastively pretrained on image-audio pairs from AudioSet [21] and VGGSound [22], respectively. Results in Table 1 demonstrate that C-MCR achieves state-of-the-art zero-shot retrieval performance. Besides, the generalization ability of AudioCLIP and WAV2CLIP is not stable. For instance, WAV2CLIP performs well on AVE but poorly on Flickr-SoundNet, while AudioCLIP achieves good results on Flickr-SoundNet but poor accuracy on AVE. Similar situations can also be observed in Table 2. In contrast, our C-MCR exhibits stronger and more stable generalization ability. Moreover, since
Table 1: Zero-shot audio-image retrieval results on AVE and Flickr-SoundNet. A-V pairs stands for whether training on paired audio-visual data; Tr. Param for Trainable parameters number.
Method	A-V Pairs	Tr. Param	AVE	Flickr-SoundNet
A2I	I2A	A2I	I2A
mAP	R@1	R@5	mAP	R@1	R@5	mAP	R@1	R@5	mAP	R@1	R@5
Random	-	-	0.25	0.02	0.12	0.25	0.02	0.12	0.17	0.02	0.06	0.17	0.02	0.06
WAV2CLIP	✓	11.7M	2.80	0.76	3.08	4.01	1.14	4.42	2.52	0.58	3.16	3.47	1.12	4.34
AudioCLIP	✓	134.1M	0.98	0.22	0.85	2.50	1.00	2.83	3.10	1.00	4.02	4.43	1.58	5.92
C-MCR	×	2.1M	4.11	1.25	4.54	4.13	1.25	4.44	4.57	1.38	5.40	4.92	1.58	5.98
Dog barking
Sizzling food
Kid's playful chatter
Figure 2: Visualization of audio-to-image retrieval on AVE and Flickr-SoundNet.
C-MCR does not utilize any paired data and has much fewer learnable parameters, AudioCLIP and WAV2CLIP are not truly "fair" baselines to compare C-MCR with. Nevertheless, C-MCR still demonstrates superior performance compared to these pre-trained audio-visual models. Figure 2 provides a few visualizations of audio-to-image retrieval.
4.3.3 Analysis on Zero-shot Audio-Visual Source Localization
Table 2 presents the zero-shot audio-visual source localization performance on MUSIC-Solo and VGGSS datasets and the comparison with previous audio-visual source localization methods. Remarkably, despite not using any audio-visual paired data and any fine-tuning, C-MCR demonstrates state-of-the-art performance, achieving a relative improvement of around  $25\%$  over the previous leading methods. Additionally, to demonstrate that the improvements are not by introducing the powerful object detector, we also performed the same zero-shot inference using audio-visual representations in AudioCLIP and WAV2CLIP. These methods exhibit unstable generalization performance on the two datasets, and our C-MCR demonstrates a significantly better overall performance than both. These results show that the state-of-the-art performances in audio-visual source localization mainly benefit from the stronger and more robust fine-grained audio-visual matching capability.
4.3.4 Analysis on Zero-shot Counterfactual Audio-Image Recognition
Table 3 shows the comparisons on counterfactual audio-image recognition. Our C-MCR significantly outperforms previous methods that trained on the training set and exhibits overall improvement compared to other audio-visual representation models. The state-of-the-art performance on zero-shot counterfactual audio-image recognition further demonstrates the superiority of our method in understanding the deep semantic relationship between audio and visual modalities.
Table 2: Zero-shot audio-visual source localization on MUSIC-Solo and VGGSS.
Table 3: Zero-shot counterfactual audio-image recognition on Ex-VGGSS and Ex-FlickrNet.
Method	MUSIC-Solo	VGGSS	Method	Ex-VGGSS	Ex-FlickrNet
cIoU	AUC	cIoU	AUC	AP	Max-F1	AP	Max-F1
Attention [32]	37.20	38.70	17.10	28.70	Attention [32]	6.70	13.10	15.98	24.00
DMC [31]	29.10	38.00	23.90	-	DMC [31]	11.53	20.30	25.56	41.80
DSOL [33]	51.40	43.60	29.91	-	DSOL [33]	16.84	25.60	38.32	49.40
TURN [28]	33.70	45.20	34.60	39.10	OGL [36]	18.73	30.90	40.20	55.70
EZ-VSL [36]	-	-	38.85	39.54	EZ-VSL [36]	27.71	34.60	48.75	56.80
SLAVC [69]	-	-	39.80	-	SLAVC [69]	34.46	41.50	52.15	60.10
WAV2CLIP [20]	47.49	53.80	36.91	39.58	WAV2CLIP [20]	33.86	47.69	60.54	66.20
AudioCLIP [19]	30.56	37.16	43.93	45.96	AudioCLIP [19]	42.59	55.43	72.78	71.98
C-MCR(Ours)	53.78	56.09	48.08	48.69	C-MCR(Ours)	50.91	58.98	73.67	74.02
4.4 Evaluation of 3D-language Representations
In order to verify the performance of the 3D-language representation obtained by connecting ULIP-2 and CLIP, we evaluate the zero-shot 3D point cloud classification accuracy on ModelNet40, and the results are shown in Table 4. Our C-MCR achieves state-of-the-art zero-shot classification results compared with methods trained on 3D-language data. The advanced performance in the 3D-language field further demonstrates the great potential of C-MCR to learn contrastive representations for modalities lacking paired data.
Table 4: Zero-shot 3D point cloud classification results on ModelNet40.
Method	Top1	Top3	Top5
ReCon [70]	61.2	73.9	78.1
CG3D [15]	48.7	60.7	66.5
ULIP [17]	60.4	79.0	84.4
ULIP-2 [18]	74.0	86.5	90.0
C-MCR	64.9	87.0	92.8
We conduct ablation studies on audio-image retrieval over AVE and Flickr-SoundNet to examine the effectiveness of our method. All the results are presented in Table 5 and Figure 3.
Semantic Consistency. We use a softmax function to softly aggregate embeddings in memory and produce the semantic consistent embedding in Equation 2. For comparison, Row I selects the embedding in the memory with the highest similarity as generated embedding, while Row J randomly selects an embedding in the memory. Compared to Row I, the significantly better results in Rows J and K highlight the necessity of inter-modality semantic consistency. Compared to the approach of hardly selecting one embedding, as in Row I, our soft clustering of memories slightly improves the performance by generating more diverse embeddings.
Semantic Completion. By comparing H and K, we can find that semantic bias in the MCR space indeed dramatically affects the learning of connections, and adding noise to embeddings can effectively alleviate this issue by enhancing semantic completeness and robustness. The results in G and K demonstrate that our re-normalized operator in Equation 3 is beneficial for learning alignment on the unit sphere. Moreover, in Figure 3, we vary the variance  $\sigma^2$  of noises and report its effect. Generally, the performance is not sensitive to changes in  $\sigma^2$ .
Inter-MCR alignment. Comparison between D, E, and K demonstrates that the connection learned from the native text-text pairs is much more crucial than from the pseudo-consistent audio and visual embeddings, and using both native text-text pairs and pseudo audio-visual pairs produces the best performance. Furthermore, if no connections are made, as in Row F, the image and audio embeddings would have no semantic relationship since the original CLIP and CLAP spaces are isolated.
Intra-MCR alignment. Results in A, B, and K indicate that alignment within either CLIP or CLAP can provide a relatively reliable connection, and aligning both leads to even better results. Conversely, not aligning CLIP and CLAP as in C results in a connection not well adapted to audio-visual input. More importantly, the results in C are similar to those in Row D, where connections are not learned from text-text pairs. This observation indicates that the primary function of intra-MCR alignment is to alleviate the modality gap, thereby enabling the more stable connections learned from text-text pairs to adapt to audio-visual inputs.
Table 5: Ablation studies in AVE and Flickr-SoundNet retrieval. We report the "mAP" metric on both A2I and I2A subtasks. Re-norm stands for the re-normalized operator in Equation 3; CLIP for intra-CLIP alignment item in Equation 9; CLAP for intra-CLAP alignment item in Equation 9; FlickrNet for Flickr-SoundNet dataset.
Rows	Consistency	Completion	Inter-MCR	Intra-MCR	AVE	FlickrNet
Re-norm	Noise	\(L_{ttc}\)	\(L_{avc}\)	CLAP	CLIP	A2I	I2A	A2I	I2A
A	softmax	✓	✓	✓	✓	×	✓	4.09	4.11	4.52	4.71
B	softmax	✓	✓	✓	✓	✓	×	3.97	4.08	4.44	4.79
C	softmax	✓	✓	✓	✓	×	×	3.14	3.22	3.63	3.51
D	softmax	✓	✓	×	✓	✓	✓	3.28	3.30	3.69	3.50
E	softmax	✓	✓	✓	×	✓	✓	4.09	4.10	4.42	4.54
F	softmax	✓	✓	×	×	✓	✓	0.22	0.23	0.18	0.19
G	softmax	×	✓	✓	✓	✓	✓	3.70	3.88	4.57	4.62
H	softmax	×	×	✓	✓	✓	✓	2.77	2.37	2.72	2.57
I	argmax	✓	✓	✓	✓	✓	✓	4.01	3.99	4.49	4.52
J	×	✓	✓	✓	✓	✓	✓	2.62	2.84	2.52	2.76
K	softmax	✓	✓	✓	✓	✓	✓	4.11	4.13	4.57	4.92
Figure 3: Effect of different variance  $\sigma^2$  of the noises in Equation 3 on AVE and Flickr-SoundNet retrieval. The average mAP is the mean value of the mAP in I2A and A2I subtasks.
This paper proposes Connecting Multi-modal Contrastive Representation (C-MCR), a new flexible and training-efficient method for learning multi-modal contrastive representation. C-MCR eliminates the need for large-scale, high-quality data pairs and instead extends the acquired multi-modal alignment knowledge in existing MCRs. By connecting existing MCRs via overlapping modality, we are able to discover more generalized contrastive representations across a broader range of modalities. Experimentally, we learn state-of-the-art audio-visual contrastive representations by connecting CLIP and CLAP through texts, and advanced 3D-language representations by connecting CLIP and ULIP via images. Despite not utilizing any paired data, the representations obtained by C-MCR significantly outperform previous representations learned from data pairs on different downstream tasks.
This work was supported in part by National Key R&D Program of China under Grant No.2022ZD0162000, National Natural Science Foundation of China under Grant No.62222211, No.62072397 and No.61836002.
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
A Ablation Study about Text Dataset.
We conduct more experiments on audio-image retrieval with training texts from different sources. Furthermore, we provide insights about selecting training data when employing our C-MCR to connect other MCRs. As discussed in Sec 4.1, our training texts are collected from three sources: image-text datasets (COCO [58] and CC3M [59]), video-text datasets (MSRVTT [60] and MAD [61]), and audio-text datasets (AudioCap [62] and Clotho [63]).
In Table 4, we exclude the text data from image-text, video-text, and audio-text datasets, respectively. The results demonstrate that combining data from all three sources achieves the best performance. Furthermore, our findings suggest that the information in video-text datasets is relatively less important than in image-text and audio-text datasets. When applying our C-MCR to connect other MCRs, it is critical to collect overlapping modality data associated with information from non-overlapping modalities to
Figure 4: Ablation studies about text datasets for training.
Dataset	AVE	Flickr
A2I	I2A	A2I	I2A
Full	4.11	4.13	4.57	4.92
w/o image-text	3.83	3.76	3.97	3.91
w/o video-text	4.04	4.05	4.41	4.78
w/o audio-text	4.07	3.88	4.31	4.59
ensure robust connections. The data from the pre-training datasets used by MCR could serve as an appropriate starting point. Combining the overlapping modality data from these sources ensures that the data used for constructing connections contains sufficient information from non-overlapping modalities. Additionally, this data is easily accessible and scalable, which greatly enhances the practicality of our C-MCR.
B Downstream Task Details.
B.1 Audio-Image Retrieval.
We consider two datasets for this task: AVE [66] and Flickr-SoundNet [32], both of which consist of semantically matched image and audio pairs that were manually curated. To more comprehensively and stably reflect the retrieval capability of the model, we use all available data in these two datasets for evaluation, resulting in 4,095 samples for AVE and 5,000 samples for Flickr-SoundNet.
B.2 Audio-Visual Source Localization.
We conduct experiments on the VGGSS [67] and MUSIC [68] datasets. VGGSS is derived from VGGSound, and its test set comprises 5,158 audio-image pairs. MUSIC consists of 489 untrimmed videos of musical solos spanning 11 instrument categories for testing. It is worth noting that we use the category names from the COCO dataset as prompts to enable the open-vocabulary object detector GLIP [3] to extract object proposals.
B.3 Counterfactual Audio-Image Recognition.
The Extended Flickr-SoundNet [69] and Extended VGGSS [69] are constructed by adding 250 and 5,158 negative samples to the test sets of the original Flickr-SoundNet and VGGSS datasets, respectively. The prompts used for the object detector GLIP [3] are also the category names from the COCO dataset. We evaluate the counterfactual Audio-Image Recognition performance using the Maximum F1 (Max-F1) and Average Precision (AP) metrics, following [69]. During inference, for the  $i$ -th image-audio pair, the proposal with the highest matching score with the audio is considered the predicted object, and its matching score is considered the confidence score  $c_{i}$ . The CIoU of the predicted object is denoted as  $I_o U_i$ . The ground-truth map is denoted as  $\mathcal{G}_i$ , and the ground-truth maps of negative samples are  $\emptyset$ . Under these definitions, the true positives  $\mathcal{T} \mathcal{P}$ , false positives  $\mathcal{F} \mathcal{P}$ , and false negatives  $\mathcal{F} \mathcal{N}$  are computed as:
$$
\mathcal {T P} (\gamma , \delta) = \{i | \mathcal {G} _ {i} \neq \emptyset , I o U _ {i} > \gamma , c _ {i} > \delta \}
$$
$$
\mathcal {F P} (\gamma , \delta) = \{i | \mathcal {G} _ {i} \neq \emptyset , I o U _ {i} \leq \gamma , c _ {i} > \delta \} \cup \{i | \mathcal {G} _ {i} = \emptyset , c _ {i} > \delta \} \tag {11}
$$
$$
\mathcal {F N} (\gamma , \delta) = \{i | \mathcal {G} _ {i} \neq \emptyset , c _ {i} \leq \delta \}
$$
where  $\gamma$  is the threshold of  $IoU$  and  $\delta$  is the threshold of confidence score. Following previous work, the  $\gamma$  is set as 0.5. The F1 score can be represented as:
$$
F 1 (\gamma , \delta) = \frac {2 * \operatorname {P r e c i s i o n} (\gamma , \delta) * \operatorname {R e c a l l} (\gamma , \delta)}{\operatorname {P r e c i s i o n} (\gamma , \delta) + \operatorname {R e c a l l} (\gamma , \delta)} \tag {12}
$$
$$
\operatorname {P r e c i s i o n} (\gamma , \delta) = \frac {| \mathcal {T P} (\gamma , \delta) |}{| \mathcal {T P} (\gamma , \delta) | + | \mathcal {F P} (\gamma , \delta) |}; \quad \operatorname {R e c a l l} (\gamma , \delta) = \frac {| \mathcal {T P} (\gamma , \delta) |}{| \mathcal {T P} (\gamma , \delta) | + | \mathcal {F N} (\gamma , \delta) |} \tag {13}
$$
In accordance with [69], we calculate F1 scores for all values of  $\delta$  and report the maximum F1 score (Max-F1). Average Precision (AP) is another commonly used metric in object detection, its computation is detailed in [58, 69].
C Model Configurations.
Table 6: Model configurations of projectors.
Module	Block	\(C_{in}\)	\(C_{out}\)
Projector1	Linear	512	1024
BatchNorm1D	1024	1024
Relu	-	-
Linear	1024	512
BatchNorm1D	512	512
Relu	-	-
Projector2	Linear	512	1024
BatchNorm1D	1024	1024
Relu	-	-
Linear	1024	512
BatchNorm1D	512	512
Relu	-	-
The model configurations of our projectors are shown in Table 6.
D Limitations and Future Work.
While C-MCR offers an efficient and effective contrastive representation learning method for modalities that lack high-quality, large-scale paired data, it still necessitates an intermediate modality to associate these modalities. Exploring ways to reduce data requirements further while maintaining representation performance is an intriguing direction for future research.
E Social Impacts.
Although C-MCR achieves outstanding performance in audio-visual learning by connecting CLIP and CLAP, further analysis of the capability boundary of this representation is necessary before applying it to additional modalities or deploying it in practice. C-MCR only requires unpaired unimodal data during training, significantly reducing the data requirements for learning a generalizable representation. However, this also means that unsuitable and harmful data in each modality are more likely to be used for training.