Connecting Multi-modal Contrastive Representations
Authors: Zehan Wang
Yang Zhao
Xize Cheng
Haifeng Huang
Jiageng Liu
Aoxiong Yin
Li Tang
Linjun Li
Yongqi Wang
Ziang Zhang
Zhou Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: multi-modal
representation learning
contrastive learning

Abstract: Multi-modal Contrastive Representation (MCR) learning aims to encode different modalities into a semantically aligned shared space. This paradigm shows remarkable generalization ability on numerous downstream tasks across various modalities. However, the reliance on massive high-quality data pairs limits its further development on more modalities. This paper proposes a novel training-efficient method for learning MCR without paired data called Connecting Multi-modal Contrastive Representations (C-MCR). Specifically, given two existing MCRs pre-trained on $(\mathcal{A}$, $\mathcal{B})$ and $(\mathcal{B}$, $\mathcal{C})$ modality pairs, we project them to a new space and use the data from the overlapping modality $\mathcal{B}$ to aligning the two MCRs in the new space. Meanwhile, since the modality pairs $(\mathcal{A}$, $\mathcal{B})$ and $(\mathcal{B}$, $\mathcal{C})$ are already aligned within each MCR, the connection learned by overlapping modality can also be transferred to non-overlapping modality pair $(\mathcal{A}$, $\mathcal{C})$. To unleash the potential of C-MCR, we further introduce a semantic-enhanced inter- and intra-MCR connection method. We first enhance the semantic consistency and completion of embeddings across different modalities for more robust alignment. Then we utilize the inter-MCR alignment to establish the connection, and employ the intra-MCR alignment to better maintain the connection for inputs from non-overlapping modalities. To demonstrate the effectiveness of C-MCR, we take the field of audio-visual and 3D-language learning as examples. Specifically, we connect CLIP and CLAP via texts to derive audio-visual representations, and integrate CLIP and ULIP via images for 3D-language representations. Remarkably, without using any paired data, C-MCR for audio-visual achieves state-of-the-art performance on audio-image retrieval, audio-visual source localization, and counterfactual audio-image recognition tasks. Furthermore, C-MCR for 3D-language also attains advanced zero-shot 3D point cloud classification accuracy on ModelNet40. Our project page is available at \url{https://c-mcr.github.io/C-MCR/}

Connecting Multi-modal Contrastive Representations
Zehan Wang'
Yang Zhao²
Xize Cheng'
Haifeng Huang'
Jiageng Liu' Li Tang'
Linjun Li
Yonggi Wang'
Aoxiong Yin' Ziang Zhang'
Zhou Zhao!.3
Zhejiang University
SShangaai Al Laboratory {wangzehan01)@zzju edu.
ByteDance
Abstract
Multi-modal Contrastive Representation (MCR) learning aims to encode different modalities into semantically aligned shared space. This paradigm shows remark- able generalization ability on numerous downstream tasks across various modalities. However, the reliance massive high-quality data pairs limits its further develop- ment on more modalities. This paper proposes a novel training-efficient method for learning MCR without paired data called Connecting Multi-modal Contrastive Representations (C-MCR). Specifically, given two existing MCRs pre-trained on (A, B) and (B, C) modality pairs, we project them data from the overlapping modality B to aligning the two MCRs in the new space. Meanwhile, since the modality pairs (A, B) and (B, C) are already aligned within each MCR, the connection learned by overlapping modality can also be transferred to non-overlapping modality pair (A,, C). To unleash the potential of C-MCR, we further introduce semantic-enhance inter- and intra-MCR connection method. We first enhance the semantic consistency and completion of embeddings across different modalities for more robust alignment. Then we utilize the inter- MCC alignment establish the connection, and employ the intra-MCR alignment better maintain the connection for inputs from non-overlapping modalities. To demonstrate the effectiveness C-MCR, we take the field audio-visual and 3D- language learning as examples. Specifically, we connect CLIP and CLAP via texts to derive audio-visual representations, and integrate CLIP and ULIP via images for 3D-language representations. Remarkably, without using any paired data, C-MCR for audio-visual achieves state-of-th thth--f--hh-rrttt performance on iadio- retrieval, audio-visual source localization, and counterfactual audio-image recognition tasks. Furthermore, C-MCR for 3D-lanuuage also attains advanced zero-shot 3D point cloud classification accuracy ModelNet40. Our project page is available htttps://c-ie github. 1c/C-MCR/
new space and the
Introduction
Multi-mon Multi-modal Contrastive Representation (MCR) learning aims map inputs from different modalities to shared representation space. With the impressive generalization performance of vision-lante language contrastive pre-training models 2 4 demonstrated on various downstream tasks [5 6 10
learning MCR spaces between multiple modalities has become promising area of research, attracting
increasing attention 11 12
13
15).
However, the generalization ability of MCR primarily benefits from the accessibility of massive data pairs from the web. For modalities where obtaining semantically matching data pairs is significantly more costly, the representations directly learned from limited data pairs are unreliable. On the other
"Correponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

hand, these modality pairs with little direct paired data often have large number of paired data with the same intermediate modality. For example, although audio-visual data are often vague, paired data of audio-language and language-image are sufficient and semantically explicit. Similarly, while 3D point-language pairs are rare, 3D point-image and image-language data are extensive.
Consider that there are already many MCRs between modalities with sufficient paired data. In this paper, we propose Connecting Multi-modal Contrastive Representations (C-MCR), novel training- MCR learning method that extends the learned alignment knowledge in existing MCRs to more modalities. With regard the overlapping modality, its representations in two MCRs are just different data views sharing the same inherent semantics, So we can take them as positive pairs to connect different MCRs. As modalities within each MCR are semantically aligned, the connections built from overlapping modalities can also be applied to -verrlaapp modalities. The advantages of our C-MCR are two-fold: (1) Flexible. C-MCR enables MCR learning on modalities with limited paired data. More importantly, C-MCR treats each learned MCR space as node and the overlapping modalities between different MCRs as links. Connecting the various isolated MCRs greatly extends the obtained multi-modaa alignment knowledge, and discovers generalized contrastive representations of broader modalities. (2) Training-Efficienn
efficient and paired-data-fee
Since C-MCR simply reprojects the learned representations into new space, only two simple projectors are learnable during training. The training parameters and costs for connecting existing MCRs are very small.
However, two factors impede the acquisition of robust and transferable connection: Firstly, em- beddings in MCR spaces are incapable of comprehensively reflecting all the semantic information of the input, and this loss of meaning would be inherited and amplified, thereby compromising the robustness of the connection. Secondly, as discussed in [16], MCR spaces exhibit modality gap phe- nomenon, i.e., the embeddings of different modalities are located in two completely separate regions in each MCR space. This poses a challenge for maintaining the connection based on overlapping modality while facing inputs from non-overlapping modalities.
Considering the above challenges, we propose semantic-enhanced inter- and intra-MCR connection method. During training, the copious amounts of easily accessible unpaired unimodal data are first encoded into embeddings two MCR spaces. We inject Gaussian noise into all the embeddings to mitigate the semantic bias, enhance the semantic completeness, and improve robustness. For directly quantifying the modality gap and the relationship between non-overlapping modalities, we exploit the inherent multi-modal alignment in MCR spaces to cluster semantic consistent embeddings and bridge different modalities. With the above strategies, we align the
embeddings across different MCR spaces in contrastive manner establish the connection. To preserve the connection for inputs from non-overlap modalities, we realign the semantic similar embeddings across modalities within each MCR space to alleviate the modality gap.
Our main contributions are summarized as follows:
(I) We propose Connecting Multi-modal Contrastive Representations (C-MCR), a novel paired-data- free and training-efffccien method for MCR learning. By connecting existing MCR spaces with simple projectors, we can mine the multi-modal alignment knowledge in existing MCR space, and extend MCRs more modalities that lack large-scale high-quality data pairs.
(2) We further propose a semantic-enhanced inter- and intra-MCR connection method to unleash our C-MCR. This approach establishes transferable connection between two MCR spaces via overlapping modality and maintains for non-overlapping modalities.
(3) To demonstrate the effectiveness of C-MCR, we connect the CLIP and CLAP through texts to acquire audio-visuaa representations, and interage CLIP and 3D-image MCR space (ULIP) via images for 3D-language representations. Remarkably, without requiring any pair data or fine-tuning, C-MCR for andio-visual achieves state-of-the-art performance on six datasets across three downstream audio- visual tasks. Furthermore, C-MCR for 3D-language also attains advanced zero-shot 3D point cloud classification accuracy on ModelNet40..
2 Related Work
Multi-modal Contrastive Representation learning. Multi-modal contrastive representation focuses on learning separate unimodal encoders for different modalities, which can map inputs from different modalities into shared representation space. These models are pre-trained on large-scale paired data

---

(a). Connecting Multi-modal Representutious Spaces
Inference
Asde
Similurity
Projectort
algestient
Projctert
Prajeckur2
CLAP
Enncdeer
4-1
Inte-MMRR
CLAP Azdai
Ensder
Excodn
tmaux
Senauatle
lati-eet aligmment low.
Audio
Image
4
42
(c), Injecting Notse
Teals
CLIP Text
Imz-MMR image sidio
140
110
alggmeee loss
Projectur2
12
12
12
Iaage
luca-CLIP aligmmnt lase
prodect
Counsion injection n-nomolization
The pipeline of connecting CLIP and CLAP using our C-MCR. During training, we take text input and encode it with frozen CLAP and CLIP text encoders, respectively. Audio(image) memory is generated by encoding lots of unimodal audio(image) data by pre-trained audio(image) encoder. Semantic enhancement enriches the semantic consistency and completion of embeddings. Then two projectors map embeddings to a new shared representation space where inter- and intra- MCR alignment establishes and maintains a stable connection between CLAP and CLIP. During inference, the audio and image are inputted to the corresponding encoder and projector.
Figure 1:
using a contrastive loss. Recent vision- Lasion-language contrastive pre-training models, such as CLIP 1 and ALIGN (2], demonstrate impressive zero-shot retrieval and classification performance and remarkable generalization capability across diverse downstream tasks 6 7 g 9 [0]. Inspired by the success of vision-language models, contrastive representation learning across more modalities has garnered increasing attention. CLAP [12 [1] construct a contrastive language-audio pre-training model by collecting large-scale audio-text pairs from diverse data sources. ULIP [17 [8] collect and generate 3D- 3D-image-text triplet data via 3D rendering and image captioning, and learn an extra 3D encoder for existing vision- laniuuge space. AudioCLIP
[99 and WAV2CLIP (0 leverage the pre-trained CLIP image encoder and acquire audio-visual representations by training on audio-image pairs from [2]] and VGGSound [22]. respectively. For certain modality pairs, such audio-visual and ID-language, the pre-training model's generalization capability is restricted by ambiguous or limited paired data. Our proposed method introduces novel method for better contrastive representation learning on these modalities.
AudioSet
Audio-Visual Learning. Audio-visual learning 23 aims to exploit the relationship between audio and visual modalities, which is an essential part of intelligent multi-modal perception research. Previous methods primarily focus on learning specific audio-visual downstream tasks (such as retrieval (24] 25 26), localization [27 28 29 30 31 32
33, 34 35| 33). or generation [37] 38 33 43 44)) within limited domains, based on the manually cleaned small-scale datasets.
40, 41 Recently, several large-scale audio-image datasets [2T 22 collected from the web are proposed. However, these datasets contain many noisy image-audio pairs due to the ambiguous nature of both images and audio and the presence of non-visible sounds and silent objects in videos. Consequently, the generalization ability of audio-visual contrastive representation [99 20 learned from these datasets is limited. Our C-MCR reduces the need for larger-scale high-quality data pairs. By extending the knowledge in CLIP and CLAP models, we acquire powerful audio-visual contrastive representations that exhibit powerful generalization capabilities across various downstream tasks.
42
3D-Januaaee Learning. 3D vision an important way for robots to perceive the rich semantic and spatial information of the real world. The 3D-language learning including recognition [45 46) 47 (8).
localization 49 (50 [T 52 53 question-answer [54] (55 and general conversation [56] 57 have
attracted increasing attentions. 3D-language contrastive representations are vital for the further development of 3D-language learning. However, due the scarcity of 3D-language paired data, the development of 3D-language representation is limited. Recent ULIP [17 [8] focus on generating 3D-image-text triplet data, but they are still limited by the relatively low quality of training datasets. C-MCR gets rid of the dependence on 3D-language pairing data, and instead connects the reliable 3D-visual representation of ULIP and the visual-language representation of CLIP via images to obtain a more robust 3D-language contrastive representation.
3

---

3 Approach
In this section, we take connecting CLIP and CLAP for audio-visual, as an example to introduce C-MCR. As depicted in (ag utilize two projectors to connect CLIP and CLAP through texts. Before delving into our method, we first introduce the mathematical formulations and revisit the multi-modal contrastive learning in Section 3.1 Then we discuss our semantic enhancement approach for robust representation alignment in Section 3.2 This is followed by the inter-MCR alignment establish the connection between CLIP and CLAP in Section 3.3] and the intra-MCR alignment to ensure the connection can be maintained for image-audio inputs in Section 3.4
3.1 Background
Problem formulation. For text inputs, the embeddings obtained by CLIP and CLAP encoder can be denoted as t' RC and A E Rd respectively. Our C-MCR method aims to leverage the inherent consistency between t' and A and the multi-modal alignment in MCR to learn two projectors (-) and f2) that map the representations from CLIP and CLAP to a new shared representation space. The connection between CLIP and CLAP, learned from texts, can effectively accommodate audio and image inputs.
Multi-modal Contrastive Learning, Given N paired instances from two different modalities, we map the i-th pair to L2-normalized embeddings x, and z; via two encoders. Multi-modal contrastive learning aims to maximize the cosine similarity between and 2; and minimize the cosine similarity between x; and 2j where * j. The contrastive loss can be formulated as:
Leons 11 2
log exp(sim(x; 2:)/T)
exp(sim(z, x,)/T) Σド=1 exp(sim(z;;xx//1)
log
(1)
Σド= exp(sim(xx,,,)) 2/)/T)
where is the temperature parameter and the sim(..-) is the operator for cosine distance. The contrastive loss based on multi-modal data pairs, and the generalization the learned representation relies on the scale and quality of the data pairs. To extend contrastive representation learning more modalities, we propose using overlapping modalities to connect two Icarned MCR spaces and extend the learned multi-modal alignment knowledge to non-overlapping modalities.
3.2 Semantic Enhancement
To achieve robust and comprehensive alignment, enhance the semantics from two perspec- tives: inter-modality semantic consistency and intra-modality semantic completion.
Inter-modality Semantic Consistency. CLIP and CLAP have already learned shared image-text and audio-text representations. To better quantify the modality gap in the MCR space and directly explore the correlation between audio and image, we first utilize the inherent modality alignment properties of CLIP and CLAP to generate semantically consistent embeddings across modalities. Specifically, encode massive unpaired images and audio using the CLIP image encoder and CLAP audio encoder, respectively. All the obtained image embeddings are served as image memory audio memory A {a) a2, am ). where
V (V1, .... VN) and the audio embeddings N, MM indicate the number of images and audios. Considering i-th text embeddings t// and t, we can generate image embedding v and audio embedding at that are semantically similar to i-th text.
VRi a exp(sim(t/, vz)/m)
exp(sim(t/) ax)/m) Σド= exp(sim( aT)/T))
&
Σj= exp(sim(t/ vx,s//
The is the temperature hyperparameter. By dynamically absorbing information from memories based on semantic similarity to the text embeddings t! and t²², we can generate more diverse and accurate mnnantically-consiste embeddings v and aç"
Intra-modality Semantic Completion. The semanties the original input data are often complex, and some information is inevitably lost when encoding it into the MCR space. When connecting and aligning existing representation spaces, this loss and bias of meaning will be inherited and amplified, affecting the robustness of alignment. To enhance the semantic completeness of each embedding,

---

we propose to serve Gaussian noise as an information augmentation method. Specifically, we add zero-mean Gaussian noises into the embeddings and re-normalize them the unit hypersphere:
i' Normalize(t ' + 01);
Normalize(y v + 02)
[ = Normalize (t + 0z); à Normalize(a + 04)
where noise items 01, O2 E _ and 03, 04 E R2 are sampled from zero-mean gaussian distribution with variance o², and they are not learnable.
Since the MCRs are L2 normalized, all embeddings are distributed on unit sphere. As illustrated in (c), each embedding can be viewed Gaussian noise can transform the point into small sphere, and re- re-normalizing projects the small sphere onto a circle on the surface of the unit sphere. Hence, aligning two embeddings with noise forces the model to acquire the ability to align all the embeddings within the two circles. In the MCR space, the closer two embeddings are to each other, the more similar their semantics are. Embeddings within the same circle share similar general semantics, and the semantics represented by the circle are more comprehensive and robust than the original embedding.
point on the unit sphere's ' surface. The addition of
3.3 Inter-MCR Alignment
To establish the connection between two MCRs, we project the eehnacee embeddings from CLIP and CLAP space to a new shared space via two learnable projectors fi(-) and fz(+), respectively.
t' fit'; v' = fi('); E = f2("(; a = fz(ā²)
(4)
In the newly projected space, our objective is to ensure that embeddings with similar semantics from different MCR spaces in close proximity to each other. The (t. .t.") from the same text is naturally semantic consistent, and it can be considered as a ground-truth pair label. Besides, there is pseudo consistency in (v. .t/) and (a .t:²) due to the multi-modal alignment properties in CLIP and CLAP. pseudo pair label. For robust and stable Thus the (V' a derived from (t/,t,² can be viewed connection of the two MCR, propose to align both (t' () and (v (aA). The text-text contrastive loss Lite and visual contrastive loss Lave are defined as:
11
cxp(sim( (t/simmt/./aa (/22)))) + log
exp(sim E.A t()/T2)
Lute = 2
log B
(5)
==1 exp(ssmm (t/, (7)22))
B Σ //=1 exppexi (t,1, (/2/T2))
i=1
1
exp(sim v 1/33)
exp(sim (a, 3/)/))))
Luve
2 log
expisim( ( + log
(6)
Σ E exp(sim(a{ +/)/T3)
B corresponds to the batch, and T2, T3 the temperature hyperparameters. The inter-MCR alignment loss Linter is the combination of the two contrastive losses:
Linter
Lite + Lave
The Lute and Lave complementary to each other. The semanties between (t',t are highly consistent, thus the connection learned from them is much more robust, but their alignment is indirect for audio-visud representation. On the other hand, (v' .aaa) pairs directly beneficial visual representation learning, but their semantic coherence is less reliable. Note that since the semantic consistency in ,) is derived from (t' ,t,), the connection learned from pseudo pair (v= can still be considered as being established via overlapping modalities.
audio-
3.4 Intra-MCR Alignment
As discussed in [16,, there exists phenomenon known as the modality gap in MCR spaces. Although the embeddings from different modalities are semantically aligned in an MCR space, they are distributed in entirely distinct regions of the representation space. This implies that the more stable connection learned from (t., t/") may not accommodate the inputs from audio and image.
To better maintain the connection, we propose closing the modality gap and guaranteeing that embeddings from different modalities with similar semantics are distributed in the same region of the representation space. The analysis in [16] suggests that the repulsive structure in contrastive loss
5

---

preserves the modality gap. Inspired by this observation, we derive the intra-MCR alignment loss by removing the repulsive structure in the contrastive loss. As introduced in 3.1 a typical contrastive item can be formulated as:
log exp(sim(x; 2/)/T) exp(sim (x,, 2,)/T)
sim(x;, 2T)/T log exp(sim(xx,
(8)
pull positive close
push negotive away
We only retain the mechanism of pulling samples closer together and remove the repulsive effect between negative pairs, which helps to close the modality gap in the newly learned MCR space. In the L2-normalized MCR space, there are (x; y;)" gradient-irrelevan constant terms,
"x)"(xx (x, ys) = 2(1 x y:). After removing the MCR alignment loss Lintra be expressed
Lintra 2 i=1
1
E(IE) - v/ l|: 112
By realigning text-guided cross-modal semantically consistent embeddings in each MCR space. i.c., aligning (i/ v for CLIP and (t:², a²²) for CLAP, the modality gap between embeddings from same MCR can be effectively alleviated in the new space. As result, the more stable connection provided by Equation [ can be maintained for audio-visual inputs.
3.5 Training and Inference
During training, all pre-trained encoders in CLIP and CLAP are frozen to preserve the semantic correspondences between image-text and audio-text, and only the two projectors are learnable. To make the training more efficient, pre-extract the text embeddings t! and t²² Since the semantic enhancements are training- ffren the inter-modality semantic consistency strategy can also be pre- computed, and the semantically consistent image v and audio embedding a²² are stored offline.
We apply a combination of inter- and intra-MCR alignment loss to optimize the two projectors for establishing stable connection between CLIP and CLAP representation spaces, formulated as:
L Linter + ALintra
(10)
À is the aperr-parameter to balance the two terms.
During inference, as shown in Figure (b), the image embedding in CLIP and the audio embedding in CLAP can be mapped into a shared space through corresponding projectors. The cosine scores in this space reflect the semantic similarity between images and audio.
4 Experiments
4.1 Details of Connecting CLAP and CLIP
Text Datasets. We collected texts from three sources; image-text datasets (COCO 58 and
CC3M (9)), video-text datasets (MSRVTT [60]. MAD [I]), and audio-text datasets (AudioCap 62
Clotho (33). ), to ensure that the texts contain sufficient visual, action, and audio information. To avoid overfitting visual information, we randomly selected one million descriptions from CC3M. In summary, the texts from image-text, video-text, and audio-text are 1.66M, 0.58M, and 77K, respectively, and there are 2.33M texts in total.
Audio/Image Memory. AudioSet (1 provides a vast collection of audio snippets from YouTube videos. All 1.8M audio data in the training set are encoded by the CLAP audio encoder to serve as the audio memory. ImageNet1K (64] is large-scale image recognition dataset. We encoded all the 1.3M images in the train set of ImageNet1K using the CLIP image encoder to construct the image memory. It is worth noting that no annotations related to the audio and images are used.
Implementation Details. We employ frozen pre-trained CLIP ViT-B/32 model and CLAP model [13]. We adopt simple multi-layer perceptrons our projectors f1(-) and fz(-). The T1, T2 and T3 in Euqation 5 are all set to 1/100. The variance o² of the noises in Equation 3 is set as 0.004. The typer-parameter X in [omation[OOisa set to 0.1. We train our projectors for 36 epochs using a batch size of 10240. We use the Adam optimizer with the initial learning rate le 3 and the cosine learning rate decay strategy.

---

4.2 Details of Connecting ULIP and CLIP
Image Datasets. The image dataset used for connecting ULIP and CLIP is ImageNetIK 64 total 1.3M images without any annotations,
Text/3D Memory. We the same 2.33M text dataset described in Section [. to construct the corresponding text memory. The 3D object point clouds from the training set of Objaverse utilized to construct 3D memory, 0.8M samples in total.
(88 are
Implementation Details, We employ a frozen pre-trained CLIP ViT-B/32 model [. and ULIP-2 PointBERT model 665 13 pre-trained ULIP-Objaverse triplets. The structure of the projector and the temperature parameters remain the same in Section 4.1 The variance o² the noises in Equation is set 0.002. The hyper- epochs using a batch size of 8 8192. We also use the Adam optimizer with the initial learning rate 5e 3 and the cosine learning rate decay strategy.
À in Equation [0 set to 0.4. We train our projectors for 24
4.3 Evaluation of Audio Visual Representations
4.3.1 Downstream Audio-Visual Tasks
We assess the quality of audio-visual representations on three downstream audio-visual tasks in a zero-shot manner. More details about the datasets and implementation details are in Appendix.
Audio-Image Retrieval. It contains two subtasks: image-to-aul au-iioo retrieval (I2A) and audio-to- image retrieval (A2I). We assess the zero-shot image-audio retrieval on the AVE [66] and Flickr- SoundNet [32] Due the small size of the test sets both datasets, we utilized all available data in the train, eval, and test sets for evaluation, resulting in 4095 samples for AVE and 5000 samples for Flickr-Sounddet. For zero-shot inference, we encode all audio and images into our newly learned audio-visual MCR space and computed the cosine similarity for all audio-image pairs. The mAP, Top-1, and Top-5 metrics are used to evaluate retrieval accuracy.
Audio Visual Source Localization. Audio-visual source localization aims to localize the visual sound sources in an image. The test sets of widely-used VGGSS 67 and MUSIC (68) benchmarks are employed for evaluation. To enable zero-shot inference, we first use pre-trained object detector B extract object proposals from the images and calculate the cosine similarity between each proposal and audio in our representations space. The proposal with the highest similarity score is token as the final prediction. We adopt Consensus Intersection over Union (cloU) and Area Under Curve(AUC)
metrics following [8 69]
Counterfactual Audio-Image Recognition. For the non-visible sounds and images with silent objects, this task requires model to distinguish the semantically unpaired audio- image from audio- image pairs. During the zero-shot inference phase, we employ an object detector [3] extract object proposals from the image. Subsequently, for each image, the proposal with the highest matching score considered the predicted object, and the matching score regarded the confidence score for this prediction. Experiments are conducted on the Extended VGGSS (Ex- V-GGGG [69] and Extended
Blickr-SoundNer Flickreet)) [69], and the comparison is based on the Average Precision (AP)
and maximum F1 (Max-F1) metrics following [69
In summary, these three tasks evaluate a group of audio-visual contrastive representations from various perspectives. Audio-image retrieval is employed to assess the ability to match coarse-grained images and audio, audio-visual source localization used evaluate the ability match grained objects and audio, and counterfactual audio-image recognition is used to evaluate the understanding and reasoning ability of audio and visual inputs.
4.3.2 Analysis on Zero-shot Image-Audio Retrieval
20 which are contrastively pre- We compared our model with AudioCLIP 19 and WAV2CLIP
trained on image-audio pairs from AudioSet 21 and VGGSound [22] respectively. Results in Table demonstrate that C-MCR achieves state-of of-te----hh-arr tht zero-shot retrieval performance. Besides, the generalization ability of AudioCLIP and WAV not stable. For instance, WAV2CLIP performs well on AVE but poorly Flickr-SoundNet, while AudioCLII achieves good results on Shond--SoundNet but poor accuracy on AVE. Similar situations can also be observed in Table 2 In contrast, our C-MCR exhibits stronger and more stable generalization ability. Moreover, since
7

---

Table 1: Zero-shot audio-image retrieval results AVE and Flickr-SoundNet. A-V pairs stands for whether training on paired audio-visual data; Tr. Param for Trainable parameters number.
A-V
Tr.
AVE
Flickr-SoundNet
Method
Pairs
Param
A2I
12A
A21
12A
MAP
@@
R@5
mAP
R@1
R@5
mAP
R@1
R@5
mAP
R@I
R@5
Random
0.25
0.02
0.12
0.25
0.02
0.12
0.17
0.02
0.06
0.17
0.02
0.06
WAV2CLIP
11.7M
2.80
0.76
3.08
4.01
1.14
4.42
2.52
0.58
3.16
3,47
1.12
4.34
AudioCLIP
134. IM
0.98
0.22
0.85
2.50
1.00
2.83
3.10
1.00
4.02
4.43
1.58
5.92
C-MCR
X
2.1M
4.11
1.25
4.54
4.13
1.25
4.44
4.57
1.38
5.40
4.92
1.58
5.98
Dog barking
d Sizzling food
o Kid's playful chatter
Figure Visualization of audio-to-image retrieval on AVE and Flickr-SoundNet..
C-MCR does not utilize any paired data and has much fewer learnable parameters, AudioCLIP and WAV are not truly "fair" baselines to compare C-MCR with. Nevertheless, C-MCR still demonstrates superior performance compared to these pre-trained audio-visual models. Figure 2 provides a few visualizations of audio-to-image retrieval.
4.3.3 Analysis on Zero-shot Audio- Visual Source Localization
Table presents the zero-shot audio-visual source localization performance on MUSIC-Solo and VGGSS datasets and the comparison with previous audio-visual source localization methods. Re- markably, despite not using any audio-visual paired data and any fine-tuning, C-MCR demonstrates state-of-the-ate performance, achieving relative improvement of around 25% over the previous lead- ing methods. Additionally, to demonstrate that the improvements are not by introducing the powerful object detector, we also performed the same zero-shot inference using audio-visual representations in AudioCLIP and WAV2CLIP. These methods exhibit unstable generalization performance on the two datasets, and our C-MCR demonstrates a significantly better overall performance than both. These results show that the state-of-the-arr performances audio-visual source localization mainly benefit from the stronger and more robust fine-gra -ineinraiedd audio-visun matching capability.
4.3.4 Analysis on Zero-shot Counterfactual Audio-Image Recognition
Table shows the comparisons on counterfactual audio- imale recognition. Our C-MCR significantly outperforms previous methods that trained on the training set and exhibits overall improvement compared to other audio-visual representation models. The state-of-the-art performance zero- shot counterfactual audio-image recognition further demonstrates the superiority of our method in understanding the deep semantic relationship between audio and visual modalities.

---

Table 2: Zero-shot audio-visual source localization on MUSIC-Solo and VGGSS.
Table 3: Zero-shot counterfactual audio-image recognition on Ex-VGGSS and Ex-FlickrNet.
MUSIC-Solo
VGGSS
EX-VGGSS
Ex-FlickrNet
Method
Method
cloU
AUC
cloU
AUC
AP
Max-F1
AP
Max-F1
Attention
32
37.20
38.70
17.10 28.70
Attention 32
6.70
13.10
15.98
24.00
DMC B1
29.10
38.00
23.90
DMC B1
11.53
20.30
25.56
41.80
DSOL 33
51.40
43.60
29.91
DSOL B3
16.84
25.60
38.32
49.40
TURN 288]
33.70
45.20
34.60
39.10
OGL [36]
18.73
30.90
40.20
55.70
EZ-VSL
36
38.85
39.54
EZ-VSL 366
27.71
34.60
48.75
56.80
SLAVC
[69]
39.80
SLAVC (69
34.46
41.50
52.15
60.10
WAV2CLIP 20
47.49
53.80
36.91
39.58
WAV2CLIP 20
33.86
47.69
60.54
66.20
AudioCLIP
19
30.56
37.16
43.93
45.96
AudioCLIP 19
42.59
55.43
72.78
71.98
C-MCR(Ours)
53.78
56.09
48.08
48.69
C-MCR(OUrs)
50.91
58.98
73.67
74.02
4.4 Evaluation of 3D-lanuuage Representations
Table 4: Zero-shot 3D point cloud
In order to verify the performance of the 3D-language rep- resentation obtained by connecting ULIP-2 and CLIP, evaluate the zero-shot 3D point cloud classification accu- racy ModelNet40, and the results shown in Table Our C-MCR achieves state-of-the-an zero-shot classifica- tion results compared with methods trained on 3D-language data. The advanced performance in the 3D-language field further demonstrates the great potential of C-MCR to learn contrastive representations for modalities lacking paired data.
classification results on ModelNet40.
Method
Top1
Top3
Top5
ReCon 70
61.2
73.9
78.1
CG3D 15]
48.7
60.7
66.5
ULIP [7
60.4
79.0
84.4
ULIP-2 [88
74.0
86.5
90.0
C-MCR
64.9
87.0
92.8
4.5
Ablation Studies
We conduct ablation studies on audio-image retrieval over AVE and Flickr-SoundNet to examine the effectiveness of our method. All the results are presented in Table 5 and Figure 3
Semantic Consistency. We and produce the semantic consistent embedding in Equation 2 For comparison, Row selects the embedding in the memory with the highest similarity generated embedding, while Row J randomly selects an embedding in the memory. Compared to Row I, the significantly better results in Rows and K highlight the necessity of inter-modality semantic consistency. Compared to the approach of hardly selecting one embedding, as in Row I, our soft clustering of memories slightly improves the performance by generating more diverse embeddings.
softmax function to softly aggregate embeddings in memory
Semantic Completion. By comparing H and K, we can find that semantic bias in the MCR space indeed dramatically affects the learning of connections, and adding noise embeddings can effectively alleviate this issue by enhancing semantic completeness and robustness. The results in G and K demonstrate that our re-normalized operator in Equation 3 is beneficial for learning alignment on the unit sphere. Moreover, in Figure 3, we vary the variance g² of noises and report its effect. Generally, the performance is not sensitive changes in a
Inter-MCR alignment. Comparison between D, E, and demonstrates that the connection learned from the native text-text pairs is much more crucial than from the pseudo-consistent audio and visual embeddings, and using both native text-text pairs and pscudo audio-visual pairs produces the best performance. Furthermore, if no connections made, as in Row F. the image and audio embeddings would have no semantic relationship since the original CLIP and CLAP spaces are isolated.
Intra-MCR alignment. Results in A, B, and K indicate that alignment within either CLIP or CLAP can provide relatively reliable connection, and aligning both leads even better results. Conversely, not aligning CLIP and CLAP as in C results in connection not well adapted to audio-visus visual input. More importantly, the results in C are similar to those Row D, where connections are not learned from text-text pairs. This observation indicates that the primary function of intra-MCR alignment is to alleviate the modality gap, thereby enabling the more stable connections learned from text-text pairs to adapt to audio-visual inputs.

---

Table 5: Ablation studies in AVE and Flickr-SoundNe retrieval. We report the "mAP" metric both A2I and 12A subtasks. Re-norm stands for the re-normalized operator in Equation 3 CLIP for alignment item in Equation 9 FlickrNet for Flickr-SoundNet dataset.
CLAP for mnta-CLAP alignment item in Equation 9
Completion Re-norm Noise
Inter-MCR
Intra-MCR
AVE
FlickrNet
Rows
Consistency
Lite
Lave
CLAP CLIP
A2I
12A
A2I
12A
A
softmax
X
4.09
4.11
4.52
4.71
softmax
/
1
V
X
3.97
4.08
4.44
4.79
C
softmax
/
X
3.14
3.22
3.63
3.51
D
softmax
V
V
3.28
3.30
3.69
3.50
E
softmax
4.09
4.10
4.42
4.54
softmax
0.22
0.23
0.18
0.19
softmax
X
3.70
3.88
4.57
4.62
H
softmax
X
2.77
2.37
2.72
2.57
I
argmax
4.01
3,99
4.49
4.52
2.62
2.84
2.52
2.76
K
softmax
4.11
4.13
4.57
4.92
50
33-
15
10
10
Figure 3: Effect different variance σ² of the noises in Equation 3 on AVE and Flickr-SoundNet retrieval. The average mAP is the mean value of the mAP in 12A and A2I subtasks.
Conclusion
This paper proposes Connecting Multi-modal Contrastive Representation (C-MCR), new flexible and training-efficient method for learning multi-modal contrastive representation. C-MCR eliminates the need for large-scale, high-quality data pairs and instead extends the acquired multi-modal aligmment knowledge in existing MCRs. By connecting existing MCRs via overlapping modality, we able to discover more generalized contrastive representations across a broader range of modalities. Experimentally, we learn state-of-the-as audio-visual contrastive representations by connecting CLIP and CLAP through texts, and advanced 3I>-language representations by connecting CLIP and ULIIP via images. Despite not utilizing any paired data, the representations obtained by C-MCR significantly outperform previous representations learned from data pairs different downstream tasks.
Acknowledgments
This work was supported in part by National Key R&D Program of China under Grant No. No.20222DD162000 National Natural Science Foundation of China under Grant 1,.62222211, No.62072397 and 63.6022.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages PMLR, 2021.
10

---

[2] Chao Jia, Yinfei Yang, Ye Xia. Yi-Ting Chen. Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and usion-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4916- PMLR, 2021.
[3] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-imag" pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975, 2022.
[4] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144, 2022.
[5] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[6] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint urXiv:2204.06/22 2022.
[7] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508:293-304, 2022.
[8] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang. Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18082-18091, 2022.
[9] Medhini Narasimhan, Anna Rohrbach, and Trevor Darrell. Clip-it! looguage-guided video summarization. Advances in Neural Information Processing Systems, 34:13988- 14000, 2021.
[10] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattrrn Recognition, pages 8552-8562, 2022.
[11] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pages 2-25. PMLR, 2022.
[12] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[13] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audin pretraining with feature fusion and keyword- to-caption augmentation. arXiv preprint arXiv::211 1.06687, 2022.
[14) Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- shot video-text understanding. arXiv preprint arXiv:2109. 14084, 2021.
[15] Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2028-2038, 2023.
[16] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:1361:111 7612-11665 2022.
[17] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Marthn Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 1 1179-1189, 2023.
[18] Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martin-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip-2: Towards scalable multimodal pre-training for 3d understanding, arXiv preprint arXiv:230..08275, 2023.
11

---

[19] Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976-980. IEFE, 2022.
[20] Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4563-4567. IEEE, 2022.
[21] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade I Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017. [22] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020.
[23] Hao Zhu, Man-Di Luo,, Rui Wang, Ai-Hua Zheng, and Ran He. Deep audio-visual learning: A survey. International Journal of Automation and Computing, 18:351-376, 2021.
[24] Donghuo Zeng, Yanan Wang, Jianming Wu, and Kazushi Ikeda. Complete cross-triplet loss in label space for audio-visual cross-modal retrieval. arXiv preprint 0rXiv:2211.034333 2022.
[25] Donghuo Zeng, Yi Yu, and Keizo Oyama. Deep triplet neural networks with cluster-cca for audio- visual cross-modal retrieval. ACM Transactions on Multimedia Computing, Communica- tions, and Applications (TOMM), 16(3):1-23, 2020.
[26] Luís Vilaça, Yi Yu, and Paula Viana. Recent advances and challenges in deep audio-visual correlation learning. arXiv preprint arXiv:2202.13673 2022.
[27] Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, and Gedas Bertasius. Vision transformers
parameter-efficient audio-visual learners. arXiv preprini arXiv:2212.07988 2022.
[28] Yang Zhao, Chen Zhang. Haifeng Huang, Haoyuan Li, and Zhou Zhao Towards effective multi-modal interchanges in zero-resource sounding object localization. Advances in Neural Information Processing Systems, 35:38089-388 -002,, 2022.
[29] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin, Multiple sound sources localization from coarse to fine. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 292-308. Springer, 2020.
[30] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self- supervised learning of audio-visual objects from video. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 208-2244 Springer, 2020.
[31] Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9248 010189927,2222 2019.
[32] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358 4366, 2018
[33] Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self-supervised audiovisual matching. Advances in Neural Information Processing Systems, 33: 13:10077-10087 2020.
[34] Xian Liu, Rui Qian, Hang Zhou, Di Hu, Weiyao Lin, Ziwei Liu, Bolei Zhou, and Xiaowei Zhou. Visual sound localization in the wild by cross-modes interference erasing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1801-1809, 2022.
[35] Arda Senocak, Hyeonggon Ryu, Junsik Kim, and In So Kweon. Learning sound localization better from semantically similar samples. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4863-4861 IEEE, 2022.
[36] Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXXVII, pages 218-234. Springer, 2022.
12

---

[37] Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, Andrew Owens, and Tae-Hyun Oh. Sound to visual scene generation by audio-to-visual latent alignment. arXiv preprint arXiv:2303. 2023.
[38] Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. arXiv preprint arXiv:2212.09477. 2022.
[39] Zhaofeng Shi. A survey on audio synthesis and audio-visual multimodal processing. arXiv
preprint
2021.
[40] Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation. arXiv preprint arXiv: 2,000008, 2022.
[41] Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete contrastive diffusion for cross- music and image generation. In The Eleventh International Conference on Learning Representations.
[42] Kun Su, Xiulong Liu, and Eli Shlizerman. Audeo: Audio generation for a silent performance video. Advances in Neural Information Processing Systems, 10:3325-3337 2020.
[43] Kun Su, Xiulong Liu, and Eli Shlizerman. How does it sound? Advances in Neural Information Processing Systems, 34:29258-299 220233,,,2,,-2 2021.
[44] Yipin Zhou, Zhaowen Wang. Chen Fang, Trung Bui, and Tamara I. Berg. Visual to sound: Generating natural sound for videos in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3550-3558, 2018.
[45] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rie -ich 3d model repository. arXiv preprint arXiv:1512.0000
2015.
[46] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotale 3d reconstructions of indoor scenes. Proceedings of the IEEE conference computer vision and pailern recognition, pages 5828-5839, 2017.
[47] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1912-1920, 2015.
[48] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13 13142-13153, 2023.
[49] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202-221. Springer, 2020.
[50] Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 422-440. 440. Springer, 2020.
[51] Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. dowg-transformer: Relation modeling for visual grounding point clouds. In Proceedings of the IEEE/CVF International Conference 2021.
on Computer Vision, pages
[52] Zehan Wang, Haifeng Huang, Yang Zhao, Linjun Li, Xize Cheng, Yichen Zhu, Aoxiong Yin, and Zhou Zhao. 3drp-net: 3d relative position-aware network for 3d visual grounding. arXiv preprint arXiv:2307. 13366333..333,, 2023.
[53] Zehan Wang, Haifeng Huang, Yang Zhao, Linjun Li, Xize Cheng. Yichen Zhu, Aoxiong Yin, and Zhou Zhao. Distilling coarse-to to-fine semantic matching knowledge for weakly supervised 3d visual grounding. ICCV 2023, 2023.
[54] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 19199-19339, 2022.
13

---

[55] Xiaojian Ma. Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song- Chug- Zhu, and Siyuan Huang. Sqa3d: Situated question answering 3d scenes. arXiv preprint arX.4:2210.07474, 2022.
[56] Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data- efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprini arXiv:2308.08769, 2023.
[57] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. arXiv preprint arXiv:2307. 12981, 2023.
[58] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[59] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556 2565, 2018
[60] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288-5296, 2016.
[61] Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem Mad: A scalable dataset for language grounding in videos from movie audio descriptions. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 5026-5035, 2022.
[62] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generat- ing captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 119-132, 2019.
[63] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736-740. IEEE, 2020.
[64] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icee, 2009.
[65] Xumin Yu, Lulu Tang. Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert:
Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19313-19322, 2022.
[66] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 247-263, 2018
[67] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16867-16876 2021.
[68] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pages 570-586, 2018
[69] Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization. arXiv preprint arXiv:2209.094 096340964,777 2022.
[70] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Con- trast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. arXiv preprint 3it.:222222228.. 2023.
14

---

A Ablation Study about Text Dataset.
We conduct more experiments on audio-image retrieval with training texts from different sources. Furthermore, we provide insights about selecting training data when employing our C-MCR to connect other MCRs. As discussed in Sec 4.1, our training texts are collected from three sources:
image-text datasets (COCO [58] and CC3M [59 video-text datasets (MSRVTT 600 and MAD 61
and audio-text datasets (AudioCap 62) and Clotho
63 ).
In Table exclude the text data from image- text, video-text, and audio-text datasets, respec- tively. The results demonstrate that combining data from all three sources achieves the best per- formance. Furthermore, our findings suggest that the information in video-text datasets is rel- atively less important than in image-text and audio-text datasets. When applying our C-MCR to connect other MCRs, it critical to collect overlapping modality data associated with in- formation from non-overlapping modalities to ensure robust connections. The data from the pre-training datasets used by MCR could serve as an appropriate starting point. Combining the overlapping modality data from these sources ensures that the data used for constructing connections contains sufficient information from non-overlapping modalities. Additionally, this data easily accessible and scalable, which greatly enhances the practicality of our C-MCR.
Figure 4: Ablation studies about text datasets for training.
AVE
Flickr
Dataset
A2I
I2A
A21
I2A
Full
4.11
4.13
4.57
4.92
w/o image-text
3.83
3.76
3.97
3.91
w/o video-text
4.04
4.05
4.41
4.78
w/o audio-text
4.07
3.88
4.31
4.59
B Downstream Task Details.
B.1 Audio-Image Retrieval.
We consider two datasets for this task: AVE 666 and Flickr-SoundNet 32 both of which consist of semantically matched image and audio pairs that were manually curated. To more comprehensively and stably reflect the retrieval capability of the model, we use all available data in these two datasets for evaluation, resulting in 4,095 samples for AVE and 5,000 samples for Flickr-SoundNet.
B.2 Audio-Visuaa Source Localization.
We conduct experiments the VGGSS [67 and MUSIC (68 datasets. VGGSS is derived from VGGSound, and its test set comprises 5,158 audio-image pairs. MUSIC consists of 489 untrimmed videos of musical solos spanning 11 instrument categories for testing. It is worth noting that the category names from the COCO dataset as prompts to enable the open-vocabulary object detector GLIP 3 to extract object proposals.
B.3 Counterfactual Audio-Image Recognition.
The Extended TTickr-SoundNet 669 and Extended VGGSS 669 are constructed by adding 250 and 5,158 negative samples to the test sets of the original Flickr-Suundeet and VGGSS datasets, respectively. The prompts used for the object detector GLIP S are also the category names from the COCO dataset. We evaluate the counterfactual Audio-Image Recognition performance using the Maximum F1 (Max-FI) and Average Precision (AP) metrics, following [69]. During inference, for the i-th image-audio pair, the proposal with the highest matching score with the audio is considered the predicted object, and its matching score is considered the confidence score c, The CloU of the predicted object denoted as ToU;, The ground-truth map is denoted Gi, and the ground-truth maps of negative samples are 0. Under these definitions, the true positives TP, false positives FP, and false negatives FN are computed as:
TP(y, 6) = {ig., + 0, LoU, > 244 > 5}
FP(Y, = [i] * 0, IoU, 27.4 8) U (ilg, 0. > o
(11)
FN(,,8) = (19 + 0,c < 8)
15

---

where is the threshold of IoU and ő is the threshold of confidence score. Following previous work, the r 0.5. The Fl score can be represented as:
2 Precision(7; 6) Recall(), Precision(7; 6) + Recall(y, 6)
F1(7,6)
(12)
where
ITP 8)1
(7 8)1 (7P( 8)1 + (F,(7,
Precision(7,0) (7P 6)1 (P((,,
Recall(7:
(13)
In accordance with [69]. calculate Fl scores for all values of å and report the maximum F1 score (Max-F1). Average Precision (AP) is another commonly used metric object detection, its computation is detailed in [58 69
C Model Configurations.
Table 6: Model configurations of projectors.
Module
Block
Cin
Cout
Linear
512
1024
BatchNorm ID
1024
1024
Relu
Projectorl
Linear
1024
512
BatchNorm
512
512
Relu
Linear
512
1024
BatchNorm
1024
1024
Relu
Projector2
Linear
1024
512
BatchNorm
512
512
Relu
The model configurations of our projectors are shown in Table 6
D Limitations and Future Work.
While C-MCR offers an efficient and effective contrastive representation learning method for modal- ities that lack high-quality, large-scale paired data, it still necessitates an intermediate modality to associate these modalities. Exploring ways to reduce data requirements further while maintaining representation performance is an intriguing direction for future research.
E
Social Impacts.
Although C-MCR achieves outstanding performance audio-visual learning by connecting CLIP and CLAP, further analysis of the capability boundary of this representation is necessary before applying it to additional modalities or deploying it in practice. C-MCR only requires unpaired unimodal data during training, significantly reducing the data requirements for learning generalizable representation. However, this also means that unsuitable and harmful data in each modality are more likely be used for training.
16


Reviews:
[{"review_id": "b2NJV6io2p", "reviewer": "Reviewer_MbUd", "summary": "This paper focuses on extending existing Multimodal Contrastive Representation (MCR) to more modalities without massive high-quality data pairs. To this end, they propose a training-efficient method for learning a new MCR without paired data. Specifical, they connect one existing MCR pre-trained on modality $(A, B)$ and one existing MCR pre-trained on modality $(B, C)$, to obtain the new MCR pre-trained on modality $(A, C)$. As a result, they connect the pre-trained CLIP and CLAD models to derive audio-visual data, which achieves significant retrieval performances on AVE and Flickr-SoundNet. ", "questions": "As shown in Weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\t**The writing is clear and well-motivated.** It is necessary to extend multimodal contrastive representation to more modalities without massive high-quality data pairs. \n\n2.\t**Technically sound and easy to understand.** The semantic-enhanced inter- and intra-MCR connection method is flexible and training-efficient.\n\n3.    **The results are impressive.** Without any paired audio-visual data, they achieve state-of-the-art performance on six datasets. More specifically, Tab.2 and Tab.3 show that C-MCR surpasses existing methods by a large margin. ", "weaknesses": "1.  **The main results are not sufficient**.  The current experimental results are only experiments on the visual-audio task. This is not enough to support the claim of connecting multi-modal contrastive representation.  This article should provide more modality alignment experiments, e.g. (3D point, image)->(text, image), (text, image)->(image, sketch/RGB-D).\n\n2.  **The comparison experiments are not fair**. As shown in Table 2 and Table 3, a column should be added to the table to discuss which pre-trained models are used. It is unfair to only compare performance with different pre-trained model usage settings.\n\n3.   **Lack of thorough analysis of the proposed method**. The inter-MCR and intra-MCR are needed for visualizing by T-SNE to understand the mechanism of modules. In addition, the intra-variance and inter-variance between different modalities should be measured w or w/o inter/intra-MCR.\n\n4.  **The proposed method is trivial**. To some extent, there have existed similar methods [1][2][3][4] that align the semantic-enhanced embeddings across different modality spaces. Can you provide a more thorough comparison to illustrate the novelty of your work?\n\n\n[1] Text-only Training for Image Captioning using Noise-injected CLIP. EMNLP 2023.\n\n[2] I Can't Believe There's No Images! Learning Visual Tasks Using Only Language Data. Arxiv 2022.\n\n[3] From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping. IJCAI 2023.\n\n[4] DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training. ICLR 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: The main results are not sufficient:**\n\nPlease refer to “response to all reviewers” for the experiment of learning 3D-text representation via image-text and image-3D pre-trained model.\n\n**W2: Unfair comparison experiments:**\n\nThank you for your suggestion. We will add a column in Table 2 and 3 to discuss the used pre-trained method.\n\nBut we should clarify that the results presented in Table 2 and 3 primarily aim to evaluate the alignment between fine-grained visual objects and audio of audio-visual representations. The comparison with previous sound source localization or counterfactual audio-image recognition methods only serves as a baseline reference, the focus is the comparison between other audio-visual representations such as AudioCLIP and WAV2CLIP.\n\nWe employed these three audio-visual representations (AudioCLIP, WAV2CLIP, and our C-MCR) for zero-shot audio-visual source localization and counterfactual audio-image recognition tasks via the same object detector and pipeline. Under identical settings, our audio-visual representations outperform AudioCLIP and WAV2CLIP on all four datasets across two tasks. Additionally, it is worth noting that AudioCLIP and WAV2CLIP exhibit significant performance fluctuations across different datasets, indicating a lack of generalization. In contrast, our method consistently demonstrates advanced performance across all four datasets.\n\n**W3: Lack of thorough analysis of the proposed method:**\n\nSome statistical metrics, as well as T-SNE visualization results, with and without inter/intra-MCR, are provided in the newly submitted PDF files.\n\nIn the original CLIP and CLAP spaces ( Figure (a) ), there exists a modality gap for both CLIP and CLAP, and the text embeddings between CLIP and CLAP are not aligned (mAP retrieval result for 1000 COCO captions is 0.14)\n\nIn the learned CMCR space ( Figure (b) ), the modality gap between different modalities is effectively eliminated, and there is significant semantic alignment among text embeddings (mAP retrieval result is 10.41)\n\nIn the CMCR space without the intra-MCR loss ( Figure (c) ), alignment is also observed between text embeddings (mAP retrieval result is 9.84). However, the modality gap persists, which poses challenges in maintaining the learned robust connection for image-audio inputs.\n\nIn the CMCR space without the inter-MCR loss ( Figure (d) ), the remaining intra-MCR loss solely focuses on pulling positive instances closer, leading to the model collapse and all inputs being mapped to the same representation.\n\n**W4: The proposed method is trivial.**\n\nOur approach differs significantly from these methods in terms of task definition, overall methods, and specific approaches for addressing the modality gap.\n\nIn terms of task definition, the mentioned papers focus on learning text generation downstream tasks using text inputs and evaluating with visual input, while our approach aims to construct a contrastive representation for novel modalities by connecting two existing MCR spaces. Our method focuses on the representation learning problem rather than a specific downstream task. This fundamental difference sets our problem setting apart from prior works. Additionally, the new experiments involving 3D-text further highlight that our approach fundamentally revolves around representation learning rather than specific downstream tasks.\n\nRegarding overall methods, although the mentioned works also discuss the modality gap in contrastive representations, their methods only aim to address the gap in a single MCR space. In contrast, our approach emphasizes building a connection between two MCR spaces while considering the modalities gaps within both spaces. Our method proposes an effective semantic enhancement approach and facilitates cross-space and cross-modal alignment, distinguishing our work from these papers.\n\nEven only considering the specific methods for addressing the modality gap, our proposed solution stands out from previous approaches. In [1][2], random noise is added to text representations, aiming to eliminate the modality gap through the non-directional offset introduced by these noises. But the offset between the modality gap is directional, the non-directional offset is unreliable for the modality gap. On the other hand, [3][4] maintain a set of text embeddings during testing, retrieving semantically related embeddings from a text bank as input of downstream head, which inevitably incurs additional inference costs. In contrast, We design an intra-MCR alignment loss by analyzing the main reason for the modality gap (e.g. the “pulling apart” term in the contrastive learning loss). Our method can directionally close the modality gap without introducing extra inference costs. Our unique approach to the modality gap problem differs from these works in terms of motivations, detailed algorithms, and advantages."}, {"review_id": "jPrKRzA1Ph", "reviewer": "Reviewer_5S9J", "summary": "- This paper proposes C-MCR, a multimodal contrastive framework that aligns multimodal representations of overlapping modality pairs to learn effective representations of the non-overlapping modality pairs. Specifically, the authors concentrate on learning the inherent semantics of visual-audio data solely through language-visual and language-audio pairs. The authors developed lightweight projectors, semantic enhancement techniques, and inter-intra-modality alignment objectives. Experiments are well designed on various downstream tasks in a zero-shot manner for C-MCR with different metrics, and C-MCR demonstrates superior performance compared to the chosen baselines. The authors further conduct ablation studies on the proposed components to evaluate the effectiveness of C-MCR.\n\n---------------------------\n\nMy score is updated after reading the authors's rebuttal and the reviews from my peer reviewers.", "questions": "- The current method is focusing on visual audio with text. Is it possible to generalize it with different or more modalities? In this case, there could be multiple overlapping pairs, rather than just two modality pair with one modality as the overlapped anchor.\n- Section 3.4 “there are (x_i - y_i)^T……”. The wording could be clarified more\n- Section 4.3 Audio-Visual Source Localization “score is token as the final prediction”, the word token causes some confusion", "limitations": "- The authors could provide a more comprehensive discussion on the limitations of the proposed method and future works. For example, they could explore the possibilities of extending the method to accommodate additional modalities and the application in other domains. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation to connect modality representations and reduce reliance on high-quality modality pairs is strong, and the proposed method is straightforward and clearly described.\n- The authors have presented illustrative figures to explain their framework\n- The proposed method demonstrates notable performance improvements over other baselines.\n- Ablation studies are well-defined and provide clear insights into the effectiveness of the proposed modules", "weaknesses": "- Figure 1 is highly informative, but it may cause confusion when simply referred to as “As illustrated in Figure 1”. Adding more detailed text or subsection numbers to label the components would enhance the clarity for the readers.\n- It would be more promising if the approach section could be generalized for additional modalities to demonstrate the extensibility and significance of the proposed methods in various applications involving multimodal data. \n- Related works could also include additional multimodal contrastive learning frameworks [a1, a2]\n- The proposed modality pair alignment is very interesting, however, the idea of projectors and contrasting components has already been explored in the past.\n\n[a1] \"Contrastive multiview coding.\" Computer Vision–ECCV 2020\n\n[a2] \"Geometric multimodal contrastive representation learning.\" International Conference on Machine Learning. PMLR, 2022.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Too informative Figure 1**\n\nThank you for your constructive suggestion. We will add subsection numbers to the subfigures in Figure 1 and make the necessary adjustments to the reference statements in the main text accordingly.\n\n**W2 & Q1: Generalize our approach to different and more modalities.**\n\nFor generalization to different modalities, please refer to “response to all reviewers” for the experiment that employs our methods in the 3D-text field. Briefly, our method also shows advanced performance in these modalities.\n\nAs for applying our method to three or more MCR modalities, due to time constraints, we regret that we were unable to conduct an evaluation experiment on applying our method to three or more MCR modalities. Such an evaluation experiment requires collecting multiple pre-trained models for different modalities, datasets for each modality, appropriate validation tasks, validation datasets, and comparable baselines for evaluating different modality combinations. However, in theory, connecting multiple modalities only requires constructing semantically enhanced embeddings through each overlapping modality anchor and simultaneously optimizing inter-MCR losses for each overlapping modality and intra-MCR losses within each MCR. Several studies [1][2][3] have demonstrated that simultaneous optimization of multiple contrastive learning losses can lead to a unified representation space. Given the successes of C-MCR in audio-visual and text-3D fields, as well as the successful examples of simultaneously optimizing multiple contrastive learning losses, our approach is promising for generalizing to more modalities.\n\n[1] ULIP-2: Towards Scalable Multimodal Pre-training For 3D Understanding. arxiv 2023\n\n[2] Any-to-Any Generation via Composable Diffusion. arxiv 2023\n\n[3] Imagebind: One embedding space to bind them all. CVPR 2023\n\n**W3: Related works:**\n\nThank you for your valuable suggestion. We will add these references to our related works.\n\n**W4: the idea of projectors and contrasting components:**\n\nTo the best of our knowledge, we are the first to propose learning contrastive representation by connecting existing contrastive representations, rather than training on large-scale data pairs. Moreover, the role of a projector in our method is a highly parameter-efficient structure for mapping representations. Our method is a very parameter-efficient learning method that requires extremely low training cost, which is also first studied in multi-modal contrastive representation learning. Please refer to the \"Response to all reviewers\" section for more analysis of the training efficiency and flexibility of our method.\n\nCould you kindly provide some references about “the idea of projectors and contrasting components” for further discussion?\n\n**Q2 The wording in Section 3.4 could be clarified more:**\n\nThanks for your advice, we will revise this in a further version.\n\n**Q3 word “token” may cause confusion:**\n\nThanks for your reminder, We will modify it to“score is selected as the final prediction”\n\n**Limitations and Future Work:**\n\nWe will add a discussion about connecting more than three modalities simultaneously and the potential of assembling knowledge from different MCRs in limitations and future Work in future versions."}, {"review_id": "ePxUmgM2tx", "reviewer": "Reviewer_XBwt", "summary": "This paper presents a Connecting Multi-modal Contrastive Representations (C-MCR) scheme that aligns the spaces of two already pre-trained MCR models by utilizing the overlapping modality between two, in a data- and training-efficient way. One interesting property of C-MCR is that aligning those two MCR spaces are feasible without the modality pairs to be aligned (e.g., image and audio) during training. To that end, two alignment objectives named Inter-MCR and Intra-MCR alignment and the Semantic Enhancement strategy are designed, while addressing the robust alignment and modality-gap issues. The Semantic Enhancement mechanism that consists of Inter-modality Semantic Consistency and Semantic Completion enables robust and comprehensive alignment. In addition, the two proposed objectives are the key to the semantic-enhanced inter- and intra-MCR connection in the C-MCR framework. As an instantiation of the C-MCR framework, the audio-visual-aligned contrastive representation is obtained using CLIP and CLAP models, and its effectiveness is verified by a series of experiments on various audio-visual tasks in a zero-shot manner. \n", "questions": "- The equations for the contrastive loss functions and the Softmax of similarity terms are too redundant. The only variations are the input and temperature parameters. To improve clarity, it is suggested to define the softmax of similarity terms and the form of contrastive losses as new functions and clean the following equations.\n\n- If time permits, I hope to see whether the proposed C-MCR method can be generalized to align the MCR representations between 3d point clouds and language via 2d image modality (as explained in the introduction section).", "limitations": "Limitations and potential negative societal impact are addressed.\n", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The motivation for the C-MCR is clear and strong. Achieving the aligned MCR spaces without using the explicit modality pairs which can be expensive in some situations seems to be a clear advantage.\n\n- The idea for aligning the MCR spaces including semantic completion mechanism and MCR alignment objectives is simple and straightforward, while it seems to be effective even lack of direct audio-visual pairs during training.\n\n- The presentation of the work is quite clear and the paper is generally well written. The codes are included in the submission.", "weaknesses": "- It would be desirable to make analysis on whether the two projectors trained to indirectly align audio-visual modalities via the proposed C-MCR scheme can also improve the alignment between image and language / audio and language compared to simple CLIP or CLAP. If it could, such discovery can strengthen the contribution of this work.\n\n- In this work, connecting only two MCR representations is explored. Discussion on whether the proposed scheme is seamlessly applied to aligning three or more MCR modalities would be helpful.\n\n- Sensitivity analysis on some important hyper-parameters (e.g., temperature scaler $\\tau$ and loss weight $\\lambda$) is missing. How those sets of hyper-parameters are determined?\n\n- For the projection layers, two layer MLPs with activations are adopted. Is the alignment ability affected by the depth of the projection layer?  Would like to see the performance changes when using single-layer MLP or MLPs with more than two.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 Whether trained projectors can improve the alignment between image and language/audio and language:**\n\nAs depicted in the visualization provided in the newly submitted PDF files, the original CLIP and CLAP space (Figure (a) ) exhibits a noticeable modality gap. After remapping by the trained projectors, the modality gap within CLIP and CLAP significantly diminishes (Figure (a) ). The L2 distance between different modalities reduces from 1.71 (CLIP), 1.99 (CLAP) to 0.53 (CLIP), 0.42 (CLAP). Closing the modality gap enables the more stable and robust connection learned from text-text to be effectively inherited by image-audio inputs.\n\n**W2 & Q2 Discussion on employing the proposed scheme to 3D-Language or more modalities**\n\nPlease refer to “response to all reviewers” for a detailed comparison and analysis of the experiment on the 3D-language field. Our method also achieves advanced performance in this field.\n\nAs for applying our method to three or more MCR modalities, due to time constraints, we are unable to conduct this evaluation experiment. Such an evaluation would require collecting multiple pre-trained models for different modalities, datasets for each modality, as well as appropriate validation tasks, validation datasets, and comparable baselines for evaluating different modality combinations. However, in theory, connecting multiple modalities only requires constructing semantically enhanced embeddings through each overlapping modality anchor and simultaneously optimizing inter-MCR losses for each overlapping modality and intra-MCR losses within each MCR. Several studies [1][2][3] have demonstrated that simultaneous optimization of multiple contrastive learning losses can lead to a unified representation space. Given the successes of C-MCR in audio-visual and text-3D representations, as well as the successful examples of simultaneously optimizing multiple contrastive learning losses, our approach is promising for generalizing to more modalities. \n\n[1] ULIP-2: Towards Scalable Multimodal Pre-training For 3D Understanding. arxiv 2023\n\n[2] Any-to-Any Generation via Composable Diffusion. arxiv 2023\n\n[3] Imagebind: One embedding space to bind them all. CVPR 2023\n\n**W3 Sensitivity analysis and selection of hyper-parameters.**\n\nThe temperature scaler is set to 1/100 which is the final learned temperature scaler in CLIP. The loss weight of 0.1 is set to ensure that the absolute values of the two losses remain comparable. Here are some sensitivity analyses of these two hyper-parameters.\n\n|  Temperature | AVE   |   Flickr  |\n| :------------: | :------------: | :------------: |\n|  1/50  |   4.11  |    4.64  |\n|  1/75  |   4.18   |   4.84  |\n|   1/100 |   4.11 |     4.57   |\n|  1/125  |  4.11  |    4.53   |\n|  1/150  |  4.08     | 4.56 |\n\n|  Loss factor  | AVE  | Flickr  |\n| :------------: | :------------: | :------------: |\n|  0.05    |  4.07    |  4.68  |\n|  0.10   |   4.11   |   4.57   |\n|  0.20   |   4.08    |  4.68  |\n|  0.30 |     4.17   |   4.96  |\n|  0.40  |    4.09   |   4.85  |\n\nOverall, our method is not sensitive to temperature scaler and loss weight. By carefully tuning these two hyper-parameters, the retrieve performance on AVE and Flickr-SoundNet even further improves from 4.11, 4.57 to 4.17, 4.96.\n\n**W4 Effect about the number of layers in MLP:**\n\nHere is the ablation study about the layer number of the MLP.\n\n|  Layer  | AVE  | Flickr  |\n| :------------: | :------------: | :------------: |\n|1-layer | 3.84  |    4.47 |\n|2-layer  |4.11   |   4.57 |\n|3-layer | 4.20  |   4.77 |\n|4-layer  |4.28  |   4.74 |\n|5-layer | 4.26   |   4.60 |\n\nOverall, our scheme is also not sensitive to the layer number of the MLP. By carefully tuning the number of linear layers in the MLP, the retrieve performance on AVE and Flickr-SoundNet further improve from 4.11, 4.57 to 4.28, 4.74.\n\n**Q1 Too redundant equation:**\n\nThanks for your constructive advice, we will revise it in the future version."}, {"review_id": "y1IXhT4BOj", "reviewer": "Reviewer_Szdi", "summary": "This paper introduced a novel technique to learn multimodal contrastive representations (MCR) without paired data from the two modalities by using a third modality to bridge between the two existing modalities. The technique assumes the existence of a learned MCR between each existing modality and the third modality, and the proposed approach brings the 2 separate MCRs together by a learned projection layer trained with both inter and intra-MCR alignments. The paper used this technique to learn an MCR between audio and images, with only unpaired audio, image and text data and existing MCR between image and text (CLIP) and between audio and text (CLAP). Then the learned MCR between audio and image was used to zero-shot on 3 audio-visual tasks: audio-image retrieval, source-localization, and counterfactual audio-image recognition. The new method is able to achieve state-of-the-art performance on all 3 tasks zero-shot, out-performing MCRs learned directly with audio-image paired data. ", "questions": "The descriptions in the semantic consistency ablation studies seems wrong. (Line 300-301) do you mean Row K is significantly better than Row I and J? Also, what exactly does Row J mean (with an X in consistency)? \n\nI also have a hard time understanding the intuition behind using a softmax for semantic consistency instead of argmax (which I assume means selecting the closest example).  Why does constructing a representation of an non-existent image/audio using softmax yield better performance? ", "limitations": "The authors did not address the limitations of the proposed method. The paper could be improved if the author discusses some limitations of their approach (such as applicability to different modality settings).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.The paper studies an important and challenging problem: learning a multimodal contrastive representation between two modalities without direct paired data.\n\n2. The proposed approach is generally quite straightforward and easy to understand. The proposed approach also only adds a very small amount of additional parameters in the projection module, and requires limited additional training.\n\n3. The proposed approach achieved state-of-the-art performance on three different audio-visual tasks zero-shot, outperforming MCRs learned directly with paired audio-visual data.\n\n4. The paper conducted a comprehensive ablation study that examines and justifies every design choice in the methodology.", "weaknesses": "Although the proposed approach does not require paired data, it does require two pre-trained that happens to bridge each of the target modalities into a third one. While this happens to be the case for audio and image modalities (where CLIP and CLAP bridges both to text), it is unclear whether this approach could be applied to a wider range of modality settings, as all experiments in this paper focused on audio-visual MCR learning.\n\nThe experiments also only included cross-modal audio-visual tasks. It would also be interesting to evaluate how good are the learned audio-visual MCR on unimodal image or audio tasks (such as image classification), since MCR like CLIP had shown great performance as unimodal image representation in addition to cross-modal performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 Apply the proposed methods to other modality settings:**\n\nPlease refer to “response to all reviewers” for a detailed comparison and analysis of the experiment on the 3D-text field. Briefly, our method also achieves state-of-the-art performance on the 3D-text field.\n\n**W2 Performance of the learned audio-visual MCR on unimodal tasks:**\n\nWe should clarify that our method does not aim to learn stronger unimodal representation. We perform image classification on the imagenet1K dataset using the visual representation in CMCR and CLIP respectively. In our implementation, we only train a linear layer to map visual representations to classification predictions. The CLIP representations attained a top-1 accuracy of 73.1% on the ImageNet1K validation set, while our CMCR representations achieved a top-1 accuracy of 72.6%.\n\nOverall, the image classification performance of CMCR closely approximates that of the original CLIP representation. Actually, the visual representation of CMCR is just a remapping of the pre-trained CLIP representation. During the process of learning CMCR, the visual encoder would not be trained. Simple remapping of the final representation hardly enhances the ability to capture visual patterns.\n\n**Q1 Question about the Row I, J, and K in Table 4**\n\nThank you for your reminder. We apologize for the typo in lines 300-301. The correct expression should be: \"Compared to Row J, the significantly better results in Rows I 301 and K highlight......”. Row J (with an X in consistency) means randomly selecting embedding from image/audio memory to calculate intra- and inter-MCR losses, resulting in semantic-inconsistent audio and visual embeddings.\n\n**Q2 Why softmax is better than argmax:**\n\nThe softmax function can be viewed as a soft information aggregation method. Using argmax to select samples involves discrete sampling within subspaces, while using softmax involves continuous sampling within subspaces. Consequently, employing softmax for interpolating the embeddings within the memory provides semantically diverse and accurate embeddings, akin to the “Mixup” technology widely used in data augmentation to enhance the diversity and accuracy of representations."}, {"review_id": "1fL29EvUvs", "reviewer": "Reviewer_3nbF", "summary": "The SOTA approach in multi-modal learning employs contrastive loss to project embeddings from different modalities into a unified embedding space. However, models trained on different modal pairs result in disparate embedding spaces. This paper introduces a novel method to align distinct multi-modal embedding spaces by leveraging the overlapping modality they share. Additionally, it proposes several techniques to ensure a more robust alignment in the aligned embedding space. The experimental results presented in the paper demonstrate impressive performance across various tasks.", "questions": "1. Could CLIP and CLAP be used to label image-audio pairs (find the image-audio pair most similar to the text embeddings, but reject the ones with distances larger than a proper threshold), using the dataset used in the proposed method, and then train an image-audio (or image-audio-text) version of CLIP? It seems that the proposed method essentially performs a similar task without explicitly creating a dataset. Inter-modality Semantic Consistency is a way to soft label noise, the step of adding Gaussian noise can be seen as a form of data augmentation, and the inter/intra alignment losses can be viewed as a type of adaptation of CLIP to three modalities.\n2. Is there a specific reason for keeping the text encoder frozen during training? Would using a small learning rate for the text encoder be beneficial in any way?\n3. The ablation studies indicate that softmax outperforms argmax in Consistency, but the paper doesn't explain why this improvement is related to better inter-modality semantic consistency. Is it possible that softmax serves as a form of label noise regularization (as mentioned in question 1)? Additionally, it would be helpful to include other baselines, such as the mean of the top 10 similar embeddings.\n4. The baseline models were trained using different datasets compared to the proposed method, which makes it difficult to convincingly demonstrate that the proposed method outperforms the baselines. I suggest creating an audio-image dataset using pretrained CLIP and CLAP (as mentioned in question 1), and then training the baselines on this dataset. This would ensure that both the baselines and the proposed model have access to the same data.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: This paper introduces a novel extension to the existing CLIP approach, enhancing its capabilities by aligning distinct embedding spaces. It incorporates innovative ideas to achieve this alignment objective, contributing to the advancement of the field.\nClarity: The paper is well-structured and written in a clear style.", "weaknesses": "Necessity of Innovative Ideas: Although the paper introduces several innovative ideas, it would benefit from stronger evidence to support their necessity. Specifically, more clarification is needed regarding the significance of aligning the shared embedding space. What advantages does this method offer over labeling a dataset using the pretrained models utilized in this paper?\nLack of Experiment Descriptions: The experiment section would benefit from more detailed descriptions regarding the evaluation tasks and datasets used. \nAdditional Baselines: To establish the significance of the proposed method, it is important to include appropriate baselines trained with the same amount of data as the proposed method in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Clarification of our core innovation and the real reason our method works:\n\nThe really effective part of our method is not constructing pseudo audio-image pairs using CLIP and CLAP. Instead, our methods focus on learning the alignment between text representations from CLIP and CLAP in a new space. Consequently, due to the intrinsic alignment of image-text and text-audio within CLIP and CLAP, the alignment learned from overlapping modality (e.g. text in the audio-visual setting) can be inherited by non-overlapping modality inputs (e.g. audio and visual inputs).\n\nTheoretically, the text encoders in CLIP and CLAP may introduce individual biases when encoding the same text （For example, encoding the same sentence \"a dog is barking\", CLIP focuses on the visual semantics of \"dog\", while CLAP focuses on the audio semantics of \"barking\".）Additionally, aggregating information from memory inevitably faces “semantic absence problem”(For example, for a text--\"a mushroom\", model can easily find a picture of a mushroom, but it is difficult to find audio related to“a mushroom”). It is hard to guarantee all the query data has corresponding semantics on different modalities. The accumulation of these errors diminishes the semantic coherence between the generated pseudo audio-image pairs, making it less reliable.\n\nOur ablation experiments provide strong evidence that pseudo audio-image pairs are not the key reason for our method's effectiveness. In row E of Table 4, when we removed the alignment between pseudo audio-image embeddings pairs from the inter-MCR loss, there is only a very slight performance decline (4.11, 4.13, 4.57, 4.92 → 4.09, 4.10, 4.42, 4.54). However, in row D, when we removed the alignment term between text embeddings from different MCR spaces in the inter-MCR loss, there was a significant performance drop (4.11, 4.13, 4.57, 4.92 → 3.28, 3.30, 3.69, 3.50). This verifies that the alignment learned from text-text pairs is the key to the effectiveness of our method, while the generated pseudo audio-visual pairs are merely supplementary.\n\nIn the 3D-text experiments, not aligning the pseudo 3D-text pairs even results in better performance (top1, top3, top5 accuracy on Objaverse and ModelNet dataset: without aligning pseudo 3D-text: 22.6, 39.3, 47.5, 64.9, 85.0, 92.8; Full method: 19.2, 34.8, 42.9, 64.1, 83.8, 90.5). On the contrary, not aligning the representations of overlapping modality results in a severe performance decline (15.4, 27.8, 35.9, 56.1, 72.6, 79.7). These results once again confirm that the connections learned from the overlapping modality are more crucial than pseudo-data pairs.\n\n**W1 The necessity of connecting MCR:**\n\nCompared to learning or extending MCR from paired data (collected or labeled with pre-trained models), connecting MCR has the following advantages:\n\n1. Connecting MCR enables learning multi-modal contrastive representations without large-scale and high-quality data pairs.\n\n2. Compared with using a pre-trained model to label a paired dataset: As discussed in the “Clarification”, utilizing pre-trained models to label data suffer serve cumulative errors, resulting in less reliable semantic coherence. Our ablation study also shows the generated pseudo audio-image pair does contribute little to the final performance. Further, we conduct the experiment using \"argmax\" (namely label image-audio pair) to retrieve a “paired image-audio dataset” and only rely on pseudo image-audio pair to learn the alignment. The corresponding result on AVE and FlickrNet are 2.74, 2.41, 3.11, 2.97, much lower than our current methods.\n\n3. Training efficiency and flexibility of C-MCR： More importantly, connecting MCRs only requires an extremely low training cost to learn contrastive representations between novel modalities. The low training cost also brings great flexibility to our method. Please refer to “Response to All Reviewers” for detailed discussions.\n\n**W2 descriptions regarding the evaluation tasks and datasets used:**\n\nIn addition to section 4.3 of the main text, we also provide more information about the datasets used in downstream tasks and the detailed calculation process of some complex metrics in section B of the appendix. Thank you for your suggestion, we will add more details in the future version.\n\n**Q1.1 Using CLIP and CLAP to label image-audio pairs and use them to train an image-audio version of CLIP:**\n\nPlease refer to \"clarification\" and \"W1\".\n\n**Q1.2 & Q2 It seems that the proposed method essentially performs a similar task without explicitly creating a dataset & Why keep the text encoder frozen?**\n\nAs mentioned in \"clarification\", the real reason for our advanced performance is not using CLIP and CLAP to label pseudo audio-image pairs. Instead, it is learning the alignment between the text of CLIP and CLAP in a new space, and the image-audio representations can be indirectly aligned due to the intrinsic image-text and text-audio alignment in CLIP and CLAP space. Therefore, once the text encoder is learnable, the alignment of image-text and text-audio in CLIP and CLAP will be destroyed. Then, the stable alignment learned from the text cannot be maintained for image-audio input.\n\n**Q3 The softmax in consistency**\n\nThank you for your reminder. “softmax” flexibly aggregates embeddings in memory and generate more accurate and diverse consistent representations, which indeed can be understood as a kind of label noise regularization.\n\nAlthough we think that the pseudo audio-image is not the key point in our method as discussed in \"Clarification\" and Q1, we still provide experiments that take the mean of the top 10 similar embeddings as consistent embeddings. The results are 4.08, 4.05, 4.56, 4.82, which is between the results using \"softmax\" and \"argmax\".\n\n**Q4 baseline trained on dataset labeled by CLIP and CLAP**\n\nPlease refer to “Clarification” and W1."}]

Author rebuttal:
# Response to all reviewers

**1 Experiment of 3D-text**

1.1 Performance comparison:

By connecting text-image representations of CLIP and image-3D representation of ULIP 2 (Objaverse), we successfully learned a contrastive 3D-text representation without explicitly using any 3D-text data. We evaluated the performance of our 3D-text representation on two  3D object classification datasets: ModelNet40 and Objaverse-LVIS. Here is the comparison of our CMCR (3D-text) with previous 3D-text representations:

Results on Objaverse-LVIS:

|   |   |Top1 |    Top3 |    Top5 |
| :------------: | :------------: | :------------: | :------------: | :------------: |
| ReCon [1] |  ICML2023  |   1.1 | 2.7 | 3.7 |
| CG3D [2] |  arxiv2023  |   5.0 | 9.5 | 11.6 |
|  ULIP [3] |   CVPR2023  |  6.2 | 13.6 |17.9  |
|  ULIP2 (objaverse) [4] | arxiv2023  |  6.3 | 11.9 | 34.5 |
|  C-MCR (3D-text) | - |  **22.6** | **39.3** | **47.5** |

Results on ModelNet40:

|   |   |Top1 |    Top3 |    Top5 |
| :------------: | :------------: | :------------: | :------------: | :------------: |
| ReCon [1] |  ICML2023  |   61.2 | 73.9 | 78.1 |
| CG3D [2] |  arxiv2023  |   48.7 | 60.7 | 66.5 |
|  ULIP [3] |   CVPR2023  | 60.4 | 79 | 84.4  |
|  ULIP2 (objaverse) [4] | arxiv2023  |  **74.0** |  86.5 | 90 |
|  C-MCR (3D-text) | - |  64.9 | **87.0** | **92.8** |

Overall, our C-MCR (3D-text) achieves advanced performance on ModelNet40 in a zero-shot setting and demonstrates significantly better performance on the more challenging and long-tail Objaverse-LVIS dataset.

The most important comparison in this table is the comparison between our C-MCR (3D-text) and ULIP 2 (Objaverse). ULIP 2 constructs image-3D-text triplets on the Objaverse dataset to learn image-text-3D representations, where the learned text-3D representation is used for classification. Our C-MCR only utilized the image and 3D encoders from ULIP 2, and by connecting them with CLIP, we obtained a new 3D-text representation. Remarkably, without using any 3D-text data, our C-MCR outperformed ULIP2, particularly on the more challenging and long-tailed Objaverse-LVIS dataset, which demands higher generalization performance.

[1] Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. ICML 2023

[2] Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. arxiv 2023

[3] Ulip: Learning unified representation of language, image and point cloud for 3d understanding. CVPR 2023

[4] ULIP-2: Towards Scalable Multimodal Pre-training For 3D Understanding. arxiv 2023


1.2 Implementation Details: 

We employ the 1.2M training images in ImageNet1K for training. The 3D object point clouds from the training set of Objaverse are utilized to construct memory (0.8M samples in total). For the text memory, we use the same dataset combination (2.3M) described in Section 4.1 of our main text.

1.3 Pre-training weight comparison： 

ULIP 2 uses the pre-trained BLIP-2 to label text for image-3D pairs, its visual and text encoder utilize an advanced version of CLIP, namely SLIP. ULIP 2 (Objaverse) is pre-trained on Objaverse 3D-image-text triplets.

Our C-MCR uses the pre-trained CLIP ViT-B/32 model, and the pre-trained visual and 3D encoders of ULIP 2 (Objaverse).

1.4 More discussion about the comparison between ULIP 2 and our C-MCR

Our C-MCR builds upon ULIP 2, even without using the pre-trained text encoder of ULIP 2，
there may be a concern that the 3D encoder might implicitly learn the knowledge in 3D-text data. From this perspective, our approach can be understood as learning stronger text representations by connecting CLIP to ULIP 2 (image-3D only), and using the newly obtained text representation to replace the original one derived from paired data. By connecting with CLIP via our method, the alignment of 3D-text in ULIP 2 is further boosted. This is a very intriguing and insightful discovery, highlighting the potential of our method to effectively reorganize knowledge of one contrastive representation and enhance its internal alignment under the guidance of another contrastive representation. Our method not only can connect two MCRs to construct contrastive representations between novel modalities, but also can assemble knowledge from different MCRs to enhance alignment in MCR which unified align three and more modalities.

**2 More discussion about the training efficiency and flexibility of C-MCR**

Connecting MCR requires an extremely low training cost. Contrastive learning is known for its huge training cost due to the large batch size required, while our method only needs to train two simple projector layers, and all the representations used can be pre-extracted and saved, thus achieving extremely low training costs. Taking learning 3D-text contrastive representations as an example, our method requires only 4G GPU memory and takes only 40 minutes for training on a single 3090 GPU. We believe that this paradigm of mining new knowledge from existing MCRs with minimal training cost can provide valuable inspiration for Parameter-Efficient Learning (PEL) in representation learning.

The extremely low training cost brings great flexibility. Any newly released stronger multi-modal contrastive pre-trained model and high-quality large-scale datasets can be easily used by our method without worrying about the training cost increase brought by the data and model scale. Combining this flexibility with the potential for creating new contrastive representations and assembling multiple MCRs (discussed in "the comparison between ULIP 2 and our C-MCR"), our approach holds significant practical value in integrating knowledge from various advanced MCRs.

Meta-review: This paper brought an active discussion. The latest results shared by the authors on the task of 3D-text representation helped emphasizing the novelty and performance of the proposed approach. The paper presents a novel approach to integrate representations from two bimodal models to allow the connection of two non-overlapping connections. The paper includes extensive set of experiments, visualizations analyses. The discussion allowed to clarify reviewer's concerns, with an agreement among reviewers that this paper is worth publication at Neurips. I support this assessment.
Decision: Accept (poster)