Extending Multi-modal Contrastive Representations
Authors: Ziang Zhang
Zehan Wang
Luping Liu
Rongjie Huang
Xize Cheng
Zhenhui Ye
Wang Lin
Huadai Liu
Haifeng Huang
Yang Zhao
Tao Jin
Siqi Zheng
Zhou Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-modal learning; Representation Learning
Primary area: other

Abstract: Multi-modal contrastive representation (MCR) of more than three modalities is critical in multi-modal learning. Although recent methods showcase impressive achievements, the high dependence on large-scale, high-quality paired data and the expensive training costs limit their further development. Inspired by recent C-MCR, this paper proposes $\textbf{Ex}$tending $\textbf{M}$ultimodal $\textbf{C}$ontrastive $\textbf{R}$epresentation (Ex-MCR), a training-efficient and paired-data-free method to build unified contrastive representation for many modalities. Since C-MCR is designed to learn a new latent space for the two non-overlapping modalities and projects them onto this space, a significant amount of information from their original spaces is lost in the projection process. To address this issue, Ex-MCR proposes to extend one modality's space into the other's, rather than mapping both modalities onto a completely new space. This method effectively preserves semantic alignment in the original space. Experimentally, we extend pre-trained audio-text and 3D-image representations to the existing vision-text space. Without using paired data, Ex-MCR achieves comparable performance to advanced methods on a series of audio-image-text and 3D-image-text tasks and achieves superior performance when used in parallel with data-driven methods. Moreover, semantic alignment also emerges between the extended modalities (e.g., audio and 3D).

Extending Multi-modal Contrastive Representations
Ziang Zhangit.2 Zehan Wang"1.2
Luping Liu'
Rongjie Huang'
Xize Cheng'
Zhenhui Ye!
Wang Lin'
Huadai Liu' Haifeng Huang!
Yang Zhao'
Tao Jin'
Siqi Zheng
Zhou 2
'Zhejiang University
Shanghai Al Laboratory 3 "Alibaba Group
{ziangzhang, wangzehan01 Jzjj.. edj..edu..ee.......... edu. cn
Abstract
Multi-modal contrastive representation (MCR) of more than three modalities is critical in multi-modal learning. Although recent methods showcase impressive achievements, the high dependence on large-scale, high-quality paired data and the expensive training costs limit their further development. Inspired by recent C- MCR, this paper proposes Extending Multimodal Contrastive Representation (Ex- MCR), craining-efficient and paired-data- dddaa-free method to build unified contrastive representation for many modalities. Since C-MCR is designed to learn new latent space for the two -on-overlapping modalities and projects them onto this space, a significant amount of information from their original spaces is lost in the projection process. To address this issue, Ex-MCR proposes to extend one modality's 's space into the other's, rather than mapping both modalities onto a completely new space. This method effectively preserves semantic alignment in the original space. Experimentally, we extend pre-trained audio-text and 3D- image representations to the existing image-text space. Without using paired data, Ex-MCR achieves comparable performance to advanced methods on a series mudio-image-tex and 3D 3D-image-tert tasks and achieves superior performance when used in parallel with data-driven methods. Morcover, semantic alignment also emerges between the extended modalities (c.g., audio and 3D). Our project page is available at https://gtthub.com/ PCC-PEEEEEPPPECCCEEEEE
Introduction
Multi-modal Contrastive Representation (MCR) learning aims to align inputs from diverse modalities within shared representation space. Recently, the high-quality contrastive representations of more than three modalities attract increasing attention 2 3
6 7], and play a fundamental role in and generation [13 14 15 16 17 18). Previous methods focused on collecting a large amount of paired data for cross-modal semantic alignment. However, the number of modalities increases, the costs associated with data preparation and model training to learn contrastive representation space escalate significantly.
4
5
many application scenarios of multi-modal understanding g
Recently, [19] introduces novel training-efficient method, called C-MCR, for learning contrastive representations between modalities that lack paired data by mining knowledge from existing semantic- aligned spaces. connects two pre-trained spaces Since the modalities of pre-trained spaces are intrinsically aligned, the connection learned from overlapping modalities without using image-audio and 3D-text data pairs, C-MCR demonstrates advanced performance in image-audio and 3D-text downstream tasks.
new shared space via overlapping modalities.
also be transferred to non-overlapping modalities. Experimentally,
Cooresponding author. +Equal Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Despite the remarkable flexibility and performance of C-MCR, its broader applications are hindered by a critical limitation: C-MCR mainly focuses on learning new space for the two non-overlapping modalities, while the modality alignments in powerful original pre-trained spaces are forgotten. As result, C-MCR faces challenges conducting continuous connection operations and fully utilizing all the knowledge in unified representation spaces. Therefore, it is difficult for C-MCR to build unified embedding space, especially with more than three modalities.
This paper introduces Extending Multi-modal Contrastive Representations (EX-MCR), a novel graining-efficient and paired-data-free unified representation learning method with excellent modality extensibility. Ex-MCR better preserves the alignment within the original pre-trained space and enhances the overall learning pipeline to align different spaces more robustly. By inheriting and reorganizing existing knowledge of the representation space, Ex-MCR achieves low training costs and data requirements. Furthermore, when used in conjunction with large-scale pre-training methods, Ex- MCR can complementarily enhance the unified representation space. Specifically, the two important designs of Ex-MCR are discussed in detail below:
1. We extend one space (called leaf' space) into another fixed space (called base space) rather than connecting two pre-trained spaces to new space. Such simple yet effective approach maximizes the preservation of modality alignment within base space, demonstrating great potential for augmenting existing unified space and integrating more pre-trained spaces.
2. We enhance the whole learning pipeline to promote stronger alignment across different spaces. Specifically: From training data perspective, since another modality cannot fully represent semantic information in one modality, we treat different modalities as queries to retrieve pseudo-data pairs (so- called different mode-centric data) and combine them form comprehensive view of multimodal semantic alignment. From the architecture perspective, we propose a decoupled projector, which reduces interference among different optimization objectives. From the learning objective perspective, we employ dense contrastive loss on pseudo-pairs between all possible modalities pairs, further enhancing the stability of learned alignments.
Utilizing Ex-MCR, we can flexibly align multiple leaf spaces onto the same base space without any paired data and with extremely low training costs. To evaluate the effectiveness of our Ex-MCR, try to extend pre-trained 31>-image and audio-text spaces onto image-text space via the overlapping image and text modality, which derive unified audio-image-text-- any paired data, Ex-MCR attains state-of-the-ar performance results across various zero-shot tasks, including audio-visual, 3D-image, audio-text, visual-text retrieval, and 3D object classification. More importantly, semantic alignment is also observed between extended modalities (e.g., audio-3D), which highlights the potential of Ex-MCR in modality extensibility.
representations. Without using
Our contributions can be summarized as three-fold:
(1) We propose Extending Multi-modal Contrastive Representations (Ex-MCR), a novel training- efficient and paired-data-
representation learning method for more than three modalities. More- over, Ex-MCR is orthogonal and complementary to previous data-driven methods, combining both can bring an enhanced space.
(2) We comprehensively augment the entire space alignment learning pipeline from the perspectives of training data, architecture, and learning objectives. These novel designs offer valuable insights about effectively integrating knowledge within existing spaces.
(3) Leveraging pre-trained models like CLIP, CLAP, and ULIP, extend audio and 3D to image-text space and obtain high-quality unified mndio-image-text-B1 representations. These representations exhibit advanced performance series of tasks.
2
Related Works
2.1
Multi-Modal Contrastive Representations
Multi- Contrastive Representations (MCR) learning aims to acquire semantically aligned cross-modal representations by pretraining the model on large-scale paired data. These aligned representations play a pivotal role in downstream comprehension and generation tasks. Inspired by the success of CLIP 20), many works try to learn contrative representations for two modalities 20) CLIP 20 and ALIGN 25 learn shared image-text representations from million-level
21 22 23 24

---

image-text pairs. CLAP [26 27 learns the audio-text representation, and CAV-MAE 28 focus on acquiring shared audio- feature space. C-MCR [9] focuses on learning new representation space by connecting the pre-trained spaces through overlapping modality.
Apart from aligning two modalities, shared representations for more than three modalities attract increasing attention. AudioCLIP [2] and WAV2CLIP [29] train an audio encoder aligned with CLIP using audio-text-image triplets data. ULIP 3 and openshape [5] construct 3D-image- text triplets data through rendering 3D mesh into 2D images and captioning images for textual description, thereby learning a corresponding 3D encoder for image-text MCR space. Furthermore, Imagebind [1] exclusively utilizes data pairs between various modalities and images to expand CLIP with multiple modal alignment encoders.
However, these methods heavily rely on large-scale, high-quality paired data collected from the internet generated automatically and exceptionally high computational resources. Due the lack of high-quality paired data for more modal combinations, such as audio-visual and text-3I,, the extensibility of representation learning is notably constrained. Furthermore, the exceedingly high computational costs also diminish the flexibility of MCR learning.
2.2 Audio -isual-Text and 3D- Visual-TTex Learning
Audio-visual-eext and -i-visual-text learning have significant applications in multi-modal recog-
nition [30
31
32 33 localization 34
35
36
37
38
39
40 4
question-answer [11] [0] 42
43
44
and generation 45
50 They also play important roles in robot-related
tasks such as human-machine interaction and synthetical information obtaining in complex environ-
ments [5 52).
Previous unified spaces, such as AudioCLIP 2 and ULIP
mainly focus on automatically
collecting or generating more paired data, but they are limited by the relatively low quality of the training datasets. Imagebind [12] employed individual vision-aligned data instead of triplets but pre-training the encoders from scratch results in high computational costs. FreeBind [53] and OmniBind (54) achieve strong modality alignment by integrating representation spaces that simultaneously contain multiple instances the same modality. These two methods mainly focus on enhancing modality alignment within existing spaces, whereas Ex-MCR is training paradigm designed to construct new modality aligmments. Our approach uses paired-free data and minimal computational resources, yet it still achieves superior performance in audio-image-text and 3D-image- text retrieval. More importantly, Ex-MCR is orthogonal to existing data-driven solutions, allowing it be flexibly used in parallel with the large-scale pre-training unified space for even stronger performance.
3 Extending Multi-modal Contrastive Representations
3.1 Extending Rather Than Connecting
employs two projectors to map them into new shared space, where the alignment of different spaces can be learned from overlapping modality B. Since each pre-trained space intrinsically contains the alignment of (A, B3) and (B,C), the alignment learned from overlapping modality theoretically can be transferred to the non-overlapping modalities.
Given two pre-trained MCR spaces on modalities (A, B) and (B, C), C-MCR 19
Specifically, for aligning different spaces, the embeddings of B are aligned in the new space, and pscudo (A,C) pairs retrieved by the same data of B are also aligned for a more comprehensive inter-space alignment. Moreover, the embeddings of different modalities within the same space realigned to close the modality gap (55). which significantly enhances the transferability of learned snter-space alignment. C-MCR shows remarkable flexibility and versatility since connecting two existing spaces only requires two learnable projectors and unpaired unimodal data.
However, as C-MCR is designed to learn new latent space for the two poveoaarlg modalities (A, C) and projects them onto this space, a significant amount of information from their original spaces is lost in the projection process.. As result, it faces challenges in concurrently establishing connections among three or more spaces. Therefore, C-MCR is not suitable for learning a unified representation space for more than three modalities.
3

---

Extending leaf-MCRs to base-MCR
Various Modality-centric Aggregating
Leaf-MCR
Bare-MCR
Laaf-MCR
a
Lincar Layer
MLP Layer
MLP Layer
Linear Loyer
0
Various Modality-centric Data Aggregating
4
T
T
V
VV
p8
V
CLAP
CLAP
CLIP
Imagoo
Audios
Texts
Images
Point Clonds
Dut
* Froocen
Leumable
Alignment loss
Dot product
Figure 1: The pipeline Ex-MCR. (a) We extend leaf spaces to base space via the overlapping modalities. The base space is frozen and the leaf spaces are aligned to the base space via projectors. (b) When extending the audio-text space to the text-image space, we iteratively use texts, audio, and images as queries retrieve and aggregate the corresponding semantically consistent embeddings. The pseudo embedding pairs generated from different modality data are shuffled together to build the final various modality-centric data pool.
To learn unified multi-modal representations in training training-efficient and paired- data--frrree manner, we propose extend one space into another space rather than connect two spaces to a new space. Considering the two spaces modalities (A, 15) and (B, C), Ex-MCR chooses one as the base space (A, B), and the other the leaf space CB, C). the "Extending" scheme, the base space frozen, and we only train one projector to map leaf space to base space via the overlapping modalities B. Specifically, we employ the native pairs of and pseudo pairs generated by to align leaf space to base space. Simultaneously, we close the modality gap between (B, C) modalities of leaf space, thereby facilitating more transferable alignments.
In contrast to C-MCR, Ex-MCR can conveniently expand more spaces and learn unified representation for three or more modalities. Benefiting from efficient training and no need for paired data, we can flexibly align multiple leaf spaces to the same base space. In addition to explicitly establishing alignment among modalities of leaf space and base space, semantic alignment also emerges between extended modalities. Ex-MCR employs base space as bridge for achieving semantic alignment among modalities multiple leaf spaces.
3.2 Enhancing Alignment Learning Pipeline
Before delving into the details our learning pipeline, we first clarify the necessary symbols and notations. We align the ULIP (3D-image) and CLAP (audio-text) onto CLIP (image-text). As shown in Fig (a), (a), the unimodal data of audios A, texts T, images V, and 3D point clouds P are input to their corresponding encoders, and the set of the extracted feature is denoted A T T V V and P where superscripts A
indicate representation space of CLAP, CLIP, ULIP, respectively. The A A {aff a ... a where n. is the number of all audio data and a represents the CLAP feature of i-th audio. Similarly, there are t." t/. v/. v!" of in T T V V pP" respectively.
In Ex-MCR, freezing base space allows us to maintain the original alignment of base space but also implies that the modality gap within base space is preserved. Consequently, it becomes necessary to map the leaf space to more suitable positions within the base space. To this end, we enhance the entire alignment learning pipeline from perspectives of data, architecture, and learning objectives.
3.2.1 Various Modality-centric Data
C-MCR only uses data of overlapping modality to retrieve semantically similar embeddings of other modalities and treats these generated embeddings pseudo pairs (we call single modality-centric data). However, is difficult to fully represent one modality with another, and retrieved embeddings by one modality often ignore some semantics of other modalities. For example, images about

---

"mushrooms" tend to be absent when retrieving embeddings by audio, and audio of "wind noise" may be ignored in embeddings aggregated by images. Therefore, aggregating embeddings from only single modality struggles to capture the entire representation space of different modalities.
To tackle the above problem, we propose various modality-centrie data strategy. By ensembling semantic consistent embeddings aggregated by multiple modalities, the final embeddings can reflect the representation space of different modalities in different MCRs more comprehensively. As depicted in Fig (b), all modalities in two spaces are iteratively employed as queries to aggregate corresponding semantic consistent embeddings. Take aligning audio-text space text-image space as an example, the consistent embeddings based on overlapping modality (e.g., text) are aggregated as follows:
E E/-
ā softmax (t. TA)/T))(( (AA))
t/s ! softmax ((t v (V!)T
where the 7i is the temperature parameter of softmax, and the softmax over all the samples in used datasets. The tilde symbols mean the features are processed to be semantically consistent. The t, and E/ are derived from the same text data, and their semantics are natively consistent. Benefiting from the modality semantic alignment within each pre-trained space, the generated a and v are also semantically relevant to the t and E/.
To capture the representation space of non-overlgpping modality more comprehensively, we further aggregate semantic consistent embeddings via data of non-overlapping modality (e.g., audio and image). The process of generating embeddings based on audio can be expressed as:
a = = E softmax T )/T)) ; softmax )/m) (T')²
v softmax (E/ v')/T)-(V')
(2)
Since the embeddings of T and T of overlapping modality are one-to-one matched, the similarity weights between ā and T can be naturally transferred to T
Based on the aforementioned formulas, when extending audio-text to text-image. we iteratively em- ploy texts, audios, and images queries to aggregate corresponding semantic consistent embeddings. During training, semantic consistent embeddings from different sources are shuffled together and the final data pool of various modality-centri data can be represented as {āg V t t J
3.2.2 Decoupled Projector
The main network structure of EX-MCR is projector, and it serves two purposes: 1) Learning the intra-space alignment to close the modality gaps within Ieaf space and prompt more stable aligmment between spaces. 2) Learning the inter-space alignment for extending leaf space to base space. Considering these two different purposes, we propose a decoupled projector alleviate the potential conflict between distinct optimization objectives and explore more reasonable mapping layer design for these two purposes. As shown in [ the projector is decoupled into linear layer fi(-) for entra-space aligmment and multi-layer perceptron layer fm (-) for inter-space alignment. For extending CLAP to CLIP, we first use fi to align ä, to Ei", the loss function is defined as:
11 B
Σ (i(a Lintro B
With the intra-space alignment loss, fi(-) learns the mapping between audio subspace and text subspace within the CLAP, thereby effectively closing the modality gap. Since the subspaces of different modalities within pre-trained spaces are very similar, linear mapping enough to bridge the modality gap. Moreover, our experiments even found that activation layers hurt bridging the modality gap.
After bridging the modality gap, the shared fm)()) are employed to map both audio and text embed- dings of CLAP space to the CLIP space, which can be expressed as:
a fm((((()))) E (t,))
(4)
5

---

Table 1: Results of audio-image-text experiments. The best results are bolded.
Audio-Image AVE
Audio-Text AudioCaps
Image-Text COCO
Method
FlickrNet
VGGSS
R@1
R@5
R@1
R@5
R@1
R@5
R@1
R@1
R@5
CLAP
40.25
76.21
CLIP
40.24
64.78
AudioCI.IP
1.37
4.91
0.61
2.65
1,25
3.94
3.53
11.30
17.51
37.50
WAV2CLIP
0.82
3.41
0.95
4.23
2.51
10.4
0.88
4.22
40.24
64.78
ImageBind
7.68
20.78
18.00
40.11
14.82
35.67
9.24
27.47
57.28
79.54
C-MCRGUIP-CLAP
1.39
5.97
1.25
4.49
1.94
7.69
15.76
41.37
16.67
37.04
Ex-MCR-ban
1.57
5.95
1.40
4.94
2.13
8.12
19.07
47.05
40.24
64.78
Ex-MCR- -hug
1.80
6.16
1.89
7.36
3.26
11.77
26.95
59.60
57.28
79.54
Ex-MCR-huge ImageBind
7.92
21.26
17.11
38.95
15,49
37.55
18.34
47.44
57.28
79.54
Table 2: Results of 3D-immage-txx experiments.
3D-Text
3D-Image Objaverse-LVV LVIIS
Image-Tex: COCO
Method
ModelNet40
Acc@1 I
Acc@3
Acc@5
R@1
R@5
R@1 1
R@5
CLIP
40.24
64.78
ULIP
60.40
79.00
84.40
1,45
4.51
28.69
53.14
ULIP v2
73.06
86.39
91.50
6.00
15.63
28.69
53.14
C-MCR
64.90
87.00
92.80
1.36
4.80
24.53
48.25
Ex-MCR-basy
66.53
87.88
93.60
2.54
8.25
40.24
64.78
3.2.3
Dense Alignment Objective
Since the modality gap within the base space is still preserved, a more robust learning objective is needed to map leaf space the appropriate position in the base space. To this end, we propose to learn the alignment densely among the quadruple semantic consistent embedding pairs described in 3.2.1 When extending CLAP to CLIP, the dense inter-space alignment losses are defined as:
Lave InfoNCE(a v(); Live InfoNCE(t v
(5)
Late InfoNCE((A (t);; Luc InfoNCE((EA,))
where the InfoNCE(s,)) is the standard contrastive loss function, which is defined as:
exp((x; 2:)/T2) + log exp((z: x:)) InfoNCE(x,2) 2B I=1 Σ exp((x; T2) Σ exp((2; x2)/T2)
B
log
(6)
where the 72 is the temperature parameter. The overall loss is defined weighted combination of the intra-space and inter-space losses:
L ALintra 1 (Lave Late Live + Lite)
where À is the
to balance the two terms.
Various modality-centric data 3.2. decoupled projector 3 3.2.2 and dense alignment loss 3 3.2.3 are also symmetrically employed to extend the 3D-image space to image-text space via images. result, we obtain unified 3D-imaaeetext-audioo representation. Considering audio, text, image, and 3D point cloud inputs, we use CLAP's audio encoder, CLIP's text and image encoder, and ULIP's 3D encoder to extract corresponding features aç" t/. v p The t/. v fA (f( (a("). unified representation learned by Ex-MCR, where the (.),,, ())) (.) are the learned projectors of CLAP and ULIP respectively.
py (S! ( (p(') are the final
'WAV2CLIP is trained on VGG-Sound. Its retrieval results on VGGSS are supervised, while other results shoo-

---

4 Experiment
4.1 Experimental Setting
Datasets For fair comparison, totaling MM texts, 1.3M images, 1.8M audio, and 0.8M 3D point clouds. More details about training datasets are provided in the Appendix.
the unimodal datasets C-MCR 19 for training,
Implementation Details For Ex-MCR-base. We employ pre-trained frozen CLIP ViT-B/32 20), CLAP (7)), and ULIPv2 (PointBERT version) [4] models. We also extend CLAP's audio encoder to OpenCLIP VIT-H [56] to build the Audio Audio-Image-Tex: space Ex Ex-MCR-huge in parallel with ImageBind 12 The temperature Ti in Eq for embedding aggregation is set to 0,01 following
19. while the 72 in 6 set 0.05. The hyper-parameter in Eq is set to 0.1. Following [19) we also add Gaussian noise with a variance of 0.004 to the semantic consistent embeddings described
in Sec 3.2.1 The linear projector fi(-) is simple linear layer, and the MLP projector fm is 2 2-layer MLP. We train our model with batch size of 4096 for 36 epochs. We employ the AdamW optimizer with an initial learning rate of le-3 and cosine learning rate decay strategy.
4.2 Audio-Image- Text Results
Downstream Tasks We employ zero-shot audio-image, audio-text, and image-text retrieval tasks to evaluate the audio-image-text representations of Ex-MCR. For audio-image retrieval, we conduct evaluations on Plickr-SoundNe (57]. VGGSS [39], and AVE [58] datasets. Due to their small dataset sizes, we utilize all their available data, comprising 5,000, 5,000, and 4,097 samples. For audio-text retrieval, we utilize the validation set from the AudioCaps [59] dataset, which includes 964 audio samples, and for each audio, there are 5 corresponding captions for retrieval. Regarding image-text retrieval, we employ the validation set of COCO [60] dataset, consisting of 5,000 images and 25,014 text captions. We calculate the cosine similarity between modalities in representation space and use Top-1 and Top-5 metrics for performance comparison.
Performance Comparison In the upper part of Fig [ we compare Ex-MCR-base to WAV2CLIP, AudioCLIP, and C-MCR. Notably, even without using audio-image paired data, Ex-MCR- achieves significantly better performance over WAV2CLIP and AudioCLIP, which illustrates that Ex-MCR is more effective representation learning method when high-quality data pairs are limited. Furthermore, compared to C-MCR, Ex-MCR not only achieves better audio-image alignment but also inherits more audio-text alignment from CLAP, fully retaining CLIP's image-text modal alignment, suggesting that Ex-MCR is generally superior to C-MCR in both establishing new Spaces and maintaining original ones. We then compare the performance of Ex-MCR- -uge and data-driven alignment-building methods in the bottom half of [ig[] Inheriting the audio-text alignment of CLAP, EX-MCR- -huge achieved better results on audio-text retrieval tasks, while ImageBind, trained directly with Audio-Image pairing data, has better audio-image performance. We were pleasantly surprised to find that using EX-MCR and data-driven methods in parallel, with very little additional cost, can complement each other to achieve a state-of-the-an unified audio-image-text representation.
4.3 ID-Imaee-Text Results
Downstream Tasks To evaluate the performance of 3D-imageetext space learned by extending ULIP to CLIP, we conduct a zero-shot 3D object classification task to assess the alignment between 3D and text. We also perform zero-shot 3D-image and image-text retrieval tasks to evaluate the alignment between 3D and image, as well as image and text. The zero-shot 3D object classification task is carried on the ModelNet40 [61] validation set, and we use the same prompt strategy as [4]. Regarding the zero-shot 3D-image retrieval task, we use the Objaverse- includes 46,054 3D objects. Additionally, we continued to use the COCO dataset's validation set for zero-shot image-text retrieval.
dataset [2]. which
It is worth noting that ULIP aligns a 3D encoder to -ssionn model called SLIP 63 (not CLIP) through 3D-image-text data. Ex-MCR only uses the aligned 3D-image representation of ULIP to extend it to different vasion-language model (i.e., CLIP) via the paired-data-free way. So we are not reproducing or refining the alignment of ULIP, but building a new alignment from scratch between the 3D representation of ULIP and CLIP.
7

---

Performance Comparison From Tab 3.2.3 we can find the following key points:
1) Even without using any 3D-text data, Ex-MCR still outperforms the advanced models (ULIP and ULIP v2) trained on 3D-text pairs in most performance metrics for 3D object classification.
2) For 3D-image retrieval, since the 3D-image space of ULIPv2 is treated as Ieaf space, it is reasonable that Ex-MCR-ban base 3D-image performance is slightly lower than ULIPv2. At the same time, the better 3D-image retrieval accuracy than ULIP and C-MCR shows that Ex-MCR effectively learns strong 3D-image alignment.
3) Ex-MCR retains the best image-text retrieval accuracy compared to these previous satte-of-the-all models. The leading performance on all these tasks further demonstrates the superiority of Ex-MCR in unified contrastive representation learning.
Table 3: Various modality -eetatittyyyeiiiccc centric data: We re- port the mAP metrics on all audio-visual and audio-text retrievals. The A. and T repre- sent pseudo data derived from audio, image, and text, respectively. The "+" between A, I, and means combining these data for training.
Table Alignment objective. A-T,T-T, A-V, and T - V represent the alignment objective between audio-text, text-text, audio-image, and text-image, respectively. "All" means using all above alignment losses simultaneously.
FlickrNet
AVE
VGGSS AudioCaps
FlickrNet AVE VGGSS AudioCaps
4.01
4.00
5.70
10.82
A
3.94
4.10
5.47
11.11
T- T
4.56
4.15
5.68
11.30
I
3.83
3.41
4.82
5.54
A-V
4.30
3.97
5.91
7.49
T
4.85
4.17
5.72
9.89
T-V
4.77
4.18
5.43
7.68
A+I
4.22
4.11
6.04
11.09
All
4.94
4.46
6.39
11.19
A+T
4.63
4.12
5.88
10.88
I+T
4.70
4.05
5.84
8.39
Table 6: Structure of fm((.)
A+I+T
4.94
4.46
6.39
11.19
fm(()
FlickrNet AVE VGGSS
AudioCaps
Table 5: Structure of ft(-). "Linear" means sin- gle linear layer, and "n MLP" indicates n-layer MLP
Linear
3.62
3.70
5.40
11.15
MLLP
4.62
4.15
5.81
10.53
f(())
FlickrNet
AVE
VGGSS
AudioCaps
2 MLP
4.94
4.46
6.39
11.19
Linear
4.94
4.46
6.39
11.19
3 MLP
4.85
4.31
6.57
11.30
MLP
4.54
4.16
6.50
10.25
4 MLP
4.95
4.35
6.55
11.07
2 MLP
4.36
4.04
6.00
9.93
5 MLP
4.79
4.42
6.59
10.93
4.4 Emergent 3D-Audio Alignment
In this section, we study whether the semantic alignment also emerges between the extended modali- ties (e.g., audio and 3D). We mutually retrieve audio in AudioCaps and 3D objects in Objaverse. In Fig. 4.4 and 3 we provide visualizations some top-5 retrieval results, and audios are described by their corresponding caption annotations. These cases effectively demonstrate the emergent semantic alignment between audio-3D in Ex-MCR space. For example, the sound of a flushing toilet and water flow can retrieve 3D objects of toilets or sinks, while sailboat 3D object can retrieve clips containing sounds of water vessels and wind.
These exciting results demonstrate that extending ULIP and CLAP onto CLIP following our Ex-MCR unified contrastive representation space. In addition the
methods derives a 3DD-image-text-ann ffte-oof-the-at performance on all possible tasks, Ex-MCR is an extremely training -effing-efficient and paired-data datred-- representation learning method, which amplifies its application value in unified multi-modal representation learning. To further support the conclusion, provide more audio-image retrieval results and original audio files supplementary material.
4.5
Ablation Studies
In this section, we analyze the main components of Ex-MCR. All experiments are conducted on extending CLAP to CLIP, and we reported the average mAP of audio-visual and audio-text retrieval, respectively. In addition, also provide ablation results full evaluation metrics in the Appendix.

---

toilet Bushing followed
brcuthing then metal clacking and a toikt fimhinng
helicopler fying as wind blows heavily inlo a microphone.
Train born sounds it moves along tracks.
Figure 2: Visualization of Audio to 3D retrieval.
"A peraoz plaping goitu shiiting""
plays guras.
loudly while moving doors "Clankiu nailan."
"A sing and pay putar,
whistling"
"Wastling accunp ngying
close
plays
thes slamsed."
AAtilleyy fire while endeettl
playing
o6t woman epeaking and dinkes changing tepether" LLual upad clakkng fillowed brief spoods."
"Cambbas while tond plass"
spokinns
brins fired"
48 woman spe ussull cluck agsinat metall part.
"A metal choking wund alora with talling."
46 pea bragi Bilowed by Jed byyag and wornn speeking"
"Men talling fuloned be fire""
ago ooman givig antoolios cooking and siming anise"
MMuie plaping felimeal by mae baking."
46 1A water vebicle move while czild zearty.
Bell rining with usces and wird blwwing
44 "Ubbaa mank paaing over lighlyy qpiniing waier wind."
LLnd anttted mumurar and bell ringag."
46 quaking voier volicle invellig.
Sevend ahoech bells nuping reperting potter""
Bhe lare crain,, water splashes Hous.
4 Cbreaa bell chiming while people un alking"
4
lips sonneting..
Figure 3: Visualization of 3D Audio retrieval.
Various modality-centric data As described in Sec 3.2.1
employ various modality centric data to train our projectors. For investigating the effect of different modality-centric data, we ablate each modality-centric data, and the results are reported in Tab The and T represent pseudo data derived from audio, image, and text respectively. Each kind of data is beneficial for audio-visual and audio-image alignment, and using all kinds of data simultaneously brings the best performance. In addition, we find that pseudo-pairs from audios are critical to the performance of audio-text retrieval, demonstrating the importance various modality- data, and proving that previous single modality-centric data really can not fully reflect the audio representation space.
Dense alignment objective To analyze the impact of different alignment objectives, train the model with each alignment objective. From the results reported in Tab[4] we find that directly aligning the pseudo audio-image audio-text embedding pairs leads sub-optimal audio alignment, whereas aligning spaces by overlapping text modality brings better alignment than learning alignment directly from pseudo pairs. This observation further suggests that overlapping modalities play key pivotal role aligning different spaces.
Structure of fi(-) our hypothesis: the representation structures between different modalities within one MCR space are similar, and simple linear layer is enough to bridge the modality gap. Moreover, the activation layer of the MLP introduces non-linearity, which may disrupt the spatial structure representations.
demonstrates the impact of different structures of fi(-). The results prove
Structure of (-)(-) The ablation studies of fm.(.) are summarized in Tab When aligning different MCR spaces, nonlinear MI.P structure with stronger expressivity better than the simple linear layer. Besides, good results are achieved no matter how many layers of MLP, which demonstrates the robustness of our method. According to more detailed experiments Tab empirically, MLP with 2 or 3 layers achieves good balance between expressivity and learning difficulty.

---

Training typerparameters 72 and A: The results of ablation experiments show that the performance insensitive the 72 in Enoland in E So the picked T2 is 0.05 which commonly used and the picked À is only to equal the absolute value of different loss terms. For detailed experimental results, please refer the table Appendix
5
Conclusion
This paper proposes Extending Multi-modal Contrastive Representations (Ex-MCR), a novel training- efficient and paired-data-fies unified constrastive representation learning method for more than three modalities. Ex-MCR effectively integrates the knowledge in pre-trained spaces through overlapping modalities between these spaces. By extending ULIP and CLAP onto CLIP via the overlapping image and text modality, respectively, we derive unified and high-quality audio-image-tex--31 representations. Additionally, Ex-MCR provides a new view to build unified representations. Even without using paired data, Ex-MCR still achieves competitive performance, and when combined with data-driven approaches, it complementarily enhances unified representation spaces, leading to state-of-the-e results across various tasks.
Acknowledgements
This work was supported in part by the National Key R&D Program of China under Grant No.2022ZD0162000.
References
[1] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180-15 190, 2023.
[2] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976-980. IEEE, 2022.
[3] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Leaminng unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1799-189. 2023.
[4] Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martín Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. arXiv preprint arXiv:2305.0827. 2023.
[5] Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding. arXiv preprint arXiv::305. /005:: 2023.
[6] Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal M Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. arXiv preprint arXiv:2303.11313, 2023.
[7] Ziyu Guo, Renrui Zhang, Xiangyang Zhu. Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao. Xianzi Li, Hongsheng Li, and Pheng-Ann Heng. Point-bind & point-llm: Aligning point cloud with multi miliiityy for 3d understanding, generation, and instruction following, 2023.
[8] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16335 2023.
[9] Hang Zhang, Xin Li, and I Lidong Bing. Video-llama: An instruction audio-visual language model for video understanding, arXiv preprint arXiv:2306.022 02858, 2023.
10

---

[10] Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang. Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding multi-moda llms. arXiv preprint arXiv:2307.0858// 2023.
[11] Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data- efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308. 08769, 2023.
[12] Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen. Ziyu Guo, al. Imagebind-llm: Multi-modalits instruction tuning. arXiv preprint arXiv:2309.03300
2023.
[13] Zineng Tang. Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion, 2023.
[14] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. arXiv preprint arXiv:2308.05734 2023.
[15] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2204.06/25, 1(2):3 2022.
[16] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[17] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In European Confer- ence on Computer Vision, pages 89-106. Springer, 2022.
[18) Rongjie Huang, Jiawei Huang, Donghhao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301. 2023.
[19] Zehan Wang, Yang Zhao, Xize Cheng, Haifeng Huang, Jiageng Liu, Li Tang, Linjun Li, Yongqi Wang, Aoxiong Yin, Ziang Zhang, et al. Connecting multi-mode contrastive representations. arXiv preprint arXiv:2305. 143381 2023.
[20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PMLR, 2021.
[21] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified uision-languse understanding and generation. In International Conference on Machine Learning, pages 12888-12900 PMLR, 2022.
[22] Jumnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
[23] Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. Vision- language pre-training: Basics, recent advances, and future trends. Foundations and Trends@ in Computer Graphics and Vision, 14(3-4): 163-352, 2022.
[24] Hu Xu, Gargi Ghosh, Po- Yao Huang, Dmytro Okhonko. Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- shot video-text understanding. arXiv preprint arXiv:2109. 14084, 2021.
[25] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vasion- representation learning with noisy text supervision. International conference on machine learning, pages 4904 4916. PMLR, 2021.
11

---

[26] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEFE, 2023.
[27] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Kirg-Kiikpatrick, and Shlomo Dubnov. Large-scale contrastive language-audi pretraining with feature fusion and keyword- to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[28] Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R Glass. Contrastive audio-visual masked autoencoder. In The Eleventh International Conference on Learning Representations, 2022.
[29] Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2002 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
IEEE, 2022.
[30] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and numan-labeles dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017.
[31] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020.
[32] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 1512.30022.0.. 2015.
[33] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. Proceedings of the IEEE conference computer vision and patlern recognition, pages 5828-5839, 2017.
[34] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202-221. Springer, 2020.
[35] Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 422 440. Springer, 2020.
[36] Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. lang-transforme:: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2928 2228-2937,40 2021.
[37] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pages 570-586, 2018
[38] Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization. Advances in Neural Information Processing Systems, 35:37524-3743 337336, 2022.
[39] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 16867-16876, 2021.
[40] Zehan Wang, Haifeng Huang, Yang Zhao, Linjun Li, Xize Cheng, Yichen Zhu, Aoxiong Yin, and Zhou Zhao. nerp-net: 3d relative position-aware network for 3d visual grounding. arXiv preprint 23Xi:23377.13666 2023.
12

---

[41] Zehan Wang, Haifeng Huang, Yang Zhao, Linjun Li, Xize Cheng. Yichen Zhu, Aoxiong Yin, and Zhou Zhao. Distilling coarse-to-fe semantic matching knowledge for weakly supervised 3d visual grounding. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2662 2671, 2023.
[42] Daichi Azuma, Taiki Miyanishi. Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 19129-191339 19139, 2022.
[43] Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, and Gedas Bertasius. Vision transformers are parameter-effiient audio- learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2299-2309 2023.
[44] Haifeng Huang, Zehan Wang, Rongje Huang. Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. arXiv preprint wwXiv2231..08/66 2023.
[45] Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10219-10228, 2023.
[46] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXhv:2209.149888 2022.
[47] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[48] Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. Frieren: Efficient video-to-audio generation with rectified flow matching. arXiv preprint ar arXiv:2406.0080 200.000200 2024.
[49] Wang Lin, Tao Jin, Wenwen Pan, Linjun Li, Xize Cheng, Ye Wang, and Zhou Zhao. Tavt: Towards transferable audio-visual text generation. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 14983-14999, 2023.
[50] Wang Lin, Tao Jin, Ye Wang, Wenwen Pan, Linjun Li, Xize Cheng, and Zhou Zhao. Exploring group video captioning with efficient relational approximation. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 15281-15290, 2023.
[51] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. /4824, 2023.
[52] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14404 2023.
[53] Zehan Wang, Ziang Zhang, Xize Cheng, Rongjee Huang, Luping Liu, Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, et al. Freebind: Free lunch in unified multimodal space via knowledge fusion. In Forty-first International Conference on Machine Learning.
[54] Zehan Wang, Ziang Zhang, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Hengshuang Zhao, and Zhou Zhao. Omnibind: Large-scale omni multimodal representation via binding spaces. arXiv preprint arXiv:2407.1189) 2024.
[55] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning, 2022.
13

---

[56] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi. Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.
[57] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming- Hsuan Yang, and In So Kweon. Learning to localize sound source visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358-4480 03864466,440 2018.
[58] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European conference on computer vision (ECCV), pages 247-263, 2018.
[59] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generat- ing captions for audios in the wild. In Proceedings of the 2019 Conference the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pages 119-132, 2019.
[60] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[61] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[62] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142-13311 2023.
[63] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In European Conference on Computer Vision, pages 529-544. Springer, 2022.
14

---

A Training Dataset
The details of our training dataset, which are mentioned in Sec 4.1 shown below.
Text Dataset To ensure that the texts contain sufficient information for other modalities, the data of text is sourced from diverse perspectives in image-text datasets (COCO, CC3M), video-text datasets (MSRVTT, MAD), and audio-text datasets (AudioCaps, Clotho). Following [1], we select 1M texts from CC3M. There 2.33M text samples in total. We extract their CLAP and CLIP features T and T using the CLAP and CLIP encoders, respectively.
Image Dataset For another modality in base space, Image, utilize ImageNet as the data source. ImageNetIK is large-scale image recognition dataset consisting of 1.3 million images. We extract their features to the sets V and V in CLIP and ULIP, using the CLIP Encoder and ULIP Encoder.
Audio Dataset AudioSet a large- scale audio dataset with 2. IM audio clips from YouTube, equiva- lent to 5.8 thousand hours of audio and encompassing over 500 sound classes. We use the CLAP audio encoder extract the feature
from the audios of the training set.
3D Point Cloud Dataset For the 3D modality, we use Objaverse, the recently released and large-scale 3D objects dataset. It has approximately 800K real-world 3D objects. All 3D data transformed into point clouds and extracted into the feature set p using the ULIP 3D encoder.
It is worth noting that do not employ any annotations provided with the datasets mentioned above as part of our training data, which means we only use the unimodal modality of data each dataset selected.
B Architecture of Projectors
Table 7: Model configurations of projectors.
Module
Block
Cin
Coul
fi(-)
Linear
512
512
Linear
512
1024
BatchNorm
1024
1024
Relu
Linear
1024
512
BatchNorm
512
512
Relu
fm(()
Linear
512
1024
BatchNormiD
1024
1024
Relu
Linear
1024
512
BatchNormiD
512
512
Relu
The model configurations of our projectors are shown in Tab 7
C Detailed Results of Ablation Study
As supplement Tab Tab Tab and Tab we provide detailed ablation experiment results on more comprehensive evaluation metrics of various datasets, as shown below.
15

---

Table 8: Detailed results of experiments data modality
Data Perspective
FlickrNet
AVE
VGGSS
AudioCaps
mAP
R@5
MAP
R@5
mAP
R@5
mAP
R@5
A
3,94
4.77
4.10
4.66
5.47
6.95
11.11
16.39
3.83
4.63
3.41
3.70
4.82
5.96
5.54
7.18
T
4.85
5,96
4.17
4.61
5.72
7.23
9.89
14.47
A+I
4.22
4.96
4.11
4.71
6.01
7.78
11.09
16.91
A+T
4.63
5.56
4.12
4.64
5.88
7.57
10.88
16.23
IT
4.70
5.82
4.05
4.34
5.84
7.36
8.39
12.09
A+I+T
4.94
5.95
4.46
4.93
6.39
8.12
11.19
16.65
Table 9: Detailed results of experiments on alignment objective.
Objective
FlickrNet
AVE
VGGSS
AudioCaps
mAP
R@5
mAP
R@5
mAP
R@5
mAP
R@5
A-T
4.01
4.78
4.00
4.56
5.70
7.28
10.82
15.87
T-T
4.56
5.33
4.15
4.54
5.68
6.86
11.30
16.93
A-V
4.30
5.34
3.97
4.51
5.91
7.30
7.49
10.35
T-V
4.77
6.03
4.18
4.92
5.43
6.93
7.68
10.36
Dense
4.94
5.95
4.46
4.93
6.39
8.12
11.19
16.65
Table 10: Detailed results of experiments on the structure of fi(-).
51(-)
FlickrNet
AVE
VGGSS
AudioCaps
mAP
R R@5
mAP
R@5
mAP
R R@5
mAP
R @5
Linear
4.94
5.95
4.46
4.93
6.39
8.12
11.19
16.65
MLLP
4.54
5.59
4.16
4.75
6.50
8.54
10.25
14.92
2 MLP
4.36
5.15
4.04
4.66
6.00
7.63
9.93
14.48
Table 11: Detailed results of experiments on the structure of fm.()).
fm)()
FlickrNet
AVE
VGGSS
AudioCaps
mAP
R R@5
mAP
R@5
mAP
R@5
mAP
R R5
Linear
3.62
4.50
3.70
4.03
5.40
6.82
11.15
16.37
1 MLP
4.62
5.79
4.15
4.76
5.81
7.28
10.53
15.87
2 MLP
4.94
5.95
4.46
4.93
6.39
8.12
11.19
16.65
3 MLP
4.85
5.93
4.31
4.88
6.57
8.70
11.30
17.10
4 MLP
4.95
6.20
4.35
4.84
6.55
8.57
11.07
16.23
5 MLP
4.79
6.02
4.42
5.15
6.59
8.63
10.93
16.21
16

---

Table 12: Detailed results of experiments the hyperparameter of 72.
T2
FlickrNet
AVE
VGGSS
AudioCaps
R@5
R@5
R@5
R@5
0.01
4.35
5.41
9.02
61.34
0.02
4.82
5.31
9.91
62.73
0.03
5.46
6.06
10.79
62.02
0.04
5.83
6,51
11.14
61.12
0.05
6.16
7.36
11.77
59.60
0.06
6.10
6.47
11.22
57.66
0.07
6.11
6.67
10.89
56.27
0.08
6.21
6.31
10.60
55.10
0.09
6.05
6,51
10.44
55.09
0.10
5.93
6.40
10.41
53.68
Table 13: Detailed results of experiments on the hyperparameter of X.
FlickrNet
AVE
VGGSS
AudioCaps
R@5
R@5
R@5
R@5
0.00
6.02
5.81
10.46
58.59
0.01
6.19
6.57
11.42
60.88
0.03
6.24
6.47
11.36
59.93
0.05
6.19
6.35
11.10
59,41
0.10
6.16
7.36
11.77
59.60
0.15
5.74
6.43
11.23
59.34
0.20
5.92
6.31
11.17
58.08
0.25
5.84
6.14
11.23
58.15
0.30
5.79
6.19
11.15
57.59
0.35
5.67
6.36
10.93
56.97
17

---

D Compute Resource
Collecting group of pseudo datasets takes about 10 hours on single 4090 while using 12GB GPU memory. The training times for projectors between two spaces are approximately 1.5 hours, on single 4090, and only requires 3GB of GPU memory.
E Potential Ethical Impact
This paper introduces Ex-MCR, paired-data-free and training-efficient method for constructing a unified multimodal representation space, While this method offers flexibility constructing unified representation space, its training process, which does not necessitate paired data, may inadvertently create unintended associations within the constructed representation space. The alignment of the representation space is primarily influenced by the pre-training space utilized and the unimodal data, both of which need to be restricted to prevent potential misuse for unethical applications.
F Limitation and Future Work
Currently, larger and more advanced models are continually emerging across various modalities, leading to increasingly stronger alignment pre-trained contrastive representations. Although the experimental results indicate that Ex-MCR already demonstrates significant advantages at comparable model scales, considering flexibility as general paradigm, utilizing these advanced models to explore the upper limits of this learning approach would be an exciting research direction.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See in introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See in Appendix F.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
No need.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Code and model checkpoints provided in supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We are using open ppen-source data and the code can be found in the supplemental
material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipps.cc/pulli///uuu guwde//ddee///
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We show the experimental detail in Sec 4.1.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in the appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The training and sets used the experiments in this paper are deterministic. and the results of the retrieval experiments are also discrete and deterministic, so no error analysis is performed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
21

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: A description of the computational resources can be found in the appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:://nuurpss... http://ppwwppppppoccccccccccccaa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
A description of the broader impacts can be found in appendix E.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the creators or original owners of assets used in the paper are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide detailed documentation in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "kavEO2yigy", "reviewer": "Reviewer_n4Yo", "summary": "This paper proposes Extending Multimodal Contrastive Representation, a training-efficient and paired-data-free method to build unified contrastive representation for many modalities. Without using paired data, Ex-MCR achieves comparable performance to advanced methods on a series of audio-image-text and 3D-image-text tasks and achieves superior performance when used in parallel with data-driven methods.", "questions": "1. In Figures 2 and 3, which are the images that are retrieved correctly, the author should mark the images that are retrieved correctly and the images that are retrieved incorrectly.\n2. In Tables 1 and 2, the author's method is lower than the compared method in some parts. How can the advantages of the author's method be explained?", "limitations": "See the weaknesses section", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors propose Extending Multi-modal Contrastive Representations (Ex-MCR), a novel training efficient and paired-data-free representation learning method for more than three modalities.\n2. The authors comprehensively augment the entire space alignment learning pipeline from the perspectives of training data, architecture, and learning objectives.", "weaknesses": "1. The contribution of this paper is the multimodal contrastive learning method. The multimodal contrastive learning method proposed by the authors is not very novel, because contrastive learning is widely used in the field of multimodal retrieval.\n2.  The method proposed by the author is too simple, and it is recommended to deepen the improvement of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1 & W2: Lack of novelty in the article & the method is too simple\n\nThe main contribution of Ex-MCR is not to \"apply contrast learning in multimodal retrieval tasks\", but to efficiently integrate multiple pre-trained contrastive multimodal representation spaces into a new and more comprehensive one. \n\nAs a novel learning paradigm, Ex-MCR is extremely flexible and able to integrate multiple pre-trained contrastive learning spaces with different modalities to achieve state-of-the-art performance.\n\nMoreover, the paper also redesigned the data aggregation methods, the structure of the projector, and the alignment objective function, all of which make Ex-MCR more powerful. We illustrate the motivation of these designs in **Section 3.2 Enhancing Alignment Learning Pipeline**. Our ablation studies in **Section 4.5 Enhancing Alignment Learning Pipeline** fully demonstrate the effectiveness of these novel designs. In the response to reviewer **1W4Z's W4&Q5**, we further show the superiority of the decoupled projectors.\n\n## Q1：Why does not mark the 3D-audio retrieval results as correct or wrong\n\nThere is no existing benchmark for Audio-3D retrieval, so there is no objective definition of right or wrong for Audio-3D pairs. To demonstrate the emergent semantic alignment in Audio-3D, we directly show some top 5 retrieval results in our paper.\n\nWe also provided more randomly selected results for audio->3D and 3D->audio retrieval in the newly uploaded PDF.\n\n## Q2:   **Ex-MCR’s performance is lower than the compared method in some parts**\n\nFor Audio-Image-Text experiments, Ex-MCR reconstructs the alignment between CLAP's audio encoder and the CLIP image encoder and text encoder. Ex-MCR outperformed ImageBind in 4 of the 6 performance metrics of Audio-Image while having much better Audio-Text alignment and the same Image-Text alignment. Therefore, comprehensively, Ex-MCR reached state-of-the-art Audio-Image-Text performance.\n\nFor 3D-Image-Text experiments, in our latest experiments(referring to the official comment to all reviewers), we use Ex-MCR to fuse more powerful models, EVA-CLIP-18B(Image-Text), and Uni3D-G(3D-Image-Text). Ex-MCR achieves leading performance in all three inter-modality alignments (3D-Image, 3D-Text, Image-Text).\n\nAdditionally, as requested by the reviewers, we provide further clarification on the 3D performance of the Ex-MCR-base. Firstly, the Ex-MCR-base demonstrates stronger Image-Text alignment and achieves a comparable level of 3D-Text alignment to ULIPv2. Therefore, overall, Ex-MCR and ULIPv2 exhibit similar alignment capabilities in 3D-Image-Text space.\n\nWe think the main reason for the performance loss in 3D-Image is the limited training data. The pseudo data for training are mainly derived from the inherent alignment of leaf space and base space. Therefore, this extending process can be regarded as distilling leaf space’s alignment knowledge to construct a new alignment for base space. The feather and ability of the distillation teacher can affect the distillation result, and when we use a stronger pre-trained space, Ex-MCR outperforms the teacher.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IKrTcecDcm", "reviewer": "Reviewer_KwrG", "summary": "The paper introduces Extending Multi-modal Contrastive Representations (Ex-MCR), a novel method for learning unified contrastive representations across multiple modalities without the need for paired data. Ex-MCR extends one modality's representation space into another, preserving semantic alignment and reducing training costs. The method enhances the learning pipeline with modality-centric data, a decoupled projector, and a dense alignment objective. Experiments show that Ex-MCR achieves state-of-the-art performance on multi-modal tasks, highlighting its efficiency in representation learning.", "questions": "Please refer to the weakness above.", "limitations": "None", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper explores an interesting problem. In practice, as the number of modalities increases, the costs of data preparation and model training for learning a contrastive representation space rise significantly. Therefore, how to purse a training-efficient and pair-data-free representation is critical but challenging.\n\n- This paper is well-written and the motivation is stated clearly.", "weaknesses": "- The effectiveness of the proposed method heavily depends on choosing hyperparameter in Eq.(7). A sensitivity analysis of hyperparameter is necessary, and discussing insights into choosing this hyperparameter is essential.\n\n- The performance improvements of the proposed method on 3D-image-text experiments are relatively limited. Especially, Ex-MCR is so far behind ULIP-v2 in the 3D image domain. The authors should explain them clearly.\n\n- Ex-MCR requires additional training cost and inference time compared to the previous method C-MCR. The authors should give more analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1: the analysis of hyperparameter choosing\n\nPlease refer to the response to all reviewers(**part 3** The analysis of hyperparameter choosing).\n\n## W2: 3D-Image-Text performance improvements are relatively limited.\n\nFor 3D-Image-Text experiments, we think there the main reason for the performance loss is the limited training data. In our latest experiments(referring to the official comment to all reviewers), we use Ex-MCR to fuse more powerful models, EVA-CLIP-18B(Image-Text), and Uni3D-G(3D-Image-Text). Ex-MCR achieves leading performance in all three domains (3D-Image, 3D-Text, Image-Text).\n\nFor the explanation of the Ex-MCR-base's 3D performance, please refer to **the response to review nY4o's Q2**.\n\n## W3：Ex-MCR requires additional training cost and inference time compared to C-MCR\n\n**Taking extending CLAP to CLIP as an example**\n\nFor training, both Ex-MCR and C-MCR can be trained for less than 4 hours on a single RTX 4090, and the cost difference in reaching convergence can be ignored.\n\nFor inferencing, Ex-MCR only projects a single modality in one space, while C-MCR needs to project three modalities in two spaces, therefore C-MCR has a higher inference overhead than Ex-MCR. Detailed statistics for inferencing overhead are shown in the following table:\n\n**Params for projecting（M）**\n\n| modality | Audio  | Image  | Text   | Total  |\n| -------- | ------ | ------ | ------ | ------ |\n| C-MCR    | 1.0532 | 1.0532 | 1.0532 | 3.1596 |\n| Ex-MCR   | 2.3690 | 0      | 0      | 2.3690 |\n\n**Computation overhead for projecting（GFlops）**\n\n| modality | Audio  | Image  | Text   | Total  |\n| -------- | ------ | ------ | ------ | ------ |\n| C-MCR    | 0.0011 | 0.0011 | 0.0011 | 0.0033 |\n| Ex-MCR   | 0.0024 | 0      | 0      | 0.0024 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y9FDB8blHo", "reviewer": "Reviewer_1W4Z", "summary": "This paper focus on the multi-modal contrastive representation of more than 3 modalities in multi-modal learning. To address the flaws of existing works, such as the dependency on large-scale, high-quality paired data and the expensive training cost, this paper introduces Ex-MCR, a training-efficient and paired-data-free method to build unified contrastive representation for multiple modalities. The key idea of the proposed method is to extend one modality's space into the other's. Experiments are conducted in audio-text and 3D image representations to the existing vision-text space.", "questions": "1. **Evidence for Information Loss**:\n   - Can the author provide experimental results or theoretical analysis to support the claim of information loss during the projection process?\n\n2. **Support for Modality Alignment Forgetting**:\n   - Are there specific examples or experiments that demonstrate the forgetting of modality alignments in original pre-trained spaces?\n\n3. **Comparison with Non-Projection-Based Methods**:\n   - How does the proposed method compare with non-projection-based methods, especially those highlighting overlapping semantic information as discussed in UniBind?\n\n4. **Validation of Contribution Claims**:\n   - Can the author provide examples or experiments that validate the claim about preserving modality alignment and integrating more pre-trained spaces?\n\n5. **Differences in Decoupled Projector**:\n   - What are the key differences between the proposed decoupled projector and existing projectors?\n\n6. **Inclusion of Related Works**:\n   - Can the author include comparisons and discussions involving the following works to provide a more comprehensive view?\n     - Imagebind: One embedding space to bind them all, CVPR 2023.\n     - Point-bind & point-llm: Aligning point cloud with multi-modality for 3D understanding, generation, and instruction following, arXiv 2023.\n     - Onellm: One framework to align all modalities with language, CVPR 2024.\n     - Meta-transformer: A unified framework for multimodal learning, arXiv 2023.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of preserving one modality's space is straight-forward and interesting.\n\n2. The writing is easy to follow and the figures clearly show the technical designs.", "weaknesses": "1. The abstract claim 'a significant amount of information from their original spaces is lost in the projection process' and the statement 'C-MCR mainly focuses on learning a new space for the two non-overlapping modalities, while the modality alignments in powerful original pre-trained spaces are forgotten' should be supported by explicit evidence, experimental results, or theoretical analysis.\n\n2. There are various methods in the multi-modal learning area, and projection/projector-based methods are not the only option for binding different modalities. Importantly, there is overlapping information across different modalities, such as semantics, as pointed out by [1], and experiments in [1] have proved this.\n\n   [1] UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All, CVPR 2024.\n\n3. The claim in the contribution part, lines 48-51, 'Such as simple yet effective approach maximizes the preservation of modality alignment within base space, demonstrating great potential for augmenting existing unified space and integrating more pre-trained spaces,' should be supported by examples conducted by the authors to validate this claim.\n\n4. The proposed method also integrates projectors, a.k.a., the decoupled projector. What are the differences between the proposed one and existing ones?\n\n5. Many related works are not mentioned, not only in the related work section but also in the experimental comparison. These include but are not limited to:\n\n   [2] Imagebind: One embedding space to bind them all, CVPR 2023.\n   \n   [3] Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following, arXiv 2023.\n   \n   [4] Onellm: One framework to align all modalities with language, CVPR 2024.\n   \n   [5] Meta-transformer: A unified framework for multimodal learning, arXiv 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1 & Q1 & Q2: Evidence for Information Loss & Support for Modality Alignment Forgetting\n\nTaking the reprojecting CLIP representations to C-MCR space as an example, the following table(as a sub-table of Table 1) shows the performance differences before and after the projection:\n\n|                     | COCO R@1 | COCO R@5 |\n| ------------------- | -------- | -------- |\n| before reprojecting | 40.24    | 64.78    |\n| after reprojecting  | 16.67    | 37.04    |\n\nIn the reprojected CLIP representations, the Image-Text retrieval performance is significantly reduced, which indicates that the modality alignment of the original CLIP space is forgotten during the reprojection.\n\n## W2 & Q3: Comparison with Non-Projection-Based Methods\n\nWe compared Ex-MCR with non-projection-based methods,  LanguageBind[1], UniBind[2], and PointBind[3] (Note that PointBind has the same models in Audio-Image-Text with ImageBind). For Audio-Image-Text experiments, we conduct the comparison under the same Image-Text performance. \n\nThe results are as follows:\n\n**Results of Audio-Image-Text experiments. The best results are bolded.**\n\n|                       | Flickr   |           | AVE       |           | VGGSS     |           | Audiocaps |           | COCO      |           |\n| --------------------- | -------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n|                       | R@1      | R@5       | R@1       | R@5       | R@1       | R@5       | R@1       | R@5       | R@1       | R@5       |\n| AudioCLIP             | 1.37     | 4.91      | 0.61      | 2.65      | 1.25      | 3.94      | 3.53      | 11.30     | 17.51     | 37.50     |\n| WAV2CLIP              | 0.82     | 3.41      | 0.95      | 4.24      | 2.51      | 10.47     | 0.88      | 4.22      | 40.24     | 64.78     |\n| C-MCR                 | 1.39     | 5.69      | 1.25      | 4.49      | 1.94      | 7.69      | 15.76     | 41.37     | 16.67     | 37.04     |\n| ImageBind(PointBind)  | 7.68     | 20.78     | **18.00** | **40.11** | 14.82     | 35.67     | 9.24      | 27.47     | 57.28     | 79.54     |\n| LanguageBind          | 1.52     | 6.36      | 1.49      | 5.96      | 2.55      | 9.86      | 12.42     | 36.7      | 53.24     | 76.48     |\n| UniBind               | 7.74     | 20.87     | 17.78     | 39.96     | 14.14     | 35.74     | 9.24      | 27.47     | 57.28     | 79.54     |\n| Ex-MCR-huge+Imagebind | **7.92** | **21.26** | 17.11     | 38.95     | **15.49** | **37.55** | **18.34** | **47.44** | **57.28** | **79.54** |\n\nEx-MCR achieved leading performance in 4 of the 6 Audio-Image retrieval metrics and a significant advantage in Audio-Text semantic alignment.\n\nIn 3D-Image-Text experiments, to explore the upper bound of performance in advanced pre-trained space, we use Ex-MCR to fuse EVA-CLIP-18B(Image-Text) and Uni3D-G(3D-Image-Text)(Please refer to the response to all reviewers). The newly constructed 3D-Image-Text space achieves state-of-the-art performance in all three domains (3D-Image, 3D-Text, Imge-Text), compared with non-projection-based methods.\n\nMoreover, since Ex-MCR is extremely flexible, its performance can easily boost, and benefit from the breakthrough of the advanced pre-training spaces. \n\n[1] LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment, ICLR 2024\n\n[2] UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All, CVPR 2024\n\n[3] Point-Bind & Point-LLM: Aligning 3D with Multi-modality, arxiv 2023\n\n## W3 & Q4: Validation of Contribution Claims\n\n**(1) Claim about preserving modality alignment:**\n\nEx-MCR preserving modal alignment in two ways: \n\nOn the one hand, by extending the modalities in one space to another space rather than constructing a new representation space, Ex-MCR preserves the base space. For example, all Image-Text modality alignment of CLIP is fully inherited in Ex-MCR.\n\nOn the other hand, by redesigning the data aggregation method, the structure of the projector, and loss functions, Ex-MCR can preserve better cross-model alignment from the leaf space(Audio-Text and 3D-Image alignment, compared to C-MCR).\n\n**(2) Claim about integrating more pre-trained spaces:**\n\nIn this unified space, we preserve advanced intermodal alignment (see Table 1, Table 2 in the paper), and obtain emergent 3D-Audio alignment (see Figure 2 in the paper and more diverse examples provided in PDF). This result proves that Ex-MCR is a unified representation space that can accommodate multiple modalities.\n\nMoreover, the result also indicates that both image and text(modalities in base space) can serve as the overlapped modality to extend the space.\n\n## W4 & Q5: Differences between decoupled projector and existing projector\n\nThe decoupled projector reduces interference among different optimization objectives, which helps the model learn modality alignment better.\n\nWe performed ablation experiments on Ex-MCR-huge, and the results are as follows:\n\n| project type            | Flickr | AVE  | VGGSS | ACaps |\n| ----------------------- | ------ | ---- | ----- | ----- |\n|                         | R@5    | R@5  | R@5   | R@5   |\n| non-decoupled projector | 7.22   | 6.53 | 11.28 | 58.36 |\n| decoupled projector     | 6.16   | 7.36 | 11.77 | 59.60 |\n\nThe above results show that the decoupled projector reaches overall performance.\n\n\n\n## W5 & Q6: Inclusion of Related Works\n\nThank you for providing more related works. We will cite these methods in the revised paper. Please refer to the response to W2 & Q3 for comparison results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wO6P6vkXOI", "reviewer": "Reviewer_V8oT", "summary": "This paper introduces the Extending Multi-modal Contrastive Representation (Ex-MCR), an efficient method for learning multi-modal contrastive representations without relying on paired modality data. Unlike the previous C-MCR scheme, which connects two pre-trained modalities to a new embedding space, Ex-MCR extends one modality (leaf) to another (base), while the base modality remains frozen. This approach has the merit of preserving the alignment of pre-trained modalities in their original space while facilitating a unified modality space via an overlapping modality (e.g., using text to map audio-text to text-image). Experimentally, pre-trained audio-text and image-3D representations are aligned within the image-text modality space without requiring paired modality data, achieving superior performance over other methods.", "questions": "- From [L303-304], the terms 'audio-visual' and 'audio-image' seem to have duplicative meanings.\n\n- As an oracle, how can the proposed Ex-MCR scheme boost performances when trained on paired modality data?", "limitations": "It adequately touches the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- It effectively addresses the limitations of previous work, which struggled to preserve multi-modal alignment in the original representation spaces. This is validated by experiments showing that the trained Ex-MCR method also yields promising results in audio-text and image-text retrieval tasks, as compared to C-MCR, detailed in Table 1.\n\n- Such improvements are possible even without paired multi-modal data and without requiring large computational resources for training, demonstrating its efficiency. This method also shows positive potential for aligning audio to 3D modalities.\n\n- The idea of treating intra-modal alignment and inter-modal alignment differently is interesting, with linear layers and MLP layers being used respectively, considering the similarity of representation spaces. This design choice is validated by experiments, in Tables 5 and 6.", "weaknesses": "- My major concern lies in the overall clarity of this paper. Although Section 3.1, which explains the underlying motivation, was clear, I found Fig. 1(b) and Section 3.2.1 difficult to understand despite several rounds of reading. \n\n  - For Figure 1(b), it is difficult to interpret what is happening in each modality. It is suggested that more illustrative figures be depicted and supplemented with detailed captions. \n\n  - For Section 3.2.1, overall clarity is limited. It seems to describe the construction of pseudo-modality pairs by retrieving similar embeddings from other modalities one by one, and incorporating various enhancing techniques. From Equation 1, the notations are inconsistent between the first and second lines. For consistent audio embeddings, T^A is used, while for consistent image embedding, V^I is used. In the formulation for retrieving similar audio samples given text as a query, should it be A^A instead of T^A in the first line? In a similar context, it would be desirable to verify the notations in Equation 2. Moreover, the clarity of the process and general description of modality-centric data pool could be further enhanced. \n\n\n- No clear evidence can be found that Ex-MCR improves the alignment of inter-modalities over C-MCR. For audio-image and 3D-image tasks in Tables 1 and 2, Ex-MCR conveys only slight improvements over C-MCR.\n\n\n- Sensitivity analysis on the hyper-parameters such as \\lambda and \\tau is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1: Statements and Figure in the “Various Modality-centric Data” part\n\nThanks for pointing out the typos and providing suggestions about figures, we have fixed them in the new version. \n\n**1.1 Unclear Figure 1(b) and its caption.**\n\nWe have uploaded the detailed captioned Figure 1(b) to the newly submitted PDF. \n\n- We replace \"softmax\" with \"A->B similarity\" in the figure to emphasize that when using Audio or Image as the query, the shared element in the text aggregation process is the similarity weight results, not the softmax operation itself.\n- We add a more detailed caption to better illustrate the overall pipeline of pseudo pair construction.\n\n**1.2 Typos in Eq(1)**\n\nWe sincerely apologize for the misunderstanding caused by the typo. The correct formula (1) is as follows:\n$$\n\\tilde{\\mathbf{t}}_i^A = \\mathbf{t}_i^A; \\ \\ \\ \\tilde{\\mathbf{t}}_i^I = \\mathbf{t}_i^I;\\ \\ \\\n    \\tilde{\\mathbf{a}}_i^{A} = \\operatorname{softmax}((\\tilde{\\mathbf{t}}_i^A \\cdot \\mathbf{T}^{A})/\\tau_1) \\cdot (\\mathbf{A}^{A})^{T};  \\\n     \\ \\ \\ \\tilde{\\mathbf{v}}_i^{I} = \\operatorname{softmax}((\\tilde{\\mathbf{t}}_i^I \\cdot \\mathbf{V}^{I})/\\tau_1) \\cdot (\\mathbf{V}^{I})^{T}\n$$\nEq(2) is correct. When audios serve as queries to construct pairs, the first step is to obtain the relevant text embeddings through the inter-modality alignment of CLAP. Note that $\\tilde{\\mathbf{t}}_i^A$ and $\\tilde{\\mathbf{t}}_i^I$ always share the similarity weights retrieved by CLAP (the softmax term in the formula) during data aggregating. Next, we use $\\tilde{\\mathbf{t}}_i^I$ and the inter-modality alignment in CLIP to obtain the relevant $\\tilde{\\mathbf{v}}_i^I$.\n\n\n\n## W2: Explanations for performance enhancement on inter-modalities when compared to C-MCR\n\nTo avoid misunderstanding, we further clarify our experiment settings in the response to all reviewers(**Part 1**). Benefit from the fully enhanced alignment learning pipeline described in **Section 3.2**, Ex-MCR makes significant progress in preserving existing inter-modalities alignments in leaf space(Audio-Text and 3D-Image) and has steadily improved the newly learned inter-modalities(Audio-Image and 3D-Text) alignments, compared to C-MCR.\n\nMost importantly, Ex-MCR fully inherits the inter-modalities alignments in base space, significantly outperforming C-MCR. The fully preserved base space brings strong modality scalability. Serving image or text as overlapping modalities, various multi-modal contrastive learning spaces can be projected into a unified Ex-MCR space.\n\n\n\n## W3: the analysis of hyperparameter choosing\n\nPlease refer to the response to all reviewers(**Part 3**).\n\n## Q1: Inconsistent wording\n\nThanks for pointing out the inconsistencies in wording. \"audio-vision\" and \"audio-image\" have the same meaning in the paper, and we have modified these two expressions into \"audio-image\".\n\n## Q2: trained on paired modality data\n\nWe finetuned Ex-MCR-huge on paired Audio-Text data (trainset of Audiocaps). The results are as follows:\n\n| Model                               | Flickr   | AVE      | VGGSS     | ACaps     |\n| ----------------------------------- | -------- | -------- | --------- | --------- |\n|                                     | R@5      | R@5      | R@5       | R@5       |\n| Ex-MCR-huge                         | 6.16     | 7.36     | 11.77     | 59.60     |\n| Ex-MCR-huge finetune on paired data | **6.46** | **7.48** | **11.94** | **60.43** |\n\nUsing paired data in training not only improves the alignment performance between the corresponding two modalities(Audio-Text) but also helps the model to obtain better alignment between other modalities(Audio-Image).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Response to all reviewers:

## 1 Further clarification of the experimental setup

The final unified Ex-MCR representations are composed of CLIP's image, text representations, projected CLAP's audio representations, and projected ULIP's 3D representations (i.e. the $\bf{t_{i}^{I}}$, $\bf{v_{i}^{I}}$ ,$\bf{f_m^A(f_l^A(a_{i}^{A}}))$, $\bf{f_m^U(f_l^U(p_{i}^{U}}))$ ) are the final audio-text-image-3D unified representation).  Ex-MCR only uses the aligned 3D-image representation of ULIP and extends it to a different image-text model (i.e., CLIP) via the paired-data-free way.

## 2 3D-Image-Text experiment on Advanced pre-trained space

We use Ex-MCR to fuse advanced pre-trained models, EVA-CLIP-18B(Image-Text)[1], and Uni3D-G(3D-Image-Text)[2]. 

To fully demonstrate the performance, we tested 3D-text alignment using three 3D classification tasks: Modelnet40[3], Objaverse-lvis[4] and Scanobjnn[5].

The details of benchmarks are shown in the following table:

| Benchmarks     | Items | Num of Classes |
| -------------- | ----- | -------------- |
| Modelnet40     | 2468  | 40             |
| Objaverse-lvis | 46832 | 1156           |
| Scanobjnn      | 2890  | 15             |

As in the paper, we use Objaverse-lvis and COCO to test the alignment between 3D-Image and Image-Text, respectively.

We compared Ex-MCR with other non-projector-based methods such as PointBind[6], and UniBind[7]:

Results of 3D-Image-Text experiments.

|                              | ModelNet40 |           | Objaverse-lvis |           | Scanobjnn |           | Objaverse 3D-image retrieval |           | COCO      |           |
| ---------------------------- | ---------- | --------- | -------------- | --------- | --------- | --------- | ---------------------------- | --------- | --------- | --------- |
|                              | R@1        | R@5       | R@1            | R@5       | R@1       | R@5       | R@1                          | R@5       | R@1       | R@5       |
| Uni3D-g                      | 87.56      | 99.27     | 53.13          | 81.59     | **64.12** | 91.63     | 43.57                        | 67.61     | 59.51     | 81.45     |
| ULIPv2                       | 73.06      | 91.50     | 30.26          | 55.01     | 46.75     | 76.33     | 7.54                         | 18.80     | 22.92     | 46.33     |
| PointBind                    | 76.18      | 97.04     | 13.83          | 30.34     | 55.05     | 86.89     | 5.86                         | 14.59     | 57.28     | 79.54     |
| UniBind                      | 63.21      | 89.59     | 8.54           | 20.29     | 41.73     | 74.39     | 3.51                         | 9.28      | 57.27     | 79.53     |
| Ex-MCR(EVA-CLIP-18B & Uni3D) | **88.09**  | **99.31** | **53.59**      | **82.25** | 63.94     | **91.97** | **46.33**                    | **70.04** | **60.61** | **82.08** |

Ex-MCR achieves leading performance in all three inter-modality alignments, indicating that its performance can easily benefit from the breakthrough of the advanced pre-training spaces.

[1] EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters, arxiv 2024

[2] Uni3d: Exploring unified 3d representation at scale}, ICLR 2024

[3] 3d shapenets: A deep representation for volumetric shapes, CVPR 2015

[4] Objaverse: A universe 347 of annotated 3d objects, CVPR 2023

[5] Revisiting point cloud classification: A new benchmark dataset and classification model on 484 real-world data, ICCV 2019

[6] Point-Bind & Point-LLM: Aligning 3D with Multi-modality, arxiv 2023

[7] UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All, CVPR 2024

## 3 The analysis of hyperparameter choosing

We conducted ablation experiments on Ex-MCR-huge for training temperature $\tau_2$ in Eq.(6) and L2 loss weight $\lambda$ in Eq.(7)(referring to the official comment to all reviewers), and the results are as follows:

Ablation results for temperature $\tau_2$ in Eq.(6).

| $\tau_2$ | Flickr   | AVE      | VGGSS     | ACaps     |
| -------- | -------- | -------- | --------- | --------- |
|          | R@5      | R@5      | R@5       | R@5       |
| 0.01     | 4.35     | 5.41     | 9.02      | 61.34     |
| 0.02     | 4.82     | 5.31     | 9.91      | **62.73** |
| 0.03     | 5.46     | 6.06     | 10.79     | 62.02     |
| 0.04     | 5.83     | 6.51     | 11.14     | 61.12     |
| 0.05     | 6.16     | **7.36** | **11.77** | 59.60     |
| 0.06     | 6.10     | 6.47     | 11.22     | 57.66     |
| 0.07     | 6.11     | 6.67     | 10.89     | 56.27     |
| 0.08     | **6.21** | 6.31     | 10.60     | 55.10     |
| 0.09     | 6.05     | 6.51     | 10.44     | 55.09     |
| 0.10     | 5.93     | 6.40     | 10.41     | 53.68     |

Ablation results for L2 loss factor $\lambda$ in Eq.(7).

| $\lambda$ | Flickr   | AVE      | VGGSS     | ACaps     |
| --------- | -------- | -------- | --------- | --------- |
|           | R@5      | R@5      | R@5       | R@5       |
| 0.00      | 6.02     | 5.81     | 10.46     | 58.59     |
| 0.01      | 6.19     | 6.57     | 11.42     | **60.88** |
| 0.03      | **6.24** | 6.47     | 11.36     | 59.93     |
| 0.05      | 6.19     | 6.35     | 11.10     | 59.41     |
| 0.10      | 6.16     | **7.36** | **11.77** | 59.60     |
| 0.15      | 5.74     | 6.43     | 11.23     | 59.34     |
| 0.20      | 5.92     | 6.31     | 11.17     | 58.08     |
| 0.25      | 5.84     | 6.14     | 11.23     | 58.15     |
| 0.30      | 5.79     | 6.19     | 11.15     | 57.59     |
| 0.35      | 5.67     | 6.36     | 10.93     | 56.97     |

The results of ablation experiments show that the performance is insensitive to the $\tau_2$ , and the commonly used 0.05 can achieve good performance. 

The $\lambda$ we picked is only to equal the absolute value of different loss terms, so we did not extensively search all values. Meanwhile, the experimental results show that further searching of hyperparameters can reach better performance.

Meta-review: The current ratings are 5, 7, 5, 5 (one of the reviewers expressed to increase his rating but did not actually did it in the system). AC saw that maybe the only concern that is left seems to be **3D-Image-Text performance remains low**. Authors provided some verbal explanations but AC remains somewhat unconvinced on that point.

However, in view of the ratings and it seems most except one concerns have been addressed, this paper looks ok to be accepted. AC urges the authors to address all the points raised in the discussion in the final manuscript.
Decision: Accept (poster)