Extending Multi-modal Contrastive Representations

Authors: ziang  zhang
zehan  wang
luping  liu
rongjie  huang
xize  cheng
zhenhui  ye
wang  lin
huadai  liu
haifeng  huang
yang  zhao
tao  jin
siqi  zheng
zhou  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 15:17:08

Abstract: Multi-modal contrastive representation (MCR) of more than three modalities is critical in multi-modal learning. Although recent methods showcase impressive achievements, the high dependence on large-scale, high-quality paired data and the expensive training costs limit their further development. Inspired by recent C-MCR, this paper proposes $\textbf{Ex}$tending $\textbf{M}$ultimodal $\textbf{C}$ontrastive $\textbf{R}$epresentation (Ex-MCR), a training-efficient and paired-data-free method to build unified contrastive representation for many modalities. Since C-MCR is designed to learn a new latent space for the two non-overlapping modalities and projects them onto this space, a significant amount of information from their original spaces is lost in the projection process. To address this issue, Ex-MCR proposes to extend one modality's space into the other's, rather than mapping both modalities onto a completely new space. This method effectively preserves semantic alignment in the original space. Experimentally, we extend pre-trained audio-text and 3D-image representations to the existing vision-text space. Without using paired data, Ex-MCR achieves comparable performance to advanced methods on a series of audio-image-text and 3D-image-text tasks and achieves superior performance when used in parallel with data-driven methods. Moreover, semantic alignment also emerges between the extended modalities (e.g., audio and 3D).

Main paper: 
 introduction: 
 Multi-modal Contrastive Representation (MCR) learning aims to align inputs from diverse modalities within a shared representation space. Recently, the high-quality contrastive representations of more than three modalities attract increasing attention [1, 2, 3, 4, 5, 6, 7] , and play a fundamental role in many application scenarios of multi-modal understanding [8, 9, 10, 11, 12] and generation [13, 14, 15, 16, 17, 18] . Previous methods focused on collecting a large amount of paired data for cross-modal semantic alignment. However, as the number of modalities increases, the costs associated with data preparation and model training to learn a contrastive representation space escalate significantly.Recently, [19] introduces a novel training-efficient method, called C-MCR, for learning contrastive representations between modalities that lack paired data by mining knowledge from existing semanticaligned spaces. It connects two pre-trained spaces onto a new shared space via overlapping modalities. Since the modalities of pre-trained spaces are intrinsically aligned, the connection learned from overlapping modalities can also be transferred to non-overlapping modalities. Experimentally, without using image-audio and 3D-text data pairs, C-MCR demonstrates advanced performance in image-audio and 3D-text downstream tasks.Despite the remarkable flexibility and performance of C-MCR, its broader applications are hindered by a critical limitation: C-MCR mainly focuses on learning a new space for the two non-overlapping modalities, while the modality alignments in powerful original pre-trained spaces are forgotten. As a result, C-MCR faces challenges in conducting continuous connection operations and fully utilizing all the knowledge in unified representation spaces. Therefore, it is difficult for C-MCR to build a unified embedding space, especially with more than three modalities. This paper introduces Extending Multi-modal Contrastive Representations (Ex-MCR), a novel training-efficient and paired-data-free unified representation learning method with excellent modality extensibility. Ex-MCR better preserves the alignment within the original pre-trained space and enhances the overall learning pipeline to align different spaces more robustly. By inheriting and reorganizing existing knowledge of the representation space, Ex-MCR achieves low training costs and data requirements. Furthermore, when used in conjunction with large-scale pre-training methods, Ex-MCR can complementarily enhance the unified representation space. Specifically, the two important designs of Ex-MCR are discussed in detail below:1. We extend one space (called leaf space) into another fixed space (called base space) rather than connecting two pre-trained spaces to a new space. Such a simple yet effective approach maximizes the preservation of modality alignment within base space, demonstrating great potential for augmenting existing unified space and integrating more pre-trained spaces.2. We enhance the whole learning pipeline to promote stronger alignment across different spaces. Specifically: From a training data perspective, since another modality cannot fully represent semantic information in one modality, we treat different modalities as queries to retrieve pseudo-data pairs (socalled different mode-centric data) and combine them to form a comprehensive view of multimodal semantic alignment. From the architecture perspective, we propose a decoupled projector, which reduces interference among different optimization objectives. From the learning objective perspective, we employ a dense contrastive loss on pseudo-pairs between all possible modalities pairs, further enhancing the stability of learned alignments.Utilizing Ex-MCR, we can flexibly align multiple leaf spaces onto the same base space without any paired data and with extremely low training costs. To evaluate the effectiveness of our Ex-MCR, we try to extend pre-trained 3D-image and audio-text spaces onto image-text space via the overlapping image and text modality, which derive unified audio-image-text-3D representations. Without using any paired data, Ex-MCR attains state-of-the-art performance results across various zero-shot tasks, including audio-visual, 3D-image, audio-text, visual-text retrieval, and 3D object classification. More importantly, semantic alignment is also observed between extended modalities (e.g., audio-3D), which highlights the potential of Ex-MCR in modality extensibility.Our contributions can be summarized as three-fold:(1) We propose Extending Multi-modal Contrastive Representations (Ex-MCR), a novel trainingefficient and paired-data-free representation learning method for more than three modalities. Moreover, Ex-MCR is orthogonal and complementary to previous data-driven methods, combining both can bring an enhanced space.(2) We comprehensively augment the entire space alignment learning pipeline from the perspectives of training data, architecture, and learning objectives. These novel designs offer valuable insights about effectively integrating knowledge within existing spaces.(3) Leveraging pre-trained models like CLIP, CLAP, and ULIP, we extend audio and 3D to image-text space and obtain high-quality unified audio-image-text-3D representations. These representations exhibit advanced performance on a series of tasks.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: See in introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
multi-modal contrastive representations: 
 Multi-modal Contrastive Representations (MCR) learning aims to acquire semantically aligned cross-modal representations by pretraining the model on large-scale paired data. These aligned representations play a pivotal role in downstream comprehension and generation tasks. Inspired by the success of CLIP [20] , many works try to learn contrative representations for two modalities [20, 21, 22, 23, 24] . CLIP [20] and ALIGN [25] learn shared image-text representations from million-level image-text pairs. CLAP [26, 27] learns the audio-text representation, and CAV-MAE [28] focus on acquiring shared audio-visual feature space. C-MCR [19] focuses on learning new representation space by connecting the pre-trained spaces through overlapping modality.Apart from aligning two modalities, shared representations for more than three modalities attract increasing attention. AudioCLIP [2] and WAV2CLIP [29] train an audio encoder aligned with CLIP using audio-text-image triplets data. ULIP [3, 4] and openshape [5] construct 3D-imagetext triplets data through rendering 3D mesh into 2D images and captioning images for textual description, thereby learning a corresponding 3D encoder for image-text MCR space. Furthermore, Imagebind [12] exclusively utilizes data pairs between various modalities and images to expand CLIP with multiple modal alignment encoders.However, these methods heavily rely on large-scale, high-quality paired data collected from the internet or generated automatically and exceptionally high computational resources. Due to the lack of high-quality paired data for more modal combinations, such as audio-visual and text-3D, the extensibility of representation learning is notably constrained. Furthermore, the exceedingly high computational costs also diminish the flexibility of MCR learning.Audio-visual-text and 3D-visual-text learning have significant applications in multi-modal recognition [30, 31, 32, 33] , localization [34, 35, 36, 37, 38, 39, 40, 41? ] , question-answer [11, 10, 42, 43, 44] , and generation [45, 46, 47, 48, 49, 50] . They also play important roles in robot-related tasks such as human-machine interaction and synthetical information obtaining in complex environments [51, 52] .Previous unified spaces, such as AudioCLIP [2] and ULIP [3, 4] , mainly focus on automatically collecting or generating more paired data, but they are limited by the relatively low quality of the training datasets. Imagebind [12] employed individual vision-aligned data instead of triplets but pre-training the encoders from scratch results in high computational costs. FreeBind [53] and OmniBind [54] achieve strong modality alignment by integrating representation spaces that simultaneously contain multiple instances of the same modality. These two methods mainly focus on enhancing modality alignment within existing spaces, whereas Ex-MCR is a training paradigm designed to construct new modality alignments. Our approach uses paired-free data and minimal computational resources, yet it still achieves superior performance in audio-image-text and 3D-imagetext retrieval. More importantly, Ex-MCR is orthogonal to existing data-driven solutions, allowing it to be flexibly used in parallel with the large-scale pre-training unified space for even stronger performance.3 Extending Multi-modal Contrastive RepresentationsQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See in Appendix F.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
extending rather than connecting: 
 Given two pre-trained MCR spaces on modalities (A, B) and (B, C), C-MCR [19] employs two projectors to map them into a new shared space, where the alignment of different spaces can be learned from overlapping modality B. Since each pre-trained space intrinsically contains the alignment of (A, B) and (B, C), the alignment learned from overlapping modality theoretically can be transferred to the non-overlapping modalities. Specifically, for aligning different spaces, the embeddings of B are aligned in the new space, and pseudo (A, C) pairs retrieved by the same data of B are also aligned for a more comprehensive inter-space alignment. Moreover, the embeddings of different modalities within the same space are realigned to close the modality gap [55] , which significantly enhances the transferability of learned inter-space alignment. C-MCR shows remarkable flexibility and versatility since connecting two existing spaces only requires two learnable projectors and unpaired unimodal data.However, as C-MCR is designed to learn a new latent space for the two non-overlapping modalities (A, C) and projects them onto this space, a significant amount of information from their original spaces is lost in the projection process.. As a result, it faces challenges in concurrently establishing connections among three or more spaces. Therefore, C-MCR is not suitable for learning a unified representation space for more than three modalities. To learn unified multi-modal representations in a training-efficient and paired-data-free manner, we propose to extend one space into another space rather than connect two spaces to a new space. Considering the two spaces on modalities (A, B) and (B, C), Ex-MCR chooses one as the base space (A, B), and the other as the leaf space (B, C). In the "Extending" scheme, the base space is frozen, and we only train one projector to map leaf space to base space via the overlapping modalities B. Specifically, we employ the native pairs of B and pseudo pairs generated by B to align leaf space to base space. Simultaneously, we close the modality gap between (B, C) modalities of leaf space, thereby facilitating more transferable alignments.In contrast to C-MCR, Ex-MCR can conveniently expand more spaces and learn unified representation for three or more modalities. Benefiting from efficient training and no need for paired data, we can flexibly align multiple leaf spaces to the same base space. In addition to explicitly establishing alignment among modalities of leaf space and base space, semantic alignment also emerges between extended modalities. Ex-MCR employs base space as a bridge for achieving semantic alignment among modalities in multiple leaf spaces.Before delving into the details of our learning pipeline, we first clarify the necessary symbols and notations. We align the ULIP (3D-image) and CLAP (audio-text) onto CLIP (image-text). As shown in Fig. 1 (a) , the unimodal data of audios A, texts T , images V , and 3D point clouds P are input to their corresponding encoders, and the set of the extracted feature is denoted as A A , T A , T I , V I , V U and P U , where superscripts A , I , U indicate representation space of CLAP, CLIP, ULIP, respectively. TheA A = {a A 1 , a A 2 , . . . a A na }where n a is the number of all audio data and a A i represents the CLAP feature of i-th audio. Similarly, there aret A i , t I i , v I i , v U i , o U i in T A , T I , V I , V U , P U respectively.In Ex-MCR, freezing base space allows us to maintain the original alignment of base space but also implies that the modality gap within base space is preserved. Consequently, it becomes necessary to map the leaf space to more suitable positions within the base space. To this end, we enhance the entire alignment learning pipeline from perspectives of data, architecture, and learning objectives.C-MCR only uses data of overlapping modality to retrieve semantically similar embeddings of other modalities and treats these generated embeddings as pseudo pairs (we call single modality-centric data). However, it is difficult to fully represent one modality with another, and retrieved embeddings by one modality often ignore some semantics of other modalities. For example, images about "mushrooms" tend to be absent when retrieving embeddings by audio, and audio of "wind noise" may be ignored in embeddings aggregated by images. Therefore, aggregating embeddings from only a single modality struggles to capture the entire representation space of different modalities.To tackle the above problem, we propose various modality-centric data strategy. By ensembling semantic consistent embeddings aggregated by multiple modalities, the final embeddings can reflect the representation space of different modalities in different MCRs more comprehensively. As depicted in Fig. 1 (b ), all modalities in two spaces are iteratively employed as queries to aggregate corresponding semantic consistent embeddings. Take aligning audio-text space to text-image space as an example, the consistent embeddings based on overlapping modality (e.g., text) are aggregated as follows:EQUATIONwhere the τ 1 is the temperature parameter of softmax, and the softmax is over all the samples in used datasets. The tilde symbols mean the features are processed to be semantically consistent. The tA i and tI i are derived from the same text data, and their semantics are natively consistent. Benefiting from the modality semantic alignment within each pre-trained space, the generated ãA i and ṽI i are also semantically relevant to the tA i and tI i . To capture the representation space of non-overlapping modality more comprehensively, we further aggregate semantic consistent embeddings via data of non-overlapping modality (e.g., audio and image). The process of generating embeddings based on audio can be expressed as:EQUATIONSince the embeddings of T A and T I of overlapping modality are one-to-one matched, the similarity weights between ãA i and T A can be naturally transferred to T I . Based on the aforementioned formulas, when extending audio-text to text-image, we iteratively employ texts, audios, and images as queries to aggregate corresponding semantic consistent embeddings. During training, semantic consistent embeddings from different sources are shuffled together and the final data pool of various modality-centric data can be represented as {ã A i , ṽI i , tA i , tI i } n i=0 .The main network structure of Ex-MCR is a projector, and it serves two purposes: 1) Learning the intra-space alignment to close the modality gaps within leaf space and prompt more stable alignment between spaces. 2) Learning the inter-space alignment for extending leaf space to base space. Considering these two different purposes, we propose a decoupled projector to alleviate the potential conflict between distinct optimization objectives and explore a more reasonable mapping layer design for these two purposes. As shown in Fig. 1 , the projector is decoupled into a linear layer f l (•) for intra-space alignment and a multi-layer perceptron layer f m (•) for inter-space alignment.For extending CLAP to CLIP, we first use f l to align ãA i to tA i , the loss function is defined as:EQUATIONWith the intra-space alignment loss, f l (•) learns the mapping between audio subspace and text subspace within the CLAP, thereby effectively closing the modality gap. Since the subspaces of different modalities within pre-trained spaces are very similar, linear mapping is enough to bridge the modality gap. Moreover, our experiments even found that activation layers hurt bridging the modality gap.After bridging the modality gap, the shared f m (•) are employed to map both audio and text embeddings of CLAP space to the CLIP space, which can be expressed as:EQUATIONTable 1 : Results of audio-image-text experiments. The best results are bolded. Objaverse-LVIS COCO Acc@1 Acc@3 Acc@5 R@1 R@5 R@1 R@ Since the modality gap within the base space is still preserved, a more robust learning objective is needed to map leaf space to the appropriate position in the base space. To this end, we propose to learn the alignment densely among the quadruple semantic consistent embedding pairs described in Sec.3.2.1. When extending CLAP to CLIP, the dense inter-space alignment losses are defined as:Audio-Image Audio-Text Image-Text Method FlickrNet AVE VGGSS AudioCaps COCO R@1 R@5 R@1 R@5 R@1 R@5 R@1 R@5 R@1 R@5 CLAP - - - - - -EQUATIONwhere the InfoNCE(•, •) is the standard contrastive loss function, which is defined as:EQUATIONwhere the τ 2 is the temperature parameter. The overall loss is defined as a weighted combination of the intra-space and inter-space losses:EQUATION)where λ is the hyper-parameter to balance the two terms.Various modality-centric data 3. A i , t I i , v I i , p U i . The t I i , v I i , f A m (f A l (a A i )), f U m (f U l (p U i )) are the final audio-text-image-3D unified representation learned by Ex-MCR, where the Implementation Details For Ex-MCR-base, We employ pre-trained frozen CLIP ViT-B/32 [20] , CLAP [27] , and ULIPv2 (PointBERT version) [4] models. We also extend CLAP's audio encoder to OpenCLIP ViT-H [56] to build the Audio-Image-Text space Ex-MCR-huge in parallel with ImageBind [12] . The temperature τ 1 in Eq.12 for embedding aggregation is set to 0.01 following [19] , while the τ 2 in Eq.6 is set to 0.05. The hyper-parameter λ in Eq.7 is set to 0.1. Following [19] , we also add Gaussian noise with a variance of 0.004 to the semantic consistent embeddings described in Sec.3.2.1. The linear projector f l (•) is a simple linear layer, and the MLP projector f m (•) is a 2-layer MLP. We train our model with a batch size of 4096 for 36 epochs. We employ the AdamW optimizer with an initial learning rate of 1e-3 and a cosine learning rate decay strategy.f A m (•), f A l (•); f U m (•), f U l (•)Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification:We are using open-source data and the code can be found in the supplemental material. Guidelines: Justification: A description of the broader impacts can be found in appendix E. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
audio-image-text results: 
 Downstream Tasks We employ zero-shot audio-image, audio-text, and image-text retrieval tasks to evaluate the audio-image-text representations of Ex-MCR. For audio-image retrieval, we conduct evaluations on Flickr-SoundNet [57] , VGGSS [39] , and AVE [58] datasets. Due to their small dataset sizes, we utilize all their available data, comprising 5,000, 5,000, and 4,097 samples. For audio-text retrieval, we utilize the validation set from the AudioCaps [59] dataset, which includes 964 audio samples, and for each audio, there are 5 corresponding captions for retrieval. Regarding image-text retrieval, we employ the validation set of COCO [60] dataset, consisting of 5,000 images and 25,014 text captions. We calculate the cosine similarity between modalities in representation space and use Top-1 and Top-5 metrics for performance comparison.Performance Comparison In the upper part of Fig. 1 , we compare Ex-MCR-base to WAV2CLIP, AudioCLIP, and C-MCR. Notably, even without using audio-image paired data, Ex-MCR-base achieves significantly better performance over WAV2CLIP and AudioCLIP, which illustrates that Ex-MCR is a more effective representation learning method when high-quality data pairs are limited. Furthermore, compared to C-MCR, Ex-MCR not only achieves better audio-image alignment but also inherits more audio-text alignment from CLAP, fully retaining CLIP's image-text modal alignment, suggesting that Ex-MCR is generally superior to C-MCR in both establishing new Spaces and maintaining original ones. We then compare the performance of Ex-MCR-huge and data-driven alignment-building methods in the bottom half of Fig. 1 . Inheriting the audio-text alignment of CLAP, Ex-MCR-huge achieved better results on audio-text retrieval tasks, while ImageBind, trained directly with Audio-Image pairing data, has better audio-image performance. We were pleasantly surprised to find that using Ex-MCR and data-driven methods in parallel, with very little additional cost, can complement each other to achieve a state-of-the-art unified audio-image-text representation.Downstream Tasks To evaluate the performance of 3D-image-text space learned by extending ULIP to CLIP, we conduct a zero-shot 3D object classification task to assess the alignment between 3D and text. We also perform zero-shot 3D-image and image-text retrieval tasks to evaluate the alignment between 3D and image, as well as image and text. The zero-shot 3D object classification task is carried on the ModelNet40 [61] validation set, and we use the same prompt strategy as [4] .Regarding the zero-shot 3D-image retrieval task, we use the Objaverse-LVIS dataset [62] , which includes 46,054 3D objects. Additionally, we continued to use the COCO dataset's validation set for zero-shot image-text retrieval.It is worth noting that ULIP aligns a 3D encoder to a vision-language model called SLIP [63] (not CLIP) through 3D-image-text data. Ex-MCR only uses the aligned 3D-image representation of ULIP to extend it to a different vision-language model (i.e., CLIP) via the paired-data-free way. So we are not reproducing or refining the alignment of ULIP, but building a new alignment from scratch between the 3D representation of ULIP and CLIP.Performance Comparison From Tab.3.2.3, we can find the following key points:1) Even without using any 3D-text data, Ex-MCR still outperforms the advanced models (ULIP and ULIP v2) trained on 3D-text pairs in most performance metrics for 3D object classification.2) For 3D-image retrieval, since the 3D-image space of ULIPv2 is treated as leaf space, it is reasonable that Ex-MCR-base 3D-image performance is slightly lower than ULIPv2. At the same time, the better 3D-image retrieval accuracy than ULIP and C-MCR shows that Ex-MCR effectively learns strong 3D-image alignment.3) Ex-MCR retains the best image-text retrieval accuracy compared to these previous state-of-the-art models. The leading performance on all these tasks further demonstrates the superiority of Ex-MCR in unified contrastive representation learning. In this section, we study whether the semantic alignment also emerges between the extended modalities (e.g., audio and 3D). We mutually retrieve audio in AudioCaps and 3D objects in Objaverse. In Fig. 4 .4 and 3, we provide visualizations of some top-5 retrieval results, and audios are described by their corresponding caption annotations. These cases effectively demonstrate the emergent semantic alignment between audio-3D in Ex-MCR space. For example, the sound of a flushing toilet and water flow can retrieve 3D objects of toilets or sinks, while a sailboat 3D object can retrieve clips containing sounds of water vessels and wind.These exciting results demonstrate that extending ULIP and CLAP onto CLIP following our Ex-MCR methods derives a 3D-image-text-audio unified contrastive representation space. In addition to the state-of-the-art performance on all possible tasks, Ex-MCR is an extremely training-efficient and paired-data-free representation learning method, which amplifies its application value in unified multi-modal representation learning. To further support the conclusion, we provide more audio-image retrieval results and the original audio files in the supplementary material.In this section, we analyze the main components of Ex-MCR. All experiments are conducted on extending CLAP to CLIP, and we reported the average mAP of audio-visual and audio-text retrieval, respectively. In addition, we also provide ablation results on full evaluation metrics in the Appendix. Various modality-centric data As described in Sec.3.2.1, we employ various modality-centric data to train our projectors. For investigating the effect of different modality-centric data, we ablate each modality-centric data, and the results are reported in Tab.3. The A, I, and T represent pseudo data derived from audio, image, and text respectively. Each kind of data is beneficial for audio-visual and audio-image alignment, and using all kinds of data simultaneously brings the best performance. In addition, we find that pseudo-pairs from audios are critical to the performance of audio-text retrieval, demonstrating the importance of various modality-centric data, and proving that previous single modality-centric data really can not fully reflect the audio representation space.To analyze the impact of different alignment objectives, we train the model with each alignment objective. From the results reported in Tab.4, we find that directly aligning the pseudo audio-image or audio-text embedding pairs leads to sub-optimal audio alignment, whereas aligning spaces by overlapping text modality brings better alignment than learning alignment directly from pseudo pairs. This observation further suggests that overlapping modalities play a key pivotal role in aligning different spaces.Structure of f l (•) Tab.5 demonstrates the impact of different structures of f l (•). The results prove our hypothesis: the representation structures between different modalities within one MCR space are similar, and a simple linear layer is enough to bridge the modality gap. Moreover, the activation layer of the MLP introduces non-linearity, which may disrupt the spatial structure of representations.The ablation studies of f m (•) are summarized in Tab.6. When aligning different MCR spaces, the nonlinear MLP structure with stronger expressivity is better than the simple linear layer. Besides, good results are achieved no matter how many layers of MLP, which demonstrates the robustness of our method. According to more detailed experiments in Tab.11, empirically, MLP with 2 or 3 layers achieves a good balance between expressivity and learning difficulty.Training hyperparameters τ 2 and λ: The results of ablation experiments show that the performance is insensitive to the τ 2 in Eq6 and λ in Eq7. So the picked τ 2 is 0.05 which is commonly used and the picked λ is only to equal the absolute value of different loss terms. For detailed experimental results, please refer to the table in Appendix C. 
conclusion: 
 This paper proposes Extending Multi-modal Contrastive Representations (Ex-MCR), a novel trainingefficient and paired-data-free unified constrastive representation learning method for more than three modalities. Ex-MCR effectively integrates the knowledge in pre-trained spaces through overlapping modalities between these spaces. By extending ULIP and CLAP onto CLIP via the overlapping image and text modality, respectively, we derive unified and high-quality audio-image-text-3D representations. Additionally, Ex-MCR provides a new view to build unified representations. Even without using paired data, Ex-MCR still achieves competitive performance, and when combined with data-driven approaches, it complementarily enhances unified representation spaces, leading to state-of-the-art results across various tasks. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All the creators or original owners of assets used in the paper are properly credited and the license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The details of our training dataset, which are mentioned in Sec.4.1, are shown below.Text Dataset To ensure that the texts contain sufficient information for other modalities, the data of text is sourced from diverse perspectives in image-text datasets (COCO, CC3M), video-text datasets (MSRVTT, MAD), and audio-text datasets (AudioCaps, Clotho). Following [19] , we select 1M texts from CC3M. There are 2.33M text samples in total. We extract their CLAP and CLIP features T A and T I using the CLAP and CLIP encoders, respectively.Image Dataset For another modality in base space, Image, we utilize ImageNet1K as the data source.ImageNet1K is a large-scale image recognition dataset consisting of 1.3 million images. We extract their features to the sets V I , and V U in CLIP and ULIP, using the CLIP Encoder and ULIP Encoder.Audio Dataset AudioSet is a large-scale audio dataset with 2.1M audio clips from YouTube, equivalent to 5.8 thousand hours of audio and encompassing over 500 sound classes. We use the CLAP audio encoder to extract the feature set A A from the audios of the training set.3D Point Cloud Dataset For the 3D modality, we use Objaverse, the recently released and large-scale 3D objects dataset. It has approximately 800K real-world 3D objects. All 3D data are transformed into point clouds and extracted into the feature set P U using the ULIP 3D encoder.It is worth noting that we do not employ any annotations provided with the datasets mentioned above as part of our training data, which means we only use the unimodal modality of data in each dataset we selected. As a supplement to Tab.3, Tab.4, Tab.5, and Tab.6, we provide detailed ablation experiment results on more comprehensive evaluation metrics of various datasets, as shown below. Collecting a group of pseudo datasets takes about 10 hours on a single 4090 while using 12GB GPU memory. The training times for projectors between two spaces are approximately 1.5 hours, on a single 4090, and it only requires 3GB of GPU memory.This paper introduces Ex-MCR, a paired-data-free and training-efficient method for constructing a unified multimodal representation space. While this method offers flexibility in constructing a new unified representation space, its training process, which does not necessitate paired data, may inadvertently create unintended associations within the constructed representation space. The alignment of the representation space is primarily influenced by the pre-training space utilized and the unimodal data, both of which need to be restricted to prevent potential misuse for unethical applications.Currently, larger and more advanced models are continually emerging across various modalities, leading to increasingly stronger alignment in pre-trained contrastive representations. Although the experimental results indicate that Ex-MCR already demonstrates significant advantages at comparable model scales, considering its flexibility as a general paradigm, utilizing these advanced models to explore the upper limits of this learning approach would be an exciting research direction.NeurIPS Paper ChecklistWAV2CLIP is trained on VGG-Sound. Its retrieval results on VGGSS are supervised, while other results are zero-shot. 


Decision: The current ratings are 5, 7, 5, 5 (one of the reviewers expressed to increase his rating but did not actually did it in the system). AC saw that maybe the only concern that is left seems to be **3D-Image-Text performance remains low**. Authors provided some verbal explanations but AC remains somewhat unconvinced on that point.

However, in view of the ratings and it seems most except one concerns have been addressed, this paper looks ok to be accepted. AC urges the authors to address all the points raised in the discussion in the final manuscript.

Reviews: review: paper_summary: paper_summary: This paper proposes Extending Multimodal Contrastive Representation, a training-efficient and paired-data-free method to build unified contrastive representation for many modalities. Without using paired data, Ex-MCR achieves comparable performance to advanced methods on a series of audio-image-text and 3D-image-text tasks and achieves superior performance when used in parallel with data-driven methods.
strength_weakness: strength_weakness: 1. The contribution of this paper is the multimodal contrastive learning method. The multimodal contrastive learning method proposed by the authors is not very novel, because contrastive learning is widely used in the field of multimodal retrieval.
2.  The method proposed by the author is too simple, and it is recommended to deepen the improvement of the method.
questions: questions: 1. In Figures 2 and 3, which are the images that are retrieved correctly, the author should mark the images that are retrieved correctly and the images that are retrieved incorrectly.
2. In Tables 1 and 2, the author's method is lower than the compared method in some parts. How can the advantages of the author's method be explained?
limitations: limitations: See the weaknesses section
paper_summary: paper_summary: The paper introduces Extending Multi-modal Contrastive Representations (Ex-MCR), a novel method for learning unified contrastive representations across multiple modalities without the need for paired data. Ex-MCR extends one modality's representation space into another, preserving semantic alignment and reducing training costs. The method enhances the learning pipeline with modality-centric data, a decoupled projector, and a dense alignment objective. Experiments show that Ex-MCR achieves state-of-the-art performance on multi-modal tasks, highlighting its efficiency in representation learning.
strength_weakness: strength_weakness: - The effectiveness of the proposed method heavily depends on choosing hyperparameter in Eq.(7). A sensitivity analysis of hyperparameter is necessary, and discussing insights into choosing this hyperparameter is essential.

- The performance improvements of the proposed method on 3D-image-text experiments are relatively limited. Especially, Ex-MCR is so far behind ULIP-v2 in the 3D image domain. The authors should explain them clearly.

- Ex-MCR requires additional training cost and inference time compared to the previous method C-MCR. The authors should give more analysis.
questions: questions: Please refer to the weakness above.
limitations: limitations: None
paper_summary: paper_summary: This paper focus on the multi-modal contrastive representation of more than 3 modalities in multi-modal learning. To address the flaws of existing works, such as the dependency on large-scale, high-quality paired data and the expensive training cost, this paper introduces Ex-MCR, a training-efficient and paired-data-free method to build unified contrastive representation for multiple modalities. The key idea of the proposed method is to extend one modality's space into the other's. Experiments are conducted in audio-text and 3D image representations to the existing vision-text space.
strength_weakness: strength_weakness: 1. The abstract claim 'a significant amount of information from their original spaces is lost in the projection process' and the statement 'C-MCR mainly focuses on learning a new space for the two non-overlapping modalities, while the modality alignments in powerful original pre-trained spaces are forgotten' should be supported by explicit evidence, experimental results, or theoretical analysis.

2. There are various methods in the multi-modal learning area, and projection/projector-based methods are not the only option for binding different modalities. Importantly, there is overlapping information across different modalities, such as semantics, as pointed out by [1], and experiments in [1] have proved this.

   [1] UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All, CVPR 2024.

3. The claim in the contribution part, lines 48-51, 'Such as simple yet effective approach maximizes the preservation of modality alignment within base space, demonstrating great potential for augmenting existing unified space and integrating more pre-trained spaces,' should be supported by examples conducted by the authors to validate this claim.

4. The proposed method also integrates projectors, a.k.a., the decoupled projector. What are the differences between the proposed one and existing ones?

5. Many related works are not mentioned, not only in the related work section but also in the experimental comparison. These include but are not limited to:

   [2] Imagebind: One embedding space to bind them all, CVPR 2023.
   
   [3] Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following, arXiv 2023.
   
   [4] Onellm: One framework to align all modalities with language, CVPR 2024.
   
   [5] Meta-transformer: A unified framework for multimodal learning, arXiv 2023.
questions: questions: 1. **Evidence for Information Loss**:
   - Can the author provide experimental results or theoretical analysis to support the claim of information loss during the projection process?

2. **Support for Modality Alignment Forgetting**:
   - Are there specific examples or experiments that demonstrate the forgetting of modality alignments in original pre-trained spaces?

3. **Comparison with Non-Projection-Based Methods**:
   - How does the proposed method compare with non-projection-based methods, especially those highlighting overlapping semantic information as discussed in UniBind?

4. **Validation of Contribution Claims**:
   - Can the author provide examples or experiments that validate the claim about preserving modality alignment and integrating more pre-trained spaces?

5. **Differences in Decoupled Projector**:
   - What are the key differences between the proposed decoupled projector and existing projectors?

6. **Inclusion of Related Works**:
   - Can the author include comparisons and discussions involving the following works to provide a more comprehensive view?
     - Imagebind: One embedding space to bind them all, CVPR 2023.
     - Point-bind & point-llm: Aligning point cloud with multi-modality for 3D understanding, generation, and instruction following, arXiv 2023.
     - Onellm: One framework to align all modalities with language, CVPR 2024.
     - Meta-transformer: A unified framework for multimodal learning, arXiv 2023.
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces the Extending Multi-modal Contrastive Representation (Ex-MCR), an efficient method for learning multi-modal contrastive representations without relying on paired modality data. Unlike the previous C-MCR scheme, which connects two pre-trained modalities to a new embedding space, Ex-MCR extends one modality (leaf) to another (base), while the base modality remains frozen. This approach has the merit of preserving the alignment of pre-trained modalities in their original space while facilitating a unified modality space via an overlapping modality (e.g., using text to map audio-text to text-image). Experimentally, pre-trained audio-text and image-3D representations are aligned within the image-text modality space without requiring paired modality data, achieving superior performance over other methods.
strength_weakness: strength_weakness: - My major concern lies in the overall clarity of this paper. Although Section 3.1, which explains the underlying motivation, was clear, I found Fig. 1(b) and Section 3.2.1 difficult to understand despite several rounds of reading. 

  - For Figure 1(b), it is difficult to interpret what is happening in each modality. It is suggested that more illustrative figures be depicted and supplemented with detailed captions. 

  - For Section 3.2.1, overall clarity is limited. It seems to describe the construction of pseudo-modality pairs by retrieving similar embeddings from other modalities one by one, and incorporating various enhancing techniques. From Equation 1, the notations are inconsistent between the first and second lines. For consistent audio embeddings, T^A is used, while for consistent image embedding, V^I is used. In the formulation for retrieving similar audio samples given text as a query, should it be A^A instead of T^A in the first line? In a similar context, it would be desirable to verify the notations in Equation 2. Moreover, the clarity of the process and general description of modality-centric data pool could be further enhanced. 


- No clear evidence can be found that Ex-MCR improves the alignment of inter-modalities over C-MCR. For audio-image and 3D-image tasks in Tables 1 and 2, Ex-MCR conveys only slight improvements over C-MCR.


- Sensitivity analysis on the hyper-parameters such as \lambda and \tau is not provided.
questions: questions: - From [L303-304], the terms 'audio-visual' and 'audio-image' seem to have duplicative meanings.

- As an oracle, how can the proposed Ex-MCR scheme boost performances when trained on paired modality data?
limitations: limitations: It adequately touches the limitations.

Comments: title: Please respond immediately, reviewer n4Yo
Additional Question: train on Audio-Image pair
comment: Dear reviewer KwrG,

Thank you for your constructive comments. We are glad we could address your concerns regarding the paper. We will discuss the related works you provided in the revised version.

Once again, thank you for your efforts in the review.

Best regards,
Paper 6854 authors
Reviewer 1W4Z:

Any chance you can provide more details in your response? This will help the authors. Please do it immediately as this is the last day for author-reviewer discussion.

-AC
Reviewer n4Yo:

Please let us know whether the authors have addressed your concerns.

Thanks.

-AC
Thanks for your efforts during the rebuttal phase. Most of my concerns have been addressed. I will raise my score accordingly. Additionally, some influential previous works are not discussed such as UMT/UME[1], QMF[2], MMParato[3], and ReconBoost[4]. I encourage authors to discuss them in the revised version.

[1] On Uni-Modal Feature Learning in Supervised Multi-Modal Learning. ICML 2023.
[2] Provable Dynamic Fusion for Low-Quality Multimodal Data. ICML2023
[3] MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance. ICML 2024
[4] ReconBoost: Boosting Can Achieve Modality Reconcilement. ICML 2024.
Dear Reviewer KwrG,

We are very thankful for your valuable feedbacks and comments! Please let us know if you have further questions regarding the rebuttal response or any other questions related to the paper. We are looking forward to your further feedbacks!

Thanks,

Authors of Paper Submission 6854
Dear Reviewer n4Yo,

We are very thankful for your valuable feedbacks and comments. Please let us know if you have further questions regarding the rebuttal response or any other questions related to the paper. We are looking forward to your further feedbacks!

Thanks,

Authors of Paper Submission 6854
Thank you for the additional experiments and further clarifications.

I believe the clarity of the paper has improved based on the authors' responses. I will increase the score to 5.
Dear Reviewer V8oT,

Ex-MCR utilizes a pre-trained CLAP audio representation extractor, where audio is inherently aligned with text, rather than image. Audio representations aligned with text and audio representations aligned with images contain complementary information. Therefore, to maintain this alignment with text, we aggregate pseudo-text embedding for the ground-truth Audio-image pair from Audioset and limit the fine-tuning steps. As the result shown below, over-fitting to the Audio-Image pair can affect the Audio-Text alignment. 

| steps | Average Audio-Image performance(R@5) | Average Audio-Text performance(R@5) |
| ----- | ------------------------------------ | ----------------------------------- |
| 0     | 8.43                                 | 59.60                               |
| 500   | 13.24                                | 62.11                               |
| 1000  | 14.10                                | 63.18                               |
| 2000  | 15.81                                | 62.74                               |
| 5000  | 16.06                                | 61.68                               |
| 8000  | 16.23                                | 59.27                               |

On the other hand, averaging the audio representations which are respectively aligned with text and image in the same space is a train-free and more economical, effective method to balance Audio-Text and Audio-Image alignment than conducting large-scale training on paired data. The experimental results are as follows:

| Model                                      | Flickr           | AVE              | VGGSS          | ACaps             |
| ------------------------------------------ | ---------------- | ---------------- | -------------- | ----------------- |
|                                            | R@5              | R@5              | R@5            | R@5               |
| ImageBind                                  | 20.78            | 40.11            | 35.67          | 27.47             |
| IB+Ex-MCR-huge                             | 21.26(+0.48)     | 38.95(-1.16)     | 37.55(+1.88)   | 47.44(+19.97)     |
| IB+Ex-MCR-huge finetune on A-V paired data | **21.85(+1.07)** | **40.15(+0.04)** | **38.55(+2.88)** | **48.18(+20.71)** |

We feel sorry for not conducting experiments on the same scale as ImageBind due to the limited time. However, the results above clearly demonstrate that Ex-MCR, by integrating representations from different sources of knowledge, achieves significantly better overall performance than ImageBind at a much lower additional computational cost.

Thank you again for your comments and we hope the experimental results and analysis provided above can address your concerns. If you have any other questions, please feel free to discuss them with us.

Best regards, Authors
Thanks for the authors' response, I will keep my initial rate
Thank you for presenting the additional experiments. 

Despite using audio-image paired datasets for training, what could be the reason for the low performance (R@5) in audio-image retrieval tasks such as Flickr and AVE? Could you provide any insights into the reasons behind this?
Dear Reviewer V8oT,

Thanks for your comments. To address your concerns, we conducted experiments on Ex-MCR-huge using paired audio and images from Audioset. As you mentioned in your review, we performed training from scratch and fine-tuning(1000 steps, less than 15 minutes) on the ”standard“ Ex-MCR-huge model in the paper, using these paired data. Please note that the relevant text embeddings are still constructed through retrieval and aggregation. The experimental results are as follows:

| Model                                   | Flickr    | AVE        | VGGSS     | ACaps     |
| --------------------------------------- | --------- | ---------- | --------- | --------- |
|                                         | R@5       | R@5        | R@5       | R@5       |
| Ex-MCR-huge                             | 6.16      | 7.36       | 11.77     | 59.60     |
| Ex-MCR-huge train on A-V paired data    | 8.63      | 12.44      | 18.05     | 62.63     |
| Ex-MCR-huge finetune on A-V paired data | **9.37** | **13.09** | **19.84** | **63.18** |

All retrieval metrics in both experimental settings outperform the "standard" Ex-MCR-huge, especially for the Audio-Image metrics.  Similar to the results from fine-tuning on Audio-Text paired data, Ex-MCR's alignment in both Audio-Image and Audio-Text tasks benefit from the Audio-Image pairs.

Furthermore, the experimental results show that fine-tuning the "standard" Ex-MCR-huge outperforms training from scratch. We believe that the pseudo pairs aggregated by the Ex-MCR method and the real data pairs provide complementary knowledge.

We would be happy to discuss our paper in detail if you have additional comments.

Best regards, Authors
Thank you for the response, which addressed many of my initial concerns.

To further clarify my previous question, I believe it is crucial to present the oracle performance to validate the uni-modal training scheme. 
Specifically, how much does the model perform when trained on paired audio-image datasets? In addition, how much does the proposed Ex-MCR model benefit from a small amount of such paired audio-image training data after fine-tuning?