OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces
Authors: Zehan Wang
Ziang Zhang
Minjie Hong
Hang Zhang
Luping Liu
Rongjie Huang
Xize Cheng
Shengpeng Ji
Tao Jin
Hengshuang Zhao
Zhou Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal representations
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recently, human-computer interaction with various modalities has shown promising applications, like GPT-4o and Gemini. Meanwhile, multimodal representation models have emerged as the foundation for these versatile multimodal understanding and generation pipeline. Models like CLIP, CLAP and ImageBind can map their specialized modalities into respective joint spaces. To construct a high-quality omni representation space that can be shared and expert in any modality, we propose to merge these advanced models into a unified space in scale. With this insight, we present \textbf{OmniBind}, advanced multimodal joint representation models via fusing knowledge of 14 pre-trained spaces, which support 3D, audio, image, video and language inputs. To alleviate the interference between different knowledge sources in integrated space, we dynamically assign weights to different spaces by learning routers with two objectives: cross-modal overall alignment and language representation decoupling. Notably, since binding and routing spaces only require lightweight networks, OmniBind is extremely training-efficient. Extensive experiments demonstrate the versatility and superiority of OmniBind as an omni representation model, highlighting its great potential for diverse applications, such as any-query and composable multimodal understanding.

Published as a conference paper at ICLR 2025
OMNIBIND: LARGE-SCALE OMNI MULTIMODAL REP- RESENTATION VIA BINDING SPACES
Zehan Wang" 1.2, Ziang Zhang"), Minjie Hong', Hang Zhang', Luping Liu", Rongjie Huang', Xize Cheng's Shengpeng Ji', Tao Jin Hengshuang Zhao?, Zhou Zhao 1,2 'Zhejiang University; Shanghai AI Lab; The University of Hong Kong
http: / ht//m/iniiinin//.. www.in..g..hh.......... github. io
a). OmmiBind
b). Omni-alignment
Vi1000
Shared Representation Space
Masiii Bar
kd
D(Chiie
CLAP
EVA-CLIP
CLIP
......
......
Scom
WayCaps
Obgn vill
Unii 3 Modell
ImagelBind
SigLIP
Uni3D
Scutkpa
"A cute yellow dog with n red collar."
[dop ranning turking)
313 object
Andio
Image/Video
Language
Figure Overview of OmniBind. OmniBind integrates diverse knowledge of various existing multimodal models, leading to large-scale omni representations, OmniBind exhibits remarkable versatility and achieves saae-offthhe-e -frt results on extensive downstream tasks over all modality pairs.
ABSTRACT
Recently, human-computer interaction with various modalities has shown promis- ing applications, like GPT-4o and Gemini. Meanwhile, multimodal representa- tion models have emerged the foundation for these versatile multimodal derstanding and generation pipeline. Models like CLIP, CLAP and ImageBind can map their specialized modalities into respective joint spaces. To construct a high-quality omni representation space that can be shared and expert in any modality, we propose to merge these advanced models into unified space scale. With this insight, we present OmniBind, advanced multimodal joint rep- resentation models via fusing knowledge of 14 pre-trained spaces, which support 3D, audio, image, video and language inputs. To alleviate the interference be- tween different knowledge sources integrated space, we dynamically assign weights to different spaces by learning routers with two objectives: cross-modal overall alignment and language representation decoupling. Notably, since bind- ing and routing spaces only require lightweight networks, OmniBind is extremely training-efficient effiiceent. Extensive experiments demonstrate the versatility of OmniBind an omni representation model, highlighting its great potential for diverse appli- cations, such as any-query and composable multimodal understanding.
INTRODUCTION
Multimodal joint representation. which aligns different modalities into shared space, forms the
foundation of current multimodal understanding (Liu et al.) 2024a 2023; Bai et al.) 2023 ZZu
al. 2023b) and generation pipelines (Rombach al.] [2022] Podell et al. [203]] Singer et al.]
2022
Huang et al.] (023b).. Recently, co-understanding and generating various modalities with
Equal Contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2025
omni model has attracted increasing attention and demonstrates promising application prospects, like GPT-4o (OpenAl 2024) and Gemini (Tea et al. 2023).
Scaling up achieves incredibly success in language (Achiam et al. 2023 Touvron et al. 2023a
Jiang ct al. 2024| Touvron et al.) 2023b) and vision-language models (Alayrac et al. 2022] Liu
al. 2023 Sun al., 2024). It consistently improves model performance and generalization by increasing the seen data and model parameters. In the field of multimodal representation models,
some recent work scale models and data, but they are limited to certain combinations of modalities,
such as image-text (Chen et al. 2023] Sun et al. (2024). video-text (Wang et al. 2022] (024a),
audio-text (Wu* al.] 2023] Mei et al. 2023). audio- -magge (Gong et a [2022] Girdhar et al. (023)
and 3D-image-tex (Zhou et al. (002). Due to the lack of large-scale datasets covering various modalities, is difficult to train omni multimodal representation models from scratch.
In this paper, we present OmniBind, framework that binds advanced multimodal models, each excelling within their respective modalities (as depicted in Fig. Do). By integrating the alignment knowledge between different modalities, our approach achieves onni-alignment in space shared by various modalities. Furthermore, by inheriting the pre-trained knowledge from these specialized models, our approach significantly lower the requirements of computational resources and training data. The entire training process can be completed with only several million unpaired data points using 4090 GPUs. The integration strategies and low resource demands makes our framework highly flexible and able to capitalize on improvements with any new pre-trained multimodal models.
However, it is non-trivial effectively integrate numerous pre-trained spaces. Methods like remap- ping and weighted-averrging as suggested in WWan et al. (2024b), fail to scale effectively when handling multiple source spaces. As more spaces are integrated, interference between the knowl- edge of different sources increases, leading to suboptimal performance. Manually adjusting the combining factors only results in trade-offs between different expertise rather than creating a truly versatile model. We attribute this interference and the associated trade-offs to the rigidity of fixed weights. Since existing spaces are trained for different purposes using varied datasets, they encapsulate knowledge specific to certain aspects. Fixed-weige weight averaging tends to favor only par- ticular aspects of knowledge, limiting the overall knowledge integration.
To unleash the potential of the integrated space that inherently contains all knowledge, we introduce a weight routing strategy designed to optimize the integration of different spaces. For each modality, we use a learnable router that dynamically determines the combining weights for different inputs. The training process of gating is guided by two primary goals: cross-modal overall alignment and language representation decoupling. The former motivates routers predict optimal weights for all modality combinations, while the latter reduces conflicts among text embeddings aligned to different modalities, ensuring the clarity and distinction of language representations.
With the above techniques, we have successfully constructed three models that bind 5,13, and 14 spaces respectively. For 3D, audio, and image classification, our model exhibits advanced zero-shot generalization capabilities. Furthermore, it achieves significantly improved cross-modal alignment across all possible modality pairs compared to existing multimodal representations. This high- quality semantic omni-alisn amni-alignment enables advanced applications, including accurate 3D-audio re- trieval, any-query object localization/audio separation, and complex compositional understanding.
Our contributions can be summarized as follows:
1) We propose OmniBind, series of omni multimodal representation models that bind 5,13, and 14 spaces respectively and support five mainstream modalities: 3D point, audio, image, video, and language. It emphasizes the value of piecing various pre-trained specialist models together.
2) We introduce routers to ensemble spaces pre-trained on various modalities and datasets, thereby mitigating interference between knowledge from different sources and further enhancing versatility.
3) We design two learning objectives for learning routers: cross-modal overall alignment and lan- guage representation decoupling, which motivate routers to dynamically predict the optimal com- bining weights for all modality pairs while reserving the discrinination of representations.
4) OmniBind exhibits state-of-the-art performance on 14 benchmarks that cover all the modality pairs, and great potential for diverse applications, such as 3D-audio retrieval and any-query separa- tion/localization, while requiring minimal training resources and data.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 MULTIMODAL JOINT REPRESENTATION
Multimodal joint representation mainly aims to map inputs from different modalities into shared space. Leveraging the semantic alignment property, pre-trained representation models are widely utilized in current multimodal large language models, such LLaVA (Liu et al. [024a
2023
ImageBind-LLM. (Han et al., 2023). and Chat-3D (Wang et al. (203aa) Huang et al.
2023a). as well
multimodal generation pipelines like Stable Diffusion (Rombach al.; 2022
SD XI. (Podell
et al. 2023), make-a-video (Singer et al. 2022) and make-an-audio (Huang et al. 2023b).
The initial multimodal representation is CLIP model (Radford et al. 2021), | which demonstrates im-
pressive generalization capabilities in various vision-language downstream tasks. Motivated by its success, successors propose stronger image-language representations by using higher-quality initial-
ization (Fang et al. 2023c Sun et al.) 2023). larger-scale datasets (Schuhmann et al. [2022] [yeon
al. 2022). improved learning objectives (Zhai al. 2023), or better model architecture (Fang et al. 2023b Li et al. 2023). Besides, some researchers employ the multimodal contrastive learning
paradigm other modalities pair. LAION-CLAP (Wu* el. 2023) and WavCaps (Mei et al. 2023
learn aligned audio-languse space from audio-text pairs.
In addition to these studies improving the alignment of two modalities, another line of work aims to develop unified spaces capable of accommodating inputs from multiple modalities (more than two). For instance, AudioCLIP (Guzhov et al., 2022) and WAV2CLIP (Wu et al., 2022) introduce additional audio encoders for CLIP, leveraging audio-image-te and audio-image pairs. ULIP (Xue iet al. 2 and Uni3D (Zhou et al. 2023) collect massive amounts of 3D-image-tex textt paired data, enabling them to learn 3D-image-text joint representations based on advanced image-text pre- trained models. More recently, ImageBind (Girdhar et al., (203) and LanguageBind (Zhu et al.) 2023a) propose to integrate multiple modalities using different data pairs that share the crucial image or language modality.
While current methods showcase certain degree of robust cross-modal semantic alignment, they are primarily explored at relatively small scales and limited to specific modality pairs. On the other hand, our work focuses on exploring effective ways to bind large number of omni representation spaces.
2.2 SCALING UP MODELS
Scaling up language or vision- language models has been incredibly successful in perception, rea- soning, and generation tasks. GPT4 (Achiam al. 2023) and LLaMA (Touvron et al., 2023a b) achieve impressive conversation capability via training billion level language models on almost the massive internet language data. Recent Intern (Chen al. 2023) and EVA-CLIP-1888 (Sun et al. 2024) try to scale the parameter of the CLIP model to the ten-billion level. These methods prove that with the growth of model parameters and seen data, models obtain consistent and non-saturating performance improvement.
However, most large-scale multimodal models are limited aligning only two three modalities. The scarcity of paired data across multiple modalities has hindered the development of large-scale omni multimodal representation models.
2.3 KNOWLEDGG FUSION
Fusing knowledge from different sources is classical and wildly-used method to develop robust AI models. Traditional ensemble learning methods (Zhou & Zhou 2021 Zounemat- Kermani et al,, 2021) train models with different sub-datasets, and combine the output of different models as the final prediction. Similar ideas are also employed by large language model research, recent works (Bansal et al. 20244 Wan et al. 2024 Yu et al., 2023) propose to merge multiple language models tuned for different downstream tasks, and the resulting model excels at all aspects. Moreover, the Mixture-of of -iarre-of-Expeeee Experts (MoE) language model (Fedus al. 2022 Jiang et al. 2024 Chowdhery ctt al. 2023) is also developing a hybrid model consisting of multiple sub-models and obtains better performance or efficiency by integrating them together.
3

---

Published as a conference paper at ICLR 2025
Lalign
dec
P
T
Sofftra
Sotmax
Sollmas
Softmax
Cannad
A1
Concat
Concet
T
Conces
PK
3D object
Audio
Image/Viden
Language
Figure 2:
The pipeline of OmniBind. The Ox denotes the router of modality X, and F'x is the i-th encoder of modality X. The losses Lalign and Ledec are the objectives for training the routers.
For multimodal representation learning, C-MCR (Wang al. 2023c) and Ex-MCR (Wang al., 2023b) first propose to fuse two bi-modality representation space via the shared modality, thereby building unified space with low data and computing resource requirements. FreeBind (Wang al.) 2024b) is a high-level abstract of the above two methods, which employs bi-modality spaces to augment pre-trained unified space. Unlike these methods that only involve a few numbers of spaces, our goal is to obtain and keep improving omni representations via binding scale of spaces, which face more severe risks of knowledge interference.
METHOD
3.1 SPACES BINDING IN SCALE
Given that vision- lasguaela models are well-resourced and play a pivotal role in the multimodal field, choose the advanced CLIP model EVA-CLIP-ISD (Sun al. 2024) the foundation, and bind additional image-text, audio-text, audio-image-text and 3D-image-ter text spaces onto it.
For space binding, FreeBind (Wang al. 2024b) firstly proposes improve representation space by integrating extra spaces. Its space binding pipeline can be summarized as two steps: 1) collecting pseudo embedding pairs across two spaces and 2) mapping one space to another. Our binding process is primarily derived from FreeBind, but we replace its pseudo embedding-pale aggregation with more efficient and robust pseudo item-pair retrieval, and scale up integrated spaces.
Concretely, FreeBind first encodes massive unpaired unimodal data into embeddings of each space, and then uses the cross-modal similarity maps to aggregate pseudo embedding pairs across two spaces. The embedding pairs are unique for each pair of spaces. Therefore, when binding extensive spaces, repeatedly aggregating embeddings needed, which very resource-intensive. Besides, the non-shared pscudo pairs are also unstable due to the varying performance of cxisting spaces.
To bind spaces robustly and efficiently, we directly retrieve pseudo pairs across all modalities at the item level. Considering lots of unpaired 3D, audio, vision, and language data, and leveraging the most advanced 3D-image-text audio-text, audio-image, and image-tex text retrieval model, we can take each modality starting point retrieve the top-1 recall of data from other modalities. This approach constructs the pseudo item pairs {p. a, v, l). For simplicity. letters a, and is correlated point cloud, audio, image and text modality, respectively.
Using the pseudo data, we train simple projectors to individually bind each space to EVA-CLIP-18B. The training objective of the projector is multimodal contrastive loss between all pairs of involved modalities. For instance, when binding CLAP with EVA-CLIP-I8B, the learning objective is:
Lhind Info(V (Aat), Tut) Info VV (Aat,, V.e) Info (T..). T.e) Info (TT())(T.)), V.t) (1)

---

Published as a conference paper at ICLR 2025
where Aut, Tat are CLAP embeddings, Vet. Tu are EVA-CLIP-188 18B embeddings and the (-()) is an multi-layer perceptron projector. The Info(.,)) :) is multimodal constrastive loss:
B (X, Y) - - E (log Σj=1 ((rr))// + log
elyxxi)/T
Ej=1 e(vex))/+
Binding all different spaces together results in hybrid model with K 3D point encoders, S audio encoders, N image encoders, and M text encoders. These encoders originate from different pre- trained models are sharing the same representation space after binding. The representations of the resulting ensemble model can be calculated as:
N
M Σ 1=1
Σ Pr; = Σ B.A
V=
% V
TT
(3)
where the P. A, V, T are the embeddings of resulting model, and P,, A,, V,, T, respectively denote the 3D point, audio, vision, and text representations from each corresponding i-th encoder. The ag Bi, Yo. ò; are the combining factors for i-th encoder of each modality.
3.2 WEIGHTS ROUTING
Existing works Wang al. 2023c b 2024b) manually set the combining factors of encoders from different spaces. While manual settings offer flexibility to customize the resulting space when inte- grating a few spaces, is increasingly complex and impractical as more spaces are added. Moreover, hand-designed combining weights also limit the deep integration across various knowledge sources, leading to simple trade-offs rather than comprehensive incorporation of different expertise.
To address this issue, drawing inspiration from the Mixture-of off-Exeert (MoE) technique in Large Language Models (LLMs). we propose dynamically assigning weights with learnable routers. As shown in Fig. 2 each modality contains one router to predict the corresponding combining factors for encoders this modality, which can formulated as:
(0,, BI...., Bs = soft.mx n n...... YN softmax(O.(( (VV))) , softmax(O,(T)) where P, A, V. T are the concatenated outputs of each modality, and the θ,, O., O,., O2 denote the router for 3D point, audio, vision and language, respectively. To develop effective and robust routers, we utilize the entire retrieved pseudo dataset {p, a, v, { and design two learning objectives.
01,.... OKK
Cross-modal Overall Alignment. To motivate routers to predict the optimal weights for all modal- ity combinations, employ contrastive losses overall modality pairs as the first learning target: Lalign Info A, P) + Info(A, A, V) + Info (A, T) + Info(P, V) + Info(P, T) + Info V, T) (5) where P. A, V. T are defined in Eq. By simply averaging the contrastive losses between all modality pairs, we cultivate balanced routers that achieve comprehensively high-quality cross-modal semantic alignment over all modalities.
Language Representation Decoupling, Compared to 3D points, audio, images, and videos, which are primarily sampled from the real world, language data is entirely artificial, cxhibiting much higher information density and stronger ideographic tendency. Therefore, textual descriptions of different modalities exhibit significant biases: image captions often describe appearances, audio captions focus on sounding actions, and 3D captions prioritize spatial structures. As result, text encoders trained to align different modalities demonstrate more specialized expertise than encoders other modalities.
Considering the significant distribution variance among the different text representations, we intro- duce an auxiliary learning objective for the language router to disentangle the language represen- tation and improve its generalization. It preserves the discrimination of text embedding space and enhances the semantic alignments with various modalities. Specifically, we drive the language router to identify which modality the input texts are likely describing and to prioritize text encoders that are specialized in the corresponding modality. To this end, we define the loss function as follows:
Ldec Σ
M log (T()))))))))) + (1 y3) log(1 +x(T))))
5

---

Published as a conference paper at ICLR 2025
where the M is the number of text encoders. The texts in the retrieved pseudo -acloo-ppreddddd dataset collected from audio-text, vision-text, and 3D-text pairs. Correspondingly. the text encoders are also derived from models pre-trat pre-trained for audio-text, vision-text, and 3D-text alignment. In Eq 6 y/ the input text and the j-th text encoder are related to the same modality and = 0 otherwise.
Finally, we linearly combine the above objectives, and the final loss can be expressed as:
MLdec + Laaiign
3.3 MODEL CONFIGURATIONS
The projectors V used for aligning spaces are simple two-layer MLPs. Additionally, we employ the mixture-of-projectan strategy following Wang et al. (2024b). The routers O are similarly designed as two-layer MLPs, with an extra sigmoid activation function at the end.
We select 14 pre-trained spaces for binding, which can be grouped into five audio-text (three Wav- Caps (Mei et al. (023)., two LAION-CLAPs (Wu* et al. 2023)), five image-text (EVA-CLIP. 18B (Sun et al.) 2024). EVA02-CLIP-E (Fang et al.) 2023b) two SigLIPs (Zhai
2023 DFN- VIT-H (Fang et al. 2023a)), three 3D 3D-image-ters (three Uni3Ds (Zhou et al. 2023) and one audio- image-text (ImageBind (Girdhar al., (023)) spaces. After binding all spaces
AVA-CLIP-188 we construct three configurations of OmniBind by combining different spaces: OmniBind-Ban Baase, OmniBind-Large and OmniBind-Full with 5, 13, and 14 individual spaces. The encoder parame-
ters for each modality and the specific spaces used in each variant are detailed in Appendix B
4 EXPERIMENT
4.1 IMPLEMENTATION
To construct the pseudo-paired data, collect unpaired 3D point, audio, vision, and text data from the training set of existing datasets. For 3D data, we use the 800k 3D point clouds from Objaverse (Deitke al. 2023). The audio and image data
Datasets &
from Au- dioSet Gemmeke et all 2017) and ImageNet (Deng et al. 2009) respectively. The text data sources from three kinds of datasets: 3D-text (Liu al. (0224)) visual-text (Lin et al.) [014 Sharma al.) 2018) and audio-text (Kim et al. 20199] Drossos et al. (020) datasets. Based on these unpaired uni- modal data, we employ state-of-the-wee audio-text (WavCaps (Mei et al., 2023)), image-text (EVA- CLIP-18B (Sun et al., 2024)), audio-image (ImageBind (Girdhar et al. 2023 and 3D-image-text
(Uni3D (Zhou et al. [2023) models to retrieve the pseudo item pairs, as discussed in Sec. 3.1 The
temperature factors contrastive losses are 0.03, and the À in Eq. 7 is 3.
Benchmarks Baselines. To compre- hensively access the performance of our omni representations, we conduct quanti- tative experiments across 14 benchmarks covering 7 downstream tasks, as summa- rized in Tab. [ In these benchmarks, compare OmniBinds with three groups of previous multimodal representation mod- els: 1) 3D-image-text models: Uni3D's pre- trained and three fine-tuned variants (Zhou)
Table 1: Statistic for evaluation tasks and bench- marks. Task
Modulity
Benchmarks
Items
AudioSet (Gemneke al
2017
19,048
ESC-30 Piczak 2015
100
Zero-shot
Image
ImageNei-IK
Deng
Classification
Objavense LVIS
Deitke et al. 2023
46,832
StanOhjNN
Ly et
2019
2,890
ModelNet40
2,468
Audio-Text
AudioCaps
Kim
2019
961
et al.) (023). 2) audio-image-tee models: C-MCR (Wang et al. 2023c), Language- Bind (Zhu et al. (023a). ImageBind (Gird- ImageBind++ (Wang et al. har et al. 2024b) and Intern VL (Wang et al.
Clotho Drossos et al:
1020
1.045
Chen et
5,158
Cross-modal
Indio-Image
FlickrNet Senocal
2018
5,000
Retrieval
Image-Text
COCO Lin al.
2014
5,000
Flickr-3KK Young et al. 2014
1,000
Text- Text-Video
MSR-VTT Nu al.
2016
2,990
2024b). 3) ---autio-image--ext models:
3D-Image
Objavens--LVIS Deitke
2023
46,205
Ex-MCR (Wang al., 2023b) and Point-
Bind (Guo et al. 2023). Moreover, we provide the best results on each benchmark achieved by the 14 source specialist spaces for reference, denoted as "Individual Best".

---

Published as a conference paper at ICLR 2025
Table 2: Cross-modal retrieval results. Best result is bolded, and second best result is underlined.
Audio-Text
Audio-Image
Image-Text
Text- Video
3D-Image
Models
AudioCaps
Clotho
VGG-SS
FlickıNet
COCO
Flickr3KK
MSRVTT
Objaverse
R@1
R@5
R@1
R@5
R@5
R@1 R@5
R@
R@5
R@
R@5
R@1
R@5
R@ R@5
Uni3D
59.51
81.45
87.85 97.55
43.57 67.61
C-MCR
15.76 41.37 8.37 24.86
1.94
7.69
1.39
5.97
16.67
37.04
34.16 63.64
LanguageBind
12.42 36.70
11.32 31.03
2.55
9.86
1.52
6.36
53.24
76.48 82.36 96.19
32.64 55.43
X
X
ImageBind
9.24 27.47 6.64
17.28
14.82
35.67
7.68
20.79
57.28
79.54 86.04 96.97
26.73 48.21
ImageBind++
29.16 62.98 13.67
33.19
15.48
39.26
8.01
21.87
57.01
79.23 85.91 97.03
X
X
Int
29.11 62.30 12.66
32.75
14.40
36.78
7.74
21.85
61.07
82.00 89,30 98.09
Ex-MCR
19.07 47.05 7.01
22.04
2.13
8.13
1.57
5.94
40.24
64.78 71.89 90.55
2.54
8.25
PointBind
9.24 27.47 6.64
17.28
14.82
35.67
7.68
20.79
57.28
79.54
86.04 96.97
26.73
48.21
5.86
14.59
OmniBind-Rase
43.61 76.02 20.94
46.77
14.11
35.74
7.67
21.65
56.94
80.11 85.99 97.02
24.23
46.08
34.34
58.40
OmmiBind-Large
47.89 79.75 23.07
49.67
14.
36.07
7,86
21.72
60.08
82.35 87.20 97. 97.40
30.05
52.91
46.09
69.11
OmniBind-Full
49.28
80.09
23.22
49.36
15.64
38.19
8.32
23.49
63.01
84.41
89.12
98.00
32.89
55.76
46.55 69.92
Individual Best
48.22 81.15 23:57 49.13
14.82
35.67 7.68 20.79
63.69 84.09 90.83 98.33
33.54 55.95 43.57 67.61
Table 3: Zero-shot classification results. Uni3D, Uni3D(O Uni3D(Scan.) and Uni3D(Model.) represent the pre-trained and three fine-tuned version of Uni3D-g. respectively.
Audio
Image
3D
Model
AudioSet
ESC-50
ImageNet Top1
Objaverse Top1
ScanObjectNN
ModelNet40
mAP
Top1
Top5
Top5
Top5
Top1
Top5
Top1
Top5
Uni3D
X
X
X
80.
95.97
53.13
81.59
64.12
91.63
87.56
99.27
Uni3D(Objav.)
X
X
80.12
95.97
54.74
82.54
58.89
88.69
84.20
98.42
Uni3D(Scan.)
X
X
x
80.12
95.97
50.99
80.00
65.81
92.70
88.05
98.58
UnidD(Model.)
X
X
X
80.12
95.97
51.21
80.14
64.43
90.66
88.05
99.11
C-MCR
11.15
70.35
96.70
24.50
52.44
X
LanguageBind
18.33
94.90
99.70
77.15
94.64
X
X
X
X
X
x
ImageBind
13.96
67.25
87.50
76.31
94.23
X
X
X
X
x
x
ImageBind++
19.69
90.30
99.30
76.01
94.36
X
InternVLng++
18.93
87.75
98.75
81.21
96.68
X
X
X
X
X
X
Ex-MCR
6.67
71.20
96.80
60.79
86.98
17.94
43.37
40.31
77.20
66.53
93.60
PointBind
13.96
67.25
87.50
76.
94.22
13.83
30.34
55.05
86.89
76.18
97.04
OmniBind-Base
21.19
92.90
99.75
76.18
94.02
53.30
81.85
57.79
89.76
82.82
97.12
OmniBind-Laage
25.57
93.25
99.80
78.87
95.32
53.97
82.90
64.67
94.15
86.55
99.03
OmniBind-Fiu
25.87
94.55
99.90
79.11
95.49
58.13
81,76
64.60
95.53
86.83
99.03
Individual Best
23.36
94.05
99.75
82.43
96.73
54.74
82.54
65.81
92.70
88.05
99.11
4.2
PERFORMANCE RESULTS
Cross-modal Retrieval. As aforementioned, OmniBind aims provide high-quality semantic alignment between all modality pairs. Therefore, we comprchensively assess the cross-modal re- trieval performance across all possible modality pairs. The quantitative results for audio-text, audio- image, vision-text, and 3D-image retrieval are presented in Tab.
Overall, OmniBind-Full and OmniBind-Large consistently outperform all previous methods across all benchmarks. Some prior approaches demonstrate competitive performance within their spe- eific domains. For instance, ImageBind++ shows similarly strong audio-image alignment, and Intern + displays comparable image-text capabilities, but they both fall short in other areas and lack support for 3D input. Compared to existing a-D-uudoo-image-.. PointBind, all OmniBind variants exhibit substantial and comprehensive advantages across all com- binations of modalities. These observations underscore the versatility and superiority of OmniBind.
models like Ex-MCR and
Moreover, OmniBind-Full achieves similar performance to the "Individual Best", demonstrating that OmniBind successfully inherits and effectively integrates the expertise of various source spaces. Remarkably, OmniBind achieves even better audio-image and 3D- 3D-image alignment, showcasing the exciting cross-space knowledge transfer. By incorporating the high-quality image representations learned from image-text data, the audio-image and 3D-image alignment can also be improved.
7

---

Published as a conference paper at ICLR 2025
a). Audio 3D Retrieval RetrievalSSappes
[A cat purrs deeply]
[A song played violin]
[The ticktock of the old clock]
PointBind
SSFI
Ex-MCR
BUFFAIES
OmniBind
b). 3D to Audio Retrieval Samples
[A triin passes by followed by horn]
PointBind
[Waming ring
pusses honkinn horn.)
[A trin hom blowing followed by mnin running on raiimad tracks pusses by.]
train
passes by:]
EX-MCR
[A wemaz speaks briefly. and muflled engine rumbles.]
trii going over tracks very slowly.]
[ligh pisches sqacaling
blowing constaat humming engine.]
OmmilBnd
short train quickly ly paeses lorving all Girly quict.]
train passes by followed by horn.]
[A vehicle engine running powering.]
PointBind
[A motor vehicle engine is idling.)
engine idling followed
male speaking.]
[A femule voice briefly speaks followed by crinking noises.]
Ex-MCR
[A woman speaches while cooking.]
[A man talking followed by plastic crinkling and crumpling.]
[A penson snoring followed by a man uilking]
OmniBind
woman breathing heavily followed
then sniffling.]
[Contimouus light snoring.]
IA bell being rang in crratic fishion no uneven tempo.]
PointBind
[A woman spenkss as water splaches.]
female speaks while sizzling crackling are ongoing.]
engine peoducing
and whine.]
Ex-MCR
[Boat motor idles then accelerates.]
[Race
revvigg through gears.)
[A penson in big boots walking down a hallway:]
OmniBind
Fooosseps pace
forth before walking away, coming
[Shoes walking on a hard surface covered with a thin layor of dirt and gravel]
Figure 3: Qualitative comparison of audio-3D object retrieval. More visualizations are provided in the Appendix C
Zero-shot Classification. To further validate the generalization ability of OmniBind, we conduct zero-shot classification on each modality, and report the results Tab,
For audio and image classification, OmniBind demonstrates overall superiority, even when com- pared models excelling the audio-image-text domains. While LanguageBind performs slightly better than OmniBind on ESC-50, all variants of OmniBind significantly outperform previous meth- ods on the more challenging AudioSet benchmark. This highlights the robustness and generalization of OmniBind. Furthermore, although Intern VI+ achieves the highest accuracy on ImageNet, it falls short in audio classification, which further showcases the versatility of OmniBind.
In 3D object classification, the three fine-tuned variants of Uni3D perform exceptionally well on their respective fine-tuned datasets. OmniBind-Bass which leverages only Uni3D(Objav.) as the 3D-image-text source space, achieves performance comparable to Uni3D(Objav.), demonstrat- ing that the binding space effectively inherits knowledge from the source space. Additionally, OmniBind-Large and OmniBind-Fin integrate the knowledge from all three fine-tuned versions, performing well across various benchmarks and achieve significant improvements on the most chal- lenging Objaverse-ILVV VISS benchmark.
8

---

Published as conference paper at ICLR 2025
a). Any-query Object Localization
Complex Composable Understanding
fwalking on gras)
(fod strlingl
"A group of kids
[Wind blowing
3D object
3D object
Audio
gather around
table with cakes."
b). Any-query Audio Separation
haskrgl-fasite
buiking
"Boys yellow shirts
bouncingl
[dog ruaning and becking)
3D object
image
Figure 4: Diverse applications enabled by OmniBind. (a) Any-query Object Localization: accu- rately matches the object proposals with audio and 3D query. (b) Any-query Audio Separation: successfully separate audio using queries from images, audio, language, and 3D models. (c) Com- plex composable understanding. OmniBind space supports complex serial embedding arithmetic to freely add and remove semantic information.
4.3 APPLICATIONS
Emergent cross-modal alignment. OmniBind reveals surprising emergent alignment between 3D and audio. In Fig. 3 and [5] we randomly sample audios from AudioCaps and retrieve rele- vant 3D objects in Objaverse, and in Fig. Gtb, we randomly select 3D objects from Objaverse and retrieve relevant audio clips from AudioCaps. OmniBind shows a deep understanding of 3D and audio samples and high- high-quality semantic alignment. For example, it accurately recognizes the audio of "cat purring" and "violin" and finds the corresponding 3D models. In the "clock ticktock" case, while PointBind and Ex-MCR struggle distinguish disk-shaped 3D models, OmniBind success- fully identifies mechanical clocks.
Any-query object localization. OmniBind also demonstrates high-quality fine-grain semantic alignment, enabling any-query object localization. By extracting object proposals using prettrained object segmentation (Kirillov et al. (023)) models, from any modality within OmniBind's space. As shown Fig. a, we can localize objects in images using sounds or 3D models queries.
match these object proposals with queries
Any-query audio separation. We replace the CLIP embeddings in CLIPSep (Dong et al. 2022) with OmniBind and fine-tune it pseudo dataset. In Fig. 4, observe that cessfully separate audios of a dog barking using queries from images, audio, language, and even 3D models of dog.
Complex composable understanding. Impressively, OmniBind's embedding space not only en- ables addition and subtraction between arbitrary embeddings but also supports complex serial em- bedding arithmetic. Three representative cases are provided in Fig. 4c. In the first case, starting with a photo of "one cat and one dog on sofa," we sequentially subtract the embeddings of 3D cat and 3D sofa to obtain "two dogs on sofa" and "two dogs not sofa." Adding the sound of walking on grass then results in "two dogs on the grass." Similar exciting capabilities are observed in two other examples.
More potentials. Previous multimodel joint representations (Zhou et al. 2023 Girdhar et al, 2023 Zhu et al. (023a) enable other applications, like point cloud painting (Zhou al. 2023 any-to-any generation (Tang et al. 2024), and omnimulti modal LLMs (Han et al.) 2023 Lin et al. 2023). Given that OmniBind is essentially homologous the representation models used previous applications and contains more diverse and generalizable knowledge, holds great potential for these applications well. We leave these explorations for future work.

---

Published as a conference paper at ICLR 2025
Table 4: Ablation study of manual weights and the two weight routing objectives: Latign and Ldee- AT high, VT high, and PVT high indicate manually assigning higher weights audio-text, image- text, and 3D-image-text spaces, respectively. Mean means averaging all spaces. The Top1 and @@1 metrics of 3D classification and cross-modal retrieval are reported. The dataset names are abbreviated.
3D classification
Audio-Text ACaps
Audio-Image VGGSS
Image-Text COCO
3D-Image Objav.
Setting
Latign
Ldee
Objav.
Scan.
Model.
Clotho
FNet
F30K
AT high
49.47
59.90
85.17
44.47
21.76
14.33
7.69
55.97
84.81
46.23
VT high
53.44
64.19
86.99
33.24
17.49
13.82
7.45
61.38
88.57
46.24
PVT high
53.35
63.08
87.07
34.95
18.08
13.18
7.34
59.86
87.84
48.18
Mean
52.29
62.46
86.47
40.31
20.33
13.40
7.35
59.61
87.61
44.61
X
53.70
64.81
87.40
48.48
24.31
13.93
7.35
61.86
88.27
44.61
Weight
52.93
63.98
85.74
43.71
20.44
15.62
8.35
59.43
87.64
46.73
Routing
58.13
64.60
86.83
49.28
23.22
15.64
8.32
63.01
89.12
46.55
4.4 ABLATION STUDY
Performance improvement with more spaces. The integrated spaces and parameters of our three variants: OmniBind-Base OmniBind-Large Large, and OmniBind-Full, sequentially increase. From Tab. 2 and 3] observe consistent overall performance improvements binding more and more models. The scalability of the integrated spaces further highlight the potential of our method.
Trade-off's in manually assigned weights. Tab. Tab. shows the performance of four types of manually assigned weights: AT high, VT high, PVT high and Mean. These four variants reveal trade-off' phenomena among different areas expertise. AT high performs well in the audio-text domain but is less effective in 3D classification and image-text tasks. Conversely. VT high (PVT high) obtains better image-text (3D-image) performance the expense of audio-text alignment. Mean exhibits more balanced performance but lacks specific expertise. Moreover, using weight routing showcases a comprehensive advantage over these manual weight variants. This observation further proves the routers can effectively alleviate interference between knowledge of different sources, leading to overall improvements performance rather than simple trade-offs.
Effectiveness of Lalign & Ldee for learning routers. We provide the ablation experiments for the two learning objectives in Tab. 44 By comparing the Mean and variants using each objective sepa- rately, we conclude that Latign brings noticeable improvements across all possible modality combi- nations, while Ldee specifically and significantly enhances language-relatte alignment. Combining the two objectives yields the best overall performance by complementing each other's strengths.
Improved discrimination with Ldec. To explore the effect of language representation decoupling on discrimination, we extract language embeddings from all AudioCaps and COCO captions. We find that employing Later reduces the cosine similarity between all the possible language embed- ding pairs from 0,0828 0.0517. As discussed in (Liang et al., 2022). lower cosine similarity between irrelevant pairs indicates the embeddings "occupy" more space in the hypersphere. There- fore, employing Liee indeed enhances the discrimination of language representations, leading to better overall performance.
CONCLUSION
In this work, we present OmniBind, a series of omni multimodal representation models that bind 5.13, and 14 spaces. Our core contributions lie in designing the weight routing strategy to miti- gate interference between knowledge of different sources, thereby successfully binding pre-trained spaces in scale obtain keep improving multimodal representation. By efficiently integrating multiple pre-training spaces, OmmiBind demonstrates impressive multimodal performance across extensive benchmarks and shows vast potential for further growth and diverse applications.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
All checkpoints for the different versions of OmniBind will be open-sourced. The source spaces used build OmniBind, as detailed in Table 8 along with the evaluation benchmarks in Table [ are publicly accessible.
ACKNOWLEDGEMENTS
This work was supported in part by the National Key R&D Program of China under Grant No.20227/D0162000.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0872 2023.
Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
Rachit Bansal, Bidisha Samanta, Siddharth Dalmia, Nitish Gupta, Shikhar Vashishth, Sriram Gana- pathy, Abhishek Bapna, Prateek Jain, and Partha Talukdar. Llm augmented lmms: Expanding 2024.
capabilities through composition. arXiv preprint
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sachoon Kim. Coyo-700m: Image-text pair dataset. https::/github.cem htt.:u/b...oomaaaabbabbaaaaaaaaa
coyo-dataset, 2022.
Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zis- serman. Localizing visual sounds the hard way. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16867-16876, 2021.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- lingu--ing tasks. arXiv preprint arX1::2312.14238, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scall hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Hao-Wen Dong, Naoya Takahashi, Yuki Mitsufuji, Julian McAuley, and Taylor Berg- Kirkpatrick. Clipsep: Learning text-queried sound separation with noisy unlabeled videos. arXiv preprint arXiv:2212.070065 2022.
11

---

Published as a conference paper at ICLR 2025
Konstantinos Drossos, Samuel Lipping. and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP). pp. 736-740. IEEE, 2020.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023a.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303. 11331, 2023b.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
19358-19366 19369, 2023c.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-33 2022.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeles dataset for audio events. In 2017 IEEE international conference acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. Contrastive audio-visual masked autoencoder. arXiv preprint
arXiv:2210.07839, 2022.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint N.Xv:22309.0061.5 2023.
Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pp. 976-980. IEEE, 2022.
Jamming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint arXiv:2309.03900 2023.
Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. arXiv preprint arXiv:2312.0816
2023a.
Rongjie Huang, Jiawei Huang. Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audioo Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pp. 13916- 133332 PMLR, 2023b.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint
2024.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume I (Long and Short Papers), pp. 119-132, 2019.
12

---

Published as a conference paper at ICLR 2025
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference Computer Vision, pp.
2023.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp.
2023.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Ad- 2022.
vances in Neural Information Processing Systems,
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311. 2023,
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, arXiv preprint arXhv:2310.03744. 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024a.
Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding, Advances in Neural Information Processing Systems, 36, 2024b.
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumb- ley, Yuexian Zou, and Wenwu Wang. Wavcaps: chatgpt-assisted weakly-labelled audio cap- tioning dataset for audio-language multimodal research. arXiv preprint (tti::2303.173999 2023.
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yin- fei Yang. Sentence-15: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint urXiv:2108.08877 2021.
OpenAl. Gpt-4o. index/hello-gpee
-pt-ll//,,p-- 2024. URL htttttt//pppeaa..: com/
Karol Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd
ACM international conference on Multimedia, pp. 1018, 2015.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 9748-8765 PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 2225278-252944 2022.
13

---

Published as a conference paper at ICLR 2025
Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang. and In So Kweon. Learning to local- ize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4358 4368-4366, 2018.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Lonng Papers), pp- 2566-2565, 2018.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. aakeea-video: Text-to-vider generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint 15338 2023.
Quan Sun, Jinsheng Wang. Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang- Eva-clip-18b: Scaling clip to 18 billion parameters. arXiv preprint arXiv:2402.04252, 2024.
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Sorieut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint wxXiv23307099288 2023b.
Mikaela Angelina Uy, Quang- Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revis- iting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1588- 1597, 2019.
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. arXiv preprint
2024.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:22/2.0319/ 2022.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403. 15377, 2024a.
Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data- efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.00 088.02.008.0866. 2023a.
Zehan Wang. Ziang Zhang, Luping Liu, Yang Zhao, Haifeng Huang, Tao Jin, and Zhou Zhao. Extending multi-modal contrastive representations, arXiv preprint trXiv:2310.08884 2023b.
Zehan Wang, Yang Zhao, Haifeng Huang, Jiageng Liu, Aoxiong Yin, Li Tang, Linjun Li, Yongqi Wang, Ziang Zhang, and Zhou Zhao. Connecting multi-modal contrastive representations. Ad- vances in Neural Information Processing Systems, 36:22099-22 1222099222144, 22,14,222 2023c.
Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang, Luping Liu, Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, and Zhou Zhao. Freebind: Free lunch in unified multimodal space via knowledge fusion. arXiv preprint arXiv:240s 04883, 2024b.
14

---

Published as a conference paper at ICLR 2025
Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4563-4561 IEEE, 2022.
Yusong Wu*, Ke Chen*, Tianyu Zhang* Yuchen Hui Taylor Berg-Kirkpatrri and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5288-5296, 2016. doi: 10.1 109/CVPR.2016.571..
Le Xue, Mingfei Gao, Chen Xing, Roberto Martin-Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, images, and point clouds for 3d understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1179-1189, 2023.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. arXiv preprint arXiv:2311.03099, 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11975-11986, 2023.
Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation scale. arXiv preprint arXiv:2300.0673,p00 2023.
Zhi-Hua Zhou and Zhi-Hua/ae Zhou. Ensemble learning. Springer, 2021.
Bin Zhu, Bin Lin, Munan Ning. Yang Yan, Jiaxi Cui, Hongla Wang, Yatian Pang. Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n- modality by language-base based semantic alignment. arXiv preprint arXiv:2310.01852, 2023a.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vasion-language understanding with advanced large language models. arXiv preprint vrXiv:2304.10592 2023b.
Mohammad Zounemat-Kerman Okke Batelaan, Marzieh Fadaee, and Reinhard Hinkelmann. En- semble machine learning paradigms in hydrology: review. Journal of Hydrology, 598:12686 108:626266, 2021.
15

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND FUTURE WORKS
Although OmniBind is already the largest multimodal representation model, demonstrating remark- able versatility and generalization, it currently utilizes only 14 existing spaces and 5 modalities: 3D, audio, images, video and text. remains to be explored whether or when this binding space-based "scaling up" will reach saturation. Additionally, this work only presents three basic applications as inspiration. Investigating whether using higher-qualinn and larger-scale omni multimodal represen- tations can enable new capabilities in downstream applications would be promising direction.
B STATISTICS OF OMNIBIND VARIANTS
The detailed source spaces used to build OmniBind-Base, OmniBind-Large, and OmniBind- presented in Tables 6 and respectively. Generally speaking, OmniBind-Bass comprises three components: three WavCaps, EVA02-CLIPEE, Uni3D (Objaverse), and ImageBind. The pri- mary difference between OmniBind-Lare and OmniBiindFull is the inclusion of EVA-CLIP- 18B in OmmniBindFuul.. Additionally, we provide statistics on the number of parameters for OmniBind- Base, OmniBindd-Laree and OmniBind-Fuu in Table g
Table 5: Source spaces for building OmniBind-Base
Orra Bind Base
Audio Text
WasCaps Mei al.
2033 WarCaps (Clotho) Maiet 2025
Cap (AudioCaps) Mei al. 2023
Image Text
EVA02 CLIP Fang al. 20336
3D-Imige-Text
Uni3D (Ohjav.
Andio-Image-Tes Test
ImageRind
Gidhar et al.
2023
Table 6: Source spaces for building OmniBind-Large
Ardho-Teet
WarCapp Me etal
War
2013
WarClap (AusdaCC go
2003
Image-Teet
20228
SuLisLane
Moce00t Lose 2003
10-Image-Tree Teet
UnITD (Dpps)
Zhre et
200 (Seas
2200 mt a01
2hem 2033
Audi--lsage Insage Tosl
IasapeBiad Kindbar
2005
Table 7: Source spaces for building OmniBind-Full!
Audio-Ted
WasCopo Mii et
LAION CLAP
2023
Ggeneral) Wut
WavCCps (Cletho)
Mei et
LAION
20233
WarCrge (AudioCapp)
(music) Wo
Mei
2023
EVA-CLIP- 18B
Sun el all 2026 EVA02
Fang et al
20236
Image Text
Fong et
2023a,, SigLIP Laago
et 2023
SIRLIP
3D Image Text Uni3D (Objun)
Zhou et al.
2023
Uni3D (Scun.)
Zhou al.
2023
Uni3D (Model)
Zhou at 2023
Audio-Imagg-Text
ImageRind
C MORE VISUALIZATIONS
To further qualitatively assess the capabilities OmniBind, we provide additional visualizations. Each sample is manually evaluated and categorized: correct samples are marked in green, incorrect ones in red, and partially correct ones in orange.
Furthermore, we present additional cases of audio 3D object retrieval in Figs. In each instance, the 3D objects retrieved by OmniBind exhibit significantly better semantic alignment with the audio query than those retrieved by Ex-MCR and PointBind.
D SOCIAL IMPACT
OmniBind is a method for learning large-scale multimodal representation models by aligning and reorganizing knowledge from existing pre-trained models. Therefore, the capability of OmniBind is
16

---

Published as a conference paper at ICLR 2025
Table 8: Statistics about parameter number of three OmniBind variants.
Parameter Number Image Encoder
Variants
3D Encoder Audio Encoder
Text Encoder Projector
Total
OmniBBindBaase
1.0B
99M
4.6B
1.3B
224M
7.2B
OmniBind-Large
3.0B
329M
6.0B
2.5B
431M
12.3B
OmniBind-Full 3.0B
329M
23.6B
3.2B
431M
30.6B
mainly inherited from the source pre-trained models. Implementing additional safety detection and filtering processes for these pre-trained models can effectively reduce the potential for misuse and mitigate negative social impacts of OmniBind.
E EXAMPLES OF PSEUDO PAIRS
In Fig. and present the examples of pseudo pairs that retrieved from 3D objects, im- ages, audio and language, respectively. There are two main findings: 1) The pseudo-pairs collected by state-of-the-arr cross-modal retrieval models exhibit strong semantic consistency. 2) Treating dif- ferent modalities the strating point improves diversity and covers different semantic biases. The typical example that there are obvious differences between the text descriptions retrieved from image (Figure 7 and audio (Figure
F VISULIZATION OF LANGUAGE REPRESENTATIONS
We present UMAP visualization of the language representations of audio captions (from Audio- Caps) and image captions (from COCO) Fig. 10 Within the high-quality self-supervised lan- guage model, T5 Ni et al. (021), ), the two kinds of captions reveal clear domain gap, supporting the claim that textual descriptions for different modalities exhibit significant biases. Similarly, in OmniBind's representation space, similar gap is observed, underscoring the discriminative capa- bility of OmniBind's representations. Notably, while removing the language decoupling objective does significantly alter the qualitative distribution map. quantitative results indicate that lan- guage decoupling reduces the representation similarity between audio and visual captions from 8.28 5.17, demonstrating improved disceriiiiiity and more dispersed distribution.
17

---

Published as a conference paper at ICLR 2025
Water that washing up beach shore very slow rate.)
Peertied
[Someone frying and scraping the pan and then throwing spatula in the pan.]
Oraitind
[Fireworks continuing to boom and crackle loudly.]
Poeztled
[Birds chirping, cawing, mating calls in forest with stream.)
Peistiied
D-MER
Orailted
[A person sharpens a knife, the blade clicking on the surface of the sharpening tool.)
MCR
Figure 5: More visualization of Audio-to-3D retrieval.
18

---

Published as a conference paper at ICLR 2025
3D object
Language
Image
Audio
"Adult elephant standing close baby elephant"
["Wild animals', 'Speech', 'Roaring cats (lions, tigers)', 'Animal']
"Yellow, small rubber duck toy squeaking"
['Quack']
"a cartoon character standing front of black square tile floor"
['Cild speech, kid speaking']
"I also drew little pocket the dress
['Choir', 'Female singing')
Figure Examples of pseudo pairs retrieved from 3D objects.
Image
Audio
3D object
Language
"a pint of beer day keeps doctors away"
['Liquid', 'Water']
0.5
.05
"A school bus parked next to the sidewalk in urban area"
['Bus', 'Illing', 'Vehicle', 'Speech', 'Child speech, kid speaking']
"a metal fan on a stand"
['Mechanical fan']
"A toucan with a brightly colored beak in a cage." "
['Bird', 'Domestic animals, pets', 'Speech']
Figure 7: Examples of pseudo pairs retrieved from images.
19

---

Published as a conference paper at ICLR 2025
Audio
3D object
Language
Image
"A an engine is idling and wind blows"
['Idling', 'Vehicle]]
['Enginc', 'Vehicle', 'Car', 'Engine starting']
"An engine idles nearby, and then belts rattle"
"musical artist plays the guitar inside his home"
['Guitar', 'Music', 'Musical instrument', 'Strum']
-05
"concert of video a mariah carey song in her early years"
['Music']
Figure 8: Examples of pseudo pairs retrieved from audios.
Language
Image
Audio
3D object
"a city by persons"
['Outside, rural or natural']
"a profile view of young beautiful white and red puppy dog standing on the lawn
['Dog', 'Animal"]
"modest living style with houses on stilts along the banks." "
['Motorboat, speedboat', 'Vehicle"]
"images vs. girls basketball game."
['Basketball bounce']
Figure 9: Examples of pseudo pairs retrieved from texts.
20

---

Published as a conference paper at ICLR 2025
T5
OmniBlin
OmniiBind (without language decoupling)
Figure 10: UMAP visualization of language representations from different domains in different pre- trained models.
21


Reviews:
[{"review_id": "HEJqth60YS", "reviewer": "Reviewer_Z2z7", "summary": "The paper proposes OmniBind, a framework that aims to merge multiple pre-trained multimodal models into a unified, large-scale representation space that can handle 3D, audio, image, video, and language inputs. This binding strategy employs weight routing to address knowledge interference from multiple sources and optimizes the integration of various models. OmniBind showcases impressive zero-shot generalization and cross-modal alignment across modality pairs.", "questions": "1. If future models introduce new modalities or more granular representations, how easily can OmniBind incorporate them without significant retraining or re-engineering?\n2. Given that 3D, audio, image, video, and text often represent different levels of semantic information (e.g., higher ideographic density in text), how does OmniBind address potential biases in multimodal alignment? How did you ensure the reliability and consistency of the pseudo pairs generated across all modalities?\n3. Can you provide more details or visualizations on how the routers assign weights in different tasks or datasets? How do these weights correlate with performance on cross-modal benchmarks?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper effectively addresses the challenge of binding multiple pre-trained multimodal spaces through dynamic weight routing. This approach moves beyond traditional fixed-weight methods by introducing a learnable routing mechanism, enhancing cross-modal alignment and reducing knowledge interference.\n2. OmniBind's performance is thoroughly evaluated across diverse benchmarks, covering a wide range of modality combinations. This strengthens the claims of improved generalization and versatility.\n3. The paper provides multiple examples of how OmniBind's representation space can be used in various applications, such as any-query object localization and audio separation, which demonstrate the practical benefits of the model.", "weaknesses": "1.  While the model uses pseudo-paired data due to a lack of real-world multimodal data pairs, this raises questions about the validity of the results in real-world applications. This reliance on pseudo-pairs could potentially limit the model's robustness in unexpected scenarios.\n2. Although the dynamic routing approach is innovative, it may introduce complexity, particularly in practical implementations or deployment, where computational costs and latency might become prohibitive with increasing scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oSHC5osmG2", "reviewer": "Reviewer_ViaF", "summary": "The manuscript presents OmniBind, a multimodal joint representation model designed to integrate and optimize the capabilities of 14 pre-trained modalities, including 3D, audio, image, video, and language inputs. By merging these specialized models into a unified representation space, the authors aim to enhance multimodal understanding and generation. The proposed framework employs a dynamic weight assignment mechanism, utilizing lightweight networks to facilitate the binding and routing of knowledge sources. The author claim that they not only improves the overall alignment across different modalities but also allows for effective language representation decoupling.", "questions": "1. For a certain modality, when different modalities need to be aligned, will it itself require different angles of representation, and is it reasonable to do a universal representation?\n2.For language representation decoupling, my understanding is that this involves separately aligning image encoders with text encoders from image-text pretraining, and audio encoders with text encoders from audio-text pretraining, etc. If the authors separate these alignments, does it contradict the goal of achieving a universal representation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.The manuscript presents a compelling approach to integrating various multimodal representation models into a unified framework, which is an advancement in the field of human-computer interaction.\n2.The introduction of dynamic weight assignment and language representation decoupling may bring benefits.\n3.The experimental validation provided in the study partially supports the authors’ claims regarding the versatility and effectiveness of the proposed model.", "weaknesses": "1. As for the proposed Language Representation Decoupling, statistics on the number of such cases can be added to support the need for such improvements.\n2. Regarding the construction of the pseudo pair data for training, for audio, the authors use audio from AudioSet and then employ a state-of-the-art audio-text model to retrieve the most similar texts from AudioCaps and Clotho (they also retrieve texts from other datasets, but as the authors mention, there is a gap in the texts across different modalities, so the most similar texts are likely to come from the audio-text dataset). First, I think this is somewhat redundant, as they could have directly used the paired audio-text data from AudioCaps, since the audio in AudioCaps comes from AudioSet. Secondly, this means that the training set includes data from Audiocaps, so please confirm whether the methods in Table 2 also have similar situations; otherwise, I believe it would be an unfair comparison. Additionally, the annotations of AudioCaps are highly correlated with the category annotations of AudioSet, which raises concerns about the potential for label leakage. Therefore, I question whether it is appropriate to use AudioSet to validate zero-shot classification in Table 3.\n3.The datasets used in the experiments are mostly quite old; it might be worth considering more updated ones.\n4.In my opinion the methods used in this paper fall under multi-task learning. Please provide comparisons with multi-task methods to demonstrate the advantages of the approach presented in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CapbGjC48G", "reviewer": "Reviewer_qoeh", "summary": "This paper presents a novel method for fusing knowledge from mutiple pre-trained spaces across different modalities (including 3D, audio, image, video and language etc.) into a single shared \"high-quality omni representation space\". The core technique involves a dynamically adjusted weighting schema (achieved via a learnable router) to alleviate interference across knowledge sources/modalities. The lightweight router network that controls binding and routing enables the system to be trained efficiently. The authors performed extensive experiments and demonstrated the superior performance of the approach on multiple relevant benchmarks.", "questions": "* The foundational step behind the proposed approach is the quality of curated pseudo item-pair across all modalities. Is there any direct or indirect quality measurement of this step? Meanwhile as the results in Table 2 and Table 3 show, after binding the omni representation sometimes beat the original \"individual best\", suggesting the cross-modality binding can even improve representation capability. It would be interesting to see qualitative examples of such improvement if possible. \n\n* Related to previous question, when a new space needs to be added to the omni representation space, does it require re-computation of all existing spaces? According to equation (4) it seems all embedding spaces need to be adjusted but please provide more clarifications. Also curious if it's possible to iterative do the binding (iteratively do binding -> pseudo item-pair update -> binding etc.) to further boost the performance, if incremental binding (one space at a time) is possible?\n\n* Some minor issues include citations wrong (L53, L60, L124, L155, etc.)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Originality\n\nThis paper presents a novel approach for binding cross-modal pre-trained latent space into a different space. Although similar ideas have been published in several prior work, this paper has made some extensions to enable more efficient pseudo item-pair construction and to be able to scale up the target spaces. \n\n* Quality\n\nThe effectiveness of the proposed framework is validated by extensive experiments over multiple benchmarks, and the model has achieved even better performance than \"individual best\" (specialist source model on the target modality only).\n\n* Clarity\n\nThe paper is in good shape with key ideas being clearly presented by text and figure illustrations. It is relatively easy to follow.\n\n* Significance\n\nThe proposed idea can be easily adapted to new modalities and may lead to more exciting applications.", "weaknesses": "* Arguably the idea presented in this paper is a natural extension thus incremental to previous work, e.g. the main change from FreeBind would be the pseudo-item generation (from pseudo-embedding pair aggregation to pseudo-item pair retrieval) and learnable weights routing. \n\n* One missing ablation (correct me if I missed anything) would be a side-by-side comparison of pseudo item pair retrieval vs. pseudo embedding aggregation (e.g. FreeBind), as the authors claim the proposed approach is more efficient and robust. The ablation can be done by applying pseudo item pair retrieval to the same experimental setup as pseudo embedding aggregation if the latter doesn't support more datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x5cAkIsyJv", "reviewer": "Reviewer_enUq", "summary": "This paper presents OmniBind, a framework designed to integrate diverse multimodal models into a unified representation space. This paper introduces a learnable router that dynamically assigns weights to different input combinations, mitigating interference between knowledge sources and enhancing versatility. By binding various pre-trained models, OmniBind achieves notable performance across multiple benchmarks and demonstrates versatility in handling different modalities.", "questions": "1. This paper mentions in the abstract that OmniBind is extremely training-efficient (line 041). How is this reflected?\n2. Did the paper consider the issue of expert load, and if so, how was it addressed?\n3. On line 199 of the paper, the authors mention the advantages compared to FreeBind. How is item-pair retrieval done here?\n4. If one wants to support a new modality under this method, how should it be carried out?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper presents an approach OmniBind leverages the strengths of specialized models.\n2. This paper addresses the limitation of fixed-weight averaging and offers a flexible way to integrate multiple knowledge sources.\n3. This paper shows the potential of OmniBind for practical applications such as 3D-audio retrieval, any-query object localization, and complex multimodal compositional understanding.", "weaknesses": "1. This paper is somehow hard to follow and not well-written. The motivation and method sections may not be sufficiently clear. This paper contains some inconsistencies in notation and symbols, which is confusing for readers. For instance, the term \"item pair {p, a, v, t}\" on line 211 is not clearly defined or explained when it first appears.\n2. This paper does not address how the introduction of OmniBind affects training and inference speeds. \n3. This paper lacks detailed discussions on certain aspects of the method. For example, expert load balancing in the Mixture of Experts (MoE) has not been explored.\n4. While the paper mentions that textual descriptions of different modalities exhibit significant biases (on line 258), it does not provide supporting evidence or references to illustrate these biases.\n5. The generalizability to new modalities is not presented in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)