Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representations

Authors: Minoh Jeong, Min Namgung, Zae Myung Kim, Dongyeop Kang, Yao-Yi Chiang, Alfred O. Hero
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: multimodal learning, representation learning, unified representation, ImageBind, shared embedding space, CentroBind

Abstract:
Multimodal learning plays a crucial role in enabling machine learning models to fuse and utilize diverse data sources, such as text, images, and audio, to support a variety of downstream tasks. A unified representation across various modalities is particularly important for improving efficiency and performance. Recent binding methods, such as ImageBind (Girdhar et al., 2023), typically use a fixed anchor modality to align multimodal data in the anchor modal embedding space. In this paper, we mathematically analyze the fixed anchor binding methods and uncover notable limitations: (1) over-reliance on the choice of the anchor modality, (2) failure to capture intra-modal information, and (3) failure to account for inter-modal correlation among non-anchored modalities. To address these limitations, we propose CentroBind, a simple yet powerful approach that eliminates the need for a fixed anchor; instead, it employs dynamically adjustable centroid-based anchors generated from all available modalities, resulting in a balanced and rich representation space.
We theoretically demonstrate that our method captures three crucial properties of multimodal learning: intra-modal learning, inter-modal learning, and multimodal alignment, while also constructing a robust unified representation across all modalities. Our experiments on both synthetic and real-world datasets demonstrate the superiority of the proposed method, showing that dynamic anchor methods outperform all fixed anchor binding methods as the former captures more nuanced multimodal interactions.

Decision: reject
Reviewer scores: 5, 8, 3, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1729403362436

thread_type:
official_review

<thread_object>
creator: reviewer_iopy

<content>
summary:
This paper proposes CentroBind, which employs a dynamically adjustable centroid-based anchors generated from all available modalities. The experiments are conducted on synthetic and real-world datasets.

strengths:
1. The paper is easy-to-follow.
2. The paper provides sufficient analysis of the proposed method.

weaknesses:
1. CentroBind computes the average of the inputs in the batch as the anchors. However, means are sensitive to outliers. When there are outliers in a batch, the performance will be influenced. Particularly, when the batch size is small, it will also influence the performance.
2. The experiments are not sufficient. There are many unified multimodal representation methods. Why do the authors only compare FABind? More recent baselines should be included.
3. I consider this paper to be incremental work, as it makes only very minor changes to FABind.
4. There are no visualizations in the experiments. Features before and after CentroBind should be visualized to better demonstrate its effectiveness.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730097960834

thread_type:
official_review

<thread_object>
creator: reviewer_ncrk

<content>
summary:
The paper addressed the issues related to fixing a particular modality as an anchor in multi-modal representations. They argue that the anchor should not be fixed since the semantic relationships among the non-anchors may be lost and also since intra-modality information  may not be captured if the anchor is fixed. Instead, they propose choosing centroids of positive augmentation pairs among multiple modalities. The paper presents theoretical analyses demonstrating the drawbacks of fixed anchors and the advantages of dynamic anchors. Experimental results show the superiority of the proposed strategy on a synthetic dataset and a multimodal dataset for cross-modal retrieval and, sarcasm and speaker classification.

strengths:
1. A strong theoretical paper that shows that fixing an image or text as an anchor for multimodal representation loses information contained in other modalities that are not selected as anchors.
2. The proposed strategy of using dynamic anchors is also supported theoretically by deriving a lower bound on CentroBind's objective function and minimizing the objective function.
3. Very good interpretations based on theoretical analysis of why FABind is insufficient and how CentroBind overcomes the former's limitations.

weaknesses:
There are no major weaknesses to speak of. 
1. To me, the claim of a closer representation to Huh et al's "Platonic representation" is not  quite evident. I tend to believe the authors are also not very positive about this aspect from their statement in line 361 that the representation is "likely closer" to the Platonic representation.
2. This is not a weakness of the paper per se. Audio, video and text are the usual modalities that are discussed in multimodal representation. It would be interesting to study datasets with additional modalities such as VIDIMU: Multimodal video and IMU kinematic dataset on daily life activities using affordable devices (https://zenodo.org/record/8210563), which has inertial sensors.

questions:
1. Line 220: The authors may clarify why equation (4) shows that FABind does not guarantee shared information fbetwen non-anchor modalities.
2. Are there any constraints on the batch B from which the anchor embeddings are derived?
3. It is not clear which part of the analysis specifically shows that using a fixed anchor affects the representativeness of all modalities (P4).
4. How would acc(All) perform in Fig 2 (a)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730481799239

thread_type:
official_review

<thread_object>
creator: reviewer_afcb

<content>
summary:
**Summary of this paper:**

This work introduces a new task: CentroBind, a novel approach in multimodal learning that addresses limitations found in fixed-anchor binding methods(FABIND). FABIND typically uses a single anchor modality to align representations across various modalities. However, this method has critical drawbacks, such as dependence on the anchor choice and lack of intra- and inter-modal information sharing. CentroBind proposes to replace fixed anchors with dynamic centroid-based anchors derived from all available modalities, creating a more balanced representation space. Theoretical analysis and experiments on synthetic and real-world datasets demonstrate that CentroBind enhances multimodal alignment and performs better in tasks requiring nuanced multimodal interactions.

**Strengths:** 

The paper is well-organized and written. The paper proposes an innovative solution to the limitations inherent in fixed-anchor binding methods by introducing CentroBind, which leverages centroid-based dynamic anchors. CentroBind's design is rooted in solid theoretical analysis, addressing crucial aspects of multimodal learning, including intra-modal, inter-modal, and multimodal alignment. The paper conducted extensive experiments on both synthetic and real-world datasets, and the results confirmed that CentroBind outperformed FABIND. The research content is innovative and holds potential application value.

**Weakness:**

As for method. The theoretical advantages of dynamic centroid-based anchors are well-articulated, the practical implementation details are somewhat limited.As for the experimental results. (1) I believe that the comparative experiments and ablation studies are not sufficiently. The paper only compares CentroBind and FABIND methods, lacking additional comparisons with other recent multi-modal alignment frameworks, which leads me to question the validity of this comparison. (2) There could be more exploration into how CentroBind performs under different data distributions and noise levels, which could provide further insights into its robustness.

**Weakness Summary.** 

CentroBind's scalability and practicality for large datasets require further consideration. In addition, broadening experimental comparisons and analysis would help address potential limitations and position the method within the broader multimodal research.

strengths:
Refer to the summary

weaknesses:
Refer to the summary

questions:
Refer to the summary
</content>
</thread_object>

</thread>
<thread>
creation date:
1730495810915

thread_type:
official_review

<thread_object>
creator: reviewer_ykp7

<content>
summary:
The paper removes the need for selecting a fixed anchor modality in Fixed-Anchor-Bind (FABIND) and uses the centroid of all modality representations as an anchor representation. The paper study their method theoretically, showing that it captures intra- and inter information and shows empirical improvements over various synthetic and real-world datasets.

strengths:
The paper addresses challenges with Fixed-Anchor Binding (FABIND) and introduces a centroid-based anchor approach as an effective alternative. This method, Centro BIND, demonstrates superior performance over FABIND across both synthetic and real-world datasets in retrieval and classification tasks, which is likely to capture the interest of the community. Additionally, the paper is concise, well-written, and easy to follow.

weaknesses:
The paper addresses challenges with Fixed-Anchor Binding (FABIND) and introduces a centroid-based anchor approach as an effective alternative. This method, Centro BIND, demonstrates superior performance over FABIND across both synthetic and real-world datasets in retrieval and classification tasks, which is likely to capture the interest of the community. Additionally, the paper is concise, well-written, and easy to follow.

* **Limited Baseline Comparison**: While the limitations of fixed anchors are well-articulated, additional baselines would help to further justify the centroid-based approach. For example, sampling a modality as an anchor randomly or choosing an anchor based on the minimum or maximum similarity score could provide useful comparisons to strengthen the motivation for centroid-based anchoring.

* **Positioning Among Related Works**: Although the paper’s primary focus is to demonstrate improvement over FABIND, it would benefit from a broader positioning among recent multimodal works [1, 2, 3]. Several studies have explored inter- and intra-modality information; a comparative discussion here would contextualize the contributions more effectively. Including a simple baseline of early or late fusion across modalities for both synthetic and real-world datasets would be insightful. Additionally, it’s unclear why Centro BIND does not utilize all modalities in Table 2 for a more comprehensive comparison.

* **Comparison with Individual Modalities**: Since the work emphasizes intra-modality information, a comparison with individual modality baselines and an ensemble of modalities, especially in Table 2 for evaluation, is essential. This would underscore how effectively Centro BIND leverages intra-modality information compared to single-modality models.

* **Clarification on FABIND Setup**: Could the authors clarify if FABIND uses pre-trained encoders in the experiments, or are the encoders randomly initialized? This detail would help to understand the experimental setup more clearly.

[1] Du et al., On Uni-Modal Feature Learning in Supervised Multi-Modal Learning.  
[2] Madaan et al., A Framework for Multi-modal Learning: Jointly Modeling Inter- & Intra-Modality Dependencies.  
[3] Zhang et al., Multimodal Representation Learning by Alternating Unimodal Adaptation

questions:
Please refer to the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730547741038

thread_type:
official_review

<thread_object>
creator: reviewer_bbki

<content>
summary:
This paper proposes a novel method to eliminate the need for an anchor in multimodal representation. Such method is based on centroid anchors computed from the modalities. The method faces a real problem in the largest part of current multimodal models that mainly rely on text anchors. The proposed method is novel and of interest to the community.

strengths:
1) The paper faces the problem of misalignment among non-anchor modalities, which is a real problem in multimodal alignment.
2) The mathematical solutions and proofs to the problem, as well as the mutual information perspective, make the paper valuable.
3) Although simple, the experiments confirm the theoretical intuitions.
4) The paper is well-written and well-presented.

weaknesses:
1) A tSNE visualization of the learned space could be strengthen the claims.
2) Maybe more detailed experiments with existing embedding models could be conducted to generalize the results. But the paper is fine also as it is now. :)


Minor comments:

1) Supplementary materials should have been submitted together with the paper.
2) The Saxon genitive should be avoided in scientific writing, although I know that both ChatGPT and Grammarly suggest to use it. However, I suggest the authors to remove all the Saxon genitives from the paper.

questions:
1) Could the authors better explain how they dynamically adjust the centroids? I mean, after an iteration, or step by step also inside the iteration?
2) How would this model scale up to multiple (more than three) modalities?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685553816

thread_type:
official_review

<thread_object>
creator: reviewer_2xlp

<content>
summary:
The paper explores limitations in current Fixed-Anchor-Bind (FABIND) methods used for multi-modal learning, which commonly align various modalities using a fixed anchor like images or text.  These methods struggle to capture essential intra-modal and shared information among non-anchored modalities.  To address these shortcomings, the authors propose a novel approach, CentroBind, which dynamically computes centroid-based anchors from all modalities, enhancing representation alignment.  Their experimental results demonstrate that CentroBind outperforms traditional FABIND methods on synthetic and real-world datasets.

strengths:
1、Innovative Approach: The introduction of CentroBind addresses key limitations in existing multi-modal learning methods by using dynamic anchors, offering a fresh perspective and improving multi-modal alignment.


2、Theoretical Rigor: The paper provides a comprehensive mathematical analysis to support the efficacy of CentroBind, ensuring a solid foundation for the proposed approach.

weaknesses:
1、Robustness of anchor generation: CentroBind takes the centroids of all modalities to generate anchors, how does this approach perform in the face of modalities with significantly different information densities? For example, if some modalities contain more or less useful information than others, does the centroid bias towards these modalities, resulting in an imbalance in the representation space?

2、Effect of different modal weights: All modes are treated as equally weighted when calculating the centroid anchor. Have methods been investigated for assigning different weights to each modality to better capture inter-modality differences? Does this weight assignment improve or optimize the overall performance of the model?

3、CentroBind aims to maximize both intra-and inter-modality information, but is there an information bottleneck effect (e.g. information may be compressed in high-dimensional embedding Spaces)? How does this bottleneck effect affect the representation power of the model, especially in tasks where multimodal interactions are very complex? Have experiments been conducted to verify the balance between information maximization and representation compactness?

4、Does CentroBind suffer from potential optimization pitfalls when multiple modalities are strongly correlated or there is obvious synergy information between modalities? For example, when two modes are highly dependent, is the centroid overly pulled towards a particular mode? In this case, how can the optimization process be adjusted or improved to capture real multimodal relationships?

5、Boundary conditions for theoretical analysis: The theoretical analysis in the paper explores the advantages of CentroBind, but are there certain boundary conditions or assumptions under which CentroBind might lose its advantage? For example, does this approach still work when the data is unevenly distributed or the amount of data is severely imbalanced?

6、Loss function Optimization: CentroBind's loss function combines multiple InfoNCE loss terms, but does this design affect the convergence speed or stability of the model? Has the effect of different temperature parameters or weight of the loss term been explored to optimize the convergence of the model?

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732328712023

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to reviewers

comment:
We would like to thank the reviewers for their thoughtful feedback and suggestions.
Overall, the reviewers highlighted the following **strengths** of our paper:
> 1. Proposing an innovative approach to multimodal alignment that overcomes the limitations of existing fixed-anchor-bind (FABind) techniques. (Reviewers: 2XLP, BBKi, NCrk)
2. Demonstrating theoretical rigor and a solid foundation through comprehensive mathematical analyses and proofs supporting the effectiveness of CentroBind. (Reviewers: 2XLP, BBKi, NCrk)
3. Providing empirical validation and showing performance improvements on both synthetic and real-world datasets. (Reviewers: 2XLP, BBKi, NCrk)
4. Addressing real-world challenges with potential for broader impact by tackling the problem of misalignment among non-anchor modalities. (Reviewers: 2XLP, BBKi, NCrk, YKP7, aFcb)
5. Presenting a well-written manuscript characterized by clarity, organization, and ease of understanding, making complex theoretical concepts accessible through a concise writing style. (Reviewers: BBKi, YKP7, aFcb, NCrk)


The reviewers' main **criticisms** can be summarized as follows followed by our general response:
> 1. General concerns about computing centroid anchors by averaging, as it may render the method sensitive to outliers, especially with small batch sizes. Additionally, when one of the modalities is extremely “bad,” this may hurt performance. Reviewers suggest trying weighted averaging. 

Thank you for the feedback; this is a valid point. In our original manuscript, we presented experiments with varying levels of modality quality in Figure 2 and observed that CentroBind demonstrated the best performance. Upon receiving this feedback, we conducted additional experiments where we pushed the modality qualities to the extreme. We compared CentroBind with weighted averaging based on prior knowledge, as well as with random anchoring both with and without intra-learning. Our new results show that CentroBind is robust to these outliers and that shared information among non-anchors—which is one of our contributions—appears to be more critical for performance in the extreme case where one modality contains most of semantic information (and hence other modalities contain partial information without unique information). For more details on these additional experiments, please refer to our response to Reviewer 2XLP. 
    
> 2. The experimental evaluations primarily compare CentroBind with FABind methods, lacking comparisons with other recent and relevant multimodal alignment frameworks.

In this paper, we focus on multimodal "representation" learning via contrastive learning that does not require annotated labels used in supervised learning. We would like to point out that FABind methods are the current state-of-the-art and the most widely used approaches in multimodal representation learning. Many recent approaches are also variants of FABind methods, and one of our main contributions is formally categorizing these methods, providing theoretical analyses, and proposing CentroBind based on these analyses. Additionally, we are currently working on adding another baselines using the UniBind [1] and AudioCLIP [2], which is one of the most recent state-of-the-art approaches with an available codebase. 


> 3. The experiments could be more extensive, exploring CentroBind's performance under varying conditions such as different data distributions, noise levels, and with more modalities. 

In response to [2XLP, YKP7, aFcb, ioPy], we compared various baseline methods for dynamic anchor selection, including weighted average and random anchor (with intra and non-intra learning). A common concern with using centroids is their sensitivity to outliers and inability to capture diverse data distributions. However, our additional experiments under varying conditions of information density and modal dependency demonstrate that CentroBind performs robustly.

Intuitively, a weighted average should outperform centroids as it incorporates additional knowledge about the dataset (e.g., information density and quality). Yet, CentroBind achieves comparable performance, suggesting that our method is robust to modality imbalance. Furthermore, since dataset quality can vary even within a single modality in practice, and modality importance is task-dependent, we argue that centroids are a reasonable choice for dynamic anchor selection within a representation learning framework where the downstream task is unknown.

Nonetheless, it is worth exploring the modality imbalance problem beyond the current setting, such as in supervised representation learning, and we have included a discussion of the modality imbalance problem in the paper.

[1] Lyu et al., UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All.

[2] Guzhov at el., Audioclip: Extending clip to image, text and audio.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This is a borderline paper, but the overall idea of this paper could meet the high quality of an ICLR Paper. This paper experienced a long discussion phase in both reviewer-author and reviewer-AC. The reviewers have clear diverse opinions about this paper, and the rating scores are 5, 8, 3, 6, and 8. Here is a summary of the discussion:  
The key characteristic of this paper is its simplicity, and we all know that a simple but effective algorithm is our ideal idea. However, the primary concerns can be summarized as follows.

***Motivation Issue***  
This paper primarily investigates learnable anchors for multimodal learning, in contrast to the claimed use of fixed anchors. The main comparisons are drawn against fixed-anchor methods, particularly ImageBind [1]. The stated limitations of ImageBind and similar approaches are outlined as follows:  
P1: over-reliance on a single anchor modality;  
P2: failure to capture intra information;  
P3: absence of shared information among non-anchored modalities;  

However, in my view, these motivations require further careful examination.  
For P1, ImageBind utilizes images as anchors to generate or associate with other modalities. This approach is primarily based on the rich and complex features embedded in the visual space, which facilitate the inference of other modalities.  
For P2, ImageBind incorporates intra-modal information in its modality alignment process.  
For P3, the shared information is also utilized in modality alignment, which is a core research challenge in multimodal learning, rather than being absent. Furthermore, ImageBind offers greater flexibility in multimodal learning by enabling seamless transitions from one modality to others, followed by their fusion.  

[1] Girdhar, Imagebind: One embedding space to bind them all. In CVPR 2023.  

***Technical Issue***  
The proposed method is straightforward yet potentially effective for large-scale inference tasks. ImageBind aligns with the principle: "One begets Two; Two begets Three; Three begets all things."  
In contrast, the authors adopt a multi-view contrastive learning approach based on instance-level representations, focusing on the mean features of each instance. However, a deeper investigation into multi-view or multimodal learning using contrastive learning [2] and its subsequent studies [3-6] reveals numerous related works that appear to have been overlooked by the authors.  

Furthermore, the authors claim that intra-modal and shared information are jointly considered using the mean features of each instance. However, in my view, ImageBind is also capable of addressing this scenario for multimodal alignment and fusion within a joint space.  

[2] Tian et al, Contrastive Multiview Coding, ECCV 2020.  
[3] Xing et al. Adaptive cross-modal few-shot learning, NeurIPS 2019.  
[4] Zeng, et al. Pan: Prototype-based adaptive network for robust cross-modal retrieval.SIGIR 2021.  
[5] Huang, et al., Clover: Towards a unified video-language alignment and fusion model, CVPR 2023.  
[6] Wang, et al. Vlmixer: Unpaired vision-language pre-training via cross-modal cutmix, ICML 2022.  

Experiments Issue For ImageBind, this work is based on scaling laws, and sufficient data must be collected to validate its effectiveness across multiple tasks.  

As another reviewer has pointed out, the authors conduct their experiments on a synthetic dataset and a claimed real-world dataset (MUStARD), which raises concerns about fairness. At present, we are unable to fully evaluate the performance comparisons between the proposed algorithm and the open-source ImageBind across the six modalities—images, text, audio, depth, thermal, and IMU data.

***Theory Issue***  
Proposition 1 states FABIND with sufficient anchors. The sufficiency is attributed to large-scale data, an InfoNCE-based encoder, and a transformer-based architecture, as also stated in the main paper. The use of information is clearly articulated, which is no need to use a Proposition to clarify.

Proposition 2 is ‘FABIND with insufficient anchor’. Yes, it is known that information bottleneck theory is always used to define or prove the representation bound Hence, all representation models would exhibit this behavior, as it is a commonly used theory to assess the capabilities of deep learning models.

For Theorem 1, while I have not thoroughly reviewed all the details, it appears to be based on the information bottleneck and bound theory derivation.

In summary, while the overall idea of this paper is straightforward, it is not fully convincing from both experimental and theoretical perspectives. Therefore, this paper will be rejected.