Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representations

## Abstract

Abstract Multimodal learning plays a crucial role in enabling machine learning models to fuse and utilize diverse data sources, such as text, images, and audio, to support a variety of downstream tasks. A unified representation across various modalities is particularly important for improving efficiency and performance. Recent binding methods, such as ImageBind (Girdhar et al., 2023 ) , typically use a fixed anchor modality to align multimodal data in the anchor modal embedding space. In this paper, we mathematically analyze the fixed anchor binding methods and uncover notable limitations: (1) over-reliance on the choice of the anchor modality, (2) failure to capture intra-modal information, and (3) failure to account for inter-modal correlation among non-anchored modalities. To address these limitations, we propose CentroBind, a simple yet powerful approach that eliminates the need for a fixed anchor; instead, it employs dynamically adjustable centroid-based anchors generated from all available modalities, resulting in a balanced and rich representation space.
We theoretically demonstrate that our method captures three crucial properties of multimodal learning: intra-modal learning, inter-modal learning, and multimodal alignment, while also constructing a robust unified representation across all modalities. Our experiments on both synthetic and real-world datasets demonstrate the superiority of the proposed method, showing that dynamic anchor methods outperform all fixed anchor binding methods as the former captures more nuanced multimodal interactions.

## 1 Introduction

Multimodal alignment is defined as identifying and exploiting relationships and correspondences between sub-components of instances from multiple modalities (e.g., text, image, audio) to establish meaningful connections between their representations (Baltrušaitis et al., ).
This process allows machine learning models to analyze heterogeneous data holistically, facilitating comprehensive decision-making. A common approach is learning a shared embedding space (Tu et al., ; Girdhar et al., ; Liang et al., ; Zhu et al., ), which aims to project data from multiple modalities into a common embedding space by clustering similar items together for direct comparison and linkage.
This approach leverages well-trained single-modal embeddings, aligning them with auxiliary objective functions like contrastive (Oord et al., ) or triplet loss (Wang et al., ) to minimize distances between similar items and maximize distances between dissimilar ones across modalities.

Instead of training separate models for each modality, ImageBind (Girdhar et al., ) pairs images with other modalities and projects them into the common image embedding space. Similarly, Zhu et al. () shows that pairing texts with other modalities (LanguageBind) improves retrieval task performance when language is specified as the anchored modality. This approach has inspired various “-Bind” methods tailored to align different modalities for specific domains, such as molecular modeling (Xiao et al., ), medical imaging (Gao et al., ), brain signals (Yang et al., ), and music selection for videos (Teng et al., ). These models commonly use image or text as the anchor embedding due to the abundance of data, with other modalities projected into this anchor representation.

We can define the aforementioned approaches as Fixed-Anchor-Bind (FABind) method, where the embedding space of the primary anchor modality remains fixed during the alignment process. Generally, the “-Bind”-like approaches maximize mutual information $I({\bf Z}_{1};{\bf Z}_{i})$ between the representation ${\bf Z}_{1}$ of the anchor modality and the representations ${\bf Z}_{i},i\in\{2,\cdots,M\}$ of other modalities.
Although these approaches are practically useful and widely adopted in learning unified multimodal representation, we theoretically and empirically demonstrate the limitations in FABind .

#### Issues with fixed anchor binding.

First, selecting an anchor modality is crucial but challenging, as it depends on both embedding quality and task suitability. Common choices like images or text may still be suboptimal, especially for less common modalities lacking high-quality embeddings.
Second, fixing an anchor can result in lost semantic information that might be better represented by other modalities. For instance, while text may describe ’a dog barks loudly,’ sound could reveal mood, and an image could add facial expression, which fixed alignment might miss.
Third, optimizing only for anchor-to-other-modality overlooks information similarity between non-anchored modalities, leading to a loss of complementary insights.
Addressing this among these non-anchored modalities would disrupt overall multimodal alignment, contradicting the primary goal of CentroBind.
We formally analyze deficiencies of the fixed anchor bind approaches in Section 2.

Figure: (a) FABind vs. CentroBind.
Refer to caption: x1.png

#### Dynamic anchor alignment.

We propose a simple yet effective modification by replacing fixed anchors with “dynamic” centroid-based anchors computed from paired samples. Our method, CentroBind, described in Section 3 removes the need for selecting a fixed anchor modality, instead calculates the centroid of all modality representations and generates an anchor representation, as shown in Figure 1(a). Encoders are then trained to minimize the ensemble of InfoNCE loss (Oord et al., ) between this dynamic anchor and other representations, using the centroid as the anchor. The main intuition is that a desirable anchor should be representative of all modalities, capturing the most comprehensive information, with well-trained encoders producing representations that naturally cluster around this shared centroid, reflecting their underlying semantic alignment.

Our theoretical analysis demonstrates that CentroBind effectively addresses three critical components of multimodal learning: 1) intra-modal mutual information, 2) inter-modal mutual information, and 3) multimodal alignment via embedding similarity. By incorporating these elements within the ‘anchor alignment’ framework, CentroBind benefits from the significant advantages of dynamic anchors across both synthetic and real-world datasets in retrieval and classification tasks.
In a sense, our approach yields an ideal unified representation space, supported by the perspective of Huh et al. (), which conjectures that multimodal representations align as they move toward a platonic representation—an ideal form that captures the semantic information of all modalities simultaneously.

## 2 Problem formulation

In this section, we describe general representation learning and representation binding problems in multimodal learning. Then, we analyze fixed-anchor-bind (FABind) methods such as ImageBind (Girdhar et al., ), that bind multimodal representation to a fixed modality of choice.

### 2.1 Representation learning
framework

#### Notation.

Boldface upper case letters (e.g., $\mathbf{X}$) denote random vectors, and a realization is denoted by the boldface lower case letters (e.g., ${\boldsymbol{x}}$);
For $n\in\mathbb{N}$, $[n]:=\{1,2,\cdots,n\}$;
$P_{{\bf X}}$ and $P_{{\bf X},{\bf Y}}$ denote the marginal and the joint distributions of ${\bf X}$ and $({\bf X},{\bf Y})$, respectively.

Given $M$ datasets $\mathcal{D}=\{\mathcal{D}_{i}\}_{i=1}^{M}$, let $\mathcal{D}_{i}=\{({\boldsymbol{x}}_{i,j},{\boldsymbol{y}}_{i,j})\}_{j=1}^{N_{
i}}$ be the dataset from the $i$-th modality, where ${\boldsymbol{x}}_{i,j}\in\mathcal{X}_{i}$ and ${\boldsymbol{y}}_{i,j}\in\mathcal{Y}_{i}$ are respectively the $j$-th input instance (e.g., feature vector) and the corresponding label in $i$-th modality, and we assume that $({\boldsymbol{x}}_{i,j},{\boldsymbol{y}}_{i,j})\overset{\rm i.i.d.}{\sim}P_{{
\bf X}_{i},{\bf Y}_{i}}$.(^1^11In self-supervised learning, labels might not exist, which corresponds to the case that ${\boldsymbol{y}}_{i,j}$’s are empty.)
We assume that $j$ indexes paired samples among modalities. For instance, ${\boldsymbol{x}}_{1,c}$ and ${\boldsymbol{x}}_{2,c}$ are features having similar semantic information (e.g., dog image and dog sound) in $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$.
The goal of representation learning is to build $M$ encoders $f_{i}:\mathcal{X}_{i}\to\mathcal{Z}_{i}$ for each modality, which maps the input instances ${\boldsymbol{x}}_{i,j}$ to its embedding ${\boldsymbol{z}}_{i,j}=f_{i}({\boldsymbol{x}}_{i,j})$, preserving as much information about ${\boldsymbol{x}}_{i,j}$ as possible.

For the uni-modal case ($M=1$), keeping maximum information about ${\boldsymbol{x}}_{1,j}$ at its embedding ${\boldsymbol{z}}_{1,j}$ is generally preferred based on the “InfoMax” principle (Linsker, ), under which the objective is to maximize mutual information $I({\bf X}_{i};f({\bf X}_{i}))$ between ${\bf X}_{i}$ and $f({\bf X}_{i})$.
Throughout the paper, we call $I({\bf X}_{i};f({\bf X}_{i}))$ intra information on ${\bf X}_{i}$.
For the multimodal case ($M\geq 2$), on top of the InfoMax principle, “minimal sufficiency” is proposed in (Tian et al., ), which suggests maximizing shared information $I(f_{i}({\bf X}_{i});f_{l}({\bf X}_{l}))$ between $f_{i}({\bf X}_{i})$ and $f_{l}({\bf X}_{l})$, while minimizing the unique information $I({\bf X}_{i};f_{i}({\bf X}_{i})|\{{\bf X}_{l}\}_{l\neq i})$. Although minimal sufficiency often leads to an efficient encoder with a better performance in numerous multimodal downstream tasks, it is not always a good strategy as there exist exceptions where the unique information on an individual modality is crucial (Liang et al., ; Wang et al., ). In other words, the optimality of minimal sufficiency is task-dependent.
To avoid task dependency, we do not consider minimal sufficiency; instead, we maximize intra and shared information without reducing unique information.
Next, we formalize the notion of sufficient embedding.

###### Definition 1 ( $\mathcal{Z}_{i}$ -Sufficient embedding of ${\bf X}_{i}$ for ${\bf X}_{l}$ ) .

For embedding space $\mathcal{Z}$, the embedding $f_{i}({\bf X}_{i})$ is $\mathcal{Z}_{i}$-sufficient for ${\bf X}_{l}$ if and only if

$$ $\displaystyle f_{i}\in{\hbox{arg}}\max_{f:\mathcal{X}_{i}\to\mathcal{Z}_{i}}I( f({\bf X}_{i});{\bf X}_{l}),$ (1) $$

and we call $f_{i}$ sufficient encoder of ${\bf X}_{i}$ for ${\bf X}_{l}$.

We note that if $i=l$, the sufficient encoder provides embeddings with maximum intra information, and if $i\neq l$, it gives embeddings with maximum shared information between $i$-th and $l$-th modalities.(^2^22With a proper choice of $\mathcal{Z}_{i}$ ensuring $\max_{f:\mathcal{X}_{i}\to\mathcal{Z}_{i}}I(f({\bf X}_{i});{\bf X}_{l})=I({\bf X }_{i};{\bf X}_{l})$, Definition 1 says that ${\boldsymbol{z}}_{i,j}=f_{i}({\boldsymbol{x}}_{i,j})$ is a sufficient statistic (Polyanskiy & Wu, ) of ${\boldsymbol{x}}_{i,j}$ for ${\boldsymbol{x}}_{l,j}$ as the encoding entails no information loss.)

In the context of contrastive representation learning, with a goal of attaining sufficient encoders in Definition 1, InfoNCE loss $I_{\rm NCE}(X;Y)$ is often employed since it relates to mutual information. Specifically, InfoNCE provides a lower bound on mutual information, i.e., $I({\bf X};{\bf Y})\geq-I_{\rm NCE}({\bf X};{\bf Y})$ (Oord et al., ), thus minimizing InfoNCE leads to an increase in mutual information. InfoNCE loss between embeddings ${\bf U}$ and ${\bf V}$ can be written as follows:

$$ $\displaystyle I_{\rm NCE}({\bf U};{\bf V}|\tau)$ $\displaystyle=\mathbb{E}_{P_{{\bf U},{\bf V}},\prod_{i=1}^{N}P_{{\bf V}_{i}}} \left[-\log\frac{\exp({\bf U}^{\top}{\bf V}/\tau)}{\exp({\bf U}^{\top}{\bf V}/ \tau)+\sum_{i=1}^{N}\exp({\bf U}^{\top}{\bf V}_{i}/\tau)}\right],$ (2) $$

where the expectation is taken with respect to $P_{{\bf U},{\bf V}}\prod_{i=1}^{N}P_{{\bf V}_{i}}$. Here, we say $({\bf U},{\bf V})\sim P_{{\bf U},{\bf V}}$ a positive pair and $({\bf U},{\bf V}_{i})\sim P_{\bf U}P_{{\bf V}_{i}}$ a negative pair. Moreover, $N\geq 1$ and $\tau>0$ are hyper-parameters, specifying the number of negative samples and the temperature parameter.
For simplicity, we assume that embeddings are normalized (Wang & Isola, ) and are of the same dimensionality in this paper.
Under the assumption of $\tau=1$, the exponent ${\bf U}^{\top}{\bf V}/\tau$ in (2) is the cosine similarity score between ${\bf U}$ and ${\bf V}$.

###### Definition 1 ( 𝒵 i subscript 𝒵 𝑖 \mathcal{Z}_{i} caligraphic_Z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT -Sufficient embedding of 𝐗 i subscript 𝐗 𝑖 {\bf X}_{i} bold_X start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT for 𝐗 l subscript 𝐗 𝑙 {\bf X}_{l} bold_X start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ) .

### 2.2 Binding representation spaces

In addition to the objective of intra and shared information, multimodal learning often takes into account multimodal alignment (Radford et al., ; Duan et al., ). Without multimodal alignment, each modality has its own embedding structure depending on its encoder.
For example, embeddings of cat and dog images, respectively, locate around $(1,0)$ and $(0,2)$ in $\mathbb{R}^{2}$, whereas embeddings of cat and dog text can lie around $(0,2)$ and $(1,0)$. Such a misalignment can happen even for sufficient encoders (Definition 1), since the mutual information is invariant to one-to-one mappings (Polyanskiy & Wu, ).

To align multimodal embedding spaces, a unified representation space (Radford et al., ; Zhou et al., ) or multimodal alignment (Wang et al., ; Liang et al., ) have gained attention and become important property in multimodal representation learning, in which embeddings of multimodal features having similar semantic should locate nearby.
Among various techniques, efficient FABind methods have been proposed (see Appendix A for a summary of FABind methods) , such as ImageBind (Girdhar et al., ).
Their main idea is to set the image modality as a fixed anchor modality, and they minimize the InfoNCE loss between the embeddings of the anchor modality and the other modalities.
More broadly, FABind (e.g., ImageBind) aims to find encoders $f_{i}^{\rm FB}$ for all modalities except the anchor modality such that

$$ $\displaystyle f^{\rm FB}_{i}={\hbox{arg}}\min_{f_{i}:\mathcal{X}_{i}\to \mathcal{Z}_{i}}I_{\rm NCE}(f_{1}({\bf X}_{1});f_{i}({\bf X}_{i})),~{}\forall i \in\{2,\cdots,M\},$ (3) $$

where $f_{1}$ is the encoder for the anchor modality (e.g., image encoder in ImageBind).
Note that FABind freezes $f_{1}$, which is initialized by an existing pretrained model, during the optimization.

### 2.3 Analysis of FABind

In this section, we characterize limitations of FABind regarding the important objective in multimodal representation learning, such as intra and shared information.
To this end, we consider (3) as

$$ $\displaystyle f^{\rm FB}_{i}={\hbox{arg}}\max_{f_{i}:\mathcal{X}_{i}\to \mathcal{Z}_{i}}I(f_{1}({\bf X}_{1});f_{i}({\bf X}_{i})),~{}\forall i\in\{2, \cdots,M\},$ (4) $$

reflecting the fact that minimizing InfoNCE loss leads to maximizing mutual information.(^3^33In contrast to (3), $f_{i}^{\rm FB}$ in (4) might not be aligned with other modalities due to the one-to-one mapping invariant property of mutual information. However, we here do not analyze multimodal alignment of FABind from (4), but rather investigate the performance of encoders in terms of the sufficiency in Definition 1.)
Let FABind encoders from (4) for each modality be $\mathcal{F}^{\rm FB}=\{f_{1},f_{2}^{\rm FB},\cdots,f_{M}^{\rm FB}\}$. The anchor encoder $f_{1}$ is fixed during the entire FABind procedure.
Moreover, we assume that $I(f_{1}({\bf X}_{1});f_{i}^{\rm FB}({\bf X}_{i}))=I(f_{1}({\bf X}_{1});{\bf X}
_{i})$ is the maximum value that can be achieved by (4) due to data processing inequality (Polyanskiy & Wu, ).
We next demonstrate that the quality of anchor embedding $f_{1}({\bf X}_{1})$ significantly impacts the performance of $\mathcal{F}^{\rm FB}$ in terms of shared information.
The following propositions show the dependency of FABind on anchor embedding quality.

###### Proposition 1 ( FABind with sufficient anchor) .

Let $f_{1}^{\rm suf}({\bf X}_{1})$ be a sufficient embedding of the anchor ${\bf X}_{1}$, and let ${\bf X}_{i},i\in[M]$ be discrete.
Assume that $f_{i}^{\rm FB},i\in\{2,\cdots,M\}$ are obtained by (4) with a sufficient anchor encoder $f_{1}=f_{1}^{\rm suf}$, i.e., $I(f_{1}^{\rm suf}({\bf X}_{1});f_{i}^{\rm FB}({\bf X}_{i}))=I(f_{1}^{\rm suf}(
{\bf X}_{1});{\bf X}_{i})$.
Then,

$$ $\displaystyle I(f_{1}^{\rm suf}({\bf X}_{1});f_{i}^{\rm FB}({\bf X}_{i}))=I({ \bf X}_{1};{\bf X}_{i}),~{}\forall i\in\{2,\cdots,M\}.$ (5) $$

###### Proof.

The proof can be found in Appendix B.1.
∎

###### Proposition 2 ( FABind with insufficient anchor) .

Let $f_{1}^{\rm ins}({\bf X}_{1})$ be insufficient embedding of the anchor ${\bf X}_{1}$ for ${\bf X}_{1}$, in the sense that there exists some $\epsilon>0$ such that $I(f_{1}^{\rm ins}({\bf X}_{1});{\bf X}_{1})<\epsilon\leq\max_{f}I(f({\bf X}_{1
});{\bf X}_{1})$.
Assume that $f_{i}^{\rm FB},i\in\{2,\cdots,M\}$ are obtained by (4) with $f_{1}=f_{1}^{\rm ins}$, i.e., $I(f_{1}^{\rm ins}({\bf X}_{1});f_{i}^{\rm FB}({\bf X}_{i}))=I(f_{1}^{\rm ins}(
{\bf X}_{1});{\bf X}_{i})$.
Then,

$$ $\displaystyle I(f_{1}^{\rm ins}({\bf X}_{1});f_{i}^{\rm FB}({\bf X}_{i}))< \epsilon,~{}\forall i\in\{2,\cdots,M\}.$ (6) $$

###### Proof.

The proof can be found in Appendix B.2.
∎

Proposition 1 shows that the FABind encoders $\mathcal{F}^{\rm FB}$ learned with a sufficient anchor embedding can achieve the maximum shared information between the anchor and the other modalities.
However, it does not guarantee the shared information between non-anchored modalities $I(f_{i}({\bf X}_{i});f_{l}({\bf X}_{l})),~{}i,l\neq 1$, which can also be seen from (4).
Proposition 2 establishes that an insufficient anchor may lead to a reduction of shared information between the anchor and the other modalities, implying that the performance of FABind solely depends on the quality of the anchor.

The analysis reveals several limitations in FABind.
Firstly, achieving maximum shared information requires sufficient anchor representation, which depends on having both an informative modality and a sufficient encoder. Without these conditions, FABind may not effectively capture shared information.
Secondly, even with sufficient anchor representation, FABind may not provide encoders with maximum intra information. This is because its objective function (4) does not take into account intra information.
Thirdly, FABind’s objective function (4) focuses solely on learning shared information between pairs of anchor and non-anchored modalities, while disregarding shared information among non-anchor modalities. This limitation renders FABind less effective when significant shared information exists among non-anchored modalities.
Lastly, the representation produced by FABind may not approximate an ideal representation, such as “Platonic representation” (Huh et al., ). The Platonic representation is an idealized depiction of reality, which generates all modalities through projections. While integrating all modalities is crucial for constructing such a comprehensive representation, FABind’s approach, which focuses solely on the anchor modality, falls short of this ideal. A more promising direction, which we pursue in this paper, involves learning representations from multiple modalities, fully leveraging fine-grained, sample-level information.

In summary, FABind exhibits the following weaknesses:

- P1:
over-reliance on a single anchor modality;
- P2:
failure to capture intra information;
- P3:
absence of shared information among non-anchored modalities;
- P4:
fundamental limitation of a fixed anchor in representativeness of all modalities.

Next, we propose our method CentroBind, which does not require any choice of anchor modality and is able to capture intra and shared information.

###### Proposition 1 ( FABind with sufficient anchor) .

###### Proof.

###### Proposition 2 ( FABind with insufficient anchor) .

###### Proof.

## 3 Toward a desirable unified representation space

Our method, CentroBind, is a variant of Bind methods, but does not require choosing a single anchor modality. The main intuition deriving CentroBind is as follows: 1) A desirable multimodal embedding should capture representation of all modalities; 2) A desirable unified embedding should attain the highest average alignment.
To this end, we generate the anchor representation from all modalities by calculating the centroid of the modalities’ representations. Then, we train the encoders toward minimizing the InfoNCE loss between anchor and other modalities, which is identical to FABind with the generated anchor instead of a fixed anchor.
Here we formally describe CentroBind, and show that the method aligns multimodal representations and simultaneously maximizes intra and shared information.

Figure: Algorithm 1 CentroBind

### 3.1 CentroBind

To present a general framework for CentroBind, we consider $M$ modalities with corresponding encoders $\{f_{i}\}_{i=1}^{M}$.
The algorithmic presentation of CentroBind is in Algorithm 1, and a graphical illustration is given in Figure 1(b). In the following, we elaborate detailed steps of CentroBind.

#### Initial encoders.

We initialize $M$ encoders $f_{i}:\mathcal{X}_{i}\to\mathcal{Z},~{}\forall i\in[M]$ for the $M$ modalities. These encoders can either be pretrained models (i.e., backbones) or parameterized models with random weights. The primary constraint for these initial encoders is that their output space must be $\mathcal{Z}$. This ensures that all data samples are mapped to the same output space $\mathcal{Z}$.
When using pretrained encoders that produce embeddings in different output spaces, we attach projection layers to standardize the shapes of the embeddings, ensuring consistency of output space across modalities.

#### Anchor embedding.

Recall that ${\boldsymbol{x}}_{i,j}\in\mathcal{D}_{i}$ denotes the $j$-th feature in the $i$-th modality, where $j$ indexes positive pairs of features (e.g., different views of the same object).
In each training iteration of CentroBind, we need to compute an anchor embedding ${\boldsymbol{a}}_{j}$ for the $j$-th multimodal positive features $\{{\boldsymbol{x}}_{i,j}\}_{i=1}^{M}$. This anchor ${\boldsymbol{a}}_{j}$ serves as a desirable aligned embedding for these features. The anchor ${\boldsymbol{a}}_{j}$ is calculated as follows:

$$ $\displaystyle{\boldsymbol{a}}_{j}$ $\displaystyle=\text{mean}\left(\{f_{i}({\boldsymbol{x}}_{i,j}^{\prime})\}_{i} \right),$ (7) $$

where $\text{mean}(\cdot)$ denotes the mean operator that computes the average of its input, and ${\boldsymbol{x}}_{i,j}^{\prime}$ represents an augmented version of ${\boldsymbol{x}}_{i,j}$.
If $\{{\boldsymbol{x}}_{i,j}\}_{i=1}^{M}$ are available in multimodal datasets, the anchor is given by ${\boldsymbol{a}}_{j}=\frac{1}{M}\sum_{i=1}^{M}f_{i}({\boldsymbol{x}}_{i,j}^{
\prime})$. If only $m<M$ positive pairs are present among $M$ modalities, the anchor is given by ${\boldsymbol{a}}_{j}=\frac{1}{m}\sum_{i\in\mathcal{I}_{j}}f_{i}({\boldsymbol{x
}}_{i,j}^{\prime})$, where $\mathcal{I}_{j}$ is the set of indices of modalities having the $m$ available features.

#### Binding encoders to the anchor.

Once anchor embeddings $\{{\boldsymbol{a}}_{j}\}_{j}$ are derived from a batch of data $B=\{{\boldsymbol{x}}_{i,j}\}_{i,j}$, CentroBind aligns each modality-specific encoder’s embedding with the anchor embedding by minimizing the InfoNCE loss. Specifically, let ${\bf A}=\text{mean}(\{f_{i}({\bf X}_{i})\}_{i})$ represent the anchor embedding variable. Then, CentroBind aims to minimize the InfoNCE loss $I_{\rm NCE}({\bf A};f_{i}({\bf X}_{i}))$ across all modalities $i\in[M]$. A detailed expression for this loss is provided in (8).

For symmetry, CentroBind optimizes the following loss function:

$$ $\displaystyle\mathcal{L}_{\rm CB}(f_{i}|\tau)$ $\displaystyle=I_{\rm NCE}({\bf A};f_{i}({\bf X}_{i})|\tau)+I_{\rm NCE}(f_{i}({ \bf X}_{i});{\bf A}|\tau),$ (8) $$

where $\mathcal{L}_{\rm CB}(f_{i}|\tau)$ denotes the loss function for the $i$-th modality. In particular, with a batch data $B=\{{\boldsymbol{x}}_{i,j}:i\in[M],j\in\mathcal{I}_{B}\}$, the loss can be computed as

$$ $\displaystyle I_{\rm NCE}({\bf A};f_{i}({\bf X}_{i})|\tau)$ $\displaystyle=-\frac{1}{|\mathcal{I}_{B}|}\sum_{k=1}^{|\mathcal{I}_{B}|}\log \frac{\exp({\boldsymbol{a}}_{k}^{\top}f_{i}({\boldsymbol{x}}_{i,k})/\tau)}{ \sum_{j\in\mathcal{I}_{B}}\exp({\boldsymbol{a}}_{k}^{\top}f_{i}({\boldsymbol{x }}_{i,j})/\tau)}~{}\text{ and }$ (9a) $\displaystyle I_{\rm NCE}(f_{i}({\bf X}_{i});{\bf A}|\tau)$ $\displaystyle=-\frac{1}{|\mathcal{I}_{B}|}\sum_{k=1}^{|\mathcal{I}_{B}|}\log \frac{\exp({\boldsymbol{a}}_{k}^{\top}f_{i}({\boldsymbol{x}}_{i,k})/\tau)}{ \sum_{j\in\mathcal{I}_{B}}\exp(f_{i}^{\top}({\boldsymbol{x}}_{i,k}){ \boldsymbol{a}}_{j}/\tau)}.$ (9b) $$

### 3.2 Theoretical analysis of CentroBind

We start by providing a lower bound of CentroBind’s objective function $\mathcal{L}_{\rm CB}(f_{i}|\tau)$ (8) in Theorem 1, followed by an analysis of minimizing $\mathcal{L}_{\rm CB}(f_{i}|\tau)$.

###### Theorem 1 .

Consider $B=\{{\boldsymbol{x}}_{i,j}:i\in[M],j\in\mathcal{I}_{B}\}$ with a set of indices $\mathcal{I}_{B}$, where ${\boldsymbol{x}}_{i,j}$ is the $j$-th sample of $i$-th modality. Then, for any encoders $\{f_{i}\}_{i}$ and for any $\tau>0$, (9a) is bounded as

$$ $\displaystyle I_{\rm NCE}\left({\bf A};f_{i}({\bf X}_{i})\;\middle|\;{\tau}\right)$ $\displaystyle\geq\frac{1}{|\mathcal{I}_{B}|}\sum_{l=1}^{M}I_{\rm NCE}\left(f_{ l}({\bf X}_{l}^{\prime});f_{i}({\bf X}_{i})\;\middle|\;\frac{{\tau}M}{| \mathcal{I}_{B}|}\right)-\frac{1}{|\mathcal{I}_{B}|}\sum_{k=1}^{|\mathcal{I}_{ B}|}\log C_{\mathcal{F},k,i},$ (10) $$

where $C_{\mathcal{F},k,i}=\frac{(c_{\mathcal{F},k,i}^{\min}+c_{\mathcal{F},k,i}^{
\max})^{2}}{4c_{\mathcal{F},k,i}^{\min}c_{\mathcal{F},k,i}^{\max}}$
with $g(l,j|k,i):=\exp\left(\frac{|\mathcal{I}_{B}|f_{l}^{\top}({\boldsymbol{x}}_{l,
k}^{\prime})f_{i}({\boldsymbol{x}}_{i,j})}{\tau M}\right)$,

$$ $\displaystyle c_{\mathcal{F},k,i}^{\min}=\min_{l\in[M],j\in\mathcal{I}_{B}}g(l ,j|k,i),~{}\text{ and }~{}c_{\mathcal{F},k,i}^{\max}=\max_{l\in[M],j\in \mathcal{I}_{B}}g(l,j|k,i).$ (11) $$

###### Proof.

The proof is provided in Appendix B.3.
∎

Theorem 1 provides a lower bound of $I_{\rm NCE}\left({\bf A};f_{i}({\bf X}_{i})\;\middle|\;{\tau}\right)$ in (9a), which is a part of CentroBind objective $\mathcal{L}_{\rm CB}(f_{i}|\tau)$.
Thus CentroBind consequentially minimizes the lower bound (10) that consists of two terms, $\sum_{l=1}^{M}I_{\rm NCE}\left(f_{l}({\bf X}_{l}^{\prime});f_{i}({\bf X}_{i})
\;\middle|\;\frac{{\tau}M}{|\mathcal{I}_{B}|}\right)$ and $-\sum_{k=1}^{|\mathcal{I}_{B}|}\log C_{\mathcal{F},k,i}$. We next provide some intuitive explanation on the effect of such a minimization of the lower bound.

###### Theorem 1 .

###### Proof.

#### The effect of minimizing ∑ l = 1 M I NCE ( f l ( 𝐗 l ′ ) ; f i ( 𝐗 i ) | τ ⁢ M | ℐ B | ) \sum_{l=1}^{M}I_{\rm NCE}\left(f_{l}({\bf X}_{l}^{\prime});f_{i}({\bf X}_{i})%
\;\middle|\;\frac{{\tau}M}{|\mathcal{I}_{B}|}\right) ∑ start_POSTSUBSCRIPT italic_l = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_M end_POSTSUPERSCRIPT italic_I start_POSTSUBSCRIPT roman_NCE end_POSTSUBSCRIPT ( italic_f start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT ( bold_X start_POSTSUBSCRIPT italic_l end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) ; italic_f start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( bold_X start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) | divide start_ARG italic_τ italic_M end_ARG start_ARG | caligraphic_I start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT | end_ARG ) .

The objective of minimizing $\sum_{l=1}^{M}I_{\rm NCE}(f_{l}({\bf X}_{l}^{\prime});f_{i}({\bf X}_{i})\mid
\frac{\tau M}{|\mathcal{I}_{B}|})$ involves reducing several InfoNCE losses. Here, each term in the sum represents the InfoNCE loss between embeddings $f_{l}({\bf X}_{l}^{\prime})$ from modality $l$ and $f_{i}({\bf X}_{i})$ from modality $i$, with $\frac{\tau M}{|\mathcal{I}_{B}|}$ being the temperature parameter for scaling the loss.
This summation can be divided into two components:
1) Intra Information: When $l=i$, the term
measures the similarity between embeddings within the same modality. Minimizing this loss enhances the representation of modality $i$, improving intra information;
2) Shared Information: When $l\neq i$, the term
measures the similarity between embeddings from different modalities. Minimizing these losses helps in learning shared information between modalities, contributing to a more coherent multimodal representation.

By optimizing this summation, CentroBind effectively captures both intra and shared information. This results in a more accurate representation for each modality.
In contrast, as noted in Section 2.3, FABind does not adequately capture intra information and shared information between non-anchored modalities. This limitation highlights the advantage of CentroBind in achieving a more integrated multimodal representation than fixed anchor binding methods.

#### The effect of minimizing − ∑ k = 1 | ℐ B | log ⁡ C ℱ , k , i superscript subscript 𝑘 1 subscript ℐ 𝐵 subscript 𝐶 ℱ 𝑘 𝑖 -\sum_{k=1}^{|\mathcal{I}_{B}|}\log C_{\mathcal{F},k,i} - ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT | caligraphic_I start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT | end_POSTSUPERSCRIPT roman_log italic_C start_POSTSUBSCRIPT caligraphic_F , italic_k , italic_i end_POSTSUBSCRIPT .

We show the effect of growing $C_{\mathcal{F},k,i}$ in terms of cosine similarity score between embeddings.
Since $C_{\mathcal{F},k,i}=\frac{1}{4}\left(\sqrt{\gamma}+\sqrt{\frac{1}{\gamma}}
\right)^{2}$ with $\gamma=\frac{c_{\mathcal{F},k,i}^{\max}}{c_{\mathcal{F},k,i}^{\min}}\geq 1$, maximizing $C_{\mathcal{F},k,i}$ is equivalent to simultaneously maximizing $c_{\mathcal{F},k,i}^{\max}$ and minimizing $c_{\mathcal{F},k,i}^{\min}$.
For ease of the analysis, we assume that the encoders are reasonably well-trained.
Then, since a positive pair of embeddings normally yields higher similarity score, $c_{\mathcal{F},k,i}^{\max}$ should be obtained by choosing $l=i$ and $j=k$ in (11) as such choices make ${\boldsymbol{x}}_{l,k}^{\prime}$ be positive pair with ${\boldsymbol{x}}_{i,j}$.
Thus, $c_{\mathcal{F},k,i}^{\max}$ is roughly proportional to the similarity score of a positive pair of embeddings.
Conversely, $c_{\mathcal{F},k,i}^{\min}$ corresponds to the similarity scores of negative pairs, which tend to be low.
Hence, minimizing $-\sum_{k=1}^{|\mathcal{I}_{B}|}\log C_{\mathcal{F},k,i}$ enhances the similarity scores for positive pairs and reduces those for negative pairs, improving the overall multimodal alignment.

The preceding analyses demonstrate that CentroBind addresses the limitations $\bf{P1,P2,P3}$, and $\bf{P4}$ of FABind identified in Section 2.3. We now conclude the analysis with a discussion that CentroBind’s unified representation is likely closer to an ideal platonic representation (Huh et al., ) compared to FABind’s.
A platonic representation is defined as an ideal representation of reality that induces information in all modalities. From this perspective, a representation derived solely from a single modality, without leveraging others, may not be sufficient. In contrast to FABind’s approach, which relies exclusively on the fixed anchor modality, CentroBind constructs its unified representation space by incorporating information from all modalities. This suggests that CentroBind’s unified space is likely to retain a more comprehensive representation of all modalities. Thus, CentroBind emerges as a promising approach for developing a Planotic representation.

## 4 Experiment

Figure: (a) Comparison of the bind methods.
Refer to caption: extracted/5897266/figures/fig_1_comp-2.png

### 4.1 Experiments with synthetic dataset

#### Synthetic datasets.

We employ a latent variable model (Bishop & Nasrabadi, ) for generating synthetic multimodal datasets. A latent variable model is a statistical model for data ${\bf X}\in\mathbb{R}^{d_{x}}$, under which ${\bf X}$ is generated according to a conditional probability distribution $P_{{\bf X}|{\bf Z}}$, where ${\bf Z}\in\mathbb{R}^{d_{z}}$ is the latent variable. In terms of the representation learning framework, ${\bf Z}$ can be seen as the true representation of ${\bf X}$. We assume that the class label ${\bf Y}\in[K]$ and the latent variable ${\bf Z}$ are jointly distributed according to $P_{{\bf Z},{\bf Y}}$.
In our setting, we exploit Gaussian mixture model (GMM) (Bishop & Nasrabadi, ) for the latent variable ${\bf Z}$, and we generate $M$ modalities ${\bf X}_{i}=g_{i}({\bf Z})+{\bf N},i\in[M]$ with random noise ${\bf N}$ and some non-linear projections $g_{i}:\mathbb{R}^{d_{z}}\to\mathbb{R}^{d_{x}}$. We choose the projections in a way such that each model can be ranked in ascending order, i.e., ${\bf X}_{1}$ is the worst, and ${\bf X}_{4}$ is the best modality in terms of their inherent correlation with the latent variable. The class label ${\bf Y}$ is set to the component id of GMM (for details, see Appendix C.1).

#### Experiment results.

Figure 2 shows the classification accuracies with a synthetic dataset of $M=4$ modalities.
To obtain the results in Figure 2(a), we initialize pretrained backbones for all modalities, apply FABind (${\bf X}_{i}$-B) with anchor ${\bf X}_{i}$ or CentroBind (CB), and evaluate accuracy (acc(${\bf Z}_{i}$)) with embeddings from $i$-th modality. We provide acc(${\bf Z}_{i}$), without any binding, for a reference. Figure 2(a) verifies our analysis of FABind (Section 2.3) and CentroBind (Section 3.2): (1) the comparison between ${\bf X}_{1}$-B and ${\bf X}_{4}$-B shows the importance of choosing anchor modality; (2) the comparison between acc(${\bf Z}_{4}$) and ${\bf X}_{1}$-B: acc(${\bf Z}_{4}$) shows a performance deterioration by FABind , demonstrating that FABind does not capture intra information; (3) CB consistently outperform all FABind , indicating that CB successfully captures elements that FABind overlooks, including intra information and shared information among non-anchored modalities.

Figure 2(b) includes accuracies of FABind and CB with random backbones. Similarly, CB outperforms all baselines. Somewhat surprisingly, CB with random backbones (green curves) also performs better than FABind with pretrained backbones (red curves). This further supports our analysis that CentroBind is robust to the backbone quality as it optimizes intra and shared information, whereas FABind is sensitive to the backbone quality. Overall, these empirical results validate our findings.
We provide additional experimental results on synthetic datasets with $M=6,8$ in Appendix C.1. With the larger number of modalities, CB still outperforms the baselines, strengthening our analysis of CB.

### 4.2 Experiments with real-world dataset

In this section, we provide experiment results with a real-world dataset. We compare CentroBind and FABind anchored at text modality. We utilize the MUStARD dataset  (Castro et al., ) for its rich combination of multimodal data with more than two modalities.
It consists of 690 video clips (including audio) and text for sarcasm detection with labels such as sarcasm indicators and speaker names.
For the backbones in FABind and CentroBind , we use the pretrained VideoMAE model (Tong et al., ) for video data, the pretrained WaveLM model (Chen et al., ) for audio data, and the pretrained BERT model (Devlin et al., ) for text data.
A detailed description of the training setting is provided in Appendix C.2.

Table: Table 1: Zero-shot one-to-one and two-to-one retrieval accuracy. ($\mathcal{V}$: video, $\mathcal{A}$: audio, $\mathcal{T}$: text)

#### Downstream tasks.

We perform evaluations in zero-shot binary and multi-class classification tasks, One-to-One cross-modal retrieval, and Two-to-One cross-modal retrieval.
For classification tasks, we use a Multi-Layer Perceptron (MLP) to perform sarcasm detection as a binary classification and speaker classification with $23$ multi-class categories.
In particular, MLP is trained on embeddings in a single modality (denoted by Tr in Table 2) and accuracy is evaluated on another modality (denoted by Ev in Table 2).
In retrieval tasks, we measure the accuracy of correct retrieval. For One-to-One case, we retrieve data sample in different modality by choosing the closest embedding from a single input embedding, while for Two-to-One case we choose the closest embedding from the centroid of two input embeddings in two modalities.
We denote input and target modalities with $\rightarrow$ in Table 1.

#### Results on cross-modal retrieval.

Table 1 shows the performance for one-to-one and two-to-one retrieval tasks. CentroBind consistently excels in one-to-one retrieval for text and audio modalities, while FABind performs better for video retrieval. This might be due to a strong dependency between text and video, which may be suitable for FABind anchored at text modality.
A notable observation is that the centroid of video and audio embeddings achieves the best text retrieval performance including one-to-one retrieval. This implies complementary information exists and is captured by CentroBind.

Table: Table 2: Accuracy results for Sarcasms and Speakers. (Tr: training modality, Ev: evaluation modality, $\mathcal{V}$: video, $\mathcal{A}$: audio, $\mathcal{T}$: text). Methods with asterisks denote accuracy evaluated in different experiment settings.

#### Results on sarcasm & speaker classification.

Table 2 presents results for sarcasm detection and speaker classification tasks, where Sar-1 indicates Top-1 accuracy for sarcasm, and Spk-1, Spk-3, and Spk-5 represent Top-1, Top-3, and Top-5 accuracies for speaker classification.
In this experiment, CentroBind consistently outperforms FABind across all pairs of train and evaluation modalities, which can be distributed to CentroBind generally learning a better shared embedding space than FABind.

Although a direct comparison is not feasible, we present the sarcasm detection accuracy of FactorCL (Liang et al., ) and SimMMDG (Dong et al., ) approaches for reference. It is important to highlight that CentroBind and FABind are trained on a single modality (Tr) and evaluated on a different modality (Ev) in a zero-shot setting. In contrast, FactorCL and SimMMDG employ supervised contrastive learning, utilizing sarcasm labels during pretraining to capture task-specific features. At inference, all three modalities are simultaneously used to predict the class.

As analyzed in Section 2.3 and Section 3.2, these results highlight the CentroBind’s ability to preserve intra and shared information among modalities, which are useful in unknown downstream tasks.
Moreover, the zero-shot setting verifies the multimodal alignment of both CentroBind and FABind.

## 5 Conclusions

In this paper, we mathematically analyze the limitations of fixed-anchor-bind methods (FABind), including over-reliance on the choice of anchor modality, and failing to capture both intra-information and shared information among non-anchored modalities. To overcome the over-reliance on an anchor modality, we propose CentroBind, which aligns multimodal embeddings to dynamic anchors constructed by centroids of the embeddings, hence removing the need for anchor modality. Moreover, we theoretically study CentroBind, showing that it captures intra- and shared information.
Extensive experiments on both synthetic and real-world datasets show that CentroBind significantly outperforms FABind, providing a robust unified representation space and validating our analysis on CentroBind and FABind.

## Appendix A Related work

### A.1 Multimodal alignment

Multimodal learning addresses four key challenges (Liang et al., ; Baltrušaitis et al., ; Liang et al., ): managing interactions among redundant, unique, and synergistic features (Dumas et al., ; Liang et al., ; ), aligning fine-grained and coarse-grained information (Wang et al., ; ), reasoning across diverse features (Yang et al., ), and integrating external knowledge (Shen et al., ; Lyu et al., ). Among these challenges, multimodal alignment is one of the core challenges that many researchers aim to solve.

A common method in multimodal alignment is using cross-modal alignment by using attention mechanisms between pairwise modalities, such as vision-language (Tan & Bansal, ) and vision-language-audio (Tsai et al., ). Another effective approach is leveraging graph neural networks to align multimodal datasets (Yang et al., ; Wilf et al., ). For instance, Yang et al. () transforms unaligned multimodal sequence data into nodes, with edges capturing interactions across modalities over time. Wilf et al. () build graph structures for each modality—visual, textual, and acoustic—and create edges to represent their interactions.

To enhance the generalizability of cross-modal representations, Xia et al. () employ a unified codebook approach, facilitating a joint embedding space for visual and audio modalities. Another prominent method (Radford et al., ) achieves cross-modal alignment by leveraging large collections of image-text pairs, making it a widely adopted strategy in multimodal learning (Zhang et al., ; Guzhov et al., ; Zhou et al., ).

### A.2 Binding methods

Recent studies have focused on aligning multimodal datasets by leveraging binding properties in various modalities. ImageBind (Girdhar et al., ) aligns multimodal data by using image representation as the anchor and aligning each modality’s embedding with the image embedding. Similarly, LanguageBind (Zhu et al., ) uses language representation as the anchor, aligning other modalities into the language space. PointBind (Guo et al., ) learns a joint embedding space across 3d point, language, image, and audio modalities by designating the point space as the central representation. Thanks to the efficacy of such a binding idea with a fixed anchor, several “-Bind” approaches have been studied in numerous domains (Teng et al., ; Xiao et al., ; Gao et al., ; Yang et al., ; Balemans et al., ; Dhakal et al., ; Yang et al., )
While these methods demonstrate strong performance in zero-shot cross-modality retrieval and classification tasks, they are constrained by their reliance on an existing single anchor modality.

Several approaches have integrated additional knowledge into multimodal representation spaces to address this limitation. Freebind (Wang et al., ) introduces bi-modality spaces to enhance a pretrained image-paired unified space. It generates pseudo-embedding pairs across diverse modality pairs and aligns them with the pre-trained unified space using contrastive learning. Omnibind (Wang et al., ) leverages multiple pretrained multimodal models to construct pseudo item-pair retrievals based on top-1 recall across various modality combinations using pairwise cross-modal alignment. Both methods show promising results in cross-modal retrieval by incorporating extra spaces into existing pairwise binding spaces. However, they still rely on fixed (pre-trained) representation spaces.

Unibind (Lyu et al., ) highlights the imbalanced representation when using image-centered representation spaces. To address this, Unibind employs large language models (LLMs) to create a unified and balanced representation space. It constructs a knowledge base with multimodal category descriptions, establishes LLM-augmented class-wise embedding centers, and aligns other modalities to these centers through contrastive learning. This approach attempts to balance representations across modalities but still depends heavily on large-scale pretrained LLMs and centers alignment around a single unified space.

## Appendix B Proofs

### B.1 Proof of Proposition 1

Using the chain rule of the mutual information, we observe that

$$ $\displaystyle I({\bf X}_{1},f_{1}^{\rm suf}({\bf X}_{1});{\bf X}_{i})$ $\displaystyle=I({\bf X}_{1};{\bf X}_{i})+I(f_{1}^{\rm suf}({\bf X}_{1});{\bf X }_{i}|{\bf X}_{1})$ $\displaystyle=I(f_{1}^{\rm suf}({\bf X}_{1});{\bf X}_{i})+I({\bf X}_{1};{\bf X }_{i}|f_{1}^{\rm suf}({\bf X}_{1})),$ (12) $$

Since $f_{1}^{\rm suf}({\bf X}_{1})$ is a deterministic function of ${\bf X}_{1}$, we have

$$ $\displaystyle I(f_{1}^{\rm suf}({\bf X}_{1});{\bf X}_{i}|{\bf X}_{1})=0.$ (13) $$

Moreover, $f_{1}^{\rm suf}$ obtained in Definition 1 with proper choice of $\mathcal{Z}$ achieves the maximum mutual information, implying together with $I({\bf X};{\bf Y})\leq\min\{H({\bf X}),H({\bf Y})\}$ that $I(f_{1}^{\rm suf}({\bf X}_{1});{\bf X}_{1})=H({\bf X}_{1})$,
where $H({\bf X}_{1})$ is the entropy of ${\bf X}_{1}$ (Polyanskiy & Wu, ). In other words, we have $H({\bf X}_{1}|f_{1}^{\rm suf}({\bf X}_{1}))=H({\bf X}_{1})-I(f_{1}^{\rm suf}({
\bf X}_{1});{\bf X}_{1})=0$.
This gives

$$ $\displaystyle I({\bf X}_{1};{\bf X}_{i}|f_{1}^{\rm suf}({\bf X}_{1}))$ $\displaystyle=H({\bf X}_{1}|f_{1}^{\rm suf}({\bf X}_{1}))-H({\bf X}_{1}|f_{1}^ {\rm suf}({\bf X}_{1}),{\bf X}_{i})$ $\displaystyle=0$ (14) $$

Substituting (13) and (B.1) into (B.1) yields

$$ $\displaystyle I({\bf X}_{1};{\bf X}_{i})$ $\displaystyle=I(f_{1}^{\rm suf}({\bf X}_{1});{\bf X}_{i}).$ (15) $$

We conclude the proof of Proposition 1 by noting that the optimality of FABind (i.e., $I(f_{1}^{\rm suf}({\bf X}_{1});{\bf X}_{i})=I(f_{1}^{\rm suf}({\bf X}_{1});f_{
i}^{\rm FB}({\bf X}_{i})),~{}\forall i\in\{2,\cdots,M\}$) yields

$$ $\displaystyle I({\bf X}_{1};{\bf X}_{i})$ $\displaystyle=I(f_{1}^{\rm suf}({\bf X}_{1});f_{i}^{\rm FB}({\bf X}_{i})).$ (16) $$

### B.2 Proof of Proposition 2

Using the chain rule of mutual information, we have

$$ $\displaystyle I(f_{1}^{\rm ins}({\bf X}_{1});{\bf X}_{1},{\bf X}_{i})$ $\displaystyle=I(f_{1}^{\rm ins}({\bf X}_{1});{\bf X}_{1})+I(f_{1}^{\rm ins}({ \bf X}_{1});{\bf X}_{i}|{\bf X}_{1})$ $\displaystyle=I(f_{1}^{\rm ins}({\bf X}_{1});{\bf X}_{i})+I(f_{1}^{\rm ins}({ \bf X}_{1});{\bf X}_{1}|{\bf X}_{i}).$ (17) $$

Moreover, since $f_{1}^{\rm ins}({\bf X}_{1})$ is a deterministic function of ${\bf X}_{1}$, we have $I(f_{1}^{\rm ins}({\bf X}_{1});{\bf X}_{i}|{\bf X}_{1})=0$, leading to $I(f_{1}^{\rm ins}({\bf X}_{1});{\bf X}_{1})=I(f_{1}^{\rm ins}({\bf X}_{1});{
\bf X}_{i})+I(f_{1}^{\rm ins}({\bf X}_{1});{\bf X}_{1}|{\bf X}_{i})$.
Then, using the assumption $I(f_{1}^{\rm ins}({\bf X}_{1});{\bf X}_{1})<\epsilon$, it follows that

$$ $\displaystyle\epsilon$ $\displaystyle>I(f_{1}^{\rm ins}({\bf X}_{1});{\bf X}_{i})+I(f_{1}^{\rm ins}({ \bf X}_{1});{\bf X}_{1}|{\bf X}_{i})$ $\displaystyle\overset{\rm(a)}{\geq}I(f_{1}^{\rm ins}({\bf X}_{1});{\bf X}_{i})$ $\displaystyle\overset{\rm(b)}{\geq}I(f_{1}^{\rm ins}({\bf X}_{1});f_{i}^{\rm FB }({\bf X}_{i})),$ (18) $$

where the labeled inequalities follow from:
$\rm(a)$ the non-negativity of mutual information;
$\rm(b)$ the data processing inequality.
This concludes the proof of Proposition 2.

### B.3 Proof of Theorem 1

To prove Theorem 1, we leverage the reverse inequality of $M$-variable Hölder inequality (Seo, , eq. (2.8)). For the sake of completeness, we state the inequality in Lemma 1.

###### Lemma 1 (Reverse inequality of the $M$ -variable Hölder inequality (Seo, 2013 ) ) .

Consider $M$ sequences $(x_{i,j})_{j\in[n]},~{}i\in[M]$ of $n$ positive scalars such that for some $0<c_{m}\leq c_{M}<\infty$,

$$ $\displaystyle 0<c_{m}\leq x_{i,j}\leq c_{M}<\infty,~{}\forall i,j.$ (19) $$

Then,

$$ $\displaystyle\prod_{i=1}^{M}\left(\sum_{j=1}^{n}x_{i,j}\right)^{\frac{1}{n}}$ $\displaystyle\leq\frac{(c_{m}+c_{M})^{2}}{4c_{m}c_{M}}\sum_{j=1}^{n}\left( \prod_{i=1}^{M}x_{i,j}\right)^{\frac{1}{n}}.$ (20) $$

Now we start by writing the summation of InfoNCE losses for each $f_{l}^{(t)}({\boldsymbol{x}}_{l,k}^{\prime}),l\in[M]$ to $f_{i}({\bf X}_{i})$ as

$$ $\displaystyle\sum_{l=1}^{M}I_{\rm NCE}(f_{l}({\bf X}_{l}^{\prime});f_{i}({\bf X }_{i})|\tau)$ $\displaystyle=-\frac{1}{|\mathcal{I}_{B}|}\sum_{k=1}^{|\mathcal{I}_{B}|}\sum_{ l=1}^{M}\log\frac{\exp\left(\frac{f_{l}^{\top}({\boldsymbol{x}}_{l,k}^{\prime} )f_{i}({\boldsymbol{x}}_{i,k})}{\tau}\right)}{\sum_{j\in\mathcal{I}_{B}}\exp \left(\frac{f_{l}^{\top}({\boldsymbol{x}}_{l,k}^{\prime})f_{i}({\boldsymbol{x} }_{i,j})}{\tau}\right)}.$ (21) $$

Then, the inner summation in (21) is bounded as

$$ $\displaystyle\sum_{l=1}^{M}\log\frac{\exp\left(\frac{f_{l}^{\top}({\boldsymbol {x}}_{l,k}^{\prime})f_{i}({\boldsymbol{x}}_{i,k})}{\tau}\right)}{\sum_{j\in \mathcal{I}_{B}}\exp\left(\frac{f_{l}^{\top}({\boldsymbol{x}}_{l,k}^{\prime})f _{i}({\boldsymbol{x}}_{i,j})}{\tau}\right)}$ $\displaystyle=\frac{1}{\tau}\sum_{l=1}^{M}f_{l}^{\top}({\boldsymbol{x}}_{l,k}^ {\prime})f_{i}({\boldsymbol{x}}_{i,k})-\log\prod_{l=1}^{M}\sum_{j\in\mathcal{I }_{B}}\exp\left(\frac{f_{l}^{\top}({\boldsymbol{x}}_{l,k}^{\prime})f_{i}({ \boldsymbol{x}}_{i,j})}{\tau}\right)$ $\displaystyle\overset{\rm(a)}{\geq}\frac{1}{\tau}\sum_{l=1}^{M}f_{l}^{\top}({ \boldsymbol{x}}_{l,k}^{\prime})f_{i}({\boldsymbol{x}}_{i,k})-\log\left(C_{ \mathcal{F},k,i}\sum_{j\in\mathcal{I}_{B}}\prod_{l=1}^{M}\exp\left(\frac{f_{l} ^{\top}({\boldsymbol{x}}_{l,k}^{\prime})f_{i}({\boldsymbol{x}}_{i,j})}{\tau| \mathcal{I}_{B}|}\right)\right)^{|\mathcal{I}_{B}|}$ $\displaystyle\overset{\rm(b)}{=}\frac{M}{\tau}{\boldsymbol{a}}_{k}^{\top}f_{i} ({\boldsymbol{x}}_{i,k})-|\mathcal{I}_{B}|\log\sum_{j\in\mathcal{I}_{B}}\exp \left(\frac{M{\boldsymbol{a}}_{k}^{\top}f_{i}({\boldsymbol{x}}_{i,j})}{\tau| \mathcal{I}_{B}|}\right)-|\mathcal{I}_{B}|\log C_{\mathcal{F},k,i}$ $\displaystyle=|\mathcal{I}_{B}|\log\exp\left(\frac{M{\boldsymbol{a}}_{k}^{\top }f_{i}({\boldsymbol{x}}_{i,k})}{\tau|\mathcal{I}_{B}|}\right)-|\mathcal{I}_{B} |\log\sum_{j\in\mathcal{I}_{B}}\exp\left(\frac{M{\boldsymbol{a}}_{k}^{\top}f_{ i}({\boldsymbol{x}}_{i,j})}{\tau|\mathcal{I}_{B}|}\right)-|\mathcal{I}_{B}| \log C_{\mathcal{F},k,i}$ $\displaystyle=|\mathcal{I}_{B}|\log\frac{\exp\left(\frac{M{\boldsymbol{a}}_{k} ^{\top}f_{i}({\boldsymbol{x}}_{i,k})}{\tau|\mathcal{I}_{B}|}\right)}{\sum_{j \in\mathcal{I}_{B}}\exp\left(\frac{M{\boldsymbol{a}}_{k}^{\top}f_{i}({ \boldsymbol{x}}_{i,j})}{\tau|\mathcal{I}_{B}|}\right)}-|\mathcal{I}_{B}|\log C _{\mathcal{F},k,i},$ (22) $$

where the labeled (in)equalities follow from:
$\rm(a)$ Lemma 1 and $C_{\mathcal{F},k,i}=\frac{(c_{\mathcal{F},k,i}^{\min}+c_{\mathcal{F},k,i}^{
\max})^{2}}{4c_{\mathcal{F},k,i}^{\min}c_{\mathcal{F},k,i}^{\max}}$ with

$$ $\displaystyle c_{\mathcal{F},k,i}^{\min}=\min_{\ell\in[M],j\in\mathcal{I}_{B}} \exp\left(\frac{f_{l}^{\top}({\boldsymbol{x}}_{l,k}^{\prime})f_{i}({ \boldsymbol{x}}_{i,j})}{\tau}\right),~{}\text{ and }$ $\displaystyle c_{\mathcal{F},k,i}^{\max}=\max_{\ell\in[M],j\in\mathcal{I}_{B}} \exp\left(\frac{f_{l}^{\top}({\boldsymbol{x}}_{l,k}^{\prime})f_{i}({ \boldsymbol{x}}_{i,j})}{\tau}\right);$ (23) $$

and $\rm(b)$ the definition of anchor embedding (7).
Substituting (B.3) into (21) gives

$$ $\displaystyle\sum_{l=1}^{M}I_{\rm NCE}(f_{l}({\bf X}_{l}^{\prime});f_{i}({\bf X }_{i})|\tau)$ $\displaystyle\leq-\frac{1}{|\mathcal{I}_{B}|}\sum_{k=1}^{|\mathcal{I}_{B}|} \left[|\mathcal{I}_{B}|\log\frac{\exp\left(\frac{M{\boldsymbol{a}}_{k}^{\top}f _{i}({\boldsymbol{x}}_{i,k})}{\tau|\mathcal{I}_{B}|}\right)}{\sum_{j\in \mathcal{I}_{B}}\exp\left(\frac{M{\boldsymbol{a}}_{k}^{\top}f_{i}({\boldsymbol {x}}_{i,j})}{\tau|\mathcal{I}_{B}|}\right)}-|\mathcal{I}_{B}|\log C_{\mathcal{ F},k,i}\right]$ $\displaystyle=|\mathcal{I}_{B}|I_{\rm NCE}\left({\bf A};f_{i}({\bf X}_{i})\; \middle|\;\frac{\tau|\mathcal{I}_{B}|}{M}\right)+\sum_{k=1}^{|\mathcal{I}_{B}| }\log C_{\mathcal{F},k,i}.$ (24) $$

Rearranging (B.3) and setting $\tilde{\tau}=\frac{\tau|\mathcal{I}_{B}|}{M}$ in (B.3) and (B.3) yield

$$ $\displaystyle I_{\rm NCE}\left({\bf A};f_{i}({\bf X}_{i})\;\middle|\;\tilde{ \tau}\right)$ $\displaystyle\geq\frac{1}{|\mathcal{I}_{B}|}\sum_{l=1}^{M}I_{\rm NCE}\left(f_{ l}({\bf X}_{l}^{\prime});f_{i}({\bf X}_{i})\;\middle|\;\frac{\tilde{\tau}M}{| \mathcal{I}_{B}|}\right)-\frac{1}{|\mathcal{I}_{B}|}\sum_{k=1}^{|\mathcal{I}_{ B}|}\log C_{\mathcal{F},k,i},$ (25) $$

which concludes the proof of Theorem 1.

###### Lemma 1 (Reverse inequality of the M 𝑀 M italic_M -variable Hölder inequality (Seo, 2013 ) ) .

## Appendix C Experiment details

### C.1 Experiments with synthetic datasets

#### Synthetic datasets.

We employ a latent variable model (Bishop & Nasrabadi, ) for generating synthetic multimodal datasets. A latent variable model is a statistical model for data ${\bf X}\in\mathbb{R}^{d_{x}}$, under which ${\bf X}$ is generated according to a conditional probability distribution $P_{{\bf X}|{\bf Z}}$, where ${\bf Z}\in\mathbb{R}^{d_{z}}$ is the latent variable. In terms of the representation learning framework, ${\bf Z}$ can be seen as a true representation of ${\bf X}$. Moreover, we assume that the class label ${\bf Y}\in[K]$ and the latent variable ${\bf Z}$ are jointly distributed according to $P_{{\bf Z},{\bf Y}}$.

For the marginal distribution of ${\bf Z}$, we make use of a Gaussian mixture model (GMM) (Bishop & Nasrabadi, ), and hence the probability density function (PDF) of ${\bf Z}$ is a weighted sum of Gaussian densities. In particular, the PDF of ${\bf Z}$ is defined as follows:

$$ $\displaystyle p_{\bf Z}({\boldsymbol{z}})$ $\displaystyle=\prod_{y=1}^{K}\pi_{y}\mathcal{N}({\boldsymbol{z}};\hbox{ \boldmath$\mu$}_{y},\hbox{\boldmath$\Sigma$}_{y}),$ (26) $$

where $K$ is the number of mixture components, $\pi_{y}=\Pr({\bf Y}=y)$ is the component prior probability, and $\mathcal{N}({\boldsymbol{z}};\hbox{\boldmath$\mu$}_{y},\hbox{\boldmath$\Sigma$
}_{y})$ denotes Gaussian PDF with mean $\hbox{\boldmath$\mu$}_{y}\in\mathbb{R}^{d_{z}}$ and covariance matrix $\hbox{\boldmath$\Sigma$}_{y}\in\mathbb{R}^{d_{z}\times d_{z}}$. This leads to the conditional PDF of ${\bf Z}$ as $p_{{\bf Z}|{\bf Y}}({\boldsymbol{z}}|y)=\mathcal{N}({\boldsymbol{z}};\hbox{
\boldmath$\mu$}_{y},\hbox{\boldmath$\Sigma$}_{y})$.

Once a latent variable ${\boldsymbol{z}}$ is generated from GMM in (26), we generate data samples $({\boldsymbol{x}}_{i,1},{\boldsymbol{x}}_{i,2},\cdots,{\boldsymbol{x}}_{i,N})$ for $i$-th modality using the conditional PDFs of ${\bf X}_{i}$ given ${\boldsymbol{z}}$, denoted by $p_{{\bf X}_{i}|{\bf Z}}({\boldsymbol{x}}_{i}|{\boldsymbol{z}})$.
Specifically, we use the model ${\bf X}_{i}=g_{i}({\bf Z}_{i})+{\bf N}$, where $g_{i}:\mathbb{R}^{d_{z}}\to\mathbb{R}^{d_{x}}$ is a non-linear projection from latent space to observation space, and ${\bf N}\sim\mathcal{N}({\bf 0},I_{d_{x}})$ is Gaussian noise with zero-mean and identity covariance matrix. To make the inherent correlation between ${\bf X}_{i}$ and ${\bf Z}_{i}$ different among modalities, we choose $g_{i}$ such that

$$ $\displaystyle g_{i}({\bf Z})$ $\displaystyle=\Theta^{(2)}_{i}{\rm sigmoid}\left(\Theta^{(1)}_{i}{\bf Z}\right),$ (27) $$

where ${\rm sigmoid}(x)=\frac{1}{1+e^{-x}}$ is applied element-wise, and $\Theta_{i}^{(1)}\in\mathbb{R}^{d_{x}\times d_{z}}$ and $\Theta_{i}^{(2)}\in\mathbb{R}^{d_{x}\times d_{x}}$ are matrices randomly generated from Gaussian distribution. Moreover, after $\Theta_{i}^{(1)},i\in[M]$ are generated, we set arbitrary columns of them all zero, so that the number of all zero columns decreases in $i$. For example, $60\%$ of columns of $\Theta_{1}^{(1)}$ are all-zero, while only $10\%$ of columns of $\Theta_{M}^{(1)}$ are all-zero. This enables approximate control the correlation between ${\bf X}_{i}$ and ${\bf Z}$, providing estimates of best modality (${\bf X}_{M}$) or worst modality (${\bf X}_{1}$).
To have meaningful labels for this latent model, which requires for downstream tasks, we set the labels ${\bf Y}$ being the component index in GMM. In particular, since there are $K$ components in GMM (26), there exists $K$ categories in ${\bf Y}$.
We conduct experiments with three different synthetic datasets by setting $M=4,6,8$.
For all synthetic datasets, we fix $d_{x}=16$, $d_{z}=8$, and $K=50$.

#### Experiment details.

We initialize two different versions of backbones for all modalities, where the first is a random backbone (highlighted by (rnd) in figures), and the second is a backbone pretrained with InfoNCE loss. For each backbone, we use a simple multilayer perceptron (MLP).
Comparing the results with these two versions of backbone provides how much both FABind and CentroBind are robust to backbone quality.
Given the backbones for $M$ modalities, we align the corresponding embedding spaces using either FABind with anchor ${\bf X}_{i}$ (denoted by ${\bf X}_{i}$-B in figures) or CentroBind (denoted by CB in figures). Finally, with the encoders aligned by either FABind or CentroBind, we evaluate classification accuracy as a measure of representation quality. We use a simple MLP for the classifier. To distinguish between accuracy with embeddings from a single modality and the one with concatenated embeddings from all modalities, we denote by acc(${\bf Z}_{i}$) the accuracy with embeddings from $i$-th modality and by acc(All) the accuracy with embeddings from all modalities.

Additional experimental results on synthetic datasets with $M=6,8$ number of modalities are shown in Figure 3 and Figure 4.

Figure: (a) FABind at ${\bf X}_{6}$ vs. CentroBind
Refer to caption: extracted/5897266/figures/IB6vsCB.png

Figure: (a) FABind at ${\bf X}_{6}$ vs. CentroBind
Refer to caption: extracted/5897266/figures/IB8vsCB.png

### C.2 Experiments with real-world datasets

#### Training details.

We utilize Low-Rank Adaptation (Hu et al., ) for training CentroBind and FABind , enhancing training efficiency and achieving impressive results with fewer iterations.
For parameter settings, we set a learning rate of $0.001$, the AdamW optimizer (Loshchilov & Hutter, ) with a batch size of $16$, and a temperature of $0.3$ for InfoNCE. Training CentroBind requires augmentation.
We augment video frames with various transformations, including random perspective shifts, random flips and rotation, color jitter, Gaussian blur, and auto-contrast adjustment.
For the audio modality, we apply a low-pass filter, speed changes, echo effect, room impulse response convolution, and background noise.
For the text modality, we generate paraphrased sentences using the Phi-3 language model served using Ollama (^4^44https://ollama.com/library/phi3).

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See Weaknesses
soundness: 3
strengths: 1、Innovative Approach: The introduction of CentroBind addresses key limitations in existing multi-modal learning methods by using dynamic anchors, offering a fresh perspective and improving multi-modal alignment.


2、Theoretical Rigor: The paper provides a comprehensive mathematical analysis to support the efficacy of CentroBind, ensuring a solid foundation for the proposed approach.
summary: The paper explores limitations in current Fixed-Anchor-Bind (FABIND) methods used for multi-modal learning, which commonly align various modalities using a fixed anchor like images or text.  These methods struggle to capture essential intra-modal and shared information among non-anchored modalities.  To address these shortcomings, the authors propose a novel approach, CentroBind, which dynamically computes centroid-based anchors from all modalities, enhancing representation alignment.  Their experimental results demonstrate that CentroBind outperforms traditional FABIND methods on synthetic and real-world datasets.
weaknesses: 1、Robustness of anchor generation: CentroBind takes the centroids of all modalities to generate anchors, how does this approach perform in the face of modalities with significantly different information densities? For example, if some modalities contain more or less useful information than others, does the centroid bias towards these modalities, resulting in an imbalance in the representation space?

2、Effect of different modal weights: All modes are treated as equally weighted when calculating the centroid anchor. Have methods been investigated for assigning different weights to each modality to better capture inter-modality differences? Does this weight assignment improve or optimize the overall performance of the model?

3、CentroBind aims to maximize both intra-and inter-modality information, but is there an information bottleneck effect (e.g. information may be compressed in high-dimensional embedding Spaces)? How does this bottleneck effect affect the representation power of the model, especially in tasks where multimodal interactions are very complex? Have experiments been conducted to verify the balance between information maximization and representation compactness?

4、Does CentroBind suffer from potential optimization pitfalls when multiple modalities are strongly correlated or there is obvious synergy information between modalities? For example, when two modes are highly dependent, is the centroid overly pulled towards a particular mode? In this case, how can the optimization process be adjusted or improved to capture real multimodal relationships?

5、Boundary conditions for theoretical analysis: The theoretical analysis in the paper explores the advantages of CentroBind, but are there certain boundary conditions or assumptions under which CentroBind might lose its advantage? For example, does this approach still work when the data is unevenly distributed or the amount of data is severely imbalanced?

6、Loss function Optimization: CentroBind's loss function combines multiple InfoNCE loss terms, but does this design affect the convergence speed or stability of the model? Has the effect of different temperature parameters or weight of the loss term been explored to optimize the convergence of the model?

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: 1) Could the authors better explain how they dynamically adjust the centroids? I mean, after an iteration, or step by step also inside the iteration?
2) How would this model scale up to multiple (more than three) modalities?
soundness: 4
strengths: 1) The paper faces the problem of misalignment among non-anchor modalities, which is a real problem in multimodal alignment.
2) The mathematical solutions and proofs to the problem, as well as the mutual information perspective, make the paper valuable.
3) Although simple, the experiments confirm the theoretical intuitions.
4) The paper is well-written and well-presented.
summary: This paper proposes a novel method to eliminate the need for an anchor in multimodal representation. Such method is based on centroid anchors computed from the modalities. The method faces a real problem in the largest part of current multimodal models that mainly rely on text anchors. The proposed method is novel and of interest to the community.
weaknesses: 1) A tSNE visualization of the learned space could be strengthen the claims.
2) Maybe more detailed experiments with existing embedding models could be conducted to generalize the results. But the paper is fine also as it is now. :)


Minor comments:

1) Supplementary materials should have been submitted together with the paper.
2) The Saxon genitive should be avoided in scientific writing, although I know that both ChatGPT and Grammarly suggest to use it. However, I suggest the authors to remove all the Saxon genitives from the paper.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to the Weaknesses section.
soundness: 2
strengths: The paper addresses challenges with Fixed-Anchor Binding (FABIND) and introduces a centroid-based anchor approach as an effective alternative. This method, Centro BIND, demonstrates superior performance over FABIND across both synthetic and real-world datasets in retrieval and classification tasks, which is likely to capture the interest of the community. Additionally, the paper is concise, well-written, and easy to follow.
summary: The paper removes the need for selecting a fixed anchor modality in Fixed-Anchor-Bind (FABIND) and uses the centroid of all modality representations as an anchor representation. The paper study their method theoretically, showing that it captures intra- and inter information and shows empirical improvements over various synthetic and real-world datasets.
weaknesses: The paper addresses challenges with Fixed-Anchor Binding (FABIND) and introduces a centroid-based anchor approach as an effective alternative. This method, Centro BIND, demonstrates superior performance over FABIND across both synthetic and real-world datasets in retrieval and classification tasks, which is likely to capture the interest of the community. Additionally, the paper is concise, well-written, and easy to follow.

* **Limited Baseline Comparison**: While the limitations of fixed anchors are well-articulated, additional baselines would help to further justify the centroid-based approach. For example, sampling a modality as an anchor randomly or choosing an anchor based on the minimum or maximum similarity score could provide useful comparisons to strengthen the motivation for centroid-based anchoring.

* **Positioning Among Related Works**: Although the paper’s primary focus is to demonstrate improvement over FABIND, it would benefit from a broader positioning among recent multimodal works [1, 2, 3]. Several studies have explored inter- and intra-modality information; a comparative discussion here would contextualize the contributions more effectively. Including a simple baseline of early or late fusion across modalities for both synthetic and real-world datasets would be insightful. Additionally, it’s unclear why Centro BIND does not utilize all modalities in Table 2 for a more comprehensive comparison.

* **Comparison with Individual Modalities**: Since the work emphasizes intra-modality information, a comparison with individual modality baselines and an ensemble of modalities, especially in Table 2 for evaluation, is essential. This would underscore how effectively Centro BIND leverages intra-modality information compared to single-modality models.

* **Clarification on FABIND Setup**: Could the authors clarify if FABIND uses pre-trained encoders in the experiments, or are the encoders randomly initialized? This detail would help to understand the experimental setup more clearly.

[1] Du et al., On Uni-Modal Feature Learning in Supervised Multi-Modal Learning.  
[2] Madaan et al., A Framework for Multi-modal Learning: Jointly Modeling Inter- & Intra-Modality Dependencies.  
[3] Zhang et al., Multimodal Representation Learning by Alternating Unimodal Adaptation

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Refer to the summary
soundness: 3
strengths: Refer to the summary
summary: **Summary of this paper:**

This work introduces a new task: CentroBind, a novel approach in multimodal learning that addresses limitations found in fixed-anchor binding methods(FABIND). FABIND typically uses a single anchor modality to align representations across various modalities. However, this method has critical drawbacks, such as dependence on the anchor choice and lack of intra- and inter-modal information sharing. CentroBind proposes to replace fixed anchors with dynamic centroid-based anchors derived from all available modalities, creating a more balanced representation space. Theoretical analysis and experiments on synthetic and real-world datasets demonstrate that CentroBind enhances multimodal alignment and performs better in tasks requiring nuanced multimodal interactions.

**Strengths:** 

The paper is well-organized and written. The paper proposes an innovative solution to the limitations inherent in fixed-anchor binding methods by introducing CentroBind, which leverages centroid-based dynamic anchors. CentroBind's design is rooted in solid theoretical analysis, addressing crucial aspects of multimodal learning, including intra-modal, inter-modal, and multimodal alignment. The paper conducted extensive experiments on both synthetic and real-world datasets, and the results confirmed that CentroBind outperformed FABIND. The research content is innovative and holds potential application value.

**Weakness:**

As for method. The theoretical advantages of dynamic centroid-based anchors are well-articulated, the practical implementation details are somewhat limited.As for the experimental results. (1) I believe that the comparative experiments and ablation studies are not sufficiently. The paper only compares CentroBind and FABIND methods, lacking additional comparisons with other recent multi-modal alignment frameworks, which leads me to question the validity of this comparison. (2) There could be more exploration into how CentroBind performs under different data distributions and noise levels, which could provide further insights into its robustness.

**Weakness Summary.** 

CentroBind's scalability and practicality for large datasets require further consideration. In addition, broadening experimental comparisons and analysis would help address potential limitations and position the method within the broader multimodal research.
weaknesses: Refer to the summary

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Line 220: The authors may clarify why equation (4) shows that FABind does not guarantee shared information fbetwen non-anchor modalities.
2. Are there any constraints on the batch B from which the anchor embeddings are derived?
3. It is not clear which part of the analysis specifically shows that using a fixed anchor affects the representativeness of all modalities (P4).
4. How would acc(All) perform in Fig 2 (a)?
soundness: 3
strengths: 1. A strong theoretical paper that shows that fixing an image or text as an anchor for multimodal representation loses information contained in other modalities that are not selected as anchors.
2. The proposed strategy of using dynamic anchors is also supported theoretically by deriving a lower bound on CentroBind's objective function and minimizing the objective function.
3. Very good interpretations based on theoretical analysis of why FABind is insufficient and how CentroBind overcomes the former's limitations.
summary: The paper addressed the issues related to fixing a particular modality as an anchor in multi-modal representations. They argue that the anchor should not be fixed since the semantic relationships among the non-anchors may be lost and also since intra-modality information  may not be captured if the anchor is fixed. Instead, they propose choosing centroids of positive augmentation pairs among multiple modalities. The paper presents theoretical analyses demonstrating the drawbacks of fixed anchors and the advantages of dynamic anchors. Experimental results show the superiority of the proposed strategy on a synthetic dataset and a multimodal dataset for cross-modal retrieval and, sarcasm and speaker classification.
weaknesses: There are no major weaknesses to speak of. 
1. To me, the claim of a closer representation to Huh et al's "Platonic representation" is not  quite evident. I tend to believe the authors are also not very positive about this aspect from their statement in line 361 that the representation is "likely closer" to the Platonic representation.
2. This is not a weakness of the paper per se. Audio, video and text are the usual modalities that are discussed in multimodal representation. It would be interesting to study datasets with additional modalities such as VIDIMU: Multimodal video and IMU kinematic dataset on daily life activities using affordable devices (https://zenodo.org/record/8210563), which has inertial sensors.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See Weaknesses.
soundness: 2
strengths: 1. The paper is easy-to-follow.
2. The paper provides sufficient analysis of the proposed method.
summary: This paper proposes CentroBind, which employs a dynamically adjustable centroid-based anchors generated from all available modalities. The experiments are conducted on synthetic and real-world datasets.
weaknesses: 1. CentroBind computes the average of the inputs in the batch as the anchors. However, means are sensitive to outliers. When there are outliers in a batch, the performance will be influenced. Particularly, when the batch size is small, it will also influence the performance.
2. The experiments are not sufficient. There are many unified multimodal representation methods. Why do the authors only compare FABind? More recent baselines should be included.
3. I consider this paper to be incremental work, as it makes only very minor changes to FABind.
4. There are no visualizations in the experiments. Features before and after CentroBind should be visualized to better demonstrate its effectiveness.

Meta-review: This is a borderline paper, but the overall idea of this paper could meet the high quality of an ICLR Paper. This paper experienced a long discussion phase in both reviewer-author and reviewer-AC. The reviewers have clear diverse opinions about this paper, and the rating scores are 5, 8, 3, 6, and 8. Here is a summary of the discussion:  
The key characteristic of this paper is its simplicity, and we all know that a simple but effective algorithm is our ideal idea. However, the primary concerns can be summarized as follows.

***Motivation Issue***  
This paper primarily investigates learnable anchors for multimodal learning, in contrast to the claimed use of fixed anchors. The main comparisons are drawn against fixed-anchor methods, particularly ImageBind [1]. The stated limitations of ImageBind and similar approaches are outlined as follows:  
P1: over-reliance on a single anchor modality;  
P2: failure to capture intra information;  
P3: absence of shared information among non-anchored modalities;  

However, in my view, these motivations require further careful examination.  
For P1, ImageBind utilizes images as anchors to generate or associate with other modalities. This approach is primarily based on the rich and complex features embedded in the visual space, which facilitate the inference of other modalities.  
For P2, ImageBind incorporates intra-modal information in its modality alignment process.  
For P3, the shared information is also utilized in modality alignment, which is a core research challenge in multimodal learning, rather than being absent. Furthermore, ImageBind offers greater flexibility in multimodal learning by enabling seamless transitions from one modality to others, followed by their fusion.  

[1] Girdhar, Imagebind: One embedding space to bind them all. In CVPR 2023.  

***Technical Issue***  
The proposed method is straightforward yet potentially effective for large-scale inference tasks. ImageBind aligns with the principle: "One begets Two; Two begets Three; Three begets all things."  
In contrast, the authors adopt a multi-view contrastive learning approach based on instance-level representations, focusing on the mean features of each instance. However, a deeper investigation into multi-view or multimodal learning using contrastive learning [2] and its subsequent studies [3-6] reveals numerous related works that appear to have been overlooked by the authors.  

Furthermore, the authors claim that intra-modal and shared information are jointly considered using the mean features of each instance. However, in my view, ImageBind is also capable of addressing this scenario for multimodal alignment and fusion within a joint space.  

[2] Tian et al, Contrastive Multiview Coding, ECCV 2020.  
[3] Xing et al. Adaptive cross-modal few-shot learning, NeurIPS 2019.  
[4] Zeng, et al. Pan: Prototype-based adaptive network for robust cross-modal retrieval.SIGIR 2021.  
[5] Huang, et al., Clover: Towards a unified video-language alignment and fusion model, CVPR 2023.  
[6] Wang, et al. Vlmixer: Unpaired vision-language pre-training via cross-modal cutmix, ICML 2022.  

Experiments Issue For ImageBind, this work is based on scaling laws, and sufficient data must be collected to validate its effectiveness across multiple tasks.  

As another reviewer has pointed out, the authors conduct their experiments on a synthetic dataset and a claimed real-world dataset (MUStARD), which raises concerns about fairness. At present, we are unable to fully evaluate the performance comparisons between the proposed algorithm and the open-source ImageBind across the six modalities—images, text, audio, depth, thermal, and IMU data.

***Theory Issue***  
Proposition 1 states FABIND with sufficient anchors. The sufficiency is attributed to large-scale data, an InfoNCE-based encoder, and a transformer-based architecture, as also stated in the main paper. The use of information is clearly articulated, which is no need to use a Proposition to clarify.

Proposition 2 is ‘FABIND with insufficient anchor’. Yes, it is known that information bottleneck theory is always used to define or prove the representation bound Hence, all representation models would exhibit this behavior, as it is a commonly used theory to assess the capabilities of deep learning models.

For Theorem 1, while I have not thoroughly reviewed all the details, it appears to be based on the information bottleneck and bound theory derivation.

In summary, while the overall idea of this paper is straightforward, it is not fully convincing from both experimental and theoretical perspectives. Therefore, this paper will be rejected.

Decision: Reject