Understanding the Emergence of Multimodal Representation Alignment
Authors: Megan Tjandrasuwita
Chanakya Ekbote
Liu Ziyin
Paul Pu Liang
Venue: ICML 2025 poster (2025)
Keywords: representation alignment
multimodal learning
vision-language models
Primary area: deep_learning->other_representation_learning
TL;DR: We study the emergence of implicit alignment between modalities and find that its impact on performance depend on factors such as modality similarity and information balance, suggesting that alignment is not always beneficial for optimal performance.

Abstract: Multimodal representation learning is fundamentally about transforming incomparable modalities into comparable representations. While prior research has primarily focused on *explicitly* aligning these representations through targeted learning objectives and model architectures, a recent line of work has found that independently trained unimodal models of increasing scale and performance can become *implicitly* aligned with each other. These findings raise fundamental questions regarding the emergence of aligned representations in multimodal learning. Specifically: (1) when and why does alignment emerge implicitly? and (2) is alignment a reliable indicator of performance? Through a comprehensive empirical investigation, we demonstrate that both the emergence of alignment and its relationship with task performance depend on several critical data characteristics. These include, but are not necessarily limited to, the degree of similarity between the modalities and the balance between redundant and unique information they provide for the task. Our findings suggest that alignment may not be universally beneficial; rather, its impact on performance varies depending on the dataset and task. These insights can help practitioners determine whether increasing alignment between modalities is advantageous or, in some cases, detrimental to achieving optimal performance.

Understanding the Emergence of Multimodal Representation Alignment
Megan Tjandrasuwita'
Chanakya Ekbote'
Liu Ziyin
Paul Pu Liang'
Abstract
Multimodal representation learning fundamen- tally about transforming incomparable modalities into comparable representations. While prior re- search primarily focused explicitly aligning these representations through targeted learning objectives and model architectures, recent line of work has found that independently trained unimodal models of increasing scale and perfor- mance can become implicitly aligned with each other. These findings raise fundamental questions regarding the emergence of aligned representa- tions in multimodal learning. Specifically: (I) when and why does alignment emerge implic- itly? and (2) is alignment a reliable indicator of performance? Through a comprehensive em- pirical investigation, we demonstrate that both the emergence of alignment and its relationship with task performance depend several criti- cal data characteristics. These include, but are not necessarily limited to, the degree of similar- ity between the modalities and the balance be- tween redundant and unique information they pro- vide for the task. Our findings suggest that align- ment may not be universally beneficial; rather, its impact on performance varies depending on the dataset and task. These insights can help practi- tioners determine whether increasing alignment between modalities is advantageous or, in some cases, detrimental to achieving optimal perfor- mance. Code is released at: https:://// /ttp::////ttttta/ com/ Bom/MeganTji. mmodamaaal..mmmmmmmmmmmmmmmmll a miggtent.....
Performance
Platonic Represinnattion Hypöshesis pessess
Increasing representation alignment
Figure 1: Emergence of multimodal alignment? Trian- gles and circles correspond different modalities. While the Platonic Representation Hypothesis (Huh et al., 2024) argues that better cross- cross-modal alignment predicts better per- formance, our findings demonstrate that the relation between alignment and performance is more nuanced and depends on several dataset characteristics including the degree of heterogeneity and interactions between modalities.
AI systems are revolutionizing predictive analytics across many applications, including in multimedia (Alayrac et al., 2022; Sun al., 2019; Ramesh al., 2021; Singer et al., 2022), healthcare (Cai et al., 2019; Muhammad et al., 2021), and physical sensing (Kirchner et al., 2019; Lee et al., 2019; Xiao et al., 2020). A large body of research in designing and training multimodal models has focused on aligning the representations from different modalities such that they are comparable some semantic representation space (Bal- trušaitis al., 2018; Liang al., 2024). Conventional wisdom posits that aligned representations cursor to multimodal fusion and representation learning (Li et al., 2021). As a result, many learning methods, such as contrastive learning and its variants (Frome et al., 2013; Jia et al., 2021; Radford et al., 2021a), and model architec- tures (Bertinetto al., 2016; Lenc Vedaldi, 2019; Bansal et al., 2021; Csiszárik al., 2021) have been proposed explicitly align incomparable modalities into comparable representation spaces for further processing.
crucial pre-
1. Introduction
Multimodal AI represents a cutting-edge paradigm in ma- chine learning that enables integrating and learning from many heterogeneous and interacting data modalities. These
Massaahussets Institute of Technology, USA 2NTT Re- search, USA. Correspondence to: Megan Tjandrasuwita <megantj @ii..edu>.
However, recent work on the "Platonic Representation Hypothesis" showed that, surprisingly, alignment could even emerge across independently pre-trained vision and language models without explicitly aligning them to-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Understanding the Emergence of Multimodal Representation Alignment
gether (Huh al., 2024). Crucially, alignment increases with model size and performance, and it has been hy- pothesized that unimodal models will become increasingly aligned. These findings raise fundamental questions re- garding the emergence of aligned representations and their implications on multimodal learning: (1) when and why does alignment emerge implicitly, and (2) is alignment a reliable indicator of performance? We illustrate these open questions in Figure 1.
Unique
X
?
?
X2
X
X2
Redundant
Platonic
?
X
representation
hypothesis
X2
In this paper, we study these questions comprehensively across two principal dimensions that taxonomize multi- modal data: interactions and heterogeneity (Baltrušaitis et al., 2018; Liang et al., 2024; Tian et al., 2020), visualized in Figure? 2. Interactions measure the information shared between two modalities for a task, from more redundant (e.g., images and corresponding captions) more unique (e.g., sensor placement). We expect alignment to emerge more easily between redundant modalities. Heterogeneity measures the degree of similarity across two modalities inde- pendent of the task, from more similar (c.g.. two languages) to more different (e.g.. text and video). We expect alignment to emerge more easily between similar modalities.
X1
X2
X1
X2
X1
X2
Homogeneous
Heterogeneous
Figure 2: Two principal dimensions of multimodal data. This study empirically evaluates data across two key di- mensions: heterogeneity and interactions. Heterogeneity, represented on the x-axis, reflects the similarity between two data modalities, X1 and X2. regardless of the task. Interac- tions, on the y-axis, indicate the balance between redundant and unique information across modalities that is relevant task Y. We expect the Platonic Representation Hypothesis to hold in cases of redundancy and similar modalities, but when and why alignment emerges implicitly, and whether alignment is a reliable indicator of performance, remain open questions.
Through extensive experiments on controlled and real-world datasets with varying degrees of interactions and heterogene- ity. we discover several key insights. First, the maximum alignment achievable depends on the degree of heterogene- ity and uniqueness in the modalities, which inherently limits alignment. Second, while alignment correlates with per- formance in datasets with high redundancy, this relation- ship breaks down when uniqueness dominates redundancy. These findings highlight that performance often does not di- rectly correspond to alignment, and the connection between them is a nuanced property of the data that varies across modalities and tasks. Therefore, our work provides impor- tant considerations for practitioners designing and training multimodal models, emphasizing that scale alone does not guarantee modality alignment and that careful assessment is necessary to determine when alignment is beneficial.
Golub Zha, 1995). along with its nonlinear extensions using kernels (Lai & Fyfe, 2000; Raghu et al., 2017) and neural networks (Andrew et al., 2013; Wang et al., 2015; Morcos et al., 2018). Among these methods, Kornblith et al. (2019) highlight the advantages of Centered Kernel Alignment (CKA), particularly its invariance to orthogonal transformations and isotropic scaling. Given mean-centered feature sets of samples, Z1, Ze Raxd from two modal- ities X, and X2, the CKA metric with linear kernel is:
ALIGN Z2)
CKA(Z1, Zz)
2. Representation Alignment
AALIG ALIGN Zz, Zz)
In this section, we review the concept of representation align- ment through prior work on measuring alignment, methods explicitly align representations, and observations regard- ing the emergence of alignment.
where ALIGN(Z,, Z2) denotes HSIC(Z,Z,, Z,Z,) with HSIC denoting an empirical estimator of the Hilbert- Schmidt Independence Criterion (Gretton et al., 2005). In- tuitively, CKA quantifies alignment by comparing the co- variance structures of two feature sets, capturing whether their representations encode similar relationships. This prop- erty ensures that even if Z₁ and Zz undergo arbitrary ro- tations (e.g., due to different initialization schemes), the CKA metric remains consistent, making it robust choice for assessing representation alignment. For large vision- language models, given the high dimensionality and large embedding sizes of learned representations, we employ a
Measuring Alignment. Measuring alignment between neu- ral network representations is a widely used approach in the research community to analyze and improve training dynamics (Huh et al., 2024; Klabunde al., 2024; Korn- blith et al., 2019). A prominent class of alignment metrics is based on canonical correlation analysis (CCA), statistical technique for comparing two subspaces (Thompson, 2005;
2

---

Understanding the Emergencc of Multimodal Representation Alignment
computationally efficient variation of CKA the mutual k- nearest neighbors (mutual KNN) method (Huh et al., 2024). Instead of directly comparing full covariance structures, this approach measures similarity by analyzing the overlap be- tween the k-nearest neighbor sets of embeddings, improving scalability. Details are provided in Appendix B.
does not always lead to improved performance. This dis- crepancy between the possibility of emergent alignment and the need for explicit alignment methods calls for system- atic exploration of the role of alignment and its downstream relationship to performance in multimodal learning.
3. Research Questions and Experimental Setup
Explicit Alignment. In addition to research on measur- ing alignment in neural networks, another line of work fo- cuses on explicitly aligning representations, a widely used technique for handling heterogeneous modalities (Liang et al., 2024). A popular approach in this domain is mul- timodal contrastive learning, where representations of the same concept across different modalities (i.e.. positive pairs) are brought closer together, while representations of differ- ent concepts (i.e., negative pairs) are pushed apart (Frome et al., 2013; Jia et al., 2021; Radford et al., 2021a). The coordination distance in contrastive learning is typically measured using cosine distance (Mekhaldi, 2007) or max- margin losses (Hu et al., 2019). Theoretical results demon- strate that contrastive learning effectively captures redun- dant information shared between modalities (Tian et al., 2020; Tosh et al., 2021). More recent extensions have been proposed to also capture unique and synergistic information, further refining multimodal representation learning (Dufu- mier et al., 2024; Liang et al., 2023b).
The recent line of work on the emergence of alignment across independently pre-trained unimodal models raises fundamental questions regarding the emergence of aligned representations and their implications on multimodal Iearn- ing. Our research seeks to understand (1) when and why alignment emerges implicitly, and (2) whether alignment a reliable indicator of performance. To reliably and com- prehensively study these questions across all types of mul- timodal data, we use two principle dimensions to taxono- mize multimodal data: interactions and heterogeneity (Bal- trušaitis et al., 2018; Liang et al., 2024; Tian et al., 2020). Interactions measure the task-relevant information shared between two modalities. We expect alignment to emerge more easily between modalities where information content is redundant. Heterogeneity measures the degree of simi- larity across two data modalities independent of the task, from more similar (e.g., two languages) to more differrnt (e.g., text and video). We expect alignment to emerge more easily between similar modalities. Our experiments aim to study the emergence of alignment and its relationship to downstream task performance by systematically varying the interactions and heterogeneity in multimodal data. To summarize, our fundamental guiding questions are:
Emergence of Implicit Alignment. In contrast to explicit alignment methods, recent findings suggest that alignment can emerge implicitly, even when neural networks differ in training objectives, datasets, and architectures (Li et al., 2015; Raghu et al., 2017; Lenc & Vedaldi, 2019; Baran- nikov et al., 2022; Bonheme & Grzes, 2022). Notably, this similarity becomes more pronounced in larger and wider networks (Raghu et al., 2017; Morcos et al., 2018; Kornblith et al., 2019). Building on the observation that latent spaces are inherently comparable, line of research explores com- posing components of different models with minimal or no additional training. Lene& & Vedaldi (2019) demonstrate that latent spaces can be stitched together using trainable stitch- ing layers, while subsequent studies (Bansal et al., 2021; Csiszárík et al., 2021) show that better-performing mod- els tend to learn more similar representations when stitched. More recently, the Platonic Representation Hypothesis (Huh et al., 2024) suggests that as vision and language models scale in capacity and performance, independently trained models exhibit increasing alignment. This finding implies that models are converging toward modality-agnostte repre- sentations, reinforcing the idea that alignment may emerge naturally as a byproduct of model scaling. However, if align- ment continues emerge, there would be need for any of the explicit alignment methods described above. That ex- plicit alignment has consistently been helpful implies either emergent alignment is not sufficient or emergent alignment
1. Does alignment emerge when uniqueness and hetero- geneity increase?
2. Does higher alignment always predict better performance when uniqueness is present?
3. How can we characterize datasets through the correlation between performance and alignment?
Based on these questions, we define our problem setting.
3.1. Problem Setting
We focus on a simplified setting with two modalities and an associated label, the generalization is straightforward. Con- cretely, we consider a scenario where we sample multimodal data and labels æ1, 22, y ~ P(X1, Y) from a data dis- tribution P(X,, X2, Y). X; represents the random variable for the i-th modality and Y for the task. Based on the re- lationships between X1, X2, and Y. these modalities can exhibit different degrees of interactions and heterogeneity.
Interactions measure the information shared between two modalities for task, from more redundant to more unique. Redundancy R represents the shared information between the two modalities and the task (Y), such as between images
3

---

Understanding the Emergencc of Multimodal Representation Alignment
and captions that describe the image (Radford al., 202 Uniqueness in modality (Ui) quantifies the amount of information present in the first modality absent in the sec- ond but critical for the downstream task (and likewise for U2). For example, feature selection is often optimized to provide new unique information and minimize redundancy to previous ones (Peng et al., 2005).
Data Generation
Training
Ej
Feature Generation
Hatooogen
Ex
U2
To investigate how alignment and performance change with respect to different interactions, we need synthetic control- lable datasets and real-world multimodal benchmarks with different interactions. For constructing synthetic data, we assume that the task-relevant information (for a particular label y) can be decomposed into Zr; Iuu+ Tuzs where I, de- notes the common or redundant information, Fu, represents information unique to the first modality, and Fuz captures information unique to the second modality.
Label
dv
Generation
Figure 3: Synthetic data generation and training. We generate synthetic data with varying levels of uniqueness and heterogeneity, The building blocks are the redundant and unique components Tu,
where [ar, Zu, are used in creating X1 and [ are for X2- The level of uniqueness is determined by the number of features from Er that are used to compute the labels Y. given that the total number of features used for label computation is held constant. X₂ transformed into a heterogeneous modality using transformation network Φ. In our experiments, we compute alignment between unimodal encoders E1, E2 trained on X1, X2 respectively.
We construct the input data as [2r, TugJ. An overview of the data generation process is shown in Figure 3. By selecting specific features to compute the label, we control the levels of redundancy and unique- ness. Specifically, Y is a nonlinear function of a subset of features, S S [ Fuz.] This enables as the number of features in S that come from Fy., and U, as the number of features that come from Tu.. for e (1, We denote the total uniqueness U = | R. By keeping [S] fixed while varying U, we generate datasets with different proportions of redundant versus non -on-rdaundant information.
[u,] and
control R
how uniqueness, redundancy, and heterogeneity influence the emergence of alignment by training encoders indepen- dently cach modality and measuring the aligmment be- tween their learned representations. Specifically, we train single-layer encoder on the first modality, denoted as E We experiment with higher depths of F3 in Appendix D.2 and find that the results are not significantly changed. For the second modality, which is transformed by the nonlinear function e(.) with varying depths (D&)). train series of encoders denoted as E2,D where DEne represents the depth of the encoder trained on the second modality and varies as DEne E (1)
Heterogeneity. Different modalities often exhibit distinct structures, qualities, and representations (Liang et al., 2024). For example, when one modality is time series and another is a static image, differences in their vocabulary tokens, and different noise or distribution shifts in each modality. We aim to investigate how alignment and performance change with different degrees of heterogeneity, from more similar (e.g.. two languages) to more different (e.g., text and video).
To generate synthetic datasets with varying heterogeneity, we start with the case where both modalities are redundant, meaning Y (the labels) is nonlinear function of x, Specifi- cally, let 1 = Ir and x2 = Φ(z,), where Φ(-) is a nonlinear function, as shown in Figure 3. In this setting, heterogene- ity is defined as the number of nonlinear transformations involved in o(-). and we assume that nonlinear transforma- tions are bijections, ensuring that the information content of the heterogeneous modality remains unchanged. Concretely, if <(-) is modeled as a multilayer perceptron (MIP), the number of layers D& quantifies the level of heterogeneity between the two modalities. We extend this definition to cases where the modalities contain unique information. Let X, = [x,, Fayl and modality that is heterogeneous with respect Xi is defined as X2 (([x,,nng))
Experimental setup for real benchmarks. In addition to experiments on synthetic data, we conduct analogous experiments vision-language models. We use the same dataset and models as Huh et al. (2024), which evaluates alignment on the Wikipedia caption dataset (Srinivasan et al., 2021) with naturally co-occurring text and images. This dataset is inherently heterogeneous (text and images are different) with high redundancy due to overlapping semantic information. To vary the amount of unique information, we leverage GPT-4 synthesize text captions with unique information that is not present in the images. For each (image, text) pair in the original dataset, we prompt GPT-4 to produce 10 captions with increasing levels of uniqueness: 10%, 20%, ... 100%, such that the final caption contains only information that is unique to the text. As uniqueness is already introduced in the text, we keep the original images in
Experimental setup for synthetic datasets. We evaluate
4

---

Understanding the Emergencc of Multimodal Representation Aligument
Dg=1,
p=-0.950
Do=3,
p=-0.833
Dg=5,
p=-0.983
Dz=7,
p=-0.983
Dy=9, p=-1.000
1.0
1.0
1.0
1.0
Alignment
0.8
Alignment
0.8
Alignment 0
Alignment
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
010
0.4
00
0.4
0.4
0.4
0.2
0.2
8000
0.2
0.2
0.0
0.0
0.0
0.0
0.0
5.0
0.0
7.5
2.5
0.0
5.0
0.0
2.5
7.5
U
U
U
Figure 4: Alignment uniqueness synthetic datasets. Alignment is computed between unimodal encoders trained on datasets with different levels of informational uniqueness U. Each dot is an independent run on a different model size on a given dataset. We see that the maximum level of achievable level of alignment, shown by the red dots, decreases as the level of uniqueness increases. Five different figures shows the different levels of nonlinear transformation we apply to the original data. We report the Spearman correlation p between the maximum alignment values and U.
the Wikipedia caption dataset. Using a pretrained sentence BERT model to quantify semantic similarity between the original caption and the GPT-4 captions, verify that the average semantic similarity monotonically decreases as the level of uniqueness increases. See Appendix C.2 for more details.
DINOv2, 0=-500
0.16
0.14
Alignment
0.12
0.10
We experiment with MultiBench (Liang et al., 2021) which collects a diverse range of real-world multimodal datasets: MOSEI (Bagher Zadeh et al., 2018), dataset for predict- ing emotions from videos (vision, audio, language); MOSI (Zadeh et al., 2016), dataset for predicting sentiment from videos (vision, audio, language), URFUNNY (Hasan et al., 2019), a humor detection dataset from videos (vision, au- dio, language); MUStARD (Castro et al., 2019), a sarcasm detection dataset from TV shows (vision, audio, language); and AVMNIST (Pérez-Rúa al., 2019), a dataset for digit classification from paired images and spoken digits. Ad- ditionally, we experiment with MM-IMDb (Arevalo et al., 2017), dataset for classifying movie genres from paired images and text. While we cannot explicitly vary the infor- mation content, past work has collected human annotations of the levels of redundancy and uniqueness in these datasets, showing that most multimodal datasets have significant amount of uniqueness (Liang et al., 2023a).
0.08
0.06
20
40
60
80
100
Unique
Figure 5: Alignment vs uniqueness on real large-scale vision-language datasets. Alignment is computed between DINOv2 vision models and large language models. Each dot is an independent run on a different model size on a dataset with given level of uniqueness. The maximum achievable alignment decreases uniqueness increases.
heterogeneity. In the synthetic setting, the level of unique- ness U denotes the number of unique features used in com- puting the label. In Figure 4, we observe that as increases, the maximum alignment decreases across different model depths and transformation depths. A similar trend is evident in Figure 5, which examines the alignment between large- scale language models and DINOV2 (Oquab et al., 2023) vision models over different levels of U is the percentage of perturbation. See Appendix D.5 for experiments with more vision models. An additional experiment, detailed in Appendix D. demonstrates that data heterogeneity is negatively correlated with the level of achievable alignment. Collectively, these experiments provide strong empirical ev- idence supporting the hypothesis that the level of alignment is indeed constrained by the degrees of heterogeneity and interactions between the modalities.
Computing Alignment. For models trained on synthetic data, we evaluate alignment using unbiased Centered Ker- nel Alignment (CKA) (Kornblith et al., 2019). See Ap- pendix D. for results with additional metrics. Following the methodology outlined in Huh et al. (2024), for large pre- trained vision and language models, we evaluate alignment using mutual KNN, variant of CKA. See Appendix B for more details.
4. RQ1: When does Alignment Emerge?
We empirically evaluate whether aligmment emerges natu- rally by systematically varying redundancy, uniqueness, and
We now investigate whether increasing model capacity can improve alignment between representations of increasingly
5

---

Understanding the Emergencc of Multimodal Representation Alignment
U=0, p=0.844
U=2, p=0.735
U=4,
p=0.432
U=6,
p=0.489
U=8, 0=-0.049
0.8
0.6
0.4
0.4
0.0125
Aligmmett
Alignment
Alignment
0.3
Alignment
Alggnment
0.0100
0.6
0.4
0.0075
0,4
98
10008
0.2
0.2
0.0050
0.2
0.2
0.1
0.1
0.0025
-5
0
5
-5
0
5
-5
0 5 Dew D4
0
Done De
Dine D
Denc
D
Dese D
Figure 6: Emergence of alignment across heterogeneity and uniqueness. We plot alignment with respect to (D Enc D2) for different levels of uniqueness and report the Spearman correlation p. When redundancy is high, we see that alignment emerges when EDene- D&) ) is high. However, as uniqueness increases, the correlation between (D D&) and alignment
vanishes.
heterogeneous and unique modalities. In Figure 6, we plot alignment scores as function of (D Face D where D Eine represents the encoder depth and D& represents the trans- formation depth of the second modality. When uniqueness is low, we observe that alignment improves significantly when the model capacity (relative to the transformation depth) greater. This suggests that increased model capac- ity effective in handling heterogeneity between modalities. Concretely, in these scenarios, alignment appears to follow the trend (Denc D)) Alignment, meaning that the relative capacity of the encoder compensates for the com- plexity introduced by the transformation depth. However, as uniqueness increases, the relationship between alignment and relative model capacity becomes much weaker. In these cases, Ene D no longer predicts higher alignment scores. This indicates that when modalities have high level of unique information, simply increasing model capacity is insufficient to achieve higher alignment. Instead, other factors-sure such as the degree of shared information- may become the limiting factor in determining alignment.
5.1. Alignment-perffrmmance vs interactions//uiqqennss
For each synthetic dataset, we analyze the relationship be- tween alignment, performance, and model capacity. We include model capacity in our analysis as increased capacity generally leads to better performance and is assumed to cor- relate with better alignment. Our findings are summarized in Figure 7, where we plot the correlations between align- ment and performance across different dataset dimensions, where U is defined in Section 4. In highly redundant set- tings, the correlation between aligmment and performance is strong, with relatively little variation across different levels of heterogeneity and random seeds. However, as uniqueness increases, the median correlation decreases toward zero, and the range of correlations expands significantly. No- tably, for U > 3, the correlation even becomes negative in some cases, suggesting that higher uniqueness can disrupt the relationship between alignment and performance. A similar trend is observed when examining the correlation between alignment and model depth in Figure 7 (right). As uniqueness increases, both the median correlation decreases and the variance in correlation increases substantially, with instances of anticorrelated alignment-depth relationships. While deeper models do not necessarily lead to better align- ment when uniqueness high, we see in Figure 7 (center) that performance and depth remain positively correlated across different levels of uniqueness, with much lower vari- ance in correlation at higher uniqueness levels. This sug- gests that while alignment may not always be a reliable predictor of performance, increasing model capacity can still improve task performance.
In summary, while model size and capacity are correlated with alignment, there exists an upper limit to the level of achievable alignment, which is fundamentally determined by the intrinsic properties of the data. This finding implies that perfect alignment cannot be simultaneously achieved with optimal performance when the data modalities inher- ently differ in their information content. Moreover, in- creasing model depth only effectively aligns heterogeneous modalities when they contain highly redundant information and can fail for high uniqueness.
We next verify whether these findings extend to large vision- language models. In Figure 8, we compute linear fits to alignment to DINOv2 and language model performance. As uniqueness increases, the slope of the linear fit decreases, showing that the relation between cross-modal alignment and performance weakens. Nevertheless, we see that the better performing language models are those with greater capacity, showing that increasing capacity can lead better performance even when alignment does not emerge. We include more analysis in Appendix D.6 involving different
5. RQ2: Is Alignment Correlated with Performance?
In this section, we systematically investigate the relationship between alignment and performance, identifying scenarios where alignment enhances performance and others where it may introduce unintended trade-offs.
6

---

Understanding the Emergence of Multimodal Representation Aligument
0.8
1.0-
0.8
Aligment/Prformancc
0.6
Performance/Deptt
0.6
Alignment/Dept
0.5
0.4
0.4
0.0-
0.2
0.0
0.0
-0.2
-0.2
2
Figure Alignment, performance, and depth correlation plots across different synthetic depths and experiment seeds. In each plot, we show the spread of Spearman correlation coefficients p for each level of uniqueness, where the orange lines the median correlations and the dots are outliers. Left: When the two modalities are fully redundant, the alignment is strongly correlated with performance. When the two modalities have high uniqueness, alignment has vanishing correlation with performance. In fact, for significant proportion of tasks, the correlation is negative. Mid: In contrast, model size measured by depth always has a strong positive correlation with performance and does not seem to change across datasets. This means that representation alignment may not be a universal phenomenon, and introduced by some special properties of data. contrast, the influence of model size on performance seems universal and is consistent with the well-observed scaling laws. Right: For each level of uniqueness, we show the variance in alignment/depth correlation. As uniqueness increases, the median alignment/depth decreases to O, and the range of correlation values increases significantly.
U-10, slope=0.112, pm0.852
U-50, slope-o.0o.... pm0.900
U-90, slope-0. slope=0.024,
pm0.612
reed mmpe mm
Momt
Honzz
Mamb
Mantia
Woort
Momaa
Wont Moodd
crail lare
smail
ovall base
Alignment to DINOV?
0.14
Aligrmeet DIVOw2
0.14
lrge
Aligrmeet DINOv2
0.14
large
glane
glane
glank
0.12
0.12
0.12
0.08
0.08
0.08
6.05
0.06
0.06
Performance
Figure 8: Aligmment vs performance across different uniqueness. We plot the vision-language alignment using DINOv2 vision models with respect to language model performance, measured using negative bi ts -eee-pyteeeleeee We show individual best fit lines for each size of vision model and the average Spearman correlation coefficient p. As increases, the slope of the linear fit decreases, showing that better performance be achieved without increased alignment.
vision model training schemes. Overall, our experiments both synthetic data and large vision-language models indi- cate that as uniqueness increases, higher-performing models with greater capacity do not necessarily exhibit stronger alignment. This reinforces the conclusion that alignment does not always predict model effectiveness, particularly when the modalities contain significant amounts of unique information.
show that while alignment and performance exhibit strong linear relationship at low levels of uniqueness, this relation- ship weakens as uniqueness increases. Specifically, with higher uniqueness, models trained on similar modalities do not consistently achieve better alignment than those trained heterogeneous modalities. This suggests that alignment does not uniformly degrade with increasing heterogeneity and that the interaction between uniqueness, heterogeneity, and alignment is more complex than a simple linear relation- ship. These results further reinforce the idea that alignment alone is not a sufficient predictor of model performance, especially in multimodal settings where modalities contain varying levels of interactions and heterogeneity.
5.2. Alignment--pefformance heterogeneity
Additionally, analyze whether alignment correlates with performance across varying levels of heterogeneity in Fig- ure 9. Intuitively, we expect higher levels of heterogeneity to result in lower performance, but is unclear whether this trend is reflected in alignment scores. Our findings

---

Understanding the Emergencc of Multimodal Representation Alignment
U-2
0.7
10
10
Aigument
0.5-
Aligmment
0.53
Algnmeet
0.3
Aligmment
0.015
0.30
0.010
0.5
0.23
9.005-
0.29
0.000
0,675 0/000
0.50
Performare
Perfor
Performance
Performance
Performance
Figure 9: Alignment performance across levels of heterogeneity. We plot the alignment and performance scores different levels of heterogeneity, with the transformed modality's 's encoder fixed to the maximum transformation depth. At high levels of uniqueness, we see that high performance correlates with high alignment, with both being greater at lower synthetic depths. Past U = 4, we see that while performance is higher at lower synthetic depths, the alignment scores on these datasets are not necessarily higher.
Vision
Audio
Vision
Text
Audio
Text
Dataset
Vision
Audio
Vision
Text
Audio
Text
MOSEI (Bagher Zadeh al., 2018)
-0.223
-0.172
-0.093
-0.482
-0.145
-0.641
MOSI (Zadeh al., 2016)
-0.014
0.313
0.116
-0.413
0.403
-0.386
URFUNNY (Hasan et al., 2019)
-0.332
-0.336
-0.232
0.284
-0.373
0.083
MUStARD (Castro et al., 2019)
0.381
0.208
0.436
0.019
0.216
0.438
AVMNIST (Pérez-Rúa et al., 2019)
0.887
0.723
Table 1: Alignment-performannn
correlations MultiBench. We compute the correlation between model performance and alignment across 4 affective computing datasets with tasks that require unique information in vision, audio, and language modalities. We additionally benchmark on AVMNIST, a dataset with high redundancy as the modalities are images of digits and spoken digits for digit classification. On the affective computing datasets, the correlation is weak and often negative, suggesting that enforcing alignment between modalities may not be desirable. In contrast, the alignment of vision and audio modalities in AVMNIST is highly correlated with performance.
6. RQ3: Aliinment-Performaaee is an Inherent Property of Datasets
Correlation
audio). See Appendix C.3 for details about the datasets. We train transformers with varying depths for each modality and compute the cross-modal alignment. See Appendix A for details on our experiment setup.
Finally, we investigate how the correlation varies across real-world multimodal datasets. Quantifying this relation is important to practitioners, as a positive
We show these results in Table 1. On sentiment analy- sis tasks that typically require unique information from language, alignment, and performance are weakly corre- lated or even negatively correlated. For given dataset, the expect that tasks involving redundant information, alignment positively correlates with performance whereas for tasks that require unique information in modalities, the correlation may be weaker and not necessarily positive.
correlation suggests that a practitioner can improve performance by explicitly align- ing modalities. As shown in our experiments in Section 5,
relationship can even vary between different modalities. For example, on MUStARD, alignment is more highly correlated with vision performance, whereas audio and text performance do not seem as correlated. On AVMNIST, alignment strongly correlates with performance for both modalities, the information content is largely redundant information about the digit identity. These results corroborate our findings that the relationship heavily depends on dataset characteristics.
6.1. MultiBench Datasets
We evaluate these hypotheses on subset of datasets from MultiBench (Liang et al., 2021) with varying degrees of task-relevant redundant and unique information content, including MOSEI (Bagher Zadeh al., 2018), a dataset for predicting emotions from videos (vision, audio, text); MOSI (Zadeh et al., 2016),& dataset for predicting senti- ment from videos (vision, audio, text), URFUNNY (Hasan et al., 2019),, a humor detection dataset from videos (vision, audio, text): MUSTARD (Castro et al., 2019), a sareasm detection dataset from TV shows (vision, audio, text); and AVMNIST (Pérez-Ráa et al., 2019), a dataset for digit clas- sification from paired images and spoken digits (vision,
6.2. Algorithmic use case for quantifying
relation
To show how quantifying the tion can impact algorithm design, we consider a practical setting where there is a large dataset of paired input data, but only a small subset of the dataset has labels due to the cost of annotation. One approach to leverage the unlabeled paired data is to finetune pretrained model using both su-
rela-
8

---

Understanding the Emergencc of Multimodal Representation Alignment
13, align-porf-... 0.094
19, wwl.nnperrf-..00)
Class Allgn-perf-0.00
6, allgnn
Class www.gnppeleteeeee rf-d.006
0.32
Rel Performance
Perfrmance
9.50
Tot Perfermanne fereemrcnce
0.41
Nurfamancc
0.78
0.50
0,480
0.76
0,675
0,070
0.05
042
0.72
CLF Fazeraedd
0.78
CJE Sepirvedd
0.465
CUP Superioed
Log CLIP Weight
Log CUP Weight
Log CLIP Weight
Log CUP Weight
Log CUP Weight
Class 15, align 0.00
1, perf=o
align perf-0.114
Class O, align perf-0
17, align pert-D 136
0:730
07:50
0.70
0.59
Perfarmanee
6-728
0.726
Pefformance
0.79
0333
0.59
0.724
0.73
0.62
0.330
0.57
0.66
0.720
0.722
0.34
0.01
0.255
0.55
0.33
0.90
0.79
<UP Seeveee
CLF Satercsed
10P -Leerree
C.P Sepervtes
0.7-
-UP-
Superveed
Log CLIP Weight
Log CUP Weight
Log CLIP Weight
Log CLIP Weight
Log CLI Weight
Figure 10: Aligmment-performaane relation predicts impact of explicit alignment downstream performance MM-IMDb. We finetune CLIP vision and language encoders MM-IMDb using C C + w LCLIP. where Coup is the supervised loss computed on top of the language embeddings and the contribution of LCLP is modulated by a weight w. Left-to-right, top-to-bottom: The classification tasks are ordered by increasing flignment--erformann slopes, which are computed using unimodal language and vision models. The dotted blue line shows the linear fit to performance across varying degrees of explicit alignment, and the dotted red line shows that finetuning with only Coup results in overfitting. Results demonstrate that perfrrrmnceeo relations predict how the amount of explicit alignment (controlled by w) impacts performance. Specifically, classes 13, 19, 10, 2, 15 have smaller aligament-performaace linear fit slopes, which correspond to a weak negative relation between w and performance. In contrast, classes 1, 16, O, and 17 have higher alignment- bigameent-pefformance performance linear fit slopes, in which increasing w generally improves performance.
linear fit
pervised loss and an explicit alignment objectivee objective, such the CLIP loss: L Soup + w LCLP. However, this raises the question of how the contribution of these two losses should be balanced to maximize performance. From our analysis, we posit that the "ideal" amount of alignment is dataset and task-spe If the geagmeent-peffrmmane relation is weak, then performance degrades does not change when increasing the weight the explicit alignment objective. Conversely, when the alignment-performance performance relation is stronger, performance should increase with larger weight on the alignment objective.
w {0, 0.25, 0.5, 1.0, 2.0, 5.0, 10.0, 50.0, 100.0). In agreement with our analysis, we demonstrate in Figure 10 that on the categories with lower slopes, increasing w leads to worse performance, whereas for classes with higher values of w improve performance. These results demon- strate the use of quantifying the relation between alignment and performance, even with unimodal models that not explicitly aligned, for deciding how much to explicitly align the modalities.
performance
slopes, high
To test this hypothesis, we run experiments on MM- IMDb (Arevalo al., 2017), a realistic dataset for classify- ing movie genres from movie posters and text descriptions of the movic plot, where there are 23 classes. As such, the multilabel classification task can be broken down into 23 bi- nary classification tasks. See Appendix D.7 for the distribu- tion of seggnmentt-efformance linear fit slopes. Compared to the original dataset size 25k examples, we use a subset of 1024 labeled examples for each of the train, validation, and test sets to simulate the scarce data scenario. For 10 different classes, we compute cross-modal alignment be- tween the same vision models and language models Huh et al. (2024), and downstream performance is measured by training a linear classifier on the language embeddings. We finetune CLIP vision and language models with L, where
7. Conclusion
This paper provides a comprehensive analysis of the rela- tionship between multimodal alignment, performance, and multimodal data characteristics. We offer nuanced perspec- tive on how aligmment emerges across different modalities and how its effectiveness is influenced by the interactions and heterogeneity within the data. Specifically, our findings show that as uniqueness and heterogeneity increase, the emergence of alignment weakens, and that alignment often fails to track performance in datasets with higher unique- ness. In the case of perfect redundancy, result supports the Platonic Representation Hypothesis, but as the amount of unique information and data heterogeneity increases, our results provide a generalization of this phenomenon.

---

Understanding the Emergencc of Multimodal Representation Alignment
Acknowledgements
Bansal, Y., Nakkiran, P., and Barak, Revisiting Model Stitching to Compare Neural Representations, June 2021. URL http: //arxiv. //arxiv.org//aaaeeo.t.to abs 210...066....2.. arXiv:2106.07688
MT is supported by the National Science Foundation (NSF) under Grant No. 2141064. We acknowledge NVIDIA's
GPU support. We thank Hengzhi Li and Minseok Jung for feedback and discussions.
Barannikov, S., Trofimov, I., Balabin, N., and Burnaev, E. Rep- resentation Topology Divergence: A Method for Comparing Neural Network Representations. In Proceedings of the 39th International Conference on Machine Learning, pp.
Impact Statement
PMLR. June 2022. URL https: / pttp///occeceinsneeooee.eeee...a mlr. press http1////aaaanni///aaaaa
htm.. ISSN: 2640-3498.
This paper presents an empirical analysis whose goal is to advance the field of multimodal representation learning. Specifically, our work opens up the possibility of character- izing and quantifying multimodal datasets via alignment- performance relationships. This can help advance our under- standing of multimodal data and inspire the design of better methods that appropriately align (or perhaps even unalign) modality representations when necessary. Our work also inspires new theoretical questions regarding why different models sometimes converge to similar representations, even though they are often overparametrized and theoretically capable of learning arbitrary representations. Answering these questions can advance our understanding of today's s large-scale multimodal AI systems. Multimodal models are broadly impactful for many real-world applications in- cluding understanding human verbal and nonverbal commu- nication, fusing multiple physical sensors, and analyzing multiple sources of medical data.
Bertinetto, I.,, Valmadre, J., Henriques, J. F. Vedaldi, A., and Torr, P. H. Fully-convolutional siamese networks for object tracking. In Computer Eici 2016 Workshops: Amsterdam, The Netherlands. October 8-10 and 15-16, 2016, Proceedings, Part II 14. pp. 850-865. Springer, 2016.
Bonheme, L. and Grzes, M. How do Variational Autoencoders Learn? Insights from Representational Similarity, Septem- ber 2022. URL http:// / //raaaaivvv..///////..////aaarla aaxiv.orrg org/ abs 2205 08399. (Xiv:2205.08899 [cs].
Cai, Q.. Wang. H., Li, Z., and Liu, X. survey multimodal data- driven smart healthcare systems: approaches and applications. IEEE Access, 0773583-133599 2019.
Castro, S., Hazarika, D., Pérez-Rosas, V., Zimmermann. R., Mi- halcea, R., and Poria, Towards Multimodal Sarcasm De- tection (An _obviously_ Perfect Paper). In Korhonen, A., Traum, D., and Mårquez, L. (cds.), Proceedings of the 57th Annuul Meeting of the Association for Computational Linguis- tics, pp. 4619-4629, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.16633v1/P19-1444 URL ht attpp:: / http///aaaaaattcccee . org/ 0t////99-14555
There are many potential long-term societal consequences of our work, but since our work is primarily an empirical analysis, we feel that none of these impacts must be specifi- cally highlighted here.
Csiszárik, A., Aetorrsi-Szzbbe P., Matszangosz, A., Papp. G., and Varga, D. Similarity and Matching of Neu- ral Network Representations. Information Processing Systems, volume 34, pp. 5656- 5668. Curran Associates, Inc.. 2021. URL https: / /ppocceeding neurips ca/ haapper/222//haaaee 2cb2 0atee//466e9A0EE/eee00012022 wttpe//bbeebeeeeee. html.
Advances in Neural
References
Alayrac. J.,.., Donahue, J., Luc, P., Miech, A., Barr, I. and Hasson, Y. Flamingo: a visual language model for few-shot learning. arXiv preprint ar Xixi:22244. 14.999, 2022.
Dufumier, B., -avvirlo-Navarro, J., Tuia, D., and Thiran, J.-P. What align in multimodal contrastive learning?, Septem- ber 2024. URL http: / //arxvvvvv///////////aa////aaavr org/ abs 2200/ arXiv:2409 00740 [cs].
Andrew, G., Arora, R., Bilmes, J., and Livescu, K. Deep canonical correlation analysis. In ICMI, 2013.
Arevalo, J., Solorio, T., Montes-y Gómez, M., and González, A. Gated Multimodal Units for Information Fusion, 2017. URL http: / /wtpiv..ivv/va////.//e.///aaa www.erggg/a...... /0//1/0000 01992.
Frome, A., Corrado, G. S., Shlens, J.. Bengio, M., and Mikolov, Devise: A deep visual- semantic embedding model. In Advances in neural information processing systems, pp. 2121-2129, 2013.
S., Dean, J., Ranzato,
Bagher Zadch, A., Liang, P. P., Poria, S., Cambria, E., and Morency, L.PP. Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. In Gurevych, I. and Miyao, Y. (eds.). Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2236-2248 Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/P8. 1208. URL org/P18 1
Geng, X. and Liu, H. OpenLLaMA: An Open Reproduction LLaMA, May 2023. URL https:/ httpss://gtttb.cooo con/ www.lln-eeearcch// wtccch/ppen__eee cptp///eenn_aaa....eeee
Golub, G. H. and Zha, H. The canonical correlations of matrix pairs and their numerical computation. Springer, 1995.
Gretton, A., Bousquet, O., Smola, A., and Schölkopf, B. Mea- suring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pp. 63-77. Springer, 2005.
Baltrušaitis, T., Ahuja, C., and Morency, L.-P. Multimodal machine learning: A survey and taxonomy. IEEE TPAMI, 41(2):233-433, 2018.
10

---

Understanding the Emergencc of Multimodal Representation Alignment
Hasan, M. K., Rahman, W., Bagher Zadeh, A., Zhong, J., Tan-
Lene, K. and Vedaldi, Understanding Image Representations by Mcasuring Their Equivariance and Equivalence. Int. Comput. Vision, (27(5) May 2019. ISSN 0920-5691. doi: 10.1007/ 1098-y. URL http::: : /ttp:p::://ci.i/// wed.i..rroo org//00
veer, M. I., Moreney, L.,P., and Hoque, M. E. UR- FRR-FUNNY: A Multimodal Language Datasct for Understanding Humor. In Inui, K., Jiang, J.. Ng, V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joins Conference on Natural Language Processing SSMNLP-IJCNLAS pp. 2046- 2056. Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.18653/1/D19-1 1211. URL https: www.a/twww.oooooooooooooooe
Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H. Align before fuse: Vision and language representa- tion learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021.
He. K., Chen, X., Xie, S., Ii, Y., Dollár, P., and Girshick, R. Masked Autoencoders Are Scalable Vision I. Learners. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16000-16009, 2022.
Li, Y., Yosinski, J., Clune, J., Lipson, H., and Hopcroft, J. Convergent I carning: Do different neural networks learn the same representations? In Proceedings of the 1st Interna- tional Workshop on Feature Extruction: Modern Questions and Challenges at NIPS 2015, pp. 196-212. PMLR, December 2015, URL https: prw.eediciinooooooo www.iingn.........mmmmm..mm. www. prwss//44///// 1 1115convrrgent.oo html. ISSN: 1938-7228.
Hu, D., Nie, F., and Li, X. Deep multimodal clustering for unsu- pervised audiovisual learning. In CVPR, pp. 9248-9257, 2019.
Huh, M., Cheung. B., Wang. T., and Isola, P. The Platonic Rep- resentation Hypothesis, July 2024. URL org/abs / arXiv:2405.07987.
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., and Chen, L. Y. Multibench: Multiscale benchmarks for multimodal representation learning. In NeurlPS Datasets and Benchmarks Track, 2021.
Jackson. Z. Jakobowskkfirre-spoknn--g digttatttt,,,ittttttttttttt January 2025. URL bttps://github. com/ Jakobovski/ free- 21T09:46:207.
Liang, P. P., Cheng. Y., Fan, X., Ling, C. Nie, S., Chen, R. J., and Deng, Z. Quantifying & modeling multimodal interactions: An information decomposition framework. In NewrIPS, 2023a.
original-date: 2016-06-
Jia, C., Yang, Y., Xia, Y., Chen, Y-T., Parckh, Z., and Pham, H. Scaling up visual and vision-language representation learning with noisy text supervision. In /CML, pp. 4914-4916. PMLR, 2021.
Liang, P. P., Deng, Z., Ma, M., Zou, J., Morency, L.-P., and Salakhutdinov, R. Factorized contrastive learning: Going be- yond multi-view redundancy. In NeurlPS, 2023b.
Liang, P. P., Zadeh, A., and Moreney, L.-P. Foundations trends in multimodal machine learning: Principles, challenges, and open questions. ACM Computing Surveys, 56(10): 42, 2024.
Kirchner, E. A.. Fairclough, S. H., and Kirchner, F. Em- bedded multimodal interfaces in robotics: applications. fit- ture trends, and societal implications. pp. ciation for Computing Machinery and Morgan & Claypool, 2019. ISBN 9781970001544. URI. https: /doii .org/ 10. 1 001114433333373 323333333395333331111 3233810.
Asso-
McFee, B., Raffel, C., Liang, D., Ellis, D. P., MiCarrr M., Bat- tenherg, E., and Nieto, O. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, volume 8, 2015.
Klabunde, M., Schumacher, T., Strohmaier. M., and Lemmerich, F.
Similarity of Neural Network Models: A Survey of Functional and Representational Measures, August 2024. URL http:// rexiv.org//e www.or////////oeeeeeeoooeeeeeeee 2305. 06329. arXiv:2305.066: (cs).
Mekhaldi, D. Multimodal document alignment: towards a fully- indexed multimedia archive. In Proceedings of the Multimedia Information Retrieval Workshop. SIGIR, Amsterdam, the Nether- lands, 2007.
Korblithh S., Norouzi, M., Lec, H., and Hinton, G. Sim- ilarity Neural Network Representations Revisited, July 2019. URL http: wwwpwwwwwwwg...//..eee..ooesse 1901.00414. 1905v:1905.00414.
Morcos, A. S., Raghu, M., and Bengio, S. Insights on repre- sentational similarity in neural networks with canonical cor- relation, October 2018 URL http:// //t.xiii.orrg////..e......o.o..e abs/ 1806. 05759. 1xXiv:880.00555) [stat].
Lai, L. and Fyfe, C. Kernel and nonlinear canonical correlation analysis. International journal of neural systems, 10(05):365- 377, 2000.
Muhammad, G., Alshehri, F., Karray, F., and El Saddik, A. A comprehensive survey on multimodal medical signals fusion for smart hcaltheare systems. Information Fusion, 76:355-375, 2021.
Lecun, Y., Bottou, I.,, Bengio, Y., and Haffner, P. Gradient- based learning applicd to document recognition. Proceedings of the IEEE, 86(1 11):2278-2324 November 1998. ISSN 1558- 2256. doi: 10. 19995777777. URL https: /ippss//eeeeeporreee ieee. org/ wwtpd//ctt/////oe/ooooooot /dccureet/126791 /72679 coedings of the IEEE
Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez. P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R.,. Huang. P.Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jogou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research, July 2023. ISSN 2835-8856. URL https: /oopnrevviii net. forum?
Conference Name: Pro-
Lee, M. A.. Zhu, Y., Srinivasan, K., Shah, P., and Savarese, S. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In ICRA, pp- 8943-8950. IEFF, 2019.
I6=a688Ut622

---

Understanding the Emergencc of Multimodal Representation Alignment
Peng, H., Long, F., and Ding, C. Feature selection based on mutual information criteria of max dex-dependency, max-relevance, and min-redundancy. IEEE Transactions on pattern analysis and machine intelligence, 27(8):1226-
Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. Videobert: A joint model for vidco and language representa- tion learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7464-7473, 2019.
2005.
Pérez-Rúa, J.-M., Vielzeuf, V., Paleux, S., Baccouche, M., and Jurie, F. Mfas: Multimodal fusion architecture search. In CVPR, pp. 6966-6975, 2019.
Thompson, B. Canonical Correlation Analysis. In Encyclopedia of Staristics in Behavioral Science. John Wiley & Sons, Ltd, 2005. ISBN 978-0-470-01319-9 doi: 10. 10010///477011192.hhaaaa
Tian, Y., Sun, C.,, Poole, B., Krishnan, D., Schmid, C., and Isola, P. What makes for good views for contrastive learning? NeurlPS, 33:6827-6839, 2020.
Pérez- Rúa, J.-M.. Vielzeuf, V., Pateux, S., Baccouche, M., and Jurie, MFAS: Multimodal Fusion Architecture Search, March 2019. URL. http:/ /a waaxiv..croo http://////////////////////////t 09649 06496. 1933v:1903 [es].
Tosh, C., Krishnamurthy, A., and Hsu, D. Contrastive learning, multi-view redundancy, and linear models. In ALT, 2021.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., and Agarwal, S. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763 PMLR, 2
Touvron, H., Martin, I.., Stone, K., Albert, P., Almahairi, A.,
Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C.,, Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J.. Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., al. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023, URL tttp: / arxiv. org/ abs /207 09288. arXiv:2307 09288 [cs].
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agar- wal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th In- ternational Conference on Machine Learning, pp. 8748-8763.
PMLR, July 2021b. URL at:ps: http//pooeeeiiigeeeee ngg...nn................... mmrr. press/ /vl.s//v39////// http//rdtoorrddd
Wang, W., Arora, R., Livescu, K., and Bilmes, J. On deep multi- view representation learning. In ICML, pp. 1083-1092. PMLR, 2015.
ISSN: 2640-3498.
Raghu, M.. Gilmer, J., Yosinski, J., and Sohl-Dickstein, SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretabiliity, Novem- ber 2017. URL http:// /ttttt///aiii //tpaaivcrg///aaaa////////oaeaea 0tt.//106..050 05806. 1rXiv:100.008006 [stat].
Wightman, R. PyTorch Image Models, 2019. URL https: github. com/ rwi www.gttan//eettt.t...o.....tt wttpa////ypoooo---eeeeeeeelelll Publication Title: GitHub repository.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C.,, Moi, A., Cistac, P., Rault, T., Louf, R., Funtowiez, M., Davison, J.. Shleifer, S., Platen, P. v., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao. T. L., Gugger, S., Drame, M., Lhoest, Q. and Rush, M. Transformers: flate-oftthe-At Natural Language Pro- cessing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Longuage Processing: System Demonstra- tions. pp. 45, Online, October 2020. Association for Com- putational Linguistics. URL ht tps / /www. aclweb. aclweb.org/ anthol ogy/ 10gg//000 22000 emn1 1 p- demos. 6.
Ramesh, A., Pavloy, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, 1. Zero-shot text-to-image generation. In ICML, pp. 8821-8831, PMLR, 2021.
Russakovsky, O.,, Deng, J., Su, H., Krause, J.. Satheesh, S., Ma,
S., Huang. Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, I.. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):
211-252. December 2015. ISSN 1573-1405 doi: 10.1007/ $11263-015-0816-y URL https::////t/. / /tw.///oio////////// orr/10.1007/ s11263- 0816--1-----.11.1111111111---111
Workshop, B., Scao, T. L., Fan, A., Akiki, C., Pavlick, E., llié, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., Tow, J., Rush, A. M.. Biderman, S., Webson, A., Ammana- manchi, P. S., Wang. T., Sagot, B., Muennighoff, N., Moral, V. d., Ruwase, O., Bawden, R., Bekman, S., McMillan- Major, A., Beltagy, I., Nguyen, H., Saulnier, I.., Tan, S., et al. BLOOM: A Par-mmerre Open- Apen-Access Multilingual Lan- guage Model, June 2023. URL http:// //t///aavvvv/////////////////a./ org/ abs// 2211. 0s100. arXiv:2211 1.05100 [cs].
Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajaling- ham, R., Issa, E. B., Kar, K., Bashivan, P., Preseott-Roo J., Schmidt, K., Yamins, D. L. K., and DiCarlo, J. J. Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?, September 2018. URL https: //www. biorxiv httpg///neet//tteeeeeeeeee 1tt.:/11..11147000000000 407007 Section: New Results.
Pages:
Xiao, Y.. Codevilla, F., Gurram, A., Urfalioglu, O., and López, A. M. Multimodal end-to-end autonomous driving. JEEE Transactions on Intelligent Transportation Systems, 23(1):337- 547, 2020.
Singer. U., Polyak. A., Hayes, T.. Yin, X.. An, J.. Zhang, S., and Hu, Q. Make-a-video: Text-to-video generation without text-vidco data, arXiv preprint arXiv:2209. 14792, 2022.
Srinivasan, K., Raman, K., Chen, J., Bendersky, M., and Na- jork, M. WIT: Wikipedia-based Image Text Dataset for Mul- timodal Multilingual Machine Learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, pp. New York, NY, USA. July 2021. Association for Computing Machinery. ISBN 978- 978-1-4503-8037-9, doi: 10. NL.1145/34448355344660 URL htwph:ttt/////. /dl http://dddlmm......ommm.mm.o org/ 346322.5........... 3463257.
Zadeh, A., Zellers, R., Pincus, E., and Morency, L.P. MOSI: Mul- timodal Corpus of Sentiment Intensity and Subjectivity Anal- ysis in Online Opinion Videos, August 2016. URL http:// arxiv, . org/ abs/160e.... 0 06259. arXiv: 160606629 [cs].
12

---

Understanding the Emergencc of Multimodal Representation Aligument
A. Experimental Details
A.I. Synthetic Data Experiments
On the synthetic dataset, we train MLPs with the AdamW optimizer with the number of hidden dimensions kept the same as the number of input features, 12. For given level of uniqueuness, we choose suitable hyperparameters across different model depths and transformation depths. Specifically, we tune the learning rate in the range (le 1, le 2, le 3, 3, 4) and weight decay in the range {0, le 1, le 2, le 3, le 4) for each modality. The depth 1 ML.P for the untransformed modality were trained for 50 epochs and the models for the transformed modality were trained for 300 epochs. We use batch size of 512 for computing alignment. To ensure robustness, we report results with five different random seeds for each dataset.
A.2. Vision-Language Alignment
We evaluate alignment using the same set of language and vision models as Huh et al. (2024). The language model families considered are BLOOM (Workshop et al., 2023), OpenL.aMM (Geng & Liu, 2023), and LLaMA (Touvron et al., 2023) downloaded from HuggingFace (Wolf et al., 2020). The vision models are vision transformer models of various sizes trained on various data and objectives. These include classification on ImageNet-21K (Russakovsky et al., 2015), MAE (He et al., 2022), DINOv2 (Oquab al., 2023), CLIP (Radford et al.. 2021b), and CLIP finetuned on ImageNet 12K. These models downloaded from PyTorch Image Models (Wightman, 2019).
A.3. MultiBench Experiments
We train transformers on the pre- pre-extracted video, audio, and text features for the affective computing datasets and the audio modality of AVMNIST, and vision transformers the AVMNIST digit images. For each modality, we vary the depth of the transformers in the range {1...., 10]. We use single head for self-attention and set the embedding size to the input dimension. For classification tasks, we append a [cls] token to the sequence with a learnable embedding. The embedding of this token is used to compute alignment between layers; otherwise, we do average pooling over the input sequence. We use the AdamW optimizer. For each dataset, we choose suitable hyperparameters across different model depths and tune the learning rate in the range {le - 3, 5e 1, - 4, - 5, - 55 and and weight decay in the range {0, 1e- 1, le 2, le 3, le- 4}.
To ensure robustness, we train each architecture across 3 different seeds, providing 30 alignment-performmnce data points. To get the alignment-performanee with respect to the modality 1 model with the highest validation score across the different seeds, and we report the correlation of these alignment scores with the performance of the modality 2 models.
correlation given modalities and 2, the alignment of every modality 2 model is computed
A.4. MM-IMDb Experiments
To compute downstream performance, we train linear classifiers on top of the final hidden layer embeddings of the language
model described in Appendix A.2 for 100 epochs. We tune the learning rate in the range (5e - 3, le - 3, 5e - 4, le - 4} and
weight decay in the range (0, le - 1, le - 2, le - 3, le - 4). For finetuning models trained with CLIP, we use a learning rate 10-4 and a cosine scheduler with final value of 10-6 and a warmup over 10 epochs. Models were optimized for 30 epochs.
B. Alignment Computation
Given mean-centered feature sets of n samples, Z1, Zz E Raxd, from two modalities X1 and X2, we first compute the covariances of these two different feature sets, and then compute the empirical estimator of the Hilbert-Schmidt Independence Criterion (Gretton et al., 2005) using a linear kernel. Hence,
1
HS (z (n 1)2
Tr (7ZIZZZZZ31
TTT
(I)
(n- 1)3
The Centered Kernel Alignment (CKA) (Komblith et al., 2019) is then obtained by normalizing HSIC to ensure scale invariance and comparability across different feature sets:
13

---

Understanding the Emergence of Multimodal Representation Aligument
HSIC(2)2,,
CKA(Z1 Z1, 22)
(2)
2,2 Z2Z44)
As demonstrated in (Huh et al., 2024), the definition of alignment can be adjusted to limit the cross-covariance measurement to only those samples identified as nearest neighbors of the current sample i. This modification prioritizes similarity over dissimilarity, thereby emphasizing local alignment:
ALIGNMKNN (2) = ΣΣ (ii.))
(3)
:.
where, a(i,)) 1[ (n,(Z1,)) Za, knn(Z2,)) A i
(4)
Here, Zi. and Z2 refer to the b4th row of Zı and Z2, respectively, while MKNN denotes Mutual KNN.
Thus, Mutual-KNN MKNN is defined as:
MKNNN Z1, Zz)
MKNN Zz, 72)
(5)
ALLINMMKNN(211 Zi) ALIGNMKNN Zz, Zz)
Following Huh et al. (2024), we use k 10 nearest neighbors over 1024 samples from the Wikipedia caption dataset. For the vision model, the class token of each layer is used, and for the language model, the embeddings of given layer average pooled to single token. la normalization is applied to the features and elements in the features that are above the 95-th percentile are truncated.
After computing the alignment between all pairs of layers between ES and Eg,d using CKA or mutual KNN, we report the best alignment score across all layer pairs (Schrimpf et al., 2018).
C. Dataset Details
C.1. Synthetic Data
We discuss in detail how construct synthetic dataset with two modalities to analyze how uniqueness, redundancy, and heterogencity influence the emergence of alignment. Let 21 = [xr, Ta, and = [xy, Tug] Here, 2, R" represents the redundant information shared between the two modalities, while Tay Buz E R denote the unique information for each modality. Both 2 and ž2 represent arbitrary data samples.
For each data sample, we generate Zy. Tuy and uwz by sampling binary vectors from uniform distribution. Specifically, H, Uniform( ((0, 1) Uniform((0, 1) and Tuz Uniform 1))).
To define the labels for this dataset, we introduce task masks MR E R and MU, Mv, Re which determine the features used in computing the output labels. These masks indicate whether a particular feature contributes to the label- generation process. Specifically, the task masks are defined as follows, where the subscript i refers to the zt entry of the respective mask vector:
if 100i<< mosiinn otherwise.
MR.
(6)
if < <<<<<<>> mg
Mus.
(7)
otherwise.
if if otherwise.
Mu2;
(8)
14

---

Understanding the Emergence of Multimodal Representation Alignment
We define the task label y as function by (-) of the masked components xy O MR Fun O MU., and uuz O M such that:
y y (z, + MR. Zu, C MU, Luz ① Mu.),
where 2, O MR captures the task-relevant redundant information, Tu, O MU, captures the task-relevant unique information from modality and uuz + Mú, captures the task-relevant unique information from modality 2.
Here, O denotes the element-wise (Hadamard) product. Intuitively, the task masks MR. Mu and MU, are essential for controlling the relative contributions of the redundant (I,,) and unique (Tay Tuz) components to the label generation process.
In our synthetic experiments, we assume the joint distribution of the components as follows:
P(Xc, XUx Xv2) = P(Xc) (((es)
(10)
Where, P(Xc) = Uniform (0, 1)mm)
(11)
P(Xu;) Uniform (0, 1)
(12)
P(Xu,) Uniform (0, 1)
(13)
This formulation assumes that the redundant information (z,)) and the unique components (Zu,, Fug are all independently distributed.
The task masks play critical role in modulating which features are used to compute the labels, thereby allowing precise control over the relative importance of shared and unique information the synthetic dataset. This design facilitates the study of how these components influence alignment and downstream task performance.
In our experiments, we fix ny (the number of features relevant for the task) while varying ng, ny to explore different dataset configurations. Concretely, ny ng ny-
When My n, both modalities have equal amounts of task-relevant information, allowing them to perform equally well on the downstream classification task. However, when na < ny, the shared information Ie becomes insufficient to fully capture the task-relevant features. By adjusting the proportion of na we heuristically vary the amount of task-specific shared information. This allows us to explore how the balance of redundant and unique information impacts alignment and downstream performance.
An example of this is when ny vr Uniform( (0, 13), Uniform ( Fuz Uniform( ((0, 1))- Given that the label function is OR function of [ J. where ny 1, we can see how the label predictions would differ based on [Te: Fuy] and [ug] in Table 2.
21
.22
91
92
y
00
00
0
0
0
01
00
1
0
1
00
01
0
0
O
01
01
1
0
10
10
1
11
10
1
1
10
11
11
1
1
Table 2: ny Predictions. d2 incorrect for 2 examples due to lacking the unique information.
In contrast, when the labels are an OR of [ ]. where ny we can see how the label predictions would differ based on 2 = [ay, and æ2 [xy, Tug] in Table
To incorporate heterogeneity into the setup, we transform the second modality (I2) using a nonlinear function <(-). Specifically, d(-) is modeled as multilayer perceptron (MIP),, where the number of layers (D₂), also referred to as transformation depth, serves as a heuristic measure of heterogeneity. A higher D& implies more complex transformation, thereby increasing the heterogeneity between the two modalities. Hence, concretely, I2,s = o([x,, Fuyl).
In all our experiments, we fix ny 8, which represents the total number of task task-relevant features. We vary nn E (O.....8) 8),
15

---

Understanding the Emergence of Multimodal Representation Alignment
91
92
y
00
00
0
0
0
01
00
1
0
0
1
01
01
1
10
10
0
0
O
11
10
1
0
10
11
0
11
11
1
1
Table 3: = 2 Predictions. Both d1 and 82 are incorrect for 2 examples due to lacking unique information in the other modality.
thereby controlling the amount redundant (shared) task-specific information. Consequently, the amount of unique task-specific information is determined as ng ny ng- We refer the level of unique information U nu.
C.2. Wikipedia-Imape Text Dataset with Uniqueness
Below the prompt GPT-4o to create captions with unique information.
Annotation Instructions
Imagine you have been assigned the task of progressively enhancing the following caption by systematically introducing unique and differentiating details:
**Original Caption:** <caption>>
### * Task Overview:** You will generate #*10 increasingly different variations** of this caption, ensuring that each version changes the semantic meaning of the *riigiaal caption** If an image is provided, ensure that the changes to the caption are semanticaly different **distinct from the visual elements in the image** ### Definitton of Changing Semantic Meaning** Changing semantic meaning means that the modified caption should **alter the image if used in generation model**
This can be achieved by changing visual cues of the original caption including but not limited to:
Identity of objects or people
Textures of objects or landscape elements
Location, time of day, weather, environment specifics
### **Task Breakdown & Structure:*
1. Incremeenal Enhancement:**
Generate ** 10 * of the caption.
Each version should introduce an increasing amount of semantic differences by increments **[10, 20, 100] (in percentage)**
**Gradual Transformation:***
**
Ensure that each step logically builds upon the previous one.
The final version should have a completely different semantic meaning from the original caption.
3. **Handling Image Input (if provided):**
If an image is provided, ensure that **the semantics of the changed captions are different from the visual clements in the **
# **Output Formatting:**
Each caption should be **separated by consistent delimiter** to ensure clarity.
Use the following format for **each generated caption:**
Caption N Uniqueness Percentage Generated Caption
Ensure that each step **logically evolves** from the previous version, creating seamless and natural transformation.
### #*Expected Output Format Example:**
*Input Caption*: Golden hues gently stretch across the horizon, deepening as the sun slowly dips, casting soft amber
16

---

Understanding the Emergence of Multimodal Representation Alignment
reflections on the tranquil sea.
Caption 10%: Crimson and violet hues gently stretch across the horizon, deepening as the slowly dips, casting reflections on the tranquil sea.
Caption 2 20%: Crimson and violet hues gently stretch across the horizon, deepening as the sun slowly dips, casting reflections on the waves.
Caption 30%: Crimson and violet hues gently stretch across the horizon, deepening as the sun rises, casting reflections on the waves.
<Captions 4-10 omitted for brevity>
### ** **Goal:** By the end, the series of 10 captions should **illustrate clear evolution* in semantic meaning both in terms of text and any provided image.
I*Iut Parameters:**
**Caption:** "caption"
**(Optional) Image:** A visual reference that must also be considered when introducing unique details. Your task is to ensure that each new version would generate an image that ""perceptibly different** from both the original caption and any provided visual input.
C.3. MultiBench Dataset
Below, we discuss the MultiBench datasets in more detail.
MUStARD (Castro et al., 2019) is a dataset for automated sarcasm discovery, compiled from popular TV shows, including Friends, The Golden Girls, The Big Bang Theory, and Sarcasmaholics Anonymous. There are 414, 138, and 138 video segments in the training, validation, and testing data, which gives total of 690 data points.
MOSI (Zadeh et al., 2016) is dataset for sentiment analysis consisting of 2,199 opinion video clips. Each vidco further split into short segments (roughly 10-20 seconds) that are annotated, resulting 1284, 229, 686 segments the train, validation, and testing sets. As the annotations are sentiment intensity, which ranges from [-3, 3], we train models the continuous labels with LI loss and evaluate positive-negativy classification accuracy.
UR-FUNNY (Hasan al., 2019) is large-scale dataset for humor detection human speech, consisting of more than 16000 video samples from TED talks collected from 1866 videos. There are total of 10,598, 2,626, and 3,290 segments in the train, validation, and testing sets. Humor is annotated as either positive negative, with homogeneous 50% split in the dataset.
MOSEI (Bagher Zadeh al., 2018) is large-scale dataset for sentence-level sentiment analysis and emotion recognition from real-world online videos, containing more than 65 hours of annotated video from more than 1,000 speakers and 250 topics. There are total of 16,265, 1,869, and 4,643 segments in the train, validation, and testing sets, resulting in 22,777 data points. As in MOSI, we train our models on continuous sentiment intensity labels with LI loss and evaluate positive-negative classification.
AVMNIST (Pérez-Rúa et al., 2019) is dataset created by pairing audio of human reading digits from the FSDD dataset (Jackson, 2025) with written digits in the MNIST dataset (Lecun et al., 1998) with task to predict the digit into one of 10 classes (0-9). While common practice (Pérez-Ráa et al., 2019) is to increase the difficulty by removing 75% of energy in the visual modality via PCA and adding noise from ESC-50 to the audio modality, we use the unnoised image and audio modalities in order to preserve the redundant information between modalities. An audio sample from FSDD with matching digit identity is paired with each image in MNIST, resulting in 55000, 5000, and 10000 examples in the train, validation, and test sets respectively. We train vision transformers on MNIST images that are converted to 4x4 patches with a sequence length of 49. We preprocess the raw FSDD audio into 36 MFCC coefficients with maximum sequence length of 20 using librosa (McFee et al., 2015).
C.4. MM-IMDb Dataset
Multimodal IMDb (Arevalo et al., 2017) is a large-scale real world dataset. curated by filtering out movies from the Movielens 20M dataset that lack poster. Each data point MM-IMDb consists of the movie poster and plot summary, as
17

---

Understanding the Emergence of Multimodal Representation Alignment
well additional metadata such year, language, director, In our experiments, the raw data instead of the preprocessed features from Multibench. In addition, consider classifying different movie genre separate downstream tasks and compute alignment-perrfrrmance linear fits for each genre.
D. Additional Figures
D.I. Synthetic Data Results with Different Alignment Metrics
In Figures 11, 12, and 13, we plot the alignment between unimodal encoders with respect to uniqueness using different aligmment metrics, including unbiased CKA (Korbblith et al., 2019) with linear and RBF kernels, SVCCA (Raghu et al., 2017), mutual k-NN (Huh et al., 2024)), well with different batch sizes.
In Figures 14, 15, and 16, plot the alignment between unimodal encoders with respect to heterogeneity. In Figures 17, 18, and 19, we plot the aligmment, performance, and depth correlations using different alignment metrics and batch sizes. Overall, our results are consistent across various alignment metrics and batch sizes.
18

---

Understanding the Emergence of Multimodal Representation Aligument
D,=
p=-0.850
D,=3,
pw-0.833
D,=5,
pw-0.950
De P 00-90.93
D, pm-0.9
1.0
1.0
1.0
1.0
1.0
Alignment
0.8
0 (000
Alignmeent
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
D.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
2.3
7.5
2.5
7.5
2.5
7.5
2.5
5.0
7.5
0.0
2.5
5.0
7.5
U
U
U
(a) Unbiased CKA with Linear Kernel, Batch Size = 256
De=1, p=-0.933
D,=3, p=-0.833
Dy=5,
p=-0.983
D5=7, p=-0.983
D,=9, p=-1.000
1.0
1.0
Alignment
000k 6000
Alignment
Alignment
Alignment
Aligmment
0.6
0.4
0.4
0.4
0.4
0.2
600
0.2
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
U
U
Unbiased CKA with RBF Kernel, Batch Size
256
De=1, 1, p=-1.000
D,=3,
p=-0.983
D,=5,
p=-1.000
Dę=7,
p=-1.000
D 9, p=.0.950
1.0
1.0
1.0
1.0
1.0
Alignment
0.8
Alignimnt
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
2.5
7.5
2.5
5.0
U
(c) SVCCA, Batch Size 256
D,=1, p=-0.850
Dy=3,
p=-0.850
D,=5,
p=-0.983
D,=7, 0=-0.983
D,=9, p=-0.983
1.0
1.0
1.0
1.0
1.0
Alignment
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
6.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
2.5
7.5
2.5
7.5
0.0
2.5
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
U
U
U
U
(d) Mutual K-NN (k = 100), Batch Size = 256
Figure 11: Alignment uniqueness with batch size = 256. Spearman correlation coefficient p is computed between the maximum alignment, shown in red, and the level of informational uniqueness U.
19

---

Understanding the Emergence of Multimodal Representation Aligument
D,=1,
pm-0.950
D,=3,
pw-0.833
D,=5.
pw-0.983
D₂=7, P -0.93.93
9, pm-1.000
1.0
1.0
1.0
1.0
1.0
Alignment
0.8
Alignmient
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
D.4
0.4
0.4
0.2
0.2
0490
0.2
0.2
0.0
0.0
0.0
0.0
0.0
2.3
7.5
2.5
7.5
2.5
7.5
2.5
5.0
7.5
0.0
2.5
5.0
7.5
U
U
U
U
(a) Unbiased CKA with Linear Kernel, Batch Size = 512
De=1, p=-0.950
D,=3, p=-0.850
Dy=5,
p=-0.983
D5=7, p=-0.983
Dæ=9, p=-1.000
1.0
1.0
Alignment
080 6040
Alignment
Alignment
Aligment
Aligmment
0.6
0.4
0.4
0.4
0.2
800
0.2
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
U
U
Unbiased CKA with RBF Kernel, Batch Size 512
De=1,
p=-1.000
D,=3,
p=-1.000
Dy=5,
p=-0.983
Dę=7,
p=-1.000
D₂= 9, p=.0.950
1.0
1.0
1.0
1.0
1.0
Alignment
0.8
Alignmmnt
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
2.5
7.5
2.5
5.0
U
U
(c) SVCCA, Batch Size 512
D,=1, pm-0.950
D,=3,
pm-0.950
Dy=5,
p=-0.983
D,=7, 0m-9.8
D,=9, p=-1.000
1.0
1.0
1.0
1.0
1.0
Alignment
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
6.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.0
2.5
7.5
2.5
7.5
0.0
2.5
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
U
U
(d) Mutual K-NN (k = 100), Batch Size = 512
Figure 12: Alignment uniqueness with batch size = 512. Spearman correlation coefficient p is computed between the maximum alignment, shown in red, and the level of informational uniqueness U.
20

---

Understanding the Emergence of Multimodal Representation Aligument
D,=1, p=-0.950
Dy=3,
p=-0.850
D,=5,
p=-0.983
D,=7, p=-0.983
D,=9, p=-0.983
1.0
1.0
1.0
1.0
1.0
0.8
Alignment
0.8
00 cm
Alignment
0.8
Alignment
0.8
Allgnment
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
U
U
U
U
(a) Unbiased CKA with Linear Kernel, Batch Size 1024
D==1, p=-0.950
D,=3, p=-0.850
Dy=5,
p=-0.983
Dx=7, p=.0.983
Dæ=9, p=.0.983
1.0
Alignment
GM SOED
Alignment
Alignment
Alignment
Alignment
0.6
0.6
0.4
0.2
600
2.5
5.0
7.5
2.5
5.0
7.5
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
(b) Unbiased CKA
RBF Kernel, Batch Size
1024
De- 0-=1, 1,
1.000000
Da- 3, p=-1ppp00 1.000000
D
D,
pw-0.983
De
7,
p0000000
D 9, p=-0.950
1.0
1.0
1.0
1.0
1.0
Alignment
0.8
Nignment
0.8
Alignment
0.8
Niigneet
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
D.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
2.5
7.5
2.5
7.5
2.5
5.0
U
U
U
U
U
(c) SVCCA, Batch Size = 1024
D,=1, p=-1.000
D,=3,
0=-0.983
D,=5,
p=-1.000
D,=7,
0=-0.983
D,=9, p=-1.000
1.0
1.0
1.0
1.0
1.0
Alignment
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4-
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.0
2.5
7.5
0.0
2.5
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
U
U
U
U
U
(d) Mutual k-NN (f 100), Batch Size 1024
Figure 13: Alignment vs uniqueness for various representation similarity metrics with batch size 1024. Spearman correlation coefficient is computed between the maximum alignment, shown in red, and the level of informational uniqueness U.
21

---

Understanding the Emergence of Multimodal Representation Aligument
U=0, p=-0.891
U=2, 0=-0.33
U=4, p=-0.394
U=6, p=-0.685
U=8, p=N/A
1.00
1.00
1.00
1.00
1.00
Alignment
0.75
600 09 osamm
6 6400 d0
09 camsuna
Alignment
0.75
60 69 omE
Alignment
0.75
Alignment
0.75
Alignment
0.75
0.50
600
e
0.50
64 600
0.50
0.50
0.50
0.25
0.25
600
0.25
0.25
0.25
Dg
D,
Dj
D₁
(a) Unbiased CKA with Linear Kernel, Batch Size = 256
U=0, p=-0.891
U=2, p=-0.685
U=4, p= 0.394
U=6, p= 0.65.
U=8, p=N/A
Ailgnment
0.75
0 600-0000
0000 emDee
Alignmeet
0.75
B ma
Alignment
0.75
Alignment
0.75
Alignnient
0.75
0.50
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.25
600
0.00
0.00
0.00
0.00
0.00
Da
D,
D
D2
D2
(b) Unbiased CKA with RBF Kernel, Batch Size 256
U- 0, pm-0.552
U=2,
-07994
U 4,
0.515
Uw6, p=-0.709
U=8, p-N/A
1.00
1.00
1.00
1.00
1.00
Aiighment
0.75
0.75
Aligmment
0.75
Aiignment
0.75
Alighment
0.75
0.25
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
5
5
0
De
D,
D
D₂
D,
(c) SVCCA, Batch Size 256
U=0, p=-0.891
U=2, 00-0.
U=4, p=-0.576
U=6, p=-0.685
U=8, p=N/A
1.00
1.00
1.00
1.00
1.00
Alignment
0.75
Aignment
0.75
Alignmen
0.75
Aligneet
0.75
Alignment
0.75
0.50
0.50
0.50
0.50
0.50
5
0
D+
D,
D
D
D2
(d) Mutual k-NN (k 100), Batch Size = 256
Figure 14: Alignment vs heterogeneity for various representation similarity metrics with batch size 256. Spearman correlation coefficient p is computed between the maximum alignment, shown in red, and heterogeneity. N/A denotes that undefined as all alignment values are 0.
22

---

Understanding the Emergence of Multimodal Representation Aligument
U=0, p=-0.915
U=2 00709
Um4, pw-0.442
U=6, pw-0.770
U=8, p-N/A
1.00
1.00
1.00
1.00
1.00
Alignment
0.75
Alggneent
0.75
Alignment
0.75
Aligneett
0.75
Alignment
0.75
o 1
64 0 40000
0.50
0.50
0.50
0.50
0.50
0.25
600
E0
610
0.00
5
5
0
De
D,
De
Dj
D2
(a) Unbiased CKA with Linear Kernel, Batch Size = 512
U=0, p=-0.915
U=2, p=-0.709
U=4, p=-0.442
U=6, p=-0.770
U=8, P=N/A
Alignment
0.75
Alignment
0.75
Alignment
0.75
Alignment
0.75
Allinnient
0.75
ermenae
649 0
@ qum
0.50
0.50
0.50
0.50
0.50
000
80
0.25
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
D₂
D,
D
D2
D₁
Unbiased CKA with RBF Kernel, Batch Size 512
U=0, 0=-0.673
U=2, 0=-0.782
U=4, p=-0.624
U=6, p=-0.576
U=8, p=N/A
1.00
1.00
1.00
1.00
1.00
Alignment
Aiignment
0.25
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
5
5
0
D
D,
D,
D2
D3
(c) SVCCA, Batch Size 512
U=0, p=-0.939
U=2, 0066.4
U=4, p=-0.309
U=6, p=-0.588
U=8, p=N/A
1.00
1.00
1.00
1.00
1.00
Alignment
0.75
0.75
Alignment
0.75
Alignment
0.75
Alignment
0.75
0.50
0.50
0.50
0.50
0.50
0.00
0.00
0.00
0.00
0.00
5
0
D+
D.
D.
D
D
(d) Mutual A-NN (k = 100), Batch Size = 512
Figure 15: Alignment vs heterogeneity with batch size 512. Spearman correlation coefficient is computed between the maximum alignment, shown in red, and heterogeneity. N/A denotes that p is undefined as all alignment values are O.
23

---

Understanding the Emergence of Multimodal Representation Aligument
U=0, p=-0.830
U=2 0m-033
Um4, pm-0.455
U=6, pw-0.758
U=8, p-N/A
1.00
1.00
1.00
1.00
1.00
Alignment
0.75
6000 4000
Alignment
0.75
Aligmment
0.75
Aignment
0.75
Alighment
0.75
0.50
0.50
0.50
0.50
0.50
0.25
600 6
0.00
5
5
0
De
D,
De
Dj
D2
(a) Unbiased CKA with Linear Kernel, Batch Size 1024
U=0, p=-0.891
U=2, p=-0.661
U=4, p=-0.442
U=6, p=-0.770
U=8, P=N/A
Alignment
0.75
00 00-000000
Alignment
0.75
Alignment
0.75
Alignment
0.75
Allinmient
0.75
0.50
0.50
0.50
0.50
0.50
0.25
0.25
60
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
Dy
D,
D
D₂
D₁
(b) Unbiased CKA with RBF Kernel, Batch Size 1024
U=0, 0=-0.673
U=2, 0=-0.794
U=4, p=-0.661
U=6, p=-0.552
U=8, p=N/A
1.00
1.00
1.00
1.00
1.00
Alignment
Alignment
Aiignment
0.25
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
5
5
0
D
D,
D
D2
D3
(c) SVCCA, Batch Size 1024
U= p=-0.818
U=2, 0m-6.4
U=4, p=-0.600
U=6, p=-0.442
U=8, p=N/A
1.00
1.00
1.00
1.00
1.00
Alignment
0.75
0.75
Alignment
0.75
Aignnent
0.75
Alignment
0.75
0.50
0.50
0.50
0.50
0.50
0.00
0.00
0.00
0.00
0.00
5
5
0
D+
D.
D.
D
D2
(d) Mutual j-NN (k 100), Batch Size = 1024
Figure 16: Alignment vs heterogeneity with batch size 1024. Spearman correlation coefficient is computed between the maximum alignment, shown in red, and heterogeneity. N/A denotes that is undefined as all alignment values are 0.
24

---

Understanding the Emergence of Multimodal Representation Aligument
0.8
0.8
Performance/eeph
0.6
Alignment/Depth
0.50-
0.4
0.2
0.2
0.0
0.0
-0.2
1
3
5
3
5
7
(a) Unbiased CKA with Linear Kernel, Batch Size = 256
0.8
1.00-
AlggmmentPerfomman
Perforaace//epptt
AlignmenUDapth
0.50-
0.4
0.25
0.2
0.00-
0.25
-0.2
-0.75-
(b) Unbiased CKA RBF Kernel, Batch Size 256
0.6
1.00
0.8
0.75-
NignmentPerrorman
0.4
Perforanne/Depphh
0.6
AllgnmenUDepph
0.50
0.4
0.00
0.2
-0.25
0.0
-0.50
-02
(c) SVCCA, Batch Size 256
Mignmmnt/Parfommanc
0.6
Perfomance/Depth
Aliggmenn/Depth
0.75
0.50
0.4
0.25
0.2
0.00
-0.25
0.0
-0.50-
-0.2
-0.75
1
i
3
i
i
3
5
i
(d) Mutual N-NN (A: 100), Batch Size 256
Figure 17: Alignment, performance, and depth correlation plots across different synthetic depths and experiment seeds for various representation similarity metries with batch size 256. In each plot, we show the spread of Spearman correlation coefficients, for each level of uniqueness.
25

---

Understanding the Emergence of Multimodal Representation Alignment
1.0
0.8
WignmnntPerfommann
0.8
Performance/eeph
0.6
Alignment/Dephh
0.5
0.4
0.0
0.2
0.0
0.0
0.2
-0.2
1
3
5
3
5
7
5
(a) Unbiased CKA with Linear Kernel, Batch Size = 512
0.8
Nignmentieeformanc
AiignmenJDaptth
Perforance//Deph
0.5
0.4
0.2
0.2
-0.5
-0.2
5
(b) Unbiased CKA RBF Kernel, Batch Size 512
NignmentPPrrorman nce
0.6
0.8
0.4
Perforann//Dephh
0.6
Alggmenn/Depth
0.5
0.2
0.4
0.0
0.2
0.2
0.0
-0.5
0.4
-02
(c) SVCCA, Batch Size 512
Perfomance//epth
Aliggmenn/Depth
0.75
0.50
0.25
0.00
0.25
-0.50
-0.75.
1
i
3
i
3
5
i
3
6
(d) Mutual N-NN (A: 100), Batch Size 512
Figure 18: Alignment, performance, and depth correlation plots across different synthetic depths and experiment seeds with batch size 512. In each plot, we show the spread of Spearman correlation coefficients p for each level of uniqueness.
26

---

Understanding the Emergence of Multimodal Representation Alignment
1.0
0.8
0.6
Performanc//eeph
0.6
Alignment/Dephh
0.5
0.4
0.4
0.2
0.0
0.2
0.0
-0.2
1
3
5
3
5
7
5
(a) Unbiased CKA with Linear Kernel, Batch Size 1024
0.8
1.0
lignmentiPeffrmand
0.6
Perforaace/DDepl
AlignmentDaptii
0.2
0.0
0.2
0.0
0.5
..2
5
(b) Unbiased CKA with RBF Kerne,, Batch Size 1024
1.00
0.8
0.75-
MignmentPPrrormanc
0.4
Perforanc//Depth
0.6
Alignnent/Depth
0.50
0.2
0.25
0.4
0.2
-0.25
-0.2
0.0
-0.50
-0.4
-0.75
-02
(c) SVCCA, Batch Size = 1024
0.8
iigrmmnn/Parfommanee
0.6
Perfomance/Dapph
Aliggmenn/DDeth
0.5
0.2
-02
-0.5
1
8
3
i
3
5
1
i
5
(d) Mutual k-NN (I: 100), Batch Size 1024
Figure 19: Alignment, performance, and depth correlation plots across different synthetic depths and experiment seeds with batch size 1024. In each plot, we show the spread of Spearman correlation coefficients p for each level of uniqueness.
27

---

Understanding the Emergence of Multimodal Representation Alignment
D.2. Synthetic Data Results with Different Ej depths
In Figures 20 and 21, provide additional experiment results showing that our results are not significantly changed when increase the depth of E to and 3. Because E₁ is trained on the untransformed modality, Ej will remain relatively easy to optimize even as the depth increases.
D,=
p=-0.950
D,
p=-0.833
5, pm-0.983
De=7,
p -0.983
D,
p=-1.000
1.0
1.0
1.0
1.0
1.0
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
000
0.4
0.4
D.4
0.4
0.4
0.2
0.2
0.2
0000
0.2
0.2
0.0
0.0
0.0
0.0
0.0
2.3
7.5
2.5
7.5
2.5
7.5
2.5
5.0
7.5
0.0
2.5
5.0
7.5
U
U
U
U
U
(a) Depth E))
D,=1, p=-0.983
D,=3, p=-0.850
D,=5, p=-0.983
D,=7, p=.0.983
D,=9, p=.0.983
Alignment
@00
Allgnment
Alignment
Alignment
Alignment
0.6
00
0.4
0.4
0.4
69
000
2.5
5.0
7.5
2.5
5.0
7.5
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
Depth E
D,=1,
p=-0.950
D,=3,
p=-0.850
D,=5,
p=-0.983
De=7, 0=-0.983
D,=9,
p=-1.000
1.0
1.0
1.0
1.0
1.0
Alignment
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0000
0.2
0.2
0.0
0.0
0.0
0.0
0.0
2.5
7.5
2.5
7.5
0.0
2.5
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
U
U
U
U
U
(c) Depth 3 Fi
Figure 20: Alignment vs uniqueness for various depths of E,. The distribution of alignment scores for various depths of Ej are nearly identical, as single level neural network is sufficient to model the untransformed modality.
28

---

Understanding the Emergence of Multimodal Representation Alignment
0.8
Aligmme Senmenn//erfommaasttt Pformance
0.6
Performance/Depth
Alignment/Depth
0.2
0.0-
0.2
0.0
-0.5
-0.4
1
3
5
1
3
7
1
3
5
(a) Depth E1
0.8
1.0
0.8
Performance e/Depph
0.6
Alignment/Depth
0.4
0.4
0.0
0.2
0.0
0.0
0.4
-02
1
1
3
(b) Depth 2 E
1.0-
0.8
WinneettPrrfmmanc
0.6
Perforaace/Despth
0.6
Alignment/Depth
0.5
0.4
0.2
0.2
-0.2
0.0
-0.4
0.
5
(c) Depth 3 F
Figure 21: Alignment, performance, and depth correlation plots across different synthetic depths and experiment seeds for various depths of E; In each plot, we show the spread of Spearman correlation coefficients for each level of uniqueness.
29

---

Understanding the Emergence of Multimodal Representation Aligument
D.3. Randomly Initialized Neural Networks Alignment
In Figure 22, we plot the alignment of randomly initialized neural networks. The alignment constant for all levels of uniqueness, except for when the dataset is fully unique. In Figure 23, show that for randomly initialized neural networks, alignment, performance, and depth do not correlate with each other.
D,=1, p.1.5
3,
3,
p=-0.450
5
p-0.000
De
p -0588
D
pm-0.433
1.0
1.0
1.0
1.0
1.0
Alignment
0.8
Alignmient
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
D.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.0
2.3
7.5
0.0
0.0
2.5
7.5
0.0
0.0
2.5
7.5
0.0
0.0
2.5
5.0
7.5
0.0
0.0
2.5
5.0
7.5
U
U
(a) Unbiased CKA with Linear Kernel, Randomly initialized neural networks
D,=1, p=0.133
Dg=3, 0=-0.450
Dx=5,
p=0.033
Dj=7, p=.0.783
D,=9, p=.0.500
1.0
Alignment
0.8
Alignment
0.8
Alignment
Alignment
Alignment
0.6
0.4
2.5
5.0
7.5
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
Unbiased CKA with RBF Kernel, Randomly initialized neural networks
D,=1,
p=-0.433
D,=3,
p=-0.467
D,=5,
p=0.067
Dx=7,
p=-0.233
9,=9,
p=-0.567
1.0
1.0
1.0
1.0
1.0
Alignment
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
Alignment
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
2.5
7.5
2.5
7.5
0.0
2.5
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
U
U
U
U
(c) SVCCA, Randomly initialized neural networks
D,=1, 0=-0.017
D==3,
0=-0.367
D,=5,
p=0.033
Dø=7,
0=511
Dj=9, 0=
1.0
1.0
1.0
Ailgnment
Ailgnment
Alignment
Nignment
Ailgnment
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
(d) Mutual I-NN (k: 100), Randomly initialized neural networks
Figure 22: Alignment uniqueness with randomly initialized neural networks. Spearman correlation coefficient p computed between the maximum alignment, shown in red, and the level of informational uniqueness U.
30

---

Understanding the Emergence of Multimodal Representation Alignment
0.75
Wiggmmnn/PPrformaat
0.2
Perfomance/Depti
0.2
0.50
Alignmen//Depi
0.5
0.0
0.00
-0.2
-0.2
-0.50-
0.4
1
3
(a) Unbiased CKA with Linear Kernel, Randomly initialized neural networks
0.3
0.75
0.2
Perfomance/Degtt
0.2
Alinnent/Deati
0.50-
0.25
0.0
0.0
0.00
0.1
-0.25
-0.50-
-0.3
-0.4
(b) Unbiased CKA with RBF Kernel, Randomly initialized neural networks
WiggnmentPPrrrmmaa nce
0.2
Perforance/Depph cD/epth
0.1
Aignment/Depti
0.0
0.25
0.0
0.00
0.2
-0.2
0.3
-0.50-
-0.4
0.75
(c) SVCCA, Batch Size 512, Randomly initialized neural networks
0.3
0.2
Performace/Depph
Diggmenn/Depth
0.0
0.00
-0.5
-0.1
-0.2
-0.50
-0.75
0.3
1
3
1
3
5
-1.00
1
6
(d) Mutual k-NN (k 100), Randomly initialized neural networks
Figure 23: Alignment, performance, and depth correlation plots across different synthetic depths with randomly initialized neural networks. each plot, we show the spread of Spearman correlation coefficients p for each level of uniqueness.
31

---

Understanding the Emergence of Multimodal Representation Alignment
D.4. Synthetic Data Siggnment-Performaant Results
In Figures 24, 25, and 26, we plot the relation between alignment and performance for individual synthetic datasets, which show that as uniqueness increases, alignment is no longer indicator of performance.
D D==3
U=0, p=0.638
U=1, p=0.790
U=2, p=0.329
0.8
10
10
Alighment
0.8
8
Alignment
8
Alignment
0.6
8
0.6
6
Depti
0.6
6
Depii
0.5
6
0.4
0.4
0.4
0.3
0.2
0.86
0.88
0.90
0.86
0.88
0.90
0.70
0.72
Performance
Performance
Performance
U=3, p=0.469
U=4, 0=2.207
U=5, p=0.000
10
10
10
0.26
0.4
Alignment
0.4
Alignment
0.24
Alignment
Depti
pep
0.3
Depth
4
0.3
0.20
2
2
0.2
0.18
0.72
0.74
0.68
0.70
0.68
0.70
0.72
Performance
Performance
Performance
U=6, 0=-0.192
U=7, 0.2221
U=8, 000000
10
0.45
10
0.03
10
0.4
Alignment
8
Alignment
0.40
8
Alignment
0.02
8
0.3
Deptf
Depth
Depth
0.35
0.2
4
0.30
0.01
0.1
2
0.25
2
0.00
0.64
0.65 Performance
0.62
0.63
0.64
0.58
0.59
0.60
Performance
Performance
Figure 24: Alignment Performance for D3 3. The alignment-performee trend is shown across different levels of uniqueness, with the Pearson's 's correlation coefficient reported for each plot.
32

---

Understanding the Emergence of Multimodal Representation Alignment
D
U=0, p=0.610
U=1, p=0.680
U=2, p=0.499
0.8
10
10
10
Alignment
0.6
Alignment
Alignment
0.6
Depai
dep
Depti
0.4
4
4
4
0.2
0.84
0.86
0.88
0.86
0.88
0.90
0.72
0.73
Performance
Performance
Performance
U=3, p=0.329
U=4, p=0.215
U=5, p=0.419
10
10
10
Alignment
0.4
Alignment
0.40
Alignment
0.35
Depth
0.35
pepth
0.30
Depth
0.3
4
0.30
-4
0.25
0.20
2
0.25
2
0.2
0.15
0.70
0.71
0.72
0.73
0.68
0.69
0.70
0.68
0.70
0.72
Performance
Performance
Performance
U=6, 0-3.364
U=7, 000.0
U=8, p=20.2
10
0.35
10
10
0.020
Alignment
8
Alignment
0.30
Alignment
0.015
Depth
Depth
Dept
0.25
0.010
0.20
0.005
2
2
2
0.15
0.000
0.62 0.63
0.64
0.62
0.63
0.64
0.65
0.57
0.58
0.59
Performance
Performance
Performance
Figure 25: Alignment vs Performance for D3 = 6. The alignment-performancc trend is shown across different levels of uniqueness, with the Pearson's 's correlation coefficient reported for each plot.
33

---

Understanding the Emergence of Multimodal Representation Alignment
99=9
U=0, p=
U=1, p=0.561
U=2, p=
10
0.7
10
Algnmmett
0.7
Alignment
Alignment
Depth
0.6
6
Depli
0.5
Depth
0.6
-4
0.5
4
0.5
0.4
2
0.4
0.80
0.85
0.80
0.85
0.675
0.700
0.725
Performance
Performance
Performance
U=3, p=0.582
U=4, 0=0.700
U=5, p=0.146
10
0.35
10
10
0.30
Alignment
0.3
Alignment
0.30
8
Alignment
0.25
Depth
0.25
Depth
Depth
0.20
0.2
4
0.20
4
0.15
0.15
2
2
0.675
0.700
0.725
0.65
0.70
0.65
0.70
Performance
Performance
Performance
U=6, 0.354
U=7, 0=0.165
U=8, p= 0.129
10
0.20
10
0.0125
10
Alignment
0.25
Depth
Alignment
0.15
8
Depth
Alignment
0.0100
8
Depth
0.20
0.0075
0.10
0.15
0.0050
2
2
0.05
0.55
0.60
0.575
0.600
0. 0.625
0.525
0.550
0.575
Performance
Performance
Performance
Figure 26: Alignment vs Performance for D3 = 9. The perfornnt-eeformmaccee trend is shown across different levels of uniqueness, with the Pearson's 's correlation coefficient reported for each plot.
34

---

Understanding the Emergence of Multimodal Representation Aligument
D.5. Vision-Language Alignment Unique
In Figure 27, we plot the relation between vision-language alignment and uniqueness, which shows that the maximum alignment decreases with uniqueness.
D.6. Vision-Language Alignment Performance
In Figure 28, plot the relation between vision-language alignment and performance for various vision and language models.
D.7. perfonnent relation on MM-IMDb
MM-IMDD (Arevalo al., 2017) is dataset for classifying movie genres from movie posters and description of the movie plot, where there are 23 classes. As such, consider 23 binary classification tasks. We compute cross-modal alignment between the same vision models and language models as Huh et al. (2024) using subset of 1024 points. To obtain classification performance for each movie genre, we train linear classifiers using the last layer hidden representation of the language models. We compute the linear fit to Intuitively, as the text describes the plot of the movie, we expect that the text modality provides many degrees of unique information compared to the image. However, not of the additional information provided by the text would be useful the given classification task, and thus the relation between alignment and performance would vary for different genres. Our analysis in Figure 29 reveals that the linear fit slopes vary depending on the movie genre. Larger linear fit slopes peeenmeet-erffommncce performance scores suggest that aligning modalities is more helpful.
scores for each downstream classification task.
alignment-
35

---

Understanding the Emergence of Multimodal Representation Alignment
DINOV2, p=-0.950
ImageNet21k, p=-0.958
0.16
small base
tiny small base large
0.12
0.14
giant
Alignment
0.12
Alignment
0.10
0.10
0.08
0.08
0.06
0.06
20
40
60
80
100
20
40
60
80
100
Unique
Unique
CLIP, p=-0.953
CLIP (I12K ft), p= -0.952
base
base large
larga huge
0.16
0.14
huge
0.14
Alignment
Alignment
0.12
0.12
0.10
0.10
0.08
0.08
0.06
0.06
40
60
100
40
60
100
Unique
Unique
MAE, p=-0.947 0.947
0.09
base huge
0.08
Alignment
03
0.07
0.06
0.05
0.04
20
60
80
100
Unique
Figure 27: Vision-Language Alignment vs Uniqueness. The alignment is computed between various vision models and large language models. We compute the Spearman correlation coefficient, p between the maximum alignment and uniqueness.
36

---

Understanding the Emergencc of Multimodal Representation Alignment
U=10, :lope=0:112, pm0.852
U=30, slope=0.J 00100. pm0.951
U-50, 3lope=0.071, p00.900
Um700 slope=0.065, p=0.000
(/=90, 0000.s 0204 pm0.612
xra
Algmmennte DWNV3
Algomenc to DROX2
Alggmment DINO2
laga
Algnment DNDx3
Iaga
012
0.1
0:22
0.30
0.0
0.10
0.30
6.30
0.08
0.2
0.5
0.08
0.08
mencormance
Performance
Performance
Performance
Performance
(a) DINOv2
U0 dlop=-0,099, p-0.899
U-30, #0p8-0.081, p-0.974
U=50, olope-0.055, 956
U-70, wwopa- p-0.794
Uw90,
p=0.503
Algnmort ImsgoNo220
0.32
Algnmerk to ImagoNuczik
0.12
Allgmmers ImageNuz21
032
Aligmmett Imagalae221
0.02
Aignmert to ImageNoczIk
2.12
legs
0.c0-
0.00
0.0
0,08
8.00
-1.6
-1.5
144
-15
-144
-1.
-15
-1.4
-1.3
-15
14
-33
-1.6
-3
Performance
Performance
Parformance
Parformace
Performance
(b) ImageNet21K
U- dlope=0.118, p=0.0.0.
U=30, 0.1114 pm8.893
V=50, 0l1pe=0.102, p00.060
U=70, slope=0.00 p=0.
U-90, slope=0.p...ooo 028, pm0.690
Algnmennt CUP
rage
Algomment CUPP
Algnmentto CUP
Puga
Algmmmntt
2.12
9.30
0.6
0.08
0.0
08c
5,56
5.06
-3.6
-3.5
-14
-15
-1.4
-1.3
-15
-1.4
-1.4
-1.3
-3.4
-12
Performance
Performance
Performance
Performanc
Performance
(c) CLIP
U- slope p-0.875
U-30, -lop--0.100 p-0.971
U-50, slape -0,000, p-o 0.3
U-70, olepa-0.067, p-0.892
(/-90, www.ssl.ppp...m -00000,,, p=0.517
to 1112X 73
0.16
kec
to CUP 1122 79
0.14
Signnenn to CUP 1122 E3
0.14
texr
Algnment CuP m12K
0.14
ruge
teee
to CUP 1122
0.14
Iuge
0.20
012
010
0.30
6.30
0.00
0.05
0.05
2:06
6.00
Performance
Performance
Performance
Performnne
Performance
(d) CLIP finetuned on ImageNet-21kk
0l00e=0.037,
200.7,p=000000 p=0.849
slope=0 0.042, p=0.9
U=50, slope=0.000 pm9.934
slope=0.033 p=0
U=90, slopemo.p.oooomm p=0.134
0.05
lase
0.00
WFnmmntt to MAE
0.02
Afgomenn MAE
0.05
2.56
Algoment to MAE
5.08
0.57
0.7
0.07
0.07
0.07
Aligmment
0.05
0.55
0.05
3:05
6.05
0.04
0.4
0.04
0.34
0.04
-3.6
-3.5
-14
-15
-1.4
-1.3
-35
-3.4
-1.3
-355
-3A
-3.3
-3.4
-13
-13
Performance
Porforma ncb
Performance
Performance
Performance
(c) MAE
Figure 28: Vision-Language Alignment vs Performance. We plot the vision-language alignment using various vision models with respect to language model performance, measured using bits-per-byte-los and show individual best fit lines for each size of vision model as well as the average Spearman correlation coefficient, p. As U increases, the relation between alignment and performance weakens.
37

---

Understanding the Emergence of Multimodal Representation Alignment
ImageNiel21k
T
T
Frecuuency
Frucusencyy
205
0.00
0.05
0.10
0.5
-0.25
5.00
5.55
0.10
0.55
-0625
3.200
0.525
0.000 5.075
0.02 0.52
0.24 0.55
-0.01
0.50
0.05
0.10
Ritedd Stopes
Fittd Slepes
Fitted Siepes
Aiteed Stppes
Fitted Slapes
Figure 29: Relationship between alignment and performances set of language and vision models Huh al. (2024), evaluate cross modal alignment MM-IMDb (Arevalo et al., 2017), dataset for movie genre prediction, where we consider two modalities: images of the movie poster and text of the plot descriptions. The task is multi-label classification, and we consider each cateogry as a separate binary classification task. To measure performance for each language model, we train a linear classification layer on the last layer hidden performance scores across categories, which varies due
MM-IMDD classification tasks. Using the
representations. We plot the slope of the linear to the different levels of information content required for each downstream classification task.
38


Reviews:
[{"review_id": "Jwm3gwDR4h", "reviewer": "Reviewer_B2rQ", "summary": "This paper empirically investigates when and why implicit alignment emerges, and whether alignment consistently predicts task performance, finding that both depend critically on modality similarity and the redundancy or uniqueness of the information provided.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive criticism and are glad that they find our analysis interesting. Below we address the reviewer’s questions and concerns.\n\n**Under “Claims And Evidence”:**\n\n> Is the used metric, HSIC, sufficient to reflect the alignment quality? … is highly sensitive to the chosen kernel, sample size and other hyper parameters. \n\nWe believe that the HSIC metric is sufficient to capture alignment quality, as we use a specific linear kernel consistently across all experiments. Moreover, this kernel has no hyperparameters, making it the simplest choice. To verify the robustness of our results, we also evaluate them using alternative alignment metrics. We perform additional experiments on the synthetic data with additional alignment metrics and with different sample sizes which demonstrate that our findings are robust to hyperparameters and consistent across different metrics. Specifically, we report results with unbiased CKA with a linear kernel (our original alignment metric), unbiased CKA with a RBF kernel, Mutual KNN [2], SVCCA [3] and run all metrics with 256, 512 (our original sample size), 1024 data points. We report our [updated results here](https://tinyurl.com/yfzwzs2s). For all metrics and batch sizes, the maximum alignment decreases with increasing uniqueness as well as increasing heterogeneity. Additionally,  the relation between alignment, performance, and depth are consistent across different batch sizes. Across all alignment metrics, performance and depth are positively correlated over different uniqueness values, whereas alignment and performance as well as alignment and depth correlations can be weak or negative for increased uniqueness, indicating that our findings are robust to different kernels and alignment metrics.\n\n[1] Kornblith et al. “Similarity of Neural Network Representations Revisited” (2019).\n[2] Huh et al. \"The platonic representation hypothesis\" (2024).\n[3] Raghu et al. “SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability” (2017).\n\n**Under “Other Strengths and Weaknesses”:**\n\n> No new method is proposed, which limits the contribution of this paper.\n\nWhile our paper does not propose a new method, we believe that our contributions are significant -- see our response to reviewer obVJ for a more in-depth discussion.\n\n> Most experimental analysis is based on the synthetic datasets … rather than a subset of MultiBench to study Emerge property.\n\nWe would like to emphasize that MultiBench datasets (used in Section 6) are real-world, with CMU-MOSEI and UR-FUNNY containing 22k and 16k video snippets respectively. In addition, we provide new experiment results on MM-IMDb and improved our experiments on the Wikipedia-Image Text dataset. We use GPT-4 to synthesize text captions with unique information that is not present in the images, ensuring that the resulting datasets retain the semantics of real-world text and images. Our findings support our paper’s key claims (see response to obVJ for more details).\n\n> It would be interesting to see the quantification of the uniqueness level in the real-world dataset.\n\nWe agree that quantifying uniqueness is an interesting direction, and our results have shown the potential for alignment-performance correlation to be used for quantification. While different pairs of modalities have varying levels of heterogeneity, which can make it difficult to quantify uniqueness across datasets, we propose that alignment-performance correlations can quantify information content between different downstream tasks within a given multimodal dataset. We present new results on MM-IMDb [1], a dataset for classifying movie genres with image and text modalities. Each movie can be labeled with 1 or more genres, and there are 23 classes. We compute cross-modal alignment using various vision models and language models. To measure performance, we train linear layers on the last layer hidden representations of the language models, resulting in F1-scores for each class. [Our results](https://tinyurl.com/5my72nnb) demonstrate that the relation between alignment and performance varies depending on the classification task — we see that the slope of the linear fit to alignment and performance scores is weak or even negative, suggesting that for certain movie genres, there is greater task relevant information that is unique to the language modality. \n\n[1] Arevalo et al. “Gated multimodal units for information fusion” (2017).\n\n> [1] proposes that different random initializations could also cause a modality gap. Will this affect the conclusion of this paper?\n\nWe would like to clarify that our experiments on synthetic data are run with 5 different seeds, and for our experiments on MultiBench datasets, we compute alignment-performance correlation over 3 seeds. Hence, we believe that our results are robust to initializations.", "claims_and_evidence": "The analysis that is conducted highly depends on the alignment quantification.  \nIs the used metric, HSIC, sufficient to reflect the alignment quality?  Such kernel-based metric is highly sensitive to the chosen kernel, sample size and other hyper parameters. If the metric cannot truly reflect the alignment level, the experiments, like Emergence of alignment across heterogeneity and uniqueness, are questionable.", "methods_and_evaluation_criteria": "No new method is proposed.", "theoretical_claims": "No theoretical claim.", "experimental_designs_or_analyses": "The experimental analysis of the synthetic dataset is interesting. However, more comprehensive experimental analysis should be performed on a large scale real-world dataset rather than a subset of MultiBench to study Emerge property.", "relation_to_broader_scientific_literature": "The results are relevant to the analysis of multimodal alignment.", "essential_references_not_discussed": "Most relevant papers are discussed.", "other_strengths_and_weaknesses": "The motivation for analyzing the alignment is interesting.\n \nWeakness:\n1. Is the used metric, HSIC, sufficient to reflect the alignment quality?  Such kernel-based metric is highly sensitive to the chosen kernel, sample size and other hyper parameters. If the metric cannot truly reflect the alignment level, the experiments, like Emergence of alignment across heterogeneity and uniqueness, are questionable. \n2. No new method is proposed, which limits the contribution of this paper. \n3. Most experimental analysis is based on the synthetic datasets, which is not convincing. More comprehensive experimental analysis should be performed on large scale real-world dataset, rather than a subset of MultiBench to study Emerge property. \n4. It would be interesting to see the quantification of the uniqueness level in the real-world dataset. \n5. [1] proposes that different random initializations could also cause a modality gap. Will this affect the conclusion of this paper?\n\n\n[1]. Liang, Victor Weixin, et al. \"Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 17612-17625.", "other_comments_or_suggestions": "see my weakness.", "questions_for_authors": "see my weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tmlYZ2l8zl", "reviewer": "Reviewer_obVJ", "summary": "This paper mainly focus on analyzing the emergence of multimodal representation alignment. Alignment between cross-modal representations has been long regarded as an important factor of improving multimodal model performance. Some recent researches have found that independently trained unimodal models can be implicitly aligned. The authors aim at finding out the opportunity and reasons for alignment emergence, and whether such alignment is a indicator of performance. Through comprehensive synthetic and real-world dataset experiments, the author reach to several conclusions. 1. The alignment may not be universally beneficial. 2. Such alignment impacts on performance differently among datasets and tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review. Below we address their questions and concerns.\n\n**Under “Claims and Evidence”:**\n>  My major question concerns the heuristic of this paper on modern multimodal model design … which however is less discussed thoughout the paper.\n\nTo the best of our knowledge, our analysis of the emergence of alignment across the dimensions of uniqueness and heterogeneity is novel and fills an important gap in the literature on cross-modal alignment. While prior work—such as the Platonic Representation Hypothesis [1]—suggests that alignment tends to emerge with increasing data scale and serves as an indicator of good performance, these claims have not been rigorously examined across key characteristics of multimodal data. In this paper, we critically evaluate these assumptions and argue that while alignment may indeed correlate with performance in settings where modalities share high redundancy (i.e., low uniqueness), this relationship breaks down when the modalities are more distinct. In such scenarios, increased alignment does not necessarily translate to better downstream performance. We believe this insight is not only novel but also practically useful, as it encourages practitioners to reconsider alignment strategies in cases where they may be counterproductive.\n\nWe additionally explore the application of alignment-performance correlation for quantifying the information content of downstream tasks. Specifically, we present results on MM-IMDb [1], a dataset for classifying movie genres with image and text modalities. [Our results](https://tinyurl.com/5my72nnb) demonstrate that the relation between alignment and performance varies depending on the classification task (see response to B2rQ for more details), which can inform practitioners when aligning modalities is beneficial. \n[1] Arevalo et al. “Gated multimodal units for information fusion” (2017).\n\n**Under Methods And Evaluation Criteria:**\n> This paper is mainly analytical without giving a method. … However, in Fig. 4 and 5, the notation  is never introduced before.\n\nThank you for the feedback. We will update our paper to define Alignment as unbiased CKA and Unique as the number of unique features used in computing the label.\n\n**Under Experimental Designs Or Analyses:**\n\n> I appreciate the designs of the synthetic experiments. The uniqueness assessment and label generation are reasonable. However, the experimental setup for real benchmark seems to be unaligned with the problem settings. … the conclusion from real-world experiments may not be convincing. \n\nWe agree that our method of perturbing the Wikipedia caption dataset is not fully aligned with our definition of uniqueness. To ensure that the perturbed dataset retains the semantics of real-world text and images, we provide new experiment results that leverage GPT-4 to synthesize text captions with unique information that is not present in the images. We keep the original image data without any additional noise. We upload our perturbed text data and code for generating the perturbations [here](https://tinyurl.com/8vxt9hby). For each (image, text) pair in the original dataset, we prompt GPT-4 to produce 10 captions with increasing levels of uniqueness: 10%, 20%, … 100%, such that the final caption contains only information that is unique to the text. As uniqueness is already introduced in the text, we keep the original images in the Wikipedia caption dataset. Using a pretrained sentence BERT model to quantify semantic similarity between the original caption and the GPT-4 captions, we find that the average semantic similarity monotonically decreases as the level of uniqueness increases. We compute the alignment between various types of vision models and LLMs. Our updated results support both claims: 1) The maximum alignment decrease with increased uniqueness. [see figure here](https://tinyurl.com/yeezwdxu) and 2) The slope of the fitted line to the alignment and performance scores decreases with increased uniqueness, showing that the relation between alignment and performance weakens. [see figure here](https://tinyurl.com/4ynj2td3).\n\n> I am also concerned about the setting of unsymmetrical encoders in line 190-195 … while the optimization of $E_2$ can be much harder.\n\nWe provide additional experiment results demonstrating that our results are unchanged when $E_1$ is a higher depth. [Here](https://tinyurl.com/4uwe4zkm), we change the depth of $E_1$ to 2 and 3 and find that the results are not significantly changed. We hypothesize that because $E_1$ is trained on the untransformed modality, $E_1$ will remain relatively easy to optimize even as the depth increases. We will include these results in our updated paper.\n\n**Under Other Strengths And Weaknesses:**\n\n> While the paper is written straight-forward, here are several weaknesses in the writing that should be improved. \n\nThank you for the feedback. We will revise our paper accordingly.", "claims_and_evidence": "The authors mainly discuss the emergence of implicit alignment in multimodal training. There are basically following claims in this paper: 1. Under low uniqueness, the alignment is significantly correlated with performance and model capacity. However, when uniqueness increases, such relationship becomes much weaker. 2. Alignment alone is not a sufficient predictor of model performance especicially in multimodal settings of uniqueness and heterogeneity. \n\nAlthough I have several concerns about the experimental designs and their support to the final conclusions, I generally agree with the claims within the paper. My major question concerns the heuristic of this paper on modern multimodal model design. Although the authors provide detailed experiments and analysis, the conclusions seem to be obvious and intuitive. The observation that alignment is less related to performance when uniqueness and heterogeneity increase is not much novel. In contrast, I am more curious about how such conclusions can impact on the design of modern models on various datasets or down-stream tasks, which however is less discussed thoughout the paper.", "methods_and_evaluation_criteria": "This paper is mainly analytical without giving a method. The evaluation metrics of the paper, for example, CKA and uniqueness is reasonable. However, in Fig. 4 and 5, the notation $r$ is never introduced before.", "theoretical_claims": "This paper is mainly analytical without giving theoretical claims.", "experimental_designs_or_analyses": "I appreciate the designs of the synthetic experiments. The uniqueness assessment and label generation are reasonable. However, the experimental setup for real benchmark seems to be unaligned with the problem settings. Accordingly, on the Wikipedia caption dataset, the uniqueness of text and image data is implemented by random deletion and Gaussian perturbation, which is actually injetced noise. Such design seems to be opposite to the definition of uniqueness in line 155-162 that \"Uniqueness in modality quantifies the amount of information present in the first modality absent in the second but critical for the downstream task\", since noise can not be crucial. Thus the conclusion from real-world experiments may not be convincing. I am also concerned about the setting of unsymmetrical encoders in line 190-195, that $E_1$ is simply a single-layer encoder while $E_2$ is a deep encoder of varying depth. While I've noticed the second modality is simulated by a nonlinear transformation, such design can lead to issue that $E_1$ can easily learn a good representation while the optimization of $E_2$ can be much harder.", "relation_to_broader_scientific_literature": "Please refer to the former parts of the review, the impact of alignment conclusions on the design of modern models on various datasets or down-stream task is less discussed in this paper.", "essential_references_not_discussed": "No other related works need to be mentioned.", "other_strengths_and_weaknesses": "While the paper is written straight-forward, here are several weaknesses in the writing that should be improved. For instance, the introduction of CKA is overlength in the second page. Since this is a contribution of previous works, details of this part is recommended to be placed in the Appendix. The explanation of $x_r$ and $x_u$ is duplicate in line 207-209 with line 171-176. The bijection $\\phi$ in line 212 has also be introduced before.", "other_comments_or_suggestions": "No other suggestions.", "questions_for_authors": "Please refer to the former parts. My major concerns include the novelty of the conclusions on implicit alignment, the impact of such conclusions on modern multimodal model designs and experimental designs.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLpec6YryJ", "reviewer": "Reviewer_wgD1", "summary": "This paper presents an empirical investigation of alignment between models with possibly different architectures and trained over different modalities. The authors investigate under which conditions the so-called Platonic Representation Hypothesis is likely to arise based on the heterogeneity of the data modalities and uniqueness in information. The empirical findings over multiple simulated data and a multi-modal benchmark reveal that alignment is not necessarily correlated to an increase in model performance, hence establishing that alignment between models can arise only under specific experimental conditions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer and are glad that they find our experimental evidence convincing. Below we address the reviewer’s comments and questions.\n\n**Under “Claims and Evidence:”**\n\n> It remains open whether and how models trained on larger datasets (consisting of multiple degrees of uniqueness) can express a higher degree of alignment. \n\nWe present new results on MM-IMDb [1], a dataset for classifying movie genres with 25k paired images and texts. [Our results](https://tinyurl.com/5my72nnb) demonstrate that the relation between alignment and performance varies depending on the classification task (see response to B2rQ for more details), suggesting that the degree of alignment depends significantly on the downstream task.\n\n[1] Arevalo et al. “Gated multimodal units for information fusion” (2017).\n\n\n**Under “Experimental Designs Or Analyses:”**\n\n> I focused more on synthetic experiments to understand the core message there. … Would it have been sensible to have it also for the first modality?\n\nWe acknowledge that there are many ways of defining heterogeneity, however, a benefit of leaving the first modality untransformed is that the representation that $E_1$ learns is an ideal one -- because it has a direct linear relationship with the labels -- aligning with this ideal representation could imply that the model has learned something really universal: a requirement for the Platonic hypothesis. Then if  $E_2$’s representation is highly aligned with $E_1$, we can infer that $E_2$ has learned to recover information that is comparable to the untransformed modality. Nevertheless, transforming both modalities may yield insightful results, and we leave the exploration of different types of heterogeneity to future work.\n\n> I suggest including random baselines when alignment is measured and plotted (RQ1).\n\nWe have run experiments computing alignment between randomly initialized neural networks [here](https://tinyurl.com/5fwdecad). Results confirm that the alignment of these neural networks is constant with respect to uniqueness and that there is no correlation between alignment and performance  on average.\n\n**Under “Relation To Broader Scientific Literature:”**\n\n> Understanding the datasets shared information or uniqueness is something relevant also in Continual Learning [1]. \n\nThank you for bringing up these related works. We will include a discussion in our updated paper.\n\n**Under “Other Comments Or Suggestions:”**\n> One minor note: I do not entirely understand how the bottom part with triangles and circles should be interpreted.\n\nIn Figures 1, the triangles and circles represent data from different modalities. In Figure 2, the triangles (and other shapes) also represent data from different modalities with varying degrees of heterogeneity. We will clarify this in the final version of our paper. \n\n> There is a repetition at the beginning of section 3, from line 118 onwards. The same sentence appears in line 65.\n\nThank you for pointing this out. We will remove the redundancy.\n\n**Under “Questions For Authors:”**\nAbout real-world experiments:\n\n> How do you measure uniqueness for MOSEI, MOSI, URFUNNY, etc? Not entirely clear how this is evaluated from human annotation\n\nWhile we do not rely on exact estimates of uniqueness for the MultiBench datasets, past work [1] has sampled several data points and asked human annotators to rate the redundancy and uniqueness of each example. These ratings are shown to agree with computational estimates of redundancy and uniqueness for various MultiBench datasets.\n\n[1] Liang et al. “Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework” (2024)\n\n> How is it the case that perturbations of the input correspond to changing uniqueness? … Is it sensible to expect there, where much information is distorted, to see any alignment at all?\n\nWe agree that our method of perturbing the Wikipedia caption dataset is not fully aligned with our definition of uniqueness. Hence, we provide new experiment results on MM-IMDb and improved our experiments on the Wikipedia-Image Text dataset. We use GPT-4 to synthesize text captions with unique information that is not present in the images, ensuring that the resulting datasets retain the semantics of real-world text and images. Our findings support our paper’s key claims (see response to obVJ for more details).\n\n> Lines 291-300 are not clear about the upper limit for alignment. How is this tested or referenced?\n\nWhile the theoretical upper bound for alignment, based on the HSIC metric, is 1, our empirical results (Figures 4 and 5, indicated by the red dot) show that the observed upper limit is significantly lower. We therefore conjecture that the maximum achievable alignment is constrained by the amount of shared information between the two modalities. We acknowledge that this argument is not rigorously formalized, and we will clarify this point in the updated version of the paper.", "claims_and_evidence": "The main claim is that the two axes the authors have proposed to measure, namely uniqueness of information and heterogeneity, are responsible for more or less alignment in trained models. This is an interesting proposal that relates to other studies in catastrophic forgetting in continual learning, see e.g. [1]. Authors provide evidence of this relation in several synthetic datasets, generated according to these two axes of variation, and on real-world data and baselines. \nThe main investigation is to uncover if alignment correlates with model performance and scale. Overall, the evidence supports the claims that this depends on uniqueness and the modality gap. \n\nIt remains open whether and how models trained on larger datasets (consisting of multiple degrees of uniqueness) can express a higher degree of alignment. This is a more challenging scenario to test, worth spelling out in the conclusions.", "methods_and_evaluation_criteria": "The methods and evaluation are clear for the synthetic experiments. \nI struggled a bit to understand how authors investigate the real-world datasets and models. I require further clarifications from the authors that can help in reading and assessing the quality of their evaluation, see questions. \nOverall, I'm leaning positively towards the analysis the authors conducted.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I focused more on synthetic experiments to understand the core message there. Why do the authors choose a non-linear transformation only for the second modality? Would it have been sensible to have it also for the first modality?\n\nI suggest including random baselines when alignment is measured and plotted (RQ1).", "relation_to_broader_scientific_literature": "Understanding the datasets shared information or uniqueness is something relevant also in Continual Learning [1]. There, this information can lead to more or less catastrophic forgetting. Also, the multimodal setup where representations are compared resembles theoretical works on identifiability for the case of independent component analysis [2]. This connection can be helpful for new theory-oriented works.\n\n\n[1] Toward Understanding Catastrophic Forgetting in Continual Learning, Nguyen et al. (2019)\n[2] The Incomplete Rosetta Stone Problem: Identifiability Results for Multi-View Nonlinear ICA, Gresele et al. (2019)", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Figures are helpful to understand the message.", "other_comments_or_suggestions": "One minor note: I do not entirely understand how the bottom part with triangles and circles should be interpreted.\n\nThere is a repetition at the beginning of section 3, from line 118 onwards. The same sentence appears in line 65.", "questions_for_authors": "About real-world experiments:\n1) How do you measure uniqueness for MOSEI, MOSI, URFUNNY, etc? Not entirely clear how this is evaluated from human annotation\n2) How is it the case that perturbations of the input correspond to changing uniqueness? This aspect is not clear and is a bit toy. There is the risk that perturbed strings and images can create out-of-distribution inputs for both vision and language models. Is it sensible to expect there, where much information is distorted, to see any alignment at all?\n3)  Lines 291-300 are not clear about the upper limit for alignment. How is this tested or referenced?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SkTQSa30Gl", "reviewer": "Reviewer_nSuf", "summary": "This paper aims to understand the properties under which alignment emerges in multi-modal models. Specifically, they studied the influence of the data similarity (heterogeneity, i.e., how similar are two modalities) and uniqueness/redundancy of information (a.k.a. information imbalance) between the modalities on alignment (Figs. 2, 4-6, 10, 14-17). Alignment is measured via Huh et al’s [1] KNN-based center kernel alignment variant (also see Appendix B). Further, they studied how alignment correlates with performance (Figs. 7-9, 11-13, 18-22, Tab. 1). To answer these questions, they designed a synthetic dataset (Fig. 3) to control for uniqueness and heterogeneity. They corroborate the synthetic results with experiments using the Wikipedia caption dataset [5] and MultiBench [2].\n\n## update after rebuttal\n\nPlease see my rebuttal comment below.\n\n---\n\n## References\n\n[1] Huh, Minyoung, et al. \"The platonic representation hypothesis.\" arXiv preprint arXiv:2405.07987 (2024).\n\n[2] Liang, Paul Pu, et al. \"Multibench: Multiscale benchmarks for multimodal representation learning.\" Advances in neural information processing systems 2021.DB1 (2021)\n\n[3] Liang, Victor Weixin, et al. \"Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 17612-17625.\n\n[4] Schrodi, Simon, et al. \"Two effects, one trigger: on the modality gap, object bias, and information imbalance in contrastive vision-language representation learning.\" arXiv preprint arXiv:2404.07983 (2024).\n\n[5] Srinivasan, Krishna, et al. \"Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning.\" Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 2021.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer and are glad that they find our experiments well-designed and motivated. Below we address the reviewer’s comments and questions.\n\n**Under “Methods And Evaluation Criteria:”**\n> The KNN-based variant … other alignment measures would be appreciated since properly measuring alignment is challenging.\n\nWe report additional results with unbiased CKA with a RBF kernel, Mutual KNN [1], SVCCA [2] with three different sample sizes, and all metrics support our paper’s main claims. See response to reviewer B2rQ for more details. \n\n[1] Huh et al. \"The platonic representation hypothesis.\" (2024). \n[2] Raghu et al. “SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability” (2017).\n\n**Under “Experimental Designs Or Analyses:”**\n> Correlation is measured by the Pearson correlation coefficient. However, rank-based correlation coefficients would be a bitter fit, like Spearman or Kendall’s $\\tau$, since they don’t assume a relationship a priori (beyond that the order should matter).\n\nWe are happy to explore other correlation metrics. [Here](https://tinyurl.com/h3fv2ez3) are the results using Spearman correlation. We find that the Spearman correlation supports our main claims: it shows a strong negative trend between maximum alignment and uniqueness and that the relation between alignment and performance is weaker or negative when there is greater uniqueness. Also, we note the Pearson correlation might actually be a suitable choice given common observations that linear relations in latent representations tend to emerge after training (e.g., see https://arxiv.org/abs/2007.00810).\n\n**Under “Essential References Not Discussed:”**\n> Schrodi et al [4] showed that information imbalance causes the modality gap and object bias. …It’d be good to discuss the similarities and differences to Schrodi et al. in future versions.\n\nThank you for bringing up this related work. We agree that [4] is related to our work in that it analyzes the effect of information imbalance on the representations learned through contrastive learning, whereas our work focuses on emerging alignment through increased model capacity. We will add a discussion of [4] in our updated paper.\n\n**Under “Other Comments Or Suggestions:”**\n\n> It’d be good to make more explicit how alignment is measured between visual-only models like DINOv2 and the LLMs, as done by Huh et al [1].\n\nThe details of the alignment computation are in Appendix B as follows: “Following Huh et al. (2024), we use $k=10$ nearest neighbors over 1024 samples from the Wikipedia caption dataset. For the vision model, the class token of each layer is used, and for the language model, the embeddings of a given layer are average pooled to a single token. $l_2$ normalization is applied to the features and elements in the features that are above the 95-th percentile are truncated.”\n\nWe’re happy to add any additional details that the reviewer thinks are missing. \n\n**Under “Questions For Authors:”**\n\n> In the synthetic data, is information always redundant or unique for each sample or can this vary per-sample? E.g., for one sample a factor is part of the data while for another sample it is not.\n\nThe proportion of redundant to unique information is constant for all samples.\n\n> Why does the first encoder in the synthetic setting only have a single layer?\n\nWe provide additional experiment results demonstrating that our results are unchanged when $E_1$ is a higher depth. [Here](https://tinyurl.com/4uwe4zkm), we change the depth of $E_1$ to 2 and 3 and find that the results are not significantly changed. We hypothesize that because $E_1$ is trained on the untransformed modality, $E_1$ will remain relatively easy to optimize even as the depth increases. We will include these results in our updated paper.\n\n> How are the models trained in the synthetic setting? I.e., what type of training method is used? CLIP loss? Captioning loss?\n\nIn the synthetic setting, the ground truth labels are available, the models are trained in a supervised manner, with cross-entropy loss.\n\n> How are correlations computed? Since there are many points per x-value across all plots, this should lower correlations, right?\n\nIn Fig. 4 and 5, the correlation is computed using only the maximum alignment rather than all points. We will clarify this in our updated paper. \n\n> What is the effect of the number of task-relevant features? Currently, it is only set to 8. What happens when you set it to 256?\n\nThe total number of task-relevant features would not impact our results -- what matters is the proportion of redundant to unique features. If we had 256 task-relevant features, out of which 128 are shared, we would expect to see that the result is similar to $U=4$ in our setting. Our results on real-world data are on much higher dimensions -- for our experiments on Wikipedia Image Text and on MM-IMDb, we use LLMs with high dimensional latent spaces of 1024 or greater.", "claims_and_evidence": "* Maximum achievable alignment is controlled by uniqueness of the input modalities. This is well-supported by the experimental evidence in Figs. 2, 4-6, 14-17 on synthetic as well as real data. The results for heterogeneity (Figs. 6, 10) are less clear.\n\n* Performance is not directly correlated to alignment (Sec. 5). This is again supported by Figs. 7, 8. Again, the results for heterogeneity seem less clear (Fig. 9).", "methods_and_evaluation_criteria": "* The KNN-based variant of center kernel alignment based on Huh et al [1] is well-suited to evaluate alignment. However, other alignment measures would be appreciated since properly measuring alignment is challenging.\n\n* The synthetic and real datasets, as well as chosen models are well-suited.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "* The synthetic dataset is well-designed and motivated to cleanly study the effect of uniqueness and heterogeneity (Fig. 3).\n\n* Correlation is measured by the Pearson correlation coefficient. However, rank-based correlation coefficients would be a bitter fit, like Spearman or Kendall’s $\\tau$, since they don’t assume a relationship a priori (beyond that the order should matter).", "relation_to_broader_scientific_literature": "Huh et al [1] put forward the platonic representation hypothesis. This paper investigates key data properties (information balance and data heterogeneity) on the alignment. I’d like to note that work by Schrodi et al [4] also investigated information imbalance in the context of the modality gap and object bias for CLIP models (see below for more details). Findings and experiments seem related, though the scope is different. Thus, I conclude that this work is a valuable contribution on understanding how data shapes the models, in this case their representational alignment.", "essential_references_not_discussed": "* Schrodi et al [4] showed that information imbalance causes the modality gap and object bias. Information (im)balance (called information redundancy/uniqueness in this work) is also the data property studied in this work (besides heterogeneity). Particularly, they hypothesized that less shared information worsens alignment that leads to the modality gap and object bias. Further, some findings and experiments share a resemblance. Thus, it’d be good to discuss the similarities and differences to Schrodi et al. in future versions.", "other_strengths_and_weaknesses": "* S: the paper is well-written and clear.", "other_comments_or_suggestions": "* It’d be good to make more explicit how alignment is measured between visual-only models like DINOv2 and the LLMs, as done by Huh et al [1].", "questions_for_authors": "* In the synthetic data, is information always redundant or unique for each sample or can this vary per-sample? E.g., for one sample a factor is part of the data while for another sample it is not.\n\n* Why does the first encoder in the synthetic setting only have a single layer?\n\n* How are the models trained in the synthetic setting? I.e., what type of training method is used? CLIP loss? Captioning loss?\n\n* How are correlations computed? Since there are many points per x-value across all plots, this should lower correlations, right?\n\n* What is the effect of the number of task-relevant features? Currently, it is only set to 8. What happens when you set it to 256?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Four reviewers submitted reviews for this submission. In the pre-rebuttal phase, reviewers raised some concerns such as:

- no new method is proposed
- is the HSIC metric used sufficient for reporting numbers
- most results are on synthetic datasets
- impact of relations and practicality of the method
- some questions to improve clarity 

All reviewers acknowledged the rebuttal and after the post-rebuttal phase, the final ratings are two accepts, a weak accept and a weak reject. Reviewers acknowledged that most of their concerns have been resolved after the post-rebuttal phase. Although a point of practical usability remained, during discussion with reviewers, it was identified that the paper contains sufficient interesting insights and advances our scientific understanding significantly and therefore this practicability aspect can be left for future work. Therefore, the decision is to recommend the acceptance of the paper. Authors are recommended to incorporate important reviewer's comments the final version.
Decision: Accept (poster)