A Closer Look at Multimodal Representation Collapse
Authors: Abhra Chaudhuri
Anjan Dutta
Tu Bui
Serban Georgescu
Venue: ICML 2025 spotlightposter (2025)
Keywords: Multimodal learning
modality collapse
Primary area: theory->domain_adaptation_and_transfer_learning
TL;DR: Modality collapse happens as a result of cross-modal polysemantic entanglements arising out of rank bottlenecks in deep multimodal models, and can thus be remedied by freeing up such bottlenecks.

Abstract: We aim to develop a fundamental understanding of modality collapse, a recently observed empirical phenomenon wherein models trained for multimodal fusion tend to rely only on a subset of the modalities, ignoring the rest. We show that modality collapse happens when noisy features from one modality are entangled, via a shared set of neurons in the fusion head, with predictive features from another, effectively masking out positive contributions from the predictive features of the former modality and leading to its collapse. We further prove that cross-modal knowledge distillation implicitly disentangles such representations by freeing up rank bottlenecks in the student encoder, denoising the fusion-head outputs without negatively impacting the predictive features from either modality. Based on the above findings, we propose an algorithm that prevents modality collapse through explicit basis reallocation, with applications in dealing with missing modalities. Extensive experiments on multiple multimodal benchmarks validate our theoretical claims. Project page: https://abhrac.github.io/mmcollapse/.

A Closer Look at Multimodal Representation Collapse
Abhra Chaudhuri'
Anjan Dutta? Tu Bui'
Serban Georgescu
Abstract
We aim to develop a fundamental understand- ing of modality collapse, a recently observed empirical phenomenon wherein models trained for multimodal fusion tend to rely only on a subset of the modalities, ignoring the rest. We show that modality collapse happens when noisy features from one modality are entangled, via a shared set of neurons in the fusion head, with pre- dictive features from another, effectively mask- ing out positive contributions from the predic- tive features of the former modality and lead- ing its collapse. We further prove that cross- modal knowledge distillation implicitly disentan- gles such representations by freeing up rank bot- tlenecks in the student encoder, denoising the husion-head outputs without negatively impact- ing the predictive features from either modality. Based on the above findings, we propose an al- gorithm that prevents modality collapse through explicit basis reallocation, with applications in dealing with missing modalities. Extensive exper- iments on multiple multimodal benchmarks vali- date our theoretical claims. Project page: ht / /bbbaacc. gi tuub. 10 aamcollapsed
ncuron cress entanglement
Loss
SLb optimal solution undie nank starvation
Demasiing
post freeing
Dimemtin Freed up by Bants Rea mullccattion
[Brokan] Loss incrase
Restrictd Dimension onder Rank Octtleverk
Denoting
Figure When noisy features of one modality exist in entangle- ment with predictive features of another in the fusion head (the probability of which increases with the number modalities), results in sub-optimal solution wherein the predictive value of the former modality is diminished hy the inevitable existence noisy features. Freeing up rank bottlenecks allows for the denoising of such features along independent dimensions without affecting the latter modality, while simultancously allowing the predictive features of the former modality to contribute to loss reduction.
model is reliant only on a subset of modalities and if it is those that specifically missing at test time, the model could end up completely non-functional. Although there have been several attempts at mitigating modality collapse based on a priori conjectures about what might be causing them, such as conflicting gradients (Javaloy et al., 2022) interactions between data distributions and the fusion strat- egy (Ma et al., 2022), to the best of our knowledge, there have been no prior efforts towards developing bottom-up understanding of the underlying learning-theoretic theoretic phenom- ena at play.
1. Introduction
number of recent works in the multimodal learning liter- ature have observed that models that aim to learn fusion of several modalities often end up relying only on subset of them (Javaloy et al., 2022; Wu et al., 2024). This phe- nomenon, termed modality collapse, has been empirically observed across a diverse range of fusion strategies (Javaloy et al., 2022; Ma et al., 2022; Zhang et al., 2022; Zhou et al., 2023; Wu et al., 2024), and has serious implications for the scenario when certain modalities can go missing test time (You et al., 2020; Ma al., 2022; Wu et al., 2024). If
We aim bridge this gap by developing mechanistic the- ory of multimodal feature encoding that is agnostic of the specific choice of fusion strategy (see Task Setup in Sec- tion 3). We start by showing that modality collapse arises as a result of an unintended entanglement among the noisy and the predictive features of different modalities through a shared set of polysemantic neurons (Elhage et al., 2022), which we observe, via Lemma 1, to increase quadratically
Fujitsu Research of Europe UUniversity of Surrey, Correspon- dence to: Abhra Chaudhuri Cabhra.chaudhun @ fujitsu.com>.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Closer Look at Multimodal Representation Collapse
Entangled Polysemantichn
Lass Trajectory under Interference
Predictive Feature (Modality
Noisy Feeture (Modality 2]
Switching Threshold
Interpolation regime
Gap duc to Modality Callapse
Disentangled
Polysemanticly
Range of Polysemantic Neuron
Patterns wise Contribution
Marginal Features
(a)
Figure 2. Polysemanticity with and without feature interference between noisy and predictive features. All horizontal axes correspond to the value of the weight of the polysemantic neuron. For (b) and (c), the vertical axes correspond to the title of the columns. When the predictive features of modality 1 (MI) and the noisy features of modality 2 (M2) activate along the region in the interpolation regime of the same neuron (a top), it prevents the predictive features of M2 from contributing to loss minimization (c- top) because of the unintended inclusion and the unwanted interference from the noisy features of M2 top), leading its collapse. However, when they are disentangled (here, by being mapped to disjoint sub-regimes in the weights-space around a switching threshold: a bottom), they result non-interfering activation patterns (b bottom), and effectively, a feature- wise separable effect on the marginal loss bottom).
with the number of modalities. implics that predictive features of some modalities cannot be learned without also including noisy features from the other modalities. The noisy features then effectively suppress the predictive value of the modality that they come from, leading to their ob- served collapse in the fused representation, process we formalize in Theorem 1.
Consequently, through Theorem 2, we arrive at the result that this gradient-rant bottleneck forces SGD to parameter- ize the fusion head neurons in a polysemantic manner.
Interestingly, observe in Theorem 3, that knowledge distillation into the modalities that collapse, from the ones
that survive, implicitly averts cross-modas polysemantic entanglements. It does so by freeing up rank bottlenecks at the level of the student encoders. As a result, as again shown in Figure 1, the noisy features of the modalities that would otherwise collapse, are allocated dedicated dimen- sions in the latent space, which the fusion operator can then leverage to denoise the output representations, This allows for the complete incorporation of predictive features from all the modalities without any noisy interference, thereby preventing collapse.
As depicted in Figure 1, in the optimization landscape, col- lapse corresponds to a suboptimal solution such that any step around it along the dimension of entanglement (which the only available dimension for optimization in the given state), would lead to a simultaneous denoising of one modality and the forgetting of predictive features from the other. If the latent factors underlying the modalities are sufficiently com- plementary, we show that cross moosl predictive predictive feature entanglements are less likely to oceur than predictive- noisy entanglements (Lemma 3) so, we mainly focus on the latter in this work.
pereitive-predictive
Under the condition of identifiability (Gulrajani Hashimoto, 2022) of modality-ppecific causal factors up to equivariances of the underlying mechanisms (Ahuja et al., 2022), we propose an algorithm called Explicit Basis Real- location (EBR), which automatically identifies cross-modal feature entanglements and learns independent denoising di- rections in the latent space to counteract their hindrance on empirical risk minimization. Consequently, the concrete feature-to-basis mapping across modalities obtained from EBR can be used identify suitable substitution candidates
We find that this cross-modal entanglement of features hap- pens due to faulty neural capacity allocation (Scherlis et al., 2022) among modalities during the optimization of the fu- sion head (illustrated in Figure 2), which we observe, in Lemma 2, to be a result of the well-known low-rank sim- plicity bias of neural networks (Huh et al., 2023) limiting the rank of the gradient updates received any given layer.
2

---

A Closer Look at Multimodal Representation Collapse
for dealing with missing modalities at test-time.
X= X2,,, Xm], Y with m modalities and labels Y, first undergo an independent modality-wise encoding through set of learnable functions fi, f2.... fm. followed by a learnable modality fusion operator where N = dim [1) + dim(f2) + + dim(fm), and M is any arbitrary integer. Note that since N is finite, sp can be considered as a neural network of bounded width and arbi- trary depth, as they are known be universal approximators (Kidger & Lyons, 2020). The output from ςp is then fed into classifier head 9 : RMM - [0, Ijc, where C' is the number of classes in Y, to produce the output label ÿ. Specifically, the neural representation of and the final prediction on a multimodal sample x = (x1. X2, X2,..,,,,.. E X is obtained as follows:
To summarize, we (i) provide a theoretical understanding of modality collapse based on polysemantic neurons in the fu- sion head leading to unwanted cross-modal entanglements, and the low-rank simplicity bias of neural networks; (ii) show that cross-modal knowledge distillation into the modal- ities undergoing collapse from the ones that survive has an implicit effect of averting modality collapse through dis- entanglement and denoising, based which we propose Explicit Basis Reallocation (EBR) for more systematic disentanglement and denoising of multimodal embeddings: (iii) extensive empirical validation of our theoretical results on multiple standard multimodal benchmarks, with EBR achieving state-of-the (SOTA) results in the application of dealing with missing modalities at test time, one of the most challenging tests of an algorithm's 's robustness to modal- ity collapse.
RV + R
3 8= (fi(x))) fa(xx). .... fm (xm)))
where {x1, (2;,,,,.,, X are the modality specific instanti- ations of the sample x. All proof's are provided in Ap- pendix B.
2. Related Works
One of the earliest reports of modality collapse was in multimodal generative models (Shi et al., 2019; Sutter et al., 2021; Ma et al., 2020), for which Nazábal et al. (2020) hypothesized the phenomenon to be a result of disparities between gradients. which was also later con- firmed by Javaloy et al. (2022). Parallely, Wang et al. (2020) showed modality collapse for multimodal classifica- tion problems, where they found unimodal models to often outperform multimodal ones. They conjectured that (i) in- creased capacity of multimodal models leads to overfitting supported later in Wu et al. (2024); Zhou et al. (2023);
Modality Collapse:
3.1. Polysemanticity and Cross-Modal Entanglements
We begin by showing that as the number of modalities in- crease, the proportion of cross-modal polysemantic neurons, i.e., those that encode features from more than one modality (as opposed to monosemantic neurons which encode exactly one feature from one modality) also increases (Lemma 1), This makes it difficult for the fusion head to independently control the contribution from a given modality without po- tential destructive interference from others (Theorem 1). It is important to note that the results presented presuppose the occurrence of polysemanticity, i.e., the number of task- relevant features in X is greater than the number of neurons in any layer, something that is most often known to hold in practice (Scherlis et al., 2022).
and (ii) different modalities generalize at different rates-
which also aligns with the findings by Nazábal et al. (2020):
Javaloy al. (2022). Going beyond the hypotheses and conjectures, we aim to develop a rigorous theoretical un- derstanding of modality collapse from the perspective of polysemanticity (Scherlis et al., 2022; Huben et al., 2024; Leconte et al., 2024) and low-rank simplicity bias (Huh et al., 2023). These theoretical tools have also so far been restricted primarily to unimodal cases, and to the best of our knowledge, we are the first to explore them for explaining failure modes in multimodal learning. Additionally, since our proposed remedies to modality collapse can be lever- aged to deal with missing modalities at test time, we provide an extended literature review on this area in Appendix A.
Lemma 1 (Cross-Modal Polysemantic Collision). As the number of modalities increase, the fraction of polysemamiic neurons encoding features from different modalities, for a given depth and width, increases quadratically in the number of modalities as follows:
p(w,) 2 m(m - 1) (dim fmin)
2.
dim fi
where p(wp) is the probability of a neuron being polyse- mantic via superposition, and fmin is the modality-ppecific encoder with the smallest output dimensionaliry.
3. Collapse Mechanisms and Remedies
Task Setup: We study the properties of representations of multimodal data learned by deep modality fusion algorithms. Specifically, based on recent literature (Wu et al., 2024: Zhang et al., 2022; Ma et al., 2022; 2021), we follow the generic setup where samples from multimodal distribution
Since we are interested in the fraction of neurons that are cross-modal polysemantic in the space of all polysemantic neurons, p(-) is defined over the space of polysemantic neurons only. Also, Lemma deals specifically with 2- semantic neurons, ie., those that simultaneously encode
3

---

A Closer Look at Multimodal Representation Collapse
features, which is the most likely form of polysemanticity in the combinatorial space of cross-modal polysemanticities for any value of m.
I(x: yl11) = [(x; ation of SGID, the norm of the difference between and the average gradient outer product (AGOP) of the complete weight matrix W is bounded as follows:
ylze), at any iter-
Definition 1 (Conjugate Features). A conjugate feature z is one that coexists, in a given modality, with another feature z" such that at least one of them has some predictive value, but they can semantically cancel each other out when considered in conjunction, i.e..
Evew (v (w)
S <7())
DEX
where 7(w) is the degree of polysemanticity of w.
y(z - [(z"; = O; yzz = 0
Theorem 2 implies that since the AGOP is known to be the low-rank subspace that W converges under SGD (Rad- hakrishnan et al., 2024), the small distance (tighter bound) between the AGOP and subspaces w with higher >(w) im- plies that W in fact converges to those subspaces w with higher degrees of polysemanticity 7(w). In other words, SGD is more likely to parameterize W with the low-rank polysemantic neurons than with high-rank monosemantic ones. The implication of this is that, we specifically con- sider the noisy-predictive type of cross-modal polysemantic neurons, they will be the first to get climinated among all cross-modal polysemantic neurons due to the low-rank sim- plicity bias, as they either do not contribute to loss reduction, or do so negatively. Below we explore ways of breaking this implicit rank bottleneck to circumvent cross-modal polyse- mantic interference among noisy and predictive features.
In other words, z and z" noisily interfere with each other. Theorem 1 (Interference). As the number of cross-modal polysemantic collisions increase, the fraction of predictive conjugate features contributing to the reduction of the task loss decreases, resulting in the following limit:
lim p(w.)
L ((p(Z)))))) = 0.
Owp p
VEyEX
where by denotes predictive conjugate features in X.
The modality facing the above marginal decrease in contri- bution to the loss reduction across its feature space, is the one that gets eliminated as part of the collapse. Next, we show how this polysemantic interference is a consequence of the low rank simplicity bias in neural networks.
3.3. Knowledge Distillation Frees Up Rank Bottlenecks
3.2. Rank Bottleneck
We propose a simple remedy to the cross-modal polyse- mantic interference that multimodal fusion model might suffer from under the default training paradigm with SGD, Based our result in Theorem 3, the solution is to re- place the modality-speciffe encoder of the modality that gets eliminated under collapse, with one that is pretrained via cross-modal knowledge distillation. Specifically, knowl- edge distillation has to be performed from the modality that survives fusion, to the one that gets ignored under fusion. When more than one modality survives, we experimentally find that distilling in a sequence starting from the weakest and finishing with the strongest provides the best results (Appendix C.3).
We establish that with increasing number of iterations, gra- dient updates in SGD tend to get restricted to low-rank manifold, the rank of which is proportional to the rank of the average gradient outer product or AGOP (Lemma 2). Consequently, in Theorem 2, we are able to derive an upper- bound of convergence for every weight subspace in given layer, which gets tighter as the neurons in that layer get increasingly polysemantic (Definition 2). It thus follows that cross-modal polysemantic interference is result of the low-rank simplicity bias.
Lemma 2 (Gradient Rank). The rank of gradient updates across iterations of SGD at layer is a convergent sequence with the following limit:
Theorem 3 (Dynamic Convergence Bound). When the in- puts to & are dynamic (for instance, when the unimodal representations are aligned via cross-modal knowledge dis- tillation) under some distance metric d, then at any itera- tion n of SGD, the norm of the difference between w and the AGOP of W is bounded as follows for all modalities i E -M and datapoints x E X:
lim rank((iCn) oR rank n-
Σ V X
where pi(x) and ViC,, are respectively the output and the gradient of the loss L at layer at the n-th iteration of SGD, and X is the set of all inputs to layer across the dataset.
Theorem 2 (Polysemantic Bottleneck). Let W be the weight matrix at a given layer of and w S W be any subspace in W. When the reduction in conditional cross- cross-entropy (xx;y) y z) provided (amount of unique la- bel information held) by each feature is the same, i.e.,
lim d(x,,x,) +c
W
Σ XEX
[xp(())V S <k-1/n,
where &; - f,, (x))). ni is a constant for a given depth proportional to the AGOP along the entire weight ma- trix W at that depth, is the maximum permissible bound

---

A Closer Look at Multimodal Representation Collapse
on the distance between any pair of speeait encod- ings, and both W and are functions of x; and x,, as they result from backpropagation their predictions X.
brings all the modalities within the e-neighborhood under d specified in Theorem 3, implementing an explicit disentan- glement of noisy and predictive features. The adversarial updates to & and 9: are continued until the final multimodal prediction loss Ly decreases, so as to retain the un- derlying causal factors that are identifiable (Gulrajani & semanti-
As per Theorem 3, the representations from different modalities get closer to each other under the distance metric d, which is what effectively happens during cross-modal knowledge distillation, the proportion of monosemantic neu- rons in W increases. This results the AGOP of W di- verging away from its polysemantic subspaces w. In other words, knowledge distillation implicitly disentangles the cross
Hashimoto, 2022), alongside modality-specifee cally relevant features (Chaudhuri et al., 2024) that arise out of equivariances shared by the underlying causal mecha- nisms (Ahuja et al., 2022). Projecting gi (x) back into the original dimensionality of S; via h; leads to a denoised representation that utilizes the compete output basis of f, for representing the predictive features of modality i, resulting in increased monosemanticityy
cross-modal modal interferences by freeing the rank bottleneck and encouraging necessary monosemanticity, allowing for inde- pendent, modality-wise denoising of features along novel dimensions. The intuition behind this observation is graphi- cally illustrated in Figure 3.
4. Experiments
3.4. Explicit Basis Reallocation
Datasets and Implementation Details: We choose the MIMIC-IV (Johnson et al., 2023) and avMNIST (Vielzeuf et al., 2018) datasets for experiments. For MIMIC-IV, we follow the same settings as (Wu et al., 2024) and that of (Wang et al., 2023; Ma et al., 2021) for avMNIST. We use Tian et al. (2020) as our cross-modal knowledge distillation (KD) algorithm of choice applied on top of MUSE (Wu et al., 2024), which also serves as our multimodal baseline for comparing EBR with SOTA. Due to space constraints, report the results on MIMIC-IV in the main manuscript and defer those on aVMNIST to Appendix C.2.
Although knowledge distillation facilitates independent de- noising of modalityyspeecinn representations in the fusion head by freeing up rank bottlenecks, the processes of dis- entanglement and denoising are implicit and hence, slow. We leverage our learnings about the disentanglement and denoising dynamics of knowledge distillation and use them as a set of inductive biases to design an algorithm for Ex- plicit Basis Reallocation, which addresses the problem in a significantly more controlled and efficient manner.
All modifications for EBR are restricted at the level of the unimodal encoders, and we do not alter the fusion operator in any way, which makes it agnostic of the choice of the fusion operator. We introduce simple encoder-decoder head h2- h on top of each modality specific encoder such that the unimodal encoding of each modality can be specified by the function fi- = ft h4 h For notational convenience, let 9: = fc h4. We also introduce a modality- discriminator network s that is trained on g, (x) to predict the modality labels. h, h-1 and sb are simple two-layer MLPs, and hence add minimal parameter overhead. Jointly, we optimize the following two criteria:
4.1. Cross-Modal Polysemantic Interference
Objective and Settings: We validate our theory on cross- modal polysemantic interference (Section 3.1) by studying the impact of the unimodal encoder corresponding to the modality that gets eliminated under fusion, on the minimiza- tion of the semantic loss. The results are depicted in Figure 4. The multimodal prefix is the modality-specific encoder of the modality that gets eliminated due to collapse. The red curve represents its semantic loss during multimodal train- ing, computed via linear evaluation on its representation. The unimodal baseline is the same encoder, but addition- ally optimized to retain unimodal semantic classification performance. Therefore, although both encoders have the same architecture and receive inputs from the same modal- ity, the multimodal prefix only receives gradient updates through the fusion head, whereas the unimodal baseline also directly optimizes the semantic loss.
371
Lmd Σ (x)( m): frem = Lc)
where Lmd and Ceem respectively stand for the modality discrimination loss and the semantic loss (of the final mul- timodal prediction) respectively. The modality-specinf pa- rameter sets are updated as follows in each iteration of SGD:
As predicted by Lemma 1, as the number of modalities increase, the number of poly- semantic features in the downstream fusion head also in- creases. Now, since polysemantic features bottleneck the fusion head (Theorem 2) due to rank-constrained gradi- ent updates (Lemma 2), backpropagated gradients through
Observations and Analyses:
a x- 2b- - V Led
9i x 9: V Lree + V. Lund
h 1 + h- Vn frem
Theoretical Rationale: The maximization of Land by gi
5

---

A Closer Look at Multimodal Representation Collapse
X
Modality (rank = 1)
Modality (rank=1 1)
Joint basis (effactive rank = +E)
Joint basis (effective rank 3)
Modality 2 (rank = 2)
Modality 2 (rank = 3)
Pre-fusion
Fusion
Pre-fusion
Fusion
(a) Cross-Modal Interference due to Rank Bottleneck
(b) Rank Bottleneck Free-up via Basis Reallocation
Figure 3. Illustration of modality collapse due to rank bottlenecks enforcing cross cross-modal polysemantic interference (a), and how freeing up such bottlenecks via basis reallocation can facilitate the elimination of noisy features (red) by encouraging monosemanticity (b).
25
Nurommdai wuft
Limmzdal Rasaline
25
Unimadal Saceline
Mutimodal Frof
Iuu Brocaaa Prafi
Unrimodal Daseline
Multmooial Prets Unimodte Boowire
20
Serr aTTO LOGS
Sear artoc Laxa
25-
Modal
Serannc LoGS
Modalie
12
20
1.5
30
100
200
300
403
502
400
100
200
300
400
500
1000
200
2010
420
500
Epoch
Esoch
Spech
Figure 4. MIMIC-IV: Semantic loss curve during training with increasing number modalities. The multimodal prefix is the semantic loss (linear) evaluation on the modality-specifie encoder corresponding to the modality that gets eliminated during multimodal training. The unimodal baseline the same encoder, but is additionally optimized to minimize its unimodal semantic loss.
the fusion head into the multimodal prefix also get rank- constrained, forcing it to allocate fractional capacities to features that would otherwise have been monosemantically represented. This makes the predictive features harder to decode, leading the observed gap between the curves. Since the unimodal model also directly minimizes its own semantic loss, it has much lower possibility of cross- interference, allowing it to successfully perform the neces- sary capacity allocations, leading to lower loss values. As the number of modalities increase, the gap between the uni- modal baseline and the multimodal prefix also increases, aligning with the conclusions of Lemma 1 and Theorem 1.
setting (w/ KD or w/o EBR) corresponds to the vanilla mul- timodal model, and the unimodal baseline refers to the rank of the representation learned by the unimodal encoder when trained in a standalone manner to minimize the semantic loss without any multimodal fusion.
Observations and Analyses: of the modality that gets eliminated by default) is increased, the multimodal rank can be seen to decrease very fast in the default setting. It happens particularly rapidly around critical point (3 = 4), exhibiting form of phase transition wherein the rank drops to values lower than the unimodal baseline. As the multimodal model is forced to incorporate more of the said modality, it is forced to select its (mostly noisy) features from the polysemantic subspaces that has already learned (Lemma 2). So, by the virtue of being rep- resented polysemantically, the rank of this feature subspace ends up being much lower than it otherwise would (as de- picted by the unimodal baseline). However, this decay in rank is not observed as we free up rank bottlenecks through basis reallocation, via KD or EBR, implying that rank bot- tlenecks causing mross-modal polysemantic interference precisely what the root of modality collapse. The rank
As the value of 33 (the strength
4.2. Presence of Rank Bottlenecks
Objective and Settings: We empirically validate our the- ory linking cross-modal polysemantic interference with the low-rank simplicity bias of neural networks (Section 3.2)
by looking at the relationship between the rank of the mul- timodal representation and the amount of upweighting (B) needed to force the multimodal model to incorporate the modality that it would otherwise eliminate under collapse. The results are visualized in Figure 5 (a) and (c). The default
6

---

A Closer Look at Multimodal Representation Collapse
(a) Mult Medal Rank (KD)
(b) Representacion Similarity (KD)
[c] Mciti-Modal Rank (CBR)
(c) Representation Similarity (CBR)
2000
0.9
1350
3800
Meczon Santantty
07
63m
Husse
Huse
1400
5420
Bassine
1000
2000
Figure MIMIC-IV: Multimodal rank and representation similarities modalities with the multimodal representation, under implicit (KD) and explicit (EBR) hasis reallocation mechanisms, across different strengths / of the modality that gets eliminated under collapse.
Vanilla
KD (Ours)
92.5
EBR (Ours)
90.0
8
AUC-ROC (Mortality)
Semantic LOSS
82.5
Grape
80.0
ShaSpec
2
M3Care
77.5
Muse
KD (Ours)
75.0
EBR (Ours)
100
200
300
400
500
10
15
25
30
35
50
Epoch
Figure 7. MIMIC-IV: With increasing noise rate, existing ap- proaches suffer from modality collapse due to noisy cross-modal entanglements. With improved strategies of basis reallocation. implicit (KD) or explicit (EBR), robustness to noise and the conse- quent prevention of modality collapse be ensured.
Figure 6. MIMIC-IV: Semantic loss minimization comparison be- tween vanilla multimodal learning and using implicit (KD) and explicit (EBR) basis reallocation.
of the default multimodal representation being bounded above by that of the unimodal baseline beyond the phase transition around the critical point, is a consequence of the upper-bound presented in Theorem
have higher rank, while EBR provides stronger buffer relative KD against the rank decay occurring around the critical point. Figure 5 (b) and (d) show the representa- tion similarities between the multimodal representation and strongest (teacher) and the weakest (student) modalities. Both the strongest and the weakest modality representations can be seen more consistently align with the multimodal representation when using EBR, while KD requires some upweighting (by increasing the value of B) to achieve this alignment. In either case, they indicate that basis realloca- tion makes the multimodal representation use information from all the modalities, instead of collapsing onto just a subset (strongest) of the modalities.
4.3. Effectiveness of Basis Reallocation
Next, we test the effectiveness of basis reallocation (both implicit, via KD, and explicit, via EBR) as a mechanism for freeing up rank bottlenecks break cross-modal polyseman- tic interferences, and eventually averting modality collapse. We report our results in Figure 5, Figures 6 and 7, and Ta- ble 2, all of which unanimously and unambiguously show the effectiveness of basis reallocation towards preventing modality collapse, confirming the result Theorem
Rank and Similarity with the Multimodal Representa- tion: Figure 5 (a) and (c) provide the most direct evidence that basis reallocation frees up rank bottlenecks, as mul- timodal representation both KD and EBR consistently
Optimization Dynamics: In Figure 6, we visualize the dy- namics of optimizing the semantic loss with or without the implicit (KD) and explicit basis reallocation (EBR) strate- gies. Generally, we observe that using basis reallocation

---

A Closer Look at Multimodal Representation Collapse
Mortality
Readmission
Method
AUC-ROC
AUC-PRC
AUC-ROC
AUC-PRC
CM-AE (ICML '1)
0.7873 # 0.40
0.3620 = 0.22
0.6007 0.31
0.3355 # 0.25
SMIL (AAAI 221)
0.7981 # 0.11
0.3536 # 0.12
0.6155 # 0.09
0.3279 + 0.15
MT (CVPR '22)
0,8176 0.10
0.3467 + 0.06
0.6278 # 0.09
0.2959 + 0,05
Grape (NeurIPS '20)
0.7657 = 0.16
0.3733 # 0.09
0.6335 0.07
0.3120 =
M3Care (SIGKDD '22)
0.8265 + 0.09
0.3830 + 0.07
0.6020=0.00
0.3870 0.05
ShaSpec (CVPR '23)
0,8100 0.13
0.3630 0.09
0.6216
0,3549 0.08
MUSE (ICLR'24)
0.8236 0.09
0.39.87 0.05
0.6781 0.05
0.4185 +4185=0000 0.07
EBR (Ours)
0.8533 # 0.09
0.4277 + 0.02
0.7030 = 0.05
0.4290 = 0.02
Table 1. MIMIC-IV: Comparison of average performance with standard deviation across multiple modality missingness rates.
provides improved overall loss minimization as opposed to not using it (vanilla). Specifically, when using implicit (KD), SGD tends to first learn noisy, polysemantic neurons before freeing up rank bottlenecks to denoise them. This leads multiple step-like structures in the loss trajectory, which could correspond to the saddle geometry of such landscapes. Such geometries are possibly smoothed out into more convex neighborhoods under EBR. providing faster convergence and a more consistent optimization dynamic.
that basis reallocation can be performed agnostic of the fu- sion strategy, we replace the unimodal encoders of a number of SOTA multimodal models with their knowledge distilled EBR counterparts and report their performance in Table 2. Irrespective of the fusion strategy, it can be seen that out- of-the-box improvement in test performance can be attained by these simple replacements, establishing the generic na- ture of our results.
4.4. Fusion with Inference- Missing Modalities
Denoising Effect of Basis Reallocation: Theorem 1 posits that cross-modal polysemantic entanglements can be harm- ful precisely due to the possibility of interference from noisy features. We do set of experiments where we corrupt the weakest modality during training with additive random uni- form noise over a range of noise rates (from 5-50%) and compare SOTA multimodal models with our proposed im- plicit (KD) and explicit basis reallocation (EBR) mecha- nisms. We report our findings in Figure 7. Unless explicitly taken care of, existing SOTA models perform notably poorly when the noise rate is increased. Since basis reallocation frees up rank bottlenecks, the novel dimensions can be uti- lized by SGD for denoising. EBR makes the denoising process explicit through the adversarial training of jb and gi to optimize Lmd, providing stronger robustness to noise.
As discussed in Section 3.4, after reallocating bases to fea- tures via EBR, since the latent factors of X are identifiable up to the equivariances shared by the underlying mecha- nisms, we leverage this property to substitute missing modal- ities test-time with those that are available. Concretely, once training with EBR converges, we proceed as follows: (1) Rank modalities wrt their similarities (computed pair- wise across all samples) with a reference modality (chosen as the strongest modality in our experiments) in terms of the latent encoding g: (x;)): (2) When a modality i of a test sample x goes missing, choose its substitution candidate as the modality j that is closest it in the ranked list; (3) Com- (gy pute the proxy unimodal encoding of x; as h h- (x, )). We validate the importance of ranking based the EBR latents by benchmarking against other substitution strategies in Appendix C.4.
Mortality
Readmissiou
Method
AUC-ROC
AUC-PRC
AUC-ROC
AUC-PRC
To evaluate our approach, we adopt the experimental setup of MUSE (Wu et al., 2024), following which we mask out the modalities in the MIMIC-IV dataset with probabilities {0.1, 0.2, 0.3, 0.4, 0.7). We then take the average and stan- dard deviation across these missingness rates and report the results in Table 1. can be seen that close to 3% improve- ments can be achieved in both Mortality and Readmission prediction AUC-ROC and AUC-PR metrics on top of SOTA, by simply replacing the unimodal encoders of the baseline MUSE with our proposed EBR variants and following the ranking and substitution strategy for dealing with missing modalities detailed above.
Grape (NeurIPS 220)
0.8837
0.4584
0.7085
0.4551
KD
0.9011
0.4620
0.7231
0,4610
FBR
0,9102
0.4799
0.7488
0.4691
M3Care (SIGKDD '22)
0.8896
0.4603
0.7067
0.4532
KD
0.8950
0.4700
0,7080
0.4562
EBR
0.8987
0,4850
0.7296
0,4832
MUSE (ICLR'24)
0.9201
0.4883
0.7351
0.4985
0.9350
0.4993
0,7402
0.5066
EBR
0.9380
0.5001
0.7597
0.5138
Table 2. MIMIC-IV: Using knowledge-distileed/ EBR backbones for the modality that would otherwise be eliminated by collapse.
Independence from Fusion Strategies: Finally, to show
8

---

A Closer Look at Multimodal Representation Collapse
5. Conclusion and Discussions
society. To the best of our knowledge, we are aware of any negative impacts of this work.
We studied the phenomenon of modality collapse from the perspective of polysemanticity and low-rank simplic- ity bias. We established, both theoretically and empirically, that modality collapse happens due to low rank gradient updates forcing the fusion head neurons to polysemantically encode predictive features of one modality with noisy fea- tures from another, leading to the eventual collapse of the latter. This work attempts to reveal that multimodal learning may be plagued in ways that are rather unexpected, and con- sequently, unexplored, thereby leaving room for number of improvements and future explorations.
References
Ahuja, K., Hartford, J., and Bengio, Y. Properties from mechanisms: equivariance perspective on identifiable representation learning. In ICLR, 2022.
Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. Invariant risk minimization. ArXiv, abs/1907. 99.999299,, 2019.
Baldi, P. Autoencoders, unsupervised learning, and deep architectures. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning, 2012.
For instance, Theorems and 3 are valid when the reduction in conditional cross-entropy provided (amount of unique la- bel information held) by each feature is the same. It remains to be explored how these results can be extended to the case when such reductions are different across features. We con- jecture that (and as also empirically evidenced) EBR turns the otherwise saddle landscape, that is obtained after the rank bottlenecks are freed up by knowledge distillation, into a convex one, enabling smoother and more predictable opti- mization. Developing an understanding of this could lead to deeper insights into the dynamics of the loss landscape geometry of modality collapse.
Bishop, C. C.M. Training with noise is equivalent to tikhonov regularization. Newral Computation, 7(1):108-116, 1995. doi: 10.1162/neco. (ttp///ccc.199..... 1995.7.1. 1...88.8.
Chaudhuri, A., Mancini, M., Chen, Y., Akata, Z.,, and Dutta, A. Cross-modal fusion distillation for fine- Ene-grained sketch- based image retrieval. In BMVC, 2022.
Chaudhuri, A., Georgescu, S., and Dutta, A. Learning conditional invariances through non-commutativity. In ICLR, 2024.
Acknowledgments
Chen, J. and Zhang, A. Hgmf: Heterogeneous graph-based fusion for multimodal data with incompleteness. In ACM SIGKDD, 2020.
We would like to thank the following individuals at Fu- jitsu Research of Europe for their independent inputs: Mo- hammed Amer (for discussions on observations of modality collapse in multimodal genomics), Shamik Bose (for inputs on polysemanticity and capacity in neural networks), and Nuria Garcfa-Santa (for help with the MIMIC-IV dataset). We would also like to thank the anonymous reviewers for their thorough analysis and detailed feedback that helped clarify and improve various aspects of our work.
De Veaux, R. D. and Ungar, L. H. Multicollinearity: A tale of two nonparametric regressions. In Selecting Models from Data, 1994.
Dou, Q., Liu, Q. Heng, P.-A., and Glocker, B. Unpaired multi-modal segmentation via knowledge distillation. IEEE Transactions on Medical Imaging, 2020.
Impact Statement
Elhage, N., Hume, T., Olsson, C.,, Schiefer, N., Henighan,
T., Kravec, S., Hatfield-Dodds Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D.,, Wattenberg, M., and Olah, C. Toy models of superposition. Transformer Circuits Thread, 2022.
This paper advances the understanding of modality collapse in multimodal fusion, providing a theoretical foundation and experimental evidence improve robustness in the presence of missing modalities. By systematically analyzing cross- modal interactions, we demonstrate that mitigating modality collapse enhances performance across diverse applications. In healthcare, our approach can be capable reliable diag- nosis even when hard / expensive to acquire modalities such as imaging or genomics might be missing. In autonomous perception, it could support safer decision-making despite sensor failures. By introducing scalable and generaliz- able multimodal learning framework, this work lays the foundation for more robust and deployable AI systems in real-worke settings, with the potential to positively impact
Galanti, T., Siegel, Z. S., Gupte, A., and Poggio, T. A. SGD and weight decay secretly minimize the rank of your neural network. In NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning, 2024.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D.,, Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Commun. ACM, 2020.
Gulrajani, I. and Hashimoto, T. Identifiability conditions for domain adaptation. In ICML. 2022.

---

A Closer Look at Multimodal Representation Collapse
Huben, R., Cunningham, H., Smith, L.. R., Ewart, A., and Sharkey, L., Sparse autoencoders find highly interpretable features in language models. In ICLR, 2024.
Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., and Sun, C. Attention bottlenecks for multimodal fusion. In NeurIPS, 2021.
Huh, M., Mobahi, H., Zhang, R., Cheung, B., Agrawal, P. and Isola, P. The low-rank simplicity bias in deep networks. Transactions on Machine Learning Research, 2023.
Nazábal, A., Olmos, P. M., Ghahramani, Z., and Valera, I. Handling incomplete heterogeneous data using vaes. Pattern Recognition, 107, 2020.
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A. Multimodal deep learning, In ICML, 2011.
Jackson, Z., Souza, C., Flaks, J., Pan, Y., Nicola, H., and Thite, A. Free spoken digit dataset (fsdd). https:/ //ttt//ggtttt//// com/ wwt.//////////eee//////e Jakobovski/ free- www.ep-diiigeeeeee dataset, 2018.
Parascandolo, G., Kilbertus, N., Rojas-Carulla, M., and Schölkopf, B. Learning independent causal mechanisms. ICMI., 2018.
Javaloy, A., Meghdadi, M., and Valera, I. Mitigating modal- ity collapse in multimodal VAES via impartial optimiza- tion. In ICML, 2022.
Poklukar, P., Vasco, M., Yin, H., Melo, F. S., Paiva, A., and Kragic, D. Geometric multimodal contrastive representa- tion learning. In ICML, 2022.
Johnson, A. E. W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horn, S., Pollard, J., Moody, B., Gow, B., wei H. Lehman, L., Celi, L. A., and Mark, R. G. Mimic-iv, a freely accessible electronic health record dataset. Scientific Data, 2023.
Radhakrishnan, A., Beaglehole, D., Pandit, P., and Belkin, M. Mechanism for feature learning in neural networks and backpropagation-free machine learning models. Sci- ence, 2024.
Kidger, P. and Lyons, T. Universal Approximation with Deep Narrow Networks. In COLT, 2020.
Ramachandram, D. and Taylor, G. W. Deep multimodal learning: A survey recent advances and trends. IEEE Signal Processing Magazine, 2017.
Kim, W., Son, B., and Kim, I. Vilt: Vision-and lnn-uuge transformer without convolution or region supervision. In ICML, 2021.
Scherlis, A., Sachan, K., Jermyn, A. S., Benton, J., and Shlegeris, B. Polysemanticity and capacity in neural networks. ArXiv, abs/2210.01892, 2022.
Lecomte, V., Thaman, K., Schaeffer, R., Bashkansky, N., Chow, T., and Koyejo, S. What causes polysemanticity? an alternative origin story of mixed selectivity from in- cidental causes, 2024. URL https::///// httpp:://araae httpp:s//axxvv.rg//// 2b12/211220 03096.
Shen, Y. and Gao, M. Brain tumor segmentation on mri with missing modalities. In Information Processing in Medical Imaging. Springer International Publishing, 2019.
Shi, Y. N, S., Paige, B., and Torr, P. Variational mixture- of-experts autoencoders for multi-modal deep generative models. In NeurlPS, 2019.
Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 1998.
Shi, Y., Paige, B., Torr, P., and N, S. Relating by contrasting: A data-efficient framework for multimodal generative models. ICLR, 2021.
Lee, K., Lee, S., Hahn, S., Hyun, H., Choi, E., Ahn, B., and Lee, J. Learning missing modal electronic health records with unified multi-modal data embedding and modality- aware attention. In Machine Learning for Healthcare Conference, 2023.
Sreelatha, S. V., Kappiyath, A., Chaudhuri, A., and Dutta, A. DenetDM: Debiasing by network depth modulation. In NeurlPS, 2024.
Ma, C., Tschiatschek, S., Hermández-Lobato J. M., Turner, R. E., and Zhang, C. Vaem: a deep generative model for heterogeneous mixed type data. In NeurlPS, 2020.
Sutter, T. M., Daunhawer, I., and Vogt, J. E. Generalized multimodal ELBO. ICLR, 2021.
Ma, M., Ren, J., Zhao, I.,, Tulyakov, S., Wu, C., and Peng, X. Smil: Multimodal learning with severely missing modality. In AAAI, 2021.
Tian, Y., Krishnan, D., and Isola, P. Contrastive representa- tion distillation. In ICLR, 2020.
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., and Salakhutdinov, R. Multimodal transformer for unaligned multimodal language sequences. In Korhonen, A., Traum, D., and Mårquez, L. (eds.), ACL., 2019.
Ma, M., Ren, J., Zhao, L., Testuggine, D., and Peng, X. Are multimodal transformers robust to missing modality? In CVPR, 2022.
10

---

A Closer Look at Multimodal Representation Collapse
Vielzeuf, V., Lechervy, A., Pateux, S., and Jurie, F. Cen- tralnet: multilayer approach for multimodal fusion. In ECCV Workshops, 2018.
Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., and Carneiro, G. Multi-modal learning with missing modality via shared-specific feature modelling. In CVPR, 2023.
Wang, W., Tran, D., and Feiszli, M. What makes training multi-modal classification networks hard? In CVPR, June 2020.
Wu, Z., Dadu, A., Tustison, N., Avants, B., Nalls, M., Sun, J., and Faghri, F. Multimodal patient representation learn- ing with missing modalities and labels. ICLR, 2024.
Xie, Q. Hovy E. H., Luong, M.-T., and Le. Q V. Self- training with noisy student improves imagenet classifica- tion. In CVPR, 2019.
Xue, Z. and Marculescu, R. Dynamic multimodal fusion. In Multi-Modal Learning and Applications Workshop (MULA), CVPR, 2023.
You, J., Ma, X., Ding, Y., Kochenderfer, M. J., and Leskovec, J. Handling missing data with graph repre- sentation learning. In NeurlPS, 2020.
Zhang, C., Chu, X., Ma, I.., Zhu, Wang, Y., Wang, J., and Zhao, M3care: Learning with missing modalities in multimodal healthcare data. In ACM SIGKDD, 2022.
Zhao, F., Zhang, C., and Geng, B. Deep multimodal data fusion. ACM Comput. Surv. 2024.
Zhou, Y., Wang, X., Chen, H., Duan, X., and Zhu, W. Intra- and model curriculum for multimodal learning. In ACM International Conference on Multimedia, 2023.

---

A Closer Look at Multimodal Representation Collapse
A. Extended Literature Review
Missing Modalities: Existing SOTA multimodal fusion approaches do not account for the possibility of missing modalities (Ramachandram & Taylor, 2017; Nagrani et al., 2021; Shi et al., 3021; Chaudhuri et al., 2022; Zhao et al., 2024). Although this limitation was identified in works as early as Ngiam et al. (2011), the pattern of missingness, i.e., which modality(ies) could go missing, were assumed to be known at training time. Later, a number of graph- techniques (You et al., 2020; Zhang et al., 2022; Wu et al., 2024), including ones that use heterogeneous graphs to model different missingness patterns (Chen& & Zhang, 2020), alongside transformer-basee (Tsai et al., 2019; Ma et al., 2022), and Bayesian meta-learning (Ma al., 2021) based approaches attempted to operate without this assumption. Other approaches such as those that facilitate direct interaction among spdality-peecific raw inputs (Kim al., 2021; Lee et al., 2023), and ones based on self-supervised domain adaptation (Shen & Gao, 2019), also provided promising results.
Interestingly, was observed by Ma et al. (2022) that multi-modal representations are strongly dependent on the fusion strategy, and that the optimal way of fusing modalities is dependent on the data, due to which, the authors recommended that fusion strategies should be iestributionsspeiitie However, the limitations introduced by such dependencies was also accounted for in related multimodal learning literature to address the challenge of resource-efficient utilization of modalities, which tackled through dynamic data-dependent fusion (Xue & Marculeseu, 2023). Despite the existence of number of bespoke techniques for dealing with missing modalities, SOTA approaches such as ShaSpec (Wang et al., 2023), in addition to such algorithms, were also benchmarked against baselines based on GANs (Goodfellow et al., 2020) and autoencoders (Baldi, 2012), which the authors found to be similarly competitive. Although there have been some works that explored the applicability of knowledge distillation to dealing with missing modalities, their purposes have been scoped to addressing issues such as compressing the extra parameter overhead due to multimodal fusion (Dou et al., 2020), or dynamically weighting data points within modalities and contributions from loss terms (Zhou et al., 2023). In this work, we knowledge distillation collapse, and show that it can avoid collapse by implicitly freeing up rank bottlenecks that lead to cross-modal entanglements between noisy and predictive features.
tool to theoretically study the fundamental processes in optimization that govern modality
B. Proofs
Lemma 1 (Cross-Modal Polysemantic Collision). As the number of modalities increase, the fraction of polysemantic neurons encoding features from different modalities, for given depth and width, increases quadratically in the number of modalities as follows:
(dim fmin)
p(wp) 2 m(m 1)
Σ dim fi
where p(w,) is the probability of a neuron being polysemantic via superposition, and fmin is the modality-specific encoder with the smallest output dimensionality.
Proof. The number of ways any two features can be selected by yp from X such that both belong to different modalities 2 (T) (dim fum) since there are (")) ways of choosing modality-pain and there are 2 (dim fmin) ways of choosing feature pairs in each such combination. Now, it is these pairs of features that lead to cross-modal polysemantic collisions m
(through neuron subspaces wp) during fusion in yp. Let the ambient dimension of the input to be Fdim dim fc. Then,
for given depth and width, the probability that a polysemantic weight subspace would represent features from two different modalities would be:
m72
(dim fmnn) m(m 1)
(dim fmnn)
p(wp)= 2
2
Fum
Σ dim
This completes the proof of the lemma.
Lemma (Entanglement by Feature Type). If the latent factors underlying the modalities are sufficiently complementary to
12

---

A Closer Look at Multimodal Representation Collapse
each other in terms of predicitvity of the label y, i.e. for any pair of modalities and
ΣΣ 2. <K P q
where p and q are indices over the latent factors of modalities? and j respectively, and is a constant, then, the noisy features from modality more likely to be entangled with predictive features of another through polysemantic weights in the fusion head, i.e., for any pair of noisy (Ze) and predictive features from the same modality, the following will hold:
by Ew
1 Ze W
where denotes weight subspaces representing a different modality y.
Proof. Since noisy features closer to random, they can get entangled with neurons representing predictive features from any modality if the corresponding neuron allows features up to (1 K)) units of deviations, according to the Johnson-Linndnssrauus lemma (Elhage et al., 2022), ie., satisfying the following:
Ze w 2 Zy <
implies that the set of noisy features z, would be more closely aligned with w than the set of predictive features Zy. making the ratio of the sums over all w y,, of the latter to the former, I 1.
With the elimination of noisy features, such entanglements following from superposition become less likely among predictive features across modalities, since they would normally require dedicated dimensions with strong orthogonality, i.e. monosemantic neurons. This completes the proof of the lemma.
Theorem 1 (Interference). As the number of cross-modal polysemantic collisions increase, the fraction of predictive conjugate features contributing to the reduction of the task loss decreases, resulting the following limit:
lim Σ
C (p(2,)))))) = =0,
Owp
Vay
where by denotes predictive conjugate features in X.
Proof. Let be the noisy conjugate of 2y. As the number of polysemantic collisions increase, does the proportion of polysemantic neurons, i.e., lim Now, from Lemma 3, we know that as the noisy features are more likely to get into p(wp)~1 cross-modal polysemantic entanglements, which implies that they would exhibit higher similarity (dot-product) with the polysemantic subspace Wp. Additionally, since Zy and z, are conjugate each other, Zy would exhibit low similarity (dot-product) with Wp. activating in the opposite direction that of z.- Again, from Lemma 3, since z, is entangled with Wp. when present in the input, it would always activate. Based on this, the conjugate activation equation (without the non-linearity) of By can be written as:
lim p(zy) = ((2y) + (p(z) p(w,)-+1
Wp Zy
Ze = 0,
large
large +ve
meaning that the net activation of ny along wp is O, which ultimately implies that:
a 8 L (p(Zz)))) Ow,
lim Σ p(w p(w,)-1 )+ V2,
This completes the proof of the theorem.
13

---

A Closer Look at Multimodal Representation Collapse
B.1. Rank Bottleneck
Definition 2 (Degree of Polysemanticity). We quantitatively define the degree of polysemanticity (Elhage et al., 2022; Scherlis et al., 2022), 7, of a weight subspace, w, the ratio of the number of features in the input distribution X encoded in the subspace and the number of dimensions of the subspace, i.e.,
w n X
(()
dim(w)
where w X| is the number of features in X that encoded in w.
Polysemantic neurons lie on a low-rank manifold that the weights of each layer converge to under SGD. So, since all optimization happens along this low-rank polysemantic manifold, it not possible for y to avert the cancellation effect among conjugate features, of which the noisy counterparts may get encoded part of polysemantic neuron.
Lemma 2 (Gradient Rank). The rank of gradient updates across iterations of SGD at layer I is convergent sequence with the following limit:
lim rank(ViCn) x rank n-000
Σ
VVx(()V
XEX
where (x)(x) and ViLn are respectively the output and the gradient of the loss L at layer / at the n-th iteration of SGD, and X is the set of all inputs to layer across the dataset.
The rank of each layer decreases with every iteration of SGD (Galanti et al., 2024). Step 2: Every layer converges a quantity proportional to the average gradient outer product (Radhakrishnan et al., 2024).
Proof. Step 1:
Theorem 4 (Depth-Rank Duality (Sreelatha et al., 2024)). Let A [Ao, AI..., An) be the attribute subspace of X with increasing ranks, i.e. rank(Aoo < rank(Ai) < < rrank such that every A E A is maximally and equally informative of the label Y, iee, I(Aa, Y) = [Ai, Y) =... = I(An, Y). Then, across the depth of the encoder é SGD yields a parameterization that optimizes the following objective:
min C(f(o(X)), + min Σ [01(()) - 22d 3 A | Φ.S
ERM
where -(:,.) is the empirical risk, [(.) is classifier head, Φ[J(.) is the output of the encoder & (optimized end- to-end) at depth 1, 11-112 is the no-nrrm, ② is the element-wise product, X is the ly-normalized version of X, Ω [L ma(t) ... (1,), 1 is random binary function that outputs with a probability n, and x((1) is the propagation probability of A, at depth! bounded as:
(1) = 0 (rank
where rank(o[1)) is the effective rank of the o|l] representation space, and r; rank(A)).
Theorem 2 (Polysemantic Bottleneck). Let W be the weight matrix at given layer of yo, and w < W be any subspace in W. When the reduction in conditional cross-entropy H (x;))zz) provided (amount of unique label information held) by each feature is the same, i.e., y/ı1 = /(x;y = = [(x; y (2x), at any iteration of SGD, the norm of the difference between w and the average gradient outer product (AGOP) of the complete weight matrix W is bounded as follows:
< <w(w)
w
where ?(w) is the degree of polysemanticity of w.
Proof. We know that the weights of a neural network when optimized with SGD converge to a value proportional to the Averge Gradient Outer Product (AGOP), which essentially represents those sets of features which when minimally perturbed, produce large change in the output (Radhakrishnan al., 2024).
14

---

A Closer Look at Multimodal Representation Collapse
Additionally, SGD with weight decay minimizes the ranks of the weight matrices (Galanti al., 2024) and that this minimization is more pronounced as we go deeper into the neural network (Huh al., 2023), formalized in Theorem 4 by Sreelatha et al. (2024).
In other words, the deeper we go into network, the more likely it is for the representations to be of a lower rank (Huh et al., 2023; Sreelatha et al., 2024), and that this rank decreases with each successive iteration (Galanti et al., 2024). In other words, for each layer, there is a subspace of a specific rank that is updated through backpropagation, and according to Lemma 2, since the rank of such updates decrease with iterations, the lower the rank of this subspace, the greater its cumulative gradient across iterations, i.e., the more likely the weights of the particular layer would converge to this subspace.
be learned by gradient descent and the more likely it is that
If two features equally minimize the empirical risk, and their joint encoding has no local improvement in the minimization of the marginal loss, extending the result by Galanti et al. (2024), SGD on the fusion operator would prioritize the encoding of the modality with the lower rank of the two as follows:
W W
min W+(
Σ WEM
W
S KKKC 2
(1)
where (x) is the subspace the weight matrix at layer I of the fusion operator yp corresponding to the modality m. μ is the learning rate, the SGD iteration, and (o(() is the target weight matrix that the fusion operator converges towards at layer such that:
rank((W))))))
= E rank((W:
mE
where Wo is the weight matrix at the output layer of the modality-specifie encoder (o) of modality and Ma is the set of modalities that survive collapse. Now, for polysemantic subspaces w, since the degree of polysemanticity 7(w) >> 1, we can extend Equation (1) as:
Wi:(1)
min WO
Σ MEM
Wi(1)
I (1 2/2)) S ((w)
(2)
According to the condition f(x;)|zz) = [(x,y(zz) = combinations all lie at the same depth, their empirical risks are essentially the same, and so are the gradients from the ERM term. Now, as a result of modality collapse, we know that one of the basins is steeper than the rest, meaning it has a higher local gradient. Since the empirical risk constant across all the basins / multimodal combinations, the steepness must come from the rank minimization term. Therefore, the combination with steep entry must lead lower rank solution.
since basins corresponding to multimodal
As observed by (Javaloy et al., 2022), no local improvement in the minimization of the marginal loss may be due to conflicting gradients in the local parameterizations for the two modalities. Note that this does not imply that the two modalities are globally conflicting. It is only the local encodings of the two that somehow conflict with each other. Specifically, following from Equations (1) and (2) and Lemma 2, the norm of the difference between the polysemantic subspace w and the AGOP of the ambient weight matrix W can be bound as:
Σ (ewls)Vew(s)) K. (1 -2 FEX
w
Therefore, for polysemantic bases, at any given iteration of SGD, the difference with the AGOP can be more tightly bound than for monosemantic bases. Formally, combining Equations (1) to (3), we have:
w-
Σ VVw(()) ZEX
< 6K-(1- (1- (1-244))) S <w(w)-1/
(4)
In other words a basis formed with polysemantic neurons is more similar to the AGOP than one formed with monosemantic neurons, provided the conditional cross-entropy H (x: reduction provided (amount of unique label information held) by
each feature is the same, iee., T(x; y 22) = T(x; yxxy|/z)) (22) = T(x;y(244
15

---

A Closer Look at Multimodal Representation Collapse
This completes the proof of the theorem.
B.2. Knowledge Distillation Frees Up Rank Bottlenecks
As described earlier, the cause for collapse cross- moosl interference between noisy and predictive features. Here, we find that knowledge distillation implicitly remedies this problem. Knowledge distillation converges when the noisy and the predictive subspaces have been sufficiently disentangled to the point that the available rank can be assigned completely towards modeling the teacher modality, after effectively having discarded as many of the noisy features as possible. There is some empirical evidence on this from the self- celf-distillation literature (Xie et al., 2019). With the disentangled and denoised representations obtained from knowledge distillation, the causal factors of the previously eliminated modalities can expand (inverse of collapse) the multimodal representation space, utilizing previously unused dimensions for encoding features that effectively reduce the loss. Previously, the effect of the semantically relevant features from the eliminated modalities would be observable since the superposition with the noisy features would cancel out (when marginalized across all features) any conditional reduction in loss that the causal factors would have induced.
Theorem 3 (Dynamic Convergence Bound). When the inputs to are dynamic (for instance, when the unimodal representa- tions are aligned via cross moosl knowledge distillation) under some distance metric d, then at any iteration n of SGD, the norm of the difference between w and the AGOP of W is bounded as follows for all modalities i, j E M and datapoints x X:
lim d(s. +
Σ XEX
(x
S 5x-1/n
where X, = fy is constant for a given depth proportional to the AGOP along the entire weight matrix W that depth, c the maximum permissible bound on the distance between any pair of modality-speciffe encodings, and both W and w are functions of x; and Xj. as they result from backpropagation on their predictions on X.
Proof. A weighted general case of the Depth-Rank Duality result by (Sreelatha et al., 2024) follows consequence of modal interferences, which can be written
min ((v((X),) + min Σ rank( P.
ERM
where a is a function measuring the compactness of the embedding space, in our case, through the pairwise distance between points, d(x; x)), computed via given iterate of s,, L..,
= (d(x;, X,))
When cross-modal polysemantic interference happens despite there being extra available dimensions the ambient space, indicates faulty capacity allocation (Scherlis et al., 2022) of the corresponding neurons by SGD, which follows from the LindeesttLnd lemma (Elhage et al., 2022). It happens because SGD, by default, performs the aforementioned implicit weighted rank-regularization aside from ERM, with the highest possible value of such that it encourages the smallest possible rank for a target ERM solution. However, if it is known a priori that the pairwise distances d(x;, X,) approach some constant neighborhood e, we get:
01 02 On hep (c) (c) A,
where A is a constant. Following from this, under the given limit, the rank regularization term in the optimization objective of Depth-Rank Duality gets relaxed as:
lim d(R. X3)
min -
Σ rank(p) min *
Σ rank((p),
ip
which ultimately implies that for any & inducing pairwise distances d(x,, x,) > will have rank(()) < rank(pp). As representations from different modalities get closer to each other under the distance metric d, which what effectively happens during cross-modal knowledge distillation, the increase in rank encourages consequent increase in the proportion
16

---

A Closer Look at Multimodal Representation Collapse
of monosemantic neurons in W. This results in the AGOP of W diverging away from its polysemantic subspaces wp. with the size of such polysemantic subspaces decreasing follows:
Wp
Σ XEX
lim
Wp
Σ XEX
(x)
d(R,8;)))) +
Given the above constraint on the size of polysemantic bases in W as d( x,) e, the size of any feature subspace w. i.e, the number of features that can be encoded by any w approaches some constant upper bound corresponding to c:
lim d(K;,;))
|w
X |w. X kw;
where |we| is the size of the feature subspace in the neigborhood e. Since both |we| and |X] are constants, low is also constant. It thus follows that, under dynamic input space approaching bounded neighborhood the number of features encoded in any polysemantic subspace also gets bounded by kw as - oc. So, the RHS in Theorem can be rewritten as:
| nxx dim ww
1/n
-(1 liw
) K 1/n
-1/m
lim ((w) -1/n n-0
(dim(w)
where x kw/ h//m/vmvv (w) is a constant as both kw and dim(w) are constants. Finally, following from the above, when d(x,,,) &2) + c, Equation (4) from the proof of Theorem 2 can be expressed in terms of as:
lim d(x., X3)
Σ x
SK- (1 11--1111100 1222)) S <(w) 1// =x 1/11/1
This completes the proof of the theorem.
B.3. Additional Remarks
Unequal conditional cross-entropy across features: According to the condition ylx,) = [xxyy/zz) = y(x,)/zz), since basins corresponding to multimodal combinations all lie at the same depth, their empirical risks are essentially the same, and so are the gradients from the ERM term. Now, as a result of modality collapse, we know that one of the basins is steeper than the rest, meaning has a higher local gradient. Since the empirical risk is constant across all the basins/ / multimodal combinations, the steepness must come from the rank minimization term in Theorem 4. Therefore, the combination with steep entry must lead to lower rank solution. When the equality is not met across all features, the low-rank/ / steepness condition is trivially satisfied by the existence of a lower-dimensional subspace of 2,s that has a lower conditional mutual information T(x: y (2:), and deriving the upper-bound on the rank in terms of the AGOP is no-longer necessary. The rank of the subspace comprising features with lower relative mutual information could act as a reasonable estimate of the rank of the final weights that SGD would converge to. By considering the condition with the equality, we analyze the boundary case that even when such a subspace with low conditional mutual information cannot be identified, possible to upper-bound the rank of the weight matrix.
Identifying Latent Factors and Substitutability; If the latent factors are identifiable from the data up to some symmetry of the latent distribution (Gulrajani Hashimoto, 2022), then the substitutability result also holds up to the actions of that symmetry group. other words, substitutability is directly contingent on identifiability, i.e., the existence of symmetries in the latent distribution can affect the substitutability of latent factors among modalities.
Segregating predictive and noisy features from the set of latent factors can be done by learning to discover independent causal mechanisms on the aggregate of all modalities (Parascandolo et al., 2018). The degree to which the latent factor representations of the individual modalities can be compressed, i.e., the value of e in Theorem 3, depends on the size / rank of the invariant (Arjovsky et al., 2019) subspace.
Every modality consists of both noisy and predictive features. If fusion collapses to a specific modality (target), it means that the modality contains more predictive information and less noise than the rest. Knowledge distillation to align the representations of the other modalities with the target would thus denoise the other modalities, allocating a larger fraction of the feature space of the modality-speciffe encodings of
Intuition behind the Dynamic Convergence Bound:
17

---

A Closer Look at Multimodal Representation Collapse
such modalities to predictive features. Since noisy features are closer random, they can get entangled with predictive features from any modality if the corresponding neuron has slight deviation from perfect orthogonality, according the Johnson-Lindensttauss lemma (Elhage et al., 2022). With the elimination of noisy features, such entanglements following from superposition become less likely among predictive features across modalities, since they would normally require dedicated dimensions with strong orthogonality, i.e, monosemantic neurons. Therefore, as the unimodal representations get closer to each other through the implicit denoising mechanism of knowledge distillation, SGD gets increasingly compelled to parameterize the fusion head in a monosemantic manner.
In other words, knowledge distillation implicitly disentangles the cross-modal interferences by freeing rank bottlenecks and encouraging necessary monosemanticity, allowing for independent, modality-wiss denoising of features along novel dimensions.
When tross-modal polysemantic weight matrices are munkbbttlenecked, the only solution to minimize the loss further is to allocate the noisy features new, independent dimensions in the latent space. However, this causes an increase in representation rank. To counteract this, knowledge distillation frees up the rank bottlenecks by down-weighting the rank-regularization term.
Distillation Denoising Conjecture: Knowledge distillation reduces the weight on the rank regularauaii term, freeing up other dimensions for exploration, potentially containing higher rank solutions with more modalities. The rank de- regularization happens due to knowledge distillation having to denoise the student modality in order to align its representation with that of the teacher (Bishop, 1995). as also indicated by our empirical observations in Section 4.3. Given this, we conjecture that knowledge distillation allows the representation of the noisy components of the teacher modality transformed version of the student noise, thereby eliminating the need for encoding noisy features from every modality in the neurons encoding the student modality. This can be formally stated as follows:
lim [e(xx))) - g(aw()) Vx. E Xa, E X.
f_(a))-f((m))
where X, and X, are respectively the teacher and student modalities, La and ft are respectively the teacher and student encoders, s) represents the noisy components of modality, and g is the transformation function relating the student noise with the teacher noise embedding.
Loss Landscape Geometry: Once the optimizer converges to the modality solution, the value of the loss and the rank of the solution balance each other out. Now, to explore further, the optimizer has to move along a novel direction in the parameter space which leads reduction in loss but simultaneous increase in rank. The reason behind this saddle- geometry is the presence of noisy features from one modality in entanglement with the predictive features from another, which results in an adversarial minimax game between the two. On either side of the saddle point along the unexplored dimensions are predictive features of the former modality which could potentially minimize the task loss, but is not taken into account due to rank regularization. Since, according to Theorem 3, subspaces in get decreasingly polysemantic when d(x,, x,) + e, it implies that knowledge distillation down-weights this regularization by disentanglement and denoising of the bases of the modalities from which the noisy features originate.
A Note on Figure 2: Under disentangled polysemanticity, although the noisy and predictive features may still be encoded via the same neuron, they map to different regions in the activation space of the neuron, leading to a feature-wise separable effect. For instance, the coefficients of the predictive features may have a relatively higher magnitude, implying that in order to activate the noisy component of the neuron, the degree of noise in the input would have be much higher than its predictive counterpart.
C. Additional Experimental Settings and Results
C.1. Experimental Settings
Dataset Details: MIMIC-IV contains information about 180,000 patients across their 431,000 admissions to the ICU, Following Wu et al. (2024), we use the clinical notes, lab values, demographies (age, gender, and ethnicity), diagnosis, procedure, and medications as the set of input modalities. The task is to perform Mortality (whether the patient will pass away in the 90 days after discharge) and Readmission (whether a patient will be readmitted within the next 15 days following discharge) prediction for a given patient. aVMNIST comprises 1500 samples of images and audio, taken from MNIST (Lecun et al., 1998) and the Free Spoken Digits Dataset (Jackson et al., 2018), where the task is to predict the labels of the
18

---

A Closer Look at Multimodal Representation Collapse
input digits from 0 9. We adopt the experimental setup of Wang et al. (2023) for aVMNIST.
Implementation Details:
The two hidden layers of t have output dimensionalities 512 and 256 respectively. The hidden
layers have output dimensionalities 1024 and 512 respectively. whereas that of h- is 512 and 1024. The model was trained for 1200 epochs, with an initial learning rate of 0.01, decayed at rate of 0.9 every 100 epochs. We interleave between the optimization of La and Liem every 10 epochs.
C.2. Results on aVMNIST
Following on from Section 4, we list the empirical results on avMNIST follows:
Presence rank bottlenecks: Figure 10 (a) and (c)
Effectiveness of basis reallocation:
Rank and Similarity with the Multimodal Representation: Figure
Optimization Dynamics: Figure 9
Denoising Effect of Basis Reallocation: Figure
Independence from Fusion Strategies: Table 4
Fusion with Inference-time Missing Modalities: Table 3
The analytical conclusions for avMNIST are the same as what discussed in Section since the patterns of observations are highly consistent between avMNIST and MIMIC-IV. The only experiment not included for avMNIST is the one corresponding to Figure 4 for MIMIC-IV, since avMNIST has only modalities, and hence, it is not possible monitor the effect of increasing the number of modalities the loss curves.
98
96
AUC-ROC (Mortality)
94
Grape
ShaSpec
Muse
KD (Ours)
(Ours)
90
10
15
20
25
30
35
40
45
50
Noise
Figure 8. avMNIST: With increasing noise rate, existing approaches suffer from modality collapse due to noisy cross-mode entanglements. With improved strategies of basis reallocation, implicit (KD) explicit (EBR), robustness noise and the consequent prevention of modality collapse can be ensured.
C.3. Sequence of Distillation
The results of various strategies for sequencing the teacher modality for cross-modal knowledge distillation are reported in Table 5, Based on these observations, we choose weakest-to toatttto-sttooot strongest as the sequence to benchmark our KD based implicit basis reallocation mechanism.
19

---

A Closer Look at Multimodal Representation Collapse
12
Vanilla
KD (Ours)
10
EBR (Ours)
ww
Semantic Loss
1 WMM
MM Vo,MM
0
175
200
Epoch
Figure avMNIST: Semantic loss minimization comparison between vanilla multimodal learning and using implicit (KD) and explicit (EBR) basis reallocation.
(4) Multi-Modal Bank (KD)
(b) Representation Similanit (KD)
(ci Multi-Modal Rank BR)
[d] Representat on Similarity (FBR)
1.00
1500
KD
1390
0.50
1900
Recressiiatto Smilaity
02
0.00
1300
0.7
Tanoten
0.80
1200
0.75
1200
Lnternttal Rameline
1109
1100
1005
0.80
Quatect
1000
-X-
FAB
0.-
FElminated under Collape
0.60
Teacher
-
wD ESR
Robiincs unser Collaatz
Figure 10. avMNIST: Multimodal rank and representation similarities of modalities with the multimodal representation, under implicit (KD) and explicit (EBR) reallocation mechanisms, across different strengths 8 of the modality that gets eliminated under collapse.
Acc @ Audio Missingness Rate
Method
95%
90 %
85 %
80% %
Autoencoder (ICMLW' 12)
89.78
89.33
89.78
88.89
GAN (ACM Comm 20)
89.11
89.78
91.11
93.11
Full2miss (IPMI' 19)
90.00
91.11
92.23
92.67
Grape (NeurlPS 20)
89.65
90.42
91.15
91.37
SMIL (AAAI'21)
92.89
93.11
93.33
94.44
ShaSpec (CVPR'23) 23)
93.33
93.56
93.78
94.67
MUSE (ICLR'24)
94.21
94.36
94.82
94.93
EBR (Ours)
95.30
95.57
95.89
95.93
Table 3. avMNIST: Comparison with SOTA on dealing with the missing audio modality across difffernt missingness rates at test time, following the baseline setup of Wang et al. (2023).
C.4. Baselines for Substitutability
We design the following baselines for comparison against our EBR-based modality substitution approach: zeros, random sampling, nearest-neighhor modality, train set average. Using four (two weak and two strong diagnosis, lab values, clinical notes, and medication respectively) modalities from MIMIC IV, we report the average with standard deviation of the 15 possible missingness patterns on the AUC-ROC metric for Mortality prediction in Table 6. The target represents the model
20

---

A Closer Look at Multimodal Representation Collapse
Ace @ Audio Missingness Rate
Method
95 %
90% %
85 %
80 %
SMIL (AAAI21)
92.89
93.11
93.33
94, 44
KD
92.95
93.97
94.06
94.70
EBR
93.77
94.02
94.51
94.96
ShaSpec (CVPR'23)
93.33
93.56
93.78
94.67
KD
95.02
95.16
95.30
95,45
+ EBR
95.30
95.57
95.89
95.93
MUSE (ICLR'24)
94.21
94.36
94.82
94.93
KD
94.98
95.02
95.40
95.55
+ EBR
95.02
95.26
95.61
95.70
Table 4. avMNIST: Using knowledge-distilled / EBR backbones for the modality that would otherwise be eliminated.
Method
AUC-ROC
AUC-PR
Strongest only
0.9196
0.4875
Strongest-to-weakke
0.8651
0.4130
Random
0.9005
0.4685
Simultaneous
0,9102
0,4786
Weakest-to-strongeet
0.9350
0.4993
Table MIMIC-IV AUC-ROC and AUC-PR on Mortality Prediction for various sequences of knowledge distillation.
Method
AUC-ROC
Target
Zeros
0.5110 0.18
Random
0.6139 # 0.15
Rep-NN
0.7150 + 0.08
Late Fusion
0.6990 0.06
0.7844 0.02
Avg w/o cls
0,5312 # 0.23
Avg w/ cls
0,7396 + 0.09
EBR (Ours)
0.7829 + 0.05
Table MIMIC-IV: Comparison of the substitutability performance EBR with baselines. Experiments performed over subset of (2 strong and 2 weak) input modalities, based on which, the average of 15 possible missingness patterns with standard deviation are reported.
trained on specifically on the subset of modalities that do not go missing, i.e., its performance would always be higher than any bascline since it does not have to deal with the distribution shift that comes from modalities going missing at test time.
C.5. Multicollinearity
We expect to see increased levels of multicollinearity as the number of modalities increase, if the dimensionality of the representation space remains constant. As correctly conjectured by the reviewer, we would expect multicollincarit to be more pronounced the deeper layers of the fusion head. The reason behind this is that although there may he dependencies among features across modalities, they may not be exactly linear. As they propagate deeper into the fusion head, is more likely that those non-linear dependencies would resolved and linearized in the final representation space prior to classification. Theoretically, the bound in Thm 2 derived based on the AGOP, i.c.. x X XV (x) View (x)". being low rank subspace in W (corresponding to independent set of features), discussed in (Radhakrishnan al., 2024), which is also required since one-to-one dimension-to- feeturr mappings needed to detect the presence of multicollinearity may exist in neural networks (De Veaux & Ungar, 1994). This aligns with the condition for regression multicollinearity that x"" X should be not a full rank matrix.
To empirically confirm this, we calculate the variance inflation factor (VIF) with increasing modalities on our trained representation space. We report the average VIF across features Table 7. With the increasing number of modalities,
21

---

A Closer Look at Multimodal Representation Collapse
multicollinearity (VIF) increases in all cases. However, basis reallocation encourages cross-modal features be encoded independently, with the explicit EBR being more efficient in controlling the level of multicollinearity relative to the implicit KD.
# Modalities
2
4
Vanilla
1.15
2.68
3.51
4.70
w/ KD
1.09
1.90
2.30
2.68
w/ EBR
1.05
1.26
1.32
1.55
Table 7. Average variance inflation factor (VIF) across features in the representation space with increasing number of modalities.
C.6. Statistical Comparisons
In Table 8 report the resulting p-values of performing the Wilcoxon rank test with Wolm-Boonferron correction (significance level a = 0.05) on the Table results between our proposed EBR and the other baseline methods. The null hypothesis that the proposed EBR and the other models follow the same distribution of AUC-ROC and AUC-PRCs with the chosen missingness rates, were rejected for the both Mortality and Readmission prediction tasks across all baselines, most often, with significantly low p-values, which in all cases, was lower than 0.01. It further provides evidence in support of the uniqueness of EBR in leveraging basis reallocation to free up rank bottlenecks as a novel mechanism to tackle missing modalities.
Method
Mortality
Readmission
AUC-ROC
AUC-PRC
AUC-ROC
AUC-PRC
CM-AE
0.0090
0.0077
0.0065
0.0035
SMIL
0.0066
0.0053
0.0042
0.0066
MT
0.0083
0.0082
0.0077
0.0065
Grape 0.0027
0.0057
0.0058
0.0042
M3-Care 0.0079
0,0031
0.0069
0.0039
ShaSpec 0.0085
0.0062
0.0049
0.0075
MUSE
0.0088
0.0079
0.0086
0.0089
Table 8. P-values of the Wilcoxon rank test with Holm- Bonferroni correction the Table results between EBR and other basclines.
C.7. Polysemanticity
Considering the results Figure 5 (a) and (c), Figure 7, and Section 4.3, since there is no external source of noise the fusion head, and encouraging monosemanticity through basis reallocation has denoising effect, the noise that leads the observed collapse must come from some cross-modal polysemantic interference.
To provide further evidence, we adapt the definition of polysemanticity based on neural capacity allocation from Scherlis et al. (2022) to measure cross-modal polysemanticity as the amount of uncertainty in the assignment of neuron to particular modality. We train a two-layer ReLU network on weights from unimodal models to classify which modality the input models are optimized on. Next, we apply this modality-classifer on the weights of our multimodal fusion head and record the average cross-entropy (CE) in its outputs. Higher values of cross-entropy indicate higher levels of cross-modal polysemanticity, since the probability masses are spread out across multiple modalities. In Table report the results on bi-modal training. The sharply lower relative CE for KD and EBR directly indicate the reduced cross-modal polysemantic interference under basis reallocation.
C.8. Comparison with Contrastive and Generative Models
Cross-Modal Polysemantic Interference in multimodal contrastive learning: We choose GMC (Poklukar et al., 2022) as our candidate multimodal contrastive learning algorithm for analyzing cross cross-modal polysemantic interference in multimodal contrastive learning. We evaluate GMC by applying their proposed contrastive objective to our baseline representation
22

---

A Closer Look at Multimodal Representation Collapse
Method
CE
Vanilla
5.66
KD
2.09
EBR 0.59
Table Empirically measuring cross moss- polysemantic interference as average cross-entropy (CB) in the modality classifier prediction.
learning setting on MIMIC-IV and report the results terms of the lowest achieved training semantic loss in Table 10. As we can see, trends similar that of our original setting reported the main manuscript, in the semantic loss gap between the Multimodal Prefix and the Unimodal Baseline, play out when we perform contrastive objective based fusion reported in Poklukar et al. (2022). It further supports the claims in Lemma and Theorem I that the number of modalities increase, the modality undergoing collapse contributes less and less to the downstream representation used to encode the semantics, irrespective of the fusion strategy.
Number of Modalities
Multimodal Prefix
27.68
52.90
91.20
167.30
Unimodal Bascline
7.97
6.55
5.33
9.55
Table 10. Lowest achieved training semantic loss with increasing number of modalities in the contrastive setting.
Rank Bottlenecks in Generative and Contrastive Models: We choose MMVAE (Shi al., 2019) as candidate generative model for analyzing rank bottlenecks. Since the objective of generative modeling is somewhat different from the downstream application that we experimented with, analyze MMVAE, we performed the experiment their proposed MNIST-SVHN dataset, while for GMC (Poklukar et al., 2022), since it is for general representation learning, we applied their proposed contrastive objective to our baseline setting on MIMIC-IV. Table 11, we report the results of our experiment, where the vanilla setting refers to the original model, without KD or EBR.
Method
2
6
8
MMVAE (Unimodal Baseline)
198
MMVAE (Vanilla)
477
421
398
110
96
MMVAE KD
482
465
390
298
270
MMVAE + EBR
485
477
431
405
395
GMC (Unimodal Baseline)
1255
GMC (Vanilla)
1877
1330
1146
930
872
GMC + KD
1905
1676
1533
1427
1390
GMC + EBR
1912
1825
1709
1600
1588
Table 11. Representation ranks with increasing in generative (MMVAF) and contrastive (GMC) models.
We can see that in both the generative and contrastive settings, the ranks consistently drop the strength of the modality undergoing collapse 3 is increased. The drop is sharp around critical point, where the rank goes below the unimodal bascline, depicting a form of phase transition, a phenomenon also observed in our original experiments (Section 4.2 Observations and Analyses). Finally, the dropping rank can be counteracted by implicit (KD), and even more effectively, by explicit basis reallocation (EBR), which results in a much more stable rank across the range of different values of
23


Reviews:
[{"review_id": "avngEPWDoy", "reviewer": "Reviewer_NmK3", "summary": "The paper investigates modality collapse in multimodal learning, where models rely only on a subset of modalities. It shows that this collapse occurs due to entanglement of noisy features from one modality with predictive features from another, leading to suboptimal solutions. The authors propose Explicit Basis Reallocation (EBR) to prevent collapse by reallocating basis vectors in the latent space. Extensive experiments validate the theoretical claims and demonstrate state-of-the-art performance in handling missing modalities.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the theoretical and empirical contributions of our work towards understanding modality collapse and providing valuable feedback. Below, we address their concerns, which we will incorporate in our final version.\n\n**Identifiability:** Indeed there are practical ways of ensuring identifiability of latent factors by ruling out common symmetries. For instance, Gulrajani & Hashimoto, 2022 show that if the CP (Canonical Polyadic) decomposition of the third moment tensor of the distribution of the underlying latent factors is unique and its rank is equal to the dimensionality of the vector space in which the factors lie, then general linear symmetries can be ruled out, ensuring identifiability, for which they also provide an efficient algorithm. Ahuja et al., 2022 also shows that looking at inductive biases of the causal mechanisms instead of the true underlying latent factors is sufficient from identifiable representation learning up to any equivariances shared by the mechanisms. So, our assumption on identifiability up to required symmetries is indeed a widespread one in the literature, and there are many practical methods to ensure this as well.\n\n**Unequal conditional cross-entropy across features:** According to the condition $I(x; y|z_1) = I(x; y|z_2) = ... = I(x; y|z_k)$, since basins corresponding to multimodal combinations all lie at the same depth, their empirical risks are essentially the same, and so are the gradients from the ERM term. Now, as a result of modality collapse, we know that one of the basins is steeper than the rest, meaning it has a higher local gradient. Since the empirical risk is constant across all the basins / multimodal combinations, the steepness must come from the rank minimization term in Theorem 4 (Depth-Rank Duality (Sreelatha et al., 2024)). Therefore, the combination with a steep entry must lead to a lower rank solution.\nWhen the equality is not met across all features, the low-rank / steepness condition is trivially satisfied by the existence of a lower-dimensional subspace of $z_i$s that has a lower conditional mutual information $I(x; y|z_i)$, and deriving the upper-bound on the rank in terms of the AGOP is no-longer necessary. The rank of the subspace comprising features with lower relative mutual information could act as a reasonable estimate of the rank of the final weights that SGD would converge to.\nBy considering the condition with the equality, we analyze the boundary case that even when such a subspace with low conditional mutual information cannot be identified, it is possible to upper-bound the rank of the weight matrix.", "claims_and_evidence": "The claims are well-supported by both theoretical analysis and empirical evidence.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I did not check the correctness of the proofs in detail, but the theoretical claims including Lemma 1, Theorem 1, and Theorem 2, appear to be sound based on the provided derivations and explanations.", "experimental_designs_or_analyses": "Yes. The authors systematically analyze the impact of increasing modalities, noise levels, and missing modalities on modality collapse. The evaluation of both implicit (knowledge distillation) and explicit (EBR) basis reallocation are presented clearly.", "relation_to_broader_scientific_literature": "The paper builds on prior work in multimodal learning, specifically addressing the problem of modality collapse. The proposed EBR algorithm contributes to the broader literature by offering new insights into improving the robustness of multimodal models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- **Strengths**: The paper provides a comprehensive theoretical analysis and empirical validation of modality collapse, a significant issue in multimodal learning. The proposed EBR algorithm is effective and achieves state-of-the-art results in handling missing modalities.\n\n- **Weaknesses**: The paper assumes that the latent factors are identifiable up to certain symmetries, which might not always hold in practice and could limit the generalizability.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Q1: How do the theoretical results change when the reduction in conditional cross-entropy provided by each feature is not the same across features? Could you provide some insights or preliminary results in this direction?\n- Q2: In real-world applications, how would you address the potential issue of non-identifiable latent factors? Are there any practical methods to ensure identifiability up to the required symmetries?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "er9CyqboUS", "reviewer": "Reviewer_BnzK", "summary": "The manuscript introduces an Explicit Basis Reallocation (EBR) approach to mitigate multimodal collapse. It first explains that multimodal collapse is driven by polysemantic neurons—which increase with the number of modalities—leading these neurons to converge into a low-rank polysemantic subspace, ultimately causing collapse. To address this, the manuscript initially proposes using knowledge distillation (KD) from the \"strongest\" modality to those that are \"weakest\". Then, it presents EBR as a better alternative that accelerates convergence and eliminates the need for separate modality-specific knowledge distillation. The experimental results on avMNIST and MIMIC-IV indicate that EBR outperforms standard and KD-based training strategies.\n\n## update after rebuttal\nThe score was increased from 3: Weak Accept to 4: Accept. The authors addressed my concern about Multicollinearity, added statistical comparisons, and improved the connection between theoretical and empirical results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to thoroughly understand our paper and providing important comments, which we believe has helped in significantly solidifying our findings. Below, we provide our response, which we will also incorporate in the final version of our manuscript.\n\n**Multicollinearity:** We indeed expect to see increased levels of multicollinearity as the number of modalities increase, if the dimensionality of the representation space remains constant. As correctly conjectured by the reviewer, we would expect multicollinearity to be more pronounced in the deeper layers of the fusion head. The reason behind this is that although there may be dependencies among features across modalities, they may not be exactly linear. As they propagate deeper into the fusion head, it is more likely that those non-linear dependencies would be resolved and linearized in the final representation space prior to classification. Theoretically, the bound in Thm 2 is derived based on the AGOP, i.e., $\\nabla\\varphi_W({x}) \\nabla\\varphi_W({x})^T$, being a low rank subspace in W (corresponding to an independent set of features), as discussed in Radhakrishnan et al., 2024, which is also required since one-to-one dimension-to-feature mappings needed to detect the presence of multicollinearity may exist in neural networks [a]. This aligns with the condition for regression multicollinearity that $X^TX$ should be not a full rank matrix.\n\nTo empirically confirm this, we calculate the variance inflation factor (VIF) with increasing modalities on our trained representation space. We report the average VIF across features below.\n\n|  |  | \\# Modalities |  |  |\n| :---- | ----- | ----- | ----- | ----- |\n|  | **2** | **3** | **4** | **5** |\n| **Vanilla** | 1.15 | 2.68 | 3.51 | 4.70 |\n| **w/ KD** | 1.09 | 1.90 | 2.30 | 2.68 |\n| **w/ EBR** | 1.05 | 1.26 | 1.32 | 1.55 |\n\nWith the increasing number of modalities, multicollinearity (VIF) increases in all cases. However, basis reallocation encourages cross-modal features to be encoded independently, with the explicit EBR being more efficient in controlling the level of multicollinearity relative to the implicit KD.\n\n[a] Veaux and Ungar. “Multicollinearity: A tale of two nonparametric regressions.”, Lecture Notes in Statistics, 1994.\n\n**Definition 1:** We apologize for not clarifying the details. $I(z)$ refers to the mutual information between a feature $z$ and the target label $y$, an abbreviation of $I(z; y)$ for notational minimality. Thm 1 holds in the context of definition 1. The latter is validated through our experiments in Fig. 4,  Sec. 4.3. The observations therein necessitate the existence of conjugate pairs $zz^*$ across modalities, which have the capacity to cancel each other out.\n\n**Statistical Comparisons:** Below we report the resulting p-values of performing the Wilcoxon rank test with Holm–Bonferroni correction (significance level $\\alpha$ = 0.05) on the Table 1 results between our proposed EBR and the other baseline methods.\n\n|  Method | Mortality |  | Readmission |  |\n| :---- | ----- | ----- | ----- | ----- |\n|  | **AUC-ROC** | **AUC-PRC** | **AUC-ROC** | **AUC-PRC** |\n| **CM-AE** | 0.0090 | 0.0077 | 0.0065 | 0.0035 |\n| **SMIL** | 0.0066 | 0.0053 | 0.0042 | 0.0066 |\n| **MT** | 0.0083 | 0.0082 | 0.0077 | 0.0065 |\n| **Grape** | 0.0027 | 0.0057 | 0.0058 | 0.0042 |\n| **M3-Care** | 0.0079 | 0.0031 | 0.0069 | 0.0039 |\n| **ShaSpec** | 0.0085 | 0.0062 | 0.0049 | 0.0075 |\n| **MUSE** | 0.0088 | 0.0079 | 0.0086 | 0.0089 |\n\nThe null hypothesis that the proposed EBR and the other models follow the same distribution of AUC-ROC and AUC-PRCs with the chosen missingness rates, were rejected for the both Mortality and Readmission prediction tasks across all baselines, most often, with significantly low p-values, which in all cases, was lower than 0.01. It further provides evidence in support of the uniqueness of EBR in leveraging basis reallocation to free up rank bottlenecks as a novel mechanism to tackle missing modalities.\n\n**Connection between theoretical and empirical results:**\nWe provide the connections for Thms 1 and 2 in our response to Reviewer xQ3V (Empirical validation of Thms 1 and 2). Here, we provide the same for Thm 3. In Sec. 4.3, we in addition to the results on rank and representation similarity (Fig. 5), we show the difference in the loss landscape geometry between implicit and explicit basis reallocation. Further, we provide evidence that collapse happens specifically due to cross-modal noisy interference, and that basis reallocation, by freeing up rank bottlenecks, allows the new dimensions to be used for denoising, leading to its effectiveness.\n\n**Experimental Details:** \\\n$\\psi$: 512 -> 256 \\\n$h$: 1024 -> 512 \\\n$h^{-1}$: 512 -> 1024 \\\n\\# Epochs: 1200 \\\nLR: Initially 0.01 decayed at a rate of 0.9 every 100 epochs \\\nWe interleave between the optimization of $L_{md}$ and $L_{sem}$ every 10 epochs.", "claims_and_evidence": "I think the paper would benefit from linking multimodal collapse to the multicollinearity problem seen in logistic regression. Since the classification head that takes concatenated features from different modalities acts like logistic regression. Explaining how multicollinearity might cause collapse would make the claims more convincing.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria make sense for the problem. EBR was compared on two multimodal datasets (avMNIST and MIMIC-IV) and versus multiple baselines. Since EBR is a model-agnostic training strategy that is applied on top of existing backbones like MUSE, the approach builds on previous work and the evaluation is well-suited.", "theoretical_claims": "- Definition 1 is vaguely defined; I is used but not defined as predictive value. Conjugate features are defined but then not used in the discussion of the experiments.", "experimental_designs_or_analyses": "- Table 1 lacks a statistical comparison between the best and other models. Usually, it requires running Wilcoxon rank test and correction for multiple comparisons (Holm)", "relation_to_broader_scientific_literature": "The solutions are similar to the ideas of DCCAE (Wang et al., 2015), where each unimodal encoder has its own additional training unsupervised objective and CCA for capturing joint information. This manuscript uses supervised objectives for each unimodal encoder and fusion joint head. Additionally, it introduces modality-specific encoder-decoder heads, which mimic autoencoder structure. Overall, previous attempts and this strategy promotes the regularization of individual unimodal encoders to ensure that we capture non-shared features from different modalities.\n\nWang, Weiran, et al. \"On deep multi-view representation learning.\" International conference on machine learning. PMLR, 2015.\n\nSimilarly, the idea of inter- and intra-modality dependencies (e.g., Madaan et al., 2024). \n\nMadaan, Divyam, et al. \"Jointly Modeling Inter-& Intra-Modality Dependencies for Multi-modal Learning.\" Advances in Neural Information Processing Systems 37 (2024): 116084-116105.", "essential_references_not_discussed": "It would be great to explore related work related to multicollinearity, and inter- and intra-modality dependencies.", "other_strengths_and_weaknesses": "Strengths:\n- The idea is interesting from an empirical results perspective\n- Multiple baselines \n- Two multimodal datasets\n\nWeaknesses:\n- Did not feel that the theoretic component was well connected to the empirical results other than rank results.", "other_comments_or_suggestions": "- It will be hard to reproduce experiments since there are no details on the \"simple two-layer MLPs\", and training schedules. I assume most of it can be found in MUSE, or other backbones, but it is essential to include the details in the appendix.", "questions_for_authors": "Overall, it is an interesting approach. I will increase the score if you can reduce confusion and clarify the details.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hkVkqFRudh", "reviewer": "Reviewer_xQ3V", "summary": "The authors propose a new explanation for the difficult problem of modality collapse. Their argument is that the low-rank bias of neural networks lead them to learn low-rank polysemantic neurons rather than high-rank monosemantic neurons. This is a problem, since as the proportion of cross-modal polysemantic features increases, it prevents the learning of conjugate features that generalize.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for noting important gaps in our initial submission. Below, we aim to address them, which will be included in the final version.\n\n**CM-AE:** We apologize for the confusion caused here and we thank the reviewer for pointing out what was an error in our reporting. While we evaluated all the remaining baselines across the 5 missingness rates (Sec. 4.4), we somehow accidentally added the numbers for CM-AE for different random seeds but no missing modality, which is the same as the first row in Tab. 1 of the MUSE paper. Below, we correct this and report the numbers on CM-AE evaluated in the same missing modality setting as that of the others.\n\n|  | Mort. |  | Rdmn. |  |\n| :---- | ----- | ----- | ----- | ----- |\n|  | **ROC** | **PRC** | **ROC** | **PRC** |\n| **CM-AE** | 0.7873 ± 0.40 | 0.3620 ± 0.22 | 0.6007 ± 0.31 | 0.3355 ± 0.25 |\n\nIn line with the other baselines, CM-AE, too, performs much worse in the missing modality setting and remains significantly below our proposed EBR.\n\n**Polysemanticity:** Considering the results in Fig. 5 (a) and (c), Fig. 7, and Sec. 4.3. Denoising Effect of Basis Reallocation, since there is no external source of noise in the fusion head, and encouraging monosemanticity through basis reallocation has a denoising effect, *the noise that leads to the observed collapse must come from some cross-modal polysemantic interference.*\n\nTo provide further evidence, we adapt the definition of polysemanticity based on neural capacity allocation from Scherlis et al., 2022, to measure cross-modal polysemanticity as the amount of uncertainty in the assignment of a neuron to a particular modality. We train a two-layer ReLU network on weights from unimodal models to classify which modality the input models are optimized on. Next, we apply this modality-classifier on the weights of our multimodal fusion head and record the average cross-entropy (CE) in its outputs. Higher values of cross-entropy indicate higher levels of cross-modal polysemanticity, since the probability masses are spread out across multiple modalities. Below, we report the results on bi-modal training:\n\n|  | CE |\n| ----- | :---: |\n| Vanilla | 5.66 |\n| KD | 2.09 |\n| **EBR** | **0.59** |\n\nThe sharply lower relative CE for KD and EBR directly indicate the reduced cross-modal polysemantic interference under basis reallocation.\n\n\n\n**Empirical validation of Thms 1 and 2:**\nWe do not claim that Fig. 4, just by itself verifies Thms 1 and 2. Below, we provide a more comprehensive explanation: \\\n**Thm 1:** Thm 1 can be factorized as: (i) as the number of modalities increase (ii) the predictive value of the weaker modality decreases. (ii) is validated by Fig. 4, where the gap in semantic loss between the multimodal prefix and the unimodal baseline increases as the number of modalities increase. (i) is validated by Section 4.3, Denoising Effect of Basis Reallocation, which shows that this drop in predictive value is indeed caused due to interference from noisy features. Combined, they imply that as the number of modalities increase, predictive features of some modalities increasingly get entangled with noisy features of another, leading to the collapse of the former. \\\n**Thm 2:** Note that by definition, the lower-rank parameterization predicted by Thm 2, is likely to be polysemantic since it has to fit in more features than the number of available dimensions. Fig 5 (a) and (c) establish the decreasing nature of multimodal rank, and Fig 4 establishes the predictivity degradation implying increased noisy polysemanticity due to the former, i.e., decreased rank. This is further discussed in Sec. 4.2 and L351-355, “The rank of the default multimodal representation being bounded above by that of the unimodal baseline beyond the phase transition around the critical point, is a consequence of the upper-bound presented in Thm 2”.\n\n\n**Beta:** We apologize for the ambiguity in the definition of $\\beta$. A clearer definition is provided in Section 4.2, stating that $\\beta$ “is the strength of the modality that gets eliminated by default”. Specifically, $\\beta$ allows for a custom weighting of the modality that would get eliminated by default, and hence, increasing $\\beta$, forces the model to incorporate it.\n\n**Terminologies and Dataset Details:** We thank the reviewer for pointing out these missing details, which we will incorporate in the final version.\n\n\n$\\mathbf{z}$**,** $\\mathbf{z^*}$**, and** $\\mathbf{zz^*}$:\n$z$ and $z^*$ are latent factors of arbitrary dimensionality, and $zz^*$ refers to the inner product between $z$ and $z^*$ in the space of latent factors. Since we try to keep our results agnostic of the vector space from which the latent factors originate, we did not concretize $zz^*$ any further. The dimensionality of $zz^*$ would depend on the nature of the inner product of the task-specific latent space. However, we agree that it is worth clarifying this point, which we will do in the final version.", "claims_and_evidence": "* The results in general support the authors claims regarding noisy features and low-rank bias leading to modality collapse.\n* The results also highlight the ability of KD/EBR to suppress this behavior and learn higher-rank features.\n* However, I think the most crucial result is not very convincing, which is that in Table 1 the downstream performance (AUROC/AUPRC) is not better than the strongest baseline (CM-AE).\n* I'm convinced by the evidence that low-rank bias leads to modality collapse. However, there's a large emphasis placed on polysemantic features in the writing, but I don't see this addressed much in the experiments.", "methods_and_evaluation_criteria": "* The authors used conventional datasets for evaluating multimodal learning.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "* Figure 4 shows that the ability of the multimodal prefix (the modality that is known to collapse) to predict the target diminishes as the number of modalities increases. It seems like a bit of a logical jump to say that this verifies the claims of Theorems 1 and 2.\n* $\\beta$ is defined as the \"amount of upweighting needed to force the multimodal model to incorporate the modality that it would otherwise eliminate under collapse.\" I don't see an explanation on what this is exactly. What is being upweighted?\n* In Table 1 MUSE is referred to as being the SOTA, but it appears CM-AE is the strongest baseline, and is arguably stronger than EBR as well.", "relation_to_broader_scientific_literature": "* The problem of modality collapse is extensively studied, well-cited in this paper, and the ideas brought by this paper seem to be a novel direction.", "essential_references_not_discussed": "* It would improve readability to define key terminology such as \"polysemantic neurons\" for those who are not familiar with the mechanistic interpretability literature.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "* Some essential details are missing, such as: in the datasets considered, what are the inputs, what are the targets, what is the size of the dataset, etc. It's simply stated as \"For MIMIC-IV, we follow the same settings as (Wu et al., 2024),\" which is fine, but you should at least include this information in the supplementary material.", "questions_for_authors": "* In definition 1, are $\\mathbf{z}$ and $\\mathbf{z}^*$ both in $R^d$? If so, what is $\\mathbf{z} \\mathbf{z}^*$? Intuitively, I see what you're saying but the clarity could improve if you defined variables and functions more precisely.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1V1ZFLxWha", "reviewer": "Reviewer_ZQQA", "summary": "In this paper, the authors contribute with a theoretical understanding of the phenomena of modality collapse in multimodal representation learning model. In particular, the authors show that modality collapse occurs when the predictive features of a given modality become entangled with noise features of another, effectively leading to the collapse of the former. Furthermore, the authors demonstrate that this cross-modal entanglement emerges from faulty neural capacity allocation and that knowledge distillation from the joint encoder into the modality encoder suffering collapse can avert this phenomena. Based on these insights, the authors propose a novel method, Explicit Basis Reallocation (EBR), that promotes the disentanglement and denoising of the multimodal embeddings. The authors evaluate extensively their method across two datasets and highlight how EBR achieves SOTA results on scenarios with missing modalities at test time.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the novelty and thoroughness of our work, as well as pointing us to important adjoining multimodal learning literature that observe modality collapse. Below, we aim to address their concerns, which we will also incorporate in the final version of the manuscript.\n\n**Comparison with generative and contrastive models:**\n\nFor the result in Section 4.1, we evaluate [4] by applying their proposed contrastive objective to our baseline representation learning setting on MIMIC-IV and report the results below in terms of the lowest achieved training semantic loss below.\n\n|   | | | Number of Modalities |  |  \n| :---- | :---: | :---: | :---: | :---: |\n|  | **2** | **3** | **4** | **5** |\n| **Multimodal Prefix** | 27.68 | 52.90 | 91.20 | 167.30 |\n| **Unimodal Baseline** | 7.97 | 6.55 | 5.33 | 9.55 |\n\nAs we can see, trends similar to that of our original setting reported in the main manuscript, in the semantic loss gap between the Multimodal Prefix and the Unimodal Baseline, play out when we perform a contrastive objective based fusion as reported in [4]. It further supports the claims in Lemma 1 and Theorem 1 that as the number of modalities increase, the modality undergoing collapse contributes less and less to the downstream representation used to encode the semantics, irrespective of the fusion strategy.\n\nUnfortunately, we could not perform this experiment on the generative models [1-3], as it would require training each of the generative models from scratch for a number of multimodal combinations, which would take several weeks on our available compute resources, and hence, is infeasible during the rebuttal timeline.\n\nHowever, we were able to perform the rank evaluation in both the contrastive and generative settings, since pretrained models are available in some cases for the latter. Due to time constraints of the rebuttal period, we chose [1] as our representative generative model. Since the objective of generative modelling is somewhat different from the downstream application that we experimented with, to analyze [1], we performed the experiment on their proposed MNIST-SVHN dataset, while for [4], since it is for general representation learning, we applied their proposed contrastive objective to our baseline setting on MIMIC-IV. Below, we report the results of our experiment, where vanilla setting refers to the original model, without KD or EBR.\n\n\n|  |  |  |  | Beta  |  |  |\n| ----- | :---- | :---: | :---: | :---: | :---: | :---: |\n|  |  | **0** | **2** | **4** | **6** | **8** |\n| | **\\[1\\] Unimodal Baseline** |  |  | 198  |  |  |\n|  | **\\[4\\] Vanilla** | 477 | 421 | 398 | 110 | 96 |\n|  | **\\[1\\] \\+ KD** | 482 | 465 | 390 | 298 | 270 |\n|  **Rank** | **\\[1\\] \\+ EBR** | 485 | 477 | 431 | 405 | 395 |\n|  | **\\[4\\] Unimodal Baseline** |  |  | 1255  |  |  |\n|  | **\\[4\\] Vanilla** | 1877 | 1330 | 1146 | 930 | 872 |\n|  | **\\[4\\] \\+ KD** | 1905 | 1676 | 1533 | 1427 | 1390 |\n|  | **\\[4\\] \\+ EBR** | 1912 | 1825 | 1709 | 1600 | 1588 |\n\nWe can see that in both the generative and contrastive settings, the ranks consistently drop as the strength of the modality undergoing collapse $\\beta$ (written as Beta in the table) is increased. The drop is sharp around a critical point, where the rank goes below the unimodal baseline, depicting a form of phase transition, a phenomenon also observed in our original experiments (Sec. 4.2 Observations and Analyses). Finally, the dropping rank can be counteracted by implicit (KD), and even more effectively, by explicit basis reallocation (EBR), which results in a much more stable rank across the range of different values of $\\beta$.", "claims_and_evidence": "The major claim of the paper is that modality collapse emerges from the entanglement of predictive features from one modality with noisy features of another modality. This claim is significantly supported by both theory and experimental evidence: the authors start by demonstrating in Section 3.1 that the proportion of cross-modal poly semantic neurons increases as the number of modalities increase. This theoretical observation is demonstrated empirically in Section 4.1. Similarly, in Section 4.1 the authors demonstrate that cross-modal interference results from the low-rank simplicity bias. Once again, this claim is empirically evaluated in Section 4.2. Finally. the authors propose Explicit Bias Rallocation (EBR) to address modality collapse. The authors evaluate the effectiveness of EBR extensively in Section 4.3., including results of dealing with missing modalities at inference time.", "methods_and_evaluation_criteria": "The proposed method appears sound and based on theoretical insights previously discussed in the paper. The evaluation datasets and baselines metrics are also in line with previous literature.", "theoretical_claims": "I reviewed the theoretical claims at a high level but did not verify the correctness of the proofs in detail. The arguments appear well-structured, though a deeper formal verification would be needed to confirm full correctness.", "experimental_designs_or_analyses": "The experimental setup presented in Section 4 is sound. Moreover, the authors do a great job in analyzing in depth each result and provide interesting insights.", "relation_to_broader_scientific_literature": "This paper contributes with a theoretical framework to understand modality collapse in multimodal representation learning models. As such, it builds on previous conjectures explored in Javaloy et al., 2022, and Ma et al., 2022. The insights from this work can also be applied to the development of novel multimodal representation learning methods, especially those that deal with large number of modalities.", "essential_references_not_discussed": "[1] also proposes a method to deal with missing modality information at test time, using a cross-modal contrastive loss.\n\n[1] Poklukar, Petra, et al. \"Geometric multimodal contrastive representation learning.\" International Conference on Machine Learning. PMLR, 2022.", "other_strengths_and_weaknesses": "One significant strength of the paper is the overall quality and completeness of the work: the authors present both interesting theoretical results and an extensive experimental setup.", "other_comments_or_suggestions": "It would further strengthen the paper if the evaluation presented in Section 4.1 and 4.2 contemplated as well other multimodal representation learning models, such as multimodal variational autoencoders [1-3], where the phenomena of modality collapse has been observed, and other contrastive learning models [4].\n\n[1] - Shi, Yuge, Brooks Paige, and Philip Torr. \"Variational mixture-of-experts autoencoders for multi-modal deep generative models.\" Advances in neural information processing systems 32 (2019).\n\n[2] - Wu, Mike, and Noah Goodman. \"Multimodal generative models for scalable weakly-supervised learning.\" Advances in neural information processing systems 31 (2018).\n\n[3] - Javaloy, Adrián, Maryam Meghdadi, and Isabel Valera. \"Mitigating modality collapse in multimodal VAEs via impartial optimization.\" International Conference on Machine Learning. PMLR, 2022.\n\n[4] - Poklukar, Petra, et al. \"Geometric multimodal contrastive representation learning.\" International Conference on Machine Learning. PMLR, 2022.", "questions_for_authors": "None.\n\n\n## Post Rebuttal Comment \nI thank the authors for their hard work in the rebuttal and for addressing my comments! I maintain my score for now, great job on the paper!", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the phenomenon of modality collapse in multimodal representation learning models. The authors first provide a theoretical analysis, showing that modality collapse occurs when predictive features from one modality become entangled with noise features from another, effectively suppressing the informative signal of the former. To address this, they propose a novel method called Explicit Basis Reallocation (EBR), which promotes disentanglement and denoising of multimodal embeddings. Extensive experiments support the theoretical findings and demonstrate state-of-the-art performance, particularly in scenarios involving missing modalities.

Reviewers found the paper makes a valuable contribution to the study of modality collapse in multimodal representation learning.
Decision: Accept (spotlight poster)