Supporting Multimodal Intermediate Fusion with Informatic Constraint and Distribution Coherence
Authors: Yi Li
Fei Song
Changwen Zheng
Jiangmeng Li
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal representation learning; Generalization error;  Informatic constraint; Distribution cohering
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Based on the prevalent intermediate fusion (IF) and late fusion (LF) frameworks, multimodal representation learning (MML) demonstrates its superiority over unimodal representation learning. To investigate the intrinsic factors underlying the empirical success of MML, research grounded in theoretical justifications from the perspective of generalization error has emerged. However, these provable MML studies derive the theoretical findings based on LF, while theoretical exploration based on IF remains scarce. This naturally gives rise to a question: **Can we design a comprehensive MML approach supported by the sufficient theoretical analysis across fusion types?** To this end, we revisit the IF and LF paradigms from a fine-grained dimensional perspective. The derived theoretical evidence sufficiently establishes the superiority of IF over LF under a specific constraint. Based on a general $K$-Lipschitz continuity assumption, we derive the generalization error upper bound of the IF-based methods, indicating that eliminating the distribution incoherence can improve the generalizability of IF-based MML methods. Building upon these theoretical insights, we establish a novel IF-based MML method, which introduces the informatic constraint and performs distribution cohering. Extensive experimental results on multiple widely adopted datasets verify the effectiveness of the proposed method.

Published as a conference paper at ICLR 2026
SUPPORTING MULTIMODAL INTERMEDIATE FUSION WITH INFORMATIC CONSTRAINT AND DISTRIBUTION COHERENCE
Li Fei Song Song Changwen Zheng' & Jiangmeng Li 1. Institute of Software Chinese Academy of Sciences
University of Chinese Academy of Sciences
(1 (1iyi2022, songfe 2022, changwen, jiangmeng2019]8iise
ABSTRACT
Based on the prevalent intermediate fusion (IF) and late fusion (LF) frameworks, multimodal representation learning (MML) demonstrates its superiority over uni- modal representation learning. To investigate the intrinsic factors underlying the empirical success of MMI., research grounded in theoretical justifications from the perspective of generalization error has emerged. However, these provable MML studies derive the theoretical findings based on LF, while theoretical exploration based on IF remains scarce. This naturally gives rise to a question: Can we de- sign comprehensive MML approach supported by the sufficient theoretical analysis across fusion types? To this end, we revisit the IF and LF paradigms from iieeggraneed dimensional perspective. The derived theoretical evidence sufficiently establishes the superiority of IF over LF under a specific constraint. Based on general K -LLipschitz continuity assumption, we derive the general- ization error upper bound of the IF-based methods, indicating that eliminating the distribution incoherence can improve the generalizability of IF-based MML meth- ods. Building upon these theoretical insights, we establish a novel IF-based MML method, which introduces the informatic constraint and performs distribution co- hering. Extensive experimental results on multiple widely adopted datasets verify the effectiveness of the proposed method.
INTRODUCTION
Given the gradually increasing data from multiple modalities, multimodal representation learning (MML) demonstrates the potential for supporting the comprehension of complex patterns. Accord- ing to the feature mapping stages Wang et al. (2020), two widely adopted multimodal fusion types exist in recent MML studies, i.c., Teature-level intermediate fusion (IF) and decision-level late fu- sion (LF) IF integrates features from various modalities in the latent space, whereas LF merges the prediction logits in the target space. MML has recently arisen popular
of research in many fields, e.g., knowledge graph Cao et al. (2022): ; Lu et al. (2022), recommendation Zhou et al. (2023): CaaphCCao@e
Wei al. (2023): Li al. (2024), sentiment analysis Hazarika et al. (2020); [Li et al. (2023); Liu
et al. (2024). and so on. Besides the documented empirical success, studies Zhang et al. (2023b);
Cao et (2024) investigate the inherent mechanisms behind MML from the generalization error perspective, thereby providing theoretical supports for the multimodal models.
However, fhus-far provable works from the generalization error perspective derive theorems based on the LF framework, while the theoretical analysis focusing on the IF framework remains in- sufficiently explored. Theoretically, according to the theory of data processing inequality Cover & Thomas (2001), IF-based methods may contain more task-dependent information. Empirically, we conduct exploratory experiments by substituting the framework of two representative L.F-aaed methods (PDF et al. (024)) and QMF Zhang et al. (2023b)) with IF. As illustrated in Figure
Equal contribution. Corresponning author.
Farly fusion aggregates the original data directly, which is impractical in real-world scenarios due to the heterogeneity of multimodal data. Therefore, we remove early fusion from the consideration.

---

Published as a conference paper at ICLR 2026
(8)
QMP
QMF (IF)
(b)
PDF
PDF (IF)
Acorcyy 1%
592
+0.27
69:67
20.44
$5.1
A6-43
1021
+9.12
92.51
V2
20:04
80.56
10.56
10.14
59.44
J1.43
#018
$6.33
10.8
1.022
15.57
2012
562
37.8
58.7
10.2
582
15.5
12.08
MVSA-Single
MVSA-Mulliple
HFM
Food101
MVSA-Single
MVSA--Muliile
Foodin1
Figure 1: QMF (IF) replaces the LF framework in QMF with the IF, and the same applies to PDF (IF). -insee HIVSA-Multiple, HFM, and Food101 are four vision- lision-languag datasets.
IF-based methods consistently outperform their LF- -a-baded counterparts on four multimodal datasets. Despite the theoretical and empirical potentials of IF's ascendancy over LF, the theoretical supports behind IF-based MML models require further exploration. To this end, we revisit the IF and LF paradigms from fine-grained dimensional perspective. With rigorous deduction, we demonstrate the superiority of IF over LF under specific constraint. Therefore, we design our model based on the IF framework and incorporate a specific informatic constraint. The informatic constraint im- poses a regularization on parameters of the linear target mapping in IF-based MML models from the information theory perspective Tishby et al. (2000). Such an informatic constraint can sufficiently guarantee the superiority of IF over LF.
To further explore the inherent mech- anism behind IF- MML models,
Sample-bas Sinkhom
150
we formalize the generalization er- ror upper bound of Ia-bd methods, which is derived by adhering to a gen- eral K K--ipschitz continuity assump- tion on the linear target mapping. Ob- serving the generalization error up- per bound, we reveal that eliminating the distribution incoherence im- prove the generalization performance of IF-based MML models. Thus, we determine to employ Wasserstein distance to conduct distribution co- hering for its favorable properties.
100
0.00
0.228
0,00
0.003
0.215
Figure 2: On the Single dataset, we leverage three methods estimate Wasserstein distance and record the av- erage time cost per 50 batches. The ground truth Wasser- stein distance is obtained by applying the Sinkhorn algo- rithm to all high-dimensional features in each batch. The results show that directly computing Wasserstein distance is impracticable because of the high time complexity. From the left column of the figure, it can be observed that com- pared to sampling-base -baed Sinkhorn and RTN, our method achieves a more accurate Wasserstein distance estimation with limited increase in time complexity.
Directly calculating Wasserstein dis- tance Cuturi (2013) between high- dimensional features requires huge computational complexity. Accord- ingly, two main categories of methods are proposed to practically estimate Wasserstein distance: (i) Sampling- bampling-based based Sinkhorn Cao et al. (2022); Li et al. [203); (ii) Radon transform-bast nonlin- ear neural network calculation (RTN) Bonneel et al. (2015): Kolouri et al. (2019); Chen et al. (2022); Sugimoto et al. (2024). Nevertheless, due to the incompleteness of the partial sampling strategy in sampling-basee based Sinkhorn and the inaccuracy of fitted non non-linear functions in RTN, current methods suffer from the degraded estimation of Wasserstein distance, as demonstrated in Figure 2 To ad- dress this issue, we propose a novel estimation method of Wasserstein distance, which introduces a restricted isometric dimensionality reduction technique, and design a Lagrange regularization to enhance robustness to the semantic disturbance during dimensionality reduction. This approach em- powers us to omit the partial sampling strategy and the nonlinear neural network, thus achieving distribution cohering effectively with limited computational complexity. The empirical evidence in Figure 2 verifies our statement.
In nutshell, we propose novel IF-based MML method with solid theoretical supports, namely Intermediate Fusion with Informatic Constraint and Distribution Coherence (IID). Our major con- tribution is four-fold:
(1) From a fine- ane-grained dimensional perspective, we rethink the two prevalent fusion types of MML, ie., IF and LF. We theoretically demonstrate the superiority of IF-dd methods over La-based counterparts based on a specific constraint. (2) Based on the K-Lipschitz continuity assumption on the linear target mapping, we derive the generalization error upper bound of IF-hased methods,
2

---

Published as a conference paper at ICLR 2026
which indicates that mitigating the distribution incoherence improve the generalizability of IF- based MML models. (3) Adhering to theoretical analyses, we propose a novel IF-based MML model, encompassing informatic linear target mapping constraint and distribution cohering with restricted isometric dimensionality reduction. (4) Empirically, we conduct extensive experimentssss on representative benchmarks to prove the effectiveness of IID.
RELATED WORK
In recent years, the expansion of available data has significantly propelled advancements in the fields of computer vision Krizhevsky et al. 22012);
; He al. 2016); Huang et al. (011)) Dosovitskiy et al. (202T) and natural language processing Pennington et al. (2014a); Vaswani et al. (2017); Devlin et al. (201bb), enabling the development of more robust and sophisticated applications. However, these models focus on the processing of unimodal data (c.g.. images and text). As the semantics ex- tracted from unimodal data approach its bottleneck, MML has garnered increasing attention from the research community. By exploring both the modality -shared and modality-specife task-dependent discriminative knowledge, MML demonstrates its superiority in fields involving various modality combinations, like audio-vidco-text Liu et al. (2024): Hazaríka et al. (2020). image-texts Li et al.
2023
Ma et al. (2024) graph-image-texts Wei al. 2023) Cao al. (2022) and on.
Besides the documented empirical success, research endeavoring to understand MML with theoret- ical justifications has started emerge. E.g.. Huang et al. (202I) rigorously demonstrates that the reason why MML outperforms unimodal methods lies in its access to a superior latent space repre- sentation. Huang et al. (2022) substantiates the existence of modality competition, which renders the joint training of multimodal networks challenging, thereby leading to suboptimal performance. Be- yond the exploration of the intrinsic mechanism MML, several works develop multimodal models under the theoretical guidance of generalization error and yield great success. Specifically, QMF Zhang et al. (2023b) is designed by the theoretical derivation that the negative correlation between a specific modality's 's fusion weight and empirical error decrease the generalization error. CDF ct al. (2024) is proposed based on the provable elucidation that the reduction of generalization error primarily stems from the negative covariance between fusion weights and the loss associated with the current modality, as well as the positive covariance between fusion weights and the loss of other modalities. Due to the inherent correspondence between the ensemble-like LF framework and the extensively investigated field of ensemble learning Qiao & Peng (2024); Wood et al. (0023), these works consistently derive the theoretical findings based on the LF framework, thus resulting in the sparse theoretical exploration based on IF. In contrast to prior research, we design comprehensive MML approach, supported by a complete theoretical analysis across fusion types.
3 THEORETICAL INSIGHTS
This section presents our theoretical insights, and we offer a concise overview of the proposed theorems with complete proofs deferred to Appendix(A.2
We first provide the basic notations of MML. We denote the input space, latent space, and target space by X, Z, and V, respectively. Given multimodal learning task, the training dataset Drain comprises instances of the form (æx, y). which are sampled from the distribution D E X x y. æ the multimodal sample and y is the corresponding label. Two mappings are defined to assist our theoretical analysis: (2) latent mapping h(-): X + Z, which takes an input from the input space X' and projects into the latent space Z: (Łi) target mapping g(-) Z V, which takes latent features from the latent space Z and maps them to the target space y. The formula f go ho(x), abbreviated as f = gh(xr), ), is a composite function of g-)) and h(-). Our objective to learn multimodal model f that performs well on the unknown test dataset Diest which is also drawn from D.
3.1 REVISITING THE IF AND LF PARADIGMS: FINE-GRAINED DIMENSIONAL PERSPECTIVE
For the sake of simplicity and without loss of generality, we perform the theoretical analysis within the scenario involving two modalities. Given the input multimodal data æ = (2), we employ latent mappings to obtain the corresponding features by 2 ht (z(1) and ø2 (x2). Given the
m-th (m E (1,2)) modality-speciff fusion weight n > 0 and D w 1, for LF, the final
3

---

Published as a conference paper at ICLR 2026
prediction logits fir(()) Em= w"" "0... h=m while for IF, fir(z) = go 100 -= w h"
(
It can be seen that each modality has its specific target mapping 9 (-) parameterized by 0n in LF. In contrast, IF leverages a common target mapping g( parameterized by for multiple modalities. Being consistent with
major baseline Zhang et al. (023a); Cao et al. (2024), we employ linear
classification layer as our target mapping, which is a widely adopted setting in multimodal learning tasks Anderson et al. (2018): Han et al. (2021): Cao et al. (2024).
We assume that 21 and Z2 share the same dimension IR", which can be realized easily by a linear transform in practice. Intuitively, in the image classification task, a specific pixel of picture either belongs to the task-dependent foreground or to the task-irrelevant background. Analogously, from a fine-grained perspective, each dimension of latent features 21, #2 (e.g., 21, and 22.n. S n S d) is either task-dependent semantics or task-independen noise. We provide the definition of task- dependent semantic and task indeepndedntt noisy dimensions.
(Semantic and noisy dimensions). If masking given dimension results in decrease of the ervor between the model's predictions and the ground truth label, the dimension is classified as task-dependent semantic dimension; conversely, the dimension is classified task-independent noisy dimension.
Definition 1
Thus, there are two partitions corresponding to per latent feature, i.e., 2 = {zI.s. 21.N, 22 = (2 22.Na ). where Sm and Nm, denote the index sets of semantic dimensions and noisy di- mensions, respectively, corresponding to the m-th modality. S₁ n N₁ 2 and Sy n N₂ = Ø since a certain dimension cannot be semantics and noise simultancously. In LF, the parameters (01, 02) of the target mappings also have two partitions corresponding to the input latent features, i... (01.s.. 0 0I.N, O2 (02.52. 02.Nz). Then the prediction logits of LF can be formalized as
+ w² N2 02.N2. (1) fur(#) 0,) w² (z262) w 11,; 01,s: + w N1,N; 01,N, + w² 22.82 02,32
While in IF, the multimodal feature is obtained in latent space by z = ===1 + w #2. thus each dimension of has four possible scenarios:
Da,s = S1 O S2, a combination of the semantic dimensions of x1 and ø2;
Ds,N, S₁ N2, combination of the semantic dimension of and the noisy dimension of #23
DNSS. NN n S2, a combination of the noisy dimension of æ and the semantic dimension of #2;
DNIN, = N₁ O N2. combination of the noisy dimensions of 2 and #2.
Briefly, can be partitioned into four components { 8, 2Ds, No 2Dx, , 2D N2 } and arbi-
trary in {Ts,, Ds, ND,,,,
Sy: ID No disjoint obviously. Corresponding to the fused multimodal feature z, the parameter of the target mapping has four partitions, i... (0p (0Ds,sy ODS N₂* 0DN 0DN,N N2 Accordingly, the prediction logits of is
fi=(x) O 202, S2 Ops,e, #0s, OnsiNg + SDN, ODx, 82 +2N 088
(2)
Then, we can derive the following Theorem [
Theorem (Prediction comparisons of IF and LF). For each input multinodal sample (x, v), there constantly exists a set of parameters A, such that the following equation holds for the linear target mapping characterized by 0 A.:
((ferr())))) S ((frr(x),)),
(3)
where L(.s-) is Cross-Entropy loss function. Given the Bayes optimal hypiiheyyttthesit f*, which achieves the infinum of the errors R' on D, i.e,, f argmin, R(f) = argmin, B(m.))=D([(( (z())) and for each E [0. [(forrr
((fir()),)) there exists corresponding O' e A s.t.
((fir(#))))
The proof of Theorem can be found in Appendix A.2. We omit the explicit notation of target mappings' parameters LF-based prediction, since Theorem holds for arbitrary parameters of target mappings in LF models (This paper follows this notation principle throughout). Theorem confirms that a simple linear target mapping characterized by the parameters in A can establish the superiority of over LF. Additionally, there theoretically exists a that allows the Ia-ed prediction be closer to Bayesian optimal prediction compared to those of the LF models.
4

---

Published as a conference paper at ICLR 2026
3.2 ANALYSIS OF THE GENERALIZATION ERROR
Based on Theorem | we present our theorem regarding the generalization error of IF and LF. The generalization error is a metric that measures the generalization performance of the learned multi- modal model f, which can be defined as: 49 = -= v)l. Theorem delineates the comparison of generalization errors between IF and LFF
Theorem (Generalization errors of IF and LF). With a linear target mapping ge in IF parame- terized by 0 E A, the following equation holds: IIF S fLF.
Theorem 2 is proven in Appendix A.2.2 which indicates that IF with linear target mapping ga(.) can exhibit lower generalization error than LF consistently. We further introduce Assumption I to investigate the factors impacting the generalization error of IF-based MML methods.
Assumption (K-Lipschitz continuity). Suppose the function ((z) = L(g(z),8) is a K- Lipschitz. continuous function in respect to input z, then for K 0. 4 a, E Do (Do is the definitional domain of Φ). we have: || [1a((a) (b)
|| K
a a- bl.
Analogous assumption has also been adopted in Qiao et al. (2025). various existing works Arjovsky & Bottou (2017); Arjovsky et al. (2017): Cao al. (2022) introduce the constraint of K K-Lipschitz continuity assumption within their theoretical analysis, demonstrating the generality of K-Lipschitz continuity constraint. Furthermore, relevant studies Yoshida & Miyato (2017); Gulrajani al. (2017) declare that K K-Liisschiz continuous function can be easily constructed. The literature indicates that K K--ipsshitz continuity constitutes mild assumption.
I [Periil be the train- Theorem (Generalization error upper bound of IF), Let Duain (a', yf Ji=1
ing datasel and DM be a complete distribution distance metric. Under the constraint condition of Assumption 7 for any fir with the linear target mapping ge parameterized by 0 E A in hypothesis space H and
1, with the probability at least 1 ö, the generalization error of fir holds:
M
SIFO S
Σ m=1
K-B(w"))
DM (M (1)
+Error (w" L(go (2.))(2m),20) ). + + Bias DIHH) 13N
Distribution incuherence
(4)
The corresponding proof of Theorem 3 can be found in Appendix A.2.3 E(w")) represents the expectation of multimodal fusion weight, DM is the complete distribution distance metric which satisfies the three essential properties (non-negativity, symmetry, triangle inequality). pm is the distribution that the features of the m-th modality are drawn from, i is the distribution that the multimodal feature z follows. Distribution incoherence quantifies the discrepancy between the dis-
tributions μ and Mm (m [1, MD. B((IF) is the empirical егrог of multimodal feature on Dinin- Bias (HH),, ONN-1/2) is the systematic bias with respect to Rademacher complexity R of the hy- pothesis space H and the size training dataset N. It's challenging to climinate the systematic bias in MML models. Error|w" [(go(=m)) y)] is an eтror term about the fusion weight w.t and unimodal loss L(go(Zm) y). which indicates that the calculation method of fusion weight w can affect the predictive performance of MML models. Recent research Zhang al. (2023a): Cao al. 2024) focuses on exploring the effective fusion weights w" to achieve better performance of MML models, leaving the diminution of the distribution incoherence term unexplored.
we determine to implement our model based on the IF framework with a linear target mapping characterized by A. According to Eq.(4) in Theorem 3, we propose to diminish the value of the distribution incoherence term, thereby further enhancing the generalizability of our method.
Consequently, inspired by Theorems 1 andd] 2
4 METHODOLOGY
Overview of IID. The framework of the proposed HID illustrated in Figure 3
Drawing upon T 2 we build our model based on the IF framework. Concretely, given a batch of input multimodal samples {(x', .y'), (x?, ..?.)). ..... Nx,, )), N is the batch size, each instance has M modalities, i.e., x² (2t, 3M (i E [1, ND. and we obtain the corresponding features by
5

---

Published as a conference paper at ICLR 2026
Fusion weights of modality
Cie
9e
(Outcast out last week. (i revirns
Interssediate fatiin and lineur target mupping with informuticc constraiu
Fusioa weights modalit 2
Distribatinn cohering with restricted isometric diseensionaity
F()
Preserving dj largest compants
Visualizariin of PO
Projection
Transport plan T
The Imput
La
features
C
X Matrix multiplication
Weight summation
F(-) Fast Fourier transform
Element-wisc multiplication
Figure 3: The overall architecture IID, which is built based on prevalent framework. The pipeline is illustrated under the scenario of two modalities without loss of generality. The proposed informatic constraint on linear target mapping and distribution cohering with restricted isometric dimensionality reduction bridges our theoretical framework and practical methodology seamlessly.
latent mappings, i.e., = = (x()), where m E [1. Then we obtain the multimodal feature x² of x in latent space via
(S)
which is prevalent IF paradigm, and we calculate the prediction logits by fu=((")) - g(21).
The derivations of Theorems[and]33 To actualize such a specific and accessible linear target mapping, we introduce the meticulously designed informatic constraint, which guarantees that the parameter of the linear target mapping is restricted to the desired set A and converges towards the theoretically optimal parameter Ø* during the training process. Under the guidance of Theorem 3 we propose the distribution cohering with restricted isometric dimensionality reduction module to diminish the distribution incoherence term in (q.44), thereby improving the generalizability of the proposed IID.
based on the linear target mapping parameterized by A.
4.1 LINEAR TARGET MAPPING WITH INFORMATIC CONSTRAINT
In this section, we introduce the informatic constraint to attain the expected linear target mapping. As delineated Theorem based on E A, we have (( S C(fir(=)) v"), which equals L(z 0,y²) = w L(=),, 0m: O':yy). Therefore, given the initial parameter Ô of the linear target mapping, can constrain the parameter Ô in A and approximate it to the optimal parameter 0* during the optimization process by:
Min L( 0.v') ΣM M
w C(z 0m.
(6)
Nevertheless, IID is established based the IF framework, which renders the unavailability of LF-based Cross-Entropy loss function (i.e., L( Om,y')), ultimately leading to incalculable (q.(6) But we note that recent research regarding information theory Tishby et al. (2000); Fed- eriei al. (2020); Li et al. (2024) maximizes mutual information by minimizing Cross-Entropy function, which manifests that higher mutual information / (z2; indicates lower L(x Ô v²). Drawing inspiration from this finding, we inversely implement (q. (6) by
Max I(>":v')
Σ I(2i: :
dz dz dy'. Although Eq. is necessary but not suf- ficient condition of (). achieving the optimization objective through the necessary condition is
where 1(>1;y') = f p(21, ( log
<''' p(yr)

---

Published as a conference paper at ICLR 2026
practical and general Jiang & Veitch (2022); Zhang al. (2024). and the empirical results in Section S confirm the effectiveness of such an implementation.
Compared to Eq. we omit the multimodal fusion weight w in Eq. since maximizing -I (zmi 3) equals maximizing - [= y) with wnth > O. Eventually, we can implement Eq.
by minimizing the loss function:
Lic - -log AKL AKL (N++IW))
Σ M m
log ( AKL (Nag, [1M7)
(8)
The derivation of Lie is detailed in Appendix AA2.44 (-I)) is the variational approximation of p(-).), which is calculated by the target mapping. À is trade-off haper-parameter. K TT)() is Kullback- Leibler divergence Van Erven et al. ()014).N (N ) is a Gaussian distribution fitted by the mean and variance of z² ((m)) N is the standard Gaussian distribution.
4.2 DISTRIBUTION COHERING WITH RESTRICTED ISOMETRIC DIMENSIONALITY REDUCTION
m=1 ((w"")D (pm, (u) is obtaining the distribution barycenter Agueh& & Carlier (01T). but such a strategy is very computationally expensive Ngguyen et al. (0255)
A direct approach to minimize Σm=
According to our derivation in Appendix [A225 utilizing the IF approach specified in Eq. for the integration of unimodal features, the tollowing equation holds for almost all the multimodal
scenarios:
mm B(w" (M(m ΣΣ
DM (M Mmg):
(9)
where m., m2 M] and + m2. 9) indicates that the distribution incoherence term is bounded by the inter-modalit, distribution discrepancy, thus we can achieve distribution cohering by minimizing the right-hand side of (9) Considering that Wasserstein distance possesses the requisite properties of complete distribution distance metric, we determine to accomplish DM by Wasserstein distance (detailed in Appendix A.3) Sinkhorn algorithm Cuturi (2013) can achieve precise Wasserstein distance calculation. But in practice, performing Sinkhorn algorithm on high- dimensional features is problematic for its excessive computational complexity.
We ascertain the underlying cause by first restating the operating mechanism of canonical Sinkhorn algorithm. Given two probability distribution P1.P2 with discrete supports u = [o]
{u Ey= U = and E Uk = 1), Wasserstein distance can be calculated as follows:
W(P. P2) min Σm Exe
h=1 TyuCsa subject to T' R x"= T'mng u. 1., 1,,
(10)
T is the transport plan and Cjk evaluates the distance between ūj and ug- During the iterative computation of the optimal transport plan T,, each element of the matrix C is derived from the pairwise distance between features. Consequently, a large feature dimensionality incurs substantial computational complexity, which renders the Sinkhorn algorithm computationally problematic for high-dimensional features.
Inspired by the studies Wright & Ma (2022); Radhakrishnan et al. (2225) indicating that the features of data from multiple sources (such as signal, image, and so on) are generally sparse in the fre- quency domain, we opt to transform nigh-dimensional sparse features into low-dimensional dense features to accelerate Sinkhorn algorithm. Beyond improving computational efficiency, to mitigate the degradation of Wasserstein distance estimation precision caused by dimensionality reduction. we particularly impose a dimensionality reduction matrix with Restricted Isometry Property (RIP 2 and can maintain the geometric structure of features during the dimensionality reduction.
Specifically, we first employ fast Fourier transform to transform feature = into the frequency do- main: 2,,, F(z!.)) = S (t) e -2xi-= dt, where (t) is the Fourier series expansion of 2 We
further strengthen the sparsity of 2,, by preserve the top-d1 (d1 < d) principal components by
if 1 2
2
(11)
O, if
| <
?A transform A satisfies RIP if (1 8') || 1r12= S || A 11 (1 (1)+8)11 + |

---

Published as a conference paper at ICLR 2026
Table 1: Results on four vision-language datasets. Bold represents the best results. D stands for dynamic fusion, i.e., the fusion weight w" is function of x In contrast, the w is a constant in the static fusion (S) method. We obtain the p-value of IID-P by performing the student t-test between IID-P and PDE, the same applies to the p-value of IID-Q and IID-L.
MVSA MVgge-Sinle
MVSA-Multiplc
HFM
Food101
Baseline
Type
Avg
Worst
Avg
Worst
Avg
Worst
Avg
Worst
Bow
s
48.79 L 7.05
35.45
64.78 1 0.81
64.18
74.22 1 0.87
73.25
82.50
1 0.18
82.32
Img
s
64.12 + 1.23
62.04
67.04
1 0.49
66.65
74.74 1 0.38
74.36
64.62
0.40
64.22
BERT
75.61 = 0.53
74.76
69.39 + 0.37
69.18
85.34 0.46
84.86
86.46
0.05
86,42
Late-fusion
76.88 1 1.30
74.76
67.94 1 0.56
67.41
85.51 1.111.
85.31
90.69
0.12
90.58
C-Bow
S
64.08 + 1.54
62.04
67.35 1 0.20
67.24
76.53 0.23
76.28
70.77
0.09
70.68
C-BERT
S
65.59 = 1.33
64.74
67.71 = 1.06
66.59
85.82 + 1.06
84.76
88.20 0.34
87.81
MMBT
D
78.50 1 0.40
78.04
69.88 1 0.31
69.71
85,39 1 0.34
85.01
91.52 1 0.10
91.38
TMC
D
74.87 + 2.24
71.10
68.41
1: 0.16
68.29
85.18 0.79
84.55
89.86 1 0.07
89.80
DYNMM
D
79.07 = 0.53
78.23
68.55
+
68.32
85.32
84.96
92.59 0.07
92.50
LCKD
62.44 1 0.30
62.27
1 0.13
65.93
82.43
81.87
85.32
84.26
QMF
D
78.07 + 1.10
76.30
68.67
0.22
68.41
85.87 0.23
85,66
92.92
0.11
92.72
UNICODE
S
66.97 = 0.39
65.94
+.32
65.98
83.37 + 0.52
82.83
88.39
0.36
87.21
SimMMDG
S
67.08 + 0.35
66.35
66.44
+: 0.23
66.19
84.13 +: 0.41
83.85
89.57
0.38
88.43
PDF
D
79.94 L 0.95
78.42
69.54
1 0.25
69.26
86.03 1 0.31
85.77
0.22 93.32
92.84
IID-L
77.78 +: 1.09
75.89
69.32 1 0,50
67.84
$5,94 0.42
85.41
91.93 0.25
91.21
p-value
5.47e
6.67e
4.34e
9.87e 3
IID-Q
80.02 = 0.40
79.58
71.08
0.30
70.76
86.61
0,23 4.97c
86.37
93. 10 0.03
93.06
p-value
1.07c
4.74c
3. 69c
IDD-P
D
81.13 + 0.84
79.98
71.23 + 0.44
70.81
86.88 1 0.39
86.32
93.73 0.14
93.52
p-value
3.34e
9.34e
6.72c
1.51e
T set to the magnitude of the d1-th -th largest component of # d1 a hyperparameter, and n E [1, d] is the dimension index. The enhancement of sparsity not only mitigates the interference of noisy semantics but also alleviates the risk of mapping two disparate high-dimensional features to an identical low-dimensional representation during dimensionality reduction.
Then we design a dimensionality reduction matrix with RIP. Let denote the Gaussian Random ma- trix, as the elements sampled from N are highly uncorrelated with the bases of the Fourier transform. V FJ can be treated as the RIP-preserved dimensionality reduction matrix Wright & Ma 2022). Thus employ mpeility- L m for the dimensionality reduction: Zy V m 2i where 2 e Idd,, 2 E Cd, and v m E Cd xd Since d1 < d and the RIP of Le m. the upper bound of distribution incoherence can be accurately measured with limited computation complexity.
Additionally, the loss of semantics derived from dimensionality reduction is inevitable. To enhance robustness such an undesirable disturbance, it plausible to relax the original hard marginal matching constraints, which allows for a flexible assignment of matching mass. Overall, we intro- duce relaxed constraint in the form of the Lagrange multiplier method, and ultimately calculate Wasserstein distance between modalities my and m2 the following manner:
W(Hmy Hmwg) = argmin Σn Σ n2 TyhCs A1 [L[TI..g u) + KL(T") 1,, [|
(12)
where Cjk- =|| V (2), V (2()) 1l2. Then distribution incoherence is bounded by Lae Em, .m W(pmy (mm)) ) and the final loss function of IID can be formalized as:
LID = aCio + 3Lde + - c(frr(a'),)))
(13)
a, 3 are the hyperparameters to control the influence of Lio and Lae. The overall training pipeline is depicted in Algorithm 1
5 RESULTS
In this section, we evaluate the performance of IID three multimodal tasks (i.e., vision-language classification, link prediction, and scene recognition) involving eight datasets. Based on Equation 4p. the calculation of " affects the predictive capability of MML models. For comprehensive and
8

---

Published as a conference paper at ICLR 2026
Table 2: The link prediction results on two multimodal knowledge graph datasets.
PB-IMG
WN9-IMG
Model
Type
MRR
H@1 1
H663
H 10
MRR
H@1
H
H 10
TransE DistMult
0.712
0.618
0.781
0.859
0.865
0.765
0.816
0.871
Unimodal
0.706
0.606
0.742
0.808
0.901
0.895
0.913
0,925
ComplEx RotatE
0.808
0.757
0.845
0.892
0,908
0.903
0.907
0,928
0.794
0.744
0.827
0.883
0,910
0.901
0.915
0.926
TransAB IKLR TBKGE
0.742
0.691
0.785
0.844
0.898
0.894
0.908
0.922
0.755
0.698
0.794
0.857
0.901
0.900
0.912
0.928
Multimodal
0.812
0.764
0.850
0,902
0.912
0.904
0.914
0,931
MMKRL
0.827
0.783
0.857
0,906
0.913
0.905
0.917
0.932
ОТКGR
0.843
0.799
0.876
0.916
0.923
0.911
0.930
0.947
MMKRL+IID OTKGE=IID
Multimodul
0.844
0.801
0.876
0,917
0.920
0.911
0.925
0.945
0.855
0.813
0.887
0.925
0.932
0.917
0.938
0.957
fair comparisons, we implement one static IID (i.e., IID-L, the w in IID-L is identical to vanilla Late-fusion), and two dynamic IIDs (i.e., IID-P and IID-Q. the w in IID-P and IID-Q are identical to PDF and QMF, respectively) across six datasets involved in vision- lasguanla classification and scene recognition tasks. As for the link prediction task on two multimodal knowledge graph datasets, we integrate the two proposed modules into competitive IF-based benchmarks. Each experiment is repeated three times. Due to the limited space, datasets, baselines, implementation details, and extended experiments are depicted in Appendix A.5
A.5 and A.6
Quantitative results. The quantitative results of vision- vision-language language classification and scene recognition are depicted in Tables [ and 3 respectively. All com-
Table 3: Results of scene recognition.
Baseline
Type
NYU Depth V2 Wont
SUN RGB
Worst
parisons performed in terms of both the average and worst-case accuracy metries. Under these metrics, the proposed IID-Q and IID-P attain the Top-2 perfor- mance on all six datasets. This outcome underscores the superior generalization capability of our models in comparison to the chosen benchmarks. Addition- ally, PDF and IID-P (QMF and IID-Q, Late-fusion and IID-L) adopt the identical implementation of fu- sion weights, thus the comparisons between these pairs can further verify the effectiveness of the proposed two modules. According to the quantitative results, the proposed IF-based models consistently outperform their LF-based counterparts. Furthermore, we conduct the Student t-test Kim (0015). in which p 0.05 indicates significant difference between the two groups of accuracy samples. Based on the results of Student (-test in Tables I 3 the p-values are all less than 0.05, thus we can attribute the performance improvement the two proposed techniques, rather than the randomness.
RGB
62.65
1:22
62.56
52.99
0.83
56.51
Depth
5
63.30
1 0.48
61.01
56.78
0.19
$1.32
Late fusion
69.14
1 1.667
68:35
02:00
0.15
00:55
Concal
30.31
+0880
69:42
62:48
0.50
61.19
Align
S
70.31
+.28
68:50
61.12
0.61
60.12
MMTM
D
71.04
1:0.4
70.18
61.72
0.67
60:94
TMC
D
71.06+
+-076
69:57
60
0.24
60.31
LCKD
S
68.01 1.3.31
66.15
56.43
0,56
56.32
QMF
70,09 + 0.57
68.11
02.09
0.56
61.30
UnICODE
70.12 0:37
68.74
59.21
= 0.55
58.55
SimMMDG
11.34+--.33
70.29
60.54
0.50
60.31
PDF
71.37 +.066
70.18
62.34
0.43
61.88
69.87 +.78
68.78
62:31
0.21
60.76
Pallu
9.12e-3
4.86g=2
IID-Q
71.61 #
71.25
62.92
0.13
62.78
paslue
5.8e-3
3.81c
LD P
72,04 1
7144
62.99
0.34
02.71
p-vsllue
188h
8.83
We employ four evaluation metrics to assess the performance on the link prediction task: the Mean Reciprocal Rank (MRR) of the correct entities, and @its@k, defined the proportion of test in- stances in which the correct entity is ranked within the top-k predictions, where k E {1, 3. 10). Big MRR and Hits k indicate good result. We present the results of the link prediction task in Table 2 For the two existing IF-based benchmarks, MMKRL Lu et al. (022) and OTKGE Cao et al. (2022), we observe that integrating IID further improves their performance. In particular, OTKGE III achieves state-of- of-he-a-tteerttttt results on both multimodal knowledge graph datasets. The results indicate that the proposed method can serve as a plug-and-play module to enhance the performance of approaches based IF framework. Overall, the quantitative results eight datasets, covering three distinct tasks with diverse modality combinations, validate the effectiveness of the proposed method and further attest to its generalization capability.
Ablation study. To investigate the contribution of each ingredient, two variants are trained for justification: i) w/o D removes the distribution cohering with restricted isometric dimensionality reduction module; ii) w/o I excludes the informatic constraint on the linear target mapping. The results of the ablation study depicted in Table It can be seen that the performance of IID drops

---

Published as a conference paper at ICLR 2026
Table 4: The ablation study on six benchmark datasets.
IID-L
W/o
IDD Q
IDD-P
SingeSiugle
7144 0.7.3
77.47
1:10
77.78
1
79.32
0.73
78,95
0.57
SI.02
0.40
80.79 = 0.80
50.46
00.75
81.13 + (US4
MVSA Mullippe
69.03 0:441
68.11
0.766
69.32
0.50
69.59
1:20
70.67
0.29
71.08
0.30
70.13 = 0.52
70.61- 1: 00:55
71.23 0.44
HPM
85.21
0.53
85.94
86.54
0.25
86.22
86.61
0.25
86.35
0.59
96.38
Food101
91.35
92.98
Deych
09:41
0.1
70:95
0.40
70.45
71.61
0.50
71.75
71.58
0.92
SUN RGB-D
62.18 0:117
62.25
0.37
62.31
0.24
62.68
0.07
62,65
0.31
62.92
0.13
62.77 0.19
62.53 1: 0.38
62.99 0.34
regardless of which module is removed, suggesting that each proposed technique has a significant impact the predictive capability of IID.
Empirical demonstrations of the theo- retical derivations. The design of IID is grounded in two theoretical derivations: (i) Lie can restrict the parameter of linear target mapping in A and render the param- eter to approximate the optimal parameter 0" during the optimization process; (dd) faie reduces the generalization error of IID by mitigating the distribution incoherence, thus enhancing the classification performance. Then, we substantiate the correctness of our theo- retical derivations with the experimental results. In Figure 5 with informatic constraint Cie on the linear target mapping, the performance improvement of the IF-based methods compared to the LF- based methods increases, which indicates that Lic can lead the initial parameter of the linear target mapping to approach the theoretically optimal 0* In Figure we present the test classification ac- curacy of IID-Q and QMF (the left subfigure of Figure(4), along with the mean Wasserstein distance between various unimodal features for each batch of samples (the right subfigure of Figure The results confirm that the classification performance improves as Wasserstein distance decreases. This demonstrates the validity of our theoretical derivation, specifically that eliminating the distribution incoherence contributes to enhanced model prediction performance on unknown test sets.
(C
diannoe)
GMF
ONB
1651 19
MESA Sigll
Figure 4: The empirical demonstrations of theoretical derivations (2/2).
QMF and its variants
Vanilla Late-fusion and its variants
MVSA-Single
MVSA-Multiple
HFM
Food101
90.0
78.81
70.32
69.5
69.02
69.50
35.4
92.5
92.31
92.89
92.98
78.5
69.0
68.67
$6.3
85 qp
92.0
21.0
78.0
76.88
77.42
68.0
69.5
35.0
85.51
K5.77
91.0
90.69
91.10
76.5
607.5
61.94
.3.4
90.5
Cie
Eie
LF
IF
IF w/
Ca
Figure 5: The empirical demonstrations of the theoretical derivations (1/2). In this figure, LF denotes the LF-based models (e.g., QMF and Late-fusion), IF denotes the LF framework is replaced by the IF framework, and w/ Lie means imposing the informatic constraint on the linear target mapping.
CONCLUSION
In this paper, we rethink the prevalent IF and LF paradigms in MML from a fine-grained dimen- sional perspective. The complete theoretical derivations sufficiently establish the superiority of IF over LF under a specific constraint. Based on the general K K-Lipschitz continuity assumption on the linear target mapping, we formalize the generalization error upper bound of ba-based methods, which indicates that the generalization error upper bound can be further decreased by mitigating the distribution incoherence. Motivated by these theoretical insights, we propose IID, an IF-based ap- proach which incorporates linear target mapping with informatic constraint and distribution cohering with restricted isometric dimensionality reduction. Empirical evidence proves that our findings are solid and IID generally effective.
ACKNOWLEDGMENTS
This work is supported by National Natural Science Foundation of China No. 62406313, Postdoc- toral Fellowship Program of China Postdoctoral Science Foundation, Grant No. YJB20250283.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We ensure reproducibility by detailing experimental settings, datasets, and hyperparameters both Section and Appendix A.5 The source code is included in the supplementary materials to repro- duce the proposed method.
REFERENCES
Martial Agueh and Guillaume Carlier. Barycenters in the wasserstein space. SIAM Journal on Mathematical Analysis, 43(2):904-924, 2011.
Alexander A. Alemi al. Deep variational information bottleneck. In 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference net forum?id=
Track Proceedings. OpenReview.net, 2017. httpttttts//pppeeeeee
HyxQzBceg
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 6077-6086, 2018.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint 1oXiv:1701.4440 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks PMLR, 2017.
In International conference on machine learning, pp.
Roman Bachmann, Oguzhan F Kar, David Mizrahi, Ali Garjani, Mingfei Gao. David Griffiths, Jiaming Hu, Afshin Dehghan, and Amir Zamir. 4m-21: An any-to-any vision model for tens of tasks and modalities. Advances in Neural Information Processing Systems, 37:61872-61911, 2024.
Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. Freebase: collaboratively created graph database for structuring human knowledge. In Jason Tsong- Li Wang (ed.), Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10-12, 2008, pp. 1247-1250. ACM, 2008 doi: 1376746
10.1145/137661 1376746. URI. hol..htps:///eo http://di.orgg/////// 100
Nicolas Bonneel et al. Sliced and radon wasserstein barycenters of measures. J. Math. Imaging Vis., 51(1):22 2015. doi: 10. 10.1007/11055-0014-05000 610851-014-0550
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data, Advances in neural information pro- cessing systems, 26, 2013.
Yitao Cai, Huiyu Cai, and Xiaojun Wan. Multi-moda sareasm detection in twitter with hierarchical fusion model. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.). Proceedingy of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. Linguistics, 2019. doi: 10.18653//11p11-11202 1239.. URL
Association for Computational
Bing Cao, Yinan Xia, Yi Ding, Changqing Zhang, and Qinghua Hu. Predictive dynamic fusion. In Proceedings of the 41st International Conference on Machine Learning, pp. 5608-5628, 2024.
Zongsheng Cao, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. OTKGE: multi-modal knowledge graph embeddings via optimal transport. In NeurIPS, 2022.
Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. In Michel François Valstar, Andrew P. French, and Tony P. Pridmore (eds.), British Machine Vision Conference. BMVC 2014, Notting- ham. UK, September 1-5, 2014. BMVA Press, 2014. URL nattps://bme arwhiver.iieeeiie .rg... html
uk http///c////////pppee///ppppp//p

---

Published as a conference paper at ICLR 2026
Xiongjie Chen et al. Augmented sliced wasserstein distances. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
net http///rmmmmiiiii/aeraee
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley, 2001. ISBN 9780471062592.
doi: 10.1002/0471200611. URL httpp:://ci....oo org 1002/
047 200611
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Christopher J. C. Burges et al. (eds.), Advances in Neural Information Processing Sys- tems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Pro- ceedings of meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 2292 2300, 2013. URL /RLps//s///p http::/sw....n.ir.o..aaar//////a
-bt-:a33dd---bbbbaaaa html
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255 IEEE URL hwt.hhttps://em http///di.oo///////////////
Computer Society, 2009, doi: 10. 1100/C/P..0009.55555 22000900..
1169/CVPR. 0...0.555504444
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA. June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019a. doi: 10.18653/v1/119-144 URL Cittps:/ddo..oree 1ttp:/11..1655///mmm3/m3
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019b. doi: 10.18653/VI/N1- 1423. URL https://dol http://....rr/10..16663///////// 1tt.:///...1133////11/
Hao Dong, Ismail Nejjar, Han Sun, Eleni Chatzi, and Olga Fink. Simmmdg: simple and effective framework for mudti-modal domain generalization. Advances in Neural Information Processing Systems, 36178674-786633 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale. In 9th International Conference Learning Representattinss ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URUaattps://ppeeeeiiiee net/ forum? id=YicbFdNTTy,
David A Edwards. On the kantorovich-rubinteii theorem. Expositiones Mathematicae, 29(4): 387-398, 2011.
Marco Federici. Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In International Conference on Learning Representations, 2020.
Werner H Greub. Linear algebra, volume 23. Springer Science & Business Media, 2012.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Zongbo Han, Changqing Zhang, Huazhu Fu, and Jocy Tianyi Zhou. Trusted multi-view classifica- tion. arXiv preprint arXiv:2102.02051 2021.
12

---

Published as a conference paper at ICLR 2026
Devamanyu Hazarika al. MISA: modality-invarre invariant and -specifie representations for multimodal sentiment analysis. In Chang Wen Chen, Rita Cucchiara, Kian-Sheng Hua, Guo-Jun Qi. Elisa Ricci. Zhengyou Zhang, and Roger Zimmermann (eds.), MM '20: The 28th ACM International Conference on Mulrimedia, Virtual Event Seattle, WA, USA, October 12-16, 26, 2020, pp. 1122 1131. ACM, 2020. doi: 00.1145/394171..441333 0194171.441367
URL https://dot: org 10 1145/
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10. 10.1109/CVPR.2016
http::::: /ttp:s://dooo doi. org/ 10. 1109/111RRR... SAI.VVRRR.......
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Yu Huang et al. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34: 10944-00996, 2021.
Yu Huang et al. Modality competition: What makes joint training of multi-modal network fail in deep learning ((proabl(......ll)... In International conference on machine learning, pp. 9226-9259. PMLR, 2022.
Yibo Jiang and Victor Veitch. Invariant and transportable representations for anti-causal domain shifts. Advances in Neural Information Processing Systems, 35:20782-20794, 2022.
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. In Visually Grounded Interaction and Language (ViGIL), NeurfPS 2019 Workshop, Vancouver, Canada, December 13, 2019, 2019. URL https:
//vigilworkshop. gi github. 10/ sstaaicccapperaa 40 pdf
Tae Kyun Kim. T test as parametric statistic. Korean journal of anesthesiology, 68(6):540-546. 2015.
Soheil Kolouri al. Generalized sliced wasserstein distances. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 261-272, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Solomon Kullback. Kullback-leible divergence, 1951.
Yi Li, Qingmeng Zhu, Hao He, Ziyin Gu, and Changwen Zheng. Moe: Multi-modal sentiment analysis via optimal transport and contrastive interactions. In International Conference on Neural Information Processing, pp. 451. Springer, 2023.
Yi Li, Qingmeng Zhu, Changwen Zheng, and Jiangmeng Li. Msi: Multi-modal recommendation via superfluous semanties discarding and interaction preserving. In Proceedings of the 2024 International Conference on Multimedia Retrieval, pp. 814-823, 2024.
Yuanyuan Liu, Haoyu Zhang, Yibing Zhan, Zijing Chen, Guanghao Yin, Lin Wei, and Zhe Chen. Nosse-resistant multimodal transformer for emotion recognition. International Journal of Com- puler Vision, pp. 1-21, 2024.
Xinyu Lu, Lifang Wang, Zejun Jiang, Shichang He, and Shizhong Liu. Mmkrl: robust embedding approach for 1-18, 2022.
knowledge graph representation learning. Applied Intelligence, pp-
Jie Ma, Jun Liu, Qi Chai, Pinghui Wang, and Jing Tao. Diagram perception networks for textbook question answering via joint optimization. International Journal of Computer Vision, 132(5): 1578-1591, 2024.
13

---

Published as a conference paper at ICLR 2026
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen-
tations in vector space. In Yoshua Bengio and Yann LeCun ). 1st International Conference Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. URL
abs/ 3013011000 .3881
Hatem Mousselly-Sergiehe Teresa Botschen, Iryna Gurevych, and Stefan Roth. A multimodal translation-basee approach for knowledge graph representation learning. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pp. 225-234, 2018.
Khai Nguyen, Hai Nguyen, and Nhat Ho. Towards marginal fairness sliced wasserstein barycenter.
In The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL http:::://ppeeeeeeeee net forum?id NCqJPPCesd
Teng Niu, Shiai Zhu, Lei Pang, and Abdulmotaleb El-Saddik. Sentiment analysis on multi-view social data. In Qi Tian, Nicu Sebe, Guo-Jun Qi, Benoit Huet, Richang Hong, and Xueliang Liu (eds.), MultiMedia Modeling 22nd International Conference, MMM 2016, Miami, FL, USA, January 4-6, 2016, Proceedings, Part II, volume 9517 of Lecture Notes in Computer Science, pp. 15-27. Springer, 2016. doi: 110007/97--3311-220000 1007/978- 3-19
Wkedi-Kizza et al. Sorption kinetics and equilibria of organic pesticides in carbonatic soils from south florida. Journal of environmental quality, 35(1):268-277 2006.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, meeting of SIGDAT, a Special Interest Group of the ACL. pp. 1532-1543. ACL, 2014a. 3115/v1/a14-115
doi: 10.3115/V1/D14- 1162. URL httph:ttp///.mm...t. /tt:::/dooo.000 .rg. /10.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Alessandro Moschitti, Bo Pang, and Walter Daclemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, Special Interest Group of the ACL, pp. 1532 1532-1543, 1543. ACL,, 2014b. doi: 10.3115/V1DD14-1112 URL httth: //doi. org. 10.
#115/v1/dl4 1162
Fengchun Qiao and Xi Peng. Ensemble pruning for out-of-distribution generalization. In Forty-first International Conference on Machine Learning, 2024.
Ziyue Qiao, Junren Xiao, Qingqiang Sun, Meng Xiao, Xiao Luo, and Hui Xiong. Towards contin- uous reuse of graph models via holistic memory diversification. In The Thirleenth International Conference on Learning Representations, ICLR 2025. Singapore, April 24-28, 2025. OpenRe- view.net, 2025. htLh:tts::// /ttp:::://///peeeeeeee net
Adityanarayanan Radhakrishnan, Mikhail Belkin, and Dmitriy Drusvyatskiy. Linear recursive fea- ture machines provably recover low-rank matrices. Proceedings of the National Academy of Sciences, 122(13):e2411325122 2025.
Hatem Mousselly Sergieh, Teresa Botschen, Iryna Gurevych, and Stefan Roth. A multimodal translation-based approach for knowledge graph representation learning. In Malvina Nissim, Jonathan Berant, and Alessandro Lenci (eds.), Proceedings of the Seventh Joint Conference Lexical and Computational Semantics, *SEM@ NAACL-IILT 2018, New Orleans, Louisiana, USA. June 5-6, 2018, pp. 225-234 Association for Computational Linguisties, 2018a. doi: 10.18653/V1/S188 2027. URL /ttp::///bbl..rrrr/0...////////1
Hatem Mousselly Sergich, Teresa Botschen, Iryna Gurevych, and Stefan Roth. A multimodal translation-based approach for knowledge graph representation learning. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, "SEM@ NAACL-HLL 2018, New Orleans. Louisiana, USA. June 5-6, 2018, pp. 225-234. Association for Computational Lin- guistics, 2018b. doi: 10.18653/v1/s18. 2027. URL https: /tEE::///ol..org/10.1B633
s 8-207
14

---

Published as a conference paper at ICLR 2026
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. URL
1409. 1556
Miyu Sugimoto, Ryo Okano, and Masaaki Imaizumi. Augmented projection wasserstein distances: Multi-dimensional projection with neural surface. Journal of Statistical Planning and Inference. 233:106188 18110066111,20 2024.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In 7th International Conference on Learning Repre- sentations, ICLR 2019, New Orleans, LA, USA. May 6-9, 2019. OpenReview.net, 2019. URL
https:://peeneeviee
Hd=
John Thickstun. Kantorovich- duality. Online manuscript available at https://courses. es. washington.
duality. pdf. 2019.
Naftali Tishby et al. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Théo Trouillon, Johannes Welbl, Sehastian Riedel, Éric Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. In Maria-Floring Balcan and Kilian Q. Weinberger (eds.). Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY. USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceed- ings, pp. 2071 2071-2080. JoLR.org, 2016. URL http: http:://proceeding mi.r.press/AB. trouillonie html
Tim Van Erven et al. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(7):3797-388 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- 5998 6008, 2017. URL httpssts/prrcceei.aaaggoo/// neurrnnn.ss..eiiine neurips wwtpppppp///////aaeeeap 3f5ee243544 dee www.eeeeeed000e1eeeeeeee
Abstract html
Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
Hu Wang, Congbo Ma, Jianpeng Zhang, Yuan Zhang, Jodie Avery, Louise Hull, and Gustavo Carneiro. Learnable cross-modal knowledge distillation for multi-modal learning with missing modality. In International Conference Medical Image Computing and Computer-Assisted In- tervention, pp. 216-226 Springer, 2023.
Jinghua Wang, Zhenhua Wang, Dacheng Tao, Simon See, and Gang Wang. Learning common and specific features for rgb-d semantic segmentation with deconvolutional networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 664-679. Springer, 2016.
Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frédéric Precioso. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia & Expo Workshops. ICME Workshops 2015, Turin, Italy, June 29 July 2015, URL https:
pp. 1-6. IEEE Computer Society, 2015. doi: 10.1 101101/CM...20..770 7/doi. org/1.0rg//0.... 1109/ICMER 2015. 7169757
Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. Deep multimodal fusion by channel exchanging. Advances in neural information processing systems, 33:4835-4848 4845, 2020.
Zikang Wang. Linjing Li, Qiudan Li, and Daniel Zeng. Multimodal data enhanced representation learning for knowledge graphs. In International Joint Conference Neural Networks, IJCNN 2019 Budapest, Hungary, July 14-19, 2019, pp. 1-8, IEEE, 2019. doi: 10. 1091 8852079. U cnttps://dof /tog//000000 1009 JCNN. 2019 8852079
15

---

Published as a conference paper at ICLR 2026
Wei Wei, Chao Huang, Lianghao Xia, and Chuxu Zhang. for recommendation. Ying Ding. Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben (eds.), Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 May 2023, pp. 790-800. ACM, 2023. doi: 10. 00.1145//54350733832266
self-supprvised learning
URL https: http://oooooooooooooooooooooo .rg. 10.
45 5555555455777 3583200
Danny Wood, Tingting Mu, Andrew M Webb, Henry WJ Reeve, Mikel Lujan, and Gavin Brown. A unified theory of diversity in ensemble learning. Journal of Machine Learning Research, 24 (359):1 49, 2023.
John Wright and Yi Ma. High-dimensional data analysis with low-dimensional models: Principles, computation, and applications. Cambridge University Press, 2022.
Yan Xia, Hai Huang, Jieming Zhu, and Zhou Zhao. Achieving cross modal generalization with mul- timodal unified representation. Advances in Neural Information Processing Systems, 36:63529- 63541, 2023.
Xiongye Xiao, Gengshuo Liu, Gaurav Gupta, Defu Cao, Shixuan Li, Yaxing Li, Tianqing Fang, Mingxi Cheng, and Paul Bogdan. Neuro-inspired information-theeretie hierarchical perception for multimodal learning. arXiv preprint urXiv:2404.09400 2024.
Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. embodied knowledge repre- sentation learning. In Carles Sierra (ed.), Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017. pp. 3140-3146. ijeai.org. 2017. doi: 10.24966/JCCA12217/448 24963/1jcai 2017/438
htLhttp:: /tt::///ooi.ooo. /tt.
Ruobing Xie, Stefan Heinrich, Zhiyuan Liu, Cornelius Weber, Yuan Yao, Stefan Wermter, and Maosong Sun. Integrating image-based and knowledge-basee representation learning. IEEE Trans. Cogn. Dev. Syst., 12(2); 177- 2020. doi: 10.100//CDSS220122666685
URL
hittss://do org.lorg//aao 1109 TCDO0//CCDD.... 201 2906685)
Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey.
IEEE Transactions on Puttern Analysis and Machine Intelligence, 11(1011:2111--11132,,,, 2023.
Zihui Xue and Radu Marculescu. Dynamic multimodal fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2574-252552 2583, 2023.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:/t:://aatt
org/ abs/ abs/ 141. 6575.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv: 10941, 2017.
Yuan Yuan, Zhaojian Li, and Bin Zhao. A survey of multimodal learning: Methods, applications, and future. ACM Computing Surveys, 57(7):1-34. 2025.
Jie Zhang. Xiaosong Ma, Song Guo, Peng Li, Wenchao Xu, Xueyang Tang, and Zicong Hong- Amend to alignment: decoupled prompt tuning for mitigating spurious correlation in vision- language models. In Forty-first International Conference Machine Learning, 2024.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng. Provable dynamic fusion for low-quality multimodal data. CoRR, abs/2306.02050, 2023a. doi: 10.48550/arxxw.22366ko 2306.02050
wttp////....///..1...... URL https: /obo:://do.... http:////....r//.........
Qingyang Zhang et al. Provable dynamic fusion for low-quality multimodal data. In International conference on machine learning, pp. 41753-41769, PMLR, 2023b.
16

---

Published as a conference paper at ICLR 2026
Xin Zhou, Hongyu Zhou, Yong Liu, Zhiwei Zeng, Chunyan Miao, Pengwei Wang, Yuan You, and Feijun Jiang. Bootstrap latent representations for multi-modal recommendation. In Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben (eds.), Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 4 May 2023, pp. 845 854. ACM, 2023. doi: 10. 2.1145/3543350733582511
URL https: /ooi org/10.
3145/3543507.3588
A APPENDIX
A.1 USE OF LLMS
For this manuscript, large language models are utilized exclusively for linguistic polishing. Beyond this function, large language models make substantive contributions the conception, analysis, or completion of this work.
A.2 THEORETICAL DERIVATION
A.2.1 PROOF THEOREM I
In this subsection, demonstrate that vanilla linear target mapping establish the superiority of over LF. For the binary classification task, the activation function is Sigmoid function:
(14)
[1.
if the prediction logits
and the predicted label ÿ (0. else
We have
OC(f(x).v)
(C(f(x),v) [a[f(x)] [a 0f(z)
0f(x)
al-yhoi - (1- 0a[f(x)]
0a
0f(x)
(15)
1
1
+ (1 V)
-1- [[- [[
o[f(x)] d[f(x) -
alf(x))) off(=)] -
a -
and a[f(=)] E (0, Therefore, the loss function L(:;)) )) is monotonically decreasing function for the samples with the label 1, and an increasing function for samples with the label = 0.
As mentioned in Section B the logits of LF can be formalized
fir(x:) w² (z) 01) 2 (22 02)
(16)
wt 21.s. 01,s, + w 21,N, OI,N + w u2222,s, 02,5, + w0 =2,2 02.N2
which equals to
fir(w)) = w Σ 01, + w
Σ
01, + 2 Σ
22, + 2 Σ
22,102,
(17)
iES,
JEN,
KES,
MEN
The prediction logits of IF can be formalized
fiF(x)
(18)
=DDs,9 ODs,32 + #Ds,N,
ODS,Ng + #DN, ODN,s, + *DN,N, ODNNNN'
17

---

Published as a conference paper at ICLR 2026
Analogously, Eq.
18
can be rewritten
[IF(
Σ
Oz
Σ
2y +
Σ
Zw
E One
,2
N'
NEDNIN
Σ
(z 21,5' + 22,r)) +
Σ
0..(w' 21,5" + u u2.2))
j'EDs
+
Σ
(x 21,8 + w² 22 +
Σ
( 21,K" + w² 22,1)
M
NEDNIN
w
8U22, +
Σ
0g 21.3") + wtt
Og 21,
Σ 0w/11,))
N' N2
WEDN, 92
MEDN, N2
+
w
0222,8 +
22,2 + + w²(
0,2225') +
22,))
S
KEDN1
J'EDs,N2
M M'D N2
(19)
Given the Bayes optimal hypothesis f*, which achieves the infimum of the errors R2 on D, i.e.: f arrgmnn R(f) argmin B(m,))> v)]
(20)
/
Equation ((f(F()),)) y) 2 (xf y) holds universally. Let A, A1, A2, As, and Ag be five scalars that are positive correlated with 51 (A, A1, 42, As, A x y 8)), where > 0 is an arbitrarily small positive constant and A = ΣE= A:
Obviously, we have the conclusion: for Ye 00, (x)))
(1fi(()),)) v) II]. there exists fir(#) + A such that the classification error [((r(()) + A,y) = [(fur(a)))) - 6. Thus core challenge lies in proving the existence of 0 which makes fe.IF( fFF(#) +A.
Considering the following linear equations:
Σ
Σ
0321.3'
Σ 21,01,1 + A
PED9,
J'E N2
iES,
Σ
Ogz +
Σ
OW2IN
E 21 01. + A
K 82
WEDN, No
JEN,
(21)
Σ
2222 +
Σ
O222
Σ 22, + As
VEDs,82
WEDN
kES2
Σ
0,
+
Σ
22
Σ
22,n h +11
N'
MEDNYN
AEN,
we treat the parameters of linear target mappings in as the coefficients to be determined, and we denote the i-th element of the set S by is, then we have:
8188,82 82
index of Ds
Das Sy 10
0109, No
Σ 21, 1,
A
1E8,
index of Ds,N2
Σ 21,01.5 + 42 JEN1
Opsyyyllas los, No
A
(22)
Σ 22, + As kES,
index of DON,
0 0p su so
Σ 22,402,4 AS NeNz
A1nn No
index of DN, No
0 0IDNy My lony Na
18

---

Published as a conference paper at ICLR 2026
where A equals
21.10g,
21, 2i,jDDe
... S2/Dssealosses 22, los.an
0
22,0x, ... 3/. P2
0
0
[2, loss
and augmented matrix can be formalized as
Silngg 1n fijee N Payls
s1,0ut A
1831
Σ 115 Ag
JEM.
$2,9 F2lDan
Σ
21,02x
ACS
#2
P2.DD, Na Day
MEN,
Obviously, the rank of A is equal to the rank of Á. According to the basic knowledge of Linear Algebra Greub (2012), there must exist parameter 0 of linear target mapping in IF such that the following equation holds:
01, 11 iES1
Σ
Σ 01, + A2 JEN
A0
(23)
Σ 22, +33 kES2
Σ 22,102,0 +11 LeN
Consequently, for Ve [o. ((f* ((fr(())))) II) v)] there exists a parameter such that ((f0,I(())))) v) [(fir(w) + A, [(fir(z).v) e, which further derives that (( v) < [(fur(x),)) y). We denote the set of parameters satisfying ((fo.1((z),v) < C(fiF(x)))) y) A. The proof of Theorem I complete.
A.2.2 PROOF OF THEORRE 2
Let (x, y) D denote the multimodal samples, the generalization error is defined as
D o = (C(.)))((((((()...)))))))))))))) u) = Eple p( (()=)()
(24)
1=1
thus the generalization emror of L.F and can be formalized as:
9FF (((.i))((((i((a)))
Σ p( (()
(25)
9FF
Σ ((Tp((()()))))))))))))))))))))))
(26)
Due to C(fr(())) w²) 0. ((fi( 0. and based the parameter O E A, we have therefore the following equation holds:
[(fr(a")) 2 (x")) ), y²),
pla w/ l ((F(r(())))))) Ep(a' w² (()
(27)
which equals
SIF,0 S SEF-
(28)
The proof of Theorem has been completed.
19

---

Published as a conference paper at ICLR 2026
A.2.3 PROOF OF THEOREM 3
In this subsection, based on Assumption | we provide the proof of Theorem 3
Let 2m, and 2mz be the latent features of two arbitrary modalities, which respectively fit the distri- butions Mmn and Jmn.. We have:
gig mms ) [z Ess, Pipting (C(g(Zm)).)) Ba Hinz [C(g( Zmz 8)] (29)
According to the Gantorovich-Rubinster Duality theorem Thickstun (2019): Edwards (201 1), we have:
E Ez m/tmy (Z(g(Z)))) y)l Ezzz ~/m [C(g(Zma))3)
S|| Φ ||rap (M (ama)
(30)
3 K ( (mg),
where (B()) is the emprical error. It's worth noting that if D M is not complete distribution distance metric such as Kullback-Leibe Divergence Kullback (195T), we need to put more discussions on E[g(z - Big Zma)) )1 & KD (P Mmz) because of Kullback-Leibler Divergence's asymmetry i.e. K [(mm (mmg)= * K KTApmag (mm))
In (30). by replacing the feature of j-th modality to the fused multimodal feature we have:
Blg(zm. (m, )] E[g(z)] & KDD M=)-
(31)
is the distribution that multimodal feature follows, and can be the index of arbitrary modality, that is, E {1, 21, M. therefore:
(lmm KD KDM (4) E(fr).
(32)
(2. indicates that the empirical error of a certain unimodal modality can be bound by the em- pirical error of the fused multimodal feature and the distribution distance between the unimodal and the fused feature.
Restating Theorem
Zhang et al. (223a) and combined with Eq. we have:
IIFO 5
B(w TM ( (lgg((m)) + Σ B(w RR (H)
Σ Coolw" L(go(2m):))
(n(1/5)
MV
2.V
M
M Σ
<
E(w m [KD [KD (1z))mmm))))>)))))))))n> + R(fi=)] +
E(w" 1m (HL)
m==1
m=1
lm(1/5)
+
Coo(w" (Zgo(Zm):)))) v)) MM
2N
m=1
Σ - (D. M2) Error(w": ((go(zm) f)ll B(Fx) + Bias (00))
(N-11/1)
(33)
Thus, the proof of Theorem is complete.
A.2.4 DERIVATION IN LINEAR TARGET MAPPING WITH INFORMATIC CONSTRAINT The objective function is:
M
Max T(z; y)
((zmit) y))
(34)
According to Alemi et al. (017)); Tishby et al. (2000); Xiao et al. (024), to avoid the collapsed representation of during the learning process of the linear target mapping parameter, introduce
20

---

Published as a conference paper at ICLR 2026
regularization term [2;; 2) and its trade-off coefficient X, then have:
[zz:y) Σ I(zmi v) = [(z:)) - \1(x;))
(I 2 ((z:8) (1 2) +
[zmi y)
M
M
(Imi Zm)
Σ
(z y)
m==1
m=1
(35)
M
M
= =(z:v) (x( -
Σ
1 (zmig) A
(x; 2m))
m=1
m=1
[(z:)) (I
[[zmig) - >x(x;; zmm)
In the following, we begin examining each term in Eq. 35) from term [(z;y).
ply. z) [(2:)) f ply, x) log p(y)p(z)
dydz / p(y. log p(u|z) dydz.
(36)
ply)
Let (y)2z) be variational approximation of p(y]z), and parameterize ((y|z) by 0. i.e., ( z). Based on the fact that the Kullback Leibler (KL) divergence is constantly positive, we have K LL[p(vl=)) [v]l2]] 2 0 p(ulz) log p(y) 2)dy f p(ulz) log go(yl= /2yd, thus
/ ply, 2) log
( z))
/ /
dydz = p(y, z) log (v) z)yzz -
ply, 2) log ply)dddd
p(y)
= -f p(u, z) log (o(oou)))dds + (Y)
/ / ((z))((/=)p((z)))) log go (0) ( dzdydz + HY)),
(37)
where (Y) is constant term and can be ignored. As for the term T(2;2), have
T(2; / p(ax, log log p(z(ar)
dzdæ.
(38)
p(z)
Let ret) be variational approximation of p( 2) and set (2)) as the standard Gaussian distribution ((O, Since KLIp(z), 2 = p(( log pl d2 2 J p(=) log (z))d2, thus:
[(z;x) - / p( p(z)p(z(()) log p(z|2) dxdz.
(39)
r(z)
As a result, we have
I(2: x=);
> [ p(a))((ll=)p(x))))ool) logg(yl (x) > / / p(z)p(z(x) log
p(z(2)
dxdydz
r(z)
= LB,
(40)
LB standards for Lower Bound. Analogously, for the upper bound (UB) of Σ [[(zmi
XI(x; zm)], we have
Σ [[(zmig) Al(x; SBB
M
Σ
pl p(z)p(u)2)) xx) log go zy ddddddz
m=1
+ 2 1
p(z)p(zml)) log p(zmlz) dodzm
r(zm)
(41)
21

---

Published as a conference paper at ICLR 2026
142
J412
J423
f41
p41
11
P113
aa)
(b)
Figure 6: The illustration of the space of probability density functions, in which each point repre-
sents probability distribution.
Then maximizing the LB UB equals to minimizing the following loss function:
M
Lic
log (o(()))) + X- KL I))
Dog (vl zm) + A KL (Ns,,,,
(42)
A.2.5 DERIVATION OF THE UPPER BOUND ON DISTRIBUTION INCOHERENCE
We provide an illustration of the space of probability density functions in Figure o to assist our theoretical derivation. Then we demonstrate that the Equation 9 holds for M (2.3). Let LHS EM E(w" )D u)) and RHS = Emngmm DM (Pms Hmm2)
For M = 2, we have
Σ m=1 DM R)) Dm((I, +DM(H2,H) RHS.
LHS
(43)
For M = 3, we have LHS < Em= D ( 14) S DM (1) + (M2 + ( 11) and RHS DM(H1, 122) + (MM, Hz) (Mu2 p3). As illustrated in Figure 6 have
H22 + DMM22, > DM (M1,H22)) H2s) = DM((1, 1) + DM P23)
(44)
and
p2() + DM us) DM),
(45)
Then the following equation holds:
DM(uI, + (M2, 123) + DM(,, (22) + D (M23,
(46)
Dm((1,)) p)) (M 123) + D DD,
which equals to
DM(H1,H2) + DM
> DM DMM(es)))
+ DD((,
(47)
Similarly, we have
DM((i, + DDM(i, DM 1) (M)(H,p)) DM m1) + (M3, (2) DM((I, + (M,
(48)
Then we have
RHS 2 (M. + DM((, + (M(HHH)) LHS.
(49)
As result, Equation holds for M = (2.3). which implies that Equation is applicable for almost all multimodal scenarios according to the recent multimodal learning survey Xu al. (2023); Yuan et al. (2025) (even the powerful model [achhmann et al. (2024) capable of handling 21 modalities can handle most 3 modalities at single time).
22

---

Published as a conference paper at ICLR 2026
Algorithm The training pseudo code of IID.
Input: The sampled minibatch samples {(x', y0ll E [I,-.. N] with batchsize N and æ' ( :.. r'r) The latent mappings (-), target mapping g(-) and multimodal fusion weights w" The hyperparameters a and B.
Output: The loss function of IID, i..., LIID.
for /= to N do
Obtain unimodal features by z (x'm)) (m E MM):
Get now-dimensional feature Zm of z
Calculate the fused multimodal feature 2 =>>" Em= w" m
Calculate the loss function ((fir(""),1) y");
end
Calculate the Lie by E8.
for m1, my E [1, M] and my # m2 do
Get the estimated Wasserstein distance between the features of my-th modality and my-th modality by (((mm:Mmm) end
Calculate the loss function Lde = E Emi W(mms Pma)s
Return LIID = otLic + BLdc + Ei= ((Tr((""))))))
A.3 WASSERSTEIN DISTANCE
Wasserstein distance has its roots in Optimal Transport theory Villani et al. (009). which is complete distance metric of distribution. Let μ be of Borel probability measures. Given Mat Mos E μ. the corresponding support sets or, .y. Wasserstein distance between fer and pas
/
inf W(uar: :(HarMMes) (ETT(,
F.g Pdd(err
(50)
where x, E Cr., E Gg dis(:;) -) is distance metric, and p = in this paper. T(r Fgg) is the set of all joint distributions (2 Fgy) that satisfies plyr = Sxg (F,, F.g) )dr, and ul2 S2, 7 (B Fi drr..
A.4 ALGORITIM
In this subsection, we elaborate on the pseudo-code of proposed IID in Algorithm
A.5 EXPERIMENTAL SETUP
A.5.1 DATASETS
Vision-language classification. We execute experiments on four vision- laiiun-lagguage classification datasets, including Food101 (Wang et al. 2015)) MVSA-Single (Niu et al. 2016), MVSA-Multiple (Niu et al., 2016) and HFM (Cai et al. 2019). Food101 comprises images sourced from Google Image Search along with corresponding textual descriptions. MVSA-Single, MVSA-Multiple, and HFM are all derived from Twitter. For Food101, there are 60101 image-text pairs in the training set, 5000 image-text pairs in the validation set, and 21695 image-text pairs the test set. For MVSA- Single, there are 1555 image-text pairs in the training set. The validation set contains 518 image-text pairs, and the test consists of 519 image-text pairs. For MVSA- -ulltipee,,,, there are 17024 image- text pairs, each annotated by three different annotators. The training set contains 13624 image-text pairs, while both the validation set and the test set contain 1700 image-text pairs. For HFM, the training set comprises 19816 image-text pairs, while the validation set contains 2410 image-text pairs, and the test set consists 2409 image-text pairs.
Link prediction. In terms of the link prediction task, we conduct the experiments and evaluate with two standard competition benchmarks, i.c., WN9-IMG Xie et al. (001)) and FB-IMG Sergieh et al. (2018)). M9-IMG dataset derived from the subset of WN18 Bordes et al. (2013), which embraces structural knowledge as triples, and multimodal knowledge including textual description and visual images. The FB-IMG dataset is derived from the subset of FB15K Mousselly-Sereee et al. (2088)
23

---

Published as a conference paper at ICLR 2026
which includes structural knowledge consisting of triples extracted from Freebase Bollacker et al. 2008), and multimodal knowledge embracing textual description and visual images.
Scene recognition. In accordance with the standard split of the NYU Depth V2 dataset, consol- idate the original 27 categories into 10 categories, encompassing 9 typical scene categories and one "other" category. For the SUN RGB-D dataset, adhere to the categorization scheme employed
by the major baseline methods (QMP (Zhang et al. (023a) and TMC (Han et al.) (02I)) utilizing the 19 primary scene categories, each containing minimum of 80 images.
A.5.2 BASELINES
Baselines of vision-language classification. To comprehensively evaluate the performance of the proposed IID, both unimodal models and multimodal models are selected as baselines. Con- cretely, unimodal models include Bow (Pennington et al. 2014a). Img (Image only, we use ResNet- 152 (He et al.] 2016) to encode the visual data) and BERT (Devlin al.) 2019a). Multimodal
baselines contain Late-fusion, ConcatBow (C-Bow), ConcatBERT (C-BERT), MMBT (Kiela et al. 2019) TMC (Han al. 2021), DYMM Xue & Marculeseul (2023), LCKD Wang al. (2023). QMP
[Zhang et al. 2023a). U niCODE Xia et al. (2023). SimMMDG Dong et al. (2023), and PDF Cao
al. (2024).
For Late-fusion and ConcatBERT fusion, we utilize the architecture of ResNet (He
it al. 2016 pretrained on ImageNet (Deng et al. 2009) as the backbone network for the visual modality and pretrained BERT (Devlin et al. 2019a) for the text modality. For ConcatBow, we replace BERT with Bow. The Late-fusion conducts an average weighted summarization between visual and textual features, and concat-based fusion concatenates the visual and textual features di- rectly. MMBT leverages the attention mechanism to execute multimodal fusion. TMC proposes novel trusted multimodal algorithm based on the Dempster- Shafer evidence theory. DYMM employs gating function to provide modality-lever or fusion-leve decisions on the fly based on multimodal features QMF designs a robust multimodal fusion method, which is connected to uncertainty learn- ing. PDF derives the multimodal model based the intra-modal negative and inter-modal positive covariance between the fusion weight and loss function, respectively.
Baselines of link prediction. For comprehensive comparison, select both unimodal methods and multi-modal methods as our benchmark baselines, including TransE [Bordes et al. (2013). MistMult
Yang al. (2013), ComplEx TTroiilon et al. (2016), RotatE [Sun al. (2019). IKRL Xie et al.
2020)) Sergich et al. (2018b), TransAE Wang et al. (2019), MMKRL Lu et al. (0022), and
ОTKGE Cao et al. (2022).
Baselines of scene recognition. For the senses recognition task, we evaluate the proposed methods against various multimodal fusion techniques, including Late-fusion, concatenation-base align-based fusion (Wang et al. (016), and the recent state-of-the-ard fusion methods, i.e., MMTM (Vaswani al., 2017)), TMC (Han al.] 2021). and QMF (Zhang et al., [023a). For Luti-fusion and concatenation-base fusion, we employ the ResNet architecture (He et al. (0166), pre-trained on ImageNet, as the backbone network for each modality. Align-based fusion intensifies the similarity of various unimodal features to achieve multimodal alignment.
fusion,
Implementation details. (I) Vision-language classification. In the proposed IID, we employ BERT and ResNet as the latent mappings for text and image modalities, respectively. In the training pro- cess, we use BertAdam for the BERT model and regular Adam for the other models. The learning rate is 55 with a warmup rate of 0.1. We adopt the early stop strategy based on validation accuracy. We elaborate on the selection of the hyperparameters a and 8 in Section A.6 (2) The structured em- beddings are produced from triples in knowledge graphs, without any external multi-modal sources. To be specific, unimodal KGE methods such as TransE [Bordes et al. (2013) and ComplEEx Trouil= lon al.] (2016) can be used learn structured embeddings. The linguistic embeddings of entities are learned by adopting the word2vec Mikolov et al. (2013) technique. For instance. we learn the linguistic embeddings of FB-IMG dataset by pre-trained word2vec while we use Glo (eneennnnnnnoonn et al.) (2014b) for the WN9-IMG dataset. The visual embeddings of entities are learned by pre- trained VGG Simonyan & Zisserman (2015) models. To be specific, visual embeddings are learned by adopting the
128CNN Chatfield et al. (2014) model in FB-IMG datasets. As for the WN9-IMM dataset, we take the VGG19 Simonyan & Zisserman (2015) model to learn visual em- beddings. (3) Scene recognition. The dimensionalities of unimodal and common representations set to 128 and 256, respectively. For abign-based fusion, we utilize cosine distance to measure the similarity of representations. For the MMTM approach, adhere to the authors' implementation,
24

---

Published as a conference paper at ICLR 2026
MVSA-Single
MVSA-Multiple
HFM
6.9
2001
2081
000
Sell
Food101
Depth V2
RGB-D
1020
100
Figure 7: The results of hyperparameters experiments.
NYU Depth V2
QMF
Acc: 113%
Acc: 48.489%
I1D
Ace: 72.171%
SUN RGB-D
QMF
Acc: 61.794%
61.988%
63.039%
Figure 8: Visualization results of the scene recognition task (the NYU Depth V2 and SUN RGB-D datasets).
setting the squeeze ratio to 4. Across all compared methods, we use the Adam optimizer with L regularization and dropout, employing learning rate x 10-4 and dropout rate of 0.
A.6 DEEP-GOING EXPERIMENTAL RESULTS
A.6.1 THE RESEARCH ON THE HYPERPARAMETERS
exist in IID, i.e., a and 3. To understand the impacts of these two hyper- parameters, conduct empirical comparisons by using various combinations of and 3 for the proposed IID. As depicted in Equation [3] a controls the impact of informatic constraint on the linear target mapping, and 3 influences the degree of distribution incoherence.
Two
25

---

Published as a conference paper at ICLR 2026
In practice, search the optimal 3 in {1e 10 1,, 1e-ee- As for a, on the NYU Depth V2 and SUN RGB-D datasets, we search a in 0.10.1,0000 0.01, 0.001), while is searched in {0.1, 0.01, 0.001) on other four vision-language classification datasets. We determine the values of a and B empirically and depict the results Figure 7 where the X axis, classification Y axis, and 7 axis represent the value of o, the value of [3, and the recognition accuracy, respectively. As we can observe, the optimal combination of a and / varies with respect different datasets, which is indicated by red pentagonal markers. For example, the optimal com- binations of a and 3 on the MVSA--Sngle, MVSA-Multiple and NYU Depth V2 are f0.01, le 1e- {0.1, 19,1e) 10), and {1, 1e (le-12), 12), respectively. Therefore, the elaborate assignment of and 3 can further help to learn informative features, thereby improving the discriminative performance of the proposed method.
1e_1+) across all six datasets.
A.6.2 VISUAL COMPARISON
To intuitively demonstrate that IID is capable of learning informative and discriminative represen- tations, we present a visualization of the learned embeddings corresponding to the samples. Specif- ically, we utilize the T-SNE technique (Nkedi-Kizza et al. 2006) visualize the feature repre- sentations of test set samples across multiple datasets visualization results of the scene recognition task (the NYU Depth V2 and SUN RGB-D datasets) are illustrated in Figure 8 We denote the distinct ground truth labels of test set samples by differ- ent colors. As we can observe from Figure 8 compared with other multimodal approaches (QMF and MMTM the NYU Depth V2 and SÜN RGB-D D datasets), the boundaries of IID between different classes are more distinct, indicating that the III) can better discriminate features across dif- ferent classes. Additionally, for the proposed IID, data points within the same class tend cluster more tightly, suggesting that the features extracted by IID have higher intra-class similarity. These observations demonstrate that the IID-learned representations facilitate the extraction of more dis- criminative features, thereby enhancing performance across various downstream tasks.
V2, and SUN RGB-D). The
A.6.3 TABLE OF NOTATIONS
We list the definitions of main notations from the main text in Table 5.
26

---

Published as a conference paper at ICLR 2026
Table 5: Main notations used this paper.
Notation
Definition
Data and Representation
X, Y.y
Input space, latent space, and target space Training dataset and test dataset Input sample Label
Drain Diest
2 E A
y E y
mm
Representation of modality m
The fused multimodal feature Number of modalities
M
N
The batch size
d
The dimension of features
Model Components
h(.): X Z
Latent mapping
g g0) : 2 22 y
Target mapping
f-gh()
The composite function of g(-) and ht-) Intermediate fusion model
fIr fLF
Late Fusion model
The modality-specific fusion weight
((.-))
Cross Coss-EEtropyy loss function
Theory-related Symbols
0
The parameter of target mapping s(-)
A
The set parameters The index sets of semantic dimensions and noisy dimensions Bayes optimal hypothesis The generalization error The definitional domain of g
Sms Nn f
D,
DM
The complete distribution distance metric Hypothesis space
H
pm
The distribution that features of the m-th modality are drawn from The distribution that the multimodal feature follows
f
B(fr)
The empirical error of multimodal feature x on Duain
IID Method
0"
The theoretically optimal parameter
The initial parameter of the linear target mapping Mutual information computing Loss of linear target mapping with informatic constraint The variational approximation of p(-l-)
I(..))
Lie
go (-1-)-)
X
The trade-off hyper- parameter
KL()
Kullback-Leibler divergence
N
The standard Gaussian distribution
W(.))
The analytical form of Wasserstein distance
T
The transport plan The cost matrix
C
F()
Fast Fourier transform The dimension index Gaussian Random matrix
n
1,
$
F
The RIP-preserved dimensionality reduction matrix
.).))
The estimation of Wasserstein distance
Ld
Loss of distribution cohering with restricted isometric dimensionality reduction
a,
The hyperparameters to control the influence loss terms
27


Reviews:
[{"review_id": "KQpA5n956I", "reviewer": "Reviewer_Daym", "summary": "This paper explores why Intermediate Fusion (IF) outperforms Late Fusion (LF) in multimodal learning. The paper analyzes: (i) an existence result showing that IF performs no worse than LF under linear target mappings; and (ii) the generalization bound of IF under the K-Lipschitz classifier, where the distribution inconsistency between the feature distributions of each modality and the fusion distribution determines performance. Based on this, the authors propose the IID algorithm, which combines information constraints and distribution consistency. In six benchmark tests (visual language, RGB-D), IID outperforms the robust LF baseline and its IF variant; ablation experiments show that both modules contribute to the performance.", "questions": "Same as mentioned in weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors prove that combining features from different modalities before classification (IF) is no worse than classifying them individually before weighting (LF). They then provide an upper bound on the generalization error, highlighting that the key factor affecting IF performance is the inconsistency between the feature distributions of different modalities and the combined distribution.\n2. The authors use FFT sparsification, RIP projection, and unbalanced OT (Sinkhorn with KL relaxation) to approximate Wasserstein distances, the method is computationally practical and achieves consistent improvements across six benchmarks; ablations indicate both modules contribute.", "weaknesses": "1. The existence result IF better than LF assumes the target mapping 𝑔 is linear. In practice, post-fusion heads are often nonlinear. Can the claim be extended to nonlinear but K-Lipschitz heads? Please also include an experiment comparing a linear head vs. a two-layer MLP head to delineate the applicable regime.\n2. The method minimizes unconditional (marginal) discrepancies between modalities, whereas decisions depend on class-conditional distributions. Under what conditions does shrinking marginal discrepancy guarantee per-class alignment and improved decision boundaries?\n3. The bounds in the method involve the distance from each modality to the fusion distribution , but the objective function minimizes the distance between pairs of modes. Please specify under what assumptions minimizing the pairwise objective function is equivalent to the objective term from each mode to the fusion distribution.\n4. The objective maximizes $I(z;y)-\\sum_{m=1}^{M} I(z_m;y)$ and minimizes pairwise modality discrepancies, yet neither term explicitly penalizes cross-modal redundancy; indeed, alignment may amplify label-irrelevant shared patterns.\n- **(i)** How does IID prevent reinforcing such redundant signals?\n- **(ii)** It's better to refer to this paper for handling redundant information in related works.\n- [1] X. Xiao, “Neuro-inspired information-theoretic hierarchical perception for multimodal learning,” in Proc. 12th Int. Conf. Learn. Represent., 2024, pp. 1–29.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bdJ2FZBESq", "reviewer": "Reviewer_Xdxj", "summary": "This paper presents a novel framework for multimodal representation learning, with a theoretical analysis centered on intermediate and late fusion mechanisms. The proposed theory further leads to the implementation of the IID method.", "questions": "1.In Table 2, the Concat method consistently outperforms IID-L. Does this indicate that modules I and D are not meaningful or contribute little to the model’s performance?\n\n2.Taking PDF as an example, within the IID-P framework, larger weights are assigned to the dominant modality while smaller weights are given to the weaker ones. Consequently, the dominant modality converges faster whereas the weaker modality remains under-optimized. In this context, is it reasonable for module D to reduce the distribution discrepancy between features of well-converged and under-optimized modalities?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.Starting from a theoretical perspective, this paper systematically compares Intermediate Fusion (IF) and Late Fusion (LF) from a dimensional viewpoint, and provides rigorous proofs for both prediction error and generalization error.\n\n2.The paper is well-structured and written in a professional academic style.", "weaknesses": "1.The errors in Figures 1 and 5 need to be corrected.\n\n2.In the case of Intermediate Fusion (IF), it is generally assumed that Concat and Sum are equivalent, which often leads to the issue of modality laziness. However, the authors did not take this into account.\n\n3.The notations are confusing. For instance, in the figures, w is shown as a vector, whereas in the main text it appears to be used as a scalar. It is recommended to include a notation table for clarification.\n\n4.Although module D is theoretically derived, it appears to be meaningless from an empirical perspective, as the hyperparameter experiments (β ∈ {1e−10, …, 1e−14}) suggest negligible effects. The rationale for including module D should be further explained.\n\n5.The proposed method is based on Intermediate Fusion (IF), yet the comparison with other methods is insufficient and somewhat unfair. Many advanced approaches within the IF framework are not considered.\n\n6.The study lacks experiments involving additional modalities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "79tTBEN9x8", "reviewer": "Reviewer_coeM", "summary": "This paper re-examines two mainstream fusion approaches in multimodal learning—Intermediate Fusion (IF) and Late Fusion (LF)—from a fine-grained perspective. It demonstrates that IF outperforms LF under specific constraints and further derives the generalization error upper bound of the IF method. Based on theoretical analysis, it proposes an IF method that incorporates information constraints and distribution consistency. Experimental results on a large number of datasets verify the effectiveness of this method.", "questions": "See the Weaknesses section for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1）The motivation is clear and the research significance is prominent. The paper identifies the issue that current theoretical research on multimodality mostly focuses on LF, while there is insufficient theoretical analysis on IF, and possesses a clear motivation to fill this gap.\n\n2）The theoretical analysis is in-depth. It provides a comparison between IF and LF from the perspective of fine-grained dimensions, derives the generalization error upper bound, and makes significant theoretical contributions.\n\n3）The ablation experiments are comprehensive and validate the necessity of the modules. Ablation experiments are conducted on the two core modules (information constraints and distribution consistency) to verify their respective effectiveness.", "weaknesses": "1）The experimental comparisons have gaps, which weakens the persuasiveness of the method's innovation. Current experiments mainly compare IID with LF-based SOTA methods (e.g., PDF, QMF) or general fusion models. However, direct comparisons with advanced methods specifically designed for the IF framework in recent years are lacking. It is suggested that the authors supplement relevant experiments, as such comparisons can more clearly highlight the unique advantages and contributions of IID under the specific paradigm of \"Intermediate Fusion\".\n\n2）The analysis of computational complexity is insufficient. Although the paper identifies the computational bottleneck of high-dimensional Wasserstein distance and proposes an efficient dimensionality reduction method, it lacks quantitative analysis of the computational overhead of the overall IID model (including the two newly proposed modules) during training and inference. This makes it difficult for readers to comprehensively evaluate the efficiency of the method in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ggcqo8xYpA", "reviewer": "Reviewer_5Xqu", "summary": "This paper provides a theoretical analysis of multimodal representation learning (MML) with a focus on intermediate fusion (IF) and late fusion (LF) frameworks. Unlike prior studies centered on LF, the authors theoretically justify the superiority of IF under certain constraints and derive a generalization error bound based on a K-Lipschitz continuity assumption. They further propose an IF-based MML method that introduces an informatic constraint and enforces distribution coherence, achieving strong empirical results on multiple benchmark datasets. However, some parts of the theoretical derivation appear to rely on assumptions that may require further clarification or justification.", "questions": "1. The theoretical analysis assumes that each latent feature dimension can be strictly partitioned into task-dependent semantics and task-independent noise. Is there any evidence in the existing experimental results that indirectly supports this assumption?\n\n2. In the proof, it is unclear how the first equality in Equation 31 connects the empirical error to the expected error. Could the authors clarify the assumptions or steps that justify this equality? Specifically, what conditions are required for this transition, and are they satisfied in the current setting?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel approach to multimodal representation learning (MML) by integrating theoretical analysis with practical model design. It provides a fine-grained theoretical investigation of intermediate fusion (IF) and late fusion (LF) frameworks, deriving a generalization error bound under a K-Lipschitz continuity assumption. Based on these theoretical insights, the authors propose an IF-based MML method that introduces an informatic constraint and performs distribution cohering to reduce distribution incoherence. This theory-driven design is validated through extensive experiments on multiple benchmark datasets, showing improved performance consistent with the theoretical predictions. The approach is innovative in that it connects theoretical guarantees with concrete architectural and training strategies, offering both conceptual and empirical contributions to the field.", "weaknesses": "While the paper presents a novel theory-driven approach to multimodal representation learning, there are some concerns regarding the theoretical assumptions. In particular, the analysis assumes that each dimension of the latent features can be strictly partitioned into task-dependent semantics and task-independent noise. This assumption provides a clear analytical framework but may be overly strong or unrealistic in practical deep learning settings, where latent features are often entangled and do not exhibit such a clean separation. As a result, some of the theoretical guarantees derived under this assumption might not fully hold in practice. Additionally, the theoretical derivations build upon prior results rather than introducing entirely new theorems, so the novelty in the theoretical contributions is somewhat incremental. Nevertheless, the empirical results indicate that the proposed IF-based method, with its informatic constraint and distribution cohering, effectively improves performance and is consistent with the theoretical intuitions, partially mitigating the concerns about the assumptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)