Efficient Quantification of Multimodal Interaction at Sample Level
Authors: Zequn Yang
Hongfa Wang
Di Hu
Venue: ICML 2025 poster (2025)
Keywords: Multimodal interactions
Information measurement
Information decomposition
Primary area: general_machine_learning->everything_else
TL;DR: We efficiently quantify sample-level multimodal interaction and leverage these interactions for understanding and improving multimodal learning.

Abstract: Interactions between modalities—redundancy, uniqueness, and synergy—collectively determine the composition of multimodal information. Understanding these interactions is crucial for analyzing information dynamics in multimodal systems, yet their accurate sample-level quantification presents significant theoretical and computational challenges. To address this, we introduce the Lightweight Sample-wise Multimodal Interaction (LSMI) estimator, rigorously grounded in pointwise information theory. We first develop a redundancy estimation framework, employing an appropriate pointwise information measure to quantify this most decomposable and measurable interaction.
Building upon this, we propose a general interaction estimation method that employs efficient entropy estimation, specifically tailored for sample-wise estimation in continuous distributions. Extensive experiments on synthetic and real-world datasets validate LSMI's precision and efficiency. Crucially, our sample-wise approach reveals fine-grained sample- and category-level dynamics within multimodal data, enabling practical applications such as redundancy-informed sample partitioning, targeted knowledge distillation, and interaction-aware model ensembling. The code is available at https://github.com/GeWu-Lab/LSMI_Estimator.

Efficient Quantification of Multimodal Interaction at Sample Level
Zequn Yang'
Hongfa Wang" 23 Hu
Abstract
richness largely derives from inter-modal interactions, the harnessing of which is crucial for advancing multimodal models. Traditionally, research has focused on capturing redundant interactions by aligning modalities and extracting consistent information among them (Rahate et al., 2022). Such methods are particularly effective when information is entirely shared across modalities, as in the case of images paired with text captions (Radford et al., 2021). However, modality inconsistency can also contain valuable informa- tion. For instance, in video understanding, certain phe- nomena (e.g., wind blowing) may provide limited or even misleading information in the visual modality but can be clearly discerned through the auditory modality. This high- lights the importance of extracting modality unique information. Moreover, when individual modalities alone are insufficient to convey information, their combination can induce additional insights. For example, sarcasm can be detected through positive expression paired with neg- ative tone. In such cases, multimodal information emerges from the collaboration of modalities, reflecting synergistic interactions. summary, multimodal interactions en- compassing redundancy, uniqueness, and synergy - play critical roles in producing multimodal information and offer credible perspectives for advancing multimodal learning.
Interactions between modalities-redundaan uniqueness, and synergy- cynlegy-collectivvlly determine the composition of multimodal information. Un- derstanding these interactions is crucial for ana- lyzing information dynamics in multimodal sys- tems, yet their accurate sample-level quantifica- tion presents significant theoretical and computa- tional challenges. To address this, we introduce the Lightweight Sample-wise Multimodal Inter- action (LSMI) estimator, rigorously grounded in pointwise information theory. We first develop redundancy estimation framework, employing appropriate pointwise information measure to quantify this most decomposable and measurable interaction. Building upon this, we propose a gen- cral interaction estimation method that employs efficient entropy estimation, specifically tailored for sample-wise estimation in continuous distri- butions. Extensive experiments on synthetic and real-world datasets validate LSMI's precision and efficiency. Crucially, our sample-wise approach reveals fine-grained sample- and category-level dynamics within multimodal data, enabling prac- tical applications such as redundancy-infe -anudddme-iffrmeed informed sample partitioning, targeted knowledge distilla- tion, and interaction-aware model ensembling. The code is available at https: //github.. com/GeWu-Late LSMI Batimator
To better characterize the dynamic interplay of these multi- modal interactions, defining and quantifying interactions in a principled manner is significant importance. Initially, Partial Information Decomposition (PID) (Williams & Beer, 2010) proposed field dedicated to investigating the nature of interactions from information-theoretic perspec- tive. Research on PID has primarily focused providing formal definitions of interactions (Lizier et al., 2018; Mages & Rohner, 2023) within discrete distributions. Additionally, several studies have extended the concept of interaction de- composition to continuous Gaussian (Venkatesh et al., 2024) and -iw-dimenioonal (Pakman et al., 2021) distributions. Building on these foundations, Liang et al. (Liang al., 2023b) propose disttribtionnoopiiiizaaioooo for interaction estimation. Their approach utilizes neural estimation to achieve interaction quantification at the distri- bution level, which has been successfully applied for dataset evaluation and model selection (Liang et al., 2023a). These advancements underscore the promising potential of inter- action quantification in advancing multimodal research.
1. Introduction
Multimodal data offer richer information sources, signifi- cantly enhancing information acquisition capabilities. This
'Gaoling School of Artificial Intelligence, Renmin Univer- sity of China, Beijing, China TTencett Data Platform Tsinnhuu University, Beijing, China 'Beijing Key Laboratory of Research on Large Models and Intelligent Governance. Beijing, China "Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE, China. Correspondence to: Di Hu <dihu @ ruc.edee. edu.cn>.
strategy
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Efficient Quantification of Multimodal Interaction at Sample Level
Sample
Sample-level Multimodal Interaction
sample-wise interaction estimation process based on this definition. By leveraging lightweight entropy estimation models (Pichler et al., 2022), we enable the efficient calcula- tion of sample-wise redundancy interactions, as well as the identification of uniqueness and synergy interactions. Ex- tensive experiments confirm the high efficiency, precision, and real-world applicability of our sample-level estimator. Our method also uncovers interaction dynamics at both sample and class levels. These insights not only enhance interpretbillity but also enable practical applications such as redundant data partitioning, interaction-guice -uidded knowledge distillation, and targeted model ensemble that effectively combine sub-models while preserving their specificity.
X1
r
1
U2
s
X2
Target
y
x2)
Multimodal information
In summary, our contributions are as follows:
Figure I: A brief illustration of multimodal interactions at the sample level, including redundancy (r), uniqueness (u,, u2). and synergy (s), which collectively constitute mul- timodal information (2;1, y)-
To the best of our knowledge, we are the first to explic- itly quantifying multimodal interaction the sample level for real-world data.
Beyond such quantifications, analyzing multimodal interac- tions at the individual sample level is both important and challenging. Given that interaction patterns can vary sub- stantially across samples (Luppi et al., 2024), sample-level analysis provides a more granular understanding of these interaction patterns and enhanced interpretability for real- world applications (Lizier al., 2013). However, exist- ing distribution-Ievel approaches (Bertschinger al., 2014; Liang et al., 2023b) primarily quantify interaction informa- tion for an entire dataset, lacking the capability for such fine-grained, sample-level analysis. Although pointwise par- tial information decomposition approaches (Finn & Lizier, 2018a) have been proposed for sample-level interaction esti- mation, they struggle to provide efficient and practical solu- tions for continuous distributions. This limitation highlights the need for novel methods capable of realizing effective sample-level interaction quantification.
2.
We propose a sample-level interaction estimation ap- proach that ensures both precision and efficiency.
Our method reveals the underlying interaction dynam- ies within the data, facilitating deeper understanding and providing guidance for multimodal learning.
2. Related Work
2.1. Multimodal Interaction Learning
Multimodal learning paradigms aim to extract various types of information embedded across different modalities. One key strategy to learn consistency, thereby exploiting redun- dant (or shared) information. Consistency learning (Zhan et al., 2018; 8; Rahate et al., 2022; Yang et al., 2021) and con- trastive learning (Radford et al., 2021; Yuan et al., 2021) are common methods to capture this redundancy. Furthermore, gradient-based modulation techniques (Peng et al., 2022; Yang et al., 2025) can mitigate learning disparities across modalities (Yang et al., 2024), bolstering consistency indi- rectly and enhancing multimodal performance. Conversely, inconsistencies can highlight unique or synergistic infor- mation. Unique information arises when heterogeneous modalities inherently contain different levels or types of in- formation (Gat et al., 2020; Zhang et al., 2023b); approaches here often enhance individual modality representations to isolate modality-specific contributions (Tsai et al., 2018; Wu & Goodman, 2018). Synergistic information emerges when the combination of modalities generates new insights not available from any single modality, especially when individual modalities are insufficient for the task. Captur- ing synergy typically involves more sophisticated fusion mechanisms and dynamic integration strategies (Fukui et al., 2016; Jayakumar et al., 2020). Understanding these fun- damental interactions redundancy, uniqueness, and syn-
To address this challenge, we propose the Lighteeight Sample-wise Multimodal Interaction (LSMI) estimation ap- proach, which enables efficient and reliable quantification of sample-leve) interactions. As illustrated in Figure 2, our estimator aim distinguish task-relevant information (cir- cles) generated from two modalities, 1 22, into redundant, unique, and synergistic interaction. We define interactions using pointwise information and further equip a lightweight model for efficient and precise measurement. On the one hand, we adopt clear and credible definition of point- wise information decomposition using sodundancy-based set-theoretic intuition. Given that pointwise mutual infor- mation can be negative, adhering to set-theoretic intuition remains challenge. To overcome this, we partition mu- tual information into target-related information components, allowing for the measurement suitable for redundancy in each component while preserving the set-theoretic intuition of redundancy. On the other hand, we employ an efficient
2

---

Efficient Quantification of Multimodal Interaction at Sample Level
ergy-is crucial for guiding the design and selection of mul- timodal learning methods. Therefore, this paper proposes efficient analytical framework to characterize multimodal interactions, aiming to provide in-depth explanations and guidance for the field.
Interaction Estimation Interaction estimation, also re- ferred to as Partial Information Decomposition, offers comprehensive framework for understanding the informa- tion conveyed through multiple modalities. In the context of multimodal information shared by X1 and X₂ with the tar- get Y, Interaction Decomposition classifies this information based on its distribution across the different modalities. distinguishes between the information that is shared across modalities (redundancy), that which is uniquely represented within each modality (uniqueness), and the information that emerges only when both modalities are present (synergy). As a result, it decomposes the total information into four distinct components: redundancy R between X1 and X2, uniqueness Un in X1, uniqueness U2 in X2, and synergy S, which emerges only when both X1 and X₂ are jointly considered. It satisfies the following equation:
2.2. Interaction Quantification
Data from multiple sources often interact to reveal how information originates across these sources. Estimation of the interactions provides valuable insights into neural science (Wibral et al., 2017; Celotto et al., 2024). Par- tial Information Decomposition (Williams& Beer, 2010) serves foundational framework, describing three types of interaction: redundancy, uniqueness, and synergy. Sub- sequent work has provided different definitions of these interactions for discrete distributions (Knuth, 2019; Mages & Rohner, 2023), but these definitions are difficult to extend to continuous distributions in real-world scenarios. Some studies have extended these methods continuous low- dimensional (Pakman al., 2021) or Gaussian (Venkatesh et al., 2024) distributions. Additionally, Liang al. (Liang et al., 2023b) have measured interactions in more complex real-world datasets by optimizing the distribution to deter- mine uniqueness. However, measuring sample-wist multi- modal interactions remains an open issue (Lizier et al., 2013; Liang et al., 2023b), and addressing this challenge could significantly enhance the fine-grained understanding of mul- timodal data and the learning preferences across modalities. To this end, this work proposes a practical approach for quantifying sample-wise interaction estimation.
I(X;; = R + U () = R U2. 1(x1, = R+ U1 + U/ +.
(1)
A widely recognized approach to interaction decomposition, based on the concept of uniqueness (Bertschinger et al., 2014; Liang et al., 2023b), determines interactions by iden- tifying a base distribution that minimizes unique informa- tion. This method, inspired by decision maaiinn theory, then leverages this base distribution to compute redundancy (R), uniqueness (U), and synergy (S). However, this distribution- level optimization presents two primary challenges. First, optimizing the distribution is computationally expensive. Second, its design is restricted to calculating interactions over the entire distribution, which impeees sample-wise wise estimation and, consequently, limits its interpretability and practical utility for fine- mne-grained analyses.
3. Method
In this section, we introduce lightweight sample-wis wise multimodal interaction estimation approach. We begin by providing detailed explanation of interaction estimation, followed by a principled definition of sample-level inter- actions from the perspective of redundancy. Subsequently, we implement efficient interaction estimation at the sample level for continuous distributions.
3.2. Redundancy- Interaction Framework
Compared with calculating interaction over the whole distri- bution, sample-level interaction estimation enables details information analysis towards each samples, which provide in-depth knowledge for multimodal interaction (Lizier et al., 2013). To achieve it, the multimodal information can be denoted in a pointwise way, in order to inspect how each event brings information about the target. Denote lower- script r, u, 8 as pointwise interaction corresponding with R, U, S, we can extend Equation event-level:
3.1. Preliminary
In this study, we primarily analyze the mutual informa- tion between two modalities X3 and X2 with respect to a target variable Y, where samples Ø1, 22 and target y can be considered as events from the joint distribution. Our framework can be naturally extended to handle mul- tiple modalities (see Appendix subsubsection A.3.1 for details). For information metrics, we use the subscript i to indicate pointwise mutual information, calculated as p(z.g) i(x;)) v) log p(=))(()' The superscript I denotes average mutual information, Y(: = Ez,p [[; y)).
i(a1;)) = + i(22: v) = = + u2
(2)
i(11, x2: = r u1 + u22 +s
This equation involves four unknown variables: т, U1, Ug. and 8. Since the mutual information terms for unimodal- ity i(x1;)) y), i(12;) y) and multimodality i(x1, y) can be estimated using discriminators or neural estimation meth- ods, the key challenge lies in determining the value of the remaining degree of freedom in Equation 2.
3

---

Efficient Quantification of Multimodal Interaction at Sample Level
X1
X2
X2i
X1iy
(x::))((2;))
X
X2 y
Larger
Smaller Quantities
Quantities
a, B
Join 0; B
Meet
Figure 2: Event-level redundancy information estimation framework. The proposed measure ensures that information quantities monotonically decrease along the decomposition path (indicated by arrows), enabling precise quantification of redundant information components.
To address this challenge, we investigate redundancy from pointwise perspective, aiming to provide reliable and intuitive quantification measure. Redundancy, as funda- mental interaction type, represents the shared information across multiple data sources. By definition, it should not ex- ceed the information present in any individual source. This property enables us to derive redundant interactions through a decomposition framework that eliminates non ronundanndnnt information components. Building on this principle, we develop an event-level decomposition framework for task- related information based on lattice structure. Let and 3 represent two distinct events, where (ox, ()) denotes their joint occurrence and (o: (9) represents their shared compo- nent. This formulation facilitates precise measurement of inter-event relationships, forming the redundancy estimation framework depicted in Figure The framework systemati- cally traces information flow through each lattice point to identify redundant information components. An appropri- ate measure that satisfies this framework would then allow the information within the shared component, specifically (#1; :2; y). to be quantified as redundancy.
sure, which highlighted by previous literature (Finn Lizier, 2018b). Note that this conclusion differs from the results obtained when averaging over the entire distribution, the average information satisfies I(Xm: Y) 2 0 and can ensure monotonicity.
Hence, there is need for an alternative feasible way for determining redundancy. Inspired by (Ince, 2017; Finn & Lizier, 2018a), we can apply redundancy on partial infor- mation component, and obtain the redundant information. In detail, we divide the information into components i(2; y) i (x;;)) - i (z; y). This division should be uniquely determined given 2, y. and both compo- nents should satisfy the monopoly condition and be positive. Accordingly. we use the following division:
information
it(x; v) = h(z)) log p(z),
(4)
i (x;;;) = h(xly) log p(rl))
Therefore, given the positive property, each component within this division adheres to monotonicity. As a result, both measures, it and i- are compatible with the lattice Figure 2. Consequently, we define the redundancies r and r- each component for
With this framework, a significant challenge arises in apply- ing reasonable measure for the framework. The measure should satisfy the monotonic decrease of redundancy de- composition each lattice (Figure 2 from left to right). An intuitive approach is to apply the pointwise information measure directly on this framework. However, pointwise mutual information ((x;y) can be negative (when & pro- vides misleading information about y), monotonicity is not always guaranteed. Concretely, according to the redundancy framework, the following inequality should hold:
+ ((13ag9)) = min (i (i*())))); (#2:
(5)
r (x::02y)) = min (x):)), (x2; (12
With this definition, we integrate redundancy from both components to obtain the event-level redundant interaction.
r .22 v) + (#1: (:::0:;11
(x1: (2:::22:00 v),
(6)
((x1:0) (((11))) (x2: < i(z1;)) i(z2:v). (3)
which uniquely determines the multimodal interaction. This enables us to accurately determine the respective values of "1, U2 and s using Equation 2. Furthermore, we can obtain the average interaction values R, U1. U2, and within the dataset by averaging over these sample-level interactions.
However, the pointwise mutual information i(02; y) can be negative, which would violate the inequality. Thus, the monotonicity over the redundancy framework cannot consistently achieved directly under the information mea-

---

Efficient Quantification of Multimodal Interaction at Sample Level
Task
XOR
OR
XOR+NOT
Interaction
R
U1
U2
S
R
U1
U2
S
R
U1
U2
S
PID-CVX
0.000
0.000
0.000
0.692
0.210
0.001
0.000
0.342
0.000
0,000
0.338
0.346
PID-Batch
0.000
0.002
0.002
0.690
0.200
0.018
0.018
0.322
0.003
0.000
0.257
0.381
LSMI (ours)
0.000
0.001
0.001
0.691
0.215
0.001
0.000
0.345
0.000
0.000
0.336
0.347
GT
0.000
0.000
0.000
0.693
0.215
0.000
0.000
0.347
0.000
0.000
0.347
0.347
Table 1: Comparison of the proposed LSMI with previous interaction estimators circuit logic (XOR, OR, and XOR+NOT).
Algorithm 1 Lightweight Sample-wise Multimodal Interac- tion Estimation (LSMI) Algorithm
for entropy estimation, thereby reducing time consumption typically associated with distribution modeling. Further- more, our lightweight method facilitates the measurement of pointwise interactions, a capability not extendable the sample level in previous approaches. Overall, this approach enhances flexibility and scalability, thereby providing in- sights and explanations for multimodal learning.
1: Input: Bimodal data T1, 22. target y; discriminative models p(y|=1 x2), p(y[=1), p(y(22))
2: Initialize: Entropy estimators he, (.), (+))
Train entropy estimators he ho, using Equation 7 on data from p(z1), p(22) respectively.
4:
Compute sample-wise h(21), h((2)) using hes hegi
then compute h(21 (), (x22ly) via Equation 8.
4. Experiment
5: Compute pointwise redundancy indicators rt, r7 via
Equation 5; then redundancy r f- -r----
4.1. Synthetic Experiment
6: Compute pointwise i(a1; y), (xx2; y), ((11, us-
4.1.1. SETUP
ing tions u1, U22 8 via Equation 2.
p(yl (11, x2); then derive interac-
In synthetic experiments, we validate the precision of our es- timation approach using two typical distributions: bitwise circuit system and mixture of Gaussians, where interac- tions can be explicitly calculated. We also conduct exper- iments with manually controlled interactions to elucidate the effects of different interaction patterns. To better adapt to real-world data scenarios, additional noise is introduced into these datasets, making their data distributions continu- ous. The generation process of these datasets illustrated in subsection A.1. Our estimation method is compared with the CVX (PID-CVX) estimator, designed for discrete distributions (Liang et al., 2023b), and the batch-level (PID- Batch) estimator, which is applicable to both discrete and continuous distributions (Liang et al., 2023b).
7: Output: Sample-wist interactions T, U1, U2,
3.3. Lightweight Interaction Estimation
Although we have defined the measure for redundancy de- composition well, how to quantify this interaction over con- tinuous distributions and real-world datasets remains ques- tion. In this work, we refer to the tool KNIFE (Pichler et al., 2022) the differential entropy estimation, which is effi- cient and suitable for sample-wise estimation over complex distributions. Denote he as entropy estimator,
Eiho [+] E[((x)] + DKL (xL(p())llo(())) 2 H(X) (7)
4.1.2. CIRCUIT LOGIC
serves as an upper bound for entropy, as the Kullback- Leibler divergence (D is inherently positive. Therefore, the estimator can be optimized by tuning parameters 0 to minimize the DKL. thereby tightening this upper bound. Consequently, we derive he, (x1) and he, (x2) as estima- tions for the positive information component i as spec- ified in Equation 4. Additionally, once the unimodal dis- criminative models p(yl=,) ) and p(y)22) are determined, the negative component: i can be estimated as follows:
We first employ bitwise circuit logic systems validating precision. We construct samples using fundamental logic operations (e.g., OR, XOR, and a mixture of NOT and OR) combined with small Gaussian noise. The crucial advan- tage this setup its capacity to provide an objective and verifiable benchmark: the deterministic nature of logic op- erations allows for sound calculation of ground truth (GT) interactions (Bertschinger et al., 2014). Consequently, a closer alignment between predicted interactions and this GT directly signifies a more accurate estimation. We com- pare our method against PID-CVX and PID- -D-BBtch& (Liang et al., 2023b). The experimental results in Table demon- strate that our estimators accurately recover the correct in- teractions, closely matching the ground truth and showing favorable performance compared to these baseline methods.
(zmi y) hem. (zm)-h(w) log p(y)=m), [2]. (8)
Details are shown in Algorithm 1. By adopting this method, we circumvent the necessity of modeling the base distribu- tion to achieve interaction decomposition. Consequently, our approach demands only a limited number of parameters
5

---

Efficient Quantification of Multimodal Interaction at Sample Level
20
32
13M0 HiDBick HD-XX
14
45-
10
10.
10.3
CT
00
10 100
35
(a) Estimation on R.
(b) Estimation on
Figure 3: Comparison of estimators data with mixture of Gaussian distributions.
(a) PID-CVX
(b) PID-Bateh
4.1.3. MIXTURE OF GAUSSIAN
40
For the simulation of continuous distribution, we design dataset based on mixture of Gaussian distributions. Specif- ically, for both œ and Z2, the distributions are defined as:
L0 4/55
K
p(mm) (xrloo) E
(9)
(c) LSMI
(d) GT
where different modalities share the same mean p( and covariance 5)() This distribution can be approximated as a K-way classification task, which is useful for measuring the interaction between 21, 32 and y.
Figure 4: Comparison of estimators with preset interactions.
(Figure 4 (a)) and PID-Batch (Figure 4 (b) show discrep- ancies in estimating Synergy interaction (within the green lines), whereas our method (Figure (c)) handles these interactions effectively, enabling more precise estimations.
To manipulate interactions between modalities, impose constraints on the covariance of Gaussian distributions for the two modalities. Specifically, we adjust the covariance p(œ1, 12) {-1, 0.8, -0.4, O, 0.4, 0.8, 1} to obtain different joint distributions. The intended effect of these adjustments that they only influence the amplitude of the synergy interaction, while the redundancy and uniqueness of interactions remain unchanged, as outlined by (Bertschinger al., 2014). The experimental comparisons, detailed in Fig- ure 3, begin with a ground truth (GT) calculated through mixture of Gaussian model. Our findings indicate that both PID-CVX and BuD-Batch fail to consistently estimate inter- actions, showing divergence from the ground truth despite exhibiting similar trends across different interaction types as p changes. In contrast, our proposed method demonstrates high capacity closely match the ground truth interactions, showcasing the precision of our estimation approach.
4.2. Validation on Real-world Datasets
We apply our LSMI approach real-world datasets to demonstrate its capability of efficiently estimating multi- modal interactions at the sample level and providing valu- able insights into interaction modeling. Additional experi- ments, which include multiple modal analyses, architectural comparisons, and distribution shifting evaluations, are de- tailed in subsection A.3.
4.2.1. 1. EXPERIMENTAL SETTING
Dataset We conduct experiments extensive multimodal datasets encompassing various tasks and modalities. These include Food-101 101 (Bossard et al., 2014), which focuses on food classification using text and image modalities; CREMA-D (Cao et al., 2014), dedicated to emotion anal- ysis with audio and visual modalities; Kinetic-Sounds (KS) (Arandjelovic Zisserman, 2017), an action recog- nition task employing audio and visual modalities: UCF- 101 (Soomro al., 2012), multimodal action recognition dataset utilizing RGB and optical flow modalities; CMU- MOSEI (Zadeh et al., 2018), which addresses binary sen- timent analysis through video (including audio and visual) and text modalities; and UR-funny (Hasan et al., 2019), aimed humor detection using video and text.
4.1.4. .. PRESET INTERACTION
We also verify the precision of manually designed datasets with controllable interactions. In this dataset, we ensure that each sample contains one type of RenertiinnnRRuundnnnnnnyy Uniqueness, or Synergy-and adjust the proportion of each mix
sample to preset the interaction. Consequently, these data in certain proportions (e.g., J + S indicates that } of the data follows Uniqueness while the remainder exhibits Synergy). The experimental results are presented in Figure 4. The ground truth the radar shown in Figure (d), which is asymmetric due to our setting. Both PID-CVX
6

---

Efficient Quantification of Multimodal Interaction at Sample Level
Dataset
KS
Food- 101
UR-Funny
CMU-MOSEI
Interaction
R
U1
U2
S
R
U1
U2
S
R
U1
U2
S
R
U1
U2
S
PID-Batch
3.16
0.02
0.19
0.01
4.23
0.24
0.00
0.14
0.02
0.03
0.01
0.06
0.18
0.34
0.02
0.03
LSMI
3.28
0.11
0.00
0.03
4.19
0.34
0.00
0.08
0.02
0.12
0.01
0.24
0.13
0.22
0.01
0.00
Human
2.32
1.61
1.45
0.48
4.06
0.92
0.05
0.00
2.30
2.73
2.33
2.50
3.27
3.37
2.87
1.03
Table 2: Comparison of average interaction over various real-world datasets.
Interaction
Categories preferred by each interaction
playing organ pushing lawnmower blowing nose
playing bagpipes shoveling snow
playing keyboard shuffling cards
playing accordion dribbling ball playing guitar blowing out
playing drums bowling playing clarinet
Uby
Ua
laughing tickling
tapping guitar
s
ripping paper
tap dancing
Table 3: Demonstration of the categories that most prefer specific types of interactions on the KS dataset.
Method
R
U1
U2
S
evidenced by minimal generalization error. Unimodal and multimodal information are then extracted by probing these learned representations. The dataset-level interactions are subsequently determined by averaging the sample-level in- teractions obtained from these selected models. To validate the reasonability of our approach, we compare our method with continuous interaction estimation (PID-batch) and hu- man judgment as references. Since there are no predefined interaction values for real-world continuous datasets, we follow previous literature that annotated several datasets, including CMU-MOSEI and UR-Funny, and use the same settings as those studies (Liang et al., 2023b) to construct human judgment for interactions on the KS and Food-101 datasets. The comparison results are shown in Table 2. We observe that our LSMI estimator is largely consistent with the PID-Batch method. Furthermore, in KS, Food-101 101, and MOSEI datasets, our method aligns with the top two highest interactions in terms of redundancy and uniqueness, respec- tively. Similarly, in the UR-Funny dataset, our method exhibits strong correlations with the expected interaction patterns. While human aamannannotated scores (on a 0-5 scale) are not direct measures of information content, we observe strong Pearson correlations between LSMI estimates and human judgments: 0.98 for redundancy and 0.95 for text uniqueness on Food-101 dataset, indicating significant align- ment in interaction quantification.
Feature-level fusion
Joint
3.165
0.143
0.000
0.122
MMIB
3.284
0.113
0,000
0.030
Bilevel
2.604
0.552
0,000
0.277
Decision-level fusion
Additive 3.397
0.006
0.000
0.029
Weighted 3.399
0.010
0.000
0.024
QMF
3.400
0.002
0.000
0.032
Additional Regulation
Mod-drop 3.163
0.134
0.000
0.116
Alignment 3.372
0.015
0.000
0.040
Recon 2.984
0.311
0.000
0.139
Table 4: Comparison of interaction components across dif- ferent multimodal learning methods on the KS dataset.
Baseline We adopt three primary types of multimodal learning paradigms, details are shown in subsection A.2.
Feature-level fusion: Joint learning (Baltrušaitis et al., 2018), MMIB (Mai al., 2022), Bilinear (Fukui et al., 2016). Decision-level fusion: Additive ensemble (Liang et al., 2021), Weighted ensemble (Shao al., 2024), QMF(Zhang et al., 2023a). Additional Regulation: Mod- drop (Hussen Abdelaziz et al., 2020), Alignment (Radford et al., 2021), Rec (Tsang et al., 2020).
4.2.3. CASE STUDY
4.2.2. DATASET INTERACTIONS
Our proposed LSMI estimator enables us to observe interac- tion variations at the sample level, providing insights into interaction differences across groups or categories. Specifi- cally, we compute the interaction preferences for each cat- egory by averaging the interactions of the samples within that category. We analyze the top categories with the high- est estimated values for redundancy r, visual uniqueness ue. auditory uniqueness ug. and synergy s. The results are presented in Table 3. Notably, instruments, which are easily
Our LSMI-estimate method can be employed to uncover the inherent multimodal interactions within datasets, which varies significantly across diverse tasks and data domains. Modeling the true data distribution of real-world datasets is inherently complex. Accurate multimodal interaction estimation requires models that faithfully capture the true data distribution. Hence, we identify and utilize trained models that approximate the underlying data distribution, as

---

Efficient Quantification of Multimodal Interaction at Sample Level
UR-Funny
CMU-MOSEI
CREMA-D
KS
UCF-101
Food- 101
Number of classes
2
2
6
31
101
101
LSMI (s)
454.4
667.1
426.1
501.5
678.9
504.0
PID-Batch(s)
1700.5
3124.4
5876.5
21928.0
48576.6
59679.5
Table 5: Comparison of time cost (s) over real-world datasets.
distinguishable by both sound and image, exhibit high
KS
CREMA-D
level of redundancy. Categories associated with visual ele- ments (e-g., grass, snow) tend to prefer the visual modality. In contrast, categories where auditory features are more dis- tinctive (e.g., blowing nose) show greater uniqueness in the audio modality. For tasks with more complex recognition (e.g., tickling), single modalities may struggle to distinguish the categories, leading these tasks to rely more on synergy. This finding aligns with human cogmition. Additionally, conduct sample-wise case studies compare human annotations with model-estimated interactions. The compar- ison, shown in subsubsection A.3.4, demonstrates that our interaction estimation aligns well with human recognition, further validating its accuracy.
Data
V+A
V
A
V+A
V
A
All
0.854
0.818
0.727
0.795
0.684
0.725
Low High
0.850
0.805
0.729
0.782
0.702
0.715
0.877
0.824
0.726
0.801
0.688
0.728
Table 6: Performance comparison of ImageBind model fine- tuned on complete dataset (AlI), low-redundancy subset (Low), and high-redundancy subset (High) across unimodal and multimodal settings.
4.3. Application
As our LSMI estimator has demonstrated both accuracy and efficiency, how such interaction estimation can offer novel perspectives and tangible benefits remains critical. In this section, we demonstrate how sample-level multimodal interaction estimation can be applied to downstream tasks to boost multimodal performance. Specifically, utility in two main directions: (I) On the learning side, we investigate how interactions can improve dataset partitioning and enable targeted model distillation; (2) On the inference side, we explore the design of efficient ensemble strategies leveraging interaction patterns.
4.2.4. INTERACTION MODELING COMPARISON
Our LSMI framework reveals distinct capabilities of vari- ous multimodal learning paradigms in modeling different information interaction types, as demonstrated in Table 4. Feature-level fusion methods exhibit comprehensive capa- bilities for learning diverse interactions, while decision-level fusion approaches specialize in capturing data redundancy. Notably, specific regulation techniques show targeted ad- vantages of interaction modeling: alignment-basee based meth- ods effectively enhance redundant interaction learning, and reconstruction-foouss approaches improve modeling of modality-specific unique information.
explore its
4.3.1. TARGETED DATA PARTITION
Effective multimodal learning depend on the nature of train- ing data, with data interactions playing crucial role. Our LSMI method addresses this by explicitly measuring these interactions and providing a quantifiable metric. This met- ric enables data partitioning based on interaction patterns suitable for specific frameworks, allowing models to learn from more tailored data subsets and ultimately boosting their performance. In this work, we employ the ImageBind model (Girdhar et al., 2023), which aligns different modali- ties into common space. Since this alignment process nat- urally maximizes shared refflttton-rlllcttin redundant interactions- we categorize samples into high-redundancy (High) and low-redundancy (Low) subsets based on their modal redundancy. We fine-tune ImageBind on these dis- tinct subsets using its contrastive loss. To validate learn- ing quality, we use simple linear layers the extracted ImageBind features for task performance evaluation. The results in Table 6 show performance across multimodal and unimodal settings. Our experiments demonstrate that fine-tuning on the high-redundann subset enhances multi- modal feature quality through improved modality alignment
4.2.5. TIME EFFICIENCY
We compare the time cost of interaction estimation between our method and the PID--atch method. The key difference lies in the way the two methods handle interaction mea- surement. Our method is designed to measure entropy by learning the mapping from samples within each individual modality to pointwise entropy. i.e., X' + R", m E [2]. In contrast, FID-Batch aims to model the entire distribution Xi x X2 x y - R", and then calculate the interaction based on this joint distribution. As shown in Table 5, our lightweight estimator demonstrates significantly higher ef- ficiency in interaction estimation compared to PID-Batch. Specifically, the computational cost of PID-Batch scales with the number of classes, resulting in substantial time complexity when the category space is large (c.g., UCF-101 and F01d-101). In contrast, our LSMI estimator avoids joint distribution modeling, ensuring consistent time efficiency across tasks, regardless of the number of classes.
8

---

Efficient Quantification of Multimodal Interaction at Sample Level
Ape m
informative knowledge compared direct distillation. This indicates that targeted distillation approach facilitates the learning of specific and distinctive information, enhanc- ing the model's 's overall performance.
47-
05
04
4.3.3. INTERACTION-GUDDE MODEL ENSEMBLE
12
Aymuh
(a) KS dataset
(b) UCF dataset
When several vell-trained models are available, an impor- tant question arises: how can we leverage interaction metrics to exploit the strengths of different models? Sample-level in- teractions can serve as indicators measure the differences in how models extract information from individual sam- ples, shedding light on more reliable ensemble approach. In this study, we conduct experiments with several well- trained. yet diverse, multimodal approaches. We compute the interaction for each sample by averaging the interac- tions across different approaches and then ensemble each unimodal model with the corresponding interaction, which we call the LSMI-based ensemble. For comparison, we also implement a weighted ensemble and provide details on the unimodal accuracy. As shown in Figure 6, we demon- strate that even when the added models have lower accuracy than the original model, they still contribute significantly to performance improvements. This because different models tend to focus different interaction patterns, and ensembling based on interaction results in capturing more accurate interactions. This explains why LSMI-based en- semble methods outperform simple weighted ensembles.
Figure 5: Validation on LSMI-based distillation approach.
42/
(a) CREMA-D dataset
(b) KS dataset.
Figure 6: Comparison between LSMI ensemble and weighted ensemble with various datasets.
facilitated by redundant interactions. This alignment pri- marily benefits information-rich modalities (e.g., Vision in KS, Audio in CREMA-D), enabling more effective learn- ing while preventing misalignment with information- modalities. Conversely, the low-redundancy subset allows more thorough exploration of information-pon modalities Audio in KS, Vision in CREMA-D), leading to their relatively stronger performance. These findings highlight the distinct roles of different data interactions training and demonstrate the practical value of our sample-level interaction estimation for data partitioning.
5. Conclusion
This work aims estimate the multimodal interaction of each sample, clarifying the quantities of interaction in redun- daney, uniqueness, and synergy. We propose a lightweight entropy- based multimodal interaction estimation approach for efficient and precise sample-wise interaction measure- ment across various continuous distributions. We demon- strate the precision and efficiency of our estimation, as well as the utility of this sample-level estimation for guiding and improving multimodal learning. Our estimation more ac- curately reveals the information generation within the data, offering finer-graine insights that empower data-driven dataset construction and sample-specific algorithm design.
4.3.2. INTERACTION-GUDDE DISTILLATION
Beyond its utility in interaction data partitioning, our method also offers valuable insights into model learning dynamics. LSMI can reflect how a multimodal model cap- tures dynamic interaction patterns unique to each sample, enabling targeted interaction adjustment. To leverage these insights, introduce efficient knowledge distillation framework that utilizes sample-level interaction patterns estimated by a nigh-performing teacher model as infor- mative signals to determine which knowledge components should be distilled into a student model trained from scratch. In particular: For redundancy (r) and uniqueness (u), we distill informative unimodal features. For synergy (s), we employ output-level distillation to capture cross-modal com- plementary effects. The distillation process is weighted by the relative magnitudes of r, and s. As baseline comparison, we also evaluate method that directly distills features from each individual modality, as illustrated in Fig- ure 5. Our experimental results demonstrate that our method learns more effectively and acquires greater amount of
Future work This work estimating sample-level inter- actions opens several avenues for future research, including: (I) investigating how models dynamically capture interac- tions within complex fusion mechanisms to elucidate adap- tive learning and synergistic information generation; (2) uncovering relationships between training strategies and learned modal interactions optimize multimodal learning systems; and (3) exploring how interaction estimation can enhance multimodal representation learning and modality- specific information acquisition, thereby enabling more fine- grained and dynamic interactive learning.

---

Efficient Quantification of Multimodal Interaction at Sample Level
Acknowledgement
for visual question answering and visual grounding. arXiv preprint arXiv: 2016.
This work sponsored by CCF-Tencent Rhino-Bine Open Research Fund, the National Natural Science Foundation of China (Grant No. No.62106272), the Public Computing Cloud of Renmin University of China, and the fund for building world-class universities (disciplines) of Renmin University of China.
Gat, I., Schwartz, I., Schwing, A., and Hazan, T. Remov- ing bias in multi-modal classifiers: Regularization by maximizing functional entropies. Advances in Neural Information Processing Systems, 33:3197-3208, 2020.
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, Imagebind: One embedding space bind them all. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 151 15180-15190, 2023.
Impact Statement
This paper presents work aimed at advancing the field of multimodal learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Hasan, M. K., Rahman, W., Zadeh, A., Zhong, J., Tanveer, M. I., Morency, I..P.,, et Ur-funny: A multimodal language dataset for understanding humor, arXiv preprint arXiv: arXiv:1904. peee:990440.010 2019.
References
Arandjelovic, R. and Zisserman, A. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pp. 609-617,20111
Hussen Abdelaziz, A., Theobald, B.-J., Dixon, P., Knothe, R., Apostoloff, N., and Kajareker, S. Modality dropout for improved performance-dree driven talking faces. In Proceed- ings of the 2020 International Conference on Multimodal Interaction, pp. 378-386, 2020.
Baltrušaitis, T., Ahuja, C., and Morency, L.-P. Multimodal machine learning: A survey and taxonomy. IEEE trans- actions on pattern analysis and machine intelligence, 41 (2):423-443, 2018
Ince. R. A. Measuring multivariate redundant information with pointwise common change in surprisal. Entropy, 19 (7):318, 2017.
Bertschinger, N., Rauh, J., Olbrich, Jost, J., and Ay, N. Quantifying unique information. Entropy, 16(4):2161- 2183, 2014.
Jayakumar, S. M., Czamecki, W. M., Menick, J., Schwarz, J., Rae, J., Osindero, S., Teh, Y. W., Harley, T., and Pas- canu, R. Multiplicative interactions and where to find them. In International conference on learning represen- tations, 2020.
Bossard, L., Guillaumin, M., and Van Gool, L. Food-101- mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Con- ference, Zurich, Switzerland, September 6-12, 2014, Pro- ceedings, Part VI 13, pp. 446-461. Springer, 2014.
Knuth, K. H. Lattices and their consistent quantification. Annalen der Physik, 531(3):1700370, 2019.
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and Verma, R. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-399 2014.
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. Multibench: Multiscale benchmarks for multimodal representation learning. arXiv preprint arXiv:2107.07502, 2021.
Celotto, M., Bim, J.. Tlaie, A., De Feo, V., Toso, A., Lemke, S., Chicharro, D., Nili, H., Bieler, M., Hlanganu-Opatz, I,, et al. An information-theoredt quantification of the con- tent of communication between brain regions. Advances in Neural Information Processing Systems, 36, 2024.
Liang, P. P., Ling, C. K., Cheng, Y., Obolenskiy, A., Liu, Y., Pandey, R., Wilf, A., Morency, L.-P., and Salakhut- dinov, R. Multimodal learning without labeled multi- modal data: Guarantees and applications. arXiv preprint SKTi:2306.04330 2023a.
Finn, C. and Lizier, T. Pointwise partial information decompositionusing the specificity and ambiguity lattices. Entropy, 20(4):297, 2018a.
Liang, P. P., Ling, C. K., Cheng, Y., Obolenskiy, A., Liu, Y., Pandey, R., Wilf, A., Morency, L.-P., and Salakhut- dinov, R. Quantifying interactions in semi-supervised multimodal learning: Guarantees and applications. In The Twelfth International Conference on Learning Repre- sentations, 2023b.
Finn, C. and Lizier, J. T. Probability mass exclusions and the directed components of mutual information. Entropy, 20611):826, 2018b.
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., and Rohrbach, M. Multimodal compact bilinear pooling
10

---

Efficient Quantification of Multimodal Interaction at Sample Level
Lizier, J. T., Flecker, B., and Williams, P. L.. Towards synergy-based approach to measuring information mod- ification. In 2013 IEEE Symposium Artificial Life (ALIFE), pp. 43 IEEE. 2013.
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. 2012.
arXiv preprint wrrxiv:112..00
Tsai, Y.-H. H.,, Liang, P. P., Zadeh, A., Morency, L.-P., and Salakhutdinov. R. Learning factorized multimodal representations. arXiv preprint arXiv: arXiv:/806. 1806:06/776 06/76, 2018.
Lizier, J. T., Bertschinger, N., Jost, J., and Wibral, M. Infor- mation decomposition of target effects from multi-source interactions: Perspectives on previous, current and future work, 2018.
Tsang, M., Cheng, D., Liu, H., Feng, X., Zhou, E., and Liu, Y. Feature interaction interpretability: A case for explain- ing ad ud-recommendation systems via neural interaction detection. ar preprint arXiv:2006. 10966, 2020.
Luppi, A. I., Rosas, F. E., Mediano, P. A., Menon, D. K., and Stamatakis, E. A. Information decomposition and the informational architecture of the brain. Trends in Cognitive Sciences, 28(4):352-368 2024.
Venkatesh, P., Bennett, C., Gale, S., Ramirez, T., Heller, G., Durand, S., Olsen, S., and Mihalas, S. Gaussian partial information decomposition: Bias correction and application to nigh-dimmnsiooal data. Advances in Neural Information Processing Systems, 36, 2024.
Mages, T. and Rohner, C. Decomposing and tracing mutual information by quantifying reachable decision regions. Entropy, 25(7):1014, 2023.
Mai, S., Zeng, Y., and Hu, H. Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations. IEEE Transactions on Multi- media, 25:4121-4134 2022.
Wibral, M., Priesemann, V., Kay, J. W., Lizier, J. T., and Phillips, W. A. Partial information decomposition as unified approach to the specification of neural goal functions. Brain and cognition, 112:25-38 2017.
Pakman, A., Nejatbakhsh, A., Gilboa, D., Makkeh, A., Mazzucato, L., Wibral, M., and Schneidman, E. Esti- mating the unique information continuous variables. Advances in neural information processing systems, 34: 20295-20307, 2021.
Williams, P. L. and Beer, R. D. Nonnegative decom- position of multivariate information. arXiv preprint
Wu, M. and Goodman, N. Multimodal generative models for scalable weakly-supervised learning. Advances in neural information processing systems, 31, 2018.
Peng, X., Wei, Y.. Deng, A., Wang, D., and Hu, D. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pullern Recognition, pp. 8238-8247, 2022.
Xu, P., Zhu, X., and Clifton, D. A. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Pichler, G., Colombo, P. J. A., Boudiaf, M., Koliander, G., and Piantanida, P. A differential entropy estimator for training neural networks. In International Conference on Machine Learning, pp. 17691-17715. PMLR, 2022.
Yang, Y., Fu, Z., Zhan, D., Liu, Z., and Jiang, Y. Semi- supervised multi multi-modal multi multi-instance multi-label deep network with optimal transport. IEEE Trans. Knowl. Data Eng., 33(2):696-709, 2021.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8787 PMLR, 2021.
Yang, Y., Wan, F., Jiang, Q. and Xu, Y. Facilitating multi- modal classification via dynamically learning modality gap. In Advances in Neural Information Processing Sys- tems 38, 2024.
Yang. Y., Pan, H., Jiang, Q., Xu, Y., and Tang, Leam- ing rebalance multi-modal optimization by adaptively masking subnetworks. IEEE Trans. Pattern Anal. Mach. Intell., 2025.
Rahate, A., Walambe, R., Ramanna, S., and Kotecha, K. Multimodal co co-learning: Challenges, applications with datasets, recent advances and future directions. Informa- fion Fusion, 81 81:203-239, 2022.
Yuan, X., Lin, Z., Kuen, J., Zhang, J., Wang, Y., Maire, M., Kale, A., and Faieta, B. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6995-7004, 2021.
Shao, Z., Dou, W., and Pan, Y. Dual-level deep evidential fusion: Integrating multimodal information for enhanced reliable decision decision-making in deep learning. Information Fusion, 103:1021 13, 2024.

---

Efficient Quantification of Multimodal Interaction at Sample Level
Zadeh, A. B., Liang, P. P., Poria, S., Cambria, E., and Morency, I..P. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fu- sion graph. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers), pp. 2236-2246, 2018.
Zhan, K., Nie, F., Wang, J., and Yang, Y. Multiview con- sensus graph clustering. IEEE Transactions on Image Processing, 28(3):1261-1270, 2018.
Zhang, Q.. Wu, H., Zhang, C., Hu, Q., Fu, H., Zhou, J. T., and Peng, X. Provable dynamic fusion for low-quality multimodal data. In International conference on machine learning, pp. 41753-41769, PMLR, 2023a.
Zhang, Y., Latham, P. E., and Saxe, A. A theory of unimodal bias in multimodal learning. arXiv preprint N.Ni:23/2.00033:: 2023b.
12

---

Efficient Quantification of Multimodal Interaction at Sample Level
A. Experiments
A.1. Synthetic Data Generation
In subsection 4.1, we employ three types of synthetic datasets to support our study. The first type is a circuit logic dataset, described in subsubsection 4.1.2, which based on Boolean logic operators. The second type a mixture of Gaussians with varying correlation coefficients (p), designed simulate scenarios where unimodal distributions remain constant while multimodal distributions change, as detailed in subsubsection 4.1.3. The third type is preset interaction dataset, where each dataset is constructed to include one or two predefined types of interactions, enabling the study of specific interaction dynamics, as outlined in subsubsection 4.1.4. Since the interactions in logic data can be easily determined (Bertschinger et al., 2014), this dataset provides a reliable ground truth for precision validation.
The circuit logic dataset, the first type in our study, is based on three fundamental Boolean logic operations: OR, XOR, and NOT. Each sample contains only one type of logic operation. For cases involving multiple logic operations (e.g. NOT)NOT). each logic type is equally represented the dataset. To increase complexity, we add noise each input dimension, requiring the evaluation model to denoise the input variables before estimating the corresponding logical relationships. As interactions in logic data can be precisely determined (Bertschinger et al., 2014), this dataset provides reliable ground truth for validating our approach's 's precision.
roo .05
mo 3
/Pa 0.5
28.
30-
2 63
65
14-
58
400
-5
-14-
-00
(a) p -0.5
(b) p 0
(c) p 0.5
Figure Illustration of bivariate Gaussian distributions with different correlation coefficients (p). From left right: negative correlation (p 0.5), correlation (p = 0), and positive correlation (p 0.5). These visualizations demonstrate how varying correlation affects the joint distribution while keeping marginal distributions constant.
The second type of dataset, a mixture of Gaussians, is designed to reflect variations in interactions when unimodal distributions remain constant while multimodal distributions change. Specifically, each individual modality follows a mixture of Gaussian distributions:
p( (m
Σ (v) (
(10)
where the unimodal distributions are identical across modalities. To simulate different interaction patterns, adjust the joint distribution by setting the covariance p(#1, 2221)) yy) to (-1, -0.8. -..4 0.4, 0.8, 1}. For illustration, component Gaussian mixture model to compute the means and variances (see Figure 7). We then apply PID decomposition (Bertschinger et al., 2014) to estimate the ground truth. When the unimodal distributions are held constant, the redundant (R) and unique (U) components remain unchanged, while only the synergistic (S) component varies, as demonstrated the results of Figure 3.
two-
For the third type, the dataset contains preset interactions, allowing for selective design of interactions evaluate the capabilities of different methods under varying interaction patterns. In this synthetic data, each sample exhibits only specific type of interaction, requiring the interaction estimation model to model the underlying distribution of each sample. As illustrated in Figure 4, each dataset is composed of samples exhibiting one two types of interactions. The proportion of each interaction type is quantified using fractional notation, such as _ - R. This indicates that 1 of the samples display Unique interactions, while the remaining samples demonstrate Redundant interactions. The data generation process is executed in two sequential steps. First, the type of interaction for each sample is determined. Next, different interactions are mapped into nigh-dimensional data through linear transformations, For redundancy, both modalities are
13

---

Efficient Quantification of Multimodal Interaction at Sample Level
UCF-101 Dataset
CMU-MOSEI Dataset
Modality Pair
R
U1
U2
S
Modality Pair
R
U1
U2
S
Visual-OF
2.111
2.483
0.000
0.000
Vision-Text (V-T)
0.121
0.000
0.163
0.005
Visual-Diff
3.474
1.121
0.000
0.000
Vision-Audio (V-A)
0.116
0.010
0.000
0.012
OF-Diff
1.998
0.003
1.476
0.239
Audio-Text (A-T)
0.127
0.000
0.248
0.002
Table 7: Pairwise interaction analysis on the UCF-101 and CMU-MOSEI datasets, with different modality combinations.
mapped simultaneously. For uniqueness, the non non-informative modality is replaced with Gaussian noise. For synergy, an XOR-like construction is employed, ensuring that information is only effective when both modalities are present. To align with real-world scenarios, additional Gaussian noise is introduced into the high-dimensional data. This setup facilitates the experimental comparisons presented Figure 4.
A.2. Baseline
We adopt three primary types of multimodal learning paradigms:
Feature-level fusion: Integration of multiple modalities the feature level. This includes: Joint learning (Baltrušaitis al., 2018): A traditional paradigm where features from different modalities are concatenated and jointly mapped into the target space. MMIB (Multimodal Information Bottleneck) (Mai et al., 2022): Application of the variational information bottleneck principle to feature fusion. Bilinear (Fukui et al., 2016); Incorporation of dynamic interactions with learnable weights.
Decision-level fusion: Integration of unimodal predictions from different modalities. This approach includes: Addi- tive (Liang et al., 2021): Ensemble of predictions by averaging. Weighted (Shao et al., 2024): Ensemble of predictions with pre-learned weights. QMF(Zhang al., 2023a): Dynamic learning for each unimodality and learning weights.
Additional Regulation: Implementation of supplementary regulations to enhance multimodal learning: Mod- drop (Hussen Abdelaziz et al., 2020): Application of dropout to partial modalities to prevent overfitting. Alignment (Radford et al., 2021): Introduction of contrastive loss to align modalities. Ree (Tsang et al., 2020): Application of unimodal recon- struction loss to strengthen unimodal capabilities.
A.3. Additional Experiments
A.3.1. EXTENSION TO MULTIPLE MODALITIES
Analyzing interactions involving more than two modalities presents considerable challenges in quantifying their complex relationships, such as the mutual information shared by three or more sources concerning target task. A significant barrier is that established theoretical PID frameworks, designed for decomposing two-way interactions into synergistic, redundant, and unique components, do not directly or uniquely extend to these higher-order scenarios (Mages & Rohner, 2023; Williams & Beer, 2010). This is primarily because the information among more than three variables (including the various modalities and the target) become substantially more intricate.
Given the theoretical limitations in decomposing interactions beyond two modalities (Mages & Rohner, 2023; Williams & Beer, 2010), we adopt the pairwise interaction analysis strategy, as detailed in Appendix C.4 of (Liang et al., 2023b). This method systematically examines interactions between each pair of modalities. We applied this approach to the UCF-101 dataset (comprising Vision, frame difference (Diff), and optical flow (OF)) and the CMU-MOSEI dataset (Vision, Audio, and Text). The quantitative results of these pairwise interactions are presented in Table 7.
For the UCF-101 dataset, the analysis in Table 7 reveals several key patterns. Vision emerges as the most informative modality in terms of unique contributions (U₃ 2.483 when paired with OF, and U1 1.121 when paired with Diff). A strong redundant relationship is observed between Vision and Diff (R 3.474). In contrast, the OF-Diff pair exhibits notable synergy (S = 0.239). This synergy is likely due to OF and Diff individually conveying less task-relevant unique information (UoF = 0.003 and UDiT = 1.476 in their pair, as U1 and U2 respectively), thus benefiting from combining their complementary aspects.
On the CMU-MOSEI dataset, Table indicates that Text the primary modality, consistently showing the highest unique
14

---

Efficient Quantification of Multimodal Interaction at Sample Level
Clean Label
Noisy Label
Sample
r
1/1
Ug
8
Total
ui
tig
8
Total
#1
4.615
-0.005
0.000
0.005
4.615
-13.719
5,495
0.000
-5.582
-138006
#2
0.048
4.567
0.000
-0.001
4.614
-1.856
-11.036
0.000
-0.581
-13.473
#3
2.466
0.000
2.148
0.001
4.615
-13.806
2.933
0.000
-2.933
-13.806
Table 8: Comparison of L.MI interaction estimation between samples with clean and noisy label.
Dataset
Distribution
R
U1
U2
S
Total
ID
3.319
1.289
0.000
0.006
4.614
UCF
OOD
2.511
0.504
0.053
0.698
3.766
ID
2.371
0.031
0.730
0.300
3.432
KS
OOD
1.864
0.083
0.386
0.559
2.892
Table Comparison of LSMI-estimated multimodal interactions between In-Domain (ID) and -iitt--Distiiittionn (OOD) data across different datasets.
information (U2- = 0.163 in the Vision-Text pair, and U2 = 0.248 the Audio-Texx Text pair). Vision and Audio generally exhibit lower unique contributions (e.g., Uvision (V-T) = 0, U Unudio (A т) 0). However, we observe synergistic effects in certain modality pairs, most notably between Vision and Audio VVA, 0.012), indicating that despite their relatively weak individual contributions, these modalities can provide complementary information when effectively combined.
A.3.2. INTERACTION ESTIMATION ON DOMAIN SHIFTING
Previous experiments primarily involved datasets under stable domain conditions. A critical aspect, however, is to understand how these interactions behave under domain shifts, We investigate this under two primary settings: (1) label noise, where the ground-truth labels of multimodal samples are deliberately corrupted, and (2) a comparison of interactions In-Domain (ID) versus cittr---isttibutionn (OOD) scenarios.
Introducing label noise, which directly modifies the task definition, significantly alters the measured multimodal information dynamics, detailed in Table 8. This alteration even cause an inversion of the informational roles of different interaction components. For instance, interaction component (e.g., redundancy or unique information from modality) that was highly informative under clean labels might yield substantial negative information (i.c., become misleading) when labels are noisy. This occurs because the learned associations from clean data become counterproductive for the task defined by noisy labels, where original samples may now provide detrimental evidence.
When comparing In-Domain (ID) and Out-of-Diitributiion (OOD) scenarios (Table 9), OOD samples inherently provide less directly usable information. This is because models are typically not adequately trained on such data, leading to diminished capacity to establish robust mapping between the OOD inputs and the target task. Notably, we observe that synergistic information (S) tends to be more crucial for OOD performance. This suggests that when confronted with unfamiliar data, particularly when individual modalities alone are insufficient for the task, the model increasingly relies on the complementary strengths derived from combining different modalities. This underscores the heightened importance of synergy for better generalization, as the model must effectively integrate potentially weaker or less familiar signals from individual modalities to achieve the desired outcome.
A.3.3. IMPACT OF FUSION STAGE ON LEARNED INTERACTIONS
To investigate how the fusion strategy affects learned multimodal interactions, applied LSMI to a Hierarchical Multimodal Transformer architecture (Xu et al., 2023) on KS dataset. The specific model variant employed in our experiments features unimodal branches, each consisting of four Transformer layers. In such architectures, modalities are typically processed through these separate unimodal pathways, with each modality undergoing layers of dedicated processing before their representations are fused. In our experiments, we systematically varied the fusion point by adjusting I, the number of these unimodal layers preceding the first cross- modal interaction. A smaller I (e.g., = O, indicating fusion at the input level)
15

---

Efficient Quantification of Multimodal Interaction at Sample Level
Unimodal Layers
R
U1
U2
.5
Total
0
1.238
0.737
0.000
1.445
3.420
1
1.844
1.011
0.000
0.566
3.421
2
1.975
1.093
0.000
0.355
3.423
3
2.299
0.870
0.000
0.250
3,419
4
2.335
0.907
0.000
0.181
3,423
Table 10: LSMI-based interaction analysis in a Hierarchical Multimodal Transformer (Xu et al., 2023), varying the fusion stage. denotes the number of unimodal layers processed before cross-mos moosl fusion is introduced. | represents the earliest fusion (at input), and I represents the latest fusion (after unimodal layers).
Text: hold buffalo chicken club sandwich Tastefully Simple
Text: RECIPE bibimbap baltimore TV Dinner healthy recipes for RA by Jamie Stelter
Text: the Headlights: Spaghetti Carbonara
18
Labelssagghett.carbae carbtnara
Labet:club_ee Labetclubbsanddwich sandwich
Label:bibimbep
(a) 4.6, = 0, 0
(b) - 115, = 6.1, u. 0, (c) r= 4.2, 0.4, m; 0,
Figure 8: Case studies on the Food-10 101 dataset showing different interaction patterns between visual and textual modalities.
corresponds to earlier fusion, while a larger (e.g., 1 4, signifying fusion after all unimodal layers) represents later fusion.
The results, presented in Table 10, demonstrate clear trend in the nature of learned interactions. While the total information captured remains relatively consistent across different fusion stages, the composition of interaction patterns varies significantly. Early fusion strategies (i.e., smaller I) tend to foster greater synergy. For instance, when fusion occurs at the input (1 = 0). the Iearned synergy (S = 1.445) is more prominent than redundancy (R = 1 1.238). Conversely, as the fusion point is delayed to later stages (i.e., larger 1), the model increasingly learns redundant information. With fusion after 4 unimodal layers (1 = 4), redundancy (R = 2.335) significantly outweighs synergy (S = 0.181). These findings suggest that introducing cross-modal connections early the architecture encourages the model to identify and combine novel, complementary information from different modalities, thereby capturing synergy, In contrast, later fusion appears focus more integrating information that has already been extensively processed within each modality, leading higher focus on redundant and overlapping information.
A.3.4. CASE STUDY
To validate the effectiveness of our pointwise method, we evaluate interaction metrics at the sample level by selecting several representative samples and models. We conduct experiments on the Food-101 dataset and present a few illustrative cases in Figure 8. As shown in Figure distinct interaction patterns are observed: Figure (a), a strong redundancy evident, where the image and text consistently describe sandwich. In Figure (b), the image information is missing, and only the text provides meaningful information. In Figure 8 (c), partial redundancy is observed, with the text containing richer information than the image. These observations align with our interaction assessments, demonstrating the capability of our method to capture nuanced multimodal interactions.
16


Reviews:
[{"review_id": "dOHfWFpSwM", "reviewer": "Reviewer_wohR", "summary": "The paper presents LMSI, a light-weight framework for estimating redundant, unique and synergistic information in multimodal tasks through sample-level partial information decomposition. While the previous approaches require computationally expensive tasks like distribution estimation, LMSI is cheaper without compromising on the accuracy.  Further, LMSI is positive, monotonic and is grounded information theory making it attractive to use. The authors supplement LMSI with many experiments, and against real-world application of model distillation.\n\n#####update after the rebuttal######\n\nI thank the authors for engaging in the rebuttal and trying to answer my queries. However, I still have my reservations about the ImageBind and OOD experiments. The authors make some very bold and interesting claims about the role of synergistic and redundant interactions in the paper. These claims might be even correct but currently stand unsubstantiated.  Since the work leaves some critical ends loose, I maintain my initial score.\n\n##############################", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions. Here are our responses:\n\n**Q1: Model biases in dataset interactions**\n\nA1: We appreciate the reviewer's valuable point. Indeed, quantifying multimodal interactions requires models, making the estimates inherently model-dependent. Consistent with prior work yielding reliable outcomes [1], our method employs fully trained multimodal models. We focused on models demonstrating strong generalization, as we posit these better reflect the underlying data distribution and better modelling the interactions within it.\n\nFollowing the reviewer's suggestion, we constructed an ensemble integrating several of best-generalizing models to measure interactions. As detailed in [Table a](https://anonymous.4open.science/r/Submission2900-7FB8/Tab_a.png), the interaction estimates derived from this ensemble were highly consistent with those from individual top-performing models, exhibiting negligible differences.\n\n**Q2: Comparison against attention-based methods**\n\nA2: Thank you for this valuable suggestion. We validated LMSI on a Multimodal Transformer with hierarchical attention (from multi-stream to single-stream)[2], where we varied the layer $l$ at which cross-modal fusion was first introduced. A smaller $l$ indicates earlier fusion architecture.\nThe experiment results are shown in [Table b](https://anonymous.4open.science/r/Submission2900-7FB8/Tab_b.png).\nOur analysis revealed that interactions learned from later fusion stages were predominantly characterized by redundancy, whereas interactions learned from earlier fusion stages demonstrated a stronger tendency to capture synergistic relationships between modalities.\n\n**Q3: Understand the absolute scores of LMSI**\n\nA3: Thank you for this insightful question. LMSI components ($r$, $s$, $u_1$, $u_2$) measure information quantity (in nats/bits) from different multimodal interactions. We can utilize relative proportions (e.g., $\\hat{r} = \\frac{r}{i(x_1,x_2;y)}$) as a bounded, standardized metric. These relative scores ($\\hat{r}$, $\\hat{s}$, $\\hat{u}_1$, $\\hat{u}_2$) sum to 1 and provide a normalized interaction view. The optimal values of interaction varies by task - video tasks show high $\\hat{r}$, while sentiment analysis favors $\\hat{u}$, as shown in Table 2 in the main paper.\n\n\n**Q4: Comparison with human judgment**\n\nA4: Thank you for this point. Direct numerical comparison is challenging because human evaluations are typically sample-specific and inherently subjective, making it difficult to accurately represent the information quantity across the entire data distribution. Despite these differences, human judgments can provide valuable insights into the perceived relative importance or ranking of interactions within specific instances. Following precedents like [1], we investigated the alignment between LMSI and human perception by focusing on the dominant interaction types identified by both. As shown in Table 2, LMSI demonstrates reasonable consistency with human assessments in identifying top interactions across datasets.\n\n\n**Q5: LSMI Behavior with Noise**\n\nA5: Thank you for your question. LSMI quantifies task information $Y$ across modalities $(x_1, x_2)$, making it sensitive to information quality. Label noise reduces the multimodal mutual information $i(x_1, x_2; y)$, affecting LSMI's interaction measures. We added noise to image-text pairs [Table c](https://anonymous.4open.science/r/Submission2900-7FB8/Tab_c.png), observing decreased interactions, especially for information-rich samples. This confirms LSMI's sensitivity to noise.\n\n**Q6: Request for concrete tasks**\n\nA6: Thank you for this suggestion. We wish to clarify that our method, LMSI, has already been validated on several concrete tasks: sentiment recognition (MOSEI), action recognition (KS), and sarcasm detection (URfunny). \n\nTo further demonstrate LSMI's practical utility, we applied it to guide ImageBind fine-tuning on the KS dataset (Audio+Visual modality). Using LSMI, we quantified sample-wise audio-visual redundancy and partitioned the KS dataset into two subsets: high-redundancy ($R$) and low-redundancy ($U+S$). We then fine-tuned ImageBind separately on these subsets as well as the full dataset ('alldata'), comparing their classification performance. The results, presented in [Table d](https://anonymous.4open.science/r/Submission2900-7FB8/Tab_d.png), demonstrate that our data partitioning strategy based on LSMI analysis can enhance fine-tuning outcomes. While our VQA task experiments are still in progress due to time limitations, these results highlight the potential of LSMI for improving model fine-tuning.\n\n[1] Paul Pu Liang et al. \"Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework.\" NeurIPS, 2023.\n\n[2] Peng Xu et al. \"Multimodal Learning with Transformers: A Survey.\" TPAMI, 2023.", "claims_and_evidence": "Multimodal biases arise from (1) dataset only, (2) model only, and (3) both (referred to as categories below). \\\nFirst the authors test the validity of their method on simulated dataset (section 4.1) which falls in the 2nd category. Next, in section 4.2.2 (Dataset Interactions), the authors quote in L354-358 \"we select models with the lowest generalization risk, which are considered to better model the underlying distribution, and average their sample-level interactions to represent the dataset-level interactions.\"  I am not convinced by this. While the dataset might have biases, the model might not be necessarily relying on those biases (that is, on the same dataset, we can have both biased and unbiased models [1]). Therefore, section 4.2.2 is actually a dataset+model (category 3). However, the section is misleadingly titled as \"Dataset Interactions\", and the discussion on model interactions is absent in the paper. Perhaps a better method to achieve sample-level interactions would be to use LMSI on (1) an overfit model or (2) ensemble of models of different families, to ameliorate the effect of model's biases.\n\n\nReferences: \\\n[1] Rawal, Ishaan Singh, et al. \"Dissecting multimodality in VideoQA transformer models by impairing modality fusion.\" ICML 2024", "methods_and_evaluation_criteria": "Can this method be used for more concrete tasks to validate its correctness? For example, does LMSI corroborate with the findings like in [1] and [2] where the authors explicitly propose debiased datasets and models (that is, decrease the redundant information in the model and dataset?)\n\n\nReferences: \\\n[1] Buch, Shyamal, et al. \"Revisiting the\" video\" in video-language understanding.\" CVPR 2022. \\\n[2] Goyal, Yash, et al. \"Making the v in vqa matter: Elevating the role of image understanding in visual question answering.\" CVPR 2017.", "theoretical_claims": "Yes, LMSI is grounded in fundamental ideas of information theory", "experimental_designs_or_analyses": "The methods section of the paper lacks many details. The implementation details, model details, details of the finetuning experiment are missing.", "relation_to_broader_scientific_literature": "The work is quite important and can have a lot of different applications in dataset collection, filtering and interpretability tasks.", "essential_references_not_discussed": "1. QUAG [1] which was also used for \"multimodal interaction quantification at the sample level for real-world data\"\n2. Comparison against and discussion of PID [2] \n\nReferences: \\\n[1] Rawal, Ishaan Singh, et al. \"Dissecting multimodality in VideoQA transformer models by impairing modality fusion.\" ICML 2024 \\\n[2] Liang, Paul Pu, et al. \"Quantifying & modeling multimodal interactions: An information decomposition framework.\" NeurIPS 2023", "other_strengths_and_weaknesses": "**Strengths:** \n1. The paper tackles an important task of multimodal interaction quantification\n2. LMSI is light-weight and accurate \n3. LMSI is grounded in theoretical and well-founded ideas in information theory\n\n\n**Weaknesses:**  \n1. Overarching claims about dataset biases without considering model biases (see Claims And Evidence)\n2. Lack of confirmatory real-world experiments (see Methods And Evaluation Criteria)\n3. No comparison against attention-based methods (early, mid and late fusion strategies)\n\nI believe that the work is headed in the right direction however it could be misinterpreted because of lack of discussions and supporting experiments.", "other_comments_or_suggestions": "The supplementary section can be improved with more examples and details.", "questions_for_authors": "1. What is the correlation between human judgement and LMSI-predicted interations in Table 2? \n2. How does LMSI account for noise/missing information? Labelling process is not perfect, especially for subjective or (semi)-automatic labelling tasks? \n3. How to make sense of absolute LMSI score? Can it be consolidated into a single number? Ideally, a measure that has well-behaved bounds. Because currently, I am not sure if u=0.1 is good or bad?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EpP0HQwRjm", "reviewer": "Reviewer_G7u5", "summary": "The paper tackles an important question of capturing interactions between modalities and proposes a lightweight\nentropy-based multimodal interaction estimation approach for efficient and precise sample-wise interaction measurement across various continuous distributions. The authors  demonstrate the efficacy of their approach using circuit logic  recovery and experimentation with real datasets.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their valuable suggestions. Here are our responses to the questions raised about our paper.\n\n**Q1: The contribution of this work compared to the work from Morency's Lab.**\n\nA1: Thank you for this question. The core contribution of our work, compared to foundational studies from Morency's Lab [1], is the development of efficient sample-wise quantification of multimodal interactions.\n\nWe build upon the same established interaction concepts (redundancy, uniqueness, synergy) common to both [1] and other literature [2,3]. However, the key distinction lies in mathematical modeling of interactions:\nMorency's work (PID) [1] typically provides dataset-level aggregate measures of interactions.\nOur method enables quantifying these interactions for **each individual sample**.\n\nThis efficient sample-level resolution is our main technical contribution and is vital for applications requiring instance-specific insights, such as targeted model integration or knowledge distillation strategies explored in our experiments (Section 4.3). Aggregate measures (PID) cannot readily support these tasks.\n\nNotably, the value of such pointwise measures was recognized as an important next step by Paul and Morency 2023: \"*A natural extension is to design pointwise measures... which could help for more fine-grained dataset and model quantification...*\" Our work provides a concrete method to realize this, demonstrating its contribution and relevance.\n\n**Q2: Application of LSMI to Downstream Tasks**\n\nThank you for the suggestion. Our LSMI method enables the estimation of sample-level interactions, which can indeed benefit downstream tasks beyond the model ensemble and distillation examples discussed in the main paper.\n\nTo demonstrate this, we applied LSMI to guide the fine-tuning process for a specific task. We used the ImageBind model, which focuses on modality alignment, and fine-tuned it on the Kinetics-Sounds (KS) dataset (Audio+Visual). Specifically, we employed LSMI to quantify the degree of redundancy between the audio and visual modalities for each sample. Based on this metric, we partitioned the KS dataset into two subsets: one containing samples with relatively high redundancy ($R$) and another with relatively low redundancy ($U+S$, comprising samples dominated by unique or synergistic information).\n\nWe then fine-tuned the ImageBind model separately on these two subsets ($R$ and $U+S$) and compared the results against fine-tuning on the entire dataset ('alldata'). The performance, evaluated by classification accuracy on KS dataset, is shown below:\n\n| Fine-tuning Set | V+A Performance | V Performance | A Performance |\n|-----------------|-----------------|---------------|---------------|\n| alldata         | 0.8539          | 0.8183        | 0.7270        |\n| U+S             | 0.8496          | 0.8045        | 0.7289        |\n| R               | 0.8772          | 0.8241        | 0.7257        |\n\nAs observed, fine-tuning specifically on the high-redundancy subset ($R$) identified by LSMI yielded the best performance for the combined modalities (V+A) and the visual modality (V). This suggests that leveraging LSMI to identify and focus on highly redundant samples can be an effective strategy for enhancing model fine-tuning, potentially by reinforcing learning when sufficient information is present in individual modalities.\n\n\n**Q3:  Result state in Table 1**\n\nA3: We appreciate you pointing it out. Table 1 focuses on evaluating the precision of different interaction estimation methods. We utilize synthetic datasets based on logic relations with additive noise, which allows for comparison against known ground truth (GT) interaction values. This discrete logic carries explicit interaction information and has been widely used in previous studies [1,4]. The objective is to assess **how closely each estimator's output matches the GT**. As the results indicate, our method achieves interaction estimates that align more closely with the GT compared to the baseline estimators evaluated. This finding underscores the robustness of our approach, demonstrating its effectiveness in capturing underlying data interactions despite the presence of noise. We have enhanced the description surrounding Table 1 in the revised manuscript to better emphasize this point.\n\n[1] Paul Pu Liang et al. \"Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework.\" NeurIPS, 2023.\n\n[2] Benoit Dufumier et al. \"What to align in multimodal contrastive learning?\" ICLR, 2025.\n\n[3] Paul Pu Liang et al. \"Multimodal learning without labeled multimodal data: Guarantees and applications.\" ICLR, 2024.\n\n[4] Nils Bertschinger et al. \"Quantifying Unique Information.\" Entropy, 2014.", "claims_and_evidence": "The authors claim that the definition of redundancy at the event (sample) level is clearer and more straightforward than that of uniqueness or synergy. Hence they suggest to obtain redundant information by partitioning mutual information into components relative to the target, allowing for the measurement suitable for redundancy in each component. They name their method Lightweight Sample wise Multimodal Interaction (LSMI).", "methods_and_evaluation_criteria": "The authors evaluate on synthetic and real world datasets (KS dataset, UCF dataset, CREMA-D dataset, KS dataset) and compare to several other methods (Weighted ensemble vs. LSMI ensemble  and PID method). However the conclusions are relatively not impresive:\n\"The comparison results are shown in Table 2. We observe that our LSMI estimator is largely consistent with the PID-Batch\nmethod. Furthermore, in the KS, Food-101, and MOSEI datasets, our method aligns with the top two highest interactions in terms of redundancy and uniqueness, respectively.\"", "theoretical_claims": "I do not think the paper has any theoretical claims", "experimental_designs_or_analyses": "I feel that paper will greatly benefit from a more precise discussion are its contribution. It might help to identify specific cases  where their \"sample\" level redundancy estimation works better and motivate the readers with those cases before providing the numbers. Also it will help to state in Table 1 if lower or higher numbers are better.", "relation_to_broader_scientific_literature": "I think the topic of capturing multimodal interactions is extremely important for a broader community that aims to develop MLLMs and align modalities. Understanding interaction between training samples' modalities might improve the training purpose. Unfortunately this paper does not  go beyond estimation of interactions.", "essential_references_not_discussed": "I'm only familiar with L.P. Morency works on capturing  multimodal interactions, and those were cited.", "other_strengths_and_weaknesses": "In overall, it was really hard to follow the  paper and its specific contributions. A year ago I was very impressed by the work from Morency's lab on  capturing the multimodal interactions, but this paper does not contribute enough on top of the original work (PID algorithm they compare to).", "other_comments_or_suggestions": "Please rewrite the paper, clearly stating the contribution and motivate the sample-level estimation of interactions. It will be helpful to see how the findings can be used in downstream tasks like ImageBind or even MLLM training.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "26xW0z5YZq", "reviewer": "Reviewer_sN2h", "summary": "The paper proposes a novel framework to resolve conflicts between pointwise mutual information (PMI) and Partial Information Decomposition (PID) axioms in multi-modal learning. Traditional PID axioms (non-negativity, monotonicity) are defined for distribution-level redundancy but conflict with sample-level PMI, which can be negative. The authors address this by decomposing PMI into two non-negative components.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for their valuable suggestions. Here are our responses to the questions.\n\n**Q1: Visualization of $r^+$ and $r^-$**\n   \nA1: Thank you for this valuable suggestion. To demonstrate the function of the $r^+$ and $r^-$ components derived from the total redundancy $r$, we conducted experiments on a Gaussian Mixture Model (GMM) dataset. We visualize the distributions of $r$, $r^+$, and $r^-$ in the provided [figure a](https://anonymous.4open.science/r/Submission2900-7FB8/Fig_a.png).\n\nAs indicated by the distributions, $r^+$ appears to capture the core information consistently present across modalities within a sample, reflecting the minimum entropy along the unimodal marginal distributions (i.e., the most reliably available information from at least one modality). It represents the inherent information content shared by the modalities. In contrast, $r^-$ quantifies the portion of redundancy where individual modalities might struggle or provide conflicting cues regarding the task. It reflects the ambiguity or potential for error introduced if relying on the worse modality.\n\n**Q2: Scalability to three modality**\n\nA2: Thank you for this thoughtful suggestion. Extending interaction analysis on more than two modalities inherently involves quantifying complex interactions, such as the mutual information shared among three modalities and the target task. However, a key challenge arises because the clear theoretical definitions used to decompose two-way interactions into distinct synergistic, redundant, and unique contributions do not readily or uniquely translate to these higher-order cases [1,2].\n\nGiven this theoretical limitation, we adopt the widely-used pairwise analysis approach (Appendix C.4 of [3]). This method systematically examines interactions between each pair of modalities. We implemented this approach on the UCF-101 dataset, analyzing the interactions among its three modalities: Vision, frame difference (Diff), and optical flow (OF). The quantitative results of these pairwise interactions are presented in the following Table.\n\n| modalityPair | $R$ | $U_1$ | $U_2$ | $S$ |\n|---|:---:|:---:|:---:|:---:|\n| Visual_OF | 2.111  | 2.483  | 0.000  | 0.000  |\n| Visual_DIFF | 3.474  | 1.121  | 0.000  | 0.000  |\n| OF_DIFF | 1.998  | 0.003  | 1.476  | 0.239  |\n\nThe pairwise analysis revealed that vision is the strongest modality, as its unique interactions consistently surpass those of other modalities. Furthermore, we observed a primarily redundant relationship between vision and frame difference, while frame difference and optical flow demonstrated notable synergy. This observed synergy likely arises because frame difference and optical flow, when considered individually, carry less task-relevant information and thus benefit significantly from combining their complementary information to achieve better task performance.\n\n \n**Q3. physical meanings of $i(x_1;x_2; y)<0$.**\n\nA3: Thank you for raising this important question. To build intuition, consider the simpler case of negative unimodal information: \n$i(x_1;y) = \\log\\frac{p(y|x_1)}{p(y)}$.\nNegativity implies $p(y|x_1) < p(y)$, meaning that observing modality $x_1$ decreases the likelihood of the target $y$ compared to its prior probability. This suggests $x_1$ provides information that contradicts $y$, a situation potentially arising from mislabeled samples.\n\nFor the interaction term, defined as:\n$i(x_1;x_2;y) = i(x_1;y) + i(x_2;y) - i(x_1,x_2;y)$\na negative value ($i(x_1;x_2;y) < 0$) signifies that the joint information $i(x_1,x_2;y)$ is greater than the sum of the individual information components ($i(x_1;y) + i(x_2;y)$). This phenomenon represents synergy: the modalities $x_1$ and $x_2$, when combined, provide more information about $y$ (or potentially, more \"mis-information\" if the individual terms are strongly negative) than would be expected from merely summing their individual contributions.\n\nRegarding the redundancy term $r(x_1;x_2;y)$ discussed in our paper (which forms part of the unimodal information, e.g., $i(x_1;y) = r + u_1$), this can also be negative. This is particularly likely if both individual modalities carry contradictory information about the label (i.e., $i(x_1;y) < 0$ and $i(x_2;y) < 0$). In such cases, the redundant information component may also reflect this contradictory concerning the target $y$.\n\n[1] Tobias Mages et al. \"Decomposing and Tracing Mutual Information by Quantifying Reachable Decision Regions.\" Entropy, 2024.\n\n[2] Paul Williams et al. \"Nonnegative Decomposition of Multivariate Information.\" arXiv, 2010.\n\n[3] Paul Pu Liang et al. \"Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework.\" NeurIPS, 2023.", "claims_and_evidence": "The paper’s ​theoretical claims (axiom compliance, decomposition structure) are well-supported by proofs. And the experiments", "methods_and_evaluation_criteria": "Yes. Its ​practical and interpretability claims are well-supported by the experiments on synthetic data, distillation experiments, and dynamic fusion experiments.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes. For example, the experiments on distillation and dynamic fusion task demonstrate the effect of LSMI.", "relation_to_broader_scientific_literature": "Existing methods focus on the distribution-level interaction quantification, this work extends them to the sample-level interaction quantification.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength:\n\n​Innovative PMI decomposition strategy: Decomposing PMI into r+ and r− to resolve the PID axiom conflict is novel in the field of information theory and PID. Existing work usually avoids the problem of sample-level negative PMI, while this paper directly reconciles the contradiction between theory and practice through decomposition.\n\n​Separation of sample and distribution-level redundancy: For the first time, the mathematical treatment of sample-level and distribution-level redundancy is clearly distinguished, which expands the scope of the application of PID.\n\nWeaknesses:\n\n​1. Visualization and examples: There is a lack of diagrams (such as the distribution of r+/r− in synthetic data) or examples (such as the redundant decomposition results of specific samples) to assist understanding.\n\n2. How is the method scalable to three or more modes? Is there any experimental analysis?\n\n3. i(x; y) = i+(x; y) − i−(x; y) in the derivation process may also be less than 0. So will there be samples with i(x1;x2; y)<0 in the actual application process? What is the physical meaning behind it?", "other_comments_or_suggestions": "see the weaness", "questions_for_authors": "see the weaness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NsHmHJMyuL", "reviewer": "Reviewer_r5Mr", "summary": "In this paper, the authors presented a novel method for efficient quantification of multimodal interaction for a single multimodal sample. Pointwise mutual information may be negative, so monotonicity over the existing redundancy framework does not hold for single samples; therefore, the authors proposed an alternative method for determining redundancy: dividing the information into two components, where each component adheres to monotaniticity, and redundancy is calculated on each component before summed together. The authors further proposed a lightweight method to compute each component for a single sample. The authors evaluated their new algorithm on both synthetic dataset and real datasets, demonstrating that their method accurately estimates redundancy, uniqueness and synergy single samples. The authors further showed that their method is time-efficient, and can be used to improve multimodal ensemble and distillation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive feedback and constructive comments on our work. We have carefully addressed the citation issue mentioned in the review, and examine other reference and formualtions carefully. Thank you again for your time and effort.", "claims_and_evidence": "The claims in the submission are supported clearly with empirical evidence.", "methods_and_evaluation_criteria": "The proposed methods makes sense.", "theoretical_claims": "Checked the correctness of the formulation of the new method (estimating the redundancy by dividing the information into two parts). Seems correct to me.", "experimental_designs_or_analyses": "I have verified the soundness/validity of the experiments, including the evaluation over synthetic datasets, real-world datasets, time efficiency experiments, and application experiments (showing that LSMI results in better distillation and ensemble performance). The experiment design are sound.", "relation_to_broader_scientific_literature": "Previous works in multimodal machine learning demonstrated the importance of quantifying and extracting unique information from each modality as well as cross-modal interactions. While previous works have successfully quantified multimodal interactions over a data distribution and applied them to improve model performance, this work is the first to propose a method that can quantify multimodal interaction on a single data sample. Being able to quantify single-sample multimodal interaction would significantly improve fine-grained understanding of multimodal data, and this paper has also demonstrated that the proposed method can be used to improve multimodal distillation and ensemble.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is well written and well presented. The explanation of the key idea and methodology is clear.", "other_comments_or_suggestions": "Minor problem: it seems like the citation for Liang 2023a and Liang 2023b are referring to the same paper in the bibliography.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: __Summary__

This submission presents a framework for capturing (and estimating) three types of interactions between two data modalities for a learning task: redundancy, uniqueness, and synergy, all under an information-theoretic framework. Previous work on multi-modal interactions relies on density estimation, which is both computationally demanding and becomes more and more challenging and inaccurate as dimensionality increases. The novelty in the framework is twofold:  

1. It estimates the three types of interactions at the sample level through partial information decomposition, and as a byproduct, the framework is computationally efficient as opposed to previous methods.

2. The framework resolves conflicts between pointwise mutual information (PMI) and partial information decomposition (PID) axioms in multi-modal learning (non-negativity, monotonicity), which are usually defined at the distribution level.  

The authors demonstrate the efficacy of the proposed framework using circuit logic recovery (synthetic data) and various experiments with real datasets. In addition, they presented additional experimental results based on reviewers' requests.

All reviewers agree that the paper proposes an interesting and important step for estimating multi-modal interactions, which can, in turn, have different applications in multi-modal ensemble learning, distillation, dataset collection, filtering, and interpretability.

__Regarding reviewers' concerns, authors' rebuttal, and reviewers' final response to authors' rebuttal.__

For Rev1 (wohR): The authors have addressed all concerns raised by the reviewer in two rounds of questions. Unfortunately, the reviewer did not respond to the last reply from the authors, although I asked the reviewer to provide feedback to the authors. The last concerns/questions raised by the reviewer are secondary or do not really affect the paper's core ideas and contributions.

For Rev2 (G7u5): The authors have addressed all concerns raised by the reviewer in two rounds. The specific concern raised by the reviewer regarding the ImageBind experiment was also addressed in detail in the second author's rebuttal. The authors also explained how the nature of contrastive learning takes advantage of the redundancy in training pairs, which explains the results in the Table provided by the authors. The reviewer did not give feedback to the authors.

Rev3 (sN2h) and Rev4 (r5Mr) have no issues with the submission, and the authors addressed the questions raised by Rev3.
Decision: Accept (poster)