Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
Authors: Paul Pu Liang
Yun Cheng
Xiang Fan
Chun Kai Ling
Suzanne Nie
Richard J. Chen
Zihao Deng
Nicholas Allen
Randy Auerbach
Faisal Mahmood
Ruslan Salakhutdinov
Louis-Philippe Morency
Venue: NeurIPS 2023 poster (2023)
Keywords: multimodal learning
feature interactions
partial information decomposition
information theory
quantification
model selection
TL;DR: We demonstrate the usefulness of information theoretic concepts of redundancy, uniqueness, and synergy in quantifying multimodal datasets, models, and model selection. Our proposed estimators allow us to scale these to large, continuous domains.

Abstract: The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities. Despite these empirical advances, there remain fundamental research questions: How can we quantify the interactions that are necessary to solve a multimodal task? Subsequently, what are the most suitable multimodal models to capture these interactions? To answer these questions, we propose an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. We term these three measures as the PID statistics of a multimodal distribution (or PID for short), and introduce two new estimators for these PID statistics that scale to high-dimensional distributions. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks where PID estimations are compared with human annotations. Finally, we demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies engaging with domain experts in pathology, mood prediction, and robotic perception where our framework helps to recommend strong multimodal models for each application.

Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
Paul Pu Liang', Yun Cheng Xiang Fan Chun Kai Ling 1.8. Suzanne Nie Richard J. Chen 4.5, Zihao Deng", Nicholas Allen", Randy Auerbach', Faisal Mahmood 4.3,
Ruslan Salakhutdinov', Louis-Philippe Morency
CCU, Princeton University, SUW, 'Harvard Medical School, 'Brigham and Women's Hospital, University of Pennsylvania, Uniiversity of Oregon, CColubbia University pliangêcs. cmu. edu, yc6206lcs princeton.
Abstract
The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities. Despite these empirical advances, there remain fundamental research questions: How can we quantify the interactions that are necessary to solve multimodal task? Subsequently, whut are the most suitable multimodal models to capture these interactions? ? To answer these questions, we propose an cermattion-thoorche approach to quantiff the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. We term these three measures as the PID statistics of multimodal distribution (or PID for short), and introduce two new estimators for these PID statistics that scale to high-dimensions distributions. To validate PID estimation, both synthetic datasets where the PID is known and on large-scale multimodal benchmarks where PID estimations are compared with human annotations. Finally, we demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies engaging with domain experts in pathology, mood prediction, and robotic perception where our framework helps to recommend strong multimodal models for each application.
conduct extensive experiments on
Introduction
core challenge in machine learning lies in capturing the interactions between multiple input modalitics. Learning different types of multimodal interactions is often quoted motivation for many successful multimodal modeling paradigms, such as contrastive learning to capture redun- dancy (54 82 modality-speciff representations retain unique information [TOO] as well tensors and multiplicative interactions to learn higher-order interactions (52, 58 115). However, several fundamental research questions remain: How can quantify the interactions that necessary solve a multimodal task? Subsequently, what are the most suitable multimodal models to capture these interactions? This paper aims to formalize these questions by proposing an approach to quantify the nature (i.e., which type) and degree (i.e., the amount) of modality interactions, a fundamental principle underpinning our understanding of multimodal datasets and models 62]
By bringing together two previously disjoint research fields of Partial Information Decomposition (PID) in information theory [3] 41 [08] and multimodal machine learning 9 62), we provide precise definitions categorizing interactions into redundancy, uniqueness, and synergy. Redundancy quantifies information shared berween modalities, uniqueness quantifies the information present in previously
only one of the modalities, and synergy quantifies the emergence of new information present in either modality. A key aspect of these four measures is that they not only quantify interactions between modalities, but also how they relate downstream task. Figure shows a depiction of these four measures, which we refer to as PID statistics. Leveraging insights from
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

neural representation learning, we propose two new estimators for PID statistics that can scale to nigh-dimensional multimodal datasets and models. The first estimator is exact, based on convex optimization, and is able to scale to features with discrete support, while the second estimator is an approximation based on sampling, which enables us to handle features with large discrete or even continuous supports. We validate our estimation of PID in 2 ways: (1) on synthetic datasets where PID statistics are known due to exact computation and from the nature of data generation, and (2) on real-world data where PID is compared with human annotation. Finally, we demonstrate that these estimated PID statistics can help in multimodal applications involving:
1. Dataset quantification:
We apply PID to quantify large-scale multimodal datasets, showing that these estimates match common intuition for interpretable modalities (e.g.. language, vision, and audio) and yield new insights in other domains (e.g. healthcare, HCI, and robotics).
2. Model quantification: Across a suite of models, we apply PID to interpret model predictions and find consistent patterns of interactions that different models capture.
3. Model selection: Given our findings from dataset and model quantification, a new research question naturally arises: given a new multimodal task, can we quantify its PID values to infer (a priori) what type of models are most suitable? Our experiments show success in model selection for both existing benchmarks and completely new case studies engaging with domain experts in computational pathology, mood prediction, and robotics to select the best multimodal model.
Finally, we make public a suite of trained models across 10 model families and 30 datasets to acceler- ate future analysis of multimodal interactions at htttps://github.. http://hwwwb.coo//llaagg/////
2 Background and Related Work
Let X; and )/ be sample spaces for features and labels. Define A to be the set of joint distributions over X2, We are concerned with features X1,X; (with support X.) and labels Y (with support }) drawn from some distribution E A. We denote the probability mass (or density) function by p(r1,, 22, ,2,y), y), where omitted parameters imply marginalization. Key to our work defining estimators that given p samples {(11, 22, Xg x X₂ x } thereof (i.e., dataset or model predictions), returns estimates for the amount of redundant, unique, and synergistic interactions.
2.1 Partial Information Decomposition
Information theory formalizes the amount of information that one variable provides about another (86). However, its extension to 3 vari- ables is an open question [34, 70, [33 [06]. In particular, the natural three- way mutual information I [(x;; (x; Y) = X2)
Clussical Information Theory
Partial Information Decomposition I(X;; ((X2)
Xy:
IXx;; ((X,)
X
X2
S
Uy U
X2 170 93 can be both positive and negative, which makes it difficult to inter- pret. In response, Partial information decom- position (PID) [108] generalizes information theory to multiple variables by decomposing ([XX,XX; Y). the total information 2 vari-
U. - XX
Ug U IX I(X:Y)
Y
I(Xg:)
Figure 1: PID decomposes I(X,, X2; Y) into redun- dancy R between X1 and X2. uniqueness U, X1 and U2 in X2, and synergy $ in both X, and X2.
ables X1, X2 provide about a task Y into quantities (see Figure I] redundancy R between X,, and X2, uniqueness U1 in X2 and U, in X2, and synergy S that only emerges when both X1 and X₂ are present. We adopt the PID definition proposed by Bertschinger et al. [3]:
R max eEA,
(2;
U min LX X|2), EEAp
U2= = min (X2;; Y|X)), qEAp
(2)
S = I2(X1 X2; Y)
min 1 Y2: gEAp
(3)
where A = = : q(.,,))) v) = p(z,, y) Vy E V. E Xe, E [2]} and the notation I,(-) and (.) disambiguates mutual information under p and q respectively. The key lies in optimizing q E Ap to satisfy the marginals q(,,, y) = p(x,, y), but relaxing the coupling between Ji and x2: q(1), x2) need not be equal to p(z1, x2)) The intuition behind this is that one should be able to infer redundancy and uniqueness given only access to p(# !/ and p( y). and therefore they should only depend on q E Ap. Synergy is the only term that should depend on the coupling p(11, and this is reflected
2

---

in B3) depending on the full distribution. This definition enjoys several useful properties in line with intuition, as we will see in comparison with related frameworks for interactions below [3]
2.2 Related Frameworks for Feature Interactions
Information-theorete definitions: Perhaps the first measure of redundancy in machine learning is co-training [14 8 23 where 2 variables are redundant if they are conditionally independent given the task: [[x1 X2 Y) = 0. As a result, redundancy can be measured by [[X1; X2; Y). The same definition of redundancy is used in multi-view learning [98] 94 [8]] which further define [Xx;: (\)))
and 2(x2; x .) as unique information in X, X2. However, TXx1; X2; Y) can be both positive and
negative [5 PID resolves this by separating R and such that R - S = I(X;; X2; Y). identifying
that prior measures confound redundancy and synergy. This crucially provides an explanation for the distinction between mediation, where one feature conveys the information already in another (i.e., R R> S), versus moderation, where one feature affects the relationship of other features (i.e.,
R) 10 36). Furthermore, if I(X1; X2; Y) = then existing frameworks are unable to distinguish between positive R and S canceling each other out.
Statistical measures: Other approaches have studied interaction measures via statistical measures, such as redundancy via distance between prediction logits using either feature [69]. statistical distribution tests on input features [4 7 or via human annotations [85]. However, it is unclear how to extend these definitions to uniqueness and synergy while remaining on the same standardized scale like PID provides. Also of interest are notions of redundant and synergistic interactions in
human and animal communication 778 79 30 85]. which we aim to formalize.
Model-based methods: Prior research has formalized definitions of non-additive interactions 32 to quantify their presence [88 [01 102 46 in trained models, or used Shapley values on trained features to measure interactions (49] Parallel research has also focused qualitative visualizations of real-
world multimodal datasets and models, such as DIME 68 ennsns [105] and MultiViz [6]]
3 Scalable Estimators for PID
PID as a framework for multimodality: Our core insight is that PID provides a formal framework understand both the nature and degree of interactions involved when two features X and X2 are used for task Y. The nature of interactions is afforded by a precise decomposition into redundant, unique, and synergistic interactions, and the degree of interactions is afforded by a standardized unit of measure (bits). However, computing PID is a considerable challenge, since it involves optimization over Ap and estimating information-theee theorretic measures. Up to now, analytic approximations of these quantities were only possible for discrete and small support [3 41 109) or continuous but low-dimensional variables [76] 81 110 Leveraging ideas in representation learning, Sections
and 3.2 are our first technical contributions enabling scalable estimation of PID for high-dimensional distributions. The first, CVX, is exact, based on convex optimization, and is able to scale to problems where | and [}]] are around 100. The second, BATCH, is an approximation based on sampling, which enables us to handle large or even continuous supports for X; and Y. Applying these estimators in Section 4 we show that PID provides path towards understanding the nature of interactions in datasets and those learned by different models, and principled approaches for model selection.
3.1
CVX: Dataset-level Optimization
Our first estimator, CVX, directly compute PID from its definitions using convex programming. Crucially, Bertschinger et al. [3 show that the solution to the max-entropy optimization problem:
q" arg max HY (Y| X2) equivalently solves - 3). When X; and ) are small and discrete,
set of tensors Q of shape |&]] x [22 x |}] with each entry representing Q ji. = p(x1 = i, X2 = j, Y = k). The problem then boils down optimizing over valid tensors Q Ap that match the marginals p(x,, v).
represent all valid distributions g(=1, 82, y)
Given a tensor Q representing g, our objective is the concave function (Y,, While Bertschinger et al. [13] report that direct optimization is numerically difficult as routines such as Mathematica's 's FINDMINIMUM do not exploit convexity, we overcome this by rewriting conditional entropy as a KL-divergence
(38) HY|Y X2) log Y KT ()), where q is auxiliary product density of 4(11, [y] enforced using linear constraints: [(21, 22, v) q(01, x22//// Finally, optimizing over Q E Ap that match the marginals can also be enforeed through linear constraints: the 3D-tensor Q summed over the second dimension gives g(x1, y) and summed over
3

---

Unnurmalized joint distribution
Sinkhurn'ss ' algorithm
Training objective
PID values
Kalpa)
org org Nx:1)
Tock
Rm Xg:1
Video
4104 -4
whew Xx)
03 (NGA))))
15 (N
22
S 6(8, (N
Forz)
A eap Core alx (Naco)
SinnnonoypAA that ( A
Sensors
Figure 2: We propose BATCHI a scalable estimator for PID over high-dimensional continuous distributions. BATCII parameterizes ÿ using matrix A learned by neural networks such that mutual information objectives over ! can be optimized via gradient-based approaches over minibatches. Marginal constraints E Ap are enforced through variant of the Sinkhorn- Sinkhom-Knoop algorithm on A.
the first dimension gives (12, y). yielding the final optimization problem:
arg mag KL(QI s.t. Q(21, x2, v) Qa11, 22)/
(4)
Q.Q
-0= v),
2)
Q-1
(5)
21-22-2
The KL-divergence objective is recognized as convex, allowing the use of conic solvers such as SCS 75 ECOS (27]. and MOSEK (6). Plugging q' into ( yields the desired PID.
Pre-processing via feature binning: In practice, X1 and X₂ often take continuous rather than discrete values. We work around this by histogramming each X₁, thereby estimating the continuous joint density by discrete distributions with finite support. We provide more details in Appendix B.1
3.2 BATCH: Batch-level Amortization
We now present BATCH, our next estimator that is suitable for large datasets where Xr is high- dimensional and continuous (3)) remains finite). To estimate PID given a sampled dataset D
{(x)), xx2 (1) y(ر)} of size n, we propose end-to-end model parameterizing marginal-matching joint distributions in Ap and a training objective whose solution returns approximate PID values.
Simplified algorithm sketch: Our goal, loosely speaking, is to optimize E Ap for objective ! through an upproximation using neural networks instead of exact optimization. We show an overview in Figure 2 To explain our approach, we first describe (1) how parameterize q using neural approaches, (2) how we ensure the marginal
networks such that it can be learned via gradient- constraints q E A, through variant of the Sinkhorn Knopp algorithm, and finally (3) how to scale this up over small subsampled batches from large multimodal datasets.
Parameterization using neural networks: The space of joint distributions A is often too large to explicitly specify. To tackle this, we implicitly parameterize each distribution @ E A using a neural network fo that takes in batches of modalities X₃ E , E X2 and the label Y E ) before returning a matrix A E mg=xnx representing an (unnormalized) joint distribution q. i.e., we want A[i]ii]b] = q((
x [i], X2 v) for each y E y. In practice, fo is implemented via a pair of encoders fo(1) and fe(2) that learn modality representations, before an outer product to learn joint relationships A, exp(Se(1) (
x2.y) for each y. yielding the desired n [}] joint distribution. As result, optimizing over ( can be performed via optimizing over parameters Ø
Respecting the marginal constraints: How do make sure the q's learned by the network satisfies the marginal constraints (i.e., ÿ E A,)? We use an unrolled version of Sinkhorn's 's algorithm [24] which projects A onto Ap by iteratively normalizing A's rows and columns to sum to 1 and rescaling to satisfy the marginals p(æ,, y). However, p(r,, y) is not easy to estimate for high-dimensional continuous x, 's. In response, we first expand p(z,, y) into p(y]z:) and p(x:) using Bayes' rule. Since A was constructed by samples x, from the dataset, the rows and columns of A are already distributed according p and p(z2) respectively. This means that it suffices to approximate p(y/ with unimodal classifiers p(y(æ;) parameterized by neural networks and trained separately, before using Sinkhorn's algorithm to normalize each row to p(yla1)
(a1) and each column to p(y) x:2).
Objective: We choose the objective (X11; Xx; Y); which equivalently solves the optimization problems in the other PID terms 13] Given matrix representing 7( x2, 3y), the objective can be computed in closed form through appropriate summation across dimensions in A to obtain ((x;). ((11, .22). ((xs1y). and 7( x2lly) and plugging into (x1;; X2; Y) [x(,; X2) [g(X; YY) We maximize 1g( (X1; X2; Y) by updating parameters via gradient- based methods. Overall, each

---

5
7.5 1e-1
7.5 1e-1
7.5 1e-1
7.5 1e-1
5.0
5.0
5.0
5.0
0
2.5
2.5
2.5
2.5
4u
-5
0
0.0
0
1
0.0
0
1
0
0.0
0
1
0.0
0
1
(a)
(b)
(C)
(d)
(e)
(1)
Figure 3: Left to right: (a) Contour plots of the GMM's density for ||l|| 2.0. Red line denotes the optimal linear classifier. (b) PID (Cartesian) computed for varying 2μ with respect to the æ axis. (c)
PID (Polar) for varying 2u, with U1 and U/ corresponding to unique information from (r, 0). Plots
(d)-(f) are similar to (a)-(c), but repeated for ||l|| 1.0. Legend: (R). (U1), (U2), (S).
(Sum). Observe how PID changes with the change of variable from Cartesian (b and e) to Polar
(c and f), as well as how a change in ||/||2 can lead a disproportionate change across PID (b vs e).
gradient step involves computing q SINKHORN (A), and updating & to maximize (1) under q. Since Sinkhorn's 's algorithm is differentiable, gradients can be backpropagated end-to-end.
Approximation with small subsampled batches:
Finally, to scale this up to large multimodal datasets where the full ạ may be too large to store, we approximate q with small subsampled batches: for each gradient iteration t, the network fo now takes in a batch of m < datapoints sampled from DD and returns A I m= for the subsampled points. We perform Sinkhorn's 'ss algorithm A and a gradient step on & as above, as if Dr was the full dataset (i.e., mini- bbtch gradient descent). While it is challenging obtain full-batch gradients since computing the full A is intractable, we found our approach to work in practice for large m. Our approach can also be informally viewed as performing amortized optimization by using Φ implicitly share information about the full batch using subsampled batches. Upon convergence of Φ, we extract PID by plugging i into (-
Implementation details such as the network architecture of f. approximation of objective via sampling from q. and estimation of I,([X,, Y) from learned are in Appendix B.3
4
Evaluation and Applications of PID in Multimodal Learning
We design experiments to (1) understand PID on synthetic data, (2) quantify real-world multimodal benchmarks, (3) understand the interactions captured by multimodal models, (4) perform model selection across different model families, and (5) applications on novel real-world tasks.
4.1 Validating PID Estimates on Synthetic Data
Our first goal is to evaluate the accuracy of our proposed estimators with respect to the ground truth (if it can be computed) human judgment (for cases where the ground truth cannot be readily obtained). We start with a suite of datasets spanning both synthetic and real-world distributions.
Synthetic bitwise features: We sam- ple from a binary bitwise distribution:
Table 1: Results on estimating PID on synthetic bitwise datasets. Both estimators exactly recover the correct PID values as reported in Bertschinger et al. 13).
21,22 {0, 1),8 T1 V Ø2, = Ø1 + ø2.. Each bitwise operator's PID
I1 A 22 y
Task
OR
AND
XOR
be solved exactly when the x;'s and labels are discrete and low-dimensional [3). Compared to the ground truth in Bertschinger et al. [33, both our estimators exactly recover the correct PID values (Table T
PID
R
U1
U2
S
R
U1
U2
$
R
U1
U2
S
Exact
0.31
0
0
0.5
0.31
0
0.5
0
0
0
CVX
0.31
0
0.5
0.31
0
0.5
0
BATCH
0.31
0.5
0.31
0
0.5
0
0
Gaussian Mixture Models (GMM): Consider a GMM, where X1, X2 E R and the label Y E {-1,+1}, comprising two equally weighted standard multivariate Gaussians centered at =y. where E R2, i.c., Y Bernoulli ( /2),. (X1, X2)
y ~ N(y I). PID estimated by sampling 1e6 points, histogramming them into 50 bins spanning [-5, +5] to give p. and then applying the CVX estimator. We term this PID MDD-Caressian. We also compute PID- which are PID computed using polar coordinates, (r. 9). We use a variant where the angle is given by the arctangent with principal values [0, π and the length and r tells us where along the line the datapoint lies on.
R could be negative. specifies line (through the origin),
We consider ||a]|2 E (1.0, 2.0), where for each |||||| we vary the angle Zu that makes with the horizontal axis. Our computed PID presented Figure Overall, we find that the PID matches what expect from intuition. For Cartesian, unique information dominates when
Results:
5

---

Table 2: Estimating PID on synthetic generative model datasets. Both CVX and BATCH measures agree with each other relative values and are consistent with ground truth interactions.
Task
DR
Du
Dua
Ds
22,
PID
R
U1
U2
S
R
U1
U2
5
RU
U2
S
R
U1
U2
5
R
U2
U2
CVX
0.16
0
0
0.05
0.
0
0.05
0
0.17
0.05
0.07
0
0.01
14
0.04
0.01
0.07
BATCH
0.29
0.02
0.02
0
0
30
0
0.30
0
0.11
0.02
0.02
0
0.
0.06
0.01
0.01
0.06
Truth
0.58
0
0
0
0.56
0
0
0
0.54
0
0
0
0
0.56
0.13
0
0 0.27
Task
y f(21, 22. 2.))
y
f21,
22,
2
1/ =
f(2)
22,
Fc)
1/
f(2,
V
f
2c)
PID
H U1 U2
U1
U2
8
R U1
U2
S
R
U1 U2
S
U1
U2
CVX
0.06
0 0.07
0.07
0
0
0.12
0.1
0
0.01
0.07
0.03
0 0.04
0.05
0.1
0
0.04
0.05
BATCH
0.04
0 0.06
0. 11
0.02
0.02
0.
0 0.11
0.02
0.02
0.05
0
0.07
0 0.06
0
0.19
0 0.06
0
Truth
0 0.25 0.25
0.18
0
0
0.36
0.22
0
0
0.22
0.21
0 0.21
0
0.34
0.17
the angle goes to 0 or 2/2 if centroids share coordinate, then observing that coordinate yields no information about y. Conversely, synergy and redundancy peak at π //4 Interestingly, synergy seems to be independent of ||1|22 For Polar, redundancy is O. Furthermore, 0 contains no unique information, since 0 shows nothing about y unless we know r (in particular, its sign). When the angle goes to π /2, almost all information is unique in The distinctions between Cartesian and Polar highlight how different representations of data can exhibit wildly different PID values, even if total information is the same. More thorough results and visualizations of q" are in Appendix
We begin with a set of latent vectors 21, 22, 2e ~ (Os,,, E3), d= 50 representing information unique to X1, X2 and common to both respectively. [21, 2e] is transformed into high-dimensional x1 using a fixed transformation Ti and likewise [22. =e] to 222 via T2. The label y is generated function of (1) only Ze, in which case we expect complete redundancy, (2) only 21 or 22 which represents complete uniqueness. (3) a combination of 21 and 22 representing complete synergy, or (4) arbitrary ratios of each of the above with 2 representing half of the dimensions from z; and therefore half of each interaction. In total, Table [ shows the 10 synthetic datasets we generated: 4 specialized datasets D,, I E Ui, U2, S) where y only depends on one interaction, and 6 mixed datasets with varying interaction ratios. We also report the ground-truth interactions as defined by the label-genee -eneerrnn process and the total capturable information using the bound in Feder and Merhav (29), which relates the accuracy of the best model on these datasets with the mutual information between the inputs to the label. Since the test accuracies for Table 2 datasets range from 67-75%, this corresponds to total MI of 0.42 - 0.59 bits.
Synthetic generative model:
Results: From Table 2 both CVX and BATCH agree in relative PID values, correctly assigning the predominant interaction type and interactions with minimal presence consistent with the ground-truti based on data generation. For example, DR has the highest R value, and when the ratio of 2 increases, Uj increases from 0.01 on y ʃ #2, x²) to 0.06 on y = ʃ(21, = **) We also note some interesting observations due to the random noise label generation, such as the non-zero synergy measure of datasets such as Drz Du, Dt, whose labels do not depend on synergy.
4.2 Quantifying Real-world Multimodal Benchmarks
We now apply these estimators to quantify the interactions in real-world multimodal datasets.
Real-world multimodal data setup: We use a large collection of real-world datasets in Multi- Bench (60] which test multimodal fusion of different input signals (including images, video, audio, text, time-series, sets, and tables) for different tasks (predicting humor, sentiment, emotions, mortality rate, ICD-9 codes, image-captions, human activities, digits, and design interfaces). We also include experiments on qmestion-answeriin test grounding of language into the visual domain. For the 4 datasets (top row of Table 3) involving images and text where modality features are available and readily clustered, we apply the CVX estimator on top of discrete clusters. For the remaining datasets (bottom row of Table 3 with video, audio, and medical time-series modalities, clustering is not easy, so we use the end-to-end BATCH estimator (see Appendix C.4 for full dataset and estimator details).
(Visual Question Answering 2.0 [9] and CLEVR (53) which
Human judgment of interactions: Real-world multimodal datasets do not have reference PID values, and exact PID computation is impossible due to continuous data. We therefore use human judgment as a reference. We design new annotation scheme where we show both modalities and the label and ask each annotator to annotate the degree of redundancy, uniqueness, and synergy on scale of 0-5, alongside their confidence in their answers on scale of 0-5. We show sample

---

Table 3: Estimating PID on real-world MultiBench 60 datasets. Many of the estimated interactions align well with human judgement as well as unimodal performance.
Task
AV-MNIST
ENRICO
VQA 2.0
CLEVR
PID
R
U1
U2
5
R
U1
U2
S
R
U1
U2
5
R
U1
U2
S
CVX
0.10
0.97
0.03
0.08
0.73
0.38
0.53
0.34
0.79
0.87
4.92
0.55
0,48
5.16
Human
0.57
0.61
0
0
0
0
6.58
0
0
0
6.19
Task
MOSEI
UR-FUNNY
MUSTARD
MIMIC
PID
R
U1
U2 S
R
U1
U2
S R
U1
U2
S
R
U1
U2 S
BATCH
0.26
0.49
0.03
0.04
0.03
0.04 0.01 0.08
0.14
0.01
0.01
0.30
0.05
0.17
0.01
Human 0.32 0.20
0.15 0.15
0.01 0.05 0.03 0.04
0.13
0.17
0.04
0.16
user interface and annotation procedures in Appendix C.5 We give 50 datapoints from each dataset (except MIMIC and ENRICO which require specialized knowledge) to 3 annotators each.
Results on multimodal fusion: From Table we find that different datasets do require different interactions. Some interesting observations: (1) (1) all pairs of modalities on MUSTARI RTARR sarcasm detection show high synergy values, which aligns with intuition since sarcasm often due to a contradiction between what is expressed in language and speech, (2) uniqueness values are strongly correlated with unimodal performance (e.g., modality 1 in AV-MNIST and MIMIC), (3) datasets with high synergy do indeed benefit from interaction modeling also seen in prior work (e.g.. MUSTARD, UR-FUNNY) 17 43), and (4) conversely datasets with low synergy are those where unimodal performance is relatively strong (e.g., MIMIC) [60]
Results on QA: We observe very high synergy values as shown in Table 3 consistent with prior work studying how these datasets were balanced (e.g.. VQA 2.0 having different images for the same question such that the answer can only be obtained through synergy) [39] and that models trained on these datasets require non non-additive interactions [46) CLEVR has a higher proportion of synergy than VQA 2.0 (83% versus 75%): indeed, CLEVR is more balanced dataset where the answer strictly depends on both the question and image with a lower likelihood of unimodal biases.
Comparisons with human judgment: For human judgment, we cannot ask humans to give a score in bits, so it is completely different scale (0-5 scale). To put them on the same scale, we normalize the human ratings such that the sum of human interactions is equal to the sum of PID estimates. The resulting comparisons are in Table 3 and we find that the with estimated PID: the highest values are the same for 4 datasets: both explain highest synergy on VQA and CLEVR, image (Ui) being the dominant modality in AV-MNIST, and language (U, being the dominant modality in MOSEI. Overall, the Krippendorff's alpha for inter-annotator agreement is high (0.72 for R. 0.68 for U1. 0.70 for U2. 0.72 for S) and the average confidence scores are also high (4.36/5 for R, 4.35/5 for U1, 4.27 /2 for U2, 4.46/5 for S), indicating that the human-annotatee results are reliable. For the remaining two datasets (UR-FUNNY and MUSTARD), estimated PID matches the second-highes human-annotated interaction. We believe this is because there is some annotator subjectivity in interpreting whether sentiment, humor, and sarcasm are present in language only (U1) or when contextualizing both language and video (S), resulting in cases of low annotator agreement in U1 and S: -0. -0.03 for TR-FUNNY and -0.08, 0..4 for MUSTARD.
interactions overall align
Comparisons with other interaction measures: Our framework allows for easy generalization to other interaction definitions: we also implemented 3 information theoretic measures I-min [108]. WMS 19. and CI 74.. These results are in Table I0 in the Appendix, where we explain the limitations of these methods as compared to PID, such as over- and under-estimation, and potential negative estimation [41). These are critical problems with the application of information theory for shared T(X;; X2; Y) and unique information T (x; Y| x2),, T(X2; YIX,) often quoted in the co-training 14] and multi-view learning [89 94 988 literature. We also tried non-info theory measures: Shapley values (67]. Integrated gradients (IG) [9]] and CCA A. which are based on quantifying interactions captured by a multimodal model. Our work is fundamentally different in that interactions are properties of data before training any models (see Appendix C.6).
4.3 Quantifying Multimodal Model Predictions
We now shift our focus to quantifying multimodal models. Do different multimodal models learn different interactions? ? A better understanding of the types of interactions that our current models struggle to capture can provide new insights into improving these models.
7

---

Table 4: Average interactions (R/U/S) learned by models alongside their average performance on interaction-sseciillieed datasets (Dn/Di Du/ (Ds). Synergy is the hardest to capture and redundancy is relatively easier to capture by existing models.
Model
EF
ADDITIVE
AGREE
ALIGN
ELEM
TENSOR
M1
MULT
LOWER
REC
AVERAGE
R
0.35
0.48
0.44
0.47
0.27
0.55
0.20
0.40
0.47
0.53
0.41 + 0.11
AC)(DR)
0.71
0.74
0.73
0.74
0.70
0.75
0.67
0.73
0.74
0.75
0.73 + 0.02
U
0.29
0.31
0.19
0.44
0.20
0.52
0.18
0.45
0.55
0.55
0.37 : 0.14
(Cc(DD)
0.66
0.55
0.60
0.73
0.66
0.73
0.66
0.72
0.73
0.73
0.68 + 0.06
0.13
0.09
0.08
0.29
0.14
0.33
0.12
0.29
0.31
0.32
0.21 = 0.10
AC)(Ds)
0.56
0.66
0.63
0.72
0.66
0.74
0.65
0.72
0.73
0.74
0.68 + 0.06
Setup: For each dataset, we train a suite of models on the train set Dirain and apply it to the validation set vat. yielding a predicted dataset Prred ((11, () E D. Running PID Preed summarizes the interactions that the model captures. We categorize and implement a comprehensive suite of models (spanning representation fusion at different feature levels, types of interaction inductive biases, and training objectives) that have been previously motivated to capture redundant, unique, and synergistic interactions (see Appendix C.7 for full model descriptions).
Results: We show results in Table 4 and highlight the following observations:
General obxervations: We first observe that model PID values are consistently higher than dataset PID. The sum of model PID is also a good indicator of test performance, which agrees with their formal definition since their sum is equal to / 11 X2): Y), the total task-relevant information.
On redundancy: Several methods succeed in capturing redundancy, with overall average of R 0.41 + 0.11 and accuracy of 73.0 + 2.0% on redundancy-speecilizae and alignment-based methods are particularly strong, and we do expect them to capture redundant shared information 266 82). Methods based on tensor fusion (synergy-basedd including lower-order interactions, and adding reconstruction objectives basedd also capture redundancy.
datasets. Additive, agreement,
On uiiqueness: Uniqueness is harder to capture than redun- dancy, with an average of U 0.37 0.14 Redundancy-basse -bsed methods like additive and agreement do poorly on unique- ness, while those designed for uniqueness (lower-order inter- actions [115] and modality reconstruction objectives (100)
High S cause drop High Ug casse drop
Performance drop
0.3
do well, with on average U = 0.55 and 73.0% accuracy on uniqueness datasets.
0.2
0.1-
On synergy: Synergy is the hardest to capture, with an av- erage score of only S 0.21 + 0.10. Some of the strong methods are tensor fusion [33, tensors with lower-order inter- actions [155, modality reconstruction [100], and multimodal transformer 112], which achieve around $ = 0.30, acc 73.0%. Additive, agreement, and element-wise interactions do not seem to capture synergy well.
0.0
-0.1
Less drop
0.2
0.5
Figure 4: We find high correlation (p 0.8) between the performance drop when X, is missing and the model's U, value: high U, coin- cides with large performance drops (red), but low U, can also lead to performance drops. The latter can be further explained by large so X; is necessary (green).
we also show connections between PID and model performance in the presence of missing modalities. We find high correlation (p- 0.8) between the performance drop when X; is missing and the model's 's U; value. Inspecting Figure we find that the implication only holds in one direc- tion: high U; coincides with large performance drops (in red), but low U, can also lead to performance drops (in green). The latter can be further explained by the presence of large 5 values: when X; is missing, synergy can no longer be learned which affects performance. For the subset of points when U; < 0.05, the correlation between S and performance drop is p = 0.73 (in contrast, the correlation for R is p = 0.01).
On robustness: Finally,
4.4
PID Agreement and Model Selection
Now that we have quantified datasets and models individually, the natural next question unifies both: what does the agreement between datasel and model PID measures tell us about model performance? We hypothesize that models able to capture the interactions necessary in given dataset should also
8

---

Table 5: Model selection results on unseen synthetic and real-world datasets. Given new dataset D, finding the closest synthetic dataset D' with similar PID values and recommending the best models on D' consistently achieves 95% 100% of the bestpperfonning model on D.
Dataset
Synthetic Datasets MIMIC ENRICO UR-FUNNT MOSEI MUSTARD MAPS
% Performance
99.91%
99.78 78%
100%
98.58%
99.35%
95.15%
100%
achieve high performance. Given estimated interactions on dataset and model ((D) trained on D, we define the agreement for each interaction I E {R. U1, U2, S} as:
Id
= (p) io= Ere(RU, U2,3) 1/2 13
(6)
which summarizes the quantity of interaction captured by model (1 ((D)) weighted by its normal-
ized importance in the dataset (fp). The total agreement sums over a D) = Σ, a)
Results: Our key finding is that PID agreement scores af,, f D) correlate (p 0.81) with model accuracy across all 10 syn- thetic datasets as illustrated in Figure This shows that PID agreement can be a useful proxy for model performance. For the specialized datasets, we find that the correlation between
0.75
Performance
0.70
ay and D, is 0.96 for R, 0.86 for U, and 0.91 for S, and neg-
0.65
atively correlated with other specialized datasets. For mixed
0.60
datasets with roughly equal ratios of each interaction, the mea-
sures that correlate most with performance are an (p 0.82) and as (p- 0.89); datasets with relatively higher redundancy see 0.89 for agg those with higher uniqueness have au, and ou,, correlate o = 0.92 and p 0.85; those with higher synergy increases the correlation of as to p 0.97.
0.55
0.0
0.1
0.2 0.3 0.4
Agreement
Figure 5: PID agreement a(S, D) between datasets and models strongly correlate with model accuracy (p = 0.81).
Using these observations, our final experiment is model selec- tion: can we choose the most appropriate model to tackle the interactions required for dataset?
Setup: Given a new dataset D, we first compute its difference in normalized PID values with respect to D' among our suite of 10 synthetic datasets, s(D, D') Er (R,EERRU.... I - T to rank the dataset D* with the most similar interactions, and return the top-3 performing models on Dr. In other words, we select models that best capture interactions that are of similar nature and degree as those in D. We emphasize that even though we restrict dataset and model search to synthetic datasets, we evaluate model selection on real-world datasets and find that it generalizes to the real world.
Results: We test selected models on 5 new synthetic datasets with different PID ratios and 6 real-world datasets, summarizing results in Table We find that the top 3 chosen models achieve 95%- 100% of the best-performing model accuracy, and > 98.5% for all datasets except 95.2% on MUSTARD. For example, UR-FUNNY and MUSTARD have the highest synergy (S = 0.13, S 0.3) and indeed transformers and higher-order interactions are helpful (MUL.T: 65%, MI: 61 61%, Tensor: 60%). ENRICO has the highest Rt = 0.73 and U2 = 0.53, and methods for redundant and unique interactions perform best (LOWER: 52%. ALIGN: 52%, AGREE: 51%). MIMIC has the highest U, 0.17, and unimodal models are mostly sufficient 60
4.5
world Applications
Finally, we apply PID to 3 real-world case studies: pathology, mental health, and robotic perception (see full details and results in Appendix C.II).
Case Study 1: Computational pathology. Cancer prognostication is challenging task in anatomic pathology that requires integration of whole-slide imaging (WSI) and molecular features for patient stratification [2 63 [7 We use The Cancer Genome Atlas (TCGA), a large public data consortium of paired WSI, molecular, and survival information [007 96). including modalities:
(1) pre-extracted histology image features from diagnostic WSIs and (2) bulk gene mutation status, copy number variation, and RNA-Seq abundance values. We evaluate on two cancer datasets in TCGA, lower-grade glioma (LGG
73
479) and pancreatic adenocarcinoma (PAAD [83] 209).

---

Results: In TCGA-LGG, most PID measures were near-zero except U2 = 0.06 for genomic features, which indicates that genomics the only modality containing task-relevant information. This conclusion corroborates with the high performance of unimodal-genome and multimodal models performance was low. In TCGA-PAAD, the uniqueness in pathology and genomic features was less than synergy (U₁ = 0.06, and U2 = 0.08 and S = 0.15), which also match the improvement of using multimodal models that capture synergy.
in Chen et al. [22], while moimooal-pptholle
Case Study 2: Mental health. Suicide the second leading cause of death among adolescents [8]] Intensive monitoring of behaviors via adolescents' frequent use of smartphones may shed new light on the early risk of suicidal idcations [37 72), since smartphones provide rich behavioral markers [59]. We used a dataset, MAPS, of mobile behaviors from high-risk consenting adolescent populations (approved by IRB). Passive sensing data is collected from each participant's 's smartphone across 6 months. The modalities include (1) text entered by the user represented bag of top 1000 words, (2) keystrokes that record the exact timing and duration of each typed character, and (3) mobile applications used per day as a bag of 137 apps. Every morning, users self-report their daily mood, which we discretized into -1, 11, O, +1. In total, MAPS has 844 samples from 17 participants.
Results: We first experiment with MAPST.K using text and keystroke features. PID measures show that MAPST,K K has high synergy (0.40), some redundancy (0.12), and low uniqueness (0.04). We found the purely synergistic dataset Ds has the most similar interactions and the suggested models LOWER REC, and TENSOR that work best on Ds were indeed the top best-performing models on MAPSy, K, indicating that model selection is effective. Model selection also retrieves the best-performing model on MAPST,A using text and app usage features.
Case Study Robotic Perception. MuJoCo PUSH [57 is a contact-rich planar pushing task in MuloCo [95]. where 7-DoF Panda Franka robot is pushing a circular puck with its end-effector in simulation. The dataset consists of 1000 trajectories with 250 steps sampled at 10Hertz. The multimodal inputs are gray-scaled images from an RGB camera, force and binary contact information from a force/torque sensor, and the 3D position of the rohot end-effector. We estimate the 2D position of the unknown object on a table surface while the robot intermittently interacts with it.
Results: We find that BATCH predicts U, = 1.79 as the highest PID value, which aligns with our ob- servation that image is the best unimodal predictor. Comparing both estimators, CVX underestimates U1 and R since the high- high-dimensional time-series modality cannot be easily described by clusters without losing information. In addition, both estimators predict a low U2 value but attribute high R. implying that multimodal model with higher-order interactions would not be much better than unimodal models. Indeed, we observe no difference in performance between these two.
5 Conclusion
Our work aims quantify the nature and degree of feature interactions by proposing scalable estimators for redundancy, uniqueness, and synergy suitable for high-dimensional heterogeneous datasets. Through comprehensive experiments and real-world applications, demonstrate the utility of our proposed framework in dataset quantification, model quantification, and model selection. We are aware of some potential limitations:
These estimators only approximate real interactions due to cluster preprocessing or unimodal models, which naturally introduce optimization and generalization errors. We expect progress in density estimators, generative models, and unimodal classifiers address these problems.
2. It is harder to quantify interactions for certain datasets, such as ENRICO which displays all interactions which makes it difficult to distinguish between R and S or U and S.
3. Finally, there exist challenges in quantifying interactions since the data generation process is never known for real- world datasets, so we have to resort to human judgment, other automatic measures, and downstream tasks such as estimating model performance and model selection.
Future work can leverage PID for targeted dataset creation, representation learning optimized for PID values, and applications of information theory to higher-dimensional data. More broadly, there are several exciting directions in investigating more applications of multivariate information theory in modeling feature interactions, predicting multimodal performance, and other tasks involving feature interactions such as privacy-- privecy-reseervig and fair representation learning from high-dimensional data 28 42). Being able to provide guarantees for faimess and privacy-preserving preserving learning can be particularly impactful.
10

---

Acknowledgements
This material is based upon work partially supported by Meta, National Science Foundation awards 1722822 and 1750439, and National Institutes of Health awards R01MH1257400, R01MH1 R01MH096951 and R21MH1 PPL. is supported in part by Siebel Scholarship and a Waibel Presidential Fellowship. RS is supported in part by ONR grant 9000142312366 and DARPA FA87502321015. One of the aims of this project is to understand the comfort zone of people for better privacy and integrity. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred. Finally, we would also like to acknowledge feedback from the anonymous reviewers who significantly improved the paper and NVIDIA's GPU support.
References
[1] Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. A rewriting system for convex optimization problems. Journal of Control and Decision, 5(1):42-60 60, 2018.
[2] Paul D Allison. Testing for interaction in multiple regression. American journal of sociology, 1977.
[3] Brandon Amos. Tutorial amortized optimization for learning to optimize over continuous domains. arXiv preprint arXiv:2202000665, 2022.
[4] Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pages 1247-1255 1255, PMLR, 2013.
[5] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh. Vqa: Visual question answering. Proceedings of IEEE international conference on computer vision, pages 2425-2433, 2015.
[6] MOSEK ApS. MOSEK Optimizer API for Python 10.0.34, 2022.
[7] Benjamin Auffarth, Maite López, and Jesús Cerquides. Comparison redundancy and relevance measures for feature selection in tissue classification of ct images. In Industrial conference on data mining, pages 248 262. Springer, 2010.
[8] Maria-Florina Balcan, Avrim Blum, and Ke Yang. Co-training and expansion: Towards bridging theory and practice. Advances in neural information processing systems, 17, 2004.
[9] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 2018.
[10] Reuben M Baron and David A Kenny. The moderator-mediator variable distinction in social psycho- logical research: Conceptual, strategic, and statistical considerations. Journal of personality and social psychology, 5176) 1986.
[11] Jan Beirlant, Edward Dudewicz, László Györf, Edward C Van der Meulen, et al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 1997.
[12] Anthony Bell. The co-information lattice. In Proceedings of the fifth international workshop on independent component analysis and blind signal separation: ICA, volume 2003, 2003.
[13] Nils Bertschinger, Johannes Rauh. Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy. 16(4):2161-2183, 2014.
[14] Avrim Blum Tom Mitchell. Combining labeled unlabeled data with co-training, In Proceedings of the eleventh anmual conference on Computational learning theory, pages 92-100, 1998.
[15] Cameron W Brennan, Roel GW Verhaak, Aaron McKenna, Benito Campos, Houtan Noushmehr, Sofie Salama, Siyuan Zheng, Debyani Chakravarty, J Zachary Sanbom, Samuel H Berman, et al. The somatic genomic landscape of glioblastoma. cell, (55(2) 477, 2013.
[16] Nancy Ann Oberheim Bush. Susan M Chang, and Mitchel S Berger. Current and future strategies for treatment of glioma. Neurosurgical review, 40(1):1-14 14, 2017.
[17] Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sareasm detection (an _obviously. perfect paper). In ACL, 2019.
[18] CDC. CDC WONDER: Underlying cause of death, 1999-2019. 2020.

---

[19] Gal Chechik, Amir Globerson, M Anderson, E Young, Israel Nelken, and Naftali Tishby. Group redun- dancy measures reveal redundancy reduction in the auditory pathway. Advances in neural information processing systems, 14, 2001.
[20] Richard J Chen, Ming Y Lu, Jingwen Wang, Drew FK Williamson, Scott Rodig, Neal Lindeman, and Faisal Mahmood. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEF Transactions on Medical Imaging, 2020.
[21] Richard Chen, Ming Y Lu, Wei-Hung Weng. Tiffany Y Chen, Drew FK Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015 4025, 2021.
[22] Richard J Chen, Ming Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Zahra Noor, Muham- mad Shaban, Maha Shady, Mane Williams, Bumjin Joo, et al. Pan-cancer integrative Histology-genoiic analysis via multimodal deep learning. Cancer Cell, 4068):865-878, 2022.
[23] C Mario Christoudias, Raquel Urtasun, and Trevor Darrell. Multi View learning in the presence of view disagreement. In Proceedings of the Twenty-Foutd Fourth Conference on Uncertainty in Artificial Intelligence. pages 88 96, 2008.
[24] Marco Cuturi. information processing systems, 26, 2013.
Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural
[25] Steven Diamond and Stephen Boyd. CVXPY: A Python-embeddeed modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1 5, 2016.
[26] Daisy Yi Ding, Shuangning Li, Balasubramanian Narasimhan, and Robert Tibshirani. Cooperative lcarning for multiview analysis. Proceedings of the National Academy of Sciences, 2022.
[27] Alexander Domahidi, Eric Chu, and Stephen Boyd. Ecos: An socp solver for embedded systems. In 2013 European Control Conference (ECC), pages 3071-3076. IEFF, 2013.
[28] Sanghamitra Dutta, Praveen Venkatesh, Piotr Mardziel, Anupam Datta, and Pulkit Grover. An information- theoretic quantification of discrimination with exempt features. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages
2020.
[29] Meir Feder and Neri Merhav. Relations between entropy and error probability. IEFE Transactions on Information theory, 40(1):259-266, 1994.
[30] Ross Flom and Lorraine E Bahrick. The development of infant discrimination of affect in multimodal and unimodal stimulation: The role of intersensory redundancy. Developmental psychology, 43(1):238, 2007.
[31] Joseph C Franklin, Jessica D Riheiro, Kathryn R Fox, Kate H Bentley, Evan M Kleiman, Xieyining Huang, Katherine M Musacchio, Adam C Jaroszewski, Bernard P Chang, and Matthew K Nock. Risk factors for suicidal thoughts and behaviors: meta-analysis of 50 years of research. Psychological bulletin, 2017.
[32] Jerome H Friedman and Bogdan E Popescu. Predictive learning via rule ensembles. The annals of applied statistics, 2(3):916 954, 2008
[33] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilincar pooling for visual question answering and visual grounding. Conference on Empirical Methods in Natural Language Processing, pages 457-468, ACL, 2016.
[34] Wendell R Garner. Uncertainty and structure as psychological concepts. 1962.
[35] Timothy Gawne and Barry J Richmond. How independent are the messages carried by adjacent inferior temporal cortical neurons? Journal of Neuroscience, 13(7):2758 2771,, 1993.
[36] AmirEmad Ghassami and Negar Kiyavash. Interaction information for causal inference: The case of directed triangle. In IEEE International Symposium on Information Theory (ISIT), 2017.
[37] Catherine R Glenn and Matthew K Nock. Improving the short-term prediction of suicidal behavior. American journal of preventive medicine, 2014.
[38] Amir Globerson and Tommi Jaakkola. Approximate inference using conditional entropy decompositions.
In Artificial Intelligence and Statistics, pages 131- 138. PMLR, 2007.
12

---

[39] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913, 2017.
[40] Michael Grant, Stephen Boyd, and Yinyu Ye. Disciplined convex programming. In Glohal optimization, pages 155-210. Springer, 2006.
[41] Virgil Griffith and Christof Koch. Quantifying synergistic mutual information. In Guided selianaaaiza::: inception, pages 159-190. Springer, 2014.
[42] Faisal Hamman and Sanghamitra Dutta. Demystifying local and global faimess trade-offs federated learning using information theory. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opporruiities, 2023.
[43] Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philiipe Morency, and Mohammed Ehsan Hoque. Ur-funny: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2046-2056, 2019.
[44] Trevor Hastie and Robert Tibshirani. Generalized additive models: some applications. Journal of the American Statistical Association, 82(398):371-388 1987.
[45] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[46] Jack Hessel and Lillian Lee. Does my multimodal model learn cross-modal interactions? it's harder to tell than you might think! In EMNLP. 2020.
[47] Ming Hou, Jiajia Tang, Jianhai Zhang, Wanzeng Kong, and Qibin Zhao. Deep multimodal multilinear fusion with high-order polynomial pooling. Advances in Neural Information Processing Systems, 32: 12136-12145, 2019.
[48] Maximilian Ilsc, Jakub Tomezak, and Max Welling. Attention- deep multiple instance lcarning. In International conference on machine learning, pages 2127-2136. PMLR, 2018
[49] Jan Ittner, Lukasz Bolikowski, Konstantin Hemker, and Ricardo Kennedy. Feature synergy. redun- dancy, and independence in global model explanations using shap vector decomposition. arXiv preprint arXiv arXiv:2107.12466, 2021.
[50] James Jaccard and Robert Turrisi. Interaction effects in multiple regression. Number 72, sage, 2003.
[51] Aleks Jakulin and Ivan Bratko. Quantifying and visualizing attribute interactions: An approach based on entropy. 2003.
[52] Siddhant M. Jayakumar, Wojeiech M. Czarnecki, Jacob Menick. Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Tch, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In International Conference on Learning Representations, 2020.
[53] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017.
[54] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: aasion-and-languee transformer without convolution region supervision. In International Conference Machine Learning, pages 5583 3593.5594. PMLR. 2021.
[55] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
[56] Matthew Michael Large, Daniel Thomas Chung, Michael Davidson, Mark Weiser, and Christopher James Ryan. In-patient suicide: selection of people at risk, failure of protection and the possibility of causation. B/Psych Open, 333):022-105 2017.
[57] Michelle A Lee, Brent Yi, Roherto Martin-Martin, Silvio Savarese, and Jeannette Bohg. Multimodal sensor fusion with differentiable filters. IROS, 2020.
[58] Paul Pu Liang, Zhun Liu, Yao Huo- Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, and Louis-Philipge Morency. Learning representations from imperfect time series data via tensor rank regularization. In ACL, 2019.
13

---

[59] Paul Pu Liang. Terrance Liu, Anna Cai, Michal Muszynski, Ryo Ishii. Nick Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Morency. Learning language and multimodal privacy-pred preeeripr markers of mood from mobile data. In ACL, 2021.
[60] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multihench: Multiscale benchmarks for multimodal representation learning. In Thirty-fith Conference on Newral Information Processing Systems Datasets and Benchmarks Track (Round) 1). 2021.
[61] Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis Chuis-Philippe Moreney, and Ruslan Salakhutdinov. Multiviz: An analysis benchmark for visualizing and understanding multimodal models. arXiv preprint arXivv2207.00056, 2022.
[62] Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges. and open questions. arXiv preprint urXi::2209.03400 2022.
[63] Jana Lipkova, Richard Chen, Bowen Chen, Ming Y Lu, Matteo Barbieri, Daniel Shao, Anurag J Vaidya, Chengkuan Chen, Luoting Zhuang, Drew FK Williamson, et al. Artificial intelligence for multimodal data integration in oncology. Cancer Cell, 40( 40(10):095--1100 2022.
[64] Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
[65] Zhun Liu, Ying Shen, Varun Bharadhwaj I. Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and I Louis-Philippe Morency. Efficient low-rank multimodal fusion with sodality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 2247-22525 2018.
[66] David N Louis, Arie Perry, Guido Reifenberger, Andreas Von Deimling, Dominique Figarella-Brane Webster K Cavenee, Hiroko Ohgaki, Otmar D Wiestler, Paul Kleihues, and David W Ellison. The 2016 world health organization classification of tumors of the central nervous system: summary. Acta neuropathologcca, 131 (31666800,-820, 2016.
[67] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[68] Yiwei Lyu, Paul Pu Liang. Zihao Deng. Ruslan Salakhutdinov, and Louis- Phii--Phi Morency. Dime: Fine-grained interpretations of multimodal models via disentangled local explanations. AIES, 2022.
[69] Alessio Mazzetto, Dylan Sam, Andrew Park, Eli Upfal, and Stephen Bach. Semi-supervised aggregation of dependent weak supervision sources with performance guarantees. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, 2021.
[70] William McGill, Multivariate information transmission. Transactions of the IRE Professional Group on Information Theory, 4(4):93-111, 1954.
[71] Pooya Mobadersany, Safoora Yousefi, Mohamed Amgad, David A Gutman, Jill S Barnholtz-Sloan, José E Velázquez Vega, Daniel J Brat, and Lee AD Cooper. Predicting cancer outcomes from histology and genomies using convolutional networks. Proceedings of the National Academy of Sciences, 2018.
[72] Inbal Nahum-Shani, Shawna N Smith, Bonnie Spring, I Linda M Collins, Katie Witkiewitz, Ambuj Tewari, and Susan A Murphy. Just-in-time adaptive interventions (jtais) in mobile health: key components and design principles for ongoing health behavior support. Annals of Behavioral Medicine, 2018
[73] Cancer Genome Atlas Research Network. Comprehensive, integrative genomic analysis of diffuse lower-grade gliomas. New England Journal of Medicine, 372(26):248) 12498, 2015.
[74] Sheila Nirenberg. Steve M Carcieri, Adam L Jacobs, and Peter E Latham. Retinal ganglion cells act largely as independent encoders. Nature, 41 1166688):688--0111 2001.
[75] Brendan O' O'Donoghue, Eric Chu, Neal Parikh, and Stephen Boyd. Conic optimization via operator splitting and homogeneous delf-dual embedding. Journal of Optimization Theory and Applications, 2016.
[76] Ari Pakman, Amin Nejatbakhsh, Dar Gilboa, Abdullah Makkeh, Luca Mazzucato, Michael Wibral, and Elad Schneidman. Estimating the unique information of continuous variables. Advances in Neural Information Processing Systems, 34:20295-20307, 2021.
[77] Liam Paninski. Estimation of entropy and mutual information. Neural computation, 2003.
14

---

[78] Sarah Partan and Peter Marler. Communication goes multimodal. Science, 283(54006:1272-12777 1999.
[79] Sarah R Partan and Peter Marler. Issues in the classification of multimodal communication signals. The American Naturalist, (66(2))231-244 2005.
[80] Fernando Pérez-Cruz. Kullback-leibler divergence estimation of continuous distributions, 2008 IEEE international symposium on information theory, pages 1666-1670. IEEE, 2008
[81] Alexandra Proca, Fernando Rosas, Andrea Luppi, Daniel Bor, Matthew Crosby, and Pedro AM Mediano. Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks, arXiv preprint ar arXiv:2210.02900 2022.
[82] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[83] Benjamin J Raphael, Ralph H Hruban, Andrew J Aguirre, Richard A Moffitt, Jen Jen Yeh, Chip Stewart, A Gordon Robertson. Andrew D Cherniack, Manaswi Gupta, Gad Getz, al. Integrated genomic charactcrization of panereatic ductal adenocarcinoma. Cancer cell, 32(2): 185-)388522, 2017.
[84] John A Rice. Mathematical statistics and data analysis. Cengage Learning, 2006.
[85] Natalie Ruiz, Ronnie Taib, and Fang Chen. Examining the redundancy of multimodal input. In Proceed- ings of the 18th Australia conference Compuler-Human Interaction: Design: Activities, Artefacts and Environments, pages 389-392, 2006.
[86] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
[87] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition, 2014. htLhhtts:://rrt htttps://arxiv..orgaaa///////// 1556
[88] Daria Sorokina, Rich Caruana, Mirek Ricdewald, and Daniel Fink. Detecting statistical interactions with additive groves of trees. In Proceedings of the 25th international conference on Machine learning. 2008.
[89] Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. In Conference on Learning Theory, 2008.
[90] Milan Studený and Jirina Vejnarová. dependence. In Learning in graphical models, pages 361-297. Springer, 1998.
The multinffrmationn function a tool for measuring stochastic
[91] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Interna- tional conference on machine learning, pages 3319-3328. PMLR, 2017.
[92] Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprins arXiv:1611.01891 2016.
[93] Han Te Sun. Multiple mutual informations and multiple interactions in frequency data. Inf. Control, 46: 26-45, 1980.
[94] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 2020.
[95] Emanuel Todorov, Tom Erez. and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 TEFF/RSJ International Conference Intelligent Robots and Systems. IERF, 2012.
[96] Katarzyna Tomezak, Patrycja Czerwińska, and Maciej Wiznerowiez. The cancer genome atlas (tcga): an immeasurable source of knowledge. Contemporary Oncology/Wspótczeena Onkologia, 2015.
[97] Giulio Tononi, Olaf Spors, and Gerald M Edelman. A measure for brain complexity: relating functional segregation and integration in the nervous system. Proceedings of the National Academy of Sciences, 91 (11):5033-555 5037, 1994.
[98] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory, pages 1179-1206. PMLR, 2021.
[99] Yao Kao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang. Zico Kolter, Louis Phiis-Phllippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558 6569, 2019.
15

---

[100] Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh. Louis-Philippe Morency. and Ruslan Salakhutdinov. Learning factorized multimodal representations. In International Conference on Learning Representations, 2019.
[101] Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. International Conference on Learning Representations, 2018
[102] Michael Tsang, Dehua Cheng, Hanpeng Liu, Xue Feng, Eric Zhou, and Yan Liu. Feature interaction interpretability: case for explaining recommendatioon systems via neural interaction detection. In International Conference on Learning Representations, 2019.
[103] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[104] Qing Wang, Sanjeev R Kulkarni, and Sergio Verdú. A nearest-neighbor approach to estimating divergence between continuous random vectors. In 2006 IEEE International Symposium on Information Theory, pages 242-2422222 246. IEEE, 2006.
[105] Xingho Wang. Jianben He, Zhihua Jin, Muqiao Yang. Yong Wang, and Huamin Qu. M2lens: Visualizing and explaining multimodal models for sentiment analysis. IEEE Transactions on Visualization and Computer Graphics, 28(1):802 812, 2021.
[106] Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of research and development, 4(1):66-82 1960.
[107] John N Weinstein, Eric A Collisson, Gordon B Mills, Kenna R Shaw. Brad A Ozenberger, Kyle Ellrott, Ilya Shmulevich, Chris Sander, and Joshua M Stuart. The cancer genome atlas pan-cancee analysis project. Nature genetics, 45(10):1113 1120, 2013.
[108] Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXiv: 1004.2515, 2010.
[109] Patricia Wollstadt, Joseph Lizier, Raul Vicente, Conor Finn, Mario Martinez- Pedro Mediano, Leonardo Novelli, and Michael Wibral. Idtxl: The information dynamics toolkit xl: a python package for the efficient analysis multivariate information dynamics in networks. Journal of Open Source Software, ((34):1081, 2019.
[110] Patricia Wollstadt, Sebastian Schmitt, and Michael Wibral. A rigorous information-theoreticc definition of redundancy and relevancy in fcature selection based on (partial) information decomposition. arXiv preprint arXiv:2 1877, 2021.
[111] Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervinn learning. Advances in Neural Information Processing Systems, 31, 2018.
[112] Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey, arXiv preprint (tt.::2200066484 2022.
[113] Shaowei Yao and Xiaojun Wan. Multimodal transformer for multimodal machine translation. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020. Association for Computational Linguistics.
[114] Lei Yu and Huan Liu. Efficient feature selection via analysis of relevance and redundancy. The Journal of Machine Learning Research, 5:1205-1222 2004.
[115] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis- Morency. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference Empirical Methods Natural Language Processing, pages 1103-1114, 2017.
[116] AmirAli Bagher Zadeh, Paul Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In ACI, 2018.
16

---

quantification:
X
D= (2.1
U2,
Unique
2. Model quantification:
Redundancyy
Synergy
((D) = (1., [())
[R, UR. Sppp)
Unique 2
(R. C, S.1s.......... (R.U......910 (1,
X2
3. Model selection:
U2. (R,),
Figure 6: The PID framework formalizes the nature of feature interactions for task. Through our proposed scalable estimators for redundancy, uniqueness, and synergy, we demonstrate their useful- ness towards quantifying multimodal datasets and multimodal model predictions, which together yield principles for model selection.
Appendix
A Partial Information Decomposition
Information theory formalizes the amount of information that a single variable (X) provides about another (Y), and is quantified by Shannon's mutual information [86] [(;; Y) which measures amount uncertainty reduced from HYY) to H()[X) when given as input. However, the
(
direct extension of information theory to 3 more variables, such as through total correlation [I06]
34 90 97 or interaction information [70] 93 12 35), both have significant shortcomings. In
particular, the three-way way mutual information I [ X2; Y) can be both negative and positive, leading to considerable difficulty in its interpretation by theoreticians and practitioners.
Partial information decomposition (PID) 108 was proposed as method to generalize information theory multiple variables, by positing a decomposition of the total information 2 variables provide about a task [(X1, X2; Y) into 4 quantities: redundancy R between X1 and X2, unique information U, in X, and U2 X2, and synergy S that should together satisfy the following consistency equations:
R+U I(X;;Y),
(7)
R+ 1(2;;) Y)
(8)
U1+5= -
(9)
U = ((x;;
(10)
5- 1211;
(11)
definition R was first proposed by Williams and Beer [08] and was subsequently improved
by Bertschinger et al. [3] Griffith and Koch [1
giving:
= max (X1;; vEAp
(12)
U1 min L,(X;;Y|XX, gEåp
(13)
U2 min Lg(X;; Y(11) gEA,
(14)
$ = IN X2; Y) A
min [x;, Y)
(15)
where A, = {q E A g(x,,))) = p(r,, y) Vy, I,, E [1,2)) and is the set of joint distributions over X1, X2, Y. We call this the set of marginal-matching distributions. The core intuition these definitions and how they enable PID lies in optimization over Ap instead of simply estimating cnormatiion-hhorree measures for the observed distribution p. Specifically, synergy S the difference between total multimodal information in p and total multimodal information in q" Ap. the non-synergistic distribution (see Figure for illustration). While there have been other definitions of PID that also satisfy the consistency equations, Bertschinger et al. [13] showed that their proposed measure satisfies several desirable properties. Optimizing over Ap be seen as going beyond purely observed data and discovering latent redundancy, uniqueness, and synergy via representation learning.
According Bertschinger et al. 13] suffices to solve for g using the following max-entropy optimization problem, the same q equivalently solves any of the remaining problems defined for
17

---

Unimodal marginalmatcching distributions:
D = (2, v) q(21. = p(.1.1)) (22.0) p))
X
Xs
X2
S (x;1),
(x X2;
Task-relsvant
Task-relevant multimodal
muttimodal info
info without synergy:
Sp- (xx, min Xx; re3,
Figure 7: We illustrate the set of marginal-matching distributions g E Ap. and an intuition behind
the definition of synergy S as the difference between total multimodal information in p and total multimodal information q² Ap, the non-synergistic distribution.
redundancy, uniqueness, and synergy. q' arg max H X2) 9EA,
(max-entropy)
(16)
q-' arg max L X2; nEá,
(redundancy)
(17)
9" arg min I (1 (EAp q' arg min (X1; X2) & 9 arg min I (X2; I| *EA,
(synergy)
(18)
(unique in X1)
(19)
(unique in X2)
(20)
B Scalable PID Estimators
This section provides algorithmic and implementation details on the estimation of PID values for high-dimensional continuous datasets.
B.1 Estimator 1: CVX for Discrete Representations
Our first proposed estimator, CVX, is exact, based on convex optimization, and is able to scale to problems where | and | are around 100. We show high-level diagram of CVX in Figure 8 and provide several implementation details here:
Implementation Details: We implemented (5) using CVXPY 25 1 The transformation from
the max-entropy objective (16) to (5) ensures adherence disciplined convex programs [40], thus convex program. All 3 conic solvers, ECOS (27). SCS (75), and MOSEK 66) were used, with ECOS and SCS being default solvers packaged with CVXPY. Our experience is that MOSEK is the fastest and most stable solver, working out of the box without any tuning. However, it comes with the downside of being commercial. For smaller problems, ECOS and SCS work just fine.
allowing CVXPY to recognize
Potential Numerical Issues: Most conic solvers will suffer from numerical instability when dealing with ill-conditioned problems, typically resulting from coefficients that are too large small. It fairly common, especially the GMM experiment that single bin contains just one sample. Since P is estimated from an empirical frequency matrix, it will contain probabilities of the order //numm-sample When the number of samples is too large (c.g.. 2 1e8), this frequency too small and causes all 3 solvers to crash.
Approximating Continuous Representation using Histogramming: The number and width of bins can affect quality of PID estimation (77). For example, it is known that with fixed number of samples, making the width of bins arbitrarily small will cause KL estimates diverge. It is known that the number of bins should grow sub-linearly with the number of samples. Rice (84) suggest setting the number of bins to be the cubed-root of number samples. In our synthetic experiments, we sampled at least le6 samples and used no more than 100 bins. Note that increasing the number of samples infinitum is not an option due to potential numerical issues described above.
18

---

Convex programming with linear constraints
p(#ul)) (ealv)
11 langaagnt Tuman
p(a (v) picaiv
Text
CVXPY
22
arg aEA,
(X,
Image
((21,
(2.1,
R, U1 U2
Figure 8: Overview of CVX estimator. Appropriate histogramming of each X₃ clustering in preprocessed feature space first performed to constrain |.t]/ By rewriting the entropy objective as KI. divergence (with auxiliary variables) which is recognized convex, this allows us conic solvers such as SCS 775] ECOS [27]. and MOSEK [6 CVXPY to solve for q" and obtain PID values.
B.2 Alternate Methods for Working with Continuous X1, X2:
Another method for discretization is based nearest neighbors 88 104], which has the ad- vantage of performing density estimation as an intermediate step, but ill-suited for our purposes of approximating q² Pakman al. (76) attempt optimize 7 directly in the space of continuous distribution by approximating an optimal copula, which canonically captures dependencies between X1, X2 in q. However, this is nontrivial due to the space of possible joint distributions, and the authors restrict themselves to Gaussian copulas.
B.3 Estimator 2: BATCH for Continuous Representations
Our next estimator, BATCH, is suitable for large datasets where the support of X1 and X₂ are high-dimensionall continuous vectors and not enumerable. Instead, we propose an end-to-end model parameterizing the joint distribution and a training objective whose solution allows us to obtain PID. Here | = | co. However, we wish to estimate the PID values given a sampled dataset (1) ()) () of size
D=
( z2
this appendix section, we provide full details deriving our BATCH estimator, see Figure P for a high- level overview. We first rewrite the optimization objective in (18) before defining neural estimators to approximate high-diment nigh-dimensional variables that we cannot compute exactly, before optimizing the (18) using learned models and obtaining the PID values.
Objective function: First, we observe that the optimization objective in 18 can be written as:
q = arg min Ig X2):3) A
(21)
log g( 22, v) ( x2)9(1)
arg min A
(22)
log (011, x213)
arg min (,A, 21.22
E
(23)
g(01,
log (102 (ri, (ogllriy))(()ll ())(()l)))
arg min q 22.22
(24)
q( (n (1))((())))))))
(222
arg min
E qEA, a a
log Σ Ey'er q( v/ (g(y an )q(x))
(25)
Our goal is to approximate the optimal q that satisfies this problem. Note that by the marginal constraints, g(=1,y) = ())1.) and {(yl = p()'(x)). The former can be obtained by sampling from the dataset and the latter is an unimodal model independent of
19

---

Unnormalized joint distribution
Sinkhorn's algorithm
1
f0(1)
p(z v) = ((yl=)))(a.))
Video
q SINKHORN,(A)
22
f0(2)
Sensors
A = exp
(X2. v)
PID values
Training objective
R= Xx)
U. = Xx, - Xx)
log q ( 11 y) Σ Ea g 2/ (a (x)
min ged, g(t
U2 = 1, X2) - XI)
S = Xx; Xx;
Figure 9: Overview of BATCH estimator: We define end-to-end model that takes in batches of features X, E X²T X2 E XTm Y E ym and learns a matrix A |pmxxx]>> [ to represent
the unnormalized joint distribution ÿ. Following ideas in multimodal alignment [99] and attention
methods [033 that learn approximate distributions over imput data, we parameterize A by computing the outer-product similarity matrix over learned features. To ensure that q Ap. we use an unrolled version of Sinkhorn's 's algorithm (24] which projects A onto Ap by iteratively normalizing all rows and columns to sum to 1 and rescaling to satisfy the marginals p. We plug in the learned into our objective in (55) and update trainable neural network parameters é using projected gradient descent. Upon convergence, we extract PID by approximating [4((X, X2); Y) by sampling and plugging into 3.. This use of mini- bathh of size m can be seen as an approximation of butc-batch gradient descent over the entire dataset of size >> m. While it is challenging to obtain an unbiased estimator of the full-batch gradient since computing the full A is intractable, we found our approach to work in practice for large m. Our approach can also be informally viewed as performing amortized optimization 3 by using @ to implicitly share information about the full batch using subsampled batches.
Neural architecture: To scale to continuous or unseen xy's we define a function approximator that takes in 11, 22 and y and outputs an unnormalized joint density q( 22, v)- Given batches X, X7m X2 E
XT Y E y"m we define learnable encoders f0(1) and fø(2) that output
matrix A E Rmxm > to represent the unnormalized joint distribution q, i.c., we want
[i], X2Ii.. 3). Following ideas in multimodal alignment [99] and attention methods [03] in
deep learning that learn approximate distributions over input data, we also parameterize our neural architecture by computing the outer-produs similarity matrix over learned features:
A = exp (fo(n) f02)) (X2.))))
(26)
Constraining A using Sinkhorn's algorithm: To ensure that the learned is a valid pdf and respects the marginal constraints i(r;, y) = p(r.,, v). we use the Sinkhorn- projects A into the space of non-negative square matrices that are row and column-normalized to satisfy the desired marginals y( and p(#2, y). Sinkhorn's 's algorithm enables us to perform this projection by iteratively normalizing all rows and columns of A to sum to 1 and rescaling the rows and columns to satisfy the marginals. Since p(x,, y/ is not easy to estimate for continuous 24, compute the marginals using Bayes' rule by expanding it as p( (p(y))(()))) (z.). Note that by sampling x; from the dataset, the rows and columns of A are already distributed p(x:). The only remaining term needed is p(y|a;; (.)), for which we use unimodal models p(y) trained before running the estimator and subsequently frozen. Finally, Sinkhorn's 's algorithm is performed by normalizing each row to p(y(z,) and normalizing each column to p(y (22)
algorithm [24), which
p(=))((p))
Optimization of the objective: We obtain the value of ((22|=1,1) by marginalizing from the above distribution ((1,, x2ly). Given matrix representing [( 22, y). the remaining terms can all be computed in closed form through appropriate summation across dimensions in A to obtain q(z,).
20

---

((11.12) x2). ( and ((21, x2l3)). We now have all the terms for the objective in 25 We optimize the objective via gradient descent.
Our final algorithm for training BATCH is outlined in Algorithm T
Extracting PID values from the learned q: To extract the PID values from learned distribution of g. we first expand the definition of mutual information under p and q, and calculate the PID values from their definitions in T-3)
Plyl a3x) ply)
)
X.)
= E
log
(27)
p(y(211, x2) p(u)
IY X1,;
E
log
(28)
21.22.9'p
p(y) Ey q(12|=11 (( )
g(22|=1, (1pp)l1)))))
((Y;X1,X,) = E
log
(29)
21.22.874
R = (Y; X1, X2)
(30)
U1 TY;Y; X1, X2) TY(1;X2
(31)
U2 = (Y; X,, X2) ((;X))
(32)
S = o X1, G X2)
(33)
Algorithm BATCH algorithm.
X2 E X3, Y E E Require: Multimodal dataset X, X2
Initialize feed-forward networks So(1): while not converged do
for sampled batch X1 E XT X2 Xm Y E ymm do
h1 vl < fe(1) (21, V), hzl: t f0(z) (I22
A,t.1./ + exp(h1 [:. yi h2(:,.1))
p(yla::) unimodal predictions.
< SINKHORN,
Marginalize and sample (22 v) from A.
Calculate the loss from (25) using (x) 22, v).
Perform a gradient step on the loss.
end for end while return network weights in f1,, f2.j for j E [C]
Important assumptions and caveats: As note, the above formulation assumes the existence of unimodal models p(y (z.) We train these models independently and freeze them when training the above estimator. After obtaining q. we extract the PID values. To do so, we need an additional multimodal model to estimate ply|21, x22 due to (28). Because part of our goal is multimodal model selection, we must assume that the existing multimodal model might not be expressive enough and could underestimate (28). However, this value is only subsequently used to calculate S, leading to a possibly underestimated S, but leaving all other PID values unaffected by the choice of the multimodal model. In practice, the possibility of S being larger than estimated should be considered, and more complex models should be investigated until 5 stops increasing. Empirically. we have observed that measured value of S by BATCH is generally larger than that measured by CVX; the underestimation is not as large as in the case of clustering in CVX.
In additional to the above, we note that the learned joined distribution ((I1,42,) y) is limited by the expressivity of the neural network and similarity matrix method used in our model. Empirically, the PID values not sensitive hyperparameters, long the network is trained to convergence. Finally, our approximation using batch sampling limits the possible joined distributions to alignments within a batch, which we alleviates by using large batch size (256 in our experiments).
Implementation Details: We separately train the unimodal and multimodal models from the dataset using the best model from Liang et al. [60] and freeze them when running Algorithm | For the
21

---

16-3
16-1
16-1
16_1
16-1
10
0.5
1.5
0.0
1.0
7.5 1e-1
1e-1
7.5 1e-1
7.5 1e-1
7.5 18-
5.0
2.5
2.5
2.5
2.5
2.5
0.0
0.0
0.0
0.0
0.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Figure 10: Results for PID decomposition under Cartesian coordinates. Within each plot: PID
+ represent the R. U1. U2. and their sum respectively. (r,0) corresspond to the unique information (U,, U2) respectively. Across columns: As μal22 varies from {0.1, 0.5, 1.0, 1.5, 2.0}. Top-Bottom: Soft and Hard labels respectively. Note the scale of the y-axis for Soft. The overall trend for each PID component as Lu varies is similar across ||1||2. However, the relative contribution of Synergy and Redundancy changes significantly as || la varies, particularly for Hard.
values for GMMs for Lu in
( E {0,.... 40} x,
neural network in Algorithm|] I we use 3-layer feedforward neural network with hidden dimension of 32. We train the network for 10 epochs using the Adam optimizer with a batch size of 256 and learning rate of 0.001.
To summarize, we propose scalable estimators for PID, each with their approximations for the parameterization of g and procedure for the estimation of PID.
1. Parameterizing q explicitly via joint probability matrix over Ap. and optimizing convex objective (5) with linear constraints. This can only be used for finite X1, X2, Y. after preprocessing by histogramming continuous domains.
Parameterizing y via neural networks and optimize their parameters & by repeatedly subsampling from data; for each subsample, perform step of projected gradient descent (using Sinkhorn's method) as this subsample was the full dataset.
C Experiments
C.1 Synthetic Bitwise Features
We show results on estimating PID values for OR. AND, and XOR interactions between X1 and X2 in Table find that both our estimators exactly recover the ground truth results in Bertschinger al.
13
C.2 Gaussian Mixture Models (GMM)
Extended Results: For each of Cartesian and Polar we consider two cases, Soft-labels, when we observe Y, and Hard-labels, where we use Y predicted by the optimal linear classifier sign (((11,2)) x)), We hope this will provide additional intuition to the bitwise modality experiments. This results are given in Figures I0 II Most of the trends are aligned with what we would expect from Redundancy, Uniqueness, and Synergy. We list the most interesting observations here.
all cases, total information (sum of PID values) is independent of Zu. This is natural consequence of rotational symmetry. It is also higher for higher ||a||2, e.g., 2.0, reflecting the fact that X₁, X₂ contain less information about labels when their centroids are close. Lastly, observe, that when ///4|2 increases, the total information approaches log(2), which is all the label information. This perfectly aligned with our intuition: if the cluster centroids are far away, then knowing a datapoint's 's cartesian coordinates will let us predict its label accurately (which is one bit of information).
For Cartesian, unique information dominates when angle goes to 0 or 2/2. This is sensible, if centroids share a coordinate, then observing that coordinate yields no information about y. Conversely, synergy and redundancy peak T/4.
22

---

16-3
16-1
16-1
16-1
16-1
10
0.5
0.0
1.0
7.5 le-
le-1
7.5 1t=1
7.5
1e-1
7.5 1e-
5.0
2.
2.5
2.5
2.5
0.0
0.0
0.0
0.0
0.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Figure 11: Results for PID decomposition under Polar coordinates where 0 E [0, π and E R,
ie.. angle is restricted the first two quadrants but distance could be negative. Within each plot:
represent the R, U1, U2. S and their sum respectively. Across columns: As Illl2 varies from {0.1,0.5, 1.0. 1.5, 2.0). Top-Bottom: Soft and Hard labels respectively. Note the scale of the y-axis for Soft. The overall trend for each PID component 2μ varies is similar across |//22, except for when ||||| is small in Soft. However, the relative contribution of Synergy and Redundancy changes significantly as ||x||2 varies, particularly for Hard.
PID values for GMMs for Zu { ğ E (o.....,40) . x,
2
0.0
0
0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
15
7.5
7e-1
7.5
7.5
7.5 1e-
7.5
7e-1
5.0
5.0
5.0
5.0
5.0
2.5
2.5
2.5
2.5
1.5
0.0
1.0
0.0
1.0
15
Figure 12: Results for PID decomposition under Polar coordinates, using the normal parameterization
where 0 E [- and r 2 0. Within each plot: PID values for GMMs for Zp in { 3 E (0. (0.....40)) 40}) x
represent the R, U1, U2. S and their sum respectively. (r,0) corresspond to the unique information (U,, U2) respectively. Across columns: As ||||/ varies from 10.1, 0.5, 1.0, 1.5, 2.0). Top-Bottom: Soft and Hard labels respectively. Note the scale of the y-axis for Soft. Observe that PID values are independent of Lu, with unique information from dominating.
Redundancy dominates Synergy as ||a|22 increases. Interestingly, synergy seems to be independent of ||æl|2 (one has to account for the scale in the vertical axis to recognize this). of total information, since Y Y| , X2 deterministic. The deficit
Hard contains 0.65 zs (2) of 0.04 nats is due to discretization near the decision boundary, and is expected to vanish if we increase the number of bins (though this ought to be accompanied by an increase in samples used to approximate PID).
However, for Soft, total information nearly 0 for small ||e||2 (and approaches log(2) when large). When the two cluster centroids are close, it is difficult to glean any information about the (soft) label, while we can predict accurately from both Tı, 22 when centroids are far apart.
For Polar, redundancy is 0. Furthermore, @ contains no unique information, since 0 shows nothing about y unless we know (in particular, its sign). When angle goes to T/2, almost all information is unique in r.
Lastly, we also experimented on the more common definition of polar coordinates, with 0 E TT x] and r 2 0 (that is, angle is allowed to be negative but lengths have to be nonnegative). The results are Figure [2] As one may expect, there is nothing exciting going on for both Softand Hard. For Hard, the optimal classifier is precisely expressed in terms of 0 only, all the information belongs to U2. i.e., 0. For Soft, the same trend remains: virtually all information is in U2 i.e., with the trend becoming sharper as ||d||/2 increases. Intuitively, when the 2 centroids are extremely far
23

---

apart, then the optimal linear classifier is extremely accurate; which may be expressed by alone without any knowledge of r. Conversely, when the centroids are close, r can provide some synergy with thehe larger is, the more "confident" we are in predictions based on 0.
Visualization of q²: The following plots of q" are provided provide some intuition behind the objectives in (6)). We focus on Cartesian for case of visualization. For each ||æ]| E {0.5, 1.0, 2.0 we vary the angle 2u and sample 1e7 labelled datapoints. We then compare q" (for both Soft and Hard) with the empirical distribution To compute PID, we use 50 bins evenly spaced between [-5, 5| for both dimensions. The results for varying ||ا|| shown in Figures 13 15
When 2μ is O, there are few differences, barring slightly "blockier" feel for q" over However, as Lu increases, q" gets skewed rather than rotated, becoming "thinner" in the process. The limiting case is when L1z = T/4 i.e., 45 degrees. Here, the objective diverges to infinity, with what appears to be solution with nonnegative entries only along the diagonal-thi this observation appears hold even as we increase the number of bins. We hypothesize that in the truly continuous case (recall performing discretization for this of experiments), the optimum is degenerate "one dimensional" GMM with mass located only at ((x, Z).
1
-5.
Figure 13: Densities for GMMs where |ll|2 2.0. Top to Bottom: Densities for p. q" (Soft), and q² (Hard). Left to right, Zp in { li E 10. 8)}
-5
-5
5
5
55
-5
5
5
55
Top to Bottom: Densities for p, 9" (Soft), and 4² (Hard). Left to right, Zu in & Z E (0.....8))
Figure 14: Densities for GMMs where ||H|| 1.0.
C.3 Synthetic Generative Model
Our third setting simulates a data generation process where there exists a set of latent concepts that either redundantly, uniquely, synergistically determine the label.
Sctup: We begin with a fixed set of latent variables 21, R2, 2e E 50 from N/ o²) with a 0.5 representing latent concepts for modality 1, modality 2, and common information respectively. The concatenated variables [21, 2e] are transformed into high-dimensional æ E 100 data space using
fixed transformation matrix Ti e 150x100 and likewise [22, øe] to 22 via Th. The label is generated
24

---

-5-
5
5
5
Figure 15: Densities for GMMs where 1|u||2 0.5. Top to Bottom: Densities for p. 7. (Soft), and q² (Hard). Left to right, 2μ in {g 1 E {o.....8)
as a transformation function of (1) only ze. which case we expect redundancy in æ and z2 with respect to y, (2) only 22 which represents uniqueness in 22 respectively, (3) the
concatenated [21, 22ح representing synergy, or (4) arbitrary ratios of each of the above. We also introduce nonlinearity and random noises to increase data complexity. In formal notation,
sigmoid 2i=o-o f())
N 0.5
where I is a fixed nonlinear transformation with dropout (p 0.1) and can be 21, 22, Ze any combination according to the cases described above. These datasets are designed test a combination of feature learning with predefined interactions between the modalities and the label.
We generated 10 synthetic datasets as shown in Table The first four are specialized datasets denoted as 2m m E {R, U1, U2, S} for which the label y/ only depends on redundancy, uniqueness, or synergy features respectively. The rest are mixed datasets with y generated from 21, 22; &e of varying dimensions. We use 27 E 1250 to denote a randomly sub-sampled feature from For example, generate the label of one data point (z( _.(" in the mixed dataset y = (2f 22, ze). we sub-sampled (z7)0), (z²)() from 21, 22 and generate y from [(=])()) (zz)(), za] following the process above.
To report the ground-truth interactions for these datasets, we first train multimodal models and obtain an estimate of test performance on them. We then convert P Paee. the test accuracy of the best model on these datasets into the mutual information between the inputs to the label using the bound in Feder
and Merhav 29
1(1, X2: S log Pase + HY)
(34)
Since the test accuracies for Table 2 datasets range from 67-75%, this corresponds to total MI of 0.42 - 0.59 bits (i.c., from log, 0.67 + to log, 0.75 + 1). The information in each interaction is then computed by dividing this total mutual information by the interactions involved in the data generation process: if the total MI is 0.6 bits and the label depends on half from the common information between modalities and half from the unique information in Z1, then the ground truth 0.3 and U1 == 0.3.
In the main paper, note the agreement between CVX and BATCH and their consis- tency with the dataset (Table In addition, we note that the sum of PID values (which sum to TY; X1, X2)) is generally larger in BATCH than in CVX, which accounts for the information loss during CVX's clustering. Moreover, in the last two cases (y = fzzz =t) and f(2 ze)), CVX predicts non-zero amount of S despite the dataset being synergy-free by construction. Because the total mutual information is small, the spurious synergy becomes large portion of the total PID values. This potentially demonstrates that when the total mutual information is small, spurious information gain could have large impact on the results.
Results:
C.4 Quantifying Multimodal Datasets
Setup: We use large collection real-world datasets in MultiBench 60) which test multimodal fusion of different input signals and requires representation learning of complex world interactions
25

---

Table 6: Full results on estimating PID on real-world MultiBench 660] datasets. Many of the estimated interactions align well with dataset construction (e.g.. MUSTARD for sarcasm) and unimodal performance (e.g., MIMIC and AV-MNIST). On QA [5] datasets, synergy is consistently the highest.
Task
AV-MNIST
ENRICO U1
VQA 2.0
CLEVR
Measure
R
UL U2 5
R
U2
S
R
U1
U/ S
R
U1
U2
S
CVX
0.10 0.97 0.03 0.08 0.73
0.38 0.53 0.34
0.79
0.87
0 4.92
0.55 0.48
O 5.16
Task
MIMIC
UR-FUNNY A.T
UR- V.T
UR-FUNNY V.A
Measure
R
U1 U2 S U1 U2 $
R U1 U2
S
R U, U2 S
BATCH
0.05 0.17 0 0.01 0.02 O 0.08 0.07
0.06 O 0.04 0.11 0.02 0.04 O 0.06
Task
MOSEI A.T
MOSEI v.T
MOSEI V.A
Measure
R
U1 U2 S
U,
U2 S
U1 U2 S
BATCH
0.22
0.04 0.09 0.13 0.30
0.70 0 0 0.26
0.74 O 0
Task
MUSTARD A.T
MUSTARD V.T
MUSTARD V.A
Measure
R
U1 U2
S
R
U2
8
R
UI
U2 S
BATCH
0.14 0.01 0.01 0.37
0.14
0.02 0.01 0.34 0.14 0.01 0.01 0.20
for different tasks. MultiBench spans 10 diverse modalities (images, video, audio, text, time-series, various robotics sensors, sets, and tables), 15 prediction tasks (humor, sentiment, emotions, mortality rate, ICD-9 codes, image-captions, human activities, digits, robot pose, object pose, robot contact, and design interfaces), and 5 research areas (affective computing, healthcare, multimedia, robotics, and HCI). These datasets are designed to test combination of feature learning and arbitrary complex interactions between the modalities and the label in the real world. We also include experiments on guestion-answering (Visual Question Answering [5 39 and CLEVR (53) which test grounding of language into the visual domain.
For quantifying multimodal datasets with CVX, we first apply PCA to reduce the high dimension of multimodal data. For each of the train, validation, and test split, we then use unsupervised clustering generate 20 clusters. We obtain clustered version of the original dataset D {I) x2, y)} as Deluster = {(c,, C22 y)} where C; E {l,.... ,0} is the ID of the cluster that zy belongs to. For datasets with 3 modalities we estimate PID separately for each of the 3 modality pairs.
Details of applying CVX:
C2,
Details of applying BATCH: For quantifying multimodal datasets with BATCH,, we first use pretrained encoders to obtain the image and text features, and then train unimodal and I multimodal classifiers on those features to predict the original answer. The final PID values are computed using the encoded features and the 3 classifiers. In our experiments we used the pretrained image and text encoders from the CLIP-VIT-B/32 (82) model, and the unimodal and multimodal classifiers 2-layer MLPs.
Results on multimodal fusion: We show full results on quantifying datasets in Table 6 For multimodal fusion, we find that different datasets do require different interactions. Among interesting observations are that (1) all pairs of modalities on sarcasm detection shows high synergy values, which aligns with intuition on sarcasm in human communication, (2) uniqueness values are strongly correlated with unimodal performance (e.g., AV-MNIST and MIMIC first modality), (3) datasets with high synergy benefit from interaction modeling (e.g., sarcasm), and (4) conversely datasets with low synergy are also those where modeling in higher-order interactions do not help (e.g., MIMIC).
To test grounding of language into the visual domain, we also run experiments on the VQA v2.0 5 39 dataset, which contains 443757 training and 214354 validation data with real-life images, and the CLEVR [53] dataset, which contains 699960 training and 149991 validation data with rendered images and generated questions. The test spilt for both datasets are not used because the labels are not publicly available. Note that for both validation datasets, we first filter out all data instances where the label is neither "yes" nor "no". On the remaining data, we use the outputs from the last feature layer of a VGG19 [87] model as image embeddings and use RoBERTa (4 to obtain the text embeddings for questions, and then apply K-means clustering to both embeddings to obtain clustered version
26

---

Table 7: Estimating PID on QA [5] datasets with both CVX and BATCH estimators. Synergy is consistently the highest and both estimators return PID values that are consistent relative to each other.
Task
VQA
2.0
CLEVR
Mcasure
R
U1
U2
R
U1
U2
S
CVX
0.79
0.87
0
4.92
0.55
0.48
5.16
BATCH
0.10
0.14
0.09
0.16
0.01
0.01
0.01
0.03
Human
0.23
0
0
4.80
0
0
0
5.00
Inoi actionn
4. The extent to which both me medaaliea prediction about the task that you make
of all
provide your cating following questions
both modalitica enable you make the some task..
2. The
modilty enabiis you make a prediction about
which medallty snables to make prediction about
your
your from
confidence)-
not otherwies
not using modally
not if uuuiing modallty 1.
modality indivisually.
Modsiity
Madiblty
Rasing
Canfilennc
Rating
Confinence
Railin
Corfidanca
Rating
Confidencc
deal dAcc tains Cht erone pas nopking they
Patwen
35000
Suddd loo uns Lie paid hou Ban Barnn mean wage, WAN
Figure 16: We show sample user interface for annotating information decomposition into redun- dancy, uniqueness, and synergy. which we give to 3 annotators each to obtain agreeing and reliable judgments of each interaction for real-world multimodal datasets.
of the original dataset D {(11, T2. y)} as Deluster = {(c1, C22 y)}. where o; E (1, are the cluster IDs. We then calculate the PID values on the clustered dataset Deluster.
Results on QA: As a whole, we observe consistently high synergy values on QA datasets as shown in Table This is consistent with prior work studying how these datasets were balanced (e.g.. VQA 2.0 having different images for the same question such that both yes/no answers are observed in the dataset that the answer can only be obtained through synergy) (39] and models trained on these datasets [46].
We carefully compare CVX and BATCH on the question answering datasets VQA and CLEVR, showing these results in Table 7 We choose these two datasets since their input modalities are images and text, which make them suitable for both clustering of representations extracted from pre-trained before input into CVX, and end-to-ent encoding and PID estimation with BATCH. We also include human judgment of PID on these datasets for comparison. From Table 7 we find that all three measures agree on the largest interaction, as well as the relative order of interaction values. We find that the absolute values of each interaction can be quite different due to the unavoidable information loss of both input modalities to the label during clustering in CVX or representation learning in BATCH. Hence, we emphasize that these estimated interactions are most useful through relative comparison to judge which interaction is most dominant in particular dataset.
Comparing CVX and BATCH:
C.5 Comparisons with Human Judgment
To quantify information decomposition for real-world tasks, we investigate whether human judgment can be used reliable estimator. We propose new annotation scheme where show both modalities and the label and ask each annotator to annotate the degree of redundancy, uniqueness, and synergy on a scale of 0-5 using the following definitions inspired by the formal definitions in information decomposition:
R: The extent which using the modalities individually and together gives the same predictions on the task,
U1: The extent which ω enables you make a prediction about the task that you would not if using I2,
U2: The extent which 22 enables you to make a prediction about the task that you would not
using I1,
S: The extent to which only both modalities enable you to make a prediction about the task that you would not otherwise make using either modality individually.
alongside their confidence their answers on a scale of 0-5. We show a sample user interface for the annotations in Figure 16
27

---

Table 8: Collection of datasets used for study of multimodal fusion interactions covering diverse modalities, tasks, and research areas in multimedia and affective computing.
Datasets MUSTARD 17
Modalities
Size
Prediction task
Research Area
[text, video, )
690
sarcasm humor
A Affective Computing
UR-FUNNY
[text, video, }
16, 514
Affective Computing
MOSEI 116
{text, video, audio)
22, 777
sentiment, emotions Affective Computing
CLEVR 5
(image, question}
853, 554
QA
Multimedia
VQA 2.0 39
(image, question)
100, 000
QA
Multimedia
Table 9: Annotator agreement and average confidence scores of human annotations of PID values for real-world multimodal datasets.
Task
Agreement
Confidence
Mcasure
U1
U2
U1
Human
0.72 0.68
0.70
0.72
4.36
4.35
4.27
4.46
Based on direct communication with our institution's IRB office, this line of user-study research is aligned with similar annotation studies at our institution that are exempt from IRB. The information obtained during our study is recorded in such a manner that the identity of the human subjects cannot readily be ascertained, directly or through identifiers linked the subjects. We do not collect any identifiable information from annotators. Participation in all annotations was fully voluntary and we obtained consent from all participants prior to annotations. Participants were not the authors nor in the same research groups as the authors. They all hold or are working towards a graduate degree in a STEM field and have knowledge of machine learning. None of the participants knew about this project before their session and each participant only interacted with the setting they were involved in.
We sample 10 datapoints with class balance from each of the 5 datasets in Table 8 and give them to 3 annotators. All the data were manually selected and do not contain offensive content. We clarified all the confusion that the annotators may have about the instructions and the definitions above at the beginning the annotation process. We summarize the results and key findings:
1. General observations interactions, agreement, and confidence: The annotated in- teractions align with prior intuitions on these multimodal datasets and do indeed explain the interactions between modalities. For example, VQA and CLEVR are annotated with significantly high synergy, and language is annotated as the dominant modality in sentiment, humor, and sareasm (high U1 values). Overall, Table Isooss that the Krippendorff's alpha for inter-annon annotator agreement in human annotations of the interactions is quite high and the average confidence scores are also quite high. This indicates that the human- -annttttd results are reliable.
2. Annotator subjectivity in interpreting presence vs absence of attribute and con- textualizing both language and video: We observe that there is some confusion between uniqueness in the language modality and synergy in the video datasets, resulting in cases low agreement in annotating U, and S: -0.09, 0.07 for MOSEI, -0.14, -0.03 for UR-FUNNY and -0.08, -0.04 for MUSTARD respectively. We believe this is due to annotator subjectivity in interpreting whether sentiment, humor, and sarcasm are present only in the language modality (Ui) present when contextualizing both language and video (S). We further investigated the agreement and confidence in the presence or absence of an attribute (c.g., humor or sarcasm). We examined videos that show and do not show an attribute separately and indeed found in general, humans reached higher agreement on annotating attribute- presset videos. The agreement of annotating 0.13 when the attribute is present, compared to 10.10 when absent.
C.6 Comparisons with Other Interaction Measures
It is worth noting that other valid information-theorre definitions of multimodal interactions also exist, but are known to suffer from issues regarding over- and under-estimation, and may even be negative [1]; these are critical problems with the application of information theory for shared
28

---

Table 10: Estimating PID on synthetic generative model datasets. Ground truth total information is computed based on an upper bound from the best multimodal test accuracy. Both CVX and BATCH measures agree with each other on relative values and are consistent with ground truth interactions. We also implement 3 other definitions: I-min can sometimes overestimate synergy and uniqueness: WMS is actually synergy minus redundancy, can he negative and when R & S are of equal magnitude WMS cancels out to be 0; CI can also he negative and sometimes incorrectly concludes highest uniqueness for S-only data.
Task
R-only data
U U--only data
Uz-only data
S-only
data
PID
R
U1
U2
s
R
U1
U2
R
U1
U2
5
R
U1
U2
S
T-MIN
0.17
0.08
0.07
0
0.23
0.06
0
0
0.25
0.08
0.07
0.03
0.04
0.17
WMS
0
0.20
0.20
-0.11
0
0.25
0.02
0.05
0
0.03
0.27
0.05
0
0.14
0.15
0.07
CI
0.34
-0.09
-0.10
0.17
0 0.23
0
0.06
0 0.01
0.25
0.07
-0.02
0.13
0.14
0.08
CVX
0.16
0.05
0.16
0.05
0
0.17
0.05
0.07
0
0.01
0.14
BATCH
0.29
0.02
0.02
0
0.30
0
0.30
0
0.11
0.02
0.02
0.15
Truth
0.58
0
0
0
0
0.56
0
0
0
0
0.54
0
0
0
0.56 0
1(11; Y) and unique information / [(X;; I(X2; () often quoted in the co-training 14 g and multi-view learning [98 94 89 literature. We additionally compare PID with other previously used measures for feature interactions on the synthetic generative model datasets. Both CVX and BATCH measures agree with each other on relative values and are consistent with ground truth interactions. We also implement 3 other definitions: I-min can sometimes overestimate synergy and uniqueness; WMS is actually synergy minus redundancy, so can be negative and when & are of equal magnitude WMS cancels out to be O: CI can also be negative and sometimes incorrectly concludes highest uniqueness for S-only data. These results are in Table TO We also tried 3 non-info theory measures: Shapley values, Integrated gradients (IG), and CCA, which are based on quantifying interactions captured by a multimodal model. Our work is fundamentally different in that interactions are properties of data before training any models.
C.7 Quantifying Model Predictions
Setup: For each dataset, we first train a suite of models on the train set Duain and apply it to the validation set Deat, yielding a predicted dataset Dpeed { I2, 9) ) Running PID on Dped summarizes the interactions that the model captures. We categorize and implement a comprehensive suite of models (spanning representation fusion at different feature levels, types of interaction inductive biases, and training objectives) that have been previously motivated in the literature capture certain types of interactions (62] In the following, we use x; to denote data or extracted features from unimodal models where appropriate. We include all implementation details
and hyperparameters in Table I1
12
13 14
15
General: We first implement the most general model, which we call early fusion:
1. EF: Concatenating data at the earliest input level, essentially treating it as a single modality, and defining suitable prediction model y f(xx1, x2)) (62.
Redundancy: Fusion at the decision level (i.e., lute fusion)) is often motivated to capture redundancy between unimodal predictors.
1. ADDITIVE: Suitable unimodal models are first applied to each modality before aggregating the outputs using an additive average: y = 1/2(f( + fx()))
2. AGREE: Prediction agreement (+A(fi - f² 26 can be added for redundancy.
3. ALIGN: Feature alignment (+>Asimxx1,2 X2) like contrastive learning
82 can also be added
for redundancy.
researchers have long sought to learn higher-order interactions across modalities to capture synergistic interactions. We implement the following models:
Synergy Exemplified by methods such as bilinear pooling 33
and tensor fusion 115
1. ELEM: For static interactions (i.c., without trainable interaction parameters), we implement
element-wise interactions: = f(x C x2) 2 50
2. TENSOR: We also implement outer-product interactions (i.e., higher-order tensors) y =
f (xxx²) [B 115
47 58 65).
29

---

3. MI: Dynamic interactions with learnable weights include multiplicative interactions W:
y f(x, Wx2) 52
4. MUI.T: Another example of dynamic interactions with learnable weights is Crossmodal self-attention typically used in multimodal Transformers: y ((softmaxxxxx)) )xx) [003
99 113..
Unique: There have been relatively fewer methods that explicitly aim to capture uniqueness, since label information is often assumed to be shared across modalities (i.e., redundancy) [98] 94], two that we implement are:
1. LOwER: Some approaches explicitly include lower-order terms in higher-order interactions to capture unique information [65] [13].
2. REC:
Finally, several methods include reconstruction objectives encourage maxi-
mization of unique information (i.e., adding an objective Liee ll (7mm) - X1ll2
1192 (mm X2 where I1, 92 are auxiliary decoders mapping mm to each raw input
modality 992 100 [II].
Results: We show the full results estimating PID for all models trained on all datasets in Table 18 To better understand the models' ability in capturing each type of interaction, we present a concise overview of the full results in Table 4 focusing the PID values of each type of interaction / for each model on /-specialized dataset Dj. We also compute the distribution of Ip, over all models. We highlight the following observations:
General observations: We first observe that model PID measures are consistently higher than dataset measures. The sum of model measures is also good indicator of test performance, which agrees with their formal definition since their sum equal I([X,, 32 Y), the total explained mutual information between multimodal data and Y.
On redundancy: Overall, several methods succeed in capturing redundancy, with an overall average of 0.41 + 0.11 and accuracy of 73.0 + 2.0% on redundancy-specialll datasets. Additive, agreement, and alignment- biignment-based methods are particularly strong, but other methods based on tensor fusion (synergy-basee based), including lower-order interactions, and adding reconstruction objectives (unique- based) also capture redundancy well.
On uniqueness: Uniqueness is harder to capture than redundancy, with an average of 0.37 0.14. Certain methods like additive and agreement do poorly uniqueness, while those designed for uniqueness (lower-orde interactions and reconstruction objectives) do well, with 0.55 and 73.0% accuracy on uniqueness datasets.
On synergy: On average, synergy the hardest to capture, with an average score of only 0.21 0.10.
The best-performing methods are tensor fusion S 0.33, acc 74.0% and multimodal transformer = 0.29, acc 73.0%. Additive, agreement, and clement-wise interactions do not seem to capture synergy well.
Finally, we also show empirical connections between estimated PID values with model performance in the presence of noisy or missing modalities. Specifically, with the original performance Acc on perfect bimodal data, we define the performance drop when X, is missing as Acc_, Ace Acc; where Acc; is the unimodal performance when X; is missing. We find high correlation (p = 0.8) between Acc.: and the model's U, value. Inspecting the graph closely in F we find that the correlation is not perfect because the implication only holds in one direction: high U. coincides with large performance drops, but low U, can also lead to performance drops. We find that the latter can be further explained by the presence of large S values: when X, is missing, these interactions can no longer be discovered by the model which decreases robustness. For the subset of points when U, < 0.05, the correlation between S and performance drop is p 0.73 while the correlation between R and performance drop is only = 0.01.
robustness:
30

---

Table 11: Table of hyperparameters for prediction on synthetic data.
Component
Model
Parameter
Value
Identity
Encoder
Input size
[200, 200
Linear
Hidden dim
512
Input dim
[200, 200]
Decoder
Linear
Hidden dim
512
Concat
With Linear
Output dim
512
MI-Matrix 52
Fusion
Output dim
512
LRTF 65
rank
32
Embed dim
512
MULT
Num heads
8
Identity
Hidden size
512
Classification head
2-Layer MLP
Activation
LeakyRel.U(0.2)
Dropout
0.1
Loss
Cross Entropy 128
EF ADDITIVE & ELEM & TENSOR MI & MULT & LOWER
Batch size
Num epochs
100
Optimizer/I Leanning rate
Adam/0.0001
Cross Entropy
Loss
+
Agree/Align Weight 128
Batch size
AGREE & ALIGN
Num epochs
200
Optimizer/Leamming rate
Adam/0.0001
Cross Entropy Weight
2.0
Training
Agree/Align Weight
1.0
Cross Entropy
Loss
Reconstruction (MSE)
Batch size
128
Num epochs
100
Optimizer
Adam
REC 10D
Learning rate
0.0001
Recon Loss Modality Weight
[1,
Cross Entropy Weight
2.0
MLP (512, 256, 256)
Intermediate Modules
MLP 512, 256, 256)
31

---

Table 12: Table of hyperparameters for prediction affective computing datasets.
Component
Model
Parameter
Value
Identity
Encoder
Input size
5, 20, 35, 300, 704
GRU
Hidden dim
32, 64, 128, 512, 1024
Input size
5,20, 35, 300, 704
Decoder
GRU
Hidden dim
[32, 64, 128, 1024
Concat
ElemMulWithhiing
Output dim
[400, 512]
MI-Matrix (22)
Fusion
Tensor Fusion 115
Output dim
512
Embed dim
40
MULT
99
Num heads
8
Idnntity
/
Hidden size
512
Classification head
2-Layer MLP
Activation
LeakyReLUUO.2))
Dropout
0.1
Loss
LI Loss
EF & ADDITIVE & ELEM & TENSOR MI MULT & LOWER
Batch size
32
Num epochs
40
Optimizer/l.camingg rate
Adam/0.0001
I Loss
Loss
+ Agree/Align Weight
Batch size
32
AGREE ALIGN
Num epochs
30
Optimizer/L Leamnin rate Agreo/Align Weight
Adam/0.0001
0.1
Training
Li Loss
Loss
Reconstruction (MSE)
Batch size
128
Num epochs
50
Optimizer
Adam
REC 100
Learning rate
0.001
Recon Loss Modality Weight
[1, 1]
MLP [600, 300, 300 ML.P (600, 300, 300)
Intermediate Modules
Table 13: Table of hyperparameters for prediction on ENRICO dataset in the HCI domain.
Model Unimodal
Parameter
Value
Hidden dim
16
Hidden dim
32
MI-Matrix
Input dims
16, 16
Hidden dim
MI
Input dims
16, 16
Hidden dim
32
LRTF
65
Input dims
16, 16
Rank
Loss
Class-weighted Cross Entropy
Batch size
32
Activation Dropout
ReLUU
Training
Optimizer
Adam
Learning Rate
10
Num epochs
30
32

---

Table 14: Table of hyperparameters for prediction on MIMIC dataset in the healthcare domain.
Component
Model
Parameter
Value
Hidden sizes
[10, 10]
Stutic Encoder
2-layer MLP
Activation
LeakyRel.U(0.2))
Layer sizes
[200, 40, 5]
Static Decoder
2-layer MLP
Activation
LeakyRel.U(0.2)
Time Series Encoder
GRU
Hidden dim
Time Series Decoder
GRU
Hidden dim
30
Hidden size
40
Clussification Head
2-yer MLP
Activation
LeakyReLUU(.22)
Output dim
100
LRTF 65
Fusion
Ranks
40
MI-Matrix 52
output dim
100
Loss
Cross Entropy
Batch size
40
TENSOR, MI
Num epochs
20
Optimizer
RMSprop
Learning rate
0.001
Cross Fntropy
Loss
+ Reconsttructinn(MSS
Batch
Num epochs
30
REC 100
Optimizer
Adam
Learning Rate
0.001
Recon Loss Modality Weights
[1, 1] 2.0
Training
Cross Entropy Weight
MLPs [200, 100, 100),
Intermediate Modules
[200, 100, 100), [400, 100, 100
33

---

Table 15: Table of hyperparameterss for prediction on MAPS.
Component
Model
Parameter
Value
Identity
Encoder
Input size
[1000, 18, 137]
Linear
Hidden dim
512
Input dim
[1000, 18, 137]
Decoder
Linear
Hidden dim
512
Concat
ElemMultWithLineaa
Output dim
512
MI-Matrix [52)
Fusion
Output dim
5T2
LRTF
rank
32
Embed dim
512
MULT 99
Num heads
8
Identity
Hidden size
512
Classification head
2-Layer MLP
Activation
LeakyReLUUO..)
Dropout
0.1
Loss
Cross Entropy 128
EF & ADDITIVE & ELEM & TENSOR MI MULT & LOWER
Batch size
Num epochs
100
Optimizer/Learmin rate
Adam/0.0001
Cross Entropy
Loss
+
Agree/Align Weight
Batch size
128
AGREE ALIGN
Num epochs
200
Optimizer/I cearning rate
Adam/0.0001
Cross Entropy Weight
2.0
Training
Agree/Align Weight
1.0
Cross Entropy
Loss
Reconstruction (MSE)
Batch size
128
Num epochs
100
Optimizer
Adam
REC 100
Learning rate
0.0001
Recon Loss Modality Weight
[1,
Cross Entropy Weight
2.0
MLP [512, 256, 256] MLP 512, 256, 256
Intermediate Modules
Table 16: Table of hyperparameters for prediction on Push dataset.
Component
Model
Parameter
Value
Pos Encoder
Linear
Hidden sizes
(64, 64, 64 (residual)
Sensors Encoder
Linear
Hidden sizes
(64, 64, 64 (residual))
Filter sizes
3,,3,3,3,3
Num filters
[32,32, 32, 16,
Image Encoder
CNN
Filter strides
Filter padding
[2, 1,1,1,11
Control Encoder
Linear
Hidden sizes
[64, 64, 64 (residual))
Hidden size
512
Unimodal LSTM
Num layers
Fusion
Hidden size
256
Late Fusion LSTM
Num layers
Classification Head
Linear
Hidden size
64
Loss
Mean Squared Error
Batch size
32
Num epochs
20
Training
Activation
ReLU
Optimizer
Adam
Learning rate
10-5
34

---

Table 17: Table of hyperparameters for prediction on AV-MNIST dataset.
Component
Model
Parameter
Value
Filter Sizes
3,333,
Num Filters
[6, 12, 48]
Image Encoder
LeNet-3
Filter Strides Filter Paddings
[1, 1, 1, 1] /22, 1,
Max Pooling
[2 2, 2, 2]
Filter Sizes
[4, 4,
Image Decoder
DeLeNet-3
Num Filters
[24, 12, 6,
Filter Strides Filter Paddings
[2, 2, 4)/11 1,,
Filter Sizes
[5,3,3,3,3,3
Num Filters
[6, 12, 28,448 96, 192]
Audio Encoder
LeNet-5
Filter Strides Filter Paddings
[,, 1, 1, 1,111
Max Pooling
2,
Filter Sizes
4,
Audio Decoder
DeL.NNet55
Num Filters
[96, 48, 24, 12,
Filter Strides Filter Paddings
22, 2,2, 2.2
Hidden size
100
Classification Head
2-Layerr MIP
Activation
LeakyRel.U(.22
Cross Entropy 40
Batch size
Training
Unimodal, LF
Num epuchs
25
Optimizer/Leamming rate/weight decay
SGD/O.05/0.00 0001
35

---

Table 18: Results on estimating PID model families trained on synthetic datasets with controlled interactions.
Category
GENERAL
REDUNDANCY
Model
EF
ADDITIVE
AGREE
Measure
R
U1
U2
Ace
R
U1
U2
S
Acc
R
U12
U2
S
Acc
Ze
0.47
0.01
0.01
0.04
0.74
0.35
0.2
0
0.04
0.71
0.48
0.01
0.01
0.06
0.74
(2)
0.41
0
0.04
0.02
0.14
0.02
0.15
0.04
0.38
(22)
0.01
0
0.47
0.04
0.72
0.05
0
0.43
0.06
0.72
0.01
0.48
0.04
0.72
(21,
0.15
0.02
0
0.29
0.72
0.15
0
0.2
0.13
0.56
0.14
0
0.26
0.09
0.66
(2T, 22,
0.22
0
0.07
0.15
0.64
0.31
0.16
0
0.08
0.64
0.21
0.11
0.15
0.58
(21, 22, Re
0.16
0.15
0.15
0.71
0.06
0.18
0.06
0.6
0.06
0.08
0.53
(21, 22,
0.16
0.05
0
0.25
0.74
0.09
0
0.23
0.11
0.52
0.02
0.33
0.06
0.62
2i zá, ze)
0.25
0.03
0
0.14
0.66
0.39
0.11
0
0.08
0.67
0.44
0.04
0.09
0.67
22, Re
0.19
0
0.12
0.06
0.65
0.15
0
0.07
0.18
0.61
0.13
0
0.24
0.01
0.64
(22 Re)
0.26
0
0.12
0.05
0.7
0.27
0
0.18
0.08
0.71
0.1
0.1
0.16
0.61
Category Model
REDUNDANCY
SYNERGY
ALIGN
ELEM
TENSOR
Mcasure
R
U1
U2
S
Acc
H
U1
U2
5
Acc
R
U1
U2
S
Acc
Ze
0.44
0.02
0
0.05
0.73
0.27
0
0.01
0.07
0.7
0.47
0.01
0.01
0.04
0.74
21
0
0.05
0.2
0.66
0
0.56
0.02
0.74
(22)
0.05
0
0.39
0.1
0.71
0.01
0
0.2
0.05
0.66
0.01
0
0.54
0.02
0.73
(21,
0.02
0
0.38
0.08
0.63
0.07
0.02
0
0.14
0.66
0.15
0.03
0.31
0.73
(2i 22
0.33
0.06
0
0.15
0.66
0.05
0
0.01
0.07
0.56
0.2
0.01
0.01
0.21
0.65
(21, 22 2e
0.05
0.14
0.04
0.05
0.09
0.24
0.13
0.71
(21, 22, 22)
0
0.26
0.04
0.64
0.08
0.02
0.12
0.66
0.17
0.06
0.27
0.72
[ii 22, zc)
0.23
0
0.09
0.08
0.64
0.1
0
0.01
0.06
0.6
0.27
0.05
0.02
0.15
0.68
22,, Re)
0.18
0.1
0
0.17
0.6
0.04
0
0.03
0.05
0.57
0.05
0.01
0.06
0.56
Ze)
0.16
0
0.2
0.06
0.69
0.15
0.03
0.06
0.65
0.14
0.03
0.05
0.65
Category Model
SYNERGY
M1
MULT
Measure
R
U1
U2
S
Acc
R
U1
U2
S
Acc
(2c)
0.2
0
0.01
0.05
0.67
0.4
0.02
0.06
0.73
[21)
0.13
0.05
0.66
0
0.48
0.02
0.73
(22)
0.01
0
0.23
0.05
0.66
0.02
0
0.42
0.06
0.71
(21, 22)
0.05
0.04
0
0.12
0.65
0.16
0.05
0.29
0.72
(2f, 22,
0.07 0.01
0.01
0.08
0.55
0.21
0.11
0.2
0.66
(21: 22, 22,
0.02 0.01
0.06
0.58
0.1
0.29
0.11
0.7
(21, 22, 22)
0.04 0.01
0
0.08
0.62
0.18
0.02
0.3
0.72
(2f, 22, 2c)
0.12
0.01
0.01
0.06
0.62
0.33
0
0.03
0.14
0.68
=2)
0.02
0
0.01
0.06
0.54
0.13
0
0.39
0.04
0.67
Rc)
0.13
0
0.01
0.06
0.64
0.36
0.18
0.04
0.72
Category Model
UNIQUE
Lower
REC
Measure
R
U1
U2
S
Acc
R
U1
U2
S
Acc
0.53
0
0.01
0.03
0.75
0.55
0.02
0.01
0.02
0.75
0.56
0
0.02
0.74
0
0.53
0.08
0.74
[22]
0.01
0
0.54
0.02
0.72
0.01
0
0.55
0.02
0.73
(21, 22)
0.15
0.03
0
0.32
0.74
0.14
0.06
0.34
0.74
(2f.
0.21
0.01
0.01
0.2
0.65
0.19
0.03
0.26
0.66
(21 22,
0.09
0.27
0.13
0.71
0.08
0.29
0.16 0.71
(21 22, Kn
0.16
0.06
0
0.27
0.73
0.21
0.05
0.31
0.72
(2f 22)
0.31
0.01
0.01
0.16
0.68
0.32
0
0.06
0.21
0.68
(2) =c)
0.13
0
0.29
0.04
0.64
0.19
0
0.36
0.04
0.66
Rc)
0.31
0
0.13
0.04
0.7
0.42
0.15
0.03
0.72
36

---

Table 19: Results estimating PID model families trained real-world multimodal datasets.
Category
GENERAL
REDUNDANCY
Model
EF
ADDITIVE
AGREE
Measure
U1
U2
S
Acc
U1
U2
S
Acc
R
U1
U2
Acc
UR-FUNNY A,T
0.02
0.02
0.06
0.59
0.02
0
0.02
0.15
0.60
0.02
0
0.02
0.05
0.59
FUNNYNYY V.T
0.05
0.01
0.17
0.62
0.11
0.08
0
0.17
0.60
0.02
0.04
0
0.24
0.59
UR-FUNNY V.A
0.01
0.27
0.19
0.59
0.02
0.15
0.56
0.02
0.02
0.15
0.53
MOSEI A.T
0.16
0.27
0.03
0
0.79
0.15
0.19
0.02
0.13
0.80
0.14
0.11
0.05
0.06
0.80
MOSEI V,T
0.01
0
0
0.63
0.11
1.21
0
0
0.80
0.1
0.84
0
0.80
MOSEI V.A
0
0.01
0.63
0.59
0.62
0
0.12
0.65
0.10
1.46
0
0.24
0.61
MUSTARD A.T
0.06
0.03
0.04
0.42
0.17
0.04
0.33
0.70
0.18
0
0.05
0.31
0.68
MUSTARD V.T
0
0.57
0.14
0.07
0.33
0.69
0.19
0
0.07
0.27
MUSTARD V,A
0
0
0
0.57
0.25
0.12
0
0.15
0.61
0.33
0.10
0
0.16
0.58
MIMIC
0.01
0.27
0
0
0.91
0.01
0.27
0
0
0.92
0.01
0.27
0
0.92
ENRICO
0.71
0.34
0.44
0.38
0.50
0.69
0.28
0.39
0.35
0.30
0.40
0.30
0.52
0.75
0.51
Category
REDUNDANCY
SYNERGY
Model
ALIGN
ELEM
TENSOR
Mcasure
R
U1
U2
S
Acc
R
U1
U2
S
Acc
R
U1
U2
S
Acc
UR-FUNNY
0.02
0.02
0.06
0.58
0.01
0.03
0.05
0.64
0.01
0
0.02
0.05
0.62
UR-FUNNY V.T
0.02
0.02
0.15
0.60
0.16
0.61
0.03
0.01
0.14
UR-FUNNY V.A
0.12
0.05
0.01
0.19
0.53
0.03
0.13
0
0.16
0.59
0.02
0.15
0
0.15
0.60
MOSEI A,T
0.12 0.26
0.14
0.16
0.65
0.09
1.07
0
0
0.80
0.18
0.17
0.06
0
0.80
MOSEI V.T
0.21 0.61
0.09
0.65
0.25
0.92
0
0.15
0.81
0.12
1.23
0
0.81
MOSEI V.A
1.15
0.79
0.40
0.65
0.12
1.07
0.65
MUSTARD A.T
0.20
0.06
0.29
0.70
0.17
0.01
0.11
0.23
0.60
0.20
0
0.13
0.25
MUSTARD V.T
0.16
0.08
0.29
0.70
0.18
0.01
0.04
0.32
0.64
0.18
0.01
0.02
0.34
0.59
MUSTARD V.A
0.27
0.12
0
0.14
0.62
0.36
0.07
0
0.16
0.55
0.33
0.05
0
0.18
0.60
MIMIC
0.01
0.28
0
0.91
0.04
0.24
0
0.01
0.91
0.28
0
0.01
0.91
ENRICO
0.37
0.76
0.52
0.43
0.29
0.73
0.44
0.38
0.48
0.32
0.50
Category Model
SYNERGY
UNIQUE LOWER
MI
Mcasure
U1
U2
S
Acc
R
U1
U2
S
Acc
UR-FUNNY A.T
0.01
0.03
0.04
0.62
0.02
0.01
0.14
0.60
UR- FUNNY V.T
0.04
0.02
0
0.14
0.64
0.04
0
0.01
0.15
0.62
UR-FUNNY V.A
0.02
0.23
0
0.15
0.61
0.02
0.15
0
0.19
0.58
MOSEI A.T
0.13 0.05
0.01
0.19
0.81
0.20
0.92
0
0.54
0.79
MOSEI V,T
0.11
0.98
0.80
0.17
1.14
0
0.08
0.80
MOSEI V,A
0.12 1.0
0
0
0.65
1.54
0.64
0
0.13
0.65
MUSTARD A.T
0.18 0
0.08
0.29
0.63
0.21
0
0.05
0.26
0.59
MUSTARD V.T
0.20 0.01
0.03
0.25
0.63
0.17
0
0.06
0.28
0.58
TUSI V.A
0.28
0.04
0.02
0.18
0.56
0.24
0.08
0
0.24
0.59
MIMIC
0.26
0.01
0.92
0.28
0.01
ENRICO
0.28 0.41
0.34
0.48
0.38
0.52
0.29
0.69
0.52
0.56
37

---

1. Datnset quantfiicttoo:
2. Model quantification
3. Model selection:
D- ((o)))0.)))
((P) (o... = +s)))
U1. po
(R. U5, (R.U3,0,.511
(M,li, Nyypssp (R.61, Mpm.)
(R.00, Ugr
Interacionn polytope
Interaction polytope
Internation polytope
(1.0,0.0)
1,000
(0.1. (.800.8
(0.1.0:7 0.2003)
Mocal "amilies trained syntheiic dats
Selects models with performance
(0,
0)
Figure 17: Model selection pipeline: (1) quantifying the interactions in different datasets, (2) quantifying interactions captured by different model families, then (3) selecting the closest model for given new dataset yields models of > 96% performance as compared to exhaustively trying all models.
C.8 Multimodal Model Selection
Setup: We show that our findings on quantifying multimodal datasets and model predictions are generalizable, and that the PID estimations informative for model selection new datasets without training all models from scratch. We simulate the model selection process on new synthetic datasets and 6 real- world multimodal datasets. Given new dataset D, we compute their PID values To and normalize it to obtain Ip:
TD
iD EVERRU, U...)
(35)
We select the most similar dataset D* from our base datasets (the 10 synthetic datasets presented in Table 2 that has the smallest difference measure
sD, D')=
Σ
11D ID.
(36)
which adds up the absolute difference between normalized PID) values on D and D'. The model selection pipeline is illustrated in Figure 17 We hypothesize that the top-3 best-performing models on D should also achieve decent performance on D due to their similar distribution of interactions. We verify the quality of the model selection by computing percentage of the performance of the selected model f with respect to the performance of the actual best-performing model f* on D, i.e. % Performance(f,
We summarize our evaluation of the model selection in Table 5 and find that the top 3 chosen models all achieve 95% 100% of the best-performing model accuracy, and above 98.5% for all datasets except MUSTARD. For example. UR-FUNNY Y and MUSTARD have the highest synergy (S 0.13, S 0.3) and indeed transformers and higher-order interactions are helpful (MUI.T: 0.65%, MI: 0.61 %. TENSOR: 0.6%). ENRICO has the highest R 0.73 and U2- = 0.53, and indeed methods for redundant and unique interactions perform best (LoWwr:: 0.52%, ALIGN: 0.52%, AGREE: 0.51%). MIMIC has the highest U₁ = 0.17, and indeed unimodal models are mostly sufficient [60). We hypothesize that model selection more difficult on MUSTARD since it has the highest synergy ratio, and it is still an open question regarding how to capture synergistic multimodal interactions. Therefore, PID values provide a strong signal for multimodal model selection.
Results:
C.9 Real-world Case Study 1: Computational Pathology
Setup: The current standard of care for cancer prognosis in anatomic pathology is the integration of both diagnostic histological information (via whole-slide imaging (WSI)) as well as molecular information for patient stratifcation and therapeutic decision-making [63]. For cancer types such as diffuse gliomas, the combination of histological subtypes (astrocytoma, oligodendrrollioma) and molecular markers (IDHI mutation, 1p19q codeletion) determines whether patients should be administered tumor resection, radiation therapy, adjuvant chemotherapy, or combination therapies [66] Recent applications of multimodal learning in pathology have found that though fusion confers benefit for majority of cancer types, unimodal models can be as competitive in patient stratification performance, which suggest potential clinical application prognosticating using only
16 20
single modality for certain cancers (2
38

---

Table 20: Results on estimating PID across the GVGG-GBMLGG and TCGA-PAAD datasets. U1, U2 correspond to pathology and genomics respectively.
Dataset
TCGA-LGG
TCGA-PAAD
Measure
R
U1
U2
S
R
U1
U2
S
CVX
0
0.02
0.06
0.02
0
0.06
0.08
0.15
Using The Cancer Genome Atlas (TCGA), PID used to quantify pathology image-omic inter- actions in downstream prognostication tasks for two cancer datasets: lower-grade glioma (TCGA- LGG [15] (n = 479) and pancreatic adenocarcinoma (TCGA-PAAD 3 (n 209). As mentioned in the main text, the modalities include: (1) sequence of pre-extracted histology image features from diagnostic WSIs (N x 1024, where N is the number of non-overlapping 256 x 256 patches at 20x magnification and 1024 is the extracted feature dimension of a pretrained vision encoder), and (2) feature vector (1 x D) of D bulk gene mutation status, copy number variation, and RNA-Scu -Seq abundance values. The label space is overall survival time converted to 4 discrete bins. Following the study design of (22). we trained an Attention-Baan Based Multiple Instance Learning (ABMIL) model and a Self- Normalzzzai Network (SNN) for the pathology and genomic modalities using a log likelihood loss for survival analysis respectively 48 (55) For ABMIL, a truncated ResNet-50 encoder pretrained on ImageNet was used for patch feature extraction of WSIs [5] Unimodal features were extracted before the classification layer, with K-means clustering (k 3) for each modality used to obtain (21, 22,22 y) pairs where Ii is pathology, 22 is genomics and y is the discrete bins, which can then be used CVX to compute PID measures.
We assess these PID measures (Table 20 in the context of current scientific understanding of pathology and genomic interactions in cancer prognostication in the literature. Specifically, compare against the findings [22]. which previously investigated unimodal and multimodal survival models in TCGA and observed that risk predictions were primarily driven by genomies across most cancer types (such as TCGA-LGG), with exceptions in a few cancer types such as TCGA- PAAD. In the evaluation of TCGA-LGG and TCGA-PAAD in [22. concordance Index performance of thimodal-patholoor 0.808) and (0.580, 0.593, 0.653) respectively. In (22), one of the original hypotheses for the lack of multimodal improvement for some cancer types was due to high mutual information between both modalities. In our results, we observe the opposite effect in which there is little mutual information between both modalities. This observation coupled with the low and high synergy values in TCGA- LGG (S 0.02) and TCGA-PAAD (S- 0.15) suggest that only TCGA-PAAD would benefit from multimodal integration, which was empirically found with both the c-Ineex improvement and statistical significant patient stratification results in TCGA-PAAS evaluation. The uniqueness values of both tasks also corroborate the high amount of task-specific information in genomics with high I-dxx performance, and vice versa with pathology. Overall, we demonstrate we can use PID measures to refine biomedical insights into understanding when multimodal integration confers benefit.
Results:
unimodal-genone genoomic, and multimodal survival models were (0.668, 0.792,
C.10 Real-world Case Study 2: Mood Prediction from Mobile Data
Setup: Suicide is the second leading cause of death among adolescents 18] Despite these alarming
statistics, there is little consensus concerning imminent risk for suicide (3] Intensive monitoring
of behaviors via adolescents' frequent use of smartphones may shed new light on the early risk of suicidal thoughts and ideations 37 72). Smartphones provide valuable and natural data source
with rich behavioral markers spanning online communication, keystroke patterns, and application usage (59] Learning these markers requires large datasets with diversity in participants, variety in features, and accuracy in annotations. As a step towards this goal, we partnered with several hospitals (approved by NIH IRB for central institution and secondary sites) to collect dataset of mobile behaviors from high-risk adolescent populations with consent from participating groups. This data monitors adolescents spanning (a) recent suicide attempters (past 6 months) with current suicidal ideation, (b) suicide ideators with no past suicide attempts, and (c) psychiatric controls with no history of suicide ideation or attempts. Passive sensing data is collected from each participant's is smartphone across a duration of 6 months. The modalities include (1) text entered by the user represented as bag of top 1000 words that contains the daily number of occurrences of each word, (2) keystroke features that record the exact timing and duration that each character was typed on mobile keyboard
39

---

Table 21: Results on estimating PID on the MAPS dataset with pairs of modalities text apps (MAPST,A) and text + keystrokes (MAPST.K).
Dataset
MAPST.A
MAPST.K
Measure
R
U1
U2
S
R
U1
U2
S
CVX
0.09
0
0.09
0.26
0.12
0
0.04
0.40
Table 22: Results on estimating PID on model families trained on the MAPS dataset.
Category
GENERAL
REDUNDANCY
Model
EF
ADDITIVE
AGREE
Mcasure
R U1
U2
Acc
R U1 U2
Acc
R U1
U2
Acc
MAPST.A
0.61 0.01
0.08 0.02
0.42
0.43
0.26
0
0.39
0.45
0.24
0
0.40
MAPST.K
0.16 0.29
0 0.08 0.51
0.09
0
0.18
0.21
0.47
0.20
0
0.24
0.17
0.42
Category
REDUNDANCY
SYNERGY
Model
ALIGN
ELEM
TENSOR
Mcasure
R U1 U2
Acc
R
U1
U2
5 Acc
U1
U2
S Acc
MAPST,A
0.69 0 0.11 0.03 0.39 0.42 0.04
0.05
0.13 0.41
0.42
0.10
0.03
0.11 0.45
MAPST.K
0.41
0.01
0.28
0.34
0.33
0.17 0.37
0.02
0.41
0.27
0.24 0.15
0
0.32
0.57
Category
SYNERGY
Model
MI
MULT
Measure
R
U1
U2
S
Acc
R
U1
U2
S Acc
MAPST,A
0.28 0.12
0.01
0.21
0.47
0.40
0.48 0.09
0
0.03
MAPST
0.25
0.24
0.02
0.40
0.36
0.10
0.27
0.19
0.50
Category
UNIQUE
Model
LOWER
REC
Mcasure
R U1
U2
S Acc
R
U2
S Acc
MAPST,A
0.53 0.04 0.04
0.09 0.41 0.28
0.06
0.07
0.10 0.56
MAPST,K
0.12
0.24
0.22
0.51
0.23
0.17
0 0.18 0.52
(including alphanumeric characters, special characters, spaces, backspace, enter, and autocorrect), and (3) mobile applications used per day, creating a bag of 137 apps for each day that are used by at least 10% of the participants.
Every day at Sam, users are asked to respond to the following question - "In general, how have you been feeling over the last day?" with an integer score between and 100, where means very negative and 100 means very positive. To construct our prediction task, we discretized these scores into the following three bins: negative follow a class distribution of 12. 22.3%, 43. 63%. and 43. 43.94% respectively. For our 3-way classification task, participants with fewer than 50 daily self-reports were removed since these participants do not provide enough data to train an effective model. In total, our dataset consists of 1641 samples, consisting of data coming from 17 unique participants.
33), neutral (34 66), and positive (67 100). which
Results: The results are shown in Table 21 22 We observe that both MAPST.A (text + apps) and MAPS (text keystrokes) have the highest synergy (S 0.26, S 0.4 respectively) and some redundancy (R = 0.09, R = 0.12) and uniqueness in the second modality (U, = 0.09, Un = 0.04). We find indeed the best-performing models are REC (ace = 0.56%) and MULT (acc 0.48%) for MAPSI,A, and TENSOR (ace = 0.57%) and REC (ace = 0.52%) for MAPS which are designed to capture synergy and uniqueness interactions. Our model selection also successfully chooses REC for MAPST.A and TENSOR for MAPST. x with high agreement of a = 0.66 and a 0.74 (98.3% and 99. 85% of the highest agreement respectively). This result further corroborates the utility and generalizability of our PID estimations and model selection.
40

---

Table 23: Results on estimating PID on robotic perception task.
Task
PUSH 1.C
Measure
R
U1
U2
S
CVX
0.24
0.03
0.06
0.04
BATCH
0.75
1.79
0.03
0.08
C.11 Real-world Case Study 3: Robotic Perception
Setup: MuJoCo PUSH [57] is a contact-riec rich planar pushing task in the MuJoCo simulation framework, in which 7-DoF Panda Franka robot is pushing circular puck with its end-effector in simulation. We estimate the 2D position of the unknown object on a table surface, while the robot intermittently interacts with the object. The final dataset consists of 1000 trajectories with 250 steps at 10Hertz, of simulated Franka Panda robot arm pushing a circular puck in MuJoCo 9 The pushing actions are generated by heuristic controller that tries to move the end-effector to the center of the object. The multimodal inputs are gray-scaled images (1 x 32 x 32) from an RGB camera, forces (and binary contact information) from force/torque sensor, and the 3D position of the robot end-effector. The task is to predict the 2-D planar object pose.
We use CNN feature extractors and LSTM as sequence model for both unimodal and multimodal models. We use hyperparameters from Table 16 Because both CVX and BATCH assumes discrete classes of labels, we discretize all continuous labels into bins of equal data points from the training set, and use the same bin cutoff's test time. The task is to predict the correct bin that contains the original label, and we use cross entropy loss as our loss function.
The results shown in Table 23 As can be seen from the table, BATCH predicts Up as the highest PID value (U,- = 1.79), which aligns with the fact that image is the best unimodal predictor for this dataset (60) Comparing both estimators, CVX underestimates Un and R since CVX clusters the data before processing, and the high-dimensional time-series modalities canmot be easily deseribed by clusters without losing information. In addition, both BATCH and CVX predict a low U2 value but attributes some mutual information to R (redundancy), implying that a multimodal model with both modalities would not be much better compared an unimodal model on modality 1, since the multimodal model would mostly get redundant information from modality 2. In our experiments, we observe no difference in performance between the multimodal model and the unimodal model with modality 1.
Results:
D Summary of Takeaway Messages and Future Work
From these results, we emphasize the main take-away messages and motivate several directions for future work:
1. Dataset quantification: PID provides reliable indicators for the nature and quantity of interactions present in both synthetic and real-world multimodal datasets, which provides a useful tool summarizing these datasets. In our opinion, creators of new datasets focused on modeling interactions between multiple features or modalities should report estimated PID values on new datasets alongside justification on whether these values are to be expected based on how the features were selected.
2. Model quantification: PID also provides reasonable indicators for the interactions captured by trained models, but naturally there is more noise due to different model parameter counts, training trajectories, and non-convex optimization artifacts that despite best efforts remain impossible to exactly control. Despite these details, we still find several consistent patterns in the types of interactions different models capture. We suggest that researchers developing new multimodal models explicitly state the assumptions on the underlying multimodal distribution their models require and the interactions their models are designed to capture, before testing exhaustively on datasets predominantly of those interactions.
3. Model selection: Our experiments on model selection for real-world datasets and applica- tions demonstrate potential utility in a rough 'user-guide" for practitioners aiming to tackle real-world multimodal datasets. Given new dataset, estimate PID values. If there high U1 and U2. just using unimodal models in the corresponding modality may be sufficient.
41

---

Otherwise. if there is high R. methods like agreement, alignment, ensemble, or co-training should be tried. If there high S, it is worth spending the time on multimodal methods that model more complex interactions based on tensors, multiplicative interactions, and self-attention.
We believe that the intersection of PID and machine learning opens the door to several exciting directions of future work:
1. Pointwise interaction measures: A natural extension is to design pointwise measures: how much does a single datapoint contribute to redundancy, uniqueness, or synergy in the context of an entire distribution? Pointwise measures could help for more fine-grained dataset and model quantification, including error analysis of incorrect predictions or active learning of difficult examples.
2. Representation learning for desired interactions: Can we design new model architectures or training objectives that better capture certain interactions? Current well-pergo models like Transformers already include some notion of multiplicative interactions, so it does seem that, heuristically, synergy between input features is playing role their success. Designing models that better capture synergy, as quantified by our PID estimators, could be a path towards learning better representations.
3. Principled approaches to fairness and invariance: Currently, PID is designed to measure the information that variables contribute towards task Y, but conversely it can also be used to remove information that one variable can have about Y, in the context of another variable. These could provide formal learning objectives for fairness, privacy, and other feature invariance tasks.
E
Limitations and Broader Impact
E.1 Limitations of our Estimators
While our estimators appear to perform well in practice, each suffers from distinct limitations. For CVX, is rarely able scale up to domains with more than several thousand classes. Furthermore, if [2] is large-which is frequently the case in real-world applications, then empirical frequencies will often be near O, causing much instability in conic solvers (see Appendix B.1 for more detailed description).
BATCH sidesteps many of the issues with CVX by applying batch gradient descent. However, it may suffer from approximation error depending on whether the network can sufficiently cover the space of joint distributions (i.e., representation capacity), alongside bias resulting from mini-batch gradient descent (as compared to full-batch gradient descent) when approximating q using bootstrapped samples. Lastly, it may suffer from the usual problems with neural estimators, such as local minima, poor or unstable convergence (in both learning q and p). Therefore, while BATCH scales to high- dimensional continuous distributions, it comes with the challenges involving training and tuning neural networks.
E.2 Broader Impact
Multimodal data and models are ubiquitous in range of real-world applications. Our proposed framework based on PID is our attempt to systematically quantify the plethora of datasets and models currently in use. While these contributions will accelerate research towards multimodal datasets and models as well as their real-world deployment, we believe that special care must be taken in the following regard to ensure that these models are safely deployed:
Care in interpreting PID values: are only an approximation the actual interactions and care should be taken to not overfit these values. Other appropriate forms of dataset visualization and quantification should still be conducted obtain holistic understanding of multimodal datasets.
Just like with any approximate estimator, the returned PID values
Privacy, security, and biases: There may be privacy risks associated with making predictions from multimodal data the datasets include recorded videos of humans or health indicators. In our experiments with real-world data where people are involved (i.e., healthcare and affective computing), the creators of these datasets have taken the appropriate steps to only access public data which participants/content creators have consented for released to the public. We also acknowledge the risk
42

---

of exposure bias due to imbalanced datasets that may cause biases towards certain genders, races, and demographic groups. Therefore, care should be taken in understanding the social risks of each multimodal dataset in parallel to understanding its interactions via PID.
Time & space complexity: Modern multimodal datasets and models, especially those pretrained on internet-scale data, may cause broader impacts resulting from the cost of hardware, electricity, and computation, as well as environmental impacts resulting from the carbon footprint required to fuel modern hardware. Future work should carefully investigate the role of size on the interactions learned by models through estimated PID values. Our preliminary experiments showed that smaller models could still capture high degrees of each interaction, which may pave away towards designing new inductive biases that enable interaction modeling while using fewer parameters.
Overall, PID offers opportunities to study the potential social and environmental issues in multimodal datasets by obtaining deeper understanding of the underlying feature interactions, providing path towards interpretable and lightweight models. We plan to continue expanding our understanding of PID via deeper engagement with domain experts and how they this framework in their work. Our released datasets, models, and code will also present step towards scalable quantification of feature interactions for future work.
43


Reviews:
[{"review_id": "VQ3WtplTcK", "reviewer": "Reviewer_oDWq", "summary": "This paper proposes an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. The authors introduce two new estimators for these PID statistics that scale to high-dimensional distributions. They validate the PID statistics on synthetic and real-world datasets, demonstrating their ability to identify meaningful interactions and select appropriate models. The authors also compare their approach to other methods for quantifying multimodal interactions, showing that their method outperforms these alternatives. They provide examples of real-world applications where the PID statistics were useful in quantifying interactions within multimodal datasets or selecting appropriate multimodal models. The authors discuss the limitations of their approach, including the need for large amounts of data and the difficulty of interpreting the results. They suggest future directions for research, such as extending the approach to continuous-valued data and incorporating causal relationships between modalities. Overall, this paper provides a useful framework for quantifying and modeling multimodal interactions that has potential applications in a wide range of fields.", "questions": "1. How does the proposed approach compare to other methods for quantifying multimodal interactions, such as correlation analysis or principal component analysis?\n2. Can the approach be applied to continuous-valued data, or is it limited to discrete data?\n3. How does the approach handle missing data or incomplete modalities?\n4. How does the approach handle interactions between more than two modalities?\n5. In Table 1, what is Exact task? Why all task and measures have the same numbers?\n6. Table 3 is hard to understand, maybe better to use the same scale?\n", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel approach: The paper proposes a novel information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. This approach is different from other methods for quantifying multimodal interactions and has the potential to provide new insights into how different modalities interact.\n\n2. Extensive validation: The authors validate their approach on both synthetic and real-world datasets, demonstrating its ability to identify meaningful interactions and select appropriate models. They also compare their approach to other methods for quantifying multimodal interactions, showing that their method outperforms these alternatives.\n\n3. Real-world applications: The paper provides examples of real-world applications where the PID statistics were useful in quantifying interactions within multimodal datasets or selecting appropriate multimodal models. This demonstrates the practical usefulness of the approach in a wide range of fields.\n\n4. Clear presentation: The paper is well-written and clearly presents the approach, the validation experiments, and the real-world applications. The authors provide detailed explanations of the methods used and the results obtained, making it easy for readers to understand and replicate their work", "weaknesses": "1. Limited scope: The paper focuses on a specific information-theoretic approach to quantifying multimodal interactions and does not consider other approaches or methods. This may limit the generalizability of the results and the usefulness of the approach in other contexts.\n\n2. Lack of comparison to ground truth: The paper validates the approach on synthetic and real-world datasets, but does not compare the results to a ground truth or gold standard. This makes it difficult to assess the accuracy of the approach and the validity of the results.\n\n3. Limited interpretability: The paper acknowledges that the results of the approach can be difficult to interpret, particularly in high-dimensional datasets. This may limit the usefulness of the approach in practice, as it may be difficult to draw meaningful conclusions from the results.\n\n4. Lack of implementation details: The paper provides a high-level overview of the approach and the methods used, but does not provide detailed implementation instructions or code. This may make it difficult for other researchers to replicate the work or apply the approach in their own research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful comments! We respond to some concerns below:\n\n[other methods] Our framework allows for easy generalization to other definitions as well: **we implemented 3 alternative definitions for interactions, including the original called I-min, WMS, and CI (see [1] for a review). These results are in Table 1 of rebuttal pdf**:\n1. I-min can sometimes overestimate synergy and uniqueness. It overestimates synergy on R-only, U1-only, and U2-only datasets, and assigns non-zero unique interactions (U1=0.08, U2=0.07 and U1=0.03, U2=0.04) even though there is no uniqueness.\n2. WMS is actually synergy minus redundancy, so when they are of equal magnitude WMS cancels out to be 0, leading one to erroneously conclude there is no S or R. For example, it is unable to detect high synergy for S-only data and assigns uniqueness instead. This term can also be negative (S=-0.11 for R-only data).\n3. CI can also be negative (U1=-0.09, U2=-0.10 for R-only data), and sometimes incorrectly concludes highest uniqueness (U1=0.13, U2=0.14) for S-only data.\n\n**Our framework also fits in with non-info theory measures, we implemented 3 more**:\n1. Shapley is based on interactions captured by a multimodal model - our work is fundamentally different in that interactions are properties of *data*. This allows us to characterize datasets and select the most appropriate model. We ran this and found 3 issues:\n- It only captures overall independence, not separate uniqueness for each modality\n- It overestimates synergy=0.5 for pure redundancy data and overestimates independence=1.0 for pure synergy data\n- We also found difficulties in estimating Shapley for large, high-dimensional datasets efficiently, which is still an open question.\n2. Integrated gradients (IG) only gives unimodal contributions of a trained model. We apply this method for computational pathology to derive importance of histology and genomics data, and find IG echoes the unique interactions of PID, but PID enables us to do so *before* model selection and training. Moreover, it is not clear how to estimate synergy with IG.\n3. CCA only gives shared information between modalities, which can be used as a proxy of redundancy, but does not give other interactions. We ran this and found that one can learn representations to maximize correlation arbitrarily high (e.g., rho=1.0 for MOSEI), even though the redundancy is bounded.\n\n[1] Griffith and Koch. Quantifying synergistic mutual information. 2014\n\n[comparison] **We tried our best to compare to ground truth in**:\n1. Table 1, bitwise operators, with ground truth PID values obtained from exactly solving the optimization problem, and both our estimators recover the ground truth.\n2. Table 2, synthetic datasets (10 types), with ground truth based on which latent features the label was sampled from (shared feature, unique to either modality, or their combination), and we show consistent trends with ground truth.\n3. Table 3, real-world datasets (8 types), with gold standard obtained from human annotation of interactions, and we show consistent trends with gold standard.\n\n[interpretability] Our approaches are designed to deal with high-dimensional data through (1) clustering followed by discrete convex optimization, and (2) end-to-end representation learning from continuous data. **Our approach does scale to high dimensions in practice through 8 large multimodal datasets and 3 additional real-world case studies, enabling domain experts to interpret these interactions.**\n\n[details] All code and data are in the supplementary material so details are transparent and reproducible, and we will put the code on github. We will add further requested details to appendix.\n\n[continuous] We propose 2 estimators, and our second BATCH estimator is designed to extend to continuous data, since we use neural networks to encode continuous inputs and learn a distribution q used to compute interaction values. The first CVX estimator assumes discrete support for distribution q so operates via a clustering preprocessing step.\n\n[missing modalities] R, U1, U2 can be estimated with access to only p(x1,y) and p(x2,y) which means they can be estimated in the presence of missing modalities. Synergy needs the full distribution p(x1,x2,y) due to the first I_p(X1,X2;Y) term. However, we can approximate synergy via the disagreement between unimodal classifiers p(y|x1) and p(y|x2), which [2] showed is related to the unique information I(X1;Y|X2) and I(X2;Y|X1). Extension of our work to missing modalities is an important avenue for future work.\n\n[2] Sridharan and Kakade. An information theoretic framework for multi-view learning. COLT 2008.\n\n[>2 modalities] Extensions to more than 2 modalities is challenging due to a larger number of interactions (e.g., x1 and x2 may exhibit synergy, but not with x3), but future work can leverage advances in information theory: e.g. one could consider decomposing features hierarchically, or approximating via pairwise synergies.\n\n[Table 1] Table 1 tests 3 multimodal distributions, where x1, x2 are 0/1 bit vectors, and the label y = x1 OR x2, y = x1 AND x2, and y = x1 XOR x2. For each of the 3 distributions the PID can be solved exactly, for example y = x1 XOR x2 requires 1 bit of S, y = x1 OR x2, y = x1 AND x2 require 0.31 bits of R and 0.5 bits of S. In comparison with this ground truth, both our estimators exactly recover the correct PID values.\n\n[Table 3] For human judgment, we cannot ask humans to give a score in bits, so it is on a completely different scale (1-5 scale). To put them in the same scale, we normalize the human ratings such that the sum of human interactions = sum of PID estimates. The resulting comparisons are in Table 2 in attached rebuttal pdf, and we find very close match between estimators and human gold standards."}, {"review_id": "GpRNYLwqSj", "reviewer": "Reviewer_4r5p", "summary": "This paper attempts to quantify the interactions between different input modalities using Principle Information Decomposition (PID) from Information Theory. Using this approach, one can quantify not only interactions within multimodal datasets but as well as interactions captured by multimodal models. \nThe paper provided a comprehensive evaluation of their approach and provided examples of its application in real-world case studies.  \n", "questions": "Are there any potential use cases of this approach outside multimodal models? Like, e.g. measuring the domain gap between synthetic and real-world data?", "limitations": "There is no dedicated section on its Limitations, nor is it focused on in the paper as far as I could find. It would be useful so that the readers would know in which cases this could potentially not be effective. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Provided a detailed explanation of its theoretical background, and thorough comprehensive evaluation. They also provided a list of real-world applications which are useful for casual users. ", "weaknesses": "The presented work, would benefit from more visuals. As this paper is quite detailed, it would benefit from a closing summary listing the main points of the paper. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful comments! We respond to some concerns below:\n\n[visuals] Thank you for pointing this out - we do plan to add more visuals and have included a rebuttal pdf with some other figures we have included to the paper:\n1. Figure 1: A venn diagram to give an intuitive explanation of multimodal interactions and the main outline of the paper (dataset quantification, model quantification, model selection).\n2. Figure 2: A figure to give an intuitive explanation of PID, distributions q \\in \\Delta_p, and computation of synergy,\n3. Figure 3: To show an illustration of model selection based on quantifying dataset interactions and model interactions.\n\nWe also have a summary section in Appendix D including key takeaway messages, and new directions for future work. At a high-level, the key ideas are:\n1. Dataset quantification: PID provides reliable indicators for interactions present in both synthetic and real-world multimodal datasets.\n2. Model quantification: PID also quantifies interactions captured by trained models, and we find several consistent patterns in the types of interactions different models capture.\n3. Model selection demonstrates potential utility as a ‘user-guide’ for practitioners aiming to tackle real-world multimodal datasets. Given a new dataset, estimate its PID values. If there is high U1 and U2, just using unimodal models in the corresponding modality may be sufficient. Otherwise, if there is high R, methods like agreement, alignment, ensemble, or co-training should be tried. If there is high S, it is worth spending the time on multimodal methods that model more complex interactions based on tensors, multiplicative interactions, and self-attention. We also show results on automatically selecting the top-3 models, which recover >96% of the best performance from exhaustively trying all models.\n\n[outside multimodal] Yes, **this framework can be used to study feature interactions more broadly even within single modality models (e.g., multiple image regions, multiple sentences) [1] as well as bias and unfairness between different interacting features** [2,3]. PID can be used to quantify the contribution of individual features as well as interacting features to unfair predictions or privacy leakage, which can then in turn be useful for minimizing the effect of these features and interactions to improve fairness and privacy. We believe that these are all ripe directions for future work. Being able to estimate and provide guarantees for fairness and privacy-preserving learning can be a particularly impactful application. There are also some connections between classical information theory and distribution shift [4], but not PID and other fine-grained interactions which could also yield new insights of future work.\n\n[1] Tsang et al., Feature interaction interpretability: A case for explaining ad-recommendation systems via neural interaction detection. ICLR 2019\n\n[2] Dutta et al., An information-theoretic quantification of discrimination with exempt features. AAAI 2020\n\n[3] Hamman et al., Demystifying Local and Global Fairness Trade-offs in Federated Learning Using Information Theory. FL-ICML 2023\n\n[4] Federici et al., An Information-theoretic Approach to Distribution Shifts. NeurIPS 2021\n\n[limitations] We have expanded on the limitations and broader impact section in Appendix E. To summarize:\n1. CVX works for smaller domains up to several thousand classes or may suffer from low empirical frequencies beyond that; BATCH may suffer from approximation error depending on whether the network can sufficiently cover the space of joint distributions, bias resulting from mini-batch gradient descent, as well as the usual local minima and unstable convergence problems with neural estimators.\n2. These estimators only approximate real interactions due to cluster preprocessing as well as the standard optimization and generalization errors of approximate models. We expect future progress in density estimators, generative models, and MI estimation to help address these problems.\n3. We are aware of challenges in evaluating PID estimation and emphasize that they should be interpreted as a relative sense of which interaction is most important, and a guideline to inspire model selection and design."}, {"review_id": "wDV9gc5Ym5", "reviewer": "Reviewer_td1f", "summary": "This paper addresses a fundamental question in multimodal modeling: given multiple input modalities, how can we quantify the ways they must interact (as part of a model) to solve a particular task. Furthermore, can we use knowledge to learn how to design models that better leverage these interactions. \n\nThe authors propose an information-theoretic approach to this problem by quantifying three properties (redundancy, uniqueness, and synergy) using mutual information across modalities. They solve for these properties using two estimators (one using convex optimization and the other sampling). The first estimator uses convex optimization and the second uses a sampling-baed approximation. \n\nThey evaluate their approaches for quantifying these properties using synthetic and real-world datasets. For the latter, annotators manually went through each dataset and quantified their intuition for overlap between modalities. They show that their models are able to do a reasonable job estimating these quantities. ", "questions": "* Sec 3.1 (preprocessing via feature binning): Is estimating this using binning a reasonable idea here? I suppose this depends on the bin size, dimensionality, size of the dataset, and the distribution of latents. \n\n* The \"Batch\" estimator is interesting. I immediately wondered if VQ-VAEs or other discrete state latent variable models could be used as an alternative here. This would give you (discrete) tokens instead of having to optimize over a continuous space. Does this seem like a reasonable idea? This idea seems simpler to me and means that you don't have to subsample batches as done here. ", "limitations": "Yes, more so in the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Overall this is a well written and carried out paper with a lot of depth. The authors approach the problem in a principled manner and have thought deeply about underlying concepts of multi-modality. \n* While I did not go through the derivations, the authors introduce a nice framework for thinking about multi-modal interactions. \n* I appreciate the combination of synthetic experiments, real-world experiments, snapshot \"case studies,\" and various ablation experiments. \n* The appendix includes a large amount of interesting additions. ", "weaknesses": "* I found the results hard to interpret and -- given synthetic results -- am wary of the validity of the claims. For example, in the second set of synthetic experiments (Table 2) the recovered values are MUCH lower than ideal (e.g., D_R set has R(CVX)=0.16, R(Batch)=0.29, and R(Truth)=1). This is especially true in the \"mixed interaction\" synthetic datasets. Is there a good reason that the estimated values are so far from the truth? Are the proposed estimators good in the absence of noise (i.e., the bit-wise experiment) but do they quickly fail once there is uncertainty? Is the issue with the estimators, the underlying generative model, or something else?\n\n* The core idea of the paper is solid, but I'm wary of the use of human judgements for creating ground truth. The models are using features/representations extracted from each input modality, and a human's perception/judgement here may differ significantly from signal in the data. I acknowledge there may not be a better solution for annotations. (aside: Thanks for putting info about the annotators' backgrounds and instructions in the appendix). \n\n* Similarly, in \"Results on multimodal fusion,\" given my experience with these kinds of models I'm hesitant to believe the rational/intuition such as \"since sarcasm is often due to a contradiction between what is expressed in language an speech.\" It seems like a natural conclusion, but my understanding is that speech emotion (or sarcasm) detectors use a more limited context to compute their predictions, which may not have enough context to encode this contradiction  (but maybe I'm wrong?)? The rational here \"make sense\" but seem cherry picked and might not be \"true.\" Couldn't there be other (potentially spurious) correlations that cause the results to look nice?\n\n* I imagine there are alternative correlations or measures one could use for evaluating redundancy/uniqueness/synergy. It would be useful to see more discussion (or comparison) with this. \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful comments! We respond to some concerns below:\n\n[scale] **The absolute value (i.e., actual bits of information) will never be fully 1 bit, because one can never extract the exact amount of task-relevant information**, which would imply 100% train and test accuracy. Noise in the data and labels, errors in feature learning and model optimization, and generalization gaps between train and test all contribute to lower estimated interaction values than the ground truth.\n\nWe can get a sense of the total capturable information using the bound from Feder and Merhav [1], which relates the accuracy of the best model on these datasets with the mutual information between the inputs to the label. **Since the test accuracies for Table 2 datasets range from 67-75%, this corresponds to total MI of 0.422-0.585 bits**. This is a tighter upper bound ground truth that our estimators should capture, and we see that they are quite close, see updated Table 2 in attached rebuttal pdf.\n\nWe are aware of challenges in evaluating PID estimation and emphasize that they should be interpreted as a relative sense of which interaction is most important. Both CVX and BATCH measures agree with each other on relative values and are consistent with ground truth interactions on relative values.\n\nFinally, we also emphasize that evaluating PID estimation is just half the picture - the more important part is using these quantities for downstream real-world case studies in multimodal model selection and estimating robustness to missing modalities for various applications.\n\n[1] Feder and Merhav. Relations between entropy and error probability. IEEE Transactions on Information theory 1994\n\n[human judgement] We are aware of the caveats regarding human annotation and annotator biases, but unfortunately for the real-world datasets (and especially 3 case studies) it may be the best choice we have. **The inter-annotator agreement is relatively high (0.72 for R, 0.68 for U1, 0.70 for U2, 0.72 for S) and the average confidence scores are also high (4.36/5 for R, 4.35/5 for U1, 4.27/5 for U2, 4.46/5 for S), indicating that human judgement is quite consistent across annotators**.\n\n[sarcasm] For sarcasm, we find that humans do estimate both synergy and unique language as relatively high - sometimes humans can tell sarcasm from language only. We will rephrase as ‘sarcasm can be (instead of often) due to a contradiction between what is expressed in language and speech’. We use sarcasm datasets that are more multimodal in nature (from the Mustard dataset of acted sarcasm in TV shows like the Big Bang Theory where the actors are more sarcastic in nature than real-life speech), which we believe explains the higher synergy in sarcasm we found.\n\n[other measures] **We implemented 3 alternative definitions for interactions, including the original called I-min, WMS, and CI (see [1] for a review). These results are in Table 1 of rebuttal pdf:**\n\n1. I-min can sometimes overestimate synergy and uniqueness. It overestimates synergy on U1-only and U2-only datasets, and assigns non-zero unique interactions (U1=0.08, U2=0.07 and U1=0.03, U2=0.04) even though there is no uniqueness.\n2. WMS is actually synergy minus redundancy, so when they are of equal magnitude WMS cancels out to be 0, leading one to erroneously conclude there is no S or R. For example, it is unable to detect high synergy for S-only data and assigns uniqueness instead. This term can also be negative (S=-0.11 for R-only data).\n3. CI can also be negative (U1=-0.09, U2=-0.10 for R-only data), and sometimes incorrectly concludes highest uniqueness (U1=0.13, U2=0.14) for S-only data.\n\n**We also implemented 3 more non-info theory measures:**\n\n1. Shapley and integrated gradients estimate interactions *captured by a multimodal model - our work is fundamentally different in that interactions are properties of data*. This allows us to characterize datasets and select the most appropriate model. We ran this and found 3 issues:\n- It only captures overall independence, not separate uniqueness for each modality\n- It overestimates synergy=0.5 for pure redundancy data and overestimates independence=1.0 for pure synergy data\n- We also found difficulties in estimating Shapley for large high-dimensional datasets efficiently\n\n2. CCA only gives shared information between modalities, which can be used as a proxy of redundancy, but does not give other interactions. We ran this and found that one can learn representations to maximize correlation arbitrarily high (e.g., rho=1.0 for MOSEI), even though the redundancy is bounded.\n\n[1] Griffith and Koch. Quantifying synergistic mutual information. 2014\n\n[binning] With a larger number of bins (e.g., >50), our estimates barely change, unless too many bins are chosen (which can lead to numerical instability as some bins become empty or without any samples). One recommendation in [2] is to select the number of bins to be the cubed root of the number of samples. We provide more details in Appendix B1. For modalities with higher dimensions, we recommend the use of our BATCH estimator instead due to exponential blowup in the number of bins wrt dimension.\n\n[2] Rice. Mathematical statistics and data analysis. Cengage Learning 2006.\n\n[batch and vq-vae] Absolutely: using the idea of discrete clusters for the CVX estimator, but with learned cluster representations of continuous data using VQ-VAE rather than fixed clusters as we do now. However, training VQ-VAE on high-dimensional multimodal data is non-trivial, and end-to-end training raw data -> VQ-VAE -> cluster representations -> convex solver -> PID objective is an even cooler idea but requires rewriting convex optimizers in pytorch. We would love to explore these in future work, and have added a discussion to our paper!"}, {"review_id": "MekbZa1swV", "reviewer": "Reviewer_xXH1", "summary": "This paper proposes an information-theoretic approach to quantify the degree of three PID(Partial Information Decomposition) statistics of a multimodal distribution: redundancy, uniqueness and synergy. To solve the difficulties in computing PID over high-dimensional continuous distributions, two scalable estimators for PID statistics are introduced, enabling the estimation of PID statistics over large multimodal datasets using small subsampled batches. To validate PID estimation, experiments are conducted on synthetic and real-world multimodal datasets. The usefulness of PID is also demonstrated in model selection and real-world case studies.", "questions": "Major concerns have been mentioned in Weakness. \n\nLine 163: missing ')'", "limitations": "The authors did not address the limitation.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ This paper proposes an useful approach to quantify the nature and degree of modality interactions based on information theory. The proposed PID estimators can scale to high-dimensional multimodal datasets and models.\n+ The definitions of redundancy, uniqueness, and synergy are intuitive and could help to better understand multimodal datasets and models.\n+ The proposed approach could generalize to real-world applications and provide reasonable interpretation for dataset properties and model selection.", "weaknesses": "- The results in Table 2 and 3 are unable to prove the accuracy of the PID estimators. Estimated results only show a rough trend. There is an order of magnitude difference between the approximation and the ground truth in most cases. Besides, it seems that the approximation becomes more inaccurate when the value distribution of R, U, S is more uniform (i.e. The second main row in Table 3). Is this method only applicable to simple and distinctive datasets?\n- Lack of comparison and discussion with existing methods. For example, Shapley Value could measure the contribution of each modality in multimodal learning. What are the similarities and differences between them? Why is PID better than them?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful comments! We respond to some concerns below:\n\n[table 2] **The absolute value (i.e., actual bits of information) will never be fully 1 bit, because one can never extract the exact amount of task-relevant information**, which would imply 100% train and test accuracy. Noise in the data and labels, errors in feature learning and model optimization, and generalization gaps between train and test all contribute to lower estimated interaction values than the ground truth.\n\nWe can get a sense of the total capturable information using the bound from Feder and Merhav [1], which relates the accuracy of the best model on these datasets with the mutual information between the inputs to the label. **Since the test accuracies for Table 2 datasets range from 67-75%, this corresponds to total MI of 0.422-0.585 bits**. This is a tighter upper bound ground truth that our estimators should capture, and we see that they are quite close, see Table 1 in attached rebuttal pdf.\n\nWe are aware of challenges in evaluating PID estimation and emphasize that they should be interpreted as a relative sense of which interaction is most important. Both CVX and BATCH measures agree with each other on relative values and are consistent with ground truth interactions on relative values.\n\n**Our method is not limited to simple and distinctive datasets, and we also emphasize that evaluating PID estimation is just half the picture - the more important part is using these quantities for downstream real-world case studies in dataset quantification, model quantification, and model selection through engagement with domain experts. Our experiments cover more than 11 datasets and 10 complex models from research areas of multimedia, affective computing, robotics, healthcare, and HCI.**\n\n[1] Feder and Merhav. Relations between entropy and error probability. IEEE Transactions on Information theory 1994\n\n[table 3] For human judgment, we cannot ask humans to give a score in bits, so it is on a completely different scale (1-5 scale). To put them in the same scale, we normalize the human ratings such that the sum of human interactions = sum of PID estimates. The resulting comparisons are in Table 2 in attached rebuttal pdf, and we find very close match between estimators and human gold standards.\n\n[other methods] Our framework allows for easy generalization to other definitions as well: **we implemented 3 alternative definitions for interactions, including the original called I-min, WMS, and CI (see [1] for a review). These results are in Table 1 of rebuttal pdf:**\n\n1. I-min can sometimes overestimate synergy and uniqueness. It overestimates synergy on U1-only and U2-only datasets, and assigns non-zero unique interactions (U1=0.08, U2=0.07 and U1=0.03, U2=0.04) even though there is no uniqueness.\n2. WMS is actually synergy minus redundancy, so when they are of equal magnitude WMS cancels out to be 0, leading one to erroneously conclude there is no S or R. For example, it is unable to detect high synergy for S-only data and assigns uniqueness instead. This term can also be negative (S=-0.11 for R-only data).\n3. CI can also be negative (U1=-0.09, U2=-0.10 for R-only data), and sometimes incorrectly concludes highest uniqueness (U1=0.13, U2=0.14) for S-only data.\n\n**Our framework also fits in with non-info theory measures, we implemented 3 more:**\n\n1. Shapley is based on interactions captured by a multimodal model - our work is fundamentally different in that interactions are properties of data. This allows us to characterize datasets and select the most appropriate model. We ran this and found 3 issues:\n- It only captures overall independence, not separate uniqueness for each modality\n- It overestimates synergy=0.5 for pure redundancy data and overestimates independence=1.0 for pure synergy data\n- We also found difficulties in estimating Shapley for large, high-dimensional datasets efficiently, which is still an open question.\n2. Integrated gradients (IG) only gives unimodal contributions of a trained model. We apply this method for computational pathology to derive importance of histology and genomics data, and find IG echoes the unique interactions of PID, but PID enables us to do so before model selection and training. Moreover, it is not clear how to estimate synergy with IG.\n3. CCA only gives shared information between modalities, which can be used as a proxy of redundancy, but does not give other interactions. We ran this and found that one can learn representations to maximize correlation arbitrarily high (e.g., rho=1.0 for MOSEI), even though the redundancy is bounded.\n\n[1] Griffith and Koch. Quantifying synergistic mutual information. 2014\n\n[limitations] We have expanded on the limitations and broader impact section in Appendix E. To summarize:\n1. CVX works for smaller domains up to several thousand classes or may suffer from low empirical frequencies beyond this; BATCH may suffer from approximation error depending on whether the network can sufficiently cover the space of joint distributions, bias resulting from mini-batch gradient descent, as well as the usual local minima and unstable convergence problems with neural estimators.\n2. These estimators only approximate real interactions due to cluster preprocessing as well as the standard optimization and generalization errors of approximate models. We expect future progress in density estimators, generative models, and MI estimation to help address these problems.\n3. We are aware of challenges in evaluating PID estimation and emphasize that they should be interpreted as a relative sense of which interaction is most important, and a guideline to inspire model selection and design."}, {"review_id": "2AXzCiOjIU", "reviewer": "Reviewer_uMC4", "summary": "This work is mainly concerned with the methodological contribution of estimators for partial information theory statistics, to quantify the levels of redundancy, uniqueness and synergy, which are expressed in information theoretic terms, through the use of mutual information. \n\nDepending on dataset size and dimensionality, the authors propose two variants of estimators: one that is exact but can only be used for small, low-dimensional data, and one that is approximate, but that scales to large, high-dimensional data.\nPID statistics can be computer for models as well, which enables the extension of to the proposed framework to deal with model selection.\n\nA very large series of experiments are discussed to support 1) the usefulness of PID statistics, 2) the quality of the proposed estimators, 3) the applicability of the proposed methods to real-world dataset. In addition, several experiments on model selection are carried out as well. Finally, three real world applications are studied through the lenses of partial information decomposition, further substantiating the impact of such statistics. When possible, evaluation is exact, given the the ground truth PID statistics value could be computed analytically. Otherwise, the authors resort to human feedback.", "questions": "* I understand that both the methodology and experiments tackle two modalities. Now, even if datasets with more than two modalities, and models capable of handling more than two modalities are not extremely common, a natural question that arises both for the proposed methodology, its foundations on PID, and the experiments is how well does the schemes proposed in this work scale with the number of modalities.\n\n* In the appendix, it is mentioned that the BATCH estimator, in its current form, relies on pre-trained classifiers, including multimodal ones. As explained, this can have the unintended effect of perturbing the estimation of PID statistics, resulting in errors on synergy. While a theoretical analysis of approximation guarantees seem to be rather difficult, could you discuss about means to keep approximation errors under control? \n\n* For the BATCH estimator, a key aspect to consider is the dimensionality of the hidden representation that is used to compute matrix A (before sinkhorn is applied). Do you have any insights concerning the effects of this hyper-parameter?\n\n* On the values obtained for R, U and S. As an overview on the various result tables, it is somehow hard to justify the choice for bold numbers. In particular, if we consider results on synthetic data for which a ground truth exists, we can see that the values for PID statistics obtained by either the exact or approximate schemes is very small. In some cases the differences between the values obtained is small, and bold numbers seem to be chosen to “comply” with group truth: see for example Table 2 $y=f(z_2^*, z_c^*)$ for the CVX scheme. More generally, the question I would like to ask is related to the “degree” aspects of PID statistics, and how to interpret correctly numerical values, especially when such values are very low, say what is the difference between R=0.07 and R=0.06 in practical terms? I understand that the first statistics indicate more redundancy than the second, but is this going to be discernible in practice?\n\n* Finally, if time allows, would it be possible to address the points mentioned in the \"weaknesses\" part of the review?", "limitations": "To the best of my knowledge, there is no explicit discussion on limitations nor societal impact. However, in the appendix, it is possible to find information about caveats, assumptions, which help understanding some limitations of the proposed methods.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Deepening our understanding of multimodal data, and models built to address tasks that rely on multimodal data is of fundamental importance. The use of partial information decomposition is, in my opinion, a good approach to achieve such goals, and this paper is convincing in both showcasing that PID statistics are indeed representative for multimodal data and modelling tasks, as well as supporting the proposed methodology to compute such PID statistics.\n\n* The experimental campaign is very extensive and overall (some comments in the questions section should be addressed though) very convincing and clear. \n\n* The proposed methodology (especially the approximate one) for computing PID statistics is simple, and easy to implement\n", "weaknesses": "* Multimodal VAEs [1,2,3] are missing. To the best of my knowledge, a large body of research on multimodal representation learning is missing from the study of PID statistics for model predictions. On the one hand, I understand that in this work a model is used to address a given downstream task, whereas multimodal VAEs are used mostly to learn latent representations of all modalities, for generative purposes, but also to solve downstream tasks. Since the literature on Multimodal VAEs clearly addresses relevant questions on coherency, and synergy, I think this work could greatly benefit from an extension to such models.\n\n* Alternatives to compute MI are not considered as “competitor” methods for the approximate scheme. Approaches such as [4, 5] have received a lot of attention as means to compute MI. Would it make sense to consider those as building blocks for alternative approximate PID estimation methods?\n\n* Some details in the experimental section need further clarification. For example, in section 4.3 on the robustness, e.g. missing modalities, it is not clear if this is considered at training or a test time.\n\n[1] Wu, et al.,”Multimodal Generative Models for Scalable Weakly-Supervised Learning”, https://arxiv.org/abs/1802.05335\n[2] Shi, et al., “Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models”, https://arxiv.org/abs/1911.03393\n[3] Palumbo, et al., “MMVAE+: Enhancing the Generative Quality of Multimodal VAEs without Compromises”, https://openreview.net/forum?id=B42UJTVdDZ5\n[4] Belghazi, et al., “MINE: Mutual Information Neural Estimation”, https://arxiv.org/abs/1801.04062\n[5] Hijelm, et al., “Learning deep representations by mutual information estimation and maximization”, https://arxiv.org/abs/1808.06670\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful comments! We respond to some concerns below:\n\n[multimodal VAEs] While multimodal VAEs are related, their goal is not to define and estimate multimodal interactions, but rather using intuition of interactions to guide model design (e.g., see figure 2 in [2] for a summary). Furthermore, training these multimodal VAEs is challenging (even harder than supervised learning). We have added these references and a discussion to our paper.\n\n[MI estimation] We do use these as an inspiration for our second batch estimator - the fact that neural networks can learn representations that are optimized for a MI term (e.g., like [4,5]). The main difference is that we’re not just estimating MI for a given distribution p, but also optimizing over distributions q \\in \\Delta_p wrt an MI objective. We are extending our work to optimize for a certain multimodal interaction (not just estimating it), which mirrors your suggested work [5].\n\n[other methods] **Our framework allows for easy generalization to other definitions: we implemented the original called I-min, WMS, and CI (see [1] for a review), as well as 3 non-info theory measures Shapley values, Integrated gradients (IG), and CCA**. These results are in Table 1 of rebuttal pdf, and to summarize:\n\n1. I-min can sometimes overestimate synergy and uniqueness. It overestimates synergy on U1-only and U2-only datasets, and assigns non-zero unique interactions (U1=0.08, U2=0.07 and U1=0.03, U2=0.04) even though there is no uniqueness.\n2. WMS is actually synergy minus redundancy, so when they are of equal magnitude WMS cancels out to be 0, leading one to erroneously conclude there is no S or R. For example, it is unable to detect high synergy for S-only data and assigns uniqueness instead. This term can also be negative (S=-0.11 for R-only data).\n3. CI can also be negative (U1=-0.09, U2=-0.10 for R-only data), and sometimes incorrectly concludes highest uniqueness (U1=0.13, U2=0.14) for S-only data.\n4. Shapley and IG are based on interactions captured by a multimodal model - our work is fundamentally different in that interactions are properties of data. This allows us to characterize datasets and select the most appropriate model.\n5. CCA only gives shared information between modalities but does not give other interactions. We ran this and found that one can learn representations to maximize correlation arbitrarily high (e.g., rho=1.0 for MOSEI), even though the redundancy is bounded.\n\n[1] Griffith and Koch. Quantifying synergistic mutual information. 2014\n\n[robustness] This is only test-time performance - models trained on all modalities but tested on data with modality imperfections and drops, which we show is highly correlated with the amount of unique and synergistic information that the trained model learns.\n\n[>2 modalities] Extensions to more than 2 modalities is challenging due to a larger number of interactions (e.g., x1 and x2 may exhibit synergy, but not with x3), but future work can leverage advances in information theory: e.g. one could consider decomposing features hierarchically, or approximating via pairwise synergies.\n\n[batch estimator approximation] We notice that under-fitting classifiers could cause the BATCH estimator to underestimate synergy. We gradually scale up our model size and batch size until PID converges. Combined with traditional regularization techniques against overfitting (such as early stopping), these techniques can mitigate approximation errors.\n\n[batch estimator dimension] The hidden dimension (d) is important because too small d limits expressivity of joint distributions q, and too large d can be computationally expensive. We scale up the hidden dimension size until the PID statistics stabilize, and reach a final value d=32.\n\nAnother important hyperparameter is the batchsize - a higher batchsize gives more accurate estimates of the joint distribution between X1 and X2, while a smaller batchsize is also important for regular updates via SGD. We tune the batchsize and reach n=256.\n\nWe have included code in the supplementary and plan to release it publicly after the review period so other readers can play with these settings.\n\n[table 2] **The absolute value of PID estimates (i.e., actual bits of information) will never be fully 1 bit, because one can never extract the exact amount of task-relevant information**, which would imply 100% train and test accuracy. We can get a sense of the total capturable information using the bound from Feder and Merhav [1], which relates the accuracy of the best model on these datasets with the mutual information between the inputs to the label. **Since the test accuracies for Table 2 datasets range from 67-75%, this corresponds to total MI of 0.422-0.585 bits**. This is a tighter upper bound ground truth that our estimators should capture, and we see that they are quite close, see Table 1 in attached rebuttal pdf.\n\n[1] Feder and Merhav. Relations between entropy and error probability. IEEE Transactions on Information theory 1994\n\n[degree aspects] We are aware of challenges in evaluating PID estimation and emphasize that they should be interpreted as a relative sense of which interaction is most important within a dataset. One should not compare interactions across different datasets, since the total information is different so they lie on different scales. In working with end users, they also generally relied on the top interaction as well as the distribution of interactions (e.g., mostly redundancy, or whether all interactions are relatively uniform) to make modeling decisions.\n\n[limitations] We have expanded on the limitations and broader impact section in Appendix E, highlighting limitations and future directions in improving the estimators and their evaluation leveraging future progress in multimodal benchmarks, generative models, and MI estimation."}]

Author rebuttal:
Dear all reviewers, we are extremely grateful for your valuable feedback and insightful comments. We are glad that you agree that our framework can be of fundamental importance for studying multimodal interactions, and that our proposed estimators are technically novel and comprehensively evaluated. Your concrete suggestions are a valuable step in this direction, and we have revised our submission accordingly to take these into account. In this short note we summarize the main changes we made to our submission:

[other methods] Our framework allows for easy generalization to other interaction definitions: we implemented 3 information theoretic measures I-min, WMS, and CI. These results are in Table 1 of the attached rebuttal pdf, and we outlined several limitations of these methods as compared to PID. We also tried 3 non-info theory measures: Shapley values, Integrated gradients (IG), and CCA, which are based on quantifying interactions captured by a multimodal model - our work is fundamentally different in that interactions are properties of data before training any models.

[table 2] The absolute value (i.e., actual bits of information) will never be fully 1 bit, because one can never extract the exact amount of task-relevant information, which would imply 100% train and test accuracy. We estimated the total capturable information based on test accuracy and we see much closer PID estimates to ground truth: see Table 1 in the attached rebuttal pdf.

[table 3] For human judgment, we cannot ask humans to give a score in bits, so it is on a different scale (1-5 scale). To put them in the same scale with PID estimates, we normalize the human ratings such that the sum of human interactions = sum of PID estimates. The resulting comparisons are in Table 2 of attached rebuttal pdf, and we find very close match between PID estimates and human gold standards.

[limitations] We have expanded on the limitations and broader impact section in Appendix E, highlighting limitations and future directions in improving the estimators and their evaluation leveraging future progress in multimodal benchmarks, generative models, and MI estimation.

[details and practice] Finally, we would like to emphasize that all code and data are in the supplementary material, and we plan to release all code and data on github after the review period so details are transparent and reproducible. We have also added all requested details regarding estimators and experiments to the appendix.

Meta-review: The reviewers are generally positive about the work, and even more so after the rebuttal. Although there are many detailed questions about the clarity, experimental design and generalizability, the reviewers overall see the work contributes a solid investigation into an important problem. The reviewers think the work is "novel" and "principled" and the experiments are "extensive". Specifically, Reviewer uMC4 stated (the work) "Deepening our understanding of multimodal data, and models built to address tasks that rely on multimodal data is of fundamental importance." The authors also did a good job at addressing the reviewers's questions during rebuttal, which strengthened the reviewers's confidence in accepting the paper.
Decision: Accept (poster)