Title: Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
Venue: NeurIPS 2023
Accepted: yes
Keywords: multimodal learning

feature interactions

partial information decomposition

information theory

quantification

model selection
Abstract: The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities. Despite these empirical advances, there remain fundamental research questions: How can we quantify the interactions that are necessary to solve a multimodal task? Subsequently, what are the most suitable multimodal models to capture these interactions? To answer these questions, we propose an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. We term these three measures as the PID statistics of a multimodal distribution (or PID for short), and introduce two new estimators for these PID statistics that scale to high-dimensional distributions. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks where PID estimations are compared with human annotations. Finally, we demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies engaging with domain experts in pathology, mood prediction, and robotic perception where our framework helps to recommend strong multimodal models for each application.

Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework

Abstract

The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities. Despite these empirical advances, there remain fundamental research questions: How can we quantify the interactions that are necessary to solve a multimodal task? Subsequently, what are the most suitable multimodal models to capture these interactions? To answer these questions, we propose an information-theoretic approach to quantify the degree of redundancy , uniqueness , and synergy relating input modalities with an output task. We term these three measures as the PID statistics of a multimodal distribution (or PID for short), and introduce two new estimators for these PID statistics that scale to high-dimensional distributions. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks where PID estimations are compared with human annotations. Finally, we demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies engaging with domain experts in pathology, mood prediction, and robotic perception where our framework helps to recommend strong multimodal models for each application.

1 Introduction

A core challenge in machine learning lies in capturing the interactions between multiple input modalities. Learning different types of multimodal interactions is often quoted as motivation for many successful multimodal modeling paradigms, such as contrastive learning to capture redundancy [54, 82], modality-specific representations to retain unique information [100], as well as tensors and multiplicative interactions to learn higher-order interactions [52, 58, 115]. However, several fundamental research questions remain: How can we quantify the interactions that are necessary to solve a multimodal task? Subsequently, what are the most suitable multimodal models to capture these interactions? This paper aims to formalize these questions by proposing an approach to quantify the nature (i.e., which type) and degree (i.e., the amount) of modality interactions, a fundamental principle underpinning our understanding of multimodal datasets and models [62].

By bringing together two previously disjoint research fields of Partial Information Decomposition (PID) in information theory [13, 41, 108] and multimodal machine learning [9, 62], we provide precise definitions categorizing interactions into redundancy , uniqueness , and synergy . Redundancy quantifies information shared between modalities, uniqueness quantifies the information present in only one of the modalities, and synergy quantifies the emergence of new information not previously present in either modality. A key aspect of these four measures is that they not only quantify interactions between modalities, but also how they relate to a downstream task. Figure 1 shows a depiction of these four measures, which we refer to as PID statistics. Leveraging insights from

neural representation learning, we propose two new estimators for PID statistics that can scale to high-dimensional multimodal datasets and models. The first estimator is exact, based on convex optimization, and is able to scale to features with discrete support, while the second estimator is an approximation based on sampling, which enables us to handle features with large discrete or even continuous supports. We validate our estimation of PID in 2 ways: (1) on synthetic datasets where PID statistics are known due to exact computation and from the nature of data generation, and (2) on real-world data where PID is compared with human annotation. Finally, we demonstrate that these estimated PID statistics can help in multimodal applications involving:

Dataset quantification : We apply PID to quantify large-scale multimodal datasets, showing that these estimates match common intuition for interpretable modalities (e.g., language, vision, and audio) and yield new insights in other domains (e.g, healthcare, HCI, and robotics).

Model quantification : Across a suite of models, we apply PID to interpret model predictions and find consistent patterns of interactions that different models capture.

Model selection : Given our findings from dataset and model quantification, a new research question naturally arises: given a new multimodal task, can we quantify its PID values to infer (a priori) what type of models are most suitable? Our experiments show success in model selection for both existing benchmarks and completely new case studies engaging with domain experts in computational pathology, mood prediction, and robotics to select the best multimodal model.

Finally, we make public a suite of trained models across 10 model families and 30 datasets to accelerate future analysis of multimodal interactions at https://github.com/pliang279/PID .

2 Background and Related Work

Let X i and Y be sample spaces for features and labels. Define ∆ to be the set of joint distributions over ( X 1 , X 2 , Y ) . We are concerned with features X 1 , X 2 (with support X i ) and labels Y (with support Y ) drawn from some distribution p ∈ ∆ . We denote the probability mass (or density) function by p ( x 1 , x 2 , y ) , where omitted parameters imply marginalization. Key to our work is defining estimators that given p or samples { ( x 1 , x 2 , y ) : X 1 × X 2 × Y} thereof (i.e., dataset or model predictions), returns estimates for the amount of redundant, unique, and synergistic interactions.

2.1 Partial Information Decomposition

Information theory formalizes the amount of information that one variable provides about another [86]. However, its extension to 3 variables is an open question [34, 70, 93, 106]. In particular, the natural three-way mutual information I ( X 1 ; X 2 ; Y ) = I ( X 1 ; X 2 ) -I ( X 1 ; X 2 | Y ) [70, 93] can be both positive and negative, which makes it difficult to interpret. In response, Partial information decomposition (PID) [108] generalizes information theory to multiple variables by decomposing I p ( X 1 , X 2 ; Y ) , the total information 2 vari-

R=I(X1;X2;Y)

I(X1;Y|X2)

I(X2; Y|X1)

S

X1

X2

U

R

U2

Y

U1 =I(X1;Y|X2)

U2 = I(X2; Y|X1)

I(X1; Y)

Y

I(X2; Y)

R=I(X1;X2;Y)

X1

X2

ables X 1 , X 2 provide about a task Y into 4 quantities (see Figure 1): redundancy R between X 1 and X 2 , uniqueness U 1 in X 1 and U 2 in X 2 , and synergy S that only emerges when both X 1 and X 2 are present. We adopt the PID definition proposed by Bertschinger et al. [13]:

where ∆ p = { q ∈ ∆ : q ( x i , y ) = p ( x i , y ) ∀ y ∈ Y , x i ∈ X i , i ∈ [2] } and the notation I p ( · ) and I q ( · ) disambiguates mutual information under p and q respectively. The key lies in optimizing q ∈ ∆ p to satisfy the marginals q ( x i , y ) = p ( x i , y ) , but relaxing the coupling between x 1 and x 2 : q ( x 1 , x 2 ) need not be equal to p ( x 1 , x 2 ) . The intuition behind this is that one should be able to infer redundancy and uniqueness given only access to p ( x 1 , y ) and p ( x 2 , y ) , and therefore they should only depend on q ∈ ∆ p . Synergy is the only term that should depend on the coupling p ( x 1 , x 2 ) , and this is reflected

in (3) depending on the full p distribution. This definition enjoys several useful properties in line with intuition, as we will see in comparison with related frameworks for interactions below [13].

2.2 Related Frameworks for Feature Interactions

Information-theoretic definitions : Perhaps the first measure of redundancy in machine learning is co-training [14, 8, 23], where 2 variables are redundant if they are conditionally independent given the task: I ( X 1 ; X 2 | Y ) = 0 . As a result, redundancy can be measured by I ( X 1 ; X 2 ; Y ) . The same definition of redundancy is used in multi-view learning [98, 94, 89] which further define I ( X 1 ; Y | X 2 ) and I ( X 2 ; Y | X 1 ) as unique information in X 1 , X 2 . However, I ( X 1 ; X 2 ; Y ) can be both positive and negative [51]. PID resolves this by separating R and S such that R -S = I ( X 1 ; X 2 ; Y ) , identifying that prior measures confound redundancy and synergy. This crucially provides an explanation for the distinction between mediation , where one feature conveys the information already in another (i.e., R > S ), versus moderation , where one feature affects the relationship of other features (i.e., S > R ) [10, 36]. Furthermore, if I ( X 1 ; X 2 ; Y ) = 0 then existing frameworks are unable to distinguish between positive R and S canceling each other out.

Statistical measures : Other approaches have studied interaction measures via statistical measures, such as redundancy via distance between prediction logits using either feature [69], statistical distribution tests on input features [114, 7], or via human annotations [85]. However, it is unclear how to extend these definitions to uniqueness and synergy while remaining on the same standardized scale like PID provides. Also of interest are notions of redundant and synergistic interactions in human and animal communication [78, 79, 30, 85], which we aim to formalize.

Model-based methods : Prior research has formalized definitions of non-additive interactions [32] to quantify their presence [88, 101, 102, 46] in trained models, or used Shapley values on trained features to measure interactions [49]. Parallel research has also focused on qualitative visualizations of realworld multimodal datasets and models, such as DIME [68], M2Lens [105], and MultiViz [61].

3 Scalable Estimators for PID

PID as a framework for multimodality : Our core insight is that PID provides a formal framework to understand both the nature and degree of interactions involved when two features X 1 and X 2 are used for task Y . The nature of interactions is afforded by a precise decomposition into redundant, unique, and synergistic interactions, and the degree of interactions is afforded by a standardized unit of measure (bits). However, computing PID is a considerable challenge, since it involves optimization over ∆ p and estimating information-theoretic measures. Up to now, analytic approximations of these quantities were only possible for discrete and small support [13, 41, 109] or continuous but low-dimensional variables [76, 81, 110]. Leveraging ideas in representation learning, Sections 3.1 and 3.2 are our first technical contributions enabling scalable estimation of PID for high-dimensional distributions. The first, CVX, is exact, based on convex optimization, and is able to scale to problems where |X i | and |Y| are around 100 . The second, BATCH, is an approximation based on sampling, which enables us to handle large or even continuous supports for X i and Y . Applying these estimators in Section 4, we show that PID provides a path towards understanding the nature of interactions in datasets and those learned by different models, and principled approaches for model selection.

3.1 CVX: Dataset-level Optimization

Our first estimator, CVX, directly compute PID from its definitions using convex programming. Crucially, Bertschinger et al. [13] show that the solution to the max-entropy optimization problem: q ∗ = arg max q ∈ ∆ p H q ( Y | X 1 , X 2 ) equivalently solves (1)-(3). When X i and Y are small and discrete, we can represent all valid distributions q ( x 1 , x 2 , y ) as a set of tensors Q of shape |X 1 | × |X 2 | × |Y| with each entry representing Q [ i, j, k ] = p ( X 1 = i, X 2 = j, Y = k ) . The problem then boils down to optimizing over valid tensors Q ∈ ∆ p that match the marginals p ( x i , y ) .

Given a tensor Q representing q , our objective is the concave function H q ( Y | X 1 , X 2 ) . While Bertschinger et al. [13] report that direct optimization is numerically difficult as routines such as Mathematica's FINDMINIMUM do not exploit convexity, we overcome this by rewriting conditional entropy as a KL-divergence [38], H q ( Y | X 1 , X 2 ) = log |Y| KL ( q || ˜ q ) , where ˜ q is an auxiliary product density of q ( x 1 , x 2 ) · 1 |Y| enforced using linear constraints: ˜ q ( x 1 , x 2 , y ) = q ( x 1 , x 2 ) / |Y| . Finally, optimizing over Q ∈ ∆ p that match the marginals can also be enforced through linear constraints: the 3D-tensor Q summed over the second dimension gives q ( x 1 , y ) and summed over

Sensors

Video

Sinkhorn's algorithm

Unnormalized

joint distribution

Training objective

PID values

p(ylc2)

C1

fo(1)

b

=

b

R=I-(X1;X2;Y)

(Ix/h)d

whereIa(X1;X2;Y)=

b

U1 =I-(X1;Y|X2)

I(X1;X2)-I(X1;X2|Y)

U2=I-(X2;Y|X1)

32

AW

S=Ip(X1,X2;Y)-I(X1,X2;Y)

f(2)

A = exp (f(1)(X1,y)fo(2)(X2,y)T)

=SINKHORNp(A)

the first dimension gives q ( x 2 , y ) , yielding the final optimization problem:

The KL-divergence objective is recognized as convex, allowing the use of conic solvers such as SCS [75], ECOS [27], and MOSEK [6]. Plugging q ∗ into (1)-(3) yields the desired PID.

Pre-processing via feature binning : In practice, X 1 and X 2 often take continuous rather than discrete values. We work around this by histogramming each X i , thereby estimating the continuous joint density by discrete distributions with finite support. We provide more details in Appendix B.1.

3.2 BATCH: Batch-level Amortization

We now present BATCH, our next estimator that is suitable for large datasets where X i is highdimensional and continuous ( |Y| remains finite). To estimate PID given a sampled dataset D = { ( x ( j ) 1 , x ( j ) 2 , y ( j ) ) } of size n , we propose an end-to-end model parameterizing marginal-matching joint distributions in ∆ p and a training objective whose solution returns approximate PID values.

Simplified algorithm sketch : Our goal, loosely speaking, is to optimize ˜ q ∈ ∆ p for objective (1) through an approximation using neural networks instead of exact optimization. We show an overview in Figure 2. To explain our approach, we first describe (1) how we parameterize ˜ q using neural networks such that it can be learned via gradient-based approaches, (2) how we ensure the marginal constraints ˜ q ∈ ∆ p through a variant of the Sinkhorn-Knopp algorithm, and finally (3) how to scale this up over small subsampled batches from large multimodal datasets.

Parameterization using neural networks : The space of joint distributions ∆ is often too large to explicitly specify. To tackle this, we implicitly parameterize each distribution ˜ q ∈ ∆ using a neural network f ϕ that takes in batches of modalities X 1 ∈ ˜ X n 1 , X 2 ∈ ˜ X n 2 and the label Y ∈ Y n before returning a matrix A ∈ R n × n ×|Y| representing an (unnormalized) joint distribution ˜ q , i.e., we want A [ i ][ j ][ y ] = ˜ q ( X 1 [ i ] , X 2 [ j ] , y ) for each y ∈ Y . In practice, f ϕ is implemented via a pair of encoders f ϕ (1) and f ϕ (2) that learn modality representations, before an outer product to learn joint relationships A y = exp( f ϕ (1) ( X 1 , y ) f ϕ (2) ( X 2 , y ) ⊤ ) for each y , yielding the desired n × n ×|Y| joint distribution. As a result, optimizing over ˜ q can be performed via optimizing over parameters ϕ .

Respecting the marginal constraints : How do we make sure the ˜ q 's learned by the network satisfies the marginal constraints (i.e., ˜ q ∈ ∆ p )? We use an unrolled version of Sinkhorn's algorithm [24] which projects A onto ∆ p by iteratively normalizing A 's rows and columns to sum to 1 and rescaling to satisfy the marginals p ( x i , y ) . However, p ( x i , y ) is not easy to estimate for high-dimensional continuous x i 's. In response, we first expand p ( x i , y ) into p ( y | x i ) and p ( x i ) using Bayes' rule. Since A was constructed by samples x i from the dataset, the rows and columns of A are already distributed according to p ( x 1 ) and p ( x 2 ) respectively. This means that it suffices to approximate p ( y | x i ) with unimodal classifiers ˆ p ( y | x i ) parameterized by neural networks and trained separately, before using Sinkhorn's algorithm to normalize each row to ˆ p ( y | x 1 ) and each column to ˆ p ( y | x 2 ) .

Objective : We choose the objective I q ( X 1 ; X 2 ; Y ) , which equivalently solves the optimization problems in the other PID terms [13]. Given matrix A representing ˜ q ( x 1 , x 2 , y ) , the objective can be computed in closed form through appropriate summation across dimensions in A to obtain ˜ q ( x i ) , ˜ q ( x 1 , x 2 ) , ˜ q ( x i | y ) , and ˜ q ( x 1 , x 2 | y ) and plugging into I ˜ q ( X 1 ; X 2 ; Y ) = I ˜ q ( X 1 ; X 2 ) -I ˜ q ( X 1 ; X 2 | Y ) . We maximize I ˜ q ( X 1 ; X 2 ; Y ) by updating parameters ϕ via gradient-based methods. Overall, each

5

0

5

(a)

5

0

5

0

1

(b)

0.0

2.5

5.0

7.5 1e

1

0

1

(c)

0.0

2.5

5.0

7.5 1e

1

5

0

5

(d)

5

0

5

0

1

(e)

0.0

2.5

5.0

7.5 1e

1

0

1

(f)

0.0

2.5

5.0

7.5 1e

1

gradient step involves computing ˜ q = SINKHORN ˆ p ( A ) , and updating ϕ to maximize (1) under ˜ q . Since Sinkhorn's algorithm is differentiable, gradients can be backpropagated end-to-end.

Approximation with small subsampled batches : Finally, to scale this up to large multimodal datasets where the full ˜ q may be too large to store, we approximate ˜ q with small subsampled batches: for each gradient iteration t , the network f ϕ now takes in a batch of m ≪ n datapoints sampled from D and returns A ∈ R m × m ×|Y| for the subsampled points. We perform Sinkhorn's algorithm on A and a gradient step on ϕ as above, as if D t was the full dataset (i.e., mini-batch gradient descent). While it is challenging to obtain full-batch gradients since computing the full A is intractable, we found our approach to work in practice for large m . Our approach can also be informally viewed as performing amortized optimization [3] by using ϕ to implicitly share information about the full batch using subsampled batches. Upon convergence of ϕ , we extract PID by plugging ˜ q into (1)-(3).

Implementation details such as the network architecture of f , approximation of objective (1) via sampling from ˜ q , and estimation of I ˜ q ( { X 1 , X 2 } ; Y ) from learned ˜ q are in Appendix B.3.

4 Evaluation and Applications of PID in Multimodal Learning

We design experiments to (1) understand PID on synthetic data, (2) quantify real-world multimodal benchmarks, (3) understand the interactions captured by multimodal models, (4) perform model selection across different model families, and (5) applications on novel real-world tasks.

4.1 Validating PID Estimates on Synthetic Data

Our first goal is to evaluate the accuracy of our proposed estimators with respect to the ground truth (if it can be computed) or human judgment (for cases where the ground truth cannot be readily obtained). We start with a suite of datasets spanning both synthetic and real-world distributions.

Synthetic bitwise features : We sample from a binary bitwise distribution: x 1 , x 2 ∼ { 0 , 1 } , y = x 1 ∧ x 2 , y = x 1 ∨ x 2 , y = x 1 ⊕ x 2 , . Each bitwise operator's PID can be solved exactly when the x i 's and labels are discrete and low-dimensional [13]. Compared to the ground truth in Bertschinger et al. [13], both our estimators exactly recover the correct PID values (Table 1).

Gaussian Mixture Models (GMM) : Consider a GMM, where X 1 , X 2 ∈ R and the label Y ∈ {-1 , +1 } , comprising two equally weighted standard multivariate Gaussians centered at ± µ , where µ ∈ R 2 , i.e., Y ∼ Bernoulli (1 / 2) , ( X 1 , X 2 ) | Y = y ∼ N ( y · µ, I ) . PID was estimated by sampling 1 e 6 points, histogramming them into 50 bins spanning [ -5 , +5] to give p , and then applying the CVX estimator. We term this PIDCartesian . We also compute PIDPolar , which are PID computed using polar coordinates , ( r, θ ) . We use a variant where the angle θ is given by the arctangent with principal values [0 , π ] and the length r ∈ R could be negative. θ specifies a line (through the origin), and r tells us where along the line the datapoint lies on.

Results: We consider || µ || 2 ∈ { 1 . 0 , 2 . 0 } , where for each || µ || 2 , we vary the angle ∠ µ that µ makes with the horizontal axis. Our computed PID is presented in Figure 3. Overall, we find that the PID matches what we expect from intuition. For Cartesian , unique information dominates when

the angle goes to 0 or π/ 2 -if centroids share a coordinate, then observing that coordinate yields no information about y . Conversely, synergy and redundancy peak at π/ 4 . Interestingly, synergy seems to be independent of || µ || 2 . For Polar , redundancy is 0 . Furthermore, θ contains no unique information, since θ shows nothing about y unless we know r (in particular, its sign). When the angle goes to π/ 2 , almost all information is unique in r . The distinctions between Cartesian and Polar highlight how different representations of data can exhibit wildly different PID values, even if total information is the same. More thorough results and visualizations of q ∗ are in Appendix C.2.

Synthetic generative model : We begin with a set of latent vectors z 1 , z 2 , z c ∼ N (0 d , Σ 2 d ) , d = 50 representing information unique to X 1 , X 2 and common to both respectively. [ z 1 , z c ] is transformed into high-dimensional x 1 using a fixed transformation T 1 and likewise [ z 2 , z c ] to x 2 via T 2 . The label y is generated as a function of (1) only z c , in which case we expect complete redundancy, (2) only z 1 or z 2 which represents complete uniqueness, (3) a combination of z 1 and z 2 representing complete synergy, or (4) arbitrary ratios of each of the above with z ∗ i representing half of the dimensions from z i and therefore half of each interaction. In total, Table 2 shows the 10 synthetic datasets we generated: 4 specialized datasets D I , I ∈ { R,U 1 , U 2 , S } where y only depends on one interaction, and 6 mixed datasets with varying interaction ratios. We also report the ground-truth interactions as defined by the label-generating process and the total capturable information using the bound in Feder and Merhav [29], which relates the accuracy of the best model on these datasets with the mutual information between the inputs to the label. Since the test accuracies for Table 2 datasets range from 67 -75% , this corresponds to total MI of 0 . 42 -0 . 59 bits.

Results : From Table 2, both CVX and BATCH agree in relative PID values, correctly assigning the predominant interaction type and interactions with minimal presence consistent with the ground-truth based on data generation. For example, D R has the highest R value, and when the ratio of z 1 increases, U 1 increases from 0 . 01 on y = f ( z ∗ 1 , z ∗ 2 , z ∗ c ) to 0 . 06 on y = f ( z 1 , z ∗ 2 , z ∗ c ) . We also note some interesting observations due to the random noise in label generation, such as the non-zero synergy measure of datasets such as D R , D U 1 , D U 2 whose labels do not depend on synergy.

4.2 Quantifying Real-world Multimodal Benchmarks

We now apply these estimators to quantify the interactions in real-world multimodal datasets.

Real-world multimodal data setup : We use a large collection of real-world datasets in MultiBench [60] which test multimodal fusion of different input signals (including images, video, audio, text, time-series, sets, and tables) for different tasks (predicting humor, sentiment, emotions, mortality rate, ICD9 codes, image-captions, human activities, digits, and design interfaces). We also include experiments on question-answering (Visual Question Answering 2.0 [5, 39] and CLEVR [53]) which test grounding of language into the visual domain. For the 4 datasets (top row of Table 3) involving images and text where modality features are available and readily clustered, we apply the CVX estimator on top of discrete clusters. For the remaining 4 datasets (bottom row of Table 3) with video, audio, and medical time-series modalities, clustering is not easy, so we use the end-to-end BATCH estimator (see Appendix C.4 for full dataset and estimator details).

Human judgment of interactions : Real-world multimodal datasets do not have reference PID values, and exact PID computation is impossible due to continuous data. We therefore use human judgment as a reference. We design a new annotation scheme where we show both modalities and the label and ask each annotator to annotate the degree of redundancy, uniqueness, and synergy on a scale of 0 -5 , alongside their confidence in their answers on a scale of 0 -5 . We show a sample

user interface and annotation procedures in Appendix C.5. We give 50 datapoints from each dataset (except MIMIC and ENRICO which require specialized knowledge) to 3 annotators each.

Results on multimodal fusion : From Table 3, we find that different datasets do require different interactions. Some interesting observations: (1) all pairs of modalities on MUSTARD sarcasm detection show high synergy values, which aligns with intuition since sarcasm is often due to a contradiction between what is expressed in language and speech, (2) uniqueness values are strongly correlated with unimodal performance (e.g., modality 1 in AV-MNIST and MIMIC), (3) datasets with high synergy do indeed benefit from interaction modeling as also seen in prior work (e.g., MUSTARD, UR-FUNNY) [17, 43], and (4) conversely datasets with low synergy are those where unimodal performance is relatively strong (e.g., MIMIC) [60].

Results on QA : We observe very high synergy values as shown in Table 3 consistent with prior work studying how these datasets were balanced (e.g., VQA 2.0 having different images for the same question such that the answer can only be obtained through synergy) [39] and that models trained on these datasets require non-additive interactions [46]. CLEVR has a higher proportion of synergy than VQA 2.0 ( 83% versus 75% ): indeed, CLEVR is a more balanced dataset where the answer strictly depends on both the question and image with a lower likelihood of unimodal biases.

Comparisons with human judgment : For human judgment, we cannot ask humans to give a score in bits, so it is on a completely different scale ( 0 -5 scale). To put them on the same scale, we normalize the human ratings such that the sum of human interactions is equal to the sum of PID estimates. The resulting comparisons are in Table 3, and we find that the human-annotated interactions overall align with estimated PID: the highest values are the same for 4 datasets: both explain highest synergy on VQAand CLEVR, image ( U 1 ) being the dominant modality in AV-MNIST, and language ( U 1 ) being the dominant modality in MOSEI. Overall, the Krippendorff's alpha for inter-annotator agreement is high ( 0 . 72 for R , 0 . 68 for U 1 , 0 . 70 for U 2 , 0 . 72 for S ) and the average confidence scores are also high ( 4 . 36 / 5 for R , 4 . 35 / 5 for U 1 , 4 . 27 / 5 for U 2 , 4 . 46 / 5 for S ), indicating that the human-annotated results are reliable. For the remaining two datasets (UR-FUNNY and MUSTARD), estimated PID matches the second-highest human-annotated interaction. We believe this is because there is some annotator subjectivity in interpreting whether sentiment, humor, and sarcasm are present in language only ( U 1 ) or when contextualizing both language and video ( S ), resulting in cases of low annotator agreement in U 1 and S : -0 . 14 , -0 . 03 for UR-FUNNY and -0 . 08 , -0 . 04 for MUSTARD.

Comparisons with other interaction measures : Our framework allows for easy generalization to other interaction definitions: we also implemented 3 information theoretic measures I-min [108], WMS [19], and CI [74]. These results are in Table 10 in the Appendix, where we explain the limitations of these methods as compared to PID, such as over- and under-estimation, and potential negative estimation [41]. These are critical problems with the application of information theory for shared I ( X 1 ; X 2 ; Y ) and unique information I ( X 1 ; Y | X 2 ) , I ( X 2 ; Y | X 1 ) often quoted in the co-training [8, 14] and multi-view learning [89, 94, 98] literature. We also tried 3 non-info theory measures: Shapley values [67], Integrated gradients (IG) [91], and CCA [4], which are based on quantifying interactions captured by a multimodal model. Our work is fundamentally different in that interactions are properties of data before training any models (see Appendix C.6).

4.3 Quantifying Multimodal Model Predictions

We now shift our focus to quantifying multimodal models. Do different multimodal models learn different interactions? A better understanding of the types of interactions that our current models struggle to capture can provide new insights into improving these models.

Setup : For each dataset, we train a suite of models on the train set D train and apply it to the validation set D val, yielding a predicted dataset D pred = { ( x 1 , x 2 , ˆ y ) ∈ D val } . Running PID on D pred summarizes the interactions that the model captures. We categorize and implement a comprehensive suite of models (spanning representation fusion at different feature levels, types of interaction inductive biases, and training objectives) that have been previously motivated to capture redundant, unique, and synergistic interactions (see Appendix C.7 for full model descriptions).

Results : We show results in Table 4 and highlight the following observations:

General observations : Wefirst observe that model PID values are consistently higher than dataset PID. The sum of model PID is also a good indicator of test performance, which agrees with their formal definition since their sum is equal to I ( { X 1 , X 2 } ; Y ) , the total task-relevant information.

On redundancy : Several methods succeed in capturing redundancy, with an overall average of R = 0 . 41 ± 0 . 11 and accuracy of 73 . 0 ± 2 . 0% on redundancy-specialized datasets. Additive, agreement, and alignment-based methods are particularly strong, and we do expect them to capture redundant shared information [26, 82]. Methods based on tensor fusion (synergy-based), including lower-order interactions, and adding reconstruction objectives (unique-based) also capture redundancy.

On uniqueness : Uniqueness is harder to capture than redundancy, with an average of U = 0 . 37 ± 0 . 14 . Redundancy-based methods like additive and agreement do poorly on uniqueness, while those designed for uniqueness (lower-order interactions [115] and modality reconstruction objectives [100]) do well, with on average U = 0 . 55 and 73 . 0% accuracy on uniqueness datasets.

On synergy : Synergy is the hardest to capture, with an average score of only S = 0 . 21 ± 0 . 10 . Some of the strong methods are tensor fusion [33], tensors with lower-order interactions [115], modality reconstruction [100], and multimodal transformer [112], which achieve around S = 0 . 30 , acc = 73 . 0% . Additive, agreement, and element-wise interactions do not seem to capture synergy well.

On robustness : Finally, we also show connections between PID and model performance in the presence of missing modalities. We find high correlation ( ρ = 0 . 8 ) between the performance drop when X i is missing and the model's U i value. Inspecting Figure 4, we find that the implication only holds in one direction: high U i coincides with large performance drops (in red), but low U i can also lead to performance drops (in green). The latter can be further explained by the presence of large S values:

when X i is missing, synergy can no longer be learned which affects performance. For the subset of points when U i ≤ 0 . 05 , the correlation between S and performance drop is ρ = 0 . 73 (in contrast, the correlation for R is ρ = 0 . 01 ).

4.4 PID Agreement and Model Selection

Now that we have quantified datasets and models individually, the natural next question unifies both: what does the agreement between dataset and model PID measures tell us about model performance? We hypothesize that models able to capture the interactions necessary in a given dataset should also

achieve high performance. Given estimated interactions on dataset D and model f ( D ) trained on D , we define the agreement for each interaction I ∈ { R,U 1 , U 2 , S } as:

which summarizes the quantity of an interaction captured by a model ( I f ( D ) ) weighted by its normalized importance in the dataset ( ˆ I D ). The total agreement sums over α ( f, D ) = ∑ I α I ( f, D ) .

Results : Our key finding is that PID agreement scores α ( f, D ) correlate ( ρ = 0 . 81 ) with model accuracy across all 10 synthetic datasets as illustrated in Figure 5. This shows that PID agreement can be a useful proxy for model performance. For the specialized datasets, we find that the correlation between α I and D I is 0 . 96 for R , 0 . 86 for U , and 0 . 91 for S , and negatively correlated with other specialized datasets. For mixed datasets with roughly equal ratios of each interaction, the measures that correlate most with performance are α R ( ρ = 0 . 82 ) and α S ( ρ = 0 . 89 ); datasets with relatively higher redundancy see ρ = 0 . 89 for α R ; those with higher uniqueness have α U 1 and α U 2 correlate ρ = 0 . 92 and ρ = 0 . 85 ; those with higher synergy increases the correlation of α S to ρ = 0 . 97 .

Using these observations, our final experiment is model selection: can we choose the most appropriate model to tackle the interactions required for a dataset?

Setup : Given a new dataset D , we first compute its difference in normalized PID values with respect to D ′ among our suite of 10 synthetic datasets, s ( D , D ′ ) = ∑ I ∈{ R,U 1 ,U 2 ,S } | ˆ I D -ˆ I D ′ | , to rank the dataset D ∗ with the most similar interactions, and return the top3 performing models on D ∗ . In other words, we select models that best capture interactions that are of similar nature and degree as those in D . We emphasize that even though we restrict dataset and model search to synthetic datasets , we evaluate model selection on real-world datasets and find that it generalizes to the real world .

Results : We test our selected models on 5 new synthetic datasets with different PID ratios and 6 real-world datasets, summarizing results in Table 5. We find that the top 3 chosen models achieve 95% -100% of the best-performing model accuracy, and > 98 . 5% for all datasets except 95 . 2% on MUSTARD. For example, UR-FUNNY and MUSTARD have the highest synergy ( S = 0 . 13 , S = 0 . 3 ) and indeed transformers and higher-order interactions are helpful (MULT: 65% , MI: 61% , TENSOR: 60% ). ENRICO has the highest R = 0 . 73 and U 2 = 0 . 53 , and methods for redundant and unique interactions perform best (LOWER: 52% , ALIGN: 52% , AGREE: 51% ). MIMIC has the highest U 1 = 0 . 17 , and unimodal models are mostly sufficient [60].

4.5 Real-world Applications

Finally, we apply PID to 3 real-world case studies: pathology, mental health, and robotic perception (see full details and results in Appendix C.9-C.11).

Case Study 1: Computational pathology. Cancer prognostication is a challenging task in anatomic pathology that requires integration of whole-slide imaging (WSI) and molecular features for patient stratification [21, 63, 71]. We use The Cancer Genome Atlas (TCGA), a large public data consortium of paired WSI, molecular, and survival information [107, 96], including modalities: (1) pre-extracted histology image features from diagnostic WSIs and (2) bulk gene mutation status, copy number variation, and RNA-Seq abundance values. We evaluate on two cancer datasets in TCGA, lower-grade glioma (LGG [73], n = 479 ) and pancreatic adenocarcinoma (PAAD [83], n = 209 ).

Results : In TCGA-LGG, most PID measures were near-zero except U 2 = 0 . 06 for genomic features, which indicates that genomics is the only modality containing task-relevant information. This conclusion corroborates with the high performance of unimodal-genomic and multimodal models in Chen et al. [22], while unimodal-pathology performance was low. In TCGA-PAAD, the uniqueness in pathology and genomic features was less than synergy ( U 1 = 0 . 06 , and U 2 = 0 . 08 and S = 0 . 15 ), which also match the improvement of using multimodal models that capture synergy.

Case Study 2: Mental health. Suicide is the second leading cause of death among adolescents [18]. Intensive monitoring of behaviors via adolescents' frequent use of smartphones may shed new light on the early risk of suicidal ideations [37, 72], since smartphones provide rich behavioral markers [59]. We used a dataset, MAPS, of mobile behaviors from high-risk consenting adolescent populations (approved by IRB). Passive sensing data is collected from each participant's smartphone across 6 months. The modalities include (1) text entered by the user represented as a bag of top 1000 words, (2) keystrokes that record the exact timing and duration of each typed character, and (3) mobile applications used per day as a bag of 137 apps. Every morning, users self-report their daily mood, which we discretized into -1 , 0 , +1 . In total, MAPS has 844 samples from 17 participants.

Results : We first experiment with MAPS T,K using text and keystroke features. PID measures show that MAPS T,K has high synergy ( 0 . 40 ), some redundancy ( 0 . 12 ), and low uniqueness ( 0 . 04 ). We found the purely synergistic dataset D S has the most similar interactions and the suggested models LOWER, REC, and TENSOR that work best on D S were indeed the top 3 best-performing models on MAPS T,K , indicating that model selection is effective. Model selection also retrieves the best-performing model on MAPS T,A using text and app usage features.

Case Study 3: Robotic Perception. MuJoCo PUSH [57] is a contact-rich planar pushing task in MuJoCo [95], where a 7 -DoF Panda Franka robot is pushing a circular puck with its end-effector in simulation. The dataset consists of 1000 trajectories with 250 steps sampled at 10 Hertz. The multimodal inputs are gray-scaled images from an RGB camera, force and binary contact information from a force/torque sensor, and the 3D position of the robot end-effector. We estimate the 2D position of the unknown object on a table surface while the robot intermittently interacts with it.

Results : We find that BATCH predicts U 1 = 1 . 79 as the highest PID value, which aligns with our observation that image is the best unimodal predictor. Comparing both estimators, CVX underestimates U 1 and R since the high-dimensional time-series modality cannot be easily described by clusters without losing information. In addition, both estimators predict a low U 2 value but attribute high R , implying that a multimodal model with higher-order interactions would not be much better than unimodal models. Indeed, we observe no difference in performance between these two.

5 Conclusion

Our work aims to quantify the nature and degree of feature interactions by proposing scalable estimators for redundancy, uniqueness, and synergy suitable for high-dimensional heterogeneous datasets. Through comprehensive experiments and real-world applications, we demonstrate the utility of our proposed framework in dataset quantification, model quantification, and model selection. We are aware of some potential limitations :

These estimators only approximate real interactions due to cluster preprocessing or unimodal models, which naturally introduce optimization and generalization errors. We expect progress in density estimators, generative models, and unimodal classifiers to address these problems.

It is harder to quantify interactions for certain datasets, such as ENRICO which displays all interactions which makes it difficult to distinguish between R and S or U and S .

Finally, there exist challenges in quantifying interactions since the data generation process is never known for real-world datasets, so we have to resort to human judgment, other automatic measures, and downstream tasks such as estimating model performance and model selection.

Future work can leverage PID for targeted dataset creation, representation learning optimized for PID values, and applications of information theory to higher-dimensional data. More broadly, there are several exciting directions in investigating more applications of multivariate information theory in modeling feature interactions, predicting multimodal performance, and other tasks involving feature interactions such as privacy-preserving and fair representation learning from high-dimensional data [28, 42]. Being able to provide guarantees for fairness and privacy-preserving learning can be particularly impactful.

References

Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. A rewriting system for convex optimization problems. Journal of Control and Decision , 5(1):42-60, 2018.

Paul D Allison. Testing for interaction in multiple regression. American journal of sociology , 1977.

Brandon Amos. Tutorial on amortized optimization for learning to optimize over continuous domains. arXiv preprint arXiv:2202.00665 , 2022.

Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning , pages 1247-1255. PMLR, 2013.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425-2433, 2015.

MOSEK ApS. MOSEK Optimizer API for Python 10.0.34 , 2022.

Benjamin Auffarth, Maite L´ opez, and Jes´ us Cerquides. Comparison of redundancy and relevance measures for feature selection in tissue classification of ct images. In Industrial conference on data mining , pages 248-262. Springer, 2010.

Maria-Florina Balcan, Avrim Blum, and Ke Yang. Co-training and expansion: Towards bridging theory and practice. Advances in neural information processing systems , 17, 2004.

Tadas Baltruˇ saitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence , 2018.

Reuben M Baron and David A Kenny. The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of personality and social psychology , 51(6):1173, 1986.

Jan Beirlant, Edward J Dudewicz, L´ aszl´ o Gy¨ orfi, Edward C Van der Meulen, et al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences , 1997.

Anthony J Bell. The co-information lattice. In Proceedings of the fifth international workshop on independent component analysis and blind signal separation: ICA , volume 2003, 2003.

Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, J¨ urgen Jost, and Nihat Ay. Quantifying unique information. Entropy , 16(4):2161-2183, 2014.

Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory , pages 92-100, 1998.

Cameron W Brennan, Roel GW Verhaak, Aaron McKenna, Benito Campos, Houtan Noushmehr, Sofie R Salama, Siyuan Zheng, Debyani Chakravarty, J Zachary Sanborn, Samuel H Berman, et al. The somatic genomic landscape of glioblastoma. cell , 155(2):462-477, 2013.

Nancy Ann Oberheim Bush, Susan M Chang, and Mitchel S Berger. Current and future strategies for treatment of glioma. Neurosurgical review , 40(1):1-14, 2017.

Santiago Castro, Devamanyu Hazarika, Ver´ onica P´ erez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an obviously perfect paper). In ACL , 2019.

CDC. CDC WONDER: Underlying cause of death, 1999-2019. 2020.

Gal Chechik, Amir Globerson, M Anderson, E Young, Israel Nelken, and Naftali Tishby. Group redundancy measures reveal redundancy reduction in the auditory pathway. Advances in neural information processing systems , 14, 2001.

Richard J Chen, Ming Y Lu, Jingwen Wang, Drew FK Williamson, Scott J Rodig, Neal I Lindeman, and Faisal Mahmood. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging , 2020.

Richard J Chen, Ming Y Lu, Wei-Hung Weng, Tiffany Y Chen, Drew FK Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015-4025, 2021.

Richard J Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell , 40(8):865-878, 2022.

C Mario Christoudias, Raquel Urtasun, and Trevor Darrell. Multi-view learning in the presence of view disagreement. In Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence , pages 88-96, 2008.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems , 26, 2013.

Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research , 17(83):1-5, 2016.

Daisy Yi Ding, Shuangning Li, Balasubramanian Narasimhan, and Robert Tibshirani. Cooperative learning for multiview analysis. Proceedings of the National Academy of Sciences , 2022.

Alexander Domahidi, Eric Chu, and Stephen Boyd. Ecos: An socp solver for embedded systems. In 2013 European Control Conference (ECC) , pages 3071-3076. IEEE, 2013.

Sanghamitra Dutta, Praveen Venkatesh, Piotr Mardziel, Anupam Datta, and Pulkit Grover. An informationtheoretic quantification of discrimination with exempt features. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3825-3833, 2020.

Meir Feder and Neri Merhav. Relations between entropy and error probability. IEEE Transactions on Information theory , 40(1):259-266, 1994.

Ross Flom and Lorraine E Bahrick. The development of infant discrimination of affect in multimodal and unimodal stimulation: The role of intersensory redundancy. Developmental psychology , 43(1):238, 2007.

Joseph C Franklin, Jessica D Ribeiro, Kathryn R Fox, Kate H Bentley, Evan M Kleiman, Xieyining Huang, Katherine M Musacchio, Adam C Jaroszewski, Bernard P Chang, and Matthew K Nock. Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychological bulletin , 2017.

Jerome H Friedman and Bogdan E Popescu. Predictive learning via rule ensembles. The annals of applied statistics , 2(3):916-954, 2008.

Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Conference on Empirical Methods in Natural Language Processing , pages 457-468. ACL, 2016.

Wendell R Garner. Uncertainty and structure as psychological concepts. 1962.

Timothy J Gawne and Barry J Richmond. How independent are the messages carried by adjacent inferior temporal cortical neurons? Journal of Neuroscience , 13(7):2758-2771, 1993.

AmirEmad Ghassami and Negar Kiyavash. Interaction information for causal inference: The case of directed triangle. In IEEE International Symposium on Information Theory (ISIT) , 2017.

Catherine R Glenn and Matthew K Nock. Improving the short-term prediction of suicidal behavior. American journal of preventive medicine , 2014.

Amir Globerson and Tommi Jaakkola. Approximate inference using conditional entropy decompositions. In Artificial Intelligence and Statistics , pages 131-138. PMLR, 2007.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6904-6913, 2017.

Michael Grant, Stephen Boyd, and Yinyu Ye. Disciplined convex programming. In Global optimization , pages 155-210. Springer, 2006.

Virgil Griffith and Christof Koch. Quantifying synergistic mutual information. In Guided self-organization: inception , pages 159-190. Springer, 2014.

Faisal Hamman and Sanghamitra Dutta. Demystifying local and global fairness trade-offs in federated learning using information theory. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities , 2023.

Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, and Mohammed Ehsan Hoque. Ur-funny: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 2046-2056, 2019.

Trevor Hastie and Robert Tibshirani. Generalized additive models: some applications. Journal of the American Statistical Association , 82(398):371-386, 1987.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Jack Hessel and Lillian Lee. Does my multimodal model learn cross-modal interactions? it's harder to tell than you might think! In EMNLP , 2020.

Ming Hou, Jiajia Tang, Jianhai Zhang, Wanzeng Kong, and Qibin Zhao. Deep multimodal multilinear fusion with high-order polynomial pooling. Advances in Neural Information Processing Systems , 32: 12136-12145, 2019.

Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning , pages 2127-2136. PMLR, 2018.

Jan Ittner, Lukasz Bolikowski, Konstantin Hemker, and Ricardo Kennedy. Feature synergy, redundancy, and independence in global model explanations using shap vector decomposition. arXiv preprint arXiv:2107.12436 , 2021.

James Jaccard and Robert Turrisi. Interaction effects in multiple regression . Number 72. sage, 2003.

Aleks Jakulin and Ivan Bratko. Quantifying and visualizing attribute interactions: An approach based on entropy. 2003.

Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In International Conference on Learning Representations , 2020.

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2901-2910, 2017.

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning , pages 5583-5594. PMLR, 2021.

G¨ unter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems , 30, 2017.

Matthew Michael Large, Daniel Thomas Chung, Michael Davidson, Mark Weiser, and Christopher James Ryan. In-patient suicide: selection of people at risk, failure of protection and the possibility of causation. BJPsych Open , 3(3):102-105, 2017.

Michelle A Lee, Brent Yi, Roberto Mart´ ın-Mart´ ın, Silvio Savarese, and Jeannette Bohg. Multimodal sensor fusion with differentiable filters. IROS , 2020.

Paul Pu Liang, Zhun Liu, Yao-Hung Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, and Louis-Philippe Morency. Learning representations from imperfect time series data via tensor rank regularization. In ACL , 2019.

Paul Pu Liang, Terrance Liu, Anna Cai, Michal Muszynski, Ryo Ishii, Nick Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Morency. Learning language and multimodal privacy-preserving markers of mood from mobile data. In ACL , 2021.

Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021.

Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multiviz: An analysis benchmark for visualizing and understanding multimodal models. arXiv preprint arXiv:2207.00056 , 2022.

Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.03430 , 2022.

Jana Lipkova, Richard J Chen, Bowen Chen, Ming Y Lu, Matteo Barbieri, Daniel Shao, Anurag J Vaidya, Chengkuan Chen, Luoting Zhuang, Drew FK Williamson, et al. Artificial intelligence for multimodal data integration in oncology. Cancer Cell , 40(10):1095-1110, 2022.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.

Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2247-2256, 2018.

David N Louis, Arie Perry, Guido Reifenberger, Andreas Von Deimling, Dominique Figarella-Branger, Webster K Cavenee, Hiroko Ohgaki, Otmar D Wiestler, Paul Kleihues, and David W Ellison. The 2016 world health organization classification of tumors of the central nervous system: a summary. Acta neuropathologica , 131(6):803-820, 2016.

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems , 30, 2017.

Yiwei Lyu, Paul Pu Liang, Zihao Deng, Ruslan Salakhutdinov, and Louis-Philippe Morency. Dime: Fine-grained interpretations of multimodal models via disentangled local explanations. AIES, 2022.

Alessio Mazzetto, Dylan Sam, Andrew Park, Eli Upfal, and Stephen Bach. Semi-supervised aggregation of dependent weak supervision sources with performance guarantees. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , 2021.

William McGill. Multivariate information transmission. Transactions of the IRE Professional Group on Information Theory , 4(4):93-111, 1954.

Pooya Mobadersany, Safoora Yousefi, Mohamed Amgad, David A Gutman, Jill S Barnholtz-Sloan, Jos´ e E Vel´ azquez Vega, Daniel J Brat, and Lee AD Cooper. Predicting cancer outcomes from histology and genomics using convolutional networks. Proceedings of the National Academy of Sciences , 2018.

Inbal Nahum-Shani, Shawna N Smith, Bonnie J Spring, Linda M Collins, Katie Witkiewitz, Ambuj Tewari, and Susan A Murphy. Just-in-time adaptive interventions (jitais) in mobile health: key components and design principles for ongoing health behavior support. Annals of Behavioral Medicine , 2018.

Cancer Genome Atlas Research Network. Comprehensive, integrative genomic analysis of diffuse lower-grade gliomas. New England Journal of Medicine , 372(26):2481-2498, 2015.

Sheila Nirenberg, Steve M Carcieri, Adam L Jacobs, and Peter E Latham. Retinal ganglion cells act largely as independent encoders. Nature , 411(6838):698-701, 2001.

Brendan O'Donoghue, Eric Chu, Neal Parikh, and Stephen Boyd. Conic optimization via operator splitting and homogeneous self-dual embedding. Journal of Optimization Theory and Applications , 2016.

Ari Pakman, Amin Nejatbakhsh, Dar Gilboa, Abdullah Makkeh, Luca Mazzucato, Michael Wibral, and Elad Schneidman. Estimating the unique information of continuous variables. Advances in Neural Information Processing Systems , 34:20295-20307, 2021.

Liam Paninski. Estimation of entropy and mutual information. Neural computation , 2003.

Sarah Partan and Peter Marler. Communication goes multimodal. Science , 283(5406):1272-1273, 1999.

Sarah R Partan and Peter Marler. Issues in the classification of multimodal communication signals. The American Naturalist , 166(2):231-245, 2005.

Fernando P´ erez-Cruz. Kullback-leibler divergence estimation of continuous distributions. In 2008 IEEE international symposium on information theory , pages 1666-1670. IEEE, 2008.

Alexandra M Proca, Fernando E Rosas, Andrea I Luppi, Daniel Bor, Matthew Crosby, and Pedro AM Mediano. Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks. arXiv preprint arXiv:2210.02996 , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Benjamin J Raphael, Ralph H Hruban, Andrew J Aguirre, Richard A Moffitt, Jen Jen Yeh, Chip Stewart, A Gordon Robertson, Andrew D Cherniack, Manaswi Gupta, Gad Getz, et al. Integrated genomic characterization of pancreatic ductal adenocarcinoma. Cancer cell , 32(2):185-203, 2017.

John A Rice. Mathematical statistics and data analysis . Cengage Learning, 2006.

Natalie Ruiz, Ronnie Taib, and Fang Chen. Examining the redundancy of multimodal input. In Proceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments , pages 389-392, 2006.

Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal , 27(3):379-423, 1948.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2014. URL https://arxiv.org/abs/1409.1556 .

Daria Sorokina, Rich Caruana, Mirek Riedewald, and Daniel Fink. Detecting statistical interactions with additive groves of trees. In Proceedings of the 25th international conference on Machine learning , 2008.

Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. In Conference on Learning Theory , 2008.

Milan Studen` y and Jirina Vejnarov´ a. The multiinformation function as a tool for measuring stochastic dependence. In Learning in graphical models , pages 261-297. Springer, 1998.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning , pages 3319-3328. PMLR, 2017.

Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint arXiv:1611.01891 , 2016.

Han Te Sun. Multiple mutual informations and multiple interactions in frequency data. Inf. Control , 46: 26-45, 1980.

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems , 2020.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems . IEEE, 2012.

Katarzyna Tomczak, Patrycja Czerwi´ nska, and Maciej Wiznerowicz. The cancer genome atlas (tcga): an immeasurable source of knowledge. Contemporary Oncology/Wsp´ ołczesna Onkologia , 2015.

Giulio Tononi, Olaf Sporns, and Gerald M Edelman. A measure for brain complexity: relating functional segregation and integration in the nervous system. Proceedings of the National Academy of Sciences , 91 (11):5033-5037, 1994.

Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory , pages 1179-1206. PMLR, 2021.

Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 6558-6569, 2019.

Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. Learning factorized multimodal representations. In International Conference on Learning Representations , 2019.

Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. In International Conference on Learning Representations , 2018.

Michael Tsang, Dehua Cheng, Hanpeng Liu, Xue Feng, Eric Zhou, and Yan Liu. Feature interaction interpretability: A case for explaining ad-recommendation systems via neural interaction detection. In International Conference on Learning Representations , 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , 2017.

Qing Wang, Sanjeev R Kulkarni, and Sergio Verd´ u. A nearest-neighbor approach to estimating divergence between continuous random vectors. In 2006 IEEE International Symposium on Information Theory , pages 242-246. IEEE, 2006.

Xingbo Wang, Jianben He, Zhihua Jin, Muqiao Yang, Yong Wang, and Huamin Qu. M2lens: Visualizing and explaining multimodal models for sentiment analysis. IEEE Transactions on Visualization and Computer Graphics , 28(1):802-812, 2021.

Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of research and development , 4(1):66-82, 1960.

John N Weinstein, Eric A Collisson, Gordon B Mills, Kenna R Shaw, Brad A Ozenberger, Kyle Ellrott, Ilya Shmulevich, Chris Sander, and Joshua M Stuart. The cancer genome atlas pan-cancer analysis project. Nature genetics , 45(10):1113-1120, 2013.

Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515 , 2010.

Patricia Wollstadt, Joseph Lizier, Raul Vicente, Conor Finn, Mario Martinez-Zarzuela, Pedro Mediano, Leonardo Novelli, and Michael Wibral. Idtxl: The information dynamics toolkit xl: a python package for the efficient analysis of multivariate information dynamics in networks. Journal of Open Source Software , 4(34):1081, 2019.

Patricia Wollstadt, Sebastian Schmitt, and Michael Wibral. A rigorous information-theoretic definition of redundancy and relevancy in feature selection based on (partial) information decomposition. arXiv preprint arXiv:2105.04187 , 2021.

Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learning. Advances in Neural Information Processing Systems , 31, 2018.

Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey. arXiv preprint arXiv:2206.06488 , 2022.

Shaowei Yao and Xiaojun Wan. Multimodal transformer for multimodal machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Online, July 2020. Association for Computational Linguistics.

Lei Yu and Huan Liu. Efficient feature selection via analysis of relevance and redundancy. The Journal of Machine Learning Research , 5:1205-1224, 2004.

Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 1103-1114, 2017.

AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In ACL , 2018.

Unique 1

Unique 2

Redundancy

Synergy

1. Dataset quantification:

2. Model quantification:

3. Model selection:

X1

Y

D={(x1,2,y)}

→{R,U1,U2,S}D

f(D) ={(c1,2,g =f(c1,2))}

{R,U1,U2,S}f(D)

{R,U,U2,S}f(D)...,{R,U,U2,S}f(Dk)

）—→{R,U1,U2,S}f

{R,U1,U2,S}D←→{R,U1,U2,S}f

X2

Appendix

A Partial Information Decomposition

Information theory formalizes the amount of information that a single variable ( X ) provides about another ( Y ), and is quantified by Shannon's mutual information [86] I ( X ; Y ) which measures the amount of uncertainty reduced from H ( Y ) to H ( Y | X ) when given X as input. However, the direct extension of information theory to 3 or more variables, such as through total correlation [106, 34, 90, 97] or interaction information [70, 93, 12, 35], both have significant shortcomings. In particular, the three-way mutual information I ( X 1 ; X 2 ; Y ) can be both negative and positive, leading to considerable difficulty in its interpretation by theoreticians and practitioners.

Partial information decomposition (PID) [108] was proposed as a method to generalize information theory to multiple variables, by positing a decomposition of the total information 2 variables provide about a task I ( X 1 , X 2 ; Y ) into 4 quantities: redundancy R between X 1 and X 2 , unique information U 1 in X 1 and U 2 in X 2 , and synergy S that should together satisfy the following consistency equations:

A definition of R was first proposed by Williams and Beer [108] and was subsequently improved by Bertschinger et al. [13], Griffith and Koch [41] giving:

where ∆ p = { q ∈ ∆ : q ( x i , y ) = p ( x i , y ) ∀ y, x i , i ∈ { 1 , 2 }} and ∆ is the set of all joint distributions over X 1 , X 2 , Y . We call this the set of marginal-matching distributions. The core intuition of these definitions and how they enable PID lies in optimization over ∆ p instead of simply estimating information-theoretic measures for the observed distribution p . Specifically, synergy S is the difference between total multimodal information in p and total multimodal information in q ∗ ∈ ∆ p , the non-synergistic distribution (see Figure 7 for an illustration). While there have been other definitions of PID that also satisfy the consistency equations, Bertschinger et al. [13] showed that their proposed measure satisfies several desirable properties. Optimizing over ∆ p can be seen as going beyond purely observed data and discovering latent redundancy, uniqueness, and synergy via representation learning.

According to Bertschinger et al. [13], it suffices to solve for q using the following max-entropy optimization problem, the same q equivalently solves any of the remaining problems defined for

Unimodalmarginal-matchingdistributions:

p= {q(x1,x2,y) : q(x1,y) = p(x1,y),q(x2,y) =p(x2,y)}

p(x1,x2,y)

X1

X2

X

X2

X1

X2

X1

X2

X1

X2

Y

Y

Y

S = Ip(X1, X2;Y) - r

min Ig(X1,X2; Y)

qEp

Task-relevant

Task-relevantmultimodal

multimodalinfo

infowithoutsynergy:

0=(xx)-（xx)=*

redundancy, uniqueness, and synergy.

B Scalable PID Estimators

This section provides algorithmic and implementation details on the estimation of PID values for high-dimensional continuous datasets.

B.1 Estimator 1: CVX for Discrete Representations

Our first proposed estimator, CVX, is exact, based on convex optimization, and is able to scale to problems where |X i | and |Y| are around 100 . We show a high-level diagram of CVX in Figure 8 and provide several implementation details here:

Implementation Details : We implemented (5) using CVXPY [25, 1]. The transformation from the max-entropy objective (16) to (5) ensures adherence to disciplined convex programs [40], thus allowing CVXPY to recognize it as a convex program. All 3 conic solvers, ECOS [27], SCS [75], and MOSEK [6] were used, with ECOS and SCS being default solvers packaged with CVXPY. Our experience is that MOSEK is the fastest and most stable solver, working out of the box without any tuning. However, it comes with the downside of being commercial. For smaller problems, ECOS and SCS work just fine.

Potential Numerical Issues : Most conic solvers will suffer from numerical instability when dealing with ill-conditioned problems, typically resulting from coefficients that are too large or small. It is fairly common, especially in the GMM experiment that a single bin contains just one sample. Since P is estimated from an empirical frequency matrix, it will contain probabilities of the order 1 / num-samples. When the number of samples is too large (e.g., ≥ 1 e 8 ), this frequency is too small and causes all 3 solvers to crash.

Approximating Continuous Representation using Histogramming : The number and width of bins can affect quality of PID estimation [77]. For example, it is known that with a fixed number of samples, making the width of bins arbitrarily small will cause KL estimates to diverge. It is known that the number of bins should grow sub-linearly with the number of samples. Rice [84] suggest setting the number of bins to be the cubed-root of the number of samples. In our synthetic experiments, we sampled at least 1 e 6 samples and used no more than 100 bins. Note that increasing the number of samples ad infinitum is not an option due to potential numerical issues described above.

Text

Image

Convex programming

with linear constraints

CVXPY

p(c2|y) ≈p(c2|y)

p(x1ly) ≈ p(c1ly)

q(x1, 2, y) ~ q(C1, C2, y)

y

C1

ilanguageir

iformationl

human

Iinguistics

C2

R,U1, U2, S

B.2 Alternate Methods for Working with Continuous X 1 , X 2 :

Another method for discretization is based on nearest neighbors [80, 11, 104], which has the advantage of performing density estimation as an intermediate step, but is ill-suited for our purposes of approximating q ∗ . Pakman et al. [76] attempt to optimize q ∗ directly in the space of continuous distribution by approximating an optimal copula , which canonically captures dependencies between X 1 , X 2 in q . However, this is nontrivial due to the space of possible joint distributions, and the authors restrict themselves to Gaussian copulas.

B.3 Estimator 2: BATCH for Continuous Representations

Our next estimator, BATCH, is suitable for large datasets where the support of X 1 and X 2 are high-dimensional continuous vectors and not enumerable. Instead, we propose an end-to-end model parameterizing the joint distribution and a training objective whose solution allows us to obtain PID. Here |X i | = |Y| = ∞ . However, we wish to estimate the PID values given a sampled dataset D = { ( x ( j ) 1 , x ( j ) 2 , y ( j ) ) } of size n .

In this appendix section, we provide full details deriving our BATCH estimator, see Figure 9 for a highlevel overview. We first rewrite the optimization objective in (18) before defining neural estimators to approximate high-dimensional variables that we cannot compute exactly, before optimizing the (18) using learned models and obtaining the PID values.

Objective function : First, we observe that the optimization objective in (18) can be written as:

Our goal is to approximate the optimal ˜ q that satisfies this problem. Note that by the marginal constraints, q ( x 1 , y ) = p ( x 1 , y ) and q ( y ′ | x 1 ) = p ( y ′ | x 1 ) . The former can be obtained by sampling from the dataset and the latter is an unimodal model independent of q .

Sensors

Video

Sinkhorn's algorithm

Unnormalized joint distribution

Training objective

PID values

C1

fΦ(1)

p(ci

q

C2

fΦ(2)

A = exp

(fΦ(1)(X1,y)fΦ(2)(X2,y)T)

p(xi,y) =p(ylxi)p(xi)

q = SINKHORNp(A)

Figure 9: Overview of BATCH estimator: We define an end-to-end model that takes in batches of features X 1 ∈ ˜ X m 1 , X 2 ∈ ˜ X m 2 , Y ∈ Y m and learns a matrix A ∈ R m × m ×|Y| to represent the unnormalized joint distribution ˜ q . Following ideas in multimodal alignment [99] and attention methods [103] that learn approximate distributions over input data, we parameterize A by computing the outer-product similarity matrix over learned features. To ensure that ˜ q ∈ ∆ p , we use an unrolled version of Sinkhorn's algorithm [24] which projects A onto ∆ p by iteratively normalizing all rows and columns to sum to 1 and rescaling to satisfy the marginals ˆ p . We plug in the learned ˜ q into our objective in (25) and update trainable neural network parameters ϕ using projected gradient descent. Upon convergence, we extract PID by approximating I ˜ q ( { X 1 , X 2 } ; Y ) by sampling and plugging into (1)-(3). This use of a mini-batch of size m can be seen as an approximation of full-batch gradient descent over the entire dataset of size n ≫ m . While it is challenging to obtain an unbiased estimator of the full-batch gradient since computing the full A is intractable, we found our approach to work in practice for large m . Our approach can also be informally viewed as performing amortized optimization [3] by using ϕ to implicitly share information about the full batch using subsampled batches.

Neural architecture : To scale to continuous or unseen x i 's, we define a function approximator that takes in x 1 , x 2 and y and outputs an unnormalized joint density ˜ q ( x 1 , x 2 , y ) . Given batches X 1 ∈ ˜ X m 1 , X 2 ∈ ˜ X m 2 , Y ∈ Y m , we define learnable encoders f ϕ (1) and f ϕ (2) that output a matrix A ∈ R m × m ×|Y| to represent the unnormalized joint distribution ˜ q , i.e., we want A [ i ][ j ][ y ] = ˜ q ( X 1 [ i ] , X 2 [ j ] , y ) . Following ideas in multimodal alignment [99] and attention methods [103] in deep learning that learn approximate distributions over input data, we also parameterize our neural architecture by computing the outer-product similarity matrix over learned features:

Constraining A using Sinkhorn's algorithm : To ensure that the learned A is a valid pdf and respects the marginal constraints ˜ q ( x i , y ) = p ( x i , y ) , we use the Sinkhorn-Knopp algorithm [24], which projects A into the space of non-negative square matrices that are row and column-normalized to satisfy the desired marginals p ( x 1 , y ) and p ( x 2 , y ) . Sinkhorn's algorithm enables us to perform this projection by iteratively normalizing all rows and columns of A to sum to 1 and rescaling the rows and columns to satisfy the marginals. Since p ( x i , y ) is not easy to estimate for continuous x i , we compute the marginals using Bayes' rule by expanding it as p ( x i ) p ( y | x i ) . Note that by sampling x i from the dataset, the rows and columns of A are already distributed as p ( x i ) . The only remaining term needed is p ( y | x i ) , for which we use unimodal models ˆ p ( y | x i ) trained before running the estimator and subsequently frozen. Finally, Sinkhorn's algorithm is performed by normalizing each row to ˆ p ( y | x 1 ) and normalizing each column to ˆ p ( y | x 2 ) .

Optimization of the objective : We obtain the value of ˜ q ( x 2 | x 1 , y ) by marginalizing from the above distribution ˜ q ( x 1 , x 2 | y ) . Given matrix A representing ˜ q ( x 1 , x 2 , y ) , the remaining terms can all be computed in closed form through appropriate summation across dimensions in A to obtain ˜ q ( x i ) ,

˜ q ( x 1 , x 2 ) , ˜ q ( x i | y ) , and ˜ q ( x 1 , x 2 | y ) . We now have all the terms for the objective in (25). We optimize the objective via gradient descent.

Our final algorithm for training BATCH is outlined in Algorithm 1.

Extracting PID values from the learned ˜ q : To extract the PID values from a learned distribution of ˜ q , we first expand the definition of mutual information under p and ˜ q , and calculate the PID values from their definitions in (1)-(3):

Algorithm 1 BATCH algorithm.

Require: Multimodal dataset X 1 ∈ ˜ X n 1 , X 2 ∈ ˜ X n 2 , Y ∈ Y n .

Initialize feed-forward networks f ϕ (1) , f ϕ (2) .

while not converged do for

sampled batch

X

h

[:

, y

]

f

ϕ

(1)

1

A

[:

˜

1

x

1

1

˜

∈

X

, y

m

,

2

(

)

h

,

X

[:

2

, y

]

∈

X

exp(

1

2

:

,

, y

y

]

h

[:

, y

]

h

[:

, y

]

)

⊤

i

unimodal predictions.

p

(

x

)

|

A

ˆ

p

SINKHORN

(

A

)

Marginalize and sample

(

x

1

, x

2

, y

)

2

˜

,

Y

m

f

∈ Y

(

ϕ

(2)

from

Calculate the loss from (25) using

(

Perform a gradient step on the loss.

end for end while

return network weights in f 1 ,j , f 2 ,j for j ∈ [ C ]

Important assumptions and caveats : As a note, the above formulation assumes the existence of unimodal models ˆ p ( y | x i ) . We train these models independently and freeze them when training the above estimator. After obtaining ˜ q , we extract the PID values. To do so, we need an additional multimodal model to estimate ˆ p ( y | x 1 , x 2 ) due to (28). Because part of our goal is multimodal model selection, we must assume that the existing multimodal model might not be expressive enough and could underestimate (28). However, this value is only subsequently used to calculate S , leading to a possibly underestimated S , but leaving all other PID values unaffected by the choice of the multimodal model. In practice, the possibility of S being larger than estimated should be considered, and more complex models should be investigated until S stops increasing. Empirically, we have observed that measured value of S by BATCH is generally larger than that measured by CVX; the underestimation is not as large as in the case of clustering in CVX.

In additional to the above, we note that the learned joined distribution ˜ q ( x 1 , x 2 , y ) is limited by the expressivity of the neural network and similarity matrix method used in our model. Empirically, the PID values are not sensitive to hyperparameters, as long as the network is trained to convergence. Finally, our approximation using batch sampling limits the possible joined distributions to alignments within a batch, which we alleviates by using large batch size (256 in our experiments).

Implementation Details : We separately train the unimodal and multimodal models from the dataset using the best model from Liang et al. [60] and freeze them when running Algorithm 1. For the

m

x

2

, y

)

A

x

1

, x

2

.

, y

)

.

do

0.0

0.5

1.0

1.5

0

2

4

6

1e

3

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0

1

2

3

1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0

2

4

1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0

2

4

6

1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

neural network in Algorithm 1, we use a 3-layer feedforward neural network with a hidden dimension of 32. We train the network for 10 epochs using the Adam optimizer with a batch size of 256 and learning rate of 0 . 001 .

To summarize , we propose 2 scalable estimators for PID, each with their approximations for the parameterization of q and procedure for the estimation of PID.

Parameterizing q explicitly via a joint probability matrix over ∆ p , and optimizing convex objective (5) with linear constraints. This can only be used for finite X 1 , X 2 , Y , or after preprocessing by histogramming continuous domains.

Parameterizing q via neural networks and optimize their parameters ϕ by repeatedly subsampling from data; for each subsample, we perform one step of projected gradient descent (using Sinkhorn's method) as if this subsample was the full dataset.

C Experiments

C.1 Synthetic Bitwise Features

We show results on estimating PID values for OR, AND, and XOR interactions between X 1 and X 2 in Table 1 and find that both our estimators exactly recover the ground truth results in Bertschinger et al. [13].

C.2 Gaussian Mixture Models (GMM)

Extended Results: For each of Cartesian and Polar we consider two cases, Soft -labels, when we observe Y , and Hard -labels, where we use ˆ Y predicted by the optimal linear classifier ˆ y = sign ( ⟨ ( x 1 , x 2 ) , µ ⟩ ) . We hope this will provide additional intuition to the bitwise modality experiments. This results are given in Figures 10, 11. Most of the trends are aligned with what we would expect from Redundancy, Uniqueness, and Synergy. We list the most interesting observations here.

In all cases, total information (sum of PID values) is independent of ∠ µ . This is a natural consequence of rotational symmetry. It is also higher for higher || µ || 2 , e.g., 2 . 0 , reflecting the fact that X 1 , X 2 contain less information about labels when their centroids are close. Lastly, observe, that when || µ || 2 increases, the total information approaches log(2) , which is all the label information. This perfectly aligned with our intuition: if the cluster centroids are far away, then knowing a datapoint's cartesian coordinates will let us predict its label accurately (which is one bit of information).

For Cartesian , unique information dominates when angle goes to 0 or π/ 2 . This is sensible, if centroids share a coordinate, then observing that coordinate yields no information about y . Conversely, synergy and redundancy peak at π/ 4 .

0.0

0.5

1.0

1.5

0

2

4

6 1e

3

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0

1

2

3

1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0

2

4

1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0

2

4

6

1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0

2

4

6 1e

3

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0

1

2

3

1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0

2

4

1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0

2

4

6

1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.5 1e

1

R

S

U

1

U

2

Sum

Redundancy dominates Synergy as || µ || 2 increases. Interestingly, synergy seems to be independent of || µ || 2 (one has to account for the scale in the vertical axis to recognize this).

Hard contains 0 . 65 ≈ log(2) nats of total information, since Y | X 1 , X 2 is deterministic. The deficit of 0 . 04 nats is due to discretization near the decision boundary, and is expected to vanish if we increase the number of bins (though this ought to be accompanied by an increase in samples used to approximate PID).

However, for Soft , total information is nearly 0 for small || µ || 2 (and approaches log(2) when large). When the two cluster centroids are close, it is difficult to glean any information about the (soft) label, while we can predict y accurately from both x 1 , x 2 when centroids are far apart.

For Polar , redundancy is 0 . Furthermore, θ contains no unique information, since θ shows nothing about y unless we know r (in particular, its sign). When angle goes to π/ 2 , almost all information is unique in r .

Lastly, we also experimented on the more common definition of polar coordinates, with θ ∈ [ -π, π ] and r ≥ 0 (that is, angle is allowed to be negative but lengths have to be nonnegative). The results are in Figure 12. As one may expect, there is nothing exciting going on for both Soft and Hard . For Hard , the optimal classifier is precisely expressed in terms of θ only, so all the information belongs to U 2 , i.e., θ . For Soft , the same trend remains: virtually all information is in U 2 , i.e., θ , with the trend becoming even sharper as || µ || 2 increases. Intuitively, when the 2 centroids are extremely far

apart, then the optimal linear classifier is extremely accurate; which may be expressed by θ alone without any knowledge of r . Conversely, when the centroids are close, r can provide some synergy with θ -the larger r is, the more 'confident' we are in predictions based on θ .

Visualization of q ∗ : The following plots of q ∗ are provided to provide some intuition behind the objectives in (16). We focus on Cartesian for ease of visualization. For each || µ || 2 ∈ { 0 . 5 , 1 . 0 , 2 . 0 } , we vary the angle ∠ µ and sample 1 e 7 labelled datapoints. We then compare q ∗ (for both Soft and Hard ) with the empirical distribution p . To compute PID, we use 50 bins evenly spaced between [ -5 , 5] for both dimensions. The results for varying || µ || 2 are shown in Figures 13-15.

When ∠ µ is 0 , there are few differences, barring a slightly 'blockier' feel for q ∗ over p . However, as ∠ µ increases, q ∗ gets skewed rather than rotated, becoming 'thinner' in the process. The limiting case is when ∠ µ 2 = π/ 4 , i.e., 45 degrees. Here, the objective diverges to infinity, with what appears to be a solution with nonnegative entries only along the diagonal-this observation appears to hold even as we increase the number of bins. We hypothesize that in the truly continuous case (recall we are performing discretization for this set of experiments), the optimum is a degenerate 'one dimensional' GMM with mass located only at q ( x, x ) .

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

C.3 Synthetic Generative Model

Our third setting simulates a data generation process where there exists a set of latent concepts that either redundantly, uniquely, or synergistically determine the label.

Setup : We begin with a fixed set of latent variables z 1 , z 2 , z c ∈ R 50 from N (0 , σ 2 ) with σ = 0 . 5 representing latent concepts for modality 1, modality 2, and common information respectively. The concatenated variables [ z 1 , z c ] are transformed into high-dimensional x 1 ∈ R 100 data space using a fixed transformation matrix T 1 ∈ R 50 × 100 and likewise [ z 2 , z c ] to x 2 via T 2 . The label y is generated

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

5

0

5

as a transformation function of (1) only z c , in which case we expect redundancy in x 1 and x 2 with respect to y , (2) only z 1 or z 2 which represents uniqueness in x 1 or x 2 respectively, (3) the concatenated [ z 1 , z 2 ] representing synergy, or (4) arbitrary ratios of each of the above. We also introduce nonlinearity and random noises to increase data complexity. In formal notation,

where f is a fixed nonlinear transformation with dropout ( p = 0 . 1 ) and z can be z 1 , z 2 , z c or any combination according to the cases described above. These datasets are designed to test a combination of feature learning with predefined interactions between the modalities and the label.

We generated 10 synthetic datasets as shown in Table 2. The first four are specialized datasets denoted as D m , m ∈ { R,U 1 , U 2 , S } for which the label y only depends on redundancy, uniqueness, or synergy features respectively. The rest are mixed datasets with y generated from z 1 , z 2 , z c of varying dimensions. We use z ∗ i ∈ R 50 to denote a randomly sub-sampled feature from z i . For example, to generate the label of one data point ( x ( j ) 1 , x ( j ) 2 ) in the mixed dataset y = ( z ∗ 1 , z ∗ 2 , z c ) , we sub-sampled ( z ∗ 1 ) ( j ) , ( z ∗ 2 ) ( j ) from z 1 , z 2 and generate y from [( z ∗ 1 ) ( j ) , ( z ∗ 2 ) ( j ) , z c ] following the process above.

To report the ground-truth interactions for these datasets, we first train multimodal models and obtain an estimate of test performance on them. We then convert P acc, the test accuracy of the best model on these datasets into the mutual information between the inputs to the label using the bound in Feder and Merhav [29]:

Since the test accuracies for Table 2 datasets range from 67 -75% , this corresponds to a total MI of 0 . 42 -0 . 59 bits (i.e., from log 2 0 . 67+1 to log 2 0 . 75+1 ). The information in each interaction is then computed by dividing this total mutual information by the interactions involved in the data generation process: if the total MI is 0 . 6 bits and the label depends on half from the common information between modalities and half from the unique information in x 1 , then the ground truth R = 0 . 3 and U 1 = 0 . 3 .

Results : In the main paper, we note the agreement between CVX and BATCH and their consistency with the dataset (Table 2). In addition, we note that the sum of PID values (which sum to I p ( Y ; X 1 , X 2 ) ) is generally larger in BATCH than in CVX, which accounts for the information loss during CVX's clustering. Moreover, in the last two cases ( y = f ( z ∗ 2 , z ∗ c ) and y = f ( z ∗ 2 , z c ) ), CVX predicts non-zero amount of S despite the dataset being synergy-free by construction. Because the total mutual information is small, the spurious synergy becomes a large portion of the total PID values. This potentially demonstrates that when the total mutual information is small, spurious information gain could have a large impact on the results.

C.4 Quantifying Multimodal Datasets

Setup : We use a large collection of real-world datasets in MultiBench [60] which test multimodal fusion of different input signals and requires representation learning of complex real-world interactions

for different tasks. MultiBench spans 10 diverse modalities (images, video, audio, text, time-series, various robotics sensors, sets, and tables), 15 prediction tasks (humor, sentiment, emotions, mortality rate, ICD9 codes, image-captions, human activities, digits, robot pose, object pose, robot contact, and design interfaces), and 5 research areas (affective computing, healthcare, multimedia, robotics, and HCI). These datasets are designed to test a combination of feature learning and arbitrary complex interactions between the modalities and the label in the real world. We also include experiments on question-answering (Visual Question Answering [5, 39] and CLEVR [53]) which test grounding of language into the visual domain.

Details of applying CVX : For quantifying multimodal datasets with CVX, we first apply PCA to reduce the high dimension of multimodal data. For each of the train, validation, and test split, we then use unsupervised clustering to generate 20 clusters. We obtain a clustered version of the original dataset D = { ( x 1 , x 2 , y ) } as D cluster = { ( c 1 , c 2 , y ) } where c i ∈ { 1 , . . . , 20 } is the ID of the cluster that x i belongs to. For datasets with 3 modalities we estimate PID separately for each of the 3 modality pairs.

Details of applying BATCH : For quantifying multimodal datasets with BATCH, we first use pretrained encoders to obtain the image and text features, and then train 2 unimodal and 1 multimodal classifiers on those features to predict the original answer. The final PID values are computed using the encoded features and the 3 classifiers. In our experiments we used the pretrained image and text encoders from the CLIP-VIT-B/32 [82] model, and the unimodal and multimodal classifiers are 2-layer MLPs.

Results on multimodal fusion : We show full results on quantifying datasets in Table 6. For multimodal fusion, we find that different datasets do require different interactions. Among interesting observations are that (1) all pairs of modalities on sarcasm detection shows high synergy values, which aligns with intuition on sarcasm in human communication, (2) uniqueness values are strongly correlated with unimodal performance (e.g., A V-MNIST and MIMIC first modality), (3) datasets with high synergy benefit from interaction modeling (e.g., sarcasm), and (4) conversely datasets with low synergy are also those where modeling in higher-order interactions do not help (e.g., MIMIC).

To test grounding of language into the visual domain, we also run experiments on the VQA v2.0 [5, 39] dataset, which contains 443757 training and 214354 validation data with real-life images, and the CLEVR [53] dataset, which contains 699960 training and 149991 validation data with rendered images and generated questions. The test spilt for both datasets are not used because the labels are not publicly available. Note that for both validation datasets, we first filter out all data instances where the label is neither 'yes' nor 'no'. On the remaining data, we use the outputs from the last feature layer of a VGG19 [87] model as image embeddings and use RoBERTa [64] to obtain the text embeddings for questions, and then apply K-means clustering to both embeddings to obtain a clustered version

Instruction:

Look at the data and provide your rating for the following questions on a

scale of 0 (none at all) - 5 (large extent)

and score your confidence in your answer from a scale of 0 (no confidence) -

5 (high confidence):

1. The extent to which

both modalities enable

you to make the same

predictions about the

task.

2. The extent to which

modality 1 enables you to

make a prediction about

the task that you would

not if using modality 2.

3. The extent to which

modality 2 enables you to

make a prediction about

the task that you would

not if using modality 1.

4. The extent to which

both modalities enable

you to make a prediction

about the task that you

would not otherwise

make using either

modality individually.

ID

Modality 1

Modality 2

1 Rating

1 Confidence

2 Rating

2 Confidence

3 Rating

3 Confidence

4 Rating

4 Confidence

video

The devil deceives false Christians into thinking they are

already safe and secure and on their way to heaven.

mosei2

video

Currently, over four-hundered thousand Americans with

disabilities are being paid less than the minimum wage, some

of them mere pennies per hour.

So once again, we call on the regime to cease these absolutely

mosei6

video

In 2008, there were very serious tribal warfare in Kenya following a disputed election.

mosei7 video They're not going to give this thing any floor time at all because they know that Donald Trump is not going to divest from his corporations mosei8 video And you saw some of the leading brands out there like McDonalds, like General Mills and Coca Cola, who are doing of the original dataset D = { ( x 1 , x 2 , y ) } as D cluster = { ( c 1 , c 2 , y ) } , where c i ∈ { 1 , . . . , 20 } are the cluster IDs. We then calculate the PID values on the clustered dataset D cluster .

such a great job of tapping into those insights and bringing them to life in their advertising. mosei9 video Know who's in it and follow the B2B CFO\u00ae golden rule; Let the Finders find, Minders mind, and Grinders grind. mosei10 video You have been invited to a special event that includes an invitation, so let's take a look at how should we respond to it. vqa1 image How many pictures are there? vqa2 image What color is the woman's shirt on the left? vqa3 image Is this a hospital? vqa4 image What is the green fruit called? vqa5 image What are these animals? Results on QA : As a whole, we observe consistently high synergy values on QA datasets as shown in Table 6. This is consistent with prior work studying how these datasets were balanced (e.g., VQA 2.0 having different images for the same question such that both yes/no answers are observed in the dataset so that the answer can only be obtained through synergy) [39] and models trained on these datasets [46].

vqa6 image Is the door closed? vqa7 image Where do these animals live? vqa8 image What game are they playing? vqa9 image Is it an overcast day? vqa10 image What  2 colors are the flowers? clevr1 image "The other small shiny thing that is the same shape as the tiny yellow shiny object is what color?" clevr2 image "How many large things are either cyan metallic cylinders or yellow blocks?" clevr3 image "There is a yellow thing to the right of the rubber thing on the left side of the gray rubber cylinder; what is its material?" clevr4 image "Are there any large matte blocks of the same color as the large metal ball?" clevr5 image "What number of cylinders are gray objects or tiny brown matte objects?" clevr6 image "Are there any large blue metallic things that have the same shape as the small brown matte object?" Comparing CVX and BATCH : We carefully compare CVX and BATCH on the 2 question answering datasets VQA and CLEVR, showing these results in Table 7. We choose these two datasets since their input modalities are images and text, which make them suitable for both clustering of representations extracted from pre-trained before input into CVX, and end-to-end encoding and PID estimation with BATCH. We also include human judgment of PID on these 2 datasets for comparison. From Table 7, we find that all three measures agree on the largest interaction, as well as the relative order of interaction values. We find that the absolute values of each interaction can be quite different due to the unavoidable information loss of both input modalities to the label during clustering in CVX or representation learning in BATCH. Hence, we emphasize that these estimated interactions are most useful through relative comparison to judge which interaction is most dominant in a particular dataset.

C.5 Comparisons with Human Judgment

To quantify information decomposition for real-world tasks, we investigate whether human judgment can be used as a reliable estimator. We propose a new annotation scheme where we show both modalities and the label and ask each annotator to annotate the degree of redundancy, uniqueness, and synergy on a scale of 0-5 using the following definitions inspired by the formal definitions in information decomposition:

R : The extent to which using the modalities individually and together gives the same predictions on the task,

U 1 : The extent to which x 1 enables you to make a prediction about the task that you would not if using x 2 ,

U 2 : The extent to which x 2 enables you to make a prediction about the task that you would not if using x 1 ,

S : The extent to which only both modalities enable you to make a prediction about the task that you would not otherwise make using either modality individually,

alongside their confidence in their answers on a scale of 0-5. We show a sample user interface for the annotations in Figure 16.

Based on direct communication with our institution's IRB office, this line of user-study research is aligned with similar annotation studies at our institution that are exempt from IRB. The information obtained during our study is recorded in such a manner that the identity of the human subjects cannot readily be ascertained, directly or through identifiers linked to the subjects. We do not collect any identifiable information from annotators. Participation in all annotations was fully voluntary and we obtained consent from all participants prior to annotations. Participants were not the authors nor in the same research groups as the authors. They all hold or are working towards a graduate degree in a STEM field and have knowledge of machine learning. None of the participants knew about this project before their session and each participant only interacted with the setting they were involved in.

We sample 10 datapoints with class balance from each of the 5 datasets in Table 8 and give them to 3 annotators. All the data were manually selected and do not contain offensive content. We clarified all the confusion that the annotators may have about the instructions and the definitions above at the beginning of the annotation process. We summarize the results and key findings:

General observations on interactions, agreement, and confidence : The annotated interactions align with prior intuitions on these multimodal datasets and do indeed explain the interactions between modalities. For example, VQA and CLEVR are annotated with significantly high synergy, and language is annotated as the dominant modality in sentiment, humor, and sarcasm (high U 1 values). Overall, Table 9 shows that the Krippendorff's alpha for inter-annotator agreement in human annotations of the interactions is quite high and the average confidence scores are also quite high. This indicates that the human-annotated results are reliable.

Annotator subjectivity in interpreting presence vs absence of an attribute and contextualizing both language and video : We observe that there is some confusion between uniqueness in the language modality and synergy in the video datasets, resulting in cases of low agreement in annotating U 1 and S : -0 . 09 , -0 . 07 for MOSEI, -0 . 14 , -0 . 03 for UR-FUNNY and -0 . 08 , -0 . 04 for MUSTARD respectively. We believe this is due to annotator subjectivity in interpreting whether sentiment, humor, and sarcasm are present only in the language modality ( U 1 ) or present when contextualizing both language and video ( S ). We further investigated the agreement and confidence in the presence or absence of an attribute (e.g., humor or sarcasm). We examined videos that show and do not show an attribute separately and indeed found in general, humans reached higher agreement on annotating attribute-present videos. The agreement of annotating S is 0 . 13 when the attribute is present, compared to -0 . 10 when absent.

C.6 Comparisons with Other Interaction Measures

It is worth noting that other valid information-theoretic definitions of multimodal interactions also exist, but are known to suffer from issues regarding over- and under-estimation, and may even be negative [41]; these are critical problems with the application of information theory for shared

I ( X 1 ; X 2 ; Y ) and unique information I ( X 1 ; Y | X 2 ) , I ( X 2 ; Y | X 1 ) often quoted in the co-training [14, 8] and multi-view learning [98, 94, 89] literature. We additionally compare PID with other previously used measures for feature interactions on the synthetic generative model datasets. Both CVX and BATCH measures agree with each other on relative values and are consistent with ground truth interactions. We also implement 3 other definitions: I-min can sometimes overestimate synergy and uniqueness; WMS is actually synergy minus redundancy, so can be negative and when R & S are of equal magnitude WMS cancels out to be 0; CI can also be negative and sometimes incorrectly concludes highest uniqueness for S -only data. These results are in Table 10. We also tried 3 non-info theory measures: Shapley values, Integrated gradients (IG), and CCA, which are based on quantifying interactions captured by a multimodal model. Our work is fundamentally different in that interactions are properties of data before training any models.

C.7 Quantifying Model Predictions

Setup : For each dataset, we first train a suite of models on the train set D train and apply it to the validation set D val, yielding a predicted dataset D pred = { ( x 1 , x 2 , ˆ y ) ∈ D val } . Running PID on D pred summarizes the interactions that the model captures. We categorize and implement a comprehensive suite of models (spanning representation fusion at different feature levels, types of interaction inductive biases, and training objectives) that have been previously motivated in the literature to capture certain types of interactions [62]. In the following, we use x i to denote data or extracted features from unimodal models where appropriate. We include all implementation details and hyperparameters in Table 11, 12, 13, 14, 15.

General : We first implement the most general model, which we call early fusion:

EF: Concatenating data at the earliest input level, essentially treating it as a single modality, and defining a suitable prediction model y = f ([ x 1 , x 2 ]) [62].

Redundancy : Fusion at the decision level (i.e., late fusion ) is often motivated to capture redundancy between unimodal predictors.

ADDITIVE: Suitable unimodal models are first applied to each modality before aggregating the outputs using an additive average: y = 1 / 2( f 1 ( x 1 ) + f 2 ( x 2 )) [44].

AGREE: Prediction agreement ( + λ ( f 1 ( x 1 ) -f 2 ( x 2 )) 2 [26]) can be added for redundancy.

ALIGN: Feature alignment ( + λ sim ( x 1 , x 2 ) like contrastive learning [82]) can also be added for redundancy.

Synergy : Exemplified by methods such as bilinear pooling [33] and tensor fusion [115], researchers have long sought to learn higher-order interactions across modalities to capture synergistic interactions. We implement the following models:

ELEM: For static interactions (i.e., without trainable interaction parameters), we implement element-wise interactions: y = f ( x 1 ⊙ x 2 ) [2, 50].

TENSOR: We also implement outer-product interactions (i.e., higher-order tensors) y = f ( x 1 x ⊤ 2 ) [33, 115, 47, 58, 65].

MI: Dynamic interactions with learnable weights include multiplicative interactions W : y = f ( x 1 W x 2 ) [52].

MULT: Another example of dynamic interactions with learnable weights is Crossmodal self-attention typically used in multimodal Transformers: y = f ( softmax ( x 1 x ⊤ 2 ) x 1 ) [103, 99, 113].

Unique : There have been relatively fewer methods that explicitly aim to capture uniqueness, since label information is often assumed to be shared across modalities (i.e., redundancy) [98, 94], two that we implement are:

LOWER: Some approaches explicitly include lower-order terms in higher-order interactions to capture unique information [65, 115].

REC: Finally, several methods include reconstruction objectives to encourage maximization of unique information (i.e., adding an objective L rec = ∥ g 1 ( z mm ) -x 1 ∥ 2 + ∥ g 2 ( z mm ) -x 2 ∥ 2 where g 1 , g 2 are auxiliary decoders mapping z mm to each raw input modality [92, 100, 111].

Results : We show the full results on estimating PID for all models trained on all datasets in Table 18. To better understand the models' ability in capturing each type of interaction, we present a concise overview of the full results in Table 4, focusing on the PID values of each type of interaction I for each model on I -specialized dataset D I . We also compute the distribution of I D I over all models. We highlight the following observations:

General observations : We first observe that model PID measures are consistently higher than dataset measures. The sum of model measures is also a good indicator of test performance, which agrees with their formal definition since their sum is equal to I ( { X 1 , X 2 } ; Y ) , the total explained mutual information between multimodal data and Y .

On redundancy : Overall, several methods succeed in capturing redundancy, with an overall average of 0 . 41 ± 0 . 11 and accuracy of 73 . 0 ± 2 . 0% on redundancy-specialized datasets. Additive, agreement, and alignment-based methods are particularly strong, but other methods based on tensor fusion (synergy-based), including lower-order interactions, and adding reconstruction objectives (uniquebased) also capture redundancy well.

On uniqueness : Uniqueness is harder to capture than redundancy, with an average of 0 . 37 ± 0 . 14 . Certain methods like additive and agreement do poorly on uniqueness, while those designed for uniqueness (lower-order interactions and reconstruction objectives) do well, with U = 0 . 55 and 73 . 0% accuracy on uniqueness datasets.

On synergy : On average, synergy is the hardest to capture, with an average score of only 0 . 21 ± 0 . 10 . The best-performing methods are tensor fusion S = 0 . 33 , acc = 74 . 0% and multimodal transformer S = 0 . 29 , acc = 73 . 0% . Additive, agreement, and element-wise interactions do not seem to capture synergy well.

On robustness : Finally, we also show empirical connections between estimated PID values with model performance in the presence of noisy or missing modalities. Specifically, with the original performance Acc on perfect bimodal data, we define the performance drop when X i is missing as Acc -i = Acc -Acc i where Acc i is the unimodal performance when X i is missing. We find high correlation ( ρ = 0 . 8 ) between Acc -i and the model's U i value. Inspecting the graph closely in Figure 5, we find that the correlation is not perfect because the implication only holds in one direction: high U i coincides with large performance drops, but low U i can also lead to performance drops. We find that the latter can be further explained by the presence of large S values: when X i is missing, these interactions can no longer be discovered by the model which decreases robustness. For the subset of points when U i ≤ 0 . 05 , the correlation between S and performance drop is ρ = 0 . 73 while the correlation between R and performance drop is only ρ = 0 . 01 .

Category

Model

Measure

(

c

z

)

(

(

(

(

(

(

(

(

(

1

2

1

)

)

, z

∗

1

1

1

2

, z

)

∗

2

, z

, z

∗

1

, z

∗

2

2

, z

∗

2

∗

2

∗

c

, z

∗

c

, z

∗

∗

2

, z

, z

c

, z

∗

c

c

z

z

z

z

z

z

z

z

z

c

)

)

0

0

0

0

0

0

0

0

0

R

.

47

0

.

01

.

15

.

22

.

16

.

16

.

25

.

19

.

26

U

GENERAL

EF

1

0

.

01

0

.

41

0

0

.

02

0

0

.

0

15

.

0

05

.

03

0

0

U

2

0

.

01

0

0

.

47

0

0

.

07

0

0

0

0

.

12

0

.

12

S

0

.

0

04

.

0

.

0

.

0

.

0

.

0

.

0

.

0

.

0

.

04

04

29

15

15

25

14

06

05

Acc

0

74

.

0

73

.

0

.

72

0

.

72

0

.

64

0

.

71

0

.

74

0

.

66

0

.

65

0

.

7

0

0

0

0

0

0

0

0

0

0

R

.

35

.

02

.

05

.

15

.

31

.

06

.

09

.

39

.

15

.

27

ADDITIVE

U

1

0

2

U

S

.

0

2

.

14

0

0

0

.

16

0

0

0

.

11

0

0

0

0

0

.

43

0

.

2

0

0

.

18

0

.

23

0

0

.

07

0

.

18

0

.

0

REDUNDANCY

Acc

R

04

.

0

.

0

.

0

.

0

.

0

.

0

.

0

.

0

.

03

06

13

08

06

11

08

18

08

0

.

71

0

.

61

0

.

72

0

.

56

0

.

64

0

.

0

6

.

52

0

.

67

0

.

61

0

.

71

0

0

0

0

0

0

0

0

.

48

.

02

.

01

.

14

.

21

.

06

.

02

.

44

0

.

13

0

.

1

U

1

0

.

01

0

.

15

0

0

0

0

0

0

.

04

0

0

AGREE

U

2

0

.

01

0

0

.

48

0

.

26

0

.

11

0

.

08

0

.

33

0

0

.

24

0

.

1

S

0

.

0

06

.

0

.

0

.

0

.

0

.

0

.

0

.

0

.

0

.

04

04

09

15

08

06

09

04

16

Acc

0

.

74

0

.

38

0

.

72

0

.

66

0

.

58

0

.

53

0

.

62

0

.

67

0

.

64

0

.

61

Category

Model

Measure

(

c

z

)

(

z

(

z

(

z

(

z

(

z

(

1

2

1

)

)

, z

∗

1

1

z

(

z

(

)

2

, z

∗

2

, z

, z

∗

1

1

∗

2

, z

∗

2

2

∗

c

, z

∗

c

, z

, z

, z

∗

∗

2

∗

c

z

c

, z

)

)

)

c

)

(

∗

2

c

z

, z

)

Category

Model

Measure

(

c

z

)

(

z

(

z

(

z

(

z

(

z

(

z

(

z

(

z

(

z

1

2

1

)

)

, z

∗

1

1

1

2

)

, z

∗

2

, z

, z

∗

1

∗

2

∗

2

, z

∗

2

2

)

∗

c

, z

∗

c

, z

, z

, z

, z

∗

∗

2

∗

c

c

c

, z

c

)

)

)

)

)

)

SYNERGY

Acc

R

0

.

67

0

.

0

.

0

.

0

.

0

.

0

.

0

.

0

.

66

66

65

55

58

62

62

54

0

0

0

.

4

0

.

02

.

0

.

0

16

21

.

0

.

0

0

1

18

.

.

33

13

0

.

64

0

.

36

UNIQUE

Acc

R

0

.

75

0

.

0

.

0

.

0

.

0

.

0

.

0

.

0

.

74

72

74

65

71

73

68

64

0

.

7

0

0

0

0

0

0

0

0

0

.

55

0

.

01

.

.

.

.

.

.

.

14

19

08

21

32

19

42

)

)

)

)

R

0

.

2

0

.

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

01

.

.

.

.

.

.

.

01

05

07

02

04

12

02

.

13

R

.

53

0

.

01

.

.

.

.

.

.

.

15

21

09

16

31

13

31

U

1

0

0

.

13

0

0

.

04

0

.

01

0

.

01

0

.

01

0

01

.

0

0

U

1

0

0

56

.

0

0

03

.

0

01

.

0

27

.

0

06

.

0

.

01

0

0

MI

U

2

0

.

01

0

0

.

23

0

0

.

01

0

0

0

.

0

01

.

01

0

.

01

LOWER

U

2

0

.

01

0

0

.

54

0

0

.

01

0

0

0

.

0

01

.

0

.

29

13

S

0

.

05

0

.

05

0

.

05

0

.

12

0

.

08

0

.

06

0

.

08

0

.

06

0

.

06

0

.

06

S

0

.

03

0

.

02

0

.

02

0

.

32

0

.

0

2

.

13

0

.

27

0

.

16

0

.

04

0

.

04

U

1

0

02

.

0

.

48

0

0

.

05

0

.

11

0

.

29

0

.

02

0

0

0

U

1

0

.

02

0

.

53

0

0

.

06

0

.

03

0

.

29

0

.

05

0

0

0

MULT

U

2

0

0

0

.

42

0

0

0

0

0

.

0

03

.

39

0

.

18

REC

U

2

0

.

01

0

0

.

55

0

0

0

0

0

.

0

06

.

0

.

36

15

S

0

.

06

0

.

02

0

.

06

0

.

29

0

.

0

2

.

11

0

.

0

3

.

14

0

.

04

0

.

04

S

0

.

02

0

.

03

0

.

02

0

.

34

0

.

26

0

.

16

0

.

31

0

.

21

0

.

04

0

.

03

Acc

0

.

73

0

.

0

.

0

.

0

.

73

71

72

66

0

.

0

.

0

7

72

.

0

.

68

67

0

.

72

Acc

0

.

75

0

.

0

.

0

.

0

.

0

.

0

.

0

.

0

.

0

.

74

73

74

66

71

72

68

66

72

1.Dataset quantification:

2.Modelquantification:

3.Model selection:

D={(x1,x2,y)}

—→{R,U1,U2,S}D

—→{R,U,U2,S}f(D)

{R,U1,U2,S}D←→{R,U,U2,S}f

{R,U,U2,S}f(D)..,{R,U,U2,S}f(Dk)—→{R,U,U2,S}f

Interactionpolytope

Interactionpolytope

Interactionpolytope

(1,0,0,0)

(0,1,0,0)

(0.1,0.7,0.2,0.3)

(0.1,0.7,0.2,0.3)

Modelfamilies trained

Selectsmodelswith

on syntheticdata

>96%performance

(0,0,1,0)

(0,0,0,1)

C.8 Multimodal Model Selection

Setup : We show that our findings on quantifying multimodal datasets and model predictions are generalizable, and that the PID estimations are informative for model selection on new datasets without training all models from scratch . We simulate the model selection process on 5 new synthetic datasets and 6 real-world multimodal datasets. Given a new dataset D , we compute their PID values I D and normalize it to obtain ˆ I D :

We select the most similar dataset D ∗ from our base datasets (the 10 synthetic datasets presented in Table 2) that has the smallest difference measure

which adds up the absolute difference between normalized PID values on D and D ′ . The model selection pipeline is illustrated in Figure 17. We hypothesize that the top3 best-performing models on D ∗ should also achieve decent performance on D due to their similar distribution of interactions. We verify the quality of the model selection by computing a percentage of the performance of the selected model f with respect to the performance of the actual best-performing model f ∗ on D , i.e. % Performance ( f, f, ∗ ) = Acc ( f ) /Acc ( f ∗ ) .

Results : We summarize our evaluation of the model selection in Table 5, and find that the top 3 chosen models all achieve 95% -100% of the best-performing model accuracy, and above 98 . 5% for all datasets except MUSTARD. For example, UR-FUNNY and MUSTARD have the highest synergy ( S = 0 . 13 , S = 0 . 3 ) and indeed transformers and higher-order interactions are helpful (MULT: 0 . 65% , MI: 0 . 61% , TENSOR: 0 . 6% ). ENRICO has the highest R = 0 . 73 and U 2 = 0 . 53 , and indeed methods for redundant and unique interactions perform best (LOWER: 0 . 52% , ALIGN: 0 . 52% , AGREE: 0 . 51% ). MIMIC has the highest U 1 = 0 . 17 , and indeed unimodal models are mostly sufficient [60]. We hypothesize that model selection is more difficult on MUSTARD since it has the highest synergy ratio, and it is still an open question regarding how to capture synergistic multimodal interactions. Therefore, PID values provide a strong signal for multimodal model selection.

C.9 Real-world Case Study 1: Computational Pathology

Setup : The current standard of care for cancer prognosis in anatomic pathology is the integration of both diagnostic histological information (via whole-slide imaging (WSI)) as well as molecular information for patient stratifcation and therapeutic decision-making [63]. For cancer types such as diffuse gliomas, the combination of histological subtypes (astrocytoma, oligodendroglioma) and molecular markers ( IDH1 mutation, 1p19q codeletion) determines whether patients should be administered tumor resection, radiation therapy, adjuvant chemotherapy, or combination therapies [66, 16, 20]. Recent applications of multimodal learning in pathology have found that though fusion confers benefit for a majority of cancer types, unimodal models can be as competitive in patient stratification performance, which suggest potential clinical application of prognosticating using only a single modality for certain cancers [22].

Using The Cancer Genome Atlas (TCGA), PID was used to quantify pathology image-omic interactions in downstream prognostication tasks for two cancer datasets: lower-grade glioma (TCGALGG [15] ( n = 479 ) and pancreatic adenocarcinoma (TCGA-PAAD [83], ( n = 209 ). As mentioned in the main text, the modalities include: (1) a sequence of pre-extracted histology image features from diagnostic WSIs ( N × 1024 , where N is the number of non-overlapping 256 × 256 patches at 20 × magnification and 1024 is the extracted feature dimension of a pretrained vision encoder), and (2) feature vector ( 1 × D ) of D bulk gene mutation status, copy number variation, and RNA-Seq abundance values. The label space is overall survival time converted to 4 discrete bins. Following the study design of [22], we trained an Attention-Based Multiple Instance Learning (ABMIL) model and a Self-Normalizing Network (SNN) for the pathology and genomic modalities using a log likelihood loss for survival analysis respectively [48, 55]. For ABMIL, a truncated ResNet-50 encoder pretrained on ImageNet was used for patch feature extraction of WSIs [45]. Unimodal features were extracted before the classification layer, with K-means clustering ( k = 3 ) for each modality used to obtain ( x 1 , x 2 , y ) pairs where x 1 is pathology, x 2 is genomics and y is the discrete bins, which can then be used in CVX to compute PID measures.

Results : We assess these PID measures (Table 20) in the context of current scientific understanding of pathology and genomic interactions in cancer prognostication in the literature. Specifically, we compare against the findings [22], which previously investigated unimodal and multimodal survival models in TCGA and observed that risk predictions were primarily driven by genomics across most cancer types (such as TCGA-LGG), with exceptions in a few cancer types such as TCGAPAAD. In the evaluation of TCGA-LGG and TCGA-PAAD in [22], concordance Index performance of unimodal-pathology, unimodal-genomic, and multimodal survival models were (0.668, 0.792, 0.808) and (0.580, 0.593, 0.653) respectively. In [22], one of the original hypotheses for the lack of multimodal improvement for some cancer types was due to high mutual information between both modalities. In our results, we observe the opposite effect in which there is little mutual information between both modalities. This observation coupled with the low and high synergy values in TCGALGG ( S = 0 . 02 ) and TCGA-PAAD ( S = 0 . 15 ) suggest that only TCGA-PAAD would benefit from multimodal integration, which was empirically found with both the c-Index improvement and statistical significant patient stratification results in TCGA-PAAD evaluation. The uniqueness values of both tasks also corroborate the high amount of task-specific information in genomics with high c-Index performance, and vice versa with pathology. Overall, we demonstrate we can use PID measures to refine biomedical insights into understanding when multimodal integration confers benefit.

C.10 Real-world Case Study 2: Mood Prediction from Mobile Data

Setup : Suicide is the second leading cause of death among adolescents [18]. Despite these alarming statistics, there is little consensus concerning imminent risk for suicide [31, 56]. Intensive monitoring of behaviors via adolescents' frequent use of smartphones may shed new light on the early risk of suicidal thoughts and ideations [37, 72]. Smartphones provide a valuable and natural data source with rich behavioral markers spanning online communication, keystroke patterns, and application usage [59]. Learning these markers requires large datasets with diversity in participants, variety in features, and accuracy in annotations. As a step towards this goal, we partnered with several hospitals (approved by NIH IRB for central institution and secondary sites) to collect a dataset of mobile behaviors from high-risk adolescent populations with consent from participating groups. This data monitors adolescents spanning (a) recent suicide attempters (past 6 months) with current suicidal ideation, (b) suicide ideators with no past suicide attempts, and (c) psychiatric controls with no history of suicide ideation or attempts. Passive sensing data is collected from each participant's smartphone across a duration of 6 months. The modalities include (1) text entered by the user represented as a bag of top 1000 words that contains the daily number of occurrences of each word, (2) keystroke features that record the exact timing and duration that each character was typed on a mobile keyboard

(including alphanumeric characters, special characters, spaces, backspace, enter, and autocorrect), and (3) mobile applications used per day, creating a bag of 137 apps for each day that are used by at least 10% of the participants.

Every day at 8 am, users are asked to respond to the following question - 'In general, how have you been feeling over the last day?' - with an integer score between 0 and 100 , where 0 means very negative and 100 means very positive. To construct our prediction task, we discretized these scores into the following three bins: negative ( 0 -33 ), neutral ( 34 -66 ), and positive ( 67 -100 ), which follow a class distribution of 12 . 43% , 43 . 63% , and 43 . 94% respectively. For our 3 -way classification task, participants with fewer than 50 daily self-reports were removed since these participants do not provide enough data to train an effective model. In total, our dataset consists of 1641 samples, consisting of data coming from 17 unique participants.

Results : The results are shown in Table 21, 22. We observe that both MAPS T,A (text + apps) and MAPS T,K (text + keystrokes) have the highest synergy ( S = 0 . 26 , S = 0 . 4 respectively) and some redundancy ( R = 0 . 09 , R = 0 . 12 ) and uniqueness in the second modality ( U 2 = 0 . 09 , U 2 = 0 . 04 ). We find indeed the best-performing models are REC (acc = 0 . 56% ) and MULT (acc = 0 . 48% ) for MAPS T,A , and TENSOR (acc = 0 . 57% ) and REC (acc = 0 . 52% ) for MAPS T,K , which are designed to capture synergy and uniqueness interactions. Our model selection also successfully chooses REC for MAPS T,A and TENSOR for MAPS T,K with a high agreement of α = 0 . 66 and α = 0 . 74 ( 98 . 3% and 99 . 85% of the highest agreement respectively). This result further corroborates the utility and generalizability of our PID estimations and model selection.

C.11 Real-world Case Study 3: Robotic Perception

Setup : MuJoCo PUSH [57] is a contact-rich planar pushing task in the MuJoCo simulation framework, in which a 7 -DoF Panda Franka robot is pushing a circular puck with its end-effector in simulation. We estimate the 2D position of the unknown object on a table surface, while the robot intermittently interacts with the object. The final dataset consists of 1000 trajectories with 250 steps at 10Hertz, of a simulated Franka Panda robot arm pushing a circular puck in MuJoCo [95]. The pushing actions are generated by a heuristic controller that tries to move the end-effector to the center of the object. The multimodal inputs are gray-scaled images (1 × 32 × 32) from an RGB camera, forces (and binary contact information) from a force/torque sensor, and the 3D position of the robot end-effector. The task is to predict the 2-D planar object pose.

We use CNN feature extractors and LSTM as a sequence model for both unimodal and multimodal models. We use hyperparameters from Table 16. Because both CVX and BATCH assumes discrete classes of labels, we discretize all continuous labels into 10 bins of equal data points from the training set, and use the same bin cutoffs at test time. The task is to predict the correct bin that contains the original label, and we use cross entropy loss as our loss function.

Results : The results are shown in Table 23. As can be seen from the table, BATCH predicts U 1 as the highest PID value ( U 1 = 1 . 79 ), which aligns with the fact that image is the best unimodal predictor for this dataset [60]. Comparing both estimators, CVX underestimates U 1 and R since CVX clusters the data before processing, and the high-dimensional time-series modalities cannot be easily described by clusters without losing information. In addition, both BATCH and CVX predict a low U 2 value but attributes some mutual information to R (redundancy), implying that a multimodal model with both modalities would not be much better compared to an unimodal model on modality 1, since the multimodal model would mostly get redundant information from modality 2. In our experiments, we observe no difference in performance between the multimodal model and the unimodal model with modality 1.

D Summary of Takeaway Messages and Future Work

From these results, we emphasize the main take-away messages and motivate several directions for future work:

Dataset quantification: PID provides reliable indicators for the nature and quantity of interactions present in both synthetic and real-world multimodal datasets, which provides a useful tool summarizing these datasets. In our opinion, creators of new datasets focused on modeling interactions between multiple features or modalities should report estimated PID values on new datasets alongside justification on whether these values are to be expected based on how the features were selected.

Model quantification: PID also provides reasonable indicators for the interactions captured by trained models, but naturally there is more noise due to different model parameter counts, training trajectories, and non-convex optimization artifacts that despite our best efforts remain impossible to exactly control. Despite these details, we still find several consistent patterns in the types of interactions different models capture. We suggest that researchers developing new multimodal models explicitly state the assumptions on the underlying multimodal distribution their models require and the interactions their models are designed to capture, before testing exhaustively on datasets predominantly of those interactions.

Model selection: Our experiments on model selection for real-world datasets and applications demonstrate potential utility in a rough 'user-guide' for practitioners aiming to tackle real-world multimodal datasets. Given a new dataset, estimate its PID values. If there is high U 1 and U 2 , just using unimodal models in the corresponding modality may be sufficient.

Otherwise, if there is high R , methods like agreement, alignment, ensemble, or co-training should be tried. If there is high S , it is worth spending the time on multimodal methods that model more complex interactions based on tensors, multiplicative interactions, and self-attention.

We believe that the intersection of PID and machine learning opens the door to several exciting directions of future work:

Pointwise interaction measures: A natural extension is to design pointwise measures: how much does a single datapoint contribute to redundancy, uniqueness, or synergy in the context of an entire distribution? Pointwise measures could help for more fine-grained dataset and model quantification, including error analysis of incorrect predictions or active learning of difficult examples.

Representation learning for desired interactions: Can we design new model architectures or training objectives that better capture certain interactions? Current well-performing models like Transformers already include some notion of multiplicative interactions, so it does seem that, heuristically, synergy between input features is playing a role in their success. Designing models that better capture synergy, as quantified by our PID estimators, could be a path towards learning better representations.

Principled approaches to fairness and invariance: Currently, PID is designed to measure the information that 2 variables contribute towards a task Y , but conversely it can also be used to remove information that one variable can have about Y , in the context of another variable. These could provide formal learning objectives for fairness, privacy, and other feature invariance tasks.

E Limitations and Broader Impact

E.1 Limitations of our Estimators

While our estimators appear to perform well in practice, each suffers from distinct limitations. For CVX, it is rarely able to scale up to domains with more than several thousand classes. Furthermore, if |X i | is large-which is frequently the case in real-world applications, then empirical frequencies will often be near 0 , causing much instability in conic solvers (see Appendix B.1 for a more detailed description).

BATCH sidesteps many of the issues with CVX by applying batch gradient descent. However, it may suffer from approximation error depending on whether the network can sufficiently cover the space of joint distributions (i.e., representation capacity), alongside bias resulting from mini-batch gradient descent (as compared to full-batch gradient descent) when approximating ˜ q using bootstrapped samples. Lastly, it may suffer from the usual problems with neural estimators, such as local minima, poor or unstable convergence (in both learning ˜ q and ˆ p ). Therefore, while BATCH scales to highdimensional continuous distributions, it comes with the challenges involving training and tuning neural networks.

E.2 Broader Impact

Multimodal data and models are ubiquitous in a range of real-world applications. Our proposed framework based on PID is our attempt to systematically quantify the plethora of datasets and models currently in use. While these contributions will accelerate research towards multimodal datasets and models as well as their real-world deployment, we believe that special care must be taken in the following regard to ensure that these models are safely deployed:

Care in interpreting PID values : Just like with any approximate estimator, the returned PID values are only an approximation to the actual interactions and care should be taken to not overfit to these values. Other appropriate forms of dataset visualization and quantification should still be conducted to obtain holistic understanding of multimodal datasets.

Privacy, security, and biases: There may be privacy risks associated with making predictions from multimodal data if the datasets include recorded videos of humans or health indicators. In our experiments with real-world data where people are involved (i.e., healthcare and affective computing), the creators of these datasets have taken the appropriate steps to only access public data which participants/content creators have consented for released to the public. We also acknowledge the risk

of exposure bias due to imbalanced datasets that may cause biases towards certain genders, races, and demographic groups. Therefore, care should be taken in understanding the social risks of each multimodal dataset in parallel to understanding its interactions via PID.

Time & space complexity : Modern multimodal datasets and models, especially those pretrained on internet-scale data, may cause broader impacts resulting from the cost of hardware, electricity, and computation, as well as environmental impacts resulting from the carbon footprint required to fuel modern hardware. Future work should carefully investigate the role of size on the interactions learned by models through estimated PID values. Our preliminary experiments showed that smaller models could still capture high degrees of each interaction, which may pave away towards designing new inductive biases that enable interaction modeling while using fewer parameters.

Overall, PID offers opportunities to study the potential social and environmental issues in multimodal datasets by obtaining a deeper understanding of the underlying feature interactions, providing a path towards interpretable and lightweight models. We plan to continue expanding our understanding of PID via deeper engagement with domain experts and how they use this framework in their work. Our released datasets, models, and code will also present a step towards scalable quantification of feature interactions for future work.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is mainly concerned with the methodological contribution of estimators for partial information theory statistics, to quantify the levels of redundancy, uniqueness and synergy, which are expressed in information theoretic terms, through the use of mutual information. \n\nDepending on dataset size and dimensionality, the authors propose two variants of estimators: one that is exact but can only be used for small, low-dimensional data, and one that is approximate, but that scales to large, high-dimensional data.\nPID statistics can be computer for models as well, which enables the extension of to the proposed framework to deal with model selection.\n\nA very large series of experiments are discussed to support 1) the usefulness of PID statistics, 2) the quality of the proposed estimators, 3) the applicability of the proposed methods to real-world dataset. In addition, several experiments on model selection are carried out as well. Finally, three real world applications are studied through the lenses of partial information decomposition, further substantiating the impact of such statistics. When possible, evaluation is exact, given the the ground truth PID statistics value could be computed analytically. Otherwise, the authors resort to human feedback.\n\nSTRENGTHS:\n* Deepening our understanding of multimodal data, and models built to address tasks that rely on multimodal data is of fundamental importance. The use of partial information decomposition is, in my opinion, a good approach to achieve such goals, and this paper is convincing in both showcasing that PID statistics are indeed representative for multimodal data and modelling tasks, as well as supporting the proposed methodology to compute such PID statistics.\n\n* The experimental campaign is very extensive and overall (some comments in the questions section should be addressed though) very convincing and clear. \n\n* The proposed methodology (especially the approximate one) for computing PID statistics is simple, and easy to implement\n\nWEAKNESSES:\n* Multimodal VAEs [1,2,3] are missing. To the best of my knowledge, a large body of research on multimodal representation learning is missing from the study of PID statistics for model predictions. On the one hand, I understand that in this work a model is used to address a given downstream task, whereas multimodal VAEs are used mostly to learn latent representations of all modalities, for generative purposes, but also to solve downstream tasks. Since the literature on Multimodal VAEs clearly addresses relevant questions on coherency, and synergy, I think this work could greatly benefit from an extension to such models.\n\n* Alternatives to compute MI are not considered as “competitor” methods for the approximate scheme. Approaches such as [4, 5] have received a lot of attention as means to compute MI. Would it make sense to consider those as building blocks for alternative approximate PID estimation methods?\n\n* Some details in the experimental section need further clarification. For example, in section 4.3 on the robustness, e.g. missing modalities, it is not clear if this is considered at training or a test time.\n\n[1] Wu, et al.,”Multimodal Generative Models for Scalable Weakly-Supervised Learning”, https://arxiv.org/abs/1802.05335\n[2] Shi, et al., “Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models”, https://arxiv.org/abs/1911.03393\n[3] Palumbo, et al., “MMVAE+: Enhancing the Generative Quality of Multimodal VAEs without Compromises”, https://openreview.net/forum?id=B42UJTVdDZ5\n[4] Belghazi, et al., “MINE: Mutual Information Neural Estimation”, https://arxiv.org/abs/1801.04062\n[5] Hijelm, et al., “Learning deep representations by mutual information estimation and maximization”, https://arxiv.org/abs/1808.06670\n\nQUESTIONS:\n* I understand that both the methodology and experiments tackle two modalities. Now, even if datasets with more than two modalities, and models capable of handling more than two modalities are not extremely common, a natural question that arises both for the proposed methodology, its foundations on PID, and the experiments is how well does the schemes proposed in this work scale with the number of modalities.\n\n* In the appendix, it is mentioned that the BATCH estimator, in its current form, relies on pre-trained classifiers, including multimodal ones. As explained, this can have the unintended effect of perturbing the estimation of PID statistics, resulting in errors on synergy. While a theoretical analysis of approximation guarantees seem to be rather difficult, could you discuss about means to keep approximation errors under control? \n\n* For the BATCH estimator, a key aspect to consider is the dimensionality of the hidden representation that is used to compute matrix A (before sinkhorn is applied). Do you have any insights concerning the effects of this hyper-parameter?\n\n* On the values obtained for R, U and S. As an overview on the various result tables, it is somehow hard to justify the choice for bold numbers. In particular, if we consider results on synthetic data for which a ground truth exists, we can see that the values for PID statistics obtained by either the exact or approximate schemes is very small. In some cases the differences between the values obtained is small, and bold numbers seem to be chosen to “comply” with group truth: see for example Table 2 $y=f(z_2^*, z_c^*)$ for the CVX scheme. More generally, the question I would like to ask is related to the “degree” aspects of PID statistics, and how to interpret correctly numerical values, especially when such values are very low, say what is the difference between R=0.07 and R=0.06 in practical terms? I understand that the first statistics indicate more redundancy than the second, but is this going to be discernible in practice?\n\n* Finally, if time allows, would it be possible to address the points mentioned in the \"weaknesses\" part of the review?\n\nLIMITATIONS:\nTo the best of my knowledge, there is no explicit discussion on limitations nor societal impact. However, in the appendix, it is possible to find information about caveats, assumptions, which help understanding some limitations of the proposed methods.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an information-theoretic approach to quantify the degree of three PID(Partial Information Decomposition) statistics of a multimodal distribution: redundancy, uniqueness and synergy. To solve the difficulties in computing PID over high-dimensional continuous distributions, two scalable estimators for PID statistics are introduced, enabling the estimation of PID statistics over large multimodal datasets using small subsampled batches. To validate PID estimation, experiments are conducted on synthetic and real-world multimodal datasets. The usefulness of PID is also demonstrated in model selection and real-world case studies.\n\nSTRENGTHS:\n+ This paper proposes an useful approach to quantify the nature and degree of modality interactions based on information theory. The proposed PID estimators can scale to high-dimensional multimodal datasets and models.\n+ The definitions of redundancy, uniqueness, and synergy are intuitive and could help to better understand multimodal datasets and models.\n+ The proposed approach could generalize to real-world applications and provide reasonable interpretation for dataset properties and model selection.\n\nWEAKNESSES:\n- The results in Table 2 and 3 are unable to prove the accuracy of the PID estimators. Estimated results only show a rough trend. There is an order of magnitude difference between the approximation and the ground truth in most cases. Besides, it seems that the approximation becomes more inaccurate when the value distribution of R, U, S is more uniform (i.e. The second main row in Table 3). Is this method only applicable to simple and distinctive datasets?\n- Lack of comparison and discussion with existing methods. For example, Shapley Value could measure the contribution of each modality in multimodal learning. What are the similarities and differences between them? Why is PID better than them?\n\nQUESTIONS:\nMajor concerns have been mentioned in Weakness. \n\nLine 163: missing ')'\n\nLIMITATIONS:\nThe authors did not address the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a fundamental question in multimodal modeling: given multiple input modalities, how can we quantify the ways they must interact (as part of a model) to solve a particular task. Furthermore, can we use knowledge to learn how to design models that better leverage these interactions. \n\nThe authors propose an information-theoretic approach to this problem by quantifying three properties (redundancy, uniqueness, and synergy) using mutual information across modalities. They solve for these properties using two estimators (one using convex optimization and the other sampling). The first estimator uses convex optimization and the second uses a sampling-baed approximation. \n\nThey evaluate their approaches for quantifying these properties using synthetic and real-world datasets. For the latter, annotators manually went through each dataset and quantified their intuition for overlap between modalities. They show that their models are able to do a reasonable job estimating these quantities.\n\nSTRENGTHS:\n* Overall this is a well written and carried out paper with a lot of depth. The authors approach the problem in a principled manner and have thought deeply about underlying concepts of multi-modality. \n* While I did not go through the derivations, the authors introduce a nice framework for thinking about multi-modal interactions. \n* I appreciate the combination of synthetic experiments, real-world experiments, snapshot \"case studies,\" and various ablation experiments. \n* The appendix includes a large amount of interesting additions.\n\nWEAKNESSES:\n* I found the results hard to interpret and -- given synthetic results -- am wary of the validity of the claims. For example, in the second set of synthetic experiments (Table 2) the recovered values are MUCH lower than ideal (e.g., D_R set has R(CVX)=0.16, R(Batch)=0.29, and R(Truth)=1). This is especially true in the \"mixed interaction\" synthetic datasets. Is there a good reason that the estimated values are so far from the truth? Are the proposed estimators good in the absence of noise (i.e., the bit-wise experiment) but do they quickly fail once there is uncertainty? Is the issue with the estimators, the underlying generative model, or something else?\n\n* The core idea of the paper is solid, but I'm wary of the use of human judgements for creating ground truth. The models are using features/representations extracted from each input modality, and a human's perception/judgement here may differ significantly from signal in the data. I acknowledge there may not be a better solution for annotations. (aside: Thanks for putting info about the annotators' backgrounds and instructions in the appendix). \n\n* Similarly, in \"Results on multimodal fusion,\" given my experience with these kinds of models I'm hesitant to believe the rational/intuition such as \"since sarcasm is often due to a contradiction between what is expressed in language an speech.\" It seems like a natural conclusion, but my understanding is that speech emotion (or sarcasm) detectors use a more limited context to compute their predictions, which may not have enough context to encode this contradiction  (but maybe I'm wrong?)? The rational here \"make sense\" but seem cherry picked and might not be \"true.\" Couldn't there be other (potentially spurious) correlations that cause the results to look nice?\n\n* I imagine there are alternative correlations or measures one could use for evaluating redundancy/uniqueness/synergy. It would be useful to see more discussion (or comparison) with this.\n\nQUESTIONS:\n* Sec 3.1 (preprocessing via feature binning): Is estimating this using binning a reasonable idea here? I suppose this depends on the bin size, dimensionality, size of the dataset, and the distribution of latents. \n\n* The \"Batch\" estimator is interesting. I immediately wondered if VQ-VAEs or other discrete state latent variable models could be used as an alternative here. This would give you (discrete) tokens instead of having to optimize over a continuous space. Does this seem like a reasonable idea? This idea seems simpler to me and means that you don't have to subsample batches as done here.\n\nLIMITATIONS:\nYes, more so in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper attempts to quantify the interactions between different input modalities using Principle Information Decomposition (PID) from Information Theory. Using this approach, one can quantify not only interactions within multimodal datasets but as well as interactions captured by multimodal models. 
The paper provided a comprehensive evaluation of their approach and provided examples of its application in real-world case studies.

STRENGTHS:
Provided a detailed explanation of its theoretical background, and thorough comprehensive evaluation. They also provided a list of real-world applications which are useful for casual users.

WEAKNESSES:
The presented work, would benefit from more visuals. As this paper is quite detailed, it would benefit from a closing summary listing the main points of the paper.

QUESTIONS:
Are there any potential use cases of this approach outside multimodal models? Like, e.g. measuring the domain gap between synthetic and real-world data?

LIMITATIONS:
There is no dedicated section on its Limitations, nor is it focused on in the paper as far as I could find. It would be useful so that the readers would know in which cases this could potentially not be effective.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. The authors introduce two new estimators for these PID statistics that scale to high-dimensional distributions. They validate the PID statistics on synthetic and real-world datasets, demonstrating their ability to identify meaningful interactions and select appropriate models. The authors also compare their approach to other methods for quantifying multimodal interactions, showing that their method outperforms these alternatives. They provide examples of real-world applications where the PID statistics were useful in quantifying interactions within multimodal datasets or selecting appropriate multimodal models. The authors discuss the limitations of their approach, including the need for large amounts of data and the difficulty of interpreting the results. They suggest future directions for research, such as extending the approach to continuous-valued data and incorporating causal relationships between modalities. Overall, this paper provides a useful framework for quantifying and modeling multimodal interactions that has potential applications in a wide range of fields.\n\nSTRENGTHS:\n1. Novel approach: The paper proposes a novel information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. This approach is different from other methods for quantifying multimodal interactions and has the potential to provide new insights into how different modalities interact.\n\n2. Extensive validation: The authors validate their approach on both synthetic and real-world datasets, demonstrating its ability to identify meaningful interactions and select appropriate models. They also compare their approach to other methods for quantifying multimodal interactions, showing that their method outperforms these alternatives.\n\n3. Real-world applications: The paper provides examples of real-world applications where the PID statistics were useful in quantifying interactions within multimodal datasets or selecting appropriate multimodal models. This demonstrates the practical usefulness of the approach in a wide range of fields.\n\n4. Clear presentation: The paper is well-written and clearly presents the approach, the validation experiments, and the real-world applications. The authors provide detailed explanations of the methods used and the results obtained, making it easy for readers to understand and replicate their work\n\nWEAKNESSES:\n1. Limited scope: The paper focuses on a specific information-theoretic approach to quantifying multimodal interactions and does not consider other approaches or methods. This may limit the generalizability of the results and the usefulness of the approach in other contexts.\n\n2. Lack of comparison to ground truth: The paper validates the approach on synthetic and real-world datasets, but does not compare the results to a ground truth or gold standard. This makes it difficult to assess the accuracy of the approach and the validity of the results.\n\n3. Limited interpretability: The paper acknowledges that the results of the approach can be difficult to interpret, particularly in high-dimensional datasets. This may limit the usefulness of the approach in practice, as it may be difficult to draw meaningful conclusions from the results.\n\n4. Lack of implementation details: The paper provides a high-level overview of the approach and the methods used, but does not provide detailed implementation instructions or code. This may make it difficult for other researchers to replicate the work or apply the approach in their own research.\n\nQUESTIONS:\n1. How does the proposed approach compare to other methods for quantifying multimodal interactions, such as correlation analysis or principal component analysis?\n2. Can the approach be applied to continuous-valued data, or is it limited to discrete data?\n3. How does the approach handle missing data or incomplete modalities?\n4. How does the approach handle interactions between more than two modalities?\n5. In Table 1, what is Exact task? Why all task and measures have the same numbers?\n6. Table 3 is hard to understand, maybe better to use the same scale?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}