Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications
Authors: Paul Pu Liang
Chun Kai Ling
Yun Cheng
Alexander Obolenskiy
Yudong Liu
Rohan Pandey
Alex Wilf
Louis-Philippe Morency
Russ Salakhutdinov
Venue: ICLR 2024 poster (2024)
Keywords: multimodal learning
multimodal interactions
information theory
self-supervised learning
multimodal fusion
Primary area: learning theory
TL;DR: We quantify multimodal interactions in a semi-supervised setting by showing new relationships between synergy, redundancy, and uniqueness, which can be used to estimate model performance, guide data collection, and select multimodal models.

Abstract: In many machine learning systems that jointly learn from multiple modalities, a core research question is to understand the nature of multimodal interactions: how modalities combine to provide new task-relevant information that was not present in either alone. We study this challenge of interaction quantification in a semi-supervised setting with only labeled unimodal data and naturally co-occurring multimodal data (e.g., unlabeled images and captions, video and corresponding audio) but when labeling them is time-consuming. Using a precise information-theoretic definition of interactions, our key contribution is the derivation of lower and upper bounds to quantify the amount of multimodal interactions in this semi-supervised setting. We propose two lower bounds: one based on the shared information between modalities and the other based on disagreement between separately trained unimodal classifiers, and derive an upper bound through connections to approximate algorithms for min-entropy couplings. We validate these estimated bounds and show how they accurately track true interactions. Finally, we show how these theoretical results can be used to estimate multimodal model performance, guide data collection, and select appropriate multimodal models for various tasks.

Published as a conference paper at ICLR 2024
MULTIMODAL LEARNING WITHOUT LABELED MULTI- MODAL DATA: GUARANTEES AND APPLICATIONS
Paul Pu Liang', Chun Kai Ling", Yun Cheng', Alex Obolenskiy', Yudong Liu', Rohan Pandey', Alex Wilf", Louiss Morency', Ruslan Salakhutdinoy 'Carnegie Mellon University. "Columbia University, 'Princeton University pliangêcs. edu, c144888columbie
ABSTRACT
In many machine learning systems that jointly learn from multiple modalities, a core research question is understand the nature of multimodal interactions: how modalities combine to provide new task-relevant information that was not present either alone. We study this challenge of interaction quantification in a semi-supervised setting with only labeled unimodal data and naturally co-occurring multimodal data (e.g., unlabeled images and captions, video and corresponding audio) but when labeling them is time-consuming. Using a precise information- theoretic definition of interactions, our key contribution is the derivation of lower and upper bounds to quantiff the amount of multimodal interactions in this semi- supervised setting. We propose two lower bounds: one based on the shared information between modalities and the other based disagreement between sepa- rately trained unimodal classifiers, and derive an upper bound through connections to approximate algorithms for min-entropy couplings. We validate these estimated bounds and show how they accurately track true interactions. Finally, we show how these theoretical results can be used to estimate multimodal model performance, guide data collection, and select appropriate multimodal models for various tasks.
INTRODUCTION
A core research question in multimodal learning is to understand the nature of interactions between modalities for task: how much information is shared between both modalities, lies in each modality alone, and the emergence of new task-relevant information during learning from both modalities that was not present in either modality alone (Liang et al., 2022b). In settings where labeled multimodal data is abundant, the study of multimodal interactions in the supervised setting has inspired fundamental advances in theoretical analysis (Hessel & Lee, 2020; Liang et al., 2023a; Sridharan & Kakade, 2008), representation learning (Jayakumar et al., 2020; Radford et al., 2021), and the selection of suitable multimodal models for various weal-world tasks (Liang et al., 2023a).
In this paper, we study the problem interaction quantification in a setting where there is only unlabeled multimodal data DM (x11x2) and some labeled unimodal data D; = (xx, y) collected separately for each modality. This multimodal semi-supervised paradigm is reminiscent of many real-world settings with separate unimodal datasets like visual recognition (Deng et al., 2009) and text classification (Wang et al., 2018), as well as naturally co co-occurring multimodal data (e.g.. news images and captions video and audio), but when labeling them is time-consuming (Hsu et al., 2018; Hu et al., 2019) or impossible due to partially observed modalities (Liang et al., 2022a) or privacy concerns (Che al., 2023). Despite these data constraints, we still want to understand how the modalities can share, exchange, and create information in order to inform our decisions on data collection and modeling (Jayakumar al., 2020; Liang al., 2023a; Zadeh et al., 2017).
Using a precise information-theores definition of interactions (Bertschinger et al., 2014), our key contributions are the derivations of lower and upper bounds to quantify multimodal interactions in this semi-supervised setting with only DD and DM- We propose two lower bounds: the first relates interactions with the amount of shared information between modalities, and the second is based on the disagreement of classifiers trained separately on each modality. Finally, we propose an upper bound through connections to approximate algorithms for min-entropy couplings (Cicalese & Vaccaro, 2002). To validate our bounds, we experiment on both synthetic and large real-world datasets with varying amounts of interactions. In addition, these theoretical results naturally yield new guarantees regarding the performance of multimodal models. By analyzing the relationship between interaction estimates and downstream task performance assuming optimal multimodal classifiers

---

Published as a conference paper at ICLR 2024
trained on labeled multimodal data, closely predict multimodal model performance, before even training the model itself. These performance estimates also help develop new guidelines for deciding when to collect additional modality data and select the appropriate multimodal fusion models. We believe these results shed light on the intriguing connections between multimodal interactions, modality disagreement, and model performance, and release our code and models at attps::/githhi com/ applanngn9//Pa
RELATED WORK AND TECHNICAL BACKGROUND
2.1 SEMI-SUPERVISED MULTIMODAL LEARNING
Let X and y be finite sample spaces for features and labels. Define A to be the set of joint distributions over (8, We are concerned with features X1, (with support X,) and labels Y (with support )) drawn from some distribution p A. We denote the probability mass function by p(11,22 T2. y). where omitted parameters imply marginalization. Many real-world applications such as multimedia and healtheare naturally exhibit multimodal data (e.g.. images and captions, video and audio, multimodal medical readings) which are difficult to label (Liang et al., 2022a; Radford al., 2021; Singh al., 2022; Yu & Liu, 2004; Zellers al., 2022). As such, rather than the full distribution from P. we only have partial datasets:
Labeled unimodal data D1 = {11 v) : 2 x )). D2 = ((12, v) X2 x y). Unlabeled multimodal data DM = ( Xi x X2).
D1, D2 and Das follow the pairwise marginals p(11. y). p p(22.8) v) and x2)) We define 4pi {g e A y) = p(a.,y) y) Vy E V,,;; X, E [2]} as the set of joint distributions which agree with the labeled unimodal data D2 and 22, and A p1.2.12 {re A ::((11 02) = p(11, ((,,)))
p(x; y)} as the set of joint distributions which agree with all D1. D2 and DM.
2.2 MULTIMODAL INTERACTIONS AND INFORMATION THEORY
The study of multimodal interactions aims to quantify the information shared between both modali- ties, in each modality alone, and how modalities can combine to form new information not present in either modality, eventually using these insights to design machine learning models to capture interac- tions from large-scale multimodal datasets (Liang et al., 2022b). Existing literature has primarily studied the interactions captured by trained models, such as using Shapley values (Itmner et al., 2021) and Integrated gradients (Sundararajan et al., 2017; Tsang et al., 2018; Liang et al., 2023b) to measure the importance model assigns to each modality, or approximating trained models with additive or non-additive functions to determine what functions are best suited to capture interactions (Friedman & Popescu, 2008; Sorokina et al., 2008; Hessel & Lee, 2020). However, these measure interactions captured by a trained model our work is fundamentllly different in that interactions are properties of data. Quantifying the interactions data, independent of trained models, allows us to characterize datasets, predict model performance, and perform model selection, prior to choosing and training a model altogether. Prior work in understanding data interactions to design multimodal models is often driven by intuition, such as using contrastive learning (Poklukar et al., 2022; Radford et al., 2021; Tosh et al., 2021), correlation analysis (Andrew et al., 2013), and agreement (Ding et al., 2022) for shared information (e.g., images and descriptive captions), or using tensors and multiplicative interac- tions (Zadeh et al., 2017; Jayakumar et al., 2020) for higher-order interactions (c.g., in expressions of sarcasm from speech and gestures).
To fill the gap in data quantification, information theory has emerged as a theoretical foundation since it naturally formalizes information and its sharing as statistical properties of data distributions. Information theory studies the information that one random variable (X1) provides about another (X2), quantified by Shannon's 's mutual information (MI) and conditional MI:
I(Xx;x2) = f (((11,2) x2) log 2(21, dx, X(Xx;x+\Y)== = f p(21, x2, v) log p(21, (2lv) dxdy. p(r))p(x2)
p(ar (1)p())))))
T(X1; measures the amount of information (in bits) obtained about X, by observing X2, and by extension, T(X;; Xy is the expected value of MI given the value of a third (e.g.. task Y).
To generalize information theory for multimodal interactions, Partial information decomposition (PID) (Williams& & Beer, 2010) decomposes the total information that two modalities X₂ provide about a task Y into 4 quantities:
= R++ U, + U2 + S, where I, , X2):)) Y) ) is the MI between the joint random variable (X1, X2) and Y. These 4 quantities are: redundancy R for
2

---

Published as a conference paper at ICLR 2024
the task-relevant information shared between X2 and X2, uniqueness U1 and U2 for the information present in only X1 or X2 respectively, and synergy S for the emergence of new information only when both X1 and X2 are present (Bertschinger et al., 2014; Griffith & Koch, 2014):
Definition 1. (Multimodal interactions) Given X1, X2, and a target Y, we define their redundant (R), unique (U1 and U2), and synergistic (S) interactions as:
R- max 1x;1;; U1 = min (x(X1;)(X2), U2 min Yx(X2; gepp1.2
(1)
g
ge
S = I,X,
min geap1,2
X2)x1)
(2)
where the notation (.) and (.) disambiguates mutual information (MI) under p and q respectively.
a multivariate extension of information theory (Bell, 2003; McGill, 1954). Most importantly, R. U1. and U2 can be computed exactly using con- vex programming over distributions q é ApI. with access only to the marginals p(z).)) y) and p(22. by solving a convex optimization problem with linear harginal-matching constraints qº' = arg max (X) (Bertschinger et al., 2014; Liang et al., 2023a), see Ap- pendix B.2 for more details. This gives us an elegant interpretation that need only labeled unimodal data in each feature from D1 and D2 to estimate redundant and unique interactions. Unfor- tunately, St is impossible to compute via equation (2) when we do not have access to the full joint distribution p, since the first term I ( Y) is unknown.
[(;; X2; x2) I I(;;;
is worth noting that other valid information theooretic definitions of multimodal interactions also exist, but are known to suffer from issues regarding over- and under-estimation, and may even be neg- ative; these are critical problems with the application of information theory for shared 7(X1; X2; Y) and unique information I(X1; 21 . 7(X2; YIXi) ) often quoted in the co-training (Blum & Mitchell, 1998; Balcan et al., 2004) and multi-view learning (Tosh et al., 2021; Tsai et al., 2020; Tian et al., 2020; Sridharan & Kakade, 2008) literature. We refer the reader to Griffith & Koch (2014) for full discussion. We choose the one in Definition above since it fulfills several desirable properties, but results can be extended other definitions as well.
ESTIMATING SEMI-SUPERVISED MULTIMODAL INTERACTIONS
Our goal is to estimate multimodal interactions R. U1. U2. and S assuming access to only semi- supervised multimodal data D,, D2, and Dar. Our first insight is that while S cannot be computed exactly, R, U1, and U2 can be computed from equation with access to only semi-supervised data. Therefore, studying the relationships between S and other multimodal interactions is key to its estimation. Using these relationships, we will then derive lower and upper bounds for synergy in the form S S SS S S. Crucially, & and depend only on D1, D2, and DM-
3.1 UNDERSTANDING RELATIONSHIPS BETWEEN INTERACTION
We start by identifying two important relationships, between S and R, and between S and U.
Synergy and redundancy Our first relationship stems from the case when two modalities contain shared information about the task. In studying these situations, a driving force for estimating S is the amount of shared information I I(X1; between modalities, with the intuition that more shared information naturally leads to redundancy which gives less opportunity for new synergistic interactions. Mathematically, we formalize this by relating S to R.
S= R IxI;
R R- Ix(( + Xx1:
(3)
implying that synergy exists when there is high redundancy and low (or even negative) three-way way MI T N1; Y). By comparing the difference in X1, X₂ dependence with and without the task (i.e., (x)xx)) X2) vs (x; X2) 2 cases naturally emerge (see left side of Figure 1):
1. S> R: When both modalities do not share a lot of information as measured by low I(X1;X2) but conditioning on Y increases their dependence: [(Xi: X2
(();;X)) then there is synergy between modalities when combining them for task Y. This setting is reminiscent of common cause structures. Examples of these distributions in the real world are multimodal question answering, where the image and question are less dependent (some questions like 'what is the color of the car' or 'how many people are there" can be asked for many images), but the
3

---

Published as a conference paper at ICLR 2024
R> 8
11
Y decreases
U>S
I1 -
y = V2 # #11
X1, X2 dependence
I22
32 4/2
Theorem
S> R
11
X2 dependence
Y incresses
Lower bound
S>U
T1 V
y y
# 92 # V1
Lower bound
1/2
12
Figure 1: We study the relationships between (left) synergy and redundancy as a result of the task Y either increasing or decreasing the shared information between X1 and X2 (i.e., common cause structures as opposed redundancy in common effect), as well as (right) synergy and uniqueness due to the disagreement between unimodal predictors resulting a new prediction y # y1 ya (rather than uniqueness where y 1/2 # yı).
answer (e.g.. 'blue car') connects the two modalities, resulting in dependence given the label. As expected, S = 4.92, R = 0.79 for the VQA 2.0 dataset (Goyal et al., 2017).
2. R> S: Both modalities share lot of information but conditioning on Y reduces their dependence: T(X;; X22) > T(X;; X2JY), (x))), which results in more redundant than synergistic information. This setting reminiscent common effect structures. A real-world example in detecting sentiment from multimodal videos, where text and video are highly dependent since they emitted by the same speaker, but the sentiment label explains away some of the dependencies between both modalities. Indeed, for multimodal sentiment analysis from text, video, and audio of monologue videos on MOSEI (Zadeh et al., 2018), = 0.26 and S = 0.04.
The second relationship arises when two modalities contain disagreeing information about the task, and synergy arises due to this disagreement in information. To illustrate this, suppose y/1 arg max, p(y(=1) is the most likely prediction from the first modality, 3/2 = arg max, m (2(2)) ) for the second modality, and y = arg max, (()(21,22)) is the true multimodal prediction. There are again 2 cases (see right side of Figure 1):
Synergy and uniqueness
1. U> S: Multimodal prediction y = arg max, y p(yl,,, x2) is the same as one the unimodal predictions (e.g.. y = 12), in which case unique information in modality 2 leads to the outcome and there is no synergy. A real-world dataset MIMIC involving mortality and disease prediction from tabular patient data and time-series medical sensors (Johnson et al., 2016) which primarily shows unique information in the tabular modality. The disagreement on MIMIC is high at 0.13, but since disagreement is due
lot of unique information, there is less synergy S 0.01.
2. S > U: Multimodal prediction y is different from both y1 and y2. then both modalities interact synergistically to give rise a final outcome different from both disagreeing unimodal predictions. This type of joint distribution is indicative of real-world expressions of sarcasm from language and speech the presence sarcasm typically detected due to contradiction between what expressed in language and speech, as we observe from the experiments on MUSTARD (Castro al., 2019) where 0.44 and disagreement 0.12 are both large.
3.2 LOWER AND UPPER BOUNDS ON SYNERGY
Given these relationships between synergy and other interactions, we now derive bounds on S. We present two lower bounds Sn and S.. which are based on redundancy and uniqueness, as well as an upper bound S. We also describe the computational complexity for computing each bound.
Remark on high dimensional, continuous modalities. Our theoretical results are concerned with finite spaces for features and labels. However, this may be restrictive when working with real-world datasets (c.g., images, video, text) which are often continuous and/or high- digendimensional. In such situations, we preprocess by performing discretization of each modality via clustering to estimate p(11, y), p(22, :) p(x1, 222), each with small, finite support. These are subsequently used for the computation of SR. Sy and S. Discretization common way to approximate information theoretic quantities like mutual information (Darbellay & Vajda, 1999; Liang et al., 2023a) and for learning representations over high- digendimeniooall modalities (Oord al., 2018).
Lower bound using redundancy Our first lower bound uses the relationship between synergy, redundancy, and dependence in equation 3. In semi-supervised settings, we can compute R exactly from p(x1,y), v)) p(22,1) y), well as the shared information X() from ((21, However, (p X2))" cannot be computed without access to the full distribution p. In Theorem 1, we obtain lower bound on (X11; X2(Y). resulting in a lower bound Sp for synergy.
4

---

Published as a conference paper at ICLR 2024
Theorem 1. (Lower-bound synergy via redundancy) We relate S to modality dependence
= R-- (x)
min 5 reAp1,
(4)
We include the full proof in Appendix B.3. This bound compares S to R via the difference of their dependence Ip(Xx;XX2) and their dependence given the task xx(xx;xxll)) distribution is not available to compute (x X2/Y), we prove a lower bound using conditional MI computed with respect to set of auxiliary distributions r Ap12.12 that are close to p. as measured by matching both unimodal marginals r(z;,) y) - =(2 3) and modality marginals (2) = (2),, If conditioning on the task increases the dependence and (, X2 Y) is large relative to 1p( X2) then we obtain a larger value of SR. otherwise if conditioning on the task decreases the dependence and (X1;; X2 is small relative to (X1;; X2) then we obtain smaller value of SR.
Since the full
Computational complexity. R and m (xxx;;xy|Y) are convex optimization problems solvable in polynomial time with off-the-shelf solvers. Ip(X1: X2) can be computed directly.
Lower bound using uniqueness Our second bound formalizes the relationship between disagree- ment, uniqueness, and synergy. The key insight is that while labeled multimodal data unavailable, the output of unimodal classifiers may be compared against each other. Consider unimodal classifiers fi X. + y and multimodal classifiers fM X1 X2 + y.. Define modality disagreement as:
Definition 2. (Modality disagreement) Given X1, X2, and target Y, well as unimodal classifiers fi and fz, we define modality disagreement as ((f1,f [x) [ f2)] where d y x y R220 distance function in label spuce scoring the disagreement of f1 and J2's predictions.
Connecting modality disagreement and synergy via Theorem yields lower bound Sv:
Theorem (Lower- on synergy via uniqueness, informal) We can relate synergy and uniqueness U to modality disagreement a f2) of optimal unimodal classifiers f1.f2 as follows:
Su =u=a(Ji.B3 c-
S
(5)
for some constant e. depending on the label dimension [ and choice of label distance fanction d.
Theorem implies that there substantial disagreement a(f1, between unimodal classifiers, it must be due to the presence of unique synergistic information. If uniqueness is small, then disagreement must be accounted for by synergy, thereby yielding a lower bound Sy. Note that the optimality of unimodal classifiers important: poorly trained unimodal classifiers could show high disagreement but would be uninformative about true interactions. We include the formal version of the theorem based on Bayes' optimality and full proof in Appendix B.4.
Computational complexity. Lower bound Sy can also be computed efficiently by estimating p(y(æ1) and (()(22) over modality clusters or training unimodal classifiers fo (y((x)) and (2(22)). U1 and U2 can be computed using a convex solver in polynomial time.
Hence, the relationships between S, R, and U yield two lower bounds Sg and S.. Note that these bounds always hold, so we could take S max S Su).
R U1 U2. However, Ip(1X,,Xy) X2): Y) cannot be computed exactly without the full distribution p. Using the same idea as lower bound we upper bound synergy by considering the worst-case maximum computed over a set of auxiliary distributions Ap1,a1 that match both unimodal marginals r(I,,)) p(x,,y) y) and modality marginals r(x1,I2) =((x),2)):
Upper bound on synergy By definition,
max 1((X,, reAp1,2.1
(X,,X + H,(Y) - (X,,, re
max
(6)
(X1, H HY)
min
(2,,,X2,
reAp1,a.2
where the second line follows from the definition of A Ap1.3,2 While the first two terms are easy to compute, the third may be difficult, as shown in the following theorem:
Theorem 3. Solving r* = arg min minl p1,.12 (X1, X2, is NP-hard, even for a fixed | 2 4.
5

---

Published as a conference paper at ICLR 2024
Theorem 3 suggests cannot tractably find a joint distribution which tightly upper bounds synergy when the feature spaces are large. Fortunately, a relaxation of r E 4p.,,1 to r E A12,," where ((11,2)) .22 = p(r1, and r(y) p(y). recovers the classic min-entropy coupling problem over (X1, X2) and Y. which is still NP-hard but admits good approximations (Cicalese & Vaccaro, 2002; Cicalese et al., 2017; Kocaoglu et al., 2017; Compton et al., 2023). Our final upper bound S is:
Theorem 4. (Upper-bound on synergy)
S s Hx( + HY(Y) min reAp12.
H,, - R- U2=
Proofs of Theorem 3, 4, and detailed approximation algorithms for min-entropy couplings included in Appendix B.5 and B.6.
Computational complexity. The upper bound S can be computed efficiently since solving the variant of the min-entropy problem in Theorem 4 admits approximations that can be computed in time Okk log k)) where k max (). [2)). All other entropy and R. U1, U2 terms are easy to compute (or have been computed via convex optimization from the lower bounds).
Practically, calculating all three bounds is extremely simple, with just few lines of code. The computation takes < minute and < 180 MB memory space on average for our large datasets (1, 0,00000 20, 000 datapoints), more efficient than training even the smallest multimodal prediction model which takes at least 3x time and 15x memory. As result, these bounds scale to large and high- digh-ddimessonal multimodal datasets found in the real world, which we verify in the following experiments.
4 EXPERIMENTS
We design comprehensive experiments to validate these estimated bounds and relationships between different multimodal interactions. Using these results, we describe applications estimating optimal multimodal performance before training the model itself, which can be used to guide data collection and select appropriate multimodal models for various tasks.
4.1 VERIFYING INTERACTION ESTIMATION IN SEMI-SUPERVISED LEARNING
= {0, 1}. We generate joint distributions A by sam- pling 100, 000 vectors from the 8-dim probability simplex and assigning them to p (11, (2, y).
Synthetic bitwise datasets Let X1 = X = =
Large real-world multimodal datasets We use collection real-world datasets from MultiBench (Liang et al., 2021) which add up to a size of more than 700, 000 datapoints.
S S
0.8
MOSI: 2, 199 videos for sentiment analysis (Zadeh et al., 2016), 2. MOSEI: 23, 000 videos for sentiment and emotion analy-
6.4
sis (Zadeh et al., 2018),
3. MUSTARD: 690 videos for sarcasm detection (Castro et al.,
0.0
2019),
0.0
4. UR-FUNNY: dataset of humor detection from 16, 000 TED talk videos (Hasan et al., 2019).
02
Su S
-0.4
5. MIMIC: 36, 212 examples predicting patient mortality and dis-
eases from tabular patient data and medical sensors (Johnson al., 2016).
SR
-0.
6. ENRICO: 1, 460 examples classifying mobile user interfaces and screenshots (Leiva et al., 2020).
Figure 2: Our two lower bounds Sg and S, track actual synergy S from below, and the upper bound S tracks S from above. We find that Sp. Sy tend to approximate S hetter than S.
7. IRFL: 6, 697 images and figurative captions (c.g, 'the car is as fast as a cheetah' describing an image with a fast car in it) (Yosef et al., 2023).
NYCaps: 1, 820 New York Yimes cartoon images and humorous captions describing these images (Hessel et al., 2022).
9. VQA: 614, 000 questions and answers about natural images (An- tol et al., 2015).
10. ScienceQA: 21, 000 questions and answers about science prob- lems with scientific diagrams (Lu al., 2022).

---

Published as a conference paper at ICLR 2024
Table I: We compute lower bounds Sg. Sy. and upper bound S in semi sumi-viid multimodal settings and compare them to S assuming knowledge of the full joint distribution p. The bounds always hold and track S well MOSEL, UR-FUNNY, MOSI, and MUSTARD: true 9 increases as estimated Sx and E. increases.
MOSEI
UR-FUNNY MOST MUSTARD MIMIC ENRICO NYCAPS IRFL. VQA SCIENCEQA
0.97
0.97
0.92
0.79
0.41
2.09
0.68
0.01
0.97
1.67
S
0.03
0.18
0.24
0.44
0.02
1.02
0.09
0.05
0.16
S
0
0
0.01
0.04
0
0.01
0
0
0.01
0.01
0.01
0.03
0.11
-0.12
-0.55
-0.03
-0.01
0
F
2
y
p
0
0
1
0.05
0
0.03
0
1
1
0.28
T1
22
y
p
0
0
0.53
0
0
0.25
0
0.25
0
1
0.03
0
1
0.25
0
1
0
0.25
J2
p
0
0.01
1
0
0.25
0
1
0.25
0
0
0.5
1
1
0.06
1
1
0.25
1
1
0.25
1
1
0.5
(a) Disagreement XOR
(b) Agreement XOR
(c) y = X1
(d) y = I1
Table 2: Four representative examples: (a) disagreement XOR has high disagreement and high synergy, (b) agreement XOR has disagreement and high synergy, (c) y has high disagreement and uniqueness but no synergy, and (d) y = 21 22 has high agreement and redundancy but synergy.
These high-dimensional and continuous modalities require approximating disagreement and mutual information: we train unimodal classifiers fo (2((x) and fo (x((z)) to estimate disagreement, and we cluster modality features approximate continuous modalities by discrete distributions with finite support to compute the lower and upper bounds. We summarize the following regarding the validity of each bound (see details in Appendix C):
1. Overall trends For the 100, 000 bitwise distributions, compute S, the true value of synergy assuming oracle knowledge of the full multimodal distribution, and compute Sp S, Su S, and S S for each point. Plotting these points as histogram in Figure 2, we find that the two lower bounds track synergy from below (Sn and S. S approaching O from below), and the upper bound tracks synergy from above (S - approaching from above). The two lower bounds are quite tight, see that for many points SR - and Su S are approaching close to 0, with average gap of 0.18. Sy seems to be tighter empirically than Sp: for half the points, Sy is within 0.14 and Sq is within 0.2 of S. For the upper bound, there is an average gap of 0.62. However, it performs especially well on high synergy data: when S > 0.6, the average gap is 0.24, with more than half of the points within 0.25 of S.
On real- world MultiBench datasets, show the estimated bounds and actual S computed assuming knowledge of full p in Table 1. The lower and upper bounds track true S: as estimated Sg and Sy increases from MOSEI to UR-FUNNY to MOSI to MUSTARD, true S also increases. For datasets like MIMIC with disagreement but high uniqueness, Su can be negative, but we can rely on Sg to give tight estimate low synergy. Unfortunately, our bounds do not track synergy well on ENRICO. We believe this is because ENRICO displays all interactions: R = 0.73, U1 = 0.38, U2 = 0.53, S = 0.34, which makes difficult to distinguish between R and using SR or and using St since no interaction dominates over others, and also quite loose. Given these general observations, we now carefully analyze the relationships between redundancy, uniqueness, and synergy.
2. Guidelines We provide guideline decide whether a lower or upper bound synergy can be considered "close enough'. It is close enough if the maximum interaction can be consistently estimated often the exact value of synergy rather synergy relative to other interactions (e.g., if we estimate S [0.2, 0.5], and exactly compute R U1 U2 0.1, then we know for sure that S is the most important interaction and can collect data or design models based on that). We find that our bounds accurately identify the same highest interaction on 10 real-world datasets as the true synergy does. Furthermore, we observed that the estimated synergy correlates very well with true synergy: as high 1.05 on ENRICO (true $ = 1.02) and as low as 0.21 on MIMIC (true S 0.02).
the most important (e.g, whether S is 0.5 0.6) but
7

---

Published as a conference paper at ICLR 2024
Table 3: Estimated lower, upper, and average bounds on optimal multimodal performance comparison with the actual best unimodal model, the best simple fusion model, and the best complex fusion model. Our performance estimates closely predict actual model performance, despite being computed only semi- supervise data and never training the model iself.
MOSEI
UR-FUNNY
MOST
MUSTARD
MIMIC
ENRICO
Estimated upper bound
1.07
1.21
1.29
1.63
1.27
0.88
Best complex multimodal
0.88
0.77
0.86
0.79
0.92
0.51
Best simple multimodal
0.85
0.76
0.84
0.74
0.92
0.49
Best unimodal
0.82
0.74
0.83
0.74
0.92
0.47
Estimated lower bound
0.52
0.58
0.62
0.78
0.76
0.48
Estimated average
0.80
0.90
0.96
1.21
1.02
0.68
3. The relationship between S and R In Table 2b we show the classic AGREEMENT XOR distribution where X1 and X₂ are independent, but Y = sets X1 X₂ to increase their dependence. [X1;; X2; Y) is negative, and Sx = 1 S 1 S is tight. On the other hand, Table 2d is extreme example where the probability mass is distributed uniformly only when y = Z1 = 22 and 0 elsewhere. As result, X1 is always equal to X₂ (perfect dependence), and yet Y perfectly explains away the dependence between X1 and X₂ so T(X;; X2(Y) = O: Se = 0 s O = S. real-world example is multimodal sentiment analysis from text, video, and audio on MOSEL, as expected the lower bound is small SR = 0 s 0.03 = S (Table 1).
0.26 and S 0.03, and
4. The relationship between S and U In Table 2a we show an example called DISAGREEMENT XOR. There is maximum disagreement between p(y(r1) and p p(y(22): the likelihood for y is high when y is the opposite bit as æ1, but reversed for z2. Given both ø1 and Ø2: takes a 'diaageement' XOR of the individual marginals, i.c. plylr,, x22) arg max, p(y|z)) XOR arg max, p (2) which indicates synergy (note that an exact XOR would imply perfect agreement and high synergy). The actual disagreement is 0.15, S is 0.16, and U is 0.02, indicating a very strong lower bound Su = 0.14 & 0.16 = S. A real-world equivalent dataset is MUSTARD, where the presence of sarcasm often due to contradiction between what expressed language and speech, so disagreement a 0.12 is the highest out of all the video datasets, giving lower bound S. 0.11 s 0.44 = S.
The lower bound is low when all disagreement explained by uniqueness (e.g.. y = 21, Table 2c), which results in S., 0 0= S (a and U cancel each other out). A real-world equivalent is MIMIC: from Table 1, disagreement is high a = 0.13 due to unique information U₁ = 0.25, so the lower bound informs us about the lack of synergy Sn; -0.12 0.02 S. Finally, the lower bound is loose when there is synergy without disagreement, such as AGR EEMEEMENT XOR (y x1 XOR 222, Table 2b) where the marginals p(y]z;) are both uniform, but there is full synergy: S, 0 < 1 S. Real-world datasets include UR- FR-FUNNY where there is low disagreement in predicting humor a = 0.03, and relatively high synergy S 0.18, which results in a loose lower bound Su- 0.01 s 0.18 S.
5. On upper bounds for synergy The upper bound for MUSTARD is close real synergy. S = 0.79 2 0.44 S. On MIMIC, the upper bound is the lowest S 0.41, matching the lowest S 0.02. Some of the other examples in Table show weaker bounds. This could be because (i) there exists high synergy distributions that match Dj and D Pas, but these are rare in the real world, or (ii) our approximation used in Theorem 4 is loose. We leave these as directions for future work.
Additional results In Appendix C and E, we also study the effect of imperfect unimodal predictors and disagreement measurements on our derived bounds, by perturbing the label by various noise levels (from no noise to very noisy) and examining the changes in estimated upper and lower bounds. We found these bounds are quite robust to label noise, still giving close trends of S. We also include more discussions studying the relationships between various interactions, and how the relationship between disagreement and synergy can inspire new sulf-supe learning methods.
4.2 IMPLICATIONS TOWARDS PERFORMANCE, DATA COLLECTION, MODEL SELECTION
Now that we have validated the accuracy of these bounds, we apply them to estimate multimodal performance in semi- supervise settings. This serves strong signal for deciding (1) whether to collect paired and labeled data from a second modality, and (2) what type of multimodal fusion method should be used. To estimate performance given Dj, D2, and DM, we first compute our lower and upper bounds S and S. Combined with the exact computation of R and U. we obtain the total information I, ({X,, X2); YY) and combine a result from Feder & Merhav (1994) with Fano's inequality (Fano, 1968) to yield tight bounds of performance function of total information.
8

---

Published as a conference paper at ICLR 2024
Theorem 5. Let (fce(fM) (0)))BB., [1[ 22) yy] denote the accuracy of the Bayes' optimal multimodal model fM (L.e, Pace 2 (F)s for all f' F.s)). We have that
241((x1.X))))))10} Parc (Jis) ((((X,X2):Y)) +1
log [
and we can plug in R U1, U2 S Ip((,, X2):V) s R U2 + and upper P Pace f'M bounds on optimal multimodal performance.
S to obtain lower Pace (fãs)
We show the proof Appendix D. Finally, we summarize estimated multimodal performance as (7+++ + Pace(fr (f))(f A high PM suggests the presence of important joint
the average PM.- = (fīx) information from both modalities (not present in each) which could boost accuracy, so it is worthwhile to collect the full distribution and explore multimodal fusion.
Setup For each MultiBench dataset, we implement a suite of unimodal and multimodel models spanning simple and complex fusion. Unimodal models trained and evaluated separately on each modality. Simple fusion includes ensembling by taking an additive or majority vote between unimodal models (Hastie & Tibshirani, 1987). Complex fusion is designed to learn higher-order interactions as exemplified by bilinear pooling (Fukui et al., 2016), multplicattive interactions (Jayakumar et al., 2020), tensor fusion (Zadch al.. 2017), and cross-modal self-attention (Tsai al., 2019). See Appendix D for models and training details. We include unimodal, simple and complex multimodal performance, as well as estimated lower and upper bounds on performance in Table 3.
RQ1: Estimating multimodal fusion performance How well could my multimodal model per- form? We find that estimating interactions enables us to closely predict multimodal model perfor- mance, before even training a model. For example, on MOSEI, we estimate the performance to be 52% based on the lower bound and 107% based on the upper bound, for average of 80% which is very close to true model performance ranging from 82% for the best unimodal model, and 85% 88% for various multimodal model. Estimated performances for ENRICO, UR-FUNNY, and MOSI are 68%, 90%, 96%, which track true performances 51%, 77%, 86%.
RQ2: Data collection Should I col- lect multimodal data? We compare estimated performance PMr with the actual difference between unimodal and best multimodal performance in Figure (left). Higher estimated PM correlates with a larger gain from uni- modal multimodal (correlation o 0.21 and rises to 0.53 if ignoring the outlier in MIMIC). MUSTARD and ENRICO show the most opportunity for multimodal modeling. Therefore, rough guideline is that if the esti- mated multimodal performance based on semi- semi-supervised data is higher, then collecting the full labeled multimodal data is worth it.
multimodal l-nimmddal
ENRICO MUSLARD
MUSIARD
MOSEI
complex plep-ll-----eeeeeeee complex-simpl simple
UR FUNNY
ENRICO
MOSI
MOSEI
MOSI
UR-FUNNY
MIMIC
MIMIC
Estimated performance Pir
Estimated performance Pir
Figure 3: Datasets with higher estimated multimodal performance
Pas tend to show improvements from unimodal to multimodal (left) and from simple to complex multimodal fusion (right).
RQ3: Model selection What model should I choose for multimodal fusion? We note strong relationships between estimated performance and the performance of different fusion methods. From Table 3, synergistic datasets like MUSTARD and ENRICO show best multimodal performance only slightly above our estimated lower bound, indicating that there is lot of room for improvement in better fusion methods. Indeed, more complex fusion methods such as multimodal transformer designed to capture synergy is the best on MUSTARD which matches its high synergy (72% accuracy). For datasets with less synergy like MOSEI and MIMIC, the best multimodal performance is much higher than the estimated lower bound, indicating that existing fusion methods may already be quite optimal. Indeed, simpler fusion methods such as feature alignment, designed capture redudnancy, are the best on MOSEI which matches its high redundancy (80% accuracy).
Figure (right) shows a visual comparison, where plotting the performance gap between complex and simple fusion methods against estimated performance PM shows a correlation coefficient 0.77. We again observe positive trends between higher estimated performance and improvements with
9

---

Published as a conference paper at ICLR 2024
complex fusion, with large gains on MUSTARD and ENRICO. We expect new methods to further improve the tfte-oofthhe-rtt on these datasets due to their generally high interaction values and low multimodal performance relative to estimated lower bound (f Therefore, rough guideline that if the estimated multimodal performance based on semi-supervised data is higher, then there is more potential for improvement by trying more complex multimodal fusion strategies.
5 CONCLUSION AND BROADER IMPACTS
We proposed estimators of multimodal interactions when observing only labeled unimodal data and some unlabeled multimodal data, general semi-supervised setting that encompasses many real-world constraints involving partially observable modalities, limited labels, and privacy concerns. Our key results draw new connections between multimodal interactions, the disagreement unimodal classifiers, and min-entropy couplings, which yield new insights for estimating multimodal model performance, data analysis, and model selection. We are aware some potential limitations:
1. These estimators only approximate real interactions due to cluster preprocessing or unimodal models, which naturally introduce optimization and generalization errors. We expect progress in density estimators, generative models, and unimodal classifiers address these problems.
2. It is harder to quantify interactions for certain datasets, such as ENRICO which displays all interactions which makes it difficult to distinguish between R and S or U and S.
3. Finally, there exist challenges in quantifying interactions since the data generation process is never known for real- wollworld datasets, so we have to resort to human judgment, other automatic measures, and downstream tasks such estimating model performance and model selection.
Future work should investigate more applications of multivariate information theory in designing self- supervised models, predicting multimodal performance, and other tasks involving feature interactions such as privacy preraccyreeeeiingg.m and fair representation learning from nigh-dimensional data (Dutta et al., 2020; Hamman & Dutta, 2023). Being able provide guarantees for fairness and ppeeccypres learning, especially for cen-ssupereised pretraining datasets, can be particularly impactful.
ACKNOWLEDGEMENT
This material is based upon work partially supported by National Science Foundation awards 1722822 and 1750439, National Institutes of Health awards R01MH125740, RO1MH132225, R01MH096951 and HHMMHH13775 and Meta. PPL is supported in part by a Siebel Scholarship and a Waibel Presidential Fellowship. RS is supported in part by ONR grant N00014231253 FA87502321015. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official en- dorsement should be inferred. Finally, we would also like to acknowledge feedback from anonymous reviewers who significantly improved the paper and NVIDIA's GPU support.
and DARPA
REFERENCES
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pp. 1247-1255, PMLR, 2013.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433 2015. 6
MOSEK ApS. MOSEK Optimizer API for Python 10.0.34, 2022. URL httph::////ccee www.s..cc......... com/ latest httpss//ttthhhpttpppttptet
html. 16
Maria-Florina Balcan, Avrim Blum, and Ke Yang. Co-training and expansion: Towards bridging theory and practice. Advances in neural information processing systems. 17, 2004. 3
Anthony J Bell. The co-information lattice. In Proceedings the fifth international workshop on independent component analysis and blind signal separation: ICA, volume 2003, 2003. 3
Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy, 2014. 1, 3, 15, 16
Abeha Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint 2 2021. 15
10

---

Published as a conference paper at ICLR 2024
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh anmual conference Computational learning theory, pp. 92-100, 1998.
Tolga Bolukhasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is computer programmer as woman is to homemaker? dehiasing word emheddings. Advances in neural information processing systems, 29, 2016. 15
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an _obviously_ perfect paper). In ACL, pp. 4619- 4629, 2019. 4, 6, 15, 26, 21
Liwei Che, Jiaqi Wang, Yao Zhou, and Fenglong Ma. Multimodal federated learning: A survey. Sensors, 23(15): 6986, 2023.
Ferdinando Cicalese and Ugo Vaccaro. Supermodularity and subadditivity properties of the entropy the majorization lattice. IEEE Transactions on Information Theory, 48(4):933 938, 2002. 6, 24, 25
Ferdinando Cicalese, Luisa Gargano, and Ugo Vaccaro. How find a joint probability distribution of minimum entropy (almost) given the marginals. In 2017 IEEE International Symposium on Information Theory (ISIT), pp. 2173-2177, IEEE, 2017. 6, 24
Spencer Compton. A tighter approximation guarantee for greedy minimum entropy coupling. In 2022 IEEE International Symposium Sympossi on Information Theory (ISIT), pp. 168-173. IEEE, 2022. 24
Spencer Compton, Dmitriy Katz, Benjamin Qi. Kristjan Greenewald, and Murat Kocaoglu, Minimum-entropy coupling approximation guarantees beyond the majorization barrier. In International Conference on Artificial Intelligence and Statistics, pp. 10445-10469, PMLR, 2023. 24
Georges Darbellay and Igor Vajda. Estimation of the information by an adaptive partitioning of the observation space. IEFE Transactions on Information Theory, 45(4):1315-13211 1999.
Jean-Benoit Delbrouck, Noé Tits, Mathilde Brousmiche, and Stéphane Dupont. A transformer-based joint- encoding for emotion recognition and sentiment analysis. In Second Grand-Challenge and Workshop on Multimodal Longuage (Challenge-HHMe HML), pp. 1-7, Seattle, USA, July 2020. Association for Computational Linguistics. doi: challengehml
URL https: http:///llaaanolg...... orgp//20..
29
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pp. 248-255 leee, 2009.
Daisy Yi Ding, Shuangning Li, Balasubramanian Narasimhan, and Robert Tibshirani. Cooperative learning for multiview analysis. Proceedings of the National Academy of Sciences, 119(38):c202111313 2022. 2
Alexander Domahidi, Eric Chu, and Stephen Boyd, Ecos: An socp solver for embedded systems. In 2013 European Control Conference (ECC), pp. 3071-3076. IEEE, 2013. 16
Sanghamitra Dutta, Praveen Venkatesh, Piotr Mardziel, Anupam Datta, and Pulkit Grover. An information- theoretic quantification of discrimination with exempt features. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 3825-3833 2020.
Shimon Even, Alon Itai, and Adi Shamir. On the complexity of time table and multi-commodity flow problems. In 16th annual symposium on foundations of computer science (xfcs 1975), pp. 184-193. IEÉE, 1975. 19, 20
Robert M Fano. Transmission of information: a statistical theory of communications. Mit Press, 1968. 28, 29
Meir Feder and Neri Merhav. Relations between entropy and error probability, IEEE Transactions on Information theory, 40(1): (00))))2992006 1994, 8, 28
Jerome H Friedman and Bogdan E Popescu. Predictive learning via rule ensembles. The annals of applied
statistics, 2(3)5916-954, 2008. 2
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Conference Empirical Methods in Natural Language Processing. ACL, 2016. 9
Amir Globerson and Tommi Jaakkola. Approximate inference using conditional entropy decompositions. In Artificial Intelligence and Statistics, pp. 131-138. PMLR, 2007. 16
11

---

Published as a conference paper at ICLR 2024
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6904-6911 2017. 4, 26
Virgil Griffith and Christof Koch. Quantifying synergistic mutual information. In Guided self-organization: inception, pp. 159-190. Springer, 2014. 27
Faisal Hamman and Sanghamitra Dutta, Demystifying local and global faimeess trade-offs in federated learning using information theory. In Federated Learning and Analytics Practice: Algorithms, Systems, Applications, and Opportunities, 2023. 10
Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis- Philippe Morency, and Mohammed Ehsan Hoque. Ur-funny: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing GEMNLP-JJCNLP pp. 2046 2056, 2019. 6, 15, 31
Md Kamrul Hasan, Sangwu Lee, Wasifur Rahman, Amir Zadeh, Rada Mihalcea, Louis-Philipee Morency, and Ehsan Hoque. Humor knowledge enriched transformer for understanding multimodal humor. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14): 12900, May 2021, doi: 10.1609/aaai. v35i14. 17534. URL https:/ /ojs. aaai. . org. http:/indxx....// php/ AAAI/AAAAa........A. CatAAA/aaailleaaeaeeeeeeeeeaaeee wttee//vewwi////// 29
Trevor Hastie and Robert Tibshirani. Generalized additive models: some applications. Journal of the American
Statistical Association, 1987. 9
Jack Hessel and Lillian Lee. Does my multimodal model learn cross- modal interactions? harder to tell than you might think! In EMNLP, 2020. 1,2
Jack Hessel, Ana Marasović, Jena D Hwang, Lillian Lee. Jeff Da. Rowan Zellers, Robert Mankoff, and Yejin Choi. Do androids laugh at electric sheep? humor" understanding" benchmarks from the new yorker caption contest. arXiv preprint arXiv:2209.062: 06293, 2022. 6,
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366 03669999666666 1989. 18
Tzu-Ming Harry Hsu, Wei-Hung Weng, Willie Boag, Matthew MeDermott, and Peter Szolovits. Unsupervised multimodal representation learning across medical images and reports. arXiv preprint arXiv: 1811.0861,, 2018.
Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9248-9257, 2019. 1
Guimin Hu, Ting-Ei En Lin, Zhao, Guangming Lu, Yuchuan Wu, and Yongbin Li. UniMSE: Towards uni- fied multimodal sentiment analysis and emotion recognition. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 7837-7851, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: //athtthhool ogy org/ 2022. emn1 1111p-mmmm in. 534. 29
Jan Ittner, Lukasz Bolikowski, Konstantin Hemker, and Ricardo Kennedy, Feature synergy, redundancy, and in- dependence in global model explanations using shap vector decomposition. arXiv preprint arXiv:2107.12436, 2021. 2
Siddhant M. Jayakumar, Wojeiech M. Czamecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where find them. In International Conference on Learning Representations, 2020. 1, 2,
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman. Mengling Feng, Mohammad Ghassemi. Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016. 4, 15, 26,
Murat Kocaoglu, Alexandros Dimakis, Sriram Vishwanath, and Babak Hassibi. Entropic causal inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017. 6, 24
Mladen Kovačević, Ivan Stanojevié, and Vojin Senk. On the entropy of couplings. Information and Computation, 242:369-382, 2015. 19
12

---

Published as a conference paper at ICLR 2024
Jie Lei. Licheng Yu, Mohit Bansal, and Tamara Berg. Tvqa: Localized, compositional video question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1369. Association for Computational Linguisties, 2018. 31
Luis A Leiva, Asutosh Hota, and Antti Oulasvirta. Enrico: A dataset for topic modeling of mobile ui designs. In 22nd International Conference Human- Homman-Computer Interaction with Mobile Devices and Services, pp. 1-4, 2020. 6, 26
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslic Yufan Chen, Peter Wu, Michelle Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. In Thirty-fifth Conference on Neurul Information Processing Systems Dutasets and Benclmarks Track (Round 1), 2021. 2,,2,,2 29
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Shengtong Mo, Dani Yogatama. Louis- Philippe Morency, and Ruslan Salakhutdinov. Highmmtt: Towards modality and task generalization for eigh-modality representation learning. arXiv preprint arXiv:2203.01311, 2022a. 1,2
Paul Pu Liang, Amir Zadeh, and Louis- Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions, arXiv preprint arXiv:2209.034333 2022b. 1,2
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nic, Richard Chen, Zihao Deng, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency. Quantifying & modeling feature interactions: An information decomposition framework. arXiv preprint arXiv:2302. 12247, 2023a. 1,3,4
Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingho Wang, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multiviz: Towards visualizing and understanding multimodal models. In International Conference on Learning Reprexentations, 2023b. URL https:/ https////peeeeeeeeeeeeeeee wwt.aeeeweeeeeeeeeeeeeeeeeeeeeee forum? ? 1id=12 IvCFnEml..
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kei-Wei Chang, Song-Chur Zhu, Oyvind Tafjord. Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-255 2521, 2022. 6
William McGill. Multivariate information transmission. Transactions of the IRE Professional Group on Information Theory, 4(4):93-111, 1954.
Brendan O' O'Dnooguue, Fric Chu, Neal Parikh, and Stephen Boyd. Conic optimization via operator splitting and homogeneous self-dual embedding. Journal of Oprimization Theory and Applications, June 2016. 16
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: arXiv:/807.00 0877033778 2018. 4,29
Petra Poklukar, Miguel Vasco, Hang Yin, Francisco S Melo, Ana Paiva, and Danica Kragic. Geometric multimodal contrastive representation learning. In International Conference on Machine Learning, pp. 17782-17800. PMLR, 2022.
Shraman Pramanick, Aniket Basu Roy, and Vishal M. Patel. Multimodal learning using optimal transport for sarcasm and humor detection. 2022 IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pp. 546-556, 2021. 29
Alce Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021. 1, 29
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher 7adeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. Integrating multimodal information large pretrained transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2359-2369, Online, July 2020. Association for Computational Linguistics. doi: 10.18655/ mai.mm..211144 URL
https: ac lanthology / 2020 acl -almmminnn 214. 29
Massimiliano Rossi. Greedy additive approximation algorithms minimum-entron entropy coupling problem. In 2019 IEEE International Symposium on Information Theory (ISIT), pp. 1127-112 1131. IEEE, 2019. 24
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Mareus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IFEE/CVF Conference on Computer Vision and Pattern Recognition, pp. -16500 2022. 29, 30
13

---

Published as a conference paper at ICLR 2024
Daria Sorokina, Rich Caruana, Mirek Riedewald. and Daniel Fink. Detecting statistical interactions with additive groves trees. In Proceedings of the 25th international conference on Machine learning, pp. 1000-1007, 2008. 2
Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. In Conference on Learning Theory, 2008. 3, 17, 18
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, In International conference on machine learning, pp. 3319 3328. PMLR, 2017. 2
Yonglong Tian, Chen Sun. Ben Poole. Dilip Krishnan, Cordelia Schmid. and Phillip Isola. What makes for good
views for contrastive learning? Advances in Neural Information Processing Systems, 33, 2020. 3
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi- view redundancy, and linear models. In Algorithmic Learning Theory, pp. 1179-1206 PMLR, 2021. 2,
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis- Phiis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6558 000--656,220 2019. 9
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis- Phiil-Philippe Morency. Self-supervised learning from multi-view perspective. In International Conference on Learning Representations, 2020. 3
Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. In International Conference on Learning Representations, 2018. 2
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi- task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.00 1804.0741,, 2018. I
Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint WxXiv1004.25155 2010. 2, 15
Kaicheng Yang, Hua Xu, and Kai Gao. Cm-bert: Cross-modal bert for text-audio sentiment analysis. In Proceedings the 28th ACM International Conference Multimedia, MM '20, pp. 521-528. New York, NY. USA, 2020. Association for Computing Machinery. ISBN 9781450379885. doi: 10.1145/3394171.3413333
URL https: : /oi crg/10. 11 3394//331111 71 3341 3690. 29
Ron Yosef, Yonatan Bitton, and Dafna Shahaf. Irfi: Image recognition of figurative language. arXiv preprint arXiv:2303. 15445, 2023.
Lei Yu and Huan Liu. Efficient feature selection via analysis of relevance and redundancy. The Journal of Machine Learning Research, 5:1205-1224 2004, 2
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis Philipp Moreney. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint 16Xi::600000655: 2016. 6, 26
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural
Language Processing, pp. 1 1103-1114 2017. 1, 1,2,9
Amir Zadeh, Michael Chan, Paul Pu Liang. Edmund Tong, and Louis-Philippes Morency. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8807-8817, 2019. 31
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 2236-2246, 2018, 4, 6, 15, 26
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjac Yu, Yampeng Zhao, Mohammadreza Salchi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
16375-16387, 2022. 2, 29, 30, 31
14

---

Published as a conference paper at ICLR 2024
APPENDIX
A BROADER IMPACT
Multimodal semi-supervised models are ubiquitous in a range of real-world applications with only labeled unimodal data and naturally co-occurring multimodal data (e.g.. unlabeled images and captions, video and corresponding audio) but when labeling them is time-consuming. This paper is our attempt at formalizing the learning setting of multimodal semi-supervised learning, allowing us to derive bounds on the information existing in multimodal semi-supervised datasets and what can be learned by models trained on these datasets. We do not foresee any negative broad impacts of our theoretical results, but we do note the following concerns regarding the potential empirical applications of these theoretical results in real-world multimodal datasets:
Biases: We acknowledge risks of potential biases surrounding gender, race, and ethnicity in large- scale multimodal datasets (Bolukbasi et al., 2016), especially those collected in semi-supervised setting with unlabeled and unfiltered images and captions (Birhane et al., 2021). Formalizing the types of bias in multimodal datasets and mitigating them an important direction for future work.
When making predictions from multimodal datasets with recorded human behaviors and medical data, there might be privacy risks of participants. Following best practices in maintaining the privacy and safety of these datasets, (1) these datasets have only been collected from public data that are consented for public release (creative commons license and following fair use guidelines of YouTube) (Castro et al., 2019; Hasan et al., 2019; Zadeh et al., 2018), or collected from hospitals under strict IRB and restricted access guidelines (Johnson et al., 2016), and (2) have been rigorously de-identified in accordance with Health Insurance Portability and Accountability Act such that all possible personal and protected information has been removed from the dataset (Johnson et al., 2016). Finally, we only use these datasets for research purposes and emphasize that any multimodal models trained to perform prediction should only be used for scientific study and should not in any way be used for real-world harm.
Privacy:
B DETAILED PROOFS
B. INFORMATION DECOMPOSITION
Partial information decomposition (PID) (Williams & Beer, 2010) decomposes of the total information 2 variables provide about a task L [({1,,X2);1) into quantities: redundancy R between X1 and X2. unique information U₁ in X, and U2 in X2, and synergy S. Williams & Beer (2010), who first proposed PIDs, showed that they should satisfy the following consistency equations:
R+ I();)))
(10)
R+U = ((X2;),,
(11)
U1+8=1(XX;
(12)
U2+5=1(Xx;
(13)
R- (xX; X2: Y).
(14)
We choose the PID definition by Bertschinger et al. (2014), where redundancy, uniqueness, and synergy are defined by the solution to the following optimization problems:
R max X2;11
(15)
Y|X2) U1 min (X1;; ((12))) <
(16)
U2 = min I (x22 Yl q
(17)
S = Ip ((X, X2):Y) gpap
min
(18)
where Ap = (q E ((,,,))))))))))))))))))(()) v) = p(a;, 3) Vy, {1, 2)}, A is the set of all joint distributions over X1, Y. and the notation (.)() and 1,(-) disambiguates MI under joint distributions p and q respectively. The key difference in this definition of PID lies in optimizing q E Ap to satisfy the marginals ( v) p(z,.)) y), but relaxing the coupling between I1 and zz: g( 222) need not be equal to p(x,, 22). The intuition behind this is that one should be able to infer redundancy and uniqueness given only access to separate marginals p(1,, y) and p(12, y), and therefore they should
15

---

Published as a conference paper at ICLR 2024
only depend o Ao which match these marginals. Synergy, however, requires knowing the coupling p(11, x2), and this is reflected in equation (18) depending on the full p distribution.
B.2 COMPUTING q', REDUNDANCY, AND UNIQUENESS
According to Bertschinger et al. (2014), it suffices to solve for q using the following max-entropy optimization problem q" arg aa, (Y X2), the same q" equivalently solves any of the remaining problems defined for redundancy, uniqueness, and synergy. This is concave maximization problem with linear constraints. When X; and y are small and discrete, we can represent all valid distributions (2 22 y)
set of tensors Q of shape [Xg] x [22] [3/] with each entry representing jjj p(X, = i, X2 = j.Y== = k). The problem then boils down to optimizing over valid tensors Q E Ap that match the marginals p( p( y) for the objective function (Y X2). We rewrite conditional entropy as KL-divergence (Globerson & Jaakkola, 2007). (Y
1 enforced using linear -o - (gll4), where q is an auxiliary product density of ((11, x2) [i
constraints: (x11,22 #2, y) ((11, The KL-divergence objective recognized as convex, allowing the use of conic solvers such as SCS (O Donoghuu et al., 2016), ECOS (Domahidi et al., 2013), and MOSEK (ApS, 2022).
Finally, optimizing over Q Ap that match the marginals can also be enforced through linear constraints: the 3D-tensor Q summed over the second dimension gives (( y) and summed over the first dimension gives (x22, y). yielding the final optimization problem:
arg max KI(QII@). s.t. Q(11, (2.9) Q.Q
(19)
-Q= v) = p v).Q2)2.0
-(= Σ Q=1.
(20)
I1 22.
After solving this optimization problem, plugging q" into (15)-(17) yields the desired estimators for redundancy and uniqueness: R= L. X2; U l.- (X1; Yi;),, Lg- (X2; Y|X)), and more importantly, can be inferred from access to only labeled unimodal data p(11, y) and p(#2, y). Unfortunately, S is impossible to compute via equation (18) when do not have access to the full joint distribution p. since the first term [X1,,, X2; Y) is unknown. Instead, we will aim to provide lower and upper bounds in the form S s S s S so that we can have a minimum and maximum estimate on what synergy could be. Crucially, S and S should depend only on D1, D2, and DM in the multimodal sup-ssuper setting.
B.3 LOWER BOUND ON SYNERGY VIA REDUNDANCY (THEOREM 1)
We first restate Theorem from the main text to obtain our first lower bound Sy linking synergy to redundancy:
Theorem 6. (Lower-bound on synergy via redundancy, same as Theorem 1) We can relate S to R as follows
Sp= R 1-1111(X1;222
min 4(K1; S reAp1,2.2
(21)
where A = fre A :: x2) p( x2); r (Z,,u) = p(x;,)))) v)). min (Xxx;XX\V1 is a max-entropy convex optimization problem which can be solved exactly using linear programming.
Proof. By consistency equation (14), S -
Note that can be computed exactly based on p(x1, ((Z11,)..((22, ((2,,12)))) and [x((; can be computed exactly based on p( 22), but I, (x11;; X2/Y') requires knowledge of the full distribution p(11, 22. y) to compute. We will instead lower bound the conditional mutual information (.(XIXX/Y'' by computing its minimum with respect to set of auxiliary distributions A p1.2112 that match both unimodal marginals r(z,, v) p(2,,8) v) and modality marginals r(11,22) pl p( (2), which yields a lower bound on synergy. We obtain:
min Xx(x)x
= min H,(X1) - - (,(Y; -, redp1,211
(22)
red21,2.1
= Hx1)))
(im)))I((XX1 (X1;)YY)
max Hx(Xi1X2,))
(23)
redp1.13
16

---

Published as a conference paper at ICLR 2024
where in the last line the P2 constraint is removed since (X,[X((2, is fixed with respect to p(22.y). To solve maXredp1.12 Hr (X2, Y), we observe that it is also a concave maximization problem with linear constraints. When X; and y are small and discrete, we can represent all valid distributions r(11, 22, 3) set of tensors It of shape [X1] x [2g] x [3] with each entry representing Ri k]] p(X1 = i, X2 = Y. = k). The problem then boils down to optimizing over valid tensors R pr.12 that match the marginals p(x1. 9) and x2)). Given a tensor R representing r, our objective is the concave function (X,X||X2. Y) which rewrite KI--divergence log - using an auxiliary distribution F r(22.2 y) and solve it exactly using convex programming with linear constraints:
arg max maxxxx (R))R)) s.t. R(21.12,8) -((=,
(24)
R.
Σ R=1.
(25)
with marginal constraints R. E Ap enforced through linear constraints on tensor R. Plugging the optimized y* into (21) yields the desired lower bound Sp R (x(1;; + (X1; O
B.4 LOWER BOUND ON SYNERGY VIA UNIQUENESS (THEOREM 2)
We first restate some notation and definitions from the main text for completeness. The key insight behind Theorem relationship between synergy and uniqueness, is that while labeled multimodal data is unavailable, the output of unimodal classifiers may be compared against each other. Let 5y = fr R211 ||]||| 1} be the probability simplex over labels V. Consider the set of unimodal classifiers F fo: X₂ + 8y and multimodal classifiers FM 3 fM : X1 x X2 + 8y.
Definition 3. (Unimodal and multimodal loss) The loss of a given unimodal classifier fi E Fi is given by L(f:) = Ep(.,s) [E(f((z:), y)1 for a loss function over the label space & y x x + R2 We denote the same for multimodal classifier Jas Frs, with a slight abuse of notation L(JM) B2(2) [f[f (x1, ) for loss function over the label space f.
Definition 4. (Unimodal and multimodal accuracy) The accuracy of a given unimodal classifier fi E Fi is given by (fi) Ep [1 [f((z,) =y]]. We denote the same for multimodal classifier fas e FMM with a slight abuse of notation (fss(fas) = Ep (11,
An unimodal classifier fü is Bayes-optimal (or simply optimal) with respect loss function L if L(f:) $ L(F)) for all f. E F,, Similarly, a multimodal classifier fūs is optimal with respect to loss L Lif T(J%) s T(f')) for all I'M e F.
Bayes optimality can also be defined with respect to accuracy, if (fi) 2 (a) for all f. e F, for unimodal classifiers, or if Pace (f&s Pace 2 Pacc for all f'a FM for multimodal classifiers.
The crux of our method is to establish a connection between modality disagreement and a lower bound on synergy.
Definition 5. (Modality disagreement) Given X1, X2, and a target Y, as well as unimodal classifiers fi and fz, we define modality disagreement as oc f2) E [d(1i, f2)] where d y x y R a distance function in label space scoring the disagreement of fi and f2 's predictions.
where the distance function d must satisfy some common distance properties, following Sridharan & Kakade (2008):
Assumption 1. (Relaxed triangle inequality) For the distance function d y y + IR20 in label space scoring the disagreement of f1 and f2 's predictions, there exists Ca 2 such that
"ūr,ū2, in E v, d(ú1,(n) S ca (d(ri, (3) + d( (2)).
(26)
Assumption 2. (Inverse Lipschitz condition) For the function d, it holds that for all J,
(11))()))))2)) J (2), s (I(()) -[([")
(27)
where f is the Bayes optimal multimodal classifier with respect loss L, and B[d(f((x:), f (a+))] s -(((f))((()))
(28)
where f is the Bayes optimal unimodal classifier with respect to loss T..
17

---

Published as a conference paper at ICLR 2024
Assumption 3. (Classifier optimality) For any unimodal classifiers f1 f2 in comparison to the Bayes' optimal unimodal classifiers f2. there exists constants c1, E2 > O such that
[L(i) L(fi)( c1.1 L(f2)) c2
(29)
We now restate Theorem 2 from the main text obtaining S.. our second lower bound on synergy linking synergy to uniqueness:
Theorem 7. (Lover-bound on synergy via uniqueness, same as Theorem 2) We can relate synergy S and uniqueness U modality disagreement f2) of optimal unimodal classifiers fr f2 as follows:
Eu =u=oc =u=a(J1.1>) c- max((xx S
(30)
for some constant e depending on the label dimension [ and choice of label distance function d.
Theorem implies that if there substantial disagreement between the unimodal classifiers fi and f2. it must be due to the presence of unique or synergistic information. If uniqueness is small, then disagreement must be accounted for by the presence synergy, which yields a lower bound.
Proof. The first part of the proof is due to an intermediate result by Sridharan & Kakade (2008), which studies how multi-view agreement can help train better multiview classifiers. We restate the key proof ideas here for completeness. The first step is to relate (X2; Y|Xi) 1) to [L(fi) L(f*)) )
the difference in errors between the Bayes' optimal unimodal classifier f with the Bayes' optimal multimodal classifier fr for some appropriate loss function L on the label space:
|2(j)) - 1(-) Exx,, (11, - Ex (1),,
(31)
EvIx, ((' (21, Ey,x, (I)
(32)
(L(plla,, 12) p(yl)))
(33)
Ex ( p())a))
(34)
(x(X2;\(x))
(35)
where we used Pinsker's inequality in (33) and Jensen's inequality in (34). Symmetrically, /L(S2 L( s (X1;; 22)), and via the triangle inequality through the Bayes' optimal multimodal classifier fr and the inverse Lipschitz condition we obtain
E Ep mx) [d(fi.
[ E (o) f5)1
(36)
s <|L(fi) (( ((
(37)
$ + (X1;;
(38)
Next, we relate disagreement (((1,f2) to (X2; and (X,; Y2X2) via the triangle inequal- ity through the Bayes' optimal unimodal classifiers fi and f
(2)
Ep(zi [2)
(39)
s ca (4
+
(40)
$ 7,;
I ca (6 1, (X2: I,(X;; 6))
(41)
+ ())))
(42)
where used classifier optimality assumption for unimodal classifiers fi, f2 in (41). Finally, we use consistency equations of PID relating U and in (12)-(13) complete the proof:
a(fi,f2) 2ca(max (x (Xxx))))))) (ax((.,6)
(43)
2ca(max + S, S) 3)+++ ()))(()))))
(44)
2c.(S + max(U,, U2) max(((,,)))
(45)
In practice, setting f1 and f2 neural network function approximators that achieve the Bayes' optimal risk (Hornik et al., 1989) results in max(é,, e/) = 0, and rearranging gives us the desired inequality.
18

---

Published as a conference paper at ICLR 2024
B.5 PROOF OF NP- HARDNESSS (THEOREM 3)
Our proof is based on reduction from the restricted timetable problem, well-known scheduling problem closely related to constrained edge coloring in bipartite graphs. Our proof description proceeds along steps.
1. Description of problem.
2. How the minimum entropy objective can engineer "classification" problems using a tech- nique from Kovačević et al. (2015).
3. Description of the RTT problem of Even et al. (1975), how to visualize RTT as a bipartite edge coloring problem, and a simple variant we call Q-RTT which RTT reduces to.
4. Polynomial reduction of Q-RTT to our problem.
B.5.1 FORMAL DESCRIPTION OF OUR PROBLEM
Recall that our problem was
min reAy1,2,12
(,,,,, X2,
where A (r A x(x1)22) p(x1, ( p(z;, Our goal is to find the minimum-entron distribution over 22 x22x)' where the pairwise marginals over X2), (X1,Y) and (X2, Y) are specified as part of the problem. Observe that this description is symmetrical, X; and Y could be swapped without loss of generality.
B.5.2 WARM UP: USING THE EIN-EY OBJECTIVE TO MIMIC MULTICLASS CLASSIFICATION
We first note the strong similarity our min-entropy problem to the classic min-entropy coupling problem in two variables. There where the goal is to find the min-entropy joint distribution over X given fixed marginal distributions p(x) and p(y). This was shown to be an NP-hard problem which has found many practical applications in recent years. An approximate solution up 1 bit can be found in polynomial time (and is in fact the same approximation we give to our problem). Our hp-haad proof involves has similar flavor Kovačević al. (2015), which is based on a reduction from the classic subset sum problem, exploiting the min-entropy objective to enforce discrete choices.
Subset sum There are d items with value ci .....C Ca 2 0, which we assume WLOG to be normalized such that Σd C₁ 1. Our target sum is 0 T' s 1. The goal is find if some subset S [d]] exists such that Eie G = T.
Reduction from subset sum to min-entropy coupling (Kovačević et al., 2015) Let X' be the d items and y be binary, indicating whether the item was chosen. Our joint distribution is of size |2] x (/ We set the following constraints on marginals.
(i) p(x:) = c; for all i, (row constraints)
(ii) (inccluee) T, p(omit) T, (column constraints)
Constraints (i) split the value of each item additively into nonnegative components to be included and not included from chosen subset, while (ii) enforees that the items included sum to T. Observe that the min-entropy objective H H(X,)) = HY|)))+ + H (X), which is solely dependent on HYYIX) since H(X) ) is constant given marginal constraints
X. Thus, H(Y]X) is nonnegative and is only equal to 0 if and only if Y is deterministic given X, i.c., r(z,, include) = 0 or r(z,, omit) = 0. If subset sum problem has solution, then this instantiation of the min-entropy coupling problem would return a deterministic solution with HY) = O, which in turn corresponds to a solution in subset sum. Conversely, subset sum has no solution, then our min-entropy coupling problem is either infeasible OR gives solutions where H/ (Y]X)) > 0 strictly, i.e., Y|X is non-deterministie, which we can detect and report.
'Strictly speaking, the marginals 72) and p( v) ought to he rational. This is not overly restrictive, since in practice these marginals often correspond to empirical distrihutions which would naturally be rational.
19

---

Published as a conference paper at ICLR 2024
Relationship to our problem Observe that our joint entropy objective may be decomposed H, X2, Y) H,(Y|X,, X2) (X1, X2).
Given that ,(11, ) is fixed under Apr our objective is equivalent to minimizing H, (Y|Y|X1, X2). Similar to before, we know that H, (Y is nonnegative and equal to zero if and only if Y is deterministic given (2).
Intuitively, we can use X1, to represent vertices in bipartite graph, such that (x) are edges (which may or may not exist), and y as colors for the edges. Then, the marginal constraints for p(11,22) 222 could be used alongside the min-entropy objective to ensure that each edge has exactly one color. The marginal constraints p(æ1, v) and p p(22. y) tell us (roughly speaking) the number of edges of each color that is adjacent to vertices in X1 and X2-
However, this insight alone is not enough; first, edge coloring problems in bipartite graphs (e.g., colorings in regular bipartite graphs) can be solved in polynomial time, so we need more difficult problem. Second, we need an appropriate choice of marginals for p(*+,,) y) that does not immediately 'reveal' the solution. Our proof uses reduction from the restricted timetable problem, one of the most primitive scheduling problems available (and closely related to edge coloring or multicommodity network flow).
B.5.3 RESTRICTED TIMETABLE PROBLEM (RTT)
The restricted timetable (RTT) problem introduced by Even et al. (1975), and has to do with how to schedule teachers to classes they must teach. It comprises the following
A collection of (Ti....: T. where T; S [3]. These represent teachers, each of which is available for the hours given in T,
students, each of which is available at any of the hours
An binary matrix (0, 1] jaxm Ray- 1 if teacher is required to teach class j. and 0 otherwise. Since Ray is binary, each class is taught by teacher at most once.
Each teacher is tight, i.e., [[]] Σx- Ray. That is, every teacher must teach whenever they are available.
Suppose there are exactly 3 hours a day. The problem is to determine if there exists a meeting function
f: [m] x [m] [3] {0, 1}.
where our goal is to have fii = if and only if teacher teaches class j at the h-th hour. We require the following conditions in our meeting function:
j,h, 11 = h T. This implies that teachers are only teaching in the hours they are available.
2. Ene[a] hh. = Ra for all e [m],j e [m]] This ensures that every class gets the teaching they are required, as specified by R.
3. Eve[n] [i,jj h) s 1 for all j e [m]] and h e [3]. This ensures no class is taught by more than one teachers once.
4. Σje(m] [(i, h) S 1 for all i E [m]] and h E [3]. This ensures no teacher is teaching more than one class simultaneously.
Even et al. (1975) showed that RTT is NP-hard via a clever reduction from 3-SAT, Our strategy is to reduce RTT to our problem.
Viewing RTT through the lens of bipartite edge coloring RTT can be visualized as variant of constrained edge coloring in bipartite graphs (Figure 4). The teachers and classes are the two different sets of vertices, while R gives the adjacency structure. There are 3 colors available, corresponding to hours in a day. The task is to color the edges of the graph with these 3 colors such that
1. No two edges of the same color adjacent. This ensures students and classes are at most tcaching/taking one session at any given hour (condition 3 and 4)
2. Edges adjacent to teacher i only allowed colors in T,. This ensures teachers are only teaching in available hours (condition 1)
If every edge is colored while obeying the above conditions, then it follows from the tightness of teachers (in the definition of RTT) that every class is assigned their required lessons (condition 2). The decision version of the problem is to return if such coloring is possible.
20

---

Published as a conference paper at ICLR 2024
1
1
1
2
2
2
2
3
3
3
3
(a) Valid coloring
(b) Invalid: class used repeated color (blue)
(c) Invalid:
teacher (d) Invalid: teacher
used repeated color (green)
2 used invalid color (green)
Figure 4: Examples of valid and invalid colorings. Left vertices are teachers 1, 2, 3. Right vertices are classes 1, 2. The colors red, green, blue are for hours 1, 2, respectively, color of teacher vertices are the hours where the teachers available (by definition of RIT. the number of distinct colors per teacher vertex is equal its degree). The color of edge (red, green or blue) says that teacher is assigned to that class at that hour. Figure 4a shows a valid coloring (or timetabling), since (i) all edges are colored, (ii) no edge of the same colors are adjacent, and (iii) edges adjacent to teachers correspond to the vertex's 's color. Figures 4b, 4c, 4d are invalid colorings because of same-colored edges being adjacent, or teacher vertex colors differing to adjacent edges.
Time Constrained RTT (Q-RTT) A variant of RTT that will be useful is when we impose restrictions on the number of classes being taught at any each hour. We call this Q-RTT, where Q (q), 92: 93) E Z Q-RTT returns true if, in addition to the usual RTT conditions, we require the meeting function to satisfy
Σ f(i.j.h) = gh.
[c[m].e[mm]
That is, the total number of hours taught by teachers in hour h is exactly 4h- From the perspective of edge coloring, Q-RTT simply imposes an additional restriction on the total number of edges of each color, i.e., there are edges of color & for each & E [3].
Obviously, RTT can be Cook reduced to Q-RTT: since there are only 3 hours and total g [i{[n] Ra total lessons to be taught, there
most O(g²) ways of splitting the required number of lessons up amongst the 3 hours. Thus, we can solve RTT by making at most Ogg²) calls to Q-RTT. This is polynomial in the size of RTT, and we conclude Q-RTT is NP-hard.
B.5.4 REDUCTION OF Q-RTT TO OUR PROBLEM
We will reduce Q-RTT normalizing constant given by the number of edges in a bipartite graph. One should think of a as an indicator of the boolean TRUE and 0 as FALSE. We use the following construction
problem. Let 1/ (E R.j 3m). where 1/a should be seen
1. Let X = [n]] U Z, where Z [2,, Zz, Za). From bipartite graph interpretation, these form one set of vertices that we will match to classes. Z₁ Zz. Zy are "holding rooms", one for each of the hours. Holding rooms
like teachers whose classes can be assigned in order to pass the time. They will not fulfill any constraints on R. but they can accommodate multiple classes once. We will explain the importance of these holding rooms later.
2. Let X2 [m.] These form the other set of vertices, one for each class.
3. Let [3=[3]] {0}. 1, 2, and are the distinct hours, corresponding to edge colors. is a special "null" color which will only be used when coloring edges adjacent to the holding rooms.
4. Let p(i.j.) a- Rúj and p(i.j) = a for all 2, j [m]] Essentially, there is an edge between a teacher and class if R dictates it. There are also always edges from every holding room to each class.
5. For i [n:] set p(i,', h)) if h T,, otherwise. For 7; Z, set
a a. qi
h=0 0 h=i
p(Z,,;h))
a
(m 9))
otherwise
21

---

Published as a conference paper at ICLR 2024
In order words, hour h, when class is not assigned to some teacher (which would to contribute to qı), they must be placed holding room Zn.
6. Let j. h) = a for E [3]. and p(: j..h a. Evenn] Ra.j- The former constraint means that for each of the 3 hours, the class must be taking some lesson with teacher OR in the holding room. The second constraint assigns the special "null" value to the holding rooms which not used by that class.
3
3
3
2
71
2
71
71
Z2
Zz
Z2
Zz
73
7.
7
Z
Valid coloring
(b) Invalid: class used repeated color (blue)
Invalid:
teacher
(d) Invalid: teacher
used repeated color (green)
2 used invalid color (green)
Figure 5: Examples of valid and invalid colorings when holding rooms included. For simplicity, we illustrate all constraints except those on Q. Left vertices are teachers 2, 3 and holding rooms Z1, 72, Zg. Right vertices are classes 1, The colors red, green, blue are for hours 1, 2, respectively, color teacher vertices are the hours where the teachers are available (by definition of RTT, the number of distinct colors per teacher vertex is equal to its degree). Border color of holding room vertices the hour that the holding room is available. The color of edge (red, green blue) says that teacher (or holding room) is assigned to that class at that hour. Gray edges are the "null" color, meaning that that waiting room coloring (or timetabling), since all edges are colored, no edge of the same colors are adjacent (other than the gray ones), and edges adjacent to teachers correspond to the vertex's 's color. Figures 5b, 5c, 5d are invalid colorings because of non-gray edges being adjacent, teacher vertices being adjacent to colors different from itself.
used by that class. Figure 5a shows valid
solution to our construction with conditional entropy implies a valid solution to Q-RTT Suppose that our construction returns a distribution such that every entry r(11, y) is either a or O. We claim that the meeting function fi.jjh) ri,, a and 0 otherwise solves Q-RTT.
Teachers only teaching in the hours they available, because of our marginal constraint on hh).
Every class gets the teaching they need. This follows from the fact that teachers are tight and the marginal constraint p(i,;, h), which forces teachers be teaching whenever they can. The students are getting the lessons from the right teachers because of the marginal constraint on j.-, since teachers who are not supposed to teach a class have those marginal values set to 0.
No class is taught by more than one teacher at once. This follows from marginal constraint h() For each of the hours, class is with either single teacher or the holding room.
No teacher is teaching more than one class simultancously. This holds again from our marginal constraint p(i,,, h).
Lastly, the total number of lessons (not in holding rooms) held in each hour is ga required by Q-RTT. To see why, we consider each color (hour). Each color (excluding the null color) is used exactly times by virtue of p(.j,)) h). Some of these in holding rooms, other
22

---

Published as a conference paper at ICLR 2024
with teachers. The former (over all classes) is given by m 2h because of our constraint p(i,,h), h), which means that exactly qı lessons in hour h as required.
A valid solution to Q-RTT implies a solution to our construction with O conditional entropy Given a solution Q-RTT, we recover a candidate solution our construction in natural way. If teacher i is teaching class j in hour h. then color edge ij with color h, i.e., r(i, /) o and rii,j.h' O if h' h. Since RTT each teacher and class can be assigned one lesson per hour at most, there will be no clashes with this assignment. For all other i E [3].]= E [m]] where R,g = O, we assign r(.. j. O. Now, we will also need assign students to holding rooms. For E [3], we set r( jZ h))
if class j was not assigned to any teacher in hour h. If class was assigned some teacher in hour h, then r(Zn. 0) = a, i.c. we give the special null color. All other entries are given a value of O. We can verify
r is a valid probability distribution. The nonnegativity of follows from the fact that > 0 strietly. We need check that r sums to 1. We break this down into two cases based on whether the first argument of is some Zn or i. In Case 1, we have
Σ
r(i,jh)=
Σ
r(,j,h)
Σ Ray.
[e[m].5e[m]]
where the first line follows from the fact that we never color a teacher-class edge with the null color, and the second line is because every class gets its teaching requirements satisfied. In Case 2, we know that by definition every class is matched to every holding room and assigned either the null color that room's 's color, hence
Σ ie 21 22 Za) he (3]u(O),.mm
ri. =3m
Summing them up, we have a- (3m Eieml]jmm R.,) 1 (by our definition of a.
This distribution has only entries in or O. This follows by definition.
This distribution has minimum conditional entropy. For fixed i, r(i.j.)) is either or 0. That is, Y is deterministic given X1,Xp, hence H (X|, =.
All marginal constraints in our construction are obeyed. We check them in turn.
Marginal constraint rii.j) pii.j). When E [3]: (i) when R.j exactly one time h is assigned to teacher i and class j. hence r(i,)) =a p(i,j) j) as required, (ii)when Ri 0 specified. Now when (Z1, Z2, Zz), J. have r(),j.)) a p(i,)) since every holding room is either assigned it's color to a class, or assigned the special null color.
Marginal constraint r(i,h) p(i, When i [3]. this follows directly from tightness. Similarly, when i E (21, Z2, Z3) ), we have by definition of Q-RTT the assignments to holding rooms equal m Qh for hour h, and consequently, qa null colors adjacent to 2, as required.
Marginal constraint r(j h(j.). For every e [3]. the class is assigned either to teacher holding room, this is equal null color, this is used exactly Σie[n] R,j times (since these were the number hours that were not assigned to teachers), as required, making its marginal Σie[n] Riy and r(j,h) h)) a Exe[m] Ray required.
required. For h 0, i.e., the
Thus, RTT returns TRUE, our construction will also return a solution with entries in {0, a), and vice versa.
Corollary The decision problem of whether there exists a distribution in r Ap12.12 such that H(Y]X,, is NP-complete. This follows because the problem is in NP since checking if Y is deterministic (i.c., x 0) can be done in polynomial time, while NP-hardness follows from the same argument above.
23

---

Published as a conference paper at ICLR 2024
B.6 UPPER BOUND ON SYNERGY (THEOREM 4)
We begin by restating Theorem 4 from the main text:
Theorem 8. (Upper-bourd synergy, same as Theorem 4).
S (x) HY()) - min H,, Y) re2p12.v
max
[2): = (46)
geáp1.2
where 4pi. {r A r((1, = ( ( =p(u))
Proof. Recall that this upper bound boils down to finding max 4 (2X1,, We have
max X2) reB21,2,1
max
(X,, HY) H Y),
(47)
re3p1.2.12
(x1, + Hp(Y) min H,
(48)
reAp1,211
HH((11)X2) H,(Y) min H, (X,, Y)
(49)
rEAPI2.
where the first two lines are by definition. The last line follows since App.. superset of r A 21,2122 which implies that minimizing over it would yield a no larger objective.
In practice, we use the slightly tighter bound which maximizes over all the pairwise marginals,
minnreAp2..) P12.3
H, X2,
max 12A.1.212
Hx H,(Y) - max
minreApp.cs P1.02
H2 X2, Y)
(50)
min
H (X1, V2
Estimating using min-entropy couplings We only show how to compute min H,,X, X2, Y). since the other variants can be computed in the same manner X as a single variable, we recover the
via symmetry. We recognize that by treating (X,, classic min-entropy coupling over X and Y, which is still NP-hard but admits good approxima- tions (Cicalese & Vaccaro, 2002; Cicalese et al., 2017; Kocaoglu et al., 2017; Rossi, 2019; Compton, 2022; Compton al., 2023).
There are many methods to estimate such a coupling, for example Kocaoglu et al. (2017) give a greedy algorithm running in lincar-lggarithmic time, which was further proven by Rossi (2019); Compton (2022) to be 1-bit approximation of the minimum coupling Another line of work was by (Cicalese al., 2017), 1),, which constructs an appropriate coupling and shows that is optimal to 1-bit to lower bound (p(#1, A p(v)), where ^ is the preatest-lower-beee bound operator, which they showed in (Cicalese Vaccaro, 2002) can be computed in lngarrlogarithmic time. We very briefly describe this method; more details may be found in (Cicalese et al., 2017; Cicalese & Vaccaro, 2002) directly.
Remark A very recent paper by Compton al. (2023) show that one can get an approximation tighter than 1-bit. We leave the incorporation of these more advanced methods as future work.
Without loss of generality, suppose that X' and y are ordered and indexed such that p(x) and p(y) are sorted in non- non-increasing order of the marginal constraints, i.e., p(X = 2z) 2 p(X Iz)) for all i J. j. We also assume WLOG that the supports of X and Y of the same size n, if they are not, then pad the smaller one with dummy values and introduce marginals that constrain these values to never occur (and set n accordingly if needed). For simplicity, we will just refer to p3 and lj for the distributions of p(X- z.) and p(Y = yj) respectively.
Given 2 distributions p, Q we say that p is majorized by q, written as p 2 q if and only
for all l...n
(51)
"This special case when there are 2 modalities. For more modalities, the hounds will depend on the sizes and number of signals.
24

---

Published as a conference paper at ICLR 2024
best fit: y= 1.095x
best fit: y=1.09x y=x
best fit: y=x-0.2 y=x
Actual Synergy
y=x
Actual Synergy
Actual Synergy
12
xir Lower Bound 5
64 Upper Bound on
66
Lower Bound
(a) SR
(b) St.
(c) S v
Figure 6: Plots of actual synergy against our estimated (a) lower bound on synergy Sg. (b) lower bound on synergy Sq. (c) upper bound S. The bounds closely track true synergy, which we show three best fit that almost exactly track true synergy: 1.095ar = 1.098z, and y x 0.2.
As Cicalese & Vaccaro (2002) point out, there is strong link between majorization and Schur-convex functions; in particular, if p q. then have H()) z (q). Indeed, if we treat z a partial order and consider the set
pa p p= (pI.....PP.) : [0,1],
1J. Σp: = 1, Piel
as the set of finite (ordered) distributions with support size with non-increasing probabilities, then we obtain a lattice with a unique greatest lower bound (^) and least upper bound (v). Then, Cicalese & Vaccaro (2002) show that that p q be computed recursively p q a(p.9) (a1....; an) where
a;
Pj.
9i
Σ
aj-i
J-1
It was shown by Cicalese et al. (2017) that any coupling satisfying the marginal constraints given by p and g, i.e.,
M C(p.9) M maj Em,s Pt.
m.j m,, Pj
has entropy (MM) 2 Hp g). In particular, this includes the min-entropy one. Since only need the optimal value of such a coupling and not the actual coupling per-se, we can use plug the value of ((p q) into the minimization term (49), which yields upper bound for max Xx];; which would form an upper bound on S itself.
C EXPERIMENTAL DETAILS
C.I VERIFYING LOWER AND UPPER BOUNDS
Synthetically generated datasets: To test our derived bounds on synthetic data, We randomly sampled 100, 000 distributions of (X,, X2. Y} to calculate their bounds and compare with their actual synergy values. We set X1, X2, and Y as random binary values, so each distribution can be represented as size vector of randomly sampled entries that sum up
We calculated the lower bound via redundancy, lower bound via disagreement, and upper bound of all distributions and plotted them with actual synergy value (Figure 6). We define a distribution to be on the boundary if its lower/upper bound is within 10% difference from its actual synergy value. We conducted the least mean-square-e error fitting on these distributions close to the boundary. We plot actual synergy against Sg in Figure 6 (left), and find that it again tracks lower bound of synergy. In fact, we can do better and fit linear trend 1.095xr on the distributions along the margin (RMSE- 0.0013).
Results:
We also plot actual synergy against computed Sy in Figure 6 (middle). As expected, the lower bound closely tracks actual synergy. Similarly, we can again fit linear model on the points along the boundary, obtaining 1.098z with a RMSE of 0.0075 (see this line in Figure 6 (middle)).
Finally, we plot actual synergy against estimated in Figure (right). Again, we find that the upper bound consistently tracks the highest attainable synergy can fit single constant y 0.2 to
25

---

Published as a conference paper at ICLR 2024
obtain RMSE of 0.0022 (see this line in Figure 6 (right)). This implies that our bound enables both accurate comparative analysis of relative synergy across different datasets, and precise estimation of absolute synergy.
Real-world datasets: We also use the large collection of real-world datasets in MultiBench (Liang et al., 2021): (1) MOSI: video-based sentiment analysis (Zadeh et al., 2016), (2) MOSEI: video- based sentiment and emotion analysis (Zadeh et al., 2018), (3) MUSTARD: video-based sarcasm detection (Castro et al., 2019), (5) MIMIC: mortality and disease prediction from tabular patient data and medical sensors (Johnson et al., 2016). and (6) ENRICO: classification of mobile user interfaces and screenshots (Leiva et al., 2020). While the previous bitwise datasets with small and discrete support yield exact lower and upper bounds, this new setting with nigh-dimensional continuous modalities requires the approximation of disagreement and anffommtiion-thooret quantities: we train unimodal neural network classifiers fo (u((a) ) and fo (1/12) to estimate disagreement, and we cluster representations of X; to approximate the continuous modalities by discrete distributions with finite support to compute lower and upper bounds.
Implementation details: We first apply PCA to reduce the dimension of multimodal data. For the test split, we use unsupervised clustering to generate 20 clusters. We obtain a clustered version of the original dataset D = 12,, y)} as Pelnster {(c), C2. y) where o; é (I....,2)) is the ID of the cluster that z; belongs to. In our experiments, where y is typically a classification task, we set the unimodal classifiers f1 (( and f2 p(y|1z) as the Bayes optimal classifiers for multiclass classification tasks.
For classification, y is the set of k-dimensional 1-hot vectors. Given two logits Φ1 822 obtained from 22 respectively, define d(01, (2) = (0) - -92)² We have that ca = I, and é = - )12 O and 62 (L(f2) L(fī) )12 = 0 for well-trained neural network unimodal classifiers f1 and fa for Theorem 2. For datasets with modalities, we perform the experiments separately for each of the 3 modality pairs, before taking
average over the 3 modality pairs. Extending the definitions of redundancy, uniqueness, and synergy, as well as our derived bounds on synergy for 3 or more modalities is an important open question for future work.
C.2 RELATIONSHIPS BETWEEN AGREEMENT, DIS AGREEMENT, AND INTERACTIONS
1. The relationship between synergy and redundancy: We give some example distributions to analyze when the lower bound based on redundancy Sx is high or low. The bound is high for distributions where X1 and X2 are independent, but Y = sets X1 # X2 to increase their dependence (i.c.. AGREEMENT XOR distribution in Table 2b). Since X1 and X2 are independent but become dependent given Y. I(X;; X2;; Y) is negative, and the bound is tight SR 1 1 S. Visual Question Answering 2.0 (Goyal et al., 2017) falls under this category, with S 4.92, R = 0.79, where the image and question are independent (some questions like 'what the color of the object' or 'how many people are there' can be asked for many images), but the answer connects the two modalities, resulting in dependence given the label. As expected, the estimated lower bound for synergy: SR = 4.03 z 4.92 = S.
Conversely, the bound is low for Table 2d with the probability mass distributed uniformly only when y = 11 22 and 0 elsewhere. As result, X, is always equal to X2 (perfeet dependence), and yet Y perfectly explains away the dependence between X1 and X₂ so T(x;; X2|Y)) O: SR = O s 0 = S. Note that this is an example of perfect redundancy and zero synergy- for example with synergy, refer back to DISAGREEMENT XOR in Table 2a - due to disagreement there is non-zero 1 (K1;X2) but the label explains some of the relationships between X1 and X2 so I x(X1; < X2):: SR = -0.3 N 1 = S. A real-world example is multimodal sentiment analysis from text, video, and audio of monologue videos on MOSEI, R = 0.26 and S 0.04, and as expected the lower bound is small SR = 0.01 < 0.04= =
2. The relationship between synergy and uniqueness: To give an intuition of the relationship between disagreement, uniqueness, and synergy, we use one illustrative example shown in Table 2a, which we call DISAGREEMENT XOR. We observe that there is maximum disagreement between marginals p(y(z1) ) and p(y)a2): the likelihood for y is high when is the same bit as x,, but reversed for Z2. Given both 24 and Zy: y seems to take a 'disagreement XOR of the individual marginals, ie. p(y]21, p P(ylæ;) XOR p(yl=2), which indicates synergy (note that an exact XOR would imply perfect agreement and high synergy). The actual disagreement is 0.15, synergy is 0.16. and uniqueness is 0.02, indicating very strong lower bound Sy 0.13 & 0.16 S. A roal-world
26

---

Published as a conference paper at ICLR 2024
Table show the full list computed lower and upper bounds on S without labeled multimodal data and compare them to the true S assuming knowledge of the full joint distribution pe the bounds track S well on MUSTARD and MIMIC, and also show general trends on the other datasets except ENRICO where estimating synergy is difficult. V video. T text, audio modalities.
MOSEL MOSElv.A MOSEL UR-FUNNY V-T UR-FUNNY UR-FUNNY,
S
0.96
0.98
0.97
0.96
0.96
0.99
0.04
0.03
0.03
0.21
0.24
0.08
0.01
0.0
0.0
0.0
0.0
0.01
0.01
0.0
0.0
0.0
0.01
MOSIyr MOSTvA
M
MUSTARDyT MUSTARDVA
MUSTARDAT MIMIC ENRICO
0.92
0.92
0.93
0.79
0.78
0.79
0.41
2.09
S
0.31
0.28
0.14
0.49
0.31
0.51
0.02
1.02
S
0.01
0.01
0.0
0.04
0.01
0.06
0.0
0.01
0.03
0.03
0.02
0.07
0.06
0.11
-0.12
-0.55
equivalent dataset is MUSTARD for sarcasm detection from video, audio, and text (Castro et al., 2019), where the presence of sarcasm often due to a contradiction between what is expressed in language and speech, disagreement 0.12 is the highest out of all the video datasets, giving lower bound S. = 0.11 s 0.44 S.
On the contrary, the lower bound is low when all disagreement explained by uniqueness (e.g.. = 3i, Table 2c), which results in S. 0 0 S (a and U cancel each other out). A real-world equivalent is MIMIC involving mortality and disease prediction from tabular patient data and time-series medical sensors (Johnson al., 2016). Disagreement is high 0.13 due unique information U/1 0.25, so the lower bound informs us about the lack of synergy Su = -0.12 & 0.02 = S.
Finally, the lower bound loose when there is synergy without disagreement, such as AGREEMENT XOR (y= XOR 22. Table 2b) where the marginals p(v(x;) are both uniform, but there is full synergy: Sy < S. Real-world datasets that have synergy without disagreement include UR-FUNNY where there is low disagreement in predicting humor 0.03, and relatively high synergy 0818, which results in a loose lower bound S. 0.01 & 0.18 S.
On upper bounds for synergy: We also run experiments obtain estimated upper bounds on synthetic and MultiBench datasets. The quality of the upper bound shows some intriguing relationships with that of lower bounds. For distributions with perfect agreement
Upper bound
Agreement XOR
Synergy
and synergy such as 2 XOR I2 (Table 2b), $ 1 1 S really close true synergy. SR =
S is also tight, but Su = 0 s S is loose. For distributions with disagreement and synergy (Table 2a), S = 0.52 0.13 S far exceeds actual synergy, SR = -0.3 < 1== S much lower than actual synergy, but Sy = 5 0.16 = S is tight (see relationships in Figure 7).
Disagreement XOR
Lower bound
Figure 7: Comparing the qualities of the bounds when there is agree- ment, disagreement, and synergy. When there is agreement and syn- ergy, Sp is tight, S₁ is loose, and S tight. When there is disagree- ment and synergy. Sp is loose, St is tight, and is loose with re- spect to true S.
Finally, while some upper bounds (e.g., MUSTARD, MIMIC) are close to true S, some of the other examples in Table show bounds that are quite weak. This could be because (i) there indeed exists high synergy distributions that match D, and Da, but these are rare in the real world, or (ii) our approximation used in Theorem 4 is mathematically loose. We leave these for future work.
C.3 COMPARISONS WITH OTHER INTERACTION MEASURES
We are not aware of other related work in mathematically formalizing multimodal interactions, much less for cemi-supervised data. Most of our results are new theoretical and empirical insights about different multimodal interactions. Other valid definitions of multimodal interactions do exist, but are known to suffer from issues regarding over- and under-estimation, and may even negative (Griffith & Koch, 2014).
In Table 5, we compare our estimators with other previously used measures for feature interactions on the synthetic datasets. We implement 6 other definitions: (1) I-min can sometimes overestimate
27

---

Published as a conference paper at ICLR 2024
Table 5: Estimating multimodal interactions on synthetic generative model datasets. Ground truth total informa- tion is computed based on an upper bound from the best multimodal test accuracy. Our estimated interactions are consistent with ground truth interactions. We also implement 3 other definitions: I-min can sometimes overestimate synergy and uniqueness; WMS is actually synergy minus redundancy, so can be negative and when R & S are of equal magnitude WMS cancels out to be O; CI can also be negative and sometimes incorrectly
concludes highest uniqueness for S-only data.
Tass
R-only data
U1 Vo-oly data
U2-only data
S-only
data
Interaction
R
U1
U2
5
R
U1
U2
5
R
U1
U2
5
R
U1
U2
S
T-MIN
0.17
0.08
0.07
0
0.23
0
0.06
0
0.25
0.08
0.07
0.03
0.04
0.17
WMS
0
0.20
0.20
-0.11
0.25
0.02
0.05
0
0.03
0.27
0.05
0
0.14
0.15
0.07
CI
0.34
-0.09
-0.10
0.17
0
0.23
0
0.06
0 0.01
0.25
0.07
-0.02
0.13
0.14
0.08
Ours
0.16
0.05
0.16
0.05
0.17
0.05
0.07
0
0.01
0.14
Truth
0.58
0
0.56
0
0
0.54
0
0
0.56
synergy and uniqueness, (2) WMS is actually synergy minus redundancy, so can be negative and when R & S are of equal magnitude WMS cancels out to O, (3) CI can also be negative and sometimes incorrectly concludes highest uniqueness for S-only data, (4) Shapley values, (5) Integrated gradients (IG), and (6) CCA, which are based on quantifying interactions captured by a multimodal model. Our work is fundamentally different in that interactions are properties of data before training any models. Our estimated interactions are more consistent with ground truth interactions.
C.4 ROBUSTNESS TO IMPERFECT UNIMODAL CLASSIFIERS
humar
masi
humar
masi
Synergy-lood lowee bound
0.5
Symergy-lower bound
dasgres
Upper bound-synergs
Upper bound synergy
0.9
0.4
0.4
0.9
0.8
0.3-
0.8
0.7
0.2
0.2
0.1
0.1
Upper
0.6
6.5
0.0
0.0
0.5
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Noise level
Noise level
Noise level
Nolse level
Figure 8: We study the effect of noisy unimodal predictors and disagreement by perturbing the label by various noise levels (x-axis from noise = 0.0 to 0.8) and examining the change in estimated upper and lower bounds (left: y-axis is true synergy- lower bound and right: y-axis is upper bound true synergy). On 2 real-world datasets (UR-FUNNY and MOSI) find bounds quite robust label noise, giving stable trends of real synergy.
We also study the effect of imperfect unimodal predictors and therefore imperfect disagreement measurements on derived bounds, by perturbing the label by various noise levels (from no noise to very noisy) and examining the change in both the estimated upper and lower bounds. In Figure 8, we find these bounds to be quite robust to imperfect unimodal classifiers, still giving close trends of real synergy.
D ESTIMATING MULTIMODAL PERFORMANCE
Formally, we estimate performance via a combination of Feder & Merhav (1994) and Fano's in- equality (Fano, 1968) together yield tight bounds of performance as a function of total information XxXX, We restate Theorem 5 from the main text:
y]l denote the accuracy of the Bayes' optimal multimodal model fūr (i.e., Pace (fū Z Pazo (fr) for all f&r FNs). We have that
Theorem 9. Let (fcc = Ep [1 [1
(2), (21,
21(((,...))))) S Pace (fis) (2):X)
(52)
log [
where can plug in R+ U1, U2 S U and upper Pass acc (f2) bounds on optimal multimodal performance.
s R U2 +5 3 to obtain lower P Pace (fis)
Proof. We use the bound from Feder & Merhav (1994), where we define the Bayes' optimal classifier
Ji is the one where given 31, 22 outputs y such that p(Y 2/21,222 is maximized over all y e )
28

---

Published as a conference paper at ICLR 2024
Table 6: Full list of best unimodal performance (f)( best simple fusion Pacc (s ) and best complex fusion Pase (J (emmmplee) as obtained from the most recent state-of-the-ar models. We also include our estimated bounds (L (fae). ; nc (oa(())) )) on optimal multimodal performance.
MOSEI
UR-FUNNY
MOSI
1.07
1.21
1.29
Pace (xccopoppp)es)
0.88 (Hu et al., 2022)
0.77 (Hasan et al., 2021) 0.86 (Hu al., 2022)
Pace (SMssipeee
0.85 (Rahman et al., 2020)
0.76 (Hasan et al., 2021)
0.84 (Rahman et al., 2020)
Pace (f)
0.82 (Delbrouck et al., 2020) 0.74 (Hasan al., 2021) 0.83 (Yang al., 2020)
P. (frr)
0.52
0.58
0.62
MUSTARD
MIMIC
ENRICO
P Pic(ffs)
1.63
1.27
0.88
( (MMemllss)
0.79 (Hasan al., 2021)
0.92 (Liang et al., 2021) 0.51 (Liang et al., 2021)
Pace (fMsimmpp))))
0.74 (Pramanick et al., 2021) 0.92 (Liang et al., 2021) 0.49 (Liang et al., 2021)
(fa)
0.74 (Hasan al., 2021)
0.92 (Liang et al., 2021) 0.47 (Liang et al., 2021)
0.78
0.76
0.48
The probability that this classifier succeeds is maxy p = y/21, 22), which is where -How ( is the min-entropy of the random variable conditioned on X2, Over all inputs (x1, the probability of accuracy is
Pace (f Em,, [2 H_ (M (1 2 E., E 17
(53)
2 22 H, H, (M. ( (2((((
(54)
The upper bound is based on Fano's inequality (Fano, 1968). Starting with H, (Y| S H (Per) Per(log] [|] -1 1) and assuming that Y is uniform over DV, rearrange the inequality obtain
Pace (f H(V) H,(V|X,, X2) + log ([(X1,X));V)
(55)
log |
log [
Finally, we summarize estimated multimodal performance the average between estimated lower and upper bounds on performance: Pru- (fir Pacc 1/2.
Unimodal and multimodal performance: Table summarizes all final performance results for each dataset, spanning unimodal models and simple complex multimodal fusion paradigms, where each type of model is represented by the most recent state-of-the-art method found in the literature.
E SELF-SUPERVISED MULTIMODAL LEARNING VIA DISAGREEMENT
Finally, we highlight an application of our analysis towards self-supervised pre-training, which is generally performed by encouraging agreement as a pre-training signal on large-scale unlabeled data (Radford et al., 2021; Singh et al., 2022) before supervised fine-tuning (Oord et al., 2018). However, our results suggest that there are regimes where disagreement can lead to synergy that may otherwise be ignored when only training for agreement. We therefore design new family of self-supervised learning objectives that capture disagreement on unlabeled multimodal data.
E. METHOP
We build upon masked prediction that is popular self-supervised pre-training: given multimodal data of the form (x1, 2()1,22) (e.g., 21 = caption and x22 image), first mask out some words (x²) before using the remaining words (x \/1) to predict the masked words via learning well as the image z2 to predict the masked words via learning fo (2)) (Singh et al., 2022; Zellers et al., 2022). other words, maximizing agreement between fe f (a1) and
fo [21 (21))
fo fo (2)) in predicting 21;
Lagre d(
+
(56)
for a distance d such as cross cross- entropy loss for discrete word tokens. To account for disagreement, we allow predictions the masked tokens a' from two different modalities i, disagree by a slack
29

---

Published as a conference paper at ICLR 2024
variable Aij. We modify the objective such that each term only incurs a loss penalty if each distance d(z,y) is larger than À as measured by a margin distance dx y) = max(0, d(x,)) A):
Ldisagree
Leugree
Σ
(57)
Isie752
These À terms are hyperparameters, quantifying the amount of disagreement we tolerate between each pair of modalities during cross-modal masked pretraining (A = O recovers full agreement). We show this visually in Figure 9 by applying it to masked pre-training on text, video, and audio using MERLOT Reserve (Zellers et al., 2022), and also apply it to FLAVA (Singh et al., 2022) for images and text experiments (see extensions modalities and details in Appendix E).
E.2 TRAINING DETAILS
We continuously pretrain and then finetune a pretrained MERLOT Re- serve Base model on the datasets with a batch size of 8. The continuous pre- training procedure is similar to Con- trastive Span Training, with the dif- ference that we add extra loss terms that correspond to modality disagree- ment. The pretraining procedure of MERLOT Reserve minimizes sum of component losses,
Modallyy predettoo con sometimes disegree, Modiaity cnagresment losses dlew this during trainig
fe MAai
② Fe (Josse
fe MADe
pacies
MASK
Image Encoder (VIT)
Word Embed (BPE)
Audio Encoder (AST)
I picture geckos in ADVERTISEMENTS
[monotone]]
*
C= Liext Landio + Lirame
(58)
where each of the component losses a contrastive objective. Each of the objectives aims match an indepen- dent encoding of masked tokens of the corresponding modality with the out- put of a Joint Encoder, which takes as input the other modalities and, possibly, unmasked tokens of the target modality.
Tooce cotways firt of smiel
Tteef is place costhet coneer suffen with gedlo
frwncfore tigeal sonefting ofeeti
Figure 9: Masked predictions do not always agree across modalitics, as shown in this example from the Social-IQ datasct. Adding slack term enabling pre-training with modality disagreement yields strong performance improvement over baselines.
We modify the procedure by adding disagreement losses between modalities to the objective. This done by replacing the tokens of a modality with padding tokens before passing them to the Joint Encoder, and then calculating the disagreement between representations obtained when replacing different modalities. For example, Liaame uses a representation of video frames found by passing audio and text into the Joint Encoder. Excluding one of the modalities and passing the other one into the Encoder separately leads to two different representations, f, for prediction using only text and f. for prediction using only audio. The distance between the representations added to the loss. Thus, the modified component loss is
L Ldisugreement frume Litrame d d ado (Pr, f.))
(59)
where d d mao (x,y) maax(0, d(x,y) >vext, audic) ), and d( d(x,y) y) is the cosine difference:
d(x, = x-y y
(60)
xx|y>
Similarly, we modify the other component losses by removing one modality at time, and obtain the new training objective
Ldisagreement ddisagreement, text Ldisagreement, andio + ddssugreement, frame
(61)
During pretraining, we train the model for 960 steps with a learning rate of 0.0001, and no warm-up steps, and use the defaults for other hyperparameters. For every dataset, we fix two of {Aeett audio, Avision, audio: Atext, vision) to be oo and change the third one, which characterizes the most meaningful disagreement. This allows us reduce the number of masked modalities required from 3 to 2 and thus reduce the memory overhead of the method. For SOCIAL-IQ, we set Asext vision set Aust vision to be 0.5. For MUSTARD, set Xvision, audio to be O. All training is done on TPU v2-8 accelerators, with continuous pretraining taking 30 minutes and using up 9GB of memory.
to be 0. For UR-FUNNY,
30

---

Published as a conference paper at ICLR 2024
Table 7: Allowing for disagreement during self-supervised masked pre-training yields performance improvements on these datasets. Over 10 runs, improvements that are statistically significant are shown in bold (p < 0.05).
SOCIAL-IQ UR-FUNNY
MUSTARD
CARTOON
TVQA
FLAVA, MERLOT Reserve
70.6 0.6
80.0 a 0.7
38.6 0.8 0.6
82.0
+ disagreement
71.1 + 0.5
80.7 a 0.5
78.1
39.3 a 0.5
83.0
0.72
Text and Audia
0.77
and vision
ACcerey
0.764
0.762
0.30
0.136
0.00
0.25
0.50
0.75
1.50
1.25
1.50
1.75
2.00
Figure 10: Impact of modality disagreement model performance. Lower t mcans train with higher disagreement between modalities: we find that disagreement between text and vision, as well audio and vision, are more helpful during self-supervised masking with performance improvements, whereas disagreement between text and audio is less suitable and can even hurt performance.
E.3 SETUP
We choose four settings with natural disagreement: (1) UR-FUNNY: humor detection from 16, 000 TED talk videos (Hasan et al., 2019), (2) MUSTARD: 690 videos for sarcasm detection from TV shows (Castro et al., 2019), (3) SOCIAL IQ: 1, 250 multi-party videos testing social intelligence knowledge (Zadeh al., 2019), (4) CARTOON: matching 704 cartoon images and captions (Hessel et al., 2022), and (5) TVQA: a large-scale video QA dataset based on 6 popular TV shows (Friends, The Big Bang Theory, How I Met Your Mother, House M.D., Grey's Anatomy, Castle) with 122, 000 QA pairs from 21, 800 video clips (Lei et al., 2018).
E.4 RESULTS
From Table 7, allowing for disagreement yields improvements on these datasets, with those on SOCIAL IQ. UR-FUNNY, MUSTARD being statistically significant (p-value < 0.05 over 10 runs). By analyzing the value of resulting in the best validation performance through hyperparameter search, we can analyze when disagreement helps for which datasets, datapoints, and modalities. On a dataset level, we find that disagreement helps for video/audio and video/text, improving accuracy by up to 0.6% but hurts for text/audio, decreasing the accuracy by up to 1%. This is in line with intuition, where spoken text is transcribed directly from audio for these monologue and dialog videos, but video can have vastly different information. In addition, we find more disagreement between text/audio for SOCIAL IQ, which we believe because it comes from natural videos while the others scripted TV shows with more agreement between speakers and transcripts. Finally, we also scaled to TVQA by incorporating the disagreement objective on top of MERLOT Reserve (Zellers et al., 2022). Unfortunately, running multiple times was not possible due the large size of the dataset, but our preliminary results show that adding disagreement improves performance from 82% to 83% as compared to the original agreement- Contrastive Span pretraining (Zellers et al., 2022). We will continue to investigate disagreement-baree based SSL in large-scale settings in future work.
E.5 DATASET LEVEL ANALYSIS
We now study the impact of pairwise modality disagreement the entire dataset model perfor- mance by fixing two modalities M1, M2 and a threshold t, and setting the modality pair-specific disagreement slack terms À according to the rule
Ne, = { i, a= M1, M₂
+00, else
31

---

Published as a conference paper at ICLR 2024
TRADEMARK PROTECTION
L
they have tradenark protection copyright protection all they have really is trademark protection
High disagreement
Low disagreement
'e all really really looking forward to hearing your story
High disagreement
Figure 11: Examples of disagreement due uniqueness (up) and synergy (down)
This allows us isolate dxw, My while ensuring that the other disagreement loss terms are 0. By decreasing f, we encourage higher disagreement between the target modalities. In Figure 10, we plot the relationship between model accuracy and t for the MUStARD dataset to visualize how pairwise disagreement between modalities impacts model performance.
E.6 DATAPOINT LEVEL ANALYSIS
Finally, we visualize the individual datapoints where modeling disagreement helps in model predic- tions. After continuously pretraining the model, fix a pair of modalities (text and video) and find the disagreement these modalities for each datapoint. We show examples of disagreement due to uniqueness and synergy in Figure 11. The first example is from UR-FUNNY dataset: the moments when the camera jumps from the speaker to their presentation slides are followed by an increase in agreement since the video aligns better with the speech. the second example MUSTARD, observe disagreement between vision and text when the speaker's face expresses the sarcastic nature of phrase. This changes the meaning of the phrase, which cannot be inferred from text only, and leads to synergy.
32


Reviews:
[{"review_id": "AEOqUkWiqw", "reviewer": "Reviewer_dPyL", "summary": "This paper explores the challenge of understanding how different modalities combine to provide new task-relevant information in a semi-supervised multimodal learning setting. The authors derive lower and upper bounds to quantify the amount of multimodal interactions, with the lower bounds based on shared information and disagreement between unimodal classifiers, and the upper bound derived through connections to min-entropy couplings. The authors also propose a practical algorithm to estimate the lower bounds in practice. The theoretical results in this paper could be used to guide data collection or select appropriate multimodal models for a specific task. Overall, this paper provides a valuable contribution to the field of multimodal learning by providing a framework for quantifying interactions between modalities.", "questions": "1. In section 3, what is D_M? Did you consider number of modalities over 2?\n\n2. In Table 3, can you give more details of multimodal or unimodal, and what are the evaluation metrics to define the best models?\n\n3. In section 4.2 RQ1, what performance to estimate? I think authors could have provided more explanations or examples to make the paper more accessible to a wider audience.\n\n4. The authors mentioned observing only labeled unimodal data and some unlabeled multimodal data, did authors conduct experiments on these kind of data or any ablation studies on incompleteness of unimodal data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a novel contribution to the field of multimodal learning by deriving lower and upper bounds to quantify the amount of multimodal interactions in a semi-supervised setting. This provides a framework for understanding how different modalities combine to provide new task-relevant information.\n\n2. The paper is theoretically rigorous, with the authors providing a detailed information-theoretic analysis of the problem. The authors also propose a practical algorithm to estimate the lower bounds in practice. The authors show that synergy bounds can be used to predict the performance of multimodal models on held-out data, and to identify modalities that are most important for synergy. This information can be used to guide data collection efforts and improve the performance of multimodal models.\n\n3. The theoretical results in this paper could be used to guide data collection or select appropriate multimodal models for a specific task. This has important practical implications for the development of multimodal learning systems in a variety of domains.", "weaknesses": "1. The synergy bounds are only approximate. The authors acknowledge that the synergy bounds are not tight, and that they may underestimate or overestimate the true amount of synergy in a dataset. This is a limitation of any information-theoretic approach to synergy quantification. While the paper provides a theoretical framework for quantifying interactions between modalities, there is limited empirical evaluation of the proposed approach. The authors only provide a proof-of-concept experiment on a small dataset, which may limit the generalizability of the results.\n\n2. The synergy bounds are sensitive to the choice of auxiliary distributions. The authors use a set of auxiliary distributions to compute their lower bound on synergy. The choice of these auxiliary distributions can affect the tightness of the bound. The authors do not provide any guidance on how to choose the auxiliary distributions, which leaves this as a practical challenge for users of the proposed method.\n\n3. The synergy bounds are not applicable to all types of synergy. The authors focus on quantifying synergy that arises from the disagreement between unimodal predictors. However, there are other types of synergy, such as synergy that arises from the complementary strengths of different modalities. The proposed synergy bounds are not applicable to these other types of synergy.\n\n4. The paper does not provide any theoretical guarantees on the performance of multimodal models trained using synergy bounds. The authors show that synergy bounds can be used to predict the performance of multimodal models on held-out data. However, they do not provide any theoretical guarantees on the accuracy of these predictions. This is a limitation of the paper, as it would be useful to know how accurately synergy bounds can predict the performance of multimodal models in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DYR1LfUEAb", "reviewer": "Reviewer_o1cs", "summary": "This paper provides two lower bounds and one upper bound to quantify the interactions between modalities for the proposed multimodal semi-supervised setting. One lower bound is based on mutual information between modalities and the other is based on disagreement between separately trained unimodal classifiers. The upper bound is approximated by min-entropy coupling methods. The performances of the bounds to track true interactions on synthetic and real-world datasets are evaluated by experimental results. Examples are shown to use the bounds to guide modality or model selection in multimodal learning.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The performances of the lower bounds are tight, which demonstrates the effectiveness.\n2. The proposed bounds can be used for modality or model selection, which are helpful for multimodal learning.", "weaknesses": "1. The performance of the proposed upper bound is not very ideal, more explanations can be provided.\n2. No related work is compared in the experimental results. The superiority of this work is not demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ofyk7mqVm3", "reviewer": "Reviewer_rHJs", "summary": "This study develop a system of information-theoretic definitions and theories that quantify multimodal interaction in a semi-supervised setting. The semi-supervised setting being investigated contains only labelled unimodal datasets and unlabelled co-occurring multimodal data. The authors derive 2 lower bounds and 1 upper bound for Synergy information - emerged information when all modalities incorporate. Additionally, the research demonstrates how these theoretical results can be applied to estimate multimodal model performance, guide data collection, and select appropriate multimodal models for various tasks.", "questions": "How to determine the closeness of the derived boundary?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Contribution:\n\n- The definitions are set out in accordance with real scenarios and datasets (e.g. MIMIC, MUSTARD,…) and in accordance with previous literatures, which increase the credit and sensibility of the definitions.\n- The line of theorem following and based on those definitions are set out reasonable and come with rigorous proof in Appendix.\n- By extracting 4 information factors given pairwise marginal distributions, the authors derive the bound of accuracy for optimal multimodal model. This can be used to analyse whether to collect more data and choose a different fusion approach\n\nPrepresentation: \n\n- The authors provides trackable notation system and use those notations throughout the manuscript, which increase its consistency and coherence.\n- Clear instructions via reference system and details in Appendix, make it more readable and trackable (e.g. we refer reader to …).", "weaknesses": "- Upon deriving lower bound using uniqueness, the reliance on optimal unimodal classifiers seem not practical in most scenarios.\n- Upon deriving the upper bound, the relaxation using min-entropy coupling problem might not produce a tight upper boundary.\n- While the empirical result support the correct boundary of *Synergy*, there is no investigation on how the bound is closed to *Synergy*, or what is the gap can be determined as close enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)