SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Estimating PID on synthetic generative model datasets. Both CVX and BATCH measures agree with each other on relative values and are consistent with ground truth interactions.
Task	\( \mathcal{D}_R \)	\( \mathcal{D}_{U_1} \)	\( \mathcal{D}_{U_2} \)	\( \mathcal{D}_S \)	\( y = f(z_1^*,z_2^*,z_c^*) \)
PID	\( R \)	\( U_1 \)	\( U_2 \)	\( S \)	\( R \)	\( U_1 \)	\( U_2 \)	\( S \)	\( R \)	\( U_1 \)	\( U_2 \)	\( S \)	\( R \)	\( U_1 \)	\( U_2 \)	\( S \)	\( R \)	\( U_1 \)	\( U_2 \)	\( S \)
CVX	0.16	0	0	0.05	0	0.16	0	0.05	0	0	0.17	0.05	0.07	0	0.01	0.14	0.04	0.01	0	0.07
BATCH	0.29	0.02	0.02	0	0	0.30	0	0	0	0	0.30	0	0.11	0.02	0.02	0.15	0.06	0.01	0.01	0.06
Truth	0.58	0	0	0	0	0.56	0	0	0	0	0.54	0	0	0	0.56	0.13	0	0	0.27
Task	y = f(z1, z2*, zc*)	y = f(z1, z2, zc*)	y = f(z1*, z2*,zc)	y = f(z*,zc*)	y = f(z2*,zc)
PID	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S
CVX	0.04	0.06	0	0.07	0.07	0	0	0.12	0.1	0	0.01	0.07	0.03	0	0.04	0.05
BATCH	0.04	0.09	0	0.06	0.11	0.02	0.02	0.10	0.11	0.02	0.02	0.05	0.07	0	0.06	0
Truth	0	0.25	0	0.25	0.18	0	0	0.36	0.22	0	0	0.22	0.21	0	0.21	0
the angle goes to 0 or  $\pi /2$  — if centroids share a coordinate, then observing that coordinate yields no information about  $y$ . Conversely, synergy and redundancy peak at  $\pi /4$ . Interestingly, synergy seems to be independent of  $\| \mu \| _2$ . For Polar, redundancy is 0. Furthermore,  $\theta$  contains no unique information, since  $\theta$  shows nothing about  $y$  unless we know  $r$  (in particular, its sign). When the angle goes to  $\pi /2$ , almost all information is unique in  $r$ . The distinctions between Cartesian and Polar highlight how different representations of data can exhibit wildly different PID values, even if total information is the same. More thorough results and visualizations of  $q^{*}$  are in Appendix C.2.
Synthetic generative model: We begin with a set of latent vectors  $z_{1}, z_{2}, z_{c} \sim \mathcal{N}(0_{d}, \Sigma_{d}^{2})$ ,  $d = 50$  representing information unique to  $X_{1}, X_{2}$  and common to both respectively.  $[z_{1}, z_{c}]$  is transformed into high-dimensional  $x_{1}$  using a fixed transformation  $T_{1}$  and likewise  $[z_{2}, z_{c}]$  to  $x_{2}$  via  $T_{2}$ . The label  $y$  is generated as a function of (1) only  $z_{c}$ , in which case we expect complete redundancy, (2) only  $z_{1}$  or  $z_{2}$  which represents complete uniqueness, (3) a combination of  $z_{1}$  and  $z_{2}$  representing complete synergy, or (4) arbitrary ratios of each of the above with  $z_{i}^{*}$  representing half of the dimensions from  $z_{i}$  and therefore half of each interaction. In total, Table 2 shows the 10 synthetic datasets we generated: 4 specialized datasets  $\mathcal{D}_{I}$ ,  $I \in \{R, U_{1}, U_{2}, S\}$  where  $y$  only depends on one interaction, and 6 mixed datasets with varying interaction ratios. We also report the ground-truth interactions as defined by the label-generating process and the total capturable information using the bound in Feder and Merhav [29], which relates the accuracy of the best model on these datasets with the mutual information between the inputs to the label. Since the test accuracies for Table 2 datasets range from 67-75%, this corresponds to total MI of 0.42 - 0.59 bits.
Results: From Table 2, both CVX and BatchH agree in relative PID values, correctly assigning the predominant interaction type and interactions with minimal presence consistent with the ground-truth based on data generation. For example,  $\mathcal{D}_R$  has the highest  $R$  value, and when the ratio of  $z_{1}$  increases,  $U_{1}$  increases from 0.01 on  $y = f(z_1^*, z_2^*, z_c^*)$  to 0.06 on  $y = f(z_{1},z_{2}^{*},z_{c}^{*})$ . We also note some interesting observations due to the random noise in label generation, such as the non-zero synergy measure of datasets such as  $\mathcal{D}_R$ ,  $\mathcal{D}_{U_1}$ ,  $\mathcal{D}_{U_2}$  whose labels do not depend on synergy.
4.2 Quantifying Real-world Multimodal Benchmarks
We now apply these estimators to quantify the interactions in real-world multimodal datasets.
Real-world multimodal data setup: We use a large collection of real-world datasets in Multi-Bench [60] which test multimodal fusion of different input signals (including images, video, audio, text, time-series, sets, and tables) for different tasks (predicting humor, sentiment, emotions, mortality rate, ICD-9 codes, image-captions, human activities, digits, and design interfaces). We also include experiments on question-answering (Visual Question Answering 2.0 [5, 39] and CLEVR [53]) which test grounding of language into the visual domain. For the 4 datasets (top row of Table 3) involving images and text where modality features are available and readily clustered, we apply the CVX estimator on top of discrete clusters. For the remaining 4 datasets (bottom row of Table 3) with video, audio, and medical time-series modalities, clustering is not easy, so we use the end-to-end BATCH estimator (see Appendix C.4 for full dataset and estimator details).
Human judgment of interactions: Real-world multimodal datasets do not have reference PID values, and exact PID computation is impossible due to continuous data. We therefore use human judgment as a reference. We design a new annotation scheme where we show both modalities and the label and ask each annotator to annotate the degree of redundancy, uniqueness, and synergy on a scale of 0-5, alongside their confidence in their answers on a scale of 0-5. We show a sample
Table 3: Estimating PID on real-world MultiBench [60] datasets. Many of the estimated interactions align well with human judgement as well as unimodal performance.
Task	AV-MNIST	ENRICO	VQA 2.0	CLEVR
PID	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S
CVX	0.10	0.97	0.03	0.08	0.73	0.38	0.53	0.34	0.79	0.87	0	4.92	0.55	0.48	0	5.16
Human	0.57	0.61	0	0	-	-	-	-	0	0	0	6.58	0	0	0	6.19
Task	MOSEI	UR-FUNNY	MUSTARD	MIMIC
PID	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S
BATCH	0.26	0.49	0.03	0.04	0.03	0.04	0.01	0.08	0.14	0.01	0.01	0.30	0.05	0.17	0	0.01
Human	0.32	0.20	0.15	0.15	0.04	0.05	0.03	0.04	0.13	0.17	0.04	0.16	-	-	-	-
user interface and annotation procedures in Appendix C.5. We give 50 datapoints from each dataset (except MIMIC and ENRICO which require specialized knowledge) to 3 annotators each.
Results on multimodal fusion: From Table 3, we find that different datasets do require different interactions. Some interesting observations: (1) all pairs of modalities on MUSTARD sarcasm detection show high synergy values, which aligns with intuition since sarcasm is often due to a contradiction between what is expressed in language and speech, (2) uniqueness values are strongly correlated with unimodal performance (e.g., modality 1 in AV-MNIST and MIMIC), (3) datasets with high synergy do indeed benefit from interaction modeling as also seen in prior work (e.g., MUSTARD, UR-FUNNY) [17, 43], and (4) conversely datasets with low synergy are those where unimodal performance is relatively strong (e.g., MIMIC) [60].
Results on QA: We observe very high synergy values as shown in Table 3 consistent with prior work studying how these datasets were balanced (e.g., VQA 2.0 having different images for the same question such that the answer can only be obtained through synergy) [39] and that models trained on these datasets require non-additive interactions [46]. CLEVR has a higher proportion of synergy than VQA 2.0 (83% versus 75%): indeed, CLEVR is a more balanced dataset where the answer strictly depends on both the question and image with a lower likelihood of unimodal biases.
Comparisons with human judgment: For human judgment, we cannot ask humans to give a score in bits, so it is on a completely different scale (0-5 scale). To put them on the same scale, we normalize the human ratings such that the sum of human interactions is equal to the sum of PID estimates. The resulting comparisons are in Table 3, and we find that the human-annotated interactions overall align with estimated PID: the highest values are the same for 4 datasets: both explain highest synergy on VQA and CLEVR, image  $(U_{1})$  being the dominant modality in AV-MNIST, and language  $(U_{1})$  being the dominant modality in MOSEI. Overall, the Krippendorff's alpha for inter-annotator agreement is high (0.72 for  $R$ , 0.68 for  $U_{1}$ , 0.70 for  $U_{2}$ , 0.72 for  $S$ ) and the average confidence scores are also high (4.36/5 for  $R$ , 4.35/5 for  $U_{1}$ , 4.27/5 for  $U_{2}$ , 4.46/5 for  $S$ ), indicating that the human-annotated results are reliable. For the remaining two datasets (UR-FUNNY and MUSTARD), estimated PID matches the second-highest human-annotated interaction. We believe this is because there is some annotator subjectivity in interpreting whether sentiment, humor, and sarcasm are present in language only ( $U_{1}$ ) or when contextualizing both language and video ( $S$ ), resulting in cases of low annotator agreement in  $U_{1}$  and  $S$ : -0.14, -0.03 for UR-FUNNY and -0.08, -0.04 for MUSTARD.
Comparisons with other interaction measures: Our framework allows for easy generalization to other interaction definitions: we also implemented 3 information theoretic measures I-min [108], WMS [19], and CI [74]. These results are in Table 10 in the Appendix, where we explain the limitations of these methods as compared to PID, such as over- and under-estimation, and potential negative estimation [41]. These are critical problems with the application of information theory for shared  $I(X_{1};X_{2};Y)$  and unique information  $I(X_{1};Y|X_{2})$ ,  $I(X_{2};Y|X_{1})$  often quoted in the co-training [8, 14] and multi-view learning [89, 94, 98] literature. We also tried 3 non-info theory measures: Shapley values [67], Integrated gradients (IG) [91], and CCA [4], which are based on quantifying interactions captured by a multimodal model. Our work is fundamentally different in that interactions are properties of data before training any models (see Appendix C.6).
4.3 Quantifying Multimodal Model Predictions
We now shift our focus to quantifying multimodal models. Do different multimodal models learn different interactions? A better understanding of the types of interactions that our current models struggle to capture can provide new insights into improving these models.
Table 4: Average interactions  $(R / U / S)$  learned by models alongside their average performance on interaction-specialized datasets  $(\mathcal{D}_R / \mathcal{D}_U / \mathcal{D}_S)$ . Synergy is the hardest to capture and redundancy is relatively easier to capture by existing models.
Model	EF	ADDITIVE	AGREE	ALIGN	ELEM	TENSOR	MI	MULT	LOWER	REC	AVERAGE
R	0.35	0.48	0.44	0.47	0.27	0.55	0.20	0.40	0.47	0.53	0.41 ± 0.11
Acc(DR)	0.71	0.74	0.73	0.74	0.70	0.75	0.67	0.73	0.74	0.75	0.73 ± 0.02
U	0.29	0.31	0.19	0.44	0.20	0.52	0.18	0.45	0.55	0.55	0.37 ± 0.14
Acc(DU)	0.66	0.55	0.60	0.73	0.66	0.73	0.66	0.72	0.73	0.73	0.68 ± 0.06
S	0.13	0.09	0.08	0.29	0.14	0.33	0.12	0.29	0.31	0.32	0.21 ± 0.10
Acc(DS)	0.56	0.66	0.63	0.72	0.66	0.74	0.65	0.72	0.73	0.74	0.68 ± 0.06
Setup: For each dataset, we train a suite of models on the train set  $\mathcal{D}_{\mathrm{train}}$  and apply it to the validation set  $\mathcal{D}_{\mathrm{val}}$ , yielding a predicted dataset  $\mathcal{D}_{\mathrm{pred}} = \{(x_1, x_2, \hat{y}) \in \mathcal{D}_{\mathrm{val}}\}$ . Running PID on  $\mathcal{D}_{\mathrm{pred}}$  summarizes the interactions that the model captures. We categorize and implement a comprehensive suite of models (spanning representation fusion at different feature levels, types of interaction inductive biases, and training objectives) that have been previously motivated to capture redundant, unique, and synergistic interactions (see Appendix C.7 for full model descriptions).
Results: We show results in Table 4 and highlight the following observations:
General observations: We first observe that model PID values are consistently higher than dataset PID. The sum of model PID is also a good indicator of test performance, which agrees with their formal definition since their sum is equal to  $I(\{X_1, X_2\}; Y)$ , the total task-relevant information.
On redundancy: Several methods succeed in capturing redundancy, with an overall average of  $R = 0.41 \pm 0.11$  and accuracy of  $73.0 \pm 2.0\%$  on redundancy-specialized datasets. Additive, agreement, and alignment-based methods are particularly strong, and we do expect them to capture redundant shared information [26, 82]. Methods based on tensor fusion (synergy-based), including lower-order interactions, and adding reconstruction objectives (unique-based) also capture redundancy.
On uniqueness: Uniqueness is harder to capture than redundancy, with an average of  $U = 0.37 \pm 0.14$ . Redundancy-based methods like additive and agreement do poorly on uniqueness, while those designed for uniqueness (lower-order interactions [115] and modality reconstruction objectives [100]) do well, with an average  $U = 0.55$  and  $73.0\%$  accuracy on uniqueness datasets.
On synergy: Synergy is the hardest to capture, with an average score of only  $S = 0.21 \pm 0.10$ . Some of the strong methods are tensor fusion [33], tensors with lower-order interactions [115], modality reconstruction [100], and multimodal transformer [112], which achieve around  $S = 0.30$ , acc = 73.0%. Additive, agreement, and element-wise interactions do not seem to capture synergy well.
On robustness: Finally, we also show connections between PID and model performance in the presence of missing modalities. We find high correlation  $(\rho = 0.8)$  between the performance drop when  $X_{i}$  is missing and the model's  $U_{i}$  value. Inspecting Figure 4, we find that the implication only holds in one direction: high  $U_{i}$  coincides with large performance drops (in red), but low  $U_{i}$  can also lead to performance drops (in green). The latter can be further explained by the presence of large  $S$  values:
when  $X_{i}$  is missing, synergy can no longer be learned which affects performance. For the subset of points when  $U_{i} \leq 0.05$ , the correlation between  $S$  and performance drop is  $\rho = 0.73$  (in contrast, the correlation for  $R$  is  $\rho = 0.01$ ).
Figure 4: We find high correlation  $(\rho = 0.8)$  between the performance drop when  $X_{i}$  is missing and the model's  $U_{i}$  value: high  $U_{i}$  coincides with large performance drops (red), but low  $U_{i}$  can also lead to performance drops. The latter can be further explained by large  $S$  so  $X_{i}$  is necessary (green).
4.4 PID Agreement and Model Selection
Now that we have quantified datasets and models individually, the natural next question unifies both: what does the agreement between dataset and model PID measures tell us about model performance? We hypothesize that models able to capture the interactions necessary in a given dataset should also
Table 5: Model selection results on unseen synthetic and real-world datasets. Given a new dataset  $\mathcal{D}$ , finding the closest synthetic dataset  $\mathcal{D}'$  with similar PID values and recommending the best models on  $\mathcal{D}'$  consistently achieves  $95\% - 100\%$  of the best-performing model on  $\mathcal{D}$ .
Dataset	5 Synthetic Datasets	MIMIC	ENRICO	UR-FUNNY	MOSEI	MUSTARD	MAPS
% Performance	99.91%	99.78%	100%	98.58%	99.35%	95.15%	100%
achieve high performance. Given estimated interactions on dataset  $\mathcal{D}$  and model  $f(\mathcal{D})$  trained on  $\mathcal{D}$ , we define the agreement for each interaction  $I\in \{R,U_1,U_2,S\}$  as:
$$
\alpha_ {I} (f, \mathcal {D}) = \hat {I} _ {\mathcal {D}} I _ {f (\mathcal {D})}, \quad \hat {I} _ {\mathcal {D}} = \frac {I _ {\mathcal {D}}}{\sum I ^ {\prime} \in \{R , U _ {1} , U _ {2} , S \}} I _ {\mathcal {D}} ^ {\prime}, \tag {6}
$$
which summarizes the quantity of an interaction captured by a model  $(I_{f(\mathcal{D})})$  weighted by its normalized importance in the dataset  $(\hat{I}_{\mathcal{D}})$ . The total agreement sums over  $\alpha(f, \mathcal{D}) = \sum_{I} \alpha_{I}(f, \mathcal{D})$ .
Results: Our key finding is that PID agreement scores  $\alpha(f, \mathcal{D})$  correlate  $(\rho = 0.81)$  with model accuracy across all 10 synthetic datasets as illustrated in Figure 5. This shows that PID agreement can be a useful proxy for model performance. For the specialized datasets, we find that the correlation between  $\alpha_I$  and  $\mathcal{D}_I$  is 0.96 for  $R$ , 0.86 for  $U$ , and 0.91 for  $S$ , and negatively correlated with other specialized datasets. For mixed datasets with roughly equal ratios of each interaction, the measures that correlate most with performance are  $\alpha_R$  ( $\rho = 0.82$ ) and  $\alpha_S$  ( $\rho = 0.89$ ); datasets with relatively higher redundancy see  $\rho = 0.89$  for  $\alpha_R$ ; those with higher uniqueness have  $\alpha_{U_1}$  and  $\alpha_{U_2}$  correlate  $\rho = 0.92$  and  $\rho = 0.85$ ; those with higher synergy increases the correlation of  $\alpha_S$  to  $\rho = 0.97$ .
Using these observations, our final experiment is model selection: can we choose the most appropriate model to tackle the interactions required for a dataset?
Figure 5: PID agreement  $\alpha (f,\mathcal{D})$  between datasets and models strongly correlate with model accuracy  $(\rho = 0.81)$
Setup: Given a new dataset  $\mathcal{D}$ , we first compute its difference in normalized PID values with respect to  $\mathcal{D}'$  among our suite of 10 synthetic datasets,  $s(\mathcal{D},\mathcal{D}') = \sum_{I\in \{R,U_1,U_2,S\}}|\hat{I}_{\mathcal{D}} - \hat{I}_{\mathcal{D}'}|$ , to rank the dataset  $\mathcal{D}^*$  with the most similar interactions, and return the top-3 performing models on  $\mathcal{D}^*$ . In other words, we select models that best capture interactions that are of similar nature and degree as those in  $\mathcal{D}$ . We emphasize that even though we restrict dataset and model search to synthetic datasets, we evaluate model selection on real-world datasets and find that it generalizes to the real world.
Results: We test our selected models on 5 new synthetic datasets with different PID ratios and 6 real-world datasets, summarizing results in Table 5. We find that the top 3 chosen models achieve  $95\% - 100\%$  of the best-performing model accuracy, and  $>98.5\%$  for all datasets except  $95.2\%$  on MUSTARD. For example, UR-FUNNY and MUSTARD have the highest synergy ( $S = 0.13$ ,  $S = 0.3$ ) and indeed transformers and higher-order interactions are helpful (MULT:  $65\%$ , MI:  $61\%$ , TENSOR:  $60\%$ ). ENRICO has the highest  $R = 0.73$  and  $U_2 = 0.53$ , and methods for redundant and unique interactions perform best (LOWER:  $52\%$ , ALIGN:  $52\%$ , AGREE:  $51\%$ ). MIMIC has the highest  $U_1 = 0.17$ , and unimodal models are mostly sufficient [60].
4.5 Real-world Applications
Finally, we apply PID to 3 real-world case studies: pathology, mental health, and robotic perception (see full details and results in Appendix C.9-C.11).
Case Study 1: Computational pathology. Cancer prognostication is a challenging task in anatomic pathology that requires integration of whole-slide imaging (WSI) and molecular features for patient stratification [21, 63, 71]. We use The Cancer Genome Atlas (TCGA), a large public data consortium of paired WSI, molecular, and survival information [107, 96], including modalities: (1) pre-extracted histology image features from diagnostic WSIs and (2) bulk gene mutation status, copy number variation, and RNA-Seq abundance values. We evaluate on two cancer datasets in TCGA, lower-grade glioma (LGG [73],  $n = 479$ ) and pancreatic adenocarcinoma (PAAD [83],  $n = 209$ ).
Results: In TCGA-LGG, most PID measures were near-zero except  $U_{2} = 0.06$  for genomic features, which indicates that genomics is the only modality containing task-relevant information. This conclusion corroborates with the high performance of unimodal-genomic and multimodal models in Chen et al. [22], while unimodal-pathology performance was low. In TCGA-PAAD, the uniqueness in pathology and genomic features was less than synergy ( $U_{1} = 0.06$ , and  $U_{2} = 0.08$  and  $S = 0.15$ ), which also match the improvement of using multimodal models that capture synergy.
Case Study 2: Mental health. Suicide is the second leading cause of death among adolescents [18]. Intensive monitoring of behaviors via adolescents' frequent use of smartphones may shed new light on the early risk of suicidal ideations [37, 72], since smartphones provide rich behavioral markers [59]. We used a dataset, MAPS, of mobile behaviors from high-risk consenting adolescent populations (approved by IRB). Passive sensing data is collected from each participant's smartphone across 6 months. The modalities include (1) text entered by the user represented as a bag of top 1000 words, (2) keystrokes that record the exact timing and duration of each typed character, and (3) mobile applications used per day as a bag of 137 apps. Every morning, users self-report their daily mood, which we discretized into  $-1, 0, +1$ . In total, MAPS has 844 samples from 17 participants.
Results: We first experiment with  $\mathrm{MAPS}_{T,K}$  using text and keystroke features. PID measures show that  $\mathrm{MAPS}_{T,K}$  has high synergy (0.40), some redundancy (0.12), and low uniqueness (0.04). We found the purely synergistic dataset  $\mathcal{D}_S$  has the most similar interactions and the suggested models LOWER, REC, and TENSOR that work best on  $\mathcal{D}_S$  were indeed the top 3 best-performing models on  $\mathrm{MAPS}_{T,K}$ , indicating that model selection is effective. Model selection also retrieves the best-performing model on  $\mathrm{MAPS}_{T,A}$  using text and app usage features.
Case Study 3: Robotic Perception. MuJoCo PUSH [57] is a contact-rich planar pushing task in MuJoCo [95], where a 7-DoF Panda Franka robot is pushing a circular puck with its end-effector in simulation. The dataset consists of 1000 trajectories with 250 steps sampled at 10Hertz. The multimodal inputs are gray-scaled images from an RGB camera, force and binary contact information from a force/torque sensor, and the 3D position of the robot end-effector. We estimate the 2D position of the unknown object on a table surface while the robot intermittently interacts with it.
Results: We find that BATCH predicts  $U_{1} = 1.79$  as the highest PID value, which aligns with our observation that image is the best unimodal predictor. Comparing both estimators, CVX underestimates  $U_{1}$  and  $R$  since the high-dimensional time-series modality cannot be easily described by clusters without losing information. In addition, both estimators predict a low  $U_{2}$  value but attribute high  $R$ , implying that a multimodal model with higher-order interactions would not be much better than unimodal models. Indeed, we observe no difference in performance between these two.
Our work aims to quantify the nature and degree of feature interactions by proposing scalable estimators for redundancy, uniqueness, and synergy suitable for high-dimensional heterogeneous datasets. Through comprehensive experiments and real-world applications, we demonstrate the utility of our proposed framework in dataset quantification, model quantification, and model selection. We are aware of some potential limitations:
1. These estimators only approximate real interactions due to cluster preprocessing or unimodal models, which naturally introduce optimization and generalization errors. We expect progress in density estimators, generative models, and unimodal classifiers to address these problems.
2. It is harder to quantify interactions for certain datasets, such as ENRICO which displays all interactions which makes it difficult to distinguish between  $R$  and  $S$  or  $U$  and  $S$ .
3. Finally, there exist challenges in quantifying interactions since the data generation process is never known for real-world datasets, so we have to resort to human judgment, other automatic measures, and downstream tasks such as estimating model performance and model selection.