SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 10: Results for PID decomposition under Cartesian coordinates. Within each plot: PID values for GMMs for  $\angle \mu$  in  $\left\{\frac{i}{40} \cdot \frac{\pi}{2} | i \in \{0, \dots, 40\}\right\}$ .  $\star, \triangle, \nabla, \bullet, +$  represent the  $R, U_1, U_2, S$  and their sum respectively.  $(r, \theta)$  correspond to the unique information  $(U_1, U_2)$  respectively. Across columns: As  $||\mu||_2$  varies from  $\{0.1, 0.5, 1.0, 1.5, 2.0\}$ . Top-Bottom: Soft and Hard labels respectively. Note the scale of the  $y$ -axis for Soft. The overall trend for each PID component as  $\angle \mu$  varies is similar across  $||\mu||_2$ . However, the relative contribution of Synergy and Redundancy changes significantly as  $||\mu||_2$  varies, particularly for Hard.
neural network in Algorithm 1, we use a 3-layer feedforward neural network with a hidden dimension of 32. We train the network for 10 epochs using the Adam optimizer with a batch size of 256 and learning rate of 0.001.
To summarize, we propose 2 scalable estimators for PID, each with their approximations for the parameterization of  $q$  and procedure for the estimation of PID.
1. Parameterizing  $q$  explicitly via a joint probability matrix over  $\Delta_p$ , and optimizing convex objective (5) with linear constraints. This can only be used for finite  $\mathcal{X}_1, \mathcal{X}_2, Y$ , or after preprocessing by histogramming continuous domains.
2. Parameterizing  $q$  via neural networks and optimize their parameters  $\phi$  by repeatedly subsampling from data; for each subsample, we perform one step of projected gradient descent (using Sinkhorn's method) as if this subsample was the full dataset.
C.1 Synthetic Bitwise Features
We show results on estimating PID values for OR, AND, and XOR interactions between  $X_{1}$  and  $X_{2}$  in Table 1 and find that both our estimators exactly recover the ground truth results in Bertschinger et al. [13].
C.2 Gaussian Mixture Models (GMM)
Extended Results: For each of Cartesian and Polar we consider two cases, Soft-labels, when we observe  $Y$ , and Hard-labels, where we use  $\hat{Y}$  predicted by the optimal linear classifier  $\hat{y} = \mathrm{sign}(\langle (x_1,x_2),\mu \rangle)$ . We hope this will provide additional intuition to the bitwise modality experiments. This results are given in Figures 10, 11. Most of the trends are aligned with what we would expect from Redundancy, Uniqueness, and Synergy. We list the most interesting observations here.
- In all cases, total information (sum of PID values) is independent of  $\angle \mu$ . This is a natural consequence of rotational symmetry. It is also higher for higher  $||\mu||_2$ , e.g., 2.0, reflecting the fact that  $X_{1}, X_{2}$  contain less information about labels when their centroids are close. Lastly, observe, that when  $||\mu||_2$  increases, the total information approaches  $\log(2)$ , which is all the label information. This perfectly aligned with our intuition: if the cluster centroids are far away, then knowing a datapoint's cartesian coordinates will let us predict its label accurately (which is one bit of information).
- For Cartesian, unique information dominates when angle goes to 0 or  $\pi /2$ . This is sensible, if centroids share a coordinate, then observing that coordinate yields no information about  $y$ . Conversely, synergy and redundancy peak at  $\pi /4$ .
Figure 11: Results for PID decomposition under Polar coordinates where  $\theta \in [0,\pi ]$  and  $r\in \mathbb{R}$  i.e., angle is restricted to the first two quadrants but distance could be negative. Within each plot: PID values for GMMs for  $\angle \mu$  in  $\{\frac{i}{40}\cdot \frac{\pi}{2}|i\in \{0,\dots ,40\} \}$ .  $\times ,\triangle ,\nabla ,\bullet ,+$  represent the  $R$ ,  $U_{1}$ ,  $U_{2}$ ,  $S$  and their sum respectively. Across columns: As  $||\mu ||_2$  varies from  $\{0.1,0.5,1.0,1.5,2.0\}$ . Top-Bottom: Soft and Hard labels respectively. Note the scale of the  $y$ -axis for Soft. The overall trend for each PID component as  $\angle \mu$  varies is similar across  $||\mu ||_2$  except for when  $||\mu ||_2$  is small in Soft. However, the relative contribution of Synergy and Redundancy changes significantly as  $||\mu ||_2$  varies, particularly for Hard.
Figure 12: Results for PID decomposition under Polar coordinates, using the normal parameterization where  $\theta \in [-\pi ,\pi ]$  and  $r\geq 0$ . Within each plot: PID values for GMMs for  $\angle \mu$  in  $\{\frac{i}{40}\cdot \frac{\pi}{2}|i\in$ $\{0,\dots ,40\} \}$  .  $\times ,\triangle ,\nabla ,\bullet , +$  represent the  $R,U_{1},U_{2},S$  and their sum respectively.  $(r,\theta)$  correspond to the unique information  $(U_{1},U_{2})$  respectively. Across columns: As  $||\mu ||_2$  varies from  $\{0.1,0.5,1.0,1.5,2.0\}$ . Top-Bottom: Soft and Hard labels respectively. Note the scale of the  $y$  -axis for Soft. Observe that PID values are independent of  $\angle \mu$ , with unique information from  $\theta$  dominating.
- Redundancy dominates Synergy as  $||\mu||_2$  increases. Interestingly, synergy seems to be independent of  $||\mu||_2$  (one has to account for the scale in the vertical axis to recognize this).
- Hard contains  $0.65 \approx \log(2)$  nats of total information, since  $Y|X_1, X_2$  is deterministic. The deficit of 0.04 nats is due to discretization near the decision boundary, and is expected to vanish if we increase the number of bins (though this ought to be accompanied by an increase in samples used to approximate PID).
- However, for Soft, total information is nearly 0 for small  $||\mu||_2$  (and approaches  $\log(2)$  when large). When the two cluster centroids are close, it is difficult to glean any information about the (soft) label, while we can predict  $y$  accurately from both  $x_1, x_2$  when centroids are far apart.
- For Polar, redundancy is 0. Furthermore,  $\theta$  contains no unique information, since  $\theta$  shows nothing about  $y$  unless we know  $r$  (in particular, its sign). When angle goes to  $\pi /2$ , almost all information is unique in  $r$ .
Lastly, we also experimented on the more common definition of polar coordinates, with  $\theta \in [-\pi, \pi]$  and  $r \geq 0$  (that is, angle is allowed to be negative but lengths have to be nonnegative). The results are in Figure 12. As one may expect, there is nothing exciting going on for both Soft and Hard. For Hard, the optimal classifier is precisely expressed in terms of  $\theta$  only, so all the information belongs to  $U_{2}$ , i.e.,  $\theta$ . For Soft, the same trend remains: virtually all information is in  $U_{2}$ , i.e.,  $\theta$ , with the trend becoming even sharper as  $||\mu||_{2}$  increases. Intuitively, when the 2 centroids are extremely far
apart, then the optimal linear classifier is extremely accurate; which may be expressed by  $\theta$  alone without any knowledge of  $r$ . Conversely, when the centroids are close,  $r$  can provide some synergy with  $\theta$ —the larger  $r$  is, the more "confident" we are in predictions based on  $\theta$ .
Visualization of  $q^{*}$ : The following plots of  $q^{*}$  are provided to provide some intuition behind the objectives in (16). We focus on Cartesian for ease of visualization. For each  $||\mu||_2 \in \{0.5, 1.0, 2.0\}$ , we vary the angle  $\angle \mu$  and sample 1e7 labelled datapoints. We then compare  $q^{*}$  (for both Soft and Hard) with the empirical distribution  $p$ . To compute PID, we use 50 bins evenly spaced between  $[-5, 5]$  for both dimensions. The results for varying  $||\mu||_2$  are shown in Figures 13-15.
When  $\angle \mu$  is 0, there are few differences, barring a slightly "blockier" feel for  $q^{*}$  over  $p$ . However, as  $\angle \mu$  increases,  $q^{*}$  gets skewed rather than rotated, becoming "thinner" in the process. The limiting case is when  $\angle \mu_{2} = \pi /4$ , i.e., 45 degrees. Here, the objective diverges to infinity, with what appears to be a solution with nonnegative entries only along the diagonal—this observation appears to hold even as we increase the number of bins. We hypothesize that in the truly continuous case (recall we are performing discretization for this set of experiments), the optimum is a degenerate "one dimensional" GMM with mass located only at  $q(x,x)$ .
Figure 13: Densities for GMMs where  $||\mu ||_2 = 2.0$ . Top to Bottom: Densities for  $p$ ,  $q^*$  (Soft), and  $q^*$  (Hard). Left to right,  $\angle \mu$  in  $\{\frac{i}{8} \cdot \frac{\pi}{4} | i \in \{0, \dots, 8\}\}$
Figure 14: Densities for GMMs where  $||\mu ||_2 = 1.0$ . Top to Bottom: Densities for  $p$ ,  $q^*$  (Soft), and  $q^{*}$  (Hard). Left to right,  $\angle \mu$  in  $\{\frac{i}{8} \cdot \frac{\pi}{4} | i \in \{0, \dots, 8\}\}$
C.3 Synthetic Generative Model
Our third setting simulates a data generation process where there exists a set of latent concepts that either redundantly, uniquely, or synergistically determine the label.
Setup: We begin with a fixed set of latent variables  $z_{1},z_{2},z_{c}\in \mathbb{R}^{50}$  from  $\mathcal{N}(0,\sigma^2)$  with  $\sigma = 0.5$  representing latent concepts for modality 1, modality 2, and common information respectively. The concatenated variables  $[z_1,z_c]$  are transformed into high-dimensional  $x_{1}\in \mathbb{R}^{100}$  data space using a fixed transformation matrix  $T_{1}\in \mathbb{R}^{50\times 100}$  and likewise  $[z_{2},z_{c}]$  to  $x_{2}$  via  $T_{2}$ . The label  $y$  is generated
Figure 15: Densities for GMMs where  $||\mu ||_2 = 0.5$ . Top to Bottom: Densities for  $p$ ,  $q^*$  (Soft), and  $q^*$  (Hard). Left to right,  $\angle \mu$  in  $\{\frac{i}{8} \cdot \frac{\pi}{4} | i \in \{0, \dots, 8\}\}$
as a transformation function of (1) only  $z_{c}$ , in which case we expect redundancy in  $x_{1}$  and  $x_{2}$  with respect to  $y$ , (2) only  $z_{1}$  or  $z_{2}$  which represents uniqueness in  $x_{1}$  or  $x_{2}$  respectively, (3) the concatenated  $[z_{1},z_{2}]$  representing synergy, or (4) arbitrary ratios of each of the above. We also introduce nonlinearity and random noises to increase data complexity. In formal notation,
$$
y = \left[ \operatorname {s i g m o i d} \left(\frac {\sum_ {i = 0} ^ {n} f (\mathbf {z}) _ {i}}{n}\right) \geq 0. 5 \right]
$$
where  $f$  is a fixed nonlinear transformation with dropout ( $p = 0.1$ ) and  $\mathbf{z}$  can be  $z_{1}, z_{2}, z_{c}$  or any combination according to the cases described above. These datasets are designed to test a combination of feature learning with predefined interactions between the modalities and the label.
We generated 10 synthetic datasets as shown in Table 2. The first four are specialized datasets denoted as  $\mathcal{D}_m$ ,  $m \in \{R, U_1, U_2, S\}$  for which the label  $y$  only depends on redundancy, uniqueness, or synergy features respectively. The rest are mixed datasets with  $y$  generated from  $z_1, z_2, z_c$  of varying dimensions. We use  $z_i^* \in \mathbb{R}^{50}$  to denote a randomly sub-sampled feature from  $z_i$ . For example, to generate the label of one data point  $(x_1^{(j)}, x_2^{(j)})$  in the mixed dataset  $y = (z_1^*, z_2^*, z_c)$ , we sub-sampled  $(z_1^*)(^j), (z_2^*)(^j)$  from  $z_1, z_2$  and generate  $y$  from  $[(z_1^*)(^j), (z_2^*)(^j), z_c]$  following the process above.
To report the ground-truth interactions for these datasets, we first train multimodal models and obtain an estimate of test performance on them. We then convert  $P_{\mathrm{acc}}$ , the test accuracy of the best model on these datasets into the mutual information between the inputs to the label using the bound in Feder and Merhav [29]:
$$
I \left(X _ {1}, X _ {2}; Y\right) \leq \log P _ {\mathrm {a c c}} + H (Y) \tag {34}
$$
Since the test accuracies for Table 2 datasets range from  $67 - 75\%$ , this corresponds to a total MI of  $0.42 - 0.59$  bits (i.e., from  $\log_2 0.67 + 1$  to  $\log_2 0.75 + 1$ ). The information in each interaction is then computed by dividing this total mutual information by the interactions involved in the data generation process: if the total MI is 0.6 bits and the label depends on half from the common information between modalities and half from the unique information in  $x_{1}$ , then the ground truth  $R = 0.3$  and  $U_{1} = 0.3$ .
Results: In the main paper, we note the agreement between CVX and BATCH and their consistency with the dataset (Table 2). In addition, we note that the sum of PID values (which sum to  $I_{p}(Y;X_{1},X_{2})$ ) is generally larger in BATCH than in CVX, which accounts for the information loss during CVX's clustering. Moreover, in the last two cases ( $y = f(z_2^*,z_c^*)$  and  $y = f(z_2^*,z_c)$ ), CVX predicts non-zero amount of  $S$  despite the dataset being synergy-free by construction. Because the total mutual information is small, the spurious synergy becomes a large portion of the total PID values. This potentially demonstrates that when the total mutual information is small, spurious information gain could have a large impact on the results.
C.4 Quantifying Multimodal Datasets
Setup: We use a large collection of real-world datasets in MultiBench [60] which test multimodal fusion of different input signals and requires representation learning of complex real-world interactions
Table 6: Full results on estimating PID on real-world MultiBench [60] datasets. Many of the estimated interactions align well with dataset construction (e.g., MUSTARD for sarcasm) and unimodal performance (e.g., MIMIC and AV-MNIST). On QA [5] datasets, synergy is consistently the highest.
Task	AV-MNIST	ENRICO	VQA 2.0	CLEVR
Measure	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S
CVX	0.10	0.97	0.03	0.08	0.73	0.38	0.53	0.34	0.79	0.87	0	4.92	0.55	0.48	0	5.16
Task	MIMIC	UR-FUNNY A,T	UR-FUNNY v,T	UR-FUNNY V,A
Measure	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S
BATCH	0.05	0.17	0	0.01	0.02	0	0.08	0.07	0.06	0	0.04	0
Task	MOSEIA,T	MOSEIV,T	MOSEIV,A
Measure	R	U1	U2	S	R	U1	U2	S	R
BATCH	0.22	0.04	0.09	0.13	0.30	0.70	0	0	0.26
Task	MUSTARD A,T	MUSTARD V,T	MUSTARD V,A
Measure	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S
BATCH	0.14	0.01	0.01	0.37	0.14	0.02	0.01	0.34	0.14	0.01	0.01	0.20
for different tasks. MultiBench spans 10 diverse modalities (images, video, audio, text, time-series, various robotics sensors, sets, and tables), 15 prediction tasks (humor, sentiment, emotions, mortality rate, ICD-9 codes, image-captions, human activities, digits, robot pose, object pose, robot contact, and design interfaces), and 5 research areas (affective computing, healthcare, multimedia, robotics, and HCI). These datasets are designed to test a combination of feature learning and arbitrary complex interactions between the modalities and the label in the real world. We also include experiments on question-answering (Visual Question Answering [5, 39] and CLEVR [53]) which test grounding of language into the visual domain.
Details of applying CVX: For quantifying multimodal datasets with CVX, we first apply PCA to reduce the high dimension of multimodal data. For each of the train, validation, and test split, we then use unsupervised clustering to generate 20 clusters. We obtain a clustered version of the original dataset  $\mathcal{D} = \{(x_1,x_2,y)\}$  as  $\mathcal{D}_{\mathrm{cluster}} = \{(c_1,c_2,y)\}$  where  $c_{i}\in \{1,\ldots ,20\}$  is the ID of the cluster that  $x_{i}$  belongs to. For datasets with 3 modalities we estimate PID separately for each of the 3 modality pairs.
Details of applying BATCH: For quantifying multimodal datasets with BATCH, we first use pretrained encoders to obtain the image and text features, and then train 2 unimodal and 1 multimodal classifiers on those features to predict the original answer. The final PID values are computed using the encoded features and the 3 classifiers. In our experiments we used the pretrained image and text encoders from the CLIP-VIT-B/32 [82] model, and the unimodal and multimodal classifiers are 2-layer MLPs.
Results on multimodal fusion: We show full results on quantifying datasets in Table 6. For multimodal fusion, we find that different datasets do require different interactions. Among interesting observations are that (1) all pairs of modalities on sarcasm detection shows high synergy values, which aligns with intuition on sarcasm in human communication, (2) uniqueness values are strongly correlated with unimodal performance (e.g., AV-MNIST and MIMIC first modality), (3) datasets with high synergy benefit from interaction modeling (e.g., sarcasm), and (4) conversely datasets with low synergy are also those where modeling in higher-order interactions do not help (e.g., MIMIC).
To test grounding of language into the visual domain, we also run experiments on the VQA v2.0 [5, 39] dataset, which contains 443757 training and 214354 validation data with real-life images, and the CLEVR [53] dataset, which contains 699960 training and 149991 validation data with rendered images and generated questions. The test spilt for both datasets are not used because the labels are not publicly available. Note that for both validation datasets, we first filter out all data instances where the label is neither "yes" nor "no". On the remaining data, we use the outputs from the last feature layer of a VGG19 [87] model as image embeddings and use RoBERTa [64] to obtain the text embeddings for questions, and then apply K-means clustering to both embeddings to obtain a clustered version
Table 7: Estimating PID on QA [5] datasets with both CVX and BATCH estimators. Synergy is consistently the highest and both estimators return PID values that are consistent relative to each other.
Task	VQA 2.0	CLEVR
Measure	R	U1	U2	S	R	U1	U2	S
CVX	0.79	0.87	0	4.92	0.55	0.48	0	5.16
BATCH	0.10	0.14	0.09	0.16	0.01	0.01	0.01	0.03
Human	0.23	0	0	4.80	0	0	0	5.00
Instruction:Look at the data and provide your rating for the following questions on a scale of 0 (none at all) - 5 (large extent)and score your confidence in your answer from a scale of 0 (no confidence) - 5 (high confidence):	1. The extent to which both modalities enable you to make the same predictions about the task.	2. The extent to which modality 1 enables you to make a prediction about the task that you would not if using modality 2.	3. The extent to which modality 2 enables you to make a prediction about the task that you would not if using modality 1.	4. The extent to which both modalities enable you to make a prediction about the task that you would not otherwise make using either modality individually.
ID	Modality 1	Modality 2	1 Rating	1 Confidence	2 Rating	2 Confidence	3 Rating	3 Confidence	4 Rating	4 Confidence
mosei1	video	The devil deceives false Christians into thinking they are already safe and secure and on their way to heaven.
mosei2	video	Currently, over four-hundred thousand Americans with disabilities are being paid less than the minimum wage, some of them mere pennies per hour.
Figure 16: We show a sample user interface for annotating information decomposition into redundancy, uniqueness, and synergy, which we give to 3 annotators each to obtain agreeing and reliable judgments of each interaction for real-world multimodal datasets.
of the original dataset  $\mathcal{D} = \{(x_1, x_2, y)\}$  as  $\mathcal{D}_{\mathrm{cluster}} = \{(c_1, c_2, y)\}$ , where  $c_i \in \{1, \dots, 20\}$  are the cluster IDs. We then calculate the PID values on the clustered dataset  $\mathcal{D}_{\mathrm{cluster}}$ .
Results on QA: As a whole, we observe consistently high synergy values on QA datasets as shown in Table 6. This is consistent with prior work studying how these datasets were balanced (e.g., VQA 2.0 having different images for the same question such that both yes/no answers are observed in the dataset so that the answer can only be obtained through synergy) [39] and models trained on these datasets [46].
Comparing CVX and BATCH: We carefully compare CVX and BATCH on the 2 question answering datasets VQA and CLEVR, showing these results in Table 7. We choose these two datasets since their input modalities are images and text, which make them suitable for both clustering of representations extracted from pre-trained before input into CVX, and end-to-end encoding and PID estimation with BATCH. We also include human judgment of PID on these 2 datasets for comparison. From Table 7, we find that all three measures agree on the largest interaction, as well as the relative order of interaction values. We find that the absolute values of each interaction can be quite different due to the unavoidable information loss of both input modalities to the label during clustering in CVX or representation learning in BATCH. Hence, we emphasize that these estimated interactions are most useful through relative comparison to judge which interaction is most dominant in a particular dataset.
C.5 Comparisons with Human Judgment
To quantify information decomposition for real-world tasks, we investigate whether human judgment can be used as a reliable estimator. We propose a new annotation scheme where we show both modalities and the label and ask each annotator to annotate the degree of redundancy, uniqueness, and synergy on a scale of 0-5 using the following definitions inspired by the formal definitions in information decomposition:
1.  $R$ : The extent to which using the modalities individually and together gives the same predictions on the task,
2.  $U_{1}$ : The extent to which  $x_{1}$  enables you to make a prediction about the task that you would not if using  $x_{2}$ ,
3.  $U_{2}$ : The extent to which  $x_{2}$  enables you to make a prediction about the task that you would not if using  $x_{1}$ ,
4.  $S$ : The extent to which only both modalities enable you to make a prediction about the task that you would not otherwise make using either modality individually,
alongside their confidence in their answers on a scale of 0-5. We show a sample user interface for the annotations in Figure 16.
Table 8: Collection of datasets used for our study of multimodal fusion interactions covering diverse modalities, tasks, and research areas in multimedia and affective computing.
Datasets	Modalities	Size	Prediction task	Research Area
MUSTARD [17]	{text, video, audio}	690	sarcasm	Affective Computing
UR-FUNNY [43]	{text, video, audio}	16, 514	humor	Affective Computing
MOSEI [116]	{text, video, audio}	22, 777	sentiment, emotions	Affective Computing
CLEVR [53]	{image, question}	853, 554	QA	Multimedia
VQA 2.0 [39]	{image, question}	1, 100, 000	QA	Multimedia
Table 9: Annotator agreement and average confidence scores of human annotations of PID values for real-world multimodal datasets.
Task	Agreement	Confidence
Measure	R	U1	U2	S	R	U1	U2	S
Human	0.72	0.68	0.70	0.72	4.36	4.35	4.27	4.46
Based on direct communication with our institution's IRB office, this line of user-study research is aligned with similar annotation studies at our institution that are exempt from IRB. The information obtained during our study is recorded in such a manner that the identity of the human subjects cannot readily be ascertained, directly or through identifiers linked to the subjects. We do not collect any identifiable information from annotators. Participation in all annotations was fully voluntary and we obtained consent from all participants prior to annotations. Participants were not the authors nor in the same research groups as the authors. They all hold or are working towards a graduate degree in a STEM field and have knowledge of machine learning. None of the participants knew about this project before their session and each participant only interacted with the setting they were involved in.
We sample 10 datapoints with class balance from each of the 5 datasets in Table 8 and give them to 3 annotators. All the data were manually selected and do not contain offensive content. We clarified all the confusion that the annotators may have about the instructions and the definitions above at the beginning of the annotation process. We summarize the results and key findings: