SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which adds up the absolute difference between normalized PID values on  $\mathcal{D}$  and  $\mathcal{D}'$ . The model selection pipeline is illustrated in Figure 17. We hypothesize that the top-3 best-performing models on  $\mathcal{D}^*$  should also achieve decent performance on  $\mathcal{D}$  due to their similar distribution of interactions. We verify the quality of the model selection by computing a percentage of the performance of the selected model  $f$  with respect to the performance of the actual best-performing model  $f^*$  on  $\mathcal{D}$ , i.e.  $\%$  Performance  $(f, f^*) = Acc(f) / Acc(f^*)$ .
Results: We summarize our evaluation of the model selection in Table 5, and find that the top 3 chosen models all achieve  $95\% - 100\%$  of the best-performing model accuracy, and above  $98.5\%$  for all datasets except MUSTARD. For example, UR-FUNNY and MUSTARD have the highest synergy ( $S = 0.13$ ,  $S = 0.3$ ) and indeed transformers and higher-order interactions are helpful (MULT:  $0.65\%$ , MI:  $0.61\%$ , TENSOR:  $0.6\%$ ). ENRICO has the highest  $R = 0.73$  and  $U_2 = 0.53$ , and indeed methods for redundant and unique interactions perform best (LOWER:  $0.52\%$ , ALIGN:  $0.52\%$ , AGREE:  $0.51\%$ ). MIMIC has the highest  $U_1 = 0.17$ , and indeed unimodal models are mostly sufficient [60]. We hypothesize that model selection is more difficult on MUSTARD since it has the highest synergy ratio, and it is still an open question regarding how to capture synergistic multimodal interactions. Therefore, PID values provide a strong signal for multimodal model selection.
C.9 Real-world Case Study 1: Computational Pathology
Setup: The current standard of care for cancer prognosis in anatomic pathology is the integration of both diagnostic histological information (via whole-slide imaging (WSI)) as well as molecular information for patient stratification and therapeutic decision-making [63]. For cancer types such as diffuse gliomas, the combination of histological subtypes (astrocytoma, oligodendroglioma) and molecular markers (IDH1 mutation, 1p19q codeletion) determines whether patients should be administered tumor resection, radiation therapy, adjuvant chemotherapy, or combination therapies [66, 16, 20]. Recent applications of multimodal learning in pathology have found that though fusion confers benefit for a majority of cancer types, unimodal models can be as competitive in patient stratification performance, which suggest potential clinical application of prognosticating using only a single modality for certain cancers [22].
Table 20: Results on estimating PID across the TCGA-GBMLGG and TCGA-PAAD datasets.  $U_{1}$ ,  $U_{2}$  correspond to pathology and genomics respectively.
Dataset	TCGA-LGG	TCGA-PAAD
Measure	R	U1	U2	S	R	U1	U2	S
CVX	0	0.02	0.06	0.02	0	0.06	0.08	0.15
Using The Cancer Genome Atlas (TCGA), PID was used to quantify pathology image-omic interactions in downstream prognostication tasks for two cancer datasets: lower-grade glioma (TCGA-LGG [15]  $(n = 479)$  and pancreatic adenocarcinoma (TCGA-PAAD [83],  $(n = 209)$ . As mentioned in the main text, the modalities include: (1) a sequence of pre-extracted histology image features from diagnostic WSIs  $(N \times 1024$ , where  $N$  is the number of non-overlapping  $256 \times 256$  patches at  $20 \times$  magnification and 1024 is the extracted feature dimension of a pretrained vision encoder), and (2) feature vector  $(1 \times D)$  of  $D$  bulk gene mutation status, copy number variation, and RNA-Seq abundance values. The label space is overall survival time converted to 4 discrete bins. Following the study design of [22], we trained an Attention-Based Multiple Instance Learning (ABMIL) model and a Self-Normalizing Network (SNN) for the pathology and genomic modalities using a log likelihood loss for survival analysis respectively [48, 55]. For ABMIL, a truncated ResNet-50 encoder pretrained on ImageNet was used for patch feature extraction of WSIs [45]. Unimodal features were extracted before the classification layer, with K-means clustering  $(k = 3)$  for each modality used to obtain  $(x_{1}, x_{2}, y)$  pairs where  $x_{1}$  is pathology,  $x_{2}$  is genomics and  $y$  is the discrete bins, which can then be used in CVX to compute PID measures.
Results: We assess these PID measures (Table 20) in the context of current scientific understanding of pathology and genomic interactions in cancer prognostication in the literature. Specifically, we compare against the findings [22], which previously investigated unimodal and multimodal survival models in TCGA and observed that risk predictions were primarily driven by genomics across most cancer types (such as TCGA-LGG), with exceptions in a few cancer types such as TCGA-PAAD. In the evaluation of TCGA-LGG and TCGA-PAAD in [22], concordance Index performance of unimodal-pathology, unimodal-genomic, and multimodal survival models were (0.668, 0.792, 0.808) and (0.580, 0.593, 0.653) respectively. In [22], one of the original hypotheses for the lack of multimodal improvement for some cancer types was due to high mutual information between both modalities. In our results, we observe the opposite effect in which there is little mutual information between both modalities. This observation coupled with the low and high synergy values in TCGA-LGG ( $S = 0.02$ ) and TCGA-PAAD ( $S = 0.15$ ) suggest that only TCGA-PAAD would benefit from multimodal integration, which was empirically found with both the c-Index improvement and statistical significant patient stratification results in TCGA-PAAD evaluation. The uniqueness values of both tasks also corroborate the high amount of task-specific information in genomics with high c-Index performance, and vice versa with pathology. Overall, we demonstrate we can use PID measures to refine biomedical insights into understanding when multimodal integration confers benefit.
C.10 Real-world Case Study 2: Mood Prediction from Mobile Data
Setup: Suicide is the second leading cause of death among adolescents [18]. Despite these alarming statistics, there is little consensus concerning imminent risk for suicide [31, 56]. Intensive monitoring of behaviors via adolescents' frequent use of smartphones may shed new light on the early risk of suicidal thoughts and ideations [37, 72]. Smartphones provide a valuable and natural data source with rich behavioral markers spanning online communication, keystroke patterns, and application usage [59]. Learning these markers requires large datasets with diversity in participants, variety in features, and accuracy in annotations. As a step towards this goal, we partnered with several hospitals (approved by NIH IRB for central institution and secondary sites) to collect a dataset of mobile behaviors from high-risk adolescent populations with consent from participating groups. This data monitors adolescents spanning (a) recent suicide attempters (past 6 months) with current suicidal ideation, (b) suicide ideators with no past suicide attempts, and (c) psychiatric controls with no history of suicide ideation or attempts. Passive sensing data is collected from each participant's smartphone across a duration of 6 months. The modalities include (1) text entered by the user represented as a bag of top 1000 words that contains the daily number of occurrences of each word, (2) keystroke features that record the exact timing and duration that each character was typed on a mobile keyboard
Table 21: Results on estimating PID on the MAPS dataset with pairs of modalities text + apps  $(\mathrm{MAPS}_{T,A})$  and text  $^+$  keystrokes  $(\mathrm{MAPS}_{T,K})$
Dataset	MAPST,A	MAPST,K
Measure	R	U1	U2	S	R	U1	U2	S
CVX	0.09	0	0.09	0.26	0.12	0	0.04	0.40
Table 22: Results on estimating PID on model families trained on the MAPS dataset.
Category	GENERAL	REDUNDANCY
Model	EF	ADDITIVE	AGREE
Measure	R	U1	U2	S	Acc	R	U1	U2	S	Acc	R	U1	U2	S	Acc
MAPST,A	0.61	0.01	0.08	0.02	0.42	0.43	0	0.26	0	0.39	0.45	0	0.24	0	0.40
MAPST,K	0.16	0.29	0	0.08	0.51	0.09	0	0.18	0.21	0.47	0.20	0	0.24	0.17	0.42
Category	REDUNDANCY	SYNERGY
Model	ALIGN	ELEM	TENSOR
Measure	R	U1	U2	S	Acc	R	U1	U2	S	Acc	R	U1	U2	S	Acc
MAPST,A	0.69	0	0.11	0.03	0.39	0.42	0.04	0.05	0.13	0.41	0.42	0.10	0.03	0.11	0.45
MAPST,K	0.41	0.01	0.28	0.34	0.33	0.17	0.37	0.02	0.41	0.27	0.24	0.15	0	0.32	0.57
Category	SYNERGY
Model	MI	MULT
Measure	R	U1	U2	S	Acc	R	U1	U2	S	Acc
MAPST,A	0.28	0.12	0.01	0.21	0.47	0.40	0.09	0	0.03	0.48
MAPST,K	0.25	0.24	0.02	0.40	0.36	0.10	0.27	0	0.19	0.50
Category	UNIQUE
Model	LOWER	REC
Measure	R	U1	U2	S	Acc	R	U1	U2	S	Acc
MAPST,A	0.53	0.04	0.04	0.09	0.41	0.28	0.06	0.07	0.10	0.56
MAPST,K	0.12	0.24	0	0.22	0.51	0.23	0.17	0	0.18	0.52
(including alphanumeric characters, special characters, spaces, backspace, enter, and autocorrect), and (3) mobile applications used per day, creating a bag of 137 apps for each day that are used by at least  $10\%$  of the participants.
Every day at 8am, users are asked to respond to the following question - "In general, how have you been feeling over the last day?" - with an integer score between 0 and 100, where 0 means very negative and 100 means very positive. To construct our prediction task, we discretized these scores into the following three bins: negative (0 - 33), neutral (34 - 66), and positive (67 - 100), which follow a class distribution of  $12.43\%$ ,  $43.63\%$ , and  $43.94\%$  respectively. For our 3-way classification task, participants with fewer than 50 daily self-reports were removed since these participants do not provide enough data to train an effective model. In total, our dataset consists of 1641 samples, consisting of data coming from 17 unique participants.
Results: The results are shown in Table 21, 22. We observe that both  $\mathrm{MAPS}_{T,A}$  (text + apps) and  $\mathrm{MAPS}_{T,K}$  (text + keystrokes) have the highest synergy ( $S = 0.26$ ,  $S = 0.4$  respectively) and some redundancy ( $R = 0.09$ ,  $R = 0.12$ ) and uniqueness in the second modality ( $U_2 = 0.09$ ,  $U_2 = 0.04$ ). We find indeed the best-performing models are REC (acc = 0.56%) and MULT (acc = 0.48%) for  $\mathrm{MAPS}_{T,A}$ , and TENSOR (acc = 0.57%) and REC (acc = 0.52%) for  $\mathrm{MAPS}_{T,K}$ , which are designed to capture synergy and uniqueness interactions. Our model selection also successfully chooses REC for  $\mathrm{MAPS}_{T,A}$  and TENSOR for  $\mathrm{MAPS}_{T,K}$  with a high agreement of  $\alpha = 0.66$  and  $\alpha = 0.74$  (98.3% and 99.85% of the highest agreement respectively). This result further corroborates the utility and generalizability of our PID estimations and model selection.
Table 23: Results on estimating PID on robotic perception task.
Task	PUSH I,C
Measure	R	U1	U2	S
CVX	0.24	0.03	0.06	0.04
BATCH	0.75	1.79	0.03	0.08
C.11 Real-world Case Study 3: Robotic Perception
Setup: MuJoCo PUSH [57] is a contact-rich planar pushing task in the MuJoCo simulation framework, in which a 7-DoF Panda Franka robot is pushing a circular puck with its end-effector in simulation. We estimate the 2D position of the unknown object on a table surface, while the robot intermittently interacts with the object. The final dataset consists of 1000 trajectories with 250 steps at 10Hertz, of a simulated Franka Panda robot arm pushing a circular puck in MuJoCo [95]. The pushing actions are generated by a heuristic controller that tries to move the end-effector to the center of the object. The multimodal inputs are gray-scaled images  $(1\times 32\times 32)$  from an RGB camera, forces (and binary contact information) from a force/torque sensor, and the 3D position of the robot end-effector. The task is to predict the 2-D planar object pose.
We use CNN feature extractors and LSTM as a sequence model for both unimodal and multimodal models. We use hyperparameters from Table 16. Because both CVX and BATCH assumes discrete classes of labels, we discretize all continuous labels into 10 bins of equal data points from the training set, and use the same bin cutoffs at test time. The task is to predict the correct bin that contains the original label, and we use cross entropy loss as our loss function.
Results: The results are shown in Table 23. As can be seen from the table, BATCH predicts  $U_{1}$  as the highest PID value ( $U_{1} = 1.79$ ), which aligns with the fact that image is the best unimodal predictor for this dataset [60]. Comparing both estimators, CVX underestimates  $U_{1}$  and  $R$  since CVX clusters the data before processing, and the high-dimensional time-series modalities cannot be easily described by clusters without losing information. In addition, both BATCH and CVX predict a low  $U_{2}$  value but attributes some mutual information to  $R$  (redundancy), implying that a multimodal model with both modalities would not be much better compared to an unimodal model on modality 1, since the multimodal model would mostly get redundant information from modality 2. In our experiments, we observe no difference in performance between the multimodal model and the unimodal model with modality 1.
D Summary of Takeaway Messages and Future Work
From these results, we emphasize the main take-away messages and motivate several directions for future work:
1. Dataset quantification: PID provides reliable indicators for the nature and quantity of interactions present in both synthetic and real-world multimodal datasets, which provides a useful tool summarizing these datasets. In our opinion, creators of new datasets focused on modeling interactions between multiple features or modalities should report estimated PID values on new datasets alongside justification on whether these values are to be expected based on how the features were selected.
2. Model quantification: PID also provides reasonable indicators for the interactions captured by trained models, but naturally there is more noise due to different model parameter counts, training trajectories, and non-convex optimization artifacts that despite our best efforts remain impossible to exactly control. Despite these details, we still find several consistent patterns in the types of interactions different models capture. We suggest that researchers developing new multimodal models explicitly state the assumptions on the underlying multimodal distribution their models require and the interactions their models are designed to capture, before testing exhaustively on datasets predominantly of those interactions.
3. Model selection: Our experiments on model selection for real-world datasets and applications demonstrate potential utility in a rough 'user-guide' for practitioners aiming to tackle real-world multimodal datasets. Given a new dataset, estimate its PID values. If there is high  $U_{1}$  and  $U_{2}$ , just using unimodal models in the corresponding modality may be sufficient.
Otherwise, if there is high  $R$ , methods like agreement, alignment, ensemble, or co-training should be tried. If there is high  $S$ , it is worth spending the time on multimodal methods that model more complex interactions based on tensors, multiplicative interactions, and self-attention.
We believe that the intersection of PID and machine learning opens the door to several exciting directions of future work:
1. Pointwise interaction measures: A natural extension is to design pointwise measures: how much does a single datapoint contribute to redundancy, uniqueness, or synergy in the context of an entire distribution? Pointwise measures could help for more fine-grained dataset and model quantification, including error analysis of incorrect predictions or active learning of difficult examples.
2. Representation learning for desired interactions: Can we design new model architectures or training objectives that better capture certain interactions? Current well-performing models like Transformers already include some notion of multiplicative interactions, so it does seem that, heuristically, synergy between input features is playing a role in their success. Designing models that better capture synergy, as quantified by our PID estimators, could be a path towards learning better representations.
3. Principled approaches to fairness and invariance: Currently, PID is designed to measure the information that 2 variables contribute towards a task  $Y$ , but conversely it can also be used to remove information that one variable can have about  $Y$ , in the context of another variable. These could provide formal learning objectives for fairness, privacy, and other feature invariance tasks.
E.1 Limitations of our Estimators
While our estimators appear to perform well in practice, each suffers from distinct limitations. For CVX, it is rarely able to scale up to domains with more than several thousand classes. Furthermore, if  $|\mathcal{X}_i|$  is large—which is frequently the case in real-world applications, then empirical frequencies will often be near 0, causing much instability in conic solvers (see Appendix B.1 for a more detailed description).
BATCH sidesteps many of the issues with CVX by applying batch gradient descent. However, it may suffer from approximation error depending on whether the network can sufficiently cover the space of joint distributions (i.e., representation capacity), alongside bias resulting from mini-batch gradient descent (as compared to full-batch gradient descent) when approximating  $\tilde{q}$  using bootstrapped samples. Lastly, it may suffer from the usual problems with neural estimators, such as local minima, poor or unstable convergence (in both learning  $\tilde{q}$  and  $\hat{p}$ ). Therefore, while BATCH scales to high-dimensional continuous distributions, it comes with the challenges involving training and tuning neural networks.
E.2 Broader Impact
Multimodal data and models are ubiquitous in a range of real-world applications. Our proposed framework based on PID is our attempt to systematically quantify the plethora of datasets and models currently in use. While these contributions will accelerate research towards multimodal datasets and models as well as their real-world deployment, we believe that special care must be taken in the following regard to ensure that these models are safely deployed:
Care in interpreting PID values: Just like with any approximate estimator, the returned PID values are only an approximation to the actual interactions and care should be taken to not overfit to these values. Other appropriate forms of dataset visualization and quantification should still be conducted to obtain holistic understanding of multimodal datasets.
Privacy, security, and biases: There may be privacy risks associated with making predictions from multimodal data if the datasets include recorded videos of humans or health indicators. In our experiments with real-world data where people are involved (i.e., healthcare and affective computing), the creators of these datasets have taken the appropriate steps to only access public data which participants/content creators have consented for released to the public. We also acknowledge the risk
of exposure bias due to imbalanced datasets that may cause biases towards certain genders, races, and demographic groups. Therefore, care should be taken in understanding the social risks of each multimodal dataset in parallel to understanding its interactions via PID.
Time & space complexity: Modern multimodal datasets and models, especially those pretrained on internet-scale data, may cause broader impacts resulting from the cost of hardware, electricity, and computation, as well as environmental impacts resulting from the carbon footprint required to fuel modern hardware. Future work should carefully investigate the role of size on the interactions learned by models through estimated PID values. Our preliminary experiments showed that smaller models could still capture high degrees of each interaction, which may pave away towards designing new inductive biases that enable interaction modeling while using fewer parameters.
Overall, PID offers opportunities to study the potential social and environmental issues in multimodal datasets by obtaining a deeper understanding of the underlying feature interactions, providing a path towards interpretable and lightweight models. We plan to continue expanding our understanding of PID via deeper engagement with domain experts and how they use this framework in their work. Our released datasets, models, and code will also present a step towards scalable quantification of feature interactions for future work.