SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. General observations on interactions, agreement, and confidence: The annotated interactions align with prior intuitions on these multimodal datasets and do indeed explain the interactions between modalities. For example, VQA and CLEVR are annotated with significantly high synergy, and language is annotated as the dominant modality in sentiment, humor, and sarcasm (high  $U_{1}$  values). Overall, Table 9 shows that the Krippendorff's alpha for inter-annotator agreement in human annotations of the interactions is quite high and the average confidence scores are also quite high. This indicates that the human-annotated results are reliable.
2. Annotator subjectivity in interpreting presence vs absence of an attribute and contextualizing both language and video: We observe that there is some confusion between uniqueness in the language modality and synergy in the video datasets, resulting in cases of low agreement in annotating  $U_{1}$  and  $S$ : -0.09, -0.07 for MOSEI, -0.14, -0.03 for UR-FUNNY and -0.08, -0.04 for MUSTARD respectively. We believe this is due to annotator subjectivity in interpreting whether sentiment, humor, and sarcasm are present only in the language modality ( $U_{1}$ ) or present when contextualizing both language and video ( $S$ ). We further investigated the agreement and confidence in the presence or absence of an attribute (e.g., humor or sarcasm). We examined videos that show and do not show an attribute separately and indeed found in general, humans reached higher agreement on annotating attribute-present videos. The agreement of annotating  $S$  is 0.13 when the attribute is present, compared to -0.10 when absent.
C.6 Comparisons with Other Interaction Measures
It is worth noting that other valid information-theoretic definitions of multimodal interactions also exist, but are known to suffer from issues regarding over- and under-estimation, and may even be negative [41]; these are critical problems with the application of information theory for shared
Table 10: Estimating PID on synthetic generative model datasets. Ground truth total information is computed based on an upper bound from the best multimodal test accuracy. Both CVX and BATCH measures agree with each other on relative values and are consistent with ground truth interactions. We also implement 3 other definitions: I-min can sometimes overestimate synergy and uniqueness; WMS is actually synergy minus redundancy, so can be negative and when R & S are of equal magnitude WMS cancels out to be 0; CI can also be negative and sometimes incorrectly concludes highest uniqueness for S-only data.
Task	R-only data	U1-only data	U2-only data	S-only data
PID	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S	R	U1	U2	S
I-MIN	0.17	0.08	0.07	0	0	0.23	0	0.06	0	0	0.25	0.08	0.07	0.03	0.04	0.17
WMS	0	0.20	0.20	-0.11	0	0.25	0.02	0.05	0	0.03	0.27	0.05	0	0.14	0.15	0.07
CI	0.34	-0.09	-0.10	0.17	0	0.23	0	0.06	0	0.01	0.25	0.07	-0.02	0.13	0.14	0.08
CVX	0.16	0	0	0.05	0	0.16	0	0.05	0	0	0.17	0.05	0.07	0	0.01	0.14
BATCH	0.29	0.02	0.02	0	0	0.30	0	0	0	0	0.30	0	0.11	0.02	0.02	0.15
Truth	0.58	0	0	0	0	0.56	0	0	0	0	0.54	0	0	0	0	0.56
$I(X_{1};X_{2};Y)$  and unique information  $I(X_{1};Y|X_{2})$ ,  $I(X_{2};Y|X_{1})$  often quoted in the co-training [14, 8] and multi-view learning [98, 94, 89] literature. We additionally compare PID with other previously used measures for feature interactions on the synthetic generative model datasets. Both CVX and BATCH measures agree with each other on relative values and are consistent with ground truth interactions. We also implement 3 other definitions: I-min can sometimes overestimate synergy and uniqueness; WMS is actually synergy minus redundancy, so can be negative and when R & S are of equal magnitude WMS cancels out to be 0; CI can also be negative and sometimes incorrectly concludes highest uniqueness for  $S$ -only data. These results are in Table 10. We also tried 3 non-info theory measures: Shapley values, Integrated gradients (IG), and CCA, which are based on quantifying interactions captured by a multimodal model. Our work is fundamentally different in that interactions are properties of data before training any models.
C.7 Quantifying Model Predictions
Setup: For each dataset, we first train a suite of models on the train set  $\mathcal{D}_{\mathrm{train}}$  and apply it to the validation set  $\mathcal{D}_{\mathrm{val}}$ , yielding a predicted dataset  $\mathcal{D}_{\mathrm{pred}} = \{(x_1, x_2, \hat{y}) \in \mathcal{D}_{\mathrm{val}}\}$ . Running PID on  $\mathcal{D}_{\mathrm{pred}}$  summarizes the interactions that the model captures. We categorize and implement a comprehensive suite of models (spanning representation fusion at different feature levels, types of interaction inductive biases, and training objectives) that have been previously motivated in the literature to capture certain types of interactions [62]. In the following, we use  $\mathbf{x}_i$  to denote data or extracted features from unimodal models where appropriate. We include all implementation details and hyperparameters in Table 11, 12, 13, 14, 15.
General: We first implement the most general model, which we call early fusion:
1. EF: Concatenating data at the earliest input level, essentially treating it as a single modality, and defining a suitable prediction model  $y = f([\mathbf{x}_1,\mathbf{x}_2])$  [62].
Redundancy: Fusion at the decision level (i.e., late fusion) is often motivated to capture redundancy between unimodal predictors.
1. ADDITIVE: Suitable unimodal models are first applied to each modality before aggregating the outputs using an additive average:  $y = 1 / 2(f_{1}(\mathbf{x}_{1}) + f_{2}(\mathbf{x}_{2}))$  [44].
2. AGREE: Prediction agreement  $(+\lambda (f_{1}(\mathbf{x}_{1}) - f_{2}(\mathbf{x}_{2}))^{2}$  [26]) can be added for redundancy.
3. ALIGN: Feature alignment  $(+\lambda \mathrm{sim}(\mathbf{x}_1,\mathbf{x}_2)$  like contrastive learning [82]) can also be added for redundancy.
Synergy: Exemplified by methods such as bilinear pooling [33] and tensor fusion [115], researchers have long sought to learn higher-order interactions across modalities to capture synergistic interactions. We implement the following models:
1. ELEM: For static interactions (i.e., without trainable interaction parameters), we implement element-wise interactions:  $y = f(\mathbf{x}_1 \odot \mathbf{x}_2)$  [2, 50].
2. TENSOR: We also implement outer-product interactions (i.e., higher-order tensors)  $y = f(\mathbf{x}_1\mathbf{x}_2^\top)$  [33, 115, 47, 58, 65].
3. MI: Dynamic interactions with learnable weights include multiplicative interactions  $\mathbb{W}$ :  $y = f(\mathbf{x}_1\mathbb{W}\mathbf{x}_2)$  [52].
4. MULT: Another example of dynamic interactions with learnable weights is Crossmodal self-attention typically used in multimodal Transformers:  $y = f(\text{softmax}(\mathbf{x}_1 \mathbf{x}_2^\top) \mathbf{x}_1)$  [103, 99, 113].
Unique: There have been relatively fewer methods that explicitly aim to capture uniqueness, since label information is often assumed to be shared across modalities (i.e., redundancy) [98, 94], two that we implement are:
1. LOWER: Some approaches explicitly include lower-order terms in higher-order interactions to capture unique information [65, 115].
2. REC: Finally, several methods include reconstruction objectives to encourage maximization of unique information (i.e., adding an objective  $\mathcal{L}_{\mathrm{rec}} = \| g_1(\mathbf{z}_{\mathrm{mm}}) - \mathbf{x}_1\| _2 +$ $\| g_{2}(\mathbf{z}_{\mathrm{mm}}) - \mathbf{x}_{2}\|_{2}$  where  $g_{1},g_{2}$  are auxiliary decoders mapping  $\mathbf{z}_{\mathrm{mm}}$  to each raw input modality [92, 100, 111].
Results: We show the full results on estimating PID for all models trained on all datasets in Table 18. To better understand the models' ability in capturing each type of interaction, we present a concise overview of the full results in Table 4, focusing on the PID values of each type of interaction  $I$  for each model on  $I$ -specialized dataset  $\mathcal{D}_I$ . We also compute the distribution of  $I_{\mathcal{D}_I}$  over all models. We highlight the following observations:
General observations: We first observe that model PID measures are consistently higher than dataset measures. The sum of model measures is also a good indicator of test performance, which agrees with their formal definition since their sum is equal to  $I(\{X_1, X_2\}; Y)$ , the total explained mutual information between multimodal data and  $Y$ .
On redundancy: Overall, several methods succeed in capturing redundancy, with an overall average of  $0.41 \pm 0.11$  and accuracy of  $73.0 \pm 2.0\%$  on redundancy-specialized datasets. Additive, agreement, and alignment-based methods are particularly strong, but other methods based on tensor fusion (synergy-based), including lower-order interactions, and adding reconstruction objectives (unique-based) also capture redundancy well.
On uniqueness: Uniqueness is harder to capture than redundancy, with an average of  $0.37 \pm 0.14$ . Certain methods like additive and agreement do poorly on uniqueness, while those designed for uniqueness (lower-order interactions and reconstruction objectives) do well, with  $U = 0.55$  and  $73.0\%$  accuracy on uniqueness datasets.
On synergy: On average, synergy is the hardest to capture, with an average score of only  $0.21 \pm 0.10$ . The best-performing methods are tensor fusion  $S = 0.33$ , acc  $= 74.0\%$  and multimodal transformer  $S = 0.29$ , acc  $= 73.0\%$ . Additive, agreement, and element-wise interactions do not seem to capture synergy well.
On robustness: Finally, we also show empirical connections between estimated PID values with model performance in the presence of noisy or missing modalities. Specifically, with the original performance  $Acc$  on perfect bimodal data, we define the performance drop when  $X_{i}$  is missing as  $Acc_{-i} = Acc - Acc_{i}$  where  $Acc_{i}$  is the unimodal performance when  $X_{i}$  is missing. We find high correlation  $(\rho = 0.8)$  between  $Acc_{-i}$  and the model's  $U_{i}$  value. Inspecting the graph closely in Figure 5, we find that the correlation is not perfect because the implication only holds in one direction: high  $U_{i}$  coincides with large performance drops, but low  $U_{i}$  can also lead to performance drops. We find that the latter can be further explained by the presence of large  $S$  values: when  $X_{i}$  is missing, these interactions can no longer be discovered by the model which decreases robustness. For the subset of points when  $U_{i} \leq 0.05$ , the correlation between  $S$  and performance drop is  $\rho = 0.73$  while the correlation between  $R$  and performance drop is only  $\rho = 0.01$ .
Table 11: Table of hyperparameters for prediction on synthetic data.
Component	Model	Parameter	Value
Encoder	Identity	/	/
Linear	Input size	[200, 200]
Hidden dim	512
Decoder	Linear	Input dim	[200, 200]
Hidden dim	512
Fusion	Concat	/	/
ElemMultWithLinear	Output dim	512
MI-Matrix [52]
LRTF [65]	Output dim	512
rank	32
MULT [99]	Embed dim	512
Num heads	8
Classification head	Identity	/	/
2-Layer MLP	Hidden size	512
Activation	LeakyReLU(0.2)
Dropout	0.1
Training	EF & ADDITIVE & ELEM & TENSOR	Loss	Cross Entropy
MI & MULT & LOWER	Batch size	128
Num epochs	100
Optimizer/Learning rate	Adam/0.0001
AGREE & ALIGN	Loss	Cross Entropy
Batch size	+ Agree/Align Weight
Num epochs	128
Optimizer/Learning rate	200
Cross Entropy Weight	Adam/0.0001
Agree/Align Weight	2.0
REC [100]	Loss	1.0
Batch size	+ Reconstruction (MSE)
Num epochs	128
Optimizer	100
Learning rate	Adam
Recon Loss Modality Weight	0.0001
Cross Entropy Weight	[1, 1]
Intermediate Modules	2.0MLP [512, 256, 256]MLP [512, 256, 256]
Table 12: Table of hyperparameters for prediction on affective computing datasets.
Component	Model	Parameter	Value
Encoder	Identity	/	/
GRU	Input size	[5,20,35,74,300,704]
Hidden dim	[32,64,128,512,1024]
Decoder	GRU	Input size	[5,20,35,74,300,704]
Hidden dim	[32,64,128,512,1024]
Fusion	Concat	/	/
ElemMultWithLinear	Output dim	[400,512]
MI-Matrix [52]
Tensor Fusion [115]	Output dim	512
MULT [99]	Embed dim	40
Num heads	8
Classification head	Identity	/	/
2-Layer MLP	Hidden size	512
Activation	LeakyReLU(0.2)
Dropout	0.1
Training	EF & ADDITIVE & ELEM & TENSOR	Loss	L1 Loss
Batch size	32
MI & MULT & LOWER	Num epochs	40
Optimizer/Learning rate	Adam/0.0001
AGREE & ALIGN	Loss	L1 Loss
Batch size	+ Agree/Align Weight
Num epochs	32
Optimizer/Learning rate	30
Agree/Align Weight	Adam/0.0001
REC [100]	Loss	0.1
Batch size	L1 Loss
Num epochs	+ Reconstruction (MSE)
Optimizer	128
Learning rate	50
Recon Loss Modality Weight	Adam
Intermediate Modules	0.001
[1,1]
MLP [600,300,300]
MLP [600,300,300]
Table 13: Table of hyperparameters for prediction on ENRICO dataset in the HCI domain.
Model	Parameter	Value
Unimodal	Hidden dim	16
MI-Matrix [52]	Hidden dim	32
Input dims	16, 16
MI	Hidden dim	32
Input dims	16, 16
LRTF [65]	Hidden dim	32
Input dims	16, 16
Rank	20
Training	Loss	Class-weighted Cross Entropy
Batch size	32
Activation	ReLU
Dropout	0.2
Optimizer	Adam
Learning Rate	10-5
Num epochs	30
Table 14: Table of hyperparameters for prediction on MIMIC dataset in the healthcare domain.
Component	Model	Parameter	Value
Static Encoder	2-layer MLP	Hidden sizes	[10, 10]
Activation	LeakyReLU(0.2)
Static Decoder	2-layer MLP	Layer sizes	[200, 40, 5]
Activation	LeakyReLU(0.2)
Time Series Encoder	GRU	Hidden dim	30
Time Series Decoder	GRU	Hidden dim	30
Classification Head	2-Layer MLP	Hidden size	40
Activation	LeakyReLU(0.2)
Fusion	LRTF [65]	Output dim	100
Ranks	40
MI-Matrix [52]	output dim	100
Training	TENSOR, MI	Loss	Cross Entropy
Batch size	40
Num epochs	20
Optimizer	RMSprop
Learning rate	0.001
REC [100]	Loss	Cross Entropy + Reconstruction(MSE)
Batch size	40
Num epochs	30
Optimizer	Adam
Learning Rate	0.001
Recon Loss Modality Weights	[1, 1]
Cross Entropy Weight	2.0
Intermediate Modules	MLPs [200, 100, 100], [200, 100, 100], [400, 100, 100]
Table 15: Table of hyperparameters for prediction on MAPS.
Component	Model	Parameter	Value
Encoder	Identity	/	/
Linear	Input size	[1000, 18, 137]
Hidden dim	512
Decoder	Linear	Input dim	[1000, 18, 137]
Hidden dim	512
Fusion	Concat	/	/
ElemMultWithLinear	Output dim	512
MI-Matrix [52]
LRTF [65]	Output dim	512
rank	32
MULT [99]	Embed dim	512
Num heads	8
Classification head	Identity	/	/
2-Layer MLP	Hidden size	512
Activation	LeakyReLU(0.2)
Dropout	0.1
Training	EF & ADDITIVE & ELEM & TENSOR	Loss	Cross Entropy
MI & MULT & LOWER	Batch size	128
Num epochs	100
Optimizer/Learning rate	Adam/0.0001
AGREE & ALIGN	Loss	Cross Entropy
Batch size	+ Agree/Align Weight
Num epochs	128
Optimizer/Learning rate	200
Cross Entropy Weight	Adam/0.0001
Agree/Align Weight	2.0
REC [100]	Loss	1.0
Batch size	+ Reconstruction (MSE)
Num epochs	128
Optimizer	100
Learning rate	Adam
Recon Loss Modality Weight	0.0001
Cross Entropy Weight	[1, 1]
Intermediate Modules	2.0
MLP [512, 256, 256]
MLP [512, 256, 256]
Table 16: Table of hyperparameters for prediction on Push dataset.
Component	Model	Parameter	Value
Pos Encoder	Linear	Hidden sizes	[64, 64, 64 (residual)]
Sensors Encoder	Linear	Hidden sizes	[64, 64, 64 (residual)]
Image Encoder	CNN	Filter sizes	[5, 3, 3, 3, 3]
Num filters	[32, 32, 32, 16, 8]
Filter strides	1
Filter padding	[2, 1, 1, 1, 1]
Control Encoder	Linear	Hidden sizes	[64, 64, 64 (residual)]
Fusion	Unimodal LSTM	Hidden size	512
Num layers	2
Late Fusion LSTM	Hidden size	256
Num layers	1
Classification Head	Linear	Hidden size	64
Training	Loss	Mean Squared Error
Batch size	32
Num epochs	20
Activation	ReLU
Optimizer	Adam
Learning rate	10-5
Table 17: Table of hyperparameters for prediction on AV-MNIST dataset.
Component	Model	Parameter	Value
Image Encoder	LeNet-3	Filter Sizes	[5,3,3,3]
Num Filters	[6,12,24,48]
Filter Strides / Filter Paddings	[1,1,1,1] / [2,1,1,1]
Max Pooling	[2,2,2,2]
Image Decoder	DeLeNet-3	Filter Sizes	[4,4,4,8]
Num Filters	[24,12,6,3]
Filter Strides / Filter Paddings	[2,2,2,4]/[1,1,1,1]
Audio Encoder	LeNet-5	Filter Sizes	[5,3,3,3,3,3]
Num Filters	[6,12,24,48,96,192]
Filter Strides / Filter Paddings	[1,1,1,1,1,1]/[2,1,1,1,1,1]
Max Pooling	[2,2,2,2,2,2]
Audio Decoder	DeLeNet-5	Filter Sizes	[4,4,4,4,4,8]
Num Filters	[96,48,24,12,6,3]
Filter Strides / Filter Paddings	[2,2,2,2,2,4]/[1,1,1,1,1,1,1]
Classification Head	2-Layer MLP	Hidden size	100
Activation	LeakyReLU(0.2)
Training	Unimodal, LF	Loss	Cross Entropy
Batch size	40
Num epochs	25
Optimizer/Learning rate/weight decay	SGD/0.05/0.0001
Table 18: Results on estimating PID on model families trained on synthetic datasets with controlled interactions.
Category	GENERAL	REDUNDANCY
Model	EF	ADDITIVE	AGREE
Measure	R	U1	U2	S	Acc	R	U1	U2	S	Acc	R	U1	U2	S	Acc
(zc)	0.47	0.01	0.01	0.04	0.74	0.35	0.2	0	0.04	0.71	0.48	0.01	0.01	0.06	0.74
(z1)	0	0.41	0	0.04	0.73	0.02	0.14	0	0.03	0.61	0.02	0.15	0	0.04	0.38
(z2)	0.01	0	0.47	0.04	0.72	0.05	0	0.43	0.06	0.72	0.01	0	0.48	0.04	0.72
(z1, z2)	0.15	0.02	0	0.29	0.72	0.15	0	0.2	0.13	0.56	0.14	0	0.26	0.09	0.66
(z1*, z2*, zc*)	0.22	0	0.07	0.15	0.64	0.31	0.16	0	0.08	0.64	0.21	0	0.11	0.15	0.58
(z1, z2*, zc*)	0.16	0.15	0	0.15	0.71	0.06	0	0.18	0.06	0.6	0.06	0	0.08	0.08	0.53
(z1, z2, zc*)	0.16	0.05	0	0.25	0.74	0.09	0	0.23	0.11	0.52	0.02	0	0.33	0.06	0.62
(z1*, z2*, zc)	0.25	0.03	0	0.14	0.66	0.39	0.11	0	0.08	0.67	0.44	0.04	0	0.09	0.67
(z2*, zc*)	0.19	0	0.12	0.06	0.65	0.15	0	0.07	0.18	0.61	0.13	0	0.24	0.04	0.64
(z2*, zc)	0.26	0	0.12	0.05	0.7	0.27	0	0.18	0.08	0.71	0.1	0	0.1	0.16	0.61
Category	REDUNDANCY	SYNERGY
Model	ALIGN	ELEM	TENSOR
Measure	R	U1	U2	S	Acc	R	U1	U2	S	Acc	R	U1	U2	S	Acc
(zc)	0.44	0.02	0	0.05	0.73	0.27	0	0.01	0.07	0.7	0.47	0.01	0.01	0.04	0.74
(z1)	0.03	0	0	0.05	0.48	0	0.2	0	0.05	0.66	0	0.56	0	0.02	0.74
(z2)	0.05	0	0.39	0.1	0.71	0.01	0	0.2	0.05	0.66	0.01	0	0.54	0.02	0.73
(z1, z2)	0.02	0	0.38	0.08	0.63	0.07	0.02	0	0.14	0.66	0.15	0.03	0	0.31	0.73
(z1*, z2*, zc*)	0.33	0.06	0	0.15	0.66	0.05	0	0.01	0.07	0.56	0.2	0.01	0.01	0.21	0.65
(z1, z2*, zc*)	0.21	0	0.05	0.14	0.66	0.04	0.05	0	0.08	0.61	0.09	0.24	0	0.13	0.71
(z1, z2, zc*)	0.14	0	0.26	0.04	0.64	0.08	0.02	0	0.12	0.66	0.17	0.06	0	0.27	0.72
(z1*, z2*, zc)	0.23	0	0.09	0.08	0.64	0.1	0	0.01	0.06	0.6	0.27	0.05	0.02	0.15	0.68
(z2*, zc*)	0.18	0.1	0	0.17	0.6	0.04	0	0.03	0.05	0.57	0.05	0	0.01	0.06	0.56
(z2*, zc)	0.16	0	0.2	0.06	0.69	0.15	0	0.03	0.06	0.65	0.14	0	0.03	0.05	0.65
Category	SYNERGY
Model	MI	MULT
Measure	R	U1	U2	S	Acc	R	U1	U2	S
(zc)	0.2	0	0.01	0.05	0.67	0.4	0.02	0	0.06
(z1)	0.01	0.13	0	0.05	0.66	0	0.48	0	0.02
(z2)	0.01	0	0.23	0.05	0.66	0.02	0	0.42	0.06
(z1, z2)	0.05	0.04	0	0.12	0.65	0.16	0.05	0	0.29
(z1*, z2*, zc*)	0.07	0.01	0.01	0.08	0.55	0.21	0.11	0	0.2
(z1, z2*, zc*)	0.02	0.01	0	0.06	0.58	0.1	0.29	0	0.11
(z1, z2, zc*)	0.04	0.01	0	0.08	0.62	0.18	0.02	0	0.3
(z1*, z2*, zc)	0.12	0.01	0.01	0.06	0.62	0.33	0	0.03	0.14
(z2*, zc*)	0.02	0	0.01	0.06	0.54	0.13	0	0.39	0.04
(z2*, zc)	0.13	0	0.01	0.06	0.64	0.36	0	0.18	0.04
Category	UNIQUE
Model	LOWER	REC
Measure	R	U1	U2	S	Acc	R	U1	U2	S
(zc)	0.53	0	0.01	0.03	0.75	0.55	0.02	0.01	0.02
(z1)	0	0.56	0	0.02	0.74	0	0.53	0	0.03
(z2)	0.01	0	0.54	0.02	0.72	0.01	0	0.55	0.02
(z1, z2)	0.15	0.03	0	0.32	0.74	0.14	0.06	0	0.34
(z1*, z2*, zc*)	0.21	0.01	0.01	0.2	0.65	0.19	0.03	0	0.26
(z1, z2*, zc*)	0.09	0.27	0	0.13	0.71	0.08	0.29	0	0.16
(z1, z2, zc*)	0.16	0.06	0	0.27	0.73	0.21	0.05	0	0.31
(z1*, z2*, zc)	0.31	0.01	0.01	0.16	0.68	0.32	0	0.06	0.21
(z2*, zc*)	0.13	0	0.29	0.04	0.64	0.19	0	0.36	0.04
(z2*, zc)	0.31	0	0.13	0.04	0.7	0.42	0	0.15	0.03
Table 19: Results on estimating PID on model families trained on real-world multimodal datasets.
Category	GENERAL	REDUNDANCY
Model	EF	ADDITIVE	AGREE
Measure	R	U1	U2	S	Acc	R	U1	U2	S	Acc	R	U1	U2	S	Acc
UR-FUNNY A,T	0.02	0	0.02	0.06	0.59	0.02	0	0.02	0.15	0.60	0.02	0	0.02	0.05	0.59
UR-FUNNY V,T	0.05	0.01	0	0.17	0.62	0.11	0.08	0	0.17	0.60	0.02	0.04	0	0.24	0.59
UR-FUNNY V,A	0.01	0.27	0	0.19	0.59	0.03	0	0.02	0.15	0.56	0.02	0	0.02	0.15	0.53
MOSEI A,T	0.16	0.27	0.03	0	0.79	0.15	0.19	0.02	0.13	0.80	0.14	0.11	0.05	0.06	0.80
MOSEI V,T	0.01	0	0	0	0.63	0.11	1.21	0	0	0.80	0.1	0.84	0	0	0.80
MOSEI V,A	0	0	0	0.01	0.63	0.59	0.62	0	0.12	0.65	0.10	1.46	0	0.24	0.61
MUSTARD A,T	0.06	0	0.03	0.04	0.42	0.17	0	0.04	0.33	0.70	0.18	0	0.05	0.31	0.68
MUSTARD V,T	0	0	0	0	0.57	0.14	0	0.07	0.33	0.69	0.19	0	0.07	0.27	0.67
MUSTARD V,A	0	0	0	0	0.57	0.25	0.12	0	0.15	0.61	0.33	0.10	0	0.16	0.58
MIMIC	0.01	0.27	0	0	0.91	0.01	0.27	0	0	0.92	0.01	0.27	0	0	0.92
ENRICO	0.71	0.34	0.44	0.38	0.50	0.69	0.28	0.39	0.35	0.30	0.40	0.30	0.52	0.75	0.51
Category	REDUNDANCY	SYNERGY
Model	ALIGN	ELEM	TENSOR
Measure	R	U1	U2	S	Acc	R	U1	U2	S	Acc	R	U1	U2	S	Acc
UR-FUNNY A,T	0.02	0	0.02	0.06	0.58	0.01	0	0.03	0.05	0.64	0.01	0	0.02	0.05	0.62
UR-FUNNY V,T	0.02	0	0.02	0.15	0.60	0.02	0.01	0	0.16	0.61	0.03	0.01	0	0.14	0.62
UR-FUNNY V,A	0.12	0.05	0.01	0.19	0.53	0.03	0.13	0	0.16	0.59	0.02	0.15	0	0.15	0.60
MOSEI A,T	0.12	0.26	0.14	0.16	0.65	0.09	1.07	0	0	0.80	0.18	0.17	0.06	0	0.80
MOSEI V,T	0.21	0.61	0	0.09	0.65	0.25	0.92	0	0.15	0.81	0.12	1.23	0	0	0.81
MOSEI V,A	1.15	0	0	0.79	0.40	0.14	1.21	0	0	0.65	0.12	1.07	0	0	0.65
MUSTARD A,T	0.20	0	0.06	0.29	0.70	0.17	0.01	0.11	0.23	0.60	0.20	0	0.13	0.25	0.59
MUSTARD V,T	0.16	0	0.08	0.29	0.70	0.18	0.01	0.04	0.32	0.64	0.18	0.01	0.02	0.34	0.59
MUSTARD V,A	0.27	0.12	0	0.14	0.62	0.36	0.07	0	0.16	0.55	0.33	0.05	0	0.18	0.60
MIMIC	0.01	0.28	0	0	0.91	0.04	0.24	0	0.01	0.91	0	0.28	0	0.01	0.91
ENRICO	0.37	0.34	0.57	0.76	0.52	0.3	0.43	0.29	0.73	0.44	0.38	0.48	0.32	0.69	0.50
Category	SYNERGY	UNIQUE
Model	MI	LOWER
Measure	R	U1	U2	S	Acc	R	U1	U2	S	Acc
UR-FUNNY A,T	0.01	0	0.03	0.04	0.62	0.02	0	0.01	0.14	0.60
UR-FUNNY V,T	0.04	0.02	0	0.14	0.64	0.04	0	0.01	0.15	0.62
UR-FUNNY V,A	0.02	0.23	0	0.15	0.61	0.02	0.15	0	0.19	0.58
MOSEI A,T	0.13	0.05	0.01	0.19	0.81	0.20	0.92	0	0.54	0.79
MOSEI V,T	0.11	0.98	0	0	0.80	0.17	1.14	0	0.08	0.80
MOSEI V,A	0.12	1.0	0	0	0.65	1.54	0.64	0	0.13	0.65
MUSTARD A,T	0.18	0	0.08	0.29	0.63	0.21	0	0.05	0.26	0.59
MUSTARD V,T	0.20	0.01	0.03	0.25	0.63	0.17	0	0.06	0.28	0.58
MUSTARD V,A	0.28	0.04	0.02	0.18	0.56	0.24	0.08	0	0.24	0.59
MIMIC	0.02	0.26	0	0.01	0.92	0.01	0.28	0	0.01	0.91
ENRICO	0.28	0.41	0.34	0.48	0.38	0.52	0.29	0.69	0.52	0.56
1. Dataset quantification:  $\mathcal{D} = \{(x_1,x_2,y)\} \longrightarrow \{R,U_1,U_2,S\}_{\mathcal{D}}$
2. Model quantification:  $f(\mathcal{D}) = \{(x_1,x_2,\hat{y} = f(x_1,x_2))\} \longrightarrow \{R,U_1,U_2,S\}_{f(\mathcal{D})}$ $\{R,U_1,U_2,S\}_{f(\mathcal{D}_k)},\dots ,\{R,U_1,U_2,S\}_{f(\mathcal{D}_k)}\longrightarrow \{R,U_1,U_2,S\}_{f}$
Figure 17: Model selection pipeline: (1) quantifying the interactions in different datasets, (2) quantifying interactions captured by different model families, then (3) selecting the closest model for a given new dataset yields models of  $>96\%$  performance as compared to exhaustively trying all models.
3. Model selection:  $\{R,U_1,U_2,S\}_{\mathcal{D}}\longleftrightarrow \{R,U_1,U_2,S\}_{f}$
C.8 Multimodal Model Selection
Setup: We show that our findings on quantifying multimodal datasets and model predictions are generalizable, and that the PID estimations are informative for model selection on new datasets without training all models from scratch. We simulate the model selection process on 5 new synthetic datasets and 6 real-world multimodal datasets. Given a new dataset  $\mathcal{D}$ , we compute their PID values  $I_{\mathcal{D}}$  and normalize it to obtain  $\hat{I}_{\mathcal{D}}$ :
$$
\hat {I} _ {\mathcal {D}} = \frac {I _ {\mathcal {D}}}{\sum I ^ {\prime} \in \{R , U _ {1} , U _ {2} , S \} I _ {\mathcal {D}} ^ {\prime}}. \tag {35}
$$
We select the most similar dataset  $\mathcal{D}^*$  from our base datasets (the 10 synthetic datasets presented in Table 2) that has the smallest difference measure
$$
s \left(\mathcal {D}, \mathcal {D} ^ {\prime}\right) = \sum_ {I \in \{R, U _ {1}, U _ {2}, S \}} \left| \hat {I} _ {\mathcal {D}} - \hat {I} _ {\mathcal {D} ^ {\prime}} \right| \tag {36}
$$