SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {i} \varphi_ {i} d _ {\mathcal {H} \Delta \mathcal {H}} (\mathbb {P} _ {i}, \mathbb {S}) \leq \max  _ {i, j} d _ {\mathcal {H} \Delta \mathcal {H}} (\mathbb {P} _ {i}, \mathbb {P} _ {j}). \tag {10}
$$
Then, for any  $h\in \mathcal{H}$
$$
\varepsilon_ {\mathbb {Q}} (h) \leq \lambda_ {\varphi} + \sum_ {i} \varphi_ {i} \varepsilon_ {\mathbb {P} _ {i}} (h) + \frac {1}{2} \min  _ {\mathbb {S} \in \mathcal {O}} d _ {\mathcal {H} \Delta \mathcal {H}} (\mathbb {S}, \mathbb {Q}) + \frac {1}{2} \max  _ {i, j} d _ {\mathcal {H} \Delta \mathcal {H}} (\mathbb {P} _ {i}, \mathbb {P} _ {j}), \tag {11}
$$
where  $\lambda_{\varphi}$  is the error of an ideal joint hypothesis,  $\varepsilon_{\mathbb{P}}(h)$  is the error for a hypothesis  $h$  on a distribution  $\mathbb{P}$ , and  $d_{\mathcal{H}\Delta \mathcal{H}}(\mathbb{P},\mathbb{Q})$  is  $\mathcal{H}$ -divergence which measures differences in distribution [6].
The proof of this theorem is provided in [60]. Here,  $\mathbb{Q}$  corresponds to the unseen out-of-distribution target domain and  $\{\mathbb{P}_i\}_{i=1}^K$  correspond to source domains.  $\lambda_{\varphi}$  is small in reality and often neglected. The term  $\sum_{i} \varphi_i \varepsilon_{\mathbb{P}_i}(h)$  is minimized by cross-entropy loss with class labels as supervision. The term  $\frac{1}{2} \max_{i,j} d_{\mathcal{H}\Delta \mathcal{H}}(\mathbb{P}_i, \mathbb{P}_j)$  measures the maximum differences among source domains. This corresponds to multi-modal supervised contrastive learning in our approach, where we map the modality-shared features of different modalities from all source domains to be as close as possible
if they have the same label. The term  $\frac{1}{2}\min_{\mathbb{S} \in \mathcal{O}} d_{\mathcal{H}\Delta \mathcal{H}}(\mathbb{S}, \mathbb{Q})$  demonstrates the importance of diverse source distributions [1], so that the unseen target  $\mathbb{Q}$  might be "near" to  $\mathcal{O}$ . Therefore, to enlarge the range of  $\mathcal{O}$ , we split the feature embedding of each modality into modality-specific and modality-shared parts by maximizing the  $\ell_2$  distance to preserve diversity.
Dataset. We use the EPIC-Kitchens dataset [16] and introduce a novel HAC dataset in this paper, which will be made publicly accessible for further research. We follow the experimental protocol used for the EPIC-Kitchens dataset in [52]. The EPIC-Kitchens dataset includes eight actions ('put', 'take', 'open', 'close', 'wash', 'cut', 'mix', and 'pour') recorded in three different kitchens, forming three separate domains D1, D2, and D3. Our HAC dataset consists of seven actions ('sleeping', 'watching tv', 'eating', 'drinking', 'swimming', 'running', and 'opening door') performed by humans, animals, and cartoon figures, forming three different domains H, A, and C. We collect 3381 video clips from the internet with around 1000 samples for each domain. We provide three modalities in our dataset: video, audio, and pre-computed optical flow. Some examples of the HAC dataset are shown in Fig. 1 (b) and more details are provided in the supplementary material.
Implementation Details. In our framework, we perform experiments on three modalities: video, audio, and optical flow. We adopt the MMAction2 [15] toolkit for experiments. To encode the visual information, we use SlowFast network [21] initialized with Kinetics-400 [35] pre-trained weights. For the audio encoder, we use ResNet-18 [29] and initialize the weights from the VGGSound pre-trained checkpoint [12]. Similarly, we use the SlowFast network with slow-only pathway and also Kinetics 400 [35] pre-trained weights for the optical flow encoder. The dimensions of the unimodal embedding  $\mathbf{E}$  for video, audio, and optical flow are 2304, 512, and 2048 correspondingly. For the projection network  $Proj(\cdot)$  in supervised contrastive learning, we instantiate it as a multi-layer perceptron with two hidden layers of size 2048 and output vector of size  $D_P = 128$ . We use a multi-layer perceptron with two hidden layers of size 2048 to instantiate the cross-modal translation  $MLP_{\mathbf{E}^i \to \mathbf{E}^j}$ . After obtaining the feature embedding from the encoder, we split the embedding into modality-shared features (the first half of the embedding) and modality-specific features (the remaining half). We use the Adam optimizer [39] with a learning rate of 0.0001 and a batch size of 16. The scalar temperature parameter  $\tau$  is set to 0.1. Additionally, we set  $\alpha_{con} = 3.0$ ,  $\alpha_{dis} = 0.7$ , and  $\alpha_{trans} = 0.1$ . We also analyze the sensitivity of different  $\alpha$  in the appendix. Finally, we train the network for 15 epochs on an RTX 2080 Ti GPU which takes about 20 hours and select the model with the best performance on the validation dataset. We report the Top-1 accuracy for all experiments.
Multi-modal Multi-source DG. Tab. 1 and Tab. 2 illustrate the results of SimMMDG under the
multi-modal multi-source DG setting, where we train on multiple source domains and test on one target domain. We first conduct experiments using video and audio modalities for experiments, as in [57]. We re-implement our framework employing the same I3D [10] and BN-Inception [32] backbone for video and audio as in RNA-Net [57] to ensure fair comparisons. The DeepAll approach implies that we feed all the data from source domains to the network without any domain generalization strategies. Tab. 1 shows that our SimMMDG significantly outperforms all the baselines. When we replace the backbone with SlowFast [21] and ResNet-18 [29],
Method	D2, D3 → D1	D1, D3 → D2	D1, D2 → D3	Mean
I3D backbone
DeepAll	43.19	39.35	51.47	44.67
IBN-Net [53]	44.46	49.21	48.97	47.55
Gradient Blending [69]	41.97	48.40	51.43	47.27
TBN [36]	42.35	47.45	49.20	46.33
AVSA [50]	42.78	47.38	51.79	47.32
Co-Attention [14]	40.87	43.57	54.88	46.44
SENet [30]	42.82	42.81	51.07	45.56
Non-Local [70]	45.72	43.08	49.49	46.10
MM-SADA [52]	39.79	52.73	51.87	48.13
RNA-Net [57]	45.65	51.64	55.88	51.06
SimMMDG (ours)	54.25	58.67	57.28	56.73
SlowFast backbone
DeepAll	47.13	55.73	57.17	53.34
MM-SADA [52]	49.20	60.40	59.14	56.25
RNA-Net [57]	52.18	59.47	60.88	57.51
SimMMDG (ours)	57.93	65.47	66.32	63.24
Table 1: Multi-modal multi-source DG on EPIC-Kitchens dataset using video and audio.
Method	Modality	EPIC-Kitchens dataset	HAC dataset
Video	Audio	Flow	D2, D3 → D1	D1, D3 → D2	D1, D2 → D3	Mean	A, C → H	H, C → A	H, A → C	Mean
DeepAll	✓	✓	47.13	55.73	57.17	53.34	66.55	72.85	45.77	61.72
MM-SADA [52]	✓	✓	49.20	60.40	59.14	56.25	65.47	72.52	44.30	60.76
RNA-Net [57]	✓	✓	52.18	59.47	60.88	57.51	60.20	73.95	48.90	61.02
SimMMDG (ours)	✓	✓	57.93	65.47	66.32	63.24	74.77	77.81	53.68	68.75
DeepAll	✓	✓	55.17	62.93	60.37	59.49	76.78	70.64	49.63	65.68
MM-SADA [52]	✓	✓	47.13	57.60	59.34	54.69	69.79	69.76	49.45	63.00
RNA-Net [57]	✓	✓	54.71	61.87	58.21	58.26	77.14	74.94	42.00	64.69
SimMMDG (ours)	✓	✓	59.31	63.33	62.73	61.79	79.31	77.04	51.29	69.21
DeepAll	✓	✓	45.28	56.40	57.08	52.92	50.04	59.71	38.97	49.57
MM-SADA [52]	✓	✓	47.36	53.47	60.27	53.70	46.58	61.81	39.15	49.18
RNA-Net [57]	✓	✓	45.74	57.73	56.47	53.31	52.05	64.13	40.35	52.18
SimMMDG (ours)	✓	✓	56.09	67.33	61.50	61.64	59.63	64.24	44.85	56.24
DeepAll	✓	✓	✓	55.63	59.20	58.01	57.61	69.07	71.30	51.47	63.95
MM-SADA [52]	✓	✓	✓	51.72	58.40	59.34	56.49	72.53	72.19	55.51	66.74
RNA-Net [57]	✓	✓	✓	52.41	57.20	60.16	56.59	69.00	73.40	51.65	64.68
SimMMDG (ours)	✓	✓	✓	63.68	70.13	67.76	67.19	77.65	79.03	56.62	71.10
Table 2: Multi-modal multi-source DG with different modalities on EPIC-Kitchens and HAC datasets.
Method	Source: Target:	EPIC-Kitchens dataset	HAC dataset
D1	D2	D3	H	A	C	Mean
D2	D3	D1	D3	D1	D2	A	C	H	C	H	A
DeepAll	52.27	51.75	44.60	54.11	48.05	56.67	51.24	66.11	43.01	63.45	37.68	46.86	58.94
MM-SADA [52]	50.67	50.10	51.49	56.57	42.99	54.00	50.97	65.89	37.22	57.75	40.90	49.82	62.91
RNA-Net [57]	45.60	46.30	43.68	57.39	49.66	55.87	49.75	65.67	42.92	61.72	38.69	47.66	61.59
SimMMDG (ours)	54.00	52.26	51.49	60.88	49.88	60.53	54.84	66.67	44.21	68.42	46.05	54.43	72.73
Table 3: Multi-modal single-source DG on EPIC-Kitchens and HAC datasets using video and audio.
the results further improve by a large margin (with an average improvement of up to  $5.73\%$ ) compared to the baselines. In the following experiments, we adopt SlowFast and ResNet-18 as our default backbones. To verify the generalization of our framework to different modalities, we conduct experiments by combining any two modalities, as well as all three modalities, and present the results in Tab. 2. Our SimMMDG outperforms all the baselines by a significant margin in all cases, with improvements of up to  $9.58\%$ . Notably, when we combine all three modalities, the performance further improves and surpasses that of any two modalities. In contrast, the baseline methods cannot achieve better results with more modalities, indicating that they fail to fully leverage the complementary information between modalities. Finally, we validate the performance of our framework on the HAC dataset, and the results are consistent with those obtained on the EPIC-Kitchens dataset, as demonstrated in Tab. 2. Our SimMMDG outperforms all the baselines by a significant margin in all cases, with improvements of up to  $7.73\%$ .
Multi-modal Single-source DG. The domain labels are not required in our SimMMDG framework. This feature makes our method readily applicable to single-source DG without modifications. In this setup, we train the model on a single source domain and test it on multiple target domains. Tab. 3 presents the results of SimMMDG under the multi-modal single-source DG setting using video and audio modalities. Despite being trained only on data from a single domain, our model demonstrates robust generalization to unseen domains, with an average improvement of up to  $5.71\%$ . In comparison, other baseline methods perform even worse than the DeepAll baseline, highlighting their limitations in the single-source DG setting. We present additional results obtained by exploring different combinations of modalities in the appendix.
Missing-modality DG. In real-world deployment scenarios, we cannot always guarantee that all modalities will be available. Hence, the network should be capable of handling missing-modality cases. One simple approach is to set the embedding of the missing modality to zero. We compare this with our proposed cross-modal translation replacement in Sec. 3.4. We retrain the cross-modal translation  $MLP_{\mathbf{E}^i \to \mathbf{E}^j}$  for 10 epochs using  $\mathcal{L}_{trans}$  in Eq. (4) and keep other parameters fixed. Tab. 4 reports on the comparison results on the EPIC-Kitchens dataset, where our solution yields significant benefits (up to  $10.47\%$  performance improvement) compared to zero-filling. By replacing the missing modality features with the translation ones, our approach also outperforms the unimodal model in most cases. In contrast, zero-filling hurts the network performance in some cases and is even worse than the unimodal model. The last six rows in Tab. 4 demonstrate that our approach is robust
Method	Video	Audio	Flow	D2, D3 → D1	D1, D3 → D2	D1, D2 → D3	Mean
DeepAll (Video-only)	✓	51.03	59.87	56.57	55.82
SimMMDG (zero-filling)	✓	X	55.40	64.00	58.32	59.24
SimMMDG (translation)	✓	X	57.24	65.07	59.65	60.65
SimMMDG (zero-filling)	✓	X	53.10	60.93	61.09	58.37
SimMMDG (translation)	✓	X	55.17	61.33	61.70	59.40
DeepAll (Audio-only)	✓	32.87	42.27	45.17	40.10
SimMMDG (zero-filling)	X	✓	29.20	37.33	45.69	37.41
SimMMDG (translation)	X	✓	37.70	46.40	52.05	45.38
SimMMDG (zero-filling)	✓	X	28.51	34.27	42.81	35.20
SimMMDG (translation)	✓	X	39.77	46.00	51.23	45.67
DeepAll (Flow-only)	✓	54.25	61.33	55.95	57.18
SimMMDG (zero-filling)	X	✓	56.32	60.67	56.57	57.85
SimMMDG (translation)	X	✓	56.32	62.13	56.98	58.48
SimMMDG (zero-filling)	X	✓	52.41	62.13	51.33	55.29
SimMMDG (translation)	X	✓	55.40	63.73	56.57	58.57
SimMMDG (zero-filling)	X	✓	✓	53.10	62.40	64.48	59.99
SimMMDG (translation)	X	✓	✓	55.86	68.27	64.58	62.90
SimMMDG (zero-filling)	✓	X	✓	62.76	66.80	57.49	62.35
SimMMDG (translation)	✓	X	✓	63.22	67.20	59.24	63.22
SimMMDG (zero-filling)	✓	✓	X	60.92	68.66	60.99	63.52
SimMMDG (translation)	✓	✓	X	62.53	68.13	61.60	64.09
SimMMDG (zero-filling)	✓	X	X	59.08	62.27	52.98	58.11
SimMMDG (translation)	✓	X	X	59.31	64.00	56.57	59.96
SimMMDG (zero-filling)	X	✓	X	33.79	37.47	44.56	38.61
SimMMDG (translation)	X	✓	X	38.39	44.53	47.74	43.55
SimMMDG (zero-filling)	X	X	✓	55.17	57.07	48.46	53.57
SimMMDG (translation)	X	X	✓	56.32	64.27	56.57	59.05
Table 4: Multi-modal multi-source DG with missing modalities on EPIC-Kitchens dataset.  $\times$  means the modality is available during training, but is missing in test time.
even when two modalities out of three are missing. We present more results on HAC dataset in the appendix.
Ablation on each proposed module. We conducted extensive ablation studies to investigate the role of each module of SimMMDG on EPIC-Kitchens dataset, as shown in Tab. 5. Incorporating the supervised contrastive learning loss alone resulted in noticeable improvements. However, the mean accuracy decreased when we integrated feature splitting without imposing any constraints. The results were further enhanced when we added the distance loss to promote diversity, and even more so when we incorporated the cross-modal translation module. Although using only supervised contrastive learning and cross-modal translation yielded satisfactory results, their performance is on average  $4.15\%$  lower than the complete approach with feature splitting and distance loss. These findings highlight the significance of segregating the feature embedding of each modality into modality-specific and modality-shared components. Although SimMMDG without supervised contrastive learning is already better than most baselines, it still has a performance gap compared with the whole framework, which means the cross-modal translation is helpful but cannot replace contrastive learning. The effect of contrastive learning is similar to explicit feature alignment. It aligns the modality-shared features of different modalities from different source domains with the same label to be as close as possible in the embedding space, while pushing away features with different labels, to make the embedding space more distinctive.
Comparison against unimodal DG. Tab. 6 presents the results in comparison to unimodal DG algorithms that exclusively rely on either video, audio, or optical flow inputs. We choose RSC [31], Mixup [72], and Fishr [59] as our baselines. By leveraging information from multiple modalities, our multi-modal DG framework delivers significant improvements (up to  $7.74\%$ ) in terms of performance as compared to unimodal DG methods.
Combine SimMMDG with other training strategies. SimMMDG can be seamlessly combined with other training strategies due to its generality and simplicity. We first combine SimMMDG with Gradient Blending [69], a strategy to improve multi-modal learning. The results are further improved compared to the original SimMMDG as shown in Tab. 7. We also combine SimMMDG with a Domain-adversarial Neural Network (DANN) [24] to align the features of source domains using
CL	FS	DL	CT	D2, D3 → D1	D1, D3 → D2	D1, D2 → D3	Mean
55.86	56.27	54.21	55.45
✓	52.18	61.60	62.22	58.67
✓	✓	51.49	62.53	63.24	59.09
✓	✓	51.72	62.67	58.93	57.77
✓	✓	✓	55.17	64.00	64.37	61.18
✓	✓	✓	54.25	63.47	43.04	60.25
✓	✓	✓	✓	57.93	65.47	66.32	63.24
Table 5: Ablations of each proposed module on EPIC-Kitchens dataset. CL: supervised contrastive learning, FS: feature splitting, DL: distance loss, CT: cross-modal translation.
Method	D2, D3 → D1	D1, D3 → D2	D1, D2 → D3	Mean
RSC (V)	50.11	62.53	58.73	57.12
RSC (A)	36.55	43.73	48.15	42.81
RSC (F)	55.17	63.33	59.65	59.38
Mixup (V)	49.20	59.73	59.96	56.30
Mixup (A)	35.17	40.80	45.07	40.35
Mixup (F)	56.32	65.60	54.62	58.85
Fishr (V)	53.79	63.47	61.09	59.45
Fishr (A)	37.47	44.80	47.43	43.23
Fishr (F)	54.25	63.87	59.14	59.09
SimMMDG (V+A+F)	63.68	70.13	67.76	67.19
Table 6: Comparison with single-modal DG methods on EPIC-Kitchens dataset. V: video, A: audio, F: optical flow.
Method	D2, D3 → D1	D1, D3 → D2	D1, D2 → D3	Mean
SimMMDG	57.93	65.47	66.32	63.24
+Gradient Blending	59.31	68.40	66.63	64.78
+DANN	60.69	66.69	64.58	64.07
Table 7: Combine SimMMDG with other training strategies on EPIC-Kitchens dataset.
Method	EPIC-Kitchens	Method	MUSTARD	UR-Funny
DeepAll	64.9	Late Fusion	61.6	63.6
TBN	73.2	LMF	65.2	63.9
Gradient Blending	74.0	MULT	60.9	63.2
SimMMDG	76.4	SimMMDG	72.5	65.6
Table 8: Evaluation under multi-modal classification setup.
domain labels and observe a similar improvement as with Gradient Blending. This indicates that our SimMMDG can be easily combined with other training strategies to get even better results.
SimMMDG as a general framework for multi-modal classification. Here, we evaluate our framework in a more general multi-modal classification setup without DG. We first evaluate our framework from three domains, we partition the training, validation, and testing data to ensure that no apparent domain shifts exit between the training and testing data. We compare our method with DeepAll, TBN [36], and Gradient Blending [69]. The results, as presented in Tab. 8 demonstrate that SimMMDG also exhibits significant advantages in the general multi-modal classification setup.
We further evaluate our framework on two multi-modal datasets: MUSTARD [11] and UR-FUNNY [27], both available in MultiBench [45]. These datasets pertain to human sentiment analysis and encompass language, video, and audio modalities. We conduct our experiments using the Multi-Bench codebase and implement SimMMDG within that environment. MultiBench treats human sentiment analysis as a regression task, whereas our framework is tailored for classification. To align the codebase with our classification task, we made the necessary modifications. For all baselines, we apply the same backbone model and solely change the fusion paradigms. In our comparisons, we evaluate our method against Late Fusion, Low-rank Tensor Fusion (LMF) [48], and Multimodal Transformer Fusion (MULT) [62]. The results reveal that our SimMMDG exhibits clear advantages, outperforming the baselines with an average improvement of  $7.3\%$  and  $1.7\%$ . This suggests that our SimMMDG serves as a versatile framework for multi-modal classification tasks and is compatible with various combinations of modalities, such as video+audio+flow and language+video+audio.
In this paper, we propose the SimMMDG framework for multi-modal DG. Our approach involves splitting the features of each modality into modality-specific and modality-shared parts and enforcing constraints on each part using supervised contrastive learning, distance loss, and cross-modal translation. The cross-modal translation module can also be applied in the case of missing-modality generalization. Our experiments on different datasets demonstrate the effectiveness of SimMMDG. Furthermore, we introduce a new challenging multi-modal dataset that can serve as a benchmark and guide future research in multi-modal DG problems.
Limitations. Currently, the number of cross-modal translation MLP in our framework is  $\mathcal{O}(|M|^2)$  and will be complex with the increase of modalities. In future work, the encoder-decoder network proposed in [19] can be used to reduce the complexity to  $\mathcal{O}(|M|)$ .
The authors acknowledge the support of "In-service diagnostics of the catenary/pantograph and wheelset axle systems through intelligent algorithms" (SENTINEL) project, supported by the ETH Mobility Initiative.
[72] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[73] Yunhua Zhang, Hazel Doughty, Ling Shao, and Cees G.M. Snoek. Audio-adaptive activity recognition across video domains. In CVPR, 2022.
[74] Han Zhao, Chen Dan, Bryon Aragam, Tommi S Jaakkola, Geoffrey J Gordon, and Pradeep Ravikumar. Fundamental limits and tradeoffs in invariant representation learning. Journal of Machine Learning Research, 23(340):1-49, 2022.
[75] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain-adversarial image generation for domain generalisation. In AAAI, 2020.
Proof of Theorem 1. Consider the joint mutual information  $I(\mathbf{E}^1, \dots, \mathbf{E}^M; y)$ . By the chain rule, we have the following decompositions:
$$
\begin{array}{l} I (\mathbf {E} ^ {1}, \dots , \mathbf {E} ^ {M}; y) = I (\mathbf {E} ^ {1}; y) + I (\mathbf {E} ^ {2}, \dots , \mathbf {E} ^ {M}; y \mid \mathbf {E} ^ {1}) \\ = \dots \\ = I \left(\mathbf {E} ^ {M}; y\right) + I \left(\mathbf {E} ^ {1}, \dots , \mathbf {E} ^ {M - 1}; y \mid \mathbf {E} ^ {M}\right). \\ \end{array}
$$
However, since  $\mathbf{E}^i$  are perfectly aligned,  $I(\mathbf{E}^2,\dots,\mathbf{E}^M;y\mid \mathbf{E}^1) = \dots = I(\mathbf{E}^1,\dots,\mathbf{E}^{M - 1};y\mid \mathbf{E}^M) = 0$ , which means  $I(\mathbf{E}^1,\dots,\mathbf{E}^M;y) = I(\mathbf{E}^1;y) = \dots = I(\mathbf{E}^M;y)$ . On the other hand, by the data processing inequality [5], we know that
$$
I (\mathbf {E} ^ {1}; y) \leq I (x _ {1}; y), \dots , I (\mathbf {E} ^ {M}; y) \leq I (x _ {M}; y).
$$
Hence, the following chain of inequalities holds:
$$
\begin{array}{l} I (\mathbf {E} ^ {1}, \dots , \mathbf {E} ^ {M}; y) = \min  \{I (\mathbf {E} ^ {1}; y), \dots , I (\mathbf {E} ^ {M}; y) \} \\ \leq \min  \left\{I \left(x _ {1}; y\right), \dots , I \left(x _ {M}; y\right) \right\} \\ \leq \max  \left\{I \left(x _ {1}; y\right),..., I \left(x _ {M}; y\right) \right\} \\ \leq I (x _ {1}, \dots , x _ {M}; y), \\ \end{array}
$$
where the last inequality follows from the fact that the joint mutual information  $I(x_{1},\dots,x_{M};y)$  is at least as large as any one of  $I(x_{i};y)$ . We use  $H(y\mid x_1,\ldots ,x_M)$  to denote the conditional entropy of  $y$  given  $x_{1},\ldots ,x_{M}$  as input. According to [74, 20] we have the following variational form:  $H(y\mid x_1,\dots,x_M) = \inf_f\mathbb{E}_p[\ell_{\mathrm{CE}}(f(x_1,\dots,x_M),y)]$ , where the infimum is over all the prediction functions that take  $x_{1},\ldots ,x_{M}$  as input to predict the target  $y$  and the expectation is taken over the joint distribution  $p$  of  $(x_{1},\dots,x_{M},y)$ . Therefore, due to the variational form of the conditional entropy, we have
$$
\begin{array}{l} \inf _ {f} \mathbb {E} _ {p} [ \ell_ {\mathrm {C E}} (f (\mathbf {E} ^ {1}, \dots , \mathbf {E} ^ {M}), y) ] - \inf _ {f ^ {\prime}} \mathbb {E} _ {p} [ \ell_ {\mathrm {C E}} (f ^ {\prime} (x _ {1}, \dots , x _ {M}), y) ] \\ = H (y \mid \mathbf {E} ^ {1}, \dots , \mathbf {E} ^ {M}) - H (y \mid x _ {1}, \dots , x _ {M}) \\ = H (y) - H (y \mid x _ {1}, \dots , x _ {M}) - (H (y) - H (y \mid \mathbf {E} ^ {1}, \dots , \mathbf {E} ^ {M})) \\ = I \left(x _ {1}, \dots , x _ {M}; y\right) - I \left(\mathbf {E} ^ {1}, \dots , \mathbf {E} ^ {M}; y\right) \\ \geq \max  \left\{I \left(x _ {1}; y\right), \dots , I \left(x _ {M}; y\right) \right\} - \min  \left\{I \left(x _ {1}; y\right), \dots , I \left(x _ {M}; y\right) \right\} \\ = \max  \left\{I \left(x _ {i}; y\right); i = 1, \dots , M \right\} - \min  \left\{I \left(x _ {i}; y\right); i = 1, \dots , M \right\} \\ \mathbf {\Delta} = \Delta_ {p}. \\ \end{array}
$$
B More Details on HAC Dataset
Our Human-Animal-Cartoon (HAC) dataset consists of seven actions ('sleeping', 'watching tv', 'eating', 'drinking', 'swimming', 'running', and 'opening door') performed by humans, animals, and cartoon figures, forming three different domains. We collect 3381 video clips from the internet with around 1000 for each domain and provide three modalities in our dataset: video, audio, and pre-computed optical flow. The dense optical flow is extracted at 24 frames per second using the TV-L1 algorithm [71].