SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our dataset was collected by 5 volunteers. For the human domain, we collect the data by selecting actions from an existing Kinetics-600 dataset [9]. We select approximately the same number of video clips for each action to ensure class balance. For the animal domain, we use the available 200 video clips in [73] and extend it to 906. We collect data from YouTube by searching keywords like 'animal sleeping', 'animal eating', 'animal running', etc. To increase the diversity of the dataset, we also specify the animal type in the keywords, such as 'cat sleeping', 'dog eating', 'horse running', etc. Each participant was asked to collect certain actions, such as Participant A for 'sleeping' and 'watching tv', Participant B for 'eating' and 'drinking', etc. For the cartoon domain, we collect all data from scratch and we collect data from popular cartoons like 'SpongeBob SquarePants', 'The Simpsons', 'Garfield and Friends', etc. Each participant was asked to collect from one or two cartoon
Figure 3: The number of each action segments for our HAC dataset.
Domain	Human	Animal	Cartoon
Training Action Segments	1111	730	870
Validation Action Segments	276	176	218
Testing Action Segments	1387	906	1088
Table 9: Number of action segments per domain.
series to avoid duplication. For each action, we annotated the start and end times in the video and then cut out video clips. The length of each video clip varies from 1 s to  $10\mathrm{s}$ . Finally, we gathered the data from all volunteers and a separate person manually discarded unqualified data like duplicate videos, videos without audio data, noisy/wrong classes, etc.
Fig. 3 and Tab. 9 illustrate the statistics of our HAC dataset. During the training phase, we train our model on one or two domains by utilizing the training action segments specific to those domains. The model selection is based on the performance of the validation action segments. Subsequently, we evaluate the model on the remaining domain, utilizing all the segments within that domain. Therefore, the total number of testing action segments is the sum of the training and validation segments. It is observed that each class within the HAC dataset exhibits a good balance across all domains. In contrast, EPIC-Kitchens dataset suffers from severe class imbalance. This implies that our proposed framework is capable of delivering high-performance results across both balanced and imbalanced datasets. Fig. 4 presents additional examples from our HAC dataset, highlighting the substantial domain shifts present in our dataset, particularly in the cartoon domain. The main purpose of this dataset is to be used for multi-modal domain generalization research. Of course, our dataset can also be used for other applications like multi-modal learning and multi-modal domain adaptation.
C Additional Implementation Details
We follow RNA-Net [57] to select the baseline methods. MM-SADA [52] is a domain adaption method and we only use the self-supervision loss without adversarial alignment as we have no access on target domain data during training. From the results in Table 1 in main paper, we know RNA-Net [57] and MM-SADA [52] are the two best baseline methods. Therefore, we select them together with DeepAll as baselines in the following experiments.
Figure 4: More examples on our HAC dataset.
eepnne eae nnnnne
Distance Type	D2, D3 → D1	D1, D3 → D2	D1, D2 → D3	Mean
Cosine	53.56	60.80	59.65	58.00
l1 norm	55.86	62.27	58.73	58.95
l2 norm	57.93	65.47	66.32	63.24
Table 10: Ablation of different distance types in distance loss on EPIC-Kitchens dataset.
D Further Ablations
Ablations of different distance types. Tab. 10 shows the ablations of different distance types in distance loss  $\mathcal{L}_{dis}$ . We explore the effects of using  $\ell_2$  norm,  $\ell_1$  norm, and Cosine similarity. Our experimental results indicate that all three distance types outperform the baselines, with  $\ell_2$  norm achieving the highest performance.
Parameter Sensitivity. We investigate the sensitivity of our method to the hyperparameters in the loss function. We perform this analysis by varying one parameter while fixing the others, and present our findings in Fig. 5. The results demonstrate that our method consistently outperforms the DeepAll baseline for all parameter settings, indicating that our approach is less sensitive to hyperparameter choices. However, we can also observe  $\alpha_{dis}$  is a little sensitive and may need attention for tuning in real applications.
Figure 5: Parameter sensitivity (D1, D2 → D3 in EPIC-Kitchens for multi-modal multi-source DG).
Method	Modality	Source: D1	Source: D2	Source: D3	Mean
Video	Audio	Flow	D1→D2	D1→D3	D2→D1	D2→D3	D3→D1	D3→D2
DeepAll	✓	✓	53.07	47.74	49.66	56.98	52.18	55.73	52.56
MM-SADA [52]	✓	✓	49.20	48.87	51.26	59.03	44.60	56.40	51.56
RNA-Net [57]	✓	✓	53.20	49.79	52.18	58.11	47.13	55.47	52.65
SimMMDG (ours)	✓	✓	56.80	54.11	53.10	59.86	50.57	64.27	56.45
DeepAll	✓	✓	41.60	47.43	41.61	51.75	45.98	56.13	47.42
MM-SADA [52]	✓	✓	42.53	47.64	42.76	50.92	42.07	51.87	46.30
RNA-Net [57]	✓	✓	46.40	49.38	42.76	54.62	43.45	52.13	48.12
SimMMDG (ours)	✓	✓	50.00	49.49	45.98	56.26	51.95	63.87	52.93
DeepAll	✓	✓	✓	51.60	53.39	44.60	60.16	45.74	53.47	51.49
MM-SADA [52]	✓	✓	✓	49.43	48.53	38.16	48.46	44.14	51.07	46.63
RNA-Net [57]	✓	✓	✓	51.20	53.90	46.67	58.73	49.43	57.47	52.90
SimMMDG (ours)	✓	✓	✓	55.33	54.83	50.80	59.45	53.33	64.40	56.36
Table 11: Multi-modal single-source DG with different modalities on EPIC-Kitchens dataset.
Method	Modality	Source: H	Source: A	Source: C	Mean
Video	Audio	Flow	H→A	H→C	A→H	A→C	C→H	C→A
DeepAll	✓	✓	59.82	41.08	73.25	35.02	60.71	64.46	55.72
MM-SADA [52]	✓	✓	66.23	46.23	69.65	42.46	59.77	57.62	56.99
RNA-Net [57]	✓	✓	58.61	37.87	73.90	48.16	60.35	59.38	56.38
SimMMDG (ours)	✓	✓	63.13	44.58	74.62	51.38	70.15	61.59	60.91
DeepAll	✓	✓	59.93	35.29	50.90	35.20	32.73	39.85	42.32
MM-SADA [52]	✓	✓	55.08	32.90	52.85	34.10	29.70	44.48	41.52
RNA-Net [57]	✓	✓	56.51	30.24	53.50	33.46	36.91	40.40	41.84
SimMMDG (ours)	✓	✓	62.25	36.40	56.60	38.79	38.72	48.12	46.81
DeepAll	✓	✓	✓	62.58	42.56	67.34	46.05	53.86	59.49	55.31
MM-SADA [52]	✓	✓	✓	62.25	44.67	60.78	43.11	41.82	44.81	49.57
RNA-Net [57]	✓	✓	✓	66.89	45.40	66.62	46.88	53.64	61.37	56.80
SimMMDG (ours)	✓	✓	✓	68.76	47.98	70.94	49.26	65.18	62.69	60.80
Table 12: Multi-modal single-source DG with different modalities on HAC dataset.
Multi-modal Single-source DG. Tab. 11 and Tab. 12 show more results under multi-modal single-source DG setting using the combination of different modalities. Our model demonstrates robust generalization to unseen domains compared to the baseline methods in most cases with a large margin up to  $4.81\%$ .
Missing-modality DG. Tab. 13 shows more results under the missing-modality DG setting on HAC dataset. Our solution yields significant benefits compared to zero-filling in most cases, similar to findings in the EPIC-Kitchens dataset. Our approach is also robust even when two modalities out of three are missing. We also validate the effectiveness of our cross-modal translation module on DeepAll, MM-SADA [52], and RNA-Net [57], as shown in Tab. 14. Compared to zero-filling,
Method	Video	Audio	Flow	A, C → H	H, C → A	H, A → C	Mean
DeepAll (Video-only)	✓	75.20	74.83	38.88	62.97
SimMMDG (zero-filling)	✓	X	73.97	75.94	51.10	67.00
SimMMDG (translation)	✓	X	77.51	77.15	53.03	69.23
SimMMDG (zero-filling)	✓	X	79.60	76.93	50.09	68.87
SimMMDG (translation)	✓	X	79.67	77.15	51.29	69.37
DeepAll (Audio-only)	✓	29.99	37.75	23.35	30.36
SimMMDG (zero-filling)	X	✓	25.16	26.05	15.99	22.40
SimMMDG (translation)	X	✓	28.26	39.74	22.15	30.05
SimMMDG (zero-filling)	✓	X	31.36	38.63	26.65	32.21
SimMMDG (translation)	✓	X	33.81	40.95	27.02	33.93
DeepAll (Flow-only)	✓	54.87	56.62	40.25	50.58
SimMMDG (zero-filling)	X	✓	54.79	51.43	40.44	48.89
SimMMDG (translation)	X	✓	59.12	60.38	41.18	53.56
SimMMDG (zero-filling)	X	✓	57.75	60.38	44.12	54.08
SimMMDG (translation)	X	✓	58.54	61.26	42.65	54.15
SimMMDG (zero-filling)	X	✓	✓	47.87	51.21	41.45	46.84
SimMMDG (translation)	X	✓	✓	52.05	61.70	41.08	51.61
SimMMDG (zero-filling)	✓	X	✓	80.32	79.36	55.88	71.85
SimMMDG (translation)	✓	X	✓	79.24	78.26	53.49	70.33
SimMMDG (zero-filling)	✓	✓	X	76.64	79.14	53.31	69.70
SimMMDG (translation)	✓	✓	X	77.58	79.91	55.88	71.12
SimMMDG (zero-filling)	✓	X	X	80.39	79.03	52.67	70.70
SimMMDG (translation)	✓	X	X	79.60	78.48	54.04	70.71
SimMMDG (zero-filling)	X	✓	X	22.71	32.45	23.62	26.26
SimMMDG (translation)	X	✓	X	30.28	41.83	29.60	33.90
SimMMDG (zero-filling)	X	X	✓	56.02	57.40	38.97	50.80
SimMMDG (translation)	X	X	✓	57.25	61.04	39.15	52.48
Table 13: Multi-modal multi-source DG with missing modalities on HAC dataset.  $\mathcal{X}$  means the modality is available during training, but is missing in test time.
Method	Video	Audio	D2, D3 → D1	D1, D3 → D2	D1, D2 → D3	Mean
DeepAll (Video-only)	✓	51.03	59.87	56.57	55.82
DeepAll (zero-filling)	✓	X	45.98	53.73	52.87	50.86
DeepAll (translation)	✓	X	47.82	54.27	54.83	52.31
MM-SADA [52] (zero-filling)	✓	X	48.97	58.00	58.52	55.16
MM-SADA [52] (translation)	✓	X	50.80	58.27	58.93	56.00
RNA-Net [57] (zero-filling)	✓	X	52.41	54.80	53.08	53.43
RNA-Net [57] (translation)	✓	X	54.25	58.13	55.85	56.08
SimMMDG (zero-filling)	✓	X	55.40	64.00	58.32	59.24
SimMMDG (translation)	✓	X	57.24	65.07	59.65	60.65
DeepAll (Audio-only)	✓	32.87	42.27	45.17	40.10
DeepAll (zero-filling)	X	✓	37.70	37.47	46.71	40.63
DeepAll (translation)	X	✓	38.39	42.13	50.10	43.54
MM-SADA [52] (zero-filling)	X	✓	36.09	40.13	46.51	40.91
MM-SADA [52] (translation)	X	✓	37.01	41.33	47.95	42.10
RNA-Net [57] (zero-filling)	X	✓	35.86	40.27	46.71	40.95
RNA-Net [57] (translation)	X	✓	39.31	42.53	48.97	43.60
SimMMDG (zero-filling)	X	✓	29.20	37.33	45.69	37.41
SimMMDG (translation)	X	✓	37.70	46.40	52.05	45.38
Table 14: Multi-modal multi-source DG with missing modalities on EPIC-Kitchens dataset for different baselines.  $\mathbf{X}$  means the modality is available during training, but is missing in test time.
replacing the features of missing modalities with the translated ones from other available modalities achieves better performance.
Statistical Significance Tests. We run each experiment three times using different seeds for multimodal multi-source DG on the EPIC-Kitchens dataset and then calculate the mean and standard deviation to show the statistical significance of our methods. As shown in Tab. 15, our framework is statistically stable and surpasses the baselines significantly.
Method	D2, D3 → D1	D1, D3 → D2	D1, D2 → D3	Mean
DeepAll	48.81 ± 1.77	55.78 ± 1.14	57.45 ± 1.19	54.01 ± 0.58
MM-SADA [52]	48.89 ± 0.43	59.15 ± 1.10	59.31 ± 0.81	55.79 ± 0.69
RNA-Net [57]	51.26 ± 3.07	58.54 ± 0.82	59.14 ± 1.24	56.31 ± 1.22
SimMMDG (ours)	57.70 ± 0.50	67.33 ± 0.99	64.41 ± 1.36	63.15 ± 0.88
Table 15: Statistical significance tests for multi-modal multi-source DG on EPIC-Kitchens dataset using video and audio.
(a) Embedding of Video
(b) Embedding of Audio
(d) Splitted Embedding of Video
Figure 6: Visualization of the learned embeddings using t-SNE (D1, D2  $\rightarrow$  D3 in EPIC-Kitchens for multi-modal multi-source DG).
(c) Embedding of Video and Audio
(e) Splitted Embedding of Audio
F Visualization
We show more visualizations of the learned embeddings using t-SNE [64] in Fig. 6. We can observe that the embeddings of video, audio, and the concatenation of video and audio are all well aligned for the source and target domains. Besides, the modality-specific and modality-shared embeddings are also well-separated and aligned across domains.
G More Intuitions behind Cross-modal Translation Module
In our SimMMDG framework, we introduce a cross-modal translation module to further regularize the learned features and facilitate missing-modality generalization. Here we give more intuitions behind this module.
Cross-modal translation won't undermine the unique features of different modalities. We want to learn an MLP projection to translate the embedding  $\mathbf{E}^i$  of the  $i$ -th modality to the embedding  $\mathbf{E}^j$  of the  $j$ -th modality. We add a translation loss to make the translated embedding  $\mathbf{E}_t^j$  to be close to  $\mathbf{E}^j$ , without any explicit alignment or constraints on  $\mathbf{E}^i$  and  $\mathbf{E}^j$ . We apply the cross-modal translation on the integrated feature of each modality  $\mathbf{E}^i = [\mathbf{E}_s^i; \mathbf{E}_c^i]$ , which is the concatenation of modality-specific feature  $\mathbf{E}_s$  and modality-shared feature  $\mathbf{E}_c$ . We still enforce a distance loss on  $\mathbf{E}_s$  and  $\mathbf{E}_c$  at the same time. Therefore, the modality-specific and modality-shared features are still forced to be separated during the training progress. The embedding visualizations shown in Fig. 6 (d) and (e) also indicate that for both video and audio, their modality-specific and modality-shared features are well disentangled and are not influenced by the cross-modal translation module.
R@1	R@5	R@10
Modality-specific Features	11.80	45.37	49.78
Modality-shared Features	76.58	90.13	92.83
Table 16: Video to Audio Retrieval.
R@1	R@5	R@10
Modality-specific Features	10.24	33.01	47.10
Modality-shared Features	67.59	89.60	93.99
Table 17: Audio to Video Retrieval.
Cross-modal translation is a type of modalities interaction, where different modality elements interact to give rise to new information when integrated together for task inference [46]. Several works [23, 55] have already demonstrated that modality interaction can help improve the performance of multi-modal tasks. For example, the proposed approach in [23] learns a coordinated similarity space between image and text to improve image classification. The approach proposed in [55] translates language into video and audio for language sentiment analysis.
Cross-modal translation can be thought of as a means to leverage the information from one modality to infer as much information as possible for the target modality. Just like when we hear a dog barking, we will fill in the picture of the puppy in our mind, and when we see a foreign language, we will automatically translate it into our native language. Although there is information loss and we can't recover all the details during this translation progress, we can still infer some useful information for the target modality. As shown in our ablation study in Tab. 5, adding the cross-modal translation module indeed improves multi-modal DG performance.
More importantly, our cross-modal translation module can be used for improving missing-modality generalization, by filling in the features of missing modality with the features inferred/translated from the available modality. The benefit of adopting cross-modal translation is demonstrated in Tab. 4. Our approach is robust even when two modalities out of three are missing. This indicates that the inferred information from the cross-modal translation module is valuable and useful for downstream tasks.
H Further Discussions on Modality-specific and Modality-shared Features
Based on our assumptions in the paper, modality-shared features reflect shared information between all modalities, like all modalities that describe the same people, objects, actions, or gestures. Modality-specific features are specific pieces of information that are unique to each modality, like texture, depth, and visual appearance in images, syntactic structure, vocabulary, and morphology in language. Our goal is to align the modality-shared features of different modalities from different source domains with the same label to be as close as possible in the embedding space, while pushing away features with different labels, to make the embedding space more distinctive. At the same time, we want the modality-specific features to be as far as possible from the modality-shared features, such that they carry different types of information.
We analyzed the meaningfulness and the ability to share features of modality-shared features by cross-modal retrieval task. The ability of cross-modal retrieval is highly related to the shareable of features, as only features that are meaningful, shareable, and highly connective can give a good recall rate. We use modality-shared features of videos to retrieve from the modality-shared features of audios and calculate the recall, and vice versa for audios. We report the R@1, R@5, and R@10 values for Video to Audio Retrieval and Audio to Video Retrieval. For example, R@5 for Video to Audio Retrieval means we retrieve 5 candidates from audio, if at least one of them has the same label as the query video, we consider the retrieval success and then we calculate the average success rate for all query videos. We also do the same thing on modality-specific features: using modality-specific features of videos to retrieve from the modality-specific features of audios, and vice versa for audios. As shown in Tab. 16 and Tab. 17, we have a very high recall rate when we use modality-shared features, while the recall rate is biased towards random when we use modality-specific features. This further indicates that the modality-shared features are truly shareable and meaningful and there are very strong relations and connections across modalities. The modality-specific features are instead with more information that is private to each single modality.
To further verify the meaningfulness of modality-shared features, we also train a classifier only on the concatenation of modality-shared features of video and audio, and discard the modality-specific features. As shown in Tab. 18, without modality-specific features, the performances drop slightly
Method	D2, D3 → D1	D1, D3 → D2	D1, D2 → D3	Mean
SimMMDG (modality-shared features only)	54.71	64.67	62.83	60.74
SimMMDG	57.93	65.47	66.32	63.24
Table 18: Multi-modal multi-source DG on EPIC-Kitchens dataset using only modality-shared features.
compared to the whole framework, but are still competitive. This indicates that modality-shared features truly have some meaningful information that can be used for prediction tasks.