SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization
Authors: Hao Dong
Ismail Nejjar
Han Sun
Eleni Chatzi
Olga Fink
Venue: NeurIPS 2023 poster (2023)
Keywords: Domain Generalization
Multi-modal Learning
Distribution Shift
Out-of-distribution Generalization
TL;DR: We propose a simple yet effective multi-modal domain generalization framework, as well as a novel multi-modal dataset for DG.

Abstract: In real-world scenarios, achieving domain generalization (DG) presents significant challenges as models are required to generalize to unknown target distributions. Generalizing to unseen multi-modal distributions poses even greater difficulties due to the distinct properties exhibited by different modalities. To overcome the challenges of achieving domain generalization in multi-modal scenarios, we propose SimMMDG, a simple yet effective multi-modal DG framework. We argue that mapping features from different modalities into the same embedding space impedes model generalization. To address this, we propose splitting the features within each modality into modality-specific and modality-shared components. We employ supervised contrastive learning on the modality-shared features to ensure they possess joint properties and impose distance constraints on modality-specific features to promote diversity. In addition, we introduce a cross-modal translation module to regularize the learned features, which can also be used for missing-modality generalization. We demonstrate that our framework is theoretically well-supported and achieves strong performance in multi-modal DG on the EPIC-Kitchens dataset and the novel Human-Animal-Cartoon (HAC) dataset introduced in this paper. Our source code and HAC dataset are available at https://github.com/donghao51/SimMMDG.

SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization
Hao Dong'
Ismail Nejjar2
Han Sun?
Eleni Chatzi'
Olga Fink?
'ETH Zürich 2EPFL
(hao. dong, chatzi baug. ethz. ch, (ismail (ismail.nejjar, han sun, olga.fink/@epfl.co
Abstract
In real-world scenarios, achieving domain generalization (DG) presents significant challenges as models are required to generalize to unknown target distributions. Generalizing unseen multi-modal distributions poses even greater difficulties due to the distinct properties exhibited by different modalities. To overcome the challenges of achieving domain generalization in multi-modal scenarios, we propose SimMMDG, a simple yet effective multi-modal DG framework. We ar- gue that mapping features from different modalities into the same embedding space impedes model generalization. To address this, we propose splitting the features within each modality into modality-specifie and modality-shared com- ponents. We employ supervised contrastive learning on the modality sharrdd fea- tures ensure they possess joint properties and impose distance constraints on modality -odaaitt--seccfic features to promote diversity. In addition, we introduce a cross- modal translation module to regularize the learned features, which can also be used for missing-modane is theoretically sell-supported and achieves strong performance multi-modal (HAC)
generalization. We demonstrate that our framework
DG on the EPIC-Kitchens dataset and the novel Caiimaa-AnimalCCaan dataset introduced in this paper. Our source code and HAC dataset are available at https///gituub.com/dnggggg I/SimMMDG.
Introduction
Domain generalization (DG) has received significant attention in the research community [66]. In real-world scenarios such autonomous driving [13, 26), robotics [17], action recognition [16],
and fault diagnosis [22], it is crucial that models trained on limited source domains perform well across novel target domains. To tackle distribution shift problems, numerous DG algorithms have been proposed, including Nomainn-nnariian
feature learning [51]. feature disentanglement [56], data augmentation [72], and meta-learning [41]. However, most of these algorithms are designed for unimodal data, such as images [40] and time series data [49]. More recently, the emergence of large-scale multi-modal datasets [16, 7] has highlighted the need to address multi-modal DG across multiple modalities, such as audio-video [36, 73], image-language [58, 33J. and LiDAR- camera [44, 18]. However, date, only RNA-Net [57] has focused the multi-modal DG problem, with relative norm alignment loss proposed to balance the feature norms of audio and video.
Multi-monl modal DG is closely related to multi-man modal representation learning [47]. Traditional multi- modal contrastive learning frameworks [58, 43] aim to project the features different modalities into common embedding space. However, this approach may not be optimal different modalities consist of both shared information that is consistent across modalities and unique information that is specific to each modality. Consequently, attempting align all modalities together can be challenging and impractical. For instance, consider descriptions of the same object that use entirely different modalities, such as video, audio, and optical flow, illustrated in Fig. (a). All modalities share some
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

information Information primurily in optical flow
Human
Motion Valecity
Information in modalltics
Information primarily video
Described objects, actions
Visual agpearance Sequential eetation
Illustativv gestures
Informattin primarily in audio
Tome Prequracy
Animal
Cartoon
Modality-ppeiifie and modality-shared information
(b)
Figure 1: (a). Different modalities possess shared information, while simultaneously containing unique information exclusive to each modality. Inspired by this, we propose to split the feature of each modality into modality-specific and modality-sarred parts in framework. (b) Our new multi-modal DG dataset consists of three domains and three modalities. For each domain, the actor of different actions (opening door in this case) could be either human, animal, or cartoon figure.
information since they describe the same object, but each modality further provides modality-speciit information. For example, videos typically convey visual appearance information and sequential relations between frames, while audio provides tone information that is closely linked to sentiment and frequency information for different sounds. Optical flow, on the other hand, offers more intuitive information motions and associated velocities. Therefore, simply mapping features from different modalities into the same embedding space would preserve only modality- shared information while overlooking modality -specffcc details. This could result in the loss of and a decline in downstream task performance.
Effective multi-modal DG frameworks that can address the challenges outlined above are urgently needed. To this end, we propose SimMMDG a Simple yet effective Multi-Modal DG framework that can be applied to two or more modalities. We first propose to split the feature embedding of each modality into modality-speciffi and modality-shared parts, ensuring that complementary information from different modalities is retained. We apply supervised contrastive learning on modality-shared features and incorporate distance constraints for modality-speciff features to ensure that the learned representations informative. Moreover, introduce cross cross-modal translation module to further regularize the learned features and facilitate missing fiissing-modality generalization. Some theoretical insights from both the multi-modal representation learning perspective and the domain generalization perspective are provided to demonstrate that our approach is well-motivated in theory. Finally, we release novel generalization, as shown in Fig. 1 (b). Our contributions can be summarized as:
Soitn-AAnial--Cattti (HAC) dataset, to promote research in multi-modal domain
We propose universal framework for multi-modal domain generalization that demonstrates both simplicity and effectiveness and outperforms existing methods on several challenging datasets. We also demonstrate its efficacy in general multi-modal classification setups.
We address the missing-modale modality generalization problem and propose cross-modal translation module as a solution, which is robust even in scenarios where multiple modalities are missing.
We provide theoretical insights in support of the efficacy of our proposed approach.
We introduce new multi-modal dataset that includes three modalities and large domain shifts, which can serve as a challenging benchmark for future research on multi-modal DG.
2 Related Work
Domain Generalization refers to the process of training model on multiple source domains in order to generalize unseen target domains. This task is more challenging than domain adaptation (DA) [67] as target domain data cannot be accessed during training. Prior research has identified three main categories of domain generalization methods [66], including data manipulation, representation learning, and learning strategies. Data manipulation techniques aim to improve generalization performance by increasing the diversity of training data. For example, Tobin el al. [61] use simulated environments for additional data generation, while Zhou el al. [75] train a domain transformation network using adversarial training to synthesize data from previously unseen domains. Mixup [72] generates new instances by performing linear interpolations between data and labels. Representation

---

learning methods seek to learn domain invariant representations through the use of domain aettainnadvvessann adversarial neural networks [25, 42], explicit feature distribution alignment [63], and instance normalization [54].
Other approaches utilize learning strategies to improve the generalization performance. Examples of
such approaches include meta-learning [41], gradient operation [31], and self-supervised learning [8].
Multi-modal Representation Learning aims to learn robust representations through two or more modalities, which can then be applied various downstream tasks. While unified models [68, 4] tokenize various input modalities into sequences [3] and employ a single Transformer [65] for joint learning, methods such as CLIP [58] and ALIGN [33] use separate encoders for each modality and leverage contrastive loss to align features. However, these methods align features from different modalities into the same embedding space and may only preserve modality-shared information. In contrast, we propose splitting the features of each modality into modality and modality- shared components to ensure that complementary information from different modalities is preserved. A recent work [34] also proposes to split the features into different components in multi-modal representation learning and provide an information thhoooeticteetcclll analysis. However, they only deal with two modalities and without considering missing- moddllty cases. They also don't 't use the label information within a batch in contrastive learning.
Multi-modal DA and DG. Several approaches exist for multi-modal DA. For instance, Munro and Damen [52] propose a supf--up alignment approach along with adversarial alignment for multi-modal DA, while Kim et al. [38] leverage cross-modal contrastive learning to align trossl and cross-domain representations. Zhang et al. [73] propose an additiv encoder and an audio-infused recognizer to address domain shifts. Notably, RNA-Net [57] is the only method known to address multi-modal DG problem, by introducing a relative norm alignment loss to balance audio and video feature norms.
3 Methodology
3.1 Problem Setting: Multi-modal Domain Generalization
We commence by presenting the definition of the multi-modat domain generalization problem based on the definition of unimodal DG, as described in [66]. Let X represent a nonempty input space, and y denote an output space. A domain, denoted as D, consists of data sampled from a distribution, Pxy, where Pxy denotes the joint distribution of input samples and output labels. X and Y represent the corresponding random variables.
D {(X,, yj 17=1
Definition 1 (Multi-modal domain generalization), In mulli-modal domain generalization, we are ni
given D training domains Dirain = (Di i = 1, ,D), where De {(x), )
j=1 denotes the
i-th domain with n4 data instances. Each data instance x3 {(x))x )k k = 1, 1....... ,M} E X is comprised of M different modalities and v¹ E y C R denotes the label. The joint distributions between each pair of domains are different: P Pxx.. $ siti D. The goal of multi-modal domain generalization is to learn robust and generalizable predictive function I X - y from D) training domains and M data modalities to achieve a minimum prediction error on an unseen test domain Dreat (i.e., Drest cannot be accessed during training and Plest + P for i {1,"., "D))
min Eix ( (x).,))
(1)
where E is the expectation and E(.y)) is the loss function.
3.2 SimMMDG
We present the SimMMDG framework for addressing the multi-modal DG problem, as depicted in Fig. 2. Our approach involves splitting the feature embedding of each modality into modality- specific and modality-shared components. We aim to map modality-shared embeddings of data samples with the same label, whether they belong to the same or different modalities, to be as close as possible (and vice versa for samples with different labels). For modality, our objective is to maximize the distance from their modality-shared features to capture unique and complementary information. Additionally, we incorporate a cross-modal translation module to regularize learned features and address the missing-modality generalization problem, which is essential for real-world testing scenarios where one or more modalities may absent due to unpredictable factors.
features within each
3

---

1 2
Feature extractor
Classifier
Modality 1
Supervised Contrastive Learning
Feature extractor
[c) Missing-modhan modalit Generalization (optional)
Modality
(c) Distance Loss
Modality-speciffe feature
Feature
Medality feature
extractor
Modalit 3
Missing-modality
(x) Feature Spliting
(d) Cross-modal Translation
Different classes
Figure 2: Overview of SiMMMDG. We split the features of each modality into modality-speciffe and modality-shared parts. For the modality-shared part, we use supervised contrastive learning map the features with the same label to be as close as possible. For modality-specific features, we use distance loss encourage them be far from modality-shared features, promoting diversity within each modality. Additionally, we introduce cross-modal translation module that regularizes features and enhances generalization across missing modalities.
3.2.1
Within-modal Feature Splitting
Traditional multi-modal learning frameworks [58, 43] aim to map features from various modalities into common embedding space using contrastive learning. However, this approach may be optimal because different modalities often contain mix of both homogeneous and heterogeneous information, making challenging and impractical align them all together. For instance, consider the language and visual modality [2]. Both modalities can describe people, objects, actions, and gestures, which reflect shared information between them. However, images can provide additional information such as texture, depth, and visual appearance that are not available in language data. Similarly, language can provide syntactic structure, vocabulary, and morphology, which are not present in image data. These are specific pieces of information that are unique to each modality.
Thus, the simple mapping of features from different modalities into the same embedding space may result in the loss of modality-specific information and consequently lead to decreased performance in downstream tasks. Guided by this intuition, we propose novel approach that involves the separation of the feature embedding of each modality into modality-specifff and modality shared components. For example, given unimodal embedding E, we denote as E Eg; Ee], where E, is modality-specific feature and E, is modality-share feature.
Multi-modal Supervised Contrastive Learning. We expect modality shared features E, to possess characteristics that are shared among distinct modalities. data instances from different modalities have the same label, we expect their modality-shared features be close as possible in the embedding space. To achieve this objective, we leverage the availability of labels for each data instance and employ supervised contrastive loss [37] for effective guidance. For set of N randomly sampled label pairs, {æ,, 1,,,,,/1,,. j= the corresponding batch used for training consists of M x N pairs, {Tx, ÿk 1x=1....Mx.. where Mxjx, 3M " IM are data instances from M different modalities in xy (j = 1,.... N) and fMxx = DM = V/-
N) be the index of arbitrary unimodal sample within batch. We define A(1) = (i), = {p E A(i): Up ūi as the set of indices of all positive samples in the batch which share the label as i. The cardinality of P(i) is denoted as P())). Then, the multi-modal supervised contrastive learning loss can be written as follows:
exp ezp 2p/T)
Loom Σ SEI
log Σ exp (2, 2a/T)
PEP())
aEA(i)
with 2N (rx)(g(#x)) E RD,, where g(-) the feature extractor, that maps x to modality-specifie
and modality-share features, E EE; E.] g(x), where E., E, RD and Proj(.) is the

---

projection network that maps E, to vector = Proj RDP,. The inner product between two projected feature vectors is denoted by :, and R is a scalar temperature parameter.
Feature Splitting with Distance. To ensure that the modality-specifie features E, carry unique and complementary information, aim maximize their dissimilarity from the corresponding modality-shared features E.. To achieve this goal, we utilize negative &g distance and formulate distance loss on and E, as:
Liis
Σ
E
E.I.2
M
where M is the number modalities, E2 and E are the modality-specific and modality-shared features of the i-th modality.
3.2.2 Cross-modal Translation
Simply increasing the distance between E, and E. may not yield optimal results. The proposed cross- modal translation module aims to ensure the meaningfulness of modalltt-speccifi features by exploit- ing the implicit relationships and approximate translation mappings that exist between the M modal- ities within the same data instance. This is achieved through multi-layer perceptron (MLP) [28] that translates the feature embedding E across modalities. For instance, E MLPE--B (E') implies that we translate the embedding E' = [ E.J of the i-th modality to the j-th modality using M ML resulting in the translated embedding E/. More intuitions behind the cross-modee translation module are discussed in the appendix. To ensure that the translated embedding E& is a meaningful representation of the j-th modality, we aim to minimize its l2 distance from the real embedding of the j-th modality, E' and the cross-modal translation loss is defined as:
Ltrans M(M
MMLP (E") E
(4)
3.3
Final Loss
The final loss is obtained the weighted sum of the previously defined losses:
L = Lats + Orooneoon + Oldiss Cais + Oltrans Ltrans:
(5)
where Cets is the cross-entropy loss for classification, and where Ocoon Odis, and Otrann are hyperparam- eters that control the relative importance of the contrastive learning, dissimilarity, and cross-modal translation terms, respectively.
3.4 Missing-modality Generalization
During the training phase, have access to data from all modalities. However, in the testing phase, one or more modalities may be absent due to unpredictable reasons, such as sensor malfunction or loss of communication. In such cases, the system's 's robustness toward potential missing modalities becomes critical. To address the baissing-modalle module. In normal circumstances, the output of the network is defined as:
scenario, we utilize our cross modal translation
y f(x) h(g(z)) A(EE E I))
(6)
where v(-) is the feature extractor and h(() is the classifier. If the i-th modality is missing during testing, one solution replace its embedding with zero. The output then given as:
= h(E''......... O...,,..,,E'.))
However, simply replacing the embeddings with null entries may have an adverse effect on the network's 's performance. To address this issue, we propose using our cross-modal translation module to predict and substitute the missing modality's embedding with information from available modalities, resulting in more robust output for the network. The output of the network is then given as:
y h(E)
E.
E
(8)
5

---

where
E
E MLPEE-BE (E'). j#i
M
In cases where multiple modalities are missing during testing, we can use the same strategy to utilize available modalities to predict and substitute the missing ones, similar Eq. (9). The benefits of approach are demonstrated in the subsequent experiments.
4 Theoretical Insights
4.1 Multi-modal Representation Learning Perspective
We first generalize Theorem 3.1 in [34] to the case of M modalities. Let p be the joint distribution of (I1, ZM 3) where 2; is the i-th modality and is the target. We define the information gap as 4g [ y); i = 1, ...; M) min (x;; y) = L,,...,,,, M) where (æ, y) is the mutual information between modality z; and the target variable The information gap serves to characterize the effectiveness of the modalities in predicting the target y. We use the cross-entropy loss, denoted by lcE, and the prediction function, denoted by f.
Theorem For M feature extractors g-(-) (i = l,.... M). if the multi-modal features E' g' (xx) are perfectly aligned in the feature space, i.e., E' infy Epllcz(f(EE),... E ",,, y)] inf (f FM), b) Ap.
E' ... EM then
The proof of this theorem be found in the appendix. Theorem indicates that the optimal prediction error, achieved with perfectly aligned features, is at least A, larger than when using the raw input modalities. In practical scenarios where one modality is less informative for prediction, the information gap Ap tends to be substantial. Consequently, this leads to a notable increase in prediction errors in downstream tasks. Furthermore, achieving perfect modality alignment enforces the aligned features to exclusively contain predictive information present in all input modalities, potentially causing the loss modality-speciinn information. By splitting the feature embedding for each modality, our model has access to both modality-shand shared features with predictive information present in both modalities, and modality-speeiffe features that contain predictive information unique to each individual modality. This enables our model to effectively capture the distinct information provided by each modality, while simultaneously leveraging the shared information across modalities. Consequently, our approach has the potential to enhance generalization capabilities and achieve better performance on downstream tasks.
4.2 Domain Generalization Perspective
Theorem 2. Let X be space and H. be class of hypotheses corresponding to this space. Let Q and the collection {P.) distributions over X and let 1K be a collection of non-negative coefficient with E. ps = 1. Let O be set of distributions s.I. vS E O, the following holds
Σ (P.,)) max dHAH (P..
(10)
1.
Then, for any h E H,
(h(h)= < Ao + Σ PIEE (h)
min duam (S, Q) max Manx (P...
1
(11)
2 SEO
where Xop is the error of an ideal joint hypothesis, Ep(h) is the foror a hypothesis h on a distribution F. and (IP Q) is H-divergence which measures differences in distribution [6].
The proof of this theorem is provided in [60]. Here, Q corresponds to the unseen cittof-dditribution target domain and {TP; K correspond to source domains. A is small in reality and often neglected. The term PtEp, (h) is minimized by cross-entropy loss with class labels as supervision. The term 2 maxi,j dHAH (P., P,) measures the maximum differences among source domains. This corresponds to multi-modal supervised contrastive learning in our approach, where we map the modality-shant shared features of different modalities from all source domains to be close possible

---

if they have the same label. The term _ minseo dwam Q) demonstrates the importance of diverse source distributions [1]. so that the unseen target Q might be "near" to O. Therefore. to enlarge the range of O, we split the feature embedding of each modality into mpdality-specifie and modality-shares parts by maximizing the la distance to preserve diversity.
5
Experiments
5.1 Experimental Setting
Dataset. We use the EPIC-Kitchens dataset [16] and introduce novel HAC dataset in this paper, which will be made publicly accessible for further research. We follow the experimental protocol used for the Kitchess dataset in [52]. The EPIC-Kitchens dataset includes eight actions ('put', "take', 'open', 'close', 'wash', "cut', 'mix', and 'pour') recorded in three different kitchens, forming three separate domains D1. D2, and D3. Our HAC dataset consists of seven actions ('sleeping' "watching tv', 'eating', 'drinking', 'swimming'. 'running', and 'opening door") performed by humans, animals, and cartoon figures, forming three different domains H, A, and C. We collect 3381 video clips from the internet with around 1000 samples for each domain. We provide three modalities in our dataset: video, audio, and pre-computed optical flow. Some examples of the HAC dataset are shown in Fig. 1 (b) and more details are provided in the supplementary material.
Implementation Details. In our framework, we perform experiments on three modalities: video, audio, and optical flow. We adopt the MMAction2 [15] toolkit for experiments. To encode the visual information, we use FlowFast network [21] initialized with Kinetics-400 [35] pre-trained weights. For the audio encoder, we use ResNet- 18 [29] and initialize the weights from the VGGSound pre-trained checkpoint [12]. Similarly, we use the SlowFast network with slow-only pathway and also Kinetics 400 [35] pre-trained weights for the optical flow encoder. The dimensions of the unimodal embedding E for video, audio, and optical flow are 2304, 512, and 2048 correspondingly. For the projection network Proj(-) in supervised contrastive learning. we instantiate it as a multi-layer perceptron with two hidden layers of size 2048 and output vector of size Dp 128. We use a multi-layer perceptron with two hidden layers of size 2048 to instantiate the cross-modal translation M LPPg:-B After obtaining the feature embedding from the encoder, we split the embedding into modality-shared features (the first half of the embedding) and modality-specifee features (the remaining half). We use the Adam optimizer [39] with learning rate of 0.0001 and a batch size of 16. The scalar temperature parameter is set to 0.1. Additionally, analyze the sensitivity of different a in the appendix. Finally, we train the network for 15 epochs on an RTX 2080 Ti GPU which takes about 20 hours and select the model with the best performance on the validation dataset. We report the Top-1 accuracy for all experiments.
Cccon 3.0, a'di 0.7, and Atrans 0.1. We also
5.2 Results
Multi-modal Multi-source DG. Tab. and Tab. 2 illustrate the results multi-modal multi-source DG setting, where we train on multiple source do- mains and test We first conduct experiments using video and audio modalities for exper- iments, as in [57]. We re-implement our framework employing the same I3D [10] and BN-Inception [32] back- bone for video and audio in RNA- Net [57] to ensure fair comparisons. The DeepAll approach implies that we feed all the data from source do- mains to the network without any do- main generalization strategies. Tab. 1 shows that our SimMMDG signifi- cantly outperforms all the baselines. When we replace the backbone with SlowFast [21] and R8sNet-18 [29].
SimMMDG under the
Method
D2, D3 + DI
DI, D3
D2
DI, D2 + D3
Mean
target domain.
L3D backbone
DeepAll
43.19
39.35
51.47
44.67
IBN-Nett [53]
44.46
49,21
48.97
47.55
Gradient Blending [69]
47.27
TBN [BN
42.35
47.45
49.20
46.33
AVSA
42.78
47.38
$1.79
47.32
Co-Attentin [14]
40.87
43,57
$4.88
46.44
SENet [30]
42.82
42.81
$1.07
45.56
LacaL [70]
MM-SADA [52]
39.79
52.73
$1.87
48.
RNA (57)
45.65
51.64
55.88
$1.06
SimMMDG (ours)
54.25
58.67
$7.28
56.73
Slow backhone
DeepAll
MM-SADA [52]
49.20
60.40
59.1 14
56.25
RNA (57)
52.18
59.47
60.88
57.51
SimMMDG (ours)
$7.93
66.32
63.24
Table 1: Multi-modi multi-source DG on EPIC-Kitchens dataset using video and audio.
7

---

Modality
EPIC-KKtchens dataset
HAC dataset
Video
Andio
D1, D2,
Meon
A.C.
H. AC
DespAll
47.13
55.73
57.17
$3.34
66.55
72.85
45.77
61.72
MM-SADA [52]
49,20
60.40
59.14
$6.25
65.44
72.52
44,30
60.76
RNA-Net (57)
52.18
59.47
60.88
57.51
60.20
73.95
48,90
61.02
SimMMDXG (ours)
57.93
65.47
66.32
63.24
74.77
77.81
68.75
DeepAll
55.17
62.91
60.37
59.49
76.78
70.64
42,63
65.68
MM-SADA [52]
47.13
57.60
59.34
54 44.69
69. 79
69.76
49:45
63.00
RNA-Net (57)
54.71
61.87
58.21
58.26
77.14
74.94
42.00
64.69
SimMMDG (ours)
61.79
79.31
77.04
DeepAll
45.28
56.40
57.08
52.92
50.04
59.71
38.97
49.57
MM-SADA [52]
47.56
53:17
60.27
$3.70
16.58
61.81
39.15
49,18
RNA-Net (57]
45.74
57.73
$6,47
$3.51
$2.05
64.13
40.35
52.18
SimMMIXi (ours)
59.63
56.24
DeepAll
55.63
59.20
58.01
57.61
69:07
$1.47
63.95
MM-SADA [52)
51.72
58.40
59.94 94
56.49
72.53
72. 12.19
55.51
66.74
RNA-Net (57)
52.41
57.20
60.16
$6.59
69.00
73.40
51.65
64.68
SIMM (ours)
63.68
67.19
77.65
Table 2: Multi-modal multi-source DG with different modalities on EPIC-Kitchens and HAC
datasets.
EPIC-Kitchens dafaset
datasel
Source:
D1
D2
D3
H
A
Method
Target:
D3
C
Mean
DeepAll
52.27
51.75
44.60
54.11
48.05
56.67
51.24
66.11
43.01
63.45
37.68
46.86
86
38.94
52.68
MM-SADA [52]
50.67
50.10 10
51.49
56.57
42.99
54.00
50.97
37,22
57.75
40.90
49.82 82
62.91
52.42
RNA-Not [57]
45.60
46.30
43.68
$7.39
49.66
55.87
49.75
63.67
42.92
61.72
38.69
47.66
61.59
53.04
SimMMDG tours)
54.00
52.26
$1.49
60.88
49.88
60.53
54.84
66.67
44.21
68.42
46.05
54.43
72.73
58.75
Table 3: Multi-modal single-source DG on EPIC-Kitchens and HAC datasets using video and audio.
the results further improve by a large margin (with an average improvement of up to 5. 5.73% compared to the baselines. In the following experiments, we adopt SlowFast and ResNet- 18 bones. To verify the generalization of our framework to different modalities, we conduct experiments by combining any two modalities, as well as all three modalities, and present the results in Tab. Our SimMMDG outperforms all the baselines by significant margin in all cases, with improvements of up to 9.58%. Notably, when we combine all three modalities, the performance further improves and surpasses that of any two modalities. In contrast, the baseline methods cannot achieve better results with more modalities, indicating that they fail to fully leverage the complementary information between modalities. Finally, we validate the performance of our framework on the HAC dataset, and the results are consistent with those obtained on the EPIC-Kitchens dataset, as demonstrated Tab. 2. Our SimMMDG outperforms all the baselines by significant margin in all cases, with improvements of up to 7.73%.
default back-
Multi-modal Single-source DG. The domain labels are not required in our SimMMDG framework. This feature makes our method readily applicable to single-source DG without modifications. In this setup, we train the model on single source domain and test it on multiple target domains. Tab. 3 presents the results of SimMMDG under the multi-modal single-source DG setting using video and audio modalities. Despite being trained only on data from a single domain, our model demonstrates robust generalization to unseen domains, with an average improvement of up to 5.71%. In comparison, other baseline methods perform even worse than the DeepAll baseline, highlighting their limitations in the single-source DG setting. We present additional results obtained by exploring different combinations of modalities in the appendix.
dissing-modalle DG. In real-world deployment scenarios, we cannot always guarantee that all modalities will be available. Hence, the network should be capable of handling modaity cases. One simple approach is set the embedding of the missing modality to zero. We compare this with our proposed cross- cross-modal translation replacement in Sec. 3.4. We retrain the cross-modal translation M MLP for 10 epochs using Lirans in Eq. (4) and keep other parameters fixed. Tab. reports on the comparison results on the EPIC-Kitchens dataset, where our solution yields significant benefits (up to 10.47% performance improvement) compared to zero-filling. By replacing the missing modality features with the translation ones, our approach also outperforms the unimodal model in most cases. In contrast, zero-filling hurts the network performance in some cases and is even worse than the unimodal model. The last six rows in Tab. 4 demonstrate that our approach is robust
8

---

Method
Video
Audio
D2,
DI.
DeepAll (Vidco-onlyy)
51,03
59.87
56.57
55.82
SimMMDG (2ero- fillngg
SimMMDG (translation)
x
57.24
65.07
59.65
60.65
SimMMDG (illng)
SimMMDG (translation)
53.17
61.33
61.70
$9.40
DeepAll Audio-oaly)
32.87
42.27
45.17
40.10
SimMMDG (zem- (rero-fillng)
29.20
37.33
45.69
37.41
SimMMDG (transtation)
37.70
46.40
52.05
45,38
SimMMDG (zero (ilin)
28.51
34.27
42.81
53.20
SimMMDG (transtation)
39.77
46,00
51.23
45.67
DeepAll (Flow-only)
54,25
61.33
55.95
57.18
SimMMDG (zero-filling)
56.32
60.67
56.57
57.85
(translation
MinMMDG (zero (illin)
52.41
62.13
$1.33
53.29
SimMMDG (translation
56.57
58.57
SimMMDG (zeto filing)
53,10
62.40
64.48
59 99
SimMMDG (translation)
55.86
68.27
64.58
62.90
SinMMDG (2cr0-illing)
62.76
66.80
57.49
62.35
SimMMDG (translation)
63.22
67.20
59.24
63,22
MimMMDG (zem (zei-iilling)
"60.92
68.66
"61.99
6131
SimMMDG (translation)
62.53
68.13
61.60
64.09
SimMMDG (zero (elinn
x
59.08
62.27
$2.98
58.11
SimMMDG (translation)
59.31
64,00
56.57
59.96
SimMMDC (zero (zeii-)illig)
33.79
57.47
44.56
58.61
SimMMDG (translation)
38.39
44.53
47.74
41.55
SinMMDD (zero-illing)
57.07
53337
SimMMDG (translation)
56.32
64.27
56.57
59.05
Table 4: Multi-modal multi-source DG with missing modalities on EPIC-Kitchens dataset. means the modality available during training, but is missing in test time.
even when two modalities out of three are missing. We present more results on HAC dataset in the appendix.
5.3
Ablation Studies
Ablation on each proposed module. We conducted extensive ablation studies to investigate the role of each module of SimMMDG on HPPCCKitchens dataset, as shown in Tab. 5. Incorporating the supervised contrastive learning loss alone resulted in noticeable improvements. However, the mean accuracy decreased when we integrated feature splitting without imposing any constraints. The results were further enhanced when we added the distance loss to promote diversity, and even more so when we incorporated the cross-modal translation module. Although using only supervised contrastive learning and cross-modal translation yielded satisfactory results, their performance is on average 4.15% lower than the complete approach with feature splitting and distance loss. These findings highlight the significance of segregating the feature embedding each modality into modality- specific and -odaaitty-shardd components. Although SimMMDG without supervised contrastive learning is already better than most baselines, it still has a performance gap compared with the whole framework, which means the cross-modal translation is helpful but cannot replace contrastive learning. The effect of contrastive learning is similar to explicit feature alignment. modality-shan shared features of different modalities from different source domains with the same label to be as close as possible in the embedding space, while pushing away features with different labels, to make the embedding space distinctive.
aligns the
Comparison against unimodal DG. Tab, presents the results in comparison to unimodal DG algorithms that exclusively rely either video, audio, or optical flow inputs. We choose RSC [31], Mixup [72], and Fishr [59] as our baselines. By leveraging information from multiple modalities, our multi-modal DG framework delivers significant improvements (up to 7. 7.74%) in terms of performance as compared to unimodal DG methods.
Combine SimMMDG with other training strategies. SimMMDG can be seamlessly combined with other training strategies due to its generality and simplicity. We first combine SimMMDG with Gradient Blending [69], strategy to improve multi-modal learning. The results are further improved compared to the original SimMMDG as shown Tab. 7. We also combine SimMMDG with Domain aamain-adversarinn Neural Network (DANN) [24] to align the features of source domains using

---

Method
D2, D3 DI
DI,
D2 DI,
Meon
CL ES DL D2.
DI
D3
D2
DI.
D2
Mean
RSC
RSC (A)
36.55
43.73
48.15
42.81
55.86
56.27
54.21
55.45
RSC
52.18
61.60
62.22
58.67
Mixup (V)
49.20
59.73
59.96
36.30
63.24
59.09
Mixup (A)
35.17
40,80
45.07
40.35
51.72
62.67
58.93
57.77
Mixup (F)
56.32
65.60
54.62
58.85
55.17
64.00
64.37
61.18
Fishr (V)
53.79
63.47
61.09
59,45
$4.25
63.47
43.04
60.25
Fishr (A)
37.47
44,80
47.43
43.23
57.93
65.47
66.32
63.24
Fishr
54.25
63.87
59.14
$9.09
SimMMDG (V+A+F)
Table 5: Ablations of each proposed module on EPIC-Kitchens dataset. CL: supervised contrastive learning, FS: feature splitting, DL: distance loss, CT: cross-modal translation.
63.68
70.13
67.76
67.19
Table 6: Comparison with single-modal DG
methods on EPIC-Kitchens dataset. V: video, A: audio, F: optical flow.
Method
D2,
D2
Mean
Method
EPIC-Kitchems
Method
MUSTARD UR-FUNNY
SimMMDG
57.93
65,47
66.32
63.24
DeepAll
TRN
73.2
64.9
1MF
Fusson
61.6
61.9
63.6
+Gradient Blending
59.31
68.40
66.63
64.78
Blending
74.0
+DANN
60.69
66.69
64.58
64.07
SimMMDG
76.4
SimMMDG
72.5
65.6
Table 7: Combine SimMMDG with other training
Table Evaluation under multi-modee classi-
strategies on EPIC-Kitchens dataset.
fication setup.
domain labels and observe a similar improvement as with Gradient Blending. This indicates that our SimMMDG can be easily combined with other training strategies to get even better results.
SimMMDG as general framework for multi-modal classification. Here, we evaluate our framework in a more general multi-modal classification setup without DG. We first evaluate our framework on the EPIC-Kitchens dataset without incorporating the DG setup. To do so, we aggregate data from three domains, we partition the training, validation, and testing data to ensure that no apparent domain shifts exit between the training and testing data. We compare our method with DeepAll, TBN [36], and Gradient Blending [69]. The results, as presented in Tab. 8 demonstrate that SimMMDG also exhibits significant advantages in the general multi-modal classification setup.
We further evaluate our framework on two multi-modal datasets: MUSTARD [11] and UR- FUNNY [27], both available in MultiBench [45]. These datasets pertain human sentiment analysis and encompass language, video, and audio modalities. We conduct our experiments using the Multi- Bench codebase and implement SimMMDG within that environment. MultiBench treats human sentiment analysis as regression task, whereas our framework is tailored for classification. To align the codebase with our classification task, we made the necessary modifications. For all baselines. we apply the same backbone model and solely change the fusion paradigms. In our comparisons, we evaluate our method against Late Fusion, Low-rank Tensor Fusion (LMF) [48], and Multimodal Transformer Fusion (MULT) [62]. The results reveal that our SimMMDG exhibits clear advantages, outperforming the baselines with an average improvement of 7.3% and 1. 1.7%. This suggests that our SimMMDG serves as versatile framework for multi-modal classification tasks and is compatible with various combinations modalities, such as video+audio+flow and language+video--audio.
6 Conclusion
In this paper, we propose the SimMMDG framework for multi-modal DG. Our approach involves splitting the features of each modality into podality-specifii and modality-shared parts and enforc- ing constraints on each part using supervised contrastive learning, distance loss, and cross-modal translation. The cross-modie translation module can also be applied in the case of dissing-modalig generalization. Our experiments on different datasets demonstrate the effectiveness of SimMMDG. Furthermore, introduce new challenging multi-modal dataset that serve benchmark and guide future research multi-modal DG problems.
Limitations. Currently, the number of cross-modal translation MLP in our framework is O( ((M))) and will complex with the increase of modalities. In future work, the encoder-decoder network proposed in [19] can be used to reduce the complexity to O[M
10

---

Acknowledgments
The authors acknowledge the support of "In-service diagnostics of the catenary/pantograph and wheelset axle systems through intelligent algorithms" (SENTINEL) project, supported by the ETH Mobility Initiative.
References
[1] Isabela Albuquerque, João Montciro, Mohammad Darvishi, Tiago Falk, and loannis Mitliagkas. Gener- alizing to unseen domains via distribution matching, arXiv preprint arXiv: 1.1:004410000, 2019.
[2] Tadas Baltrušaitis. Chaitanya Ahuja, and Quis-Philippe Moreney. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):433 443, 2019.
[3] Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2022.
[4] Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision- laigu-la pre-training with mixture-of of poddllityexpeerts NeurIPS, 2022.
Normand Beaudry and Renato Renner. An intuitive proof of the data processing inequality. Quantum Information Computation, 12(5-6):4324444 14332-441,20 2012.
[6] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Percira, and Jennifer Wortman Vaughan. theory of learning from different domains. Machine learning. 79(1):151-775, 2010.
[7] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[8] Fabio M Carlucci, Antonio D' I'noccente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In CVPR, 2019.
[9] Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv: 01.400 2018.
[10] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset.
In CVPR, 2017.
[11] Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and
Soujanya Poria. Towards multimodal sarcasm detection. arXiv preprint arXiv:1906.018/5. 2019.
[12] Honglie Chen. Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio-visual dataset. In ICASSP, 2020.
[13] Xiaozhi Chen, Huimin Ma, Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In CVPR, 2017.
[14] Ying Cheng, Ruize Wang, Zhihao Pan, Rui Feng, and Yuejie Zhang. Look, listen, and attend: Co- attention network for self supervised audio-visual representation learning. In Proceedings of the 28th ACM International Conference on Multimedia, pages
2020.
[15] MMAction2 Contributors. Openmmlah's 's next generation video understanding toolbox and benchmark. https: //ta::///iitiie htt.tttub....oppoeeeeeeeeet aappeenmmab/mmmeeeeeee htt//mmmaaiiinoooo 2020.
[16] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens datasct. In ECCV, 2018.
[17] Hao Dong, Xieyuanli Chen, Simo Särkkä, and Cyrill Stachniss. Online pole segmentation range images for long-term lidar localization in urban environments. Robotics and Autonomous Systems, 159:104283, 2023.
[18] Hao Dong, Xianjing Zhang, Jintao Xu, Rui Ai, Weihan Gu, Huimin La,, Juho Kannala, and Xieyuanli Chen. Superfusion: Multilevel lidar-camera fusion for long-range hd map generation. arXiv preprint arXiv:2211.16666, 2022.
[19] Mihai Dusmanu, Ondrej Miksik, Johannes Schönberger, and Mare Pollefeys. Cross-descriptor visual localization and mapping. In ICCV, 2021.
[20] Farzan Farnia and David Tse. A minimax approach supervised learning. In NewrlPS, 2016.
[21] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.

---

[22] Olga Fink, Qin Wang, Markus Svensén, Pierre Dersin, Wan-Jui Lee, and Melanie Ducoffe. Potential, challenges and future directions for deep learning in prognostics and health management applications. Engineering Applications of Artificial Intelligence, 92:103678, 2020.
[23] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio. Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In NeurlPS, 2013.
[24] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML.
2015.
[25] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette,
Mario Marchand, and Vietor Lempitsky. aaversiadvvrsariall training of neural networks. The journal of machine learning research, 17(1):2096-200 2030, 2016.
[26] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
[27] Md Kamrul Hasan, Wasifur Rahman, Amir Zadch, Jianyuan Zhong, Md lftckhar Tanveer, Louis-Philippe Moreney, et al. Ur-funny: A multimodal language dataset for understanding humor. arXiv preprint (eteii1044066661 2019.
[28] Trevor Hastie, Rohert Tibshirani, and Jerome Friedman. The Flements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[30] Jie Hu, Li Shen, and Gang Sun. Squeere-and-exciiation networks. In CVPR, 2018.
[31] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In ECCV, 2020.
[32] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[33] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoe Iec, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling visual and vision-language representation learning with noisy text supervision. In /CML. 2021.
[34] Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Understanding and constructing latent modality structures in multi multi- representation learning. In CVPR, 2023.
[35] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video datasct. arXiv preprini 10Sv:1705.006500, 2017.
[36] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In ICCV, 2019.
[37] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NewrlPS, 2020.
[38] Donghyun Kim, Yi-Hsuan Tsai, Bingbing Zhuang, Xiang Yu, Stan Sclaroff, Kate Saenko, and Manmohan Chandraker. Learning cross-modal contrastive features for video domain adaptation. In ICCV, 2021.
[39] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2015.
[40] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV, 2017.
[41] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Learning generalize: Meta-learning for domain generalization. In AAAI, 2018.
[42] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR, 2018.
[43] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS, 2021.
[44] Ming Liang. Bin Yang. Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-senser 3d object detection. In F.CVV, 2018.
[45] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, al. Multibench: Multiscale benchmarks for multimodal representation learning. arXiv preprint arXiv:2107.075:0 2021.
[46] Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.03430) 2022.
12

---

[47] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurlPS, 2022.
[48] Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshninarasimhann, Paul Pu Liang, Amir Zadeh, and Louis- Philippe Morency. Efficient low-rank multimodal fusion with modality factors. arXiv preprint arXiv: 00064, 2018.
[49] Wang Lu, Jindong Wang, Xinwei Sun, Yiqiang Chen, and Xing Xie. Out-of-distribution representation learning for time series classification. In ICI.R, 2023.
[50] Pedro Morgado. Yi Li, and Nuno Nvasconcelos. Learning representations from audio-visual spatial alignment. In NeurlPS, 2020.
[51] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICML, 2013.
[52] Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine-grained action recognition. In CVPR, 2020.
[53] Hyeonseob Nam and Hyo-Eun Kim. Batch-instance normalization for adaptively style- inverinvariantt neural networks. In NeurlPS, 2018.
[54] Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing leamning and generaliza- tion capacities via ien-net. In ECCV, 2018.
[55] Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Moreney, and Barnbáss Póezos. Found in translation: Learning robust joint representations by cyclic translations between modalities. In AAAI, 2019.
[56] Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common- specific low-rank decomposition. In ICML, 2020.
[57] Mirco Planamente, Chiara Plizzari, Emanuele Alberti, and Barbara Caputo. Domain generalization through audio-visual relative norm alignment first person action recognition. In WACV, 2022.
[58] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In JCML, 2021.
[59] Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of- distribution generalization. In /CML, 2022.
[60] Anthony Sicilia, Xingchen Zhao, and Seong Jae Hwang. Domain adversarial neural networks for domain generalization: When works and how improve. Machine Learning, pages 1-37, 2023.
[61] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. IROS, 2017.
[62] Yao-HHung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In ACL, 2019.
[63] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint 111.::441..3474, 2014.
[64] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
[65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Ilaa Polosukhin. Attention all you need. In NewrIPS, 2017.
[66] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing unseen domains: A survey domain generalization. IEEE Transactions on Knowledge and Data Engineering, pages 1-1, 2022.
[67] Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312:135-153, 2018.
[68] Peng Wang. An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to- sequence learning framework. In ICMI, 2022.
[69] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In CVPR, 2020.
[70] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
[71] Christopher Zach, Thomas Pock, and Horst Bischof. A duality based approach for realtime tv-1 I optical flow, In Pattern Recognition, 2007.
13

---

[72] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[73] Yunhua Zhang, Hazel Doughty, Ling Shao, and Cees G.M. Snoek. Audio-adaptive activity recognition
across video domains. In CVPR, 2022.
[74] Han Zhao, Chen Dan. Bryon Aragam, Tommi Jaakkola, Geoffryy J Gordon, and Pradeep Ravikumar. Fundamental limits and tradeoffs in invariant representation learning. Journal of Machine Learning Research, 23(340):1-45 2022.
[75] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep generation for domain generalisation. AAAI, 2020.
image
14

---

A Proof of Theorem 1
Proof of Theorem I. Consider the joint mutual information (E', E.; By the chain rule, we have the following decompositions:
T(E', BM,u) = T(E':)) + T(E?. EM ;y |
= =IEEM + I(E .... EM-11 y E.M
However, since E' are perfectly aligned, I(E² EM; ; E
I(E ..... EM-1.y EE
O, which means I [EE ..... EM; :y) I [E': :y)
I(E :y). On the other hand, by the data
processing inequality [5], we know that
(EE y) S 1 (z v), ((EM S [(Mii))
Hence, the following chain of inequalities holds:
E M :) min{/(E': :v), 1IM ;y}}
min(/(x1;))) I..,...,,,,, ((
S max | (x); [(
(,., TMyy).
where the last inequality follows from the fact that the joint mutual information [(x1, JMr y), at least as large as any one of [[ y). We use H(y J1,..., to denote the conditional entropy of y given I1, ... ZM as input. According to [74, 20] we have the following variational form: ( I1, IM) = inf Ep (f functions that take T1, .... ZM input to predict the target y and the expectation is taken over the joint distribution of (I1,.... FMM y). Therefore, due to the variational form of the conditional entropy, we
IM ). y)]. where the infimum is over all the prediction
have
inf EplceE(f(E E.), - inf -inffE (f (((,,,......))))) f
-H E EM.,,M)) - (y [1,....
-H) Hy I1, ..., TM) (ld H E E)))
[II,, EMM
2
- min(/((zig)) [ 4)
max()(z))); :=::;;1111-- - ( 1,... M}
Ap-
B More Details on HAC Dataset
Our Human-Animal-Carronn (HAC) dataset consists of seven actions ('sleeping'. 'watching tv', 'eating', 'drinking', 'swimming', 'running', and 'opening door') performed by humans, animals, and cartoon figures, forming three different domains. We collect 3381 video clips from the internet with around 1000 for each domain and provide three modalities in our dataset: video, audio, and pre-computed optical flow. The dense optical flow is extracted at 24 frames per second using the TV-L. algorithm [71].
Our dataset was collected by volunteers. For the human domain, collect the data by selecting actions from an existing Kinetics-600 dataset [9]. We select approximately the same number of video clips for each action ensure class balance. For the animal domain, we use the available 200 video clips in [73] and extend it to 906. We collect data from YouTube by searching keywords like "animal sleeping', "animal eating', 'animal running', etc. To increase the diversity of the dataset, we also specify the animal type in the keywords, such as 'cat sleeping', 'dog eating', 'horse running', etc. Each participant was asked to collect certain actions, such as Participant A for 'sleeping" and "watching tv', Participant B for 'eating' and 'drinking'. etc. For the cartoon domain, we collect all data from scratch and we collect data from popular cartoons like 'SpongeBob SquarePants', 'The Simpsons', "Garfield and Friends', etc. Each participant was asked to collect from one two cartoon
15

---

Human Animal
200
Cartoon
175
Number of Action Segnetts
150
125
100-
75
50
25
sleeping
watching
cating
drinking Action Class
swimming
running
opening door
Figure 3: The number of each action segments for our HAC dataset.
Domain
Human
Animal Cartoon
Training Action Segments
1111
730
870
Validation Action Segments
276
176
218
Testing Action Segments
1387
906
1088
Table 9: Number of action segments per domain.
series to avoid duplication. For each action, we annotated the start and end times the video and then cut out video clips. The length of each video clip varies from 1s to 10 Finally, gathered the data from all volunteers and a separate person manually discarded unqualified data like duplicate videos, videos without audio data, noisy/wrong classes, etc.
Fig. 3 and Tab. illustrate the statistes of our HAC dataset. During the training phase, we train our model on one or two domains by utilizing the training action segments specific to those domains. The model selection is based on the performance of the validation action segments. Subsequently, we evaluate the model the remaining domain, utilizing all the segments within that domain. Therefore, the total number testing action segments the sum of the training and validation segments. is observed that each class within the HAC dataset exhibits good balance across all domains. In contrast, CetC-Kitheens dataset suffers from severe class imbalance. This implics that our proposed framework is capable of delivering regh-performance results across both balanced and imbalanced datasets. Fig. 4 presents additional examples from our HAC dataset, highlighting the substantial domain shifts present in our dataset, particularly in the cartoon domain. The main purpose of this dataset is to be used for multi-mode domain generalization research. Of course, our dataset can also be used for other applications like multi-modai learning and multi-modal domain adaptation.
C Additional Implementation Details
We follow RNA-Net [57] to select the baseline methods. MM-SADA [52] is domain adaption method and we only use the self-supervision loss without adversarial alignment as we have no access on target domain data during training. From the results in Table in main paper, we know RNA-Net [57] and MM-SADA [52] are the two best baseline methods. Therefore, we select them together with DeepAll baselines in the following experiments.
16

---

aleeping
watching tv
cating
drinking
swimming
running
opening door
Human
Animal
Cartoon
Figure 4: More examples on our HAC dataset.
Distance Type D2, D3 -+ DI DI, D3 - D2 DI, D2 - D3 Mean
Cosine
53.56 55.86 57.93
60.80 62.27
59.65
58.00
&
58.73
58.95
65.47
66.32
63.24
Table 10: Ablation of different distance types in distance loss on EPIC-Kitchens dataset.
D Further Ablations
Ablations of different distance types. Tab. shows the ablations of different distance types in distance loss Laie. We explore the effects of using & norm, li norm, and Cosine similarity. Our experimental results indicate that all three distance types outperform the baselines, with Eg norm achieving the highest performance.
Parameter Sensitivity. We investigate the sensitivity our method the hyperparameters in the loss function. We perform this analysis by varying one parameter while fixing the others, and present our findings Fig. The results demonstrate that our method consistently outperforms the DeepAll baseline for all parameter settings, indicating that our approach is less sensitive to hyperparameter choices. However, we can also observe Oatis is little sensitive and may need attention for tuning in real applications.
17

---

66
SuDBBD DvesAll
66
SuSDID
Degui
06
61
61
Accmray (90
400020c (%0
4000000Y (%0
02-
Su
Depail
60-
68
58
3
5
10
0.1
0.5
0.7
1.0
0.010.1
0.5
1.0
Qwn
Ods
Opuns
Figure 5: Parameter sensitivity (DI, D2 + D3 in EPIC- Kitchens for mudti-modal multi-source DG).
Modality
Source: D1
Source: D2
Source: D3
Method
Video Audio Flow D1
D2 DI
D2
D3 D3
D2 Mean
DeepAll
53.07
47.74
49.66
56.98
52.18
55.73
52.56
MM-SADA [52]
49.20
48.87
51.26
59.03
44.60
56.40
51.56
RNA-Net [57]
53.20
49.79
52.18
58.11
47.13
55.47
52.65
SimMMDG (ours)
56.80
54.11
53.10
59.86
50.57
64.27
56.45
DeepAll
/
41.60
47,43
41.61
51.75
45,98
56.13
47.42
MM-SADA [52]
V
42.53
47.64
42.76
50,92
42.07
51.87
46.30
RNA-Net [57]
V
46.40
49.38
42.76
54.62
43.45
52.13
48.
SimMMDG (ours)
V
50.00
49.49
45.98
56.26
51.95
63.87
52.93
DeepAll
51.60
53.39
44.60
60.16
45.74
53.47
51.49
MM-SADA [52]
49.43
48.53
38,16
48.46
44.14
51.07
46.63
RNA-Net [57]
$1.20
53,90
46.67
58.73
49.43
57.47
52.90
SimMMDG (ours)
55.33
54.83
50.80
59.45
53.33
64.40
56.36
Table 11: Multi-modal single-source DG with different modalities on EPIC-Kitchens dataset.
Modality
Source: H
Source: A
Source:
Method
Vidco Audio Flow H-+ H-> C
A +
H
A +
C
C-+ H C-+ Mean
DeepAll
59.82
41.08
73.25
35.02
60.71
64.46
55.72
MM-SADA [52)
66.23
46.23
69.65
42.46
59.77
57.62
56.99
RNA-Net [57]
58.61
37.87
73.90
48.16
60.35
59.38
56.38
SimMMDG (ours)
63.13
44.58
74.62
51.38
70.15
61.59
60.91
DeepAll
59.93
35.29
50.90
35.20
32.73
39.85
42.32
MM-SADA [52]
55.08
32.90
52.85
34.10
29.70
44.48
41.52
RNA-Net [57]
56.51
30.24
53.50
33.46
36.91
40.40
41.84
SimMMDG (ours)
v
62.25
36.40
56.60
38.79
38.72
48.12
46.81
DeepAll
62.58
42.56
67.34
46.05
53.86
59.49
55.31
MM-SADA [52]
62.25
44.67
60.78
43.11
41.82
44.81
49.57
RNA-Net [57]
66.89
45.40
66.62
46. 88
53.64
61.37
56.80
SimMMDG (ours)
68.76
47.98
70.94
49.26
65.18
62.69
60.80
Table 12:
single-source DG with different modalities on HAC dataset.
E Other Experimental Results
Multi-modal Single-source DG. Tab. 11 and Tab. 12 show more results under multi-modal single- source DG setting using the combination of different modalities. Our model demonstrates robust generalization to unseen domains compared to the baseline methods in most cases with a large margin up to 4.81%.
Missing-modality DG. Tab. 13 shows more results under the missing-modality DG setting on HAC dataset. Our solution yields significant benefits compared to zero-filling most cases, similar to findings in the EPIC-Kitchens dataset. Our approach is also robust even when two modalities out of three are missing. We also validate the effectiveness of our cross-modal translation module on DeepAll, MM-SADA [52], and RNA-Net [57], shown in Tab. 14. Compared to zero-filling,
18

---

Method
Video Audio Flow A, H
H. - C Mean
DeepAll (Video-only)
75.20
74.83
38.88
62.97
SimMMDG (zero-filling filling)
73.97
75.94
51.10
67.00
SimMMDG (translation)
77.51
77.15
53.03
69.23
SimMMDG (zero-filling)
79.60
76.93
50.09
68.87
SimMMDG (translation)
79.67
77.15
51.29
69.37
DeepAll (Audio-only)
29.99
37.75
23.35
30.36
SimMMDG fero--fllligg)
26.05
15.99
22.40
SimMMDG (translation)
28.26
39.74
22.15
30.05
SimMMDG (zero- fillini
31.36
38.63
26.65
32.21
SimMMDG (translation)
33.81
40.95
27.02
33.93
DeepAll (Flow-only)
54.87
56,62
40.25
50.58
SimMMDG (zero-filling filling)
54.79
51.43
40.44
48.89
SimMMDG (translation)
59.12
60,38
41.18
53.56
SiniMNDD (zero-filiing
X
57.75
60.38
44.12
34,08"
SimMMDG (translation)
x
58.54
61.26
42.65
54.15
SimMMDG
(zero-filling)
$1.21
41.45
46.84
SimMMDG (translation)
52.05
61.70
41.08
51.61
SimMMDG (zero-filling)
80.32
79.36
55.88
71.85
SimMMDG (translation)
79.24
78.26
53.49
70.33
SimMMDG (zero-filling
V
76.64
79.14"
53.31
69.70
SimMMDG (translation)
77.58
79.91
55.88
71.12
SimMMDG (zero-filling)
X
80.39
79.03
$2.67
70.70
SimMMDG (translation)
x
79.60
78.48
54.04
70.71
SimMMDG (Zero-fillin)
X
/
X
22.71
32,45
23.62
26.26
SimMMDG (translation)
/
30.28
41.83
29.60
33,90
SimMMDG (zero-filling. filling)
X
X
56.02
57,40
38.97
50.80
SimMMDG (translation)
X
57.25
61.04
39.15
52.48
Table 13: Multi-modal multi-soure DG with missing modalities HAC dataset. X means the
modality is available during training, but is missing in test time.
Method
Video Audio D2, D3 - DI DI, D3 - D2 DI, D2
D3 Mean
DeepAll (Video-only)
$1.03
59.87
56.57
55.82
DeepAll (zero-tilling)
X
45.98
53.73
52.87
50.86
DeepAll (translation)
x 47.82
54.27
54.83
52.31
MM-SADA [52] (zero-fillin)
48.97
58.00
58.52
55.16
MM-SADA (translation)
50.80
58.27
58.93
56.00
RNA-Net [37] (zero-filling)
52.41
54,80
53.08
53.43
RNA-Net [57] (translation)
54.25
58.13
55.85
56.08
SIMMMDG (Zero-filling)
x
55.40
"67.00
58732
39.24
SimMMDG (translation)
x
$7.24
65.07
59.65
60.65
DeepAll (Audio-only)
32.87
42.27
45.17
40.10
DeepAll (zero-filling)
X
37.70
37.47
46.71
40.63
DeepAll (translation)
X
38,39
42.13
50.10
43.54
MM-SADA [52) (zcro-älling)
X
36.09
40.13
46.51
40.91
MM-SADA [52] (translation)
37.01
41.33
47.95
42.10
RNA-Net [57] (zero-filling)
40.27
46.71
40.95
RNA-Net [57] (translation)
42.53
48.97
43.60
SimMMDG (zero-fillig)
29.20
37.33
45.69
37.41
SimMMDG (translation)
37.70
46.40
52.05
45.38
Table 14:
Multi-modal multi-source DG with missing modalities on EPIC-Kitchens dataset for
different baselines. X means the modality is available during training, but is missing in test time.
replacing the features of missing modalities with the translated ones from other available modalities achieves better performance.
Statistical Significance Tests. We run each experiment three times using different seeds for multi- modal multi-source DG on the EPIC-Kitchens dataset and then calculate the mean and standard deviation to show the statistical significance of our methods. As shown Tab. 15, our framework statistically stable and surpasses the baselines significantly.
19

---

Method
D2, D3 + D1 D1, D3
D2
DI. D2
D3
Mean
DeepAll
48.81 + 1.77
65.78
57.45
- 1.19 54.01 0.58
MM-SADA [52]
48.89 0.43
59.15
59.31
- 0.81 55.79
0.69
RNA-Net [57]
51.26 3.07
58.54
0.82
59.14
1.24
56.31 1.22
SimMMDG (ours)
57.70 0.50
67.33
+ 0.99
64.41
+ 1.36 63.15 0.88
Table 15: Statistical significance tests for multi-modal multi-source DG on using video and audio.
dataset
treun yrond
(a) Embedding Video
(b) Embedding of Audio
(c) Embedding of Audio
-5
-30
(d) Splitted Embedding of Video
(0) Splitted Embodding of Audio
Figure 6: Visualization of the learned embeddings using t-SNE (D1, D2 D3 in EPiC-Kitchens for multi-modal multi-source DG).
F
Visualization
We show more visualizations of the learned embeddings using t-SNE [64] in Fig. 6. We can observe that the embeddings of video, audio, and the concatenation of video and audio are all well aligned for the source and target domains. Besides, the modality-speciftt and modality-shared embeddings are also well-separated and aligned across domains.
G More Intuitions behind Cross-modal Translation Module
In our SimMMDG framework, we introduce a cross-modal translation module to further regularize the learned features and facilitate behind this module.
modality generalization. Here we give more intuitions
Cross-modal translation won't undermine the unique features of different modalities. We want to learn an MLP projection to translate the embedding E' of the i-th modality the embedding E' of the j-th modality. We add a translation loss make the translated embedding E to be close to E without any explicit alignment or constraints on E' and E². We apply the cross-modal translation on the integrated feature of each modality E' E EA], which is the concatenation of modality-speciffi feature E, and modality-share feature E.. We still enforce distance loss on E, and E, at the same time. Therefore, the modality-specific and modality-shared features are still forced to be separated during the training progress. The embedding visualizations shown in Fig. 6 (d) and (e) also indicate that for both video and audio, their modality-specific and modality-shared features are well disentangled and are not influenced by the cross mods- translation module.
20

---

R@1 1 R@5 R@ 10
R@1 1 R@5 R@ 10
Modality-specific Features
11.80 45.37 49.78
Modalityspeciifc Features
10.24 33.01 47.10
Modality-hhared Features
76.58 90.13 92.83
Modality-shane Features 67.59 89.60 93.99
Table 16: Video Audio Retrieval.
Table 17: Audio to Video Retrieval.
Cross-modal translation is a type of modalities interaction, where different modality elements interact give rise to new information when integrated together for task inference [46]. Several works [23, 55] have already demonstrated that modality interaction can help improve the performance of multi-modal tasks. For example, the proposed approach in [23] learns a coordinated similarity space between image and text to improve image classification. The approach proposed in [55] translates language into video and audio for language sentiment analysis.
Cross-modal translation can be thought of as a means to leverage the information from one modality to infer much information possible for the target modality. Just like when hear a dog barking, we will fill in the picture of the puppy in our mind, and when we see foreign language, we will automatically translate it into our native language. Although there is information loss and we can't 't recover all the details during this translation progress, we can still infer some useful information for the target modality. As shown in our ablation study in Tab. 5, adding the cross-modal translation module indeed improves multi-modal DG performance.
More importantly, cross-modal translation module can be used for improving missing- modality generalization, by filling in the features of missing modality with the features in- ferred/translated from the available modality. The benefit of adopting cross-modal translation demonstrated in Tab. 4. Our approach is robust even when two modalities out of three are missing. This indicates that the inferred information from the cross-modal translation module is valuable and useful for downstream tasks.
H Further Discussions on Modality-speciffe and Modality-shared Features
Based on our assumptions in the paper, modality-shared features reflect shared information between all modalities, like all modalities that describe the same people, objects, actions, or gestures. Modality- specific features are specific pieces of information that are unique to each modality, like texture, depth, and visual appearance in images, syntactic structure, vocabulary, and morphology in language. Our features of different modalities from different source domains
goal is to align the modality-shane with the same label to be as close as possible the embedding space, while pushing away features with different labels, to make the embedding space more distinctive. At the same time, we want the sepecitt-specificc features to be as far as possible from the modality-shared features, such that they carry different types of information.
We analyzed the meaningfulness and the ability to share features of modality-shared features by cross-modal retrieval task. The ability of cross-modal retrieval is highly related to the shareable of features, as only features that are meaningful, shareable, and highly connective can give good recall rate. We use codallty-sshaed features of videos to retrieve from the modality-shared features of audios and calculate the recall, and vice versa for audios. We report the R@ 1, R @@5, and R@ 10 values for Video to Audio Retrieval and Audio to Video Retrieval. For example, R 5 for Video to Audio Retrieval means we retrieve candidates from audio, if at least one of them has the same label as the query video, we consider the retrieval success and then we calculate the average success rate for all query videos. We also do the same thing on modality-speeciff features: using features of videos to retrieve from the modality-speciff features of audios, and vice versa for audios. As shown in Tab. 16 and Tab. 17, we have a very high recall rate when we use modality-shared features, while the recall rate is biased towards random when use modality-speciff features. This further indicates that the modality shared features are truly shareable and meaningful and there are very strong relations and connections across modalities. The modality-spechi features are instead with more information that is private to each single modality.
To further verify the meaningfulness of modality-shared features, we also train classifier only on the concatenation of modality-share features of video and audio, and discard the modality-specine features. As shown in Tab. 18, without modality-specie features, the performances drop slightly
21

---

Method
D2, D3 + D1 D1, D3 + D2 D1, D2 - D3 Mean
SimMMDG (modality-shared fcatures only) SimMMDG
54.71
64.67 65.47
62.83 66.32
60.74
57.93
63.24
Table 18: Multi-modal multi-source DG on EPIC-Kitchens dataset using only modality-share features.
compared to the whole framework, but are still competitive. This indicates that modality-shared features truly have some meaningful information that can be used for prediction tasks.
22


Reviews:
[{"review_id": "HaipjfjADH", "reviewer": "Reviewer_93nE", "summary": "The paper tackles the problem of multimodal domain generalization in which the goal is to learn a model from multi-modal multi/single-source domain data that can generalize to unseen multimodal distributions. The main motivation is that mapping data from multiple modalities to a shared embedding space only harnesses the common features among different modalities while overlooks the modality-specific information. The paper proposes three main technical contributions to tackle the aforementioned problem and address the multimodal domain generalization problem. At first, each modality features are split into modality-shared and modality-specific components and then a supervised contrastive loss is used for alignment. Next, a distance loss is leveraged to repel the modality-specific features from each modality. Also, a cross-modal translation module is proposed to regularize the learned features and facilitate the missing modality issue. Finally, the paper also introduces a new multimodal DG dataset, namely Human-Animal-Cartoon (HAC). Results on EPIC-kitchens and HAC datasets claim improved DG performance under both multi-source and single-source settings.", "questions": "The paper tackles a challenging and a relatively new problem of how to learn a cross-domain generalizable model under multi-modal settings. Overall, the paper presents an effective framework comprised of three components and alongside a new multimodal dataset, however, there are some important questions/concerns as listed in weaknesses (1-5) due to which my initial rating for the paper is ‘weak accept’. \n\nAlso:\n\nIt would be interesting to know the performance of the method under:\n\na) different types of domain shifts, such as texture, background etc.\n\nb) increasing domain shifts.\n\nAre the results reported as the average of different trials and if yes, what are the standard deviations?\n\n\n\n\n\n", "limitations": "The paper mentions one limitation regarding the increasing complexity of cross-modal translation module with increasing number of modalities. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The problem of multimodal domain generalization is relatively new and mostly underxplored and carries practical relevance because in many real-world scenarios the data is often observed via multiple modalities.\n\n2) The cross-modal translation module is capable of regularizing the learned features and handle missing modalities issue.\n\n3) The paper proposes a new Human-Animal-Cartoon (HAC) dataset, which consists of different action classes and three different domains, to facilitate research in multimodal DG.\n\n4) Results on two different datasets and under both multi-source and single-source DG setting show that the proposed framework provides notable gains over the baselines and existing methods.", "weaknesses": "1) The paper seem to be missing the domain generalization aspect, like which pertinent DG problem in the context of multimodal data the paper is trying to solve, because all the technical contributions mostly aim at either bringing the modality-shared component or repel the modality-specific component which seems effective compared to the only relevant baseline [44]. \n\n2) The two important technical contributions i.e. learning shared embedding space and pushing modality-specific feature is not very new in the context of DG and even broader scope [A],[B]. This is also because the supervised contrastive loss [29] as well as the distance based loss are the existing components taken from literature.\n\n3) The unimodal DG comparison is made with rather old approaches RSC [23] and MixUp [55] which weakens the claim of significant performance gain. There are several new unimodal DG approaches which have shown state-of-the-art performances such as [C], [D] and [E].\n\n4) Fig. 2 that displays the overall architecture of the proposed method is not very helpful towards grasping the mid-level idea. For instance, one thing is to improve the patterns and colors. Also, the some symbols from the text in methodology should be used in the diagram to better connect the two.\n\n5) Details on how the HAC datsets is being developed is missing. For instance, how many volunteers were involved? what were the quality assurance mechanisms in place? I was expecting these details at least in the supplementary material, but the supp material only provides some high-level statistics of the dataset such as number of videos or to a little bit class-imbalance issue.\n \n[A] Bui, M.H., Tran, T., Tran, A. and Phung, D., 2021. Exploiting domain-specific features to enhance domain generalization. Advances in Neural Information Processing Systems, 34, pp.21189-21201.\n\n[B] Chattopadhyay, P., Balaji, Y. and Hoffman, J., 2020. Learning to balance specificity and invariance for in and out of domain generalization. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16 (pp. 301-318). Springer International Publishing.\n\n[C] Cha, J., Chun, S., Lee, K., Cho, H.C., Park, S., Lee, Y. and Park, S., 2021. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34, pp.22405-22418.\n\n[D] Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P. and Cord, M., 2022. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35, pp.10821-10836.\n\n[E] Rame, A., Dancette, C. and Cord, M., 2022, June. Fishr: Invariant gradient variances for out-of-distribution generalization. In International Conference on Machine Learning (pp. 18347-18377). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful reviews, and we appreciate your valuable suggestions! We address your concerns and questions as follows:\n>**Q1**: The paper seem to be missing the domain generalization aspect, like which pertinent DG problem in the context of multimodal data the paper is trying to solve, because all the technical contributions mostly aim at either bringing the modality-shared component or repel the modality-specific component which seems effective compared to the only relevant baseline [44].\n\n**A1**: This paper aims to solve the general multi-modal DG problem, an extension of traditional DG with multiple modalities as input. In this setup, we train the model on source domains with at least two modalities as input data, and test on target domains that have the same modalities but have large distribution shifts compared to the source domains.\n\nDue to the simplicity of our framework and the proposed cross-modal translation module, our method can also be extended to multi-modal single-source DG and missing-modality DG directly without any modification. \n\nMulti-modal DG is an under-explored problem. [44] is the only known work trying to solve it and is, therefore, the only relevant baseline. However, we also selected several strong multi-modal learning baselines, such as Gradient Blending [53] and MM-SADA [40], in order to verify the effectiveness of the proposed framework.\n___\n>**Q2**: The two important technical contributions i.e. learning shared embedding space and pushing modality-specific feature is not very new in the context of DG and even broader scope [A],[B]. This is also because the supervised contrastive loss [29] as well as the distance based loss are the existing components taken from literature.\n\n**A2**: [A,B] learn invariant and specific features at the domain level, while our proposed approach learns at the modality level. This is a key difference compared to previous schemes and, thus, a major contribution of this work. In future work, it would be also interesting to combine [A,B] with our proposed framework to learn disentangled features at both domain and modality levels. \n\nTraditional multi-modal learning frameworks use contrastive loss [29] on integrated features of different modalities, while our proposed approach only uses contrastive loss on modality-shared features and adds a distance loss on modality-specific features in order to learn complementary information from different modalities. \n\nIn addition to these contributions, we address the missing-modality generalization problem with our proposed cross-modal translation module, which is robust even in scenarios where multiple modalities are missing. Moreover, we also provide theoretical insights in support of the efficacy of our proposed approach.\n___\n\n>**Q3**: The unimodal DG comparison is made with rather old approaches RSC [23] and MixUp [55] which weakens the claim of significant performance gain. There are several new unimodal DG approaches which have shown state-of-the-art performances such as [C], [D] and [E].\n\n**A3**: In response to the reviewer's comment, we add experiments with one of the suggested SOTA unimodal DG baselines: Fishr [E] (ICML, 2022), as shown in the table below. Fishr outperforms RSC and Mixup in most cases. However, it still yields a large classification accuracy gap with an average of 7.74\\% compared to our multi-modal DG method. These additional results further emphasize the contribution of our proposed multi-modal DG framework.\n\n| Method  | D2,D3→D1 | D1,D3→D2 | D1,D2→D3 | mean |\n|---------|----------|----------|----------|------|\n| RSC (V) |50.11    |      62.53    |       58.73   |  57.12    |\n| RSC (A) |36.55    |    43.73      |      48.15    |    42.81  |\n| RSC (F) |55.17   |        63.33  |     59.65     |  59.38    |\n| Mixup  (V) |49.20     |     59.73     |    59.96      |    56.30  |\n| Mixup  (A) |35.17      |        40.80  |     45.07     |   40.35   |\n| Mixup  (F) |56.32    |    65.60      |      54.62    |     58.85 |\n| Fishr (V) |53.79   |     63.47     |    61.09      |  59.45    |\n| Fishr (A) |37.47   |   44.80       |   47.43       |    43.23  |\n| Fishr (F) |54.25    |   63.87       |      59.14    |   59.09   |\n| SimMMDG(V+A+F) |      **63.68**    |     **70.13**     |    **67.76**      |  **67.19**    |\n\n___\n\n>**Q4**: Fig. 2 that displays the overall architecture of the proposed method is not very helpful towards grasping the mid-level idea. For instance, one thing is to improve the patterns and colors. Also, the some symbols from the text in methodology should be used in the diagram to better connect the two.\n\n**A4**: Thanks for your suggestion! We will indeed change the patterns and colors and add the text symbols in the final version of the paper for better illustration.\n___\n\n>**Q5**: Details on how the HAC datsets is being developed is missing. \n\n**A5**: Due to space limits, we put more details on our HAC dataset in the **global response** at the top of this page.\n___\n\n>**Q6**: It would be interesting to know the performance of the method under: a) different types of domain shifts, such as texture, background etc. b) increasing domain shifts. Are the results reported as the average of different trials and if yes, what are the standard deviations?\n\n**A6**: In fact, the EPIC-kitchens and HAC datasets suffer different types of domain shifts. For EPIC-kitchens, the domain shifts come from different backgrounds, e.g. three kitchens from distinct countries with different layouts. The shifts of the HAC dataset mostly come from different textures, e.g. cartoons vs real-world videos. Our SimMMDG shows strong results under different types of domain shifts. Indeed, the results are reported as the average of three different trials (Table 7 in Supplementary Material) with the standard deviations lying within reasonable values, which indicates that our framework has statistical improvements compared to other baselines."}, {"review_id": "iWcYZyE1i5", "reviewer": "Reviewer_Zavq", "summary": "This paper proposes a method for multi-model domain generalization. Rather than mapping features of different modality into the same embedding space, it splits features within each modality into modality-specific feature and modality-shared feature. Then it applies different learning strategies to different split features. Besides, a new dataset called Human-Animal-Cartoon is proposed in this paper. Experiments a re conducted on two datasets and shows good performance. ", "questions": "See the weaknesses.", "limitations": "The cross-modal translation does not make sense to me. Will rise my rating if rebuttal can address my concern.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a novel method for multi-model domain generalization and a new dataset for this task. The motivation and method are clear to me. The results are convincing to me. ", "weaknesses": "I am confused to the section 3.2.2 Cross-model Translation. The design is translating the i-th modality feature to j-th modality feature. However, like the paper states that, different modality features contains share-features and unique features. Does the translation will undermine the unique features in different modalities? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful reviews, and we appreciate your valuable suggestions! We address your concerns and questions as follows:\n\n> **Q1:** I am confused to the section 3.2.2 cross-modal Translation. The design is translating the i-th modality feature to j-th modality feature. However, like the paper states that, different modality features contains share-features and unique features. Does the translation will undermine the unique features in different modalities?\n\n**A1:** \n\n1. **Cross-modal translation won't undermine the unique features of different modalities.** We want to learn an MLP projection to translate the embedding $\\mathbf{E}^{i}$ of the $i$-th modality to the embedding $\\mathbf{E}^{j}$ of the $j$-th modality. We add a translation loss to make the translated embedding $\\mathbf{E}^{j}_{t}$ to be close to $\\mathbf{E}^{j}$, without any explicit alignment or constraints on $\\mathbf{E}^{i}$ and $\\mathbf{E}^{j}$. We apply the cross-modal translation on the integrated feature of each modality $\\mathbf{E}^{i} = [\\mathbf{E}^{i}_s; \\mathbf{E}^{i}_c]$, which is the concatenation of modality-specific feature $\\mathbf{E}_s$ and modality-shared feature $\\mathbf{E}_c$. We still enforce a distance loss on $\\mathbf{E}_s$ and $\\mathbf{E}_c$ at the same time. Therefore, the modality-specific and modality-shared features are still forced to be separated during the training progress. The embedding visualization shown in Figure 4 (d) and (e) in the Supplementary Material also indicates that for both video and audio, their modality-specific and modality-shared features are well disentangled and are not influenced by the cross-modal translation module.\n\n2. **Cross-modal translation is a type of modalities interaction,** where different modality elements interact to give rise to new information when integrated together for task inference [1]. Several works [2,3] have already demonstrated that modalities interaction can help improve the performance of multi-modal tasks. For example, the proposed approach in [2] learns a coordinated similarity space between image and text to improve image classification. The approach proposed in [3] translates language into video and audio for language sentiment analysis.\n\n3. **Cross-modal translation can be thought of as means to leveraging the information from one modality to infer as much information as possible for the target modality.** Just like when we hear a dog barking, we will fill in the picture of the puppy in our mind, and when we see a foreign language, we will automatically translate it into our native language. Although there is information loss and we can't recover all the details during this translation progress, we can still infer some useful information for the target modality. As shown in our ablation study (Table 5 in the main paper), adding the cross-modal translation module indeed improves multi-modal DG performance. \n\n4. More importantly, our **cross-modal translation module can be used for improving missing-modality generalization,** by filling in the features of missing modality with the features inferred/translated from the available modality. The benefit of adopting cross-modal translation is demonstrated in Table 4 of the main paper. Our approach is robust even when two modalities out of three are missing. This indicates that the inferred information from the cross-modal translation module is valuable and useful for downstream tasks.\n\n[1] Paul Pu Liang, et al. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions, arXiv preprint arXiv:2209.03430, 2022.\n\n[2] Andrea Frome, et al. Devise: A deep visual-semantic embedding model. In NeurIPS, 2013.\n\n[3] Hai Pham, et al. Found in translation: Learning robust joint representations by cyclic translations between modalities. In AAAI, 2019. "}, {"review_id": "yl5KXWhEHX", "reviewer": "Reviewer_kbvb", "summary": "This paper presents a framework for domain generalization for models trained with multi-modal/ multi-source data: SimMMDG. The approach is motivated by the observation that each modality contains features shared with other modalities as well as features specific to that modality. The paper proposes to split per-modality features such that modality-shared features should be similar while modality-specific features should be complementary. A multi-loss strategy is adopted to implement this idea. The author evaluates the approach's performance on EPIC-Kitchen and a proposed  Human-Animal-Cartoon dataset.", "questions": "See weaknesses", "limitations": "Author should discuss the potential impact regarding their newly proposed dataset HAC.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach, SimMMDG seems simple yet novel. It relies on a simple intuition that modalities should have modality-specific features, which had been overlooked in previous literature.  The approach seems very general and has the potential to become a popular framework used by any researcher working with multi-modal representation learning. Its simplicity also enables combining this framework with previous approaches.\n- On the empirical results, the method seems to provide non-trivial gain over baselines and previous approaches targeting multi-modal supervised classification. \n- The paper is clearly written; it feels one can directly implement the proposed idea without too much trouble.", "weaknesses": "- Despite how much I appreciate the idea proposed by the paper, some important experiments are missing, which leads me to question the technical soundness of the work. For example, the learning rate of CL (supervised contrastive loss) is set to 3.0; this essentially increased the effective learning rate of the model significantly. I wonder if authors have tuned the baseline learning rate accordingly to ensure that the performance gain is not influenced by the learning rate change. In Table5, it seems all results adds CL; what if we remove CL, can CT effectively learn similarly as CL? In addition, the work is often motivated with the recent use of text + vision, but the evaluation does not include any text+vision task. \n- Impact of the work. The proposed framework seems very general, but the author eventually chose to test only on Domain Generalization. This seems to undersell the work, since it seems feasible to implement this idea to perform multi-modal classification tasks in a generic fashion. I wonder if authors have tried more evaluation apart from domain generalization, and if the methods provide improvement there.\n- Lack of discussion on HAC. One of the contribution of the paper is the HAC dataset. This is a new evaluation benchmark, yet studies/ analysis/ details on this dataset is missing. I also checked the supplementary of the paper, but found very limited information on this benchmark, such as dataset statistics, how are videos selected, how is the annotation performed, is there any QA adopted. Without details and studies (e.g. comprehensive baseline evaluation) on HAC, it will be hard to count HAC as a sound contribution and trust evaluation on this new benchmark. \n- SimMMDG with other baseline methods. A major contribution author claims is on the simplicity. This makes me wonder if the pipeline is actually compatible with many of the baseline adopted, such as Gradient Blending [53] and Non-Local [54]. I wonder if authors have tried their approach with these methods to show if the proposed framework is complementary to existing techniques.\n\nIn general, although I really enjoy reading this paper and likes the intuitions, it seems the work needs more polish and empirical evaluation to prove its soundness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful reviews, and we appreciate your valuable suggestions! We address your concerns and questions as follows:\n\n>**Q1**: The learning rate of CL is set to 3.0; this essentially increased the effective learning rate of the model significantly. I wonder if authors have tuned the baseline learning rate accordingly to ensure that the performance gain is not influenced by the learning rate change.\n\n**A1**: We apologize for the confusion. In fact, 3.0 is not the learning rate (lr) of CL, but the weight to balance different loss terms, as indicated in formula (5) in the main paper. For SimMMDG, we choose a lr of 0.0001 for all experiments by grid search. For each baseline method, we also tune the lr using grid search and select the one with the best performance.\n___\n\n>**Q2**: In Table5, it seems all results adds CL; what if we remove CL, can CT effectively learn similarly as CL?\n\n**A2**: We additionally evaluated the contributions of the supervised contrastive loss (CL) and report the results below. \n|Method|D2,D3→D1|D1,D3→D2|D1,D2→D3|mean|\n|---------|----------|----------|----------|------|\n|SimMMDG w/o CL|54.25|63.47|63.04|60.25|\n|SimMMDG|**57.93**|**65.47**|**66.32**|**63.24**|\n\nAlthough SimMMDG without CL is already better than most baselines, it still has a performance gap compared with the whole framework, which means CT is helpful but cannot replace CL. The effect of CL is similar to explicit feature alignment. It aligns the modality-shared features of different modalities from different source domains with the same label to be as close as possible in the embedding space while pushing away features with different labels, to make the embedding space more distinctive.\n___\n\n>**Q3**: In addition, the work is often motivated with the recent use of text + vision, but the evaluation does not include any text+vision task.\n\n**A3**: Text is not a usual modality in DG setup and there is no public benchmark on text-based DG, not to mention multi-modal DG. Therefore, we don't include any text+vision DG task. However, as you suggest, we indeed added several multi-modal classification tasks on text+video+audio below.\n___\n\n>**Q4**: I wonder if authors have tried more evaluation apart from domain generalization, and if the methods provide improvement there.\n\n**A4**: Thanks for your suggestion! Following your suggestion, we evaluated our framework in a more general multi-modal classification setup without DG. We first evaluated the proposed framework on the EPIC-Kitchens dataset without including the DG setup. For that, we mixed the data from three domains together and split the training, validation, and testing data accordingly, such that there are no apparent domain shifts between training and testing data. The results indicate that our proposed SimMMDG has obvious advantages in the general multi-modal classification setup.\n\n|  |EPIC-Kitchens| \n|---------|----------|\n|DeepAll|64.89| \n|TSN [1]|66.33|   \n|TBN [28]|73.18|  \n|Gradient Blending [53]|73.97|   \n|SimMMDG|**76.42**|  \n\nWe further evaluate our framework on two multi-modal datasets, namely MUSTARD [2] and UR-FUNNY [3] in MultiBench [4]. These two datasets are for human sentiment analysis and involve language, video, and audio modalities. We use the code base in MultiBench for experiments and implement SimMMDG on that. MultiBench treats human sentiment analysis as a regression task, while our framework is designed for classification. Therefore, we modified the code base to fit the classification task. For all baselines, we chose the same backbone and only change the fusion paradigms. Our SimMMDG shows obvious advantages and outperforms the previous SOTA methods with an average gain of 7.3\\% and 1.7\\%. This indicates that our SimMMDG can also be used as a general framework for multi-modal classification and is compatible with different combinations of modalities (video+audio+flow, language+video+audio). The results further strengthen the contributions of our proposed approach.\n\n|   | MUSTARD | UR-FUNNY | \n|---------|----------|----------|\n|Late Fusion|61.6|63.6|      \n|Low-rank Tensor Fusion [5]|65.2|63.9|    \n|MULT [6]|60.9|63.2|     \n|SimMMDG|**72.5**|**65.6**|  \n\n___\n\n>**Q5**: Lack of discussion on HAC.\n\n**A5**: Due to space limits, we put more details on our dataset in the **global response** at the top of this page.\n___\n\n>**Q6**: SimMMDG with other baseline methods. \n\n**A6**: Following your suggestion, we combined SimMMDG with Gradient Blending and the results are further improved. As suggested by Reviewer c1hR, we also combine SimMMDG with a Domain-adversarial Neural Network (DANN) [7] to align the features of source domains using domain labels and observe a similar improvement. This indicates that our SimMMDG can be seamlessly combined with other training strategies due to its general and simplicity to get even better performance improvements.\n| Method  | D2,D3→D1 | D1,D3→D2 | D1,D2→D3 | mean |\n|---------|----------|----------|----------|------|\n| SimMMDG |57.93|65.47|66.32|63.24|\n| SimMMDG+Gradient Blending |59.31|68.40|66.63|**64.78**|\n| SimMMDG+DANN |60.69|66.93|64.58|**64.07**|\n___\n[1] Limin Wang, et al. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016. \n\n[2] Santiago Castro, et al. Towards multimodal sarcasm detection. In ACL, 2019. \n\n[3] Md Kamrul Hasan, et al. Ur-funny: A multimodal language dataset for understanding humor. In EMNLP-IJCNLP, 2019.\n\n[4] Paul Pu Liang, et al. MultiBench: Multiscale Benchmarks for Multimodal Representation Learning. In NeurIPS, 2021.\n\n[5] Zhun Liu, et al. Efficient low-rank multimodal fusion with modality-specific factors. In ACL, 2018.\n\n[6] Yao-Hung Hubert Tsai, et al. Multimodal transformer for unaligned multimodal language sequences. In ACL, 2019.\n\n[7] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015."}, {"review_id": "UaiVAPZd8r", "reviewer": "Reviewer_c1hR", "summary": "This paper addresses the challenge of domain generalization in a multi-modal setting. The core idea is that aligning features from multiple modalities in a shared embedding space can hinder generalization. To overcome this limitation, the paper proposes a more effective approach: it separates the features within each modality into modality-specific and modality-shared components and then aligns only the modality-specific components. This alignment is achieved through supervised contrastive learning and cross-modal feature translation techniques. To validate the proposed approach, the paper introduces a new dataset called HAC, which consists of video, audio, and optical flow data from three domains: Human, Animal, and Cartoon. Extensive experiments conducted on both the HAC dataset and the EPIC-Kitchens dataset demonstrate the impressive performance of the proposed method when compared to existing approaches.", "questions": "1. Can the usage of domain labels and explicit domain alignment strategies improve the generalization performance?\n2. Can you clarify the contradiction pointed out in Weakness-1?", "limitations": "The authors adequately addressed the limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Multi-modal domain generalization is an important practical problem and the proposed method and dataset introduced in this paper will help in advancing research in this direction.\n2. From a methodological standpoint, the idea of splitting the features into modality-specific and modality-shared and aligning them separately seems new. Interestingly, this goes against the strategy followed by popular multi-modal models such as CLIP.\n3. The paper showcases a comprehensive range of experiments to validate the effectiveness of the proposed method. It explores various combinations of modalities (e.g., Audio+Video, Audio+Video+OpticalFlow), different numbers of training domains, and even evaluates performance in scenarios with missing modalities during inference. This thorough evaluation provides a robust understanding of the method's capabilities and limitations.\n4. The paper is very well written and easy to follow. The diagram in Figure 2 is especially well done and effectively illustrates the framework in a concise manner.", "weaknesses": "1. The paper highlights an apparent contradiction in its argument regarding modality-specific information. On one hand, it acknowledges that certain modalities, such as video and flow, possess distinct and unalignable features, such as lighting and motion information. However, the paper introduces a cross-modal transformation module that seemingly enables seamless transformation between these features. This raises a critical question: Does this transformation not contradict the initial premise that modality-specific information is inherently unalignable?\n2. The paper notably overlooks the utilization of domain labels or any explicit feature alignment techniques across domains. This omission raises concerns about the effectiveness of the proposed method in achieving robust domain generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful reviews, and we appreciate your valuable suggestions! We address your concerns and questions as follows:\n\n>**Q1**: The paper highlights an apparent contradiction in its argument regarding modality-specific information. On one hand, it acknowledges that certain modalities, such as video and flow, possess distinct and unalignable features, such as lighting and motion information. However, the paper introduces a cross-modal transformation module that seemingly enables seamless transformation between these features. This raises a critical question: Does this transformation not contradict the initial premise that modality-specific information is inherently unalignable?\n\n**A1**: We don’t aim at aligning modality-specific features across modalities. Instead, we focus on learning an approximate mapping from the embedding space of one modality to another.\n\n1. **Cross-modal translation won't undermine the unique features of different modalities.** We want to learn an MLP projection to translate the embedding $\\mathbf{E}^{i}$ of the $i$-th modality to the embedding $\\mathbf{E}^{j}$ of the $j$-th modality. We add a translation loss to make the translated embedding $\\mathbf{E}^{j}_{t}$ to be close to $\\mathbf{E}^{j}$, **without any explicit alignment or constraints on $\\mathbf{E}^{i}$ and $\\mathbf{E}^{j}$.** We apply the cross-modal translation on the integrated feature of each modality $\\mathbf{E}^{i} = [\\mathbf{E}^{i}_s; \\mathbf{E}^{i}_c]$, which is the concatenation of modality-specific feature $\\mathbf{E}_s$ and modality-shared feature $\\mathbf{E}_c$. We still enforce a distance loss on $\\mathbf{E}_s$ and $\\mathbf{E}_c$ at the same time. Therefore, the modality-specific and modality-shared features are still forced to be separated during the training progress. The embedding visualization shown in Figure 4 (d) and (e) in the Supplementary Material also indicates that for both video and audio, their modality-specific and modality-shared features are well disentangled and are not influenced by the cross-modal translation module.\n\n2. **Cross-modal translation is a type of modalities interaction,** where different modality elements interact to give rise to new information when integrated together for task inference [1]. Several works [2,3] have already demonstrated that modalities interaction can help improve the performance of multi-modal tasks. For example, the proposed approach in [2] learns a coordinated similarity space between image and text to improve image classification. The approach proposed in [3] translates language into video and audio for language sentiment analysis.\n\n3. **Cross-modal translation can be thought of as means to leveraging the information from one modality to infer as much information as possible for the target modality.** Just like when we hear a dog barking, we will fill in the picture of the puppy in our mind, and when we see a foreign language, we will automatically translate it into our native language. Although there is information loss and we can't recover all the details during this translation progress, we can still infer some useful information for the target modality. As shown in our ablation study (Table 5 in the main paper), adding the cross-modal translation module indeed improves multi-modal DG performance. \n\n4. More importantly, our **cross-modal translation module can be used for improving missing-modality generalization,** by filling in the features of missing modality with the features inferred/translated from the available modality. The benefit of adopting cross-modal translation is demonstrated in Table 4 of the main paper. Our approach is robust even when two modalities out of three are missing. This indicates that the inferred information from the cross-modal translation module is valuable and useful for downstream tasks.\n___\n\n>**Q2**: The paper notably overlooks the utilization of domain labels or any explicit feature alignment techniques across domains. This omission raises concerns about the effectiveness of the proposed method in achieving robust domain generalization?\n\n**A2**: The main goal of the paper is to propose a general framework for multi-modal (multi-/single- source) DG, since in many real cases domain labels are not available. But the reviewer raises a valid point; when the domain labels are available, we can use explicit feature alignment techniques across domains to improve the generalization ability.\n\nFollowing your suggestion, we combined SimMMDG with a Domain-adversarial Neural Network (DANN) [4] to align the features of source domains using domain labels. This setup has resulted in a further improvement compared to our proposed setup (without using the domain labels). As suggested by Reviewer kbvb, we also combined SimMMDG with Gradient Blending [53] and observed a similar improvement. This indicates that our proposed SimMMDG can be seamlessly combined with other training strategies due to its general and simplicity to get even better performances.\n\n| Method  | D2,D3→D1 | D1,D3→D2 | D1,D2→D3 | mean |\n|---------|----------|----------|----------|------|\n| SimMMDG |57.93|65.47|66.32|63.24|\n| SimMMDG+Gradient Blending |59.31|68.40|66.63|**64.78**|\n| SimMMDG+DANN |60.69|66.93|64.58|**64.07**|\n___\n[1] Paul Pu Liang, et al. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions, arXiv preprint arXiv:2209.03430, 2022.\n\n[2] Andrea Frome, et al. Devise: A deep visual-semantic embedding model. In NeurIPS, 2013.\n\n[3] Hai Pham, et al. Found in translation: Learning robust joint representations by cyclic translations between modalities. In AAAI, 2019. \n\n[4] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015."}, {"review_id": "6s7oeF7cIG", "reviewer": "Reviewer_8EtQ", "summary": "This work focuses on addressing a special case of multimodal learning, i.e., multi-modal domain generalization, where one or more modalities will be absent during test phase. \n\nThis work propose a approach to solve multimodal DG from a disentanglement perspective.  Modality-specific features contain modality-specific information, while Modality-shared feature contains information that is shared across domain such as class labels. \n\nSupervised contrastive learning is employed to learn better modality-shared features while a distance-based regularize is used to enlarge the discrepancy between modality-specific and modality-shared features. In addition, a cross-modal translation module is proposed to enhance the modality-shared features.\n\nThis work also release a dataset named Human-Animal-Cartoon (HAC).", "questions": " ", "limitations": " ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work has following strengths:\n\n- This paper released a dataset, HAC, which is helpful for multimodal domain generalization community.\n\n- This paper is easy to follow.\n\n- Multi-modal DG is interesting and has research potential since real-world task is complicated that often involves multiple modalities.", "weaknesses": "This work has following weaknesses:\n\n- Is the name of the setting appropriate? Typically, domain generalization refer to a setting that testing domain is unseen during training. However, the setting in this work is that the testing domain/modality exists in training data but some training modalities might be missing.\n\n- Lack of proof that the network truly learn modality-specific and modality-shared features.\n\n- It's not clear that why Cross-modal Translation can enhance modality-specific features, since the MLP use both features for translation. How to guarantee the two types features are well disentangled.\n\n- The technique contribution of this work is somehow not strong. Supervised contrastive loss is a mature technique. What's the essence to use supervised contrastive loss here? Can CE loss obtain similar performance? Translation idea is also appeared in [1].\n\n[1] Ge, Yunhao, et al. \"Zero-shot synthesis with group-supervised learning.\" arXiv preprint arXiv:2009.06586 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful reviews, and we appreciate your valuable suggestions! We address your concerns and questions as follows:\n>**Q1**: However, the setting in this work is that the testing domain/modality exists in training data but some training modalities might be missing.\n\n**A1**: In our setting, the testing domain data is completely unseen during training and we only train our model on source domains. This is exactly the same setup as traditional DG, the only difference is that we take multiple modalities as input. In this setup, we train the model on source domains with at least two modalities as input data and test on target domains that have the same modalities but have large distribution shifts compared to source domains. \n\nBenefiting from the simplicity of our framework and the proposed cross-modal translation module, our method can also be used in multi-modal single-source DG and missing-modality DG directly without any modification.\n___\n>**Q2**: Lack of proof that the network truly learn modality-specific and modality-shared features. How to guarantee the two types features are well disentangled.\n\n**A2**: The feature disentanglement is ensured by the proposed distance loss, where we maximize the dissimilarity between modality-shared and modality-specific features. According to the embedding visualization shown in Figure 4 (d) and (e) in the Supplementary Material, for both video and audio modalities, their modality-specific and modality-shared features are well disentangled. One of the goals of our proposed framework is to maximize the dissimilarity between modality-specific and modality-shared features, which means the network truly learns such properties.\n___\n>**Q3**: It's not clear that why Cross-modal Translation can enhance modality-specific features.\n\n**A3**: \n1. **Cross-modal translation is a type of modalities interaction**, where different modality elements interact to give rise to new information when integrated together for task inference [1]. Several works [2,3] have already demonstrated that modalities interaction can help improve the performance of multi-modal tasks.\n\n2. **Cross-modal translation can be thought of as means to leveraging the information from one modality to infer as much information as possible for the target modality.** Just like when we hear a dog barking, we will fill in the picture of the puppy in our mind, and when we see a foreign language, we will automatically translate it into our native language. Although there is information loss and we can't recover all the details during this translation progress, we can still infer some useful information for the target modality. As shown in our ablation study (Table 5 in the main paper), adding the cross-modal translation module indeed improves multi-modal DG performance. \n\n3. More importantly, our **cross-modal translation module can be used for improving missing-modality generalization,** by filling in the features of missing modality with the features inferred/translated from the available modality. The benefit of adopting cross-modal translation is demonstrated in Table 4 of the main paper. Our approach is robust even when two modalities out of three are missing. This indicates that the inferred information from the cross-modal translation module is valuable and useful for downstream tasks.\n\n[1] Paul Pu Liang, et al. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions, arXiv preprint, 2022.\n\n[2] Andrea Frome, et al. Devise: A deep visual-semantic embedding model. In NeurIPS, 2013.\n\n[3] Hai Pham, et al. Found in translation: Learning robust joint representations by cyclic translations between modalities. In AAAI, 2019. \n___\n\n>**Q4**: The technique contribution of this work is somehow not strong.\n\n**A4**: Traditional multi-modal learning frameworks use contrastive loss (CL) on the integrated features of different modalities, while we only use CL on modality-shared features and add distance loss on modality-specific features to learn complementary information from different modalities. The effect of CL is similar to explicit feature alignment. It aligns the modality-shared features of different modalities from different source domains with the same label to be as close as possible in the embedding space while pushing away features with different labels, to make the embedding space more distinctive.\n\nWe evaluated additionally the contributions of the supervised contrastive loss (CL) and the results are reported below. Although SimMMDG without CL is already better than most baselines, it still has a performance gap compared with our proposed entire framework (which includes CL), which indicates the importance of CL.\n\n|Method|D2,D3→D1|D1,D3→D2|D1,D2→D3|mean|\n|---------|----------|----------|----------|------|\n|SimMMDG w/o CL|54.25|63.47|63.04|60.25|\n|SimMMDG|**57.93**|**65.47**|**66.32**|**63.24**|\n\nThe approach proposed in [1] **swaps** disentangled features in the latent space across examples, while we **translate** features from one modality to another. This is a key difference with respect to existing schemes and a main contribution of this work. Moreover, [1] disentangles features at the level of single data samples, e.g. identity, pose, and background. However, we disentangle features at the level of the different modalities, e.g. modality-shared and modality-specific features.\n\nWe also address the missing-modality generalization problem with our proposed cross-modal translation module, which is robust even in scenarios where multiple modalities are missing. We also provide theoretical insights in support of the efficacy of our proposed approach (section 4 in the main paper).\n\nWe show further findings in the global response that: **SimMMDG is a general framework for multi-modal classification tasks, not limited to domain generalization setup, and can be seamlessly combined with other training strategies to achieve better performances.**"}]

Author rebuttal:
>**Q1**: More details on HAC dataset.

**A1**: The detailed dataset statistics such as the name of each action, the number of each action segment, and training-testing splits are in the supplementary material. Here, we provide more details on how we collect the dataset.

Our dataset was collected by 5 volunteers. For the human domain, we collect the data by selecting actions from an existing Kinetics-600 dataset. We select approximately the same number of video clips for each action to ensure class balance. For the animal domain,  we collect data from YouTube by searching keywords like ‘animal sleeping’, 'animal eating', 'animal running', etc. To increase the diversity of the dataset, we also specify the animal type in the keywords, such as ‘cat sleeping’, 'dog eating', 'horse running', etc.  Each participant was asked to collect certain actions, such as participant A for ‘sleeping’ and ‘watching tv’, participant B for ‘eating’ and ‘drinking’, etc. For the cartoon domain, we collect data from popular cartoons like 'SpongeBob SquarePants', 'The Simpsons', 'Garfield and Friends', etc. Each participant was asked to collect from one or two cartoon series to avoid duplication. 

For each action, we annotated the start and end times in the video and then cut out video clips. The length of each video clip varies from 1s to 10s. Finally, we gathered the data from all volunteers and a separate person manually discarded unqualified data like duplicate videos, videos without audio data, noisy/wrong classes, etc. 

The main purpose of this dataset is to be used for multi-modal domain generalization research. Of course, our dataset can also be used for other applications like multi-modal learning and multi-modal domain adaptation. For comprehensive baseline evaluation on our new dataset, we also provided selected multi-modal DG baseline evaluations in the main paper (DeepAll, MM-SADA, RNA-Net, and SimMMDG). Here, we provide additional evaluations on two baselines including TSN[1] and TBN[28].

||A,C→H|H,C→A|H,A→C|mean|
|---------|----------|----------|----------|------|
|TSN[1]|63.23|71.96|41.82|59.00|
|TBN[28]|70.65|75.71|47.47|64.61|
___
>**Q2: More strengths of SimMMDG framework.**

**A2**: We have a deeper study on our SimMMDG framework based on the comments by Reviewer kbvb and c1hR and report our interesting findings here.

1. **SimMMDG is a general framework for multi-modal classification tasks, not limited to domain generalization setup.** Following the suggestion from Reviewer kbvb, we evaluated our framework in a more general multi-modal classification setup without DG. We first evaluated the proposed framework on the EPIC-Kitchens dataset (video+audio) without including the DG setup. For that, we mixed the data from three domains together and split the training, validation, and testing data accordingly, such that there are no apparent domain shifts between training and testing data. The results indicate that our proposed SimMMDG has obvious advantages in the general multi-modal classification setup.

|  |EPIC-Kitchens| 
|---------|----------|
|DeepAll|64.89| 
|TSN [1]|66.33|   
|TBN [28]|73.18|  
|Gradient Blending [53]|73.97|   
|SimMMDG|**76.42**|  

We further evaluate our framework on two multi-modal datasets, namely MUSTARD [2] and UR-FUNNY [3] in MultiBench [4]. These two datasets are for human sentiment analysis and involve language, video, and audio modalities. We use the code base in MultiBench for experiments and implement SimMMDG on that. MultiBench treats human sentiment analysis as a regression task, while our framework is designed for classification. Therefore, we modified the code base to fit the classification task. For all baselines, we chose the same backbone and only change the fusion paradigms. Our SimMMDG shows obvious advantages and outperforms the previous SOTA methods with an average gain of 7.3\% and 1.7\%. This indicates that our SimMMDG can also be used as a general framework for multi-modal classification and is compatible with different combinations of modalities (video+audio+flow, language+video+audio). The results further strengthen the contributions of our proposed approach.

|   | MUSTARD | UR-FUNNY | 
|---------|----------|----------|
|Late Fusion|61.6|63.6|      
|Low-rank Tensor Fusion [5]|65.2|63.9|    
|MULT [6]|60.9|63.2|     
|SimMMDG|**72.5**|**65.6**|  

2. **SimMMDG can be seamlessly combined with other training strategies to achieve better performances.** Following the suggestion from Reviewer kbvb, we combined SimMMDG with Gradient Blending and the results are further improved. As suggested by Reviewer c1hR, we also combine SimMMDG with a Domain-adversarial Neural Network (DANN) [7] to align the features of source domains using domain labels and observe a similar improvement. This indicates that our SimMMDG can be seamlessly combined with other training strategies due to its general and simplicity to get even better performance improvements.
| Method  | D2,D3→D1 | D1,D3→D2 | D1,D2→D3 | mean |
|---------|----------|----------|----------|------|
| SimMMDG |57.93|65.47|66.32|63.24|
| SimMMDG+Gradient Blending |59.31|68.40|66.63|**64.78**|
| SimMMDG+DANN |60.69|66.93|64.58|**64.07**|
___
[1] Limin Wang, et al. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016. 

[2] Santiago Castro, et al. Towards multimodal sarcasm detection. In ACL, 2019. 

[3] Md Kamrul Hasan, et al. Ur-funny: A multimodal language dataset for understanding humor. In EMNLP-IJCNLP, 2019.

[4] Paul Pu Liang, et al. MultiBench: Multiscale Benchmarks for Multimodal Representation Learning. In NeurIPS, 2021.

[5] Zhun Liu, et al. Efficient low-rank multimodal fusion with modality-specific factors. In ACL, 2018.

[6] Yao-Hung Hubert Tsai, et al. Multimodal transformer for unaligned multimodal language sequences. In ACL, 2019.

[7] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015.

Meta-review: The topic of multi-modal domain generalization is considered of interest, the effectiveness of the simple method is appreciated, and the proposed HAC dataset is valuable. The author-responses to the remaining reviewer concerns have been well received and post-rebuttal all reviewers are supportive of acceptance. The AC agrees and encourages the authors to include all the promised additions and experiments in the camera-ready paper.
Decision: Accept (poster)