Title: SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Domain Generalization

Multi-modal Learning

Distribution Shift

Out-of-distribution Generalization
Abstract: In real-world scenarios, achieving domain generalization (DG) presents significant challenges as models are required to generalize to unknown target distributions. Generalizing to unseen multi-modal distributions poses even greater difficulties due to the distinct properties exhibited by different modalities. To overcome the challenges of achieving domain generalization in multi-modal scenarios, we propose SimMMDG, a simple yet effective multi-modal DG framework. We argue that mapping features from different modalities into the same embedding space impedes model generalization. To address this, we propose splitting the features within each modality into modality-specific and modality-shared components. We employ supervised contrastive learning on the modality-shared features to ensure they possess joint properties and impose distance constraints on modality-specific features to promote diversity. In addition, we introduce a cross-modal translation module to regularize the learned features, which can also be used for missing-modality generalization. We demonstrate that our framework is theoretically well-supported and achieves strong performance in multi-modal DG on the EPIC-Kitchens dataset and the novel Human-Animal-Cartoon (HAC) dataset introduced in this paper. Our source code and HAC dataset are available at https://github.com/donghao51/SimMMDG.

SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization

Abstract

In real-world scenarios, achieving domain generalization (DG) presents significant challenges as models are required to generalize to unknown target distributions. Generalizing to unseen multi-modal distributions poses even greater difficulties due to the distinct properties exhibited by different modalities. To overcome the challenges of achieving domain generalization in multi-modal scenarios, we propose SimMMDG , a simple yet effective multi-modal DG framework. We argue that mapping features from different modalities into the same embedding space impedes model generalization. To address this, we propose splitting the features within each modality into modality-specific and modality-shared components. We employ supervised contrastive learning on the modality-shared features to ensure they possess joint properties and impose distance constraints on modality-specific features to promote diversity. In addition, we introduce a crossmodal translation module to regularize the learned features, which can also be used for missing-modality generalization. We demonstrate that our framework is theoretically well-supported and achieves strong performance in multi-modal DG on the EPIC-Kitchens dataset and the novel Human-Animal-Cartoon (HAC) dataset introduced in this paper. Our source code and HAC dataset are available at https://github.com/donghao51/SimMMDG.

1 Introduction

Domain generalization (DG) has received significant attention in the research community [66]. In real-world scenarios such as autonomous driving [13, 26], robotics [17], action recognition [16], and fault diagnosis [22], it is crucial that models trained on limited source domains perform well across novel target domains. To tackle distribution shift problems, numerous DG algorithms have been proposed, including domain-invariant feature learning [51], feature disentanglement [56], data augmentation [72], and meta-learning [41]. However, most of these algorithms are designed for unimodal data, such as images [40] and time series data [49]. More recently, the emergence of large-scale multi-modal datasets [16, 7] has highlighted the need to address multi-modal DG across multiple modalities, such as audio-video [36, 73], image-language [58, 33], and LiDARcamera [44, 18]. However, to date, only RNA-Net [57] has focused on the multi-modal DG problem, with a relative norm alignment loss proposed to balance the feature norms of audio and video.

Multi-modal DG is closely related to multi-modal representation learning [47]. Traditional multimodal contrastive learning frameworks [58, 43] aim to project the features of different modalities into a common embedding space. However, this approach may not be optimal as different modalities consist of both shared information that is consistent across modalities and unique information that is specific to each modality. Consequently, attempting to align all modalities together can be challenging and impractical. For instance, consider descriptions of the same object that use entirely different modalities, such as video, audio, and optical flow, as illustrated in Fig. 1 (a). All modalities share some

Information primarily in video

•

Visual appearance

•

Sequential relation

•

...

Information primarily in audio

•

Tone

•

Frequency

•

...

Information primarily in optical flow

•

Motion

•

Velocity

•

...

Information in all modalities

•

Described objects, actions

•

Illustrative gestures

•

...

H

uman

(b) HAC dataset

(a) Modality-specific and modality-shared information

A

nimal

C

artoon

information since they describe the same object, but each modality further provides modality-specific information. For example, videos typically convey visual appearance information and sequential relations between frames, while audio provides tone information that is closely linked to sentiment and frequency information for different sounds. Optical flow, on the other hand, offers more intuitive information on motions and associated velocities. Therefore, simply mapping features from different modalities into the same embedding space would preserve only modality-shared information while overlooking modality-specific details. This could result in the loss of modality-specific information and a decline in downstream task performance.

Effective multi-modal DG frameworks that can address the challenges outlined above are urgently needed. To this end, we propose SimMMDG - a Sim ple yet effective M ultiM odal DG framework that can be applied to two or more modalities. We first propose to split the feature embedding of each modality into modality-specific and modality-shared parts, ensuring that complementary information from different modalities is retained. We apply supervised contrastive learning on modality-shared features and incorporate distance constraints for modality-specific features to ensure that the learned representations are informative. Moreover, we introduce a cross-modal translation module to further regularize the learned features and facilitate missing-modality generalization. Some theoretical insights from both the multi-modal representation learning perspective and the domain generalization perspective are provided to demonstrate that our approach is well-motivated in theory. Finally, we release a novel H umanA nimalC artoon (HAC) dataset, to promote research in multi-modal domain generalization, as shown in Fig. 1 (b). Our contributions can be summarized as:

We propose a universal framework for multi-modal domain generalization that demonstrates both simplicity and effectiveness and outperforms existing methods on several challenging datasets. We also demonstrate its efficacy in general multi-modal classification setups.

We address the missing-modality generalization problem and propose a cross-modal translation module as a solution, which is robust even in scenarios where multiple modalities are missing.

We provide theoretical insights in support of the efficacy of our proposed approach.

We introduce a new multi-modal dataset that includes three modalities and large domain shifts, which can serve as a challenging benchmark for future research on multi-modal DG.

2 Related Work

Domain Generalization refers to the process of training a model on multiple source domains in order to generalize to unseen target domains. This task is more challenging than domain adaptation (DA) [67] as target domain data cannot be accessed during training. Prior research has identified three main categories of domain generalization methods [66], including data manipulation, representation learning, and learning strategies. Data manipulation techniques aim to improve generalization performance by increasing the diversity of training data. For example, Tobin et al. [61] use simulated environments for additional data generation, while Zhou et al. [75] train a domain transformation network using adversarial training to synthesize data from previously unseen domains. Mixup [72] generates new instances by performing linear interpolations between data and labels. Representation

learning methods seek to learn domain-invariant representations through the use of domain-adversarial neural networks [25, 42], explicit feature distribution alignment [63], and instance normalization [54]. Other approaches utilize learning strategies to improve the generalization performance. Examples of such approaches include meta-learning [41], gradient operation [31], and self-supervised learning [8].

Multi-modal Representation Learning aims to learn robust representations through two or more modalities, which can then be applied to various downstream tasks. While unified models [68, 4] tokenize various input modalities into sequences [3] and employ a single Transformer [65] for joint learning, methods such as CLIP [58] and ALIGN [33] use separate encoders for each modality and leverage contrastive loss to align features. However, these methods align features from different modalities into the same embedding space and may only preserve modality-shared information. In contrast, we propose splitting the features of each modality into modality-specific and modalityshared components to ensure that complementary information from different modalities is preserved. A recent work [34] also proposes to split the features into different components in multi-modal representation learning and provide an information-theoretical analysis. However, they only deal with two modalities and without considering missing-modality cases. They also don't use the label information within a batch in contrastive learning.

Multi-modal DA and DG. Several approaches exist for multi-modal DA. For instance, Munro and Damen [52] propose a self-supervised alignment approach along with adversarial alignment for multi-modal DA, while Kim et al. [38] leverage cross-modal contrastive learning to align cross-modal and cross-domain representations. Zhang et al. [73] propose an audio-adaptive encoder and an audio-infused recognizer to address domain shifts. Notably, RNA-Net [57] is the only method known to address multi-modal DG problem, by introducing a relative norm alignment loss to balance audio and video feature norms.

3 Methodology

3.1 Problem Setting: Multi-modal Domain Generalization

We commence by presenting the definition of the multi-modal domain generalization problem based on the definition of unimodal DG, as described in [66]. Let X represent a nonempty input space, and Y denote an output space. A domain, denoted as D , consists of data sampled from a distribution, D = { ( x j , y j ) } n j =1 ∼ P XY , where P XY denotes the joint distribution of input samples and output labels. X and Y represent the corresponding random variables.

̸

̸

Definition 1 (Multi-modal domain generalization) . In multi-modal domain generalization, we are given D training domains D train = {D i | i = 1 , · · · , D } , where D i = { ( x i j , y i j ) } n i j =1 denotes the i -th domain with n i data instances. Each data instance x i j = { ( x i j ) k | k = 1 , · · · , M } ∈ X is comprised of M different modalities and y i j ∈ Y ⊂ R denotes the label. The joint distributions between each pair of domains are different: P i XY = P j XY , 1 ≤ i = j ≤ D . The goal of multi-modal domain generalization is to learn a robust and generalizable predictive function f : X →Y from D training domains and M data modalities to achieve a minimum prediction error on an unseen test domain D test (i.e., D test cannot be accessed during training and P test XY = P i XY for i ∈ { 1 , · · · , D } ):

̸

where E is the expectation and ℓ ( · , · ) is the loss function.

3.2 SimMMDG

We present the SimMMDG framework for addressing the multi-modal DG problem, as depicted in Fig. 2. Our approach involves splitting the feature embedding of each modality into modalityspecific and modality-shared components. We aim to map modality-shared embeddings of data samples with the same label, whether they belong to the same or different modalities, to be as close as possible (and vice versa for samples with different labels). For modality-specific features within each modality, our objective is to maximize the distance from their modality-shared features to capture unique and complementary information. Additionally, we incorporate a cross-modal translation module to regularize learned features and address the missing-modality generalization problem, which is essential for real-world testing scenarios where one or more modalities may be absent due to unpredictable factors.

Modality 1

Modality 3

Modality 2

Feature

extractor

Feature

extractor

Feature

extractor

(a) Feature Splitting

(b) Supervised Contrastive Learning

Modality-specific feature

Modality-shared feature

(c) Distance Loss

(d) Cross-modal Translation

Classifier

1

→

2

2

→

3

3

→

2

2

→

1

3

→

1

1

→

3

1

→

2

1

→

3

(e) Missing-modality Generalization (optional)

Missing-modality

Different classes

3.2.1 Within-modal Feature Splitting

Traditional multi-modal learning frameworks [58, 43] aim to map features from various modalities into a common embedding space using contrastive learning. However, this approach may not be optimal because different modalities often contain a mix of both homogeneous and heterogeneous information, making it challenging and impractical to align them all together. For instance, consider the language and visual modality [2]. Both modalities can describe people, objects, actions, and gestures, which reflect shared information between them. However, images can provide additional information such as texture, depth, and visual appearance that are not available in language data. Similarly, language can provide syntactic structure, vocabulary, and morphology, which are not present in image data. These are specific pieces of information that are unique to each modality.

Thus, the simple mapping of features from different modalities into the same embedding space may result in the loss of modality-specific information and consequently lead to decreased performance in downstream tasks. Guided by this intuition, we propose a novel approach that involves the separation of the feature embedding of each modality into modality-specific and modality-shared components. For example, given a unimodal embedding E , we denote it as E = [ E s ; E c ] , where E s is modality-specific feature and E c is modality-shared feature.

Multi-modal Supervised Contrastive Learning. We expect modality-shared features E c to possess characteristics that are shared among distinct modalities. If data instances from different modalities have the same label, we expect their modality-shared features to be as close as possible in the embedding space. To achieve this objective, we leverage the availability of labels for each data instance and employ supervised contrastive loss [37] for effective guidance. For a set of N randomly sampled label pairs, { x j , y j } j =1 ,...,N , the corresponding batch used for training consists of M × N pairs, { ˜ x k , ˜ y k } k =1 ,...,M × N , where ˜ x M × j , ˜ x M × j -1 , ... , ˜ x M × j -M +1 are data instances from M different modalities in x j ( j = 1 , ..., N ) and ˜ y M × j = ... = ˜ y M × j -M +1 = y j .

Let i ∈ I ≡ { 1 , ..., M × N } be the index of an arbitrary unimodal sample within a batch. We define A ( i ) ≡ I \{ i } , P ( i ) ≡ { p ∈ A ( i ) : ˜ y p = ˜ y i } as the set of indices of all positive samples in the batch which share the same label as i . The cardinality of P ( i ) is denoted as | P ( i ) | . Then, the multi-modal supervised contrastive learning loss can be written as follows:

with z k = Proj ( g ( ˜ x k )) ∈ R D P , where g ( · ) is the feature extractor, that maps x to modality-specific and modality-shared features, E = [ E s ; E c ] = g ( x ) , where E s , E c ∈ R D E , and Proj ( · ) is the

projection network that maps E c to a vector z = Proj ( E c ) ∈ R D P . The inner product between two projected feature vectors is denoted by · , and τ ∈ R + is a scalar temperature parameter.

Feature Splitting with Distance. To ensure that the modality-specific features E s carry unique and complementary information, we aim to maximize their dissimilarity from the corresponding modality-shared features E c . To achieve this goal, we utilize negative ℓ 2 distance and formulate a distance loss on E s and E c as:

where M is the number of modalities, E i s and E i c are the modality-specific and modality-shared features of the i -th modality.

3.2.2 Cross-modal Translation

Simply increasing the distance between E s and E c may not yield optimal results. The proposed crossmodal translation module aims to ensure the meaningfulness of modality-specific features by exploiting the implicit relationships and approximate translation mappings that exist between the M modalities within the same data instance. This is achieved through a multi-layer perceptron (MLP) [28] that translates the feature embedding E across modalities. For instance, E j t = MLP E i → E j ( E i ) implies that we translate the embedding E i = [ E i s ; E i c ] of the i -th modality to the j -th modality using MLP E i → E j , resulting in the translated embedding E j t . More intuitions behind the cross-modal translation module are discussed in the appendix. To ensure that the translated embedding E j t is a meaningful representation of the j -th modality, we aim to minimize its ℓ 2 distance from the real embedding of the j -th modality, E j and the cross-modal translation loss is defined as:

̸

3.3 Final Loss

The final loss is obtained as the weighted sum of the previously defined losses:

where L cls is the cross-entropy loss for classification, and where α con , α dis, and α trans are hyperparameters that control the relative importance of the contrastive learning, dissimilarity, and cross-modal translation terms, respectively.

3.4 Missing-modality Generalization

During the training phase, we have access to data from all modalities. However, in the testing phase, one or more modalities may be absent due to unpredictable reasons, such as sensor malfunction or loss of communication. In such cases, the system's robustness toward potential missing modalities becomes critical. To address the missing-modality scenario, we utilize our cross-modal translation module. In normal circumstances, the output of the network is defined as:

where g ( · ) is the feature extractor and h ( · ) is the classifier. If the i -th modality is missing during testing, one solution is to replace its embedding with zero. The output is then given as:

However, simply replacing the embeddings with null entries may have an adverse effect on the network's performance. To address this issue, we propose using our cross-modal translation module to predict and substitute the missing modality's embedding with information from available modalities, resulting in a more robust output for the network. The output of the network is then given as:

where

Then, for any h ∈ H ,

where λ φ is the error of an ideal joint hypothesis, ε P ( h ) is the error for a hypothesis h on a distribution P , and d H ∆ H ( P , Q ) is H -divergence which measures differences in distribution [6].

The proof of this theorem is provided in [60]. Here, Q corresponds to the unseen out-of-distribution target domain and { P i } K i =1 correspond to source domains. λ φ is small in reality and often neglected. The term ∑ i φ i ε P i ( h ) is minimized by cross-entropy loss with class labels as supervision. The term 1 2 max i,j d H ∆ H ( P i , P j ) measures the maximum differences among source domains. This corresponds to multi-modal supervised contrastive learning in our approach, where we map the modality-shared features of different modalities from all source domains to be as close as possible

̸

In cases where multiple modalities are missing during testing, we can use the same strategy to utilize available modalities to predict and substitute the missing ones, similar to Eq. (9). The benefits of our approach are demonstrated in the subsequent experiments.

4 Theoretical Insights

4.1 Multi-modal Representation Learning Perspective

We first generalize Theorem 3.1 in [34] to the case of M modalities. Let p be the joint distribution of ( x 1 , ..., x M , y ) , where x i is the i -th modality and y is the target. We define the information gap as ∆ p := max { I ( x i ; y ); i = 1 , ..., M } -min { I ( x i ; y ); i = 1 , ..., M } , where I ( x i ; y ) is the mutual information between modality x i and the target variable y . The information gap serves to characterize the effectiveness of the modalities in predicting the target y . We use the cross-entropy loss, denoted by ℓ CE, and the prediction function, denoted by f .

Theorem 1. For M feature extractors g i ( · ) ( i = 1 , ..., M ), if the multi-modal features E i = g i ( x i ) are perfectly aligned in the feature space, i.e., E 1 = ... = E i = ... = E M , then inf f E p [ ℓ CE ( f ( E 1 , ..., E M ) , y )] -inf f ′ E p [ ℓ CE ( f ′ ( x 1 , ..., x M ) , y )] ≥ ∆ p .

The proof of this theorem can be found in the appendix. Theorem 1 indicates that the optimal prediction error, achieved with perfectly aligned features, is at least ∆ p larger than when using the raw input modalities. In practical scenarios where one modality is less informative for prediction, the information gap ∆ p tends to be substantial. Consequently, this leads to a notable increase in prediction errors in downstream tasks. Furthermore, achieving perfect modality alignment enforces the aligned features to exclusively contain predictive information present in all input modalities, potentially causing the loss of modality-specific information. By splitting the feature embedding for each modality, our model has access to both modality-shared features with predictive information present in both modalities, and modality-specific features that contain predictive information unique to each individual modality. This enables our model to effectively capture the distinct information provided by each modality, while simultaneously leveraging the shared information across modalities. Consequently, our approach has the potential to enhance generalization capabilities and achieve better performance on downstream tasks.

4.2 Domain Generalization Perspective

Theorem 2. Let X be a space and H be a class of hypotheses corresponding to this space. Let Q and the collection { P i } K i =1 be distributions over X and let { φ i } K i =1 be a collection of non-negative coefficient with ∑ i φ i = 1 . Let O be a set of distributions s.t. ∀ S ∈ O , the following holds

if they have the same label. The term 1 2 min S ∈O d H ∆ H ( S , Q ) demonstrates the importance of diverse source distributions [1], so that the unseen target Q might be 'near' to O . Therefore, to enlarge the range of O , we split the feature embedding of each modality into modality-specific and modality-shared parts by maximizing the ℓ 2 distance to preserve diversity.

5 Experiments

5.1 Experimental Setting

Dataset. We use the EPIC-Kitchens dataset [16] and introduce a novel HAC dataset in this paper, which will be made publicly accessible for further research. We follow the experimental protocol used for the EPIC-Kitchens dataset in [52]. The EPIC-Kitchens dataset includes eight actions ('put', 'take', 'open', 'close', 'wash', 'cut', 'mix', and 'pour') recorded in three different kitchens, forming three separate domains D1, D2, and D3. Our HAC dataset consists of seven actions ('sleeping', 'watching tv', 'eating', 'drinking', 'swimming', 'running', and 'opening door') performed by humans, animals, and cartoon figures, forming three different domains H, A, and C. We collect 3381 video clips from the internet with around 1000 samples for each domain. We provide three modalities in our dataset: video, audio, and pre-computed optical flow. Some examples of the HAC dataset are shown in Fig. 1 (b) and more details are provided in the supplementary material.

Implementation Details. In our framework, we perform experiments on three modalities: video, audio, and optical flow. We adopt the MMAction2 [15] toolkit for experiments. To encode the visual information, we use SlowFast network [21] initialized with Kinetics-400 [35] pre-trained weights. For the audio encoder, we use ResNet-18 [29] and initialize the weights from the VGGSound pre-trained checkpoint [12]. Similarly, we use the SlowFast network with slow-only pathway and also Kinetics 400 [35] pre-trained weights for the optical flow encoder. The dimensions of the unimodal embedding E for video, audio, and optical flow are 2304 , 512 , and 2048 correspondingly. For the projection network Proj ( · ) in supervised contrastive learning, we instantiate it as a multi-layer perceptron with two hidden layers of size 2048 and output vector of size D P = 128 . We use a multi-layer perceptron with two hidden layers of size 2048 to instantiate the cross-modal translation MLP E i → E j . After obtaining the feature embedding from the encoder, we split the embedding into modality-shared features (the first half of the embedding) and modality-specific features (the remaining half). We use the Adam optimizer [39] with a learning rate of 0 . 0001 and a batch size of 16 . The scalar temperature parameter τ is set to 0 . 1 . Additionally, we set α con = 3 . 0 , α dis = 0 . 7 , and α trans = 0 . 1 . We also analyze the sensitivity of different α in the appendix. Finally, we train the network for 15 epochs on an RTX 2080 Ti GPU which takes about 20 hours and select the model with the best performance on the validation dataset. We report the Top-1 accuracy for all experiments.

5.2 Results

Multi-modal Multi-source DG.

Tab. 1 and Tab. 2 illustrate the results of SimMMDG under the multi-modal multi-source DG setting, where we train on multiple source domains and test on one target domain. We first conduct experiments using video and audio modalities for experiments, as in [57]. We re-implement our framework employing the same I3D [10] and BN-Inception [32] backbone for video and audio as in RNANet [57] to ensure fair comparisons. The DeepAll approach implies that we feed all the data from source domains to the network without any domain generalization strategies. Tab. 1 shows that our SimMMDG significantly outperforms all the baselines. When we replace the backbone with SlowFast [21] and ResNet-18 [29],

the results further improve by a large margin (with an average improvement of up to 5 . 73% ) compared to the baselines. In the following experiments, we adopt SlowFast and ResNet-18 as our default backbones. To verify the generalization of our framework to different modalities, we conduct experiments by combining any two modalities, as well as all three modalities, and present the results in Tab. 2. Our SimMMDG outperforms all the baselines by a significant margin in all cases, with improvements of up to 9 . 58% . Notably, when we combine all three modalities, the performance further improves and surpasses that of any two modalities. In contrast, the baseline methods cannot achieve better results with more modalities, indicating that they fail to fully leverage the complementary information between modalities. Finally, we validate the performance of our framework on the HAC dataset, and the results are consistent with those obtained on the EPIC-Kitchens dataset, as demonstrated in Tab. 2. Our SimMMDG outperforms all the baselines by a significant margin in all cases, with improvements of up to 7 . 73% .

Multi-modal Single-source DG. The domain labels are not required in our SimMMDG framework. This feature makes our method readily applicable to single-source DG without modifications. In this setup, we train the model on a single source domain and test it on multiple target domains. Tab. 3 presents the results of SimMMDG under the multi-modal single-source DG setting using video and audio modalities. Despite being trained only on data from a single domain, our model demonstrates robust generalization to unseen domains, with an average improvement of up to 5 . 71% . In comparison, other baseline methods perform even worse than the DeepAll baseline, highlighting their limitations in the single-source DG setting. We present additional results obtained by exploring different combinations of modalities in the appendix.

Missing-modality DG. In real-world deployment scenarios, we cannot always guarantee that all modalities will be available. Hence, the network should be capable of handling missing-modality cases. One simple approach is to set the embedding of the missing modality to zero. We compare this with our proposed cross-modal translation replacement in Sec. 3.4. We retrain the cross-modal translation MLP E i → E j for 10 epochs using L trans in Eq. (4) and keep other parameters fixed. Tab. 4 reports on the comparison results on the EPIC-Kitchens dataset, where our solution yields significant benefits (up to 10 . 47% performance improvement) compared to zero-filling. By replacing the missing modality features with the translation ones, our approach also outperforms the unimodal model in most cases. In contrast, zero-filling hurts the network performance in some cases and is even worse than the unimodal model. The last six rows in Tab. 4 demonstrate that our approach is robust

Method

Video

Audio

Flow

D2, D3

→

D1

D1, D3

→

D2

D1, D2

→

D3

Mean

DeepAll (Video-only)

✓

51.03

59.87

56.57

55.82

SimMMDG (zero-filling)

✓

✗

55.40

64.00

58.32

59.24

SimMMDG (translation)

✓

✗

57.24

65.07

59.65

60.65

SimMMDG (zero-filling)

✓

✗

53.10

60.93

61.09

58.37

SimMMDG (translation)

✓

✗

55.17

61.33

61.70

59.40

DeepAll (Audio-only)

✓

32.87

42.27

45.17

40.10

SimMMDG (zero-filling)

✗

✓

29.20

37.33

45.69

37.41

SimMMDG (translation)

✗

✓

37.70

46.40

52.05

45.38

SimMMDG (zero-filling)

✓

✗

28.51

34.27

42.81

35.20

SimMMDG (translation)

✓

✗

39.77

46.00

51.23

45.67

DeepAll (Flow-only)

✓

54.25

61.33

55.95

57.18

SimMMDG (zero-filling)

✗

✓

56.32

60.67

56.57

57.85

SimMMDG (translation)

✗

✓

56.32

62.13

56.98

58.48

SimMMDG (zero-filling)

✗

✓

52.41

62.13

51.33

55.29

SimMMDG (translation)

✗

✓

55.40

63.73

56.57

58.57

SimMMDG (zero-filling)

✗

✓

✓

53.10

62.40

64.48

59.99

SimMMDG (translation)

✗

✓

✓

55.86

68.27

64.58

62.90

SimMMDG (zero-filling)

✓

✗

✓

62.76

66.80

57.49

62.35

SimMMDG (translation)

✓

✗

✓

63.22

67.20

59.24

63.22

SimMMDG (zero-filling)

✓

✓

✗

60.92

68.66

60.99

63.52

SimMMDG (translation)

✓

✓

✗

62.53

68.13

61.60

64.09

SimMMDG (zero-filling)

✓

✗

✗

59.08

62.27

52.98

58.11

SimMMDG (translation)

✓

✗

✗

59.31

64.00

56.57

59.96

SimMMDG (zero-filling)

✗

✓

✗

33.79

37.47

44.56

38.61

SimMMDG (translation)

✗

✓

✗

38.39

44.53

47.74

43.55

SimMMDG (zero-filling)

✗

✗

✓

55.17

57.07

48.46

53.57

SimMMDG (translation)

✗

✗

✓

56.32

64.27

56.57

59.05

even when two modalities out of three are missing . We present more results on HAC dataset in the appendix.

5.3 Ablation Studies

Ablation on each proposed module. We conducted extensive ablation studies to investigate the role of each module of SimMMDG on EPIC-Kitchens dataset, as shown in Tab. 5. Incorporating the supervised contrastive learning loss alone resulted in noticeable improvements. However, the mean accuracy decreased when we integrated feature splitting without imposing any constraints. The results were further enhanced when we added the distance loss to promote diversity, and even more so when we incorporated the cross-modal translation module. Although using only supervised contrastive learning and cross-modal translation yielded satisfactory results, their performance is on average 4 . 15% lower than the complete approach with feature splitting and distance loss. These findings highlight the significance of segregating the feature embedding of each modality into modalityspecific and modality-shared components. Although SimMMDG without supervised contrastive learning is already better than most baselines, it still has a performance gap compared with the whole framework, which means the cross-modal translation is helpful but cannot replace contrastive learning. The effect of contrastive learning is similar to explicit feature alignment. It aligns the modality-shared features of different modalities from different source domains with the same label to be as close as possible in the embedding space, while pushing away features with different labels, to make the embedding space more distinctive.

Comparison against unimodal DG. Tab. 6 presents the results in comparison to unimodal DG algorithms that exclusively rely on either video, audio, or optical flow inputs. We choose RSC [31], Mixup [72], and Fishr [59] as our baselines. By leveraging information from multiple modalities, our multi-modal DG framework delivers significant improvements (up to 7 . 74% ) in terms of performance as compared to unimodal DG methods.

Combine SimMMDG with other training strategies. SimMMDG can be seamlessly combined with other training strategies due to its generality and simplicity. We first combine SimMMDG with Gradient Blending [69], a strategy to improve multi-modal learning. The results are further improved compared to the original SimMMDG as shown in Tab. 7. We also combine SimMMDG with a Domain-adversarial Neural Network (DANN) [24] to align the features of source domains using

domain labels and observe a similar improvement as with Gradient Blending. This indicates that our SimMMDG can be easily combined with other training strategies to get even better results.

SimMMDG as a general framework for multi-modal classification. Here, we evaluate our framework in a more general multi-modal classification setup without DG. We first evaluate our framework on the EPIC-Kitchens dataset without incorporating the DG setup. To do so, we aggregate data from three domains, we partition the training, validation, and testing data to ensure that no apparent domain shifts exit between the training and testing data. We compare our method with DeepAll, TBN [36], and Gradient Blending [69]. The results, as presented in Tab. 8 demonstrate that SimMMDG also exhibits significant advantages in the general multi-modal classification setup.

We further evaluate our framework on two multi-modal datasets: MUSTARD [11] and URFUNNY [27], both available in MultiBench [45]. These datasets pertain to human sentiment analysis and encompass language, video, and audio modalities. We conduct our experiments using the MultiBench codebase and implement SimMMDG within that environment. MultiBench treats human sentiment analysis as a regression task, whereas our framework is tailored for classification. To align the codebase with our classification task, we made the necessary modifications. For all baselines, we apply the same backbone model and solely change the fusion paradigms. In our comparisons, we evaluate our method against Late Fusion, Low-rank Tensor Fusion (LMF) [48], and Multimodal Transformer Fusion (MULT) [62]. The results reveal that our SimMMDG exhibits clear advantages, outperforming the baselines with an average improvement of 7 . 3% and 1 . 7% . This suggests that our SimMMDG serves as a versatile framework for multi-modal classification tasks and is compatible with various combinations of modalities, such as video+audio+flow and language+video+audio.

6 Conclusion

In this paper, we propose the SimMMDG framework for multi-modal DG. Our approach involves splitting the features of each modality into modality-specific and modality-shared parts and enforcing constraints on each part using supervised contrastive learning, distance loss, and cross-modal translation. The cross-modal translation module can also be applied in the case of missing-modality generalization. Our experiments on different datasets demonstrate the effectiveness of SimMMDG . Furthermore, we introduce a new challenging multi-modal dataset that can serve as a benchmark and guide future research in multi-modal DG problems.

Limitations. Currently, the number of cross-modal translation MLP in our framework is O ( | M | 2 ) and will be complex with the increase of modalities. In future work, the encoder-decoder network proposed in [19] can be used to reduce the complexity to O ( | M | ) .

Acknowledgments

The authors acknowledge the support of "In-service diagnostics of the catenary/pantograph and wheelset axle systems through intelligent algorithms" (SENTINEL) project, supported by the ETH Mobility Initiative.

References

Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. Generalizing to unseen domains via distribution matching. arXiv preprint arXiv:1911.00804 , 2019.

Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41(2):423-443, 2019.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR , 2022.

Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. NeurIPS , 2022.

Normand J Beaudry and Renato Renner. An intuitive proof of the data processing inequality. Quantum Information & Computation , 12(5-6):432-441, 2012.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning , 79(1):151-175, 2010.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR , 2020.

Fabio M Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In CVPR , 2019.

Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01340 , 2018.

Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR , 2017.

Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection. arXiv preprint arXiv:1906.01815 , 2019.

Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP , 2020.

Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In CVPR , 2017.

Ying Cheng, Ruize Wang, Zhihao Pan, Rui Feng, and Yuejie Zhang. Look, listen, and attend: Coattention network for self-supervised audio-visual representation learning. In Proceedings of the 28th ACM International Conference on Multimedia , pages 3884-3892, 2020.

MMAction2 Contributors. Openmmlab's next generation video understanding toolbox and benchmark. https://github.com/open-mmlab/mmaction2 , 2020.

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In ECCV , 2018.

Hao Dong, Xieyuanli Chen, Simo Särkkä, and Cyrill Stachniss. Online pole segmentation on range images for long-term lidar localization in urban environments. Robotics and Autonomous Systems , 159:104283, 2023.

Hao Dong, Xianjing Zhang, Jintao Xu, Rui Ai, Weihao Gu, Huimin Lu, Juho Kannala, and Xieyuanli Chen. Superfusion: Multilevel lidar-camera fusion for long-range hd map generation. arXiv preprint arXiv:2211.15656 , 2022.

Mihai Dusmanu, Ondrej Miksik, Johannes L Schönberger, and Marc Pollefeys. Cross-descriptor visual localization and mapping. In ICCV , 2021.

Farzan Farnia and David Tse. A minimax approach to supervised learning. In NeurIPS , 2016.

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV , 2019.

Olga Fink, Qin Wang, Markus Svensén, Pierre Dersin, Wan-Jui Lee, and Melanie Ducoffe. Potential, challenges and future directions for deep learning in prognostics and health management applications. Engineering Applications of Artificial Intelligence , 92:103678, 2020.

Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In NeurIPS , 2013.

Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML , 2015.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research , 17(1):2096-2030, 2016.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR , 2012.

Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, et al. Ur-funny: A multimodal language dataset for understanding humor. arXiv preprint arXiv:1904.06618 , 2019.

Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning . Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR , 2018.

Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In ECCV , 2020.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML , 2015.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021.

Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Understanding and constructing latent modality structures in multi-modal representation learning. In CVPR , 2023.

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017.

Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In ICCV , 2019.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS , 2020.

Donghyun Kim, Yi-Hsuan Tsai, Bingbing Zhuang, Xiang Yu, Stan Sclaroff, Kate Saenko, and Manmohan Chandraker. Learning cross-modal contrastive features for video domain adaptation. In ICCV , 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR , 2015.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV , 2017.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI , 2018.

Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR , 2018.

Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS , 2021.

Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In ECCV , 2018.

Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. arXiv preprint arXiv:2107.07502 , 2021.

Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.03430 , 2022.

Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS , 2022.

Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and LouisPhilippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064 , 2018.

Wang Lu, Jindong Wang, Xinwei Sun, Yiqiang Chen, and Xing Xie. Out-of-distribution representation learning for time series classification. In ICLR , 2023.

Pedro Morgado, Yi Li, and Nuno Nvasconcelos. Learning representations from audio-visual spatial alignment. In NeurIPS , 2020.

Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICML , 2013.

Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine-grained action recognition. In CVPR , 2020.

Hyeonseob Nam and Hyo-Eun Kim. Batch-instance normalization for adaptively style-invariant neural networks. In NeurIPS , 2018.

Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In ECCV , 2018.

Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos. Found in translation: Learning robust joint representations by cyclic translations between modalities. In AAAI , 2019.

Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via commonspecific low-rank decomposition. In ICML , 2020.

Mirco Planamente, Chiara Plizzari, Emanuele Alberti, and Barbara Caputo. Domain generalization through audio-visual relative norm alignment in first person action recognition. In WACV , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-ofdistribution generalization. In ICML , 2022.

Anthony Sicilia, Xingchen Zhao, and Seong Jae Hwang. Domain adversarial neural networks for domain generalization: When it works and how to improve. Machine Learning , pages 1-37, 2023.

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IROS , 2017.

Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In ACL , 2019.

Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 , 2014.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering , pages 1-1, 2022.

Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing , 312:135-153, 2018.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-tosequence learning framework. In ICML , 2022.

Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In CVPR , 2020.

Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR , 2018.

Christopher Zach, Thomas Pock, and Horst Bischof. A duality based approach for realtime tv-l 1 optical flow. In Pattern Recognition , 2007.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR , 2018.

Yunhua Zhang, Hazel Doughty, Ling Shao, and Cees G.M. Snoek. Audio-adaptive activity recognition across video domains. In CVPR , 2022.

Han Zhao, Chen Dan, Bryon Aragam, Tommi S Jaakkola, Geoffrey J Gordon, and Pradeep Ravikumar. Fundamental limits and tradeoffs in invariant representation learning. Journal of Machine Learning Research , 23(340):1-49, 2022.

Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain-adversarial image generation for domain generalisation. In AAAI , 2020.

A Proof of Theorem 1

Proof of Theorem 1. Consider the joint mutual information I ( E 1 , ..., E M ; y ) . By the chain rule, we have the following decompositions:

However, since E i are perfectly aligned, I ( E 2 , ..., E M ; y | E 1 ) = ... = I ( E 1 , ..., E M -1 ; y | E M ) = 0 , which means I ( E 1 , ..., E M ; y ) = I ( E 1 ; y ) = ... = I ( E M ; y ) . On the other hand, by the data processing inequality [5], we know that

Hence, the following chain of inequalities holds:

where the last inequality follows from the fact that the joint mutual information I ( x 1 , ..., x M ; y ) , is at least as large as any one of I ( x i ; y ) . We use H ( y | x 1 , ..., x M ) to denote the conditional entropy of y given x 1 , ..., x M as input. According to [74, 20] we have the following variational form: H ( y | x 1 , ..., x M ) = inf f E p [ ℓ CE ( f ( x 1 , ..., x M ) , y )] , where the infimum is over all the prediction functions that take x 1 , ..., x M as input to predict the target y and the expectation is taken over the joint distribution p of ( x 1 , ..., x M , y ) . Therefore, due to the variational form of the conditional entropy, we have

B More Details on HAC Dataset

Our Human-Animal-Cartoon (HAC) dataset consists of seven actions ('sleeping', 'watching tv', 'eating', 'drinking', 'swimming', 'running', and 'opening door') performed by humans, animals, and cartoon figures, forming three different domains. We collect 3381 video clips from the internet with around 1000 for each domain and provide three modalities in our dataset: video, audio, and pre-computed optical flow. The dense optical flow is extracted at 24 frames per second using the TV-L1 algorithm [71].

Our dataset was collected by 5 volunteers. For the human domain, we collect the data by selecting actions from an existing Kinetics-600 dataset [9]. We select approximately the same number of video clips for each action to ensure class balance. For the animal domain, we use the available 200 video clips in [73] and extend it to 906 . We collect data from YouTube by searching keywords like 'animal sleeping', 'animal eating', 'animal running', etc. To increase the diversity of the dataset, we also specify the animal type in the keywords, such as 'cat sleeping', 'dog eating', 'horse running', etc. Each participant was asked to collect certain actions, such as Participant A for 'sleeping' and 'watching tv', Participant B for 'eating' and 'drinking', etc. For the cartoon domain, we collect all data from scratch and we collect data from popular cartoons like 'SpongeBob SquarePants', 'The Simpsons', 'Garfield and Friends', etc. Each participant was asked to collect from one or two cartoon

sleeping

watching tv

eating

drinking

swimming

running

opening door

0

25

50

75

100

125

150

175

200

Number of Action Segments

Human

Animal

Cartoon

Action Class

series to avoid duplication. For each action, we annotated the start and end times in the video and then cut out video clips. The length of each video clip varies from 1 s to 10 s. Finally, we gathered the data from all volunteers and a separate person manually discarded unqualified data like duplicate videos, videos without audio data, noisy/wrong classes, etc.

Fig. 3 and Tab. 9 illustrate the statistcs of our HAC dataset. During the training phase, we train our model on one or two domains by utilizing the training action segments specific to those domains. The model selection is based on the performance of the validation action segments. Subsequently, we evaluate the model on the remaining domain, utilizing all the segments within that domain. Therefore, the total number of testing action segments is the sum of the training and validation segments. It is observed that each class within the HAC dataset exhibits a good balance across all domains. In contrast, EPIC-Kitchens dataset suffers from severe class imbalance. This implies that our proposed framework is capable of delivering high-performance results across both balanced and imbalanced datasets. Fig. 4 presents additional examples from our HAC dataset, highlighting the substantial domain shifts present in our dataset, particularly in the cartoon domain. The main purpose of this dataset is to be used for multi-modal domain generalization research. Of course, our dataset can also be used for other applications like multi-modal learning and multi-modal domain adaptation.

C Additional Implementation Details

We follow RNA-Net [57] to select the baseline methods. MM-SADA [52] is a domain adaption method and we only use the self-supervision loss without adversarial alignment as we have no access on target domain data during training. From the results in Table 1 in main paper, we know RNA-Net [57] and MM-SADA [52] are the two best baseline methods. Therefore, we select them together with DeepAll as baselines in the following experiments.

Human

Animal

Cartoon

sleeping

watching tv

eating

drinking

swimming

running

opening door

D Further Ablations

Ablations of different distance types. Tab. 10 shows the ablations of different distance types in distance loss L dis . We explore the effects of using ℓ 2 norm, ℓ 1 norm, and Cosine similarity. Our experimental results indicate that all three distance types outperform the baselines, with ℓ 2 norm achieving the highest performance.

Parameter Sensitivity. We investigate the sensitivity of our method to the hyperparameters in the loss function. We perform this analysis by varying one parameter while fixing the others, and present our findings in Fig. 5. The results demonstrate that our method consistently outperforms the DeepAll baseline for all parameter settings, indicating that our approach is less sensitive to hyperparameter choices. However, we can also observe α dis is a little sensitive and may need attention for tuning in real applications.

1

3

5

10

αcon

58

60

62

64

66

accuracy (%)

SimMMDG

DeepAll

0

.

1

0

.

5

0

.

7

1

.

0

αdis

58

60

62

64

66

accuracy (%)

SimMMDG

DeepAll

0.010.1

0.5

1.0

αtrans

58

60

62

64

66

accuracy (%)

SimMMDG

DeepAll

E Other Experimental Results

Multi-modal Single-source DG. Tab. 11 and Tab. 12 show more results under multi-modal singlesource DG setting using the combination of different modalities. Our model demonstrates robust generalization to unseen domains compared to the baseline methods in most cases with a large margin up to 4 . 81% .

Missing-modality DG. Tab. 13 shows more results under the missing-modality DG setting on HAC dataset. Our solution yields significant benefits compared to zero-filling in most cases, similar to findings in the EPIC-Kitchens dataset. Our approach is also robust even when two modalities out of three are missing. We also validate the effectiveness of our cross-modal translation module on DeepAll, MM-SADA [52], and RNA-Net [57], as shown in Tab. 14. Compared to zero-filling,

replacing the features of missing modalities with the translated ones from other available modalities achieves better performance.

Statistical Significance Tests. We run each experiment three times using different seeds for multimodal multi-source DG on the EPIC-Kitchens dataset and then calculate the mean and standard deviation to show the statistical significance of our methods. As shown in Tab. 15, our framework is statistically stable and surpasses the baselines significantly.

(a) Embedding of Video

(b) Embedding of Audio

(c) Embedding of Video and Audio

(d) Splitted Embedding of Video

(e) Splitted Embedding of Audio

ource Domain (Video)

Domain (Video+ Asdio)

40

40

20

20

40

Source Domain (Audio)

Target Domain (Audio)

60

60

20

bU

40

20

0

20

40

Source Domain (Video-shared)

Target Domain (Video-shared)

75

(podsoopA) uemo a

50

40

25

20

25

20

50

Source Domain (Audio-shared)

75

Source Domain (Audio-specific)

Target Domain (Audio-shared)

100

Target Domain (Audio-specific)

20

20

60

40

20

0

20

40

60

F Visualization

We show more visualizations of the learned embeddings using t-SNE [64] in Fig. 6. We can observe that the embeddings of video, audio, and the concatenation of video and audio are all well aligned for the source and target domains. Besides, the modality-specific and modality-shared embeddings are also well-separated and aligned across domains.

G More Intuitions behind Cross-modal Translation Module

In our SimMMDG framework, we introduce a cross-modal translation module to further regularize the learned features and facilitate missing-modality generalization. Here we give more intuitions behind this module.

Cross-modal translation won't undermine the unique features of different modalities. We want to learn an MLP projection to translate the embedding E i of the i -th modality to the embedding E j of the j -th modality. We add a translation loss to make the translated embedding E j t to be close to E j , without any explicit alignment or constraints on E i and E j . We apply the cross-modal translation on the integrated feature of each modality E i = [ E i s ; E i c ] , which is the concatenation of modality-specific feature E s and modality-shared feature E c . We still enforce a distance loss on E s and E c at the same time. Therefore, the modality-specific and modality-shared features are still forced to be separated during the training progress. The embedding visualizations shown in Fig. 6 (d) and (e) also indicate that for both video and audio, their modality-specific and modality-shared features are well disentangled and are not influenced by the cross-modal translation module.

Cross-modal translation is a type of modalities interaction, where different modality elements interact to give rise to new information when integrated together for task inference [46]. Several works [23, 55] have already demonstrated that modality interaction can help improve the performance of multi-modal tasks. For example, the proposed approach in [23] learns a coordinated similarity space between image and text to improve image classification. The approach proposed in [55] translates language into video and audio for language sentiment analysis.

Cross-modal translation can be thought of as a means to leverage the information from one modality to infer as much information as possible for the target modality. Just like when we hear a dog barking, we will fill in the picture of the puppy in our mind, and when we see a foreign language, we will automatically translate it into our native language. Although there is information loss and we can't recover all the details during this translation progress, we can still infer some useful information for the target modality. As shown in our ablation study in Tab. 5, adding the cross-modal translation module indeed improves multi-modal DG performance.

More importantly, our cross-modal translation module can be used for improving missingmodality generalization, by filling in the features of missing modality with the features inferred/translated from the available modality. The benefit of adopting cross-modal translation is demonstrated in Tab. 4. Our approach is robust even when two modalities out of three are missing. This indicates that the inferred information from the cross-modal translation module is valuable and useful for downstream tasks.

H Further Discussions on Modality-specific and Modality-shared Features

Based on our assumptions in the paper, modality-shared features reflect shared information between all modalities, like all modalities that describe the same people, objects, actions, or gestures. Modalityspecific features are specific pieces of information that are unique to each modality, like texture, depth, and visual appearance in images, syntactic structure, vocabulary, and morphology in language. Our goal is to align the modality-shared features of different modalities from different source domains with the same label to be as close as possible in the embedding space, while pushing away features with different labels, to make the embedding space more distinctive. At the same time, we want the modality-specific features to be as far as possible from the modality-shared features, such that they carry different types of information.

We analyzed the meaningfulness and the ability to share features of modality-shared features by cross-modal retrieval task. The ability of cross-modal retrieval is highly related to the shareable of features, as only features that are meaningful, shareable, and highly connective can give a good recall rate. We use modality-shared features of videos to retrieve from the modality-shared features of audios and calculate the recall, and vice versa for audios. We report the R@1, R@5, and R@10 values for Video to Audio Retrieval and Audio to Video Retrieval. For example, R@5 for Video to Audio Retrieval means we retrieve 5 candidates from audio, if at least one of them has the same label as the query video, we consider the retrieval success and then we calculate the average success rate for all query videos. We also do the same thing on modality-specific features: using modality-specific features of videos to retrieve from the modality-specific features of audios, and vice versa for audios. As shown in Tab. 16 and Tab. 17, we have a very high recall rate when we use modality-shared features, while the recall rate is biased towards random when we use modality-specific features. This further indicates that the modality-shared features are truly shareable and meaningful and there are very strong relations and connections across modalities. The modality-specific features are instead with more information that is private to each single modality.

To further verify the meaningfulness of modality-shared features, we also train a classifier only on the concatenation of modality-shared features of video and audio, and discard the modality-specific features. As shown in Tab. 18, without modality-specific features, the performances drop slightly

compared to the whole framework, but are still competitive. This indicates that modality-shared features truly have some meaningful information that can be used for prediction tasks.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on addressing a special case of multimodal learning, i.e., multi-modal domain generalization, where one or more modalities will be absent during test phase. \n\nThis work propose a approach to solve multimodal DG from a disentanglement perspective.  Modality-specific features contain modality-specific information, while Modality-shared feature contains information that is shared across domain such as class labels. \n\nSupervised contrastive learning is employed to learn better modality-shared features while a distance-based regularize is used to enlarge the discrepancy between modality-specific and modality-shared features. In addition, a cross-modal translation module is proposed to enhance the modality-shared features.\n\nThis work also release a dataset named Human-Animal-Cartoon (HAC).\n\nSTRENGTHS:\nThis work has following strengths:\n\n- This paper released a dataset, HAC, which is helpful for multimodal domain generalization community.\n\n- This paper is easy to follow.\n\n- Multi-modal DG is interesting and has research potential since real-world task is complicated that often involves multiple modalities.\n\nWEAKNESSES:\nThis work has following weaknesses:\n\n- Is the name of the setting appropriate? Typically, domain generalization refer to a setting that testing domain is unseen during training. However, the setting in this work is that the testing domain/modality exists in training data but some training modalities might be missing.\n\n- Lack of proof that the network truly learn modality-specific and modality-shared features.\n\n- It's not clear that why Cross-modal Translation can enhance modality-specific features, since the MLP use both features for translation. How to guarantee the two types features are well disentangled.\n\n- The technique contribution of this work is somehow not strong. Supervised contrastive loss is a mature technique. What's the essence to use supervised contrastive loss here? Can CE loss obtain similar performance? Translation idea is also appeared in [1].\n\n[1] Ge, Yunhao, et al. \"Zero-shot synthesis with group-supervised learning.\" arXiv preprint arXiv:2009.06586 (2020).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of domain generalization in a multi-modal setting. The core idea is that aligning features from multiple modalities in a shared embedding space can hinder generalization. To overcome this limitation, the paper proposes a more effective approach: it separates the features within each modality into modality-specific and modality-shared components and then aligns only the modality-specific components. This alignment is achieved through supervised contrastive learning and cross-modal feature translation techniques. To validate the proposed approach, the paper introduces a new dataset called HAC, which consists of video, audio, and optical flow data from three domains: Human, Animal, and Cartoon. Extensive experiments conducted on both the HAC dataset and the EPIC-Kitchens dataset demonstrate the impressive performance of the proposed method when compared to existing approaches.\n\nSTRENGTHS:\n1. Multi-modal domain generalization is an important practical problem and the proposed method and dataset introduced in this paper will help in advancing research in this direction.\n2. From a methodological standpoint, the idea of splitting the features into modality-specific and modality-shared and aligning them separately seems new. Interestingly, this goes against the strategy followed by popular multi-modal models such as CLIP.\n3. The paper showcases a comprehensive range of experiments to validate the effectiveness of the proposed method. It explores various combinations of modalities (e.g., Audio+Video, Audio+Video+OpticalFlow), different numbers of training domains, and even evaluates performance in scenarios with missing modalities during inference. This thorough evaluation provides a robust understanding of the method's capabilities and limitations.\n4. The paper is very well written and easy to follow. The diagram in Figure 2 is especially well done and effectively illustrates the framework in a concise manner.\n\nWEAKNESSES:\n1. The paper highlights an apparent contradiction in its argument regarding modality-specific information. On one hand, it acknowledges that certain modalities, such as video and flow, possess distinct and unalignable features, such as lighting and motion information. However, the paper introduces a cross-modal transformation module that seemingly enables seamless transformation between these features. This raises a critical question: Does this transformation not contradict the initial premise that modality-specific information is inherently unalignable?\n2. The paper notably overlooks the utilization of domain labels or any explicit feature alignment techniques across domains. This omission raises concerns about the effectiveness of the proposed method in achieving robust domain generalization.\n\nQUESTIONS:\n1. Can the usage of domain labels and explicit domain alignment strategies improve the generalization performance?\n2. Can you clarify the contradiction pointed out in Weakness-1?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework for domain generalization for models trained with multi-modal/ multi-source data: SimMMDG. The approach is motivated by the observation that each modality contains features shared with other modalities as well as features specific to that modality. The paper proposes to split per-modality features such that modality-shared features should be similar while modality-specific features should be complementary. A multi-loss strategy is adopted to implement this idea. The author evaluates the approach's performance on EPIC-Kitchen and a proposed  Human-Animal-Cartoon dataset.\n\nSTRENGTHS:\n- The proposed approach, SimMMDG seems simple yet novel. It relies on a simple intuition that modalities should have modality-specific features, which had been overlooked in previous literature.  The approach seems very general and has the potential to become a popular framework used by any researcher working with multi-modal representation learning. Its simplicity also enables combining this framework with previous approaches.\n- On the empirical results, the method seems to provide non-trivial gain over baselines and previous approaches targeting multi-modal supervised classification. \n- The paper is clearly written; it feels one can directly implement the proposed idea without too much trouble.\n\nWEAKNESSES:\n- Despite how much I appreciate the idea proposed by the paper, some important experiments are missing, which leads me to question the technical soundness of the work. For example, the learning rate of CL (supervised contrastive loss) is set to 3.0; this essentially increased the effective learning rate of the model significantly. I wonder if authors have tuned the baseline learning rate accordingly to ensure that the performance gain is not influenced by the learning rate change. In Table5, it seems all results adds CL; what if we remove CL, can CT effectively learn similarly as CL? In addition, the work is often motivated with the recent use of text + vision, but the evaluation does not include any text+vision task. \n- Impact of the work. The proposed framework seems very general, but the author eventually chose to test only on Domain Generalization. This seems to undersell the work, since it seems feasible to implement this idea to perform multi-modal classification tasks in a generic fashion. I wonder if authors have tried more evaluation apart from domain generalization, and if the methods provide improvement there.\n- Lack of discussion on HAC. One of the contribution of the paper is the HAC dataset. This is a new evaluation benchmark, yet studies/ analysis/ details on this dataset is missing. I also checked the supplementary of the paper, but found very limited information on this benchmark, such as dataset statistics, how are videos selected, how is the annotation performed, is there any QA adopted. Without details and studies (e.g. comprehensive baseline evaluation) on HAC, it will be hard to count HAC as a sound contribution and trust evaluation on this new benchmark. \n- SimMMDG with other baseline methods. A major contribution author claims is on the simplicity. This makes me wonder if the pipeline is actually compatible with many of the baseline adopted, such as Gradient Blending [53] and Non-Local [54]. I wonder if authors have tried their approach with these methods to show if the proposed framework is complementary to existing techniques.\n\nIn general, although I really enjoy reading this paper and likes the intuitions, it seems the work needs more polish and empirical evaluation to prove its soundness.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nAuthor should discuss the potential impact regarding their newly proposed dataset HAC.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes a method for multi-model domain generalization. Rather than mapping features of different modality into the same embedding space, it splits features within each modality into modality-specific feature and modality-shared feature. Then it applies different learning strategies to different split features. Besides, a new dataset called Human-Animal-Cartoon is proposed in this paper. Experiments a re conducted on two datasets and shows good performance.

STRENGTHS:
The paper proposes a novel method for multi-model domain generalization and a new dataset for this task. The motivation and method are clear to me. The results are convincing to me.

WEAKNESSES:
I am confused to the section 3.2.2 Cross-model Translation. The design is translating the i-th modality feature to j-th modality feature. However, like the paper states that, different modality features contains share-features and unique features. Does the translation will undermine the unique features in different modalities?

QUESTIONS:
See the weaknesses.

LIMITATIONS:
The cross-modal translation does not make sense to me. Will rise my rating if rebuttal can address my concern.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of multimodal domain generalization in which the goal is to learn a model from multi-modal multi/single-source domain data that can generalize to unseen multimodal distributions. The main motivation is that mapping data from multiple modalities to a shared embedding space only harnesses the common features among different modalities while overlooks the modality-specific information. The paper proposes three main technical contributions to tackle the aforementioned problem and address the multimodal domain generalization problem. At first, each modality features are split into modality-shared and modality-specific components and then a supervised contrastive loss is used for alignment. Next, a distance loss is leveraged to repel the modality-specific features from each modality. Also, a cross-modal translation module is proposed to regularize the learned features and facilitate the missing modality issue. Finally, the paper also introduces a new multimodal DG dataset, namely Human-Animal-Cartoon (HAC). Results on EPIC-kitchens and HAC datasets claim improved DG performance under both multi-source and single-source settings.\n\nSTRENGTHS:\n1) The problem of multimodal domain generalization is relatively new and mostly underxplored and carries practical relevance because in many real-world scenarios the data is often observed via multiple modalities.\n\n2) The cross-modal translation module is capable of regularizing the learned features and handle missing modalities issue.\n\n3) The paper proposes a new Human-Animal-Cartoon (HAC) dataset, which consists of different action classes and three different domains, to facilitate research in multimodal DG.\n\n4) Results on two different datasets and under both multi-source and single-source DG setting show that the proposed framework provides notable gains over the baselines and existing methods.\n\nWEAKNESSES:\n1) The paper seem to be missing the domain generalization aspect, like which pertinent DG problem in the context of multimodal data the paper is trying to solve, because all the technical contributions mostly aim at either bringing the modality-shared component or repel the modality-specific component which seems effective compared to the only relevant baseline [44]. \n\n2) The two important technical contributions i.e. learning shared embedding space and pushing modality-specific feature is not very new in the context of DG and even broader scope [A],[B]. This is also because the supervised contrastive loss [29] as well as the distance based loss are the existing components taken from literature.\n\n3) The unimodal DG comparison is made with rather old approaches RSC [23] and MixUp [55] which weakens the claim of significant performance gain. There are several new unimodal DG approaches which have shown state-of-the-art performances such as [C], [D] and [E].\n\n4) Fig. 2 that displays the overall architecture of the proposed method is not very helpful towards grasping the mid-level idea. For instance, one thing is to improve the patterns and colors. Also, the some symbols from the text in methodology should be used in the diagram to better connect the two.\n\n5) Details on how the HAC datsets is being developed is missing. For instance, how many volunteers were involved? what were the quality assurance mechanisms in place? I was expecting these details at least in the supplementary material, but the supp material only provides some high-level statistics of the dataset such as number of videos or to a little bit class-imbalance issue.\n \n[A] Bui, M.H., Tran, T., Tran, A. and Phung, D., 2021. Exploiting domain-specific features to enhance domain generalization. Advances in Neural Information Processing Systems, 34, pp.21189-21201.\n\n[B] Chattopadhyay, P., Balaji, Y. and Hoffman, J., 2020. Learning to balance specificity and invariance for in and out of domain generalization. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16 (pp. 301-318). Springer International Publishing.\n\n[C] Cha, J., Chun, S., Lee, K., Cho, H.C., Park, S., Lee, Y. and Park, S., 2021. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34, pp.22405-22418.\n\n[D] Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P. and Cord, M., 2022. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35, pp.10821-10836.\n\n[E] Rame, A., Dancette, C. and Cord, M., 2022, June. Fishr: Invariant gradient variances for out-of-distribution generalization. In International Conference on Machine Learning (pp. 18347-18377). PMLR.\n\nQUESTIONS:\nThe paper tackles a challenging and a relatively new problem of how to learn a cross-domain generalizable model under multi-modal settings. Overall, the paper presents an effective framework comprised of three components and alongside a new multimodal dataset, however, there are some important questions/concerns as listed in weaknesses (1-5) due to which my initial rating for the paper is ‘weak accept’. \n\nAlso:\n\nIt would be interesting to know the performance of the method under:\n\na) different types of domain shifts, such as texture, background etc.\n\nb) increasing domain shifts.\n\nAre the results reported as the average of different trials and if yes, what are the standard deviations?\n\nLIMITATIONS:\nThe paper mentions one limitation regarding the increasing complexity of cross-modal translation module with increasing number of modalities.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}