Cross-modal Representation Flattening for Multi-modal Domain Generalization

Authors: yunfeng  fan
wenchao  xu
haohao  wang
song  guo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:05:09

Abstract: Multi-modal domain generalization (MMDG) requires that models trained on multi-modal source domains can generalize to unseen target distributions with the same modality set. Sharpness-aware minimization (SAM) is an effective technique for traditional uni-modal domain generalization (DG), however, with limited improvement in MMDG. In this paper, we identify that modality competition and discrepant uni-modal flatness are two main factors that restrict multi-modal generalization. To overcome these challenges, we propose to construct consistent flat loss regions and enhance knowledge exploitation for each modality via cross-modal knowledge transfer. Firstly, we turn to the optimization on representation-space loss landscapes instead of traditional parameter space, which allows us to build connections between modalities directly. Then, we introduce a novel method to flatten the high-loss region between minima from different modalities by interpolating mixed multi-modal representations. We implement this method by distilling and optimizing generalizable interpolated representations and assigning distinct weights for each modality considering their divergent generalization capabilities. Extensive experiments are performed on two benchmark datasets, EPIC-Kitchens and Human-Animal-Cartoon (HAC), with various modality combinations, demonstrating the effectiveness of our method under multi-source and single-source settings. Our code is open-sourced.

Main paper: 
 introduction: 
 Domain generalization (DG) aims to equip models with the ability to perform robustly across unseen domains when trained only on several source domains, thereby enhancing their adaptability and utility in real-world scenarios, such as autonomous driving [1, 2] , medical health [3, 4] , person re-identification [5, 6] and brain-computer interface [7, 8] . Methods on how to deal with domain shift have been extensively proposed in the literature, including domain alignment [9] , meta-learning [10, 11] , data augmentation [12, 13] and ensemble learning [14] . Despite the remarkable achievements of DG in recent years, most of research still focuses on uni-modal data. The emergence of various multi-modal datasets and the requirement to complete a variety of multi-modal tasks highlight the need to address multi-modal domain generalization (MMDG) problems.Due to the complementary information that exists between modalities, MMDG aims to exploit generalization capabilities from each modality simultaneously. According to [15] , the generalization capability of deep neural networks (DNNs) is closely related to their flatness of minima on loss landscape (as shown in Fig. 1 (a) ), which motivates penalizing sharpness [16] and rewarding flatness [17] . Sharpness-aware minimization (SAM) [18] and its variants [14, 19] have been proposed to seek flatter minima and achieve better generalization across domains. Despite their success on uni-modal scenarios, in this paper, we argue that they are not compatible well in MMDG since the distinct properties between modalities pose two challenges (more details can be found in Sec.3.2).(1) Modality competition: according to [20] , multiple modalities will compete with each other during joint training, leading to inadequate knowledge exploitation for each modality [21, 22] , i.e, larger minima of loss as shown in Fig. 1 (b), and consequently worse generalization. (2) Discrepant uni-modal flatness: the generalization gap between modalities makes it hard to find their flat minima simultaneously, resulting in multi-modal networks incapable of utilizing generalization capabilities from all modalities, as illustrated in Fig. 1 (c ). Hence, existing methods can not fully exploit the generalization potential of each modality, which inevitably leads to sub-optimal solutions for MMDG.To overcome these challenges, we propose to construct consistent flat loss regions and enhance knowledge exploitation for each modality via cross-modal knowledge transfer. Traditional SAMbased methods are analyzed on parameter space. However, due to the heterogeneity between modalities, their parameter spaces could be extremely different (e.g., different model structures and parameter numbers), making it challenging to represent their correlation. Instead, we turn to optimization on representation-space loss landscape [23] as representations of different modalities can be mapped into a shared space, so that we can build their connections directly. Based on this, we propose a novel Cross-Modal Representation Flattening (CMRF) method to achieve consistent representation flat minima. As shown in Fig. 1 (d), we construct the interpolations by mixing paired multi-modal representations and then optimize them to flatten the high-loss regions between minima from different modalities. Specifically, we obtain more stable and generalizable cross-modal interpolations from moving averaged teacher model and then employ feature distillation to regularize the learning of each modality. The interpolations between modalities bring their flat regions closer, alleviating their flatness discrepancy. Moreover, the cross-modal knowledge transfer also helps to promote each modality and alleviate their competition. Our contributions can be summarized as:• To the best of our knowledge, we are the first to extend the uni-modal flatness analysis to MMDG, and empirically attribute the reasons for limited MMDG performance to two problems: modality competition and discrepant uni-modal flatness. • We construct shared representation space instead of parameter space to build connections between modalities directly and propose to flatten high-loss representation regions between modalities by interpolating mixed multi-modal representations and performing knowledge distillation to regularize the learning of each modality. • Extensive experiments verify the effectiveness and superiority of our framework on two benchmark datasets of EPIC-Kitchens and Human-Animal-Cartoon (HAC) under various modalities combinations on both multi-and single-source MMDG. 
related work: 
 Flat Minimum of Loss Landscape for DG. Domain generalization refers to the ability of models to perform well on new, unseen domains that are dissimilar with domains they were trained on. Numerous methods have been proposed to tackle the domain shift, while one type among them is to search for flat minima in loss landscapes [18, 24, 19] . Jiang et al. [15] conducted comprehensive measures and found that a sharpness-based measure has highest correlation with generalization. Based on that, Foret et al. [18] proposed sharpness-aware minimization (SAM) to seek parameters that lie in neighborhoods with uniformly low loss via perturbed gradients, while Wang et al. [25] further proposed to align the gradient directions between the empirical risk and the perturbed loss. Moreover, average weights during training has also shown to yield flatter minima [17] , which motivates more elegant average methods such as SWAD [14] and EoA [26] . In this paper, we try to optimize consistent flat minima for different modalities in representation-space loss landscapes instead of traditional parameter space.Multi-modal DG. Although uni-modal DG has been extensively studied in recent years, the research on MMDG is severely insufficient, while only few works have been done. Planamente et al. [27] proposed RNA-Net to balance audio and video feature norms via a relative norm alignment loss. Dong et al. [28] proposed a unified framework to achieve domain generalization in various multi-modal scenarios including multi-source, uni-source, and modality missing DG. In this paper, we extend the uni-modal flatness analysis to MMDG and address two particular problems in multi-modal scenarios.Mixup. Mixup [29] is a data augmentation technique introduced to improve the generalization performance of models. Traditional mixup and its variant CutMix [30] are performed on input data, while Verma et al. [31] further introduced Manifold Mixup that mixes the representations in each layer to produce smoother decision boundaries. However, Manifold Mixup and its variants [32, 33] are designed for uni-modal data, and only few works are on multi-modal scenarios [34, 35] . STEMM [34] aims to align speech and text features by mixing them, but is limited with its architecture-specific design. Oh et al. [35] introduced m 2 -Mix aiming at generating hard negative samples by mixing image and text embeddings to fine-tuning CLIP. Compared with them, our mixed multi-modal representations has no architecture restrictions and are used as teacher signals to guide various modalities to learn consistent flat minima.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Limitations can be found in Sec. 5Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 We follow the definition of multi-modal domain generalization problem as in [28] . In MMDG, we are given D source domains for trainingD train = D i |i = 1, • • • , D , where D i = x i j , y i j ni j=1 ∼ P iXY denotes the i-th domain with n i data instances sampled from a joint distribution of input samples and output labels P i XY . X and Y represent the corresponding random variables. Each input instancex i j = x i j k |k = 1, • • • , M ∈ X consists of M different modalities and y i j ∈ Y ⊂ R denotes corresponding label,where X and Y represent input and output space. The joint distributions in D train are different from each other: P i XY ̸ = P j XY , 1 ≤ i ̸ = j ≤ D.EQUATION)where E is the expectation and ℓ (•, •) is the loss function, e.g., cross-entropy loss for multi-modal classification tasks. In this paper, we use θ = {θ 1 , • • • , θ M } to denote the parameters of the neural network f , where θ i indicates the parameters for i-th modality. Therefore, the training loss over all training domains D train is defined as follows: The empirical risk minimization (ERM) of Eq. 2 tends to converge to sharp minima and SAM [18] is proposed to seek flatter minima on loss landscape with the following optimization:L (θ; D train ) = 1 D i=1 n i D i=1 ni j=1 ℓ f x i j ; θ , y i j (2)EQUATION)where ρ is a predefined constant controlling the radius of the neighborhood.MMDG aims to comprehensively exploit the generalization capabilities from each modality to learn more robust and generalized models. However, the generalization behavior of each modality in multi-modal networks has not been well explored. Here, we analyze the behavior of each modality and find the challenges for generalizable multi-modal networks.Modality competition leads to larger minima. As demonstrated in Tab. 1, we compare naive joint training and SAM about their uni-and multi-modal performance. SAM can clearly improve generalization on both uni-modal and multi-modal training. However, the uni-modal generalization from multi-modal trained network is worse than uni-modal trained network, whether or not SAM is applied (e.g, 56.65% vs. 58.73% without SAM and 58.80% vs. 61.68% with SAM on EPIC-Kitchens video). This phenomenon can be explained by modality competition [20, 36] that modalities in joint training compete with each other, making each modality under-explored. Our empirical results show that it not only degrades in-domain performance for each modality as discussed in [37, 38] , but also weakens their out-of-domain generalization, resulting in larger minima of loss as shown in Fig. 1 (b) .Generalization gap results in discrepant uni-modal flatness. We observe that applying SAM can only improve generalization of better modality in multi-modal network but has marginal benefit or even harm on weak modality (e.g., video generalization is improved from 56.65% to 58.80% on EPIC-Kitchens while the number of audio drops from 38.62% to 37.77%). According to [38] , the better modality will dominate multi-modal gradients. Hence, in Eq. 3, the gradient perturbation ε in SAM could also be dominated by the better modality, which means this optimization on multi-modal network tends to search for flatter regions for modality with better generalization but ignores other weak modalities. This suggests that conventional uni-modal SAM-based methods cannot find the coexisting flat minima for each modality due to their generalization gap, leading to discrepant flatness and consequently under-utilization of generalization from all modalities, as shown in Fig. 1 (c ). More results with other modality combinations can be found in Sec. 4.2 and Appendix. B.Based on the analyses above, in this paper, we aim to 1) accomplish consistent flat minima for all modalities in multi-modal network and 2) alleviate the competition between modalities to utilize their generalization comprehensively. Considering the correlation and complementary information between modalities, we propose to leverage cross-modal knowledge transfer to enhance MMDG.Representation-space loss landscape. Previous analysis of loss landscapes usually happens on parameter space [19, 39] . However, the network structures and sizes for different modalities are The overall framework of our method. The projectors map features with different dimensions to the same representation space. The teacher model is moving averaged from online model and generates cross-modal mixed representations as interpolations to distill the student representations. Uni-modal classifier is used to lower the loss of distilled features for each modality and a contrastive loss aims to alleviate gap between modalities. Only the online student model back propagates gradients. The teacher model is used for evaluation finally.commonly different, leading to disparate parameter spaces. This makes it difficult to catch correlations between modalities and produce consistent flat loss regions in parameter space. Inspired by [23] that introduces representation-space loss landscape, we turn to analyze loss landscapes of different modalities in representation space. Specifically, given a data point x i j = x i j k |k = 1, • • • , M , feature extractors are usually applied to transform input data into features with different dimensions:EQUATIONwhere g k is feature extractor for k-th modality, d k is feature dimension size and ∃k ̸ = l, d k ̸ = d l .In this paper, we use a projector P roj k (•) for k-th modality that maps its features into a shared representation space for all modalities with the same dimension d (omit superscript and subscript of domain and instance index for simplicity):EQUATIONGiven that each point in the representation space corresponds to a specific loss value, it is feasible to construct a landscape that maps each representation point to its associated loss value (e.g., horizontal axis indicates representation and vertical axis indicates loss in Fig. 1 (d) ). After training, each representation extracted from each training sample can be viewed as a minimum. And we can judge whether a representation minimum is flat or sharp according to its neighboring loss distribution. In the shared representation loss landscape, we can build connections between different modalities directly.Cross-modal representation interpolation. As discussed in Sec. 3.2, the discrepant uni-modal flatness severely impedes the utilization of generalization capability from each modality. The conclusion also applies to representation-space loss landscape since better modality still dominates gradients of representations, which optimizes weak modalities at sharp regions. Therefore, to obtain flat minima for various modalities simultaneously, we aim to flatten the high-loss regions between minima from different modalities. Given the paired multi-modal representations z k and z l , k ̸ = l, we construct interpolated representations between them by cross-modal representation mixup:EQUATION)where δ is mixing ratio. If the loss of mixed representations can be optimized to lower values, we would get a flatter region between modalities, as demonstrated in Fig. 1 (d ). However, according to [31] , directly optimization on mixed representations requires mixup at multiple eligible layers to be effective. It is impractical in multi-modal scenarios because representations of each layer for different modalities are generally at different scales, converting all them into a shared space is costly. In this paper, we propose a simple yet effective method that distills the knowledge from mixed representations to each modality and then optimize the learned representations. Firstly, we perform simple moving average (SMA) [26] for the online updated network θ k of each modality to establish the teacher network θt k , which can produce more stable and generalizable representations:EQUATIONwhere θ t k is the online model's state at iteration t of k-th modality. t 0 is the start iteration for SMA. Hence, the representation from teacher network is denoted as ẑk and the mixed representation of Eq. 6 should be rewritten as:EQUATIONwhere α is a hyperparameter in Beta distribution. Considering the semantic gap between modalities, we let interpolation closer to k-th modality act as its teacher signal, so distillation loss should be:EQUATIONThen, we assign specific classifier for each modality before P roj k (•) to online models and optimize the features by classification loss L k cls . The combination L k dis + L k cls flattens the neighboring representation-space loss landscape of k-th modality to other modalities. Further, we employ a multi-modal supervised contrastive loss on shared representation space, which can help to narrow the gap between modalities and make it conducive to flatten the region between them. For a random batch B with M × B uni-modal samples, we let i as the index of a uni-modal instance in the batch, and define P (i) as the set of uni-modal samples that have the same label with i (except itself). The supervised contrastive loss can be written as (notably, subscript here does not denote modality index but the index of each sample):EQUATIONwhere τ ∈ R + is the temperature parameter.Adaptive weight. As demonstrated in Tab. 1, the generalization capabilities between modalities may have significant gaps, so we propose to assign stronger flattening weights to better modalities. We compare the uni-modal validation accuracy from teacher model (calculated by the moving averaged uni-modal classifier) as a rough estimate of the difference in generalization ability between modalities (the performance of different modalities on in-domain validation set can generally reflect their strength in generalization capability, as shown in Appendix. B). The distillation loss can be modified as:EQUATIONwhere Âk denotes the validation accuracy of k-th modality by teacher model, µ is a hyperparameter (default 1.2 in this paper). In this way, the teacher signal with stronger generalization ability is applied with a larger distillation weight. Finally, we can get our final loss as follows:EQUATION)where L cls is the multi-modal classification loss, and λ 1 , λ 2 and λ 3 are hyperparameters to control the strength of each loss. Finally, we use teacher model for evaluation as it averages learned knowledge from student for better generalization.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: These information can be found in Appendix. A.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
experimental setting: 
 Dataset and implementation details. We utilize two benchmark datasets, EPIC-Kitchens [40] and Human-Animal-Cartoon (HAC) [28] , both of them have video, optical flow, and audio data. Three distinct domains for EPIC-Kitchens are D1, D2, and D3 and for HAC are humans (H), animals (A), and cartoon figures (C). Our experiment setup follow [28] . Training details including model structures, hyperparameters, and experimental environment can be found in Appendix. A.Baselines. We compare our CMRF with seven different baselines that can be divided into four groups: 1) Base, naive multi-modal joint training without any domain generalization strategies, 2) SAM [18] and SAGM [25] , searching for flat minima in parameter loss landscapes, 3) SWAD [14] and EoA [26] , ensemble-based methods for flat minima, and 4) RNA-Net [27] and SimMMDG [28] , domain generalization methods specifically designed for MMDG. SAM, SAGM, SWAD and EoA are initially designed for uni-modal DG and we extent them into MMDG. For all methods, we follow [41] and select the model with best validation (in-domain) accuracy to evaluate generalization on test (out-of-domain) data. We report the Top-1 accuracy for all results. Multi-modal multi-source DG. Tab. 2 illustrate the results of our CMRF and all baselines on EPIC-Kitchens and HAC under multi-modal multi-source domain generalization setting, where the models are trained on multiple source domains and test on one target domain. We conduct experiments by combining any two modalities, as well as all three modalities, to validate the generalization of our method. As we can see from Tab. 2, our CMRF outperforms all baselines on almost all settings and achieves great improvement on the average results (by up to 3.52% with video-audio modalities on HAC). The uni-modal DG methods, especially SAGM and EoA, can improve the generalization of multi-modal network to a certain extent, but their improvements are limited as they do not consider modality competition and inconsistent flatness between modalities. Two MMDG methods RNA-Net and SimMMDG also perform less than satisfactory since they do not fully exploit the generalization capability of each modality.Multi-modal single-source DG. Our CMRF does not requires domain labels for training, making it feasible to perform multi-modal single-source domain generalization, where models are trained on a single source domain and test on other multiple target domains. The results trained with three modalities are presented in Tab. 3. Our CMRF still apparently outperforms all baselines on average accuracy, despite being trained only on single-source domain data. For baselines with domain generalization strategies, they can not improve consistently across datasets, e.g., SimMMDG achieves the second best on EPIC-Kitchens but has limited improvement on HAC, showing their unstable generalization and their limitations in the single-source DG setting.Uni-modal performance in MMDG. As we discussed in Sec. 3.2, exploiting the generalization capability of each modality simultaneously is the key to improving multi-modal domain generalization performance. Therefore, we evaluate the uni-modal performance from multi-modal trained networks to show the superiority of our method. We freeze the trained uni-modal feature extractor and train a linear classifier to test uni-modal performance. The results of average multi-source accuracy on EPIC-Kitchens are shown in Tab. 4 . We can see that our CMRF not only improves the multi-modal domain generalization, but also greatly promotes its uni-modal domain generalization, even better than that of uni-modal training (60.66% vs. 58.73% and 43.12% vs. 40.04% for video and audio on EPIC-Kitchens), indicating the effectiveness of CMRF to use cross-modal knowledge to promote the Ablation on each design. Our CMRF contains five main modules: distillation loss L k dis , uni-modal classification loss L k cls , multi-modal supervised contrastive loss L con , adaptive weight, and SMA for teacher model. We conduct extensive ablation experiments to verify the effectiveness of each proposed module on EPIC-Kitchens with video-audio data under multi-source domain generalization setting. The results are illustrated in Tab. 5. Only applying distillation loss or uni-modal classification loss improves slightly and their combination leads to noticeable increase, highlighting the importance of flattening representation loss landscape between modalities for domain generalization. However, it does not guarantee steady improvement, e.g., the accuracy decreases from 54.94% to 52.75% in D2, D3 → D1 setting. Multi-modal supervised contrastive loss can enhance the average generalization by a small margin. Adaptive weight and using SMA network as teacher can both improve MMDG by a large margin, suggesting that it is necessary to emphasize the more generalized modality and obtain more stable distillation signals. Finally, combining all of them achieves the best results for multi-modal domain generalization, hence, each of them is indispensable. Ablation on interpolations. In this paper, we mix multimodal representations in the random ratio generated from Beta distribution as teacher signals, and choose interpolations closer to current modality for distillation, as in Eq. 9.We conduct experiments by using different forms of teacher signals to verify our method's effectiveness, as presented in Tab. 6. For k-th modality, we set δ to 1, 0, 0.5 for self-modal distillation, cross-modal distillation, and distillation with fixed mixing ratio. Since self-modal distillation can enhance learning for each modality via more generalizable signals, it achieves great performance next to ours. The heterogeneous knowledge between modalities makes cross-mode distillation worse. Fixed mixing ratio only locates one interpolation while our random ratio covers all possible points, resulting in our better performance.Comparison with methods designed for modality competition. Here, we conduct experiments with three baselines Gradient Blending [42] , OGM-GE [37] , and PMR [38] for modality competition as we attribute it as one challenge for MMDG. We not only report out-of-domain test accuracy but also in-domain validation results, as shown in Tab. 7 . We can see that these methods can actually promote their performance on multi-modal validation set since they mitigate the competition. However, they tend to locate at sharp minima and the generalization gap between modalities still makes it hard to build consistent flat minima for different modalities. Hence, their performance increase on test set is limited, while our method achieves significant improvement on both validation and test sets.Parameter sensitivity. Fig. 3 shows the results of different values on loss weights λ 1 , λ 2 , and λ 3 . Since our method uses the moving averaged teacher model for evaluation, it is insensitive to hyperparameters. 
conclusion: 
 In this paper, we analyze the behavior of multi-modal domain generalization and find that modality competition and discrepant uni-modal flatness restrict the generalization capability of multi-modal network. To address these challenges, we propose cross-modal representation flattening (CMRF) to construct consistent flat regions in a shared representation-space loss landscape. Our method builds interpolations by mixing multi-modal representations from moving averaged teacher model and use feature distillation to optimize the high-loss regions between modalities. Our extensive experiments on two benchmark datasets demonstrate the effectiveness of our method to promote multi-modal domain generalization, as well as uni-modal domain generalization in multi-modal network.Limitations. Currently, we need to test on validation set to estimate generalization of each modality for Eq. 11, which can be time-consuming with the scale increase of validation set. In future work, we can add low-frequency noise as in [23] for domain shifting to evaluate the generalization. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: No new data or models are released.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the papaer [28] that our code is based on.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Dataset. We utilize two benchmark datasets: EPIC-Kitchens [40] and Human-Animal-Cartoon (HAC) [28] . Our experimental setup follows the protocols established for the EPIC-Kitchens dataset in [43] and for the HAC dataset in [28] . The EPIC-Kitchens dataset encompasses eight actions ('put', 'take', 'open', 'close', 'wash', 'cut', 'mix', and 'pour') captured across three different kitchens, forming three distinct domains: D1, D2, and D3. The HAC dataset comprises seven actions ('sleeping', 'watching tv', 'eating', 'drinking', 'swimming', 'running', and 'opening door') executed by humans (H), animals (A), and cartoon figures (C), resulting in three separate domains: H, A, and C. The HAC dataset includes 3381 video clips sourced from the internet, with approximately 1000 samples per domain. Both datasets offer three modalities: video, audio, and optical flow.Baselines. In our experiments, we compare our CMRF with seven different baselines that can be divided into four groups: 1) Base, naive multi-modal joint training without any domain generalization strategies, 2) SAM [18] and SAGM [25] , searching for flat minima in parameter loss landscapes, 3) SWAD [14] and EoA [26] , ensemble-based methods for flat minima, and 4) RNA-Net [27] and SimMMDG [28] , domain generalization methods specifically designed for MMDG. SAM, SAGM, SWAD and EoA are initially designed for uni-modal DG and we extent them into MMDG. For all methods, we follow [41] and select the model with best validation (in-domain) accuracy to evaluate generalization on test (out-of-domain) data. We report the Top-1 accuracy for all results.Implementation Details. In our framework, we conduct experiments across three modalities: video, audio, and optical flow, adhering to the implementation described in [28] . We leverage the MMAction2 toolkit [44] for our experimental setup. To encode visual information, we utilize the SlowFast network [45] , initialized with pre-trained weights on Kinetics-400 [46] . For the audio encoder, we employ ResNet-18 [47] , initialized with weights from the VGGSound pre-trained checkpoint [48] . The optical flow encoder uses the SlowFast network's slow-only pathway with Kinetics-400 pre-trained weights. The dimensions of the uni-modal feature h are 2304 for video, 512 for audio, and 2048 for optical flow. For the projector P roj k (•), we implement a multi-layer perceptron with two hidden layers of size 2048 and output size 128. We use the Adam optimizer [49] with a learning rate of 0.0001 and a batch size of 16. The scalar temperature parameter τ is set to 0.1. Additionally, we set λ 1 = 2.0, λ 2 = λ 3 = 3.0, α in the Beta distribution to 0.1, and the SMA start iteration t 0 to 400 for EPIC-Kitchens and 100 for HAC respectively. All experiments were conducted on an NVIDIA GeForce RTX 3090 GPU with a 3.9-GHz Intel Core i9-12900K CPU. The model is trained with 15 epochs, taking two hours.Uni-modal in-domain validation performance. Modal competition refers to the mutual inhibition between modalities in joint training, which is reflected in in-domain performance straightforwardly as studied in previous literature. In Tab.8 we give the uni-modal validation results (in-domain) on EPIC-kitchens with video and audio data. Modal competition is manifested in that each single modality of Base performs worse than uni-modal training, which further leads to worse out-of-domain performance as shown in Tab. 9. Our method achieves the best uni-modal in-domain performance, indicating that it optimizes modal competition effectively, which in turn improves the generalization ability to other domains as in Tab. 4. Flatness visualization. To evaluate the loss flatness, we can apply low-frequency perturbation from the Gaussian Distribution on representations, where the variance controls the perturbation strength. With the increase of Variance, our method has the smallest performance drop on each modality, indicating that our method achieves flatter loss landscape for both modalities simultaneously and in turn provides flatter multi-modal loss landscape.Uni-modal out-of-domain performance. Here, we give the detailed results of uni-modal performance comparison on EPIC-Kitchens in Tabs. 9, 10, and 11, which form the results in Tab. 4 in the main paper. The results for HAC dataset are demonstrated in Tabs. 12, 13, and 14. Our method can achieve the best uni-modal, as well as multi-modal, performance on both datasets with various modality combinations. Validation and test comparison with uni-modal training. In Tab. 15 and Tab. 16, we report the in-domain validation and out-of-domain test results on EPIC-kitchens and HAC datasets for each modality. We can see that for each modality, its validation performance is strongly positive correlated to its test performance, i.e., modalities that perform better on the validation set usually perform better on the test set. This provides empirical support for us to use validation set accuracy in Eq. 11 to evaluate the generalization ability of different modalities. Justification: The contributions and scope can be found in Sec. 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We follow the Code Of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of the work performed as we aim to train networks on public datasets.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper identifies two main issues related to multi-modal domain generalization (MMDG) methods -- modality competition and discrepant uni-modal flatness. The authors propose a CMRF method, which mixes multi-modal representations from a teacher model and uses feature distillation to optimize high-loss regions between modalities. It uses a supervised multi-modal contrastive loss to enhance the flat loss region in the shared feature space. Furthermore, CMRF does adaptive weighting based on modal validation set accuracy to better utilize each modality. The effectiveness of the proposed method is validated through experiments.

The reviews of the paper are overall positive. The paper is recommended for acceptance as a poster paper. The authors are advised to address the remaining feedback of the reviewers, which were not handled during author feedback time.

Reviews: review: paper_summary: paper_summary: In this paper, the authors identify two primary limitations in multi-modal domain generalization (MMDG): modality competition and discrepant uni-modal flatness. To address these challenges, they propose a novel approach called Cross-Modal Representation Flattening (CMRF). CMRF constructs interpolations by mixing multi-modal representations from a teacher model and uses feature distillation to optimize high-loss regions between modalities. It adopts a supervised multi-modal contrastive loss to enhance the flat loss region in the shared feature space. Furthermore, CMRF employs adaptive weighting based on modal validation set accuracy to better utilize each modality. The effectiveness of the proposed method is validated through extensive experiments.
strength_weakness: strength_weakness: 1. This paper lacks a detailed discussion on the practical implementation aspects of the proposed method, such as computational requirements and scalability.
2. Although the authors' claims about addressing modality competition and discrepant uni-modal flatness are intuitive, the paper does not adequately address how to evaluate the loss flatness for multi-modal data. Additionally, the proposed method lacks sufficient evidence and empirical results to convincingly demonstrate its effectiveness in optimizing these specific limitations within multi-modal domain generalization.
questions: questions: 1. What does the comparative data in Table 1 represent?
limitations: limitations: The authors have discussed the limitations in the paper.
paper_summary: paper_summary: The paper addresses the problem of multi-modal domain generalization (MMDG). The authors identify two main challenges in MMDG: modality competition and discrepant uni-modal flatness and propose a novel method called Cross-Modal Representation Flattening (CMRF). This method optimizes the representation-space loss landscapes instead of the traditional parameter space, allowing for direct connections between modalities. The authors demonstrate the effectiveness of CMRF through extensive experiments on benchmark datasets EPIC-Kitchens and Human-Animal-Cartoon (HAC), showing superior performance in both multi-source and single-source settings.
strength_weakness: strength_weakness: 1. Are there any visualization or quantitative results to show that after CMRF training the loss landscape of different modalities get consistent flat region?
2. What is the influence of α in the Beta distribution? What if it is larger than 1?
questions: questions: Are there any visualization or quantitative results to show that after CMRF the loss landscape of different modalities get consistent flat region? What is the influence of α in the Beta distribution? What if it is larger than 1?
limitations: limitations: Currently, the paper need to test on validation set to estimate generalization of each modality for Eq. 11, which can be time-consuming with the scale increase of validation set.
paper_summary: paper_summary: This paper extends the analysis of unimodal flatness to the domain of multi-modal domain generalization (MMDG) for the first time, and proposes the Cross-Modal Representation Flattening (CMRF) method. By constructing a shared representation space and cross-modal knowledge distillation, it addresses the issues of competition and inconsistency in flatness across multiple modalities. Experimental results demonstrate that CMRF significantly outperforms all baseline methods in the multi-modal multi-source domain generalization setting, effectively enhancing the model's generalization capability. In particular, it achieves an average improvement of up to 3.52% in the video-audio modality combination.
strength_weakness: strength_weakness: 1. The paper mentions using simple moving average to build the teacher network, but how to extract knowledge from the mixed representation and how to transfer this knowledge to each modality is not clear enough.

2. The paper points out that inter-modal competition is a key issue, but the specific performance of the competition and the detailed evaluation of the proposed mitigation strategies are not thorough. It is hoped that the authors can provide more quantitative analysis on inter-modal competition and validation of the effectiveness of the mitigation strategies.
questions: questions: 1. In Section 3.3, the authors proposed a method for cross-modal representation flattening, but further explanation is needed on how to construct interpolation representations and how to optimize these interpolations to achieve flattening of the loss region.

2. The authors should provide more details on how to dynamically adjust weights based on the differences in the generalization ability of each modality.
limitations: limitations: Authors have adequately addressed the limitations.

Comments: comment: The authors' rebuttal dispelled some of my concerns, and I choose to maintain the score.
Thank the authors for addressing my concerns. I tend to increase my score to 5.
Thank the authors for their rebuttal. Most of my concerns have been well-addressed and I thus increased my score to 6.