SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hao Dong $^{1}$  Ismail Nejjar $^{2}$  Han Sun $^{2}$  Eleni Chatzi $^{1}$  Olga Fink $^{2}$
$^{1}$ ETH Zürich  $^{2}$ EPFL
{hao.dong, chatzi}@ibk.baug.ethz.ch, {ismail.nejjar, han.sun, olga.fink}@epfl.ch
In real-world scenarios, achieving domain generalization (DG) presents significant challenges as models are required to generalize to unknown target distributions. Generalizing to unseen multi-modal distributions poses even greater difficulties due to the distinct properties exhibited by different modalities. To overcome the challenges of achieving domain generalization in multi-modal scenarios, we propose SimMMDG, a simple yet effective multi-modal DG framework. We argue that mapping features from different modalities into the same embedding space impedes model generalization. To address this, we propose splitting the features within each modality into modality-specific and modality-shared components. We employ supervised contrastive learning on the modality-shared features to ensure they possess joint properties and impose distance constraints on modality-specific features to promote diversity. In addition, we introduce a cross-modal translation module to regularize the learned features, which can also be used for missing-modality generalization. We demonstrate that our framework is theoretically well-supported and achieves strong performance in multi-modal DG on the EPIC-Kitchens dataset and the novel Human-Animal-Cartoon (HAC) dataset introduced in this paper. Our source code and HAC dataset are available at https://github.com/donghao51/SimMMDG.
Domain generalization (DG) has received significant attention in the research community [66]. In real-world scenarios such as autonomous driving [13, 26], robotics [17], action recognition [16], and fault diagnosis [22], it is crucial that models trained on limited source domains perform well across novel target domains. To tackle distribution shift problems, numerous DG algorithms have been proposed, including domain-invariant feature learning [51], feature disentanglement [56], data augmentation [72], and meta-learning [41]. However, most of these algorithms are designed for unimodal data, such as images [40] and time series data [49]. More recently, the emergence of large-scale multi-modal datasets [16, 7] has highlighted the need to address multi-modal DG across multiple modalities, such as audio-video [36, 73], image-language [58, 33], and LiDAR-camera [44, 18]. However, to date, only RNA-Net [57] has focused on the multi-modal DG problem, with a relative norm alignment loss proposed to balance the feature norms of audio and video.
Multi-modal DG is closely related to multi-modal representation learning [47]. Traditional multi-modal contrastive learning frameworks [58, 43] aim to project the features of different modalities into a common embedding space. However, this approach may not be optimal as different modalities consist of both shared information that is consistent across modalities and unique information that is specific to each modality. Consequently, attempting to align all modalities together can be challenging and impractical. For instance, consider descriptions of the same object that use entirely different modalities, such as video, audio, and optical flow, as illustrated in Fig. 1 (a). All modalities share some
(a) Modality-specific and modality-shared information
(b) HAC dataset
Figure 1: (a). Different modalities possess shared information, while simultaneously containing unique information exclusive to each modality. Inspired by this, we propose to split the feature of each modality into modality-specific and modality-shared parts in our framework. (b) Our new multi-modal DG dataset consists of three domains and three modalities. For each domain, the actor of different actions (opening door in this case) could be either a human, animal, or cartoon figure.
information since they describe the same object, but each modality further provides modality-specific information. For example, videos typically convey visual appearance information and sequential relations between frames, while audio provides tone information that is closely linked to sentiment and frequency information for different sounds. Optical flow, on the other hand, offers more intuitive information on motions and associated velocities. Therefore, simply mapping features from different modalities into the same embedding space would preserve only modality-shared information while overlooking modality-specific details. This could result in the loss of modality-specific information and a decline in downstream task performance.
Effective multi-modal DG frameworks that can address the challenges outlined above are urgently needed. To this end, we propose SimMMDG - a Simple yet effective Multi-Modal DG framework that can be applied to two or more modalities. We first propose to split the feature embedding of each modality into modality-specific and modality-shared parts, ensuring that complementary information from different modalities is retained. We apply supervised contrastive learning on modality-shared features and incorporate distance constraints for modality-specific features to ensure that the learned representations are informative. Moreover, we introduce a cross-modal translation module to further regularize the learned features and facilitate missing-modality generalization. Some theoretical insights from both the multi-modal representation learning perspective and the domain generalization perspective are provided to demonstrate that our approach is well-motivated in theory. Finally, we release a novel Human-Animal-Cartoon (HAC) dataset, to promote research in multi-modal domain generalization, as shown in Fig. 1 (b). Our contributions can be summarized as:
- We propose a universal framework for multi-modal domain generalization that demonstrates both simplicity and effectiveness and outperforms existing methods on several challenging datasets. We also demonstrate its efficacy in general multi-modal classification setups.
- We address the missing-modality generalization problem and propose a cross-modal translation module as a solution, which is robust even in scenarios where multiple modalities are missing.
- We provide theoretical insights in support of the efficacy of our proposed approach.
- We introduce a new multi-modal dataset that includes three modalities and large domain shifts, which can serve as a challenging benchmark for future research on multi-modal DG.
Domain Generalization refers to the process of training a model on multiple source domains in order to generalize to unseen target domains. This task is more challenging than domain adaptation (DA) [67] as target domain data cannot be accessed during training. Prior research has identified three main categories of domain generalization methods [66], including data manipulation, representation learning, and learning strategies. Data manipulation techniques aim to improve generalization performance by increasing the diversity of training data. For example, Tobin et al. [61] use simulated environments for additional data generation, while Zhou et al. [75] train a domain transformation network using adversarial training to synthesize data from previously unseen domains. Mixup [72] generates new instances by performing linear interpolations between data and labels. Representation
learning methods seek to learn domain-invariant representations through the use of domain-adversarial neural networks [25, 42], explicit feature distribution alignment [63], and instance normalization [54]. Other approaches utilize learning strategies to improve the generalization performance. Examples of such approaches include meta-learning [41], gradient operation [31], and self-supervised learning [8].
Multi-modal Representation Learning aims to learn robust representations through two or more modalities, which can then be applied to various downstream tasks. While unified models [68, 4]okenize various input modalities into sequences [3] and employ a single Transformer [65] for joint learning, methods such as CLIP [58] and ALIGN [33] use separate encoders for each modality and leverage contrastive loss to align features. However, these methods align features from different modalities into the same embedding space and may only preserve modality-shared information. In contrast, we propose splitting the features of each modality into modality-specific and modality-shared components to ensure that complementary information from different modalities is preserved. A recent work [34] also proposes to split the features into different components in multi-modal representation learning and provide an information-theoretical analysis. However, they only deal with two modalities and without considering missing-modality cases. They also don't use the label information within a batch in contrastive learning.
Multi-modal DA and DG. Several approaches exist for multi-modal DA. For instance, Munro and Damen [52] propose a self-supervised alignment approach along with adversarial alignment for multi-modal DA, while Kim et al. [38] leverage cross-modal contrastive learning to align cross-modal and cross-domain representations. Zhang et al. [73] propose an audio-adaptive encoder and an audio-infused recognizer to address domain shifts. Notably, RNA-Net [57] is the only method known to address multi-modal DG problem, by introducing a relative norm alignment loss to balance audio and video feature norms.
3.1 Problem Setting: Multi-modal Domain Generalization
We commence by presenting the definition of the multi-modal domain generalization problem based on the definition of unimodal DG, as described in [66]. Let  $\mathbf{X}$  represent a nonempty input space, and  $\mathcal{Y}$  denote an output space. A domain, denoted as  $\mathcal{D}$ , consists of data sampled from a distribution,  $\mathcal{D} = \{(\mathbf{x}_j, y_j)\}_{j=1}^n \sim P_{XY}$ , where  $P_{XY}$  denotes the joint distribution of input samples and output labels.  $X$  and  $Y$  represent the corresponding random variables.
Definition 1 (Multi-modal domain generalization). In multi-modal domain generalization, we are given  $D$  training domains  $\mathcal{D}_{train} = \{\mathcal{D}^i\mid i = 1,\dots ,D\}$ , where  $\mathcal{D}^i = \{(\mathbf{x}_j^i,y_j^i)\}_{j = 1}^{n_i}$  denotes the  $i$ -th domain with  $n_i$  data instances. Each data instance  $\mathbf{x}_j^i = \{(\mathbf{x}_j^i)_k\mid k = 1,\dots ,M\} \in \mathbf{X}$  is comprised of  $M$  different modalities and  $y_{j}^{i}\in \mathcal{V}\subset \mathbb{R}$  denotes the label. The joint distributions between each pair of domains are different:  $P_{XY}^{i}\neq P_{XY}^{j},1\leq i\neq j\leq D$ . The goal of multi-modal domain generalization is to learn a robust and generalizable predictive function  $f:\mathbf{X}\to \mathcal{V}$  from  $D$  training domains and  $M$  data modalities to achieve a minimum prediction error on an unseen test domain  $\mathcal{D}_{test}$  (i.e.,  $\mathcal{D}_{test}$  cannot be accessed during training and  $P_{XY}^{test}\neq P_{XY}^{i}$  for  $i\in \{1,\dots ,D\}$ ):
$$
\min  _ {f} \mathbb {E} _ {(\mathbf {x}, y) \in \mathcal {D} _ {\text {t e s t}}} [ \ell (f (\mathbf {x}), y) ], \tag {1}
$$
where  $\mathbb{E}$  is the expectation and  $\ell (\cdot ,\cdot)$  is the loss function.
3.2 SimMMDG
We present the SimMMDG framework for addressing the multi-modal DG problem, as depicted in Fig. 2. Our approach involves splitting the feature embedding of each modality into modality-specific and modality-shared components. We aim to map modality-shared embeddings of data samples with the same label, whether they belong to the same or different modalities, to be as close as possible (and vice versa for samples with different labels). For modality-specific features within each modality, our objective is to maximize the distance from their modality-shared features to capture unique and complementary information. Additionally, we incorporate a cross-modal translation module to regularize learned features and address the missing-modality generalization problem, which is essential for real-world testing scenarios where one or more modalities may be absent due to unpredictable factors.
Figure 2: Overview of SimMMDG. We split the features of each modality into modality-specific and modality-shared parts. For the modality-shared part, we use supervised contrastive learning to map the features with the same label to be as close as possible. For modality-specific features, we use a distance loss to encourage them to be far from modality-shared features, promoting diversity within each modality. Additionally, we introduce a cross-modal translation module that regularizes features and enhances generalization across missing modalities.
3.2.1 Within-modal Feature Splitting
Traditional multi-modal learning frameworks [58, 43] aim to map features from various modalities into a common embedding space using contrastive learning. However, this approach may not be optimal because different modalities often contain a mix of both homogeneous and heterogeneous information, making it challenging and impractical to align them all together. For instance, consider the language and visual modality [2]. Both modalities can describe people, objects, actions, and gestures, which reflect shared information between them. However, images can provide additional information such as texture, depth, and visual appearance that are not available in language data. Similarly, language can provide syntactic structure, vocabulary, and morphology, which are not present in image data. These are specific pieces of information that are unique to each modality.
Thus, the simple mapping of features from different modalities into the same embedding space may result in the loss of modality-specific information and consequently lead to decreased performance in downstream tasks. Guided by this intuition, we propose a novel approach that involves the separation of the feature embedding of each modality into modality-specific and modality-shared components. For example, given a unimodal embedding  $\mathbf{E}$ , we denote it as  $\mathbf{E} = [\mathbf{E}_s; \mathbf{E}_c]$ , where  $\mathbf{E}_s$  is modality-specific feature and  $\mathbf{E}_c$  is modality-shared feature.
Multi-modal Supervised Contrastive Learning. We expect modality-shared features  $\mathbf{E}_c$  to possess characteristics that are shared among distinct modalities. If data instances from different modalities have the same label, we expect their modality-shared features to be as close as possible in the embedding space. To achieve this objective, we leverage the availability of labels for each data instance and employ supervised contrastive loss [37] for effective guidance. For a set of  $N$  randomly sampled label pairs,  $\{\pmb{x}_j, y_j\}_{j=1,\dots,N}$ , the corresponding batch used for training consists of  $M \times N$  pairs,  $\{\tilde{\pmb{x}}_k, \tilde{y}_k\}_{k=1,\dots,M \times N}$ , where  $\tilde{\pmb{x}}_{M \times j}, \tilde{\pmb{x}}_{M \times j-1}, \dots, \tilde{\pmb{x}}_{M \times j-M+1}$  are data instances from  $M$  different modalities in  $\pmb{x}_j$  ( $j=1,\dots,N$ ) and  $\tilde{y}_{M \times j} = \dots = \tilde{y}_{M \times j-M+1} = y_j$ .
Let  $i \in I \equiv \{1, \dots, M \times N\}$  be the index of an arbitrary unimodal sample within a batch. We define  $A(i) \equiv I \setminus \{i\}$ ,  $P(i) \equiv \{p \in A(i) : \tilde{y}_p = \tilde{y}_i\}$  as the set of indices of all positive samples in the batch which share the same label as  $i$ . The cardinality of  $P(i)$  is denoted as  $|P(i)|$ . Then, the multi-modal supervised contrastive learning loss can be written as follows:
$$
\mathcal {L} _ {\text {c o n}} = \sum_ {i \in I} \frac {- 1}{| P (i) |} \sum_ {p \in P (i)} \log \frac {\exp \left(\boldsymbol {z} _ {i} \cdot \boldsymbol {z} _ {p} / \tau\right)}{\sum_ {a \in A (i)} \exp \left(\boldsymbol {z} _ {i} \cdot \boldsymbol {z} _ {a} / \tau\right)}, \tag {2}
$$
with  $\mathbf{z}_k = \text{Proj}(g(\tilde{\mathbf{x}}_k)) \in \mathcal{R}^{D_P}$ , where  $g(\cdot)$  is the feature extractor, that maps  $\mathbf{x}$  to modality-specific and modality-shared features,  $\mathbf{E} = [\mathbf{E}_s; \mathbf{E}_c] = g(\mathbf{x})$ , where  $\mathbf{E}_s, \mathbf{E}_c \in R^{D_E}$ , and  $\text{Proj}(\cdot)$  is the
projection network that maps  $\mathbf{E}_c$  to a vector  $z = Proj(\mathbf{E}_c) \in \mathcal{R}^{D_P}$ . The inner product between two projected feature vectors is denoted by  $\cdot$ , and  $\tau \in \mathcal{R}^+$  is a scalar temperature parameter.
Feature Splitting with Distance. To ensure that the modality-specific features  $\mathbf{E}_s$  carry unique and complementary information, we aim to maximize their dissimilarity from the corresponding modality-shared features  $\mathbf{E}_c$ . To achieve this goal, we utilize negative  $\ell_2$  distance and formulate a distance loss on  $\mathbf{E}_s$  and  $\mathbf{E}_c$  as:
$$
\mathcal {L} _ {d i s} = \frac {- 1}{M} \sum_ {i = 1} ^ {M} \left| \left| \mathbf {E} _ {s} ^ {i} - \mathbf {E} _ {c} ^ {i} \right| \right| _ {2} ^ {2}, \tag {3}
$$
where  $M$  is the number of modalities,  $\mathbf{E}_s^i$  and  $\mathbf{E}_c^i$  are the modality-specific and modality-shared features of the  $i$ -th modality.
3.2.2 Cross-modal Translation
Simply increasing the distance between  $\mathbf{E}_s$  and  $\mathbf{E}_c$  may not yield optimal results. The proposed cross-modal translation module aims to ensure the meaningfulness of modality-specific features by exploiting the implicit relationships and approximate translation mappings that exist between the  $M$  modalities within the same data instance. This is achieved through a multi-layer perceptron (MLP) [28] that translates the feature embedding  $\mathbf{E}$  across modalities. For instance,  $\mathbf{E}_t^j = MLP_{\mathbf{E}^i \to \mathbf{E}^j}(\mathbf{E}^i)$  implies that we translate the embedding  $\mathbf{E}^i = [\mathbf{E}_s^i; \mathbf{E}_c^i]$  of the  $i$ -th modality to the  $j$ -th modality using  $MLP_{\mathbf{E}^i \to \mathbf{E}^j}$ , resulting in the translated embedding  $\mathbf{E}_t^j$ . More intuitions behind the cross-modal translation module are discussed in the appendix. To ensure that the translated embedding  $\mathbf{E}_t^j$  is a meaningful representation of the  $j$ -th modality, we aim to minimize its  $\ell_2$  distance from the real embedding of the  $j$ -th modality,  $\mathbf{E}^j$  and the cross-modal translation loss is defined as:
$$
\mathcal {L} _ {\text {t r a n s}} = \frac {1}{M (M - 1)} \sum_ {i = 1} ^ {M} \sum_ {j \neq i} | | M L P _ {\mathbf {E} ^ {i} \rightarrow \mathbf {E} ^ {j}} (\mathbf {E} ^ {i}) - \mathbf {E} ^ {j} | | _ {2} ^ {2}. \tag {4}
$$
3.3 Final Loss
The final loss is obtained as the weighted sum of the previously defined losses:
$$
\mathcal {L} = \mathcal {L} _ {c l s} + \alpha_ {c o n} \mathcal {L} _ {c o n} + \alpha_ {d i s} \mathcal {L} _ {d i s} + \alpha_ {t r a n s} \mathcal {L} _ {t r a n s}, \tag {5}
$$
where  $\mathcal{L}_{cls}$  is the cross-entropy loss for classification, and where  $\alpha_{\mathrm{con}}$ ,  $\alpha_{\mathrm{dis}}$ , and  $\alpha_{\mathrm{trans}}$  are hyperparameters that control the relative importance of the contrastive learning, dissimilarity, and cross-modal translation terms, respectively.
3.4 Missing-modality Generalization
During the training phase, we have access to data from all modalities. However, in the testing phase, one or more modalities may be absent due to unpredictable reasons, such as sensor malfunction or loss of communication. In such cases, the system's robustness toward potential missing modalities becomes critical. To address the missing-modality scenario, we utilize our cross-modal translation module. In normal circumstances, the output of the network is defined as:
$$
y = f (x) = h (g (x)) = h \left(\left[ \mathbf {E} ^ {1}, \dots , \mathbf {E} ^ {i}, \dots , \mathbf {E} ^ {M} \right]\right), \tag {6}
$$
where  $g(\cdot)$  is the feature extractor and  $h(\cdot)$  is the classifier. If the  $i$ -th modality is missing during testing, one solution is to replace its embedding with zero. The output is then given as:
$$
y = h \left(\left[ \mathbf {E} ^ {1}, \dots , \mathbf {0}, \dots , \mathbf {E} ^ {M} \right]\right). \tag {7}
$$
However, simply replacing the embeddings with null entries may have an adverse effect on the network's performance. To address this issue, we propose using our cross-modal translation module to predict and substitute the missing modality's embedding with information from available modalities, resulting in a more robust output for the network. The output of the network is then given as:
$$
y = h ([ \mathbf {E} ^ {1}, \dots , \mathbf {E} _ {t} ^ {i}, \dots , \mathbf {E} ^ {M} ]), \qquad \qquad (8)
$$
$$
\mathbf {E} _ {t} ^ {i} = \frac {1}{M - 1} \sum_ {j \neq i} ^ {M} M L P _ {\mathbf {E} ^ {j} \rightarrow \mathbf {E} ^ {i}} \left(\mathbf {E} ^ {j}\right). \tag {9}
$$
In cases where multiple modalities are missing during testing, we can use the same strategy to utilize available modalities to predict and substitute the missing ones, similar to Eq. (9). The benefits of our approach are demonstrated in the subsequent experiments.
4 Theoretical Insights
4.1 Multi-modal Representation Learning Perspective
We first generalize Theorem 3.1 in [34] to the case of  $M$  modalities. Let  $p$  be the joint distribution of  $(x_{1},\ldots ,x_{M},y)$ , where  $x_{i}$  is the  $i$ -th modality and  $y$  is the target. We define the information gap as  $\Delta_p\coloneqq \max \{I(x_i;y);i = 1,\dots,M\} -\min \{I(x_i;y);i = 1,\dots,M\}$ , where  $I(x_{i};y)$  is the mutual information between modality  $x_{i}$  and the target variable  $y$ . The information gap serves to characterize the effectiveness of the modalities in predicting the target  $y$ . We use the cross-entropy loss, denoted by  $\ell_{\mathrm{CE}}$ , and the prediction function, denoted by  $f$ .
Theorem 1. For  $M$  feature extractors  $g^{i}(\cdot)$ $(i = 1,\dots,M)$ , if the multi-modal features  $\mathbf{E}^i = g^i (x_i)$  are perfectly aligned in the feature space, i.e.,  $\mathbf{E}^1 = \ldots = \mathbf{E}^i = \ldots = \mathbf{E}^M$ , then  $\inf_{f}\mathbb{E}_{p}[\ell_{CE}(f(\mathbf{E}^{1},\dots,\mathbf{E}^{M}),y)] - \inf_{f^{\prime}}\mathbb{E}_{p}[\ell_{CE}(f^{\prime}(x_{1},\dots,x_{M}),y)]\geq \Delta_{p}$ .
The proof of this theorem can be found in the appendix. Theorem 1 indicates that the optimal prediction error, achieved with perfectly aligned features, is at least  $\Delta_p$  larger than when using the raw input modalities. In practical scenarios where one modality is less informative for prediction, the information gap  $\Delta_p$  tends to be substantial. Consequently, this leads to a notable increase in prediction errors in downstream tasks. Furthermore, achieving perfect modality alignment enforces the aligned features to exclusively contain predictive information present in all input modalities, potentially causing the loss of modality-specific information. By splitting the feature embedding for each modality, our model has access to both modality-shared features with predictive information present in both modalities, and modality-specific features that contain predictive information unique to each individual modality. This enables our model to effectively capture the distinct information provided by each modality, while simultaneously leveraging the shared information across modalities. Consequently, our approach has the potential to enhance generalization capabilities and achieve better performance on downstream tasks.
4.2 Domain Generalization Perspective
Theorem 2. Let  $\mathcal{X}$  be a space and  $\mathcal{H}$  be a class of hypotheses corresponding to this space. Let  $\mathbb{Q}$  and the collection  $\{\mathbb{P}_i\}_{i=1}^K$  be distributions over  $\mathcal{X}$  and let  $\{\varphi_i\}_{i=1}^K$  be a collection of non-negative coefficient with  $\sum_i \varphi_i = 1$ . Let  $\mathcal{O}$  be a set of distributions s.t.  $\forall \mathbb{S} \in \mathcal{O}$ , the following holds