SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yunhua Zhang, Hazel Doughty, Cees G.M. Snoek
Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. In this paper, we challenge this modality-complete assumption for multimodal learning and instead strive for generalization to unseen modality combinations during inference. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved. This allows the information to be accumulated with a simple summation operation across available modalities. To reduce overfitting to less discriminative modality combinations during training, we further improve the model learning with pseudo-supervision indicating the reliability of a modality's prediction. We demonstrate that our approach is effective for diverse tasks and modalities by evaluating it for multimodal video classification, robot state regression, and multimedia retrieval. Project website: https://xiaobai1217.github.io/Unseen-Modality-Interaction/.
Real-world machine learning models must operate on a wide variety of platforms ranging from autonomous vehicles to wearable devices. This requires effective combination of data from the diverse suite of sensors available. Multimodal learning, which learns how to combine multiple different data sources to make a prediction, has therefore become a central problem in machine learning [3]. Learning correspondences between modalities is a challenging problem as modalities differ in their input representations, learning dynamics, and discriminability for a target task. Recent multimodal learning methods [16, 28, 36, 32, 22] thus require the training data to have all modalities available for every sample. We call this modality-complete learning. However, this constraint will not always hold in the real world as it may be infeasible to collect sufficient training data which jointly measures data across all sensors. Similarly, sensors could be added or removed at any time. It is thus important to make multimodal learning methods able to handle the mismatch between modality combinations at training and inference, which is the topic of this paper.
We propose the multimodal learning problem that is able to infer predictions over modality combinations unseen during training. Several previous works [45, 23, 26, 32, 22] have increased robustness to the data with missing modalities, i.e., modality-incomplete data. For example, Miech et al. [26] predict a weighting to combine unimodal predictions, while Shvetsova et al. [32] utilize cross-modal attention for multiple modality combinations to obtain the prediction. Albeit successful, these methods still require a portion of the data to have all modalities of interest to learn the relative importance of different modalities [26] or the cross-modal correspondences [32, 22, 45, 23]. Thus, they cannot handle unseen modality combinations at inference. Different from previous works, we propose to relieve the need for modality-complete training data and generalize to unseen modality combinations.
In this paper, our main contribution is to establish the problem of unseen modality interaction, where we strive to learn a multimodal model from only modality-incomplete data that can handle unseen modality combinations at test time. The challenge is to effectively combine modalities without any training data of the modality combinations that will be seen in inference. We propose to address this problem by effectively collecting information from available modalities while not explicitly
learning cross-modal correspondences. Specifically, we project multidimensional features of different modalities into a common space, while keeping as much distinguishing information as possible. To reduce overfitting to unreliable modality combinations during training, we further improve the model learning by pseudo-supervision with dual branch prediction. To evaluate this new problem, we reorganize existing datasets and tasks for video classification, robotic state regression, and multimedia retrieval on a variety of modality combinations. Our approach enables effective unseen modality interaction and is more robust to scenarios where some modalities are corrupted by severe noise.
Modality-Complete Learning. The field of multimodal learning is extensive, e.g., [16, 14, 20, 32, 44, 5, 36, 34, 38, 31, 33, 2, 30], as different modalities can provide complementary information. A simple approach is to integrate information across modalities inputting concatenated unimodal features to linear layers [16, 44, 5]. Several works [14, 20, 34, 38, 31, 2, 30] instead capture the correlations between different modalities with cross-modal attention, enhancing the discriminative features for each modality. Some recent works [9, 28, 32, 36] utilize self-attention to gain both cross-modal and intra-modal attention. For instance, Gabeur et al. [9] directly use the features of different modalities as the inputs to a transformer. Nagrani et al. [28] make this more efficient by condensing the relevant information per modality with attention bottlenecks. Wang et al. [36] instead dynamically replace uninformative feature tokens among transformer layers with inter-modal features. Another line of research learns multimodal representations by utilizing the correspondences in multimodal data, e.g., [43, 33]. Zhang et al. [43] use audio to adapt their visual model to distribution shift, while Swetha et al. [33] learn projection functions to transform unimodal features into a joint embedding space with the semantic structures preserved. Although these works achieve impressive performance, they assume all data in both training and testing have the same modalities. We instead learn a model suitable for unseen modality interactions at test-time, which are not present in the training data.
Modality-Incomplete Learning. Some prior works [23, 42, 45, 22, 26, 32, 37] do not assume all data to be modality-complete and are more robust to missing modalities at test-time [45, 22, 42, 29, 37] or in training [23, 26, 32]. To obtain test-time predictions with a single modality, Recasens et al. [29] apply masks to the attention in the multimodal transformer. Ma et al. [22] improve robustness to missing modalities during testing by searching for the optimal multimodal learning strategies. Miech et al. [26] instead learn from modality-incomplete training data, by predicting a weighting to combine unimodal predictions. Alternatively, Ma et al. [23], Zhao et al. [45] and Wang et al. [37] hallucinate the missing modalities according to the available ones. Shvetsova et al. [32] leave out the positional encodings and use a combinatorial loss so that the transformer can accept any modality combination. While these works are robust to data with missing modalities, they still require some training data to be modality-complete. We remove this assumption and learn a model able to make predictions at test-time for modality combinations not seen during training.
Modality-Agnostic Models. While the multimodal learning methods discussed above make use of unimodal encoders, several works [1, 15, 11] study modality-agnostic models, which possess the flexibility to take any modality as input. For instance, Jaegle et al. [15] propose the Perceiver model that can handle various modalities, by leveraging an asymmetric attention mechanism to distill inputs into a tight latent bottleneck. Akbari et al. [1] present a way for video, audio, and text modalities to share the same transformer model when trained by a self-supervised contrastive objective. Girdhar et al. [11] develop a modality-agnostic vision model trained by unpaired image, video and single-view 3D data as various visual modalities have a lot in common (e.g., shapes). Even though these works achieve state-of-the-art results on various tasks, they only extract unimodal representations [15, 1, 11]. We instead learn multimodal representations from modality-incomplete data.
Federated Multimodal Learning. Similar to our task, federated learning [17, 25, 4, 40, 19] also aims to learn from unpaired data, collected from different devices. Specifically, the goal is to collaboratively train a model with data from multiple devices without centralizing the data. Commonly this is solved by training models on local devices separately and aggregating these models with the server, which creates an updated global model. Most relevant to our work, is the work by Yang et al. [41]. This goes a step further and handles local devices with different sensors, although it assumes some modality-complete data are available for model learning. Our task has a similar inspiration in that data may be collected from different devices. However, we assume that all training data are centrally available, but are modality-incomplete and we focus on learning for unseen modality interactions.
Figure 1: Learning Unseen Modality Interactions. The connected squares represent a sample, with each color indicating a different modality. Our goal is to learn from a modality-incomplete training set to make predictions for unseen modality combinations during inference. The modality combinations at inference could be either a union of or subsets of the modalities used in training.
Our goal is to learn from a modality-incomplete training set, where some modality combinations do not have data available, to make predictions on unseen modality combinations during inference (as depicted in Figure 1). We denote a set of  $n$  samples with  $k$  modalities  $\mathcal{M} = \{m_1, m_2, \dots, m_k\}$  by  $\mathcal{X}_{\mathcal{M}} = \{x_{\mathcal{M}}^1, \dots, x_{\mathcal{M}}^n\}$ . We aim to make predictions for a set of samples  $\mathcal{X}_{\mathcal{M}^*}$  with available modalities  $\mathcal{M}^*$ , without having access to training data with the same complete set of modalities. Instead, the task is to learn from multiple sets of samples containing subsets of the modalities  $\mathcal{M}^*$ . For ease of explanation, we consider the scenario where we learn from two sets of samples  $\mathcal{X}_{\mathcal{M}_1}$  and  $\mathcal{X}_{\mathcal{M}_2}$  with different sets of modalities, i.e.,  $\mathcal{M}_1 \neq \mathcal{M}_2$ . However, this can easily be extended to more than two modality sets. Each modality set in training is a subset of the modalities seen at inference, i.e.,  $\mathcal{M}_1, \mathcal{M}_2 \subset \mathcal{M}^*$ , and together have all available modalities, i.e.,  $\mathcal{M}^* = \mathcal{M}_1 \cup \mathcal{M}_2$ . The objective is to make predictions using the available modalities during inference even though this modality combination never appears during model learning. Two main challenges exist. The first is integrating modality-specific features from different feature spaces without any modality-complete training data containing all modalities of interest. The second is reducing the overfitting to the specific modality combinations from the modality-incomplete training data.
The inputs to our model are the raw signals of available modalities for a sample, from  $\mathcal{X}_{\mathcal{M}_1} \cup \mathcal{X}_{\mathcal{M}_2}$  during training and  $\mathcal{X}_{\mathcal{M}^*}$  at test time. After training, our model is able to integrate information from different modalities for prediction even though the modality combinations are unseen in training.
Figure 2 shows our method overview. Each modality  $m$  of a sample is first encoded by a unimodal encoder to generate the unimodal features  $\mathbf{F}_m$ . These features from different modalities will be in different spaces with different dimensions. To tackle the challenge of integrating different modality combinations, we propose to project features of available modalities into a common space by our feature projection, while keeping as much distinguishing information as possible. To reduce overfitting to the modality combinations in training, we improve the supervision by introducing pseudo-labels, which indicate the reliability of different modalities, to learn our dual branch prediction.
Feature Projection. We use pre-trained unimodal encoders  $\mathcal{E}_m(\cdot)$  to extract the features of each modality. These features are then flattened to tokens  $\mathbf{F}_m = [\mathbf{f}_1,\dots ,\mathbf{f}_{k_m}],\mathbf{F}_m\in \mathbb{R}^{k_m\times d_m}$  with  $k_{m}$  tokens and  $d_{m}$  feature dimension for modality  $m$ . The feature tokens of different modalities are unaligned and in different feature spaces with different lengths  $k_{m}$  and dimensions  $d_{m}$ . For example, audio tokens correspond to a specific range of time and frequency, while video tokens represent a specific spatial-temporal video segment. Existing works with modality-complete training data concatenate features of different modalities as input to a multimodal transformer. However, without modality-complete data in training, a multimodal transformer cannot learn to find cross-modal correspondences. Thus, we need a model that can handle a flexible combination of modalities to effectively learn from multiple sets of modality-incomplete data  $\mathcal{X}_{\mathcal{M}_1}$  and  $\mathcal{X}_{\mathcal{M}_2}$  and make predictions for modality-complete data  $\mathcal{X}_{\mathcal{M}^*}$ . Instead of concatenation we use the summation operation, which provides a natural way to accumulate information from any number of available modalities, as long as the modality-specific features are in a common feature space. However, projecting modality-specific features into a single vector in the common space as in previous works [24, 27] loses a lot of
Figure 2: Method Overview. Given a sample collected from multiple sensors, each modality is first encoded by a unimodal encoder. To generalize to unseen modality combinations, our feature projection projects the unimodal features into a common space while keeping as much differentiating information as possible. We encourage feature alignment by applying  $L_{\mathrm{align}}$  to each modality in training. Then, we can obtain a multimodal representation by adding all modalities available for the input sample together. To reduce overfitting to the modality combinations in training, we propose our dual branch prediction. Features are first divided into two groups corresponding to the two branches. One branch is supervised by groundtruth labels via  $L_{\mathrm{supervised}}$  and the other by pseudo-labels via  $L_{\mathrm{pseudo}}$ . During inference, the final prediction is the average of the two branches.
information, thus it cannot enable effective model learning. We instead introduce a feature projection module for projecting multidimensional features into a common space. This approach allows the projected features of different modalities to have the same length and dimension with each feature element having the same semantic meaning over multiple modalities.
Our feature projection module takes the unimodal features  $\mathbf{F}_m = [\mathbf{f}_1, \dots, \mathbf{f}_{k_m}]$ ,  $\mathbf{F}_m \in \mathbb{R}^{k_m \times d_m}$  as inputs and outputs the features in the common space  $\hat{\mathbf{F}}_m = \mathbf{F}_m^{\mathrm{T}}\mathbf{O}_m$ ,  $\hat{\mathbf{F}}_m \in \mathbb{R}^{k^* \times d^*}$ , where  $k^*$  and  $d^*$  indicate the length of the feature tokens and their dimension which are shared across modalities. To this end, we first project different unimodal features of different lengths  $k_m$  to a fixed length  $k^*$  via an attention matrix  $\mathbf{O}_m = [\mathbf{o}_1, \dots, \mathbf{o}_{k_m}]$ ,  $\mathbf{O}_m \in \mathbb{R}^{k_m \times k^*}$ , which is obtained from  $\mathbf{F}_m$  through several transformer layers. The matrix  $\mathbf{O}_m$  is normalized using the Softmax operation so that  $\sum_{j=1,\dots,k_m} o_{k_m,j} = 1$ .  $\mathbf{O}_m$  indicates how to combine the input feature tokens to obtain each output feature token. Then, we obtain the reorganized features by  $\mathbf{F}_m' = \mathbf{F}_m^{\mathrm{T}}\mathbf{O}_m$ ,  $\mathbf{F}_m' \in \mathbb{R}^{k^* \times d_m}$ , and each feature token of  $\mathbf{F}_m'$  is a combination of input tokens from  $\mathbf{F}_m$ . To project the reorganized features  $\mathbf{F}_m'$  into the common space with the dimension  $d^*$ , we pass them through a fully connected layer to get  $\hat{\mathbf{F}}_m = [\hat{\mathbf{f}}_m^1, \dots, \hat{\mathbf{f}}_m^{k^*}]$ ,  $\hat{\mathbf{F}}_m \in \mathbb{R}^{k^* \times d^*}$ . While each feature token in the resulting  $\hat{\mathbf{F}}_m$  has the same semantic meaning across available modalities, we can add them together to obtain a multimodal representation  $\hat{\mathbf{F}} = \sum_{m} \hat{\mathbf{F}}_m$ ,  $\hat{\mathbf{F}} \in \mathbb{R}^{k^* \times d^*}$ . This allows for integrating the information from any combination of modalities, i.e., the modalities in  $\mathcal{M}_1, \mathcal{M}_2$  or  $\mathcal{M}^*$ .
To ensure the output unimodal features  $\hat{\mathbf{F}}_m$  are in a common space, we apply a feature alignment loss  $L_{\mathrm{align}}$ , to all modalities in  $\mathcal{M}_1$  and  $\mathcal{M}_2$ . Specifically, we additionally learn a series of tokens  $[\mathbf{u}_1,\dots ,\mathbf{u}_{n_u}]$ ,  $\mathbf{u}_i\in \mathbb{R}^{d^*}$  and encourage the average feature  $\bar{\mathbf{f}}_m = \sum_i\hat{\mathbf{f}}_m^i$  of modality  $m$  to be close to one learnable token for each sample by minimizing their Euclidean distance. For example, in a classification task, we learn one learnable token per class and encourage the average feature  $\hat{\mathbf{f}}_m$  to be close to the corresponding token. For other tasks, we simply encourage the average feature to be close to its nearest learnable token. Thus, for a sample in  $\mathcal{X}_{\mathcal{M}_1}$ , we formulate our feature alignment loss as:
$$
L _ {\text {a l i g n}} = \sum_ {m \in \mathcal {M} _ {1}} \| \mathbf {f} _ {m} - \mathbf {u} _ {n _ {m}} \| _ {2} ^ {2}, \tag {1}
$$
where  $\mathbf{u}_{n_m}$  is the selected learnable token of the current sample for modality  $m$ . Features from different modalities can be assigned to the same learnable token  $\mathbf{u}_t$ , when  $\mathbf{u}_t$  is the nearest token for both modalities. With the shared learnable tokens, we use the average feature to ensure features occupy a common space while still allowing for modality-specific information.
Dual Branch Prediction. From the multimodal features, we could directly learn to make predictions using only groundtruth labels, as in existing multimodal learning methods [28, 9, 32]. However, we find this strategy to be ineffective for modality-incomplete training data, as some modality combinations are less discriminative for the target task. For example, for activity recognition in video, the RGB modality is often more discriminative than audio and optical flow. For the activity wash carrot, the sound of water indicates the verb could be wash or rinse, but the motion of hands cannot distinguish the object, which could be courgette, carrot, fruit, etc. In contrast, the RGB modality can make fine-grained distinctions via the appearance. By forcing the model to predict the activity wash carrot with audio and optical flow, it may overfit to some unrelated features such as background noise. To mitigate the overfitting of less discriminative modalities, we propose to generate pseudo-supervision which reflects the discriminability of a modality combination and incorporates it alongside the groundtruth labels. To allow separate predictions for the groundtruth and pseudo-supervision, we adopt dual branches with each trained by one type of supervision. We reduce the number of parameters by letting the two branches share the model parameters but have different input features. Specifically, we divide the multimodal feature representation  $\hat{\mathbf{F}}$  into two groups, i.e.,  $\hat{\mathbf{F}}_{\mathrm{gt}} \in \mathbb{R}^{\frac{k^{*}}{2} \times d^{*}}$  and  $\hat{\mathbf{F}}_{\mathrm{pseudo}} \in \mathbb{R}^{\frac{k^{*}}{2} \times d^{*}}$ , to form the inputs of the dual branches. Since the two groups of features are trained with different supervision, they learn to capture different semantic information from the input. To reduce the computation cost, we avoid using multiple forward passes and instead send the whole representation  $\hat{\mathbf{F}}$  into the transformer layers  $\mathcal{H}(\cdot)$  where the attention operations are masked to obtain the prediction of each branch separately. Specifically, for each layer, the output attention becomes:
$$
a _ {i j} = \frac {I _ {i j} \exp \left(\frac {q _ {i} ^ {\mathrm {T}} k _ {j}}{\sqrt {D}}\right)}{\sum_ {\{j ^ {\prime} , I _ {i j ^ {\prime}} = 1 \}} \exp \left(\frac {q _ {i} ^ {\mathrm {T}} k _ {j ^ {\prime}}}{\sqrt {D}}\right)}, \tag {2}
$$
where  $q$  and  $k$  indicate the query and key,  $i$  and  $j$  indicate the  $i$ th and  $j$ th tokens in the query and key, and  $D$  indicates the dimension of the key. We set  $I_{ij} = 1$  if  $j$  is in the same group as  $i$ , otherwise  $I_{ij} = 0$ . As a result, we obtain the outputs of the dual branches by a single forward pass.
For the branch trained using groundtruth labels, we simply adopt the standard cross-entropy loss for classification, a triplet loss for retrieval and an L2 loss for regression. For the pseudo-labels used to train the second branch, we average the unimodal predictions across training epochs to get modality-specific pseudo-labels. We obtain the pseudo-labels in such a way as the correctness of predictions of each sample across training epochs can indicate its difficulty [35]. For example, if a sample is often misclassified, it may be too difficult for the available modalities to accurately classify this sample, i.e., there is ambiguous information for recognition. By using average predictions as pseudo-labels to provide a distribution over classes, the model is able to incorporate important distinguishing information while avoiding overfitting as it allows uncertainty between multiple classes. When  $\mathcal{M}_1$  or  $\mathcal{M}_2$  contain multiple modalities, we select the modality-specific pseudo-label closest to the groundtruth annotation with respect to the cosine similarity. For the pseudo-labels, we adopt the KL divergence loss for classification tasks, while we keep the triplet loss for retrieval and the L2 loss for regression. During testing, we average predictions across branches to obtain the final prediction.
Overall Training Objective. For each sample in  $\mathcal{X}_{\mathcal{M}_1}$  and  $\mathcal{X}_{\mathcal{M}_2}$ , our training objective contains three parts: (1) feature alignment loss  $L_{\mathrm{align}}$  to ensure that the features of all modalities are projected into the common space with our feature reorganization module; (2) supervised loss  $L_{\mathrm{supervised}}$  for the branch using label annotations; (3) pseudo-supervised loss  $L_{\mathrm{pseudo}}$  for the other branch using pseudo-labels. The overall training objective is:
$$
L = \lambda L _ {\text {a l i g n}} + L _ {\text {s u p e r v i s e d}} + \alpha L _ {\text {p s e u d o}}, \tag {3}
$$
where  $\lambda$  and  $\alpha$  are hyperparameters to balance the weights of the loss terms. Since we have multiple training sets for learning, we form batches using randomly selected samples from all training sets.
5 Evaluating Unseen Modality Interaction
Since unseen modality interaction is a new problem, we repurpose and reorganize existing multimodal datasets [6, 39, 18] which contain a variety of modalities, as summarized in Table 1.
Video Classification. For classification, we use action recognition in EPIC-Kitchens-100 [6] by Damen et al. This dataset contains 75,820 first-person video clips captured in 32 different environments. There are three available modalities: RGB, optical flow, and audio. To use this dataset for
Task	Training Set 1	Training Set 2	Validation	Testing
Video Classification [6]	RGB	Audio	RGB, Audio	RGB, Audio
RGB	Flow	RGB, Flow	RGB, Flow
Audio	Flow	Audio, Flow	Audio, Flow
Robotic State Regression [18]	Image, Depth	Proprioception, Force	All	All
Multimedia Retrieval [39]	RGB, Object, Scene, Face	RGB, Audio, OCR, Speech	All	All