Learning Unseen Modality Interaction
Authors: Yunhua Zhang
Hazel Doughty
Cees G. M. Snoek
Venue: NeurIPS 2023 poster (2023)
Keywords: Multimodal Learning
TL;DR: We pose the problem of unseen modality interaction and introduce a first solution that is able to infer predictions over modality combinations unseen during training.

Abstract: Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. In this paper, we challenge this modality-complete assumption for multimodal learning and instead strive for generalization to unseen modality combinations during inference. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved. This allows the information to be accumulated with a simple summation operation across available modalities. To reduce overfitting to less discriminative modality combinations during training, we further improve the model learning with pseudo-supervision indicating the reliability of a modality’s prediction. We demonstrate that our approach is effective for diverse tasks and modalities by evaluating it for multimodal video classification, robot state regression, and multimedia retrieval. Project website: https://xiaobai1217.github.io/Unseen-Modality-Interaction/.

Learning Unseen Modality Interaction
Yunhua Zhang, Hazel Doughty, Cees G.M. Snoek University of Amsterdam
Abstract
Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. In this paper, we challenge this
assumption for multimodal learning and instead strive for generalization to unseen modality combinations during inference. We pose the problem of unseen modality interaction and introduce a first solution. It exploits module that projects the multidimensional features of different modalities into a common space with rich information preserved. This allows the information to be accumulated with simple summation operation across available modalities. To reduce overfitting to less discriminative modality combinations during train- indicating
ing, we further improve the model learning with pseudo-supervisie the reliability of modality's prediction. We demonstrate that our approach is effective for diverse tasks and modalities by evaluating it for multimodal video classification, robot state regression, and multimedia retrieval. Project website: https://xiaabai1211 www.ggg..b....... io/Unseen Moodelity -ooeelly---yy--tterrttttt.oo.o.
Introduction
Real-world machine learning models must operate wide variety of platforms ranging from autonomous vehicles to wearable devices. This requires effective combination of data from the diverse suite of sensors available. Multimodal learning, which Iearns how combine multiple different data sources to make prediction, has therefore become a central problem in machine learning [3]. Learning correspondences between modalities
challenging problem as modalities differ in their input representations, learning dynamics, and discriminability for a target task. Recent multimodal learning methods [16, 28, 36, 32, 22] thus require the training data have all modalities available for every sample. We call this modality-complest learning. However, this constraint will not always hold in the real world as may be infeasible to collect sufficient training data which jointly measures data across all sensors. Similarly, sensors could be added or removed at any time. It is thus important to make multimodal learning methods able handle the mismatch between modality combinations at training and inference, which is the topic of this paper.
We propose the multimodal learning problem that able infer predictions over modality combina- tions unseen during training. Several previous works [45, 23, 26, 32, 22] have increased robustness to the data with missing modalities, i.e., modality-incompplet data. For example, Miech et al. [26] predict a weighting to combine unimodal predictions, while Shvetsova er al. [32] utilize cross-modal attention for multiple modality combinations to obtain the prediction. Albeit successful, these meth- ods still require portion of the data to have all modalities of interest to learn the relative importance of different modalities [26] or the cross-modal correspondences [32, 22, 45, 23]. Thus, they cannot handle unseen modality combinations at inference. Different from previous works, we propose to relieve the need for
training data and generalize to unseen modality combinations.
In this paper, our main contribution is to establish the problem of unseen modality interaction, where we strive to learn multimodal model from only modality-incompleee data that can handle unseen modality combinations at test time. The challenge is to effectively combine modalities without any training data of the modality combinations that will be seen in inference. We propose address this problem by effectively collecting information from available modalities while not explicitly
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

learning cross- moosl correspondences. Specifically, we project multidimensional features of different modalities into a common space, while keeping as much distinguishing information as possible. To reduce overfitting to unreliable modality combinations during training, we further improve the model learning by pseudo-supervision with dual branch prediction. To evaluate this new problem, we reorganize existing datasets and tasks for video classification, robotic state regression, and multimedia retrieval on a variety of modality combinations. Our approach enables effective unseen modality interaction and is more robust to scenarios where some modalities are corrupted by severe noise.
Related Work
Coddalty-Compleet Learning. The field of multimodal learning is extensive, c.g., [16, 14, 20, 32, 44, 5,36, 34, 38, 31, 33, 2, 30]. as different modalities can provide complementary information. A simple approach is to integrate information across modalities inputring concatenated unimodal features to linear layers [16, 44, 5J. Several works [14, 20, 34, 38, 31, 2, 30] instead capture the correlations between different modalities with cross-modal attention, enhancing the discriminative features for each modality. Some recent works [9, 28, 32, 36] utilize self-attention to gain both cross- mross-modal and intra-modal attention. For instance, Gabeur el al. [9] directly use the features of different modalities as the inputs a transformer. Nagrani et al. [28] make this more efficient by condensing the relevant information per modality with attention bottlenecks. Wang el al. [36] instead dynamically replace uninformative feature tokens among transformer layers with inter-modal features. Another line of research learns multimodal representations by utilizing the correspondences in multimodal data, e.g.. [43, 33]. Zhang et al. [43] use audio to adapt their visual model to distribution shift, while Swetha et al. [33] learn projection functions to transform unimodal features into joint embedding space with the semantic structures preserved. Although these works achieve impressive performance, they assume all data in both training and testing have the same modalities. We instead learn a model suitable for unseen modality interactions at test-time, which are not present in the training data.
Incomlee Learning, Some prior works [23, 42, 45, 22, 26, 32, 37] do not assume all data to be modality-complete and are more robust to missing modalities at test-time [45, 22, 42, 29,37] 37] or
in training [23, 26, 32]. To obtain test-time predictions with single modality, Recasens et al. [29] apply masks to the attention in the multimodal transformer. Ma et al. [22] improve robustness to missing modalities during testing by searching for the optimal multimodal learning strategies. Miech et al. [26] instead learn from modality-incomple unimodal predictions. Alternatively, Ma er al. [23], Zhao et al. [45] and Wang et al. [37] hallucinate the missing modalities according to the available ones. Shvetsova et al. [32] leave out the positional encodings and use combinatorial loss so that the transformer can accept any modality combination.
training data, by predicting a weighting to combine
While these works are robust to data with missing modalities, they still require some training data to be modality-complet, We remove this assumption and learn model able to make predictions at test-time for modality combinations not seen during training.
Modality-Agnostte Models. While the multimodal learning methods discussed above make use of unimodal encoders, several works [1, 15, 11] study modality-agnostic models, which possess the flexibility to take any modality as input. For instance, Jaegle et al. [15] propose the Perceiver model that can handle various modalities, by leveraging an asymmetric attention mechanism to distill inputs into a tight latent bottleneck. Akbari et al. [1] present a way for video, audio, and text modalities to share the same transformer model when trained by a sulf-super contrastive objective. Girdhar er vision model trained by unpaired image, video and single-view
al. [11] develop modality- 3D data as various visual modalities have a lot in common (e.g., shapes). Even though these works achieve state-of-the-art results on various tasks, they only extract unimodal representations [15, 1, 11]. We instead learn multimodal representations from
-inalitt--iooopltte data.
Federated Multimodal Learning. Similar to our task, federated learning [17, 4, 40, 19] also aims to learn from unpaired data, collected from different devices. Specifically, the goal is to collaboratively train a model with data from multiple devices without centralizing the data. Commonly this is solved by training models on local devices separately and aggregating these models with the server, which creates an updated global model. Most relevant to our work, is the work by Yang et al. [41]. This goes a step further and handles local devices with different sensors, although it assumes some modality- complete data are available for model learning. Our task has similar inspiration in that data may be collected from different devices. However, we assume that all training data are centrally available, but are podality-incompltee and we focus on learning for unseen modality interactions.
2

---

Training Set
Testing Set
Modality
Modality
Modality
Figure 1: Learning Unseen Modality Interactions. The connected squares represent a sample, with each color indicating a different modality. Our goal is to learn from a modality-incompleet training set make predictions for unseen modality combinations during inference. The modality combinations at inference could be either a union of or subsets of the modalities used in training.
3 Problem Definition
training set, where some modality combinations do not have data available, to make predictions on unseen modality combinations during inference (as depicted in Figure 1). We denote set of n samples with k modalities MM m2,. ... m by X-=(w*) M: .MM.. We aim to make predictions for a set of samples XM with available modalities M without having access to training data with the same complete set of modalities. Instead, the task is to learn from multiple sets of samples containing subsets the modalities M For ease of explanation, XM, with different sets of modalities, Le., M1 * M22. However, this can easily be extended to more than modality sets. Each modality set in training is subset of the modalities seen at inference, Le., MM, M2 C M", and together have all available modalities, iee, =M UM2 The objective is to make predictions using the available modalities during inference even though this modality combination never appears during model learning. Two main challenges exist. The first is integrating modality-specifff features from different feature spaces without any modality-complete training data containing all modalities of interest. The second is reducing the overfitting to the specific modality combinations from the modality-incompll training data.
Our goal is to learn from a
consider the scenario where we learn from two sets of samples XM, and
4 Method
The inputs to model are the raw signals of available modalities for a sample, from XM, U UM, during training and XM at test time. After training, our model is able to integrate information from different modalities for prediction even though the modality combinations
unseen in training.
Figure shows our method overview. Each modality m of a sample first encoded by unimodal encoder to generate the unimodal features F.. These features from different modalities will be in different spaces with different dimensions. To tackle the challenge integrating different modality combinations, we propose to project features of available modalities into a common space by our feature projection, while keeping as much distinguishing information as possible. To reduce overfitting to the modality combinations in training. we improve the supervision by introducing pscudo-labele which indicate the reliability of different modalities, to learn our dual branch prediction.
Feature Projection. We use pre-trained unimodal encoders Em, (-) to extract the features of each modality. These features are then flattened to tokens Fm= f F Fm. E RR xd with km tokens and dm feature dimension for modality m. The feature tokens of different modalities unaligned and different feature spaces with different lengths kn and dimensions dny- For example, audio tokens correspond to specific range of time and frequency, while video tokens represent a specific anttial-temporal video segment. Existing works with modality-complet training data concatenate features of different modalities as input to a multimodal transformer. However, without modality-complee data in training, a multimodal transformer cannot learn to find cross-modal correspondences. Thus, we need model that can handle a flexible combination of modalities to effectively learn from multiple sets of modality- incomlllte data X'M, and X'M, and make predictions for modality-compler data XM- Instead concatenation we use the summation operation, which provides a natural way to accumulate information from any number of available modalities, long as the modality-specifff features are in a common feature space. However, projecting modality- specific features into single vector in the common space in previous works [24, 27] loses a lot of
3

---

Feature Projection
Dual Branch Prediction
Lalign
Transformer Layers (-)
Modality m
Encoder E2()
supervised
CLS
Modality my
Encoder Ex())
l'pseudo
Lalign
CLS
Figure 2: Method Overview. Given a sample collected from multiple sensors, each modality is first encoded by a unimodal encoder. To generalize to unseen modality combinations, our feature projection projects the unimodal features into common space while keeping as much differentiating information as possible. We encourage feature alignment by applying Lalign to each modality in training. Then, we can obtain a multimodal representation by adding all modalities available for the input sample together. To reduce overfitting to the modality combinations training, we propose our dual branch prediction. Features are first divided into two groups corresponding to the two branches. One branch is supervised by groundtruth labels via Tsppervised and the other by pseudo-labels via /peeado During inference, the final prediction is the average of the two branches.
information, thus it cannot enable effective model learning. We instead introduce feature projection module for projecting multidimensional features into a common space. This approach allows the projected features of different modalities to have the same length and dimension with each feature element having the same semantic meaning over multiple modalities.
Our feature projection module takes the unimodal features F Fm fRm.,, Fm E km xdm as inputs and outputs the features in the common space =F Om F, E IR* " xd" where k' and dM indicate the length the feature tokens and their dimension which are shared across modalities. To this end, we first project different unimodal features of different lengths kim to fixed length ke via an attention matrix 0 Okmn Om E RR xk" xk"", which is obtained from Fm, through several transformer layers. The matrix O... normalized using the Softmax operation so that Ej=1, Am Ok J =1. O., indicates how to combine the input feature tokens to obtain each output feature token. Then, we obtain the reorganized features by F',
m FT m Om F',, E IR* xd.., and each feature token of F',, is a combination of input tokens from F F... To project the reorganized features F', m into the common space with the dimension d', we pass them through fully connected layer to get 10.F==FF f ... P Fm E R² xd", While each feature token in the resulting Fm has the same semantic meaning across available modalities, we can add them together obtain multimodal representation F= En F F e R xd" This allows for integrating the information from any m. combination of modalities, i.e., the modalities in M1,Mp or M
To ensure the output unimodal features F are in a common space, we apply a feature alignment loss Talignn to all modalities in Mi and M2. Specifically, we additionally learn series of tokens [uj,... 1. u, E Rd' and encourage the average feature f... - f;, of modality m to be close to one learnable token for each sample by minimizing their Euclidean distance. For example, in classification task, we learn one learnable token per class and encourage the average feature f, to be close to the corresponding token. For other tasks, simply encourage the average feature to be close to its nearest learnable token. Thus, for a sample in XM,, we formulate our feature alignment loss as:
Latign
Σ mEM,
[I Up
where Un,,, the selected learnable token of the current sample for modality m. Features from different modalities can be assigned to the same learnable token u,, when is the nearest token for both modalities. With the shared learnable tokens, we use the average feature to ensure features occupy common space while still allowing for vodality-specifie information.

---

Dual Branch Prediction. From the multimodal features. we could directly learn to make predictions using only groundtruth labels, as in existing multimodal learning methods [28, 9, 321. However, training data, as some modality
we find this strategy to he ineffective for modality-incomplee combinations are less discriminative for the target task. For example, for activity recognition in video, the RGB modality is often more discriminative than audio and optical flow. For the activity wash carrot, the sound of water indicates the verb could be wash or rinse, but the motion of hands cannot distinguish the object, which could be courgette, carrot, fruit, etc. In contrast, the RGB modality can make fine-grained distinctions via the appearance. By forcing the model to predict the activity wash carrot with audio and optical flow, it may overfit some unrelated features such as background noise. To mitigate the overfitting of less discriminative modalities, we propose to generate -speeviiiiiiiiiionn which reflects the discriminability of modality combination and incorporates it alongside the groundtruth labels. To allow separate predictions for the groundtruth and pseudo-supervision, adopt dual branches with each trained by one type of supervision. We reduce the number of parameters by letting the two branches share the model parameters but have different input features. Specifically, we divide the multimodal feature representation F into two groups, i.e., F R xd" and F pscudo E R 5 xd" form the inputs of the dual branches. Since the two groups of features are trained with different supervision, they learn to capture different semantic information from the input. To reduce the computation cost, we avoid using multiple forward passes and instead send the whole representation F into the transformer layers H(-)) where the attention operations are masked to obtain the prediction of each branch separately. Specifically, for each layer, the output attention becomes:
(okk VD
Isj exp
Jij =
Ee exp
where q and ie indicate the query and key, i and j indicate the ith and jth tokens in the query and key, and D indicates the dimension of the key. We set 1, if j is in the same group as i, otherwise 1y=0, As a result, we obtain the outputs of the dual branches by a single forward pass.
For the branch trained using groundtruth labels, we simply adopt the standard cross-entropy loss for classification, a triplet loss for retrieval and L.2 loss for regression. For the pseudo-labels used to train the second branch, we average the unimodal predictions across training epochs to get modality-speciife pseudo-labels. We obtain the pseudo-labels in such way as the correctness of predictions of each sample across training epochs can indicate its difficulty [35]. For example, if a sample is often misclassified, it may be too difficult for the available modalities to accurately classify this sample, i.e., there is ambiguous information for recognition. By using average predictions as pseudo-labels to provide a distribution over classes, the model is able to incorporate important distinguishing information while avoiding overfitting as it allows uncertainty between multiple classes. When Mi or M2 contain multiple modalities, we select the modality-speciff pseudo-label closest to the groundtruth annotation with respect to the cosine similarity. For the pseudo-labels, we adopt the KL divergence loss for classification tasks, while we keep the triplet loss for retrieval and the 1.2 loss for regression. During testing. we average predictions across branches to obtain the final prediction.
Overall Training Objective. For each sample in XM, and XM2 our training objective contains three parts: (1) feature alignment loss Lalign to ensure that the features of all modalities are projected into the common space with our feature reorganization module; (2) supervised loss Lsspervised for the branch using label annotations; (3)
loss Lpseudo for the other branch using
pseudo-labels. The overall training objective is:
L = ALalign + Laupervised + L.
(3)
where À and are hyperparameters to balance the weights of the loss terms. Since have multiple training sets for learning, we form batches using randomly selected samples from all training sets.
Evaluating Unseen Modality Interaction
Since unseen modality interaction is new problem, repurpose and reorganize existing multimodal datasets [6, 39, 18] which contain a variety of modalities, as summarized in Table 1.
Video Classification. For classification, we use action recognition in PPIC-Kitchens-100 100 [6] by Damen el al. This dataset contains 75,820 first-person video clips captured in 32 different environ- ments. There are three available modalities: RGB, optical flow, and audio. To use this dataset for
5

---

Task
Training Set
Training Set 2
Validation
Testing
RGB
Audio
RGB, Audio
RGB, Audio
Video Classification [6]
RGB
Flow
RGB, Flow
RGB, Flow
Audio
Flow
Audio, Flow
Audio, Flow
Robotic State Regression [18]
Image, Depth
Proprioception, Force
All
All
Multimedia Retrieval [39] RGB, Object, Scene, Face
RGB, Audio, OCR, Speech
All
All
Table I: Unseen modality interaction benchmarks for video classification, robotic state regression
and multimedia retrieval with the datasets and modalities per split.
unseen modality interaction, we first divide it into training, validation and testing so that each contains different environments. Training has 62,429 samples, validation 6,750 and testing 6,641. For the data in validation and testing, two modalities are used. The training set is further divided into two equal partitions, each with a different modality and different environments. This results in three settings with different modality combinations: RGB & audio, RGB & optical flow, and audio & optical flow.
Robot State Regression. For regression, we use the robotics dataset by Lee el al. [18]. The goal is to predict future end-effector positions. The dataset contains 100K states of a real robot collected in manipulation tasks, each with RGB, depth, force, and proprioception modalities. For unseen modality interactions, the training set has 105,554 states, and we divide it into two splits, each containing 52,577 samples. One split has image and depth modalities, while the other contains proprioception and force. Validation and testing contain 20,874 and 17,738 samples and use all four modalities.
Multimedia Retrieval. For retrieval, we use the video-text retrieval dataset MSR- -TTT [39] by Xu er al. This contains 10K YouTube videos with 200K text descriptions. The dataset has seven available modalities [20]: RGB video, audio, object, scene, face, OCR and speech. To evaluate unseen modality interactions, we use the original validation and testing sets with all available modalities but divide the training set into two splits. Since many of the modalities are noisy or missing for some samples, we use the RGB modality in both splits. The first training split contains the 1,702 samples with audio, OCR, speech and RGB. The second split contains the remaining 4,811 video samples with RGB, object, scene and face. The validation and test sets have 127 and 765 video samples respectively.
Evaluation Criteria. Following standard practice [6, 10, 20], we report the top-1 accuracy for classification and the mean absolute error (MAE) for regression. For retrieval, we summarize the results with mean rank averaged between video-to-texx and text-to-vides retrieval in the main paper, we also report full recall, median rank and mean rank metrics in the supplementary.
6 Results
Implementation Details. For video classification, we use Swin-T [21] pre-trained by Girdhar et al. [11] for the RGB unimodal encoder, ResNet [13, 8] for the optical flow encoder, and AST [12] for the audio encoder. For robotic state regression, we use the same feature extractors as in [18], ie., a
series of convolution layers for image and depth and fully connected layers for proprioception and force. For multimedia retrieval, we directly use the unimodal features provided by Liu et al. [20].
Both of our feature projection module and transformer layers for prediction consist of six transformer layers [7], each with 8 heads and hidden dimension d' of 256. The number of learnable tokens for the feature alignment loss is set to =3,66 EPIC-Kitchens, and n,,=125 on MSR-VTT and the robot dataset. The length of the feature tokens after feature projection is set as =5= on EPIC-Kitchens, and =1= on MSR-VTT and the robot dataset. We obtain the pseudo-labels by averaging the predictions from the last epochs of the pretrained unimodal encoders. For video classification, e-10. For robot state regression and multimedia retrieval, e=20. For the overall training objective (Eq. 3), we set a=3000 for video classification, while a=1 for regression and retrieval. X=0.001 used for all the three tasks. We use three NVIDIA RTX A6000 GPUs to train our model with batch size of 96 for video classification, while a single NVIDIA RTX 2080Ti GPU with batch size of 128 for robotic state regression and multimedia retrieval. Our method trained with 120 epochs on video classification with a learning rate of 10 reduced to 10-5 for the last 50 epochs. On robot state regression and multimedia retrieval, we train our method with 50 epochs and a learning rate of 10

---

RGB & Audio
RGB & Flow
Audio & Flow
Mean
Unimodal
RGB
18.2
18.2
18.2
Flow
8.6
8.6 11.8
8.6
Audio
10.9
11.4
Multimodal
Late fusion
20.2
20.7
13.2
18.0
Vanilla transformer
19.3
20.1
12.2
17.2
This paper: Unseen Modality Interaction
Feature projection
23.4
24.0
16.6
21.3
+ Dual branch prediction
25.7
26.2
19.3
23.7
Table 2: Ablation of model components for video classification. We report the top-1 accuracy
(%). Note that our proposed feature projection and dual branch prediction are added on top of vanilla multimodal transformer. Both components contribute to performance improvement for unseen modality combinations and the improvement over a vanilla transformer is considerable.
Tealign
Tppeudo
RGB & Audio
RGB & Flow
Audio & Flow
Mean
22.1
23.2
15.1
20.2
23.5
24.2
16.3
21.4
25.7
26.2
19.3
23.7
Table 3: Ablation of training objective using video classification. We report the top-1 accuracy (%).
Both loss terms improve the generalization unseen modality interactions.
Ablation of Model Components. We ablate the effect of our two proposed model components, i.e., feature projection and dual branch prediction, on our unseen modality interaction for video classification. We compare with: (i) Unimodal encoders which use the unimodal features of a single modality with linear layer for prediction, (ii) Lafe fusion which uses the average of the predictions from the unimodal encoders, (iii) Vanilla transformer which uses the multimodal fusion from [9]. To use this vanilla transformer with -onolity-incomplete training data, we use a series of per-modality learnable tokens in place of the missing modality. The results are shown in Table 2. Note that for each column, models only see unimodal data in training and consider the unseen combination of both modalities at inference. Combining unimodal features by either late fusion or vanilla transformer is beneficial for all modality combinations as different modalities provide complementary information. However, using the vanilla transformer is worse than simply averaging the predictions for our task of unseen modality interactions, since the vanilla transformer cannot learn the cross- modal correspondences among different modalities without modality-complee data. By projecting the features from different modalities into a common space before applying the vanilla transformer, model obtains 4.1% %% average improvement over the vanilla transformer. This demonstrates that combining modality-specific features in the common space improves the generalizability to unseen modality combinations. Adopting dual branch prediction in the transformer delivers further 2.4% improvement on average. This highlights the importance of using overfitting to modalities. Overall, model achieves top-1 accuracy of 23.7% on average with training data. When considering an oracle with all training data for all modalities available, the accuracy would be 26.1% 1.. Thus, model closes the gap with the oracle considerably.
to avoid
Ablation of Training Objective. Our training objective in Eq. consists of three terms. We ablate the effectiveness of adding the alignment loss Lalign and the eseudo-superviis loss Tupeudo to the task-spcc loss. We use our full model with both the feature projection and the dual branch prediction. When not using the pseudo-supervisee loss, branches. Table 3 shows the results. The +1 +1.2% improvement when adding our feature alignment loss indicates that this loss facilitates the projection of different modalities into a common space so that the model can better generalize towards unseen modality combinations. Our loss further improves the accuracy by +2.4% as it eliminates the overfitting to groundtruth labels, which can be harmful when a particular modality combination cannot give a reliable prediction.
the same supervised loss for both
Ablation of Number of Parameters. Since our feature projection module introduces additional parameters over vanilla multimodal transformer [9], we further demonstrate that the benefit of our model comes from our proposals, not the additional parameters used. Specifically, we reduce the
7

---

Test Modalities
Test Modalities
Depth, Proprioception, Force
RGB, Object, Speech, OCR RGB, Scene, Audio, OCR RGB, Scene, Speech RGB, Object, Audio
Image, Proprioception, Force Depth, Force Depth, Proprioception Image, Force
RGB, Speech
Image, Proprioception
RGB, Audio
0.20
Improvement in Mean Absolute Error
Improvement Mean Rank
Robotic State Regression
(b) Multimedia Retrieval
Figure 3: Benefit for Snttality-Inompplett Testing with robot state regression and multimedia retrieval. The modality combinations used in training are the same as Table 1. We show the improvement over vanilla multimodal transformer. Our model improves the robustness for all unseen modality combinations.
number of parameters from 14.4M to 7.2M, which is equal to the number used a vanilla multimodal transformer. With equal number of parameters, outperforming the vanilla counterpart by 4.2%. Thus, the improvement of our model is due to the effective method for learning unseen modality interactions.
model achieves 21.4% EPIC-Kitchens, still
Alternative Feature Projection. Previous works [24, 27] also consider projecting features of different modalities into common space. However, they only project the feature vector for the final prediction, instead of the multidimensional features, which contain much richer information. To further demonstrate the effectiveness of our feature projection, the multidimensional features, followed by a linear layer as in previous methods. Then, we obtain 19.0% EPIC-Kitchens with RGB, optical flow, and audio compared to 23.7% by our method. Thus, our feature projection module is more effective as it keeps rich information for accurate prediction.
replace it with an average pool on
Beyond Two Unimodal Subsets. Up to this point, we have learned model using two training sets with distinet modalitics. Here we confirm our model extends beyond this to more distinct training and modality sets. We test this with video classification, where partition the training set into three equal splits, with RGB, optical flow and audio respectively. We keep the testing and validation sets unchanged but all three modalities. We obtain 23.8% top-1 accuracy, surpassing the late fusion with 18.4 18.4% and the vanilla transformer with 17.5%. Thus, our model can solve the general unseen modality interaction task where there are than two training splits with distinct modalities.
Benefit for Modality-Incomplet Testing. While our model learns from experiments up to this point use modality-comple data in testing. Here we investigate whether our model is also beneficial to modality -inaaitt--i data at inference. We test this for robot state regression and multimedia retrieval these have many modalities. Figure shows the improvement of our approach over a vanilla transformer for six different modality combinations for each task. the
data,
Our model can handle any combination of input modalities at inference and improves vanilla transformer for all unseen combinations. We also note that our model is most effective when combining larger number of modalities as this provides more diverse information.
Benefit for Noisy Modalities. We further demonstrate the benefit of our model by testing it on samples with noisy modalities for unseen modality interactions with video classification. To this end, we apply Gaussian noise on one of the modalities for all samples across the test set. While late fusion delivers 1 11.2% accuracy, the vanilla multimodal transformer shows worse accuracy of 10.1% 1%, since the noise makes cross-modal correspondences harder to find. contrast, our model is more robust to noise and obtains 18.0% Our model's robustness comes from the feature projection module which selects the informative features during the projection and can thus reduce the noise.
Comparative Results. As there are prior methods for unseen modality interaction, we compare our approach with recent multimodal learning methods which assume all data to be modality-comple Gabeur ef al. [9], Nagrani et al. [28], Wang et al. [36], as well as methods which are robust to some modality-incommpete data: Recasens el [29] and Shvetsova el al. [32]. We use publicly
available implementations, except for the work by Recasens et al. [29], which re-implement

---

Video Classification
Robot State Regression
Multimedia Retrieval
Top-1 1 (%) f
MAE 1
MnR +
Late fusion
18.1
1.29
72.3
Modality Complete
Gabeur et al.
17.3
1.37
88.8
Nagrani et al.
17.5
1.35
86.2
Wang et al.
17.3
1.37
87.8
Modality Incomplete
Shvetsova et al.
18.0
1.25
72.3
Recasens el al.
18.5
1.22
72.2
Unseen Modality Interaction
This paper
23.7
66.2
Averaging the final predictions from unimodal encoders.
Based on our re-implementatton.
Table 4: Comparison with multimodal learning methods for video classification, robot state re- gression and multimedia retrieval tasks. While previous multimodal learning methods need modality- complete data to learn the cross cross-modal correspondences, our method gives more effective cross-modal fusion for unseen modality combinations.
ourselves. For all models, we use the same unimodal feature extractors as ours. To adapt modality- complete methods [9, 28, 36] to the unseen modality interaction, we learn series of learnable feature tokens for each modality and use these learnable tokens in place of the missing modality. Table shows the results for classification, regression, and retrieval tasks. We find that modality-complee methods perform worse than late fusion as they cannot learn the cross- pross-modal correspondences without modality-complete data, therefore cannot effectively make predictions for unseen modality combinations during inference. While the modality-incomplle approaches are robust to some missing modality data, they are not for unseen modality interactions where there is no modality-compler data in training. Thus, we outperform these approaches by effectively accumulating the information from any modality combinations. Overall, existing approaches not robust unseen modality combinations whereas our model obtains better generalization ability from training data. It does so for many different modalities including RGB, audio, depth, and force, and diverse set of tasks including video classification, robot state regression, and multimedia retrieval.
7 Discussion
Conclusion. We pose the problem of unseen modality interaction, where the modality combinations that appear during inference have not been seen during training. To facilitate further analysis and progress, repurpose and reorganize three existing multimodal datasets evaluate our approach on a wide variety of modalities, domains, and tasks. Experiments show that we can effectively make predictions for unseen modality interactions by accumulating different modalities projected common space and incorporating sscudo-supervisii representation. Our approach is suitable for classification, regression, and retrieval, can handle a wide variety of modality combinations, and is more robust to noisy modalities.
indicating the reliability of the multimodal
Linitations. Most of our experiments in the number of samples or number of modalities in different training subsets would be interesting to method comes with longer training time. We also explore only fully
equally sized training subsets. The effect of an imbalance
examine. While effective, supervised learning. Future work should tackle the extension to self-supervised learning framework.
Broader impact. Learning from data with missing modalities is important for machine learning, as collecting sufficient training data with all modalities of interest can be difficult. A multimodal model that can make predictions with unseen modality combinations increase robustness when applied to real-world applications. Learning unseen modality interaction orthogonal to other machine learning challenges like continual learning, federated learning, well as domain adaptation and generalization, whose combination opens up new lines of research.
Acknowledgement, This work is financially supported by the Inception Institute of Artificial Intelligence, the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.

---

References
[1] Н. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F. Chang, Y. Cui, and B. Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In NeurIPS, 2021.
[2] D. M. Argaw, J.-Y. Lee, M. Woodson, S. Kweon, and F. C. Heilbron. Long-range multimodal pretraining for movie understanding, ICCV, 2023.
[3] T. Baltrušaitis, C. Ahuja, and L.-P. Moreney, Multimodal machine learning: A survey and taxonomy. TPAMI, 41(2):233 443, 2018.
[4] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and
K. Seth. Practical secure aggregation for privacy-preservig machine learning. In ACM CCS, 2017.
[5] C. Chen, U. Jain, C. Schissler, S. V. A. Gari, Z. Al-Halah, V. K. Ithapu, P. Robinson, and K. Grauman. Soundspaces: Audio-visual navigation in 3d environments. In ECCV, 2020.
[6] D. Damen, H. Doughty, G. M. Farinella, A. Furnari, J. Ma, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-Kitchens-100 IJV,, 130:33-55, 2022.
[7] A. Dasovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, Heigold, Gelly, et al. image is worth 16x words: Transformers for image recognition at scale. In ICLR, 2021.
[8] C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In ICCV, 2019.
V. Gabeur, C. Sun, K. Alahari, and Schmid. Multi-modal transformer for video retrieval. In ECCV, 2020.
[10] A. Géron. Hands-on machine learning with scikit-leam and tensorflow: Concepts. Tools, and Techniques to build intelligent systems, 2017.
[11] R. Girdhar, M. Singh, N. Ravi, L. der Maaten, A. Joulin, and Misra. Omnivore: A single model for many visual modalities. CVPR, 2022.
[12] Y. Gong, Y.A. Chung, and J. Glass. Ast: Audio spectrogram transformer. In Interspeech, 2021.
[13] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[14] R. Hu, A. Singh, T. Darrell, and M. Rohrbach. Iterative answer prediction with pointer-augmented multimodal transformers for textvqa. In CVPR, 2020.
[15] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception with iterative attention. In ICMI, 2021.
[16] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In ICCV, 2019.
[17] J. Konečný, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon. Federated learning: Strategies for improving communication efficiency. In NIPS Workshop on Private Multi- Parti- Muchine Learning, 2016.
[18] M. A. Lee, Y. Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei. A. Garg, and J. Bohg. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. IEEE Transactions on Robotics, 36(3):582 596, 2020.
[19] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization in heterogeneous networks. MLSys, 2:429-4: 450, 2020.
[20] Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman. Use what you have: Video retrieval using representations from collaborative experts. In BMVC, 2019.
[21] Z. Liu, J. Ning, Y. Cao, Y. Wei. Z. Zhang. S. Lin, and H. Hu. Video swin transformer. In CVPR, 2022.
[22] M. Ma. J. Ren, L. Zhao. D. Testuggine, and X. Peng. Are multimodal transformers robust to missing modality? In CVPR, 2022.
[23] M. Ma. J. Ren, L. Zhao, S. Tulyakov, C. Wu, and X. Peng. Smil: Multimodal learning with severely missing modality. AAAI, 2021.
10

---

[24] S. Ma, Z. Zeng, D. McDuff, and Y. Song. Active contrastive learning of audio-visual video representations. In ICLR, 2021.
[25] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication--ffiieett learning of deep networks from decentralized data. In AISTATS, 2017.
[26] A. Miech, I. Laptev, and J. Sivic. Learning text-video embedding from incomplete and heterogeneous data. arXiv preprint a 800.02216, 2018.
[27] P. Morgado, I. Misra, and N. Vasconeelos. Robust audio-visual instance discrimination. In CVPR, 2021.
[28] A. Nagrani, S. Yang, A. Amab, A. Jansen, C. Schmid, and C. Sun. Attention bottlenecks for multimodal fusion. In NeurIPS, 2021.
[29] A. Recasens, J. Lin, J. Carreira, D. Jacgle, L. Wang, J.-b. Alayrac, P. Luc, A. Miech, L. Smaira, R. Hemsley, ct al. Zorro: the masked multimodal transformer. arXiv preprint 2.0Xix::300.00555, 2023.
[30] N. Sadoughi, X. Li, A. Vajpayee, D. Fan, B. Shuai, H. Cantos-Villalooss V. Bhat, and R. MV. Mega: Multimodal alignment aggregation and distillation for cinematic video segmentation. In ICCV, 2023.
[31] A. Senocak, T.-H. Oh, J. Kim, M.-H. Yang, and I. S. Kweon. Learning to localize sound source in visual scenes. In CVPR, 2018.
[32] N. Shvetsova, B. Chen, A. Rouditchenko, S. Thomas, B. Kingsbury, R. S. Feris, D. Harwath, J. Glass, and
H. Kuchne. Everything once-multi-modal fusion transformer for video retrieval. In CVPR, 2022.
[33] S. Swetha, M. N. Rizve, N. Shvetsova, H. Kuehne, and M. Shah. Preserving modality structure improves multi modal learning. In ICCV, 2023.
[34] Y. Tian, D. Li, and C. Xu. Unified multisensory perception: wcakly-supervised audio-visual video parsing. In ECCV, 2020.
[35] M. Toneva, A. Sordoni, R. T. d. Combes, A. Trischler, Y. Bengio, and G. J. Gordon. An empirical study of example forgetting during deep neural network leamnin.. In ICLR, 2019.
[36] Y. Wang, X. Chen, L. Cao, W. Huang, F. Sun, and Y. Wang. Multimodal token fusion for vision transformers. In CVPR, 2022.
[37] Y. Wang. Z. Cui, and Y. Li. Distribution-consisseet modal recovering for incomplete multimodal learning. In ICCV, 2023.
[38] Y. Wu and Y. Yang. Exploring heterogeneous clues for weakly- superv-s andio-visual video parsing, In CVPR, 2021.
[39] J. Xu. T. Mei, T. Yao, and Y. Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
[40] Q. Yang. Y. Liu, T. Chen, and Y. Tong. Federated machine learning: Concept and applications. TIST. 10(2):1-19, 2019.
[41] X. Yang, B. Xiong, Y. Huang, and C. Xu. Cross-modal federated human activity recognition via modality- agnostic and modality-specinn representation learning. In AAAI, 2022.
[42] J. Zeng. T. Liu, and J. Zhou. Tag- multimodal sentiment analysis under uncertain missing modalities. In SIGIR, 2022.
[43] Y. Zhang, H. Doughty, L. Shao, and C. G. M. Snoek. Audio-adaptive activity recognition across video domains. In CVPR, 2022.
[44] Y. Zhang, L. Shao, and C. G. M. Snoek. Repetitive activity counting by sight and sound. In CVPR, 2021.
[45] J. Zhao, R. L.,, and Q. Jin. Missing modality imagination network for emotion recognition with uncertain missing modalities. In ACI, 2021.


Reviews:
[{"review_id": "9bnecbD085", "reviewer": "Reviewer_ctBz", "summary": "This paper tackles the issue of unseen modality interaction, which challenges the conventional assumption of modality completion during training. The approach taken in this study involves formulating a training setting that accounts for modality incompleteness. Subsequently, the proposed method focuses on projecting multiple modalities into a shared feature space through an alignment objective and leveraging a pseudo-labeling strategy to alleviate the model's tendency to overfit to unreliable modalities. The experimental results on multiple benchmarks demonstrate the efficacy of the proposed framework.", "questions": "- In a similar context, is there any reason to split $\\hat{F}$ into exactly half? Since the contribution of $L_{pseudo}$ is extremely far from $L_{supervised}$, may be dividing $\\hat{F}$ into half is not the optimal solution.\n- In line 146, the paper mentions selecting the modality-specific pseudo-label that is closest to the ground truth annotation. How is the distance measured between the label and ground truth annotation when using a one-hot vector label?\n- Why the number of video classification accuracy in Table 2, 3 (23.7%) and Table 4 (23.8%)?", "limitations": "The paper sufficiently deals with the limitation of the paper in the conclusion.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses a practical problem, since incompleteness of modality occurs often in reality.\n- The authors conduct their experiments on the datasets with various modalities.", "weaknesses": "- There are some ambiguous parts in the manuscript.\n    - The authors mention that L_{align} ensures the projection of features from different modality spaces into the same feature space. However, based on Eq (1), it appears that the loss is computed as the summation of the difference between modality-specific features and modality-specific learnable tokens. In light of this, how can we ensure that the features from all modalities are projected into the same feature space?\n    - Regarding the dual branch prediction part, please provide further elaboration on the intuition behind the pseudo-labeling strategy to address less discriminative modalities. Can we consider less discriminative modalities as unreliable modalities? Even if a modality is less discriminative than others, it may still contain important information for the model. I am curious about the purpose of using pseudo labels to suppress such less discriminative modalities or encourage them.\n- The numbering of tables is incorrect in the experimental section. For instance, the table referenced in line 160 should be Table 1, not Table 6. Most of the tables are misnumbered.\n- Line 273 needs to be ended with ‘.’", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***We thank the reviewer for their time and effort and are glad that the reviewer found our paper addresses a practical problem and appreciate the experiments conducted on various modalities***. \n\n&nbsp;\n\n**Alignment Loss.** We apologize for the unclear text. The learnable tokens are not modality-specific but are shared across modalities. We make this clearer by modifying Eq.1 as: \n\n\\begin{aligned}\nL_{align} &= \\sum_{m \\in M_1} || \\bar{f_m} - u_{n_m} ||^2_2,\n\\end{aligned}\n\nwhere $u_{n_m}$ is the learnable token from $[u_1, …, u_{n_u}]$ selected for feature $\\bar{f}_m$. With the shared learnable tokens, the modality-specific features are encouraged to be projected into a common feature space. \n\n\n**Intuition behind pseudo-labeling strategy.** We agree with the reviewer that even if the modality is less discriminative, it can still contain important information. Our intuition for the pseudo-labeling strategy is inspired by the observation that a single modality alone often cannot provide enough information for accurate prediction. Take the example of conducting activity recognition with audio and video modalities, the audio is often less discriminative than video. For instance, the audio modality can be crucial in distinguishing that the activity is one of *swimming*, *surfing* or *water skiing*, but cannot make fine-grained distinctions. By forcing the model to predict the ground-truth activity *swimming*, it may overfit to some unrelated features such as background noise. By using average predictions as pseudo-labels to provide a distribution over classes, the model is able to incorporate the important distinguishing information while avoiding such overfitting as it allows uncertainty between multiple classes. We will make this motivation clearer in the final version. \n\n\n**Typos.** We thank the reviewer for highlighting the incorrect table indexes and the typo. We will correct them.  \n\n**Dividing $\\hat{F}$ in half.** We add experiments to test the model’s performance with different partition strategies on EPIC-Kitchens and report the results in the table below. \n\n| Ratio of $L_{pseudo}$ to $L_{supervised}$ | Top-1 (%) |\n|-------|---|\n| 30:70 |   22.9   |\n| 50:50 | **23.7** |\n| 70:30 |    22.5    |\n\n\nDividing the tokens into half delivers the best performance. While the pseudo-labels can refine the overconfident predictions trained by groundtruth labels, they may also be noisy for some difficult samples while the groundtruth labels provide the correct supervision. Thus, making the pseudo supervision and the groundtruth supervision equally important is beneficial. \n\n\n**Distance measurement for one-hot labels.** When using one-hot vector labels, the distance is measured by the cosine similarity between the modality-specific prediction and the groundtruth one-hot vector label. We will add this to the final version.\n\n**Number inconsistency.** We apologize for this mistake, the accuracy in Table 4 should be 23.7%. "}, {"review_id": "6uQawkUcf2", "reviewer": "Reviewer_adMX", "summary": "This paper introduces a method that can enhance the performance of multimodal models in scenarios involving unseen modality interactions.", "questions": "See Weaknesses", "limitations": "I don't think the experimental results of this paper convince me, so I tend to reject this paper. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1, The issue of \"unseen modality interaction\" explored in this paper is quite novel.\n\n2, This paper maps the features of different modalities into the same space and merges different modalities by simple addition. As the number of modalities increases, the number of parameters required for fusion does not significantly increase.", "weaknesses": "1, The performance of the baseline in this paper is so low that it makes the entire method proposed by the paper unconvincing. Specifically, although the method of this paper surpasses its own set baseline methods on EPIC-Kitchens, it only achieves an accuracy of 23.7%. Meanwhile, the most naive baseline in the paper [1] also has an accuracy of 23.7%. Therefore, it's hard for me to be convinced by the experimental conclusions of this paper.\n\n2, In Table 2, the authors compare the performance of different methods on different datasets, but why can't many of these methods outperform a simple late-fusion? Is it because there are significant differences in certain settings? If the author cannot clarify the situation here, I would consider the experimental comparison to be very unfair.\n\n3, Also in Table 2, the performance of DEQ Fusion in MM-IMDB is 61.52/53.38, while the performance of MMBTl[2] is 66.8/61.6 (Micro F1/Macro F1). Why choose not to report this method? It's a well-known classic approach that has already one hundred of citations.\n\n[1] What Makes Training Multi-modal Classification Networks Hard?\n\n[2] Supervised Multimodal Bitransformers for Classifying Images and Text", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***We thank the reviewer for their time and effort. We are glad to hear the reviewer found our proposed problem of “unseen modality interaction” novel and that the reviewer appreciates that  number of parameters required for fusion does not significantly increase as the number of modalities increases***.\n\n&nbsp;\n\n**Clarification on the experimental setups.** Our results on EPIC-Kitchens are not comparable to the results of Wang et al. [1] as we aim for unseen modality interaction where we do not have access to data with all modalities present. The A/V baseline from Table 8 in [1] instead assumes all samples have all modalities present. \n\nSpecifically, we modify the training set of EPIC-Kitchens by dividing the training data into two splits with each split having a different set of modalities. The A/V baseline is trained with the full training set. Training the naive A/V from [1] with our unseen modality training splits, we get 19.0%, much lower than our result (23.7%). We will add this baseline to the final version. \n\n**Late fusion vs. recent methods.** Our late-fusion baseline outperforms recent multi-modal fusion methods as late fusion simply averages the predictions from unimodal models. Previous multi-modal fusion models learn correspondences under the assumption that some or all of the data is modality-complete. When there is no modality-complete data for model learning, these models cannot learn cross-modal correspondences and instead overfit to the modality combinations seen during training. We realize the term ‘late-fusion’ can be ambiguous and will clarify it refers to averaging the final predictions from unimodal encoders in our results.\n\n**Comparison with MMBT.** Even though this comment seems to be for another paper, we further compare our paper with MMBT on EPIC-Kitchens, and report the performance in the table below:\n\n\n| | Top-1 (%) |\n|-------|---|\n| MMBT |   17.4  |\n| This paper| 23.7 |\n\n\nWhile MMBT is an effective method for modality-complete multimodal fusion, we conclude that our method is more effective in unseen modality interaction.\n\n[1] Wang et al. \"What makes training multi-modal classification networks hard?.\" In CVPR 2020.\n"}, {"review_id": "TOoVz1zWRr", "reviewer": "Reviewer_2qSh", "summary": "The paper is about multimodal learning and deals in particular with the mismatch between modality combinations at training and inference time. \nThe authors propose to project the multimodal features in a shared space, apply an alignment, and enforce the discriminative ability of the method with a dual branch prediction with full and pseudo supervision. \nThe method is evaluated on three scenarios, ablation studies and comparative analysis are reported.", "questions": "- I find it confusing having a separation of the considerations on the existing literature at the beginning and at the end of the manuscript, I would group them\n- On the same topic, the authors mention existing approaches and how their method is different. However (at least in the first part) only for some of them ([25, 30]) they explicitly mention limitations, what about the others?\n- As reported above: the statement \"Our approach enables effective unseen modality interaction and is more robust to scenarios where some modalities are corrupted by severe noise.\" would require a deeper investigation I think \n- What about the ability to generalize to new tasks/settings, different from the ones of the training set? What I mean is: if you have in training data collected with a certain combination of sensors from a certain platform to solve an action classification problem for instance, could the method be able to generalize to test data for the very same problem and possibly different sensors combinations acquired with a different platform?\n- The section on method lacks details in the description that are probably then reported at the beginning of the section on experimental analysis. For instance, it is said in Section 3 that the attention matrix Om is obtained through several transformer layers. Be sure all the details are properly reported. This is very important for the reproducibility of the results\n- I don’t understand when the alignment is applied, before or after the integration via the sum.\n- Eq. (2) appears without the appropriate context, I find\n- “… we propose to generate pseudo-supervision which reflects the discriminability a modality combination” Sentence to be rephased \n- Are the splittings in the training-validation-test used for the experiments provided with the datasets? If yes it should be stated, if not it should be justified\n- How many epochs for the training stage? Learning rate? Again, this is important to share as many implementation details as possible to favour the reproducibility\n- References to tables 7 and 8 should be 2 and 3 instead\n- “While our model learns from modality-incomplete data, previous experiments use modality-complete data in testing” It would be better to cite the methods\n- In the section “Benefit for Modality-Incomplete Testing” it is not quite clear how the experiment has been designed. Are the combinations in Figure 3 used at inference time? What was the training? I think some details are missing. The same considerations can be done for other sections, for instance “Benefit for Noisy Modalities“\n- In the experiments: “We use publicly available implementations where available, otherwise re-implementing ourselves.” Please be more specific: for which ones you used public implementations?", "limitations": "The authors explicitly mention some limitations of their method in the last section.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Overall, the paper is well-written, with some originality with respect to previous approaches. What is reported in the document is clear, figures are appropriate and descriptive of the concepts.", "weaknesses": "My main concerns on the paper are the following.\nOn the main goal: while I understand the benefit of learning from multiple modalities while being able to use a single modality at inference time,  I am not sure I fully understand the benefit of learning according to the set defined in Sect. 2, where at training time the method learns from a group of modalities which is a subset of what is seen at inference time. \nI expected to see in the empirical evaluation a quantitative justification that using at inference time a superset of the training modalities helps to improve the results. For instance, what if at inference time I use the combination used at training time, with no additional modalities? Is it really helpful to have these extra modalities at inference time? This is not clear to me.\nOn the objectives: in my opinion, some somehow strong statements of the authors on the abilities of their method are not appropriately justified with an empirical evaluation. For instance, the authors state that their method is more robust to scenarios where some modalities are corrupted by severe noise. More robust with respect to who and what? There is only one experiment tackling this challenge where only the proposed method is used. Moreover, the authors state that one of their challenges \"...is reducing the overfitting to the specific modality combinations from the modality-incomplete training data.\" Again, I think this would deserve more attention on the comments to the experimental analysis\nI am not sure I fully understood how the architecture I structured: I assume the architecture is accommodating a maximum number of input modalities while disregarding (or having learnable tokens for) the ones that are not present. From what I understand this requires the input modalities to be provided always in the same order. Is this interpretation correct?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***We thank the reviewer for their time and effort and are glad the reviewer found our paper well-written***.\n\n&nbsp;\n\n\n**Benefit of supersetting training modalities.** We show the benefit in Table 2 where we compare our model and multimodal baselines, which use all available modalities, to unimodal results which use the single modalities seen in training. For example, our method achieves 25.7% accuracy with RGB & audio as opposed to 18.2% using only RGB and 10.9% using only audio. We will highlight this. Furthermore, when using only RGB or audio, our method gives 19.6% and 12.3%, worse than using both modalities (25.7%). Thus, using a superset at inference is beneficial. \n\n**Benefit for Modality-Incomplete Testing.** We use the same model as in Table 4: trained on the modality-incomplete data splits as described in Section 4. In Fig 3 we test the model on different modality combinations at inference. For “Benefit for Noisy Modalities”, we also use the same trained model but add noise to some modalities during inference. We will add these clarifications.\n\n**Robustness to noise.**. In our paper, we show that when applying Gaussian noise $N(0,1)$ on one modality for all test samples in video classification, our method is more robust to the noise than late fusion or a vanilla multimodal transformer. We further compare our approach with more methods under different amounts of noise:\n|Model|N(0,0.5)|N(0,1)|N(0,2)|\n|-|-|-|-|\n|Late fusion|14.1|11.2 |10.0|\n|Gabeur et al.|13.2|10.1|9.3|\n|Nagrani et al.|16.4|15.5|14.4|\n|Wang et al.|14.9|12.3|10.8|\n|Shvetsova et al.|15.5|13.4|11.3|\n|Recasens et al.|15.3|13.1|12.0|\n|Ours|**20.5**|**18.0**|**16.2**|. \nWe conclude that our model achieves better results when modalities are corrupted by severe noise than these prior works and will add this table.\n\n**Reducing overfitting to specific modality combinations.** In Fig 3, we show our model improves robustness to various unseen modality combinations over a vanilla multimodal transformer. To further demonstrate our generalizability to different modality combinations and demonstrate that prior works indeed overfit to the combinations in training, we expand this experiment. Specifically, we compare our method on both seen and unseen modality combinations with previous fusion works and report the results in the tables below. Note for both mean rank (multimedia retrieval) and MAE (robot state regression), lower is better.\n|Model|Gabeur et al.|Nagrani et al.|Wang et al.|Shvetsova et al.|Recasens et al.|Ours|\n|-|-|-|-|-|-|-|\n|RGB, Audio, OCR, Speech|90.6|89.7|90.0|90.6|90.5|**79.4**|\n|RGB, Object, Scene, Face|89.1|88.8|88.9| 89.3|89.4|**80.3**|\n|RGB, Object, Speech, OCR|92.4|90.2|91.5|78.1|77.4|**70.2**|\n|RGB, Scene, Audio, OCR|91.0|89.9|90.7|75.3|74.2|**70.3**|\n|RGB, Scene, Speech|95.6|94.5|95.1|80.2|79.3|**74.3**|\n|RGB, Object, Audio|96.1|96.0|96.3|82.1|80.3|**76.9**|\n|RGB, Speech|98.3|98.0|98.8|84.6|83.0|**81.4**|\n|RGB, Audio|98.0|97.3|98.4|85.3|84.5|**79.8**|\nTable: Multimedia retrieval\n\n|Model|Gabeur et al.|Nagrani et al.|Wang et al.|Shvetsova et al.|Recasens et al.|Ours|\n|-|-|-|-|-|-|-|\n|Image, Depth|1.40|1.39|1.41|1.38|1.41|**1.29**|\n|Force, Proprioception|1.39|1.37|1.39|1.37|1.39|**1.27**|\n|Depth, Proprioception, Force|1.48|1.45|1.44|1.37|1.34|**1.19**|\n|Image, Proprioception, Force|1.47|1.43|1.45|1.32|1.30|**1.18**|\n|Depth, Force|1.77|1.72|1.79|1.62|1.58|**1.47**|\n|Depth, Proprioception|1.60|1.52|1.55|1.49|1.43|**1.38**|\n|Image, Force|1.71|1.64|1.70|1.58|1.52|**1.44**|\n|Image, Proprioception|1.54|1.50|1.52|1.38|1.35|**1.27**|\nTable: Robotic State Regression\n\nSince Gabeur et al., Nagrani et al. and Wang et al. assume modality-complete data, they obtain worse performance with unseen combinations than with seen combinations (top two rows), even with additional modalities. Thus we conclude these methods overfit to seen combinations. Since Shvetsova et al. and Recasens et al. aim to be robust to some modality-incomplete training data, they benefit from some modality combinations. However, these methods are not generalizable to all combinations. In contrast, our method is the most effective on all seen and unseen modality combinations. \n\n**Generalizability.** The tables above also show the generalizability of our method to different sensor combinations.\n\n**Input modality order.** We do not require input modalities to be provided in a set order since we project the features of each modality into a shared space before summation (L105-107).\n\n**Literature grouping.** We will move the related work to be section 2 to group the literature discussion.\n\n**Limitations of prior work.** The limitations in [30] also exist in [21,22,40] as these works assume modality-complete data is available like [30]. We will add clarification.\n\n**Method Details.** We will add more context to Eq. 2 and add other requested details to the method. We train our method with 120 epochs on video classification with an lr of $10^{-4}$, reduced to $10^{-5}$ for the last 50 epochs. On robot state regression and multimedia retrieval, we train with 50 epochs and an lr of $10^{-2}$. We will release the code provided in the supplementary on publication to ensure reproducibility.\n\n**Alignment.** We project the features of different modalities into the same space by the alignment loss before fusing them via a sum.\n\n**Dataset splits.** We use the same validation and test splits as provided with the datasets. To facilitate our research on unseen modality interaction, we divide the original training set of each dataset into multiple splits where each split contains different modalities. We will release our training set division.\n\n**Table numbering.** We will correct table indexes.\n\n**Modality-complete data in testing.** We mean we use modality-complete data at inference in Tables 1-3. We will make this clearer.\n\n**Implementations.** We re-implement Recassens et al. since their code is not available. We use released code for all others. We will clarify this.\n"}, {"review_id": "VvSKmFGhN1", "reviewer": "Reviewer_d7nB", "summary": "This work studies the problem of learning interactions of unseen modality combinations. Specifically, all training data is modality-incomplete, and the model must learn to perform inference on modality-complete data. The paper claims to be the first to study inference under such settings, and proposed two novel improvements to tackle this challenge: (1) feature projection layer that projects all encoded modalities into the same dimensionality and constrained by an alignment loss, and (2) a dual-branch prediction layer that predicts a pseudo-label in addition to the real label. The paper performed experiments on 3 datasets that contains a diverse set of modalities, domains and tasks (including classification, retrieval and regression). The paper included thorough ablation studies on their methods to justify all of their design choices, and they showed that under the their new setting, their method significantly outperforms previous modality-complete and modality-incomplete approaches. ", "questions": "(1) I think figure 1 can be a bit misleading, since in this paper, we focused on the setting where the test set is always the union of all modalities.\n\n(2) The dimension of F'm on line 96 and 102 are inconsistent. \n\n(3) For feature projection, is it correct that we simply do a linear combination to change the sequence dimension? When we change dm to d*, do we have a fully connected layer that maps each dm-dimensional vector to a d* dimensional vector individually, or does it map the entire k*x dm matrix to a k* x d* matrix?\n\n(4) On line 111, why do we average the features within each modality? It seems to me that the alignment loss is sort of like a vector-qualitization process, and it is difficult for me to understand why quantizing each modality separately (as opposed to, for example, quantizing each d-dimensional vector in the sequence, or the average of the vectors in the same corresponding locations across different modalities) helps features from different modalities occupy a common space. Wouldn't the current approach prevent the projected features from occupying a common space? For example, we could have modality 1 always close to the first few u in the dictionary, and modality 2 always close to the next few u, etc.\n\n(5) In equation 1, the notation of u_m is a bit confusing. I think it is supposed to mean the closest vector in [u1, u2,... ] to fm, but it could be confused with the mth vector in [u1,u2,...]\n\n(6) What exactly does \"average across training epochs\" mean? Since we need the pseudo-labels during training, do we just average the first-branch prediction on this data point from each previous epoch? How do we obtain this for the first epoch? Also, is it correct that the pseudo-labels are probability-distributions across all labels in the classification case? I also have a hard time understanding the intuition behind dual-branch prediction helping with overfitting problem. Perhaps a more clear and detailed description on how the pseudo-labels are obtained could make it more clear.\n\n(7) In line 167, since there are 3 modalities in this dataset, how exactly are they partitioned into the two training set partitions?", "limitations": "Limitation discussion is adequate.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposed a new setting: learning to infer from unseen combinations of modalities, and proposed a new method to tackle this challenge.\n\n2. The paper's experiments and evaluations are very comprehensive. The experiments involved 3 datasets that contains a very diverse set of different domains (kitchen videos, robotics, Youtube videos), modalities (14 total modalities), and task (classification, regression, retrieval), and the diversity shows that their method generalizes well. There is a comprehensive ablation study that justifies the design choice of each model component as well as loss function. Since this is a new setting, the authors re-implemented or re-ran several existing approaches on the new setting and showed that the new method outperforms all of them.", "weaknesses": "The presentation/clarity of the paper needs improvement, especially in some parts of the methods section. For example, the use of different variable names in the feature projection are inconsistent (e.g. F'm  in line 96 and line 102 have different dimensions); the notation of the alignment process is confusing, and the intuition behind the whole alignment process is unclear; and the description of how exactly the psudo-labels are obtained is very vague and unclear. See Questions section for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***We thank the reviewer for their time and effort. We are glad that the review appreciated the new setting of unseen modality interaction, the new method proposed for this and found  the experiments and evaluations comprehensive***. \n\n&nbsp;\n\n**Figure 1.** We will make Figure 1 clearer by presenting two scenarios where all the modalities are available and where subsets of the modalities are available during inference. While our experiments in Tables 1-3 do focus on the test set having the union of all modalities, in Figure 3 we show that our method is beneficial to subsets of  modalities. We will also expand the analysis in Figure 3 in the final version with the comparison to previous multimodal learning methods provided in the response to reviewer 2qSh. \n\n**Dimension of $F’_m$.** The reviewer is correct. We made a mistake here and F’m on line 96 should be modified to $\\hat{F}_m$. \n\n**Feature projection.** We have a fully connected layer that maps each dm-dimensional vector to a d dimensional vector individually. \n\n**Averaging the features within each modality.** We average the modality specific features for the alignment loss instead of aligning each individual feature vector as many feature vectors may be uninformative to the target problem. We demonstrate that averaging is better than aligning individual features with an experiment on EPIC-Kitchens. When encouraging each d-dimensional vector to be close to one of the learnable tokens, we obtain 20.7%. When using the individual features averaged across modalities, we get 19.8%. Both are worse than our 23.7%. Thus, our alignment strategy is more effective. \n\nThe reviewer is correct that it could be possible to have one modality always close to the first few u in the dictionary and another modality always close to other u in the dictionary. However, we observed that this didn’t happen and therefore didn’t find it necessary to discourage such cases in the alignment loss. Instead, we observe that each modality covers the majority of the tokens across the training samples, allowing the predictions to be diverse and satisfy the groundtruth supervision. \n\n**Notation $u_m$.** Yes, we mean the closest vector in [u1, u2, …] to $\\bar{f_m}$. We agree that the notation of $u_m$ is confusing and modify it as:\n$$L_{align} = \\sum_{m \\in M_1} ||\\bar{f_m} - u_{n_m}||^2_2,$$\nwhere $u_{n_m}$ is the learnable token from $[u_1, …, u_{n_u}]$ selected for feature $\\bar{f}_m$.  \n\n**Average across training epochs.** We obtain the pseudo-labels by averaging the predictions from the last e epochs of the pretrained unimodal encoders. For video classification e=10, for robot state regression e=20 and for multimedia retrieval e=20. We will add these details to the paper. Yes, the pseudo-labels are probability distributions across all labels in the classification case. \n\n**Intuition behind dual-branch prediction.** Our intuition for the pseudo-labeling strategy is inspired by the observation that a single modality alone often cannot provide enough information for accurate prediction. Take the example of conducting activity recognition with audio and video modalities, the audio is often less discriminative than video. For instance, the audio modality can be crucial in distinguishing that the activity is one of *swimming*, *surfing* or *water skiing*, but cannot make fine-grained distinctions. By forcing the model to predict the ground-truth activity *swimming*, it may overfit to some unrelated features such as background noise. By using average predictions as pseudo-labels to provide a distribution over classes, the model is able to incorporate the important distinguishing information while avoiding such overfitting as it allows uncertainty between multiple classes. We will make this clearer in the final version. \n\n**Training set partitions.** We have three settings on video classification with the EPIC-Kitchens dataset each using two of the three modalities. Therefore, we divide the training set into two splits with each containing only one modality. "}]

Author rebuttal:

Meta-review: The reviewers found the paper to be making a valuable contribution, to a large degree by exploring a practical issue that has received little attention so far, and recommend it for acceptance, trusting the authors to make the manuscript clarifications they promised in their rebuttals.
Decision: Accept (poster)