SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Unseen modality interaction benchmarks for video classification, robotic state regression and multimedia retrieval with the datasets and modalities per split.
unseen modality interaction, we first divide it into training, validation and testing so that each contains different environments. Training has 62,429 samples, validation 6,750 and testing 6,641. For the data in validation and testing, two modalities are used. The training set is further divided into two equal partitions, each with a different modality and different environments. This results in three settings with different modality combinations: RGB & audio, RGB & optical flow, and audio & optical flow.
Robot State Regression. For regression, we use the robotics dataset by Lee et al. [18]. The goal is to predict future end-effector positions. The dataset contains 100K states of a real robot collected in manipulation tasks, each with RGB, depth, force, and proprioception modalities. For unseen modality interactions, the training set has 105,554 states, and we divide it into two splits, each containing 52,577 samples. One split has image and depth modalities, while the other contains proprioception and force. Validation and testing contain 20,874 and 17,738 samples and use all four modalities.
Multimedia Retrieval. For retrieval, we use the video-text retrieval dataset MSR-VTT [39] by Xu et al. This contains 10K YouTube videos with 200K text descriptions. The dataset has seven available modalities [20]: RGB video, audio, object, scene, face, OCR and speech. To evaluate unseen modality interactions, we use the original validation and testing sets with all available modalities but divide the training set into two splits. Since many of the modalities are noisy or missing for some samples, we use the RGB modality in both splits. The first training split contains the 1,702 samples with audio, OCR, speech and RGB. The second split contains the remaining 4,811 video samples with RGB, object, scene and face. The validation and test sets have 127 and 765 video samples respectively.
Evaluation Criteria. Following standard practice [6, 10, 20], we report the top-1 accuracy for classification and the mean absolute error (MAE) for regression. For retrieval, we summarize the results with mean rank averaged between video-to-text and text-to-video retrieval in the main paper, we also report full recall, median rank and mean rank metrics in the supplementary.
Implementation Details. For video classification, we use Swin-T [21] pre-trained by Girdhar et al. [11] for the RGB unimodal encoder, ResNet [13, 8] for the optical flow encoder, and AST [12] for the audio encoder. For robotic state regression, we use the same feature extractors as in [18], i.e., a series of convolution layers for image and depth and fully connected layers for proprioception and force. For multimedia retrieval, we directly use the unimodal features provided by Liu et al. [20].
Both of our feature projection module and transformer layers for prediction consist of six transformer layers [7], each with 8 heads and hidden dimension  $d^{*}$  of 256. The number of learnable tokens for the feature alignment loss is set to  $n_u = 3806$  on EPIC-Kitchens, and  $n_u = 128$  on MSR-VTT and the robot dataset. The length of the feature tokens after feature projection is set as  $k^{*} = 512$  on EPIC-Kitchens, and  $k^{*} = 16$  on MSR-VTT and the robot dataset. We obtain the pseudo-labels by averaging the predictions from the last  $e$  epochs of the pretrained unimodal encoders. For video classification,  $e = 10$ . For robot state regression and multimedia retrieval,  $e = 20$ . For the overall training objective (Eq. 3), we set  $\alpha = 3000$  for video classification, while  $\alpha = 1$  for regression and retrieval.  $\lambda = 0.001$  is used for all the three tasks. We use three NVIDIA RTX A6000 GPUs to train our model with a batch size of 96 for video classification, while a single NVIDIA RTX 2080Ti GPU with a batch size of 128 for robotic state regression and multimedia retrieval. Our method is trained with 120 epochs on video classification with a learning rate of  $10^{-4}$ , reduced to  $10^{-5}$  for the last 50 epochs. On robot state regression and multimedia retrieval, we train our method with 50 epochs and a learning rate of  $10^{-2}$ .
RGB & Audio	RGB & Flow	Audio & Flow	Mean
Unimodal
RGB	18.2	18.2	-	18.2
Flow	-	8.6	8.6	8.6
Audio	10.9	-	11.8	11.4
Multimodal
Late fusion	20.2	20.7	13.2	18.0
Vanilla transformer	19.3	20.1	12.2	17.2
This paper: Unseen Modality Interaction
Feature projection	23.4	24.0	16.6	21.3
+ Dual branch prediction	25.7	26.2	19.3	23.7
Table 2: Ablation of model components for video classification. We report the top-1 accuracy  $(\%)$ . Note that our proposed feature projection and dual branch prediction are added on top of a vanilla multimodal transformer. Both components contribute to performance improvement for unseen modality combinations and the improvement over a vanilla transformer is considerable.
L align	L pseudo	RGB & Audio	RGB & Flow	Audio & Flow	Mean
22.1	23.2	15.1	20.2
✓	23.5	24.2	16.3	21.4
✓	✓	25.7	26.2	19.3	23.7
Table 3: Ablation of training objective using video classification. We report the top-1 accuracy  $(\%)$ . Both loss terms improve the generalization to unseen modality interactions.
Ablation of Model Components. We ablate the effect of our two proposed model components, i.e., feature projection and dual branch prediction, on our unseen modality interaction for video classification. We compare with: (i) Unimodal encoders which use the unimodal features of a single modality with a linear layer for prediction, (ii) Late fusion which uses the average of the predictions from the unimodal encoders, (iii) Vanilla transformer which uses the multimodal fusion from [9]. To use this vanilla transformer with modality-incomplete training data, we use a series of per-modality learnable tokens in place of the missing modality. The results are shown in Table 2. Note that for each column, models only see unimodal data in training and we consider the unseen combination of both modalities at inference. Combining unimodal features by either late fusion or a vanilla transformer is beneficial for all modality combinations as different modalities provide complementary information. However, using the vanilla transformer is worse than simply averaging the predictions for our task of unseen modality interactions, since the vanilla transformer cannot learn the cross-modal correspondences among different modalities without modality-complete data. By projecting the features from different modalities into a common space before applying the vanilla transformer, our model obtains a  $4.1\%$  average improvement over the vanilla transformer. This demonstrates that combining modality-specific features in the common space improves the generalizability to unseen modality combinations. Adopting dual branch prediction in the transformer delivers a further  $2.4\%$  improvement on average. This highlights the importance of using pseudo-supervision to avoid overfitting to modalities. Overall, our model achieves a top-1 accuracy of  $23.7\%$  on average with modality-incomplete training data. When considering an oracle with all training data for all modalities available, the accuracy would be  $26.1\%$ . Thus, our model closes the gap with the oracle considerably.
Ablation of Training Objective. Our training objective in Eq. 3 consists of three terms. We ablate the effectiveness of adding the alignment loss  $L_{\mathrm{align}}$  and the pseudo-supervised loss  $L_{\mathrm{pseudo}}$  to the task-specific loss. We use our full model with both the feature projection and the dual branch prediction. When not using the pseudo-supervised loss, we use the same supervised loss for both branches. Table 3 shows the results. The  $+1.2\%$  improvement when adding our feature alignment loss indicates that this loss facilitates the projection of different modalities into a common space so that the model can better generalize towards unseen modality combinations. Our pseudo-supervised loss further improves the accuracy by  $+2.4\%$  as it eliminates the overfitting to groundtruth labels, which can be harmful when a particular modality combination cannot give a reliable prediction.
Ablation of Number of Parameters. Since our feature projection module introduces additional parameters over a vanilla multimodal transformer [9], we further demonstrate that the benefit of our model comes from our proposals, not the additional parameters used. Specifically, we reduce the
(a) Robotic State Regression
(b) Multimedia Retrieval
Figure 3: Benefit for Modality-Incomplete Testing with robot state regression and multimedia retrieval. The modality combinations used in training are the same as Table 1. We show the improvement over a vanilla multimodal transformer. Our model improves the robustness for all unseen modality combinations.
number of parameters from 14.4M to 7.2M, which is equal to the number used in a vanilla multimodal transformer. With an equal number of parameters, our model achieves  $21.4\%$  on EPIC-Kitchens, still outperforming the vanilla counterpart by  $4.2\%$ . Thus, the improvement of our model is due to the effective method for learning unseen modality interactions.
Alternative Feature Projection. Previous works [24, 27] also consider projecting features of different modalities into a common space. However, they only project the feature vector for the final prediction, instead of the multidimensional features, which contain much richer information. To further demonstrate the effectiveness of our feature projection, we replace it with an average pool on the multidimensional features, followed by a linear layer as in previous methods. Then, we obtain  $19.0\%$  on EPIC-Kitchens with RGB, optical flow, and audio compared to  $23.7\%$  by our method. Thus, our feature projection module is more effective as it keeps rich information for accurate prediction.
Beyond Two Unimodal Subsets. Up to this point, we have learned our model using two training sets with distinct modalities. Here we confirm our model extends beyond this to more distinct training and modality sets. We test this with video classification, where we partition the training set into three equal splits, with RGB, optical flow and audio respectively. We keep the testing and validation sets unchanged but use all three modalities. We obtain  $23.8\%$  top-1 accuracy, surpassing the late fusion with  $18.4\%$  and the vanilla transformer with  $17.5\%$ . Thus, our model can solve the general unseen modality interaction task where there are more than two training splits with distinct modalities.
Benefit for Modality-Incomplete Testing. While our model learns from modality-incomplete data, our experiments up to this point use modality-complete data in testing. Here we investigate whether our model is also beneficial to modality-incomplete data at inference. We test this for robot state regression and multimedia retrieval as these have many modalities. Figure 3 shows the improvement of our approach over a vanilla transformer for six different modality combinations for each task. Our model can handle any combination of input modalities at inference and improves over the vanilla transformer for all unseen combinations. We also note that our model is most effective when combining a larger number of modalities as this provides more diverse information.
Benefit for Noisy Modalities. We further demonstrate the benefit of our model by testing it on samples with noisy modalities for unseen modality interactions with video classification. To this end, we apply Gaussian noise on one of the modalities for all samples across the test set. While late fusion delivers  $11.2\%$  accuracy, the vanilla multimodal transformer shows a worse accuracy of  $10.1\%$ , since the noise makes cross-modal correspondences harder to find. In contrast, our model is more robust to noise and obtains  $18.0\%$ . Our model's robustness comes from the feature projection module which selects the informative features during the projection and can thus reduce the noise.
Comparative Results. As there are no prior methods for unseen modality interaction, we compare our approach with recent multimodal learning methods which assume all data to be modality-complete: Gabeur et al. [9], Nagrani et al. [28], Wang et al. [36], as well as methods which are robust to some modality-incomplete data: Recasens et al. [29] and Shvetsova et al. [32]. We use publicly available implementations, except for the work by Recasens et al. [29], which we re-implement
Video Classification	Robot State Regression	Multimedia Retrieval
Top-1 (%) ↑	MAE ↓	MnR ↓
Late fusion‡	18.1	1.29	72.3
Modality Complete
Gabeur et al.	17.3	1.37	88.8
Nagrani et al.	17.5	1.35	86.2
Wang et al.	17.3	1.37	87.8
Modality Incomplete
Shvetsova et al.	18.0	1.25	72.3
Recasens et al.†	18.5	1.22	72.2
Unseen Modality Interaction
This paper	23.7	1.07	66.2
$\ddagger$  Averaging the final predictions from unimodal encoders.
Based on our re-implementation.
Table 4: Comparison with multimodal learning methods for video classification, robot state regression and multimedia retrieval tasks. While previous multimodal learning methods need modality-complete data to learn the cross-modal correspondences, our method gives more effective cross-modal fusion for unseen modality combinations.
ourselves. For all models, we use the same unimodal feature extractors as ours. To adapt modality-complete methods [9, 28, 36] to the unseen modality interaction, we learn a series of learnable feature tokens for each modality and use these learnable tokens in place of the missing modality. Table 4 shows the results for classification, regression, and retrieval tasks. We find that modality-complete methods perform worse than late fusion as they cannot learn the cross-modal correspondences without modality-complete data, therefore cannot effectively make predictions for unseen modality combinations during inference. While the modality-incomplete approaches are robust to some missing modality data, they are not for unseen modality interactions where there is no modality-complete data in training. Thus, we outperform these approaches by effectively accumulating the information from any modality combinations. Overall, existing approaches are not robust to unseen modality combinations whereas our model obtains better generalization ability from modality-incomplete training data. It does so for many different modalities including RGB, audio, depth, and force, and a diverse set of tasks including video classification, robot state regression, and multimedia retrieval.
Conclusion. We pose the problem of unseen modality interaction, where the modality combinations that appear during inference have not been seen during training. To facilitate further analysis and progress, we repurpose and reorganize three existing multimodal datasets to evaluate our approach on a wide variety of modalities, domains, and tasks. Experiments show that we can effectively make predictions for unseen modality interactions by accumulating different modalities projected in a common space and incorporating pseudo-supervision indicating the reliability of the multimodal representation. Our approach is suitable for classification, regression, and retrieval, can handle a wide variety of modality combinations, and is more robust to noisy modalities.
Limitations. Most of our experiments use equally sized training subsets. The effect of an imbalance in the number of samples or number of modalities in different training subsets would be interesting to examine. While effective, our method comes with a longer training time. We also explore only fully supervised learning. Future work should tackle the extension to a self-supervised learning framework.
Broader impact. Learning from data with missing modalities is important for machine learning, as collecting sufficient training data with all modalities of interest can be difficult. A multimodal model that can make predictions with unseen modality combinations can increase robustness when applied to real-world applications. Learning unseen modality interaction is orthogonal to other machine learning challenges like continual learning, federated learning, as well as domain adaptation and generalization, whose combination opens up new lines of research.
Acknowledgement. This work is financially supported by the Inception Institute of Artificial Intelligence, the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.
[24] S. Ma, Z. Zeng, D. McDuff, and Y. Song. Active contrastive learning of audio-visual video representations. In ICLR, 2021.
[25] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In AISTATS, 2017.
[26] A. Miech, I. Laptev, and J. Sivic. Learning a text-video embedding from incomplete and heterogeneous data. arXiv preprint arXiv:1804.02516, 2018.
[27] P. Morgado, I. Misra, and N. Vasconcelos. Robust audio-visual instance discrimination. In CVPR, 2021.
[28] A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, and C. Sun. Attention bottlenecks for multimodal fusion. In NeurIPS, 2021.
[29] A. Recasens, J. Lin, J. Carreira, D. Jaegle, L. Wang, J.-b. Alayrac, P. Luc, A. Miech, L. Smaira, R. Hemsley, et al. Zorro: the masked multimodal transformer. arXiv preprint arXiv:2301.09595, 2023.
[30] N. Sadoughi, X. Li, A. Vajpayee, D. Fan, B. Shuai, H. Santos-Villalobos, V. Bhat, and R. MV. Mega: Multimodal alignment aggregation and distillation for cinematic video segmentation. In ICCV, 2023.
[31] A. Senocak, T.-H. Oh, J. Kim, M.-H. Yang, and I. S. Kweon. Learning to localize sound source in visual scenes. In CVPR, 2018.
[32] N. Shvetsova, B. Chen, A. Rouditchenko, S. Thomas, B. Kingsbury, R. S. Feris, D. Harwath, J. Glass, and H. Kuehne. Everything at once-multi-modal fusion transformer for video retrieval. In CVPR, 2022.
[33] S. Swetha, M. N. Rizve, N. Shvetsova, H. Kuehne, and M. Shah. Preserving modality structure improves multi-modal learning. In ICCV, 2023.
[34] Y. Tian, D. Li, and C. Xu. Unified multisensory perception: weakly-supervised audio-visual video parsing. In ECCV, 2020.
[35] M. Toneva, A. Sordoni, R. T. d. Combes, A. Trischler, Y. Bengio, and G. J. Gordon. An empirical study of example forgetting during deep neural network learning. In ICLR, 2019.
[36] Y. Wang, X. Chen, L. Cao, W. Huang, F. Sun, and Y. Wang. Multimodal token fusion for vision transformers. In CVPR, 2022.
[37] Y. Wang, Z. Cui, and Y. Li. Distribution-consistent modal recovering for incomplete multimodal learning. In ICCV, 2023.
[38] Y. Wu and Y. Yang. Exploring heterogeneous clues for weakly-supervised audio-visual video parsing. In CVPR, 2021.
[39] J. Xu, T. Mei, T. Yao, and Y. Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
[40] Q. Yang, Y. Liu, T. Chen, and Y. Tong. Federated machine learning: Concept and applications. TIST, 10(2):1-19, 2019.
[41] X. Yang, B. Xiong, Y. Huang, and C. Xu. Cross-modal federated human activity recognition via modality-agnostic and modality-specific representation learning. In AAAI, 2022.
[42] J. Zeng, T. Liu, and J. Zhou. Tag-assisted multimodal sentiment analysis under uncertain missing modalities. In SIGIR, 2022.
[43] Y. Zhang, H. Doughty, L. Shao, and C. G. M. Snoek. Audio-adaptive activity recognition across video domains. In CVPR, 2022.
[44] Y. Zhang, L. Shao, and C. G. M. Snoek. Repetitive activity counting by sight and sound. In CVPR, 2021.
[45] J. Zhao, R. Li, and Q. Jin. Missing modality imagination network for emotion recognition with uncertain missing modalities. In ACL, 2021.