Title: MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities
Venue: NeurIPS 2024
Accepted: yes
Keywords: Out-of-Distribution Detection

Multimodal Learning
Abstract: Detecting out-of-distribution (OOD) samples is important for deploying machine learning models in safety-critical applications such as autonomous driving and robot-assisted surgery. Existing research has mainly focused on unimodal scenarios on image data. However, real-world applications are inherently multimodal, which makes it essential to leverage information from multiple modalities to enhance the efficacy of OOD detection. To establish a foundation for more realistic Multimodal OOD Detection, we introduce the first-of-its-kind benchmark, MultiOOD, characterized by diverse dataset sizes and varying modality combinations. We first evaluate existing unimodal OOD detection algorithms on MultiOOD, observing that the mere inclusion of additional modalities yields substantial improvements. This underscores the importance of utilizing multiple modalities for OOD detection. Based on the observation of Modality Prediction Discrepancy between in-distribution (ID) and OOD data, and its strong correlation with OOD performance, we propose the Agree-to-Disagree (A2D) algorithm to encourage such discrepancy during training. Moreover, we introduce a novel outlier synthesis method, NP-Mix, which explores broader feature spaces by leveraging the information from nearest neighbor classes and complements A2D to strengthen OOD detection performance. Extensive experiments on MultiOOD demonstrate that training with A2D and NP-Mix improves existing OOD detection algorithms by a large margin. To support accessibility and reproducibility, our source code and MultiOOD benchmark are available at https://github.com/donghao51/MultiOOD.

MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities

Abstract

Detecting out-of-distribution (OOD) samples is important for deploying machine learning models in safety-critical applications such as autonomous driving and robot-assisted surgery. Existing research has mainly focused on unimodal scenarios on image data. However, real-world applications are inherently multimodal, which makes it essential to leverage information from multiple modalities to enhance the efficacy of OOD detection. To establish a foundation for more realistic Multi modal OOD Detection, we introduce the first-of-its-kind benchmark, MultiOOD, characterized by diverse dataset sizes and varying modality combinations. We first evaluate existing unimodal OOD detection algorithms on MultiOOD, observing that the mere inclusion of additional modalities yields substantial improvements. This underscores the importance of utilizing multiple modalities for OOD detection. Based on the observation of Modality Prediction Discrepancy between in-distribution (ID) and OOD data, and its strong correlation with OOD performance, we propose the Agree-to-Disagree ( A2D ) algorithm to encourage such discrepancy during training. Moreover, we introduce a novel outlier synthesis method, NP-Mix , which explores broader feature spaces by leveraging the information from nearest neighbor classes and complements A2D to strengthen OOD detection performance. Extensive experiments on MultiOOD demonstrate that training with A2D and NP-Mix improves existing OOD detection algorithms by a large margin. To support accessibility and reproducibility, our source code and MultiOOD benchmark are available at https://github.com/donghao51/MultiOOD.

1 Introduction

Most existing machine learning (ML) models are trained under the closed-world assumption, where the test data is assumed to be drawn i.i.d. from the same distribution as the training data, referred to as in-distribution (ID). However, in open-world scenarios, test samples can be out-of-distribution (OOD), thus impacting model robustness and safety [67]. OOD detection aims to detect samples with semantic shifts that are undesirable for the model to generalize [67] and is critical for deploying ML models in safety-critical domains such as autonomous driving [21], robotics [18, 4], and diagnostics for critical assets [24]. Numerous OOD detection algorithms have been developed, ranging from classification-based to distance-based methods [66]. Classification-based methods typically derive confidence directly from the classifier, employing post-hoc processing techniques such as Maximum Softmax Probability (MSP) [31] and Energy [43] or training strategies such as logit normalization [65] and outlier synthesis [22]. Distance-based methods typically measure distances in high-dimensional feature spaces to distinguish between ID and OOD [41, 59]. Additionally, other methodologies explore density estimation [1, 50] and reconstruction techniques [70] for OOD detection.

models [53] to enhance OOD performance, their evaluations are still limited to benchmarks containing solely images . Consequently, existing methods fall short in fully leveraging the complementary information from various modalities, such as LiDAR and camera in autonomous driving [21], as well as video, audio, and optical flow in action recognition [56]. To underscore the importance of using multiple modalities in OOD detection, we evaluate representative OOD algorithms across various modalities on the HMDB51 [39] dataset within our MultiOOD benchmark. This is an action recognition task and all models are trained solely using cross-entropy loss between a one-hot target vector and the softmax output. Results in Fig. 1 show that even a simple fusion of video and optical flow modalities can substantially enhance OOD detection performance.

To establish a foundation for more realistic Multi modal OOD Detection, we introduce a novel OOD benchmark named MultiOOD (Fig. 2), which is the first benchmark for Multimodal OOD Detection and covers diverse dataset sizes and modalities. MultiOOD comprises five video datasets with over 85 , 000 video clips in total. The datasets vary in the number of classes, ranging from 7 to 229 , and in size, spanning from 3 k to 57 k . Video , optical flow , and audio are used as different types of modalities. While most existing unimodal OOD algorithms designed for images can be directly applied to Multimodal OOD Detection, such approaches may yield suboptimal results without accounting for the interaction and complementary nature of diverse modalities. As depicted in Fig. 1, the AUROC performance is very close for all unimodal baselines, underscoring the importance of developing OOD detection algorithms tailored to effectively exploit information from multiple modalities.

In this work, we first identify and illustrate the Modality Prediction Discrepancy phenomenon on the MultiOOD benchmark, where the discrepancies of softmax predictions across different modalities are shown to be negligible for ID data while significant for OOD data (Fig. 3). We discover a strong correlation between such discrepancies and the OOD detection performance (Fig. 4). Motivated by these observations, we introduce the Agree-to-Disagree ( A2D ) algorithm, which aims to enhance such discrepancies during training. The algorithm is designed so that different modalities should Agree on the prediction of the ground-truth class, and Disagree on other classes by maximizing the distance between their predictions. Additionally, we propose a novel outlier synthesis algorithm named NP-Mix , designed to use the information from nearest neighbor classes to explore broader feature spaces and complement A2D to strengthen the OOD detection performance.

Extensive experiments on the MultiOOD benchmark demonstrate the superiority of A2D and NP-Mix . The integration of A2D and NP-Mix yields substantial performance enhancements over existing unimodal OOD detection algorithms. For instance, on the UCF101 [57] dataset within MultiOOD, our approach reduces the FPR95 from 32 . 14% to 10 . 68% for ASH [16] method, representing a noteworthy absolute 21 . 46% improvement over the baseline. Our contributions include:

We highlight the significance of integrating more modalities for OOD detection and introduce MultiOOD, the first benchmark for Multimodal OOD Detection encompassing diverse dataset sizes and various combinations of modalities.

We conduct comprehensive evaluations of existing unimodal OOD algorithms on MultiOOD, revealing their limitations in multimodal scenarios.

We propose a novel A2D training algorithm, inspired by the observation of the Modality Prediction Discrepancy phenomenon, alongside a new outlier synthesis algorithm NP-Mix that explores broader feature spaces and complements A2D to strengthen the OOD detection performance.

Extensive experiments conducted on MultiOOD underscore the effectiveness of A2D and NP-Mix . Our source code and MultiOOD benchmark will be made publicly available, facilitating future research endeavors in Multimodal OOD Detection.

EPIC-Kitchens

4/4

HMDB51

25/26

UCF101

50/51

129/100

HMDB51

4

×

Multimodal Near-OOD benchmark

2

×

Multimodal Far-OOD benchmark

In-Distribution

Out-of-Distribution

catch

clap

dribble

fencing

laugh

somersault

situp

punch

kiss

jump

golf

pullup

In-Distribution

Out-of-Distribution

UCF101

HAC

HMDB51

EPIC-Kitchens

Video

Optical Flow

Audio

2 Preliminaries: Multimodal Out-of-distribution Detection

Multimodal OOD Detection aims to detect samples with semantic shifts using multiple modalities . We consider a training set D in = { ( x i , y i ) } n i =1 drawn i.i.d. from the joint data distribution P XY , where X is the input space and Y = { 1 , 2 , ..., C } is the discrete label space. Each training sample x i is comprised of M modalities, denoted as x i = { x k i | k = 1 , · · · , M } . Let P in denote the marginal distribution on X and f : X ↦→ R C be a neural network trained on samples in P XY that predicts the label of each input sample. The f in Multimodal OOD Detection comprises of M feature extractors g k ( · ) and a classifier h ( · ) . Each feature extractor g k ( · ) extracts an embedding Z k for its corresponding modality k , and the classifier h ( · ) takes the combined embeddings from all modalities as input and outputs a prediction probability ˆ p :

where δ ( · ) is the softmax function. We further include a separate classifier h k ( · ) for each modality k to get predictions from each modality separately, with the prediction probability from the k -th modality as ˆ p k = δ ( h k ( g k ( x k ))) .

When deploying f in the real world, it should not only accurately classify known samples as ID, but also identify any 'unknown' sample as OOD. A separate score function S ( x ) is often used to decide whether a sample x ∈ X is from P in (ID) or not (OOD):

where samples with higher scores S ( x ) are classified as ID and vice versa, η is the threshold. Existing OOD detection studies predominantly focus on unimodal scenarios, with a detailed literature review offered in Appendix A. To establish a foundation for more realistic Multimodal OOD Detection, we introduce a novel MultiOOD benchmark (Sec. 3) and propose an effective multimodal training strategy (Sec. 4) that yields significant enhancements over existing unimodal approaches.

3 MultiOOD Benchmark

We create the MultiOOD benchmark to understand the existing gap in Multimodal OOD Detection research. OOD detection primarily focuses on detecting semantic shifts, with two main approaches used for constructing OOD benchmarks. A common method involves considering an entire dataset as in-distribution (ID) and further collects datasets, which comprise similar tasks but are disconnected from any ID categories, as OOD datasets. In this scenario, both semantic and domain shifts are present between the ID and OOD samples. We term this setup as Far-OOD in our benchmark. Another approach is to partition the categories of existing datasets into two subsets, referred to as closed (ID) and open set (OOD). Here, both ID and OOD samples originate from the same distribution, with only semantic shifts existing between them. We denote this setup as Near-OOD within our benchmark;

(a) Prediction for ID data

(b) Prediction for OOD data

Distance: 0.16

Distance: 1.90

a setup that poses greater challenges compared to Far-OOD . This setup is also referred to as open set recognition (OSR) in some studies [61, 37, 9]. Notably, OSR and OOD detection both share the same goal of identifying test samples with semantic shifts without compromising the accuracy of ID classification [67]. In our benchmark, we treat OSR and OOD as synonymous concepts and adopt OOD as the general term. MultiOOD comprises five action recognition datasets (EPIC-Kitchens [48], HAC [20], HMDB51 [39], UCF101 [57], and Kinetics-600 [6]) with over 85 , 000 video clips in total. The datasets vary in the number of classes, ranging from 7 to 229 , and in size, spanning from 3 k to 57 k . Video , optical flow , and audio are used as different types of modalities. An overview of the MultiOOD benchmark is provided in Fig. 2, with additional details available in Appendix B.

3.1 Multimodal Near-OOD Benchmark

In the Near-OOD setup, we include four datasets. EPIC-Kitchens 4/4 is derived from the EPICKitchens Domain Adaptation dataset [48], where the dataset is partitioned into four classes for training as ID and four classes for testing as OOD, with a total of 4 , 871 video clips. Similarly, HMDB51 25/26 and UCF101 50/51 are constructed based on HMDB51 [39] and UCF101 [57], with a total of 6 , 766 and 13 , 320 video clips respectively. In the case of Kinetics-600 129/100 , we select 229 action classes from the Kinetics-600 dataset [6], with each class comprising approximately 250 video clips and a total of 57 , 205 video clips. Within this setup, 129 classes are designated for training as ID, while the remaining 100 classes are allocated for testing as OOD.

3.2 Multimodal Far-OOD Benchmark

In the Far-OOD setup, we include HMDB51 and Kinetics-600 as ID datasets.

HMDB51 dataset as ID. For the OOD datasets, we utilize UCF101, EPIC-Kitchens, HAC, and Kinetics-600 datasets. All of these datasets are carefully curated to remove samples that belong to ID classes in HMDB51. Given the relatively small number of classes in the EPIC-Kitchens and HAC datasets, we remove 8 classes in the HMDB51 dataset that overlap with EPIC-Kitchens and HAC, with 43 classes left as ID classes. For UCF101, we remove 31 overlapping classes with HMDB51, resulting in 70 classes designated as OOD classes for evaluation. For other datasets, no class overlap exists and we utilize their original classes as OOD.

Kinetics-600 dataset as ID. Similarly, we adopt UCF101, EPIC-Kitchens, HAC, and HMDB51 datasets as OOD datasets, with careful selection undertaken to exclude samples belonging to ID classes in Kinetics-600. We carefully selected a subset of 229 action classes from Kinetics-600 in the Near-OOD setup to mitigate the potential overlap with other datasets. For the UCF101 dataset, we remove 11 overlapping classes with Kinetics-600, leaving 90 classes as OOD classes for evaluation. For other datasets, there are no class overlap issues and we use their original classes as OOD.

4 Methodology

In this section, we first identify the Modality Prediction Discrepancy phenomenon on the MultiOOD benchmark and demonstrate its substantial correlation with the OOD detection performance (Sec. 4.1). Subsequently, we introduce the Agree-to-Disagree ( A2D ) algorithm aimed at enhancing such discrep-

ancies during training (Sec. 4.2). Finally, we propose the novel outlier synthesis algorithm named NP-Mix that complements A2D to further strengthen the OOD detection performance (Sec. 4.3).

4.1 Modality Prediction Discrepancy

We first explore the predictive behaviors demonstrated by various modalities when using both ID and OOD data as input on MultiOOD. We compute the prediction probabilities employing classifiers for video and optical flow on the same ID and OOD samples and calculate their L 1 distances. As depicted in Fig. 3, for ID data, the prediction probabilities of both video ˆ p 1 and optical flow ˆ p 2 are generally exhibited consistent with each other on the ground-truth label, consequently yielding a small L 1 distance ∥ ˆ p 1 -ˆ p 2 ∥ 1 between them. Conversely, for OOD data, each modality tends to express varying confidence preferences towards distinct classes, resulting in a notable increase in the L 1 distance between their predictions. We refer to this phenomenon as Modality Prediction Discrepancy between ID and OOD data. This discrepancy can be attributed to the unavailability of semantic information on OOD data during model training, stimulating each modality to generate conjectures based on its unique characteristics upon encountering OOD data during testing.

To verify the universality of this phenomenon, we calculate the average L 1 distance between predictions of video ˆ p 1 and optical flow ˆ p 2 on both ID and OOD data within the HMDB51 dataset. The average prediction L 1 distance is 0 . 63 for ID data ( l ID ) and 1 . 42 for OOD data ( l OOD ), revealing a substantial discrepancy. In addition, we evaluate other datasets in the Near-OOD setup within the MultiOOD benchmark and observe similar discrepancies ( l OOD -l ID ). Such discrepancies have a positive correlation with the OOD detection performance, as illustrated in Fig. 4.

4.2 Agree-to-Disagree Algorithm

Motivated by the Modality Prediction Discrepancy and its strong correlation with Multimodal OOD Detection performance, we introduce the Agree-to-Disagree ( A2D ) algorithm to foster the amplification of such discrepancies during training. The underlying idea is that different modalities should Agree on the prediction regarding the ground-truth class, while they should Disagree on the remaining classes by maximizing their prediction distance. A2D enables the model to diversify predictions across modalities, consequently yielding high prediction discrepancies for OOD data during testing.

Given a training sample x with label c , we obtain prediction probabilities ˆ p from the combined embeddings of all modalities, and ˆ p 1 , ˆ p 2 from individual modality, all of which are of shape [1 , C ] , where C represents the number of classes. By removing the c -th value from ˆ p 1 and ˆ p 2 (different modalities should Agree on the prediction regarding the ground-truth class), we derive new prediction probabilities without ground-truth classes, denoted as ¯ p 1 and ¯ p 2 with shapes [1 , C -1] . Subsequently, we aim to maximize the discrepancy between ¯ p 1 and ¯ p 2 , which can be defined as:

Modality 1

Modality 2

A2D

Training

Agree-to-Disagree (A2D)

Outlier Synthesis Using NP-Mix

Prototypes

Nearest Neighbor

Synthesized Outliers

0.5

0.5

+

Z1

h1

Z

h

p

pl

pl

p2

p2

h2

Z1

h1

pl

p2

h2

Z

where Discr ( · ) is a distance metric quantifying the similarity between two probability distributions. We utilize the Hellinger distance [49] and explore the efficacy of alternative distance metrics in our ablation study. The Hellinger distance between two probability distributions is defined as:

Furthermore, to ensure that the ground-truth classes possess the highest probabilities, we incorporate the cross-entropy loss CE ( · ) for each prediction, defined as:

The final loss is obtained as the weighted sum of the previously defined losses:

where the hyperparameter γ controls the relative importance of the discrepancy term.

4.3 Nearest Neighbor Prototype-based Mixup for Outlier Synthesis

Outlier synthesis [22, 60] has demonstrated its efficacy in OOD detection by imposing regularization on the model's decision boundary during training. Introducing the discrepancy loss in A2D to the synthesized outlier data for model regularization has the potential to further enhance OOD detection performance. However, existing outlier synthesis methods [22, 60] typically generate outliers near the ID data (Fig. 7), neglecting to explore the broader embedding spaces, thereby potentially leading to suboptimal performance. Inspired by the recent approach introduced in [19], we introduce a novel algorithm termed N earest Neighbor P rototype-based Mi xup ( NP-Mix ), aimed at synthesizing outliers capable of spanning wider embedding spaces by leveraging the information from nearest neighbor classes, as shown in Fig. 5 and Fig. 7. To synthesize outliers, we concatenate the embeddings of all modalities ( Z = [ Z 1 , Z 2 ] ) and treat them as a unified entity. Subsequently, we compute a prototype embedding ˆ Z c for each class c by calculating the mean of all embeddings within that class. For each prototype embedding ˆ Z c , we identify its top N nearest neighbor prototypes and randomly select one prototype ˆ Z s from them for mixing. Two samples, Z 1 and Z 2 , are randomly chosen from class c and class s respectively. The outlier ˜ Z is generated by their convex combination:

where λ ∼ Beta ( α, α ) , for α ∈ (0 , ∞ ) . In our experiments, we opt for α > 1 to ensure the synthesized outliers reside in the intermediary space between two prototypes, rather than near the ID data. We then partition ˜ Z into different modalities as ˜ Z = [ ˜ Z 1 , ˜ Z 2 ] , where ˜ Z 1 and ˜ Z 2 represent the synthesized outlier embeddings for each modality. Subsequently, the corresponding prediction probabilities are computed as ˜ p 1 = δ ( h 1 ( ˜ Z 1 )) and ˜ p 2 = δ ( h 2 ( ˜ Z 2 )) .

For synthesized outliers, we aim to maximize the prediction discrepancies between different modalities, similar to Eq. (2) for ID training samples. In this context, there is no need to remove any value from the prediction, as the outliers are assumed not to belong to any ID class. The discrepancy loss between ˜ p 1 and ˜ p 2 can be defined as:

Moreover, we seek to mitigate the overconfidence of predictions for synthesized outliers towards any existing ID class. Therefore, we maximize the entropy of predictions:

where H ( ˜ p ) is the entropy of prediction ˜ p and can be calculated as H ( ˜ p ) = -∑ c ˜ p c log ˜ p c . The final loss is obtained as the weighted sum of the previously defined losses:

5 Experiments

5.1 Experimental Setting

Baselines. Our objective is to enhance existing unimodal OOD algorithms through multimodal training utilizing the proposed A2D and NP-Mix . To validate the efficacy and versatility of our proposed approaches, we include representative algorithms that leverage information from the probability space (MSP [31], GEN [44]), logit space (Energy [43], MaxLogit [30]), feature space (Mahalanobis [41], KNN [59]), both logit and feature spaces (VIM [62]), penultimate activation manipulations (ReAct [58], ASH [16]), and training-time regularization (LogitNorm [65]).

Evaluation metrics. We evaluate the performance via the use of the following metrics: (1) the false positive rate (FPR95) of OOD samples when the true positive rate of ID samples is at 95% , (2) the

area under the receiver operating characteristic curve (AUROC), and (3) ID classification accuracy (ID ACC). For each baseline algorithm, we report the results both with and without A2D training and NP-Mix outlier synthesis. Additional implementation details are provided in Appendix D.

5.2 Multimodal Near-OOD Detection

We commence our evaluation by assessing the efficacy of A2D training and NP-Mix outlier synthesis within the Multimodal Near-OOD Detection setup. As presented in Tab. 1, the simple incorporation of A2D yields substantial enhancements in OOD performance across nearly all scenarios. The average prediction L 1 distances between ID and OOD data ( l OOD -l ID ) before and after A2D training across various datasets are depicted in Fig. 4. Notably, across all datasets, A2D training serves to further enlarge the Modality Prediction Discrepancy and consequently improve OOD detection performance, verifying the strong correlation between them. Specifically, A2D training reduces the FPR95 by up to absolute 19 . 62% on UCF101 50/51 dataset for ASH and increases AUROC up to 14 . 82% for Mahalanobis on Kinetics-600 129/100. Combining A2D training with NP-Mix yields additional improvements in OOD detection performance, underscoring the efficacy of outlier synthesis in model regularization. Additionally, A2D training and NP-Mix outlier synthesis exhibit notable efficacy across all baseline OOD detection algorithms despite their different underlying principles, indicating the versatility of our proposed method.

5.3 Multimodal Far-OOD Detection

Tab. 2 and Tab. 3 present the results of Multimodal Far-OOD Detection with HMDB51 and Kinetics600 as ID datasets, respectively. Similar to the Near-OOD setup, training with A2D and NP-Mix yields considerable enhancements in OOD detection performance across most cases for all baseline algorithms. Specifically, training with both A2D and NP-Mix achieves reductions in FPR95 of up to absolute 23 . 38% on the HMDB51 dataset for ASH and up to 14 . 43% for ReAct on the Kinetics-600 dataset. Due to space limits, we provide comprehensive results in Appendix E (Tab. 7 and Tab. 8). Interestingly, we observe that the performance on the HMDB51 dataset generally surpasses that of the Kinetics-600 dataset. This finding aligns with observations from unimodal OOD detection benchmarks [66], where performance on datasets with fewer classes (e.g., CIFAR-10 [38]) tends to outperform those on datasets with a greater number of classes (e.g., ImageNet [14]).

5.4 Ablation Studies

Multimodal Near-OOD Detection with other modalities. We demonstrate the efficacy of A2D training and NP-Mix outlier synthesis across various combinations of modalities, not limited to video and optical flow, as detailed in Tab. 4 and Tab. 9 in Appendix E. Notably, the performance of most algorithms is consistently improved with A2D and NP-Mix , regardless of which combinations of modalities are used.

Effectiveness of other distance functions. We substitute the distance metric for measuring probability distributions in the discrepancy loss with alternative distance functions, including L 1 , L 2 , and Wasserstein [2] distances. As depicted in Tab. 5, A2D training exhibits robustness across various distance functions. Regardless of the specific distance metric employed, substantial improvements are consistently observed compared to the baseline approach without A2D training.

Compared with other outlier synthesis methods. To evaluate the effectiveness of other outlier synthesis methods, we replace NP-Mix with VOS [22], NPOS [60], and Mixup [69] and train with A2D . All baseline methods yield improvements in OOD performance, underscoring the significance of outlier synthesis for model regularization. Notably, NP-Mix emerges as the most effective among the various baselines by synthesizing outliers that span broader embedding spaces.

6 Conclusion

In this paper, we introduce the Multimodal Out-of-distribution Detection problem and present a novel benchmark, MultiOOD, which includes diverse dataset sizes and various combinations of modalities. Motivated by the Modality Prediction Discrepancy phenomenon observed within MultiOOD, we propose a novel A2D training algorithm to encourage the enlargement of such discrepancy during model training, along with a new outlier synthesis algorithm, NP-Mix , that complements A2D . Extensive experiments on MultiOOD and under Near-OOD and Far-OOD setups verify the efficacy of the proposed approaches. We hope our work will inspire future research endeavors in Multimodal OOD Detection.

Limitation and Future Work. The performance on Near-OOD benchmarks and on datasets with a large number of classes still exhibits potential for enhancement. Moreover, the exploration of Outlier Exposure [32] deserves attention as a potential to better learn ID/OOD discrepancy.

Acknowledgments

The authors acknowledge the support of "In-service diagnostics of the catenary/pantograph and wheelset axle systems through intelligent algorithms" (SENTINEL) project, supported by the ETH Mobility Initiative.

References

Abati, D., Porrello, A., Calderara, S., Cucchiara, R.: Latent space autoregression for novelty detection. In: CVPR (2019)

Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: ICML (2017)

Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., Gall, J.: SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences. In: ICCV (2019)

Blum, H., Sarlin, P.E., Nieto, J., Siegwart, R., Cadena, C.: The fishyscapes benchmark: Measuring blind spots in semantic segmentation. International Journal of Computer Vision 129 (11), 3119-3135 (2021)

Breunig, M.M., Kriegel, H.P., Ng, R.T., Sander, J.: Lof: identifying density-based local outliers. In: Proceedings of the 2000 ACM SIGMOD international conference on Management of data. pp. 93-104 (2000)

Carreira, J., Noland, E., Banki-Horvath, A., Hillier, C., Zisserman, A.: A short note about kinetics-600. arXiv preprint arXiv:1808.01340 (2018)

Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: CVPR (2017)

Chan, R., Lis, K., Uhlemeyer, S., Blum, H., Honari, S., Siegwart, R., Fua, P., Salzmann, M., Rottmann, M.: Segmentmeifyoucan: A benchmark for anomaly segmentation. arXiv preprint arXiv:2104.14812 (2021)

Chen, G., Peng, P., Wang, X., Tian, Y.: Adversarial reciprocal points learning for open set recognition. arXiv preprint arXiv:2103.00953 (2021)

Chen, H., Xie, W., Vedaldi, A., Zisserman, A.: Vggsound: A large-scale audio-visual dataset. In: ICASSP (2020)

Contributors, M.: Openmmlab's next generation video understanding toolbox and benchmark. https://github.com/open-mmlab/mmaction2 (2020)

Cortinhal, T., Tzelepis, G., Aksoy, E.E.: Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds for autonomous driving. arXiv preprint arXiv:2003.03653 (2020)

Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., Wray, M.: Scaling egocentric vision: The epic-kitchens dataset. In: ECCV (2018)

Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR (2009)

Deng, L.: The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine 29 (6), 141-142 (2012)

Djurisic, A., Bozanic, N., Ashok, A., Liu, R.: Extremely simple activation shaping for out-ofdistribution detection. arXiv preprint arXiv:2209.09858 (2022)

Dong, H., Chatzi, E., Fink, O.: Towards multimodal open-set domain generalization and adaptation through self-supervision. In: ECCV (2024)

Dong, H., Chen, X., Särkkä, S., Stachniss, C.: Online pole segmentation on range images for long-term lidar localization in urban environments. Robotics and Autonomous Systems 159 , 104283 (2023)

Dong, H., Frusque, G., Zhao, Y., Chatzi, E., Fink, O.: Nng-mix: Improving semi-supervised anomaly detection with pseudo-anomaly generation. arXiv preprint arXiv:2311.11961 (2023)

Dong, H., Nejjar, I., Sun, H., Chatzi, E., Fink, O.: SimMMDG: A simple and effective framework for multi-modal domain generalization. In: NeurIPS (2023)

Dong, H., Zhang, X., Xu, J., Ai, R., Gu, W., Lu, H., Kannala, J., Chen, X.: Superfusion: Multilevel lidar-camera fusion for long-range hd map generation. arXiv preprint arXiv:2211.15656 (2022)

Du, X., Wang, Z., Cai, M., Li, Y.: Vos: Learning what you don't know by virtual outlier synthesis. In: ICLR (2022)

Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition. In: ICCV (2019)

Fink, O., Wang, Q., Svensén, M., Dersin, P., Lee, W.J., Ducoffe, M.: Potential, challenges and future directions for deep learning in prognostics and health management applications. Engineering Applications of Artificial Intelligence 92 , 103678 (2020). https://doi.org/https://doi.org/10.1016/j.engappai.2020.103678

Gong, Y., Chung, Y.A., Glass, J.: Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778 (2021)

Gorishniy, Y., Rubachev, I., Khrulkov, V., Babenko, A.: Revisiting deep learning models for tabular data. NeurIPS (2021)

Gupta, S., Hoffman, J., Malik, J.: Cross modal distillation for supervision transfer. In: CVPR (2016)

Han, S., Hu, X., Huang, H., Jiang, M., Zhao, Y.: Adbench: Anomaly detection benchmark. In: Neural Information Processing Systems (NeurIPS) (2022)

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)

Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., Song, D.: Scaling out-of-distribution detection for real-world settings. ICML (2022)

Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: ICLR (2017)

Hendrycks, D., Mazeika, M., Dietterich, T.: Deep anomaly detection with outlier exposure. In: ICLR (2019)

Jaritz, M., Vu, T.H., de Charette, R., Wirbel, E., Pérez, P.: xMUDA: Cross-modal unsupervised domain adaptation for 3D semantic segmentation. In: CVPR (2020)

Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al.: The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 (2017)

Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. In: NeurIPS (2020)

Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. ICLR (2015)

Kong, S., Ramanan, D.: Opengan: Open-set recognition via open data generation. In: ICCV (2021)

Krizhevsky, A., Nair, V., Hinton, G.: Cifar-10 and cifar-100 datasets. URl: https://www. cs. toronto. edu/kriz/cifar. html 6 (1), 1 (2009)

Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: Hmdb: a large video database for human motion recognition. In: ICCV (2011)

LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., Huang, F.: A tutorial on energy-based learning. Predicting structured data 1 (0) (2006)

Lee, K., Lee, K., Lee, H., Shin, J.: A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In: NeurIPS (2018)

Liang, S., Li, Y ., Srikant, R.: Enhancing the reliability of out-of-distribution image detection in neural networks. In: ICLR (2018)

Liu, W., Wang, X., Owens, J.D., Li, Y.: Energy-based out-of-distribution detection. In: NeurIPS (2020)

Liu, X., Lochman, Y., Zach, C.: Gen: Pushing the limits of softmax-based out-of-distribution detection. In: CVPR (2023)

Liu, Z., Ning, J., Cao, Y ., Wei, Y ., Zhang, Z., Lin, S., Hu, H.: Video swin transformer. In: CVPR (2022)

Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9 (11) (2008)

Ming, Y., Cai, Z., Gu, J., Sun, Y ., Li, W., Li, Y .: Delving into out-of-distribution detection with vision-language representations. In: NeurIPS (2022)

Munro, J., Damen, D.: Multi-modal domain adaptation for fine-grained action recognition. In: CVPR (2020)

Pardo, L.: Statistical inference based on divergence measures. Chapman and Hall/CRC (2018)

Pidhorskyi, S., Almohsen, R., Adjeroh, D.A., Doretto, G.: Generative probabilistic novelty detection with adversarial autoencoders. In: NeurIPS (2018)

Planamente, M., Plizzari, C., Alberti, E., Caputo, B.: Domain generalization through audiovisual relative norm alignment in first person action recognition. In: WACV (2022)

Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V., Gulin, A.: Catboost: unbiased boosting with categorical features. NeurIPS (2018)

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: ICML (2021)

Ruff, L., Vandermeulen, R., Goernitz, N., Deecke, L., Siddiqui, S.A., Binder, A., Müller, E., Kloft, M.: Deep one-class classification. In: International conference on machine learning. pp. 4393-4402. PMLR (2018)

Ruff, L., Vandermeulen, R.A., Görnitz, N., Binder, A., Müller, E., Müller, K., Kloft, M.: Deep semi-supervised anomaly detection. In: ICLR (2020)

Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NeurIPS (2014)

Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)

Sun, Y., Guo, C., Li, Y.: React: Out-of-distribution detection with rectified activations. In: NeurIPS (2021)

Sun, Y., Ming, Y., Zhu, X., Li, Y.: Out-of-distribution detection with deep nearest neighbors. ICML (2022)

Tao, L., Du, X., Zhu, X., Li, Y.: Non-parametric outlier synthesis. arXiv preprint arXiv:2303.02966 (2023)

Vaze, S., Han, K., Vedaldi, A., Zisserman, A.: Open-set recognition: A good closed-set classifier is all you need. In: ICLR (2022)

Wang, H., Li, Z., Feng, L., Zhang, W.: Vim: Out-of-distribution with virtual-logit matching. In: CVPR (2022)

Wang, H., Li, Y., Yao, H., Li, X.: Clipn for zero-shot ood detection: Teaching clip to say no. In: ICCV (2023)

Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., Van Gool, L.: Temporal segment networks: Towards good practices for deep action recognition. In: ECCV (2016)

Wei, H., Xie, R., Cheng, H., Feng, L., An, B., Li, Y.: Mitigating neural network overconfidence with logit normalization. In: ICML (2022)

Yang, J., Wang, P., Zou, D., Zhou, Z., Ding, K., Peng, W., Wang, H., Chen, G., Li, B., Sun, Y., et al.: Openood: Benchmarking generalized out-of-distribution detection. In: NeurIPS (2022)

Yang, J., Zhou, K., Li, Y., Liu, Z.: Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334 (2021)

Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime tv-l 1 optical flow. In: Pattern Recognition (2007)

Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: ICLR (2018)

Zhou, Y.: Rethinking reconstruction autoencoder-based out-of-distribution detection. In: CVPR (2022)

Zhou, Y., Song, X., Zhang, Y., Liu, F., Zhu, C., Liu, L.: Feature encoding with autoencoders for weakly supervised anomaly detection. IEEE Transactions on Neural Networks and Learning Systems 33 (6), 2454-2465 (2021)

Zhuang, Z., Li, R., Jia, K., Wang, Q., Li, Y ., Tan, M.: Perception-aware multi-sensor fusion for 3d lidar semantic segmentation. In: ICCV (2021)

A Related Work

Out-of-Distribution (OOD) Detection aims to detect test samples with semantic shift without losing the ID classification accuracy. Numerous OOD detection algorithms have been developed, with post hoc methods and training-time regularization as major categories [66]. Post hoc methods aim to design OOD scores based on the classification output of neural networks, offering the advantage of being easy to use without modifying the training procedure and objective. Early approaches include utilizing Maximum Softmax Probability (MSP) [31] as OOD score, often coupled with temperature scaling and input perturbation [42]. Instead of using softmax probabilities, MaxLogit [30] employs maximum logit as OOD scores rather than softmax. Energy-based algorithm [43] demonstrated the efficacy of energy function [40] in quantifying OOD-ness. Other approaches like ReAct [58] improved existing scoring functions by truncating the activations with high values. Similarly, ASH [16] prunes a large portion of the input activations and adjusts the remaining activations using pruning, binarizing, or scaling. Methods like Mahalanobis [41] and k -Nearest Neighbor (KNN) [59] use distance metrics in feature space for OOD detection, while Virtual-logit Matching (VIM) [62] integrates information from both feature and logit spaces to define the OOD score. Recently, Generalized Entropy (GEN) [44] proposed an entropy-based score function that proves to be both simple and effective.

Training-time regularization methods such as LogitNorm [65] address prediction overconfidence by imposing a constant vector norm on the logits during training. Outlier Exposure [32] leverages external OOD samples from other datasets during training to facilitate the learning of better ID and OOD discrepancy. Additionally, some approaches [22, 60] proposed synthesizing virtual outliers for training-time regularization. However, all previous approaches were designed for unimodal scenarios, without accounting for the interaction and complementary nature of diverse modalities.

Multimodal OOD Detection. Recent endeavors [47, 63] have explored vision-language models [53] to enhance OOD performance, which are also referred to as multimodal in some of the studies. Maximum Concept Matching (MCM) [47] defines OOD score by aligning visual features with textual concepts. CLIPN [63] equips CLIP [53] with the capability of distinguishing OOD and ID samples using positive-semantic prompts and negation-semantic prompts. However, the evaluations of all these works are still limited to benchmarks exclusively containing images . Consequently, existing methods fall short in fully leveraging the complementary information from various modalities, such as LiDAR and camera in autonomous driving [21], as well as video, audio, and optical flow in action recognition [56]. There is also a lack of benchmark datasets that facilitate the evaluation of Multimodal OOD Detection. Therefore, we aim to develop a more practical and challenging benchmark incorporating multiple combinations of modalities (i.e., video, audio, and optical flow). This enables the creation of OOD detection algorithms that are specifically designed to leverage the complementary nature of various modalities effectively.

Anomaly Detection and Open Set Recognition are two closely related fields to OOD Detection. Anomaly Detection (AD) aims to detect patterns that deviate from the predefined normality during testing [67] and treats all in-distribution samples as a single class. Therefore, AD algorithms can be applied to OOD detection by ignoring all the labels for ID data. Typical AD algorithms include unsupervised [54, 5], semi-supervised [55, 71], and supervised [26, 52], depending on the availability of labels [28]. Open Set Recognition (OSR) [61, 37, 9, 17] focuses on accurately classifying test samples from 'known known classes' (ID) and detecting test samples from 'unknown unknown classes' (OOD). While OOD detection benchmarks always take one dataset as ID and find several other datasets with non-overlapping categories as OOD, OSR benchmarks usually split one multiclass classification dataset into ID and OOD parts according to classes. However, both OSR and OOD detection share the same goal of identifying test samples with semantic shifts without compromising the accuracy of ID classification [67]. Therefore, in our benchmark, we treat OSR and OOD as synonymous concepts and adopt OOD as the general term. Our Near-OOD Benchmark is similar to traditional OSR setup and our Far-OOD Benchmark is the same as general OOD setup.

OODBenchmarks. Early works [31] in OOD detection primarily focus on small-scale image datasets such as MNIST [15] and CIFAR-10/100 [38]. Recognizing the need for evaluating OOD detection at scale, studies such as [62] introduce new OOD datasets based on ImageNet [14]. Additionally, some OOD benchmarks [30, 8] are specifically designed for semantic segmentation tasks. OpenOOD [66] offers a comprehensive OOD benchmark comprising datasets from previous works and incorporating over 30 common OOD methods. However, all of these benchmarks are limited to image data. In contrast, our MultiOOD is the first public OOD benchmark that encompasses different combinations of modalities, facilitating future research endeavors in Multimodal OOD Detection.

HAC

HMDB51

EPIC-Kitchens

WritingOnBoard

YoYo

BabyCrawling

Pullups

Push ups

Rock ClimbingIndo

Haircut

Head Massage

UCF101

B More Details on the MultiOOD Benchmark

B.1 Datasets Used in MultiOOD Benchmark

Action recognition is inherently multimodal and serves as the primary task within our benchmark, and we include five action recognition datasets accordingly of varying sizes, as shown in Fig. 6.

EPIC-Kitchens [13]. The EPIC-Kitchens dataset is a large-scale egocentric dataset collected by 32 participants in their native kitchen environments. The participants were asked to capture all their daily kitchen activities and record sequences regardless of their duration. The start and end times for each action are annotated. We use a subset of the EPIC-Kitchens dataset introduced in the Multimodal Domain Adaptation paper [48], which comprises 4 , 871 video clips from 8 largest action classes in sequence P22. These actions include 'put', 'take', 'open', 'close', 'wash', 'cut', 'mix', and 'pour', with provided modalities including video , optical flow , and audio .

HAC [20]. The HAC dataset encompasses seven actions ('sleeping', 'watching tv', 'eating', 'drinking', 'swimming', 'running', and 'opening door') performed by humans, animals, and cartoon figures. There are 3 , 381 video clips in total from seven actions. Modalities provided in this dataset include video , optical flow , and audio .

HMDB51 [39]. The HMDB51 dataset is a video action recognition dataset, comprising 6 , 766 video clips spanning 51 action categories. The video clips are extracted from a variety of sources ranging from digitized movies to YouTube. Available modalities in this dataset include video and optical flow .

UCF101 [57]. UCF101 is another video action recognition dataset collected from YouTube, comprising 13 , 320 video clips from 101 actions. UCF101 offers substantial diversity in action types and encompasses significant variations in camera motion, object appearance and pose, object scale, viewpoint, cluttered background, illumination conditions, etc. Modalities provided in this dataset include video and optical flow .

Kinetics-600 [6]. Kinetics-600 is a large-scale action recognition dataset, featuring approximately 480 k videos spanning 600 action categories. Each video in the dataset is a 10 -second clip of action moment annotated from YouTube videos. In our benchmark, we carefully selected a subset of 229 action classes from Kinetics-600 to mitigate the potential category overlap with other datasets, with each class comprising roughly 250 video clips, yielding a total of 57 , 205 video clips. The original dataset provides video and audio modalities. To make it consistent with the other dataset, our benchmark further provides the extracted optical flow for all video clips, amounting to a total of 114 , 410 optical flow samples. The dense optical flow is extracted at 24 frames per second using the TV-L1 algorithm [68].

B.2 Multimodal Near-OOD Benchmark

In the Near-OOD setup, we incorporate four datasets. EPIC-Kitchens 4/4 is derived from the EPICKitchens Domain Adaptation dataset [48], where the dataset is randomly partitioned into four classes for training as ID and four classes for testing as OOD. Similarly, HMDB51 25/26 and UCF101 50/51

(a) VOS

(b) NPOS

(d) NP-Mix (ours)

(c) Mixup

are constructed based on HMDB51 [39] and UCF101 [57], respectively. In the case of Kinetics-600 129/100 , we curate 229 action classes from the Kinetics-600 dataset [6], with each class comprising approximately 250 video clips. Within this setup, 129 classes are randomly designated for training as ID, while the remaining 100 classes are allocated for testing as OOD.

B.3 Multimodal Far-OOD Benchmark

In the Far-OOD setup, we incorporate HMDB51 and Kinetics-600 as ID datasets.

HMDB51 dataset as ID. For the OOD datasets, we utilize UCF101, EPIC-Kitchens, HAC, and Kinetics-600 datasets. All of these datasets are carefully curated to remove samples that should belong to ID classes. Given the relatively small number of classes in the EPIC-Kitchens and HAC datasets, we remove 8 classes in the HMDB51 dataset that overlap with EPIC-Kitchens and HAC, including 'chew', 'climb_stairs', 'drink', 'eat', 'pick', 'pour', 'ride_horse', 'run', leaving 43 classes as ID classes. In the case of UCF101, we remove 31 overlapping classes with HMDB51, resulting in 70 classes designated as OOD classes for evaluation. For Kinetics-600, we use the same subset of 229 classes as in the Near-OOD setup, which are carefully selected to mitigate the potential category overlap with other datasets. For other datasets, no class overlap exists and we utilize their original classes as OOD.

Kinetics-600 dataset as ID. Similarly, we adopt UCF101, EPIC-Kitchens, HAC, and HMDB51 datasets as OOD datasets, with careful curation undertaken to exclude samples belonging to ID classes. We carefully selected a subset of 229 action classes from Kinetics-600 in the Near-OOD setup to mitigate the potential overlap with other datasets. For the UCF101 dataset, we remove 11 overlapping classes with Kinetics-600, leaving 90 classes as OOD classes for evaluation. For other datasets, there are no class overlap issues, and we use their original classes as OOD.

(a) Energy

(e) Energy++

(c) KNN

(b) ASH

(d) LogitNorm

(h) LogitNorm++

(g) KNN++

(f) ASH++

(a) Embeddings without A2D and NP-Mix

(b) Embeddings with A2D and NP-Mix

C Visualization of Results

Visualization of Synthesized Outliers. We visualize the outliers generated by different outlier synthesis algorithms, including VOS [22], NPOS [60], Mixup [69], and our proposed NP-Mix . As shown in Fig. 7, VOS and NPOS generate outliers near the ID data, neglecting to explore the broader embedding space. Mixup randomly selects samples from all classes to mix and introduces unwanted noise samples within the distribution of ID data. NP-Mix excels at generating synthesized outliers by effectively utilizing information from neighboring classes and spanning wider embedding spaces.

Score Distributions. Fig. 8 illustrates the score distributions generated by various baseline methods on the HMDB51 25/26 dataset before and after training with A2D and NP-Mix . Score distributions produced by A2D and NP-Mix lead to better ID/OOD separation, resulting in strengthened OOD detection performance.

Visualization of Learned Embeddings for ID and OOD Data. Fig. 9 shows the visualization of the learned embeddings using t-SNE [46] on the HMDB51 25/26 dataset before and after training with A2D and NP-Mix . The embedding of ID and OOD data are more separable after A2D training and NP-Mix outlier synthesis.

D More Implementation Details

D.1 Training Details

We conduct experiments across three modalities: video, audio, and optical flow. We adopt the MMAction2 [11] toolkit for experiments. To encode the visual information, we utilize SlowFast network [23], initialized with pre-trained weights from Kinetics-400 [34]. For the audio encoder, we employ ResNet-18 [29] , initializing the weights from the VGGSound pre-trained checkpoint [10]. Similarly, we use the SlowFast network with a slow-only pathway, again leveraging pre-trained weights from Kinetics-400 [34] for the optical flow encoder. We use the Adam optimizer [36] with a learning rate of 0 . 0001 and a batch size of 16 . Additionally, we set the hyperparameters as follows: γ = 0 . 5 , mixup α = 10 . 0 , nearest neighbor N = 2 . We train the network for 50 epochs on an RTX 3090 GPU and select the model with the best performance on the validation dataset.

D.2 Extension to More Modalities

Our framework is not limited to two modalities and can be easily extended to M modalities. Given a training sample x with label c and M modalities, we obtain prediction probabilities ˆ p from the combined embeddings of all modalities, and ˆ p 1 , ˆ p 2 , ..., ˆ p M from each modality, all of which are of shape [1 , C ] , where C represents the number of classes. By removing the c -th value from each prediction, we derive new prediction probabilities without ground-truth classes, denoted as ¯ p 1 , ¯ p 2 , ..., ¯ p M with shapes [1 , C -1] . Subsequently, we aim to maximize the discrepancy between ¯ p 1 , ¯ p 2 , ..., ¯ p M , which can be defined as:

where Discr ( · ) is a distance metric quantifying the similarity between two probability distributions. Similarly, for L cls , L Discr _ syn and L Ent , we can define as:

The final loss is obtained as the weighted sum of the previously defined losses:

D.3 Inference Details

For algorithms that define OOD score or truncate activations leveraging information from the feature space (Mahalanobis [41], KNN [59], VIM [62], ReAct [58], ASH [16]), we use the combined embedding Z = [ Z 1 , Z 2 , ..., Z M ] from all modalities. For algorithms that define the OOD score leveraging information from the probability space or logit space (MSP [31], GEN [44], Energy [43], MaxLogit [30], VIM [62], LogitNorm [65]), we use the prediction probabilities ˆ p or prediction logits h ( Z ) obtained from the combined embeddings of all modalities.

E Further Experimental Results

Multimodal Far-OOD Detection. Tab. 7 and Tab. 8 present comprehensive results for Multimodal Far-OOD Detection on the HMDB51 and Kinetics-600 datasets. Training with A2D and NP-Mix significantly improves OOD performance in most cases across all baseline algorithms, underscoring the versatility of our proposed method.

Multimodal Near-OOD Detection with Other Combination of Modalities. We demonstrate the effectiveness of A2D and NP-Mix across various combinations of modalities, not limited to video and optical flow, as shown in Tab. 9. The performance of different baseline algorithms improves significantly with A2D training and NP-Mix outlier synthesis, regardless of whether the input modalities are video-audio, flow-audio, or video-audio-flow combinations.

F More Ablations

Compared with Other Multimodal Tasks. We compare A2D and NP-Mix with other multimodal self-supervised training tasks, including Contrastive Loss [35], Relative Norm Alignment (RNA) Loss [51], Cross-modal Distillation [27], and Cross-modal Translation [20], as shown in Tab. 10. While contrastive loss demonstrates effectiveness in enhancing OOD performance, other tasks significantly decrease the performance. A2D and NP-Mix show substantial superiority over other multimodal self-supervised tasks.

Influences of N and α in NP-Mix . In this section, we investigate the parameter sensitivity of NP-Mix on the HMDB51 25/26 dataset. For the Nearest Neighbor parameter N , we test values of 1 , 2 , 3 , and 4 . Regarding the Mixup parameter α , we evaluate values of 2 . 0 . 4 . 0 , and 10 . 0 . As shown

in Fig. 10 and Fig. 11, NP-Mix demonstrates robustness across different parameter settings and yields substantial enhancements in OOD performance for all cases.

Different Architectures. In this section, we demonstrate that the strong performance of training with A2D and NP-Mix is maintained across different network architectures. We replaced the architecture of the video backbone with Inflated 3D ConvNet (I3D) [7] and the optical flow backbone with Temporal Segment Network (TSN) [64]. As shown in Tab. 12, A2D and NP-Mix consistently deliver significant improvement in OOD detection performance with these new architectures. Furthermore, we replaced the architecture of the video backbone architecture with Video Swin Transformer (Swin-T) [45] and the audio backbone with Audio Spectrogram Transformer (AST) [25]. As illustrated in Tab. 13, A2D and NP-Mix also consistently achieve significant improvement in OOD detection performance using Transformer-based architectures.

Table 11: Ablation on the ensemble of different OOD scores on a single modality.

2

4

10

Mixup

α

30

32

34

36

38

40

42

44

46

FPR95

↓

Energy

Energy++

2

4

10

Mixup

α

86

87

88

89

90

91

92

AUROC

↑

Energy

Energy++

1

2

3

4

Nearest Neighbor

N

30

32

34

36

38

40

42

44

46

FPR95

↓

Energy

Energy++

1

2

3

4

Nearest Neighbor

N

86

87

88

89

90

91

92

AUROC

↑

Energy

Energy++

Ensemble of Multiple Unimodal OOD Methods. In this section, we add evaluations on HMDB51 25/26 for the ensemble of multiple unimodal OOD methods for each modality to demonstrate the importance of studying the multimodal OOD detection problem. We first evaluate the ensemble of different OOD scores on a single modality. We choose three scores for the ensemble: probability space (MSP), logit space (Energy), and feature space (Mahalanobis). For all scores, we normalize their values between 0 and 1 and calculate the

ensemble score as: score = α * score 1 + ( 1 -α ) * score 2 . For α , we do a grid search from 0 . 1 to 0 . 9 with a 0 . 1 interval and report the one with the best performance. As shown in Tab. 11, combining MSP or Energy with Mahalanobis can bring significant improvement, especially for video. However, there is still a large gap compared with our proposed solution, demonstrating the importance of studying the multimodal OOD detection problem. We then evaluate the ensemble of OOD scores on different modalities and calculate the ensemble score as: score = α * score video + ( 1 -α ) * score flow . For α , we also do a grid search from 0 . 1 to 0 . 9 with a 0 . 1 interval and report the one with the best performance. As shown in Tab. 14, combining more modalities always brings performance improvements, but still has a large gap compared with our proposed solution, further demonstrating the importance of studying multimodal OOD detection problems.

Robustness under Missing-modalities. In our framework, we train a classifier for each modality to get predictions from each modality separately. By default, we use the predictions obtained from the combined embeddings of all modalities to calculate the OOD score. However, when one modality is missing, we can use the predictions from the remaining modality to calculate the OOD score. We add evaluations on HMDB51 25/26 under this challenging condition and use Energy as the OOD score. As shown in Tab. 15, when one

modality is missing, the performance drops a little, especially in the case when the video is missing (A2D+NP-Mix (Flow)). However, compared with training on one modality alone (Video-only and Flow-only), training with A2D and NP-Mix can also bring significant improvements for each modality when another modality is missing. For example, A2D+NP-Mix (Video), the case when optical flow is missing, yields a 16 . 56% relative improvement on FPR95 compared with Video-only. This underscores the importance of cross-modal training for multimodal OOD detection.

Beyond Action Recognition Task. To further demonstrate the versatility of the proposed A2D training, we add another task on 3D semantic segmentation using LiDAR point cloud and RGB images. We evaluate on SemanticKITTI [3] dataset and set all vehicle classes as OOD classes. During training, we set the labels of OOD classes to void and ignore them. During inference, we aim to segment the known classes with high Intersection over Union (IoU) score, and detect OOD classes as unknown. We adopt three metrics for evaluation, including FPR95, AUROC, and mIOU c (mean Intersection over Union for known classes). We use ResNet-34 [29] and

SalsaNext [12] as the backbones of the camera stream and LiDAR stream. We compare our A2D with basic LiDAR-only and Late Fusion, as well as two multimodal 3D Semantic Segmentation baselines PMF [72] and XMUDA [33]. As shown in Tab. 16, our A2D also demonstrates strong performance under this new task (3D Semantic Segmentation) with different combinations of modalities (LiDAR point cloud and RGB images).

G Statistical Significance Tests

Different Random Seeds. We run each experiment three times using different seeds for Multimodal Near-OOD Detection on the HMDB51 25/26 dataset in our MultiOOD benchmark, and then calculate the mean AUROC and FPR95 to demonstrate the statistical significance of our methods. As shown in Fig. 12, training with A2D and NP-Mix is statistically stable and significantly surpasses the baselines under different random seeds.

Different Random Splits. We run each experiment five times using different dataset splits for Multimodal Near-OOD Detection on the HMDB51 25/26 dataset in our MultiOOD benchmark, and then calculate the mean AUROC and FPR95 to demonstrate the statistical significance of our methods. As shown in Fig. 13, training with A2D and NP-Mix is statistically stable and surpasses the baselines significantly under different dataset splits.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction clearly state the claims made.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the conclusion part, we discussed limitations and future work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our paper discloses all the information needed to reproduce the main experimental results. Our source code and benchmark datasets are provided in Supplementary Material and will also be made publicly available.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code in Supplementary Material to reproduce the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all the training and test details in the paper and in the provided code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide Statistical Significance Tests in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide details in implementation details part.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: They are properly credited in our paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The new assets introduced in the paper are well documented.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first proposes a MultiOOD benchmark for multimodal OOD detection. It then further proposes methods including A2D and NP-Mix for better tacking the multimodal OOD detection task.\n\nSTRENGTHS:\n1. I appreciate the formulation of the multimodal OOD detection benchmark.\n\n2. This work presents extensive experiments and achieves good results.\n\nWEAKNESSES:\n(See the questions for more details.)\n\nQUESTIONS:\nOverall, I am now hesitating around the borderline for this paper and below are my concerns:\n\n1. Generally, I believe that the relationship between this work and other multi-modality works can be very close. I thus suggest the authors to also discuss with other multi-modality works in their related works. I will also elaborate on this more in below concerns.\n\n2. As for the benchmark, I believe that it is important for the evaluation to be conducted in a more statistically significant manner. In other words, from my perspective, the authors may consider to at least evaluate methods on different ID/OOD category splits and report standard deviation.\n\n3. W.r.t. A2D in Sec 4.2, from my perspective, it seems that sometimes, the orders of non-ground-truth classes matter. For example, while for a cat input, both fox and truck are non-ground-truth classes, it seems natural for fox to have a higher softmax score than truck. Thus, I am curious, what if the discrepancy is maximized in an order-preserved manner. I appreciate if this can be performed as an ablation study to see whether this alternative of A2D can be more effective.\n\n4. W.r.t. Sec 4.3, the authors claim that they get inspiration from [16]. I thus hope to see a more detailed comparison between the proposed method and the method in [16]. Besides, it seems to also worth comparing with Learning Placeholders for Open-Set Recognition CVPR 2021. If I am not wrong, it seems that the proposed NP-Mix is somehow similar to its data placeholder.\n\n5. As for the evaluation, as mentioned in the first concern, this work is closedly related to those multi-modaity ones. Thus, to better validate the efficacy of A2D and NP-Mix, besides comparing with only uni-modal OOD methods, it is also suggeted to create baselines that combine uni-modal OOD methods and existing typical multi-modality methods outside the OOD area. This can lead the proposed method to be better evaluated.\n\nLIMITATIONS:\nThis work has discussed its potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MultiOOD, a new benchmark for multimodal out-of-distribution (OOD) detection. The authors propose two new techniques: 1) Agree-to-Disagree (A2D), which encourages discrepancy between modality predictions during training, and 2) NP-Mix, a novel outlier synthesis method. Extensive experiments on the MultiOOD benchmark demonstrate significant improvements over existing unimodal OOD detection methods.\n\nSTRENGTHS:\nOriginality:\n-First benchmark for multimodal OOD detection (MultiOOD).\n-A2D algorithm leveraging modality prediction discrepancy.\n-Outlier synthesis method (NP-Mix).\n\nQuality:  \n-Strong performance improvements over baselines.\n-Thoughtful analysis of modality prediction discrepancy phenomenon.\n-Comprehensive experiments on diverse datasets.\n\nClarity:  \n-Well-organized structure.\n-Clear explanations of key concepts and methods.\n\nWEAKNESSES:\nWeaknesses:\n1) Lacks important baselines (e.g. ensemble of multiple singleOOD methods for each modality). This also relates to the question of \"why we should study MultiOOD?\" The importance of studying MultiOOD and its practical impact on real-world applications could be further demonstrated.\n2) Limited theoretical analysis of proposed methods. A2D and NP-Mix need more theoretical analysis. \n3) The experiments are only conducted on the task of action recognition. In this case, the title \"MultiOOD\" seems over-claiming. Please specify the task in the title and the main text. In addition, it only focuses on specific modalities (video, optical flow, audio). This makes the scope of the paper somewhat limited.\n5) Some figures (e.g. Fig. 4) could be improved for clarity, with consistent color-coding and same x-axis scales.\n\nQUESTIONS:\n1) Why \"α > 1\" can ensure the synthesized outliers reside in the intermediary space between two prototypes, rather than near the ID data? Please give more explanations.\n2) How sensitive are the A2D and NP-Mix methods to hyperparameter choices?\n\nLIMITATIONS:\nThe authors acknowledge some limitations, such as the focus on specific modalities (video, optical flow, audio) and action recognition tasks. However, they could further discuss potential limitations in scaling to a larger number of modalities or very different types of data.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel OOD detection benchmark for multimodal data (called MultiOOD), covering diverse dataset sizes and modalities. Based on this benchmark, the authors first demonstrate the Modality Prediction Discrepancy phenomenon, which means that \nthe discrepancies of softmax predictions are shown to be negligible for in-distribution data (across different modalities) and significant for OOD data. Based on these observations, the authors introduce a novel Agree-to-Disagree (A2D) algorithm, which aims to enhance\nsuch discrepancies during training. Additionally, the authors propose a new outlier synthesis algorithm NP-Mix that explores broader feature spaces and complements A2D to strengthen the OOD detection performance. Experimental validation is extensive and confirms the improvements in OOD detection due to the new proposed algorithms.\n\nSTRENGTHS:\nPaper's strengths:\n- the paper has a significant scientific contribution, by introducing the first multi-modal benchmark for OOD detection\n- besides this, it contributes to the improvement of OOD detection by introducing two new algorithms:  A2D algorithm and a new outlier synthesis algorithm NP-Mix that explores broader feature spaces and complements A2D\n- the paper is clearly written and well-documented\n- the review of the state of the art is comprehensive and covers most of the relevant work\n- experimental validation is extensive and underline the improvements introduced by the A2D and NP-Mix algorithms for OOD detection\n\nWEAKNESSES:\n- Some more details are required regarding some aspects.\n\nQUESTIONS:\nHere are my concerns:\n- In the video modality, what features do you extract to characterize the stream? Do you extract per-frame features, or per-video features.\n- Same question for the audio modality\n- You use 5 datasets, all of them having the video and optical flow modalities, but audio modality is missing from 2 of them. In this case, what protocol do you adopt? Do you ignore them and perform the audio analysis only on the remaining three?\n- Is your approach robust when one modality is missing?\n\nLIMITATIONS:\nThe identified limitations are: (i) the performance on Near-OOD benchmark with a large number of classes can be further improved; and (ii) the ID/OOD discrepancy could also be further improved. The paper does not represent any negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-07"}