MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities
Authors: Hao Dong
Yue Zhao
Eleni Chatzi
Olga Fink
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Out-of-Distribution Detection
Multimodal Learning
Primary area: safety_in_machine_learning
TL;DR: We introduce a novel benchmark MultiOOD for more realistic Multimodal OOD Detection, as well as an Agree-and-Disagree (A2D) algorithm and an NP-Mix outlier synthesis method to improve Multimodal OOD Detection performances.

Abstract: Detecting out-of-distribution (OOD) samples is important for deploying machine learning models in safety-critical applications such as autonomous driving and robot-assisted surgery. Existing research has mainly focused on unimodal scenarios on image data. However, real-world applications are inherently multimodal, which makes it essential to leverage information from multiple modalities to enhance the efficacy of OOD detection. To establish a foundation for more realistic Multimodal OOD Detection, we introduce the first-of-its-kind benchmark, MultiOOD, characterized by diverse dataset sizes and varying modality combinations. We first evaluate existing unimodal OOD detection algorithms on MultiOOD, observing that the mere inclusion of additional modalities yields substantial improvements. This underscores the importance of utilizing multiple modalities for OOD detection. Based on the observation of Modality Prediction Discrepancy between in-distribution (ID) and OOD data, and its strong correlation with OOD performance, we propose the Agree-to-Disagree (A2D) algorithm to encourage such discrepancy during training. Moreover, we introduce a novel outlier synthesis method, NP-Mix, which explores broader feature spaces by leveraging the information from nearest neighbor classes and complements A2D to strengthen OOD detection performance. Extensive experiments on MultiOOD demonstrate that training with A2D and NP-Mix improves existing OOD detection algorithms by a large margin. To support accessibility and reproducibility, our source code and MultiOOD benchmark are available at https://github.com/donghao51/MultiOOD.

MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities
Hao Dong'
Yue Zhao?
Eleni Chatzi!
Olga Fink
'ETH Zürich
University Southern California 9EPFL.
(hao.dong, chatzi]@ibk. baug. ethz.ch, yzhao010@usc. olga.fink@epfl..
Abstract
Detecting out-of-distribution (OOD) samples is important for deploying machine learning models safety-critical applications such as autonomous driving and robot-assisted surgery. Existing research has mainly focused unimodal scenarios on image data. However, real-world applications are inherently multimodal, which makes it essential leverage information from multiple modalities enhance the efficacy of OOD detection. To establish a foundation for more realistic Multimodal introduce the first-of-its-kind benchmark, MultiOOD, char-
OOD Detection, acterized by diverse dataset sizes and varying modality combinations. We first evaluate existing unimodal OOD detection algorithms MultiOOD, observing that the mere inclusion of additional modalities yields substantial improvements. This underscores the importance of utilizing multiple modalities for OOD de- tection. Based on the observation of Modality Prediction Discrepancy between m-distribution (ID) and OOD data, and its strong correlation with OOD perfor- mance, propose the
(A2D) algorithm to encourage such discrepancy during training. Moreover, we introduce a novel outlier synthesis method, NP-Mix, which explores broader feature spaces by leveraging the infor- mation from nearest neighbor classes and complements A2D to strengthen OOD detection performance. Extensive experiments on MultiOOD demonstrate that training with A2D and NP-Mix improves existing OOD detection algorithms by large margin. To support accessibility and reproducibility, our source code and Martoogghhho///MOOOOO00 I/MultiOD.
MultiOOD benchmark are available at
Introduction
Most existing machine learning (ML) models are trained under the closed-world assumption, where the test data is assumed to be drawn i.i.d. from the same distribution as the training data, referred as in-distribution (ID). However, in open-world scenarios, (OOD), thus impacting model robustness and safety [67]. OOD detection aims to detect samples with semantic shifts that are undesirable for the model generalize [67] and is critical for deploying ML models in crittcci domains such as autonomous driving [21]. robotics [18, 4], and diagnostics critical assets [24]. Numerous OOD detection algorithms have been developed, ranging from plassficattion-base to distance-based methods [66]. Classification-basee methods typically derive confidence directly from the classifier, employing post-hoc processing techniques such as Maximum Softmax Probability (MSP) [31] and Energy [43] or training strategies such logit normalization [65] and outlier synthesis [22]. Distance- based methods typically measure distances in high- dimensionnl feature spaces distinguish between ID and OOD [41, 59]. Additionally, other methodologics explore density estimation [1, 50] and reconstruction techniques [70] for OOD detection.
samples can be out-of-distribution
Current research in OOD detection has predominantly focused unimodal settings, often involving images as inputs [66]. While several recent works [47, 63] have investigated vision- vasion-language
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

96
90
Vidoo
Video
RPR95
70-
Vidiose Flow
85
Ville- Flow
60-
80
75
50-
40
MSP
Enssgy
ReAct
ASU
MSP
Ebergy
ReAct
ASII
Figure 1: The FPR95 (lower is better) and AUROC (higher is better) HMDB51 dataset across
various modalities. Multimodal OOD substantially improves unimodal OOD w/o bells and whistles.
models [53] to enhance OOD performance, their evaluations are still limited benchmarks containing solely images. Consequently, existing methods fall short in fully leveraging the complementary information from various modalities, such as LiDDAR and camera in autonomous driving [21], as well as video, audio, and optical flow in action recognition [56]. To underscore the importance of using multiple modalities various modalities on the HMDB51 [39] dataset within our MultiOOD benchmark. This is an action recognition task and all models are trained solely using cross-entropy loss between a one-hot target vector and the softmax output. Results in Fig. 1 show that even a simple fusion of video and optical flow modalities can substantially enhance OOD detection performance.
OOD detection, we evaluate representative OOD algorithms across
To establish a foundation for more realistic Multimodal OOD Detection, we introduce novel OOD benchmark named MultiOOD (Fig. 2), which the first benchmark for Multimodal OOD Detection and covers diverse dataset sizes and modalities. MultiOOD comprises five video datasets with over 85, 000 video clips in total. The datasets vary in the number classes, ranging from 7 to 229, and in size, spanning from 3k to 57k. Video, optical flow, and audio are used as different types of modalities. While most existing unimodal OOD algorithms designed for images can be directly applied to Multimodal OOD Detection, such approaches may yield suboptimal results without accounting for the interaction and complementary nature of diverse modalities. As depicted in Fig. 1, the AUROC performance is very close for all unimodal baselines, underscoring the importance of developing OOD detection algorithms tailored to effectively exploit information from multiple modalities.
In this work, we first identify and illustrate the Modality Prediction Discrepancy phenomenon on the MultiOOD benchmark, where the discrepancies of softmax predictions across different modalities are shown to be negligible for ID data while significant for OOD data (Fig. 3). We discover a strong correlation between such discrepancies and the OOD detection performance (Fig. 4). Motivated by these observations, we introduce the Agree-to-Disagree (A2D) algorithm, which aims enhance such discrepancies during training. The algorithm designed so that different modalities should Agree the prediction of the ground- -ruth class, and Disagree other classes by maximizing the distance between their predictions. Additionally, we propose a novel outlier synthesis algorithm named NP-Mix. designed to use the information from nearest neighbor classes to explore broader feature spaces and complement A2D to strengthen the OOD detection performance.
Extensive experiments on the MultiOOD benchmark demonstrate the superiority of A21) and NP-Mix. The integration of A2D and NP-Mix yields substantial performance enhancements over existing unimodal OOD detection algorithms. For instance, on the UCF101 [57] dataset within MultiOOD, our approach reduces the FPR95 from 32.14% to 10.68% for ASH [16] method, representing a noteworthy absolute 21. 21% improvement over the baseline. Our contributions include:
We highlight the significance of integrating more modalities for OOD detection and introduce MultiOOD, the firs/ benchmark for Multimodal OOD Detection encompassing diverse dataset sizes and various combinations of modalities.
We conduct comprehensive evaluations of existing unimodal OOD algorithms on MultiOOD, revealing their limitations in multimodal scenarios.
We propose novel A2D training algorithm, inspired by the observation of the Modality Prediction Discrepancy phenomenon, alongside a new outlier synthesis algorithm NP-Mix that explores broader feature spaces and complements A2/) strengthen the OOD detection performance.
Extensive experiments conducted on MultiOOD underscore the effectiveness of A2D and NP-Mix. Our source code and MultiOOD benchmark will be made publicly available, facilitating future research endeavors in Multimodal OOD Detection.

---

4x Multimodal Near-OOD benchmark
Multimodal Far-OOD benchmark
Optical
Audio
HNDBSI Kimedles-600
4'4
25/26
3W/11
129/100
An-Distributtin
In- Distribiibbtn
Kinetlcs-600
Nomber of videce
calchh
clle
Fencing
400
Chil-df-ratribatiion
300
goll
justop
laugh
UCPPIE
Out-uf- Distribution
HMDBS1
UCF101
16k
IMMERS:
pallup
punch
situp
somerssult
EPIC Kitchens
HAC
51
101
229
Number clissos
Figure 2: An overview of MultiOOD Benchmark.
2 Preliminaries: Multimodal Out-of-distributiio Detection
Multimodal OOD Detection aims to detect samples with semantic shifts using multiple modalities. We consider training set aa... {xx; v)) drawn i.i.d. from the joint data distribution Pxy. where is the input space and y 21, C} the discrete label space. Each training sample X; comprised of M modalities, denoted as x; = {at = 1, ....... X' and f: X R be a neural network trained on samples in Pxy that predicts the label of each input sample. The f in Multimodal OOD Detection comprises of M feature extractors (+x(-) and classifier h(-). Each feature extractor g&(-) extracts an embedding Z for its corresponding modality k, and the classifier h(-) takes the combined embeddings from all modalities as input and outputs a prediction probability p::
M)- Let Pnn denote the
marginal distribution
ý (x(())
5(M((Z) I²))))
(1)
where 8(-) is the softmax function. We further include separate classifier ha(-) ()) for each modality k to get predictions from each modality separately, with the prediction probability from the k-th modality as 5a (x")))(<z)))
When deploying f in the real world, should not only accurately classify known samples as ID, but also identify any "unknown" sample OOD. A separate score function S(x) is often used to decide whether a sample x E X' is from Pin (ID) or not (OOD):
ID
S(x)
OOD
S(x) < n)
where samples with higher scores S(x) are classified as ID and vice versa, is the threshold. Existing OOD detection studies predominantly focus on unimodal scenarios, with detailed literature review offered in Appendix A. To establish a foundation for more realistic Multimodal OOD Detection, we introduce novel MultiOOD benchmark (Sec. 3) and propose an effective multimodal training strategy (Sec. 4) that yields significant enhancements over existing unimodal approaches.
3 MultiOOD Benchmark
We create the MultiOOD benchmark to understand the existing gap in Multimodal OOD Detection research. OOD detection primarily focuses detecting semantic shifts, with main approaches used for constructing OOD benchmarks. A common method involves considering an entire dataset as in-distribution (ID) and further collects datasets, which comprise similar tasks but are disconnected from any ID categories, as OOD datasets. In this scenario, both semantic and domain shifts are present between ID and OOD samples. We term this setup as Far-OOD in our benchmark. Another approach is to partition the categories of existing datasets into two subsets, referred to as closed (ID) and open set (OOD). Here, both ID and OOD samples originate from the same distribution, with only semantic shifts existing between them. We denote this setup Near- OOD within our benchmark;

---

L Distance: 0.16
L Distamce: 1,90
Vidno Profecien
Vióro Predictio
Viox Predictoc
0,5
Foo Prederana
0.3
Probahitty
Probebiitty
0.4
0.6
0.2
0.2
0.1
10
15
20
25
10
15
20
25
Class Label
Class Labsl
(a) Prediction for ID data
(b) Prediction for ODD data
Figure 3: An example of softmax outputs for ID and OOD data. The predictions from video and
optical flow demonstrate uniformity across ID data and exhibit variability actoss OOD data.
setup that poses greater challenges compared to Far-OOD. This setup is also referred to open set recognition (OSR) in some studies [61, 37, 9]. Notably, OSR and OOD detection both share the same goal of identifying test samples with semantic shifts without compromising the accuracy of ID classification [67]. In our benchmark, we treat OSR and OOD as synonymous concepts and adopt OOD the general term. MultiOOD comprises five action recognition datasets LEPIC-Kitchens [48], HAC [20], HMDB51 [39], UCF101 [57], and Kinetics- 600 [6]) with over 85, 000 video clips in total. The datasets vary in the number of classes, ranging from to 229, and in size, spanning from 3k to 57k. Video, optical flow, and audio are used as different types of modalities. An overview of the MultiOOD benchmark provided in Fig. 2, with additional details available in Appendix B.
3.1 Multimodal Near-OOD Benchmark
In the Near-OOl setup, we include four datasets. EPIC-Kitchens 4/4 is derived from the EPIC- Kitchens Domain Adaptation dataset [48], where the dataset is partitioned into four classes for training as ID and four classes for testing as OOD, with total of 4, 871 video clips. Similarly, HMDB51 25/26 and UCF101 50/51 are constructed based on HMDB51 [39] and UCF101 [57]. with total of 6, 766 and 13, 320 video clips respectively. In the case of Kinetics-600 129/100, we select 229 action classes from the Kinetics-600 dataset [6], with each class comprising approximately 250 video clips and total of 57, 205 video clips. Within this setup, 129 classes are designated for training as ID, while the remaining 100 classes are allocated for testing as OOD.
3.2 Multimodal Far-OOD Benehmark
In the Far-OOD setup, include HMDB51 and Kinetics-600 as ID datasets.
HMDB51 dataset as ID. For the OOD datasets, we utilize UCF101, EPiC-Kitchens, HAC, and Kinetics-600 datasets. All of these datasets are carefully curated to remove samples that belong to ID classes in HMDB51. Given the relatively small number of classes the EPIC-Kitcher and HAC datasets, remove classes in the HMDB51 dataset that overlap with EPIC-Kitchens and HAC, with 43 classes left as ID classes. For UCF101, we remove 31 overlapping classes with HMDB51, resulting in 70 classes designated as OOD classes for evaluation. For other datasets, class overlap exists and we utilize their original classes as OOD.
Kinetics-600 dataset as ID. Similarly, we adopt UCF101, EPIC-Kitchens, HAC, and HMDB51
datasets as OOD datasets, with careful selection undertaken to exclude samples belonging to ID classes in Kinetics-600. We carefully selected a subset of 229 action classes from Kinetics-600 in the Near-OOD setup to mitigate the potential overlap with other datasets. For the UCF101 dataset, we remove 11 overlapping classes with Kinetics-600, leaving 90 classes as OOD classes for evaluation. For other datasets, there are no class overlap issues and we use their original classes as OOD.
4 Methodology
In this section, we first identify the Modality Prediction Discrepancy phenomenon on the MultiOOD benchmark and demonstrate its substantial correlation with the OOD detection performance (See. 4.1). Subsequently, introduce the Agree-to- to-Dsaaree (A2D) algorithm aimed at enhancing such discrep-

---

100
100
IDD35
95
0OD Performazos (AUROC f)
VCP10]
0OD Pedlarmnii (AUROC
SPIC Kiirbssa
EPIC-Kidder
Kinctis 100
Kaclics-00
90
IDDD51+A20
UCPI01-AID
855
85
KPtc-Kiidura----
Kaclics-0D A22)
80-
75-
75
63
65
0.4 0.6
0.8
0.4 0.6 0.8
1.0
Average Prodiction L1 Distamees (before A2D training)
Average Prodbction 1.1 Distamoes (afte A2D training)
Figure 4: Average prediction L1 distances between ID and OOD data (lOOD ITD) before and after A2D training across various datasets within the MultiOOD, where Energy [43] is used as score function. The distances highly correlated to the ultimate OOD performance. A2D training amplifies such distances, consequently enhancing the efficacy of OOD detection.
ancies during training (Sec. 4.2). Finally, we propose the novel outlier synthesis algorithm named NP-Mix that complements A2D to further strengthen the OOD detection performance (Sec. 4.3).
4.1 Modality Prediction Discrepancy
We first explore the predictive behaviors demonstrated by various modalities when using both ID and OOD data as input on MultiOOD, We compute the prediction probabilities employing classifiers for video and optical flow on the same II) and OOD samples and calculate their T,, distances. As depicted Fig. 3, for ID data, the prediction probabilities of both video g² and optical flow p² are generally exhibited consistent with each other on the ground-truth label, consequently yielding small L1 distance |p= - p² || between them. Conversely, for OOD data, each modality tends to express varying confidence preferences towards distinct classes, resulting in notable increase in the L1 distance between their predictions. We refer to this phenomenon as Modality Prediction Discrepancy between ID and OOD data. This discrepancy can be attributed the unavailability of semantic information on OOD data during model training, stimulating each modality to generate conjectures based its unique characteristics upon encountering OOD data during testing.
To verify the universality of this phenomenon, we calculate the average I.₁ distance between predic- tions of video 6² and optical flow ø² on both ID and OOD data within the HMDB51 dataset. The average prediction L1 distance is 0.63 for ID data (l)p) and 1.42 for OOD data (looD), revealing a substantial discrepancy. In addition, MultiOOD benchmark and observe similar discrepancies (loOD - LID). Such discrepancies have a positive correlation with the OOD detection performance, as illustrated in Fig. 4.
evaluate other datasets in the Near-OOD setup within the
4.2 Agree-to-Disageee Algorithm
Motivated by the Modality Prediction Discrepancy and its strong correlation with Multimodal OOD Detection performance, introduce the perre-to-Disagres (A2D) algorithm to foster the amplification of such discrepancies during training. The underlying idea that different modalities should Agree the prediction regarding the ground-truth class, while they should Disagree the remaining classes by maximizing their prediction distance. A2D enables the model to diversify predictions across modalities, consequently yielding high prediction discrepancies for OOD data during testing.
Given training sample with label c. we obtain prediction probabilities ý from the combined embeddings of all modalities, and p² ji from individual modality, all of which are shape [1, C), where C represents the number of classes. By removing the c-th value from f and 1 (different modalities should Agree on the prediction regarding the ground-trutt -ruth class), we derive new prediction probabilities without ground-trute classes, denoted pt and 8² with shapes [1, C - 1]. Subsequently, we aim to maximize the discrepancy between p² and j², which can be defined as:
D -Discr pp)²)
(2)
5

---

h1
8
B
91
Z
Modolay
h
Di
AED
i
Trainng
92
Z.
h2
Modalin
Agree-to-Disse DoDiggreee (A2D)
Outlier Syuthesis Using NP-Mix
F
Egat D
Prototypes Ze
Nearest Neighber Z.
Synthesized Outliers Z
2
Figure 5: An overview the proposed framework for Multimodal OOD Detection. We introduce A2D algorithm to encourage enlarging the prediction discrepancy across modalities, Additionally, we propose novel outlier synthesis algorithm, NP-Mix, designed to explore broader feature spaces, which complements A2D to strengthen the OOD detection performance.
where Discr(-) a distance metric quantifying the similarity between two probability distributions. We utilize the Hellinger distance [49] and explore the efficacy of alternative distance metrics in ablation study. The Hellinger distance between two probability distributions is defined as:
D
Σ
P
p
Furthermore, ensure that the ground-truth classes possess the highest probabilities, incorporate the cross-entropy loss C-)(-) for each prediction, defined as:
Cats (CF(p.)) CF .) CF((2,)))
The final loss obtained the weighted sum of the previously defined losses:
[ Cats Discrs
(5)
where the hyperparameter controls the relative importance of the discrepancy term.
4.3 Nearest Neighbor Prototype-based Mixup for Outlier Synthesis
Outlier synthesis [22, 60] has demonstrated its efficacy in OOD detection by imposing regularization on the model's decision boundary during training. Introducing the discrepancy loss in A21) to the synthesized outlier data for model regularization has the potential to further enhance OOD detection performance. However, existing outlier synthesis methods [22, 60] typically generate outliers near the ID data (Fig. 7), neglecting to explore the broader embedding spaces, thereby potentially leading to suboptimal performance, Inspired by the recent approach introduced in [19], we introduce a novel algorithm termed Nearest Neighbor Prototype-basee Mixup (NP-Mix), aimed synthesizing outliers capable of spanning wider embedding spaces by leveraging the information from nearest neighbor classes, as shown in Fig. and Fig. 7. To synthesize outliers, we concatenate the embeddings of all modalities (Z = I'' 2 and treat them as unified entity. Subsequently, we compute prototype embedding åe for each class by calculating the mean of all embeddings within that class. For each prototype embedding 2. we identify its top N nearest neighbor prototypes and randomly select one prototype 2e from them for mixing. Two samples, Z₁ and Zz, randomly chosen from class and class respectively. The outlier Z is generated by their convex combination:
Ž XZ1 + (1 - A)Z2,

---

Table 1: Multimodal Near-OOD Detection using video and optical flow. indicates larger values better and vice versa. Training with A2D and NP-Mix yields considerable performance enhancements.
Metholls
HENDHSI 25/26
UCF101 58/51
EPIC- 4/4
Kinetics 60D 12W100
AUHOOCY
1D ACU
1PR9N
AURUL
10 DCACT
PURSS
AURDE?
10 ACC
19RR95
AUROC,
IDP ACC
Without A20 Training
MSP
44.05
87.24
69.32
22.14
59.22
16.31
67.59
71.46
66.05
75.16
90.11
Energy
455
69-32
22.92
96.05
76 68
71.40
08:25
75.05
90.1 11
MasLogii
76.68
75.05
Mabulanobis
99.22
71.46
Redlct
09.22
76.87
70.15
23.79
09.22
76.87
71.46
KNN
99.22
75.95
63.60
71.46
VIM
96.83
59.32
12.52
97.65
59.22
72.05
68.00
71.46
75.47
90.11
LagitNurm
48.84
87.65
39
19661
95.85
99.51
30.97
63.11
71.83
67.32
75.84
80.23
With
A2D Training
MSP+
38.78_2.48
58.3
08.37.440
50.63
7.00-11.00
18. 19...4
99.90
00
11.04.0.45
71:46
13-00-1.33
70.47 70.47
70.50
Energy+
59.22 4.15
58.55
00.63
9.51
18.
09.61
19 05.98
F2:45
71:46
61.50 4.10
70:45 10.5
70.50
Manilcgpis
59.22
58.50
00.65
9.51
18.15
09.61
05.08
72.25
71.46
64.57
76, 02 you
70.50
Matalonneise
40.88.4.0v
56.30
90.65
6.74 6.74--444
58.007
99.51
00.52
44.43....1
71.46
92.55
50.00....0
79.50
Refute
57.91-4.14
50.00 11.00
50.6
10:30 10:39--5334
11.34
18.122.5.7
59.51
08 11:30
72.00....4
00.52
71.25. 1:14
74 18.8)....
79.64
ASI
22405 11:14
50.41
12.02 -115
631 11:00
74.73... mm
67:02
644
4.90
77 18 1117
79.15
2:50
mm
90.63
8,53 149
26.12 mas
65.56 1:
T3,05 1.
71.46
02.35
6:70
77 mm
79.50
$0.331-..90
23.1
90.63
--45-111
16.20-1.45
99:51
72.08-2244
07,83.1.11
71.46
05.71-2.90
79.50
52.77-2.455
+1.31
8,90-171
43-2.31
71.46
64.20-4.15
79.50
LagitNueme
6.32
12.03_1135
1172
07.00_1.35
77.90....0
With AID Training NP-Mixx Outlier Svatheds
MSP-+
13.99-19.47
98.24.2.51
59.71
T1.52-2.11
71.64
Enszgy+s
50.38
58.91
$9.98
0.50
18.15
09.71
07.91
13.79
71.64
63.60
77.11
90.53
MasLogiiiss
56.58
50.06,
$9.98
0.50-
18.60.2.47
09.71
06.05
T3.65 13.65....1
71.64
63.65
77 77.50
90.52
Mahslansiiss
41.61-1.30
57.60.2.1
$9.98
7,86 7.86-128
97.50..as
99.71
D4.75-
64.37-1.31
T1.64
92.69-1.22
5000--15
90.52
ReActee
S7.47
59.63
50.20
12:43 1510
59
00.00
11:1l1777
72.19
17.00
75.55
90.70
ASS
36.17
50.30 1114
59-32
1008
(0.50 1une
09:81
60:23
T1.00 1.33
66:33
13:77
8.47
77.44
70-44
GEN++
$5.96 1:4
50.78
06
7.67
[8-96
09.71
65.50
T5.17
71.64
62.05 6.36
70. 96.96
80.53
KNN++
53.77-6415
90,85.3.90
59.98
12.04.3.99
97.663.4722
99.71
71.27 71.27-041 68
09.8...34
71.04
06,81-1.9
74.19-c..
80.52
VIM--
34.64-2.16
58.80.9.74
39.98
1.56-799 7:9
18.771.177
59:71
67.72-931
71.64
63.67-8.0 8830
77.12 27.32
80.52
LogiNNem--
36:35 12
20.38 25.75
39.36
9.81 .8
16.14,139
59.71
72.01 9.91
72.0
71.46
03.02 4.99
77.57
90.33
where ~ Beta(a, a), for a E (0, In our experiments, we opt for a > 1 to ensure the synthesized outliers reside in the intermediary space between two prototypes, rather than near the ID data. We then partition Z into different modalities as Z Z z²), where Zt and Z represent the synthesized outlier embeddings for each modality. Subsequently, the corresponding prediction probabilities are computed as pd 5(h1 (Z))) and pd 8(h₂( (z²)
For synthesized outliers, we aim to maximize the prediction discrepancies between different modali- ties, similar to Eq. (2) for ID training samples. In this context, there is no need to remove any value from the prediction, as the outliers are assumed not to belong to any ID class. The discrepancy loss between p' and p" can be defined as:
Disser_syn = -Dicrr (p)
Moreover, we seek to mitigate the overconfidence of predictions for synthesized outliers towards any existing ID class. Therefore, we maximize the entropy of predictions:
LE - (HG))+ (
(8)
where (p) is the entropy of prediction p and can be calculated as H()) = - -. Po log The final loss is obtained as the weighted sum of the previously defined losses:
L Lets + (L + L DDiser Din syn + L
Experiments
5.1 Experimental Setting
Baselines. Our objective is to enhance existing unimodal OOD algorithms through multimodal training utilizing the proposed A2D and NP-Mix. To validate the efficacy and versatility of our proposed approaches, we include representative algorithms that leverage information from the probability space (MSP [31], GEN [44]), logit space (Energy [43], MaxLogit [30]), feature space (Mahalanobis [41], KNN [59]), both logit and feature spaces (VIM [62]). penultimate activation manipulations (ReAct [58], ASH [16)), and training-time regularization (LogitNorm (65]).
Evaluation metrics. We evaluate the performance via the use of the following metrics: (1) the false positive rate (FPR95) of OOD samples when the true positive rate of ID samples is at 95%, (2) the
7

---

Table 2: Multimodal Far-OOD Detection using video and optical flow, with HMDB51 as ID.
Datasets
Methods
Kinetics-6d0 600
UCF101
EPIC-
HAC
Average
ACC1
FPR95L
AUROCT
FPRO5
AUROC!
FPR9S]
AUROCT
FPR95L
AUROCI
FPR95.
AUROCI
Without A2D Training
Energy
32.95
02.48
44.91
57.95
8.10
97.70
32.95
92.28
29.73
92.60
87.23
ASH
51.20
$7.81
53.93
84.22
19.95
95,92
42.99
90.25
42,02
89.55
$6.20
GEN
41.51
46.18
87.91
91.28
33.55
91.
87.23
KNN
39.34
57.33
VIM
13.68
97.01
33.87
91:45
5,93
13:45
97.12
16.73
57.23
With A2D Training and NP-Mix Outlier Synthesis
Energy+
24.52 1.01
99.96
36
89:07
-1:99
1.966-1111
97.53 n.
22.92 10.15
94.41-9.1
22.72 0.
50.80.,1.9
$6.89
ASCI+
27.82
90.17
89
6.84-15311
08 25 50
25.05
94.35 94.35-4122
24.00
53. $3.8444429
$6.20
GEN+
25,06
55L501
37.40-
91.19
6.20--200
24.63-
94.28
20.24
544 94..49-2.54
56.89
KNN-+
15.05-744
96.96
$3.06
91.92
6.47
08.97
13:46-140
16.76
96
$6.89
VIM++
9:24
25.45-7.4
92.34
6.306...2
98.09-..0
6.04.7.4
98.56.1.44
11.77 6.0
96.70.3.41
$6.89
Table 3: Multimodal Far-OOD Detection using video and optical flow, with Kineties-600 as II.
OOD Datasets
Methods
HMDRSI
UCF101
EPIC-Kiiehle Kitchens
HAC
Average
TD ACC
FPRB5
ALROCT
FPR951
AUROCT
FFR951
AUROC1
FFR95L
AUROC1
FPR951
AUROCT
Without AID Training
Energy
$2.05
61.50
ASH
71,62
76.66
72.38
72.20
GFN
68.47
78.43
73.97
83.67
$4.90
80.30
73:14
KNN
71.08
78.84
68.62
74.33
41.83
K2.32
57.00
82.53
59.63
79.51
73.14
VIM
72.25
71.88
70.72
70.58
43.14
$2.69
59.48
75:46
61.40
75.15
73.14
With A2D Training and NP-Mix Outlier Synthesds
Enery++
63.27
74.17
67.20
71.50
34.07
87 67.49...
56 60
65.91
70.09,410
73.67
AS
663.65-7.55
79.14.2.40
62.15-77.2
70.322
36.0811.71
46.47
85.21+1.77
52.09-5771
82.09.1.4
72.53
GEN++
80.63.2.2 22
67.09
77.97-10
40.00_125
65.10-caa
41.97-122
96.98.5.51
50.19-4.1
82.07 2237
73.67
KNN-+
1:40
67.44-1.4
77.95-3.22
37.65
-1.55-1.11
53.63-1.37
65.51,008)
55.83-23.6
1.77
73.67
VIM++
63.25 9m0
75. (2
66.31 4:01
74.08.4330
34.45 5.99
87.67 -18
53 03.82..1 6.16
$1,06,5.10
54.46....
70.19,
73.67
area under the receiver operating characteristic curve (AUROC), and (3) ID classification accuracy (ID ACC). For each baseline algorithm, we report the results both with and without A2D training and NP-Mix outlier synthesis. Additional implementation details are provided in Appendix D.
5.2 Multimodal Near-OOD Detection
We commence our evaluation by assessing the efficacy of A2D training and NP-Mix outlier synthesis within the Multimodal Near-OOD Detection setup. As presented Tab, I, the simple incorporation of A2D yields substantial enhancements in OOD performance across nearly all scenarios. The average prediction L1 distances between ID and OOD data (lOOD ( before and after A2D training across various datasets are depicted in Fig. 4. Notably, across all datasets, A2D) training serves to further enlarge the Modality Prediction Discrepancy and consequently improve OOD detection performance, verifying the strong correlation between them. Specifically, A2D training reduces the FPR95 by up to absolute 19.62% on UCF101 50/51 dataset for ASH and increases AUROC up to 14.82% for Mahalanobis on Kinetics-600 129/100. Combining A2D training with NP-Mix yields additional improvements OOD detection performance, underscoring the efficacy of outlier synthesis model regularization. Additionally, A2D training and NP-Mix outlier synthesis exhibit notable efficacy across all bascline OOD detection algorithms despite their different underlying principles, indicating the versatility of our proposed method.
5.3 Multimodal Far-OOD Detection
Tab. 2 and Tab. present the results of Multimodal Far-OOD Detection with HMDB51 and Kinetics- 600 as ID datasets, respectively. Similar to the Near-OOD setup, training with A2D and NP-Mix yields considerable enhancements in OOD detection performance across most cases for all baseline algorithms. Specifically, training with both A2D and NP-Mix achieves reductions in FPR95 of up to absolute 23.38% on the HMDB51 dataset for ASH and up to 14.43% for ReAct on the Kinetics-600 dataset. Due to space limits, we provide comprehensive results in Appendix E (Tab. 7 and Tab. 8). Interestingly, we observe that the performance on the HMDB51 dataset generally surpasses that of the Kinetics-600 dataset. This finding aligns with observations from unimodal OOD detection benchmarks [66], where performance on datasets with fewer classes (e.g., CIFAR-10 [38]) tends to outperform those on datasets with greater number of classes (e.g., ImageNet [14]).
8

---

Table 4: Multimodal Near-OOD Detection using video, audio, and optical flow.
Methods
Modality
E.PIC-Kitchens 4/4
Kinetics-600 129/100
Video
Audão
Flow
FPR951
AUROCT
ID ACC1 1
FPR951
AUROCT
ACC
Without
Training
Energy ASH
69.22
72. 39
73.13
66.42
76,60
80.35
v
70.52
69:
68.10
63.4%
78.11
79:35 54
GEN KNN VIM
80.22
60.56
73.13
75.03
65.97
80.33
76.12
59.03
73.13
66.38
76.59
80.33
With A2D Training and NP-Mix Outlier Synthesis
Enengy++ ASH++
<
6.56
74.85.2250
63-81-241
77.80-1100
80.82
69.78 0.74
60
67.72
61.22
78.57 0.40
80:05
GEN++
63.62
73.8442255
71,46
63.35-..4
77.92-034
80.82
KNN++
08.47-111 11
67.87-7.31
71.46-a57
68.87_2.0
VIM+i
70.51
50.57.4054
71.46
63.42-2
77.90-1.31
80.82
Table 5: Ablation on Distance Functions for Near- Detection on HMDB51 dataset.
Methods
Haseline
12
Wasserstein
Hellinger
FPR95]
AUROC1
FPR95L
AUROC1
FPR951
AUROC1
FPR95]
AUROC
FPR95_
AUROC1
Energy
43.36
87.46
37.04
88.52
37.69
88.54
36.17
88.61
36.38
88.91
GEN
43.79
KNN
42.92
88.46
35.73
90.24
35.51
89.78
36,60
88.57
33.77
90.05
VIM
36.82
88.06
32.24
89.36
33,99
89.03
33.99
89.24
34.64
88.80
Table 6: Ablation on Outlier Synthesis Methods for Near-OOD Detection on HMDB51 dataset.
Methods
Baseline
NPOS
Mixup
FPR95]
AUROC1
FPR95L
AUROCC
[PR95]
AUROCI
FPR95]
AUROC
[PR95
AUROC1
Energy
43.36
87.46
37.69
87.94
43.14
87.59
42.27
87.77
36.38
88.91
GEN
43.79
88.11
42.70
35.95
KNN
42.92
88-46
38.
88.55
40.52
88.81
36.17
89.61
33.77
90.05
VIM
36.82
88.06
35.51
88.93
37.69
88.17
37.04
88.44
34.64
58.80
5.4 Ablation Studies
Multimodal Near-OOD Detection with other modalities. We demonstrate the efficacy of A2D training and NP-Mix outlier synthesis across various combinations of modalities, not limited to video and optical flow, as detailed Tab. 4 and Tab. 9 in Appendix E. Notably, the performance of most algorithms is consistently improved with A2D and NP-Mix, regardless of which combinations of modalities are used.
Effectiveness of other distance functions. We substitute the distance metric for measuring probabil- ity distributions in the discrepancy loss with alternative distance functions, including L1, L2, and Wasserstein [2] distances. As depicted in Tab. 5, A2D training exhibits robustness across various distance funetions. Regardless of the specific distance metric employed, substantial improvements are consistently observed compared to the baseline approach without A2D training.
Compared with other outlier synthesis methods. To evaluate the effectiveness of other outlier synthesis methods, we replace NP-Mix with VOS [22], NPOS [60], and Mixup [69] and train with A2D. All baseline methods yield improvements in OOD performance, underscoring the significance of outlier synthesis for model regularization. Notably, NP-Mix emerges as the most effective among the various baselines by synthesizing outliers that span broader embedding spaces.
Conclusion
In this paper, we introduce the Multimodal Out-of -ittr--disttibution Detection problem and present a novel benchmark, MultiOOD, which includes diverse dataset sizes and various combinations of modalities. Motivated by the Modality Prediction Discrepancy phenomenon observed within MultiOOD, propose a novel A2D training algorithm to encourage the enlargement of such discrepancy during model training, along with new outlier synthesis algorithm, NP-Mix, that complements A2D. Extensive experiments on MultiOOD and under Near-OOD and of the proposed approaches. We hope work will inspire future research endeavors in Multimodal OOD Detection.
setups verify the efficacy
Limitation and Future Work. The performance on Near-OOD benchmarks and on datasets with large number of classes still exhibits potential for enhancement. Moreover, the exploration Outlier Exposure [32] deserves attention potential to better learn ID/OOD discrepancy.

---

Acknowledgments
The authors acknowledge the support of "In-service diagnostics of the catenary/pantograph and wheelset axle systems through intelligent algorithms" (SENTINEL) project, supported by the ETH Mobility Initiative.
References
[1] Abati, D., Porrello, A., Calderara, S., Cucchiara, R.: Latent space autoregression for novelty detection. In: CVPR (2019)
[2] Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: ICML (2017)
[3] Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., Gall, J.: Se- manticKITTI: Dataset for Semantic Scene Understanding of LiDAR Sequences. In: ICCV (2019)
[4] Blum, H., Sarlin, P.E., Nieto, J.. Siegwart, R., Cadena, C.: The fishyscapes benchmark: Measur- ing blind spots in semantic segmentation. International Journal of Computer Vision 129(11), 3119-3135 (2021)
[5] Breunig, M.M., Kriegel, H.P., Ng, R.T., Sander, J.: Lof: identifying density- baased local outliers. In: Proceedings of the 2000 ACM SIGMOD international conference on Management of data. pp. 93-104 (2000)
[6] Carreira, J., Noland, E.,, Banki-Horvath, A., Hillier, C., Zisserman, A.: short note about kinetics-600. arXiv preprint arXiv: 1808.01 000000040 (2018)
[7] Carreira, J., Zisserman, A.: Quo vadis, action recognition? new model and the kinetics dataset. In: CVPR (2017)
[8] Chan, R., Lis, K., Uhlemeyer, S., Blum, H., Honari, S., Siegwart, R., Fua, P., Salzmann, M., Rottmann, M.: Segmentmeifyoucaan: A benchmark for anomaly segmentation. arXiv preprint arXiv:2104. 14812 (2021)
[9] Chen, G., Peng, P., Wang, X., Tian, Y:: Adversarial reciprocal points learning for open set recognition. arXiv preprint #xXvv2203.00095. (2021)
[10] Chen, H., Xie, W., Vedaldi, A., Zisserman, A.: Vggsound: A large-scale audio-visual dataset. In: ICASSP (2020)
[11] Contributors, M.: Openmmlab's next generation video understanding toolbox and benchmark. https://git thub. cwm/open-teeeeec mm www.b//aam...mmmmmmmmmmmmm wwtpaactiionnn (2020)
[12] Cortinhal, T., Tzelepis, G., Aksoy, E.E.: Salsanext: Fast, uncertainty-aware semantic segmenta- tion of lidar point clouds for autonomous driving. arXiv preprint arXiv:2003.0365: (2020)
[13] Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D.,
Munro, J., Perrett, T., Price, W., Wray, M.: Scaling egocentric vision: The epic-kitchens dataset. In: ECCV (2018)
[14] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. CVPR (2009)
[15] Deng, L.: The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine 29(6), 141-142 (2012)
[16] Djurisic, A., Bozanic, N., Ashok, A., Liu, R.: Extremely simple activation shaping for out-of- distribution detection. arXiv preprint arXiv:2209.009: 008858 (2022)
[17] Dong, H., Chatzi, E.. Fink, O.: Towards multimodal open-set domain generalization and adaptation through self-supervision. In: ECCV (2024)
[18] Dong, H., Chen, X., Särkkä, S., Stachniss, C.: Online pole segmentation on range images for long-term lidar localization in urban environments. Robotics and Autonomous Systems 159, 104283 (2023)
[19] Dong, H., Frusque, G., Zhao, Y., Chatzi, E.,, Fink, O.: Nng-mix: Improving semi-supervised anomaly detection with pseudo-anomaly generation. arXiv preprint arXiv:2311.1191 (2023)
10

---

[20] Dong, H., Nejjar, I., Sun, H., Chatzi, E., Fink, O.: SimMMDG: A simple and effective framework for multi mudal domain generalization. In: NeurIPS (2023)
[21] Dong, H., Zhang, X., Xu, J., Ai, R.,, Gu, W., Lu, H., Kannala, J., Chen, X.: Superfusion: Multi- level lidar-camera fusion for long-range hd map generation. arXiv preprint arXiv:2211. 15666 (2022)
[22] Du, X., Wang. Z., Cai, M., Li, Y:: Vos: Learning what you don't know by virtual outlier synthesis. In: ICLR (2022)
[23]
Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition. In: ICCV
(2019)
[24] Fink, O., Wang. Q., Svensén, M.. Dersin, P., Lee, W.J. Ducoffe, M.: Potential, challenges and future directions for deep learning in prognosties and health manage- ment applications. Engineering Applications of Artificial Intelligence 92, 103678 (2020). e0g.///.ggaaa 103678
(25] Gong, Y., Chung, Y.A., Glass, J.: Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778 (2021)
[26] Gorishniy, Y., Rubachev, I., Khrulkov, V., Babenko, A.: Revisiting deep learning models for
tabular data. NeurIPS (2021)
[27] Gupta, S., Hoffman, J., Malik, J.: Cross modal distillation for supervision transfer. In: CVPR (2016)
(28) Han, S., Hu, X., Huang, H., Jiang, M., Zhao, Y.: Adbench: Anomaly detection benchmark. In: Neural Information Processing Systems (NeurIPS) (2022)
[29] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
[30] Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., Song,
D.: Sealing out-of-distribution detection for real-world settings. ICML (2022)
[31] Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: ICLR (2017)
[32] Hendrycks, D., Mazeika, M., Dietterich, T.: Deep anomaly detection with outlier exposure. In: ICLR (2019)
[33] Jaritz, M., Vu, T.H., de Charette, R., Wirbel, E., Pérez, P.: xMUDA: Cross-modal unsupervised domain adaptation for 3D semantic segmentation. In: CVPR (2020)
[34] Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T.. Back, T.. Natsev, P., et al.: The kinetics human action video dataset. arXiv preprint arXiv:170s 705.06950 (2017)
[35] Khosla, P., Teterwak, P., Wang. C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C.,
Krishnan, D.: Supervised contrastive learning. In: NeurlPS (2020)
[36] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. ICLR (2015)
[37] Kong, S., Ramanan, D.: Opengan: Open-set recognition via open data generation. In: ICCV (2021)
[38] Krizhevsky, A., Nair, V., Hinton, G.: Cifar- 10 and cifar-100 datasets. URI: https://www. es. toronto. edu/kriz/eifar, html 6(1), (2009)
[39] Kuchne, H.. Jhuang, H., Garrote, E.. Poggio, T., Serre, T.: Hmdb: large video database for human motion recognition. In: ICCV (2011)
[40] LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., Huang, F: tutorial on energy-based learning. Predicting structured data 1(0) (2006)
[41] Lee, K., Lee, K., Lee, H., Shin, J.: A simple unified framework for detecting out-of distribbbtii samples and adversarial attacks. In: NeurIPS (2018)
[42] Liang, S., Li, Y., Srikant. R.: Enhancing the reliability of out-of distribbittiitooonn image detection in neural networks. In: ICLR (2018)
[43] Liu, W., Wang, X., Owens, J.D., Li, Y.: Energy-based out-of-distribution detection, In: NeurIPS (2020)
11

---

[44] Liu, X., Lochman, Y.. Zach, C.: Gen: Pushing the limits of softmax- out-of-distribuite detection. In: CVPR (2023)
[45] Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin transformer. In: CVPR (2022)
[46] Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11) (2008)
[47] Ming, Y., Cai, Z., Gu, J., Sun, Y., Li, W., Li, Y: Delving into out-of-distribttion detection with vision-language representations. In: NeurIPS (2022)
[48] Munro, J., Damen, D.: Multi-modal domain adaptation for fine-grained action recognition. In:
CVPR (2020)
[49] Pardo, L.: Statistical inference based on divergence measures. Chapman and Hall/CRC (2018)
[50] Pidhorskyi, S., Almohsen, R., Adjeroh, D.A., Doretto, G.: Generative probabilistic novelty detection with adversarial autoencoders. In: NeurIPS (2018)
[51] Planamente, M., Plizzari, C., Alberti, E., Caputo, B.: Domain generalization through audio- visual relative norm alignment in first person action recognition. In: WACV (2022)
[52] Prokhorenkova, L.,, Gusev, G., Vorobev, A., Dorogush, A. Gulin, A.: Catboost: unbiased boosting with categorical features. NeurlPS (2018)
[53] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A.,
Mishkin, P., Clark, J., Krueger, G., Sutskever, L: Learning transferable visual models from natural language supervision. In: ICML (2021)
[54] Ruff, L., Vandermeulen, R., Goernitz, N., Deecke, L., Siddiqui, S.A., Binder, A., Müller, E., Kloft, M.: Deep one-class classification, In: International conference on machine learning. pp- 1393-4402 PMLR (2018)
[55] Ruff, L., Vandermeulen, R.A., Görnitz, N., Binder, A., Müller, E., Müller, K., Kloft, M.: Deep
semi-supervised anomaly detection. In: ICLR (2020)
[56] Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NeurIPS (2014)
[57] Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: (2012)
[58] Sun, Y. Guo, C., Li, Y:: React: Out-of-distribution detection with rectified activations. In: NeurIPS (2021)
[59] Sun, Y., Ming, Y., Zhu, X., Li, Y:: Out-of-distributtin detection with deep nearest neighbors. ICML (2022)
[60] Tao, L., Du, X., Zhu, X., Li, Y.: Non-parametric outlier synthesis. arXiv preprint arXiv:2303.0290
(2023)
[61] Vaze, S., Han, K., Vedaldi, A., Zisserman, A.: Open-set recognition: A good closed-sed set classifier is all you need. In: ICLR (2022)
[62] Wang, H., Li, Z., Feng, L.., Zhang, W.: Vim: Out-of-distribution with virtual-logit matching. In: CVPR (2022)
[63] Wang, H., Li, Y., Yao, H., Li, X.: Clipn for zero-shot ood detection: Teaching clip say In: ICCV (2023)
[64] Wang, L., Xiong, Y.. Wang, Z., Qiao, Y., Lin, D., Tang, X., Van Gool, L.: Temporal segment networks: Towards good practices for deep action recognition. In: ECCV (2016)
[65] Wei. H., Xie, R., Cheng, H., Feng, L., An, B., Li, Y:: Mitigating neural network overconfidence with logit normalization. In: ICML (2022)
[66] Yang, J., Wang, P., Zou, D.,, Zhou, 7., Ding, K., Peng, W., Wang, H., Chen, G., Li, B., Sun, Y., et al.: Openood: Benchmarking generalized out-of-distribution detection. In: NeurIPS (2022)
[67] Yang, J., Zhou, K., Li, Y., Liu, Z.. Generalized out-of-distiibution detection: A survey. arXiv preprint arXiv:21 10. 1334 (2021)
[68] Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime tv-1 optical flow. In: Pattern Recognition (2007)
12

---

[69] Zhang, H., Cisse. M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimiza-
tion. In: ICLR (2018)
[70] Zhou, Y.: Rethinking reconstruction autoencoder-based out-of-distribution detection. In: CVPR (2022)
[71] Zhou, Y., Song, X., Zhang, Y., Liu, F., Zhu, C., Liu, L.: Feature encoding with autoencoders for weakly supervised anomaly detection. IEEE Transactions on Neural Networks and Learning Systems 33(6), 2454-2465 (2021)
[72] Zhuang, Z., Li, R., Jia, K., Wang, Q., Li, Y., Tan, M.: Perception-aware multi-sensor fusion for 3d lidar semantic segmentation. In: ICCV (2021)
13

---

A Related Work
Auttof-Distribution (OOD) Detection aims to detect test samples with semantic shift without losing the ID classification accuracy. Numerous OOD detection algorithms have been developed, with post hoc methods and training-time regularization as major categories [66]. Post hoc methods aim to design OOD scores based on the classification output of neural networks, offering the advantage of being easy to use without modifying the training procedure and objective. Early approaches include utilizing Maximum Softmax Probability (MSP) [31] as OOD score, often coupled with temperature scaling and input perturbation [42]. Instead of using softmax probabilities, MaxLogit [30] employs maximum logit as OOD scores rather than softmax. Energy-based algorithm [43] demonstrated the efficacy of energy function [40] in quantifying OOD-ness. Other approaches like ReAct [58] improved existing scoring functions by truncating the activations with high values. Similarly, ASH [16] prunes large portion of the input activations and adjusts the remaining activations using pruning, binarizing, or scaling. Methods like Mahalanobis [41] and K-Nearest Neighbor (KNN) [59] use distance metrics in feature space for OOD detection, while Virtual-logit Matching (VIM) [62] integrates information from both feature and logit spaces to define the OOD score. Recently, Generalized Entropy (GEN) [44] proposed an entropy- bbssed score function that proves to be both simple and effective.
Truining-time regularization methods such as LogitNorm [65] address prediction overconfidence by imposing constant vector norm on the logits during training. Outlier Exposure [32] leverages external OOD samples from other datasets during training to facilitate the learning of better ID and OOD discrepancy. Additionally, some approaches [22, 60] proposed synthesizing virtual outliers for training-tine time regularization. However, all previous approaches were designed for unimodal scenarios, without accounting for the interaction and complementary nature of diverse modalities.
Multimodal OOD Detection. Recent endeavors [47, 63] have explored vision-language models [53] to enhance OOD performance, which are also referred to as multimodal in some of the studies. Maximum Concept Matching (MCM) [47] defines OOD score by aligning visual features with textual concepts. CLIPN [63] equips CLIP [53] with the capability of distinguishing OOD and ID samples using positive-semantic prompts and negation-semantic prompts. However, the evaluations of all these works are still limited to benchmarks exclusively containing images. Consequently, existing methods fall short in fully leveraging the complementary information from various modalities, such as LiDAR and camera in autonomous driving [21], as well as video, audio, and optical flow in action recognition [56]. There is also a lack of benchmark datasets that facilitate the evaluation of Multimodal OOD Detection. Therefore, we aim to develop more practical and challenging benchmark incorporating multiple combinations of modalities (i.e., video, audio, and optical flow). This enables the creation of OOD detection algorithms that are specifically designed to leverage the complementary nature of various modalities effectively.
Anomaly Detection and Open Set Recognition are two closely related fields to OOD Detection. Anomaly Detection (AD) aims to detect patterns that deviate from the predefined normality during testing [67] and treats all in-distribution samples as a single class. Therefore, AD algorithms can be applied to OOD detection by ignoring all the labels for ID data. Typical AD algorithms include unsupervised [54, 51. semi-supervised [55, 71], and supervised [26, 52], depending on the availability of labels [28]. Open Set Recognition (OSR) [61, 37, 9, 17] focuses on accurately classifying test samples from "known known classes" (ID) and detecting test samples from "unknown unknown classes" (OOD). While OOD detection benchmarks always take one dataset as ID and find several other datasets with con-overlap categories as OOD, OSR benchmarks usually split one multi- class classification dataset into ID and OOD parts according to classes. However, both OSR and OOD detection share the same goal of identifying test samples with semantic shifts without compromising the accuracy of ID classification [67]. Therefore, in our benchmark, we treat OSR and OOD as synonymous concepts and adopt OOD as the general term. Our Near-OOD Benchmark is similar to traditional OSR setup and our Far-OOD Benchmark is the same as general OOD setup.
OOD Benchmarks. Early works [31] in OOD detection primarily focus on small-scale image datasets such as MNIST [15] and CIFAR-10/100 [38]. Recognizing the need for evaluating OOD detection at scale, studies such as [62] introduce new OOD datasets based on ImageNet [14]. Additionally, some
OOD benchmarks [30, 8] are specifically designed for semantic segmentation tasks. OpenOOD [66] offers a comprehensive OOD benchmark comprising datasets from previous works and incorporating over 30 common OOD methods. However, all of these benchmarks are limited to image data. In contrast, our MultiOOD is the first public OOD benehmark that encompasses different combinations of modalities, facilitating future research endeavors in Multimodal OOD Detection.
14

---

EPIC-Kitchens
HAC
HMDB51
Choeling
UCF101
Kinetics-600
Figure 6: Visualization of action recognition datasets used in our MultiOOD benchmark.
B More Details on the MultiOOD Benchmark
B.1 Datasets Used in MultiOOD Benchmark
Action recognition is inherently multimodal and serves as the primary task within our benehmark, and we include five action recognition datasets accordingly of varying sizes, shown in Fig. 6.
EPIC-Kitchens [13]] The EPIC-Kitchens dataset is a large-scale egocentric dataset collected by 32 participants in their native kitchen environments. The participants were asked to capture all their daily kitchen activities and record sequences regardless of their duration. The start and end times for each action are annotated. We use a subset of the Domain Adaptation paper [48], which comprises 4, 871 video clips from 8 largest action classes in sequence P22. These actions include 'put', 'take', 'open', 'close", 'wash', 'cut", 'mix", and 'pour', with provided modalities including video, optical flow, and audio.
dataset introduced in the Multimodal
HAC [20]. The HAC dataset encompasses seven actions ('sleeping', 'watching tv', 'cating', 'drink- ing', 'swimming'. 'running', and 'opening door') performed by humans, animals, and cartoon figures. There are 3, 381 video clips in total from seven actions. Modalities provided in this dataset include video, optical flow, and audio.
HMDB51 [39]. The HMDB51 dataset is a video action recognition dataset, comprising 6, 766 video clips spanning 51 action categories. The video clips are extracted from variety of sources ranging from digitized movies to YouTube. Available modalities in this dataset include video and optical flow.
UCF101 [57]. UCF101 is another video action recognition dataset collected from YouTube, com- prising 13, 320 video clips from 101 actions. UCF101 offers substantial diversity in action types and encompasses significant variations in camera motion, object appearance and pose, object scale, viewpoint, cluttered background, illumination conditions, etc. Modalities provided this dataset include video and optical flow.
Kinetics-600 [6]. Kinetics-600 is large-scale action recognition dataset, featuring approximately 480k videos spanning 600 action categories. Each video in the dataset is 10-second clip of action moment annotated from YouTube videos. our benchmark, we carefully selected subset of 229 action classes from Kinetics-600 to mitigate the potential category overlap with other datasets, with each class comprising roughly 250 video clips, yielding a total 57, 205 video clips. The original dataset provides video and audio modalities. To make it consistent with the other dataset, our benchmark further provides the extracted optical flow for all video clips, amounting to a total of 114, 410 optical flow samples. The dense optical flow is extracted at 24 frames per second using the TV-L1 algorithm [68].
B.2 Multimodal Near-OOD Benchmark
In the Near-00D setup, we incorporate four datasets. EPICCKKtchenss 4/4 is derived from the EPIC- Kitchens Domain Adaptation dataset [48], where the dataset randomly partitioned into four classes for training as ID and four classes for testing as OOD, Similarly, HMDB51 25/26 and UCF101 50/51
15

---

(a) VOS
(b) NPOS
TD Gonarced Dlttar
Coumened Dutle
(c) Mixup
(d) NP-Mix (ours)
Figure 7: Visualization of synthesized outiers compared against other methods. VOS and NPOS tend to generate outliers near the ID data, neglecting to explore the broader embedding space. Mixup randomly selects samples from all classes mix and inadvertently introduces unwanted noise samples within the distribution of ID data. NP-Mix excels at generating synthesized outliers by effectively utilizing information from neighbor classes and spanning wider embedding spaces.
are constructed based on HMDB51 [39] and UCF101 [57]. respectively. In the case of Kinetics-600 129/100, we curate 229 action classes from the Kinetics-600 dataset [6], with each class comprising approximately 250 video clips. Within this setup, 129 classes are randomly designated for training as ID,, while the remaining 100 classes are allocated for testing as OOD.
B.3 Multimodal Far-OOD Benchmark
In the Far-OOD setup, incorporate HMDB51 and Kinetics-600 ID datasets.
HMDB51 dataset as ID. For the OOD datasets, we utilize UCF101, EPIC-Kitchens, HAC, and Kinetics-600 -000 datasets. All of these datasets are carefully curated to remove samples that should belong ID classes. Given the relatively small number of classes the EPIC-Kitchens and HAC datasets, we remove classes in the HMDB51 dataset that overlap with EPIC-Kitchens and HAC, including 'chew", "climb cliibbsstirr', 'drink', 'eat', 'pick", 'pour", ride_horse"" 'run', leaving 43 classes as ID classes. In the case of UCF101, we remove 31 overlapping classes with HMDB51, resulting in 70 classes designated as OOD classes for evaluation. For Kinetics-600, we use the same subset of 229 classes in the Near-OOD setup, which are carefully selected to mitigate the potential category overlap with other datasets. For other datasets, no class overlap exists and we utilize their original classes OOD.
Kinetics-600 dataset as ID. Similarly, we adopt UCF101, EPIC- Fitc-Ki,hhens, HAC, and HMDB51 datasets as OOD datasets, with careful curation undertaken to exclude samples belonging to ID classes. We carefully selected subset of 229 action classes from Kinetics-600) in the Near-OOD setup mitigate the potential overlap with other datasets. For the UCF101 dataset, we remove 11 overlapping classes with Kinetics-600, leaving classes as OOD classes for evaluation. For other datasets, there class overlap issues, and use their original classes OOD.
16

---

00
(8)A50)
KONNN
LoptNomm
140
1.70
115
69) Rorrgy-
(6-ASH++
KKN+
(ha LeghtNomes
Figure 8: Score distributions of different baseline methods on the HMDB51 25/26 dataset before and after training with A2D and NP-Mix.
OOD
OOD
Embeddings without A2D and NP-Mix
(b) Embeddings with A2D and NP-Mix
Figure Visualization the learned embeddings on ID and OOD data using t-SNE on the HMDB51 25/26 dataset before and after training with A2D and NP-Mix.
C Visualization of Results
Visualization of Synthesized Outliers. We visualize the outliers generated by different outlier synthesis algorithms, including VOS [22], NPOS [60], Mixup [69], and our proposed NP-Mix. As shown in Fig. 7, VOS and NPOS generate outliers near the ID data, neglecting to explore the broader embedding space. Mixup randomly selects samples from all classes mix and introduces unwanted noise samples within the distribution of ID data. NP-Mix excels at generating synthesized outliers by effectively utilizing information from neighboring classes and spanning wider embedding spaces.
Score Distributions, Fig. 8 illustrates the score distributions generated by various baseline methods on the HMDB51 25/26 dataset before and after training with A2D and NP-Mix. Score distributions produced by A2D and NP-Mix lead to better ID/OOD separation, resulting in strengthened OOD detection performance.
Visualization of Learned Embeddings for and OOD Data. Fig. shows visualization the learned embeddings using t-SNE [46] on the HMDB51 25/26 dataset before and after training with A2D and NP-Mix. The embedding ID and OOD data are more separable after A2D training and NP- Mix outlier synthesis.
17

---

D More Implementation Details
D.1 Training Details
We conduct experiments across three modalities: video, audio, and optical flow. We adopt the MMAction2 [11] toolkit for experiments. To encode the visual information, we utilize SlowFast network [23]. initialized with pre-trained weights from Kinetics-400 [34]. For the audio encoder, we
employ ResNet-18 [29]. initializing the weights from the VGGSound pre-trained checkpoint [10].
Similarly, we use the SlowFast network with slow-only pathway, again leveraging pre-trained weights from Kinetics-400 [34] for the optical flow encoder. We use the Adam optimizer [36] with learning rate of 0.0001 and a batch size of 16. Additionally,
set the hyperparameters as follows:
y = 0.5, mixup a 10.0, nearest neighbor N = 2. We train the network for 50 epochs on an RTX 3090 GPU and select the model with the best performance on the validation dataset.
D.2 Extension to More Modalities
Our framework is not limited two modalities and can be easily extended to M modalities. Given training sample x with label e and M modalities, we obtain prediction probabilities ý from the combined embeddings of all modalities, and 8¹, p² ...., PM from each modality, all of which are of shape [1, C). where C represents the number of classes. By removing the e-th value from each prediction, we derive new prediction probabilities without ground-truth classes, denoted as pt. f ...... Id with shapes [1, C 1]. Subsequently, we aim to maximize the discrepancy between p', P
pM which can be defined as:
M-1
M
Diser M(M 1)
Σ
Σ j=i+1
Discr .'))).
(10)
where Discr(-) is a distance metric quantifying the similarity between two probability distributions. Similarly, for Lots. D and LEnt, we can define as:
M
Cete M ++11
(CE(P.)) v) + Σ CE( v)),
(11)
1=1
2
DDisor_eyn
Discr ;,)).
(12)
M(M - 1)
j=it1
M
LEa
1
Σ H())
(13)
M
The final loss is obtained the weighted sum of the previously defined losses:
= Lote + (C + Diecr_egn + LEnt).
(14)
D.3
Inference Details
For algorithms that define OOD score truncate activations leveraging information from the feature
space (Mahalanobis [41], KNN [59]. VIM [62], ReAct [58], ASH [16]), we use the combined embedding Z [Z] Z ..., ZM from all modalities. For algorithms that define the OOD score leveraging information from the probability space or logit space (MSP [31], GEN [44], Energy [43], the prediction probabilities ş or prediction logits h(Z) obtained from the combined embeddings of all modalities.
MaxLogit [30], VIM [62], LogitNorm [65]),
E Further Experimental Results
Multimodal Far-OOD Detection. Tab. and Tab. present comprehensive results Multimodal Far-OOD Detection on the HMDB51 and Kinetics-600 datasets. Training with A2D and NP-Mix significantly improves OOD performance most cases across all baseline algorithms, underscoring the versatility of our proposed method.
18

---

Table 7: Multimodal Far-OOD Detection using video and optical flow, with HMDB51 ID.
Datasets
Kinetics-600
UCFI01
EPIC-KKichens
BAC
Average
ACC
FPROS]
AUROC1
FPRO5L
AUROCT
FPR95]
AUROC*
FPR95]
AUROC1
FPR95]
AUROCT
Without A2D Training
MSP
96.11
88.78
46.64
86.40
17.33
95. 09
19.91
89.10
35.75
90.07
87.23
Energy
32.95
92.45
$7.95
3.10
97.70
32,95
92.28
29.33
92.60
87.23
Maxt
33.07
92.31
Malsalanobis
91
ReAct
93.02
ASH
87.81
GEN
41.51
90.34
46.18
81.91
8.21
98. 26
39.31
91.28
33.55
91.95
87.23
KNN
22.69
95.01
39
89.20 28
9.92
97. 97.92
20.75
96.02
23 18
94.96
87.23
VIM
13.68
97.01
33.87
91.45
3.93
98,
13.45
97.12
16.73
95.93
87.23
LegitNorm
46.07
87.41
49.03
85.96
15.96
35.30
47,09
87.64
39.54
89.33
86.09
With A2D Training und NP-Mix Outlier Synthesis
MSP++
4.82
13.34....9
28-16...
91.6342.55
27.70
01.72,11.0
Energ++
24.52
36.40
69.67.1.72
6.96--1.14
53-1.11
12:12
04.41.2.15
02.772-7:43
Marl agit=*
03.60 02.00, 1:39
36.60 x.13
6.06-3.14
97.67 -3:10
16-14
94.22,y9.0
-1-84
03.9 03.92;11
86.87
Mahnllnsbiss
9.01-1322 1,02
27.94-15.22 97.72.1.08
91,09:444
12.77-5.99
50, -..9
7:64-4:55
98.254..35
14.34 14.34--7.00
95. 95.75.22.5
86.89
ReAet4+
21.09 -6.00
94.72
57.51-4.22
80.66..1.4
7.30--0.21
57.38 -3.33
20064-16.17
90.01 +0.10
21.60 21.63-1.0 1.90
94.19.1.11
86.66
ASI+++
27 52
03.17 45.36
38:
89.5 69.52...30
6.84 6.81--13.11
98. 2,.1
23.00
04.45.4..2
24.05.
00. 13.86..22
86.20
GEN++
25.06
93.50 55..5....0
37.
91.19
0.25
588.96
24.63
94.25
23.24
914.49 994.99,153
86.89
KNN++
15.05
96.96 96.96.1..6
33.00
91.92
5.47
98.97
13.45.-7.9
87.25.1.25
16.70
06. 06.28
86.89
VIM=+
9.22
98.34.1.05
26.45-7744
98.545
11.77-4.94
916. 966.766-0.8
LogiNNemms
16.94
18.91
81.37
96.81
21.23
05.16,5.55
14:00
03.20,a.9
Table 8: Multimodal Far-OOD Detection using video and optical flow, with Kinetics-600 ID.
0OD Datasets
Methods
HMDB51
LCF101
EPIC KitchKes
HAC
Average
ID ACC1
JPXU5]]
AL KOUU
HPRSS
AUROCT
EPRSS,
AURUC"
HPRS5
AUROUT
HPRR5
AUROCT
Without A2D Training
MSP
66.85
67.32
71.15
43.37
56.66
56.17
73.14
Energy
43.66
$2.05
75.07
MaxLogit
73.14
Mahulanobis
89:01
$6.24
81.85
64.49
93:12
48.85
$2.79
73. 13.14
ReAct
$2.17
67.09
78.44
67.64
53.49
75.07
75.11
70.04
72.30
70.71
13.27
ASH
71.62
76.66
69. 36
12.35
34.38
83 85.05
47.85
$3.40
55.80
90.15
72.20
GEN
68.47
78.43
64.80
73.97
36.81
85.
49.3
83.67
$1,90
80.30
73.14
KNN
71.08
78.84
68.62
74.33
41.83
N2
57.00
$2.53
59.63
79.51
73.14
VIM
72.25
70.72
70.58
43.14
92 12.09
59.48
75.46
73.14
LogitNorm
66.48
74,47
Training
Synthesis
MSP++
62.76-1.77
7..5
66.33--0.0
73.13.2.00
122844 9.40
83, 78.7788
53.37 5.40
68.83,0.41
78.82.0.50
73.67
Enegy++
63:27 19:37
74.17 42:40
67:20 22100-1000
70.50 o1
$1.07 1.30
87 .5.49 4:4
56 4.51
80.20 60:20
55.51 -6
70.00.2.02
73.67
MatLogit=+
61.70 10:36
75.30
87.48
74.58 mm
02:22
30. 23.74...
52.52
81.90, (4.06
53.48 6..0
801.41
73.67
Mahulanahisss
85.95
67. 67.92..100
$1.96...11
62.
80.05_..04
20.37 01.37..1
93.38
4.8
88.09
64.60.
73.67
ReAs+++
73.00
70.09.2.00
76.09
71. 71.30.a.m
30.00
85,
70.09
76.04.6..0
64.56
75.74.....
73.65
ASi+++
65.65
70. 10.14:9.00
62.15
75.32
96 96.09;1.11
88
46. 16-47-1.95
21:2
52.00
82.00
72.53
GEN++
57.03-7.77
77.07-...
85,10 25.10.
41.97-7.66
60.19-4471
73.67
KNN-++
64.60
80.25....4 ...4
07.44.1.11
77.9
57.
-2.22.2
53.63
83.34 4m.m3
81.28.1
73.67
VIM++
63.25-..m0
25.96....o
86.31-444
34.45-6.00
-4.38
5.46
41.06....00
51.46-...0
LogitNorme
07.07 11
79.15,g
00.14 4.00
77.01
31.07 7:46
57.51 y3.99
78.92 9.01
53.97
81.19,0.9
70.81
Multimodal Near-OOD Detection with Other Combination of Modalities. We demonstrate the effectiveness of A2D and NP-Mix across various combinations of modalities, not limited to video and optical flow, as shown in Tab. 9. The performance of different baseline algorithms improves significantly with A2D training and NP-Mix outlier synthesis, regardless of whether the input modalities are video-audio, flow-audio, or video-audio-feee flow combinations.
F
More Ablations
Compared with Other Multimodal Tasks. We compare A2D and NP-Mix with other multimodal self-supervised training tasks, including Contrastive Loss [35], Relative Norm Alignment (RNA) Loss [51], Cross-modal Distillation [27], and Cross-modal Translation [20], as shown in Tab. 10. While contrastive loss demonstrates effectiveness enhancing OOD performance, other tasks significantly decrease the performance. A2D and NP-Mix show substantial superiority over other multimodal self-supervised tasks.
Influences of N and a in NP-Mix. In this section, we investigate the parameter sensitivity of NP-Mix on the HMDB51 25/26 dataset. For the Nearest Neighbor parameter N, we test values of 1, 2, 3, and 4. Regarding the Mixup parameter a, evaluate values of 2.0. 4.0, and 10.0. As shown
19

---

Table 9: Multimodal Near-OOD Detection using different combination of modalities.
Modality
EPICCKitchens 4/4
Kinetics-600 129/100
Methods
Video
Audio
Flow
FPR95]
AUROCT
ACC
FPR95]
AUROCI
ACC
Without A2D
Truining
Energy ASH GEN KNN
69.59
73.58
71.08
65.69
76.68
81.44
75.75
63.89
62.69
63.32
78.04
80.86
69.22
69.04
71.U8
66.18
77.65
81.44
97.20
31.97
71,08
77.38
64.36
81.44
37.
76.69
With A2D Training and NP-Mix Outlier Synthesis
Energy++ ASII+
67.72-1x8
72.66--0.5
61.12-1.57
77.15.0047
75.93
67 67.83
60.44
62.67 0.65
78.03
79.80
GEN++
67.35-144
72.66
62 a0
78.01+0.06
KNN++
79. -1.666--7.
65.5 65.52-33.a5
69.40
73 -3.57
66.09
81.23
VIM++
77.80 14.55
57.50.20..4
69.
03.65 2.12
77.23; y0.03
81.23
Withont A Training
Energy
62.59
77.76
18
ASH
78.36
60.74
62.13
77.27
65.43
$4,47
GEN
74.25
61.48
67.91
64.73
18
KNN
89.55
84
67.91
92.02
50.70
56, 18
VIM
45,99
77.78
65.09
With A2D Training and NP-Mix Outlier Synthesis
Energy++
70.90 2.68
68.64 66.0
68.47
76.11
65.55 yo
55.90
ASH++
72.37-5mm
66,201 3.145..00
60.45
75.68
65.86
$4.55
GEN++
73. 13.
66, 66.38.4450
68.47
7-148__2.0
65.33
55.90
KNN++
52. -2.00
58. 1 14.-
68.47
89.98
53.944 nx
$5.91
VIM++
83. 40....
19.4043.4
68.47
76.13-1.8
65.57
55,90
Without A2D Training
Energy
69.22
72.39
73.13
66.42
76.60
80.33
ASII
70.52
69.70
68.10
63.48
78.11
79.54
GEN
70.34
70.99
73.13
64.24
77.54
80.33
KNN
80.22
60.56
73.13
75.03
65.97
80.33
VIM
76.12
59.03
73,13
66.38
76.59
80.33
With A2D Training and NP-MIx Outller Synthesis
Energy++ ASH++ GEN+
62.69
74.95
71.46
63
77 17.80,
80.82
69.78 0.74
0.21
2.26
78.57 10:40
63.622
73.944
71.46
633.55
77.92 +0.922+0.a
80.82
KNN++
47 11.75
67.87
71.46-1.57
68.87
VIM++
T3.51
59.57+0.54
71.46
63..42
77.90)
80.82
Table 10: Ablation on Multimodal Training Tasks for Near-OOD Detection on HMDB51 dataset.
Mathoos
Baseline
Contrastive Loss
RNA Loss
Cross-modal Ditillation
Cross-mtdal Translazion
A2D4NP-Mix (ours)
HPR05]
AUROCCT
PPRU5]
AURROC
HPRO51
AUROC:
FPR25]
AUROCT
FPR95_
AUROC!
RPROS
AUROCT
Energy
43.36
$7.46
39:65
$8.72
55.55
$4.33
53.81
85.18
$6.41
$7.98
36.38
$8.91
GEN
43.79
87.49
41.61
89.24
58.61
83.18
55.34
85.10
48.55
87.66
35.95
89.78
57.69
43.29
88.75
90005
30.87
88.32
in Fig. 10 and Fig. 11, NP-Mix demonstrates robustness across different parameter settings and yields substantial enhancements in OOD performance for all cases.
Different Architectures. this section, we demon- strate that the strong performance of training with A2D and NP-Mix is maintained across different net- work architectures. We replaced the architecture of the video backbone with Inflated 3D ConvNet (I3D) [7] and the optical flow backbone with Tem- poral Segment Network (TSN) [64]. As shown in Tab. 12, A2D and NP-Mix consistently deliver sig- nificant improvement in OOD detection performance with these new architectures. Furthermore, re- placed the architecture of the video backbone archi- tecture with Video Swin Transformer (Swin-T) [45] and the audio backbone with Audio Spectrogram Transformer (AST) [25]. As illustrated in Tab. 13, A2D and NP-Mix also consistently achieve significant improvement in OOD detection performance using Transformer-basee architectures.
FPR95
AUROCT
Video-only (MSP)
60.78
84.39
Flow-only (MSP)
70.37
72.97
Video-only (Energy)
64.05
83.14
Flow-only (Energy)
71.46
75.51
Video-only (Mahalanohis)
31.20
81.25
Flow-only (Mahalanobis)
89.98
59.38
Vidco-only (MSP+Energy)
63.18
84.19
Vidco-only (MSS+MMahallnohis)
41.61
86.44
Vidco-only (Enncgy+Mabhalnnbbb)
44.44
86.62
Flow-only (MSP+Energy)
70.15
73.40
Flow-only Mahallansa)))
66.88
73.59
Flow-only
65.36
74.69
A2D-NP-Mix lours, Energy)
36.38
88.91
Table 11: Ablation the ensemble of differ- ent OOD scores on a single modality.
20

---

46
Eaage Eangec+
Loerge Furng++
41-
40-
AUROCT
FP285
33-
63
36
68
31-
87
32
30-
80
Mixup
Mixup
Figure 10: Influences of Mixup a for OOD performance in NP-Mix.
Enegyy Seesay
Enegy Eneay
44-
91-
42
90-
40-
AIROCT
FPR85
88
89
36-
88-
34
32-
30
Neurest Neighbor N
Neavest Neighbor N
Figure 11: Influences of Nearest Neighbor N for OOD performance in NP-Mix.
Ensemble of Multiple Unimodal OOD Methods. In this section, we add evalu-
FPR95 | AUROCT
Video (MSP) Flow (MSP)
50.98
85,40
ations on HMDB51 25/26 for the ensem-
Video (Energy) + Flow (Energy)
49.89
85.38
ble of multiple unimodal OOD methods
Video (Mahalanobis) + Flow (Mahalanobis)
52.07
81.27
Video (MSP) + Flow (Energy)
46,62
86,25
for each modality to demonstrate the im- portance of studying the multimodal OOD detection problem. We first evaluate the en- semble of different OOD scores on a single modality. We choose three scores for the ensemble: probability space (MSP), logit space (Energy), and feature space (Maha- lanobis). For all scores, we normalize their values between 0 and 1 and calculate the ensemble score as: score = a score1 + (1 - a:) * score2. For a, we do a grid search from 0.1 to 0.9 with 0.1 interval and report the one with the best performance. As shown in Tab. 11, combining MSP or Energy with Mahalanobis can bring significant improvement, especially for video. However, there is still a large gap compared with our proposed solution, demonstrating the importance of studying the multimodal OOD detection problem. We then evaluate the ensemble of OOD scores on different modalities and calculate the ensemble score score scoreflor For a, we also do a grid search from 0.1 to 0.9 with a 0.1 interval and report the one with the best performance. As shown in Tab. 14, combining more modalities always brings performance improvements, but still has large gap compared with our proposed solution, further demonstrating the importance of studying multimodal OOD detection problems.
Vidco (Energy) Flow (MSP)
50.98
83.69
Video (MSP) Flow (Mahalanobis)
57.30
84.68
Video (Mahalanobis) + Flow (MSP)
49.02
82.92
Video (Mahalanobis) + Flow (Energy)
47.71
83.51
Video (Energy) + Flow (Mahalanobis) A2D+NP-Mix (ours, Energy)
59.91
81.96
36.38
88.91
Table 14: Ablation on the ensemble of different OOD scores different modalities.
scorevideo + (1 a) *
21

---

Table 12: Multimodal Near-OOD Detection using video and flow on HMDB51 25/26 with I3D and TSN backbones.
Table 13: Multimodal Near-OOD Detection using video and audio on EPIC-Kitchens 4/4 with Swin-T T and AST backbones.
HMDBS1 25/26
EPIC-KKtthee 4/4
Methods
Methods
FPR95
AUROCT ID ACC
FPR95
AUROC* ID ACC
Without A2D Training
Without A2D Training
MSP
51.42
85,00
88.02
MSP
87 87.87
53.67
61.01
Energy
50.76
85.93
88.02
Energy
83.40
56.44
61.01
MaxLogit
50.98
85.95
88.02
MaxLogit
83.40
56.31
61.01
Mahalanobis
60.13
79.00
88.02
Mahalanobis
92.72
54.12
61.01
ReAct
86.64
87,80
ReAct
83.21
$6.45
60.45
ASH
59.26
85.95
88.02
ASH
94.96
54.23
52.61
GEN
53.38
86.13
88.02
GEN
86.38
54.37
61.01
KNN
43.57
88.66
88.02
KNN
91.04
52.03
61.01
VIM
41.61
86.58
88.02
VIM
85.45
56.68
61.01
LogitNorm
46.62
86,03
88.45
LogitNorm
81.34
59.37
58.40
With A2D Training and NP-Mix
With A2D Training and NP-Mix
MSP++
40.09-11.35
.,2.1
90.63
MSP++
77.24-10.63
67 07.022+33.5 13.33
62.31
Energy++
36.60-14.16 14.16 87.48 -1.55
90.63
Energy++
69.78 12.66
68.89 +1.89
62.31
MaxLogit++
36.60 14.38 87 07.69..1.74
90.63
MaxLogit++
69.96-13.44
48.99
62.31
Mahalanobis++
54.68
21.36.2.35
90.63
Mahalanobis++
89.93-2270
56.58+2.40
62.31
ReAct++
39.22
87_ 40.47
90.85
ReAct++
70.52-12.60
68. 68.72412.27
62.13
ASH++
45.32-13.94
87 -7.002-1..0
87.80
ASH++
89.74
56.65
14.93
GEN++
36.60 16.78
88.48 88.49 4,.39
90.63
GEN++
73.13
69.46 15.01
62.31
KNN++
37.25 37.25-6.52
88.51 -0.1
90.63
KNN++
77.05
03.35+11.32
62.31
VIM++
39.00-2.61
85.70 0.88
90.63
VIM++
71.83-13.22
67.87
62.31
LogitNorm++
39.00-7.62
87.79 1.76
87.58
LogitNorm++
12.84.1.50
57.54
58.58
Robustness under Missing-mddllities In our framework, we train a classifier for each modality to get predictions from each modality separately. By default, we use the predictions
FPR95
AUROCT
Video-only
64.05
83.14
Flow-only
71.46
75.51
obtained from the combined embeddings of all modalities to calculate the OOD score. However, when one modality is missing,
A2D+NP-ME (Video)
47.49
86.48
A2D=NP-Mix (Flow)
66.01
77.23
A2D+NP-Mix
36.38
88.91
the predictions from the remaining modality to calculate the OOD score. We add evaluations on HMDB51 25/26 under this challenging condition and use Energy as the OOD score. As shown in Tab. 15, when one modality is missing, the performance drops little, especially in the when the video is missing (A2D+NP-Mix (Flow)). However, compared with training on one modality alone (Video-only and Flow-only). training with A2D and NP-Mix can also bring significant improvements for each modality when another modality is missing, For example, A2D+NP-Mix (Video), the case when optical flow missing, yields a 16.56% relative improvement on FPR95 compared with Video-only. This underscores the importance of cross-modal training for multimodal OOD detection.
Table 15: Ablation the robustness under missing-modalities.
Beyond Action Recognition Task. To further demonstrate the versatility of the proposed A2D training, we add another task on 3D semantic segmentation using LiDAR point cloud and RGB images. We evaluate on SemanticKITTI [3] dataset and set all vehicle classes as OOD classes. During training, we set the labels of OOD) classes to void and ignore them. During inference, we aim to segment the known classes with high Intersection over Union (IoU) score, and detect OOD classes as unknown. We adopt three metries for evaluation, including FPR95, AUROC, and mIOU. (mean Intersection over Union for known classes). We use ResNet-34 [29] and SalsaNext [12] as the backbones of the camera stream and LiDAR stream. We compare our A2D with basic LiDAR-only and Late Fusion, as well as two multimodal 3D Semantic Segmentation baselines PMF [72] and XMUDA [33]. As shown in Tab. 16, our A2D also demonstrates strong performance under this new task (3D Semantic Segmentation) with different combinations of modalities (LiDAR
FPR95
AUROC
MOOOO f
LiDAR- 57.78
84.76
59.81
Late Fusion
53.43
86,98
61.43
PMF
$1.57
88.13
61. 3.
XMUDA
55.49
89,99
61.45
A2D
49.02
91.12
61.98
Table 16: Ablation the 3D seman- tic segmentation using LiDAR point cloud and RGB images.
point cloud and RGB images).
22

---

wio A2D and NP Mix
w/jo AED and NP-S
55-
w/
92
50-
AUROC
F7R86
05-
88
A
86
40-
84
35-
80
Ensegy
GEN
KNN
VIM
Energy
GEN
KNN
VIM
Figure 12: Experiments using three random seeds for Multimodal Near-OOD Ser Detection on the HMDB51 25/26. Foreground points bold show results averaged across three different seeds while background points, shown feint, indicate results from the underlying individual seeds.
wylo AED and NP Mix
92.5
wo ADD and NP. Mix
NP-Xix
87.5
AUROC
P97195
60-
86.0
82.5
60-
80.0
77.5
75.0
Enetgy
GEN
KNN
VIM
Ecsegy
GEN
KNN
VIM
Figure 13: Experiments using five random dataset splits for Multimodal Near-OOn Detection on the HMDB51 25/26. Foreground points in bold show results averaged across five different splits while background points, shown feint, indicate results from the underlying individual splits.
G Statistical Significance Tests
Different Random Seeds. We run each experiment three times using different seeds for Multimodal Near-OOD Detection the HMDB5 25/26 dataset in our MultiOOD benchmark, and then calculate the mean AUROC and FPR95 to demonstrate the statistical significance of our methods. As shown in Fig. 12, training with A2D and NP-Mix is statistically stable and significantly surpasses the baselines under different random seeds.
Different Random Splits. We each experiment five times using different dataset splits for Multimodal Near-OOD Detection on the HMDB51 25/26 dataset in our MultiOOD benchmark, and then calculate the mean AUROC and FPR95 to demonstrate the statistical significance of methods. As shown in Fig. 13, training with A2D and NP-Mix is statistically stable and surpasses the baselines significantly under different dataset splits.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction clearly state the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In the conclusion part, we discussed limitations and future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Our paper discloses all the information needed to reproduce the main exper- imental results. Our source code and benchmark datasets are provided Supplementary Material and will also be made publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the code Supplementary Material reproduce the main experi- mental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ httppsscccppbpiccccceeceeecc guides/ CodeeSbbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and details in the paper and in the provided code.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide Statistical Significance Tests in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details in implementation details part.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : nnurrps....pp// oc cc/public/ htte/ttttiitiielleeeeeeeeaee
Answer: [Yes]
Justification: Our paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They properly credited in our paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new assets introduced in the paper are well documented.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "8jatLcNSOZ", "reviewer": "Reviewer_Lt2v", "summary": "The paper introduces a novel OOD detection benchmark for multimodal data (called MultiOOD), covering diverse dataset sizes and modalities. Based on this benchmark, the authors first demonstrate the Modality Prediction Discrepancy phenomenon, which means that \nthe discrepancies of softmax predictions are shown to be negligible for in-distribution data (across different modalities) and significant for OOD data. Based on these observations, the authors introduce a novel Agree-to-Disagree (A2D) algorithm, which aims to enhance\nsuch discrepancies during training. Additionally, the authors propose a new outlier synthesis algorithm NP-Mix that explores broader feature spaces and complements A2D to strengthen the OOD detection performance. Experimental validation is extensive and confirms the improvements in OOD detection due to the new proposed algorithms.", "questions": "Here are my concerns:\n- In the video modality, what features do you extract to characterize the stream? Do you extract per-frame features, or per-video features.\n- Same question for the audio modality\n- You use 5 datasets, all of them having the video and optical flow modalities, but audio modality is missing from 2 of them. In this case, what protocol do you adopt? Do you ignore them and perform the audio analysis only on the remaining three?\n- Is your approach robust when one modality is missing?", "limitations": "The identified limitations are: (i) the performance on Near-OOD benchmark with a large number of classes can be further improved; and (ii) the ID/OOD discrepancy could also be further improved. The paper does not represent any negative societal impact.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Paper's strengths:\n- the paper has a significant scientific contribution, by introducing the first multi-modal benchmark for OOD detection\n- besides this, it contributes to the improvement of OOD detection by introducing two new algorithms:  A2D algorithm and a new outlier synthesis algorithm NP-Mix that explores broader feature spaces and complements A2D\n- the paper is clearly written and well-documented\n- the review of the state of the art is comprehensive and covers most of the relevant work\n- experimental validation is extensive and underline the improvements introduced by the A2D and NP-Mix algorithms for OOD detection", "weaknesses": "- Some more details are required regarding some aspects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful reviews and great support of our paper! We provide the responses to your questions as follows:\n\n>**Q1**: In the video modality, what features do you extract to characterize the stream? Do you extract per-frame features, or per-video features.\n\n**A1**: Thanks for your valuable question! We use the SlowFast [a] network as the backbone to extract per-video features. The SlowFast model involves a Slow pathway operating at a low frame rate to capture spatial semantics and a Fast pathway operating at a high frame rate to capture motion at fine temporal resolution. \n\n[a] Feichtenhofer, et al. Slowfast networks for video recognition. In: ICCV, 2019\n___\n\n>**Q2**: Same question for the audio modality\n\n**A2**: We also extract per-video features for the audio modality. Each 10-second audio waveform is converted into one spectrogram and then inputted to the ResNet-18 audio encoder. Audios that are less than 10 seconds are padded to 10 seconds.\n___\n\n>**Q3**: You use 5 datasets, all of them having the video and optical flow modalities, but audio modality is missing from 2 of them. In this case, what protocol do you adopt? Do you ignore them and perform the audio analysis only on the remaining three?\n\n**A3**: All datasets have video and optical flow modalities. Therefore, we create four Multimodal Near-OOD benchmarks and two Multimodal Far-OOD benchmarks using video and optical flow, as shown in Figure 2 in the paper. Only EPIC-Kitchens, HAC, and Kinetics have audio modality. We create two challenging Multimodal Near-OOD benchmarks using EPIC-Kitchens and Kinetics with different combination of modalities (video-audio, flow-audio, video-flow-audio), as shown in Table 9 in the paper.\n___\n\n>**Q4**: Is your approach robust when one modality is missing?\n\n**A4**: Thanks for your interesting comment! In our framework, we also train a classifier for each modality to get predictions from each modality separately. By default, we use the predictions obtained from the combined embeddings of all modalities to calculate the OOD score. However, when one modality is missing, we can use the predictions from the remaining modality to calculate the OOD score. We added the evaluations on HMDB51 25/26 under this challenging condition as below. We use Energy as the OOD score.\n\n|   | FPR95$\\downarrow$ | AUROC$\\uparrow$ |\n|---------|----------|----------|\n| Video-only |      64.05    |     83.14    |\n| Flow-only |      71.46    |    75.51  |\n| A2D+NP-Mix (Video) |       47.49  |        86.48    |\n| A2D+NP-Mix (Flow) |     66.01     |     77.23  |\n| A2D+NP-Mix |    36.38     |        88.91   |\n\nWhen one modality is missing, the performance drops a little, especially when the video is missing (A2D+NP-Mix (Flow)). However, compared with training on one modality alone (Video-only and Flow-only), training with A2D and NP-Mix can also bring significant improvements for each modality when another modality is missing. For example, A2D+NP-Mix (Video), the case when optical flow is missing, yields a 16.56\\% relative improvement on FPR95 compared with Video-only. This underscores the importance of multimodal training for Multimodal OOD Detection.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vGxUfrxoAV", "reviewer": "Reviewer_gP94", "summary": "This paper introduces MultiOOD, a new benchmark for multimodal out-of-distribution (OOD) detection. The authors propose two new techniques: 1) Agree-to-Disagree (A2D), which encourages discrepancy between modality predictions during training, and 2) NP-Mix, a novel outlier synthesis method. Extensive experiments on the MultiOOD benchmark demonstrate significant improvements over existing unimodal OOD detection methods.", "questions": "1) Why \"α > 1\" can ensure the synthesized outliers reside in the intermediary space between two prototypes, rather than near the ID data? Please give more explanations.\n2) How sensitive are the A2D and NP-Mix methods to hyperparameter choices?", "limitations": "The authors acknowledge some limitations, such as the focus on specific modalities (video, optical flow, audio) and action recognition tasks. However, they could further discuss potential limitations in scaling to a larger number of modalities or very different types of data.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n-First benchmark for multimodal OOD detection (MultiOOD).\n-A2D algorithm leveraging modality prediction discrepancy.\n-Outlier synthesis method (NP-Mix).\n\nQuality:  \n-Strong performance improvements over baselines.\n-Thoughtful analysis of modality prediction discrepancy phenomenon.\n-Comprehensive experiments on diverse datasets.\n\nClarity:  \n-Well-organized structure.\n-Clear explanations of key concepts and methods.", "weaknesses": "Weaknesses:\n1) Lacks important baselines (e.g. ensemble of multiple singleOOD methods for each modality). This also relates to the question of \"why we should study MultiOOD?\" The importance of studying MultiOOD and its practical impact on real-world applications could be further demonstrated.\n2) Limited theoretical analysis of proposed methods. A2D and NP-Mix need more theoretical analysis. \n3) The experiments are only conducted on the task of action recognition. In this case, the title \"MultiOOD\" seems over-claiming. Please specify the task in the title and the main text. In addition, it only focuses on specific modalities (video, optical flow, audio). This makes the scope of the paper somewhat limited.\n5) Some figures (e.g. Fig. 4) could be improved for clarity, with consistent color-coding and same x-axis scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful reviews, and we appreciate your valuable suggestions! Please find the responses to your questions below:\n\n>**Q1**: Lacks important baselines (e.g. ensemble of multiple singleOOD methods for each modality). This also relates to the question of \"why we should study MultiOOD?\"\n\n**A1**: Thanks for your suggestions! We added evaluations on HMDB51 25/26 for the ensemble of multiple unimodal OOD methods for each modality to demonstrate the importance of studying MultiOOD. Due to space limits, we put the detailed results in the **global response** at the top of this page. The ensemble of multiple unimodal OOD methods always brings performance improvement, but still has a large gap compared with our multimodal solution (A2D+NP-Mix), further demonstrating the importance of studying MultiOOD.\n___\n>**Q2**: Limited theoretical analysis of proposed methods. A2D and NP-Mix need more theoretical analysis.\n\n**A2**: Thanks for bringing up this point. Our A2D is based on the empirical observation of Modality Prediction Discrepancy on our MultiOOD benchmark. This discrepancy can be attributed to the unavailability of semantic information on OOD data during model training, stimulating each modality to generate conjectures based on its unique characteristics upon encountering OOD data during testing. We demonstrate that such discrepancy is highly correlated to the ultimate OOD performance. We also show through extensive experiments that A2D can amplify such discrepancy and enhance the efficacy of OOD detection. For NP-Mix, we give an additional analysis on decision boundaries, as shown in Figure 2 in attached PDF in global response. Vanilla Mixup causes manifold intrusion due to the injection of noisy synthesized outliers. Instead, our NP-Mix only generates synthesized outliers in the intermediary space between two classes, thus avoiding the manifold intrusion and helping the network learn better decision boundaries.\n\nIn summary, Multimodal OOD Detection is in its very early era. Our MultiOOD benchmark aims to fill this gap and our proposed A2D+NP-Mix solution is mostly based on empirical observation from extensive experiments. More interesting solutions and theoretical analyses are expected in future work.\n___\n>**Q3**: The experiments are only conducted on the task of action recognition. the title \"MultiOOD\" seems over-claiming. In addition, it only focuses on specific modalities (video, optical flow, audio). This makes the scope of the paper somewhat limited.\n\n**A3**: Thanks for your insightful comments! To further increase the scope of our paper and demonstrate the versatility of the proposed A2D training, we add another task of 3D Semantic Segmentation using LiDAR point cloud and RGB images. We evaluate on SemanticKITTI [a] dataset and set all vehicle classes as OOD classes. During training, we set the labels of OOD classes to void and ignore them. During inference, we aim to segment the known classes with high Intersection over Union (IoU) score, and detect OOD classes as unknown. We adopt three metrics for evaluation, including FPR95, AUROC, and $mIOU_c$ (mean Intersection over Union for known classes). We use ResNet-34 and SalsaNext [b] as the backbones of the camera stream and LiDAR stream. We compare our A2D with basic LiDAR-only and Late Fusion, as well as two multimodal 3D semantic segmentation baselines PMF [c] and XMUDA [d]. Our A2D also shows strong performance under this new task (**3D Semantic Segmentation**) with different combinations of modalities (**LiDAR point cloud and RGB images**). We will integrate these new benchmark results in our paper to further increase its scope.\n\n||FPR95$\\downarrow$|AUROC$\\uparrow$|$mIOU_c\\uparrow$|\n|-|-|-|-|\n|LiDAR-only|57.78|84.76|59.81|\n|Late Fusion|53.43|86.98|61.43|\n|PMF|51.57|88.13|61.38|\n|XMUDA|55.49|89.99|61.45|\n|A2D (ours)|**49.02** |**91.12**|**61.98**|\n\n[a] Behley, et al. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In ICCV, 2021\n\n[b] Cortinhal, et al. Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds. In ISVC, 2020\n\n[c] Zhuang, et al. Perception-aware multi-sensor fusion for 3d lidar semantic segmentation. In ICCV, 2021\n\n[d] Jaritz, et al. xmuda: Cross-modal unsupervised domain adaptation for 3d semantic segmentation. In CVPR, 2020\n___\n>**Q4**: Some figures (e.g. Fig. 4) could be improved for clarity, with consistent color-coding and same x-axis scales.\n\n**A4**: Thanks for your suggestion! We have improved the figures and will update them in the final version of the paper.\n___\n>**Q5**: Why \"$\\alpha$ > 1\" can ensure the synthesized outliers reside in the intermediary space between two prototypes, rather than near the ID data?\n\n**A5**: Given a $\\lambda$ sampled from distribution Beta($\\alpha$, $\\alpha$), as shown in Figure 1 in the attached PDF in global response, when $\\alpha$ < 1, $\\lambda$ has a very high probability of being close to 0 or 1. As a result, the synthesized data Z = $\\lambda$ * Z1 + ($1-\\lambda$) * Z2 will be close to Z1 or Z2. Instead, when $\\alpha$ > 1, $\\lambda$ has the highest probability near 0.5. As a result, the synthesized data Z = $\\lambda$ * Z1 + ($1-\\lambda$) * Z2 will be in the intermediary space between Z1 and Z2.\n___\n\n>**Q6**: How sensitive are the A2D and NP-Mix methods to hyperparameter choices?\n\n**A6**: For A2D, we analyzed the choices of different distance functions, as shown in Table 5 in our main paper. A2D training exhibits robustness across various distance functions. Regardless of the specific distance metric employed, substantial improvements are consistently observed compared to the baseline approach without A2D training.\n\nWe investigated the parameter sensitivity of NP-Mix on Nearest Neighbor parameter N and Mixup parameter $\\alpha$, as shown in Figure 10 and Figure 11 in our main paper. NP-Mix demonstrates robustness across different parameter settings and yields substantial enhancements in OOD performance for all cases.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A8bdFOX9mP", "reviewer": "Reviewer_z2YR", "summary": "This paper first proposes a MultiOOD benchmark for multimodal OOD detection. It then further proposes methods including A2D and NP-Mix for better tacking the multimodal OOD detection task.", "questions": "Overall, I am now hesitating around the borderline for this paper and below are my concerns:\n\n1. Generally, I believe that the relationship between this work and other multi-modality works can be very close. I thus suggest the authors to also discuss with other multi-modality works in their related works. I will also elaborate on this more in below concerns.\n\n2. As for the benchmark, I believe that it is important for the evaluation to be conducted in a more statistically significant manner. In other words, from my perspective, the authors may consider to at least evaluate methods on different ID/OOD category splits and report standard deviation.\n\n3. W.r.t. A2D in Sec 4.2, from my perspective, it seems that sometimes, the orders of non-ground-truth classes matter. For example, while for a cat input, both fox and truck are non-ground-truth classes, it seems natural for fox to have a higher softmax score than truck. Thus, I am curious, what if the discrepancy is maximized in an order-preserved manner. I appreciate if this can be performed as an ablation study to see whether this alternative of A2D can be more effective.\n\n4. W.r.t. Sec 4.3, the authors claim that they get inspiration from [16]. I thus hope to see a more detailed comparison between the proposed method and the method in [16]. Besides, it seems to also worth comparing with Learning Placeholders for Open-Set Recognition CVPR 2021. If I am not wrong, it seems that the proposed NP-Mix is somehow similar to its data placeholder.\n\n5. As for the evaluation, as mentioned in the first concern, this work is closedly related to those multi-modaity ones. Thus, to better validate the efficacy of A2D and NP-Mix, besides comparing with only uni-modal OOD methods, it is also suggeted to create baselines that combine uni-modal OOD methods and existing typical multi-modality methods outside the OOD area. This can lead the proposed method to be better evaluated.", "limitations": "This work has discussed its potential limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. I appreciate the formulation of the multimodal OOD detection benchmark.\n\n2. This work presents extensive experiments and achieves good results.", "weaknesses": "(See the questions for more details.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful reviews, and we appreciate your valuable suggestions! We address your concerns and questions as follows:\n\n>**Q1**: Discuss with other multi-modality works in related works. \n\n**A1**: Thanks for your insightful suggestion! As shown in **Table 10** in our main paper, we indeed compared A2D and NP-Mix with other multimodal self-supervised training tasks, including Contrastive Loss, Relative Norm Alignment, Cross-modal Distillation, and Cross-modal Translation. A2D and NP-Mix show substantial superiority over other multimodal tasks. We will add a subsection on Multimodal Learning in related work to discuss these works.\n___\n\n>**Q2**: Evaluate methods on different ID/OOD category splits.\n\n**A2**: Thanks for your valuable comment! As shown in **Figure 13** in our main paper, we indeed evaluated our method on **five different ID/OOD category splits**. Training with our A2D and NP-Mix is statistically stable and surpasses the baselines significantly under different dataset splits. We also evaluated our method on **three different random seeds**, as shown in **Figure 12** in our main paper. Our method significantly surpasses the baselines under different random seeds.\n___\n\n>**Q3**: W.r.t. A2D, the orders of non-ground-truth classes matter. what if the discrepancy is maximized in an order-preserved manner.\n\n**A3**: Thanks for your interesting idea! There are two small issues with the idea. One issue is the conflict object with A2D training. A2D aims to enlarge the prediction distance between different modalities for all non-ground-truth classes. If we want to preserve the order of predictions for different modalities, for example, we make the most similar non-ground-truth class have the second-largest prediction for both video and optical flow, the prediction distance between them will decrease as a result. The second issue is on the implementation. For each class, we need to manually select its most similar classes to define the order, which is complicated for datasets with a large number of classes. \n\nWe implement this idea on EPIC-Kitchens 4/4 using video and optical flow and use Energy as OOD score. EPIC-Kitchens 4/4 has four ID classes:  ‘put’, ‘take’, ‘open’, ‘close’. We define 'put' and 'take' as paired classes (most similar) and ‘open’ and ‘close’ as another paired classes. Given an input sample, we preserve its prediction order for both modalities by making the paired classes have the second-largest prediction. As shown below, A2D with order-preserved prediction is better than w/o A2D, but with a large gap compared with the original A2D. Besides, because of the conflict object with A2D, the prediction discrepancy $l_{OOD}-l_{ID}$ for order-preserved A2D is significantly reduced compared with the original A2D.\n\n||FPR95$\\downarrow$|AUROC$\\uparrow$|$l_{OOD}-l_{ID}$|\n|-|-|-|-|\n|w/o A2D|76.68|68.29|0.2696|\n|A2D (order-preserved)|73.13|69.39|0.2866|\n|A2D|**66.98**|**72.45**|0.3987|\n___\n\n>**Q4**: Detailed comparison between the proposed method and the method in [16]. it seems to also worth comparing with Learning Placeholders for Open-Set Recognition CVPR 2021.\n\n**A4**: Thanks for your useful suggestion! The major difference between [16] and our NP-Mix is the availability of real outliers during training. In [16], few labeled outliers are available and they generate more synthesized outliers based on the labeled outliers and unlabeled data. However, in our case, we assume only ID data is available during training and we want to generate synthesized outliers using only ID data. Therefore, [16] can't be used in our case directly. We make a small modification to it and implement it as a baseline. Instead of randomly selecting one sample from real outliers and another sample from its nearest neighbor in unlabeled data for Mixup, we randomly select a sample from one ID class and another sample from its nearest neighbor in other ID classes for Mixup. We also include PROSER [a] as a baseline based on your suggestion. PROSER [a] conducts manifold mixup with the pairs from different classes without considering neighborhood information and will inject manifold intrusion problem, as shown in Figure 2 in the attached PDF in global response. Instead, our NP-Mix explores broader feature spaces by leveraging the information from nearest neighbor classes without noisy synthesized outliers injection and achieves the best performances.\n\n||FPR95$\\downarrow$|AUROC$\\uparrow$|\n|-|-|-|\n|A2D+NNG-Mix [16]|38.78|88.07|\n|PROSER|44.01|87.52|\n|A2D+NP-Mix (ours)|**36.38**|**88.91**|\n\n[a] Zhou, et al. Learning placeholders for open-set recognition. In: CVPR, 2021\n___\n\n>**Q5**: create baselines that combine uni-modal OOD methods and existing typical multi-modality methods outside the OOD area.\n\n**A5**: Thanks for your suggestions! We added evaluations on HMDB51 25/26 for the ensemble of multiple unimodal OOD methods for each modality to demonstrate the importance of studying MultiOOD. Due to space limits, we put the detailed results in the **global response** at the top of this page. The ensemble of multiple unimodal OOD methods always brings performance improvements, but still has a large gap compared with our multimodal solution (A2D+NP-Mix), further demonstrating the importance of studying MultiOOD.\n\nFor baselines on typical multimodal methods outside the OOD area, we already compared A2D and NP-Mix with four multimodal self-supervised training tasks in Table 10 in our main paper. Here, we further include two multimodal baselines Gradient Blending [b] and SimMMDG [c]. A2D and NP-Mix show substantial superiority over other multimodal methods.\n\n||FPR95$\\downarrow$|AUROC$\\uparrow$|\n|-|-|-|\n|Gradient Blending|42.92|87.28|\n|SimMMDG|42.05|87.91|\n|A2D+NP-Mix (ours)|**36.38**|**88.91**|\n\n[b] Wang, et al. What makes training multi-modal classification networks hard? In: CVPR, 2020\n\n[c] Dong, et al. SimMMDG: A simple and effective framework for multi-modal domain generalization. In: NeurIPS, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their encouraging and insightful comments. We have carefully read through them and provided global and individual responses, respectively. In global responses here, we first add evaluations for a general question on the **ensemble of multiple unimodal OOD methods**. Then, we add our new findings of A2D on a new task (**3D Semantic Segmentation**) with different combinations of modalities (**LiDAR point cloud and RGB images**). We also attach a PDF with figures to better illustrate answers for Reviewer gP94's questions on "Why $\alpha$ > 1 can ensure the synthesized outliers reside in the intermediary space between two prototypes, rather than near the ID data?" and "more detailed analysis on NP-Mix".

>**Ensemble of multiple unimodal OOD methods**

Based on the suggestions from Reviewer gP94 and Reviewer z2YR, we added evaluations on HMDB51 25/26 for the ensemble of multiple unimodal OOD methods for each modality to demonstrate the importance of studying MultiOOD. We first evaluate the ensemble of different OOD scores on a single modality. Specifically, we choose three different OOD scores for ensemble: probability space (MSP), logit space (Energy), and feature space (Mahalanobis). For all three scores, we normalize their values between 0 and 1 and calculate the ensemble score as score = $\alpha$ * score\_1 + ($1-\alpha$) * score\_2. For $\alpha$, we do a grid search from 0.1 to 0.9 with a 0.1 interval and report the best performance. As shown below, combining MSP or Energy with Mahalanobis can bring significant improvement, especially for video. However, there is still a large gap compared with our best multimodal OOD detection solution (A2D+NP-Mix), demonstrating the importance of studying MultiOOD.

||FPR95$\downarrow$|AUROC$\uparrow$|
|-|-|-|
|Video-only (MSP)|60.78|84.39|
|Flow-only (MSP)|70.37|72.97|
|Video-only (Energy)|64.05|83.14|
|Flow-only (Energy)|71.46|75.51|
|Video-only (Mahalanobis)|51.20|81.25|
|Flow-only (Mahalanobis)|89.98|59.38|
|Video-only (MSP+Energy)|63.18|84.19|
|Video-only (MSP+Mahalanobis)|41.61|86.44|
|Video-only (Energy+Mahalanobis)|44.44|86.62|
|Flow-only (MSP+Energy)|70.15|73.40|
|Flow-only (MSP+Mahalanobis)|66.88|73.59|
|Flow-only (Energy+Mahalanobis)|65.36|74.69|
|A2D+NP-Mix (ours best)|**33.77**|**90.05**|

We then evaluate the ensemble of various OOD scores on different modalities and calculate the ensemble score as score = $\alpha$ * score\_video + ($1-\alpha$) * score\_flow. In this case, we use the Video-only and Flow-only models as above. For $\alpha$, we also do a grid search from 0.1 to 0.9 with a 0.1 interval and report the one with the best performance. As shown below, combining more modalities always brings performance improvements, but still has a large gap compared with our A2D+NP-Mix, further demonstrating the importance of studying MultiOOD.

||FPR95$\downarrow$|AUROC$\uparrow$|
|-|-|-|
|Video (MSP) + Flow (MSP)|50.98|85.40|
|Video (Energy) + Flow (Energy)|49.89|85.38|
|Video (Mahalanobis) + Flow (Mahalanobis)|52.07|81.27|
|Video (MSP) + Flow (Energy)|46.62|86.25|
|Video (Energy) + Flow (MSP)|50.98|83.69|
|Video (MSP) + Flow (Mahalanobis)|57.30|84.68|
|Video (Mahalanobis) + Flow (MSP)|49.02|82.92|
|Video (Mahalanobis) + Flow (Energy)|47.71|83.51|
|Video (Energy) + Flow (Mahalanobis)|59.91|81.96|
|A2D+NP-Mix (ours best)|**33.77**|**90.05**|
___

>**Scope of the paper beyond action recognition using video, optical flow, and audio**

Based on the suggestions from Reviewer gP94, we add another task of 3D Semantic Segmentation using LiDAR point cloud and RGB images, to further increase the scope of our paper and demonstrate the versatility of the proposed A2D training. We evaluate on SemanticKITTI [a] dataset and set all vehicle classes as OOD classes. During training, we set the labels of OOD classes to void and ignore them. During inference, we aim to segment the known classes with high Intersection over Union (IoU) score, and detect OOD classes as unknown. We adopt three metrics for evaluation, including FPR95, AUROC, and $mIOU_c$ (mean Intersection over Union for known classes). We use ResNet-34 and SalsaNext [b] as the backbones of the camera stream and LiDAR stream. We compare our A2D with basic LiDAR-only and Late Fusion, as well as two multimodal 3D semantic segmentation baselines PMF [c] and XMUDA [d]. Our A2D also shows strong performance under this new task (**3D Semantic Segmentation**) with different combinations of modalities (**LiDAR point cloud and RGB images**). We will integrate these new benchmark results in our paper to further increase its scope.

||FPR95$\downarrow$|AUROC$\uparrow$|$mIOU_c\uparrow$|
|-|-|-|-|
|LiDAR-only|57.78|84.76|59.81|
|Late Fusion|53.43|86.98|61.43|
|PMF|51.57|88.13|61.38|
|XMUDA|55.49|89.99|61.45|
|A2D (ours)|**49.02** |**91.12**|**61.98**|

[a] Behley, et al. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In ICCV, 2021

[b] Cortinhal, et al. Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds. In ISVC, 2020

[c] Zhuang, et al. Perception-aware multi-sensor fusion for 3d lidar semantic segmentation. In ICCV, 2021

[d] Jaritz, et al. xmuda: Cross-modal unsupervised domain adaptation for 3d semantic segmentation. In CVPR, 2020

Meta-review: This paper looks at the problem of out-of-distribution detection in multi-modal settings, presenting a new benchmark MultiOOD that has a range of diverse dataset sizes and modalities stemming from five different video datasets. After benchmarking on this dataset, the authors observe that softmax predictions across the modalities are similar within distribution but disagree widely out-of-distribution, and show empirically that this tends to correlate with poor OOD performance. An Agree-to-Disagree algorithm is developed to drive agreement on the correct class and disagreement on the incorrect classes, along with outlier synthesis, showing state of art results on this benchmark.

  The reviewers all agreed that the paper introduces an interesting benchmark, has good contributions with respect to the algorithmic improvements, and is clearly written. Some clarifications and concerns were raised, including clarification of the architecture (e.g. per-frame or per-video feature usage, protocol and robustness to missing modalities, lack of comprehensiveness of baselines (e.g. ensembles), and extensibility to domains other than activity recognition. The authors provided a comprehensive rebuttal including new results that, as mentioned by the reviewers, addressed most of the concerns. 

  Overall, taking into account all of the information I recommend acceptance of this paper. The benchmark is likely to be highly impactful as multi-modal models in general become ubiquitous, and the additional algorithmic contributions are both interesting, uniquely adapted to the problem, and effective. I recommend that the authors include the many additional results and exposition in the camera-ready paper.
Decision: Accept (spotlight)