Out-Of-Distribution Detection with Diversification (Provably)
Authors: Haiyun Yao
Zongbo Han
Huazhu Fu
Xi Peng
Qinghua Hu
Changqing Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: OOD detection
Primary area: safety_in_machine_learning
TL;DR: Our theory and experiments demonstrate that training with diverse auxiliary outliers enhances OOD detection performance.

Abstract: Out-of-distribution (OOD) detection is crucial for ensuring reliable deployment of machine learning models. Recent advancements focus on utilizing easily accessible auxiliary outliers (e.g., data from the web or other datasets) in training. However, we experimentally reveal that these methods still struggle to generalize their detection capabilities to unknown OOD data, due to the limited diversity of the auxiliary outliers collected. Therefore, we thoroughly examine this problem from the generalization perspective and demonstrate that a more diverse set of auxiliary outliers is essential for enhancing the detection capabilities. However, in practice, it is difficult and costly to collect sufficiently diverse auxiliary outlier data. Therefore, we propose a simple yet practical approach with a theoretical guarantee, termed Diversity-induced Mixup for OOD detection (diverseMix), which enhances the diversity of auxiliary outlier set for training in an efficient way. Extensive experiments show that diverseMix achieves superior performance on commonly used and recent challenging large-scale benchmarks, which further confirm the importance of the diversity of auxiliary outliers.

Out-Of-Distribution Detection with Diversification
(Provably)
Haiyun Yao', Zongbo Han', Huazhu Fu², Xi Peng', Qinghua Hu', Changqing Zhang College of Intelligence and Computing, Tianjin University'
Institute of High Performance Computing, A A*STAR?
College of Computer Science, Sichuan University? (yaohaiyun, zongbo, huqinghua, zhangchangqing) @rju.edu.cn, hzfu ieee.org, pengx.gm gm gmail.com
Abstract
cuttof-diitribution (OOD) detection is crucial for ensuring reliable deployment of machine learning models. Recent advancements focus on utilizing easily ac- cessible auxiliary outliers (e.g., data from the web or other datasets) in training. However, we experimentally reveal that these methods still struggle to generalize their detection capabilities to unknown OOD data, due to the limited diversity of the auxiliary outliers collected. Therefore, we thoroughly examine this problem from the generalization perspective and demonstrate that more diverse set of auxiliary outliers essential for enhancing the detection capabilities. However, in practice, it is difficult and costly to collect sufficiently diverse auxiliary outlier data. Therefore, we propose a simple yet practical approach with a theoretical guar- antee, termed inversity-inducee enhances the diversity of the auxiliary outlier set for training in an efficient way. Extensive experiments show that diverseMix achieves superior performance on commonly used and recent challenging large-scale benchmarks, which further confirm the importance of the diversity of auxiliary outliers. Our code is available at htttss//giitub..om/Haaiiy Yao/diverse/i/
Mixup for OOD detection (diverseMixx, which
Introduction
The OOD problem occurs when machine learning models encounter data that differs from the dis- tribution of training data. In such scenarios, models may make incorrect predictions, leading to safety-critical issues in real-world applications, c.g., autonomous driving [14] and medical diagno- sis (27). To ensure the reliability of the outputs of model, it is essential not only to achieve good performance on in-distribution (ID) samples, but also to detect potential OOD samples, thus avoiding making erroneous decisions in test. Therefore, OOD detection has become critical challenge for the secure deployment of machine learning models [
Several significant studies 19: 23 25 focus detecting OOD examples using only ID data in training. However, due to a lack of supervision information from unknown OOD data, difficult for these methods to achieve satisfactory performance in detecting OOD samples. Recent methods 20 46 6, 34] involve training model with casily available auxiliary outliers (c.g., data from the web other datasets), with the hope that the detection ability generalize to unknown OOD. However, as shown in Fig. T [b), we experimentally find that while the use of outlier datasets can enhance performance in OOD detection, the generalization capabilities of these methods remain significantly limited. Specifically, there a considerable risk the model overfitting to the auxiliary outliers,
CCoresspodiing authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ID data (abel )
data laabe)
ID data ( label: 2
auxiliary outliers
mixed outliers
0OD seare
high
low
(a) auxiliary outliers
(h) with auxiliary outilers
(c) diverse outilers
(d) mixed outilers (mumber commpponntss 10)
number of components: (0) ( (number of compenents: 1000)
Figure 1: OOD score for different training strategies. The ID data Xin C R² is sampled from three distinct Gaussian distributions, each representing a different class. The auxiliary outliers are sampled from Gaussian mixture model away from the ID data, where the number of mixture components indicates the number of classes contained in auxiliary outliers. (a) The model trained without auxiliary outliers fails to detect OOD, (b) Incorporating less diverse set of auxiliary outliers (I0 classes) during training enables partial OOD detection, but overfits auxiliary outliers. (c) OOD detection is improved with more diverse set of auxiliary outliers (1000 classes). (d) diverseMix enriches the diversity of outliers (10 classes) through creating significantly distinct mixed outliers.
consequently failing identify OOD samples that deviate markedly. The above limitation motivates the following important yet under-explored question: What are the theoretical principles underlying these methods that enable better utilization of outliers?
In this work, we theoretically investigate this crucial question from the perspective of generalization ability 3. Specifically, we first conduct theoretical analysis to demonstrate how the distribution shift between auxiliary outlier training set and test OOD data affects the generalization capability of OOD detector. Accordingly, a generalization bound is induced on the test-time OOD detection error of classifier, considering both empirical error and the error caused by the distribution shift between deduce intuitive conclusion that
test OOD data and auxiliary outliers. Based the theory, more diverse set of auxiliary outliers can reduce the distribution shift error and effectively lower the upper bound of the OOD detection error. As shown in Figure [b)((c). big)re([b))(c)., the model trained with more diverse set of auxiliary outliers achieves better OOD detection. However, in practice, is difficult and costly to collect sufficiently diverse outlier data. Therefore, natural question arises how to guarantee the effective utilization of a f.fix set of auxiliary outliers?
Inspired from the theoretical principles, we propose simple yet effective method called Diversity- induced Mixup (diverseMix) for OOD detection, which introduces and improves the mixup strategy to enhance the outlier diversity. Specifically, diverseMix employs semantic-level interpolation to generate mixed samples, creating new outliers that significantly deviate from their original coun- terparts. Given the risk that random interpolation strategy (merely sampling from a predefined prior distribution) might produce mixed outliers that are unhelpful for the model (as the model can already detect them effectively), diverseMix dynamically adjusts its interpolation strategy based on original samples. This adjustment ensures that the generated outliers are novel and distinct from those previously encountered by the model, thereby enhancing diversity throughout the training process. As shown in Figure (b)-d), diverseMix effectively boosts the diversity of auxiliary outliers, leading improved OOD detection performance. The contributions of this paper are summarized follows:
We provide a theoretical analysis of the generalization error linked to methods trained with auxiliary outliers. By establishing an upper bound for expected error, we reveal the connection between auxiliary outlier diversity and the upper bound of OOD detection error. Our theoretical insights emphasize the importance of leveraging diverse auxiliary outliers to enhance the generalization capacity of the OOD detector.
Constrained by the prohibitive cost of collecting outliers with sufficient diversity, we propose the Diversity-induced Mixup (diverseMix) for OOD detection, a simple yet effective strategy which theoretically guaranteed improve OOD detection performance.
The proposed diverseMix achieves state-of-the-art OOD detection performance, outperform- ing existing methods on both standard and recent large-scale benchmarks. Remarkably, our method exhibits significant improvements over advanced methods, showing relative

---

performance improvements of 24.4% and 43.8% (in terms of FPR95) on the CIFAR-10 and CIFAR-100 datasets, respectively.
2
Related Works
We provide brief review of prior research relevant to our work followed by comparison.
Mukiliary-Outlie--Pie OOD Detection. One early work by [99 pioneered the field of OOD detection, introducing a baseline method based on maximum softmax probability. However, it has since been established, as noted by [36. that this approach is not quite suitable for OOD detection. To address this, various methods have been developed that operate in the logit space to enhance
OOD detection. These include ODIN [25], energy score 46 28 45
ReAct [4). logit normalization
48 Mahalanobis distance [23]_ and KNN-based scoring 41
However, post-hoc OOD detection
methods that do not involve pre-training on a substantial dataset generally exhibit poorer performance
compared to methods that leverage auxiliary datasets for model regularization [13]
OOD Detection with Auxiliary Outliers. Recent advancements in OOD detection have focused on incorporating easily available auxiliary outliers into the model regularization process. Outlier exposure 20 encourage models to predict uniform distributions for outliers, and Energy-based learning [46] widens the energy gap between ID and OOD distribution. However, performance heavily depends on outlier quality. ATOM (6, POEM 34 and DOS [22 enhance performance by
improving the sampling strategy for auxiliary outliers. DivOE [59] and DAL [47] improve outlier quality in a learnable manner, either in the sample space or feature space, respectively. Additionally, DOE implicitly enhances outlier informativeness through model perturbation. Incorporating outliers during training often achiveves superior performance, as shown in many other works 44 2
Comparison with Existing Methods. Several existing methods have explored the utilization of mixup in OOD detection. MixOE (56) and OpenMix (58) perform mixup between ID data and outliers, linearly representing the transition from ID to OOD and thus enhancing the model capturing the uncertainty from outliers. Meanwhile, MixOOD (5I) employ mixup on ID data to generate outliers for training. Different from existing research which primarily focuses on refining mixup strategy or designing outlier regularization method, we place emphasis on the theoretical significance of auxiliary outlier diversity. Our approach advances this concept by enhancing outlier diversity via mixup based strategy, guaranteed by a robust theoretical framework. This focus on enhancing the diversity of auxiliary outliers distinguishes our research from prevailing studies in this area.
Theory: Diverse Auxiliary Outliers Boost OOD Detection
In this section, we lay the foundation for our analysis of OOD detection. We begin by introducing the key notations for OOD detection in Sec. 3.1 In Sec. 3.2 we establish a generalization bound which highlights the critical role for auxiliary outliers in influencing the generalization capacity of OOD detection methods. Finally, in Sec. 3.3] we demonstrate how a diverse set of auxiliary outliers effectively mitigate the distribution shift errors, consequently lowering the upper bound of error. For detailed proofs, please refer to Appendix
3.1 Preliminaries
We consider the multi-class classification task and each sample in the training set Dia { y/ M is drawn i.i.d. from the joint distribution PxsxxXis. where X'a denotes the input space of ID data, and Vid = [1, ...... K} represents the label space. OOD detection can be formulated as binary classification problem to learn a hypothesis h from hypothesis space H C {h X' + {}} such that f outputs for any E Xid and 0 for any x Xood, where Xood X' Xid represent the input space of OOD data and X' represents the entire input space in the open- world setting. To address the challenge posed by the unknown and arbitrariness of OOD distribution Px.... we leverage an auxiliary dataset Daua drawn from the distribution PA.... to serve as partial OOD data, where Xaux C Xxodd Due to the diversity of real-world OOD data, auxiliary outliers cannot fully represent all OOD data, so Px,... + Px.... We aim to train a model on data sampled from Px= = ktrain Px., + +11 ktrain ( Pxx... to obtain a reliable hypothesis that can effectively generalize to the unknown test-time distribution Px AxexPP. + (1 krest) ) Xonay where Kerain and kteat
3

---

determine the proportion of ID and OOD data used for training and testing, respectively. Note that krest unknown due to unpredictable test data distribution.
3.2 Generalization Error Bound in OOD Detection
Basic Setting. We define an OOD label function which provides ground truth labels (OOD or ID) for inputs as f X' + [0. 1]. The expectation that a hypothesis h disagrees with f with respect to a distribution P is defined as:
cp (p f) Ee<p(1h(x) -())))))))))))]])--)))
(1)
The set of ideal hypotheses on the training data distribution Px= and test-time data distribution Px is defined as:
H Houz= h = arg min
( (1, f); Hood h = ang min <px [), bEH
hER
and we define hand and haur as the element in Hood and Hauz: respectively, which can be denoted as bood Hoodd Naue E HHass Considering that Xaaz Xood, it follows that H C H reflecting the reality that hypotheses perform well on real-world OOD data also perform well on auxiliary outliers, conditioning that auxiliary outliers subset of real-world OOD data. The generalization error of an OOD detector defined as:
GError (h) = CxMPx (h,
(3)
Now, we present our first main result regarding OOD detection (training with auxiliary outliers).
Theorem (Generalization Bound of OOD Detector). We let Derasn Did U consisting of M samples. For any hypothesis h E H and 0 < 8 < 1, with probability of at least - 8, the following inequality holds:
In(
GError(h) S PxMP (h, + c(h, (h, + sup CxNPx (h, haod) + Rm (H) heHfau
+1
+ 3,
2M
(4)
empirical error
reducible
complexity
distribution shift evior
where & (h, is the empirical error. We define ch,, hanz) (xx Wa (x) as the reducible error, where éx and Φx is the density function of Pxx and Px respec- tively. sup <z~Px (h, tod) is the distribution shift error, R., (H) represents the Rademacher heH
complexity, and 3 some constant condition on the error related to ideal hypotheses.
Minimizing empirical risk optimizes the model to l E Huua leading to a reduction in the reducible error, which tends to zero. However, the inherent distribution shift error between auxiliary outliers and real-world OOD data remains constant and non-negligible. This limitation fundamentally restricts the generalization of OOD detection methods trained with auxiliary outliers. To address this limitation, we investigate the effect of outlier diversity on mitigating the distribution shift error.
3.3 Generalization with Auxiliary OOD Diversification
In this paper, the diversity refers to semantic diversity, where a formal definition is given as follows.
Definition 1 (Diversity of Outliers). We assume Xoaz can be divided into distinct semantic groups: Kauæ X _UXU,, where each group X contains data points with label y, Considering a dataset Daio sampled from the distribution Pxa.. where Xaiv C Xood encompasses Xaus and an additional group Xnow Dm++ U X with different semantic compared to Naua. i.e. Xaso Xauz U Knew, we define D div is more diverse than D aue terms of the range of semantic classes covered.
We consider the hypothesis set H. to consist of fully-connected ReLU network with width dm, S n n 44 where n is the input dimension.

---

Suppose we could use this diverse auxiliary outliers dataset for training, the ideal hypotheses achieved by training with Daxo are denoted as:
Hadiv h = arg min C NEH
P Kas (h. f),
(5)
with Px Px kitrin Px, + (1 ktrain ) Xace performing well on Px Xas also perform well on Px Xxaus giving rise to Hai C Haue Consequently, we have:
Because Naux C Xdiv holds, the hypotheses
sup Ca~Px (h, hood) S sup he NEN
CxmPx (h, h Nood),
which indicates that training with more diverse set of auxiliary outliers reduce the distribution shift error. Furthermore, effective training leads to sufficient small empirical error and reducible error, and the intrinsic complexity of the model remains constant. Consequently, more diverse set of auxiliary outliers results in a lower generalization error bound. This theorem is formalized as:
Theorem (Diverse Outliers Enhance Generalization). Let (GEEror())) and D(GError(hato)) represent the upper bounds of the generalization error of detector training with vanilla auxiliary outliers D Daua and diverse auxiliary outliers Daio, respectively. For any hypothesis h and havv in H. and 0 < 5 < 1, with a probability of at least 1 - 8, the following inequality holds
O(GError(baae)) < ((GError(t))
Remark. Theorem 2 highlights that the diversity of the outlier set a critical factor in reducing the upper bound of generalization error. However, despite the fundamental improvement in model generalization achieved by increasing the diversity of auxiliary outliers, collecting a more diverse set of auxiliary outliers is expensive, and the auxiliary outliers we can use are limited in practical scenarios, which hinders the application of outlier exposure based methods for OOD detection. This raises an intuitive question: can we enhance the diversity of a fixed outlier set for better utilization?
4 Method: Diversity-induced Mixup (diverseMixx)
In this section, we show how diverseMix addresses the challenge of effective training when the outlier diversity is limited. We begin with a theoretical analysis demonstrating the effectiveness of mixup in enhancing outlier diversity to improve OOD detection performance, providing a reliable guarantee for our mixup-based method. Then, introduce simple yet effective framework implementing our method diverseMix to enhance OOD detection performance.
4.1 Theoretical Insights: Semantic Interpolation Guarantees Enhanced Diversity of Outliers
Mixup (55 is widely used machine learning technique augment training data by creating synthetic samples, which has been extensively utilized in various studies [7 7 [2]. It involves generating virtual training samples (referee to as mixed samples) through linear interpolations between data points and corresponding labels, given by:
f = Ar; + (1 A)æz,
i Ays + (1 A)95-
(8)
where (ry, y:) and (xy. are two samples drawn randomly from the empirical training distribution, and X E [0, 1] is usually sampled from Beta distribution with parameter a denoted as Beta(o, a). This technique assumes a linear relationship between semantics (labels) and features (in data), allowing us to create new mixed samples that deviate significantly from the semantics of the original ones by combining features from samples with distinct semantics. These new mixed samples are situated outside of the original data manifold [16]. We summarize this assumption as follows:
Assumption (Semantic Change under Mixup). Let xs and x, be any two data points from imput spaces X and X respectively, where yi and yj are corresponding semantic labels and ys # yj- If Ç < << 1-C, then there exists a positive value Ç such that the mixed data point àt Xx, + (1 A)a does not belong to either X or X
This assumption suggests that we can enhance the diversity of outliers by generating new outliers with distinct semantics using mixup. Specifically, applying mixup to outliers in Xave results in some
5

---

generated mixed outliers having different semantics, suggesting that they belong to novel (unknown or unnamed) semantic classes outside of Xaux- Consequently, these mixed outliers can be considered as samples from a broader region within the input space. As per Definition [ the mixed outliers exhibit greater diversity than the original outliers. This lemma formally presented as follows:
Lemma 1 (Diversity Enhancement with Mixup). For a group of mixup transform g acting on the imput space Nouz to generate an augmented input space GXauzz defined as GNaus (#/s
Xz + (1 21, 22 E Xauzz X E [0, }]}, the following relation holds:
Raux C. GRaux.
Lemma establishes that mixed outliers Dini exhibits greater diversity compared to Daurs where Diirr is drawn from distribution Pgx....- Consequently, according to Theorem 2 mixup outliers contribute to reduction in generalization error. We can formalize this relationship as follows, and the detailed proof's can be found in Appendix
Theorem (Mixed Outlier Enhances Generalization), Let ((GEror)))) and PGGError(hmssse represent the upper bounds of the generalization error of detector training with vanilla auxiliary outliers D Daus and mixed auxiliary outliers ' Dinan respectively. For any hypothesis and hmiia in H. and 0 < 8 < 1, with probability of at least 1 - 6, we have
((EError(Mmie)) S O(GError(h)).
(10)
Theorem 3 demonstrates that mixup enhances auxiliary outlier diversity, reducing the upper bound of generalization error in OOD detection, which provides a reliable guarantee of mixup's 's effectiveness in improving OOD detection. However, the vanilla mixup lacks flexibility, which may generate outliers that not necessarily beneficial to the model. Next, we will provide implementation of our method which dynamically adjusts the interpolation strategy in a data- adaativeee manner.
4.2 Implementation
Considering a classifier network and F(x, 0) denotes the logit outputs for input æ, our goal is to use the scoring function ((x, 8) to develop an OOD detector:
G(x) ID (S( 0) 2 7 + OOD 1[5(z,0) <>},
(11)
where 1(-} is the indicator function, 7 is the threshold, typically chosen to ensure that significant proportion (e.g., 95%) of ID data is accurately identified. The training objective is given by:
arg min (ee Fx. 0), 3)] + w Laux;
(12)
where (-E(-) is the cross entropy loss, Laux serves as regularization term enabling model to learn from auxiliary outliers with low-confidence predictions, and controls the strength of regularization.
Our previous analysis showed that semantic interpolation can increase the diversity of outliers, thereby enhancing the model's OOD detection performance. However, the interpolation weights in vanilla mixup is randomly sampled from a preset prior distribution (e.g. beta distribution), which may result in generating mixed outliers that are not necessarily beneficial to the model. To efficiently increase the diversity of auxiliary outliers, we dynamically adjust the mixup strategy based on the original outliers, thereby generating novel mixed outliers which are more likely be unfamiliar the model.
During each training epoch, outliers are regularized, prompting the model 0 to assign lower scores to previously encountered outliers. Consequently, outliers that achieve higher scores S(x, 0) are more likely to be novel or previously unseen outliers. We expect the generated outliers to be located in the vicinal space of the novel outliers that have not yet been encountered by the model. To achieve this, we adjust the prior distribution based on scores. Specifically, for outlier samples x; and x; randomly drawn from the empirical auxiliary outlier distribution, the mixed outliers are formulated as follows:
f λx; + (1 A)æ5, X Beta(š,a, Sya),
(13)
where 8; adjusts the original Beta distribution according æ; and zj. which is defined as follows:
exp(S(x,,0)TT) EkEt,,) exp(((#x:))) 0)/T)*
(14)
The set of all possible combinations of data points and all possible values of for mixup.

---

Algorithm I: diverseMix for OOD Detection
Input: ID dataset Dia, outlier dataset Daux, batch size N, distribution parameter o, temperature T.
Output: model parameters 0. for each iteration do
for each mini-batch do
Sample N ID data from Dia as Bia and outliers from Daux as Baxx respectively.
Evaluate the auxiliary outliers Baux using the current model 0 to obtain the scores S. Randomly shuffle Baxx and the corresponding scores S to generate Bixx and S' Generate prior adjustment strategies based scores S and S' according to Eq. 14
Sample the interpolation weight from the adjusted prior distribution and generate mixed outliers mic according to Eq. [3
Train the model 0 using the objective function defined in Eq. 12
end for end for
with T representing the temperature parameter. This adaptive strategy assigns higher weights the outliers that contain more information unknown to the current model, ensuring the generation of novel outliers, thereby increasing diversity throughout the training process. After constructing the mixed auxiliary outliers, they are used for the training objective proposed method shown in Alg. [
12 The whole pseudo code of the
Compatibility with different OOD regularization method. DiverseMix is a general method that is suitable for series of OOD regularization methods. One representative method is the energy-based method 46 which employs the following OOD regularization loss:
Laux = P, (max (0, Min ((;; 0)) + Eza |mmax (0, S(x;
m) (15)
where m4n and Mout are margin hyperparameters, and S(z; = log Σ exp(F. 0)) is the corresponding scoring function. More details for regularization methods provided in Appendix B.3
5 Experiments
In this section, we outline our experimental setup and conduct experiments on common OOD detection benchmarks to answer the following questions: Q1. Effectiveness (I): Does our method outperform its counterparts in OOD detection? Q2. Effectiveness (II): Does method retain its superior performance across various settings including large-scale benchmarks? Q3. Practicability (I): Does our method demonstrate effectiveness across different OOD regularization methods? Q4. Practicability (II): Does our method demonstrate effectiveness in low-quality auxiliary outlier datasets? Q5. Ablation study: (I) Does diverseMix truly offer distinct advantage other data augmentation methods? (II) What is the key factor contributing to performance improvement in our method? Q6. Reliability: Do the experimental results provide strong support for established theory?
5.1 Experimental Setup
We briefly present the experimental setup here, including the experimental datasets and evaluation metrics. Further experimental details can be found in Appendix B
Following the commonly used benchmark in OOD detection literature, CIFAR-10. 10, CIFAR-100 and ImageNet-200 as ID datasets. Auxiliary outlier datasets. For CIFAR experiments, the downsampled version of ImageNet (ImageNettRRC) auxiliary outliers. For ImageNet-200 experiments, the remaining 800 categories from ImageNet-Ik (ImageNer-800) serve as auxiliary outliers. o OOD test sets. For CIFAR benchmark, we use diverse datasets including SVHN (37), Textures (8) P6aces365 (57), LSUN-crop. LSUN-resize (53), and iSUN
Datasets. ID datasets.
employed as
49 For ImageNet benchmark, We use datasets such as SSB-hard [33. NINCO [5), iNaturalist 42
Textures g and Openlmage-O 444)
Evaluation metrics. Following common practice, we report: (1) OOD false positive rate at 95% true positive rate for ID samples (FPR95) (26), (2) the area under the receiver operating characteristic
7

---

Table 1: Main results. Comparison with competitive OOD detection methods trained with the same DenseNet backbone. The performance metrics are averaged (%) over six OOD test datasets from Section [.1 The best results are in bold. diverseMix not only demonstrates stteeoofthe--a OOD detection performance on the CIFAR benchmark but also maintains high accuracy in ID classification.
More details are provided in the Appendix B
CIFAR 10
CIFAR 100
Method
FPR95 (1)
AUROC (1)
AUPR (1)
ID-ACC
FPR95 (L)
AUROC (t)
AUPR (t)
ID-ACC
w/w.D. DANE
MSP
58.98
90.63
93.18
94.39
80.30
73.13
76.97
74,05
ODIN
26.55
94.25
95.34
94.39
56.31
84.89
85.88
74.05
X
Mahulanobis
29.47
89.96
89.70
94.39
47.89
85.71
K7.15
74.05
Energy
28.53
94.39
95.56
94.39
65.87
81.50
84.07
74.05
SSD+
7.22
98.48
98.59
NA
38.32
88.91
89.77
NA
OE
9.66
98.34
98.55
94.12
19.54
94.93
95.26
74.25
SOFL
5.41
98.98
99.10
93.68
19.32
96.32
96.99
73.93
CCU
8.78
98.41
98.69
93.97
19.27
95.02
95.41
74.49
Energy Donz)
4.62
98.93
99.12
92.92
19.25
96.68
97.44
72.39
NTOM
4.00
99.09
98.61
94.26
18.77
96.69
96.49
74.52
POEM
2.54
99.40
99.50
93.49
15.14
97.79
98.31
73.41
MixOE
14.54
97.16
97.41
94.48
27.71
92.93
93.81
75.
DivOE
11.41
97.76
98.18
93.74
18.91
95.00
95,26
74.08
DiveneMix (ours)
1.92
99.42
99.51
94.16
8.51
98.24
98.46
74,60
Table Main results on large-scale ImageNet benchmark. Comparison with competitive OOD
detection methods trained with the same ResNet backbone. For better presentation, the best and
second-best results are in bold and underline respectively. Consistent with CIFAR experiment results, diverseMix demonstrates strong OOD detection capabilities for both near-OOD and far- ROOD test sets, achieving ofate-of-the the---rrr OOD detection performance. Details are provided in the Appendix B
Method
Near OOD
Far-OOD AUROC (f)
Averuge AUROC (1)
ID-ACC
FPR
AUROC
AUPR (1)
FPR
AUPR (1)
FPR
AUPR (1)
MSP
70.35
82.75
88.58
54.51
88.81
91.86
60.85
86.39
90.54
85.81
Energy
70.35
81,88
88.56
53.87
89,30
91.59
60.46
86,33
90.38
85.81
Logits
82.25
88.69
52.49
59.28
85.81
ODIN
69.06
82.20
88.75
50.90
89.90
92.50
58. 16
86.82
91.00
85.81
OE
59.12
86.86
92.69
54.95
90.51
91.20
$6.61
$9.05
91.79
85.52
Energy (w. Dase)
60.67
85,95
91.75
58.07
89.73
89.67
59.11
58.22
90.50
84.94
DPN
63.39
84,94
91.46
61.31
89.85
90.16
62. 14
87,89
90.68
85.27
MixOE
50.51
90.62
92.31
DiverseMix (ours)
59.81
86.36
91.76
48.58
91.35
92.38
$3.07
89.36
92.13
85.95
curve (AUROC) 10, (3) the area under the precision-recall curve (AUPR) [31.
We also provide II)
classification accuracy (ID-ACC).
5.2 Experimental Results and Discussion
DiverseMix achieves superior performance the common benchmark (Q1). Our method outperforms existing competitive methods, establishing state-of-the-ari performance both on CIFAR- 10 and CIFAR- R00 datasets. Table provides a comprehensive comparison with methods grouped into:
(1) ID-only training: MSP 19). ODIN (25). Mahalanobis
(23), Energy 46); ; (2) Utilizing auxiliary
outliers: OE
20 SOFL 35 CCU (32). Energy with outliers (46). NTOM 66. POEM (34), MixOE
DivOE 159 Methods that utilizing auxiliary outliers generally achieve significantly better empirical performance on OOD detection. This implies that leveraging auxiliary outliers essential for enhancing OOD detection performance. Our method diverseMix significantly outperforms the top baseline, reducing the FPR95 by 0.62% and 6.63% on CIFAR- 10 and CIFAR-1OO, respectively. These reductions correspond to relative error reductions of 24.4% and 43.8%. These notable improvements can be attributed to the enhanced diversity auxiliary outliers offered by diverseMix, which lowers the generalization error bound and significantly improves the OOD detection performance.
Diverse is effective on the large-scale benchmark (Q2). Recent studies (50 have suggested that methods leveraging outlier data tend to underperform in more demanding, large-scale OOD detection tasks. To evaluate the effectiveness of our method, we conduct experiments on the ImageNet benchmark. Following
We categorize the OOD test set into two distinct groups: near-OOD and far-OOD. For each group, we report the average performance metrics. Furthermore, we also present the overall average performance on the OOD test sets. Table 2 illustrates that methods requiring outliers during training tend to excel in near OOD detection but fall short in far-OOD detection, sometimes even performing worse than methods that do not require outliers during training. Although MixOE improves far-OOD detection performance to some extent, fails to fully leverage auxiliary outliers to enhance near-OOD detection. In contrast, our method not only maintains strong performance near-OOD detection but also significantly improves performance in far-OOD scenarios. We speculate that the virtual auxiliary outlier data generated by diverseMix may be more
8

---

(a) different outlier size
(b) different outlier diversity
25.0
Energy
DiverseMiix (Energy)
25.0
Energy
Diverselix (Energy)
OE
DiverseMix (OE)
OE
Diverseiix (OE)
k+1
DiverseMix (K+1)
k+1
DiverseMix (K+1)
21.0
FPR95 ())
21.0
FPR95
17.0
17.0
13.0
13.0
8.0
100
70
55
40
25
10
100 number of categories in auxiliary outliers
1000
850
550 400
number of auxiliary outliers (%)
Figure 2: Comparison of OOD detection performance CIFAR-100 with decreased quality of auxiliary outlier datasets (a) With constant diversity of auxiliary outliers (1000 categories), the dataset size is decreased. The x-axis represents the percentage of the original outlier dataset's size used for training. (b) With fixed dataset size (10% of auxiliary outliers), the diversity of outliers is decreased, with the x-axis displaying the number of categories. See Appendix B.4 for more details.
Table 3: Ablation study. Performance are averaged (%) over six OOD test datasets from Section 5.1
The best results are in bold. More details about the comparison methods provided in Appendix B
(a) different data augmentation method.
(b) different semantic interpolation strategy.
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-I00
FPR (4)
AUROC (*)
(4)
AUROC (1)
FPR ({)
AUROC (1)
FPR (()
AUROC (1)
Gaussian noise
98.64
95.69
Vanilla
5.43
16.30
96.87
Cutout
7.20
98.57
19.12
96.64
Mixup
4.00
99.02
12.56
97.42
Color jätter
8.83
98.15
21.36
95.01
Cutmix
5.76
98.79
15.95
97.06
DivenseMix (ours)
1,92
99.42
8.51
98.24
DiverseMix (ours)
1.92
99.42
8.51
98.24
representative of far-OOD data. While most OOD detection methods face difficulties in achieving satisfactory performance across both near-OOD and far-OOD, our method excels in detecting both types of OOD, significantly surpassing other methods the average OOD detection performance.
DiverseMix is general method that achieves good performance across different OOD reg- ularization methods (Q3). To investigate the generality of diverseMix across different OOD regularization methods, we replace the original energy loss with the K+1 loss and the OE loss. The experimental results presented in Figure 2 reveal that diverseMix achieves consistent effectiveness re- gardless of the OOD regularization method employed. These findings not only suggest the versatility of our method but also provide substantial empirical evidence supporting our theoretical framework.
DiverseMix remains effective even when the auxiliary outlier data is of low quality (Q4), In Figure 2 the quality of auxiliary outliers used for training is decreased by gradually decreasing their quantity or their diversity. Our method diverseMix consistently outperforms previous methods by enhancing the diversity of auxiliary outliers across different dataset sizes and diversity levels. This suggests that diverseMix remains effective even when the auxiliary outliers are of low quality.
Sample adaptive semantic interpolation contributing to unique ad adnatages of diverseMix (Q5). We compared MiverseMix with other data augmentation methods. As shown in Table B a). diverseMix demonstrates superior performance for OOD detection over other data augmentation methods that preserve the semantics of outliers. Additionally, the ablation study in Table compares diverseMix with different mixup strategies. DiverseMix outperforms both vanilla mixup and cutmix by adaptively adjusting its interpolation strategy based on the given outliers, thereby efficiently generating novel mixed outlier samples enhance diversity. The advantages of diverseMix lie in 1) enhancing the diversity of outliers at the semantic level, and 2) efficiently boosting diversity by adaptively adjusting its strategy for the given outlier samples. For detailed comparisons, please see Appendix B.6
Our theory effectively demonstrates that the diversity of auxiliary outliers is a key factor to ensure OOD detection performance (Q6). In Figure 2 when maintaining the diversity relatively constant and changing the quantity of data, the performance of different methods remains relatively stable. However, when the number of outliers is fixed and the diversity of the outliers dataset is reduced, there is a significant decrease in performance across all methods. This suggests that diversity is a key quality factor for the auxiliary outliers, providing substantial empirical support for theory.
MiverseMix has the potential for application across a wide range of task domains. Our theory is not rely any assumptions specific to the task domain. Given the successful implementation of

---

mixup across different fields [15 54]. task domains beyond just computer vision tasks. We have investigated the application of diverseMix in the NLP domain through experiments. For additional details, please see Appendix
diverseMix also has the potential for application in multiple
C.2
6 Conclusions and Future Work
In this study, we demonstrate that the performance of OOD detection methods is hindered by the distribution shift between unknown test OOD data and auxiliary outliers. Through rigorous theoretical analysis, we demonstrate that enhancing the diversity of auxiliary outliers can effectively mitigate this problem. Constrained by limited access to auxiliary outliers and the high cost of data collection, we introduce diverseMix, an effective method that enhances the diversity of auxiliary outliers and significantly improves model performance. The effectiveness diverseMix supported by both theoretical analysis and empirical evidence. Furthermore, our theory enables future research to design new OOD detection method. We hope that our research can bring more attention to the diversity in OOD detection.
7
Acknowledgements
This work supported by the National Natural Science Foundation of China (Grant No. No.62376193), the National Science Fund for Distinguished Young Scholars (Grant No.61925602) and the H. Fu's Agency for Science, Technology and Research (A (TAR) Central Research Fund ("Robust and Trustworthy Al system for Multi-modality Healthcare"). The authors also appreciate the suggestions from NeurlPS anonymous peer reviewers.
References
[1] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06550 2016.
[2] Yichen Bai, Zongbo Han, Bing Cao, Xiaoheng Jiang, Qinghua Hu, and Changqing Zhang. Id- like prompt learning for few-shot out-of- distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[3] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 2002.
[4] Shai Ben-David. John Blitzer, Koby Crammer, and Fermando Pereira. Analysis of representations for domain adaptation. Advances in Neural Information Processing Systems, 2006.
[5] Julian Bitterwolf, Maximilian Mueller, and Matthias Hein. In or out? fixing imagenet out- of-distribution detection evaluation. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models, 2023.
[6] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of- distribution detection using outlier mining. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, 2021.
[7] Hsin-Ping Chou, Shih-Chieh Chang, Jia-Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: Rebalanced mixup. In European Conference on Computer Vision, 2020
[8] Mircea Cimpoi, Subhransu Maji, Jasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Conference on Computer Vision and Pattern Recognition, 2014.
[9] Koby Crammer, Michael Kearns, and Jennifer Wortman. Learning from multiple sources. Advances in Neural Information Processing Systems, 2006.
[10] Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc eurves. In International Conference Machine Learning, 2006.
10

---

[11]
Terrance VeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 000:1708804555, 2017.
[12]
Thomas G Dietterich. Steps toward robust artificial intelligence. Ai Magazine, 2017.
[13] Stanislav Fort, Jie Jessie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of- distribution detection. In Advances in Neural Information Processing Systems, 2021.
[14] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition, 2012.
[15] Hongyu Guo. Nonlinear mixup: Out-of-manifold data augmentation for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[16] Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regularization. In Proceedings of the AAAI conference on artificial intelligence, 2019.
[17] Zongbo Han, Zhipeng Liang, Fan Yang. Liu Liu, Lanqing Li, Yatao Bian, Peilin Zhao, Bingzhe Wu, Changqing Zhang, and Jianhua Yao. Umix: Improving importance weighting for subpopu- lation shift via uncertainty- mixup. Advances in Neural Information Processing Systems, 2022.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, 2016.
[19] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribun examples neural networks. International Conference on Learning Representations, 2017.
[20] Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019.
[21] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In Conference on Computer Vision and Pattern Recognition, 2017.
[22] Wenyu Jiang, Hao Cheng, MingCai Chen, Chongjun Wang, and Hongxin Wei. Dos: Diverse outlier sampling for out-o--distribution detection. In International Conference on Learning Representations, 2023.
[23] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, 2018.
[24] Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. Trustworthy ai: From principles to practices. ACM Computing Surveys, 2023.
(25] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of detection in neural networks. In International Conference on Learning Representations, 2018.
image
[26] Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018.
[27] Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, L. Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou. Advances, challenges and opportunities in creating data for trustworthy Al. Nature Machine Intelligence, 2022.
[28] Ziqian Lin, Sreya Dutta Roy, and Yixuan Li. Mood: Multi-level out-of-distribution detection. In Conference on Computer Vision and Pattern Recognition, 2021.
[29] Haochen Liu, Yigi Wang, Wenqi Fan, Xiaorui Liu, Yaxin Li, Shaili Jain, Yunhao Liu, Anil Jain, and Jiliang Tang. Trustworthy Al: A Computational Perspective. ACM Transactions on Intelligent Systems and Technology, 2023.
[30] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. Advances neural information processing systems, 2017.
11

---

[31] Christopher Manning and Hinrich Schutze. Foundations of statistical natural language process- ing. MIT press, 1999.
[32] Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't know. In International Conference on Learning Representations, 2019.
[33] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing Istm language models. In International Conference Learning Representations, 2018.
[34] Yifei Ming, Ying Fan, and Yixuan Li. Poem: Out-of-distribution detection with posterior sampling. In International Conference Machine Learning, 2022.
[35] Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distributie detection. In The AAAI Conference on Artificial Intelligence, 2020.
[36] Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detection. The AAAI Conference on Artificial Intelligence, 2022.
[37] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[38] Salah Rifai, Xavier Glorot, Yoshua Bengio, and Pascal Vincent. Adding noise to the input of a model trained with regularized objective. 2011.
[39] Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: unified framework for self-supervised outlier detection. International Conference Learning Representations, 2021.
[40] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribtiion detection with rectified activations. In Advances in Neural Information Processing Systems, 2021.
[41] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distributioo detection with deep nearest neighbors. In International Conference on Machine Learning, 2022.
[42] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard. Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Conference on Computer Vision and Pattern Recognition, 2018.
[43] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set set recognition: A good closed-set classifier is all you need? arXiv preprint arXiv:2110.06200 2021.
[44] Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distributin with virtual-logit matching. In Conference on Computer Vision and Pattern Recognition, 2022.
[45] Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't 't know? In Advances in Neural Information Processing Systems, 2021.
[46] Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Energy-based detection for multi-label classification. International Conference Learning Representations, 2021.
[47] Qizhou Wang. Zhen Fang, Yonggang Zhang, Feng Liu, Yixuan Li, and Bo Han. Learning to augment distributions for out-of-distribution detection. Advances in Neural Information Processing Systems, 2024.
[48] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overeonfidence with logit normalization. In International Conference on Learning Representations, 2022.
[49] Pingmei Xu, Krista A. Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R. Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv, 2015.
12

---

[50] Jingkang Yang, Pengyun Wang. Dejian Zou, Zitang Zhou, Kunyuan Ding, WenXuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. OpenOOD: Benchmarking generalized out-of- distribution detection. Tiith Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[51] Taocun Yang, Yaping Huang, Yanlin Xie, Junbo Liu, and Shengchun Wang. Mixood: Improving out-of-distribution detection with enhanced data mixup. ACM Transactions on Multimedia Computing, Communications and Applications, 2023.
[52] Huaxiu Yao, Yiping Wang, Linjun Zhang, James Y Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. Advances in neural information processing systems, 2022.
[53] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv: Computer Vision and Pattern Recognition, 2015.
[54] Sangdoo Yun, Scong Joon Oh, Byeongho Heo, Dongyoon Han, and Jinhyung Kim. Videomix: Rethinking data augmentation for video classification. arXiv preprint arXiv:2012.03457, 2020.
[55] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018.
[56] Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mix- ture outlier exposure: Towards out-of-distribution detection in fine-grained environments. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023.
[57] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding. http://artivoorg/as///00000
2016.
[58] Fei Zhu, Zhen Cheng, Xu- Yao Zhang, and Cheng-Lin Liu. Openmix: Exploring outlier samples for misclassification detection. In Conference on Computer Vision and Pattern Recognition, 2023.
[59] Jianing Zhu, Yu Geng, Jiangehao Yao, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Bo Han. Diversified outlier exposure for out-of-distribution detection via informative extrapola- tion. Advances in Neural Information Processing Systems, 2024.
13

---

Appendix
Contents
Theoretical Analysis
14
A.1 Proof of Hood C
14
on2
A.2 Proof of Theorem!
15
Proof of Theorem
17
Proof of Lemmal
18
Experimental Details
18
Details of Dataset
18
B.2
Training Details.
19
B.3
Details of OOD Regularization Method,
19
B.4
Details of Main Experiment.
19
B.5
Details of Figure
20
B.6
Details of Q5 Ablation Study,
20
Additional Results
21
Hyperparameter Analysis.
21
DiverseMix for OOD Detection Natural Language Processing.
................................
22
Experiments on Computational Cost.
22
C.4 Impact Statements
22
D Hardware and Software
23
A Theoretical Analysis
In this section, we provide detailed proof's of our theories and the proposed method, including the proof of Hood S H the establishment the generalization error bound for OOD detection (Theorem [ a more diverse set of auxiliary outliers leads to reduced generalization error (Theorem 2), and the proof of diversity enhancement with mixup (Lemma Tp.
A.1 Proof of Ho.d S Houz
In this section, demonstrate that if 7 consist of fully-connected ReLU network with width dm I n + 4, where n is the input dimension, and given that that X C Xoods follows that Hoad C 7t* H This reflects the reality that hypotheses that perform well real-world OOD data also perform well on auxiliary outliers, conditioning on that auxiliary outliers are a subset of real-world OOD data.
Proof. We first express the expected error of hypotheses h on the training data distribution Px and the unknown test-time data distribution Px follows:
f) (h,
&x [h(x) f(x)ldr Sxcus Ih(x) f(x)ldx SX. [h(z) - f(x) c1,
Sxmaa Xine [1(x) - f(x)
(xx fh, Fx [bxx) f(x)ddx= Ix.. [h(x) - fxx) Sx MMxx) f(x))
Sxie (1(x)) [(x) + Sx... [h(x)) [x))ddx - Sxccd) [h(x) - (xx) dr 61 62-
14

---

From the above expressions, we obtain:
Haua { arg arg min 6);
Hother {h : arg min e2]
Hooo = (h. arg argmin(e + x2)).
Let f' be function that minimizes both E and e2, considering that Jx (z))lrr co, which implies that f' is Lebesgue-interrable on X' The H. represent the fully-connected ReLU networks with width dm I nn+ + 4, where n the input dimension. According to the Universal Approximation Theorem for Width-Bounded ReLU Networks [0). for any > O, there exists a h H such that: E H that simultaneously
Sx [xx) - [xx) de < 6. Consequently, there exists hypothesis minimizes both 6 and 62- leading to the condition Hauzz Hother * 0. In this case, we have min(c1 + (2) = min + min 62- We denote Hood = Hous O Hother thus establishing that Hood C. Haur.
h
A.2 Proof of Theorem
In this section, analyze the generalization error of the OOD detector training with auxiliary outliers. First, we recall the setting from Sec. 3.1] our goal is to train a detector with auxiliary outliers that perform well real-world OOD data. In other words, we aim to train model data sampled from P = Atrainnxx.. + (1 (trrii)Px..... to obtain a reliable hypothesis h that can effectively generalize to the unknown test-time distribution Px krest + (1 k Xood"
Next, we develop bounds on the OOD detection performance of a detector training with auxiliary outliers, which can be formulated as follow:
(Generalization Bound of 0OD Detector). Let Derain Dia aue, consisting of M samples. For any hypothesis h E H. and 0 < 8 < 1, with a probability of at least - 8, the following inequality holds:
In(
GError(h) S. (h, e(h, 1(au))
+ sup ExmPx (h, hood) + Rm +V
+ 3, (16)
heHour
2M
empirical evror
reducibic error
complexity
distribution shift error
where Eh, () is the empirical error. We define e(h, ((oz))
havr (z)ldr /d the reducible error, ox and Φx is the density function of Px and Pg respectively.
sup Cx~Px (h, hood) is the distribution shift color, Rm, (H) represents the Rademacher complexity,
heN
is the етrоr related to ideal hypotheses. The roadmap of our analysis is as follows:
Roadmap. We first show how to bound the OOD detection error in terms of the generalization enror on Px and the maximum distribution shift error as well as the reducible error which can be reduced to small value as the model is optimized. Then, we study the generalization bound from the perspective of Rademacher complexity. We use complexity-basee learning theory to quantify the generalization error on Px. In the end, we bound the OOD detection generalization error in terms of the empirical error the training data, the reducible error, the maximum distribution shift error, and the complexity. We also provide detailed proof steps as follows:
Proof. This proof relies on the triangle inequality for classification error 4 9. which implies that for
any labeling functions fI, fz. and fa. we have f2) S e(fi, fa)
e e fz).
GError(h) = CamPx (h,
<xmPy (h, koud) + ExNPx (hoodd f)
= CewPx (h, haod) + ExNPy (hoods i f) + CowP x (hi hood) E px (h, hiod)
= CewPy (h, Nood) + CemPx (hoods 1) + ExNPx (hi, Wood) Cx x (h, 1ood)
< (h, <xxPxx (hood: - 1) + CxNPx (hoods f) + Campy (h, hood) Exup (h, h Wood) Let ox and ox be the density functions of Px and P P respectively.
15

---

GError(h) s CxNpy (h, f) + Cowpy (hod f) + ComPx (Bood f) Sowpp((nnt + f ox(x))((()) (ood dr - 0z(x))((22) - hood (2)) dx
ExxPx (h, (h, f) + Ce f) + EaNPx (Bood: f)+ / (xx(x) - +x((x) (k(x)) - (x)d(()) da + (xx(x)) - (xx(2)]((())) - haur (x)) dæ + =fl0xe(0) 0x(xx) (Mau (a) dx I CxNPx + CoNpx 2 f) + CowPx (honds f) + (xx(x) - (xx(x)]1((2) - hawr (x) dx + *xx(1z) (2) - (ood(a)) dr + / (xx(()) ( (x) dx = =x (h. + Pxp (hoods f) + CxmPx (Hooods f) + / (x((x) - 2 (x((x) - haur (x)1 dx teamPx (Waner Nood) + cx=Px (hauz: Nood) S ExmPy (h, f) EoWPx (hood f) + CxnPx (hood: f) + (xx(x) +x(x))l1(()) hauz (x)1 dæ texm ( (hauz: Mood) + EENP (hanary f) + EoNP (hoods f),
I Cenpy (h. + ComPy Compy (hoods f) + CowPx (Honds
Given that miner=px (h, f) and mine (h, f) represent the error of haod and hluz on distribu- NEW tions Px and P x. respectively, we have E (hand: f) mine (h, f) and EzWP (hauzi f) kEW
hEH
minc (h. f). Considering that Hood C Hanz. follows that for any h E Hoods h E Wiuw is kEH
holds. As a result, we have Ex~Px (1oodr J) mine [h, Thus, we obtain the following:
hER
GError(h) S CoNPy (h, f) mine (h, f) minez~P, (h, f) hEH
heh
+/ * (0x((0) - -x((2) h(x))
hane (I) dr EzWPx (hauz: hood)
+mine NCH
(h, f) mine (h, f),
x
heft
We can demonstrate that minc Px (h, 2 minc x (h. f) as follows: hEH
hEH
min / [M(x)) f(x))de
minez=Px (b, NEW
NEA
x
( [b(x) (xx)lde + / x
min KEN
(1(x)) - [(x)(dr)
min NEW
[h(x)) - f(x))dx + min
[1(x)) - fx))ddx
x
xxx
min NCW
[h(x) fx))ldx
x
-mine (h,f) keH
Thus, We obtain:
GError(h) S (xm
(h, (6x(x) - -xxx)) [(x) - liaua dx + Ex (hauer lijoa)
+4 mine (h, f), heR
16

---

We denote 3 = 4 min (h, f),
hCH
GError(h) CamPx (hr
1 - howr (x)) do + campy xe Mauus + Bg
Consider an upper bound on the distribution shift error CxuPy ( (h hood)
GError(h) S (xmPx(h, (xx(x) (x(2))1((2) haur( dx
+ sup CxNPx (h, Mand) + B, bEH2ue
Next, we recap the Rademacher complexity measure for model complexity. We use complexity-base learning theory
(Theorem 8) quantify the generalization error. Let Dtrain = Dia Dauz consisting of M samples, C (h, f) is the empirical error of h. Then for any hypothesis in H (i.e..H X + {0, 1), H) and >> 8 > O, with probability at least 6, we have
In(
P (h, < E (h, (H) + V
2M
where Rm (H) is the Rademacher complexities. Finally, it holds with probability of at least 1 - that Cxpx f) (1. + (h,, Nows) + sup
ln(3)
Ca (h, Bood) + R. (H)))) +V
+ B
be
2M
empirical reducible error
complexity
distribution shift error
where c(h, hawar) - (x)) [h(x)- (a))(a)) (x)) da represents the reducible error and / is the error related to ideal hypotheses. Notably, when is large, there exists no detector that performs well on Px, making it unfeasible to find good hypothesis through training with auxiliary outliers.
A.3 Proof of Theorem
In this section, proof that diverse outliers enhance generalization, which can be formulated as follows:
Let ((GEroo((h) and O(GError(haii)) represent the upper bounds of the generalization error of detector training with vanilla auxiliary outliers Duux and diverse auxiliary outliers Daie, respectively.
For any hypothesis h and haso in H. and O < 5 < 1, with probubility of al least 1 - ô, the following inequality holds
((GError(baa)) < O(GEror(h)))
(17)
The detailed proof proceeds follows:
Proof. At first, we prove that diverse outliers correspond smaller distribution shift error than vanilla outliers. Because Naur C Xeiv holds, the hypotheses performing well on Px also perform well on Px giving rise to Hisso C Hauzz
sup ExpPx (ood) S maxt sup Ex~PA (h, hood): Ne
sup heH H
EzNPA (fi, bood) }
he
note that
max sup ConPx (h, Nood); NEH
sup kEH
CoNPx (h, hood))
sup CxNPx (h, hood), REH
Hai
Consequently, we have
sup CxuPx (h, hood)
sup CoNPy (h, haod): he heH:...
(18)
hE
Furthermore, model effective training leads to small empirical error and small reducible error, if we continue use the same model architecture, the intrinsic complexity of the model R,, (H) remains invariant, consider that 3 is a small constant value, therefore, it holds that
((GError((aaie)) I O(EError(h))
(19)
with probability of at least 1-6 8.
17

---

A.4 Proof of Lemmal 1
In this section, we give the proof of the Lemma] which can be formalized as follow:
(Diversity Enhancement with Mixup). For a group of mixup transform 9 acting on the inpur space Xaux generate an augmented input space GKaua defined as GXauz { Xrn (1
X)æ2; 11, 12 E Xaua, X E [0, 1],, the following relation holds:
Kaux C Gaaux.
(20)
Proof. Kaur X U... U XY: U... ... U Nuún U... U X
mis. Consider performing mixup to
obtain a mixed outlier = Ax; + (1 ,)r3, where Z; E X ouu. Ij E Xoaz and y * y/. According to assumption [ there exists À such that & exhibits different semantics from the original, i.e., & & X and 2 & X Clearly, the semantic of à is also inconsistent with other outliers in Xauz- Therefore, f & X Kawn- We define Xmiz { 1 Xauz, â Xx; (1 A)r Fi, 23 Xaux } to represents the input space of mixed outliers with distinct semantic to the original. Consequently, GXauz= Xana U Umiz, leading G.a.ux Xanx.
B Experimental Details
B.1 Details of Dataset
Auxiliary OOD datasets. For CIFAR experiments, we employ the downsampled ImageNet dataset (ImageNet 64 x 64) as a variant of the original ImageNet dataset, comprising 1,281,167 images with dimensions of 64x64 pixels and organized into 1000 distinct classes. Notably, there is overlap between some of these classes and those present in CIFAR-10 and CIFAR- 100 datasets. It is important to emphasize that we abstain from utilizing any label information from this dataset, thereby regarding it as an unlabeled auxiliary OOD dataset. To augment the dataset, we apply random cropping procedure to the 64 64 images, resulting in 32x32 pixel images with a 4-pixel padding. This operation performed with high probability ensures that the resulting images are unlikely to contain objects corresponding to the ID classes, even if the original images featured such objects. Consequently, we retain substantial quantity of OOD data for training purposes, yielding low proportion of ID data within the auxiliary outliers. For conciseness and clarity, we refer to this dataset as ImageNet-RR..o For ImageNet experiments, we selected a subset from ImageNet- KK which includes 200 classes, to serve as the ID data. Images from the other 800 classes are used as auxiliary datasets, following the setting of [50] The resolution for both IDD and auxiliary images are 224 224.
Test OOD datasets. For CIFAR experiments, we follow the setting in 6 34 Specifically, we employ six different natural image datasets
OOD test datasets, while CIFAR- 10 and CIFAR-100 100 serve as our ID test datasets. These six datasets are SVHN B7], Textures S Places365 (57). LSUN (crop), LSUN (resize) [53], and iSUN [49] Below, provide detailed information about these OOD test datasets, all of which consist of 32 x 32 pixel images. o SVHN. The SVHN dataset (37] comprises color images of house numbers, encompassing ten different digit classes from 0 to 9. For our evaluation, we randomly select 1,000 test images from each digit class, creating a new test dataset with 10,000 images. o Textures. The Describable Textures Dataset [8] consists of textural images in the wild. We include the entire collection of 5,640 images for evaluation. o Places365. The Places365 dataset (57] comprises a large-scale photographs depicting scenes classified into 365 scene categories. In the test set, there are 900 images per category. We randomly sample 10,000 images from the test set for our evaluation. o LSUN (crop) and LSUN (resize). The Large-scale Scene Understanding dataset (LSUN) [53] offers a testing set containing 10,000 images from 10 different scenes. We create two variants of this dataset, namely LSUN (crop) and LSUN (resize). LSUN (crop) is generated by randomly cropping image patches to the size of 32 x 32 pixels, while LSUN (resize) involves downsampling each image to the same size. o iSUN. The iSUN dataset (49 is a subset of SUN images. We incorporate the entire collection of 8,925 images from iSUN for our evaluation.
43) In ImageNet experiments, we follow the settings of [50] where Openlmage-O 44 SSB-hard
Textures [8) iNaturalist 42 and NINCO [5) are selected as OOD test datasets. We include SSB-hard and NINCO in the near-OOD group, while the far-OOD group considers iNaturalist, Textures, and OpenImage-O. Openlmage-O contains 17632 manually filtered images and is 7.8 x larger than
'The set of all possible combinations of data points and all possible values of for mixup.
18

---

Table 4: Main results with standard deviation. Comparison with competitive OOD detection methods trained with the same DenseNet backbone. The performance metrics are averaged (%) over six OOD test datasets from Section 5.1 Some baseline results sourced from 34. The best results are in bold. diverseMix not only demonstrates state-of-the OOD detection performance on the CIFAR benchmark but also maintains high accuracy in ID classification.
Method
FPR95 (1)
AUROC
CIFRR-1
AUPR (1)
ID-ACC
FPR55 (.)
AUROC (1)
CIEAR-I00
AUPR(1)
ID-ACC
w/w.a. PeN.
MSP
58.98
90.63
93,18
94.39
80.30
73,13
76.97
74,015
ODIN
36.55
94.25
95.34
94.39
$6.31
$4.89
85.88
74,05
Maholanohis
85.71
74.05
Enengy
7:23
88.91
NA
OB
9.06
98. 38.34
58.35
94.12
19.54
94,93
95.26
74,25
SOFI.
5.41
98 99
99.10
93.68
19.32
96.32
96.99
73.93
CCU
8.78
98
98.(1)
93.97
19.27
95.022
95.41
74.49
Energy (w. Dase)
4.62
98. 93
99.12
92.92
19.25
96.68
97.44
72.39
NEOM
4:00
94.25
18.77
74.52
0.31
0.07
0.16
27.71
0.09
75.15
D6.0E
11.41
0.88
97.76
0.16
18
0.12
94.07
0.34
18.91
95.00
0.72
95.26
74.08
DiveseMix (0ur5)
1.92 0.14
99.42
0.01
99.51 + 0.00
94.16
0.12
8.51 + 0.68
98.24
0.10
98.46
0.10
74.60 + 0.32
the ImageNet- O dataset. o SSB-hard is selected from ImageNet-21K, consists of 49K images and covers 980 categories. &iNaturalist consists of 859000 images from over 5000 different species of plants and animals. o NINCO consists with a total of 5879 samples of 64 classes which are non-overlapped with ImageNet-IK.
B.2 Training Details.
CIFAR experiments. We use DenseNet-101 21 the backbone for all methods, employing stochastic gradient descent with Nesterov momentum (momentum = 0.9) over 100 epochs. The initial learning rate of 0.1 decreases by factor of 0.1 at 50, 75, and 90 epochs. Batch sizes are 128 for both ID data and OOD data. For DiverseMix, we set a = 4, T' = 10. Experiments are run over five times to report the means and standard deviations. ImageNet experiments. We use 18 [8] as the backbone network. We use SGD optimizer to train all the models. The momentum is set to 0.9. Model is obtained by training ResNet18 for 100 epochs with initial learning rate of 0.1, utilizing cosine annealing strategy to adjust the learning rate. The weight decay is set to 0.0005. Batch size is set 8, T 0.1. We use OE loss as
set to 256 both ID data and OOD data. For DiverseMix, regularization loss. Experiments are run over five times to report the means and standard deviations.
B.3 Details of OOD Regularization Method.
In addition to the Energy loss mentioned in Section 4.2 our method can be extended to different
OOD regularization methods, such OE 20 and K+1
6 The details are follows:
Outlier Exposure (OE). OE introduces a promising approach towards OOD detection by utilizing outliers to force apart the distributions of ID and OOD. Its scoring function and corresponding regular function can be expressed as:
(zz B) max softmax(F(xx, 0)), Laux EzwDenn F (FF
(21)
where 2/ is the uniform distribution over classes.
(Ky1)-way regularization method. Considering a (K+l)-way classifier network F, where the (K+1)-th label indicates OOD class. Its scoring function and regular function can be expressed as:
S(z, 0) sofumaxx- (F (x, 0)), Laux Ex- (Co((F (), 1)],
(22)
where softmaxg.1(> (-) represents the softmax output in the K+1 dimension.
B.4 Details of Main Experiment.
Full Results with Standard Deviation. In Tab. 4 and Tab. 5 we present the experimental results for all evaluation metrics along with the corresponding standard deviations. From the experimental results draw similar conclusions those in Sec.
Results on Individual OOD Dataset. We also provide the performance of our method on individual OOD dataset in tabel 6
19

---

Table 5: Main results on large-scale datasets. Comparison with competitive OOD detection methods trained with the same ResNet backbone. We divide the OOD test set into two distinct groups: near-OOD and far- aar-ODD. For each group, we report the average performance metrics with standard deviation. The best and second-best results are in bold and underline respectively. Echoing the findings from our CIFAR experiments, diverseMix demonstrates strong OOD detection capabilities for both near-OOD and far-OOD test sels, achieving slale-of-thearrr OOD delection performance.
Mechod
EPR (1)
AUROC (1)
NeteOOD
AUPR (1)
FPR (3)
AUROC (1)
FarOODD
AUPR (?)
FPR (1)
AUROC (1)
Average
AUPR (*)
ID-ACC
MSP
70.35+0.67
82.75+0.19
88 58.00
34.51+3.30
58.81+0.67
91 ...50.999
60.85+2.23
86 19+-0.48
90.54+0.38 0.33
85.81
Energy
70,35.10,54
$1.8810.12
68.56.0.044
53.87.14771
$9.30.10.95
91. 11.5910.74
60:46.1.89
86.33.10.62
90.38.10.45
85.8110.14
Max Lagits
69,55=0.668
82.25:0.16
33.020.334
52.89=4.48
59.60:20.88
92. 92.13+00.67
59. 29.28+2211
80.66+01.00
90.75=00.22
85. #5.14
ODIN
69 69,06+-0.62
82.20:0017
88.75+0.044
50.90=24.33
80.90-10.89
92. 0.66
58. 18.16
86.82.00.00
91.00:0.41
85881+0.1 14
59.121.0.57
$6.86.10.32
92.6910.27
54.95
90.111021
56,61 2010..33
19.0510.24
91.7910.18
53.5210.18
Energ Dosa)
60.67=1188
65,95+003
91.75+144
58.07=3.89
89.73+00.7
89 99.6740.55
59.11=1.66
22.22:00000
90.50=0.088
54.94+0666
63.3911.115
91.46.0.05
61.31:1204
89.8510031
161.0.66.
62.14:1160
67.89.10.21
85.271.0.00 90.68.00.18
MixOE
68.42:00:12
83.4240.18
88.7420.34
90.6210.19
92.3140.14
57.082023
87.28.1:.18
56.35:0.12 90.8920.18
DiversMiic ((uis))
50.81+00788
56. 36.16-0022
91.76H0.25
48.58+1.51
91.35+0.26
02.38+0.299
53.17
89.36+0014
02.13-0.00
85.95+0.03
Table 6: main results on individual OOD dataset. We provide the results of diverseMix on each
individual OOD dataset from Section [.1 The reported performance of our method is based on five independent training runs using different random seeds.
OOD dataset
CIFAR-10
CIFAR-100
FPR (1)
AUROC (1)
AUPR (1)
ID-ACC
EPR (L)
AUROC (1)
AUPR (*)
ID-ACC
LSUN-C
3.39 = 0.80
99:21 0.14
99.29 99.29-00.13 0.13
99.11 + 0.02
8.16
98.56 + 0.35
98 + 0.32
98.19 08.19:0..10
LSUN-R
0.00 + 0.000
100.00 + 0.00
100.00 - 0.00
50.41
+ 0.02
0.01 + 0.01
99.93 + 0.13
98.95 + 0.08
99.83 +
ISUN
0.00
100.00
100.00 0.00
0.001
99.91
98.95
DTD
1.13 0: 0.14
99.58 1. 0.06
99.72 1 0.06
99.98 1: 0.01
5.77 4: 0.61
98.47 1: 0.15
98.98 0.111
99.96 0.01
Places365
5.30 + 0.64
98.53 +0117
98.00 + 0.25
98.41 + 0.10
26.55 + 2.45
94.95
+ 0.41
95.32 + 0.46
93,92 + 0..42
SVHN
1.66
99.21
+:
97
97.93
Average
1.92 + 0.14
90.42 + 0.01
90.50 + 0.03
90.41 + 0.02
8.51
0.68
98.24 + 0.10
98.46
+ 0.10
98.19 0.10
B.5 Details of Figure
In this experiment, we aim to explore the effect of outlier quality on OOD detection performance. We analyzed this from two perspectives: the sample size of outliers and the diversity of outliers. Specifically, we constructed series of subsets from the Imagener-RC dataset to generate low-quality auxiliary outliers datasets with diffcrent sample size and diversity. Afterwards, we used these constructed low-quality subsets as the auxiliary outliers dataset train the model. All experimental results are run over three times and averaged. The experimental details are as follows:
Decreasing the sample size of auxiliary outliers. To explore the impact of sample size on our experimental results, we keep the number of classes constant and decrease the size of the auxiliary outliers dataset. This is achieved by applying downsampling techniques, result- ing subsets with the same classes as the original Imagenet-RC dataset but with sizes of {100%, 85%, 70%, 55%, 40%, 25% 10%} compared to the original auxiliary outliers dataset.
Decreasing the diversity of auxiliary outliers. To investigate the effect of outlier diversity on OOD detection performance, we further reduce the number of classes included in the subset. Specifically, we keep the sample size of the subset at 10% of the original outliers dataset, but gradually decrease the number of classes included (as the number of classes decreases, the number of samples per class increases, ensuring a consistent overall sample size). We constructed a series of subsets with {1000, 850, 700, 550, 400, 250, 100} classes to serve as auxiliary outliers for experimental evaluation.
B.6 Details of Q5 Ablation Study.
In this section, we conduct an ablation study from two perspectives. Firstly, we compare our method with traditional data augmentation techniques
pressvviii) to demonstrate that our method effectively enhances the diversity of outliers by altering their semantics. Secondly, considering that our method is an improved variant of mixup, we investigate different mixup strategies to explore what factors contribute to the performance gains. Detailed experimental results are shown in Table 7
A Ablation study (I): Ablation study with different data augmentation method. To investigate if diverseMix offers unique advantages over other data augmentation techniques in enhancing the diver- sity of outliers, we select different data augmentation methods to process the auxiliary outliers and validate their impact on performance. Specifically, we choose semantic-invarfe methods: Gaussian noise (38), cutout [I]] and color jitter for comparison with our method.
data augmentation
20

---

Table 7: Ablation study on different data augmentation methods. Performance are averaged (%) over six OOD test datasets from Section [5.]] The best results are in bold. The reported OOD detection performance is based on five independent training runs using different random seeds.
Metbod
FPR ())
AUROC (7)
CIFAR-10
AUPR
ID-ACC
FPR
AUROC (t)
CIFAR-100
AUPR (7)
ID-ACC
Vanilla
5.13 4 0.18
95.80
98.92 + 0.04
91.37
0.110
16.30 = 1.74
96.87 =0.4
97.35 38 0.30
74.19 + 0.29
Gaussian noise
6.09 1 0.19
38.641.0.5
98.75
0.08
91.27 1 0.02
19.94
2.19
95.09
0.55
96,07 4 0.55
74.78 4: 0.18
Cutout
0.87
0.14
98.72
94.17
0.07
19.12
0.82
96.64
0.20
97.18
0.26
Color jittriing
4.34 0.72
00.04+00.0
99.09 + 0.16
94.22
0.06
14:47
= 1.75
97.02
10:02-00 0.33
91.33
0.36
T4.46
0:20
Mixup
4.00 1: 0.12
99.02
0.06
98.07 1: 0.09
91.16 1 0.08
12.56
0.30
97.12 0.09
97.63
+ 0.05
71.76 4: 0.19
Cutmix
0.75
98.65
98.80
94.10
0.16
17.07
0.97
96.87
97.30
0.19
DiverseMix (ours)
1.92 0.14
99.42
0.01
99.50 + 0.03
94.16 + 0.12
8,51
0.68
98.24
0.10
98.46
0.30
74.60 + 0.32
Gaussian noise. Here, we introduce an appropriate level of noise to the training data to augment its diversity and quantity. We incorporate Gaussian noise with a mean of 0 and a variance of 0.1. To effectively mitigate the risk of model overfitting to Gaussian noise, wherein the model incorrectly classifies any image with Gaussian noise as an OOD input and any noise-free image as an ID sample, this type of noise is applied to only half of the outlier samples during the model training phase.
Cutout. Cutout is data augmentation technique that introduces random masking of small regions in input images, preventing the model from relying on specific features. In our study, we apply the cutout augmentation half of the auxiliary outlier samples. This involves randomly masking out small regions within these outlier images by setting all pixel values in the masked regions to zero
Color jittering. Color jittering is widely adopted data augmentation technique in image processing. It introduces random variations to the brightness, contrast, saturation, and hue of an image, simulating the diverse conditions encountered in real-world scenarios, such as different lighting environments or camera settings. Specifically, for each auxiliary outlier image, we randomly adjust its brightness within a range of =0.4, its contrast within range of +0.4 and its saturation within range of +0.4, while rotating the hue by +0.1 radians. This data augmentation strategy preserves the semantic content of the original outlier image while introducing controlled variations in color properties.
A Ablation study (II) Ablation study with different semantic interpolation method. To explore how diverseMix differs from other mixup-based methods, we compared the performance to vanilla mixup and cutmix. We set the hyperparameter a 4, consistent with our method diverseMix.
Vanilla mixup. Vanilla mixup involves generating virtual training examples (referred to as mixed samples) through linear interpolations between data points and corresponding labels, given by:
f = Ar; + (1 A)æz
Ay; + (1 A)y5
(23)
where (z 91) and (z).3;)) are two samples drawn randomly from the empirical training distribution, and X E [0, 1] is usually sampled from a Beta distribution with parameter a denoted as Beta (a, a).
Cutmix. Cutmix is a data augmentation method that constructs virtual training examples by perform- ing cutting and replacing the cutted region with the corresponding region from the other image:
t- ( 3 x; + (1 - (A)) C zy; ÿ Ay + -))y,
(24)
where (A) is binary mask randomly chosen covering À proportion of the input, and @ represents the element-wise product. Here, Å is usually sampled from a preset beta distribution Beta(a, a).
C Additional Results
C.1 Hyperparameter Analysis.
In this section, we analyze the main hyperparameters involved in our method. The experimental results are shown in the table g From the experimental results, we find that diverseMix is more effective with larger values of a. A larger a means that the model will adopt a more aggressive interpolation strategy, generating mixed outliers that deviates further from the original samples. This aligns with our expectations. The temperature T controls diverseMix's 's sensitivity to the samples, an appropriate T' allows diverseMix to accurately perceive the model's familiarity with the samples. controls the strength of regularization, an excessively large w may impair the classification performance. In addition, we provide our strategy for hyperparameter adjustment in practice as follows:
hyper-paramettr tuning. We can first determine the largest possible value of w for the original baseline model while maintaining the ID classification accuracy. Then, we can select more suitable
21

---

Table 8: Hyperparameter analysis. Performance averaged (%) over six OOD test datasets from The performance reported are averaged over different random seeds.
Section 5.T
T
FPR (L)
AUROC (?)
CIFAR-10
AUPR (1)
ID-ACC
FPR
AUROC (1)
CIFAR-100
AUPR (1)
ID-ACC
0.5
10
0.01
3.67.1041
99.15
99.24:10.66
94.19::0.08
9.39
98.03=0.21
98.23:10.19
74.67:0.20
10
0.01
3.16:10.18
99.23-0.03
99.33-00004
94.13:10.08
9.5510.99
98.10-0.16
98.3610.19
74.66:1.10
10
2.55++0.88
99.32--0110
99.41-0.11
94.20--0.05
8.44+00.1
98.17-0.05
98.41+0.04
74.70+0.22
4
5
0.01
1.87.1031
99.08.1.005
99.19_0.011
94.42.10.09
10.27-00.97
96.04_00.16
98.29100.11
74.67
5.4320.59
98.7820.16
98.93=0.16
94,01=2.22
15.62=1.71
97.18=0.29
97.6310.20
74.90:0.12
4
20
0.01
2.75.10.38
99.26.10.11
99.31-10.12
94.15.10.06
9.49..0.9
98.04_
98.29110.04
74.43.1.339
10
0.05
1.76:0002
99.47=0.02
99.56:20.02
93.89=00115
8.12+1.03
98.36=0.17
98.5810.14
73.94:1.28
10
0.1
2.60+0.62
99.33=00.99
99.45=00.07
92.51=0.44
9.04=1.00
98. 18.1900.08
98,45:20.05
71.96+0.60
4
10
0.01
1.22 0.14
99.42 0.01
99.50 1:: 0.03
94.16 =: 0.12
8.51 = 0.68
98.24
0.10
98.46 1: 0.10
74.60.10.32
Table 9: Experimental Results on NLP OOD detection task. The best results are in bold. The
same network architecture used for all three detectors, All results are represented in percentages.
Our method diverseMix also achieves good performance in the field of natural language proceeding.
FPR95 (4)
AUROC (1)
AUPR (1)
0OD testset
MSP
OE
diverseMix
MSP
OE
diverseMix
MSP
OE
diverseMix
SNLI
52.61
27,05+22.23
21,58-14.51
76.19
17.63--1.03
90.12 0.66
33.83
50,80++3.21
58,45+1..2 +.27
Multi30k
76.00
35.69:2..0
19.38:13.83
60.69
87.09:1488
92.43::1.05
22.08
56.93114,24
68.37:15.53
WMT16
68.66
14.28 1.70
11.90
67.30
94.59+0.44
95.59+0.00
26.36
25.65
79.80-1.003
Yelp Reviews
82.98
5.36:10.71
4.23:12.96
56,38
96.98:1.70
9,.92:10.50
20.45
78.09:16.34
85.09:5.87
Average
70.06
20.60
14.27:12.56
65.14
91.57-0.72
94.02:10.45
25.68
65.3743.60
72.93:14.82
Table 10: Time and memory cost of different methods. We compare the computational overhead of DiverseMix and other methods on CIFAR-100 under the same setting. Best results are in bold.
Computational Cost
0OD Detection Performance
Method
w./.oo. Doux
Times (hours) GPU Memory (MB)
FPR95 (L)
AUROC (*)
AUPR (1)
ID-ACC
Energy NTOM
1.99
13017
19.25
96.68
97.44
72.39
2.51
15549
18.77
96.69
96.49
74.52
POEM
8.47
19851
15.14
97.79
98.31
73.41
V
DivOR
4.88
13163
18.91
95.00
95.26
74.08
DiverseMix
2.20
13213
8.51
98.24
98.46
74.60
parameters for a and T, with adjustments made using an OOD validation set distinct from the testing OOD dataset. For example, subset from the auxiliary outliers could serve an OOD validation set.
C.2 DiverseMix for OOD Detection in Natural Language Processing.
To further validate the applicability of our method non-image domains, we explore the use of diverseMix in the task of Natural Language Processing, following the setting of OE 22)
Experimental Setting. We use the SST dataset as the ID data, while utilizing the WikiTexi-2 dataset as auxiliary outlier data. We employ the SNLI, Mulri30K, WMT16. and Yelp Reviews datasets as OOD test set. We use QRNN [33) language models as baseline OOD detectors. Initially, we train vanilla models for 50 epochs and subsequently fine-tune them on the WikiText-2 dataset using OE or DiverseMix for an additional epochs. Outlier Exposure is implemented by adding the cross entropy to the uniform distribution on tokens from sequences in Dauz as an additional loss term. For DiverseMix, we apply mixup strategy at embedding level, and the loss function is consistent with OE.
Experimental Results. The results presented in table 9 highlight that: 1) The incorporation of auxil- iary outliers enhances OOD detection performance in non-image domains. 2) Our method increases the diversity of auxiliary outliers, further enhancing the model's 's OOD detection performance.
C.3 Experiments on Computational Cost.
To better understand the computational budget, summarize the time and memory cost results in Table 10 which shows that diverseMix can achieve better performance with relatively low time and memory overhead compared with other OOD detection methods that train with auxiliary outliers.
C.4
Impact Statements
Our work focuses on enhancing Al safety and trustworthiness by improving the robust performance of machine learning models on OOD data, which is crucial for high-stakes tasks in real-world scenarios.
22

---

However, biases in benchmark OOD detection data, such as ImageNet, necessitate careful auxiliary outlier selection for safety-critical applications ensure the proposed method's reliability and safety.
D Hardware and Software
We run all the experiments on NVIDIA GeForce RTX 3090 GPU, Our implementations based on Ubuntu Linux 18.04 with Python 3.8.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's 's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper has discussed the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
The paper has provided the full set of assumptions and a complete (and correct)
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
25

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
aublic/guudees/Cdeettbii.. wtt.:eewwwwbiiiiiooooooooooogg for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppppclgguiuaeaeaeccuuee
Ctteeee//ooSSSSSSSSS.ee.a www.ssssiicgcyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details necessary understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:://ettiippps.ccccm httpe/rrpppp.cpppupcb//heecccccc itw.e
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive socictal impacts and negative socictal impacts of the work performed.
Guidelines:
27

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We have not released data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and the license and terms of use explicitly are mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
There is no new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
There is no crowdsourcing experiments and research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is no research with Human Subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "jj0Eydn022", "reviewer": "Reviewer_E172", "summary": "This study aims to explore the reasons behind the effectiveness of out-of-distribution (OOD) regularization methods by linking the auxiliary OOD dataset to generalizability. The researchers show that the variety within the auxiliary OOD datasets significantly influences the performance of OOD detectors. Moreover, they introduce a straightforward approach named diverseMix which is designed to enhance the diversity of the auxiliary dataset used for OOD regularization.", "questions": "The primary question of the reviewer is why not leverage the entire ImageNet-1k dataset for ID whilst leveraging other unlabeled datasets for the auxiliary data.", "limitations": "The authors have adequately addressed any potential negative societal impacts of this work, and the reviewer does not anticipate any such negative impacts.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n- The paper is well-composed and presents an extensive array of experiments across multiple OOD detection benchmarks.\n- This study offers important insights into the significance of auxiliary datasets in OOD regularization, addressing a frequently neglected aspect of OOD regularization techniques.\n- The authors provide a robust theoretical foundation for diverseMix. Additionally, they show strong empirical evaluations which further highlight the effectiveness of diverseMix across a range of OOD experiments.", "weaknesses": "Weakness:\n- The reviewer has some concerns regarding the empirical evaluations of diverseMix. In particular, the choice of how the ImageNet-1k is split into ImageNet-200 as ID while the remaining data is leveraged as OOD seems arbitrary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your valuable comments and appreciate your recognition of the effective method as well as sufficient theoretical guarantees. We provide detailed responses to the constructive comments.\n\n## W1. The reviewer has some concerns regarding the empirical evaluations  of diverseMix. In particular, the choice of how the ImageNet-1k is split into ImageNet-200 as ID while the remaining data is leveraged as OOD  seems arbitrary.\n\nThank you for raising this important question. We sincerely appreciate your attention to the details of our experiments and pleasure to provide further clarification.\n\n(i) **Our splitting strategy was not arbitrary but carefully considered.** We randomly selected 200 classes from ImageNet-1K as the ID categories, while the remaining 800 classes were used as OOD categories. This setting ensures the validity of our experiments.\n\n(ii) **This experiment setting is consistency with prior work.** We followed the benchmark [1] to set ImageNet-200 as the ID dataset for our experiments. \n\n## Q1. The primary question of the reviewer is why not leverage the entire  ImageNet-1k dataset for ID whilst leveraging other unlabeled datasets  for the auxiliary data?\n\nThanks for the constructive suggestions. We are grateful for this insight and have conducted the additional experiments as suggested. Specifically, we used the entire ImageNet-1K dataset as the ID dataset and employed the SSB-hard dataset as auxiliary outliers. The experimental results are shown below, with columns representing different OOD datasets. The values in the table are presented in the format (FPR $\\downarrow$ / AUROC $\\uparrow$). **From the experimental results, it is evident that our method remains effective even when ImageNet-1K is used as the ID dataset.**\n\n| Method | dtd             | iNaturalist     | ninco           | average         |\n| :----: | :-------------: | :-------------: | :-------------: | :-------------: |\n| OE     | 73.90/76.82     | 49.33/89.53     | 76.03/80.52     | 66.42/82.29     |\n| Energy | 69.80/82.56     | 74.40/85.58     | 81.66/77.32     | 75.29/81.82     |\n| Mixoe | 69.48/78.07 | 46.61/89.72 | 74.17/80.79 | 63.42/82.86 |\n| Ours   | **68.17/78.69**   | **42.71/90.98**   | **73.29/81.27**   | **61.39/83.65**|\n\nReferences:\n> [1] OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4R42eOcQHG", "reviewer": "Reviewer_hmUc", "summary": "The paper proposed Diversity-induced Mixup for OOD detection (diverseMix), which enhances the diversity of auxiliary outlier set for training in an efficient way.", "questions": "see above", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is written well and is easy to understand.\n\n2. The studied problem is very important.\n\n3. The results seem to outperform state-of-the-art.", "weaknesses": "1. My biggest concern is that there are already some papers that theoretically analyze the effect of auxiliary outliers and proposed some complimentary algorithms based on mixup, such as [1], [2] and [3]. It might be useful to clarify the differences. \n\n[1] Out-of-distribution Detection with Implicit Outlier Transformation\n[2] Learning to augment distributions for out-of-distribution detection\n[3] Diversified outlier exposure for out-of-distribution detection via informative extrapolation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your reviews. We are encouraged that you appreciate our studied problem and state-of-the-art result. We address your concerns as follows. \n\n## W1. My biggest concern is that there are already some papers that theoretically analyze the effect of auxiliary outliers and proposed some complimentary algorithms based on mixup, such as [1], [2] and [3]. It might be useful to clarify the differences.\n\nThanks for mentioning [1] [2] [3]. We first review these related works and then clarify the differences from the perspectives of motivations, techniques, and theory. Additionally, we have incorporated these related works into the manuscript.\n\n**(i) Reviews of related works.**\n\nDOE [1] proposes a novel and effective approach for improving OOD detection performance by implicitly synthesizing virtual outliers via model perturbation. \n\nDAL [2] introduces a novel and effective framework for learning from the worst cases in the Wasserstein ball to enhance OOD detection performance. \n\nDivOE [3] is an innovative and effective method for enhancing OOD detection performance by using informative extrapolation to generate new and informative outliers.\n\n**(ii) Differences in motivations.** DiverseMix has a different motivation from DOE, DAL and DivOE. Our DiverseMix is proposed for enhancing the diversity of auxiliary outliers set  through semantic-level interpolation to enhance OOD detection performance. In comparison, DOE focuses on improving the generalization of the original outlier exposure by exploring model-level perturbation. DAL focus on crafting an OOD distribution set in a Wasserstein ball centered on the auxiliary OOD distribution to alleviating the distribution discrepancy between auxiliary outliers and unseen OOD data. DivOE focuses on extrapolating auxiliary outliers to generate new informative outliers for enhance OOD detection performance. \n\n**(iii) Differences in technique.** Our method, DiverseMix, has a different algorithmic design compared to others. Specifically, DiverseMix adaptively generates interpolation strategies based on outliers to create new outliers. In contrast, DOE, DAL, and DivOE primarily rely on adding perturbations. Specifically, DOE, DAL, and DivOE apply perturbations at the model level, feature level, and sample level, respectively, to mitigate the OOD distribution discrepancy issue.\n\n**(iv) Differences in theory.** We prove that a more diverse set of auxiliary outliers could improve the detection capacity from the generalization perspective, and this theoretical insight inspired our method DiverseMix. We also provide insightful theoretical analysis verifying the superiority of DiverseMix. In comparison, DOE revealing that model perturbation leads to data transformation  to enhance the generalization of OOD detector. DAL finds that the distribution discrepancy between the auxiliary and the real OOD data affecting the OOD detection performance. DivOE is based on the perspective of sample complexity to demonstrate its effectiveness.\n\nReferences:\n>[1] Out-of-distribution Detection with Implicit Outlier Transformation.\n> \n>[2] Learning to augment distributions for out-of-distribution detection.\n> \n>[3] Diversified outlier exposure for out-of-distribution detection via informative extrapolation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dEuBILciU7", "reviewer": "Reviewer_1vuo", "summary": "The authors propose DiverseMixup, a Mixup data augmentation technique applied to auxiliary OOD data to improve the OOD detection capacities of classifiers trained with the Outlier Exposure technique. They provide a theoretical analysis justifying their approach and demonstrate the superior empirical performances of their technique.", "questions": "I am puzzled because, on the one hand, the paper demonstrates strong empirical results, and the evaluation methodology is extensive and thorough, but on the other hand, I suspect that the authors' theoretical work is flawed - which does not affect the strength of the presented method but the validity of the paper. I am ready to improve my rating to acceptance if the authors prove my suspicions wrong during the rebuttal.", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method appears to be novel and competitive compared to other OOD data augmentation methods while remaining quite simple and easy to implement.\n- The ablation and complementary experiments are satisfying.\n- The experiments answer many practical questions.", "weaknesses": "One of the main contributions of the paper is the theoretical analysis. However, there appear to be critical flaws in both the demonstrations and the hypotheses.\n\n### Major\n\n1. The quantities $h^*_{ood}$ and $h^*_{aux}$ are never defined. We can guess out of commonly used notation in optimization that $h^*_{ood} = \\\\underset{h\\\\in \\\\mathcal{H}}{\\\\operatorname{argmin}} \\\\epsilon_{P_{\\\\mathcal{X}}}(h,f)$ (same for $h^*_{aux}$ ) but this is the definition of $\\\\mathcal{H}^*_{ood}$, which is defined as a set (which is not straightforward - why would an argmin be a set in that case ?). It adds a lot of confusion, and we never know exactly what we are talking about, which is critical for a demonstration. \n2. In the demonstration of theorem 1, the authors define $\\\\beta_1$ and $\\\\beta_2$ as constants, but 1) they depend on $P_{\\\\mathcal{X}}$ which is supposed to be affected by the later-introduced $\\\\mathcal{X_div}$ and 2) the parts of l.530 that are replaced do not seem to match the definition of $\\\\beta$'s. \n3. Demonstrations of Theorems 2 and 3 seem to rely on one argument, which is: \"Since $\\\\mathcal{X_aux} \\\\subset \\\\mathcal{X_ood}$, then  $\\\\mathcal{H}^*_{ood} \\\\subset \\\\mathcal{H}^*_{aux}$\". I am concerned with the validity of this assumption (assuming that the definition of $\\\\mathcal{H}^*$ as sets make sense, which is not clear). As a counterexample, let's consider $\\\\phi_{\\\\mathcal{X}} = Unif(0,3)$ and $\\\\phi_{\\\\tilde{\\\\mathcal{X}}} = Unif(0,1)$ (which implies that $\\\\mathcal{X_aux} = (0,1)$ and  $\\\\mathcal{X_ood} = (0,3)$).  Now, let's consider $h^*_{ood}$ and $h^*_{aux}$ such that \n\n$\n\\\\begin{dcases}\n  \\\\int_{\\\\mathcal{X_ood} \\\\setminus \\\\mathcal{X_aux}} |h^*_{ood}(x) -f(x)|dx = 0\\\\\\\\\n  \\\\epsilon_{P_X}(h^*_{ood},f) = \\\\int_{\\\\mathcal{X_aux}} |h^*_{ood}(x) -f(x)|dx + \\\\int_{\\\\mathcal{X_ood} \\\\setminus \\\\mathcal{X_aux}} |h^*_{ood}(x) -f(x)|dx= \\\\int_{\\\\mathcal{X_aux}} |h^*_{ood}(x) -f(x)|dx =  \\\\epsilon_1,\\\\\\\\ \\\\end{dcases}$  \n\nand \n\n$\\\\begin{dcases}\n\\\\epsilon_{P_\\\\tilde{X}}(h^*_{aux},f) = \\\\int_{\\\\mathcal{X_aux}} |h^*_{aux}(x) -f(x)|dx =  \\\\epsilon_2 < \\\\epsilon_1,\\\\\\\\\n  \\\\int_{\\\\mathcal{X_aux} \\\\setminus \\\\mathcal{X_aux}} |h^*_{aux}(x) -f(x)|dx > \\\\epsilon_1 - \\\\epsilon_2\n  \\\\end{dcases}$\n\n  where for simplicity, we omit $\\\\mathcal{X_id}$, assuming that the behavior is similar on this input space region for $h^*_{ood}$ and $h^*_{aux}$ . In that case, clearly, $h^*_{ood}$ minimizes $\\\\epsilon_{P_X}(h,f)$ (thanks to the inequality above that keeps $h^*_{aux}$ sub-optimal for $P_X$) but does not minimize $\\\\epsilon_{P_\\\\tilde{X}}(h,f)$.\n\n### Minor\n\n1. l. 111 perhaps you meant $\\\\mathcal{X_id}$ instead of $\\\\mathcal{Y_id}$?\n2. l.122 $\\\\epsilon$ is called a probability, whereas it is an expectation\n3. Some typos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our novel method and satisfying experiments. We appreciate your support and constructive suggestions and address your concerns as follows.\n\n---\n## W1. The quantities $h^*_{ood}$ and $h^*_{aux}$ are never defined. Why would an  argmin be a set in that case?\n\nThank you for your valuable comment. We agree that clearer definitions of $h^*_{ood}$ and $h^*_{aux}$ would improve our manuscript. Our choice to define the argmin as a set was purposeful and well-founded and we appreciate to clarify this as follow:\n\n### **1) The definition of quantities $h^{*}_{aux}$** **and $h^{*}_{aux}$**.\n\n$h^*_{ood}$ and $h^*_{aux}$ is defined as the element in $\\mathcal H^*_{ood}$ and $\\mathcal H^*_{aux}$, respectively, which can be denoted as: $h^*_{ood}\\in \\mathcal H^*_{ood}, h^*_{aux} \\in \\mathcal H^*_{aux}$, where $\\mathcal H^*_{ood}$ and $\\mathcal H^*_{aux}$ refers to the sets of ideal hypotheses on the training data distribution $P_{\\widetilde{\\mathcal{X}}}$ and test-time data distribution $P_{\\mathcal{X}}$. We will add the definition in  manuscript.\n\n### **2) Why would an argmin be a set?** \n\n**(i) Our setting requires us to represent the optimal hypothesis as a set.** \nIn our setting, $\\mathcal H_{aux}$ contains all hypotheses optimal for the training data distribution. However, these hypotheses may perform inconsistently on the test-time data distribution, necessitating the use of a set rather than a single $h^*_{aux}$.\n\n**(ii) The optimization problem behind deep neural networks is highly non-convex and the optimal solution is not unique [1] [2].** Therefore, defining argmin as a set offers generality.\n\n**(iii) Representing argmin as a set is not unprecedented in the field.** For example, the theoretical analysis in [3] also employs this set-based representation.\n\n---\n## W2. The authors define $\\beta_1$ and $\\beta_2$ as constants, but 1) they depend on  $P_\\mathcal X$  which is supposed to be affected by the later-introduced $\\mathcal X_{div}$ and 2) the parts of line 530 that are replaced do not seem to match the definition of  $\\beta$'s.\n\nSincerely thank you for your thorough review. We realize our derivation omitted some explanations, leading to misunderstandings.  We're happy to clarify in detail as follow:\n\n### **1) Clarification of $\\beta_1$and $\\beta_2$.**\n\n**(i) We do not define $\\beta_1$and $\\beta_2$ as constants.** This misunderstanding may have arisen from our definition of $\\beta$ as a constant in Theorem 1. We will clarify this in the revised paper.\n\n**(ii) $\\beta_1$ is not affected by $\\mathcal X_{div}$.** $\\beta_1$ depends on the unknown test data distribution $P_{\\mathcal X}$. The later-introduced $\\mathcal X_{div}$ only affects the training data distribution, thus $\\beta_1$ is unaffected by $\\mathcal X_{div}$.\n\n**(iii) While $\\beta_2$ is indeed influenced by $\\mathcal X_{div}$, we can prove that $\\beta_2$ is bounded by a very small constant. Therefore, in subsequent analyses, the effect of $\\mathcal X_{div}$ on $\\beta_2$ is negligible.** We have put the detailed discussion and proof in the official comment (1).\n\n### **2) The parts of l.530 that are replaced do not seem to match the definition of  $\\beta$'s.**\n\n**Upon careful examination, we believe the derivation is accurate.** However, we recognize that the presentation could be enhanced for clarity. We have provided a detailed derivation in the official comment (2).\n\n---\n## W3. Demonstrations of Theorems 2 and 3 seem to rely on one argument, which is: \"Since $\\mathcal X_{aux}\\subset\\mathcal X_{ood}$ , then $\\mathcal H^*_{ood}\\subset \\mathcal H^*_{aux}$\" I am concerned with the validity of this assumption.\n\nThank you for your valuable feedback and we greatly appreciate your attention to detail. We illustrate the validity of this assumption as follow:\n\n**(i) The assumption is reasonable in the over-parameterized setting.**\n Considering that we can decomposing the ideal error on $\\mathcal X_{ood}$ into two components: one for $\\mathcal X_{aux}$ and another for the remaining OOD data $\\mathcal X_{ood} \\setminus \\mathcal X_{aux}$. \nConsidering that the model is over-parameterized (line 126), there exists the ideal hypothesis minimizing both error simultaneously. \nIn other word, $\\mathcal X_{ood}$ is the intersection of the sets of optimal hypothesis for auxiliary and remaining data, thereby $\\mathcal H^*_{ood}\\subset\\mathcal H^*_{aux}$. We have provided a detailed derivation in the official comment (3).\n\n**(ii) Discussion of the Counterexample.**\nThe counterexample suggests that the model's optimal solution in $\\mathcal X_{ood}$ may not be optimal in $\\mathcal X_{aux}$ because it balances performance in $\\mathcal X_{ood} \\setminus \\mathcal X_{aux}$. This assumes there is no hypothesis $h \\in \\mathcal H$ that achieves optimal performance in both $\\mathcal X_{aux}$ and $\\mathcal X_{ood} \\setminus \\mathcal X_{aux}$. However, in over-parameterized setting, there exists the ideal hypothesis that is optimal in both $\\mathcal X_{aux}$ and $\\mathcal X_{ood} \\setminus \\mathcal X_{aux}$. Therefore, **the counterexample does not hold in our setting**.\n\n---\n## 4. line 111 perhaps you meant $\\mathcal X_{id}$ instead of $\\mathcal Y_{id}$?\n## 5. line 122 $\\epsilon$ is called a probability, whereas it is an expectation.\n## 6. Some typos\n\nThank you for your valuable suggestions, we have addressed your concerns as follows:\n\n(i) We would like to clarify that $\\mathcal X_{ood}$ is outside the support of $\\mathcal Y_{id}$. In other words, $\\mathcal X_{id}$ is within the support of $\\mathcal Y_{id}$.\n\n(ii) We acknowledge the error on line 122 and have corrected it.\n \n(iii) We have thoroughly reviewed and revised all representation issues throughout the paper.\n\nReferences:\n>[1] The Loss Surface of Deep and Wide Neural Networks.\n>\n>[2] On the Quality of the Initial Basin in Overspecified Neural Networks.\n>\n>[3] Agree To Disagree: Diversity Through Dis-Agreement For Better Transferability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission received three ratings (6, 6 and 7), averaging 6.33, which is above the acceptance. After rebuttal, all reviewers' concerns have been well addressed, especially about the experiments, difference from previous works and the clarity of the theoretical notations and proof. After carefully checking the response of all reviewers, I suggest the acceptance. Hope the authors carefully follow the reviewers' advice to improve the submission finally.
Decision: Accept (poster)