Diversified Outlier Exposure for Out-of-Distribution Detection via Informative Extrapolation
Authors: Jianing Zhu
Geng Yu
Jiangchao Yao
Tongliang Liu
Gang Niu
Masashi Sugiyama
Bo Han
Venue: NeurIPS 2023 poster (2023)
Keywords: out-of-distribution detection
outlier exposure

Abstract: Out-of-distribution (OOD) detection is important for deploying reliable machine learning models on real-world applications. Recent advances in outlier exposure have shown promising results on OOD detection via fine-tuning model with informatively sampled auxiliary outliers. However, previous methods assume that the collected outliers can be sufficiently large and representative to cover the boundary between ID and OOD data, which might be impractical and challenging. In this work, we propose a novel framework, namely, Diversified Outlier Exposure (DivOE), for effective OOD detection via informative extrapolation based on the given auxiliary outliers. Specifically, DivOE introduces a new learning objective, which diversifies the auxiliary distribution by explicitly synthesizing more informative outliers for extrapolation during training. It leverages a multi-step optimization method to generate novel outliers beyond the original ones, which is compatible with many variants of outlier exposure. Extensive experiments and analyses have been conducted to characterize and demonstrate the effectiveness of the proposed DivOE. The code is publicly available at: https://github.com/tmlr-group/DivOE.

Diversified Outlier Exposure for Out-of-Distribution Detection via Informative Extrapolation
Jianing Zhu'
Geng Yu²
Jiangchao Yao Tongliang Liu
Gang Niu
Masashi Sugiyama 5.6 Bo Han 1,5
Hong Kong Baptist University 2CMIC, Shanghai Jiao Tong University
Shanggaai AI Laboratory "Sydney Al Centre, The University of Sydney
"RIKEN Center for Advanced Intelligence Project TThe University of Tokyo
(esjuzhu, bhanml| @ww@e..hkb..du....... edu.hk (warriors30, sunarker @sjtu.edu.cn gang.niu.mlo gmail.com
tongliang. liu @ sydney.edu.au
sugi u.tooyoo..cc
Abstract
Out-of-distribution (OOD) detection is important for deploying reliable machine learning models on real-world applications. Recent advances in outlier exposure have shown promising results OOD detection via fine-tuning model with infor- matively sampled auxiliary outliers. However, previous methods assume that the collected outliers be sufficiently large and representative cover the boundary between ID and OOD data, which might be impractical and challenging. In this work, we propose novel framework, namely, Diversified Outlier Exposure (Di- vOE), for effective OOD detection via informative extrapolation based on the given auxiliary outliers. Specifically, DivOE introduces new learning objective, which diversifies the auxiliary distribution by explicitly synthesizing more informative out- liers for extrapolation during training. It leverages multi-step optimization method to generate novel outliers beyond the original ones, which is compatible with many variants of outlier exposure. Extensive experiments and analyses have been con- ducted to characterize and demonstrate the effectiveness of the proposed DivOE. The code publicly available at: https: // / thub. com/tmlr- -aaarr-rruuuD.D.....
Introduction
Out-of-distribution (OOD) detection [Nguyen et al., 2015, Hendrycks and Gimpel, 2017] gains increasing attention in deploying machine learning models into open-world scenarios, as deep learning systems are expected to conduct reliable predictions identifying OOD inputs [Bommasani et al., 2021, Hendrycks et al., 2022]. It becomes more critical in safety-critical applications like financial or medical intelligence, where the false prediction for samples out of pre-defined label space can sometimes be a disaster. Extensive explorations [Hendrycks al., 2019, Liu al., 2020, Tack al., 2020, Mohseni et al., 2020, Sehwag et al., 2021, Yang et al., 2021, 2022] in recent years has been contributed to improving the model ability for OOD detection.
in-distribution (ID) data while
Compared with host-hoc methods [Hendrycks and Gimpel, 2017, Liang et al., 2018. Liu et al., 2020, Huang et al., 2021, Sun et al., 2022] which designs different score functions for OOD uncertainty estimation, Outlier Exposure (OE) [Hendryeks et al., 2019] is another kind of method owning better performance improvement on OOD detection [Mohseni et al., 2020, Sehwag et al., 2021, Ming et al., 2022, Katz-Samuels al., 2022, Wang al., 2023). The latter engages surrogate OOD data during
CCorrespondence to Bo Han (bhanml @ @ome@/oomppphc www....bb..eduuh....
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Daut
CIFARIO
CFARR00
PCCH
Dube
Din
Dout
4000
Numiten of Aami Oualees
4002 2008
Figure I: Illustration about the research problem on auxiliary outliers studied in our work for OE Left panel: the general assumption on the auxiliary outliers, in which the surrogate OOD distribution Dou can not always be broad enough to well represent the unseen OOD distribution Pout since it is impractical or even infeasible to accurately pre-define and collect those boundary outliers (especially those close to Di,); Right panel: an empirical verification on OOD detection performance (evaluated by FPR95, the lower is the better) with the gradually decreased number of given auxiliary outliers. Our proposed DivOE can perform better via extrapolation than previous sampling- based methods, given different numbers of auxiliary outliers. More experimental details are provided in Appendix C.
training and regularizes the model via fine-tuning those auxiliary outliers. The conceptual idea is to learn the knowledge from auxiliary outliers for effectively identifying OOD inputs. Under such learning framework, there is intuitive gap between the surrogate OOD data and the unseen OOD inputs [Fang et al., 2022, Ming et al., 2022] as illustrated in Figure 1. It will be a fundamental problem for boosting the model's discriminative capability auxiliary outliers, it naturally motivates the following critical research question: How could we utilize the given outliers for effective OOD detection if the auxiliary outliers are informative enough?
the OOD inputs. Given the finite
Addressing the essential distribution gap between surrogate OOD data and the unseen OOD inputs remains challenging [Hendrycks et al., 2019, Fang al., 2022, Du et al., 2022, Yang et al., 2021], as is hard to know the prior knowledge of potential OOD inputs would be encountered at inference stage [Hendrycks et al., 2022], and intentionally collect them. As the unseen OOD inputs in the real- world scenarios are complex [Cimpoi et al., 2014, Yu et al., 2015, Yang et al., 2022], the performance of OE-based methods is heavily affected by the given auxiliary outliers (as empirically presented in Figure 2). Given the finite auxiliary outliers, the surrogate OOD data are expected to have less discrepancy [Hendrycks et al., 2019, Fang et al., 2022] with the unseen OOD inputs, especially for the ones that are close to the decision boundary. Therefore, a potential idea is to expand the given outliers to cover more informative distributions and better generalize to the unseen OOD) inputs.
Based on the previous analysis, we propose a new learning framework, i.e., Divsified Qutlier Exposure (DivOE), to alleviate the pessimism of limited informative auxiliary outliers. At the high level, we aim to diversify the current distribution represented by the surrogate OOD data (e.g., Figure 2). In detail, we introduce novel learning objective (i.e., Eq. (4)) that conducts informative extrapolation based on the given auxiliary outliers. Through a multi-step optimization target for generating a fraction of more informative outliers, DivOE adaptively extrapolate and learn beyond the original OOD distribution. It realizes expanding the overall surrogate distribution of OOD data toward a broader coverage by maximizing the difference between the extrapolated one with its original counterpart.
We conduct extensive experiments to characterize and present our proposed methods. We have verified its effectiveness with series of OOD detection benchmarks mainly two common ID datasets, i.e. CIFAR-IO and CIF,R-100, and also demonstrated its scalability using the ImageNet dataset with the large-scaled auxiliary outliers sampled from ImageNet21K. Under the various evaluations, our DivOE via informative extrapolation, can obtain the better OOD discriminative capability for the models and consistently achieve the lower averaged FPR95 compared with different baselines. Finally, range of ablation studies from various aspects of the learning framework and further discussions from different perspectives are provided. Our main contributions summarized as follows,
Conceptually, we study a more general and practical research setting in outlier exposure for OOD detection, considering the auxiliary outliers having limited information that can not well represent the whole decision areas for ID and OOD data. (in Section 3.1)
Technically, we propose novel learning framework, namely Diversified Outlier Exposure (DivOE), for facilitating fine-tuning with auxiliary outliers, which conducts informative extrapolation to adaptatively diversify outlier exposure. (in Sections 3.2 and 3.3)

---

Empirically, extensive explorations from different perspectives are conducted to verify the effectiveness of DivOE in improving OOD detection performance, and we perform various ablations or further discussions to provide thorough understanding. (in Section 4)
2 Background
In this section, we briefly introduce the preliminaries and related work in OOD detection.
2.1 Preliminaries
We consider multi-class classification as the original training task [Nguyen et al., 2015], where X' C IR denotes the input space and J (1...., C} denotes the label space. A reliable classifier should be able to figure out the OOD input, which can be considered a binary classification problem. Given P. the distribution over X' x V. we consider Din as the marginal distribution of P for X, namely, the distribution of ID data. At test time, the environment can present a distribution Don over X' of OOD data. In general, the OOD distribution Dour is defined as an irrelevant distribution of which the label set has no intersection with Y and thus should not be predicted by the model f(.). A decision model g(t) can be made with the threshold A:
[ID
S(z; 2 A
91 (2; f) OOOD
(I)
S(x; f) < X
Building upon the model f H : + RR trained on ID data with the logit outputs, the goal of decision is to utilize the scoring function S X' + R to distinguish the inputs of Din from that of Domt by S(x; f). If the score value larger than the threshold X, the associated input æ is classified as ID and vice versa. We consider several representative scoring functions designed for OOD detection our exploration, e.g., MSP [Hendryeks and Gimpel, 2017], ODIN [Liang et al., 2018], and Energy [Liu al., 2020]. More detailed definitions are provided in Appendix A.
To mitigate the issue of over-confident predictions for [Hendrycks and Gimpel, 2017, Liu al., 2020] some OOD data, another line of research directions [Hendrycks et al., 2019, Tack et al., 2020] utilize the auxiliary unlabeled dataset to regularize the model behavior. Among them, one representative baseline is outlier exposure (OE) [Hendrycks et al., 2019]. OE can further improve the detection performance by making the model f(-) finetuned from a surrogate OOD distribution Dx D and its corresponding learning objective is defined as follows,
(2) where X is the balancing parameter, (cs(.) is the Cross-Entropy (CE) loss, and fog(.) is the Kullback-Leible divergence to the uniform distribution, which can be written Lou (h())) Ln sof tmax; fxx)/C. where sof toft,ax,,) (-) denotes the k-th element of a softmax output. The OE loss (-E(.) is designed for model regularization, encouraging the model leam knowledge from the surrogate OOD inputs and return low-confident predictions [Hendrycks and Gimpel, 2017].
Lor = ED,, [f v)] + AE [(o)([(x)]
2.2 Related Work
OOD Detection. [Hendryeks and Gimpel, 2017] formally benchmarks the OOD detection problem, proposing to use softmax prediction probability as conventional baseline method. Subsequent works [Sun et al., 2021] keep focusing on designing post-hoc metrics to distinguish ID samples from OOD samples, among which ODIN [Liang et al., 2018] introduces small perturbations into input images to facilitate the separation of softmax score, Mahalanobis distance based confidence score [Lee et al., 2018b] exploits the feature space by obtaining conditional Gaussian distributions, energy-based score [Liu et al., 2020] aligns better with the probability density. Except directly designing new score functions, some works [Lin et al., 2021, Sun et al., 2022, Djurisic et al., 2023] pay attention to various aspects to enhance the OOD detection such that LogitNorm [Wei et al., 2022] produces confidence scores by training with a constant vector norm on the logits, DICE [Sun and Li, 2022] reduces the variance of the output distribution by leveraging the model sparsification, and ASH [Djurisic et al., 2023] explores the manipulation of feature representation to enhance OOD detection. In addition, Zhu et al. [2023] improves detection performance by investigating the quality of ID data.
OOD Detection with Auxiliary Outliers. Another promising direction toward OOD detection involves the auxiliary outliers for model regularization. From the data perspective, some works
3

---

180
175
17.
EPROS
Pest
Orgaa DIAT
CfferentNuxdlary Oudlers
Figure 2: Empirical demonstration about informative outliers in OE and the diversification effect of the proposed DivOE. In the two left panels, we illustrate auxiliary outliers in different informative levels and divide them into five groups according to the degree of diversity. Empirically, we show that the OOD detection performance of OE can be benefited by the most diversified Groupl among these trials. We leave the experimental details in Appendix C for reference. In the two right panels, we compare the TSNE visualization of the original outliers with our extrapolated outliers DivOE.
explore generating virtual outliers such that VOS [Du et al., 2022] and NPOS [Tao et al., 2023] regularize the decision boundary by adaptively sampling virtual outliers from the lilliiil region. On the other hand, other works exploit information from natural outliers, such that outlier exposure introduced by [Hendrycks al., 2019], given that real OOD data are available in enormous quantities. Recently, some works highlight the importance of sampling strategy, such that NTOM [Chen et al., 2021] greedily utilizes informative auxiliary data tighten the decision boundary for OOD detection, and POEM [Ming et al., 2022] adopts Thompson sampling to contour the decision boundary precisely. The performance of training with outliers usually superior to that without outliers, shown in many other works [Fort et al., 2021, Salehi et al., 2021, Sun et al., 2023]. From the optimization perspective, Wang et al. [2023] introduces implicit regularization via model perturbation, and Wang et al. [2022], Yang et al. [2023] present early trials to improve OOD performance in terms of the AUROC with tighter generalization bounds and new evaluation metrics.
3 Method: Diversified Outiler Exposure
In this section, we introduce our new framework, i.e., Diversified Outlier Exposure (DivOE), which conducts informative extrapolation during fine-tuning with auxiliary outliers. First, we present and discuss the critical motivation that inspires our method (Section 3.1). Second, we introduce its newly derived learning objective and explain the underlying implications (Section 3.2). Lastly, we present the algorithmic realization DivOE and discuss compatibility with other methods (Section 3.3).
3.1 Motivation
First, collecting OOD samples near the boundary still remains challenging as we can hardly know which kind samples are truly located the decision boundary between ID and OOD space. Second, sufficiently
empirically shown in the left-middle panel of Figure 2, if the auxiliary outliers outlier exposure will be limited due to the
informative (e.g.. less diversified), the performance under-represented OOD distributions, which be reflected by the higher FPR95 score (indicating higher error on OOD detection). Thus, it naturally arises the following research question,
How could we utilize the given outliers for effective OOD detection if the auxiliary outliers are not informative enough to represent the unseen OOD distribution?
Especially for those OOD inputs that are close to the decision boundary. the auxiliary outliers may well characterize such broad distributional area [Fang et al., 2022, Wang et al., 2023, Zhang et al., 2023]. Therefore, a new mechanism is required to diversify the outlier exposure by exploring more potential OOD distribution for effective OOD detection. As in the right two panels of Figure DivOE realizes this expectation by explicitly extrapolating the OOD data engaged during training.
3.2 DivOE via Informative Extrapolation
As aforementioned, the OE paradigm heavily relies on the auxiliary outliers sampled from the surro- gate OOD distribution. Given the auxiliary outliers that not informative enough to represent the

---

unseen OOD data, the model is expected to learn beyond the current surrogate OOD distribution. One conceptual idea to achieve this goal extrapolate based on the current surrogate distribution. Under such a concurrent learning paradigm, the model can be regularized by the original surrogate OOD distribution and simultaneously generalize beyond the given auxiliary outliers for OOD detection. To this intuition, we consider an adaptively evolved version of the surrogate OOD distribution D under the learning framework of OE as follows,
fOr = D.. [fer(f(x), 3)1 + AE ((x)))
(3)
where Dau indicates the extrapolated surrogate distribution during training and X indicates the newly manipulated samples that are different from those in the original objective (i.e., Eq. (2)) of OE, In the above learning objective, the left part is for the original classification task, and the right part is for the newly proposed informative extrapolation in our DivOE. With the conceptual target for diversifying the current surrogate OOD distribution, we consider reformulating the specific objective as
EDi.. LOEE (f(3)) (1 ()) D(m) + 3 A (foz(f: D lor(f: Dou))
(4)
Informative Extrapolation
where 3 indicates a balancing factor for controlling the extrapolation ratio and D represents the synthesized distribution with manipulation A. Intuitively, the overall objective defined in Eq. (4) not only learns from the original surrogate OOD distribution but also generalizes to the different OOD distributions via the maximization part. The underlying insight is to expand the surrogate OOD distribution towards a more diversified one. The informative extrapolation is realized by the maximization part that maximizes the differences between the original surrogate OOD distribution and the generated one. Note that, Eq. (4) corresponds to an expectation formulation based on the whole population for diversifying outlier exposure, which is prohibitively expensive in realization. To achieve that more efficiently, we introduce an instance-level version for practical realization, and its optimization target can be formulated as the following hybrid loss,
(on(s())
n rn Σ (J)))
1
TT Σ max lor +)),
Original Distribution
Informative Extrapolation
where n indicates the sample number of auxiliary outliers and r indicates the extrapolation ratio that corresponds to 3 in Eq. (4). Except for the original loss learning with the given outliers, the latter part of Eq. (5) defines a surrogate optimization part for realizing the informative extrapolation.
Here we adopt instance-level maximization for manipulating the expected OOD samples based on the original one, which the specific is obtained by the following optimization objective,
arg max fo (f + +)),
(6)
Intuitively, DivOE will generate outliers closer to the decision boundary expand the coverage area of auxiliary outliers, and simultaneously be more informative to shape the decision area between ID and OOD data. The specific optimization process of 5 is characterized as,
8 t- 8 + (asign(Vsoott (o + 8)), sign Dign(Valozza (Volo + 6)) sign( V (5 (x + 8));
where sign is the function that extracts the sign of the tensor elements and S(: indicates the general OOD score functions. The optimization process also indicates that our extrapolation framework is general and can adopt different score functions to generate targeted outliers. Here we also provide the theoretical implication of our proposed objective based on previous work [Ming et al., 2022].
Theorem 3.1. Given simple Gaussian mixture model in binary classification with the hypothesis class H. sign(0" 2), E IRd, There exists conslant a, and o(e) that satisfy.
n T 1
||u| o |||||
o (a-r(e))
(8)
10 1
2 n 1 ||
From the perspective of sample complexity, we present the effects of our informative extrapolation. 1² 0'
u² T ""
As FPR(B er
crf(olle monotonically decreasing, the lower bound of a 0 m2
will
5

---

o8ginal
CFAR1S
CEPARO
Terares
Toduus PlaciCRS
FDS
Plonee365
p-2.1
6-2.15
0.6
0.0
02
Frwngy Soor
Erergy Soor
Energy Store
Figure 3: Empirical exploration of informative extrapolation. Left panel: visualization of extrapolated outliers w.r.t different diversified strength E [0, 0.15]; Left-middle panel: the corresponding density of the energy score distributions on the different extrapolated outliers; Right-midda panel: compari- son of ID/OOD distributions based on Energy score of the OE baseline; Right panel: comparison of ID/OOD distributions based on Energy score using the model fine-tuned by the proposed DivOE.
increase with the constraint of (a T(E)) (which corresponds to our DivOE that extrapolates the outlier boundary towards ID data) decrease in our methods, the upper bound of FPR(0", (0)..)))))) will decrease. The above indicates the benefit of DivOE with the informative extrapolation on shaping boundary decision areas. The completed definition and the analysis can be referred in Appendix
3.3 Realization of DivOE
In this part, we introduce the realization of the whole learning framework of DivOE in detail. Here we formally present the learning objective of DivOE via informative extrapolation as follows,
DDvO
m E (cE((())))) X( n TTL Σ (oe(f(x))
- rT Σ loe(f(> + +))),
where and indicate the sample numbers of ID data and auxiliary outliers respectively, other notations keep the same meaning as the previous equations. Concretely, the maximization part of DivOR adopts the multi-step optimization (e.g., projected gradient decent [Bottou, 2012, Madry et al., 2018]) to realize the targeted outlier synthetics. The specific process is defined as,
(z))++1) 1B va] (x(*) + (esign(V..t) (x=((((())))))
(10)
where E N, ((() is the extrapolated OOD data at step c, and B (-) is the projection function
that projects the new OOD data back into the e-ball centered at (0)) if necessary. Different from the conceptual idea of constraining the perturbation (for guman-imperceptiee adversarial robustness [Madry et al., 2018, Zhang al., 2019], the projection operation in DivOE provides a quantification way to study the extrapolated distribution and its diversification. As the generation process starts from the given outliers, the semantical class (belongs ID or OOD) is hard to change, as empirically shown in the left-most panel of Figure (we adopted a very large = 0.15). More exploration about this perspective will leave to Section 4.3 for further discussion.
in the literature on
Here we summarize the whole procedure of the proposed DivOE Algorithm 1, which consists of multi-round training iterations. To be specific, on each round of fine-tuning, DivOE first samples a mini-batch with a specific ratio r E [0, 1] from the auxiliary outliers and conduct multi-step optimization following Eq. (10) generate the new outliers. With the ID training data, the original outliers, and the extrapolated outliers, DivOE then fine-tune the model with the loss defined in Eq. (9).
In practical realization, for the specific extrapolation strength DivOE can also generate the extrapo- lation pool with multiple candidates within the extrapolated sub-batch. The specific definition of extrapolation pool is as follows,
0(X) = {x thi
Ed kern E,
(11)
Ep
where the extrapolation strength with its corresponding sample number can be flexibly decided in the extrapolated sub-batch with a total sample number equal diversification target. As primary exploration, we visualize the extrapolated data with its distribution quantified by Energy score [Liu et al., 2020] in the left two panels in Figure 3. Intuitively, even with a large extrapolation strength with will not change its classification results in OOD detection. can represent a more different OOD distribution compared with the given outliers the distribution shifted gradually to the right.
n. which may better facilitate the
pixel noise added in the original outliers, it

---

Algorithm Diversified Outlier Exposure (DivOE) via Informative Extrapolation
given model: 0, fine-tuning epochs T, training samples of ID data: Di training samples of auxiliary outliers: x² ~ Dan extrapolated ratio the original outliers: r, extrapolate epsilon: e, optimization step number: k, extrapolation pool: O(X);
Input:
Output: fine-tuned model '": for epoch ......... T do
2: for mini-batch 1, I...., . do
3: Sample mini-batch { 24) ))) from Din sample mini-batch ((rj)] from D
out
4: Sample sub-batch with ratio r from the batch of outliers {(a])}}=
5: for k = 1, ........
(in parallel) do Obtain the extrapolated outliers žे by Eq. (10) that following max fo (f (ft))
6:
7:
Aggregate all the extrapolated samples with uniform or different e in ( end for
8:
9:
0 0 n/
1 [ECE(f(=).))) Man- (oee(f(=') I (oE((((()))) m
10: end for
end for
Comparison and Compatibility. Compared with the conventional OE algorithm [Hendrycks et al., 2019], the critical difference behind DivOE is trying to extrapolate based on the given auxiliary outliers for diversifying outlier exposure. To be specific, it provides general framework (under the guidance of the learning objective in Eq. (5)) for modeling the informative unknown inputs beyond the original ones. The discriminative feature regularized by our DivOE can be utilized by those advanced post-hoc scoring functions [Sun al., 2021, Huang al., 2021, Sun and Li, 2022, Djurisic et al., 2023]. For OE- based methods, the informative extrapolation introduced in DivOE is orthogonal to current uning objectives [Hendrycks and Gimpel. 2017, Wang et al., 2023] and also compatible to those different data augmentation [Zhang et al., 2018, Yun et al., 2019], synthesizing [Kong and Ramanan, 2021, Lee al., 8a], and also sampling strategies [Chen et al., 2021, Ming et al., 2022] adopted based on either ID data [Du et al., 2022, Tao al., 2023] or auxiliary outliers.
4 Experiments
In this section, we present the comprehensive verification of the proposed DivOE in the OOD detection scenario. First, we provide the experimental setups in detail (in Section 4.1). Second, we provide the performance comparison of our DivOE with series of post-hoc scoring functions and the OE-based methods with different strategies on the auxiliary outliers (in Section 4.2). Third, we conduct various ablation studies and further discussions to understand our DivOE (in Section 4.3).
4.1 Setups
Here we present several critical parts of experimental setups and leave more details in Appendix C.
Datasets. Following the common benchmarks used in previous work [Liu et al., 2020, Ming et al., 2022], we adopt 100 CIFAR-100 [Krizhevsky, 2009] as our ID datasets. We use series of different image datasets as the OOD datasets, namely Textures [Cimpoi et al., 2014], Places365 [Zhou et al., 2017], iSUN [Xu et al., 2015], LSUN_ Crop [Yu et al., 2015), and LSUN _esizee [Yu et al., 2015]. For the surrogate OOD data used in OE-based methods [Hendrycks et al., 2019, Chen et al., 2021, Ming al., 2022, Zhang et al., 2023], adopt TinyImmaeeNe [Le and Yang. 2015]. We also conduct the experiments on the ImageNet, where we utilize the large-scaled ImageNet21K [Ridnik al., 2021] the surrogate OOD like previous works [Wang et al., 2023) for OF-based methods.
Evaluation metrics. We employ the following three common metrics to evaluate the performance of OOD detection: (i) Area Under the Receiver Operating Characteristic curve (AUROC) [Davis and Goadrich, 2006] can be interpreted as the probability for positive sample to have higher discriminating score than a negative sample [Fawcett, 2006]; (ii) Area Under the Rrecision-Recall curve (AUPR) [Manning and Schütze, 1999] is ideal metric to adjust the extreme difference
7

---

Table I: Main Results (%). Comparison with competitive OOD detection baselines. All methods trained on the same backbone. Values are percentages. Bold numbers are superior results. 1 indicates larger values are better, and indicates smaller values better. (averaged by multiple trials)
Method
FPR95]
AUROC1
AUPR
ID:ACC1
w//.o Pac
Operation for Pas
MSP [Handrycks and Gimpel, 2017]
$2.21
90.61
97.88
94,84
ODIN [Liang el al., 2018]
52.84
90.67
97.29
94.81
Maltalanohii [Lce
2018b]
92,60
94.84
Fnergy [1 et al., 2020]
32.82
91.99
97. .6
ASH [Djurisic et ol., 2023]
33.01
92.11
97.92
94.84
CIBAR-10
OE [Hendryaks al. 2019]
13.76+40.29
97.53+00.33
99 49.43-00.01
94.51=00.06
Vanilla
Energy P [Lio al., 2020]
18.15:40.32
90.90110.18
94.51
Vanilla
VOS [Du et al. 2022]
34.20:10.02
91.27:10.0]
97 97.001
91.02200001
Virtually Synthesized
NTOM [Chen et 2002)
17.76:10.84
91 11.08::044
96 00..00.1..18
05,044::0003
Groody Sampled
POEM [Ming 20.2]
17.8110.78
90.9610.33
96 16..6.0515
94.90:1:.04
Thompson Sumpled
MixOE IZhang 2013)
13.57:10.25
97.17:0.007
91.98:10:02
Misup with Data
DivOE (Ouns)
11.66:10.04
97.82::0.04
99.48:10.01
94.66:20.12
Extrapotated
MSP [Hendrycks and Gimpel, 2017]
79.71
76.13
94,077
75.95
ODIN [ang
2018]
63.29
75.95
Mahalanobis [Loc et al., 201888
52,90
83,86
95.73
75.95
Energy [Lái et al., 20200
71.95
80:27
95.001
75.95
ASH [Djurisic et ol. 2023]
61.82
83.42
95. 84.84
75.87
CIFAR-100
[Heendryckk et al., 2019]
27.67:10.33
01.89:1.122
97.93-10.04
75.41100.10
Vanilla Vanilla
Energy (w. Pa..) [Lin et al., 2020]
27.86:10.26
90.70:10.07
97.10.10.03
75.03:10.11
VOS [Du et al. 2022]
70.87:10.00
80.00:00.00
94 $9.10.000
76.25::0.00
Vistually Synthesized
NTOM [Chen et 2021]
27.58.10.88
91.0610.34
97 51.52.00000
75.41
Greedy Sampled
POEM [Ming et 2022]
26.50:10.10
91.3311.006
58:00000
75.711:0.14
Thompoon Sampled
MixOE 12hang al., 2023)
35.26:1:0.80
10
97 58:10.00
75.741:0:04
Mixup with Data Extrapolated
DivOE (Ouns)
24.80.10.51
92.911.0111
98.22:10.03
75.24
MSP [Hendryoks and Glimpel, 2017]
75.23
76.67
94. 34.35
68.72
ODIN [Liang el al., 2018]
67.61
80.36
95.15
68.72
Mahulamobis [1.oz at al., 2018b]
83.41
58.47
87.72
68.72
Fnergy [1 et al., 3020)
71.44
79.52
94.98
68.72
ASH [Djurisic et 2023]
73.51
93.43
ImageNet
OB (Hendryaks et al., 2019)
61.94:30.03
81.58:00.01
95 49:10.00
75. 75,90:00:00
Vanilla Vanilla Virtually Synthesized
Energy (w. Pin.) [Lis et al., 2020]
66.941:0.01
79.93110:02
94, 9.100.000
74. 74.19:10.08
VOS [Du 2022]
71.00.10.02
78.97:00.01
91.82210.00
05.10210.02
NTOM [Chen al. 2021]
66.32-40.24
80.144:0.08
95-4000222
74.26=20001
Greedy Sampled
POEM [Ming al., 20.21
67.81110.44
79.36:10.24
94.73:10.06
74.21-10.03
Thompson Sampled
MixDE (Zhang 2023)
70.12:1:0.6
78.8510.005
91 91.85:10.02
76.1810.011
Misup with ID Data Extrapolated
DivDE (Ouns)
60.12::0.11
81,96200.00
95.59-10.00
75.73::0022
between positive and negative base rates; (iii) False Positive Rate (FPR) at 95% True Positive Rate (TPR) [Liang et al., 2018] indicates the probability for negative sample to be misclassified as positive when the true positive rate at 95%. We also include in-distribution testing accuracy (ID-ACC) to reflect the preservation level of the performance for the original classification task on ID data.
OOD detection baselines. We compare the proposed method with several competitive baselines in the two directions. Specifically, we adopt Maximum Softmax Probability (MSP) [Hendrycks and Gimpel, 2017]. ODIN [Liang et al., 2018). Mahalanobis score [Lee et al., 2018b], and Energy score [Liu al., 2020] scoring function baselines; We adopt OE [Hendrycks et al., 2019], Energy- bounded learning [Liu et al., 2020]. NTOM[Chen et al., 2021], POEM [Ming et al., 2022] and MixOE |Zhang et al., 2023] as baselines with outliers. For all scoring function methods, we assume the accessibility of well-trained models. For all methods involving outliers, we constrain all major experiments to a fine-tuning scenario, which is more practical in real cases. Different from training a dual-task model at the very beginning, equipping deployed models with OOD detection ability is a much more common circumstance. We leave more definitions and implementations in Appendix A.
4.2
Main Results
In this part, present the major performance comparison with some representative baseline methods for OOD detection to demonstrate the effectiveness of the proposed DivOE. Specifically, we consider several post-hoc methods of scoring functions as the performance reference based on the pre-trained model and some OE-based methods for specific comparison on fine-tuning with auxiliary outliers. Note that for all the experiments here, we utilize the whole auxiliary outliers (following the general setup of OE [Hendryeks et al., 2019, Liu et al., 2020] and ensuring the same number of outliers engaged in each method) instead of the reduced ones adopted in previous figures for illustrations.
In Table 1, we present the overall results using different methods for OOD detection. Since the OE-based methods engage the auxiliary outliers during training, the model will generally gain better
8

---

Table 2: OOD detection performance on compatibility experiments. All methods are trained on the same backbone. f indicates larger values are better, and I indicates smaller values are better.
D2
Method
CIFAR-10
CIFAR-100
FPR95]
AUROCT
AUPRI
ID-ACCT
FPR95
AUROCT
AUPR
ID-ACC1
DE
13.76::0.23
97.53:10033
99,43-0.01
94.51100.06
27.67=10.43
91.89-00.12
97.93=0.04
75.41=00.10
DiVOE
11.66-00.04
97.82:0.004
99.48:0.01
94.66+0.12
24.80
92.91-00111
98.22-0.03
75.24=0.01
Energy [w. Den)
18.15:20.32
90.90100118
96.34:2.100
91.51100.01
27.86:20.26
90.70=0.07
97.40=0.03
75.03=0.111
Energy Paad-DivEE
16.03::0.35
92.23310.10
9..87.0.03
0.05
0..29
92,04-0.199
97.82-0.08
05.25
NTOM
17.76-+0.84
91 08++0,44
96.40=0.18
95.04+0.033
27.58-00.88
91.08--0.1
97.52-0.10
NTOM-DWOO
15.30::0.57
92.21:10.57
96.73:20.28
94.75200.02
25.97=20.74
91.62:0.29
9716
75.39:20.15
POEM
17.81=00.8
90 90-96-200.77
W6. 15
94.99:200.04
26. 06.50==0.10 10
91.33==0.16
VT. S8 0.00
75.74=0014
POEM+DiVOE
15.22::0.20
92.42::0.08
91.66:10.07
25.05=00003
91.91=0.07
97.77-0.03
DECC
17.65::0.06
96 68:10.03
99. 19.19
94,50:10.07
28.51
91.08:10.08
97.70=00.03
74.95
ORCCADIVOR
14.18:0.04
97.26-+0.07
99.32=-0.03
94 15
26.13-00.99
92.07=00114
97.97
74.74+0036
7.39:10.46
98 29::0.08
92.82:+:0008
30.69=00992
93.13::0115
98.
DOE+DivOE
6.51110.10
98.60:10.01
99.65 0.01
93.45110.02
20.26:10.20
95.41::0.05
0.002
71.43:20.00
empirical performance on OOD detection, reflected by the evaluation metrics. For those OE-based methods, we provide a specific comparison on their detailed operation for Daux (c.g., Dia) in the right of Table 1. Since VOS virtually synthesizes the outliers based on the data, doesn't perform better than those using the real collected outliers. Among the rest, NTOM and POEM adopt different strategies for informative sampling, which obtain different levels of detection performance gains across different ID datasets. MixOE that mixups ID and surrogate OOD data does not perform well on some complexed tasks like CIFAR- 100 and ImageNet, since the original algorithm is designed for fine-grained OOD detection. Without sacrificing much classification performance (i.e., ID-ACC) on ID data, our DivOE can consistently achieve better OOD detection performance across the three datasets, which verifies the effectiveness of our methods with the newly proposed informative extrapolation for outlier exposure. We also summarize the fine- grained results on OOD detection to Tables 12 and 13 in Appendix C, which provides a better understanding of the improvement.
In Table 2, we report the results of compatibility experiments, in which we compare those OE-based methods with their variants, incorporating our DivOE for informative extrapolation. Specifically, we consider some of the previous OE-based methods and also add two related methods, e.g., OECC [Pa- padopoulos et al., 2021] and DOE [Wang et al., 2023], in the comparison. Although those methods can have different level of performance enhancement compared with the conventional OE baseline, We can find that our DivOE can consistently help them gain better OOD detection performance across three evaluation metrics, and keep the ID-ACC comparable with the original pre-trained model. As for DOE, it utilizes the model perturbation on implicit outlier synthesizing, which is also orthogonal and compatible with our DivOE for explicit extropolation. In this table, we can find the plain DivOE can achieve lower FPR95 on CIFAR- 100 with DOE, while the performance of DOE is affected by different ID datasets. The above demonstrates the effectiveness and algorithmic robustness of our proposed method. In addition, also provide further discussion on the differences in Appendix 1.
4.3 Ablation and Further Discussions
In this part, we conduct further explorations to provide a thorough understanding of our DivOE. For the extra results and discussions (e.g., impact and limitations), we leave more details in Appendix C.
Importance of extrapolation ratio in diversifying the distribution. As a critical aspect of our proposed learning objective of DivOE in Eq. (9), the extrapolation ratio is introduced to control the manipulated portion of the given auxiliary outliers. DivOE aims to extrapolate based on the surrogate OOD distribution without total loss of the original representation. In Figure 4(a), we show the performance by varying the ratio r. It is worth noting that adopting extrapolation on the full batch of data may even degrade the performance, indicating the vital role of the extrapolation ratio in the whole learning framework. It performs diversifying as the maximization part introduced in Eq. (5) instead of an overall augmentation. The results verify the rationality of pursuing diversified outliers instead of only most informative ones. We provide more comparison on the concepts of "diversified" and "informative" in Appendix A.2, and leave more results in Appendix C to further explain it.
Extending the diversified strength in the informative extrapolation. To better characterize the effects of informative extrapolation in DivOE, we consider fine-tuning the given model with

---

912
Dx0C
37.5
CEARI00
CHANTE
F7785
FPR05
FIRRSS
230
12.4
153
12.
20
032
0224
2:5
0.1
671
Terk youy Techhrre
1edF
Extrapciation
Augnertstion
(a) Extrapolated Ratio
(b) Diversified Strength
(c) Extrapolation Method
(d) Compare with Augs.
Figure 4: Ablation Study. (a) performance of using different extrapolated ratios of DivOE; (b) explo-
ration of different diversified strength e; (c) using different OOD scores as informative extrapolation targets; (d) comparison between DivOE and OE adopting other augmentations for diversification.
different (with consistent extrapolation ratio, 0.5), which is used constrain the diversified strength in maximizing the optimization target in Eq. (6). In Figure 4(b), we can observe that an appropriate diversified strength can help OE performs better on OOD detection, while using large may empirically return to the original baseline case. One possible reason may be that the extrapolated outliers with large may fail to become more informative as previously reflected in Figure 3.
Generality of using different OOD scores. Since DivOE introduces a general learning framework of outlier exposure for model finetuning with auxiliary outliers, the specific realization for informative extrapolation can have multiple choices. For example, different scoring functions can all perform OOD uncertainty estimation, indicating the multiple candidate target for Eq. (6) to optimize. Here report the performance using different optimization targets (e.g.. MSP or Energy score) in Figure 4(c), where they have different performance improvements compared with the original OE baseline.
Comparison with different augmentation-basee methods for OF. In Figure 4(d), we perform the comparison of our DivOE with the original OE adopting (e.g., Mixup [Zhang et al., 2018) and CutMix (uun et al., 2019]) for diversifying the given outliers. The results verify the superiority of DivOE on OOD detection over the direct adaptation of those strategies since they lack a critical diversification guidance. On the other hand, our framework has no strict constraints for the detailed manipulating operations. In other words, DivOE can also utilize those augmentation methods for extrapolation, for which we leave more discussion in Appendix C.
representative augmentation methods
Conclusion
In this paper, we propose a novel learning framework, i.e., Diversified Outlier Exposure (DivOE), that promotes diversity in outlier exposure with the given auxiliary outliers. To empower the model with more knowledge about the decision boundary between and OOD data, DivOE conducts informative extrapolation guided by newly designed objective. Through the diversification target, our method adaptively synthesizes meaningful outliers during the training process, which helps the model better shape the decision demonstrate the effectiveness of our proposed DivOE and its compatibility with a range of OE-basce methods, and also various ablation studies with further explorations characterize the framework.
for OOD detection. We have conducted extensive experiments to
Acknowledgments
JNZ and BH were supported by the NSFC Young Scientists Fund No. 62006202, NSFC Gen- eral Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A151501 1652, RIKEN Collaborative Research Fund, HKBU Faculty Niche Research Areas No. RCC-FNRA-IG/2-223SCCA were supported by the National Key R&D Program of China (No. 2022ZD016070S 111 plan (No. BP0719010) and National Natural Science Foundation of China (No. 62306178). TLL was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527. LP220200949, and I IC190100031.
and HKBU CSD Departmental Incentive Scheme. GY and JCY
10

---

References
Second Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. In arXiv, 2021.
Karsten M Borgwardt, Arthur Gretton, Malte Rasch, Hans-Peter Kriegel, Bernhard Schölkopf; and Alex J Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 2006.
Léon Bottou. Stochastic gradient descent tricks. Neural Networks: Tricks of the Trade: Edition, 2012.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of- distribution detection using outlier mining. In ECML PKDD, 2021.
Mircea Cimpoi, Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In ICML, 2006.
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In ICLR, 2023.
Xuefeng Du Zhaoning Wang, Mu Cai, and Yixuan Li. VOS: learning what you don't 't know by virtual outlier synthesis. In ICLR, 2022.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 2011.
Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is -iito--distribution detection learaall?? In NeurlPS, 2022.
Tom Fawcett, An introduction roc analysis. Pattern Recognition Letters, 2006.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In ICLR, 2021.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-diitrrbuttin
detection. In NeurlPS, 2021.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribttion examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mosta- jabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. In ICML, 2022.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In NeurIPS, 2021.
Julian Katz-Samuels, Julia Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors in their natural habitats. ICML, 2022.
Kiefer and J. Wolfowitz. Stochastic estimation of the maximum of a regression function. Ann. Math. Statist, 1952.
Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In CVPR, 2021.
Alex Krizhevsky, Learning multiple layers of features from tiny images. In arXiv, 2009.

---

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. In CS 231N, 2015.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. ICLR, 20.8a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, 2018b.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
Ziqian Lin, Sreya Dutta Roy, and Yixuan Li. Mood: Multi-level out-of-distribution detection. In CVPR, 2021.
Energy- based out-of-distribution Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li.
detection. In NeurlPS, 2020.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ICLR, 2018.
Christopher D. Manning and Hinrich Schütze. Foundations of Statistical Natural Language Process- ing. The MIT Press, 1999.
Yifei Ming, Ying Fan, and Yixuan Li. Poem: Out-of-distribution detection with posterior sampling. In ICML, 2022.
Sina Mohseni, Mandar Pitale, J. B. S. Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable distri-iistiitution detection. In AAAI, 2020.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, 2015.
Aristotelis-Angelae Papadopoulos. Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang. Outlier exposure with confidence control for out-of-distribution detection. Neurocomputing, 2021.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. In NeurIPS Datasets and Benchmarks Track, 2021.
Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out-o--distribution detection: Solutions and future challenges. In arXiv, 2021.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. SSD: A unified framework for self-supervised outlier detection. In ICLR, 2021.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In ECCV, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In NewrlPS, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In JCML, 2022.
Yiyou Sun, Zhenmei Shi, Yingyu Liang, and Yixuan Li. When and how does known class help discover unknown ones? provable understandings through spectral analysis. In ICML, 2023.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: Novelty detection via contrastive learning on distributionally shifted instances. In NeurIPS, 2020.
Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. In ICLR, 2023.
12

---

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two sample. Biometrika, 1933.
Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander, Tongliang Liu, Jianye HAO, and Bo Han. Out-of-distribution detection with implicit outlier transformation. In ICLR, 2023.
Zitai Wang. Qianqian Xu, Zhiyong Yang. Yuan He, Xiaochun Cao, and Qingming Huang. Openaue: Towards auc-oriented open-set recognition. In NeurlPS, 2022.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICML, 2022.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. In arXiv, 2015.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection:
survey. In arXiv, 2021.
Jingkang Yang. Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, WENXUAN PENG, Haoqi Wang. Guangyao Chen, Bo Li, Yiyou Sun, et al. Openood: Benchmarking generalized oft-o--distribution detection. In NeurIPS Datasets and Benchmarks Track, 2022.
Zhiyong Yang, Qianqian Xu, Shilong Bao, Peisong Wen, Yuan He, Xiaochun Cao, and Qingming Huang. Auc-oriented domain adaptation: From theory algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. In arXiv, 2015.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In CVPR, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. In ICLR, 2018.
Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards diitr--distiibttionn detection in fine-grained environments. In WACV, 2023.
Bolei Zhou, Agata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding. Journal of Vision, 2017.
Jianing Zhu, Hengzhuang Li, Jiangchao Yao, Tongliang Liu, Jianliang Xu, and Bo Han. Unleashing mask: Explore the intrinsic aut-of-distriuuton detection capability. In ICML, 2023.
13

---

Appendix for DivOE
The whole Appendix is organized follows. In Appendix A, present the detailed definitions and implementation of those post-hoe score functions and several OE-based methods that are considered in our exploration. In Appendix B, we formally present the theoretical analysis about DivOE. In Appendix C, we provide our extra experimental details and more comprehensive results with further discussion on the underlying implications. Finally, in Appendix D, we discuss the potential broader impact and limitations of our work.
Reproducibility Statement
Below summarize several important aspects facilitate reproducible results:
The datasets we used are all publicly accessible, which is introduced in Section 4.1. For methods involving auxiliary outliers, strictly follow previous works [Liu et al., 2020, Sun et al., 2021] to avoid overlap between the auxiliary dataset (Tiny-ImageNer and any other OOD datasets.
Datasets,
Assumption. We set our experiments to a fine-tuning scenario where well-trained model the original classification task available, and the training samples are also available for subsequent fine-tuning [Hendrycks et al., 2019].
Open source. We open our source code the repository: DivOE
Environment. All experiments conducted with multiple runs NVIDIA GeForce RTX 3090 GPUs with Python 3.7 and PyTorch 1.12.
A Details about Considered Baselines and Metrics
In this section, we provide the details about the baselines for the scoring functions and fine-tuning with auxiliary outliers, as well as the corresponding hyper-parameters and other related metrics that are considered in our work. Additionally, we provide further discussion on the comparison with two related works, e.g., MixOE and DOE, and also present the critical differences between the diversifying with previous concept of informative.
Maximum Softmax Probability (MSP). [Hendrycks and Gimpel. 2017] proposes to use maximum softmax probability to discriminate ID and OOD samples. The score is defined as follows,
(2; f) max Ply = c|a; f) = max tof
(12)
where f' represents the given well-trained model and is one of the classes y = [1,.... CJ. The larger softmax score indicates the larger probability for sample to ID data, reflecting the model's confidence on the sample.
ODIN.
[Liang al., 2018] designed the ODIN score, leveraging the temperature scaling and tiny perturbations to widen the gap between the distributions of ID and OOD samples. The ODIN score is defined as follows,
SODIN (z; f) max Ply f) max softmax [(2)
(13) where represents the perturbed samples (controled by e). T represents the temperature. For fair comparison, we adopt the suggested hyperparameters [Liang et al., 2018]: = 1.4 x 10 10-3, T' 1.0 x 104.
T
Mahalanobis. [Lee et al., 2018b] introduces a Mahalanobis distance- based confidence score, exploiting the feature space of the neural networks by inspecting the class conditional Gaussian distributions. The Mahalanobis distance score defined as follows.
Muhhalanohis (v; /) max (J(x) A "
(U(x)-ii)
(14)
where fec represents the estimated mean of multivariate Gaussian distribution class e, E represents the estimated tied covariance of the C' class-conditions Gaussian distributions.
14

---

Energy. [Liu et al., 2020] proposes to use the Energy of the predicted logits to distinguish the ID and OOD samples. The Energy score is defined as follows,
Senrggy (z: f) = -T' log Σ M
(15)
where T' represents the temperature parameter. As theoretically illustrated in Liu al. [2020], a lower Energy score indicates higher probability for sample to be ID. Following [Liu et al., 2020], we fix the T 1.0 throughout all experiments.
ASH. [Djurisic et al., 2023] designs a extremely simple, post-hoc method called Activation SHaping for OOD detection. removes large portion of input's activation at a late layer and adjusts the rest of the activation values by scaling them up or assigning them a constant value. The simplified representation then passed throughout the rest of the network. The logit output is used classify ID samples and calculate scores for OOD detection as usual. We adopt the energy score and apply the ASH-S version with the hyperparameter p = for CIFAR- 10 and p = for CIFAR-100, as suggested by [Djurisic et al., 2023].
Outlier Exposure (OE). [Hendrycks et al., 2019] initiates promising approach towards OOD detections by involving outliers to force apart the distributions of ID and OOD samples. In the experiments, we use the cross-entropy from (3 )o the uniform distribution as the Loe [Lee et al., 201 8a],
Lj = ED, (I), + AEDS log Σ M()) - -EDA (1(1))
(16)
Energy (w. D P.ux). In addition using the Energy as post-hoc score to distinguish ID) and OOD samples, [Liu et al., 2020] proposes an distributions. The OE objective as follows,
objective to further separate the two
Cor Epy (ma SEeeryy (I, f) m)n)) + ET (max(0, Mont S (2, f))))
(17)
We keep the thresholds same to [Liu et al., 2020]: min - 23.0 for CIFAR-10 and Min = 25.0 for CIFAR-100, Mout 550 for both CIFAR-10 10 and CIFAR-100.
[Chen et al., 2021] greedily exploits informative auxiliary data to tighten the decision boundary between ID and OOD samples. It samples hard outliers with lower OOD scores to regularize the model for OOD detection. For a fair comparison, we adopt the energy score for both outlier sampling and model training and choose the suggested sampling parameter p = 0 [Chen et al., 2021].
NTOM.
POEM.
[Ming et al., 2022] explores the Thompson sampling strategy [Thompson, 1933] to make the most use of outliers to learn tight decision boundary. The details of Thompson sampling can refer to Ming et al. [2022].
MixOE. [Zhang al., 2023] proposes mix ID data and auxiliary outliers cover the broad region where fine-grained OOD samples lic. The model is then trained such that the prediction confidence linearly declines as the input shifts from ID distribution to OOD distribution. Considering [Zhang et al., 2023] chooses different OOD test datasets from our work, we set the hyperparameters and 3 0.1 which are verified to be optimal by our tuning experiments.
DOE. [Wang et al., 2023] proposes to implicitly synthesize virtual outliers via model perturbation, which enhances the generalization of OE to better optimize the learning target [Foret al., 2021].
A.1 Closer Look at Comparison with MixOE and DOE
In this part, we discuss more about the difference of our DOE with previous MixOE and DOE.
Conceptually, DivOE has different underlying motivation from MixOE and DOE. Our DivOE is proposed for diversifying outlier exposure by extrapolating auxiliary outliers. comparison, MixOE focuses on enbancing the fine-grained OOD detection by interpolation between ID samples and OOD
15

---

samples; DOE focuses on improving the generalization of the original outlier exposure by exploring model-level perturbation.
Technically, DivOE has a different algorithmic design from the others. Specifically, DivOE was proposed as general framework for outlier extrapolation, which bases OOD samples. It is different from MixOE which directly mix-up ID and OOD samples; It is also different from DOE that constructs the model perturbation for better optimization. It is worthy noting our DivOE is from a different perspective (data-level) compared with DOE (model-level), and they are orthogonal and combinable. DOE regards regularizing training on the same outliers with different weight perturbations as implicitly synthesized but does not change the outliers. The optimization of DOE is constrained by the original training samples and is also sensitive tuning the model perturbation.
Table 3: Comparison of DivOE and DOE across different ID classification tasks.
Dataset
Method
FPR95
AUROCT
AUPRT
ID-ACC1
OE
61.91
81.57
95.49
75,90
DivOE (OE backbone)
60.01
81.96
95.59
75.74
ImageNet
DOE
65.90
79.32
90.61
75.38
DivOE (DOE backbone)
64.23
79.46
91.21
75.10
OE
3.99
99.33
99.85
91.86
DivOE (OE backbone)
2.24
99.60
99.91
91.72
SVHN
DOE
8.93
97.47
99.44
91.11
DivOE (DOE backbone)
4.24
98.68
99.62
91.50
OE
27.67
91.89
97.93
75.41
DivOE (OE backbone)
24.80
92.91
98.22
75.24
CIFARI00
DOE
30.69
93.13
98.39
71.41
DivOE (DOE backbone)
20.26
95.41
98.91
71.43
OE
13.76
97.53
99.43
94,51
DivOE (OF backbone)
11.66
97.82
99.48
94.66
CI
DOE DivOE (DOE backbone)
7.39
98.29
99.58
92.82
6.51
98.60
99.65
93.45
Empirically, shown in Table 1, have directly compared MixOE with our DivOE. The results show its performance is worse than our DivOE even the original OE. For DOE, since it is from different perspective and orthogonal, we compare DOE mainly in Table of the original submission, and here we present Table 3 for further discussion. It shows: (1) DOE is sensitive to the ID classification task and even performs worse than the original OE in CIFAR- 100////N//mmaeee (2) DivOE improves both OE and DOE under the corresponding backbones; (3) Even under unfair comparison between DivOE (OE backbone) vs. DOE, DivOE (OE backbone) still achieves the comparable or better performance ImageNeuSSVHNNIIFAE 100 datasets.
A.2 Conceptual Comparison of Diversified and Informative
In this part, we provide further discussion on the comparison of diversified and informative.
Conceptually, "diversified" is proposed to characterize one internal property of auxiliary outliers, while the "informative" in [Chen et al., 2021] behind "diversified" is that we hope the outliers can span more informative areas beyond the original ones, unlike sampling the most "informative" outliers (i.e., near- boundary OOD samples that are close to ID data). Intuitively, Group 1 in Figure 2 indicates the auxiliary outliers which span more informative areas among those other groups In other words, the most diversified Group also covers the areas of other groups. Here also utilize the Maximum Mcan Discrepancy (MMD) [Borgwardt et al., 2006] to show the difference between "diversified" in our submission and "informative" in Table 4. We measure the discrepancy between the sampled/manipulated outliers with the original ones, termed as MMD[new, original], to indicate the "informative": measure the discrepancy half by half within the sampled/manipulated outliers, termed as MMD/half new, half new], to indicate the "diversified". The results show that DivOE owns lower "informative" due to the partial outlier extrapolation but higher "diversified" the inner dispersion of the outliers.
concept for relative comparison. The intuition
Technically, in our DivOE, "diversified" guides the training to expand the auxiliary outliers to the extrapolated distributions but also consider the original ones. We re-summarized the results
16

---

Table "Diversified" and "Informative" measured by MMD scores.
Dataset
Method
MMD[new, original] originnll (informative)
MMD[half new, half new] (diversified)
OF
0.059 0.001
NTOM
0.032 0.000
0.056 10.000
CIFARIO0
ATOM
0.088 0.001
0.056 0.000
DivOE
0.053 + 0.001
0.113 + 0.002
OE
0,052 + 0.000
NTOM
0.030 + 0.000
0.049 + 0.001
CIFARIO
ATOM
0.285 1 0.005
0.069 1 0.001
DivOE
0.118 = 0.001
0.304 + 0.007
of Figure 4(a) to Table 5 and show that our DivOE with the pursuit of "diversified" (conduct partial extrapolation) performs better than those totally considered "informative" (conduct total extrapolation). The results demonstrate the effectiveness and rationality of DivOE considering partial extrapolation for diversification.
Table 5: Comparison of DivOE pursuing "diversified" and "informative"
Dataset
Method
FPR95
AUROCT
AUPR1
ID-ACC1
OE
27.67
91.89
97.93
75,41
CIFARIO0
DivOE (r=1.0 for "informative")
43.18
88.11
97.11
75,43
DivOR (r=0.5 for "diversified")
24.80
92.91
98.22
75.24
OF
13.76
97.53
99.43
94.51
DivOE (r=1.0 for "informative")
22.05
96.24
99.14
94.81
CIFARIO
DivOE (r=0.5 for "diversified")
11.66
97.82
99,48
94.66
B Theoretical Analysis and Discussion
In this section, we formally present the theoretical implications with detailed definitions and assump- tions. The complete theoretical analysis is based the perspectives of sample complexity and OOD distribution gap the previous works [Ming et al., 2022, Fang et al., 2022]. Different from the existing methods, focus the informative extrapolation proposed in our DivOE for diversifying the given auxiliary outliers towards the ID decision area. Intuitively, we provide an illustration of the conceptual comparison in Figure sampling- based methods which assume that the sampling space of the surrogate OOD distribution be broad enough cover the OOD decision area, DivOE is targeted conduct informative extrapolation to extend the given outlier exposure towards the ID decision area. In the following parts, we first introduce some preliminary setups and notations for the analyses.
explain the differences. Different from the previous
Preliminary setups and notations. Following the prior works [Lee et al., 2018a, Sehwag et al., 2021, Ming et al., 2022], we assume the extracted feature approximately follows Gaussian mixture model (GMM) with the equal class priors as [ TI + oN(-H,, To be specific, Dm N(u,, and D (-p,,, I). Considering the hypothesis class as H sign(0" z), Rd. The classifier outputs if x ~ Din and outputs -1 if x De
First, we introduce the assumption about informative extrapolation on D= out The assumption is empirically supported to be valid by the previous exploration on the informative extrapolation in Figures 2 and 3. where the extrapolated outliers are more informative and close to the decision boundary towards ID samples.
Assumption B.1 (Informative Extrapolation on D Considering the extrapolated version D of the D in our proposed DivOE, we assume that the data points Dr. satisfy the extended constraint based on the boundary scores - [foutiier (x)) defined in POEM Ming et al. [2022]:
Soutleer S (a--7- ((E))n
(18)
17

---

Duut
Dout
Dm
Dout
Din
Dout
derive the following extended lemma based on that adopted in Figure 5: Illustration about the conceptual comparison of the previous assumption auxiliary outliers in OE and that studied in our work. Left panel: the benign assumption on the auxiliary outliers, in which the surrogate OOD distribution Päur can be broad enough cover the informative boundary outliers close to the decision area of ID data (in Din. Right panel: more general assumption on the auxiliary outliers, considering that they can well represent the unseen OOD distribution Dour since it is impractical or even infeasible to accurately pre-define and collect those boundary outliers.
where the foutier is a function parameterized by some unknown ground truth weights and maps the high-dimensional input x into a scalar. Generally, it represents the discrepancy between the De and the true Dont. indicated with the constraint r(c) results from the informative extrapolation. The T(c) represents the extended diversity, which positively related to the extrapolated strength in general.
Given the above assumption, POEM [Ming et al., 2022].
Lemma B.2 (Constraint of Extrapolated D Assume the data points x~ extrapolated De satisfy the following constraint for resulting in the following varied boundary margin: Σi=1 [2x pl S no² (a- T(c)).
proof of Lemma. B.2. Given the Gaussian mixture model described in the previous setup, we can obtain the following expression by Bayes' rule of P(outlier/z).
(ooutiier\z) = [P(sjoutlier)P(uutirr))
(19)
P(x)
1 + 20d (dumtr(at)) (a))))))
where dnuttie (x:) = (x + n²) (ar + n), din d=n(x) = (x - 1)) (x (1), and (P(outlee(xz) Ite Iusinta
according to its definition. Then have:
1
Sautter
20 (duutier (x) din():))
(20)
2
-Isaatier|
1x
Tx
(x 11) - (x + 11) T [xx + 11)1 = G2 Tul.
ml.
(21)
2x
Therefore, get the constraint as: D 12x u1| < no² (a T(e)).
With the previous assumption and lemma, then we present the analysis below.
Complexity analysis on Di With the above lemma and the assumptions of informative extrapo- lation, we can derive the results to understand the benefit of our proposed DivOE.
Consider the given classifier detined 0* 0 ni (E xi Dn xx),, assume each 2 is drawn i..d. from D., and each x² is drawn i.i.d from D and assume the signal/noise ratio is Iell 1, the dimentionality/sanne By decomposition, we can rewrite 8" + n1
size ratio is = r1, as well exist some constant 1.
81 + na 02 with the following 0 and fz:
1 (-
0,
2 4,
02
4.
(22)
m1
112
18

---

Since 01 (0, I), we have that ||0:112 concentration bounds as:
a ny
X3 and
N'(0, e²) form the standard
[alll
o² + ) P(10.11 2
P( 1ut 0nl ||l|| 2 (a]ln|]) S 2e الll
(23)
The distribution of 02 can be treated as a truncated distribution of 0, as x² drawn i.d. from the extrapolated De Without losing the generality, we replace with n, and have the following inequality with a finite positive constant a:
0² (d - - )) S ae R
P(1102112
(24)
n1
According to Lemma B.2, we have that 1 021 v || 2 + o((a) Now can have ||0:11 < - (d )) ||0 v (d+ H), Mall
(a |m|]] simultaneously hold and derive the
following recall the decomposition,
10 10
112
|1
71
01
021 2
0²
(d
1 =) + +1a111
(25)
12 + n2
n1 n2
n
and
0² T()))
| 0 mg
2 2 (11x112 - |b||
(26)
2
With the above incquality derived in Eq. (25) and Eq. (26), we can have the following bound with the probability least 1 (1 a)e
2e "Ille!
||a||2 oF | (a())(())
(27)
o 0110% n1.n22
2 0² (d + :) 1|411
is monotonically decreasing, as the lower bound will increase the constraint from the extrapolated D changed accordingly in our DivOE, the upper bound of FPR(0", R. n2, will decrease. From the above analysis, we can observe the benefit of DivOE when conducting the informative extrapolation.
Since (PR -r
er f( 1² 10 !mg!
a oz
- wy ma
Additional Discussion with Extrapolation Condition. The extrapolation of outlier data towards inliers may always be possible. Similar the condition analysis in [Fang al., 2022], the informative extrapolation that targets better characterize the decision boundary between ID and OOD also needs some requirements or realization assumptions. Intuitively, it mainly relies on auxiliary outliers and specific extrapolation techniques. For the former, the original outliers should not be far away from the ID sample; For the latter, the extrapolation techniques should able add enough perturbation or semantic changes to the original outlier. It is worthy further exploring the condition theoretical way by referring the analysis in Fang et al. [2022].
C Additional Experimental Results and Further Discussion
In this section, we provide more experiment results from different perspectives to characterize our proposed DivOE. First, we introduce the additional experimental setups for the empirical verification in previous figures and our learning framework. Second, provide more analyses of the manipulated data via informative extrapolation. Finally, comprehensive results with discussions are provided.
C.1 Additional Experimental Setups
Figure 1. In the right-panel of Figure 1, we conduct experiments to evaluate the OOD detection performance of different Oa-based methods with the gradually reduced number auxiliary outliers. We first sample particular number of the least informative auxiliary outliers to simulate the different cases in collecting those surrogate OOD data. With the highest OOD scores, we sample the selected auxiliary outliers from the whole auxiliary dataset. Specifically, we choose the negative boundary
19

---

score generated by POEM [Ming et al., 2022] as the OOD score. For the fair comparison, we adjust the size of the mini-batch for OOD samples so that all methods actually train on 90% of the least informative auxiliary dataset by random sampling, or their custom sampling strategies during every epoch while keeping the number of ID training data unchanged.
Figure 2. In the left-middle panel of Figure 2, we evaluate the OOD detection perfor- mance of Energy-OE [Liu et al., 2020] with auxiliary outliers with different level of diver- sity. To simulate the collected auxiliary outliers from different levels of diversity, we first sample the 40000 least informative outliers from Iiny-ImageNet [Le and Yang. 2015] ac- cording to the boundary score generated by POEM [Ming et al., 2022]. Then we choose 10000 informative outliers different but not-overlapped boundary score intervals, namely 10, 10000], [10000, 20000), [30000, 40000), [40000, 50000], [50000, 60000), and combine them with the 40000 least informative outliers to form 5 groups of auxiliary outliers with different levels of diversity. In the two right panels of Figure 2, we present TSNE visualization of feature embedding (taken at the penultimate layer of the network) on ID, the original outliers, and extrapolated outliers after the first training epoch in DivOE, with extrapolated ratio 0.5, extrapolate epsilon 0.05 and optimization step number 5.
Figure 3. In the two left panel of Figure 3, we keep the extrapolated ratio r fixed to 0.5 and optimization step number & fixed to 5. In the right-middle panel, we compare ID/OOD distributions based on Energy score after training with DivOE, with hyperparameters
0.5,e 0.05, and & 5.
Figure In Figure 4(a), the extrapolated epsilon is set to 0.5 and optimization step number is set to 5. In Figure 4(b), we fix the extrapolated ratio r to 0.5 and optimization step number k to 5.
Training details. We conduct all major experiments on pre-trained WedeResNNt IZagoruyko and Komodakis, 2016]with 40 depth and 2 widen factor and fix the number of fine-tuning epochs to 10, following the previous research work [Hendrycks trained using stochastic gradient descent [Kiefer and Wolfowitz, 1952] with Nesterov momentum [Duchi et al., 20111 1]. We adopt Cosine Annealing [Loshchilov and Hutter, 2017] to schedule the learning rate. which begins at 0.001. We set the momentum and weight decay to be 0.9 and 10" respectively for all experiments. The size of the mini-batch is 128 for both ID samples and OOD samples when training and 200 for both when testing.
al., 2019, Liu et al.. 2020J. The models are
of DivOE. Throughout the paper, all the experiments with DivOE are conducted with extrapolated ratio r = 0.5, extrapolate epsilon e = 0.05, and optimization step number k = 5, unless stated otherwise.
C.2 Analyses on the Extrapolated Data
In this part, we further compare the extrapolated outliers with the given auxiliary outliers during the fine-tuning process of OE. In Figure 6, we adopt the TSNE visualization to check the feature representation of the extrapolated outliers during training. Through the comparison with the given outliers (represented by the blue circles), the extrapolated outliers (represented by the orange cricles) manipulated by our introduced target (e.g.. Eq. (6)) can better cover the decision area between ID and OOD data. Note that the extrapolated OOD samples are optimized to be closer to the ID samples (represented by the green circles), which indicates this extrapolated portion can perform better on more accurately shaping the decision boundary of the binary classification task for ID and OOD.
In Figure 8, we present the energy score of the extrapolated outliers compared with the original ones. It can be found that the newly synthesized outliers can generally casier to be recognized as ID samples than the original auxiliary outliers. In Figure 9 and Table 28, we report the loss differences corresponding to the maximization part that verifies operation indeed achieves the conceptual idea of diversified outlier exposure. Intuitively, we visualize the extrapolated outliers in Figure 10. The extrapolated samples still keep the semantic information that belongs to the OOD class space.
In addition, we also conduct another quantitative investigation on the extrapolated outliers with the original ones. In Table 6, we adopt the Maximum Mean Discrepancy (MMD) [Borgwardt et al., 2006] to check the distance between the distributions of the given auxiliary outliers with the extrapolated ones. By varying the specific that is adopted in the multi-step optimization (c.g.. Eq. (9)), we can
20

---

Original ooD
Orginal
Organal con Excapckocs 000
Extropolated 00D
Eurassae 000
(a) Epoch
(b) Epoch
(c) Epoch
(d) Epoch
(e) Epoch 5
Figure 6: TSNE visualization of CIFARIO ID data and the extrapolated outliers generated during OE.
observe that the manipulated outliers with large perform large discrepancy with the original outliers. intuitive and reasonable with more changes on semantic levels as shown in the left panel of Figure 3. However, previously illustrated in the left-middl: panel of Figure 3, also find the larger may not always indicates more informative outliers as the semantic information may be lost when the pixel perturbations arbitrarily large. is worth be further explored in the future.
Table 6: MMD value between original outliers and the extrapolated outliers with different in DivOE
0.01
0.025
0.05
0.075
0.1
0.15
0.5
MMD value
0.0155
0.0176
0.0208
0.0233
0.0253
0.0286
0.0439
C.3
Discussion on Using PGD-based Attack
Note that, PGD is replaceable choice in the framework of DivOE. To better understand the effect of these attack- noise, we compare it with other inner-attack methods and show that PGD with the uniform distribution loss is generally the better way to achieve higher performance.
We would like to explain that DivOE was proposed as a general framework for diversifying outlier exposure with partially conduct informative extrapolation, which allows any appropriate noise or augmentation techniques to incorporated.
show the effectiveness of PGD, we conduct the following experiments. On the one hand, we compare the random noise, re-adversarial perturbation, and adversarial perturbation to justify rationality. The results are summarized in the Table and show the adversarial perturbation achieve better performance than other types of perturbations. On the other hand, we compare with other adversarial attacks and data augmentation methods for inner-maxinn mamiiiaaimiittion part in Table 8 and show that PGD with the uniform distribution loss is generally a better way to achieve higher performance due to the optimization originality.
How would play effect in DivOE As for it is a step size adopted in the informative extrapolation. To better understand its effect, we conducted the experiments by changing the step size in the PGD attack in Table 9, Since the step size is only related to the optimization of the constrained
21

---

Table Rationality of adopting PGD attack in the inner maximization.
Dataset
Method
Perturbation Direction & method
FPR95
AUROC
AUPR
ID-ACC
OE
27.67
91.89
97.93
75.41
DivOE
Re-PGD (most uninformative)
27.36
92.02
97.98
75.62
CIFARI00
DivOE
Random Noise
26.16
92.39
98.06
75.56
DivOE
Mixup
25.70
92.46
98.07
76.06
DivOE PGD (most informative)
24.80
92.91
98.22
75.24
OE
13.76
97.53
99.43
94.51
DivOE
Re-PGD (most uninformative)
13.20
97.59
99.43
94.80
CIFARIO
DivOE
Random Noise
13.19
97.60
99.43
94.81
DivOE Mixup
12.87
97.59
99.42
DivOB
PGD (most informative)
11.66
97.82
99.48
94.66
Table 8: DivOE incorporating different attacked noise
Dataset
Method
Attack Method
FPR95
AUROC1
AUPRT
ID-ACC1
OE
27.67
91.89
97.93
75.41
DivOE
Random
26.16
92.39
98.06
75.56
DivOE
FGSM
25.58
92.60
98.14
75.59
DivOE
PGD
24.80
92.91
98.22
75.24
CIFARIO0
DivOE
C&W
24.20
93.02
98.25
75.18
DivOB
TRADES
24.88
92.85
98.20
75.13
DivOE
Mixup
25.70
92.46
98.07
76.06
DivOE
CutMix
28.46
91.70
97.89
75.62
DivOE
AugMix
25.40
92.60
98.12
75.58
OE
13.76
97.53
99,43
94.51
DivOE
Random
13.19
97.60
99.43
94.81
DivOE
FGSM
12.71
97.70
99.45
94.78
DivOE
PGD
11.66
97.82
99.48
94.66
CIFARIO
DivOE
C&W
11.73
97.74
99.46
94.64
DivOE
TRADES
12.47
97.67
99.44
94.54
DivOE
Mixup
12.87
97.59
99.42
94.83
DivOE
CutMix
13.95
97.54
99.42
94.78
DivOR
AugMix
12.41
97.82
99.49
94.71
maximization, the effects on the final performance of DivOE are not significant. By enlarging the step size, we may save more extra time on the inner maximization to reduce the time cost relatively, which is beneficial for algorithm deployment.
Table The effect of a in training of DivOE
Dataset
Method
Attack Method
Attack Steps
FPR95]
AUROCT
AUPRT
ID-ACCT
OE
27.67
91.89
97.93
75.41
DivOE
PGD (alpha=0.005)
20
25.02
92.69
98.15
75.27
CIFAR-100
DivOE
PGD (alpha=0.01)
10
25.08
92.75
98.17
74.82
DivOE
PGD (alpha=0.02)
5
24.41
92.99
98.23
75.34
DivOE
PGD (alpha=0.04)
3
25.26
92.71
98.17
75.44
OE
13.76
97.53
99.43
94.51
DivOE
PGD (alpha=0.005)
20
11.92
97.78
99.47
94.57
CIFAR-10
DivOE
PGD (alpha=0.01)
10
12.20
97.61
99.42
94.59
DivOE
PGD (alpha=0.02)
12.11
97.80
99.47
94.64
DivOE
PGD (alpha=0.04)
3
12.23
97.76
99.46
94.79
Visual illustration of Extrapolation Process. From the algorithmic perspective, the adversarial perturbation with uniform distribution loss a non-target perturbation. The intuition to perturb the OOD sample to be more like ID regarded by the model. Maximizing uniform distribution loss has no requirement for each OOD sample to be perturbed into some specific ID classes. Thus, is actually automatically decided by the gradient for the specific perturbation directions. To better
22

---

original
0.005
0.01
0.1
0.5
Figure 7: Prediction on the perturbed outlier with different extrapolation strength é.
understand the perturbation effect, we conduct the case study on the specific example in Figure 7 for their specific prediction results during the adversarial perturbation process. It can be find that after the adversarial perturbation, the prediction result of the second example will be concentrate on one specific class with high confidence. By enlarging the perturbation strength, the example will be perturbed to be confidently predicted on different classes.
C.4 Rationality of Partially Extrapolation
Considering the initial motivation to diversify outliers, we explore to extrapolate more potential OOD distribution beyond the original auxiliary ones. For the unperturbed OOD samples used in training, we hope the training can still learn the knowledge from the original auxiliary outliers, for the perturbed OOD samples, we hope the training can find different but informative outliers beyond the original ones to improve the generalization of outlier exposure.
Formally, Eq. (5) is an instance-level realization of Eq. (4) that actually contains the perturbed part and the unperturbed part. The deduction from Eq. (4) to Eq. (5) requires to merge the same terms in Eq. (4) first, and then specifically set with a subsequent rescaling. Finally, we have the formulation that a portion of data to conduct maximization part for diversifying the existing data distribution and the remaining other portion of data corresponds to learning on the original outliers.
To better understand the specific effects of different samples, it can be referred to the left panel of Figure 4, in which we show the OOD detection performance of DivOE under different portions of perturbed data during training. Generally, using small part of perturbed samples can achieve the extrapolation target and improve the performance. However, if we totally perturb all the data, it is even worse than the original OE.
Intuitively, we hope the extrapolation can realize the diversification target and cover more potential OOD distributions beyond the original auxiliary ones. Under such high-level guidance, generally, the extrapolation ratio is expected to be not too large to totally change the whole original auxiliary distribution, which is empirically demonstrated to negative for OE. for diversified strength, we kindly refer to the left middle panel of Figure 3, which serves search space for synthesizing different outliers using different. practical hyper-paramette tuning, we may generally suggest starting with the original OE baseline (i.e., extrapolation ratio equals O and diversified strength equals 0) and enlarging the extrapolation ratio or diversified strength step by step, finding a stage where the OOD detection performance the validation set will not be further enhanced. For the goarse-grained augmentations and specific OOD scores, may be further decided by other.
For four extrapolation factors, we re- summaaare the previous results and discuss the general guidance for tuning. First, MSP score and PGD are empirically demonstrated be the appropriate choice for extrapolation, which can be fixed (refer to the results in Figure 4(c)). Second, extrapolation ratio and strength (i.e., e) have general trends either individually or jointly considering them Table 10. The performance of the extrapolation ratio is monotonous regarding different e. The large values would degrade performance due to the loss of diversity on the auxiliary outliers. Then, the extrapolation strength could be tuned after the other factors are set to appropriate choices. Our empirical results
23

---

show that very small or large should be avoided, and epsilons from 0.05 to 0.1 can generally achieve satisfactory results by referring previous experimental results.
Table DivOE with different extrapolation ratiosand strengths.
Dataset
Method
Extrapolation ratio
Extrapolation strength
FPR95.
OE
27.67
DivOE
0.01
30,52
DivOE
0.5
0.01
29.06
DivOE
0.25
0.01
26.21
DivOE
0.1
0.01
24.57
DivOE
1
0.05
43.21
CIFAR-100
DivOE
0.5
0.05
24.80
DivOE
0,25
0.05
24.21
DivOE
0.1
0.05
23.54
DivOE
1
0.125
49.69
DivOE
0.5
0,125
24.94
DivOE
0.25
0.125
25.36
DivOE
0.125
24.77
C.5 Extra Experimental Results and Discussions
In this part, we present the extra experimental results in the following perspectives to characterize the learning framework of DivOE for the OOD detection task.
Experiments on Add-on Modules. For adding other methods to Table 2, we conduct experiments accordingly and summarize the results in Table 11. The results show that DivOE consistently improve them on the original basis compared with other counterparts across different setups.
Table 11: Extra comparison with various add-on methods.
Datasct
Method
FPR95
AUROC1
AUPRT
ID-ACC1
OE
27.67
91.89
97.93
75.41
DOE
30.69
93.13
98.39
71.43
MixOE
35.26
90.31
97.58
75.74
DivOE
24.80
92.91
98.22
75.24
C1FAR-100
Energy
27.86
90.70
97.40
75.41
Energy+DOE
44.91
88.80
97.22
73.79
Energy+MixOF
38.14
85.12
95.40
73.87
Energy+DivOE
24.71
92.04
97.82
75.25
OE
13.76
97.53
99.43
94.51
DOE
7.39
98.29
99.58
92.82
MixOE
13.57
97.47
99.39
94.66
DivOE
11.66
97.82
99.48
94.66
CIFAR-10
Energy
18.15
90.90
96.34
94.02
Energy+DOF
9.65
97.70
99.37
94.83
Energy+MixOE
31.90
83.72
93.50
93.72
Energy+DivOE
16.03
92.23
96.87
94.53
Experiments on Fine-grained OOD Test Sets. In order further understand the effectiveness of our DivOE on different OOD datasets, we report the fine-grained results of our experiments on CIFAR-IO/CIFAL 100 with OOD datasets (Textures, Places365, iSUN, LSUN-r, LSUN-c). The results the 5 OOD datasets demonstrate the general effectiveness of our DivOE.
In Tables 12 and 13, we summarize the fine-grained detection performance. The results reflect the performance improvement of each method compared with those representative baselines. Our DivOE can significantly reduce the FPR95 score on some OOD testing sets which those previous baselines do not perform well (e.g., Textures and Places365), and achieve comparable performance on the rest ones. Note that general to have some tradeoffs [Hendrycks et al., 2019, Chen al., 2021, Ming et al., 2022, Fang et al., 2022] on the performance change when using different methods under the OE framework. DivOE shows a good balance gaining significant improvement without sacrificing the other performance too much via the informative extrapolation. On the other hand, the significant

---

Table 12: OOD detection performance CIFAR-10 [Krizhevsky, 2009] ID. All methods trained on the same backbone. Values are percentages. 1 indicates larger values are better, and indicates smaller values are better. (averaged by multiple trials)
Method
Textures
Places365
ISUN
LSUN-r
LSUN-c
ID ACC
PPR95,
AUROO-
FPR9S]
AUROCT
FPR95,
AUROCT
FPR95].
AUROCT
HPR95_
AUROCT
89.53
ODIN
54.84
80.22
48.89
85 66
28.68
93.85
23.17
95.05
9.96
97.90
94.84
Mahalanobis
16.99
96.69
69.90
82.54
31,62
94.44
12.41
94.70
30.50
95.06
94.84
Energy
GrudNorm
73.17
58.11
78.27
60.45
70.28
70.82
65.63
73.23
11.41
97.02
94.84
OF
21.81
96.15
44.86
91.77
0.90
100.01
0.00
100.90
0.99
99.85
94.56
Energy (w:
33.59
85.47
58.39
68.32
0.90
100.00
0.00
100.00
0.31
99.83
94.50
55.45
81.72
4.21
99.13
33.33
92.47
90.82
NTOM
32.78
$6.30
$9.03
67.50
0.00
100.00
0.00
100.90
1.17
99.18
95.06
POEM
34.73
85,28
57.82
67.93
0.00
100.00
0.00
100.00
0.43
99.74
95.10
92.88
0.26
DivOR (Ours)
12:28
97.73
44.97
91.51
0.07
99.98
0.02
99.99
3.03
99.47
94.56
Table 13: OOD detection performance on CIFAR-100 [Krizhevsky, 2009] as ID. All methods are trained on the same backbone. Values are percentages. i indicates larger values are better, and indicates smaller values are better. (averaged by multiple trials)
OOD dataset
Method
Places365
LSUN-r
LSUN-c
ACC
FPR95,
AUROCT
FPR95]
AUROC1
FPR95,
AUROC:
FPR951
AUROC1
FPR95,,
AUROCT
MSP
$2.90
73.53
82.31
74.08
83.28
75.10
83.06
74.08
66.41
83.85
75.96
93.31
Mahalanobis
38.99
90.64
88.58
67.92
25.33
94.57
20.89
96.05
91.26
69.73
75.96
Energy
79.70
76.34
80.48
75.78
82.44
77.87
81.01
77.67
35.69
93.43
75.96
GradNorm
54.11
$4.69
77.42
76.27
0.07
99,99
0.04
100.00
4.58
99.06
75.50
Energy (w. Daxc)
56.93
81.22
80.53
72.76
0.00
100.00
0.00
100.00
0.52
99.88
75.14
VOS
78.50
74,02
82.21
73.40
85.24
75.86
83.59
75.58
25.65
94.88
7494
POEM
51.72
84.16
79.94
72.84
0.00
100.00
0.00
100.00
0.34
99.92
75.88
MixOE
61.78
82.13
72.95
78.20
2,96
99.32
1.05
99.75
33,56
92.88
75.77
(Ours)
99.94
99.99
performance improvement of DivOE on the Textures and Places365 (that OOD test sets that the original OE does not perform well) exactly verifies that the informatively extrapolated outliers better represent the unseen OOD inputs that are close to the ID decision area.
Experiments on Sample Comparison may be hard to align the sample number as different methods have different synthesis motivations and but samples. To facilitate the comparison, we summarize the specific outlier numbers that were manipu- lated in the adopted algorithm with its performance in OOD detection in Table 14. The results show that, compared with previous methods, either need go through all the outliers for sampling or change small portion of
actually present the number of generated
the training all the auxiliary outliers, DivOE conduct extrapolation original outliers but achieve generally better performance across different ID classification tasks.
Table 14: Numbers of outliers manipulated sampled during training
Dataset
Method
Numbers
FPR95
AUROCT
AUPR1
ID-ACC1
OF
0
27.67
91.89
97.93
75.41
NTOM
50000
27.58
91.08
97.52
75.41
CIFAR100
POEM
50000
26.50
91.33
97.58
75.74
MixOE
50000
35.26
90.31
97.58
75.74
DivOE
25000
24.80
92.91
98.22
75.24
OE
13.76
97.53
99.43
94.51
NTOM
50000
17.76
91.08
96.40
95.04
CIFAR100
POEM
50000
17.81
90.96
96.36
94.90
MixOE
50000
13.57
97.47
99.39
94.66
DivOE
25000
11.66
97.82
99.48
94.66
25

---

Experiments on Smaller Number of Auxiliary Samples. Here, we added the experiments under different numbers auxiliary outliers used for all OE-based methods. The experimental results are summarized in Table 15, which indicates that the performance of OOD detection can be generally better we use more outliers. Under smaller number of samples, i.e., from 50000 1000 (relatively smaller than 50000 ID data), DivOE can consistently perform better than OF.
Table 15: Performance comparison by changing the numbers of auxiliary outliers.
Dataset
Method
Number (ratio to ID samples)
FPR95
AUROC
AUPR
ID-ACC
OE
26.72
91.12
97.51
75.58
NTOM
29.18
90.72
97.45
75.77
POEM
50000 (1.0)
27.20
90.79
97.41
75.77
MixOE
35.28
90.29
97.57
75.67
DivOE
26.27
92.71
98.18
75.34
OE
33.64
89.98
97.36
75.92
CIFART00
NTOM
33.23
88.92
96.86
75.87
POEM
5000 (0.1)
33.27
88.59
96.73
75.64
MixOE
37.32
89.39
97.31
76.09
DivOE
32.80
90.89
97.71
75.62
38.40
88.93
97.17
76.14
NTOM
38.11
89.12
97.45
76.12
POEM
1000 (0.02)
37.69
89.47
97.56
75.98
MixOE
40.78
87.98
96.89
76.10
DivOE
36.74
89.63
97.36
76.08
OE
12.64
97.76
99.47
94.85
NTOM
19.84
90.12
96.23
94.80
POEM
50000 (1.0)
20.21
90.15
96.10
94.97
MixOE
14.26
97.31
99.35
94.92
DivOE
12.04
97.76
99.47
94.80
OE
15.16
97.31
99.35
94.95
NTOM
28.91
84.26
93.40
94.63
CIFAR1O
POEM
5000 (0.1)
27.46
86.56
94.68
94.78
MixOE
97.06
99.29
94.98
DivOE
14.41
97.43
99.40
94.81
OE
16.74
97.11
99.31
95.03
NTOM
31.90
81.93
91.99
94.86
POEM
1000 (0.02)
30.67
82.56
92.33
94.78
MixOE
15.80
97.07
99.29
95.11
DivOE
15.38
97.25
99.37
95.03
Experiments on Comprehensive Explorations of the Extrapolated Data In Tables 20 and 21, we present more comprehensive comparison of the extrapolated outliers under the learning framework of our DivOE. To be more specific, we control the specific extrapolation ratio r and the extrapolated strength e characterize the algorithm. From one perspective, we gradually increase the extrapolated strength with the fixed extrapolation ratio r = 1, which means we conduct the full-batch manipula- tion on the original surrogate OOD distribution. The results demonstrate the failure of such a overall distribution perturbation with the significant performance drop on several OOD test sets (e.g., I.UN-r and LSUN-c). Intuitively, the overall distribution manipulation for the auxiliary outliers can not perform well towards the diversification target, as it is also inconsistent with the conceptual idea of our DivOF. From the other perspective, we decrease the extrapolation ratio with the enlarged strength on informative extrapolation. Fortunately, with the diversified guidance introduced by proposed objective DivOE, we can find that even using e = 0.125 can also enhance the performance of outlier exposure with different OOD test sets, resulting in a better performance in average.
Experiments on Using Different Synthetic Mechanism Here, we consider some advanced tech- niques using Generative Adversarial Network (GAN) to synthesizing new outliers for outlier exposure. Those methods [Lee et al., 2018a, Kong and Ramanan, 2021] additionally train generative models for generating extra outliers for data augmentation, which depends on the sample quality and actually are orthogonal to our extrapolation. We conduct experiments comparing and incorporating the method [Lee et al., 201 8a] for informative extrapolation in Table 12 in attached PDF. We find that the
26

---

Table 16: Fine- Fine-grained OOD detection performance of DivOE CIFAR-10 [Krizhevsky, 2009] as
ID.
Method
Averago
21.88
100.00
1.02
0.01
15.32
102.00
14.53
102.00
12.25
DiOE
101100
102.00
12.92
102.00
-0005
102.00
-0005
19.42
102.00
13.40
DiOEE -0.6.-
99.68
22.05
DiOE -0.5)
91.65
99.5%
Di.E -0.25)
12.29
99.59
99.99
11.57
13.45
91.73
Di.EE: -0125
89.24
14.15
91.79
100,00
1.85
12.05
Di-EE:
15.92
92.08
102,00
1.39
12.21
Di-E
91.82
102000
12.42
DiOEE 0.03
12.28
Table 17: Fine- tine-grained OOD detection performance of DivOE on CIFAR-100 as ID.
OUD dataset
Method
Testures
Places365
ISUN
LSUN-r
LSUN-c
FPR95
FFRUSL
AUROC-
FPR
AUROCT
FPR95]
AURROCT
FPROS]
AUROCT
FPRUS)
AUROCT
OE
$4.31
$4.59
77.18
76.30
03.09
29.99
0103
100.00
4.84
99.04
27.29
Di.E 0.01
36,11
50.30
77.73
76.50
0.:66
99.54
11.02
99.59
38.55
90.11
30.52
DiOE ( 0.01. 0.5)
36,30
50.34
76.79
76.17
02:14
29.96
0101
99.99
30.08
93.72
29.06
DaOE ( 0.01. 0.251
37.70
$9.65
75.04
76.51
0257
59.50
0100
102.00
15.26
96.79
26.21
DaOE ( 0.0.* 0.13
38,89
$9.23
75.94
17.08
0.08
99.58
0,00
102.00
7.92
95.22
24.57
DaOE ( 0003, 17
41.33
$7.64
77.51
16.54
0.02
99.99
0.00
102.00
5.72
95,55
24.95
DoOE ( 0003, 0.30
40.33
$1.00
78.2
16.15
0.05
99.98
0.00
100.00
9.38
07.97
25, 39
DiOE 00005, I1.11
44.13
56.83
75.66
76.
0.06
29.58
11.00
102,00
4.53
98.39
24.94
DaOE ( 0.00.
62.51
$2.89
$1.42
75.20
7.43
98.23
3,69
99.17
60.99
85 85.11
43.21
D&OE ( 0.00. 0.50
33.56
90.75
75.14
17.23
0.20
99.4
0.03
99.99
12.50
97.17
24.29
DaOE ( 0.0. 0.25)
38.29
90.20
74.44
77.50
0.09
99.97
0.01
99.99
11.25
97.48
24.21
DaOE ( 0.8. 0.13
$4.33
90.37
73.18
77.92
0.10
99.97
0,00
100.00
10.08
97.71
23.56
DoOE (r 0.125
71.13
$0.04
$1.22
14.76
19.69
55.29
13.69
95.79
02.35
64.94
49.69
DaOE (1 0.125 ILS1
41.54
59.72
73.64
11.43
02.88
99.56
B103
192.00
9.31
97.97
24.94
D&OE ( 0.125 (125)
45.16
$7.92
74.25
77.66
0.10
99.98
0,00
192.00
7.29
95.45
25.36
DaOE (t 0.125 0..1
43.73
$8.11
73.94
77.77
0..99
99.98
0,00
100,00
6.07
98.70
24.77
DiOE T 0.01 0.125)
31,67
91.12
76.19
17.08
0.17
99.05
0102
99.99
24.67
94.75
26.54
GAN-synthesized method hard train under our setups. When we incorporate DivOE into the generated sample, the performance can better enhanced than the original one of GAN-based method.
Experiments Computational Cost To better understand the computational budget, conduct additional experiments and summarize the time and memory results in Table 18. which shows that DivOR can achieve better performance with comparable time and memory costs with other OE-based methods. In addition, since the extra time requirement compared with the original OE is mainly from the mmermmaxxiizztion of the informative extrapolation, noise on the OOD detection performance in Table 19. The results show that we can further save extra time by reducing the outlier manipulation times with maintaining the improvement.
also explore the effect of perturbation
Table 18: Time and memory cost of different methods.
Datasct
Method
Time for one iteration (s)
GPU Memory (MiB)
FPR95
AUROC
AUPR
ID-ACC
OE NTOM
0.22
3769
27.67
91.89
97.93
75.41
0.39
3775
27.58
91.08
97.52
75.41
CIFAR100
POEM
0.79
3845
26.50
91.33
97.58
75.74
MixOE
3769
35.26
90.31
97.58
75.74
DivOE
3781
24.80
98.22
75.24
OE
0.21
3769
13.76
97.53
99.43
94.51
NTOM
0.34
3733
17.76
91.08
96.40
95.04
CIFARIO
POEM
0.66
3835
17.81
90,96
96.36
94,90
MixOF
0.47
3731
13.57
97.47
99.39
94.66
3779
11.66
97.82
99.48
94.66
27

---

Table 19: Perturb number and average time the extrapolation in DivOE.
Dataset
Method Perturb Number
Average Time for One lteration (s)
FPR95
AUROC
AUPR
ID-ACC
OE DivOE
0.22
27.67
91.89
97.93
75.41
0.41
25.58
92.60
98.14
73.59
C0FAR100
DivOR DivOB
0.47
24.94
92.74
98.17
75.18
0.57
75.24
OE DivOE
0.21
13.76
97.53
99.43
94,51
0.38
12.71
97.70
99.45
94.78
CIFARIO
DivOE
0.44
12.10
97.68
99.45
95.59
DivOE
0.53
11,66
97.82
99.48
94.66
Experiments on Compatibility with other OE-based Methods Tables 20 and 21, we provide the fine-grained performance OOD detection of those OE-based methods [Chen et al., 2021, Ming et al., 2022, Wang et al., 2023] with our DivOE. The overall results again confirm the effectiveness of the previously proposed OE-based methods incorporating the informative extrapolation introduced by our DivOE. Especially, DivOE via the informative extrapolation helps these methods better represent the unseen OOD inputs close to the ID decision area. The overall averaged results can refer to Table 2.
Table 20: Fine-grained results of compatibility experiments with other OE-based methods on CI- FARIO [Krizhevsky, 2009] as ID.
OOD dataset
Method
Textures
Place365
ISUN
LSUN-r
LSUN-c
Average FPR95
AUROCT
EPRO5_
FPRO5
AUROC1
RPRUS]
21,88
45.22
0.00
100.00
100.00
1,02
NTOM
32.78
86.
59.11
67:50
0.00
100.00
0.00
100.00
1.17
99:33
18.50
NTOM-DivOE
14.49
94.01
53.88
72.31
0.00
100.00
0.00
100.00
5.29
97.53
14.73
POEM
34,73
85.28
57.82
67.93
0.00
100.00
100.00
0.45
99.74
18.59
MiXOE
22.23
95.74
39.15
92.86
0.26
99.92
0.27
99.92
4,40
99.21
13.32
MixOE-DivOE
18.63
96.34
42.55
91.68
0.54
99.86
0.40
99 86
9.77
98.34
14.38
DOE
21.97
94.87
15.43
96.74
0.30
99.89
0.22
99.
1.27
99.67
7.34
DOEsDIVOR
16.28
0.25
Table 21: Fine-grained results of compatibility experiments with other OE-based methods on CI- FARI00 [Krizhevsky, 2009] as ID.
OOD dataset
Method
Textures
Places65
ISUN
LSUN-r
LSUN-c
Average FPR95
FPRYS
AUROC!
EPRO5
AUROC1
FPR95
ALROC!
FPR95]
AUROCT
FPRUS]
AUROCT
72.46
100.00
NTOM-DivOE
41.65
88.21
79:45
72.80
0.00
100.00
0.00
100.00
5.04
98.54
25.23
POEM
51.72
$4.16
79.94
72.84
0.00
100.00
0.00
100.00
0.34
99.92
26.40
DOEM-DiVOE
43.61
87.13
30.09
72.52
0.00
100.00
100.00
1.69
99.55
25.06
MIXOE
61.78
72.95
78.20
99.32
99.75
33.56
92.85
34.46
DOE
59.92
83.93
40.50
91.53
26.37
95.11
18.35
96.59
12.90
97.72
31.61
DOE+DivOE
13.77
89.24
10.18
91,44
0.75
99.71
0.17
99.81
17.16
97.01
20.16
Experiments with Limited Informative Auxiliary Outliers In Table 22, present the detailed performance results of the experiments with limited informative auxiliary outliers in CIFAR-10. The experiments are designed to demonstrate the consistent effectiveness our proposed DivOE with informative extrapolation. Different from the previous sampling-basee based methods (c.g.. NTOM [Chen al., 2021] and POEM [Ming et al., 2022]), which assume that the potential surrogate OOD space can be arbitrarily large and cover the boundary decision area for ID and OOD inputs. our DivOE introduces the diversification target which is more general and robust given different auxiliary outliers. With such a general learning target, DivOE can perform learning and extrapolation given surrogate OOD distribution iteratively under a single learning framework during the fine-tuning process. Empirically, it results in better performance improvement across the different setups on the auxiliary outliers in Table 20. Since the NTOM and POEM both adopt the sampling strategy sample part of outliers (90%), we keep the same number in OE and our DivOB for the fair comparisons.
28

---

Table 22: Experiments limited informative auxiliary outliers CIFAR1O 10.
Method
Number of auxiliary outliers
FPR95
50000
19.87
40000
19.86
OE
30000
21.90
10000
26.19
5000
31.28
50000
19.90
40000
22.29
NTOM
30000
24.12
10000
25.89
5000
29.03
50000
20.30
40000
21.39
POEM
30000
21.77
10000
28.10
5000
27.42
50000
15.95
40000
15.75
DivOE
30000
16,82
10000
17.52
5000
20.96
Experiments on Extrapolation via Data Augmentation In Tables 24, 25, 26 and 27, we replace the detailed realization of the maximmaxxmizttion of the informative extrapolation with the conven- tional data augmentation method, i.e., Mixup [Zhang et al., 2018]. and also explore the effectiveness of combining the CutMix [Yun et al., 2019] in the learning framework. The results also confirm that the high-level idea of our DivOR is general to adopt different data manipulation or synthetic strategies achieve the diversification target. The averaged and fine-grained performance results demonstrate the effectiveness of the proposed DivOE, which diversifies the given outlier for effective OOD detection. In addition, we also conduct experiments and combine them with the inner-maximization function. Compared with the pixel level manipulation, more coarse-grained data augmentation can also boost the performance if can be conducted in an optimization way. the original outliers are far away from the ID samples, data augmentation
as the basis for inner- maxemmaaxxizi...n....
Table 23: DivOE with different inner extrapolation methods.
Dataset
Method
Extrapolation Method
FPR95
AUROC
AUPR
ID-ACC
OE
27.67
91.89
97.93
75.41
GyN-synthesized [c]
87.09
66.70
90.40
31.73
CIFAR100
DivOE
PGD
24.80
92.91
98.22
75.24
DivOE
Mixup
25.70
92.46
98.07
76.06
DivOE
GAN-synthesized [c]
69.69
81.82
95.16
29.11
OE
13.76
97.53
99.43
94,51
GAN-synthesized [c]
78.75
72.97
91.88
50.67
CIFARIO
DivOE
PGD
11.66
97.82
99.48
94.66
DivOE
Mixup
12.87
97.59
99.42
94.83
DivOE
CAN-syythesized [c]
61.34
80.14
93.23
61.23
D Broader Impact
OOD detection is an important aspect of deploying reliable deep learning systems in the real world INguyen al., 2015, Hendryeks al., 2022]. It is especially important for those safety-critical applications [Bommasani et al., 2021] like financial medical intelligence, in which reliable
29

---

Table 24: The Augmentation Method
combined with different augmentation methods.
Dataset
Method
FPR95
AUROC
AUPR
ID-ACC
OE
27.67
91.89
97.93
75.41
DivOE
Random
26. 16
92.39
98.06
75.56
DivOE
PGD
24.80
92.91
98.22
75.24
DivOE
Mixup
25.70
92.46
98.07
76.06
CI
DivOE
CutMix
28.46
91.70
97.89
75.62
DivOE
AugMix
25.40
92.60
98.12
75.58
DivOE
Mixup+PGD
25.14
92.79
98.18
75.78
DivOE
CutMix+PGD
26.15
92.37
98.06
75.51
DivOE
AugMix+PGD
24.92
93.01
98.24
75.48
OE
13.76
97.53
99.43
94,51
DivOE
Random
13.19
97.60
99,43
94.81
DivOE
PGD
11.66
97.82
99.48
94.66
DivOE
Mixup
12.87
97.59
99.42
94.83
CIFARIO
DivOE
CutMix
13.95
97.54
99.42
94.78
DivOE
AugMix
12.41
97.82
99.49
94.71
DivOE
Mixup+PGD
12.71
97.70
99.47
94.68
DivOE
CutMix+PGD
13.12
97.72
99.47
94.74
DivOE
AugMix+PGD
11.49
97.87
99.49
94.71
model should have the capability distinguish those samples having different label spaces (e.g. animals) instead of giving a prediction based on existing classes (e.g., finance products or disease). Our research work studies general and practical research problem in outlier exposure for effective OOD detection, considering the limited informative auxiliary outliers that can not well represent the boundary outliers for the pre-trained model on the classification tasks with ID samples. is important for effective OOD detection to gain better empirical performance through learning from the auxiliary outliers, and also conceptually figure out the critical problem in this general learning framework.
Although we take a step forward in fine- mui-ttning with auxiliary outliers for effective OOD detection, it is not the end of this direction since there are still many practical problems to be addressed. For example, although learning with the surrogate OOD data can boost the performance of the given model in identifying the OOD inputs, fine-tuning requires extra training time and computational cost, which may bring extra resources for adjusting. Same to other ba-ba methods, how to reduce the extra training cost for the model regularization is also future direction that is worth exploring.
0,45
pm1 25
25
EC
535
0.35
0.26
132
2.0
0.30
0.30
511
511
0.10
0.10
115
225
00b
000
100g
0.00
0.00
Energy Score
Energy Score
Entrgy Soore
Energy Soom
Figure 8: Verification on the effect of extrapolation via Energy Score. The density of the energy score distributions the e = 0.5 extrapolated outliers under different numbers of outlier samples, 100000 for left panel, 50000 for left-middle panel, 10000 for right-middle panel, 1000 for right panel.
30

---

Table 25: Experiments of DivOE via Mixup (%). (averaged by five OOD test sets)
Di
Method
FPR95
AUROC1
AUPR
ID-ACC1
Energy (w. Daux)
18.69
90.51
96.15
94.59
Mixup (w. Dax)
14.82
92.72
96.96
94.84
Energy+DivOEE (Mixup) Daa))
15.02
92.35
96.78
94.98
Energy+CCttMix (w. D
19.28
90.30
96.04
94.90
NTOM (with MSP)
12.96
97.73
99.46
95.11
NTOM+Mixup
12.25
97.85
99.50
95,04
NTOM+DivOE (Mixup)
12.20
97.81
99.48
94.98
POEM
18.32
90.81
96.27
94.70
CIFAR-10
POEM+Mixup
14,70
92.76
97.00
94.98
POEM+DivOE (Mixup)
15.39
91.89
96.62
94.77
POEM+CutМix
17.70
91.25
96.49
94.88
OE
13.62
97.52
99.41
94.57
OE+Mixup
13.08
97.68
99.46
OE+DivOE (Mixup)
12.87
97.59
99.42
94.83
OE+CutMix
14.74
97.45
99.41
94.93
Energy (w. Daux)
27.49
90.72
97.39
75,23
Energy+Mixup (w. Dax)
25.27
92.02
97.81
75.49
Energy+DivOE (Mixup) a
24.47
92.04
97.79
75.53
Emergy+CuttMix (w. Dan))
29.87
89.72
97.13
75.59
NTOM (with MSP)
26.13
92.32
98.07
75.54
NTOM+Mixup
25.41
92.75
98.18
75,60
NTOM+DivOE (Mixup)
24.44
92.73
98.15
75,84
POEM
25.57
91.73
97.69
76.01
CIFAR-100
POEM+Mixup
24.46
92.19
97.84
75.57
POEM+DivOE (Mixup)
24.14
92.16
97.81
75.57
POEM+CuMix
28.23
90.75
97.41
75.90
OE
27.29
91.98
97.96
75.51
ME+Mixup
27.20
92.22
98.03
75.87
OE+DivOE (Mixup)
25.70
92.46
98.07
76.06
OE+CutMix
31.14
90.98
97.70
75.66
Table 26: OOD detection performance on CIFAR-10 [Krizhevsky, 2009] ID. All methods trained on the same backbone. Values are percentages. 1 indicates larger values are better, and indicates smaller values are better. (averaged by multiple trials)
Places365
I.SUN-r
T.SUN-c
FPRO5]
FPRO5)
Energy Pax)
Encgy>Mixup Pac)
Faergy-Divtl (Mispp) (w. (P.))
NTOM&eith MSP)
NTGM+Misup
42.13
100.00
0.00
100.00
1.12
09.70
95.04
NEOM (Mixup)
18.12
90.76
42.16
92.40
0.00
100.00
0.80
100.00
0.71
59.89
94.9%
POFM
34.52
54.54
$7.00
60 55
0.00
100.00
0.80
100.00
0.09
99 06
94.70
POEM-Mixup
21.19
89.99
43.59
75.45
0.00
100.00
0.00
100.00
3.70
95. 35
94.9%
POEM-DiVOE (Misup)
22.36
89.03
$4.29
70.61
0.00
100.00
0.00
100.00
0.32
99.83
94.77
UE
21.88
96.14
45.22
91.64
0.000
100.00
0.00
100.00
1.02
99.84
94.57
OF-Mixup
19.01
96.68
44.70
91.97
0.00
100.00
0.03
100.00
1.67
90 74
94.89
OF4DNOF (Mixup)
19.92
96.40
43.59
91
0.00
100.00
0.00
100.00
0.82
99.86
94.83
31

---

Table 27: OOD detection performance CIFAR-100 [Krizhevsky, 2009] as ID, All methods trained on the same backbone. Values are percentages. 1 indicates larger values are better, and indicates smaller values are better. (averaged by multiple trials)
000
Method
Testures
L.UUN-
TSUNN-
AURROC
FPRO05)
AUTROC
Energy Pan)
75.33
Mixup Pac)
mmcxy-Divcl (Mixep) (w: (Pa))
NTOM/with MSP)
NTOM+Misup
100.00
10.53
75.60
OI (Mixup)
44.37
57.21
74.32
77.20
0.02
99.99
0.01
100.00
3.50
99.23
73.84
PORM
51.55
53.56
76.23
75.10
0.00
100.00
0.80
100.00
0.09
99.98
76.01
POEM-Maxup
39:70
53.09
77.06
74.39
8.00
100.00
0.00
100.00
5.53
95.47
73.57
POEM-DIvOE (Mixup)
43.30
$6.40
77.12
74.43
0.00
100.00
0.00
100.00
0.28
99.90
75.57
OE
54.31
$4.59
77.18
76.30
0.09
99.95
0.03
100.00
4.84
99.
75.51
OF-Mixup
43.11
86.13
74.27
77.66
0.22
99.93
0.09
99.99
14.30
97 00
75.87
0E4D6OO (Mixup)
48.35
86.07
75.04
77.33
0.13
99.37
0.02
100.00
4.97
95.94
76.06
24
e
Loss difference
0.01
10.84
Loss Difference
0.025
16.64
18
0.05
18.81
0.075
18.64
0.1
18.50
0.15
19.03
0.2
20.29
8
0.01
0.075 0.1
0.15
0.2
0.5
24.16
Figure 9: OOD loss difference (i.e., the max- imization part in Eq. (4)) between original outliers and the extrapolated outliers with dif- ferent in DivOE.
Table 28: OOD loss difference (i.e., the imization part in Eq. (4)) between original outliers and the extrapolated outliers with dif- ferent in DivOE.
32

---

original
E 0.025
E 0.05
0.075
=
= 0.15
=
= 0.5
Figure 10: Examples of the extrapolated auxiliary outliers by controlling the e in Eq. (10) from 0 (i.e., original) to 0.5. By different levels of pixel manipulation, the extrapolated samples can cover more potential decision areas for OOD inputs.
33


Reviews:
[{"review_id": "H7Wjerir31", "reviewer": "Reviewer_aWLd", "summary": "This paper improves the methods of utilizing auxiliary outlier data for fine-tuning. Specifically, it synthesizes informative outlier samples close to in-distribution at the decision boundary by adding noise to existing auxiliary data and utilizes them in learning. As a result, it shows improved results by applying the proposed method to various outlier exposure methodologies.", "questions": "1. What is the difference from the most important outlier synthesis methodologies (e.g., MixOE [Zhang et al., 2023] and DOE [Wang et al., 2023]).\n2. The fact that the outlier close to the boundary in the left figure of Figure 2 is a diversified outlier is not clearly explained. It is necessary to explain how the informative and diversified are different.\n3.\tThe process of moving from Equation 4 to Equation 5 is not clear. Additional detailed explanations are needed.\n4. Equation 4 synthesizes all outlier data and leverages only loss, while Equation 5 synthesizes and calculates loss for a portion of outlier data. It seems that the two equations are different methods. Please explain how the two equations are connected.\n5. It is necessary to discuss direct comparison and difference with DOE in TABLE1.\n6. Please add the comparison results of each when combined with DOE and MixOE in TABLE2.\n7.\tPlease add experiments on the ImageNet benchmark [A], as discussed in DOE.\n8.\tPlease add experiments on the application of DivOE to OECC [B] in TABLE2.\n\n[A] Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik. Imagenet-21k pretraining for the masses. In NeurIPS Datasets and Benchmarks, 2021.\n\n[B] Papadopoulos, Aristotelis-Angelos, et al. Outlier exposure with confidence control for out-of-distribution detection. Neurocomputing, 2021, 441: 138-150.\n", "limitations": "Yes. The author adequately addressed the limitations", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper's method is simple and clearly explained. This paper's approach is novel in explicitly adding noise to outlier data to synthesize and utilize outlier data. \n2. Theoretically, the proposed method looks a reasonable way to select samples closer to in-distribution data at the decision boundary than existing methods. \n3. The experiments are generally fair and show performance improvements when this method is applied as an add-on to a variety of methods. In particular, this paper suggests a method of synthesizing and utilizing outlier data, which presents the possibility of improving the performance of fine-tuning using outlier data. \n4. Additionally, this paper shows that this method is an effective way to improve the performance of fine-tuning using outlier data by applying it to a variety of outlier exposure methodologies and showing improved results.", "weaknesses": "1. The novelty and superiority in outlier synthesis against DOE [Wang et al., 2023] are not clear against DOE [Wang et al., 2023]. \n2. The term \"diversified\" is not well-defined, and it is not clear how it differs from the term \"informative\" used in ATOM [Chen et al., 2021]. \n3. The experiments also lack comparison and discussion with the most similar papers MixOE [Zhang et al., 2023] and DOE [Wang et al., 2023] that use outlier synthesis. For example, there is no comparison between the proposed method and DOE [Wang et al., 2023] when it is applied as an add-on to a variety of existing methods. \n4. The rationale for using PGD (Projected Gradient Descent) based noise is not well-explained. It is not clear if it has superiority over other attack-based noise. \n5. It is not clear how the proposed method of explicitly synthesizing outlier samples differs from implicitly synthesizing them in terms of new effects or novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n\n> **W1, Q1, W5, W3, Q5:** about DivOE, MixOE, DOE\n\nThank you for the special recommendation of MixOE and DOE for clarity . We would like to explain their differences as follows.\n\nConceptually, DivOE has a **different underlying motivation** from MixOE and DOE. Our DivOE is proposed for diversifying outlier exposure by extrapolating auxiliary outliers. In comparison, MixOE focuses on enhancing the fine-grained OOD detection by interpolation between ID samples and OOD samples; DOE focuses on improving the generalization of the original outlier exposure* by exploring model-level perturbation. \n\nTechnically, DivOE has a **different algorithmic design** from the others. Specifically, DivOE was proposed as a general framework for outlier extrapolation, which bases on OOD samples. It is different from MixOE which directly mix-up ID and OOD samples; It is also different from **DOE that constructs the model perturbation for better optimization**. It is worthy noting **our DivOE is from a different perspective (data-level) compared with DOE (model-level), and they are orthogonal and combinable**. DOE regards regularizing training on the same outliers with different weight perturbations as implicitly synthesized but does not change the outliers. The optimization of DOE is constrained by the original training samples and is also sensitive to tuning the model perturbation.\n\nEmpirically, as shown **in Table 1 of the original submission**, we have directly **compared MixOE with our DivOE**. The results show its performance is worse than our DivOE or even the original OE. **For DOE, since it is from a different perspective and orthogonal, we compare DOE mainly in Table 2 of the original submission, and here we present Table 1 in attached PDF for further discussion.** It shows: **(1)** DOE is sensitive to the ID classification task and even performs worse than the original OE in CIFAR-100/SVHN/ImageNet; **(2)** DivOE improves both OE and DOE under the corresponding backbones; **(3)** Even under unfair comparison between DivOE (OE backbone) vs. DOE, DivOE (OE backbone) still achieves the comparable or better performance on ImageNet/SVHN/CIFAR100.  We will add more discussion about the above in our draft.\n\n> **W2, Q2:** \"diversified\" and \"informative\"\n\nThank you for the question.\n\nConceptually, **\"diversified\"** is proposed to **characterize one internal property** of auxiliary outliers, while the \"informative\" in [Chen et al., 2021] is a concept for relative comparison. **The intuition behind \"diversified\"** is that we **hope the outliers can span more informative areas beyond the original ones**, unlike sampling the most \"informative\" outliers (i.e., near-boundary OOD samples that are close to ID data).\n\nIntuitively, **Group 1 in Figure 2(a)** indicates the auxiliary outliers which **span more informative areas among those other groups**. In other words, the most diversified Group 1 also covers the areas of other groups. We are sorry for the unclear part and will make the caption of Figure 2 clearer.\n\nHere we also utilize the **Maximum Mean Discrepancy (MMD) to show the difference between \"diversified\" in our submission and \"informative\" in Table 2 of attached PDF**. We measure the discrepancy between the sampled/manipulated outliers with the original ones, termed as MMD[new, original], to indicate the \"informative\"; measure the discrepancy half by half within the sampled/manipulated outliers, termed as MMD[half new, half new], to indicate the \"diversified\". The results show that DivOE owns lower \"informative\" due to the partial outlier extrapolation but higher \"diversified\" on the inner dispersion of the outliers.\n\nTechnically, in our DivOE, **“diversified” guides the training to expand the auxiliary outliers to the extrapolated distributions but also consider the original ones.** We re-summarized the results of Figure 4(a) to Table 3 in attached PDF and show that our DivOE with the pursuit of “diversified” (conduct partial extrapolation) performs better than those totally considered “informative” (conduct total extrapolation). The results demonstrate the effectiveness and rationality of DivOE considering partial extrapolation for diversification.\n\n> **W4:** rational of PGD\n\nNote that, **PGD is a replaceable choice in the framework of DivOE**. To better understand the effect of these attack-based noise, **we compare it with other inner-attack methods in Table 4 in the attached PDF** and show that **PGD with the uniform distribution loss is generally the better way to achieve higher performance**. We will include these results and discussion in our draft.\n\n> **Q3,Q4:** Explain Eq.4 and Eq.5\n\nThank you for the insightful comments. **Eq.(4) corresponding to an expectation formulation** based on the whole population for diversifying outlier exposure, and **Eq.(5) is an instance-level realization** as directly conducting distributional manipulation in Eq.(4) is prohibitively expensive. The deduction from Eq.(4) to Eq.(5) requires to merge the same terms in Eq.(4) first, and then specifically set $\\beta=\\frac{n-rn}{n}$ with a subsequent rescaling. Finally, we have the formulation that a portion of data to conduct maximization part for diversifying the existing data distribution and the remaining other portion of data corresponds to **learning on the original outliers**. We will add more clear explanation between Eq.(4) and Eq.(5).\n\n> **Q6, Q7, Q8**\n\nDue to the space limit, we place our answer in the general response."}, {"review_id": "BRaqpOrlw5", "reviewer": "Reviewer_vD9G", "summary": "The paper tries to solve the problem that sampled auxiliary informative outliers may not be sufficient and diverse enough to recover the data boundary in the OOD detection setting. To achieve this, the authors propose a new learning objective with information extrapolation, where second term expands the surrogate OOD distributions towards a more diversified one. The authors have provided theoretical analysis to show that for Gaussian mixture model and binary classification problem, DivOE can extrapolate the\noutlier boundary towards ID data. In the experiments, the authors adopt image ID datasets such as CIFAR10, CIFAR100, and results show that DivOE achieves better performances over several evaluation metrics.", "questions": "Q1. The experiments are only performed on two simple image classification tasks with CIFAR10 and CIFAR100. The reviewer is wondering whether the authors have used other image datasets or tabular datasets for evaluation. \n\nQ2. Is there a general guideline in how to choose the extrapolation ratio, augmentation techniques, diversified strength, OOD scores when using DivOE on unseen OOD detection task? It seems that the four factors play an important role in detection performance.\n\nQ3. How would $\\eta$ and $\\alpha$ play an effect in the training steps of DivOE? \n\nQ4. Could different data augmentation techniques be used in combination with the inner-maximization function? Would the augmentation function boost up the performance?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work, as listed in the NeurIPS checklist. The reviewer would appreciate if the authors can address the question and weakness section.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The paper targets an important research problem within the OOD detection research community, and proposes a new auxiliary outlier generation and learning objective to target the research problem that surrogate auxiliary outliers are not sufficient and diverse enough.\n\nS2. The learning objective is simple but adaptable to different post-hoc scoring functions, augmentation techniques and sampling techniques to auxiliary outliers.\n\nS3. The paper has provided a solid theoretical analysis that shows the effectiveness of DivOE within a simplified binary classification setting. The result of Theorem 3.1 is consistent with the authors' explanations and observations in the previous sections.\n\nS4. The paper provides good experiment comparison to exciting methods. They have used several evaluation metrics to demonstrate the effectiveness of the approach.\n\nS5. The writing is very clear and presentation is easy to understand.", "weaknesses": "W1. In Theorem 3.1, The hypothesis class of the binary classification problem is overly simplified to only linear decision boundary. It would be nicer if the authors can provide theoretical results that generalize to more complex hypothesis class.\n\nW2. As Figure 4 shows, the extrapolation ratio and diversified strength are two variables that affect the OOD detection results. If extrapolation ratio is between 0.9~1.0, the performance may degrade and become worse than OE. However, the authors have not mentioned any general guidance to tune those two variables, especially for unseen OOD detection problems.\n\nW3. The authors should consider to include a table comparing the number of outliers generated by DivOE and other benchmark methods in order to achieve similar detection performance scores during experiments.\n\nW4. The authors should also include bold numbers for the best performing algorithms for Table 11, Table 12 and Table 13.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n\n> **W1:** About Theorem 3.1\n\nThanks for the constructive comments. In this work, we follow POEM [Ming et al., 2022] to provide an explanation about DivOE under the linear case. We will generalize the theoretical analysis into the non-linear counterpart by combining the theoretical advancement of deep learning.\n\n> **W2, Q2:** About guidance for extrapolation factors\n\nThanks for the constructive question. Intuitively, we hope the extrapolation can realize the diversification target and cover more potential OOD distributions beyond the original auxiliary ones. Under such high-level guidance, generally, the extrapolation ratio **is expected to be not too large** to totally change the whole original auxiliary distribution, which is empirically demonstrated to be negative for OE, as Figure 4(a) shows. As for diversified strength, we kindly refer the reviewer to the left middle panel of Figure 3, which serves as a search space for synthesizing different outliers using different $\\epsilon$.\n\nIn practical hyper-parameter tuning, we may generally suggest starting with the original OE baseline (i.e., extrapolation ratio equals 0 and diversified strength equals 0) and **enlarging the extrapolation ratio or diversified strength step by step, finding a stage where the OOD detection performance on the validation set will not be further enhanced**. For the coarse-grained augmentations and specific OOD scores, it may be further decided by other conditions on data quality. We will add these discussions to our submission for clarity.\n\n> **W3:** The authors should consider to include a table comparing the number of outliers generated by DivOE and other benchmark methods in order to achieve similar detection performance scores during experiments.\n\n**It may be hard to align the sample number as different methods have different synthesis motivations and but we can actually present the number of generated samples**. To facilitate the comparison, we summarize the specific outlier numbers that were manipulated in the adopted algorithm with its performance in OOD detection in Table 8 in attached PDF. The results show that, compared with previous methods, either need go through all the outliers for sampling or change the training on all the auxiliary outliers, our DivOE can conduct extrapolation on a small portion of original outliers but achieve generally better performance across different ID classification tasks.\n\n> **W4:** The authors should also include bold numbers for the best performing algorithms for Table 11, Table 12 and Table 13.\n\nWe will revise Tables 11, 12, and 13 to bold the numbers of the best-performing algorithms. \n\n> **Q1:** The experiments are only performed on two simple image classification tasks with CIFAR10 and CIFAR100. The reviewer is wondering whether the authors have used other image datasets or tabular datasets for evaluation.\n\nThanks for the question. We accordingly conduct **the experiments on ImageNet, and adopt ImageNet 21k as auxiliary outliers**. We summarize the results in Table 6 in attached PDF, and the results again confirm that our DivOE achieves a competitive performance than other counterparts. We will update all the complete results with the corresponding discussion in our draft.\n\n> **Q3:** How would $\\eta$ and $\\alpha$ play an effect in the training steps of DivOE?\n\nThanks for the question. If we understand correctly, $\\eta$ in our algorithm indicates the learning rate, which we **kept the same** across different OE methods. As for $\\alpha$, it is a step size adopted in the informative extrapolation. To better understand its effect, we conducted the experiments by changing the step size in the PGD attack in Table 9 in attached PDF. Since the step size is only related to the optimization of the constrained maximization, **the effects on the final performance of DivOE are not significant**. By enlarging the step size, we may save more extra time on the inner maximization to reduce the time cost relatively, which is beneficial for algorithm deployment.\n\n> **Q4:** Could different data augmentation techniques be used in combination with the inner-maximization function? Would the augmentation function boost up the performance?\n\nFor different data augmentation techniques, we accordingly conduct experiments and combine them with the inner-maximization function in Table 10 in attached PDF. First, we simply replace the PGD attack-based noise with other data augmentation methods. It seems that in our experiments, **those augmentation methods can improve the performance of the original OE. In comparison, PGD can achieve the better maximization target for extrapolation**. Compared with the pixel level manipulation, more coarse-grained data augmentation can also boost the performance if it can be conducted in an optimization way. In addition, if the original outliers are far away from the ID samples, data augmentation can serve as the basis for inner-maximization."}, {"review_id": "vyyvKwiYfE", "reviewer": "Reviewer_1Wzd", "summary": "The manuscript studies image-wide OOD detection in presence of auxiliary negative data. The negative data is often limited and therefore cannot fully encompass the distribution of inliers. Consequently, contemporary learning procedures fail to deliver classifiers resilient to outliers. To overcome this issue, the manuscript presents a method for extrapolating the negative data towards all modes of the inlier distribution. The proposed method first calculates the gradient of arbitrary OOD score over the input. Then, the sign of the gradient is used to direct the negative input samples towards the inlier distribution. The final learning algorithm uses both initial and extrapolated auxiliary negatives to train the classifier resilient to outliers. The proposed method outperforms relevant related works on small image benchmarks.", "questions": "C1. Can the proposed method work on large-scale experiments [d]\n\nC2. Is the extrapolation of outlier data towards inliers always possible or there are some requirements that should be met?\nAnalysis similar to [e] could improve the manuscript.\n\n[d] Haoqi Wang, Zhizhong Li, Litong Feng, Wayne Zhang:\nViM: Out-Of-Distribution with Virtual-logit Matching. CVPR 2022.\n\n[e] Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, Feng Liu:\nIs Out-of-Distribution Detection Learnable? NeurIPS 2022", "limitations": "Although promised in Appendix D (L435), the limitations are not clearly stated. One possible limitation might be W1.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The manuscript deals with an important issue.\n\nS2. Extrapolation of auxiliary negative data towards modes of inlier distribution intuitively makes sense.\n\nS3. The developed method achieves competitive results on considered benchmarks.\n\nS4. The developed method can be combined with existing OOD detectors (e.g. Energy, MSP, ... )", "weaknesses": "W1.  The manuscript does not discuss the effectiveness of the method when there is only a small auxiliary dataset available. It seems that the developed method still requires a broad OE dataset (Tiny-ImageNet as stated in L236).\n\nW2. The manuscript does not consider relevant related works which use synthetic negatives created by generative models [a,b,c]. Synthetic negatives are an effective way for augmenting the auxiliary dataset and the proposed method should outperform methods trained on a mixture of real and synthetic negative data.\n\nW4. The manuscript does not reflect on the additional computational budget (time and memory) required by the method over the OE baseline.\n\n[a] Shu Kong, Deva Ramanan: OpenGAN: Open-Set Recognition via Open Data Generation. ICCV 2021\n\n[b] Matej Grcic, Petra Bevandic, Sinisa Segvic: Dense Open-set Recognition with Synthetic Outliers Generated by Real NVP. VISAPP 2021.\n\n[c] Kimin Lee, Honglak Lee, Kibok Lee, Jinwoo Shin: Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples. ICLR 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n\n> **W1:** The manuscript does not discuss the effectiveness of the method when there is only a small auxiliary dataset available. It seems that the developed method still requires a broad OE dataset (Tiny-ImageNet as stated in L236).\n\nHere, we added the experiments under different numbers of auxiliary outliers used for all OE-based methods. The experimental results are summarized in Table 11 in attached PDF, which indicates that the performance of OOD detection can be generally better if we use more outliers. **Under smaller number of samples, i.e., from 50000 to 1000 (relatively smaller than 50000 ID data), DivOE can consistently perform better than OE**.\n\n> **W2:** The manuscript does not consider relevant related works which use synthetic negatives created by generative models [a,b,c]. Synthetic negatives are an effective way for augmenting the auxiliary dataset and the proposed method should outperform methods trained on a mixture of real and synthetic negative data.\n\nThanks for recommending these related works [a,b,c], and we will add the proper discussion and comparison in the revision. Those methods additionally train generative models for generating extra outliers for data augmentation, which **depends on the sample quality and actually are orthogonal to our extrapolation**. We conduct experiments comparing and incorporating the method [c] for informative extrapolation in Table 12 in attached PDF. We find that the GAN-synthesized method is hard to train under our setups. **When we incorporate DivOE into the generated sample, the performance can better enhanced than the original performance of GAN-based method**. More results about [a,b,c] will be conducted and added to the submission.\n\n[a] Shu Kong, Deva Ramanan: OpenGAN: Open-Set Recognition via Open Data Generation. ICCV 2021\n\n[b] Matej Grcic, Petra Bevandic, Sinisa Segvic: Dense Open-set Recognition with Synthetic Outliers Generated by Real NVP. VISAPP 2021.\n\n[c] Kimin Lee, Honglak Lee, Kibok Lee, Jinwoo Shin: Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples. ICLR 2018.\n\n> **Actually W3:** The manuscript does not reflect on the additional computational budget (time and memory) required by the method over the OE baseline.\n\nThanks for the constructive comments. To better understand the computational budget, we conduct additional experiments and summarize the time and memory results in Table 13 in attached PDF, which shows that **DivOE can achieve better performance with comparable time and memory costs with other OE-based methods**.\n\nIn addition, since the extra time requirement compared with the original OE is mainly from the inner-maximization of the informative extrapolation, we also explore the effect of perturbation noise on the OOD detection performance in Table 14 in attached PDF. The results show that **we can further save extra time by reducing the outlier manipulation times with maintaining the improvement**.\n\nWe will update the complete results and corresponding discussion in our draft.\n\n> **Q1:** Can the proposed method work on large-scale experiments\n\nThanks for the question. We added additional experiments on ImageNet (with ImageNet 21k as the auxiliary outlier dataset), and present the results in Table 6 in attached PDF. **The results demonstrate the consistent effectiveness of DivOE on the large-scale data**.\n\nWe will add the complete results with the corresponding discussion in our submission.\n\n> **Q2:** Is the extrapolation of outlier data towards inliers always possible or there are some requirements that should be met? Analysis similar to [e] could improve the manuscript.\n\nThanks for the constructive question and insightful suggestion! The extrapolation of outlier data towards inliers **may not always be possible**. Similar to the condition analysis in [e], the informative extrapolation that targets better characterize the decision boundary between ID and OOD also needs some requirements or realization assumptions. Intuitively, it mainly **relies on auxiliary outliers and specific extrapolation techniques**. For the former, the original outliers **should not be far away from the ID sample**; For the latter, the extrapolation techniques should be able to **add enough perturbation or semantic changes to the original outlier**. We will extend the above discussion with more conditional derivation by referring to the analysis in [e].\n\n[e] Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, Feng Liu: Is Out-of-Distribution Detection Learnable? NeurIPS 2022"}, {"review_id": "jjxqAtG0Cd", "reviewer": "Reviewer_Mh63", "summary": "The authors of this paper present an innovative methodology for out-of-distribution (OOD) detection. While existing methodologies typically involve the direct use of given OOD samples, this paper introduces a new approach that applies perturbations to OOD samples, allowing the model to experience a more diverse set of OOD samples during training.\n\nThe authors define these perturbations using an adversarial loss based on the uniform distribution loss commonly applied to OOD samples. This creates directional guidance for each instance, determining the direction of the gradient and thus defining the perturbations applied to OOD samples.\n\nThe results demonstrate that using perturbed OOD samples improves the performance of OOD detection across several key metrics, including False Positive Rate at 95% Recall (FPR95), Area Under the Receiver Operating Characteristic Curve (AUROC), and Area Under the Precision-Recall Curve (AUPR), when compared to existing methodologies.", "questions": "Please see the section weaknesses.", "limitations": "Please see the section weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper's novel approach of applying perturbations to a given data instance in order to utilize a broader array of samples and those tightly located at the decision boundary is indeed a reasonable and intriguing choice. The authors' concept of adversariality against the uniform loss, which implies a concentration of prediction towards a particular class, exhibits an interesting property worth exploring further.\n\nExperimentally, the authors provide a substantial ablation study on the hyperparameters used in loss composition, which adds to the comprehensiveness of their methodology. Notably, the use of t-SNE based plotting allows for a clear visualization of how the OOD samples are perturbed to be situated very closely to in-class samples. It also arise the question of \"how would ood behave for the different perturbation types?\".", "weaknesses": "The authors have presented an interesting methodology that applies adversarial perturbations based on a particular loss function to enhance the performance of out-of-distribution detection. However, it would be important for the authors to provide empirical evidence that demonstrates the superiority of this adversarial perturbation over other types of perturbations. This would require a systematic and rigorous experimentation design and would ideally be conducted across various datasets and under different conditions to ensure the results are robust and generalizable.\n\nIn the final implementation of the authors' methodology, it's noted that both the perturbed and unperturbed out-of-distribution (OOD) samples are simultaneously considered in the loss function. However, the manuscript doesn't sufficiently explain the rationale behind this particular choice. It would be particularly interesting to see a comparison of results when using only perturbed samples, only unperturbed samples, or both, in the loss function. \n\na crucial aspect that needs further clarification and discussion is the ratio of in-distribution to OOD samples used in the training process. The choice of this ratio can significantly affect the performance and robustness of the model. For instance, a too high proportion of OOD samples could make the model overly sensitive to outliers, while a too low proportion might not adequately expose the model to OOD scenarios.\n\nWhile the paper shows promising results in the specific contexts tested, it would be beneficial for the authors to provide a more extensive analysis covering a range of different OOD datasets. If the authors can provide empirical evidence demonstrating the consistent performance of their method across diverse OOD datasets, it would significantly strengthen their claims. \n\nit's unclear from the manuscript how each OOD sample is directed towards a specific class during this adversarial perturbation process. Specifically, they should explain how the gradient direction, which determines the perturbations applied to the OOD samples, correlates with the movement of these samples towards specific classes.\n\nI am more than eager to increase my score if the questions above are adequately answered.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n\n> **Q1:** However, it would be important for the authors to provide empirical evidence that demonstrates the superiority of this adversarial perturbation over other types of perturbations.....\n\nThanks for the constructive comments. We would like to clarify that **DivOE was proposed as a general framework** for diversifying outlier exposure with **partially conduct informative extrapolation**, which allows any appropriate noise or augmentation techniques to be incorporated. \n\nTo show the effectiveness of PGD, we conduct the following experiments. On the one hand, we compare the random noise, re-adversarial perturbation, and adversarial perturbation to justify its rationality. The results are summarized in the Table 15 in attached PDF and show **the adversarial perturbation achieve better performance than other types of perturbations**. On the other hand, we compare with other adversarial attacks and data augmentation methods for inner-maximization part in Table 4 in attached PDF and show that **PGD with the uniform distribution loss is generally a better way** to achieve higher performance due to the optimization originality. We will add the complete results with the corresponding discussion in our draft.\n\n> **Q2:** ...However, the manuscript doesn't sufficiently explain the rationale behind this particular choice...\n\nConsidering the initial motivation to diversify outliers, we explore to extrapolate more potential OOD distribution beyond the original auxiliary ones. For the unperturbed OOD samples used in training, we hope **the training can still learn the knowledge from the original auxiliary outliers**, for the perturbed OOD samples, we hope **the training can find different but informative outliers beyond the original ones** to improve the generalization of outlier exposure.\n\nFormally, Eq.(5) is an instance-level realization of **Eq.(4) that actually contains the perturbed part and the unperturbed part**. The deduction from Eq.(4) to Eq.(5) requires to merge the same terms in Eq.(4) first, and then specifically set $\\beta=\\frac{n-rn}{n}$ with a subsequent rescaling. Finally, we have the formulation that a portion of data to conduct maximization part for diversifying the existing data distribution and the remaining other portion of data corresponds to learning on the original outliers. We will add more explanation about the final implementation.\n\nTo better understand the specific effects  of different samples, it can be referred to the left panel of Figure 4 (in the original submission), in which we show the OOD detection performance of DivOE under different portions of perturbed data during training. **Generally, using a small part of perturbed samples can achieve the extrapolation target and improve the performance**. However, if we totally perturb all the data, it is even worse than the original OE.\n\n> **Q3:** a crucial aspect that needs further clarification and discussion is the ratio of in-distribution to OOD samples used in the training process...\n\nThanks for pointing out this. We generally **keep the same auxiliary outlier across different methods for fair comparison**. For the effects under different ratios of auxiliary outliers to ID samples, we conduct the extra experiments in Table 11 in attached PDF to show the results. Specifically, we vary the outlier ratio from 1.0 (50000) to 0.02 (1000) and keep the ID data number=50000 unchanged. Generally, the empirical results show that all the OE-based methods can achieve better OOD detection performance, and **our proposed DivOE can consistently improve the performance under different ratios. In comparison, with less auxiliary outliers, the model performance of ID classification will be less affected**. We will add the complete results with the corresponding discussion in our draft.\n\n> **Q4:** it would be beneficial for the authors to provide a more extensive analysis covering a range of different OOD datasets...\n\nWe would like to re-explain that all the OOD detection results in Table 1 and Table 2 are averaged on five different OOD test datasets, i.e., Textures, Places365, iSUN, LSUN-r, LSUN-c, which are also benchmarked OOD test sets used in previous works. For the direct comparison under those OOD test sets, we list the performance of our DivOE with the original OE in Table 16 in attached PDF, which show that **DivOE can generally perform better across different OOD test sets**. For more results on other OE-based methods, due to the space limit, we kindly refer the reviewer to Tables 4-8 in our Appendix for more results and analysis.\n\n> **Q5:** how each OOD sample is directed towards a specific class during this adversarial perturbation process...\n\nDue to the space limit, we place the answer to the general response."}]

Author rebuttal:
## General Response

We appreciate all the reviewers for their thoughtful comments and suggestions on our paper.

We are very glad to see that the reviewers find our focused problem is **important** (R2,R3) within the OOD detection research, and the method is **novel** (R1,R2,R3,R4), **theoretical reasonable** (R1,R2) and **simple but adaptable** (R1,R2,R3) to various other techniques, and the experiments are **good**, **comprehensive** and demonstrate the **general effectiveness** of our DivOE (R1,R2,R3,R4). We are also pleased that the reviewers find our writing is **very clear** and **easy to understand** (R2).

We have tried our best to address the reviewers' comments and concerns in **individual responses to each reviewer** with comprehensive experimental justification. The reviews allowed us to improve our draft and **the contents added** in the revised version and **the attached PDF** are summarized below:

**From Reviewer aWLd**

- Clarify and compare the difference of DivOE with MixOE and DOE (see Tables 1,2 in original submission and Table 1 in PDF)
- Explain and compare the difference between "diversified" and "informative" (see Tables 2,3 in PDF)
- Conduct a comparison to explain the rationality of PGD-based attack (see Table 4 in PDF)
- Extend Table 2 with all the add-on varieties (see Table 5 in PDF)
- Conduct experiments on the Large-scaled ImageNet dataset (see Table 6 in PDF)
- Add application of DivOE with OECC (see Table 7 in PDF)

**From Reviewer vD9G**

- Compare the specific outlier numbers affected by each method (see Table 8 in PDF)
- Conduct experiments on the Large-scaled ImageNet dataset (see Table 6 in PDF)
- Show the effect of $\alpha$ in the training/synthetic steps of DivOE (see Table 9 in PDF)
- Incorporating data augmentation methods under the framework of DivOE (see Table 10 in PDF)

**From Reviewer 1Wzd**

- Check the performance when the number of auxiliary outliers is changed (see Table 11 in PDF)
- Discuss and compare performance with GAN-based methods (see Table 12 in PDF)
- Include a comparison of time and memory cost (see Table 13,14 in PDF)
- Conduct experiments on the Large-scaled ImageNet dataset (see Table 6 in PDF)

**From Reviewer Mh63**

- Design additional experiments for the rationality of attack-based inner-maximization (see Tables 4,15 in PDF)
- Check the performance when the number of auxiliary outliers is changed (see Table 11 in PDF)
- Clarify the results considering non-perturbed, perturbed, or both samples in DivOE (see Figure 4(a) in the original submission)
- Clarify the results across five different OOD test sets (see Table 16 in PDF)
- Visualize prediction results of the adversarial perturbation process (see Figure 1 in PDF)

**We appreciate your comments and time!** We have tried our best to address your concerns and revised the paper following the suggestions. **Would you mind checking it and confirming if there are any unclear parts?**

--- 
### Some rest answers:

For **Reviewer aWLd**

> **Q6,Q7,Q8:** about experiments...

Thanks for the constructive suggestions. **For adding other methods to Table 2**, we conduct extra experiments accordingly and summarize the results **in Table 5 in attached PDF**. The results show that our proposed DivOE can consistently improve performance on the original basis compared with other counterparts across different setups. We will update the complete results in our draft. **For the large-scale experiments**, We accordingly add experiments on the ImageNet benchmark **in Table 6 in attached PDF**, which adopts ImageNet 21k as auxiliary outliers like DOE. The results confirm the effectiveness of our DivOE. **For OECC**, we add the experiments on OECC with our DivOE **in Table 7 in attached PDF**, and will merge them into the original Table 2.

For **Reviewer Mh63**

> **Q5:** about the adversarial perturbation process...

Thanks for the constructive question. From the algorithmic perspective, **the adversarial perturbation with uniform distribution loss is a non-target perturbation**. The intuition is to perturb the OOD sample to be more like ID regarded by the model. Maximizing uniform distribution loss has no requirement for each OOD sample to be perturbed into some specific ID classes. Thus, it is actually **automatically decided by the gradient for the specific perturbation directions**.

To better understand the perturbation effect, we conduct the **case study on the specific example in Figure 1 in attached PDF** for their specific prediction results during the adversarial perturbation process. It can be find that after the adversarial perturbation, **the prediction result of the second example will be concentrate on one specific class with high confidence**. By enlarging the perturbation strength $\epsilon$, the example will be perturbed to be confidently predicted on different classes. We will update more similar visualization to illustrate the movement of the perturbation process on specific samples in our draft.

To provide an overview of the prediction result change, we also check it in our experiments and show in the following table, the results show that after our adversarial perturbation, **most of the OOD samples will be regarded as different ID classes before the perturbation**. It also demonstrates the our extrapolation can finding different outlier distributions beyond the original ones. 

| Dataset | Same class | Different class|
| :---: |  :---: |  :---: | 
| CIFAR-100 | 4507 | **20517** |
| CIFAR-10  | 9802 | **15222** |

Meta-review: This paper focuses on out-of-distribution detection and addresses the weakness of previous outlier exposure-based methods when dealing with a small number of auxiliary outliers. The authors add perturbations to the given outliers to expand the surrogate OOD (out-of-distribution) distributions. The synthesized OOD distributions help bridge the gap between the limited original outliers and in-distribution samples, thereby improving the model's ability to detect OOD samples. During the author-reviewer discussion, most of the previous concerns were addressed, including the comparison with other OE (outlier exposure) works, the method's details, and the sufficiency of the experiments. The reviewers consistently provided positive comments on this paper. If the authors incorporate the suggestions and additional results into the manuscript, this paper could be further improved. Overall, I recommend accepting this paper.
Decision: Accept (poster)