Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources
Authors: Haotian Zheng
Qizhou Wang
Zhen Fang
Xiaobo Xia
Feng Liu
Tongliang Liu
Bo Han
Venue: NeurIPS 2023 poster (2023)
Keywords: out-of-distribution detection

Abstract: Out-of-distribution (OOD) detection discerns OOD data where the predictor cannot make valid predictions as in-distribution (ID) data, thereby increasing the reliability of open-world classification. However, it is typically hard to collect real out-of-distribution (OOD) data for training a predictor capable of discerning ID and OOD patterns. This obstacle gives rise to *data generation-based learning methods*, synthesizing OOD data via data generators for predictor training without requiring any real OOD data. 
Related methods typically pre-train a generator on ID data and adopt various selection procedures to find those data likely to be the OOD cases. However, generated data may still coincide with ID semantics, i.e., mistaken OOD generation remains, confusing the predictor between ID and OOD data. To this end, we suggest that generated data (with mistaken OOD generation) can be used to devise an *auxiliary OOD detection task* to facilitate real OOD detection. Specifically, we can ensure that learning from such an auxiliary task is beneficial if the ID and the OOD parts have disjoint supports, with the help of a well-designed training procedure for the predictor. Accordingly, we propose a powerful data generation-based learning method named *Auxiliary Task-based OOD Learning* (ATOL) that can relieve the mistaken OOD generation. We conduct extensive experiments under various OOD detection setups, demonstrating the effectiveness of our method against its advanced counterparts.

Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources
Haotian Zheng 1.2+
Qizhou Wang' Zhen Fang"
Xiaobo Xia*
Feng Liu
Tongliang Liu Bo Han
'Department of Computer Science, Hong Kong Baptist University
School Electronic Engineering, Xidian University
Australian Artificial Intelligence Institute, University of Technology Sydney *Sydney AI Centre, The University of Sydney
School of Computing and Information Systems, The University of Melbourne
htzheng.xxe xdu @ gmail.com (csqzwang, bhanml) @oomp xiaoboxia. gmail.com tongliang. liu @ @ydney.edu.au
@ecom..kbb..com. www...bu........... edu..kk
zhen.fang @utt. edu.edu.au fengliu.ml @ gmail.com
Abstract
Out-of-distribution (OOD) detection discerns OOD data where the predictor cannot make valid predictions in-distribution (ID) data, thereby increasing the reliability open-world classification. However, it is typically hard to collect real out-of- distribution (OOD) data for training predictor capable of discerning ID and OOD patterns. This obstacle gives rise to data generation-based learning methods, synthesizing OOD data via data generators for predictor training without requiring any real OOD data. Related methods typically pre-tras pre-train a generator on ID data and adopt various selection procedures to find those data likely to be the OOD cases. However, generated data may still coincide with ID semantics, i.e., mistaken OOD generation remains, confusing the predictor between ID and OOD data. To this end, we suggest that generated data (with mistaken OOD generation) can be used devise an auxiliary OOD detection task facilitate real OOD detection. Specifically, we can ensure that learning from such an auxiliary task is beneficial if the ID and the OOD parts have disjoint supports, with the help of well-designed training procedure for the predictor. Accordingly. we propose a powerful data generation-biie based learning method named Auxiliary Task-based OOD Learning (ATOL) that can relieve the mistaken OOD generation. We conduct extensive experiments under various OOD detection setups, demonstrating the effectiveness of our method against its advanced counterparts. The code is publicly available at: https: // thub. com/ homm/mmrree httpt/mmlllrrr/rreeeee/rtereeret grooppATTOO.
Introduction
Deep learning in the open world should not only make accurate predictions for in-distribution (ID) data meanwhile should detect out-of- ootto--distribttion (OOD) data whose semantics are different from ID cases [15, 56, 22, 81, 85, 82]. It drives recent studies OOD detection [43, 23, 33, 66, 80, 24], which is important for many safety- cafety-critical applications such as autonomous driving and medical analysis. Previous works have demonstrated that well-trained predictors can wrongly take many OOD data as ID cases [1, 15, 21, 48], motivating recent studies towards effective OOD detection.
* Equual contributions. Correspondence to Bo Han (thamml@comp.hkhuu.chl
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Generated Data
Auxillary Task
Proportion
Joint Train
Predictor Learn
FRRS5
60
Ship
Automobiic
Aunibary ID
Duall
Audilary DOD
Aligrment
Real Data
Epach
Roal Task
(a) Mistaken Semantics
(b) Adverse Impacts
Figure 1: Illustrations of mistaken OOD generation. Fig- ure ((a) indicates the generated OOD data (colored in red) may wrongly possess semantics of the real ID data (colored in green), e.g., generated data with the ID seman- ties of "Horse". Figure I(b) demonstrates that the OOD performance of predictors OOD data, described Appendix E.2. With the propor- tion of OOD data possessing ID semantics increasing, the OOD performance (measured by FPR95) degrades.
Figure Illustration of our method. ATOL makes the predictor learn from the real task and our auxiliary task jointly via a well-designed learning procedure, leading to the separation between the OOD and ID cases. The predictor can benefit from such an aux- iliary task for real OOD detection, re- lieving mistaken OOD generation.
impaired by unreliable
In the literature, an effective learning scheme to conduct model regularization with OOD data, making predictors achieve low-confidence predictions on such data [3, 16, 36, 42, 49, 75, 79]. Overall, they directly let the predictor learn to discern ID and OOD patterns, thus leading improved performance in OOD detection. However, it is generally hard to access real OOD data [10, 11, 69, 70], hindering the practical usage scenarios for such a promising learning scheme.
Instead of collecting real OOD data, we can generate OOD data via data generators to benefit the predictor learning. motivating the data generation-base methods [31. 50, 55, 62, 67]. Generally speaking, existing works typically fit data generator [12] on ID data (since real OOD data are inaccessible). Then, these strategies target on finding the OOD-like cases from generated data by adopting various selection procedures. For instance, Lee et al. [31] and Vernekar et al. [62] take boundary data that lie far from ID boundaries OOD-like data; PourReza et al. [50] select those data generated in the early stage of generator training. However, since the generators are trained on ID data and these selection procedures can make mistakes, one may wrongly select data with ID semantics as OOD cases (cf., Figure (a)), i.e., mistaken OOD generation remains. It will misguide the predictors to confuse between ID and OOD data, making the detection results unreliable, as in Figure (bb). Overall, it is hard to devise generator training procedures or data selection strategies to overcome mistaken OOD generation, thus hindering practical usages of previous data generation-baat methods.
To this end, we suggest that generated data can be used to devise an auxiliary task, which can benefit real OOD detection even with data suffered from mistaken OOD generation. Specifically, such an auxiliary task is a crafted OOD detection task, thus containing ID and OOD parts of data (termed auxiliary ID and auxiliary OOD data, respectively). Then, two critical problems arise -how to make such an auxiliary OOD detection lask learnable w.r.l the predictor and how to ensure that the auxiliary OOD detection task is beneficial w.r.l. the real OOD detection. For the first question, we refer to the advanced OOD detection theories [11], suggesting that the auxiliary ID and the auxiliary OOD data should have the disjoint supports in the input space (i.c., without overlap w.r.t. ID and OOD distributions), cf., Cl (Condition 1 for short). For the second question,
justify that, for predictor, if real and auxiliary ID data follow similar distributions, cf., C2, the auxiliary OOD data reliable OOD data w.r.t. the real ID data. In summary, if C1 and C2 hold, we can ensure that learning from the auxiliary OOD detection task can benefit the real OOD detection, cf., Proposition 1.
Based on the auxiliary task, we propose Auxiliary data generation- based OOD learning method. For C1, it is generally hard to directly ensure the disjoint supports between the auxiliary ID and auxiliary OOD data due to the high-dimensional input space. Instead, we manually craft two disjoint regions in the low-dimensional latent space (i.e., the input space of the generator). Then, with the generator having distance correlation between input
OOD Learning (ATOL), an effective

---

space and latent space [63]. we can make the generated data that belong to two different regions in the latent space have the disjoint supports in the input space, assigning to the auxiliary ID and the auxiliary OOD parts, respectively. Furthermore, to fulfill C2, we propose a distribution alignment risk for the auxiliary and the real ID data, alongside the OOD detection risk to make the predictor benefit from the auxiliary OOD detection task. Following the above learning procedure, we can relieve the mistaken OOD generation issue and achieve improved performance over previous works in data generation-baser OOD detection, cf., Figure 2 for heuristic illustration.
To verify the effectiveness of ATOL, we conduct extensive experiments across representative OOD detection setups, demonstrating the superiority of our method over previous data generation based methods, such as Vernekar et al. [62], with 12.02%, 18. 18.32%. and 20.70% improvements measured by average FPR95 on CIFAR-10. CIFAR-100 and ImageNet datasets. Moreover, compared with more advanced OOD detection methods (cf., Appendix F), such as Sun et al. [57], our ATOI, can also have promising improvements, which reduce the average FPR95 by 4.36%, 15.03%, and 12.02% on CIFAR-10, 10, CIFAR-100, and ImageNet datasets.
Contributions. We summarize our main contributions into three folds:
We focus on the mistaken OOD generation in data generation-basee detection methods, which are largely overlooked previous works. To this end, we introduce an auxiliary OOD detection task to combat the mistaken OOD generation and suggest a set of conditions in Section to make such an auxiliary task useful.
Our discussion about the auxiliary task leads to a practical learning method named ATOL. Over existing works in data generation- based methods, our ATOL introduces small costs of extra computations but is less susceptible to mistaken OOD generation.
We conduct extensive experiments across representative OOD detection setups, demonstrat- ing the superiority of our method over previous data generation- baaeed methods. Our ATOL is also competitive over advanced works that use real OOD data for training, such as outlier exposure [16], verifying that the data generation-base methods are still worth studying.
2 Preliminary
Let X' C R." the input space, y= (1....,c) the label space, and h p R. - R the predictor, where Φ : 1² - d is the feature extractor and p _R + Re is the classifier. We consider P.Pyy and PiP the joint and the marginal ID distribution defined over X' x y and X, respectively. We also denote POOD the marginal OOD distribution over X. Then, the main goal of OOD detection is to find the proper predictor h-.) and the scoring function s(: h): : IR" + R such that the OOD detector
ID, if (x; h) 2 8,
fa(x) - OOD, sf: h) < 3,
can detect OOD data with a high success rate, where /3 is a given threshold. To get a proper predictor ht.), outlier exposure [16] proposes regularizing the predictor to produce low-confidence predictions for OOD data. Assuming the real ID data X and label yiD are randomly drawn from PlD then the learning objective of outlier exposure can be written as:
min Ep [&cu (Xw: VID: h)] + AEp (x; h)]
h
where À is the trade-off hyper- papper-prramter. fcE(.) is the cross-entropy loss, and (oE(.) is the Kullback- Leibler divergence of softmax predictions to the uniform distribution. Although outlier exposure remains one of the most powerful methods in OOD detection, its critical reliance on OOD data hinders its practical applications.
Fortunately, data generation-bast OOD detection methods can overcome the reliance on real OOD data, meanwhile making the predictor learn to discern ID and OOD data. Overall, existing works [31, 62, 67] leverage the generative adversarial network (GAN) [12, 51] to generate data, collecting those data that are likely to be the OOD cases for predictor training. As a milestone work, Lee et al. [31] propose selecting the "boundary" data that lie in the low-density area of PIP, with the following learning objective for OOD generation:
min max G
Egxx [fon h)]
Epp (log D(xm)) + Bax (log(1 D(x))],
(3)
D
(a) generating boundary data
(b) training data generator
3

---

where Gx denotes the OOD distribution of the generated data w.r.t the generator G : R" + R" (m the dimension of the latent space). Note that the term (a) forces the generator G(+) to generate
low-density data and the term (b) is the standard learning objective for GAN training [12]. Then, with the generator trained for OOD generation, the generated data are used for predictor training, following the outlier exposure objective, given by
min Epp, (XID, yID; h)] + ABEx [xor(x; h)]
h
Drawbacks. Although promising line of works, the generated data therein may still contain many ID semantics (cf., Figure ((a)). It stems from the lack of knowledge about OOD data, where one can only access ID data to guide the training of generators. Such mistaken OOD generation is common in practice, and its negative impacts are inevitable in misleading the predictor (cf., Figure 1(b)). Therefore, how to overeome the mistaken OOD generation is one of the key challenges data generation-based OOD detection methods.
3 Motivation
Note that getting generators to generate reliable OOD data difficult, since there no access to real OOD data in advance and the selection procedures make mistakes. Instead, given that generators suffer from mistaken OOD generation, we aim to make the predictors alleviate the negative impacts raised by those unreliable OOD data.
Our key insight is that these unreliable generated data can be used to devise a reliable auxiliary task which can benefit the predictor in real OOD detection. In general, such an auxiliary task is a crafted OOD detection task, of which the generated data therein are separated into the ID and the OOD parts. Ideally, the predictor can learn from such an auxiliary task and transfer the learned knowledge (from the auxiliary task) to benefit the real OOD detection, thereafter relieving the mistaken OOD) generation issue. Therein, two issues require our further study:
Based on unreliable OOD sources, how to devise a learnable auxiliary OOD detection task?
Given the well-designed auxiliary task, how to make benefit the real OOD detection?
For the first question, we refer the advanced work [11] in OOD detection theories, which provides necessary conditions for which the OOD detection task is learnable. In the context of the auxiliary
OOD detection task, we will separate the generated data into the auxiliary ID parts and auxiliary OOD parts, following the distributions defined by GPP and gooD, respectively. Then, we provide the following condition for the separable property of the auxiliary OOD detection task.
Condition 1 (Separation for the Auxiliary Task). There is no overlap between the auxiliary ID distribution and the auxiliary OOD distribution, i.e., supp G n supp GoOD 0, where supp denotes the support set of a distribution.
Overall, CI states that the auxiliary ID data distribution GP and the auxiliary OOD data distribution GoOD should have the disjoint supports. Then, according Fang et al. [11] (Theorem 3), such an auxiliary OOD detection might be learnable, that the predictor can have a high success rate in discerning the auxiliary ID and the auxiliary OOD data. In general. the separation condition is indispensable for us to craft a valid OOD detection task. Also, as in the proof of Proposition 1, the separation condition is also important to benefit the real OOD detection.
For the second question, we need to first formally define if a data point can benefit the real OOD detection. Here, we generalize the separation condition in Fang et al. [11], leading to the following definition for the reliability of OOD data.
Definition 1 (Reliability of OOD Data). An OOD data point is reliable (w.Et. real ID data) to a mapping function & (-), if o' (x) & suppo' # PiP, where d'g is the distribution transformation associated with ø' (.).
With identity mapping function ('()), the above definition states that x is a reliable OOD data point if it is not in the support set of the real ID distribution w.r.t the transformed space. Intuitively, if real ID data are representative almost surely [11], ('1))'() Φ'(-) far from the ID support also indicates that d' (.). will not possess ID semantics, namely, reliable. Furthermore, of (-) is not limited to identity mapping,

---

which can be defined by any transformed space. The feasibility of such generalized definition is supported by many previous works [10, 41] in OOD learning, demonstrating that reliable OOD data defined in the embedding space (or in our context, the transformed distribution) can also benefit the predictor learning OOD detection.
Based on Definition I, we want to make auxiliary OOD data reliable w.r.t. real ID data, such that the auxiliary OOD detection task is ensured to be beneficial, which motivates the following condition.
Condition (Transferability of the Auxiliary Task). The auxiliary OOD detection task is transfer- able w.r.t. the real OOD detection task if H5 PVP 2 d'4...
Overall, C2 states that auxiliary ID data approximately follow the same distribution as that of the real ID data in the transformed space, i.e., ø' (.t). Note that, Auxiliary ID and OOD data arbitrarily differ from real ID and OOD data in the data space, i.e., auxiliary data are unreliable OOD sources. However, if C2 is satisfied, the model "believes" the auxiliary ID data and the real ID data the same. Then, we are ready to present our main proposition, demonstrating why C2 can ensure the reliability of auxiliary OOD data.
Proposition 1. Assume the predictor h o Φ can separate ID and OOD data, namely,
Egg (X YID; hi] + AR [eogo( h)]
(5)
approaches O. Under Cl and C2, auxiliary OOD data are reliable w.r.t. real ID data given ø'
Φ.
The proof can be found in Appendix C. In heuristics, Eq. (5) states that the predictor should excel at such an auxiliary OOD detection task, where the model can almost surely discern the auxiliary ID and the auxiliary OOD cases. Then, if we can further align the real and the auxiliary ID distributions (i.e.. C2), the predictor will make no difference between the real ID data and the auxiliary ID data. Accordingly, since the auxiliary OOD data are reliable OOD cases w.r.t. the auxiliary ID data, the auxiliary OOD data are also reliable w.r.f. the real ID data. Thereafter, one can make the predictor learn to discern the real ID and auxiliary OOD data, where no mistaken OOD generation affects. Figure summarizes our key concepts. As we can see, learning from the auxiliary OOD detection task and further aligning the ID distribution ensure the separation between ID and OOD cases, benefiting the predictor learning from reliable OOD data and improving real OOD detection performance.
There are two parallel definitions for real OOD data [70], related to different goals towards effective OOD detection. One definition is that exact distribution of real OOD data exists, and the goal is to mitigate the distribution discrepancy w.r.t. the real 0OD distribution for effective OOD detection. Another definition is that all those data whose true labels not in the considered label space are real OOD data, and the goal is to make the predictor learn to see as many OOD cases as possible. Due to the lack of real OOD data for supervision, the latter definition is more suitable than the former one in our context, where we can concentrate on mistaken OOD generation.
Remark 1.
4 Learning Method
Section 3 motivates new data generation- based method named Auxiliary Task-based OOD Learning (ATOL), overcoming the mistaken OOD generation via the auxiliary OOD detection task. Referring to Theorem 1, the power of the auxiliary task is built upon CI and C2. It makes ATOL a two-staged learning scheme, related to auxiliary task crafting (crafting the auxiliary task for C1) and predictor training (applying the auxiliary task for C2), respectively. Here, we provide the detailed discussion.
4.1 Crafting the Auxiliary Task
We first need to construct the auxiliary OOD detection task, defined by auxiliary ID and auxiliary OOD data generated by the generator G(.). .) In the following, we assume the auxiliary ID data XiD drawn from GDP and the auxiliary OOD data ROoD drawn from goon Then, according CI, we need fulfill the condition that GiD and Good should have the disjoint supports.
Although all distribution pairs (GR, GOOD) with disjoint supports can make Cl hold, it is hard to realize such a goal due to the complex data space R. of images. Instead, we suggest crafting such a distribution pair in the latent space 1.Th of the generator, of which the dimension m is much lower
5

---

than n. We define the latent ID and the latent OOD data as ZID and ZOOD. respectively. Then, we assume the latent ID data are drawn from the high-density region of a Mixture of Gaussian (MoG):
ZID E 2 with z = {z ~ Mz)lM(z) > T}
(6)
where is the threshold and Mz is pre-defined MoG, given by the density function of MoG M(z) = Ei=1 (2 o.), with the mean l4, and the covariance o, for the i-th sub-Gaussian Furthermore, we demonstrate in Section 4.2, we also require the specification of labels for latent ID data. Therefore, we assume that Mz is constructed by c sub-distributions of Gaussian, and data belong to each of these sub-distributions should have the same label ÿID-
Remark 2. MoG provides a simple way to generate data, yet the key point is to ensure that auxiliary ID/OOD data should have the disjoint support, i.e, CI. Therefore, CI is satisfied properly, other noise distributions, such the beta mixture models| [o]] and the uniform distribution, can also be used. Further, our ATOL is different from previous data generation-base methods in that we do not require generated data to be reliable in the data space, ATOL, does not involve fitting the MoG to real ID data, where the parameters can be pre-defined and fixed. Therefore, we do not consider overfitting and accuracy of MoG in our paper.
Furthermore, for the disjoint support, the latent OOD data are drawn from a uniform distribution except for the region with high MoG density, i.e.,
ZOOD E 200D with gooD = {z ~ Uzl2((z) S t),
with U2 pre-defined uniform distribution. For now, we can ensure that 2DD n 2OOD_ 0.
However, 21D n 2000 @ does not imply G(Z n G(ZOOD) = 0. i... C1 is not satisfied given arbitrary G(-). To this end, we suggest further ensuring the generator to be a distance- pereenerreserinng function [83], which is sufficient condition to ensure that the disjoint property can transformed
from the latent space into the data space. For distance preservation, we suggest regularization term for the generator regularizing. which given by
Eu xuzz [d(z), 22) d(G(z)), G(zz))
(Zog(Z1,
[Z1, Z2: G)
(8)
v EEzz xitz [d(z), 22) ) E2g xlz [d] (((()) G(Zz))]
where d(z1, Z2) is the centralized distance given by ||z Zg 1|| E |zz z zell2- Overall, Eq. (8) is the correlation for the distances between data pairs measured in the latent and the data space, stating that the closer distances in the latent space should indicate the closer distances in the data space.
In summary, to fulfill CI, we need to specify the regularizing procedure for the generator, overall summarized by the following two steps. First, we regularize the generator with the regularization in Eq. (8) to ensure the sestance-preservin property. Then, we specify two disjoint regions in the latent space following Eqs. (6)-(7), of which the data after passing through the generator are taken as auxiliary ID and auxiliary OOD data, respectively. We also summarize the algorithm details in crafting the auxiliary OOD detection task in Appendix D.
4.2 Applying the Auxiliary Task
We have discussed general strategy to construct the auxiliary OOD detection task, which can be learned by the predictor following an outlier exposure-based learning objective, similar to Eq. (2). Now, we fulfill C2, transferring model capability from the auxiliary task real OOD detection.
In general. C2 states that we should align distributions between the auxiliary and the real ID data such that the auxiliary OOD data can benefit real OOD detection, thus overcoming the negative effects of mistaken OOD generation. Following Tang et al. [59], we align the auxiliary and the real ID distribution via supervised contrastive learning [29], pulling together data belonging to the same class in the embedding space meanwhile pushing apart data from different classes. Accordingly, we suggest the alignment loss for the auxiliary ID data ÅID. following,
explo(<m)) Φ(x1))
Fatign (&DD Iw:: Φ) - -log
Σ X XEENDD
(9)
ExBEM, exp ( x(xx)))
NG is a set of data of size drawn from the real IDD distribution, namely, Mi = (xio (x víD) ~ Phy, for i e (I....,)) (]). Furthermore, Nr' is subset of N₁ with ID data that below to the label of

---

Accordingly, based namely, NV [> (xiD= VíD) PPxyy and vin = .DD for i {1..... (.). on Theorem I, we can ensure that learning from the auxiliary OOD detection task, i.e. small Eq. (5), can benefit real OOD detection, freeing from the mistaken OOD generation.
4.3 Overall Algorithm
We further summarize the training and the inferring procedure for the predictor of ATOL, given the crafted auxiliary ID and the auxiliary OOD distributions that satisfy CI via Eqs. (6)-(7). The pseudo codes are further summarized in Appendix due to the space limit.
Training Procedure. The overall learning objective consists of the real task learning, the auxiliary task learning, and the distribution alignment, which is given by:
(a) real task learning
(b) auxiliary task learning
min h=poo
E Ep
(XI, MID: h)] + Egg, [fcu MiD; h)] + AlBaggr (oB(XO0D: h)]
(10)
aEgg, [Katign (XI, ŌID: Φ)1.
distribution alignment
where À 2 O are the trade-off parameters. By finding the predictor that leads to the minimum of the Eq. (10), our ATOL can ensure the improved performance of real OOD detection. Note that, Eq. (10) can be realized in a stochastic manner, suitable for deep model training.
Inferring Procedure. We adopt the MaxLogit scoring [17) in OOD detection. Given test input x, the MaxLogit score is given by:
SMI. (x; h) = max (x),
(11)
where ha(-) denotes the k-th logit output. In general, the MaxLogit scoring is better than other commonly-used scoring functions, such as maximum softmax prediction [15], when facing large semantic space. Therefore, we choose MaxLogit instead of MSP for OOD scoring in realization.
5
Experiments
This section conducts extensive experiments for ATOL in OOD detection. In Section 5.1, we describe the experiment setup. In Section 5.2, demonstrate the main results of our method against the data generation- based counterparts on both the CIFAR [30] and the ImageNet [8] benchmarks. In Section 5.3, further conduct ablation studies to comprehensively analyze method.
5.1 Setup
Backbones setups. For the CIFAR benchmarks, we employ the WRN-40-2 [78] as the backbone model. Following [36], models have been trained for 200 epochs via empirical risk minimization. with batch size 64, momentum 0.9, and initial learning rate 0.1. The learning rate is divided by 10 after 100 and 150 epochs. For the ImageNet, we employ pre-trained ResNet-50 [13] on ImageNet, downloaded from the PyTorch official repository.
Generators setups. For the CIFAR benchmarks, following [31], adopt the generator from the Deep Convolutional (DC) GAN [51]). Following the vanilla generation objective, DCGAN has been trained the ID data, where the batch size is 64 and the initial learning rate 0.0002. For the ImageNet benchmark, we adopt the generator from the BigGAN [2] model, designed for scaling generation of high nigh-resolution images, where the pre-trained model biggan-256 can be downloaded from the TensorFlow hub. Note that our method can work on various generators except those mentioned above, even with a random- ceridoo--ppammtttricceee one, later shown Section 5.3.
Baseline methods. We compare our ATOL with advanced data generation based methods in OOD detection, including (1) BoundaryGAN [31], (2) ConfGAN [55], (3) ManifoldGAN [62], (4) G2D [50] (5) CMG [67]. For a fair comparison, all the methods use the same generator and pre-trained backbone without regularizing with outlier data.
Auxiliary task setups. Hyper-parameters are chosen based on the OOD detection performance on validation datasets. In latent space, the auxiliary ID distribution is the high density region of the
7

---

Table I: Comparison in OOD detection on the CIFAR and ImageNet benchmarks. (or indicates smaller (or larger) values are preferred; bold font indicates the best results in a column.
CIFAR-10
CIFAR-100
ImageNet
Methods
FPR95 4
AUROC t
FPR95
AUROC
FPR95 4
AUROC
BoundaryGAN [31]
55.60
86.46
76.72
75.79
85,48
66.78
ConfGAN [55]
31.57
93.01
74.86
77.67
74.88
77.03
ManifoldGAN [62]
26.68
94.09
73.54
77.40
72.50
77.73
G2D [50]
31.83
91.74
70.73
79.03
74.93
77.16
CMG [67]
39.83
92.83
79.60
77.51
72.95
77.63
ATOL (ours)
14.66
97.05
55.22
87.24
51.80
85.82
Mixture of Gaussian (MoG), Each sub-Gaussian in the MoG has the mean (f1,, ................................ Mm and the same covariance matrix a I, where m is the dimension latent space, μ4; is randomly selected from the set {-H, jr), and is the identity matrix. The auxiliary OOD distribution is the uniform distribution except for high MoG density region, where each dimension has the same space size u. For the CIFAR benchmarks, the value of is set to 1, is 5, is 0.1, and is 8. For the ImageNet benchmarks, the value of set to 1, lr is 5, a is 0.8, and u is 8.
Training details. For the CIFAR benchmarks, ATOL is run for 10 epochs and uses SGD with an initial learning rate 0.01 and the cosine decay [37]. The batch size 64 for real ID cases, 64 for auxiliary ID cases, and 256 for auxiliary OOD cases. For the ImageNet benchmarks, ATOL is run for 10 epochs using SGD with an initial learning rate 0.0003 and momentum 0.9. The learning rate is decayed by factor of 10 at 30%, 60%, and 90% of the training steps. Furthermore, the batch size is fixed to 32 for real ID cases, 32 for auxiliary ID cases, and 128 for auxiliary OOD cases.
Evaluation metries. The OOD detection performance of a detection model is evaluated via two representative metrics, which both threshold-independennt [7]: the false positive rate of OOD data when the true positive rate of ID data is at 95% (FPR95); and the area under the receiver operating characteristic curve (AUROC), which can be viewed as the probability of the IDD case having greater score than that of the OOD case.
5.2 Main Results
We begin with our main experiments on the CIFAR and ImageNet benchmarks. Model performance is tested on commonly-used OOD datasets. For the CIFAR cases, we employed Texture [6], SVHN [47], Places365 [84], LSUN-Crop [76), L.SUN Res-ee [76], and iSUN [74]. For the ImageNet case, we employed iNaturalist [18]. SUN [74]. Places365 [84], and Texture [6]. In Table 1, report the average performance (i.e.. FPR95 and AUROC) regarding the OOD datasets mentioned above. Please refer to Tables 17-18 and 19 in Appendix F.1 for the detailed reports.
CIFAR benchmarks. Overall, our ATOL can lead to effective OOD detection, which generally demonstrates better results than the rivals by a large margin. In particular, compared with the advanced data generation- based method, e.g., ManifoldGAN [62], ATOL significantly improves the performance in OOD detection, revealing 12.02% and 2.96% average improvements w.r.t. FPR95 and AUROC on the CIFAR-10 dataset and 18.32% and 9. 9.84% of the average improvements on the CIFAR-100 dataset. This highlights the superiority of our novel data generation strategy, relieving mistaken OOD generation as in previous methods.
ImageNet benchmark. We evaluate large-scale OOD detection task based on ImageNet dataset. Compared to the CIFAR benchmarks above, the ImageNet task more challenging due the large semantic space and large amount of training data. As we can see, previous data generation- based methods reveal poor performance due to the increased difficulty in searching for the proper OOD-like data in ImageNet, indicating the critical mistaken OOD generation remains (cf., Appendix F.9). In contrast, our ATOL can alleviate this issue even for large-scale datasets, e.g., ImageNet, thus leading to superior results. In particular, our ATOL outperforms the best baseline ManifoldGAN [62] by 20. 70% and 8.09% average improvement w.r.t. FPR95 and AUROC, which demonstrates the advantages of our ATOL to relieve mistaken OOD generation and benefit to real OOD detection.

---

Table 2: Effectiveness of auxiliary task crafting and learning scheme of ATOL. + (or 1) indicates smaller (or larger) values are preferred; bold font indicates the best results in a row
Ablation
ATOL w/o r
ATOL w/o Euz
ATOL w/o Ealign
ATOL
FPR95 + AUROC f
87.64
41.13
22.29
14.54
CIFAR-I0
59.32
90.38
94.08
97.01
FPR95
92.90
82.80
74.20
55.06
CIFAR-100
AUROC f
51.34
73.95
80.56
87.26
Table Performance comparisons with different setups of the generator on CIFAR benchmarks; (or 1) indicates smaller (or larger) values are preferred; a bold font indicates the best results in a row.
Generators
DCGAN
Rand- Mond-DCGAN
StyleGAN
BigGAN
FPR95 L AUROC f
14.66
20.78
13.65
8.61
CIFAR-10
97.05
95.57
96.63
97.90
FPR95
55.52
65.69
36.72 42.62
CIFAR 1OO
AUROC 1
86.21
83.26
89.17
91.33
5.3
Ablation Study
To further demonstrate the effectiveness of our ATOI,, conduct extensive ablation studies to verify the contributions of each proposed component.
Auxiliary task crafting schemes. In Section 4.1, we introduce the realization of the auxiliary OOD detection task in a tractable way. Here, we study two components generation: the disjoint supports for ID and OOD distributions in the latent space and the distance- preenncviggee generator. In particular, we compare ATOL with two variants: 1) ATOL w/o T. where the ID and OOD distributions may overlap in latent space without separation via the threshold 2) ATOL w/o legg where we directly use the generator to generate the auxiliary data without further regularization. As we can see in Table 2, the overlap between the auxiliary ID and the auxiliary OOD distributions in latent space will lead to the failure of the auxiliary OOD detection task, demonstrating the catastrophic performance (e.g., 92.20% FPR95 and 51 ..4% AUROC CIFAR-100). Moreover, without the regularized generator as a distance- preserving function, the ambiguity of generated data will compromise the performance of ATOL, which verifies the effectiveness of generation scheme.
Auxiliary task learning schemes. In Section 4.2, we propose ID distribution alignment to transfer model capability from auxiliary OOD detection task to real OOD detection. In Table 2, conduct experiments on variant, namely, ATOL w/o faligns where the predictor directly learns from the auxiliary task without further alignment. Accordingly, ATOL w/o Ealign can reveal better results than ATOL w/o freg. with 18.84% and 3.70% further improvement w.r.t. FPR95 and AUROC on CIFAR-10 10 and 8.60% and 6.61% CIFAR-100. However, the predictor learned the auxiliary task cannot fit into the real OOD detection task. Thus, we further align the distributions between the auxiliary and real ID data the transformed space, significantly improving the OOD detection performance with 7.75% on CIFAR-10 and 19.14% on CIFAR-100 w.r.t. FPR95.
Generator Setups. On CIFAR benchmarks, ID data in our ATOL. We further investigate whether ATOL remains effective when using generators with other setups, summarized in Table 3. First. find that even with a random- caaddo-pprrmettrrizeee DCGAN generator, i.c., Rand-DCGAN our ATOL can still yield reliable performance surpassing its advanced counterparts. Such observation suggests that method requires relatively low training costs, a property not shared by previous methods. Second, we adopt more advanced generators, ie., BigGAN [2] and StyleGAN-v2 [27]. As
the generator of the DCGAN model fitting the
see, ATOL can benefit from better generators, leading to large improvement 18. 80% and 5.12% w.r.t FPR95 and AUROC on CIFAR-100 over the DCGAN case. Furthermore, even with complicated generators, our method demonstrates promising performance improvement over other methods with acceptable computational resources, Please refer the Appendix E.8 and Appendix F.7 for more details.

---

Quantitative analysis on computation. Data generation-
based methods are relatively expensive in computation, so the consumed training time is also of our interest. In Table 4, we compare the time costs of training for the con- sidered data ceneration-based methods. Since our method does not need a complex generator training procedure during the OOD learning step, we can largely alleviate the expensive cost of computation inherited from the data generation- based approaches. Overall, our method quires minimal training costs compared to the advanced counterparts, which further reveals the efficiency of our method. Therefore, our method can not only lead to im- proved performance over previous methods, but it also requires fewer computation resources.
Table 4: Computational comparison with other counterparts. We report the per- epoch training time (measured by sec- onds) on CIFAR benchmarks.
Methods
Training time (s)
BoundaryGAA
106.98 + 7.82
ConfGAN
111.37 + 5.72
ManifoldGAN
159.23 + 9.23
G2D
73.29 4.09
CMG
254.22 +: 2.42
ATOL
57.22 = 2.28
6 Conclusion
Data generation-based methods in OOD detection are a promising way to make the predictor learn to discern ID and OOD data without real OOD data. However, mistaken OOD generation significantly challenges their performance. To this end, we propose powerful data generation-base method by the proposed auxiliary OOD detection task, largely relieving mistaken OOD generation. Extensive experiments show that our method can notably improve detection performance compared to advanced counterparts. Overall, our method can benefit applications for data generation-bai based OOD detection, and we intend further research to extend our method beyond image classification. We also hope our work can draw more attention from the community toward data generation-basee methods, and we anticipate OOD detection, such as wild OOD detection [28].
learning
auxiliary task-based learning scheme can benefit more directions in
Acknowledgments and Disclosure of Funding
HTZ, QZW and BH were supported by the NSFC Young Scientists Fund No. 62006202, NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A15150) 1652, HKBU Faculty Niche Research Areas No. RC-FNRA-IGG22-24SSI004 and HKBU CSD Departmental Incentive Scheme. XBX and TLLL were partially supported by the following Australian Research Council projects: F8220100318, DP220102121, LP220100527, LP220200949, and IC190100031, FL was supported by Australian Research Council (ARC) under Award No. DP230101540, and by NSF and CSIRO Responsible AI Program under Award No.
2303037.
References
[1] Abhijit Bendale and Terrance Boult. Towards open set deep networks. In CVPR, 2016.
[2] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
[3] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. ATOM: robustifying out-of- distribution detection using outlier mining. In ECML, 2021.
[4] Ling-Hao Chen, He Li, Wanyuan Zhang, Jianbin Huang, Xiaoke Ma, Jiangtao Cui, Ning Li, and Jaesoo Yoo. Anomman: Detect anomaly multi-view attributed networks. arXiv preprint arXiv:2201.02822 2022.
[5] Ling-Hap Hao Chen, Jiawei Zhang, Yewen Li, Yiren Pang. Xiaobo Xia, and Tongliang Liu. Human- mac: Masked motion completion for human motion prediction. In ICCV, 2023.
[6] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
10

---

[7] Jesse Davis and Mark Goadrich. The relationship between precision-recal and ROC curves. In ICML, 2006.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[9] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In ICLR, 2023.
[10] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. VOS: learning what you don't 't know by virtual outlier synthesis. In ICLR, 2022.
[11] Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribution detection learnable? In NeurlPS, 2022.
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[14] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In CVPR, 2019.
[15] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples neural networks. In ICLR, 2017.
[16] Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
[17] Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real- world settings. In ICML, 2022.
[18] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
[19] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[20] Rui Huang and Yixuan Li. MOS: towards scaling outto--disttibution detection for large semantic space. In CVPR, 2021.
[21] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In NeurlPS, 2021.
[22] Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. In NeurIPS, 2021.
[23] Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-o--distribution examples via augmenting content and style. In ICLR, 2023.
[24] Zhuo Huang, Miaoxi Zhu, Xiaobo Xia, Li Shen, Jun Yu, Chen Gong, Bo Han, Bo Du, and Tongliang Liu. Robust generalization against photon-limited corruptions via worst-case sharp- ness minimization. In CVPR, 2023.
[25] Conor Igoe, Youngseog Chung, Ian Char, and Jeff Schneider. How useful are gradients for ood detection really? arXiv preprint arXiv:2205.10433 2022.
[26] Taewon Jeong and Heeyoung Kim. OOD-MAMI: meta-learning for few-shot out-of-distribution detection and classification. In NewrIPS, 2020.

---

[27] Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020.
[28] Julian Katz-Samuels, Julia B. Nakhleh, Robert D. Nowak, and Yixuan Li. Training OOD detectors in their natural habitats. In ICML, 2022.
[29] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Neur/PS, 2020.
[30] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009.
[31] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibratedd classifiers for detecting out-of-distriiution samples. In ICLR, 2018.
[32] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting diitt--diitiitttionn samples and adversarial attacks. In NeurIPS, 2018.
[33] Yewen Li, Chaojic Wang. Xiaobo Xia, Tongliang Liu, Xin Miao, and Bo An. distri-disttituutin detection with an adaptive likelihood ratio informative hierarchical VAE. In NeurlPS, 2022.
[34] Yi Li and Nuno Vasconcelos. Background data resampling for outlier-aware classification. In CVPR, 2020.
[35] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distributton image detection in neural networks. In ICLR, 2018.
[36] Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-basse based outto--distribution detection. NeurlPS, 2020.
[37] Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR, 2017.
[38] Yadan Luo, Zijian Wang, Huang, and Mahsa Baktashmotlagh. Progressive graph learning for open-set domain adaptation. In ICML, 2020.
[39] Yadan Luo, Zijian Wang, Zhuoxiao Chen, Zi Huang, and Mahsa Baktashmotlagh. Source-free progressive graph learning for open-set domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9): 1ee:::1220-12255 2023.
[40] Zhanyu Ma and Arne Leijon. Beta mixture models and the application to image classification. In ICIP, 2009.
[41] Akshay Mehra, Bhavya Kailkhura, Pin-Yu Chen, and Jihun Hamm. On certifying and improving generalization to unseen domains. arXiv preprint arXiv:2206. 12364, 2022.
[42] Yifei Ming. Ying Fan, and Yixuan Li. Poem: Out-of-distribution detection with posterior sampling. ICML, 2022.
[43] Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-of- distribution detection. In AAAI, 2022.
[44] Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embed- dings for nut-of-distribution detection? In ICLR, 2023.
[45] Sina Mohseni, Mandar Pitale, J. S. Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable distriiiittiitnoonn detection. In AAAI, 2020.
[46] Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detection. In AAAI, 2022.
[47] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
12

---

[48] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, 2015.
[49] Aristotelis-Angells Papadopoulos, Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang. Outlier exposure with confidence control for -isttr--ittiittionnn detection. Neurocomputing. 441:138- 150, 2021.
[50] Masoud PourReza, Bahram Mohammadi, Mostafa Khaki, Samir Bouindour, Hichem Snoussi, and Mohammad Sabokrou. G2D: generate detect anomaly. In WACV, 2021.
[51] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
[52] Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set consistency regulariza- tion for semi-supervised learning with outliers. arXiv preprint arXiv:2/05.14148 2021.
[53] Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distriibtion examples with gram matrices. In JCML, 2020.
[54] Vikash Sehwag, Mung Chiang, and Prateek Mittal. SSD: A unified framework for self- supervised outlier detection. In ICLR, 2021.
[55] Kumar Sricharan and Ashok Srivastava. Building robust classifiers through generation of confident out of distribution examples. arXiv preprint arXhv:1812..02239, 2018.
[56] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In NeurIPS, 2021.
[57] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribttion detection with deep nearest neighbors. In ICML, 2022.
[58] Jihoon Tack, Sangwoo Mo. Jongheon Jeong, and Jinwoo Shin. CSI: novelty detection via contrastive learning on distributionally shifted instances. In NeurIPS, 2020.
[59] Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu, Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In ICML, 2022.
[60] Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In ICLR, 2023.
[61] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, ((86):2579-2605 2008
[62] Sachin Vernekar, Ashish Gaurav, Vahdat Abdelzad, Taylor Denouden, Rick Salay, and Krzysztof Czarnecki. Out-of-distribution detection in classifiers via generation. arXiv preprint 1rXiv:1910.04241, 2019.
[63] Martin Wainwright. High-dimensional statistics: A non-asymptotic viewpoint. Cambridge University Press, 2019.
[64] Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In CVPR, 2022.
[65] Haotao Wang, Aston Zhang, Yi Zhu, Shuai Zheng, Mu Li, Alex J. Smola, and Zhangyang Wang. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. In ICML, 2022.
[66] Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, et al. On the robustmess of chatgpt: An adversarial and out-of-distribution perspective. arXiv preprint arXiv:2302.12090 2023.
[67] Mengyu Wang, Yijia Shao, Haowei Lin, Wenpeng Hu, and Bing Liu. CMG: class-mixed generation approach to out-of-distribution detection. In ECML, 2022.
13

---

[68] Qizhou Wang, Feng Liu, Yonggang Zhang, Jing Zhang, Chen Gong, Tongliang Liu, and Bo Han. Watermarking for -ustt--distiiuttio detection. NeurIPS, 2022.
[69] Qizhou Wang, Zhen Fang, Yonggang Zhang, Feng Liu, Yixuan Li, and Bo Han. Learning to augment distributions for out-of-distribution detection. In NeurIPS, 2023.
[70] Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Mareus Kalander, Tongliang Liu, Jianye Hao, and Bo Han. Out-of-distribution detection with implicit data transformation. In ICLR, 2023.
[71] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. ICML, 2022.
[72] Xiaobo Xia, Wenhao Yang, Jie Ren, Yewen Li, Yibing Zhan, Bo Han, and Tongliang Liu. Pluralistic image completion with gaussian mixture models. In NeurIPS, 2022.
[73] Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. ICLR, 2023.
[74] Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1544.000
2015.
[75] Jingkang Yang, Haoqi Wang. Litong Feng. Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distribution detection. In ICCV, 2021.
[76] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: /506.03365, 2015.
[77] Alireza Zaeemzadeh, Niccolò Bisagno, Zeno Sambugaro, Nicola Conei, Nazanin Rahnavard, and Mubarak Shah. iittof-diistrbbutin detection using union of 1-dimensional subspaces. In CVPR, 2021.
[78] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[79] Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments. In WACV, 2023.
[80] Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Ryan Luley, Yiran Chen, and Hai Li. Sio: Synthetic in-distribution data benefits out-of-distribution detection. arXiv preprint arXiv:2303. 145311, 2023.
[81] Yongqi Zhang, Zhanke Zhou, Quanming Yao, and Yong Li. Efficient typer-parament search for knowledge graph embedding. ACL, 2022.
[82] Yongqi Zhang, Zhanke Zhou, Quanming Yao, Xiaowen Chu, and Bo Han. Adaprop: Learning adaptive propagation for graph neural network based knowledge graph reasoning. In KDD, 2023.
[83] Xingjian Zhen, Zihang Meng, Rudrasis Chakraborty, and Vikas Singh. On the versatile uses of partial distance correlation in deep learning. ECCV. 2022.
[84] Bolei Zhou, Ägata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6): 2018.
[85] Zhanke Zhou, Jiangchao Yao, Jiaxu Liu, Xiawei Guo, Quanming Yao. Li He, Liang Wang, Bo Zheng, and Bo Han. Combating bilateral edge noise for robust link prediction. In NeurIPS, 2023.
14


Reviews:
[{"review_id": "INiONdOWKs", "reviewer": "Reviewer_Q6fc", "summary": "One of the techniques for detecting OOD instances is to train a model on OOD data. However, that task is not easy due to difficulty inherent with collecting such OOD data. Rather than collecting such data, this paper proposes instead to generate it, and to train an auxiliary task to improve the OOD detection capabilities of deep learning networks. The proposed approach is called ATOL, for Auxiliary Task-based OOD Learning, and proposes to address one fundamental flaw in existing data-generation based detection methods: the collection of OOD instances from ID data that can mistakenly be labeled as OOD data while being in reality ID.", "questions": "Based on the limitations I raised above, I would suggest the authors to perform a comparative study with some of the distance-based OOD detection approaches like CIDER. Additionally, evaluating the performance of the MoGs they used could help validate further the effectiveness of their approach.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper addresses an interesting problem that I think is quite overlooked in the research community. Indeed, many studies focus on collecting OOD instances without necessarily evaluating the possible side effects of the collected data. This paper identified that instances that are collected and labeled as OOD instances can in fact be ID instances, which could lead to poor generalization performance on ID data, and poor OOD detection capabilities. The approach the authors devised to address the problem is sound and quite intuitive, and the evaluation is quite strong.  ", "weaknesses": "Although this paper addresses the OOD detection problem from a data-generation perspective, I would have very much liked to see how their approach fair with other techniques like distance-based OOD detection methods. Some interesting distance-based OOD detection mechanisms have been proposed in the recent past. For instance, CIDER [1] has achieved state-of-the-art performance OOD detection capabilities that shouldn't be overlooked by the researchers approaching OOD detection from a data-generation perspective.  This would help educate mainstream readers more on exactly what techniques to pursue to robustify their models against OOD samples. \n\nA more fundamental limitation of this study is the fact that it heavily relies on a mixture of Gaussian to decide on what latents to consider as OOD and which ones to consider as ID. As MoGs can be sensitive to outliers, have a rather limited expressive power, their accuracy needs to be presented in the paper to showcase their effectiveness in helping collect the data to train the auxiliary task on.    \n\n[1]: How To Exploit Hyperspherical Embeddings For OOD detection? https://arxiv.org/pdf/2203.04450.pdf\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comments and generous supports! Please find our responses below.\n\n> Q1. Although this paper addresses the OOD detection problem from a data-generation perspective, I would have very much liked to see how their approach fair with other techniques like distance-based OOD detection methods, e.g., CIDER.\n\nA1. In the main context, we mainly compare our ATOL with other data generation-based approaches. However, we also conduct more evaluations in Appendix E (e.g., Tables 15 to 17), comparing with regularization-based and post-hoc approaches. We include several representative distance-based approaches in these experiments, such as KNN [1] and CSI [2]. As shown in Appendix E, Our ATOL outperforms these approaches, revealing our superiority.\n\nFollowing your kind suggestions, we further compare with CIDER and ViM [3] (both distance-based methods) on CIFAR-10/100 benchmarks, summarizing the results in the following tables. **Comparing with the SOTA and vital baseline CIDER, our ATOL can still reveal superior results**. We will add the related discussions for CIDER and other distance-based methods to Appendix E in our revision.\n\n\n\nCIFAR-10\n|   Method   |  SVHN |       | LSUN-Crop |       | LSUN-Resize |       |  iSUN |       | Texture |       | Places365 |       | Average |       |\n|:----------:|:-----:|:-----:|:---------:|:-----:|:-----------:|:-----:|:-----:|:-----:|:-------:|:-----:|:---------:|:-----:|:-------:|:-----:|\n|            | FPR95 | AUROC |   FPR95   | AUROC |    FPR95    | AUROC | FPR95 | AUROC |  FPR95  | AUROC |   FPR95   | AUROC |  FPR95  | AUROC |\n|    ViM    | 56.97 | 89.77 |   49.96   | 91.43 |    63.54    | 88.15 | 62.20 | 88.47 |  45.20  | 91.76 |   47.86   | 91.45 |  54.29  | 90.17 |\n|    CIDER    | **5.86** | **98.36** |   7.35   | 98.50 |    47.58    | 93.64 | 47.15 | 93.60 |  28.04  | 94.79 |   41.10   | 91.03 |  29.51   | 94.99 |\n| Our Method | 12.75 | 96.92 |   **4.60**   | **98.92** |    **0.65**    | **99.78** | **0.55** | **99.83** |  **10.25**  | **97.12** |   **22.85**   | **94.80** |  **8.61**  | **97.90** |\n\nCIFAR-100\n|   Method   |  SVHN |       | LSUN-Crop |       | LSUN-Resize |       |  iSUN |       | Texture |       | Places365 |       | Average |       |\n|:----------:|:-----:|:-----:|:---------:|:-----:|:-----------:|:-----:|:-----:|:-----:|:-------:|:-----:|:---------:|:-----:|:-------:|:-----:|\n|            | FPR95 | AUROC |   FPR95   | AUROC |    FPR95    | AUROC | FPR95 | AUROC |  FPR95  | AUROC |   FPR95   | AUROC |  FPR95  | AUROC |\n|    ViM    | 72.32 | 82.92 |   74.03   | 81.57 |    84.89    | 77.03 | 84.15 | 76.69 |  33.51  | 91.72 |   64.17   | 79.57 |  68.78  | 81.58 |\n|    CIDER    | **52.21** | 88.44 |   46.88   | 90.18 |    52.23    | 89.89 | 47.57 | 89.91 |  84.67  | 70.62 |   84.67   | 71.82 |  61.37  | 83.48 |\n| Our Method | 54.65 | **89.69** |   **43.95**   | **91.27** |    **7.80**    | **98.57** | **9.60** | **98.13** |  **37.45**  | **89.51** |   **66.90** | **80.82** |  **36.72**  | **91.33** |\n\n\n\n> Q2. A more fundamental limitation of this study is the fact that it heavily relies on a mixture of Gaussian to decide on what latents to consider as OOD and which ones to consider as ID. As MoGs can be sensitive to outliers, have a rather limited expressive power, their accuracy needs to be presented in the paper to showcase their effectiveness in helping collect the data to train the auxiliary task on.\n\nA2. Sincerely apologize for your confusion. We would like to answer your questions as follows. The related discussion will be added in our revision, especially for Section 3.\n\n- **Reliance on MoG Assumption.** MoG just provides a simple way to generate data, yet the key point is to ensure that auxiliary ID/OOD data should have the disjoint support (i.e., Condition 1). Therefore, if Condition 1 is satisfied properly, other noise distributions, such as the beta mixture models and the uniform distribution, can also be used. We will explore different choices of noise distributions in the future. \n\n- **Fitting the MoG distribution.** Our ATOL is different from previous data generation-based methods in that we do not require generated data to be reliable in the data space. ATOL does not involve fitting the MoG to real ID data, where the parameters can be pre-defined and fixed. Therefore, we do not consider overfitting and accuracy of MoG in our paper.\n\nAlthough generated data are unreliable in the data space (e.g., auxiliary ID data may differ from its real counterparts), they can still benefit the predictor when Conditions 1-2 and Eq. 5 are satisfied. Heuristically, these conditions and constraints make the predictor take no difference between auxiliary and real ID data. Therefore, auxiliary OOD data are beneficial from the predictor perspective since they have the disjoint support over the real ID data. \n\n\n\n[1]: Yiyou Sun, et al. \"Out-of-distribution detection with deep nearest neighbors.\" ICML 2022.\n\n[2]: Kimin Lee, et al. \"A simple unified framework for detecting out-of-distribution samples and adversarial attacks.\" NeurIPS 2018.\n\n[3]: Haoqi Wang, et al. \"Vim: Out-of-distribution with virtual-logit matching.\" CVPR 2022."}, {"review_id": "9mNMkFjNpD", "reviewer": "Reviewer_3oBY", "summary": "The paper tries to overcome the impact of directly applying incorrect OOD data on the OOD model through auxiliary tasks, thereby improving the performance of OOD tasks. The theoretical part is hard to follow, and the experimental part proves the effectiveness of the theory.", "questions": "There are too many Mathematical notations. It is suggested to sort out a table to make it clearer.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper introduces an auxiliary OOD detection task to combat mistaken OOD generation.\n2. The proposed method requires a small additional calculation cost.\n3. Experiments show the effectiveness of the proposed method.", "weaknesses": "1. The proof of C2 is not clear enough to allow me to clearly understand why achieving C2 can better utilize OOD data. If the model is strong enough or fully trained, it can still confuse incorrect OOD data even if formula 5 is met. Therefore, I judge that the training steps of the algorithm should not be too many and the model should not be too large. Does the author have an explanation for this aspect?\n\n2. There is a writing error in part b of Formula 10.\n\n3. Many aspects of the experiment followed the settings of reference [31], but why not include them in the comparison?\n\n4. The paper is really hard to follow, please polish the paper carefully.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comments and generous supports! Please find our responses below.\n\n> Q1. The proof of C2 is not clear enough to allow me to clearly understand why achieving C2 can better utilize OOD data. \n\nA1. Thank you for your valuable concern. We would like to answer your questions as follows. The related discussions will be added in our revision, especially for Section 3. \n\n- **How to interpret Condition 2?** Measured in the embedding space of the predictor, Condition 2 states that auxiliary ID data approximately follow the same distribution as that of the real ID data. Therefore, a proper predictor should be trained to satisfy such a condition, motivating Eq. 9 in our realization. \n\n- **Why is Condition 2 important?** Auxiliary ID/OOD data can arbitrarily differ from real ID/OOD data in the data space, i.e., auxiliary data are unreliable OOD sources. However, if Condition 2 is satisfied, the model \"believes\" the auxiliary ID data and the real ID data are the same. Then, the auxiliary OOD data will have the disjoint support over the real ID data and thus achieve reliability w.r.t. the predictor (cf., Proposition 1). Therefore, Condition 2 is critical for our ATOL.\n\n\n> Q2. If the model is strong enough or fully trained, it can still confuse incorrect OOD data even if formula 5 is met. Therefore, I judge that the training steps of the algorithm should not be too many and the model should not be too large. Does the author have an explanation for this aspect?\n\nA2. Many thanks for your question. Conditions 1-2 ensure that unreliable sources will not mislead our predictor in OOD detection. In fact, if the model is stronger, we can better satisfy Condition 2 and Eq. 5. Then, according to Proposition 1, auxiliary OOD data are more likely to be reliable and will not confuse our predictor. The following experimental results on CIFAR can support our above claims, where using a more complex model (i.e., DenseNet-121) can lead to better performance in OOD detection. We will add the related discussion below Proposition 1.\n\nCIFAR-10\n|Method|SVHN||LSUN-Crop||LSUN-Resize||iSUN||Texture||Places365|| Average||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|\n|DenseNet-121|18.05|96.27|1.45|99.58|4.35|99.88|4.45|98.86|20.90|95.70|25.85|94.39|12.51|97.28|\n|WRN-40-2|20.60|96.03|1.48|99.59|5.20|98.78|5.00|98.76|26.05|95.03|27.55|94.33|14.31|97.09|\n\n\nCIFAR-100\n|Method|SVHN||LSUN-Crop||LSUN-Resize||iSUN||Texture||Places365|| Average||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|\n|DenseNet-121|70.30|85.96|61.55|87.93|22.15|95.87|22.30|95.59|48.30|87.87|77.15|79.28|50.29|88.75|\n|WRN-40-2|70.85|84.70|13.45|97.52|51.85|90.12|55.80|89.02|63.10|83.37|75.30|78.86|55.06|87.26|\n\n\n> Q3. There is a writing error in part b of Formula 10.\n\nA3. Thank you for your kind correction. We will correct Eq. 10 in our revision.\n\n> Q4. Many aspects of the experiment followed the settings of reference [31], but why not include them in the comparison?\n\nA4. In our evaluation, we **have conducted** experiments for Free Energy (the method suggested in reference [31] of our paper). We will make it clearer in our revision.\n\n\n> Q5. The paper is really hard to follow, please polish the paper carefully.\n\nA5. Sincerely apologize for your confusion. Following your kind suggestion, we will summarize the key concepts and heuristics in our revision. Besides, we will add the notation table in Appendix and remarks for theories in Section 3, hoping they can help you better understand our paper. If you have any further suggestions, we look forward to discussion and are happy to answer any questions that might arise.\n\n> Q6. There are too many Mathematical notations. It is suggested to sort out a table to make it clearer.\n\nA6. Thank you for the kind suggestion. We summarize the adopted notations in the following tables. We will add the mathematical notations table in the Appendix.\n|Notation|Description|\n|:--:|:--:|\n|**Spaces**||\n|$\\mathcal{X}$ and $\\mathcal{Y}$|the data space and the label space \n|**Distributions**||\n|$\\mathcal{P}\\_{\\text{X,Y}}^{\\text{ID}}$ and $\\mathcal{P}\\_{\\text{X}}^{\\text{ID}}$|the joint and the marginal real ID distribution|\n|$\\mathcal{P}\\_{\\text{X}}^{\\text{OOD}}$|the marginal real OOD distribution|\n|$\\mathcal{G}\\_{\\text{X,Y}}^{\\text{ID}}$ and $\\mathcal{G}\\_{\\text{X}}^{\\text{ID}}$|the joint and the marginal auxiliary ID distribution|\n|$\\mathcal{G}\\_{\\text{X}}^{\\text{OOD}}$|the marginal auxiliary OOD distribution|\n|$\\mathcal{M}\\_{\\text{Z}}$|the specified MoG distribution|\n|$\\mathcal{U}\\_{\\text{Z}}$ |the specified uniform distribution|\n|**Data**||\n|$\\mathbf{x}\\_{\\text{ID}}$ and $y\\_{\\text{ID}}$ |the real ID data and label|\n|$\\hat{\\mathbf{x}}\\_{\\text{ID}}$ and $\\hat{y}\\_{\\text{ID}}$|the auxiliary ID data and label|\n|$\\hat{\\mathbf{x}}\\_{\\text{OOD}}$|the auxiliary OOD data|\n|$\\mathbf{z}\\_{\\text{ID}}$ and $\\mathbf{z}\\_{\\text{OOD}}$|the latent ID data and the latent OOD data|\n|$\\mathcal{Z}^{\\text{ID}}$ and $\\mathcal{Z}^{\\text{OOD}}$|the latent ID data sets and the latent OOD data sets|\n|**Models**||\n|$\\mathbf{h}$|the predictor: $\\mathbb{R}^{n} \\rightarrow \\mathbb{R}^c$|\n|$\\boldsymbol\\phi$ and $\\boldsymbol\\rho$|the feature extractor and the classifier|\n|$s(\\cdot;\\mathbf{h})$ |the scoring function: $\\mathbb{R}^{n}\\rightarrow \\mathbb{R}$|\n|$f_{\\beta}(\\cdot)$ | the OOD detector: $\\mathbb{R}^{n}\\rightarrow \\\\{\\text{ID}, \\text{OOD}\\\\}$, with threshold $\\beta$|\n|$G$ |the generator: $\\mathbb{R}^m\\rightarrow\\mathbb{R}^n$|\n|**Loss and Function**||\n|$\\ell_{\\text{CE}}$ and $\\ell_{\\text{OE}}$|ID loss and OOD loss|\n|$\\ell_{\\text{reg}}$|the generator regularization loss|\n|$\\ell_{\\text{align}}$|the alignment loss|\n|$\\boldsymbol{\\phi}'(\\cdot)$|the mapping function|\n|$\\mathcal{M}(\\cdot)$|the density function of MoG|\n"}, {"review_id": "ivGCvThRls", "reviewer": "Reviewer_eqWq", "summary": "The paper propose to fix the mistaken OOD generation issue in generative model based approach to out-of-distribution data detection, where the mistaken OOD generation means generated OOD data have semantics of ID data. To fix this issue, auxiliary task-based OOD learning (ATOL) is proposed, which is claimed to have the effect of satisfying two key conditions, i.e., auxiliary ID and auxiliary OOD data have disjoint supports in the input space, and auxiliary OOD data are reliable. To achieve this goal, ATOL adds auxiliary task learning loss and ID distribution alignment loss to the real task learning loss. The empirical study shows the non-trivial improvement of OOD detection performance when using ATOL. Ablation study further confirms the effectiveness of each loss in ATOL.", "questions": "1. What is M(z) in Equation 6 and 7? Is it density function of MoG?\n\n2. It is kind of misleading to make the ATOL result bold in Tab. 17 since it is not the best result in terms of AUROC. The best one should be CSI.\n\n3. What is the proportion in Fig. 1b and how is it computed? I tried to search for this information in the paper but it seems that there is no such information. I believe this information is quite important to motivate this paper. ", "limitations": "Limitations are not discussed in the main paper (please correct me if I am wrong about this). ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The motivation to fix the mistaken OOD generation makes sense to me as shown in Fig. 1. \n\nThe two new losses in the proposed ATOL are reasonable and are clearly presented.\n\nI have to say that I like the empirical studies in the paper since they are quite comprehensive and strong, though many interesting results are presented in the supplementary as a result of page limits. ", "weaknesses": "The drawback and strength of generative model based OOD detection and its comparison with other approaches like scoring or regularized training is not fully discussed in the paper, e.g., performance and efficiency. \n\nIn Table 17, the benefit of ATOL is not quite clear when compared with ReAct and CSI. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comments and generous supports! Please find our responses below.\n\n> Q1: The drawback and strength of generative model based OOD detection and its comparison with other approaches like scoring or regularized training is not fully discussed in the paper, e.g., performance and efficiency.\n\nA1: **Data Generation-based Methods vs. Post-hoc Methods**. Post-hoc OOD detection establishes the basis of OOD detection, where various scoring functions in post-hoc methods can be used to improve data generation-based methods. For example, In Appendix E.6, we test ATOL with different scoring strategies, demonstrating that proper scoring functions can lead to improved performance for data generation-based OOD detection. Therefore, post-hoc methods are typically orthogonal to data generation-based approaches. \n\n**Data Generation-based Methods vs. Regularization-based Methods.** Data generation-based OOD detection improves traditional regularization-based methods, in which we do not require real OOD data to fine-tune our predictor. However, due to mistaken OOD generation, previous generation-based methods can make mistakes. It motivates our ATOL to improve conventional data generation-based methods, leading to superior performance over both data generation-based and regularization-based methods. \n\n**Experimental Comparison.** In the main context, our primary focus is on comparing data generation-based approaches. However, we also conduct more extensive evaluations in Appendix E (e.g., Tables 15 to 17), comparing with regularization-based and post-hoc approaches. As we can see, Our ATOL outperforms these approaches, revealing our superiority. Moreover, we also compare the training time for a set of representative methods on CIFAR-100 (similar results on CIFAR-10), summarizing the results in the following table. As we can see, our method demonstrates promising performance improvement over other methods with acceptable computational resources. We will add the related discussion in the Appendix.\n\n\n| Methods | FPR95 | AUROC |Training Time / Epoch|\n| -------- | -------- | -------- | -------- |\n| CSI     | 80.08     | 85.23     | 98.88     |\n| LogitNorm     | 63.45     | 80.18     | **25.16**     |\n| VOS     | 75.41     | 78.20     | 38.97     |\n| NPOS     | 62.72     | 84.17     | 61.54     |\n| ATOL     | 55.22     | 87.24     | 58.33     |\n| ATOL-S     | 43.18    | 88.43     | 70.28     |\n| ATOL-B     | **36.72**     | **91.33**     | 65.33     |\n\n\n\n> Q2: In Table 17, the benefit of ATOL is not quite clear when compared with ReAct and CSI.\n\nA2: In Table 17, we conduct experiments on the ImageNet benchmark, one of the most challenging setups in OOD detection (due to its vast semantic space) [1,2]. Therefore, about $5.55\\%$ to $6.28\\%$ improvements of our ATOL over CSI and ReAct is relatively promising. How to improve data generation-based OOD detection for tasks with large semantic space remains an open question, which will motivate our future studies. \n\n\n\n> Q3: What is M(z) in Equation 6 and 7? Is it density function of MoG?\n\nA3: Yes, $\\mathcal{M}(\\mathbf{z})$ in Eqs. 6-7 represents the density function of MoG, namely, $\\mathcal{M}(\\mathbf{z}) = \\sum_{i=1}^c \\frac{1}{c} \\mathcal{N}(\\mathbf{z}|\\boldsymbol{\\mu}_i, \\boldsymbol{\\sigma}_i)$, with $\\boldsymbol{\\mu}_i$ the mean and\n$\\boldsymbol{\\sigma}_i$ the covariance for the $i$-th sub-Gaussian. We will make it clearer in our revision. \n\n> Q4: It is kind of misleading to make the ATOL result bold in Tab. 17 since it is not the best result in terms of AUROC. The best one should be CSI. \n\nA4. Sincerely apology for our mistakes. We will modify the related parts in our revision. \n\n> Q5: What is the proportion in Fig. 1b and how is it computed? I believe this information is quite important to motivate this paper.\n\nA5. We apologize for the missing description. The proportion in Figure 1(b) is the percentage of ID data mixed in the OOD data, reflecting the severity of wrong OOD data during training. We will describe more about the experimental setups for Figure 1 in the Appendix. \n\n> Q6: Limitations are not discussed in the main paper (please correct me if I am wrong about this).\n\nA6. Due to space limitations, our discussion about limitations is given in Appendix F. Here, we list two factors that may motivate our future works. First, our current realization of ATOL is relatively intricate, requiring training constraints on both the generator and the predictor. Further studies will explore more advanced conditions that can ease our realization and further reduce computing costs. Second, we observe that the diversity of generated data is closely related to the final performance (cf., Appendix D.6). However, in our current version, we do not consider diversity for the generator in either theories or algorithms, which will motivate our following exploration. \n\n[1]: Rui Huang, et al. \"MOS: towards scaling out-of-distribution detection for large semantic space.\" CVPR 2021.\n\n[2]: Dan Hendrycks, et al. \"Scaling out-of-distribution detection for real world settings.\" ICML 2022.\n"}, {"review_id": "nskaeBfkgK", "reviewer": "Reviewer_iqYB", "summary": "This paper contributes a new data-generation method to train a OOD detector. The paper proposes to first create an auxiliary task for a generator to generate in-distribution samples and OOD samples by using regions of disjoint latent space (equ 6-7). This can lead to disjoint support set by enforcing a distance preserving loss (equ 8). In order to transfer auxiliary task to real OOD detection, it suggests to use a contrastive learning to bring together generated in-dist samples and real in-dist samples. With these two settings, the OOD detector can learn from generated OOD samples. ", "questions": "$\\bullet$ Can the author change the term predictor to OOD detector? \n\n$\\bullet$ While it is a good idea to generate in-dist and OOD samples using high MoG density region, does it also make the task too easy? I imagine that high MoG region samples may look like in-dist samples, while the rest may look like noise. Does the predictor can perform better if it can learn from more difficult (harder to discern) generated in-dist and generated OOD samples?  \n\n$\\bullet$ Does this method consider the case when high MoG density region still have some OOD samples? \n\n$\\bullet$ The performance gain of the (OOD) predictor may be comparable to the generative model used for auxiliary task. Can the auxiliary generative model(s) be used for OOD detection? Can you provide its (or their) performance? If the generative model(s) performs worse, what could contribute to the advantage of the predictor (maybe the contrastive learning loss or maybe the auxiliary task)? Can you identify where the advantage comes from? ", "limitations": "The paper states there is a limitation discussion. Where is it? ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This discussion is complete and clear. The idea of crafting the auxiliary task using generative model(s) is promising in general. By dividing the samples into high and low density regions, this method bypasses the issue of mistaken OOD samples.  ", "weaknesses": "The major concern is the performance of the generator (or possibly I miss that piece of information). While section 3 is convincing, in practice (section 4) how to guarantee high MoG density region has high concentration of in-distribution samples. This may happen when a generator does not perform well. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comments and generous supports! Please find our responses below.\n\n> Q1. The major concern is the performance of the generator (or possibly I miss that piece of information). While section 3 is convincing, in practice (section 4) how to guarantee high MoG density region has high concentration of in-distribution samples. This may happen when a generator does not perform well.\n\nA1. Sincerely apologize for your confusion. We would like to address your concerns as follows. \n\n\n- **The performance of the generator does not matter.** Our ATOL is different from previous data generation-based methods in that we do not assume the generated data are reliable. Then, even with the unreliable data, they can still benefit OOD detection when it satisfies Conditions 1-2 and Eq. 5. Therefore, we do not need to care about the performance of the generator, making our proposal more attractive than previous works in data generation-based OOD detection. \n\n- **Auxiliary data can differ from real data in the input space.** Generally, any data distribution can benefit our predictor if they can be separated into two disjoint supports, playing the role of auxiliary ID and OOD data, respectively (i.e., Condition 1 should be satisfied). Therefore, if a high MoG density region does not correspond to ID data (which is often the case, as we demonstrated in Figures 4 to 9 in Appendix E.8), it can still benefit our predictor following Proposition 1. \n\nIn summary, the key motivation of our ATOL is that: **unreliable data (i.e., auxiliary ID and OOD data) in the input space can still benefit OOD detection, if we align distributions between auxiliary and real data w.r.t. the predictor (i.e., Condition 2).** \n\n\n\n> Q2. Can the author change the term predictor to OOD detector?\n\nA2. As a common practice [1,2], OOD detector is defined by both the predictor and the scoring function. In our paper, we emphasize our focus on the training procedure for the predictor, thus using \"predictor\" more commonly than \"OOD detector\". We will further consider the terminologies in our revision, trying to make descriptions clearer.\n\n> Q3. While it is a good idea to generate in-dist and OOD samples using high MoG density region, does it also make the task too easy? I imagine that high MoG region samples may look like in-dist samples, while the rest may look like noise. Does the predictor can perform better if it can learn from more difficult (harder to discern) generated in-dist and generated OOD samples?\n\nA3. Yes. Using hard samples in auxiliary data seems to be a promising way, which may boost detection performance and training efficiency. However, there may still be a lack of a principal way of selecting those data in low density regions, and thus studying how to make the data selection more effective remains an open question. Your constructive suggestions will enlighten us to further improve ATOL, and thanks again for your comments.\n\n\n> Q4. The performance gain of the (OOD) predictor may be comparable to the generative model used for auxiliary task. Can the auxiliary generative model(s) be used for OOD detection? Can you provide its (or their) performance? If the generative model(s) performs worse, what could contribute to the advantage of the predictor (maybe the contrastive learning loss or maybe the auxiliary task)? Can you identify where the advantage comes from?\n\nA4. Sincerely apologize for the confusion. We would like to address your concerns as follows. Related discussions will be added in our revision. \n\n \n- **Generators cannot be used for OOD detection.** In our current realization, the generator is used to generate auxiliary OOD data. It is either randomly initialized or pre-trained on ID data, further satisfying the distance-preserving constraint in Eq. 8. Therefore, the generator in ATOL is not trained particularly for OOD detection, which is hard to be used for OOD detection. However, we also believe that using the generator to detect OOD detection is attractive, which will motivate our following studies. \n\n- **A poor generator can still benefit OOD detection.** Our ATOL is different from previous data generation-based methods in that we do not assume the generator is reliable in generating high-quality data. Especially, in Section 3, we demonstrate that even if the generated data are not reliable, they can still benefit the predictor if Conditions 1-2 and Eq. 5 are satisfied. We also provide the experimental supports in Section 5.3, justifying that a randomly initialized generator can still benefit our method to improve OOD detection. Therefore, the main advantages of our improvements come from the introduced conditions in Section 3 and the learning strategies in Section 4 instead of the adopted generators. \n\n\n\n\n\n> Q5: The paper states there is a limitation discussion. Where is it?\n\nA5. Due to space limitations, our discussion about limitations is given in Appendix F. Here, we list two factors that may motivate our future works. First, our current realization of ATOL is relatively intricate, requiring training constraints on both the generator and the predictor. Further studies will explore more advanced conditions that can ease our realization and further reduce computing costs. Second, we observe that the diversity of generated data is closely related to the final performance (cf., Appendix D.6). However, in our current version, we do not consider diversity for the generator in either theories or algorithms, which will motivate our following exploration. \n\n\n[1]: Weitang Liu, et al. \"Energy-based out-of-distribution detection.\" NeurIPS 2020.\n\n[2]: Du Xuefeng, et al. \"Vos: Learning what you don't know by virtual outlier synthesis.\" ICLR 2022.\n"}]

Author rebuttal:

Meta-review: The paper aims to address the mistaken OOD generation issue (i.e., the generated OOD data are not OOD) in data-generation-based learning methods for OOD detection. The authors proposed to use the generated ID/OOD data as an auxiliary task, not a direct task, to improve the OOD detector for real OOD data. Reviewers generally found the problem important, the proposed approach promising, and the experiments solid.

After the rebuttal, many of the reviewers' concerns were addressed, and several reviewers raised the ratings. The paper now receives a positive rating: average 5.75 (5, 5, 6, 7). The AC thus recommends an acceptance. The AC suggests the authors incorporate their rebuttal responses into the final version and improve parts of the writing/explanations, as requested by the reviewers.
Decision: Accept (poster)