Title: Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources
Venue: NeurIPS 2023
Accepted: yes
Keywords: out-of-distribution detection
Abstract: Out-of-distribution (OOD) detection discerns OOD data where the predictor cannot make valid predictions as in-distribution (ID) data, thereby increasing the reliability of open-world classification. However, it is typically hard to collect real out-of-distribution (OOD) data for training a predictor capable of discerning ID and OOD patterns. This obstacle gives rise to *data generation-based learning methods*, synthesizing OOD data via data generators for predictor training without requiring any real OOD data. 
Related methods typically pre-train a generator on ID data and adopt various selection procedures to find those data likely to be the OOD cases. However, generated data may still coincide with ID semantics, i.e., mistaken OOD generation remains, confusing the predictor between ID and OOD data. To this end, we suggest that generated data (with mistaken OOD generation) can be used to devise an *auxiliary OOD detection task* to facilitate real OOD detection. Specifically, we can ensure that learning from such an auxiliary task is beneficial if the ID and the OOD parts have disjoint supports, with the help of a well-designed training procedure for the predictor. Accordingly, we propose a powerful data generation-based learning method named *Auxiliary Task-based OOD Learning* (ATOL) that can relieve the mistaken OOD generation. We conduct extensive experiments under various OOD detection setups, demonstrating the effectiveness of our method against its advanced counterparts.

Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources

Abstract

Out-of-distribution (OOD) detection discerns OOD data where the predictor cannot make valid predictions as in-distribution (ID) data, thereby increasing the reliability of open-world classification. However, it is typically hard to collect real out-ofdistribution (OOD) data for training a predictor capable of discerning ID and OOD patterns. This obstacle gives rise to data generation-based learning methods , synthesizing OOD data via data generators for predictor training without requiring any real OOD data. Related methods typically pre-train a generator on ID data cases. However, generated data may still coincide with ID semantics, i.e., mistaken OOD generation remains, confusing the predictor between ID and OOD data. To this end, we suggest that generated data (with mistaken OOD generation) can be used to devise an auxiliary OOD detection task to facilitate real OOD detection. Specifically, we can ensure that learning from such an auxiliary task is beneficial if the ID and the OOD parts have disjoint supports, with the help of a well-designed training procedure for the predictor. Accordingly, we propose a powerful data generation-based learning method named Auxiliary Task-based OOD Learning (ATOL) that can relieve the mistaken OOD generation. We conduct extensive experiments under various OOD detection setups, demonstrating the effectiveness of our method against its advanced counterparts. The code is publicly available at: https://github.com/tmlr-group/ATOL .

1 Introduction

Deep learning in the open world should not only make accurate predictions for in-distribution (ID) data meanwhile should detect out-of-distribution (OOD) data whose semantics are different from ID cases [15, 56, 22, 81, 85, 82]. It drives recent studies in OOD detection [43, 23, 33, 66, 80, 24], which is important for many safety-critical applications such as autonomous driving and medical analysis. Previous works have demonstrated that well-trained predictors can wrongly take many OOD data as ID cases [1, 15, 21, 48], motivating recent studies towards effective OOD detection.

Ship

Automobile

Horse

(a) Mistaken Semantics

Ship

Automobile

Horse

(b) Adverse Impacts

Real Data

Ship

Automobile

Horse

Auxiliary Task

Auxiliary Task

Real Task

Joint Train

Predictor Learn

Auxiliary OOD

Real ID

Auxiliary ID

Alignment

Joint Train

Predictor Learn

Auxiliary OOD

Real ID

Auxiliary ID

Alignment

Auxiliary Task

Real Task

Joint Train

Predictor Learn

Auxiliary OOD

Real ID

Auxiliary ID

Alignment

Real Task

In the literature, an effective learning scheme is to conduct model regularization with OOD data, making predictors achieve low-confidence predictions on such data [3, 16, 36, 42, 49, 75, 79]. Overall, they directly let the predictor learn to discern ID and OOD patterns, thus leading to improved performance in OOD detection. However, it is generally hard to access real OOD data [10, 11, 69, 70], hindering the practical usage scenarios for such a promising learning scheme.

Instead of collecting real OOD data, we can generate OOD data via data generators to benefit the predictor learning, motivating the data generation-based methods [31, 50, 55, 62, 67]. Generally speaking, existing works typically fit a data generator [12] on ID data (since real OOD data are inaccessible). Then, these strategies target on finding the OOD-like cases from generated data by adopting various selection procedures. For instance, Lee et al. [31] and Vernekar et al. [62] take boundary data that lie far from ID boundaries as OOD-like data; PourReza et al. [50] select those data generated in the early stage of generator training. However, since the generators are trained on ID data and these selection procedures can make mistakes, one may wrongly select data with ID semantics as OOD cases (cf., Figure 1(a)), i.e., mistaken OOD generation remains. It will misguide the predictors to confuse between ID and OOD data, making the detection results unreliable, as in Figure 1(b). Overall, it is hard to devise generator training procedures or data selection strategies to overcome mistaken OOD generation, thus hindering practical usages of previous data generation-based methods.

To this end, we suggest that generated data can be used to devise an auxiliary task , which can benefit real OOD detection even with data suffered from mistaken OOD generation. Specifically, such an auxiliary task is a crafted OOD detection task, thus containing ID and OOD parts of data (termed auxiliary ID and auxiliary OOD data , respectively). Then, two critical problems arise how to make such an auxiliary OOD detection task learnable w.r.t. the predictor and how to ensure that the auxiliary OOD detection task is beneficial w.r.t. the real OOD detection . For the first question, we refer to the advanced OOD detection theories [11], suggesting that the auxiliary ID and the auxiliary OOD data should have the disjoint supports in the input space (i.e., without overlap w.r.t. ID and OOD distributions), cf., C 1 (Condition 1 for short). For the second question, we justify that, for our predictor, if real and auxiliary ID data follow similar distributions, cf., C 2, the auxiliary OOD data are reliable OOD data w.r.t. the real ID data. In summary, if C 1 and C 2 hold, we can ensure that learning from the auxiliary OOD detection task can benefit the real OOD detection, cf., Proposition 1.

Based on the auxiliary task, we propose Auxiliary Task-based OOD Learning (ATOL), an effective data generation-based OOD learning method. For C 1, it is generally hard to directly ensure the disjoint supports between the auxiliary ID and auxiliary OOD data due to the high-dimensional input space. Instead, we manually craft two disjoint regions in the low-dimensional latent space (i.e., the input space of the generator). Then, with the generator having distance correlation between input

space and latent space [63], we can make the generated data that belong to two different regions in the latent space have the disjoint supports in the input space, assigning to the auxiliary ID and the auxiliary OOD parts, respectively. Furthermore, to fulfill C 2, we propose a distribution alignment risk for the auxiliary and the real ID data, alongside the OOD detection risk to make the predictor benefit from the auxiliary OOD detection task. Following the above learning procedure, we can relieve the mistaken OOD generation issue and achieve improved performance over previous works in data generation-based OOD detection, cf., Figure 2 for a heuristic illustration.

To verify the effectiveness of ATOL, we conduct extensive experiments across representative OOD detection setups, demonstrating the superiority of our method over previous data generation-based methods, such as Vernekar et al. [62], with 12 . 02% , 18 . 32% , and 20 . 70% improvements measured by average FPR 95 on CIFAR10 , CIFAR100 , and ImageNet datasets. Moreover, compared with more advanced OOD detection methods (cf., Appendix F), such as Sun et al. [57], our ATOL can also have promising improvements, which reduce the average FPR 95 by 4 . 36% , 15 . 03% , and 12 . 02% on CIFAR10 , CIFAR100 , and ImageNet datasets.

Contributions. We summarize our main contributions into three folds:

We focus on the mistaken OOD generation in data generation-based detection methods, which are largely overlooked in previous works. To this end, we introduce an auxiliary OOD detection task to combat the mistaken OOD generation and suggest a set of conditions in Section 3 to make such an auxiliary task useful.

Our discussion about the auxiliary task leads to a practical learning method named ATOL. Over existing works in data generation-based methods, our ATOL introduces small costs of extra computations but is less susceptible to mistaken OOD generation.

We conduct extensive experiments across representative OOD detection setups, demonstrating the superiority of our method over previous data generation-based methods. Our ATOL is also competitive over advanced works that use real OOD data for training, such as outlier exposure [16], verifying that the data generation-based methods are still worth studying.

2 Preliminary

Let X ⊆ R n the input space, Y = { 1 , . . . , c } the label space, and h = ρ ◦ ϕ : R n → R c the predictor, where ϕ : R n → R d is the feature extractor and ρ : R d → R c is the classifier. We consider P ID X,Y and P ID X the joint and the marginal ID distribution defined over X ×Y and X , respectively. We also denote P OOD X the marginal OOD distribution over X . Then, the main goal of OOD detection is to find the proper predictor h ( · ) and the scoring function s ( · ; h ) : R n → R such that the OOD detector

can detect OOD data with a high success rate, where β is a given threshold. To get a proper predictor h ( · ) , outlier exposure [16] proposes regularizing the predictor to produce low-confidence predictions for OOD data. Assuming the real ID data x ID and label y ID are randomly drawn from P ID X,Y , then the learning objective of outlier exposure can be written as:

where λ is the trade-off hyper-parameter, ℓ CE ( · ) is the cross-entropy loss, and ℓ OE ( · ) is the KullbackLeibler divergence of softmax predictions to the uniform distribution. Although outlier exposure remains one of the most powerful methods in OOD detection, its critical reliance on OOD data hinders its practical applications.

Fortunately, data generation-based OOD detection methods can overcome the reliance on real OOD data, meanwhile making the predictor learn to discern ID and OOD data. Overall, existing works [31, 62, 67] leverage the generative adversarial network (GAN) [12, 51] to generate data, collecting those data that are likely to be the OOD cases for predictor training. As a milestone work, Lee et al. [31] propose selecting the 'boundary' data that lie in the low-density area of P ID X , with the following learning objective for OOD generation:

where G X denotes the OOD distribution of the generated data w.r.t. the generator G : R m → R n ( m is the dimension of the latent space). Note that the term (a) forces the generator G ( · ) to generate low-density data and the term (b) is the standard learning objective for GAN training [12]. Then, with the generator trained for OOD generation, the generated data are used for predictor training, following the outlier exposure objective, given by

Drawbacks. Although a promising line of works, the generated data therein may still contain many ID semantics (cf., Figure 1(a)). It stems from the lack of knowledge about OOD data, where one can only access ID data to guide the training of generators. Such mistaken OOD generation is common in practice, and its negative impacts are inevitable in misleading the predictor (cf., Figure 1(b)). Therefore, how to overcome the mistaken OOD generation is one of the key challenges in data generation-based OOD detection methods.

3 Motivation

Note that getting generators to generate reliable OOD data is difficult, since there is no access to real OOD data in advance and the selection procedures can make mistakes. Instead, given that generators suffer from mistaken OOD generation, we aim to make the predictors alleviate the negative impacts raised by those unreliable OOD data.

Our key insight is that these unreliable generated data can be used to devise a reliable auxiliary task which can benefit the predictor in real OOD detection. In general, such an auxiliary task is a crafted OOD detection task, of which the generated data therein are separated into the ID and the OOD parts. Ideally, the predictor can learn from such an auxiliary task and transfer the learned knowledge (from the auxiliary task) to benefit the real OOD detection, thereafter relieving the mistaken OOD generation issue. Therein, two issues require our further study:

Based on unreliable OOD sources, how to devise a learnable auxiliary OOD detection task?

Given the well-designed auxiliary task, how to make it benefit the real OOD detection?

For the first question, we refer the advanced work [11] in OOD detection theories, which provides necessary conditions for which the OOD detection task is learnable. In the context of the auxiliary OOD detection task, we will separate the generated data into the auxiliary ID parts and auxiliary OOD parts, following the distributions defined by G ID X and G OOD X , respectively. Then, we provide the following condition for the separable property of the auxiliary OOD detection task.

Condition 1 ( Separation for the Auxiliary Task ) . There is no overlap between the auxiliary ID distribution and the auxiliary OOD distribution, i.e., supp G ID X ∩ supp G OOD X = ∅ , where supp denotes the support set of a distribution.

Overall, C 1 states that the auxiliary ID data distribution G ID X and the auxiliary OOD data distribution G OOD X should have the disjoint supports. Then, according to Fang et al. [11] (Theorem 3), such an auxiliary OOD detection might be learnable, in that the predictor can have a high success rate in discerning the auxiliary ID and the auxiliary OOD data. In general, the separation condition is indispensable for us to craft a valid OOD detection task. Also, as in the proof of Proposition 1, the separation condition is also important to benefit the real OOD detection.

For the second question, we need to first formally define if a data point can benefit the real OOD detection. Here, we generalize the separation condition in Fang et al. [11], leading to the following definition for the reliability of OOD data.

Definition 1 ( Reliability of OOD Data ) . An OOD data point x is reliable (w.r.t. real ID data) to a mapping function ϕ ′ ( · ) , if ϕ ′ ( x ) / ∈ supp ϕ ′ # P ID X , where ϕ ′ # is the distribution transformation associated with ϕ ′ ( · ) .

With identity mapping function ϕ ′ ( · ) , the above definition states that x is a reliable OOD data point if it is not in the support set of the real ID distribution w.r.t. the transformed space. Intuitively, if real ID data are representative almost surely [11], ϕ ′ ( · ) far from the ID support also indicates that ϕ ′ ( · ) will not possess ID semantics, namely, reliable. Furthermore, ϕ ′ ( · ) is not limited to identity mapping,

which can be defined by any transformed space. The feasibility of such a generalized definition is supported by many previous works [10, 41] in OOD learning, demonstrating that reliable OOD data defined in the embedding space (or in our context, the transformed distribution) can also benefit the predictor learning in OOD detection.

Based on Definition 1, we want to make auxiliary OOD data reliable w.r.t. real ID data, such that the auxiliary OOD detection task is ensured to be beneficial, which motivates the following condition.

Condition 2 ( Transferability of the Auxiliary Task ) . The auxiliary OOD detection task is transferable w.r.t. the real OOD detection task if ϕ ′ # P ID X ≈ ϕ ′ # G ID X .

Overall, C 2 states that auxiliary ID data approximately follow the same distribution as that of the real ID data in the transformed space, i.e., ϕ ′ ( X ) . Note that, Auxiliary ID and OOD data can arbitrarily differ from real ID and OOD data in the data space, i.e., auxiliary data are unreliable OOD sources . However, if C 2 is satisfied, the model " believes " the auxiliary ID data and the real ID data are the same. Then, we are ready to present our main proposition, demonstrating why C 2 can ensure the reliability of auxiliary OOD data.

Proposition 1. Assume the predictor h = ρ ◦ ϕ can separate ID and OOD data, namely,

approaches 0. Under C 1 and C 2, auxiliary OOD data are reliable w.r.t. real ID data given ϕ ′ = ϕ .

The proof can be found in Appendix C. In heuristics, Eq. (5) states that the predictor should excel at such an auxiliary OOD detection task, where the model can almost surely discern the auxiliary ID and the auxiliary OOD cases. Then, if we can further align the real and the auxiliary ID distributions (i.e., C 2), the predictor will make no difference between the real ID data and the auxiliary ID data. Accordingly, since the auxiliary OOD data are reliable OOD cases w.r.t. the auxiliary ID data, the auxiliary OOD data are also reliable w.r.t. the real ID data. Thereafter, one can make the predictor learn to discern the real ID and auxiliary OOD data, where no mistaken OOD generation affects. Figure 2 summarizes our key concepts. As we can see, learning from the auxiliary OOD detection task and further aligning the ID distribution ensure the separation between ID and OOD cases, benefiting the predictor learning from reliable OOD data and improving real OOD detection performance.

Remark 1. There are two parallel definitions for real OOD data [70], related to different goals towards effective OOD detection. One definition is that an exact distribution of real OOD data exists, and the goal is to mitigate the distribution discrepancy w.r.t. the real OOD distribution for effective OOD detection. Another definition is that all those data whose true labels are not in the considered label space are real OOD data, and the goal is to make the predictor learn to see as many OOD cases as possible. Due to the lack of real OOD data for supervision, the latter definition is more suitable than the former one in our context, where we can concentrate on mistaken OOD generation.

4 Learning Method

Section 3 motivates a new data generation-based method named Auxiliary Task-based OOD Learning (ATOL), overcoming the mistaken OOD generation via the auxiliary OOD detection task. Referring to Theorem 1, the power of the auxiliary task is built upon C 1 and C 2. It makes ATOL a two-staged learning scheme, related to auxiliary task crafting (crafting the auxiliary task for C 1) and predictor training (applying the auxiliary task for C 2), respectively. Here, we provide the detailed discussion.

4.1 Crafting the Auxiliary Task

We first need to construct the auxiliary OOD detection task, defined by auxiliary ID and auxiliary OOD data generated by the generator G ( · ) . In the following, we assume the auxiliary ID data ˆ x ID drawn from G ID X and the auxiliary OOD data ˆ x OOD drawn from G OOD X . Then, according to C 1, we need to fulfill the condition that G ID X and G OOD X should have the disjoint supports.

Although all distribution pairs ( G ID X , G OOD X ) with disjoint supports can make C 1 hold, it is hard to realize such a goal due to the complex data space R n of images. Instead, we suggest crafting such a distribution pair in the latent space R m of the generator, of which the dimension m is much lower

than n . We define the latent ID and the latent OOD data as z ID and z OOD, respectively. Then, we assume the latent ID data are drawn from the high-density region of a Mixture of Gaussian (MoG):

where τ is the threshold and M Z is a pre-defined MoG, given by the density function of MoG M ( z ) = ∑ c i =1 1 c N ( z | µ i , σ i ) , with the mean µ i and the covariance σ i for the i -th sub-Gaussian. Furthermore, as we demonstrate in Section 4.2, we also require the specification of labels for latent ID data. Therefore, we assume that M Z is constructed by c sub-distributions of Gaussian, and data belong to each of these c sub-distributions should have the same label ˆ y ID.

Remark 2. MoG provides a simple way to generate data, yet the key point is to ensure that auxiliary ID/OOD data should have the disjoint support, i.e, C 1. Therefore, if C 1 is satisfied properly, other noise distributions, such as the beta mixture models[40] and the uniform distribution, can also be used. Further, our ATOL is different from previous data generation-based methods in that we do not require generated data to be reliable in the data space. ATOL does not involve fitting the MoG to real ID data, where the parameters can be pre-defined and fixed. Therefore, we do not consider overfitting and accuracy of MoG in our paper.

Furthermore, for the disjoint support, the latent OOD data are drawn from a uniform distribution except for the region with high MoG density, i.e.,

with U Z a pre-defined uniform distribution. For now, we can ensure that Z ID ∩ Z OOD = ∅ .

However, Z ID ∩ Z OOD = ∅ does not imply G ( Z ID ) ∩ G ( Z OOD ) = ∅ , i.e., C 1 is not satisfied given arbitrary G ( · ) . To this end, we suggest further ensuring the generator to be a distance-preserving function [83], which is a sufficient condition to ensure that the disjoint property can be transformed from the latent space into the data space. For distance preservation, we suggest a regularization term for the generator regularizing, which is given by

where d ( z 1 , z 2 ) is the centralized distance given by ∥ z 1 -z 2 ∥ 2 -E ∥ z 1 -z 2 ∥ 2 . Overall, Eq. (8) is the correlation for the distances between data pairs measured in the latent and the data space, stating that the closer distances in the latent space should indicate the closer distances in the data space.

In summary, to fulfill C 1, we need to specify the regularizing procedure for the generator, overall summarized by the following two steps. First, we regularize the generator with the regularization in Eq. (8) to ensure the distance-preserving property. Then, we specify two disjoint regions in the latent space following Eqs. (6)-(7), of which the data after passing through the generator are taken as auxiliary ID and auxiliary OOD data, respectively. We also summarize the algorithm details in crafting the auxiliary OOD detection task in Appendix D.

4.2 Applying the Auxiliary Task

We have discussed a general strategy to construct the auxiliary OOD detection task, which can be learned by the predictor following an outlier exposure-based learning objective, similar to Eq. (2). Now, we fulfill C 2, transferring model capability from the auxiliary task to real OOD detection.

In general, C 2 states that we should align distributions between the auxiliary and the real ID data such that the auxiliary OOD data can benefit real OOD detection, thus overcoming the negative effects of mistaken OOD generation. Following Tang et al. [59], we align the auxiliary and the real ID distribution via supervised contrastive learning [29], pulling together data belonging to the same class in the embedding space meanwhile pushing apart data from different classes. Accordingly, we suggest the alignment loss for the auxiliary ID data ˆ x ID, following,

N l is a set of data of size l drawn from the real ID distribution, namely, N l = { x i ID | ( x i ID , y i ID ) ∼ P ID X,Y , for i ∈ { 1 , . . . , l }} . Furthermore, N y l is a subset of N l with ID data that below to the label of

y , namely, N y l = { x i ID | ( x i ID , y i ID ) ∼ P ID X,Y , and y i ID = ˆ y ID , for i ∈ { 1 , . . . , l }} . Accordingly, based on Theorem 1, we can ensure that learning from the auxiliary OOD detection task, i.e., small Eq. (5), can benefit real OOD detection, freeing from the mistaken OOD generation.

4.3 Overall Algorithm

We further summarize the training and the inferring procedure for the predictor of ATOL, given the crafted auxiliary ID and the auxiliary OOD distributions that satisfy C 1 via Eqs. (6)-(7). The pseudo codes are further summarized in Appendix D due to the space limit.

Training Procedure. The overall learning objective consists of the real task learning, the auxiliary task learning, and the ID distribution alignment, which is given by:

where α, λ ≥ 0 are the trade-off parameters. By finding the predictor that leads to the minimum of the Eq. (10), our ATOL can ensure the improved performance of real OOD detection. Note that, Eq. (10) can be realized in a stochastic manner, suitable for deep model training.

Inferring Procedure. We adopt the MaxLogit scoring [17] in OOD detection. Given a test input x , the MaxLogit score is given by:

where h k ( · ) denotes the k -th logit output. In general, the MaxLogit scoring is better than other commonly-used scoring functions, such as maximum softmax prediction [15], when facing large semantic space. Therefore, we choose MaxLogit instead of MSP for OOD scoring in our realization.

5 Experiments

This section conducts extensive experiments for ATOL in OOD detection. In Section 5.1, we describe the experiment setup. In Section 5.2, we demonstrate the main results of our method against the data generation-based counterparts on both the CIFAR [30] and the ImageNet [8] benchmarks. In Section 5.3, we further conduct ablation studies to comprehensively analyze our method.

5.1 Setup

Backbones setups. For the CIFAR benchmarks, we employ the WRN-40-2 [78] as the backbone model. Following [36], models have been trained for 200 epochs via empirical risk minimization, with a batch size 64 , momentum 0 . 9 , and initial learning rate 0 . 1 . The learning rate is divided by 10 after 100 and 150 epochs. For the ImageNet, we employ pre-trained ResNet-50 [13] on ImageNet, downloaded from the PyTorch official repository.

Generators setups. For the CIFAR benchmarks, following [31], we adopt the generator from the Deep Convolutional (DC) GAN [51]). Following the vanilla generation objective, DCGAN has been trained on the ID data, where the batch size is 64 and the initial learning rate is 0 . 0002 . For the ImageNet benchmark, we adopt the generator from the BigGAN [2] model, designed for scaling generation of high-resolution images, where the pre-trained model biggan-256 can be downloaded from the TensorFlow hub. Note that our method can work on various generators except those mentioned above, even with a random-parameterized one, later shown in Section 5.3.

Baseline methods. We compare our ATOL with advanced data generation-based methods in OOD detection, including (1) BoundaryGAN [31], (2) ConfGAN [55], (3) ManifoldGAN [62], (4) G2D [50] (5) CMG [67]. For a fair comparison, all the methods use the same generator and pre-trained backbone without regularizing with outlier data.

Auxiliary task setups. Hyper-parameters are chosen based on the OOD detection performance on validation datasets. In latent space, the auxiliary ID distribution is the high density region of the

Mixture of Gaussian (MoG). Each sub-Gaussian in the MoG has the mean µ = ( µ 1 , µ 2 , . . . , µ m ) and the same covariance matrix σ · I , where m is the dimension of latent space, µ i is randomly selected from the set {-µ, µ } , and I is the identity matrix. The auxiliary OOD distribution is the uniform distribution except for high MoG density region, where each dimension has the same space size u . For the CIFAR benchmarks, the value of α is set to 1 , µ is 5 , σ is 0 . 1 , and u is 8 . For the ImageNet benchmarks, the value of α is set to 1 , µ is 5 , σ is 0 . 8 , and u is 8 .

Training details. For the CIFAR benchmarks, ATOL is run for 10 epochs and uses SGD with an initial learning rate 0 . 01 and the cosine decay [37]. The batch size is 64 for real ID cases, 64 for auxiliary ID cases, and 256 for auxiliary OOD cases. For the ImageNet benchmarks, ATOL is run for 10 epochs using SGD with an initial learning rate 0 . 0003 and a momentum 0 . 9 . The learning rate is decayed by a factor of 10 at 30% , 60% , and 90% of the training steps. Furthermore, the batch size is fixed to 32 for real ID cases, 32 for auxiliary ID cases, and 128 for auxiliary OOD cases.

Evaluation metrics. The OOD detection performance of a detection model is evaluated via two representative metrics, which are both threshold-independent [7]: the false positive rate of OOD data when the true positive rate of ID data is at 95% (FPR 95 ); and the area under the receiver operating characteristic curve (AUROC), which can be viewed as the probability of the ID case having greater score than that of the OOD case.

5.2 Main Results

We begin with our main experiments on the CIFAR and ImageNet benchmarks. Model performance is tested on commonly-used OOD datasets. For the CIFAR cases, we employed Texture [6], SVHN [47], Places 365 [84], LSUN-Crop [76], LSUN-Resize [76], and iSUN [74]. For the ImageNet case, we employed iNaturalist [18], SUN [74], Places 365 [84], and Texture [6]. In Table 1, we report the average performance (i.e., FPR 95 and AUROC) regarding the OOD datasets mentioned above. Please refer to Tables 17-18 and 19 in Appendix F.1 for the detailed reports.

CIFAR benchmarks. Overall, our ATOL can lead to effective OOD detection, which generally demonstrates better results than the rivals by a large margin. In particular, compared with the advanced data generation-based method, e.g., ManifoldGAN [62], ATOL significantly improves the performance in OOD detection, revealing 12 . 02% and 2 . 96% average improvements w.r.t. FPR 95 and AUROC on the CIFAR10 dataset and 18 . 32% and 9 . 84% of the average improvements on the CIFAR100 dataset. This highlights the superiority of our novel data generation strategy, relieving mistaken OOD generation as in previous methods.

ImageNet benchmark. We evaluate a large-scale OOD detection task based on ImageNet dataset. Compared to the CIFAR benchmarks above, the ImageNet task is more challenging due to the large semantic space and large amount of training data. As we can see, previous data generation-based methods reveal poor performance due to the increased difficulty in searching for the proper OOD-like data in ImageNet, indicating the critical mistaken OOD generation remains (cf., Appendix F.9). In contrast, our ATOL can alleviate this issue even for large-scale datasets, e.g., ImageNet, thus leading to superior results. In particular, our ATOL outperforms the best baseline ManifoldGAN [62] by 20 . 70% and 8 . 09% average improvement w.r.t. FPR 95 and AUROC, which demonstrates the advantages of our ATOL to relieve mistaken OOD generation and benefit to real OOD detection.

5.3 Ablation Study

To further demonstrate the effectiveness of our ATOL, we conduct extensive ablation studies to verify the contributions of each proposed component.

Auxiliary task crafting schemes. In Section 4.1, we introduce the realization of the auxiliary OOD detection task in a tractable way. Here, we study two components of generation: the disjoint supports for ID and OOD distributions in the latent space and the distance-preserving generator. In particular, we compare ATOL with two variants: 1) ATOL w/o τ , where the ID and OOD distributions may overlap in latent space without separation via the threshold τ ; 2) ATOL w/o ℓ reg, where we directly use the generator to generate the auxiliary data without further regularization. As we can see in Table 2, the overlap between the auxiliary ID and the auxiliary OOD distributions in latent space will lead to the failure of the auxiliary OOD detection task, demonstrating the catastrophic performance (e.g., 92 . 20% FPR 95 and 51 . 34% AUROC on CIFAR100 ). Moreover, without the regularized generator as a distance-preserving function, the ambiguity of generated data will compromise the performance of ATOL, which verifies the effectiveness of our generation scheme.

Auxiliary task learning schemes. In Section 4.2, we propose ID distribution alignment to transfer model capability from auxiliary OOD detection task to real OOD detection. In Table 2, we conduct experiments on a variant, namely, ATOL w/o ℓ align, where the predictor directly learns from the auxiliary task without further alignment. Accordingly, ATOL w/o ℓ align can reveal better results than ATOL w/o ℓ reg, with 18 . 84% and 3 . 70% further improvement w.r.t. FPR 95 and AUROC on CIFAR10 and 8 . 60% and 6 . 61% on CIFAR100 . However, the predictor learned on the auxiliary task cannot fit into the real OOD detection task. Thus, we further align the distributions between the auxiliary and real ID data in the transformed space, significantly improving the OOD detection performance with 7 . 75% on CIFAR10 and 19 . 14% on CIFAR100 w.r.t. FPR 95 .

Generator Setups. On CIFAR benchmarks, we use the generator of the DCGAN model fitting on the ID data in our ATOL. We further investigate whether ATOL remains effective when using generators with other setups, summarized in Table 3. First, we find that even with a random-parameterized DCGAN generator, i.e., Rand-DCGAN, our ATOL can still yield reliable performance surpassing its advanced counterparts. Such an observation suggests that our method requires relatively low training costs, a property not shared by previous methods. Second, we adopt more advanced generators, i.e., BigGAN [2] and StyleGAN-v2 [27]. As we can see, ATOL can benefit from better generators, leading to large improvement 18 . 80% and 5 . 12% w.r.t. FPR 95 and AUROC on CIFAR100 over the DCGAN case. Furthermore, even with complicated generators, our method demonstrates promising performance improvement over other methods with acceptable computational resources. Please refer to the Appendix E.8 and Appendix F.7 for more details.

Quantitative analysis on computation. Data generationbased methods are relatively expensive in computation, so the consumed training time is also of our interest. In Table 4, we compare the time costs of training for the considered data generation-based methods. Since our method does not need a complex generator training procedure during the OOD learning step, we can largely alleviate the expensive cost of computation inherited from the data generation-based approaches. Overall, our method requires minimal training costs compared to the advanced counterparts, which further reveals the efficiency of our method. Therefore, our method can not only lead to improved performance over previous methods, but it also requires fewer computation resources.

6 Conclusion

Data generation-based methods in OOD detection are a promising way to make the predictor learn to discern ID and OOD data without real OOD data. However, mistaken OOD generation significantly challenges their performance. To this end, we propose a powerful data generation-based learning method by the proposed auxiliary OOD detection task, largely relieving mistaken OOD generation. Extensive experiments show that our method can notably improve detection performance compared to advanced counterparts. Overall, our method can benefit applications for data generation-based OOD detection, and we intend further research to extend our method beyond image classification. We also hope our work can draw more attention from the community toward data generation-based methods, and we anticipate our auxiliary task-based learning scheme can benefit more directions in OOD detection, such as wild OOD detection [28].

Acknowledgments and Disclosure of Funding

HTZ, QZW and BH were supported by the NSFC Young Scientists Fund No. 62006202, NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011652, HKBU Faculty Niche Research Areas No. RC-FNRA-IG/22-23/SCI/04, and HKBU CSD Departmental Incentive Scheme. XBX and TLL were partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031. FL was supported by Australian Research Council (ARC) under Award No. DP230101540, and by NSF and CSIRO Responsible AI Program under Award No. 2303037.

References

Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. In CVPR , 2016.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR , 2019.

Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. ATOM: robustifying out-ofdistribution detection using outlier mining. In ECML , 2021.

Ling-Hao Chen, He Li, Wanyuan Zhang, Jianbin Huang, Xiaoke Ma, Jiangtao Cui, Ning Li, and Jaesoo Yoo. Anomman: Detect anomaly on multi-view attributed networks. arXiv preprint arXiv:2201.02822 , 2022.

Ling-Hao Chen, Jiawei Zhang, Yewen Li, Yiren Pang, Xiaobo Xia, and Tongliang Liu. Humanmac: Masked motion completion for human motion prediction. In ICCV , 2023.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR , 2014.

Jesse Davis and Mark Goadrich. The relationship between precision-recall and ROC curves. In ICML , 2006.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In ICLR , 2023.

Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. VOS: learning what you don't know by virtual outlier synthesis. In ICLR , 2022.

Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribution detection learnable? In NeurIPS , 2022.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS , 2014.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In CVPR , 2019.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR , 2017.

Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. In ICLR , 2019.

Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for realworld settings. In ICML , 2022.

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In CVPR , 2018.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR , 2017.

Rui Huang and Yixuan Li. MOS: towards scaling out-of-distribution detection for large semantic space. In CVPR , 2021.

Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In NeurIPS , 2021.

Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. In NeurIPS , 2021.

Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distribution examples via augmenting content and style. In ICLR , 2023.

Zhuo Huang, Miaoxi Zhu, Xiaobo Xia, Li Shen, Jun Yu, Chen Gong, Bo Han, Bo Du, and Tongliang Liu. Robust generalization against photon-limited corruptions via worst-case sharpness minimization. In CVPR , 2023.

Conor Igoe, Youngseog Chung, Ian Char, and Jeff Schneider. How useful are gradients for ood detection really? arXiv preprint arXiv:2205.10439 , 2022.

Taewon Jeong and Heeyoung Kim. OOD-MAML: meta-learning for few-shot out-of-distribution detection and classification. In NeurIPS , 2020.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR , 2020.

Julian Katz-Samuels, Julia B. Nakhleh, Robert D. Nowak, and Yixuan Li. Training OOD detectors in their natural habitats. In ICML , 2022.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS , 2020.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto , 2009.

Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In ICLR , 2018.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS , 2018.

Yewen Li, Chaojie Wang, Xiaobo Xia, Tongliang Liu, Xin Miao, and Bo An. Out-of-distribution detection with an adaptive likelihood ratio on informative hierarchical V AE. In NeurIPS , 2022.

Yi Li and Nuno Vasconcelos. Background data resampling for outlier-aware classification. In CVPR , 2020.

Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR , 2018.

Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurIPS , 2020.

Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR , 2017.

Yadan Luo, Zijian Wang, Zi Huang, and Mahsa Baktashmotlagh. Progressive graph learning for open-set domain adaptation. In ICML , 2020.

Yadan Luo, Zijian Wang, Zhuoxiao Chen, Zi Huang, and Mahsa Baktashmotlagh. Source-free progressive graph learning for open-set domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(9):11240-11255, 2023.

Zhanyu Ma and Arne Leijon. Beta mixture models and the application to image classification. In ICIP , 2009.

Akshay Mehra, Bhavya Kailkhura, Pin-Yu Chen, and Jihun Hamm. On certifying and improving generalization to unseen domains. arXiv preprint arXiv:2206.12364 , 2022.

Yifei Ming, Ying Fan, and Yixuan Li. Poem: Out-of-distribution detection with posterior sampling. In ICML , 2022.

Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-ofdistribution detection. In AAAI , 2022.

Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out-of-distribution detection? In ICLR , 2023.

Sina Mohseni, Mandar Pitale, J. B. S. Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distribution detection. In AAAI , 2020.

Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detection. In AAAI , 2022.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning , 2011.

Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR , 2015.

Aristotelis-Angelos Papadopoulos, Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang. Outlier exposure with confidence control for out-of-distribution detection. Neurocomputing , 441:138-150, 2021.

Masoud PourReza, Bahram Mohammadi, Mostafa Khaki, Samir Bouindour, Hichem Snoussi, and Mohammad Sabokrou. G2D: generate to detect anomaly. In WACV , 2021.

Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR , 2016.

Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set consistency regularization for semi-supervised learning with outliers. arXiv preprint arXiv:2105.14148 , 2021.

Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In ICML , 2020.

Vikash Sehwag, Mung Chiang, and Prateek Mittal. SSD: A unified framework for selfsupervised outlier detection. In ICLR , 2021.

Kumar Sricharan and Ashok Srivastava. Building robust classifiers through generation of confident out of distribution examples. arXiv preprint arXiv:1812.00239 , 2018.

Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In NeurIPS , 2021.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In ICML , 2022.

Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: novelty detection via contrastive learning on distributionally shifted instances. In NeurIPS , 2020.

Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In ICML , 2022.

Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In ICLR , 2023.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(86):2579-2605, 2008.

Sachin Vernekar, Ashish Gaurav, Vahdat Abdelzad, Taylor Denouden, Rick Salay, and Krzysztof Czarnecki. Out-of-distribution detection in classifiers via generation. arXiv preprint arXiv:1910.04241 , 2019.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint . Cambridge University Press, 2019.

Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In CVPR , 2022.

Haotao Wang, Aston Zhang, Yi Zhu, Shuai Zheng, Mu Li, Alex J. Smola, and Zhangyang Wang. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. In ICML , 2022.

Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, et al. On the robustness of chatgpt: An adversarial and out-of-distribution perspective. arXiv preprint arXiv:2302.12095 , 2023.

Mengyu Wang, Yijia Shao, Haowei Lin, Wenpeng Hu, and Bing Liu. CMG: A class-mixed generation approach to out-of-distribution detection. In ECML , 2022.

Qizhou Wang, Feng Liu, Yonggang Zhang, Jing Zhang, Chen Gong, Tongliang Liu, and Bo Han. Watermarking for out-of-distribution detection. In NeurIPS , 2022.

Qizhou Wang, Zhen Fang, Yonggang Zhang, Feng Liu, Yixuan Li, and Bo Han. Learning to augment distributions for out-of-distribution detection. In NeurIPS , 2023.

Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander, Tongliang Liu, Jianye Hao, and Bo Han. Out-of-distribution detection with implicit data transformation. In ICLR , 2023.

Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICML , 2022.

Xiaobo Xia, Wenhao Yang, Jie Ren, Yewen Li, Yibing Zhan, Bo Han, and Tongliang Liu. Pluralistic image completion with gaussian mixture models. In NeurIPS , 2022.

Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In ICLR , 2023.

Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755 , 2015.

Jingkang Yang, Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distribution detection. In ICCV , 2021.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Alireza Zaeemzadeh, Niccolò Bisagno, Zeno Sambugaro, Nicola Conci, Nazanin Rahnavard, and Mubarak Shah. Out-of-distribution detection using union of 1-dimensional subspaces. In CVPR , 2021.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC , 2016.

Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments. In WACV , 2023.

Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Ryan Luley, Yiran Chen, and Hai Li. Sio: Synthetic in-distribution data benefits out-of-distribution detection. arXiv preprint arXiv:2303.14531 , 2023.

Yongqi Zhang, Zhanke Zhou, Quanming Yao, and Yong Li. Efficient hyper-parameter search for knowledge graph embedding. In ACL , 2022.

Yongqi Zhang, Zhanke Zhou, Quanming Yao, Xiaowen Chu, and Bo Han. Adaprop: Learning adaptive propagation for graph neural network based knowledge graph reasoning. In KDD , 2023.

Xingjian Zhen, Zihang Meng, Rudrasis Chakraborty, and Vikas Singh. On the versatile uses of partial distance correlation in deep learning. In ECCV , 2022.

Bolei Zhou, Àgata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(6):1452-1464, 2018.

Zhanke Zhou, Jiangchao Yao, Jiaxu Liu, Xiawei Guo, Quanming Yao, Li He, Liang Wang, Bo Zheng, and Bo Han. Combating bilateral edge noise for robust link prediction. In NeurIPS , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper contributes a new data-generation method to train a OOD detector. The paper proposes to first create an auxiliary task for a generator to generate in-distribution samples and OOD samples by using regions of disjoint latent space (equ 6-7). This can lead to disjoint support set by enforcing a distance preserving loss (equ 8). In order to transfer auxiliary task to real OOD detection, it suggests to use a contrastive learning to bring together generated in-dist samples and real in-dist samples. With these two settings, the OOD detector can learn from generated OOD samples.\n\nSTRENGTHS:\nThis discussion is complete and clear. The idea of crafting the auxiliary task using generative model(s) is promising in general. By dividing the samples into high and low density regions, this method bypasses the issue of mistaken OOD samples.\n\nWEAKNESSES:\nThe major concern is the performance of the generator (or possibly I miss that piece of information). While section 3 is convincing, in practice (section 4) how to guarantee high MoG density region has high concentration of in-distribution samples. This may happen when a generator does not perform well.\n\nQUESTIONS:\n$\\bullet$ Can the author change the term predictor to OOD detector? \n\n$\\bullet$ While it is a good idea to generate in-dist and OOD samples using high MoG density region, does it also make the task too easy? I imagine that high MoG region samples may look like in-dist samples, while the rest may look like noise. Does the predictor can perform better if it can learn from more difficult (harder to discern) generated in-dist and generated OOD samples?  \n\n$\\bullet$ Does this method consider the case when high MoG density region still have some OOD samples? \n\n$\\bullet$ The performance gain of the (OOD) predictor may be comparable to the generative model used for auxiliary task. Can the auxiliary generative model(s) be used for OOD detection? Can you provide its (or their) performance? If the generative model(s) performs worse, what could contribute to the advantage of the predictor (maybe the contrastive learning loss or maybe the auxiliary task)? Can you identify where the advantage comes from?\n\nLIMITATIONS:\nThe paper states there is a limitation discussion. Where is it?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose to fix the mistaken OOD generation issue in generative model based approach to out-of-distribution data detection, where the mistaken OOD generation means generated OOD data have semantics of ID data. To fix this issue, auxiliary task-based OOD learning (ATOL) is proposed, which is claimed to have the effect of satisfying two key conditions, i.e., auxiliary ID and auxiliary OOD data have disjoint supports in the input space, and auxiliary OOD data are reliable. To achieve this goal, ATOL adds auxiliary task learning loss and ID distribution alignment loss to the real task learning loss. The empirical study shows the non-trivial improvement of OOD detection performance when using ATOL. Ablation study further confirms the effectiveness of each loss in ATOL.\n\nSTRENGTHS:\nThe motivation to fix the mistaken OOD generation makes sense to me as shown in Fig. 1. \n\nThe two new losses in the proposed ATOL are reasonable and are clearly presented.\n\nI have to say that I like the empirical studies in the paper since they are quite comprehensive and strong, though many interesting results are presented in the supplementary as a result of page limits.\n\nWEAKNESSES:\nThe drawback and strength of generative model based OOD detection and its comparison with other approaches like scoring or regularized training is not fully discussed in the paper, e.g., performance and efficiency. \n\nIn Table 17, the benefit of ATOL is not quite clear when compared with ReAct and CSI.\n\nQUESTIONS:\n1. What is M(z) in Equation 6 and 7? Is it density function of MoG?\n\n2. It is kind of misleading to make the ATOL result bold in Tab. 17 since it is not the best result in terms of AUROC. The best one should be CSI.\n\n3. What is the proportion in Fig. 1b and how is it computed? I tried to search for this information in the paper but it seems that there is no such information. I believe this information is quite important to motivate this paper.\n\nLIMITATIONS:\nLimitations are not discussed in the main paper (please correct me if I am wrong about this).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper tries to overcome the impact of directly applying incorrect OOD data on the OOD model through auxiliary tasks, thereby improving the performance of OOD tasks. The theoretical part is hard to follow, and the experimental part proves the effectiveness of the theory.

STRENGTHS:
1. The paper introduces an auxiliary OOD detection task to combat mistaken OOD generation.
2. The proposed method requires a small additional calculation cost.
3. Experiments show the effectiveness of the proposed method.

WEAKNESSES:
1. The proof of C2 is not clear enough to allow me to clearly understand why achieving C2 can better utilize OOD data. If the model is strong enough or fully trained, it can still confuse incorrect OOD data even if formula 5 is met. Therefore, I judge that the training steps of the algorithm should not be too many and the model should not be too large. Does the author have an explanation for this aspect?

2. There is a writing error in part b of Formula 10.

3. Many aspects of the experiment followed the settings of reference [31], but why not include them in the comparison?

4. The paper is really hard to follow, please polish the paper carefully.

QUESTIONS:
There are too many Mathematical notations. It is suggested to sort out a table to make it clearer.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOne of the techniques for detecting OOD instances is to train a model on OOD data. However, that task is not easy due to difficulty inherent with collecting such OOD data. Rather than collecting such data, this paper proposes instead to generate it, and to train an auxiliary task to improve the OOD detection capabilities of deep learning networks. The proposed approach is called ATOL, for Auxiliary Task-based OOD Learning, and proposes to address one fundamental flaw in existing data-generation based detection methods: the collection of OOD instances from ID data that can mistakenly be labeled as OOD data while being in reality ID.\n\nSTRENGTHS:\nThis paper addresses an interesting problem that I think is quite overlooked in the research community. Indeed, many studies focus on collecting OOD instances without necessarily evaluating the possible side effects of the collected data. This paper identified that instances that are collected and labeled as OOD instances can in fact be ID instances, which could lead to poor generalization performance on ID data, and poor OOD detection capabilities. The approach the authors devised to address the problem is sound and quite intuitive, and the evaluation is quite strong.\n\nWEAKNESSES:\nAlthough this paper addresses the OOD detection problem from a data-generation perspective, I would have very much liked to see how their approach fair with other techniques like distance-based OOD detection methods. Some interesting distance-based OOD detection mechanisms have been proposed in the recent past. For instance, CIDER [1] has achieved state-of-the-art performance OOD detection capabilities that shouldn't be overlooked by the researchers approaching OOD detection from a data-generation perspective.  This would help educate mainstream readers more on exactly what techniques to pursue to robustify their models against OOD samples. \n\nA more fundamental limitation of this study is the fact that it heavily relies on a mixture of Gaussian to decide on what latents to consider as OOD and which ones to consider as ID. As MoGs can be sensitive to outliers, have a rather limited expressive power, their accuracy needs to be presented in the paper to showcase their effectiveness in helping collect the data to train the auxiliary task on.    \n\n[1]: How To Exploit Hyperspherical Embeddings For OOD detection? https://arxiv.org/pdf/2203.04450.pdf\n\nQUESTIONS:\nBased on the limitations I raised above, I would suggest the authors to perform a comparative study with some of the distance-based OOD detection approaches like CIDER. Additionally, evaluating the performance of the MoGs they used could help validate further the effectiveness of their approach.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}