Title: Learning to Augment Distributions for Out-of-distribution Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: OOD Detection
Abstract: Open-world classification systems should discern out-of-distribution (OOD) data whose labels deviate from those of in-distribution (ID) cases, motivating recent studies in OOD detection. Advanced works, despite their promising progress, may still fail in the open world, owing to the lacking knowledge about unseen OOD data in advance. Although one can access auxiliary OOD data (distinct from unseen ones) for model training, it remains to analyze how such auxiliary data will work in the open world. To this end, we delve into such a problem from a learning theory perspective, finding that the distribution discrepancy between the auxiliary and the unseen real OOD data is the key to affect the open-world detection performance. Accordingly, we propose Distributional-Augmented OOD Learning (DAOL), alleviating the OOD distribution discrepancy by crafting an OOD distribution set that contains all distributions in a Wasserstein ball centered on the auxiliary OOD distribution. We justify that the predictor trained over the worst OOD data in the ball can shrink the OOD distribution discrepancy, thus improving the open-world detection performance given only the auxiliary OOD data. We conduct extensive evaluations across representative OOD detection setups, demonstrating the superiority of our DAOL over its advanced counterparts.

Learning to Augment Distributions for Out-of-Distribution Detection

Abstract

Open-world classification systems should discern out-of-distribution (OOD) data whose labels deviate from those of in-distribution (ID) cases, motivating recent studies in OOD detection. Advanced works, despite their promising progress, may still fail in the open world, owing to the lack of knowledge about unseen OOD data in advance. Although one can access auxiliary OOD data (distinct from unseen ones) for model training, it remains to analyze how such auxiliary data will work in the open world. To this end, we delve into such a problem from a learning theory perspective, finding that the distribution discrepancy between the auxiliary and the unseen real OOD data is the key to affecting the open-world detection performance. Accordingly, we propose Distributional-Augmented OOD Learning (DAL), alleviating the OOD distribution discrepancy by crafting an OOD distribution set that contains all distributions in a Wasserstein ball centered on the auxiliary OOD distribution. We justify that the predictor trained over the worst OOD data in the ball can shrink the OOD distribution discrepancy, thus improving the open-world detection performance given only the auxiliary OOD data. We conduct extensive evaluations across representative OOD detection setups, demonstrating the superiority of our DAL over its advanced counterparts. The code is publicly available at: https://github.com/tmlr-group/DAL .

1 Introduction

Deep learning in the open world often encounters out-of-distribution (OOD) data of which the label space is disjoint with that of the in-distribution (ID) cases (Hendrycks and Gimpel, 2017; Fang et al., 2022). It leads to the well-known OOD detection problem, where the predictor should make accurate predictions for ID data and detect anomalies from OOD cases (Bulusu et al., 2020; Yang et al., 2021). Nowadays, OOD detection has attracted intensive attention in reliable machine learning due to its integral role in safety-critical applications (Cao et al., 2020; Shen et al., 2021).

OOD detection remains challenging since predictors can make over-confidence predictions for OOD data (Hendrycks et al., 2019), motivating recent studies towards effective OOD detection. Therein, outlier exposure (Hendrycks et al., 2019; Ming et al., 2022) is among the most potent ones, learning from auxiliary OOD data to discern ID and OOD patterns. However, due to the openness of the OOD task objective (Wang et al., 2023), auxiliary OOD data can arbitrarily differ from the (unseen) real OOD data in the open world. So, to formally understand their consequences, we model the difference

†

between auxiliary and real OOD data by their distribution discrepancy, measured by the Wasserstein distance (Villani, 2021, 2008). Then, we reveal the negative impacts of such OOD distribution discrepancy on the real detection power, with a larger distribution discrepancy indicating a lower performance on real OOD data, cf., Eq. (4).

The OOD distribution discrepancy threatens the open-world detection performance for outlier exposure. Therefore, we raise a natural question in how to alleviate such an OOD distribution discrepancy . Hence, this paper establishes a promising learning framework named DistributionalAugmented OOD Learning (DAL). Therein, we augment the auxiliary OOD distribution by crafting an OOD

Real OOD Data

Discrepancy

DistributionDiscrepancy

after DAL

Auxiliary,OOD Data

WassersteinBall

distribution set containing all distributions in a Wasserstein ball (Villani, 2021, 2008), centered on the auxiliary OOD distribution. Then, by making the predictor learn from the worst OOD distribution in the set, cf., Eq. (8), one can alleviate the negative impacts of the distribution discrepancy. Moreover, our proposed framework enjoys the learning guarantees towards the expected risk with respect to the real OOD distribution, making OOD detection stay effective when facing unseen data (cf., Theorem 3). Figure 1 provides a conceptual explanation: learning from the worst OOD distribution ensures the uniformly well performance inside the Wasserstein ball, enlarging the influence of the auxiliary OOD distribution. Thus, one can shrink the OOD distribution discrepancy between the auxiliary and the real OOD data and improve OOD detection.

In realization, the primal learning objective in Eq. (8) is generally intractable due to the infinitedimensional optimization for the worst OOD distribution search. Instead, we adopt the dual form with respect to the original learning problem (cf., Theorem 1), transforming it into a tractable problem of the worst OOD data search in a finite-dimensional space. Furthermore, following Du et al. (2022a); Mehra et al. (2022), the data search procedure is conducted in the embedding space, which can benefit the open-world performance of OOD detection with decent costs of additional computation.

We conduct extensive experiments over representative OOD detection setups, revealing the openworld performance of our method toward effective OOD detection. For example, our DAL reduces the average FPR95 by 1.99 to 13.46 on CIFAR benchmarks compared with the conventional outlier exposure (Hendrycks et al., 2019). Overall, we summarize our contributions into three folds:

We measure the difference between the auxiliary and the real OOD data by the Wasserstein distance, and establish an effective learning framework, named DAL, to mitigate the OOD distribution discrepancy issue. We further guarantee our performance with respect to unseen real OOD data via Theorem 3, which is new to previous works.

DAL leads to a practical method in Algorithm 1, learning from the worst cases in the Wasserstein ball to improve the open-world detection performance. Overall, our method solves the dual problem, which performs the worst-case search in the embedding space, which is simple to compute yet effective in OOD detection.

We conduct extensive experiments in Section 5 to evaluate our effectiveness, ranging from the well-known CIFAR benchmarks to the challenging ImageNet settings. The empirical results comprehensively demonstrate our superiority over advanced counterparts, and the improvement is mainly attributed to our distributional-augmented learning framework.

A detailed overview of existing OOD detection methods and theories can be found in Appendix A, and a summary of the important notations can be found in Appendix B.

2 Outlier Exposure

Let X denote the feature space and Y = { 1 , . . . , C } denote the label space with respect to the ID distribution. We consider the ID distribution D X I Y I , a joint distribution defined over X × Y , where X I and Y I are random variables whose outputs are from spaces X and Y . We also have an OOD joint distribution D X O Y O , where X O is a random variable from X , but Y O is a random variable whose outputs do not belong to Y , i.e., Y O / ∈ Y (Fang et al., 2022).

The classical OOD detection (Hendrycks and Gimpel, 2017; Yang et al., 2021) typically considers an open-world setting, where the real OOD data drawn from D X O Y O are unseen during training. Recently, Fang et al. (2022) have provided several strong conditions necessary to ensure the success of the classical OOD setting. Furthermore, to increase the possibility of success for OOD detection and weaken the strong conditions proposed by Fang et al. (2022), advanced works (Hendrycks et al., 2019; Chen et al., 2021) introduce a promising approach named outlier exposure , where a set of auxiliary OOD data is employed as a surrogate of real OOD data. Here, we provide a formal definition.

Problem 1 (OOD Detection with Outlier Exposure) . Let D X I Y I , D X O , and D X A be the ID joint distribution, the OOD distribution, and the auxiliary OOD distribution, respectively. Given the sets of samples called the ID and the auxiliary OOD data, namely,

outlier exposure trains a predictor f by using the training data S and T , such that for any test data x : 1) if x is an observation from D X I , the predictor f can classify x into its correct ID label; otherwise 2) if x is an observation from D X O , the predictor f can detect x as an OOD case.

OOD Scoring. Many existing methods detect OOD data by using various score-based strategies (Hendrycks and Gimpel, 2017; Lee et al., 2018a; Liu et al., 2020; Sun et al., 2022). In general, given a model f : X → R C and a scoring function s ( · ; f ) : X → R , the OOD detector g λ is given by:

where λ is a given threshold. For example, as a well-known baseline scoring function, the maximum softmax prediction (MSP) (Hendrycks and Gimpel, 2017) is given by:

with softmax k ( · ) denoting the k -th dimension of the softmax output.

Model and Risks. We denote f w : X → R C the predictor with parameters w ∈ W , with W the parameter space. We consider the loss functions ℓ and ℓ OE w.r.t. the ID and the OOD cases, respectively. Then, the expected and the empirical ID risks of the model f w can be written as:

The expected and the empirical auxiliary OOD risks are then given by

and the expected real OOD risk is given by R O ( w ) = E x ∼ D X O ℓ OE ( f w ; x ) . Accordingly, we can define the expected detection risk with respect to real OOD data, following

where α is the trade-off parameter.

Learning Strategy. After the scoring function is selected, one can obtain the OOD detector if the model f w is given. Under the Problem 1 of outlier exposure, a common learning strategy is to optimize the empirical ID and auxiliary OOD risk jointly (Hendrycks et al., 2019), namely,

Note that the auxiliary OOD data are employed in Eq. (3), which can arbitrarily differ from the real OOD cases. Then, it is generally expected that the predictor f w , trained over the auxiliary OOD data, can perform well even on unseen OOD data, i.e., a small value of R D ( w ) is expected.

3 Motivation

To the general learning strategy in Eq. (3), intuitively, if the auxiliary data are sampled from a distribution similar to real ones, the predictor will perform well for real OOD data. However, auxiliary and real OOD data differ in practice, posing us to suspect their open-world detection performance. To formally study the problem, we measure the difference between auxiliary and real OOD data in the distribution level, motivating our discussion of OOD distribution discrepancy .

Distribution Discrepancy. In this paper, we adopt a classical measurement for the distribution discrepancy-Optimal Transport Cost (Sinha et al., 2018; Mehra et al., 2022).

Definition 1 (Optimal Transport Cost and Wasserstein-1 Distance (Villani, 2021, 2008)) . Given a cost function c : X × X → R + , the optimal transport cost between two distributions D and D ′ is

where Π( D,D ′ ) is the space of all couplings for D and D ′ . Furthermore, if the cost c is a metric , then the optimal transport cost is also called the Wasserstein-1 distance.

Based on Definition 1, we use the distribution discrepancy to measure the difference between the auxiliary and the real OOD data, namely, W c ( D X O , D X A ) . Then, we can formally study the impacts of such a discrepancy on the detection performance of the predictor. Under certain assumptions (cf., Corollary 1), we can prove that with high probability, the following generalization bound holds:

where ̂ w is the parameter learned by Eq. (3), i.e., ̂ w ∈ arg min w ∈W ̂ R I ( w ) + α ̂ R A ( w ) , L c is the Lipschitz constant of ℓ OE w.r.t. the cost function c ( · , · ) (see Theorem 3). In general, the expected detection risk R D ( ̂ w ) measures the expected performance on unseen OOD data given the predictor trained on the auxiliary OOD data. Then, due to the upper bound, the impacts of the OOD distribution discrepancy are reflected by the Wasserstein-1 distance between the auxiliary and the real OOD data, i.e., W c ( D X O , D X A ) . Therefore, although classical outlier exposure can improve OOD detection to some extent, it fails to ensure reliable detection of unseen OOD data, in that a larger distribution discrepancy generally indicates a worse guarantee for open-world OOD detection.

The key to improve the detection performance is mitigating the negative impact induced by the OOD distribution discrepancy. To tackle this problem, a simple lemma inspires us:

Lemma 1. Let d ( · , · ) be the distance to measure the discrepancy between distributions. Given a space D consisting of some OOD distributions, if D X A ∈ D , then

If d ( · , · ) is the Optimal Transport Cost in Definition 1, the cost function c is a continuous metric, and D is the Wasserstein-1 ball with a radius ρ > 0 , i.e., D = { D X ′ :W c ( D X ′ , D X A ) ≤ ρ } , then

In the light of Lemma 1, we introduce a specific set of distributions D , augmented around the auxiliary OOD distribution. It makes it possible to mitigate the distribution discrepancy, following Eqs. (5) and (6). Therefore, instead of choosing a model f w that directly minimizes the empirical risk in Eq. (3), we target augmenting the auxiliary OOD data within the distribution space D , namely,

where ̂ D X A is the empirical form of D X A , i.e., ̂ D X A = 1 m ∑ m i =1 δ x i A and δ x i A is the dirac measure.

4 Learning Framework

This section proposes a general learning framework to mitigate the OOD distribution discrepancy. As aforementioned, we consider an augmented set of OOD distributions to improve OOD detection, thus named Distributional-Augmented OOD Learning (DAL).

To begin with, we need to select a suitable distribution space D for the tractable solutions of Eq. (7). Generally, the choice of D influences both the richness of the auxiliary data as well as the tractability of the resulting optimization problem. Previous works have developed a series of distribution spaces, e.g., the distribution ball based on f -divergences (Namkoong and Duchi, 2016; Michel et al., 2021) and maximum mean discrepancy (MMD) (Staib and Jegelka, 2019). However, there are several drawbacks for the distribution balls based on f -divergences and MMD: 1) any f -divergence-based space D contains only distributions within the same support set as ̂ D X A ; and 2) the effective solutions in the MMD-based space have not been provided (Staib and Jegelka, 2019).

Instead, motivated by Sinha et al. (2018); Mehra et al. (2022); Dai et al. (2023) and Theorem 1, we consider the Wasserstein ball. For any ρ > 0 , we define the augmented OOD distribution set as

and consider the following optimization problem:

However, the optimization problem in Eq. (8) is intractable due to the infinite-dimensional search for the distribution D X ′ . Fortunately, the following dual theorem provides a solution:

Theorem 1 (Blanchet and KarthyekRajhaaA. (2016)) . Let c ( · , · ) be a continuous metric and ϕ γ ( w ; x ) = sup x ′ ∈X { ℓ ( f w ; x ′ ) -γc ( x ′ , x ) } be the robust surrogate function. Then, for any ρ > 0 ,

Theorem 1 provides a feasible surrogate for the original optimization problem in Eq. (8), transforming the infinite-dimensional problem to its finite counterpart, i.e., the data feature search. We use Eq. (10) to design our algorithm, cf., Section 4.2.

4.1 Theoretical Supports

This section provides the theoretical support for our DAL. Specifically, 1) Theorem 2 shows that the empirical model given by Eq. (8) can achieve consistent learning performance, and 2) Theorem 3 further demonstrates the expected detection risk estimation, i.e., R D ( w ) , with respect to the empirical model given by Eq. (8). All the proofs can be found in Appendix C. To state our theoretical results, we use the notation R D ( w ; ρ ) to represent the ideal form of ̂ R D ( w ; ρ ) , which is defined by

Similar to Sinha et al. (2018), our results rely on the covering number (cf., Appendix C.1) for the model classes F = { ℓ ( f w ; · ) : w ∈ W} and F OE = { ℓ OE ( f w ; · ) : w ∈ W} to represent their complexity. Intuitively, the covering numbers N ( F , ϵ, L ∞ ) and N ( F OE , ϵ, L ∞ ) are the minimal numbers of L ∞ balls of radius ϵ > 0 needed to cover the model classes F and F OE , respectively. Now, we demonstrate that DAL can achieve consistent performance under mild assumptions.

Theorem 2 (Excess Generalization Bound) . Assume that 0 ≤ ℓ ( f w ; x , y ) ≤ M ℓ , 0 ≤ ℓ OE ( f w ; x ) ≤ M ℓ OE , and c ( · , · ) : X × X → R + is a continuous metric. Let ̂ w be the optimal solution of Eq. (8) , i.e., ̂ w ∈ arg min w ∈W ̂ R D ( w ; ρ ) . Then with the probability at least 1 -4 e -t > 0 ,

where where

for any ρ > 0 , where

where b 0 , b 1 and b 2 are uniform constants.

Algorithm 1 Distributional-Augmented OOD Learning (DAL)

Furthermore, under proper conditions, one can show that the bound in Eq. (11) can attain O (1 / √ n ) + O (1 / √ m ) , i.e., R D ( ̂ w ; ρ ) -min w ∈W R D ( w ; ρ ) ≤ O (1 / √ n )+ O (1 / √ m ) . Corollary 1 in Appendix C.5 gives an example to support the above claim. Next, we give a learning bound to estimate the expected detection risk in Eq. (2) w.r.t. the model f ̂ w given by Eq. (8).

Theorem 3 (Risk Estimation) . Given the same conditions in Theorem 2 and let ̂ w be the solution of Eq. (8) , which is given by ̂ w ∈ arg min w ∈W ̂ R D ( w ; ρ ) . If ℓ OE ( f w ; x ) is L c -Lipschitz w.r.t. c ( · , · ) , i.e., | ℓ OE ( f w ; x ) -ℓ OE ( f w ; x ′ ) | ≤ L c c ( x , x ′ ) , then with the probability at least 1 -4 e -t > 0 ,

for any ρ > 0 , where ϵ ( n, m ; t ) is defined in Theorem 2.

The bias term αL c max { W c ( D X O , D X A ) -ρ, 0 } = 0 when ρ is large enough. Hence, a large ρ implies a small estimation error. Although a larger ρ leads to better generalization ability, the approximate risk min w ∈W R D ( w ; ρ ) may become larger. It implies that for practical effectiveness, i.e., small R D ( ̂ w ) , there is a trade-off between the approximate risk min w ∈W R D ( w ; ρ ) and the bias αL c max { W c ( D X O , D X A ) -ρ, 0 } across different choices of ρ . Hence, we need to choose a proper ρ for open-world detection with unseen data (cf., Section 5.3).

4.2 Proposed Algorithm

In this section, we introduce the algorithm design for DAL, summarized in Algorithm 1. Due to the space limit, we provide further discussions in Appendix E.

Losses and Cost Function. Following Hendrycks et al. (2019), we adopt the cross entropy loss to realize ℓ and the KL-divergence between model predictions and uniform distribution for ℓ OE . We also define the cost function c by the l 1 norm, namely, c ( x , x ′ ) = ∥ x -x ′ ∥ 1 .

Algorithm Design. By Theorem 1, we can address the primary problem in Eq. (8) by the dual problem in Eq. (9). Additionally, following Du et al. (2022a), we perturb for the worst OOD data in the embedding space. Denote the model f w = h ◦ e with h the classifier and e the feature extractor, we find the perturbation p for the embedding features, i.e., e ( x ) , of the associated data x . The perturbation p should lead to the worst OOD case for the surrogate function in Theorem 1, namely,

where E denotes the space of embedding features. Note that we abuse the definition of ℓ OE , emphasizing that we perturb the embedding features of e ( x ) by p .

Training and Inference. Our definition of ϕ γ ( w ; e ( x )) leads to a particular realization of Eq. (10), which is the learning objective of our DAL. It can be solved by stochastic gradient optimization for deep models, e.g., mini-batch stochastic gradient descent. After training, we use the MSP scoring function by default and discuss the possibility of other scoring functions in Appendix F.3.

Stochastic Realization. Algorithm 1 gives a stochastic realization of DAL, where ID and auxiliary OOD mini-batches are randomly sampled in each stochastic iteration, denoted by S B and T B respectively. Therein, we first find the perturbation p that leads to the maximal ϕ γ ( w , e ( x )) . The value of p is initialized by random Gaussian noise with the standard deviation σ and updated by gradient ascent for num _ search steps with the perturbation strength ps . Then we update γ by one step of gradient descent with the learning rate β , and further clipping between 0 and γ max to avoid extreme values. Finally, given the proper perturbations for the auxiliary OOD data in T B , we update the model parameter w by one step of mini-batch gradient descent.

5 Experiments

In this section, we mainly test DAL on the CIFAR (Krizhevsky and Hinton, 2009) benchmarks (as ID datasets). To begin with, we introduce the evaluation setups.

OODDatasets. We adopt the 80 Million Tiny Images (Torralba et al., 2008) as the auxiliary OOD dataset; Textures (Cimpoi et al., 2014), SVHN (Netzer et al., 2011), Places 365 (Zhou et al., 2018), LSUN (Yu et al., 2015), and iSUN (Xu et al., 2015) as the (test-time) real OOD datasets. We eliminate those data whose labels coincide with ID cases.

Pre-training Setups. We employ Wide ResNet-40-2 (Zagoruyko and Komodakis, 2016) trained for 200 epochs via empirical risk minimization, with a batch size 64 , momentum 0 . 9 , and initial learning rate 0 . 1 . The learning rate is divided by 10 after 100 and 150 epochs.

Hyper-parameters Tuning Strategy. The hyper-parameters are tuned based on the validation data, separated from the training ID and auxiliary OOD data, which is a common strategy in OOD detection with outlier exposure field (Hendrycks et al., 2019; Chen et al., 2021). Specifically, we fix σ = 0 . 001 , num _ search = 10 , and adopt the grid search to choose γ max from { 0 . 1 , 0 . 5 , 1 , 5 , 10 , 50 } ; β from { 1 e -3 , 5 e -3 , 1 e -2 , 5 e -2 , 1 e -1 , 5 e -1 , 1 , 5 } ; ρ from { 1 e -2 , 1 e -1 , 1 , 10 , 100 } ; ps from { 1 e -3 , 1 e -2 , 1 e -1 , 1 , 10 , 100 } ; α from { 0 . 1 , 0 . 5 , 1 . 0 , 1 . 5 , 2 . 0 } .

Hyper-parameters Setups. For CIFAR-10, DAL is run for 50 epochs with the ID batch size 128 , the OOD batch size 256 , the initial learning rate 0 . 07 , γ max = 10 , β = 0 . 01 , ρ = 10 , ps = 1 , and α = 1 . For CIFAR-100, DAL is run for 50 epochs with the ID batch size 128 , the OOD batch size 256 , the initial learning rate 0 . 07 , γ max = 10 , β = 0 . 005 , ρ = 10 , and ps = 1 , and α = 1 . For both cases, we employ cosine decay (Loshchilov and Hutter, 2017) for the model learning rate.

Baseline Methods. We compare DAL with representative methods, including MSP (Hendrycks and Gimpel, 2017), Free Energy (Liu et al., 2020), ASH (Djurisic et al., 2023), ReAct (Sun et al., 2021), Mahalanobis (Lee et al., 2018a), KNN (Sun et al., 2022), KNN+ (Sun et al., 2022), CSI (Tack et al., 2020), VOS (Du et al., 2022a), Outlier Exposure (OE) (Hendrycks et al., 2019), Energy-OE (Liu et al., 2020), ATOM (Chen et al., 2021), DOE (Wang et al., 2023), and POEM (Ming et al., 2022). We adopt their suggested setups but unify the backbones for fairness.

Evaluation Metrics. The detection performance is evaluated via two representative metrics, which are both threshold-independent: the false positive rate of OOD data when the true positive rate of ID data is at 95% (FPR 95 ); and the area under the receiver operating characteristic curve (AUROC), which can be viewed as the probability of the ID case having greater score than that of the OOD case.

Due to the space limit, we test our DAL with more advanced scoring strategies in Appendix F.3 and conduct experiments on the more complex ImageNet (Deng et al., 2009) dataset in Appendix F.10.

5.1 Main Results

The main results are summarized in Table 1, where we report the detailed results across the considered real OOD datasets. First, we reveal that using auxiliary OOD data can generally lead to better results than using only ID information, indicating that outlier exposure remains a promising direction worth studying. However, as demonstrated in Section 3, the OOD distribution discrepancy can hurt its openworld detection power, while previous works typically oversee such an important issue. Therefore, our DAL, which can alleviate the OOD distribution discrepancy, reveals a large improvement over the original outlier exposure. Specifically, comparing with the conventional outlier exposure, our method reveals 1.99 and 0.13 average improvements w.r.t. FPR95 and AUROC on the CIFAR-10 dataset, and 13.46 and 3.65 of the average improvements on CIFAR-100 dataset. For advanced works that consider the OOD sampling strategies, e.g., ATOM and POEM, DAL can achieve much better results, especially for the CIFAR-100 case. The reason is that these methods mainly consider the situations

where the model capacity is not enough to learn from all the auxiliary OOD data, deviating from our considered issue in OOD distribution discrepancy. Moreover, for the previous works that adopt similar concepts in the worst-case OOD learning, e.g., VOS and DOE, DAL also reveals better results, with 1.29 and 30.89 improvements on the CIFAR-10 dataset and 11.27 and 44.12 improvements on the CIFAR-100 dataset w.r.t. FPR95. It indicates that our theoretical-driven scheme can also guide the algorithm designs with practical effectiveness. Note that many previous works use advanced scoring strategies other than MSP, and thus our experiment above is not completely fair to us. Therefore, in Appendix F.3, we also combine DAL with many advanced scoring strategies other than MSP, which can further improve our performance.

5.2 Hard OOD Detection

We further consider hard OOD scenarios (Sun et al., 2022), of which the test OOD data are very similar to that of the ID cases. Following the common setup (Sun et al., 2022) with the CIFAR10 dataset being the ID case, we evaluate our DAL on three hard OOD datasets, namely, LSUN-Fix (Yu et al., 2015), ImageNet-Resize (Deng et al., 2009), and CIFAR100 . Note that data in ImageNet-Resize ( 1000 classes) with the same semantic space as TinyImageNet ( 200 classes) are removed. We select a set of strong baselines that

are competent in hard OOD detection, summarizing the experiments in Table 2. As we can see, our method can beat these advanced methods across the considered datasets, even for the challenging

fter DRO

tein Ball

FPR95

Real OOD Data

Original Distribution

Discrepancy

𝜌

Distribution

Discrepancy after DRO

(a) Before DAL Training

(b) After DAL Training

Auxiliary OOD Data

Real OOD Data

Original Distribution

Discrepancy

FPR95

𝜌

(a)

ps = 1

(b)

ps = 10

𝜌

(a) Before DAL Training

(b) After DAL Training

CIFAR10 versus CIFAR100 setting. The reason is that our distributional augmentation directly learns from OOD data close to ID pattern, which can cover hard OOD cases.

5.3 Ablation Study

We further conduct an ablation study to demonstrate two mechanisms that mainly contribute to our open-world effectiveness, namely, OOD data generation and Wasserstein ball constraint.

OOD Data Generation. DAL learns from the worst OOD data to mitigate the OOD distribution discrepancy. To understand such an OOD generation scheme, we employ the t-SNE visualization (Van der Maaten and Hinton, 2008) for the ID, the auxiliary OOD, and the worst OOD data. Figure 2 summarizes the results before and after DAL training. Before training, the ID and auxiliary OOD data overlap largely, indicating that the original model is not effective at distinguishing between them. Then, DAL does not directly train the model on auxiliary OOD data but instead perturbs it to further confuse the model beyond the overlap region. After DAL training, the overlap region between ID and auxiliary OOD data shrinks. Additionally, perturbing the original OOD data becomes more difficult, indicating that the model has learned to handle various worst-case OOD scenarios.

Wasserstein Ball Constraint. The choice of ρ determines the radius of the Wasserstein ball. Larger values of ρ reduce estimation error and improve model generalization, as stated in Theorem 3. However, larger values of ρ also increase the approximate risk min w ∈W R D ( w ; ρ ) as it becomes more challenging to ensure uniform model performance with increased distributional perturbation. Figure 3 shows the FPR95 curves on the CIFAR-100 dataset for both the real and the surrogate OOD data, revealing the trade-off in selecting ρ . Here, we consider two setups of ρ , i.e., ps = 1 (default) and ps = 10 (large perturbation strength). First, when the perturbation strength is very large (i.e., ps = 10 ), the model can easily fail for training if the value of ρ is also large (e.g., ρ = 100 ), indicating that large value of ρ can lead to a large approximation error. However, such an issue can be overcome by selecting a relatively small value of ρ (e.g., ρ = 0 . 5 ).

6 Conclusion

Outlier exposure is one of the most powerful methods in OOD detection, but the discrepancy between the auxiliary and (unseen) real OOD data can hinder its practical effectiveness. To address such an issue, we have formalized it as the OOD distribution discrepancy and developed an effective learning framework to mitigate its negative impacts. Specifically, we consider a specific distribution set that contains all distributions in a Wasserstein ball centered on the auxiliary OOD distribution. Then, models trained over worst-case OOD data in the ball can ensure improved performance toward open-world OOD detection. Overall, as pioneers in critically analyzing the open-world setting with theoretical analysis, we are committed to raising attention to the OOD distribution discrepancy issue and encouraging further research in this direction.

𝜌

Acknowledgments and Disclosure of Funding

QZW, YGZ, and BH were supported by the NSFC Young Scientists Fund No. 62006202, NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011652, HKBU Faculty Niche Research Areas No. RC-FNRA-IG/22-23/SCI/04, and HKBU CSD Departmental Incentive Scheme. FL was supported by Australian Research Council (ARC) under Award No. DP230101540, and by NSF and CSIRO Responsible AI Program under Award No. 2303037. YXL was supported by the AFOSR Young Investigator Program under award number FA9550-23-1-0184, National Science Foundation (NSF) Award No. IIS-2237037 & IIS-2331669, and Office of Naval Research under grant number N00014-23-1-2643.

References

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR , 2017.

Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribution detection learnable? In NeurIPS , 2022.

Saikiran Bulusu, Bhavya Kailkhura, Bo Li, P Varshney, and Dawn Song. Anomalous instance detection in deep learning: A survey. Technical report, Lawrence Livermore National Lab., 2020.

Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: a survey. arXiv preprint arXiv:2110.11334 , 2021.

Tianshi Cao, Chin-Wei Huang, David Yu-Tung Hui, and Joseph Paul Cohen. A benchmark of medical out of distribution detection. arXiv preprint arXiv:2007.04250 , 2020.

Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624 , 2021.

Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. In ICLR , 2019.

Yifei Ming, Ying Fan, and Yixuan Li. POEM: Out-of-distribution detection with posterior sampling. In ICML , 2022.

Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander, Tongliang Liu, Jianye Hao, and Bo Han. Out-of-distribution detection with implicit outlier transformation. In ICLR , 2023.

Cédric Villani. Topics in Optimal Transportation . American Mathematical Society, 2021.

Cédric Villani. Optimal Transport: Old and New . Springer, 2008.

Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. VOS: Learning what you don't know by virtual outlier synthesis. In ICLR , 2022a.

Akshay Mehra, Bhavya Kailkhura, Pin-Yu Chen, and Jihun Hamm. On certifying and improving generalization to unseen domains. arXiv preprint arXiv:2206.12364 , 2022.

Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. ATOM: Robustifying out-ofdistribution detection using outlier mining. In ECML , 2021.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS , 2018a.

Weitang Liu, Xiaoyun Wang, John D Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurIPS , 2020.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In ICML , 2022.

Aman Sinha, Hongseok Namkoong, and John C. Duchi. Certifying some distributional robustness with principled adversarial training. In ICLR , 2018.

Hongseok Namkoong and John C. Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. In NIPS , 2016.

Paul Michel, Tatsunori Hashimoto, and Graham Neubig. Modeling the second player in distributionally robust optimization. In ICLR , 2021.

Matthew Staib and Stefanie Jegelka. Distributionally robust optimization and generalization in kernel methods. In NeurIPS , 2019.

Rui Dai, Yonggang Zhang, Zhen Fang, Bo Han, and Xinmei Tian. Moderately distributional exploration for domain generalization. In ICML , 2023.

José H. Blanchet and M. KarthyekRajhaaA. Quantifying distributional model risk via optimal transport. Risk Management eJournal , 2016.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto , 2009.

Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence , 30(11):1958-1970, 2008.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR , 2014.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop , 2011.

Bolei Zhou, Àgata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(6):1452-1464, 2018.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755 , 2015.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC , 2016.

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR , 2017.

Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In ICLR , 2023.

Yiyou Sun, Chuan Guo, and Yixuan Li. ReAct: out-of-distribution detection with rectified activations. In NeurIPS , 2021.

Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: novelty detection via contrastive learning on distributionally shifted instances. In NeurIPS , 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: a large-scale hierarchical image database. In CVPR , 2009.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research , 9(11):2579-2605, 2008.

Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR , 2018.

Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't know? In NeurIPS , 2021a.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS , 2017.

Rui Huang and Yixuan Li. MOS: towards scaling out-of-distribution detection for large semantic space. In CVPR , 2021.

Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In ICML , 2020.

Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. ViM: Out-of-distribution with virtuallogit matching. In CVPR , 2022a.

Ziqian Lin, Sreya Dutta Roy, and Yixuan Li. MOOD: Multi-level out-of-distribution detection. In CVPR , 2021.

Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detection. In AAAI , 2022.

Yadan Luo, Zijian Wang, Zhuoxiao Chen, Zi Huang, and Mahsa Baktashmotlagh. Source-free progressive graph learning for open-set domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(9):11240-11255, 2023.

Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In NeurIPS , 2021.

Conor Igoe, Youngseog Chung, Ian Char, and Jeff Schneider. How useful are gradients for ood detection really? arXiv preprint arXiv:2205.10439 , 2022.

Jianing Zhu, Hengzhuang Li, Jiangchao Yao, Tongliang Liu, Jianliang Xu, and Bo Han. Unleashing mask: Explore the intrinsic out-of-distribution detection capability. In ICML , 2023a.

Vikash Sehwag, Mung Chiang, and Prateek Mittal. SSD: A unified framework for self-supervised outlier detection. In ICLR , 2021.

Haotao Wang, Aston Zhang, Yi Zhu, Shuai Zheng, Mu Li, Alex J Smola, and Zhangyang Wang. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. In ICML , 2022b.

Haotian Zheng, Qizhou Wang, Zhen Fang, Xiaobo Xia, Feng Liu, Tongliang Liu, and Bo Han. Out-of-distribution detection learning with unreliable out-of-distribution sources. In NeurIPS , 2023.

Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li. SIREN: Shaping representations for detecting out-of-distribution objects. In NeurIPS , 2022b.

Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out-of-distribution detection? In ICLR , 2023.

Alireza Zaeemzadeh, Niccolò Bisagno, Zeno Sambugaro, Nicola Conci, Nazanin Rahnavard, and Mubarak Shah. Out-of-distribution detection using union of 1-dimensional subspaces. In CVPR , 2021.

Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICML , 2022.

Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distribution examples via augmenting content and style. In ICLR , 2023a.

Jianing Zhu, Geng Yu, Jiangchao Yao, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Bo Han. Diversified outlier exposure for out-of-distribution detection via informative extrapolation. In NeurIPS , 2023b.

Yi Li and Nuno Vasconcelos. Background data resampling for outlier-aware classification. In CVPR , 2020.

Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield highconfidence predictions far away from the training data and how to mitigate the problem. In CVPR , 2019.

Taewon Jeong and Heeyoung Kim. OOD-MAMl: Meta-learning for few-shot out-of-distribution detection and classification. In NeurIPS , 2020.

Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In ICML , 2020.

Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In ICLR , 2018b.

Sachin Vernekar, Ashish Gaurav, Vahdat Abdelzad, Taylor Denouden, Rick Salay, and Krzysztof Czarnecki. Out-of-distribution detection in classifiers via generation. In NeurIPS Workshop , 2019.

Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. In ICLR , 2023.

Yadan Luo, Zijian Wang, Zi Huang, and Mahsa Baktashmotlagh. Progressive graph learning for open-set domain adaptation. In ICML , 2020.

Lily H. Zhang, Mark Goldstein, and Rajesh Ranganath. Understanding failures in out-of-distribution detection with deep generative models. In ICML , 2021.

Zhen Fang, Jie Lu, Anjin Liu, Feng Liu, and Guangquan Zhang. Learning bounds for open-set learning. In ICML , 2021.

Julian Bitterwolf, Alexander Meinke, Maximilian Augustin, and Matthias Hein. Breaking down out-of-distribution detection: Many methods based on OOD training data estimate a combination of the same core quantities. In ICML , 2022.

Roman Vershynin. High-Dimensional Probability . Springer, 2018.

Jose Blanchet, Yang Kang, and Karthyek Murthy. Robust wasserstein profile inference and applications to machine learning. Journal of Applied Probability , 56(3):830-857, 2019.

Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In COLT , 2018.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018.

Qizhou Wang, Feng Liu, Bo Han, Tongliang Liu, Chen Gong, Gang Niu, Mingyuan Zhou, and Masashi Sugiyama. Probabilistic margins for instance reweighting in adversarial training. In NeurIPS , 2021b.

Zhuo Huang, Miaoxi Zhu, Xiaobo Xia, Li Shen, Jun Yu, Chen Gong, Bo Han, Bo Du, and Tongliang Liu. Robust generalization against photon-limited corruptions via worst-case sharpness minimization. In CVPR , 2023b.

Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(12):10045-10067, 2021.

Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang. Adversarial robustness through the lens of causality. In ICLR , 2022.

Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. ImageNet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972 , 2021.

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The iNaturalist species classification and detection dataset. In CVPR , 2018.

Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. In ICML , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper noticed that the discrepancy between the auxiliary and (unseen) real OOD data hindered the effectiveness of outlier exposure, which is a potential method for OOD detection. To tackle this issue, the authors proposed a learning framework that augments the auxiliary OOD dataset using a Wasserstein ball. By training the model on the worst-case OOD data in the ball, the performance of the model toward open-world OOD detection will be improved.\n\nSTRENGTHS:\n- The paper is well-motivated: the authors formalized the challenges in existing outlier exposure methods for OOD detection and provide a solution to the challenges.\n- The proposed DAL framework is proposed with firm theoretical support with a convergence guarantee.\n- Experiments show that using ID and OOD auxiliary samples and using the proposed DAL can effectively improve the performance of OOD detection, even when ID and real OOD samples are very close. The results confirm the design of DAL.\n\nWEAKNESSES:\n- Experiments are conducted on relatively simple datasets and models. It is worth to be shown that the proposed method still has improvements on more complicated samples.\n\nQUESTIONS:\n- Given the Wasserstein ball is applied to the feature distribution, is it possible to train the classifier only and leave the encoder as pre-trained?\n- Will the false negative rate of ID samples be increased under DAL, i.e., ID samples being misclassified as OOD samples, especially in hard OOD detections?\n- Why choose 10 searches in experiments? Will increased searches improve performance? How long does it take for the DAL training?\n\nLIMITATIONS:\nThe authors do not address the limitations of the method. Possible improvements can be: can the model be easily applied to (1) regression tasks? (2) models that are hard to be fully fine-tuned, i.e., transformers.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a theory and an algorithm to address the OOD detection problem better. The whole paper is well-structured, and the algorithm is designed via the developed theory. The basic idea is to simulation the OOD distributions as representative as possible, thus we can expect that the detector can work well in the most scenarios. Experiments verify the effectiveness of the proposed method. All theoretical results are proved in the appendix.\n\nSTRENGTHS:\n1. This paper's structure is very clear, and readers enjoy reading this paper. It starts from a classical but powerful solution of OOD detection problem. Then, a theory follows to demonstrate the effectiveness of this kind of solution. Finally, am algorithm is designed via the proposed theory.\n  \n2. The proposed algorithm is verified in many benchmark datasets, including hard OOD situation, which is a strong evidence to support the claims made in this paper.\n\n3. The studied problem is very important, and the focused research direction is practical. Although there are many non-outlier methods used to detect OOD data, we can always obtain some outliers to help our training, which is practical.\n\nWEAKNESSES:\n1. There might be some gaps between the algorithm and theory. One might be addressed during the response phase: In Theorem 2, there are two assumptions regarding the loss function. How are the both assumptions satisfied in your algorithm? Some discussions should be added here.\n\n2. In lines 218-219, the reason why we need to perturb data in the embedding space should be clarified. You indeed follow previous work, but more explanations are needed here.\n\n3. In lines 228-235, more justifications should be added here. I did not see any support (from literature or experiments) for this kind of design.\n\n4. Experiments regarding CIFAR-10 can be moved to appendix, as all methods have good performance. It is not meaningful to see that one method is better than the other by 0.2%. In this way, more experiments in appendix can be moved back to the main content.\n\n5. It is very interesting to see the real validation set here, more details can be included in the main content.\n\nQUESTIONS:\nSee the weakness.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper aims to address the outlier exposure in OOD detection from theoretical perspective. By developing a novel generalization bound based on the distributional augment principle, the paper proposes a novel outlier exposure method. Experiments indicate the effectiveness of the proposed method.

STRENGTHS:
1. A novel generalization bound to understand outlier exposure. It seems this theory is the first theory in outlier exposure field.  
2. The proposed algorithm is theoretical guided and guarantee. From my personal view, this is important  to reliable ML.
3. The features trained based on proposed DAL can be used with other post-hoc OOD detection method. This method can be combined with previous works well and naturally.
4. Extensive experiments have shown the effectiveness of the method. It seems the performance has achieved SOTA performance compared with latest works.

WEAKNESSES:
1. I am confused that it seems the radius rho becomes larger, the estimation error in Theorem 3 will decrease. So, from the theoretical view, why not selected rho larger enough? 
2. How to tune the parameters rho, gamma….? Validate set is allowed in outlier exposure?
3. It seems your algorithm and theory have a small gap. Could you clarity why you do that? It is better to give more clear explanations.

QUESTIONS:
Please answer the questions in Weakness.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to tackle the problem of out-of-distribution (OOD) detection in supervised learning through a learning theoretic approach. The authors propose a method termed DAL, which intends to reduce the discrepancy between the detector's training distribution (auxiliary OOD distribution) and true OOD distribution by expanding the set of auxiliary distributions over which the detector is trained. The paper outlines a rigorous approach to open-world OOD detection. DAL is evaluated experimentally and shown to outperform competing techniques.\n\nPost-rebuttal update: increasing score to 7\n\nSTRENGTHS:\nStrengths:\n1. The authors recognize the shortcomings in prior work on OOD detection and motivate the problem towards their proposed solution\n2. While I am not an expert in the area, using Wasserstein type metrics and controlling the distribution discrepency between true and auxiliary OOD distributions appeals to mean\n3. Through Theorems 2 and 3, the paper shows that DAL has a theoretical grounding and gives an algorithm to achieve the DAL objective\n4. While I have not checked the proofs in sufficient detail, I believe that the analysis is rigorous\n5. The experiments show the improvements achieved by DAL over several recent competing approaches\n\nWEAKNESSES:\nWeaknesses:\n1. Theorems 2 and 3 can benefit from a detailed discussion in the main body\n2. This could be a question as well - are common corruptions of CIFAR10 or Imagenet not used as standard datasets for OOD detection?\nAlso, please comment on my questions below\n\nQUESTIONS:\nQuestions:\n1. Line 88: Does this mean that the labels in ID and OOD are mutually exclusive? Could you please explain if this is a significant assumption\n2. Line 124: Please provide references for the statement \"the auxiliary and the real OOD data differ largely in practice\". I would like to understand how significant is the typical difference.\n3. As a follow-up, how large can the upper bound in (4) grow? Are there any standard assumptions made for theoretical analysis?\n4. What does the assumption in Theorem 2 regarding l <= M imply- suppose l is cross-entropy or exponential loss, does the bound mean that f_w needs to be a reasonably good solution? Otherwise, the loss function could be unbounded, if I am not mistaken. How mild are these assumptions?\n\nLIMITATIONS:\nIt would be useful if the authors discussed a few most relevant open challenges, in light of their findings", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}