The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection

Authors: qingyang  zhang
qiuxuan  feng
joey tianyi zhou
yatao  bian
qinghua  hu
changqing  zhang
tencent ai lab
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-28 04:09:44

Abstract: Out-of-distribution (OOD) detection is essential for model trustworthiness which aims to sensitively identity semantic OOD samples and robustly generalize for covariate-shifted OOD samples. However, we discover that the superior OOD detection performance of state-of-the-art methods is achieved by secretly sacrificing the OOD generalization ability. The classification accuracy frequently collapses catastrophically when even slight noise is encountered. Such a phenomenon violates the motivation of trustworthiness and significantly limits the model's deployment in the real world. What is the hidden reason behind such a limitation? In this work, we theoretically demystify the "\textit{sensitive-robust}" dilemma that lies in previous OOD detection methods. Consequently, a theory-inspired algorithm is induced to overcome such a dilemma. By decoupling the uncertainty learning objective from a Bayesian perspective, the conflict between OOD detection and OOD generalization is naturally harmonized and a dual-optimized performance could be expected. Empirical studies show that our method achieves superior performance on commonly used benchmarks. To our best knowledge, this work is the first principled OOD detection method that achieves state-of-the-art OOD detection performance without sacrificing OOD generalization ability. Our code is available at https://github.com/QingyangZhang/DUL.

Main paper: 
 introduction: 
 Endowing machine learning models with out-of-distribution (OOD) detection and OOD generalization ability are both essential for their deployment in the open world [1, 2, 3 ]. We borrow an example of autonomous driving from [4] to demonstrate the motivation of these two tasks. Given a machine learning model trained on in-distribution (ID) data (top image in Fig. 1 (a) ), OOD detection aims to sensitively perceive uncertainty arising upon outliers that do not belong to any known classes of training data [5] (bottom right image in Fig. 1 (a) ). While OOD generalization expects machine learning models to be robust in the presence of unexpected noise or corruption, e.g., rainy or snowy weather (bottom left image in Fig. 1 (a) ). In this paper, we reveal that many previous methods pursue OOD detection performance at a secret cost of sacrificing OOD generalization ability. To make things worse, we observe that some SOTA OOD detection methods may result in a catastrophic collapse in classification performance (∼15% accuracy degradation) when encountering even slight noise. One pioneering work [4] makes a trade-off between OOD detection and OOD generalization, but the relationship between these two tasks is still largely unexplored. The learning objectives of these two tasks are seemingly conflicting at first glance. OOD detection encourages sensitive uncertainty awareness (highly uncertain prediction) on unseen data, while generalization expects the prediction to be confident and robust under unforeseeable distributional shifts. Previous work in OOD detection Figure 1 : (a): Models trained on in-distribution (ID) data inevitably encounter distributional shifts during their deployment. OOD generalization expects the model to correctly classify covariate-shifted data that undergoes noise or corruption due to environmental issues. OOD detection aims to identify samples that do not belong to any known classes for trustworthiness consideration. (b): Limitations of current advanced OOD detection methods. We consider 8 representative OOD detection methods including the baseline method MSP [6] (without any OOD detection regularization), Entropy [7] , EBM [8] , Bayesian [9] , SOTA OOD detection methods WOODS [10] , POEM [11] , recent advanced SCONE [4] which aims to seek for a good trade-off and the proposed DUL. All these methods exhibit a degraded generalization ability compared to baseline method MSP and lie in a trade-off area except our DUL. The goal of this paper is to understand and mitigate this phenomenon.research area [4] characterizes the relationship between OOD detection and OOD generalization as a trade-off and thus striking for a balanced performance. However, this trade-off significantly limits the employment of current state-of-the-art OOD detection methods. Naturally, one might require the model to be aware of the OOD input for ensuring safety, but certainly does not expect to sacrifice the generalization ability, not to mention that the catastrophically collapsed classification performance under noise or corruption.In this work, we first uncover the potential reason behind this limitation by characterizing the generalization error lower bound of previous OOD detection methods, which is referred to sensitiverobust dilemma. To overcome the dilemma, we devise a novel Decoupled Uncertainty Learning (DUL) framework for dual-optimal performance. The decoupled uncertainties are separately responsible for characterizing semantic OOD (detection) and covariate-shifted OOD (generalization). Thanks to the decoupled uncertainty learning objective, dual-optimal OOD detection and OOD generalization performance could be expected. Our emphasis lies on a particular category of OOD detection methods in the classification task, including max softmax probability (MSP) based model [6] , energy-based model (EBM) [8] and Bayesian methods [9] . This selection offers two-fold advantages. First, MSP, EBM and Bayesian detectors encompass major OOD detection advances in classification task [5] . Second, numerous OOD detection works in diverse learning tasks (classification, object detection [12] , time-series prediction and image segmentation) are all roughly related to classification [13] . The contributions of this paper are summarized as follows:• This paper reveals that existing SOTA OOD detection methods may suffer from catastrophic degradation in terms of OOD generalization. That is, their superior OOD detection ability is achieved by (secretly) sacrificing OOD generalization ability. We theoretically demystify the sensitive-robust dilemma in learning objectives as the main reason behind such a limitation.• In contrast to previous works that characterize OOD detection and generalization as conflictive learning tasks and thus implying an inevitable trade-off, we propose a novel learning framework termed Decoupled Uncertainty Learning (DUL) to successfully break through the limitation beyond a simple trade-off. Our DUL substantially harmonizes the conflict between OOD detection and OOD generalization, which achieves the best OOD detection performance without sacrificing the OOD generalization ability.• We conduct extensive experiments on standard benchmarks to validate our findings. Our DUL achieves dual-optimal OOD detection and OOD generalization performance. To our best knowledge, DUL is the first method that gains state-of-the-art OOD detection performance without sacrificing OOD generalization ability.Uncertainty estimation in Bayesian framework. In the Bayesian framework, predictive uncertainty can be regarded as an indicator of whether the input sample is prone to be OOD. Since OOD samples are unseen during training and thus should be of higher uncertainty than ID. The overall predictive uncertainty of a classification model can be decomposed into three factors according to their source, including data (aleatoric) uncertainty (AU), distributional uncertainty (DU), and model (epistemic) uncertainty (EU) [31, 9] . AU measures the natural complexity of the data (e.g., class overlap, label noise) and EU results from the difficulty of estimating the model parameters with finite training data. DU arises due to the mismatch between the distributions of test and training data. A line of classic measurement can be used to capture various types of uncertainty including entropy, mutual information, and differential entropy [9] .Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract and introduction accurately reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In conclusion and Appendix. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 We consider K-class classification task with classifier f : X → R K parameterized by θ, where X is the input space and Y = {1, 2, ..., K} denotes the target space. The model output f θ (x) is considered as logits. The k-th element in logits is denoted as f k (x) indicates the confidence of predicting x to class k. The predicted distribution F (x) is obtained by normalizing f (x) with the softmax function.We first formalize all possible distributions that the model might encounter.• In-distribution P ID X Y which denotes the distribution of labeled training data. • Covariate-shifted OOD P COV X Y which is relevant to OOD generalization. P COV X Y is of the same label space with ID. However, its marginal distribution P COV X encounters shifts due to unexpected noise or corruption.• Semantic OOD P SEM X Y ′ is the distribution of data that do not belong to any known class. Its label space has no overlap with the known ID label space, i.e., Y ′ ∩ Y = ∅.In the following paper, we omit the subscript for simplicity. The goal of OOD detection is to build a detector G : X → [IN, OUT] to decide whether an input x is semantic OOD data or not through a thresholding function G deduced from classifier fEQUATION)where γ is the threshold. g f is an OOD scoring function deduced from f , which is expected to assign a higher value to OOD than ID. For example, in MSP detectors, g f (x) = -max k F (x) where F (x) is the predicted softmax probability (negative max softmax probability). In EBM detectors, g f is realized by the energy function E(x; f ) := -log K i=1 e f k (x) and the semantic input of OOD should be of high energy [8] . Since it is difficult to foresee P SEM one will encounter, a board line of OOD detection works [7, 8, 9, 11, 17, 12, 32, 33] regularize the model on some auxiliary OOD data P SEM train during training (e.g., data from the web or other datasets), and expect the model can learn useful heuristic to handle unknown test-time OOD P SEM test . The learning objective is shown as followsEQUATIONwhere L CE is the standard cross entropy loss for the original classification task. L reg is the OOD detection regularization term depending on the detector used, which generally encourages a high uncertainty on P SEM train . For example, L reg is set to cross entropy between F (x) and the uniform distribution for MSP detector [7] . In EBM detectors [8] , L reg is realized as a margin ranking loss to explicitly encourage a large energy gap between ID and semantic OOD. In this paper, we are interested in this setting for the following reasons:1) In contrast to labeled data in supervised learning literature, auxiliary OOD data can be unlabeled and easy-to-collected in practice [11] . 2) Most SOTA methods involve auxiliary outliers [5, 20] for superior performance.3) Even under some strict assumptions that P SEM train is unavailable, recent works utilize GAN [15] , diffusion model [34] or sampling strategy [12] to generate "virtual" outliers for training.Thus we believe this setting is promising and the cost of auxiliary outliers is minor given the importance of ensuring model trustworthiness. At test-time, the model is evaluated in terms of • ID accuracy (ID-Acc ↑) which measures the model's performance on P ID , • OOD accuracy (OOD-Acc ↑) measures the OOD generalization ability on P COV , • False positive rate at 95% true positive rate (FPR95↓) := E x∼P SEM test (I(G γ (x) = IN)) measures the OOD detection ability, where γ is chosen when true positive rate (TPR) is 95%. I is the indicator function. In OOD detection, ID samples are considered as negative.It is worth noting that in the standard OOD detection setting [11, 4] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]  
sensitive-robust dilemma of out-of-distribution detection: 
 In this section, we detail the limitation of current OOD detection methods: their OOD detection performance is achieved at the cost of generalization ability. This limitation implies the potential risk of SOTA OOD detection methods and underscores the urgent need for a better solution. Firstly, we re-examine representative OOD detection methods of six different types, including 1) baseline model MSP that is trained without any OOD detection regularization [6] , 2) entropy-regularization (Entropy) that encourages high predictive entropy on OOD [7] , which is devised for MSP detectors, 3) energy-regularization for EBM detectors that enforces the output with high energy score for OOD input [8] , 4) Bayesian uncertainty learning that encourages high overall uncertainty on OOD [9] , 5) state-of-the-art OOD detection methods WOODS [10] and POEM [11] 6) the most related SCONE [4] that seeks for a trade-off between OOD detection and generalization performance.Limitation of current OOD detection methods. In Fig. 1 (b), we investigate current OOD detection methods in terms of OOD classification error and FPR95. The expected classifier should yield both low OOD classification error and FPR95. As it is observed, despite the superior OOD detection performance, all above methods significantly underperform the baseline MSP in terms of OOD generalization. By contrast, our method (DUL) successfully overcomes the limitation.Theoretical justification. Toward understanding the limitation, we provide theoretical analysis for two types of most popular OOD detection methods, i.e., MSP and EBM detectors. Our analysis identifies the "sensitive-robust" dilemma as the main reason behind such a limitation. The roadmap of our analysis is: (1) inspired by transfer learning theory, we first reveal that OOD detection regularization applied on semantic OOD may also affect the behavior of model on covariate-shifted OOD; (2) then we demonstrate why MSP detectors suffer from poor generalization by characterizing its generalization error bound; (3) we further identify that EBM methods [8] suffer from a similar drawback when incorporating with gradient-based optimization. First of all, we recap the definition of disparity discrepancy in transfer learning theory [35, 27] . Definition 1 (Disparity with Total Variation Distance). Given two hypotheses f ′ , f ∈ F and distribution P , we define the Disparity with Total Variation Distance between them asEQUATIONwhereF f ′ , F f are the class distributions predicted by f ′ , f respectively. T V (•||•) is the total variation distance, i.e., T V (F f ||F f ′ ) = 1 2 K k=1 ||F f,k -F f ′ ,k ||.Definition 2 (Disparity Discrepancy with Total Variation Distance, DD with TVD). Given a hypothesis space F and two distributions P, Q, the Disparity Discrepancy with Total Variation Distance (DD with TVD) is defined asEQUATIONDisparity discrepancy (DD) measures the "distance" between two distributions P, Q which considers the hypothesis space. DD is one of the most fundamental conceptions in transfer learning theory which constrains the behavior of hypothesis in F should not be dissimilar substantially on different distributions P and Q. 3 If the DD between semantic OOD and covariate-shifted OOD is limited, one can suppose that OOD detection regularization applied to semantic OOD samples will also influence the model's behavior on covariate-shifted OOD. Thus encouraging high uncertainty on semantic OOD may also result in highly uncertain prediction on covariate-shifted OOD, which is potentially harmful to generalization ability. We first formalize this intuition for MSP detectors. Theorem 1 (Sensitive-robust dilemma). Let P COV , P SEM test be the covariate-shifted OOD and semantic OOD distribution. GError P COV (f ) denotes standard cross entropy loss taking expectation on P COV , i.e., generalization error. Then we haveGError P COV (f ) OOD generalization error ≥ C - 1 8κ 2 E P SEM test [ L reg (f )OOD detection lossEQUATIONwhere L reg is the OOD detection loss devised for MSP detectors defined in [7] , i.e., cross-entropy between predicted distribution F (x) and uniform distribution. d F (P COV , P SEM test ) is DD with TVD that measures the dissimilarity of covariate-shifted OOD and semantic OOD. C and κ are both some constants depending on hypothesis space F, P COV and P SEM test .The proof is deferred in Appendix A. ) is large enough. However, since the semantic OOD samples can be any samples that do not belong to ID classes, one can suppose that semantic OOD samples are extremely diverse and some are of high similarity with ID and covariate-shifted OOD [37] . Detecting these "ID-like" OOD samples is inherently the core challenge of OOD detection [11, 17, 38] . Thus, it is reasonable to assume a limited d F (P COV , P SEM test ). We provide more discussions in the Appendix D.2.As presented before, the key limitation of MSP detectors is that they enforce high-entropy prediction on semantic OOD. We proceed to reveal that EBM detectors suffer from similar issues due to the natural property of gradient-based optimization. For EBM detectors, L reg is defined by [8] isEQUATIONwhich constrains the energy score E(x; f ) := -log K k=1 e f k (x) of ID sample x to be lower than that of OOD sample x. m IN , m OUT are manually selected margins. Although such regularization does not indicate high entropy prediction at first glance, unfortunately, we demonstrate that EBM detectors also tend to uncertain prediction when equipped with gradient-based optimization. Here we focus on Gradient Descent (GD) as a showcase. In each training epoch t, model f θ is updated with GD as θ t+1 = θ t -η∇ θ L reg , where η is the learning rate. For any sample x drawn from P SEM , the gradient ∇ θ L reg can be written asEQUATIONwhere m OUT -E(x) > 0 (otherwise the gradient is zero) and f k (x) is the predicted logits on the k-th class. For sample x, when class k has larger predicted logit, it contributes more to the overall gradient ∇ θ L reg and thus could obtain more optimization efforts during backpropagation. Eventually, one can infer that when an EBM detector is about to converge, it tends to high-entropy prediction on P SEM train accordingly. Incorporating this into the established Theorem 1, this is likely to harm the generalization ability of P COV . Empirical evidence can support this supposition (see Table 14 in Appendix C). Therefore both MSP and EBM detectors face the "sensitive-robust" dilemma. 
decoupled uncertainty learning: 
 We demonstrate how to handle the dilemma between OOD detection and generalization by decoupled uncertainty learning in the Bayesian framework. Unlike the most related work [4] which aims to seek a good trade-off, our method successfully gets out of the aforementioned sensitive-robust dilemma.Uncertainty Estimation in Bayesian Framework. We first revisit the theoretical properties of different types of uncertainty in a Bayesian framework. Non-Bayesian classifiers consider the model's output f (x) as logits, which is then normalized with softmax to directly model predictive categoricals p(ŷ|x). While in Bayesian framework [9] , f (x) is considered as parameters of a Dirichlet distribution p(µ|x) firstly, which is used to model the prior of predictive categoricals p(ŷ|x) byEQUATIONwhere Dir(µ|α) is Dirichlet distribution and α is the concentration parameters of Dirichlet. The sum of all α k ∈ α (noted as α 0 ) is so called the strength of the Dirichlet distribution, i.e., α k > 0, α 0 = k α k . After obtaining prior p(µ|x, θ), the final predicted posterior p(ŷ|x) over class labels is given by calculating the mean of the Dirichlet prior p(ŷ|x, θ)EQUATIONFrom a Bayesian perspective, given deterministic parameters θ, the overall uncertainty of final prediction p(ŷ|x) can be decomposed into two factors, including data (aleatoric) uncertainty (AU) and distributional uncertainty (DU). DU lies in p(µ|x, θ) which is defined as uncertainty due to the mismatch between the distributions of test and train data. AU is described by p(ŷ|µ) which captures the natural complexity of the data (e.g., class-overlap) [9] . By definition, OOD detection is primarily associated with DU which is only a part of the overall uncertainty. While generalization is related to the overall uncertainty of p(ŷ|x) as we mentioned in related works (both AU and DU). One essential property of DU is that it can be high even if the expected categorical p(ŷ|x, θ) expresses low overall uncertainty. Such a property is well suited to achieve OOD detection and generalization jointly since high DU no longer necessarily indicates high overall uncertainty.Decoupled Uncertainty Learning. While the aforementioned Bayesian framework enjoys theoretical potentiality, its learning object [9] lacks consideration of OOD generalization. Similar to other OOD detection methods, it also directly enforces high overall uncertainty on OODEQUATIONwhere p(y|x), p(ŷ|x) are the ground-truth distribution and predicted distribution on ID. The model's prediction on OOD is enforced to be close to a rather flat Dirichlet distribution. It is worth noting that Dir(µ|α = 1) means all classes are equiprobable, and the entropy of the final prediction is maximized. As shown in Fig. 1 (b), the vanilla Bayesian method [9] also suffers from degraded OOD generalization performance. To this end, we propose Decoupled Uncertainty Learning (DUL), a novel OOD detection regularization method that explicitly encourages high DU on OOD samples without affecting the overall uncertainty. Similarly to previous OOD detection methods [8] , our DUL is also devised in a finetune manner for effectiveness. Given a classifier f θ0 well pre-trained on P ID , the goal of DUL lies in enhancing its OOD detection performance without sacrificing any generalization ability. Specifically, we finetune the model by encouraging higher DU but non-increased overall uncertainty on P SEM train . The learning objective of DUL ismin θ E (x,y)∼P ID [L CE (f (x), y)] ID classification +λ E x∼P SEM train ||max(0, (h 0 + m OUT ) -h)|| τ high distributional uncertainty (detection) s.t. H(p(ŷ|x)) = H(p 0 (ŷ|x))non-increased overall uncertainty (generalization)EQUATIONwhere H(•) is the entropy. p(ŷ|x) and p 0 (ŷ|x) are the predicted distribution on semantic OOD data x after and before finetuning. The first term is the original ID classification loss. The second term is OOD detection loss, which encourages high DU on outlier x. m OUT and τ > 0 are hyperparameters. h 0 , h are DU on x before and after finetuning. Here we measure DU with the differential entropy(h[p(µ|x, θ)] = -S K-1 p(µ|x)ln(p(µ|x))dµ, S is a K-simplex).We refer interested readers to the Appendix D.1 for mathematical details. The third term constraining on H(p(ŷ|x)) avoids increment of overall uncertainty during finetuning and thus the generalization ability can be retained. Considering the difficulty of constrained optimization, we convert Eq. 11 into an unconstrained form and get our final minimizing objectiveEQUATIONwhere γ is hyperparameter. In contrast to previous Bayesian method [9] , DUL only encourages high DU rather than overall uncertainty on OOD and explicitly discourages high entropy in the final prediction. The implementation details are in Appendix D.1. 
experiment: 
 We conduct experiments to validate our analysis and the superiority of DUL. The questions to be verified are Q1 Motivation. To what extent does OOD detection conflict with OOD generalization in previous methods? Q2 Effectiveness. Does DUL achieve better OOD detection and generalization performance compared to its counterparts? Q3 Interpretability. Does the proposed method well decouple uncertainty as expected?Our settings follow the common practice [8, 11, 20, 5] in OOD detection. Here we present a brief description and more details about datasets, metrics, and implementation are in Appendix B.1 and B.2.• ID datasets P ID . We train the model on different ID datasets including CIFAR-10, CIFAR-100 and ImageNet-200 (a subset of ImageNet-1K [39] with 200 classes). • Auxiliary OOD datasets P SEM train . In CIFAR experiments, we use ImageNet-RC as P SEM train . ImageNet-RC is a downsampled variant of the original ImageNet-1K which is widely adopted in previous OOD detection Table 1 : OOD detection and generalization performance comparison. Substantial (≥ 0.5) improvement and degradation compared to the baseline MSP [6] (training without any OOD detection regularization) are highlighted in blue or red, respectively. The best and second-best results are in bold or underlined. DUL is the only method that achieves SOTA OOD detection performance (mostly the best or second best) without sacrificing generalization i.e., the value of the entire row is either blue or black. Full results with standard deviation and diverse types of corruption are in Appendix C. works [8, 11, 17] . We also conduct experiments on the recent TIN-597 [20] as an alternative.When ImageNet-200 is ID, the remaining 800 classes termed ImageNet-800 are considered as P SEM train . • OOD detection test sets P SEM test are a suite of diverse datasets introduced by commonly used benchmark [5] . In CIFAR experiments, we use SVHN [40] , Places365 [41] , Textures [42] , LSUN-R, LSUN-C [43] and iSUN [44] as P SEM test . When P ID is ImageNet-200, P SEM test consists of iNaturlist [45] , Open-Image [46] , NINCO [47] and SSB-Hard [48] . It is worth noting that in standard OOD detection settings, there should be no overlapped classes between P ID , P SEM train and P SEM test , otherwise OOD detection is a trivial problem. • OOD generalization test sets P COV is the original ID test set corrupted with additive Gaussian noise of N (0, 5), following [4] . Besides, we also conduct experiments on CIFAR10-C, CIFAR100-C and ImageNet-C which involve 15 diverse types of different noise or corruption (e.g., snow, rain, frost, fog...) in Appendix C.For OOD detection performance evaluation, we report the average FPR95, AUROC and AUPR to be consistent with [11] . OOD generalization ability is compared in terms of classification accuracy (OOD-Acc). Besides, we also report classification accuracy on ID test sets (ID-Acc).Compared methods. We compare DUL with a board line of OOD detection methods, including auxiliary OOD required and auxiliary OOD free methods. • Auxiliary OOD-free methods do not require P SEM train during training, including MSP [6] , Maxlogits [49] , pretrained EBM [8] and Mahalaobis [18] . • Auxiliary OOD-required methods explicitly regularize the model on P SEM train , including entropy-regularization (Entropy) [7] , finetuned EBM [8] , DPN of Bayesian framework [9] , POEM [11] and WOODS [10] . We also compare our DUL to recent advanced SCONE [4] which aims to keep a balance between OOD detection and generalization.Dilemma between OOD detection and generalization (Q1). We validate the dilemma mentioned before in Fig. 1 . As shown in Tab. 1, though many advanced methods establish superior OOD detection performance, their OOD generalization degrades a lot. For example, recent SOTA POEM achieves nearly perfect OOD detection performance on CIFAR10 when ImageNet-RC serves as P SEM train with 3.32% false positive error rate (FPR95). However, its OOD-Acc drops a lot (about 10%) compared to baseline MSP. This phenomenon is also observed in other advanced methods. To further detail this phenomenon, we reduce the weight of OOD detection regularization terms in Entropy and finetuned EBM and show the performance on both OOD detection and generalization. As shown in Table 3 , when the regularization strength increases, OOD detection performance improves (lower FPR.), while the OOD generalization performance degrades (higher error rate).OOD detection and generalization ability (Q2). As shown in Tab. 1, DUL establishes strong overall performance in terms of both OOD detection and generalization. We highlight a few essential observations: 1) Compared to auxiliary OOD free methods, DUL establishes substantial improvement due to additional regularization on auxiliary outliers. 2) Compared to auxiliary OOD required methods, our method achieves superior OOD detection performance without sacrificing generalization ability. Meanwhile, previous OOD detection methods commonly exhibit severely degraded classification accuracy, with many cases increasing by more than 10% error rate. 3) Comparison to the most related work SCONE [4] . Despite recent advanced SCONE simultaneously considering both two targets, we observe that it can be hard to find a good trade-off. In contrast, dual-optimal OOD detection and generalization performance is achieved by our DUL. Noted that DUL is the only method that achieves state-of-the-art detection performance (mostly the best or second best) without degraded generalization ability (no red values in the entire row). The sensitive-robust dilemma is no longer observed in our method. These observations justify our expectation of DUL. 4) Combining with existing methods. Besides, to further demonstrate the effectiveness of the proposed DUL, we also add the unchanged overall uncertainty term in Eq.12 to the original Entropy and finetuned EBM. The results in Table 2 show that DUL regularization can also benefit EBM. However, combining Entropy with our regularization can not improve the accuracy substantially. This is not surprising, since the target of Entropy (high entropy prediction) and our DUL (non-increased entropy) directly conflict according to Theorem 1. 5) Comparison to methods with an extra OOD detect branch. Different from aforementioned methods, a line of recent OOD detectors [50, 51, 52, 17] employ extra output branches aside from the classification logits (with a shared backbone for feature extraction). For these OOD detectors, our theoretical analysis is not directly applicable and further analysis from a feature learning perspective may be needed in future work. However, the proposed DUL is devised in a finetune manner. Compared to OOD detectors with extra output branches that requires re-training the classifier from scratch, DUL can be applied to any pre-trained model (e.g., from torchvision, huggingface), with modest computation overhead.To evaluate the uncertainty estimation, we visualize the distribution of ID (CIFAR-10) and OOD (SVHN) samples in terms of uncertainty. As we can see in Fig. 2 (b), our DUL establishes a distinguishable (distributional) uncertainty gap between test-time ID and OOD data, which indicates a good sensitiveness for OOD detection. By contrast, the baseline method MSP (Fig. 2 (a) ) can not effectively discriminate ID and OOD. Besides, we visualize the predictive entropy (overall uncertainty) on covariate-shifted OOD (CIFAR-10 with Gaussian noise) in Fig. 2 (c), our DUL yields much lower entropy compared to other methods. Besides, we visualize the data uncertainty on semantic OOD test data (Textures) when CIFAR-10 is ID in Fig. 6 .2. The investigated methods are 1) pretrained model training on ID dataset only, 2) finetuned model with OOD detection regularization (ablating the last term in Eq.12), and 3) finetuned model with the full DUL method described by Eq.12. As shown in Fig. 6 .2, to keep the overall uncertainty and enlarge the distributional uncertainty (for OOD detection), the data uncertainty must be reduced. We use Eq.17 from [9] to calculate data uncertainty. The distributional uncertainty is shifted by subtracting that on ID dataset. These results meet our expectation.  
conclusion: 
 This paper provides both theoretical and empirical analysis towards understanding the dilemma between OOD detection and generalization. We demonstrate that the superior OOD detection performance of current advances are achieved at the cost of generalization ability. The theory-inspired algorithm successfully removes the conflict between previous OOD detection and generalization methods. For SOTA OOD detection performance, our implementation assumes that auxiliary outliers are available during training. This limitation is noteworthy for our DUL as well as the most existing SOTA OOD detection methods. We argue that this added cost is minor and reasonable given the significance of ensuring model trustworthiness in open-environments. Reducing the dependency on auxiliary OOD data can be an interesting research direction for the future exploration.  
 
 Appendix: First, we recap the definitions of Disparity with Total Variation Distance and Disparity Discrepancy. Definition 3 (Disparity with Total Variation Distance). Given two hypotheses f ′ , f ∈ F and distribution P , we define the Disparity with Total Variation Distance between them asEQUATIONwhere F f , F f ′ are the class distributions predicted by f ′ , f respectively. T V (•||•) is the total variation distance (TVD), i.e., T V (F f ||F f ′ ) = 1 2 K k=1 ||F f,k -F f ′ ,k || 1 .Definition 4 (Disparity Discrepancy with Total Variation Distance, DD with TVD). Given a hypothesis space F and two distributions P, Q, the Disparity Discrepancy with Total Variation Distance (DD with TVD) is defined asEQUATION)Since TVD is a distance measurement of two distribution. It yields the triangle equality. That is, for any distribution P X support on X and hypotheses f 1 , f 2 and f 3 ∈ F, we haveEQUATIONTo prove Theorem 1, we need the following lemmas. Lemma 1. For any f ∈ F, we haveEQUATION)where λ is a constant independent of f . U is the K-classes uniform distribution. P COV is the covariate-shifted OOD distribution. P SEM test is the semantic OOD distribution.Proof. Let f * be the hypothesis which jointly minimizes the total variance distance between the predicted distribution F f with uniform distribution U taking expectation on P COV and P SEM test , which is to sayEQUATION) Set λ = E x∼P COV [T V (F f * (x)||U )] + E x∼P SEM test [T V (F f * (x)||U )],then by the triangle equality we haveEQUATIONIntuitively speaking, Lemma 1 demonstrates that if the classifier f express high overall uncertainty on P SEM test (i.e., the predicted distribution F f is close to uniform distribution), it will also tend to high uncertain prediction on P COV given a limited d F (P SEM test , P COV ). Lemma 2. [Inequality between KL and TV] For any K-class distribution P and Q on {1, • • • , K} and κ > 0, the following inequality holdsEQUATIONProof. The proof can be found in [53] page 8.Lemma 3. Denote the OOD detection loss used for MSP detectors as L reg , then we haveEQUATIONwhere T V (•||•) is total variance distance (TVD). U denotes uniform distribution support on Y = {1, 2 • • • K}. L reg is defined in [7] is the cross-entropy between predicted distribution F f (x) and uniform distribution U .Lemma 2 means that minimizing the OOD detection loss will constrains the predicted distribution to be close to uniform distribution, which is a intuitive and straightforward result.Proof. In K-classes classification task, for any sample x drawn from P SEM X , we haveEQUATIONApplying Pinsker's Inequality, the following inequality holdsEQUATIONNoted that H(U ) = log K, we can re-write above inequality asEQUATIONThen, by taking expectation on P SEM X we can get the result. Now we are ready to present the proof of Theorem 1.Proof. By the definition, the generalization error can be written asEQUATIONwhere P true is the target distribution given input x (i.e., the true class distribution) and L CE (•) denotes the cross-entropy loss.Applying Lemma 2, for any x we haveEQUATIONBy the sub-additivity of TVD, we haveEQUATIONTaking expectation on P COV X Y , we haveEQUATIONApplying Lemma 1 and Lemma 2,EQUATIONGiven the fact that P COV X Y , P true are both fixed, H(P true ) and T V (P true ||U ) are constants for each x. Finally, we getEQUATIONwhereC = 1 2κ (E P COV X T V (P true ||U ) -λ + 8) + E P COV X Y H(P true ) is a constant.B.1 Datasets details ID datasets P ID . ID datasets are chosen following common practice in OOD detection. We use CIFAR-10, CIFAR-100 and ImageNet-200 as P ID . ImageNet-200 is a subset of the original ImageNet-1K introduced by [20, 5] .Auxiliary OOD datasets P SEM train . For CIFAR experiments, we use ImageNet-RC and TIN-597 as auxiliary datasets. ImageNet-RC is a down-sampled variant of the ImageNet-1K, which consists of 1000 classes and 1,281,167 images. We also conduct experiments on TIN-597 as an alternative for ImageNet-RC. TIN-597 is introduced by recent work [5] . The resolutions of ID and auxiliary samples are both 64 × 64. For ImageNet experiments, we use a subset of ImageNet-1K consisting of 200 classes as ID datasets. The remaining images belong to other 800 classes are utilized as auxiliary datasets. The resolutions of ID and auxiliary images are both 224 × 224.OOD detection test datasets P SEM test . In CIFAR experiments, following standard practice [8] , we use SVHN [40] , Textures [42] , Places365 [41] , iSUN [44] , LSUN-C and LSUN-R [43] to evaluate the OOD detection performance. • The SVHN test set comprises 26,032 color images of house numbers. • Textures (Describable Textures Dataset, DTD) consists of 5,640 images depicting natural textures. • Places365 dataset consists scenic images of 365 different categories. Each class consists of 900 images. • The iSUN dataset is a subset of the SUN database with 8,925 images. • The Large-scale Scene Understanding dataset (LSUN) comprises a testing set with 10,000 images of 10 different scenes. LSUN offers two datasets, LSUN-C and LSUN-R. In LSUN-C, the original high-resolution images are randomly cropped into 32 × 32. Meanwhile, in LSUN-R, the images are resized to 32 × 32. In ImageNet experiments, we follow the settings of [5] , where OpenImage-O [54] , SSB-hard [55] , Textures [42] , iNaturalist [45] and NINCO [47] OOD generalization test datasets P COV . Following previous work [4] , we corrupt the original test data with Gaussian noise of zero mean and variance of 5 in the main paper. In appendix, we conduct additional experiments involving CIFAR10-C, CIFAR100-C and ImageNet-C [56] with 15 diverse types of noise.We use WideResNet- 40-10 [57] as the backbone network, which comprises 40 layers. The widen factor is set to 10. We use SGD optimizer to train all methods with dropout strategy. The dropout rate is 0.3. The momentum is set to 0.9 and weight decay is set to 0.0005.Pretraining details. The pretrained model is obtained by training WideResNet-40-10 for 200 epochs with an initial learning rate of 0.1. We decay the learning rate by a factor of 0.2 at the 60-th, 120-th, and 160-th epochs. Batch size is set to 128.Finetuning details. • For Entropy and EBM, we finetune the pretrained model for 20 epochs with an initial learning rate of 0.001, utilizing a cosine annealing strategy to adjust the learning rate. Following the official implementation, the weight of OOD detection regularization term is set to 0.5 and 0.1 for Entropy and EBM (finetune) respectively. The hyperparameters m ID and m OOD in EBM regularization learning are set to -25 and -7 respectively. The ID batch size is 128 and the OOD batch size is set to 256. • For SCONE, we finetune the pretrained model for 10 epochs with an initial learning rate of 0.0002, utilizing a cosine annealing strategy to adjust the learning rate. The batch size is 32, the OOD batch size is 64. The margin of the OOD detection boundary is set to 1. To be aligned with most previous works in OOD detection and generalization, we assume P COV X is unavailable during finetuning. • For WOODS, we finetune the pretrained model for 10 epochs with an initial learning rate of 0.0002, utilizing a cosine annealing strategy to adjust the learning rate. The ID batch size is 32, the OOD batch size is 64. Other hyperparameters settings are consistent with SCONE. • For DUL, α 0 is set to 12. While finetuning on CIFAR10, the m ID and m OOD are set to 10 and 30 respectively. The weight λ, γ are set to 0.3 and 2. We train for 20 epochs with an initial learning rate of 0.00005, utilizing a cosine annealing strategy to adjust the learning rate. While finetuning on CIFAR100/TIN-597, the m ID and m OOD are set to 10 and 30 respectively. The weights λ, γ are set to 0.05 and 2 respectively. We finetune for 20 epochs with an initial learning rate of 0.00005, utilizing a cosine annealing strategy to adjust the learning rate. While finetuning on CIFAR100/ImageNet-RC, we set h 0 = 0. The m ID and m OOD are set to -430 and -370 respectively. We train for 30 epochs with an initial learning rate of 0.0001, utilizing a cosine annealing strategy to adjust the learning rate.The weights λ, γ are set to 0.1 and 1 respectively. For CIFAR-100/ImageNet-RC, we set τ = 2 and otherwise τ = 1. • For DUL † , we use Thompson sampling strategy [11] for OOD informativeness mining. The sampling hyperparameters are consistent with that of POEM.Training from scratch details. • For POEM, we train from scratch for 200 epochs with an initial learning rate of 0.1, and decay the learning rate by a factor of 0.2 at the 60-th, 120-th, and 160-th epochs following [57] . The ID and OOD batch size are set to 128 and 256 respectively. Following the official implementation, the pool of outliers consists of randomly selected 400,000 samples from auxiliary datasets, and only 50,000 samples (same size as the ID training set) are selected for training based on the boundary score. • For DPN, we train for 200 epochs with an initial learning rate of 0.1, and decay the learning rate by a factor of 0.2 at the 60-th, 120-th, and 160-th epochs. The Dirichlet parameters α are calculated by performing ReLU plus one on the model's outputs, i.e., α = ReLU(f (x)) + 1. α 0 is set to 15 and 12 respectively when training on CIFAR10 and CIFAR100. The auxiliary datasets are ImageNet and TIN-597. The ID and OOD batch size are set to 128 and 256 respectively. When training on CIFAR100/TIN-597, the OOD regularization weight λ is set to 0.05. In other cases, λ is set to 0.5.We use ResNet18 [58] as the backbone network. We use SGD optimizer to train all the models. The momentum is set to 0.9.Pretraining details. The pretrained model is obtained by training ResNet18 for 100 epochs with an initial learning rate of 0.1, utilizing a cosine annealing strategy to adjust the learning rate. The weight decay is set to 0.0001. Batch size is set to 64.Finetuning details. • For Energy regularized learning, we finetune the pretrained model for 10 epochs with an initial learning rate of 0.001, utilizing a cosine annealing strategy to adjust the learning rate. The weight decay is set to 0.0001. Following the official implementation, the weights of OOD detection regularization term are set to 0.1. Specifically, the m ID and m OOD in energy regularization method are set to -25 and -7 respectively. The ID batch size is 64 and the OOD batch size is set to 128. • For Entropy, we finetune the pretrained model for 10 epochs with an initial learning rate of 0.001, utilizing a cosine annealing strategy to adjust the learning rate. The weight decay is set to 0.0001. The ID and OOD batch size are set to 64 and 128 respectively. Following the official implementation, the weights of OOD detection regularization term are set to 0.5.• For SCONE, we finetune the pretrained model for 10 epochs with an initial learning rate of 0.0002, utilizing a cosine annealing strategy to adjust the learning rate. The weight decay is set to 0.0005. The batch size is 32, the OOD batch size is 64. The margin of the OOD detection boundary is set to 1. To be aligned with most previous works in OOD detection and generalization, we assume P COV X is unavailable during finetuning. • For WOODS, we finetune the pretrained model for 10 epochs with an initial learning rate of 0.0002, utilizing a cosine annealing strategy to adjust the learning rate. The batch size is 32, the OOD batch size is 64. Other hyperparameters of WOODS are consistent with SCONE. • For our DUL, the Dirichlet parameters α are calculated by performing ReLU and exp operation on the model's outputs, i.e., α = exp(ReLU(f (x))). For numerical stability on large scale benchmark, we measure the distributional uncertainty by the strength of Dirichlet distribution. λ, γ are set to 0.1 and 4 respectively. We set τ = 1 in large-scale ImageNet experiments.Training from scratch details. • For DPN, we train ResNet18 for 100 epochs with an initial learning rate of 0.1, utilizing a cosine annealing strategy to adjust the learning rate. The weight decay is set to 0.0001. Batch size is set to 64. The Dirichlet parameters α are calculated by performing ReLU plus one on the model's outputs, i.e., α = ReLU(f (x)) + 1. The ID classification loss is set to KL-divergence between predicted class distribution under Dirichlet prior and target distribution because of the inconvenience of directly setting α 0 . The target distribution is obtained by label smoothing strategy with parameter of 0.01 [9] . The weight of regularization term applied on OOD auxiliary samples is 1.Input : ID data P ID , auxiliary outliers P SEM train , classifier f θ0 pretrained on P ID . Output : finetuned classifier f θ 1 Initialize θ = θ 0 ; 2 for each iteration do 3 Obtain ID sample (x, y) from P ID and auxiliary outlier x from P SEM train ;4 Update model parameters θ by minimizing objective defined in Eq. 12; 5 endC.1 Uncertainty estimation.We add Gaussian noise with zero mean and varying variance ϵ on CIFAR-10 and investigate the estimated distributional uncertainty and overall uncertainty. Distributional uncertainty is measured by differential entropy. It clear that with DUL regularization, the prediction yields a low overall uncertainty and high distributional uncertainty on covariate-shifted data. We conduct experiments on CIFAR-10/ImageNet-RC and CIFAR-10/TIN-597, tabular results are shown in Tab. 4. Table 4 : Mean value of estimated uncertainty on CIFAR-10-C with varying severity of Gaussian noise with zero mean and variance of ϵ. Full results with standard deviation are presented this section. In CIFAR experiments, we report the mean and standard deviation in 5 random runs. In ImageNet experiments, we report the mean and standard deviation in 3 random runs to be consist with [5] . CIFAR experimental results are shown in Tab. 6. Large-scale ImageNet results are shown in Tab. 7. Here we provide empirical supports to our intuition about energy-based OOD detection regularization. We calculate the entropy of predicted distribution before and after finetuning with Energy regularization [8] . The results show can support our claim in Section 4. where α 0 is the strength of Dirichlet, i.e., α 0 = K α k . α k denotes the k-th element in α. Γ is the Gamma function and ψ is the digamma function. Here we provide a PyTorch implementation on how to calculate distribution uncertainty measured by differential entropy.def diff_entropy ( alphas ) : alpha0 = torch . sum ( alphas , dim = 1 ) return torch . sum ( torch . lgamma ( alphas ) -( alphas -1 ) * ( torch . digamma ( alphas )torch . digamma ( alpha0 ) . unsqueeze ( 1 ) ) , dim = 1 ) -torch . lgamma ( alpha0 ) logits = model ( x ) alpha = torch . Relu ( logits ) + 1 diff_entropy = diff_entropy ( alpha )We refer interested readers to [9] and Gal's PhD Thesis [59] for more detailed math derivations.In section 4, we claim that a limited disparity discrepancy between test-time semantic OOD and covariate-shifted OOD is practical. Here we provide some empirical evidence and discussion to support such a claim.The key challenge of OOD detection lies in identifying ID-like semantic OOD. As mentioned in recent works [38] , effectively distinguishing between the most challenging OOD samples that are much like in-distribution (ID) data is the core challenge of OOD detection. Recent works regularize models on ID-like OOD to enhance the OOD detection performance. Since the ID-like semantic OOD samples are more difficult to be detected and more informative. For example, NTOM [17] and POEM [11] utilizes greedy and Thompson sampling strategies to find semantic OOD samples which are more closely to ID. [38] proposes to explicitly discover outliers near ID by prompt learning.Semantic OOD and covariate OOD can be very similar in practice. As shown in Fig. 4 . There exists many similar samples from semantic OOD and covariate OOD in large-scale commonly used benchmarks. We borrow some examples from recent works [47] to show case. AI safety and trustworthiness are closely related to our work. This paper presents work to harmonize the conflicts between out-of-distribution detection methods and model generalization. The proposed method puts effort to enhance machine learning models for their safely deployment on out-ofdistribution data, avoiding both undesirable behavior and degraded performance in challenging high-stake tasks. However, due to the bias from data used by current OOD detection benchmark, e.g., large-scale ImageNet, the ones using the proposed method need to carefully consider the selection of auxiliary outliers for safety-critical applications.Encompassed by[36] as a special case, our definition is realized using TVD for theoretical convenience. 


Decision: This paper investigated a very interesting problem: the relationship between OOD detection ability and OOD generalization ability, suggesting us be aware of purely pursuing the OOD detection ability in practice and theory. All reviewers agree that this contribution is valuable to the field.

Reviews: review: paper_summary: paper_summary: This paper addresses out-of-distribution (OOD) detection and generalization problems. The authors show the sensitive-robust dilemma in learning objectives of OOD detection and generalization and propose a decoupled uncertainty learning (DUL) method to harmonize the above conflict. The proposed method only encourages high distribution uncertainty on OOD data and explicitly discourages high entropy in the final prediction. Experiments on some datasets show the DUL method achieves better performance compared to several baselines.
strength_weakness: strength_weakness: 1. While analyzing the sensitive-robust dilemma via transfer learning theory is novel, the phenomenon that existing SOTA OOD detection methods may suffer from catastrophic degradation in terms of OOD generalization has been reported in [4]. 

2. The decoupled uncertainty learning objective introduces an additional regularization term named unchanged overall uncertainty. It is difficult to understand why DUL even performs better than Entropy [7] and EBM (finetune) [8] on OOD detection tasks.

3. Only simple Gaussian noise is used to reflect the OOD generalization ability, while there are many types of out-of-distributions like corruptions. Therefore, the results in Table 1 can not represent the performance of evaluated methods because different kinds of OOD data often behave differently under the same model [A]. 

[A] OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization, CVPR 2022.
questions: questions: 1. According to Eq.9, if keep the overall uncertainty and enlarge the distributional uncertainty (for OOD detection), the data uncertainty must be reduced. Could the authors provide the visualization of data uncertainty?

2. The reviewer suggests two more baselines. (1) In the original Entropy [7] or EBM (finetune) [8], reduce the weight of outlier regularization terms and show the performance on both OOD detection and generalization. (2) Add the unchanged overall uncertainty term in Eq.12 to the original Entropy [7] or EBM (finetune) [8], and report the performance. Those two baselines would help to demonstrate the effectiveness of the bayesian framework in decoupled uncertainty learning.
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: This paper reveals the trade-off dilemma of OOD detection and OOD generalization for current SOTA OOD detectors from both theoretical and empirical perspectives. The authors employ a transfer learning framework to analyze the generalization error bound for MSP-based OOD detectors and identify the optimization conflicts between the training objectives of OOD detection and OOD generalization. They find similar patterns for Energy-based detectors. From the theoretical analysis, this paper derives a decoupled uncertainty learning technique for the dual-optimal performance for OOD detection and generalization. Experiments on CIFAR10/100 and ImageNet-200 validate its effectiveness.
strength_weakness: strength_weakness: 1. This paper only discusses the MSP- and Energy-based OOD detectors, which adopt exactly the same outputs (i.e., logits) from a neural network to perform ID classification (and OOD generalization) and OOD detection, where the optimization conflict is intuitive. However, other superior OOD detectors employ extra output branches aside from the classification logits (even if they share the same backbone for feature extraction) for OOD detection, such as [1][2][3], which shows promising robustness for covariate shifts[2]. Is the theoretical analysis and DUL technique still applicable to those types of OOD detectors?
2. As the authors claim,  DUL can lead to a dual-optimal performance for both OOD detection and OOD generalization. However, according to Table 1, DUL does not always win first place across different experimental settings. For instance, DUL and $\dagger$ DUL are surpassed by POEM on OOD detection performance by a non-negligible margin in the CIFAR10/100+ImageNet-RC setting. It seems DUL just achieves a better trade-off rather than the dual-optimal results. A proper explanation should be given.
3. DUL seems to basically add a normalization/regularization on the model's output logits with knowledge (measured by Bayesian framework) distilled from a pre-trained ID classifier. It limits the application and novelty of this paper. What happens if the model is trained from scratch?
4. Besides, adding Gaussian noise is a too trivial setting to evaluate the robustness of covariate shifts. How about evaluating OOD detectors in standard domain adaptation settings?

[1] J Bitterwolf et al. “Breaking Down Out-of-Distribution Detection: Many Methods Based on OOD Training Data Estimate a Combination of the Same Core Quantities”. ICML 2022.

[2] L Kai et al. "Category-Extensible Out-of-Distribution Detection via Hierarchical Context Descriptions". NeurIPS 2023.

[3] W Miao et al. "Out-of-Distribution Detection in Long-Tailed Recognition with Calibrated Outlier Class Learning". AAAI 2024.
questions: questions: Although OOD detection and OOD generalization are both frequently discussed in the literature, it is still weird when they come up together. It is basically because the D(istribution) for OOD detection and OOD generalization has different meanings. For OOD detection, it means the collection of label y. For OOD generalization, it means the data distribution for sample x. A proper clarification or modification can be taken into consideration
limitations: limitations: Please see weakness.
paper_summary: paper_summary: This paper theoretically characterizes the underlying dilemma in SOTA OOD detection method.  The authors find that OOD detection performance of state-of-the-art methods is achieved with tradeoff between OOD detection and classification. Accordingly, the authors provide an uncertainty-based strategy which decouples these two tasks and thus addresses the dilemma. Plenty of experiments are conducted which well support the theory and validate the effectiveness of the proposed model.
strength_weakness: strength_weakness: I am not sure are there any existing works implicitly considering decoupling these two tasks, i.e., OOD detection and generalization. Please provide more clarifications. 

The theory is very promising. So it will be better if the authors could provide more discussion on other potential strategies that could also addressing the dilemma. I think it will be very useful for readers.
questions: questions: 1. Figure 1 is very helpful and intuitive, while it seems not clear - which dataset the results are reported in the right figure.
2. The authors should explain more about the “tradeoff area”. Although it seems ok, but I think more detailed description is necessary. 
3. “Reducing the dependency on auxiliary OOD data can be an interesting research direction for the future exploration.” I think it is interesting to reduce the dependency on auxiliary OOD data. However, could the authors provide some potential directions?
limitations: limitations: See Weaknesses
paper_summary: paper_summary: This paper proposes a novel method for finetuning a classifier for both OOD generalization and OOD detection. The paper presents a novel objective function for OOD generalization and detection based on the Bayesian framework and OOD generalization theory. Extensive experimental results suggest that the proposed method is effective.
strength_weakness: strength_weakness: * It is possible that I missed it, but the paper does not provide a direct theoretical argument supporting the main objective function (Eq. (11)). The most important term in Eq. (11) is the constraint that the entropy of p(y|x) should not change before and after the fine-tuning. This term maintains OOD generalization, but the justification is weak. 
* Related to the above point, the statement in Lines 244-245 ("While generalization is related to 245 the overall uncertainty of p(yˆ|x) as we mentioned in related works (both AU and DU).") argues that this entropy (i.e., "overall uncertainty") is connected to the generalization performance. Still, I could not find the supporting information in the related work section.
questions: questions: * Is there a reason why we should enforce the "non-increased overall uncertainty" constraint (the constraint in Eq. (11)) only on semantic OOD data? For example, what if we enforce this on the in-distribution dataset? Is there a theoretical justification for this choice, or is it more empirical?
limitations: limitations: * The paper addresses its limitation in the conclusion section. The limitation of the paper does not raise a concern.

Comments: title: Thanks for your response
Discussion has begun
comment: Thanks for your positive feedback. If there are any additional questions that you would like to discuss, please feel free to let us know, we will continuously work on them and actively address your concerns. Your previous suggestions, especially the two suggested new baselines and visualization of uncertainty, have improved even our own understanding on the proposed method. Thanks again for your review and invaluable suggestions.
I appreciate author's feedback. Some of my concern are resolved. I keep my original positive score.
Thank you for your detailed rebuttal. The author addressed my problem so well that I kept my rating on the work.
Thank you for your suggestion. We are pleased to hear that most of your questions have been addressed. It is our responsibility to present mathematical notations clearly and avoid ambiguity. In lines 135-136, we describe in words that $L_{reg}$ is the cross-entropy between the prediction and the uniform distribution for MSP detectors. According to your advice, we will use $H(F(x), U)$ instead, where U denotes the uniform distribution and $H(\cdot , \cdot)$ represents the cross-entropy. This notation is consistent with the rest of this paper. We will also carefully revise the relative parts.

If there are any additional questions that you would like to discuss with us, please feel free to post, we will continuously work on them and actively address your concerns.
Thank you for your detailed response.  Overall, all my other questions are addressed well. Regarding W1, I missed that the OOD detection loss is the cross-entropy with respect to the uniform distribution. Using the same mathematical symbol for representing the same quantity could potentially improve the clarity.
Thanks for your positive feedback! We will carefully revise the related parts according to your suggestion.
I appreciated the author's responses and have raised my score to 6. It is highly recommended to add those discussions in the revised version.
Dear Reviewers,

Thank you for taking the time to review the paper. The discussion has begun, and active participation is highly appreciated and recommended.

Thanks for your continued efforts and contributions to NeurIPS 2024.

Best regards,

Your Area Chair