SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
(x) ^ {(t + 1)} = \Pi_ {\mathcal {B} [ x ^ {(0)}; \epsilon ]} \left(x ^ {(t)} + \alpha \operatorname {s i g n} \left(\nabla_ {x ^ {(t)}} \ell_ {\mathrm {O E}} (f (x ^ {(t)}))\right)\right), \tag {10}
$$
where  $t \in \mathbb{N}$ ,  $x^{(t)}$  is the extrapolated OOD data at step  $t$ , and  $\Pi_{\mathcal{B}[x^{(0)};\epsilon]}(\cdot)$  is the projection function that projects the new OOD data back into the  $\epsilon$ -ball centered at  $x^{(0)}$  if necessary. Different from the conceptual idea of constraining the perturbation (for human-imperceptibility) in the literature on adversarial robustness [Madry et al., 2018, Zhang et al., 2019], the projection operation in DivOE provides a quantification way to study the extrapolated distribution and its diversification. As the generation process starts from the given outliers, the semantical class (belongs to ID or OOD) is hard to change, as empirically shown in the left-most panel of Figure 3 (we adopted a very large  $\epsilon = 0.15$ ). More exploration about this perspective will leave to Section 4.3 for further discussion.
Here we summarize the whole procedure of the proposed DivOE in Algorithm 1, which consists of multi-round training iterations. To be specific, on each round of fine-tuning, DivOE first samples a mini-batch with a specific ratio  $r \in [0,1]$  from the auxiliary outliers and conduct multi-step optimization following Eq. (10) to generate the new outliers. With the ID training data, the original outliers, and the extrapolated outliers, DivOE then fine-tune the model with the loss defined in Eq. (9).
In practical realization, for the specific extrapolation strength  $\epsilon$ , DivOE can also generate the extrapolation pool with multiple  $\epsilon$  candidates within the extrapolated sub-batch. The specific definition of extrapolation pool is as follows,
$$
\mathcal {O} (\tilde {X}) = \left\{x _ {\epsilon_ {1}} ^ {* k _ {1}}, \dots , x _ {\epsilon_ {p}} ^ {* k _ {p}} \right\} _ {\epsilon_ {1}, \dots , \epsilon_ {p} \in [ 0, 1 ]} ^ {\sum_ {i = 1} ^ {p} k _ {i} = r n} \tag {11}
$$
where the extrapolation strength with its corresponding sample number can be flexibly decided in the extrapolated sub-batch with a total sample number equal to  $r * n$ , which may better facilitate the diversification target. As a primary exploration, we visualize the extrapolated data with its distribution quantified by Energy score [Liu et al., 2020] in the left two panels in Figure 3. Intuitively, even with a large extrapolation strength with human-perceptible pixel noise added in the original outliers, it will not change its classification results in OOD detection. It can represent a more different OOD distribution compared with the given outliers as the distribution shifted gradually to the right.
Algorithm 1 Diversified Outlier Exposure (DivOE) via Informative Extrapolation
Input: given model:  $\theta$ , fine-tuning epochs:  $T$ , training samples of ID data:  $x \sim \mathcal{D}_{\mathrm{in}}^{\mathrm{s}}$ , training samples of auxiliary outliers:  $x^{*} \sim \mathcal{D}_{\mathrm{out}}^{\mathrm{s}}$ , extrapolated ratio of the original outliers:  $r$ , extrapolate epsilon:  $\epsilon$ , optimization step number:  $k$ , extrapolation pool:  $\mathcal{O}(\tilde{X})$ ;  
Output: fine-tuned model  $\theta^T$ ;  
1: for epoch = 1, ..., T do  
2: for mini-batch = 1, ..., M do  
3: Sample a mini-batch  $\{(x_i, y_i)\}_{i=1}^m$  from  $\mathcal{D}_{\mathrm{in}}$ , sample a mini-batch  $\{(x_j^*)\}_{j=1}^n$  from  $\mathcal{D}_{\mathrm{out}}^s$   
4: Sample a sub-batch with ratio  $r$  from the batch of outliers  $\{(x_j^*)\}_{j=1}^n$   
5: for  $k = 1, \ldots, r * n$  (in parallel) do  
6: Obtain the extrapolated outliers  $\tilde{x}_k^*$  by Eq. (10) that following max  $\ell_{\mathrm{OE}}(f(\tilde{x}_k^*))$   
7: Aggregate all the extrapolated samples with uniform or different  $\epsilon$  in  $\mathcal{O}(\tilde{X})$   
8: end for  
9:  $\theta \gets \theta - \eta \nabla_\theta \left\{\frac{1}{m} \sum \ell_{\mathrm{CE}}(f(x), y) + \lambda \left(\frac{1}{n-rn} \sum \ell_{\mathrm{OE}}(f(x^*)) + \frac{1}{rn} \sum \ell_{\mathrm{OE}}(f(\tilde{x}^*))\right)\right\}$   
10: end for  
11: end for
Comparison and Compatibility. Compared with the conventional OE algorithm [Hendrycks et al., 2019], the critical difference behind DivOE is trying to extrapolate based on the given auxiliary outliers for diversifying outlier exposure. To be specific, it provides a general framework (under the guidance of the learning objective in Eq. (5)) for modeling the informative unknown inputs beyond the original ones. The discriminative feature regularized by our DivOE can be utilized by those advanced post-hoc scoring functions [Sun et al., 2021, Huang et al., 2021, Sun and Li, 2022, Djurisic et al., 2023]. For OE-based methods, the informative extrapolation introduced in DivOE is orthogonal to current tuning objectives [Hendrycks and Gimpel, 2017, Wang et al., 2023] and also compatible to those different data augmentation [Zhang et al., 2018, Yun et al., 2019], synthesizing [Kong and Ramanan, 2021, Lee et al., 2018a], and also sampling strategies [Chen et al., 2021, Ming et al., 2022] adopted based on either ID data [Du et al., 2022, Tao et al., 2023] or auxiliary outliers.
In this section, we present the comprehensive verification of the proposed DivOE in the OOD detection scenario. First, we provide the experimental setups in detail (in Section 4.1). Second, we provide the performance comparison of our DivOE with a series of post-hoc scoring functions and the OE-based methods with different strategies on the auxiliary outliers (in Section 4.2). Third, we conduct various ablation studies and further discussions to understand our DivOE (in Section 4.3).
Here we present several critical parts of experimental setups and leave more details in Appendix C.
Datasets. Following the common benchmarks used in previous work [Liu et al., 2020, Ming et al., 2022], we adopt CIFAR-10, CIFAR-100 [Krizhevsky, 2009] as our ID datasets. We use a series of different image datasets as the OOD datasets, namely Textures [Cimpoi et al., 2014], P1aces365 [Zhou et al., 2017], iSUN [Xu et al., 2015], LSUN_Crop [Yu et al., 2015], and LSUN_Resize [Yu et al., 2015]. For the surrogate OOD data used in OE-based methods [Hendrycks et al., 2019, Chen et al., 2021, Ming et al., 2022, Zhang et al., 2023], we adopt Tiny-ImageNet [Le and Yang, 2015]. We also conduct the experiments on the ImageNet, where we utilize the large-scaled ImageNet21K [Ridnik et al., 2021] as the surrogate OOD set like previous works [Wang et al., 2023] for OE-based methods.
Evaluation metrics. We employ the following three common metrics to evaluate the performance of OOD detection: (i) Area Under the Receiver Operating Characteristic curve (AUROC) [Davis and Goadrich, 2006] can be interpreted as the probability for a positive sample to have a higher discriminating score than a negative sample [Fawcett, 2006]; (ii) Area Under the Precision-Recall curve (AUPR) [Manning and Schütze, 1999] is an ideal metric to adjust the extreme difference
Table 1: Main Results (\%). Comparison with competitive OOD detection baselines. All methods are trained on the same backbone. Values are percentages. Bold numbers are superior results.  $\uparrow$  indicates larger values are better, and  $\downarrow$  indicates smaller values are better. (averaged by multiple trials)
Din	Method	FPR95↓	AUROC↑	AUPR↑	ID-ACC↑	w./w.o Daux	Operation for Daux
CIFAR-10	MSP [Hendrycks and Gimpel, 2017]	52.21	90.61	97.84	94.84
ODIN [Liang et al., 2018]	32.84	90.67	97.29	94.84
Mahalanobis [Lee et al., 2018b]	36.27	92.69	98.31	94.84
Energy [Liu et al., 2020]	32.82	91.99	97.86	94.84
ASH [Djurisic et al., 2023]	33.01	92.11	97.92	94.84
OE [Hendrycks et al., 2019]	13.76±0.23	97.53±0.03	99.43±0.01	94.51±0.06	✓	Vanilla
Energy (w. Daux) [Liu et al., 2020]	18.15±0.32	90.90±0.18	96.34±0.10	94.51±0.01	✓	Vanilla
VOS [Du et al., 2022]	34.30±0.02	91.27±0.01	97.64±0.01	94.02±0.01	✓	Virtually Synthesized
NTOM [Chen et al., 2021]	17.76±0.84	91.08±0.44	96.40±0.18	95.04±0.03	✓	Greedy Sampled
POEM [Ming et al., 2022]	17.81±0.78	90.96±0.37	96.36±0.15	94.90±0.04	✓	Thompson Sampled
MixOE [Zhang et al., 2023]	13.57±0.25	97.47±0.07	99.39±0.02	94.98±0.02	✓	Mixup with ID Data
DivOE (Ours)	11.66±0.04	97.82±0.04	99.48±0.01	94.66±0.12	✓	Extrapolated
CIFAR-100	MSP [Hendrycks and Gimpel, 2017]	79.71	76.13	94.07	75.95
ODIN [Liang et al., 2018]	63.29	82.44	95.49	75.95
Mahalanobis [Lee et al., 2018b]	52.90	83.86	95.73	75.95
Energy [Liu et al., 2020]	71.93	80.27	95.00	75.95
ASH [Djurisic et al., 2023]	61.82	83.42	95.84	75.87
OE [Hendrycks et al., 2019]	27.67±0.43	91.89±0.12	97.93±0.04	75.41±0.10	✓	Vanilla
Energy (w. Daux) [Liu et al., 2020]	27.86±0.26	90.70±0.07	97.40±0.03	75.03±0.11	✓	Vanilla
VOS [Du et al., 2022]	70.87±0.00	80.00±0.00	94.89±0.00	76.25±0.00	✓	Virtually Synthesized
NTOM [Chen et al., 2021]	27.58±0.88	91.08±0.34	97.52±0.10	75.41±0.10	✓	Greedy Sampled
POEM [Ming et al., 2022]	26.50±0.10	91.33±0.06	97.58±0.02	75.74±0.14	✓	Thompson Sampled
MixOE [Zhang et al., 2023]	35.26±0.80	90.31±0.16	97.58±0.04	75.74±0.04	✓	Mixup with ID Data
DivOE (Ours)	24.80±0.51	92.91±0.11	98.22±0.03	75.24±0.01	✓	Extrapolated
ImageNet	MSP [Hendrycks and Gimpel, 2017]	75.23	76.67	94.35	68.72
ODIN [Liang et al., 2018]	67.61	80.36	95.15	68.72
Mahalanobis [Lee et al., 2018b]	83.41	58.47	87.72	68.72
Energy [Liu et al., 2020]	71.44	79.52	94.98	68.72
ASH [Djurisic et al., 2023]	73.51	77.03	93.43	68.70
OE [Hendrycks et al., 2019]	61.94±0.03	81.58±0.01	95.49±0.00	75.90±0.00	✓	Vanilla
Energy (w. Daux) [Liu et al., 2020]	66.94±0.01	79.93±0.02	94.90±0.00	74.19±0.08	✓	Vanilla
VOS [Du et al., 2022]	74.00±0.02	78.97±0.01	94.82±0.00	76.15±0.02	✓	Virtually Synthesized
NTOM [Chen et al., 2021]	66.32±0.24	80.14±0.08	94.95±0.02	74.26±0.01	✓	Greedy Sampled
POEM [Ming et al., 2022]	67.81±0.44	79.36±0.24	94.73±0.06	74.21±0.03	✓	Thompson Sampled
MixOE [Zhang et al., 2023]	70.12±0.26	78.85±0.05	94.85±0.02	76.18±0.01	✓	Mixup with ID Data
DivOE (Ours)	60.12±0.11	81.96±0.00	95.59±0.00	75.73±0.02	✓	Extrapolated
between positive and negative base rates; (iii) False Positive Rate (FPR) at  $95\%$  True Positive Rate (TPR) [Liang et al., 2018] indicates the probability for a negative sample to be misclassified as positive when the true positive rate is at  $95\%$ . We also include in-distribution testing accuracy (ID-ACC) to reflect the preservation level of the performance for the original classification task on ID data.
OOD detection baselines. We compare the proposed method with several competitive baselines in the two directions. Specifically, we adopt Maximum Softmax Probability (MSP) [Hendrycks and Gimpel, 2017], ODIN [Liang et al., 2018], Mahalanobis score [Lee et al., 2018b], and Energy score [Liu et al., 2020] as scoring function baselines; We adopt OE [Hendrycks et al., 2019], Energy-bounded learning [Liu et al., 2020], NTOM[Chen et al., 2021], POEM [Ming et al., 2022] and MixOE [Zhang et al., 2023] as baselines with outliers. For all scoring function methods, we assume the accessibility of well-trained models. For all methods involving outliers, we constrain all major experiments to a fine-tuning scenario, which is more practical in real cases. Different from training a dual-task model at the very beginning, equipping deployed models with OOD detection ability is a much more common circumstance. We leave more definitions and implementations in Appendix A.
In this part, we present the major performance comparison with some representative baseline methods for OOD detection to demonstrate the effectiveness of the proposed DivOE. Specifically, we consider several post-hoc methods of scoring functions as the performance reference based on the pre-trained model and some OE-based methods for specific comparison on fine-tuning with auxiliary outliers. Note that for all the experiments here, we utilize the whole auxiliary outliers (following the general setup of OE [Hendrycks et al., 2019, Liu et al., 2020] and ensuring the same number of outliers engaged in each method) instead of the reduced ones adopted in previous figures for illustrations.
In Table 1, we present the overall results using different methods for OOD detection. Since the OE-based methods engage the auxiliary outliers during training, the model will generally gain better
Table 2: OOD detection performance on compatibility experiments. All methods are trained on the same backbone.  $\uparrow$  indicates larger values are better, and  $\downarrow$  indicates smaller values are better.
Method	Din
CIFAR-10	CIFAR-100
FPR95↓	AUROC↑	AUPR↑	ID-ACC↑	FPR95↓	AUROC↑	AUPR↑	ID-ACC↑
OE	13.76±0.23	97.53±0.03	99.43±0.01	94.51±0.06	27.67±0.43	91.89±0.12	97.93±0.04	75.41±0.10
DivOE	11.66±0.04	97.82±0.04	99.48±0.01	94.66±0.12	24.80±0.51	92.91±0.11	98.22±0.03	75.24±0.01
Energy (w. Daux)	18.15±0.32	90.90±0.18	96.34±0.10	94.51±0.01	27.86±0.26	90.70±0.07	97.40±0.03	75.03±0.11
Energy (w. Daux)+DivOE	16.03±0.35	92.23±0.10	96.87±0.03	94.53±0.05	24.71±0.29	92.04±0.19	97.82±0.08	75.25±0.11
NTOM	17.76±0.84	91.08±0.44	96.40±0.18	95.04±0.03	27.58±0.88	91.08±0.34	97.52±0.10	75.41±0.10
NTOM+DivOE	15.30±0.57	92.21±0.57	96.73±0.28	94.75±0.02	25.97±0.74	91.62±0.29	97.68±0.10	75.39±0.15
POEM	17.81±0.78	90.96±0.37	96.36±0.15	94.90±0.04	26.50±0.10	91.33±0.06	97.58±0.02	75.74±0.14
POEM+DivOE	15.22±0.20	92.42±0.08	96.90±0.03	94.66±0.07	25.05±0.03	91.91±0.07	97.77±0.03	75.62±0.05
OECC	17.65±0.06	96.68±0.03	99.19±0.01	94.50±0.07	28.51±0.11	91.08±0.08	97.70±0.03	74.95±0.30
OECC+DivOE	14.18±0.04	97.26±0.07	99.32±0.03	94.49±0.15	26.13±0.39	92.07±0.14	97.97±0.05	74.74±0.36
DOE	7.39±0.46	98.29±0.08	99.58±0.02	92.82±0.08	30.69±0.93	93.13±0.15	98.39±0.03	71.41±0.38
DOE+DivOE	6.51±0.10	98.60±0.01	99.65±0.01	93.45±0.02	20.26±0.20	95.41±0.05	98.91±0.02	71.43±0.60
empirical performance on OOD detection, reflected by the evaluation metrics. For those OE-based methods, we provide a specific comparison on their detailed operation for  $\mathcal{D}_{\mathrm{aux}}$  (e.g.,  $\mathcal{D}_{\mathrm{out}}^s$ ) in the right of Table 1. Since VOS virtually synthesizes the outliers based on the ID data, it doesn't perform better than those using the real collected outliers. Among the rest, NTOM and POEM adopt different strategies for informative sampling, which obtain different levels of detection performance gains across different ID datasets. MixOE that mixups ID and surrogate OOD data does not perform well on some complexed tasks like CIFAR-100 and ImageNet, since the original algorithm is designed for fine-grained OOD detection. Without sacrificing much classification performance (i.e., ID-ACC) on ID data, our DivOE can consistently achieve better OOD detection performance across the three datasets, which verifies the effectiveness of our methods with the newly proposed informative extrapolation for outlier exposure. We also summarize the fine-grained results on OOD detection to Tables 12 and 13 in Appendix C, which provides a better understanding of the improvement.
In Table 2, we report the results of compatibility experiments, in which we compare those OE-based methods with their variants, incorporating our DivOE for informative extrapolation. Specifically, we consider some of the previous OE-based methods and also add two related methods, e.g., OECC [Papadopoulos et al., 2021] and DOE [Wang et al., 2023], in the comparison. Although those methods can have different levels of performance enhancement compared with the conventional OE baseline, We can find that our DivOE can consistently help them gain better OOD detection performance across three evaluation metrics, and keep the ID-ACC comparable with the original pre-trained model. As for DOE, it utilizes the model perturbation on implicit outlier synthesizing, which is also orthogonal and compatible with our DivOE for explicit extrapolation. In this table, we can find the plain DivOE can achieve lower FPR95 on CIFAR-100 with DOE, while the performance of DOE is affected by different ID datasets. The above demonstrates the effectiveness and algorithmic robustness of our proposed method. In addition, we also provide further discussion on the differences in Appendix A.1.
4.3 Ablation and Further Discussions
In this part, we conduct further explorations to provide a thorough understanding of our DivOE. For the extra results and discussions (e.g., impact and limitations), we leave more details in Appendix C.
Importance of extrapolation ratio in diversifying the distribution. As a critical aspect of our proposed learning objective of DivOE in Eq. (9), the extrapolation ratio is introduced to control the manipulated portion of the given auxiliary outliers. DivOE aims to extrapolate based on the surrogate OOD distribution without total loss of the original representation. In Figure 4(a), we show the performance by varying the ratio  $r$ . It is worth noting that adopting extrapolation on the full batch of data may even degrade the performance, indicating the vital role of the extrapolation ratio in the whole learning framework. It performs diversifying as the maximization part introduced in Eq. (5) instead of an overall augmentation. The results verify the rationality of pursuing diversified outliers instead of only most informative ones. We provide more comparison on the concepts of "diversified" and "informative" in Appendix A.2, and leave more results in Appendix C to further explain it.
Extending the diversified strength in the informative extrapolation. To better characterize the effects of informative extrapolation in DivOE, we consider fine-tuning the given model with
(a) Extrapolated Ratio
(b) Diversified Strength
Figure 4: Ablation Study. (a) performance of using different extrapolated ratios of DivOE; (b) exploration of different diversified strength  $\epsilon$ ; (c) using different OOD scores as informative extrapolation targets; (d) comparison between DivOE and OE adopting other augmentations for diversification.
(c) Extrapolation Method
(d) Compare with Augs.
different  $\epsilon$  (with a consistent extrapolation ratio,  $r = 0.5$ ), which is used to constrain the diversified strength in maximizing the optimization target in Eq. (6). In Figure 4(b), we can observe that an appropriate diversified strength can help OE performs better on OOD detection, while using a large  $\epsilon$  may empirically return to the original baseline case. One possible reason may be that the extrapolated outliers with a large  $\epsilon$  may fail to become more informative as previously reflected in Figure 3.
Generality of using different OOD scores. Since DivOE introduces a general learning framework of outlier exposure for model finetuning with auxiliary outliers, the specific realization for informative extrapolation can have multiple choices. For example, different scoring functions can all perform OOD uncertainty estimation, indicating the multiple candidate target for Eq. (6) to optimize. Here we report the performance using different optimization targets (e.g., MSP or Energy score) in Figure 4(c), where they have different performance improvements compared with the original OE baseline.
Comparison with different augmentation-based methods for OE. In Figure 4(d), we perform the comparison of our DivOE with the original OE adopting some representative augmentation methods (e.g., Mixup [Zhang et al., 2018] and CutMix [Yun et al., 2019]) for diversifying the given outliers. The results verify the superiority of DivOE on OOD detection over the direct adaptation of those strategies since they lack a critical diversification guidance. On the other hand, our framework has no strict constraints for the detailed manipulating operations. In other words, DivOE can also utilize those augmentation methods for extrapolation, for which we leave more discussion in Appendix C.
In this paper, we propose a novel learning framework, i.e., Diversified Outlier Exposure (DivOE), that promotes diversity in outlier exposure with the given auxiliary outliers. To empower the model with more knowledge about the decision boundary between ID and OOD data, DivOE conducts informative extrapolation guided by a newly designed objective. Through the diversification target, our method adaptively synthesizes meaningful outliers during the training process, which helps the model better shape the decision areas for OOD detection. We have conducted extensive experiments to demonstrate the effectiveness of our proposed DivOE and its compatibility with a range of OE-based methods, and also various ablation studies with further explorations to characterize the framework.
JNZ and BH were supported by the NSFC Young Scientists Fund No. 62006202, NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011652, RIKEN Collaborative Research Fund, HKBU Faculty Niche Research Areas No. RC-FNRA-IG/22-23/SCI/04, and HKBU CSD Departmental Incentive Scheme. GY and JCY were supported by the National Key R&D Program of China (No. 2022ZD0160703), 111 plan (No. BP0719010) and National Natural Science Foundation of China (No. 62306178). TLL was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. In arXiv, 2021.
Karsten M Borgwardt, Arthur Gretton, Malte J Rasch, Hans-Peter Kriegel, Bernhard Scholkopf, and Alex J Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 2006.
Léon Bottou. Stochastic gradient descent tricks. Neural Networks: Tricks of the Trade: Second Edition, 2012.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of-distribution detection using outlier mining. In ECML PKDD, 2021.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In ICML, 2006.
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In ICLR, 2023.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. VOS: learning what you don't know by virtual outlier synthesis. In ICLR, 2022.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 2011.
Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribution detection learnable? In NeurIPS, 2022.
Tom Fawcett. An introduction to roc analysis. Pattern Recognition Letters, 2006.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In ICLR, 2021.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. In NeurIPS, 2021.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. In ICML, 2022.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In NeurIPS, 2021.
Julian Katz-Samuels, Julia Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors in their natural habitats. In ICML, 2022.
J. Kiefer and J. Wolfowitz. Stochastic estimation of the maximum of a regression function. Ann. Math. Statist., 1952.
Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In CVPR, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. In arXiv, 2009.