Strengthen Out-of-Distribution Detection Capability with Progressive Self-Knowledge Distillation
Authors: Yang Yang
Haonan Xu
Venue: ICML 2025 poster (2025)
Keywords: Out-of-distribution Detection
Self-distillation
Primary area: deep_learning->robustness

Abstract: Out-of-distribution (OOD) detection aims to ensure AI system reliability by rejecting inputs outside the training distribution. Recent work shows that memorizing atypical samples during later stages of training can hurt OOD detection, while strategies for forgetting them show promising improvements. However, directly forgetting atypical samples sacrifices ID generalization and limits the model's OOD detection capability. To address this issue, we propose Progressive Self-Knowledge Distillation (PSKD) framework, which strengthens the OOD detection capability by leveraging self-provided uncertainty-embedded targets. Specifically, PSKD adaptively selects a self-teacher model from the training history using pseudo-outliers, facilitating the learning of uncertainty knowledge via multi-level distillation applied to features and responses. As a result, PSKD achieves better ID generalization and uncertainty estimation for OOD detection. Moreover, PSKD is orthogonal to most existing methods and can be integrated as a plugin to collaborate with them. Experimental results from multiple OOD scenarios verify the effectiveness and general applicability of PSKD.

In real-world scenarios, deep learning models are inevitably exposed to unseen classes of samples that lie outside the training distribution (Yang et al., 2021), known as OOD data. Persisting in classifying OOD data into one of the predefined training classes is meaningless. Such unreliable behavior could lead to disastrous consequences, particularly in safety-critical fields like autonomous driving (Geiger al., 2012) and medical diagnosis (Litjens et al., 2017). Strengthen Out-of-Distribution Detection Capability with Progressive Self- Knowldnowe Distillation
Yang Yang'
Haonan Xu
Abstract
100
100
Rist acouracyy
Near-COD. CIFAR-100
Out-of-distribution (OOD) detection aims to en- sure AI system reliability by rejecting inputs outside the training distribution. Recent work shows that memorizing atypical samples dur- ing later stages of training can hurt OOD detec- tion, while strategies for forgetting them show promising improvements. However, directly for- getting atypical samples sacrifices ID general- ization and limits the model's OOD detection capability. To address this issue, we propose Progressive Self- Saff-Knowledge Distillation (PSKD) framework, which strengthens the OOD detection capability by leveraging self-provided uncertainty- embedded targets. Specifically, PSKD adaptively selects a self-teacher model from the training his- tory using pseudo-outliers, facilitating the learn- ing of uncertainty knowledge via multi-level dis- tillation applied to features and responses. As result, PSKD achieves better ID generalization and uncertainty estimation for OOD detection. Moreover, PSKD orthogonal to most existing methods and can be integrated as a plugin to col- laborate with them. Experimental results from multiple OOD scenarios verify the effectiveness and general applicability of PSKD.
80
FPR954
Fan0OD: Textures
Accuracy
60
60
40
Epochs
Typical samples
Atypical samples
Method
ID 0OD
Vanilla
learning
memorizing
Unleashing Mask
learning
forgetting
Ours
learning
learning
Figure I: Upper: The FPR95 (lower values indicate better OOD detection) curves for various OOD datasets and the ID test accuracy curve, tracked when CIFAR-10 is used as the ID dataset for training. Lower: Comparisons existing methods and our PSKD for handling atypical samples.
This highlights the necessity of OOD detection to ensure Al system reliability by enabling models to identify and reject predictions for OOD inputs.
Up to now, many efforts have been made in pursuing re- liable detection methods. Early methods, represented by MSP (Hendrycks & Gimpel, 2017), ODIN (Liang et al., 2018) and Energy (Liu et al., 2020), focus on developing suitable scoring functions for OOD uncertainty estimation. However, having well-trained model with solid basis for OOD detection is essential prerequisite. To this end, series of studies delve into training time regularization methods, which can be categorized into two main areas (Zhang et al., 2023b). The first involves exploring alterna- tive training strategies to train stronger models with better discriminative representations (DeVries & Taylor, 2018: Se- hwag et al., 2021: Wei al., 2022: Pei, 2024). The second involves introducing realistic outliers help OOD detectors learn more robust decision boundaries (Hendrycks et al., 2019; Liu et al., 2020; Wang et al., 2023; Jiang et al., 2024), Despite the promising performance achieved by these meth- ods, limited attention is given to the models' intrinsic OOD detection capabilities, which constrains their full potential.
1. Introduction
Naajing University Science and Technology, Correspon- dence to: Yang Yang <yyang @ njust.edu.en>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Strengthen Nut-of-Distributioo Detection Capability with Progressive Self-Knowledge Distillation
To provide an intuitive explanation of this issue, we monitor the model's performance on the ID classification and OOD detection throughout the entire training process, shown in Figure 1. The results show that the model's optimal OOD detection performance occurs at an intermediate stage he- fore convergence, rather than at the final well-trained state. Moreover, Zhu et al. (2023) have confirmed that this phe- nomenon is prevalent across variety of training settings. They attribute this to the model's tendency memorize atypical samples during later stages of training, which bene- fits ID generalization but leads the model to become more confident about OOD data, thereby hurting OOD detection (Zhu et al., 2023). In response, they propose Unleashing Mask/ Unleashing Mask Adopts Pruning (UM/UMAP), which uses gradient ascent forget these atypical samples, thereby backtracking the model to an earlier state with better OOD detection capabilitiiss However, determining the tent of model backtracking is labor-intensive and needs to be empirically found by the users. More importantly, directly forgetting atypical samples can sacrifice ID generalization and limit the model's OOD detection capability.
input perturbations and temperature scaling. Energy score (Liu et al., 2020) utilizes the logsumexp of the output logits, which is provably aligned with the density of inputs. Our work is orthogonal to these methods, as it focuses on estab- lishing a stronger model foundation for OOD detection.
Training-Tine Regularization Methods aim to calibrate the model to address OOD detection. One major category of methods involves altering training strategies to provide better discriminative representations for OOD detection (Zhou et al., 2021; Huang & Li, 2021; Zhang & Xiang, 2023). LogitNorm (Wei et al., 2022) and T2Fnorm (Regmi et al., 2024) provide alternatives to cross-entropy loss by decoupling the impact of logits and feature norms, respec- tively. SNN (Ghosal et al., 2024) learns the most relevant subspace to tackle the curse-of- suise-ff-dimensionaliy problem in baasance-based OOD detection. Zhu et al. (2023) propose UM/UMAP to revert a well-trained model to an earlier stage, aiming to forget memorized atypical samples which hurt OOD detection. Another category introduces realis- tic outliers to facilitate the model in learning more robust decision boundaries (Li & Vasconcelos, 2020; Ming et al., 2022; Yang et al., 2023; 2024). OE (Hendrycks et al., 2019) forces the predictive distribution of outliers to be uniform. MixOE (Zhang et al., 2023a) mixes ID data and outliers to expand the coverage of different OOD granularities. DOE (Wang et al., 2023) utilizes model perturbations to implicitly broaden the coverage of outliers for better generalization. While effective in OOD detection, these methods often de- grade ID task performance, and realistic outliers are typi- cally costly to acquire.
Targeting these important problems, we propose a novel self- learning framework termed Progressive Self-Knowledge Distillation (PSKD), designed to leverage self- ooff-proviee uncertainty-embeddeed targets for more effective learning from both typical and atypical samples. Concretely, PSKD adaptively selects the teacher model with the most reliable uncertainty estimation from the current training history, which can be assisted by generated pseudo-outliers. Multi- level distillation, applied to both features and responses, is then used to facilitate more effective learning of OOD knowledge from uncertainty-embeddee targets provided by the teacher model. This can mitigate overconfident pre- dictions caused by treating training samples with varying uncertainties uniformly via one-hot targets. As a result, PSKD achieves more reliable uncertainty estimation for OOD detection, while also offering the additional benefit of improved ID generalization. Moreover, PSKD can be integrated as plugin, leveraging self-distillation to restore the model's 's intrinsic OOD detection capability. This estab- lishes a solid OOD detection model basic for most existing methods, thereby pushing their performance further.
2.2. Knowledge Distillation
Knowledge distillation is a popular learning method where student model is trained to mimic the behavior of more powerful teacher model (Gou et al., 2020; Cao et al., 2023). The idea originally stems from Bucila et al. (2006), and is later formally popularized by Hinton et al. (2015). Dif- ferent knowledge can be utilized, represented by response- based knowledge (Hinton et al., 2015) and based knowledge (Romero et al., 2015; Ahn et al., 2019). Self- distillation is a special case of knowledge distillation, where the same network is adopted for both the teacher and stu- dent models. Intuitively, it enables the model to refine itself via extended learning of its self- kooleknoweddge. In this con- text, Zhang et al. (2019) explore distilling knowledge from the deeper layers of the network into its shallower layers. Yang et al. (2019) propose transferring knowledge from the carly stages (teacher) of the network to the later stages (student) to facilitate the supervised training process. More- over, Mobahi et al. (2020) provide a theoretical analysis of self dufddistillation, and Zhang & Sabuncu (2020) empirically demonstrate its improved performance.
2. Related Work
2.1. cittr--Disttibution Detection
OOD Scoring Methods aim to provide suitable measures to indicate the likelihood that a sample originates from the OOD distribution (Lee et al., 2018; Huang et al., 2021; Djurisic et al., 2023; Xu et al., 2024; Xu & Yang, 2025). For example, MSP (Hendrycks & Gimpel, 2017) directly uses the maximum SoftMax score as the criterion. ODIN (Liang et al., 2018) improves the MSP score by introducing
2

---

Strengthen Dut-of-Distribution Detection Capability with Progressive Self-Knowledge Distillation
3. Preliminaries
samples, which improves ID generalization but simultane- ously makes the model more confident about OOD data, thereby hindering OOD detection. Accordingly, Zhu et al. (2023) propose UM and its variant UMAP, which directly forget atypical samples to revert the model to an earlier state. Conversely, in this paper, we explore how to learn from valuable atypical samples effectively.
Setup. In this paper, we focus on the setting of K-way image classification. Formally, let X' be the input space and Iin = 21, ..., } be the ID label space. The learner has access to labeled training set Duain = x+); Bim and validation set vaa, where the samples are drawn i..dd from a joint distribution Pxy.. Let fe X + R2K denote the classification model parameterized by 0, which is learned by minimizing the empirical risk: 8" = arg ming (fo(.)) where (fo = P.... (x), y) and is the loss function. Generally, a SoftMax layer with temperature T (normally set to 1) is applied for the prediction fx)):
Typically, the training set in supervised learning consists of pairs of input image and its corresponding one-hot target. However, in the context of uncertainty estimation for OOD detection, assigning all training samples an absolutely confident one-hot target can be a potential hindrance. This could hinder the model's ability to effectively learn to differ- entiate between samples with varying levels of uncertainty, potentially leading to overconfidence in its uncertainty esti- mates when encountering OOD data. Therefore, providing sample-level uncertainty knowledge as the learning objec- tive for OOD detectors is crucial. Intuitively, atypical sam- ples should be assigned higher uncertainty soft targets for learning compared to typical samples. However, a practi- cal challenge is that hand-crafting uncertainty- targets prohibitively costly, requiring substantial expert knowledge and labor-intensive efforts.
exp(21 (2)( P: Σ3 eXp(2,/T)
where 2; is the logit for the i-th class. A larger T can make the probability distribution softer, which can be beneficial for knowledge distillation (Hinton et al., 2015).
OOD Detection aims discem OOD data arising from irrelevant distribution whose label set has no intersection with Di (Yang et al., 2021). In general, the OOD detector gr (-) is given by:
To this end, we propose a self-learning paradigm PSKD that leverages uncertainty-embeddee targets, self -rovidd by the teacher model selected from the current training history, as a substitute to facilitate the model's learning of uncertainty knowledge. Figure illustrates the framework of our proposal. Conceptually, PSKD jointly optimizes for both: (1) accurate ID classification, and (2) reliable uncertainty estimation. Formally, let the self-selected OOD teacher model be denoted as fø., the overall risk can be formalized with a weight factor À as follows:
if S fo) > Ti OOD, if; fo) S
ID,
gx(x)=
(2)
where T is a threshold typically chosen to correctly classify the majority of ID data (e.g., 95%), and S(x; fo) X' - TR is the OOD scoring function, which quantifies model uncer- tainty as a scalar value. Representative scoring functions include the MSP (Hendrycks & Gimpel, 2017) and Energy score (Liu al., 2020), which are defined follows:
SMSP(x: xx So) = max Pt, SEneryy (x; se) = log Σ exp(=)).
arg min (1 - X) (fo +
A . RooD ( (fo; fo-) 1.
8
ID classification
Self OOD knowledge distillation
(3) Since OOD labels lie outside the training label space, the model tends to produce lower expected scores compared to ID cases. However, OOD detection remains non-trivial because deep models can be overconfident when faced with OOD data (Nguyen et al., 2015; Bendale & Boult, 2016).
(4)
where the term [w)(S) is intended to learn ID classifica- tion capability, which can be cross-entropy loss in vanilla training, while ROOD (fo fe-) is designed to learn uncer- tainty estimation knowledge from the self-selected OOD teacher model. To prevent large bias the teacher model's responses during early training, we dynamically adjust the weights of ID learning and self-distillation for effective ini- tial learning. Mathematically, the weight factor À at epoch e is computed using a cosine annealing schedule as follows:
4. Proposed Method
4.1. Motivation and Framework
As illustrated in Figure 1, the model's optimal OOD de- tection performance is achieved at an intermediate stage of training before full convergence, rather than at the final well-trained state. In other words, trade-off exists between pursuing better performance of the main task (ID prediction) and the OOD detection task. Zhu et al. (2023) attribute this inconsistency to the model's tendency to memorize atypical
Attnal 2
(1 cos
E (ET)
(5)
where Atina is the final self-distillation weight, and E' is the total number of training epochs. To make PSKD feasible, we have two key challenges that need to be addressed: (1) how to identify and select desired teacher model fo., and
3

---

Strengthen cutt-o--Distribution Detection Capability with Progressive Belf-Knowledge Distillation
Training Phase
Validation Phase
OOD Teacher Model (self-selection)
O
O O O Spftt tangees
Feature Extractor
Festures
Classiier SeftMax
Update if the student outperforms
Validation set (ID)
Training
Image perturbed
Student Model
(ID)
o
Classi SoftiMa
Predi Prddiction
Uncertainty Estimation
Feature Extractor
O
Actual
Teacher Model
Pseudo-outlier set (OOD)
OOD delection
Student Model (currently in training)
capability comparison
Forward flow
Supervision from hints
Supervision from soft targets
Supervision from one hot targets
Figure 2: Illustration of model training in a self-learning paradigm with PSKD. During the training phase, the model is jointly supervised by both actual targets and mncertainty-embeddee teacher signals. In the validation phase, pseudo-outliers are used with the validation set for OOD detection capability comparison. The OOD teacher model is replaced by the student model when the student performs OOD detection better. This process can be iteratively repeated over time.
(2) how effectively learn OOD knowledge from the self- selected teacher model.
periodically after certain number of training steps (e.g., after one epoch in the validation stage).
4.2. Self-Selection of the OOD Teacher Model
4.3.
Distillation
To acquire the OOD teacher model, the first erucial step is to define explicit criteria for assessing and comparing the OOD detection performance of different models. In this work, we adopt the AUROC (Area Under the Receiver Operating Characteristic curve), widely recognized metric in OOD detection, provides a comprehensive evaluation across various decision thresholds. Given model fe, ID dataset DID. OOD dataset DOOD. and the OOD scoring function S(: fo), the OOD detection capability of fo can be approximately quantified using the AUROC as follows:
To efficiently learn OOD knowledge from the teacher model, we focus on knowledge at the response and penultimate fea- ture levels, which are most relevant and impactful for OOD detection (Zhu et al., 2022). At the response level, PSKD leverages self-provided uncertainty-embeddee targets de- rived from the teacher's predictions, enabling the model to learn uncertainty knowledge that reflects the teacher's 's reli- able behavior in uncertain scenarios. This can be achieved by minimizing the Kullback-Leibler (KI.) divergence be- tween the output probabilities of the classifiers:
Rangis (fo; fa. = Ex KL (prş (x) (x)), (7)
A(fe) ExED ExpDoo I(S(x; fo) 5(x::fo)
where Pış (x) and pfv (x) represent the softmax outputs from the teacher and student models, respectively.
[DDl (DooD]
where II(-) denotes the indicator function and |D] denotes the cardinality of the DD. The validation dataset Vaal can naturally serve reference for DID- As for DOoD, we adopt generic outlier generation strategy by perturbing or corrupting images from Dval to create the OOD data, as obtaining realistic outliers is often costly or impractical. The generation details and the impact of using pseudo-outliers from different sources, as well as the choice of ((: fo), are further discussed in Section 5.4.
On the other hand, feature-level supervision comes from the deepest classifier's 's hint, defined as the teacher model's feature representation guiding the student's 's learning (Hinton et al., 2015). This achieved by reducing the distance between feature maps in the teacher model and the student model:
Riea( fer) ExeDrn | (x) ha (x) (8)
where hg- (x) and (x() denote the penultimate feature rep- resentations of the teacher and student models, respectively.
In the context of the evaluation criteria, strong OOD de- tector is expected to assign higher scores to ID cases than
To up, the self-OOD knowledge distillation risk be formalized with weighting factor follows:
OOD cases, resulting in a higher A(-). To maintain the advanced uncertainty estimation capability of the teacher, PSKD dynamically selects the teacher model that achieves the best A(-) determined from the current training history,
RooD fo-) = Regins Sa-) + a Rreat ReeaeJJs JJo su-). (9)
The pseudo code of PSKD is available in Appendix A.

---

Strengthen cutt-f--Distiibttion Detection Capability with Progressive Self-Knowledge Distillation
100
our proposed leverages the potential within the model's own learning process to generate soft targets that inherently capture sample-level uncertainty. Specifically, PSKD alle- viates overfitting to atypical samples by learning from the targets provided by the self-selected teacher model, thereby effectively enhancing the model's ability to perceive uncertainty. As shown in Figure 4, PSKD significantly reduces the model's overconfidence in OOD samples, as evidenced by the decrease in the right tail of the OOD score distribution for OOD data, thereby improving ID-OOD separation.
Vanilla: ID ACC
PSKD: ID ACC
80
Vanilla: OOD FPR95
95
80
PSKD: 00D FPR95
FPR95
60
ACC1
60
80
90
100
40
40
20
20
40
60
100
Epochs
Figure 3: Performance tracking curves for ID prediction and OOD detection tasks. The OOD results are averaged over near- and far-OOD groups on the CIFAR-10 10 benchmark.
5. Experiments
2.8
S
Bat Teatuins
In this section, we first describe our experimental setup, then present the main results on multiple OOD detection bench- marks, followed by ablation studies and further analysis.
Proguercy
FPR95:72.669 72.69%
30.17%
#PR95: 60.22%
Eneegy Scores
Energy Scores
Fre Scorss
5.1. Experimental Setup
(a) Vanilla training with cross-entropy loss
Following standard practices (Zhang et al., 2023b), we eval- uate our PSKD using the OpenOOD v1.5 benchmark on both small-scale CIFAR and large-scale ImageNet, cover- ing near-OOD scenarios with semantic shifts and far-OOD scenarios with further obvious covariance shifts! !
SPROS: 34.14%
FPRO5: 14.96%
FPRSS: 21.77%
3.2
Energy Scores
Energy Scores
Energy Scores
(b) Training with our proposed PSKD
Datasets. For small-scale experimental setups, CIFAR- 10/100 (Krizhevsky, 2009) is adopted as the ID dataset. The near-OOD group contains CIFAR-100/10 and Tiny- ImageNet (Le & Yang, 2015), while the far-OOD group consists of MNIST (Deng, 2012), SVHN (Netzer et al., 201 Textures (Cimpoi et al., 2014), and Places365 (Zhou et al., 2018). For the realistic outlier dataset, we follow the OpenOOD standards and use TinylmagNett-597 (Le & Yang. 2015), which has no category overlap with CIFAR- 10/100 and the OOD test sets.
Figure 4: Changes in OOD score distribution using PSKD,
4.4. Insight Justification
The following remarks are provided to empirically elaborate on the insights of PSKD.
Remark 1. PSKD restores the model's intrinsic OOD de- fection capability and also benefits ID generalization. As shown in Figure 3, in the later stages of vanilla training, the model's OOD detection capability weakens due to its mem- orization of atypical samples. In this regard, PSKD restores the model's 's intrinsic OOD detection capability through self- distillation and further enhances it by effectively learning certainty from atypical samples. Moreover, PSKD achieves additional improvements in the original task by making bet- ter use of atypical samples, without requiring a trade-off between ID task performance and OOD detection perfor- mance, as would occur when directly forgetting valuable atypical samples.
Large-scale experiments are conducted on ImageNet-200.. a 200-class subset of Image IK (Deng et al., 2009), as the ID dataset. Evaluations cover near-OOD scenarios, includ- ing SSB-hard (Vaze et al., 2022) and NINCO (Bitterwolf et al., 2023), as well as far-OOD scenarios, including iNat- uralist (Horn et al., 2018), Textures (Cimpoi al., 2014), and OpenImage- O (Wang al., 2022). The rest 800 classes from ImageNet-1K 1K are used as realistic outlier data.
Evaluation Metrics. We report three widely adopted met- rics: (1) FPR95, the false positive rate of OOD data at a 95% true positive rate of ID data; (2) AUROC, commonly referred to as AUC, is used here following (Zhang et al., 2023b) to denote the area under the receiver operating char- acteristic curve; and (3) ID ACC, the classification accuracy the ID test set.
Remark 2. PSKD alleviates the overconfidence of OOD data by learning with uncertainty-embeddeed targets. Traditional supervision is provided via one-hot vectors, which fails in uncertainty learning by assigning absolute confidence to samples with varying levels of uncertainty. This can make the model prone to overconfidence when confronted with OOD samples. To address this problem,
'The code for this work is available at https: /tte:://gittie /gtthbb. com/ hon/njustkngg CCM125 2-255P
5

---

Strengthen Out-of-Distributiin Detection Capability with Progressive Self-Knowledge Distillation
Table 1: Comparison on CIFAR benchmarks. All values are percentages, and OOD detection results are averaged over multiple OOD test datasets. The best results are in bold, with the second-best underlined. Detailed results for each OOD dataset are provided in Appendix G. f indicates that larger values are better, while + indicates that smaller values are better.
CIFAR-10
CIFAR-100
Method
Near-OOD
Far-OOD
Ncar-OOD
Far-OOD
AUROCT ID ACC1
ID ACC1
FPR95
AUROCT
FPR95
FPR95 AUROCT
FPR95
AUROCT
OOD scoring methods (vanilla training with cross-entropy)
MSP
48. 18.19
88.03+022
31.72=11.4
90.73+0.43
95.06=030
54.30
80.27+0.11
58.70_1.06
77.76+004
77.25+0010
ODIN
76. 19.108
82.87:185
57.62:2422
87 0.
95,06
57.91
79,90
58.86-0.9
79. 79.281011 1021
77.25:0010
Energy
61.34:1463
87.58+046
41.69
91.21
95.06=0.30
55.62#00.1
80.91
56.59=1.38
79.77
77.25+0.10
Energy+UM
13.12:10:47
90.601037
22.95=1.65
93.67 20.68
92.33
65.86:12.06
77.14:062
51.90_204
72.21 81.63
Energy+UMAP
33.01
91.00+aor
21.70
94.20+0.6
95.06=0330
59.71
79,49
52.11=2.36
81.62+1.37
77.25+0.0
ESSD
31.67.10.78
91.71
20.48:11300
94.56.1029 95.14:000
54.8312.79
81.45::0.4
51.56-2.39
82.40::.52
77.44:..09
Training- time regularization methods (w/ realistic outliers)
LogitNorm
29.34::0.1
92.33++008
13.81 =0.20 96.7440.06
94,30=02s
62.89+0.77
78.47:031
53.61=345
11.5
76.34
SNN
37.21
90.25
26.05 22.34 92,49
9..1
60.32.11.41
80,33
53.52-1.77
12.17
77.561007
T2FNorm
26.47+0035
92.79++013
12.75=0.73
94.69 96.98+0.23
58.47+1135
79.84+0.0
51.25-252
82.73+101
76.4
T2FNom+UM
27.79
92.7410.16
14.0621.31
96.80.0.28
94.02
59 59.42.1074
79.32.024
51.09_0.98
83.71
75.671041
22FNom++UMA
26.78
23.08+0.28
13.11
=7.01
94.69
59.77=000
79.63+028
49.92-4.16
83,81+184
76.43++013
T2FNormPPSKD
25.72.1028
93.1810.12
13.07
96.941..11 95.06::001
57.26:10.6
80.50.025
49.15
84.13::0.9
77.14
Training time regularization methods (w/ realistic outliers)
MixOE
51.45
88.73+082
33.84=4.77
91.93=t0.0
94.55+032
55.22:t0.9
80,95+0.0
63.88_248
76.40++144 1:44
DOE
20.3910.15
94,841007
15.59
94.67:0.69
94,32::019
1..8
86,61
45.38
84.301081
75:6910026
OE
19.84
94.821021
13.13
96.00
94.6310026
30.7310.11
08.30
54.82-2.79
81.41
1..84
OE+UM
18.18
95.044024
12.55=2.63
96.36220.7
94,51
30.71#0046
88.28+023
53.42_2.29
81.92+1.5
76.51 +0.37
OE+UMAP OE+PSKD
18.20
15.06110.4
12.12:2.24
96.37.006
94.63
30.1610.08
88,44:00s
54.07=121
82,42.1087
16.84
18.55+0221
95.08++.11
12.21
96.58=:0.44
94.75=a..s
31.55+00.7
88,24
48.72-0.7
84.60+0.43
76.99=...5
Baselines. We compare our proposed method with a suite of competitive baselines, including: (1) OOD scoring meth- ods with vanilla training (using cross-entropy loss): MSP (Hendryeks & Gimpel, 2017), ODIN (Liang et al., 2018), and Energy (Liu et al., 2020); (2) training-time regulariza- tion methods without realistic outliers: LogitNorm (Wei et al., 2022). T2Fnorm (Regmi et al., 2024), and SNN (Ghosal et al., 2024); and (3) training-time regularization methods with realistic outliers: OE (Hendrycks et al., 2019), MixOE (Zhang et al., 2023a), and DOE (Wang et al., 2023). Moreover, we compare the most relevant methods, UM and UMAP (Zhu et al., 2023), both of which, same as our PSKD, function as plug-in solutions designed to restore the model's intrinsic OOD detection capability.
rameter selection is performed via grid search using the ID and OOD validation samples from OpenOOD, and the parameters achieving the best AUROC are used for the final test. We leave more implementation details in Appendix B.
5.2. Main Results
In this section, we present a comprehensive evaluation of our proposed PSKD method on both small-scale CIFAR benchmarks, detailed in Table 1, and the more realistic and challenging ImageNet benchmark, summarized in Ta- ble 2. The experimental results demonstrate that: (1) On small-scale CIFAR benchmarks, PSKD can consistently im- prove baseline methods by restoring the model's intrinsic OOD detection capabilities across different OOD scenar- ios, while also achieving additional gains in ID accuracy. (2) On the large-scale ImageNet benchmark, PSKD con- tinues to exhibit competitive OOD detection performance while achieving the best ID accuracy among the compared methods. (3) Overall, PSKD outperforms the UM/UMAP method across a range of benchmark tests. This demon- strates the efficacy of the self-learning mechanisms, which leverage the continuously evolving self-selection teacher model to effectively push the model beyond its intrinsic OOD detection capability limits.
Implementation Details. In line with OpenOOD (Zhang et al., 2023b), we adopt R8 (He al., 2016) as the backbone architecture. Models are trained with stochastic gradient descent (SGD) for 100 epochs, using a learning rate of 0.1 with cosine annealing decay schedule (Loshchilov & Hutter, 2017), momentum of 0.9, and weight decay of x 10-4, The batch size is set to 128 for CIFAR- 100 and 256 for ImageNet-200. For validation, 1000 samples from the official test set are used, while the remainder are for testing. The model's 's self-selection mechanism is applied after the validation step at the end of each epoch. Hyperpa-
6

---

Strengthen Cit-oo-DDittriutionn Detection Capability with Progressive Self-Knowledge Distillation
Table 2: Comparison on large-scale ImageNet benchmark. All values are percentages, and OOD detection results are averaged over multiple OOD test datasets. Detailed results for each OOD dataset are provided in Appendix G.
Near-OOD AUROCT
Far-OOD
Average
Method
ID ACC1
FPR954
FPR95
AUROCT
FPR95
AUROC1
0OD scoring methods (vanilla training with cross-entrupy)
MSP
54.82:10.35
03.34:00066
35.43::0.38
90.13
45.13
86.37 86.74_0.77
ODIN
66.76
80.27
34,23=1.05
90.71
50.50=0.55
85.99=0.13
86.37=0008
Energy
60.2440.57
82.50=00.05
34.86=1.30
90.86=0.21
40.55
86.68=0.12
86.37=0008
Energy+UM
60.23=+1.13
81.79:0.38
32.46=1130
91.68=0.37
46.34
86. 06.74
85.01=033
Energy+UMAP
60.81
81.08.10.33
32.47
91.62 1.029
46.64
86.35
86.37.0.0
Energy+PSKD
57.12.1063
82.84
31.6410.87
10.16
44.38 1.022
87.12 0.15
86.791.0 0.225
Training-tima regularization methods (w/o realistic outliers)
LogitNorm
1..80
82.21
20.21
93.3 15.31:100 10.15
41.56:10.1
87.76=0.31
1 18:000
SNN
59.85
81.33310.19
28.04:1064
92.28
43.95
86.80=0.16 16
86.56610.08
T2FNorm
55.65-1020
82.72:10.05
25.25-10.48
23.38-10111
40.45-10.33
88.06=00.
86.52.00.1
T2FNorm+UM
55.20++0.71
02.89
26.31=0091
=3.21
40.76:20.11
88.05
=.221
T2FNorm+ UMAP
16.1441..99
82.52:+0.22
26.25=20.00
93.02=0.099
41.20::0.43
87.77=0.19
86.52=
12FNorm+PSKD
55.06:10466
82.83-+0.12
24.77:10.55
93.61
39.91:1051
88.22
8..74:0.09
Training-ting regularization methods (w/ realistic outliers)
MixOE
57.25
82.57:10.23
40.12:10.66
88.3911.00
49.03:10.44
85.73 85.49=0.12
DOE
54.14+0.51
83.23:10.60
37.60+22995
88.24=1.45
45.87=1.72
85.73=1.00
79.87=3.12
OE
52.55=00.1
84.51+0021
18.183
88.21
43.86=20.99
86,36=0.20
OE+UM
51.89+0064
84.94=0017
34.76=00.56
88.68=0.07
43.33=0031
86.81=0.08
85.7 00223
OE+UMAP
52.34.10.22
84.56.10.08
35.03
88.39
43.68
86.48 1011
85.78 10.12
OE
51.88
84.98.10.10
34.14
88.61 10.10
43.01 110.12
86. 0.00
86.31 0.11
AUROC
ACC
96
94-
AUROC
96
94
AUROCT
92
94
AUROC
92
94
AUROC1
92
ACC
AUROC
92
ACC1
90
32
92
Const
Linear
CO5
0.00
0.01
0.10
0.50
90
08
20
10
20
Schedul
Asrol
(a) Diff. Schedules for X
(b) Diff. Final Weight Atial
(c)
Diff. Weight a
(d) Diff. Temperature
Figure 5: Ablation studies on the hyperparameters: (a) influence of various adjustment strategies for X; (b) impact of final
weight Afina in cosine annealing schedule; (c) impact of varying the weighting factor a in ROoD: (d) effects of different temperature scaling T for self-distillation.
5.3. Ablation Study
tion, with Cos showing the most pronounced improvements. Figure 5(b) illustrates the impact of varying Xrial in Equa- tion (5) within the Cos strategy. The significant decline in performance when Agiall = 0 highlights the importance of PSKD in OOD learning. Moreover, since the learning objective of PSKD aims to provide secondary information, relatively smaller parameter setting is most beneficial, whereas excessively large settings may lead to misleading results and degraded performance.
Effect of Adjustment Strategies for A. Figures 5(a) an- alyze the impact of different X-adjustment strategies in Equation (4), specifically considering the constant strategy (Const), where À remains fixed; the exponential strategy (Exp), which leads rapid initial growth of X; the linear strategy (Linear), where increases at constant rate; and the cosine strategy (Cos) with an initial soft decline de- scribed in Equation (5). Appendix B.3 provides the details of various adjustment strategies. The results indicate that Const and Exp strategies over-rely on the OOD teacher early in training, preventing the model from effectively learning ID knowledge. thereby constraining the development of OOD discrimination capabilities. In contrast, Linear and Cos strategies better promote ID learning and OOD detec-
Effecc of the Weighting Factor a. In Figure 5(c), we evalu- ate the impact of varying the weighting factor a in Equation (9), which governs the balance between response-level and feature-level knowledge in distillation. As performance markedly worsens when feature-level distilla- tion is excluded (i.e., = 0), underscoring the critical role
see, the
7

---

Strengthen cuttoo-DDistibuuie Detection Capability with Progressive Self-Knowledee Distillation
Table 3: Impact of varying teacher selection intervals on CIFAR- 10 benchmark. OOD results are averaged over near- and far-OOD groups. * denotes the default setting.
Table 4: Teacher selection strategy comparison. PSKD-S refers to the strategy without progressive selection. The best results are in bold, and the second-best are underlined.
Teacher Selection Interval
FPR95 AUROCT ID ACC1
Dataset
Method
FPR95
AUROCT ID ACC1
10 selections per epoch
26.25:::.46
93.11:0.22
95.08:0.22
Vanilla
51.52:3382
89.40
95.06+10.30
selections per epoch selection per 1 epoch*
25.79::.17 93.29:0.30 95.30
CIFAR-I0
+PSKD-S
18.18:13.29
92.82
95.06..0.05
26.08=11022
93. 04 95. 05.14
+PSKD
26.08=1.02
93.14=0021
95. 0..048
selection per 5 epochs
26.86+0091 92.96+0037 95.14
Vanilla
16.10.10.99
10.34....34
77.25 100100
CIFAR-100
+PSKD-S
53.40
81.78 1.066
77.57 1.007
+PSKD
53.19=1.13
81.93=0013
77.44+0..9
of feature-level OOD discrimination knowledge. Further- more, large a places too much emphasis on feature-level distillation, leading suboptimal performance. Therefore, striking a moderate balance is more beneficial for uncer- tainty learning, resulting in better performance.
Vanilla
47.55
16.68
86.37.10.0
ImageNet-200
+PSKD-S
46.32=0:43
86.90=0.16
86.86+10.13
+PSKD
44.38= 07.1 86. 05
Table 5: Impact of OOD scoring selection on AUROC based on the CIFAR-10 benchmark.
Effect of Temperature Scaling T. Figure 5(d) examines the impact of temperature scaling T on the uncertainty- embedded target learning in distillation. Compared to the baseline (i.e. 7 1), higher 7 facilitates knowledge trans- fer by softening targets, thereby providing richer discrim- ination signals for the distillation process and ultimately improving performance. In contrast, excessively high tem- peratures can overly smooth the teacher signals, resulting in a loss of information. This undermines the intended purpose and degrades overall performance. Thus, a mild temperature provides the best effect for OOD knowledge learning.
Self-selection OOD score
Test-time OOD score
Method
MSP
Energy
KNN
Vanilla
89.38:10.18
89.40+0442
91.99+0.10
MSP Energy KNN
+PSKD
91.2911043 92.79:1040
92.10
+PSKD
91.57:10.2 93.14
92.54
+PSKD
91.24:10.22 92.5210.88 92.71
Impact of OOD Scoring Selection. Table 5 examines the generalization performance of PSKD under different OOD scores selected during self-selection and testing time. The analysis covers various OOD scoring functions, including logit- based methods like the baseline MSP (Hendrycks & Gimpel, 2017), Energy (Liu et al., 2020), and the feature- based KNN (Sun et al., 2022). The results reveal that: (1) Aligning self-selection and test OOD scores (diagonal en- tries) generally enhances performance. Notably, within the same level, superior self- OOD scores better guide teacher self-selection. For example, using Energy scores for self-selection improves MSP test AUROC by 0.28% over matching MSP scores. (2) Even when the score is based on different levels, the model can still drive performance gains through the use of PSKD. For example, using logit-based Energy scores for self-selection and feature-based KNN scores for testing results in a 0.55% AUROC boost over the baseline. (3) Better OOD scores are more critical and lead to improved performance, i.e., the Energy scores for self-selection and testing achieve the best performance.
Effect of Teacher Update Frequency. In Table 3, we exam- ine the effect of teacher model update intervals training stability and final performance. The results demonstrate that PSKD maintains stable training dynamics across a range of update settings. Notably, appropriately increasing the update frequency can enhance OOD detection performance by providing the model with more frequent guidance. This enables the model to better explore and refine its intrinsic OOD detection capacity. However, such gains come at the cost of increased computational overhead, highlighting trade-off between performance and efficiency.
5.4. Further Analysis
Impact of Teacher Selection Strategies. To validate the effectiveness of progressive teacher selection, we design another version of the teacher selection strategy: the sta- tionary strategy (PSKD-S), wherein models are sequentially compared in pairs based on the value of A(.) with Equa- tion (6) throughout the training process identify the one with the best OOD performance. The model exhibiting the best OOD detection capability is selected as fo- and used to guide the retraining of models to learn better OOD dis- crimination capabilities. The results in Table 4 validate that PSKD performs better than PSKD-S in all OOD detection tasks, which verifies that progressive distillation can further enhance performance by continually refining itself.
Impact of Pseudo-Outlier Generation Strategies. Table 6 explores the effect of different sources of pseudo-outlies for the teacher model self-selection. Several cost-effective strategies are considered, including image rotation (Rot) to enhance diversity, image distortion (Dist) through resiz- ing followed by restoration, and the addition of Gaussian noise (Gauss) at varying intensities. Specific details are provided in Appendix B.4. The results show that: (1) The
8

---

Strengthen oulloo--Distiibttion Detection Capability with Progressive Self-Knowledge Distillation
Table 6: The impact of different sources of pseudo-outliers on the CIFAR-10 10 benchmark. OOD results are averaged over near- and far-OOI) groups.
References
Ahn, S., Hu, S. X., Damianou, A. C., Lawrence, N. D., and
Dai, Z. Variational information distillation for knowledge transfer. In CVPR, pp. 9163-9171, 2019.
Source
FPR951
AUROCT ID ACCT
Rot
Dist Gauss
Bendale, A. and Boult, T. E. Towards open set deep net- works. In CVPR, pp. 1563-1572, 2016.
51.52:3.82
89,4010422
95.06.10.30
28.52.11.26
92.6510.13
95.07.10.12
Bitterwolf, J., Müller, M., and Hein, M. In or out? fixing im- agenet outff-dittoo detection evaluation. In ICMI., pp. 2471-2506, 2023.
X
v
30.88.11.66
92.4110.16
95.15
x
V
v
20.78
92.87110.33
95.111000
V
v
26.08=1102
93.14+0021
95.14
Bucila, C., Caruana, R., and Niculescu-Mizil, A. Model compression. In KDD, pp. 535-541, 2006.
Dist and Gauss strategies achieved promising results by dis- rupting image features to construct pseudo-outliers data. (2) Combining the Rot strategy further improves performance by augmenting diversity and expanding the pseudo-outliers distribution. (3) When all considered strategies are com- the most diverse
Cao, S., Li, M., Hays, J., Ramanan, D., Wang, Y., and Gui, L. Learning lightweight object detectors via multi-teache progressive distillation. In ICML,, pp. 3577-3598, 2023.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. CVPR. pp. 3606-3613, 2014.
bined, the generated pscudo-outliess and informative, leading the best performance. Overall, performance remains relatively stable across different data quality levels, with high-quality validation sets generally yielding the most significant performance gains.
Deng. J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
6. Conclusion
Deng. L. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6): 14--144, 2012.
In this paper, we explore storing models' intrinsic OOD detection capabilities via self-learning mechanisms, aim- ing to establish robust model basic for OOD detection. To address the issue of suboptimal OOD detection perfor- mance during the later stages of training, we propose the PSKD framework. This framework progressively refines the model's OOD discrimination capability by leveraging self-selection teacher model. Extensive experiments demon- strate that PSKD consistently improves OOD detection per- formance and further enhances the effectiveness of other OOD detection methods.
Veiris,, T. and Taylor, G. W. Learning confidence for out-of-distribution detection in neural networks. CORR, abs/1 abs/1802.048655 2018
Djurisic, A., Bozanic, N., Ashok, A., and Liu, R. Extremely simple activation shaping for outoo--distribution detection. In ICLR, 2023.
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Acknowledgements
National Key RD Program of China (2022YFF0712100, NSFC (62276131), Natural Science Foundation of Jiangsu Province of China under Grant (BK20240081), the Fun- damental Research Funds for the Central Universities (N0.30922010317, No.30923011007) Key Laboratory of Target Cognition and Application Technology (2023-XXPT- LC-005).
Geiger, A., Lenz, P., and Urtasun, R. Are we ready for autonomous driving? the KITTI vision benchmark suite, In CVPR, pp. 3354-3361 2012.
Ghosal, S. S., Sun, Y., and Li, How to overcome curse- of-dimensionality for out-of-distribution detection? In AAAI, pp. 19849-19857, 2024.
Impact Statement
Gou, J., Yu, B., Maybank, S. J., and Tao, D. Knowledge distillation: A survey, CoRR, abs/2006.05525, 2020.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
9

---

Strengthen Cut-of-Distributiin Detection Capability with Progressive Self-Knowledge Distillation
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and -f-ttiiiuitttnoonn examples in neural networks. In ICLR, 2017.
Ming, Y., Fan, Y., and Li, Y. POEM: out-of-distribution detection with posterior sampling. In ICML, pp. 15650- 15665, 2022.
Hendrycks, D., Mazeika, M., and Dietterich, T. G. Deep anomaly detection with outlier exposure. In ICLR, 2019.
Mobahi, H., Farajtabar, M., and Bartlett, distillation amplifies regularization in hilbert space. In NeurIPS, 2020.
Self-
Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, Reading digits in natural images with unsuper- vised feature learning. 2011.
Horn. G. V., Aodha, O. M., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In CVPR, pp. 8769-8778, 2018.
Nguyen, A. M., Yosinski, J., and Clune, J. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, pp. 427-436, 2015.
Pei, S. Image background serves good proxy for out-of- distribution data. In ICLR, 2024.
Huang, R. and Li, Y. MOS: towards scaling out-of- distribution detection for large semantic space. In CVPR, pp.
2021.
Regmi, S., Panthi, B., Dotel, S., Gyawali, P. K., Stoyanov, D., and Bhattarai, B. T2fnorm: Train-time feature nor- malization for OOD detection in image classification. In CVPRW, pp. 153-162, 2024.
Huang, R., Geng, A., and Li, Y. On the importance of gradients for detecting distributional shifts in the wild. In NeurlPS, pp. 677-689, 2021.
Romero, A., Ballas, N., Kahou, E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Jiang, W., Cheng, H., Chen, M., Wang, C., and Wei, H. DOS: diverse outlier sampling for out-tt-disttibuttinn detection. In ICLR, 2024.
Sehwag, V., Chiang, M., and Mittal, SSD: A unified framework for self-supervised outlier detection. In ICLR, 2021.
Krizhevsky, Learning multiple layers of features from tiny images. 2009.
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Sun, Y., Ming, Y., Zhu, X., and Li, Y. -fitof-distribution detection with deep nearest neighbors. In ICMI., pp. 20827-20840, 2022.
Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS, pp. 7167-7177, 2018.
Vaze, S., Han, K., Vedaldi, A., and Zisserman, A. Open-set recognition: A good closed-set classifier all you need. ICLR, 2022.
Li, Y. and Vasconcelos, N. Background data resampling for outlier-aware classification. CVPR, pp. 33215-13224, 2020.
Wang. H. Li, Z., Feng, L., and Zhang, W. Vim: Out-of- distribution with virtual-logit matching. In CVPR, pp. 4911 4911-4920, 2022.
Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of diitrr--iistibutioon image detection in neural networks. In ICLR, 2018.
Wang, Q. Ye, J., Liu, F, Dai, Q., Kalander, M., Liu, T., Hao, J., and Han, B. Out-of-distribttion detection with implicit outlier transformation. In ICLR, 2023.
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., Van Der Laak, J. A., Van Gin- neken, B., and Sánchez, C. I. A survey on deep learning in medical image analysis. Medical image analysis, 42: 60-88, 2017.
Wei, H., Xie, R., Cheng, H., Feng, L., An, B., and Li, Y. Mitigating neural network overconfidence with logit normalization. In ICML, pp. 23631-23644, 2022.
Xu, H. and Yang, Y. ITP: instance-aware test pruning for detection. In AAAI, pp. 21743-21751,
Liu, W., Wang, X., Owens, J. D., and Li, Y. Energy-based out-of-distribution detection. In NeurIPS, 2020.
2025.
Xu, K., Chen, R., Franchi, G., and Yao, A. Scaling for training time and post-hoc distribiitt detection enhancement. ICLR, 2024.
Loshchilov, I. and Hutter, F. SGDR: stochastic gradient descent with warm restarts. In ICLR, 2017.
10

---

Strengthen Out-of-Distribution Detection Capability with Progressive Self-Knowledge Distillation
Yang, C., Xie, L.., Su, C., and Yuille, A. L.. Snapshot dis- tillation: Teacher-student optimization in generation. In CVPR, pp. 2859-2868, 2019.
Yang. J., Zhou, K., Li, and Liu, Z. Generalized out-of- distribution detection: A survey. CoRR, abs/21 10.1 2021.
Yang. Y., Zhang. Y., Song, X., and Xu, Not all out-of- distribution data are harmful to open-set active learning. In NeurIPS, 2023.
Yang. Y., Jiang, N., Xu, Y., and Zhan, D. Robust semi- supervised learning by wisely leveraging open-set data. IEEE Transactions on Pattern Analysis and Machine In- telligence, 46(12):8334-832) 2024.
Zhang. J., Inkawhich, N.. Linderman, R., Chen, Y., and Li, H. Mixture outlier exposure: Towards ouitt--distribution detection in fine-grained environments. WACV, pp. 5520-5529, 2023a.
Zhang, J., Yang, J., Wang, P., Wang. H., Lin, Y., Zhang, H., Sun, Y., Du, X., Zhou, K., Zhang, W., Li, Y., Liu, Z.,, Chen, Y., and Li, H. Openood v1.5: Enhanced benchmark for
detection. CoRR, abs/2306.00 093011,
2023b.
Zhang. L., Song, J., Gao, A., Chen, J., Bao, C., and Ma. K. Be your own teacher: Improve the performance of convolutional neural networks via self distillation.
ICCV. pp. 3712-3721, 2019.
Zhang, Z. and Sabuncu, M. R. Self-distillation as instance- specific label smoothing. In NeurIPS, 2020.
Zhang, Z. and Xiang, X. Decoupling maxlogit for out-of- distribution detection. In CVPR, pp. 3388-33977 2023.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Tor- ralba, A. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
Zhou, Z. Guo, L., Cheng. Z., Li. Y., and Pu, STEP: nut-of-distribution detection in the presence of limited in- distribution labeled data. In NeurIPS, pp. 29168-29180. 2021.
Zhu, J., Li, H., Yao, J., Liu, T., Xu, J., and Han, B. Un- leashing mask: Explore the intrinsic -iiti--disttibution detection capability. In ICML, pp.
2023.
Zhu, Y. Chen, Y.. Xie, C., Li, X., Zhang, R., Xue, H., Tian, X., Zheng, B., and Chen, Y. Boosting nut-of-distribution detection with typical features. In NeurIPS, 2022.

---

Strengthen Out-of-Distribution Detection Capability with Progressive Self-Knowledge Distillation
Appendix
A. Pseudo Code of PSKD
The Pseudo code of PSKD available in Algorithm
Algorithm PKSD: Progressive Self- KofflKnoweedge Distillation
Input: Training dataset priin validation dataset Daal- initial model parameters 0
Output: Trained student model fo
1: Initialize teacher model: 0" =
2: Generate pseudo-outlier dataset DOoD from Dval
3: while student model fe has not converged do
4: Update fo to minimize both (ID(Jo and Roo So-) with weight factor in Equation (4) over Drain-
5: if a predefined number steps (e.g., one epoch) is completed then
6: Quantify the OOD detection capability of fo and fo- based on val and DooD following Equation (6): A(fo) and
A(fe-)
7:
if A(fo-) A(fo) then
8:
Update teacher model: 0" 0 end if
9:
10:
end if
11:
Adjust weight factor following Equation (5)
end while
Return fe
B. Implementation details
B.1. Software and Hardware
All experiments conducted using Python 3.8.19 and PyTorch 2.0.1 on workstation equipped with dual 2.20 GHz CPUs, 384 GB of RAM, and six NVIDIA RTX 4090 GPUs.
B.2. Validation Strategy
We use the ID and OOD validation samples from OpenOOD for hyperparameters selection of PSKD. We utilize grid search over all possible values of Amna x x to determine the optimal settings, where vary Xŭnal (0.01, 0.02, 0.03, 0.05, 0.1), 5). T {3, 5}. The hyperparameter that yields the best AUROC is used for the final test. We adopt the same hyperparameters in the same model in all experiments. For CIFAR benchmarks, the optimal settings are Ainal 0.01, = and 7' 3. For ImageNet-200, Atnal 0.05, a 1 and T = 3 is optimal.
B.3. Details of Adjustment Strategies for X
We provide details regarding the various À adjustment strategies in Equation (4) below.
Let Àiina represent the final weight, and let denote the total number of training epochs. Four adjustment strategies are considered: constant strategy (Const), exponential adjustment strategy (Exp), linear adjustment strategy (Linear), and cosine annealing adjustment strategy (Cos). The mathematical expressions for these strategies are as follows:
[ (1-exp)
E (
e Acos
Aginal (1 (1- cos (G)7) x
Aonnst = Annall
AxEp
Atinal
exp
ALincar = Attnal
(10)
E"
where ACons remains fixed at the final weight Atnal- Axpp increases following an exponential growth influenced by parameter (set to 5 in our experiment), ALiearr increases linearly with the epoch number, and Aces increases according to a cosine annealing schedule. The visualizations of these adjustment strategies are presented in Figure
12

---

Strengthen oittr--Distribution Detection Capability with Progressive Self-Knowledge Distillation
1.0
0.8
0.6
0.4
Acoet
Apsp
Auhoar
0.0
Aces
0
20
40
60
80
100
Figure 6: Curves of À under four different adjustment strategies, with the parameters fixed at Asial - and E' = 100.
B.4. Details of Pseudo-Outlier Generation Strategies
We consider three cost-effective strategies for generating pseudo-outliers: rotation, distortion, and Gaussian noise. The detailed implementation settings are provided below:
Rotation: We apply random rotation to enhance the diversity of pscudo-outliers and introduce shifts, with the rotation angle uniformly sampled from (90", 180%, 270%).
Distortion: To generate challenging pseudo-outliers that resemble ID samples, we apply a distortion technique. Given an original RGB image of size (3, h, w) and distortion factor E [0, 1]. the image is first downsampled to (3, [y/h,, [yw] J) using bilinear interpolation. It then upsampled back its original dimensions (3, h, w) using bilinear interpolation. This procedure leads loss of fine- ane-grained feature information while retaining the essential scene-level content from the original image. For the CIFAR benchmark, the distortion factor set to 0.16, which corresponds downsampling image from its original size of 32 32 pixels to 5 x 5 pixels and then upsampling it back to 32 32 pixels. For the ImageNet benchmark, the distortion factor is set to 0.05, meaning the original 224 224 pixel image is downsampled to 11 11 pixels and then upsampled back to 224 x 224 pixels.
Gaussian Noise:
Noise with varying intensities is added to achieve different levels random corruption to original ID images, simulating both near-OOD and far-OOD scenarios. Gaussian noise with a mean of and variance of
added the normalized images, where a takes each of the 10 values from the set (0.00, 0.02, 0.04, 0.06, 0.08, 0.10, 0.12, 041 0.16, 0.18). This process expands the dataset by factor of 10, aiming to address the limited size of the validation set provided by OpenOOD, ultimately facilitating accurate identification of the desired OOD teacher model.
C. Teacher Selection with Realistic Outlier
To ensure fair comparison, we intentionally avoid using realistic OOD data as auxiliary information the main paper. To further explore this aspect, Table examines the impact of using realistic outliers for teacher model selection. The results indicate that both realistic and pseudo-outlies data consistently lead significant performance improvements. Moreover, realistic OOD data can more accurately reflect the actual OOD distribution, thereby enhancing the robustness of teacher model selection and further improving performance.
Table 7: The impact of different sources of OOD data for teacher selection ImageNet-200 benchmark. Following the settings of the OpenOOD benchmark for the OOD validation set, Openlmage-O is adopted as the realistic OOD data. The OOD results are averaged over near- and far-OOD groups
Methods
FPR95L
AUROCT
ID ACC1
Vanilla
47.55
86.68
86.07.10.08
+PSKD w/ Pseudo
44.38.1022
87.12.0.15
06.79..0.25
+PSKD w/ Realistic
43,93+0.27
87.64=0.18
86.76=0.16
13

---

Strengthen
Detection Capability with Progressive Self-Knowledge Distillation
D. Analysis of Computational Overhead
Table presents the additional training overhead introduced by PSKD on both small-scale and large-scale datasets. The results indicate that: (1) the overhead of the teacher selection process accounts for only a small fraction of the total training cost, which is affordable. Notably, for large-scale datasets, the additional overhead of PSKD is even less significant due the relatively higher ratio of training samples validation samples. (2) The generation of pseudo-outliers is conducted only once at the beginning of training, and its computational cost is minimal compared to the overall training time.
Table 8: Overhead analysis introduced by PSKD. The training setup follows the OpenOOD benchmark, consisting a total of 100 epochs for CIFAR-I0 (Small scale) and 90 epochs for ImageNet-200 (Large scale). The results are averaged over five independent runs. Appendix B.I outlines the software and hardware configurations.
Dataset
One-Epoch Training
One-Time Teacher Selection
Total Training
Pseudo-Outlice Preprocessing
CIFAR-10
10.50 seconds
1.18 seconds
21.96 minutes
0.96 minutes
ImageNet-200
252.30 seconds
8.72 seconds
397.49 minutes
1.42 minutes
E. Architectural Variants
Table 9 analyzes the robustness of PSKD across different architectures, including the CNN-based ResNet-18 and the munsformer-baseee Vision Transformer (Dosovitskiy et al., 2021) (ViT-B/16), The results indicate that PSKD consistently improves the model's OOD detection performance across different architectures and demonstrates general applicability.
Table 9; The robustness of PSKD across different architectures on the ImageNet-200 benchmark. The Energy score adopted for OOD scoring and the OOD results are averaged over near- and far-OOD groups.
Model
Methods
FPR95
AUROCT
ID ACC1
Vanilla +PSKD
47.55+40.66
86.68 12
86.37
ResNet-18
44.38++0.22
87.12:20115
86.79
Vanilla +PSKD
28.8011041
93.61:10.19
93.9011.110
Vit-B/16
27.79+00.00
93.87=0024
94.01
F. Statistical Tests for Comparison
In this section, we report the statistical test results using AUROC as the OOD detection metric on the CIFAR-10 benchmark. First, we rank the performance of all methods across multiple datasets, with the results recorded in Table 10. We then perform the Friedman test to determine whether there is a significant difference in the average rankings of the methods. The Friedman test yields a test statistic of 25.524 and a p-value of 0.00011 (which is less than the significance level of 0.05), indicating a statistically significant difference among the methods. To further identify specific methods with significant performance differences, conduct Nemenyi test, obtaining Critical Difference (CD) value of 3.078, and present the comparative analysis of ranking differences between our PSKD and other methods in Table 11.
The statistical test results indicate that (1) compared to Energy, Energy+PSKD effectively enhances the model's OOD detection capability and demonstrates a significant performance improvement; and (2) the lack of significant difference between our PSKD and Unleashing Mask/Unleashing Mask Adopts Pruning (UM/UMAP) can be attributed to our shared goal of restoring the model's 's intrinsic OOD detection capability. The distinction lies our method, which utilizes an uncertainty-embedde target learn valuable atypical samples, whereas UM/UMAP directly discards them, resulting in loss of ID generalization performance and limiting the model's OOD detection capability.
Table 10: Average ranking results of methods across multiple datasets.
MSP
ODIN
Energy
Energy+UM
Energy+UMAP
Energy+PSKD
Average Rank
4.67
5.67
4.67
2.67
1.83
1.50
14

---

Strengthen oitto--Disttibution Detection Capability with Progressive Self-Knowledge Distillation
Table 11: Ranking differences between our PSKD method and other comparison methods, with an asterisk (*) indicating a
significant difference, where the value exceeds the CD value of 3.078.
MSP
ODIN
Energy
Energy+UM
Energy+UMAP
Ranking Difference
3.167*
4.167
3.167*
1.167
0.333
G. Fine-grainee Results
We present the fine-grained results of our experiments on the CIFAR benchmarks, including six OOD datasets (CIFAR- 10/100, TinylmageNet, MNIST, SVHN, Textures, Place365), as well as on ImageNet-200, with five OOD datasets (SSB-hard, NINCO, iNaturalist, Textures, OpenImage-O), The results on the CIFAR- 10 benchmark are reported in Table 12 and Table 13, while the results on the CIFAR- benchmark are shown in Table 14 and Table 15, and the results on the ImageNet-200 benchmark are presented in Table 16 and Table 17. Our findings include: (1) By equipping with PSKD, the baseline can achieve significant improvements and outperform other methods most near- and far-OOD datasets. For example, on the CIFAR- 10 benchmark, PSKD can reduce FPR95 by 31.26%
CIFAR- 100 (near- -ODD and 28.26% on Textures (far-OOD). (2) In both w/o and w/ realistic outliers scenarios, the results consistently demonstrate that as plugin, PSKD effectively excavates the model's 's intrinsic capabilities and pushes their performance further. For instance, on the CIFAR-100 100 benchmark, PSKD reduced the FPR95 of OE on SVHN by 17. 7.1%. (3) In addition to the OOD detection performance improvement, PSKD can also improve ID accuracy by reasonably learning atypical samples. PSKD consistently achieved improvements of 0.08%, 0.19% and 0.42% in ID test accuracy on CIFAR-I0, 10, CIFAR-I00, and ImageNett220,, respectively. This contrasts with UM/UMAP, which directly forgets valuable atypical samples. Our proposed PSKD leverages the uncertainty-embeddes targets to learn from both typical and atypical samples, which achieves dual success in both OOD detection performance and main task performance.
Analysis of Performance Disparities. The performance improvement of PSKD is inherently dependent on the model's s intrinsic OOD capability relative to the training data. Compared to the small-scale CIFAR dataset, the large-scale ImageNet dataset encompasses a larger and more complex semantic space. Under the same ResNet-18 architecture used in our paper, models trained on the more challenging ImageNet dataset tend to learn a more crowded feature space, increased class overlap, and unreliable decision boundaries. These factors contribute to the model's relatively weak intrinsic OOD) detection capability (Huang & Li, 2021), limiting PSKD's potential restore the model's intrinsic OOD detection capability. Consequently, the performance improvements on ImageNet are less pronounced, while substantial gains are observed on the simpler CIFAR-10 10 benchmark. Although the improvements vary across datasets of different scales, PSKD consistently enhances the model's OOD detection performance, proving its effectiveness.
15

---

Strengthen Out-of-Distribution Detection Capability with Progressive Belf-Knowledge Distillation
Table 12: Fine-graina results (AUROC1) on the CIFAR-10 benchmark.
Near-OOD
Far-OOD
Method
ID ACC
CIFAR-100
Tiny Image Net
Average
MNIST
SVHN
Textures
Places365
Average
OOD scoring methods
(vanilla fraining with cross-entropy)
MSP
87.19+0..33
88.87=00119
88.03=+.225
92.63+1.557
91.46==0.40 89.89=0.71
88.92+0.47
90.73++0.43
95.06+0.30
ODIN
82.18:11.87
83.55 1.84
82.87.11.85
95.24:11.96
84,5810.77
86.9412.26
85.07.11.24
87.96.10.11
95.06.10.30
Energy
86.56.10.58
88.80 80.00.336
87.58100.4
94.32:12.53
91.79:10988
89.47:0.70
89.2510.78
91.21110.2
95.06:10.30
Energy+UM
89.08:10.33
92.12440.4
90.60:10.37
96.86:0.87
92.2942.86
92.1010.53
93.41+0.59
93.671:0.88
92.33::.41
Energy+UMAP
89.67-+0.18
92.33-1-0.09
91.00-+0.00
0..50
9..30+1.69
91.23++..2
93.58
94.20-10.33
95.06
Energy+PSKe PSKD
90.62.10.16
92.80.10.16
91.71
97.20:10.30 94.98.11. 93.38 10.37
92.69.1040
94.56.10.29
95.1410.08
Training-time regularization methods (w/o realistic outlers)
LogitNorm
90.95+0.22
93.70+00.06
92.33++0.08
99.14+0.45
98.25+0.411
94.77+0.43
94.79+0.166
96.74+00006
94,30+0.25
SNN
89.32+00.09
91.18+0.09
90.25++0.09
93.53++1.04
1.79
92.94+0.34
91.49+0.23
92.49+0.78
+5.11+00.18
T2FNorm
91.56:+0110
94.02:+0.16
92.79:1013
99.28+10.27
98.81=0.22
95.44=0.77
94.40:1.32
96.98:1022
94.69:1.007
12FNomm+UM
91.21+0..00
94.28
92.74=0.116
99.19::0.21
98. 08.12=00.08
95.10=0.64
94.79=+030
96. 06.80-00.28
94.02+0.13
12FNorm+UMAP
91.69+0.28
94.47+00.28
93.08-+0.28
99227++0222
98.63 08.63=0.335
95.30=0.13
94.74+0.27
97.01=+0.11
94.69+0.077
1ZENorm+PSKD
91.95:10.11
94.41
93.18:10.12
99.2311.119
98.5010..4 95.47.10.71
94.5411..11
96.94.10.11
95.06:10.01
Training-tiin time regularization methods (w/ realistic outliers)
MixOE
87.47:10.97
90.00:1.0.3
88,73:10.82
91.66:1..22
93.82:t1.27
91.84:10.51
90.38
0.69
94.55:10.32
DOE
92.63.10.21
97.04.10.27
94.84.10.07
15.16.22.20
99.20110.09 97.77.0.31 96.5710.11
94.67-10.69
94.32.10.19
OE
90.54110.53
99.11110.34
92.1..11
90.22:11.11
99,6010.14
97.58.0.27
96.5810.70
96.00-10.13
94.63110.26
OR+UM
90.65++0.66
99.42-+0.34
95.04++0.24
92.85+2.116
99.72+0.17
97.03+1.14
95.83++2.16
96.36++0.37
94.51+0.31
OR-UMAP
90.58+00.60
99.55++0.32
95.06++0.14
92.97+3.57
99.78+ 0.08
97.34+1.14
95.37+2.27
96.37+0.65
94.63+0.26
OE+PSKD
90.66:10.26
99.50-10.15
95.08:10.12
91.8012.19
99.73=0.11
97.8910.11
96.9110.37
96.58.10.44
94.75110118
Table 13: Fine-grained results (FPR95L) on the CIFAR-10 benchmark.
Near-OOD Tiny TinyImageNet
Far-OOD
Method
ID ACC
CIFAR-100
Average
MNIST
SVHN
Textures
Places365
Average
00D
scoring methods (vanilla truining with cross-entropy)
MSP
53.08.14.86
43.27:13.00
48.1743.92
23.6415.81
25.82:11 64 34.96
42.47=3.81
31.72-1.84
95.06:10.30
ODIN
77.00115.74
75.38:16.42
76.1926.08
23.83:112.34
68.6110.52
67.70=11.00
70.36=6.96
57.62-4.24
95.06:0.30
Energy
66.60:14.46
56.08:14.83
61.3424.63
24.99:112.93
35. 35.12:1..11
51.82:6.11
54.85=6.52
41.69-5.32
95.06:10330
Encrgy+UM
38.33:10.27
27.92:1.94
33.12=0..47
11.58=3.50
26.30=7.448
27.2442.78
26.67=1.99
22.95=1.65
92.33-0.41
Energy+UMAN
47.49:10:42
28. 44.444
33.01
8,5911881
20.1111..72
31.0011.088
26.49=1.88
21.70=1.57
95.06.10.30
Energy+PSKI
35.34:10.94
28.01
31,67:10,78
11.41:0.50
17.60-14.85
23.56:1148
29.34=2.41
20.48-1.30
95.14:10.08
Training regularization methods (w/o realistic outliers)
LogitNorm
34.37+11.0
24.30=0.54
29. 29.34=0.81
1.93
1.33
21.94+00.5
21.04=0.71
13.81=0.20
94.30=00225
SNN
41.00++0.76
33.43++0.2
37.21+0.70
22.92+3.86
25.33++3.64
24.34++1.77
31.60=1.04
26.05=2.34
95.11=00.13
T2FNorm
30.60+00.45
22.33+0.33
26.47+0.35
3.50++1.33
5.72+
19.49+2.58
22.27-1.28
12.75-0.7 0.73
94.69+0.07
T2FNorm*UM
32.60++0.80
22.98+0.33
27.79++0.55
3.63++0.96
8.39
21.81++2.79
22.39-0.91
14.06-1.31
94.02+0.13
U2FNor=+UMAP
31.61:10.72
10.79
26.78.10.75
2.95:11.12
6.87112.01
20.95:10.17
21767=0.72
13.11=0.81
94.69-10.07
T2FD
29.72:10.51
21.72.10.52
25.72.10.28
3.67:10.93
7.03:11011
19.48
22.11=0.33
13.07=0.17
95.06:10.01
Training-time regularization methods (w/ realistic outliers)
MixOE
58.2911.25
44.62.1..57
51.45:7.78
38.28:113:40
20.3613.99
33.19:14.28
43.54=4.95
33.84-4.77
94.55-1.32
DOE
28.27:10.84
12.50=10.95
20.39=10115
35.70-1.33
2.55:10.88
15.35+1.50
13.77=0.47
15.59=1.47
94.32-0.19
OE
36.71+2.00
1.97
19.84=0.95
24.67=2.55
1.25+00.66
12.0742.14
14.53=2.80
13.13-0.53
94.63-0.26
OB+UM
34.34422007
2.01+1.188
18.18=00.97
19.20=1.74
0.99:10.60
13.87455.65
16.12-6.99
12.55-2.63
94.51=0.31
OF+UMAP
34.78++1.68
1.36
18.20+00.51
16.86=4.94
0.77++0.18
12.70++4.77
18.16-
12.12-2.24
94.63+0.26
OE+PSKD
35.58-+0.71
1.54
18.55+0.21
22.87+3.83
0.78++0.11
1..68
14.00- 1.54
12.21-0.53
94.75 0.188
16

---

Strengthen Out-of-Distribution Detection Capability with Progressive Belf-Knowledge Distillation
Table 14; Fine-grained results (AUROC1) on the CIFAR-100 benchmark.
Near-OOD TinyImageNet
Far-OOD
Method
ID ACC
CIFAR-10
Average
MNIST
SVHN
Textures
Places365
Average
OOD scoring methods
(vanilla training with cross-enropy)
MSP
78.4710.07
82.07.10.17
80.2710.11
76.08
77.32.10.71 7.89
79.22.1..99
77.76.10.4
17.2510.10
ODIN
78.18:10.14
81.63110008
79.90.0.111
83.7911.31
7.754_0.76
19.33.1.00
79.4510.26
79.28:1.21
77.2510.10
Energy
79.05-00111
82.76+0008
80.91+0.08
19.18+1.37
82.03-1.774
78.35+0.83
79.52+0.23
79.77+0.61
77,25+0.10
Unergy+UM
72.86+1.00 1.03
81.43+0.58
77.14+0.62
86.79+2.97
86.00-3.61
16.58+1.033
77.54+1.44
11.63+1.70
72.21+0.43
Energy+UMAB
76.96=20.46
82.01=0.115
79.4921.22
84.9014.24
85.25-3.16
78, 1.36
78.17:0:42
81.624137
77.2540.10
EDeDgy+PSKD
0.93
8.174
81,45:0.44
0.7
86.20
1.32
0..64
80.59:0067
82.40-10.52
77.44
Truining-time regularization methods (w/o realistic outliers)
LogitNorm
74.57=0.39
02.37+0.24
78.47+0.31
90.69=1.38
82.80=4.57
72.37+0.67
80.25+0.61
1..53+1.26
76.34+0.117
SNN
76.99-0.25
83.68+0.20
80.33++022
82.90+1.21
83.06-2.44
82.98+0.37
79.74+00.27
82.17+0.69
77.56
T2FNorm
76.09=0.881
83.5940.02
79.8420.40
86.22+22.29
86.04=1.04
17.32+1163
81.35=0.33
82.73+1001
76.43-20.13
T2FNomm+UM
0.32
83.43+00.3
79.32-+0.24
90.89:1.97
36.22-1.95
77.63++0.82
80.10:+0.67
83.71-10.77
75.67
T2FNomm+ UMAP
75.69
83. 1.337
79.63 1..28
90.58100.8
85.67 5.89
1..72.11.34
81.28:10.47
83.8111884
76.43.10.13
12FNorm+PSKD
76.95=0..33
04.04=0.18
80.50=00.25
87.17=3.19
89.94=1441
77.80=0.444
81.60:+0.31
84.13++0.99
77.14+0.08
Training-ting regularization methods (w/ realistic outliers)
MixOE
78.17
83.7310.12
80,95
76.06
72.28=0.81
77.3410.91
79.92=0.30
76.4011.44
75.13:10.06
DOE
75.47=+0.55
97.76++0.28
86.61+00.22
72.54++4.38
0.42
86.34
0.95
84.30++0.81
10.69
OE
76.70.10.19
99.8910.00
88.30.10.10
80.68.15.88
84.37-1.34
82.18.10.68
78.3910.41
81,4111499
70.84.10.42
OE+UM
76.67=00.48
09.89+0.03
88.28-0.23
81.44+3.24
84.32-2.00
83.16+1.35
78.76+00.8
1.592+1558
76.51+0.37
OE+UMAP
76.95110.16
99.9210.00
88.4410.08
85.2711.68
83.92.3.116
82.1410.68
78.35.10.35
82.4210.87
76.84110.42
OE+PSKD
76.56.1..31
99.9110.02
88.2410.155
82.6410.85
92.38-1.67
83.75.0.80
79.64.10.17
84.60-10.43
76.9910.05
Table 15: Fine-grained results (FPR95J) on the CIFAR-100 benchmark.
Near-OOD Tiny ImagNNe
Far-OOD
Method
ID ACC
CIFAR-10
Average
MNIST
SVHNN
Textures
Places365
Averuge
0OD scoring methods (vanilla training with proxs-entropyy
MSP
08.91+0..3
50.70++0.34
54,80 0.333
57.23++4.68
59.07+2253
61.88+1.22
56.62+0.87
58.70+1.06
77.25-0.10
ODIN
60.64+0.56
55. 0.57
$7.91
45.94:+3.29
67.41+33.88
62.37+22.96
59.71+0.92
58.86 0.78
77.25-0.10
Energy
59.71
52.03:10.50
55.62:10.61
52.62.13.83
53.62.1..14
62.35.12.06
57.75:10.66
$6.59-1.38
7..25-0.10
UMM
14.09
1..8111.004
65. 6..86-12.06
36.56.15.81
39.72.1.886
68. 2..16.
63.0013..26
51.90012.04
7..21-0.43
Energy+UMAN
64.73:10.97
54.70.10.33
59.71
42.80.17.56
40.03:16.52
65 65.02.14.09
60.6111661
52.11-1..36
77.25-0.10
Energy+PSKI
60.73++4.96
48.93+00.3
54.83+2.77
42.91+2.95
44.76++4.73
63.23++2.45
5..34++060
51.56+2.39
77.44-0.09
Truining-time regularization methods (wlo realistic outleers)
LogitNorm
73.88
$1.89.10.10
62.89
34.12.18.32
47.5211.02
77.38:12.99
55,441 1.45 $3.61
76.34_0.17
SNN
71.5612.17
0..78
60.32:1.44
46.31111.23
54.1116.62
54.14:11.00
59.5110.91
13.52-1.77
77.56_0.27
T2FNorm
67.07
49,88 89.88
18.47=1.35
39.39115.88
44.29:3.14
66.82.14.61
54.5010.52
51.25-2.52
76.43=0.13
UM
69.07=0.79
49.7710.68
59.42-0.74
34.6411.554
46.5616.07
67.1411.49
16.0011.26
51.09-0.98
75.67=0.41
T2FNorm+UMAP
69.30=0066
50.25=0060
59.77=0.03
32.48:12222
46.25=15.16
66.8842.55
54.06-0.600
49.92-4.16
76.43=0.13
T2FNorm+PSKD
65.18:0.77
49.33
$7.26
10.19
35.43
6..46.11.46
5.700
49.15 1.67
77.14
Training-time regularization methods (w/ realistic outliers)
MixOE 61.1211.08
49.32-1.366
55.22-10:49
59 39.49
73.0914.00
66.04110.8
56.9310.78
63.88-2.449
75.13-0.06
DOE
00,55
1..8
3..84=1.05
57.97:14.42
20.27=1.01
50.29:12.73
52.97=1.56
46.38-0.52
75.69=0.26
OE
61.26=20.2
0.21=0001
30.73=0.11
53.31:1991
51.84=3.45
55.83:+1.22
58.30+0.72
54.82-2.79
76.84=0.42
OE+UM
65.95
0.27=0.03
30.71=0.46
50.43:+4.09
49.87=3.58
54.79:2.52
58.59+
53.42-2.29
76.51=0.37
OE+UMAP
60.1510.12
0.18
30.16:0008
49.14.11.83
52.76-15.43
55.77
58.62.1.71
14.07-11.21
76.84_0.42
OE+PSKD
6..91=1.113
0.19=0010
31.55
50.30-14.22
34.67+3.87
52.9411.222
56.9511:40
48.72
7.09
17

---

Strengthen cut-of-Distribbttinn Detection Capability with Progressive Self-Knowledge Distillation
Table 16: Fine-grained results (AUROC1) on the ImageNet-200) benchmark.
Near-OOD
Far-OOD
Method
ID ACC
SSB-hard
NINCO
Average
Naturalist
Textures
OpenImage-O
Average
OOD scoring methods
(vanilla training with cruss-entrepy)
MSP
0.03
86.29.1.111 83.34.1.006
92.80:10.25
88.36.1.13
89.24:10.02
90.13:10.99
86.37:10.08
ODIN
77.19=0.06
83,34+0.12
80.27+0.08
94.37+0.41
90.65+0.20
90.11+0.155
91.71+0.19
86.37+00008
Energy
79.83-10.02
85.1710.11
82.5010.005
92.55:10.50
90.79:10.16
89.23:1.226
90.86:10.21
86.37:10.08
Energy+UM
79.15:10:46
84.43:10.29
81.79:1.38
93.7510.56
92.1110.441
89.18:0:46
91.68:0.37
85.01
Unergy+UMAP
77.94+0.42
84.22+0066
81.08+0.39
93.25+0069
93.19++0.16
88.42+0.35
91.62++0.29
86.37+0.08
ENeD
79.8710.29
85.8110.42
82.8410.32
92.97:1.19 91.8010.31
89.40:10.66
91.3911.11
86.79-10.25
Training-time regularization methods (w/o realistic outliers)
LogitNorm
08.19
86.2410.46
82.2110.52
96,40.1.233
92.36:10.24
91.17:0.47
93.3110.155
0..18:00.00
SNN
77.47+00.24
85.18+0.31
81.33+0.19
92.82+00.66
94.55+0.10
89.45+0.19
92.28
86.56+00003
T2FNorm
78.82+00.08
86.63+0.04
82.72+0.05
96.72+0.21
91.97+0.28
91.47+0.21
93.38+0.11
0..52+0021
T2FNorm+UM
79.25100.31
86.5310.11
82.8910.20
96.5211.25
91.8010.26
91.32:10.26
93.2110.11
86.31
12FNorm+UMAP
78.77+00.24
86.27+0.41
82.52+0.32
96.36+0.10
91.65
91.06+0.08
93.02+00.09
86.52+00.11
T2FNorm+PSKD
78.82+00.11
86.83+0.20
82.83+0.12
96.79+0.10
92.28+00.3
91.75++0.08
93.61+0011
86.74+00009
Training-tin regularization methods (w/ realistic outliers)
MixOE
80.15+0..2
84,99+0.28
82.57+0023
90.94+0.17
87.02++0222
87.22++0.14
88.39+0.02
85.73+0009
DOE
80.6410.71
85.81
83.2310.60
90.641:121
1..0411.774
87.05=11.41
88. 88.24
19.87
OE
82.14+0.11
86.89+0.22
84.51+0.21
89.08+0.18
87.33++0.22
88.22+0019
88.21+0.19
85.78++0112
OE+UM
82.57:0.31
87.32:0.118
84.9410.17
89.9610.05
87.5011..21
88.59:10.07
88.68-10.07
85.71
OE+UMAP
82.30=1.116
86.83:10.08
84.56110.08
89.3711.28
87.4410.06
88.371:020
88.39110.18
85.78:0.12
OE+PSKD
82.56=0.12
87.39+0.13
84.98+0.10
89.51+0.10
0..85
88.47+00.07
88.6140.10
0.13
Table 17: Fine-grained results (FPR95.) on the ImageNet-200 benchmark.
Near-OOD
Far-OOD
Method
ID ACC
SSB-hard
NINCO
Average
iNaturalist
Textures
Openlmage-O
Average
OOD scoring methods
(vanilla training with cross-entropy)
MSP
66.00:10.10
43.6511..75 54.82.1.35
26.4811073
44.5810.68
0..2
35.4311.38
86.3710.08
ODIN
73.51+-0.38
60.00=00880
66.76+0.26
22.39++187
12.99+
37.30+
34.23++1005
86.37+00.08
Energy
69.77-10.2
50.70:1.89
60.24110.57
26.4112.29
1.84
36.74::1.14
34.86=t1.30
86.37=0008
Energy+UM
68.91=1.49
51.55:10.82
60.23+1113
24.69:1140
35.46:11.99
3..23+1.30
32.46:11.30
85.01
UMeP
70.33=00.55
51.28=1..55
60.81+0.84
26.23++1.76
31.71
39.46=t1.02
32.47+40.67
86.37-00088
Energy+PSKD
67.8810.33
1..3 57.12:0..3
24.111:093
3..0
35.77:11.19
31.6410.87
86.79:10.25
Training-time regularization methods (w/o realistic outliers)
LogitNorm
68.07:0.77
47.52
1.28
15.1011.26
1..21
29.6 29.62:11088
25.31+0.20
86.10:00.0
SNN
73.11+00.3 46.60+0028
59.85++0466
25.66++0.76
24.814:0.66
33.64
28.0440064
86.56++0.03
T2FNorm
66.41110.26 04.88:1.54
00.65
13.99+40.6
32.39-+1.41
20.38+0.60
25.25
86.52+0021
T2FNorm+UM
65.27=10.88
13.00.5.65
55.20+1.771
14.73+1.12
34.32+1.87
29.89-0.39
26.31+0.91
86.311:.24
T2FNorm+UMAP
66.62:10.55
45.66:10.93
56.14:10.69
15.07
30.69
30.114:.226
26.25+10.30
86.52+10.11
T2FNorm+PSKD
65.97-1.30
44.14.10.63 55.06:10.46
13.5410.08 31.92++1.78
28.84-1042
24.77-10.75
86.74110.99
Training-time regularization methods (w/ realistic outliers)
MixOE
67.9411.43
47.95+0.63
57.95+0.22
29.97+0.20
1..23+:1.21
40.1740.67
40.12.4..66
85.73+10.09
DOE
63.70:0.04
44.5910.99 54.14:0.51
29.1343.75
1..87+1.93
38.81+3.35
37.60+2..9
79.87+3.12
OE
64.20-1045 40.90-1..6
52.55
28.93:10.58
41.82+1109
34.78.10.24
18.16:1..63
85.78-10.12
OD+UM
63.07-1.83 40.71
51.69
27.5911.31
12.1941..64
34.5140.36
34.76:10.56
85.71
OE+UMAP
63.48+0.10
41.20:10.5
52.34+0.22
28.8140.64
41.62+0.85
34.66+0.59
35.03+40.68
85.78+0.12
OE+PSKD
63.57-0.79 40.19-1..55 51.8810.47
27.9710.38
40.70-10.88
33.76.10.29
34.14110.33
86.31
18


Reviews:
[{"review_id": "ctIkpI13Ck", "reviewer": "Reviewer_ioAG", "summary": "This paper proposes Progressive Self-Knowledge Distillation (PSKD), a framework to enhance out-of-distribution (OOD) detection by leveraging uncertainty-embedded targets from a self-selected teacher model. The authors argue that models tend to memorize atypical samples during later training stages, harming OOD detection. PSKD dynamically selects a teacher model from training history using pseudo-outliers and applies multi-level distillation (feature and response) to learn uncertainty knowledge. Experiments on CIFAR and ImageNet benchmarks demonstrate improved OOD detection and in-distribution (ID) classification.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment and helpful feedback. \n\n**Comment 1. Teacher Selection with Realistic Outlier:** \nTo ensure a fair comparison, we intentionally avoid using realistic OOD data as auxiliary information following the setting of [1]. To further explore this, we introduce realistic OOD data for teacher selection to evaluate the performance of PSKD, with the results recorded in Table 1. The results indicate that: (1) both realistic and pseudo-outlier data consistently lead to significant performance improvements; (2) realistic OOD data more accurately reflect the actual OOD distribution, thereby enhancing the robustness of teacher model selection and further improving performance.\n\nTable 1. The impact of different sources of OOD data for teacher selection on ImageNet-200 benchmark. Following the settings of the OpenOOD benchmark [1] for the OOD validation set, OpenImage-O is adopted as the realistic OOD data. The OOD results are averaged over near- and far-OOD groups.\n|Methods|FPR95↓|AUROC↑|ID ACC↑\n|-|-|-|-|\n|Vanilla|47.55 ± 0.76|86.68 ± 0.12|86.37 ± 0.08|\n|PSKD w/ Pseudo|44.38 ± 0.22|87.12 ± 0.15|**86.79 ± 0.25**|\n|PSKD w/ Realistic|**43.93 ± 0.27**|**87.64 ± 0.18**|86.76 ± 0.16|\n\n**Comment 2. Architectural Variants:** Table 2 analyzes the robustness of PSKD across different architectures, including the CNN-based ResNet-18 and the transformer-based Vision Transformer [2] (ViT-B/16). The results indicate that PSKD consistently improves the model's OOD detection performance across different architectures and demonstrates general applicability.\n\nTable 2. The robustness of PSKD across different architectures on the ImageNet-200 benchmark. The Energy score is adopted for OOD scoring and the OOD results are averaged over near- and far-OOD groups.\n|Model|Methods|FPR95↓|AUROC↑|ID ACC↑|\n|-|-|-|-|-|\n|ResNet-18|Vanilla|47.55 ± 0.76|86.68 ± 0.12|86.37 ± 0.08|\n|ResNet-18|PSKD|**44.38 ± 0.22**|**87.12 ± 0.15**|**86.79 ± 0.25**|\n|Vit-B/16|Vanilla|28.80 ± 0.41|93.61 ± 0.19|93.90 ± 0.10|\n|Vit-B/16|PSKD|**27.79 ± 0.50**|**93.87 ± 0.24**|**94.01 ± 0.06**|\n\n**Comment 3. Handling of Non-Image Data:**\nA general and feasible way for handling non-image data (e.g., text, audio) is to induce distribution shifts by adding noise after converting the data into embedding vectors. To verify the effectiveness of this strategy, we conduct an OOD detection task in an audio classification setting based on the well-known Kinetics-Sound dataset [3]. The results presented in Table 3 provide empirical evidence that our PSKD is also suitable for other types of non-image data.\n\nTable 3. The applicability of PSKD to audio data, with the Energy score used for OOD scoring.\n|Methods|FPR95↓|AUROC↑|ID ACC↑|\n|-|-|-|-|\n|Vanilla|76.95 ± 1.64|69.67 ± 0.59|68.06 ± 0.60|\n|PSKD|**72.93 ± 1.84**|**71.04 ± 0.48**|**68.33 ± 0.37**|\n\n**Comment 4. Sensitivity to the Quality of the Validation Set:** \nThe ablation analysis in Table 5 of the original paper examines the impact of pseudo-outlier quality on OOD detection performance. The results reveal the following: (1) even when using simple pseudo-outlier construction strategies (such as Gaussian noise or distortions), there is a notable improvement in OOD detection performance; (2) increasing the diversity of pseudo-outliers (e.g., by incorporating rotations, distortions, and noise) provides a marginal but further performance boost. Additionally, as shown in Table 1, high-quality, realistic OOD data results in a slight yet consistent improvement in performance. Overall, performance remains relatively stable across different data quality levels, with high-quality validation sets generally yielding the most significant performance gains.\n\n**Comment 5. Computational Overhead of Teacher Selection:** \nTable 4 records the overhead of teacher selection in PSKD on both small-scale and large-scale datasets. The results show that the overhead of the teacher selection process accounts for only a small fraction of the total training cost, which is affordable. Notably, for large-scale datasets, the additional overhead of PSKD is even less significant due to the relatively higher ratio of training samples to validation samples.\n\nTable 4. Analysis of teacher selection overhead in PSKD. The results are averaged over five independent runs. Software and hardware configurations remain consistent with those detailed in Appendix B.1 of our paper.\n|Dataset|One-Epoch Training Cost|One-Time Teacher Selection Cost|\n|-|-|-|\n|CIFAR-10 (Small scale)|10.50 seconds|1.18 seconds|\n|ImageNet-200 (Large scale)|252.30 seconds|8.72 seconds|\n\n**References:**\n\n[1] Yang, et al. OpenOOD: Benchmarking Generalized Out-of-Distribution Detection. NeurIPS 2022.\n\n[2] Dosovitskiy, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021.\n\n[3] Arandjelovic, et al. Look, listen and learn. ICCV 2017.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence including methodology, comprehensive experiments.", "methods_and_evaluation_criteria": "Yes, PSKD makes sense for the OOD detection problem.", "theoretical_claims": "This paper does not include theoretical analysis.", "experimental_designs_or_analyses": "Yes, the soundness/validity of experimental designs and analysis is checked.", "relation_to_broader_scientific_literature": "Yes, the method proposed in the paper is highly inspiring to the field of OOD detection.", "essential_references_not_discussed": "No, the references of the paper are comprehensive.", "other_strengths_and_weaknesses": "Strengths:\n1. Dynamic Teacher Selection: The AUROC-based criterion (Eq. 6) for selecting teachers is intuitive and aligns with OOD detection objectives.\n2. Strong Empirical Validation: Comprehensive experiments across near- and far-OOD scenarios demonstrate PSKD’s superiority over existing methods, e.g., reducing FPR95 by 31.26% on CIFAR-100.\n3. Practical Plug-and-Play Design: PSKD’s compatibility with existing OOD scoring methods (e.g., Energy, MSP) and training paradigms (e.g., OE) enhances its applicability.\n4. Reproducibility: Open-sourced code and hyperparameter details (Appendix B) ensure transparency.\nWeaknesses:\n1. Pseudo-Outlier Dependency: Performance hinges on artificially generated pseudo-outliers (Table 5); robustness to realistic OOD data remains understudied.\n2. Overlooked Architectural Variants: Evaluations are limited to ResNet-18; performance on transformers or attention-based models is unexplored.", "other_comments_or_suggestions": "How does PSKD handle distribution shifts in high-dimensional or non-image data (e.g., text or audio)?", "questions_for_authors": "Could the teacher selection mechanism be sensitive to the quality of pseudo-outliers? How robust is PSKD to noisy validation data?\nWhat is the computational overhead of the teacher selection process, especially for large-scale training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VsDLcumbuC", "reviewer": "Reviewer_nFr5", "summary": "This paper concerns the out-of-distribution (OOD) detection task. Recent work shows that memorizing atypical samples during later stages of training can hurt OOD detection, while strategies for forgetting them show promising improvements. However, directly forgetting atypical samples sacrifices ID generalization and limits the model's OOD detection capability. To address this issue, this paper proposes Progressive Self-Knowledge Distillation (PSKD), which strengthens the OOD detection capability by leveraging self-provided uncertainty-embedded targets. Specifically, PSKD adaptively selects a self-teacher model from the training history using pseudo-outliers, facilitating the learning of uncertainty knowledge via multi-level distillation applied to features and responses. Moreover, PSKD is orthogonal to most existing methods and can be integrated as a plugin to collaborate with them. Extensive experiments including main comparison and ablation study verify the effectiveness of PSKD.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment and helpful feedback.\n\n**Comment 1. Analysis of Performance Disparities:** \nThe performance improvement of PSKD is inherently dependent on the model's intrinsic OOD capability relative to the training data. Compared to the small-scale CIFAR dataset, the large-scale ImageNet dataset encompasses a larger and more complex semantic space. Under the same ResNet-18 architecture used in our paper, models trained on the more challenging ImageNet dataset tend to learn a more crowded feature space, increased class overlap, and unreliable decision boundaries. These factors contribute to the model's relatively weak intrinsic OOD detection capability [1], limiting PSKD's potential to restore the model's intrinsic OOD detection capability. Consequently, the performance improvements on ImageNet are less pronounced, while substantial gains are observed on the simpler CIFAR-10 benchmark. Although the improvements vary across datasets of different scales, PSKD consistently enhances the model's OOD detection performance, demonstrating its effectiveness.\n\n**Comment 2. Insight Justification of PSKD:** \nOur motivation stems from the observation that the model's OOD detection performs optimally at an intermediate stage of training rather than in the final well-trained state. We attribute this to the issue of label assignment in traditional supervised learning, where samples with varying levels of uncertainty are assigned a uniform, absolutely confident learning target (i.e., one-hot labels). This forces the model to learn overly confident predictions for atypical samples, leading to an overestimation of certainty. Consequently, the model becomes prone to overconfidence regarding OOD data, which hurts OOD detection. \n\nTo address this problem, our PSKD leverages the potential within the model's own learning process to generate soft targets that inherently capture sample-level uncertainty. Specifically, PSKD alleviates overfitting to atypical samples by learning from the uncertainty-embedded targets provided by the self-selected teacher model, thereby effectively enhancing the model's ability to perceive uncertainty. As a result, PSKD can effectively learn from atypical samples to restore the model's intrinsic OOD detection capability. Further empirical analysis and interpretative insights regarding PSKD can be found in Section 4.4 of the original paper.\n\n**Reference:**\n\n[1] Huang, et al. MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space. CVPR 2021.", "claims_and_evidence": "Clear. The main claims have been supported from methodology and comprehensive experiments.", "methods_and_evaluation_criteria": "Make sense.", "theoretical_claims": "This paper does not include theoretical analysis.", "experimental_designs_or_analyses": "The experimental designs or analyses make sense and are validate", "relation_to_broader_scientific_literature": "The key contributions of the paper are related to the OOD detection, which is an interesting research issue.", "essential_references_not_discussed": "No, the author provides extensive references on OOD detection.", "other_strengths_and_weaknesses": "Strengths:\nNovelty: The proposed PSKD framework is innovative and can effectively solve the problem of overfitting atypical samples in the late training period, thereby improving the OOD detection capability.\nEffectiveness: The experimental design was comprehensive, covering multiple datasets and baseline methods, verifying the effectiveness and generality of PSKD. The authors have provided code to enhance the reproducibility of the paper.\nWriting and Paper Organization: The writing of the paper is clear, with well-structured arguments and a logical flow that enhances readability and comprehension. And the experimental results makes the paper more convincing.\nCons:\nPerformance: Although PSKD performed well on multiple datasets, performance improvements on some specific datasets were not significant and may require further analysis for reasons.\nDiscussion: There is a lack of in-depth discussion on why PSKD can effectively improve OOD detection capability.", "other_comments_or_suggestions": "It is suggested that the author add the analysis of PSKD performance differences on different datasets in the experimental part, especially those data sets with insignificant performance improvement.", "questions_for_authors": "The performance of PSKD varies greatly on different data sets. Are there some data set characteristics that affect the performance of PSKD? Can you further analyze the reasons for these differences?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c92Rk3Gu0y", "reviewer": "Reviewer_MqyY", "summary": "This paper proposes Progressive Self-Knowledge Distillation (PSKD), a framework that leverages self-distillation and dynamic teacher selection to enhance a model’s intrinsic OOD detection capability. PSKD uses pseudo-outlier samples generated through rotation, distortion, and Gaussian noise to iteratively refine the student model’s OOD performance by distilling knowledge from a dynamically updated teacher model. Extensive experiments demonstrate the superiority of the PSKD, including (1) main comparison with SOTA OOD detection baselines, (2) ablation study for adjustment strategy, weighting factor, and temperature scaling, (3) other deep analysis. The paper is well-written and easy to follow.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment and helpful feedback. \n\n**Comment 1. Analysis of Training Stability:** \nTables 6-11 in Appendix C of the original paper report the performance of PSKD across multiple independent training runs on various benchmarks. The results show that PSKD exhibits a standard deviation comparable to that of Vanilla training, suggesting that the introduction of PSKD has a negligible effect on the stability of the training process. Below, we provide an analysis of the effects of teacher update frequency and λ adjustment on training stability:\n\n- **Teacher Update Frequency:** Table 1 examines the effect of teacher model update intervals on training stability. The results indicate that PSKD is robust to update frequency, with frequent updates having a limited effect on training stability. Moreover, appropriately increasing the update frequency may help improve OOD detection performance by providing more opportunities for the model to explore its optimal intrinsic OOD detection capability. \n\n- **λ adjustments:** The purpose of adjusting λ is to minimize the influence of self-distillation in the early stages of training. This helps prevent the large bias introduced by the teacher model, which may not have been adequately trained, from destabilizing the training process. As training progresses, λ gradually increases to emphasize the objective of learning the uncertainty-embedded targets. As analyzed in Figure 5(c) of the original paper, failing to adjust λ dynamically may compromise training stability due to excessive interference from an undertrained teacher model.\n\n\nTable 1. The impact of varying teacher selection intervals on training stability on CIFAR-10 benchmark. The OOD results are averaged over near- and far-OOD groups on the CIFAR-10 benchmark.\n|Teacher Selection Interval|FPR95↓|AUROC↑|ID ACC↑\n|-|-|-|-|\n|10 selections per 1 epoch|26.25 ± 1.46|93.11 ± 0.22|95.08 ± 0.22|\n|5 selections per 1 epoch|**25.79 ± 2.17**|**93.29 ± 0.30**|**95.30 ± 0.07**|\n|1 selection per 1 epoch (default)|26.08 ± 1.02|93.14 ± 0.21|95.14 ± 0.08|\n|1 selection per 5 epochs|26.86 ± 0.91|92.96 ± 0.37|95.14 ± 0.17|\n|Single selection throughout training (PSKD-S)|28.18 ± 3.29|92.82 ± 0.68|95.06 ± 0.05|\n\n\n**Comment 2. Analysis of Computational Overhead:**  \nTable 2 presents the additional training overhead introduced by PSKD on both small-scale and large-scale datasets. The results indicate that: (1) the overhead of the teacher selection process accounts for only a small fraction of the total training cost, which is affordable. Notably, for large-scale datasets, the additional overhead of PSKD is even less significant due to the relatively higher ratio of training samples to validation samples. (2) The generation of pseudo-outliers is conducted only once at the beginning of training, and its computational cost is minimal compared to the overall training time.\n\nTable 2. Overhead analysis introduced by PSKD. The training setup follows the OpenOOD benchmark [1], consisting of a total of 100 epochs for CIFAR-10 and 90 epochs for ImageNet-200. The results are averaged over five independent runs. Software and hardware configurations remain consistent with those detailed in Appendix B.1 of our paper.\n|Dataset|One-Epoch Training Cost|One-Time Teacher Selection Cost|Total Training Cost|Pseudo-Outlier Preprocessing|\n|-|-|-|-|-|\n|CIFAR-10 (Small scale)|10.50 seconds|1.18 seconds|21.96 minutes|0.96 minutes|\n|ImageNet-200 (Large scale)|252.30 seconds|8.72 seconds|397.49 minutes|1.42 minutes|\n\n**Reference:**\n\n[1] Yang, et al. OpenOOD: Benchmarking Generalized Out-of-Distribution Detection. NeurIPS 2022.", "claims_and_evidence": "Yes. The key claims made in the paper are well-supported by relevant methodological and experimental evidence. The authors thoroughly explain their approach, ensuring that each assertion is backed by rigorous theoretical analysis and empirical validation. The experimental results are comprehensive and demonstrate the effectiveness of the proposed method across different scenarios, further strengthening the credibility of the findings.", "methods_and_evaluation_criteria": "Yes. The method proposed by the author addresses OOD problems and effectively tackles related challenges in practical applications.", "theoretical_claims": "N/A. The paper does not include any theoretical claims.", "experimental_designs_or_analyses": "Yes. I have evaluated the rationality and effectiveness of the experimental design in the paper. The experiments are well-designed, and the author has provided the relevant code. Based on the experimental setup and results, I think relevant results are reasonable and convincing.", "relation_to_broader_scientific_literature": "OOD detection is a crucial area in machine learning. This paper addresses the limitation of directly forgetting atypical samples in previous studies.", "essential_references_not_discussed": "Key references are cited in this paper. The author provides a comprehensive and well-structured discussion of related work.", "other_strengths_and_weaknesses": "Strengths:\n1. Novelty: The authors design a novel strategy to adaptively select a self-teacher model. PSKD integrates self-distillation with dynamic teacher selection, addressing the “forgetting” problem in traditional regularization methods.\n2. Simplicity and Efficiency: Avoids complex architectures or additional data, relying solely on pseudo-outliers and parameter tuning.\n3. Comprehensive Experiments: The authors provide comprehensive experiments to verify the superiority of the proposed method. Furthermore, the authors also report the ablation study, deeply analysis experimental results to further reveal the justifiability and effectiveness.\n4. Clearly Writing: This paper is well-written and easy to follow. The logic is clear, and the structure is well-organized, allowing readers to grasp the main ideas efficiently.\nWeaknesses:\n1. Training Instability: Frequent teacher updating and adaptive $\\lambda$ adjustments might be leading to training instability in practice.\n2. Computation Overhead: The PSKD requires frequent teacher updates and pseudo-outlier generation, potentially increasing training time.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Please analyze the training instability problem.\n2. Please analyze the training computation overhead problem.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G4YTniJIAC", "reviewer": "Reviewer_JJ2e", "summary": "To address the issue of suboptimal OOD detection performance during the later stages of training, this paper proposes Progressive Self-Knowledge Distillation (PSKD) framework. PSKD strengthens the OOD detection capability by leveraging self-provided uncertainty embedded targets. PSKD is orthogonal to most existing methods and thus can be used to further enhance the effectiveness of other OOD detection methods.\n\n## update after rebuttal\nAfter reading the authors' responses, I decide to keep my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment and helpful feedback.\n\n**Comment 1. Statistical Tests for Comparison:** \nThank you for your valuable suggestion. The discussion on statistical tests for result comparison will be included in the final version. Here, we report the statistical test results using AUC as the OOD detection metric on the CIFAR-10 benchmark. First, we rank the performance of all methods across multiple datasets, with the results recorded in Table 1. We then perform the Friedman test to determine whether there is a significant difference in the average rankings of the methods. The Friedman test yields a test statistic of 25.524 and a p-value of 0.00011 (which is less than the significance level of 0.05), indicating a statistically significant difference among the methods. To further identify specific methods with significant performance differences, we conduct a Nemenyi test, obtaining a Critical Difference (CD) value of 3.078, and present the comparative analysis of ranking differences between our PSKD and other methods in Table 2.\n\nThe statistical test results indicate that (1) compared to Energy, Energy+PSKD effectively enhances the model's OOD detection capability and demonstrates a significant performance improvement; and (2) the lack of a significant difference between our PSKD and Unleashing Mask/Unleashing Mask Adopts Pruning (UM/UMAP) can be attributed to our shared goal of restoring the model’s intrinsic OOD detection capability. The distinction lies in our method, which utilizes an uncertainty-embedded target to learn valuable atypical samples, whereas UM/UMAP directly discards them, resulting in a loss of ID generalization performance and limiting the model's OOD detection capability.\n\nTable 1. Average ranking results of methods across multiple datasets.\n||MSP|ODIN|Energy|Energy+UM|Energy+UMAP|Energy+PSKD(Ours)|\n|-|-|-|-|-|-|-|\n|Average Rank|4.67|5.67|4.67|2.67|1.83|**1.50**|\n\nTable 2. Ranking differences between our PSKD method and other comparison methods, with an asterisk (*) indicating a significant difference, where the value exceeds the CD value of 3.078.\n||MSP|ODIN|Energy|Energy+UM|Energy+UMAP|\n|-|-|-|-|-|-|\n|Ranking Difference|3.167*|4.167*|3.167*|1.167|0.333|\n\n**Comment 2. Clarity and Consistency:** \nThank you for pointing out these mistakes, and we will revise the manuscript accordingly. (1) The full name will be provided upon the first occurrence of any abbreviation to enhance clarity. (2) Special attention will be given to word choice to ensure consistency throughout the manuscript.\n\n**Comment 3. Why Use AUROC Instead of AUC?** \nThank you for your correction. The abbreviation \"AUROC\" (Area Under the Receiver Operating Characteristic Curve) used in our paper follows the OpenOOD benchmark [1]. Indeed, the more commonly used abbreviation is \"AUC\". We will make this revision in the final version.\n\n**Reference:**\n\n[1] Yang, et al. OpenOOD: Benchmarking Generalized Out-of-Distribution Detection. NeurIPS 2022.", "claims_and_evidence": "Yes, all claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, both proposed methods and evaluation criteria make sense for the problem and application.", "theoretical_claims": "Yes, both proofs for the theoretical claims are correct.", "experimental_designs_or_analyses": "Yes, the experimental design and analyses in this paper are solid, with a clear comparison to state-of-the-art methods.", "relation_to_broader_scientific_literature": "Out-of-distribution (OOD) detection aims to ensure AI system reliability by rejecting inputs outside the training distribution. This paper proposes Progressive Self-Knowledge Distillation (PSKD) framework to improve OOD detection performance and further enhance the effectiveness of other OOD detection methods.", "essential_references_not_discussed": "No, there are no essential references not discussed.", "other_strengths_and_weaknesses": "Strengths:\n1) To address the issue of suboptimal OOD detection performance during the later stages of training, this paper proposes Progressive Self-Knowledge Distillation (PSKD) framework. PSKD strengthens the OOD detection capability by leveraging self-provided uncertaintyembedded targets.\n2) The authors conduct extensive experiments to verify the effectiveness of the proposed PSKD on both small-scale CIFAR and large-scale ImageNet, covering near-OOD scenarios with semantic shifts and far-OOD scenarios with further obvious covariance shifts.\n3) The paper is generally a good paper with a clear central idea. The organization of the paper is quite good and it is easy to follow the topic and the proposed algorithms.\n\nWeaknesses:\n1) With regard to the comparison results, statistical tests are needed in the comparison results. The detailed description about statistical tests for comparisons of multiple algorithms on multiple datasets can be found from the following paper: Statistical comparisons of classifiers over multiple data sets.\n2) In the current version, the authors use the words \"approach\", \"method\", “technique”, “framework” and \"strategy\" a little casually. I know, it is very difficult to distinguish these words thoroughly. At least, the author should try to unify the use of these words in a paper. \n3) In the paper, the authors use AUROC to denote the area under the receiver operating characteristic curve. Why? To my knowledge, the area under the receiver operating characteristic curve is widely denoted as AUC. It is not necessary to change the well-known abbreviation.", "other_comments_or_suggestions": "Please give the full names for all abbreviations when they occur for the first time. For example, UM/UMAP. I have not any other comments or suggestions.", "questions_for_authors": "1) With regard to the comparison results, statistical tests are needed in the comparison results. The detailed description about statistical tests for comparisons of multiple algorithms on multiple datasets can be found from the following paper: Statistical comparisons of classifiers over multiple data sets.\n2) In the current version, the authors use the words \"approach\", \"method\", “technique”, “framework” and \"strategy\" a little casually. I know, it is very difficult to distinguish these words thoroughly. At least, the author should try to unify the use of these words in a paper. \n3) In the paper, the authors use AUROC to denote the area under the receiver operating characteristic curve. Why? To my knowledge, the area under the receiver operating characteristic curve is widely denoted as AUC. It is not necessary to change the well-known abbreviation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel and practical PSKD framework that enhances OOD detection while preserving ID generalization. The approach is novel and practical, with good potential for integration into existing methods. I recommend accept.
Decision: Accept (poster)