Kernel PCA for Out-of-Distribution Detection
Authors: Kun Fang
Qinghua Tao
Kexin Lv
Mingzhen He
Xiaolin Huang
JIE YANG
Venue: NeurIPS 2024 poster (2024)
Keywords: out-of-distribution detection
kernel principal component analysis
random Fourier features
Primary area: safety_in_machine_learning

Abstract: Out-of-Distribution (OoD) detection is vital for the reliability of Deep Neural Networks (DNNs).
Existing works have shown the insufficiency of Principal Component Analysis (PCA) straightforwardly applied on the features of DNNs in detecting OoD data from In-Distribution (InD) data.
The failure of PCA suggests that the network features residing in OoD and InD are not well separated by simply proceeding in a linear subspace, which instead can be resolved through proper non-linear mappings.
In this work, we leverage the framework of Kernel PCA (KPCA) for OoD detection, and seek suitable non-linear kernels that advocate the separability between InD and OoD data in the subspace spanned by the principal components.
Besides, explicit feature mappings induced from the devoted task-specific kernels are adopted so that the KPCA reconstruction error for new test samples can be efficiently obtained with large-scale data.
Extensive theoretical and empirical results on multiple OoD data sets and network structures verify the superiority of our KPCA detector in  efficiency and efficacy with state-of-the-art detection performance.

Kernel PCA for Out-of-Distribution Detection
Kun Fang'
Qinghua Tao?
Kexin Ly
Mingzhen He'
Xiaolin Huang'
Jie Yang'
Shanghai Jiao Tong University "ESAT-STADIUS, KU Leuven
*China Mobile Shanghai ICT Co., Ltd
(fanghenshao, nicchhao,mingge mingzhen he deeexiialinnnang hataaoiinhannniieaaageeeeee eduu.edu.cco qinghua. taoθesat kuleuven. be 1vkexin@cmsr, chinamobile com
Abstract
Out-of-Distribution (OoD) detection is vital for the reliability Deep Neural Networks (DNNs). Existing works have shown the insufficiency of Principal Component Analysis (PCA) straightforwardly applied on the features of DNNs detecting OoD data from In- D--Ditribbttinn (InD) data. The failure of PCA suggests that the network features residing in OoD and InD are not well separated by simply proceeding in linear subspace, which instead non-linear mappings. In this work, we leverage the framework of Kernel PCA (KPCA) for OoD detection, and seek suitable non-linear kernels that advocate the separability between InD and OoD data the subspace spanned by the principal components. Besides, explicit feature mappings induced from the devoted task- specific kernels are adopted so that the KPCA reconstruction error for new test samples can be efficiently obtained with large-scale data. Extensive theoretical and empirical results on multiple OoD data sets and network structures verify the superiority our KPCA detector in efficiency and efficacy with state-of-the-un detection performance.
be resolved through proper
Introduction
With the rapid advancement of the powerful learning abilities of Deep Neural Networks (DNNs) (12), the trustworthiness of DNNs in security-sensitivv scenarios has attracted considerable attention in recent years 33. Generally, samples from the training and test sets of DNNs are viewed as data from some In Distribution (InD) Fin, while samples from other data sets are regarded as coming from a different distribution P Pout. i.e., out-of di-sttiibution (OoD) data. In the practical deployment, DNNs trained on InD data would inevitably encounter OoD data and thus yield unreliable results with potential risks. Therefore, detecting whether a new sample is from Pi, or P has been valuable research topic of trustworthy deep learning, namely OoD detection
Existing OoD detection methods exploit different outputs from DNNs to unveil the hidden disparities between InD and OoD data, e.g., logits (5). gradients 6 and features g. In this work, we address OoD detection from a perspective of utilizing the feature spaces learned by the backbone of DNNs. To be specific, given a DNN I > IR. takes x E Rd as inputs and learns penultimate layer features R before the last linear layer. Principal Component Analysis (PCA) is investigated in [ calculate the reconstruction eиror in the s-space as the OoD detection score. That is. PCA is executed on the penultimate features of InD training samples and learns linear subspace spanned by the principal components the z-space. Then, given features of an unknown sample á, one can obtain the reconstructed counterpart of 2 by projecting & to the linear subspace and re-projecting it back. The reconstruction error is computed as the Euclidean distance between & and the reconstructed counterpart. In [9], similar ideas adopted with energy-base models using
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

CIFAR10
C PAR10
LSUN
LSUN
ISUN
ISUN
(CIFAR1O) D. SUNI OoD (SUN)
(CFARRO)
(USUN)
Oo (ISUN)
(a) T-SNE of z and PCA reconstruction errors.
(b) T-SNE of ((2) and KPCA reconstruction errors.
Figure 1: The t-SNE [10] visualization on the original features (left) and the mapped features ((2) (right). Our KPCA detector alleviates the linearly inseparability between InD and OoD features in the original z-space via explicit feature mapping 4, and thus substantially improves the Ool detection performance, illustrated by the much more distinguishable reconstruction errors.
auto-encoder structure, where the neural networks are trained from scratch and the reconstruction is conducted in the decoder. For good OoD detection performance, it is expected that InD features are compactly allocated along the linear principal components with high variances for capturing most of the informative patterns of Inl data, leading to small reconstruction errors, while OoD features are not supposed be well matched with the learned subspace, causing large reconstruction errors.
However, has been empirically observed in 8 that such PCA reconstruction errors alone cannot distinctively differentiate OoD data from InD data, leading poor detection performance of PCA in the z-space. Nevertheless, S did not take further explorations on the reasons behind, and instead proposed practical fusion trick boost PCA by multiplying with other existing powerful detection scores. Therefore, in this work, more in-depth analyses are undertaken to improve the limitations of PCA for OoD detection with insightful understandings on the distribution of InD and OoD features. It is widely acknowledged that PCA falls inferior in dealing with those mmecaly-inseparabee data, which inspires us explore the non-linearity existing in the z-space of InD and OoD features under the help of the celebrated Kernel PCA (KPCA).
KPCA has long been powerful technique in learning the non-linear patterns of data 12 By deploying KPCA, a non-linear feature mapping & is imposed on the z-space in this setup, so that the linear inseparability can be alleviated in the mapped (xz)-space. KPCA is generally conducted through a kernel function & induced by the feature mapping, i.e., k( z2) = (0(21) (22))) )), in order avoid explicit calculations in the mapped (b2) (x))-space. prior on the unknown non-linear data distribution, e.g., the non-linearity in the z-space of InD and OoD features. Therefore, finding an appropriate &o or that well adapts the data always remains non-trivial issue for KPCA. For example, the mostly common Gaussian kernel is shown to be unable to separate the InD and OoD features, and leads to terrible OoD detection performance, see details in Appendix C In addition, KPCA also faces the challenge of calculating and storing the Nir x Nur kernel matrix on millions of training samples with very large size N which significantly hinders its application in practical tasks with a huge amount of data.
most cases, researchers have no
The proposed KPCA detection method well addresses the aforementioned issues of KPCA for OoD detection. On the one hand, to better understand the non-linear patterns in InD and OoD features, we take kernel perspective on an existing OoD detector 77). which searches k-th Nearest Neighbors (KNN) on the l normalized features By decoupling and analyzing key components of the KNN method, we acquire two effective kernels, a cosine kernel and a Gaasssan kernel, for our KPCA detector to promote the linear separability between InD and OoD features in the subspace of the principal components, leading to substantially improved distinguishable KPCA reconstruction errors, as shown in Figure [ On the other hand, for computationally -friiittiooallyyrennllly implementation, two explicit feature mappings induced from the cosine and Posine-Gaussine kernels are executed on the original features followed by PCA in the mupped (<z)-space to obtain the reconstruction errors without calculations on the kernel matrix. Specifically, the celebrated Random Fourier Features (RFFs) [ are introduced to approximate the Gaussian kernel, allowing an O(M) computation
2

---

complexity in inference, which significantly outperforms the (Nri)) computation complexity of the and the kemel-maarix- KPCA (M the number of RFFs and M N
KNN
Extensive experiments verify the effectiveness of the devoted two kernels, which we hope could bring inspirations for the research community in exploring the non-linearity in InD and OoD data from kernel perspective. For example, the two kernels can even serve as a beneficial prior on advocating learning more and stronger kernels for OoD detection. In addition, we supplement our method with its implementation via the kernel matrix, and illustrate the advantageous effectiveness and efficiency The contributions of this work are summarized below:
of explicit feature mappings in Section 6
To the best of our knowledge, this is the first work that explores suitable kernels seize the non-linearity in InD and OoD features in the post-hoe stage on well-trained DNNs.
Two task-specific kernels are carefully devised for OoD detection. Particularly, two explicit feature mappings induced from the kernels are adopted for the KPCA detector, and lead to separable KPCA reconstruction errors with significantly-rdducee
complexity in inference.
Theoretical and experimental comparisons indicate the effects of kernels in our KPCA detector with SOTA detection performance and remarkably reduced time complexity.
In the remainder of this work, related works and research backgrounds outlined in Section 2 and Section [ respectively. Section delves into details of the proposed KPCA detector. Comparison experiments with prevailing Ool detection methods and KPCA via kernel functions presented in Section and Section 6 respectively. Conclusions and limitations are drawn in Section 7
Related work
Generally, out-of-distribution detection has been formulated as binary classification problem including decision function (.)) and scoring function S(-):
InD, S(x) > 8, D(x) { OoD, S(x) <
(I)
The scoring function S(-) assigns a score S())) for new sample If ((t) is greater than threshold the decision function D.)) would view & as an InD sample, and vice versa. The key to effectually detecting OoD samples a well-designed scoring function. Existing OoD detectors adopt different outputs from DNNs and design justified scores to measure the disparity between InD and OoD data.
Logits-based detectors exploit the abnormal responses reflected in the predictive logits or probabilities from DNNs to detect OoD data. Typical methods adopt either the maximum logits [4] or probability or the energy function on logits [5] the detection score.
15 16
Gradients-based methods investigate differences on gradients w.r. InD and OoD data for Ool)
detection. For example, gradient norms 6 or low-dimensional representations 17 studied to devise the detection score.
Features-based detectors try to capture the feature information causing over-confidence of OoD predictions in different ways. Feature clipping [88
20
21 22), feature distances [23 7 24
feature norms (25), rank-1 features (26). feature subspace 3 etc., have been explored with excellent
performance.
Aside from methods above, other existing OoD detectors cover the training regularization 27 28
the ensemble technique [29 and theoretical understandings 30 31,. Refer to Appendix for more
detailed descriptions on the compared methods in experiments 32
34 35
Background
3.1 PCA for OoD detection
The PCA detector with the reconstruction error as the detection score summarized firstly. Given the penultimate features 2; 1 learned by well-trained DNN f _ - R of the InD training
3

---

data 2; E Nu. the covariance matrix Σ is calculated as:
Σ
(2) 11) (zi 14)
Z
Through the eigendecompasition Σ UAU the dimensionality reduction matrix U, Rmxq is obtained by taking the first q columns of the eigenvector matrix w.r. the top-q largest eigenvalues.
In inference, given new sample å E Rd and its feature 2 R from the DNN f. the reconstruction error computed as:
e( = U 14) - (8 15-14) (1)
By projecting centralized (# (1) to the -sususae and re-projecting back, we can obtaín the (4) and the reconstruction error e(ã), which then can be set as e An ideal case is that U, contains informative principal the OoD detection score: S()
reconstructed features U, U,U, q
components of InD data and causes projections of OoD data far away from that of InD data, leading to separable reconstruction errors between OoD and InD data.
3.2 Random Fourier features
A concise description is firstly given on the Random Fourier Features (RFFs) 13] which will be adopted in our method later. RFFs are proposed to approximate the kernel function so as to alleviate the heavy computation cost in large-scale kernel machines. In kernel methods, N N kernel matrix w.r. N samples requires ((N2) kernel manipulations, ((N2) space complexity and O(N3) time complexity to calculate the inverse of the kernel matrix, which leads overwhelmed computation costs for a large data size N. Therefore, RFFs are introduced by building an explicit feature mapping directly approximate the kernel function for efficient kernel machines large-scale data.
RFFs are built on the Bochner's ss theorem [6): A continuous and shift-invariant kernel k(21, z2) k(21 z2) on Rn is positive definite if and only if k(-) is the Fourier transform of a non-negative measure. An explicit feature mapping ΦRFF induced from the kernel is derived in [3]
2 V M (z), Øar(2)), (x(2) = cos (2 T wi + us),
ΦRFF (2)
M,
(4)
where wI, WM RT are i.i.d. sampled from the Fourier transform of k(-), and w1,... UM I are i.i.d. sampled from uniform distribution (((0, 2T). For example, the sampling distribution for w; of Gaussian kernel function kgau = = is NN m2) Such feature mapping ORFF satisfies Kgau (=1, 22) R ÓRFF(E ( ΦRFF(=2) and is known the random Fourier features (RFFs) mapping. Refer to [3] for a detailed convergence analysis. RFFs have been widely utilized
in kernel learning (37), optimization (38), etc.
Methodology
As empirically observed in H the aforementioned PCA reconstruction error in the z-space is not effective score in detecting OoD data from Inl data. We propose that the reason behind is possibly due the linearly-inseprrahe this issue, we propose to explore the non-lincarity in z-space via kernel PCA. Then, through kernel perspective on an existing KNN detector 77. we put forward characterize the non-linear patterns in z-space of InD and OoD data: a cosine kernel (Section 4.T) and a Gaussiann kernel (Section 4.2). Particularly, we adopt two explicit feature mappings induced from the two kernels, and execute PCA in the mapped ((z)-space, which leads to an informative principal subspace and distinct reconstruction errors for efficacious OoD detection.
features of InD and OoD data, shown in Figure Ta To address
efficacious kernels that well
4.1
Cosine kernel
In the KNN detector 77 the nearest neighbor searching is executed on the ey-normalized penultimate features, i.e., - In inference, given new sample å its feature å is firstly normalized as 2

---

then the negative of its (k-th) shortest &g distance to the fz-normalized features is set as the detection score:
21_ of training data
||::|1
2
2;
Skna (2) =
min
(5)
a: Ner
||
|zi
The ablations in KNN demonstrate the indispensable significance of the fz-normalization: the nearest neighbor searching directly on & shows notably drop in detection performance. The critical role of the fz-normalization in KNN attracts our attention in the sense of kernel. From a kernel perspective, the fg-normalization is exactly the non-linear feature mapping Pcos inducing the cosine kernel kcus:
2 22
Koos (21,
|| | = 2o0a(21) (2o)) (cos(2)
(6)
||
It indicates that justified (cos(2) (=)-space with such non-linear mapping, instead of the original z-space, contributes to the success of nearest neighbor searching in detecting OoD.
Notice that the key of KPCA for OoD detection lies in an appropriate non-linear feature space that captures the non-linearity in InD and OoD features, either through the kernel or the associated explicit feature mapping + Motivated by the KNN detector, we apply Φf-) - dews(.) as the feature mapping in KPCA to introduce non-linearity. Then, PCA is executed on mapped features Φcos (z), following the procedures described in Section 3.1 All the features z are now mapped to ((z) to formulate the covariance matrix ΣΦ, for computing non-linear principal components with matrix U and the corresponding reconstruction error e This detection scheme is dubbed as CoP (Cosine mapping followed by PCA), shown in Algorithm| [ normalization the cosine kernel is left Appendix C.I
An in-depth analysis on the effect of the
4.2 Cosine-Gaussian kernel
The success of KNN (Equation (5) suggests that the lz distance on | is effective distinguishing OoD data from InD data. In other words, the &g distance relation between samples in the ~pace preserves useful information that benefits the separation of OoD data from InD data. This motivates us to seek non- linear feature spaces that can retain the lg distance relation. Hence, we propose to introduce KPCA with non-linearity built upon &eos (2), through which the useful Eg distance in Ócos-Space can be preserved to further separate InD and OoD data.
In this regard, we deploy the shift-invariant Gaussian kernel keep the distance information:
Kgau (21, x2) = - -->|==-==
The feature mapping associated with kgau is dimeiit-ldiessional, but it can be efficiently approxi- mated through random Fourier features (RFFs, [3)), i.e., ÓRFF defined in Equation 4 In this way,
the inner product two mapped samples ΦurF (z1) @xFF(=z) provides the approximate Gaussian kernel, so that through the Gaussian kernel.
leverage the RFFs mapping Φnrr to preserve the f distance information
Hence, cosine-Gaussian kernel is adopted for OoD detection, the Gaussian kernel Kgau (or ÓRFF) imposed on top of the cosine kernel keas (or doos), further exploiting the E distance relationships beyond the Space for OoD detection. As we work with the explicit feature mapping, the non- linearity to is achieved by +t.) ΦRFF ( (f)cs(.)). With mappings ΦRFF @ (z)). PCA is then exccuted to compute the reconstruction eиrors for OoD detection. This detection scheme is dubbed as CoRP (Cosine and RFFs mappings followed by PCA). Algorithm [ illustrates the complete procedure the proposed CoP and CoRP for OoD detection. Alternative choices for more kernels
exploited in Appendix
To warp up, we devise two effective feature mappings induced from a cosine kernel and a cosine- Gaussian kernel to promote the separability of InD data and OoD data in non-linear feature spaces, inspired by effectiveness of the f2 normalization and the l2 distance from a kernel perspective on the KNN detector 7 Our proposed two feature mappings well characterize the non-linearity in penultimate features z of DNNs between InD and OoD data, enabling PCA extract an informative subspace w.zt the mapped features through principal components and the reconstruction errors.
5

---

Algorithm Kernel PCA for Out-of-Distribution Detection
1: if CoP then
2: ((+) Deas (ecos()) (ooa(())
- T=||2*
3: else if CoRP then
4: Sampling wi NN, m)) 8 1.... M.
5: Sampling u, ((, 2T), ; 1,... M.
6: (..)) 4 ORFF (dcos (-)).
7: end if
8: Calculating the covariance matrix in the mapped -space::
== E (0())) 1 (0x)) )²*)² T. 1
No. ΣN ((zi).
9: Applying cigendecoomoosition: EΦ U A U
10: Taking the first y columns of U w.r. the top-q largest eigenvalues in A U - = -U:::00 dl.
Ensure:
caaensiionnait--rddettn matrix U@:
11: Given a new sample å and its features 2.
12: Calculating the reconstruction error:
U (T(2) n²) ($( 11 //
(
Ensure: Reconstruction error e
Computation complexity our method, given any new sample & with the penultimate features in inference, to compute the reconstruction error e (.t) only need the feature mapping b the projection matrix U and the mean mapped training feature vector 1= Both U and 1x8h can be pre-calculated and stored in preparation for inference. Therefore, the entire computation cost of CoP and CoRP comes from the construction of the explicit feature mapping on new features ε.
For CoP, its feature mapping dees is an in-place operation and does not require additional computations. Therefore, the time and memory complexity of CoP is ((1).
For CoRP, the feature mapping ORFF of the Gaussian kernel requires 2M samplings for w; and uy, respectively, and M dot-products and M additions. Accordingly, the time and memory complexity of CoRP is (M).
In contrast, regarding the Equation 5 of the KNN detector, all the training features have to be stored at hand and iterated in inference, which implics a heavy O(Nv,) time and memory complexity. The ((L)O(M) of CoP/CoRP significantly outperforms the O(N,)) of KNN (M < N₂y). Detailed empirical comparisons are provided Section 5.T
In the following, Section 5 exhibits the SOTA performance of our KPCA detector over multiple prevailing detection methods. Section gives an analytical discussion between our covariance-basse KPCA and classic KPCA via kernel functions for OoD detection. Due to space limitation, more in-depth investigations on the kernel properties are left in Appendix [ covering ablation studies (Appendix C.1 alternative kernels (Appendix C.2) and sensitivity analysis (Appendix C.3)
5 Experiments on OoD detection
In experiments, our KPCA-based detectors, CoP and CoRP, are firstly compared with KNN in Section 5. and show stronger detection performance and cheaper computation costs. In Section 5.2 CoP and CoRP are further compared with the regularized PCA reconstruction error [8), and achieve SOTA OoD detection performance over various prevailing methods. The source code of this work has been publicly released! All the experiments are executed on 1 NVIDIA GeForce RTX 3090 GPU.
Datasets Experiments are executed on the commonly-used small-scale CIFARIO 39 and large- scale ImageNet- 1K benchmarks sets include SVHN [1,, LSUN 42 iSUN
400 following the settings in [78 For CIFAR10 as InD, OoD data Textures and Places365 For ImageNet-1K
as InD, OoD data sets contain iNaturalist 46) SUN 47 Places 45 and Textures
44

---

Table 1: The detection performance of different methods (ResNet50 trained on ImageNet-1K).
OoD sets
AVERAGE
method
iNaturalist
SUN
Places
Textures
FPR4
AUROC
FPR
AUROCT
FPR.
AUROCT
FPR.
AUROCT
FPR,
AUROCT
Standard Training
MSP 15
54.99
87.74
70.83
80.86
73.99
79.76
68.00
79.61
66.95
81.99
ODIN
16
47.66
89.66
60.15
84.59
67.89
81.78
50.23
85.62
56.48
85.41
Energy
5
55.72
89.95
59.26
85.89
64.92
82.86
53.72
85.99
58.41
86.17
GODIN
27
61.91
85.40
60.83
85.60
63.70
83.81
77.85
73.27
66.07
82.02
Mahala
97.00
52.65
98.50
42.41
98.40
41.79
55.80
87.43
55.47
KNN
59.00
86.47
68.82
80.72
76.28
75.76
11.77
97.07
53.97
85.01
CuP (ours)
67.25
83.41
75.53
79.93
82.48
73.83
8.33
98.29
$8.40
83.86
CoRP (ours)
50.07
89,32
62,56
83.
72.76
78,91
9.02
98.14
48.60
87.53
Supervised
Contrastive 84,21
Learning
MSP 15
32.18
93.30
60.36
61.68
83,94
50.62
84,68
51.21
86.53
ODIN 16
23.48
95.80
50.73
88.43
53.99
87.30
41.88
88.60
42.52
90.03
Energy 5
23.00
95.94
47.56
88.86
51.59
87.58
39.15
89.05
40.33
90.36
SSD
57.16
87.77
78.23
73.10
81.19
70.97
63.24
KNN
30.18
94.89
48.99
88.63
59.15
84.71
15.55
95.40
38.47
90.91
CoP (ours)
29.85
94.79
44.99
90.62
56.77
86.19
10.28
97.35
35.47
92.24
CoRP (ours)
23.61
95.86
41.07
91.25
53,52
87.27
10.23
97.04
32.11
92.86
Table Comparisons on the computation complexity between KNN 7 and our CoRP (ResNet50
on ImageNet-1K), Experiments executed on the same machine for fair comparison. The nearest neighbor searching of KNN is implemented via Faiss [88.
method
time and memorty complexity
time consuming (ms, per sample)
storage
KNN
O(Nw)
2J 15.59
20 GiB
CoP
O(1)
0.035
22 MiB
CoRP
((M)
~ 0.086
2 29 MiB
Metrics For the evaluation metrics, employ the commonly-used (i) False Positive Rate of OoD samples with 95% true positive rate InD samples (FPR), and (ii) Area Under the Receiver Operating Characteristic curve (AUROC). The average FPR95 and AUROC values over the selected multiple OoD data sets are viewed as the final comparison metrics.
5.1 Comparisons with nearest neighbor searching
The comparisons with KNN 7 cover both the benchmarks. Following the setups in KNN, for fair comparisons, we evaluate models trained via the standard cross-entropy loss and models trained via the supervised contrastive learning [49], and adopt the same checkpoints released by KNN: ResNet18 50 on CIFAR 10 and ResNet50 on ImageNet-Iee Here, the scoring function CoP and CoRP is (())
( in Algorithm
Table presents empirical results of ResNer50 on the ImageNet- 1K benchmark. In the standard training, our CoRP shows superior detection performance over KNN with lower FPR and higher AUROC values averaged over multiple OoD data sets. In supervised contrastive learning, both CoP and CoRP outperform other baseline results on each OoD data set. These results show that the proposed KPCA exploring non-linear patterns more advantageous than the nearest neighbor searching and all compared methods. Besides, the further improvements of CoRP over CoP also verify the effectiveness of the distance-preserviig property the Gaussian kernel kgau on top of the cosine kernel Kcos for OoD detection.
On the other hand, regarding the computational complexity in inference, Table 2 empirically shows the superior ( ((M))(MM time and memory complexity of CoP/CoRP over the O(N.z) of KNN, including: (i) the inference time of the nearest neighbor search in KNN and the reconstruction error calculation in CoP/CoRP:
(ii) the storage the InD training features KNN and the U and 2³ in CoP/CoRP. To be specific, for KNN, storing and iterating all the N = 1, 281, 167 features of ImageNet- 1K training set requires nearly 20 GiB and ms, respectively, while our CoP and CoRP directly compute the reconstruction error for each new sample with the creccaluulate projection

---

Table 3: Comparisons the detection performance between the regularized reconstruction error g and our CoP and CoRP fused with other OoD scores (MSP, Energy, ReAct and BATS) on each OoD data set (ResNet50 trained on ImageNet-IK). Best average results are highlighted with underlines.
OoD data sets
AVERAGE
method
iNaturalist
SUN
Places
Textures
FPR]
AUROCT
FPR.
AUROCT
[PR
AUROCT
FPR.
AUROC
FPR]
AUROCT
MSP 115
54,99
87.74
70.83
80.86
73.99
79.76
68.00
79.61
66.95
81,99
PCA 8
51.47
88.95
67.64
82.71
71.20
80.87
60.53
85.86
62.71
84.60
CoP
50.84
89.21
67.35
82.81
70.96
81.08
59.96
86.21
62.28
84,83
CoRP
43.70
91.70
61.79
85.43
66.67
83.07
45.67
91.86
54.46
88.02
Energy 5
55.72
89.95
59.26
85.89
64,92
82.86
53.72
85,99
58.41
86.17
PCA
50.36
91.09
54.19
87.55
64.13
84.00
29.33
92.59
49.50
88.81
CoP
45.13
92.15
52.33
88.01
61.49
84.96
29.13
92.57
47.02
89,42
CoRP
26.85
95.15
40.38
90.76
51.26
87.35
12.11
97.17
32.65
92.61
ReAct
IK
20.38
96.22
24.20
94.20
33.85
91.58
47.30
89.80
31.43
92.95
PCA 8
10.17
97.97
18.50
95.80
27.31
93.39
18.67
95.95
18.66
95.76
CoP
13.30
97.44
19.80
95.37
29.92
92,64
15.90
96.51
19.73
95.49
CoRP
10.77
97.85
18.70
95.75
28.69
93.13
12.57
97.21
17.68
95.9%
BATS 19
42.26
92.75
44.70
90.22
55.85
86.48
33.24
93.33
44.01
90.69
PCA
29.66
38.11
90.03
$1.70
87.25
13.46
97.09
33.23
92.56
CoP
27.14
94.87
34.36
91.96
47.68
87.87
11.97
97.33
30.29
93.01
CoRP
18.74
96.31
28.02
93,49
41.41
89,78
9.45
97.79
24.41
94.34
ODIN 16
47.66
89.66
60.15
84.59
67.89
81.78
50.23
85,62
56.48
85.41
Mahala
24
97.00
$2.65
98.50
42.41
98.40
41.79
55.80
85.01
87.43
$5.47
ViM 35
87.13
79.62
81.67
83.81
77.80
14.95
96.74
61.81
85.83
DICE 15
26.66
94.49
36.08
90.98
47.63
87.73
32.46
90.46
35.71
90,92
DICE-ReAct
20.08
96.11
26.50
93.83
38.34
90.61
29.36
92.65
28.57
93.30
NNGuide
24
25.73
95.12
37.18
91.21
46.97
88.67
27.70
92.30
34.39
91.82
DML+ 28
13.57
97.50
30.21
94.01
39.06
91.42
36.31
89.70
29.79
93.16
ASII-B
14.21
22.08
95.10
33.45
92.31
21.17
95.50
22.73
95.06
ASH-S
11.49
97.87
27.98
94.02
39.78
90.98
11.93
97.60
22.80
95.12
SCALE 21
9.50
98.17
23.27
95.02
34.51
92.26
12.93
97.37
20.05
95.71
DDCS 22
11.63
97.85
18.63
95.68
28.78
92.89
18.40
95.77
19.36
95.55
matrix and the mean vector from the training data, resulting in much higher processing speed and far less storage. The number of RFFs M for CoRP in this experiment M 4, 096 (M < Nuz).
addition, KPCA also outperforms KNN on the CIFARIO benchmark with improved OoD detection performances, which we leave Appendix B for more details.
5.2 Comparisons with regularized reconstruction errors
In [), to alleviate the weak detection performance of PCA reconstruction error e(λ) of Equation (3) the authors proposed to regularize e(2) by the feature norm || and fusion strategy to boost its detection performance by introducing existing OoD scores. Firstly, the regularized reconstruction
error Creg 2 is calculated the original x-space as Greg (2) |0, ($- (2- Then, the
|| authors claimed that such regularized version Greg (8) )) is still insufficient for OoD detection, and designed fusion strategy to combine Ereg with other existing OoD scores. For example, to fuse ereg with the Energy [ score, the final scoring function is S( (1 Crog - (2).
In this section, we show that our KPCA reconstruction error Φ outperforms the regularized PCA reconstruction сиrог Greg under the same fusion framework. Following the settings 88, for fair comparison, the fused OoD scores include MSP [15,, Energy (5), ReAct [18] and BATS 19 The detection experiments are executed on the ImageNet- benchmark with pre-trained ResNet50 and MobileNet [5I] checkpoints from PyTorch [52]
Table presents the comparisons between and ours on the ImageNet- benchmark of ResNet50. When fused with MSP, Energy and BATS, both the KPCA-based CoP and CoRP outperform the regularized reconstruction error g on almost all the OoD data sets with substantially improved FPR and AUROC values. Specifically, when fused with the ReAct method [18], the CoRP achieves new SOTA OoD detection performance among various prevailing detectors. Experiments on MobileNet also show superior performance of CoP and CoRP, details in Appendix B

---

All these experiment results indicate that an appropriately mapped ((z)-space benefits the OoD detection, as the non-linearity in x s-spac gets alleviated by the feature mapping D. Our work provides 2 viable selections for & with empirical validations, which we hope could attract attentions towards the non-linearity in InD and Ool features for the research community from a kernel perspective.
6 Analytical discussions with KPCA via kernel functions
In CoP and CoRP, KPCA is executed with the covariance matrix of mapped features ((z). In contrast, in the classic KPCA [TT [2), such feature mappings & are not explicitly given, and it rather works with a kernel function applied to original features z. In this section, we supplement our covariance-basee based KPCA with its kernel function implementation, including theoretical discussions and empirical comparisons on OoD detection. Our CoP and CoRP are shown to be more effective and efficient than their counterparts that employ kernel functions.
In the classic KPCA, the kernel trick enables projections to the principal subspace via kernel functions without calculating 4 However, how to map the projections in the principal subspace back to the original z-space remains a non-trivial issue, known as the pre-image problem [53], which makes it problematic to calculate reconstructed features via kernel functions. To address this issue, the following Proposition [ shows a flexible way to directly calculate reconstruction errors without building reconstructed features, as to apply the kernel trick, shown in the subsequent Proposition 2 Proposition 1. The KPCA reconstruction error ex (2) can be represented as the norm of features projected the residual subspace, i.e,, the U -subspace with U [U q: U
22) 1 (2) - 1²)
(8)
Proposition | implies that the reconstruction error equals to the norm of projections in the residual Uh "s'ssuuppee,, ie., the subspace consisting of those principal components that are not kept, see proofs in Appendix(D D Accordingly, as typically done in the classic KPCA, we can introduce a kernel function to perform dimension reduction, but to the residual subspace, and then calculate the norms of the reduced features as the reconstruction error, illustrated by Proposition] 2
Given a kernel function k(,,-) I x RT + P. we have a kernel matrix K E IR.N...NN on training data with Ki, k(z,, 2y), and vector Kg E R--rr with the i-th element k( 2) for a new sample 2. Proposition Sisiiionnsssosssssss shows how to calculate the KPCA reconstruction error via the kernel function k. Proposition 2. The KPCA reconstruction error (2) w.r.t kernel function k can be calculated as:
(²)
where A E TR: Ni includes eigenvectors of the kernel matrix K w.r.f the top-l smallest eigenvalues.
According to Proposition now CoP and CoRP can be implemented via kernel functions. For CoP, we just directly apply the cosine kernel function kicos on features compute the kernel matrix K and the projection matrix A, so as to obtain (2) following Equation For CoRP, we should adopt the Gaussian kernel function kgau on the no-ommalede inputs = 2
to calculate K, A and
(²)). Figure 2 kernel function implementations.
shows comparisons on the detection performance between CoP/CoRP and their
In Figure 2 the detection performance of KPCA with kernel functions is evaluated by varying the explained variance ratio of the kernel matrix K. The larger the explained variance ratio, the smaller the dimension / of A. The best detection results achieved by CoP/CoRP are illustrated as the dashed lines. Clearly, regarding the OoD detection performance, reconstruction errors e calculated by kernel functions are less effective than those calculated explicitly in the mapped ((x) (x)-spacee.
Aside from the detection performance, KPCA with kernel functions is far less computationally efficient than CoP/CoRP in two aspects. On the one hand, the time expense of eigendecomposition on the N₁ x Nrr kernel matrix K by the former is more expensive than that on the m x m or M x M covariance matrix Σ by the latter, since No. > M and Nu. >> m. For example, on the ImageNet- IK benchmark with MobileNet, these settings are Nur = 281, 167, M = 2560 and m 1280, on which KPCA with kernel functions actually nearly prohibitive. On the other hand, in the inference stage, KPCA via kernel functions yet requires an O(N,,) time and memory complexity in calculating kg, as all the training data has to be stored and iterated, which much higher than the (L))O(M) complexity of our CoP/CoRP.
9

---

cosine kernel
cosine-Gaussian kernel
85
kemel func.
CoP
kernel func.
CoRP
FPR95 (%)
65
25
0.95 0.96 0.97
0.98
0.99
1.0
0.8
0.85
0.9
0.95 0.99
explained variance ratio of K
explained variance ratio of K
Figure 2: Comparisons on the average detection FPR values between CoP/CoRP and their kernel function implementations in the CIFARIO benchmark. experiments. 5,000 images the CIFARIO training set and 1, 1,000 images of the CIFAR10 1 test set and OoD data sets are randomly selected.
Conclusion
As PCA reconstruction errors fail to distinguish OoD data from InD data the penultimate features of DNNs, kernel PCA introduced for its non-linearity in the manner of employing explicit feature mappings. To find appropriate kernels that can characterize the non-linear patterns in InD and OoD features, we take kernel perspective
decouple and analyze key components of an existing KNN detector [7], and thus propose cosine kernel and cosine-Gaussian kernel for KPCA. Specifically, two explicit feature mappings 2(-) induced from the two kernels are leveraged on original features For the cosine kernel, its explicit feature mapping can be directly obtained. For the Gaussian kernel, we adopt the celebrated random Fourier features to approximate the Gaussian kernel. The mapped (=z)-space enables PCA to extract principal components that well separate InD and OoD data, leading to distinguishable reconstruction errors. Extensive empirical results have verified the improved effectiveness and efficiency of the proposed KPCA with new SOTA Ool) detection performance. Besides, more in-depth analyzes are drawn on the individual effects of the cosine kernel and the Gaussian kernel, and the involved multiple syper-parammttrs. In addition, theoretical discussions and associated experiments are provided to bridge the relationships between our covariance-batte KPCA and its kernel function implementation advantages of our method.
to further illustrate the
One limitation of the KPCA detector is that the two specific kernels still manually selected with carefully-tuned parameters. It remains a valuable topic in the OoD detection task whether the parameters of kernels could be learned from data according to some optimization objective. For example, deep kernel learning [54] could be considered as an alternative choice so as to pursue stronger kernels that better characterize InD and OoD with enhanced detection performance by an additional learning step on the features. We hope that the proposed two effective kernels verified empirically in work could benefit the research community solid example for future studies.
Acknowledgments and Disclosure of Funding
This work is jointly supported by National Natural Science Foundation of China (62376153, 62376155), and Shanghai Municipal Science and Technology Research Major Project (201ISHZDZX0102,,
Societal impacts
The societal impacts of this work are mainly positive, as it aims at detecting OoD samples in inference or deployment stage of DNNs, which benefits researches in trustworthy deep learning. Through our work, we hope that new inspirations on the non-linearity in data could be drawn from kernel perspective so to highlight the safety issue in real-world machine learning applications.
10

---

References
[I] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-660 (1168006655,,, 2020.
[2] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[3] Clark Barrett, Brad Boyd, Elie Bursztein, Nicholas Carlini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodoreseu, Anupam Datta, Soheil Feizi, et al. Identifying and mitigating the security risks of generative ai. Foundations and Trends@) in Privacy and Security, 6(1):1-52.
2023.
[4] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. International Journal of Computer Vision, pages 1-28, 2024.
[5] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- based out-of-distribution detection. Advances in neural information processing systems, 14:21464-21777 2020.
[6] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances Neural Information Processing Systems, 34:677- 689, 2021.
[7] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pages 20827-20840. PMLR, 2022.
[8] Xiaoyuan Guan, Zhouwu Liu, Wei-Shi Zheng, Yuren Zhou, and Ruixuan Wang. Revisit pea- based technique for out-of-distribution detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19431-19439 2023.
[9] Francesco Tonin, Arun Pandey, Panagiotis Patrinos, and Johan AK Suykens. Unsupervised energy-based out-o--distributtin detection using stiefel-restrieteed kernel machine. In 2021 International Joint Conference on Neural Networks (JCCNN), pages 1-8. IEEE, 2021.
[10] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
[11] Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Kernel principal component analysis. In International conference on artificial neural networks, pages 583-588. Springer, 1997.
[12] Bernhard Schölkopf, Alexander Smola, and Klaus-Rober Robert Müller. Nonlinear component analysis kernel eigenvalue problem. Neural computation, 1065):1299-1319 1319, 1998.
[13] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
[14] Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling ouiot--distribution detection for real- world settings. In International Conference on Machine Learning, pages 8759-8773. PMLR, 2022.
[15] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference Learning Representations, 2016.
[16] Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of -ittof-distribution image detection in neural networks. In International Conference on Learning Representations, 2018.
[17] Yingwen Wu, Tao Li, Xinwen Cheng, Jie Yang, and Xiaolin Huang. Low-dimensional gra- dient helps distr--iittintttio detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.

---

[18] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out- -ittof-diitrrbutionn detection with rectified activations. Advances in Neural Information Processing Systems. 34:144-157, 2021.
[19] Yao Zhu, Yuefeng Chen, Chuanlong Xie, Xiaodan Li, Rong Zhang. Hui Xue, Xiang Tian, Yaowu Chen, et al. Boosting -uito--distribution detection with typical features. Advances in Neural Information Processing Systems, 35:20758-20769, 2022.
[20] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In The Eleventh International Conference Learning Representations, 2023.
[21] Kai Xu, Rongyu Chen, Gianni Franchi, and Angela Yao. Scaling for training time and post- hoc out-of-distribution detection enhancement. In The Twelfth International Conference Learning Representations, 2024.
[22] Yue Yuan, Rundong He, Yicong Dong Zhongyi Han, and Yilong Yin. Discriminabilty--diie channel selection for out-of-distribution detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26171-26180 2024.
[23] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-o--distribution samples and adversarial attacks, Advances neural information processing systems, 31, 201 8.
[24] Jaewoo Park, Yoon Gyo Jung, and Andrew Beng Jin Teoh. Nearest neighbor guidance for out-of-distribution detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1686-1695, 2023.
[25] Yeonguk Yu, Sungho Shin, Seongju Lee, Changhyun Jun, and Kyoobin Lee. Block selection method for using feature norm out-of-distributiin detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15701-15711 2023.
[26] Yue Song, Nicu Sebe, and Wei Wang. Rankfeat: Rank-1 feature removal for out distribbuuion detection. Advances in Neural Information Processing Systems, 35:1 35:17885-17898, 2022.
[27] Yen- Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from di-or-ddistibuttinn data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10951-10960, 2020.
[28] Zihan Zhang and Xiang Xiang. Decoupling maxlogit for mut-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3388 0108-337,2220 2023.
[29] Kun Fang, Qinghua Tao, Xiaolin Huang, and Jie Yang. Revisiting deep ensemble for out-of- distribution detection: A loss landscape perspective. International Journal of Computer Vision, pages 1-20, 2024.
[30] Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards a theoretical framework of out-of distribuiiin generalization. Advances in Neural Information Processing Systems, 34:23519-255 12-331,,,,, 2021.
[31] Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribution detection learnable? Advances in Neural Information Processing Systems, 35:37199-37213, 2022.
[32] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 33:11839 1113999-1112,1111111111 2020.
[33] Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2020.
[34] Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for ouits--diitiiiuttin detection. In European Conference on Computer Vision, pages 691-708. Springer, 2022.
12

---

[35] Haoqi Wang. Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-disti with virtual-logit matching. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4921 4930, 2022.
[36] Walter Rudin. Fourier analysis groups, volume 121967. Wiley Online Library, 1962.
[37] Kun Fang, Fanghui Liu, Xiaolin Huang, and Jie Yang. End-to-end kernel learning via generative random fourier features, Pattern Recognition, 134:109057, 2023.
[38] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconeiling modern machine- learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 1 116(32):15844 15854, 2019.
[39] Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
[40] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248-24882 255, 2009.
[41] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[42] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint xxxv:11000 50.5500000..,.. 2015.
[43] Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755. 2015.
[44] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conferenc computer vision and pattern recognition, pages 3606-3613, 2014.
[45] Bolci Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-1466: 2017.
[46] Grant Van Horn. Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769-887 8769-8778, 8778, 2018.
[47] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[48] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 733):335-547 2019.
[49] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition, pages 770-778. 2016.
[51] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
13

---

[52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[53] JT-Y Kwok and IW-H Tsang. The pre-image problem in kernel methods. IEEE transactions on neural networks, 15(6):111-1125. 2004.
[54] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378. PMLR, 2016.
[55] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-299888 29834, 2021.
[56] Ninh Pham and Rasmus Pagh. Fast and scalable polynomial kernels via explicit feature maps. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 239-247, 2013.
[57] Heiko Hoffmann. Kernel pea for novelty detection. Pattern recognition, 40(3):863-874, 2007.
14

---

A Details of representative OoD detectors
In this section, we elahorate the scoring function S(-) of the OoD detectors included in the comparison experiments of Section 5 Given well-trained DNN f I + R with inputs E Rd, the outputs
are c-dimension logits f(x) E IR w.rd c classes. The DNN f learns features z E IR" of æ before the
last linear layer, i.e., the penultimate features
MSP [15 employs the softmax function on the output logits and takes the maximum probability as the scoring function. Given a new sample a E R its MSP score is
SMSP ( = max (((()
(10)
ODIN 16 introduces the temperature scaling and adversarial examples into the MSP score:
f(ta)
SODIN( (+) max [sotmmax(
(11)
T
where T denotes the temperature and a denotes the perturbed adversarial examples of &
Mahala (23 employs the Mahalanobis score to perform OoD detection. The DNN outputs at different mixture of multivariate Gaussian distributions, and the Mahalanobis distance is calculated. Then, a linear regressor is trained to achieve a weighted Mahalanobis distance at different layers the final detection score. To train the linear regressor, the training data and the corresponding adversarial examples are adopted as positive and negative samples, respectively.
layers are modeled
(f) max ( 1 (() p!),
(12)
Smahal (+) = Σ m
where (d( denotes the output features at the 1-th layer with the associated mean feature vector pu of class-i and the covariance matrix Σ,, and aj denotes the linear regression coefficients.
Energy uses an energy function logits energy is well aligned with input probability densities:
Senergy (+) ) = log ex)().()))
(13)
where f.(t) denotes the i-th element in the c-dimension output logits f()).
GODIN [77 improves ODIN from 2 aspects: decomposing the probabilities and modifying the input pre- preppeocesinng On the one hand, a two-branch structure with learnable parameters imposed after the logits to formulate the decomposed probablities. On the other hand, the magnitude of the adversarial examples is optimized instead of manually tuned in ODIN.
proposes activation truncation on the penultimate features z of DNNs, as the authors observe that features of OoD data generally hold high unit activations in the penultimate layers. The feature clipping is implemented in a simple way:
ReAct 18
(14) where is pre-defined constant. The clipped features & then pass through the last linear layer and yield modified logits. Other logits-based OoD methods such as Energy could be applied on the modified logits to produce detection score.
min {z, a).
KNN is a simple but time-consuming and memory-inefficien detector since it performs nearest neighbor search on the lz-ormalld penultimate features between the test sample and all the training samples. The negative of the (k-th) shortest f2 distance set as the score for new sample
VIM 335 combines information from both logits and features in a complicated way for OoD detection. Firstly, penultimate features are projected the residual space obtained by PCA. Then the norm of projected features gets scaled together with the logits via the softmax function. Finally the scaled feature norm is selected as the detection score.
OoD detector by preserving the most important weights in the last linear layer. Denote the weights W Rm and the bias b E R in the last linear layer, the forward propagation of DICE defined as:
DICE 34 is a sparsification-bass
fDICE( A (M O T) b.
(15)
15

---

C is the element-win multiplication, and M E m is masking matrix whose elements are determined by the element-wise multiplication between the i-th column w, R in W and the penultimate features 2: u, ④ E. Then, similar as ReAct, logits-based detectors could be executed on the modified logits SDICE( (+) to produce a detection score.
BATS [19 proposes to truncate the extreme outputs of Batch Normalization (BN) layers via the estimated mean and standard deviations stored in BN layers, those extreme features would lead to ambiguity and should be rectified. However, in the released code, the authors actually does not use any information from the BN layers, but instead simply perform clipping on the penultimate features via the feature mean and standard deviations.
PCA 8 re-formulates the reconstruction errors and empirically shows the inseparativity via the re-formulated errors between InD and Ool data in the primal z-space. The authors further propose a regularized reconstruction error and fusion strategy to boost the OoD detection performance.
exploits the guidance of the Energy score on logits to boost the detection performance of the nearest neighbor search on features. Specifically, the training features are firstly scaled by their corresponding Energy scores, then KNN is executed on such re-scaled features for the new sample. The final detection score is set as the multiplication of the searched distance and the Energy score. The time and memory complexity of NNGuide is still the same ((,,,) as that of KNN (7].
NNGuide 24
DML (8 decouples the maximum logits into two parts: the maximum cosine similarity (MaxCosine) and the maximum norm (MaxNorm), and employs their ensemble as the detection score. DMI. reveals that the cosine similarity and the feature norm jointly contribute to the effectiveness of the previous MSP [5] and MaxLogit [1] methods and designs new training losses from the perspective 55
of feature collapse
so as to further improve the performance of MaxCosine and MaxNorm.
ASH (00 is a post-hoc detection method that removes the abnormal information in features. ASH includes two stages: removing a large portion of the features, and adjusting the remaining feature values by scaling up or assigning constant value. ASH exhibits advantages over the classic ReAct method [8] no global thresholds and stronger flexibility, and shows better detection performance.
SCALE 2 analyzes the rectification and scaling components of the ASH method, and improves ASH by only a scaling process in the post-hoc stage.
DDCS 22 investigates the effects of different channels based on existing feature- clappreg detection methods, and proposes a channel-leve anomalous activations pre-rectifying module so as to clip features more carefully for better detection performance.
We follow the settings in KNN 7 and include CSI 22 and SSD +
into the comparisons in Section 3.I. The 2 methods adopt the contrastive losses to train DNNs. In the comparison results of Table [ and the following Table 4 the reported detection results of CSI and SSD are directly from
7 and are obtained by executing the Mahalanobis detector on learned features of DNNs trained by
CSI and SSD. Refer 7 for more details.
B Supplementary experiment results on OoD detection
Table 4 Aillustrates the comparison results on the CIFARIO benchmark between our CoP/CoRP and the KNN detector 77). Similar as the comparisons the ImageNet-IX IK benchmark in Table | CoRP outperforms other baselines in both the standard training and the supervised contrastive learning with lower FPR and higher AUROC average values.
Table 5jshoss the comparison results on the ImageNet- 1K benchmark between our CoP/CoRP and the regularized reconstruction error [8] of MobileNet [51]. Similar as the case of ResNet50 in Table 3 under the same fusion trick with other detection scores, our KPCA reconstruction errors of CoP and CoRP significantly outperform the regularized PCA reconstruction error of [8], implying the substantial improvements by characterizing the non-linear data distribution of the InD and OoD
features via the devised two proper non-linear kernels.
16

---

Table 4: The detection performance of different methods (ResNet18 trained CIEAR10).
OnD data
method
SVHN
LSUN
ISUN
Textures FPR
Placea365
AVERAGE
FPR.
AUROCT
FPR
AUROCT
FPR.
AUROCT
AUROCT
FPR.
AUROCT
FPR]
AUROCT
Standard Training
MSP 15
59.66
91.25
45.21
91.80
54.57
92.
66.45
88.50
62.46
88.64
57.67
90.86
ODIN
Energy
54.41
91.22
10.19
98.05
27.52
95.59
55.23
89.37
42.77
91.02
38.02
93.05
GODIN
F2
96.60
4.90
99.07
94.94
46.91
62.63
32.80
93.52
Mahala
9.24
97.80
67.73
73.61
6.02
98.63
23.21
92.91
83.50
69.56
37.94
86.50
KNN
24.53
95.96
25.29
95. 69
25.55
95.26
27.57
94.71
50,90
89.14
30.77
94.15
(ours)
11.56
97.57
CoRP (ours)
20.68
96.53
19.19
96.71
21.49
96.26
21.61
96.08
53.73
89.14
27.34
94,95
Supervised Contrastive
Learning
CSI 32
37.38
94.69
5.88
98.86
10.36
98.00
28, 85
941.8
38.31
93.04
24.16
95.89
SSD 33
1.51
99.68
6.09
98.48
33.60
95.16
12.98
97.70
28.41
94.72
16.52
97.15
KNN
2,42
99.52
1.78
99.48
20.06
96.74
8.09
98.56
23,02
95.36
11.07
97.93
Col (ours)
0.55
99.85
1.12
99.67
23.91
96.11
4.79
99.06
19.92
95.63
10.06
98.07
(ours)
Table 5: Comparisons on the detection performance between the regularized reconstruction error g and our CoP and CoRP fused with other OoD scores (MSP, Energy, ReAct and BATS) on each Ool data set (MobileNet trained on Net Best average results are highlighted with underlines.
OoD data
AVERAGE
method
iNaturalist
SUN
Places
Textures
FPR.
AUROCT
FPR.
AUROC1
FPR.
AUROC1
FPR.
AUROC
FPR.
AUROCT
MSP 15
64.29
85.32
77.02
77.10
79.23
76.27
73.51
77.30
73.51
79.00
+ PCA 8
59.49
86.87
73.75
79.41
76.79
77.94
65.71
83.46
68.93
81.92
CoP
$7.14
87.62
72.86
79.45
76.17
77.77
60.71
86.42
66.72
82.82
CoRP
55.71
80.77
75.33
78.90
58.90
87.13
65.36
83.73
Energy 5
59.50
88.91
62.65
84.50
69.37
81.19
58.05
85.03
62.39
84,91
PCA
56.92
89.62
60.07
85. $5.80
69.23
81.72
34,22
91.66
55.11
87.20
CoP
51.21
90.79
59.88
85.84
68.62
81.74
23.16
94.55
50.72
88.23
CoRP
43.85
91.96
52.17
87.91
63.75
83.59
19.02
95.41
44,70
89.72
ReAct 18
43.07
92.72
52.47
87.26
59.91
84.07
40.20
90.96
48.91
88.75
PCA 8
35.84
93.66
40.35
90.77
52.38
86.76
18.44
95.39
36.75
91.65
CoP
60.62
84.45
12.62
96.97
39.30
CoRP
31.72
94.27
40.77
90.98
$5.69
86.42
10.48
97.49
34,66
92.29
BATS 19
49.57
91.50
57.81
85.
64.48
82.83
39.77
91.17
52.91
87.87
PCA 8
50.51
90.86
55.41
87.00
66.43
82.60
23.26
94.70
48.90
88.77
CoP
42.68
92.24
55.01
86.89
65.70
82.44
13.78
96.77
44,29
89.58
CoRP
36.10
93.37
45.92
89.47
59.82
84,83
11.37
97.24
38.30
91.23
ODIN 16
58.54
87.51
$7.00
85.8%
59.87
84.77
$2.07
85.04
56.87
85,79
Mahala
23
62.11
81.00
47.82
83.66
52.09
83.63
92.38
33.06
63.60
71.01
VIM 35
91.83
77.47
94.34
70.24
93.97
68.26
37.62
92.65
79.44
77.15
DICE 3
43.28
90.79
38.86
90.41
53.48
85.67
33.14
91.26
42.19
89.53
DICF-ReAct
41.75
89.84
39.07
90.39
$4.41
84.03
19.98
95.86
38.80
90.03
NNGuide 24
45.73
91.19
51.03
87.87
60.60
84.44
29.50
92.47
46.72
88.99
ASH-B
31.46
94.28
38.45
91.61
$1.80
87.56
20.92
95.07
35.66
92.13
ASH-S
39.10
91.94
43.62
90.02
58.84
84.73
13.12
97.10
38.67
90.95
C Analytical discussions on kernels
In this section, more in-depth discussions are drawn on the effects of cosine and Gaussian kernels in CoP and CoRP for OoD detection in Appendix C. and Appendix C.2) respectively. A comprehensive sensitivity analysis on the involved hyper- parameters CoP and CoRP are presented in Appendix
C.1 Effects of the cosine kernel
The cosine kernel in CoP and CoRP appears an indispensable basis in alleviating the linear insepara- bility in InD and OoD features. The reason for effectiveness lies the imbalanced feature norms ||z|| between InD and OoD features, which has also been observed in preceding works 7 6 32 Figure shows the feature norms multiple InD and OoD data sets, from which one can find clear disparities of the InD and OoD feature norms. The cosine kernel koos and the corresponding feature
17

---

CI
ImageNet-1K
LSUN
SUN
10
12
20
Figure 3: A density histogram of the imbalanced norms of InD and OoD features. InD: CIFARIO and ImageNet- 1K. OoD: LSUN and places365, SUN and Textures.
InD (CIFAR1O) (LSUN)
OoD (ISUN)
Figure 4: T-SNE visualization of the original features (left), mapped features W.E.& a Gaussian kernel (middle) and mapped features w.e cosine kernel (right).
mapping Pcos in Equation between InD and OoD data.
(6) thereby normalize the feature norms and facilitate the separability
Figure 4 illustrates the t-SNE visualization on the InD and OoD features further imply the importance of the cosine kernel. As shown in the middle panel of Figure 4 the Gaussian kernel alone fails in creating separable InD and OoD features in the mapped space and actually leads to complete mess of the mapped features. In contrast, the cosine kernel significantly alleviates the inseaalyisseparrablity of features, shown in the right panel of Figure 4
Ablation studies on the cosine feature mapping &ee in CoP and CoRP are executed to verify its indispensability for OoD detection. Specifically, CoP without &ees reduces standard PCA the z-space, and CoRP without dess reduces to KPCA with a Gaussian kernel. Table 6 shows the corresponding detection FPR and AUROC values each OoD data set in the ImageNet- K benchmark of ablations on écos- Both the standard PCA and the Gaussian KPCA exhibit worse detection performance than CoP and CoRP. Particularly, the single Gaussian kernel in KPCA actually results in a complete failure in detecting OoD samples with nearly 95% FPR values. Therefore, the cosine kernel is essential in characterizing the non-linearity in InD and OoD features and critical for the superior performance of the KPCA detector.
C.2 Alternative kernels
The design of the Gaussian kernel in CoRP is motivated by the useful &g distance on TहÎr in the KNN detector (7). The Gaussian kernel preserves the beneficial & distance relationship in dcce (z)- space through the RFFs mapping. In this section, provide two alternative choices aside from the Oosine-Gaussian kernel.
The cosine- Laplacian kernel explores the e distance in dens (2) (2)=(2))pppct -space via the Laplacian
kernel Atap:
(21 (2) e >1 =
(16)
To construct the RFFs for kiap. the sampling distribution of the wi Equation (4) is
2
Cauchy distribution p(w)
18

---

Table 6: The detection results among variety of kernels (ResNet50 trained on ImageNet-1K).
OoD data
AVERAGE
kernel
iNaturalist FPR.
SUN
Places
Textures
AUROCT
F
AUROCT
FPR.
AUROC
FPR
AUROCT
FPR
AUROCT
PCA (no kernels)
95.46
52.01
97.98
44,86
97.99
45.19
46.22
87.77
84.41
57.46
Polynomial
96.03
53.07
98.26
42.84
97.85
45.02
95.50
47.96
96.91
47.22
Laplacian
94.65
50.25
94.68
50.29
95.28
49.
94.66
50.34
94.82
50.17
Gaussion
94,46
50.83
95.17
50.33
94,80
50.46
95.09
50,80
94.88
50.60
Cosine (CoP)
67.25
83.41
75.53
79.93
82.48
73.83
8.33
98.29
58.40
83.86
Cosine-Polymomial
54.10
84.48
75.97
75.04
82.82
69.01
59.15
83.27
68.01
77.95
Cosine-I aplacian
76.18
77.95
77.54
76.70
84.47
70.16
11.97
97.57
62.54
80.60
Cosine-Gaussian (CoRP)
50.07
89.32
62.56
83.74
72.76
78.91
9.02
98. 14
48.60
87,53
The cosine posiin-polynomial kernel does not hold the l nor lg Distance-presernve property for Ocos )s)spac,, as the polynomial kernel defined as:
Apoly(21: x2) = (z x2 + c)
(17)
To obtain an explicit feature mapping for Appll.. we do not adopt the RFFs and take the Tensor Sketch approximation (56) instead for simplicity.
Table filluttissssssrtes the comparisons on the detection performance among multiple alternative kernels. Actually, both the cosine-l.aplacaa kernel and the cosine-polynomen kernel cannot bring detection performance gains on top of the cosine kernel (CoP), which indicates that the é fiddsttncce relationship characterized by the Laplacian kernel and the inner-product information characterized by the poly- nomial kernel in the &eos -ppce are less effective promoting the separability between InD and OoD features. Thus, the cosine-Gaussie kernel is used and recommended in the proposed KPCA method for OoD detection.
C.3 Sensitivity analysis
comprehensive sensitivity analysis is executed to show the effects of hyper-parameters in CoP and CoRP. A common hyper-parameter in CoP and CoRP is the number of columns q of the dimeensionality-reduttioo matrix U Additional hyper-parameters of CoRP include the bandwidth of the Gaussian kernel and the number of RFFs M. In the following, we discuss the influence of each parameerr,, and report experiment results of the detection performance by varying one hyper-parameter with the others fixed.
Effect of q q indicates the number of preserved principal components and determines how much information captured by the subspace where the InD and OoD data is projected onto. q is selected as the minimal number of principal components with the amount of information that exceeds the given explained variance ratio.
Figure 5 iillustrates the detection performance of CoP and CoRP under varied explained variance ratios. On CIFARIO and ImageNet- K benchmarks, for CoP, a mild value of the explained variance ratio is suggested with around 90% for keeping the components. Regarding CoRP, sufficiently large value of the explained variance ratio is no longer essential for CoRP on the ImageNet- IK benchmark, which might be due to that the concatenated kernels make the useful information for distinguishing OoD samples more concentrated less principal components.
Effect of y The Gaussian kernel width directly affects the mapped data distribution. For large - se O for 21 * ø2, which indicates that the mapped features of 1 and #2 are (nearly) mutually-orthhogonal. In this case, a PCA would become meaningless. For a small y, the KPCA-base reconstruction emrors will approach the standard PCA-based ones, shown by (57). Figure iillustrates the detection FPR95 and AUROC values CoRP w.nl varied Gaussian kernel width 7 on CIFAR10 and ImageNet- IK benchmarks. Clearly, neither too large too small kernel width benefits the detection performance, and a mild value of y should be carefully tuned for different in-distribution data.
Y, kgau (Z1, (21,
Efffect M The number of RFFs M determines the approximation ability of RFFs towards the Gaussian kernel. As proved in [1] the larger the M, the better the RFFs approximate kgau-
19

---

CIFARIO, CoP
ImageNet-IK 1K, Cop
90
95
FPR95 (%6
70
AUROC (96)
FPR95 (%)
65
85
AUROC (%6
50
75
60
80
30
65
55
10
AUROC
55
50
70
0.6
0.7
1.0
0.94
0.98
1.00
explained variance ratio
explained variance ratio
CIFAR10,
ImageNet-IK, CoRP
45
-88
FPR95 (%)
-95
AUROC (%)
FPR95 (%)
AUROC (96)
35
90
88
50
FPR95 AUROC
885
25
-80
0.5
0.6
0.7
0.8
0.9
1.0
0.3
0.4
0.5
0.6
0.7
0.8
explained variance
explained variance
Figure 5: A sensitivity analysis on the explained variance ratio of CoP (top) and CoRP (bottom). The average FPR and AUROC values of Ool) data sets in CIFAR10 10 and ImageNet-Ik I benchmarks are reported. The Gaussian kernel width y and the dimension M of RFFs in CoRP are fixed.
CIFAR10, CoRP
ImageNet-1K, CoRP
45
75
85
FPR95 (%)
AUROC (%))
FPR95 (%)
AUROC (%)
65
80
90
75
55
AUROC
45
1.0 1.5 2.5 3.0 3.5 4.0 5.0
0.5
2.0 2.5 3.0 4.0 4.5 5.0 Gaussian kernel width y
Gausslan kernel width
Figure 6: A sensitivity analysis on the Gaussian kernel width of CoRP. The average detection FPR and AUROC values of OoD data sets in CIFARIO and ImageNet- IK benchmarks are reported. The explained variance ratio and the dimension M of RFFs are fixed.
Figure indicates the detection FPR95 and AUROC values of CoRP w.r.f. multiple values of the RFFs dimension M on CIFARIO and ImageNet-IK benchmarks. As M increases, the detection performance gets improved since the RFFs better converge to the Gaussian kernel. Considering the computation efficiency of eigendecomposition on the covariance matrix R. M M in the comparison experiments, we adopt M = 4m on CIFAR10 with ImageNet-1K IK with m 2048 for ResNet50 and m 1280 for MobileNet.
512 for ResNet18, 18, and M 2m on
D Supplementary theoretical results
The proof Proposition D
presented.
Proof. Recall Th and suppose + T R and U E M is the eigenvector matrix of the covariance matrix of the training data with U = U q. U and + = M. For the reconstruction error e (2) of new test sample m in the mapped (zz)-space, we have:
20

---

CIFAR1O, CoRP
ImageNet-1K 1K, CoRP
45
75
85
FPR95 (%)
AUROC (%)
FPR95 (%6
AUROC (%)
65
80
35
93
75
55
FPR95
.22
700
25
AUROC
45
512
1024
1536
2048
512
2048
4096
dimension of RFFs
dimension M of RFFs
Figure 7: A sensitivity analysis on the dimension M of RFFs of CoRP. The average detection FPR and AUROC values of OoD data sets in CIFARIO and ImageNet- IK benchmarks are reported. The explained variance ratio and the Gaussian kernel width are fixed.
(e (x)) (I(2) )*) UU" (0( - 1 1
(0( H )²) U (0( v²)) (0( -²) -09-UU (0(2) ($(2) ) (0(2) 1²) (8(2) TU (0(2) 11)
U u²))
(0()) "²) (I- (I-UNU?T (0(2) )
(0(2) n²) "UTUT )²)
|0 (UT((((3)) (8(() n ))
(18)
Obviously (2) T (0(2) 11
and the proof is finished.
The key in the proof of Proposition of the covariance matrix, thereby which leads to U -UT
is U +U
p I. Since U is the eigenvector matrix
U is unitary matrix and satisfies U U T
- - =
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have made clear claims the paper's contributions and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of this work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
The paper provide the full set of assumptions and a complete and correct proof for each theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have fully disclosed the information needed to reproduce the main experimental results in the appendix and submitted anonymous code for the reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have submitted anonymous code for the reproduction of the proposed algorithm this work. The adopted data public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc)
Soblic//uiddes/ooeeeubbiiss ott.eeeuuuuiiiiiooooooooioooii for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppplccguua.eeaeaecccuea
Saleee//CoSSSSbmsssS.S.. www.sssoiicgyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the experiment details have been specified in both the appendix and the released anonymous code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
The involved DNNs are pre-trained and the results of the PCA operation are deterministic.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :ttppp:://neuuree newrips........ hwt.c///bbccctthccccc..eeeeeeee.
Answer: [Yes]
Justification:
The research in the paper conforms with the NeurIPS Code of Ethics in every
aspect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the societal impacts of the work in Section 7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: No new models nor datasets are released in this work, and thus the paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the creators and original owners of assets used in the paper have been properly credited. The license and terms of use have been explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
New asserts introduced in the paper have been well documented and provided.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "Py0aCErlcy", "reviewer": "Reviewer_sSKR", "summary": "In this work, the authors leverage the framework of Kernel Principal Component Analysis (KPCA)\nfor Out-of-Distribution (OoD) detection, aiming to enhance the separability between In-Distribution\n(InD) and OoD data within the subspace defined by the principal components. The study introduces\ntwo task-specific kernels: a cosine kernel and a cosine-Gaussian kernel, both meticulously designed\nfor OoD detection. To address the computational challenges associated with large-scale kernel\nmachines, the authors employ Random Fourier Features (RFFs) to approximate the Gaussian kernel\nfunctions. This approach not only maintains the effectiveness of the kernel methods but also\npotentially reduces the time complexity of KPCA detectors, making them more feasible for practical\napplications.", "questions": "- **Evidence Against PCA Method:** The paper mentions the failure of PCA in OoD detection due to its linear mapping. Can you provide more detailed evidence or experiments that explicitly show this limitation of PCA?\n- **Extended Experimental Analysis:** Conduct a more comprehensive experimental analysis, including comparisons with the latest state-of-the-art methods in OoD detection. This will help validate the effectiveness and competitiveness of your proposed method in the context of recent advancements in the field.", "limitations": "- Manual Selection of Kernels and Parameters: One significant limitation of the proposed method is the manual selection and careful tuning of the cosine and cosine-Gaussian kernels. This process may not generalize well across different datasets and requires extensive manual intervention, which can be time-consuming and prone to human bias.\n- Comparative Analysis with Outdated Baselines: The study compares the proposed KPCA-based method with outdated OoD detection baselines. This limits the assessment of the method's competitiveness and effectiveness in the context of recent advancements in the field. Including comparisons with the latest state-of-the-art methods would provide a more accurate evaluation.\n\n---------------------------------------------------------------------------------------\nAfter Rebuttal\n\nThe author's detailed response has addressed most of my concerns. I have increased my score. However, I cannot concur with certain aspects, particularly the selection of the kernel and hyperparameter tuning. The FPR cannot serve as a tuning target during training because the OOD datasets are test data. In the setup of this study, only ID data are available for kernel selection and hyperparameter tuning.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Innovative Kernel Perspective on Existing KNN Detector: The authors creatively take a kernel perspective on an existing k-nearest neighbor (KNN) detector. This novel approach opens new avenues for improving OoD detection performance by leveraging advanced kernel methods.\n- Introduction of Cosine and Cosine-Gaussian Kernels: The proposal of a cosine kernel and a\ncosine-Gaussian kernel for Kernel Principal Component Analysis (KPCA) is a significant\ncontribution. These kernels provide alternative ways to capture non-linear patterns in the data,\nenhancing the effectiveness of the OoD detection process.\n- Utilization of Explicit Feature Mappings: The paper leverages two explicit feature mappings $\\Phi(\\cdot)$ induced from the proposed kernels on the original features $z$. This methodical approach showcases the practical application of kernel methods in transforming the feature space for improved detection accuracy.\n- Well-Written and Insightful: The paper is well-written, clearly presenting complex concepts and\nmethodologies. It provides valuable insights into the application of kernel methods for OoD\ndetection.", "weaknesses": "- **Lack of Evidence Against PCA Method:** There is insufficient evidence to substantiate the claim that the failure of the Principal Component Analysis (PCA) method in OOD detection is due to its reliance on linear mapping. Additional experiments and analyses are necessary to support this assertion.\n- **Outdated Baselines in OOD Detection:** While the experiments on ImageNet-1K show that the proposed Kernel Principal Component Analysis (KPCA) method, which explores non-linear patterns, is advantageous compared to nearest neighbor searching, it remains uncompetitive within the broader out-of-distribution (OOD) detection community. The baselines used for comparison are outdated, undermining the reliability and relevance of the results.\n- **Insufficient Demonstration of $\\ell$2-Normalization Significance:** The demonstration of the indispensable significance of $\\ell$2-normalization is inadequate. More thorough evidence and experiments are required to convincingly establish its critical role in the proposed methodology.\n- **Manual Selection of Kernels and Parameter Tuning**: The specific kernels employed in the study are manually selected and require carefully-tuned parameters. This manual selection process raises concerns about the generalizability and robustness of the proposed method, as it may not perform optimally across diverse datasets and conditions without significant manual intervention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed comments, each of which will be answered pointwisely below.  \nDue to length limit, we\n- use \"W\", \"Q\", \"L\" for Weakness, Question, Limitation,\n- put references and mentioned figures in global response and one-page rebuttal PDF file,\n- only show the average FPR in tables and put the results on each OoD dataset in global response.\n\n---\n### **(W1 & Q1) Evidence against PCA** \nThe primary goal in OoD detection is to differentiate OoD and InD data. To this end, we provide the following evidence on the significance of non-linear KPCA upon linear PCA.\n- As shown in **Fig.1(a)**, InD features from the DNN backbone are not compactly located nor easily separable w.r.t OoD features. Then, it is difficult to rely on  PCA to find a new axis system simply through linear transformation, where InD and OoD features can be well separated, since both InD and OoD features can have worse reconstruction along the axes.\n- In contrast, in **Fig.1(b)** with our KPCA-based method, InD features are located compactly. In this way, InD features can be well reconstructed along the informative axes extracted for InD, while OoD features cannot be well reconstructed with such axes and thus are easily separated from InD.\n- An ablation study is also given in **Table sSKR-1**, further verifying the significance of KPCA over PCA. \n\n**Table sSKR-1** *Comparisons on PCA and KPCA with ResNet18 on CIFAR10 as InD.*\nMethod|Average-FPR ($\\downarrow$)\n:-:|:-:|\nPCA|66.76\nKPCA w. Cosine|37.78\nKPCA w. Cosine-Gaussian|**27.34**\n\n---\n### **(W2 & Q2 & L2) Outdated Baselines**\nAs suggested, we additionally compare with four latest detection methods, SCALE [1] (ICLR'2024), DDCS [2] (CVPR'2024), ASH [3] (ICLR'2023) and DML [4] (CVPR'2023). **Table sSKR-2** shows that our method maintains superior performances and will be incorporated into next version.\n\n**Table sSKR-2** *Comparisons on latest baselines with ResNet50 on ImageNet-1K as InD, where our CoRP is set upon ReAct [5,6] similar to Table 3 in the paper.*\nMethod|Average-FPR ($\\downarrow$)\n:-:|:-:|\nSCALE (ICLR'2024)|20.05\nDDCS (CVPR'2024)|19.36\nASH-B (ICLR'2023)|22.73 \nASH-S (ICLR'2023)|22.80\nDML+ (CVPR'2023)|29.79\nOurs|**17.68**\n\n---\n### **(W3) Insufficient demonstration on $\\ell_2$-normalization significance**\nWe will elaborate the significance of $\\ell_2$-normalization, i.e., Cosine kernel, from two aspects.\n- **Promoting separability between InD and OoD features**  \n  - **Motivation.** Kernels can measure (dis)similarities between samples, e.g., a pair of InD features, such that $k(z_1,z_2)=\\phi(z_1)^\\top \\phi(z_2)$. The norms of InD features from the DNN backbone can vary drastically, even after feature mapping $\\phi$. This may pose numerical defects to kernel methods, as any two InD features $z_1,z_2$ can have varied magnitudes with very large $\\phi(z_1)^\\top\\phi(z_2)$, despite of their true (dis)similarities. This does not help reveal the intrinsic connections between samples, hindering the extraction of informative principal components of InD features. Then, both InD and OoD features can have bad reconstruction and are thereby not distinguishable.\n  -  **Numerical Tests.** In **Fig.4** and **Fig.5** of **Appendix D.1**, detailed discussions are presented. More specifically,\n       - **Fig.4 (left)** shows the inseparability of InD and OoD features without normalization.\n       - **Fig.4 (middle)** still shows the inseparability of InD and OoD features even applied with a Gaussian kernel, indicating that a non-linear mapping alone fails to capture informative patterns of InD features, nor to separate InD and OoD features.\n       - **Fig.4 (right)**  shows distinctive separability of $\\ell_2$-normalized (with Cosine kernel) InD and OoD features, indicating that $\\ell_2$-normalization is critical to bring meaningful measures in kernel methods for InD features.\n       - **Ablation studies** are provided in **Table sSKR-3** with thorough results in **Fig.5**.\n\n**Table sSKR-3** *Ablation on Cosine kernel in CoP and CoRP with ResNet18 on CIFAR10 as InD.*\nMethod|Average-FPR ($\\downarrow$)\n:-:|:-:|\nCoP|**37.78**\nCoP w/o Cosine|66.76\nCoRP|**27.34**\nCoRP w/o Cosine|94.76\n-  **Alleviating imbalanced feature norms for distinguishable reconstruction errors**  \nThe norms of InD features are commonly larger than that of OoD features, as shown in **Fig.3** in **Appendix D.1**, which was also pointed in [7]. Recall the reconstruction error in Eqn.(4), where $\\mu$ is for centering:\n$$e(z)=||U_qU_q^{\\top}(z-\\mu)-(z-\\mu)||_2\\leq||U_qU_q^{\\top}-I||_2\\cdot||z-\\mu||_2$$\nAs $||z-\\mu||_2$ of InD are commonly larger than OoD, _the reconstruction error of OoD features can still be small_, due to the imbalanced norms shown in **Fig.3**, which brings difficulty to differentiate OoD and InD through reconstruction errors. With our deployed $\\ell_2$-normalization, the issue of imbalanced norms is greatly alleviated.\n\n---\n### **(W4 & L1) Manual kernel selection and parameter tuning**\nThe choices of kernels and their tuning parameters are common in kernel methods. Our comprehensive evaluations on OoD detection show:\n- the $\\ell_2$-normalization from Cosine kernel plays a pivotal role, where the explicit feature mapping has no hyper-parameters;\n- the $\\ell_2$-distance preserving property of the shift invariant Gaussian kernel further promotes separability between InD and OoD, where only $M$ (dimensionality in RFF) and $\\gamma$ (band width) need to be tuned. Our practice shows that our default settings of $M=2m$ ($m$ is the InD feature dimension) and $\\gamma=1$ with superior performances in most cases can be a mild suggestion for practitioners.\n\nIn Sec.6, we have discussed this limitation as mentioned by the Reviewer. Nevertheless, it seems that in practical setups, the kernel choice is relatively simple and does not suffer from exhaustive tuning. We would also like to highlight that our work takes the first step of casting a kernel perspective into OoD detection, advocating more future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z6yXaDJetk", "reviewer": "Reviewer_LuQe", "summary": "The paper introduces a method that applies Kernel Principal Component Analysis (KPCA) to the output of the penultimate layer of a Deep Neural Network (DNN) for out-of-distribution (OOD) detection. Inspired by previous approaches in OOD detection that utilize K-nearest neighbors (KNN) on $\\ell_{2}$ normalized features, two kernels are designed to incorporate these features. The study explores the effectiveness of these kernels and discusses experimental results.", "questions": "1. How can one select a kernel for OOD detection beyond those discussed in this work? Please refer to my previous comment as well.\n\n2. Could you provide a mathematical justification for why KPCA with a Gaussian cosine kernel or a cosine kernel itself performs better than the KNN method with $\\ell_{2}$ normalization, as discussed in [1]?\n\n[1]Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep 358 nearest neighbors. In International Conference on Machine Learning, pages 20827–20840. 359 PMLR, 2022.", "limitations": "The authors discussed the limitations of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The problem of OOD detection is important in ML and relevant to the Neurips community. \n\n3. Many experiments are conducted and some are promising. For instance, the proposed approach using the Cosine kernel and Gaussian-Cosine kernel performs better than KNN work [1] which the method is inspired.", "weaknesses": "1. It is not clear how the proposed method can adapt to a wider variety of feature maps and effectively leverage corresponding kernels. The current feature map used in this study draws inspiration from [1] and closely resembles it and the main observation of the paper is to consider to corresponding kernel to it. As a result, the effectiveness and novelty of the kernel method in this context are not entirely clear and need to be clarified. \n\n2. The authors mention ‘kernel learning’ as a potential area for future exploration under the study's limitations. However, pursuing kernel learning in this context may appear redundant given the extensive training already performed on the DNN from scratch for the OOD detection task.\n\n[1]Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep 358 nearest neighbors. In International Conference on Machine Learning, pages 20827–20840. 359 PMLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed comments, each of which will be answered pointwisely below.\n\n---\n### **(Weakness 1 & Question 1) Selection of kernels/feature mappings**\nWe would like to firstly elaborate the effectiveness and novelty of our KPCA method (*Weakness 1*), and then discuss more kernel selections (*Question 1*).\n\n**(A1) Novelty of KPCA with the designed two effective kernels for OoD detection**  \n- The key of our KPCA detector is to find qualified kernels (or their explicit mappings) that can capture informative patterns of InD and meanwhile well distinguish InD and OoD in the mapped space.  \nFrom our extensive evaluations, we have highlighted that: *the generic KPCA method can adapt to a wide variety of kernels, but only specific designs of kernels are competent for OoD detection*, which is justified in detail in **A2** below. Thus, although the KPCA detector is not entirely new in the sense of kernel methods, *we have brought new insights into OoD detection from a kernel perspective, which is novel to the research community and the first trial on practicable and efficacious kernel methods for OoD detection*.\n- The efficiency of our KPCA detector is underlined for its cheap O(1) complexity from the reconstruction-error score in OoD inference over the O(N) of KNN. Besides, explicit feature mappings instead of expensive kernel matrix computation are deployed in our detector, resolving the inefficiency of KPCA on large-scale data.\n- Despite the used $\\ell_2$-normalization technique, our method is significantly different from KNN, which will be discussed in the response to **Question 2**.\n\n**(A2) Investigations on more kernel choices**  \nThe devoted two kernels provide the following in-depth insights about InD and OoD that should be considered in designing more viable kernels:\n- The $\\ell_2$-normalization from Cosine kernel plays a pivotal role in distinguishing OoD from InD by balancing the norm of InD and OoD features.\n- The $\\ell_2$-distance preserving property of the shift invariant Gaussian kernel can further promote the separability that benefits OoD detection performances.\n\nTherefore, when devising or learning new kernels, we should consider those kernels that can normalize feature norms or preserve the distance relationships. For example, the ablation studies in **Fig.6** of **Appendix D.2** discuss an alternative **cosine-Laplacian kernel**, where the $\\ell_1$-distance gets preserved via the Laplacian kernel. Another common **polynomial** kernel is also investigated in **Table LuQe-1**.\n\n**Table LuQe-1** *More kernel choices with ResNet18 on CIFAR10 as InD.*\nkernel|AVERAGE-FPR ($\\downarrow$)\n:-:|:-:|\nCosine (CoP)|37.78\npolynomial|79.47\nLaplacian|62.31\nGaussian|94.76\nCosine-polynomial|44.86\nCosine-Laplacian|38.78\nCosine-Gaussian (CoRP)|**27.34**\n\n\n---\n### **(Weakness 2) Limitation of learning kernels as future works**\nLearning kernels as a potential area may not be redundant for OoD detection.\n- Despite the training on DNNs from scratch, it is also a widely-accepted research paradigm to further perform learning with features from a pre-trained DNN. A typical example is the celebrated Deep Kernel Learning (DKL, [1]), where a kernel learning procedure is imposed upon the features learned from a pre-trained DNN.  \nWith particular regards to OoD detection, an additional learning step can be taken for the specified goal. Here, DKL could be considered as an alternative choice so as to pursue stronger kernels that can better characterize InD and OoD with enhanced detection performance. In contrast, we currently leverage the kernel method under typical non-parametric setups, rather than learning-based ones.\n- Although we leverage the non-parametric kernel method, it still requires a few hyper-parameters. In this work, we tune the parameters $q$, $M$ and $\\gamma$ of our proposed CoP and CoRP and use the suggested default settings according to our extensive evaluations. In future work, it would be of greater convenience and potentials, if such kernels can be learned with data available at hands.\n\n[1] Wilson, et al. \"Deep kernel learning.\" AISTATS'2016.\n\n---\n###  **(Question 2) Justification between KNN and KPCA**\nKNN relies on the discrete distance, while the analysis on KPCA comes from the feature space w.r.t Mercer kernel operators. Thus, it is difficult to align theoretical justifications between such two methods, nor to make comparisons straight away. Despite the $\\ell_2$-normalization technique, our KPCA detector has significant differences with KNN, regarding the designed kernel tricks and O(1)-complexity detection score. With particular interest to OoD detection, the following two aspects are elaborated to discuss the superiority of our KPCA detector over the KNN one.\n\n**OoD detection score design.** KNN sets the Euclidean distance between samples as the detection score, while KPCA adopts the reconstruction error along the extracted principal components. For the former, the k-th smallest Euclidean distance to InD data  determines whether a sample is OoD or not; thus, only the nearest affinity neighborhood in InD data is used, where  the full information of InD is missing in calculating the detection score. In contrast, in the latter, all InD data contributes to learning the principal components for the reconstruction; accordingly, our KPCA-based method goes beyond the affinity neighborhood in KNN, but makes full use of InD data in designing the detection score, facilitating stronger OoD detection.\n\n**Low-dimensional subspace modeling.** Another key advantage of KPCA comes from the low-dimensional subspace learned from InD data, where the most informative patterns of InD data (e.g., 99% explained variance) is kept and redundant information in InD data is removed. Such a subspace is learned with specification to InD data and is thereby less sensitive in revealing the pattern of OoD data than using the original all dimensions of KNN. Hence, OoD data can get more easily differentiated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ajXPjVvgJp", "reviewer": "Reviewer_w1ZW", "summary": "The paper everage the framework of Kernel PCA (KPCA) for OoD detection, and seek suitable non-linear kernels that advocate the separability between InD and OoD data in the subspace spanned by the principal components. Besides, explicit feature mappings induced from the devoted task specific kernels are adopted so that the KPCA reconstruction error for new test samples can be efficiently obtained within an O(1) time and memory complexity in inference.", "questions": "see above", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is written well and is easy to understand.\n\n2. The studied problem is very important.\n\n3. The results seem to outperform state-of-the-art.", "weaknesses": "1. It might be more useful to include more effective post-hoc OOD detection methods, such as ReAct, ASH for comparison. Also, the training-based approach might be useful to be included.\n\n2. It might be more useful to include the ablation results on more kernel designs. \n\n3. I am wondering why linear separable space between ID and OOD matters, if we have a non-linear classifier, it can still perform accurate OOD detection. \n\n4. What are some important assumptions here for the theory? Might be useful to provide some justifications if there are any.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed comments, each of which will be answered pointwisely below.  \nDue to length limit, we\n- use \"W\", \"Q\", \"L\" for Weakness, Question, and Limitation,\n- put the references in global response,\n- put the mentioned figures in the one-page rebuttal PDF file.\n\n---\n### **(W1) More comparisons**\nAs suggested, we add the post-hoc method ASH [3] and a training-based method DML [4] for more comprehensive comparisons, shown in **Table w1ZW-1**. Another suggested method ReAct [5] has been considered into comparisons in **Table 3** and **Table 5**. Besides, we also supplement two latest post-hoc methods SCALE [1] and DDCS [2], as suggested by *Reviewer sSKR*.\n\n**Table w1ZW-1** *Additional comparisons with more latest methods with ResNet50 on ImageNet-1K as InD data. Our CoRP detector is set upon ReAct [5,6] similar to Table 3 in the paper.*\nmethod|iNaturalist|SUN|Places|Textures|AVERAGE-FPR ($\\downarrow$)\n:-:|:-:|:-:|:-:|:-:|:-:|\nASH-P|44.57|52.88|61.79|42.06|50.32\nASH-B|14.21|22.08|33.45|21.17|22.73 \nASH-S|11.49|27.98|39.78|11.93|22.80\nDML|47.32|57.40|61.43|52.80|54.74\nDML+|13.57|30.21|39.06|36.31|29.79\nSCALE|9.50|23.27|34.51|12.93|20.05\nDDCS|11.63|18.63|28.78|18.40|19.36\nCoRP+ (ours)|10.77|18.70|28.69|12.57|**17.68**\n\n---\n### **(W2) Ablation on more kernel designs**\nIn our work, we have demonstrated the indispensable importance of the $\\ell_2$-normalization feature mapping from Cosine kernel. On such basis, upon the Cosine kernel, we further deploy a Gaussian kernel to pursue greater separability between InD and OoD, where the $\\ell_2$-distance along samples are preserved: $k_{\\rm gau}(z_1,z_2)=e^{-\\gamma||z_1-z_2||_2}$. Further investigations on more kernel designs are shown as follows. More details can also be found in **Appendix D.1 and D.2**.\n- **Cosine-Laplacian kernel.** Aside from the Gaussian kernel preserving the $\\ell_2$-distance on top on the indispensable Cosine kernel, we have considered another *Cosine-Laplacian* kernel, where the Laplacian kernel could keep the $\\ell_1$-distance: $k_{\\rm lap}(z_1,z_2)=e^{-\\gamma||z_1-z_2||_1}$. Comparisons in **Table w1ZW-2** demonstrate the superiority of Gaussian kernel, and refer to **Fig.6** of Appendix D.2 for thorough results.\n- **Cosine-polynomial kernel.** The polynomial kernel $k_{\\rm poly}(z_1,z_2)=(z_1^\\top z_2+c)^d$ is also investigated via the Tensor Sketch approximation [8] in **Table w1ZW-2**, and is shown to be unable to bring superior detection performances.\n- **Effects of individual kernels.** We have also investigated the individual effect of each kernel in **Table w1ZW-2**, where the $\\ell_2$-normalization mapping from Cosine kernel shows indispensable significance for OoD detection and the preserved $\\ell_2$-distance from Cosine-Gaussian kernel brings the best results. More thorough results can refer to **Fig.5** of Appendix D.1.\n\n**Table w1ZW-2** *Comparisons among different kernels with ResNet18 on CIFAR10 as InD data.*\nkernel|SVHN|LSUN|iSUN|Textures|Places365|AVERAGE-FPR ($\\downarrow$)\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|\nCosine (CoP)|11.56|23.24|53.71|26.28|74.11|37.78\npolynomial|46.80|97.43|95.25|61.97|95.91|79.47\nLaplacian|46.63|72.65|73.00|49.15|70.12|62.31\nGaussian|94.93|95.09|94.53|94.49|94.78|94.76\nCosine-polynomial|26.89|37.01|50.32|33.16|76.95|44.86\nCosine-Laplacian|37.80|25.58|42.20|31.05|57.28|38.78\nCosine-Gaussian (CoRP)|20.68|19.19|21.49|21.61|53.73|**27.34**\n\n---\n### **(W3) Why not non-linear classifier & why linear separability matters**\nWe would like to address this point from the following two aspects:\n- **Non-linear classifier is impractical for general OoD detection tasks.** OoD detection is an *unsupervised* task, as the DNN is trained on InD data without knowledge about OoD data. In OoD detection, OoD data can be any data of any size drawn from different distributions w.r.t InD. Thus, it is impractical to particularly train a non-linear binary classifier with sufficient OoD data being another class. Then, it is justified to consider *unsupervised* models that brings separability between InD and OoD, such as the KNN-based detector in [7], and our KPCA-based detector.\n- **Linearity separability matters.** In our work, we find that the linear principal components by simply applying PCA  are insufficient to depict the informative patterns of InD features,  as InD features are not even located compactly nor linearly inseparable with OoD features, as shown in **Fig.1(a)**. Thus, PCA cannot achieve good reconstruction performances for all InD features, so that the reconstruction of InD and OoD features along such extracted principal components from linear PCA can both be bad, failing to well differentiate OoD from InD.\n\n    Therefore, we introduce non-linear kernels with KPCA which map InD and OoD features into a new space, where the corresponding InD mappings are located more compactly and are almost linearly separable from OoD mappings. Thanks to such separability between InD and OoD, informative principal components are learned for InD and only works for InD with good reconstruction, as the separable OoD fails to be well characterized along such principal components, as shown in **Fig.1(b)**.   In this sense, good OoD detection performances are attained through the reconstructions along new axes by the extracted components in the mapped feature space with KPCA.\n\n---\n### **(W4) Important assumptions for the theory**\nImportant assumptions for the theory in the paper are listed.\n- For the RFF approximation outlined in Sec.2.2, the theory basis of RFF methods is **the Bochner's theorem**, which assumes *a continuous and shift-invariant kernel* (**line 118**). In our work, we apply RFF to approximate the Gaussian and Laplacian kernels, both of which satisfy the assumption in the Bochner's theorem. \n- For the analytical analysis in Sec.5, the two propositions have no particular assumptions, and can be applied to any standard KPCA methods under both covariance and kernel setups.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ESm0pFIYM4", "reviewer": "Reviewer_6eWv", "summary": "The paper describes a method for out-of-distribution (OOD) detection. Kernel PCA is applied to the penultimate features of a neural network. Two kernels are chosen motivated by prior work on nearest-neighbour-based detection. Namely the cosine kernel and a cosine-Gaussian kernel. In experiments, both methods outperform baseline methods on image-based OOD detection.", "questions": "- it is unclear how the proposed methods achieve O(1) time and memory complexity with RFF. Please elaborate on this.\n- line 133: how is $q$ chosen?\n- in Figure 2, please clarify what the kernel function in this experiment exactly is.\n\nSmaller things:\n- line 282: here the standard solution from [1] should be mentioned\n- line 192: wrap up\n- line 259: \"under the same framework\" unclear phrasing\n- line 299: \"we adopt\" remove \"should\"\n- for automatic learning of features from data, a suggestion is to consider Recursive Feature Machines [2].\n\nReferences:\n[1] G. H. Bakır, J. Weston, and B. Schölkopf, “Learning to find pre-images,” in Proc. Adv. Neural Inf. Process. Syst., 2004, pp. 449–456.\n\n[2] A. Radhakrishnan et al., Mechanism for feature learning in neural networks and backpropagation-free machine learning models. Science 383, 1461-1467 (2024).", "limitations": "For reproducibility the main limitation lies in the missing information about the RFF implementation as no information is provided about the number of random Fourier features. Furthermore, it is not clear what other kernel hyperparameters are to be tuned and how they are selected.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. The work is put into context, clearly motivated and the methods are explained clearly.\n- The method itself is fairly simple and works well which is highly appreciated.\n- Extensive experiments and comparisons with prior work are conducted to showcase the method's performance.", "weaknesses": "- in section 2 it is not entirely clear whether the RFF approximation is used for the the cosine-Gaussian kernel alone or also for the cosine kernel. Also please, elaborate on the details of how the kernel is approximated. \n\n- in relation to the last point, in the experiments, it is important to highlight the dimensionality of the penultimate layer features $c$ and the number of random Fourier features that you select. It is known that for medium to high-dimensional problems, the approximation with RFF requires a very high number of features.\n\n- line 198: the computational complexity of the RFF is not discussed in this paragraph.\n\n- section 2.2 is less related work and more background. Thus, sections 2.2 and 3.1 could be better summarized in a \"background\" section. \n\n- It is not entirely clear how Proposition 2 is necessary to implement CoP and CoRP. Please clarify this in the manuscript.\n\n- at the start of the paper it is unclear why you two kernels (lines 65, 68, ...). Only from the experiments and algorithm 1, it becomes clear that these are two different choices/methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed comments, each of which will be answered pointwisely below.  \nDue to length limit, we\n- use \"W\", \"Q\" for Weakness, Question,\n- put the references in global response,\n- put the mentioned figures in the one-page rebuttal PDF file.\n\n---\n### **(W1) The usage of RFF approximation**\n- For the *Cosine* kernel, the RFF approximation is *not* needed, since the Cosine kernel $k_{\\rm cos}(z_1,z_2)=\\frac{z_1^\\top z_2}{||z_1||\\_2||z_2||\\_2}$ has its exact feature mapping $\\phi_{\\rm cos}(z)=\\frac{z}{||z||\\_2}$, such that $k_{\\rm cos}(z_1,z_2)=\\phi_{\\rm cos}(z_1)^\\top\\phi_{\\rm cos}(z_2)$.\n- For the *Cosine-Gaussian* kernel, _the RFF is adopted to only approximate the Gaussian kernel part_, i.e.,  $k_{\\rm gau}(z_1,z_2)\\approx\\phi_{\\rm RFF}(z_1)^\\top\\phi_{\\rm RFF}(z_2)$ in Eqn.(2) in the paper. Specifically, the adopted feature mapping $\\phi_{\\rm RFF}(\\phi_{\\rm cos}(z))$ for the entire *Cosine-Gaussian* kernel is a composite of the exact feature mapping $\\phi_{\\rm cos}$ for the Cosine kernel part and the approximate RFF mapping $\\phi_{\\rm RFF}$ for the Gaussian kernel $k_{\\rm gau}$ part.\n\n---\n### **(W2) Dimensionality of penultimate layer features $m$ and the number of RFFs $M$**\nThe settings of  $m$ and $M$  are provided in **Table 6eWv-1**, where $M$ is  set as $4m$ for the CIFAR10 InD and $2m$ for the ImageNet-1K InD for all experiments in the paper. \n\n**Table 6eWv-1** *The values of $m$ and $M$*\ndataset|network|feat. dim. $m$|num. of RFFs $M$\n:-:|:-:|:-:|:-:|\nCIFAR10|ResNet18|512|2048\nImageNet-1K|ResNet50|2048|4096\nImageNet-1K|MobileNet|1280|2560\n\nIndeed, as $M$ increases, RFF can better approximate the Gaussian kernel. We have conducted sensitivity analysis on $M$ in **line 605 and Fig.9 of Appendix D.3**, showing that increasing $M$ boosts and then maintains rather steady OoD detection performances. Hence, we note that _a substantially high number of RFF is not of significant necessity in the addressed task_. To balance efficiency and effectiveness, we adopt $M$ as in **Table 6eWv-1** with SOTA results under varied setups.\n\n---\n### **(W3) Computation complexity of RFF**\nWith Eqn.(2), for the RFF mapping $\\phi_{\\rm RFF}$, $2M$ samplings for $\\omega_i$ and $u_i$ respectively are required, and $\\phi_{\\rm RFF}$ includes $M$ dot products and $M$ additions. Accordingly, the memory and time complexity of RFF is ${\\cal O}(M)$.\n\n---\n### **(W4) Organization of Sec.2.2 and Sec.3.1**\nThanks for the nice suggestion. We will merge Sec.2.2 and Sec.3.1 as a 'background' section in the revised version.\n\n---\n### **(W5) Proposition 2 and the implementation of CoP and CoRP**\nProposition 2 supplements the analysis for kernel implementation of CoP and CoRP.\n\nCoP and CoRP leverage the covariance-based KPCA and are implemented via explicit feature mappings induced from the kernel, rather than via the more expensive kernel implementation. Proposition 2 provides the reconstruction errors when implemented with kernels, and serves as a supplementary analysis on our more efficient covariance-based CoP and CoRP.\n\n---\n### **(W6) Elaboration on the two kernels**\nThe proposed OoD detector is a KPCA-based method with **two alternative choices of kernels** explained in the main context, i.e., Cosine kernel and Cosine-Gaussian kernel.\n\n---\n### **(Q1) How O(1) complexity is achieved with RFF**\nThanks for the valuable comment. To align with the complexity analysis of KNN detector [7], the O(1) time and memory complexity in our work also refers to the inference procedure of calculating the OoD detection score, i.e., the reconstruction error, when given with features, so it indeed does not include the computation of the mapping. \nRegarding the raised comment in **W3**, we specify the complete computation complexity of CoP and CoRP:\n- For CoP without RFF involved, the time and memory complexity in inference is  O(1), since the feature mapping $\\phi_{\\rm cos}$ is obtained right away as the normalized $z$ without additional cost.\n- For CoRP with RFF involved, when considering the computation of the feature mapping  $\\phi_{\\rm RFF}(\\phi_{\\rm cos}(\\cdot))$, we have the complete time and memory complexity in inference as ${\\cal O}(M)$, since $\\phi_{\\rm RFF}$ requires an ${\\cal O}(M)$ complexity.\n\n---\n### **(Q2) How to choose $q$**\nIn CoP and CoRP, $q$ is selected as the number of principal components sufficing a certain explained variance ratio (e.g., 99%), which is also a common technique in (K)PCA-based methods. For example, given the ResNet50 on ImageNet-1K with a penultimate feature dimension $m=2048$, to preserve an explained variance ratio of 99%,  $q=1346$ principal components are required in CoP.\n\nWe have conducted sensitivity analysis on $q$ in CoP and CoRP to comprehensively evaluate its effect. More details can refer to **line 587 and Fig.7 in Appendix D.3**.\n\n---\n### **(Q3) Kernel function in Fig.2**\nIn the left panel of Fig.2, the adopted kernel function is the Cosine kernel function $k_{\\rm cos}$. In the right panel of Fig.2, the adopted kernel function is the Gaussian function $k_{\\rm gau}$ on $\\ell_2$-normalized features, i.e., Cosine-Gaussian kernel.\n\n---\n### **(Q4) Smaller things**\n- **Typos and literatures.** We will revise all the typos and add the recommended literature [9] in the updated version. Also, we will consider the suggested Recursive Feature Machines [10] in learning algorithms for OoD detection in future works.\n- **Unclear phrasing.** Regarding the 'under the same framework' in line 259, it indicates that our KPCA reconstruction error and the regularized reconstruction error in [6] are evaluated under the same fusion trick as [6] for a fair comparison.\n\n---\n### **Limitation on reproducibility**\nThe source code for reproduction has been provided in the supplementary material, and will also be released publicly in github upon acceptance. In **Appendix D.3**, all  hyper-parameters of CoP and CoRP are discussed in detail and evaluated with ablation studies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Program Chairs, Area Chairs, and Reviewers,

First of all, we would like to thank you for your time, constructive suggestions, which greatly help us improve the work. In this global response, we
- provide full results of tables in the response to Reviewer **sSKR**,
- gather the shared references in the responses to Reviewers **sSKR**, **w1ZW** and **6eWv**,
- put the figures mentioned in responses in the one-page PDF for reference.

---
### **Full results of Tables sSKR-1, sSKR-2 and sSKR-3**

**Table sSKR-1** *Comparisons on PCA and KPCA. The FPR values of each OoD data set are presented with ResNet18 trained on CIFAR10 as InD data.*
Method|SVHN|LSUN|iSUN|Textures|Places365| Average-FPR ($\downarrow$)
:-:|:-:|:-:|:-:|:-:|:-:|:-:|
PCA|30.22|78.15|85.88|46.29|93.27|66.76
KPCA w. Cosine|11.56|23.24|53.71|26.28|74.11|37.78
KPCA w. Cosine-Gaussian|20.68|19.19|21.49|21.61|53.73|**27.34**

---
**Table sSKR-2** *Comparisons on the supplemented recent baseline methods. The FPR values of each OoD data set are presented with ResNet50 trained on ImageNet-1K as InD data. Our CoRP+ detector is set upon ReAct [5,6] and the results can refer to Table 3 in Sec.4.2 of the manuscript.*
method|iNaturalist|SUN|Places|Textures|AVERAGE-FPR ($\downarrow$)
:-:|:-:|:-:|:-:|:-:|:-:|
SCALE (ICLR'2024)|9.50|23.27|34.51|12.93|20.05
DDCS (CVPR'2024)|11.63|18.63|28.78|18.40|19.36
ASH-P (ICLR'2023)|44.57|52.88|61.79|42.06|50.32
ASH-B (ICLR'2023)|14.21|22.08|33.45|21.17|22.73
ASH-S (ICLR'2023)|11.49|27.98|39.78|11.93|22.80
DML (CVPR'2023)|47.32|57.40|61.43|52.80|54.74
DML+ (CVPR'2023)|13.57|30.21|39.06|36.31|29.79
CoRP+ (ours)|10.77|18.70|28.69|12.57| **17.68**

---
**Table sSKR-3** *Ablation on Cosine kernel in CoP and CoRP. The FPR values of each OoD data set are presented with ResNet18 trained on CIFAR10 as InD data.*
Method|SVHN|LSUN|iSUN|Textures|Places365|Average-FPR ($\downarrow$)
:-:|:-:|:-:|:-:|:-:|:-:|:-:|
CoP|11.56|23.24|53.71|26.28|74.11|**37.78**
CoP w/o Cosine|30.22|78.15|85.88|46.29|93.27|66.76
CoRP|20.68|19.19|21.49|21.61|53.73|**27.34**
CoRP w/o Cosine|94.93|95.09|94.53|94.49|94.78|94.76

---
### **References throughout the responses to Reviewers sSKR, w1ZW and 6eWv**

[1] Xu, et al. Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement. ICLR'2024.  
[2] Yuan, et al. Discriminability-Driven Channel Selection for Out-of-Distribution Detection. CVPR'2024.  
[3] Djurisic, et al. Extremely Simple Activation Shaping for Out-of-Distribution Detection. ICLR'2023.  
[4] Zhang, et al. Decoupling maxlogit for out-of-distribution detection. CVPR'2023.  
[5] Sun, et al. React: Out-of-distribution detection with rectified activations. NeurIPS'2021.  
[6] Guan, et al. Revisit pca-based technique for out-of-distribution detection. ICCV'2023.  
[7] Sun, et al. Out-of-distribution detection with deep nearest neighbors. ICML'2022.  
[8] Pham, et al. Fast and scalable polynomial kernels via explicit feature maps. KDD'2013.  
[9] Bakır, et al. Learning to find pre-images. NeurIPS'2004.  
[10] Radhakrishnan, et al. Mechanism for feature learning in neural networks and backpropagation-free machine learning models. Science 2024.

Meta-review: This paper proposes to use a kernel PCA for OOD with deep neural net features. As a choice of kernel, this study suggests to use a cosine kernel and a cosine-Gaussian kernel for the OOD tasks. In addition to that, the usage of Random Fourier Features (RFFs) is also suggested to reduce the computational complexity. The practical effectiveness of the proposed method is examined through several numerical experiments.   

Although the usage of kernel PCA is not very novel, the authors enhanced their proposal by several numerical experiments. In addition to the main submission, they conducted some more ablation studies in the rebuttal to address reviewers' concerns.  
The authors experimentally demonstrated the effectiveness of the proposed method against the existing nearest neighbor method, which would be informative to the community.  

This paper was quite on the borderline, but based on the discussions with the reviewers after rebuttal, I would like to recommend acceptance of this paper.
Decision: Accept (poster)