Kernel PCA for Out-of-Distribution Detection

Authors: kun  fang
qinghua  tao
kexin  lv
mingzhen  he
xiaolin  huang
jie  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 04:27:48

Abstract: Out-of-Distribution (OoD) detection is vital for the reliability of Deep Neural Networks (DNNs).
Existing works have shown the insufficiency of Principal Component Analysis (PCA) straightforwardly applied on the features of DNNs in detecting OoD data from In-Distribution (InD) data.
The failure of PCA suggests that the network features residing in OoD and InD are not well separated by simply proceeding in a linear subspace, which instead can be resolved through proper non-linear mappings.
In this work, we leverage the framework of Kernel PCA (KPCA) for OoD detection, and seek suitable non-linear kernels that advocate the separability between InD and OoD data in the subspace spanned by the principal components.
Besides, explicit feature mappings induced from the devoted task-specific kernels are adopted so that the KPCA reconstruction error for new test samples can be efficiently obtained with large-scale data.
Extensive theoretical and empirical results on multiple OoD data sets and network structures verify the superiority of our KPCA detector in  efficiency and efficacy with state-of-the-art detection performance.

Main paper: 
 introduction: 
 With the rapid advancement of the powerful learning abilities of Deep Neural Networks (DNNs) [1, 2] , the trustworthiness of DNNs in security-sensitive scenarios has attracted considerable attention in recent years [3] . Generally, samples from the training and test sets of DNNs are viewed as data from some In Distribution (InD) P in , while samples from other data sets are regarded as coming from a different distribution P out , i.e., out-of-distribution (OoD) data. In the practical deployment, DNNs trained on InD data would inevitably encounter OoD data and thus yield unreliable results with potential risks. Therefore, detecting whether a new sample is from P in or P out has been a valuable research topic of trustworthy deep learning, namely OoD detection [4] .Existing OoD detection methods exploit different outputs from DNNs to unveil the hidden disparities between InD and OoD data, e.g., logits [5] , gradients [6] and features [7, 8] . In this work, we address OoD detection from a perspective of utilizing the feature spaces learned by the backbone of DNNs. To be specific, given a DNN f : R d → R c , f takes x ∈ R d as inputs and learns penultimate layer features z ∈ R m before the last linear layer. Principal Component Analysis (PCA) is investigated in [8] to calculate the reconstruction error in the z-space as the OoD detection score. That is, PCA is executed on the penultimate features of InD training samples and learns a linear subspace spanned by the principal components in the z-space. Then, given features ẑ of an unknown sample x, one can obtain the reconstructed counterpart of ẑ by projecting ẑ to the linear subspace and re-projecting it back. The reconstruction error is computed as the Euclidean distance between ẑ and the reconstructed counterpart. In [9] , similar ideas are adopted with energy-based models using an Figure 1 : The t-SNE [10] visualization on the original features z (left) and the mapped features Φ(z) (right). Our KPCA detector alleviates the linearly inseparability between InD and OoD features in the original z-space via an explicit feature mapping Φ, and thus substantially improves the OoD detection performance, illustrated by the much more distinguishable reconstruction errors.auto-encoder structure, where the neural networks are trained from scratch and the reconstruction is conducted in the decoder. For good OoD detection performance, it is expected that InD features are compactly allocated along the linear principal components with high variances for capturing most of the informative patterns of InD data, leading to small reconstruction errors, while OoD features are not supposed to be well matched with the learned subspace, causing large reconstruction errors.However, it has been empirically observed in [8] that such PCA reconstruction errors alone cannot distinctively differentiate OoD data from InD data, leading to poor detection performance of PCA in the z-space. Nevertheless, [8] did not take further explorations on the reasons behind, and instead proposed a practical fusion trick to boost PCA by multiplying with other existing powerful detection scores. Therefore, in this work, more in-depth analyses are undertaken to improve the limitations of PCA for OoD detection with insightful understandings on the distribution of InD and OoD features.It is widely acknowledged that PCA falls inferior in dealing with those linearly-inseparable data, which inspires us to explore the non-linearity existing in the z-space of InD and OoD features under the help of the celebrated Kernel PCA (KPCA).KPCA has long been a powerful technique in learning the non-linear patterns of data [11, 12] . By deploying KPCA, a non-linear feature mapping Φ is imposed on the z-space in this setup, so that the linear inseparability can be alleviated in the mapped Φ(z)-space. KPCA is generally conducted through a kernel function k induced by the feature mapping, i.e., k(z 1 , z 2 ) = ⟨Φ(z 1 ), Φ(z 2 )⟩, in order to avoid explicit calculations in the mapped Φ(z)-space. In most cases, researchers have no prior on the unknown non-linear data distribution, e.g., the non-linearity in the z-space of InD and OoD features. Therefore, finding an appropriate k or Φ that well adapts the data always remains a non-trivial issue for KPCA. For example, the mostly common Gaussian kernel is shown to be unable to separate the InD and OoD features, and leads to terrible OoD detection performance, see details in Appendix C. In addition, KPCA also faces the challenge of calculating and storing the N tr × N tr kernel matrix on millions of training samples with a very large size N tr , which significantly hinders its application in practical tasks with a huge amount of data.The proposed KPCA detection method well addresses the aforementioned issues of KPCA for OoD detection. On the one hand, to better understand the non-linear patterns in InD and OoD features, we take a kernel perspective on an existing OoD detector [7] , which searches k-th Nearest Neighbors (KNN) on the ℓ 2 normalized features z. By decoupling and analyzing key components of the KNN method, we acquire two effective kernels, a cosine kernel and a cosine-Gaussian kernel, for our KPCA detector to promote the linear separability between InD and OoD features in the subspace of the principal components, leading to substantially improved distinguishable KPCA reconstruction errors, as shown in Figure 1 . On the other hand, for a computationally-friendly implementation, two explicit feature mappings Φ induced from the cosine and cosine-Gaussian kernels are executed on the original features z, followed by PCA in the mapped Φ(z)-space to obtain the reconstruction errors without calculations on the kernel matrix. Specifically, the celebrated Random Fourier Features (RFFs) [13] are introduced to approximate the Gaussian kernel, allowing an O(M ) computation complexity in inference, which significantly outperforms the O(N tr ) computation complexity of the KNN [7] and the kernel-matrix-based KPCA (M is the number of RFFs and M ≪ N tr ).Extensive experiments verify the effectiveness of the devoted two kernels, which we hope could bring inspirations for the research community in exploring the non-linearity in InD and OoD data from a kernel perspective. For example, the two kernels can even serve as a beneficial prior on advocating learning more and stronger kernels for OoD detection. In addition, we supplement our method with its implementation via the kernel matrix, and illustrate the advantageous effectiveness and efficiency of explicit feature mappings in Section 6. The contributions of this work are summarized below:• To the best of our knowledge, this is the first work that explores suitable kernels to seize the non-linearity in InD and OoD features in the post-hoc stage on well-trained DNNs.• Two task-specific kernels are carefully devised for OoD detection. Particularly, two explicit feature mappings induced from the kernels are adopted for the KPCA detector, and lead to separable KPCA reconstruction errors with significantly-reduced complexity in inference.• Theoretical and experimental comparisons indicate the effects of kernels in our KPCA detector with SOTA detection performance and remarkably reduced time complexity.In the remainder of this work, related works and research backgrounds are outlined in Section 2 and Section 3, respectively. Section 4 delves into details of the proposed KPCA detector. Comparison experiments with prevailing OoD detection methods and KPCA via kernel functions are presented in Section 5 and Section 6, respectively. Conclusions and limitations are drawn in Section 7. 
related work: 
 Generally, out-of-distribution detection has been formulated as a binary classification problem including a decision function D(•) and a scoring function S(•):EQUATIONThe scoring function S(•) assigns a score S(x) for a new sample x. If S(x) is greater than a threshold s, the decision function D(•) would view x as an InD sample, and vice versa. The key to effectually detecting OoD samples is a well-designed scoring function. Existing OoD detectors adopt different outputs from DNNs and design justified scores to measure the disparity between InD and OoD data.Logits-based detectors exploit the abnormal responses reflected in the predictive logits or probabilities from DNNs to detect OoD data. Typical methods adopt either the maximum logits [14] or probability [15, 16] or the energy function on logits [5] as the detection score.Gradients-based methods investigate differences on gradients w.r.t InD and OoD data for OoD detection. For example, gradient norms [6] or low-dimensional representations [17] are studied to devise the detection score.Features-based detectors try to capture the feature information causing over-confidence of OoD predictions in different ways. Feature clipping [18, 19, 20, 21, 22] , feature distances [23, 7, 24] , feature norms [25] , rank-1 features [26] , feature subspace [8] , etc., have been explored with excellent performance.Aside from methods above, other existing OoD detectors cover the training regularization [27, 28] , the ensemble technique [29] and theoretical understandings [30, 31] . Refer to Appendix A for more detailed descriptions on the compared methods in experiments [32, 33, 34, 35] .3 Background 
pca for ood detection: 
 The PCA detector with the reconstruction error as the detection score is summarized firstly. Given the penultimate features z i ∈ R m learned by a well-trained DNN f :EQUATIONThrough the eigendecomposition Σ = U ΛU ⊤ , the dimensionality reduction matrix U q ∈ R m×q is obtained by taking the first q columns of the eigenvector matrix U w.r.t the top-q largest eigenvalues.In inference, given a new sample x ∈ R d and its feature ẑ ∈ R m from the DNN f , the reconstruction error is computed as:EQUATIONBy projecting centralized (ẑ -µ) to the U q -subspace and re-projecting back, we can obtain the reconstructed features U q U ⊤ q (ẑ -µ) and the reconstruction error e(x), which then can be set as the OoD detection score: S(x) = -e(x). An ideal case is that U q contains informative principal components of InD data and causes projections of OoD data far away from that of InD data, leading to separable reconstruction errors between OoD and InD data.A concise description is firstly given on the Random Fourier Features (RFFs) [13] , which will be adopted in our method later. RFFs are proposed to approximate the kernel function so as to alleviate the heavy computation cost in large-scale kernel machines. In kernel methods, an N ×N kernel matrix w.r.t N samples requires O(N 2 ) kernel manipulations, O(N 2 ) space complexity and O(N 3 ) time complexity to calculate the inverse of the kernel matrix, which leads to overwhelmed computation costs for a large data size N . Therefore, RFFs are introduced by building an explicit feature mapping to directly approximate the kernel function for efficient kernel machines on large-scale data.RFFs are built on the Bochner's theorem [36] : A continuous and shift-invariant kernelk(z 1 , z 2 ) = k(z 1 -z 2 ) on R m is positive definite if and only if k(•)is the Fourier transform of a non-negative measure. An explicit feature mapping ϕ RFF induced from the kernel k is derived in [13] :EQUATIONwhere ω = e -γ∥z1-z2∥ 2 2 is ω ∼ N (0, √ 2γI m ). Such a feature mapping ϕ RFF satisfies k gau (z 1 , z 2 ) ≈ ϕ RFF (z 1 ) ⊤ ϕ RFF (z 2 )and is known as the random Fourier features (RFFs) mapping. Refer to [13] for a detailed convergence analysis. RFFs have been widely utilized in kernel learning [37] , optimization [38] , etc. 
methodology: 
 As empirically observed in [8] , the aforementioned PCA reconstruction error in the z-space is not an effective score in detecting OoD data from InD data. We propose that the reason behind is possibly due to the linearly-inseparable features of InD and OoD data, as shown in Figure 1a . To address this issue, we propose to explore the non-linearity in z-space via kernel PCA. Then, through a kernel perspective on an existing KNN detector [7] , we put forward two efficacious kernels that well characterize the non-linear patterns in z-space of InD and OoD data: a cosine kernel (Section 4.1) and a cosine-Gaussian kernel (Section 4.2). Particularly, we adopt two explicit feature mappings Φ induced from the two kernels, and execute PCA in the mapped Φ(z)-space, which leads to an informative principal subspace and distinct reconstruction errors for efficacious OoD detection.In the KNN detector [7] , the nearest neighbor searching is executed on the ℓ 2 -normalized penultimate features, i.e., z ∥z∥2 . In inference, given a new sample x, its feature ẑ is firstly normalized as ẑ ∥ẑ∥2 , then the negative of its (k-th) shortest ℓ 2 distance to the ℓ 2 -normalized features zi ∥zi∥2 of training data is set as the detection score:EQUATIONThe ablations in KNN demonstrate the indispensable significance of the ℓ 2 -normalization: the nearest neighbor searching directly on z shows a notably drop in detection performance. The critical role of the ℓ 2 -normalization in KNN attracts our attention in the sense of kernel. From a kernel perspective, the ℓ 2 -normalization is exactly the non-linear feature mapping ϕ cos inducing the cosine kernel k cos :EQUATION)It indicates that a justified ϕ cos (z)-space with such non-linear mapping, instead of the original z-space, contributes to the success of nearest neighbor searching in detecting OoD.Notice that the key of KPCA for OoD detection lies in an appropriate non-linear feature space that captures the non-linearity in InD and OoD features, either through the kernel k or the associated explicit feature mapping Φ. Motivated by the KNN detector, we apply Φ(•) ≜ ϕ cos (•) as the feature mapping in KPCA to introduce non-linearity. Then, PCA is executed on mapped features ϕ cos (z), following the procedures described in Section 3.1. All the features z are now mapped to Φ(z) to formulate the covariance matrix Σ Φ , for computing non-linear principal components with matrix U Φ q and the corresponding reconstruction error e Φ . This detection scheme is dubbed as CoP (Cosine mapping followed by PCA), as shown in Algorithm 1. An in-depth analysis on the effect of the normalization of the cosine kernel is left in Appendix C.1.The success of KNN (Equation ( 5)) suggests that the ℓ 2 distance on z ∥z∥2 is effective in distinguishing OoD data from InD data. In other words, the ℓ 2 distance relation between samples in the ϕ cos -space preserves useful information that benefits the separation of OoD data from InD data. This motivates us to seek non-linear feature spaces that can retain the ℓ 2 distance relation. Hence, we propose to introduce KPCA with non-linearity built upon ϕ cos (z), through which the useful ℓ 2 distance in ϕ cos -space can be preserved to further separate InD and OoD data.In this regard, we deploy the shift-invariant Gaussian kernel to keep the ℓ 2 distance information:EQUATIONThe feature mapping associated with k gau is infinite-dimensional, but it can be efficiently approximated through random Fourier features (RFFs, [13] ), i.e., ϕ RFF defined in Equation (4) . In this way, the inner product of two mapped samples ϕ RFF (z 1 ) ⊤ ϕ RFF (z 2 ) provides the approximate Gaussian kernel, so that we can leverage the RFFs mapping ϕ RFF to preserve the ℓ 2 distance information through the Gaussian kernel.Hence, a cosine-Gaussian kernel is adopted for OoD detection, as the Gaussian kernel k gau (or ϕ RFF ) is imposed on top of the cosine kernel k cos (or ϕ cos ), further exploiting the ℓ 2 distance relationships beyond the ϕ cos -space for OoD detection. As we work with the explicit feature mapping, the nonlinearity to z is achieved by Φ(•) ≜ ϕ RFF (ϕ cos (•)). With mappings ϕ RFF (ϕ cos (z)), PCA is then executed to compute the reconstruction errors for OoD detection. This detection scheme is dubbed as CoRP (Cosine and RFFs mappings followed by PCA). Algorithm 1 illustrates the complete procedure of the proposed CoP and CoRP for OoD detection. Alternative choices for more kernels are exploited in Appendix C.2.To warp up, we devise two effective feature mappings induced from a cosine kernel and a cosine-Gaussian kernel to promote the separability of InD data and OoD data in non-linear feature spaces, inspired by effectiveness of the ℓ 2 normalization and the ℓ 2 distance from a kernel perspective on the KNN detector [7] . Our proposed two feature mappings well characterize the non-linearity in penultimate features z of DNNs between InD and OoD data, enabling PCA to extract an informative subspace w.r.t the mapped features through principal components and the reconstruction errors. Φ(•) ← ϕ cos (•), ϕ cos (z) = z ∥z∥2 . 3: else if CoRP then 4: Sampling ω i ∼ N (0, √ 2γI m ), i = 1, • • • , M . 5: Sampling u i ∼ U(0, 2π), i = 1, • • • , M . 6: Φ(•) ← ϕ RFF (ϕ cos (•)). 7: end if 8:Calculating the covariance matrix in the mapped Φ(z)-space:Σ Φ = Ntr i=1 (Φ(z i ) -µ Φ )(Φ(z i ) -µ Φ ) ⊤ , µ Φ = 1Ntr Ntr i=1 Φ(z i ). 9: Applying eigendecomposition: Σ Φ = U Φ Λ Φ U Φ⊤ . 10: Taking the first q columns of U Φ w.r.t the top-q largest eigenvalues in Λ Φ : U Φ q = U Φ [:, : q]. Ensure: Dimensionality-reduction matrix U Φ q . 11: Given a new sample x and its features ẑ. 12: Calculating the reconstruction error:e Φ (x) = U Φ q U Φ⊤ q (Φ(ẑ) -µ Φ ) -(Φ(ẑ) -µ Φ ) 2 .Ensure: Reconstruction error e Φ .Computation complexity In our method, given any new sample x with the penultimate features ẑ in inference, to compute the reconstruction error e Φ (x), we only need the feature mapping Φ, the projection matrix U Φ q and the mean mapped training feature vector µ Φ . Both U Φ q and µ Φ can be pre-calculated and stored in preparation for inference. Therefore, the entire computation cost of CoP and CoRP comes from the construction of the explicit feature mapping Φ on new features ẑ.• For CoP, its feature mapping ϕ cos is an in-place operation and does not require additional computations. Therefore, the time and memory complexity of CoP is O(1).• For CoRP, the feature mapping ϕ RFF of the Gaussian kernel requires 2M samplings for ω i and u i , respectively, and M dot-products and M additions. Accordingly, the time and memory complexity of CoRP is O(M ).In contrast, regarding the Equation ( 5 In the following, Section 5 exhibits the SOTA performance of our KPCA detector over multiple prevailing detection methods. Section 6 gives an analytical discussion between our covariance-based KPCA and classic KPCA via kernel functions for OoD detection. Due to space limitation, more in-depth investigations on the kernel properties are left in Appendix C, covering ablation studies (Appendix C.1), alternative kernels (Appendix C.2) and sensitivity analysis (Appendix C.3).Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We have fully disclosed all the information needed to reproduce the main experimental results in the appendix and submitted anonymous code for the reproduction. 
experiments on ood detection: 
 In experiments, our KPCA-based detectors, CoP and CoRP, are firstly compared with KNN [7] in Section 5.1, and show stronger detection performance and cheaper computation costs. In Section 5.2, CoP and CoRP are further compared with the regularized PCA reconstruction error [8] , and achieve SOTA OoD detection performance over various prevailing methods. The source code of this work has been publicly released 1 . All the experiments are executed on 1 NVIDIA GeForce RTX 3090 GPU.Datasets Experiments are executed on the commonly-used small-scale CIFAR10 [39] and largescale ImageNet-1K benchmarks [40] , following the settings in [7, 8] . For CIFAR10 as InD, OoD data sets include SVHN [41] , LSUN [42] , iSUN [43] , Textures [44] and Places365 [45] . For ImageNet-1K as InD, OoD data sets contain iNaturalist [46] , SUN [47] , Places [45] and Textures [44] . Metrics For the evaluation metrics, we employ the commonly-used (i) False Positive Rate of OoD samples with 95% true positive rate of InD samples (FPR), and (ii) Area Under the Receiver Operating Characteristic curve (AUROC). The average FPR95 and AUROC values over the selected multiple OoD data sets are viewed as the final comparison metrics.The comparisons with KNN [7] cover both the benchmarks. Following the setups in KNN, for fair comparisons, we evaluate models trained via the standard cross-entropy loss and models trained via the supervised contrastive learning [49] , and adopt the same checkpoints released by KNN: ResNet18 [50] on CIFAR10 and ResNet50 on ImageNet-1K. Here, the scoring function of CoP and CoRP is In addition, KPCA also outperforms KNN on the CIFAR10 benchmark with improved OoD detection performances, which we leave to Appendix B for more details.S(x) = -e Φ (x) in Algorithm 1.In [8] , to alleviate the weak detection performance of PCA reconstruction error e(ẑ) of Equation ( 3), the authors proposed to regularize e(ẑ) by the feature norm ∥ẑ∥ 2 and a fusion strategy to boost its detection performance by introducing existing OoD scores. Firstly, the regularized reconstruction error e reg (ẑ) is calculated in the original z-space as : e reg (ẑ) =∥U q U ⊤ q (ẑ-µ)-(ẑ-µ)∥2 ∥ẑ∥ 2. Then, the authors claimed that such a regularized version e reg (ẑ) is still insufficient for OoD detection, and designed a fusion strategy to combine e reg with other existing OoD scores. For example, to fuse e reg with the Energy [5] score, the final scoring function is S(x) = (1 -e reg (ẑ)) • S Energy (ẑ).In this section, we show that our KPCA reconstruction error e Φ outperforms the regularized PCA reconstruction error e reg under the same fusion framework. Following the settings in [8] , for a fair comparison, the fused OoD scores include MSP [15] , Energy [5] , ReAct [18] and BATS [19] . The detection experiments are executed on the ImageNet-1K benchmark with pre-trained ResNet50 and MobileNet [51] checkpoints from PyTorch [52] .Table 3 presents the comparisons between [8] and ours on the ImageNet-1K benchmark of ResNet50. When fused with MSP, Energy and BATS, both the KPCA-based CoP and CoRP outperform the regularized reconstruction error [8] on almost all the OoD data sets with substantially improved FPR and AUROC values. Specifically, when fused with the ReAct method [18] , the CoRP achieves new SOTA OoD detection performance among various prevailing detectors. Experiments on MobileNet also show superior performance of CoP and CoRP, see details in Appendix B.All these experiment results indicate that an appropriately mapped Φ(z)-space benefits the OoD detection, as the non-linearity in z-space gets alleviated by the feature mapping Φ. Our work provides 2 viable selections for Φ with empirical validations, which we hope could attract attentions towards the non-linearity in InD and OoD features for the research community from a kernel perspective.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We have submitted anonymous code for the reproduction of the proposed algorithm in this work. The adopted data is public.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
analytical discussions with kpca via kernel functions: 
 In CoP and CoRP, KPCA is executed with the covariance matrix of mapped features Φ(z). In contrast, in the classic KPCA [11, 12] , such feature mappings Φ are not explicitly given, and it rather works with a kernel function applied to original features z. In this section, we supplement our covariance-based KPCA with its kernel function implementation, including theoretical discussions and empirical comparisons on OoD detection. Our CoP and CoRP are shown to be more effective and efficient than their counterparts that employ kernel functions.In the classic KPCA, the kernel trick enables projections to the principal subspace via kernel functions without calculating Φ. However, how to map the projections in the principal subspace back to the original z-space remains a non-trivial issue, known as the pre-image problem [53] , which makes it problematic to calculate reconstructed features via kernel functions. To address this issue, the following Proposition 1 shows a flexible way to directly calculate reconstruction errors without building reconstructed features, so as to apply the kernel trick, shown in the subsequent Proposition 2. Proposition 1. The KPCA reconstruction error e Φ (ẑ) can be represented as the norm of features projected in the residual subspace, i.e., the U Φ p -subspace withEQUATION)Proposition 1 implies that the reconstruction error equals to the norm of projections in the residual U Φ p -subspace, i.e., the subspace consisting of those principal components that are not kept, see proofs in Appendix D. Accordingly, as typically done in the classic KPCA, we can introduce a kernel function to perform dimension reduction, but to the residual subspace, and then calculate the norms of the reduced features as the reconstruction error, illustrated by Proposition 2.Given a kernel function k(•, •) : R m × R m → R, we have a kernel matrix K ∈ R Ntr×Ntr on training data with K i,j = k(z i , z j ), and a vector k ẑ ∈ R Ntr with the i-th element k(z i , ẑ) for a new sample ẑ. Proposition 2 shows how to calculate the KPCA reconstruction error via the kernel function k. Proposition 2. The KPCA reconstruction error e k (ẑ) w.r.t a kernel function k can be calculated as:EQUATIONwhere A ∈ R Ntr×l includes l eigenvectors of the kernel matrix K w.r.t the top-l smallest eigenvalues.According to Proposition 2, now CoP and CoRP can be implemented via kernel functions. For CoP, we just directly apply the cosine kernel function k cos on features z to compute the kernel matrix K and the projection matrix A, so as to obtain e k (ẑ) following Equation (9) . For CoRP, we should adopt the Gaussian kernel function k gau on the ℓ 2 -normalized inputs z ∥z∥ 2 to calculate K, A and e k (ẑ). Figure 2 shows comparisons on the detection performance between CoP/CoRP and their kernel function implementations.In Figure 2 , the detection performance of KPCA with kernel functions is evaluated by varying the explained variance ratio of the kernel matrix K. The larger the explained variance ratio, the smaller the dimension l of A. The best detection results achieved by CoP/CoRP are illustrated as the dashed lines. Clearly, regarding the OoD detection performance, reconstruction errors e k calculated by kernel functions are less effective than those calculated explicitly in the mapped Φ(z)-space.Aside from the detection performance, KPCA with kernel functions is far less computationally efficient than CoP/CoRP in two aspects. On the one hand, the time expense of eigendecomposition on the N tr × N tr kernel matrix K by the former is more expensive than that on the m × m or M × M covariance matrix Σ Φ by the latter, since N tr ≫ M and N tr ≫ m. For example, on the ImageNet-1K benchmark with MobileNet, these settings are N tr = 1, 281, 167, M = 2560 and m = 1280, on which KPCA with kernel functions is actually nearly prohibitive. On the other hand, in the inference stage, KPCA via kernel functions yet requires an O(N tr ) time and memory complexity in calculating k ẑ , as all the training data has to be stored and iterated, which is much higher than the O(1)/O(M ) complexity of our CoP/CoRP. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All the experiment details have been specified in both the appendix and the released anonymous code.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 As PCA reconstruction errors fail to distinguish OoD data from InD data on the penultimate features z of DNNs, kernel PCA is introduced for its non-linearity in the manner of employing explicit feature mappings. To find appropriate kernels that can characterize the non-linear patterns in InD and OoD features, we take a kernel perspective to decouple and analyze key components of an existing KNN detector [7] , and thus propose a cosine kernel and a cosine-Gaussian kernel for KPCA. Specifically, two explicit feature mappings Φ(•) induced from the two kernels are leveraged on original features z. For the cosine kernel, its explicit feature mapping can be directly obtained. For the Gaussian kernel, we adopt the celebrated random Fourier features to approximate the Gaussian kernel. The mapped Φ(z)-space enables PCA to extract principal components that well separate InD and OoD data, leading to distinguishable reconstruction errors. Extensive empirical results have verified the improved effectiveness and efficiency of the proposed KPCA with new SOTA OoD detection performance. Besides, more in-depth analyzes are drawn on the individual effects of the cosine kernel and the Gaussian kernel, and the involved multiple hyper-parameters. In addition, theoretical discussions and associated experiments are provided to bridge the relationships between our covariance-based KPCA and its kernel function implementation so as to further illustrate the advantages of our method.One limitation of the KPCA detector is that the two specific kernels are still manually selected with carefully-tuned parameters. It remains a valuable topic in the OoD detection task whether the parameters of kernels could be learned from data according to some optimization objective. For example, deep kernel learning [54] could be considered as an alternative choice so as to pursue stronger kernels that can better characterize InD and OoD with enhanced detection performance by an additional learning step on the features. We hope that the proposed two effective kernels verified empirically in our work could benefit the research community as a solid example for future studies.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The involved DNNs are pre-trained and the results of the PCA operation are deterministic.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We have provided sufficient information on the computer resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have discussed the societal impacts of the work in Section 7.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: No new models nor datasets are released in this work, and thus the paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All the creators and original owners of assets used in the paper have been properly credited. The license and terms of use have been explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: New asserts introduced in the paper have been well documented and provided.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we elaborate the scoring function S(•) of the OoD detectors included in the comparison experiments of Section 5. Given a well-trained DNN f : R d → R c with inputs x ∈ R d , the outputs are c-dimension logits f (x) ∈ R c w.r.t c classes. The DNN f learns features z ∈ R m of x before the last linear layer, i.e., the penultimate features z.MSP [15] employs the softmax function on the output logits and takes the maximum probability as the scoring function. Given a new sample x ∈ R d , its MSP score isEQUATIONODIN [16] introduces the temperature scaling and adversarial examples into the MSP score:EQUATIONwhere T denotes the temperature and xa denotes the perturbed adversarial examples of x.Mahala [23] employs the Mahalanobis score to perform OoD detection. The DNN outputs at different layers are modeled as a mixture of multivariate Gaussian distributions, and the Mahalanobis distance is calculated. Then, a linear regressor is trained to achieve a weighted Mahalanobis distance at different layers as the final detection score. To train the linear regressor, the training data and the corresponding adversarial examples are adopted as positive and negative samples, respectively.EQUATIONwhere f l (x) denotes the output features at the l-th layer with the associated mean feature vector µ l i of class-i and the covariance matrix Σ l , and α l denotes the linear regression coefficients.Energy [5] uses an energy function on logits as energy is well aligned with input probability densities:EQUATIONwhere f i (x) denotes the i-th element in the c-dimension output logits f (x).GODIN [27] improves ODIN from 2 aspects: decomposing the probabilities and modifying the input pre-processing. On the one hand, a two-branch structure with learnable parameters is imposed after the logits to formulate the decomposed probablities. On the other hand, the magnitude of the adversarial examples is optimized instead of manually tuned in ODIN.ReAct [18] proposes activation truncation on the penultimate features z of DNNs, as the authors observe that features of OoD data generally hold high unit activations in the penultimate layers. The feature clipping is implemented in a simple way:EQUATIONwhere α is a pre-defined constant. The clipped features z then pass through the last linear layer and yield modified logits. Other logits-based OoD methods such as Energy could be applied on the modified logits to produce a detection score. KNN [7] is a simple but time-consuming and memory-inefficient detector since it performs nearest neighbor search on the ℓ 2 -normalized penultimate features between the test sample and all the training samples. The negative of the (k-th) shortest ℓ 2 distance is set as the score for a new sample x.ViM [35] combines information from both logits and features in a complicated way for OoD detection. Firstly, penultimate features z are projected to the residual space obtained by PCA. Then the norm of projected features gets scaled together with the logits via the softmax function. Finally the scaled feature norm is selected as the detection score.DICE [34] is a sparsification-based OoD detector by preserving the most important weights in the last linear layer. Denote the weights W ∈ R m×c and the bias b ∈ R c in the last linear layer, the forward propagation of DICE is defined as: EQUATIONThe cosine kernel in CoP and CoRP appears an indispensable basis in alleviating the linear inseparability in InD and OoD features. The reason for its effectiveness lies in the imbalanced feature norms ∥z∥ 2 between InD and OoD features, which has also been observed in preceding works [7, 6, 32] . Figure 3 shows the feature norms of multiple InD and OoD data sets, from which one can find clear disparities of the InD and OoD feature norms. The cosine kernel k cos and the corresponding feature mapping ϕ cos in Equation ( 6) thereby normalize the feature norms and facilitate the separability between InD and OoD data.Figure 4 illustrates the t-SNE visualization on the InD and OoD features to further imply the importance of the cosine kernel. As shown in the middle panel of Figure 4 , the Gaussian kernel alone fails in creating separable InD and OoD features in the mapped space and actually leads to a complete mess of the mapped features. In contrast, the cosine kernel significantly alleviates the linearly-inseparability of features, shown in the right panel of Figure 4 .Ablation studies on the cosine feature mapping ϕ cos in CoP and CoRP are executed to verify its indispensability for OoD detection. Specifically, CoP without ϕ cos reduces to a standard PCA on the z-space, and CoRP without ϕ cos reduces to KPCA with a Gaussian kernel. Table 6 shows the corresponding detection FPR and AUROC values on each OoD data set in the ImageNet-1K benchmark of ablations on ϕ cos . Both the standard PCA and the Gaussian KPCA exhibit worse detection performance than CoP and CoRP. Particularly, the single Gaussian kernel in KPCA actually results in a complete failure in detecting OoD samples with nearly 95% FPR values. Therefore, the cosine kernel is essential in characterizing the non-linearity in InD and OoD features and critical for the superior performance of the KPCA detector.The design of the Gaussian kernel in CoRP is motivated by the useful ℓ 2 distance on z ∥z∥2 in the KNN detector [7] . The Gaussian kernel preserves the beneficial ℓ 2 distance relationship in ϕ cos (z)space through the RFFs mapping. In this section, we provide two alternative choices aside from the cosine-Gaussian kernel.• The cosine-Laplacian kernel explores the ℓ 1 distance in ϕ cos (z)-space via the Laplacian kernel k lap :EQUATION) To construct the RFFs for k lap , the sampling distribution of the ω i in Equation ( 4) is a Cauchy distribution ω ∼ p(ω) = • The cosine-polynomial kernel does not hold the ℓ 1 nor ℓ 2 distance-preserving property for ϕ cos (z)-space, as the polynomial kernel is defined as:EQUATIONTo obtain an explicit feature mapping for k poly , we do not adopt the RFFs and take the Tensor Sketch approximation [56] instead for simplicity.Table 6 illustrates the comparisons on the detection performance among multiple alternative kernels. Actually, both the cosine-Laplacian kernel and the cosine-polynomial kernel cannot bring detection performance gains on top of the cosine kernel (CoP), which indicates that the ℓ 1 -distance relationship characterized by the Laplacian kernel and the inner-product information characterized by the polynomial kernel in the ϕ cos (z)-space are less effective in promoting the separability between InD and OoD features. Thus, the cosine-Gaussian kernel is used and recommended in the proposed KPCA method for OoD detection.A comprehensive sensitivity analysis is executed to show the effects of hyper-parameters in CoP and CoRP. A common hyper-parameter in CoP and CoRP is the number of columns q of the dimensionality-reduction matrix U Φ q . Additional hyper-parameters of CoRP include the bandwidth γ of the Gaussian kernel and the number of RFFs M . In the following, we discuss the influence of each hyper-parameter, and report experiment results of the detection performance by varying one hyper-parameter with the others fixed.Effect of q q indicates the number of preserved principal components and determines how much information captured by the subspace where the InD and OoD data is projected onto. q is selected as the minimal number of principal components with the amount of information that exceeds the given explained variance ratio.Figure 5 illustrates the detection performance of CoP and CoRP under varied explained variance ratios. On CIFAR10 and ImageNet-1K benchmarks, for CoP, a mild value of the explained variance ratio is suggested with around 90% for keeping the components. Regarding CoRP, a sufficiently large value of the explained variance ratio is no longer essential for CoRP on the ImageNet-1K benchmark, which might be due to that the 2 concatenated kernels make the useful information for distinguishing OoD samples more concentrated in less principal components.Effect of γ The Gaussian kernel width γ directly affects the mapped data distribution. For a large γ, k gau (z 1 , z 2 ) = e -γ∥z1-z2∥ 2 2 ≈ 0 for z 1 ̸ = z 2 , which indicates that the mapped features of z 1 and z 2 are (nearly) mutually-orthogonal. In this case, a PCA would become meaningless. For a small γ, the KPCA-based reconstruction errors will approach the standard PCA-based ones, shown by [57] . Figure 6 illustrates the detection FPR95 and AUROC values of CoRP w.r.t varied Gaussian kernel width γ on CIFAR10 and ImageNet-1K benchmarks. Clearly, neither a too large nor a too small kernel width benefits the detection performance, and a mild value of γ should be carefully tuned for different in-distribution data.Effect of M The number of RFFs M determines the approximation ability of RFFs towards the Gaussian kernel. As proved in [13] , the larger the M , the better the RFFs approximate k gau . 0.5 0.6 0.7 0.8 0.9 1.0 explained variance ratio e Φ (ẑ)EQUATION(18) Obviously e Φ (ẑ) = U Φ⊤ p (Φ(ẑ) -µ Φ ) 2 and the proof is finished.The key in the proof of Proposition 1 is U Φ q U Φ⊤ q +U Φ p U Φ⊤ p = I. Since U Φ is the eigenvector matrix of the covariance matrix, thereby U Φ is a unitary matrix and satisfiesU Φ U Φ⊤ = U Φ⊤ U Φ = I, which leads to U Φ q U Φ⊤ q + U Φ p U Φ⊤ p = I.Justification: The paper provide the full set of assumptions and a complete and correct proof for each theoretical result.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Justification: The research in the paper conforms with the NeurIPS Code of Ethics in every aspect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://github.com/fanghenshaometeor/ood-kernel-pca 


Decision: This paper proposes to use a kernel PCA for OOD with deep neural net features. As a choice of kernel, this study suggests to use a cosine kernel and a cosine-Gaussian kernel for the OOD tasks. In addition to that, the usage of Random Fourier Features (RFFs) is also suggested to reduce the computational complexity. The practical effectiveness of the proposed method is examined through several numerical experiments.   

Although the usage of kernel PCA is not very novel, the authors enhanced their proposal by several numerical experiments. In addition to the main submission, they conducted some more ablation studies in the rebuttal to address reviewers' concerns.  
The authors experimentally demonstrated the effectiveness of the proposed method against the existing nearest neighbor method, which would be informative to the community.  

This paper was quite on the borderline, but based on the discussions with the reviewers after rebuttal, I would like to recommend acceptance of this paper.

Reviews: review: paper_summary: paper_summary: In this work, the authors leverage the framework of Kernel Principal Component Analysis (KPCA)
for Out-of-Distribution (OoD) detection, aiming to enhance the separability between In-Distribution
(InD) and OoD data within the subspace defined by the principal components. The study introduces
two task-specific kernels: a cosine kernel and a cosine-Gaussian kernel, both meticulously designed
for OoD detection. To address the computational challenges associated with large-scale kernel
machines, the authors employ Random Fourier Features (RFFs) to approximate the Gaussian kernel
functions. This approach not only maintains the effectiveness of the kernel methods but also
potentially reduces the time complexity of KPCA detectors, making them more feasible for practical
applications.
strength_weakness: strength_weakness: - **Lack of Evidence Against PCA Method:** There is insufficient evidence to substantiate the claim that the failure of the Principal Component Analysis (PCA) method in OOD detection is due to its reliance on linear mapping. Additional experiments and analyses are necessary to support this assertion.
- **Outdated Baselines in OOD Detection:** While the experiments on ImageNet-1K show that the proposed Kernel Principal Component Analysis (KPCA) method, which explores non-linear patterns, is advantageous compared to nearest neighbor searching, it remains uncompetitive within the broader out-of-distribution (OOD) detection community. The baselines used for comparison are outdated, undermining the reliability and relevance of the results.
- **Insufficient Demonstration of $\ell$2-Normalization Significance:** The demonstration of the indispensable significance of $\ell$2-normalization is inadequate. More thorough evidence and experiments are required to convincingly establish its critical role in the proposed methodology.
- **Manual Selection of Kernels and Parameter Tuning**: The specific kernels employed in the study are manually selected and require carefully-tuned parameters. This manual selection process raises concerns about the generalizability and robustness of the proposed method, as it may not perform optimally across diverse datasets and conditions without significant manual intervention.
questions: questions: - **Evidence Against PCA Method:** The paper mentions the failure of PCA in OoD detection due to its linear mapping. Can you provide more detailed evidence or experiments that explicitly show this limitation of PCA?
- **Extended Experimental Analysis:** Conduct a more comprehensive experimental analysis, including comparisons with the latest state-of-the-art methods in OoD detection. This will help validate the effectiveness and competitiveness of your proposed method in the context of recent advancements in the field.
limitations: limitations: - Manual Selection of Kernels and Parameters: One significant limitation of the proposed method is the manual selection and careful tuning of the cosine and cosine-Gaussian kernels. This process may not generalize well across different datasets and requires extensive manual intervention, which can be time-consuming and prone to human bias.
- Comparative Analysis with Outdated Baselines: The study compares the proposed KPCA-based method with outdated OoD detection baselines. This limits the assessment of the method's competitiveness and effectiveness in the context of recent advancements in the field. Including comparisons with the latest state-of-the-art methods would provide a more accurate evaluation.

---------------------------------------------------------------------------------------
After Rebuttal

The author's detailed response has addressed most of my concerns. I have increased my score. However, I cannot concur with certain aspects, particularly the selection of the kernel and hyperparameter tuning. The FPR cannot serve as a tuning target during training because the OOD datasets are test data. In the setup of this study, only ID data are available for kernel selection and hyperparameter tuning.
paper_summary: paper_summary: The paper introduces a method that applies Kernel Principal Component Analysis (KPCA) to the output of the penultimate layer of a Deep Neural Network (DNN) for out-of-distribution (OOD) detection. Inspired by previous approaches in OOD detection that utilize K-nearest neighbors (KNN) on $\ell_{2}$ normalized features, two kernels are designed to incorporate these features. The study explores the effectiveness of these kernels and discusses experimental results.
strength_weakness: strength_weakness: 1. It is not clear how the proposed method can adapt to a wider variety of feature maps and effectively leverage corresponding kernels. The current feature map used in this study draws inspiration from [1] and closely resembles it and the main observation of the paper is to consider to corresponding kernel to it. As a result, the effectiveness and novelty of the kernel method in this context are not entirely clear and need to be clarified. 

2. The authors mention ‘kernel learning’ as a potential area for future exploration under the study's limitations. However, pursuing kernel learning in this context may appear redundant given the extensive training already performed on the DNN from scratch for the OOD detection task.

[1]Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep 358 nearest neighbors. In International Conference on Machine Learning, pages 20827–20840. 359 PMLR, 2022.
questions: questions: 1. How can one select a kernel for OOD detection beyond those discussed in this work? Please refer to my previous comment as well.

2. Could you provide a mathematical justification for why KPCA with a Gaussian cosine kernel or a cosine kernel itself performs better than the KNN method with $\ell_{2}$ normalization, as discussed in [1]?

[1]Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep 358 nearest neighbors. In International Conference on Machine Learning, pages 20827–20840. 359 PMLR, 2022.
limitations: limitations: The authors discussed the limitations of their work.
paper_summary: paper_summary: The paper everage the framework of Kernel PCA (KPCA) for OoD detection, and seek suitable non-linear kernels that advocate the separability between InD and OoD data in the subspace spanned by the principal components. Besides, explicit feature mappings induced from the devoted task specific kernels are adopted so that the KPCA reconstruction error for new test samples can be efficiently obtained within an O(1) time and memory complexity in inference.
strength_weakness: strength_weakness: 1. It might be more useful to include more effective post-hoc OOD detection methods, such as ReAct, ASH for comparison. Also, the training-based approach might be useful to be included.

2. It might be more useful to include the ablation results on more kernel designs. 

3. I am wondering why linear separable space between ID and OOD matters, if we have a non-linear classifier, it can still perform accurate OOD detection. 

4. What are some important assumptions here for the theory? Might be useful to provide some justifications if there are any.
questions: questions: see above
limitations: limitations: n/a
paper_summary: paper_summary: The paper describes a method for out-of-distribution (OOD) detection. Kernel PCA is applied to the penultimate features of a neural network. Two kernels are chosen motivated by prior work on nearest-neighbour-based detection. Namely the cosine kernel and a cosine-Gaussian kernel. In experiments, both methods outperform baseline methods on image-based OOD detection.
strength_weakness: strength_weakness: - in section 2 it is not entirely clear whether the RFF approximation is used for the the cosine-Gaussian kernel alone or also for the cosine kernel. Also please, elaborate on the details of how the kernel is approximated. 

- in relation to the last point, in the experiments, it is important to highlight the dimensionality of the penultimate layer features $c$ and the number of random Fourier features that you select. It is known that for medium to high-dimensional problems, the approximation with RFF requires a very high number of features.

- line 198: the computational complexity of the RFF is not discussed in this paragraph.

- section 2.2 is less related work and more background. Thus, sections 2.2 and 3.1 could be better summarized in a "background" section. 

- It is not entirely clear how Proposition 2 is necessary to implement CoP and CoRP. Please clarify this in the manuscript.

- at the start of the paper it is unclear why you two kernels (lines 65, 68, ...). Only from the experiments and algorithm 1, it becomes clear that these are two different choices/methods.
questions: questions: - it is unclear how the proposed methods achieve O(1) time and memory complexity with RFF. Please elaborate on this.
- line 133: how is $q$ chosen?
- in Figure 2, please clarify what the kernel function in this experiment exactly is.

Smaller things:
- line 282: here the standard solution from [1] should be mentioned
- line 192: wrap up
- line 259: "under the same framework" unclear phrasing
- line 299: "we adopt" remove "should"
- for automatic learning of features from data, a suggestion is to consider Recursive Feature Machines [2].

References:
[1] G. H. Bakır, J. Weston, and B. Schölkopf, “Learning to find pre-images,” in Proc. Adv. Neural Inf. Process. Syst., 2004, pp. 449–456.

[2] A. Radhakrishnan et al., Mechanism for feature learning in neural networks and backpropagation-free machine learning models. Science 383, 1461-1467 (2024).
limitations: limitations: For reproducibility the main limitation lies in the missing information about the RFF implementation as no information is provided about the number of random Fourier features. Furthermore, it is not clear what other kernel hyperparameters are to be tuned and how they are selected.

Comments: title: Summary of Rebuttal and Discussions
Response to W2 and W3
Response to W1
Thank you for your response.
Official comment by reviewer 6eWv
Official comment by reviewer 6eWv
comment: Dear Chairs and Reviewers,

Thanks for the constructive comments and suggestions during the review process. As the discussion period is approaching the end, we summarize the outcomes below.

---
**Strengths & Reviewers' support:**
1. *Contribution to the kernel perspective for the OoD detection community.*

   All reviewers agree on our contribution of introducing a kernel perspective into OoD detection, which has not been extensively explored in the community.

   Our main contribution lies in the novel kernel perspective on OoD detection with carefully-devised kernels for effectiveness and approximate explicit feature mappings for efficiency,  as recognized by the reviewers on our detection performances and computation efficiency.


2. *Thorough experimental evaluations with strong performances and efficiency.*

   Our method has been  extensively evaluated  with comparisons to many very recent SOTA methods. In rebuttal, as suggested by *Reviewer sSKR* and *Reviewer w1ZW*,  **four very latest methods**, i.e.,  SCALE [1] (ICLR'2024), DDCS [2] (CVPR'2024), ASH [3] (ICLR'2023) and DML [4] (CVPR'2023), have also been added into comparisons in **Table sSKR-2**,  further demonstrating our superior performances.

   Aside of the performances, careful ablation studies have also been presented, as shown in **Table sSKR-3** and **Fig.5**, **Fig.6** in **Appendix D.1** and **Appendix D.2**, covering the imbalanced norms of InD and OoD features, our specified kernel designs, and also comprehensive sensitivity analysis of the hyperparameter as presented in **Appendix D.3**, facilitating more in-depth understandings on our  method.


---
In the rebuttal, we have responded to each of raised comments. *Reviewer 6eWv and Reviewer LuQe recognized that our responses have addressed their questions during the discussions; Reviewer LuQe mentioned to vote to accept our work and Reviewer 6eWv increased the score.*

**Remaining comments in the review:**  
Below, we summarize the remaining comments which we may have no time further discuss about.

> Reviewer sSKR: Could you provide a complete statement regarding the motivation, and explanations on the boosting scheme and the hyper-parameter selection?

We have elucidated the motivation from PCA to KPCA in the latest rebuttal of **Response to W1**, which mainly lies in:
- the advantage of the **low-dimensional** subspace learned by (K)PCA in boosting robustness;
- the **non-linearity** of kernels that improves the limitation of PCA in inseparable cases.

Besides, in the rebuttal of **Response to W2 and W3**, we have also clarify the relationship between our method and the boosting scheme, and the principal governing the hyper-parameter selection.

> Reviewer w1ZW: The studied problem is very important. The results seem to outperform state-of-the-art.  
> Reviewer w1ZW also raised suggestions to add comparisons with more methods and kernel designs, elaborations on the significance of linear separable space, and our assumptions.

We thank the *Reviewer w1ZW*'s acknowledgement on the significance and effectiveness of our work. We have responded to each comment with both empirical evidence and explanations. By far, we have not received the feedback from the reviewer, though we would be very willing to discuss and see if any further advice would be received.

---
**Updates to the manuscript:**

We thank all reviewers for the suggestions in the review and discussion periods. Accordingly, the new experimental results and the more clarified elaboration on our work in our responses will be added to update the manuscript, which we believe are durable in the current status. We hope that our summaries above and pointwise responses below will be helpful for the final decision of our work.

Best wishes,  
Authors of Paper 5729

[1] Xu, et al. Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement. ICLR'2024.  
[2] Yuan, et al. Discriminability-Driven Channel Selection for Out-of-Distribution Detection. CVPR'2024.  
[3] Djurisic, et al. Extremely Simple Activation Shaping for Out-of-Distribution Detection. ICLR'2023.  
[4] Zhang, et al. Decoupling maxlogit for out-of-distribution detection. CVPR'2023.
### **(W2) Our method and the boosting scheme**

Our proposed method is *not* simply a boosting scheme, but a detection method via the KPCA reconstruction error being the detection score.

**The "*bad*" results in Table 1**  
In **Table 1**, CoRP is under a fair comparison with the KNN method [2], where the detection procedures are both straightforwardly proceeded with the DNN features. Besides, the results in Table 1 are not truly bad, since CoRP outperforms the related KNN method in performances and also efficiency as in **Table 2**.  Note that CoRP also achieves better detection than other compared popular baselines, which do not involve boosting techniques.

**The boosting scheme**  
In **Table sSKR-2** of our rebuttal and **Table 3** of Sec.4.2, the boosting scheme is adopted, as we aim to present a fair comparison with the relevant PCA-based method [6]. The PCA method alone cannot achieve reasonable detection performance, unless its detection score gets fused with other methods as shown in [6], e.g., ReAct+PCA. Therefore, we report the results of our KPCA detector *under the same boosting scheme*, e.g., ReAct+CoRP, which refers to the "ours" in **Table sSKR-2** above.

*Remark:* We would like to point out that the strong SOTA methods compared in **Table sSKR-2** and **Table 3** are specifically designed with incorporating boosting schemes, e.g., the "DICE+ReAct" in Table 3 and SCALE [1], DDCS [2] in **Table sSKR-2**, where the feature-pruning-based detectors SCALE [1] and DDCS [2] are boosting detection scores on logits by clipping features. Therefore, in this set of experiments, our boosting implementation is fair for comparisons. Without the boosting technique, our method also achieves superior performances than the compared methods in Sec.4.1.

---
### **(W3) Hyper-parameter selection**

Maintaining a high true positive rate (TPR) and our hyper-parameter tuning principle are elaborated below.

- **Maintaining a 95% TPR** is a common setup in current evaluations on OoD detection methods. Specifically, for any detection method, it is required to achieve a 95% TPR on the InD test set firstly to assign the value of $s$ in Eqn.(1), then, based on the assigned $s$, the detection results on OoD datasets (FPR and AUROC) are determined, which is a widely-adopted procedure in evaluating OoD detection methods.

- **The hyper-parameter selection** is to obtain the best detection FPR values on OoD datasets, in order to find the best performance that can be achieved by the detection method, which is also a common setup in existing works. For example, in prevailing feature-pruning-based OoD detectors [1,2,3,5] with the pruning thresholds as hyper-parameters, their detection results are recorded under a variety of different thresholds and the best results are reported finally, which provides a comprehensive understanding on the feature pruning. Regarding our KPCA detector, we also give a detailed analysis on the effect of each hyper-parameter in our method in **Appendix D.3**, for in-depth insights on the KPCA detector.
Dear Reviewer,

Thanks for your kind reply. Below we provide a pointwise response to each of the  further comments.

---
### **(W1) Motivation behind our work**

**(A1)** The motivations from PCA to KPCA are elaborated below.
- The linear transformations by simply applying PCA [6] cannot capture informative principal components of InD features. As shown in **Fig.1(a)**, InD features are not even located compactly nor linearly-separable with OoD features. In this case, the axes extracted from PCA are insufficient to depict the intrinsic patterns of InD features, so the both InD features and OoD features can have bad reconstructions, hindering to well differentiating OoD from InD. 
- Despite of the current bad performance of PCA, the **low-dimensional subspace** property is commonly considered to boost robustness and thus should be of great interest in OoD detection, as long as informative components can be extracted, such that the intrinsic patterns of InD features are kept, while removing redundant information. Hence, introducing appropriate **non-linear** feature mappings is promising as a remedy to the existing obstacle, motivating us to construct our KPCA-based detector. 
- With KPCA, we can proceed the task in a new **non-linear** feature space, where with proper kernels (new feature spaces), the corresponding InD mappings are located more compactly and even almost linearly separable from OoD mappings, as shown in **Fig.1(b)**. In this mapped space, KPCA can learn informative principal components from InD data which produce distinguishable reconstruction errors w.r.t OoD.

For the raised two questions, we answer below.

**(Q1) Does PCA significantly impact OoD detection?**  
Yes. (K)PCA learns a **low-dimensional** subspace spanned by the (nonlinear) principal components from the InD data. All InD data are utilized to learn such a low-dimensional subspace, while KNN [2] only utilizes the nearest affinity neighborhood in InD data. With the low-dimensional subspace technique, it is expected that the most informative patterns of InD are kept and redundant information in InD is removed, and meanwhile OoD features are easily separable in such space. In this way, based on projections onto the subspace,  the good reconstruction of InD could well differentiate InD and OoD and serve as an effective OoD detection score. Here, the critical problem is how to find an effective low-dimensional subspace in an efficient way, which is well resolved by our proposed KPCA-based detector. **Figure 1** can be referred for illustration.

**(Q2) Is the importance of non-linear, low-dimensional structures evident in OoD detection?**  
Yes. The importance of **low-dimensional structures** has been elaborated in the response to **(Q1)** above. This can also be observed from the improved performance of CoRP than KNN, which all utilize the $\ell_2$ distance/similarities between normalized InD features in **Table 1**. The **non-linearity** lies in the non-linear kernel which is in correspondence to a mapped feature space, elaborated in **(A1)** above. With proper non-linear transformations, significantly improved separability between InD and OoD is attained, as shown by the improved performance of CoP/CoRP than PCA in **Table sSKR-1**.
[W1] Could you provide a complete statement regarding the motivation behind this work? Specifically, addressing two critical questions is essential: 1. Does PCA significantly impact OOD detection? 2. Is the importance of non-linear, low-dimensional structures evident in OOD detection?

[W2] Does "Ours" refer to "ReAct+CoRP"? 

Is your proposed method a boosting scheme? According Table 1, the performance of CoRP appears to be bad.

[W3] The proposed method necessitates the selection of a kernel function and bandwidth, with training conducted without access to out-of-distribution (OOD) data. The tuning objective is to maintain a high true positive rate (TPR). However, for any chosen kernel function and bandwidth, a threshold can be determined to achieve a 95% TPR. What is the principle governing hyperparameter selection in your method?
Thanks, this answers all my concerns and I have updated my score accordingly.
Dear Reviewer,

We hope that our responses above could address your concerns and be helpful for final evaluations on our work.

As the author-reviewer discussion is approaching the end, we would like to inquiry  if there is any question, and we would be willing to discuss. 

Sincerely, Authors.
Dear Reviewer,

We hope that our responses above could address your concerns and be helpful for final evaluations on our work.

As the author-reviewer discussion is approaching the end, we would like to inquiry  if there is any question, and we would be willing to discuss. 

Sincerely, Authors.
Dear Reviewer,

We would like to thank you for recognizing our method being interesting to the OoD detection community, and your support on the acceptance of our work. 

Your suggestions on future works are inspirational to further explorations with kernel methods for OoD detection:
- **Computation advantage on a broader class of kernels.** Our detection method is intentionally devised as a KPCA via *finite-dimensional (approximate) explicit feature mappings*,  which *avoids the exact computation on the Gram (kernel) matrix*, guaranteeing the efficient inference time. If only an arbitrary Gram/kernel matrix is available, it remains an interesting topic to be explored for efficient OoD detection with our KPCA-based method, as an (approximate) explicit feature mapping w.r.t the Gram (kernel) matrix is required with acceptable dimensions.
- **Priori on more viable kernels.** In our work, we are grounded on the $\ell_2$-normalization property from the Cosine kernel for InD features and the $\ell_2$ distance-preserving property from the Gaussian kernel, jointly contributing to effective OoD detector with KPCA. Aside of the kernels discussed in rebuttal  **(A2)** above, we agree that a even broader class of kernels and more specific priori would be of interest  in future work for further understandings on kernel methods for OoD detection tasks.

In the final version, as suggested, we will incorporate the corresponding discussions in the rebuttal  to elaborate our work and also the potential future directions.

Sincerely, Authors
Dear Authors,

Thank you for the rebuttal and for addressing my questions.

The interesting aspect of the paper is that the KPCA perspective can reduce the computational cost of [1] while achieving comparable performance. I think this would be of interest to the OOD detection community. Additionally, the extensive experiments and evaluations provided are helpful.

On the downside, I am still not convinced how your approach can be extended to a broader class of kernels while maintaining the computational advantage and/or providing new scores for OOD detection beyond the two kernels discussed. For example, if one uses the associated Gram matrix (which is common in KPCA), the computational complexity advantage of your method may be lost. Furthermore, it is not clear a priori how to select an appropriate kernel for the OOD detection task. However, this aspect could be explored more thoroughly in future work.

Overall, I would like to maintain my score and vote to accept the paper. I would suggest to please discuss the limitations of your approach and the scope of your contributions in the abstract and introduction as mentioned in the above and in other reviews.

Thank you,
Reviewer

[1] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pages 20827–20840. PMLR, 2022.
Dear Reviewer,

Thanks for your reply. We agree with your understandings on the efficient inference time with the exact feature mapping and the approximated RFF mapping in our deployed Cosine and Cosine-Gaussian kernels, respectively. 

Regarding the extension to more general classes of kernels, our method can still achieve such efficient OoD inference, *as long as (the approximate) feature mappings w.r.t the kernel can  be obtained*, since the reconstruction error can be calculated straightly with an efficient ${\cal O}(1)$ complexity on mapped features. Several examples on explicit feature mappings of different types of kernels are discussed below.
- The adopted *Gaussian* and *Laplacian* kernels are actually in correspondence to infinite-dimensional feature spaces and can be approximated via RFF, as in our work. 
- Another alternative is the *polynomial* kernel $k_{\rm poly}(z_1,z_2)=(z_1^\top z_2+c)^d$ w.r.t a high-dimensional space, which however does not satisfy the conditions of RFF. In this case, we can adopt relevant approximation methods for particular kernels, e.g., the Tensor Sketching in [1] for polynomial kernels as in **Table LuQe-1** in our rebuttal above.
- A more general solution for generic kernels is provided: One can firstly use a sum of Gaussian kernels to approximate any kernel, and then adopt RFF to approximate the approximated Gaussian kernels, see an example in [2].

Given these approximation methods, we would like to further highlight that the required dimension of the mapped features is not very high to ensure good OoD detection performances. Take the Gaussian kernel with infinite-dimensional spaces as an example, the dimension of approximated RFFs $M$ are set as $M=4m$ for the CIFAR10 InD and $M=2m$ for the ImageNet-1K InD throughout the paper, where $m$ is the dimension of DNN backbone features. Please kindly refer to **Fig.9** in our attached PDF in rebuttal and **Appendix D.3**, and our response **(W2)** and **Table 6eWv-1** to *Reviewer 6eWv*.

We hope the responses above could well resolve your concerns and help the final evaluations on our work. If there is any further comment, we would like to discuss.

Sincerely, Authors

[1] Pham, et al. Fast and scalable polynomial kernels via explicit feature maps. KDD'2013.  
[2] Pennington et al. Spherical random features for polynomial kernels. NeurIPS'2015.
Dear Reviewer,

We agree with your suggestions. In the final version, we will polish up our manuscript with elaboration and clarification on 
- the distinction between CoP and CoRP in the beginning of introducing our method and also in abstract,
- the use of RFF in Gaussian kernel approximation in CoRP and also the corresponding two distinctive contributions of our kernel designs specified for OoD detection,
- the clarified computation complexity of CoRP as ${\mathcal O}(M)$ with RFF as explained in our above rebuttal,

which are viable to be included into the final version.

We thank the Reviewer for recognizing our rebuttal responses and the helpful advice. We would appreciate if our responses are helpful for the final evaluations on our work.

Best regards,  
Authors
Dear Authors,

Thank you for your response. I have a follow-up question regarding your answer about extending to a more general class of kernels. Specifically, regarding computational complexity: as I understand it, efficient inference time is achieved only for the two specific kernels considered in your work (e.g., Cosine and Cosine-Gaussian kernels). For the Cosine kernel, the feature map has the same dimension as the feature space, while for the Cosine-Gaussian kernel, the feature map is approximated using Random Fourier Features (RFF).

Could you comment on whether this efficient inference time can be achieved for other classes of kernels, such as those with potentially infinite-dimensional or high-dimensional feature spaces, beyond the two specific cases considered in your work?

Sincerely,
Reviewer
I would like to thank the authors for their responses. My questions have been addressed. 

The last remaining point is to ask the authors, to phrase their manuscript such that a clear distinction between CoP and CoRP is made from the beginning. This regards the use of RFF, the complexity and that these are two distinct contributions. Thus, claims about the complexity, such as in the abstract, need to be clarified (and rectified) with the $\mathcal{O}(M)$ complexity.