Title: Quantifying Statistical Significance of Deep Nearest Neighbor Anomaly Detection via Selective Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Anomaly Detection

k-Nearest Neighbors

Statistical Test

Selective Inference

Deep Learning
Abstract: In real-world applications, anomaly detection (AD) often operates without access to anomalous data, necessitating semi-supervised methods that rely solely on normal data.
Among these methods, deep $k$-nearest neighbor (deep $k$NN) AD stands out for its interpretability and flexibility, leveraging distance-based scoring in deep latent spaces.
Despite its strong performance, deep $k$NN lacks a mechanism to quantify uncertainty—an essential feature for critical applications such as industrial inspection.
To address this limitation, we propose a statistical framework that quantifies the  significance of detected anomalies in the form of $p$-values, thereby enabling control over false positive rates at a user-specified significance level (e.g.,0.05).
A central challenge lies in managing selection bias, which we tackle using Selective Inference—a principled method for conducting inference conditioned on data-driven selections.
We evaluate our method on diverse datasets and demonstrate that it provides reliable AD well-suited for industrial use cases.


Peer reviews:
comments: SUMMARY:
This paper proposes a framework to quantify the statistical significant of anomalies detected through deep kNN based anomaly detection in the form of p-values for hypothesis testing via Selective Inference (SI)

STRENGTHS AND WEAKNESSES:
The paper is thoughtfully written and easy to understand. 

Experiments are comprehensive, covering a variety of data types (tabular, image) and a range of datasets.

It is difficult to find fault with this paper, though I would say it would be beneficial to make more effort explaining the details of other SI methods to provide clearer grounding of this work within the context of other literature. Also, Figure 4 is not particularly easy to comprehend. The authors may consider alternative ways to plot their results.

QUESTIONS:
1. Is this the first paper to consider SI for AD? Or only for deep kNN-based AD? If so, how does this methodology differ from others proposed in other settings?

2. How does this result translate to other relevant tasks such as OOD detection?

LIMITATIONS:
No discussion of negative societal impact, though model limitations are discussed.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper applies selective inference techniques to KNN anomaly detection, aiming to provide statistical guarantees for the detection process. The authors present a methodological framework, prove supporting theorems, and evaluate the approach on synthetic and real-world datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper clearly articulates its core contribution: establishing a selective inference framework tailored for kNN-based anomaly detection, supported by theoretical results and empirical validation. The inclusion of asymptotic results and experiments helps bridge a gap in the statistical understanding of kNN anomaly detection methods.\n\nWeaknesses:\n\nThe theoretical contributions appear limited in novelty. The paper primarily establishes asymptotic distributions under standard Gaussian assumptions, following a relatively well-established framework. It does not clearly demonstrate the unique advantage of the proposed selective inference method theoretically over alternatives such as empirical p-value–based methods or bonferroni correction or multiple testing techniques like FDR.\n\nThe Gaussian assumptions throughout the theory are too strong. This restricts the generality of the results and may undermine applicability to real-world anomaly detection settings, where sub-Gaussian, sub-exponential, or heavy-tailed distributions are common. This issue also extends to the synthetic experiments, which are very limited to parametric and semi-parametric settings under Gaussian assumptions.\n\nMoreover, the paper mentions general applicability of the SI framework across learning methods, yet does not sufficiently articulate what makes this adaptation to kNN unique. This raises questions about the novelty and specificity of the proposed approach.\n\nCertain components, such as the anomaly score in Equation (1), appear to be adopted from existing literature without strong motivation or justification. Similarly, the test statistic used is tightly coupled to the Gaussian assumption without any data-driven adjustment, further weakening its practical utility.\n\nQUESTIONS:\nPlease see the weakness part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a selective inference framework for quantifying the statistical significance of the a deep kNN anomaly detector. The proposed method studies what would the p-value of a test observation be if we assume that the signal component (which is modelled non-parametric) is the same as the k-nearest neighbour. Nonetheless, since the computation of this p-value is intractable because it'll mean to model a complex neural network, the authors use Selective Inference to achieve a tractable solution (by conditioning the significance test). Numerical examples are performed considering synthetic, tabular and image data, showing that the selective inference provides a good measure of statistical significance while having the greatest power among all of the other baseline methods. Overall, this work is an applied work about selective inference for the anomaly distribution task.\n\nSTRENGTHS AND WEAKNESSES:\nThe work stablishes a good motivation on why using Selective Inference for anomaly detection could give a valid statistical significance measure of a prediction for an anomaly detector. The mathematical framework is mostly well defined and explained, although there are some parts that lack rigorosity. The experiments show that Selective Inference is in fact a good method for getting an accurate p-value. Therefore, I think the work is technically solid and addresses an interesting problem.\n\nNonetheless, I still found some weaknesses that makes me lean towards a reject. I have three main criticisms: (1) the work is limited to a single method for anomaly detection (deep kNN); (2) it is not clear how this actually helps Anomaly Detection; and (3) I think there might be an error in the proof of Theorem 4.1, which could potentially invalidate some of the analsys . For (1), there are several methods for anomaly detection, and choosing a single method makes me question the significante of the work. Several methods have shown to work as good or better than deep kNN for AD, so it'll be great to discuss how SI could be applied to these methods (if it can). Additionally, the evaluation is carried out in rather limited datasets. I find the synthetic dataset not sufficiently strong to explore the gains of SI in real-life scenarios, and the analysis of a single method in the MVTec dataset is also underwhelming, since this dataset has been practicallty solved using different methods. For (2), I think that while the authors clearly state the motivation, it would be great to discuss how this actually could help in anomaly detection. Regarding (3), in the proof of Theorem 4.1, you have this:\n\n\\begin{equation}\n(I_{(n+1)d}-P)\\mathbf{Y} = \\mathcal{U}(\\mathbf{y})\n\\end{equation}\n\nThen we'll have:\n\\begin{equation}\n\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) + P\\mathbf{Y} \n\\end{equation}\n\nWhich would be equivalent to\n\\begin{equation}\n\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) + \\frac{\\lVert P\\mathbf{Y}  \\rVert P\\mathbf{Y}}{\\lVert P\\mathbf{Y} \\rVert}\n\\end{equation}\n\nWhich, according to your notation, it'd be:\n\\begin{equation}\n\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) + \\lVert P\\mathbf{Y}  \\rVert \\mathcal{V}(\\mathbf{Y})\n\\end{equation}\n\nBut you have \n\\begin{equation}\n\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) +  \\mathcal{V}(\\mathbf{Y}) \\mathbf{z} =  \\mathcal{U}(\\mathbf{y}) + \\lVert P\\mathbf{Y}  \\rVert_2^2 \\mathcal{V}(\\mathbf{Y})\n\\end{equation}\n\nI think there might be a problem here that probably would invalidate most of the analysis. \n\nAdditionally, I also think that the work could be explained better. For instance, adding more detail into where do the condfitions in the equation between lines 197 and 198 come from. I understand that you might refer to a detailed work about SI, and that;s fine, but for the sake of the analysis in this paper I think it is important to discuss it. Is that nuisance parameter the same for all the SI problems? finally, I think some qualitative examples in the main paper would be good. \n\nOther minor weaknesses:\n- It'd be a good idea to add quantitative results in the abstract\n- L18: Tsehe -> This\n- L36: tereby -> thereby, by making...\n- L66: It's important to mention why quantifying statistical significance for such AD methods would be important\n- L102: I think it's more self-supervised rather than semi-supervised, but I leave it to the author\n- L107: $x_i$ -> $\\mathbf{x}_i$\n- L123-124. Any ref aboyut the heuristic method for choosing k?\n- Please add numbering to *all* equations\n- Eq between L175 and L176: I think $\\boldsymbol{\\eta}_y$ depends on $k$ but it's neither   $\\boldsymbol{\\eta}_k$ or  $\\boldsymbol{\\eta}_y (k)$. I think there should be a way to explicitly show that it depends on k.\n- Fig 4: you cannot see some error bars, could you please improve this?\n\nQUESTIONS:\n1. Could you please elaborate on how you reached to $\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) + \\lVert P\\mathbf{Y}  \\rVert_2^2 \\mathcal{V}(\\mathbf{Y})$ andh why it is not actually $\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) + \\lVert P\\mathbf{Y}  \\rVert \\mathcal{V}(\\mathbf{Y})$ (as per above)? If indeed you made a mistake, how does that would affect your analysis? (If I'm making a mistake please ignore this question). \n\n2. How could this improve upon Anomaly Detection in industrial settings?\n\n3. Could you provide more results for a different image dataset? The MVTec is already solved.\n\n4. How would this analysis translate to other Anomaly Detection methods? Such as Teacher-Student models?\n\n5. Isn't Eq (4) the same as Eq (5)? But you say Eq (4) is intractable but Eq (5) can be computed. I might be wrong, but could you clarify this?\n\nI'd be happy to change my score if these questions are answered satisfactory, but I'm also going to consider other factors such as discussion with other reviewers and the main weaknesses I found.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies semi-supervised anomaly detection, where the model is trained solely on normal data. It focuses on the existing \"deep k-NN AD\" method, which maps input data into a latent feature space using a pre-trained CNN, and then measures the \n  distances between a test sample and the training points. While the paper does not propose a new detection algorithm, its main contribution lies in introducing a statistical framework to assess the significance of detections made by deep k-NN. Specifically, the authors formulate the AD process as a hypothesis testing problem and propose a selective p-value based on the selective inference framework. This approach aims to control the false positive rate and correct for selection bias in the AD setting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper identifies a fundamental challenge in deep k-NN-based anomaly detection that computing the conventional p-value is intractable. To address this, the authors propose an alternative: a selective p-value derived from the selective inference framework, which is computationally feasible, and thus capable of effectively quantifying the false positive rate. This is a meaningful contribution, as most anomaly detection research (especially those on the theoretical side) tends to focus primarily on overall misclassification rate. However, in many real-world applications such as cybersecurity, controlling false positives is often more critical.  I believe this work makes a significant contribution to the anomaly detection literature.\n\nWeaknesses:\n\n1. I am not very familiar with the issue of \"double-dipping\", which the authors cite as a primary motivation for adopting selective inference to quantify statistical significance in deep k-NN AD. I understand that \"double-dipping\" refers to using the same dataset for both hypothesis selection and testing, why this issue cannot be addressed by simply splitting the dataset? Since this work specifically studies semi-supervised AD which trained only on normal data, the lack of labeled anomalies shouldn't prevent such a split. It would help the general audience if the authors could explain more precisely why double-dipping is unavoidable in this context, and whether the challenge is specific to the k-NN approach or applies more broadly.\n\n2. Perhaps related to the above point, I understand that this work prioritizes minimizing the false positive rate, which is indeed critical in AD. However, the overall misclassification rate (including both false positives and false negatives) is also an important metric in AD research. I wonder if there is a trade-off between reducing the false positive rate and increasing the false negative rate in this setting? It would be helpful if the authors could comment on whether their method maintains a good balance, or if minimizing Type-I errors comes at the cost of missing true anomalies.\n\n3. While reading the numerical experiment results, I was particularly struck by the consistently zero Type-I error achieved by the Bonferroni correction. It seems almost too good to be true. Can the authors provide a discussion comparing their proposed method to Bonferroni correction beyond what is shown in the experiments? Are there advantages that are not immediately visible from the reported metrics? Otherwise, based solely on the experimental results, one could argue that Bonferroni performs better than the proposed method, and that there may be little justification for adopting the new approach. \n\n4. One thing of the synthetic data experiments left me confused. It appears that all the synthetic datasets were generated from Gaussian distributions. However, the authors mention that the Hotelling’s \n$t^2$ test fails to control the Type-I error because it relies on the Gaussian assumption. If the data is Gaussian, shouldn't the test perform well under its assumed conditions? \n\nMinor comment: There is a typo on line 18 on page 1. It should be \"The\" instead of \"Tsehe\".\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}