Quantifying Statistical Significance of Deep Nearest Neighbor Anomaly Detection via Selective Inference
Authors: Mizuki Niihori
Shuichi Nishino
Teruyuki Katsuoka
Tomohiro Shiraishi
Kouichi Taji
Ichiro Takeuchi
Venue: NeurIPS 2025 poster ()
Keywords: Anomaly Detection
k-Nearest Neighbors
Statistical Test
Selective Inference
Deep Learning
Primary area: probabilistic_methods
TL;DR: We propose a method to quantify the statistical significance of anomalies detected by deep k-nearest neighbor approach.

Abstract: In real-world applications, anomaly detection (AD) often operates without access to anomalous data, necessitating semi-supervised methods that rely solely on normal data.
Among these methods, deep $k$-nearest neighbor (deep $k$NN) AD stands out for its interpretability and flexibility, leveraging distance-based scoring in deep latent spaces.
Despite its strong performance, deep $k$NN lacks a mechanism to quantify uncertainty—an essential feature for critical applications such as industrial inspection.
To address this limitation, we propose a statistical framework that quantifies the  significance of detected anomalies in the form of $p$-values, thereby enabling control over false positive rates at a user-specified significance level (e.g.,0.05).
A central challenge lies in managing selection bias, which we tackle using Selective Inference—a principled method for conducting inference conditioned on data-driven selections.
We evaluate our method on diverse datasets and demonstrate that it provides reliable AD well-suited for industrial use cases.

Quantifying Statistical Significance of Deep Nearest Neighbor Anomaly Detection via Selective Inference
Mizuki Nithori!
Shuichi Nishino Teruyuki Katsuoka'
Tomohiro Shiraishi!.2
Kouichi Taji' Ichiro Takeuchi
'Nagoya University PRIKEN
niihori mizuki www.yaallggmii..coomoommmmm takeuchi.. werichi..chize sichiicc...... n60f mail. nagoya-u..c..pp
Abstract
In real-world applications, anomaly detection (AD) often operates without access anomalous data, necessitating semi-supervised methods that rely solely on normal data. Among these methods, deep k-nearest neighbor (deep kNN) AD stands out for its interpretability and flexibility, leveraging distance- based scoring in deep latent spaces. Despite its strong performance, deep &NN lacks mechanism quantify uncertaintyan essential feature for critical applications such industrial inspection. To address this limitation, we propose a statistical framework that quantifies the significance of detected anomalies in the form of p-values, thereby enabling control over false positive rates at user-specified significance level (e.g.0.055). A central challenge lies in managing selection bias, which we tackle using Selective Inference-a a principled method for conducting inference condi- tioned data-driven selections. We evaluate method diverse datasets and demonstrate that it provides reliable AD well-suited for industrial use cases.
1 Introduction
In many practical anomaly detection (AD) problems, anomalous data is often unavailable in advance; therefore, AD algorithms must be developed using only normal data-a setting known as semi- supervised AD (13) Among various ceni-supervised AD methods, we focus on the k-nearest neighbor (kNN) approach ( The iNN approach simple yet effective, offering flexibility,, minimal assumptions about the data, and adaptability to different distance metrics. Especially, by applying KNN in a latent feature space identified by deep learning models, anomalies can be detected more flexibly using task-specific distance metricsan approach we refer to as deep kNN-based AD. Deep kNN-based AD combines the power deep learning models with interpretable kNN scoring latent space, making it a promising method for practical applications [5)
However, its dependence on complex deep learning-base detection procedures means that no established method currently exists for rigorously quantifying uncertainty or ensuring reliability. This limitation is particularly critical in high-stakes applications such as industrial inspection, where reliable uncertainty estimation is essential. Despite the interpretability of deep i-NN-based scoring, it lacks a principled way assess how confidently a test case deviates from normal cases. Therefore, developing a statistical framework that complements the high detection accuracy of deep kNN with uncertainty quantification is a crucial step toward practical and reliable deployment real-world practical applications.
To address this issue, we propose method that can quantify the statistical significance of the results obtained by deep ANN-based ADs. Specifically, we formulate the kNN-based AD as statistical hypothesis testing problem and develop method quantify the statistical significance of detected anomalies in the form of p-values. The p-value of a detected anomaly represents the probability that the anomaly is a false positive. By making decisions based on anomalies with p-values below
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

certain threshold (e.g., 0.05), we can ensure that the probability of erroneous decisions remains below the significance level of 5%. In this study, we refer to the proposed testing method as the deep-kNN test. From a practical perspective, the ability to explicitly control the false positive rate highly beneficial in real-world anomaly detection. safety-critical applications such as medical diagnosis or industrial inspection, false positives often lead to unnecessary interventions, costs, or workflow disruptions. By providing a statistically significant upper bound on the false positive rate, our framework enables practitioners to make anomaly detection decisions with quantifiable level of reliability.
Accurately quantifying the statistical significance of anomalies identified by deep kNN is non- trivial challenging task from the following two perspectives. The first challenge arises from the fact that both AD and testing are performed on the same dataset, leading to selection bias called double- dipping 66 The second challenge is that when anomalies are detected based on the latent feature space of deep learning model, it becomes necessary account for the complex process of the deep learning model computation. The core idea of our deep kNN test is to address these two challenges
by introducing statistical testing framework known as Selective Inference (SI) (7 SI has recently gained attention as method for statistically testing data-driven hypotheses, by conditioning statistical inference on the event that the hypothesis has been selected. Figure SaareeTeeenne demonstrates how the proposed deep kNN test can be applied to industrial visual inspection task.
Normal Image
Anomaly Image
Patch
Patch
550
5 C.O
Cactavtee
naive p 0.045 selective p 0.394
naive = 0.000 selective p 0.011
Figure 1: Examples of anomaly patches extracted from Capsule images using ANN-based AD shown (see $ [ for detailed settings). For both the normal image (left) and the anomaly image (right), two types of p-values derived from different statistical tests presented. The "naive p" represents the p-value obtained using a conventional method, while the "selective p" denotes the p-value computed using the method proposed in this study. At significance level of a 0.05, the conventional naive p-value for the normal image (left) falls below the threshold, resulting in a false positive detection. In contrast, the proposed selective p-value correctly identifies it true negative. For the anomaly image (right), both p-values fall below the threshold, correctly identifying the patch anomalous (true positive). In this study, we show that that conventional naive p-values are invalid as measures of statistical significance, whereas the proposed selective p-values serve as valid uncertainty measures for assessing the significance of anomalies detected by I:NN-based ADs.
Related Works Semi-supervised AD algorithms can generally be divided into three main cate- gories 94 1O]. The first group comprises AD methods based on parametric probabilistic models. These methods assume that normal data follow specific statistical distribution, such as multi- variate Gaussian. common traditional technique involves computing the Mahalanohis distance and assigning p-values using the x² distribution, allowing for statistical significance testing. These methods are interpretable and computationally efficient but may perform poorly with complex or Gau-Gassiian data distributions 12]. The second group includes AD methods based on classical machine learning algorithms. Representative methods include One-Clan SVM (13), Isolation For- est [14), and k-Nearest Neighbors (kNN) (15). While these methods are more flexible than parametric probabilistic poddel-based methods, they often lack principled mechanism to quantify statistical significance of the detected anomalies, making it difficult to assess confidence in AD. The third group comprises deep learning-based AD methods, which use neural networks to capture complex patterns normal data. Representative approaches include autoencoders or variational autoencoders (VAEs) [16. GAN- method [7], and diffusion-basee method [18] Deep SVDD [[] and deep

---

kNN-based AD [19] are also widely used as AD methods based on deep learning models. While these deep learning-bar based approaches often achieve high detection performance, most still lack a well-established framework for quantifying statistical significance, such as assigning p-values.
Selective Inference (SI) -also known post-selection inference- provides valid statistical inference for data-driven hypotheses. By leveraging the conditional distribution of test statistic given that particular hypothesis was selected based on the data, SI corrects the selection bias introduced by cherry-picking findings. This ensures that the reported p-values -known as selective p-values- maintain valid false positive rates, even when the same data is used for both hypothesis selection and testing, which would otherwise inflate the error rates due to double dipping (7). Early work in SI primarily focused on feature selection in linear regression. A seminal contribution by Lee al (8) introduced an exact SI procedure for the Lasso, deriving valid selective p-values for selected regression coefficients by conditioning on the selection event induced by the Lasso solution. Since then, extensive research has extended SI to various other feature selection settings, including marginal screening (20).
stepwise feature selection [2T], generalized linear models [22], and many others 23) Recent developments have also focused on improving the power of SI methods through new theoretical insights and algorithmic innovations [34/38). In parallel, SI has been adapted to problems beyond feature selection, finding applications in range of domains, such clustering [39 and many others (43 47). In the context of deep learning, SI has been applied to provide statistical inference for segmentation tasks [48]. saliency maps such CAM [49]. and attention weights in Vision Transformers [50]. In recent years, this line of research has begun to explore the reliability of various deep learning model components from a statistical inference perspective (31)53). In contrast, applications of SI in AD settings remain limited. One related work is the application of SI to change- point detection in time series data (54 58). However, that line of research focuses on testing the significance of global changes in the entire sequence, which differs from our setting where the goal to assess the abnormality of individual data points. Another related work is the application of SI to robust regression [59], where inference is performed on regression coefficients obtained after excluding outliers. However, the statistical significance of the detected outliers themselves is not addressed. In conclusion, no studies have yet explored the SI framework to quantify the statistical significance of kNN-based AD, and its deep &NN variants remain entirely unexplored, leaving a significant gap in the literature.
Our contributions In this paper, as a proof of concept, we address the problem of quantifying the statistical significance of results produced by simple deep k-N-baaeed anomaly detection (AD) approach, as studied in [9). where anomalies are identified by thresholding the kNN distance in the latent feature space of CNN classifier. Unlike prior studies on deep -NN-based AD, our contribution is not the development of new AD algorithm aimed at improving detection accuracy, but rather the introduction of method to quantify the statistical uncertainty of detected anomalies. Our first contribution is to formulate kNN-based semi-supervised AD as a statistical test within the SI framework, enabling rigorous reliability assessment of detected anomalies. The second contribution to develop a computational method to incorporate the distance measure derived from a deep learning model into the SI framework. Finally, we validate the effectiveness of the proposed deep-kNN various datasets and industrial inspection scenarios, demonstrating its practical utility and robustness. Furthermore, to facilitate reproducibility and further research, we release an open-source implementation of our proposed method, including code and exper- imental scripts, available at lattps://githu/ com/Takeuchi -aw--------------/---------- Stw.---oo/ooouutiitiiiiiie Statistical Sarill__nnnntttnteeee of Deep Nearest Neighbor Anomaly Detectioeeetono via SI.
test through experiments
Problem setup: deep --Ndd anomaly detection
In this section, we describe the problem setting of simple deep ANN-based AD, nearly identical to that in [19]. as a proof of concept for our statistical testing framework. In semi-supervised AD problems, the available training dataset consists only of the set of normal instances. Let ......, Fn e Rd represent the set of d-dimensional input feature vectors for n normal training instances, where n is the number of instances. If an instance is an image, for example, d is the number of pixels in the image and I,, é E [n] is the d-dimensional vector of pixel values. We assume that a preterined CNN is available, and let us denote its feature representation as 6( b(x;), i E [r]] using feature extractor
Φ Rd + D. where D is the dimension size of the latent feature vectors. We measure the distance
3

---

between two imput instances x' E Id in the latent feature space of a pre-trained CNN classifier as
distø(z, x²) (| <0<')(())))))) 1l2
(1)
where we adopt the T- distance within the latent feature space this study for concreteness, although the choice of distance metric is flexible and discussed further in the 6
Given test instance œt Rd, the kNN-based AD is formulated as follows. Consider an order of indices by ascending distance with respect to a distance function disto(-.-) such that
dist o test (o(1)) S dist test To S ... S dist. test Fo(n))
(2)
Then, (o(()) is called the th nearest neighbor instance of x: test, Since the choice of it affects the distance magnitude, we adopt the following well-known anomaly score 60]:
log k D
a(x
Best) log dist (zlest, (zl(et,,o(())
(3)
where the first term represents the loalscale distance, whereas the second term adjusts for the influence of k's selection" In #NN-based AD, if Eq. (3) exceeds certain threshold 6, the test instance tesst is selected as an anomaly. The threshold 0 is typically determined based on the empirical distribution of anomaly scores among normal instances. The choice of k greatly affects the results in
AD. Users can set i based on domain knowledge or experience. However, when domain know ledge is limited data is complex, a systematic approach is desirable. In semi-supervised AD, unlike supervised learning such as ANN classification or regression, it is not possible to determine & through data splitting such cross-validation. One commonly used heuristic select k is to calculate the anomaly scores for various k values per test instance tt choosing the k that maximizes this score. Our proposed &NN-test is valid regardless of the method used to determine k.
3 Statistical testing framework for ANN-base AD
In this section, we formulate the kNN-based AD problem as a statistical hypothesis testing problem in order to quantify the statistical significance of the detected anomalies. First, consider each input feature vector æi, i E [nx] as a realization of a random vector X,, E [nx] The statistical model for the random vector X; is written as
X, = 5, Es, E [rz].
where 8; E Rd represents the signal component, and e; E Ied represents the noise component. In parametrri assumption for the statistical model described in Eq.
this study, we adopt Specifically, we place no restrictions on the distribution of the signal components 8; for i E [n) treating them in a completely -arrameetiiic fashion. In contrast, we assume that the noise com- ponent follows a Gaussian distribution ((0 r²)), where o² either known or can be estimated from an independent dataset. This setup differs from traditional AD approaches based parametric probabilistic models, which assume that the signal components follow a specific parametric dis- tribution. In semi- remi-parametric framework, the distribution of the signal components is entirely unknown and unrestricted, allowing the model to remain valid even when the signals exhibit complex or multimodal characteristics. For example, if the input feature vectors I; represent images each element corresponding to a pixel value- then our model in Eq.(4) allows a set of arbitrary original images each of which is contaminated by Gaussian noise. Our goal is to determine, within statistical hypothesis testing framework, whether an observed anomaly originates from the underlying signal or merely a consequence of the noise.
Let the fcature vector of a test instance be denoted as test and its corresponding random version as ttest We assume X k-nearest instance Tock) is selected from the n available training instances. Our interest in the test is determine whether the signal at the representative instance of k nearest neighbor points K. = (Xx(1) Xo(2). Xo(1 is statistically different from the signal of the test instance.
guast + e sest in the same way Eq. In bNN-based AD, the
x Fmed(k)
= argmin {æ, (SEK)
(at,
(5)
'The choice of Eq. 3 beyond the scope of this paper. For further information, refer to
is based on certain assumptions and heuristics in the literature, but its details are

---

and denote its corresponding random version as X (k) This problem can be formulated as hypothesis testing problem using the following null hypothesis Hg and alternative hypothesis H₁:
Ho 8 teat = mmd((k)
vS. H1 Lest * mmed(k)
(6)
The null hypothesis Ho states that the true signal of the k-medoid normal training instance equals the true signal of the test instance, while the alternative hypothesis H1 asserts they are different. By performing statistical test for these hypotheses, the false detection rate of anomaly can be quantified using p-values.
As reasonable test statistic, consider
1
T (xx test {X,)
X
ml)
V2
The p-value defined as the probability observing test statistic greater than or equal to the one actually observed under the null hypothesis Ho, i..,
p = PHo (T (Xtesst
test (ei))(()) 2 T (a: test (xi))
(8)
Unfortunately, the probability in Eq. ( is computationally intractable, as it depends on the complex computation process of deep kNN. To address this issue, we introduce the SI framework and define significance quantification measure called the selective p-values. In the following section, we present the concept of selective p-values and demonstrate that this measure can appropriately quantify the statistical significance of anomalies detected by i-N-bbsed AD.
4 Selective Inference (SI) for -Nse AD
In this section, we propose selective p-values, based on the framework of SI, as measure of the statistical significance of anomalies detected by k-N--based AD. These selective p-values can be interpreted in the same way as conventional p-values. For example, if the significance level is set to a 0.05 and we consider anomalies with selective p-values less than 0.05, it is theoretically guaranteed that the proportion falsely detected anomalies will remain below 0.05.
4.1 Alternative formulations of the test statistic in Eq.
First, let us denote the (1 + r.)d-dimensional vector obtained by concatenating the test instance x test and n training instance ......, n2, all of which are d-dimensional vectors,
y vec (x: 2,,...,, F.)) E R (1+)dd
where vec is the operation that concatenates multiple vectors into a single column vector. Similarly, random vectors is denoted as
the (1 + n.dd-dimensional vector obtained by concatenating
Y vec
X
E RR (R++n)
(10)
With these notations, we can rewrite the test statistic in Eq. as
T(Y)
|IPY 1|2,
(11)
where P E ReL++))xxL+x)) is symmetric block matrix with each block of size (d x d), defined as follows:
Ta O
Ta
0
O
O
0
P
(12)
2
-Ia O
Ia
:
0
Note that the only non-zero blocks are the identity matrix Ta in ((O, 0)), (med(() + 1, med(k) + 1)) and -I in ((0, med(k) + 1)), ((med(k) + 1, 0)), and all other blocks are zero matrices.
5

---

4.2 Naive p-values
Here, we discuss a measure referred to as the naive p-value. Although this measure is invalid as an indicator of statistical significance, it serves as a contrasting concept that helps introduce the notion of selective p-values. The naive p-value defined as
Pnaive PHo (l 12)
(13)
where we emphasize the distinction between the random vector Y and the observed vector y. Under the statistical model in Eq.(4) the random vector Y follows multivariate normal distribution. Therefore, the statistic T(Y) follows X distribution with d degrees of freedom. Consequently, Eq. (3) can be easily computed as the tail probability of x(d) distribution. Unfortunately, this easily computable naive p-value is invalid in the sense that it does account for the fact that the k-nearest neighbor instance To(k) is selected based on the same observed data. If naive p-values are used for decision-making in the same way ordinary p-values, the false detection rate cannot be properly controlled as intended.
4.3
Selective p-values
The basic idea of SI, pioneered by the seminal work by Lee al. (8) is to address the problem of the naive p-values by employing the framework of conditional testing, based on the key insight that the sampling distribution of test statistic be tractable if the statistic conditioned on.
In order to define selective p-value, us represent an event that the k-nearest neighbor index o(k) is selected based on the random vector Y as "Ey = o(k)". Then, selective p-value is defined as
Psclective Hoo (IPY 2 |yyle &y Ey. Qr =2y)
(14)
where the first condition "Ey Ey" indicates the event that the k-nearest neighbor index o(k) obtained from the random data vector Y the same as that obtained from the observed data vector y. The second condition "Qr Qy" indicates that the sufficient statistic of the nuisance parameter defined as
Qx = ( (IPPI] -Pltn
PY
(15)
where P = My E [[++))ddx(++ndd is the same for both the random data vector Y and the observed data vector y. Here, the key idea is, by the first conditioning on the k-nearest neighbor index o(k), 7/y is fixed as Myy making the computation of the probability in (14) tractable with the use of x distribution as with the case of the naive p-value. Due to space limitations, we omit
the mathematical details and statitical rationale of SI, including the role of the event related to the
nuisance parameter? For further details, we refer the reader to the literature such as 7 8 34
The computation of selective p-value in Eq. (14 is reduced to a tail probability computation of truncated x distribution formally stated in the following theorem.
Theorem 4.1. The following conditional test statistic
PY Ey
(16)
follows a truncated x distribution with (1 n)d degrees of freedom, where the truncation is determined by the constraint "Ex Ey and the domain of the distribution is the one-dimensional subspace defined by {Y Qr = )
The proof of Theorem 4.. is deferred Appendix A.2 Furthermore, the selective p-value in Eq. (14) is valid p-value in the sense that, for any significance level a E (0, 1), the false detection rate of the anomalies with Psolective < is exactly a formally stated in the following theorem.
In this context, the nuisance parameter refers to a parameter that included the null distribution but not of direct inferential interest. To characterize the null distribution, it is necessary to eliminate the influence of the nuisance parameter. In our approach, this is achieved by conditioning on its sufficient statistic. This is standard technique in the field of selective inference
34

---

Theorem 4.2. The selective p-values defined in Eq. satisfies
HHo (Prelective S a) a, Vo (0, 1).
(17)
The proof of Theorem deferred to Appendix A.2 The statement of Theorem 4.2 indicates that the selective p-value in Eq 14 can be used measure quantity the statistical significance of the detected anomalies.
4.4 Selection event characterization
The main technical challenge in computing selective p-values lies characterizing the selection event "Ey = Egy" Conditioning on this selection event means identifying the set vectors Y E ((++ such that the same normal instance [a(k). which was selected as the k-nearest neighbor based on the observed data vector y, is also selected as the k-nearest neighbor. To realize this, one must appropriately characterize and account for several factors, including the computation of latent feature distances between the test instance and
vectors in the trained CNN model and the comparison normal instances in the latent feature space. Due to space limitations, the detailed characterization of the selection event provided in Appendix B where we adopt the parametric programmingbased the core computational framework.
approach proposed in (88
5 Numerical Experiments
In this section, we demonstrate that the proposed method exhibits high power (true positive rate) while controlling the Type I error rate (false positive rate) below the significance level compared to other methods. First, experiments are conducted on synthetic datasets, followed by similar experiments on two types of real datasets. All experiments are conducted with a significance level a = 0.05. We examined the case where Φ,, mapping to a latent feature space, the identity function simple settings for the synthetic and tabular data experiments, while for the image data experiments, we considered the case where & is defined by DNN model. We executed all experiments on AMD) EPYC 9474F processor, 48-core 3.6GHz CPU and 768GB memory.
5.1 Baseline Methods
In the experiments synthetic datasets and tabular datasets, compared the proposed method (proposed) with four other baseline methods: Hotelling_tt t2, w/o a/o-pp. naive, and Bonf erroni. Subsequently, in the experiments on image datasets, additionally compare two further ablation studies: OpA1 and OpA2.
Hotelling_ t2: This method is not based on kNN but instead performs classical anomaly detection by computing a p-value using Hotellings T test. Unlike proposed, which assumes semi-parametric conditions on the training data as shown in Eq.(4). this method makes a stricter parametric assumption that all training samples are i.i.d. from a single Gaussian distribution. It uses the squared Mahalanobis distance 2 ttsst) as the test
statistic and computes the p-value as Photelling = IHH, (T²2 (X test 2 (x test)). where the null distribution assumed follow the x2 distribution.
Bonf erroni: We chose the Bonferroni erroni correction as the most basic multiple comparison method This is method to control the Type error rate by using the Bonferroni correction. There are (X) ways to choose the neighbors N,, then we compute the Bonferroni corrected
p-value Phonferroni min(1, (w) Pnaive).
naive: This is statistical test that uses the naive p-value defined in Eq. 13
w/o-pp: An ablation study that excludes the parametric programming technique described in Appendix B
OpA1: Another ablation study that excludes the selection events for ANN (i.c., My, Ny. Ky, and Sy in Appendix B).
0pA2: Another ablation study that excludes the selection events for DNN (i.c., Dy in Appendix B).
*Other multiple testing procedures, such as Holm's 's method, require computing nominal p-values for all possible hypotheses, which is intractable task due the combinatorial explosion.
7

---

5.2 Synthetic Datasets
To evaluate the Type I error rate, we varied the training dataset size n {100, 200, 500, 1000) and set the data dimension d = 5. The number of neighbors k was adaptively selected in data- driven manner from 21, 2, 5, 10}. See Appendix f.r results when and are varied. For each configuration, we conducted 1,000 independent experiments. In each iteration, instance are and training instances T; for E [n], sampled from the same distribution under one of the following two settings. In the parametric setting, all instances were drawn from standard Gaussian distribution (O, Iz)- the semi-parametric setting, each instance was drawn from N(s, Ia), where is randomly generated mean vector. For more details about data generation, see Appendix C.I evaluate power, we generated data in the same way, except that a signal 5 E (2, 4, 6, 8 was added to randomly selected coordinate of the test instance w/'est We set d 5, n = 100 and & was adaptively selected in the same way. See Appendix C.3 for results when d, k, and n are varied. The results of Type I error rate shown in Figure The proposed, w/o-pp. and Bonferroni successfully controlled the Type error rate under the significance level, whereas the naive could not. Since naive failed control the Type error rate, we excluded it from the power experiment. Hotelling. t2 also fails the semi-parametric setting (where 8; follows a non-Gaussian distribution), because it assumes that all samples i.i.d. from single Gaussian distribution. So, it is likewise excluded from the power experiment in the semi-parametrie setting. The results of power shown in Figure 3 Among the methods that controlled the Type I error rate, the proposed has the highest power.
generated test
1.0
1.0
proposed
proposed
w/o-pp
w/C-pp
Type Error Rate
0.8
Bonferrani
Type Error Rate
0.8
Barferranii
Hotelling_az 12
Hotelling_e 12
0.6
naive Significance Level (a=0.05)
0.6
haive
Signi Levll (a=0005)
0.4
0.4
0.2
0.2
0.0
0.0
100
200
500
1000
100
200
500
1000
Training Datasize
Training Datasize
(a) Parametric
(b) Semi-Parametris
Figure 2: Results of Type I error rate when changing the dataset size n. proposed, w/o-pp, and Bonferroni successfully control the Type error rate across all settings. Their lines are almost overlapping. naive fails and the results of Bonferroni are almost zero, because is too conservative. Hotelling_ t2 also fails in the semi- cemi-parametric setting.
1.0
1.0
proposed
proposed
0.8
w/o-pp
0.8
w/o-pp
Bonferroni
Bonferroni
Power
0.6
Motelling_te t2
Power
0.6
0.4
0.4
0.2
0.0
0.0
2
4
6
8
4
6
8
5
5
(a) Parametric
Semi-Parametrit
Figure 3: Power when varying signal strength 8. proposed and Hotelling ingg t2 outperformed other methods. However, botelling_tt t2 failed to control the Type error rate in the semi paramerrcttiic setting so is not shown in the power results.

---

proposed
we pp
Bonferron
Hotellng_
raive
OpA1
OpA2
Significance Level (x-0..5)
0.6
Power
1pp
04
0.4
02-
0.2
0.0
Heart
Money
Riee
Cancer
Credit
Student
Bantrupcy
Drink
Huddear
Netwark
0.0
Heart
Money
Cancer
Credit
Student
Bankrupcy
Drink
Nuztear
Network
Dotaset Nome
Dataset Nome
Type Error Rate Results on Tabular Data
Power Results on Tabular Data
1..
1.0
Rate
0.-
0.
1ypn|
0.4
02
1
0.0
Cameet
Gni
Leather
Tite
Wood
Bottle
Capsule
Metal Not
Tranistor
Zipper
Carpet
Grid
Leather
Tile
Wond
Bottle
Capsule
Motal Not
Transistor
Zipper
Name
Type Error Rate Results on Image Data
Power Results on Image Data
Figure 4: Results real datasets. The proposed method (proposed) outperformed the other methods in terms of power, while controlling the Type I error rate below the significance level across all datasets. The Type I error rate, power, and error bars of the conferroni almost zero, because too conservative.
5.3 Real Datasets I: Tabular Data
conducted evaluations using 10 tabular real-world datasets. These datasets reflect various real-world problems from different domains. The datasets used in our experiments are listed in Appendix C.4 Only numerical features from each dataset were used in the experiments. The datasets vary in dimensionality, ranging from 4 to 10 dimensions. The number of neighbors & was adaptively selected in data-driven manner from {1, 2, 5, 10}. Before conducting the experiments, All datasets standarized with each feature having mean 0 and variance 1. The results of the Type I error rate and power are shown in Figure 4 The proposed method outperformed the other methods in terms of power, while controlling the Type I error rate.
5.4 Real Datasets II: Image Data
The dataset consists of 15 classes, and we chose 10 classes for the experiments which seem to follow normal distribution. The datasets used in our experiments are listed in Appendix C.3] Before conducting the experiments, All datasets are standarized with each feature having mean and variance 1. Following the conventional Deep the ImageNet dataset as feature
In this experiment, used the MVTec AD dataset 66] 62
kNN approach [19]. we employed ResNet model pre-trained extractor in this experiment. As a preprocessing step, the original image, which has size of 900 x 900 1024 x 1024, was divided into 30 30 patches, and the patch For the training instances, we used 100 patches from the same position as the test instance. We set the number of neighbors The proposed method outperformed the other methods in terms of power, while controlling the Type I error rate below the significance level. Some examples of the experimental results are shown in Figure 5 and Figure 15
used as the test instance.
The results of the Type error rate and power shown Figure 4

---

Bottle
Capsule
Anunsay
Aeunally
aeomaty
Anornayy
800
SD0
SDD
CANEECERAL weadliy LOBE
whilhhergs 5.800
MONGRER 2.000
seocdrep GORKS
Leather
Metal
Anomaly
Largo Anemaly
Anemety
Large Anoewly
SIVVEOT & R022
AdVEa 8.990
GURNS PMACT
GUICE BRFNCIVOOS
Figure 5: Experimental results of 4 datasets from MVTec AD dataset. For each dataset, one normal example (left) and two anomaly examples (center, right) are showed. For each example, the top row displays the original image used for testing along with the patch location (marked in red), while the bottom row presents the extracted patch image. For all normal examples, the naive p-value below the significance level a = 0.05 (false positive), whereas the proposed selective p-value correctly results in a true negative. For all anomaly examples, the selective p-value successfully detects anomalies.
6 Scope, limitations and conclusions
In this study, we proposed a method for quantifying uncertainty in &NN-based AD by assessing statistical significance. Uncertainty quantification in the outputs of deep learning models remains a major challenge in machine learning community, and our work contributes toward addressing this gap. This is particularly important for AD because it is often used in high-stakes applications such as medical diagnosis and industrial inspection, where evaluating statistical significance is crucial for practical reliability. Therefore, our proposed method, which enables the quantification of statistical significance of the detected anomalies, has substantial practical importance. Our SI-based fundamental idea is not limited to the specific i:NN-based AD algorithm presented as a proof of concept in this study; it is also applicable to various other variants. For SI of deep ANN-based AD, it essential to characterize both the computations by the deep learning model and the selection event of the &NN instances. The former is applicable to many CNN-type networks, while the latter applies to distance metrics such as L1, L2, and Loo. is, in principle, extendable to other deep AD methods, as long as the selection event (i.e., Ey Ey in Eq 14 (4) can be characterized in a tractable form.
One limitation of the proposed KNN test lies in the form of the cemi-parametric model discussed in Section 3 In this model, the signal components are entirely non non-parametric, offering significantly greater flexibility than conventional statistical AD. However, the need assume a distribution for the noise component remains limitation. Additional experimental results on robustness when the noise distribution deviates from the normal distribution are presented in Appendix C, in which we observe that when the deviation is small, the false detection rate can be maintained at approximately the desired level. Another limitation arises when the selection event becomes more complex, rendering the current framework inapplicable in its existing form. This issue, for instance, occurs when
10

---

Transformer is used as the deep learning model for identifying the latent feature space. This remains an important challenge for our future work.
Acknowledgments and Disclosure of Funding
This work was partially supported by JST CREST (JPMJCR2ID3, JPMJCR22N2), JST Moonshot R&D (JPMJMS2033-055, RIKEN Center for Advanced Intelligence Project, and JSPS KAKENHI Grant Number 15080.
References
[1] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning. pages 4393-4402. PMLR, 2018.
[2] Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, and Ehsan Adeli. Adversarially learned one-class classifier for novelty detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3379-3388, 2018.
[3] Samet Akcay, Amir Alapour-Abarghouei and Toby Breckon. Ganomaly: Semi-supervised anomaly detection via adversaria training. In Asian conference on computer vision, pages 622 637. Springer, 2018.
[4] Kishan Mehrotra, Chilukuri K Mohan, HuaMing Huang, Kishan G Mchrotra, Chilukuri K Mohan, and HuaMing Huang. Anomaly detection. Springer, 2017.
[5] Shuai Lyu, Dongmei Mo, and Wai keung Wong. Reb: Reducing biases in representation for industrial anomaly detection. Knowledge-Batt Based Systems, 290:1 1563, 2024.
[6] Nikolaus Kriegeskorte, W Kyle Simmons, Patrick SF Bellgowan, and Chris I Baker. Cireu- lar analysis in systems neuroscience: the dangers of double dipping. Nature neuroscience, 12(5))535-54) 540, 2009.
[7] Jonathan Taylor and Robert J Tibshirani. Statistical learning and selective inference. Proceedings of the National Academy of Sciences, 112(25):762-76333 2015.
[8] Jason D Lee, Dennis L Sun, Yuekai Sun, and Jonathan E Taylor. Exact post-selection inference, with application to the lasso. Annals of Statistics, 2016.
[9] Sridhar Runaswamy, Rajeev Rastogi, and Kyuseok Shim. Efficient algorithms for mining outliers from large data sets. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 427-438, 2000.
[10] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93-104, 2000.
[11] Bo Du and Liangpei Zhang. discriminative metric learning based anomaly detection method. IEEE Transactions on Geoscience and Remote Sensing, 52(1 18:844 6857, 2014.
[12] Thibaud Ehret, Axel Davy, Jean-Michel Morel, and Mauricio Delbracio. Image anomalies: review and synthesis of detection methods. Journal of Mathematical Imaging and Vision, 61:710-743, 2019.
[13] Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex Smola, and Robert C Williamson. Estimating the support a high-dimensional distribution. Neural computation, 137))1443-1171, 2001.
[14] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining, pages 413-422 IEEE, 2008
[15] Fabrizio Angiulli and Clara Pizzuti. Fast outlier detection in high dimensional spaces. In European conference on principles of data mining and knowledge discovery, pages 15-27. Springer, 2002.
[16] Jinwon and Sungzoon Cho. Variational autoencoder based anomaly detection using recon- struction probability. Special lecture on IE, 2(1):1-18, 2015.
11

---

[17] Mikael Sabuhi, Ming Zhou, Cor-Paul Bezemer, and Petr Musilek. Applications of genera- tive adversarial networks in anomaly detection: A systematic literature review. leee Access, 9:161003-161029, 2021.
[18) Julia Wolleb, Florentin Bieder, Robin Sandkühler, and Philippe C Cattin. Diffusion models for medical anomaly detection. In International Conference on Medical image computing and computer-assisted intervention, pages 35-45. Springer, 2022.
[19] Liron Bergman, Niv Cohen, and Yedid Hoshen. Deep nearest neighbor anomaly detection. arXiv preprint arXiv:2002. 10445, 2020.
[20] Jason D Lee and Jonathan E Taylor. Exact post model selection inference for marginal screening. Advances in neural information processing systems, 27, 2014.
[21] Ryan J Tibshirani, Jonathan Taylor, Richard Lockhart, and Robert Tibshirani. Exact post- selection inference for sequential regression procedures. Journal of the American Staristical Association, 11 111(514):600-44
2016.
[22] Jonathan Taylor and Robert Tibshirani. Post-selection inference for-penalized likelihood models. Canadian Journal of Statistics, 166,,, 2018.
[23] Joshua R Loftus. Selective inference after cross-validation. arXiv preprint arXiv: 15/ 1.08866, 2015.
[24] Ali Charkhi and Gerda Claeskens. Asymptotic post-selection inference for the akaike informa- tion criterion. Biometrika, 105(3):645-664 2018.
[25] Fan Yang, Rina Foygel Barber, Prateck Jain, and John Lafferty. Selective inference for group- sparse linear models. In Advances in Neural Information Processing Systems, pages 2469-2477, 2016.
[26] Shinya Suzumura, Kazuya Nakagawa, Yuta Umezu, Koji Tsuda, and Ichiro Takeuchi. Selective inference for sparse high-order interaction models. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 3338-3347. JMLR. org, 2017.
[27] Sangwon Hyun, Max Gsell, and Ryan J Tibshirani. Exact post-selection inference for the generalized lasso path. Electronic Journal of Statistics, 1253:10553 2018.
[28] Kazuya Sugiyama, Vo Nguyen Le Duy, and Ichiro Takeuchi. More powerful and general selective inference for stepwise feature selection using the homotopy continuation approach. In Proceedings of the 38th International Conference on Machine Learning. 2021.
[29] David Rügamer and Sonja Greven. Inference for 1 2-boosting. Statistics and computing, 50(2):279-288) 2020.
[30] Diptesh Das. Vo Nguyen Le Duy, Hiroyuki Hanada, Koji Tsuda, and Ichiro Takeuchi. Fast and more powerful selective inference for sparse high-order interaction model. arXiv preprint arXiv:2/06.00 04929, 2021.
[31] David Rügamer, Philipp FM Baumann, and Sonja Greven. Selective inference for additive and linear mixed models. Computational Statistics & Data Analysis, 167:107350, 2022.
[32] Snigdha Panigrahi, Peter W MacDonald, and Daniel Kessler. Approximate post-selective inference for regression with the group lasso. Journal of machine learning research, 24(79):1-49, 2023.
[33] Thang Loi Nguyen, Loe Duong, and Vo Nguyen Le Duy. Statistical inference for feature selection after optimal transport-based domain adaptation. In International Conference on Artificial Intelligence and Statistics. PMLR, 2025.
[34] William Fithian, Dennis Sun, and Jonathan Taylor. Optimal inference after model selection. arXiv preprint 1410.259. 2014.
[35] Yoshikazu Terada and Hidetoshi Shimodaira. Selective inference for the problem of regions via multiscale bootstrap. arXiv preprini arXiv: 171.0099 2017.
[36] Xiaoying Tian and Jonathan Taylor. Selective inference with a randomized response. The Annals of Statistics, 46(2):679-710, 2018
[37] Snigdha Panigrahi, Jonathan Taylor, and Asaf Weinstein. Bayesian post-selection inference in the linear model. arXiv preprint 1rXiv:1605.08824, 28, 2016.
12

---

[38] Vo Nguyen Le Duy and Ichiro Takeuchi. More powerful conditional selective inference for generalized lasso by parametric programming. The Journal of Machine Learning Research, 23(1):13544-3333
2022.
[39] Jason D Lee, Yuekai Sun, and Jonathan E. Taylor. Evaluating the statistical significance of biclusters. Advances in neural information processing systems, 28, 2015.
[40] Chihiro Watanabe and Taiji Suzuki. Selective inference for latent block models, electron. Stat, 15(1), 2021.
[41] Lucy L. Gao, Jacob Bien, and Daniela Witten. Selective inference for hierarchical clustering. Journal of the American Statistical Association, pages 1-11, 2022.
[42] Yiqun T Chen and Daniela M Witten. Selective inference for k-means clustering. Journal of Machine Learning Research, 24(152):1-41, 2023.
[43] Kosuke Tanizaki, Noriaki Hashimoto, Yu Inatsu, Hidekata Hontani, and Ichiro Takeuchi. Computing valid p-values for image segmentation by selective inference. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9553-9566 2020.
[44] Vo Nguyen Le Duy and Ichiro Takeuchi. Exact statistical inference for time series similarity using dynamic time warping by selective inference. arXiv preprint arXiv:2202.0633 06593, 2022.
[45] Toshiaki Tsukurimichi,
Inatsu, Vo Nguyen Le Duy, and Ichiro Takeuchi. Conditional selective inference for robust regression and outlier detection using piecewise- homotopy continuation. Annals of the Institute of Statistical Mathematics, 74(6):119--122 2022.
[46] Vo Nguyen Le Duy. Hsuan-Tiee Lin, and lchiro Takeuchi. Cad-da: Controllable anomaly detection after domain adaptation by statistical inference. In International Conference Artificial Intelligence and Statistics,, pages 1828-1836. PMLR, 2024.
[47] Tatsuya Matsukawa, Tomohiro Shiraishi, Shuichi Nishino, Teruyuki Katsuoka, and Ichiro Takeuchi. Statistical test for auto feature engineering by selective inference. In International Conference on Artificial Intelligence and Statistics. PMLR, 2025.
[48] Vo Nguyen Le Duy, Shogo lwazaki, and Ichiro Takeuchi. Quantifying statistical significance of neural network-based image segmentation by selective inference. Advances in Neural Information Processing Systems, 2022.
[49] Miwa Daiki, Vo Nguyen Le Duy, and Ichiro Takeuchi. Valid p-value for deep learning-driven salient region. In Proceedings of The International Conference on Learning Representations, 2023.
[50] Tomohiro Shiraishi, Daiki Miwa, Terayuki Katsuoka, Vo Nguyen Le Duy, Kouichi Taji, and Ichiro Takeuchi. Statistical test for attention maps in vision transformers. In Proceedings of the 4/st International Conference on Machine Learning, Proceedings of Machine Learning Research, 2024.
[51] Teruyuki Katsuoka, Tomohiro Shiraishi, Daiki Miwa, Vo Nguyen Le Duy, and Ichiro Takeuchi. Statistical test for generated hypotheses by diffusion models. arXiv preprint arXiv:2402.1178) 2024.
[52] Daiki Miwa. Tomohiro Shiraishi, Vo Nguyen Le Duy, Teruyuki Katsuoka, and Ichiro Takeuchi. Statistical test for anomaly detections by variational auto-encoders. arXiv preprint arXiv:2402.03724 2024.
[53] Teruyuki Katsuoka, Tomohiro Shiraishi, Daiki Miwa, Shuichi Nishino, and Ichiro Takeuchi. si4onnx: A python package for selective inference in deep learning models. arXiv preprint arXiv:2501.1444 17415, 2025.
[54] Sangwon Hyun, Max G' G'Sell, and Ryan Tibshirani. Exact post- inference for change- point detection and other generalized lasso problems. arXiv preprint 1rXiv:1606.0 1605.0355, 2016.
[55] Vo Nguyen Le Duy, Hiroki Toda, Ryota Sugiyama, and Ichiro Takeuchi. Computing valid value for optimal changepoint by selective inference using dynamic programming. In Advances in Neural Information Processing Systems, pages 11356-11367, 2020.
[56] Sean Jewell, Paul Fearnhead, and Daniela Witten Testing for a change in mean after change- point detection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 14(4):102-11144 2022.
13

---

[57] Tomohiro Shiraishi, Daiki Miwa, Vo Nguyen Le Duy, and Ichiro Takeuchi. Selective inference for change point detection by recurrent neural network. Neural Computation, 37(1):160-192.
2024.
[58] Akifumi Yamada, Tomohiro Shiraishi, Shuichi Nishino, Terayuki Katsuoka, Kouichi Taji, and Ichiro Takeuchi. Change point detection in the frequency domain with statistical reliability. Transactions on Machine Learning Research, 2025.
[59] Shuxiao Chen and Jacob Bien. Valid inference corrected for outlier removal, 2019.
[60] Don O Loftsgaarden and Charles Quesenberry.
nonparametric estimate a multivariate density function. The Annals of Muthematical Statistics, 36(3):1049-1051, 1965.
[61] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mytec ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9592-9600, 2019.
[62] Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger, and Carsten Steger. The mytec anomaly detection dataset: comprchensive real-world dataset for unsupervised anomaly detection. International Journal of Computer Vision, 129(4): 1030-105999 2021.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (12 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Our paper includes the discussion of limitations in 6
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We provided complete proofs all the theoretical claims in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
All the experiment settings are fully described.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We submitted the code as supplementary materials. In the experiments, we only used datasets that are publicly available or can be synthetically generated.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
/mips.cc/publlc/guiide wot.eedeee//oeeeemmmmmmoeo
wwt.hlmmmmiiioooooooiiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper provides all necessary training and test details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: This study is paper on new statistical testing method and properly discusses statistical significance of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We described the information on computer resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps//neurips.c httpr//.....cpullleccccceeecaaae
Answer: [Yes]
Justification: All the studies conducted under NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
This work, which focuses on statistical tests for kNN-based anomaly detection, aims to enhance the reliability of AI and has the potential to broadly influence the machine learning community. On the other hand, it does not present significant ethical concerns or foreseeable societal consequences because this work is theoretical and, as of now, has no direct applications that might impact society or raise ethical considerations.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
19

---

Justification:
This study is on the statistical reliability of Al and poses no risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We appropriately credited all the code, data, and models used in this study.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi wwwdee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
This paper does not release new assets. (Details of the experiments performed by the code for reproducibility provided in the supplementary material are given in the paper.)
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
20

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This study does not involve research with human subjects, including crowd- sourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve studies on human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
We did not LLMs in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy hatose//nerrrjs.cc/ooffeeeeeeeee for what should or should not be described.
21

---

A Proof of Theorems
A.1 Proof of Theorem 4.1
According to the condition on Ey = Ey and Qr = ey i.c, U(Y) = u(y) and V(Y) = V(y).
where
V(Y) = PY
|PY| E ([a+1)dd
u(Y) = (L(n+1)d P)Y E R(+
we have
u(Y) =(yy)
<> (l(n+1)d
PJY U(y)
+ Y =M(y) + V(Y)²
+ Y -1(y) V(y)= (Y V(v))
$ Y = a + bb,
where a U(y). b V(y), and
T(Y) ||PY||2
Then, have
(Y E R((nn)d
&3 = Ey, 2()) 2(y)}
=Y E p(l+n)d
[3 Ey, Y
bz, R}
=(Y be E (R(ta)d
Eatbz = Eyy E RR]
={Y bz E (((+n)d
2 2 Z}:
where 2 is the truncation region defined
= {= E R Eatb: = Eyy].
Therefore, by noting that ||s||2 is zero, we obtain
T(Y) (&y Ey: Q(Y) Q(v)) ()),,
where (P(t 2) is a truncated x-distribution with the degrees of freedom d, whose domain is the truncation region Z.
A.2 Proof of Theorem
The sampling distribution of the test statistic conditional on Ey Ey and Q(Y) Q denoted by T(Y) (yr Ey, Q(Y) 2(y)}
is truncated x- x-distribution with the degrees of freedom (1 n and the truncation region Z defined in Theorem 4.1 Thus, by applying the probability integral transform, under the null hypothesis,
Pselective {yy- Ey. Q(Y) Q(y)} Unif 1),
which leads to
PHo (Pselective Sa = Ey, Q(Y) = Q(v)) = Va E (0, 1). Next, for any a (0, 1), we have
PHo (Pselective S a Ex = Ey)
= -f PHo (Peolective 5 a Ey = Ey, = Q(Y)) PH, (Q(Y) = 2(Y) Er = dg)))(
Pro (2 Q(y) Er = Ey)dQ(y)
= a.
Therefore, we obtain the result in Theorem 4.2 as follows:
PHo (Peelective 5 aa Σ
IHo (Pselective S a Sa Eny) PHo (Ex En)
Ey
PHo (1x Ex)
E
22

---

B Selection Event Characterization
In this section, we characterize the selection events &y Ey of deep kNN-based anomaly detection (AD). The selection event of deep kNN-based AD consists of two components: the selection event related to the kNN-based AD, and the selection event related to the deep learning models that perform the transformation into latent features. The former is described in Appendix [B.]] and the latter in Appendix B.2 Finally, in Appendix B.3 we describe how to identify the data space that satisfies the selection event and how compute the selective p-values.
B.1 Selection Event for ANN Anomaly Detection
In the selection events of kNN-AD, it is necessary to consider events such as selecting the nearest instances, the anomaly score exceeding threshold, and determining based the data. In the following, we describe these events one by one. It is worth noting that all the events described below be collectively represented by set of linear inequalities, which facilitates the computation of truncation regions for the truncated normal distribution used in selective p-value calculations.
Selection event for 4thh nearest neighbor The anomaly detection decision based on Eq. (3) and the test statistic of Eq. (7) depend on the selection of the t nearest neighbor instance of the test instance X test Therefore, the condition on hth nearest neighbor instance is required. Specifically, by conditioning on
dist X test Xo(H)) 2 dist, (X test X Xo(Br)
(18)
for k' 1,...,k and
dist.p
X test X o(K)) S dist. Xtest test X Xo(k))
(19)
for = + ,,...,7 we can consider only cases where the k-the nearest neighbor is the same as the observed case. Hereafter, the conditions in Eq. (18) and Eq. 19) are collectively represented as Ny = Ny
Selection Events for k-Medoids The test statistic in Eq. also depends the selection of the k-medoid instance. Therefore, conditions on the k-medoid instance are also necessary. Specifically, using Eq. (5), it is expressed as follows:
dista (X,, X mod(k))
Σ
dist & (X,, K,j # med(k).
(20)
iEK/ (k))
NEK/U/
Hereafter, the condition in equation 20 is expressed as .My MMy
Selection event for anomaly score Since the statistical test is performed only on test instances selected in the AD, it is essential to consider the selection events associated with it. A test instance is selected and if its anomaly score, as defined in Eq. (3). exceeds a threshold 0. The condition for the anomaly score is written as
log k d
log dist (X X Xo(1))
0.
(21)
With the conditions 2q. we can characterize the selection event that the test case X test is selected in AD. Hereafter, the condition in 21) is represented as Ky Kay.
Selection event for data-driven selection of l: In the case of the data-driven option for determining the number of neighbors k. its effect must also be appropriately considered For example, consider the scenario where ky.... 1xx are candidate values for it, and the candidate that maximizes the anomaly score in Eq. (3 is selected. Let the selected k {KI,.... kx be denoted as k* Then, the selection event simply given by log dast test -Bo(R=) log >
selection event.
log !i (o(kk)) d' to the four selection events mentioned above, this event must also be incorporated as an additional condition. Hereafter, we denote this selection event Sy Sy-
lo AI [K]. In the case of data-driven option to determine k, in addition
23

---

B.2 Selection Event for Deep Learning Models
When using i--nerest neighbors AD with feature representations from a pre-trained deep learning model, the influence of the model should be considered as a selection event. SI for deep learning has been discussed in prior studies, and tools like the software facilitate the analysis of selection events in these models. In this study, we employ methods from earlier research to calculate selective p-values, taking into account selection events related to deep learning models. The basic idea in these methods involves decomposing the model into components and representing each as a piecewise linear function. For example, operations in CNN such as convolution, ReLU activation, max pooling, and up-sampling are represented as piecewise linear functions. In the experiment, we utilize the feature representation represented precisely composition of piecewise linear functions. We explain the selection events regarding the deep learning model that transforms an image instance 2; E R" to a latent feature vector 2, E IRd. We consider deep learning model that consists of sequential piecewise-lineat functions (e.g., convolution, ReLU activation, max pooling, and up-sampling). Obviously, the composite function of those piecewise-linear functions maintains its piecewise-liness nature. Thus, within a specific real space in Rd, the deep learning model simplifies to a linear function, which can be expressed as:
CNN model pre-trained on the ImageNet database. This model is
(n, = B + Wa; if af E,,
(22)
where B E Rd and W RRdxd represent the bias and weight matrices, and P C Rd is polytope where ØDL acts as linear function. The polytope can be characterized by set of linear inequalities. For details on computing these linear inequalities, see (53]. Let denote the of polytopes for all instances in Y as:
Dy = (P] x, YYYY,,E EP).
(23)
Hereafter, we denote the selection event as Dy Dy.
B.3
Computing Selective p-values
Based on the discussions in Appendix B.1 and B.2
selective p-values in (14 can rewritten as
follows:
Pselective == Poo (T(Y) 2 T(y) - -N,, My My, Ky, Sr Sy, Dr Dr
Dys 2r 2y)
(24)
Although the calculation of this selective p-values is complex, the marginalization of NY Ny. Dy = Dy. S, is effectively utilized and existing SI research methods. We specifi- cally use the parametric programming (pp)-based method from previous studies (38]. In SI, statistical inference is based on the probability measure within the subspace Z of the data space (l++a)d where selection event conditions are met. By conditioning on the selection event for the nuisance component, Qy = Qy Z reduces to a one-dimensional subspace (see Theorem 4.1 and its proof in Appendix A.Ip. The selection events are formulated as unions of intersections linear or quadratic inequalities, suitable when using L1 or L2 distances for k-nearest neighbors. 2 consists of finite number of intervals along line the (1 n) didddmmmnio space, and the pp-based method systematically enumerates all intervals that meet these conditions.
Since the noise is Gaussian, the statistic T(Y) under the null hypothesis Ho follows one- dimensional truncated Gaussian distribution within the subspace 2, comprising finite intervals along line. The selective p-value is calculated as the tail probability of this truncated distribution. Early SI research often simplified calculations by assuming Z as a single interval under additional conditions, which still controls the false detection probability but reduces detection power. In our problem, similar simplification can be considered by enforcing Z to be a single interval. In the experiments in we conduct an ablation study comparing this simple approach (denoted as w/o-pp) as one the baselines.
$
C Details of the Experiments
C.1 Details of Synthetic Data Generation
This subection provides additional details regarding the generation of synthetic datasets used in Section 5.2 We describe both the parametric and semi-parametric settings. To illustrate the two
24

---

data- generatiinn settings, we present in Figure the distributions of the training samples in two- dimensional example (d=2) 2). In the parametric setting, all samples are centered around the origin. In contrast, in the semi-parametric setting, the samples are distributed around different mean vectors 81 producing a mixture of Gaussian clusters. This visualization clarifies the structural difference between the two settings and the increased heterogeneity in the semi mani-parametric case.
(a) Parametric setting
(b) Semi canni--aramtricc setting
Figure 6: Visualization of the data-generation process in the parametric and semi-parametric settings
for d = 2. In the parametric case, all training samples are drawn from a single Gaussian distribution centered the origin. In the semi-paraen cemi-paraettric case, each training sample is drawn from a Gaussian distribution with randomly shifted mean vector s;, resulting in heterogeneous distribution.
C.2 Additional Type I Error Rate Results
We also conducted experiments investigate the Type error rate when the data dimension n, d and the number neighbors I were varied in the parametric and semi-parametric setting. Specifically, we varied d E (5, 10, 15, 20). default parameters as d = 5, k and way as in the experiments on synthetic datasets (5.)) The results shown Figures and 9
{1, 2, 5, 10} and n E {100, 200, 500, 1000), while setting the 100. In all cases, we generated the datasets in the same
To further assess the robustness of our method, we conducted experiments on datasets that deviate from the normal distribution. Specifically, data sampled from the exponentially modified Gaussian (EMG), generalized normal distribution (GND), skew normal distribution (SND), and Student's /- distribution. The degree of deviation from the normal distribution is quantified using the Wasserstein distance and we evaluate the Type error rate for each case by varying E {0.01, 0.02, 0.03, 0.04). The results are shown in Figure 10
C.3 Additional Power Results
We also conducted experiments investigate the power when the number of training data n, the data dimension d and the number of neighbors l are varied in the parametric and semi- eemi-parametric setting. We varied E {100, 200, 500, 1000), default parameters as 100, d = 5. k = 3 and signal strength additional experiments where n and was varied, considering the case where & was adaptively selected from E {1, 2, 10) in data-driven manner. In all cases, we generated the datasets in the
(5, 10, 15, 20) and & f1, 5, 10) while setting the
5. Furthermore, we conducted
same way as the experiments on synthetic datasets 5.2). The results are shown in Figures
13
and 14
25

---

1.0
proposed
proposed
w/o-pp
w/O-pp
Type Error Rate
0.8
Bonferreni
Type Emro Rate
0.8
Bonferroni
Hotelling_tt _2
Hotelling_L _2
naive
naive
Significance Level (a=0.05)
Sig Significance Level 0.0).
0.4
0.4
0.0
0.0
5
10
15
20
10
15
20
Data Dimension
Data Dimension
(a) Parametric
(b) Pami-Parametric
Figure 7: Results of Type error rate when varying the date dimension d. proposed, w/o-pp. and Bonfer erroni successfully control the Type eиror rate across all settings. naive fails and the results of Bonferroni are almost zero, because too conservative. Since Hotelling_ t2 does not involve a parameter k, its value remains unchanged. Hotelling_ t2 also fails in the semi- carammrratttric setting.
1.0
proposed
proposed
w/o-pp
w/o-pp
Type Error Rate
0.8
Bonferroni
Type Error Rate
0.8
Bonferroni
Hotelling_12
Hotelling
0.6
naive
naive
Significance Level (a=0.05)
Significance Level (a=0.05)
0.4
0.4
0.24
0.0
0.0
1
10
10
Number of Neighbors
Number of Neighbors
(a) Parametric
(b) Semi-Parametric
Figure 8: Results of Type error rate when varying the number of neighbors kt proposed, w/o- -pp, and Bonferroni successfully control the Type еттог rate across all settings. naive fails and the results of Bonf erroni are almost zero, because it is too conservative. Since Hotelling. t2 does not involve parameter k. its value remains unchanged in the both settings. In the semi-parametric setting, Hotelling ing. t2 fails control the Type error rate.
26

---

1.0
1.0
proposed
proposed
w/o-pp
w/o-pp
Type Error Rate
0.8-
Bonferroni
Type Error Rate
0.8
Bonferroni
Hotelling_t2
Hatelling_
0.6
naive
0.6
naive
Significance Level (a-0.05)
Significance Levet+a-00005
0.4
0.4
0.2
0.2
0.0
0.0
100
200
500
1000
Training Datasize
Training Datasize
(a) Parametric
(b) Semi-Paramette
Figure 9: Results of Type error rate when varying the number of datasize n. proposed, w/o-pp. and Bonf orroir successfully control the Type eπror rate across all settings. naive fails and the results of Bonferroni are almost zero, because is too conservative. Hotelling_ t2 also fails in the semi-parametric setting.
1.0
1.0
a 0.05
a 0.05
Type Error Rate
0.8
a 0.1
Type Error Rate
0.8
a 0.1
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.01
0.03 Wasserstein Distance
0.02
0.04
0.01
0.02
0.03
0.04
Wasserstein Distance
(a) EMG
(b) GND
1.0
1.0
0.05
a = 0.05
Type Error Rate
0.8
0.1
Type Error Rate
0.8
a 0.1
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.01
0.02
0.03
0.04
0.01
0.02
0.03
0.04
Wasserstein Distance
Wasserstein Distance
(c) SND
(d) Student's r-distribution
Figure 10: Results of Type error rate when varying the Wasserstein distance l. proposed success- fully control the Type error rate in both significance levels a
27

---

0.8
0.8-
proposed
proposed
Power
0.6
w/o-pp
Power
0.6
w/o-pp
Bonferroni
Bonferroni
0.4
0.4
Hotelling_tt
Hotelling_t2
0.2
0.2
200
500
1000
15
Training Datasize
Data Dimension
(a) Datasize
Data Dimension
0.8
proposed
Power
0.6
w/o-pp
Bonferroni
0.4
Hotelling 12
0.2
10
Number of Neighbors
(c) Number of Neighbors
Figure 11: These are in the parametric setting. Power for a fixed number of neighbors k. The results show the effect of varying the training dataset size n, the data dimension and k. Our proposed method (proposed) and Hotelling t2
t2 outperformed other methods across all settings.
proposed
w/o-pp
proposed
Bonferroni
Power
0.6
Power
0.6
w/o-pp
0.4
Bonferroni
0.4
0.2
0.2
0.0
100
200
só0
1000
15
20
Training Datasize
Data dimension
(a) Datasize
(b) Data Dimension
proposed
0.8
w/o-pp
Bonferroni
Power
0.6
0.4
0.2
0.0
Number of Neighbors
(c) Number of Neighbors
Figure 12: These are in the semi-pane cami-paraettric setting. Power for a fixed number of neighbors k. The results show the effect of varying the training dataset size n, the data dimension d, and k. Our proposed method (proposed) outperformed other methods across all settings.
28

---

proposed
0.8
0.8-
w/o-pp
Bonferroni
Power
0.6
Power
0.6
Natelling 12,
0.4
proposed
0.4
w/o-pp
0.2
Bonferroni
0.2
Hotelling_t? t2
100
200
500
1000
5
15
Training Datasize
Data Dimension
(a) Datasize
Data Dimension
Figure 13: These are in the parametric setting. Power for an adaptively selected number of neighbors k. The results show the effect of varying the training dataset size n and the data dimension d. Our proposed method (proposed) and Hotelling t2 outperformed other methods across all settings.
1.0
1.0
proposed
0.8
0.8
w/o-pp
Bonferrani
Power
Power
0.4
proposed
w/o-pp
Bonferroni
0.0
0.0
100
200
500
1000
5
10
15
20
Training Datasize
Data Dimension
(a) Datasize
(b) Data Dimension
Figure 14: These are in the semi-parametric setting. Power for an adaptively selected number of neighbors k. The results show the effect of varying the training dataset size and the data dimension d. Our proposed method (proposed) outperformed other methods across all settings.
C.4 Details of Tabular Datasets
We used the following 10 real datasets from the Kaggle Repository. All datasets are licensed under the CC BY 4.0 license.
Heart: Dataset for predicting heart attacks
Money: Dataset on financial transactions in a virtual environment
Fire: Dataset on fires in the MUGLA region in June
Cancer: Dataset related to breast cancer diagnosis
Credir: Dataset on credit card transactions
Student: Dataset related to student performance
Bankruptcy: Dataset on company bankruptcies
Drink: Dataset the quality of drinking water
Nuclear: Dataset pressurized nuclear reactors
Network: Dataset on anomaly detection in virtual network environments
29

---

C.5 Experimental Results on Image Data Examples
We evaluated proposed and naive on the 10 datasets from MVTec AD dataset. The datasets used in this study are Carpet, Grid, Leather, Tile, Wood, Bottle, Capsule, Metal Nut, Transistor, and Zipper. Examples except for those shown the Figure from each dataset are shown Figure [3] In each example, we present patches corresponding to true negative and true positive cases, along with both the naive p-value and the selective p- p-value. Carpet
Grid
Smalt Anomaly
Large Anemaly
Anemety
Large Anorely
sdrep CORD
JONCheR 0.000
eefecivoss 6.093
Tile
Transistor
Nomaal
Teall Anvesaly
Large Asersaly
Termol
Sesill Beornaly
Large Anoonoly
AAVESS BELLS
DEPER GUMER
Wood
Zipper
Normal
Small Anematy
Large Aneratyy
Normal
Smell Anemely
Large Anemely
Figure 15: Experimental results of 6 datasets from MVTec AD dataset. For each dataset, one normal example (left) and two anomaly examples (center, right) showed. For each example, the top row displays the original image used for testing along with the patch location (marked in red), while the bottom row presents the extracted patch image. For all normal examples, the naive p-value is below the significance level = 0.05 (false positive), whereas the proposed selective p-value correctly results in a true negative. For all anomaly examples, the selective p- p-value successfully detects anomalies.
30


Reviews:
[{"review_id": "Bq5Qa8IAlK", "reviewer": "Reviewer_baZz", "summary": "This paper studies semi-supervised anomaly detection, where the model is trained solely on normal data. It focuses on the existing \"deep k-NN AD\" method, which maps input data into a latent feature space using a pre-trained CNN, and then measures the \n  distances between a test sample and the training points. While the paper does not propose a new detection algorithm, its main contribution lies in introducing a statistical framework to assess the significance of detections made by deep k-NN. Specifically, the authors formulate the AD process as a hypothesis testing problem and propose a selective p-value based on the selective inference framework. This approach aims to control the false positive rate and correct for selection bias in the AD setting.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis paper identifies a fundamental challenge in deep k-NN-based anomaly detection that computing the conventional p-value is intractable. To address this, the authors propose an alternative: a selective p-value derived from the selective inference framework, which is computationally feasible, and thus capable of effectively quantifying the false positive rate. This is a meaningful contribution, as most anomaly detection research (especially those on the theoretical side) tends to focus primarily on overall misclassification rate. However, in many real-world applications such as cybersecurity, controlling false positives is often more critical.  I believe this work makes a significant contribution to the anomaly detection literature.\n\nWeaknesses:\n\n1. I am not very familiar with the issue of \"double-dipping\", which the authors cite as a primary motivation for adopting selective inference to quantify statistical significance in deep k-NN AD. I understand that \"double-dipping\" refers to using the same dataset for both hypothesis selection and testing, why this issue cannot be addressed by simply splitting the dataset? Since this work specifically studies semi-supervised AD which trained only on normal data, the lack of labeled anomalies shouldn't prevent such a split. It would help the general audience if the authors could explain more precisely why double-dipping is unavoidable in this context, and whether the challenge is specific to the k-NN approach or applies more broadly.\n\n2. Perhaps related to the above point, I understand that this work prioritizes minimizing the false positive rate, which is indeed critical in AD. However, the overall misclassification rate (including both false positives and false negatives) is also an important metric in AD research. I wonder if there is a trade-off between reducing the false positive rate and increasing the false negative rate in this setting? It would be helpful if the authors could comment on whether their method maintains a good balance, or if minimizing Type-I errors comes at the cost of missing true anomalies.\n\n3. While reading the numerical experiment results, I was particularly struck by the consistently zero Type-I error achieved by the Bonferroni correction. It seems almost too good to be true. Can the authors provide a discussion comparing their proposed method to Bonferroni correction beyond what is shown in the experiments? Are there advantages that are not immediately visible from the reported metrics? Otherwise, based solely on the experimental results, one could argue that Bonferroni performs better than the proposed method, and that there may be little justification for adopting the new approach. \n\n4. One thing of the synthetic data experiments left me confused. It appears that all the synthetic datasets were generated from Gaussian distributions. However, the authors mention that the Hotelling’s \n$t^2$ test fails to control the Type-I error because it relies on the Gaussian assumption. If the data is Gaussian, shouldn't the test perform well under its assumed conditions? \n\nMinor comment: There is a typo on line 18 on page 1. It should be \"The\" instead of \"Tsehe\".", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your feedback.\n\n> (Weaknesses-1) I am not very familiar with the issue of \"double-dipping\", which the authors cite as a primary motivation for adopting selective inference to quantify statistical significance in deep k-NN AD. I understand that \"double-dipping\" refers to using the same dataset for both hypothesis selection and testing, why this issue cannot be addressed by simply splitting the dataset? Since this work specifically studies semi-supervised AD which trained only on normal data, the lack of labeled anomalies shouldn't prevent such a split. It would help the general audience if the authors could explain more precisely why double-dipping is unavoidable in this context, and whether the challenge is specific to the k-NN approach or applies more broadly.\n\nIn our problem setting, anomaly detection is performed in the latent space of a pre-trained deep model. Our focus is on the *test phase*, where we input a test instance and evaluate its anomaly score based on its distance to nearest neighbors. Here, \"double-dipping\" arises because the same test instance is used twice: first to *select* whether it is anomalous (i.e., if its distance exceeds a threshold), and second to *test* whether that distance is statistically significant. This re-use violates the standard assumption of independence between selection and inference. Unlike other selective inference (such as feature selection), we cannot split the test instance (i.e., an image patch) for selection and inference. Our selective inference framework addresses this issue by conditioning on the selection of anomaly. This challenge is not specific to k-NN and applies broadly to anomaly detection relying on test-instance-derived statistics.\n\n> (Weaknesses-2) Perhaps related to the above point, I understand that this work prioritizes minimizing the false positive rate, which is indeed critical in AD. However, the overall misclassification rate (including both false positives and false negatives) is also an important metric in AD research. I wonder if there is a trade-off between reducing the false positive rate and increasing the false negative rate in this setting? It would be helpful if the authors could comment on whether their method maintains a good balance, or if minimizing Type-I errors comes at the cost of missing true anomalies.\n\nWe fully agree that in anomaly detection, a trade-off between false positive and false negative rates is inevitable. In high-stakes applications such as medical diagnosis or industrial inspection, it is essential to explicitly control this trade-off to ensure reliable decision-making. Our contribution lies in developing a statistically principled framework that enables such control. Specifically, our method computes p-values for individual instances and makes decisions based on a user-specified significance level (e.g., 0.05). As theoretically guaranteed in Theorem 4.2, this approach allows for strict control of the false positive rate below the desired threshold. Once the false positive rate is controlled at the specified level, we assess the trade-off by analyzing the statistical power, i.e., 1 − false negative rate (see Figs. 3 and 4). In summary, our method is not designed merely to minimize false positives, but rather to support decision-making under explicitly guaranteed control of the false positive rate under the trade-off.\n\n> (Weaknesses-3) While reading the numerical experiment results, I was particularly struck by the consistently zero Type-I error achieved by the Bonferroni correction. It seems almost too good to be true. Can the authors provide a discussion comparing their proposed method to Bonferroni correction beyond what is shown in the experiments? Are there advantages that are not immediately visible from the reported metrics? Otherwise, based solely on the experimental results, one could argue that Bonferroni performs better than the proposed method, and that there may be little justification for adopting the new approach.\n\nThe objective of statistical hypothesis testing is to control the Type I error rate (false positive rate) at a specified significance level, while maximizing statistical power as much as possible. Our experimental results indicate that the Bonferroni correction tends to be overly conservative, reducing the Type I error rate well below the specified significance level (0.05), which results in a substantial loss of statistical power (Fig. 4). This suggests that the Bonferroni correction may not be practical from the perspective of the aforementioned trade-off. In contrast, our proposed method offers the advantage of theoretically controlling the Type I error rate at the specified significance level while maintaining practically useful statistical power.\n\n> (Weaknesses-4) One thing of the synthetic data experiments left me confused. It appears that all the synthetic datasets were generated from Gaussian distributions. However, the authors mention that the Hotelling's $t^2$ test fails to control the Type-I error because it relies on the Gaussian assumption. If the data is Gaussian, shouldn't the test perform well under its assumed conditions?\n\nIn our problem setup, we distinguish between the *signal component* $s$ and the *noise component* $\\epsilon$ in the data, as described in Eq.(2). Hotelling's $T^2$ test assumes a fully parametric model in which the entire observation follows a multivariate Gaussian distribution. In contrast, our setting is *semi-parametric* in the sense that only the noise is assumed to follow a Gaussian distribution, while the signal is allowed to follow an arbitrary, potentially complex distribution. This semi-parametric setting is substantially less restrictive than that of the fully parametric setting. In the left panels of Figures 2 and 3, we simulate data under a parametric condition where the signal is Gaussian, and Hotelling's test performs as expected. In contrast, the right panels illustrate a semi-parametric condition where the signal is multimodal and non-Gaussian, leading to inflated Type-I error in Hotelling's method. We agree that this distinction may not have been sufficiently clear and will provide a more detailed explanation in the revised version.\n\n> Minor comment: There is a typo on line 18 on page 1. It should be \"The\" instead of \"Tsehe\".\n\nThank you for pointing this out. We will make the necessary correction in the camera-ready version."}, {"review_id": "tc3XHJ8GcT", "reviewer": "Reviewer_ZerL", "summary": "This work introduces a selective inference framework for quantifying the statistical significance of the a deep kNN anomaly detector. The proposed method studies what would the p-value of a test observation be if we assume that the signal component (which is modelled non-parametric) is the same as the k-nearest neighbour. Nonetheless, since the computation of this p-value is intractable because it'll mean to model a complex neural network, the authors use Selective Inference to achieve a tractable solution (by conditioning the significance test). Numerical examples are performed considering synthetic, tabular and image data, showing that the selective inference provides a good measure of statistical significance while having the greatest power among all of the other baseline methods. Overall, this work is an applied work about selective inference for the anomaly distribution task.", "questions": "1. Could you please elaborate on how you reached to $\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) + \\lVert P\\mathbf{Y}  \\rVert_2^2 \\mathcal{V}(\\mathbf{Y})$ andh why it is not actually $\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) + \\lVert P\\mathbf{Y}  \\rVert \\mathcal{V}(\\mathbf{Y})$ (as per above)? If indeed you made a mistake, how does that would affect your analysis? (If I'm making a mistake please ignore this question). \n\n2. How could this improve upon Anomaly Detection in industrial settings?\n\n3. Could you provide more results for a different image dataset? The MVTec is already solved.\n\n4. How would this analysis translate to other Anomaly Detection methods? Such as Teacher-Student models?\n\n5. Isn't Eq (4) the same as Eq (5)? But you say Eq (4) is intractable but Eq (5) can be computed. I might be wrong, but could you clarify this?\n\nI'd be happy to change my score if these questions are answered satisfactory, but I'm also going to consider other factors such as discussion with other reviewers and the main weaknesses I found.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The work stablishes a good motivation on why using Selective Inference for anomaly detection could give a valid statistical significance measure of a prediction for an anomaly detector. The mathematical framework is mostly well defined and explained, although there are some parts that lack rigorosity. The experiments show that Selective Inference is in fact a good method for getting an accurate p-value. Therefore, I think the work is technically solid and addresses an interesting problem.\n\nNonetheless, I still found some weaknesses that makes me lean towards a reject. I have three main criticisms: (1) the work is limited to a single method for anomaly detection (deep kNN); (2) it is not clear how this actually helps Anomaly Detection; and (3) I think there might be an error in the proof of Theorem 4.1, which could potentially invalidate some of the analsys . For (1), there are several methods for anomaly detection, and choosing a single method makes me question the significante of the work. Several methods have shown to work as good or better than deep kNN for AD, so it'll be great to discuss how SI could be applied to these methods (if it can). Additionally, the evaluation is carried out in rather limited datasets. I find the synthetic dataset not sufficiently strong to explore the gains of SI in real-life scenarios, and the analysis of a single method in the MVTec dataset is also underwhelming, since this dataset has been practicallty solved using different methods. For (2), I think that while the authors clearly state the motivation, it would be great to discuss how this actually could help in anomaly detection. Regarding (3), in the proof of Theorem 4.1, you have this:\n\n\\begin{equation}\n(I_{(n+1)d}-P)\\mathbf{Y} = \\mathcal{U}(\\mathbf{y})\n\\end{equation}\n\nThen we'll have:\n\\begin{equation}\n\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) + P\\mathbf{Y} \n\\end{equation}\n\nWhich would be equivalent to\n\\begin{equation}\n\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) + \\frac{\\lVert P\\mathbf{Y}  \\rVert P\\mathbf{Y}}{\\lVert P\\mathbf{Y} \\rVert}\n\\end{equation}\n\nWhich, according to your notation, it'd be:\n\\begin{equation}\n\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) + \\lVert P\\mathbf{Y}  \\rVert \\mathcal{V}(\\mathbf{Y})\n\\end{equation}\n\nBut you have \n\\begin{equation}\n\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) +  \\mathcal{V}(\\mathbf{Y}) \\mathbf{z} =  \\mathcal{U}(\\mathbf{y}) + \\lVert P\\mathbf{Y}  \\rVert_2^2 \\mathcal{V}(\\mathbf{Y})\n\\end{equation}\n\nI think there might be a problem here that probably would invalidate most of the analysis. \n\nAdditionally, I also think that the work could be explained better. For instance, adding more detail into where do the condfitions in the equation between lines 197 and 198 come from. I understand that you might refer to a detailed work about SI, and that;s fine, but for the sake of the analysis in this paper I think it is important to discuss it. Is that nuisance parameter the same for all the SI problems? finally, I think some qualitative examples in the main paper would be good. \n\nOther minor weaknesses:\n- It'd be a good idea to add quantitative results in the abstract\n- L18: Tsehe -> This\n- L36: tereby -> thereby, by making...\n- L66: It's important to mention why quantifying statistical significance for such AD methods would be important\n- L102: I think it's more self-supervised rather than semi-supervised, but I leave it to the author\n- L107: $x_i$ -> $\\mathbf{x}_i$\n- L123-124. Any ref aboyut the heuristic method for choosing k?\n- Please add numbering to *all* equations\n- Eq between L175 and L176: I think $\\boldsymbol{\\eta}_y$ depends on $k$ but it's neither   $\\boldsymbol{\\eta}_k$ or  $\\boldsymbol{\\eta}_y (k)$. I think there should be a way to explicitly show that it depends on k.\n- Fig 4: you cannot see some error bars, could you please improve this?", "was_revised": true, "final_justification": "The authors addressed my main concern about the derivation of Theorem 4.1, mainly regarded as a typo, which they claim it does not affect the performance. The authors also compromised to fix all the typos and provide further clarification in the final version of the paper. Therefore, I increase my original rating from kBorderline Reject to Accept.", "rebuttal": "We thank the reviewer for your valuable feedback and for carefully reading the contents of the appendix as well.\n\n> (Weakness 1-1) there are several methods for anomaly detection, ... \n\nWe agree with the reviewer that deep learning-based anomaly detection is rapidly evolving with many promising methods. Our goal, however, is to establish a statistically principled framework for controlling false positives at a desired significance level. To clearly demonstrate the effectiveness of selective inference, we chose deep kNN—a simple, widely used, nonparametric baseline—as a proof of concept. In principle, our framework can extend to other methods, provided the selection event (i.e., \\$\\varepsilon\\_{\\boldsymbol{Y}} = \\varepsilon\\_{\\boldsymbol{y}}\\$ in Eq.(6)) is well characterized. We will add a discussion on this generalizability in the revised version.\n\n> (Weakness 1-2) Additionally, the evaluation is carried out in rather limited datasets. ...\n\nWe agree that extending the evaluation to more diverse datasets would strengthen the empirical analysis. However, since our goal is not to develop SOTA anomaly detection methods, but rather to establish the theoretical foundation of selective inference for anomaly detection with guaranteed Type I error control (see Theorem 4.2), we chose the MVTec dataset, which is the most widely used benchmark in this field. We believe that experiments presented in the main paper and appendix --- including those on the MVTec dataset and ten tabular datasets shown in Figure 4 --- serve to empirically validate the correctness and robustness of our theoretical claims.\n\n> (Weakness 2) I think that while the authors clearly state the motivation, it would be great to discuss how this actually could help in anomaly detection. \n\nOur goal is not to develop a more accurate anomaly detection algorithm, but to offer a statistically principled framework for controlling false positives. Anomaly detection inherently involves a trade-off between false positives and false negatives, which is critical in high-stakes areas like medical diagnosis or industrial inspection. Our framework supports such controlled decision-making by computing p-values for individual test instances and making decisions based on a user-defined significance level (e.g., 0.05), thereby ensuring that the false positive rate stays below the target threshold (see Theorem 4.2). We will clarify this practical benefit in the revised version.\n\n> (Weakness 3) in the proof of Theorem 4.1, you have this: ...\n\nWe apologize for the confusion caused by our typo. There is indeed the following typographical error in Appendix A.1, line 834:\n\n* Incorrect：$z = T(\\boldsymbol{Y}) =||\\boldsymbol{\\eta}\\_{\\boldsymbol{Y}}^T \\boldsymbol{Y}||\\_2^2 = ||P \\boldsymbol{Y}||\\_2^2$  \n* Correct：$z = \\sqrt {T(\\boldsymbol{Y})} = ||\\boldsymbol{\\eta}\\_{\\boldsymbol{Y}}^T \\boldsymbol{Y}|| = ||P \\boldsymbol{Y}||$  \n\nWith this correction, the equation pointed out by the reviewer becomes valid as follows:\n\n$$\\mathbf{Y} = \\mathcal{U}(\\mathbf{y}) +  \\mathcal{V}(\\mathbf{Y}) \\mathbf{z} =  \\mathcal{U}(\\mathbf{y}) + \\lVert P\\mathbf{Y}  \\rVert \\mathcal{V}(\\mathbf{Y})$$\n\nWe will correct this typo in the camera-ready version. We would like to emphasize that this correction does not affect the conclusions of the paper.\n\n> (Weakness 4) Additionally, I also think that the work could be explained better ...\n\nThe statistic between lines 197 and 198,\n$$\n\\mathcal{Q}\\_{\\boldsymbol{Y}} = \\left( \\frac{P\\boldsymbol{Y}}{\\lVert P\\boldsymbol{Y} \\rVert},\\ (I\\_{(n+1)d} - P)\\boldsymbol{Y} \\right),\n$$\nis derived from the sufficient statistic for the nuisance parameter. The concept of a nuisance parameter is fundamental in statistics; in this context, it refers to a parameter that is included in the null distribution but is not of direct inferential interest. To characterize the null distribution, it is necessary to eliminate the influence of the nuisance parameter. In our approach, this is achieved by conditioning on its sufficient statistic. This is a standard technique in the field of selective inference ; for more details, please refer to Section 3.1 of [1]. Although the formulation of $\\mathcal{Q}$ may differ depending on the problem setting, the one we use is widely adopted in prior SI literature. For example, the quantities $\\text{dir}\\_{\\mathcal{L}}(Y)$ and $\\mathcal{P}\\_{\\mathcal{L}}^{\\perp}Y$ in Section 2 of [2] correspond to our formulation of $\\mathcal{Q}\\_{\\boldsymbol{Y}}$. In the camera-ready version, we plan to add further explanations about nuisance parameters and how they are treated in the selective inference literature.\n\n[1]https://arxiv.org/pdf/1410.2597  \n[2]https://arxiv.org/pdf/1607.08211\n\n> (Weakness 5) finally, I think some qualitative examples in the main paper would be good.\n\nRegarding qualitative results, we currently present a variety of examples using the MVTec dataset in Appendix C.4. In the camera-ready version, we plan to consider moving a subset of these results into the main paper.\n\n> (minor weakness 1) L66: It's important to mention why quantifying statistical significance for such AD methods would be important\n\nOur proposed method enables rigorous control of the false positive rate in anomaly detection by quantifying statistical significance through hypothesis testing. This property is particularly important in high-stakes applications such as medical diagnostics and industrial quality control, where false positives may lead to substantial risks or costs. We intend to clarify this point more explicitly in the camera-ready version.\n\n> (minor weakness 2) L102: I think it's more self-supervised rather than semi-supervised, but I leave it to the author\n\nIn the problem setting of anomaly detection, there are three possible scenarios: 1) when both normal and anomalous instances are available, 2) when only normal instances are available, and 3) when neither normal nor anomalous instances are available. The second scenario is sometimes referred to as semi-supervised anomaly detection in the literature, and we follow this convention in this paper.\n\n> (minor weakness 3) L123-124. Any ref about the heuristic method for choosing k?\n\nIn this paper, we adopt a heuristic approach in which the value of $k$ is selected to maximize the anomaly score. This strategy has also been employed in previous studies on anomaly detection. For example, Eq (3) in reference [3] adopts the same heuristic.  \n\n[3]https://www.sciencedirect.com/science/article/pii/S0375674221001540\n\n> (minor weakness 4) Fig 4: you cannot see some error bars, could you please improve this?\n\nAs correctly noted, the error bars for the Bonferroni method in Fig 4 are barely visible. This is due to the highly conservative nature of the Bonferroni correction, which yields values so small that the corresponding error bars become negligible or vanish in the plot. We will address this issue in the camera-ready version by improving the visualization or providing clarification in the figure caption.\n\n> Other minor weaknesses\n\nRegarding the other minor weaknesses, we fully agree with the reviewers' comments and will properly address all of them in the revised version.\n\n>(Question 1)\n\nSee our response to Weakness 3.\n\n>(Question 2)\n\nSee our response to Weakness 2.\n\n>(Question 3) \n\nSee our response to Weakness 1-2.\n\n>(Question 4)\n\nSee our response to Weakness 1-1.\n\n>(Questions5). Isn't Eq (4) the same as Eq (5)? ... \n\nEq (4): \n$$\np = P\\_{\\rm H\\_0}(T(\\boldsymbol{X}^{\\text{test}}, \\{ \\boldsymbol{X}\\_i \\}\\_{i \\in [n]} ) \\geq T(\\boldsymbol{x}^{\\text{test}}, \\{ \\boldsymbol{x}\\_i \\}\\_{i \\in [n]})),\n$$\ncan be rewritten using the same notation as in Eq (5) as follows:\n$$\np = P\\_{\\rm H\\_0} \\left( \\lVert \\boldsymbol{\\eta}\\_{\\boldsymbol{Y}} \\boldsymbol{Y} \\rVert\\_2^2 \\geq \\lVert \\boldsymbol{\\eta}\\_{\\boldsymbol{Y}} \\boldsymbol{y} \\rVert\\_2^2 \\right).\n$$\nHere, $\\boldsymbol{\\eta}\\_{\\boldsymbol{Y}}$ is a random variable that depends on the deep $k$-NN inference, making the null distribution difficult to characterize analytically. In contrast, Eq (5) is given by:\n$$\np = P\\_{\\rm H\\_0} \\left( \\lVert \\boldsymbol{\\eta}\\_{\\boldsymbol{y}} \\boldsymbol{Y} \\rVert\\_2^2 \\geq \\lVert \\boldsymbol{\\eta}\\_{\\boldsymbol{y}} \\boldsymbol{y} \\rVert\\_2^2 \\right),\n$$\nwhich corresponds to a incorrect but intuitive approach. This naive method neglects the randomness of $\\boldsymbol{\\eta}\\_{\\boldsymbol{Y}}$ by treating it as a fixed quantity $\\boldsymbol{\\eta}\\_{\\boldsymbol{y}}$. In this naive formulation, $\\boldsymbol{\\eta}\\_{\\boldsymbol{y}} \\boldsymbol{Y}$ can be viewed as a linear transformation of the Gaussian variable $\\boldsymbol{Y}$, and hence also follows a Gaussian distribution. This allows for the null distribution to be easily characterized."}, {"review_id": "cdbIfmUvkR", "reviewer": "Reviewer_Awbd", "summary": "This paper applies selective inference techniques to KNN anomaly detection, aiming to provide statistical guarantees for the detection process. The authors present a methodological framework, prove supporting theorems, and evaluate the approach on synthetic and real-world datasets.", "questions": "Please see the weakness part.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe paper clearly articulates its core contribution: establishing a selective inference framework tailored for kNN-based anomaly detection, supported by theoretical results and empirical validation. The inclusion of asymptotic results and experiments helps bridge a gap in the statistical understanding of kNN anomaly detection methods.\n\nWeaknesses:\n\nThe theoretical contributions appear limited in novelty. The paper primarily establishes asymptotic distributions under standard Gaussian assumptions, following a relatively well-established framework. It does not clearly demonstrate the unique advantage of the proposed selective inference method theoretically over alternatives such as empirical p-value–based methods or bonferroni correction or multiple testing techniques like FDR.\n\nThe Gaussian assumptions throughout the theory are too strong. This restricts the generality of the results and may undermine applicability to real-world anomaly detection settings, where sub-Gaussian, sub-exponential, or heavy-tailed distributions are common. This issue also extends to the synthetic experiments, which are very limited to parametric and semi-parametric settings under Gaussian assumptions.\n\nMoreover, the paper mentions general applicability of the SI framework across learning methods, yet does not sufficiently articulate what makes this adaptation to kNN unique. This raises questions about the novelty and specificity of the proposed approach.\n\nCertain components, such as the anomaly score in Equation (1), appear to be adopted from existing literature without strong motivation or justification. Similarly, the test statistic used is tightly coupled to the Gaussian assumption without any data-driven adjustment, further weakening its practical utility.", "was_revised": true, "final_justification": "The rebuttal addressed my concern.", "rebuttal": "We thank the reviewer for your feedback.\n\n>(Weakness-1) The theoretical contributions appear limited in novelty. The paper primarily establishes asymptotic distributions under standard Gaussian assumptions, following a relatively well-established framework. It does not clearly demonstrate the unique advantage of the proposed selective inference method theoretically over alternatives such as empirical p-value–based methods or bonferroni correction or multiple testing techniques like FDR.\n\nFirst of all, we emphasize that Theorem 4.2 provides a finite-sample guarantee of Type I error control, which distinguishes our framework from many existing asymptotic approaches. Our Gaussian assumptions are also not overly restrictive (please refer to our response to Weakness-2 for further details). Regarding comparisons with alternative methods, we experimentally demonstrate that our approach achieves higher statistical power than Bonferroni correction, which is known to be overly conservative and often results in reduced sensitivity (see Figures 3 and 4). Unlike Bonferroni, which lowers the Type I error rate at the expense of power, our selective inference framework controls the Type I error rate exactly at the user-specified level, as guaranteed by Theorem 4.2. We selected Bonferroni as a baseline because it is applicable when the number of hypotheses is known. In contrast, other multiple testing procedures, such as Holm's method, require computing nominal p-values for all possible hypotheses, which is an intractable task due to the combinatorial explosion. Furthermore, since we aim to determine whether a single test instance is anomalous, permutation-based tests are not applicable. We will clarify these points in the revised manuscript.\n\n> (Weakness-2) The Gaussian assumptions throughout the theory are too strong. \n\nWe would like to clarify that our framework does not assume the overall distribution of normal instances to be Gaussian. As described in Eq.(2), our setting is semi-parametric in the sense that we make no distributional assumptions on the signal component, which can follow an arbitrary and potentially complex distribution. The Gaussian assumption is imposed only on the additive noise component. Unlike fully parametric tests such as Hotelling's $T^2$, which assume the entire observation is Gaussian, our assumption is significantly less restrictive. Furthermore, to assess the robustness of our method under noise model misspecification, we provide additional experiments in Appendix C.2 (Figure 8), where the noise component follows non-Gaussian distributions. These experimental results indicate that our method remains effective and continues to control the Type I error rate even when the Gaussian noise assumption is violated.\n\n> (Weakness-3) Moreover, the paper mentions general applicability of the SI framework across learning methods, yet does not sufficiently articulate what makes this adaptation to kNN unique. This raises questions about the novelty and specificity of the proposed approach.\n\nOur main contribution is to introduce a statistically principled framework for deep learning–based anomaly detection that explicitly controls the trade-off between false positives and true positives. To the best of our knowledge, no existing work provides such a general and reliable framework in this context. We chose the deep k-NN method as a proof-of-concept because it is simple, widely used, and allows us to clearly illustrate our framework's utility. While our selective inference approach is general in principle, adapting it to other anomaly detection methods requires characterizing the selection event specific to each algorithm (i.e., $\\varepsilon\\_{\\boldsymbol{Y}} = \\varepsilon\\_{\\boldsymbol{y}}$ in Eq (6)), which involves nontrivial engineering effort. Building a systematic pipeline for applying our framework across broader classes of deep anomaly detectors is an important direction for future work.\n\n>(Weakness-4) Certain components, such as the anomaly score in Equation (1), appear to be adopted from existing literature without strong motivation or justification. \n\nThe anomaly score used in our study is based on a classical definition. Specifically, it corresponds to the log-transformed version of Eq (3.2) in [1] up to constant terms. We will explicitly clarify this point in the camera-ready version.  \n\n[1] Loftsgaarden and Quesenberry. A Nonparametric Estimate of a Multivariate Density Function. Ann. Math. Statist. 36(3): 1049-1051 (1965)\n\n> (Weakness5) Similarly, the test statistic used is tightly coupled to the Gaussian assumption without any data-driven adjustment, further weakening its practical utility.\n\nWe are afraid that there may be a misunderstanding regarding the role of the Gaussian assumption and the nature of our test statistic. As mentioned in our response to Weakness-2, our method does not assume Gaussianity for the data distribution (see Eq.(2)). While the noise model used in calculating the sampling distribution of the test statistic assumes Gaussianity (a common and analytically tractable choice), the test statistic itself is not specialized for Gaussian data. Moreover, contrary to the reviewer's comment, our test statistic is in fact data-driven. Specifically, $T(\\boldsymbol{Y}) = \\\\|\\boldsymbol{\\eta}\\_{\\boldsymbol{y}}^t \\boldsymbol{Y}\\\\|_2^2$ is constructed using the vector $\\boldsymbol{\\eta}\\_{\\boldsymbol{y}}$, which depends on the deep k-NN structure conditioned on the observed data point $\\boldsymbol{y}$. This data-dependent construction is central to our methodological contribution, which enables selective inference for deep data-driven hypotheses. Within the selective inference framework, we condition on the selection event associated with this $\\boldsymbol{\\eta}\\_{\\boldsymbol{y}}$, making the sampling distribution of the data-dependent statistic tractable."}, {"review_id": "Wl7Buei2Ub", "reviewer": "Reviewer_VcWS", "summary": "This paper proposes a framework to quantify the statistical significant of anomalies detected through deep kNN based anomaly detection in the form of p-values for hypothesis testing via Selective Inference (SI)", "questions": "1. Is this the first paper to consider SI for AD? Or only for deep kNN-based AD? If so, how does this methodology differ from others proposed in other settings?\n\n2. How does this result translate to other relevant tasks such as OOD detection?", "limitations": "No discussion of negative societal impact, though model limitations are discussed.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is thoughtfully written and easy to understand. \n\nExperiments are comprehensive, covering a variety of data types (tabular, image) and a range of datasets.\n\nIt is difficult to find fault with this paper, though I would say it would be beneficial to make more effort explaining the details of other SI methods to provide clearer grounding of this work within the context of other literature. Also, Figure 4 is not particularly easy to comprehend. The authors may consider alternative ways to plot their results.", "was_revised": true, "final_justification": "Technically strong and novel contribution and the authors responses are convincing.", "rebuttal": "We thank the reviewer for your feedback.\n\n>(Weakness-1-1) I would say it would be beneficial to make more effort explaining the details of other SI methods to provide clearer grounding of this work within the context of other literature. \n\nIn the camera-ready version, we plan to expand the related works section to include more detailed discussion of other SI methods.  \n\n> (Weakness-1-2) Also, Figure 4 is not particularly easy to comprehend. The authors may consider alternative ways to plot their results.\n\nAdditionally, we will revise the visualization and caption of Fig. 4 to make the differences between our method and the baseline methods more easily understandable.\n\n>(Question-1). Is this the first paper to consider SI for AD? Or only for deep kNN-based AD? If so, how does this methodology differ from others proposed in other settings?\n\nTo the best of our knowledge, this is the first study to apply SI to anomaly detection. While SI has been extensively studied in the statistical literature, its application to machine learning methods remains very limited. One related work is the application of SI to change-point detection in time series data [1]. However, that line of research focuses on testing the significance of global changes in the entire sequence, which differs from our setting where the goal is to assess the abnormality of individual data points. Another related work is the application of SI to robust regression [2], where inference is performed on regression coefficients obtained after excluding outliers. However, the statistical significance of the detected outliers themselves is not addressed. Moreover, to the best of our knowledge, our work is also the first to apply SI in the context of data analysis performed in a latent space induced by deep learning models. We plan to expand the discussion of these points in the related works section of the camera-ready version.  \n\n[1] https://arxiv.org/pdf/1606.03552  \n[2] https://arxiv.org/abs/1711.10635\n\n>(Question-2). How does this result translate to other relevant tasks such as OOD detection?\n\nWe appreciate the suggestion highlighting this interesting research direction. While it would be necessary to formulate the corresponding selection event (e.g., $\\varepsilon_{\\boldsymbol{Y}} = \\varepsilon_{\\boldsymbol{y}}$ as in Eq (6)), we believe that our framework is in principle applicable to other tasks such as OOD. For example, OOD detection based on Mahalanobis distance in the latent space [3] shares structural similarities with our setting, and thus may allow for a natural extension of our method.  \n\n[3] https://arxiv.org/pdf/2505.18032"}]

Author rebuttal:

Meta-review: The paper proposes a selective inference framework for quantifying statistical significance in deep k-NN anomaly detection. The contribution is well-motivated, addressing the lack of uncertainty quantification in distance-based anomaly detectors by providing p-values with finite-sample error guarantees. The work is technically sound, and the committee generally appreciated the articulation of the problem, the theoretical analysis, and the empirical demonstrations on synthetic, tabular, and image datasets. 

At the same time, several concerns were raised, including the relatively narrow scope of the method (restricted to deep k-NN), reliance on Gaussian noise assumptions, limited theoretical novelty beyond adapting existing selective inference tools, and a somewhat modest empirical evaluation. The rebuttal was effective in clarifying some technical details in question (e.g., proof corrections, robustness under noise misspecification, etc.). 

Overall, I share the view that this is a timely and valuable contribution to the anomaly detection literature. The authors are encouraged, when preparing the next revision, to better highlight the scope and unique contributions of this work in the broader landscape of selective inference for anomaly detection, and to further clarify limitations and generalizability. This will help position the paper more strongly and mitigate potential perceptions of overlap with related concurrent efforts.
Decision: Accept (poster)