An Upper Bound for the Distribution Overlap Index and Its Applications
Authors: Hao Fu
Prashanth Krishnamurthy
Siddharth Garg
Farshad Khorrami
Venue: Submitted to ICLR 2023 (2023)
Keywords: Distribution Overlap
Finite-Sample Approximation
One-Class Classification
Domain Shift Analysis

Abstract: The overlap index between two probability distributions has various applications in statistics, machine learning, and other scientific research. However, approximating the overlap index is challenging when the probability distributions are unknown (i.e., distribution-free settings). This paper proposes an easy-to-compute upper bound for the overlap index without requiring any knowledge of the distribution models. We first utilize the bound to find the upper limit for the accuracy of a trained machine learning model when a domain shift exists.  We additionally employ this bound to study the distribution membership classification of given samples. Specifically, we build a novel, distribution-free, computation-efficient, memory-efficient one-class classifier by converting the bound into a confidence score function. The proposed classifier does not need to train any parameters and requires only a small number of in-class samples to be accurate. The classifier shows its efficacy on various datasets and outperforms many state-of-the-art methods in different one-class classification scenarios, including novelty detection, out-of-distribution detection, and backdoor detection.  The obtained results show significant promise toward broadening the applications of overlap-based metrics. 

Under review as a conference paper at ICLR 2023
AN UPPER BOUND FOR THE DISTRIBUTION OVERLAP INDEX AND ITS APPLICATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
The overlap index between two probability distributions has various applications in statisties, machine learning, and other scientific research. However, approxi- mating the overlap index is challenging when the probability distributions are un- known (i.e., distribution-fres settings). This paper proposes an easy-to-compute upper bound for the overlap index without requiring any knowledge of the distri- bution models. We first utilize the bound to find the upper limit for the accuracy of trained machine learning model when domain shift exists. We additionally employ this bound to study the distribution membership classification of given samples. Specifically, build novel, distribution-free free, cimputation-efficienn memory-efficient one-class classifier by converting the bound into a confidence score function. The proposed elassifier does not need to train any parameters and is empirically accurate with only a small number of in-class samples. The classi- fier shows its efficacy and outperforms many state-of-the-a methods on various datasets in different one-class classification scenarios, including novelty detection, out-of-distribution detection, and backdoor detection. The obtained results show significant promise toward broadening the applications of overlap-based metrics.
INTRODUCTION
The distribution overlap index refers to the area intersected by two probability density functions (i.e.. Fig. [[] and measures the similarity between the two distributions. high overlap index value implies high similarity. Although the overlap index has various applications in many areas, such as biology (Langøy et af. 2012 Utne et al. 2012), economies (Milanovic Yitzhaki, 2002). and statisties (Inman & Bradley Jr 1989). the literature on approximating it under distribution- free settings thin. This work proposes an upper bound for the overlap index with distribution- free settings to broaden the potential applications of overlap-based metrics. The bound is easy to compute and contains three terms: a constant number, the norm of the difference between the two distributions' means, and variation distance between the two distributions over subset. Even though finding such an upper bound for the distribution overlap index is already valuable, we further explore additional applications of our bound discussed below.
One application of our bound is for domain shift analysis. Specifically, a domain shift is a change in the dataset distribution between model's training dataset and the testing dataset encountered during implementation (i.e., the overlap index value between the distributions of the two datasets is less than 1). We calculated the model's testing accuracy terms of the overlap index between the distributions of the training and testing datasets and further found the upper limit of the accuracy using our bound for the overlap index. Knowing the upper bound for model's testing accuracy helps measure the model's potential and compare it with other models. We validated the calculated upper limit accuracy with experiments in backdoor attacks.
Another application of our bound is for one-class classification. Specifically, one-class classifica- tion refers to a model that outputs positive for in-class samples and negative for out-class samples that are absent, poorly sampled, or not well defined (i.e., Fig. [b)) We propose novel one- class classifier by converting our bound into a confidence score function to evaluate if sample is in-class out-class. The proposed classifier has many advantages. For example, implementing deep neural network-basee classifiers requires training thousands of parameters and large memory, whereas implementing our classifier does not. only needs sample norms to calculate the confi-

---

Under review as a conference paper at ICLR 2023
Stop
In-Class Samples
Right Turn
Overlap
STOP
Clean Sample
Polsoned Sample
Trigger
(a)
(b)
(c)
Figure 1: (a): Overlap of two distributions. (b): One-class classification. (c): Backdoor attack.
dence score. Besides, deep neural network-based classifiers need relatively large amounts of data to avoid under-fitting or over-fitting, whereas our method empirically accurate with only a small number of in-class samples. Therefore, classifier is computation--effcient, memory-efficiem and data-efficient. Additionally, compared with other traditional one-class classifiers, such as Gaussian distribution-bassed classifier, Mahalanobis distance-based classifier (Lee al. 2018)) and one-class support vector machine (Schölkopf et al. 2001), our classifier is distribution- ffree, explainable, and easy to understand.
Overall, the contributions of this paper include:
Finding distribution- upper bound for the overlap index.
Applying this bound the domain shift analysis problem with experiments.
Proposing novel one-class classifier with the bound being the confidence score function.
Evaluating the proposed one-class classifier through comparison with various state-of- the-art methods on several datasets, including UCI datasets, CIFAR-100, sub-ImageNet, etc., and different one-class classification scenarios, such as novelty detection, out-of- distribution detection, and neural network backdoor detection.
1.1 BACKGROUND AND RELATED WORKS
Measuring the similarity between distributions: Gini & Livada 1943) and Weitzman (970))
introduced the concept of the distribution overlap index. Other measurements for the similarity between distributions include the total variation distance, Kullback-Leebler divergence (Kullback &
Leibler 1951). Bhattacharyya's distance (Bhattacharyya 1943), and Hellinger distance (Hellinger) 1909). In psychology, some effect size measures' definitions involve the concept of the distribution overlap index, such as Cohen's 's U index (Cohen 2013), McGraw and Wong's 's CL measure (McGraw & Wong 1992). and Huberty's 's degree of non-overlap index (Huberty & Lowman, 20000. However, they all have strong distribution assumptions (e.g., symmetry or unimodality) regarding the overlap index. Pastoree & Calcagni (2099) approximates the overlap index via kernel density estimators.
One-class classification: Moya & Hush 1996) coined the term one-class classification. One-class classification intersects with novelty detection, anomaly detection, iit-r--distribution detection, and outlier detection. Yang et al. (202I) explains the differences among these detection areas. Khan & Madden (014) discusses many traditional non neural network-based one-class classifiers, such as one-class support vector machine (Schölkopf et al., 2001), decision-tree (Comité et al. 1999). and one-class nearest neighbor (Tax) 2002). Two neural network based one-class classifiers are (Ruff) et al. 2018) and OCGAN (Perera et al., 2019). Morteza & Li (202) introduces Gaussian mixture- based energy measurement and compares with several other score functions, including maximum softmax score (Hendrycks & Gimpel, 2017). maximum Mahalanobis distance (Lee et al. 2018). and energy score (Liu et al. (220a)) for one-class classification.
Neural network backdoor attack and detection: Gu et al. (2019) and Liu et al. 20188) mentioned the concept of the neural network backdoor attack. The attack contains two steps: during training, the attacker injects triggers into the training dataset; during testing, the attacker leads the network to
misclassify by presenting the triggers (i.e., Fig. [c)). The data poisoning attack (Biggio et al. 2012)
and adversarial attack (Goodfellow et al. 2014) overlap with the backdoor attack. Some proposed
2

---

Under review as a conference paper at ICLR 2023
trigger types Wanet (guyyen & Tran 2021). invisible sample-specite (Li et al. 2021 smooth Zeng et al. 2021), and reflection (Liu et al., 2020b). Some methods protecting neural networks from backdoor attacks include neural cleanse Waag et al. 2019), fin-pruniing (Liu et al., 201 8a), and STRIP (Gao et al., 2019). NNoculation (Veldanda et al., 2021) and RAID (Fu et al. 2022) utilize online samples intersects with one-class classification. Therefore, some one-class classifiers can detect poisoned samples against the neural network backdoor attack.
improve their detection methods. The backdoor detection problem also
Organization of the Paper: We first provide preliminaries and derive the proposed upper bound for the overlap index in Sec. 2 We next apply our bound to domain shift analysis in See. We then propose. analyze, and evaluate our novel one-class classifier in Sec. 4 We finally conclude the
paper in Section 5
AN UPPER BOUND FOR THE OVERLAP INDEX
2.1 PRELIMINARIES
For simplicity, we consider the I= space and continuous random variables. We also define P and Q as two probability distributions R" with fP and fo being their probability density functions. the two distributions is defined:
Definition (Overlap Index). The overlap 17 R x R - [0. 1] the
n(P.Q) minlfp(=))
JRn
Definition 2 (Total Variation Distance). The total variation distance R R - |0, 1] of the two distributions is defined as
1 [fp(z) fQ(x))
8(P - Rm
(2)
Definition (Variation Distance on Subsets). Given subset A R", we define 8 XR" [0, 1] to the variation distance of the two distributions on A, which is
1 (P Q) = 2 J [fr(x) - fo(=)ldr
(3)
Remark 1. One can prove that 1 and 5 satisfy the following equation:
(P,, Q) = 100----PPP000 Q) = - SA Q) A
(4)
The quantity A defined in will play an important role in deriving our upper bound for v).
2.2 THE UPPER BOUND FOR THE OVERLAP INDEX
We now proceed with deriving our proposed upper bound.
Theorem 1. Without loss of generality, assume D and D are two probability distributions on bounded domain B C Ih with defined norm || 1I| (i.e., supzen || < +oo), then for any subset
A C with complementary set A B A, we have
1 n(D BO+,D Ar
T TA 8
||ND+ PD-1l
(5)
A
where T = SuPzEA || and r = supzEA= ||]|| jDD+ and are the means of D+ and D- and A is the variation distance on set A defined in Definition then we have
Moreover, let rB sup.e ||||
nD+ 51
1
ppp+ MD-ll
TB - TA 8..
(6)
2rB
TB
Since 6 holds for any A, tighter bound can be written as
n(D D < 2rB
|
max
TB - TA 5
(7)
rB
In this paper, use the Ta norm. However, the choice of the norm is not unique and the analysis can be carried out using other norms well.
3

---

Under review as a conference paper at ICLR 2023
Proof. Let fn+ and fn- be the probability density functions for D and D From (4 we have
nD* ) =
5a(D*, D-) 5Λ- (D= D.).
(8)
Using 8 triangular inequality, and boundedness, we obtain
||D+ PDD J, (JD+(x) - SD- - (x)) dxyl
|<x(S++ (x) - SD- (x)
(9)
/ |||||| 1fn+ (x) - (z)-()) + ar a + Ar ~rr + 2r 8^ - 11D .
||||| [|]]|-Ifn++00 (2) fD- (z)
(10)
A"
(11)
which implies (5. Replacing TA= with rB in 11 implies
(6)
Remark 2. The only ussumption in this theorem is that the probability distribution domain is
bounded. However, almost all real-world applications satisfy the boundedness assumption since the data is bounded. Therefore, тв can always be found (or at least a reasonable approximation can be found). Additionally, Although the proof of this theorem involves probability density functions, the computation does not require knowing the probability density functions but only finite samples because we can use the law of large numbers to estimate p
can constrain A be bounded ball that r is also known.
MDD-1 || and 8 which will be shown next.
2.3 APPROXIMATING THE BOUND WITH FINITE SAMPLES
Let 9: B (0, 1} be condition functior and define A for [)())= = 1, BJ. According to the definition of d and triangular inequality, we have
5A(D+, ( 2 f/
1
|SD -JD-(x) 2 2 / (D-(x)-SD- -D- (DD((r)
(12)
1
/
fDI (x)g(x)) - /.51 -J, fo (z)g(x)dr
LE [gl - ED [g]
(13)
2
Calculating EDi [g] and IED [g] is easy: one just needs to draw samples from D and D", and then average their g values. Applying (1) into Th gives the following corollary:
Corollary 1. Given D+, D B. and ||- used in Theorem ! let A(g) { g(x) = 1, e B}
with any condition function B > 10, Then, an upper bound for nn n(D* D-) that can be
obtained by our approximation
FB r(())
(D+ S1-
2r Il "D max
PD [4] ED 1All-
(14)
2rB
Given several condition functions {9) 1/ and finite sample sets (i.., {a, Hat D+ and {2, mm D-), Alg. shows how to compute the RHS of (4p.
Algorithm ComputeBound (- ) ta, Mm (
B -
22 .... Im and rn + max ||]|||
...
A -
1==1 x7
n
m
for
1 k
do
A = (x (z)
1, E B) and P t- max |
8j +-
Ia
1 E (xt) 1 -m 15
end for Return:
2r
1
A
max, 8
Remark 3. The choice of condition functions is not unique. In this work, we use the indicator function g(x) [[]x=]] < rr. which outputs if ||=||| < and 0 otherwise. By setting different values forr, we generate a family of condition functions. fundttionnnnctinsss The motivation for choosing such indicator function form is that it is the most simple way to separate space nonlinearly and it saves compu- tations by directly applying r into Corollary I However, other indicator functions, such as RBF kernel-bast indicator functions, are worth exploring and will be considered in our future works.
Thhe condition function is indicator function 1 (comdition) ) that outputs I when the input satisfies the given condition and 0 otherwise.
4

---

Under review as a conference paper at ICLR 2023
3 APPLICATION OF OUR BOUND TO DOMAIN SHIFT ANALYSIS
We now apply our bound to domain shift analysis.
Theorem 2. Assume that D and D' are two different data distributions (i.e., T( D*) < 1). If model is trained on D with p accuracy on D and accuracy on D D, then the overall accuracy of the model on D is pn(D, + ((,))) which is upper bounded because n,D, D*) is
upper bounded by (14).
Remark 4. To prove the theorem, let fD and fo- be their probability density functions, then
Accuracy= = -.ND (
minn(JD()) SD- (x)]
+ 9 +1 (1(11 min(fo(x) SD (m)) )) [ (x)dx (15)
fD (x) = =pn(D, q( (D,1)(D, = D,, D²) +
fD-
(16)
Without loss of generality, assume p > 9. Then (16) shows that a large domain shift (i.e., a small (DD,, )) leads to low overall accuracy of the model the testing data distribution D". If
and q = 0, then the overall accuracy is equal to ny (D, D*).
Theorem 2 in Backdoor Attack Scenarios: A backdoor attack scenario (Fig. [bb) considers that the model has a zero accuracy on poisoned data distribution as the attack success rate is almost 100%. Define the clean data distribution as D, poisoned data distribution distribution D' composed by D and D", i.e., D = o + (1-a) oP, where a E [o, 1] is the purity ratio (i.e., the ratio of clean samples the entire testing samples). With all the settings above, we know that q = 0 on DP and (66) becomes
DP, and a testing data
rA(s)
Accuracy pn(D, D*) S p(
IAD LDD
|| max
Ep[g
EE [gll)
(17)
2rB
9
2rB
= p(1 - 1
TB
-A(a)) Eplgl - EE [g]])
PDD - PD=ll (1 σ)
max
(18)
2r
2r
17 shows that the actual model accuracy on the contaminated testing data should be bounded by the multipli multiplication of its accuracy p on clean data and our upper bound for rD, D*) with samples. (18)
shows that the upper limit of the model accuracy the contaminating data should linearly increase with the purity ratio a (i.e., the percentage of clean samples over the entire testing samples).
Validating Theorem 2 in Backdoor Attack Scenarios: The considered datasets are MNIST (Le- Cun et al. 2010), GTSRB (Stallkamp et al. 2011), YouTuue Face (Wolf et al. 2011). and sub- IeageNet (Deng et al.) 2009). We composed the testing datasets D with O, 0.1. ..... 0.9, 1 and calculated the upper bound for 71D, DD) using L1, L2, and Lis norms in the raw data space, model output space, and intermediate layer space. The actual model accuracy and corresponding upper bounds are plotted in Fig. 2 The actual model accuracy is below all the calculated upper limits, validating Theorem [2] Additionally, the upper limits grow linearly with a, supporting (18). In other scenarios except for backdoor attacks, the model accuracy, p, and may be known, then the lower bound for 1) can be estimated by Theorem 2 Theorem can also help in finding g by knowing p. 7), and model accuracy. Therefore, Theorem has practical relevance and usefulness.
4 APPLICATION OF OUR BOUND TO ONE-CLASS CLASSIFICATION
4.1 PROBLEM FORMULATION FOR ONE- CLASS CLASSIFICATION
Given Rd space and n samples { that lie in unknown probability distribution, would like build a test V Rdd - (+1} that for any new input x., Vxx)) outputs 1 when æ is from the same unknown probability distribution, and outputs -1, otherwise. Some applications of v are novelty detection, out-of-distribution detection, and backdoor detection (e.g., Fig. (b. c)).
4.2 A NOVEL CONFIDENCE SCORE FUNCTION
Given some in-class samples { one can pick several condition functions {0, 10 where (x) = 1 1[[[=]] < ry} } for different r,, that f(z) ComputeeBoundd (z), {21)" {g 1/=)) defined in Alg. is a score function that measures the likelihood of any input, x, being an in-class sample. Alg. shows the overall one-class classification algorithm. a = 10 in our experiments.
5

---

Under review as a conference paper at ICLR 2023
MNIST
GTSRB
1.0
1.0
Accuracy of Model
0.8
Accuracy of Model
0.8
0.6
0.6
Actual
Out
Actual
Out
0.4
Li
0.4
In.
La
4
4
0.2
L. In
Lg Hid.
L. in
La Hid.
4 Out
Hid
4 Out
Hid
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
YouTube Face
ImageNet
1.0
0.8
0.7
Accuracy of Model
0.8
Accuracy Model
0.6
0.6
0.5
0.4
Actual
Out
Actual
Out
0.4
Li
Out.
0.3
In.
La
41
0.2
4 Hid
0.2
L. In
Lg Hid.
0.1
L. in.
4 Hid.
4 Out
Hid.
4g Out
Hid.
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 2: The actual model accuracy (dot) vs. (66) (solid) calculated with L1, L2, and L norms in input, output, and hidden spaces. x: the ratio of clean samples to the entire testing samples.
Algorithm The Novel One-Class Classifier for the Input æ
Given in-class samples {= select several condition functions {g, - set a threshold To if Boundlt {an h (0, )/-1) 2 To then
is an in-class sample else
is an out-class sample end if
Remark 5. The score function f measures the maximum similarity between the new input and the available in-class samples { Different To lead to different detection accuracy. However, we will show that the proposed one-class classifier has an overall high accuracy under different To-
4.3 COMPUTATION AND SPACE COMPLEXITIES
Our algorithm can pre-compute and store 1 E ii x4 and 1 Σm (I())) with j 2,,,... k. There- fore, the total space complexity O(k + 1). Assume that the total number of new online inputs I; then, for every new input z, our algorithm needs to calculate ||æ||| once and s for & times. Therefore, the total computation complexity is ( + 1)). Empirically, we restricted
be small number (e.g., 10) so that even devices without strong computation power can run our algorithm efficiently. Therefore, our classifier is computation-efficient and memory-efficient.
4.4 EVALUATION
Overall Setup: All the baseline algorithms with optimal hyperparameters, related datasets, and models were acquired from the corresponding authors' websites. The only exception is backdoor detection, in which created own models. However, we have carefully fine-tuned the baseline

---

Under review as a conference paper at ICLR 2023
Mean AUROC(%) + Std.
100
90
80
70
60
40
Our
LB
KC
Pa
Ga
KM
IN
KN
AE
LP
PC
LR
NN
ST
MC
50
SVM
Min.
Mix.
LOF
Nai.
LCI
Figure 3: The method is listed in the same order in Table
methods' hyperparameters ensure their best performance over other hyperparameter choices. Our approach used ten indicator functions for all the experiments.
4.4.1
CLASSIFICATION FOR NOVELTY DETECTION
We evaluated our classifier on 100 small UCI datasets (UCI, Dua & Graff 2017) and recorded the area under the receiver operating characteristic curve (AUROC). Fig. shows the mean and standard deviation of AUROC for ours and other classifiers. Detailed numerical results can be found Table in Appendix A The implementation code provided in the supplementary material.
Our classifier is the most consistent with the smallest standard deviation. Except for Gaussian, K-Nearest Neighbor, and Minimum Covariance Determinant, our classifier outperforms the other methods by showing the highest AUROC mean and the lowest AUROC standard deviation. Among Gaussian, K-Nearest Neighbor, and Minimum Covariance Determinant, Gaussian the best clas- sifier by having the highest mean and lowest standard deviation of AUROC. However, our method comparable to Gaussian by showing a close mean and a smaller standard deviation.
Besides the results, our classifier is distribution-free compuration-efficient, and memory-efficient whereas some other classifiers do not. Additionally, our method is also easy to explain and under- stand: the score measures the maximum similarity between the new input and the available in-class samples. Therefore, we conclude that our classifier is valid for novelty detection.
4.4.2 ONE-CLASS CLASSIFICATION FOR OUT-OF-DISTRIBUTION DETECTION
We used CIFAR-I0 and CIFAR- testing datasets (Krihheskyy al., 2009 as the in-distribution datasets. The compared methods contain MSP (Hendrycks & Gimpel (017), Mahalanobis (Lee) al. 2018), Energy score (Liu et al. 2020a), and GEM (Morteza & 2022). We used WideResNet Zagoruyko & Komodakis, 2016)
extract features from the raw data. The WideResNet models (well-trained on CIFAR- 10 and CIFAR- training datasets) and corresponding feature extractors acquired from Morteza & Li 20022 All the methods were evaluated in the same feature spaces with their optimal hyperparameters for fair comparisons. To fit the score function's 's parameters for all the methods, we formed small dataset by randomly selecting 10 samples from each class, The out-of-distributinn datasets include Textures (Cimpoi et al.) 2014), SVHN (Netzer et al. 2011), LSUN-Crop (Yu et al. 2015), LSUN-Resize YYu al. 2015), and ISUN (Xu al. 2015). We used three metrics: the detection accuracy for distrr-iistttbttion samples when the detection accuracy for in-distribution samples 95% (TPR95), AUROC, and area under precision and recall (AUPR).
Table shows the average results for CIFAR- 10 and CIFAR- 100. The details for each individual out- of-distribution dataset can be found in Table and Table G Appendix B Our method outperforms the other methods by using the least memory and showing the highest AUROC on average. Our approach is also one of the fastest methods: the execution time of our approach for each sample is less than one millisecond (ms). For the CIFAR-10 case, our method also shows the highest average TPR95, and the average AUPR our method is over 92%. For the CIFAR- 100 case, the average TPR95 of method is 0.3% close to the highest average TPR95, and the average AUPR of our method is over 85%. For each individual -f--ofribitbbtiionn dataset, our method always outperforms no less than half of the methods in TPR95 and AUROC, and the total average AUPR of our method
7

---

Under review as a conference paper at ICLR 2023
Table 1: Average performance on various out-of-distribution datasets. Our method can be further improved with an iterative approach as shown in Table 2
In-Distributions Method
TPR95
AUROC
AUPR
Time/Sample
Memory
Ours
72.33%
92.76% %
92.32%
0.60ms
1048.22MiB
MSP
50.63%
91.46%
98.07% %
0.02ms
1825.21MiB
CIFAR-10
Mahala.
46.83%
90.46%
97.92%
30.61ms
1983.17MiB
Energy
68.31%
92.32%
97.96%
0.22ms
1830.01
GEM
50.81%
90.45%
97.91%
25.62ms
1983.51
Ours
48.08%
87.63 %
85.72%
0.60ms
1134.32MiB
MSP
19.87%
75.97%
94.09%
0.02ms
1825.98MM
CIFAR-100
Mahala. 48.35%
84.90%
96.37%
56.24ms
108B.
Energy
27.91%
80.44%
95.15%
0.2
1838.23MiB
GEM
48.36%
84.94%
96.38%
56.27ms
1984.77MiB
TPR95
AUROC
AUPR
90
85
80
75
93
Yoture
Voture
70
92
65
01
60
90
75 100
175
50
100
50 75 125
indicator functions
indicator functions
indicator functions
Figure 4: Improvements with more indicator functions with CIFAR- 10 being the in-distribution data.
over all cases is 89.02%. Our method can be further improved with an iterative approach as shown in Table details discussed at the end. We noticed that the out-of-distribution datasets are much smaller in size than the in-distribution datasets. Therefore, although the current AUPR is sufficient to ensure that our approach is valid, we see a potential improvement in our method to increase the AUPR for heavily imbalanced problems in our prospective work. On balanced datasets, our approach shows higher AUPR than the baseline methods as shown in Table 3ith details in Table 7 in Appendix C Therefore, our approach performs better on balanced datasets than on unbalanced datasets. We also empirically observed that the compared baseline methods reported errors when data dimensions are dependent because the compared baseline methods need to calculate the inverse of the estimated covariance matrices that will not be full rank if data dimensions are dependent. We have reported this observation in Table 7 in the appendix. In contrast, our approach works since does not require finding the inverse of any matrices. Further, Table and Table B together show that the baseline methods perform well only for out-of-distribution detection, whereas our approach performs well for both out-of-distributtin detection and backdoor detection (details are explained in next subsection). In summary, our classifier is a valid out-of ddistribbtiion detector.
Our approach used ten indicator functions in all experiments. However, we have evaluated our approach by using more indicator functions (i.e., more r;)) and plotted the results in Fig. From the figure, the performance of our approach increases with more indicator functions being used and eventually converges to limit. This limit is determined by the out-of distrribbutii dataset, the tightness of our bound Corollary [ and the form of utilized indicator functions (i.e., g(x) = [||=r|| < r) in this work). To increase this limit on given out- of-distribution dataset with the current form of our upper bound, a more advanced type of indicator
Improvement with more indicator functions:
function is required, which will be our future work as mentioned in Remark
Improvement with an iterative approach: Except for using more indicator functions, our approach can also be improved by interative approach with only the original ten indicator functions being used. Assume that the confidence score of an input x is s(z), then its iterative confidence score, s' (z), can be calculated by Alg. [ with the condition function g(z) (1s(z) < T.}. where T, represents different thresholds. Table 2 shows the performance of our approach by applying the
iterative score s' (xx to Alg. The results show considerable improvement compared to Table for
8

---

Under review as a conference paper at ICLR 2023
Table 2: Performance of approach with the iterative approach.
In n--Distributions
Metrics (%)
Texture
SVHN
LSUN-C
LSUN-R
ISUN
Ave.
TPR95
64.20
94.10
83.63
85.41
79.62
81.39
CIFAR-10
AUROC
92.00
96.88
95.29
95.48
94.46
94.82
AUPR
92.02
98.78
95.45
95.65
94.65
95.31
TPR95
42.5
93.75
57.76
88.49
82.15
72.93
CIFAR-100
AUROC
85.21
96.61
89.23
96.09
94.72
92.37
AUPR
85.22
98.69
89.70
96.19
94.99
92.95
Table 3: Average performance for backdoor detection over various backdoor triggers and datasets.
Metrics (%)
Ours
STRIP
Mahalanobis
GEM
MSP
TPR95
89.40
39.60
56.97
91.57
39.24
AUROC
96.68
70.30
75.94
58.08
54.92
AUPR
95.42
68.76
76.37
75.88
60.52
the average and to Table 5 and Table 6 for each individual out-of-distrriution dataset. Exploring the potential improvement of our approach with more rounds iterations will also be our future work.
4.4.3 ONE-CLASS CLASSIFICATION FOR BACKDOOR DETECTION
The utilized datasets are MNIST (LeCun et al. 2010). CIFAR-10 (Krizhessky et al. 2009), GTSRB
(Stallkamp et al. 2011), YouTuue Face Wolf et al.
2011), and sub-ImageNNet (Deng et al. 2009).
The adopted backdoor attacks include naive triggers, all-label attacks (Gu ct al. 2019). moving trig- gers (Fu et al. 2020), Wanet (Nguyen & Tran 2021). combination attacks, large- triggers, filter triggers, and invisible sample-specific triggers (Li et al. 2021). as listed in Fig. in Appendix C The neural network architecture includes Network in Network (Lin et al. 2014), Resnet (He et al. 2016
and other networks from (Wang et al.] 2019 Gu et al.) 2019). For each backdoor attack, we assume that small clean validation dataset available (i.e., 10 samples from each class) at the beginning. Therefore, the poisoned samples (i.e., samples attached with triggers) can be considered out-class samples, whereas the clean samples can be considered in-class samples. We used the backdoored network to extract data features. Then, we evaluated our one-class classifier and compared it with the previous baseline methods and STRIP (Gao et al., 2019) in the feature space, The metrics used are the same: TPR95 (i.e., the detection accuracy for poisoned samples when the detection accuracy for clean samples is 95%). AUROC, and AUPR. Table shows the average performance. Details on each individual trigger can be found in Table 7 in Appendix C
From the table, our classifier outperforms other baseline methods on average by showing higher AUROC, and AUPR. As for TPR95, our approach is very close to GEM. Compared with STRIP on the overall average performance, our classifier is 49.8% higher in TPR95, 26.38% higher in AUROC, and 26.66% higher in AUPR. For each individual trigger, the TPR95 of our method is over 96% for most cases, the AUROC of our method is over 97% for most cases, and the AUPR of our method over 95% for most cases. It is also that classifier is robust against the latest advanced backdoor attacks, such as Wanet, invisible trigger, all label attack, and filter attack, whereas the baseline methods show low performance on those attacks. Therefore, conclude that our classifier is valid for backdoor detection.
5 CONCLUSION
This paper proposes an easy-to-compen distribution-free upper bound for the distribution overlap index. Two applications of the bound are explored. The first application for domain shift analy- sis with a proposed theorem and discussion. The second application is for one-class classification. Specifically, this paper introduces novel distribution-fee one-class classifier with the bound being its confidence score function. The classifier is sample-efficient, computation-efficiee and memory- efficient. The proposed classifier is evaluated on novelty detection, outlof-distribution detection, and backdoor detection various datasets and compared with many state-of-the-art methods. The ob- tained results show significant promise toward broadening the application of overlap- based metrics.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
One-class classifier results.
html
Accessed: 2022-08-30.
Anil Bhattacharyya. On a measure of divergence between two statistical populations defined by their probability distributions. Bulletin of the Calcutta Mathematical Society, 35:99-109, 1943.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector ma- chines. In Proceedings of the International Conference on International Conference on Machine Learning. pp. 1467-1474, Edinburgh, Scotland, June 2012.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613 3613, Columbus, OH, June 2014.
Jacob Cohen. Statistical power analysis for the behavioral sciences. Routledge, 2013.
Francesco De Comité, François Denis, Rémi Gilleron, and Fabien Letouzey. Positive and unlabeled examples help learning. Proceedings of International Conference on Algorithmic Learning Theory, pp. 219-2122222 1900,00,,1100 Tokyo, Japan, Dec. 1999.
J. Deng, W. Dong. R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchi- cal image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. Miami, FL, 2009.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http: ics. uci. sici.edu/m/.
Hao Fu, Akshaj Kumar Veldanda, Prashanth Krishnamurthy, Siddharth Garg, and Farshad Khor- rami. Detecting backdoors neural networks using novel feature- based anomaly detection. arXiv preprint ar arXiv:201 / 1.02526, 2020.
Hao Fu, Akshaj Kumar Veldanda, Prashanth Krishnamurthy, Siddharth Garg, and Farshad Khorrami. by iterative demarcation. IEEE
A reature-based on-line detector to remove Access, 10:5545 5558, 2022.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks deep neural networks. In Proceedings of the Anmual Computer Security Applications Conference, pp. 113-125, Austin, TX, Dec. 2019.
Gini and G Livada. Nuovi contributi alla teoria della transvariazione. Roma: Atti della VI Riunione della Società Italiana di Statistica, 1943.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of Advances in Neural Information Processing Systems, volume 27, Montreal, Canada, Dec. 2014.
Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-47244, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, Miami, FL, June 2016.
Ernst Hellinger. Neue begründung der theorie quadratischer formen von unendlichvielen veränderlichen. Journal für die reine und angewandte Mathematik, 1909(136):11-271,, 1909.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and duitt--dis examples in neural networks. In Proceedings of International Conference Learning Represen- tations, Toulon, France, Apr. 2017.
Carl Huberty and Laureen Lowman. Group overlap as a basis for effect size. Educational and Psychological Measurement, 60(4)5543-533, 2000.
10

---

Under review as a conference paper at ICLR 2023
Henry F Inman and Edwin L Bradley Jr. The overlapping coefficient measure of agreement between probability distributions and point estimation of the overlap two normal densities.
Communicattoons in Statistics-theory and Methods, 18(10):3851-3874 1989.
Shehroz S Khan and Michael G Madden. One-class classification: laxonomy of study and review of techniques. The Knowledge Engineering Review, 29(3):345-374 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Solomon Kullback and Richard Leibler. On information and sufficieney. The Annals of Mathe- matical Statistics, 22(1):79-86, 1951.
Herdis Langey, Leif Nøttestad, Georg Skaret, Cecilie Broms, and Anders Fernö. Overlap in dis- tribution and diets of atlantic mackerel (scomber scombrus), norwegian spring-spawning herring (clupea harengus) and blue whiting (micromesistius poutassou) in the norwegian during late summer. Marine Biology Research, 8(5-6))442-44 460, 2012.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.leee lewun. http://www...u..o////d////i wwwpexdb///i/stt..omommmmmm///.. 2010.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detect- ing out-of-distribution samples and adversarial attacks. In Proceedings of Advances in Neural Information Processing Systems, volume 31, Montreal, Canada, Dec. 2018.
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16463-16472, Virtual, June 2021.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In Proceedings of the International Conference on Learning Representations, pp. 1-10, Banff, Canada, Apr. 2014.
Kang Liu, Brendan Davint and Siddharth Garg. Fine-pruning: Defending against backdoor- ing attacks on deep neural networks. In Proceedings of International Symposium on Research in Attacks, Intrusions, and Defenses, pp. 273-294, Crete, Greece, Sep. 2018a
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. anergy-based out-of-distrbbttio de- tection. In Proceedings of Advances in Neural Information Processing Systems, volume 33, pp. 21475-21475, Virtual, Dec. 2020a.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In Proceedings of the Network and Distributed Systems Security Symposium, San Diego, CA, Feb. 2018b.
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. Proceedings of the European Conference on Computer Vision, pp. 182-199, Virtual, Aug. 2020b.
Kenneth O McGraw and Seok P Wong. A common language effect size statistic. Psychological Bulletin, 111(():361, 1992.
Branko Milanovic and Shlomo Yitzhaki. Decomposing world income distribution: Does the world have a middle class? Review of Income and Wealth, 48(2):155-178, 2002.
Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-o--distriuution detec- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 8, Virtual, Feb. 2022.
Mary Moya and Don Hush. Network constraints and multi-ohjeetivv multi-ohhjetiveeopiiiizaooo for one-class classification. Neural Networks, 0(3):463-474, 1996.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Proceedings of NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Granada, Spain, Dec. 2011.

---

Under review as a conference paper at ICLR 2023
Tuan Anh Nguyen and Anh Tuan Tran. Wanet imperceptible warping- based backdoor attack. In Proceedings of the International Conference on Learning Representations, Virtual, May 2021.
Massimiliano Pastore and Antonio Calcagni. Measuring distribution similarities between samples: A distribution-ffee overlapping index. Frontiers in Psychology, 10: 10:1089, 2019.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Oegan: One-class novelty detection using gans with constrained latent representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2898-2906, Long Beach, CA, June 2019.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In Proceedings of International Conference Machine Learning, pp. 4393-44994433 4402, Stockholm, Sweden, July 2018.
Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. Estimating the support of high-dimensional distribution. Neural Computation, 1443) 2001.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recog- nition benchmark: A multi-class classification competition. In Proceedings of the International Joint Conference on Neural Networks, pp. 1453-1460, San Jose, CA, July 2011.
David Martinus Johannes Tax. One-class classification: Concept learning in the absence of counter- examples. PhD thesis, Technische Universiteit Delft, 2002.
Kjell Rong Utne, Geir Huse, Geir Ottersen, Jens Christian Holst, Vladimir Zabavnikov, Jan Arge Jacobsen, Gumundur J Óskarsson, and Leif Nottestad. Horizontal distribution and overlap of planktivorous fish stocks in the norwegian sea during summers 1995-2006. Marine Biology Re- search, 8(5-6):420- 441, 2012.
Akshaj Kumar Veldanda, Kang Liu, Benjamin Tan, Prashanth Krishnamurthy. Farshad Khorrami, Ramesh Karri, Brendan Dolan-Gavin and Siddharth Garg. Nnoculation: Catching badnets in the wild. Proceedings of the ACM Workshop on Artificial Intelligence and Security, pp. 49-60, Virtual, Nov. 2021.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In Pro- ceedings of IEEE Symposium on Security and Privacy, pp- 707-723, San Francisco, CA, May 2019.
Murray S Weitzman. Measures of overlap of income distributions of white and Negro families the United States, volume 3. US Bureau of the Census, 1970.
Lior Wolf, Tal Hassner, and Itay Maoz. Face recognition in unconstrained videos with matched background similarity. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 529-534, Colorado Springs, CO, June 2011.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsoureing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755. 2015.
Jingkang Yang, Kaiyang Zhou, Yixuan Li. and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2 10. 1134, 2021.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 03365,, 2015.
Sergey Zagoruyko and Nikos Komodakis, arXiv:1605.077166 2016.
Wide residual networks. arXiv preprint
Yi Zeng, Won Park, Z Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 16473-16481, 2021.
12

---

Under review as a conference paper at ICLR 2023
Clean
Triggers
Clean
Triggers
MNIST
All label
Naive.1
Naive.2
ImageNet
Invisible
7
7
CIFAR-10
TCA.1
TCA.2
Wanet
Sunglasses
YouTube Face
GTSRB
Moving
Filter
Wanet
50
Lipstick
Figure 5:
Pictures under "Triggers" are poisoned samples regarding different backdoored attacks. Pictures under "Clean" are clean samples for each dataset.
A DETAILS FOR NOVELTY DETECTION
The method in Table is the same order as shown in Fig. 3
Table 4: Mcans and standard deviations of AUROC (%) for different methods on 100 UCI datasets.
Ours
LI-Ball
K-Center
Parzen
Gaussian
K-Mean
79 + 15.6
65.4 + 25.4
72.2 + 17.6
77.6 + 18.6
81.1 + 16.6
76.3 + 17.9
I-Nearest
K-Nearest
Auto-Encoder
Linear Programming 77.5 18.9
Principal
Lof Range
Neighbor
Neighbor
Network
Component
78.8 + 18.3
79.2 + 18.3
73.7 = 17.9
75.1 + 18.3
56.5 30.8
Nearest
Minimum
Minimum
Self
Support Vector Machine 69.4 + 24.6
Minimax
Neighbor
Spanning Tree
Covariance
Organizing Map 77.6 + 17.4
Probability
Distance
Determinant
Machine 66.4 + 33.3
77.5 + 16.6
78.8 + 17.4
80.4 + 17.1
Mixture
Local Outlier
Naive Parzen 78.4 + 16.2
Local Correlation
Gaussians
Factor 76.1 19.7
Integral 60.7 23.9
78.7 18.1
B DETAILS FOR OUT-OF-DISTRRIUUIIO DETECTION
Table is for CIFAR-10 and Table for CIFAR-100 case.
C DETAILS FOR BACKDOOR DETECTION
Fig. shows the used triggers and the corresponding clean samples. Table 7 shows the details for backdoor detection.
13

---

Under review as a conference paper at ICLR 2023
Table Results for CIFAR-10 in-distribution case (higher number implies higher accuracy). Bold-
face shows the best performing algorithm, whereas underline shows the second best algorithm.
Out-of-Distriuttion Datasets
Method
TPR95 (%)
AUROC (%)
AUPR (%)
Ours
58.41
90.97
85.64
MSP
40.75
88.31
97.08
Texture
Mahalanobis
62.38
94.46
98.75
Energy Score
47.47
85.47
95.58
GEM
72.61
94.59
98.79
Ours
88.31
96.23
98.49
MSP
52,41
92.11
98.32
SVHN
Mahalanobis
79.34
95.72
99.04
Energy Score
64.20
91.05
97.66
GEM
79
95.65
99.01
Ours
79.45
94.33
94.60
MSP
69.07
95.64
99.13
LSUN-Crop
Mahalanobis
30.06
86.15
97.05
Energy Score
91.89
98.40
99.67
GEM
30.20
86.09
97.03
Ours
69.92
92.03
92.54
MSP
47.45
91.30
98.11
LSUN-Resize
Mahalanobis
35.64
88.12
97.45
Energy Score
71.75
94.12
98.64
GEM
35.45
88.09
97.43
Ours
65.60
90.25
90.33
MSP
43.40
89.72
97.72
iSUN
Mahalanobis
26.77
87.87
97.33
Energy Score
66.27
92.56
98.25
GEM
36.80
87.85
93.33
Ours
72.33
92.76
92.32
MSP
50.63
91.46
98.07
Average Performance
Mahalanobis
46.83
90.46
97.92
Energy Score
68.31
92.32
97.96
GEM
50.81
90.45
97.91
14

---

Under review as a conference paper at ICLR 2023
Table Results for CIFAR-100 in-distribution case (higher number implies higher accuracy). Bold-
face shows the best performing algorithm, whereas underline shows the second best algorithm.
Out-of-Distriuttion Datasets
Method
TPR95 (%)
AUROC (%)
AUPR (%)
Ours
46.13
87.52
80.57
MSP
16.71
73.58
93.02
Texture
Mahalanobis
57.62
90.14
97.62
Energy Score
20.38
76.46
93.68
GEM
57.40
90.17
97.63
Ours
90.69
96.41
98.58
MSP
15.66
71.37
92.89
SVHN
Mahalanobis
51.35
89.25
97.52
Energy Score
14.59
74.10
93.65
GEM
51.51
89.40
97.57
Ours
27.93
80.09
79.32
MSP
33.44
83.71
96.32
LSUN-Crop
Mahalanobis
1.53
58.48
89.73
Energy Score
64.01
93.41
98.59
GEM
1.70
58.42
89.70
Ours
39.80
88.15
86.79
MSP
16.54
75.32
94.03
LSUN-Resize
Mahalanobis
67.20
93.97
98.70
Energy Score
21.38
79.29
94.97
GEM
67.09
94.01
98.70
Ours
35.87
86.01
83.37
MSP
17.02
75.87
94.20
iSUN
Mahalanobis
64.07
92.69
98.32
Energy Score
19.20
78.98
94.90
GEM
64.10
92.73
98.32
Ours
48.08
87.63
85.72
MSP
19.87
75.97
94.09
Average Performance
Mahalanobis
48.35
84.90
96.37
Energy Score
27.91
80.44
95.15
GEM
48.36
84.94
96.38
15

---

Under review as a conference paper at ICLR 2023
Table 7: Comparison results for backdoor detection (higher number implies higher accuracy).
Datasets
Trigger
Metrics (%)
Ours
STRIP
Mahalanobis
GEM
MSP
TPR95
83.05
2.58
50.83
100
100
MNIST
All label
AUROC
96.13
44.69
90.78
50.43
AUPR
94.20
35.47
86.71
70.94
70.83
TPR95
100
98.85
99.86
100
5.11
MNIST
Naive.1
AUROC
97.50
97.32
97.49
53.95
51.64
AUPR
96.17
95.95
96.38
74.74
50.41
TPR95
96.53
67.46
35.16
100
14.69
MNIST
Naive.2
AUROC
97.28
93.67
78.63
53.51
58.14
AUPR
95.75
89.85
78.65
74.62
64.16
TPR95
100
35.68
100
100
4.38
CIFAR-10
TCA.1
AUROC
97.50
83.00
97.49
50
49.23
AUPR
95.47
73.22
97.84
76.32
52.64
TPR95
100
27.86
100
100
0.02
CIFAR-10
TCA.2
AUROC
97.50
68.79
97.49
50
29.90
AUPR
97.63
72.41
95.86
67.86
18.05
TPR95
37.87
0.07
20.35
22.90
100
CIFAR-10
Wanet
AUROC
92.74
34.97
50.61
57.81
AUPR
89.95
37.42
57.30
68.48
74.87
TPR95
99.99
54
GTSRB
Moving
AUROC
85.39
7.29
Fail: dependent data dimensions
AUPR
96.96
89.07
TPR95
85.39
7.29
GTSRB
Filter
AUROC
96.54
38.92
Fail: dependent data dimensions
AUPR
95.42
38.81
TPR95
100
1.24
0.51
100
100
GTSRB
Wanet
AUROC
97.50
36.31
54.46
50
AUPR
97.62
39.53
48.92
75.23
75.23
TPR95
73.37
83.03
71.64
98.58
13,06
YouTube Face
Sunglasses
AUROC
95.21
94.80
94.38
84.29
66.55
AUPR
93.00
95.54
94.63
88.83
53.27
TPR95
96.64
90.14
90.88
94.18
3.73
YouTube Face
Lipstick
AUROC
97.21
93.15
93.26
80.80
50.14
AUPR
96.30
94.98
95.09
86.53
53.27
TPR95
100
7.01
0.5
100
51.40
sub-ImageNet
Invisible
AUROC
97.49
66.26
4.78
50
93.61
AUPR
96.53
62.83
12.27
75.26
92.46
TPR95
89.40
39.60
56.97
91.57
39.24
Average Performance
AUROC
96.68
70.30
75.94
58.08
54.92
AUPR
95.42
68.76
76.37
75.88
60.52
16


Reviews:
[{"review_id": "yKCfp_eelE", "reviewer": "Reviewer_yR91", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed an upper bound to overlap index to measure the distributional distance between two distributions in a distribution-free setting, i.e. when we are only given access to finite samples from the underlying distributions. The authors first present a theoretical result to approximate the accuracy of a model with their upper bound on the overlap index.  Second, the authors employ this bound to study the distribution membership of given examples. Experimental results on out-of-distribution and novelty detection show that the proposed bound outperform existing heuristics on average in the AUROC metric. ", "summary_of_the_review": "Overall, the paper presents an interesting yet simple upper bound of the overlap index. The upper bound is distribution-free and can be computed only with samples. An interesting application for one-class classification is discussed in the paper. However, the section with the application of accuracy estimation under distribution shift is a bit skim, and currently, it is unclear if the proposed upper bound on the overlap index can be leveraged in any non-trivial manner. Moreover, the paper lacks some crucial experimental and baseline details.  ", "strength_and_weaknesses": "**Strengths** \n-  The paper presents an interesting yet simple upper bound of the overlap index. The upper bound is distribution-free and can be computed only with samples. \n-  An interesting application of the bound on one-class classification is presented in the paper.  Experimental evidence for novelty detection, out-of-distribution detection, and backdoor detection shows the efficacy of the method. Moreover, the proposed one-way classifier does not need to train any parameters. \n\n\n\n**Weaknesses** \n- In Theorem 2, authors present an approximation to the accuracy of a model under distribution shift. However, it is unclear if this bound is practically useful. Moreover, there are no experiments highlighting the efficacy or practicality of the bound. Specifically, this may happen because we can not compute the term q in the bound. \n- Abstract and introduction of the paper should be toned down with respect to the above contribution. In particular, it should be made clear that the application to accuracy estimation is only of theoretical interest.  \n- Several crucial experimental details and design choices are missing in the paper. Please refer to the response below for details. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qClidicIT2", "reviewer": "Reviewer_NVdx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to compute the distance between two distributions. \nThe authors show how this can be used for novelty detection, out-of-distribution classification, and the like. The authors present results primarily focusing on these tasks, and compare them with traditional novelty detection methods.", "summary_of_the_review": "The paper tries to attack an important problem of measuring the distance between two distributions. However, the technique that the paper proposes is quite similar to the statistics-based method for distribution distance computation. The results section of the paper is missing a comparison with some commonly used techniques, and the proposed method does not stand out as the clear winner in any of the settings. ", "strength_and_weaknesses": "Strengths\n- The paper is nicely written\n\nWeakness\n- The results on novelty class detection are not convincing. Traditional methods perform almost similarly to the proposed method\n- The performance for out-of-distribution detection is also not convincing. There is no clear winner, and the proposed method is only good on half of the datasets. Similar comment for Table 3.\n- The authors do not compare with Deep One-Class Classification, which is a relevant work.\n- Figure 3 is not well presented. The captions should be self-explanatory.\n- Overall it is not clear what is the utility of this work compared to other works in the literature, as it hardly introduces a new concept, findings, or better performance. \n \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "faA_habTGgJ", "reviewer": "Reviewer_S8uA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors define an upper bound for the overlap index and then by converting it into confidence score they can train a one-class classifier. Since the bound is estimated on unknown distribution then the classifier is also distribution free. ", "summary_of_the_review": "Given the comments, I vote for borderline reject. ", "strength_and_weaknesses": "Strength: \nInteresting idea\nWeakness:\nMinor improvement in results AUPR is not better than others or not that much increased. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject