Title: Unsupervised Anomaly Detection with Rejection
Venue: NeurIPS 2023
Accepted: yes
Keywords: Anomaly Detection

Learning with Rejection

Unsupervised Learning
Abstract: Anomaly detection aims at detecting unexpected behaviours in the data. Because anomaly detection is usually an unsupervised task, traditional anomaly detectors learn a decision boundary by employing heuristics based on intuitions, which are hard to verify in practice. This introduces some uncertainty, especially close to the decision boundary, that may reduce the user trust in the detector's predictions. A way to combat this is by allowing the detector to reject predictions with high uncertainty (Learning to Reject). This requires employing a confidence metric that captures the distance to the decision boundary and setting a rejection threshold to reject low-confidence predictions. However, selecting a proper metric and setting the rejection threshold without labels are challenging tasks. In this paper, we solve these challenges by setting a constant rejection threshold on the stability metric computed by ExCeeD. Our insight relies on a theoretical analysis of such a metric. Moreover, setting a constant threshold results in strong guarantees: we estimate the test rejection rate, and derive a theoretical upper bound for both the rejection rate and the expected prediction cost. Experimentally, we show that our method outperforms some metric-based methods.

Unsupervised Anomaly Detection with Rejection

Abstract

Anomaly detection goal is to detect unexpected behaviours in the data. Because anomaly detection is usually an unsupervised task, traditional anomaly detectors learn a decision boundary by employing heuristics based on intuitions, which are hard to verify in practice. This introduces some uncertainty, especially close to the decision boundary, which may reduce the user trust in the detector's predictions. A way to combat this is by allowing the detector to reject examples with high uncertainty (Learning to Reject). This requires employing a confidence metric that captures the distance to the decision boundary and setting a rejection threshold to reject low-confidence predictions. However, selecting a proper metric and setting the rejection threshold without labels are challenging tasks. In this paper, we solve these challenges by setting a constant rejection threshold on the stability metric computed by EXCEED. Our insight relies on a theoretical analysis of this metric. Moreover, setting a constant threshold results in strong guarantees: we estimate the test rejection rate, and derive a theoretical upper bound for both the rejection rate and the expected prediction cost. Experimentally, we show that our method outperforms some metric-based methods.

1 Introduction

Anomaly detection is the task of detecting unexpected behaviors in the data [6]. Often, these anomalies are critical adverse events such as the destruction or alteration of proprietary user data [29], water leaks in stores [50], breakdowns in gas [65] and wind [64] turbines, or failures in petroleum extraction [45]. Usually, anomalies are associated with a cost such as a monetary cost (e.g., maintenance, paying for fraudulent purchases) or a societal cost such as environmental damages (e.g., dispersion of petroleum or gas). Hence, detecting anomalies in a timely manner is an important problem.

When using an anomaly detector for decision-making, it is crucial that the user trusts the system. However, it is often hard or impossible to acquire labels for anomalies. Moreover, anomalies may not follow a pattern. Therefore anomaly detection is typically treated as an unsupervised learning problem where traditional algorithms learn a decision boundary by employing heuristics based on intuitions [22, 55, 61, 41], such as that anomalies are far away from normal examples [3]. Because these intuitions are hard to verify and may not hold in some cases, some predictions may have high uncertainty, especially for examples close to the decision boundary [32, 28]. As a result, the detector's predictions should be treated with some circumspection.

One way to increase user trust is to consider Learning to Reject [12]. In this setting, the model does not always make a prediction. Instead, it can abstain when it is at a heightened risk of making a mistake thereby improving its performance when it does offer a prediction. Abstention has the drawback that no prediction is made, which means that a person must intervene to make a decision. In the literature, two types of rejection have been identified [25]: novelty rejection allows the model to abstain when given an out-of-distribution (OOD) example, while ambiguity rejection enables abstention for a test example that is too close to the model's decision boundary. Because anomalies

often are OOD examples, novelty rejection does not align well with our setting as the model would reject all OOD anomalies (i.e., a full class) [10, 63, 30]. On the other hand, current approaches for ambiguity rejection threshold what constitutes being too close to the decision boundary by evaluating the model's predictive performance on the examples for which it makes a prediction (i.e., accepted), and those where it abstains from making a prediction (i.e., rejected) [9, 44, 16]. Intuitively, the idea is to find a threshold where the model's predictive performance is (1) significantly lower on rejected examples than on accepted examples and (2) higher on accepted examples than on all examples (i.e., if it always makes a prediction). Unfortunately, existing learning to reject approaches that set a threshold in this manner require labeled data, which is not available in anomaly detection.

This paper fills this gap by proposing an approach to perform ambiguity rejection for anomaly detection in a completely unsupervised manner. Specifically, we make three major contributions. First, we conduct a thorough novel theoretical analysis of a stability metric for anomaly detection [49] and show that it has several previously unknown properties that are of great importance in the context of learning to reject. Namely, it captures the uncertainty close to the detector's decision boundary, and only limited number of examples get a stability value strictly lower than 1 . Second, these enabls us to design an ambiguity rejection mechanism without any labeled data that offers strong guarantees which are often sought in Learning to Reject [12, 60, 7] We can derive an accurate estimate of the rejected examples proportion, as well as a theoretical upper bound that is satisfied with high probability. Moreover, given a cost function for different types of errors, we provide an estimated upper bound on the expected cost at the prediction time. Third, we evaluate our approach on an extensive set of unsupervised detectors and benchmark datasets and conclude that (1) it performs better than several adapted baselines based on other unsupervised metrics, and (2) our theoretical results hold in practice.

2 Preliminaries and notation

Wewill introduce the relevant background on anomaly detection, learning to reject, and the EXCEED's metric that this paper builds upon.

Anomaly Detection. Let X be a d dimensional input space and D = { x 1 , . . . , x n } be a training set, where each x i ∈ X . The goal in anomaly detection is to train a detector f : X → R that maps examples to a real-valued anomaly score, denoted by s . In practice, it is necessary to convert these soft scores to a hard prediction, which requires setting a threshold λ . Assuming that higher scores equate to being more anomalous, a predicted label ˆ y can be made for an example x as follows: ˆ y = 1 (anomaly) if s = f ( x ) ≥ λ , while ˆ y = 0 (normal) if s = f ( x ) < λ . We let ˆ Y be the random variable that denotes the predicted label. Because of the absence of labels, one usually sets the threshold such that γ × n scores are ≥ λ , where γ is the dataset's contamination factor (i.e., expected proportion of anomalies) [51, 50].

Learning to Reject. Learning to reject extends the output space of the model to include the symbol ®, which means that the model abstains from making a prediction. This entails learning a second model r (the rejector) to determine when the model abstains. A canonical example of ambiguity rejection is when r consists of a pair [confidence M s , rejection threshold τ ] such that an example is rejected if the detector's confidence is lower than the threshold. The model output becomes

̸

A standard approach is to evaluate different values for τ to find a balance between making too many incorrect predictions because τ is too low (i.e., ˆ y = y but M s > τ ) and rejecting correct predictions because τ is too high (i.e., ˆ y = y but M s ≤ τ ) [9, 44, 16]. Unfortunately, in an unsupervised setting, it is impossible to evaluate the threshold because it relies on having access to labeled data.

EXCEED's metric. Traditional confidence metrics (such as calibrated class probabilities) quantify how likely a prediction is to be correct, This obviously requires labels [9] which are unavailable in an unsupervised setting. Thus, one option is to move the focus towards the concept of stability : given a fixed test example x with anomaly score s , perturbing the training data alters the model learning, which, in turn, affects the label prediction . Intuitively, the more stable a detector's output is for a test example, the less sensitive its predicted label is to changes in the training data. On the other hand, when P ( ˆ Y = 1 | s ) ≈ P ( ˆ Y = 0 | s ) ≈ 0 . 5 the prediction for x is highly unstable, as training the

detector with slightly different examples would flip its prediction for the same test score s . Thus, a stability-based confidence metric M s can be expressed as the margin between the two classes' probabilities:

where the lower M s the more unstable the prediction.

Recently, Perini et al. introduced EXCEED to estimate the detector's stability P ( ˆ Y = 1 | s ) . Roughly speaking, EXCEED uses a Bayesian formulation that simulates bootstrapping the training set as a form of perturbation. Formally, it measures such stability for a test score s in two steps.

First , it computes the training frequency ψ n = |{ i ≤ n : s i ≤ s }| n ∈ [0 , 1] , i.e. the proportion of training scores lower than s . This expresses how extreme the score s ranks with respect to the training scores. Second , it computes the probability that the score s will be predicted as an anomaly when randomly drawing a training set of n scores from the population of scores. In practice, this is the probability that the chosen threshold λ will be less than or equal to the score s . The stability is therefore estimated as

Assumption. EXCEED's Bayesian formulation requires assuming that Y | s follows a Bernoulli distribution with parameter p s = P ( S ≤ s ) , where S is the detector's population of scores. Note that the stability metric is a detector property and, therefore, is tied to the specific choice of the unsupervised detector f .

3 Methodology

This paper addresses the following problem:

Given: An unlabeled dataset D with contamination γ , an unsupervised detector f , a cost function c ;

Do: Introduce a reject option to f , i.e. find a pair (confidence, threshold) that minimizes the cost.

We propose an anomaly detector-agnostic approach for performing learning to reject that requires no labels . Our key contribution is a novel theoretical analysis of the EXCEED confidence metric that proves that only a limited number of examples have confidence lower than 1 -ε (Sec. 3.1). Intuitively, the detector's predictions for most examples would not be affected by slight perturbations of the training set: it is easy to identify the majority of normal examples and anomalies because they will strongly adhere to the data-driven heuristics that unsupervised anomaly detectors use. For example, using the data density as a measure of anomalousness [5] tends to identify all densely clustered normals and isolated anomalies, which constitute the majority of all examples. In contrast, only relatively few cases would be ambiguous and hence receive low confidence (e.g., small clusters of anomalies and normals at the edges of dense clusters).

Our approach is called REJEX (Rejecting via ExCeeD) and simply computes the stability-based confidence metric M s and rejects any example with confidence that falls below threshold τ = 1 -ε . Theoretically, this constant reject threshold provides several relevant guarantees. First, one often needs to control the proportion of rejections (namely, the rejection rate ) to estimate the number of decisions left to the user. Thus, we propose an estimator that only uses training instances to estimate the rejection rate at test time . Second, because in some applications avoiding the risk of rejecting all the examples is a strict constraint, we provided an upper bound for the rejection rate (Sec. 3.2). Finally, we compute a theoretical upper bound for a given cost function that guarantees that using REJEX keeps the expected cost per example at test time low (Sec. 3.3).

3.1 Setting the Rejection Threshold through a Novel Theoretical Analysis of EXCEED

Our novel theoretical analysis proves (1) that the stability metric by EXCEED is lower than 1 -ε for a limited number of examples (Theorem 3.1), and (2) that such examples with low confidence are the ones close to the decision boundary (Corollay 3.2). Thus, we propose to reject all these uncertain examples by setting a rejection threshold

where 2 e -T is the tolerance that excludes unlikely scenarios, and T ≥ 4 is required for Theorem 3.1.

We motivate our approach as follows. Given an example x with score s and the proportion of lower training scores ψ n , Theorem 3.1 shows that the confidence M s is lower than 1 -2 e -T (for T ≥ 4 ) if ψ n belongs to the interval [ t 1 , t 2 ] . By analyzing [ t 1 , t 2 ] , Corollary 3.2 proves that the closer an example is to the decision boundary, the lower the confidence M s , and that a score s = λ (decision threshold) has confidence M s = 0 .

Remark. Perini et al. performed an asymptotic analysis of EXCEED that investigates the metric's behavior when the training set's size n → + ∞ . In contrast, our novel analysis is finite-sample and hence provides more practical insights, as real-world scenarios involve having a finite dataset with size n ∈ N .

Theorem 3.1 (Analysis of EXCEED) . Let s be an anomaly score, and ψ n ∈ [0 , 1] its training frequency. For T ≥ 4 , there exist t 1 = t 1 ( n, γ, T ) ∈ [0 , 1] , t 2 = t 2 ( n, γ, T ) ∈ [0 , 1] such that

Proof. See the Supplement for the formal proof.

The interval [ t 1 , t 2 ] has two relevant properties. First, it becomes narrower when increasing n (P1) and larger when increasing T (P2). This means that collecting more training data results in smaller rejection regions while decreasing the tolerance ε = 2 e -T has the opposite effect. Second, it is centered (not symmetrically) on 1 -γ (P3-P4), which means that examples with anomaly scores close to the decision threshold λ are the ones with a low confidence score (P5). The next Corollary lists these properties.

Corollary 3.2. Given t 1 , t 2 as in Theorem 3.1, the following properties hold for any s , n , γ , T ≥ 4 :

P2. t 1 and t 2 are, respectively, monotonic decreasing and increasing as functions of T ;

P3. the interval always contains 1 -γ , i.e. t 1 ≤ 1 -γ ≤ t 2 ;

P4. for n →∞ , there exists s ∗ with ψ n = t ∗ ∈ [ t 1 , t 2 ] such that t ∗ → 1 -γ and M s → 0 .

Proof sketch. For P1, it is enough to observe that t 1 , t 2 → 1 -γ for n → + ∞ . For P2 and P3, the result comes from simple algebraic steps. P4 follows from the surjectivity of M s when n → + ∞ , the monotonicity of P ( ˆ Y = 1 | s ) , from P1 with the squeeze theorem. Finally, P5 follows from ψ n ∈ [ t 1 , t 2 ] = ⇒ s ∈ [ ψ -1 n ( t 1 ) , ψ -1 n ( t 2 ) ] , as ψ n is monotonic increasing, where ψ -1 n is the inverseimage of ψ n . Because for P3 1 -γ ∈ [ t 1 , t 2 ] , it holds that ψ -1 n ( t 1 ) ≤ ψ -1 n (1 -γ ) = λ ≤ ψ -1 n ( t 2 ) . This implies that s ∈ [ λ -u 1 , λ + u 2 ] , where u 1 = λ -ψ -1 n ( t 1 ) , u 2 = λ -ψ -1 n ( t 2 ) .

3.2 Estimating and Bounding the Rejection Rate

It is important to have an estimate of the rejection rate, which is the proportion of examples for which the model will abstain from making a prediction. This is an important performance characteristic for differentiating among candidate models. Moreover, it is important that not all examples are rejected because such a model is useless in practice. We propose a way to estimate the rejection rate and Theorem 3.5 shows that our estimate approaches the true rate for large training sets. We strengthen our analysis and introduce an upper bound for the rejection rate, which guarantees that, with arbitrarily high probability, the rejection rate is kept lower than a constant (Theorem 3.6).

Definition 3.3 (Rejection rate) . Given the confidence metric M s and the rejection threshold τ , the rejection rate R = P ( M s ≤ τ ) is the probability that a test example with score s gets rejected.

We propose the following estimator for the reject rate:

Definition 3.4 (Rejection rate estimator) . Given anomaly scores s with training frequencies ψ n , let g : [0 , 1] → [0 , 1] be the function such that P ( ˆ Y = 1 | s ) = g ( ψ n ) (see Eq. 1). We define the rejection rate estimator ˆ R as

where g -1 is the inverse-image through g , and, for u ∈ [0 , 1] , ˆ F ψ n ( u ) = | i ≤ n : ψ n ( s i ) ≤ u | n is the empirical cumulative distribution of ψ n .

Note that ˆ R can be computed in practice, as the ψ n has a distribution that is arbitrarily close to uniform, as stated by Theorem A.1 and A.2 in the Supplement.

Theorem 3.5 (Rejection rate estimate) . Let g be as in Def. 3.4. Then, for high values of n , ˆ R≈R .

Proof. From the definition of rejection rate 3.3, it follows

where F ψ n ( · ) = P ( ψ n ≤ · ) is the theoretical cumulative distribution of ψ n . Because the true distribution of ψ n for test examples is unknown, the estimator approximates F ψ n using the training scores s i and computes the empirical ˆ F ψ n . As a result,

Theorem 3.6 (Rejection rate upper bound) . Let s be an anomaly score, M s be its confidence value, and τ = 1 -2 e -T be the rejection threshold. For n ∈ N , γ ∈ [0 , 0 . 5) , and small δ > 0 , there exists a positive real function h ( n, γ, T, δ ) such that R≤ h ( n, γ, T, δ ) with probability at least 1 -δ , i.e. the rejection rate is bounded.

Proof. Theorem 3.1 states that there exists two functions t 1 = t 1 ( n, γ, T ) , t 2 = t 2 ( n, γ, T ) ∈ [0 , 1] such that the confidence is lower than τ if ψ n ∈ [ t 1 , t 2 ] . Moreover, Theorems A.1 and A.2 claim that ψ n has a distribution that is close to uniform with high probability (see the theorems and proofs in the Supplement). As a result, with probability at least 1 -δ , we find h ( n, γ, T, δ ) as follows:

3.3 Upper Bounding the Expected Test Time Cost

In a learning with reject scenario, there are costs associated with three outcomes: false positives ( c fp > 0 ), false negatives ( c fn > 0 ), and rejection ( c r ) because abstaining typically involves having a person intervene. Estimating an expected per example prediction cost at test time can help with model selection and give a sense of performance. Theorem 3.8 provides an upper bound on the expected per example cost when (1) using our estimated rejection rate (Theorem 3.5), and (2) setting the decision threshold λ as in Sec. 2.

Definition 3.7 (Cost function) . Let Y be the true label random variable. Given the costs c fp > 0 , c fn > 0 , and c r , the cost function is a function c : { 0 , 1 } × { 0 , 1 , ® } → R such that

Note that defining a specific cost function requires domain knowledge. Following the learning to reject literature, we set an additive cost function. Moreover, the rejection cost needs to satisfy the inequality c r ≤ min { (1 -γ ) c fp , γc fn } . This avoids the possibility of predicting always anomaly for an expected cost of (1 -γ ) c fp , or always normal with an expected cost of γc fn [52].

Theorem 3.8. Let c be a cost function as defined in Def. 3.7, and g be as in Def. 3.4. Given a (test) example x with score s , the expected example-wise cost is bounded by

where A = ˆ F ψ n ( g -1 ( e -T ) ) and B = ˆ F ψ n ( g -1 ( 1 -e -T ) )

Proof. We indicate the true label random variable as Y , and the non-rejected false positives and false negatives as, respectively,

Using Theorem 3.5 results in

where A = ˆ F ψ n ( g -1 ( e -T ) ) , B = ˆ F ψ n ( g -1 ( 1 -e -T ) ) come from Theorem 3.5. Now we observe that setting a decision threshold λ such that n × γ scores are higher implies that, on expectation, the detector predicts a proportion of positives equal to γ = P ( Y = 1) . Moreover, for ε = 2 e -T ,

FP ≤ P ( ˆ Y = 1 | M s > 1 -ε ) = 1 -B as false positives must be less than total accepted positive predictions;

FN ≤ γ and FN ≤ P ( ˆ Y = 0 | M s > 1 -ε ) = A , as you cannot have more false negatives than positives ( γ ), nor than accepted negative predictions ( A ).

From these observations, we conclude that E x [ c ] ≤ min { γ, A } c fn +(1 -B ) c fp +( B -A ) c r .

4 Related work

There is no research on learning to reject in unsupervised anomaly detection. However, three main research lines are connected to this work.

Supervised methods. If some labels are available, one can use traditional supervised approaches to add the reject option into the detector [11, 38]. Commonly, labels can be used to find the optimal rejection threshold in two ways: 1) by trading off the model performance (e.g., AUC) on the accepted examples with its rejection rate [24, 1], or 2) by minimizing a cost function [46, 7], a risk function [18, 27], or an error function [35, 33]. Alternatively, one can include the reject option in the model and directly optimize it during the learning phase [60, 12, 31].

Self-Supervised methods. If labels are not available, one can leverage self-supervised approaches to generate pseudo-labels in order to apply traditional supervised learning to reject methods [26, 59, 19, 37]. For example, one can employ any unsupervised anomaly detector to assign training labels, fit a (semi-)supervised detector (such as DEEPSAD [57] or REPEN [47]) on the pseudo labels, compute a confidence metric [14], and find the optimal rejection threshold by minimizing the cost function treating the pseudo-labels as the ground truth.

3) Optimizing unsupervised metrics. There exist several unsupervised metrics (i.e., they can be computed without labels) for quantifying detector quality [43]. Because they do not need labels, one can find the rejection threshold by maximizing the margin between the detector's quality (computed using such metric) on the accepted and on the rejected examples [54]. This allows us to obtain a model that performs well on the accepted examples and poorly on the rejected ones, which is exactly the same intuition that underlies the supervised approaches. Some examples of existing unsupervised metrics (see [43]) are the following. EM and MV [20] quantify the clusterness of inlier scores, where more compact scores indicate better models. STABILITY [48] measures the robustness of anomaly detectors' predictions by looking at how consistently they rank examples by anomalousness. UDR [15] is a model-selection metric that selects the model with a hyperparameter setting that yields consistent results across various seeds, which can be used to set the rejection threshold through the analogy [hyperparameter, seed] and [rejection threshold, detectors]. Finally, ENS [56, 67] measures the detector trustworthiness as the ranking-based similarity (e.g., correlation) of a detector's output to the 'pseudo ground truth', computed via aggregating the output of an ensemble of detectors, which allows one to set the rejection threshold that maximizes the correlation between the detector's and the ensemble's outputs.

5 Experiments

We experimentally address the following research questions:

Q1. How does REJEX's cost compare to the baselines?

Q2. How does varying the cost function affect the results?

Q3. How does REJEX's CPU time compare to the baselines?

Q4. Do the theoretical results hold in practice?

Q5. Would REJEX's performance significantly improve if it had access to training labels?

5.1 Experimental Setup

Methods. We compare REJEX 1 against 7 baselines for setting the rejection threshold. These can be divided into three categories: no rejection, self-supervised, and unsupervised metric based.

We use one method NOREJECT that always makes predictions and never rejects (no reject option).

We consider one self-supervised approach SS-REPEN [47]. This uses (any) unsupervised detector to obtain pseudo labels for the training set. It then sets the rejection threshold as follows: 1) it creates a held-out validation set (20 % ), 2) it fits REPEN, a state-of-the-art (semi-)supervised anomaly detector on the training set with the pseudo labels, 3) it computes on the validation set the confidence values as the margin between REPEN's predicted class probabilities | P ( Y = 1 | s ) -P ( Y = 0 | s ) | , 4) it finds the optimal threshold τ by minimizing the total cost obtained on the validation set.

We consider 5 approaches that employ an existing unsupervised metric to set the rejection threshold and hence do not require having access to labels. MV [20], EM [20], and STABILITY [48] are unsupervised metric-based methods based on stand-alone internal evaluations that use a single anomaly detector to measure its quality, UDR [15] and ENS [56] are unsupervised consensus-based metrics that an ensemble of detectors (all 12 considered in our experiments) to measure a detector's quality. 2 We apply each of these 5 baselines as follows. 1) We apply the unsupervised detector to assign an anomaly score to each train set example. 2) We convert these scores into class probabilities using [34]. 3) We compute the confidence scores on the training set as difference between these probabilities: | P ( Y = 1 | s ) -P ( Y = 0 | s ) | . 4) We evaluate possible thresholds on this confidence by computing the considered unsupervised metric on the accepted and on the rejected examples and select the threshold that maximizes the difference in the metric's value on these two sets of examples. This aligns with the common learning to reject criteria for picking a threshold [9, 54] such that the model performs well on the accepted examples and poorly on the rejected ones.

Data. We carry out our study on 34 publicly available benchmark datasets, widely used in the literature [23]. These datasets cover many application domains, including healthcare (e.g., disease diagnosis), audio and language processing (e.g., speech recognition), image processing (e.g., object identification), and finance (e.g., fraud detection). To limit the computational time, we randomly sub-sample 20 , 000 examples from all large datasets. Table 3 in the Supplement provides further details.

Anomaly Detectors and Hyperparameters. We set our tolerance ε = 2 e -T with T = 32 . Note that the exponential smooths out the effect of T ≥ 4 , which makes setting a different T have little impact. We use a set of 12 unsupervised anomaly detectors implemented in PYOD [66] with default hyperparameters [62] because the unsupervised setting does not allow us to tune them: KNN [3], IFOREST [42], LOF [5], OCSVM [58], AE [8], HBOS [21], LODA [53], COPOD [39], GMM [2], ECOD [40], KDE [36], INNE [4]. We set all the baselines' rejection threshold via Bayesian Optimization with 50 calls [17].

Setup. For each [dataset, detector] pair, we proceed as follows: (1) we split the dataset into training and test sets (80-20) using 5 fold cross-validation; (2) we use the detector to assign the anomaly scores on the training set; (3) we use either REJEX or a baseline to set the rejection threshold;

(4) we measure the total cost on the test set using the given cost function. We carry out a total of 34 × 12 × 5 = 2040 experiments. All experiments were run on an Intel(R) Xeon(R) Silver 4214 CPU.

5.2 Experimental Results

Inne

IForest

Hbos

Knn

Ecod

Ocsvm

Ae

Gmm

Copod

Lof

0.12

0.13

0.14

0.15

0.16

Average Cost per Example

Inne

IForest

Hbos

Knn

Ecod

Ocsvm

Ae

Gmm

Copod

Lof

2

3

4

5

6

Average Ranks

RejEx

SS-Repen

Mv

Ens

Em

Stability

NoReject

Q1: REJEX against the baselines. Figure 1 shows the comparison between our method and the baselines, grouped by detector, when setting the costs c fp = c fn = 1 and c r = γ (see the Supplement for further details). REJEX achieves the lowest (best) cost per example for 9 out of 12 detectors (left-hand side) and similar values to SS-REPEN when using LODA, LOF and KDE. Averaging over the detectors, REJEX reduces the relative cost by more than 5% vs SS-REPEN, 11% vs ENS, 13% vs MV and UDR, 17% vs EM, 19% vs NOREJECT. Table 4 (Supplement) shows a detailed breakdown.

For each experiment, we rank all the methods from 1 to 8 , where position 1 indicates the lowest (best) cost. The right-hand side of Figure 1 shows that REJEX always obtains the lowest average ranking. We run a statistical analysis separately for each detector: the Friedman test rejects the null-hypothesis that all methods perform similarly (p-value < e -16 ) for all the detectors. The ranking-based post-hoc Bonferroni-Dunn statistical test [13] with α = 0 . 05 finds that REJEX is significantly better than the baselines for 6 detectors (INNE, IFOREST, HBOS, KNN, ECOD, OCSVM).

Hbos

IForest

Inne

Ocsvm

Ecod

Copod

Knn

Ae

Lof

Gmm

0.5

0.6

0.7

0.8

0.9

Average Cost per Example

cfp

=10

cfn

=1

cr

=min{(1

)

cfp

,

cfn

}

Hbos

IForest

Inne

Ocsvm

Ecod

Copod

Knn

Ae

Lof

Gmm

cfp

=1

cfn

=10

cr

=min{(1

)

cfp

,

cfn

}

Hbos

IForest

Inne

Ocsvm

Ecod

Copod

Knn

Ae

Lof

Gmm

cfp

=5

cfn

=5

cr

=

RejEx

SS-Repen

Mv

Ens

Em

Stability

NoReject

Q2. Varying the costs c fp , c fn , c r . The three costs c fp , c fn , and c r are usually set based on domain knowledge: whether to penalize the false positives or the false negatives more depends on the application domain. Moreover, the rejection cost needs to satisfy the constraint c r ≤ min { (1 -

γ ) c fp , γc fn } [52]. Therefore, we study their impact on three representative cases: (case 1) high false positive cost ( c fp = 10 , c fn = 1 , c r = min { 10(1 -γ ) , γ ), (case 2) high false negative cost ( c fp = 1 , c fn = 10 , c r = min { (1 -γ ) , 10 γ ), and (case 3) same cost for both mispredictions but low rejection cost ( c fp = 5 , c fn = 5 , c r = γ ). Note that scaling all the costs has no effect on the relative comparison between the methods, so the last case is equivalent to c fp = 1 , c fn = 1 , and c r = γ/ 5 .

Figure 2 shows results for the three scenarios. Compared to the unsupervised metric-based methods, the left plot shows that our method is clearly the best for high false positives cost: for 11 out of 12 detectors, REJEX obtains both the lowest (or similar for GMM) average cost and the lowest average ranking position. This indicates that using REJEX is suitable when false alarms are expensive. Similarly, the right plot illustrates that REJEX outperforms all the baselines for all the detectors when the rejection cost is low (w.r.t. the false positive and false negative costs). Even when the false negative cost is high (central plot), REJEX obtains the lowest average cost for 11 detectors and has always the lowest average rank per detector. See the Supplement (Table 6 and 7) for more details.

Q3. Comparing the CPU time. Table 1 reports CPU time in milliseconds per training example aggregated over the 34 datasets needed for each method to set the rejection threshold on three unsupervised anomaly detectors (IFOREST, HBOS, COPOD). NOREJECT has CPU time equal to 0 because it does not use any reject option. REJEX takes just a little more time than NOREJECT because computing EXCEED has linear time while setting a constant threshold has constant time. In contrast, all other methods take 1000 × longer because they evaluate multiple thresholds. For some of these (e.g., STABILITY), this involves an expensive internal procedure.

Http

Fraud

WDBC

Pendigi

Thyroid

Optdigi

Vowels

Wavefor

Musk

Mammogr

Lymphog

ALOI

WBC

Wilt

Letter

Wine

Annthyr

Shuttle

Cardio

Census

PageBlo

Vertebr

Campaig

Interne

Landsat

Cardiot

WPBC

Donors

Satelli

Yeast

Pima

Fault

0.0

0.3

0.6

Cost per example

Fraud

Http

Census

Campaig

ALOI

Shuttle

Donors

Pendigi

Optdigi

Mammogr

Thyroid

Wavefor

Annthyr

Landsat

Satelli

Wilt

Musk

PageBlo

Vowels

Interne

Letter

Cardiot

Cardio

Fault

Yeast

WDBC

Pima

WBC

Vertebr

Lymphog

Wine

WPBC

Rejection Rate

Theoretical Upper Bound

Empirical value

Theoretical Expected value

Q4. Checking on the theoretical results. Section 3 introduces three theoretical results: the rejection rate estimate (Theorem 3.5), and the upper bound for the rejection rate (Theorem 3.6) and for the cost (Theorem 3.8). We run experiments to verify whether they hold in practice. Figure 3 shows the results aggregated over the detectors. The left-hand side confirms that the prediction cost per example (blue circle) is always ≤ than the upper bound (black line). Note that the upper bound is sufficiently strict, as in some cases it equals the empirical cost (e.g., Census, Wilt, Optdigits).

The right-hand side shows that our rejection rate estimate (orange star) is almost identical to the empirical rejection rate (orange circle) for most of the datasets, especially the large ones. On the other hand, small datasets have the largest gap, e.g., Wine ( n = 129 ), Lymphography ( n = 148 ), WPBC ( n = 198 ), Vertebral ( n = 240 ). Finally, the empirical rejection rate is always lower than the theoretical upper bound (black line), which we compute by using the empirical frequencies ψ n .

Q5. Impact of training labels on REJEX. We simulate having access to the training labels and include an extra baseline: ORACLE uses EXCEED as a confidence metric and sets the (optimal) rejection threshold by minimizing the cost function using the training labels. Table 2 shows the average cost and rejection rates at test time obtained by the two methods. Overall, REJEX obtains an average cost that is only 0 . 6% higher than ORACLE's cost. On a per-detector basis, REJEX obtains a 2 . 5% higher cost in the worst case (with LODA), while getting only a 0 . 08% increase in the best case (with KDE). Comparing the rejection rates, REJEX rejects on average only ≈ 1 . 5 percentage points more examples than ORACLE ( 12 . 9% vs 11 . 4% ). The supplement provides further details.

6 Conclusion and Limitations

This paper addressed learning to reject in the context of unsupervised anomaly detection. The key challenge was how to set the rejection threshold without access to labels which are required by all existing approaches We proposed an approach REJEX that exploits our novel theoretical analysis of the EXCEED confidence metric. Our new analysis shows that it is possible to set a constant rejection threshold and that doing so offers strong theoretical guarantees. First, we can estimate the proportion of rejected test examples and provide an upper bound for our estimate. Second, we can provide a theoretical upper bound on the expected test-time prediction cost per example. Experimentally, we compared REJEX against several (unsupervised) metric-based methods and showed that, for the majority of anomaly detectors, it obtained lower (better) cost. Moreover, we proved that our theoretical results hold in practice and that our rejection rate estimate is almost identical to the true value in the majority of cases.

Limitations. Because REJEX does not rely on labels, it can only give a coarse-grained view of performance. For example, in many applications anomalies will have varying costs (i.e., there are instance-specific costs) which we cannot account for. Moreover, REJEX has a strictly positive rejection rate, which may increase the cost of a highly accurate detector. However, this happens only in ≈ 5% of our experiments.

References

M. R. Abbas, M. S. A. Nadeem, A. Shaheen, A. A. Alshdadi, R. Alharbey, S.-O. Shim, and W. Aziz. Accuracy rejection normalized-cost curves (arnccs): A novel 3-dimensional framework for robust classification. IEEE Access , 7:160125-160143, 2019.

C. C. Aggarwal. An introduction to outlier analysis. In Outlier analysis , pages 1-34. Springer, 2017.

F. Angiulli and C. Pizzuti. Fast outlier detection in high dimensional spaces. In European conference on principles of data mining and knowledge discovery , pages 15-27. Springer, 2002.

T. R. Bandaragoda, K. M. Ting, D. Albrecht, F. T. Liu, Y. Zhu, and J. R. Wells. Isolationbased anomaly detection using nearest-neighbor ensembles. Computational Intelligence , 34(4): 968-998, 2018.

M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data , pages 93-104, 2000.

V. Chandola, A. Banerjee, and V. Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR) , 41(3):1-58, 2009.

N. Charoenphakdee, Z. Cui, Y. Zhang, and M. Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning , pages 1507-1517. PMLR, 2021.

Z. Chen, C. K. Yeo, B. S. Lee, and C. T. Lau. Autoencoder-based network anomaly detection. In 2018 Wireless telecommunications symposium (WTS) , pages 1-5. IEEE, 2018.

C. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory , 16(1):41-46, 1970.

L. Coenen, A. K. Abdullah, and T. Guns. Probability of default estimation, with a reject option. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA) , pages 439-448. IEEE, 2020.

D. Conte, P. Foggia, G. Percannella, A. Saggese, and M. Vento. An ensemble of rejecting classifiers for anomaly detection of audio events. In 2012 IEEE Ninth International Conference on Advanced Video and Signal-Based Surveillance , pages 76-81. IEEE, 2012.

C. Cortes, G. DeSalvo, and M. Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory . Springer, 2016.

J. Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research , 7:1-30, 2006.

C. Denis and M. Hebiri. Consistency of plug-in confidence sets for classification in semisupervised learning. Journal of Nonparametric Statistics , 32(1):42-72, 2020.

S. Duan, L. Matthey, A. Saraiva, N. Watters, C. P. Burgess, A. Lerchner, and I. Higgins. Unsupervised model selection for variational disentangled representation learning. arXiv preprint arXiv:1905.12614 , 2019.

R. El-Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research , 11(5), 2010.

P. I. Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811 , 2018.

Y. Geifman and R. El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International conference on machine learning , pages 2151-2159. PMLR, 2019.

M.-I. Georgescu, A. Barbalau, R. T. Ionescu, F. S. Khan, M. Popescu, and M. Shah. Anomaly detection in video via self-supervised and multi-task learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12742-12752, 2021.

N. Goix. How to evaluate the quality of unsupervised anomaly detection algorithms? arXiv preprint arXiv:1607.01152 , 2016.

M. Goldstein and A. Dengel. Histogram-based outlier score (hbos): A fast unsupervised anomaly detection algorithm. KI-2012: poster and demo track , 9, 2012.

D. Guthrie, L. Guthrie, B. Allison, and Y. Wilks. Unsupervised anomaly detection. In IJCAI , pages 1624-1628, 2007.

S. Han, X. Hu, H. Huang, M. Jiang, and Y. Zhao. Adbench: Anomaly detection benchmark. In Neural Information Processing Systems (NeurIPS) , 2022.

B. Hanczar. Performance visualization spaces for classification with rejection option. Pattern Recognition , 96:106984, 2019.

K. Hendrickx, L. Perini, D. Van der Plas, W. Meert, and J. Davis. Machine learning with a reject option: A survey. arXiv preprint arXiv:2107.11277 , 2021.

H. Hojjati, T. K. K. Ho, and N. Armanfard. Self-supervised anomaly detection: A survey and outlook. arXiv preprint arXiv:2205.05173 , 2022.

L. Huang, C. Zhang, and H. Zhang. Self-adaptive training: beyond empirical risk minimization. Advances in neural information processing systems , 33:19365-19376, 2020.

E. Hüllermeier and W. Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning , 110(3):457-506, 2021.

H. Jmila and M. I. Khedher. Adversarial machine learning for network intrusion detection: A comparative study. Computer Networks , page 109073, 2022.

M. U. K. Khan, H.-S. Park, and C.-M. Kyung. Rejecting motion outliers for efficient crowd anomaly detection. IEEE Transactions on Information Forensics and Security , 14(2):541-556, 2018.

M. A. Kocak, D. Ramirez, E. Erkip, and D. E. Shasha. Safepredict: A meta-algorithm for machine learning that uses refusals to guarantee correctness. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(2):663-678, 2019.

B. Kompa, J. Snoek, and A. L. Beam. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digital Medicine , 4(1):4, 2021.

Ł. Korycki, A. Cano, and B. Krawczyk. Active learning with abstaining classifiers for imbalanced drifting data streams. In 2019 IEEE international conference on big data (big data) , pages 2334-2343. IEEE, 2019.

H.-P. Kriegel, P. Kroger, E. Schubert, and A. Zimek. Interpreting and unifying outlier scores. In Proceedings of the 2011 SIAM International Conference on Data Mining , pages 13-24. SIAM, 2011.

S. Laroui, X. Descombes, A. Vernay, F. Villiers, F. Villalba, and E. Debreuve. How to define a rejection class based on model learning? In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 569-576. IEEE, 2021.

L. J. Latecki, A. Lazarevic, and D. Pokrajac. Outlier detection with kernel density functions. In International Workshop on Machine Learning and Data Mining in Pattern Recognition , pages 61-75. Springer, 2007.

C.-L. Li, K. Sohn, J. Yoon, and T. Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9664-9674, 2021.

S. Li, X. Ji, E. Dobriban, O. Sokolsky, and I. Lee. Pac-wrap: Semi-supervised pac anomaly detection. arXiv preprint arXiv:2205.10798 , 2022.

Z. Li, Y. Zhao, N. Botta, C. Ionescu, and X. Hu. Copod: copula-based outlier detection. In 2020 IEEE International Conference on Data Mining (ICDM) , pages 1118-1123. IEEE, 2020.

Z. Li, Y. Zhao, X. Hu, N. Botta, C. Ionescu, and G. Chen. Ecod: Unsupervised outlier detection using empirical cumulative distribution functions. IEEE Transactions on Knowledge and Data Engineering , 2022.

O. Lindenbaum, Y. Aizenbud, and Y. Kluger. Probabilistic robust autoencoders for outlier detection. arXiv preprint arXiv:2110.00494 , 2021.

F. T. Liu, K. M. Ting, and Z.-H. Zhou. Isolation-based anomaly detection. ACM Transactions on Knowledge Discovery from Data (TKDD) , 6(1):1-39, 2012.

M. Q. Ma, Y. Zhao, X. Zhang, and L. Akoglu. A large-scale study on unsupervised outlier model selection: Do internal strategies suffice? arXiv preprint arXiv:2104.01422 , 2021.

C. Marrocco, M. Molinara, and F. Tortorella. An empirical comparison of ideal and empirical roc-based reject rules. In International Workshop on Machine Learning and Data Mining in Pattern Recognition , pages 47-60. Springer, 2007.

L. Martí, N. Sanchez-Pi, J. M. Molina, and A. C. B. Garcia. Anomaly detection based on sensor data in petroleum industry applications. Sensors , 2015.

V.-L. Nguyen and E. Hullermeier. Reliable multilabel classification: Prediction with partial abstention. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5264-5271, 2020.

G. Pang, L. Cao, L. Chen, and H. Liu. Learning representations of ultrahigh-dimensional data for random distance-based outlier detection. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2041-2050, 2018.

L. Perini, C. Galvin, and V. Vercruyssen. A ranking stability measure for quantifying the robustness of anomaly detection methods. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 397-408. Springer, 2020.

L. Perini, V . Vercruyssen, and J. Davis. Quantifying the confidence of anomaly detectors in their example-wise predictions. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 227-243. Springer, 2020.

L. Perini, V . Vercruyssen, and J. Davis. Transferring the contamination factor between anomaly detection domains by shape similarity. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 4128-4136, 2022.

L. Perini, P.-C. Bürkner, and A. Klami. Estimating the contamination factor's distribution in unsupervised anomaly detection. In International Conference on Machine Learning , pages 27668-27679. PMLR, 2023.

L. Perini, D. Giannuzzi, and J. Davis. How to allocate your label budget? choosing between active learning and learning to reject in anomaly detection. arXiv preprint arXiv:2301.02909 , 2023.

T. Pevn` y. Loda: Lightweight on-line detector of anomalies. Machine Learning , 102(2):275-304, 2016.

A. Pugnana and S. Ruggieri. Auc-based selective classification. In International Conference on Artificial Intelligence and Statistics , pages 2494-2514. PMLR, 2023.

C. Qiu, T. Pfrommer, M. Kloft, S. Mandt, and M. Rudolph. Neural transformation learning for deep anomaly detection beyond images. In International Conference on Machine Learning , pages 8703-8714. PMLR, 2021.

S. Rayana and L. Akoglu. Less is more: Building selective anomaly ensembles. Acm transactions on knowledge discovery from data (tkdd) , 10(4):1-33, 2016.

L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694 , 2019.

B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson. Estimating the support of a high-dimensional distribution. Neural computation , 13(7):1443-1471, 2001.

V. Sehwag, M. Chiang, and P. Mittal. Ssd: A unified framework for self-supervised outlier detection. arXiv preprint arXiv:2103.12051 , 2021.

S. Shekhar, M. Ghavamzadeh, and T. Javidi. Binary classification with bounded abstention rate. arXiv preprint arXiv:1905.09561 , 2019.

T. Shenkar and L. Wolf. Anomaly detection for tabular data with internal contrastive learning. In International Conference on Learning Representations , 2021.

J. Soenen, E. Van Wolputte, L. Perini, V. Vercruyssen, W. Meert, J. Davis, and H. Blockeel. The effect of hyperparameter tuning on the comparative evaluation of unsupervised anomaly detection methods. In Proceedings of the KDD , volume 21, pages 1-9, 2021.

D. Van der Plas, W. Meert, J. Verbraecken, and J. Davis. A reject option for automated sleep stage scoring. In Workshop on Interpretable ML in Healthcare at International Conference on Machine Learning (ICML) , 2021.

L. Xiang, X. Yang, A. Hu, H. Su, and P. Wang. Condition monitoring and anomaly detection of wind turbine based on cascaded and bidirectional deep learning networks. Applied Energy , 305: 117925, 2022.

N. Zhao, X. Wen, and S. Li. A review on gas turbine anomaly detection for implementing health management. Turbo Expo: Power for Land, Sea, and Air , 2016.

Y. Zhao, Z. Nasrullah, and Z. Li. Pyod: A python toolbox for scalable outlier detection. Journal of Machine Learning Research , 20(96):1-7, 2019. URL http://jmlr.org/papers/v20/ 19-011.html .

A. Zimek, R. J. Campello, and J. Sander. Ensembles for unsupervised outlier detection: challenges and research questions a position paper. Acm Sigkdd Explorations Newsletter , 15(1): 11-22, 2014.

Supplement

In this supplementary material we (1) provide additional theorems and proofs for Section 3, and (2) further describe the experimental results.

A Theoretical Results

Firstly, we provide the proof for Theorem 3.1.

Theorem 3.1 (Analysis of EXCEED) Let s be an anomaly score, and ψ n ∈ [0 , 1] the proportion of training scores ≤ s . For T ≥ 4 , there exist t 1 = t 1 ( n, γ, T ) ∈ [0 , 1] , t 2 = t 2 ( n, γ, T ) ∈ [0 , 1] such that

Proof. We split this proof into two parts: we show that the reverse inequalities, i.e. that (a) if ψ n ≤ t 1 , then M s ≥ 1 -2 e -T , and (b) if ψ n ≥ t 2 , then M s ≥ 1 -2 e -T , hold and prove the final statement because P ( ˆ Y = 1 | s ) is monotonic increasing on s .

(a) The probability P ( ˆ Y = 1 | s ) (as in Eq. 1) can be seen as the cumulative distribution F of a binomial random variable B ( q s , n ) with at most nγ -1 successes out of n trials, with q s = n (1 -ψ n )+1 2+ n as the success probability. By applying Hoeffding's inequality, we obtain the upper bound

that holds for the constraint ψ n ≤ 2+ n n 2 + 1 -2 γ n +(1 -γ ) . Because P ( ˆ Y = 1 | s ) ≤ e -T implies that M s ≥ 1 -2 e -T , we search for the values of ψ n such that the upper bound is ≤ e -T . Forcing the upper bound ≤ e -T results in

which is satisfied for ( I 1 ) 0 ≤ ψ n ≤ A 1 - √ B 1 and ( I 2 ) A 1 + √ B 1 ≤ ψ n ≤ 1 , where

However, for T ≥ 4 , no values of n , γ , and T that satisfy the constraint on ψ n also satisfy I 2 . Moving to I 1 , we find out that if ψ n satisfies I 1 , then it also satisfies the constraint on ψ n for any n , γ , and T . Therefore, we we set t 1 ( n, γ, T ) = A 1 - √ B 1 . As a result,

(b) Similarly, P ( ˆ Y = 0 | s ) can be seen as the cumulative distribution F of B ( p s , n ) , with n (1 -γ ) successes and p s = 1+ nψ n ( s ) 2+ n . By seeing the binomial as a sum of Bernoulli random variables, and using the property of its cumulative distribution F ( n (1 -γ ) , n, p s ) + F ( nγ -1 , n, 1 -p s ) = 1 , we apply the Hoeffding's inequality and compare such upper bound to the e -T . We obtain

that holds with the constraint ψ n ≥ (2+ n )(1 -γ ) -1 n . The quadratic inequality in ψ n has solutions for ( I 1 ) 0 ≤ ψ n ≤ A 2 - √ B 2 and ( I 2 ) A 2 + √ B 2 ≤ ψ n ≤ 1 , where A 2 = (2+ n )(1 -γ ) -1 n , and B 2 = T ( n +2) 2 2 n 3 . However, the constraint limits the solutions to I 2 , i.e. for ψ n ≥ A 2 + √ B 2 . Thus, we set t 2 ( n, γ, T ) = A 2 + √ B 2 and conclude that

Secondly, Theorem 3.6 relies on two important results: given S the anomaly score random variable, (1) if ψ n was the theoretical cumulative of S , it would have a uniform distribution (Theorem A.1), but because in practice (2) ψ n is the empirical cumulative of S , its distribution is close to uniform with high probability (Theorem A.2). We prove these results in the following theorems.

Theorem A.1. Let S be the anomaly score random variable, and ψ = F S ( S ) be the cumulative distribution of S applied to S itself. Then ψ ∼ Unif (0 , 1) .

Proof. We prove that, if ψ ∼ Unif (0 , 1) , then F ψ ( t ) = t for any t ∈ [0 , 1] :

Theorem A.2. Let ψ be as in Theorem A.1, and F ψ n be its empirical distribution obtained from a sample of size n . For any small δ > 0 and t ∈ [0 , 1] , with probability > 1 -δ

Proof. For any ε > 0 , the DKW inequality implies

By setting δ = 2exp ( -2 nε 2 ) , i.e. ε = √ ln 2 δ 2 n , and using the complementary probability we conclude that

B Experiments

Data. Table 3 shows the properties of the 34 datasets used for the experimental comparison, in terms of number of examples, features, and contamination factor γ . The datasets can be downloaded in the following link: https://github.com/Minqi824/ADBench/tree/main/ datasets/Classical .

Q1. REJEX against the baselines. Table 4 and Table 5 show the results (mean ± std) aggregated by detectors in terms of, respectively, cost per example and ranking position. Results confirm that REJEX obtains an average cost per example lower than all the baselines for 9 out of 12 detectors, which is similar to the runner-up SS-REPEN for the remaining 3 detectors. Moreover, REJEX has always the best (lowest) average ranking position.

Q2. Varying the costs c fp , c fn , c r . Table 6 and Table 7 show the average cost per example and the ranking position (mean ± std) aggregated by detectors for three representative cost functions, as discussed in the paper. Results are similar in all three cases. For high false positives cost ( c fp = 10 ), REJEX obtains an average cost per example lower than all the baselines for 11 out of 12 detectors and always the best average ranking position. For high false negative cost ( c fn = 10 ) as well as for low rejection cost ( c fp = 5 , c fn = 5 , c r = γ ), it has the lowest average cost for all detectors and always the best average ranking. Moreover, when rejection is highly valuable (low cost), REJEX's cost has a large gap with respect to the baselines, which means that it is particularly useful when rejection is less expensive.

RANKING POSITION

(MEAN

±

STD.)

DET.

REJEX

SS-REPEN

MV

ENS

UDR

EM

STABILITY

NOREJECT

AE

2.63

±

1.63

3.96

±

2.94

4.78

±

2.10

3.81

±

2.11

4.98

±

1.88

5.15

±

1.80

4.92

±

1.78

5.77

±

1.84

COPOD

2.49

±

1.65

3.44

±

2.75

3.97

±

1.92

4.25

±

2.17

5.24

±

1.70

5.13

±

1.67

5.59

±

1.48

5.89

±

2.13

ECOD

2.43

±

1.44

3.62

±

2.86

3.73

±

1.96

4.13

±

2.29

5.53

±

1.57

4.75

±

1.62

5.74

±

1.39

6.07

±

1.93

GMM

2.12

±

1.08

3.05

±

2.49

5.26

±

1.92

3.49

±

2.00

4.43

±

1.66

6.26

±

1.24

5.20

±

1.43

6.20

±

1.45

HBOS

2.29

±

1.57

3.64

±

2.98

4.54

±

2.11

4.39

±

2.06

4.95

±

1.79

5.04

±

1.80

5.61

±

1.40

5.52

±

1.88

IFOR

2.23

±

1.48

3.78

±

2.78

4.12

±

1.90

4.26

±

2.08

5.10

±

1.88

5.27

±

1.66

5.34

±

1.38

5.91

±

2.22

INNE

1.73

±

1.14

3.18

±

2.74

5.86

±

2.42

5.57

±

1.40

4.94

±

1.62

5.57

±

1.60

5.32

±

1.37

3.83

±

1.63

KDE

2.33

±

1.42

3.99

±

2.86

4.74

±

2.06

3.79

±

2.03

5.01

±

1.90

5.43

±

1.59

4.87

±

1.92

5.84

±

1.80

KNN

2.02

±

1.29

3.58

±

2.87

4.87

±

1.81

3.94

±

1.94

4.41

±

1.83

5.75

±

1.49

5.22

±

1.62

6.21

±

1.48

LODA

2.89

±

1.77

3.17

±

2.30

4.15

±

2.26

4.36

±

2.14

4.99

±

2.00

5.50

±

2.04

4.98

±

2.11

5.95

±

1.73

LOF

2.04

±

1.01

3.16

±

2.73

5.68

±

1.40

3.32

±

1.71

3.96

±

1.63

6.15

±

1.19

5.47

±

1.49

6.22

±

1.31

OCSVM

2.33

±

1.29

3.92

±

2.84

4.89

±

1.98

3.85

±

2.17

4.86

±

1.89

5.31

±

1.80

5.06

±

1.89

5.78

±

1.66

AVG.

2.29

±

1.40

3.54

±

2.76

4.72

±

1.99

4.10

±

2.01

4.87

±

1.78

5.44

±

1.63

5.28

±

1.60

5.77

±

1.76

COST PER EXAMPLE FOR THREE COST FUNCTIONS

(MEAN

±

STD)

DET.

REJEX

SS-REPEN

MV

ENS

UDR

EM

STABILITY

NOREJECT

FALSE POSITIVE COST

= 10

, FALSE NEGATIVE COST

= 1

, REJECTION COST

= min

{

10(1

-

γ

)

, γ

}

AE

0.504

±

0.626

0.584

±

0.723

0.697

±

0.763

0.661

±

0.830

0.703

±

0.829

0.766

±

0.841

0.768

±

0.826

0.825

±

0.873

COPOD

0.491

±

0.637

0.593

±

0.706

0.686

±

0.746

0.618

±

0.726

0.707

±

0.788

0.778

±

0.825

0.785

±

0.801

0.781

±

0.833

ECOD

0.479

±

0.628

0.584

±

0.727

0.625

±

0.705

0.642

±

0.755

0.711

±

0.774

0.748

±

0.803

0.770

±

0.783

0.771

±

0.817

GMM

0.568

±

0.713

0.589

±

0.752

0.823

±

0.878

0.715

±

0.929

0.790

±

0.925

0.941

±

0.948

0.889

±

0.929

0.950

±

0.967

HBOS

0.475

±

0.595

0.569

±

0.758

0.666

±

0.693

0.697

±

0.732

0.709

±

0.764

0.776

±

0.803

0.771

±

0.770

0.809

±

0.816

IFOR

0.477

±

0.602

0.575

±

0.712

0.665

±

0.718

0.634

±

0.731

0.683

±

0.786

0.776

±

0.818

0.763

±

0.788

0.808

±

0.831

INNE

0.479

±

0.592

0.567

±

0.698

0.752

±

0.724

0.820

±

0.795

0.815

±

0.787

0.819

±

0.793

0.818

±

0.792

0.823

±

0.799

KDE

0.602

±

0.827

0.589

±

0.704

0.819

±

0.947

0.740

±

0.913

0.793

±

0.939

0.897

±

0.945

0.774

±

0.906

0.914

±

0.945

KNN

0.498

±

0.577

0.596

±

0.726

0.741

±

0.734

0.669

±

0.720

0.669

±

0.736

0.777

±

0.747

0.739

±

0.735

0.800

±

0.749

LODA

0.518

±

0.619

0.574

±

0.709

0.574

±

0.647

0.689

±

0.729

0.701

±

0.748

0.762

±

0.774

0.697

±

0.682

0.827

±

0.797

LOF

0.539

±

0.623

0.603

±

0.742

0.898

±

0.840

0.685

±

0.773

0.715

±

0.790

0.891

±

0.813

0.831

±

0.821

0.887

±

0.808

OCSVM

0.479

±

0.599

0.589

±

0.705

0.745

±

0.790

0.632

±

0.752

0.694

±

0.782

0.760

±

0.775

0.695

±

0.737

0.818

±

0.806

FALSE POSITIVE COST

= 1

, FALSE NEGATIVE COST

= 10

, REJECTION COST

= min

{

1

-

γ,

10

γ

}

AE

0.730

±

0.747

0.761

±

0.756

0.909

±

0.882

0.784

±

0.825

0.780

±

0.805

0.819

±

0.843

0.789

±

0.825

0.797

±

0.821

COPOD

0.761

±

0.767

0.765

±

0.770

0.930

±

0.888

0.794

±

0.805

0.800

±

0.801

0.844

±

0.842

0.802

±

0.815

0.827

±

0.832

ECOD

0.739

±

0.759

0.767

±

0.766

0.900

±

0.858

0.789

±

0.811

0.788

±

0.787

0.840

±

0.839

0.791

±

0.803

0.821

±

0.819

GMM

0.670

±

0.676

0.765

±

0.767

0.845

±

0.782

0.754

±

0.755

0.739

±

0.736

0.785

±

0.757

0.760

±

0.753

0.766

±

0.750

HBOS

0.687

±

0.684

0.776

±

0.782

0.824

±

0.808

0.750

±

0.768

0.744

±

0.747

0.785

±

0.787

0.749

±

0.765

0.753

±

0.766

IFOR

0.679

±

0.680

0.775

±

0.776

0.847

±

0.824

0.755

±

0.771

0.743

±

0.745

0.761

±

0.772

0.757

±

0.774

0.763

±

0.770

INNE

0.660

±

0.685

0.772

±

0.779

0.695

±

0.620

0.774

±

0.742

0.748

±

0.722

0.758

±

0.737

0.744

±

0.716

0.773

±

0.754

KDE

0.691

±

0.692

0.791

±

0.773

0.887

±

0.836

0.754

±

0.760

0.755

±

0.744

0.785

±

0.807

0.758

±

0.754

0.759

±

0.760

KNN

0.706

±

0.657

0.767

±

0.764

0.839

±

0.779

0.791

±

0.736

0.769

±

0.710

0.778

±

0.736

0.799

±

0.736

0.803

±

0.729

LODA

0.750

±

0.714

0.781

±

0.775

0.880

±

0.850

0.811

±

0.768

0.806

±

0.761

0.804

±

0.783

0.827

±

0.780

0.838

±

0.784

LOF

0.738

±

0.679

0.764

±

0.764

0.999

±

0.833

0.826

±

0.757

0.810

±

0.739

0.871

±

0.770

0.867

±

0.799

0.846

±

0.747

OCSVM

0.730

±

0.711

0.780

±

0.774

0.953

±

0.878

0.791

±

0.786

0.795

±

0.773

0.845

±

0.833

0.787

±

0.772

0.796

±

0.783

FALSE POSITIVE COST

= 5

, FALSE NEGATIVE COST

= 5

, REJECTION COST

=

γ

AE

0.534

±

0.611

0.618

±

0.666

0.671

±

0.716

0.644

±

0.741

0.655

±

0.736

0.707

±

0.748

0.705

±

0.740

0.738

±

0.762

COPOD

0.545

±

0.619

0.627

±

0.673

0.676

±

0.724

0.629

±

0.674

0.666

±

0.716

0.719

±

0.747

0.719

±

0.730

0.731

±

0.739

ECOD

0.529

±

0.609

0.625

±

0.675

0.629

±

0.687

0.638

±

0.702

0.662

±

0.705

0.701

±

0.736

0.708

±

0.716

0.724

±

0.727

GMM

0.534

±

0.599

0.626

±

0.687

0.719

±

0.709

0.656

±

0.716

0.675

±

0.720

0.776

±

0.736

0.746

±

0.731

0.780

±

0.743

HBOS

0.499

±

0.572

0.622

±

0.694

0.632

±

0.661

0.650

±

0.669

0.641

±

0.681

0.695

±

0.706

0.688

±

0.686

0.710

±

0.709

IFOR

0.497

±

0.569

0.623

±

0.677

0.643

±

0.680

0.620

±

0.667

0.629

±

0.692

0.696

±

0.712

0.688

±

0.701

0.714

±

0.719

INNE

0.491

±

0.569

0.617

±

0.668

0.622

±

0.572

0.718

±

0.691

0.691

±

0.674

0.709

±

0.685

0.705

±

0.675

0.726

±

0.698

KDE

0.562

±

0.639

0.642

±

0.673

0.709

±

0.739

0.666

±

0.711

0.684

±

0.726

0.748

±

0.742

0.679

±

0.689

0.761

±

0.742

KNN

0.521

±

0.544

0.628

±

0.677

0.687

±

0.667

0.657

±

0.646

0.634

±

0.651

0.701

±

0.664

0.693

±

0.657

0.728

±

0.664

LODA

0.550

±

0.595

0.627

±

0.677

0.601

±

0.649

0.670

±

0.668

0.665

±

0.680

0.708

±

0.698

0.683

±

0.645

0.757

±

0.711

LOF

0.554

±

0.580

0.628

±

0.681

0.809

±

0.737

0.678

±

0.682

0.674

±

0.688

0.792

±

0.703

0.759

±

0.718

0.788

±

0.698

OCSVM

0.523

±

0.582

0.631

±

0.671

0.704

±

0.728

0.634

±

0.685

0.657

±

0.695

0.709

±

0.704

0.660

±

0.674

0.733

±

0.716

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the topic of rejection of samples in an unsupervised anomaly detection setup. Their approach focuses on determining a constant rejection threshold, which allows the detector to reject examples with high uncertainty. The new proposed method introduces this rejection threshold based on a confidence score given by another existing model (ExCeed). The authors provide theoretical analyses as well as empirical experiments and show that it is possible to set a constant rejection threshold with strong theoretical guarantees.\n\nSTRENGTHS:\n- The paper provides a good structure but also requires some pre-knowledge in this area to be able to follow the presented thoughts e. g. research questions are later connected according to sections. \n- In-depth theoretical methodology as well as empirical evaluation.\n- Detailed overview of single results in the supplement is given.\n\nWEAKNESSES:\n- It was not always straightforward to follow the paper, especially because a lot of variables are introduced but are defined much later in the paper (e.g. t_1(n, \\gamma, T) (line 131) and  \\gamma = \\mathds{P}(Y = 1) (line 199)). Starting with a more explanatory part would let the reader build an intuition about which factors are important to calculate the rejection threshold. With all the factors in mind, it will get easier to follow the complex theoretical contribution. \n- in line 135 \\epsilon is defined as 1 - 2e^(-T). In the formula between lines 123 and 124, the rejection threshold is defined as \\mathcal{T} = 1 - \\epsilon = 1 - 2e^(-T), which would result in \\epsilon = 2e^(-T).\n\nMinor comments: \n- \"Our approach is called **RejEx** (Rejecting via ExCeed)(line 110)\n- In Theorem 3.8 it is refed to Theorem 3.5 for the definition of g. Theorem 3.5, however, refers to Theorem 3.4; It would be easier to directly refer to Theorem 3.4\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nDifficult write-up.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a rejection scheme for the task of unsupervised anomaly detection.  Learning to reject enables a predictor to withhold from making a prediction; this paradigm is more common in unsupervised learning. Here, the authors extend the rejection idea to the unsupervised anomaly detection task.  The idea is to reject samples based on a stability metric; namely, of the prediction is unstable to small changes in the feature space, the prediction is rejected. This type of stability metric was recently proposed and is termed EXCEED. The authors present a theoretical analysis of the EXCEED metric and derive upper bounds for the test rejection rate and expected prediction cost. The new scheme is evaluated for several anomaly detectors on real datasets and outperforms other rejection schemes.\n\nSTRENGTHS:\nThe paper is well-written, and easy to follow.  Overall, the presentation is scientifically sound. The problem of unsupervised anomaly detection is extremely challenging and important; the paper presents a rejection scheme that could improve trust in commonly used detectors. The idea of using stability and specifically the EXCEED metric, makes sense. The theoretical analysis strengthens the work and offers bounds on the expected values of the presented scheme. The empirical evidence presented in the paper is promising and demonstrates the merits of the method.\n\nWEAKNESSES:\nBackground on the EXCEED method is missing; adding more information on this metric could help the reader. Some recently proposed NN anomaly detectors are missing from the evaluation, for example:\n[1] Qiu, Chen, et al. \"Neural transformation learning for deep anomaly detection beyond images.\" International Conference on Machine Learning. PMLR, 2021.\n[2] Shenkar, T., & Wolf, L. (2021, October). Anomaly detection for tabular data with internal contrastive learning. In International Conference on Learning Representations.\n[3] Lindenbaum, et al. (2021). Probabilistic robust autoencoders for outlier detection. arXiv preprint arXiv:2110.00494.\nThere are many other NN that could be included, I think some NN baselines should be considered. \nThe description of the experiments conducted is too brief; it would be good if the authors could expand on the implementation and evaluation protocol. For example, what is $\\lambda$ in all experiments? Or how is it tuned?\n\nSample size is limited in the evaluation to 20K.\n\nQUESTIONS:\nThe analysis assumes that $\\gamma$ is known, in practice how can you gain access to this value? Is there a way to overcome this limitation?\nPlease expand on the ECEED metric, equation 1.\nA comma is missing after this equation.\nHow is the cost influenced when changing \\lambda? It is not clear from the text what do you do with this value.\nWhy are you not mentioning the subsampling in the main text? Is it so computationally demanding to evaluate the method on large datasets?\n\nLIMITATIONS:\nLimitations are discussed in the last section, I would however add information on the cases in which rejection increases the cost, for example using statistic across datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper suggests applying the stability metric computed by EXCEED for anomaly detection. The authors present theoretical findings regarding this metric, including the test rejection rate, as well as upper bounds for both the rejection rate and the expected prediction cost. Furthermore, comprehensive experiments are conducted to validate the effectiveness of the proposed method.

STRENGTHS:
1. The presentation of the paper is clear, and the proposed method is simple but effective.
2. This paper offers a theoretical analysis of EXCEED, deriving the upper bounds for both the rejection rate and the expected prediction cost.
3. The effectiveness of the proposed method and the validity of the theoretical results are confirmed through comprehensive experiments.

WEAKNESSES:
1. The methods compared in Figure 1 appear to be significantly dated. It would be valuable if the paper could include additional results pertaining to recently proposed methods.

QUESTIONS:
Please refer to [Weakness].

LIMITATIONS:
Please refer to [Weakness].
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

comments: SUMMARY:
- The authors proposed a selective predictor (learning to reject) for fully unsupervised setting in anomaly detection problems given an unsupervised anomaly detector.
- The proposed method is based on the theoretical supports and the threshold can be selected without any labeled data.
- The experimental results show that the proposed method can significantly reduce the cost of selective prediction.

STRENGTHS:
- The proposed method is grounded on the theoretical supports that could be beneficial on generalization.
- The experimental sections are extensive and multiple ablation studies show its superiority across various settings.

WEAKNESSES:
- It seems like the proposed method is only applicable when the anomaly ratio is given. In some cases, anomaly ratio itself is not provided.
- It would be great if the authors can provide more extensive experiments when we have some labeled data in comparison to the baselines.

QUESTIONS:
1. EXCEED metric
- I can understand the brief concepts of the EXCEED metric.
- However, it would be good to further explain how the training data is perturbed. 
- Also, it would be good to add additional explanations of Equation (1) - like the motivations of this equation.

2. Problem settings
- So, here, do we assume that we have an access to the contamination ratio (gamma)?
- If yes, can we extend this method without access to the contamination ratio?

3. With some labels
- As discussed in Related works, if we have some labels, we can easily optimize the rejection function using two ways that the authors explained.
- In that case, can we analyze how many samples do we need to have similar performance with the proposed unsupervised method?

LIMITATIONS:
Limitations are clearly stated.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach to perform learning to reject for anomaly detection in a completely unsupervised manner. The authors make three major contributions: (1) a novel theoretical analysis of a stability metric for anomaly detection, (2) a mechanism for designing an ambiguity rejection mechanism without any labeled data that offers strong guarantees, and (3) an evaluation of the proposed approach on an extensive set of unsupervised detectors and benchmark datasets. The authors show that their method outperforms several adapted baselines based on other unsupervised metrics and that their theoretical results hold in practice.\n\nSTRENGTHS:\nOriginality: The paper proposes a novel approach to perform ambiguity rejection for anomaly detection in a completely unsupervised manner. The authors provide a novel theoretical analysis of a stability metric for anomaly detection and show that it has several previously unknown properties that are of great importance in the context of learning to reject.\n\nQuality: The paper provides a thorough theoretical analysis of the proposed approach and demonstrates its effectiveness through experiments on an extensive set of unsupervised detectors and benchmark datasets. The authors also provide strong guarantees for their proposed method.\n\nClarity: The paper is well-written and easy to follow. The authors provide clear explanations of the proposed approach and the theoretical analysis.\n\nSignificance: The proposed approach addresses the challenge of uncertainty in traditional anomaly detectors and provides a solution through Learning to Reject. The authors show that their method outperforms several adapted baselines based on other unsupervised metrics and that their theoretical results hold in practice. The proposed approach has significant implications for anomaly detection in various domains.\n\nWEAKNESSES:\nThe author may provide more intuition on how EXCEED works to estimate the stability.\nThe paper could benefit from a more detailed discussion of the limitations of the proposed approach and potential directions for future research. \nWhile the paper provides a thorough theoretical analysis of the proposed approach, it could benefit from more detailed explanations of the experimental setup and results. Specifically, the paper could provide more information on the hyperparameters used in the experiments and how they were selected, as well as more detailed comparisons with other state-of-the-art methods.\n\nQUESTIONS:\n1. Is there any other method to estimate the detector's stability, in addition to EXCEED? If so, how does EXCEED outperform other methods?\n2. In Definition 3.7, the cost function is defined as a simple addition. What if the cost function is of a more complex form?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}