SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lorenzo Perini
DTAI lab & Leuven.AI,
KU Leuven, Belgium
lorenzo.perini@kuleuven.be
Jesse Davis,
jesse.davis@kuleuven.be
Anomaly detection goal is to detect unexpected behaviours in the data. Because anomaly detection is usually an unsupervised task, traditional anomaly detectors learn a decision boundary by employing heuristics based on intuitions, which are hard to verify in practice. This introduces some uncertainty, especially close to the decision boundary, which may reduce the user trust in the detector's predictions. A way to combat this is by allowing the detector to reject examples with high uncertainty (Learning to Reject). This requires employing a confidence metric that captures the distance to the decision boundary and setting a rejection threshold to reject low-confidence predictions. However, selecting a proper metric and setting the rejection threshold without labels are challenging tasks. In this paper, we solve these challenges by setting a constant rejection threshold on the stability metric computed by EXCEED. Our insight relies on a theoretical analysis of this metric. Moreover, setting a constant threshold results in strong guarantees: we estimate the test rejection rate, and derive a theoretical upper bound for both the rejection rate and the expected prediction cost. Experimentally, we show that our method outperforms some metric-based methods.
Anomaly detection is the task of detecting unexpected behaviors in the data [6]. Often, these anomalies are critical adverse events such as the destruction or alteration of proprietary user data [29], water leaks in stores [50], breakdowns in gas [65] and wind [64] turbines, or failures in petroleum extraction [45]. Usually, anomalies are associated with a cost such as a monetary cost (e.g., maintenance, paying for fraudulent purchases) or a societal cost such as environmental damages (e.g., dispersion of petroleum or gas). Hence, detecting anomalies in a timely manner is an important problem.
When using an anomaly detector for decision-making, it is crucial that the user trusts the system. However, it is often hard or impossible to acquire labels for anomalies. Moreover, anomalies may not follow a pattern. Therefore anomaly detection is typically treated as an unsupervised learning problem where traditional algorithms learn a decision boundary by employing heuristics based on intuitions [22, 55, 61, 41], such as that anomalies are far away from normal examples [3]. Because these intuitions are hard to verify and may not hold in some cases, some predictions may have high uncertainty, especially for examples close to the decision boundary [32, 28]. As a result, the detector's predictions should be treated with some circumspection.
One way to increase user trust is to consider Learning to Reject [12]. In this setting, the model does not always make a prediction. Instead, it can abstain when it is at a heightened risk of making a mistake thereby improving its performance when it does offer a prediction. Abstention has the drawback that no prediction is made, which means that a person must intervene to make a decision. In the literature, two types of rejection have been identified [25]: novelty rejection allows the model to abstain when given an out-of-distribution (OOD) example, while ambiguity rejection enables abstention for a test example that is too close to the model's decision boundary. Because anomalies
often are OOD examples, novelty rejection does not align well with our setting as the model would reject all OOD anomalies (i.e., a full class) [10, 63, 30]. On the other hand, current approaches for ambiguity rejection threshold what constitutes being too close to the decision boundary by evaluating the model's predictive performance on the examples for which it makes a prediction (i.e., accepted), and those where it abstains from making a prediction (i.e., rejected) [9, 44, 16]. Intuitively, the idea is to find a threshold where the model's predictive performance is (1) significantly lower on rejected examples than on accepted examples and (2) higher on accepted examples than on all examples (i.e., if it always makes a prediction). Unfortunately, existing learning to reject approaches that set a threshold in this manner require labeled data, which is not available in anomaly detection.
This paper fills this gap by proposing an approach to perform ambiguity rejection for anomaly detection in a completely unsupervised manner. Specifically, we make three major contributions. First, we conduct a thorough novel theoretical analysis of a stability metric for anomaly detection [49] and show that it has several previously unknown properties that are of great importance in the context of learning to reject. Namely, it captures the uncertainty close to the detector's decision boundary, and only limited number of examples get a stability value strictly lower than 1. Second, these enables us to design an ambiguity rejection mechanism without any labeled data that offers strong guarantees which are often sought in Learning to Reject [12, 60, 7] We can derive an accurate estimate of the rejected examples proportion, as well as a theoretical upper bound that is satisfied with high probability. Moreover, given a cost function for different types of errors, we provide an estimated upper bound on the expected cost at the prediction time. Third, we evaluate our approach on an extensive set of unsupervised detectors and benchmark datasets and conclude that (1) it performs better than several adapted baselines based on other unsupervised metrics, and (2) our theoretical results hold in practice.
2 Preliminaries and notation
We will introduce the relevant background on anomaly detection, learning to reject, and the EXCEED's metric that this paper builds upon.
Anomaly Detection. Let  $\mathcal{X}$  be a  $d$  dimensional input space and  $D = \{x_{1},\ldots ,x_{n}\}$  be a training set, where each  $x_{i}\in \mathcal{X}$ . The goal in anomaly detection is to train a detector  $f\colon \mathcal{X}\to \mathbb{R}$  that maps examples to a real-valued anomaly score, denoted by  $s$ . In practice, it is necessary to convert these soft scores to a hard prediction, which requires setting a threshold  $\lambda$ . Assuming that higher scores equate to being more anomalous, a predicted label  $\hat{y}$  can be made for an example  $x$  as follows:  $\hat{y} = 1$  (anomaly) if  $s = f(x)\geq \lambda$ , while  $\hat{y} = 0$  (normal) if  $s = f(x) < \lambda$ . We let  $\hat{Y}$  be the random variable that denotes the predicted label. Because of the absence of labels, one usually sets the threshold such that  $\gamma \times n$  scores are  $\geq \lambda$ , where  $\gamma$  is the dataset's contamination factor (i.e., expected proportion of anomalies) [51, 50].
Learning to Reject. Learning to reject extends the output space of the model to include the symbol  $\mathbb{R}$ , which means that the model abstains from making a prediction. This entails learning a second model  $r$  (the rejector) to determine when the model abstains. A canonical example of ambiguity rejection is when  $r$  consists of a pair [confidence  $\mathcal{M}_s$ , rejection threshold  $\tau$ ] such that an example is rejected if the detector's confidence is lower than the threshold. The model output becomes
$$
\hat {y} _ {\mathbb {R}} = \left\{ \begin{array}{l l} \hat {y} & \text {i f} \mathcal {M} _ {s} > \tau ; \\ \mathbb {R} & \text {i f} \mathcal {M} _ {s} \leq \tau ; \end{array} \right. \quad \hat {y} _ {\mathbb {R}} \in \{0, 1, \mathbb {R} \}.
$$
A standard approach is to evaluate different values for  $\tau$  to find a balance between making too many incorrect predictions because  $\tau$  is too low (i.e.,  $\hat{y} \neq y$  but  $\mathcal{M}_s > \tau$ ) and rejecting correct predictions because  $\tau$  is too high (i.e.,  $\hat{y} = y$  but  $\mathcal{M}_s \leq \tau$ ) [9, 44, 16]. Unfortunately, in an unsupervised setting, it is impossible to evaluate the threshold because it relies on having access to labeled data.
EXCEED's metric. Traditional confidence metrics (such as calibrated class probabilities) quantify how likely a prediction is to be correct. This obviously requires labels [9] which are unavailable in an unsupervised setting. Thus, one option is to move the focus towards the concept of stability: given a fixed test example  $x$  with anomaly score  $s$ , perturbing the training data alters the model learning, which, in turn, affects the label prediction. Intuitively, the more stable a detector's output is for a test example, the less sensitive its predicted label is to changes in the training data. On the other hand, when  $\mathbb{P}(\hat{Y} = 1|s) \approx \mathbb{P}(\hat{Y} = 0|s) \approx 0.5$  the prediction for  $x$  is highly unstable, as training the
detector with slightly different examples would flip its prediction for the same test score  $s$ . Thus, a stability-based confidence metric  $\mathcal{M}_s$  can be expressed as the margin between the two classes' probabilities:
$$
\mathcal {M} _ {s} = | \mathbb {P} (\hat {Y} = 1 | s) - \mathbb {P} (\hat {Y} = 0 | s) | = | 2 \mathbb {P} (\hat {Y} = 1 | s) - 1 |,
$$
where the lower  $\mathcal{M}_s$  the more unstable the prediction.
Recently, Perini et al. introduced EXCEED to estimate the detector's stability  $\mathbb{P}(\hat{Y} = 1|s)$ . Roughly speaking, EXCEED uses a Bayesian formulation that simulates bootstrapping the training set as a form of perturbation. Formally, it measures such stability for a test score  $s$  in two steps.
First, it computes the training frequency  $\psi_{n} = \frac{|\{i\leq n:s_{i}\leq s\}|}{n}\in [0,1]$ , i.e. the proportion of training scores lower than  $s$ . This expresses how extreme the score  $s$  ranks with respect to the training scores. Second, it computes the probability that the score  $s$  will be predicted as an anomaly when randomly drawing a training set of  $n$  scores from the population of scores. In practice, this is the probability that the chosen threshold  $\lambda$  will be less than or equal to the score  $s$ . The stability is therefore estimated as
$$
\mathbb {P} (\hat {Y} = 1 | s) = \sum_ {i = n (1 - \gamma) + 1} ^ {n} \binom {n} {i} \left(\frac {1 + n \psi_ {n}}{2 + n}\right) ^ {i} \left(\frac {n (1 - \psi_ {n}) + 1}{2 + n}\right) ^ {n - i}. \tag {1}
$$
Assumption. EXCEED's Bayesian formulation requires assuming that  $Y|s$  follows a Bernoulli distribution with parameter  $p_s = \mathbb{P}(S \leq s)$ , where  $S$  is the detector's population of scores. Note that the stability metric is a detector property and, therefore, is tied to the specific choice of the unsupervised detector  $f$ .
This paper addresses the following problem:
Given: An unlabeled dataset  $D$  with contamination  $\gamma$ , an unsupervised detector  $f$ , a cost function  $c$ ;
Do: Introduce a reject option to  $f$ , i.e. find a pair (confidence, threshold) that minimizes the cost.
We propose an anomaly detector-agnostic approach for performing learning to reject that requires no labels. Our key contribution is a novel theoretical analysis of the EXCEED confidence metric that proves that only a limited number of examples have confidence lower than  $1 - \varepsilon$  (Sec. 3.1). Intuitively, the detector's predictions for most examples would not be affected by slight perturbations of the training set: it is easy to identify the majority of normal examples and anomalies because they will strongly adhere to the data-driven heuristics that unsupervised anomaly detectors use. For example, using the data density as a measure of anomalousness [5] tends to identify all densely clustered normals and isolated anomalies, which constitute the majority of all examples. In contrast, only relatively few cases would be ambiguous and hence receive low confidence (e.g., small clusters of anomalies and normals at the edges of dense clusters).
Our approach is called REJEX (Rejecting via ExCeeD) and simply computes the stability-based confidence metric  $\mathcal{M}_s$  and rejects any example with confidence that falls below threshold  $\tau = 1 - \varepsilon$ . Theoretically, this constant reject threshold provides several relevant guarantees. First, one often needs to control the proportion of rejections (namely, the rejection rate) to estimate the number of decisions left to the user. Thus, we propose an estimator that only uses training instances to estimate the rejection rate at test time. Second, because in some applications avoiding the risk of rejecting all the examples is a strict constraint, we provided an upper bound for the rejection rate (Sec. 3.2). Finally, we compute a theoretical upper bound for a given cost function that guarantees that using REJEX keeps the expected cost per example at test time low (Sec. 3.3).
3.1 Setting the Rejection Threshold through a Novel Theoretical Analysis of EXCEED
Our novel theoretical analysis proves (1) that the stability metric by EXCEED is lower than  $1 - \varepsilon$  for a limited number of examples (Theorem 3.1), and (2) that such examples with low confidence are the ones close to the decision boundary (Corollay 3.2). Thus, we propose to reject all these uncertain examples by setting a rejection threshold
$$
\tau = 1 - \varepsilon = 1 - 2 e ^ {- T} \quad \text {f o r} T \geq 4,
$$
where  $2e^{-T}$  is the tolerance that excludes unlikely scenarios, and  $T \geq 4$  is required for Theorem 3.1.
We motivate our approach as follows. Given an example  $x$  with score  $s$  and the proportion of lower training scores  $\psi_{n}$ , Theorem 3.1 shows that the confidence  $\mathcal{M}_s$  is lower than  $1 - 2e^{-T}$  (for  $T \geq 4$ ) if  $\psi_{n}$  belongs to the interval  $[t_1, t_2]$ . By analyzing  $[t_1, t_2]$ , Corollary 3.2 proves that the closer an example is to the decision boundary, the lower the confidence  $\mathcal{M}_s$ , and that a score  $s = \lambda$  (decision threshold) has confidence  $\mathcal{M}_s = 0$ .
Remark. Perini et al. performed an asymptotic analysis of EXCEED that investigates the metric's behavior when the training set's size  $n \to +\infty$ . In contrast, our novel analysis is finite-sample and hence provides more practical insights, as real-world scenarios involve having a finite dataset with size  $n \in \mathbb{N}$ .
Theorem 3.1 (Analysis of EXCEED). Let  $s$  be an anomaly score, and  $\psi_n \in [0,1]$  its training frequency. For  $T \geq 4$ , there exist  $t_1 = t_1(n,\gamma,T) \in [0,1]$ ,  $t_2 = t_2(n,\gamma,T) \in [0,1]$  such that
$$
\psi_ {n} \in [ t _ {1}, t _ {2} ] \Rightarrow \mathcal {M} _ {s} \leq 1 - 2 e ^ {- T}.
$$
Proof. See the Supplement for the formal proof.
The interval  $[t_1, t_2]$  has two relevant properties. First, it becomes narrower when increasing  $n$  (P1) and larger when increasing  $T$  (P2). This means that collecting more training data results in smaller rejection regions while decreasing the tolerance  $\varepsilon = 2e^{-T}$  has the opposite effect. Second, it is centered (not symmetrically) on  $1 - \gamma$  (P3-P4), which means that examples with anomaly scores close to the decision threshold  $\lambda$  are the ones with a low confidence score (P5). The next Corollary lists these properties.
Corollary 3.2. Given  $t_1, t_2$  as in Theorem 3.1, the following properties hold for any  $s, n, \gamma, T \geq 4$ :
P1.  $\lim_{n\to +\infty}t_1 = \lim_{n\to +\infty}t_2 = 1 - \gamma$
P2.  $t_1$  and  $t_2$  are, respectively, monotonic decreasing and increasing as functions of  $T$ ;
P3. the interval always contains  $1 - \gamma$ , i.e.  $t_1 \leq 1 - \gamma \leq t_2$ ;
P4. for  $n\to \infty$  , there exists  $s^*$  with  $\psi_{n} = t^{*}\in [t_{1},t_{2}]$  such that  $t^{\ast}\rightarrow 1 - \gamma$  and  $\mathcal{M}_s\to 0$
P5.  $\psi_{n}\in [t_{1},t_{2}]$  iff  $s\in [\lambda -u_1,\lambda +u_2]$  , where  $u_{1}(n,\gamma ,T),u_{2}(n,\gamma ,T)$  are positive functions.
Proof sketch. For P1, it is enough to observe that  $t_1, t_2 \to 1 - \gamma$  for  $n \to +\infty$ . For P2 and P3, the result comes from simple algebraic steps. P4 follows from the surjectivity of  $\mathcal{M}_s$  when  $n \to +\infty$ , the monotonicity of  $\mathbb{P}(\hat{Y} = 1|s)$ , from P1 with the squeeze theorem. Finally, P5 follows from  $\psi_n \in [t_1, t_2] \implies s \in [\psi_n^{-1}(t_1), \psi_n^{-1}(t_2)]$ , as  $\psi_n$  is monotonic increasing, where  $\psi_n^{-1}$  is the inverse-image of  $\psi_n$ . Because for P3  $1 - \gamma \in [t_1, t_2]$ , it holds that  $\psi_n^{-1}(t_1) \leq \psi_n^{-1}(1 - \gamma) = \lambda \leq \psi_n^{-1}(t_2)$ . This implies that  $s \in [\lambda - u_1, \lambda + u_2]$ , where  $u_1 = \lambda - \psi_n^{-1}(t_1)$ ,  $u_2 = \lambda - \psi_n^{-1}(t_2)$ .
3.2 Estimating and Bounding the Rejection Rate
It is important to have an estimate of the rejection rate, which is the proportion of examples for which the model will abstain from making a prediction. This is an important performance characteristic for differentiating among candidate models. Moreover, it is important that not all examples are rejected because such a model is useless in practice. We propose a way to estimate the rejection rate and Theorem 3.5 shows that our estimate approaches the true rate for large training sets. We strengthen our analysis and introduce an upper bound for the rejection rate, which guarantees that, with arbitrarily high probability, the rejection rate is kept lower than a constant (Theorem 3.6).
Definition 3.3 (Rejection rate). Given the confidence metric  $\mathcal{M}_s$  and the rejection threshold  $\tau$ , the rejection rate  $\mathcal{R} = \mathbb{P}(\mathcal{M}_s \leq \tau)$  is the probability that a test example with score  $s$  gets rejected.
We propose the following estimator for the reject rate:
Definition 3.4 (Rejection rate estimator). Given anomaly scores  $s$  with training frequencies  $\psi_{n}$ , let  $g\colon [0,1]\to [0,1]$  be the function such that  $\mathbb{P}(\hat{Y} = 1|s) = g(\psi_n)$  (see Eq. 1). We define the rejection rate estimator  $\hat{\mathcal{R}}$  as
$$
\hat {\mathcal {R}} = \hat {F} _ {\psi_ {n}} \left(g ^ {- 1} \left(1 - e ^ {- T}\right)\right) - \hat {F} _ {\psi_ {n}} \left(g ^ {- 1} \left(e ^ {- T}\right)\right) \tag {2}
$$
where  $g^{-1}$  is the inverse-image through  $g$ , and, for  $u \in [0,1]$ ,  $\hat{F}_{\psi_n}(u) = \frac{|i \leq n : \psi_n(s_i) \leq u|}{n}$  is the empirical cumulative distribution of  $\psi_n$ .
Note that  $\hat{\mathcal{R}}$  can be computed in practice, as the  $\psi_{n}$  has a distribution that is arbitrarily close to uniform, as stated by Theorem A.1 and A.2 in the Supplement.
Theorem 3.5 (Rejection rate estimate). Let  $g$  be as in Def. 3.4. Then, for high values of  $n$ ,  $\hat{\mathcal{R}} \approx \mathcal{R}$ .
Proof. From the definition of rejection rate 3.3, it follows
$$
\begin{array}{l} \mathcal {R} = \mathbb {P} \left(\mathcal {M} _ {s} \leq 1 - 2 e ^ {- T}\right) = \mathbb {P} \left(\mathbb {P} (\hat {Y} = 1 | s) \in [ e ^ {- T}, 1 - e ^ {- T} ]\right) = \mathbb {P} \left(g \left(\psi_ {n}\right) \in [ e ^ {- T}, 1 - e ^ {- T} ]\right) \\ = \mathbb {P} \left(\psi_ {n} \in \left[ g ^ {- 1} \left(e ^ {- T}\right), g ^ {- 1} \left(1 - e ^ {- T}\right) \right]\right) = F _ {\psi_ {n}} \left(g ^ {- 1} \left(1 - e ^ {- T}\right)\right) - F _ {\psi_ {n}} \left(g ^ {- 1} \left(e ^ {- T}\right)\right). \\ \end{array}
$$
where  $F_{\psi_n}(\cdot) = \mathbb{P}(\psi_n \leq \cdot)$  is the theoretical cumulative distribution of  $\psi_n$ . Because the true distribution of  $\psi_n$  for test examples is unknown, the estimator approximates  $F_{\psi_n}$  using the training scores  $s_i$  and computes the empirical  $\hat{F}_{\psi_n}$ . As a result,
$$
\mathcal {R} \approx \hat {F} _ {\psi_ {n}} \left(g ^ {- 1} \left(1 - e ^ {- T}\right)\right) - \hat {F} _ {\psi_ {n}} \left(g ^ {- 1} \left(e ^ {- T}\right)\right) = \hat {\mathcal {R}}.
$$
Theorem 3.6 (Rejection rate upper bound). Let  $s$  be an anomaly score,  $\mathcal{M}_s$  be its confidence value, and  $\tau = 1 - 2e^{-T}$  be the rejection threshold. For  $n \in \mathbb{N}$ ,  $\gamma \in [0,0.5)$ , and small  $\delta > 0$ , there exists a positive real function  $h(n,\gamma,T,\delta)$  such that  $\mathcal{R} \leq h(n,\gamma,T,\delta)$  with probability at least  $1 - \delta$ , i.e. the rejection rate is bounded.
Proof. Theorem 3.1 states that there exists two functions  $t_1 = t_1(n,\gamma,T)$ ,  $t_2 = t_2(n,\gamma,T) \in [0,1]$  such that the confidence is lower than  $\tau$  if  $\psi_n \in [t_1,t_2]$ . Moreover, Theorems A.1 and A.2 claim that  $\psi_n$  has a distribution that is close to uniform with high probability (see the theorems and proofs in the Supplement). As a result, with probability at least  $1 - \delta$ , we find  $h(n,\gamma,T,\delta)$  as follows:
$$
\begin{array}{l} \mathcal {R} = \mathbb {P} \left(\mathcal {M} _ {s} \leq 1 - 2 e ^ {- T}\right) \overset {\text {T 3 . 1}} {\underset {\leq} {\longrightarrow}} \mathbb {P} \left(\psi_ {n} \in \left[ t _ {1}, t _ {2} \right]\right) = F _ {\psi_ {n}} \left(t _ {2}\right) - F _ {\psi_ {n}} \left(t _ {1}\right) \\ \stackrel {\mathrm {T A . 2}} {\leq} F _ {\psi} (t _ {2}) - F _ {\psi} (t _ {1}) + 2 \sqrt {\frac {\ln \frac {2}{\delta}}{2 n}} \stackrel {\mathrm {T A . 1}} {=} t _ {2} (n, \gamma , T) - t _ {1} (n, \gamma , T) + 2 \sqrt {\frac {\ln \frac {2}{\delta}}{2 n}} = h (n, \gamma , T, \delta). \\ \end{array}
$$
3.3 Upper Bounding the Expected Test Time Cost
In a learning with reject scenario, there are costs associated with three outcomes: false positives  $(c_{fp} > 0)$ , false negatives  $(c_{fn} > 0)$ , and rejection  $(c_r)$  because abstaining typically involves having a person intervene. Estimating an expected per example prediction cost at test time can help with model selection and give a sense of performance. Theorem 3.8 provides an upper bound on the expected per example cost when (1) using our estimated rejection rate (Theorem 3.5), and (2) setting the decision threshold  $\lambda$  as in Sec. 2.
Definition 3.7 (Cost function). Let  $Y$  be the true label random variable. Given the costs  $c_{fp} > 0$ ,  $c_{fn} > 0$ , and  $c_r$ , the cost function is a function  $c: \{0,1\} \times \{0,1,\mathbb{R}\} \to \mathbb{R}$  such that
$$
c (Y, \hat {Y}) = c _ {r} \mathbb {P} (\hat {Y} = \circledast) + c _ {f p} \mathbb {P} (\hat {Y} = 1 | Y = 0) + c _ {f n} \mathbb {P} (\hat {Y} = 0 | Y = 1)
$$
Note that defining a specific cost function requires domain knowledge. Following the learning to reject literature, we set an additive cost function. Moreover, the rejection cost needs to satisfy the inequality  $c_{r} \leq \min \{(1 - \gamma)c_{fp}, \gamma c_{fn}\}$ . This avoids the possibility of predicting always anomaly for an expected cost of  $(1 - \gamma)c_{fp}$ , or always normal with an expected cost of  $\gamma c_{fn}$  [52].
Theorem 3.8. Let  $c$  be a cost function as defined in Def. 3.7, and  $g$  be as in Def. 3.4. Given a (test) example  $x$  with score  $s$ , the expected example-wise cost is bounded by
$$
\mathbb {E} _ {x} [ c ] \leq \min  \left\{\gamma , A \right\} c _ {f n} + (1 - B) c _ {f p} + (B - A) c _ {r}, \tag {3}
$$
where  $A = \hat{F}_{\psi_n}(g^{-1}(e^{-T}))$  and  $B = \hat{F}_{\psi_n}(g^{-1}(1 - e^{-T}))$  are as in Theorem 3.5.
Proof. We indicate the true label random variable as  $Y$ , and the non-rejected false positives and false negatives as, respectively,
$$
F P = \mathbb {P} \left(\hat {Y} = 1 | Y = 0, \mathcal {M} _ {s} > 1 - 2 e ^ {- T}\right) F N = \mathbb {P} \left(\hat {Y} = 0 | Y = 1, \mathcal {M} _ {s} > 1 - 2 e ^ {- T}\right)
$$
Using Theorem 3.5 results in
$$
\mathbb {E} _ {x} [ c ] = \mathbb {E} _ {x} [ c _ {f n} F N + c _ {f p} F P + c _ {r} \mathcal {R} ] = \mathbb {E} _ {x} [ c _ {f n} F N ] + \mathbb {E} _ {x} [ c _ {f p} F P ] + c _ {r} (B - A)
$$
where  $A = \hat{F}_{\psi_n}(g^{-1}(e^{-T}))$ ,  $B = \hat{F}_{\psi_n}(g^{-1}(1 - e^{-T}))$  come from Theorem 3.5. Now we observe that setting a decision threshold  $\lambda$  such that  $n \times \gamma$  scores are higher implies that, on expectation, the detector predicts a proportion of positives equal to  $\gamma = \mathbb{P}(Y = 1)$ . Moreover, for  $\varepsilon = 2e^{-T}$ ,