SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  $FP \leq \mathbb{P}\left(\hat{Y} = 1|\mathcal{M}_s > 1 - \varepsilon\right) = 1 - B$  as false positives must be less than total accepted positive predictions;
-  $FN \leq \gamma$  and  $FN \leq \mathbb{P}\left(\hat{Y} = 0|\mathcal{M}_s > 1 - \varepsilon\right) = A$ , as you cannot have more false negatives than positives  $(\gamma)$ , nor than accepted negative predictions  $(A)$ .
From these observations, we conclude that  $\mathbb{E}_x[c] \leq \min \{\gamma, A\} c_{fn} + (1 - B)c_{fp} + (B - A)c_r$ .
There is no research on learning to reject in unsupervised anomaly detection. However, three main research lines are connected to this work.
1) Supervised methods. If some labels are available, one can use traditional supervised approaches to add the reject option into the detector [11, 38]. Commonly, labels can be used to find the optimal rejection threshold in two ways: 1) by trading off the model performance (e.g., AUC) on the accepted examples with its rejection rate [24, 1], or 2) by minimizing a cost function [46, 7], a risk function [18, 27], or an error function [35, 33]. Alternatively, one can include the reject option in the model and directly optimize it during the learning phase [60, 12, 31].
2) Self-Supervised methods. If labels are not available, one can leverage self-supervised approaches to generate pseudo-labels in order to apply traditional supervised learning to reject methods [26, 59, 19, 37]. For example, one can employ any unsupervised anomaly detector to assign training labels, fit a (semi-)supervised detector (such as DEEPSAD [57] or REPEN [47]) on the pseudo labels, compute a confidence metric [14], and find the optimal rejection threshold by minimizing the cost function treating the pseudo-labels as the ground truth.
3) Optimizing unsupervised metrics. There exist several unsupervised metrics (i.e., they can be computed without labels) for quantifying detector quality [43]. Because they do not need labels, one can find the rejection threshold by maximizing the margin between the detector's quality (computed using such metric) on the accepted and on the rejected examples [54]. This allows us to obtain a model that performs well on the accepted examples and poorly on the rejected ones, which is exactly the same intuition that underlies the supervised approaches. Some examples of existing unsupervised metrics (see [43]) are the following. EM and Mv [20] quantify the clusterness of inlier scores, where more compact scores indicate better models. STABILITY [48] measures the robustness of anomaly detectors' predictions by looking at how consistently they rank examples by anomalousness. UDR [15] is a model-selection metric that selects the model with a hyperparameter setting that yields consistent results across various seeds, which can be used to set the rejection threshold through the analogy [hyperparameter, seed] and [rejection threshold, detectors]. Finally, ENS [56, 67] measures the detector trustworthiness as the ranking-based similarity (e.g., correlation) of a detector's output to the "pseudo ground truth", computed via aggregating the output of an ensemble of detectors, which allows one to set the rejection threshold that maximizes the correlation between the detector's and the ensemble's outputs.
We experimentally address the following research questions:
Q1. How does REJEX's cost compare to the baselines?
Q2. How does varying the cost function affect the results?
Q3. How does REJEX's CPU time compare to the baselines?
Q4. Do the theoretical results hold in practice?
Q5. Would REJEX's performance significantly improve if it had access to training labels?
Methods. We compare  $\mathbf{REJEx}^1$  against 7 baselines for setting the rejection threshold. These can be divided into three categories: no rejection, self-supervised, and unsupervised metric based.
We use one method NOREJECT that always makes predictions and never rejects (no reject option).
We consider one self-supervised approach SS-REPEN [47]. This uses (any) unsupervised detector to obtain pseudo labels for the training set. It then sets the rejection threshold as follows: 1) it creates a held-out validation set  $(20\%)$ , 2) it fits REPEN, a state-of-the-art (semi-)supervised anomaly detector on the training set with the pseudo labels, 3) it computes on the validation set the confidence values as the margin between REPEN's predicted class probabilities  $|\mathbb{P}(Y = 1|s) - \mathbb{P}(Y = 0|s)|$ , 4) it finds the optimal threshold  $\tau$  by minimizing the total cost obtained on the validation set.
We consider 5 approaches that employ an existing unsupervised metric to set the rejection threshold and hence do not require having access to labels. MV [20], EM [20], and STABILITY [48] are unsupervised metric-based methods based on stand-alone internal evaluations that use a single anomaly detector to measure its quality, UDR [15] and ENS [56] are unsupervised consensus-based metrics that an ensemble of detectors (all 12 considered in our experiments) to measure a detector's quality. We apply each of these 5 baselines as follows. 1) We apply the unsupervised detector to assign an anomaly score to each train set example. 2) We convert these scores into class probabilities using [34]. 3) We compute the confidence scores on the training set as difference between these probabilities:  $|\mathbb{P}(Y = 1|s) - \mathbb{P}(Y = 0|s)|$ . 4) We evaluate possible thresholds on this confidence by computing the considered unsupervised metric on the accepted and on the rejected examples and select the threshold that maximizes the difference in the metric's value on these two sets of examples. This aligns with the common learning to reject criteria for picking a threshold [9, 54] such that the model performs well on the accepted examples and poorly on the rejected ones.
Data. We carry out our study on 34 publicly available benchmark datasets, widely used in the literature [23]. These datasets cover many application domains, including healthcare (e.g., disease diagnosis), audio and language processing (e.g., speech recognition), image processing (e.g., object identification), and finance (e.g., fraud detection). To limit the computational time, we randomly sub-sample 20,000 examples from all large datasets. Table 3 in the Supplement provides further details.
Anomaly Detectors and Hyperparameters. We set our tolerance  $\varepsilon = 2e^{-T}$  with  $T = 32$ . Note that the exponential smooths out the effect of  $T \geq 4$ , which makes setting a different  $T$  have little impact. We use a set of 12 unsupervised anomaly detectors implemented in PYOD [66] with default hyperparameters [62] because the unsupervised setting does not allow us to tune them: KNN [3], IFOREST [42], LOF [5], OCSVM [58], AE [8], HBOS [21], LODA [53], COPOD [39], GMM [2], ECOD [40], KDE [36], INNE [4]. We set all the baselines' rejection threshold via Bayesian Optimization with 50 calls [17].
Setup. For each [dataset, detector] pair, we proceed as follows: (1) we split the dataset into training and test sets (80-20) using 5 fold cross-validation; (2) we use the detector to assign the anomaly scores on the training set; (3) we use either REJEx or a baseline to set the rejection threshold;
$^{1}$ Code available at: https://github.com/Lorenzo-Perini/RejEx.
Sec. 4 describes these approaches.
(4) we measure the total cost on the test set using the given cost function. We carry out a total of  $34 \times 12 \times 5 = 2040$  experiments. All experiments were run on an Intel(R) Xeon(R) Silver 4214 CPU.
Figure 1: Average cost per example (left) and rank (right) aggregated per detector (x-axis) over all the datasets. Our method obtains the lowest (best) cost for 9 out of 12 detectors and it always has the lowest (best) ranking position for  $c_{fp} = c_{fn} = 1$ ,  $c_r = \gamma$ .
Q1: REJEx against the baselines. Figure 1 shows the comparison between our method and the baselines, grouped by detector, when setting the costs  $c_{fp} = c_{fn} = 1$  and  $c_{r} = \gamma$  (see the Supplement for further details). REJEx achieves the lowest (best) cost per example for 9 out of 12 detectors (left-hand side) and similar values to SS-REPEN when using LODA, LOF and KDE. Averaging over the detectors, REJEx reduces the relative cost by more than  $5\%$  vs SS-REPEN,  $11\%$  vs ENS,  $13\%$  vs MV and UDR,  $17\%$  vs EM,  $19\%$  vs NOREJECT. Table 4 (Supplement) shows a detailed breakdown.
For each experiment, we rank all the methods from 1 to 8, where position 1 indicates the lowest (best) cost. The right-hand side of Figure 1 shows that REJEX always obtains the lowest average ranking. We run a statistical analysis separately for each detector: the Friedman test rejects the null-hypothesis that all methods perform similarly  $(\mathrm{p - value} < e^{-16})$  for all the detectors. The ranking-based post-hoc Bonferroni-Dunn statistical test [13] with  $\alpha = 0.05$  finds that REJEX is significantly better than the baselines for 6 detectors (INNE, IFOREST, HBOS, KNN, ECOD, OCSVM).
Figure 2: Average cost per example aggregated by detector over the 34 datasets when varying the three costs on three representative cases: (left) false positives are penalized more, (center) false negatives are penalized more, (right) rejection has a lower cost than FPs and FNs.
Q2. Varying the costs  $c_{fp}, c_{fn}, c_r$ . The three costs  $c_{fp}, c_{fn},$  and  $c_r$  are usually set based on domain knowledge: whether to penalize the false positives or the false negatives more depends on the application domain. Moreover, the rejection cost needs to satisfy the constraint  $c_r \leq \min \{(1 -$
$\gamma)c_{fp},\gamma c_{fn}\}$  [52]. Therefore, we study their impact on three representative cases: (case 1) high false positive cost  $(c_{fp} = 10,c_{fn} = 1,c_r = \min \{10(1 - \gamma),\gamma \})$  , (case 2) high false negative cost  $(c_{fp} = 1,c_{fn} = 10,c_r = \min \{(1 - \gamma),10\gamma \})$  , and (case 3) same cost for both mispredictions but low rejection cost  $(c_{fp} = 5,c_{fn} = 5,c_r = \gamma)$  . Note that scaling all the costs has no effect on the relative comparison between the methods, so the last case is equivalent to  $c_{fp} = 1$ $c_{fn} = 1$  , and  $c_{r} = \gamma /5$
Figure 2 shows results for the three scenarios. Compared to the unsupervised metric-based methods, the left plot shows that our method is clearly the best for high false positives cost: for 11 out of 12 detectors, REJEX obtains both the lowest (or similar for GMM) average cost and the lowest average ranking position. This indicates that using REJEX is suitable when false alarms are expensive. Similarly, the right plot illustrates that REJEX outperforms all the baselines for all the detectors when the rejection cost is low (w.r.t. the false positive and false negative costs). Even when the false negative cost is high (central plot), REJEX obtains the lowest average cost for 11 detectors and has always the lowest average rank per detector. See the Supplement (Table 6 and 7) for more details.
Table 1: Average CPU time (in ms) per training example  $(\pm$  std) to set the rejection threshold aggregated over all the datasets when using IFOREST, HBOS, and COPOD as unsupervised anomaly detector. REJEX has a lower time than all the methods but NOREJECT, which uses no reject option.
DETECTOR	CPU time in ms (mean ± std.)
NOREJECT	REJEX	SS-REPEN	MV	EM	UDR	ENS	STABILITY
IFOREST	0.0±0.0	0.06±0.22	90±68	89±128	155±161	120±132	122±135	916±900
HBOS	0.0±0.0	0.13±0.93	89±53	39±81	80±129	200±338	210±358	142±242
COPOD	0.0±0.0	0.04±0.04	84±53	21±28	81±60	119±131	123±138	140±248
Q3. Comparing the CPU time. Table 1 reports CPU time in milliseconds per training example aggregated over the 34 datasets needed for each method to set the rejection threshold on three unsupervised anomaly detectors (IFOREST, HBOS, COPOD). NOREJECT has CPU time equal to 0 because it does not use any reject option. REJEX takes just a little more time than NOREJECT because computing EXCEED has linear time while setting a constant threshold has constant time. In contrast, all other methods take  $1000 \times$  longer because they evaluate multiple thresholds. For some of these (e.g., STABILITY), this involves an expensive internal procedure.
Figure 3: Average cost per example (left) and average rejection rate (right) at test time aggregated by dataset over the 12 detectors. In both plots, the empirical value (circle) is always lower than the predicted upper bound (continuous black line), which makes it consistent with the theory. On the right, the expected rejection rates (stars) are almost identical to the empirical values.
Q4. Checking on the theoretical results. Section 3 introduces three theoretical results: the rejection rate estimate (Theorem 3.5), and the upper bound for the rejection rate (Theorem 3.6) and for the cost (Theorem 3.8). We run experiments to verify whether they hold in practice. Figure 3 shows the results aggregated over the detectors. The left-hand side confirms that the prediction cost per example (blue circle) is always  $\leq$  than the upper bound (black line). Note that the upper bound is sufficiently strict, as in some cases it equals the empirical cost (e.g., Census, Wilt, Optdigits).
The right-hand side shows that our rejection rate estimate (orange star) is almost identical to the empirical rejection rate (orange circle) for most of the datasets, especially the large ones. On the other hand, small datasets have the largest gap, e.g., Wine  $(n = 129)$ , Lymphography  $(n = 148)$ , WPBC  $(n = 198)$ , Vertebral  $(n = 240)$ . Finally, the empirical rejection rate is always lower than the theoretical upper bound (black line), which we compute by using the empirical frequencies  $\psi_{n}$ .
Table 2: Mean ± std. for the cost per example (on the left) and the rejection rate (on the right) at test time on a per detector basis and aggregated over the datasets.
DETECTOR	COST PER EXAMPLE (MEAN ± STD.)	REJECTION RATE (MEAN ± STD.)
REJEX	ORACLE	REJEX	ORACLE
AE	0.126 ± 0.139	0.126 ± 0.139	0.131 ± 0.132	0.118 ± 0.125
COPOD	0.123 ± 0.140	0.121 ± 0.140	0.123 ± 0.131	0.101 ± 0.114
ECOD	0.119 ± 0.138	0.118 ± 0.138	0.125 ± 0.130	0.107 ± 0.114
GMM	0.123 ± 0.135	0.122 ± 0.134	0.139 ± 0.143	0.132 ± 0.136
HBOS	0.118 ± 0.129	0.118 ± 0.129	0.139 ± 0.148	0.114 ± 0.128
IFOREST	0.118 ± 0.129	0.118 ± 0.128	0.127 ± 0.131	0.118 ± 0.130
INNE	0.115 ± 0.129	0.115 ± 0.128	0.132 ± 0.132	0.122 ± 0.125
KDE	0.129 ± 0.140	0.129 ± 0.139	0.121 ± 0.129	0.105 ± 0.120
KNN	0.119 ± 0.123	0.118 ± 0.123	0.127 ± 0.129	0.112 ± 0.117
LODA	0.125 ± 0.133	0.122 ± 0.130	0.126 ± 0.124	0.110 ± 0.114
LOF	0.126 ± 0.131	0.125 ± 0.131	0.129 ± 0.126	0.118 ± 0.115
OCSVM	0.120 ± 0.131	0.120 ± 0.131	0.126 ± 0.128	0.107 ± 0.115
AVG.	0.122 ± 0.133	0.121 ± 0.133	0.129 ± 0.132	0.114 ± 0.121
Q5. Impact of training labels on REJEX. We simulate having access to the training labels and include an extra baseline: ORACLE uses EXCEED as a confidence metric and sets the (optimal) rejection threshold by minimizing the cost function using the training labels. Table 2 shows the average cost and rejection rates at test time obtained by the two methods. Overall, REJEX obtains an average cost that is only  $0.6\%$  higher than ORACLE's cost. On a per-detector basis, REJEX obtains a  $2.5\%$  higher cost in the worst case (with LODA), while getting only a  $0.08\%$  increase in the best case (with KDE). Comparing the rejection rates, REJEX rejects on average only  $\approx 1.5$  percentage points more examples than ORACLE ( $12.9\%$  vs  $11.4\%$ ). The supplement provides further details.
This paper addressed learning to reject in the context of unsupervised anomaly detection. The key challenge was how to set the rejection threshold without access to labels which are required by all existing approaches. We proposed an approach REJEx that exploits our novel theoretical analysis of the EXCEED confidence metric. Our new analysis shows that it is possible to set a constant rejection threshold and that doing so offers strong theoretical guarantees. First, we can estimate the proportion of rejected test examples and provide an upper bound for our estimate. Second, we can provide a theoretical upper bound on the expected test-time prediction cost per example. Experimentally, we compared REJEx against several (unsupervised) metric-based methods and showed that, for the majority of anomaly detectors, it obtained lower (better) cost. Moreover, we proved that our theoretical results hold in practice and that our rejection rate estimate is almost identical to the true value in the majority of cases.
Limitations. Because REJEX does not rely on labels, it can only give a coarse-grained view of performance. For example, in many applications anomalies will have varying costs (i.e., there are instance-specific costs) which we cannot account for. Moreover, REJEX has a strictly positive rejection rate, which may increase the cost of a highly accurate detector. However, this happens only in  $\approx 5\%$  of our experiments.
This research is supported by an FB Ph.D. fellowship by FWO-Vlaanderen (grant 1166222N) [LP], the Flemish Government under the "Onderzoeksprogramma Artificièle Intelligentie (AI) Vlaanderen" programme [LP,JD], and KUL Research Fund iBOF/21/075 [JD].
[1] M. R. Abbas, M. S. A. Nadeem, A. Shaheen, A. A. Alshdadi, R. Alharbey, S.-O. Shim, and W. Aziz. Accuracy rejection normalized-cost curves (arnccs): A novel 3-dimensional framework for robust classification. IEEE Access, 7:160125-160143, 2019.
[2] C. C. Aggarwal. An introduction to outlier analysis. In Outlier analysis, pages 1-34. Springer, 2017.
[3] F. Angiulli and C. Pizzuti. Fast outlier detection in high dimensional spaces. In European conference on principles of data mining and knowledge discovery, pages 15-27. Springer, 2002.
[4] T. R. Bandaragoda, K. M. Ting, D. Albrecht, F. T. Liu, Y. Zhu, and J. R. Wells. Isolation-based anomaly detection using nearest-neighbor ensembles. Computational Intelligence, 34(4): 968–998, 2018.
[5] M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93-104, 2000.
[6] V. Chandola, A. Banerjee, and V. Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3):1-58, 2009.
[7] N. Charoenphakdee, Z. Cui, Y. Zhang, and M. Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning, pages 1507-1517. PMLR, 2021.
[8] Z. Chen, C. K. Yeo, B. S. Lee, and C. T. Lau. Autoencoder-based network anomaly detection. In 2018 Wireless telecommunications symposium (WTS), pages 1-5. IEEE, 2018.
[9] C. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory, 16(1):41-46, 1970.
[10] L. Coenen, A. K. Abdullah, and T. Guns. Probability of default estimation, with a reject option. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pages 439-448. IEEE, 2020.
[11] D. Conte, P. Foggia, G. Percannella, A. Saggese, and M. Vento. An ensemble of rejecting classifiers for anomaly detection of audio events. In 2012 IEEE Ninth International Conference on Advanced Video and Signal-Based Surveillance, pages 76-81. IEEE, 2012.
[12] C. Cortes, G. DeSalvo, and M. Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory. Springer, 2016.
[13] J. Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30, 2006.
[14] C. Denis and M. Hebiri. Consistency of plug-in confidence sets for classification in semi-supervised learning. Journal of Nonparametric Statistics, 32(1):42-72, 2020.
[15] S. Duan, L. Matthew, A. Saraiva, N. Watters, C. P. Burgess, A. Lerchner, and I. Higgins. Unsupervised model selection for variational disentangled representation learning. arXiv preprint arXiv:1905.12614, 2019.
[16] R. El-Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010.
[17] P. I. Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811, 2018.
[55] C. Qiu, T. Pfrommer, M. Kloft, S. Mandt, and M. Rudolph. Neural transformation learning for deep anomaly detection beyond images. In International Conference on Machine Learning, pages 8703-8714. PMLR, 2021.
[56] S. Rayana and L. Akoglu. Less is more: Building selective anomaly ensembles. Acm transactions on knowledge discovery from data (tkdd), 10(4):1-33, 2016.
[57] L. Ruff, R. A. Vandermeulen, N. Gornitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.
[58] B. Scholkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson. Estimating the support of a high-dimensional distribution. Neural computation, 13(7):1443-1471, 2001.
[59] V. Sehwag, M. Chiang, and P. Mittal. Ssd: A unified framework for self-supervised outlier detection. arXiv preprint arXiv:2103.12051, 2021.
[60] S. Shekhar, M. Ghavamzadeh, and T. Javidi. Binary classification with bounded abstention rate. arXiv preprint arXiv:1905.09561, 2019.
[61] T. Shenkar and L. Wolf. Anomaly detection for tabular data with internal contrastive learning. In International Conference on Learning Representations, 2021.
[62] J. Soenen, E. Van Wolputte, L. Perini, V. Vercruyssen, W. Meert, J. Davis, and H. Blockeel. The effect of hyperparameter tuning on the comparative evaluation of unsupervised anomaly detection methods. In Proceedings of the KDD, volume 21, pages 1-9, 2021.
[63] D. Van der Plas, W. Meert, J. Verbraecken, and J. Davis. A reject option for automated sleep stage scoring. In Workshop on Interpretable ML in Healthcare at International Conference on Machine Learning (ICML), 2021.
[64] L. Xiang, X. Yang, A. Hu, H. Su, and P. Wang. Condition monitoring and anomaly detection of wind turbine based on cascaded and bidirectional deep learning networks. Applied Energy, 305: 117925, 2022.
[65] N. Zhao, X. Wen, and S. Li. A review on gas turbine anomaly detection for implementing health management. Turbo Expo: Power for Land, Sea, and Air, 2016.
[66] Y. Zhao, Z. Nasrullah, and Z. Li. Pyod: A python toolbox for scalable outlier detection. Journal of Machine Learning Research, 20(96):1-7, 2019. URL http://jmlr.org/papers/v20/19-011.html.
[67] A. Zimek, R. J. Campello, and J. Sander. Ensembles for unsupervised outlier detection: challenges and research questions a position paper. Acm Sigkdd Explorations Newsletter, 15(1): 11-22, 2014.
In this supplementary material we (1) provide additional theorems and proofs for Section 3, and (2) further describe the experimental results.
Firstly, we provide the proof for Theorem 3.1.
Theorem 3.1 (Analysis of EXCEED) Let  $s$  be an anomaly score, and  $\psi_{n} \in [0,1]$  the proportion of training scores  $\leq s$ . For  $T \geq 4$ , there exist  $t_1 = t_1(n,\gamma,T) \in [0,1]$ ,  $t_2 = t_2(n,\gamma,T) \in [0,1]$  such that
Proof. We split this proof into two parts: we show that the reverse inequalities, i.e. that  $(\mathbf{a})$  if  $\psi_n \leq t_1$ , then  $\mathcal{M}_s \geq 1 - 2e^{-T}$ , and  $(\mathbf{b})$  if  $\psi_n \geq t_2$ , then  $\mathcal{M}_s \geq 1 - 2e^{-T}$ , hold and prove the final statement because  $\mathbb{P}(\hat{Y} = 1|s)$  is monotonic increasing on  $s$ .
(a) The probability  $\mathbb{P}(\hat{Y} = 1|s)$  (as in Eq. 1) can be seen as the cumulative distribution  $F$  of a binomial random variable  $\mathcal{B}(q_s,n)$  with at most  $n\gamma - 1$  successes out of  $n$  trials, with  $q_{s} = \frac{n(1 - \psi_{n}) + 1}{2 + n}$  as the success probability. By applying Hoeffding's inequality, we obtain the upper bound
$$
\mathbb {P} (\hat {Y} = 1 | s) \leq \exp \left(- 2 n \left(\frac {n (1 - \psi_ {n}) + 1}{2 + n} - \frac {n \gamma - 1}{n}\right) ^ {2}\right)
$$
that holds for the constraint  $\psi_{n} \leq \frac{2 + n}{n^{2}} + \frac{1 - 2\gamma}{n} + (1 - \gamma)$ . Because  $\mathbb{P}(\hat{Y} = 1|s) \leq e^{-T}$  implies that  $\mathcal{M}_s \geq 1 - 2e^{-T}$ , we search for the values of  $\psi_{n}$  such that the upper bound is  $\leq e^{-T}$ . Forcing the upper bound  $\leq e^{-T}$  results in