SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
2 n \left(\frac {n (1 - \psi_ {n}) + 1}{2 + n} - \frac {n \gamma - 1}{n}\right) ^ {2} - T \geq 0,
$$
which is satisfied for  $(I_1) 0 \leq \psi_n \leq A_1 - \sqrt{B_1}$  and  $(I_2) A_1 + \sqrt{B_1} \leq \psi_n \leq 1$ , where
$$
A _ {1} = \frac {2 + n (n + 1) (1 - \gamma)}{n ^ {2}} \qquad B _ {1} = \frac {2 n \left(- 3 \gamma^ {2} - 2 n (1 - \gamma) ^ {2} + 4 \gamma - 3\right) + T (n + 2) ^ {2} - 8}{2 n ^ {3}}.
$$
However, for  $T \geq 4$ , no values of  $n, \gamma$ , and  $T$  that satisfy the constraint on  $\psi_{n}$  also satisfy  $I_{2}$ . Moving to  $I_{1}$ , we find out that if  $\psi_{n}$  satisfies  $I_{1}$ , then it also satisfies the constraint on  $\psi_{n}$  for any  $n, \gamma$ , and  $T$ . Therefore, we set  $t_1(n,\gamma ,T) = A_1 - \sqrt{B_1}$ . As a result,
$$
\psi_ {n} \leq t _ {1} \Rightarrow \mathbb {P} (\hat {Y} = 1 | s) \leq e ^ {- T} \Rightarrow \mathcal {M} _ {s} \geq 1 - 2 e ^ {- T}.
$$
(b) Similarly,  $\mathbb{P}(\hat{Y} = 0|s)$  can be seen as the cumulative distribution  $F$  of  $\mathcal{B}(p_s,n)$ , with  $n(1 - \gamma)$  successes and  $p_{s} = \frac{1 + n\psi_{n}(s)}{2 + n}$ . By seeing the binomial as a sum of Bernoulli random variables, and using the property of its cumulative distribution  $F(n(1 - \gamma),n,p_{s}) + F(n\gamma -1,n,1 - p_{s}) = 1$ , we apply the Hoeffding's inequality and compare such upper bound to the  $e^{-T}$ . We obtain
$$
2 n \left(\frac {1 + \psi_ {n} n}{2 + n} - (1 - \gamma)\right) ^ {2} - T \geq 0
$$
that holds with the constraint  $\psi_n \geq \frac{(2 + n)(1 - \gamma) - 1}{n}$ . The quadratic inequality in  $\psi_n$  has solutions for  $(I_1) 0 \leq \psi_n \leq A_2 - \sqrt{B_2}$  and  $(I_2) A_2 + \sqrt{B_2} \leq \psi_n \leq 1$ , where  $A_2 = \frac{(2 + n)(1 - \gamma) - 1}{n}$ , and  $B_2 = \frac{T(n + 2)^2}{2n^3}$ . However, the constraint limits the solutions to  $I_2$ , i.e. for  $\psi_n \geq A_2 + \sqrt{B_2}$ . Thus, we set  $t_2(n, \gamma, T) = A_2 + \sqrt{B_2}$  and conclude that
$$
\psi_ {n} \geq t _ {2} \Longrightarrow \mathbb {P} (\hat {Y} = 1 | s) \geq 1 - e ^ {- T} \Longrightarrow \mathcal {M} _ {s} \geq 1 - 2 e ^ {- T}.
$$
Secondly, Theorem 3.6 relies on two important results: given  $S$  the anomaly score random variable, (1) if  $\psi_{n}$  was the theoretical cumulative of  $S$ , it would have a uniform distribution (Theorem A.1), but because in practice (2)  $\psi_{n}$  is the empirical cumulative of  $S$ , its distribution is close to uniform with high probability (Theorem A.2). We prove these results in the following theorems.
Theorem A.1. Let  $S$  be the anomaly score random variable, and  $\psi = F_S(S)$  be the cumulative distribution of  $S$  applied to  $S$  itself. Then  $\psi \sim \text{Unif}(0, 1)$ .
Proof. We prove that, if  $\psi \sim Unif(0,1)$ , then  $F_{\psi}(t) = t$  for any  $t \in [0,1]$ :
$$
F _ {\psi} (t) = \mathbb {P} (\psi \leq t) = \mathbb {P} \left(F _ {S} (S) \leq t\right) = \mathbb {P} \left(S \leq F _ {S} ^ {- 1} (t)\right) = F _ {S} \left(F _ {S} ^ {- 1} (t)\right) = t \Rightarrow \psi \sim U n i f (0, 1).
$$
Theorem A.2. Let  $\psi$  be as in Theorem A.1, and  $F_{\psi_n}$  be its empirical distribution obtained from a sample of size  $n$ . For any small  $\delta > 0$  and  $t \in [0,1]$ , with probability  $>1 - \delta$
$$
F _ {\psi_ {n}} (t) \in \left[ F _ {\psi} (t) - \sqrt {\frac {\ln \frac {2}{\delta}}{2 n}}, F _ {\psi} (t) + \sqrt {\frac {\ln \frac {2}{\delta}}{2 n}} \right].
$$
Proof. For any  $\varepsilon > 0$ , the DKW inequality implies
$$
\mathbb {P} \left(\sup  _ {t \in [ 0, 1 ]} | F _ {\psi_ {n}} (t) - F _ {\psi} (t) | > \varepsilon\right) \leq 2 \exp \left(- 2 n \varepsilon^ {2}\right).
$$
By setting  $\delta = 2\exp \left(-2n\varepsilon^{2}\right)$ , i.e.  $\varepsilon = \sqrt{\frac{\ln\frac{2}{\delta}}{2n}}$ , and using the complementary probability we conclude that
$$
\mathbb {P} \left(\sup  _ {t \in [ 0, 1 ]} | F _ {\psi_ {n}} (t) - F _ {\psi} (t) | \leq \sqrt {\frac {\ln \frac {2}{\delta}}{2 n}}\right) > 1 - \delta .
$$
Data. Table 3 shows the properties of the 34 datasets used for the experimental comparison, in terms of number of examples, features, and contamination factor  $\gamma$ . The datasets can be downloaded in the following link: https://github.com/Minqi824/ADBench/tree/main/datasets/Classical.
Q1. REJEX against the baselines. Table 4 and Table 5 show the results (mean ± std) aggregated by detectors in terms of, respectively, cost per example and ranking position. Results confirm that REJEX obtains an average cost per example lower than all the baselines for 9 out of 12 detectors, which is similar to the runner-up SS-REPEN for the remaining 3 detectors. Moreover, REJEX has always the best (lowest) average ranking position.
Q2. Varying the costs  $c_{fp}, c_{fn}, c_r$ . Table 6 and Table 7 show the average cost per example and the ranking position (mean ± std) aggregated by detectors for three representative cost functions, as discussed in the paper. Results are similar in all three cases. For high false positives cost ( $c_{fp} = 10$ ), REJEX obtains an average cost per example lower than all the baselines for 11 out of 12 detectors and always the best average ranking position. For high false negative cost ( $c_{fn} = 10$ ) as well as for low rejection cost ( $c_{fp} = 5, c_{fn} = 5, c_r = \gamma$ ), it has the lowest average cost for all detectors and always the best average ranking. Moreover, when rejection is highly valuable (low cost), REJEX's cost has a large gap with respect to the baselines, which means that it is particularly useful when rejection is less expensive.
Table 3: Properties (number of examples, features, and contamination factor) of the 34 benchmark datasets used for the experiments.
DATASET	#EXAMPLES	#FEATURES	γ
ALOI	20000	27	0.0315
ANNTHYROID	7062	6	0.0756
CAMPAIGN	20000	62	0.1127
CARDIO	1822	21	0.0960
CARDIOTOCOGRAPHY	2110	21	0.2204
CENSUS	20000	500	0.0854
DONORS	20000	10	0.2146
FAULT	1941	27	0.3467
FRAUD	20000	29	0.0021
GLASS	213	7	0.0423
HTTP	20000	3	0.0004
INTERNETADS	1966	1555	0.1872
LANDSAT	6435	36	0.2071
LETTER	1598	32	0.0626
LYMPHOGRAPHY	148	18	0.0405
MAMMOGRAPHY	7848	6	0.0322
MUSK	3062	166	0.0317
OPTDIGITS	5198	64	0.0254
PAGEBLOCKS	5393	10	0.0946
PENDIGITS	6870	16	0.0227
PIMA	768	8	0.3490
SATELLITE	6435	36	0.3164
SATIMAGE	5801	36	0.0119
SHUTTLE	20000	9	0.0725
THYROID	3656	6	0.0254
VERTEBRAL	240	6	0.1250
VOWELS	1452	12	0.0317
WAVEFORM	3443	21	0.0290
WBC	223	9	0.0448
WDBC	367	30	0.0272
WILT	4819	5	0.0533
WINE	129	13	0.0775
WPBC	198	33	0.2374
YEAST	1453	8	0.3310
Table 4: Cost per example (mean ± std) per detector aggregated over the datasets. Results show that REJEX obtains a lower average cost for 9 out of 12 detectors and similar average cost as the runner-up SS-REPEN for the remaining 3 detectors. Moreover, REJEX has the best overall average (last row).
COST PER EXAMPLE (MEAN ± STD.)
DET.	REJEX	SS-REPEN	MV	ENS	UDR	EM	STABILITY	NoREJECT
AE	0.122 ± 0.139	0.124 ± 0.133	0.136 ± 0.143	0.134 ± 0.151	0.138 ± 0.148	0.143 ± 0.150	0.143 ± 0.149	0.148 ± 0.152
COPOD	0.123 ± 0.138	0.125 ± 0.134	0.135 ± 0.142	0.133 ± 0.140	0.140 ± 0.144	0.143 ± 0.148	0.145 ± 0.147	0.146 ± 0.148
ECOD	0.120 ± 0.136	0.124 ± 0.135	0.129 ± 0.136	0.133 ± 0.143	0.139 ± 0.142	0.140 ± 0.145	0.143 ± 0.144	0.145 ± 0.145
GMM	0.122 ± 0.135	0.124 ± 0.137	0.144 ± 0.141	0.136 ± 0.146	0.142 ± 0.145	0.156 ± 0.148	0.151 ± 0.147	0.156 ± 0.149
HBOS	0.116 ± 0.129	0.123 ± 0.138	0.131 ± 0.132	0.134 ± 0.136	0.135 ± 0.137	0.139 ± 0.141	0.140 ± 0.139	0.142 ± 0.142
IFOR	0.115 ± 0.128	0.123 ± 0.135	0.130 ± 0.136	0.129 ± 0.136	0.134 ± 0.139	0.140 ± 0.143	0.139 ± 0.141	0.143 ± 0.144
INNE	0.113 ± 0.129	0.122 ± 0.133	0.145 ± 0.134	0.146 ± 0.140	0.145 ± 0.138	0.147 ± 0.140	0.146 ± 0.139	0.145 ± 0.140
KDE	0.127 ± 0.140	0.127 ± 0.134	0.143 ± 0.145	0.138 ± 0.145	0.144 ± 0.145	0.150 ± 0.148	0.145 ± 0.143	0.152 ± 0.148
KNN	0.119 ± 0.123	0.125 ± 0.135	0.140 ± 0.131	0.135 ± 0.131	0.135 ± 0.130	0.144 ± 0.132	0.141 ± 0.131	0.146 ± 0.133
LODA	0.125 ± 0.133	0.125 ± 0.134	0.131 ± 0.130	0.139 ± 0.137	0.140 ± 0.136	0.146 ± 0.141	0.141 ± 0.131	0.151 ± 0.142
LOF	0.126 ± 0.131	0.126 ± 0.136	0.155 ± 0.140	0.140 ± 0.139	0.142 ± 0.138	0.157 ± 0.140	0.151 ± 0.139	0.158 ± 0.140
OCSVM	0.120 ± 0.131	0.125 ± 0.133	0.138 ± 0.138	0.132 ± 0.140	0.138 ± 0.140	0.141 ± 0.140	0.137 ± 0.136	0.147 ± 0.143
AVG.	0.121 ± 0.133	0.125 ± 0.135	0.138 ± 0.137	0.136 ± 0.140	0.139 ± 0.140	0.146 ± 0.143	0.144 ± 0.140	0.148 ± 0.144
Table 5: Ranking positions (mean ± std) per detector aggregated over the datasets. Results show that REJEX obtains always the lowest average rank, despite being close to the runner-up SS-REPEN when the detector is LODA.
DET.	RANKING POSITION (MEAN ± STD.)
REJEX	SS-REPEN	MV	ENS	UDR	EM	STABILITY	NOREJECT
AE	2.63 ± 1.63	3.96 ± 2.94	4.78 ± 2.10	3.81 ± 2.11	4.98 ± 1.88	5.15 ± 1.80	4.92 ± 1.78	5.77 ± 1.84
COPOD	2.49 ± 1.65	3.44 ± 2.75	3.97 ± 1.92	4.25 ± 2.17	5.24 ± 1.70	5.13 ± 1.67	5.59 ± 1.48	5.89 ± 2.13
ECOD	2.43 ± 1.44	3.62 ± 2.86	3.73 ± 1.96	4.13 ± 2.29	5.53 ± 1.57	4.75 ± 1.62	5.74 ± 1.39	6.07 ± 1.93
GMM	2.12 ± 1.08	3.05 ± 2.49	5.26 ± 1.92	3.49 ± 2.00	4.43 ± 1.66	6.26 ± 1.24	5.20 ± 1.43	6.20 ± 1.45
HBOS	2.29 ± 1.57	3.64 ± 2.98	4.54 ± 2.11	4.39 ± 2.06	4.95 ± 1.79	5.04 ± 1.80	5.61 ± 1.40	5.52 ± 1.88
IFOR	2.23 ± 1.48	3.78 ± 2.78	4.12 ± 1.90	4.26 ± 2.08	5.10 ± 1.88	5.27 ± 1.66	5.34 ± 1.38	5.91 ± 2.22
INNE	1.73 ± 1.14	3.18 ± 2.74	5.86 ± 2.42	5.57 ± 1.40	4.94 ± 1.62	5.57 ± 1.60	5.32 ± 1.37	3.83 ± 1.63
KDE	2.33 ± 1.42	3.99 ± 2.86	4.74 ± 2.06	3.79 ± 2.03	5.01 ± 1.90	5.43 ± 1.59	4.87 ± 1.92	5.84 ± 1.80
KNN	2.02 ± 1.29	3.58 ± 2.87	4.87 ± 1.81	3.94 ± 1.94	4.41 ± 1.83	5.75 ± 1.49	5.22 ± 1.62	6.21 ± 1.48
LODA	2.89 ± 1.77	3.17 ± 2.30	4.15 ± 2.26	4.36 ± 2.14	4.99 ± 2.00	5.50 ± 2.04	4.98 ± 2.11	5.95 ± 1.73
LOF	2.04 ± 1.01	3.16 ± 2.73	5.68 ± 1.40	3.32 ± 1.71	3.96 ± 1.63	6.15 ± 1.19	5.47 ± 1.49	6.22 ± 1.31
OCSVM	2.33 ± 1.29	3.92 ± 2.84	4.89 ± 1.98	3.85 ± 2.17	4.86 ± 1.89	5.31 ± 1.80	5.06 ± 1.89	5.78 ± 1.66
AVG.	2.29 ± 1.40	3.54 ± 2.76	4.72 ± 1.99	4.10 ± 2.01	4.87 ± 1.78	5.44 ± 1.63	5.28 ± 1.60	5.77 ± 1.76