SlowGuess/NeurIPS_2023_OCR huggingface dataset:
C. Jung, M. Kearns, S. Neel, A. Roth, L. Stapleton, and Z. S. Wu. An algorithmic framework for fairness elicitation, 2019. URL https://arxiv.org/abs/1905.10660.
M. P. Kim, O. Reingold, and G. N. Rothblum. Fairness through computationally-bounded awareness. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 4847-4857, Red Hook, NY, USA, 2018. Curran Associates Inc.
R. Kohavi. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD'96, page 202-207. AAAI Press, 1996.
B. Kulis. Metric learning: A survey. 2013.
P. Lahoti, K. P. Gummadi, and G. Weikum. Operationalizing individual fairness with pairwise fair representations. Proc. VLDB Endow., 13(4):506-518, dec 2019. ISSN 2150-8097. doi: 10.14778/3372716.3372723. URL https://doi.org/10.14778/3372716.3372723.
P. Moutafis, M. Leng, and I. A. Kakadiaris. An overview and empirical comparison of distance metric learning methods. IEEE Transactions on Cybernetics, 47(3):612-625, 2017. doi: 10.1109/TCYB.2016.2521767.
D. Mukherjee, M. Yurochkin, M. Banerjee, and Y. Sun. Two simple ways to learn individual fairness metrics from data. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
I. Niño-Adan, D. Manjarres, I. Landa-Torres, and E. Portillo. Feature weighting methods: A review. Expert Systems with Applications, 184:115424, 2021. ISSN 0957-4174.
J. L. Suárez-Díaz, S. García, and F. Herrera. A tutorial on distance metric learning: Mathematical foundations, algorithms, experimental analysis, prospects and challenges (with appendices on mathematical background and detailed algorithms explanation), 2018. URL https://arxiv.org/abs/1812.05944.
O. Tamuz, C. Liu, S. Belongie, O. Shamir, and A. T. Kalai. Adaptively learning the crowd kernel. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, page 673-680. Omnipress, 2011. ISBN 9781450306195.
L. van der Maaten and K. Weinberger. Stochastic triplet embedding. In 2012 IEEE International Workshop on Machine Learning for Signal Processing, pages 1-6, 2012. doi: 10.1109/MLSP.2012.6349720.
H. Wang, N. Grgic-Hlaca, P. Lahoti, K. P. Gummadi, and A. Weller. An empirical study on learning fairness metrics for compas data with human supervision, 2019. URL https://arxiv.org/abs/1910.10255.
M. Wilber, I. Kwak, and S. Belongie. Cost-effective hits for relative similarity comparisons. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 2(1):227-233, Sep. 2014. URL https://ojs.aaai.org/index.php/HCOMP/article/view/13152.
I. Yeh and C.-H. Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36:2473-2480, 03 2009. doi: 10.1016/j.eswa.2007.12.020.
G. Yona and G. Rothblum. Probably approximately metric-fair learning. In International Conference on Machine Learning, pages 5680-5688. PMLR, 2018.
Proof of Theorem 4.2. For two distinct groups  $\ell$  and  $\ell'$ , consider what will happen when we are asked to compare some  $x \in \mathcal{D}_{\ell}$  and  $y \in \mathcal{D}_{\ell'}$ . Properties  $\mathcal{M}_1$  and  $\mathcal{M}_2$  of  $\sigma_{\ell, \ell'}$  imply
$$
Q \leq \sigma_ {\ell , \ell^ {\prime}} (x, y) \leq P, \text {w h e r e}
$$
$$
P := d _ {\ell} (x, \pi (x)) + \sigma_ {\ell , \ell^ {\prime}} (\pi (x), \pi (y)) + d _ {\ell^ {\prime}} (y, \pi (y))
$$
$$
Q := \sigma_ {\ell , \ell^ {\prime}} (\pi (x), \pi (y)) - d _ {\ell} (x, \pi (x)) - d _ {\ell^ {\prime}} (y, \pi (y))
$$
Note that when  $d_{\ell}(x,\pi (x))\leq 3\epsilon$  and  $d_{\ell^{\prime}}(y,\pi (y))\leq 3\epsilon$ , the above inequalities and the definition of  $f_{\ell ,\ell^{\prime}}(x,y)$  yield  $|f_{\ell ,\ell^{\prime}}(x,y) - \sigma_{\ell ,\ell^{\prime}}(x,y)|\leq 6\epsilon$ . Thus, we just need upper bounds for  $\mathcal{A}:= \operatorname*{Pr}_{S_{\ell},x\sim \mathcal{D}_{\ell}}[\forall x^{\prime}\in S_{\ell}:d(x,x^{\prime}) > 3\epsilon ]$  and  $\mathcal{B}:= \operatorname*{Pr}_{S_{\ell^{\prime}},y\sim \mathcal{D}_{\ell^{\prime}}}[\forall y^{\prime}\in S_{\ell}:d(y,y^{\prime}) > 3\epsilon ]$ , since the previous analysis and a union bound give  $\operatorname*{Pr}[\operatorname{Error}_{(\ell ,\ell ')}]\leq \mathcal{A} + \mathcal{B}$ . In what follows we present an upper bound for  $\mathcal{A}$ . The same analysis gives an identical bound for  $\mathcal{B}$ .
Before we proceed to the rest of the proof, we have to provide an existential construction. For the sake of simplicity we will be using the term dense for elements of  $\mathcal{D}_{\ell}$  that are not  $(\epsilon, \delta)$ -rare. For every  $x \in \mathcal{D}_{\ell}$  that is dense, we define  $B_{x} := \{x' \in \mathcal{D}_{\ell} : d_{\ell}(x, x') \leq \epsilon\}$ . Observe that the definition of dense elements implies  $\operatorname{Pr}_{x' \sim \mathcal{D}_{\ell}}[x' \in B_{x}] \geq \delta$  for every dense  $x$ . Next, consider the following process. We start with an empty set  $\mathcal{R} = \{\}$ , and we assume that all dense elements are unmarked. Then, we choose an arbitrary unmarked dense element  $x$ , and we place it in the set  $\mathcal{R}$ . Further, for every dense  $x' \in \mathcal{D}_{\ell}$  that is unmarked and has  $B_{x} \cap B_{x'} \neq \emptyset$ , we mark  $x'$  and set  $\psi(x') = x$ . Here the function  $\psi$  maps dense elements to elements of  $\mathcal{R}$ . We continue this picking process until all dense elements have been marked. Since  $B_{z} \cap B_{z'} = \emptyset$  for any two  $z, z' \in \mathcal{R}$  and  $\operatorname{Pr}_{x' \sim \mathcal{D}_{\ell}}[x' \in B_{z}] \geq \delta$  for  $z \in \mathcal{R}$ , we have  $|\mathcal{R}| \leq 1/\delta$ . Also, for every dense  $x$  we have  $d_{\ell}(x, \psi(x)) \leq 2\epsilon$  due to  $B_{x} \cap B_{\psi(x)} \neq \emptyset$ .
Now we are ready to upper bound  $\mathcal{A}$ .
$$
\begin{array}{l} \mathcal {C} := \Pr_ {S _ {\ell}, x \sim \mathcal {D} _ {\ell}} \left[ \forall x ^ {\prime} \in S _ {\ell}: d (x, x ^ {\prime}) > 3 \epsilon \wedge x \text {i s} (\epsilon , \delta) \text {- r a r e} \right] \\ \leq \Pr_ {x \sim \mathcal {D} _ {\ell}} [ x \text {i s} (\epsilon , \delta) \text {- r a r e} ] = p _ {\ell} (\epsilon , \delta) \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {D} := \Pr_ {S _ {\ell}, x \sim \mathcal {D} _ {\ell}} [ \forall x ^ {\prime} \in S _ {\ell}: d (x, x ^ {\prime}) > 3 \epsilon \wedge x \text {i s d e n s e} ] \\ \leq \Pr_ {S _ {\ell}} [ \exists r \in \mathcal {R}: B _ {r} \cap S _ {\ell} = \emptyset ] \\ \leq \sum_ {r \in \mathcal {R}} \Pr_ {S _ {\ell}} \left[ B _ {r} \cap S _ {\ell} = \emptyset \right] \leq | \mathcal {R} | (1 - \delta) ^ {| S _ {\ell} |} \leq | \mathcal {R} | e ^ {- \delta | S _ {\ell} |} \leq \delta \\ \end{array}
$$
The upper bound for  $\mathcal{C}$  is trivial. We next explain the computations for  $\mathcal{D}$ . For the transition between the first and the second line we use a proof by contradiction. Hence, suppose that  $S_{\ell} \cap B_r \neq \emptyset$  for every  $r \in \mathcal{R}$ , and let  $i_r$  denote an arbitrary element of  $S_{\ell} \cap B_r$ . Then, for any dense element  $x \in \mathcal{D}_{\ell}$  we have  $d_{\ell}(x, \pi(x)) \leq d_{\ell}(x, i_{\psi(x)}) \leq d_{\ell}(x, \psi(x)) + d_{\ell}(\psi(x), i_{\psi(x)}) \leq 2\epsilon + \epsilon = 3\epsilon$ . Back to the computations for  $\mathcal{D}$ , to get the third line we simply used a union bound. To get from the third to the fourth line, we used the definition of  $r \in \mathcal{R}$  as a dense element, which implies that the probability of sampling any element of  $B_r$  in one try is at least  $\delta$ . The final bound is a result of numerical calculations using  $|\mathcal{R}| \leq \frac{1}{\delta}$  and  $|S_{\ell}| = \frac{1}{\delta} \log \frac{1}{\delta^2}$ .
To conclude the proof, observe that  $\mathcal{A} = \mathcal{C} + \mathcal{D}$ , and using a similar reasoning as the one in upper-bounding  $\mathcal{A}$  we also get  $\mathcal{B} \leq p_{\ell'}(\epsilon, \delta) + \delta$ .
Proof of Theorem 2.3. Given any  $\epsilon, \delta$ , consider the following instance of the problem. We have two groups represented by the distributions  $\mathcal{D}_1$  and  $\mathcal{D}_2$ . For the first group we have only one element belonging to it, and let that element be  $x$ . In other words, every time we draw an element from  $\mathcal{D}_1$  that element turns out to be  $x$ , i.e.,  $\operatorname*{Pr}_{x' \sim \mathcal{D}_1}[x' = x] = 1$ . For the second group we have that every  $y \in \mathcal{D}_2$  appears with probability  $\frac{1}{|\mathcal{D}_2|}$ , and  $|\mathcal{D}_2|$  is a huge constant  $c \gg 0$ , with  $\frac{1}{c} \ll \delta$ .
Now we define all similarity values. At first, the similarity function for  $\mathcal{D}_1$  will trivially be  $d_1(x,x) = 0$ . For the second group, for every distinct  $y,y^{\prime}\in \mathcal{D}_{2}$  we define  $d_{2}(y,y^{\prime}) = 1$ . Obviously, for every  $y\in \mathcal{D}_2$  we set  $d_2(y,y) = 0$ . Observe that  $d_{1}$  and  $d_{2}$  are metric functions for their respective groups. As for the across-groups similarities, each  $\sigma (x,y)$  for  $y\in \mathcal{D}_2$  is chosen independently, and it is
drawn uniformly at random from  $[0,1]$ . Note that this choice of  $\sigma$  satisfies the necessary metric-like properties  $\mathcal{M}_1$  and  $\mathcal{M}_2$  that were introduced in Section 2.
Further, since  $\epsilon ,\delta \in (0,1)$  , any  $y\in \mathcal{D}_2$  will be  $(\epsilon ,\delta)$  -rare:
$$
\operatorname * {P r} _ {y ^ {\prime} \sim \mathcal {D} _ {2}} \left[ d _ {2} \left(y, y ^ {\prime}\right) \leq \epsilon \right] = \operatorname * {P r} _ {y ^ {\prime} \sim \mathcal {D} _ {2}} \left[ y ^ {\prime} = y \right] = \frac {1}{| \mathcal {D} _ {2} |} <   \delta
$$
The first equality is because the only element within distance  $\epsilon$  from  $y$  is  $y$  itself. The last inequality is because  $\frac{1}{|\mathcal{D}_2|} < \delta$ . Therefore, since all elements are  $(\epsilon, \delta)$ -rare, we have  $p_2(\epsilon, \delta) = 1$ .
Consider now any learning algorithm that produces an estimate function  $f$ . For any  $y \in \mathcal{D}_2$ , let us try to analyze the probability of having  $|f(x,y) - \sigma (x,y)| = \omega (\epsilon)$ . At first, note that when  $y \in S_2$ , we can always get the exact value  $f(x,y)$ , since  $x$  will always be in  $S_{1}$ . The probability of having  $y \in S_{2}$  is  $1 - (1 - 1 / |\mathcal{D}_2|)^N$ , where  $N$  is the number of used samples. Since we have control over  $|\mathcal{D}_2|$  when constructing this instance, we can always set it to a large enough value that will give  $1 - (1 - 1 / |\mathcal{D}_2|)^N = \epsilon$ ; note that this is possible because  $1 - (1 - 1 / |\mathcal{D}_2|)^N$  is decreasing in  $|\mathcal{D}_2|$  and  $\lim_{|\mathcal{D}_2| \to \infty} (1 - (1 - 1 / |\mathcal{D}_2|)^N) = 0$ . Hence,
$$
\Pr_ {y} [ | f (x, y) - \sigma (x, y) | = \omega (\epsilon) ] = (1 - \epsilon) \Pr_ {y} [ | f (x, y) - \sigma (x, y) | = \omega (\epsilon) \mid y \notin S _ {2} ] + \epsilon \tag {3}
$$
When  $y$  will not be among the samples, the algorithm needs to learn  $\sigma(x,y)$  via some other value  $\sigma(x,y')$ , for  $y'$  being a sampled element of the second group. However, due to the construction of  $\sigma$  the values  $\sigma(x,y)$  and  $\sigma(x,y')$  are independent. This means that knowledge of any  $\sigma(x,y')$  (with  $y \neq y'$ ) provides no information at all on  $\sigma(x,y)$ . Thus, the best any algorithm can do is guess  $f(x,y)$  uniformly at random from  $[0,1]$ . This yields  $\operatorname*{Pr}[|f(x,y) - \sigma(x,y)| = \omega(\epsilon) \mid y \notin S_2] = 1 - \operatorname*{Pr}[|f(x,y) - \sigma(x,y)| = O(\epsilon) \mid y \notin S_2] = 1 - O(\epsilon) = p_2(\epsilon, \delta) - O(\epsilon)$ . Combining this with (3) gives the desired result.
Proof of Theorem 4.3. For two distinct groups  $\ell$  and  $\ell'$ , consider comparing some  $x \in \mathcal{D}_{\ell}$  and  $y \in \mathcal{D}_{\ell'}$ . To begin with, let us assume that  $d_{\ell}(x, \pi(x)) \leq 3\epsilon$  and  $d_{\ell'}(y, \pi(y)) \leq 3\epsilon$ . Furthermore, the execution of the algorithm implies  $H_{\pi(x)}^{\ell} \cap H_{r_{\ell}(\pi(x))}^{\ell} \neq \emptyset$ , and thus the triangle inequality and the definitions of the sets  $H_{\pi(x)}^{\ell}, H_{r_{\ell}(\pi(x))}^{\ell}$  give  $d_{\ell}(\pi(x), r_{\ell}(\pi(x))) \leq 16\epsilon$ . Similarly  $d_{\ell'}(\pi(y), r_{\ell'}(\pi(y))) \leq 16\epsilon$ . Eventually:
$$
d _ {\ell} (x, r _ {\ell} (\pi (x))) \leq d _ {\ell} (x, \pi (x)) + d _ {\ell} (\pi (x), r _ {\ell} (\pi (x))) \leq 1 9 \epsilon
$$
$$
d _ {\ell^ {\prime}} (y, r _ {\ell^ {\prime}} (\pi (y))) \leq d _ {\ell^ {\prime}} (y, \pi (y)) + d _ {\ell^ {\prime}} (\pi (y), r _ {\ell^ {\prime}} (\pi (y))) \leq 1 9 \epsilon
$$
For notational convenience, let  $A \coloneqq d_{\ell}(x,r_{\ell}(\pi (x)))$  and  $B \coloneqq d_{\ell^{\prime}}(y,r_{\ell^{\prime}}(\pi (y)))$ . Then, the metric properties  $\mathcal{M}_1$  and  $\mathcal{M}_2$  of  $\sigma_{\ell ,\ell^{\prime}}$  and the definition of  $f_{\ell ,\ell^{\prime}}(x,y)$  yield
$$
\left| \sigma_ {\ell , \ell^ {\prime}} (x, y) - f _ {\ell , \ell^ {\prime}} (x, y) \right| \leq A + B \leq 3 8 \epsilon
$$
Overall, we proved that when  $d_{\ell}(x,\pi (x)) \leq 3\epsilon$  and  $d_{\ell^{\prime}}(y,\pi (y)) \leq 3\epsilon$ , we have  $|f_{\ell ,\ell^{\prime}}(x,y) - \sigma_{\ell ,\ell^{\prime}}(x,y)| \leq 38\epsilon$ . Finally, as shown in the proof of Theorem 4.2, the probability of not having  $d_{\ell}(x,\pi (x)) \leq 3\epsilon$  and  $d_{\ell^{\prime}}(y,\pi (y)) \leq 3\epsilon$ , is at most  $2\delta +p_{\ell}(\epsilon ,\delta) + p_{\ell^{\prime}}(\epsilon ,\delta)$ .
Proof of Lemma 4.5. Consider a group  $\ell$ , and let  $C^*$  be its optimal solution for the problem of Definition 4.4. We first claim that each  $\mathcal{H}_c^\ell$  with  $c \in C^*$  contains at most one element of  $R_\ell$ . This is due to the following. For any  $c \in C^*$ , we have  $d_\ell(z, z') \leq d_\ell(z, c) + d_\ell(c, z') \leq 8\epsilon$  for all  $z, z' \in \mathcal{H}_c^\ell$ . In addition, the construction of  $R_\ell$  trivially implies  $d_\ell(x, x') > 8\epsilon$  for all  $x, x' \in R_\ell$ . Thus, no two elements of  $R_\ell$  can be in the same  $\mathcal{H}_c^\ell$  with  $c \in C^*$ . Finally, since Definition 4.4 requires all  $x \in R_\ell$  to be covered, we have  $|R_\ell| \leq |C^*| = OPT_\ell$ .
Proof of Lemma 4.6. Consider a group  $\ell$ . Initially, through Definition 4.4 it is clear that  $OPT_{\ell} \leq |S_{\ell}| = N$ . For the second statement of the lemma we need to analyze  $OPT_{\ell}$  in a more clever way.
Recall the classification of elements  $x \in \mathcal{D}_{\ell}$  that was first introduced in the proof of Theorem 4.2. According to this, an element can either be  $(\epsilon, \delta)$ -rare, or dense. Now we will construct a solution  $C_{\ell}$  to the problem of Definition 4.4 as follows.
At first, let  $S_{\ell,r}$  be the set of  $(\epsilon, \delta)$ -rare elements of  $S_{\ell}$ . We will include all of  $S_{\ell,r}$  to  $C_{\ell}$ , so that all  $(\epsilon, \delta)$ -rare elements of  $S_{\ell}$  are covered by  $C_{\ell}$ . Further:
$$
\mathbb {E} \left[ | S _ {\ell , r} | \right] = p _ {\ell} (\epsilon , \delta) N \tag {4}
$$
Moving on, recall the construction shown in the proof of Theorem 4.2. According to that, there exists a set  $\mathcal{R}$  of at most  $\frac{1}{\delta}$  dense elements from  $\mathcal{D}_{\ell}$ , and a function  $\psi$  that maps every dense element  $x \in \mathcal{D}_{\ell}$  to an element  $\psi(x) \in \mathcal{R}$ , such that  $d_{\ell}(x, \psi(x)) \leq 2\epsilon$ . Let us now define for each  $x \in \mathcal{R}$  a set  $G_x := \{x' \in \mathcal{D}_{\ell} : x'$  is dense and  $\psi(x') = x\}$ , and note that  $d_{\ell}(z, z') \leq d_{\ell}(z, x) + d_{\ell}(z', x) \leq 4\epsilon$  for all  $z, z' \in G_x$ . Thus, for each  $x \in \mathcal{R}$  with  $G_x \cap S_{\ell} \neq \emptyset$ , we place in  $C_{\ell}$  an arbitrary  $y \in G_x \cap S_{\ell}$  and that  $y$  gets all of  $G_x \cap S_{\ell}$  covered. Finally, since the sets  $G_x$  induce a partition of the dense elements of  $\mathcal{D}_{\ell}$ ,  $C_{\ell}$  covers all dense elements of  $S_{\ell}$ .
Equation (4) and  $|\mathcal{R}|\leq \frac{1}{\delta}$  yield  $\mathbb{E}\big[|C_{\ell}|\big]\leq \frac{1}{\delta} +p_{\ell}(\epsilon ,\delta)N$  . Also, since  $C_\ell$  is shown to be a feasible solution for problem of Definition 4.4, we get
$$
O P T _ {\ell} \leq | C _ {\ell} | \Rightarrow \mathbb {E} [ O P T _ {\ell} ] \leq \frac {1}{\delta} + p _ {\ell} (\epsilon , \delta) N
$$
Proof of Theorem 2.5. We are given accuracy and confidence parameters  $\epsilon, \delta \in (0,1)$  respectively. For the sake of simplifying the exposition in the proof, let us assume that  $\frac{1}{\delta}$  is an integer; all later arguments can be generalized in order to handle the case of  $1 / \delta \notin \mathbb{N}$ .
We construct the following problem instance. We have two groups represented by the distributions  $\mathcal{D}_1$  and  $\mathcal{D}_2$ . In addition, for both of these groups we assume that the support of the corresponding distribution contains  $\frac{1}{\delta}$  elements, and  $\operatorname{Pr}_{x' \sim \mathcal{D}_1}[x = x'] = \delta$  for every  $x \in \mathcal{D}_1$  as well as  $\operatorname{Pr}_{y' \sim \mathcal{D}_2}[y = y'] = \delta$  for every  $y \in \mathcal{D}_2$ .
For every  $x, x' \in \mathcal{D}_1$  let  $d_1(x, x') = 1$ , and  $d_1(x, x) = 0$  for every  $x \in \mathcal{D}_1$ . Similarly, for every  $y, y' \in \mathcal{D}_2$  we set  $d_2(y, y') = 1$ , and for every  $y \in \mathcal{D}_2$  we set  $d_2(y, y) = 0$ . The functions  $d_1$  and  $d_2$  are clearly metrics. As for the across-groups similarity values, each  $\sigma(x, y)$  for  $x \in \mathcal{D}_1$  and  $y \in \mathcal{D}_2$  is chosen independently, and it is drawn uniformly at random from [0, 1]. Note that this choice of  $\sigma$  satisfies the necessary properties  $\mathcal{M}_1, \mathcal{M}_2$  introduced in Section 2.
In this proof we are also focusing on a more special learning model. In particular, we assume that the distributions  $\mathcal{D}_1$  and  $\mathcal{D}_2$  are known. Hence, there is no need for sampling. The only randomness here is over the random arrivals  $x\sim \mathcal{D}_1$  and  $y\sim \mathcal{D}_2$ , where  $x$  and  $y$  are the elements that need to be compared. Obviously, the similarity function  $\sigma$  would still remain unknown to any learner. Finally, the queries required for learning in this model cannot be more than the queries required in the original model, and this is because this model is a special case of the original.
Consider now an algorithm with the error guarantees mentioned in the Theorem statement, and focus on a fixed pair  $(x,y)$  with  $x\in \mathcal{D}_1$  and  $y\in \mathcal{D}_2$ . If the algorithm has queried the oracle for  $(x,y)$ , it knows  $\sigma (x,y)$  with absolute certainty. Let us study what happens when the algorithm has not queried the oracle for  $(x,y)$ . In this case, because the values  $\sigma (x',y')$  with  $x'\in \mathcal{D}_1$  and  $y'\in \mathcal{D}_2$  are independent, no query the algorithm has performed can provide any information for  $\sigma (x,y)$ . Thus, the best the algorithm can do is uniformly at random guess a value in  $[0,1]$ , and return that as the estimate for  $\sigma (x,y)$ . If  $\bar{\mathcal{Q}}_{x,y}$  denotes the event where no query is performed for  $(x,y)$ , then
$$
\begin{array}{l} \mathcal {P} := \Pr \left[ | f _ {\ell , \ell^ {\prime}} (x, y) - \sigma_ {\ell , \ell^ {\prime}} (x, y) | = \omega (\epsilon) \mid \bar {\mathcal {Q}} _ {x, y} \right] \\ = 1 - \Pr \left[ | f _ {\ell , \ell^ {\prime}} (x, y) - \sigma_ {\ell , \ell^ {\prime}} (x, y) | = O (\epsilon) \mid \bar {Q} _ {x, y} \right] \\ = 1 - O (\epsilon) \\ = \Omega (1) \\ \end{array}
$$
where the randomness comes only from the algorithm.
For the sake of contradiction, suppose the algorithm uses  $q = o(1 / \delta^2)$  queries. Since each  $(x,y)$  is equally likely to appear for a comparison, the overall error probability is
$$
\left(\frac {1 / \delta^ {2} - q}{1 / \delta^ {2}}\right) \cdot \mathcal {P} = \Omega (1) \cdot \Omega (1) = \Omega (1)
$$
Contradiction; the error probability was assumed to be  $O(\delta)$ .
Proof of Corollary 4.8. When every  $p_{\ell}(\epsilon, \delta)$  is very close to 0, Lemma 4.6 gives  $\mathbb{E}[OPT_{\ell}] \leq \frac{1}{\delta}$ . Thus, by inequality (1) the expected queries are  $\frac{\gamma(\gamma - 1)}{\delta^2}$ . Theorem 2.5 concludes the proof.
Theorem A.1. The function  $\sigma$  defined in Equation (2) is not metric, and satisfies properties  $\mathcal{M}_1, \mathcal{M}_2$  for  $d_1(x,y) = d_2(x,y) = \sqrt{\sum_{i\in[d]}(x_i - y_i)^2}$ , when  $\alpha_i, \beta_i \in [0,1]$  for all  $i \in [d]$ .
Proof. The easiest way to see that  $\sigma(x, y)$  is not a metric, is by realizing that it does not satisfy the symmetry property and also  $x = y$  does not necessarily imply  $\sigma(x, y) = 0$ . Both of these issues stem from the offsets  $\theta_i$ . To verify that symmetry is violated let  $x, y$  be 1-dimensional, and let  $x = 1, y = 2, \alpha = 1, \beta = 1, \theta = 2$ . Then  $\sigma(1, 2) = 1$ , while  $\sigma(2, 1) = 3$ . Next, let  $x = 1, y = 1, \alpha = 1, \beta = 1, \theta = 1$ . In this example, although  $x = y$  we have  $\sigma(x, y) = 1 \neq 0$ .
In the following we are going to show that  $\sigma$  satisfies  $\mathcal{M}_1$ . The proof for  $\mathcal{M}_1$  is identical. At first, take  $x,y,z\in \mathbb{R}^d$ , such that  $x,z\in \mathcal{D}_1$  and  $y\in \mathcal{D}_2$ . Then:
$$
\begin{array}{l} \sigma (x, y) = \sqrt [ 3 ]{\sum_ {i \in [ d ]} | \alpha_ {i} \cdot x _ {i} - \beta_ {i} \cdot y _ {i} + \theta_ {i} | ^ {3}} = \sqrt [ 3 ]{\sum_ {i \in [ d ]} | (\alpha_ {i} \cdot x _ {i} - \alpha_ {i} \cdot z _ {i}) + (\alpha_ {i} \cdot z _ {i} - \beta_ {i} \cdot y _ {i} + \theta_ {i}) | ^ {3}} \\ \leq \sqrt [ 3 ]{\sum_ {i \in [ d ]} | (\alpha_ {i} \cdot x _ {i} - \alpha_ {i} \cdot z _ {i}) | ^ {3}} + \sqrt [ 3 ]{\sum_ {i \in [ d ]} | (\alpha_ {i} \cdot z _ {i} - \beta_ {i} \cdot y _ {i} + \theta_ {i}) | ^ {3}} \\ = \sqrt [ 3 ]{\sum_ {i \in [ d ]} a _ {i} ^ {3} \left| x _ {i} - z _ {i} \right| ^ {3}} + \sigma (z, y) \\ \leq \sqrt [ 3 ]{\sum_ {i \in [ d ]} | x _ {i} - z _ {i} | ^ {3}} + \sigma (z, y) \\ \leq d _ {1} (x, z) + \sigma (z, y) \\ \end{array}
$$
To get the first inequality we used the triangle inequality for  $\ell_3$ . The second to last inequality is because  $\alpha_i^3 \leq 1$  for all  $i$ , and the last inequality is because the  $\ell_3$  norm of a vector is always smaller than its  $\ell_2$  norm.
Table 3: Error statistics for Adult
Algorithm	Average Relative Error %	SD of Relative Error %	Average (Absolute Error)/ε	SD of (Absolute Error)/ε
NAIVE	1.319	3.381	0.847	2.382
CLUSTER	1.321	3.383	0.849	2.383
MLP	4.119	6.092	2.306	1.977
RF	3.519	6.657	2.020	2.987
XGB	2.248	5.351	1.215	1.655
Table 4: Error statistics for Give Me Some Credit
Algorithm	Average Relative Error %	SD of Relative Error %	Average (Absolute Error)/ε	SD of (Absolute Error)/ε
NAIVE	1.630	4.106	2.644	7.710
CLUSTER	1.645	4.118	2.648	7.665
MLP	5.819	9.073	7.588	8.750
RF	5.692	12.404	7.258	34.508
XGB	5.614	13.355	6.516	7.754
(b) (Absolute Error) /  $\epsilon$
Figure 3: Empirical CDF for Adult
(a) Relative Error Percentage
Figure 4: Empirical CDF for Give Me Some Credit
(b) (Absolute Error) /  $\epsilon$