SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Regarding the difficulty in computing similarity between members of different groups, we are only aware of a brief result by Dwork et al. [2012]. In particular, given a metric  $d$  over the whole feature space, they mention that  $d$  can only be trusted for comparisons between elements of the same group, and not for across-groups comparisons. In order to achieve the latter for groups  $\ell$  and  $\ell'$ , they find a new similarity function  $d'$  that approximates  $d$ , while minimizing the Earthmover distance between the distributions  $\mathcal{D}_{\ell}, \mathcal{D}_{\ell'}$ . This is completely different from our work, since here we assume the existence of across-groups similarity values, which we eventually want to learn. On the other hand, the approach of Dwork et al. [2012] can be seen as an optimization problem, where the across-groups similarity values need to be computed in a way that minimizes some objective. Also, unlike our model, this optimization approach has a serious limitation, and that is requiring  $\mathcal{D}_{\ell}, \mathcal{D}_{\ell'}$  to be explicitly known (recall that here we only need samples from these distributions).
Finally, since similarity as distance is difficult to compute in practice, there has been a line of research that defines similarity using simpler, yet less expressive structures. Examples include similarity lists Chakrabarti et al. [2022], graphs Lahoti et al. [2019] and ordinal relationships Jung et al. [2019].
4 Theoretical results
All proofs for this section can be found in the supplementary material
4.1 A simple learning algorithm
Given any confidence and accuracy parameters  $\delta, \epsilon \in (0,1)$  respectively, our approach is summarized as follows. At first, for every group  $\ell$  we need a set  $S_{\ell}$  of samples that are chosen i.i.d. according to  $\mathcal{D}_{\ell}$ , such that  $|S_{\ell}| = \frac{1}{\delta} \log \frac{1}{\delta^2}$ . Then, for every distinct  $\ell$  and  $\ell'$ , and for all  $x \in S_{\ell}$  and  $y \in S_{\ell'}$ , we ask the expert oracle for the true similarity value  $\sigma_{\ell,\ell'}(x,y)$ . The next observation follows trivially.
Observation 4.1. The algorithm uses  $\frac{\gamma}{\delta}\log \frac{1}{\delta^2}$  samples, and  $\frac{\gamma(\gamma - 1)}{\delta^2}\log^2\frac{1}{\delta^2}$  queries to the oracle.
Suppose now that we need to compare any  $x \in \mathcal{D}_{\ell}$  and  $y \in \mathcal{D}_{\ell'}$ . Our high level idea is that the properties  $\mathcal{M}_1$  and  $\mathcal{M}_2$  of  $\sigma_{\ell, \ell'}$  (see Section 2), will actually allow us to use the closest element to  $x$  in  $S_{\ell}$  and the closest element to  $y$  in  $S_{\ell'}$  as proxies. Thus, let  $\pi(x) = \arg \min_{x' \in S_{\ell}} d_{\ell}(x, x')$  and  $\pi(y) = \arg \min_{y' \in S_{\ell'}} d_{\ell'}(y, y')$ . The algorithm then sets
$$
f _ {\ell , \ell^ {\prime}} (x, y) := \sigma_ {\ell , \ell^ {\prime}} (\pi (x), \pi (y))
$$
where  $\sigma_{\ell, \ell'}(\pi(x), \pi(y))$  is known from the earlier queries.
Before we proceed with the analysis of the algorithm, we need to recall some notation which was introduced in Section 2.1. Consider any group  $\ell$ . An element  $x \in \mathcal{D}_{\ell}$  with  $\operatorname{Pr}_{x' \sim \mathcal{D}_{\ell}}[d_{\ell}(x,x') \leq \epsilon] < \delta$  is called an  $(\epsilon, \delta)$ -rare element of  $\mathcal{D}_{\ell}$ , and also  $p_{\ell}(\epsilon, \delta) \coloneqq \operatorname{Pr}_{x \sim \mathcal{D}_{\ell}}[x$  is  $(\epsilon, \delta)$ -rare for  $\mathcal{D}_{\ell}$ .
Theorem 4.2. For any given parameters  $\epsilon, \delta \in (0,1)$ , the simple algorithm produces similarity approximation functions  $f_{\ell,\ell'}$  for every  $\ell$  and  $\ell'$ , such that
where  $\operatorname*{Pr}[Error_{(\ell,\ell')}]$  is as in Theorem 2.1.
Observation 4.1 and Theorem 4.2 directly yield Theorem 2.1.
A potential criticism of the algorithm presented here, is that its error probabilities depend on  $p_{\ell}(\epsilon, \delta)$ . However, Theorem 2.3 shows that such a dependence is unavoidable (see appendix for proof).
4.2 Optimizing the number of expert queries
Here we modify the earlier algorithm in a way that improves the number of queries used. The idea behind this improvement is the following. Given the sets of samples  $S_{\ell}$ , instead of asking the oracle for all possible similarity values  $\sigma_{\ell, \ell'}(x, y)$  for every  $\ell, \ell'$  and every  $x \in S_{\ell}$  and  $y \in S_{\ell'}$ , we would rather choose a set  $R_{\ell} \subseteq S_{\ell}$  of representative elements for each group  $\ell$ . Then, we would ask the oracle for the values  $\sigma_{\ell, \ell'}(x, y)$  for every  $\ell, \ell'$ , but this time only for every  $x \in R_{\ell}$  and  $y \in R_{\ell'}$ . The choice of the representatives is inspired by the  $k$ -center algorithm of Hochbaum and Shmoys [1985]. Intuitively, the representatives  $R_{\ell}$  of group  $\ell$  will serve as similarity proxies for the elements of  $S_{\ell}$ , such that each  $x \in S_{\ell}$  is assigned to a nearby  $r_{\ell}(x) \in R_{\ell}$  via a mapping function  $r_{\ell}: S_{\ell} \mapsto R_{\ell}$ . Hence, if  $d_{\ell}(x, r_{\ell}(x))$  is small enough,  $x$  and  $r_{\ell}(x)$  are highly similar, and thus  $r_{\ell}(x)$  acts as a good approximation of  $x$ . The full details for the construction of  $R_{\ell}, r_{\ell}$  are presented in Algorithm 1.
Suppose now that we need to compare some  $x \in \mathcal{D}_{\ell}$  and  $y \in \mathcal{D}_{\ell'}$ . Our approach will be almost identical to that of Section 4.1. Once again, let  $\pi(x) = \arg \min_{x' \in S_{\ell}} d_{\ell}(x, x')$  and  $\pi(y) = \arg \min_{y' \in S_{\ell'}} d_{\ell'}(y, y')$ . However, unlike the simple algorithm of Section 4.1 that directly uses  $\pi(x)$  and  $\pi(y)$ , the more intricate algorithm here will rather use their proxies  $r_{\ell}(\pi(x))$  and  $r_{\ell'}(\pi(y))$ . Our prediction will then be
$$
f _ {\ell , \ell^ {\prime}} (x, y) := \sigma_ {\ell , \ell^ {\prime}} \left(r _ {\ell} (\pi (x)), r _ {\ell^ {\prime}} (\pi (y))\right)
$$
where  $\sigma_{\ell,\ell'}(r_{\ell}(\pi(x)), r_{\ell'}(\pi(y)))$  is known from the earlier queries.
Theorem 4.3. For any given parameters  $\epsilon, \delta \in (0,1)$ , the new query optimization algorithm produces similarity approximation functions  $f_{\ell,\ell'}$  for every  $\ell$  and  $\ell'$ , such that
where  $\operatorname*{Pr}[Error_{(\ell ,\ell ')}]$  is as in Theorem 2.1.
Since the number of samples used by the algorithm is easily seen to be  $\frac{\gamma}{\delta} \log \frac{1}{\delta^2}$ , the only thing left in order to prove Theorem 2.4 is analyzing the number of oracle queries. To that end, for every group  $\ell \in [\gamma]$  with its sampled set  $S_\ell$ , we define the following Set Cover problem.
Algorithm 1 Training Phase
Accuracy and confidence parameters  $\epsilon, \delta$ . For every group  $\ell \in [\gamma]$ , a set  $S_{\ell}$  of i.i.d. samples chosen according to  $\mathcal{D}_{\ell}$ , such that  $|S_{\ell}| = \frac{1}{\delta} \log \frac{1}{\delta^2}$ .
1: for each  $\ell \in [\gamma]$  do
2:  $H_{x}^{\ell}\gets \{x^{\prime}\in S_{\ell}:d_{\ell}(x,x^{\prime})\leq 8\epsilon \}$  for each  $x\in S_{\ell}$
3:  $U\gets S_{\ell},R_{\ell}\gets \emptyset$  and  $r_\ell (x)\gets x$  for each  $x\in S_{\ell}$
4: while  $U\neq \emptyset$  do
5: Choose an arbitrary  $x \in U$ .
6: Set  $R_{\ell}\gets R_{\ell}\cup \{x\}$
7:  $W_{x}\gets \{x^{\prime}\in U:H_{x}^{\ell}\cap H_{x^{\prime}}^{\ell}\neq \emptyset \}$
8:  $r_{\ell}(x') \gets x$  for every  $x' \in W_x$ .
9:  $U\gets U\setminus W_{x}.$
10: end while
11: end for
12: For every distinct  $\ell$  and  $\ell'$ , and for every  $x \in R_{\ell}$  and  $y \in R_{\ell'}$ , ask the oracle for  $\sigma_{\ell, \ell'}(x, y)$  and store all these values.
13: For every  $\ell$ , return the set  $R_{\ell}$  and the function  $r_{\ell}$ .
Definition 4.4. Let  $\mathcal{H}_x^\ell \coloneqq \{x' \in S_\ell : d_\ell(x, x') \leq 4\epsilon\}$  for all  $x \in S_\ell$ . Find  $C \subseteq S_\ell$  minimizing  $|C|$ , with  $\bigcup_{c \in C} \mathcal{H}_c^\ell = S_\ell$ . We use  $OPT_\ell$  to denote the optimal value of this problem. Using standard terminology, we say  $x \in S_\ell$  is covered by  $C$  if  $x \in \bigcup_{c \in C} \mathcal{H}_c^\ell$ , and  $C$  is feasible if it covers all  $x \in S_\ell$ .
Lemma 4.5. For every  $\ell \in [\gamma]$  we have  $|R_{\ell}| \leq OPT_{\ell}$ .
Lemma 4.6. Let  $N = \frac{1}{\delta}\log \frac{1}{\delta^2}$ . For each group  $\ell \in [\gamma ]$  we have  $OPT_{\ell}\leq N$  with probability 1, and  $\mathbb{E}[OPT_{\ell}]\leq \frac{1}{\delta} +p_{\ell}(\epsilon ,\delta)N$ . The randomness here is over the samples  $S_{\ell}$ .
The proof of Theorem 2.4 follows since all pairwise queries for the elements in the sets  $R_{\ell}$  are
$$
\sum_ {\ell} \left(| R _ {\ell} | \sum_ {\ell^ {\prime} \neq \ell} | R _ {\ell^ {\prime}} |\right) \leq \sum_ {\ell} \left(O P T _ {\ell} \sum_ {\ell^ {\prime} \neq \ell} O P T _ {\ell^ {\prime}}\right) \tag {1}
$$
where the inequality follows from Lemma 4.5. Combining (1) and Lemma 4.6 proves Theorem 2.4.
Remark 4.7. The factor 8 in the definition of  $H_{x}^{\ell}$  at line 2 of Algorithm 1 is arbitrary. Actually, any factor  $\rho = O(1)$  would yield the same asymptotic guarantees, with any changes in accuracy and queries being only of an  $O(1)$  order of magnitude. Specifically, the smaller  $\rho$  is, the better the achieved accuracy and the more queries we are using.
Finally, we are interested in lower bounds on the queries required for learning. Thus, we give Theorem 2.5, showing that any algorithm with accuracy  $O(\epsilon)$  and confidence  $O(\delta)$  needs  $\Omega (\gamma^2 /\delta^2)$  queries. This immediately leads to the following corollary characterizing the optimality of our algorithm.
Corollary 4.8. When for every  $\ell \in [\gamma]$  the value  $p_{\ell}(\epsilon, \delta)$  is arbitrarily close to 0, the algorithm presented in this section achieves an expected number of oracle queries that is asymptotically optimal.
5 Experimental evaluation
We implemented all algorithms in Python 3.10.6 and ran our experiments on a personal laptop with Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz 2.90 GHz and 16.0 GB memory.
Algorithms: We implemented the simple algorithm from Section 4.1, and the more intricate algorithm of Section 4.2. We refer to the former as NAIVE, and to the latter as CLUSTER. For the training phase of CLUSTER, we set the dilation factor at line 2 of Algorithm 1 to 2 instead of 8. The reason for this, is that a minimal experimental investigation revealed that this choice leads to a good balance between accuracy guarantees and oracle queries. As explained in Section 3, neither the existing similarity learning algorithms [Ilvento, 2019, Mukherjee et al., 2020, Wang et al., 2019] nor the Earthmover minimization approach of Dwork et al. [2012] address the problem of finding similarity values for heterogeneous data. Furthermore, if the across-groups functions  $\sigma$  are not metric, then no approach from the metric learning literature can be used. Hence, as baselines we used three general regression
models; an MLP, a Random Forest regressor (RF) and an XGBoost regressor (XGB). The MLP uses 4 hidden layers of 32 relu activation nodes, and both RF and XGB use 200 estimators.
Number of demographic groups: All our experiments are performed for two groups, i.e.,  $\gamma = 2$ . The following reasons justify this decision. At first, this case captures the essence of our algorithmic results; the  $\gamma > 2$  case can be viewed as running the algorithm for  $\gamma = 2$  multiple times, one for each pair of groups. Secondly, as the theoretical guarantees suggest, the achieved confidence and accuracy of our algorithms are completely independent of  $\gamma$ .
Similarity functions: In all our experiments the feature space is  $\mathbb{R}^d$ , where  $d \in \mathbb{N}$  is case-specific. In line with our motivation which assumes that the intra-group similarity functions are simple, we define  $d_1$  and  $d_2$  to be the Euclidean distance. Specifically, for  $\ell \in \{1, 2\}$ , the similarity between any  $x, y \in \mathcal{D}_{\ell}$  is given by
$$
d _ {\ell} (x, y) = \sqrt {\sum_ {i \in [ d ]} (x _ {i} - y _ {i}) ^ {2}}
$$
For the across-groups similarities, we aim for a difficult to learn non-metric function; we purposefully chose a non-trivial function in order to challenge both our algorithms and the baselines. Namely, for any  $x \in \mathcal{D}_1$  and  $y \in \mathcal{D}_2$ , we assume
$$
\sigma (x, y) = \sqrt [ 3 ]{\sum_ {i \in [ d ]} \left| \alpha_ {i} \cdot x _ {i} - \beta_ {i} \cdot y _ {i} + \theta_ {i} \right| ^ {3}} \tag {2}
$$
where the vectors  $\alpha, \beta, \theta \in \mathbb{R}^d$  are basically the hidden parameters to be learned (of course non of the learners we use has any insight on the specific structure of  $\sigma$ ).
The function  $\sigma$  implicitly adopts a paradigm of feature importance [Niño-Adan et al., 2021]. Specifically, when  $x$  and  $y$  are to be compared, their features are scaled accordingly by the expert using the parameters  $\alpha, \beta$ , while some offsetting via  $\theta$  might also be necessary. For example, in the college admissions use-case, certain features may have to be properly adjusted (increase a feature for a non-privileged student and decrease it for the privileged one). In the end, the similarity is calculated in an  $\ell_3$ -like manner. To see why  $\sigma$  is not a metric and why it satisfies the necessary properties  $\mathcal{M}_1$  and  $\mathcal{M}_2$  from Section 2, refer to the last theorem in the Appendix.
In each experiment we choose all  $\alpha_{i},\beta_{i},\theta_{i}$  independently. The values  $\alpha_{i},\beta_{i}$  are chosen uniformly at random from [0, 1], while the  $\theta_{i}$  are chosen uniformly at random from  $[-0.01,0.01]$ . Obviously, the algorithms do not have access to the vectors  $\alpha ,\beta ,\theta$  which are only used to simulate the oracle and compare our predictions with the corresponding true values.
Datasets: We used 2 datasets from the UCI ML Repository, namely Adult (48,842 points - 14 features) [Kohavi, 1996] and Credit Card Default (30,000 points - 23 features) [Yeh and Lien, 2009], and the publicly available Give Me Some Credit dataset (150,000 points - 11 features) [Credit Fusion, 2011]. We chose these datasets because this type of data is frequently used in applications of issuing credit scores, and in such cases fairness considerations are of utmost importance. For Adult, where categorical features are not encoded as integers, we assigned each category to an integer in  $\{1,\# \text{categories}\}$ , and this integer is used in place of the category in the feature vector [Ding et al., 2021]. Finally, in every dataset we standardized all features through a MinMax re-scaller. Due to space constraints, the figures for Adult and Give me Some Credit are moved to the supplementary material. However, the observed traits there the same as the ones shown here for Credit Card Default.
Choosing the two groups: For Credit Card Default and Adult, we defined groups based on marital status. Specifically, the first group corresponds to points that are married individuals, and the second to points that are not married (singles, divorced and widowed are merged together). In Give Me Some Credit, we partition individuals into two groups based on whether or not they have dependents.
Choosing the accuracy parameter  $\epsilon$ : To use a meaningful value for  $\epsilon$ , we need to know the order of magnitude of  $\sigma(x, y)$ . Thus, we calculated the value of  $\sigma(x, y)$  over 10,000 trials, where the randomness was of multiple factors, i.e., the random choices for the  $\alpha, \beta, \theta$ , and the sampling of  $x$  and  $y$ . Figure 1 shows histograms for the empirical frequency of  $\sigma(x, y)$  over the 10,000 runs. In addition, in those trials the minimum value of  $\sigma(x, y)$  observed was 1) 0.1149 for Credit Card Default, 2) 0.1155 for Adult, and 3) 0.0132 for Give Me Some Credit. Thus, aiming for an accuracy parameter that is at least an order of magnitude smaller than the value to be learned, we choose  $\epsilon = 0.01$  for Credit Card Default and Adult, and  $\epsilon = 0.001$  for Give Me Some Credit.
(a) Credit Card Default
(b) Adult
(c) Give Me Some Credit
Figure 1: Frequency counts for  $\sigma (x,y)$
Table 1: Error statistics for Credit Card Default
Algorithm	Average Relative Error %	SD of Relative Error %	Average (Absolute Error)/ε	SD of (Absolute Error)/ε
NAIVE	1.558	3.410	0.636	1.633
CLUSTER	1.593	3.391	0.646	1.626
MLP	3.671	4.275	1.465	1.599
RF	3.237	4.813	1.306	2.318
XGB	3.121	4.606	1.273	1.869
Confidence  $\delta$  and number of samples: All our experiments are performed with  $\delta = 0.001$ . In NAIVE and CLUSTER, we follow our theoretical results and sample  $N = \frac{1}{\delta} \log \frac{1}{\delta^2}$  points from each group. Choosing the training samples for the baselines is a bit more tricky. For these regression tasks a training point is a tuple  $(x, y, \sigma(x, y))$ . In other words, these tasks do not distinguish between queries and samples. To be as fair as possible when comparing against our algorithms, we provide the baselines with  $N + Q$  training points of the form  $(x, y, \sigma(x, y))$ , where  $Q$  is the maximum number of queries used in any of our algorithms.
Testing: We test our algorithms over 1,000 trials, where each trial consists of independently sampling two elements  $x, y$ , one for each group, and then inputting those to the predictors. We are interested in two metrics. The first is the relative error percentage; if  $p$  is the prediction for elements  $x, y$  and  $t$  is their true similarity value, the relative error percentage is  $100 \cdot |p - t| / t$ . The second metric we consider is the absolute error divided by  $\epsilon$ ; if  $p$  is the prediction for two elements and  $t$  is their true similarity value, this metric is  $|p - t| / \epsilon$ . We are interested in the latter metric because our theoretical guarantees are of the form  $|f(x, y) - \sigma(x, y)| = O(\epsilon)$ .
Table 1 shows the average relative and absolute value error, together with the standard deviation for these metrics across all 1,000 runs. It is clear that our algorithms dominate the baselines since they exhibit smaller errors with smaller standard deviations. In addition, NAIVE appears to have a tiny edge over CLUSTER, and this is expected (the queries of CLUSTER are a subset of NAIVE's). However, as shown in Table 2, CLUSTER leads to a significant decrease in oracle queries compared to NAIVE, thus justifying its superiority. To further demonstrate the statistical behavior of the errors, we present Figure 2. This depicts the empirical CDF of each error metric through a bar plot. Specifically, the height of each bar corresponds to the fraction of test instances whose error is at most the value in the x-axis directly underneath the bar. Once again, we see that our algorithms outperform the baselines, since their corresponding bars are always higher than those of the baselines.
This paper was prepared for informational purposes in part by the Artificial Intelligence Research group of JPMorganChase. and its affiliates ("JP Morgan"), and is not a product of the Research Department of JPMorganChase. JPMorganChase makes no representation and warranty whatsoever and
Table 2: Percent of decrease in queries when using CLUSTER instead of NAIVE
Credit Card Default	Adult	Give Me Some Credit
81.01%	80.40%	84.87%
(a) Relative Error Percentage
(b) (Absolute Error) /  $\epsilon$
Figure 2: Empirical CDF for Credit Card Default
disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.
A. Bellet, A. Habrard, and M. Sebban. A Survey on Metric Learning for Feature Vectors and Structured Data. Research report, Laboratoire Hubert Curien UMR 5516, 2013. URL https://hal.inria.fr/hal-01666935.
D. Chakrabarti, J. P. Dickerson, S. A. Esmaeili, A. Srinivasan, and L. Tsepenekas. A new notion of individually fair clustering:  $\alpha$ -equitable  $k$ -center. In G. Camps-Valls, F. J. R. Ruiz, and I. Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 6387-6408. PMLR, 28-30 Mar 2022.
W. C. Credit Fusion. Give me some credit, 2011. URL https://kaggle.com/competitions/GiveMeSomeCredit.
F. Ding, M. Hardt, J. Miller, and L. Schmidt. Retiring adult: New datasets for fair machine learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 6478-6490. Curran Associates, Inc., 2021.
C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS '12, 2012.
A. Frome, Y. Singer, F. Sha, and J. Malik. Learning globally-consistent local distance functions for shape-based image retrieval and classification. In 2007 IEEE 11th International Conference on Computer Vision, pages 1-8, 2007. doi: 10.1109/ICCV.2007.4408839.
D. S. Hochbaum and D. B. Shmoys. A best possible heuristic for the k-center problem. Mathematics of Operations Research, 10(2):180-184, 1985. ISSN 0364765X, 15265471. URL http://www.jstor.org/stable/3689371.
C. Ilvento. Metric learning for individual fairness, 2019. URL https://arxiv.org/abs/1906.00250.
K. G. Jamieson and R. D. Nowak. Low-dimensional embedding using adaptively selected ordinal data. In 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1077-1084, 2011. doi: 10.1109/Allerton.2011.6120287.