Title: Comparing Apples to Oranges: Learning Similarity Functions for Data Produced by Different Distributions
Venue: NeurIPS 2023
Accepted: yes
Keywords: individual fairness; similarity learning; active learning
Abstract: Similarity functions measure how comparable pairs of elements are, and play a key role in a wide variety of applications, e.g., notions of Individual Fairness abiding by the seminal paradigm of Dwork et al., as well as Clustering problems. However, access to an accurate similarity function should not always be considered guaranteed, and this point was even raised by Dwork et al. For instance, it is reasonable to assume that when the elements to be compared are produced by different distributions, or in other words belong to different ``demographic'' groups, knowledge of their true similarity might be very difficult to obtain. In this work, we present an efficient sampling framework that learns these across-groups similarity functions, using only a limited amount of experts' feedback. We show analytical results with rigorous theoretical bounds, and empirically validate our algorithms via a large suite of experiments.

Learning Similarity Functions for Data Produced by

Abstract

Similarity functions measure how comparable pairs of elements are, and play a key role in a wide variety of applications, e.g., notions of Individual Fairness abiding by the seminal paradigm of Dwork et al. [2012], as well as Clustering problems. However, access to an accurate similarity function should not always be considered guaranteed, and this point was even raised by Dwork et al. [2012]. For instance, it is reasonable to assume that when the elements to be compared are produced by different distributions, or in other words belong to different 'demographic' groups, knowledge of their true similarity might be very difficult to obtain. In this work, we present an efficient sampling framework that learns these across-groups similarity functions, using only a limited amount of experts' feedback. We show analytical results with rigorous theoretical bounds, and empirically validate our algorithms via a large suite of experiments.

1 Introduction

Given a feature space I , a similarity function σ : I 2 ↦→ R ≥ 0 measures how comparable any pair of elements x, x ′ ∈ I are. The function σ can also be interpreted as a distance function, where the smaller σ ( x, x ′ ) is, the more similar x and x ′ are. Such functions are crucially used in a variety of AI/ML problems, and in each such case σ is assumed to be known .

The most prominent applications where similarity functions have a central role involve considerations of individual fairness. Specifically, all such individual fairness paradigms stem from the the seminal work of Dwork et al. [2012], in which fairness is defined as treating similar individuals similarly . In more concrete terms, such paradigms interpret the aforementioned abstract definition of fairness as guaranteeing that for every pair of individuals x and y , the difference in the quality of service x and y receive (a.k.a. their received treatment) is upper bounded by their respective similarity value σ ( x, y ) ; the more similar the individuals are, the less different their quality of service will be. Therefore, any algorithm that needs to abide by such concepts of fairness, should always be able to access the similarity score for every pair of individuals that are of interest.

Another family of applications where similarity functions are vital, involves Clustering problems. In a clustering setting, e.g, the standard k -means task, the similarity function is interpreted as a distance

function, that serves as the metric space in which we need to create the appropriate clusters. Clearly, the aforementioned metric space is always assumed to be part of the input.

Nonetheless, it is not realistic to assume that a reliable and accurate similarity function is always given. This issue was even raised in the work of Dwork et al. [2012], where it was acknowledged that the computation of σ is not trivial, and thus should be deferred to third parties. The starting point of our work here is the observation that there exist scenarios where computing similarity can be assumed as easy (in other words given), while in other cases this task would be significantly more challenging. Specifically, we are interested in scenarios where there are multiple distributions that produce elements of I . We loosely call each such distribution a 'demographic' group, interpreting it as the stochastic way in which members of this group are produced. In this setting, computing the similarity value of two elements that are produced according to the same distribution, seems intuitively much easier compared to computing similarity values for elements belonging to different groups . We next present a few motivating examples that clarify this statement.

Individual Fairness: Consider a college admissions committee that needs access to an accurate similarity function for students, so that it provides a similar likelihood of acceptance to similar applicants. Let us focus on the following two demographic groups. The first being students from affluent families living in privileged communities and having access to the best quality schools and private tutoring. The other group would consist of students from low-income families, coming from a far less privileged background. Given this setting, the question at hand is 'Should two students with comparable feature vectors (e.g., SAT scores, strength of school curriculum, number of recommendation letters) be really viewed as similar, when they belong to different groups?' At first, it appears that directly comparing two students based on their features can be an accurate way to elicit their similarity only if the students belong to the same demographic (belonging to the same group serves as a normalization factor). However, this trivial approach might hurt less privileged students when they are compared to students of the first group. This is because such a simplistic way of measuring similarity does not reflect potential that is undeveloped due to unequal access to resources. Hence, accurate across-groups comparisons that take into account such delicate issues, appear considerably more intricate.

Clustering: Suppose that a marketing company has a collection of user data that wants to cluster, with its end goal being a downstream market segmentation analysis. However, as it is usually the case, the data might come from different sources, e.g., data from private vendors and data from government bureaus. In this scenario, each data source might have its own way of representing user information, e.g., each source might use a unique subset of features. Therefore, eliciting the distance metric required for the clustering task should be straightforward for data coming from the same source, while across-sources distances would certainly require extra care, e.g., how can one extract the distance of two vectors containing different sets of features?

As suggested by earlier work on computing similarity functions for applications of individual fairness [Ilvento, 2019], when obtaining similarity values is an overwhelming task, one can employ the advice of domain experts. Such experts can be given any pair of elements, and in return produce their true similarity value. However, utilizing this experts' advice can be thought of as very costly, and hence it should be used sparingly. For example, in the case of comparing students from different economic backgrounds, the admissions committee can reach out to regulatory bodies or civil rights organizations. Nonetheless, resorting to these experts for every student comparison that might arise, is clearly not a sustainable solution. Therefore, our goal in this paper is to learn the across-groups similarity functions, using as few queries to experts as possible.

2 Preliminaries and contribution

For the ease of exposition, we accompany the formal definitions with brief demonstrations on how they could relate to the previously mentioned college applications use-case.

Let I denote the feature space of elements; for instance each x ∈ I could correspond to a valid student profile. We assume that elements come from γ known 'demographic' groups, where γ ∈ N , and each group ℓ ∈ [ γ ] is governed by an unknown distribution D ℓ over I . We use x ∼ D ℓ to denote a randomly drawn x from D ℓ . Further, we use x ∈ D ℓ to denote that x is an element in the support of D ℓ , and thus x is a member of group ℓ . In the college admissions scenario where we have two demographic groups, there will be two distributions D 1 and D 2 dictating how the profiles of

̸

privileged and non-privileged students are respectively produced. Observe now that for a specific x ∈ I , we might have x ∈ D ℓ and x ∈ D ℓ ′ , for ℓ = ℓ ′ . Hence, in our model group membership is important, and every time we are considering an element x ∈ I , we know which distribution produced x , e.g., whether the profile x belongs to a privileged or non-privileged student.

For every group ℓ ∈ [ γ ] there is an intra-group similarity function d ℓ : I 2 ↦→ R ≥ 0 , such that for all x, y ∈ D ℓ we have d ℓ ( x, y ) representing the true similarity between x, y . In addition, the smaller d ℓ ( x, y ) is, the more similar x, y . Note here that the function d ℓ is only used to compare members of group ℓ (in the college admissions example, the function d 1 would only be used to compare privileged students with each other). Further, a common assumption for functions measuring similarity is that they are metric 1 [Yona and Rothblum, 2018, Kim et al., 2018, Ilvento, 2019, Mukherjee et al., 2020, Wang et al., 2019]. We also adopt the metric assumption for the function d ℓ . Finally, based on the earlier discussion regarding computing similarity between elements of the same group, we assume that d ℓ is known, and given as part of the instance.

Moreover, for any two groups ℓ and ℓ ′ there exists an unknown across-groups similarity function σ ℓ,ℓ ′ : I 2 ↦→ R ≥ 0 , such that for all x ∈ D ℓ and y ∈ D ℓ ′ , σ ℓ,ℓ ′ ( x, y ) represents the true similarity between x, y . Again, the smaller σ ℓ,ℓ ′ ( x, y ) is, the more similar the two elements, and for a meaningful use of σ ℓ,ℓ ′ we must make sure that x is a member of group ℓ and y a member of group ℓ ′ . In the college admissions scenario, σ 1 , 2 is the way you can accurately compare a privileged and a nonprivilaged student. Finally, to capture the metric nature of a similarity function, we impose the following mild properties on σ ℓ,ℓ ′ , which can be viewed as across-groups triangle inequalities:

Property M 1 : σ ℓ,ℓ ′ ( x, y ) ≤ d ℓ ( x, z ) + σ ℓ,ℓ ′ ( z, y ) for every x, z ∈ D ℓ and y ∈ D ℓ ′ .

Property M 2 : σ ℓ,ℓ ′ ( x, y ) ≤ σ ℓ,ℓ ′ ( x, z ) + d ℓ ′ ( z, y ) for every x ∈ D ℓ and y, z ∈ D ℓ ′ .

In terms of the college admissions use-case, M 1 and M 2 try to capture reasonable assumptions of the following form. If a non-privileged student x is similar to another non-privileged student z , and z is similar to a privileged student y , then x and y should also be similar to each other.

Observe now that the collection of all similarity values (intra-group and across-groups) in our model does not axiomatically yield a valid metric space. This is due to the following reasons. 1) If σ ℓ,ℓ ′ ( x, y ) = 0 for x ∈ D ℓ and y ∈ D ℓ ′ , then we do not necessarily have x = y . 2) It is not always the case that d ℓ ( x, y ) ≤ σ ℓ,ℓ ′ ( x, z ) + σ ℓ,ℓ ′ ( y, z ) for x, y ∈ D ℓ , z ∈ D ℓ ′ . 3) It is not always the case that σ ℓ,ℓ ′ ( x, y ) ≤ σ ℓ,ℓ ′′ ( x, z ) + σ ℓ ′′ ,ℓ ′ ( z, y ) for x ∈ D ℓ , y ∈ D ℓ ′ , z ∈ D ℓ ′′ .

However, not having the collection of similarity values necessarily produce a metric space is not a weakness of our model. On the contrary, we view this as one of its strongest aspects. For one thing, imposing a complete metric constraint on the case of intricate across-groups comparisons sounds unrealistic and very restrictive. Further, even though existing literature treats similarity functions as metric ones, the seminal work of Dwork et al. [2012] mentions that this should not always be the case. Hence, our model is more general than the current literature.

Goal of Our Problem: We want for any two groups ℓ, ℓ ′ to compute a function f ℓ,ℓ ′ : I 2 ↦→ R ≥ 0 , such that f ℓ,ℓ ′ ( x, y ) is our estimate of similarity for any x ∈ D ℓ and y ∈ D ℓ ′ . Specifically, we seek a PAC (Probably Approximately Correct) guarantee, where for any given accuracy and confidence parameters ϵ, δ ∈ (0 , 1) we have:

The subscript in the above probability corresponds to two independent random choices, one x ∼ D ℓ and one y ∼ D ℓ ′ . In other words, we want for any given pair our estimate to be ϵ -close to the real similarity value, with probability at least 1 -δ , where ϵ and δ are user-specified parameters.

As for tools to learn f ℓ,ℓ ′ , we only require two things. At first, for each group ℓ we want a set S ℓ of i.i.d. samples from D ℓ . Obviously, the total number of used samples should be polynomial in the input parameters, i.e., polynomial in γ, 1 ϵ and 1 δ . Secondly, we require access to an expert oracle, which given any x ∈ S ℓ and y ∈ S ℓ ′ for any ℓ and ℓ ′ , returns the true similarity value σ ℓ,ℓ ′ ( x, y ) . We refer to a single invocation of the oracle as a query. Since there is a cost to collecting expert feedback, an additional objective in our problem is minimizing the number of oracle queries.

2.1 Outline and discussion of our results

In Section 4 we present our theoretical results. We begin with a simple and very intuitive learning algorithm which achieves the following guarantees.

Theorem 2.1. For any given parameters ϵ, δ ∈ (0 , 1) , the simple algorithm produces a similarity approximation function f ℓ,ℓ ′ for every ℓ and ℓ ′ , such that:

The randomness here is of three independent sources. The internal randomness A of the algorithm, a choice x ∼ D ℓ , and a choice y ∼ D ℓ ′ . The algorithm requires 1 δ log 1 δ 2 samples from each group, and utilizes γ ( γ -1) δ 2 log 2 1 δ 2 oracle queries.

In plain English, the above theorem says that with a polynomial number of samples and queries, the algorithm achieves an O ( ϵ ) accuracy with high probability, i.e., with probability ω ( 1 -δ -p ℓ ( ϵ, δ ) -p ℓ ′ ( ϵ, δ ) ) . The definition of the functions p ℓ is presented next.

Definition 2.2. For each group ℓ , we use p ℓ ( ϵ, δ ) to denote the probability of sampling an ( ϵ, δ ) -rare element of D ℓ . We define as ( ϵ, δ ) -rare for D ℓ , an element x ∈ D ℓ for which there is a less than δ chance of sampling x ′ ∼ D ℓ with d ℓ ( x, x ′ ) ≤ ϵ . Formally, x ∈ D ℓ is ( ϵ, δ ) -rare iff Pr x ′ ∼D ℓ [ d ℓ ( x, x ′ ) ≤ ϵ ] < δ , and p ℓ ( ϵ, δ ) = Pr x ∼D ℓ [ x is ( ϵ, δ ) -rare for D ℓ ] . Intuitively, a rare element should be interpreted as an 'isolated' member of the group, in the sense that it is at most δ -likely to encounter another element that is ϵ -similar to it. For instance, a privileged student is considered isolated, if only a small fraction of other privileged students have a profile similar to them.

Clearly, to get a PAC guarantee where the algorithm's error probability for ℓ and ℓ ′ is O ( δ ) , we need p ℓ ( ϵ, δ ) , p ℓ ′ ( ϵ, δ ) = O ( δ ) . We hypothesize that in realistic distributions each p ℓ ( ϵ, δ ) should indeed be fairly small, and this hypothesis is actually validated by our experiments. The reason we believe this hypothesis to be true, is that very frequently real data demonstrate high concentration around certain archetypal elements. Hence, this sort of distributional density does not leave room for isolated elements in the rest of the space. Nonetheless, we also provide a strong no free lunch result for the values p ℓ ( ϵ, δ ) , which shows that any practical PAC-algorithm necessarily depends on them. This result further implies that our algorithm's error probabilities are indeed almost optimal .

Theorem 2.3 (No-Free Lunch Theorem) . For any given ϵ, δ ∈ (0 , 1) , any algorithm using finitely many samples, will yield similarity approximations f ℓ,ℓ ′ with Pr [ | f ℓ,ℓ ′ ( x, y ) -σ ℓ,ℓ ′ ( x, y ) | = ω ( ϵ ) ] = Ω(max { p ℓ ( ϵ, δ ) , p ℓ ′ ( ϵ, δ ) } -ϵ ) ; the probability is over the independent choices x ∼ D ℓ and y ∼ D ′ ℓ as well as any potential internal randomness of the algorithm.

In plain English, Theorem 2.3 says that any algorithm using a finite amount of samples, can achieve ϵ -accuracy with a probability that is necessarily at most 1 -max { p ℓ ( ϵ, δ ) , p ℓ ′ ( ϵ, δ ) } + ϵ , i.e., if max { p ℓ ( ϵ, δ ) , p ℓ ′ ( ϵ, δ ) } is large, learning is impossible.

Moving on, we focus on minimizing the oracle queries. By carefully modifying the earlier simple algorithm, we obtain a new more intricate algorithm with the following guarantees:

Theorem 2.4. For any given parameters ϵ, δ ∈ (0 , 1) , the query-minimizing algorithm produces similarity approximation functions f ℓ,ℓ ′ for every ℓ and ℓ ′ , such that:

Pr[ Error ( ℓ,ℓ ′ ) ] is as defined in Theorem 2.1. Let N = 1 δ log 1 δ 2 . The algorithm requires N samples from each group, and the number of oracle queries used is at most

̸

where Q ℓ ≤ N and E [ Q ℓ ] ≤ 1 δ + p ℓ ( ϵ, δ ) N for each ℓ .

At first, the confidence, accuracy and sample complexity guarantees of the new algorithm are the same as those of the simpler one described in Theorem 2.1. Furthermore, because Q ℓ ≤ N , the

queries of the improved algorithm are at most γ ( γ -1) N , which is exactly the number of queries in our earlier simple algorithm. However, the smaller the values p ℓ ( ϵ, δ ) are, the fewer queries in expectation. Our experimental results indeed confirm that the improved algorithm always leads to a significant decrease in the used queries.

Our final theoretical result involves a lower bound on the number of queries required for learning.

Theorem 2.5. For all ϵ, δ ∈ (0 , 1) , any learning algorithm producing similarity approximation functions f ℓ,ℓ ′ with Pr x ∼D ℓ ,y ∼D ℓ ′ [ | f ℓ,ℓ ′ ( x, y ) -σ ℓ,ℓ ′ ( x, y ) | = ω ( ϵ ) ] = O ( δ ) , needs Ω( γ 2 δ 2 ) queries.

Combining Theorems 2.4 and 2.5 implies that when all p ℓ ( ϵ, δ ) are negligible, i.e., p ℓ ( ϵ, δ ) → 0 , the expected queries of the Theorem 2.4 algorithm are asymptotically optimal.

Finally, Section 5 contains our experimental evaluation, where through a large suite of simulations we validate our theoretical findings.

3 Related work

Metric learning is a very well-studied area [Bellet et al., 2013, Kulis, 2013, Moutafis et al., 2017, Suárez-Díaz et al., 2018]. There is also an extensive amount of work on using human feedback for learning metrics in specific tasks, e.g., image similarity and low-dimensional embeddings [Frome et al., 2007, Jamieson and Nowak, 2011, Tamuz et al., 2011, van der Maaten and Weinberger, 2012, Wilber et al., 2014]. However, since these works are either tied to specific applications or specific metrics, they are only distantly related to ours.

Our model is more closely related to the literature on trying to learn the similarity function from the fairness definition of Dwork et al. [2012]. This concept of fairness requires treating similar individuals similarly. Thus, it needs access to a function that returns a non-negative value for any pair of individuals, and this value corresponds to how similar the individuals are. Specifically, the smaller the value the more similar the elements that are compared. Even though the fairness definition of Dwork et al. [2012] is very elegant and intuitive, the main obstacle for adopting it in practice is the inability to easily compute or access the crucial similarity function. To our knowledge, the only papers that attempt to learn this similarity function using expert oracles like us, are Ilvento [2019], Mukherjee et al. [2020] and Wang et al. [2019]. Ilvento [2019] addresses the scenario of learning a general metric function, and gives theoretical PAC guarantees. Mukherjee et al. [2020] give theoretical guarantees for learning similarity functions that are only of a specific Mahalanobis form. Wang et al. [2019] simply provide empirical results. The first difference between our model and these papers is that unlike us, they do not consider elements coming from multiple distributions. However, the most important difference is that these works only learn metric functions. In our case the collection of similarity values (from all d ℓ and σ ℓ,ℓ ′ ) does not necessarily yield a complete metric space ; see the discussion in Section 2. Hence, our problem addresses more general functions.

Regarding the difficulty in computing similarity between members of different groups, we are only aware of a brief result by Dwork et al. [2012]. In particular, given a metric d over the whole feature space, they mention that d can only be trusted for comparisons between elements of the same group, and not for across-groups comparisons. In order to achieve the latter for groups ℓ and ℓ ′ , they find a new similarity function d ′ that approximates d , while minimizing the Earthmover distance between the distributions D ℓ , D ℓ ′ . This is completely different from our work, since here we assume the existence of across-groups similarity values, which we eventually want to learn. On the other hand, the approach of Dwork et al. [2012] can be seen as an optimization problem, where the across-groups similarity values need to be computed in a way that minimizes some objective. Also, unlike our model, this optimization approach has a serious limitation, and that is requiring D ℓ , D ℓ ′ to be explicitly known (recall that here we only need samples from these distributions).

Finally, since similarity as distance is difficult to compute in practice, there has been a line of research that defines similarity using simpler, yet less expressive structures. Examples include similarity lists Chakrabarti et al. [2022], graphs Lahoti et al. [2019] and ordinal relationships Jung et al. [2019].

4 Theoretical results

All proofs for this section can be found in the supplementary material

4.1 A simple learning algorithm

Given any confidence and accuracy parameters δ, ϵ ∈ (0 , 1) respectively, our approach is summarized as follows. At first, for every group ℓ we need a set S ℓ of samples that are chosen i.i.d. according to D ℓ , such that | S ℓ | = 1 δ log 1 δ 2 . Then, for every distinct ℓ and ℓ ′ , and for all x ∈ S ℓ and y ∈ S ℓ ′ , we ask the expert oracle for the true similarity value σ ℓ,ℓ ′ ( x, y ) . The next observation follows trivially.

Observation 4.1. The algorithm uses γ δ log 1 δ 2 samples, and γ ( γ -1) δ 2 log 2 1 δ 2 queries to the oracle.

Suppose now that we need to compare any x ∈ D ℓ and y ∈ D ℓ ′ . Our high level idea is that the properties M 1 and M 2 of σ ℓ,ℓ ′ (see Section 2), will actually allow us to use the closest element to x in S ℓ and the closest element to y in S ℓ ′ as proxies. Thus, let π ( x ) = arg min x ′ ∈ S ℓ d ℓ ( x, x ′ ) and π ( y ) = arg min y ′ ∈ S ℓ ′ d ℓ ′ ( y, y ′ ) . The algorithm then sets

where σ ℓ,ℓ ′ ( π ( x ) , π ( y )) is known from the earlier queries.

Before we proceed with the analysis of the algorithm, we need to recall some notation which was introduced in Section 2.1. Consider any group ℓ . An element x ∈ D ℓ with Pr x ′ ∼D ℓ [ d ℓ ( x, x ′ ) ≤ ϵ ] < δ is called an ( ϵ, δ ) -rare element of D ℓ , and also p ℓ ( ϵ, δ ) := Pr x ∼D ℓ [ x is ( ϵ, δ ) -rare for D ℓ ] .

Theorem 4.2. For any given parameters ϵ, δ ∈ (0 , 1) , the simple algorithm produces similarity approximation functions f ℓ,ℓ ′ for every ℓ and ℓ ′ , such that

where Pr[ Error ( ℓ,ℓ ′ ) ] is as in Theorem 2.1.

Observation 4.1 and Theorem 4.2 directly yield Theorem 2.1.

A potential criticism of the algorithm presented here, is that its error probabilities depend on p ℓ ( ϵ, δ ) . However, Theorem 2.3 shows that such a dependence is unavoidable (see appendix for proof).

4.2 Optimizing the number of expert queries

Here we modify the earlier algorithm in a way that improves the number of queries used. The idea behind this improvement is the following. Given the sets of samples S ℓ , instead of asking the oracle for all possible similarity values σ ℓ,ℓ ′ ( x, y ) for every ℓ, ℓ ′ and every x ∈ S ℓ and y ∈ S ℓ ′ , we would rather choose a set R ℓ ⊆ S ℓ of representative elements for each group ℓ . Then, we would ask the oracle for the values σ ℓ,ℓ ′ ( x, y ) for every ℓ, ℓ ′ , but this time only for every x ∈ R ℓ and y ∈ R ℓ ′ . The choice of the representatives is inspired by the k -center algorithm of Hochbaum and Shmoys [1985]. Intuitively, the representatives R ℓ of group ℓ will serve as similarity proxies for the elements of S ℓ , such that each x ∈ S ℓ is assigned to a nearby r ℓ ( x ) ∈ R ℓ via a mapping function r ℓ : S ℓ ↦→ R ℓ . Hence, if d ℓ ( x, r ℓ ( x )) is small enough, x and r ℓ ( x ) are highly similar, and thus r ℓ ( x ) acts as a good approximation of x . The full details for the construction of R ℓ , r ℓ are presented in Algorithm 1.

Suppose now that we need to compare some x ∈ D ℓ and y ∈ D ℓ ′ . Our approach will be almost identical to that of Section 4.1. Once again, let π ( x ) = argmin x ′ ∈ S ℓ d ℓ ( x, x ′ ) and π ( y ) = arg min y ′ ∈ S ℓ ′ d ℓ ′ ( y, y ′ ) . However, unlike the simple algorithm of Section 4.1 that directly uses π ( x ) and π ( y ) , the more intricate algorithm here will rather use their proxies r ℓ ( π ( x )) and r ℓ ′ ( π ( y )) . Our prediction will then be

where σ ℓ,ℓ ′ ( r ℓ ( π ( x )) , r ℓ ′ ( π ( y ))) is known from the earlier queries.

Theorem 4.3. For any given parameters ϵ, δ ∈ (0 , 1) , the new query optimization algorithm produces similarity approximation functions f ℓ,ℓ ′ for every ℓ and ℓ ′ , such that

where Pr[ Error ( ℓ,ℓ ′ ) ] is as in Theorem 2.1.

Since the number of samples used by the algorithm is easily seen to be γ δ log 1 δ 2 , the only thing left in order to prove Theorem 2.4 is analyzing the number of oracle queries. To that end, for every group ℓ ∈ [ γ ] with its sampled set S ℓ , we define the following Set Cover problem.

Algorithm 1 Training Phase

Accuracy and confidence parameters ϵ, δ . For every group ℓ ∈ [ γ ] , a set S ℓ of i.i.d. samples chosen according to D ℓ , such that | S ℓ | = 1 δ log 1 δ 2 .

1: for each ℓ ∈ [ γ ] do

2: H ℓ x ←{ x ′ ∈ S ℓ : d ℓ ( x, x ′ ) ≤ 8 ϵ } for each x ∈ S ℓ .

3: U ← S ℓ , R ℓ ←∅ and r ℓ ( x ) ← x for each x ∈ S ℓ .

̸

4: while U = ∅ do

5: Choose an arbitrary x ∈ U .

6: Set R ℓ ← R ℓ ∪ { x } .

̸

8: r ℓ ( x ′ ) ← x for every x ′ ∈ W x .

7: W x ←{ x ′ ∈ U : H ℓ x ∩ H ℓ x ′ = ∅} .

9: U ← U \ W x .

10: end while

11: end for

12: For every distinct ℓ and ℓ ′ , and for every x ∈ R ℓ and y ∈ R ℓ ′ , ask the oracle for σ ℓ,ℓ ′ ( x, y ) and store all these values.

13: For every ℓ , return the set R ℓ and the function r ℓ .

Definition 4.4. Let H ℓ x := { x ′ ∈ S ℓ : d ℓ ( x, x ′ ) ≤ 4 ϵ } for all x ∈ S ℓ . Find C ⊆ S ℓ minimizing | C | , with ⋃ c ∈ C H ℓ c = S ℓ . We use OPT ℓ to denote the optimal value of this problem. Using standard terminology, we say x ∈ S ℓ is covered by C if x ∈ ⋃ c ∈ C H ℓ c , and C is feasible if it covers all x ∈ S ℓ .

Lemma 4.5. For every ℓ ∈ [ γ ] we have | R ℓ | ≤ OPT ℓ .

Lemma 4.6. Let N = 1 δ log 1 δ 2 . For each group ℓ ∈ [ γ ] we have OPT ℓ ≤ N with probability 1 , and E [ OPT ℓ ] ≤ 1 δ + p ℓ ( ϵ, δ ) N . The randomness here is over the samples S ℓ .

The proof of Theorem 2.4 follows since all pairwise queries for the elements in the sets R ℓ are

̸

̸

where the inequality follows from Lemma 4.5. Combining (1) and Lemma 4.6 proves Theorem 2.4.

Remark 4.7. The factor 8 in the definition of H ℓ x at line 2 of Algorithm 1 is arbitrary. Actually, any factor ρ = O (1) would yield the same asymptotic guarantees, with any changes in accuracy and queries being only of an O (1) order of magnitude. Specifically, the smaller ρ is, the better the achieved accuracy and the more queries we are using.

Finally, we are interested in lower bounds on the queries required for learning. Thus, we give Theorem 2.5, showing that any algorithm with accuracy O ( ϵ ) and confidence O ( δ ) needs Ω( γ 2 /δ 2 ) queries. This immediately leads to the following corollary characterizing the optimality of our algorithm.

Corollary 4.8. When for every ℓ ∈ [ γ ] the value p ℓ ( ϵ, δ ) is arbitrarily close to 0 , the algorithm presented in this section achieves an expected number of oracle queries that is asymptotically optimal.

5 Experimental evaluation

We implemented all algorithms in Python 3.10.6 and ran our experiments on a personal laptop with Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz 2.90 GHz and 16.0 GB memory.

Algorithms: Weimplemented the simple algorithm from Section 4.1, and the more intricate algorithm of Section 4.2. We refer to the former as NAIVE, and to the latter as CLUSTER. For the training phase of CLUSTER, we set the dilation factor at line 2 of Algorithm 1 to 2 instead of 8 . The reason for this, is that a minimal experimental investigation revealed that this choice leads to a good balance between accuracy guarantees and oracle queries. As explained in Section 3, neither the existing similarity learning algorithms [Ilvento, 2019, Mukherjee et al., 2020, Wang et al., 2019] nor the Earthmover minimization approach of Dwork et al. [2012] address the problem of finding similarity values for heterogeneous data. Furthermore, if the across-groups functions σ are not metric, then no approach from the metric learning literature can be used. Hence, as baselines we used three general regression

models; an MLP, a Random Forest regressor (RF) and an XGBoost regressor (XGB). The MLP uses 4 hidden layers of 32 relu activation nodes, and both RF and XGB use 200 estimators.

Number of demographic groups: All our experiments are performed for two groups, i.e., γ = 2 . The following reasons justify this decision. At first, this case captures the essence of our algorithmic results; the γ > 2 case can be viewed as running the algorithm for γ = 2 multiple times, one for each pair of groups. Secondly, as the theoretical guarantees suggest, the achieved confidence and accuracy of our algorithms are completely independent of γ .

Similarity functions: In all our experiments the feature space is R d , where d ∈ N is case-specific. In line with our motivation which assumes that the intra-group similarity functions are simple, we define d 1 and d 2 to be the Euclidean distance. Specifically, for ℓ ∈ { 1 , 2 } , the similarity between any x, y ∈ D ℓ is given by

For the across-groups similarities, we aim for a difficult to learn non-metric function; we purposefully chose a non-trivial function in order to challenge both our algorithms and the baselines. Namely, for any x ∈ D 1 and y ∈ D 2 , we assume

where the vectors α, β, θ ∈ R d are basically the hidden parameters to be learned (of course non of the learners we use has any insight on the specific structure of σ ).

The function σ implicitly adopts a paradigm of feature importance [Niño-Adan et al., 2021]. Specifically, when x and y are to be compared, their features are scaled accordingly by the expert using the parameters α, β , while some offsetting via θ might also be necessary. For example, in the college admissions use-case, certain features may have to be properly adjusted (increase a feature for a non-privileged student and decrease it for the privileged one). In the end, the similarity is calculated in an ℓ 3 -like manner. To see why σ is not a metric and why it satisfies the necessary properties M 1 and M 2 from Section 2, refer to the last theorem in the Appendix.

In each experiment we choose all α i , β i , θ i independently. The values α i , β i are chosen uniformly at random from [0 , 1] , while the θ i are chosen uniformly at random from [ -0 . 01 , 0 . 01] . Obviously, the algorithms do not have access to the vectors α, β, θ , which are only used to simulate the oracle and compare our predictions with the corresponding true values.

Datasets: We used 2 datasets from the UCI ML Repository, namely Adult (48,842 points - 14 features) [Kohavi, 1996] and Credit Card Default (30,000 points - 23 features) [Yeh and Lien, 2009], and the publicly available Give Me Some Credit dataset (150,000 points - 11 features) [Credit Fusion, 2011]. We chose these datasets because this type of data is frequently used in applications of issuing credit scores, and in such cases fairness considerations are of utmost importance. For Adult, where categorical features are not encoded as integers, we assigned each category to an integer in { 1 , # categories } , and this integer is used in place of the category in the feature vector [Ding et al., 2021]. Finally, in every dataset we standardized all features through a MinMax re-scaller. Due to space constraints, the figures for Adult and Give me Some Credit are moved to the supplementary material. However, the observed traits there the same as the ones shown here for Credit Card Default.

Choosing the two groups: For Credit Card Default and Adult, we defined groups based on marital status. Specifically, the first group corresponds to points that are married individuals, and the second to points that are not married (singles, divorced and widowed are merged together). In Give Me Some Credit, we partition individuals into two groups based on whether or not they have dependents.

Choosing the accuracy parameter ϵ : To use a meaningful value for ϵ , we need to know the order of magnitude of σ ( x, y ) . Thus, we calculated the value of σ ( x, y ) over 10 , 000 trials, where the randomness was of multiple factors, i.e., the random choices for the α, β, θ , and the sampling of x and y . Figure 1 shows histograms for the empirical frequency of σ ( x, y ) over the 10 , 000 runs. In addition, in those trials the minimum value of σ ( x, y ) observed was 1) 0 . 1149 for Credit Card Default, 2) 0 . 1155 for Adult, and 3) 0 . 0132 for Give Me Some Credit. Thus, aiming for an accuracy parameter that is at least an order of magnitude smaller than the value to be learned, we choose ϵ = 0 . 01 for Credit Card Default and Adult, and ϵ = 0 . 001 for Give Me Some Credit.

(a) Credit Card Default

(b) Adult

(c) Give Me Some Credit

Frequency countfor across-groups similarities

Frequency countfor across-groupssimilarities

Frequency countfor across-groups similarities

600

500

600

Count

500

Frequency Count

400

Frequency Count

500

Frequency

400

300

400

300

300

200

200

200

100

100

100

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Value for across-groups similarities

Value for across-groups similarities

Value for across-groups similarities

Confidence δ and number of samples: All our experiments are performed with δ = 0 . 001 . In NAIVE and CLUSTER, we follow our theoretical results and sample N = 1 δ log 1 δ 2 points from each group. Choosing the training samples for the baselines is a bit more tricky. For these regression tasks a training point is a tuple ( x, y, σ ( x, y )) . In other words, these tasks do not distinguish between queries and samples. To be as fair as possible when comparing against our algorithms, we provide the baselines with N + Q training points of the form ( x, y, σ ( x, y )) , where Q is the maximum number of queries used in any of our algorithms.

Testing: We test our algorithms over 1 , 000 trials, where each trial consists of independently sampling two elements x, y , one for each group, and then inputting those to the predictors. We are interested in two metrics. The first is the relative error percentage; if p is the prediction for elements x, y and t is their true similarity value, the relative error percentage is 100 · | p -t | /t . The second metric we consider is the absolute error divided by ϵ ; if p is the prediction for two elements and t is their true similarity value, this metric is | p -t | /ϵ . We are interested in the latter metric because our theoretical guarantees are of the form | f ( x, y ) -σ ( x, y ) | = O ( ϵ ) .

Table 1 shows the average relative and absolute value error, together with the standard deviation for these metrics across all 1 , 000 runs. It is clear that our algorithms dominate the baselines since they exhibit smaller errors with smaller standard deviations. In addition, NAIVE appears to have a tiny edge over CLUSTER, and this is expected (the queries of CLUSTER are a subset of NAIVE's). However, as shown in Table 2, CLUSTER leads to a significant decrease in oracle queries compared to NAIVE, thus justifying its superiority. To further demonstrate the statistical behavior of the errors, we present Figure 2. This depicts the empirical CDF of each error metric through a bar plot. Specifically, the height of each bar corresponds to the fraction of test instances whose error is at most the value in the x-axis directly underneath the bar. Once again, we see that our algorithms outperform the baselines, since their corresponding bars are always higher than those of the baselines.

Disclaimer

This paper was prepared for informational purposes in part by the Artificial Intelligence Research group of JPMorganChase. and its affiliates ('JP Morgan'), and is not a product of the Research Department of JPMorganChase. JPMorganChase makes no representation and warranty whatsoever and

The CDFof the Relative Error

1.0

Cluster

Naive

MLP

RF

0.8

XGB

Empirical Probability

0.6

0.4

0.2

0.0

0.5

Relative Error Value

5

10

20

30

100

(a) Relative Error Percentage

The CDF of (Absolute Error/epsilon)

1.0

Naive

Cluster

MLP

RF

0.8

XGB

Empirical Probability

0.6

0.4

0.2

0.0

0.5

Value of (Absolute Error / epsilon)

5

10

20

30

100

(b) (Absolute Error) / ϵ

disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.

References

Bellet, A. Habrard, and M. Sebban. A Survey on Metric Learning for Feature Vectors and Structured Data. Research report, Laboratoire Hubert Curien UMR 5516, 2013. URL https: //hal.inria.fr/hal-01666935 .

Chakrabarti, J. P. Dickerson, S. A. Esmaeili, A. Srinivasan, and L. Tsepenekas. A new notion of individually fair clustering: α -equitable k -center. In G. Camps-Valls, F. J. R. Ruiz, and I. Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pages 6387-6408. PMLR, 28-30 Mar 2022.

C. Credit Fusion. Give me some credit, 2011. URL https://kaggle.com/competitions/ GiveMeSomeCredit .

Ding, M. Hardt, J. Miller, and L. Schmidt. Retiring adult: New datasets for fair machine learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 6478-6490. Curran Associates, Inc., 2021.

Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference , ITCS '12, 2012.

Frome, Y. Singer, F. Sha, and J. Malik. Learning globally-consistent local distance functions for shape-based image retrieval and classification. In 2007 IEEE 11th International Conference on Computer Vision , pages 1-8, 2007. doi: 10.1109/ICCV.2007.4408839.

S. Hochbaum and D. B. Shmoys. A best possible heuristic for the k-center problem. Mathematics of Operations Research , 10(2):180-184, 1985. ISSN 0364765X, 15265471. URL http://www. jstor.org/stable/3689371 .

Ilvento. Metric learning for individual fairness, 2019. URL https://arxiv.org/abs/1906. 00250 .

G. Jamieson and R. D. Nowak. Low-dimensional embedding using adaptively selected ordinal data. In 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1077-1084, 2011. doi: 10.1109/Allerton.2011.6120287.

Jung, M. Kearns, S. Neel, A. Roth, L. Stapleton, and Z. S. Wu. An algorithmic framework for fairness elicitation, 2019. URL https://arxiv.org/abs/1905.10660 .

P. Kim, O. Reingold, and G. N. Rothblum. Fairness through computationally-bounded awareness. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, page 4847-4857, Red Hook, NY, USA, 2018. Curran Associates Inc.

Kohavi. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining , KDD'96, page 202-207. AAAI Press, 1996.

Kulis. Metric learning: A survey. 2013.

Lahoti, K. P. Gummadi, and G. Weikum. Operationalizing individual fairness with pairwise fair representations. Proc. VLDB Endow. , 13(4):506-518, dec 2019. ISSN 2150-8097. doi: 10.14778/3372716.3372723. URL https://doi.org/10.14778/3372716.3372723 .

Moutafis, M. Leng, and I. A. Kakadiaris. An overview and empirical comparison of distance metric learning methods. IEEE Transactions on Cybernetics , 47(3):612-625, 2017. doi: 10.1109/TCYB. 2016.2521767.

Mukherjee, M. Yurochkin, M. Banerjee, and Y. Sun. Two simple ways to learn individual fairness metrics from data. In Proceedings of the 37th International Conference on Machine Learning , ICML'20. JMLR.org, 2020.

Niño-Adan, D. Manjarres, I. Landa-Torres, and E. Portillo. Feature weighting methods: A review. Expert Systems with Applications , 184:115424, 2021. ISSN 0957-4174.

L. Suárez-Díaz, S. García, and F. Herrera. A tutorial on distance metric learning: Mathematical foundations, algorithms, experimental analysis, prospects and challenges (with appendices on mathematical background and detailed algorithms explanation), 2018. URL https://arxiv. org/abs/1812.05944 .

Tamuz, C. Liu, S. Belongie, O. Shamir, and A. T. Kalai. Adaptively learning the crowd kernel. In Proceedings of the 28th International Conference on International Conference on Machine Learning , ICML'11, page 673-680. Omnipress, 2011. ISBN 9781450306195.

van der Maaten and K. Weinberger. Stochastic triplet embedding. In 2012 IEEE International Workshop on Machine Learning for Signal Processing , pages 1-6, 2012. doi: 10.1109/MLSP.2012. 6349720.

Wang, N. Grgic-Hlaca, P. Lahoti, K. P. Gummadi, and A. Weller. An empirical study on learning fairness metrics for compas data with human supervision, 2019. URL https://arxiv.org/ abs/1910.10255 .

Wilber, I. Kwak, and S. Belongie. Cost-effective hits for relative similarity comparisons. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , 2(1):227-233, Sep. 2014. URL https://ojs.aaai.org/index.php/HCOMP/article/view/13152 .

Yeh and C.-H. Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications , 36:2473-2480, 03 2009. doi: 10.1016/j.eswa.2007.12.020.

Yona and G. Rothblum. Probably approximately metric-fair learning. In International Conference on Machine Learning , pages 5680-5688. PMLR, 2018.

A Missing proofs

Proof of Theorem 4.2. For two distinct groups ℓ and ℓ ′ , consider what will happen when we are asked to compare some x ∈ D ℓ and y ∈ D ℓ ′ . Properties M 1 and M 2 of σ ℓ,ℓ ′ imply

Note that when d ℓ ( x, π ( x )) ≤ 3 ϵ and d ℓ ′ ( y, π ( y )) ≤ 3 ϵ , the above inequalities and the definition of f ℓ,ℓ ′ ( x, y ) yield ∣ ∣ f ℓ,ℓ ′ ( x, y ) -σ ℓ,ℓ ′ ( x, y ) ∣ ∣ ≤ 6 ϵ . Thus, we just need upper bounds for A := Pr S ℓ ,x ∼D ℓ [ ∀ x ′ ∈ S ℓ : d ( x, x ′ ) > 3 ϵ ] and B := Pr S ℓ ′ ,y ∼D ℓ ′ [ ∀ y ′ ∈ S ℓ : d ( y, y ′ ) > 3 ϵ ] , since the previous analysis and a union bound give Pr[ Error ( ℓ,ℓ ′ ) ] ≤ A + B . In what follows we present an upper bound for A . The same analysis gives an identical bound for B .

̸

Before we proceed to the rest of the proof, we have to provide an existential construction. For the sake of simplicity we will be using the term dense for elements of D ℓ that are not ( ϵ, δ ) -rare . For every x ∈ D ℓ that is dense, we define B x := { x ′ ∈ D ℓ : d ℓ ( x, x ′ ) ≤ ϵ } . Observe that the definition of dense elements implies Pr x ′ ∼D ℓ [ x ′ ∈ B x ] ≥ δ for every dense x . Next, consider the following process. We start with an empty set R = {} , and we assume that all dense elements are unmarked. Then, we choose an arbitrary unmarked dense element x , and we place it in the set R . Further, for every dense x ′ ∈ D ℓ that is unmarked and has B x ∩ B x ′ = ∅ , we mark x ′ and set ψ ( x ′ ) = x . Here the function ψ maps dense elements to elements of R . We continue this picking process until all dense elements have been marked. Since B z ∩ B z ′ = ∅ for any two z, z ′ ∈ R and Pr x ′ ∼D ℓ [ x ′ ∈ B z ] ≥ δ for z ∈ R , we have |R| ≤ 1 /δ . Also, for every dense x we have d ℓ ( x, ψ ( x )) ≤ 2 ϵ due to B x ∩ B ψ ( x ) = ∅ .

̸

Now we are ready to upper bound A .

̸

The upper bound for C is trivial. We next explain the computations for D . For the transition between the first and the second line we use a proof by contradiction. Hence, suppose that S ℓ ∩ B r = ∅ for every r ∈ R , and let i r denote an arbitrary element of S ℓ ∩ B r . Then, for any dense element x ∈ D ℓ we have d ℓ ( x, π ( x )) ≤ d ℓ ( x, i ψ ( x ) ) ≤ d ℓ ( x, ψ ( x )) + d ℓ ( ψ ( x ) , i ψ ( x ) ) ≤ 2 ϵ + ϵ = 3 ϵ . Back to the computations for D , to get the third line we simply used a union bound. To get from the third to the fourth line, we used the definition of r ∈ R as a dense element, which implies that the probability of sampling any element of B r in one try is at least δ . The final bound is a result of numerical calculations using |R| ≤ 1 δ and | S ℓ | = 1 δ log 1 δ 2 .

To conclude the proof, observe that A = C + D , and using a similar reasoning as the one in upper-bounding A we also get B ≤ p ℓ ′ ( ϵ, δ ) + δ .

Proof of Theorem 2.3 . Given any ϵ, δ , consider the following instance of the problem. We have two groups represented by the distributions D 1 and D 2 . For the first group we have only one element belonging to it, and let that element be x . In other words, every time we draw an element from D 1 that element turns out to be x , i.e., Pr x ′ ∼D 1 [ x ′ = x ] = 1 . For the second group we have that every y ∈ D 2 appears with probability 1 |D 2 | , and |D 2 | is a huge constant c ≫ 0 , with 1 c ≪ δ .

Now we define all similarity values. At first, the similarity function for D 1 will trivially be d 1 ( x, x ) = 0 . For the second group, for every distinct y, y ′ ∈ D 2 we define d 2 ( y, y ′ ) = 1 . Obviously, for every y ∈ D 2 we set d 2 ( y, y ) = 0 . Observe that d 1 and d 2 are metric functions for their respective groups. As for the across-groups similarities, each σ ( x, y ) for y ∈ D 2 is chosen independently, and it is

drawn uniformly at random from [0 , 1] . Note that this choice of σ satisfies the necessary metric-like properties M 1 and M 2 that were introduced in Section 2.

Further, since ϵ, δ ∈ (0 , 1) , any y ∈ D 2 will be ( ϵ, δ ) -rare:

The first equality is because the only element within distance ϵ from y is y itself. The last inequality is because 1 |D 2 | < δ . Therefore, since all elements are ( ϵ, δ ) -rare, we have p 2 ( ϵ, δ ) = 1 .

Consider now any learning algorithm that produces an estimate function f . For any y ∈ D 2 , let us try to analyze the probability of having | f ( x, y ) -σ ( x, y ) | = ω ( ϵ ) . At first, note that when y ∈ S 2 , we can always get the exact value f ( x, y ) , since x will always be in S 1 . The probability of having y ∈ S 2 is 1 -(1 -1 / |D 2 | ) N , where N is the number of used samples. Since we have control over |D 2 | when constructing this instance, we can always set it to a large enough value that will give 1 -(1 -1 / |D 2 | ) N = ϵ ; note that this is possible because 1 -(1 -1 / |D 2 | ) N is decreasing in |D 2 | and lim |D 2 |→∞ ( 1 -(1 -1 / |D 2 | ) N ) = 0 . Hence,

̸

When y will not be among the samples, the algorithm needs to learn σ ( x, y ) via some other value σ ( x, y ′ ) , for y ′ being a sampled element of the second group. However, due to the construction of σ the values σ ( x, y ) and σ ( x, y ′ ) are independent. This means that knowledge of any σ ( x, y ′ ) (with y = y ′ ) provides no information at all on σ ( x, y ) . Thus, the best any algorithm can do is guess f ( x, y ) uniformly at random from [0 , 1] . This yields Pr[ | f ( x, y ) -σ ( x, y ) | = ω ( ϵ ) | y / ∈ S 2 ] = 1 -Pr[ | f ( x, y ) -σ ( x, y ) | = O ( ϵ ) | y / ∈ S 2 ] = 1 -O ( ϵ ) = p 2 ( ϵ, δ ) -O ( ϵ ) . Combining this with (3) gives the desired result.

̸

Proof of Theorem 4.3. For two distinct groups ℓ and ℓ ′ , consider comparing some x ∈ D ℓ and y ∈ D ℓ ′ . To begin with, let us assume that d ℓ ( x, π ( x )) ≤ 3 ϵ and d ℓ ′ ( y, π ( y )) ≤ 3 ϵ . Furthermore, the execution of the algorithm implies H ℓ π ( x ) ∩ H ℓ r ℓ ( π ( x )) = ∅ , and thus the triangle inequality and the definitions of the sets H ℓ π ( x ) , H ℓ r ℓ ( π ( x )) give d ℓ ( π ( x ) , r ℓ ( π ( x ))) ≤ 16 ϵ . Similarly d ℓ ′ ( π ( y ) , r ℓ ′ ( π ( y ))) ≤ 16 ϵ . Eventually:

For notational convenience, let A := d ℓ ( x, r ℓ ( π ( x ))) and B := d ℓ ′ ( y, r ℓ ′ ( π ( y ))) . Then, the metric properties M 1 and M 2 of σ ℓ,ℓ ′ and the definition of f ℓ,ℓ ′ ( x, y ) yield

Overall, we proved that when d ℓ ( x, π ( x )) ≤ 3 ϵ and d ℓ ′ ( y, π ( y )) ≤ 3 ϵ , we have ∣ ∣ f ℓ,ℓ ′ ( x, y ) -σ ℓ,ℓ ′ ( x, y ) ∣ ∣ ≤ 38 ϵ . Finally, as shown in the proof of Theorem 4.2, the probability of not having d ℓ ( x, π ( x )) ≤ 3 ϵ and d ℓ ′ ( y, π ( y )) ≤ 3 ϵ , is at most 2 δ + p ℓ ( ϵ, δ ) + p ℓ ′ ( ϵ, δ ) .

Proof of Lemma 4.5. Consider a group ℓ , and let C ∗ be its optimal solution for the problem of Definition 4.4. We first claim that each H ℓ c with c ∈ C ∗ contains at most one element of R ℓ . This is due to the following. For any c ∈ C ∗ , we have d ℓ ( z, z ′ ) ≤ d ℓ ( z, c )+ d ℓ ( c, z ′ ) ≤ 8 ϵ for all z, z ′ ∈ H ℓ c . In addition, the construction of R ℓ trivially implies d ℓ ( x, x ′ ) > 8 ϵ for all x, x ′ ∈ R ℓ . Thus, no two elements of R ℓ can be in the same H ℓ c with c ∈ C ∗ . Finally, since Definition 4.4 requires all x ∈ R ℓ to be covered, we have | R ℓ | ≤ | C ∗ | = OPT ℓ .

Proof of Lemma 4.6 . Consider a group ℓ . Initially, through Definition 4.4 it is clear that OPT ℓ ≤ | S ℓ | = N . For the second statement of the lemma we need to analyze OPT ℓ in a more clever way.

Recall the classification of elements x ∈ D ℓ that was first introduced in the proof of Theorem 4.2. According to this, an element can either be ( ϵ, δ ) -rare, or dense. Now we will construct a solution C ℓ to the problem of Definition 4.4 as follows.

At first, let S ℓ,r be the set of ( ϵ, δ ) -rare elements of S ℓ . We will include all of S ℓ,r to C ℓ , so that all ( ϵ, δ ) -rare elements of S ℓ are covered by C ℓ . Further:

̸

Moving on, recall the construction shown in the proof of Theorem 4.2. According to that, there exists a set R of at most 1 δ dense elements from D ℓ , and a function ψ that maps every dense element x ∈ D ℓ to an element ψ ( x ) ∈ R , such that d ℓ ( x, ψ ( x )) ≤ 2 ϵ . Let us now define for each x ∈ R a set G x := { x ′ ∈ D ℓ : x ′ is dense and ψ ( x ′ ) = x } , and note that d ℓ ( z, z ′ ) ≤ d ℓ ( z, x ) + d ℓ ( z ′ , x ) ≤ 4 ϵ for all z, z ′ ∈ G x . Thus, for each x ∈ R with G x ∩ S ℓ = ∅ , we place in C ℓ an arbitrary y ∈ G x ∩ S ℓ , and that y gets all of G x ∩ S ℓ covered. Finally, since the sets G x induce a partition of the dense elements of D ℓ , C ℓ covers all dense elements of S ℓ .

Equation (4) and |R| ≤ 1 δ yield E [ | C ℓ | ] ≤ 1 δ + p ℓ ( ϵ, δ ) N . Also, since C ℓ is shown to be a feasible solution for problem of Definition 4.4, we get

Proof of Theorem 2.5 . We are given accuracy and confidence parameters ϵ, δ ∈ (0 , 1) respectively. For the sake of simplifying the exposition in the proof, let us assume that 1 δ is an integer; all later arguments can be generalized in order to handle the case of 1 /δ / ∈ N .

We construct the following problem instance. We have two groups represented by the distributions D 1 and D 2 . In addition, for both of these groups we assume that the support of the corresponding distribution contains 1 δ elements, and Pr x ′ ∼D 1 [ x = x ′ ] = δ for every x ∈ D 1 as well as Pr y ′ ∼D 2 [ y = y ′ ] = δ for every y ∈ D 2 .

For every x, x ′ ∈ D 1 let d 1 ( x, x ′ ) = 1 , and d 1 ( x, x ) = 0 for every x ∈ D 1 . Similarly, for every y, y ′ ∈ D 2 we set d 2 ( y, y ′ ) = 1 , and for every y ∈ D 2 we set d 2 ( y, y ) = 0 . The functions d 1 and d 2 are clearly metrics. As for the across-groups similarity values, each σ ( x, y ) for x ∈ D 1 and y ∈ D 2 is chosen independently, and it is drawn uniformly at random from [0 , 1] . Note that this choice of σ satisfies the necessary properties M 1 , M 2 introduced in Section 2.

In this proof we are also focusing on a more special learning model. In particular, we assume that the distributions D 1 and D 2 are known. Hence, there is no need for sampling. The only randomness here is over the random arrivals x ∼ D 1 and y ∼ D 2 , where x and y are the elements that need to be compared. Obviously, the similarity function σ would still remain unknown to any learner. Finally, the queries required for learning in this model cannot be more than the queries required in the original model, and this is because this model is a special case of the original.

Consider now an algorithm with the error guarantees mentioned in the Theorem statement, and focus on a fixed pair ( x, y ) with x ∈ D 1 and y ∈ D 2 . If the algorithm has queried the oracle for ( x, y ) , it knows σ ( x, y ) with absolute certainty. Let us study what happens when the algorithm has not queried the oracle for ( x, y ) . In this case, because the values σ ( x ′ , y ′ ) with x ′ ∈ D 1 and y ′ ∈ D 2 are independent, no query the algorithm has performed can provide any information for σ ( x, y ) . Thus, the best the algorithm can do is uniformly at random guess a value in [0 , 1] , and return that as the estimate for σ ( x, y ) . If ¯ Q x,y denotes the event where no query is performed for ( x, y ) , then

where the randomness comes only from the algorithm.

For the sake of contradiction, suppose the algorithm uses q = o (1 /δ 2 ) queries. Since each ( x, y ) is equally likely to appear for a comparison, the overall error probability is

Contradiction; the error probability was assumed to be O ( δ ) .

Proof of Corollary 4.8. When every p ℓ ( ϵ, δ ) is very close to 0 , Lemma 4.6 gives E [ OPT ℓ ] ≤ 1 δ . Thus, by inequality (1) the expected queries are γ ( γ -1) δ 2 . Theorem 2.5 concludes the proof.

Theorem A.1. The function σ defined in Equation (2) is not metric, and satisfies properties M 1 , M 2 for d 1 ( x, y ) = d 2 ( x, y ) = √∑ i ∈ [ d ] ( x i -y i ) 2 , when α i , β i ∈ [0 , 1] for all i ∈ [ d ] .

̸

Proof. The easiest way to see that σ ( x, y ) is not a metric, is by realizing that it does not satisfy the symmetry property and also x = y does not necessarily imply σ ( x, y ) = 0 . Both of these issues stem from the offsets θ i . To verify that symmetry is violated let x, y be 1-dimensional, and let x = 1 , y = 2 , α = 1 , β = 1 , θ = 2 . Then σ (1 , 2) = 1 , while σ (2 , 1) = 3 . Next, let x = 1 , y = 1 , α = 1 , β = 1 , θ = 1 . In this example, although x = y we have σ ( x, y ) = 1 = 0 .

In the following we are going to show that σ satisfies M 1 . The proof for M 1 is identical. At first, take x, y, z ∈ R d , such that x, z ∈ D 1 and y ∈ D 2 . Then:

To get the first inequality we used the triangle inequality for ℓ 3 . The second to last inequality is because α 3 i ≤ 1 for all i , and the last inequality is because the ℓ 3 norm of a vector is always smaller than its ℓ 2 norm.

B Additional experimental results

(a) Relative Error Percentage

(b) (Absolute Error) /

ϵ

The CDF of the Relative Error

The CDF of (Absolute Error / epsilon)

1.0

1.0

0.8

0.8

Empirical Probability

Empirical Probability

MLP

Naive

Cluster

RF

0.6

Naive

Cluster

MLP

RF

0.4

XGB

0.4

XGB

0.2

0.2

0.0

0.5

Relative Error Value

5

10

20

30

100

0.0

0.5

Value of (Absolute Error / epsilon)

S

10

20

30

100

(a) Relative Error Percentage

(b) (Absolute Error) /

ϵ

The CDF of the Relative Error

The CDF of (Absolute Error/ epsilon)

1.0

Naive

1.0

Naive

Cluster

Cluster

MLP

RF

111

MLP

RF

0.8

XGB

0.8

XGB

Empirical Probability

Empirical Probability

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.5

Relative Error Value

5

10

20

30

100

0.0

0.5

2

Value of (Absolute Error / epsilon)

5

10

20

30

100

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers PAC-learnable algorithms for estimating a intra-group\nsimilarity function where:\n1. intra-group similarity functions are metrics and given in advance,\n2. inter-group similarities satisfy a notion of the triangle inequality.\nIt is assumed that points are always labeled by the group to which they belong.\n\nThe authors present two algorithms Naive and Cluster, and\ngive PAC-learning guarantees for them (i.e., the sample complexity needed for\nthe learned similarity function to be $\\varepsilon$-close to the true\nsimilarity function with probability at least $\\delta$). The clustering\nalgorithm uses a ``representative'' subset of each groups samples before\nquerying inter-group similarities to reduce the query complexity. The authors\ngive substantive experiments to empirically study their algorithms.\n\nSTRENGTHS:\n- The \"simple learning algorithm\" is indeed simple and clean with strong\n  guarantees (Theorem 2.1).\n- The idea of using $k$ centers from each group's samples as representatives is\n  a natural improvement over the simple algorithm.\n- Very good experiments where the authors define a challenging similarity\n  function to try to recover:\n  * Creating two groups based on the \"married\" feature is an excellent instance\n    of what the paper aims to study.\n  * The benchmark algorithms MLP, RF, and XGB each see at least as many samples\n    as the proposed algorithms, and are therefore fair (assuming they fully\n    trained).\n\nWEAKNESSES:\n- In section 2.1, you say you begin with a simple algorithm, but it is never\n  described before Theorem 2.1. Consider giving a quick description to the\n  reader before they take in the theorem statements.\n- The presentation could be improved slightly: less motivation in the intro and\n  more details in the main text (e.g., proof sketches).\n\nQUESTIONS:\n- [line 22] If two objects $x$ and $y$ are identical, they have large\n  similarity? Maybe it makes sense to use the distance-version of $\\sigma$ here\n  so that it is clear the upper bound on the difference is zero for identical\n  objects.\n- [line 171] Is $Q_\\ell$ a random variable? If so, it might be worthwhile to\n  make that more explicit.\n- [line 229] Is a factor of $1/2$ missing in the query complexity? Are we\n  assuming that $\\sigma(x,y)$ is symmetric?\n- Does the similarity function in Eq. (2) have the property that inter-group\n  points are further away on average than intra-group points?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is about a mechanism to compare elements  of a set that belong to two distinct distributions when two distinct sets of features for each distribution.   This is a natural problem that arises in many practical settings.  The paper presents the underlying problem and the formal framework with some guarantees assuming some 3rd database (oracle) has answers about comparability of any pair of elements.\n\nSTRENGTHS:\nCompared to other papers I have to review for NeurIPS this year, the authors of this paper go out of their way to explain what they're trying to do and motivate the theoretical results they obtain.\n\nThe topic is a good one -- often the same data appear in different databases with different attributes and this paper provides a mechanism to construct a similarity function across different features *assuming* there is an oracle that can compare across different features.\n\nWEAKNESSES:\nThe basic idea of this paper is well explained and illustrated.  However, unlike much of the rest of the discussion in this paper, the \"no free lunch\" theorem is not explored in full.  A key question anyone would have is how rare or common is it for max { p_l, p_l' } is large and therefore the framework introduced in this paper is not applicable?  The authors could use the datasets they used for experimental results to shed some light on this question.  This is actually the 1st question anyone wishing to use this result would be asking themselves.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThe major limitation of this work is the assumption of the availability of an oracle that can provide the ground truth similarity between any pair of elements of the set across a broader set of features than what's given.  Eg, in the sport (say) of 100 meter sprint, on the one hand we have the age, height, weight, sex/gender, ethnicity, etc., of the competitors and on the other hand, we have the diet and medications for some.   Naturally a good similarity function to use to predict the athletes' performance needs to incorporate both sets of features.  So in such cases, where does one look for this all-knowing oracle even if the number of queries to it are limited?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning a similarity function of items generated from different distributions. Precisely, there are two groups of items, and the items in each group follow a specific distribution and the group has an intra-cluster metric. This paper wants to learn a cross-group metric from limited number of samples with fairness and PAC error guarantee. This problem is critical for creating fair comparisons of items in heterogeneous groups. \nFurther, this paper provides upper bound and lower bound analysis on both i) error rates of algorithms with polynomial sample complexity; and ii) the sample complexity of a PAC (\\epsilon,\\delta) algorithm. From the analysis, I think the bounds on both error rates and sample complexity are reasonably good though still have rooms for improvements. \nFinally, the authors present the numerical results on real-world data sets. The numerical results show out-performed performance of the proposed algorithms.\n\nSTRENGTHS:\nNovelty: This paper studies an interesting, novel, and theoretical-fundamental problem. The algorithms are not very new, but already show good enough performance in this new problem, so it is okay. Overall, this paper is of high novelty. \n\nQuality: The paper has a high quality. The results are complete. It is not common to see well-established lower bound analysis, but this paper does it for both error rates and sample complexity. Also, the upper bounds and lower bounds are reasonably close, and the gaps between them are not very high, which is good enough for initial research. The paper also conducts nice numerical experiments on real-world data to confirm the theoretical conclusions, which strengthen this paper's quality.\n\nClarity: This paper is well written. It has a clear story-telling: it introduces the problems, explain the mathematical formulation, state the theoretical results, and present the proofs well. \n\nSignificance: In my personal opinion, this is a very important problem in terms of theory, as it is actually understanding the fundamental learnability of aggregating two or multiple metric spaces together. In terms of practice, I am not very confident, but the examples such as the student case given in the paper are convincing. Overall, the I think this paper is significant enough.\n\nWEAKNESSES:\nNo major weakness found. The paper is good overall. One minor question: How often would the real-world scenarios have non-complete metrics, as this is an improvement of this paper?\n\nQUESTIONS:\nAs stated in Weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper involves finding an inter-group similarity function with (limited) oracle advice when one has access to the intra-group metrics. The authors give theoretical guarantees about their algorithm and claim to have proven lower bounds on the performance of any algorithm.\n\nSTRENGTHS:\nI like this problem. The proposed problem is a nice contribution - but only if its solution improves on the more general learning of the whole metric (without knowledge of the intra-cluster similarities) in the case that the similarities are a metric (the fact that the assumptions of this paper are slightly less restricted than a metric is by itself not enough novelty in my opinion)\n\nThe experiments show this algorithm outperforms certain baselines (although similar works are not considered in the experiments).\n\nThe paper is well written from a language point of view.\n\nWEAKNESSES:\nI do not think that the algorithms themselves are innovative enough for a conference like NeurIPS.\n\nFrom Line 437 of the appendix it seems that the true definition of a rare element should be one whose probability that ALL of the N samples lie at distance more than epsilon away is greater than delta (not a single sample as is defined in the paper). Although the definition used doesn’t make Theorem 4.1 incorrect it means the bound is much weaker than it could be.\n\nThere seems to be something very wrong with Theorem 2.3…\n- Firstly, If I limit \\delta to 1 then in most cases every element is (\\epsilon, \\delta) rare so that p_l(\\epsilon,delta)=1 and hence the theorem essentially states that with a finite number of samples it is impossible to achieve \\epsilon-accuracy with probability no more than \\epsilon. This is clearly incorrect.\n- Secondly, if the instance space is finite then we can always choose a large enough (but still finite) number of samples such that w.h.p. every possible instance is sampled. If we apply the oracle to every pair we have, w.h.p. (a probability arbitrarily close to 1) found the exact similarity function. This contradicts the theorem.\n\nIt is not clear how other works compare to the result of this paper when it is a true metric space (since in this case the multiple distribution idea reduces to a single distribution problem). I do not think that handling the slightly more general non-metric case alone is sufficient for acceptance to a conference like NeurIPS.\n\nIn Line 287 the authors state that the metric algorithms cannot be used for non-metric. I suspect that they can still actually run but the bounds don’t necessarily hold (please correct me if I’m wrong). Since there is no theoretical comparison to the metric algorithms (when the similarity is a metric) I would have liked to see empirical comparisons to them in the case that it is a metric.\n\nA minor weakness is that in many applications you would be able to query instances without actually having sampled them.\n\nQUESTIONS:\nSome questions and comments/tips…\n\nLine 81 says that if x is in the support of D_l then it is a member of group l - so can the supports not overlap? This contradicts line 84\n\nIn Line 109 I would write “does not need to axiomatically…” - this phrasing points out that this property is a strength rather than a weakness. Maybe also point out that all metric spaces obey your properties.\n\nThe functions p_l should be defined before Theorem 2.1 is presented.\n\nIn Line 162 it’s not “\\epsilon accuracy and a probability at most 1-max{..}” since you use \\omega and \\Omega.\n\nIn Line 74 you mean N^2 not N\n\nLIMITATIONS:\nAuthors have addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-25"}