Fairness under Competition
Authors: Ronen Gradwohl
Eilam Shapira
Moshe Tennenholtz
Venue: NeurIPS 2025 poster ()
Keywords: fairness
classification
strategic machine learning
competition
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We study fair classification when multiple classifiers compete, and show that even if individual classifiers are fair the outcome may not be.

Abstract: Algorithmic fairness has emerged as a central issue in ML, and it has become standard practice to adjust ML algorithms so that they will satisfy fairness requirements such as Equal Opportunity. In this paper we consider the effects of adopting such fair classifiers on the overall level of _ecosystem fairness_. Specifically, we introduce the study of fairness with competing firms, and demonstrate the failure of fair classifiers in yielding fair ecosystems. Our results quantify the loss of fairness in systems, under a variety of conditions, based on classifiers' correlation and the level of their data overlap. We show that even if competing classifiers are individually fair, the ecosystem's outcome may be unfair; and that adjusting biased algorithms to improve their individual fairness may lead to an overall decline in ecosystem fairness. In addition to these theoretical results, we also provide supporting experimental evidence. Together, our model and results provide a novel and essential call for action.

Fairness under Competition
Ronen Gradwohl Department Economics University of Haifa
Eilam Shapira
Faculty of Data and Decision Sciences
Technion - Israel Institute of Technology
rgradwohl@econ. hww.a/leeeoooooaa..aaaaaeee hwi.a.
am. shapira@gmail com
Moshe Tenneholtz
Faculty of Data and Decision Sciences
Technion Israel Institute of Technology moshet@ie technion..iii.i....m ac.
Abstract
Algorithmic fairness has emerged central issue in MI., and it has become stan- dard practice to adjust MI algorithms so that they will satisfy fairness requirements such as Equal Opportunity. In this paper consider the effects of adopting such fair classifiers on the overall level of ecosystem fairness. Specifically, we introduce the study of fairness with competing firms, and demonstrate the failure of fair classifiers in yielding fair ecosystems. Our results quantify the loss of fairness in systems, under a variety of conditions, based on classifiers' correlation and the level of their data overlap. We show that even if competing classifiers are individually fair, the ecosystem's outcome may be unfair; and that adjusting biased algorithms to improve their individual fairness may lead to an overall decline in ecosystem fairness. In addition these theoretical results, we also provide supporting experi- mental evidence. Together, our model and results provide novel and essential call for action.
1 Introduction
Algorithmic ceciision-making systems are not immune to human prejudices. This has been demon- strated by ample empirical evidence: For example, the use of algorithmic docision-making in determining loan approval and interest rates has led to minority applicants facing higher loan rejection rates and higher interest rates than non-minonn minority applicants (Bartlett et al., 2022; Fuster et al., 2022), and credit card issuers offering lower credit limits for women than men (Gupta, 2019; Telford, 2019). Additional contexts in which such algorithmic bias has been documented are as diverse as job recruitment (Doleac and Hansen, 2016), insurance premiums and payouts (Angwin et al., 2017), college admissions (Baker and Hawn, 2022; Gándara et al., 2024), and more (O (O'Neil, 2016).
To contextualize, consider a bank or other financial institution that issues loans and is subject to regulatory oversight. The bank employs historical data train classifier-a decision rule-to determine loan eligibility, with the aim of maximizing profit through successful loan repayments. However, such classifiers frequently exhibit biases against protected groups, such as ethnic minorities. In response, regulators can mandate faimess restrictions to mitigate these disparities. Bias is not only normatively problematic, but also has tangible adverse effects on the utility of affected individuals and, more broadly, on the welfare of the disadvantaged group.
The regulator's goal in imposing fairness adjustments to classifiers is thus to enhance the welfare of historically disadvantaged populations, One principled approach to fairness adjustment involves specifying a utility function, and requiring the classifier ensure equal welfare across groups.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prominent fairesss criteria such as Demographic Parity (DP) (Agarwal et al., 2018; Dwork et al., 2012), as well as Equal Opportunity (EO) and Equalized Odds (ED) (Hardt et al., 2016) can be interpreted as special cases within this welfare-based framework. For instance, DP corresponds to a utility specification where each approved applicant gains utility of one and zero otherwise. EO and ED, correcting for some criticisms about DP, does the same but only for the restricted set of deserving applicants.
Fairness interventions are intended to promote equity. While even widely adopted constraints like DP and EO may lead to suboptimal long-term outcomes (Liu et al., 2018), common wisdom holds that proper fairness adjustments of ML algorithms is an essential requirement.
But is the adoption of fair algorithms by firms sufficient to guarantee a fair Al ecosystem, in which multiple firms interact? Recent work in AT has recognized that Al systems should be evaluated in the context of multi-agent system, in which several stakeholders 2022; Boutilier et al., 2024). In lending, multiple financial institutions underwrite loans and issue credit cards; in employment, many firms vie for the same pool of job applicants; in insurance, numerous insurance companies offer policies; and in education, a large number of colleges compete over the same cohorts of students. In such multi-firm environments, equity not about whether single firm's 's decisions are fair, but whether there is overall discrimination in the number and quality of offers issued and opportunities afforded to loan, job, insurance, and college applicants.
active (Kurland and Tennenholtz,
In the world of algorithmic fairness, this translates into a novel question. Suppose there are several competing firms, each of which adopts fair classifier. Does the adoption of fair classifiers induce fair ecosystem, in which there is no overall discrimination?
In this paper, we introduce the study of fairness with competing firms. We provide fundamental definitions, and demonstrate the failure of fair classifiers in obtaining fair ecosystems. In particular, we show that even if competing classifiers are individually fair, the ecosystem's 's outcome may be unfair; and that adjusting biased algorithms to improve their individual fairness may lead to an overall decline in ecosystem fairness.
Contributions We begin in Section 2 by developing model of ecosystem faimess. We suppose there are multiple lenders who make loan offers to borrowers; however, the model applies analogously to job candidates and employers, insurance buyers and insurers, or student applicants and colleges. For this model we define our main notion of fairness under competition, competition analogue of EO that we call Equal Opportunity under Competition (EOC). The level of EOC here is a measure of how far the ecosystem is from satisfying EOC, where lower level implies higher ecosystem fairness. We also define a second, welfare-based version of EOC. An ecosystem satisfies this second notion if the welfare of different groups of borrowers is equalized when the lenders use their classifiers to make loan offers.
In Section 3 we turn to study EOC. Our main results are that, even when the individual classifiers satisfy EO, the ecosystem may be far from EOC. We quantify how far, based on model primitives, and provide worst-case bounds. In particular, we identify two distinct forces under which EO classifiers do not satisfy EOC. The first force, analyzed in Section 3.1, arises from differences in the correlations between the classifiers on the protected groups. The second force, analyzed in Section 3.2, arises when the lenders' pools of potential borrowers are overlapping but not identical. We quantify the extent to which each force decreases ecosystem faimeess.
To illustrate the first force, consider the following example:
There are two lenders and two groups of borrowers. Each lender has vast, distinct data on group 1, and trains a classifier to predict loan repayment. Neither lender has sufficient data on group 2, so both outsource third-party, who provides each lender with an (identical) classifier. Each lender uses her own and the third-party classifiers to offer loans to individuals predicted to repay the loan.
Example 1
In this example, if the lenders' and haird-party's 's classifiers have the same accuracies, then each lender's decisions satisfy EO. Note, however, that the lenders' predictions on group 2 are identical, as they are using the same classifier, and so the Pearson correlation between them is high. On the other hand, since the lenders use different classifiers with different training data on group 1, their respective predictions on this group have lower correlation. In Proposition and Corollary I we show that this difference in correlations leads to a positive level of EOC, of the same ofder-of-magee magnitude as

---

the classifiers' false-negative rates. Hence, competition between EO classifiers leads to non non-EOC outcome.
To illustrate the second force, consider the following example:
There are two lenders and two groups of borrowers. Each lender serves all applicants of group but only subset of applicants of group 2, perhaps because the lenders have lower market penetration in the latter population. For served applicants, lenders use an EO classifier to predict loan repayment and offer loans to individuals predicted to repay.
Example 2
In this example, the overlap between applicants served by the two classifiers in group is higher than in group 2. In Proposition 4 and Corollary 3 we show that such difference in overlaps leads to positive level of EOC, again of the same order-of-magniuus magnitude as the classifiers' false-negative rates, and again despite the fact that individual classifiers do satisfy EO.
Section 3 focuses on EOC when there are two lenders. However, Sections 3.1.1 and 3.1.2 we extend some results to the welfare-based notion of fairness and an arbitrary number of lenders.
In Section 4 we compare competition between classifiers that are not EO with competition between the same classifiers after undergoing a fairness adjustment (making them EO). We describe two examples in which the ecosystem EOC before the fairesss adjustment, but not EOC afterwards. These examples show that imposing fairness adjustment on individual classifiers can lead to a decline in ecosystem fairness.
Next, in order to explore the empirical prevalence of our theoretical results, in Section 5 we describe several experiments we ran Lending Club loan data. In these experiments compared the level of EOC before and after fairness adjustments under several variations. Our results indicate that, surprisingly, imposing fairness adjustments often leads to a higher level of EOC, and so lower ecosystem fairness.
Finally, in Appendix B we extend our results to other notions of fairness, namely, ED and DP.
Related literature Our paper is part of large and growing literature on fairness in machine learning (see, e.g., Chouldechova and Roth. 2018; Mehrabi et al.. 2021; Pessach and Shmueli, 2022; Barocas et al., 2023, among others). The literature includes a plethora of demonstrations of bias in machine learning, large number of fairness metries, and many algorithms for bias mitigation (the last of which is surveyed extensively by Hort et al., 2024).
Within this large literature, our paper fits within subset of papers that question whether fair machine learning algorithms actually achieve fairness (see Ruggieri et al., 2023, for a survey). One paper related ours in this vein is that of Liu al. (2018), who show that imposing fairness criteria may have adverse long-term effects. This can be seen as illustrating the failure of imposed fairness criteria due to temporal or sequential effects. Our paper, in contrast, can be seen as illustrating the failure due to competition or parallel effects.
Our research is more closely related to papers that examine fairness in settings with multiple classifiers. Two such papers, Bower et al. (2017) and Dwork and Ilvento (2019), also discuss the insight that fairness of individual classifiers does not imply fairness of a system composed of multiple classifiers. Our paper differs along multiple dimensions. First, in terms of motivation, these papers ask when central platform (e.g., an ad platform) will be fair when handling task that consists of several subtasks (advertisers on the platform), where each subtask is required to be fair. Our motivation is more about competing firms in decentralized market, where the joint activity determines the user's utility. More specifically, Dwork and Ilvento (2019) largely focus on individual fairness. They do have some extensions to group fairness, and their main results here are show that there exist distributions for which individual faimess does not imply joint fairness. However, in their model the classifiers are assumed to always be independent, and they cannot capture the correlations between classifiers that lead to ecosystem unfairness. In addition, their notion of group faimess does not include EO. Bower al. (2017) do focus on EO, but here the main difference is that classifiers are composed sequentially: one classifier makes a prediction or decision, the outcome is then passed on to the next classifier, and so on. our setting, contrast, the classifiers run in parallel. Finally, although both papers contain the insight that individual fairness does not suffice for joint fairness, in our paper we also identify the forces that lead the joint unfairness, and quantify the extent of this unfaimess as function of correlations and overlaps.
3

---

Another somewhat related paper is that of Ustun et al. (2019), who compare the benefits to protected groups when there is one classifier with imposed fairness criteria, as opposed many group-tailore tailored individual classifiers. Although the latter case involves multiple classifiers, individuals are not classified by more than one, and so there is no competition between the classifiers.
Our paper is also part of a smaller literature on competition between machine learning algorithms (see,e.g., Ben-Porat and Tennenholtz, 2017; Feng et al., 2022; Jagadeesan al., 2023). While that literature demonstrates that competition has some counter-intuitive effects, it has not examined issues surrounding fairness.
Finally, Example I above is related to the insight of Kleinberg and Raghavan (2021), who show that algorithmic monoculture the use of the same algorithms by different firms can lead to a decrease in utilities. In Example the use of the third-party classifier by the two firms leads to a difference in correlations, which can then lead to ecosystem bias.
Model
There are two types of players, borrowers and lenders. Each borrower is characterized by a triplet (z, a, y) E X x A x Y. where x is a vector of observable features, a is an observable protected attribute (group membership), and y is unobservable outcome variable. For simplicity, assume A = Y = {0. 1}. There is an underlying distribution DD over X x A x Y. We slightly abuse notation by using X, A, and Y as the sets of features, protected attributes, and outcome variables, and also as random variables with joint distribution D over the respective sets. Throughout, we think of each Y an individual borrower.
(x, a, v) E X
There is a set I of lenders. Each lender L E I. decides whether or not to offer a loan to each borrower based on observables (x, a). To do the lender C : x A + {0, 13. where cz( a) if the lender offers borrower (æ, a) a loan, and cr(2,, a) 0 otherwise. The unobservable outcome variable Y signifies whether or not the borrower repays the loan, and we call borrowers with = deserving borrowers. Naturally, lenders prefer to offer loans only to deserving borrowers. Hence, perfect classifiers are ones where cy(2,, a) = y for each borrower (æ, a. y). In general, however, firms' classifiers are not perfect. For given classifier ce, denote its neggaive rate by 3r Pr (X, A) O|Y 1] and its false-positive rate by ag = Pr (X, A) = 1 = 0j. Conversely, 1 - Be and 1 ag are the classifier's 's true-negative and true-positive rates, respectively.
potentially randomized classifier
Borrowers' utilities are described by a function v : X x A x Y x N + R, where N is the set of non-negative integers. v(I, a, y, is the utility of borrower (I, a, y) when she receives offers.
Fairness with single lender We begin by stating two notions of fairness for the setting of single 2 The first is the popular notion of EO (Hardt et al., 2016), which requires the classifier's false-negative rates to be equal across groups The second is based on the welfare-eoe eefflari-eullzzingg notion of Ben-Porat et al. (2021) that we call v-EO, and which requires the expected utilities of deserving borrowers to be equal across groups.
EO: The EO level of c is JE (c,(X,)))> = ference in the fraction of offers made each of the groups' deserving borrowers. A classifier is EO if its EO level is O.
= 1, IJI. the dif- A)IY =
v-EO: For given utility function of the borrowers and classifier o of lender, denote by We,ee (a) E[,(X,A, (X, A))lY = 1, A == a] the welfare of deserving borrowers from group a. The v-EO level of a classifier is [W... W.,ce((0 (0) W... (1)]. J. the difference in welfare between the deserving borrowers of the two groups under classifier ez.
Fairness with multiple lenders We generalize the definitions of faimesss a setting with mul- tiple competing lenders, EO under competition (EOC) and v-EO under competition (u-EOC). Fix a utility function v and classifiers (ci, 91)). and denote by R(x, ( a) Σter Cr(2r, a) the num-
More generally, borrowers' utilities could also depend on which lenders extended offers. Our definitions and results apply to this more general setting as well.
These notions are the foci of the paper, but in Appendix B extend our results to other notions.

---

ber offers made to borrower with observables (z, a). Also, denote by d(z, a) = Pr [R(z, a) 2 1], the probability that at least one lender offers a borrower with observables (x,, 0) loan.
EOC: The EOC level of classifiers e is |E [d(X, A)IY - 1, - - [d(X, A))Y = 1, A Il|
the difference between the two groups in the fraction of deserving borrowers who obtained at least one offer. Classifiers are EOC if their EOC level is O.
B-EOC: For a given utility function of the borrowers and classifiers c of the lenders, denote by (a) E (u(X, , A, Y, R(X, A)) = 1, al. The v-EOC level of classifiers is then W, W (1)l
In some of our analyses we will impose several assumptions. First, we sometimes focus on the case of two lenders. Second, sometimes focus the case in which borrowers do not how many offers they obtain, as long as they obtain at least one. In this case, their preferences satisfy the following:
Assumption (0-1 preferences) v(z a. a. = 1if 1 and v(z, a, y, 0) = 0, Vz, a,
Observe that under Assumption 1, EO is equivalent to e-EO and EOC is equivalent to v-EOC. For more general utility functions EO and EOC may differ from v-EO and v-EOC.
3 Equal Opportunity under Competition
In this section we ask the following question: Given two EO classifiers, are they guaranteed to be EOC? We show that the answer negative, and identify two distinct forces that drive this result. We also quantify the level of EOC, and bound the worst case. The first force, analyzed in Section 3.1, arises from differences the correlation between the classifiers on the two protected groups. The second force, analyzed in Section 3.2, arises when each classifier serves only subset of the borrowers.
3.1 Correlations between classifiers
Fix two EO classifiers O1 and c with false- negenivet rates B1 and 32. We define two Bernoulli random variables that capture the true positives the classifiers on the two groups a E A: For each a E A and each E {1, let the Bernoulli random variable
B ce (X, A) ((Y=1,A=a)-
Each B& is the output of classifier Gg on random instances with A and Y 1, and so E [Ba]] = - Be is the true-positive rate classifier ct. Denote by ar = / Be - 3r) the standard deviation By, and note that, because Gj EO, aę is also the standard deviation of B}. Finally, for each let fd denote the Pearson correlation coefficient between Bí and Bg. The Pearson correlation captures the extent to which the true positives
classifiers correlate with one another.
We now quantify the EOC level of two classifiers using these correlation coefficients, and determine the worst case.
Proposition 1 For two EO classifiers 01 and 02 with filh false-negative rates 31 and B2, the level of EOC 0i 02 1000 p'). In the worst case, the level of EOC is min min (2) max (0, B1 32 1}.
The intuition is the following. Under high (resp., low) correlation, if borrower is misclassified by one classifier, she is likely also (resp., not) misclassified by the other. In the low correlation case, each deserving borrower intuitively has "two chances" to get an offer, whereas the high correlation case she only has "one chance". . If in one group deserving borrowers get two chances and in the other they only get one, then in the former there a lower chance of being misclassificd.
The proofs of this and all other propositions are deferred to Appendix A due to space constraints. Proposition 1 yields the following simple corollary.
Corollary 3 = 31 = 32 and B < 1/2, then the worst-case level of EOC is B.
An implication of Proposition and Corollary is that the more accurate the classifiers get, the lower the worst-case level FOC. Thus, one way to minimize the EOC train more accurate classifiers, and this desideratum is aligned with the general goal of machine learning.
5

---

We can also use Proposition 1 to analyze Example 1.
Example 1 continued Suppose all classifiers in the example-the the third-party -arty classifier, well as the two firms' individual classifiers- the same false-negative rate B. Thus, a firm that uses its own classifier A = and the third-party classifier on A = 0 is in practice using Suppose also that the firms' individual classifiers e1 and C2 are uncorrelated on A = 1, namely, that p' = O. Finally, note that, since both firms use the same classifier on A we also have p 1. Thus, even though each firm uses an EO classifier, the level of EOC is 0102 J0- - 1| 3 - 8).
EO classifier.
3.1.1 Generalization: E-EOC
Recall that, under Assumption 1, EOC is equivalent to v-BOC, in which case all our results apply to the latter notion as well. In this section we study v-EOC when utility functions do not satisfy Assumption 1. In particular, we drop the assumption that v(x, a, y, 1) = v(z, a, y, 22 1, and instead assume the following:
Assumption (0-1-k preferences) v(2,a, a, = k 1 for r > 1, a(2,0.y.1) = 1, and ((2, a, y, 0) = 0, Vz, a. y.
Assumption 1 imposes k 1, but Assumption 2, a classifier is still EO if and only if is v-EO. However, for a pair of classifiers e₁ and 022 the definitions of EOC and e-EOC are longer equivalent.
consider all values of k: 2 1. Observe that under
We first generalize Proposition to this setting, and then apply it to Example 1.
Under Assumption 2, for two EO classifiers e and 0g with false-neangati rates 31 and {2, the level of v-EOC is 0 (k: 2) p²){- In the worst case, the level of v-EOC is max 31 82 1}).
Proposition 2
|k 2| (min 32]
There are a few interesting observations to note. First, in all cases except k 2, the level of v-EOC is positive, even though both classifiers are EO (and so, under Assumption 2, also e-HO). Hence, our insight is. in sense, robust. Interestingly, however, there is qualitative difference between the case k E [1, 2) and k > 2. in that the group with lower utility under competition is different in the two parameter intervals. This last point further explained at the end of this sub-section, in the context of the example.
Proposition 2 yields the following simple corollary.
Corollary If 31 Bg and B 1/2, then the worst-case level of v-EOC is 3 1k 2/.
We can also use Proposition 2 to further analyze Example 1.
Example 1 continued Suppose again that all classifiers in the example- the third-party classifier, well as the two firms' individual classffirss-hai the same false-negative rate B. Thus, firm that uses its own classifier on A = 1 and the third-party classifier on A = 0 is in practice using an EO classifier. Suppose also that the firms' individual classifiers e₁ and C2 uncorrelated on A = 1, namely, that pa = O. Finally, note that, since both firms use the same classifier on = O we also have p 1. Thus, even though each firm uses EO classifier, the level of v-EOC is B(1 (3) k 2. Furthermore, the utility of borrowers in group A = 0 is k(1 - 3), since all borrowers who receive an offer actually receive two offers. In group A 1, on the other hand, the expected utility of borrowers is k(1 - 3)² + 2,311 - 3).
Now,
A01 3)2 + 2 -3001-3))--1
This implics that the level of v-EOC is 3(1-3) |2 k). However, note that, if E [1, 2) then the difference is positive, whereas if k > 2 then the difference is negative. This implies that, in the former case, the expected utility higher in group A = 1, whereas in the latter case, the expected utility is higher in group A 0.
The main insights from this section are, first, that two E-EO classifiers can lead to a positive v-EOC even for more general utility functions, and second, that whether or not the "disadvantaged" group is the one that suffers under a positive v-EOC depends on the utility function (i.e., whether E (1) or k >>> 2).

---

3.1.2 Generalization: more than two classifiers
Suppose that instead of two classifiers, the set L contains classifiers. We provide two results. First, we characterize the worst-case EOC with n classifiers. Second, we generalize the analysis of Example and show that, when classifiers are uncorrelated on one group but fully correlated on the other group, the level of EOC strictly increases (and so worsens) with n..
Proposition 3 For EO classifiers CI...... Cn with
rates .I...., Bu, the worst-case
level of EOC is miniet B; max 0, 0, -1 1 If 3: 3 s 1 for all i, then the worst-case level of EOC 3.
We now analyze further extension of Example 1, in which we show that the level of HOC is strictly increasing in the number of lenders.
Example I continued Suppose there are n lenders. As in the original example, each lender has vast, distinct data on group A and trains a classifier to predict loan repayment. No lender has sufficient data on group A = 0, so both outsource to a third-party. who provides each lender with an (identical) classifier. Suppose all classifiers have the same False-negative rate 3. Thus, firm that uses its own classifier on A and the third- third-party classifier on A = 0 is in practice using an EO classifier. Suppose also that, on 1, the firms' individual classifiers misclassify positive instances independently of other classifiers' predictions:
Pr(ci(X,A)
cn (X, = A 1, = 1] 1| 8".
Additionally, since all lenders the same classifier A 0, the probability that positive
instance from A = is misclassified, Pr [cr (X, A)= _... (X, = 0| = Y,Y =1. is equal
to 3. Thus, even though each firm an EO classifier, the level of EOC is 3 - 8 Observe that this is increasing in n..
The main insight from this section is that rather than improving the situation, increasing the number of (EO) classifiers can actually lead to a larger EOC.
3.2 Different sets of borrowers
Recall that each borrower is triplet (z, a, E XxAXY, and that there is an underlying distribution D over X x Y. In this section we modify the model and suppose that each classifier & serves only a subset Sg C supp(D) of borrowers. Let us assume that S1 U.S, supp(D). In Section 3.1 we assumed that S S2 supp(D), (D). but this section we assume that S and S₂ are only partially overlapping, and so that S1 * Sg. In the extreme case, the two subsets of borrowers are disjoint.
Formally, assume each classifier faces an underlying distribution Dr of borrowers, where 'D₂ is equal to the distribution D conditional on St. Denote by 7 PRp, [A = a| the fraction of borrowers belonging to group A that are served by classifier E, and by 7 the fraction of borrowers of group a served by both lenders. Note that 7 + +2 -7 1.
In general, classifiers may serve different sets of borrowers and at the same time differ in other whys-they may have different error rates on borrowers served by both lenders, and may also have correlation between them (as in Section 3. In order to focus the former, however, in this section we make two simplifying assumptions. First, we assume that each classifier's same for borrowers that served exclusively by that classifier and for those that are served by both classifiers. Second, we assume that on instances (I. a, v) E S1 O S2 the classifiers are uncorrelated. Formally,
negattvee rate is the
Definition Classifiers C1, 22 with fals--poisssiii rates 31, B2 are uncorrelated if for each f, = 1,
Pr A) = Pr ()) (X,
8 S
St Sz-e),
and for every (z, a, S, SS2 it holds that Pr (r, c2(2, a) = = -1
In particular, if S = S2 (D) as in Section 3.1 then uncorrelated classifiers satisfy nd = p²= 0. neutralizing the force identified by Proposition This assumption thus isolates the effect of having different sets of borrowers. We now quantify this effect, and determine the worst case.
7

---

Proposition 4 For two uncorrelated EO classifiers and with false rates B1 and B2. the level of EOC is
[61 -0--) 22) + (7) -1)33 + (7* >
In the worst case, the level of EOC is max [22 ,2
The intuition for Proposition 4 is similar that of Proposition 1, with high (resp., low) overlap in the former playing the same role as high (resp.. low) correlation in the latter. To more clearly see the effect of different sets of served borrowers, consider the following corollary:
Corollary For two uncorrelated EO classifiers 61 and 02 with nelse-negative rates B1 = B2 = B. the level of EOC is 3 (1 3)
Thus, a large EOC occurs when there is large imbalance in the respective sizes of the overlaps in served borrowers between the two groups, 7 and y².
4 Harmful fairness adjustment
In trading off fairness and welfare (accuracy), it is quite intuitive that increasing the fairness properties of a classifier decreases its welfare properties. In this section, however, we show that increasing the fairness properties of individual classifiers may lead decreases in the fairness properties under competition. In particular, we show that the level of EOC can worsen following fairness- innnesssmmprovingg post-processing of the individual classifiers.
In the following, we provide two theoretical examples where this occurs. The first is based on an imbalance in correlations, as in Section 3.1, and the second imbalance in the sets of served borrowers, as in Section 3.2. While the examples are stylized, in Section we show that not only is this phenomenon empirically plausible, it is even likely.
For the following examples, we assume that the fairness adjustment is derived via post-processing (Hardt et al., 2):6): Given learned classifier the EO classifier is derived from c, namely, it depends only on A and the predictions A). We also assume that minimizes squared-loss relative to all such derived EO classifiers. These simplifying assumptions imply the following lemma:
Lemma Fix a classifier with false-negativ rates 3 on group = and 3¹ on group = 1. Then for each { 333 there exists distribution 'D and false positive-rates under which the optimal derived EO classifier has false-negative rate 33 on both groups.
The proof of Lemma follows from the fact that the optimal derived classifier can be found by linear program (Hardt et al., 2016), together with two observations: (i) when minimizing squared-loss subject EO, solution can be found vertex of the polytope formed by the constraints, and (ii) for some D and false-positive rates these vertices contain points where 3 E 1
We now turn the difference in correlations between classifiers, as described in Section 3.1.
first example. In this example, the reason the faimess adjustment worsens EOC is
Example 3 Both classifiers cj have false-ne rates 88 = 0.1 and B = 0.2. The correlations are 0 - and p² = 0.375 (perhaps both lenders outsource to the same third-party for predictions about group A 0. as Example 1). Furthermore, suppose that D and false calse-positive rates are such that the fairness adjustments lead to classifiers ě with false-negative rates 30 3 = 0.1 (guaranteed to exist by Lemma 1). Such fairness adjustments are accomplished by randomizing each C 's predictions only when A and cx(X, A) and this leads to new correlations 7o and م < 1.
Observe that, before fairness adjustment, neither classifier is EO, but after fairness adjustment, both classifiers are EO. However, observe also that, before fairness adjustment, the classifiers EOC: First, Pr [ci (X, A) = c2(X, [XX)] A) = Y = 1, A 0] = 0.1, because the classifiers are fully correlated = 0. Second, by (1),
Pr[c A) c2(X.A) A) = 0/Y 1,1
= =po0002 + 3133 0.375 0.2 0.8 + 0.2 0.2 0.1.
Finally, observe that after fairness adjustment, the classifiers are no longer FOC. This is because the probability both classifiers misclassify positive instance in A 0 remains the same, namely,
8

---

= = 0/Y = 1,A = 0] 0.1. The probability of misclassification in A = 1,
however, now different:
Pr Pr( ( 4) = 0|V = =1,4 =
= - 0.9 0.1 0.1 < 0.1,
where the inequality holds since p² < 1.
We now turn to our second example, in which the reason the faimess adjustment worsens EOC is the difference in sets of served borrowers, as described in Section 3.2.
Example 4 Let S (D) and S2 supp 1). Note that this implies that 70 0 and y' = 1. Suppose 09 is a perfect classifier, with 82 = 02 = 0. Classifier c1 is a perfect classifier on group A = 0 only, and has dalse-negative rate 8 0 on group A = 0 and false- negativv rate 3 = 3 > 0 on group A = 1. Note that C1 is not EO, and for C2 the issue is moot since C2 serves only borrowers with A = 1. However, note that the classifiers are EOC: deserving borrower from = 1 will always get loan offer from (2, and a deserving borrower from A = 0 will always get a
loan offer from C1-
Now suppose e1 undergoes fairness adjustment to č by post-processing, namely, it is derived from C1. Suppose also that false-positive rates and D are such that, after adjustment, the false-negative rates on both groups are 31 B (guaranteed to exist by Lemma 1). In this case, deserving borrowers from A = 1 will still always get a loan offer from c2, but deserving borrowers from A will only get offer from ε with probability 1 3 (and never get offer from e2). Thus, the classifiers are no longer EOC.
Experiments
In order to explore the prevalence of our theoretical results several experiments." 3 We describe and report results from the first three here, and the remaining ones in Appendix C. For the first three we used Lending Club loan data for the years 2007-2015, dataset that includes roughly 890,000 peer-to-peer loans through Lending Club (2021). We used simple classifiers to predict the Loan_status outcome, and, in particular, whether not the loan was fully paid, given set of features that included the loan amount, the interest rate, the number of installments, and the borrower's annual income. As there is no demographic information in the public data, the protected attributes A we used in our experiments were whether or not individuals have a mortgage. We compared the performance in terms of EO and EOC before and after the corresponding fairness adjustment. The fairness adjustment was implemented using the open spurcsource Fairlearn (2025) package.
In the experiments simulated a situation in which two firms compete over the same pool of borrowers, S = S2, as in Proposition 1. The differences between the firms' classifiers were captured in three different ways in the respective experiments:
Exp. Cj was a logistic regression and 02 a decision tree, but their training data was identical.
Exp. 2 Both classifiers were logistic regressions, but their training data disjoint: one trained on random examples with a short loan term, and one with a long loan term.
Exp. e1 was a logistic regression and d2 a decision tree, and their training data was disjoint (as in Exp. 2).
In all three experiments, the training data consisted of range of 300 100, 000 random examples. We ran each experiment with each size of training set 500 times." For each run we calculated the EO of each classifier as well as the EOC level, both before and after the fairness adjustments. The most relevant statistic was then the comparison between the EOC level before fairness adjustments and the EOC level after fairness adjustments. We calculated the percentage times the EOC level was higher after adjustment than before, as well standard errors of the mean.
To get a sense of the results, consider the following selected run from experiment 3: Before adjust- ments, the values were EO, 0.597192%, EO2 = 3.430634%, and EOC = 0.020774% Thus,
1 *oode available at https: //githbb com/ htt//eilmmeeee amanapppti/BBtrrreetttrrrrpppttp 'Running all experiments took few hours home computer.

---

Table 1: 95%-CI CI-CIIII for likelihood EOC level increased following fairness adjustment
300
lk
3k
10k
30k
100k
Exp. 1
[75.0, 82.2]
[68.0, 0, 76,4]
[55.6, 64.2]
[49.4, 58.2]
[42.0, 50.6]
[26.2, 34.0]
Exp. 2
[75.6, 82.8]
[65.2, 73.8]
[51.8, 60.8]
[35.4, 44.2]
[25.8. 33.8]
[12.6, 19.0]
Exp. 3
[74.2, 81.2]
[63.4, 71.2]
[50.8, 59.6]
[35.6, 43.8]
[27.6, 36.2]
[14.2, 20.6]
even though both classifiers were somewhat biased, under competition the EOC was nearly 0. How- ever, once the individual classifiers were adjusted for fairness, the values were FO1 = 0.4034188 EO2 = 0.985511 %,, and EÔC = 4440522. Thus, even though each individual classifier became more fair after adjustment, the result under competition became worse.
To examine the prevalence this phenomenon we counted the percentage of times the EOC became worse after faimess adjustment. Table 1 reports the results: The rows indicate the experiment number, the columns indicate the size of the training set, and the values in the table are the 95% confidence intervals, in percentages, for the probability that the EOC level was worse after adjustment. We found this to be surprisingly common even for large training sets. For instance, for training sets of size 100k (out of dataset of size roughly 900k) it occurred 30.1% %, 15.8% and 17.4% of the time in experiments 1, 2, and 3, respectively.
We note that the effect diminishes as the size of the training set increases. The diminishing of the effect for large training sets holds a similar message as the results from Section 3, where we show that the magnitude of the EOC level is roughly the same that of the false-negative rates. For very large training sets the false-negative rates diminish to O, and so the levels of EO and BOC diminish in turn.
In addition to examining the prevalence of the effect, we also measured its magnitude. For each run, if the EOC level was higher after fairness adjustment than before, we measured the factor by which the EOC level increased. For each experiment we then calculated the average factor by which the EOC level increased (conditional on increasing), as well as the standard error of the mean. The results are reported in Appendix C due to space constraints. However, to get a sense of the orders-of- orders-of-magnitude magnitude consider the following. For training sets of size 100k (out of a dataset of size roughly 900k), the EOC level increased by a mean factor of 19, 1.3, and 3.1 in experiments 1, 2, and 3, respectively.
In Appendix C we also report on additional experiments we ran. In particular, we ran an experiment with S1 * S2, as in Proposition 4, and simulated the situation described in Example 4. We also ran experiments in which classifiers' training sets were independently chosen, experiments using different dataset (specifically, Becker and Kohavi, 1996), and experiments with three rather than two competing classifiers. For all experiments, the results were similar to the ones reported here.
Discussion and future work
Our main insight is that individual fairness is neither necessary nor sufficient for ecosystem fairness. Thus, when making fairness adjustments, their effects the ecosystem should be considered. An interesting question relates to the kinds of regulations that could lead to ecosystem fairness while also maintaining the benefits of having multiple competing classifiers.
In our analysis we made several simplifying assumptions. First, we focused on EO and EOC. In Appendix B, however, we extend the definitions and analyses to two other common notions of fairness-namd namely, Equalized Odds and Demographic Parity. In particular, we define variants of these notions for setting with multiple classifiers, and then show that versions of Propositions and hold for them as well: If the correlations or overlaps in served borrowers between classifiers differ across groups, then classifiers that are fair will not be fair under competition.
Second, for our welfare-basee based notions of E-EO and E-EOC. we focused on simple utility functions that satisfy Assumptions or 2. A natural direction that we leave for future work is to consider more general preferences, including ones where deserving borrowers' utilities may differ from others', and where the utility function may depend on any of (Ir, a, v).
DDividing responsibility, for example by allowing only one EO classifier to serve each borrower, would lead ecosystem fairness, but would not deliver the benefits of having multiple classifiers.
10

---

Acknowledgments and Disclosure of Funding
Research supported by funding from the Israel Science Foundation, award number 2039/24 (Grad- wohl), and by funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme, grant number 740435 (Tennenholtz).
References
AGARWAL, A., BEYGELZIMER, A., DUDÍK, M., LANGFORD, and WALLACH, H. (2018). A reductions approach to fair classification.. In International Conference Machine Learning.
PMLR, 60-69.
ANGWIN, J., LARSON, J., KIRCHNER, L. and MATTU, S. (2017). Minority neighborhoods pay higher car insurance premiums than white areas with the same risk. ProPublica (5 2017).
BAKER, R. S. and HAWN, A. (2022). Algorithmic bias in education. International Journal of Artificial Intelligence in Education 1-41.
BAROCAS, S., HARDT, M. and NARAYANAN, A. (2023). Fairness and machine learning: Limita- tions and opportumities. MIT press.
BARTLETT, R., MORSE, A., STANTON, R. and WALLACE, N. (2022). Consumer-lending discrimi- nation in the FinTech era. Journal of Financial Economics, 143 30-56.
BECKER, B. and KOнAVI, R. (1996). Adult. UCI Machine Learning Repository, DOI: http:://oo.or//0.2442//SC3
BEN-PORAT, O., SANDOMIRSKIY, F. and TENNENHOLTZ, M. (2021). Protecting the protected group: Circumventing harmful faimess. In Proceedings of the AAAI Conference Artificial
Intelligence, vol. 35. 5176-5184.
BEN-PORAT, O. and TENNENHOLTZ, M. (2017). Best response regression. Advances in Neural Information Processing Systems, 30.
BOUTILIER, C., MLADENOV, M. and TENNENHOLTZ, G. (2024). Recommender ecosystems: A mechanism design perspective on holistic modeling and optimization. In Proceedings of the AAAI Conference Artificial Intelligence, vol. 38. 22575 225833.
BOWER, A., KITCHEN, S. N., NISS, L., STRAUSS, M. J., VARGAS, A. and VENKATASUBRAMA- NIAN, S. (2017). Fair pipelines. arXiv preprint vrXiv:1707.00311
CHOULDECCOVA, A, and ROTH, (2018). The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.0880
DOLEAC, L. and HANSEN, B. (2016). Does "ban the box" help hurt low-skilled workers? Statistical discrimination and employment outcomes when criminal histories are hidden. Tech. rep., National Bureau of Economic Research.
DwORK, C., HARDT, M., PITASSI, T., REINGOLD, O. and ZEMEL, R. (2012). Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference. 214-226.
DWORK, C. and ILVENTO, C. (2019). Fairness under composition. In 10th Innovations in Theoretical Computer Science Conference.
FAIRLEARN (2025). https: fttaps:/ffarre h/ta://eaarrrrraaeeeeaaaaaaaaaae org/ Accessed: 2025-04-30.
FENG, Y., GRADWOHL, R., HARTLINE, J., JOHNSEN, A. and NEKIPELOV, D. (2022). Bias-variance games. In Proceedings of the 23rd ACM Conference Economics and Computation. 328-329,
FUSTER, A., GOL.DMMIT- unequal? The effects of machine learning credit markets. The Journal of Finance, 77 5-47.
P., RAMADORAT, T. and WALTHER, A. (2022). Predictably

---

GÁNDARA, D., ANAHIDEH, H., ISON, M. P. and PICCHIARINI, L. (2024). Inside the black box: Detecting and mitigating algorithmic bias across racialized groups in college student-success prediction. AERA Open, 10 1-15.
GUPTA, A. (2019). Are algorithms sexist. The New York Times.
HARDT, M., PRICE, E. and SREBRO, N. (2016). Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems, 29.
HORT, M., CHEN, Z., ZHANG, J. M., HARMAN, M. and SARRO, F. (2024). Bias mitigation for machine learning classifiers: A comprehensive survey, ACM Journal on Responsible Computing, 1 1-52.
JAGADEESAN, M., JORDAN, M.. STEINHARDT, J. and HAGHTALAB, N. (2023), Improved Bayes risk can yield reduced social welfare under competition. Advances in Neural Information Processing Systems, 36 66940-66955
KLEINBERG, J. and RAGHAVAN. M. (2021). Algorithmic monoculture and social welfare. Proceed- ings of the National Academy of Sciences, 118 e20183401
KURLAND, O. and TENNENHOLTZ, M. (2022). Competitive search. In Proceedings of the 45th International ACM SIGIR Conference Research and Development in Information Retrieval.
2838-2849.
LENDING CLUB (2021).
attps://wwx http://www..ggl...eee http://glleecmmttttttttt/s//////
/ aaae//aaarsssg//////e Open Database License (ODbL) 1.0. Accessed:
lending-cluu club-loan- wwwllllnnnndaaeeee data-csv/data 2025-04-30.
LIU, L. T., DEAN, S., ROLF, E., SIMCHOWITZ, M. and HARDT, M. (2018). Delayed impact of fair machine learning. In International Conference on Machine Learning. PMLR, 3150-3158.
MEHRABI, N., MORSTATTER, F., SAXENA, N., LERMAN, K. and GALSTYAN, A. (2021). A survey bias and fairness in machine learning. ACM computing surveys (CSUR), 54 1-35.
NELSEN, R. B. (2006). An Introdaction Copulas. Springer.
O'NEIL, C. (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Penguin Books Limited.
PESSACH, D. and SHMUELI, E. (2022). A review on faimess in machine learning. ACM Computing Surveys (CSUR), 55 1 1-44.
RUGGIERI, S., ALVAREZ, J. M., PUGNANA, A., TURINI, F. ET AL. (2023). Can we trust fair-AI? In Proceedings of the AAAI Conference Artificial Intelligence, vol. 37. 15421-15430.
TELFORD, T. (2019). Apple Card algorithm sparks gender bias allegations against Goldman Sachs. Washington Post.
USTUN, B., LIU, Y. and PARKES, D. (2019). Fairness without harm: Decoupled classifiers with
preference guarantees. In International Conference on Machine Learning. PMLR, 6373-6388
12

---

Appendix
A Proofs
Proof of Proposition For each a, the Pearson correlation between Bi and Bi is
Pr Pr
-
B =
Pr [3 .r
Pr
- -32)
Pr
0102
0102
Thus,
Pr [B]
= (1- B2)
and
Pr [Bf = -
Pr BB B 1])
=1- -1- 32) 000
+ (1 (1) - (1)
+ 8132.
(1)
The level EOC's is E [d(, 4) - -
E|d
= 1, 1//. where
E [d(X, Y
1, A a = Pr (a =) (2 A) 0 1, A
= - - Pr B = =
Putting the above together yields
E [d((,, = 1, - - 1,
= Pr = = Pr B = =
= 10o102 + 81 - p 0102 3132 01 02 1000 - p'1,
as claimed.
Now, the worst case scenario maximizes 100 - p-1 and for example, maximizes p while minimizing p² (or vice versa). For any a, the correlation can written as pa f where max(0, 31 + 32 - 13 I r < min 31, {2] by the Tréchet-Hoeffdig bounds (see, e.g., Nelsen, max(0, 31 82 yields the result.
did2
2006). Plugging in p min (2) and
To see that this worst case can be achieved, and obtain an intuitive explanation for the Frèchet- Hoeffding bounds in our setting, recall that
nd Pr [BT = B3 = 1] =
1, -,
the probability that both classifiers misclassify an instance with Y = 1. Given that false- negative rates are 81 and 82, the probability that both classifiers misclassify such instance is at most T0 = min (11, }3). and this occurs when the set of one classifier's misclassified instances are contained in the other's.
The probability that both classifiers misclassify an instance with Y 1 minimal when the instances on which the classifiers misclassify are maximally disjoint, and in this case the probability that both misclassify an instance is r max(0 (0, 81 + B2 -13. 1}. partieular, if 31 32 S then the classifiers never misclassify the same positive instance.
Proof of Proposition 2 Recall from the proof of Proposition that
Pr Pr = B 1]
p" 0000 (1 11B1(((1 82)
and
Pr [ B = p² 0102 + 81.2.
Thus, have that
Pr B n B3 = 0 = Pr = 1| Pr [3 = B = 1]
=1 31 0 00000022222200000000000000 (1 - 3:)(1 -B2)
F2 - -1)-33p-p00 -
13

---

and, analogously,
Pr = 8 82) - 020
Thus, for each a, A, Y, B cx - 1, 1,A=d
+ (()
k Pr
=pp + = +(
- 2000002
(k 2100 - (k -
The level of v-HOC is thus
E A,Y, R(X,
EE A,Y -(X
= - 2 0102 (k 2
= 0100 (k: 2p p²)l-
As in the proof of Proposition 1, the worst case scenario maximizes |0" p'|, and so, for example, maximizes p while minimizing p (or vice versa). Again, as before, the worst-case difference is (min (2) - max(0, 81 32 1)), which leads to the claimed worst case level of v-EOC.
0102
Proof of Proposition 3 The worst case occurs when the probability that all classifiers misclassify positive instances is minimal in group A = and maximal in group A = 0 (or vice versa). To maximize Pr [cq (X, .... (x, A) 0 O, 1] the overlap in instances on which each e, misclassifies has to be maximal, which occurs when the instances are contained in one anther.
Formally, if 31 S ... S An, then
Ca(x, = 1= ... (z.
In this case,
Pr (x, = ... Gn (X, A) 0/4 Y, 1| B, min 9
iEL
To minimize Pr (X, =.. Cn(X, A) 1 = 1] the overlap in instances on which each o misclassifies has to be minimal, which occurs when the instances are maximally disjoint. Equivalently, the overlap in positive instances in which the c, correctly classify is maximally disjoint. If E1 3,) then
1,, e1 (z, =... (z, 1) = 0} = 0.
Otherwise, in the worst case each classifier c; correctly classifies unique set of 1 -3 positive instances, and this case
Pr(ci(X, =...= =... A) = =
= Σ Bg -
JEL
Thus, the worst-case level of EOC is min 3, -
max f0. E.. jEL Bg - 13.
Finally, if 8; = B < 1/nn for all i then minger B, = 3 and max 0. (.EEe G3j - 11] = 0.
Proof of Proposition 4 For any a A.
E [id(X,))l* = 1, A al = -
( 1,A a
(P (x) S 11A,
+ =) ( (=
+ "TP.....0==.0....=-01188.1111 1= 1,A=
- (Bff-++)a--00-7)3)3--3-3
= 11 - (1==3))---
14

---

The level of EOC is thus
E 1 - = 1]
1 1, = 1(72 - 12)81 - (7) - -1) (7' - 1/
The worst case is constructed as follows. Classifier & arg max; 3r serves all borrowers from group A and the other classifier (3 f) does not serve any borrower from this group. Additionally, both classifiers serve all borrowers with A = 1. Formally, of = % = 1 and 28- = 0 and 73-1 1. This implies also that 70 0 and 7 1, and so yields a level of EOC equal to max [ 82] - -31
To see that this maximal, observe that
0 72 + 7 71 = 7 + 7 - (12 71) = 1- (1 -1+-)
Thus, (72 72)) (7p (09--71)83 3/ s 0 - - and so the level of BOC is at most (70-7") 7²) (max{31, - 3132) S max 323 31
Proof of Lemma 1 For a classifier с and a group a E A, let (c) = 1-", where at
Pr Pr[ce (X, A. = 1Y 0, = a] the false-positive rate of the classifier in group a, and 1 1-3" Pr lez [X A, = - -1 1, = a. is the ine-poositvee rate of the classifier in group a. Hardt et al.
c). (1, 1)}. where (1 c) is the classifier but with predictions flipped. They then show that the optimal derived classifier can found using the following linear program:
(2016) show that any derived classifier satisfies X (C) E convhull {(0. 0), (c), X
min E(((C, c)]
s.L. (c) E convhull ((L,)),) Va A
x 22
The second constraint above states that the true-positive rates of the classifier are equal on both a E A, and so the classifier must be EO. The optimization problem is squared-loss minimization.
Let us fix the distribution D to be uniform over all of x Y x A. Thus, for each group a and label E {0, 1}. we have that Pr [A al Pr Y yl 1/2. We also assume, without loss of generality, that 30 > B).
The optimal derived classifier č is then one of the following:
1. Let ē = except that, on some fraction of instances with c(x, 0) = 0 fix 0(z, 0) 1, so that (a) = (1- - (c) + 50 (1,1) = (ō", 1 31 3
for 50 = (55 5 and some a"_ In this case, 30 = 3¹. The cost of this in terms of
squared loss is the fraction of instances where c(2, 0) = 0 that were flipped to 2(x, 0) 1,
namely, 80 - a + //4
2. Let = c. except that, on some fraction of instances with c(x; 1) 1 fix a(x, 1) = 0, so that (²()) (1 8t (c) 5t (0,0) (ā 1 30)
for 81 (30 /11 - 31) and some â'. In this case, β² f30 The cost of this in
terms of squared loss is the fraction of instances where c(x, 1) 1 that were flipped to C(x, 1) = O, namely, 8 (a + //4
To see that one the above is an optimal derived classifier, note first that the optimal solution cannot be in the strict interior of the polytope. Furthermore, if (1 a + +/3 # (a + /1 3t) then considering a mixture of (1) and (2) some fraction of instances with c(x, 0) = 0 setting c(z, 0) 1 and on some fraction of instances with c(z, 1) 1 setting č(ι, 1) 0-is suboptimal.
Finally, note that setting some fraction of instances with c(x, 0) = 1 to 0(z, 0) = 0 only increases
the EO, does setting some fraction of instances with c(z, 1) 0 ē(z, 1) 1, and so cannot be part of an optimal classifier.
Finally, an optimal derived EO classifier in which the false-ne rate is B (resp., (0) on both groups is one where false-posee positive rates satisfy (1 a + //30 < (a+ +1 //1 31) (resp., (1 - a + /3" > (a + 1 80) B²)). Under equality, both classifiers are optimal.
15

---

B Additional fairness notions
In this section we discuss extensions of our results to two additional, common notions of fairness. We state the two notions for the setting of a single classifier, extend the definitions to multiple competing classifiers, and then show that versions of Proposition 1 and 4 hold for these notions as well: If the correlations or overlaps in served borrowers between classifiers differ across groups, then classifiers that are fair will not be fair under competition.
The first notion we consider is Equalized Odds (ED), a strengthening of EO, which requires both the classifier's s dalse-negative and false-positive rates to be equal across groups (Hardt et al., 2016). The second we consider Demographic Parity (DP), which requires the total fraction of approved borrowers be equal across groups (Agarwal al., 2018; Dwork al., 2012).
ED: The ED level of is
max E [Xx A)IY y. A = 0] E(c((X, y, 4= 1]]} --1=
A classifier is ED if its ED level is O.
DP: The DP level of eg is E [ex(X, A) = 0] E [cx(X A = 1] A classifier is DP if its DP level is O.
We now generalize the above definitions of fairesss to setting with multiple competing lenders, ED under competition (EDC) and DP under competition (DPC). Fix classifiers c = (c1,..., (1)), and denote by R(x, a) = =EELL DEEL Ce (x. a) the number offers made to borrower with observables (ær, a). Also, denote by (1r a) Pr R[x, a) I], the probability that at least one lender offers borrower with observables (x, a) a loan.
BDC: The EDC level of classifiers c is
max (
{E (()x = y, A = -X) -
Classifiers are EDC if their EDC level is 0.
DPC: The DPC level of classifiers e is |F] [d(x, A)I4 = 0] E[d(x, 4) 1] Classifiers are
DPC if their DPC level
B.1 Equalized Odds under Competition
In this section we show that classifiers that are ED need not be EDC. Consider the following simple observation, which follows from the fact that the definition of ED is strictly stronger than that of EO:
Observation 1 Fix classifiers c == (c,,..., Ct.).
If ct is ED, then it is also EO.
The EDC level of classifiers is at least their EOC level.
Fix two ED classifiers 61 and 62 and define Be, ot, and pa as in Section 3.1. Observation implies the following corollary of Proposition 1.
Corollary For two classifiers e1 and 02 with false-negative rates 31 and B2, the level of EDC at least 01 .2 1pp - p²)). In the worst case, the level of EDC is at least (31, max (0, B B2 - 1}.
Similarly, fix Se, 7/, and 7" in Section 3.2. Observation implies the following corollary of
Proposition 4.
Corollary For two uncorrelated ED classifiers e₁ and 02 with walse rates B1 and B2, the level of EDC is at least
1622 + (7 -
In the worst case, the level of EDC is at least max{ (2) 31,3.
16

---

B.2
Demographic Parity under Competition
In this section we show that classifiers that are DP need not be DPC.
B.2.1 Correlations between classifiers
Fix two DP classifiers C1 and C22 and denote their approval probabilities by ne Pr(cccc 1]. We define two Bernoulli random variables that capture the probability of approval on the groups E A: For each a E A and each & E {1, 2}. let the Bernoulli random variable
C = Cj (X, A) ((Am)).
Each Cg is the output of classifier on random instances with A = and so E [[G]] = Tj. Denote by de // Mi the standard deviation of C² and note that, because t₁ is DP, de is also the standard deviation of C/. Finally, for each let pa denote the Pearson correlation coefficient between Cf and Cg. The Pearson correlation captures the extent to which the approvals of the classifiers correlate with one another.
We now quantify the DPC level of two classifiers using these correlation coefficients, and determine the worst case.
Proposition 5 For two DP classifiers e and 02 with approval probabilities Tn and 122, the level of DPC is 01 02 100 p²l. In the worst case, the level of DPC is min - 1,, 72) max (0, M1 72)
The proof of Proposition 5 is nearly identical to that of Proposition I, with Of replacing B3 1 m78 replacing Be, and without conditioning any of the probabilities and expectations Y 1.
Proposition 5 yields the following simple corollary.
Corollary Suppose = m 72. If 2 1/2, then the worst-case level of DPC is 1 - 17. If 7 S 1/2, then the worst-case level of DPC is
Recall that, under EOC, the more accurate EO classifiers get, the lower the worst-case level of EOC.
With DPC this also holds. To see this, observe that DP classifiers get more cettuattt-specffiici as their false-positive and felse-negative rates approach 0-the correlations pd and p between the classifiers also approach O. By Proposition 5, this implies that the level of DPC approaches O.
B.2.2 Different sets of borrowers
Fix two DP classifiers с₁ and 62 with approval probabilities ? and n2, and Se, 72, and 7at be as in Section 3.2. Consider the following variant of Definition I, which essentially removes the conditioning on =
Definition 2 DP classifiers and e2 with approval probabilities mn and 1/2 uncorrelated (i) for each E,
Pr = A, Y) Sz S
1|(X, A,) S1 S
= (X)
-) -|
and for every v. E S S2 it holds that Pr [c1(x, a) c2 a) = =] = = 172
Proposition 6 For two uncorrelated DP classifiers ej and 02 with approval probabilities mn and m2. the level of DPC is
(n2 - -1) 10mmmm) m) (7f +1) -2) + (7) - 70) m1 -2)) (1)))))]1 In the worst case, the level of EOC is max 7/, - (1 -72).
The proof Proposition 6 is nearly identical to that of Proposition 4, with The replacing 3 and without conditioning any of the probabilities and expectations on Y =
Proposition 6 yields the following simple corollary.
Corollary For two uncorrelated DP classifiers ei and ez with approval probabilities tn = 1/2 = ".
the level of DPC is n (1-1) 1/20 -/
17

---

C Additional experiments
In this section we describe and report the results of additional experiments that ran.
C.1 Effect size
In addition to examining the prevalence of the effect of fairness adjustment on the EOC, we also measured its magnitude. For each run in Experiments 1-3, if the EOC level was higher after faireess adjustment than before, we measured the factor by which the EOC level increased. For each experiment then calculated the average factor by which the EOC level increased (conditional on increasing), as well as the standard error of the mean. Table reports the results: The values in the table the 95% confidence intervals for the factor by which the EOC level increased after adjustment (conditional on increasing).
Table 2: 95%-CI CI for effect size (ratio only)
300
lk
3k
10k
30k
100k
Exp. Exp. 2
[10.8, 30.8]
[7.8, 16.3]
[7.6, 121.0]
[6.0, 19.9]
[7.3, 23.0]
[7.5, 31.0]
[21.1, 111.8]
[8.3, 30.1]
[2.9, 9.4]
[1.8, 2.2]
[1.6, 1.8]
[1.3, 1.4]
Exp. 3
[14.3, 27.2]
[10.0, 74.2]
[8.2, 16.7]
[4.8, 32.4]
[2.5, 3.6]
[1.9, 4.3]
C.2 Experiment with different subsets
In this experiment we simulated a situation in which S # S2, as in Proposition 4. In particular, we simulated the situation described in Example 4, where S supp(D) and S2 supp(D) and where A = is the set of individuals with a mortgage. Classifier was trained on random of examples taken from the entire dataset, whereas classifier C2 was trained on random set of examples consisting only of individuals with mortgage. Both classifiers were logistic regressions, but since only classifier Ci served individuals with both values of A, only that classifier underwent fairness adjustment. As in the previous experiments, we measured the level of EOC before and after the fairness adjustment.
1),
Our results for experiment are reported in Tables and 4, and are the following: For training sets of size up 10k, fairness adjustment leads worse EOC in significant fraction of runs. For larger training sets, however, the effect disappears. This is in line with the previous experiments, except that the effect diminishes for smaller training sets. Even when the effect is prevalent, however, its size is rather small.
Table 3: 95%-CI CT for likelihood EOC level increased following faireess adjustment
300
1k
3k
10k
30k
100k
Exp. 4
[52.2, 60.4]
[34.2, 42.4]
[17.8, 25.2]
[5.4, 10.0]
[0.8, 3.2]
[0.0, 0.0]
Table 4: 95%-CI for effect size (ratio > only)
300
lk
3k
10k
30k
100k
Exp. 4
[1.5, 1.8]
[1.2, 1.3]
[1.1, 1.2]
[1.1, 1.1]
[1.0, 1.1]
[N/A]
C.3
Independent samples
We ran two experiments that were similar to Experiments 1-3, except that the training data was chosen independently for each classifier.
Exp. Both classifiers were decision trees, and their training data was sampled independently. Exp. Classifier Ci was a logistic regression and e2 a decision tree, and their training data was sampled independently.
18

---

The results are reported in Tables 5 and 6.
Table 95%-CI CI harm likelihood following fairness adjustment
300
lk
3k
10k
30k
100k
Exp. 5
[56.2, 65.0)
[57.0, 65.6]
[50.2, 58.8]
[51.0, 59.6]
[41.2, 49.6]
[24.0, 31.4]
Exp. 6
[71.8, 79.2]
[67.0, 74.4]
[55.0, 63.6]
[49.6, 58.4]
[45.0, 54.4]
[28.0, 36.0]
Table 6: 95%-CI for effect size (ratio > only)
300
1k
3k
10k
30k
100k
Exp.
[5.4, 9.4] [10.7, 858.3)
[6.9. 14.0]
[6.4, 44.8]
[7.3, 14.1]
[5.7, 18.5]
[3.2, 18.7]
Exp.
[8.3, 24.5]
[5.8, 10.3]
[11.3, 35.8)
[5.6, 11.4]
[5.7, 12.9]
C.4
Different dataset
We ran Experiments 1-3 on a different dataset from the Lending Club loan data the Adult Census Income dataset from the UC Irvine Machine Learning Repository (Becker and Kohavi, 1996). This dataset contains data on roughly 40,000 individuals. We used simple classifiers to predict whether individual's income is above $50,000, given of features that included age, education, work hours week, occupation, and relationship status. The protected attribute A was gender.
Exp. 1' Classifier 61 was a logistic regression and 0o a decision tree, but their training data was identical.
Exp. 2' on random examples of White individuals, and one on non- Won-White individuals.
Both classifiers were logistic regressions, but their training data was disjoint: one was trained
Exp. 3' Classifier 61 logistic regression and e2 decision tree, and their training data was disjoint (as in Exp. 2).
The results are reported in Tables 7 and 8.
Table 7: 95%-CI CI for harm likelihood following fairness adjustment
300
1k
3k
10k
Exp. 1'
[43.8. 52.6]
[29.6, 37.4]
[13.4, 20.2]
[2.0, 5.2]
Exp. 2'
[52.8, 63.1]
[45.4, 54.4]
[13.6, 20.2]
[0.0, 1.2]
Exp. 3'
[43.2, 52.8]
[39.4, 48.1]
[34.4, 43.0]
[18.4, 25.4]
Table 8: 95%-CI for effect size (ratio > only)
300
1k
3k
10k
Exp. 1'
[3.9, 11.8]
[4.4, 8.8]
[2.2, 14.2]
[1.1, 1.3]
Exp. 2'
[4.7, 7.9]
[4.7, 12.1]
[1.8, 2.5]
2.00
Exp. 3'
[3.1, 26.2]
[4.4, 8.2]
[6.1,
[2.3,
C.5 More than two classifiers
We several experiments that included three, rather than two, classifiers. As before, measured the level of EOC before and after all three underwent a fairness adjustment. In particular, we ran the following three experiments on the Lending Club data:
Exp. Classifier Cj was a logistic regression, C2 decision tree, and 03 a random forest, and their training data was identical.
Exp. 8 All classifiers were logistic regressions, and their training data was independently sampled.
19

---

Exp. 9 Classifier was logistic regression, decision tree, and es random forest, and their training data was independently sampled.
Tables 9 and 10 report the results.
Table 9: 95%-CI for harm likelihood following fairness adjustment
300
lk
3k
10k
30k
100k
Exp. 7
[74.2, 81.2]
[71.2, 78.6]
[60.0, 68.2]
[49.6, 58.2]
[44.6, 53.4]
[28.4, 36.4]
Exp. 8
[80.4, 87.0]
[79.0, 85.8]
[77.4, 84.2]
[65.4, 73.0]
[54.6, 63.4]
[30.8, 39.0]
Exp. 9
[74.2, 81.4]
[73.0, 80.8]
[65.0, 73.2]
[63.6. 71.6]
[52.0, 60.8]
[34.2, 43.2]
Table 10: 95%-CI for effect size (ratio >> only)
300
1k
3k
10k
30k
100k
Exp.
(7.77,
[8.8, 19.1]
[7.5, 19.7]
[6.2, 22.8]
[5.5, 11.6]
[5.4, 22.7]
Exp.
[25.2, 133.7]
[12.4, 32.5]
[6.0, 10.9]
[3.2, 3.7]
2.5] [7.2, 15.2]
[1.5, 1.7]
Exp.
[17.1, 106.7]
[13.6, 26.3]
[9.6, 28.9]
[7.6, 16.1]
[5.9, 14.3]
In addition, Experiments 7', 8", and 9", which were the same 8, and 9 except that they used the Adult Census Income dataset. The results are reported in Tables 11 and 12.
Table 95%-CI for harm likelihood following fairness adjustment
300
1k
10k
30k
Exp. 7'
[41.2, 50.2]
[31.2, 39.0]
[9.8, 15.6]
[0.4, 2.2]
[0.0, 0.0]
Exp. 8'
[49.2, 57.8]
[30.2, 38.0]
[7.8, 12.8]
[0.0, 0.6]
[0.0. 0.0]
Exp. 9'
[47.8, 56.4]
[34.6, 43.2]
[16.6, 23.6]
[5.2, 10.0]
[0.0, 1.0]
Table 12: 95%-CI for effect size (ratio only)
300
1k
3k
10k
30k
Exp. 7'
[2.7,
[2.9, 8.6]
[1.3, 2.0]
[1.1, 1.7]
[N/A]
Exp. 8'
[5.7, 53.2]
[3.5,
2.5)
[1.0, 1.0]
[N/A]
Exp. 9'
[3.8, 15.2]
[3.4, 12.7]
[1.9,
(1.1,
[1.0, 1.1]
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction describe the contributions and point to specific results to justify claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Most complete proofs appear in the paper. The one exception is the proof of Lemma 1, for which there is only a proof outline. The formal proof appears in the technical appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The code used in the experiments is part of the technical appendix, and the data is publicly available.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code used in the experiments is part of the technical appendix, and the
data is publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
onPolicyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /twpss.ccppublll/le/ee wwtpcccppppcccceecc.cccc.ccc.clt /ellic//uiiee httpguuddess//ee https//ooeSSuuiiseeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The "Experiments" section describes the experimental setting. The code used in the experiments is part of the technical appendix, and the data is publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our experiments include confidence intervals for all results, and factors of variability are explained.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
As noted in Section 5, running all experiments took a few hours on home computer.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https pettps:/nneuuu htt////eerurr.eeeee//eee http.///ccc/tthcccccieeeeeeeee
Answer: [Yes]
Justification: We do not believe there were any harms during the research process, and do not foresee harmful consequences of the research.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make eration due to laws or regulations in their jurisdiction).
preserve anonymity (e.g., if there is special consid-
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: The main societal positive societal impact of the paper (and the main argument in the paper) is that current fairness constraints are insufficient, and it advocates for greater care. We do not foresee any negative societal impacts.

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper is largely theoretical and no models are released; the data used is open-source and publicly available.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Data is open open-source and publicly available, and database license is included.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comddataees has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No LLMs were used in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pettpa://neurbae cc/ 20///oneeeenccce22255/////////// for what should or should not be described.
27


Reviews:
[{"review_id": "M63njY6v5z", "reviewer": "Reviewer_V2R5", "summary": "The paper studies the fairness of AI-based decision-making in a multi-agent system, for instance, different financial institutions offering loans or different employers hiring candidates. The paper presents theoretical and empirical results that in such systems fairness of individual systems used by different firms does not guarantee fairness of the overall ecosystem. More simply, even when the classifiers used by different banks or employers for making loan or hiring decisions satisfy fairness criteria such as equal opportunity, the overall system might not result in equal opportunity for different classes. The paper discusses two factors that may result in this behavior – the classifiers used by competing firms may have different correlations on the protected group, or the pool of candidates served by the competing firms may be overlapping, but not identical. The paper also presents experiments that show that imposing fairness adjustments to individual classifiers can worsen the fairness of the ecosystem (compared to when individual classifiers are not adjusted for fairness).", "questions": "Main suggestion: \n\n1. The main paper very briefly (in the last 3 lines of the paper) alludes to the generalized results with more than two lenders and a more general utility function, which are included in the appendix. I would instead recommend that the authors bring this up earlier in the main paper. The detailed proofs can be kept in the appendix, but it would be useful to mention the results in the main paper, as it adds strength to the paper's claims. Furthermore, the empirical experiments in the paper are limited to 2 lender settings, whereas it seems useful and straightforward to try it on more than 2 lenders to see how the empirical findings generalize to a multi-agent setting with many agents. \n\nMain questions: \n\n2. What are the implications of the findings in the paper? The paper raises an important problem that imposing fairness in individual classifiers may degrade fairness across the ecosystem involving “competing” firms. But it is not clear what the practical takeaway should be. Should we be thinking differently about imposing a fairness adjustment? Should we refrain from making any fairness adjustments at all? Do the correlation in classifiers and overlap in applicants exhaustively cover all factors that could lead to a decline in fairness under competition, in which case, this could be used as an early signal that the fairness of the system could be compromised? I would suggest adding a discussion about the implications of the findings in the paper. \n\n3. The result section provides an example where two somewhat biased classifiers result in EOC under competition of zero, that is, the system is fair under competition without any fairness adjustment to the individual classifiers. Was EOC=0 a coincidence, or was this experimentally designed to exhibit that two classifiers can lead to EOC=0 without adjustment, but EOC>0 after adjustment? Generally speaking, the paper discusses what conditions of unbiased classifiers result in a biased system under competition. However, can we define this relationship the other way around? Basically, generalizing the above example, what relationship between the biased classifiers would result in an unbiased system under competition? \n\nOther questions: \n\n4. In “fairness with a single lender” subsection where welfare-equalizing (WE) and equalized opportunity (EO) are defined, the paper mentions that EO is derived from WE by fixing $v(x, y, a, r)=1$ if $y=r=1$, and 0 otherwise, which makes sense. But then, in the definition of WE, why does r seem to be fixed to 1? Isn’t WE supposed to be general, where r can take any value between 1 to |L|?\n\n5. The paper introduces the notation $v(x, a, y, o_1, \\ldots, o_{|L|})$ as “the utility of borrower $(x, a, y)$ when he receives offers from all lenders $l$ with $o_l=1$”. What is $o_l$  here? This notation is not introduced anywhere. After staring at it for a while, I am assuming you mean that $o_l$ is whether lender $l$ makes an offer to the borrower $(x, a, y)$? \n\nMinor formatting issue: \n\n6. The paper mentions a Figure 1 in the results section (line 357), but this is not present in the main paper or the appendix.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The paper discusses a very interesting and useful problem, which, to my knowledge, has not been studied previously. I will add here, though, that I am not intimately familiar with this area, but based on my search, the content of this paper seems to be novel. The theoretical and empirical results presented in the paper support their main claims well and can be impactful for thinking about fairness at the level of the overall ecosystem, rather than the fairness of individual classifiers. Overall, I believe this paper addresses an important question, provides sufficient evidence to support its claims, and provides important insights into fairness in multi-agent systems, which is applicable in many real-world scenarios.  \n\nMy main concerns with the paper involve a lack of intuition and clarity of notation in a few places, which made following the theoretical results a little difficult. The examples included in the paper help understand what scenarios would correspond to different results; however, the \"why\" of it is not immediately clear to me. For instance, intuitively, why does a difference in correlations between classifiers result in a decline in fairness? I would recommend including more intuition along with examples and proofs, and possibly moving complete proofs to the appendix.  \n\nMy other main concern with the presentation is that most assumptions and simplifications are noted in text, making it hard to track all specific assumptions that the theoretical results are based on. For instance, “For simplicity, we will assume henceforth that borrowers’ utilities depend only on the number of offers they received”. Does this mean that the results only follow in these settings, or is it only for simplifying discussion? I would strongly recommend including all assumptions in a well-defined “Assumptions” environment (similar to proposition or corollary) and then referring to specific assumptions needed for each theorem, for instance, “if Assumptions 1 and 2 hold, then …..”", "was_revised": true, "final_justification": "The authors have acknowledged that my concerns regarding the need for more intuition and appropriate exposition of assumptions needed for each result can be addressed in the final version of the paper. Thus, I will maintain my positive recommendation for this paper.", "rebuttal": "Thank you very much for all your comments and suggestions!\n\nIn the following, we first address your questions, and then also your comments from the \"Strengths and Weaknesses\" section.\n\n### Questions\n\n1. These are all very helpful recommendations! We will make the generalized results more prevalent, and will add experiments involving three lenders.\n2. Our main insight is that individual fairness is neither necessary nor sufficient for ecosystem fairness. We typically care about the latter, and so the main implication is that the ecosystem-level is the level at which we should analyze fairness. This is not to say that we should not make any fairness adjustments, but rather that the effects of adjustment on the ecosystem should be considered. Correlation and overlap are exhaustive in the sense that at least one is necessary for individual fairness to lead to unfairness in competition, and so as you note they can be used as early signals. Another way to frame it is to say that we should not complacently accept individual fairness as the end-all be-all of fairness.\n3. Examples 3 and 4 were designed so that $\\mathrm{EOC}=0$ before adjustment, in order to drive home the point that the EOC can increase after adjustment. The experimental section is meant to give a sense of the prevalence of this phenomenon. We agree that it would be an  interesting challenge to uncover the relationship between biased classifiers that lead to EOC. One thought is that it would require the group with greater correlation to also be the one that has lower false-negative rates. Finding a general characterization is beyond the scope of the current paper, but certainly a worthwhile direction for future research.\n4. The definition of WE in that subsection looks at a setting where there is only one isolated classifier, and so $r$ is always either 1 or 0. That is, the utility of a borrower is $v(x,a,y,1)$ when getting an offer, and 0 otherwise. (We use the general utility function with $r$ for consistency with the subsequent subsection with multiple lenders.)\n5. Yes, this is exactly the meaning. We will make it clearer.\n6. This is a typo – it is supposed to be Table 1.\n\n\n### Other comments\n\n**Comment starting with \"My main concerns with the paper...\":** Thanks for this comment. We can certainly add more intuition. For example, for the difference in correlation: under high (low) correlation, if a borrower is misclassified by one classifier, she is likely also (not) misclassified by the other. In the latter case, each deserving borrower intuitively has “two chances” to get an offer, whereas in the former case she only has “one chance”. If in one group deserving borrowers get two chances and in the other they only get one, then in the former there is a lower chance of being misclassified. This is what creates the decline in fairness.\n\n**Comment starting with \"My other main concern with the presentation...\":** The assumption to which you point is the one we use in the main text of the paper. The results do hold in more general settings, which we describe and analyze in the appendix. We will take your advice and be more explicit about this and other assumptions in the paper."}, {"review_id": "KhIaY6HhJG", "reviewer": "Reviewer_vyhi", "summary": "This paper takes an ecosystem perspective on algorithmic fairness, focusing on how fairness is affected by competition. The paper studies this question in a stylized model for lending where there are multiple competing lenders. The key finding is that even if each lender deploys a classifier satisfying fairness, then the ecosystem may not be fair. \n\nThe paper studies in this in a stylized lending model with borrowers and two lenders. Borrowers are captured by observable features, binary protected attributes, and binary outcomes for whether they repay loans. Lenders use a classifier to determine whether to offer a loan to the borrowers based on the features and attributes. They focus on fairness for individual classifiers as given by equal opportunity and generalizations. They focus on ecosystem fairness in terms of the set of all loans available to borrowers, defining an analogue of equal opportunity. \n\nThe paper theoretically demonstrate that even if each lender’s classifier satisfies equal opportunity, the ecosystem need not satisfy equal opportunity under competition. They identify two forces: first, differences in correlations across groups, and second, differences in which borrowers are served. They then show that fairness-improving post-processing of individual classifiers can hurt equal opportunity under competition. They empirically validate these findings on the Lending Club loan dataset.", "questions": "- The paper does not include a discussion of how their work relates to older works that study ecosystem fairness. Can the authors discuss how their papers relate to e.g., [1] and [2]? \n- Is there a different way of dividing responsibility across lenders to guarantee ecosystem fairness? \n\n[1] Dwork and Ilvento. Fairness Under Composition. ITCS 2019.\n\n[2] Bower, Kitche, Niss, Strauss, Vargas, and Venkatasubramanian. Fair Pipelines. FAT/ML 2017.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper shows through a nice mix of theoretical and empirical results that there is a big disconnect between fairness for each individual classifiers and ecosystem fairness. \n- The paper clearly distills two distinct forces which create this disconnect, and connects them to existing intuition. As the authors note, the first force is related to algorithmic monoculture. The second force relates to the size of each lender’s market base.\n- The paper is very well written.  \n\nWeaknesses:\n- The insight that fairness of individual classifiers does not guarantee ecosystem fairness has been established (in different models) in prior work. See e.g., [1] and [2]. This makes the results in this paper a bit less surprising.\n- The results are limited to a stylized lending model. (However, I do not see this is a major weakness, since the key forces are clearly distilled within this model.) \n- While enforcing standard fairness criteria for individual classifiers does not guarantee ecosystem fairness, it is possible that there is a different way of dividing responsibility across lenders (i.e., not using standard fairness criteria) to guarantee ecosystem fairness. The paper does not seem to discuss this possibility.  \n\n[1] Dwork and Ilvento. Fairness Under Composition. ITCS 2019.\n\n[2] Bower, Kitche, Niss, Strauss, Vargas, and Venkatasubramanian. Fair Pipelines. FAT/ML 2017.", "was_revised": true, "final_justification": "After discussions with the AC and other reviewers, I think that this paper provides an interesting analysis of how fairness is affected by competition, and I find the paper to be well-written. However, the analysis is limited to a stylized lending model, and I also didn't find the main message particularly surprising in light of prior work (even though the mechanisms/setting are different, as the authors explained in the author response). My assessment is that this paper is borderline, and I would lean slightly towards acceptance.", "rebuttal": "Thank you for your review of the paper, and for the helpful comments!\n\nIn the following, we first address your questions, and then also your comments from the \"Strengths and Weaknesses\" section.\n\n### Questions\n\n* Thank you for pointing out these papers. As you note, both of them also discuss the insight that fairness of individual classifiers does not imply fairness of a system composed of multiple classifiers. Our paper differs along multiple dimensions. First, in terms of motivation, these papers ask when a central platform (e.g., ad platform) will be fair when handling a task that consists of several subtasks (advertisers on the platform), where each subtask is required to be fair. Our motivation is more about competing firms in a decentralized market, where the joint activity determines the user’s utility. More specific differences are the following:  \n\n    [1] largely focus on individual fairness. They do have some extensions to group fairness, and their main results here are to show that there exist distributions for which individual fairness does not imply joint fairness. However, in their model the classifiers are assumed to always be independent, and they cannot capture the correlations between classifiers that lead to ecosystem unfairness. In addition, their notion of group fairness does not include EO.  \n\n    [2] do focus on EO, but here the main difference is that classifiers are composed sequentially: one classifier makes a prediction or decision, the outcome is then passed on to the next classifier, and so on. In our setting, in contrast, the classifiers run in parallel.  \n\n\n    Finally, although both [1] and [2] have the insight that individual fairness does not suffice for joint fairness, in our paper we also identify the forces that lead to the joint unfairness, and quantify the extent of this unfairness as a function of correlations and overlaps.\n\n* This is a great question. Since our motivating environment is one where lenders compete with one another, we are interested in the kinds of regulations that a regulator could implement in order to guarantee ecosystem fairness. And indeed, dividing responsibility – for example, by allowing only one lender to serve each borrower – would work. However, this would require a high level of centralization, which may not be feasible in a competitive market. In addition, we would lose out on the benefits of having multiple lenders, namely, that the possibility of multiple lenders serving the same borrower leads to fewer borrowers with $y=1$ being denied a loan.\n\n\n### Other comments\n\n**Comment starting with \"The results are limited to a stylized lending model...\":** Our main modeling assumptions would also fit a setting where borrowers are job candidates, insurance buyers,  or student applicants, and lenders are employers, insurers, or colleges, respectively."}, {"review_id": "0fr4uggjJY", "reviewer": "Reviewer_e5fy", "summary": "If two (or more) classifiers are fair (e.g., satisfy equal opportunity), is the ecosystem containing both classifiers also fair? The authors show theoretically and empirically that fairness is not maintained in this setting.\n\nThe authors' main contributions are (1) defining what ecosystem fairness means, showing that ecosystem fairness isn’t guaranteed (2) when individual classifiers have different levels of correlation on different subpopulations or (3) when individual classifiers serve different subpopulations, and (4) empirically showing that the theoretical results hold on a real dataset.", "questions": "* In the experiments, was accuracy comparable across all dataset sizes? In particular, 300 samples seems like it might not be sufficient for good accuracy\n* Are there different definitions of ecosystem outcomes that wouldn't reduce to ensembles with \"one-yes\" voting? \n* The finding that large dataset sizes causes EOC to increase less is interesting. Do you have any thoughts on why this trend occurs?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n* Important problem, real-world relevance.\n* Theoretical contributions are clear. The examples are helpful. \n* The authors are straightforward about limitations (theoretical assumptions), but what they do present is sufficient to illustrate the general idea and additional details on generalization are in the appendix.\n\n### Weaknesses\n* It’s a well-known result that fairness does not always compose [1-2], e.g., in ensembles. The authors should cite this work and describe how their contributions are novel. The problem setups are different but the specific ecosystem framework described in the paper is very related to ensembles, just with a voting mechanism of “one-yes” rather than majority vote. Are there other instantiations of the ecosystem framework that would be distinct from ensembles? Describing these (even if it's not possible to evaluate them experimentally) would help strengthen the paper's novely.\n* Clarity in Section 5 could be improved. Specifically, mention Experiment 4 in the bulleted list with the other experiments and make Figure 1 be self-contained (lines 357-359 describe it this but is hard to find)\n* Experiments only looked at one dataset, and it is a dataset without a true protected attribute (“having a mortgage” feels different than race or gender, because one could make an argument that it could causally impact rather than just correlate with the outcome)\n\n\n[1] Dwork et al., https://arxiv.org/abs/1806.06122 \n\n[2] Bower et al., https://arxiv.org/abs/1707.00391", "was_revised": true, "final_justification": "I'm going to bump up my score to a 5- my biggest concern was the redundancy with previous work on fairness not composing, but the distinctions highlighted by the authors make sense. I'd encourage the authors to include some version of that part of the rebuttal in the camera-ready.", "rebuttal": "We very much appreciate your comments. Thank you!\n\nIn the following, we first address your questions, and then also your comments from the \"Strengths and Weaknesses\" section.\n\n### Questions\n\nOn the accuracy across different dataset sizes: Accuracy improved with dataset size, and indeed, with 300 samples the accuracy was not great. What we found interesting is that our finding in this section, that the EOC-level worsens after fairness adjustment, holds for a range of accuracies, even though the effect decreases somewhat as accuracy increases. \n\nOn definitions of ecosystem outcomes that don't reduce to ensembles with \"one-yes\" voting: Such definitions indeed exist, and, in fact, our utility-based approach provides such a definition. Although in the main text of the paper we assume that borrowers’ utility functions are one-yes – 1 if they have at least one offer, 0 otherwise – our general definition of WEC (middle of page 4) does not reduce to ensembles with one-yes voting. A particular formulation is analyzed in the appendix (Proposition 3), where we extend our results to a utility function that equals 0 if a borrower obtains no offers, 1 if she obtains 1 offer, and $k\\in[0,\\infty)$ if she obtains 2 offers.\n\nOn the finding that large dataset sizes causes EOC to increase less: One guess about why this trend occurs is that this has to do with the classifiers’ accuracies. As the dataset size increases, the classifiers become more accurate. At the limit, when they are perfectly accurate, fairness is not really an issue – neither before nor after adjustment. Perhaps in these high-accuracy regimes the fairness adjustment does not do much, since the classifiers are already close to being fair.\n\n### Other comments\n\n**Comment starting with \"It’s a well-known result...\":** Thank you for pointing out these papers. As you note, both of them also discuss the insight that fairness of individual classifiers does not imply fairness of a system composed of multiple classifiers. Our paper differs along multiple dimensions. First, in terms of motivation, these papers ask when a central platform (e.g., ad platform) will be fair when handling a task that consists of several subtasks (advertisers on the platform), where each subtask is required to be fair. Our motivation is more about competing firms in a decentralized market, where the joint activity determines the user’s utility. More specific differences are the following:\n\n[1] largely focus on individual fairness. They do have some extensions to group fairness, and their main results here are to show that there exist distributions for which individual fairness does not imply joint fairness. However, in their model the classifiers are assumed to always be independent, and they cannot capture the correlations between classifiers that lead to ecosystem unfairness. In addition, their notion of group fairness does not include EO.  \n\n[2] do focus on EO, but here the main difference is that classifiers are composed sequentially: one classifier makes a prediction or decision, the outcome is then passed on to the next classifier, and so on. In our setting, in contrast, the classifiers run in parallel.  \n\nFinally, although both [1] and [2] have the insight that individual fairness does not suffice for joint fairness, in our paper we also identify the forces that lead to the joint unfairness, and quantify the extent of this unfairness as a function of correlations and overlaps.\n\n**Comment starting with \"Clarity in Section 5...\":** Thanks, we'll make these changes.\n\n**Comment starting with \"Experiments only looked at one dataset...\":** Following this suggestion (also made by another reviewer), we ran the same experiments on a different dataset: The Adult Income dataset from the UC Irvine Machine Learning Repository. The dataset is smaller, and contains 48,000 instances. We ran our experiments to predict whether or not income is above \\$50,000, using age, education, work hours per week, occupation, and family status as features. The protected attribute was gender. The results were broadly similar to our experiment, in terms of orders-of-magnitude. For brevity, we report here the 95%-CIs for a sample size of 3000:\n\n> Exp1:  [13.4, 20.2]\n>\n> Exp2:  [34.4, 43.0]\n>\n> Exp3:  [45.4, 54.4] \n>\n> Exp4:  [61.8, 69.8]\n\nWe plan to include the full set of results in the paper."}, {"review_id": "QaVN7CZv6C", "reviewer": "Reviewer_7AYg", "summary": "This work proposes a novel concept of *fairness under competition*, where the goal is not only to study the *fairness of individual predictive models* but to analyze the overall fairness of the *deployment ecosystem* where different competing decision-makers may own such varying predictive models. The authors identify two primary sources of ecosystem unfairness in the presence of multiple predictive models: i) the correlation of the prediction scores between the classifiers, and ii) the data overlap between the different models in deployment. Using illustrative theoretical examples, the authors further show how certain fairness modifications on the predictive models can improve the fairness of *individual predictors* but deteriorate the *ecosystem's overall fairness*. The authors conduct experiments on one real-world dataset to show how different ecosystem setups and varying data sizes can possibly impact the fairness under competition.", "questions": "1. Are there any fairness notions that the general welfare formulation cannot capture?\n2. Can the authors comment on the potential relationship of the different decision-makers using different subsamples and selection labeling? If the two subsamples are i.i.d., is there still a chance of having ecosystem unfairness stemming from it?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strenghts\n\n- This work proposes a novel perspective on the well-established research space of algorithmic fairness. While most works focus on studying the fairness of individual predictive models, this work provides a novel approach to studying the fairness of the entire market ecosystem.\n- The theoretical framework helps to identify two critical components that can affect ecosystem unfairness. This aspect is important in improving the practical applicability of these new notions and informing potential mitigation strategies.\n- The authors provide some theoretical analysis in the appendix regarding how multiple classifiers in an ecosystem can impact the ecosystem's unfairness. \n\n\n## Weaknesses\n\n- The evaluations are limited, focusing on only one dataset and very specific subsamples from the underlying data distribution. A more thorough evaluation would have helped. Specifically, it would also be interesting to study the effects across multiple datasets, in the presence of multiple classifiers, and for different subsampling of data by each of the classifiers.\n- The factors discussed by the authors that impact ecosystem unfairness are also related to other well-studied aspects like selective labeling, predictive multiplicity, etc. For instance, the gap in overall fairness from different subsampling of data is very related to the issue of the selective labeling problem and how that interacts with fairness issues.\n- The analyses are conducted using only one fairness notion (equal opportunity). While this notion is very relevant to the lending scenario explained in the paper, it would be helpful to understand further ecosystem unfairness from the lens of other fairness notions and their empirical evaluations. \n- The evaluations reported the likelihood of individual fairness interventions worsening the overall ecosystem unfairness; however, one cannot infer the magnitude of the worsening of the ecosystem unfairness.", "was_revised": true, "final_justification": "The authors proposed a novel paradigm to study fairness where the focus is not a single predictive or decision-making model but an entire ecosystem, as is the case in most real-world settings. This paradigm can open new directions to pursue in algorithmic fairness. Through the rebuttal, the authors also conducted additional evaluations that further strengthen their claims and provide more empirical grounding.", "rebuttal": "Thank you for all your comments and suggestions!\n\nIn the following, we first address your questions, and then also your comments from the \"Strengths and Weaknesses\" section.\n\n### Questions\n\n1. The general welfare formulation captures existing notions of group fairness, but cannot capture notions of individual fairness.\n\n2. The possible connection to selection labeling is interesting, but as of now we are not able to draw a clear analogy. However, we did run our experiment with two i.i.d. subsamples, and the results were very similar to the experiments in the paper in terms of ecosystem unfairness. More specifically, we ran two additional experiments – in ExpA there were two identical classifiers (decision trees), using iid data, and in ExpB there were two different classifiers (decision tree and logistic regression), using iid data. The results were the following. (We report only sample size 30k for brevity, with 95% CIs and also effect size. The effect size is described below, in the reply to the comment starting with \"The evaluations reported...\").\n\n    > ExpA: 45.4% [41.2%, 49.6%]  | effect size: 10.90x [5.73, 18.53]\n    >\n    > ExpB: 49.8% [45.0%, 54.4%]  | effect size: 8.06x [5.56, 11.35]\n\n    We plan to include the full set of new experimental results in the paper.\n\n\n### Other comments\n\n**Comment starting with \"The evaluations are limited...\":** Following this suggestion (also made by another reviewer), we ran the same experiments on a different dataset: The Adult Income dataset from the UC Irvine Machine Learning Repository. The dataset is smaller, and contains 48,000 instances. We ran our experiments to predict whether or not income is above \\$50,000, using age, education level, work hours per week, occupation, and family status as features. The protected attribute was gender. The results were broadly similar to our experiment, in terms of orders-of-magnitude. For brevity, we report here the 95%-CIs for a sample size of 3000:\n\n> Exp1:  [13.4, 20.2]\n>\n> Exp2:  [34.4, 43.0]\n>\n> Exp3:  [45.4, 54.4] \n>\n> Exp4:  [61.8, 69.8]\n\n\n**Comment starting with \"The analyses are conducted using only one fairness notion...\":** We agree. However, the other relevant fairness notions we considered also introduced additional subtleties, which we felt crowded out the main message of the current paper. It is certainly a direction we are pursuing in subsequent work.\n\n**Comment starting with \"The evaluations reported...\":** Following this comment, we ran the experiments again, this time recording the magnitudes of the effect. Specifically, for each experiment and sample size, we look at all the runs where the ecosystem fairness worsened, calculated the factor by which it worsened, and took an average of these factors. The results for sample size 30k were the following, with 95% CIs. (Other sizes were similar, reporting here only one size for brevity.)\n\n> Exp1: 13.82x [7.25, 22.98]\n>\n> Exp2: 1.66x [1.57, 1.76]\n>\n> Exp3: 3.05x [2.54, 3.62]\n>\n> Exp4: 1.06x [1.03, 1.08]\n\nInterestingly, in Exp1 the level of unfairness grew by an order or magnitude, in Exp2 and Exp3 it grew by a small factor, and in Exp4 it barely grew."}]

Author rebuttal:

Meta-review: ## Summary

The paper seeks to model fairness "under competition". In this setting, k competing lenders decide who to offer loans to from a population. A borrower, receiving between 0 and k offers from lenders, then has to decide what to do and accrue utility based on that. The question the authors ask is:  is it possible for individual lenders to implement loan offer schemes that are fair, but that overall the outcome from borrowers in the marketplace is not. They introduce a measure of equal opportunity under competition by asking if, for each demographic group of concern, if the fraction of borrowers receiving at least one offer is more or less the same. 

Their conclusions are that indeed fairness "does not aggregate": there are scenarios (and they describe two) where lenders competing in the marketplace with algorithms that look like they are fair on their own can actually lead to unfair outcomes for borrowers. They back this up with experimental evidence showing that adding fairness adjustments to individal lender algorithms could actually make things worse. 

## Strengths
Reviewers generally appreciated the question being investigated, and the theoretical modeling used to answer it. The paper is relatively well written, and the theory, examples, and empirical results all reinforce each other well. 

## Weaknesses
Reviewers were unconvinced that the results were surprising, and cited (almost unanimously and independently) prior work on fairness under composition that makes similar observations (namely, that when fair classifiers compose, the results will in general not be fair). The reviewers were a little concerned that the theoretical model was too stylized to be predictive of the real world, and were therefore a little concerned about the relevance of the results. One way this concern played out was in reviewer questions about how to act on the results of the paper, and what recommendations the authors would have for better design to support fairness in ecosystems. 

## Author Discussion
The authors had a robust and fruitful discussion with the reviewers, where all the above points were addressed. The authors in my view were a little too hasty to dismiss the comparison to prior work. It felt like they were trying to defend their claims of novelty when the reviewers merely wanted them to place their work in the right context. I'd encourage the authors to not just insert this discussion in the conclusions (as they indicated) but actually in the related work section where it's more appropriate, and also as context setting for the broader claim of "individual fairness actions don't lead to global fairness". 

In other regards, the reviewers were satisfied with the explanations provided by the authors, and the authors have incorporated changes based on the discussion. 

## Justification
My overall assessment is that the paper adds a useful dimension to the broader question of how fairness methods work in an ecosystem, and their results are interesting. The reviewers shared this view, and the discussion was productive. All of these factors influence my final decision.
Decision: Accept (poster)