Title: Fairness under Competition
Venue: NeurIPS 2025
Accepted: yes
Keywords: fairness

classification

strategic machine learning

competition
Abstract: Algorithmic fairness has emerged as a central issue in ML, and it has become standard practice to adjust ML algorithms so that they will satisfy fairness requirements such as Equal Opportunity. In this paper we consider the effects of adopting such fair classifiers on the overall level of _ecosystem fairness_. Specifically, we introduce the study of fairness with competing firms, and demonstrate the failure of fair classifiers in yielding fair ecosystems. Our results quantify the loss of fairness in systems, under a variety of conditions, based on classifiers' correlation and the level of their data overlap. We show that even if competing classifiers are individually fair, the ecosystem's outcome may be unfair; and that adjusting biased algorithms to improve their individual fairness may lead to an overall decline in ecosystem fairness. In addition to these theoretical results, we also provide supporting experimental evidence. Together, our model and results provide a novel and essential call for action.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a novel concept of *fairness under competition*, where the goal is not only to study the *fairness of individual predictive models* but to analyze the overall fairness of the *deployment ecosystem* where different competing decision-makers may own such varying predictive models. The authors identify two primary sources of ecosystem unfairness in the presence of multiple predictive models: i) the correlation of the prediction scores between the classifiers, and ii) the data overlap between the different models in deployment. Using illustrative theoretical examples, the authors further show how certain fairness modifications on the predictive models can improve the fairness of *individual predictors* but deteriorate the *ecosystem's overall fairness*. The authors conduct experiments on one real-world dataset to show how different ecosystem setups and varying data sizes can possibly impact the fairness under competition.\n\nSTRENGTHS AND WEAKNESSES:\n## Strenghts\n\n- This work proposes a novel perspective on the well-established research space of algorithmic fairness. While most works focus on studying the fairness of individual predictive models, this work provides a novel approach to studying the fairness of the entire market ecosystem.\n- The theoretical framework helps to identify two critical components that can affect ecosystem unfairness. This aspect is important in improving the practical applicability of these new notions and informing potential mitigation strategies.\n- The authors provide some theoretical analysis in the appendix regarding how multiple classifiers in an ecosystem can impact the ecosystem's unfairness. \n\n\n## Weaknesses\n\n- The evaluations are limited, focusing on only one dataset and very specific subsamples from the underlying data distribution. A more thorough evaluation would have helped. Specifically, it would also be interesting to study the effects across multiple datasets, in the presence of multiple classifiers, and for different subsampling of data by each of the classifiers.\n- The factors discussed by the authors that impact ecosystem unfairness are also related to other well-studied aspects like selective labeling, predictive multiplicity, etc. For instance, the gap in overall fairness from different subsampling of data is very related to the issue of the selective labeling problem and how that interacts with fairness issues.\n- The analyses are conducted using only one fairness notion (equal opportunity). While this notion is very relevant to the lending scenario explained in the paper, it would be helpful to understand further ecosystem unfairness from the lens of other fairness notions and their empirical evaluations. \n- The evaluations reported the likelihood of individual fairness interventions worsening the overall ecosystem unfairness; however, one cannot infer the magnitude of the worsening of the ecosystem unfairness.\n\nQUESTIONS:\n1. Are there any fairness notions that the general welfare formulation cannot capture?\n2. Can the authors comment on the potential relationship of the different decision-makers using different subsamples and selection labeling? If the two subsamples are i.i.d., is there still a chance of having ecosystem unfairness stemming from it?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIf two (or more) classifiers are fair (e.g., satisfy equal opportunity), is the ecosystem containing both classifiers also fair? The authors show theoretically and empirically that fairness is not maintained in this setting.\n\nThe authors' main contributions are (1) defining what ecosystem fairness means, showing that ecosystem fairness isn’t guaranteed (2) when individual classifiers have different levels of correlation on different subpopulations or (3) when individual classifiers serve different subpopulations, and (4) empirically showing that the theoretical results hold on a real dataset.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* Important problem, real-world relevance.\n* Theoretical contributions are clear. The examples are helpful. \n* The authors are straightforward about limitations (theoretical assumptions), but what they do present is sufficient to illustrate the general idea and additional details on generalization are in the appendix.\n\n### Weaknesses\n* It’s a well-known result that fairness does not always compose [1-2], e.g., in ensembles. The authors should cite this work and describe how their contributions are novel. The problem setups are different but the specific ecosystem framework described in the paper is very related to ensembles, just with a voting mechanism of “one-yes” rather than majority vote. Are there other instantiations of the ecosystem framework that would be distinct from ensembles? Describing these (even if it's not possible to evaluate them experimentally) would help strengthen the paper's novely.\n* Clarity in Section 5 could be improved. Specifically, mention Experiment 4 in the bulleted list with the other experiments and make Figure 1 be self-contained (lines 357-359 describe it this but is hard to find)\n* Experiments only looked at one dataset, and it is a dataset without a true protected attribute (“having a mortgage” feels different than race or gender, because one could make an argument that it could causally impact rather than just correlate with the outcome)\n\n\n[1] Dwork et al., https://arxiv.org/abs/1806.06122 \n\n[2] Bower et al., https://arxiv.org/abs/1707.00391\n\nQUESTIONS:\n* In the experiments, was accuracy comparable across all dataset sizes? In particular, 300 samples seems like it might not be sufficient for good accuracy\n* Are there different definitions of ecosystem outcomes that wouldn't reduce to ensembles with \"one-yes\" voting? \n* The finding that large dataset sizes causes EOC to increase less is interesting. Do you have any thoughts on why this trend occurs?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper takes an ecosystem perspective on algorithmic fairness, focusing on how fairness is affected by competition. The paper studies this question in a stylized model for lending where there are multiple competing lenders. The key finding is that even if each lender deploys a classifier satisfying fairness, then the ecosystem may not be fair. \n\nThe paper studies in this in a stylized lending model with borrowers and two lenders. Borrowers are captured by observable features, binary protected attributes, and binary outcomes for whether they repay loans. Lenders use a classifier to determine whether to offer a loan to the borrowers based on the features and attributes. They focus on fairness for individual classifiers as given by equal opportunity and generalizations. They focus on ecosystem fairness in terms of the set of all loans available to borrowers, defining an analogue of equal opportunity. \n\nThe paper theoretically demonstrate that even if each lender’s classifier satisfies equal opportunity, the ecosystem need not satisfy equal opportunity under competition. They identify two forces: first, differences in correlations across groups, and second, differences in which borrowers are served. They then show that fairness-improving post-processing of individual classifiers can hurt equal opportunity under competition. They empirically validate these findings on the Lending Club loan dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper shows through a nice mix of theoretical and empirical results that there is a big disconnect between fairness for each individual classifiers and ecosystem fairness. \n- The paper clearly distills two distinct forces which create this disconnect, and connects them to existing intuition. As the authors note, the first force is related to algorithmic monoculture. The second force relates to the size of each lender’s market base.\n- The paper is very well written.  \n\nWeaknesses:\n- The insight that fairness of individual classifiers does not guarantee ecosystem fairness has been established (in different models) in prior work. See e.g., [1] and [2]. This makes the results in this paper a bit less surprising.\n- The results are limited to a stylized lending model. (However, I do not see this is a major weakness, since the key forces are clearly distilled within this model.) \n- While enforcing standard fairness criteria for individual classifiers does not guarantee ecosystem fairness, it is possible that there is a different way of dividing responsibility across lenders (i.e., not using standard fairness criteria) to guarantee ecosystem fairness. The paper does not seem to discuss this possibility.  \n\n[1] Dwork and Ilvento. Fairness Under Composition. ITCS 2019.\n\n[2] Bower, Kitche, Niss, Strauss, Vargas, and Venkatasubramanian. Fair Pipelines. FAT/ML 2017.\n\nQUESTIONS:\n- The paper does not include a discussion of how their work relates to older works that study ecosystem fairness. Can the authors discuss how their papers relate to e.g., [1] and [2]? \n- Is there a different way of dividing responsibility across lenders to guarantee ecosystem fairness? \n\n[1] Dwork and Ilvento. Fairness Under Composition. ITCS 2019.\n\n[2] Bower, Kitche, Niss, Strauss, Vargas, and Venkatasubramanian. Fair Pipelines. FAT/ML 2017.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the fairness of AI-based decision-making in a multi-agent system, for instance, different financial institutions offering loans or different employers hiring candidates. The paper presents theoretical and empirical results that in such systems fairness of individual systems used by different firms does not guarantee fairness of the overall ecosystem. More simply, even when the classifiers used by different banks or employers for making loan or hiring decisions satisfy fairness criteria such as equal opportunity, the overall system might not result in equal opportunity for different classes. The paper discusses two factors that may result in this behavior – the classifiers used by competing firms may have different correlations on the protected group, or the pool of candidates served by the competing firms may be overlapping, but not identical. The paper also presents experiments that show that imposing fairness adjustments to individual classifiers can worsen the fairness of the ecosystem (compared to when individual classifiers are not adjusted for fairness).\n\nSTRENGTHS AND WEAKNESSES:\nThe paper discusses a very interesting and useful problem, which, to my knowledge, has not been studied previously. I will add here, though, that I am not intimately familiar with this area, but based on my search, the content of this paper seems to be novel. The theoretical and empirical results presented in the paper support their main claims well and can be impactful for thinking about fairness at the level of the overall ecosystem, rather than the fairness of individual classifiers. Overall, I believe this paper addresses an important question, provides sufficient evidence to support its claims, and provides important insights into fairness in multi-agent systems, which is applicable in many real-world scenarios.  \n\nMy main concerns with the paper involve a lack of intuition and clarity of notation in a few places, which made following the theoretical results a little difficult. The examples included in the paper help understand what scenarios would correspond to different results; however, the \"why\" of it is not immediately clear to me. For instance, intuitively, why does a difference in correlations between classifiers result in a decline in fairness? I would recommend including more intuition along with examples and proofs, and possibly moving complete proofs to the appendix.  \n\nMy other main concern with the presentation is that most assumptions and simplifications are noted in text, making it hard to track all specific assumptions that the theoretical results are based on. For instance, “For simplicity, we will assume henceforth that borrowers’ utilities depend only on the number of offers they received”. Does this mean that the results only follow in these settings, or is it only for simplifying discussion? I would strongly recommend including all assumptions in a well-defined “Assumptions” environment (similar to proposition or corollary) and then referring to specific assumptions needed for each theorem, for instance, “if Assumptions 1 and 2 hold, then …..”\n\nQUESTIONS:\nMain suggestion: \n\n1. The main paper very briefly (in the last 3 lines of the paper) alludes to the generalized results with more than two lenders and a more general utility function, which are included in the appendix. I would instead recommend that the authors bring this up earlier in the main paper. The detailed proofs can be kept in the appendix, but it would be useful to mention the results in the main paper, as it adds strength to the paper's claims. Furthermore, the empirical experiments in the paper are limited to 2 lender settings, whereas it seems useful and straightforward to try it on more than 2 lenders to see how the empirical findings generalize to a multi-agent setting with many agents. \n\nMain questions: \n\n2. What are the implications of the findings in the paper? The paper raises an important problem that imposing fairness in individual classifiers may degrade fairness across the ecosystem involving “competing” firms. But it is not clear what the practical takeaway should be. Should we be thinking differently about imposing a fairness adjustment? Should we refrain from making any fairness adjustments at all? Do the correlation in classifiers and overlap in applicants exhaustively cover all factors that could lead to a decline in fairness under competition, in which case, this could be used as an early signal that the fairness of the system could be compromised? I would suggest adding a discussion about the implications of the findings in the paper. \n\n3. The result section provides an example where two somewhat biased classifiers result in EOC under competition of zero, that is, the system is fair under competition without any fairness adjustment to the individual classifiers. Was EOC=0 a coincidence, or was this experimentally designed to exhibit that two classifiers can lead to EOC=0 without adjustment, but EOC>0 after adjustment? Generally speaking, the paper discusses what conditions of unbiased classifiers result in a biased system under competition. However, can we define this relationship the other way around? Basically, generalizing the above example, what relationship between the biased classifiers would result in an unbiased system under competition? \n\nOther questions: \n\n4. In “fairness with a single lender” subsection where welfare-equalizing (WE) and equalized opportunity (EO) are defined, the paper mentions that EO is derived from WE by fixing $v(x, y, a, r)=1$ if $y=r=1$, and 0 otherwise, which makes sense. But then, in the definition of WE, why does r seem to be fixed to 1? Isn’t WE supposed to be general, where r can take any value between 1 to |L|?\n\n5. The paper introduces the notation $v(x, a, y, o_1, \\ldots, o_{|L|})$ as “the utility of borrower $(x, a, y)$ when he receives offers from all lenders $l$ with $o_l=1$”. What is $o_l$  here? This notation is not introduced anywhere. After staring at it for a while, I am assuming you mean that $o_l$ is whether lender $l$ makes an offer to the borrower $(x, a, y)$? \n\nMinor formatting issue: \n\n6. The paper mentions a Figure 1 in the results section (line 357), but this is not present in the main paper or the appendix.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}