Formalising Anti-Discrimination Law in Automated Decision Systems

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 15:27:43

Abstract: We study the legal challenges in automated decision-making by analysing conventional algorithmic fairness approaches and their alignment with anti-discrimination law in the United Kingdom and other jurisdictions based on English common law. By translating principles of anti-discrimination law into a decision-theoretic framework, we formalise discrimination and propose a new, legally informed approach to developing systems for automated decision-making. Our investigation reveals that while algorithmic fairness approaches have adapted concepts from legal theory, they can conflict with legal standards, highlighting the importance of bridging the gap between automated decisions, fairness, and anti-discrimination doctrine.

Main paper: 
 introduction: 
 Automated decision-making using predictive models is becoming increasingly important in many areas of society, including lending [60, 100, 107] , criminal justice [31, 14, 151] , hiring [64, 59, 25] , and welfare eligibility [41, 56, 113] . Instances of large-scale failures, from disproportionately harming vulnerable people in welfare eligibility assessments [113] to bias in consumer lending [80] , highlight the need for lawful implementation. Scrutiny of ML-based decisions is heightened by concerns about replicating human biases and historical inequality [97, 41, 93] .Concerns about algorithmic bias have spurred research into fair ML. Early discourse on fairness in ML was relatively narrow due to technical constraints [29, 61] . More recently, researchers have developed formal definitions of fairness in algorithmic decisions and methods to measure fairness in predictive models [49, 31, 27, 147, 87, 85] . Algorithmic fairness definitions generally measure prediction disparities across groups with different legally protected characteristics [90, 136, 87, 17] .This research has resulted in several proposals, including statistical metrics to assess the fairness of individual predictive models [136, 111, 22, 24] , fairness for model auditing [70, 103, 63, 89, 98] , and fairness constraints on models [31, 148, 50, 145, 12] . These criteria simplify fairness into measurements of disparity that do not inherently map to unlawful discrimination. The usefulness of these metrics in practice is limited as incomplete or even irrelevant measures for legal investigations. There have been important efforts to bridge the gap between legal and technical approaches to fair ML [81, 55, 51, 144, 139, 1, 46] . Lawyers have highlighted the challenges of the narrow construction of fairness metrics focusing on disparity in predictions rather than more nuanced definitions of discriminatory conduct and the broader context of the automated decision-making process [55, 51, 144, 1] . We aim to contextualise and formalise legal concepts of algorithmic discrimination beyond the narrow construction of statistical disparity.The predominance of US analysis of fairness and discrimination in ML, lack of non-US ML datasets [78] , and the limited legal scholarship translating these concepts, has inadvertently fostered a series of misconceptions that pervade the field. However, very few papers have engaged with Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.anti-discrimination laws outside of the United States [143, 139, 1, 140, 67, 76] . We aim to introduce new principles and methods to deal with the issues identified in this literature. By avoiding the nuanced legal realities of other jurisdictions, models designed to comply with US laws may breach UK laws or those in comparable jurisdictions. Our paper addresses this gap by providing a rigorous analysis of UK discrimination law, correcting some mischaracterisations, and establishing a more accurate foundation for developing fair ML in the UK and its related jurisdictions.Let x i ∈ R p be a vector of observed attributes for individual i. A decision-maker must choose a decision a ∈ A, where A is closed. Further, we assume that the decision-maker wants to decide based on a future outcome y i ∈ Y for individual i. Here, we assume Y = N, which can be relaxed.Decision-making under uncertainty has long been studied in statistical decision theory [108, 32, 13, 99] . Let u(y, a) be a utility function that summarises the utility for the decision-maker. The optimal decision is then a ⋆ = arg max a∈A y∈Y u(a, y)p(y|a) .(EQUATIONThe decision-maker usually neither knows y i nor p(y|a) at the time of the decision. Hence, the decision must be based solely on x i . In an SML setting, a prediction model p(y|x) is trained to compute the predicted probability distribution (pmf) πi = p(y|x i ) for individual i, with the support on Y. Further, let ŷ( πi ) ∈ Y be the classification made based on πi . In simple settings, the decision can be formulated as a decision function d( πi ) ∈ A that is used to choose an appropriate action based on πi . In the binary y and a case, it reduces to a simple threshold τ , i.e., d(π) = I(π ≤ τ ),where I is the indicator function and πi = p(y = 1 | x i ). We often train a model p(y|x) based on previous data D = (y, X), drawn from a population p(y, x), where both x i and y i are known.Replacing p(y|x i ) with the predictive model p(y|x i ) in Eq. 1 gives an optimal decision.To define algorithmic fairness, we separate x i into protected and legitimate features x i = (x pi , x li );we drop i to simplify notation. Here, x p ∈ C indicates protected attributes, with C being the set of different groups. Legally protected characteristics commonly identified in datasets include gender, race, and age. Many fairness metrics aim to evaluate the fairness of an SML model for commonly identified protected characteristics in datasets, including gender and race [49, 31, 82, 85] .Statistical parity, or demographic parity, is one of the central algorithmic fairness metrics [31, 136, 90, 82] . For statistical parity to hold, it requires thatEQUATIONsuch that the model predictions, in expectation over x, need to be the same for the different groups [31, 136] . Given that the decision function d(π) is the same for the different groups, statistical parity results in equal decisions for the different groups. However, we discuss later in this paper that, in practice, statistical parity may exacerbate inequality or even result in unlawful discrimination [10, 74, 63] .Conditional statistical parity extends statistical parity to account for legitimate features x l . The model predictions should only differ across protected groups to the extent that the difference is conditional on legitimate factors [31, 136, 23] . This can be formalised as,EQUATIONso that, conditional on legitimate features x l , there should not be any difference in predictions between groups given by the protected attribute. Below, we discuss the legitimacy of variables that correlate to protected attributes [34, 77] .Other similar group comparison metrics have been proposed, such as error parity, balanced classification rate, and equalised odds [49, 31, 136, 90, 82, 30] . Also, more individual approaches to parity have considered whether otherwise identical individuals are treated differently if they have different protected attributes [34, 68] . Finally, ideas from causal inference and counterfactual analysis have also been proposed to measure outcome consistency for individuals across protected groups [75, 69, 106, 149, 26, 142, 92, 6] The algorithmic fairness literature largely identifies statistical disparities in predicted outcomes for binary marginalised groups. Legally, discrimination is both broader and more detailed. Not all actions perceived as discriminatory are unlawful, and some non-obvious actions may be prohibited.Anti-discrimination law only applies to select duty-bearers in certain conditions [65] . Individuals' friendship choices being based on race are not legally regulated, despite sometimes seeming unfair [36, 65] . It only applies to protected attributes. An algorithm that rejects a loan application because the applicant uses an Android phone rather than an iOS device may seem unfair because it does not reflect the true default risk but is a proxy for the applicant's income [2, 76] . However, in isolation, this would not be unlawful discrimination under UK law because poverty is not a protected attribute [96] .The prohibition on discrimination traces its legal roots to the Universal Declaration of Human Rights, which established equality and freedom from discrimination as fundamental human rights, further advanced in several international treaties [132, 88] , and enacted as legislation worldwide spurred by the Civil Rights Movement [86, 65] . The United Kingdom implemented several anti-discrimination laws in the 20th century [118, 119, 116 fairness literature has often oversimplified these protected characteristics as simply identifying visible traits when each has complex social meanings [57] . One complexity is, for example, the difference between a person with a protected attribute by biological fact or by identifying with a protected group [73] . UK anti-discrimination law distinguishes between direct discrimination and indirect discrimination. While analogous to the US disparate treatment and disparate impact doctrine, there are important distinctions, meaning they should not be so easily elided [1] .Direct discrimination occurs when an individual is treated less favourably than another based on a protected characteristic [40, s 13]. To establish direct discrimination, it is necessary to identify the specific protected characteristic involved, demonstrate the less favourable treatment (by real or hypothetical comparison), and prove that this treatment was caused "but for" the protected attribute.The intention of the decision-maker is not required or necessary [123, 131] .Indirect discrimination refers to a policy, criterion, or practice (PCP) that disproportionately disadvantages a group with a particular protected attribute compared to those without [40, s 19]. To prove indirect discrimination, one must identify such a PCP, show that it puts a group defined by its protected attribute at a particular disadvantage compared to those without such attribute, and evaluate whether it is justifiable as a proportionate means of achieving a legitimate aim.English common law is either in force or is the dominant influence in 80 legal systems that govern approximately 2.8 billion people, not including the US [28] . UK anti-discrimination law is very similar to numerous Commonwealth and common law jurisdictions, including Australia [3] , Canada [20] ,India [47], New Zealand [91] , South Africa [110] , and the pending bill in Bangladesh [9] . EuropeanUnion law also has broadly the same principles and discrimination case law evolved in parallel during the UK's membership [45] . It is increasingly important to gain a nuanced understanding of unlawful discrimination in AI systems as new laws aim to prevent future harms [44, 16] .This paper makes four core contributions at the intersection of automated decision-making, fairness, and anti-discrimination doctrine.1. We formalise critical aspects of anti-discrimination doctrine into decision-theoretic formalism.2. We analyse the legal role of the data-generating process (DGP) and develop the DGP as a theoretical framework to formalise the legitimacy of the prediction target y and the features x in supervised models for automated decisions.3. Further, we consider the legal and practical effects of approximating the DGP in supervised models. We propose conditional estimation parity as a new, legally informed target.4. Finally, we provide recommendations on creating SML models that minimise the risk of unlawful discrimination in automated decision-making.Our paper is formally limited to analysing and providing novel recommendations for the UK. While we discuss related jurisdictions that are functionally similar and based on English common law, specific legal advice should be followed with respect to different jurisdictions. Accountability varies by jurisdiction and context, which is why our paper underscores the importance of careful, informed classification by experts with appropriate legal advice.2 Automated Decisions and Discrimination 
legitimacy of true differences: 
 In SML, it is crucial to differentiate unlawful discrimination from mere statistical disparities and concepts of algorithmic fairness. While formal equality may map to statistical parity, anti-discrimination laws in the UK and related jurisdictions aim to achieve substantive equality. Despite the general rule that individuals should not receive less favourable treatment based on their protected attributes, courts acknowledge that treating all groups the same can actually disadvantage a protected group and minimise important structural and true differences [137, 143] . Therefore, substantive equality may sometimes require legitimate differential treatment because of the true differences among individuals [137, 52, 144, 139] .For instance, insurance decisions that might otherwise be construed as discriminatory -specifically concerningTherefore, an important aspect from the legal perspective that is overlooked in the existing literature is the distinction between a "true data-generating" process (DGP) and the estimated model p(y|x).To formalise, we assume that there exists a true DGP, D ∼ p(y, x), where D i = (y i , x i ). Further, we use p(y|x true i ) to denote the true probability (pmf) for individual i, given the true features x true i .We make multiple observations on the role of the "true" model and its use in connecting predictive modelling and legal reasoning.First, understanding the limits of predictive models is crucial to explore inherent uncertainties and limitations in predictions. The true model is, in practice, never observed or known. When developing p(y|x), the target is often to select the model with the best predictive performance, which is closely connected to the role of the true DGP [15, 134, 135, 133] . For this reason, the "true" model may include features in x true i that are not observed in the data, sometimes referred to as an M-open setting when the "true" model is not included in the set of candidate models [15, 135] .Second, we assume that p(y|x i ) is a probability distribution over Y, introducing some level of aleatoric uncertainty in the true underlying process [95, 58, 114] . This means that perfect prediction of y i may not be possible, even with knowledge of the true DGP. The distinction between aleatoric and epistemic uncertainty is important from a legal perspective. The reason is simple: the uncertainty coming from estimation is the (legal) responsibility of the modeller, while the aleatoric uncertainty can instead be considered a true underlying general risk.Third, the true DGP connects to judicial legal reasoning. Courts must engage theoretically with legal and normative conceptions of what is justifiable and what constitutes unlawful discrimination.Judges consider legitimacy, proportionality, and necessity when evaluating actions, and hypothetical alternatives, that led to less favourable treatment. Although, courts are not oracles. Discrimination case law may not pinpoint what the perfect decision should have been. However, courts will engage in a similar theoretical process of reasoning about the decision-making process to the true DGP to understand whether the actions were justified or unlawful. We explain legal reasoning within this framework throughout the paper and in a real-world case on unlawful discrimination in algorithmic decision-making (see Appendix A).Legally, distinguishing between a true difference and an estimated one is important. We approximate the true DGP with a model p(y|x) based on training data when training an SML model. The approximation introduces estimation errorEQUATIONAlgorithmic fairness literature often assumes the absence of estimation error [see e.g., 49] or assumes that the true causal structure is known [150, 68, 26, 21] . In practice, this is rarely the case. Hence, it is crucial, both practically and legally, to distinguish between the true underlying probabilities π i and the estimated probabilities πi . While the true underlying probability may sometimes be defensible (Section 2.2), introducing an estimation error that disadvantages individuals based on protected attributes invokes discrimination liability.As the model will try to approximate the true data-generating process, modellers' expectations are difficult to ascertain. The law is unlikely to set a deterministic standard that any adverse effects of estimation will make a modeller liable. The modeller should try to approximate the true model as much as possible [see 4, 141, 135, 133, for discussions on model misspecification]. However, where an estimation disparity reaches a threshold for discriminatory effects, the legal evaluation would require analysing the steps taken to test and mitigate estimation disparity (even though the intent is immaterial).The potential bias in training data presents a risk that the estimation model will introduce bias against individuals with protected attributes (Section 2.6). Historical discriminatory lending practices, for example, could be perpetuated through biased training data [18, 104] . Such biased estimations may introduce biased outcomes that are not reflective of true differences, potentially leading to discriminatory outcomes. Therefore, we introduce "Conditional Estimation Parity" to formalise the legal context of estimation.Conditional Estimation Parity is the difference in estimation error between groups with a protected attribute, given legitimate features, i.e.,EQUATIONReducing the error in Eq. 4 is expected to diminish the risk of conditional estimation disparity.However, assessing conditional estimation parity is complex due to inherent challenges in evaluating estimation error.It is crucial to examine both mathematical and legal causal theories of why certain differences are legitimate bases to make classification distinctions [71] . We examine the mathematical basis for identifying statistical disparities in the context of unlawful discrimination. In Section 2.5, 2.7, and 2.6we consider the causal relationships between legitimate differentiation and unlawful discrimination.To initiate a claim for discrimination, a claimant must establish a prima facie case [37, 40, s 136] .Sufficient evidence must be produced to show that unlawful discrimination may have occurred, including by showing discriminatory effects or harm against an individual or group caused by the decision-maker's action [37, 65] . Statistical evidence can be used to prove less favourable treatment or particular disadvantage, but by design, it shows correlations, and "a correlation is not the same as a causal link" [130, para. 28] . We explain the threshold for legal causation at the trial stage in Section 2.5. Although, at this stage, a mere correlation between the adverse effect on the person and the decision-maker's action will suffice [65] . The size of the disparity is relevant. Smaller and often resisted by courts to avoid excessive dependence on data [138] . The UK has specifically avoided thresholds like those used to measure statistically significant disparity in the US [105, 10] .Statistical disparities, as identified through algorithmic fairness metrics, may indicate a reason to consider whether discrimination has arisen. However, without taking context and potential true and legitimate differences into account, these disparities hold little legal weight (see Section 2.1). We can formalise this as the legal target being to minimise the conditional estimation disparityEQUATIONwhere || • || 2 is the euclidean norm. This target generalises the idea of minimising conditional statistical parity. If we assume true conditional statistical parity, i.e.EQUATIONthen the target in Eq. 6 will be reduced to minimise the conditional statistical parity (see Eq. 3).Although, this is only true as long as there are no true differences.Hence, if true statistical parity does not hold, it is explained by true differences between groups. If there is a true difference, such as age in financial services, forcing conditional statistical parity would harm the protected group, most likely resulting in unlawful discrimination. This result aligns with previous observations about the risks of forcing parity metrics [31, 144, 54] . Courts may need to be more flexible in the type of statistical data they consider to establish a prima facie case by considering non-comparative adverse effects in their assessment. Therefore, deferring to conditional estimation parity provides an avenue for a contextually informed assessment.To lawyers, causation is the relationship between an act, i.e., an action or decision, and its effect, which requires two questions: (1) factually, but for the act, would the consequences have occurred;(2) is the act a substantial cause of the consequence to apply responsibility. We are concerned with the first question. Direct discrimination "requires a causal link between the less favourable treatment and the protected characteristic"; indirect discrimination "requires a causal link between the PCP and the particular disadvantage suffered by the group and individual" [130, para. 25] . In an algorithmic context, this causal link requires asking whether i would have received the same action or decision a, but for their protected attribute x p or the PCP that indirectly relates to their protected attributex p [122, 123] . For instance, whether an individual would have suffered the disadvantage but for the protected attribute would be discriminatory regardless of the decision-maker's intention [1] . This is a notable distinction from certain aspects of US discrimination doctrine.From a decision-theoretic perspective, the protected attribute x p can affect the decision a either through the utility function u(a, y) or through the model p(y|x). Discrimination may occur if the utility function in Eq. 1 differs for different groups defined by the protected attribute. Such a difference would mean that an individual or whole group with a protected attribute is treated less favourably than those without a protected attribute given the same model p(y|x). Such a difference in the utility function would risk unlawful discrimination. Specifically, if u(a, y) is changed for different persons, either directly based on a protected attribute or indirectly has the effect of disproportionately disadvantaging a group with a protected characteristic without justification (see Section 2.6).Having different p(y|x), on the other hand, would mean that there is a legal causation between the decision a and x p . This might either be motivated by true differences (see Section 2.1) or a result of conditional estimation disparity. In the latter case, this might be a case of legal causation, i.e., that the model is poor, and hence, the modelling has resulted in disadvantaging a protected group. Therefore, we can view the causal structure of p(y|x) as central to avoiding unlawful discrimination. However, not considering causal structures could lead to conditional estimation disparity, and potentially result in unlawful discrimination.Legal causation focuses on the legal causal link between x p and the decision a. In addition, legal causation is less formal than common definitions of causal effects in ML. Courts, at least outside of the US, are effects-orientated, and a wide range of forms of a "legal causal link" could be identified [109, 65] . Much of the causal-based fairness literature formulates "causation" on the true causal model structure in p(y|x), i.e., the study of the causal effect of x, due to outside interventions on y [101, 10, 149, 21] . However, this formulation is not the same as that of legal causation.In this discussion, the parallels to other discrimination studies become evident in how it would affect automated decision-making, particularly taste-based and statistical discrimination. Taste-based discrimination [11] , could arise if only the utility function u(a, y) unjustifiably disfavours a group based on protected attributes x p . Statistical discrimination, on the other hand, arises when decisionmakers use group-level statistics as proxies for individual characteristics due to imperfect information [7, 102] . Statistical discrimination parallels the disadvantaging of a group due to having different p(x|y). While these types of discrimination are generally prohibited, statistical discrimination can be legally permissible in some circumstances (see Section 2.1).Decision-makers must consider the legitimacy of using an SML model by explicitly defining its purpose and the outcome variable y. In algorithm design, social implications should be considered [87, 57, 63] . Additionally, this aligns the model's use with legal expectations.If the court believes sufficient evidence of discrimination exists, the burden shifts to the respondent to disprove allegations of unlawful discrimination [38] . [35] .The legitimacy of the aim depends on the decision-makers' raison d'être [65] . In Homer, the Court established a legitimate aim must "correspond to a real need and the means used must be appropriate with a view to achieving the objective and be necessary to that end" [128, 35, 39] . In lending, it is a legitimate aim to protect the repayment of their loans or at least secure their loans. In fact, "the mortgage market could not survive without that aim being realised" [126, para. 79].For a legitimate y to be an exception to indirect discrimination, the PCP must be a proportionate means of achieving the legitimate y [40, s 19(2)(d)]. To be proportionate, it must be an appropriate means of achieving the legitimate aim and (reasonably) necessary to do so [128] . Such analysis will turn on the facts of each case. However, it will require evaluating whether the design choices were "appropriate with a view to achieving the objective and be necessary" by weighing the need against the seriousness of detriment to the disadvantaged group [39, para. 151 ]. This will require considering whether non-discriminatory alternatives were available [128] . Measures to improve accuracy, maximise benefits over costs, minimise estimation error, or condition for protected attributes may all be relevant considerations for whether the modeller's choices were proportionate means of achieving a legitimate y.If the estimated outcome ỹ approximates the true outcome y, this can lead to biased predictions. LetEQUATIONthen, if the expectation of γ condition on x l shows a disparity, i.e.,EQUATIONit suggests the use of ỹ is inappropriate and might be discriminatory.To illustrate with an example, if a bank's training data is outdated or sourced from a different country, it may not accurately represent the current population relevant to the model. This discrepancy can lead to biased estimations, particularly if the data reflects historical prejudices. For instance, the model might unjustly associate certain demographics with higher default risk, not because of true differences but biased historical data [as warned in 33].One of the more crucial aspects of SML for automated decision-making is the choice of features x.The aim and y will help inform the choice of features to include in the model. We can separate three types of features from a legal perspective: features with protected attributes x p , legitimate features x l , and non-legitimate or illegitimate features x n . The distinction between x l and x n depends on whether the feature can be considered legitimately related to y (see Section 2.5). Causal fairness literature has engaged with questions of discriminatory variables through the lens of proxy discrimination [69, 115] . Proxy discrimination has a specific legal meaning under UK law that relates to direct discrimination, unlike much of the US literature on proxy discrimination that relates to indirect forms of discrimination. Here, we explain the UK legal implications of such causal relationships between variables and we provide a real-world example in Appendix A.Direct discrimination in automated decisions may arise when members of, or an entire protected group, is affected. Where a model p(y|x) uses a protected attribute x p , and there is a difference in predictions between the protected groups defined by x p , this risk arises. Models have directly used protected characteristics, giving rise to direct discrimination [94, see discussion in Appendix].Direct discrimination may arise when a feature is an exact proxy for a protected attribute. In Lee v Ashers, Lady Hale explained that the risk of direct discrimination also arises if a decision is based on a feature that "is not the protected characteristic itself but some proxy for it" [129] . Therefore, direct discrimination can arise even where x p has been removed because there is a feature which is an exact proxy that is "indissociable" or has an "exact correspondence" to x p [130, 129] . Formally, we can define an exact proxy as a feature xp with a perfect or almost perfect correlation with x p [115] .UK courts have accepted that an exact proxy would be pregnancy because "pregnancy is unique to the female sex" [121, 125] . If a model uses pregnancy or maternity leave as a feature, collected from CV information, for example, it would have the effect of using an exact proxy xp that could hypothetically be the basis for a direct discrimination claim.Given the relevance of x p to direct discrimination, modellers have been encouraged to remove protected attributes when designing ML models [105, 62, 48] . These claims are usually based on the US Equal Protection Clause, which subjects classifications based on certain protected characteristics, such as race, to strict scrutiny [146] . The focus on excluding certain data inputs is one form of discrimination prevention [146, 46] , but not under UK law. Further, simply removing protected characteristics reduces accuracy and utility [150, 66] , and does not remove the risk of discrimination [34, 79, 72] .This reasoning connects to the true DGP. If a protected attribute like gender is inherent in the DGP, removing it does not eliminate discrimination but instead may introduce it. Taking a gender-neutral approach to recidivism predictions may have the adverse effect of discrimination against women who would otherwise have received lower risk scores [31] . In Loomis, the Court accepted that in recidivism algorithms, "if the inclusion of gender promotes the accuracy, it serves the interests of institutions and defendants, rather than a discriminatory purpose" [112, 766] . Hence, if the inclusion of x p improves the accuracy and benefits the protected group, it may avoid the risk of discriminatory purposes. There is an absence of any legal guidance in the UK on the relationship between true probabilities and protected attributes in automated decision-making. Pending further legal guidance, it is important to carefully consider whether including x p is relevant to promote accuracy and conditional estimation parity. Removing protected attributes often ignores the true probabilities for the legitimate differences between protected groups, affecting the lawfulness of its outcomes.Therefore, removing x p will not avoid liability for unlawful direct discrimination by itself. Even if a model ignores x p , in practice, it may rely on other data points acting as proxies with "exact correspondence" to a protected characteristic xp . Importantly, this diverges from US law and highlights that intention is immaterial to UK direct discrimination [Cf. e.g., 5, 115] . UK law focuses on the discriminatory effects rather than a formalistic view of whether x p is considered or not.Indirect discrimination may arise if a PCP appears to apply equally to everyone but disadvantages members of a protected group. Both forms of discrimination can arise using an exact proxy or a weak proxy in a PCP. Therefore, identifying legitimate features is challenging when many features correlate to protected groups. We define non-legitimate features x n as features not legitimate in the context of the true DGP (Section 2.2). In practice, this means a non-legitimate feature is one that, if included, would not contribute to the predictive performance of the optimal model, i.e., the one with the lowest estimation error (Section 2.3). Therefore, x n would not improve the predictive performance if a modeller had the true features.For example, hair length strongly correlates to gender in many cultural contexts but is unlikely to contribute to the consumers' true default risk. Boyarskaya et al. explain the absence of a "causal story" between hair length and loan repayment because hair length would not be part of a true model for the risk of default [19] . Therefore, hair length is an example of x n in a lending context.For comparison, the legitimacy of zip codes illustrates the nuanced nature of legitimate features.While a zip code may correlate with race in some contexts, it might be a legitimate variable in other situations. For example, in an application for home insurance covering flood risk, zip codes are invaluable proxies for granular information such as geographical features, land topography and historical flooding. Therefore, in the best model for property flood insurance decisions, zip code will improve the predictive performance as a legitimate proxy for data within the true DGP. However, in a university application, there should be no predictive or causal relationship to merit for acceptance. In such cases, zip code likely acts as a proxy for race or the unprotected characteristic of socio-economic status and would be x n . So, in some circumstances, the zip code would be legitimate x l , but in others, it may not be x n . It will also be relevant to consider whether a less discriminatory feature is available, i.e., one with less correlation to a protected attribute that is equally predictive.As explored in Appendix A, in lending, information about income and debts are likely to be legitimate features x l . Credit scores can be a proxy for a person's financial position, as well as protected attributes [18, 60] . However, the complexity of calculating credit scores means it is more valuable for inferring income, debt repayments, and history of credit. Credit scores, or related features, would have a material impact on the true model for default, and then would be a legitimate feature x l .Given that nearly, all features may contain some information on protected attributes, even legitimate factors [30] , this approach explains the need to assess the strength of this dependence and whether the feature contributes significantly to the model's prediction and can be argued to be part of a true DGP.The distinction between x l and x n also gives rise to problems in automatic feature construction, such as using deep neural networks. If features are constructed automatically using a combination of x l and x n , indirect and direct discrimination are risks. As an example, an applicant's resume contains legitimate features x l for recruitment prediction. However, the detailed granularity of many resumes also gives rise to the problem of non-legitimate information, such as maternity leave or women-only sports or other information that may contain information on other protected attributes. Hence, there needs to be an active choice of only including legitimate features x l from available data in the model.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Section 1.4 sets out the limitations of this paper. 
conclusions: 
 Minimising unlawful discrimination in automated decision-making requires a nuanced and contextual approach. While it is beyond our scope to offer specific legal advice, our findings underscore several key considerations to identify and mitigate potential discrimination effectively: (a) Identify whether conditional statistical parity is unattainable or undesirable based on true group differences. This requires stringent analysis into whether differences stem from prior injustice or legitimate variation.(b) Incorporate further legitimate features x l that could minimise statistical disparities by "explaining away" the performance gained by the protected attribute with legitimate features.(c) Avoid using the model due to unmitigated discrimination risks.While these guidelines cannot guarantee lawful automated decisions, they provide meaningful recommendations and abstractions to help identify and mitigate unlawful discrimination risks.In conclusion, this work bridges a critical gap between the technical aspects of automated decisions and the complexities of anti-discrimination law. By translating these nuanced legal concepts into decision theory, we underscore the importance of accurately modelling true data-generating processes and the innovative concept of estimation parity. This interdisciplinary approach enhances the understanding of automated decision-making and sets a foundation for future research that aligns technological advancements with legal and ethical standards.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Our research contributes to bridging the gap between legal standards and algorithmic fairness, aiming to enhance the integrity and fairness of automated decisionmaking systems. This has significant implications for improving equity in critical areas where algorithmic decisions are increasingly prevalent. We also the risks of unfair treatment based on model or data biases or misinterpretation of the legal doctrines we study. We explore the potential for unintended consequences even when the technology functions as intended, such as the reinforcement of existing societal biases under the guise of legal compliance. To mitigate these risks, we propose specific safeguards to prevent unlawful discrimination in systems.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Overview of Finnish Anti-Discrimination Law Finnish anti-discrimination law bears many similarities to UK and EU laws. We briefly set out the relevant provisions that show the similarities to the Equality Act set out in Section 1. It is worth noting that this definition is broader than in the UK Equality Act. Some protected characteristics are outlined more explicitly; for example, a person discriminated against on the basis of language may be able to bring a claim based on racial discrimination [120] . Unlike many Nordic countries, the Equality Act does not explicitly protect political activity, trade union activity, and does not include "or other personal characteristics" [53] .Direct discrimination is defined in Section 10: 2Discrimination is direct if a person, on the grounds of personal characteristics, is treated less favourably than another person was treated, is treated or would be treated in a comparable situation.Indirect discrimination is defined in Section 13: 3 Discrimination is indirect if an apparently neutral rule, criterion or practice puts a person at a disadvantage compared with others as on the grounds of personal characteristics, unless the rule, criterion or practice has a legitimate aim and the means for achieving the aim are appropriate and necessary.Section 11(1) defines justifications for different treatment as: 4Different treatment does not constitute discrimination if the treatment is based on legislation and it otherwise has an acceptable objective and the measures to attain the objective are proportionate.The first case regarding automated decision-making and discrimination was in Finland. The person, referred to as A, was denied credit for online purchases based on a credit rating system employed by a bank. Person A reported the case to the Non-Discrimination Ombudsman (Yhdenvertaisuusvaltuutettu), who brought the case before the National Non-Discrimination and Equality Tribunal (Yhdenvertaisuus-ja tasa-arvolautakunta). The Tribunal found that the bank's statistical scoring model resulted in direct discrimination based on multiple protected characteristics and was not 1 Official translation from Finnish, although only legally binding in Swedish (not included) and Finnish: "Syrjinnän kielto Ketään ei saa syrjiä iän, alkuperän, kansalaisuuden, kielen, uskonnon, vakaumuksen, mielipiteen, poliittisen toiminnan, ammattiyhdistystoiminnan, perhesuhteiden, terveydentilan, vammaisuuden, seksuaalisen suuntautumisen tai muun henkilöön liittyvän syyn perusteella. Syrjintä on kielletty riippumatta siitä, perustuuko se henkilöä itseään vai jotakuta toista koskevaan tosiseikkaan tai oletukseen" 2 "Syrjintä on välitöntä, jos jotakuta kohdellaan henkilöön liittyvän syyn perusteella epäsuotuisammin kuin jotakuta muuta on kohdeltu, kohdellaan tai kohdeltaisiin vertailukelpoisessa tilanteessa." 3 "Syrjintä on välillistä, jos näennäisesti yhdenvertainen sääntö, peruste tai käytäntö saattaa jonkun muita epäedullisempaan asemaan henkilöön liittyvän syyn perusteella, paitsi jos säännöllä, perusteella tai käytännöllä on hyväksyttävä tavoite ja tavoitteen saavuttamiseksi käytetyt keinot ovat asianmukaisia ja tarpeellisia." justified by an acceptable objective achieved by proportionate measures. Consequently, the Tribunal prohibited the bank from continuing this practice and imposed a conditional fine to enforce compliance.The decision-making system in question is for online store financing, which is a purchase-bound, fast and automated credit type very different from regular consumer credit. The credit applied for by the consumer in each situation is also always bound to the purchase and its value, which means that it is more difficult, or even impossible, to undertake detailed requests for information and background checks. The individual investigation of the creditworthiness of customers using personal information and documents, such as salary and tax certificates, may not be suitable for this type of credit.The company made credit decisions based on data from the internal records of the credit company, information from the credit file, and the score from the company's internal scoring system. The bank's scoring system assessed creditworthiness. The scoring system used population statistics and personal attributes to calculate the percentage of people in certain groups with bad credit history and awarded points proportionate to how common bad credit records were in the group in question.The variables used included race, first language, age, and place of residence. The company did not require or investigate the applicant's income or financial situation.The bank's scoring model was based on statistical correlations calculated population and groups, including gender, language, age and place of residence, meaning the model is more or less p(y|x p ). This model cannot be said to have attempted to model the true underlying data-generating process and instead relied on data that was available regarding protected attributes. It is reasonable to expect that the bank was aware of other legitimate factors that could explain the credit score. Therefore, the model introduces epistemic uncertainty stemming from the lack of information that could have been used to make better predictions, i.e. reasonable legitimate features x l .By solely using the data available, rather than identifying what data would be best to reduce estimation error, the modellers built an automated decision-making system that unlawfully discriminated. We now evaluate how the Tribunal came to those conclusions about the legitimacy of y and x for such a model.The bank argued that the "different treatment does not constitute discrimination if the treatment is based on legislation and has an otherwise acceptable objective and the measures to attain the objective are proportionate." The Tribunal agreed that "the provision of credit to customers is a business, the purpose of which is to gain profit" and that "the investigation of creditworthiness is as such based on law and that it has the acceptable and justified objective as defined in section 11 of the Non-Discrimination Act". Therefore, creditworthiness assessment is a legitimate y.However, the Tribunal clarified that "the individual assessment required by the legislation means expressly the assessment of an individual's credit behaviour, credit history, income level and assets, and not the extension of the impact of models formed on the basis of probability assessments created with statistical methods using the behaviour and characteristics of others, to the individual applying for the credit in the credit decision in such a way that assessment is solely based on such models." Therefore, to be appropriate and necessary to achieve that aim, the model must consider legitimate features x l .Protected, Legitimate, and Non-Legitimate Variables x Four protected attributes were used as variables in this model x p : age, language, other personal characteristic (place of residence), and gender.The Tribunal acknowledged that age may be a legitimate variable if it had been used in the assessment of creditworthiness mainly when applied to young persons. However, it was not justified in this assessment, given the age of the credit applicant.The Tribunal agreed with the position under European law that gender is prohibited from being used as an actuarial factor in financial services [42] .Therefore, these features did not contribute to the accuracy of the model's prediction in a way that could be argued as part of the true DGP. Therefore, in this case, these x p variables are also non-legitimate variables x n .As explained by the Tribunal, to achieve the legitimate y of undertaking an individual assessment of creditworthiness and ability to repay, the model should have considered, for example, income, expenditure, debt, assets, security and guarantee liabilities, employment and type of employment contract (i.e., permanent or temporary). These features would have been legitimate variables x l by improving the predictive performance of the model to achieve more accurate decisions.Using the legitimate variables identified above, we can now consider conditional estimation parity, the difference in estimation error between groups with a protected attribute, given legitimate features.Reducing the error in Eq. 4 is expected to diminish the risk of conditional estimation disparity.However, assessing conditional estimation parity is complex due to inherent challenges in evaluating estimation error.Judges engage this type of reasoning through statistical or theoretical means. In this case, the Ombudsman brought evidence of the effects of the protected characteristics x p on the true prediction.Person A was negatively affected by his age. He was in the age group of 31-40 years old, but if he had been at least 51 years old, he would have received a higher score sufficient for the credit application.If person A spoke Swedish as his first language, he would have received a sufficient score for granting the loan. Finnish-speaking residents received a lower score compared to Swedish-speaking residents.Further, ethnic minorities with an official first language other than Finnish or Swedish were put in an unfavourable position.A would have earned more points based on his residential area if he had lived in a population centre. The bank's statistical method, which is based on a grid of residential areas, gave A the lowest score because he lives in a sparsely populated area that has not yielded any statistically significant information.Gender impacted the model, where women received a higher score than men. The Tribunal agreed that if the person A had been a woman, he would have been granted the credit.This case study demonstrates the intersection between judicial reasoning and our formalisation. To avoid liability for unlawful multiple direct discrimination in this algorithmic decision-making process, the company should have:1. Assessed data legitimacy. While the Tribunal agreed with the target variable (y) as a legitimate aim, they did not believe the features (x) were legitimate for the specific context (Section 2.7).Built an accurate model. The bank did not strive to approximate the true DGP p(y|x), and did not use legitimate features x l . Reasonable, necessary, and proportionate steps should have been taken to minimise estimation error and aim for estimation parity (Section 2.3).3. Evaluate differences. The bank should have considered whether there were true and legitimate differences based on protected characteristics and whether they could have been "explained away" by legitimate features (x l ) to minimise statistical disparities.These recommendations should be used to help identify and mitigate unlawful discrimination within the specific context of each jurisdiction. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations."Erilainen kohtelu ei ole syrjintää, jos kohtelu perustuu lakiin ja sillä muutoin on hyväksyttävä tavoite ja keinot tavoitteen saavuttamiseksi ovat oikeasuhtaisia." 


Decision: This paper explores the legal challenges related to algorithmic fairness from the perspective of UK anti-discrimination law. Most reviewers are positive about the work. However, concerns have been raised regarding the novelty and technical contributions of this study. Additional comments were provided during the discussion period, highlighting several key points:

* Much of the content appears to be a review of existing literature, a point already noted in two reviews. Some of the comments suggest that the considerations and recommendations provided in the paper are not novel.

* It is generally hard to identify practical benefits of this work. It is important for the paper to clearly explain how systems developed under American anti-discrimination laws could contravene UK legislation, specifically pointing out the differences between US and UK anti-discrimination laws that impact ML practitioners.

* The paper would benefit from positioning its contributions relative to existing definitions and concepts of fairness, clarifying why UK anti-discrimination law's requirements are not addressed by these concepts.

* Although the paper could be of interest to researchers working on algorithmic fairness, its accessibility to the broader NeurIPS audience is questionable. The case study in the appendix could improve clarity, though it is unclear if this would necessitate a major rewrite.

Many of these concerns appear to be reasonable, and it is generally hard to evaluate whether the authors would adequately address these concerns without seeing a revision. Hence, I strongly encourage the authors to try to revise their paper accordingly.

Reviews: review: paper_summary: paper_summary: This paper provides a UK-and-European-law-based view of anti-discrimination law as it relates to fair machine learning and automated decision systems. It does a good job laying out the doctrine, arguing correctly that work in this area to-date is very centered on US legal concepts such as disparate treatment vs. disparate impact. Although I am willing to believe that there are subtle differences that drive important aspects of fair ML analysis, as the paper claims, I think the specifics of these differences could be made much clearer and need to be for the paper to have the impact it should.

Of particular note, the paper is very well situated in the surrounding literature. Although this contextualization should make the contributions more clearly offset from prior work, as presented I find the opposite: it is difficult to tell what is new as a contribution here. For example, while the contributions are clearly identified in 1.4, I think it would aid the paper if they appeared higher in the intro and were clearer about what is new and why it matters. The example in Appendix A could be used as a running example to show where new concepts are needed and what about existing work does not capture this different legal regime. In particular, after claiming that disparate treatment/disparate impact are distinct to direct & indirect discrimination, the definitions given from 105-114 seem to align tightly to the former. And while I'm not a lawyer, I don't believe that disparate impact claims require a showing of intent under US law either, so I found that distinction somewhat confusing.

On the technical level, the discussion of the true data generating process should really be contextualized in the literature on measurement and construct validity, specifically with respect to work by Jacobs & Wallach, which in particular encompasses the material in 2.3 on estimation parity (at least in part). Also, the causal analysis components of the discussion of data generation could cite more of the work of Kohler-Hausmann and also Hu (one paper from these authors is cited, but others are also relevant and speak more directly to causality and counterfactual fairness claims).

As a final observation, although the ML community talks in terms of "fair" outcomes, it is often conceptually clearer (and more in line with legal analysis) to use the same techniques as tools for identifying "unfair" activities or outcomes. Phrasing some of the claims this way may condense some arguments and tighten the presentation overall. Related to this, the discussion of these tools as part of an overall practical strategy for risk management is important and should receive more attention. For example, it would be good to discuss how the measures proposed would be used in real legal analysis of an example, such as in litigation or a regulatory proceeding.

I was also a bit confused about the analysis of constructed proxies for protected variables in 2.7. I understand that it's necessary to look beyond a formalistic view of whether a specific attribute is considered, but what happens if the proxy for a protected attribute is (say) the sum of two legitimate attributes? Why is it good enough to use only legitimate features? Also, at 393-394 it might be valuable to look at the recent paper on "Less Discriminatory Algorithms" and compare the approaches and outlooks.

Incredibly minor: 
* There is a missing period at 81.
* At 284-288, there is a latent call to questions of ecological validity which could be made more explicit
strength_weakness: strength_weakness: * Novelty is at times hard to identify. I think it's there, but the claims on what it covers should be clearer. In particular, the discussion of the decision-theoretic framing seems a bit under-attended even though it's potentially very useful.
* Some important concepts are missed, notably theories of measurement and construct validity/reliability are at least partially re-invented when they should just be treated as background.
questions: questions: * Can the example in Appendix A be a running example?
* What is new over and above existing literature on ML fairness? How can this novelty be more clearly offset in the presentation? Here I think specifically about the conclusions at 415-430, which seem rather anodyne in light of existing literature.
limitations: limitations: I believe the limitations are expressed well.
paper_summary: paper_summary: This paper addresses the issues around existing fairness metrics and bias detection/mitigation methods not corresponding with legal notions of fairness, specifically under UK anti-discrimination law. The authors propose a theoretical framework for a data-generating process that aims to formalise the legitimacy of decisions and features in the data. Further, they propose a new metric "conditional estimation parity" which compares estimation errors for different protected groups.
strength_weakness: strength_weakness: 1. A lot of the paper is background or a collation of existing literature. The main contribution is the new conditional estimation metric metric but this metric relies on the true DGP and evaluating the estimation error which, as stated, can be complex in practice. This could make it difficult to use the metric in practice.
2. I understand it would be hard to use the metric for evaluating discrimination in existing datasets for the reasons specified above and also due to the inherent context-dependency of the metric (which is a benefit) but it could be useful to include some experimentation or results in a hypothetical scenario to show how it might be used in practice. As there are no results as such to comment on, it is difficult to assess it's significance.
3. The conclusions drawn such as "Assess data legitimacy" or "Build an accurate model", although justified with evidence in the paper, are not novel and are pretty standard, common-sense recommendations. 
4. Overall, the main novel contribution is the new metric but this is a small part of the paper. The rest of the paper is a nice collation and narrative of existing literature but I am not sure it significantly advances the field.

Other comments:
1. I can't see where SML terminology is introduced - I assume this means supervised machine learning?
2. In Section 1.4, DGPs are mentioned for the first time. It would be useful to have some more background to them before this - what exactly is a DGP? I do not believe it is ever explained.
questions: questions: 1. See weaknesses above. How would you go about using the metric in a real scenario?
2. Do you have any thoughts about how your metric relates to other notions of fairness such as individual fairness metrics?
3. Could you explain "taste-based discrimination" further?
limitations: limitations: The authors are honest about the strengths and weaknesses of their work (although some are hidden away and not pointed towards in the checklist). It would be useful to improve the discussion of limitations in Section 1.4 as it only mentions the limitation of applicability only in the UK.
paper_summary: paper_summary: - There is a gap between the definitions of fairness studied in the computer science literature, and the definitions of fairness operationalized by courts adjudicating discrimination claims. This limits the usefulness of the CS definitions.
- Amongst work attempting to reconcile legal and computational definitions of fairness, little has focused on anti-discrimination law outside the US.
- This paper makes four contributions in this context:
    - (1) It formalizes elements of anti-discrimination law into a decision-theoretic formalism
    - (2) If analyzes the legal role of the data-generation process
    - (3) It proposes conditional estimation parity as a legally-informed target
    - (4) It provides recommendations on creating SML models that minimize the risk of unlawful discrimination in automated decision-making
strength_weakness: strength_weakness: - Much of the paper reads like a review of anti-discrimination law. This makes it difficult to parse out (1) what the technical contributions are, (2) why they’re novel, and (3) why they matter. 
- It’s extremely unclear what the technical payoff of the paper’s modeling choices are. The fairness field is overwhelmed with different definitions/frameworks. Why is the one proposed by the author’s meaningful over others? 
- It seems like an essential point to the paper’s argument is that prior work hasn’t studied UK anti-discrimination law. But if the paper wants to successfully extend that into an argument about modeling choices, I think it needs to explain why the existing definitions of fairness do not work for UK law.
- The recommendations provided are extremely general. Are these new or different from the many recommendations that already exist in the fairness/responsible AI literature?
questions: questions: The comments in the weaknesses section list the relevant questions!
limitations: limitations: NA
paper_summary: paper_summary: The paper maps existing literature and law on algorithmic fairness onto a decision-theoretic framework. It describes various desiderata (e.g. statistical parity) and legal restrictions (e.g., legitimate aims) in terms of expectations, distributions, estimation error, etc.
strength_weakness: strength_weakness: n/a
questions: questions: I regret to say that my expertise does not extend to the paper's two principal areas (anti-discrimination law and decision theory). I cannot form a sufficiently educated opinion about the correctness or the novelty of the paper's results. This is my fault, not the authors'.
limitations: limitations: n/a
paper_summary: paper_summary: The paper presents a formalization of fairness metrics intended to ease analysis of discrimination by automated decision making systems in the UK. While there is a relatively applied angle, the bulk of the contribution is intended to be a generic and re-targetable mathematical formalism.
strength_weakness: strength_weakness: I do not have any major scientific critiques, though there were areas where the clarity of the paper could improve.

Lines 240-274 were written in harder to parse prose than the bulk of the rest of the paper. I had to reread that area multiple times.

The case study in Appendix A was actually very useful for understanding the authors' formalism and it is a shape that some of that context was not woven into the paper as concrete examples of how to understand the math.

The discussion on proxy discrimination never seemed to finish? I wasn't able to understand its meaning under UK law.


Missing a ref to Homer on L299.

All these are very minor issues. I'm substantially in favour of accepting this paper.
questions: questions: Where does proxy discrimination sit under UK law?

On L428 the authors make the suggestion to incorporate legitimate features that substantively create the same outcome as the protected features–this seems like a recipe for disaster in a world where we care about the ultimate effects? I'm wondering how we can square the circle there!
limitations: limitations: Ultimately, adherence to a formalism is *not* what courts generally take into account. While statistical analyses may be used to advance a given line of argument, the standards used are open-textured–and this is an inherent limitation of this line of work.
It also would have been good to see where this formalism sits under EU law (or representative EU-member law) or perhaps a discussion of how civil law jurisdictions handle these sorts of issues.

Comments: title: Response to Rebuttal
Rebuttal by Authors (2/2)
comment: I don't have any other comments or questions, and very much appreciate the responses to the review. I remain very happy with this paper and believe the revision will address the smaller issues raised in all the reviews.

I leave as a comment to the area chairs that the issues I raise and the issues raised by reviewer 4EE7 are very similar but our scores are very different. To this, I offer that while the recommendations are very general, using a case study through the paper to answer the critical question of why the difference in legal regimes matters in practice shows why my outlook is positive (also, I believe that contributions need not be a definition or a model, but could be a review of relevant law that unpacks the requirements on how existing work can be applied in a different and understudied context).

On the point about making more use of the decision-theoretic framework, adding material about choices between available models and incorporating a running example provides the opportunity to show how the decision-maker's choices can be thought of as decision-theoretic optimization under the legal constraints described here. I believe that change is feasible straightforwardly, since the necessary material is all in the paper (just some is in the appendix).

Hu gives commentary on earlier work of Kohler-Hausmann here: https://www.phenomenalworld.org/analysis/disparate-causes-i/. You are likely also aware of another paper focusing specifically on the construction of sensitive groups and the structuring of the data generating process in which they collaborated: https://dl.acm.org/doi/abs/10.1145/3351095.3375674. The general point is that getting at the question of which group differences are causally meaningful vs. which are protected by anti-discrimination law requires either importing an exogenous ontology of protected groups (which the law provides but may not define in as much detail as is needed) or ignoring the contextual construction of subgroup structure in a population for a use case. There are many deep philosophical questions here, and this paper can't reckon with all of them, but it's important to point out where the lines are still blurry to make the scope as clear as possible
Thank you for your detailed rebuttal and clarifications and apologies for the late response. 

I appreciate that this is theoretic work and not applied, however I would expect when talking about the law and fairness there should be some way for the reader to understand how it might be useful in practice. I think adding the existing case study throughout the text will help this. Given this addition and the clarifications on the contributions, I am happy to increase my score as I think this work could be valuable for the Neurips and fairness community. 

As for adding to the limitations section (Section 1.4) - I would make it clear about the difficulties of finding the true model, or in other words stress the inherent challenges in evaluating the estimation error.
Thank you–I really appreciate the response. I will stick with my score however.
Thanks for the detailed rebuttal.

I still maintain my positive score.

Regarding A2, if you could really work on clarifying that in the paper, I think it would go a long way to assuaging my concerns.

Overall though, nice work!
> **W2** “Some important concepts are missed, notably theories of measurement and construct validity/reliability are at least partially re-invented when they should just be treated as background.”

**R2** We will expand our discussion to include references to relevant work on measurement and construct validity, particularly based on Jacobs & Wallach, incorporating in Section 2.3. We will also incorporate more of Kohler-Hausmann and Hu’s research on causality. Could you please specify which papers by Kohler-Hausmann and Hu you believe are most relevant? We're familiar with the paper by Kohler-Hausmann and Dembroff on causality in the US context, which we could reference in a comparative sense. 

## Questions

> **Q1** “Can the example in Appendix A be a running example? 

**A1** Yes, we fully agree with this suggestion and will include specific references to the case study throughout the text. We believe this will also aid the understanding of the practical use of these formalisations and how the measures would be used in real legal analysis. 

> **Q2** “What is new over and above existing literature on ML fairness? How can this novelty be more clearly offset in the presentation?”

**A2** Please see comment in **R1** above.

## Limitations
Thank you for confirming that “the limitations are expressed well.”

## Summary 
We appreciate your thorough feedback and are committed to improving the paper based on your valuable suggestions. We hope you will continue to support our paper for acceptance. Do you have any further questions or comments?