Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a general framework for evaluating the fairness of a decision-making system, based on concepts from theories of distributive justice. The authors first point out the limitations and shortcomings of the current fairness metrics used in the literature, namely “the levelling down objection”, “non-consideration of consequences”, and a limited set of fairness definitions in the literature of fairness in AI. The framework they propose is based on utility values for individuals based on different combinations of decision D and the variable Y. They show that most popular fairness metrics present in the literature can be interpreted as special cases of their proposed frameworks.  
strengths_and_weaknesses: Originality: This paper brings an interesting and fresh perspective to the discussion of fairness concept in AI field. I lack the understanding of ethics studies from the philosophical field that is discussed in this paper.  Therefore, I cannot comment on the originality of the ideas from that perspective. However, I believe that this paper would bring in an original enough perspective to the AI community working on fairness and ethical AI topics and would contribute to and enrich the discussions. 

Quality: The submission is technically sound. The claims appear to be theoretically supported but lacks the empirical evidence.  

Clarity: The paper is well written and organized. 

Significance: This paper addresses an important issue, since most fairness metrics currently under consideration in the fairness literature suffer from levelling down objections. 

questions: Have the authors consider provide experimental comparison of the proposed metrics and the existing metrics? 
limitations: The paper lacks empirical evidence for their claims. While discussing downsides of current metrics, providing specific cases for which their claims might hold true would strengthen their claims. 
Similarly, in addition to the theoretical and qualitative approach of their framework, providing empirical evidence for why and when their framework can be helpful in quantifying its effectiveness.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a unifying framework for group fairness metrics which exposes different issues and justice patterns as choices in a mathematical equation. The key idea is to use explicitly the notion of utility (and expectation of it) in the definitions of fairness. By doing this they show that not only it is possible to represent all the most important fairness metrics in their formulation, but it is possible to construct new, more complex forms of fairness which address common criticisms of fairness criteria.
strengths_and_weaknesses: Strengths:

The article aims to be formal, mathematical, and precise. Within this goal, it works well, and reads as easily as it would be expected in this context. The authors make a good job of pointing to other articles (included in the additional material) which have a deeper philosophical perspective and are, to some extent, easier for a less technical audience to read.

Weaknesses:

It would be really good to have the authors' view of what the drawbacks of their definition are. Often, unifying frameworks leave out some methods used in practice, and it would be great to have the authors' own view about what they have left behind. If nothing is left outside their framework, the authors' should clearly state that claim.
questions: Are there fairness formalisms and practical methods which are not covered by the unifying framework? Which are those? Why?  
limitations: The authors should be more clear whether there are fairness formalisms which are not representable in their framework and why.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper presents a general framework for understanding group fairness metrics that is connected to relevant literature in philosophy underpinning the relevant values and moral perspectives. Specifically, the paper proposes to interpret group fairness as the “just distribution of utility among relevant groups”, as opposed to simply “equality (of decisions) between socio-demographic groups”. This highlights the several components of the proposed approach:
- utility (as opposed to decisions)
- some notion of justice (as opposed to necessarily equality)
- comparing relevant groups (as opposed to simply demographic groups)

In slightly more detail: The starting point of the proposed approach is to focus on the utility of the individuals (here referred to as decision subjects) and to introduce the concept of a “claims differentiator” J, which identifies individuals with equal claims utility for the sake of the evaluation. Technically,  this is a binary variable where now fairness means comparing the expected utilities of individuals with the same J=j but different group membership A=a. There is then the question of how to compare these quantities - here comes the component of “pattern of justice”. The paper considers several such patterns studied in the philosophical literature and studies their implications in terms of the implied fairness metrics. For example, “egalitarianism” would seek to equalize the above-mentioned quantities, but other principles (Maximin, Prioritarianism) could require different things. 

strengths_and_weaknesses: Strengths: The paper is generally clearly written, and bridging the gap between philosophical ideas and fair ML is an important objective.

Weaknesses: The main weakness of the paper in my opinion is that the main contributions of the paper are w.r.t the authors’ portrayal of the notion of group fairness (“equality (of decisions) between socio-demographic groups”), which I’m not sure is a actually fair w.r.t the *current* state of the literature. Specifically:

1. Confusion between group fairness as a metric and group fairness as an objective: Even with the narrow interpretation of equality of outcomes (or some other statistic that depends on the outcomes), an  important distinction in the literature is between group fairness as a metric and group fairness as an objective. I (and I believe many of my colleagues) view group notions of fairness as an important red flag, that if not satisfied, suggests one should look into the entire algorithmic pipeline for issues - be it the problem specification, the data collection, etc. In particular this means that the “solution” may be something like collecting better data, which does not require making any group’s welfare worse off. 
2. A narrow interpretation of group fairness: There are many more notions of fairness studied in the literature that (i) utility based, (ii) that take preferences into account, or (iii) that speak to notions of fairness that do not necessarily involve parity (such as making sure no groups is “worse off” than what it would be had there been no other groups).  None of them were mentioned in the paper, and I found the perspective that the proposed unifying view is novel to be hindered by this. 


Another weakness is that while the approach sets out to expose the normative and moral assumptions under-pinning group fairness, there is a bit of a “pick and choose” nature to this, as some assumptions which I think are not inherent (and encode some moral assumptions) are taken for granted and not discussed. For example, it is stated that since we are interested in systematic differences between groups, fairness only means we are “interested in the expectation value E(U_DS) of the individual utility”. Isn’t this also an assumption? Two groups may very well be systematically different even if their expectations are similar (but their variance, for example, is not). Does this not come up in the philosophical literature? 	
questions: Minor points: 
- The notation U_{DS,i} seems overly complex, why not simply use u_i?
- I couldn’t follow some of the choices for the degree to which fairness is fulfilled. For example in Maximin section, the choice of (5) seemed a bit weird - in particular, wouldn’t it be non-zero even if fairness is met? I would expect something like the difference in the terms in Eq (4).
- Technically I think it would be helpful to clarify some of the mathematical notation. For example the notion of independence is used throughout for what I understand are realizations (e.g. w_dy independent of a in Table 1), and I’m not sure this is well-defined.

limitations: Yes.
review_title: null

Meta-review: metareview: The paper grounds several fairness notions used in machine learning in principles of distributive justice. The stated motivation is to understand the normative choices behind each and to combat the shortcoming of some of these notions. The main concerns of the reviews were that this grounding is very limited in terms of its scope and there is little actionable insight that follows. Furthermore, many of the connections have already been acknowledged in the literature, e.g. [11, 29]. Philosophical underpinnings of the sciences are very important, as they can help advance both the questions we ask and the answers we offer. The effort of this paper is thus appreciated. However, as it falls somewhat short of advancing either the philosophy or the science, it may be of limited significance to the community. To garner better appreciation of their work, the authors are advised to elaborate on how their grounding could guide the field (e.g., How could one make algorithmic fairness choices in light of this perspective? Have there been instances where the wrong choice was made (algorithmically) relative to the stated intent (normatively)? Are there limitations to this perspective, perhaps in terms of assumptions that should be challenged? etc.)
confidence: Less certain
Decision: Reject