Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper considers the problem of fairness. Given a binary quality, a classifier, and populations groups, how does the interaction of the evolution of the quality and the classifier accuracy evolve? The authors propose a model for this that crucially avoids tying group membership with particular attributes (like race or religion), and instead assume just that the groups are isolated. They then analyze the dynamical system and show that if there are initial disparities, they may persist forever, and finally apply fairness interventions from literature to their model.
main_review: This is an excellent paper and was a pleasure to read, despite it being outside of my area. It addresses a key problem and does so crisply and elegantly. I cannot speak to the technical depth of the results (which is unfortunate for a theory paper) but the quality of the technical writing is great. I don't have many comments, except for I would add a graphic or something of the model to help the reader, and that the interventions section currently is almost incomprehensible for someone not familiar with the related work.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The authors consider how qualification disparities between groups can arise without structural differences between different groups. The authors formulate the problem in terms of a classifier that interacts with groups of agents, where the agents change their strategy over time in response to the classifier, and the classifier learns new classification thresholds. The authors characterize the dynamics of this system and its fixed points, showing that there exist fixed points with unequal qualification rates between groups. The authors then discuss how imposing different fairness constraints on the classifier affect the trajectory of the system, as well as a feedback control intervention that explicitly aims to reduce qualification disparities.
main_review: The paper is very well-written, and the claims are well-supported. The statements are refreshingly precise. My main concern is that the paper could do more to relate its results to existing work in the classical theory of statistical discrimination from economics and with more recent work in fair ML that has built on it.

Most pressingly, I think it is important to discuss how this model is similar or different from the Statistical Discrimination model (https://en.wikipedia.org/wiki/Statistical_discrimination_(economics)) proposed on the 70's by Phelps (https://www.jstor.org/stable/1806107) and Arrow (https://econpapers.repec.org/paper/priindrel/30a.htm, https://www.jstor.org/stable/2646963). In addition, there has been work in this line that considers the impacts of various interventions such as affirmative action (see Coate and Loury 1993 https://www.jstor.org/stable/2117558). The evolution dynamics seem quite similar between these models, so I would not be surprised if they are equivalent. I think the paper also needs to engage more with the Hu and Chen paper that was cited.

However, even if the evolutionary model here is equivalent to previous statistical discrimination models, this paper makes several contributions in terms of presenting this evolution in terms that are more accessible terms, providing a transparent analytical strategy, generalizing these models to an arbitrary number of groups, and showing how current proposals for fairness fit into their dynamics.

One angle that I think the authors could play up is the idea that the only effective intervention here is the feedback control intervention that explicitly considers the dynamics of the system, whereas current fairness metrics take a static view of the system and can interact poorly (or not at all) with the dynamics. Several papers including Liu et al 2018 Delayed Impact (in the context of a Markov model), D'Amour et al 2019 Fairness is not Static (in terms of simple simulations) and Fazelpour and Lipton 2020 Fairness from a Non-Ideal Perspective (from a more abstract philosophical point of view) have argued this point. The authors discuss some of this in related work, but the idea that there are clear unintended consequences of applying fairness criteria obliviously to dynamics is, IMO, underplayed in the current manuscript.

Some smaller concerns:

In my opinion, the motivation of replicator dynamics is quite heavy-handed, and does not characterize prior work as carefully as it could. First, this paper is not unique in assuming that there are no structural differences between groups except for this initial conditions. In addition to the statistical discrimination work cited above, the Liu et al 2018 delayed impact paper is similar (contrary to the description provided with the citation at the top of section 2.2). Secondly, it seems no less contentious to axiomatically assume replicator dynamics in this paper as it does to assume that there are incentives for investment in qualification in other work. In a social system, "fitness" will end up determined by the costs and benefits of qualification, so whether these assumptions are stated up front or tucked into the parameters chosen to characterize a system seems to make little different. I think it would make more sense to frame this work as an extension of previous work (especially if connections can be drawn to the classical statistical discrimination models) where social dynamics can lead to poor equilibria even if there are few structural disparities.

At the very least, I would recommend focusing more of the discussion 2.2 on a plausible story for the emergence of the replicator dynamics. There is a mention of "exchange of qualification strategies" before Assumption 8; elaboration on this point would be useful.

 In (16) and (20), the p-norms of D are missing exponents inside of the summation. The norm notation is also mixed up in the proof of Remark 7 in the appendix (e.g., LHS of eq 63a).

The statement of Remark 7 seems incomplete. The RHS of the iff arrow is just an expression, not a claim.

Theorem 14 should perhaps be rephrased to say that the *decision rule \hat Y* satisfies equality of opportunity under the stated assumptions.
limitations_and_societal_impact: This is adequate.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors are introducing a dynamical system (agents, groups, classifiers, etc.) The agents seem to have fuzzy-like membership of the groups (randomly assigned with some prob.) and they have qualification rates for groups, etc. They have a classifier built upon a specific utility function and it is shown that it is equivalent to having a policy that works by using a simple threshold on the probability of the label conditioned on the feature (Theorem 1). This idea of fitness and replicator equations is used (Eq. 8) to build a system to analyze the stability of the dynamic system. Section 3 which covers that part is a multitude of mathematical arguments showing stability and equilibrium. Section 4 discusses what might happen if there were interventions to make the classifier follow some fairness criterion, like the equal opportunity classification. And it seems to be a direct consequence of this framework. 




main_review: 
The new contribution is the feedback control mechanism which seems to do much better at bringing everyone to a good equilibrium compared to group based fairness measures (The Fig. 3 shows this behavior in an empirical setting as well). Section 4 seems to be interesting. It discusses what might happen if there were interventions to make the classifier follow some fairness setup. Equal opportunity classification seems to be a direct consequence of this kind of setup. 

The goal is clear, but a paragraph or section on the applications/practicality in real-world settings could make the significance more pronounced.


limitations_and_societal_impact: The writing is technical but not unclear. The writing has to be technical in this case. However, some sections could still use some updates. For example, the introduction could be written such that it sets the stage to explain what is being addressed in this paper, and why it would be important and novel in more details. 

The impact it would have due to the assumptions that might constraint the practical applicability of it should be addressed in a paragraph or two as well. For example, the global knowledge of the classifier is one of the assumptions that might make it impractical. In a way, the setup is just relying on the conditional distribution to make predictions. Knowing the conditional probability, is it a common assumption in this space? This needs more detailed explanations and justifications. Also, the claim that this, in practice, can be learned from enough data should be followed by some tangible example or clarification.

All in all, section 5 is good in the sense that the authors discuss their own limitations.
review_title: null

Decision: Accept (Spotlight)