Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper engages with the use of risk assessment instrument (RAI) datasets as benchmarks in fair ML. The paper shows how the criminal justice field, psychology, law, and other fields have come together to create RAIs; what assumptions and values are embedded in the creation and use of RAIs; and how those embedded assumptions and values are unavoidable if RAI datasets are being used. The authors show how the typical framing of benchmark datasets hides these assumptions, which may lead to harmful outcomes---as well as statistically ill-informed results. Given the importance of the criminal justice context, the popularity of ("fair") ML, and the ubiquity of the COMPAS dataset, these authors provide important context and guidelines for the fair ML community.
strengths: The most significant contribution of this work highlights the importance of the criminal justice context, the popularity of ("fair") ML, and the ubiquity of the COMPAS dataset---and how failing to consider the context and values of RAI datasets creates new risks and harms. This is even more true as "fair" systems can exacerbate biases and harms. 

This work shows the context and values embedded in using RAI datasets for benchmarking. This paper draws from lessons from criminal justice and psychology and shows how these are present in the fair ML use case, even if they are not explicitly considered. This work shows the structure of benchmark problems in fair ML and where understanding from criminal justice, psychology, and science and technology studies can help us understand where values and harms are embedded. This benchmark problem structure is combined with some technical references (although see notes about S3 in weakness).

If this paper is (lightly) improved to better integrate its technical discussions, this paper could provide an important warning and call to arms for fair ML, as well as lay out an interdisciplinary strategy to explore benchmark datasets from other domains in fair ML.

Some additional key strengths: 
- Reflects deep disciplinary engagement with criminal justice research. 
- Provides a framework for understanding the structure of benchmark problems, and where it may not align with practical use.
- Provides important warnings about such use, particularly given the ubiquity/importance of RAI dataset use and the lack of engagement with the values implicit in such use. 
- Provides guidelines for the fair ML community (dos and don'ts), some weaknesses of the fair ML community (S6), and some other suggestions for practical use (S5).
weaknesses: Lack of integration or synthesis: Several areas of the paper list technical ideas/results without connecting them to the rest of the paper. This paper does important bridging work, so providing this type of insight would be a major contribution; without it, this paper won't achieve as much in this venue or from this audience.

- In section 3, every subsection ends with a list of technical results without interpretation. Are we supposed to understand that these results solve the problems just introduced? Or are they supposed to show that the field is/is not answering the appropriate question? What should be happening instead, or what work is needed to make this more/less useful? Given the bridging work this paper is supposed to be doing, providing this type of insight would be a major contribution; without it, this paper won't achieve as much in this venue.
- Section 4: This section is about limitations of "fairness," but seems quite focused on points of discretion. How should the reader understand how this section fits in with the rest of the work? Do the authors intend to make a measurement argument (in which case it belongs better integrated with S3) or is the primary argument about limitations of mathematical/algorithmic definitions of "fairness"? Later references to this section as the latter, but the section does not seem focused as such. 
- Section 5 contains some of the core arguments about values that are relevant for this audience; can any of these ideas be integrated earlier or combined with section 4? In addition the section on "fairness objectives" seems very important for this paper, but it is short and the last two sentences are not clearly written.
- Section 6 has good content but it is hard to see the takeaways, and it feels out of place. Can these ideas be combined elsewhere, moved into background, or even just swapped with section 7?

Some ideas are introduced without context, and new terms are sometimes introduced unnecessarily instead of stating the intended conclusion. This will hinder uptake of these ideas by the fair ML community. Some salient examples:

- S3: what is "measurement bias" vs. measurement error vs. other types of threats to validity? Are these problems with the construct itself or its operationalization [re-arrest as re-offense]? 
- S3: Does the term "label bias" mean "bias in the assignment of labels"? If so, then here and in similar instances, new terms appear to be introduced but not defined, and instead could be said more clearly.
- S3: Predictive multiplicity makes sense in its later context, but it is less clear what it is adding here
- S3: "may contribute to distributional harms" -- this could have multiple meanings, and the use of "distributional" is especially unclear for ML audiences -- please translate this point for a broader audience
- S2: relevance, solvability, bioinformatics -- this last sentence is not clear - what is this accomplishing for us?

For S3, two recommended citations for demonstrating different types of lack of validity/biases, particularly for Y and X: 
- A Roberts. 2018. Arrests as guilt. Ala. L. Rev. 70 
- AL Hoffmann. 2019. Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse. Information, Communication & Society

The conclusion seems to be written for researchers who identify as doing criminal justice research, but the target of the paper seems to be anyone using RAIs. Your claim might be that these are actually not so distinct, but this conclusion doesn't make that claim. Some of the stronger conclusions/takeaways are more buried (e.g., last paragraph of S2, last sentence of S5) throughout the text.

correctness: Solid reasoning and arguments, deep disciplinary engagement
clarity: See the "weaknesses" discussion: technical ideas from both the ML literature and the fairness literature are introduced without context, which makes this more difficult for a broader audience to read. Some of the stronger conclusions/takeaways are more buried throughout the text. To successfully do this bridging work across fields, this paper needs to do more of the interpretation (show not tell).

- Section 5 contains some of the core arguments about values that are relevant for this audience; can any of these ideas be integrated earlier or combined with section 4? In addition the section on "fairness objectives" seems very important for this paper, but it is short and the last two sentences are not clearly written.
- Section 6 has good content but it is hard to see the takeaways, and it feels out of place. Can these ideas be combined elsewhere, moved into background, or even just swapped with section 7?

Lesser details:
- S3: problem with FTA as binary outcome - the last sentence seems out of place
- S3, last section: This first sentence is hard to parse and understand, and needs to be better connected to the second.
- S5: this section title is hard to parse. Can this be "Normative Values Embedded in Use of RAI Datasets" or similar?
- S5: this text is hard to parse, technical terms undefined, last two sentences difficult to read
- S6 has some spelled-out references rather than numeric.
Some unnecessary acronyms:
- DUA, STS, PSA (as well as typo in footnote 4); CS instead of spelling out or instead of AI/ML
- could drop acronyms from "educational and psychological research associations AERA, APA, and NCME" 
- FTA should be introduced where it is being used regularly; right now it is introduced twice, and the first time out of useful context

relation to prior work: Yes - integrates criminal justice, science and technology studies, psychology, and fair ML. 
additional feedback: This could be important work: my comments point to ways to clarify and help ML/CS readers take the full messages away from this paper. I believe this is achievable editing before a camera-ready and would make this paper much more impactful.
review_title: Integrated and interdisciplinary perspectives are needed to understand how sensitive datasets are used as benchmarks in fair ML

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper discusses the usage of RAI datasets (such as the COMPAS datasets) in CS research. It highlights what issues may arise when using such datasets and critiques their usage for benchmarking purposes, which typically comes at the cost of understanding the data’s context. This leads both the reader and the paper to ask whether RAI datasets can even be used in an ethical manner in CS research.

The paper covers a lot of ground and discusses the following topics:
- biases in the dataset (X, Y and A)
- differences between deploying RAI systems in theory and in practice
- implicit consequences of using RAI datasets (e.g., legitimizing current CJ system instead of working towards reform)
- how other fields work with RAI datasets and what CS researchers can learn from them
- critique of research practices in ML fairness (e.g., focus on methods papers and the disconnectedness from the data context)
- suggestions for what to do about these issues in the future
strengths: The paper provides a comprehensive analysis of the potential issues that arise when using RAI datasets to e.g. test new methods. It brings together work from CS, CJ, psychology and other fields to provide this overview. This intersectionality is extremely valuable as it offers a perspective on how CS’s usage of RAI datasets differs from that of other fields - which have been using RAI datasets for much longer than CS. These insights are particularly helpful for researchers who are considering using e.g. the COMPAS dataset, but haven’t thought through what the consequences of this might be. It can thus create an increased awareness of potential ethical issues. Lastly and most importantly, the paper raises very interesting questions and could start a much-needed debate about the usage of real-world benchmarking data that comes from socio-technical systems.
weaknesses: The paper covers a lot of topics and for the most part achieves to do this in an accessible manner. However, some parts appear too condensed (potentially due to the space limit) and could profit from a bit of elaboration, in particular:
- lines 103-105: The explanation of construct (in)validity in recidivism predictions could profit from a brief definition of construct validity as CS researchers are in my experience not familiar with the concept unless they are familiar with [27] and related work.
- Methodological approaches regarding the reduction of biases could be explained in more detail. For example, “Wang et al. [41] studies robust evaluation and classification.“ (line 140 f.) might be a helpful reminder to readers familiar with that work, but otherwise appears difficult to follow.

correctness: The claims made in the paper are backed up through research from various fields, such as CJ and psychology.
clarity: The paper is a pleasure to read as it is well-structured and well-written.
relation to prior work: To my knowledge, previous contributions have rarely looked at the potential issues of using RAI datasets in experimental CS studies. The paper draws on existing work to demonstrate these issues. This is done in a clear manner.
additional feedback: Here is a list of typos / smaller issues I noticed while reading the paper:
- reference [122] could be updated to the FAccT paper, which should include Maria De-Arteaga as an author
- line 81: is there a word missing? “ML studies have mainly explored the former through “benchmark misalignment” which arises with systematically unstable labels [22, 23], but misalignment in the context of socio-technical systems is *rarely(?)* discussed.“
- reference [27] could be updated to the FAccT version
- delete one “in the first place” in line 177: “RAI datasets suffer from selective labels, e.g. that outcomes (Y ) recorded in datasets are only recorded post-selection into the dataset in the first place since observing re-arrest or non-re-arrest is only possible for individuals not detained in the first place [55].”
- line 187: institution → institutions
- line 237: still stands and is → still stand and are
- line 243: tends to favor → tend to favor
- line 243 f.: favor equality of process of deciding an outcome rather than the outcome itself → favor equality in the process of deciding an outcome rather than in the outcome itself

Personally, I’d be very curious about the authors’ thoughts on how simulated data could be helpful in addressing the discussed issues, but I recognize that this might be difficult to add due to page limit constraints.
review_title: Raises very important questions on benchmarking in fair ML

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper highlights the problems of overused datasets in the field of fair machine learning, especially when such datasets pertain to the field of criminal justice (CJ). I commend the authors for choosing this problem to expand on, as the COMPAS dataset released by ProPublica has become the de-facto benchmark for almost every paper describing a technique on fair classification. I particularly enjoyed the discussions about the different metrics used by Psychology than CS for CJ datasets, the specific focus on the pitfalls of method papers wrt conference cycles and blindly trying to apply those ideas to sensitive CJ datasets. 

My main point of contention is that in my opinion, this paper is lacking an empirical analysis or even a value-based, methodological analysis of the pitfalls of the COMPAS dataset (since it is the most representative of the widely used CJ datasets). The abstract made it seem like this would exist somewhere in the paper, and yet after going through the entire paper (and the appendix), it seems like an actual dataset discussion, for instance, a table summarizing the data distribution, the features -- and what how these features were defined by CJ folks vs how they are misinterpreted by computer scientists without domain knowledge, is missing. I am aware that this has been critiqued widely in the past, and that the authors have profusely cited a lot of previous literature which looks into this, but considering the goals of the Datasets and Benchmarks Track, this paper narrowly fits into the "Frameworks for responsible dataset development" silo. As described above, a table or section- even somewhere in the appendix - would make the paper several times stronger because it would add an audit and fit the venue much better. Fortunately, this is addressable and I am willing to give the paper the high score it deserves if the authors do so.

Other nits: 

- "For example, it is not uncommon for the experiment design to be so disconnected from context that a prediction of high-risk is treated as the preferred outcome for an individual in an equal opportunity model simply because high-risk is the “positive” label in the dataset." Citations?
-" It is also not uncommon for “granted bail” to be described as the positive decision, implying that the many individuals who are unable to afford bail [citations] being incarcerated during the pretrial period is positive.". I had to read this sentence several times and then search about it to understand the (brilliant) point that the authors have made. Point being, this sentence assumes a lot of domain knowledge about the US prison bail system from the reader and could be reworded or expanded to make it slightly more accessible.
- Section 6 and Section 8 are both guidelines, and I wonder if they could be reworded/rearranged to highlight their individual contributions more. Perhaps rewording section 6 as established norms and section 8 as proposed guidelines (although call to arms is great too) might be better?
- Citations assume that CJ is almost entirely used in classification problems. Unfortunately in my experience, they are also retrofitted by authors in other use cases like ranking, clustering, recommendations, etc. Citations on those and talking briefly about those harms might be great too if it can fit the narrative of the paper. 
- "Some methodological approaches suggest using proxy measures (e.g. covariate-based proxies such as the distribution of race by geographic area, or viewing observed...". There's a bunch of empirical and policy work you could cite here, for instance 
      - https://arxiv.org/pdf/1912.06171.pdf
      - https://arxiv.org/abs/2011.02282
      - https://arxiv.org/abs/2105.02091
      - https://www.latimes.com/business/la-fi-rand-elliott-20160824-snap-story.html
strengths: - Very strong focus on the foundations/sources of CJ datasets 
- Illuminating takes on the differences in metrics in CS and other domains 
- Good points about the conference cycle and the impetus towards doing benchmark studies
weaknesses: - As discussed above, would really love to see a case study, specifically show the pain points in the COMPAS data as a summary 
- Citations could be improved here and there, I provided some suggestions
correctness: Submission is neither a dataset nor a benchmark. Other claims made in the paper hold up.
clarity: Yes
relation to prior work: Yes
additional feedback: -
review_title: Review for paper

Meta-review: This paper examines limitations and broader social consequences of the widespread use of risk assessment datasets -- specifically COMPAS-- within algorithmic fairness benchmarking. The authors demonstrate how these datasets embed values and assumptions that can lead to harmful outcomes, invalid results, and are poor methods of measuring real-world impact. 

There is clear consensus amongst the reviewer that this paper is a strong contribution to this track. Reviewers note that the authors engage deeply with relevant literature, provide a tangible framework for understanding the structure of benchmarking problems, and provide clear guidelines for ML fairness practitioners. 

Reviewers note that the paper is quite dense and the camera-ready draft could be improved (and made more accessible to a wider audience)  by elaborating on technical ideas from ML and algorithmic fairness literature as well as concepts that might not be familiar to the average ML practitioner, e.g. concepts relating to measurement modeling. 

Decision: Accept