Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This dataset contains information on the personnel, activities, use of force, and complaints in from over ~35,000 police officers in the Chicago Police Department spanning over a decade. Prior to the authors’ extensive and well-documented procedures for cleaning of this data obtained from several FOIA requests, the data was unlinked, inconsistent, and furthermore undocumented. This dataset – which takes vasts amounts of data that should be made available to the public but is often inaccessible due to the significant amounts of time needed for both FOIA requests and preparation/cleaning – could enable a variety of new empirical research on patterns of law enforcement as well as the potential impact of new interventions that has prior to its launch not been available to the public. The dataset *and* the documentation surrounding how it’s processed – centers on a important and relevant location in the US where investigations of policy activity are common but empirical evaluation is thin – in part due to the prior inaccessibility of sufficient data. The authors outline several important possible strands of research that this important and extensive dataset enables, ranging from prediction to causal inference and beyond. 
strengths: This dataset – which takes vasts amounts of data that should be made available to the public but is often inaccessible due to the significant amounts of time needed for both FOIA requests and preparation/cleaning – could enable a variety of new empirical research on patterns of law enforcement as well as the potential impact of new interventions that has prior to its launch not been available to the public. This dataset is the culmination of several years of perseverance and commitment to unveiling what is often obfuscated. 
weaknesses: The dataset on this important city is very relevant. If only we could have access to similar datasets across the country!

correctness: The dataset appears to be constructed soundly and is well documented. Where assumptions were needed in order to proceed with processing, those are clearly indicated in the dataset. Observed limitations are made explicit.
clarity: The paper is well written and clear. The graphics are also helpful and well-explained.
relation to prior work: The manuscript reflects a clear linkage to the existing problems in the literature and a knowledge of what their dataset has the potential to enable as distinct from other difficult to access datasets that are also limited in scope.
additional feedback: Bravo! This is an exceptional dataset and cleaning protocol which has been built on many years of perseverance and teamwork.
review_title: Valuable dataset and processing code to support analyses of critical social issue in an important geography

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The researchers provide a cleaned and pre-processed dataset about individuals in law enforcement. This is a novelty because data about police interactions so far has focused on just one side: the members of the public that have interacted with the police – but not the police itself. Because of this skewed access to data, it is incredibly difficult to study police and usually only indirectly possible. While the data that the paper publishes is not new (it has been collected, used and published by Invisible Institute), the paper makes the data easier to use for other researchers. This paper therefore makes a significant contribution in providing clean and easy-to-use data on police officers.
strengths: - The authors provide data on police’s use of force and possible complaints – this is the opposite of the typical approach which is to collect data about the people interacting with police.
- The authors did a great job in preprocessing the data and documenting the process.
- I really appreciate the datasheet in Appendix A. I was wondering if that could be provided in a more prominent place in the repository – perhaps as its own .md file at the top level of the repo?
weaknesses: - The original repository by Invisible Institute doesn’t seem to be particularly active at the moment. Why did the researchers feel compelled to create a new repository instead of contributing to the existing one?
- Could the authors say something about to what extent we can expect findings from this dataset to generalize? For example, does this data generalize to other cities / states / countries? What’s different about the CPD that would limit generalizations?

correctness: The dataset has been constructed in a sound way from the existing raw data. The authors clearly highlight limitations of the dataset, such as missing data and difficulties with linking the dataset.
clarity: Overall, the paper is written in a very clear way. There were just some smaller things that would be helpful to clarify:

- Line 67 ff.: “Furthermore, the Department of Justice (DOJ) only began last year to collect data on arrest-related deaths through the implementation of the “Death in Custody Reporting Act” of 2013 [10].” --> It is unclear to me why the collection of data only started last year if the Act is from 2013 – is this because of delays in the implementation of the Act?
- Line 220 f.: “Unless otherwise specified, an officer is considered “active” if their resignation date is after 2019-01-01.” --> I don’t understand why officers who have resigned are considered active. Could you provide your reasoning for that?
- Figure 1, bottom: This figure is unclear to me. Only the red dot appears in the legend, what do the blue dots represent? And do the red dots show us that for officers born before 1950, the resignation date is basically always missing? The insight from this figure could be stated in the text in 1 or 2 sentences.
- For people unfamiliar with (American) law enforcement: What are awards and when are they typically requested?
relation to prior work: Related work is appropriately discussed. One work to potentially bring up is “Studying Up: Reorienting the study of algorithmic fairness around issues of power” by Barabas et al. as the CPD dataset allows for what Barabas et al. call “studying up,” i.e., studying people and institutions with power, such as the police.
additional feedback: - Line 211 f.: “and set as the end date of each assignment, the day immediately preceding the start date of the following assignment” --> the comma after “assignment” seems to be misplaced
review_title: Important work that allows researchers to study police misconduct

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper addresses a lack of open and reproducible software and data for investigating police activity. The authors describe a method for cleaning and linking Freedom of Information Act (FOIA)-acquired data about the Chicago Police Department. They provide code to easily perform this cleaning and linking task. The paper provides brief exploratory analyses and discusses intended uses of the dataset and some ethical concerns with predictive work in this domain.
strengths: - Well written paper and organized code repository.
- The work described is already a great contribution to the research community and has a high likelihood of creating positive impact. Open and reproducible approaches to cleaning and linking data are very valuable. The effort to do so with a high level of transparency (e.g. open code, open GitHub issues outlining future directions) and many worked examples (e.g. Jupyter notebook files) will be helpful. 
- The general approach can be useful for other contexts, especially for linking various documents and releases obtained via FOIA or similar procedures. It may even be possible that some of the software and methods choices are directly re-usable.
- The topic is of high societal importance.
- Good overview of challenges with this kind of data.
weaknesses: - Some of the insights gleaned from exploratory analysis seemed brief and weakly supported. More discussion of this point in "Correctness" below.
- References to CPD data infrastructure were a bit unclear (e.g. "probably due to internal errors in the CPD data infrastructure", ) Is there any way to confirm the "data infrastructure bug" hypothesis in the future (e.g. a new FOIA request aimed specifically at answering this question?), or is this likely the best we can do?
- Cursory connection to journalistic coverage on the topic, which is a highly relevant area of related work. For instance, is there any corroborating evidence for decline in complaints or an explanation for why this happened? Some specific examples below in "Relation to Prior Work"
- Discussion of negative impacts could be expanded, especially regarding the extent to which many of the problems with using police-generated data for ML will persist for using this data. Some specific examples below in "Ethics".
correctness: Data cleaning choices seems uncontroversial. Procedure to build a database of officer UIDs seems correct, although it may help to further describe any manual validation procedures (e.g. did authors look at a sample of merged entities?).

The procedure for generating "canonical profiles" of officers seems reasonable and is explained well.
Documentation of each FOIA data file is important for researchers seeking to understand the history of the data.

The conclusions of the exploratory analysis were introduced very briefly. Overall, it was not clear if the data as provided was enough to support these conclusions (e.g. the claims about why complaint volume has fallen, the explanation for disparities in complaints by officer demographics). It may strengthen the paper to be make more specific suggestions around what external data sources or new analysis techniques are needed to answer these questions (e.g. more FOIA requests? Scraping data? Using causal inference techniques or quantitative fairness techniques? etc.)
clarity: Organization of the paper is very clear and the contribution is communicated well. No concerns here.
relation to prior work: Quick summary of work drawing on CPD FOIA files is provided (specifically, refs 12,13, and 14 from the paper). It may be useful to additionally specify what kind of findings this work produced, i.e. to contrast and compare with the intended uses of this dataset.

Regarding connections to journalistic coverage of police complaints, etc., articles along these lines could be relevant to discuss in the body of the paper. Or, if the authors think this is out of scope for this particular paper, explicitly mentioning this kind of work could be a helpful lead for users of the dataset.
- This article from the Chicago Tribune [1] mentions that "Nearly 60 percent of all the complaints were thrown out without being fully investigated because the alleged victims failed to sign required affidavits."
- Another Tribune article [2] mentions that "a Tribune analysis of police complaint data showed about 60% of approximately 17,700 complaints over a four-year period ending in mid-December 2014 were thrown out because there were no signed affidavits."


[1] Gorner and Hing, 2015. Tribune analysis: Cops who pile up complaints routinely escape discipline https://www.chicagotribune.com/news/ct-chicago-police-citizen-complaints-met-20150613-story.html 


[2] Gorner, 2021. Filing a complaint against a Chicago cop? You wouldn’t have to sign your name to it under proposed law https://www.chicagotribune.com/news/criminal-justice/ct-chicago-police-affidavits-anonymous-complaints-20210129-str7gjrkkzaf3bqypclw7na3t4-story.html 
additional feedback: Very minor comment: the Jupyter notebooks in current version of the repo are "un-run" versions. The authors may want to consider generating some examples as HTML or PDF for some interested users.

I was also curious if there was a strong reason for not providing the `final` data files directly in the repo (e.g., to make sure users have the most up-to-date version?). This may be useful for some dataset users.
review_title: Describes an approach and provides code for reliably cleaning and linking various FOIA-requested data about the Chicago Police Department.

Decision: Accept