SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Elliott Ash $^{1*}$  Naman Goel $^{2*}$  Nianyun Li $^{1*}$  Claudia Marangon $^{1*}$  Peiyao Sun $^{1*}$
<sup>1</sup>Center for Law & Economics, ETH Zurich
$^{2}$ Department of Computer Science, University of Oxford
[elliott.ash, claudia.marangon]@gess.ethz.ch, naman.goel@cs.ox.ac.uk
Machine learning based decision-support tools in criminal justice systems are subjects of intense discussions and academic research. There are important open questions about the utility and fairness of such tools. Academic researchers often rely on a few small datasets that are not sufficient to empirically study various real-world aspects of these questions. In this paper, we contribute WCLD, a curated large dataset of 1.5 million criminal cases from circuit courts in the U.S. state of Wisconsin. We used reliable public data from 1970 to 2020 to curate attributes like prior criminal counts and recidivism outcomes. The dataset contains large number of samples from five racial groups, in addition to information like sex and age (at judgment and first offense). Other attributes in this dataset include neighborhood characteristics obtained from census data, detailed types of offense, charge severity, case decisions, sentence lengths, year of filing etc. We also provide pseudo-identifiers for judge, county and zipcode. The dataset will not only enable researchers to more rigorously study algorithmic fairness in the context of criminal justice, but also relate algorithmic challenges with various systemic issues. We also discuss in detail the process of constructing the dataset and provide a datasheet. The WCLD dataset is available at https://clezdata.github.io/wcld/.
Machine learning algorithms are increasingly being used or proposed to be used in various sensitive use-cases, including criminal justice. In such scenarios, a major concern is that the algorithms can perpetuate and amplify different forms of social biases. There is plenty of empirical evidence supporting this concern in the literature (1). For example, the seminal work by ProPublica (2) found COMPAS, a recidivism risk estimation tool, to be unfair against black defendants: making different errors for different demographic groups and at different rates. While any component in the machine learning pipeline can introduce bias and there are also fundamental limitations of algorithms (3; 4), it is also widely acknowledged that training and evaluation datasets are major contributors (5; 6) to unfairness in algorithms. Therefore, it is important for academic researchers to have access to large datasets to study the problem rigorously. Further, the datasets should also contain rich information to allow understanding the interaction of data and algorithmic issues with various systemic issues. A standard research dataset used in this context is the COMPAS dataset (7). While this dataset has been extremely useful so far in bringing attention to some of the issues with machine learning algorithms, it is often limiting in improving our understanding of the algorithms' behavior because of its small size and other limitations (discussed in Section 2). Further, researchers often use synthetic datasets and simulations (8; 9; 10; 11; 12) to complement their analysis. Availability of large scale real-world datasets will bring more grounding to this complementary analysis approach (13; 9).
In this paper, we contribute a large scale curated dataset of criminal cases from circuit courts in the U.S. state of Wisconsin. The dataset has been constructed from publicly available raw case records, to make that information easy to use in machine learning research. The dataset has a defined set of attributes and prediction variable for research. The attributes include type of offense, sex, race, prior criminal count (for each offense type), age (at judgment and at first offense) and neighbourhood characteristics like population density, proportion who attended college, proportion eligible for food stamp, African American population share, Hispanic population share, proportion of male, proportions who live in rural and urban area and median household income. The prediction variable for research is the observed recidivism within a follow-up period of 2 years, measured in different ways. We also provide more information for each row in the dataset that can be used for further analysis. This information includes charge severity, case decisions, year of filing, sentence length, charge severity for previous cases, sentence lengths for previous cases, pseudo-identifiers for judges, county and zipcode. Mappings for county and zipcode can be made available to the researchers on request. Information that we have excluded from the dataset are name of the defendant, their address, date of birth, dates of case events etc. There are about 1.5 millions rows in the dataset. We will also strive to add more features to this dataset in the future.
We hope that the academic research community will find this dataset useful not only for benchmarking machine learning and fairness algorithms, but also for critically understanding the strengths and weaknesses of various algorithms and how these issues interact with biases due to geography, time, judges, sentencing etc.
In the rest of the paper, we describe the process of constructing the dataset from raw case records accessed through the Wisconsin Circuit Courts Access API. We discuss various choices made while curating the dataset and their relevance in using this dataset for research. We provide summary statistics of various features in the dataset. For reference, we also report the performance of two basic machine learning classifiers (logistic regression and XGBoost) on the dataset. Appendix F contains datasheet (14) for this dataset.
The COMPAS dataset (7) is one of the most frequently used dataset in machine learning fairness literature. It has around 7000 observations from a single court (Broward County, Florida) over two years (2013 and 2014). It has very few samples for racial groups other than Caucasian and African-American. The dataset is limited to the set of defendants assessed with COMPAS at the pre-trial stage, and it does not include information on judges or other officials. The validity of this dataset as a reliable fairness benchmark is often disputed (15).
(16) contribute datasheets for 15 publicly available datasets in criminal justice from the US: such as survey data from victims, policing data, court records, data about enforcement agencies and officers, data about extremists etc. The datasets listed Section 4.4 of their paper are the most relevant in the context of our contribution. However, those are mostly raw case records, not datasets with curated attributes and recidivism outcomes. (13) survey a few other datasets in criminal justice domain that have similar limitations. (17) contribute a dataset that contains information on the personnel, activities, use of force, and complaints in the Chicago Police Department (CPD).
In the field of machine learning fairness more broadly, there are several notable works that introduce new larger datasets or make larger datasets more usable. For example, (18) construct five datasets related to income, employment, health coverage, commute time, and housing. They observe that many results vary with the data highlighting the importance of more data-centric empirical research. (15) release a toolkit to make the Home Mortgage Disclosure Act datasets more easily usable. They also conclude that the variance in small datasets like COMPAS can muddle the reliability of conclusions about algorithmic fairness interventions. We refer the readers to (19) for a detailed survey on datasets used in algorithmic fairness research.
In a recent paper (13), we reported results of a study that was made possible on subsets of the WCLD dataset. In that study, we used simulations to create several semi-synthetic datasets from the WCLD dataset to understand the impact of a few data-centric factors on algorithmic fairness. However, this is only one example and we believe that researchers will use this dataset for understanding various other issues as well. Further, the study in (13) used only some of the information that we are releasing with this dataset. For example, the additional information on judge, county, detailed offense
types (wcisclass), charge severity, probation, detention, sentencing etc can be used for understanding various systemic biases and how they interact with algorithmic fairness interventions. We have now also curated a variable on violent recidivism. With this submission, we also provide a detailed datasheet for the dataset, lists its various limitations and describe in detail the origin and the process of curating the dataset. This datasheet and all the other details will be useful for researchers while considering conducting further research with this dataset, and even to inform decisions about further possible improvements in the dataset.
3.1 Wisconsin Circuit Courts Access (WCCA)
WCCA was created in April 1999, and it contains public case records and docket information from the 72 county courts of the U.S. state of Wisconsin. The WCCA website allows the public to search and read the records. The court record summaries provided by WCCA are public records under Wisconsin open records law sections 19.31-19.39 of the Wisconsin Statutes.
The REST interface of WCCA is a subscription based service provided by WCCA to allow automated processes to download and monitor public circuit courts information. We purchased a subscription to this service in May 2020 for the purpose of academic research. We paid 12,500 USD to the 'Wisconsin Supreme Court' from our research budget, for 12 months of electronic access to WCCA information through the REST interface.
3.2 Case Records
Using the REST interface subscription, we were able to download dockets from 1970 to 2020. We queried all case numbers in each county during the period and subsequently, using these case numbers, we queried individual cases for all available information. There are around 11 million records, out of which around 2.5 million are criminal. The case records include the charges in current offense, the outcomes and sentences of cases and the defendants' demographic information (e.g. sex, race, address, and date of birth). There is additional information available about various case events (e.g. hearings, bail decisions, bail amounts etc), information on the associated attorneys and government officials involved, including prosecutors and judges.
3.3 Attributes
From these raw case records, we set out to create a curated dataset with numerical and categorical attributes and prediction variable for research. Attributes like type of offense (felony, misdemeanor and criminal traffic), wiscclass (more specific categorization of offense), year of filing, defendant sex, and race do not require much effort to extract as these are directly available in the case records. As per WCCA declaration (please see the datasheet in appendix F): "In criminal cases, any designation in any race field contains subjective information generally provided by the agency that filed the case." We found that for each case, the defendants are assigned exactly one race. For a small number of defendants, different cases assign a different race to the same defendant. This doesn't necessarily imply mixed-race but could be (and perhaps more likely to be) a data quality issue. More specifically, for  $\approx 0.3\%$  of the defendants, there were more than two races reported (across different cases). For another  $\approx 0.3\%$  of the defendants, there were two races reported (across different cases) and none of the two races was caucasian. There was perhaps no obvious way to determine the race in these records. These records are not included in the dataset. For  $\approx 5\%$  of defendants, there were two races reported (across different cases) and one of the two races was caucasian. We observed that for most of these defendants, non-caucasian race was the most frequently reported race in cases associated with the respective defendant. For reporting model (un)fairness in Section 4.2, we used the non-caucasian race, but a separate column called 'all Cases Races' in the dataset indicates all the races reported for the defendants across cases.
We construct the rest of the attributes and the variable for prediction from the information that is indirectly available in the case records. We use a combination of first name, last name, and date of birth as a unique identifier for a defendant. This identifier allows us to conduct a search in the
database of case records to match the defendant across multiple cases and construct the additional attributes  ${}^{2}$  . The detailed process of constructing the additional attributes is described as follows.
3.3.1 Prior Criminal Counts
Using the database search, we obtain the prior count of each of the three crime types - felony, misdemeanor and criminal traffic - of the defendant for each of the cases. We were able to collect cases from as early as 1970. We use all these case records for constructing the prior criminal count.
3.3.2 Age
We also infer age at judgment and age at first offense for each case. Age at judgment is calculated based on the date of birth of the defendant and judgment disposition date of the case. Age at first offense for each case is the age when the defendant committed the first crime found in the database.
3.3.3 Neighborhood Characteristics
We link 9 local demographics variables to our data from a zipcode level dataset processed from 2010 census data (20). These are population density, proportion who attended college, proportion eligible for food stamp, African American population share, Hispanic population share, proportion of male, proportions who live in rural and urban areas and median household income. For around  $3\%$  of the cases in our dataset, the records didn't contain zipcode. For such cases, we have used average county characteristics. Note that the neighborhood characteristics are from the year 2010, not necessarily from the respective years of individual cases. However, we include the year of filing for each case in the dataset. Researchers can use the year variable to link with external datasets for respective years (not only neighborhood characteristics but other information that they may consider interesting and relevant). Zipcode for the cases can also be made available to the researchers on request for this purpose, for example.
3.3.4 Charge Severity
Felonies and misdemeanors are sub-classified by severity level in Wisconsin. Charge severity in raw case records accessed through the REST interface can be Felony A, B, BC, C, D, E, F, G, H, I, U and Misdemeanor A, B, C, U. We map each charge severity to a unique numerical value. The numerical values reflect the increasing charge severity. Mapping is provided in Table 5 in Appendix G. There can be multiple charges in a case; we only keep the highest charge severity for each case.
Using the defendant identifier, we also created the prior counts of charge severity for each prior charge of the defendants. These columns (for each of the 15 charge severity types) are also available in the dataset.
3.4 Sentence Length
We calculate the sentence length using the available information about the number of assigned days of jail. In case of multiple charges in the same date, we take the maximum of jail sentence for all charges. If this value is zero, we use the available information about the number of assigned days of house correction (maximum value in case of multiple charges).
For each row, we also provide summary statistics for the previous sentence lengths in four additional columns (max, min, mean, and median sentence length).
3.4.1 Other Columns
The dataset contains two other binary variables: not_detained and probation. not_detained = 0 implies that the defendant was detained in the case, and not_detained = 1 implies released, probation or sentence. The other variable, probation = 1 implies that the defendant got probation.
It may be noted that this identifier is not robust to name change by some of the defendants (which we cannot rule out). We provide this identifier in the dataset as a separate column.
3.5 Recidivism
The choice of the prediction variable in machine learning is a difficult and value-laden decision (21). In this dataset, we provide a commonly used variable in machine learning research, which is whether the defendant recidivates or not. There are well-founded ethical concerns about recidivism prediction task. Please see Section 5.3 for important discussion on ethics. We provide this variable in the dataset to facilitate empirical research on (fundamental) limitations and/or opportunities for improvement. Please see Section 5.2 for discussion on intended uses and limitations. We emphasise that the researchers can use this dataset for other research questions as well and do not have to necessary use the recidivism variable (as a prediction variable or otherwise).
To construct this variable, we had to decide on a follow-up period within which committing a new offense is deemed as recidivism. We pick the usual choice (2; 22) of 2 year follow-up period. Specifically, we measure whether the defendant would commit a crime within two years from the date of judgment.
In any dataset, whether we observe recidivism or not for a defendant is fundamentally biased by the decisions of judges and sentence. This is a common problem (23; 24). Further, defendants serve different sentence lengths and depending on the sentence, it may not be possible to observe recidivism within 2 year follow-up period. Assigning a missing outcome to every case with a sentence throws away a lot of useful data. Yet extending the follow up period for two years after the assigned sentence period instead of the judgment date is also problematic because defendants often serve more or less than the assigned sentence. Since there is not a comparable data source that has the exact jail record of every defendant in Wisconsin, we don't observe the actual (served) sentence length. There is no consensus in the literature about how to deal with this problem. We use a cutoff for sentence length, of 180 days, such that we don't have to throw away a lot of useful data and still leave enough time in the follow-up period for the defendant to reveal crime potential. Above this cutoff, we treat the defendants' outcome as missing, even if they do not re-offend within the follow-up period. As an alternate, we can also define the recidivism variable by varying the sentence length cut-off from 180 days to 2 years, and extending the follow-up period of 2 years by adding the sentence length. Note that the second way takes into account the sentence length given by the judge, which creates even more bias in recidivism variable. Both these recidivism variables are available in our dataset.
3.6 Violent Recidivism
In machine learning fairness research, violent recidivism (as opposed to recidivism without any specific criminal context) may also be considered (7; 25). In the FBI's Uniform Crime Reporting (UCR) Program, violent crime is composed of four offenses: murder and nonnegligent manslaughter, forcible rape, robbery, and aggravated assault. In our setting, the challenge in constructing a violent recidivism variable is annotating the large number of records as violent/non-violent. In the raw case records, we have textual descriptions for various charges in a case. There were about  $\approx 45,000$  charge descriptions across records, making manual annotation difficult. To the best of our knowledge, there are no publicly available violent/non-violent classifications of these charge descriptions. Note that charge descriptions are different from type of offense (felony, misdemeanor and criminal traffic) and wiscclass columns. We used GPT-4 (26), a pre-trained large language model, for categorising charge descriptions as referring to violent or non-violent crimes. We provided GPT-4 with the FBI's UCR definition of violent crime in the prompt and asked it to classify a given charge description as violent/non-violent while spelling out the thought process or explanation for the classification. Appendix H provides further details about the process. Due to lack of any ground truth, we do not have a quantitative assessment of the accuracy of the violent/non-violent labels thus obtained. But we did observe several limitations of this approach. We found that GPT often struggled to label charge descriptions that were ambiguous, had insufficient context or contained abbreviations. We also noticed that if prompted multiple times, GPT could classify some charge descriptions differently due to ambiguity and provided plausible explanations for both classifications. We manually chose classifications for a very small fraction of these instances but there can be more such inconsistencies that were not caught. In some other cases, it explicitly stated that due to insufficient information, it could not provide a classification. Further, charge descriptions may have standard meanings in Wisconsin courts which GPT may not have access to in its training data or may not have taken into consideration due to our prompt's focus on FBI's UCR definition. Thus, the process is not perfect and certainly not a replacement for expert annotations.
Using the classifications provided by GPT, we identified whether at least one of the charge descriptions in the case referred to a violent crime. Similar to the recidivism variables in the previous section, the violent recidivism variable is also binary (1 meaning a violent recidivism was observed within a two year follow-up period, with 180 days sentence length cutoff) and contains missing values. Due to space constraints, we do not discuss violent recidivism variable in the rest of this paper, unless explicitly stated. In addition to violent recidivism, another column in the dataset marks whether the current case had a violent charge. We also make available the violent/non-violent classifications for charge descriptions and the explanations provided by GPT as a separate file.
3.7 Data Filtering
Through the REST interface, we were able to download case records from as early as 1970, but the records in earlier years tend to be incomplete and the number of cases much smaller. Therefore, we only keep the cases from 2000 in the dataset. It only means that the rows in the curated dataset are only those cases that appear in the courts from 2000. The pre-2000 information for such cases is still included in the form of prior criminal count of defendants. For around  $0.005\%$  cases in the dataset, WCCA filing date of older cases is during or after 2000. We have not removed these cases from the dataset. Given the 2 year follow-up period, we exclude cases that are disposed after 2018 since there is not enough time to observe recidivism (we had case records until 2020 only). Note that the post 2018 information is used for observing recidivism outcomes in the dataset. We also point out that during pre-processing, we excluded dismissed cases that do not result in conviction. We also deleted records of defendants that do not have sex and/or race data available or if race could not be determined from the available conflicting race information across cases (see Section 3.3 for details.). We also excluded cases that only have forfeiture (non-crime) charge. Information excluded in pre-processing step is not used in the dataset.
3.8 Redacted Information
While the case records on the WCCA website contain personal information of the defendants, the website also displays notice to the employers informing about the violation of state law to discriminate against a job applicant because of an arrest or conviction record. As an ethics consideration on our end, we have removed personal information from the curated dataset that we contribute in this paper. In particular, we have excluded the names of the defendants, their addresses, dates of birth, dates of case events, case numbers etc. We have also replaced information about judge, zipcode and county with pseudo-identifiers, but mappings for zipcode and county can be made available to researchers on request.
4 Exploratory Analysis
4.1 Summary Statistics