SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1 presents summary statistics of the dataset thus constructed with around 1.5 million cases from 2000 to 2018. There are five race groups in the dataset, with around  $65\%$  Caucasian,  $23\%$  African American and  $7\%$  Hispanic. The 'Asian or Pacific Islander' and 'American Indian or Alaskan Native' groups are abbreviated as 'Asian' and 'Native American' respectively. The proportion of male defendants in the dataset  $(80\%)$  is significantly higher than female, and most crimes in the dataset are committed at a younger age (below 60). Misdemeanors are the most frequent crime type except for Hispanic with criminal traffic  $(45\%)$  being the most common crime type.
Table 2 shows the summary statistics for the two recidivism variables, when the sentence length cutoff is 180 days and 2 years as discussed in Section 3.5. While the rate of missing prediction variable in the dataset varies significantly in the two variables, the recidivism rate (excluding 'missing') doesn't show the same level of significant difference. In the rest of the paper, we use the recidivism variable with 180 days cut-off length, unless otherwise stated. The observed recidivism rate is the highest ( $\sim 56\%$ ) with Native American.
Figure 1 shows the recidivism rate for different races over the years. Overall recidivism rate and the recidivism rate difference between groups is decreasing over time. We also observed marginal shifts in the distribution of offense types, group proportions, sex, and age. We discuss the limitations of this interpretation in Section 5.2.
Table 1: Summary of the Dataset
Full sample	Caucasian	African American	Hispanic	Native American	Asian
Sample size	1,476,967	964,922	333,036	101,607	63,862	13,540
Sample share	Sex	65.33%	22.55%	6.88%	4.32%	0.92%
Male	80.40%	79.05%	83.47%	88.88%	69.65%	87.57%
Age	Below 30	51.38%	49.45%	54.13%	56.91%	53.71%
30 to 60	47.44%	49.09%	45.17%	42.61%	45.58%	30.85%
Case type	Felony	32.18%	30.76%	39.98%	21.09%	29.80%
Misdemeanor	43.04%	43.67%	43.14%	34.12%	47.55%	40.89%
Criminal Traffic	24.78%	25.57%	16.88%	44.79%	22.66%	22.73%
Table 2: Statistics of Two Recidivism Variables
Missing Variable Rate
Sentence Cut-off	Overall	Caucasian	African American	Hispanic	Native American	Asian
180 Days	0.0807	0.0667	0.1286	0.0653	0.0686	0.0767
2 Years	0.0471	0.0367	0.0824	0.0365	0.0374	0.0513
Recidivism Rate (excluding 'missing')
Sentence Cut-off	Full sample	Caucasian	African American	Hispanic	Native American	Asian
180 Days	0.4221	0.4034	0.4643	0.3876	0.5647	0.3780
2 Years	0.4168	0.3996	0.4534	0.3825	0.5593	0.3724
Figure 1: Change in Data Distribution with Time
For each type of offense in the dataset, there are more specific categorization of offense in the dataset under the wcisclass variable. For misdemeanor, there are 55 distinct wcisclass values (and  $0.33\%$  missing). For felony, there are 60 distinct wcisclass values (and  $0.34\%$  missing), and for criminal traffic, there are 39 distinct wcisclass values (and  $0.49\%$  missing). We list the 10 most common wcisclasses for each offense type in Table 3. We provide summary statistics about highest charge severity in Table 6 in appendix G. Figure 2 in appendix G also compares the
distributions of different offense types in the two recidivism variables, when the recidivism value is missing and when it is not not-missing.
Table 3: 10 Most Frequent wcisclass for Each Type of Offense
Misdemeanor	Felony	Criminal Traffic
wcisclass	Percentage	wcisclass	Percentage	wcisclass	Percentage
Disorderly Conduct	15.51	Drug Possession	15.98	OAR/OAS	42.74
Battery	14.44	Bail Jumping	10.43	Operating While Intoxicated	42.25
Resisting Officer	13.62	Burglary	7.49	Operate Without License	8.37
Drug Possession	8.76	Drug Manufacture/Deliver	6.94	Hit and Run	1.98
Bail Jumping	8.75	Theft	6.28	Unidentified Misdemeanor Traffic	1.16
Theft	6.31	Operating while intoxicated	5.83	Bail Jumping	0.9
Retail Theft(Shoplifting)	5.62	Other Felony	5.03	Other Misdemeanor	0.78
Criminal Damage	5.61	Forgery	3.76	nan	0.49
Drug Paraphernalia	2.96	Battery	3.25	Resisting Officer	0.41
Worthless Checks	2.69	Substantial/Aggravated Battery	3.06	Drug Possession	0.21
4.2 Machine Learning Predictions
The purpose of this section is not to indicate that the following is the "right" way or the only way to use this dataset in research. But for reference only, we report performance metrics of two machine learning classifiers on this dataset. The predictors included in the models were sex, type of offense, prior criminal count (for each type), and age (at judgment and at first offense). Logistic Regression and XGBoost classifiers were from Python's scikit-learn (27) and XGBoost (28) packages respectively. The packages are public and freely available. LR learns a linear model while XGBoost learns a decision tree ensemble. We split the entire data into  $70\%$  train and  $30\%$  test, and the results are reported on the test data. For LR classifier, we include L2 regularization and select the regularization parameter via 10-fold cross-validation. For XGBoost, we include both L1 and L2 regularization and tune the hyperparameters via grid search and 5-fold cross-validation.  $95\%$  confidence intervals are constructed by multiple train and test splits. Details of compute resources can be found at https://scicomp.ethz.ch/wiki/Euler#Introduction.
Table 4 shows the performance of the LR classifier and the XGBoost classifier on this dataset across all five race groups. FPR stands for False Positive Rate, FNR for False Negative Rate, PR for the rate of positive classification, AUC stands for Area under the ROC Curve. We observe from the table that Native American is the most disadvantaged group in the sense that it has the highest FPR and PR, and the lowest FNR. The largest FPR difference is around  $13\%$ , the largest FNR difference is around  $18\%$ , and the largest PR difference is around  $20\%$ , all between Native American and Hispanic with XGBoost classifier. Hispanic and Caucasian groups receive the most favorable decisions (1-PR) in both models, followed by Asian and African American groups. The overall accuracy of XGBoost and LR is not very different but the FNR, FPR and PR differences between groups is higher for XGBoost. These results are for the recidivism variable with 180 days sentence cut-off. In Table 7 in appendix G, we also provide results for the recidivism variable with 2 years sentence cut-off. We did not observe a notable difference in performance metrics for the two recidivism variables but it may be worth investigating the differences at a more granular level.
In addition to the discussion in Section 2 on the limitations of the COMPAS dataset, it is useful to consider the following as well for comparison with our dataset. (29) report about COMPAS/ProPublica
dataset: "The (unconstrained) logistic regression classifier leads to an accuracy of 0.668. However, the classifier yields false positive rates of 0.35 and 0.17, respectively, for blacks and whites (i.e.,  $\mathrm{DFPR} = 0.18$ ), and false negative rates of 0.31 and 0.61 (i.e.,  $\mathrm{DFNR} = 0.30$ . The COMPAS dataset has the following number of samples from different racial groups. African-American: 3175, Asian: 31, Caucasian: 2103, Hispanic: 509, Native American: 11. Due to very small numbers, racial groups other than African-American and Caucasian are generally not analysed in the literature using the COMPAS. Our dataset significantly addresses that limitation.
Table 4: Recidivism Prediction Performance on the Dataset, with  $95\%$  Confidence Intervals
Overall	Caucasian	African American
XGBoost
Accuracy	0.6588 ± 0.0018	0.6648 ± 0.0020	0.6459 ± 0.0034
AUC	0.7039± 0.0018	0.7044± 0.0023	0.7033± 0.0035
FPR	0.2244 ± 0.0041	0.2159 ± 0.0042	0.2454 ± 0.0048
FNR	0.5008 ± 0.0040	0.5113 ± 0.0045	0.4792 ± 0.0067
PR	0.3405 ± 0.0035	0.3261 ± 0.0038	0.3734 ± 0.0041
Logistic Regression
Accuracy	0.6461 ± 0.0013	0.6560 ± 0.0020	0.6206 ± 0.0027
AUC	0.6822 ± 0.0013	0.6825 ± 0.0022	0.6806 ± 0.0030
FPR	0.1532 ± 0.0022	0.1479 ± 0.0026	0.1667 ± 0.0031
FNR	0.6282 ± 0.0029	0.6334 ± 0.0035	0.6244 ± 0.0039
PR	0.2456 ± 0.0021	0.2363 ± 0.0024	0.2638 ± 0.0023
Hispanic	Native American	Asian
XGBoost
Accuracy	0.6567 ± 0.0055	0.6303 ± 0.0076	0.6760 ± 0.0108
AUC	0.6719± 0.0068	0.6878± 0.0090	0.7079 ± 0.0122
FPR	0.2016 ± 0.0073	0.3272 ± 0.0145	0.2203 ± 0.0147
FNR	0.5674 ± 0.0096	0.4025 ± 0.0106	0.4944 ± 0.0225
PR	0.2910 ± 0.0060	0.4800 ± 0.0100	0.3283 ± 0.0128
Logistic Regression
Accuracy	0.6535 ± 0.0048	0.6026 ± 0.0057	0.6735 ± 0.0116
AUC	0.6558 ± 0.0056	0.6746 ± 0.0068	0.6946 ± 0.0136
FPR	0.1292 ± 0.0051	0.2395 ± 0.0081	0.1386 ± 0.0118
FNR	0.6903 ± 0.0085	0.5189 ± 0.0080	0.6350 ± 0.0199
PR	0.1991 ± 0.0049	0.3761 ± 0.0065	0.2243 ± 0.0106
5.1 License
The dataset is distributed under the Creative Commons 4.0 BY-NC-SA (Attribution-NonCommercial-ShareAlike) license. The choice of this license is motivated by the intended uses of the dataset and other considerations that we discuss next.
5.2 Intended Uses and Limitations
The dataset is intended to be used for academic research only. We focus in this paper on algorithmic fairness analysis as a specific use-case, but it doesn't have to be restricted to this. We have provided pseudo-identifiers for judge, county and zipcode and as stated earlier, mappings for county and zipcode can be provided to researchers on request to explore different research use-cases or to explore the interaction of algorithmic or data centric factors with systemic issues such as biases of judges, county or location factors etc. There is more information available in the raw case records that is more difficult to parse and thus, not available at present in the curated dataset. As we discuss in Section 5.4, we will strive to update the dataset in the near future if we find this additional information to be of sufficiently good quality. This will further extend the scope of research uses of the dataset. These
may include, for e.g., studying the process of sentencing itself, or measuring the impact of defendant attributes on the severeness of sentencing or bail.
The dataset addresses several limitations of prior datasets (larger size, large number of samples from different racial groups, different courts, counties, more attributes, less variance, observations over longer time etc). At the same time, there are several fundamental limitations (11) that are difficult or even impossible to address in any dataset. Examples include missing values and biases encoded in the recidivism variable, as discussed in Section 3.5, and in the attributes. Such biases in the data can not be avoided. Depending on the context, they may be handled in other ways, for e.g., through appropriate algorithmic intervention or even critically rethinking the use of algorithms in the first place. We argue that since these limitations are fundamental and can not be addressed easily through data collection or curation processes, it is important that the datasets are available to researchers for critical research. The researchers must be aware of the limitations of the datasets however. For example, the unavailability of prison/jail records implies that the reported recidivism is not guaranteed to be accurate, and we are not able to measure the extent of this inaccuracy. In addition, since we only have cases from Wisconsin Circuit Courts, we can not capture recidivism elsewhere, also, an error that is hard to quantify the impact of. Moreover, the fact that convictions do not faithfully represent criminality is another source of bias and a fundamental issue with the prediction task definition itself. We further bring attention to the fact that the WCLD dataset is built using case records for a long period of time. There are several changes in the society and external variables over such a long time period that can not be captured in a dataset. This temporal factor must be kept in mind while using the dataset for research. However, it is also a strength of the dataset since it can be used to study the algorithms' behavior under distributions shifts. We note that not all distribution shifts are reflective of the real-world societal changes but are nevertheless realities of the data collection and curation processes, and therefore, availability of such datasets for research is important. In case of WCCA, case records from earlier years like late 1900s tend to be less in number (there can be multiple possible reasons for this), and that may have caused underestimation of prior criminal counts for older cases in the WCLD dataset.
As part of the subscription information, WCCA also informed of a few other limitations of the data available through the REST interface. We list those limitations in the datasheet (Section F.3 in appendix F).
5.3 Ethical Considerations
The court record summaries provided by WCCA are public records under Wisconsin open records law sections 19.31-19.39 of the Wisconsin Statutes. Court records not open to public inspection by law are not available. WCCA information does not include information that may be confidential, sealed, or redacted in accordance with all applicable statutes, court orders, and rules related to confidentiality, sealing, and redaction. We refer the reader to the WCCA FAQs (30) for further information. We took additional steps to redact sensitive information as discussed in Section 3.8
Further, readers should not interpret summary statistics of this dataset (for example, those listed in Tables 1 and 2) as 'ground truth' but rather as characteristics of the dataset only.
Researchers using this dataset are expected to carefully take into account the intended uses and limitations listed in Section 5.2 and follow practices for responsible use of criminal justice data in research (11). Beyond limitations of our dataset and data more broadly in this context, fundamental limitations of algorithms (including but not limited to mathematical limitations (3; 4)) pose challenge to addressing the ethical concerns (11; 31; 32) related to recidivism prediction task itself (33). Such limitations and ethical concerns about the task should also be taken into account while drawing conclusions from the research using this dataset.
5.4 Future Work
In the case records that we downloaded using the REST interface, there is additional information available about various case events (e.g. hearings, bail decisions, bail amounts, sentences), information on the associated attorneys and government officials involved, including prosecutors. It may be useful to parse this information and update the dataset. We will strive to do these and update the dataset depending on the quality and usefulness of the additional information available. Readers interested in contributing to the dataset are encouraged to reach out to us.
F Datasheet
For what purpose was the dataset created? The raw case records were created presumably for public inspection and for record keeping in the courts.
The external census dataset used in curation was created for census.
The final curated dataset, contributed in this paper, was created for academic research on algorithmic fairness.
Who created the dataset, and on behalf of which entity? According to Wisconsin Circuit Courts Access website, the raw data records that we accessed through their website are an exact copy of the case information entered into the circuit court case management system by court staff in the counties where the case files are located.
Nianyun Li, Claudia Marangon and Peiyao Sun curated the dataset in its current form with the help of Elliott Ash and Naman Goel.
Who funded the creation of the dataset? The funding of creation of original data records is unclear, but presumably the state funded the court staff in the counties where the case files are located.
The creation of the curated dataset was funded by ETH Zurich.
F.2 Composition
What do the instances that comprise the dataset represent? The instances represent case and defendant information.
How many instances are there in total (of each type)? There are around 1.5 million instances.
Does the dataset contain all possible instances or is it a sample of instances from a larger set? The dataset is a sample of instances from a larger set.
What data does each instance consist of? Each instance contains defendant's new_id, age, sex, type of offense, wiscclass, year of filing, race, age at judgment, age at first offense, 9 neighborhood characteristics (population density, proportion who attended college, proportion eligible for food stamp, African American population share, Hispanic population share, proportion who live in rural and urban area, median household income, highest charge severity, not_detained, probation, recid (180 days sentence length cutoff), recid (2 years sentence length cutoff), violent_recid (180 days sentence length cutoff), jail, county, violent_crime. The details of each of these variables are provided in the accompanying paper and the metadata file in the data directory. Each instance also contains prior criminal count for each type of offense, prior sentence length statistics, prior charge severity counts. The counts were created from the available raw case records by performing database search, and are therefore possibly underestimated.
Is there a label or target associated with each instance? For research, recid (180 days sentence length cutoff) and recid (2 years sentence length cutoff) are two variables associated with each instance. There is also a violent_recid (180 days sentence length cutoff) variable available for research. These are not ground truth labels in a traditional sense but only variables defined by the authors. Details in Section 3.5 and ethics discussion in Section 5.3 of the accompanying paper. The first variable is recidivism as observed in the case records by performing database search, within a 2 year follow-up period since judgment disposition date, using a 180 days cut-off for sentence. The second is obtained by using a 2 year cut-off sentence and extending the follow-up period of 2 years by adding the sentence length.
Is any information missing from individual instances? Yes, the recidivism variables can not be observed for defendants depending on their sentence. Thus, it is missing for some defendants. Details in Section 3.5.
Are relationships between individual instances made explicit? The dataset is anonymized and therefore some relationship between individual instances may be lost. We have included defendant pseudo-identifier in the dataset constructed based on first name, last name and date of birth.
Are there recommended data splits? No. But we have included two possible random splits in the dataset (one is completely random thus only ensuring different cases in train and test splits, the other also ensures different defendants in train and test splits).
Are there any errors, sources of noise, or redundancies in the dataset? The errors are possible in the raw case information entered by the court staff. Known errors in the curated dataset construction are discussed in Section 5.2.
Is the dataset self-contained, or does it rely on external resources? The dataset has been curated using case records from WCCA and census data from 2010. The curated dataset is self-contained.
Does the dataset contain data that might be considered confidential? No
Does the dataset contain data that, if viewed directly, might be offensive, insulting, or threatening? No
Does the dataset relate to people? Yes, the raw case records relate to the defendants. However, directly identifiable information such as names, addresses, date of birth, case numbers etc has been removed in the curated dataset.
Does the dataset identify any subpopulations? Yes, the dataset contains data from five racial groups as marked by WCCA, sex and age groups.
Is it possible to identify individuals? The raw case records available on WCCA are public information and it is possible to identify individuals there. For the curated dataset that we release, we have removed directly identifiable information such as names, addresses, date of birth, case numbers etc.
Does the dataset contain data that might be considered sensitive in any way? The raw case records available on WCCA are public information and some of the information such as defendant's personal information may be considered sensitive. For the curated dataset that we release, we have removed directly identifiable information such as names, addresses, date of birth, case numbers etc.
F.3 Collection Process
How was the data associated with each instance acquired? Through the REST interface of WCCA.
What mechanisms or procedures were used to collect the data? Through the REST interface of WCCA. We queried all case numbers in each county during the period and subsequently, using these case numbers, we queried individual cases for all available information. Different attributes were then derived using the process described in the accompanying paper.
If the data are a sample from a larger set, what was the sampling strategy? The dataset is composed primarily of new cases filed between 2000-2018. The dataset excludes dismissed cases that do not result in conviction, records of defendants that do not have sex and/or race data and cases that only have forfeiture (non-crime) charge. https://wcca.wicourts.gov/faq.html provides more information on records that might have been deleted. WCCA also informed us of a few limitations of the data as part of the subscription. These are listed as follows:
1. WCCA Information includes only court records open to public view under Wisconsin's Open Records Law, Wis. Stat. 19.31-19.39. Court records not open to public inspection by law are not available.
2. WCCA Information does not include information that may be confidential, sealed, or redacted in accordance with all applicable statutes, court orders, and rules related to confidentiality, sealing, and redaction.
3. WCCA Information consists of information entered into the CCAP  $^{3}$  case management system by the Clerk of Circuit Court or Register in Probate in each county. CCAP is not responsible for the accuracy or timeliness of WCCA information.
4. WCCA Information does not comprise the complete court record. Copies of documents must be obtained from the Clerk of Circuit Court or Register in Probate.
5. WCCA Information is only a snapshot of the information accessible in the CCAP case management system on the date the information is downloaded by the Subscriber.
$^{3}$ WCCA was formerly CCAP.
6. WCCA Information is not the Judgment and Lien Docket under Wis. Stat. 806.10. The Judgment and Lien Docket is available from the Clerk of Circuit Court.
7. Court records which predate the implementation of the CCAP case management system in the county in which the records were created are not accessible under this Agreement, except to the extent such records have been back loaded.
8. In criminal cases, any designation in any race field contains subjective information generally provided by the agency that filed the case.
9. Searching WCCA Information by a particular field or code may not return all cases in which a particular event occurred unless at the time the record was created the case management system required the field or code to be completed in order to proceed to make the rest of the record.
Who was involved in the data collection process and how were they compensated? The case records were created by court staff in respective county courts and were presumably, compensated by state.
The authors of this paper collected the case records from WCCA and were employees of ETH Zurich during the data curation process. They were compensated by ETH Zurich in the form of fixed monthly salaries. Nianyun Li, Naman Goel, Peiyao Sun, Claudia Marangon were/are on fixed-term contracts with ETH Zurich.
Over what time frame was the data collected? Authors had access to the raw case records through WCCA REST interface during the period July 2020 - July 2021. However, data collection was finished by Feb 2021.
Were any ethical review processes conducted? The authors are not aware of the ethical review process followed in WCCA or county courts for creation of the case records. As part of the curation of the dataset that we contribute, no formal/institutional ethical review process was conducted.