AI and Criminal Justice — Introduction (Part 1/3)
By: Daniel Thinfen
Tags: Artificial Intelligence, AI, Criminal Justice

In this series we will be exploring the intersections of AI and ethics. We’ll be using the COMPAS dataset, a dataset of arrest records with demographic, prior conviction, and other information with the goal of predicting whether criminals will reoffend.

COMPAS, which stands for Correctional Offender Management Profiling for Alternative Sanctions, is an algorithmic risk-assessment tool that attempts to predict whether a defendant is likely to reoffend (recidivate). This tool is specifically intended to be used in the pre-trial detention phase, in which bail determinations are made. Ideally, based on particular characteristics about a defendant, the model would predict whether or not someone is likely to commit another crime if released. This software has been used in many U.S. jurisdictions, including New York, Wisconsin, California, and Broward County, FL.

Racial tensions and specifically anti-Blackness has existed in the U.S. for centuries. In the modern day, racial bias in the U.S. criminal justice system is a particularly difficult sociological problem. The idea was that a machine learning model would be able to be “less biased” than a human judge, and therefore provide better outcomes for all people. However, the adoption of COMPAS has not been without controversy, and has been challenged on legal and ethical grounds perhaps most prominently in the Wisconsin State Supreme Court case Loomis v. Wisconsin (2012). In the models built we will try to reduce bias as much as possible while still getting accurate results.

This is a snapshot of what data from the COMPAS database includes

The data includes:

sex: Male or female.

age: Age.

age_category: Age group; i.e. under 25, 25 to 45, or over 45.

race: Categorical variable; takes on values African-American, Caucasian, Asian, Native American, Hispanic, and Other.

juvenile_felony_count: Number of times defendant has previously been convicted of a juveline felony.

juvenile_misdemeanor_count: Number of times defendant has previously been convicted of a juveline misdemeanor.

juveline_other_count: Number of times defendant has been convicted of another juvenile charge.

prior_convictions: Number of prior convictions for a defendant.

current_charge: Felony, misdemeanor, or other (F, M, or O).

charge_description: The charge for which they were arrested, and took a risk assessment.

recidivated_last_two_years: Recidivated within two years of completing the assessment. This variable is the focus of our task, and what we’ll be predicting.

I made 2 simple models to get started, a Logistic Regression Model and a Random Forest Classifier Model.

The first way I chose to evaluate the models fairness was to compare the results from different races. I first split the data into different racial groups and evaluated the differences between how the models treated African-Americans vs. Caucasians using confusion matrixes.

Splitting the Data

Caucasian Confusion Matrix

African-American Confusion Matrix

The largest issue in the Confusion Matrixes above is that the false negative rate for Caucasians is extremely high around 60%. This means that Caucasian defendants that would have reoffended were incorrectly released. The false positive rate was around 15% for Caucasians and the false positive and negative rate for African-Americans was around 30% for each. Just looking at this there is obviously a level of racial bias that has already developed in the Logistic Regression and Random Forest models, which are extremely simple models. The first way I attempted to remove bias was to simply take the race label out of the data so race wouldn’t affect how the model made its predictions.

The new results are:

Caucasian Confusion Matrix

African-American Confusion Matrix

Even after removing racial bias it seems that the results have gotten even worse! The false positive rate for African-Americans is now 3 times more than the Caucasian group. Clearly there is no easy solution to reducing bias with AI in criminal justice.

Stay tuned for further posts about this project!