Hierarchical Approaches to Text-based Offense Classification

Jay Choi1 David Kilmer2 Michael Mueller-Smith1 Sema Taheri2

1University of Michigan 2Measures for Justice

Fall 2022

1 / 18
Motivation

Offense classification plays critical roles for society, research, and public administration:

1. Criminal background checks (employment, public benefit eligibility, security credentialing,

firearm purchases)

2. Evaluating sentencing disparities, testing behavioral theories, etc

3. Public funding, resource allocation, election outcomes

Yet, no comprehensive standard currently exists to cover the universe of illicit activity nor a

mechanism to map free-entry descriptions into systematic codes, leading to:

– More discretion with little oversight

– Inconsistent definitions and mappings

– Irreproducible research

2 / 18
Project summary

In this paper, we do the following:

1. Introduce the Uniform Crime Classification Standard (UCCS)

– Schema satisfying 4 design principles laid out by the National Academy of Sciences

2. Develop the Text-based Offense Classification (TOC) tool

– Hierarchical machine learning model trained on over 300k hand coded training records

3. Experiment with model variation in TOC to assess determinant of prediction success

– Help identify potential lessons for other classification applications

3 / 18
Four design principles for offense classification

National Academy of Sciences (2016) laid out four goals of offense schema:

1. Encompass new and emerging crime types

2. Fully realized classification for statistical purposes

3. Attribute-based classification

4. Enable comparisons between jurisdictions across time

Summary Reporting System (SRS)

National IncidentBased Reporting

System (NIBRS)

National Corrections Reporting

Program (NCRP)

Uniform Crime

Classification

Standard (UCCS)

Principle 1: x X x X

Principle 2: x x x X

Principle 3: X X X X

Principle 4: x x x X

4 / 18
Uniform Crime Classification Standard

Each UCCS code has four digits:

– First digit → Broad code

– Digits 2 and 3 → Offense code

– Fourth digit → Offense modifier

Example UCCS codes: Table S9: Uniform Crime Classification Standard Schema

UCCS Code/Description Broad Code/Description Offense Code/Description Offense Modifier Code/Description

1010 Murder 1 Violent 01 Murder 0

1011 Attempted Murder 1 Violent 01 Murder 1 Attempt

1012 Conspiracy to Commit Murder 1 Violent 01 Murder 2 Conspiracy

1020 Unspecified Homicide 1 Violent 02 Unspecified homicide 0

1021 Unspecified Homicide, Attempted 1 Violent 02 Unspecified homicide 1 Attempt

1022 Unspecified Homicide, Conspiracy 1 Violent 02 Unspecified homicide 2 Conspiracy

1030 Voluntary Manslaughter 1 Violent 03 Voluntary/nonnegligent manslaughter 0

1031 Voluntary Manslaughter, Attempted 1 Violent 03 Voluntary/nonnegligent manslaughter 1 Attempt

1032 Voluntary Manslaughter,Conspiracy 1 Violent 03 Voluntary/nonnegligent manslaughter 2 Conspiracy

1040 Vehicular Manslaughter 1 Violent 04 Voluntary/nonnegligent manslaughter 0

1041 Vehicular Manslaughter, Attempted 1 Violent 04 Voluntary/nonnegligent manslaughter 1 Attempt

1042 Vehicular Manslaughter, Conspiracy 1 Violent 04 Voluntary/nonnegligent manslaughter 2 Conspiracy

1050 Involuntary Manslaughter 1 Violent 05 Manslaughter - non-vehicular 0

1051 Involuntary Manslaughter, Attempt 1 Violent 05 Manslaughter - non-vehicular 1 Attempt

1052 Involuntary Manslaughter, Conspiracy 1 Violent 05 Manslaughter - non-vehicular 2 Conspiracy

1060 Kidnapping 1 Violent 06 Kidnapping 0

1061 Kidnapping, Attempted 1 Violent 06 Kidnapping 1 Attempt

1062 KidiCi1 Vilt 06 Kidi2 Ci

5 / 18
Text-based Offense Classification tool

TOC tool maps free-entry offense description fields to UCCS codes

Built through a partnership between Measures for Justice (MFJ) and CJARS

– MFJ hand coded significant volume of offense descriptions, creating training data to

develop TOC

– 313,209 hand-coded offense descriptions from 24 states

Freely available to general public online: https://cjars-toc.isr.umich.edu

6 / 18
Text-based Offense Classification tool

7 / 18
Prediction models

8 / 18
Prediction models: flat

8 / 18
Prediction models: flat

8 / 18
Prediction models: hierarchical

8 / 18
Prediction models: hierarchical

8 / 18
Model details

– 75% - 25% random split of hand-coded data for training vs testing purposes

– Avoids known problems arising from data leakage

– Modest preprocessing: remove articles, punctuation, capitalization, word normalization

– Tokenization: 4-grams

– Feature selection: Term Frequency-Inverse Document Frequency

– Optimization algorithm: neural network

– Multi-layer perceptron model with 1 hidden layer and 100 neurons

9 / 18
Feature extraction via n-grams

Figure 3: Feature extraction

(a) Contiguous sequence of 4 characters

theft from person > 65 value >300k<10k

theft from person > 65 value >300k<10k

theft from person > 65 value >300k<10k

...

theft from person > 65 value >300k<10k

theft from person > 65 value >300k<10k

theft from person > 65 value >300k<10k

(b) Extracted features

’thef’, ’heft’, ’eft ’, ’ft f’, ’t fr’, ’ fro’,

’from’, ’rom ’, ’om p’, ’m pe’, ’

per’, ’pers’, ’erso’, ’rson’, ’son>’,

’on>6’, ’n>65’, ’>65 ’, ’65 v’, ’5

va’, ’ val’, ’valu’, ’alue’, ’lue>’,

’ue>3’, ’e>30’, ’>300’, ’300<’,

’00<1’, ’0<10’, ’<10k’, ....

Notes: This figure shows a visual representation of tokenization using 4-grams for the offense description “theft from

person > 65 value >200k<10k.” Using this method, contiguous sequence of 4 characters are selected from the beginning

to the end of the text as highlighted by yellow box in (Figure 3a). Each of the extracted tokens, or features, in (Figure 3b)

are then scored using a vectorizer. For Count Vectorizer, each feature is assigned an integer to represent the number

of times it appears in the entire data. In the case of TF-IDF, each feature is assigned a score from 0 to 1. The values

10 / 18
Defining success

We evaluate three related standard performance statistics:

– Precision: % of positive predictions that are correctly labelled

– Recall: % of labels that are correctly made

– F1 Score: harmonic mean of precision and recall

Model performance is assessed at both the broad crime type and full UCCS code levels

11 / 18
Performance Statistics

Broad Crime Type Full UCCS Code

Precision Recall F1 Score Precision Recall F1 Score

All Crime Types 0.983 0.983 0.983 0.963 0.963 0.963

Broad Crime Type Code:

Violent 0.997 0.994 0.995 0.993 0.989 0.991

Property 0.927 0.990 0.957 0.884 0.944 0.913

Drug 0.999 0.960 0.979 0.862 0.828 0.845

DUI 0.987 0.986 0.986 0.942 0.941 0.941

Public Order 0.993 0.938 0.965 0.977 0.923 0.949

Criminal Traffic 0.987 0.991 0.989 0.986 0.991 0.988

12 / 18
Out-of-state performance

Broad crime type Full crime type

13 / 18
Comparison with existing resources

Distribution of offense types for felony charges in CJARS 0 20 40 60 80 100

Violent Property Drug DUI Public Order Traffic Unknown

UCCS NCRP NCIC

14 / 18
Comparison with existing resources

Distribution of offense types for misdemeanor charges in CJARS 0 20 40 60 80 100

Violent Property Drug DUI Public Order Traffic Unknown

UCCS NCRP NCIC

14 / 18
Evaluating optimal model parameterization

– Variation in size of training data: 5,000 – 200,000 training observations link

– Feature unit (n-grams, bag-of-words) and selection mechanism (Count Vectorizer vs.

TF-IDF) link

– Total number of features: 100 – 10,000 selected features link

– Machine learning algorithm: Random Forest vs. Neural Network link

15 / 18
Next steps:

1. Expand geographic coverage of training data to non-covered jurisdictions

2. Crowdsource prediction errors to improve corpus of training records via TOC portal

3. Incorporate feasibility to process cited statute numbers

– Requires comprehensive database mapping statutes to offense descriptions

4. Develop mechanism to evaluate updates to schema

16 / 18
Summary and conclusion

This project introduces the UCCS schema and the TOC tool to map free entry offense

descriptions to standardized codes

Goal is to raise the bar for research and statistical reporting:

– Reduce researcher discretion as well as data wrangling burden

– Improve common definitions and comparability across jurisdictions

– Expand coverage of historically understudied (but large!) portion of the justice system:

misdemeanor caseload

– Lower research barriers to increase diverse perspectives on justice system

17 / 18
Join our growing user base!

18 / 18
Appendix Slides

1 / 5
Variation in size of training data

(a) Broad Crime Type .7 .75 .8 .85 .9 .95 1 F1 Score

0 50,000 100,000 150,000 200,000

Number of observations

Weighted Unweighted

(b) Full UCCS Code .7 .75 .8 .85 .9 .95 1 F1 Score

0 50,000 100,000 150,000 200,000

Number of observations

Weighted Unweighted

back

2 / 5
Feature unit selection mechanism

(a) Broad Crime Type .55 .6 .65 .7 .75 .8 .85 .9 .95 1 F1 Score

1-gram 2-gram 3-gram 4-gram 5-gram 6-gram word CV TF-IDF

(b) Full UCCS Code .55 .6 .65 .7 .75 .8 .85 .9 .95 1 F1 Score

1-gram 2-gram 3-gram 4-gram 5-gram 6-gram word CV TF-IDF

back

3 / 5
Total number of features

(a) Broad Crime Type .5 .75 1 F1 Score

0 2,000 4,000 6,000 8,000 10,000

Number of features

Weighted Unweighted

(b) Full UCCS Code .5 .75 1 F1 Score

0 2,000 4,000 6,000 8,000 10,000

Number of features

Weighted Unweighted

back

4 / 5
Machine learning algorithm

(a) Broad Crime Type .9 .925 .95 .975 1 F1 Score

Flat Hierarchical Random Forest Multi-layer Perceptron

(b) Full UCCS Code .9 .925 .95 .975 1 F1 Score

Flat Hierarchical Random Forest Multi-layer Perceptron

back

5 / 5