A Times Series Analysis of Crime in Pittsburgh Pennsylvania
By: Matthew Mccarroll
Tags: Python, Pandas, Data Analysis, Time Series Analysis

Figure 1. Image of Downtown Pittsburgh. The Monongahela (right) and Allegheny (left ) rivers come together to form the Ohio River at what is known as “The Point”. Image provided by Bobak Ha’Eri found on Wikipedia.

This notebook provides analysis of the police arrests and the crimes that have occurred in the city of Pittsburgh Pennsylvania. The data being used comes from the Western Pennsylvania Regional Data Center and can be obtained by following this link. In this notebook I address how to read in, clean, and manipulate a database, create visualizations to help understand and convey important features of the data, and carry out statistical modeling/analysis to discover trends that the data holds. A detailed Jupyter notebook of this code and its outputs including plots and dataframe printouts can be found here. Lets begin!

import packages

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

%matplotlib inline

import seaborn as sns



# see more columns and rows

pd.set_option('display.max_columns', 500)

pd.set_option('display.max_rows', 500)

pd.set_option('display.width', 1000)



from IPython.core.display import display, HTML

display(HTML('<style>.container { width:100% !important; }</style>'))

read in and assess data of missing values

# read in data

arrests=pd.read_csv('pittsburgh_arrests_123119.csv')



# look at column types, missing values, identification numbers...

arrests.info()

print(arrests.isna().sum())

# Pk and CCR are identification values of individuals

print('

number of unique PK: '+str(arrests['PK'].unique().shape))

print('number of unique CCR: '+str(arrests['CCR'].unique().shape))



arrests.head()

Output:

<class 'pandas.core.frame.DataFrame'>

RangeIndex: 38158 entries, 0 to 38157

Data columns (total 16 columns):

PK 38158 non-null int64

CCR 38158 non-null int64

AGE 37859 non-null float64

GENDER 38158 non-null object

RACE 38158 non-null object

ARRESTTIME 38158 non-null object

ARRESTLOCATION 38158 non-null object

OFFENSES 38155 non-null object

INCIDENTLOCATION 38158 non-null object

INCIDENTNEIGHBORHOOD 36727 non-null object

INCIDENTZONE 38100 non-null object

INCIDENTTRACT 36749 non-null float64

COUNCIL_DISTRICT 33250 non-null float64

PUBLIC_WORKS_DIVISION 33250 non-null float64

X 36810 non-null float64

Y 36810 non-null float64

dtypes: float64(6), int64(2), object(8)

memory usage: 4.7+ MB

PK 0

CCR 0

AGE 299

GENDER 0

RACE 0

ARRESTTIME 0

ARRESTLOCATION 0

OFFENSES 3

INCIDENTLOCATION 0

INCIDENTNEIGHBORHOOD 1431

INCIDENTZONE 58

INCIDENTTRACT 1409

COUNCIL_DISTRICT 4908

PUBLIC_WORKS_DIVISION 4908

X 1348

Y 1348

dtype: int64



number of unique PK: (38158,)

number of unique CCR: (31308,)

Data Cleaning

# remove nans in offenses

arrests = arrests[pd.notnull(arrests['OFFENSES'])]

# drop CCR value repeats

arrests=arrests.drop_duplicates(subset='CCR',keep='first')

# convert arrest times to datetime values and sort

arrests['ARRESTTIME'] = pd.to_datetime(arrests['ARRESTTIME'], yearfirst=True)

arrests= arrests.sort_values(by= ['ARRESTTIME']).reset_index(drop=True) # look at arrest totals

print('arrest totals per year')

for yr in np.arange(1998,2021,1):

print(str(yr)+': '+str(len(arrests.loc[arrests['ARRESTTIME'].dt.year==yr])))

Output:

arrest totals per year

1998: 1

1999: 0

2000: 0

2001: 0

2002: 0

2003: 0

2004: 0

2005: 1

2006: 1

2007: 3

2008: 1

2009: 0

2010: 0

2011: 2

2012: 1

2013: 2

2014: 6

2015: 10

2016: 3564

2017: 9927

2018: 8890

2019: 8733

2020: 166

In this instance I want data from September 2016 up until and including December 2019. September was when this data began to be tabulated according to https://data.wprdc.org/dataset/arrest-data.

#select data pertaining to desired time frame

arrests=arrests.loc[arrests['ARRESTTIME']>= '2016-09'].reset_index(drop=True)

arrests=arrests.loc[arrests['ARRESTTIME']< '2020-01'].reset_index(drop=True)

There are a lot of missing x,y coordinates. This is because sex crimes do not receive coordinates as to where it happened, rather it is assigned the police zone location. A mapping dictionary was created to assign x,y coordinates to police zone values to fill the missing data regarding sex crimes.

# I looked up police zone coordinates

# 1

# Latitude: 40.4569565

# Longitude: -80.0160637

# 2

# Latitude: 40.4435808

# Longitude: -79.9800111

# 3

# Latitude: 40.421685

# Longitude: -79.9924763

# 4

# Latitude: 40.4414787

# Longitude: -79.9212047

# 5

# Latitude: 40.4742729

# Longitude: -79.9091618

# 6

# Latitude: 40.4420734

# Longitude: -80.0322484 X_coord = {'Zone 1': -80.0160637 , 'Zone 2': -79.9800111, 'Zone 3': -79.9924763, 'Zone 4':-79.9212047 ,

'Zone 5':-79.9091618 , 'Zone 6':-80.0322484 } Y_coord = {'Zone 1': 40.4569565, 'Zone 2': 40.4435808 , 'Zone 3': 40.421685, 'Zone 4': 40.4414787 ,

'Zone 5': 40.4742729, 'Zone 6':40.4420734 }



# map zone coordinates to all incident locations labeled with a zone description.

# if not 'zone', puts NaN

arrests['ZONEX'] = arrests['INCIDENTLOCATION'].map(X_coord)

arrests['ZONEY'] = arrests['INCIDENTLOCATION'].map(Y_coord) # fills in x,y coordinates of zone incident locations

arrests['X']=arrests['X'].fillna(arrests['ZONEX'], axis=0)

arrests['Y']=arrests['Y'].fillna(arrests['ZONEY'], axis=0)

# remove ZONEX and ZONEY columns

arrests = arrests.drop(['ZONEX'], axis=1)

arrests = arrests.drop(['ZONEY'], axis=1) print('missing x coordinates: '+str(len(arrests.loc[arrests['X'].isna()])))

Output:

missing x coordinates: 8

The missing coordinates belong to out of city locations.

# remove incidents that occur outside of Pittsburgh

print(arrests.loc[arrests['X'].isna()]['INCIDENTLOCATION'].unique())

# remove mislabeled and/or data entry error.

arrests=arrests[arrests['INCIDENTLOCATION'] !='Zone OSC']

arrests=arrests[arrests['INCIDENTLOCATION'] !='Zone ']

print('missing x coordinates: '+str(len(arrests.loc[arrests['X'].isna()])))

The column labeled offenses contains all the charges of the individual upon arrest separated by a /

arrests['OFFENSES'].head()

Output:

0 2706 Terroristic Threats. / 5104 Resisting Arr...

1 2702 Aggravated Assault. / 2706 Terroristic Th...

2 9501 Bench Warrant

3 13(a)(16) Possession of Controlled Substance /...

4 3934 Theft from Vehicle

Name: OFFENSES, dtype: object

I count the total number of charges per arrest per individual and also create a dataframe that records each charge per each individual.

# find total number of offenses per individual

arrests['NUMOFFENSES'] = arrests['OFFENSES'].apply(lambda x: len(x.split(' / ')))# records the indivdiual charges pertaining to each felon

df=[]

for index, row in arrests.iterrows():

# each offense is separated by a ''/''. Split on this.

charges = row['OFFENSES'].split(' / ')

for charge in charges:

row['CHARGES']=charge

df.append(row)

row=row.drop('CHARGES',axis=0)



arrests_cleaned=pd.DataFrame(df).reset_index(drop=True)

# isinstance(arrests_cleaned, pd.DataFrame) # check if dataframe was made

arrests_cleaned.head()

Nice! Now the new dataframe can be cleaned up. I assign the dates of arrests as the index. This will greatly help in analysis moving forward using Python’s Pandas package.

# remove period

arrests_cleaned['CHARGES'] = arrests_cleaned['CHARGES'].str.replace('.','',regex=True) # remove period from string

# make column for arrest codes

arrests_cleaned['ARRESTCODE'] = arrests_cleaned['CHARGES'].apply(lambda x: x.split(" ",1)[0])

# make column for arrest descriptions

arrests_cleaned['ARRESTDESCRIPT'] = arrests_cleaned['CHARGES'].apply(lambda x: x.split(" ",1)[1]) # remove unwanted columns

arrests_cleaned = arrests_cleaned.drop(['CHARGES','OFFENSES'],axis=1)

# arrests_cleaned = arrests_cleaned.drop(['OFFENSES'],axis=1)

# make index relfective of arrest times

arrests_cleaned.index = arrests_cleaned['ARRESTTIME']

del arrests_cleaned.index.name

# arrests_cleaned.to_csv('arrests_cleaned.csv', index=False)

arrests_cleaned.head()

Data Analysis

Now I will begin to look at what charges are being assigned to arrests.

# count the number of individual charges and how they compare to all the others.

charge_info=arrests_cleaned['ARRESTDESCRIPT'].value_counts()

charge_info=pd.DataFrame(charge_info).reset_index()

charge_info=charge_info.rename(columns={'index': 'crime', 'ARRESTDESCRIPT': 'count'})

charge_info['cumsum']=charge_info['count'].cumsum()

charge_info['overall_pct']=(charge_info['count']/charge_info['count'].sum())*100

charge_info['cumsum_pct']=charge_info['overall_pct'].cumsum()

charge_info.head()

A bar graph visualization will help better see these values. Let’s evaluate the top 20 charges. Code to make this and all other graphs can be found on my github page here.

The percentage value associated with each crime represents the percentage of that crime in relation to all total crimes committed over the time span. It seems that assault and drug related charges dominate arrests. A good question to investigate is whether an arrest has multiple charges or a single charge associated with them. I will break down the information into monthly values.

The plot of marijuana possession under 30 grams is interesting because there is an underlying occurrence of arrests with only that charge. Let’s look more closely at this.

The graphs show that close to 700 (678 approximately) people were arrested solely for possession of marijuana of less than 30 grams since September 2016 to December 2019. That’s approximately 17 arrest per month. It seems that arrests with single charge drops after June 2018. Let’s explore this.

import warnings

warnings.filterwarnings('ignore')

upto_june2018=individ_month_arrests.loc['2017-01':'2018-06']

upto_june2018['ID']='up to june 2018'

after_june2018=individ_month_arrests.loc['2018-07':]

after_june2018['ID'] = 'after june 2018'

labeled=pd.concat([upto_june2018,after_june2018])

labeled.head()

Yup it looks like there is a difference in means. Let’s approach this from a statistics viewpoint. The null hypothesis of that there is no difference in single charge monthly arrest totals between Jan 2017-June 2018 and after June 2018.

First, test for normality using qq-plots.