Crime Location Analysis and Prediction Using Python and Machine Learning | by Will Keefe | Towards Data Science

The author of this article is a proud West Virginia University Mountaineer and spent four years in Morgantown, West Virginia. Morgantown is a beautiful city in the Mountain State, and the campus of WVU is very safe. That being said, like in all cities safety incidents do occur, and the University Police Department reports these events daily in a crime log. With analysis via coding packages in Python, these events can be better visualized geographically to assess trends in location, type of incident, the likelihood of incidents based on time of day, and trends during unordinary events such as a football weekend, where the population of the town temporarily doubles.
At the end of 2019, the author analyzed the safety event log in detail for the prior fall semester and created a dataset in JSON schema for the preceding four months as follows.
{"case": casenumber (Integer),"crime": text (String),"occurred": date and time (String),"comments": text (String),"building": place (String),"address": place (String),"disposition": disposition (String),},
A total of approximately 1500 unique events occurred during this time, and the events are saved within a JSON file.
To begin our project we will first need to import the packages below.
# Basic Analysis and Visualizationimport pandas as pdimport numpy as npimport matplotlib as mplimport matplotlib.pyplot as pltimport mathfrom datetime import timedelta# Mappingimport geopandasimport geopyfrom geopy.geocoders import Nominatimimport foliumfrom geopy.extra.rate_limiter import RateLimiterfrom folium import pluginsfrom folium.plugins import MarkerCluster# Statistical OLS Regression Analysis%matplotlib inlineimport statsmodels.api as smfrom statsmodels.compat import lzipfrom statsmodels.formula.api import ols#Scipy sklearn Predictionsfrom sklearn.ensemble import GradientBoostingRegressor
Next, we import all of the events to our program from the file.
# Pull in JSON and set index as case number (unique key)df = pd.read_json("MC.json",orient="records")df = df.set_index("case")df
We should also convert our occurrence column to fit new date, time, and day of week columns with date-time schema. We should also remove any unfounded events from our dataset as well.
# Convert time objectsdf['occurred'] = pd.to_datetime(df['occurred'])df['date'] = [d.date() for d in df['occurred']]df['time'] = [d.time() for d in df['occurred']]df['day'] = df['occurred'].dt.day_name()# Find Fractions of Daydf['timeint'] = (df['occurred']-df['occurred'].dt.normalize()).dt.total_seconds()/timedelta(days=1).total_seconds()# Remove unfounded eventsdf = df[df['disposition'] != 'Unfounded']
Let’s now get geographical data for our dataset as well. Abdishakur has a great article here we can follow along with to collect this data, implemented below.
locator = Nominatim(user_agent='myGeocoder')location = locator.geocode('Morgantown, West Virginia')# 1 - conveneint function to delay between geocoding callsgeocode = RateLimiter(locator.geocode, min_delay_seconds=0.5)# 2- - create location columndf['location'] = df['address'].apply(geocode)# 3 - create longitude, latitude and altitude from location column (returns tuple)df['point'] = df['location'].apply(lambda loc: tuple(loc.point) if loc else None)# 4 - split point column into latitude, longitude and altitude columns and remove empty pointsdf = df[df['point'].notnull()]df[['latitude', 'longitude', 'altitude']] = pd.DataFrame(df['point'].tolist(), index=df.index)df.head()
Assessing our now-complete dataset at a glance, we can start to notice some trends.
df.groupby("crime")["crime"].count().sort_values()
Most of the recorded incidences here are the result of poor decision-making under the influence of alcohol, destruction of university property in dorm rooms, or parking in illegal parking areas such as reserved lots.
VEHICLE ACCIDENT           28ALARM CONDITION            30LARCENY                    32FIRE ALARM                 43DESTRUCTION OF PROPERTY    55DRUG INCIDENT              69TALK WITH OFFICER          74ABCC VIOLATION             84ASSIST EMS POLICE          86BACK TICKET TOW            99
The most frequently visited locations for these incidences also reflect the challenging transition to maturity and adulthood, with most occurring within the dormitories.
df.groupby("building")["building"].count().sort_values()BLUE LOT                                     21WVU MOUNTAINLAIR                             23WVU BOREMAN SOUTH                            23COLLEGE PARK                                 23WVU SENECA HALL                              25WVU POLICE DEPARTMENT                        28WVU SUMMIT HALL                              28WVU BROOKE TOWER                             29HEALTH SCIENCE CENTER                        55WVU DADISMAN HALL                            56
Finally, we can also note that some of the days with the most safety events reported were days with unordinary events, such as football game days.
df.groupby("date")["date"].count().sort_values()2019-11-01    21 # WVU vs Baylor2019-10-05    27 # WVU vs Texas2019-11-09    31 # WVU vs Texas Tech
Having analyzed the data frames, let’s now visualize the data on maps using Folium. The below code plots points on Open Street Map with indicators for the type of event that occurred.
# Mark events with names on mapm = folium.Map([39.645,-79.96], zoom_start=14)for index, row in df.iterrows():    folium.CircleMarker([row['latitude'], row['longitude']],                        radius=3,                        popup=row['crime'],                        fill_color="#3db7e4", # divvy color                       ).add_to(m)m
While these indicators are useful and we can zoom in on the map to see them in detail, an additional heat map would improve the visibility of high-frequency areas where multiple indicators may overlap.
# convert to (n, 2) nd-array format for heatmapdfmatrix = df[['latitude', 'longitude']].values# plot heatmapm.add_child(plugins.HeatMap(dfmatrix, radius=15))m
From the rendered heatmap of safety incidences we can see areas of high-frequency on the Evansdale campus near the densely-populated Towers dormitories and football tailgating lots, and on the Downtown campus near Frat Row and High Street where a lot of nightlife occurs, not surprising for a college town.
Earlier we had noted that the days with the most safety incidences happened to be football game days. Our data can help assess optimal locations to stage emergency services for future gamedays. The most active game was that against Texas Tech, on November 9, 2019. We can filter our dataset for just day with the code below.
incidents_on_nov09 = df[df['date'] == pd.datetime(2019,11,9).date()]
Let’s now visualize these filtered events reusing a script from earlier.
# Map points of eventsm2 = folium.Map([39.645,-79.96], zoom_start=14)for index, row in incidents_on_nov09.iterrows():    folium.CircleMarker([row['latitude'], row['longitude']],                        radius=5,                        popup=row['crime'],                        fill_color="#3db7e4", # divvy color                       ).add_to(m2)# convert to (n, 2) nd-array format for heatmapdfmatrix = incidents_on_nov09[['latitude', 'longitude']].values# plot heatmapm2.add_child(plugins.HeatMap(dfmatrix, radius=15))m2
We can see that the events happened all over town. Using the latitude and longitude of these events, we can find a centroid for optimal resource allocation. If we have one ambulance available, it would be most efficient to place it in the middle of every point weighted by frequency on standby throughout the day for quickest average response time. We can find the centroid with this code.
lat = []long = []for index, row in incidents_on_nov09.iterrows():  lat.append(row["latitude"])  long.append(row["longitude"])lat1=sum(lat)/len(lat)lat2=sum(long)/len(long)folium.CircleMarker([lat1,lat2],                        radius=5,                        popup="CENTER LOCATION",                        color='black',                        fill_color="#3db7e4", # divvy color                       ).add_to(m2)m2
The black circle in the map above shows the centroid the incidences on November 9th. If we were to stage emergency services on a future gameday, it would be worthwhile to stage them between Univeristy Avenue and Jones Avenue for fastest average response.
What if we want to change our time frame? What would the centroid be for the evening on the day of a game after the game?
after9PM = incidents_on_nov09[incidents_on_nov09['time'] >= pd.datetime(2019,11,9,21,0,0).time()]lat = []long = []for index, row in after9PM.iterrows():  lat.append(row["latitude"])  long.append(row["longitude"])lat1=sum(lat)/len(lat)lat2=sum(long)/len(long)folium.CircleMarker([lat1,lat2],                        radius=5,                        popup="LATE HOURS LOCATION",                        color='red',                        fill_color="#3db7e4", # divvy color                       ).add_to(m2)m2
Notice now the optimal staging location shifted further downtown, towards 1st Street and the Life Sciences Building.
Can we use time series data and machine learning to predict locations of next incidences? To explore this further we need to configure two prediction models — one for latitude and one for longitude. We will be using the “timeint” column, or time interval or fraction of day, and scikit for predicting using gradient boosting regression. The following code is inspired by this example, and it is developing a prediction model from our data. This tutorial is not meant to perfectly explain the math behind our predictions, scikitlearn does a decent job of that, but what is important is that we are using the data set to create a function with a 90% confidence interval derived from the mean squared error.
#----------------------------------------------------------------------#  First the noiseless caseX = np.atleast_2d(df['timeint'].values).T# Observationsy = np.atleast_2d(df['latitude'].values).T# Mesh the input space for evaluations of the real function, the prediction and# its MSExx = np.atleast_2d(np.linspace(0, 1, 913)).Txx = xx.astype(np.float32)alpha = 0.95clf = GradientBoostingRegressor(loss='quantile', alpha=alpha,                                n_estimators=250, max_depth=3,                                learning_rate=.1, min_samples_leaf=9,                                min_samples_split=9)clf.fit(X, y)# Make the prediction on the meshed x-axisy_upper = clf.predict(xx)clf.set_params(alpha=1.0 - alpha)clf.fit(X, y)# Make the prediction on the meshed x-axisy_lower = clf.predict(xx)clf.set_params(loss='ls')clf.fit(X, y)# Make the prediction on the meshed x-axisy_pred = clf.predict(xx)# Plot the function, the prediction and the 90% confidence interval based on# the MSEfig = plt.figure()plt.figure(figsize=(20,10))plt.plot(X, y, 'b.', markersize=10, label=u'Observations')plt.plot(xx, y_pred, 'r-', label=u'Prediction')plt.plot(xx, y_upper, 'k-')plt.plot(xx, y_lower, 'k-')plt.fill(np.concatenate([xx, xx[::-1]]),         np.concatenate([y_upper, y_lower[::-1]]),         alpha=.5, fc='b', ec='None', label='90% prediction interval')plt.xlabel('$Time of Day by Fraction$')plt.ylabel('$Latitude$')plt.ylim(39.6, 39.7)plt.legend(loc='upper right')plt.show()ypred2 = y_pred
We can see that our data is almost bimodal on the Latitude axis, and that the time of day does not significantly predict a shift in direction neither North nor South.
#----------------------------------------------------------------------#  First the noiseless caseX = np.atleast_2d(df['timeint'].values).T# Observationsy = np.atleast_2d(df['longitude'].values).T# Mesh the input space for evaluations of the real function, the prediction and# its MSExx = np.atleast_2d(np.linspace(0, 1, 913)).Txx = xx.astype(np.float32)alpha = 0.95clf = GradientBoostingRegressor(loss='quantile', alpha=alpha,                                n_estimators=250, max_depth=3,                                learning_rate=.1, min_samples_leaf=9,                                min_samples_split=9)clf.fit(X, y)# Make the prediction on the meshed x-axisy_upper = clf.predict(xx)clf.set_params(alpha=1.0 - alpha)clf.fit(X, y)# Make the prediction on the meshed x-axisy_lower = clf.predict(xx)clf.set_params(loss='ls')clf.fit(X, y)# Make the prediction on the meshed x-axisy_pred = clf.predict(xx)# Plot the function, the prediction and the 90% confidence interval based on# the MSEfig = plt.figure()plt.figure(figsize=(20,10))plt.plot(X, y, 'b.', markersize=10, label=u'Observations')plt.plot(xx, y_pred, 'r-', label=u'Prediction')plt.plot(xx, y_upper, 'k-')plt.plot(xx, y_lower, 'k-')plt.fill(np.concatenate([xx, xx[::-1]]),         np.concatenate([y_upper, y_lower[::-1]]),         alpha=.5, fc='b', ec='None', label='90% prediction interval')plt.xlabel('$Time of Day by Fraction$')plt.ylabel('$Longitude$')plt.ylim(-80, -79.9)plt.legend(loc='upper right')plt.show()ypred1 = y_pred
Similarly, we can see a near-absence of trend for Longitude East and West, however the data is not nearly as split and has more evenly distributed points. If we were to try and predict subsequent events for the end of the day, we would also expect to notice a lack of a trend, and expect predicted safety incidences near the centroid, or the means of both latitude and longitude.
# Map points of eventsm5 = folium.Map([39.645,-79.96], zoom_start=14)for i in range(len(ypred1)):    folium.CircleMarker([ypred2[i], ypred1[i]],                        radius=4,                        popup=str(i),                        fill_color="#3db7e4", # divvy color                       ).add_to(m5)m5
Evaluating “predicted” timeseries for our data presented no clear conclusion aside from confirming the lack of causality between safety incident time of occurrence and location in town on average. While this could vary from day to day or with extraordinary circumstances, we did notice our trends in Latitude show bimodality, or a partial distinction between North and South Morgantown. If we assess these two areas separately, perhaps we can gather new areas of focus for resource allocation, such as a northern standby location and southern standby location.
dfDowntown = df[df['latitude']<=39.6425]
We can first separate our data on the approximate function line from the last dataset. Then, we will complete our model regression a second time on just the downtown data.
#----------------------------------------------------------------------#  First the noiseless caseX = np.atleast_2d(dfDowntown['timeint'].values).T# Observationsy = np.atleast_2d(dfDowntown['latitude'].values).T# Mesh the input space for evaluations of the real function, the prediction and# its MSExx = np.atleast_2d(np.linspace(0, 1, 535)).Txx = xx.astype(np.float32)alpha = 0.95clf = GradientBoostingRegressor(loss='quantile', alpha=alpha,                                n_estimators=250, max_depth=3,                                learning_rate=.1, min_samples_leaf=9,                                min_samples_split=9)clf.fit(X, y)# Make the prediction on the meshed x-axisy_upper = clf.predict(xx)clf.set_params(alpha=1.0 - alpha)clf.fit(X, y)# Make the prediction on the meshed x-axisy_lower = clf.predict(xx)clf.set_params(loss='ls')clf.fit(X, y)# Make the prediction on the meshed x-axisy_pred = clf.predict(xx)# Plot the function, the prediction and the 90% confidence interval based on# the MSEfig = plt.figure()plt.figure(figsize=(20,10))plt.plot(X, y, 'b.', markersize=10, label=u'Observations')plt.plot(xx, y_pred, 'r-', label=u'Prediction')plt.plot(xx, y_upper, 'k-')plt.plot(xx, y_lower, 'k-')plt.fill(np.concatenate([xx, xx[::-1]]),         np.concatenate([y_upper, y_lower[::-1]]),         alpha=.5, fc='b', ec='None', label='90% prediction interval')plt.xlabel('$Time of Day by Fraction$')plt.ylabel('$Latitude$')plt.ylim(39.6, 39.7)plt.legend(loc='upper right')plt.show()ypred2 = y_pred
#----------------------------------------------------------------------#  First the noiseless caseX = np.atleast_2d(dfDowntown['timeint'].values).T# Observationsy = np.atleast_2d(dfDowntown['longitude'].values).T# Mesh the input space for evaluations of the real function, the prediction and# its MSExx = np.atleast_2d(np.linspace(0, 1, 535)).Txx = xx.astype(np.float32)alpha = 0.95clf = GradientBoostingRegressor(loss='quantile', alpha=alpha,                                n_estimators=250, max_depth=3,                                learning_rate=.1, min_samples_leaf=9,                                min_samples_split=9)clf.fit(X, y)# Make the prediction on the meshed x-axisy_upper = clf.predict(xx)clf.set_params(alpha=1.0 - alpha)clf.fit(X, y)# Make the prediction on the meshed x-axisy_lower = clf.predict(xx)clf.set_params(loss='ls')clf.fit(X, y)# Make the prediction on the meshed x-axisy_pred = clf.predict(xx)# Plot the function, the prediction and the 90% confidence interval based on# the MSEfig = plt.figure()plt.figure(figsize=(20,10))plt.plot(X, y, 'b.', markersize=10, label=u'Observations')plt.plot(xx, y_pred, 'r-', label=u'Prediction')plt.plot(xx, y_upper, 'k-')plt.plot(xx, y_lower, 'k-')plt.fill(np.concatenate([xx, xx[::-1]]),         np.concatenate([y_upper, y_lower[::-1]]),         alpha=.5, fc='b', ec='None', label='90% prediction interval')plt.xlabel('$Time of Day by Fraction$')plt.ylabel('$Longitude$')plt.ylim(-80, -79.9)plt.legend(loc='upper right')plt.show()ypred1 = y_pred
We can notice now a tighter, more responsive prediction and prediction interval. Mapped out, we also see areas of focus where we could potentially want to assess focusing safety response availability.
# Map points of eventsm7 = folium.Map([39.645,-79.96], zoom_start=14)for i in range(len(ypred2)):    folium.CircleMarker([ypred2[i], ypred1[i]],                        radius=4,                        popup=str(i),                        fill_color="#3db7e4", # divvy color                       ).add_to(m7)# convert to (n, 2) nd-array format for heatmapmatrix = np.column_stack((ypred2,ypred1))# plot heatmapm7.add_child(plugins.HeatMap(matrix, radius=15))m7
The same can now be done for Evansdale.
dfEvansdale = df[df['latitude']>39.6425]
From this data, we can see a good location for staging safety services on the Evansdale Campus is near the Towers dormitories and recreation centers near the football stadium.
It can be very challenging to model machine learning based on timeseries data, and it is much easier to implement with clear causality. Future steps for this project could be to search for more indicators of causality or to filter data further, such as on game days or weekends specifically, or even by nature of event. The purpose of this data is not to perfectly predict where and when a near random event will occur, but to have a better understanding of how to prepare for an event and finding good locations for staging rapid service of events. This application can be applied to multiple other sources as well, such as event data in other universities or even cities if the API or data reporting is available.
For further learning please check out some of my other articles and forecasting tools for financial data! Please follow me and let me know what you think!