Predicting vehicle accident severity using ensemble classifiers and AutoML | by Ranganath Venkataraman | Towards Data Science

INTRODUCTION / BUSINESS PROBLEM
This essay details work that uses a publicly available dataset on vehicle accidents registered by the Seattle City Department of Transportation. This work benefits the American public by identifying those factors conducive to various types of traffic incidents, predicting accident severity, and using this information to minimize this risk.
The full code is found at my GitHub repo
OBJECTIVE
Predict the severity of accident as ‘1’ or ‘2’ using features including the number of cars and people, traffic, weather conditions, etc. This is a classification problem that — once modeled — gives Seattle City authorities previously unavailable insight into accident risk factors, and also gives this author the necessary completion to gain a Coursera Professional Certification.
DATA USED and STEPS TAKEN
The raw data used is provided by the Seattle DOT with the following steps applied:
Step 1: data loading and preliminary insights
Step 2: detailed analysis of feature visualization. The goal is to understand potentially leveraging input variables
Step 3: feature engineering and selection
Step 4: model fitting and training
Step 5: model evaluation, with sensitivity check using auto-machine learning (also referred to as AutoML)
Step 6: concluding takeaways for the client
Steps 1 and 2: taken together here. The traffic data is loaded and analyzed using Seaborn visualization tools — our goal is to understand data distributions, attributes, and implications for predicting vehicle accident severity.
The Seaborn visualizations were uninformative and can be found at the repo — they are not published here, given the absence of value addition to this post. Therefore, I analyzed the data and some explanatory Seattle DOT literature to decipher features.
What did I learn?
Many of the features were unique identifiers used by the Department of Transportation and provide in predicting accidents. These include incident key, intersection key, collision detail key, seg lane key, and crosswalk key.
ObjectID is unique in that while it is also a classifier courtesy of the DOT, it does not duplicate information explicitly available in another feature.
There are many features that are binary and non-numerical i.e. answerable via a simple yes or no. These include a check of whether: speeding was a factor, the pedestrian had right of way, inattention or DUI were contributing factors or a parked car was hit.
Finally, there were other non-numerical features, which had more than two possible entries. These covered factors like lighting and road conditions, the type of junction that saw the accident, and the date/time of the incident. I’m classifying time as non-numerical for purposes of this article since it’s a datetime object per Python.
Step 3: the list specified above compels a methodical approach when considering what to include/exclude and what type of feature engineering might be needed. We want to ensure the inclusion of influential factors without increasing unnecessary complexity that leads to overfitting
What approach preserves the most useful information and removes unnecessary features that add complexity?
Let’s first remove features that are administrative tools for the Seattle DOT. These include report number, incident key, and intersection key, and status. Let’s also delete the columns ‘EXCEPTRSNCODE’ and ‘EXCEPTRSNDESC’ after removing the rows that had entries in these columns — these existed simply to inform the reader that an incident lacked enough information.
Let’s also remove the coordinates x and y since they are centered around one to two specific areas and are assumed to hold no predictive insight.
Any non-numeric feature will require encoding to convert to numbers that can be processed by a machine learning algorithm. Encoding can be either ordinal or one-hot: I pursued the latter since none of these variables held any inherent order that I wanted to preserve. Features impacted with this approach include address type, road and lighting conditions, whether a parked car was hit, collision key, weather, and junction type.
Finally, I used the incident date and time to extract the month and time. The time was translated to morning, noon, or evening and subsequently, this detail was one-hot encoded. I saw this as relevant, given driving accidents can increase at late nights / early mornings or when seasons render roads trickier to navigate
Before we move on to model evaluation, let’s take a look at how the steps above changed some key features in our dataset. Here’s a snapshot of the before:
Here’s a snapshot of the dataset after — in this excerpt, we see that address type is encoded as is speeding and whether pedestrian had right of way. This is on the training data, having earlier completed a train-test-split:
MODEL FITTING AND TRAINING
With the requisite Feature Engineering and Selection completed, I then selected a list of classifiers — including Dummy for baseline, trained them, and evaluated their performance on the test set using a number of metrics. The code snippet follows.
dum = DummyClassifier()xgb = XGBClassifier()dt = DecisionTreeClassifier()gbc = GradientBoostingClassifier()rfc = RandomForestClassifier()clfs = [dum,xgb,dt,gbc,rfc]f1_Scores = []jaccard_Scores = []log_loss_Scores = []for clf in clfs:    clf.fit(x_train,y_train)    ypred = clf.predict(x_test)    ypred_proba = clf.predict_proba(x_test)    f1score = f1_score(y_test,ypred)    logloss = log_loss(y_test,ypred_proba)    jaccard = jaccard_score(y_test,ypred)    f1_Scores.append(f1score)    jaccard_Scores.append(jaccard)    log_loss_Scores.append(logloss)
Step 5: I’ve summarized model performance in the following Excel table. Best practices suggest using multiple metrics to evaluate performance:
The table above demonstrates F1 scores in the mid-80s and log-loss scores below 0.5, which is an improvement over the Dummy Classifier’s baseline performance. We’ll talk about the implications for the metrics above for our client in the conclusion, but I wanted to touch on two other topics first:
GridSearchCV on XGBoost didn’t offer any improvement on the scores above
I used the AutoML package TPOT. See code snippets below with resulting performance:
%pip install tpot --userimport tpotfrom tpot import TPOTClassifier# define model evaluationcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)# define searchmodel = TPOTClassifier(generations=5, population_size=50, cv=cv, scoring='f1', verbosity=2, random_state=1, n_jobs=-1)# perform the searchmodel.fit(x_train,y_train)# export the best modelmodel.export('tpot_capstone_best_model.py')
The exported model is then reintroduced for training in the dataset and evaluation on the test-set. As observed in the code snippet below, TPOT chose to use a DecisionTree classifier on the top 49% performing features, with resulting F1 score of 0.84 and logloss of 0.5. AutoML is quicker , but didn’t return results that were any more promising than traditional methods.
from sklearn.pipeline import make_pipelinefrom tpot.export_utils import set_param_recursivefrom sklearn.feature_selection import SelectPercentile, f_classifexported_pipeline = make_pipeline(    SelectPercentile(score_func=f_classif, percentile=49),    DecisionTreeClassifier(criterion="entropy", max_depth=6, min_samples_leaf=19, min_samples_split=7))# Fix random state for all the steps in exported pipelineset_param_recursive(exported_pipeline.steps, 'random_state', 1)exported_pipeline.fit(x_train, y_train)tpot_y_pred = exported_pipeline.predict(x_test)tpot_y_pred_prob = exported_pipeline.predict_proba(x_test)f1score = f1_score(y_test,tpot_y_pred)logloss = log_loss(y_test,tpot_y_pred_prob)jaccard = jaccard_score(y_test,tpot_y_pred)
Step 6: we can give our client — the Seattle DOT — a model that demonstrates performance in identifying what does and equally important, what doesn’t lead to accidents of particular severities.
Our model is better than blind-guessing since the log_loss score of 0.47 is better than a “dumb score” of 0.59 based on class distributions. It’s not only predicting severities accurately but able to distinguish between the different classes — as measured by the high F1-score that exceeds that of the baseline performer, the Dummy Classifier.
Insights for the Seattle Department of Transportation:
the month of the year is the second-most influential variable when predicting accident severity. Makes sense that certain months see poorer weather conditions, less focus by drivers. The City of Seattle should consider safety programs, increased traffic accident avoidance efforts in problematic times of the year
the number of people and vehicles involved in an accident is influential on severity. The City of Seattle should consider options to limit passengers in a vehicle during certain times of the month and day, to minimize the magnitude of risk.
Seglane key and crosswalk key represent the area of the incident — understanding these specific areas mean that the city can install more cameras, more signage to discourage reckless behavior at these specific intersections.
Inattention is significantly less impactful than those above, but one that the City can still influence. The City of Seattle should consider increasing fines for distracted driving and putting out more PSAs to discourage this behavior.
The type of collision has the greatest influence in predicting severity. This is not surprising but not actionable by itself
As always, I welcome ideas, suggestions, and feedback.