Using Classification Models To Predict Vaccinations
By: Ryan Werth
Tags: Machine Learning, Pandas, Python, Classification, Data Science

We are currently in the seventh month of a global pandemic. I think I can safely say no one is enjoying it. The quicker this ends the better. One of the most important tools the medical field has when it comes to mitigating the spread of a virus is vaccination. However, vaccinations don’t work if no one gets vaccinated. Now, I realize that sounds like an obvious statement, and in many ways it is. However, it is nonetheless a hugely important fact. This is the underlying idea of “herd immunity”, and it is vital to be able to identify members of the community who are unlikely to get vaccinated.

Fortunately for us aspiring data scientists this is not our first pandemic. In response to the H1N1 Flu in 2009, the Centers for Disease Control and Prevention (CDC) conducted a survey “in order to monitor and evaluate flu vaccination efforts among adults and children”. This phone survey asked people whether they had received H1N1 and seasonal flu vaccines, in conjunction with information they shared about their lives, opinions, and behaviors. DrivenData provided a large chunk of this dataset and posed the question: Using the survey results can you make a model that predicts who will get either vaccine? I figured I would give it a shot (pun intended).

For my last project, I created a linear regression model to predict a baseball team’s total wins. The target there (wins) is a continuous value and so it was perfectly suited for a regression model. This problem is not so simple (or perhaps it’s more simple?). Here our target value is binary, whether or not a participant got the vaccine. So for this problem, we will be looking at classification models rather than regression models. These are models that predict the likelihood of one result or the other, rather than trying to predict a continuous variable.

In this post I will walk through the process I took to build my model as well as explain some of the different classification models I ended up not using.

Data Cleaning

Fair warning, this is the boring part, so if you aren’t very interested you will not lose anything by skipping this section.

The data set consisted of 35 different questions. Twenty-three of the questions had numerical answers, with either a range on an opinion scale (1–5) or a simple binary yes(1) or no(0). Twelve of the questions however had categorical answers, i.e. income level, education level, race, city, etc. Most classification models will not work with pure categorical values, so I had to use a few different techniques to convert these categorical values into numerical inputs.

The first method I used is to create dummy features for each category. For example, one question asked for a home city. Instead of keeping that one column with all the cities (in text form), I turned each city into its own column. Each city column had a 0 if the respondee did not live there and a 1 if they did. This can be a dangerous approach as it can greatly increase the number of columns, and thus the dimensionality of your dataset.

The second method is to convert the text data to ordinal numeric values. This is done when there is some ordered relationship between the values. For example, when looking at education, the data was in the form of “Graduated High School”, “Completed Some College”, “Graduated College”, etc. These values can be converted to 14, 16, and 18 respectively to give a rough estimation of “years in school”. Again, this is only really appropriate where there is already some underlying relationship between the values, where one is “larger” than the other.

On top of these transformations, I also had to deal with all of the “Null” values. In particular, for a question on whether or not a participant had health care, nearly half of the responses were “Null” i.e. no data. My first inclination was to just toss this column altogether, there’s not enough data so just get rid of it. However, upon closer look, I saw that while slightly over 20% of all the respondents got the H1N1 vaccine, of the people with “Null” healthcare only 13% got the H1N1 vaccine.

This told me that there may actually be value in those “Null” values, so I decided to create “dummy variables” out of that column. I ended up with three columns, “Has Healthcare”, “Doesn’t Have Healthcare”, and “We Don’t Know”.

Was this the right approach? Perhaps not, and with more time I would have loved to play around with this feature some more. But this process really exemplified how data cleaning is not a cut and dry process, but rather each decision is unique and requires thought out reasoning. Anyways, now that I had a numerical full data set with no “Null” values I was ready to move to the fun part.

Model Introduction

Welcome back to those that left us.

There are quite a few classification models to choose from. For simplicity (and time) I will be focusing on three classifiers in this post: K-Nearest Neighbors (KNN), Random Forest Classifiers, Logistic Regression.

KNN

Let’s start with KNN. This model uses clustering to create a prediction. Let’s first envision a problem with only two features (X and Y) and a target (Orange and Blue). We could graph those features like so:

Example KNN

Now if we are given a new data point with an X and Y but not color, we could plot it and see what colors are around it. If it is near a lot of blues (in the top right) we will categorize it as blue, if it winds up near a lot of oranges (in the bottom left) we will categorize it as orange. We can fairly easily conceptualize how this would work if we added in a third dimension. And thanks to distance formulas in math, this idea can be scaled up to theoretically infinite dimensions. This is exactly what KNN models are doing.

The main parameter you can set in a KNN model is the number of neighbors it will look for to categorize the data point. If you set K to 1, then the new data will be categorized as whatever the closest point is categorized as. This would lead to an extremely overfit model. Likewise, if you set K too high then the model will tend to underfit. Finding an appropriate K is imperative for a good model.

Random Forest Classifier

The second model I looked at was a Random Forest Classifier (RFC). An RFC is just a collection of Decision Trees, so let’s first breakdown what a decision tree is.

Here’s a definition straight from Wikipedia: “A decision tree is a flowchart-like structure in which each internal node represents a “test” on an attribute”. Huh? Ok, maybe a visual representation would be a bit easier.

Example Decision Tree

The above is an example decision tree to decide what a person should do with their day. The questions are the nodes of the tree, and depending on their answers you work your way down different branches of the “tree”. At a certain point, you reach the “leaf node” which is the category that the model predicts. What does this mean for my model? Well, each survey question will be a different node. So if I feed in a set of survey answers, it will run it down the decision tree model and land at a leaf node which would be the prediction.

One main variable you can use to “mold” this model is the tree’s “depth”. The initial node is set at depth 0, and each ensuing branch adds a layer of depth. The “deeper” a tree is the more overfit it can become. A really deep tree would do a great job of explaining the training data, but a poor job of generalizing for new data. There is no “one size fits all” for how deep a tree should be. It will differ for each dataset and each problem. Again, it’s helpful to have a good understanding of your problem in order to understand how to play with this variable.

One dataset can have thousands of Decision Trees that give slightly different results. An RFC aggregates a certain number of these trees into a “Forest” and takes the mode of all of the Decision Trees’ results. For example, I made an RFC of 100 trees. If I feed in a set of answers and 70 trees predict “No Vaccine” and 30 predict “Yes Vaccine”, then the RFC will return a “No Vaccine” Prediction.

Logistic Regression

The last model I’ll discuss in the Logistic Regression model. Of the three this is definitely the hardest to conceptualize. Whereas Linear Regression assumed that there was a linear relationship between our features and our prediction, Logistic Regression assumes a linear relationship between the features and the log-odds of the prediction. So ultimately, the output of a logistic regression model is the percent chance the data entries correspond to one of two binary predictions.

Example of Linear regression

As you can see in the above image, logistic regression will give you a value between 1 and 0. The closer the value is to 1, the more likely it is to belong to that class. In my case 1 represents a “Yes Vaccine”, so if a logistic regression model returned a 0.85 it was telling me that the given survey inputs corresponded to an 85% chance of getting vaccinated.

For a logistic regression model, it is very important to understand your “threshold”. The threshold determines at what cut-off to split your data. If you set your threshold to 0.6 then anything under that would be predicted as a 0. There are a few different methods to determine what threshold works best for your dataset that I will get into later, but again the most important thing to understand is it will vary for every problem.

Model Selection

Once I had a good understanding of how each model worked, I needed to determine which model would work best for my problem. My model would need to predict two separate targets, H1N1 vaccination, and Seasonal-Flu Vaccination. In the initial data exploration, I quickly noticed that the H1N1 target was highly imbalanced.

The proportion of the population that got each vaccine

As you can see from the above graph, while close to 50% of the respondents got the Seasonal Flu Vaccination, only about 20% of the respondents got the H1N1 vaccination. This means that we can’t rely on a metric like accuracy to determine how good our model is. Why not? Well, if we made a “dumb model” that just predicted a respondent would not get the H1N1 every single time regardless of the inputs, it would still have an accuracy of almost 80 percent!

For this problem, I decided that the False Positives would be the term I would like to limit. These would be the people that my model predicted WOULD get a vaccination that ended up NOT getting a vaccination. This number is important because ultimately I want to be capturing as many of the non-vaccinators as possible. The more False Positives I have, the more non-vaccinators are slipping through the crack.

Let’s take a look at how my models did. For the sake of this blog, I am just showing what I did for the unbalanced H1N1 target, but in actuality, I performed all these steps for both target variables.

Here you can see three different confusion matrixes for my three models predicting H1N1 vaccines. The False Positives are shown in the top right corner. These are the people the model predicted would get the vaccine that did not. You can see that the Nearest Neighbors method fared the worst, misidentifying 241 non-vaccinators as vaccinators. In addition, it also had the highest number of False Negatives, people it predicted to not get the vaccine that did get it.

So it was clear to see that the KNN model would not be the best model in predicting this problem. The other two models had much closer values. I had to look a bit further into these to decide which I would use.

Recall versus Precision

Recall and precision are two vital metrics when analyzing a classification model’s predictions. They are also very easily mixed up. Here are there formulas:

So what do they mean? Precision can be thought of as the percentage of correct identifications out of all of the true values. While the recall is the percentage of correct identifications out of all of the models predicted identifications. In my case, I wanted to limit the false negatives so I wanted to maximize the recall. There will always be a trade-off between the two. So in order to get a higher recall value, sometimes you have to sacrifice a bit of precision.

I realize those are a lot of numbers to look at. But focusing on the No H1N1 vaccination recall we see that both models had a score of .95. This means that of all the people that didn't get vaccinated, the models were able to identify 95% of them. Comparing this to the roughly 80% we would have gotten from just guessing that EVERYONE wouldn't have gotten vaccinated we see that our models are doing definitely doing better than a “dumb model”.

It turns out my two models were pretty neck and neck when it came to their predictive abilities. By using some fancy ensembling techniques (combining multiple models and taking their averages) I was able to get to a model that consistently (albeit slightly) outperformed these more basic models. However, the last thing I had to factor in was model complexity.

For me, this meant how big my actual model was (in actual MBs when I stored it) and how long the model took to make predictions. Logistic regression was an order of ten times smaller than the Random Forest model (remember the random forest was comprised of 100 decision trees) and could produce very quick predictions. So ultimately I decided to use the logistic regression model as my final model for my app. Try it out and see if anything surprises you!

Takeaways

I found that the most important features for my model were whether or not vaccines were recommended by a doctor and whether or not the respondent had health insurance. This is noteworthy because it tells us that across all ages, education levels, and socio-economic levels a doctor's recommendation is possibly the biggest tool we have. It may not be the most exciting discovery, but I believe it is important none the less. It seems imperative that doctors provide vaccine recommendations during any and every check-up. And if the doctors better understand the impact their recommendations have, I’m sure they would make sure to do so.