Extra Trees, please
By: Diane Tunnicliffe
Tags: Machine Learning, Flatiron School, Classification, Gridsearchcv, Data Science

One of the first things I realized when embarking on this project was that pipelines are basically magical creations to streamline the process of fitting a classifier to the data. Pipelines take all the steps for scaling, preprocessing, and setting a classifier, and combine them into a single object that completes the process cleanly and eliminates some human error. Since there were many different algorithms I wanted to explore, I began by writing a function to take in a classifier of your choosing and return a pipeline object ready to be fitted to an X and y. (Please note that my class_weight parameter is set to ‘balanced’ to account for my imbalanced target variable; there were many more Normal health babies than At-Risk babies in my data.)

Gist by author.

Now, arguably even more amazing than pipelines is the ability to grid search. GridSearchCV allows you to check many different hyperparameters for a model and optimize your model by choosing the best params for a chosen metric. In this case, the metric I was prioritizing was recall. By optimizing for recall, I was minimizing the amount of false negatives or Type II errors in my model’s predictions. While I was okay with the model sometimes falsely predicting that a baby was at risk, I was less okay with the model sometimes falsely predicting that the baby was healthy. Because we are talking about human lives here, the recall metric was the most important metric for this project, far above precision, accuracy, or any other measures of evaluation. To this end, I wrote a function that takes in a pipeline object (from the function above, for instance) and a set of hyperparameters to be tuned, and returns a grid search object. If you are optimizing for something other than recall, such as accuracy, you’d just have to change the scoring parameter to reflect that.

Gist by author.

Finally, I wrote a function to take in a grid search object (like the one created from the function above )and return relevant information to be used for evaluation. Now this is where the real magic happens. This last function takes in that grid search object from above and gets right to work on optimizing your model. You sit and you wait. Your computer makes some scary noises. You back away slowly. Your husband apprehensively asks if he should be able to hear your computer from three rooms away. You smile and say everything’s fine while you sweat profusely and calculate the cost of a new MacBook in your head. You occasionally peek into the room and see that the ominous hourglass is still on the screen, the dark gray circle still in the upper righthand corner of the Jupyter notebook, the whirring and growling only increasing in volume. But then, after some time has passed, everything is bright and quiet again. And there waiting for you on the screen, in your output cell, is:

the very best possible hyperparameters for your model, based on your chosen evaluation metric,

a classification report with your precision, recall, F1 score, and accuracy values for this optimized model,

and a color-coded confusion matrix to tell you the number of true positives, true negatives, false positives, and false negatives based on this model’s predictions.

Gist by author.

Image by author.

Then, once you have this, you get to do it all again! Over and over, as many times as necessary, until you have your best possible model. (And no, the anxiety that happens when the computer makes those noises does not go away; you just gradually come to accept the fact that your computer may blast off at any moment. It makes life exciting.) Personally, I tried out a couple of Decision Trees, some Logistic Regression, a Random Forest, a Support Vector Machine, and even some AutoML with TPOT Classifier (which did not go as well as I had hoped). This image above is from the best model I was able to accomplish, using an Extra Trees classifier. It has a recall score of 97%, which while not perfect, is much better than my baseline model’s recall score of 77%.

Wait, but what exactly is this Extra Trees classifier?

If you are familiar with Random Forest, then you know that it is an algorithm comprised of an ensemble of Decision Trees. Extra Trees is a very similar algorithm that uses a collection of Decision Trees to make a final prediction about which class or category a data point belongs in. Extra Trees differs from Random Forest, however, in the fact that it uses the whole original sample as opposed to subsampling the data with replacement as Random Forest does. Another difference is in how the nodes are split. While Random Forest is built to always choose the best possible split, Extra Trees chooses random splits. Both Extra Trees and Random Forest are both programmed to optimize the end result.

I was surprised by the fact that I was able to build a better model with Extra Trees than Random Forest. While my Random Forest model with my grid search’s best params returned a pretty good recall score of 94%, my Extra Trees model returned a better one. The lesson here is that there is much to be gained from working with different algorithms and classifiers, so it’s always a good idea to keep iterating through different models and algorithms until you find the most optimal one for your particular dataset and evaluation metric.