Introduction to Yellowbrick: A Python Library to Visualize the Prediction of your Machine Learning Model | by Khuyen Tran | Towards Data Science

Congratulation! You have just trained a model and improved your f1-score to 98%! But what does it really mean? Will the increase in the f1 score indicate that your model is performing better?
You know f1-score is the harmonic between recall and precision, but how many of them are false among the positive predictions? And how many of them are false among the negative predictions?
If you want to hypertune your machine learning model to improve the f1-score to 99%, which category should you focus on improving to have an increase of 1%?
Gaining these insights would help you to understand your machine learning results and know which action you should take to improve the model. One of the best ways to understand machine learning is through plots. That is when Yellowbrick becomes helpful.
Yellowbrick is a machine learning visualization libary. Essentially, Yellowbrick makes it easier for you to:
Select features
Tuning hyperparameters
Interpret the score of your models
Visualize text data
Being able to analyze your data and model with plots would make it much easier for you to understand your model and figure out the next steps to increase the scores that are meaningful to your goal.
In this article, we will play with a classification problem to learn which tools yellowbrick provides that can help you interpret your classification results.
To install Yellowbrick, type
pip install yellowbrick
We will use occupancy, the experimental data used for binary classification (room occupancy) from Temperature, Humidity, Light, and CO2. Ground-truth occupancy was obtained from time stamped pictures that were taken every minute.
How correlated are for each pair of features in the data? A two-dimensional ranking of features utilizes a ranking algorithm that takes into account pairs of features at a time. We use Pearson correlation to score to detect colinear relationships.
Based on the data, humidity is strongly correlated to the relative humidity. Light is strongly correlated to the temperature. This makes sense since these features often come hand in hand with each other.
One of the biggest challenges for classification models is the imbalance of classes in training data. Our high f1-score might be not a good evaluation score for an imbalanced class because the classifier can simply guess all the majority class to get a high score.
Thus, it is important to visualize the distribution of the class. We could utilize ClassBalance to visualize the distribution of the class with a bar chart
It seems like there are a lot more data classified as unoccupied than occupied. Knowing this, we can utilize several techniques for dealing with class imbalance such as stratified sampling, weighting to get a more informative result.
Now we come back to the question: What does the f1-score of 98% really mean? Does the increase in f1-score result in more profit for your company?
Yellowbrick provides several tools you can use to visualize the results for classification problems. Some of them you might have or have not heard of, that can be extremely helpful for interpreting your model.
What is the percentage of false predictions mong the unoccupied class? What is the percentage of false predictions among the occupied class? The confusion matrix helps us to answer this question
It seems like the occupied class is with a higher percentage of wrong predictions; thus, we could try to improve the number of the right predictions in the occupied class to improve the score.
Imagine we improve our f1 score to 99%, how do we know that it is actually better? Confusion matrix could help but instead of comparing the percentage of right prediction in each class between two models, is there an easier way to compare to the performance of two models? That is when ROC AUC would be helpful.
A ROCAUC plot allows the user to visualize the tradeoff between the classifier’s sensitivity and specificity. A ROC curve displays the true positive rate on the Y axis and the false positive rate on the X axis.
The ideal point is therefore the top-left corner of the plot: false positives are zero and true positives are one. The higher the area under the curve (AUC), the better the model generally is.
Considering that our ROC curves are near the top-left corner, the performance of our model is really good. If we observe that a different model or a different hyperparameter result in ROC curves are closer to the top left corner then our current one, we can assure that the performance of our model actually improves.
Now we understand the performance of our model, how do we go about improving the model? To improve our model, we might want to
Prevent our model from underfitting or overfitting
Find the features are the most important to the estimator
We will explore the tools provided by Yellowbrick to help us figure out how to improve our model
A model can have many hyperparameters. We might select the hyperparameters that accurately predict the training data. The good way to find the best hyperparameters is to choose a combination of those parameters with a grid search.
But how do we know that those hyperparameters will also accurately predict the test data? It is useful to plot the influence of a single hyperparameter on the training and test data to determine if the estimator is underfitting or overfitting for some hyperparameter values.
Validation Curve could help us to find the sweet spot where lower or higher value than this hyperparameter will result in underfitting or overfitting the data
As we can see from the plot, while the higher number of max depth results in the higher training score, but it also results in the lower cross-validation score. This makes sense since decision trees become more overfitting the deeper they are.
Thus, the sweet spot will be where the cross-validation score does not decrease, which is 1.
Will more data result in a better performance of the model? Not always, the estimator may be more sensitive to error due to variance. That is when the learning curve is helpful.
A learning curve shows the relationship of the training score versus the cross-validated test score for an estimator with a varying number of training samples.
From the graph, we can see that the number of training instances of around 8700 results in the best f1-score. The higher number of training instances results in a lower f1-score.
Having more features is not always equivalent to a better model. The more features the model has, the more sensitive the model is to errors due to variance. Thus, we want to select the minimum required features to produce a valid model.
A common approach to eliminate features is to eliminate the ones that are the least important to the model. Then we re-evaluate if the model actually performs better during cross-validation.
Feature importance is perfect for this task since it helps us to visualize the relative importance of the features for the model.
It seems like the light is the most important feature to DecisionTreeClassifier, followed by CO2, temperature.
Considering there are not many features in our data, we will not eliminate humidity. But if there are many features in our model, we should eliminate the ones that are not important to the model to prevent errors due to variance.
Congratulations! You have just learned how to create plots that help you to interpret the results of your model. Being able to understand your machine learning results will make it easier for you to find out the next steps to improve its performance. There are more functionalities of Yellowbrick than what I mention here. To learn more about how to use Yellowbrick to interpret your machine learning model, I encourage you to check the doc here.
The source code of this repo could be found here.
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: