Part 2: Step-by-Step Guide to leave-one-person-out cross validation with Random Forests in Python — Plotting Importance
By: Brinnae Bent
Tags: Machine Learning, Python, Random Forest, Data Science, Data Analysis

This is the second part in the series on leave-one-person-out cross validation with random forests in Python. The first part details the algorithm that we will use today in part two to plot importances.

Just a reminder that these methods have been compiled into a function that is published on the Digital Biomarker Discovery Pipeline (DBDP). If you want to implement this cross validation in your own work with random forests, please visit the repository on the DBDP GitHub. The ‘how-to’ of using the function is well documented and can be easily implemented with only one line in a number of projects.

In this tutorial, I am going to describe a process for plotting importances from leave-one-person-out cross validation in Python for you to implement in your own projects!

Getting Started

Some basic assumptions before we get started: I am assuming you have, at minimum, a basic understanding of functions in python and you have a working knowledge of pandas. I am running under the assumption that you have either followed along with Part 1 of this tutorial or you have found the source code and documentation here.

Getting Importances in the Model

In Part 1 of this tutorial, I recommended that you define a dataframe with the importances from the model. If you did not implement this in Part 1, I recommend doing it now, as shown below:

The entire function is shown below. Note that you are returning the importances dataframe.

Importances DataFrame

Continuing to follow along with the tutorial in Part 1 (feel free to skip if you followed Part 1 exactly), initialize an empty dataframe to hold your importances.

Just a reminder that you need to iterate through all of your participants in your dataset. At each iteration, you are going to append your returns to their respective lists and you will append the importances to the dataframe you initialized previously.

Make sure to return the importances dataframe. The entire function is shown below:

Plotting Importances

Finally, what you all came to this tutorial for! Call the above function. Now you have a dataframe with the importances for each fold.

Next, you will build a function that takes this importances dataframe as an input and outputs a bar chart of the importances of a defined fold. (Reminder that each fold will have different importances- please see Part 1 for an explanation of leave-one-person-out cross validation and why this is).

First, let’s set up a function and set the ID for the fold. This will be whatever you indicated it to be in the previous steps. The function will take as inputs the importances dataframe, a user-defined ID to plot, and a filepathout (this allows you to run this iteratively and just save the plot for each fold).

Next, establish the x values. Set this to be a list of range of length of the importances column in that particular fold.

Here is where you get to have fun and be creative, setting style choices for your barplot. This is a handy site for deciding on styles for your plots.

Define the barplot using the x_values list we made and the importances column for the fold. Define x_ticks, which adds your feature names as labels on the x-axis.

I have added a simple ylabel here. Feel free to add a title and make other stylistic adjustments.

I recommend saving the figure inside your function. This way, you can run the function for every participant ID and have them saved neatly for each fold (participant ID “left out”).

Take a look at the entire function, below:

Here is the output of one fold of our variable importance plot. (Note that in your function, “Feature#” will be your feature names from your original dataset). Add your creativity with color, style, and labeling!

Keep in mind that you have the importances as a dataframe now. This means you can look at average importance of features, you can plot histograms of the importances for a particular feature, and you can start to get a better understanding of how your model is working.