Boruta Feature Selection (an Example in Python) | by Aaron Lee | Towards Data Science

If you aren’t using Boruta for feature selection, you should try it out. It can be used on any classification model. Boruta is a random forest based method, so it works for tree models like Random Forest or XGBoost, but is also valid with other classification models like Logistic Regression or SVM.
Boruta iteratively removes features that are statistically less relevant than a random probe (artificial noise variables introduced by the Boruta algorithm). In each iteration, rejected variables are removed from consideration in the next iteration. It generally ends up with a good global optimization for feature selection which is why I like it.
We will take a look at a simple random forest example for feature selection. The only intention of this story is to show you an easy working example so you too can use Boruta.
This example will use the breast_cancer dataset that comes with sklearn. You can use the optional return_X_y to have it output arrays directly as shown.
Simply fit the data to your chosen model, and now it is ready for Boruta.
Note: I fit entire dataset when doing feature selection. You can do the train/test split after you have eliminated features.
Now create a BorutaPy feature selection object and fit your entire data to it. During the fit, Boruta will do a number of iterations of feature testing depending on the size of your dataset. Boruta creates random shadow copies of your features (noise) and tests the feature against those copies to determine if it is better than the noise, and therefore worth keeping. It automatically checks for interactions that might hurt your model. Boruta will output confirmed, tentative, and rejected variables for every iteration.
After being fit, the Boruta object has useful attributes and methods:
.support_ attribute is a boolean array that answers — should feature should be kept?
.ranking_ attribute is an int array for the rank (1 is the best feature(s))
.transform(X) method applies the suggestions and returns an array of adjusted data. In this way, you could just let Boruta manage the entire ordeal.
Note: If you get an error (TypeError: invalid key), try converting your X and y to numpy arrays before fitting them to the selector. Let me know if you run into this error and need help.
Here is some quick code I wrote to look output Boruta’s results. Looks like 5 of my 30 features were recommended to be dropped. The output of the code is shown below.
Feature: mean radius               Rank: 1,  Keep: TrueFeature: mean texture              Rank: 1,  Keep: TrueFeature: mean perimeter            Rank: 1,  Keep: TrueFeature: mean area                 Rank: 1,  Keep: TrueFeature: mean smoothness           Rank: 1,  Keep: TrueFeature: mean compactness          Rank: 1,  Keep: TrueFeature: mean concavity            Rank: 1,  Keep: TrueFeature: mean concave points       Rank: 1,  Keep: TrueFeature: mean symmetry             Rank: 1,  Keep: TrueFeature: mean fractal dimension    Rank: 2,  Keep: FalseFeature: radius error              Rank: 1,  Keep: TrueFeature: texture error             Rank: 2,  Keep: FalseFeature: perimeter error           Rank: 1,  Keep: TrueFeature: area error                Rank: 1,  Keep: TrueFeature: smoothness error          Rank: 3,  Keep: FalseFeature: compactness error         Rank: 1,  Keep: TrueFeature: concavity error           Rank: 1,  Keep: TrueFeature: concave points error      Rank: 1,  Keep: TrueFeature: symmetry error            Rank: 2,  Keep: FalseFeature: fractal dimension error   Rank: 2,  Keep: FalseFeature: worst radius              Rank: 1,  Keep: TrueFeature: worst texture             Rank: 1,  Keep: TrueFeature: worst perimeter           Rank: 1,  Keep: TrueFeature: worst area                Rank: 1,  Keep: TrueFeature: worst smoothness          Rank: 1,  Keep: TrueFeature: worst compactness         Rank: 1,  Keep: TrueFeature: worst concavity           Rank: 1,  Keep: TrueFeature: worst concave points      Rank: 1,  Keep: TrueFeature: worst symmetry            Rank: 1,  Keep: TrueFeature: worst fractal dimension   Rank: 1,  Keep: True
Now that we have identified the features to drop, we can confidently drop them and proceed with our normal routine. You can even use the .transform()method to automatically drop them. You may want to try other feature selection methods to suit your needs, but Boruta uses one of the most powerful algorithms out there, and is quick and easy to use.
Good luck!
Feel free to reply if you run into trouble, and I will help out if I can.