Feature Selection With BorutaPy. This post will serve as a tutorial on... | by Jason Wong | Towards Data Science

This post will serve as a tutorial on how to implement BorutaPy when performing feature selection for a predictive classification model. I will go through some strengths as well as a few weaknesses when choosing to go with BorutaPy. I will be using the Tanzania well classification dataset to try and build a classification model to predict if a well is functional, functional but in need of repair, or not functional.
Feature selection is one of the most important steps in machine learning. When inputting features into a model, the goal is to feed the model with features that are relevant for predicting a class. Including irrelevant features poses the problem of unnecessary noise in the data, resulting in lower model accuracy. Generally, we use statistical-based feature selection methods such as ANOVA or the Chi-squared test, evaluating the relationship between each predictor variable with the target variable.
“Boruta is an all relevant feature selection method, while most other are minimal optimal; this means it tries to find all features carrying information usable for prediction, rather than finding a possibly compact subset of features on which some classifier has a minimal error.” -Miron B. Kursa
With boruta, the features are curating down to the ‘all relevant’ stopping point and not the ‘minimal optimal’ stopping point. An all relevant variable being not redundant when used for prediction. Boruta is based on two ideas, shadow features and binomial distribution. When using boruta, the features are not evaluated with themselves but with a randomized version of them. For the binomial distribution idea, boruta takes a feature that we aren’t aware of being useful or not and either refuses or accepts the feature based on three areas defined by selecting the tails of the distribution, 0.5 percent for an example.
Area of refusal: area where features are dropped as they are considered noise.
Area of irresolution: area where boruta is indecisive about the features
area of acceptance: area where the features are considered predictive
For an example, I will build a ternary classification model to be used with the Tanzanian Water Well Data. This classifier will predict the status of the water well, the dataset can be retrieved here. I will start by going through how I processed the data for the model.
To install BorutaPy run the following code in your notebook or terminal. The dependencies for Boruta are numpy, scipy, and scikit-learn.
###Installing BorutaPypip install BorutaPy###Importing the libraries for data processingimport pandas as pdimport numpy as np###Reading in the datatrain_data = pd.read_csv(‘../data/train_data.csv’)train_target = pd.read_csv(‘../data/train_targets.csv’)test = pd.read_csv(‘../data/test_set_values.csv’)###Adding training label to training dataframetrain = train_data.merge(train_target, on='id', how='inner')###‘status group' needs to be converted to numerical values for the ###classification model and not ‘functional’, ‘non-functional’, and ###‘functional in need of repair’.status_group_numeric = {'functional': 2,                        'functional needs repair': 1,                        'non functional': 0}train['status'] = train.status_group.replace(status_group_numeric)
Checking for any missing values in the dataset
train.isna().sum()
Around 69 percent of the values in the permit field are true and 31 percent are false. I also chose to fill in the missing values with the same ratio. I also need to convert the values to be numerical.
train['permit'].fillna(False, limit=947, inplace=True)train['permit'].fillna(True, inplace=True)###Function to take values in permit field and change them to ###numerical valuesdef permit(row):    if row['permit'] == True:        return 1    else:        return 0train['permit'] = train.apply(lambda x: permit(x), axis=1)
The public meeting field holds 91 percent True and 9 percent False. I chose to fill the missing values with the same ratio. I also need to change the values to be numerical.
train['public_meeting'].fillna(False, limit=300, inplace=True)train['public_meeting'].fillna(True, inplace=True)###Function to take values in public field and convert them to ###numerical valuesdef public(row):    if row['public_meeting'] == True:        return 1    else:        return 0train['public_meeting'] = train.apply(lambda x: public(x), axis=1)
The funder column shows who funded the well. There are 1,897 different outcomes but it looks like only the top 7 fields have values that occur more over a thousand times. I’m going to divide this column into 7 different categories with these, the rest of the fields will be categorized as ‘other’.
def categorize_funder(train):    '''This function will go through every row in    the dataframe column funder and if the value    is any of the top 7 fields, will return those    values. If the value does not equal any of these    top 7 fields, it will return other.'''    if train['funder'] == 'Government Of Tanzania':        return 'govt'    elif train['funder'] == 'Danida':        return 'danida'    elif train['funder'] == 'Hesawa':        return 'hesawa'    elif train['funder'] == 'Rwssp':        return 'rwssp'    elif train['funder'] == 'World Bank':        return 'world_bank'    elif train['funder'] == 'Kkkt':        return 'kkkt'    elif train['funder'] == 'World Vision':        return 'world_vision'    else:        return 'other'train['funder'] = train.apply(lambda x: categorize_funder(x), axis=1)
Binning the installer field into 7 categories
def categorize_installer(train):    '''This function will go through    every row in the installer column    and if the value is equal to any of    the top 7, will return those values.    If not, will return other.'''    if train['installer'] == 'DWE':        return 'dwe'    elif train['installer'] == 'Government':        return 'govt'    elif train['installer'] == 'RWE':        return 'rwe'    elif train['installer'] == 'Commu':        return 'commu'    elif train['installer'] == 'DANIDA':        return 'danida'    elif train['installer'] == 'KKKT':        return 'kkkt'    elif train['installer'] == 'Hesawa':        return 'hesawa'    else:        return 'other'train['installer'] = train.apply(lambda x: categorize_installer(x), axis=1)
Binning the scheme management field into 7 categories
def categorize_scheme(row):    '''This function will go through each    row in the scheme management column    and if the value is equal to any of the     top 7, will return those values. If not,    will categorize the value as other.'''    if row['scheme_management'] == 'VWC':        return 'vwc'    elif row['scheme_management'] == 'WUG':        return 'wug'    elif row['scheme_management'] == 'Water authority':        return 'water_authority'    elif row['scheme_management'] == 'WUA':        return 'wua'    elif row['scheme_management'] == 'Water Board':        return 'water_board'    elif row['scheme_management'] == 'Parastatal':        return 'parastatal'    elif row['scheme_management'] == 'Private operator':        return 'private_operator'    else:        return 'other'train['scheme_management'] = train.apply(lambda x: categorize_scheme(x), axis=1)
It looks like the construction year field will be pretty hard for our model to categorize since we have 55 different construction years in the dataset. I am going to try binning these years into decades.
def categorize_contruction(row):    if row['construction_year'] < 1970:        return '1960s'    elif row['construction_year'] < 1980:        return '1970s'    elif row['construction_year'] < 1990:        return '1980s'    elif row['construction_year'] < 2000:        return '1990s'    elif row['construction_year'] < 2010:        return '2000s'    elif row['construction_year'] < 2020:        return '2010s'train['construction_year'] = train.apply(lambda x: categorize_contruction(x), axis=1)
I chose to drop the fields that had values similar to another and the fields holding values that I could not find enough information relevant for the predictive model. The dropped features could hold important information, I’d recommend taking a closer look into the data. I’m going to move forward without them for the purpose of this example.
- train = train.drop(columns=['subvillage'], axis=1)- train = train.drop(columns=['scheme_name'], axis=1)- train = train.drop(columns=['wpt_name'], axis=1)- train = train.drop(columns=['region'], axis=1)- train = train.drop(columns=['extraction_type', 'extraction_type_group'],                  axis=1)- train = train.drop(columns=['management_group'], axis=1)- train = train.drop(columns=['payment_type'], axis=1)- train = train.drop(columns=['water_quality'], axis=1)- train = train.drop(columns=['quantity_group'], axis=1)- train = train.drop(columns=['source_type'], axis=1)- train = train.drop(columns=['waterpoint_type_group'], axis=1)
pivot_train = pd.pivot_table(train, index=['status_group'],                             values='status',                            aggfunc='count')pivot_train
X = train.drop(columns=['id', 'status_group', 'status', 'date_recorded'], axis=1)y = train.statusX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
I encoded the categorical features with one Sklearn’s hot encoder and scaled the continuous features Sklearn’s standard scaler.
###Instantiating One Hot Encoderohe = OneHotEncoder()###Defining features to encodeohe_features = ['funder', 'installer', 'basin',                'region_code', 'district_code', 'lga', 'public_meeting',               'scheme_management', 'permit', 'construction_year',                'extraction_type_class', 'management',               'payment', 'quality_group',               'quantity', 'source', 'waterpoint_type']###Defining continuous numerical featurescont_features = ['amount_tsh', 'gps_height', 'longitude',                        'latitude', 'population'###Creating series for categorical test and trainX_train_cat = X_train[ohe_features]X_test_cat = X_test[ohe_features]###Fitting encoder to training categorical features and transforming ###test and trainX_train_ohe = ohe.fit_transform(X_train_cat)X_test_ohe = ohe.transform(X_test_cat)###Converting series to dataframescolumns = ohe.get_feature_names(input_features=X_train_cat.columns)X_train_processed = pd.DataFrame(X_train_ohe.todense(), columns=columns)X_test_processed = pd.DataFrame(X_test_ohe.todense(), columns=columns)###Instantiating Standard Scalerss = StandardScaler()###Converting continuous feature values to floatsX_train_cont = X_train[cont_features].astype(float)X_test_cont = X_test[cont_features].astype(float)###Fitting scaler to training continuous features and transforming ###train and testX_train_scaled = ss.fit_transform(X_train_cont)X_test_scaled = ss.transform(X_test_cont)###Concatenating scaled and encoded dataframesX_train_a2 = pd.concat([pd.DataFrame(X_train_scaled), X_train_processed], axis=1)X_test_a2 = pd.concat([pd.DataFrame(X_test_scaled), X_test_processed], axis=1)
###Classification Algorithmfrom sklearn.ensemble import RandomForestClassifier###Model evaluationfrom sklearn.model_selection import cross_val_scorefrom sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score###Feature Selectionfrom boruta import BorutaPy
For this example, I chose to use random forest classifier, an ensemble tree-based learning algorithm. It is a set of decision trees created with a randomly selected subset of the training data. The random forest classifier then aggregates the votes from different decision trees to choose the final class of the test object. This first run with the random forest classifier will include all of the features then I will run BorutaPy.
###Instantiating Random Forest Classifierrf = RandomForestClassifier(n_estimators=500, random_state=42)###Fitting Random Forest Classifier to training datarf.fit(X_train_a2, y_train)###Print accuracy and recall scores for both test and train###Average is set to micro for recall score since this is a ###multi-class classification model. Micro-average aggregates the ###contributions of all classes to compute the average metric. Micro ###is preferred for data with class imbalance.print('Test Accuracy:')print(accuracy_score(y_test, rf.predict(X_test_a2)))print('Test Recall:')print(recall_score(y_test, rf.predict(X_test_a2), average='micro'))print('Train Accuracy:')print(accuracy_score(y_train, rf.predict(X_train_a2)))print('Train Recall:')print(recall_score(y_train, rf.predict(X_train_a2),  average='micro'))####Output:Test Accuracy:0.80006734006734Test Recall:0.6729820394303477Train Accuracy:0.9955555555555555Train Recall:0.9902567898799216
This resulted with a test score (mean accuracy) of 80% and a training score of 96%, now I will run the features through BorutaPy and see which ones are selected as relevant.
Begins by providing randomness to the features by creating duplicate features and shuffling the values in each column to remove their correlations with the response. (Shadow Features)
Trains a random forest classifier on the dataset and calculates the relevance/importance by gathering the Z-scores.
Finds the maximum Z-score among shadow attributes and assigns a hit to every attribute with a Z-score higher than the maximum Z-score of its shadow feature. (accuracy loss divided by the standard deviation of accuracy loss)
Takes each attribute that has not yet been determined important and perform a two-sided test of quality with the maximum Z-score among shadow attributes.
Tags the attributes with an importance level lower than MZSA as ‘unimportant’.
Tags the attributes with an importance level higher than MZSA as ‘important’.
Removes all shadow attributes
Repeats this process until the importance is computed for all attributes or the algorithm reaches the set number of iterations
###define X and y for boruta, algorithm takes numpy arrays as inputs ###and not dataframe (why you see .values)X_boruta = train_all.drop(columns=['status'], axis=1).valuesy_boruta = train_all.status.values###define random forest classifier, set n_jobs parameter to -1 to ###utilize all processors and set n_estimators parameter to 500, ###number of trees in the forest.rf = RandomForestClassifier(n_jobs=-1, n_estimators=500, oob_score=True, max_depth=6)###Define borutapy with rf as estimator and verbose parameter set to ###2 to output which features have been selected alreadyfeat_selector = BorutaPy(rf, n_estimators='auto', verbose=2, random_state=42)###fit boruta selector to X and y borutafeat_selector.fit(X_boruta, y_boruta)Output:Iteration: 	1 / 100Confirmed: 	0Tentative: 	277Rejected: 	0Iteration: 	2 / 100Confirmed: 	0Tentative: 	277Rejected: 	0Iteration: 	3 / 100Confirmed: 	0Tentative: 	277Rejected: 	0Iteration: 	4 / 100Confirmed: 	0Tentative: 	277Rejected: 	0Iteration: 	5 / 100Confirmed: 	0Tentative: 	277Rejected: 	0Iteration: 	6 / 100Confirmed: 	0Tentative: 	277Rejected: 	0Iteration: 	7 / 100Confirmed: 	0Tentative: 	277Rejected: 	0Iteration: 	8 / 100Confirmed: 	110Tentative: 	49Rejected: 	118Iteration: 	9 / 100Confirmed: 	110Tentative: 	49Rejected: 	118Iteration: 	10 / 100Confirmed: 	110Tentative: 	49Rejected: 	118...Iteration: 	90 / 100Confirmed: 	122Tentative: 	11Rejected: 	144Iteration: 	91 / 100Confirmed: 	122Tentative: 	11Rejected: 	144Iteration: 	92 / 100Confirmed: 	122Tentative: 	11Rejected: 	144Iteration: 	93 / 100Confirmed: 	122Tentative: 	11Rejected: 	144Iteration: 	94 / 100Confirmed: 	122Tentative: 	11Rejected: 	144Iteration: 	95 / 100Confirmed: 	122Tentative: 	11Rejected: 	144Iteration: 	96 / 100Confirmed: 	122Tentative: 	11Rejected: 	144Iteration: 	97 / 100Confirmed: 	122Tentative: 	11Rejected: 	144Iteration: 	98 / 100Confirmed: 	122Tentative: 	11Rejected: 	144Iteration: 	99 / 100Confirmed: 	122Tentative: 	11Rejected: 	144BorutaPy finished running.Iteration: 	100 / 100Confirmed: 	122Tentative: 	2Rejected: 	144
Checking accepted features
###Print accepted features as well as features that boruta did not ###deem unimportant or important (area of irresolution)accept = X.columns[feat_selector.support_].to_list()irresolution = X.columns[feat_selector.support_weak_].to_list()###Call transform on boruta to update X_boruta to selected featuresX_filtered = feat_selector.transform(X_boruta)print('Accepted features:')print('----------------------------')print(list(accept))print('----------------------------')print(list(irresolution))
By using these selected features and tuning the hyperparameter max_depth to 6, we get improved scores.
###Creating new dataframe from original X dataframe with only the ###selected features from BorutaPynew_x = train_all[accept]X2_boruta = new_x.drop(['status'], axis=1)y2_boruta = new_x.status###Train test split on updated XX_t, X_val, y_t, y_val = train_test_split(new_x, y, random_state=42)###Instantiating Random Forest Classifierrf2 = RandomForestClassifier(n_jobs=-1, n_estimators=500, oob_score=True, max_depth=6, random_state=42)###Fitting Random Forest Classifier to train and testrf2.fit(X_t, y_t)###Predicting on test datay_pred = rf2.predict(X_val)###Test Scorerf2.score(X_val, y_val)###Training Scorerf2.score(X_t, y_t)###Print accuracy and recall scores for both test and train ###Average is set to micro for recall score since this is a ###multi-class classification model. Micro-average aggregates the ###contributions of all classes to compute the average metric. Micro ###is preferred for data with class imbalance.print('Test Accuracy:')print(accuracy_score(y_test, rf.predict(X_test_a2)))print('Test Recall:')print(recall_score(y_test, rf.predict(X_test_a2), average='micro'))print('Train Accuracy:')print(accuracy_score(y_train, rf.predict(X_train_a2)))print('Train Recall:')print(recall_score(y_train, rf.predict(X_train_a2),  average='micro'))###Output:Test Accuracy:0.797068340067343Test Recall:0.769820394303477Train Accuracy:0.818649338720538Train Recall:0.810256789879921
This second run with only the selected features results in a testing score of 80 percent and a training score of 81 percent. Definitely an improvement from our first iteration with a testing score of 80 percent and a training score of 96 percent. We are minimizing the gap between the testing and training scores (model is becoming less overfit).
Boruta is powerful feature selection algorithm which you can implement across most datasets. It can be super helpful in a time crunch as well as datasets containing a large amount of weakly correlated predictor variables. The main disadvantage is the computation time, while a lot of algorithms can be executed in seconds or milliseconds, boruta’s execution time is measured in hours. This can make it extremely difficult to adjust the parameters as each tune will require significant additional time. Boruta may not be the best option for the dataset you’re working with and I’d recommend testing other algorithms as well and comparing the results.
References:
Bhattacharyya, I. (2018, September 18). Feature Selection (Boruta /Light GBM/Chi Square)-Categorical Feature Selection. Retrieved from https://medium.com/@indreshbhattacharyya/feature-selection-categorical-feature-selection-boruta-light-gbm-chi-square-bf47e94e2558
Danielhomola. (2016, February 08). BorutaPy — an all relevant feature selection method. Retrieved from http://danielhomola.com/2015/05/08/borutapy-an-all-relevant-feature-selection-method/
Boruta. (n.d.). Retrieved from https://pypi.org/project/Boruta/