How to Find the Best Predictors for ML Algorithms | Towards Data Science

So now, you have your data cleaned and are ready to feed it to machine learning (ML) algorithms. But what if you have a substantial number of input features? Are they all useful and predictive enough for model learning? Will it make your model less interpretive if you use all the features? Having a large number of predictors are also likely to increase the development and model training time, while at the same time utilizing a large amount of system memory.
Feature selection techniques aim to systematically select the best subset of input features for model training to predict the target variable. Do not confuse feature selection with other dimensionality reduction techniques (e.g., PCA or use of information theory). Feature selection techniques do not amend the original semantics of the predictors, but simply select a subset of them, thereby resulting in a more interpretable model.
According to Sayes et al.1: “The objectives of feature selection are manifold, the most important ones being:
to avoid overfitting and improve model performance, i.e., prediction performance in the case of supervised classification and better cluster detection in the case of clustering,to provide faster and more cost-effective models, andto gain a deeper insight into the underlying processes that generated the data.”
to avoid overfitting and improve model performance, i.e., prediction performance in the case of supervised classification and better cluster detection in the case of clustering,
to provide faster and more cost-effective models, and
to gain a deeper insight into the underlying processes that generated the data.”
Although feature selection is applicable for both supervised and unsupervised learning, I will focus here on supervised feature selection, where the target variable is known beforehand.
We can define three broad categories in the context of supervised feature selection:
Embedded methods: Sometimes also called intrinsic methods, these techniques are built-in into the model training phase of a classification or regression model. Examples include penalized regression models (e.g., Lasso) and tree-based models. Since such embedded methods are an integral part of model training, we will not be going over these in this post.
Filter methods: These techniques look at the intrinsic properties of features and use statistical techniques to evaluate the relationship between a predictor and the target variable. The subset of best ranked or scored features is then used for model training. We will go over these various statistical techniques in this post.
Wrapper methods: These methods utilize ML algorithms as part of the feature evaluation process to identify and select the best subset of features iteratively and according to a specific performance metric. For example, scikit-learn’s Recursive Feature Elimination (RFE) class that we will go over later in this post.
Filter feature selection techniques utilize various statistical tests and measures to infer the relationship between input and target variables before selecting the best subset of input features.
Note that these statistical measures are univariate, and accordingly, the correlations between various pairs of input features are not accounted for. It might be a good idea to first start with a correlation matrix to weed out redundant correlated features. Alternatively, specific advanced multivariate filter techniques can be utilized, for example, Correlation-based Feature Selection (CFS), Markov Blanket Filter (MBF), and Fast Correlation-based Feature Selection (FCBF). However, there are currently no widely used and supported python packages out there that support these advanced multivariate feature selection techniques.
Two conventional statistical feature techniques that can be used for categorical features in a classification problem are:
Chi-Squared Test of IndependenceMutual Information
Chi-Squared Test of Independence
Mutual Information
The Chi-Squared test is used to determine the extent of relationship or dependence between two categorical variables — in our case, one categorical input feature, and the other, a categorical target variable.
In probability and information theories, the Mutual Information of two random variables measures the level of independence between them. Higher values mean higher dependency.
For categorical features encoded as integers (e.g., through OrdinalEncoder), scikit-learn’s SelectKBest class is used in conjunction with either the chi2 or mutual_info_classif functions to identify and select the top k most relevant features. The higher the score returned by SelectKBest, the better that feature’s predictive power will be. The shortlisted features are then fed into the ML algorithm for model development.
A complete working example showing both chi2 and mutual_info_classif follows:
Note that SelectKBest can also be used as part of a Pipeline for cross-validation purposes. However, depending upon the cardinality of your input features, using OrdinalEncoder in a Pipeline can sometimes not be feasible when the model comes across new value in the test set that was not available in the train set and was therefore not used for encoding. A workaround in such situations could be to run train_test_split multiple times with different test_size and random_state, and take an average of the selected validation metric.
For object type categorical features that have not been numerically encoded (maybe because they will be one-hot encoded or converted to dummy variables at a later stage), SciPy’s chi2_contingency class is used to calculate the chi statistic and p values of each pair of categorical features and the target variable. Features with the lowest p values are then shortlisted for modeling.
A complete working example follows:
Two conventional statistical feature techniques that can be used for numerical features in a classification problem are:
ANOVA F-StatisticMutual Information (as explained above)
ANOVA F-Statistic
Mutual Information (as explained above)
The Analysis of Variance (ANOVA) F-statistic calculates the ratio of variances of the means of two or more samples of data. The higher this ratio between a numerical input feature and a categorical target feature, the lower the independence between the two and more likely to be useful for model training. Refer to this excellent article for details on the ANOVA F-statistic.
ANOVA F-statistic for feature selection is similarly implemented in Python as a chi-squared test of independence through sci-kit learn’s f_classif function. Given that we are dealing with numerical features this time around, we can easily implement cross-validation through pipelines as demonstrated below:
Ah, this is the easiest type of feature selection: identifying the most suitable numeric input feature for a numerical target variable (regression problem). Possible techniques include:
Correlation statisticsMutual Information (same as explained earlier, with just a different score_func to be used: mutual_info_regression)
Correlation statistics
Mutual Information (same as explained earlier, with just a different score_func to be used: mutual_info_regression)
Correlation measures the degree of change in one variable as a result of another. Perhaps the best-known correlation measure is Pearson’s correlation that assumes a Gaussian distribution of the data. Correlation coefficients range between -1 (perfect negative correlation) and 1 (perfect positive correlation) with 0 representing no relationship between the variables whatsoever.
In addition to plotting the correlation matrix, automatic correlation-based feature selection can be implemented similar to a chi-squared test of independence or ANOVA F-statistic, using sci-kit learn’s f_regression function in conjunction with the SelectKBest class. Further, just like the ANOVA F-statistic function, we can easily implement cross-validation through pipelines as demonstrated below:
Recursive Feature Elimination (RFE) from scikit-learn is the most widely used wrapper feature selection method in practice. RFE is feature-type agnostic that iteratively selects the best number of features through a given supervised learning model (estimator).
From scikit-learn documentation: “First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through a coef_ attribute or through a feature_importances_ attribute. Then, the least important features are pruned from current set of features. That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.”
As evident from the above, RFE can only be used with an algorithm that has either a coef_ or feature_importances_ attribute to evaluate feature importance. Since RFE is a wrapper feature selection technique, it can use any given estimator (that can be different from the algorithm that is applied to the selected features) to identify the most suitable features.
Given that RFE can be used for both categorical and numerical features; its implementation is similar in case of both classification and regression problems.
The main parameter of RFE that requires tuning is n_features_to_select. A practical demonstration to identify the best number of features based on RFE using DecisionTreeClassifier for both feature selection and the classification algorithm is as below:
The above code snippet will print out the mean and STD of Accuracy scores for each number of features utilized in RFE, something like below:
>2: 0.710>3: 0.815>4: 0.872>5: 0.884>6: 0.891>7: 0.888>8: 0.888>9: 0.884
Based purely on the Accuracy score, selecting six features appears to be ideal in this scenario.
However, how do we know that DecisionTreeClassifier is the best algorithm to be used within RFE? What if we would like to evaluate various algorithms with RFE? RFECV comes to our help here.
The primary purpose of RFECV is to select the best number of features through cross-validation automatically. Since the ideal number of features will be auto-selected by RFECV (therefore, not requiring the for loop in the last code snippet), we can efficiently train and assess multiple models for RFECV feature selection. The model with the best validation metric can then be used moving forward for making predictions.
Let us look at a practical example now:
The above code snippet will print out the mean and STD of Accuracy scores for each of the model utilized in RFECV, something like below:
>LR: 0.891>DT: 0.882>RF: 0.888>XGB: 0.886
The simple logistic regression model wins for wrapper feature selection. We can then use this for making predictions:
# create pipelinerfecv = RFECV(estimator = LogisticRegression(), cv = 10, scoring =     'accuracy')model = DecisionTreeClassifier()pipeline = Pipeline(steps=[('features', rfecv), ('model', model)])# fit the model on all available datapipeline.fit(X, y)# make a prediction for one exampledata = #load or define any new data unseen data that you want to make predictions uponyhat = pipeline.predict(data)print('Predicted: %.3f' % (yhat))
The majority of ML algorithms have a built-in attribute to view the relative importance of the features used during model learning. Some examples include:
Linear Regression: coef_ gives the list of coefficients. Higher the coefficient, the more of an impact that input feature has on the target variable
Logistic Regression: coef_ provides the list of coefficients, to be interpreted the same as that in Linear Regression
Decision Trees and Random Forests: feature_importanes_ — higher the better
That is it from me this time. Feel free to reach out to me if you would like to discuss anything related to data analytics, traditional machine learning, or credit analysis.
Till next time, rock on!
Inspired by Dr. Jason Brownlee of Machine Learning Mastery
[1] Saeys Y, Inaki I, Larranaga P. A Review of Feature SelectionTechniques in Bioinformatics. Bioinformatics. 2007; 23(19): 2507-2517.