Feature Selection and Dimensionality Reduction | by Tara Boyle | Towards Data Science

According to wikipedia, “feature selection is the process of selecting a subset of relevant features for use in model construction” or in other words, the selection of the most important features.
In normal circumstances, domain knowledge plays an important role and we could select features we feel would be the most important. For example, in predicting home prices the number of bedrooms and square footage are often considered important.
Unfortunately, here in the Don’t Overfit II competition, the use of domain knowledge is impossible as we have a binary target and 300 continuous variables “of mysterious origin” forcing us to try automatic feature selection techniques.
The full notebook can be found here.
Often, feature selection and dimensionality reduction are grouped together (like here in this article). While both methods are used for reducing the number of features in a dataset, there is an important difference.
Feature selection is simply selecting and excluding given features without changing them.
Dimensionality reduction transforms features into a lower dimension.
In this article we will explore the following feature selection and dimensionality reduction techniques:
Remove features with missing values
Remove features with low variance
Remove highly correlated features
Univariate feature selection
Recursive feature elimination
Feature selection using SelectFromModel
PCA
We’ll use logistic regression as our baseline model. We first split into test and train sets and scale the data:
# prepare for modelingX_train_df = train.drop(['id', 'target'], axis=1)y_train = train['target']# scaling datascaler = StandardScaler()X_train = scaler.fit_transform(X_train_df)lr = LogisticRegression(solver='liblinear')lr_scores = cross_val_score(lr,                            X_train,                            y_train,                            cv=5,                            scoring='roc_auc')print('LR Scores: ', lr_scores)LR Scores:  [0.80729167 0.71875    0.734375   0.80034722 0.66319444]
We can see the model is overfitting from the variation in cross validation scores. We can attempt to improve these scores through feature selection.
Checking for missing values is a good first step in any machine learning problem. We can then remove columns exceeding a threshold we define.
# check missing valuestrain.isnull().any().any()False
Unfortunately for our dimensionality reduction efforts, this dataset has zero missing values.
In sklearn’s feature selection module we find VarianceThreshold. It removes all features whose variance doesn’t meet some threshold. By default it removes features with zero variance or features that have the same value for all samples.
from sklearn import feature_selectionsel = feature_selection.VarianceThreshold()train_variance = sel.fit_transform(train)train_variance.shape(250, 302)
The competition description stated that our features are all continuous. We can see from above there are no features with the same value in all columns, so we have no features to remove here.
We can always revisit this technique and consider removing features with low variance.
Features that are highly correlated or co-linear can cause overfitting.
When a pair of variables are highly correlated we can remove one to reduce dimensionality without much loss of information. Which one should we keep? The one with a higher correlation to the target.
Let’s explore correlations among our features:
# find correlations to targetcorr_matrix = train.corr().abs()print(corr_matrix['target'].sort_values(ascending=False).head(10))target    1.00000033        0.37360865        0.293846217       0.207215117       0.19749691        0.19253624        0.173096295       0.17050173        0.167557183       0.164146
Here we see the features that are most highly correlated with our target variable. Feature 33 has the highest correlation to the target, but with a correlation value of only 0.37, it is only weakly correlated.
We can also check the correlation of features to other features. Below we can visualize a correlation matrix. It looks like none of our features are very highly correlated.
Let’s try to drop features with a correlation value greater than 0.5:
# Find index of feature columns with high correlationto_drop = [column for column in matrix.columns if any(matrix[column] > 0.50)]print('Columns to drop: ' , (len(to_drop)))Columns to drop: 0
We have no columns to drop using highly correlated features. Let’s continue to explore other strategies.
Univariate feature selection works by selecting the best features based on univariate statistical tests.
We can use sklearn’s SelectKBest to select a number of features to keep. This method uses statistical tests to select features having the highest correlation to the target. Here we will keep the top 100 features.
from sklearn.feature_selection import SelectKBest, f_classif# feature extractionk_best = SelectKBest(score_func=f_classif, k=100)# fit on train setfit = k_best.fit(X_train, y_train)# transform train setunivariate_features = fit.transform(X_train)
Recursive feature selection works by eliminating the least important features. It continues recursively until the specified number of features is reached. Recursive elimination can be used with any model that assigns weights to features, either through coef_ or feature_importances_
Here we will use Random Forest to select the 100 best features:
from sklearn.feature_selection import RFE# feature extractionrfe = RFE(rfc, n_features_to_select=100)# fit on train setfit = rfe.fit(X_train, y_train)# transform train setrecursive_features = fit.transform(X_train)
Like recursive feature selection, sklearn’s SelectFromModel is used with any estimator that has a coef_ or feature_importances_ attribute. It removes features with values below a set threshold.
from sklearn.feature_selection import SelectFromModelfrom sklearn.ensemble import RandomForestClassifier# define modelrfc = RandomForestClassifier(n_estimators=100)# feature extractionselect_model = feature_selection.SelectFromModel(rfc)# fit on train setfit = select_model.fit(X_train, y_train)# transform train setmodel_features = fit.transform(X_train)
PCA (Principle Component Analysis) is a dimensionality reduction technique that projects the data into a lower dimensional space.
While there are many effective dimensionality reduction techniques, PCA is the only example we will explore here.
PCA can be useful in many situations, but especially in cases with excessive multicollinearity or explanation of predictors is not a priority.
Here we will apply PCA and keep 90% of the variance:
from sklearn.decomposition import PCA# pca - keep 90% of variancepca = PCA(0.90)principal_components = pca.fit_transform(X_train)principal_df = pd.DataFrame(data = principal_components)print(principal_df.shape)(250, 139)
We can see that we are left with 139 features that explain 90% of the variance in our data.
Feature selection is an important part of any machine learning process. Here we explored several methods for feature selection and dimensionality reduction that can aid in improving model performance.