Your Ultimate Data Mining & Machine Learning Cheat Sheet
By: Andre Ye
Tags: Machine Learning, Data Science, AI, Data Analysis, Statistics

Feature Importance

Feature Importance is the process of finding the most important feature to a target. Through PCA, the feature that contains the most information can be found, but feature importance concerns a feature’s impact on the target. A change in an ‘important’ feature will have a large effect on the y-variable, whereas a change in an ‘unimportant’ feature will have little to no effect on the y-variable.

Permutation Importance is a method to evaluate how important a feature is. Several models are trained, each missing one column. The corresponding decrease in model accuracy as a result of the lack of data represents how important the column is to a model’s predictive power. The eli5 library is used for Permutation Importance.

import eli5

from eli5.sklearn import PermutationImportance

model = PermutationImportance(model)

model.fit(X,y)

eli5.show_weights(model, feature_names = X.columns.tolist())

In the data that this Permutation Importance model was trained on, the column lat has the largest impact on the target variable (in this case the house price). Permutation Importance is the best feature to use when deciding which to remove (correlated or redundant features which actually confuse the model, marked by negative permutation importance values) in models for best predictive performance.

SHAP is another method of evaluating feature importance, borrowing from game theory principles in Blackjack to estimate how much value a player can contribute. Unlike permutation importance, SHapley Addative ExPlanations use a more formulaic and calculation-based method towards evaluating feature importance. SHAP requires a tree-based model (Decision Tree, Random Forest) and accommodates both regression and classification.

import shap

explainer = shap.TreeExplainer(model)

shap_values = explainer.shap_values(X)

shap.summary_plot(shap_values, X, plot_type="bar")

PD(P) Plots, or partial dependence plots, are a staple in data mining and analysis, showing how certain values of one feature influence a change in the target variable. Imports required include pdpbox for the dependence plots and matplotlib to display the plots.

from pdpbox import pdp, info_plots

import matplotlib.pyplot as plt

Isolated PDPs: the following code displays the partial dependence plot, where feat_name is the feature within X who will be isolated and compared to the target variable. The second line of code saves the data, whereas the third constructs the canvas to display the plot.

feat_name = 'sqft_living'

pdp_dist = pdp.pdp_isolate(model=model,

dataset=X,

model_features=X.columns,

feature=feat_name)

pdp.pdp_plot(pdp_dist, feat_name)

plt.show()

The partial dependence plot shows the effect of certain values and changes in the number of square feet of living space on the price of a house. Shaded areas represent confidence intervals.

Contour PDPs: Partial dependence plots can also take the form of contour plots, which compare not one isolated variable but the relationship between two isolated variables. The two features that are to be compared are stored in a variable compared_features .

compared_features = ['sqft_living', 'grade'] inter = pdp.pdp_interact(model=model,

dataset=X,

model_features=X.columns,

features=compared_features)

pdp.pdp_interact_plot(pdp_interact_out=inter,

feature_names=compared_features),

plot_type='contour')

plt.show()

The relationship between the two features shows the corresponding price when only considering these two features. Partial dependence plots are chock-full of data analysis and findings, but be conscious of large confidence intervals.