Linear Regression — Feature step selection based on p-value for your model
By: Datascience Sue Mir
Tags: Python, Data Science, Linear Regression, Stepwise Selection, Scikit Learn

Working on data with many features and training models may take way too long when iterating through all the combinations for stepwise selection.

sklearn (nor Python)seem to have a forward selection algorithm/library(yet). However, it(sklearn) does provide recursive feature elimination, which is a greedy feature elimination algorithm similar to sequential backward selection. See the documentation here:

http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html

If you still want vanilla stepwise regression to determine the most important features for a model by using recursive feature elimination, it is easier to base it on statsmodels , since this package calculates p-values for you.

The parameters or coefficients we’re calculating have a p-value or significance attached to them. The p-value represents the probability that the coefficient is actually zero.

In stepwise selection, start with an empty model (which only includes the intercept), and each time, the variable that has an associated parameter estimate with the lowest p-value is added to the model (forward step). After adding each new variable in the model, the algorithm will look at the p-values of all the other parameter estimates which were added to the model previously, and remove them if the p-value exceeds a certain value (backward step). The algorithm stops when no variables can be added or removed given the threshold values.

This example of code was taken for use from stackexchange: https://datascience.stackexchange.com/questions/937/does-scikit-learn-have-forward-selection-stepwise-regression-algorithm

Running the above function to list the features for your model: (you will need to pass the target column that you would like to calculate the best fit features for, in this example below my target is price)

Heres the Code if you wish to cut and paste:

— — — — — — — — — — — — — — — — — — — — — — — — — —

# Perform stepwise selection

import statsmodels.api as sm

def stepwise_selection(X, y,

initial_list=[],

threshold_in=0.01,

threshold_out = 0.05,

verbose=True):

“”” Perform a forward-backward feature selection

based on p-value from statsmodels.api.OLS

Arguments:

X — pandas.DataFrame with candidate features

y — list-like with the target

initial_list — list of features to start with (column names of X)

threshold_in — include a feature if its p-value < threshold_in

threshold_out — exclude a feature if its p-value > threshold_out

verbose — whether to print the sequence of inclusions and exclusions

Returns: list of selected features

Always set threshold_in < threshold_out to avoid infinite looping.

See https://en.wikipedia.org/wiki/Stepwise_regression for the details

“””

included = list(initial_list)

while True:

changed=False

# forward step

excluded = list(set(X.columns)-set(included))

new_pval = pd.Series(index=excluded)

for new_column in excluded:

model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included+[new_column]]))).fit()

new_pval[new_column] = model.pvalues[new_column]

best_pval = new_pval.min()

if best_pval < threshold_in:

best_feature = new_pval.idxmin()

included.append(best_feature)

changed=True

if verbose:

print(‘Add {:30} with p-value {:.6}’.format(best_feature, best_pval))

# backward step

model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included]))).fit()

# use all coefs except intercept

pvalues = model.pvalues.iloc[1:]

worst_pval = pvalues.max() # null if pvalues is empty

if worst_pval > threshold_out:

changed=True

worst_feature = pvalues.argmax()

included.remove(worst_feature)

if verbose:

print(‘Drop {:30} with p-value {:.6}’.format(worst_feature, worst_pval))

if not changed:

break

return included

— — — — — — — — — — — — — — — — — — — — — — — — — —

def stepwise_selection_results():

“”” Call stepwise_selection() and print the results forward-backward feature selection

Arguments:

Returns: none

“””

X = df

X = df.drop(‘price’, axis=1)

y = pd.DataFrame(df, columns=[‘price’])

result = stepwise_selection(X, y, verbose = True)

print(‘resulting features:’)

print(result)



stepwise_selection_results()

— — — — — — — — — — — — — — — — — — — — — — — — — —