Data Science Debrief: My First Multiple Regression Analysis + 4 Questions About Stepwise Addition
By: Marvin Lee
Tags: Regression Analysis, Data Science, Tips, Big Data Analysis, Feature Engineering

What is stepwise addition?

In my words: it’s an automated process for adding and dropping independent variables based on an underlying test

Something more official (from Investopedia): Stepwise regression is the step-by-step iterative construction of a regression model that involves the selection of independent variables to be used in a final model. It involves adding or removing potential explanatory variables in succession and testing for statistical significance after each iteration.

There are three main approaches (from Wikipedia):

Forward selection — start with no variables, perform statistical tests to find the variable giving the most statistically significant improvement to the model, and repeating this until there are no more variables that improve the model

— start with no variables, perform statistical tests to find the variable giving the most statistically significant improvement to the model, and repeating this until there are no more variables that improve the model Backward elimination — start with all variables, choosing to delete a variable based on a certain criterion, and repeating this until no more variables can be deleted in a way that brings on statistically significant loss to the model

— start with all variables, choosing to delete a variable based on a certain criterion, and repeating this until no more variables can be deleted in a way that brings on statistically significant loss to the model Bidirectional elimination — pretty much a combination of both Forward and Backward

The third approach is the one I took.

Check out the repository for this project if you’d like!

How valuable is stepwise addition to regression modeling?

Well, what I can say here is that I saw value in being able to use an automated process for choosing variables. This is especially because I one-hot coded a number of times and so had a ton of dummy variables.

One consequence of having many dummy variables is that when one variable is removed — due to a p-value greater than 0.05, for example — all of the other variables adjust somehow to the absence of the variable dropped, and so the p-values of the variables remaining can then change!

Moreover, with so many independent variables, I didn’t want to drop a variable without having a solid reason.

This is why I imagine the stepwise method is so often used. It’s an industry-accepted way to save time.

How does stepwise addition impact real world solutions?

In a world full of Big Data, the number of potential explanatory variables can be large, and so from this angle, it perhaps helps to make problem-solving a more efficient endeavor?

But, on the other hand, there is concern that stepwise addition is causing more harm than good. In fact, this paper from the Journal of Big Data uses a series of Monte Carlo simulations to demonstrate that stepwise regression is a poor solution to a surfeit of variables. The author’s claim is that, in fact, the larger the number of potential explanatory variables, the more likely stepwise regression is to be misleading.

What are some important things to remember when it comes to using stepwise addition during regression modeling?

Here is what I’ll be keeping in mind going forward, and I hope this helps one of you reading this as well:

Consider stepwise addition as a means, not as the end

After doing some stepwise addition my model had no independent variables with a p-value above 0.05, however, I believed there were still too many variables and decided to continue iterating and trying other methods, so that I could hopefully get to the goal of having a solid R-squared value with as few variables as possible. I also sometimes asked myself why a particular variable was left out, such as the feature I’d engineered (it’s a feature that quantifies the ratio of a home’s living space to its lot size). I think that, without a clear business case or objective in mind, and without proper domain expertise, it could be easy to overfit the stepwise method into a data science practice.

If you log-transform or scale one or many of your variables, include them both in the stepwise addition process

This is what I ended up doing on one occasion. I chose to scale a couple log-transformed variables, thinking that the scaling would improve the variables’ performance when running the models. However, stepwise addition revealed that my former variables had a better “fit” and so eventually I dropped the scaled variables. I also had two variables which had strong multicollinearity and the stepwise method showed how one of these variables’ dummy variables consistently was better “fit” to the model. So, I’ve found stepwise addition useful in this regard.