Multiple Linear Regression Python 101
By: Chuck Utterback
Tags: Getting Started, Predictive Analytics, Deep Dives, Python, Analytics

The chart shows living space “taking off” during the 1970’s in the Seattle area. For homes built since-1970, we see a 32% increase in living square footage, a 9% increase in average bedrooms and a 21% increase in square footage per bedroom. The feature sqft_per_bedroom made the final model cut as a complimentary feature to living_sqft.

Pairplot Association Analysis

Another data profiling technique is to visually inspect intersections of variables for associations. Here I’ve used Seaborn’s pairplot which creates a scatterplot for each combination, leaving a histogram on the diagonals.

Linear with price: Grade, Condition, Bathrooms

Inconclusive or non-linear with price: Year Built, Indicators, Bedrooms

One-Hot Encoding

A common approach is one-hot encoding (OHE) to pivot categorical variables into features. I tested several OHE variables such as condition, grade, bedrooms and bathrooms to see if they performed better than as a continuous variable. I found each of those was better off as a continuous variable.

I ended up with two OHE variables: zipcode and grade_grp (simplifying the residential building groups from 3–13 into 4 categories). I used Pandas .get_dummies() method to pivot the feature to OHE columns, dropping the first class to minimize pure multicollinearity risk.

Step 4 — Correlation Analysis

The next step is looking at correlations with these goals:

Validate which predictor features (X) correlate to target (y)

Check correlations between predictors for multicollinearity

Pearson’s Correlation Coefficient (r) — Target y

Review Pearson correlation coefficients for each feature (X) against the predictor price_log (y). Correlations will provide insight into features that may perform well in a linear regression model. I used these results as well as correlation matrix heat maps as a guide on which features to include.

The four starred features had moderate or strong correlations and made the final OLS model.

Strong correlations (0.6–0.8): sqft_living, grade, sqft_above, zip_psf_median, sqft_living15

Moderate correlations (0.4–0.6): bathrooms, view

Check Multicollinearity

When independent variables are correlated, it indicates that changes in one variable are associated with shifts in another variable. The stronger the correlation, the more difficult it is to change one variable without changing another. — Jim Frost “Regression Analysis”

Reviewing correlations between predictors is a key step to avoid picking features that have strong multicollinearity that may invalidate the model. The code block creates pairs of variables and sorts them by highest correlation.

Highlighting two different scenarios (starred above):

Sqft_living with sqft_above 0.876 correlation — I tried in my Model #3 to incoporate sqft_above but abandoned this due to multicollinearity.

Grade, Sqft_living at 0.76 correlation — even though highly correlated, I was able to include both features without significant issues.

The recommendation here is to be highly aware of predictor correlations using Pearsons or other statistics like Variance Inflation Factor (VIF).

Step 5 — Engineer New Features

The first time through a data set when creating a linear regression model, you may not initially need any new features. But after your first model iteration, potential transformations include:

Add a confounding variable to reduce model bias

Convert a feature to per-capita or express as a rate

Power transform a feature using inverse, log, square root or box-cox

Add a polynomial feature (e.g. X²) to better fit curvature

Add an interaction term to tease out shifts in linear relationships between two predictors

For the King County home price sales predictions, I explored all of the above except polynomials. For my baseline Model #1 — let’s just see how simple untransformed data works first.

Target variable: price (untransformed)

Predictor variables (top 6 correlations): sqft_living, sqft_above, sqft_living15, bathrooms, view, grade (all untransformed)

Yikes!

While Model #1 registered a 0.577 adjusted r-squared and a significant p-value, it had massive skew and multicollinearity as the QQ-plot of overall model residuals show.

We clearly need to look at predictor distributions and consider other features.

Check Normality and Create Log Functions

Given non-normal residuals and skewed price distribution, I created Python functions check_normality and create_log which takes in a dataframe column, creates a log series and displays dual histograms for comparison.

Using these functions, I log transformed the King County price series to price_log. The log series as shown below is more normalized with skew close to 0 and kurtosis of just under 1. Prior to transformation, price had a skew over 4, kurtosis of 35 and nearly 200 observations above the 4th standard deviation. As a guide, ideal normality would target a skew near 0 and kurtosis between 0–6 with 3 being ideal.

Linear regression assumptions do not require that dependent or independent variables have normal distributions, only normal model residuals. I settled on price_log as my dependent variable for model iterations Model #2 through Model #8 since it improved performance and removed residual plots skew.

Create Indicator Features — Renovation and Basement

After data profiling and trying to use the year renovated as a feature, I settled on including categorical indicator features renovation_ind and basement_ind with 1’s representing ‘yes’.

Median Price Per Square Foot By Zip Code

Incorporating zipcode in the model is important since prices vary greatly by geographic area. In Model #6, I first used 70 zip codes as one-hot encoded categorical variables, but abandoned this model as residuals were very poor. Instead I added a feature “median price per square foot” that I applied to the data by zipcode. The box-cox transformed version of this feature ended up significant so was included in the final Model #8.

Miles From Seattle City Center

This feature used each home’s latitude and longitude to measure the distance to the center of Seattle. This feature was actually created before the above zip-code price-per-square-foot based on the idea that prices are generally higher closer to urban population centers. Initially this feature was very significant, but lost most of it’s influence once included alongside the zip-code median price-per-square-foot.

Step 6 — Select Features

The final step of data preparation is to chose which features to include in the next iteration of the regression model. This experimentation process is informed by your completed data analysis and prior model results.

My approach is to make a copy of train and test data containing only the features selected so I have a set of dataframes and model results dedicated to each model iteration.

I saved 8 model iterations in my notebook but ran at least twice that many experimenting on features, transformations, scaling and so forth. Four of my iterations I considered statistically valid while four I abandoned due to performance, residuals or multicollinearity. I stopped at Model #8 which contained the following features.