On the role of dummy variables and interactions in linear regression
By: Adrià Luz
Tags: Data Science, Linear Regression, Getting Started, Machine Learning, Statistics

On the role of dummy variables and interactions in linear regression

We’re all familiar with the quintessential example of linear regression: predicting house prices based on house size, number of rooms and bathrooms, and so on. However, we may often want to introduce categorical variables into our model too, such as whether the house has got a swimming pool or its neighbourhood.

One issue with linear regression models is that they can only interpret numerical inputs. Thus, we need a way of translating words like neighbourhood names to numbers that the model can understand.

The most common way of doing this is by creating dummy variables. Let’s say we’re looking at Spanish houses in three main cities and we have a categorical variable which captures the city the house is in. This categorical variable can take on values “Barcelona”, “Madrid”, and “Valencia”, and therefore we need to transform it if we want our model to be able to interpret it. The way we do this is by creating m-1 dummy variables, where m is the total number of unique cities in our dataset (3 in this case). Hence, we would substitute our “city” variable for the two dummy variables below:

Image by author

These dummy variables are very simple. The first one will be equal to 1 if the city is Barcelona — otherwise it will be 0. Likewise, the second will be equal to 1 if and only if the city is Madrid. Can you see why we only needed to add m-1=2 dummy variables to represent all possible cases? If these two dummies are both 0, it must be the case that the city is Valencia. If we naïvely included three dummy variables, we would’ve created a multicollinearity problem for ourselves since the three variables would be perfectly collinear. Note that:

Image by author

where cᵥ represents the dummy variable for the city of Valencia. Hence, we should only create m-1 dummy variables to avoid over-parametrising our model.

Now, let’s look at the famous Iris flower data set that Ronald Fisher introduced in his 1936 paper “The use of multiple measurements in taxonomic problems”. Although he used it to show his linear discriminant and it is popularly used for teaching classification techniques, here we’ll use it to show the importance and interpretation of dummy variables and interactions in multiple linear regression.

The data set consists of 50 samples from each of three species of Iris flowers: Iris setosa, Iris virginica, and Iris versicolor. There are four measurements taken for each flower: sepal length, sepal width, petal length, and petal width.

Photo by Zdeněk Macháček on Unsplash

Let’s assume we’re interested in the relationship between sepal length ( sepalL ) and petal length ( petalL ). Here’s what that relationship looks like:

Image by author

To better understand the relationship between sepal length and petal length we may want to fit the following simple linear regression model:

Image by author

That is, we model the petal length of flower i as a function of its sepal length (plus an intercept term). Fitting this model to our data we find the line of best fit shown below:

Image by author

However, by looking at this line, we can already see how we’re overestimating the petal length of Iris setosa — notice how our model (the line of best fit) is almost in its entirety above the red points. At the same time, we’re also mostly underestimating the petal lengths of the other two species. We can also see this by looking at the standardised residuals of our model:

Image by author

Recall that the residuals are the difference between the observed and predicted values. Therefore, a negative residual implies that the predicted value was higher than the observed value (over-estimation) while a positive residual implies that the predicted value was lower than the observed value (under-estimation). Ideally, we’d like to see the standardised residuals randomly scattered around 0, with no clear patterns. Unfortunately, we can clearly see that this is not the case in the two charts above — we need to do better.

It seems like we could benefit from adding a dummy variable to represent the species of the flower. Hence, in a similar fashion to what we did before with the Spanish cities, we could create two dummy variables:

Image by author

Hence, we would now move from a simple to a multiple linear regression. The new model would be of the following form:

Image by author

Now, note how this will result in three different lines depending on the species of the flower. If the flower is an Iris setosa:

Image by author

If the flower is Iris versicolor:

Image by author

And if the flower is Iris virginica:

Image by author

Notice how Iris virginica is our reference category. When the flower is Iris setosa or versicolor, the dummy variables have the effect of altering the intercept. Since the slopes are not changing, this means that fitting this model will give us three parallel lines — one for each species. Let’s see how that performs:

Image by author

And the standardised residuals:

Image by author

This model is much, much better than the previous one. Allowing for three different intercepts gives our model a lot more flexibility. Now it can represent the three species much better — we can see this in both plots. But we may still be able to do a little bit better. It looks like it could be valuable to also allow for different slopes, particularly when looking at Iris setosa flowers.

In mathematical terms, what that means is that we need to add an interaction between the species dummies and the sepal length. This is our final model:

Image by author

Again, note that this will result in three unique lines depending on the species of the flower. The difference is that this time around, both the intercepts and the slopes will be allowed to be different. If the flower is Iris setosa:

Image by author

If the flower is Iris versicolor:

Image by author

And finally, if the flower is Iris virginica:

Image by author

In this case, we can see how adding interaction terms to our previous model allows the model to provide three lines with both different intercepts and different slopes. Let’s fit this model and see what we get:

Image by author

And the standardised residuals:

Image by author

Now we can see that the our model is fitting the data very well. Look at that residuals plot — it’s almost perfect!

For completeness, here’s a table showing the adjusted R-squared (higher is better) and the residual standard error (lower is better) of the three models: