Why Interacting Variables Are Multiplied Together in Linear Models
Tags: Math, Mathematics, Data Science, Statistics, Linear Models

Why Interacting Variables Are Multiplied Together in Linear Models ___ Follow Jun 14 · 4 min read

Introduction

In this article, I will explain why when building a linear model, we add an x₁x₂ term if we think the variables x₁ and x₂ interact and conclude with a principled method to add interaction terms.

The content of this article is based on Chapter 8 of [1].

I assume the reader has a basic understanding of how linear models work.

A No Interaction Model

Let’s begin by building a model with no interaction terms.

Suppose we would like to model y as a function of x₁ and x₂. Then, a linear model that describes this relationship is:

Figure 1: A linear model with no interaction terms

We call α the intercept and β₁and β₂ the slope of x₁ and x₂ respectively.

A Model With Variable Slopes

Suppose we think that the slope of x₁ actually depends on x₂. How can we incorporate this belief into the model?

Since we are working with linear models, one way to do it is to model β₁ as another linear model that is a function of x₂, like so:

Figure 2: The effect of assuming the slope of x1 depends on x2

Substituting the equation in Figure 2 into Figure 1, we end up fitting a model that looks like:

Figure 3: Fitting a linear model that assumes the effect of x₁ depends on the value of x₂

The model in Figure 3 is exactly the same as the one in Figure 1 except that it has an extra term, bx₁x₂.

This way of building a linear model shows that the interaction term is a natural consequence of the assumption that the effect of x₁ on y is linearly dependent on the current value of x₂.

x₁ Depending On x₂ Is The Same As x₂ Depending On x₁

The previous section built on the assumption that the effect of x₁ depends on the current value of x₂. What if we had made the assumption in the other direction i.e. the effect of x₂ depends on the current value of x₁?

Using the same approach, the assumption means that:

Figure 4: Assuming the slope of x₂ depends on x₁

We end up with fitting a model that looks like:

Figure 5: Fitting a linear model that assumes the effect of x₂ depends on the value of x₁

Notice that the model in Figure 5 is the same as the one in Figure 4 (they only differ in the names assigned to the coefficients).

This is a point worth highlighting to emphasize that adding an x₁x₂ term does not say anything about the causal relationship between x₁ and x₂.

Interactions Involving More Than Two Variables

This approach of adding interaction terms shows that interactions involving more than two variables can be had by recursively applying the assumption that the slope of a variable is dependent on another variable.

For example, suppose we want to model the relationship between y and x₁, x₂, and x₃ using a linear model.

Now, if we assume that the slope of x₁ depends on x₂, we have:

Figure 6: Assume the slope of x₁ depends on x₂

and if we further assume that the slope of x₂ in Figure 6 is dependent on x₃, we get:

Figure 7: Assume the effect of x₂ on β₁ also depends on x₃

So, given:

Figure 8: Modelling y as a function of x₁, x₂ and x₃

we end up with:

Figure 9: A model assuming the effect of x₁ depends on x₂ which depends on x₃

Notice the presence of the 3-way interaction term (the rightmost term) in Figure 9.

What if we had assumed a different order i.e. slope of x₁ depends on x₃ which depends on x₂? Then, given:

Figure 10: Same assumptions as in Figure 8 except different order

We will get:

Figure 11: A model based on the assumptions in Figure 10

Notice that the model in Figure 11 and Figure 9 are different. They differ in the 5th term.

Conclusion

This article has shown that interaction terms can be interpreted as putting in the assumption that the slope of a particular variable is dependent on the value of another variable.

Using this approach, we have a systematic way of using our domain knowledge to intelligently add interaction terms compared to tacking on all possible combinations of variables in our dataset. The latter approach could lead to models that overfit and/or give erroneous causal inference.

References

[1] Statistical Rethinking. McElreath. 2020