Multiple linear regression with interactions. | Towards Data Science

We all had some sort of experience with linear regression. It’s one of the most used regression techniques used. Why? Because it is simple to explain and it is easy to implement. But what happens when you have more than one variable? How can you deal with this increased complexity and still use an easy to understand regression like this? And what happen if the system is even more complicated? Let’s imagine when you have an interaction between two variables.
Here is where multiple linear regression kicks in and we will see how to deal with interactions using some handy libraries in python. Finally we will try to deal with the same problem also with symbolic regression and we will enjoy the benefits that come with it!
If you want to have a refresh on linear regression there are plenty of resources available and I also wrote a brief introduction with coding. What about symbolic regression? In this article we will be using gplearn. See its documentation for more informations or, if you like, see my other article about how to use it with complex functions in python here.
We will explore two use cases of regression. In in the first case we will just have four variables (x1 to x4) which adds up plus some predetermined interactions: x1*x2, x3*x2 and x4*x2.
Note that in our dataset “out_df” we don’t have the interactions terms. What we will be doing will try to discover those relationships with our tools. This is how the variables look like when we plot them with seaborn, using x4 as hue (figure 1):
The y of the second case (figure 2) is given by:
y_true = x1+x2+x3+x4+ (x1*x2)*x2 - x3*x2 + x4*x2*x3*x2 + x1**2
Pretty complex sceneario!
The first step is to have a better understanding of the relationships so we will try our standard approach and fit a multiple linear regression to this dataset. We will be using statsmodels for that. In figure 3 we have the OLS regressions results.
import statsmodels.api as sm Xb = sm.add_constant(out_df[['x1','x2','x3','x4']])mod = sm.OLS(y_true, Xb)res = mod.fit()res.summary()
Ouch, this is clearly not the result we were hoping for. R2 is just 0.567 and moreover I am surprised to see that P value for x1 and x4 is incredibly high. We need some different strategy.
What we can do is to import a python library called PolynomialFeatures from sklearn which will generate polynomial and interaction features. For example, if an input sample is two dimensional and of the form [a, b], the degree-2 polynomial features are [1, a, b, a2, ab, b2].
from sklearn.preprocessing import PolynomialFeaturesimport scipy.specialpoly = PolynomialFeatures(interaction_only=True)X_tr = poly.fit_transform(Xb)Xt = pd.concat([Xb,pd.DataFrame(X_tr,columns=poly.get_feature_names()).drop([‘1’,’x0',’x1',’x2',’x3',’x4'],1)],1)
With “interaction_only=True” only interaction features are produced: features that are products of at most degree distinct input features (so not x[1] ** 2, x[0] * x[2] ** 3, etc.). The default degree parameter is 2.
With the same code as before, but using Xt now, yields the results below.
mod = sm.OLS(y_true, Xt)res = mod.fit()res.summary()
Now R2 in Figure 4 is 1 which is perfect. Too perfect to be good? In fact there are a lot of interaction terms in the summary statistics. Some that we did not even be aware of. Our equation is of the kind of: y = x1+05*x2+2*x3+x4+ x1*x2 — x3*x2 + x4*x2 So our fit introduces interactions that we didn’t explicitly use in our function. Even if we remove those with high p-value (x1 x4), we are left with a complex scenario. This might be a problem for generalization. We can exploit genetic programming to give us some advice here.
With genetic programming we are basically telling the system to do its best to find relationships in our data in an analytical form. If you read the other tutorial some functions I will call here will be clearer. However what we basically want to do is to import SymbolicRegressor from gplearn.genetic and we will use sympy to pretty formatting our equations. Since we are at it, we will also import RandomForest and DecisionTree regressors to compare the results between all those tools later on. Below the code to get it working:
The converter dictionary is there to help us map the equation with its corrispondent python function to let simpy do its work. We also do train_test split of our data so that we will compare our predictions on the test data alone. We defined a function set in which we use standard functions from gplearn’s set. At the 40th generation the code stops and we see that R2 is almost 1, while the formula generated is now pretty easy to read.
If you compare it with the formula we actually used you will see that its a close match, refactoring our formula becomes:
y = -x3 (x2–2) + x2 (x1+x4+ 0.5)+x1+x4
All algorithms performed good on this work: here are the R2.
statsmodels OLS with polynomial features 1.0, random forest 0.9964436147653762, decision tree 0.9939005077996459, gplearn regression 0.9999946996993035
In this case the relationship is more complex as the interaction order is increased:
X = np.column_stack((x1, x2, x3, x4))y_true = x1+x2+x3+x4+ (x1*x2)*x2 - x3*x2 + x4*x2*x3*x2 + x1**2out_df['y'] = y_true
We do basically the same steps as in the first case, but here we already start with polynomial features:
poly = PolynomialFeatures(interaction_only=True)X_tr = poly.fit_transform(out_df.drop('y',1))Xt = pd.concat([out_df.drop('y',1),pd.DataFrame(X_tr,columns=poly.get_feature_names()).drop(['1','x0','x1','x2','x3'],1)],1)Xt = sm.add_constant(Xt)mod = sm.OLS(y_true, Xt)res = mod.fit()res.summary()
In this scenario our approach is not rewarding anymore. It is clear that we don’t have the correct predictors in our dataset. We could use polynomialfeatures to investigate higher orders of interactions but the dimensionality will likely increase too much and we will be left with no much more knowledge then before. Besides, if you had a real dataset and you did not know the formula of the target, would you increase the interactions order? I guess not!
In the code below we again fit and predict our dataset with decision tree and random forest algorithms but also employ gplearn.
X_train, X_test, y_train, y_test = train_test_split(out_df.drop('y',1), y, test_size=0.30, random_state=42)est_tree = DecisionTreeRegressor(max_depth=5)est_tree.fit(X_train, y_train)est_rf = RandomForestRegressor(n_estimators=100,max_depth=5)est_rf.fit(X_train, y_train)y_gp = est_gp.predict(X_test)score_gp = est_gp.score(X_test, y_test)y_tree = est_tree.predict(X_test)score_tree = est_tree.score(X_test, y_test)y_rf = est_rf.predict(X_test)score_rf = est_rf.score(X_test, y_test)y_sm = res.predict(Xt)est_gp.fit(X_train, y_train)print('R2:',est_gp.score(X_test,y_test))next_e = sympify((est_gp._program), locals=converter)next_e
The result is incredible: again after 40 generations we are left with an incredibly high R2 and even better a simple analytical equation.
The original formula is like this:
So we see that there are indeed differences on the terms which involves x1 and its interactions. While the terms which don’t depend on it are perfectly there. Neverthless, if compared with the polynomialfeatures approach, we’re dealing with a much less complicated formula here.
What is the error of the different systems? Well for gplearn it is incredibly low if compared with other. In figure 8 the error in the y-coordinate versus the actual y is reported. While the x axis is shared, you can notice how different the y axis become. The maximum error with GPlearn is around 4 while other methods can show spikes up to 1000.
In the first part of this article we saw how to deal with multiple linear regression in the presence of interactions. We used statsmodels OLS for multiple linear regression and sklearn polynomialfeatures to generate interactions. We then approached the same problem with a different class of algorithm, namely genetic programming, which is easy to import and implement and gives an analytical expression.
In the second part we saw that when things get messy, we are left with some uncertainty using standard tools, even those from traditional machine learning. However, this class of problems is easier to face with the use of gplearn. With this library we were given an analytical formula for our problem directly.
[1] statsmodels[2] sklearn polynomial features[3] gplearn