What is Model Complexity? Compare Linear Regression to Decision Trees to Random Forests
By: Susan Maina
Tags: Machine Learning, Algorithms, Data Science, Data, Programming

A machine learning model is a system that learns the relationship between the input (independent) features and the target (dependent) feature of a dataset to be useful in making predictions in the future. To test the effectiveness of the model, a completely new similar dataset is introduced which only contains the input features, and the model should predict the target variable based on the insights it gained during training. An appropriate performance metric is then used to compare how good the predicted values are compared to the actual target values. In this article, we are going to test the effectiveness of 3 popular models that vary in complexity;

Linear regression Decision Tree Random Forest

We are then going to check the their performances using two metric systems: Mean squared error, and Mean absolute error.

To accomplish the above we need a dataset, and we are going to create a brand new one ourselves. To understand why I created my own dataset, let us first understand what machine learning is. Machine learning is the process of identifying patterns in data. You can think of a pattern as the ‘way that things work’. Machine learning does this by understanding the relationship between the features (X) and the label (y). Some patterns are simple for example high temperatures leading to increased ice cream sales is a simple linear relationship, while others are complex such as details about a house and how they affect the house price, and this is where machine learning shines. In my case, I created a dataset where the relationship between the X and y features is a cosine curve.

Image by author

Step 1: Dataset simulation

In general, everything has a certain way in which it works and a great model should reveal this pattern. In our case, the ‘true state’ is the cosine curve which is y=cos(x). We will then add some noise to simulate ‘real state’ values, since the real world situations are never perfect.

The first step is to initialize our x values. I used np.linspace(start, end, length). We want our axis to start at 0, end at 2*pi (which is 6.3), and have 100 evenly spaced steps.

x = np.linspace(0, 2*np.pi, 100)

To get the true cosine curve above, I used y=np.cos(x). However, we will add noise in the data all around the cos curve to create our simulated y values.

true_y = np.cos(x)

To create the noise, I used np.random.normal(mean,std,length). This creates normally distributed random values which are centered around the mean, and only vary from the mean by the std(standard deviation). I used 0 for the mean and 0.5 for the std, so the noise values will be between -0.5 to 0.5. I set a random seed of 567 so that the generated sample noise was reproducible, otherwise every time the code was run, new data would be generated.

np.random.seed(567)

noise = np.random.normal(0, 0.5, 100)

The last step was to add the true y mapping to the noise to create our target variable y.

y = np.cos(x) + noise

Now our values are ready and we will create a dataframe to hold the x and y values. We will call this dataframe train, since this will be the training set to train our models on.

train = pd.DataFrame({'x':x, 'y':y})

Now we can plot both the cosine curve (true relationship) and our simulated train data.

plt.scatter(train.x, train.y)

plt.plot(train.x, np.cos(train.x), color='k')

plt.xlabel('x')

plt.ylabel('y')

plt.show()

Image by author

Test set

We will also simulate a test set. This will be important for testing the effectiveness of the model, by checking how well it can predict y values using unseen x values from the test set. We create the test set exactly the same way as the train, except for the noise which we will create using a different random seed of 765.

x = np.linspace(0, 2*np.pi, 100) np.random.seed(765)

noise = np.random.normal(0, 0.5, 100) y = np.cos(x)+noise test = pd.DataFrame({'x':x, 'y':y})

Let us plot the train (blue dots) and test (red dots) sets together.

Plot of train and test simulated data by author

Step 2: Model creation

Simple Linear regression

This model fits a straight line through the data. y^=β0+β1x is the formula for representing a simple linear regression model; β0 is the y intercept and β1 is the coefficient for x, or the slope of the line. y^ is the predicted value. The intercept and the coefficient are learnt from the data by the model. Therefore the model’s task is to get the best estimates of β0 and β1 that represent the true patterns in the data.

I used sci-kit learn to implement the model. First we import the library then initialize an instance of the model.

from sklearn.linear_model import LinearRegression

basic_linear_model = LinearRegression()

Next we separate our features (x) from the target variable (y) in the dataset. I used df.drop(‘y’,axis=1). It is common practice to just drop (remove) the target feature and use the rest of the columns as the input features.

features = train.drop('y', axis=1)

target = train.y

The final step is to fit the model to the data. This is also called training the model and the best estimates of β0 and β1 are determined.

basic_linear_model.fit(features, target)

You can get the β0 and β1 by calling the intercept_ and coef_ methods of the linear regression model.

print(basic_linear_model.intercept_)

print(basic_linear_model.coef_) ### Results

0.09271652228649466

[-0.0169538]

To make predictions we use model.predict(features). The underlying function is β0+β1x and therefore our predictions are represented by a straight line which we can plot, together with our simulated data set.

y_preds = basic_linear_model.predict(features)

plt.scatter(df.x, df.y)

plt.plot(df.x, y_preds, 'k')

plt.show()

Image by author

As observed, the linear regression model assumes a linear relationship in the data, which is not a good representation for our data.

Polynomial Linear Regression — adding complexity

Unlike a simple linear regression, polynomial models add curves to the data by adding a polynomial factor of x for example x².

Let us first create a second-order polynomial model, by adding x² as another column to our data set. We will call the new column x2 and the formula now is y^=β0 + β1x + β2x².

I created a copy of the data frame to preserve the original. To get x2, I used np.power(value,power)

df2 = train.copy()

df2['x2'] = np.power(df2['x'], 2)

Create reusable function: To avoid a lot of code, I created a function that initializes a model, separates the data set into features and target variables, then plots the predictions from the model. The function takes in a dataframe and a model, and returns the trained model.

def model_fitter(data, model):

features = data.drop('y', axis=1)

target = data.y

model.fit(features, target)

y_preds = model.predict(features)

plt.scatter(data.x, data.y)

plt.plot(data.x, y_preds, 'r--')

return model

Let us now call the function and create our second-order polynomial model.

polynomial_second_order = model_fitter(df2, LinearRegression())

Second order polynomial model by author

Wow, look at that! It models the curve in the data very well. But we will know more when we make predictions on completely new unseen data.

I also created a more complex third-order polynomial model whose results turned out almost identical to the second-order model. As a general rule, if two models perform equally well, it is better to choose the less complex model as it usually generalizes better to new data.

Decision Trees

Decision trees build a model by splitting the data into subsets based on the target variable. This branching nature allows for very complex relationships in the data. We will first create a Decision tree that does not have any parameters. This will create an unconstrained decision tree that will keep splitting into smaller and smaller subsets until the final leaf nodes have only 1 value.

Image from www.datasciencecentral.com

Let us see this concept in code:

from sklearn.tree import DecisionTreeRegressor

decision_tree_unconstrained = model_fitter(train,DecisionTreeRegressor())

Unconstrained Decision Tree by author

The lines represent our predicted values, and clearly the model has completely learnt the train data and predicted all the points perfectly. This is called overfitting and this model will not be able to generalize well when provided with new data. The idea for a model is to ignore the noise and learn the actual signal, but this model learnt the noise as well.

Constrained decision tree by depth

We will add some complexity to our decision tree by limiting the number of levels that the tree can go down to. In our case, we will set a maximum depth of 3, meaning the tree will only be able to branch down 3 levels then arrive at a prediction.

decision_tree_by_depth = model_fitter(train,DecisionTreeRegressor(max_depth = 3))

Decision Tree with max_depth=3 by author

This was a better performance. You can see the branching nature of the decision tree by the steps, unlike the smooth curve of the polynomial linear regression. Let us use another parameter to constraint the model.

Constrained decision tree by leaf

Here we will set the minimum number of samples that each leaf can have before reaching a prediction to 5. This means that the smallest subset will have 5 values of x, preventing the scenario we had with the unconstrained decision tree which had one sample per leaf node.

decision_tree_by_leaf = model_fitter(train,DecisionTreeRegressor(min_samples_leaf = 5))

Decision Tree with min_samples_leaf = 5 by author

There are several parameters you can use with decision trees. The process of choosing the best model parameters is called hyperparameter optimization or tuning.

Random Forests

You can think of a random forest as an ensemble of decision trees.

Random forests train many decision trees in parallel. Each decision tree is trained on only a random subset of observations, and the predictions are combined into one decision tree. This is very effective in preventing overfitting. We need to set a random_state when initializing the model to get reproducible results because each tree is trained on a random set of observations.

from sklearn.ensemble import RandomForestRegressor

random_forest_unconstrained = model_fitter(train,RandomForestRegressor(random_state=111))

Unconstrained Random Forest by author

The unconstrained random forest is still overfit, but not as much as the unconstrained decision tree.

We can also constraint the random forest the same way we did the decision tree; max_depth=3 and min_samples_leaf=5.

random_forest_by_depth = model_fitter(train, RandomForestRegressor(random_state=111,max_depth=3))

Random forest with max_depth=3 by author

random_forest_by_leaf = model_fitter(train, RandomForestRegressor(random_state=111, min_samples_leaf=5))

Random Forest with min_samples_leaf=5 by author

As observed with the constrained random forests above, there is a general smoothing effect of the predictions curve compared to the decision trees which appear as steps. This is because random forest trees are trained on subsets of the observations, then these predictions are combined creating better generalized predictions.

We have only explored two parameters of random forests; max_depth and min_samples_leaf. Find the expansive list here.

Step 3: Testing model performance on unseen test data set

We will evaluate our models using two commonly used metrics for regression machine learning: the mean squared error (MSE) and mean absolute error (MAE). MSE takes the average of the squared errors, and larger errors are scaled up therefore penalized more because of the exponent. MAE takes the average of the absolute errors. A lower MSE and MAE is preferred.

MSE is generally a bit faster to calculate, while MAE is easier to interpret.

from sklearn.metrics import mean_squared_error

from sklearn.metrics import mean_absolute_error

Let us now make predictions on our test data. I created a function to automate this. The function takes in a test dataset, the trained model, and the name to be printed. It separates the test set into features and target variable, makes predictions on the test set, calculates the MAE and MSE, and prints them.

def model_performance(data, model, name):

test_features = data.drop('y', axis=1)

test_target = data.y

test_preds = model.predict(test_features)

mae = mean_absolute_error(test_preds, test_target)

mse = mean_squared_error(test_preds, test_target)

print(name)

print('MAE', np.round(mae,3))

print('MSE', np.round(mse, 3))

For the second order polynomial linear model, we need to engineer the same features as those for the train data set by adding another column to our test data that contains x² values. We will first create a copy of the test data then add the x2 column.

test2 = test.copy()

test2['x2'] = np.power(test2.x, 2)

Now we will run the function for every model and return the mean squared errors and mean absolute errors.

model_performance(test, basic_linear_model, 'Basic linear regression')

model_performance(test2, polynomial_second_order, 'Second Order Polynomial Model')

model_performance(test, decision_tree_unconstrained, 'Uncontrained Decision Tree Model')

model_performance(test, decision_tree_by_depth, 'Decision Tree Constrained by max_depth = 3')

model_performance(test, decision_tree_by_leaf, 'Decision Tree Constrained by min_samples_leaf = 5')

model_performance(test, random_forest_unconstrained, 'Unconstrained Random Forest')

model_performance(test, random_forest_by_depth, 'Random Forest Constrained by max_depth = 3')

model_performance(test, random_forest_by_leaf, 'Random Forest Constrained by min_samples_leaf = 5')

Below are the results

Basic linear regression

MAE 0.677

MSE 0.669

Second Order Polynomial Model

MAE 0.427

MSE 0.277

Uncontrained Decision Tree Model

MAE 0.523

MSE 0.434

Decision Tree Constrained by max_depth = 3

MAE 0.441

MSE 0.289

Decision Tree Constrained by min_samples_leaf = 5

MAE 0.427

MSE 0.287

Unconstrained Random Forest

MAE 0.47

MSE 0.333

Random Forest Constrained by max_depth = 3

MAE 0.424

MSE 0.28

Random Forest Constrained by min_samples_leaf = 5

MAE 0.416

MSE 0.276

We can see that for the two metrics, The Random Forest constrained by leaf has the lowest error, hence the best performing. As expected, the simple linear regression was the worst performing as the data clearly lacked a linear relationship. However the second-order polynomial model was the second best performing model, even though the curve seemed to fit so well. This goes to show why the random forest is so vastly used in applied machine learning.