Creating a simple statistical learning model from scratch: emphasizing prediction in regression | by Alex daSilva | Towards Data Science

Learning about statistical/machine learning can be intimidating, especially if you’re like me and coming from another field (eg., social sciences, life sciences, business, etc). There are hundreds of complex models to choose from and numerous schemes to validate your data. Here, we’re going to take a closer look at a simple model for prediction you probably came across if you completed a statistics course, regression. Why so simple? Regression may not be as sexy as some other methods, but gaining a deeper understanding of the basics is undoubtedly important for moving on to more complex models.
When learning about regression, the emphasis probably wasn’t on prediction (or where predicted values come from) but rather on jumping to the output, checking coefficients, and praying that p-values associated with said coefficients fell below .05. Regression need not be just a tool for inferential statistics. As I touched on, if you’ve ran a regression analysis you’ve already completed a computation that involved calculating predicted values.
Below, we’ll unpack how prediction unfolds in two different contexts. In the first context, we’ll be working with data where the y value is present — a common situation where an analyst is making inferences about the relationship between the outcome and predictor variables. In the second context, we’ll be predicting new y values on left out data. We’ll do this all without the help of functions or libraries to illustrate how linear regression can be used as a basic predictive tool with the classic iris data set.
The first thing to do is to load the iris data set:
data("iris")

gridExtra::grid.table(head(iris, 10))

Before we get into our own calculations, let’s begin by looking at how we can fit a linear model and use that to predict some new data in base R with the “lm” function. We’ll attempt to predict Sepal Length from the other 3 numeric variables in the iris data set. To begin, we’ll grab some training data (70/30 split) and fit our model.
iris_sub <- iris[,c(1:4)]

cut <- .7*nrow(iris)

iris_train <- iris_sub[1:cut, ]

fit <- lm(Sepal.Length ~ ., data = iris_train )

Now, we’ll create some test data and combine that with our model from the training data to create some predictions.
iris_test <- iris_sub[(cut+1):nrow(iris_sub), ]

preds <- predict(fit, iris_test[, -1])

plot(preds, iris_test[, 1], main = paste('Cor =',round(cor(iris_test[, 1], preds ),2)), xlab = "Observed Values", ylab = "Predicted Values")

The predictions look great! If someone accidentally deleted a bunch of information regarding iris’ sepal lengths in a flower database, we can safely say we’d do a sound job recovering that information.
Of course, our concern here is to determine what “predict” is doing to come up with those values. We’ll take a look at that in a second; but first, we’ll examine prediction in the context of linear regression models in a more general sense.
We’ll start by examining how new values can be predicted on a set of data where both the y values and predictor variables are present (using the iris training data we created as an example). First, we need to separate our x values from our y values. We’ll pad the x matrix with a column of ones to represent the intercept.
x <- iris_train[, -1]

x <- cbind(Intercept = rep(1, nrow(x)), x)

x <- as.matrix(x)

y <- as.matrix(iris_train[, 1])

Now, we need to transpose x and multiply it by itself (X’X). This results in a sum of squares/cross-product matrix, SSCP. It turns out with a few manipulations, this matrix can depict the variance/co-variance, correlation, or cosine association between a set of variables. If you’re interested in hearing a bit more about that, check out my post here.
XtX <-t(x) %*% x

XtX

##              Intercept Sepal.Width Petal.Length Petal.Width
## Intercept        105.0      324.80       314.50       89.10
## Sepal.Width      324.8     1027.66       930.13      259.79
## Petal.Length     314.5      930.13      1188.37      364.22
## Petal.Width       89.1      259.79       364.22      115.75

Our calculations match the output from the actual cross-product function.
crossprod(x) 

##              Intercept Sepal.Width Petal.Length Petal.Width
## Intercept        105.0      324.80       314.50       89.10
## Sepal.Width      324.8     1027.66       930.13      259.79
## Petal.Length     314.5      930.13      1188.37      364.22
## Petal.Width       89.1      259.79       364.22      115.75

Next, we are going to calculate a projection matrix or hat matrix (on it’s diagonal are observation leverages).
h <- x %*% solve(XtX) %*% t(x)

head(data.frame(our_leverages = diag(h), model_leverages = hatvalues(fit)), 5)

##   our_leverages model_leverages
## 1    0.02139771      0.02139771
## 2    0.02869219      0.02869219
## 3    0.02416413      0.02416413
## 4    0.02373262      0.02373262
## 5    0.02397262      0.02397262

all.equal(diag(h), hatvalues(fit)) #matches output from hatvalues function

## [1] TRUE

This will allow us to map y into predicted y values.
our_fitted_vals <- h %*% y

head(data.frame(our_fitted_vals, model_fitted_vals = as.numeric(fit$fitted.values)), 5)

##   our_fitted_vals model_fitted_vals
## 1        5.035358          5.035358
## 2        4.738074          4.738074
## 3        4.786883          4.786883
## 4        4.867636          4.867636
## 5        5.094815          5.094815

all.equal(as.numeric(our_fitted_vals), as.numeric(fit$fitted.values)) #our calculated predicted values match the values accessed from the model

## [1] TRUE

So, if we have access to the y values, we can combine them with a projection matrix to obtain predictions. That’s all well and good, but how can we predict new values of y in our test data using our training data? To do that, we need to calculate beta coefficients for our training data that contain information about the relationship between our y and x values.
XtY <- t(x) %*% y

betas <- solve(XtX) %*% XtY

We can ensure that out calculated beta values are indeed correct by comparing them to the values produce by “lm”.
data.frame(model_betas = summary(fit)$coefficients[, 1], our_betas = betas) # our calculated betas match betas from lm

##              model_betas  our_betas
## (Intercept)    2.0952291  2.0952291
## Sepal.Width    0.5945672  0.5945672
## Petal.Length   0.7010446  0.7010446
## Petal.Width   -0.6115940 -0.6115940

We can add the model intercept through to our test data and multiply that by the beta coefficients (intercept excluded) to find our predicted values.
our_preds <- betas[1] + as.matrix(iris_test[,-1]) %*% betas[-1]

head(data.frame(model_preds = as.numeric(preds), our_preds = as.numeric(our_preds)), 5)

##   model_preds our_preds
## 1    7.221478  7.221478
## 2    5.696638  5.696638
## 3    7.135186  7.135186
## 4    6.546837  6.546837
## 5    6.983058  6.983058

all.equal(as.numeric(preds), as.numeric(our_preds)) #our predicted values for our test data match that of the predict function 

## [1] TRUE

What we’ve learned so far can easily be extended to a kfold cross validation scheme. The function below will create folds and return a list containing the original data set with the folds as a new column. In addition, a list containing the fold indices themselves will also be returned.
Let’s create the folds and make sure they make sense.
And fit our model using 5 fold cross-validation:
We can extract and examine the predictive summary statistics.
Finally, we can pass our fold indices to the popular predictive modeling package, caret, and confirm our calculations.
Averaging across folds, our statistics match caret’s output.
In sum, we took a closer look at how prediction functions in the context of regression. We were ultimately able to apply the computations we covered to make predictions on left out data. Hopefully this tutorial provided a bit of clarity into what happens when you’re pressing ctrl+enter on a line of code containing the “lm” function.