PCA and OLS in matrix form with R | by Serafim Petrov | Apr, 2021 | Towards Data Science | Towards Data Science

Principal Component Analysis (PCA) and Ordinary Least Squares (OLS) are two important statistical methods. They are even better when performed together. We will explore these methods using matrix operations in R and introduce a basic Principal Component Regression (PCR) technique.
We will generate a simple data set of four highly correlated exploratory variables from the Gaussian distribution, and a response variable which will be a linear combination of them with added random noise.
> library(‘MASS’)> mu=rep(3,4)> sigma=matrix(.9, nrow=4, ncol=4) + diag(4)*0.1> set.seed(2021)> data <- as.data.frame(mvrnorm(20, mu = mu, Sigma = sigma), + empirical = T)> y <- apply(data, 1, sum)+rnorm(20, 1, 1)
We may observe the correlation matrix and confirm that the exploratory variables are highly correlated. In this case, the regression coefficients might be biased.
> cor(data)
This statistical method helps us to deal with multicollinearity and high dimensionality. Using R, we might obtain PCs just in 4 steps.
First, we need to scale the data. In our example we know that the data are of the equivalent magnitude; therefore, we will only center it.
data.scaled <-  scale(data, scale=F)
Second, we need to compute the covariance matrix:
data.cov <- cov(data.scaled)
Third, we need to compute eigenvalues to obtain eigenvectors. In R the function eigen() returns both results. The eigenvalue is a scale parameter; the eigenvector is a rotation parameter.
data.eigen <- eigen(data.cov)
Finally, we need to perform a matrix multiplication of our data and eigenvectors. The result would bring Principal Components: these are coordinates of original data on the new axis.
> data.custom.pca <- data.scaled%*%(data.eigen$vectors)
The components are orthogonal, and we may confirm this by observing the correlation matrix:
> round(cor(data.custom.pca),5)
We may also compute the proportion of explained variation captured by each component. The first component captures 93.88% of variation; the first + second components capture 97.06%, and so on.
> cumsum(data.eigen$values)/sum(data.eigen$values)
This function automates the four-step approach we performed earlier. From the results below, we may see, that the components are equivalent to the ones we have just obtained:
> data.pca <- prcomp(data)
Some components have opposite signs, but if we check the corresponding eigenvectors (named rotation in this package), we notice that their signs are opposite as well, and when we multiply them to obtain original data, everything would work well.
> data.pca$rotation
The eigenvalues could be obtained by squaring the standard deviation from this package.
> data.pca$sdev^2
Finally, the summary function would return the importance of components and the proportion of variance captured by each component. We may observe that the results are consistent with our previous estimations.
> summary(data.pca)
Without further details, the coefficients for the regression equation could be obtained using this formula:
To compute it, we need to know three operations in R: how to invert matrices, how to transpose matrices, and how to multiply matrices:
# Matrix multiplication:  %*% # Transpose matrix:       t(x)# Invert matrix:          solve(x)
But before we finalize our equation, we need to keep in mind, that this formula assumes that x0 = 1, therefore we to add a unit column to our data. Then we run the equation in matrix form and obtain the coefficients:
> x <- as.matrix(cbind(1,data))> solve(t(x)%*%x) %*% t(x) %*% y
We may see, that this equation in R directly repeats the one we stated at the beginning of the topic. Let us compare it with the built-in function in R:
> model <- lm(y~x[,2:5])
The coefficients are identical for both methods because it is essentially the same method.
Now we know how the PCA’s and OLS methods are performed on simple data set. This completes our prerequisite for the following topic.
We will use the PCs we have obtained earlier and since they had different signs for some components, we will show, that when we transform the data to the original form, the coefficients would be equivalent.
# Adding components from both methods to the data setdata.new <-  cbind(data, data.custom.pca, data.pca$x, y)colnames(data.new)[5:8] <- c('cust1', 'cust2', 'cust3', 'cust4')
For demonstration purpose, the models would be estimated using all four components, even though usually after PCA we reduce the number of predictors:
model.cust <- lm(y~cust1+cust2+cust3+cust4, data.new)model.pca <- lm(y~PC1+PC2+PC3+PC4, data.new)
Now we need to convert the coefficients from the component rotation to the original space. This could be done by multiplying them by corresponding eigenvectors:
beta.pca <- as.matrix(model.pca$coefficients[2:5])beta.original.pca <- as.matrix(data.pca$rotation) %*% beta.pcabeta.cust.pca <- as.matrix(model.cust$coefficients[2:5])beta.original.cust <- as.matrix(data.custom.eigenvector) %*% beta.cust.pca
This confirms that the opposite signs of PCs would be compensated by signs of eigenvectors when we transform the points to the original system.
The function pcr() has several parameters to tune, but we will construct the most basic version, and obtain the coefficients.
library(pls)fit <- pcr(y ~.,  data = as.data.frame(data.scaled), center=F)
The function computes all PCs, eigenvalues, and many more, as well as coefficients for the different number of components. Above we slice it for all four components, and we may see, that they are not even slightly different from the ones we have obtained earlier.
But how many components are needed? We may plot our estimated PCs or the PCR model for searching for ‘elbow’ on the graph or try to make a prediction of the test set/cross-validation or use some more formal approaches.
plot(fit, "validation", val.type = "MSEP")# orplot(data.pca, type='l')
From the plot, we may see, that one PC is sufficient enough to capture the variance in the data. We also know from the previous observations, that the first component captures ~ 93.88% of the variation. Therefore if we wanted to run a regression on the real data set and obtained similar results, we would have used only the first component to escape multicollinearity in the data set.
In this article, we explored a matrix form approach to compute PCs and OLS coefficients, compared this custom approach with the built-in functions in R, and summarized it by performing the PCR. We found no difference in both techniques. For practical purposes, it is convenient to use the PLS library, but for theoretical purposes, it is useful to know the math behind the scene.
The full code can be found on GitHub.
If you have any questions or spotted a mistake or typo, please leave a comment below.
Connect on LinkedIn