What is the difference between PCA and Factor Analysis? | by Joos Korstanje | Towards Data Science

PCA, short for Principal Component Analysis, and Factor Analysis, are two statistical methods that are often covered together in classes on Multivariate Statistics.
In this article, you will discover the mathematical and practical differences between the two methods.
Multivariate Statistics is a group of statistical methods that focus on studying multiple variables together while focusing on the variation that those variables have in common.
Multivariate Statistics deals with the treatment of data sets with a large number of dimensions.
Its goals are therefore different from supervised modeling, but also different from segmentation and clustering models.
There are many models in the family of Multivariate Statistics. In this article, I will focus on the difference between PCA and Factor Analysis, two commonly used Multivariate models.
Modern datasets often have a very large number of variables. This makes it difficult to inspect each of the variables individually, due to the practical fact that the human mind cannot easily digest data on such a large scale.
When a dataset contains a large number of variables, there is often a serious amount of overlap between those variables.
The components that are found by PCA are ordered from the highest information content to the lowest information content.
PCA is a statistical method that allows you to “regroup” your variables into a smaller number of variables, called components. This regrouping is done based on variation that is common to multiple variables.
The goal of PCA is to regroup variables in such a way that the first (newly created) component contains a maximum of variation. The second component contains the second-largest amount of variation, etc etc. The last component logically contains the smallest amount of variation.
Thanks to this ordering of components, it is made possible to retain only a few of the newly created components, while still retaining a maximum amount of variation. We can then use the components rather than the original variables for data exploration.
The mathematical definition of the PCA problem is to find a linear combination of the original variables with maximum variance.
This means that we are going to create a (new) component. Let’s call it z. This z is going to be computed based on our original variables (X1, X2, ...) multiplied by a weight for each of our variables (u1, u2, ...).
This can be written as z = Xu.
The mathematical goal is to find the values for u that will maximize the variance of z, with a constraint of unit length on u. This problem is mathematically called a constrained optimization using Lagrange Multiplier, but in practice, we use computers to do the whole PCA operation at once.
This can also be described as applying matrix decomposition to the correlation matrix of the original variables.
PCA is efficient in finding the components that maximize variance. This is great if we are interested in reducing the number of variables while keeping a maximum of variance.
Sometimes, however, we are not purely interested in maximizating variance: we might want to give the most useful interpretations to our newly defined dimensions. And this is not always easiest with the solution found by a PCA. We can then apply Factor Analysis: an alternative to PCA that has a little bit more flexibility.
Just like PCA, Factor Analysis is also a model that allows reducing information in a larger number of variables into a smaller number of variables. In Factor Analysis we call those “latent variables”.
Factor Analysis tries to find latent variables that make sense to us. We can rotate the solution until we find latent variables that have a clear interpretation and “make sense”.
Factor Analysis is based on a model called the common factor model. It starts from the principle that there a certain number of factors in a data set, and that each of the measured variables captures a part of one or more of those factors.
An example of Factor Analysis is given in the following schema. Imagine many students in a school. They all get grades for many subject matters. We could imagine that these different grades are partly correlated: a more intellectually gifted student would have higher grades overall. This would be an example of a latent variable.
But we could also imagine having students who are overall good in languages, but bad in technical subjects. In this case, we could try to find a latent variable for language ability and a second latent variable for technical ability.
We now have latent variables that measure the general ability of a student for Language and Technical subjects. But it would still be possible that some students are great at languages overall, but that they are just bad at German. This is why the Common Factor Model has specific factors: they measure the impact of one specific measured variable on the measured variable. We could describe it as “Ability for learning German while taking into account the general ability for learning languages”.
As said, the mathematical model in Factor Analysis is much more conceptual than the PCA model. Where the PCA model is more of a pragmatic approach, in Factor Analysis we are hypothesizing that latent variables exist.
In a case with two latent variables, we can compute our original variables X by attributing one part of its variation to our first common latent variable (let’s call it k1), part to the second common latent variable (k2), and part to a specific factor (specific to this variable; called d).
In a case with 4 original variables, the Factor Analysis model would be as follows:
X1 = c11 * k1 + c12 * k2 + d1X2 = c21 * k1 + c22 * k2 + d2X3 = c31 * k1 + c32 * k2 + d3X4 = c41 * k1 + c42 * k2 + d4
c are each coefficient of the coefficient matrix, which are the values that we need to estimate.
To solve this, the same mathematical solution as in PCA is used, except for a small difference. In PCA we apply matrix decomposition to the correlation matrix. In Factor Analysis, we apply matrix decomposition to a correlation matrix in which the diagonal entries are replaced by 1 — var(d), one minus the variance of the specific factor of the variable.
So in short, the mathematical difference between PCA and Factor Analysis is the use of specific factors for each original variable.
Let’s get back to the example with students in a school who take 4 exams: two languages exams and two technical exams. We expect two underlying factors: language ability and technical ability.
PCA does not estimate specific effects, so it simply finds the mathematical definition of the “best” components (components who maximize variance). Those could be a component for language ability and a component for technical ability, but it also could be something else.
Factor Analysis will also estimate the components, but we now call them common factors. Besides that, it also estimates the specific factors. It will therefore give us two common factors (language and technical) and four specific factors (abilities on test 1, test 2, test 3, and test 4 that are unexplained by language or technical ability).
Even though we don’t really care for the specific factors, the fact that they have been estimated gives us a different definition of the common factors / components.
Resulting from this mathematical difference, we have also a big difference between the application of PCA and Factor Analysis. In PCA, there is one fixed outcome that orders the components from the highest explanatory value to the lowest explanatory value.
In Factor Analysis, we can apply rotations to our solution, which will allow for finding a solution that has a more coherent business explication to each of the factors that was identified.
The possibility to apply rotation to a Factor Analysis makes it a great tool for treating multivariate questionnaire studies in marketing and psychology.
The fact that Factor Analysis is much more flexible for interpretation makes it a great tool for exploration and interpretation. Two examples:
In marketing: resume product evaluation questionnaire with many questions in a few latent factors for product improvement.
In psychology: reduce very long personality test responses into a small number of personality traits.
PCA allows us to find the components that contain the maximum amount of information in fewer variables. This makes it a great tool for dimension reduction.
PCA on the other hand is used in cases where we want to retain the largest amount of variation in the smallest number of variables possible. This can for example be used to simplify further analysis. PCA is also much used in data preparation for Machine Learning tasks, where we want to help the Machine Learning model by already “summarising” the data in an easier to digest form.
To get started with PCA and Factor Analysis in Python or R, here are a few quick pointers to useful libraries that you will need. The implementations of PCA are as good in Python as in R. However, Factor Analysis is not yet very mature in Scikit Learn and it may be useful to check out the R alternative.
To get started with PCA in Python, you can use the PCA function from sklearn.decomposition.
It can be used as follows:
from sklearn.decomposition import PCAfrom sklearn import datasetsiris = datasets.load_iris()X = iris.datamy_pca = PCA(n_components=2)my_pca.fit(X)print(my_pca.explained_variance_ratio_)print(my_pca.singular_values_)
To get started with Factor Analysis in Python, you can use the FactorAnalysis function from sklearn.decomposition.
In version 0.23 of scikit-learn, rotation is not yet available. But varimax rotation will soon be available and is already available in the nightly build of version 0.24.
It can be used as follows:
from sklearn.decomposition import FactorAnalysisfrom sklearn import datasetsiris = datasets.load_iris()X = iris.datamy_fa = FactorAnalysis(n_components=2)# in new version of sklearn:# my_fa = FactorAnalysis(n_components=2, rotation='varimax') X_transformed = my_fa.fit_transform(X)
In R, there are multiple implementation of PCA available, but prcompis a good option. It can be used as follows:
data(iris)iris_data <- iris[, 1:4] my_pca <- prcomp(iris_data, center = TRUE, scale = TRUE)print(my_pca)
Factor Analysis is much more developed in the R Psych package when compared to the Scikit Learn implementation. It can be used as follows:
install.packages(“psych”)library(psych)data(iris)iris_data <- iris[, 1:4]iris_cor <- cor(iris)my_fa <- fa(r = iris_cor, nfactors = 2)print(my_fa)# or the alternative with varimax rotation:my_fa <- factor.pa(iris, nfactors=3, rotation="varimax")print(my_fa)# or the alternative with proimax rotationmy_fa <- factor.pa(iris, nfactors=3, rotation="promax")print(my_fa)
Another great R library for PCA and Factor Analyis is FactoMineR. It has a really large range of additional options for going further in Multivariate Statistics.
So in conclusion, we observe the difference between PCA and Factor Analysis in three points:
Firstly, the goal is different. PCA has as a goal to define new variables based on the highest variance explained and so forth.
FA has as a goal to define new variables that we can understand and interpret in a business / practical manner.
Then as a consequence, the mathematics behind the two methods are, while close to each other, not exactly the same. Although both methods use decomposition, they differ in the details.
This also causes the fact that Factor Analysis has an additional possibility for rotation of the final solution, while PCA does not.
As Factor Analysis is more flexible for interpretation, due to the possibility of rotation of the solution, it is very valuable in studies for marketing and psychology.
PCA’s advantage is that it allows for dimension reduction while still keeping a maximum amount of information in a data set. This is often used to simplify exploratory analyses or to prepare data for Machine Learning pipelines.
I hope this article has been useful to you. Thanks for reading and don’t hesitate to stay tuned for more!