Exploratory Factor Analysis vs Principal Components: from concept to application | by Rafael Valdece Sousa Bastos | Towards Data Science

In data science, we often want to measure variables such as social-economic status (SES). Some variables have a lot of parameters (or items), for example, SES can be measured based on income, education, etc. Then, to proceed with the analysis, it is common to reduce the number of parameters to fewer components through Principal Components Analysis (PCA). However, we will see why some variables cannot be reduced by PCA and we will learn how to use Exploratory Factor Analysis in our favor.
Both of them are used to reduce the number of parameters to fewer variables. Also, both methods assume that the variance of a parameter is divided into specific variance, common variance, and error variance.
In PCA, when we retain a component, we take into account both specific variance and common variance. While in EFA we only take into account common variance. Seeing the next figure, we can think that A’s are specific variances, B is the common variance, and C’s are error variances. In PCA we use A’s + B while in EFA we only use B.
PCA is based on the formative model, where the variation in the component is based on the variation in item responses (i.e. level of income will affect the social-economic status). While EFA is based on the reflective model, where the variation of the items is based on the variation of a construct (i.e. a person's happiness will change their response to the items, not the contrary). We can see this representation with the following figure.
With that being said, the common application of EFA is to measure psychological variables. For example, if you want to measure a person’s level of happiness we will use only the common variance because the items of the instrument are trying to measure what they have in common (i.e. the level of happiness).
PCA has mostly three main steps.
Compute the covariance matrixCompute eigenvalues and eigenvectorsRotation of components
Compute the covariance matrix
Compute eigenvalues and eigenvectors
Rotation of components
While in EFA we have:
Verification of data adequacyComputation of covariance/correlation matrix (Factor Extraction)Selection of factors to retainRotation of factors
Verification of data adequacy
Computation of covariance/correlation matrix (Factor Extraction)
Selection of factors to retain
Rotation of factors
Since there are a lot of posts here on Towards Data Science regarding PCA, I will focus on EFA from now on. In the next section, I will describe each step from EFA.
We usually use two tests to measure if our data is adequate to proceed with EFA.
Bartlett’s test of sphericity
This test verifies the hypothesis that variables are not correlated in the population. Therefore, the null hypothesis is that the correlation matrix is equal to an identity matrix. If the correlation matrix is equal to an identity matrix, we cannot proceed with EFA, since there is no correlation between variables. The statistical analysis behind this test goes as follows:
χ2 =- [(n-1)-(2v+5)/6]ln|R|
Wheren is the sample sizev is the number of variables|R| is the determinant of the correlation matrix
In the literature, we can see that if the level of significance equals p < 0.05 that means we can proceed with EFA.
Kaiser-Meyer-Olkin (KMO)
Verify the proportion of variance of items that can be caused by factors. The test verifies if the inverse correlation matrix is close to a diagonal matrix, comparing the values of linear correlations with values of partial correlations.
Where:
rjk = correlation coefficient between Xj and Xk
pjk = correlation coefficient between Xj and Xk, controlling for other Xs.
Values under 0.5 are considered unacceptable, between 0.50 and 0.70 are mediocre, between 0.70 and 0.80 are good, above 0.80 are considered great and above 0.90 are excellent.
In EFA we have a lot of methods of extraction to choose from. If data are normally distributed, it is recommended to use maximum likelihood, since it enables a variety of goodness of fit indices, significance test of factor loadings, calculation of confidence intervals, etc. However, if the data doesn’t follow a normal distribution, it is recommended to use principal axis factoring.
There are a lot of methods we can use to select the number of items, I will focus mainly on three of them.
a) Kaiser criterion: it proposes if a factor’s eigenvalue is above 1.0, we should retain that factor. The logic behind it is: if a factor has an eigenvalue = 3.0, that means that the factor explains the same amount of variance as 3 items.
Watch out, this criterion is known to over and underestimate the number of factors. I would not recommend using it alone.
b) Scree plot: where we evaluate when there is a substantial decline in the magnitude of the eigenvalues. This method also has some limitations, because it can generate ambiguous results and are open to subjective interpretation.
c) Parallel analysis: the eigenvalue of the sample and eigenvalue of random data are calculated. The number of factors is selected when the number of eigenvalues of real data is bigger than from simulated data. This method usually works well.
Now we will see a tutorial for computing an EFA in R. We will be using the package psych to make our computations. Also, we will be using the Positive and Negative Affects Scale, which consists of items regarding negative and positive emotions.
We will read the dataset using the R function read.delim
Affects <- read.delim(“https://raw.githubusercontent.com/rafavsbastos/data/main/Afetos.dat")
The name we choose for our data frame was Affets. Using View(Affects) we will see our dataset:
To start manipulating data, we will need to download the psych package. Just run the following code.
install.packages(“psych”)
Ok. Now the package is on your computer. Now, we need to make it start working with library(psych) .
To see if it’s ok to do EFA with our data, we will first calculate Bartlett’s test of sphericity.
First, we calculate the correlation matrix:
correlation <- cor(Affects)
Then, we will calculate Bartlett’s test with cortest.bartlett(correlation, n=1033) , where the second argument is the sample size. We will have the following output:
The significance level was smaller than 0.05, which means we can proceed with EFA (if we assume values under 0.05 indicate the adequacy of our data).
Now we will compute KMO. Use the following code KMO(Affects) .
We can see that the overall Measure of Sample Adequacy (MSA) was 0.94, which means that it’s excellent. We can also see the MSA of each item below the overall MSA. Based on these results, we can proceed with EFA.
Using only one line of code, we will be able to extract the number of factors and select which factors we are going to retain.
fa.parallel(Affects,fm=”pa”, fa=”fa”,  main = “Parallel Analysis Scree Plot”, n.iter=500)
Where:
the first argument is our data framefm is the extraction method; we are using principal axis factoring (“pa”)fa = “fa” ; if we write “fa” we will be doing an EFA. If we write “pc” we will be doing a PCA. Since we are using the reflective model, we will be doing an EFA.main = title of our image.n.iter = number of interactions we wish to do.
the first argument is our data frame
fm is the extraction method; we are using principal axis factoring (“pa”)
fa = “fa” ; if we write “fa” we will be doing an EFA. If we write “pc” we will be doing a PCA. Since we are using the reflective model, we will be doing an EFA.
main = title of our image.
n.iter = number of interactions we wish to do.
We will have the following output:
It tells us that we need to retain 3 factors based on Parallel Analysis. Another output from the same line of code is the next figure:
We can see our data eigenvalues (in blue) and the simulated data eigenvalues (in red). Look at the intersection between the red and blue eigenvalues and notice that the fourth eigenvalue of some random data explains more variance than the fourth eigenvalue from our data. However, if we consider the Kaiser criterion (black line crossing the figure), we will be extracting just two factors. Since we want to retain as few factors as possible, we will proceed with the analysis with only two factors.
Before we jump to the next section, I want to show you the difference between PCA and EFA when extracting the eigenvalues.
fa.parallel(Affects, fm = "pa", cor = "cor", fa= "both", main = "Parallel Analysis Scree Plot", n.iter=500)
In the figure above, PC represents principal components and FA represents factor analysis. If we use the Kaiser criterion in PCA, we will retain three factors, while if we use Parallel Analysis criterion, we will retain 2 factors. We can also notice that eigenvalues of PCA are higher than from the EFA, that’s because of that thing I said before, PCA takes into account common and specific variance, while EFA only takes common variance.
We can see which items explain more of the factor based on factor loadings, commonalities. Also, with the following code we can see explained variance and goodness of fit indices of our two-factor model:
fit <- fa(Affects, nfactors = 2, n.obs = 1033 , rotate = “oblimin”, residuals = TRUE, fm = “pa”)print(fit, sort = TRUE)
That prints out:
In the first table, we can see that Positive Affects Items (AP’s) load strongly on the first factor (PA1), while Negative Affects Items (AN’s) load strongly on the second factor (PA2). The commonality is represented by h2.
Bellow the first table, we have the proportion of explained variance for each factor. The factor of Positive Affects (PA1) explained 25% of the data variance, while the Negative Affects (PA2) explained 24%.
The goodness of fit indices is also calculated. Although they are widely used in Confirmatory Factor Analysis, their use in EFA is a bit of a mystery. There are few studies that evaluate the use of goodness of fit indices in EFA, therefore it may be difficult to interpret this part of the data.
To obtain a visual representation of the factor loadings, we will use the function fa.diagram(fit).
We can also see a geometric visualization of factors in their axis with plot(fit) .
In the graphic above, we want our Positive Affects items to be closer together while staying far from the cluster of Negative Affects items.
Given all steps so far, we can see that our measure presented a two-factor structure, which is the same as was theorized by previous authors. In addition, we found that items loaded in their given factors, and the factors explained a good portion of the variance.
I must highlight that, although we have statistical methods to select the number of factors to retain, only the researcher can choose which method will be the best one. That means the criterion to select the best method to retain factors is open to subjective interpretation. Also, I presented a statistical tool to analyze psychological data where the reflective model is more adequate.
Feel free to contact me by
Gmail: rafavsbastos@gmail.comWebsite for consulting and partnerships: rafavsbastos.wixsite.com/websiteLinkedIn: linkedin.com/in/rafael-valdece-sousa-bastos/Github: github.com/rafavsbastos