Factor Analysis 101. Can we reduce the number of variables... | by Jeppe Andersen | Towards Data Science

Even though this is an introductory look at Factor Analysis, I assume that the reader has some knowledge in math and statistics and want to learn about multivariate statistical analysis. This article will focus on the theoretical understanding of Factor Analysis and I will thus leave out the coding related to this subject — except for a simple R illustration.
Make sure to follow my profile if you enjoy this article and want to see more!
In real life, data tends to follow some patterns but the reasons are not apparent right from the start of the data analysis. The essential purpose of Factor Analysis is to describe the covariance relationships between several variables in terms of a few underlying and unobservable random components that we will call factors.
We will assume that the variables can be grouped by looking at their correlations. That is, assume that all variables in a specific group have high correlation among themselves, but low correlation with variables in other groups. In that case we can think of each group of variables as a representation of a single underlying construction, or a factor, which is responsible for the observed correlation.
As an example, correlation from a group consisting of the variables english, math and biology scores could come from an underlying “intelligence factor” and another group of variables representing fitness scores could correspond to another underlying factor.Factor Analysis is an extension of Principal Component Analysis (PCA). Both models tries to approximate the covariance matrix Σ, but Factor Analysis questions whether the data are consistent with some prescribed structure. If you haven’t heard about PCA, I recommend this article that illustrates it with R:
towardsdatascience.com
The Factor Analysis Model in matrix form is:
X is an observable stochastic vector with p components, mean vector μ and covariance matrix Σ. The factor model says that X is linear dependent on a few m unobservable stochastic variables,
called common factors and p sources of variation,
called errors or specific factors. The coefficients l_ij is called the loadings of the variable i on the factor j, so that the matrix L is the matrix of factor loadings (I will use “_” to denote lowercase letters).The i specific factors ε_i are only connected with the i variables X_i. The p deviations,
are expressed in terms of the p+m stochastic variables,
which are all unobservable. This is what seperates the factor model from the multivariate regression model, where the independent variables can be observed. All of these unobservable components provides some complications. We cannot directly verify the factor model if we have unobservable components, but with some assumptions on F and ε, the model implements covariance relationships and we can try to interpret them.
We will then assume that the mean and covariance for F and ε are:
and that F and ε are independent, so that Cov(ε, F) = E(εF’) = 0.Here, Cov(ε) includes Ψ which is the uniqueness for each variable, i.e. the amount of variance that cannot be explained by any factor.
If the unobservable random vectors F and ε satisfy these conditions, we have a factor model with m common factors:
The covariance structure of this model is then:
The factor model is linear in the common factors. If the relationship between X and the underlying factors are not linear, the covariance structure above is not sufficient.
The proportion of variance of the ith variable contributed by the m common factors is called the ith communality, h2_i. The proportion of Var(X_i) = σ_ii that is due to the specific factor is the uniqueness or specific variance. We will see that:
(I.e. Var(X_i) = communality + specific variance)Or that
The ith communality is then the sum of squares of loadings of the ith variable on the m common factors.
If the common and specific factors can be assumed normal distributed, we can use MLE for the factor loadings and specific variances. The likelihood function is not pretty, so I won’t include it here, but understand that it depends on L and Ψ through Σ = LL^T + Ψ. This model is not defined because of the many choices for L and this is where the uniqueness condition comes to use:
It’s a diagonal matrix and it secures one maximum so that estimates for ^L and ^Ψ can be found (I will use ^ in front of a letter to denote a “hat” operator). From here, the proportion of total variance included in the jth factor can be explained by the estimated loadings. The trouble here is that the maximum likelihood solution for factor loadings is limited by the uniqueness condition and clear patterns in the factors can be difficult to see until they are rotated.
Our structure is orthogonal (e.g. two vectors are considered orthogonal if they form a 90-degree angle). An orthogonal transformation of factor loadings (and hence as implied orthogonal rotation of the factors) is called factor rotation. If ^L is the matrix of estimated factor loadings, then
is a matrix of “rotated” loadings. The estimated covariance matrix stays the same and this rotation can give a simpler structure and make the factors easier to interpret, just like a microscope. An analytical solution is the Varimax Criterion (Google it) that chooses the orthogonal transformation for maximizing. Factor rotation is especially good for estimates found by MLE because of the uniqueness condition used there.
Alright, so what did we just spend 10 minutes reading about?Dealing with multidimensional data can be tough, so if we can “cut” down the number of dimensions, we can make the data way easier to look at and we can subjectively try to intepret the underlying factors.
We will look at the Olympic decathlon data (Johnson and Wichern p.499) that you can get with
od.data <- read.table("http://static.lib.virginia.edu/statlab/materials/data/decathlon.dat")
Edit: If you are trying this out for the first time, you will get an error if you are not adding a matrix translation. This is done by:
od.data <- as.matrix(od.data)rownames(od.data) <- colnames(od.data) <- c("100m","LJ","SP","HJ","400m","100mH","DS","PV","JV","1500m")
First let’s try 3 factors. The output is:
By default, factanalprovides ML estimates and Varimax rotation. First is the uniqueness. It represents ^Ψ from earlier. If the uniqueness is high, the variance cannot be described by any of the factors, thus the name uniqueness or specific variance. It cannot be explained by some underlying factor. This is the case for the 1500m run and vica versa for SP (Shot put). If we subtract the uniqueness from 1, we get the communality and it will then tell how much variance is explained by the 3 factors.
The loadings represent ^L from earlier. I have sorted the data and we can easily see that shot put (SP) and discus (DS) have high loadings relative to the other variables in factor 1, implying thename “arm strength” for this factor. Specifically it means that e.g. shot put has correlation of 0.915 with factor 1 and a smaller correlation with the other two factors.
We are also interested in “Cumulative var”, the cumulative proportion of variance explained, and it should be a “high” number, where high is subjective. We should probably try fitting 4 factors, as I don’t think that 0.554 is that high and we might be able to do better. The low p-value in the end also rejects the hypothesis that 3 factors are sufficient.The output is now:
This looks more promising. 4 factors are sufficient (according to the p-value) and we have succesfully reduced the number of variables from 10 to 4. The interpretation of the factors is subjective, we cannot explicitly name them, but a good guess could be “arm strength”, “leg strength”, “speed/acceleration” and “running endurance” by looking at where the correlation is high for the variables. Javelin and 100m hurdles have high uniqueness and does not fit well into any of the factors. Maybe a little bit in factor 1 and 2, respectively.
The most important part is finding the number of factors. From here you can play around with estimation and rotation methods and sharpen the focus of the microscope, but it’s not that crucial.
Johnson and Wichern state that “the vast majority of attempted factor analyses do not yield clearcut results.” (p. 526) There is no guarantee that a factor analysis will lead to a satisfying description of meaningful factors. If you find yourself confused over the results of a factor analysis because it didn’t seem to “work”, there’s a good chance you did nothing wrong and that the factor analysis simply didn’t find anything interesting.
Also, factor analysis can lead to controversy if you’re trying to measure quantities such as “intelligence”, “compassion” or “potential”. Those quantities are difficult to measure and can they be responsible for events that we know we can measure, such as high SAT scores? Factor analysis cannot answer this. The quality of a factor analysis depends more on a “Wow” criterion, as the quality has not been quantified and if you can say “wow, I understand these factors” the application has probably been a success.
As I said before, the cumulative variance is a subjective measure and it is hard to know whether 0.621 is good or bad and intuitively, 60% is probably not that good. Either way, if you are able to reduce the dimensions from, say, 50 to 8 (or 10 to 4 in our case), it could help you explore your data better and make it more managable by understanding the covariance between the variables.
Please keep in mind that this is purely introductory and isn’t supposed to be a conclusion or an exhaustive analysis. Feel free to correct me if I missed something or did something wrong (I probably did). There are a lot of different methods for dealing with multivariate data and there is even more to Factor Analysis than in the scope of this article.
Make sure to follow my profile if you enjoy this article and want to see more!