A Gentle Introduction To Dimensionality Reduction
By: Antonieta Mastrogiuseppe
Tags: Correlation Analysis, Principal Component, Factor Analysis, Unsupervised Learning, Dimensionality Reduction

A Gentle Introduction To Dimensionality Reduction

Taken by the author

As the word exploratory suggest, Exploratory Factor Analysis (EFA) is that preliminary examination seeking to understand relationship between variables. When first exposed to the data, the researcher might not hold any assumption about the input variables and their relationship. The researcher starting point is to understand patterns and factors influencing such patterns to identify relationship between variables. EFA, the topic of this article is a statistical technique aiming to identify underlying factors influencing such variables.

To develop on this topic, I will next step into the field of model building (my favorite subject) to explain the fundamental of EFA. In building a model, a given number of input variables (independent variables) are collected which are identified as relevant to explain variations in the output variable (also referred as the dependent variable). Assuming a relationship of some sort between the inputs and the output variable, one seeks to measure the amount of variation of the output variable that can be explained by an input variable holding all the other constants, “ceteris paribus”. Because of “noise” such as unobserved human behavior that one is not able to capture, lack of time and resources to collect the totality of inputs influencing the output variable, an error term representing anything else that is not included into the model is introduced. Mathematically, this is formulated as follow:

On running the above linear regression model, one thing to examine is the degree of connection between the input variables (if they explain variations of the dependent variable there must be a relationship of some sort between them). Known as ‘Correlation analysis’, we aim to measure the degree of the relationship between two or more input variables. Because too many input variables will lead to a too perfectly fitted model, one needs to make sure only relevant variables, namely those explaining a significant proportion of variation of the dependent variable are included in the model. If overfitted, a model becomes too specific to the data it has been trained on as it memorizes the relationship being modelled. Not generalized enough, when the model is run on unseen data it is then unable to understand the ‘updated’ relationship now present in the data hence leading to unreliable predictions. Thus, too many variables is not good but one wants to make sure the relevant ones are included. This is where EFA comes into play. In machine learning terms, techniques such as EFA, Principal Component Analysis (PCA) fall into the category of unsupervised learning methods, the latter a type of technique where ONLY input variables are analyzed to learn from the data and understand patterns between the inputs and identify factors with significant influence.

Now, let us step back and ask: can one identify a given number of ‘factors’ with significant influence that is common to multiple input variables? Could it be the case that some of the variation among correlated inputs can be explained by ‘common factors’? If this is the case, is it also possible that a ‘unique factor’ variation type is identified as influencing a single input variable? Is also there an inherited proportion of error in this correlated relationship that simply cannot be measured? In EFA, the input variables are a function of: common factors, unique factors and an error term. Technically, this is written:

In here, common variance are the ones we are interested! EFA focus on common factors that influence multiple variables so that these factors can be used instead allowing to reduce the number of variables into the model hence reducing its complexity without compromising its predictive power.

A key concept under exploratory factor analysis, is the one on ‘latent variables’. In the form of variables, these are unobserved but influential factors that explain a significant proportion of the variation common among the input variables. In this article, I will use the definitions ‘latent variables’ and ‘factors’ interchangeably. At the very top level, latent variables are generated from the combination of the original input variables. Thus, the number of factors that are generated equals the number of original variables. Based on the proportion of variance contributed by each original input, a weighted value is produced, known as factor loadings. These are important! Factor loadings are coefficients which range from -1 to 1 measuring the correlation, magnitude of influence of a given factor on a given input. The main objective is to identify the number of latent variables where the largest amount of information contained in the original input variables is preserved as measured by the factor loadings.

OK, so How are factor loadings calculated? Ok time to step back again and go in more detail. In EFA, the number of latent variables (new variables) generated are equal to the number of original input variables. On this, a well known correlation matrix between variables and eigenvector and eigenvalue for the correlation matrix are derived. In pair, and for each latent variable an Eigenvector and Eigenvalue is generated. The former, a vector of weight for each eigenvalue indicates a direction and the latter measuring the amount of variance explained by a given latent variable in that direction. Taking the square root of the eigenvalue (which are the standard deviations of the latent variables) are the singular value decomposition that when multiplied by the Eigenvector generates the factor loading. Thus, the above transformation generates a factor loading for each original input variable across factors. This is important! Factor loadings reflect the degree of correlation of each original input variable with the latent variable. The eigenvector with the highest eigenvalue is the first latent variable. Thus, in decreasing order the first latent variable is the most significant latent variable accounting for most of the (common) variance in the data, followed by the second one and so on.

As additional latent variables are added less variance in the data is explained. Conventionally, a factor loading with a value greater than 0.3 identifies a relevant variable. As in correlation, a “high” correlation value is a subjective point when defined. Now, a factor loading greater than 1 are sometime the case. These are taken as an indication of spurious results, caused by what is known as improper solutions, a sign of something wrong with the output.

Next, I will illustrate the above by implementing EFA and Principal Component Analysis (PCA). In doing this, I aim to emphasis the power of these statistic techniques in unveiling and describing the underlying structure of a dataset. In comparing these two, only key similarities and differences are covered in this article mainly to highlight the use of one or the other based on the analysis at hand.

This paper uses a dataset of 98,586 observations and a total of 22 input variables from a private company. Due to confidentiality reasons data source cannot be disclosed. However, the analysis can be easily reproduced using similar variable types. By performing EFA and PCA on the above dataset, I aim to establish a sensible approach when implementing a dimensionality reduction technique rather than focus on the output per se.

The analysis is composed of three phases: Phase I: Analysis of first output based on 14 variables on demographic data. In this first group of input variables a relatively low number of missing values are found hence making available a relatively larger dataset to base the analysis on. Phase II: Some adjustments on input treatment are proposed based on first results. As opposed to phase I, input variables are now normalized to scale variables so they all range between (0–1). Phase III: Additional variables are incorporated based on marketing, consumer behavior data (i.e. consumption, communication channel) totaling 22 variables for analysis. These are added to account for model misspecification as the latter has been identified as a relevant element likely to result in improper solutions. However, larger number of missing values are present on these second set of input variables added leading to a smaller dataset to run the analysis on.

Phase I: Initial results

Variable treatment:

· Entries with missing values for certain input variables are removed as these represented only 0.03% of total data points reducing the number of observations from 98,594 to 98,586.

· Due to the presence of outliers, using the interquartile range approach, data points above Q3+(1.5*IQR) are capped hence removing values lying on the boxplot beyond the upper whisker.

Main Result:

Heywood cases detected. Heywood cases are the result of factor loadings which variance is constrained to negative or near zero value. As we have seen above, factor analysis is based on the correlation matrix of the variables under analysis. Under Heywood cases, any linear combination of a group of variables produces a not positive definite covariance matrix. Technically speaking, as variance is a squared root output of standard deviation, it cannot be negative, being this an indication of biased/unreliable factor loading estimates.

This is what is known as an improper solution where the estimates (factor loadings) are constrained within a defined range leaving the underlying problem (negative variance) unrectified hence allowing for invalid estimates. In here, factor loadings are biased hence in need to be rectified as interpretation of latent variables does not sensibly fit the purposes of the analysis. Under this scenario, the software yields the following message (or similar one): “covariance matrix of latent variables is not positive definite”. In here, the researcher with domain on the input variables could opt to run a summary of the model with a covariance table session to inspect variables that are highly correlated and collate/remove them accordingly.

Elements to look at to rectify Heywood cases mainly are: 1) high values within variables, 2) model misspecification and 3) sample size. As I am using a relatively large dataset, and assuming no domain on the variables, I will first attempt different inputs models to account for model misspecification, namely adding first and (maybe) removing later input variables as my interest is to identify latent variables highlighting the most significant input variables among all available ones. Before this, I will next first normalize the input variables to make sure they all range within the same band of values thus removing the probability of higher values leading to biased results.

Based on the above, the first rectification is undertaken.

Phase II: Variables Normalization.

Variable treatment:

· In this phase, variables are normalized keeping everything else the same. Using the min-max Scaling variables are scaled and data points fixed to a range 0–1. The above aims to improve the algorithm’s performance by preventing biased towards features with larger values hence facilitating minima gradient descendent and convergence.

Main Result:

Heywood cases are still present. Further adjustments in terms of model specification are performed.

Phase III: Normalization and Model Specification.

Variable treatment:

· In the third and final phase, more variables are added. This set of input variables are found to have a greater proportion of missing values. In this phase and again, I am removing entries with missing values for the latest set of variables added bringing the total number of observations down to 65,605. Median infill for variables with missing values is of common practice in order to preserve the original number of observations. In this article, I am removing missing values given the relatively large dataset left after this so I make sure the analysis is kept under its purposes and no other bias elements are brought into the analysis.

· Because machine learning techniques are proven to allocate higher weight to features with higher value range, variables are left normalized as performed on previous phase.

Main Result:

No improper solution found.

What does the final output look like? How is this interpreted? Depending on the package used, different information and metrics are displayed within the EFA output. However, the analysis from final output should still reach the same conclusion in terms of relevant factors. Among the different packages, I am using the factanal function from R, which is based on maximum likelihood estimator (MLE). Even though MLE might be better known, other packages have a parameter enabling to set up a given estimator. The reader should familiarize himself(herself) with the different estimation methods performed by the other packages, which are out of the scope of this article.

The below line of code shows the main arguments from the factanal function:

From the factanal function, the following arguments are defined: dataframe, number of factors (one suspects as relevant) and factor rotation method. Simply put, for a given number of factors, there is no unique factor loading matrix solution for all variables involved but rather an infinite (factor loading) number of solutions. In a two-dimension space and for visualization and interpretation purposes, two factors can be thought of two axes with factor loadings as values representing the variance contribution from each original input variable. The method of rotation (for the best fit based on the variance direction) of these axes is stated as ‘varimax’ for orthogonal factors, perpendicular factors (axis), namely an assumption of no correlation between factors. From the Greek word meaning ‘upright angle’, this simply means axes that are perpendicular in a coordinate space hence no correlated. Again, I suggest for the reader to familiarize herself(himself) with the different rotation methods.

Output from running the factanal function is shown below. Let us remind ourselves, one is selecting the number of latent variables that capture the highest amount of (common) variance among the original input variables. Thus, EFA aims to determine an optimal number of latent variables which is less than the total number of original variables. Output below is based on a 3-factor solution. As can be seen, the uniqueness metric is the first provided. This ranges from 0 to 1. Based on the unique variance type contribution of each original variable, one way to interpret this metric is by obtaining what is known as the communality measure. Communality is the proportion of variance, common to a set of the original input variables, contributed by variable ith and accounted for by the number of selected factors (in this case 3). This is calculated by deducting 1 from the uniqueness value.

So yes, lowest the uniqueness value, greater the variance contribution of a given input variable. For instance, from the output below, education has a significant contribution of 92% of total variance in the data that is common among the original variables, this is (1–0.079 = 0.921). In fact, this variable has a factor loading close to 1, namely 0.949 hence highly correlated to factor 1 followed by a managerial role variable. Again, let us remind ourselves, the factor loadings represent the correlation of a given variable with respect to a given factor. It ranges from -1 to 1. As we have discussed all along this analysis and as we will demonstrate next, factor loadings are used to understand the variable with the highest common variance contribution within a factor. Importantly, this facilitates the description hence labelling of the latent/unobserved variable/factor.

Now, how is ‘high’ variance defined? This is part of the subjectivity embedded in EFA. Thus, this should mainly be stablished by the researcher’s interpretation and how this fit the analysis at hand. OK so given this, how are the loadings interpreted so that these are described and defined?

From the output, variables where factor loadings are greater than 0.3 are highlighted. Higher this value, higher the correlation of a given input variable relative to the factor.

As it can be seen, Factor 1 identifies variables such as level of education, social status, house price, income, demand, managerial role position as relevant, which they are all associated with a latent variable of an individual’s Socioeconomic level.

Factor 2, identifies variables such as renting, never worked or unemployed status, area density, number of people in the house, benefit status as relevant to a latent variable on social security transfer, social benefits, allowances indicator.

Finally, Factor 3 identifies variables such as number of people in a household, house size, income from other assets for a latent variable measuring standard of living, purchasing power.

The final part of the output shows the SS loading, proportion variance, cumulative variance and the hypothesis test. The SS loading is the sum squared loadings and where this value is greater than 1 is taken as an indication of a factor with significant explanation power. These decrease in value as shown by the yellow cells as subsequent factors explain less variation in the data. Factors with values above this threshold are kept. The cumulative variance tells us that 46% of the common variance in the data is explained when the first three factors are accounted for. This might appear low and an indicative of the need to include more factors, greater than 3 in this case. Again, this is based on the analysis at hand. Tests with varies number of factors as well as different number of original input variables can be run to validate how this last section changes based on subsequent adjustments.

The eigenvalues graph can be plotted to visualize the above. In R, Dino’s ‘paran’ package can be used to run a parallel analysis Scree Plot. As shown by the graph, the eigenvalue is plotted in the y-axis (total common variance explained by a factor) and number of factors in the x-axis. As we have learned, eigenvalues represent the total communality variance across all original variable explained by a given factor. Thus, we want to choose the number of factors with a value greater than 1. This is used as a criterion for factor selection where 1 factor is expected to explain more variance that 1 original variable. As expected, one can see factor number 1 accounting for the highest variance and as shown by the blue curve this decreases from the first to the last factor. In fact, the three-factor solution is shown and validated as sufficient number of factors to be used for this analysis.

Both being statistical techniques employed for dimensionality reduction, for comparison purpose this article additionally touches on Principal Component Analysis (PCA) thus to highlight key differences when compared to EFA.

Equivalent to a latent variable, a principal component is a linear weighted combination of the observed original variables. Thought of an axis in a coordinate space, PCA minimizes the sum of squared distance of the data with respect to the principal component (the axis). The number of components extracted aim to retain most of the variance observed and brought about by the original variables. In here and similar to EFA, variables that are highly correlated hence contributing to most variance in the data are represented in the first component, this is the one explaining most of the variance in the data followed by the second component and subsequently where less variance is explained as more components are added. For PCA, the total number of original variables are transformed into a new coordinate system with an equal number of new variables (principal components). From the linear transformation, the eigenvalues and eigenvectors are generated. Eigenvalues representing the amount of variance contributed by each variable, and eigenvectors as weights representing the direction of the variance that best fit the data, components are iteratively derived which in nature are orthogonal. Principal component and EFA share similar assumptions which in this analysis have been accounted when treating the original input variables as described above. Detailed description of these are out of the scope of this article but a subject for further investigation for the researcher to undertake.

Among the different packages in R, the princomp() function is used to perform PCA and output is shown below.

where parameter ‘cor’ defines within the function the use of the correlation matrix to derive the principal components.

As one can see, 22 components have been created, which equals the number of original variables. In here, we read the “Proportion of Variance” as the maximum amount of variance explained by a given component. “Cumulative Proportion” will provide a running total, maximum amount of variation that is explained for a given number of components. For Comp.3 this is 51%. In fact, by extracting all the 22 components one will be accounting for all variation in the data, this is where “Cumulative Proportion” equals to 1. However, we have learned PCA aims for an optimal number of components which is less than the number of original variables. Now, and this is important: here the word maximum becomes relevant when it comes to understand a key difference between PCA and EFA. Simply put, PCA reports on the maximum amount of variation in the observed original input variables explained by the components. In contrast, EFA reports on the total amount of variation that is common among a given number of the original input variables as a result of factors that cannot be measured and/or observed. Thus, PCA holds onto the strong assumption that common variance is equal to total variance. This is,

Thus, which one to use is mainly based on the analysis at hand.

Eigenvalues on a scree plot are shown below for PCA. Variance plotted in the y-axis and number of components in the x-axis, this clearly shows the decreasing rate at which variance is explained as more principal components are added.

Conclusion

· This article employed a data set of size 98,594 x 22. It performed EFA and PCA as statistic techniques to reduce dimensionality in a dataset.

· Under EFA, the number of latent variables represent the common variance present in the data as a result of unobserved factors.

· Under PCA, the number of components extracted explain the maximum variance present in the data from the observed original variables.

· Both techniques aim for an optimal solution where the number of components or the number of factors is less than the number of original input variables.

· For demonstration purposes this article used the same dataset to run both techniques. However, based on the analysis at hand the researcher should aim to identify the most appropriate technique to use.