Robust & Non-Robust Methods For Data Reduction That You Should Know | by K.A. | Towards Data Science

The rapid advancement in networking technologies, emergence of cloud computing, mobile internet, and the advent of IoT devices the like of home appliances to self-driving vehicles have led to the creation, exchange, and storage of massive amount of data. But, much of these data may contain large amount of redundancy and noise. To improve prediction accuracy in Machine Learning (ML) models lies in selecting the smallest and most compact useful data set; a pain point that follows the data cleaning process. Reasons for reducing the size (dimension) of the data include achieving reduction in CPU and GPU computational working hour. Needless to say, no one desires to spend hundreds of dollars running cloud computing powers before one could settle and say that the prediction model is approximately accurate. Data reduction could be performed on the dataset in several ways. In data analysis and machine learning predictive modeling, we often ask questions like do all the attributes, a subset or at least one of the predictors help to explain the response?
Observations Reduction
In the case of whittling down the observations, the crude method is to slice the data in half, or any other ratio for that matter. Provided that the observations are truly randomized, it is not advisable to perform this despite the upside. For example, if we are to take large collection of tracks- and user-level music metadata then we could select the first few rows in the dataset and still would have good enough representation of the overall data population. Yet, this could spell trouble if the data are not randomized. Let’s say we have a dataset of music records; the first rows could be of data points specific only to a genre like “Jazz” — in this case the sample is biased. The safest option is in the form of stratified sampling. Let’s say, our model requires ‘genres’ as an explanatory attribute which consists of class variables; Jazz, Rock, Classical, Pop, etc. We could sample the same distribution percentage for each variable where it appears in the original dataset.
Reducing data observations is a no-brainer, but we cannot say the same with attributes. Dataset come in as many as hundreds of attributes, which shouldn’t be jam-packed inside the predictive model.
Simple Attribute Reduction Technique
When it comes to attributes reduction the tools and concepts get rather complicated. We could decide removing attributes by using specialized knowledge of the dataset, visualizing and performing simple statistical analysis, or use one of the robust statistical tools for data reduction. Let’s take at an example demonstrating visualization and easy statistical analysis on IMDB 5000 movie dataset. We want to examine whether ‘Genres’ has any effect on the IMDB score. That is, if we wanted to fit a model to predict IMDB Score, would we need ‘Genres’ as an explanatory attribute, or remove it altogether?
We begin by downloading IMDB dataset from the following link and run the below codes.
library(data.table)library(GGally)IMDB <- read.csv("movie_metadata.csv")sum(duplicated(IMDB))IMDB <- IMDB[!duplicated(IMDB), ]ggplot(imdb_score, aes(imdb_score)) + geom_histogram(binwidth = 0.5, color = "dark gray")
The IMDB score shows a distribution skewed to the right between 6 to 8. Except for some low frequency genres, this leads us to think that given enough sample count of genres we tend to find their average IMDB Score falls also between this range. This is a good start to consider that the attribute has little explanatory value on IMDB Score, but we shall confirm that through simple statistics using the mean.
‘Genres’ is a categorical attribute consisting of characters string of one or more combinations of genres. In this analysis, we avoid parsing the genres in the string and calculate their means to avoid convoluted addition of averages. Instead, we pick the topmost commonly used movie genres recipe in the movie industry.
sort(summary(IMDB$genres),dec=T) #topmost commonly used movie genres excluding ‘others’gnr.df <- as.data.frame(IMDB[,c("genres", "imdb_score")])gnr.df$'Comedy|Drama' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Comedy|Drama") 1 else 0)gnr.df$'Comedy' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Comedy") 1 else 0)gnr.df$'Horror' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Horror") 1 else 0)gnr.df$'Drama' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Drama") 1 else 0)gnr.df$'Documentary' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Documentary") 1 else 0)gnr.df$'Comedy|Drama|Romance' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Comedy|Drama|Romance") 1 else 0)gnr.df$'Comedy|Romance' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Comedy|Romance") 1 else 0)gnr.df$'Drama|Romance' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Drama|Romance") 1 else 0)gnr.df$'Crime|Drama|Thriller' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Crime|Drama|Thriller") 1 else 0)gnr.df$'Action|Crime|Drama|Thriller' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Action|Crime|Drama|Thriller") 1 else 0)gnr.df$'Action|Crime|Thriller' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Action|Crime|Thriller") 1 else 0)gnr.df$'Drama|Thriller' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Drama|Thriller") 1 else 0)gnr.df$'Crime|Drama' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Crime|Drama") 1 else 0)gnr.df$'Horror|Thriller' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Horror|Thriller") 1 else 0)gnr.df$'Crime|Drama|Mystery|Thriller' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Crime|Drama|Mystery|Thriller") 1 else 0)gnr.df$'Action|Adventure|Sci-Fi' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Action|Adventure|Sci-Fi") 1 else 0)gnr.df$'Comedy|Crime' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Comedy|Crime") 1 else 0)gnr.df$'Documentary' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Action|Adventure|Sci-Fi") 1 else 0)gnr.df$'Drama' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% "Comedy|Crime") 1 else 0)gnr.df$'Comedy' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% " Action|Adventure|Thriller") 1 else 0)gnr.df$'Horror' <- sapply(1:length(IMDB$genres), function(x) if (IMDB[x,10] %like% " Drama|Mystery|Thriller ") 1 else 0)means <- rep(0,17)for (i in 1:17) {means[i] <- mean(gnr.df$imdb_score[gnr.df[i+2]==1])}plot(means, type = "b", col = "dark red", ylim = c(1,8), xlab = "Genres")
The figure above shows that Genres, as a predictor, does not explain the response IMDB score very well, because on average IMDB Score for the highest used combination of genres fall narrowly between 6 and 7. In effect, we can reasonably and confidently reduce the dimension of our dataset by eliminating genres.
Attribute reduction: Correlation Analysis
The idea of correlation test is to identify the linear strength between two attributes or the intensity of the correlation between them. This analysis helps us to determine the interdependence of the attributes. On one hand, there may be complete independence, while on the other we may have collinearity; with one variable being a linear function of the other. To put it another way, as the two attributes tend to increase or decrease together, it can be difficult to separate the individual effect on the response for a given predictive model.
It must be pointed out that in correlation analysis attributes quantify the association between a single pair of attributes (i.e., two attributes) rather than between a larger number of them. When dealing with large dataset with several hundreds of attributes there will be a matrix of thousands of correlations. Even if the data is completely random you will find a high correlation among some pairs of attributes simply by chance.
It is worth to grasp the technical concepts behind the correlation analysis. Feel free to skip the definitions if they are not to your liking. We begin by defining two statistical concepts; Covariance and Correlation.
Covariance, correlation and the correlation coefficient
Covariance: Analogous to variance where it measures the squared deviation of a single random variable from its mean (Var (X) = E (X2) − (E (X))2). The covariance measures how two attributes fluctuate together. In other words, if the first attribute is large the second is large too for both to fluctuate together.
Correlation is simply the normalized version of covariance. The interpretation for this is that covariance quantity is rarely used as descriptive statistics to infer the strength of the relationship between two attributes because the covariance quantity depends on the unit measures of the two attributes. In other words, if we have two attributes that are measured using two separate units then the resulting quantity is a number which we could not relate to either units. Therefore, covariance is often standardized (i.e., zero mean and variance of one) by dividing it by the product of the standard deviations of the two attributes to give us a quantity called the correlation coefficient or Pearson’s coefficient of correlation.
It can be also shown mathematically that the correlation coefficient is the cosine of the angle between the two standardized column vectors (i.e., attributes’ values). For example, if the two individual attributes values are close, then the angle between the two vectors will be small and its cosine is close to +1, indicating high positive correlation. As can be seen in the figure below, the vectors (arrow-head), in opposite directions (positive and negative) are non-orthogonal or relatively parallel with reference to the horizontal axis (dark red) attribute to indicate some quantity of relationship.
Computing the correlation coefficient
To run correlation analysis, we compute the correlation coefficient which is rather straightforward operation using a statistical software like R. The correlation coefficient is a number between +1 and -1. The positive or the negative sign denotes the direction of the correlation. The positive sign signifies direct correlation; the increase of a unit value of the first attribute increases the second attribute by a unit value, while the negative sign indicates inverse correlation; the increase of a unit value of the first attribute deceases the second attribute by a unit value. However, if there is no apparent linear relationship between the attributes, then the correlation is zero or close to zero.
The previous statement comes with a few caveats:
1 — A near zero or zero correlation coefficient does not necessarily mean there exist no relation, but rather to mean no linear relation. This basically tells us that the two attributes have zero or near zero correlation and yet are dependent on one another, often in a nonlinear fashion.
2 — While the correlation coefficient is intended to quantify the strength of a linear association between a pair of variables, nonlinear association sometimes produce correlations that do not reflect the strength of the relationship.
3 — It is generally unwise to compute the correlation coefficient for a pair of variables that has obvious nonlinear relationship. Therefore, correlation is not an exhaustive summary of relationships within the attributes. It is a good practice to plot the attributes and give it visual inspection along with its statistical data summaries.
4 — The usage of correlation coefficient is rather dependent on the fact that a pair of attributes are a sample taken from a Gaussian (normal) distribution. Because some has much higher scatter than others it is worth inspecting the form of the distribution of each attribute and deliberate whether any of the attributes need to be transformed. For instance, the logarithmic transformation is often used to reduce positive skewness and produces a distribution closer to normal.
5 — The analysis ultimately helps identify some potential redundancy while examining the random attributes. But some degree of correlation should not always mean that one of it is redundant, although if the correlation is very high then one of it may be ‘nearly redundant’.
Take the following code sample to plot a dataset to illustrate points 1, 2 and 3.
set.seed(1)x = rnorm(600) #vector withdrawn from a normal distributiony = 1 + x + 0.5*x^2 #nonlinear function (1)plot(x,y)legend(-3,12, c("Pearson",round(cor(x,y),2)))y = 1 + x^2 #nonlinear function (2)plot(x,y)legend(-3,15, c("Pearson",round(cor(x,y),2)))
The left side of the figure, Pearson correlation coefficient is large (>0.8) and yet the attributes exhibit a nonlinear relationship. The right side of the figure another nonlinear relationship where the correlation coefficient is close to zero. without inspecting the attributes visually, we would not be able see the true nature of the association.
Let’s run another code sample but this time on Automobile dataset to show redundancy. The answer which we want to get from the dataset comes from the following: Could miles per gallon (mpg) be predicted from the other attributes?
library(GGally)path = "http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data"auto_data <- read.csv(path, header = F, sep = "", quote = "\"'", na.strings ='?')colnames(auto_data) <- c('mpg', 'cylinders', 'displacement','horsepower', 'weight', 'acceleration', 'model_year', 'origin', 'name')str(auto_data) #we see there is '?' value in the datasetround(cor(auto_data[,c(-7,-8,-9)]),2)ggcorr(auto_data[,c(-7,-8,-9)], label = T)
Cylinders and Displacement are highly correlated (+1). The presence of collinearity between them implies that the information cylinders provides about the response mpg is redundant in the presence of displacement. This is true even if we are to analyze the two attributes from a mechanical point of view.
Multicollinearity problem in high-dimensional space
We have discussed thus far that linear correlation can be detected through the correlation coefficient. But, in a high-dimensional space (i.e. large set of attributes) we could still see collinearity between three or more attributes without exhibiting high correlation between each pair. This phenomenon is called multicollinearity. We will not discuss this topic further, however, the solution may be relatively simple is that to use Principal Components Analysis (PCA), although that’s not all to it.
Principal Component Analysis (PCA)
PCA is perhaps the most widely used technique for data reduction. Even though the technique does not actually involve removing attributes in itself but rather achieving the reduction via transforming a set of correlated attributes to what is called principal components. Each principal component is a linear combination of the original attributes, and all the principal components are orthogonal to each other, so there is no redundant information. Furthermore, each new principal component should be pointing into the direction of the largest variance possible among the original data. Simply put, the principle components would give out the direction and magnitude of the variance which would help us predict the response better.
But why PCA is concerned about the variance in the dataset? Remember that information is all about variability. When an attribute has a zero variance it will not give away any valuable information because it means all its observations are the same. Also, PCA is concerned with transforming continuous attributes (i.e., quantitative).
The other benefit of PCA is that it also allows us to visualize large set of attributes. Imagine that we have a set of 10 attributes, if we are to create a scatterplot for each combination of attributes, to identify any possible correlation, we would end up having 45 plots! This is not a feasible endeavor when the number of attributes is much larger.
PCA is a form of unsupervised method because it deals with set of attributes without considering the response attribute. That is, without guidance on their effect on the response. We will come back to this point later. Furthermore, PCA does not assume the underlying probability distribution of the original dataset. That is, whether it is a normal (Gaussian)l distribution or otherwise. Fortunately, in the real-world, data are Gaussian distributed and PCA is typically a robust technique so we could breathe a sigh of relief.
To visualize the process, the following figure shows Plot 1A of a dataset expressed in x, y, z axes in a four-experiment dataset colored and shaped differently. We could see that the dataset exhibits some linearity. The first and second principal components, PC1 and PC2, together transform the dataset to a new set of basis vectors in the direction of the most variability. Plot 1B is the transformation of the dataset onto 2-dimensional space represented by PC1 and PC2 axes. We could see that the largest variance, or the dynamics, of the dataset is explained mostly along PC1 while PC2 variance is relatively closer to zero giving it less rank. The last sentence would be understood better if we projected the data in Plot 1B along the PC1 line in which it shows that PC1 is still close to the original dataset and has the largest variance, while PC2 projection would give less variance and is farthest away from the original dataset.
Many methods exist for the computation of principal components like the power method, spectral decomposition, eigenvalue decomposition (ED), singular value decomposition (SVD), and neural network algorithms. R software package offers us the built-in function prcomp() to compute PCA which uses the singular value decomposition (SVD) technique. Prior to calculating the principal components of a set of attributes they should be standardized to have variance of one and zero mean. In effect, the calculation finds the principal components from the correlation matrix rather than from the covariance matrix (Please see above on the correlation and covariance concepts).
Solving PCA: Eigenvectors and Covariance
This part can be skipped if it is not to your liking. We could explain PCA through eigenvalue decomposition (ED) technique to help us develop the technical intuition. The gist of it boils down to the following:
1- Subtract off the mean for each attribute in our dataset X.
2- Find the eigenvectors of the covariance of X.
3- Perform change of basis: The dataset that lies in an m-dimensional space (i.e., >= 2). PCA asks is there another set of basis vectors which is a linear combination of the original basis vectors that would express the dataset better? And by better we mean, the direction with the largest variance and the projection is closest to the original dataset. In linear algebra words, the change of basis is a matrix P that would transform the dataset X {V1, V2,...,Vn} into Y {V1,V2,...,Vn} where V is a vector. This new representation of data is expressed in this new set of basis vectors which are the principle components that are orthogonal to each other and pointing toward the direction of the maximal variance of the transformed dataset. Basically, we are asking for a matrix P that would fulfill this transformation.
a. Compute the covariance of the dataset Y and let’s name it CY which captures the covariance between all possible pairs of attributes. This square covariance matrix would have off-diagonal line the covariance values and on the diagonal line the variance values.
b. Optimize the covariance CY to achieve the goal of minimizing redundancy in the dataset by diagonalizing CY by the matrix P. That is, all off-diagonal values (covariance values) in CY should be zero.
c. The previous point refers to a linear algebra method that says an orthogonal matrix P which diagonalized CY are the eigenvectors of dataset X’s covariance and therefore its principle components.
d. The diagonal values of CY are the variances of the X dataset attributes.
e. Rank-order CY according to largest variance to help find the most important corresponding principle components in P.
While PCA is still a robust technique it comes certainly with a drawback which is that PCA’s decision about the importance of each principal component does not depend on the response attribute. In other words, the principal components with low variances that get relegated in rank may also be important for predicting the response, and in some cases even more important. Hence, there are other methods which could help overcome this bias. One could use the robust method Partial Least Squares (PLS) which in a nutshell makes use of the response to identify new attributes that not only estimate the old attributes well but are also related to the response, in effect, it finds a direction which explains the response and the predictor.
Summary
We have discussed various methods of data reduction for performing predictive modeling. The aim was to narrow down the choice of attributes and observations that would explain the response succinctly and to help run the chosen model rather quickly. Some reductions methods may require specialized knowledge of the dataset or visualizing and performing statistical analysis. Some methods are non-robust including observation reduction via stratified sampling, or attributes reduction by means of correlation analysis for 2 or more-dimensional space, and lastly, using PCA, a robust technique, for reducing the dimension space of the dataset without losing important information in a high dimensional space. We explained also that each method comes with its unique challenge or drawback, but this shouldn’t deter one to explore the methods further.