Dimensionality Reduction in Data Mining | by Uditha Maduranga | Towards Data Science

Big data is the large scale of data sets that have multi-level variables and that grow really fast. Volume is the most important aspect of big data. With the recent technological advancements happened in data processing and computer science field, the recent explosion of big data, in both number of records and attributes, has triggered few challenges in the data mining and overall in data science. Extremely big size of data in big data forms multidimensional datasets. Having multiple dimensions for the in a large data set makes the job of analyzing those or looking for any kind of patterns in the data really hard. High dimensional data can be obtained from various sources, depending on what kind of process one is interested in. Any process in nature progresses as a result of many different variables, some of which are observable or measurable and some are not. When we are to get data of any kind of simulation accurately, we get to deal with higher dimensional data.
Dimensionality reduction is the process of reducing the number of random variables or attributes under consideration. High-dimensionality data reduction, as part of a data pre-processing-step, is extremely important in many real-world applications. High-dimensionality reduction has emerged as one of the significant tasks in data mining applications. For an example you may have a dataset with hundreds of features (columns in your database). Then dimensionality reduction is that you reduce those features of attributes of data by combining or merging them in such a way that it will not loose much of the significant characteristics of the original dataset. One of the major problem that occurs with high dimensional data is widely known as the “Curse of Dimensionality”. This pushes us to reduce the dimensions of our data if we want to use them for analysis.
The curse of dimensionality refers to the phenomena that occur when classifying, organizing, and analyzing high dimensional data that does not occur in low dimensional spaces, specifically the issue of data sparsity and “closeness” of data.
Above sequence of graphs shows the issue of closeness of data when the underlying dimension increases. As the data space seen above moves from one dimension to two dimensions and finally to three dimensions, the given data fills less and less of the data space. The volume of data that is needed in order to maintain an accurate representation of the space, grows exponentially with the dimension.
When the dimension increases, with the sparsity, the distance between two independant points increases. That results in less similarity among the data points which will result in more error when it comes to most of the machine learning and other techniques used in data mining. To compensate we will have to feed very large number of data points but with higher dimensions it’s practically impossible and even it’s possible it will be inefficient.
In order to overcome above mentioned challenges that come with higher dimensional data, there is this need of reducing the dimensions of the data that is planned to be analysed and visualized.
Dimensionality reduction is accomplished based on either feature selection or feature extraction. Feature selection is based on omitting those features from the available measurements which do not contribute to class separability. In other words, redundant and irrelevant features are ignored. Feature extraction, on the other hand, considers the whole information content and maps the useful information content into a lower dimensional feature space.
One can differentiate the techniques used for dimensionality reduction as linear techniques and non-linear techniques as well. But here those techniques will be described based on the feature selection and feature extraction standpoint.
As a stand-alone task, feature selection can be unsupervised (e.g. Variance Thresholds) or supervised (e.g. Genetic Algorithms). You can also combine multiple methods if needed.
This technique looks for the variance from one observation to another of a given feature and then if the variance is not different in each observation according to the given threshold, feature that is responsible for that observation is removed. Features that don’t change much don’t add much effective information. Using variance thresholds is an easy and relatively safe way to reduce dimensionality at the start of your modeling process. But this alone will not be sufficient if you want to reduce the dimensions as it’s highly subjective and you need to tune the variance threshold manually. This kind of feature selection can be implemented using both Python and R.
Here the features are taken into account and checked whether those features are correlated to each other closely. If they are, the overall effect to the final output of both of the features would be similar even to the result we get when we used one of those features. Which one should you remove? Well, you’d first calculate all pair-wise correlations. Then, if the correlation between a pair of features is above a given threshold, you’d remove the one that has larger mean absolute correlation with other features. Like the previous technique, this is also based on intuition and hence the burden of tuning the thresholds in such a way that the useful information will not be neglected, will fall upon the user. Because of the those reasons, algorithms with built-in feature selection or algorithms like PCA are preferred over this one.
They are search algorithms that are inspired by evolutionary biology and natural selection, combining mutation and cross-over to efficiently traverse large solution spaces. Genetic Algorithms are used to find an optimal binary vector, where each bit is associated with a feature. If theth bit of this vector equals 1, then the that feature is allowed to participate in classification.If the bit is a 0, then the corresponding feature does not participate. In feature selection, “genes” represent individual features and the “organism” represents a candidate set of features. Each organism in the “population” is graded on a fitness score such as model performance on a hold-out set. The fittest organisms survive and reproduce, repeating until the population converges on a solution some generations later. You can have a further understanding on genetic algorithm here.
[Start] Generate random population of n chromosomes (suitable solutions for the problem)[Fitness] Evaluate the fitness f(x) of each chromosome x in the population[New population] Create a new population by repeating following steps until the new population is complete :      [Selection] Select two parent chromosomes from a population according to their fitness (the better fitness, the bigger chance to be selected)      [Crossover] With a crossover probability cross over the parents to form a new offspring (children). If no crossover was performed, offspring is an exact copy of parents.      [Mutation] With a mutation probability mutate new offspring at each locus (position in chromosome).      [Accepting] Place new offspring in a new population[Replace] Use new generated population for a further run of algorithm[Test] If the end condition is satisfied, stop, and return the best solution in current population[Loop] Go to step 2 [Fitness]
This can efficiently select features from very high dimensional datasets, where exhaustive search is unfeasible. But in most of the cases one might think that this is not worth the hassle which is quite true depending on the context as using PCA or built-in selection would be far more simpler.
In statistics, stepwise regression is a method of fitting regression models in which the choice of predictive variables is carried out by an automatic procedure. In each step, a variable is considered for addition to or subtraction from the set of explanatory variables based on some prespecified criterion. Usually, this takes the form of a sequence of F-tests or t-tests, but other techniques are possible, such as adjusted R2, Akaike information criterion, Bayesian information criterion, Mallows’s Cp, PRESS, or false discovery rate. You can get a better understanding of stepwise regression here.
This has two flavors: forward and backward. For forward stepwise search, you start without any features. Then, you’d train a 1-feature model using each of your candidate features and keep the version with the best performance. You’d continue adding features, one at a time, until your performance improvements stall. Backward stepwise search is the same process, just reversed: start with all features in your model and then remove one at a time until performance starts to drop substantially.
This is a greedy algorithm and commonly has a lower performance than the supervised methods such as regularizations etc.
Feature extraction is for creating a new, smaller set of features that still captures most of the useful information. This can come as supervised(e.g. LDA) and unsupervised(e.g. PCA) methods.
LDA uses the information from multiple features to create a new axis and projects the data on to the new axis in such a way as to minimizes the variance and maximizes the distance between the means of the classes. LDA is a supervised method that can only be used with labeled data. It consists of statistical properties of your data, calculated for each class. For a single input variable (x) this is the mean and the variance of the variable for each class. For multiple variables, this is the same properties calculated over the multivariate Gaussian, namely the means and the covariance matrix. The LDA transformation is also dependent on scale, so you should normalize your dataset first.
LDA is a supervised, so it needs labeled data. It offers variations (i.e. quadratic LDA) to tackle specific roadblocks. But the new features that are being created are hard to interpret using LDA.
PCA is a dimensionality reduction that identifies important relationships in our data, transforms the existing data based on these relationships, and then quantifies the importance of these relationships so we can keep the most important relationships. To remember this definition, we can break it down into four steps:
We identify the relationship among features through a Covariance Matrix.Through the linear transformation or eigendecomposition of the Covariance Matrix, we get eigenvectors and eigenvalues.Then we transform our data using Eigenvectors into principal components.Lastly, we quantify the importance of these relationships using Eigenvalues and keep the important principal components.
We identify the relationship among features through a Covariance Matrix.
Through the linear transformation or eigendecomposition of the Covariance Matrix, we get eigenvectors and eigenvalues.
Then we transform our data using Eigenvectors into principal components.
Lastly, we quantify the importance of these relationships using Eigenvalues and keep the important principal components.
The new features that are created by PCA are orthogonal, which means that they are uncorrelated. Furthermore, they are ranked in order of their “explained variance.” The first principal component (PC1) explains the most variance in your dataset, PC2 explains the second-most variance, and so on. you can reduce dimensionality by limiting the number of principal components to keep based on cumulative explained variance. The PCA transformation is also dependent on scale, so you should normalize your dataset first. PCA is a find linear correlations between the features given. This means that only if you have some of the variables in your dataset that are linearly correlated, this will be helpful.
t-SNE is non-linear dimensionality reduction technique which is typically used to visualize high dimensional datasets. Some of the main applications of t-SNE are Natural Language Processing (NLP), speech processing, etc.
t-SNE works by minimizing the divergence between a distribution constituted by the pairwise probability similarities of the input features in the original high dimensional space and its equivalent in the reduced low dimensional space. t-SNE makes then use of the Kullback-Leiber (KL) divergence in order to measure the dissimilarity of the two different distributions. The KL divergence is then minimized using gradient descent.
Here the lower dimensional space is modeled using t distribution while the higher dimensional space is modeled using Gaussian distribution.
Autoencoders are a family of Machine Learning algorithms which can be used as a dimensionality reduction technique. Autoencoders also use non-linear transformations to project data from a high dimension to a lower one. Autoencoders are neural networks that are trained to reconstruct their original inputs. Basically autoencoders consist with two parts.
Encoder: takes the input data and compress it, so that to remove all the possible noise and unhelpful information. The output of the Encoder stage is usually called bottleneck or latent-space.Decoder: takes as input the encoded latent space and tries to reproduce the original Autoencoder input using just it’s compressed form (the encoded latent space).
Encoder: takes the input data and compress it, so that to remove all the possible noise and unhelpful information. The output of the Encoder stage is usually called bottleneck or latent-space.
Decoder: takes as input the encoded latent space and tries to reproduce the original Autoencoder input using just it’s compressed form (the encoded latent space).
In the above diagram the middle layer represents the large number of input features within smaller number of neurons thus giving a dense and smaller representation of the inputs. Since it’s a neural network based solution for feature extraction it might need large sets of data to train.
I hope you enjoyed this article, thank you for reading!