Dimension Reduction Techniques with Python | by Dr. Dataman | Towards Data Science

(Revised on October 12, 2021)
Why Do We Need to Reduce Dimensionality?
A high-dimensional dataset is a dataset that has a great number of columns (or variables). Such a dataset presents many mathematical or computational challenges. The good news is that variables (or called features) are often correlated — high-dimensional data are dominated “superficially” by a small number of simple variables. We can find a subset of the variables to represent the same level of information in the data or transform the variables to a new set of variables without losing much information. Although high-power computing can somehow handle high-dimensional data, in many applications it is still necessary to reduce the dimensionality of the original data.
Principal Component Analysis (PCA) is probably the most popular technique when we think of dimension reduction. In this article, I will start with PCA, then go on to introduce other dimension reduction techniques. Python code will be included in each technique.
Dimensionality Reduction Can Also Find Outliers
Data scientists can use dimension reduction techniques to identify anomalies. Why? Don’t we just want to reduce the dimensionality? The intuition lies in outliers themselves. D.M.Hawkins said:” An outlier is an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a different mechanism.” Once dimensions are reduced to fewer principal dimensions, patterns are identified and then the outliers are revealed. We can say outlier detection is a by-product of dimension reduction, as described in the article “Anomaly Detection with Autoencoders Made Easy”.
I have written articles on a variety of data science topics. For ease of use, you can bookmark my summary post “Dataman Learning Paths — Build Your Skills, Drive Your Career” that lists the links to all articles.
It is worth to mention that in real life many classification problems are multi-class. If you even face the need to model a multi-class problem, see my post “A Wide Variety of models for Multi-class Classification”.
Principal Component Analysis (PCA)
The idea of principal component analysis (PCA) is to reduce the dimensionality of a dataset consisting of a large number of related variables while retaining as much variance in the data as possible. PCA finds a set of new variables that the original variables are just their linear combinations. The new variables are called Principal Components (PCs). These principal components are orthogonal: In a 3-D case, the principal components are perpendicular to each other. X can not be represented by Y or Y cannot be presented by Z.
Figure (A) shows the intuition of PCA: it “rotates” the axes to line up better with your data. The first principal component will capture most of the variance in the data, then followed by the second, third, and so on. As a result the new data will have fewer dimensions.
Let’s use the iris dataset to illustrate PCA:
# Use the iris dataset to illustrate PCA:import pandas as pdurl = “https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data"# load dataset into Pandas DataFramedf = pd.read_csv(url, names=[‘sepal length’,’sepal width’,’petal length’,’petal width’,’target’])df.head()
Notice this IRIS dataset comes with the target variable. In PCA, you only transform the X variables without the target Y variable.
Standardization: All the variables should be on the same scale before applying PCA, otherwise, a feature with large values will dominate the result. This point is further explained my post “Avoid These Deadly Modeling Mistakes that May Cost You a Career”. Below I use StandardScaler in scikit-learn to standardize the dataset’s features onto the unit scale (mean = 0 and variance = 1).
from sklearn.preprocessing import StandardScalervariables = [‘sepal length’, ‘sepal width’, ‘petal length’, ‘petal width’]x = df.loc[:, variables].valuesy = df.loc[:,[‘target’]].valuesx = StandardScaler().fit_transform(x)x = pd.DataFrame(x)
There are four features in the original data. So PCA will provide the same number of principal components.
from sklearn.decomposition import PCApca = PCA()x_pca = pca.fit_transform(x)x_pca = pd.DataFrame(x_pca)x_pca.head()
What are the variances explained by each of the principal components? Use pca.explained_variance_ratio_ to return a vector of the variance:
explained_variance = pca.explained_variance_ratio_explained_variancearray([0.72770452, 0.23030523, 0.03683832, 0.00515193])
It shows the first principal component accounts for 72.22% variance, the second, third and fourth account for 23.9%, 3.68%, and 0.51% variance respectively. We can say 72.22 + 23.9 = 96.21% of the information is captured by the first and second principal components. We often want to keep only the significant features and drop the insignificant ones. A rule of thumb is to keep the top principal components that capture significant variance and ignore the small ones.
We can plot the results using the first two components. Let’s append the target variable y to the new data x_pca:
x_pca[‘target’]=yx_pca.columns = [‘PC1’,’PC2',’PC3',’PC4',’target’]x_pca.head()
The outcome shows the data are separable in the new space.
import matplotlib.pyplot as pltfig = plt.figure()ax = fig.add_subplot(1,1,1) ax.set_xlabel(‘Principal Component 1’) ax.set_ylabel(‘Principal Component 2’) ax.set_title(‘2 component PCA’) targets = [‘Iris-setosa’, ‘Iris-versicolor’, ‘Iris-virginica’]colors = [‘r’, ‘g’, ‘b’]for target, color in zip(targets,colors): indicesToKeep = x_pca[‘target’] == target ax.scatter(x_pca.loc[indicesToKeep, ‘PC1’] , x_pca.loc[indicesToKeep, ‘PC2’] , c = color , s = 50)ax.legend(targets)ax.grid()
How do we use PCA to detect outliers? Let me give you the intuition. After the transformation, the “normal” data points will align along the eigenvectors (new axes) with small eigenvalues. The outliers are far away from the eigenvectors with large eigenvalues. Therefore the distances between each data point to the eigenvectors become a measure for the outlier. A large distance indicates an anomaly. For more information, see “Anomaly Detection with PyOD”.
Kernel PCA (KPCA)
PCA applies linear transformation, which is just its limitation. Kernel PCA extends PCA to non-linearity. It first maps the original data to some nonlinear feature space (usually higher dimension), then applies PCA to extract the principal components in that space. This can be understood by Figure (B). The graph in the left shows the blue and red dots can not be separated using any linear transformation. But if all the dots are projected onto a 3D space, the result becomes linearly separable! We then apply PCA to separate the components.
Where does intuition come from? Why does component separation become easier in a higher-dimensional space? This has to go back to the Vapnik-Chervonenkis (VC) theory. It says mapping into a higher dimensional space often provides greater classification power.
The following Python code makes a circle plot consisting of red and blue dots. Obviously there is no way to separate the red and blue dots with a line (linear separation).
print(__doc__)import numpy as npimport matplotlib.pyplot as pltfrom sklearn.decomposition import PCA, KernelPCAfrom sklearn.datasets import make_circlesnp.random.seed(0)X, y = make_circles(n_samples=400, factor=.3, noise=.05)plt.figure(figsize=(10,10))plt.subplot(2, 2, 1, aspect='equal')plt.title("Original space")reds = y == 0blues = y == 1plt.scatter(X[reds, 0], X[reds, 1], c="red",s=20, edgecolor='k')plt.scatter(X[blues, 0], X[blues, 1], c="blue",s=20, edgecolor='k')plt.xlabel("$x_1$")plt.ylabel("$x_2$")
However, when we project the circle to a higher dimensional space and separate using PCA, the data observations against the first and second principal components are separable! Below is the result that the dots are plotted against the first and second principal components. I draw a line to separate the red and blue dots. In KernelPCA we specify kernel=’rbf’, which is the Radial Basis Function, or the Euclidean distance. The RBFs are commonly used as a kernel in machine learning techniques such as the Support Vector Machine (SVM).
kpca = KernelPCA(kernel=”rbf”, fit_inverse_transform=True, gamma=10)X_kpca = kpca.fit_transform(X)pca = PCA()X_pca = pca.fit_transform(X)plt.scatter(X_kpca[reds, 0], X_kpca[reds, 1], c=”red”,s=20, edgecolor=’k’)plt.scatter(X_kpca[blues, 0], X_kpca[blues, 1], c=”blue”,s=20, edgecolor=’k’)x = np.linspace(-1, 1, 1000)plt.plot(x, -0.1*x, linestyle=’solid’)plt.title(“Projection by KPCA”)plt.xlabel(r”1st principal component in space induced by $\phi$”)plt.ylabel(“2nd component”)
If we specify the kernel to be “linear” as the code below (KernelPCA(kernel=’linear’), it becomes the standard PCA with only linear transformation, and the red and blue dots are not separable.
kpca = KernelPCA(kernel=”linear”, fit_inverse_transform=True, gamma=10)X_kpca = kpca.fit_transform(X)pca = PCA()X_pca = pca.fit_transform(X)plt.scatter(X_kpca[reds, 0], X_kpca[reds, 1], c=”red”,s=20, edgecolor=’k’)plt.scatter(X_kpca[blues, 0], X_kpca[blues, 1], c=”blue”,s=20, edgecolor=’k’)x = np.linspace(-1, 1, 1000)plt.plot(x, -0.1*x, linestyle=’solid’)plt.title(“Projection by KPCA”)plt.xlabel(r”1st principal component in space induced by $\phi$”)plt.ylabel(“2nd component”)
Linear Discriminant Analysis (LDA)
The origin of LDA is different from PCA. PCA is an unsupervised learning method that transforms the original features to a set of new features. We do not care about whether the new set of features can provide the best discriminatory power for the target variable. In contrast, Linear Discriminant Analysis (LDA) seeks to preserve as much discriminatory power as possible for the dependent variable, while projecting the original data matrix onto a lower-dimensional space. LDA is a type of supervised learning technique. It utilizes the classes in the dependent variable to divide the space of predictors into regions. All the regions should have linear boundaries. Thus the name linear comes from. The model predicts that all observations within a region belong to the same class of the dependent variable.
LDA achieves the above goal in three major steps. First, it calculates the separability between different classes of the dependent variable, which is called the between-class variance, as shown in (1) of Figure LDA. Second, it calculates the distance between the mean and the samples of each class, which is called the within-class variance, as shown in (2). Then it constructs the lower-dimensional space with this criterion: maximizing the between-class variance and minimizing the within-class variance. The solution to this criterion is to compute the eigenvalues and eigenvectors. The resulting eigenvectors represent the directions of the new space, and the corresponding eigenvalues represent the length of the eigenvectors. Thus, each eigenvector represents one axis of the LDA space, and the eigenvalue represents the length of that eigenvector.
I will use the “Red Wine Quality” dataset in the Kaggle competition. This dataset has 11 input variables and one output variable ‘quality’.
import matplotlib.pyplot as pltfrom sklearn.decomposition import PCAfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysiswine = pd.read_csv(‘winequality-red.csv’)wine.head()
For the sake of simplicity, I re-group the output variable into three values. wine[‘quality2’] = np.where(wine[‘quality’]<=4,1, np.where(wine[‘quality’]<=6,2,3)).
The following code performs PCA and LDA.
X = wine.drop(columns=[‘quality’,’quality2'])y = wine[‘quality2’]target_names = np.unique(y)target_namespca = PCA(n_components=2)X_r = pca.fit(X).transform(X)lda = LinearDiscriminantAnalysis(n_components=2)X_r2 = lda.fit(X, y).transform(X)
Then plot both the results of PCA and LDA:
# Percentage of variance explained for each componentsprint(‘explained variance ratio (first two components): %s’ % str(pca.explained_variance_ratio_))plt.figure()colors = [‘navy’, ‘turquoise’, ‘darkorange’]lw = 2for color, i, target_name in zip(colors, target_names, target_names): plt.scatter(X_r[y == i, 0], X_r[y == i, 1], color=color, alpha=.8, lw=lw, label=target_name)plt.legend(loc=’best’, shadow=False, scatterpoints=1)plt.title(‘PCA of WINE dataset’)plt.figure()for color, i, target_name in zip(colors, target_names, target_names): plt.scatter(X_r2[y == i, 0], X_r2[y == i, 1], alpha=.8, color=color, label=target_name)plt.legend(loc=’best’, shadow=False, scatterpoints=1)plt.title(‘LDA of WINE dataset’)plt.show()
LDA is suitable for a multi-class classification problem. If you even face the need to model a multi-class problem, see my post “A Wide Variety of models for Multi-class Classification”.
Singular Value Decomposition (SVD)
SVD is a data summary method similar to PCA. It extracts important features from data. But there is one more advantage of SVD: reconstructing the original dataset into a small dataset. So it has wide applications such as image compression. For example, if you have a 32*32 = 1,024-pixel image, SVD can summary it into 66 pixels. The 66 pixels can retrieve 32*32 pixel images without miss any important information.
SVD has been instrumental in linear algebra, but it seems “not nearly as famous as it should be” as stated in the classic textbook “Linear Algebra and Its Applications” by Gilbert Strang. To introduce SVD properly, it is essential to start with matrix operation. If A is a symmetric real n × n matrix, there exists an orthogonal matrix V and a diagonal D such that
Columns V are eigenvectors for A and the diagonal entries of D are the eigenvalues of A. This process is called the Eigenvalue Decomposition, or EVD, for matrix A. It tells us how to choose orthonormal bases so that the transformation is represented by a matrix with the simplest possible form, that is, diagonal. (For readers who would like to go over the steps to diagonalize a matrix, here is a good example.) The term orthonormal means two vectors are orthogonal or perpendicular.
Extending the symmetric matrix, the SVD works with any real m × n matrix A. Given a real m× n matrix A, there exists an orthogonal m × m matrix U, an orthogonal matrix m × m V, and a diagonal m × n matrix Σ such that
Note that an orthogonal matrix is a square matrix such that the produce of itself and its inverse matrix is an identity matrix. A diagonal matrix is a matrix in which the entries other than the diagonal are all zero.
Below I will again use the iris dataset to show you how to apply SVD.
from numpy import *import operatorimport matplotlib.pyplot as pltimport pandas as pdfrom numpy.linalg import *url = “https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data"# load dataset into Pandas DataFramedf = pd.read_csv(url, names=[‘sepal length’,’sepal width’,’petal length’,’petal width’,’target’])# Only the X variablesdata = df[[‘sepal length’,’sepal width’,’petal length’,’petal width’]]#calculate SVDn = 2 # We will take two Singular ValuesU, s, V = linalg.svd( data )# eye() creates a matrix with ones on the diagonal and zeros elsewhereSig = mat(eye(n)*s[:n])newdata = U[:,:n]newdata = pd.DataFrame(newdata)newdata.columns=[‘SVD1’,’SVD2']newdata.head()
You can compare the result of SVD to that of PCA. Both achieve similar outcomes.
# Add the actual target to the data in order to plot itnewdata[‘target’]=df[‘target’]fig = plt.figure()ax = fig.add_subplot(1,1,1) ax.set_xlabel(‘SVD 1’) ax.set_ylabel(‘SVD 2’) ax.set_title(‘SVD’) targets = [‘Iris-setosa’, ‘Iris-versicolor’, ‘Iris-virginica’]colors = [‘r’, ‘g’, ‘b’]for target, color in zip(targets,colors): indicesToKeep = newdata[‘target’] == target ax.scatter(newdata.loc[indicesToKeep, ‘SVD1’] , newdata.loc[indicesToKeep, ‘SVD2’] , c = color , s = 50)ax.legend(targets)ax.grid()
t-distributed Stochastic Neighbor Embedding (t-SNE)
t-SNE is developed by Laurens van der Maaten and Geoggrey Hinton. It is a machine learning algorithm for visualization that presents embedding high-dimensional data in a low-dimensional space of two or three dimensions.
What is the best way to present the above three-dimensional Swiss roll to two-dimensional? Intuitively we want to “unroll” the Swiss roll to a flat cake. In mathematics, it means similar points will become nearby points and dissimilar points will become distance points.
Figure (C) shows another example. It is a 3-dimensional tetrahedron with data points clustering in the vertex corners. If we just collapse the 3-dimensional graph to a 2-dimensional graph like Panel (A) does, it does not work well because the group (A) becomes the center cluster. In contrast, Panel (B) is probably a better 2-D exhibit that that preserves the far distances between Cluster (A)-(E) while keeps the local distances of points in each cluster. t-SNE, a nonlinear dimension reduction technique, is designed to preserve the local neighborhoods. If a set of points cluster together on a t-SNE plot, we can be fairly certain that these points are close to each other.
t-SNE models the similarities among points. How does it define similarities? First, it is defined by the Euclidean distance between point Xi and Xj. Second, it is defined as the conditional probability that “the similarity of data point i to point j is the conditional probability p that point i would pick data j as its neighbor if other neighbors were picked according to their probabilities under a Gaussian distribution.” In the following conditional expression, if point j is closer to point i than other points, it has a higher probability (notice the negative sign) to be chosen.
t-SNE aims to match the above conditional probability p between j and i as well as possible by a low-dimensional space q between point Yi and Yj, as shown below. The probability q follows a fat-tailed Student-t distribution, thus the “t” in t-SNE comes from.
The next step is to find Yi such that the distribution q will be as close to the distribution p as possible. t-SNE uses the gradient decent technique, an optimization technique, to find the values.
Below I demonstrate how the t-SNE technique is used with the iris dataset.
from sklearn.manifold import TSNEfrom sklearn.datasets import load_irisfrom sklearn.decomposition import PCAimport matplotlib.pyplot as pltiris = load_iris()X_tsne = TSNE(learning_rate=100).fit_transform(iris.data)X_pca = PCA().fit_transform(iris.data)plt.figure(figsize=(10, 5))plt.subplot(121)plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=iris.target)plt.subplot(122)plt.scatter(X_pca[:, 0], X_pca[:, 1], c=iris.target)
For more information, this post “how to use t-SNE more effectively” provides more discussions.