Principal Components Analysis (PCA) In Python In Under 5 Minutes | by Wei Hao Khoong | Towards Data Science

Existing knowledge of linear algebra and matrix factorization
Python 3 programming proficiency
In a nutshell, PCA is arguably the most popular dimensionality reduction algorithm for datasets with a large number of features. It serves to remove highly correlated features and redundant ones, and also trims away noise in the data.
Recent machine learning (ML) algorithms and neural networks work well with datasets of high dimensions, often times with hundreds of thousands or over a million features. Note that due to recent advancements in compute technology (PCs/GPUs), it has been increasingly efficient to deal with high-dimensional data without the need to reduce their dimensions for use due to compute constraints. However, dimensionality reduction algorithms like PCA are still used for various considerations, such as reducing overfitting of the ML model by eliminating features with high correlation, by creating new uncorrelated variables that successively maximize variance.
Recall from literature that we can obtain the principal components matrix V via Singular Value Decomposition (SVD), a matrix factorization technique. Here, we have
where there are n principal components in V.
In other words, SVD decomposes the data matrix X into three matrices:
where U comprises of the left singular vectors, Σ is a diagonal matrix with the same dimensions as X containing the singular values, and V contains the right singular vectors/principal components.
In Python, we utilize Numpy’s svd() function to obtain all the principal components of X:
U, S, V_T = np.linalg.svd(X)# transpose to get V, with number of components as V.shape[1]V = V_T.T# first componentc1 = V[:, 0]# last (n-th) componentcn = V[:, -1]
Now, we are ready to reduce the dimensions of X using the components found! To reduce the dimension to d, we have to project the data onto the hyperplane defined by the first d components. Doing so results in the hyperplane preserving as much of the variance as possible. Mathematically, we compute the multiplication of X by the matrix W, which is the matrix comprising of the first d principal components:
In Python, we have the following:
W = V[:, :d] # d is the number of componentsX_d = X.dot(W) 
Fortunately for us, the Scikit-learn Python library has all this set up for us. Suppose that we have defined d and X previously. We can then use the following code to obtain the resulting X matrix after dimensionality reduction by utilizing the first d principal components:
from sklearn.decomposition import PCApca = PCA(n_components = d)X_d = pca.fit_transform(X)
To access the individual components, it is slightly different. We have to use the components_ variable:
# first componentc1 = pca.components_.T[:, 0]
A side remark here is that the components are horizontal vectors, so you’ll have to transpose them with a .T as above.
A common practice is to choose a number of dimensions that add up to a sufficiently large portion of the variance. Usually, a convention will be to retain 95% of the variance, but that depends on your needs. For example, if you seek to reduce the dimensions to visualize the data better, you will have to reduce it to 2 or at most 3 dimensions as humans cannot read more than 3 dimensions visually.
In this approach, we iterate through the proportions of variance explained by the components until a certain threshold is reached — the desired proportion of variance explained to be retained. We do so using the following function:
def get_pca_components(pca, var):    cumm_var = pca.explained_variance_ratio_    total_var = 0.    N_COMPONENTS = 0    for i in cumm_var:        N_COMPONENTS += 1        total_var += i        if total_var >= var:            break    return N_COMPONENTS
Here, cumm_var contains the proportions of variance explained by each component, from the largest to the smallest (n-th component). Next, we fit the PCA() object to the data X and extract the number of components such that 95% of the variance is retained:
pca = PCA().fit(X)n_components = get_pca_components(pca, 0.95)
Finally, we can transform the data to reduce its dimensions:
pca = PCA(n_components=n_components)X_d = pca.fit_transform(X)
Luckily for us, Scikit-learn has made life for us much simpler. We only need two lines of code:
pca = PCA(n_components=0.95)X_d = pca.fit_transform(X)
In the PCA() object, you can specify n_components to be a floating point between 0 and 1, to specify the ratio of variance to be preserved. You could also specify the number of components as a positive integer up to d, should you wish to approach the dimensionality reduction that way.
Now that you have obtained the transformed data X_d, you can proceed to analyze/visualize it and fit a ML model on it!