Andrew Ng’s Machine Learning Course in Python (Kmeans-Clustering, PCA) | by Benjamin Lau | Towards Data Science

Second last part of the series, we look at unsupervised learning algorithms which consist of data without labels. Let’s get right into the assignment as we are going through two algorithms today.
K-means Clustering is one of a cluster analysis technique that allows grouping of data into groups called clusters. As labels are not provided for each training data, clusters are determined by the similarity of the data from each other.
We will start by implementing the K-means algorithms. Since K-means is an iterative process that assigns training examples to their closest centroids and then recomputing the centroids, we need two main functions that do just that.
import numpy as npimport matplotlib.pyplot as pltfrom scipy.io import loadmatmat = loadmat("ex7data2.mat")X = mat["X"]
findClosestCentroids find the closest centroids by evaluating the distance between the training example from each centroid and assign the centroid to the training example with the smallest distance.
def findClosestCentroids(X, centroids):    """    Returns the closest centroids in idx for a dataset X where each row is a single example.    """    K = centroids.shape[0]    idx = np.zeros((X.shape[0],1))    temp = np.zeros((centroids.shape[0],1))        for i in range(X.shape[0]):        for j in range(K):            dist = X[i,:] - centroids[j,:]            length = np.sum(dist**2)            temp[j] = length        idx[i] = np.argmin(temp)+1    return idx# Select an initial set of centroidsK = 3initial_centroids = np.array([[3,3],[6,2],[8,5]])idx = findClosestCentroids(X, initial_centroids)print("Closest centroids for the first 3 examples:\n",idx[0:3])
np.argmin find the index with the smallest distance and assign it to the training example. +1 is used here to number the centroid from 1 instead of 0.
The print statement will print:
Closest centroids for the first 3 examples: [[1.] [3.] [2.]]
To compute the centroid means after the assignment, we sum up the training examples assigned to a particular centroid and divide by the number of examples in each centroid.
def computeCentroids(X, idx, K):    """    returns the new centroids by computing the means of the data points assigned to each centroid.    """    m, n = X.shape[0],X.shape[1]    centroids = np.zeros((K,n))    count = np.zeros((K,1))        for i in range(m):        index = int((idx[i]-1)[0])        centroids[index,:]+=X[i,:]        count[index]+=1        return centroids/countcentroids = computeCentroids(X, idx, K)print("Centroids computed after initial finding of closest centroids:\n", centroids)
The print statement will print:
Centroids computed after initial finding of closest centroids: [[2.42830111 3.15792418] [5.81350331 2.63365645] [7.11938687 3.6166844 ]]
Now to visualize the whole process, I created a subplot for each iteration of the algorithms to monitor the movement of the centroids and assignment of the training examples.
def plotKmeans(X, centroids, idx, K, num_iters):    """    plots the data points with colors assigned to each centroid    """    m,n = X.shape[0],X.shape[1]        fig, ax = plt.subplots(nrows=num_iters,ncols=1,figsize=(6,36))        for i in range(num_iters):            # Visualisation of data        color = "rgb"        for k in range(1,K+1):            grp = (idx==k).reshape(m,1)            ax[i].scatter(X[grp[:,0],0],X[grp[:,0],1],c=color[k-1],s=15)# visualize the new centroids        ax[i].scatter(centroids[:,0],centroids[:,1],s=120,marker="x",c="black",linewidth=3)        title = "Iteration Number " + str(i)        ax[i].set_title(title)                # Compute the centroids mean        centroids = computeCentroids(X, idx, K)                # assign each training example to the nearest centroid        idx = findClosestCentroids(X, centroids)        plt.tight_layout()m,n = X.shape[0],X.shape[1]plotKmeans(X, initial_centroids,idx, K,10)
Since K-means algorithms do not always give the optimal solution, random initialization is important.
def kMeansInitCentroids(X, K):    """    This function initializes K centroids that are to beused in K-Means on the dataset X    """    m,n = X.shape[0], X.shape[1]    centroids = np.zeros((K,n))        for i in range(K):        centroids[i] = X[np.random.randint(0,m+1),:]            return centroidscentroids = kMeansInitCentroids(X, K)idx = findClosestCentroids(X, centroids)plotKmeans(X, centroids,idx, K,10)
The code above will run the visualization again but with random initialization. You can run the code multiple times to see the impact of random initial centroids.
Now that we finish coding the algorithm, we can start implementing it with other datasets. In this exercise, we will use the algorithm to select 16 clusters to represent the image (down from thousands of colors) to compress the image.
mat2 = loadmat("bird_small.mat")A = mat2["A"]# preprocess and reshape the imageX2 = (A/255).reshape(128*128,3)def runKmeans(X, initial_centroids,num_iters,K):        idx = findClosestCentroids(X, initial_centroids)        for i in range(num_iters):                # Compute the centroids mean        centroids = computeCentroids(X, idx, K)# assign each training example to the nearest centroid        idx = findClosestCentroids(X, initial_centroids)return centroids, idx
Now to run k-means algorithm on the dataset
K2 = 16num_iters = 10initial_centroids2 = kMeansInitCentroids(X2, K2)centroids2, idx2 = runKmeans(X2, initial_centroids2, num_iters,K2)m2,n2 = X.shape[0],X.shape[1]X2_recovered = X2.copy()for i in range(1,K2+1):    X2_recovered[(idx2==i).ravel(),:] = centroids2[i-1]# Reshape the recovered image into proper dimensionsX2_recovered = X2_recovered.reshape(128,128,3)# Display the imageimport matplotlib.image as mpimgfig, ax = plt.subplots(1,2)ax[0].imshow(X2.reshape(128,128,3))ax[1].imshow(X2_recovered)
This is a side-by-side comparison of the original image and the compressed image with only 16 colors.
The next part of the assignment makes use of a 2D dataset to get intuition on the process of Principal Component Analysis (PCA) and then conduct PCA on a face image dataset to perform dimensionality reduction.
Load and visualize the 2D dataset
mat3 = loadmat("ex7data1.mat")X3 = mat3["X"]plt.scatter(X3[:,0],X3[:,1],marker="o",facecolors="none",edgecolors="b")
To implement the PCA algorithm, PCA also consists of two computational steps where we will code one of it to compute the covariance matrix and utilize numpy library for the other to obtain the eigenvectors.
Before that, feature normalization is needed to ensure that data are in the same range.
def featureNormalize(X):    """    Returns a normalized version of X where the mean value of each feature is 0 and the standard deviation is 1.    """    mu = np.mean(X,axis=0)    sigma = np.std(X,axis=0)        X_norm = (X - mu)/sigma        return X_norm, mu , sigmadef pca(X):    """    Computes eigenvectors of the covariance matrix of X    """    m,n = X.shape[0], X.shape[1]        sigma = 1/m * X.T @ X        U,S,V = svd(sigma)        return U,S,V
np.linalg.svd is similar to svd function in matlab and return the same U,S,V matrices. The official documentation can be found here.
from numpy.linalg import svdX_norm,mu,std = featureNormalize(X3)U,S = pca(X_norm)[:2]plt.scatter(X3[:,0],X3[:,1],marker="o",facecolors="none",edgecolors="b")plt.plot([mu[0],(mu+1.5*S[0]*U[:,0].T)[0]],[mu[1],(mu+1.5*S[0]*U[:,0].T)[1]],color="black",linewidth=3)plt.plot([mu[0],(mu+1.5*S[1]*U[:,1].T)[0]],[mu[1],(mu+1.5*S[1]*U[:,1].T)[1]],color="black",linewidth=3)plt.xlim(-1,7)plt.ylim(2,8)
The code block above implement PCA on the dataset and visualize the eigenvectors on the data. I found that Wikipedia has a good source of information for most of the learning algorithms and definitely worth looking if you wish to delve deeper into the algorithms.
print("Top eigenvector U(:,1) =:",U[:,0])`
The print statement print: Top eigenvector U(:,1) =: [-0.70710678 -0.70710678]
To reduce the dimension of the dataset, we project the data onto the principal components (eigenvectors) found.
def projectData(X, U, K):    """    Computes the reduced data representation when projecting only on to the top k eigenvectors    """    m = X.shape[0]    U_reduced = U[:,:K]    Z = np.zeros((m,K))        for i in range(m):        for j in range(K):            Z[i,j] = X[i,:] @ U_reduced[:,j]        return Z# Project the data onto K=1 dimensionK=1Z = projectData(X_norm, U, K)print("Projection of the first example:",Z[0][0])
The print statement will print: Projection of the first example: 1.4963126084578515
Data also can be approximately reconstructed by projecting them back onto the original dimension space.
def recoverData(Z, U, K):    """    Recovers an approximation of the original data when using the projected data    """    m,n = Z.shape[0],U.shape[0]    X_rec = np.zeros((m,n))    U_reduced = U[:,:K]        for i in range(m):        X_rec[i,:] = Z[i,:] @ U_reduced.T        return X_recX_rec  = recoverData(Z, U, K)print("Approximation of the first example:",X_rec[0,:])
The print statement will print: Approximation of the first example: [-1.05805279 -1.05805279]
To visualize the whole process,
plt.scatter(X_norm[:,0],X_norm[:,1],marker="o",label="Original",facecolors="none",edgecolors="b",s=15)plt.scatter(X_rec[:,0],X_rec[:,1],marker="o",label="Approximation",facecolors="none",edgecolors="r",s=15)plt.title("The Normalized and Projected Data after PCA")plt.legend()
Finally, we move on to a more complex dataset — Face image dataset. To load and visualize the data,
mat4 = loadmat("ex7faces.mat")X4 = mat4["X"]fig, ax = plt.subplots(nrows=10,ncols=10,figsize=(8,8))for i in range(0,100,10):    for j in range(10):        ax[int(i/10),j].imshow(X4[i+j,:].reshape(32,32,order="F"),cmap="gray")        ax[int(i/10),j].axis("off")
These images contain 32 X 32 pixels in grayscale, resulting in a dimension of 1,024 features and our task is to reduce the dimension to around 100 principal components that best describe our data.
X_norm2 = featureNormalize(X4)[0]# Run PCAU2 =pca(X_norm2)[0]#Visualize the top 36 eigenvectors foundU_reduced = U2[:,:36].Tfig2, ax2 = plt.subplots(6,6,figsize=(8,8))for i in range(0,36,6):    for j in range(6):        ax2[int(i/6),j].imshow(U_reduced[i+j,:].reshape(32,32,order="F"),cmap="gray")        ax2[int(i/6),j].axis("off")
Above is the visualization of 36 principal components that describe the largest variations in the dataset.
Next, we project the data onto the first 100 principal components, effectively reduce the dimension down to a hundred, recover the data and try to understand what is lost in the dimension reduction process.
K2 = 100Z2 = projectData(X_norm2, U2, K2)print("The projected data Z has a size of:",Z2.shape)# Data reconstructionX_rec2  = recoverData(Z2, U2, K2)# Visualize the reconstructed datafig3, ax3 = plt.subplots(10,10,figsize=(8,8))for i in range(0,100,10):    for j in range(10):        ax3[int(i/10),j].imshow(X_rec2[i+j,:].reshape(32,32,order="F"),cmap="gray")        ax3[int(i/10),j].axis("off")
This is the end of unsupervised learning. Please look forward to the last article in the series. The Jupyter notebook will be uploaded to my GitHub at (https://github.com/Benlau93/Machine-Learning-by-Andrew-Ng-in-Python).
For other python implementation in the series,
Linear Regression
Logistic Regression
Regularized Logistic Regression
Neural Networks
Support Vector Machines
Anomaly Detection
Thank you for reading.