The essence of eigenvalues and eigenvectors in Machine Learning | by Ranjeet Singh | Towards Data Science

The word, Eigen is perhaps most usefully translated from German which means Characteristic. So when we talk about Eigenvalues and eigenvectors of a Matrix, we’re talking about finding the characteristics of the matrix.
Before diving deep into Eigenvectors, let's understand what is a matrix except being a rectangular array of numbers, What does it represent?
So a matrix is simply a linear transformation applied to a vector. There can be different types of transformation applied to a vector, for example-
Let's see some matrix transformations
img = cv2.imread(path_to_image,flags=cv2.IMREAD_UNCHANGED)img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)rows,cols,_ = img.shapeM = np.float32([[1, -np.sin(.1), 0],[0,  np.cos(.1), 0]])out = cv2.warpAffine(img,M,(cols,rows))
What does this matrix M do with the image? It introduced a horizontal shear to every vector in the image.
#Let's try this oneM = cv2.getRotationMatrix2D((cols/2,rows/2),45,1)out = cv2.warpAffine(img,M,(cols,rows))print(M)=> [[   0.70710678    0.70710678 -124.36235483] [  -0.70710678    0.70710678  501.76271632]]
So what has the matrix M has done to the images? So this linear transformation M rotates every vector in the image by 45 degrees.
M = [[  1.   0. 100.] [  0.   1. 100.]]out = cv2.warpAffine(img,M,(cols,rows))
It translates the image in both horizontal and vertical directions.
The rotation has no eigenevector[except the case of 180-degree rotation]. For pure shear, the horizontal vector is an eigenvector. The factor by which the length of vector changes is called eigenvalue.
The concept of eigenvalues and eigenvectors is used in many practical applications. I will discuss only a few of these.
PCA is a very popular classical dimensionality reduction technique which uses this concept to compress your data by reducing its dimensionality since curse of dimensionality has been very critical issue in classical Computer Vision to deal with images and even in Machine Learning, features with high dimensionality increase model capacity which in turn requires a large amount of data to train.
It is a method that uses simple matrix operations and statistics to calculate a projection of the original data into the same number or fewer dimensions.
Let the data matrix X be of n×p size, where n is the number of samples and p is the dimensionality of each sample. In PCA, essentially we diagonalize the covariance matrix of X by eigenvalue decomposition since the covariance matrix is symmetric-
C = VLVTIn Python-from numpy import covfrom numpy.linalg import eigX = array([[-2, -2], [0, 0], [2, 2]])C = cov(X)#To do eigendecomposition of Cvalues, vectors = eig(C)#Project data into pricipal directionsP = vectors.T.dot(X.T)print(P.T)Output-[[-2.82842712  0.        ] [ 0.          0.        ] [ 2.82842712  0.        ]]
where V is a matrix of eigenvectors (each column is an eigenvector) and L is a diagonal matrix with eigenvalues λi in the decreasing order on the diagonal.
Eigenvectors of a symmetric matrix, covariance matrix here, are real and orthogonal.
The eigenvectors are called principal axes or principal directions of the data. Projections of the data on the principal axes are called principal components.
We reduce the dimensionality of data by projecting it in fewer principal directions than its original dimensionality.
K-Means is the most popular algorithm for clustering but it has several issues associated with it such as dependence upon cluster initialization and dimensionality of features. Also, it faces problems if your clusters are not spherical as seen below-
from sklearn.datasets import make_moonsX_mn, y_mn = make_moons(150, noise=.07, random_state=21)fig, ax = plt.subplots(figsize=(9,7))ax.set_title('Data with ground truth labels ', fontsize=18, fontweight='demi')ax.scatter(X_mn[:, 0], X_mn[:, 1],c=y_mn,s=50, cmap='viridis')
Spectral clustering is a family of methods to find K clusters using the eigenvectors of a matrix. It handles these issues and easily outperforms other algorithms for clustering.
Here data is represented in the form of a graph. Now clustering can be thought of making graph cuts where Cut(A,B) between 2 clusters A and B is defined as the sum of weight connections between two clusters. For example-
To find optimum clusters, we need MinCut and the objective of a MinCut method is to find two clusters A and B which have the minimum weight sum connections.
In spectral clustering, this min-cut objective is approximated using the Graph Laplacian matrix computed from the Adjacency and degree matrix of the graph. For proof, see this
Given: A graph with n vertices and edge weights Wij , number of desired clusters k
Construct (normalized) graph Laplacian L G V, E = D − W
Find the k eigenvectors corresponding to the k smallest eigenvalues of L
Let U be the n × k matrix of eigenvectors
Use k-means to find k clusters C′ letting xi ′ be the rows of U 5. Assign data point xi to the j’th cluster if xi ′ was assigned to cluster j
from sklearn.neighbors import radius_neighbors_graphfrom scipy.sparse import csgraphfrom sklearn.cluster import KMeans#Create adjacency matrix from the datasetA = radius_neighbors_graph(X_mn,0.4,mode='distance', metric='minkowski', p=2, metric_params=None, include_self=False)A = A.toarray()'''Next find out graph Laplacian matrix, which is defined as the L=D-A where A is our adjecency matrix we just saw and D is a diagonal degree matrix, every cell in the diagonal is the sum of the weights for that point'''L = csgraph.laplacian(A, normed=False)eigval, eigvec = np.linalg.eig(L)
Now, use k-means to find k clusters C letting xi be the rows of eigvec. Finally to assign data points into clusters, assign xi to the j’th cluster if xi was assigned to cluster j.
The second smallest eigenvector , also called Fiedler vector is used to recursively bi-partition the graph by finding the optimal splitting point.
See full example code here
Variants of spectral clustering are used in Region Proposal based Object Detection and Semantic Segmentation in Computer Vision.
At last, I will discuss my favorite field under AI, which is Computer Vision. In Computer Vision, Interest points in an image are the points which are unique in their neighborhood. Such points play a significant role in classical Computer Vision where these are used as features.
Corners are useful interest points along with other more complex image features such as SIFT, SURF, and HOG, etc. I would discuss one such method of corner detection.
Corners are easily recognized by looking through a small window.
Shifting the window should give a large change in intensity E if the window has a corner inside it.
The algorithm-
Compute image gradients over a small region
imggray = cv2.imread('checkerboard.png',0)i_x = cv2.Sobel(imggray,cv2.CV_64F,1,0,ksize=5)i_y = cv2.Sobel(imggray,cv2.CV_64F,0,1,ksize=5)
Compute the covariance matrix
# Calculate the product of derivates in each directioni_xx, i_xy, i_yy = multiply(i_x, i_x), multiply(i_x, i_y), multiply(i_y, i_y)# Calculate the sum of product of derivatess_xx, s_xy, s_yy = cv2.GaussianBlur(i_xx, (5,5), 0), cv2.GaussianBlur(i_xy, (5,5), 0), cv2.GaussianBlur(i_yy, (5,5), 0)
Compute eigenvectors and eigenvalues.
Harris described a way for a faster approximation — Avoid computing the eigenvalues, just compute Trace and Determinant. Combing these 2 properties, we calculate a measure of cornerness-R
Determinant of a matrix = Product of eigen values
Trace of a matrix = Sum of eigen values
# Compute the response of the detector at each pointk = .04 # Recommended value between .04 and .06det_h = multiply(s_xx, s_yy) - multiply(s_xy, s_xy)trace_h = s_xx + s_yyR = det_h - k*multiply(trace_h, trace_h)
Use threshold on eigenvalues to detect corners
ratio = .2 # Number to tweakthresh = abs(R) > ratio * abs(R).max()
If either eigenvalue is close to 0, then this is not a corner, so look for locations where both are large. E is almost constant in all directions.
λ2 >> λ1 or λ1 >> λ2 depicts an Edge
λ1 and λ2 are large, λ1 ~ λ2 E increases in all directions
See full example code here
Normalized Cuts and Image Segmentation. J. Shi and J. Malik, 2000
A Combined Combined and Edge Detector, Chris Harris & Mike Stephens, 1988
Algebraic Connectivity of Graph M. Fiedler, 1973