Eigenvector Computation and Low-Rank Approximations - GeeksforGeeks

30 May, 2021
Prerequisites: Eigen Values and Eigen Vectors
Before getting into the in-depth math behind computations involved with Eigenvectors, let us briefly discuss what an eigenvalue and eigenvector actually are.
The word ‘eigen’ means ‘characteristics‘. In general terms, the eigenvalues and eigenvectors give the characteristics of a matrix or a vector.
Eigenvector: It is a vector represented by a matrix X such that when X is multiplied with any matrix A, then the direction of the resultant matrix remains the same as vector X. Observe Fig 1 carefully to look at a graphical representation of an eigenvector. 
Fig 1 : Eigenvector representation
In Fig 1, we observe the following scalar transformations:
After applying these transformations, the vectors v1‘ and v2‘ are in the same direction as v1 and v2. So as per our definition, these are considered as eigenvectors. But resultant vector v3‘ is not in the same direction as v3. Hence it cannot be considered as a eigenvector.
Eigenvalues: It tells us about the extent to which the eigenvector has been stretched or diminished. In the above case, the eigenvalues will be 1.5 and 0.5.
We can calculate eigenvalues of any matrix using the characteristic equation of the matrix (as discussed in the prerequisite article) that is:
The roots of the above equation (i.e. values of λ) gives us the eigenvalues. 
Using the values of λ obtained, we can find the corresponding eigenvectors using the equation given below.
Example Problem
Consider the following example for a better understanding. Let there be a 3×3 matrix X defined as:
Find the eigen values and eigen vectors corresponding to the matrix A. 
1. Finding the eigen values.
Thus, the eigen values obtained are 1, 2 and 3. 
2. Finding the eigen vectors.
Using the formula given above, we will calculate a corresponding eigen vector xi for each value of λi. 
Thus, we obtain the eigen vectors X1, X2, X3 corresponding to each value of λ.
Rank of a (m x n) matrix is determined by the number of linearly independent rows present in the matrix. Consider the example given below for a better understanding. 
All 3 rows of matrix A are linearly independent. Therefore, Rank ( A ) = 3.
Rank ( B ) = 2. This is because Row 3 is dependent on R1 and R2. [R3 <- R1 + R2]
For any matrix A of shape m x n, the following rank properties are applicable:
Rank (A) = Rank (AT)Rank (BAC) = Rank (A) provided B and C are invertible matrices.Rank (AB) ≤ min{ Rank (A) + Rank (B) }
Rank (A) = Rank (AT)
Rank (BAC) = Rank (A) provided B and C are invertible matrices.
Rank (AB) ≤ min{ Rank (A) + Rank (B) }
Before getting into Low-Rank Approximation, it is important to understand the following:
Matrix Factorization 
Any matrix A of shape m x n having rank (A) = r is said to be factorized when it takes the form:
where, the shapes of the matrices are:
 
mat(A) -> m x n
mat(B) -> m x r 
mat(C) -> n x r
Low Rank
We say that matrix A has a low rank if r << min {m,n}. 
Based on the above two discussed method, we define LRA as:
For a matrix A of shape m x n and rank (A) << min {m, n}, the low-rank approximation of A is to find another matrix B such that rank (B) ≤ rank (A). Intuitively, we tend to see how linearly similar matrix B is to the input matrix A. Mathematically, LRA is a minimization problem, in which we measure the fit between a given matrix (the data) and an approximating matrix (the optimization variable).
Let us assume 3 matrices X, Y, Z of dimensions (50 x 40), (50 x 10), (40 x 10) respectively where rank(X) = 10. Therefore, as per matrix factorization, the three matrices are of the form A = BCT. We observe that:
Amxn = BmxrCTrxn

No of elements/pixels in mat(A) = (50 x 40) = 2000
while
No of elements/pixels in mat(BCT) = (50 x 10) + (10 x 40) = 900.
Thus, by using low-rank approximation, we can reduce the number of pixels of input data by a significant amount. (1100 fewer pixels than input image in the above case)
This method is commonly used in Image Processing tasks where the input image (data) is very large and needs to be compressed before any processing task. Observe the two images of the same person given below. 
After LRA image vs Input image
After applying LRA, we are able to obtain an image that is very similar to the original image by discarding some unimportant pixels, hence reducing the size of the image as well. After doing this, it becomes much easier to handle such large datasets and perform a variety of pre-processing tasks on them. 
Machine Learning
Machine Learning
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Support Vector Machine Algorithm
k-nearest neighbor algorithm in Python
Singular Value Decomposition (SVD)
Intuition of Adam Optimizer
ML | Logistic Regression using Python
Principal Component Analysis with Python
CNN | Introduction to Pooling Layer
Normalization vs Standardization
Python | Decision Tree Regression using sklearn
Introduction to Recurrent Neural Network