Visual Introduction to Singular Value Decomposition (SVD) | by Hadrien Jean | Towards Data Science

In this article, you’ll learn about Singular value decomposition (SVD), which is a major topic of linear algebra, data science, and machine learning. It is for instance used to calculate the Principal Component Analysis (PCA).
You’ll need some understanding of linear algebra basics (feel free to check the previous article and the book Essential Math for Data Science).
You can only apply eigendecomposition to square matrices because it uses a single change of basis matrix, which implies that the initial vector and the transformed vector are relative to the same basis. You go to another basis with Q to do the transformation, and you come back to the initial basis with Q^-1.
As eigendecomposition, the goal of singular value decomposition (SVD) is to decompose a matrix into simpler components: orthogonal and diagonal matrices.
You also saw that you can consider matrices as linear transformations. The decomposition of a matrix corresponds to the decomposition of the transformation into multiple sub-transformations. In the case of the SVD, the transformation is converted to three simpler transformations.
You’ll see here three examples: one in two dimensions, one comparing the transformations of the SVD and the eigendecomposition, and one in three dimensions.
You’ll see the action of these transformations using a custom function matrix_2d_effect() (you can find this function here). This function plots the unit circle (you can find more details about the unit circle in Chapter 05 of Essential Math for Data Science) and the basis vectors transformed by a matrix.
To represent the unit circle and the basis vectors before the transformation, let’s use this function using the identity matrix:
Let’s now use the function to see the effect of the following matrix A.
It will plot the unit circle and the basis vectors transformed by the matrix:
Figure 2 illustrates the effect of A on your two-dimensional space. Let’s compare this to the sub-transformations associated with the matrices of the SVD.
You can calculate the SVD of A using Numpy:
Remember that the matrices U, Σ, and V contain respectively the left singular vectors, the singular values, and the right singular vectors. You can consider V^T as the first change of basis matrix, Σ as the linear transformation in this new basis (this transformation should be a simple scaling since Σ is diagonal), and U another change of basis matrix. You can see in Chapter 10 of Essential Math for Data Science that SVD constraints both change of basis matrices U and V^T to be orthogonal, meaning that the transformations will be simple rotations.
To summarize, the transformation corresponding to the matrix A is decomposed into a rotation (or a reflection, or a retroflection), a scaling, and another rotation (or a reflection, or a retroflection).
Let’s see the effect of each matrix successively:
You can see in Figure 3 that the basis vectors have been rotated by the matrix V^T.
Then, Figure 4 shows that the effect of Σ is a scaling of the unit circle and the basis vectors.
Finally, a third rotation is applied by U. You can see in Figure 5 that the transformation is the same as the one associated with the matrix A. You have decomposed the transformation into a rotation, a scaling, and a rotoreflection (look at the basis vectors: a reflection has been done because the yellow vector is on the left of the green vector, which was not the case initially).
Since the matrix A was square, you can compare this decomposition with eigendecomposition and use the same type of visualization. You’ll get more insights about the difference between the two methods.
Remember from Chapter 09 of Essential Math for Data Science that the eigendecomposition of the matrix A is given by
Let’s calculate the matrices Q and Λ (pronounced “capital lambda”) with Numpy:
Note that, since the matrix A is not symmetric, the eigenvectors are not orthogonal (their dot product is not equal to zero):
-0.16609095970747995
Let’s see the effect of Q^-1 on the basis vectors and the unit circle:
You can see in Figure 6 that Q^-1 rotates and scales the unit circle and the basis vectors. The transformation of a non-orthogonal matrix is not a simple rotation.
The next step is to apply Λ.
The effect of Λ, shown in Figure 7 is a stretching and a reflection through the y-axis (the yellow vector is now on the right of the green vector).
The last transformation, shown in Figure 8 corresponds to the change of basis back to the initial one. You can see that it leads to the same result as the transformation associated with A: both matrices A and Q Λ Q^-1 are similar: they correspond to the same transformation in different bases.
It highlights the differences between eigendecomposition and SVD. With SVD, you have three different transformations, but two of them are only rotation. With eigendecomposition, there are only two different matrices, but the transformation associated with Q is not necessarily a simple rotation (it is only the case when A is symmetric).
Since the SVD can be used with non square matrices, it is interesting to see how the transformations are decomposed in this case.
First, non square matrices map two spaces that have a different number of dimensions. Keep in mind that m by n matrices map an n-dimensional space with an m-dimensional space.
Let’s take the example of a 3 by 2 matrix, mapping a two-dimensional space to a three-dimensional space. This means that input vectors are two-dimensional and output vectors three-dimensional. Take the matrix A:
To visualize the effect of A, you’ll still use the unit circle in two dimensions and you’ll calculate the output for points on this circle. Each point is considered as an input vector and you can observe the effect of A on each of these vectors. The function matrix_3_by_2_effect() can be found here.
As expected, and as represented in Figure 9, the two-dimensional unit circle is transformed into a three-dimensional ellipse. Feel free to use the notebook to be able to move the plot and have a better idea of the three-dimensional shape.
What you can note is that the output vectors land all on a two-dimensional plane. This is because the rank of A is two (more details about the rank of a matrix in Section 7.6 of of Essential Math for Data Science).
Now that you know the output of the transformation by A, let’s calculate the SVD of A and see the effects of the different matrices, as you did with the two-dimensional example.
The shape of the left singular vectors (U) is m by m and the shape of the right singular vectors (V^T) is n by n. There are two singular values in the matrix Σ.
The transformation associated with A is decomposed into a first rotation in Rn (associated with V^T, in the example, R2), a scaling going from Rn to Rm (in the example, from R2 to R3), and a rotation in the output space Rm (in the example, R3).
Let’s start to inspect the effect of V^T on the unit circle. You stay in two dimensions at this step:
You can see in Figure 10 that the basis vectors have been rotated.
Then, you need to reshape Σ because the function np.linalg.svd() gives a one-dimensional array containing the singular values. You want a matrix with the same shape as A: a 3 by 2 matrix to go from 2D to 3D. This matrix contains the singular values as the diagonal, the other values are zero.
Let’s create this matrix:
array([[9.99274669, 0. ], [0. , 4.91375758], [0. , 0. ]])
You can now add the transformation of Σ to see the result in 3D in Figure 11:
Finally, you need to operate the last change of basis. You stay in 3D because the matrix U is a 3 by 3 matrix.
You can see in Figure 12 that the result is identical to the transformation associated with the matrix A.
Figure 13 summarizes the decomposition of a matrix A into three matrices. The transformation associated with A is done by the three sub-transformations. The notation is the same as in Figure 13 and illustrates the geometric perspective of the SVD.
Singular Value Decomposition can be used for instance to find transformations that approximates well a matrix (see Low-Rank Matrix Approximation in Section 10.4 of Essential Math for Data Science)
This post is a sample of my book Essential Math for Data Science!
Get the book here: https://www.essentialmathfordatascience.com/!