Linear Algebra for Data Science: The Bigger Picture of Linear Algebra— Part 2
By: Anuranjan Kumar
Tags: Machine Learning, Linear Algebra, Musings Of Data Science, Data Science

A quick introduction, in this series, I am revisiting topics discussed by Prof. Gilbert Strang in his latest lecture series, “A 2020 Vision of Linear Algebra”. This post is the second part of the series “Linear Algebra for Data Science”, I advise you to go through Part 1 so that you don’t miss the flow.

My aim here is to share with you my understanding of the topics with relevant details which I believe will aid in understanding the lectures in a better and nuanced way.

So let’s get going!

Topics Covered in Part 2:

In this part, I will mostly talk about four fundamental subspaces of Linear Algebra and how they relate to each other.

A subspace of a vector space Four fundamental Subspaces Basis of fundamental Subspaces The Bigger Picture!

Subspace

If V is a vector space on the field of Real numbers, defines as and W is a subset of V, then W is a subspace of V. For example, let the vector space V be the real coordinate space R³ and W be the set of all vectors in V whose last component is 0. Then W is a subspace of V. In terms of matrices, let A be a 3 by 2 matrix, then the column space is a subspace if R³ and row space is a subspace if R².

In R³, the intersection of two distinct two-dimensional subspaces. By Fred the Oyster, CC BY-SA 4.0

Four Fundamental Subspaces

The Fundamental Theorem of Linear Algebra by Gilbert Strang describes the action of an m by n matrix A. The matrix A produces a linear transformation from R^n to R^m but this picture by itself is too large. The “truth” about Ax = b is expressed in terms of four subspaces (two of R^n and two of R^m). The first step is to see Ax as a combination of the columns of A. This step raises the viewpoint to subspaces. We see Ax in the column space. Solving Ax = b means finding all combinations of the columns that produce b in the column space.

The four fundamental subspaces are:

The Column Space C(A), a subspace of R^m

C(A), a subspace of R^m The Row Space C(A’), a subspace of R^n (A’ = Transpose of A)

C(A’), a subspace of R^n (A’ = Transpose of A) The Null Space N(A), a subspace of R^n

N(A), a subspace of R^n The Left Null Space N(A’), a subspace of R^m

Basis of Column Space

From Part 1, we know that dimension of column space is equal to the dimension of row space and is equal to the rank r of Matrix A, i.e. dim(C(A)) = dim(C(A’)) = r.

We also saw, to compute the basis of the column space, we need to reduce the matrix A in its row echelon form and pick the row corresponding to the pivot elements. Now let’s see how to calculate the basis for the other three sub-spaces.

Basis of Row Space

Before proceeding forward, let’s be consistent with the notations we will be using:

A is an m x n matrix

A’ is transpose if A

R is the reduced row echelon form of A. This is a special case when each leading 1 is the only nonzero entry in its column.

Matrix A and its Reduced Row Echelon form R

To find the basis of the Row Space we first reduce the matrix in its reduced row echelon form and pick the rows in R which constitutes an identity matrix of size equal to rank r. As shown in the figure above, the basis of the row space is the first two rows. So, here the basis of row space is the first two rows of R.

Please note, the row space remains unchanged in the transformation of A to R but the column space changes.

Basis of Null Space

To obtain the basis of Null Space of A, we need to solve the homogenous system of equation, Ax = 0. We first obtain the reduced row echelon form matrix corresponding to the matrix A then find the coefficients corresponding to the free-variables (variable corresponding to non-pivot columns in the reduced row echelon form). Let’s try to understand with the help of an example:

Matrix A and its Reduced Row Echelon form R

Solving Ax = 0, we get the above solution

Basis of Left Null Space

The Left Null Space can also be seen as the Null Space if A’. So, in this case, we solve A’ y = 0. You may we wondering why “Left” Null Space? Well, there is no mathematical reason behind it but just to justify our choice let’s build an explanation (all credits to Prof. Strang!). Let’s take transpose on both sides of the equation A’y = 0. It is now equivalent to y’ A = 0. Please note, here the vector of Zeros is a row vector now. Since y’ now appears to the left of matrix A, we call this space the Left Null Space.

One obvious way to find the Left Null Space is to solve the equation as we did for Null Space but wait, DRY! (Do Not Repeat Yourself!). Let’s do it in another way! Yay!

We know from our second formulation that y’ is now a row vector to the left of A and it produces a row vector of Zeros on the right-hand side. Let’s use that.

Remember what we did while determining the row space, we transformed our matrix A to R. Let, augment an m x m Identity matrix to m x n matrix A and perform the same steps, this will now transform [ A | I ] to [ R | E ], where R is m x n and E is m x m. Let’s just take a min and observe what happened… We applied the same row transformations we applied to matrix A to get R to matrix I to get E. It means, E[ A | I ] = [ R | E ] or EA = R

Please Note: When A is invertible, R = I and E = inverse(A)

Reducing [ A | I ] to [ R | E ]

As the above equation, EA = R is the same as solving y’ A = 0, and from the matrix R we can see that rank(A) = 2. So, the dimension of left null space = 3-2 = 1, i.e. dimension of column space-rank. Hence, the basis of Left Null Space of A corresponds to the row in E which produces the zero row in R, which is [ -1 0 1 ].

The Bigger Picture!

Now, after seeing the Big Picture, you can guess why it is called the “Big Picture”. It is the essence of all the four fundamental subspaces all at a place! So let’s note what information does it contain: