What is Column Space? — Example, Intuition & Visualization | by Aerin Kim | Towards Data Science

You might already know this, but for some quick background — when we see mathematical expressions like X ∈ R2, X ∈ R5 or X ∈ R100, what do they mean?
When you see these expressions, they are easy to understand if you visualize X as a column vector with n components. We use R because its components are the Real numbers.
Then, R2 is represented by 2 numbers (coordinates), the good old x-y plane. In the same way, the three components of a vector in R3 is a point in 3-D space.
One of the reasons why we’re fond of Linear Algebra is that the extension to n-dimensions is straightforward. For example, if we want to define a vector in R7, all you need are seven real numbers (i.e. [4, 1, 8, 5, 9, 5, 6]), even though it is hard to visualize 7-D space.
Let’s say you wrote a vector with 100 random numbers. Then your vector belongs to R100.
Space is short for subspace. A subspace is a subset that is “closed” under addition and scalar multiplication, which is basically closed under linear combinations. These two operations keep the output within the subspace always.
Definition of Subspace:A subspace of a vector space is a subset that satisfies the requirements for a vector space -- Linear combinations stay in the subspace.(i) If any two vectors x and y are in the subspace, x + y is in the subspace as well.(ii) If we multiply any vector x in the subspace by any scalar c, cx is in the subspace as well.
Understanding the concept by only reading the definition doesn’t really work for me. In order to solidify our understanding, let’s try to answer this question:
The first quadrant of the x-y plane: Is it a subspace?
Let’s check if the rule (i) holds. If we add any two vectors a (a1>0, a2>0) and b (b1>0, b2>0) in the first quadrant, a + b will be in the subspace. Ok, so far so good. How about the rule (ii)? Let’s pick any scalar c = -3 and the vector x = [2, 4] in the first quadrant. Now, cx = [-6, -12] is in the third quadrant, not the first. The first quadrant is not a subspace.
If we include the third quadrant along with the first, scalar multiplication is all right. Every multiple cx will stay in this subset. However, now the rule (i) is violated, since adding [3, 5]+[-9, -1] will result in [-6, 4], which is not in either quadrant. Hence, the smallest subspace containing the first quadrant is the whole R2 space.
A column space (or range) of matrix X is the space that is spanned by X’s columns. Likewise, a row space is spanned by X’s rows.
In the above picture, [0,1] and [1,0] spans the whole plane (R2). However, vectors don’t need to be orthogonal to each other to span the plane. As long as they are two non-parallel vectors, their linear combinations will fill (“SPAN”) the whole plane.
Column space of X = Span of the columns of X = Set of all possible linear combinations of the columns of X
Multiplying the matrix X by any vector θ gives a combination of the columns. Hence, the vector Xθ is in the column space.
In Eq.(a), there are 2 unknowns [θ1, θ2] but 3 equations. When we have more equations than unknowns, usually there is no solution.
Notice that the number of equations determines the dimension of the column vectors. (If we have 10 equations, instead of 3, then we’ll be solving a 10-dimensional problem.)
Eq.(a) can be written as eq.(b).
The right-hand side y can be any combinations of the columns of X. [3,6,9] is just one example among many possible vectors.If you choose θ1 = 1, θ2 = 0, then y will be the X1 vector itself.Likewise, for θ1 = 0, θ2= 1, y will be the same as X2.
However, notice, if y lies off the plane C(X), then it is not the combination of the two columns. In that case, Xθ = y has no solution.
The spanned plane C(X) is not just a subset of R3. It is a subspace. It consists of every combination of the columns and satisfies the rule (i) and (ii).
Xθ = y can be solved only when y lies in the plane that is spanned by the two column vectors, the combination of the columns of X. Then we say y is in the column space.
Few things to note:
1. What is a good example for X, y and [θ1, θ2]?
Linear regression. (Let’s predict the housing price.)
X is a feature matrix or input variables (# of bedrooms, square feet, location, etc). The number of rows in X is the number of training examples. y is a target variable (the housing price). θ is the coefficient that we are trying to fit.
Therefore, “y lies in the column space” means the error of the linear regression is zero, which is never the case in real life.
2. Why do [2,0,9] and [1,5,3] span the plane? They are 3-D vectors.
Because they are 2 vectors 😜. Just because they are 3-D vectors, they don’t span the 3-D plane, you need 3 vectors to span 3-D space.
Two 2-D vectors [1,0] and [4,1] will span the plane.Two 7-D vectors [2,0,9,0,1,4,2] and [7,7,0,1,8,4,8] will still span the plane.
3. Any n by n matrix that is non-singular will have R^n as its columns space. If we allow singular matrices, or rectangular matrices of any shape, then C(X) will be somewhere between the zero space and R^n.
Finally, the “Machine Learning” part begins. Before continue reading, make sure you are familiar with the concept of “orthogonality”.
medium.com
When y lies off the plane (= when y is not in the column space of X), then Xθ = y has no solution. Because the system is inconsistent.
However, in real life, we still need to find a solution — the best approximation of θ. So we use linear regression. 99.99999% of the time, there is no way the data points y will lie exactly on the spanned plane C(X). Therefore, we will be approximating the plane that is the closest to where y data points lie.
1 is solvable when y1, y2, y3 are in the ratio 1:5:3.
When y1, y2, y3 are not in the ratio 1:5:3, we can still “solve” (it’s rather “fit”) Xθ = y by minimizing the least square error.
When there is an exact solution, the minimum error will be absolute zero. However, most likely y won’t be exactly proportional to X, and the graph of (Error)2 will be a parabola.
The minimum error will be at where the derivative of (Error)2 is zero.
If you look at the derived θ in step 4, it matches with “the normal equation” that we derived in the previous post.
Taking the derivative of (Error2) to find a minimum is a calculus technique. However, orthogonality of the dot product X and (y - Xθ) is a geometric interpretation.
Geometry confirms Calculus!
When we learn linear regression, we learn in an analytical way; however, there is a geometric interpretation as well. I think this is a beautiful connection between the two concepts, which solidifies understanding.
Let’s project y onto a subspace (plane), instead of just onto a line.
Going back to eq.(a), X is a 3 by 2 matrix and θ is 1 by 2 matrix (no longer a scalar). Think of X as a design matrix for which the number of samples is 3, the number of features is 2.
The number of samples (3 in this example, or any m) usually will be much greater than the number of features (2, or any n). So we expect that there will be no exact solution. In other words, y won’t be a combination of columns of X. y will be outside of the column space C(X).
Error = ‖ y - Xθ ‖This is the distance between y to the point Xθ which lies in the columns space of X.
Searching for the least square solution (θ) that minimizes the error is the same as LOCATING the point Xθ as close as possible in the column space than any other point in the column space.
All vectors perpendicular to the column space lie in the left null space. Thus the error vector y - Xθ is in the null space of Xθ.
If you like my post, could you please clap? It gives me motivation to write more. :)