Essential Math for Data Science: Basis and Change of Basis | by Hadrien Jean | Towards Data Science

In this article, you’ll learn about the concept of basis, which is an interesting way to understand matrix factorization methods like eigendecomposition or singular value decomposition (SVD).
The basis is a coordinate system used to describe vector spaces (sets of vectors). It is a reference that you use to associate numbers with geometric vectors.
To be considered as a basis, a set of vectors must:
Be linearly independent.
Span the space.
Every vector in the space is a unique combination of the basis vectors. The dimension of a space is defined to be the size of a basis set. For instance, there are two basis vectors in R2 (corresponding to the x and y-axis in the Cartesian plane), or three in R3.
As shown in section 7.4 of Essential Math for Data Science, if the number of vectors in a set is larger than the dimensions of the space, they can’t be linearly independent. If a set contains fewer vectors than the number of dimensions, these vectors can’t span the whole space.
Vectors can be represented as arrows going from the origin to a point in space. The coordinates of this point can be stored in a list. The geometric representation of a vector in the Cartesian plane implies that we take a reference: the directions given by the two axes x and y.
Basis vectors are the vectors corresponding to this reference. In the Cartesian plane, the basis vectors are orthogonal unit vectors (length of one), generally denoted as i and j.
For instance, in Figure 1, the basis vectors i and j point in the direction of the axis x and y respectively. These vectors give the standard basis. If you put these basis vectors into a matrix, you have the following identity matrix (for more details about identity matrices, see 6.4.3 in Essential Math for Data Science):
Thus, the columns of I2 span R2. In the same way, the columns of I3 span R3 and so on.
Orthogonal basis
Basis vectors can be orthogonal because orthogonal vectors are independent. However, the converse is not necessarily true: non-orthogonal vectors can be linearly independent and thus form a basis (but not a standard basis).
The basis of your vector space is very important because the values of the coordinates corresponding to the vectors depend on this basis. By the way, you can choose different basis vectors, like in the ones in Figure 2 for instance.
Keep in mind that vector coordinates depend on an implicit choice of basis vectors.
You can consider any vector in a vector space as a linear combination of the basis vectors.
For instance, take the following two-dimensional vector v:
The components of the vector v are the projections on the x-axis and on the y-axis ( v_x and v_y, as illustrated in Figure 3). The vector v corresponds to the sum of its components: v = v_x + v_y, and you can obtain these components by scaling the basis vectors: v_x = 2 i and v_y = -0.5 j. Thus, the vector v shown in Figure 3 can be considered as a linear combination of the two basis vectors i and j:
The columns of identity matrices are not the only case of linearly independent column vectors. It is possible to find other sets of n vectors linearly independent in Rn.
For instance, let’s consider the following vectors in R2:
and
From the definition above, the vectors v and w are a basis because they are linearly independent (you can’t obtain one of them from combinations of the other) and they span the space (all the space can be reached from the linear combinations of these vectors).
It is critical to keep in mind that, when you use the components of vectors (for instance v_x and v_y, the x and y components of the vector v), the values are relative to the basis you chose. If you use another basis, these values will be different.
You’ll see in Chapter 09 and 10 of Essential Math for Data Science that the ability to change the bases is fundamental in linear algebra and is key to understand eigendecomposition or Singular Value Decomposition.
You saw that to associate geometric vectors (arrows in the space) with coordinate vectors (arrays of numbers), you need a reference. This reference is the basis of your vector space. For this reason, a vector should always be defined with respect to a basis.
Let’s take the following vector:
The values of the x and y components are respectively 2 and -0.5. The standard basis is used when not specified.
You could write Iv to specify that these numbers correspond to coordinates with respect to the standard basis. In this case, I is called the change of basis matrix.
You can define vectors with respect to another basis by using another matrix than I.
Vector spaces (the set of possible vectors) are characterized in reference to a basis. The expression of a geometrical vector as an array of numbers implies that you choose a basis. With a different basis, the same vector v is associated with different numbers.
You saw that the basis is a set of linearly independent vectors that span the space. More precisely, a set of vectors is a basis if every vector from the space can be described as a finite linear combination of the components of the basis and if the set is linearly independent.
Consider the following two-dimensional vector:
In the R2 Cartesian plane, you can consider v as a linear combination of the standard basis vectors i and j, as shown in Figure 5.
But if you use another coordinate system, v is associated with new numbers. Figure 6 shows a representation of the vector v with a new coordinate system (i’ and j’).
In the new basis, v is a new set of numbers:
You can use a change of basis matrix to go from a basis to another. To find the matrix corresponding to new basis vectors, you can express these new basis vectors (i’ and j’) as coordinates in the old basis (i and j).
Let’s take again the preceding example. You have:
and
This is illustrated in Figure 7.
Since they are basis vectors, i’ and j’ can be expressed as linear combinations ofi and j.:
Let’s write these equations under the matrix form (more details about the matrix form of systems of equations in Chapter 08 of Essential Math for Data Science):
To have the basis vectors as columns, you need to transpose the matrices. You get:
This matrix is called the change of basis matrix. Let’s call it C:
As you can notice, each column of the change of basis matrix is a basis vector of the new basis. You’ll see next that you can use the change of basis matrix C to convert vectors from the output basis to the input basis.
Change of basis vs linear transformation
The difference between change of basis and linear transformation is conceptual. Sometimes it is useful to consider the effect of a matrix as a change of basis; sometimes you get more insights when you think of it as a linear transformation.
Either you move the vector or you move its reference. This is why rotating the coordinate system has an inverse effect compared to rotating the vector itself.
For eigendecomposition and SVD, both of these views are usually taken together, which can be confusing at first. Keeping this difference in mind will be useful throughout the end of the book.
The main technical difference between the two is that change of basis must be invertible, which is not required for linear transformations.
Finding the Change of Basis Matrix
A change of basis matrix maps an input basis to an output basis. Let’s call the input basis B1 with the basis vectors i and j, and the output basis B2 with the basis vectors i’ and j’. You have:
and
From the equation of the change of basis, you have:
If you want to find the change of basis matrix given B1 and B2, you need to calculate the inverse of B1 to isolate C:
In words, you can calculate the change of basis matrix by multiplying the inverse of the input basis matrix (B1^{-1}, which contains the input basis vectors as columns) by the output basis matrix (B2, which contains the output basis vectors as columns).
Be careful, this change of basis matrix allows you to convert vectors from B2 to B1 and not the opposite. Intuitively, this is because moving an object is the opposite to moving the reference. Thus, to go from B1 to B2, you must use the inverse of the change of basis matrix C^{-1}.
Note that if the input basis is the standard basis (B1=I), then the change of basis matrix is simply the output basis matrix:
Since the basis vectors are linearly independent, the columns of C are linearly independent, and thus, as stated in section 7.4 of Essential Math for Data Science, C is invertible.
Let’s change the basis of a vector v, using again the geometric vectors represented in Figure 6.
You’ll change the basis of v from the standard basis to a new basis. Let’s denote the standard basis as B1 and the new basis as B2. Remember that the basis is a matrix containing the basis vectors as columns. You have:
and
Let’s denote the vector v relative to the basis B1 as [v]B1:
The goal is to find the coordinates of v relative to the basis B2, denoted as [v]B2.
Square bracket notation
To distinguish the basis used to define a vector, you can put the basis name (like B1) in subscript after the vector name enclosed in square brackets. However, due to Medium subscript limitations, we’ll write for instance, [v]B1 in text to denote the vector v relative to the basis B1. It is also called the representation of v with respect to B1.
Let’s express the vector v as a linear combination of the input and output basis vectors:
The scalars c1 and c2 are weighting the linear combination of the input basis vectors, and the scalars d1 and d2 are weighting the linear combination of the output basis vectors. You can merge the two equations:
Now, let’s write this equation in matrix form:
The vector containing the scalars c1 and c2 corresponds to [v]B1 and the vector containing the scalars d1 and d2 corresponds to [v]B2. You have:
That’s good, this an equation with the term you want to find: [v]B2. You can isolate it by multiplying each side by B2^{-1}:
You have also:
The term B2^{-1} B1 is the inverse of B1^{-1} B2, which is the change of basis matrix C described before. This shows that C^{-1} allows you to convert vectors from an input basis B1 to an output basis B2 and C from B2 to B1.
In the context of this example, since B1 is the standard basis, it simplifies to:
This means that, applying the matrix B2 ^{-1} to [v]B_1 allows you to change its basis to B2.
Let’s code this:
array([ 0.86757991, -1.00456621])
These values are the coordinates of the vector v relative to the basis B2. This means that if you go to 0.86757991 i’ — 1.00456621 j’ you arrive to the position (2, 1) in the standard basis, as illustrated in Figure 6.
Conclusion
Understanding the concept of basis is a nice way to approach matrix decomposition (also called matrix factorization), like eigendecomposition or singular value decomposition (SVD). In these terms, you can think of matrix decomposition as finding a basis where the matrix associated with a transformation has specific properties: the factorization is a change of basis matrix, the new transformation matrix, and finally the inverse of the change of basis matrix to come back into the initial basis (more details in Chapter 09 and 10 of Essential Math for Data Science).
This post is a sample of my book Essential Math for Data Science!
Get the book here: https://bit.ly/2WVf4CR!