Linear Algebra for Machine Learning: Solve a System of Linear Equations | by Khuyen Tran | Towards Data Science

Let’s start with the common plot in data science: Scatter plot
The plot above represents the correlation between the diameter and the height of the tree. Each dot is a sample of a tree. Our task is to find the best fit line to predict the height provided the diameter.
How can we do that? That is when Linear Algebra is needed.
Linear regression is an example of linear systems of equations. Linear Algebra is about working on linear systems of equations. Rather than working with scalars, we start working with matrices and vectors.
Linear Algebra is the key to understanding the calculus and statistics you need in machine learning. If you can understand machine learning methods at the level of vectors and matrices, you will improve your intuition for how and when they work. The better linear algebra will lift your game across the board.
And what is the best way to understand linear algebra? Implement it. There are 2 methods to solve a system of linear equations: direct methods and iterative methods. In this article, we will use the direct methods, in particular, Gauss-method.
Since most often time we work with data with many features (or variables). We will make our system of linear equations more general by working with a 3-dimensional data instead.
Let’s generate an example for the plot above:
where the coefficients of x_0, x_1, and x_2 and the corresponding values 8, 4, 5 are the samples of points in the plot. The equations can be split into matrices A, x, and b
where A and b are the matrices of known constants, x is the vector of unknown variables.
A = np.array([[2, 1, 5],            [4, 4, -4],            [1, 3, 1]])b= np.array([8,4,5])
Concatenate matrix A and b to get
n = A.shape[0]    C=np.c_[A,b.reshape(-1,1)]
Now we are ready to tackle our problems with 2 steps:
Apply Gaussian Elimination to reduce the matrix above to triangular matrix
Apply Gaussian Elimination to reduce the matrix above to triangular matrix
which could be represented by the equation:
2. Apply backward substitution to obtain the result
Let’s start with the first step
To obtain this matrix:
the idea is simple:
We start with the pivot value in the first row and first column: row =0, column = 0
Find the maximum absolute value of the column of the pivot. If all the values in that column are 0, we stop.
Else, we exchange E_0 and E_1
Next, apply equivalent transformations to convert all the entries below the pivot to 0 by:
Find the ratio between element j,i and pivot i,i (.i.e, 2/4 = 1/2).Multiply all the elements in E0 with 1/2. Subtract all the elements in row 1 by 1/2 E0 (i.e, 2-(4*1/2)=2–2 =0)
Find the ratio between element j,i and pivot i,i (.i.e, 2/4 = 1/2).
Multiply all the elements in E0 with 1/2. Subtract all the elements in row 1 by 1/2 E0 (i.e, 2-(4*1/2)=2–2 =0)
#rowfor j in range(i+1, n):                c = C[j,i]/C[i,i]    C[j,:] = C[j,:] - c*C[i,:]
After each column is simplified, we proceed to the next column to the right.
Repeat the procedure:
Pivot: row = 1, column = 1. Maximum absolute value: 2 in row 2. Then, permute E_1 and E_2
Applying the equivalent transformation to get all the entries below the pivot converted to 0
Put everything together
Nice! Now we have a system of equations:
Once we get here, this system of equation is incredibly easy to solve using backsubstitution
From Gaussian Elimination, we obtain a triangular matrix
The idea is to solve the system of equations above by solving from the bottom up. Use the value obtained from the last equation to find the other values
Start with row 3. Divided 8 by 8 to get the value of x_3.
X[n-1] = T[n-1,n]/T[n-1,n-1]
Now in the second row, we have:
x_2 could be solved easily by
Repeat with x1
Thus, in general, backsubstution can be represented as:
Awesome! We obtain a solution as we predict. To make sure this is correct when working with a larger matrix, we can use the built-in function in NumPy
>>> np.linalg.solve(A,b)array([1., 1., 1.])
What we get is a vector of solutions where each element corresponding to x_0, x_1, x_2
Congratulation on getting this far! I hope this article helps you understand what is linear algebra and one of the mechanisms of solving a system of linear equations. I try to make this article understandable as much as possible. But I understand that this could be challenging if you are not familiar with linear algebra. That is okay! One step at a time. The more you are exposed to linear algebra, the more you will understand it.
You could play and experiment with the codes above in my Github.
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: