Introduction to Linear Algebra with NumPy | by Soner Yıldırım | Towards Data Science

Data is collected in many different formats from numbers to images, from categories to sound waves. However, we need the data represented with numbers to be able analyze it on computers. Machine learning and deep learning models are data-hungry. The performance of them is highly dependent on the amount of data. Thus, we tend to collect as much data as possible in order to build a robust and accurate model. As the amount of data increases, the operations done with scalars start to be inefficient. We need vectorized or matrix operations to make computations efficiently. That’s where linear algebra comes into play.
Linear algebra is one of the most important topics in data science domain. In this post, we will cover some basic terms in linear algebra and go through examples using NumPy, a scientific computing library for Python.
There are different types of objects (or structures) in linear algebra:
Scalar: Single number
Vector: Array of numbers
Matrix: 2-dimensional array of numbers
Tensor: N-dimensional array of numbers where n > 2
Let’s create an example for each of these objects. I will skip “scalar” since it is just a number. We first import numpy:
import numpy as np
A vector that has 4 elements:
A 2x4 matrix:
A 2x2x4 tensor:
In terms of numpy representations, vector is 1-D array, matrix is 2-D array, and tensor is n-D array (n>2):
We can also create higher dimensional arrays with numpy but those are hard visualize. Let’s create a 4-D array with a size of 2 in each dimension.
The total size of an array is the product of sizes in each dimension. The printed version of T2:
We have covered the basic structures. Now it is time to discuss basic operations that can be done on these structures.
When a vector is multiplied by a scalar, each element of the vector is multiplied with the scalar:
Another common operation is dot product. The dot product of two vectors is the sum of the products of elements with regards to position. The first element of one vector is multiplied by the first element of the second vector and so on. The sum of these products is the dot product which can be done with np.dot() function.
Dot product is an essential part in matrix multiplication that we will discuss later in this post.
Element-wise summation and subtraction on vectors are done with standard math operations.
Matrix multiplication is a widely-used operation in linear algebra. When solving a system of linear equations, matrix multiplication comes in very handy. In complex machine learning and deep learning models, many matrix multiplications are done under the hood. The optimization algoritms are performing numerous matrix multiplications to find model parameters. Thus, it is a must-know operation.
Before we dive into matrix multiplication, it is better to understand some properties of matrices. A matrix is a 2-dimensional array. If the number of elements (size) in both dimensions are the same, then we have a square matrix.
Identity matrix is a square matrix that contains ones on the main diagonal and zeros in all other positions.
We define the shape of a matrix in terms of the number of rows and columns. For instance, matrix A below has 3 rows and 4 columns. The indices start from zero so we can access the first row by A[0], and the first column by A[:,0]. A specific element can be accessed by passing both row and column indices. For instance, A[1,1] gives us the element at the second row and second column.
Matrix multiplication of two matrices is based on dot products of rows of one matrix with the columns of other matrix. Each element of the product matrix is a dot product of a row in first matrix and a column in the second matrix. Consider matrices A1 and A2 below. The first row of A1 is [2,2] and the first column of A2 is [1,4]. Dot product of these two vectors is (2x1)+(2x4)=10 which is the element of the product matrix at position [0,0]. The dot product of third row of A1 and third column of A2 gives us the element at position [2,2] which is 12.
Since the rows of the first matrix are multiplied by the columns of the second matrix, the number of rows of first matrix must be equal to the number of columns of the second matrix. If we multiply mxn matrix (m rows, n columns) with nxk matrix, the shape of the product matrix is mxk:
Thus, matrix multiplication is not commutative which means we cannot switch the order of matrices in the multiplication. Thus, AB is not equal to BA.
When we multiply a matrix with its inverse, we get the identity matrix. Thus,
Please note that not all the matrices have an inverse. For a matrix to have an inverse, it must be a square matrix (i.e. number of rows equal to the number of columns). However, not all the square matrices have an inverse either.
If a square matrix have an inverse, it is called invertible or non-singular. If a square matrix does not have an inverse, it is a singular matrix. Non-square matrices do not have an inverse.
One important feature of the identity matrix is that it does not change a matrix when multiplied. So, AI = A. Let’s confirm with numpy:
As we can see, the order does not matter for identity matrix. It does not change a matrix when multiplied from right or left. Thus, left inverse of a matrix is equal to the right inverse:
The inverse of the inverse of a matrix is itself:
There are different ways to find the inverse of matrix. One way is called elimination. It is pretty simple but I will leave it for another post not to make this one too long.
One of the basic operations on matrices is transpose which means switching rows and columns. It can be done using one letter (T) with numpy:
We have covered some of the basic concepts in linear algebra. There are, of course, more complex concepts and operations. However, it is always a good practice to build the knowledge step-by-step from basic to advance.
Thank you for reading. Please let me know if you have any feedback.