Illustrated Introduction to Linear Algebra using NumPy
By: Kanishk Srivastava
Tags: Linear Algebra, Numpy, Data Science, Data Visualization, Machine Learning

Matrix and Vector Products

In this section, we will discuss the various types of matrix and vector products we can calculate through the in-built functions of NumPy.

Dot Product

We can use np.dot(a,b) to perform a dot product of two matrices.

We can pass an optional output ( out ) argument to describe what kind of output we need. In particular, it must have the right type, must be C-contiguous, and it’s dtype must be what was returned for the dot product. This optional argument is a performance feature therefore, rather than being flexible, an exception is returned in case of any inconsistencies.

For a detailed description, check the documentation page for the function.

Note: If both a and b are 2-D matrices, using matmul is preferred.

Inner Product

We can use np.inner(a,b) to perform an ordinary inner product of two vectors.

When higher-dimension matrices are passed as arguments, we get a sum product of the last axes.

For a detailed description, check the documentation page for the function.

Outer Product

We can use the np.outer(a,b) to perform an ordinary outer product of two vectors. If higher dimensional matrices are passed on as arguments, they are flattened into a 1-D array before performing the operation.

We can use the optional output ( out ) argument. It works similarly to the output argument in the dot product function.

For a detailed description, check the documentation page for the function.

Matrix Multiplication

We can use np.matmul(a,b) to perform matrix multiplication on two matrices. If higher dimensional matrices are passed on an input, we process them as a stack of matrices with the dimensions of the last two axes. Broadcasting takes care of any dimensionality differences.

Look at the animation below to get a better understanding of the process of matrix multiplication.

For a detailed description, check the documentation page for the function.

Einstein Summation

We can implement Einstein Summation in NumPy through np.einsum .

Einstein Summation is generally used to speed up complex operations for arrays and matrices. In order to get a basic understanding of the effectiveness of this function, let’s look at a small example.

Consider two matrices A and B; let’s suppose we want to multiple them element-wise and then we want the sum along axis-1 (rows of the array).

Normally, in NumPy, we would first need to reshape A, and then we will have to broadcast it onto B to multiply ‘0’ with the first row, ‘1’ with the second row and ‘2’ with the third row. This would give us a new array where we could perform the summation along axis-1.

However, if we use einsum , we do not need to reshape A at all, and more importantly, we won’t have to make a temporary array while performing the operations. Instead, einsum simply sums up the products along the row as it goes. Look at the code snippet below to see how both of these methods are implemented in code.

Even in such a small-scale example, the einsum implementation is thrice as fast as the “basic approach”. Einstein summation does solve the problem of speed and space, however, it can be hard to understand the syntax and call signatures used to perform these operations. Here’s a handy cheat-sheet to help with some basic operations.

When A and B are compatible one-dimensional arrays; Source

When A and B are compatible two-dimensional arrays; Source

Einstein summation is a very lengthy topic to get into in this blog, therefore we won’t be covering it in depth. However, in order to get a better understanding of how einsum works in NumPy, have a look at this very helpful blog by Alex Riley.

Kronecker Product

We can compute the Kronecker product of two arrays by using np.kron . In mathematics, Kronecker product (denoted by ⊗ ) is an operation where two matrices of arbitrary size are given as inputs and we get a block matrix as the output. It is a generalisation of the outer product (which is also denoted by ⊗ ). This operation is not to be confused with basic matrix multiplication. Look at the illustration and code snippet below to understand how it works.

For a detailed description, check the documentation of this function.