AI Essentials: Working with Matrices
By: Ross Bulat
Tags: Python, Neural Networks, Artificial Intelligence, Deep Learning, AI

Matrix Multiplication

Matrix multiplication follows a slightly more verbose methodology than simple addition or subtraction. When multiplying matrices, each row of the previous matrix multiplies with each column of the next matrix. This will become clearer in the example to follow.

One way to think of this process is that each row of the first matrix processes each column of the second matrix. Let’s firstly present two 2 x 2 matrices to multiply:

To calculate the final matrix (that is currently empty) each row of the first matrix needs to multiply with every column of the second matrix. The resulting matrix is known as the dot product of the matrices being multiplied.

Dot products are widely used in machine learning, with NumPy supplying their own dot function to leverage.

This process is admittedly is extremely hard to visualise, so let’s break it down into stages. The first row and column process resembles the following:

Notice that we’re starting with the first row of matrix 1, and the first column of matrix 2. The corresponding values are multiplied, with their products added together resulting in one dot product value.

Since there are more columns to multiply, row 1 then multiples with the next column of matrix 2:

Now row 1 of matrix 1 has processed all columns of matrix 2, we can repeat the whole process for row 2:

And finally process the last column of matrix 2 with the currently active row of matrix 1:

And this concludes the dot product process. We can now simplify the resulting matrix with its true values:

This concludes the dot product process! However, there are some requirements in order for a multiplication to be valid. The above example multiplies two matrices of shape 2 x 2 for simplicity, but there are a couple of rules that one must be aware of when dealing with matrix multiplication that pertains to the shapes and ordering of the matrices being multiplied. Let’s briefly take a look at these requirements.

The requirements of matrix multiplication

Even though the size of the matrices being multiplied does not need to be the same, the inner numbers must match. This requirement is obviously always met in the case of square matrices, but when the dimensions are different, then the column count of matrix 1 must match the row count of matrix 2.

The shape of the resulting matrix is also determined by the shapes of the multiplying matrices— their outer numbers to be exact. These two rules are summarised in the following multiplication:

Outer values determine the resulting shape, and the inner values must be the same for the multiplication to be valid, or defined. If the inner values are not the same, the resulting matrix will be undefined.

Because of these requirements, it is not possible to multiply a non-square matrix with itself! The result would be undefined. This would entail multiplying, for example, a 2 x 3 matrix with a 2 x 3 matrix, and the inner numbers here do not match. We could however do an element-wise multiplication, that will be mentioned further down.

With multiplication, ordering also matters

Unlike scaler products, the resulting matrix will differ depending on which way round the matrices are in your formula. In other words, matrix multiplication is not commutative.

If we take the product of two regular numbers, it does not matter which way round they are multiplied. E.g:

6 x 4 = 24 A x B

4 x 6 = 24 B x A

== Same result

This is not the case with matrices. Re-ordering even the simplest of matrices yields a drastically different dot product:

Dot product vs element-wise multiplication

Another type of matrix multiplication is known as element-wise multiplication. This is where each index of matrix A is multiplied with matrix B. For this type of multiplication to be valid, both matrices must be the same shape.

Element-wise operations are useful for adjusting a matrix in some way, such as normalising the elements with element-wise multiplication, or adjusting magnitude by increasing or decreasing all the elements uniformly.

Did you notice? — matrix addition and subtraction are also element-wise operations.

Working with scalers

Matrices can also be manipulated using plain scaler numbers, with all the operations previously mentioned. This is a nice shortcut for manipulating every matrix element in the same way without having to define another matrix to achieve the same result — and works great in code. NumPy supports such operations out of the box:

""" Scaler operations on matrices """ import numpy as np arr1 = np.array([[2, -3], [5, 9], [-4, -0]]) # adds 5 to each matrix element

print(arr1 + 5)

>>> [[ 7 2]

[10 14]

[ 1 5]] # subtracts 20 from each matrix element

print(arr1 - 20)

>>> [[-18 -23]

[-15 -11]

[-24 -20]] # multiplies each matrix element by 5

print(5 * arr1)

>>> [[ 10 -15]

[25 45]

[-20 0]]

Let’s explore some key interesting properties with matrix multiplication next.

What about division? There is no such thing as division when working with matrices. You can add, subtract, and multiply matrices, but you cannot divide them. There are concepts that loosely resemble division such as inverse matrices — this will be covered further down.