Python Numba or NumPy: understand the differences
By: Mohammed Alhamid
Tags: Numpy, Towards Data Science, Numba, Python, Machine Learning

Python Numba or NumPy: understand the differences

Photo by Patrick Tomasso on Unsplash

NumPy and Numba are two great Python packages for matrix computations. Both of them work efficiently on multidimensional matrices. In Python, the creation of a list has a dynamic nature. Appending values to such a list would grow the size of the matrix dynamically. NumPy works differently. It builds up array objects in a fixed size. Adding or removing any element means creating an entirely new array in the memory. In this article, we are looking into finding an efficient object structure to solve a simple problem.

What is essential to discuss is not only how the array objects are created, but how to apply scientific operations on those arrays, particularly scanning arrays. Performance is the principal motivation of having those libraries when we apply some expensive logic to them. For instance, when we develop Machine Learning (ML) models, especially in production environments, we spend a reasonable amount of time optimizing the code that generates the training data applying any required data transformation or any other ETL operation. Using some compiled programming languages like C or Fortran is ideal, but it would need us to build some wrappers here and there to bring the pipeline back to Python.

Implementing Arrays in Python

Let us have a simple example: First, we will create a simple list in python with ten million values. The big number would highlight the differences in performance easily.

# We need to import the random package to fillup the array with some random values. import random

array = []

I am using IPython; if you are running this code on Jupyter Notebook, then I recommend using built-in magic (time).

%%time

for i in range(0,10000000):

val = random.randint(0,1000)

array.append(val)

Computing the Frequency

Let us search in this list how many rows contain the value 999?

%%time

freq = 0

for value in array:

if value == 999:

freq = freq + 1

It took my machine 461 ms, and the function found 10184 instances of the value 999. Now let us see how to do the same job using NumPy arrays.

import numpy as np # We use the same list created earlier:

np_array = np.array(array)

# To get a sense of the size of the bumpy array, simply call up the function 'shape'

print('The size of the numpy array: ', np_array.shape)

Searching how many rows contain the value 999 in the NumPy array is only one line of code:

%%time

result = np.where(np_array == 999)

In addition to just writing a few instructions, it took my machine 12.6 ms for doing the same job as the list array.

Please note that the indexing mechanism of the NumPy array is similar to any ordinary Python list.

Now we will make the example a little bit more interesting by introducing some mathematical operations on the array values. First, we will construct three vectors (X, Y, Z) from the original list and then will do the same job using NumPy.

def apply_operation_list(array):

# We need to build an X vector from the original array

# The math operations in this function represents some random logic.

x_array = []

for array_v in array:

x_array.append(array_v * 2)



# Building the Y vector

y_array = []

for array_v, x_array_v in zip(array, x_array):

y_array.append(array_v + x_array_v)



# Building the Z vector

z_array = []

for array_v, x_array_v, y_array_v in zip(array, x_array, y_array):

if x_array_v == 0:

z_array.append(0)

else:

z_array.append((array_v - x_array_v ) + y_array_v)



return x_array, y_array, z_array %%time

x_array, y_array, z_array = apply_operation_list(array)

Applying the operation on the list took 3.01 seconds.

Let us define the same function with Numpy:

def apply_operation_numpy(array):

# We need to build an X vector from the original array

# The math operations in this function represent some random logic.

x_array = array * 2



# Building the Y vector

y_array = array + x_array



# Building the Z vector

z_array = ( array - x_array ) + y_array



return x_array, y_array, z_array

Using Numpy, it took only 132 ms.

Numba

Original photo is taken by Faris Mohammed on Unsplash

Numba works perfectly with Python and gives you the privilege to use your favourite math libraries but compiled to native machine instructions [2]. Using Numba is straightforward and does not require you to change the way you wrote the function:

# Loading the Numba package

# Jit tells numba the function we want to compile

from numba import jit

Note that all we have to change compared to Numpy function defined above.

@jit

def apply_operation_numba(array):

# We need to build an X vector from the original array

# The math operations in this function represents some random logic.

x_array = array * 2



# Building the Y vector

y_array = array + x_array



# Building the Z vector

z_array = ( array - x_array ) + y_array



return x_array, y_array, z_array def apply_operation_numba(array):# We need to build an X vector from the original array# The math operations in this function represents some random logic.x_array = array * 2# Building the Y vectory_array = array + x_array# Building the Z vectorz_array = ( array - x_array ) + y_arrayreturn x_array, y_array, z_array

Using Numba, the calculation of the three vectors took only 71.5 ms.

Which one to choose?

“NumPy is the fundamental package for scientific computing with Python. It contains among other things: a powerful N-dimensional array object, sophisticated (broadcasting) functions, tools for integrating C/C++ and Fortran code, useful linear algebra, Fourier transform, and random number capabilities” [1]

NumPy is a enormous container to compress your vector space and provide more efficient arrays. The most significant advantage is the performance of those containers when performing array manipulation.

Numba, on the other hand, is designed to provide native code that mirrors the python functions. Python can be looked at as a wrapper to the Numba API code. From my experience, we use Numba whenever an already provided Numpy API does not support the operation that we execute on the vectors. If the implemented customized function is not fast enough in our context, then Numba can help us to generate the function inside the Python interpreter. This is also the recommendation available from the Numba documentation.

The example provided earlier does not show how significant the difference is? This is true since we only search for the frequency of a single value. Let’s repeat the experiment by computing the frequency of all the values in a single column.

In Python, the most efficient way to avoid a nested loop, which is O^2 is the use of a function count(). With only one line of code, we can compute the frequencies of the full column:

%%time

count = {x:x_array.count(x) for x in x_array}

However, depending on your processing power, this function may take hours to complete 10-million records. Let’s see next what Numpy could offer:

%%time

x_elements_np, x_counts_np = np.unique(x_array_np, return_counts=True)

Computing the frequency of a million-value column took 388 ms using Numpy. A big performance relief! Note that this function is enhanced by computing the frequency of distinct values only. This is an example that shows how unrealistic to use a nested loop in a big data environment.

Until recently, Numba was not supporting np.unique() function, but still, you won’t get any benefit if used with return_counts. This just to show sometimes Numpy could be the best option to pick.

Finally, the next two figures show the runtime performance of using different data object structure. The x-axis represents the incremental increase of the size of the data from 10,000 rows to 1-billion rows.

The next figure shows the performance of the Numby with Numba library. Note that the number may vary depending on the data size. The numbers in the graph show the average of repeating the experiment for five times.

More Examples

Computing Singular Value Decomposition (SVD)

The frequency example is just one application that might not be enough to draw an impression, so let us pick SVD as another example. SVD is a well known unsupervised learning algorithm. It allows us to decompose a big matrix into a product of multiple smaller matrices. SVD has many application in ML and used to reduce the dimensionality. Here is a recommended article for further readings.

The example written below only uses two dimensions (columns) with the same number of rows as in our earlier example.

from scipy.linalg import svd # We will consider in this example only two dimensions.

# m x 2

# U: m x r

# s: Diagonal matrix r x r

# VT: n x r

U, s, VT = svd(array_np[:, 0:2])

Sigma = np.zeros((array_np.shape[0], 2))

Sigma[:2, :2] = np.diag(s)

If you try to run the code, you probably will get a similar error like the following failure: “ValueError: Too large work array required — computation cannot be performed with standard 32-bit LAPACK.”. That’s because the internal implementation of lapack-lite uses int for indices. With a size like our array, it definitely will cause an overflow. We either have to reduce the size of the vector or use an alternative algorithm. If the SVD function used with Numba, we will not get any noticeable benefits either since we are calling the LAPACK SVD function.

Matrix Multiplications

Matrix multiplication is another example that shows how Numba could be useful to boost up the processing time. Even without Cuda, we could achieve better performance. Let us take the example step by step. As we did before, we will implement a function using Python list.

def matrix_multiplication(A,B):

row, col_A = A.shape

col_B = B.shape[1] result = np.zeros((row,col_B)) for i in range(0,row):

for j in range(0,col_B):

for k in range(0,col_A):

result[i,j] += A[i,k]*B[k,j]

return result

For 10-million row, the list is pretty quick to process the multiplications. The runtime is only 1min and 7 seconds. Using Numpy, it took 95 seconds to the do the same job.

def matrix_multiplication_numpy(A,B):

result = np.dot(A,B)

return result

%%time

result = matrix_multiplication_numpy(array_np, array_np)

Now replacing Numby with Numba, we reduced the costly multiplications by a simple function which led to only 68 seconds that is 28% time reduction.

@jit

def matrix_multiplication_numba(A, B, result):

for i in range(result.shape[0]):

for j in range(result.shape[1]):

tmp = 0.

for k in range(A.shape[1]):

tmp += A[i, k] * B[k, j]

result[i, j] = tmp def matrix_multiplication_numba(A, B, result):for i in range(result.shape[0]):for j in range(result.shape[1]):tmp = 0.for k in range(A.shape[1]):tmp += A[i, k] * B[k, j]result[i, j] = tmp

The next figure shows the performance of matrix multiplication using a Python list, with Numby, and with Numba library.

The examples provided in this publication have been run on 15-inch 2018 MacBook Pro with 16 GB and using anaconda distribution. The code used in these examples can be found in my Github repo. The performance could be enhanced using a GPU environment, which was not considered in this comparison.

References

[1] Official NumPy website, available online at https://numpy.org

[2] Official Numba website, available online at http://numba.pydata.org