Array Oriented Programming with Python NumPy | by Semi Koen | Towards Data Science

NumPy is the most popular Python library for high-performance array implementation: operations on arrays are a lot faster than those on lists, which in the world of big-data it can make an amplified runtime difference. According to libraries.io (as of Apr 2020) over 4K libraries depend on NumPy, including the most popular Data Science packages, Pandas and SciPy.
The beauty of NumPy is the array-oriented programming style it offers. That is, instead of processing the array elements using conditional for-loops (or nested for-loops when it comes to n-dimensions), it provides functional-style, vectorised operations with internal iterations, which make the array manipulations less elaborative and more succinct.
In this tutorial you will discover all you need to know to get from total beginner to advanced, over 10 comprehensive topics. But if you are already advanced, fear not! You may jump straight to the last two sections (Advanced & Expert) where the ‘crux’ of array orientation will be addressed, together with essential performance tips. By the end you will be able to change your procedural programming style to the elegant, fast and loop-free, array-based style.
Let’s start . . .
After importing NumPy import numpy as np, we are ready to roll:
We can create an array by passing a list of elements; in this example integers:array = np.array([1, 2, 3, 5, 8, 13, 21])
📌 Can you notice something interesting in the output?
All the comma separated values are of the same width and right aligned. Since the maximum number, 21, occupies 2 positions, all the other values are formatted as two-character fields. So now you know why there is a leading space between the [ and the 1. 😏
We will create a 2 row by 3 column array by passing two nested lists of three elements each:array2x3 = np.array([[1, 2, 3], [10, 20, 30]])
The format of the output is based on the number of dimensions, aligning the columns within each row: as we can see the 1 and 10 are aligned, the 2 and 20 etc.
We can determine the array’s shape with the shape attribute, which returns a tuple with the dimensions:
And the number of dimensions (i.e. the array’s rank) with the ndim attribute, like so:
The ndim is the same as the number of axes or the length (len) of the output of the array’s shape:
Using the attribute size we can get the array’s total number of elements:
As NumPy is written in C, it uses its data types. As such integers are stored as int64 values — which correspond to 64-bit (i.e. 8-byte) integers in C.
Determining the array’s element type is easy, by accessing the dtype attribute:
and the number of bytes required to store each element, by accessing the itemsize:
The memory footprint (nbytes)is the number of elements times the number of bytes.
To fill an array with specific values, NumPy provides three special functions: zeros, ones and full, which respectively create arrays containing 0s, 1s or a specified value. Please note that zeros and ones contain float64 values, but we can obviously customise the element type.
A 1D array of 0s: zeros = np.zeros(5)
A 1D array of 0s, of type integer: zeros_int = np.zeros(5, dtype = int)
A 2D array (2x5) of 1s: ones = np.ones((2, 5))
A 2D array (2x5) of 7s: full = np.full((2, 5), 7)
NumPy provides optimised functions for creating arrays from ranges. The two most important functions to create evenly spaced ranges are arange and linspace, for integers and floating points respectively.
◼️ Integers: Given the intervalnp.arange(start, stop, step): Values are generated within the half-open interval [start, stop) — i.e. the interval including start but excluding stop. The default start value is 0 and the default step size is 1.
Range from 0 to 4, with an increment of 1:np.arange(5)
Range from 5 to 9, with an increment of 1:np.arange(5, 10)
Range from 2 to 8, with an increment of 2:np.arange(2, 10, 2)
Range from 8 to 2, with an increment of -2:np.arange(8, 0, -2)
◼️ Floats: Given the number of elementsnp.linspace(start, stop, num, endpoint): Returns num evenly spaced samples, calculated over the interval [start, stop]. The endpoint of the interval can optionally be excluded. The default num value is 50 and the default endpoint is True.
5 evenly spaced elements from 1.0 to 2.0 (excluding 2.0):np.linspace(1, 2, num = 5, endpoint = False)
5 evenly spaced elements from 1.0 to 2.0 (including 2.0):np.linspace(1, 2, num = 5, endpoint = True)
💡 Although we can create arrays by passing ranges as arguments, always prefer the built-in functions as they are optimised for performance.
To generate random ranges, NumPy provides a few options, but here are the most popular:
◼️ Random samples from a uniform distribution over [0, 1)np.random.rand(d0, d1, ...) where dn are the array dimensions:
1D array with 5 random samples:np.random.rand(5)
2D array with 2 rows and 5 random samples each:np.random.rand(2, 5)
◼️ Random integersnp.random.randint(low, high, size): Returns random integers from low (inclusive) to high (exclusive). If high is None (the default), then results are from [0, low).
10 random integers from 1 to 99:np.random.randint(low = 1, high = 100, size = 10)
We can reference an array element using its zero based index enclosed in square brackets, so for instance in a five element array, the first element is accessed by [0] and the last one is accessed by [4].
Arrays can also be accessed from the end by using negative indices, starting from -1 for the last element. So in the previous example, the last element ([4]), can be accessed with [-1] and its first element ([0]) with [-5]:
So in this array:array1D = np.array([0, 1, 2, 3, 4])
we can select the 3rd element: array1D[2]
or starting from the end: array1D[-3]
2D arrays can be accessed with this notation: [row_index, column_index].
So in this instance: array2D = np.array([[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]])
we can select the element at the 2nd row and 3rd column by: array2D[1, 2]
Slicing arrays results to a subset of the original elements, using the [first:last] notation, which returns a sub array with elements from index first to last-1.
→ If first is omitted, 0 is assumed, so the elements from the beginning till the last-1 are returned.→ If last is omitted, the array’s length is assumed, so the elements from first till the end are returned.→ If both first and last are omitted, the whole array is returned.
Carrying on the previous array1D example:
Sub array with 2nd and 3rd elements (from index 1 to 2):array1D[1:3]
Sub array with first 3 elements (from beginning to index 2):array1D[:3]
Sub array with last 3 elements (from 2nd index to end):array1D[2:]
Sub array with all the elements:array1D[:]
Similar principles are applied in the 2D arrays, so slicing uses this notation:[row_first:row_last, column_first:column_last].
→ To select multiple rows we use: [row_first:row_last, :].→ To select multiple columns we use: [:, column_first:column_last].
Carrying on the previous array2D example:
Sub 2D array from row indices 0 to 1 and column indices 1 to 2array2D[0:2, 1:3]
Sub 2D array with multiple columns from beginning to index 2:array2D[:, :3]
Sub 2D array with multiple rows from index 1 to end:array2D[1:, :]
Sub array with all the elements:array2D[:, :]
Moving to the next section we will explain why, Numpy Sub Arrays are just views!
Slicing does not modify the original array. The newly created array makes shallow copies (or views) of the original elements — that means it copies the elements’ references but not the objects they point to.
In other words, any modification to the newly created array will be reflected in the original array too.
Here is a newly created array with 10 elements:originalArray = np.arange(0, 10)
We slice it by selecting the last 5 elements:subArray = originalArray[5:]
We observe that the original array and the sliced one are indeed two different objects, by using the built-in id function:display(id(originalArray)) and display(id(subArray)).
To prove that subArray views the same data as originalArray, we will modify the first element of the subArray:subArray[0] = subArray[0] * 10
then display both arrays — the original value has also changed:
Although shallow copies save memory as they share data, sometimes it is necessary to create independent copies of the original data. This operation is called deep copying and is useful in multi-threading programming where separate parts of our program could attempt to modify the data at the same time — possibly corrupting it.
NumPy provides the method copy, which returns a new array object with a deep copy of the original array object’s data.
Repeating the previous example, we can see that the original array is not impacted when modifying the sub array.
We first copy the originalArray to a newArray:newArray = originalArray.copy()
We then modify the 6th element:newArray[5] = newArray[5] * 10
and finally we display both arrays — see how original array’s value is unchanged?
NumPy provides many operators which enable us to write simple expressions that perform operations on entire arrays. This is our stepping stone to becoming advanced later and eliminate for-loops in our programs!
We can perform element-wise arithmetic operations with arrays and scalars. During these operations the scalar is applied to every array element, so this snippet adds 5 to every element: array + 5.
Each operation returns a new array containing the result (i.e. does not modify the original array).
Augmented assignments modify every element in the left operand. In this example: array += 1, the elements of the original array are now incremented by 1. No memory allocation happens here, contrary to the previous case, as such in-place operations are preferred.
Multiplication, subtraction, division and exponentiation are as easy as the previous addition.
📌 Operating with scalars is the simplest form of Broadcasting that we will be discussing in the next section.
We can perform element-wise arithmetic operations between arrays of the same shape. The result is a new formed array with the elements of both arrays combined.
In this example, we have two arrays with 5 elements each:
arrayA = np.arange(2, 12, 2)arrayB = np.arange(0, 5) + 1
When we add them: arrayA + arrayB, we can see the new combined array:
📌 Please note that array multiplication is not matrix multiplication. The elements are solely component-wise multiplied. Using the previous arrays, we can calculate the matrix multiplication by using the dot function:np.dot(arrayA, arrayB)
When comparing two arrays, we get a Boolean array in which each element is True or False, indicating the respective comparison result.
Given two arrays of random integers:
compA = np.random.randint(low = 1, high = 10, size = 5)compB = np.random.randint(low = 1, high = 10, size = 5)
we can check whether the elements of the 1st array are greater than the elements of the second one: compA > compB.
📌 To compare complete arrays for equality we use the array_equal function, which returns True if two arrays have the same shape and same elements:np.array_equal(compA, compB)
Additionally, we can apply logical OR and AND to arrays element-wise, by using the logical_or and logical_and functions respectively.
Given two Boolean arrays:
logA = np.array([True, True, False, False])logB = np.array([True, False, True, False])
we get:np.logical_or(logA, logB) and np.logical_and(logA, logB).
Moreover, NumPy offers many standalone universal functions (or ufuncs) that perform various element-wise operations. Each returns a new array containing the results.
The official documentation lists five categories — math, trigonometry, bit manipulation, comparison and floating point. A quick browse is recommended so you know what is available.
Well done! You made it to the next level!
So far these operations require as operands two arrays of the same size and shape. Broadcasting relaxes these constraints when the arrays’ shapes are compatible, enabling some concise and powerful manipulations.
The smaller array becomes ‘broadcast’ across the larger array. It allows us to avoid loops and also create unnecessary copies of our data.
Based on the official documentation:
When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions, and works its way forward.Two dimensions are compatible when:• they are equal, or• one of them is 1If these conditions are not met, a ValueError: operands could not be broadcast together exception is thrown.
We will demonstrate the operating principles of broadcasting in a few of step-by-step multiplication examples. Pretty much, the smaller array ‘stretches’ so that it is multiplied with each row of the larger array.
It is not literally stretched in memory; it is the computation that is repeated.
In this example we have a 3x3 table and a 1x3 table.
ex1A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])ex1B = np.array([2, 5, 10])
We can observe that the broadcasting conditions are met and both dimensions are compatible:
• Row dimension: Although they differ (i.e. 3 vs. 1), one of them is 1.• Column dimension: Both are the same (i.e. 3)
So the result of the multiplication (ex1A * ex1B) is:
In the 2nd example we have a 3x3 table and a 3x1 table.
ex2A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])ex2B = np.array([[2], [5], [10]])
Again, the arrays are compatible:
So the result of the multiplication (ex2A * ex2B) is:
In the 3rd example we have a 1x3 table and a 3x1 table.
ex3A = np.array([1, 2, 3])ex3B = np.array([[2], [5], [10]])
Again, the arrays are compatible:
So the result of the multiplication (ex3A * ex3B) is:
In this example, the number of rows is different (i.e. 3 vs. 2) and none of them is 1.
Similarly, the second from last dimensions mismatch:
After understanding broadcasting, another important concept is to manipulate the shape. Let’s see a few techniques:
It is common practice to create a NumPy array as 1D and then reshape it to multiD later, or vice versa, keeping the total number of elements the same.
📌 The reshape returns a new array, which is a shallow copy of the original.
Here is a 1D array with 9 elements: array09 = np.arange(1, 10).
We can reshape it as a 3x3 array: array09.reshape(3, 3).
or as a 9x1 array: array09.reshape(9, 1).
If we need to change the total number of elements, then we need to resize the array. If we are enlarging it, it will add trailing zeroes, until it reaches the new size. Otherwise it is truncated to the new size.
📌 The resize modifies the original array.
Carrying on this example, we can enlarge the 9-element array to 15 elements:array09.resize(15, refcheck = False).
or we can truncate it down to 5 elements:array09.resize(5, refcheck = False).
📌 Another desired behaviour of enlarging an array is to have it repeat itself until it reaches the new size. In this case we can use this ‘static’ version of resize, which takes the original array as an input:np.resize(array09, 10)
Inserting a new axis into an array increases its dimensionality. newaxis is usually used to make arrays compatible for broadcasting.
In this example we will see how we can transpose a 1-row table to a 1-column:
array1x4 = np.array([1, 2, 3, 4])print(array1x4)print(np.shape(array1x4))array4x1 = array1x4[:, np.newaxis]print(array4x1)print(np.shape(array4x1))
Quite neat, right?
Flattening is the inverse operation to reshaping. We can flatten a multidimensional array into a single dimension with the methods flatten and ravel. Method flatten deep copies the original array’s data, while ravel shallow copies it, [as such flatten is slower].
Here is a 2x2 array: array2x2 = np.arange(1, 5).reshape(2, 2).
We flatten it — deep copy: array2x2.flatten()
then quickly re-set it to 2x2: array2x2 = array2x2.reshape(2, 2)
and then unravel it again — shallow copy: array2x2.ravel()
Transposing an array i.e. permuting its dimensions (meaning turning its shape from (X,Y) to (Y,X)) is also very straight forward.
📌 The transpose returns a new array, which is a shallow copy of the original.
Here is an array:toBeTransposed = np.arange(1, 5).reshape(2, 2)
and here is its transposed copy:toBeTransposed.transpose()
Joining or Concatenating means putting contents of two or more arrays in a single array, along a specified axis, by ‘stacking’ them under (axis = 1) or next to (axis = 0) each other. There are a few options to do that, but the concatenate function is the most popular.
Here are two 3x3 arrays, arr1 and arr2 :
arr1 = np.arange(9).reshape(3, 3)arr2 = arr1 * 2
◼️ Vertical Concatenation [Row-wise | axis = 0]np.concatenate((arr1, arr2))
◼️ Horizontal Concatenation [Column-wise | axis = 1]np.concatenate((arr1, arr2), axis = 1)
Splitting is the reverse operation of Joining, i.e. we split the contents of an array to multiple sub-arrays, along a specified axis.We can either split them into arrays of the same shape or indicate the position after which the split should occur.
Using this nine element array (arr3), we will see these two variations (on axis = 0):
◼️ Even Splitnp.split(arr3, 3)will produce 3 equal-sized sub-arrays:
◼️ Uneven Split, based on the positions indicated in a 1D arraynp.split(arr3, [2, 7]))will produce 3 sub-arrays, split after the 2nd and 7th elements:
'Map, Filter and Reduce' are paradigms of functional programming. They allow us to write simpler, shorter code, without needing to bother about for-loops or if-else branching. Even if they are powerful on their own, 'the cherry on top' is that they can be chained together.The next three sections will cover exactly that, when it comes to NumPy arrays.
To apply a transformation into NumPy arrays, we will employ the vectorize function. A new array is returned with the mapping taken effect.
Here is a simple array:toBeMapped = np.arange(0, 10)
We will now double all the elements of the array, by passing a lambda anonymous function:np.vectorize(lambda x: x * 2)(toBeMapped)
If we had a more complex calculation (with many lines of code), we can pass a function instead.
📌 According to the documentation, ‘the vectorize function is provided primarily for convenience, not for performance. The implementation is essentially a for loop’. Nevertheless readability is key here, so although under the hood it uses a for loop, our code doesn't!
Filter, tests each element with a unary predicate. NumPy provides the extract function for this. Elements that satisfy the predicate are kept; those that don’t are removed. A new array is returned; filter doesn’t modify the original array.
Here is our example array:toBeFiltered = np.arange(0, 9).reshape(3, 3)
We would like to select the odd numbers:np.extract(toBeFiltered % 2 == 1, toBeFiltered)
An equivalent way of achieving the same result is as follows:toBeFiltered[toBeFiltered % 2 == 1]
When faced with a large amount of data, a desired step is to reduce the dimensions by applying a calculation across the whole array or over one of its axis.
Here is a 2x3 array:toBeReduced = np.arange(1, 7).reshape(2, 3)
◼️ Over the whole arraytoBeReduced.sum()will produce a scalar representing the sum of all the elements:
◼️ Column-wise [axis = 0]toBeReduced.sum(axis = 0)will produce a 1x3 array, where each element is the sum of the respective column:
◼️ Row-wise [axis = 1]toBeReduced.sum(axis = 1)will produce a 3x1 array, where each element is the sum of the respective row:
They all work the same way. For example:
◼️ Min/MaxtoBeReduced.min() and toBeReduced.max()
◼️ OtherTwo other important categories include:
Logical Reductions (Truth Value Testing)
Statistics
which I suggest you skim through and then learn as you go.
My compliments! You are ready to bid for-loops Adieu!
To optimise the performance of our programs the first thing we do is to move computations or memory allocations outside a for-loop. We have already covered a lot of material on NumPy techniques, but let’s distill these tips and tricks and make them explicit, to help you write more performant code.
Vectorised are the operations that process entire arrays in an element-wise fashion. They turn the Python explicit for-loops into implicit, low-level, optimised and compiled C implementations, and therefore you should prefer them.
You should avoid unnecessary deep copies; use shallow copies (or views) and in-place operations whenever possible.
You should use broadcasting to do operations on arrays as small as possible before combining them; the smallest dimension is internally stretched to match the other dimension, but this operation does not involve any memory copy.
Numpy arrays are stored in row-major order — i.e. the flattened memory is represented row-by-row. As such, storing data in a contiguous block of memory ensures that the architecture of modern CPUs is used optimally, in terms of memory access patterns and CPU cache.Consequently, accessing or traversing an array in a contiguous way is much faster than random access, so you should choose wisely how to represent the data in n-dimensions. As an illustrative example, summing elements along rows is faster than along columns.
You should use Map, Filter, Reduce to avoid for-loops and conditional programming.
Wonderful! You made it to the end!
100 Numpy Exercises
Real Python: Array Programming With NumPy
The NumPy array: a structure for efficient numerical computation** Must read **
SciPy Lectures: Advanced NumPy
Beautiful Code [Essay 19: ‘Multidimensional Iterators in Numpy’]
I hope you feel enlightened by how powerful NumPy is and the flexibility it evokes. You are now equipped with knowledge and skills needed to help you optimise the performance of your code and eliminate the for-loops as much as possible. All you need to do is:Practise — Practise — Practise!
Thanks for reading!
I regularly write about Leadership, Technology & Data on Medium — if you would like to read my future posts then please ‘Follow’ me!