NumPy Crash Course — Zero to Hero
By: Prejith Premkumar
Tags: Machine Learning, Python, Analytics, Numpy, Data Science

Setting up numPy

>> import numpy as np

The sole reason that numpy is imported as np is convention. You are free to use another alias but it's not recommended as this is what you will find everywhere and it's better to stick to standards

>> np.__version__ '1.18.1'

nd-array

The primary reason that numpy is fast is because of the nd-array type that it uses to store and manipulate data

An ndarray is a generic multidimensional container for homogenous data. It provides vectorized arithmetic operations and sophisticated broadcasting capabilities. Every ndarray has 2 properties: shape and dtype. shape is a tuple providing the dimension of the array and dtype provides you the datatype of the array.

The dtype of the array can also be explicitly specified while defining the array giving you fine-tuned control over the array.

Let's create a numpy array from an array. This is possible by passing the array as input to the np.array function

>> nparray = np.array([1,2,3]) array([1, 2, 3])

A numpy array has numerous properties which gives more information about them.

Datatype of the array

>> nparray.dtype dtype('int32')

Size of the array

>> nparray.size 3

Shape of the array

>> nparray.shape (3,)

The below 2 cells of code warrant a detailed explanation. The itemsize parameter returns the size of a single item in the array. In this case, we have an integer array taking up 32 bits of space for a single item and this is equivalent to 4 bytes(1 byte = 8 bits). The following cell showcases the nbytes parameter which returns the size in bytes of the entire array, thereby providing a value of 12 bytes(4 bytes * 3 items).

To put it short: itemsize provides the size of a single item in the array while nbytes returns the size of the entire array.

>> nparray.itemsize 4 >> nparray.nbytes 12

There is a reason that numpy’s nd-array is faster than Python’s native list. Let's look at this in depth in the following cells

>> %timeit pythonList = [i for i in range(10000)] 545 µs ± 24 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) >> %timeit npList = np.arange(10000) 7.82 µs ± 256 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

numpy arrays are homogenous and is handled faster in memory. Now compare this to a Python List where you can put anything in; every entry in a Python list is a Python object and this causes overhead in computations. This is the primary reason why numpy arrays are significantly faster than the traditional Python lists

Lets now look at some of the functions which makes numpy a flexible and handy library.

Generating data with numpy

arange generates a list of numbers within the range of the digit passed

>> np.arange(10) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

linspace returns a set of linearly-spaced items within the range passed as input. In linspace, the starting digit, ending digit along with the number of digits required as passed as input. Basically, it returns an array with the required number of digits in a specified interval

>> np.linspace(0, 10, 5) array([ 0. , 2.5, 5. , 7.5, 10. ])

ones creates an array filled with ones. The parameter passed as input is the size of the required array

>> np.ones(5) array([1., 1., 1., 1., 1.])

zeros creates an array filled with zeroes. The parameter passed as input is the size of the required array

>> np.zeros(5) array([0., 0., 0., 0., 0.])

zeros_like creates an array with the same size as the array passed as input. The generated array will have zeros as elements

>> np.zeros_like(np.arange(5)) array([0, 0, 0, 0, 0])

eye creates an identity matrix. The generated matrix will be of the dimensions of the integer passed as input

>> np.eye(5) array([[1., 0., 0., 0., 0.],

[0., 1., 0., 0., 0.],

[0., 0., 1., 0., 0.],

[0., 0., 0., 1., 0.],

[0., 0., 0., 0., 1.]])

empty creates an array filled with garbage values, usually zeroes. The parameter passed as input is the size of the required array

>> np.empty(5) array([1., 1., 1., 1., 1.])

Indexing

numpy follows the usual rules of Python when it comes to indexing and slicing. I am laying out a few examples below that you can play around with and experiment with:

>> nparray[1] 2 >> nparray[-1] 3

Slicing

>> nparray[1:2] array([2]) >> nparray[:2] array([1, 2]) >> nparray[:] array([1, 2, 3]) >> nparray[1:] array([2, 3]) >> largeArray = np.arange(100) >> largeArray[::20] array([ 0, 20, 40, 60, 80]) >> largeArray[1:10:2] array([1, 3, 5, 7, 9]) >> largeArray[::-20] array([99, 79, 59, 39, 19]) >> largeArray[10:1:-2] array([10, 8, 6, 4, 2])

An important thing to keep in mind while using slicing in numpy is that the slices are essentially references(views) and hence, any changes that you make to the sliced data will reflect in the parent. Lets look at an example

>> smallArray = largeArray[:10] >> smallArray array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >> largeArray[:10] array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >> smallArray[0] = 666 >> smallArray array([666, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >> largeArray[:10] array([666, 1, 2, 3, 4, 5, 6, 7, 8, 9])

The copy() method can be used to create copies instead of such views

Axes

A numpy array can be multi-dimensional. It also gives you the ability to change an existing array to a shape of your liking provided that it meets multiple constraints

reshape lets you do exactly what the name says. It lets you shape the array into the dimensions passed as input. If you do not pass a dimension that the array can be reshaped into, then the function will return an error. Say, the array has 10 elements and you try to reshape it to an array of shape 3x5, then reshape will return an error

>> np.arange(1, 7).reshape((2, 3)) array([[1, 2, 3],

[4, 5, 6]]) >> np.arange(1, 4) array([1, 2, 3]) >> np.arange(1, 4).reshape(1,3) array([[1, 2, 3]])

newaxis is used to create a new axis in the data. It is commonly used when working on modelling techniques as models require the data to be shaped in a certain manner. As you can see below, if the newaxis parameter is in the first position, then a new row vector will be generated. If its in the second position, then a column will be created with each of the elements being a separate vector.

>> np.arange(1, 4)[np.newaxis, :] array([[1, 2, 3]]) >> np.arange(1, 4)[:, np.newaxis] array([[1],

[2],

[3]])

Array Concatenation

Arrays can be concatenated in numpy using the concatenate method. The list of arrays to be concatenated is to be passed as input to the concatenate function.

>> np.concatenate([smallArray, largeArray]) array([666, 1, 2, 3, 4, 5, 6, 7, 8, 9, 666, 1, 2,

3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,

16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28,

29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41,

42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54,

55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,

68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80,

81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93,

94, 95, 96, 97, 98, 99]) >> np.concatenate([smallArray, largeArray, [888, 999]]) array([666, 1, 2, 3, 4, 5, 6, 7, 8, 9, 666, 1, 2,

3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,

16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28,

29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41,

42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54,

55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,

68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80,

81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93,

94, 95, 96, 97, 98, 99, 888, 999])

numpy Ufuncs

The primary purpose of Ufunc is to be able to speed up repeated operations on values in a numpy array. It can work both between a scalar value & an array and between 2 arrays

>> 3 * np.arange(0, 10) array([ 0, 3, 6, 9, 12, 15, 18, 21, 24, 27]) >> np.arange(0, 10) + np.arange(20, 30) array([20, 22, 24, 26, 28, 30, 32, 34, 36, 38])

This is not just syntactically better & intuitive, Its also faster. Lets try that out below

>> %timeit 3 * smallArray 1.22 µs ± 61.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) >> %%timeit



>> for i in range(len(smallArray)):

3 * smallArray[i] 7.15 µs ± 182 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

As can be seen above, the ufunc is multitudes faster than the looped version of the code. This difference becomes more and more pronounced as the computational logic involved gets more complex

Aggregation

In this section, we will explore the various aggregation functions that numpy provides. numpy ships with a standard sum() function which returns the sum of all elements in the array. You may ask what the difference is, between the native Python function and the numpy function! After all, they are doing the same functionality; provide the sum of elements. Let’s check it out below

>> np.sum(smallArray) 711 >> sum(smallArray) 711 >> hugeArray = np.random.randint(100000, size=1000000)

>> %timeit np.sum(hugeArray)

>> %timeit sum(hugeArray) 526 µs ± 9.48 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

271 ms ± 11 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

From the above code block, it's pretty evident how fast numpy is, compared to the native functions. This applies to most, if not all, aggregation functions available in numpy

>> np.min(smallArray) 1 >> np.max(smallArray) 666 >> np.std(smallArray) 198.31512801599376 >> np.mean(smallArray) 71.1 np.median(smallArray) 5.5

One thing to keep in note while using the aggregation functions is that these are prone to NaN values while you are using them, i.e., if you have a NaN-value in your array, these aggregation functions will fail. In such cases, you can use their NaN-safe alternatives. Just to give you an example: nansum() is the NaN-safe alternative for the sum function

>> np.nanmean(np.array([1, 2, np.nan])) 1.5 >> np.mean(np.array([1, 2, np.nan])) nan

Broadcasting

We are now going to look at an operation that you might have used a lot without having a conceptual understanding. In broadcasting, you perform an operation on an entity that has a different shape/dimensions.

A simple example of this could be adding a scalar to a numpy array. You can basically think of the values being duplicated to match the dimension of the array followed by the operation that is to be performed.

Broadcasting is an operation that can be elaborated on, but that is not my goal here with this post

>> a = np.arange(1, 7)

>> a array([1, 2, 3, 4, 5, 6]) >> a + 1 array([2, 3, 4, 5, 6, 7])

Logical Operations

There will be times when you would like to perform logical checks on a piece of data. numpy provides all the normal logical operations that you would expect: greater than, less than & equal to checks. The logical functions return an array of results in Boolean indicating if they satisfied the condition or not

>> x = np.array([1, 2, 3, 4]) >> x > 2 array([False, False, True, True]) >> x == 2 array([False, True, False, False])

numpy also provides useful functions such as any() and all() which are used to perform the following checks: if there is any element that fulfilled the condition or if all the elements satisfy the condition, respectively. They provide a single Boolean value as output which indicates the result

>> np.any(x == 2) True >> np.all(x == 2) False

From the above 2 code blocks: the following can be understood:

any(x == 2): It checks if any of the elements in the array met the condition passed

all(x == 2): It checks if all the elements in the array met the condition passed

Say we want a count of values that satisfy this particular condition, you can use the sum() method as follows. It counts the number of True values in the array

>> np.sum(x == 2) 1

This can also be compounded to check for multiple conditions. The same can be done for the any and all functions

>> np.sum((x == 2) | (x == 3)) 2 >> np.any((x == 2) | (x == 3)) True

Masking

You might have seen the above arrays which provide values as a Boolean array and wondered how that’s helpful because you still need to provide further operations to make sense of the output. Here is where masking comes in; the True/False array can be passed into the array to provide only those values which meet the conditions

>> x[x == 2] array([2]) >> x[x > 2] array([3, 4])

Fancy Indexing

Fancy indexing is nothing but the ability to access multiple elements of the array at once

>> x array([1, 2, 3, 4]) >> x[[0, 2, 3]] array([1, 3, 4])

As you might have guessed, we can pass in an array as the list of indices too

>> indexList = [0, 2, 3]

>> x[indexList] array([1, 3, 4])

The true power of fancy indexing comes out when it is combined with the likes of slicing, indexing and broadcasting

Sorting

The sorting algorithm provided by numpy is very efficient. There are 2 ways you can sort a numpy array; in-place and by calling the numpy sort function that returns the sorted array

Let’s shuffle our array first so that we can sort it and play around with it

>> np.random.shuffle(x)

>> x array([1, 3, 4, 2])

Calling np.sort on the array will return a copy of the array that is sorted and does not sort the array in place as demonstrated below

>> np.sort(x) array([1, 2, 3, 4]) >> x array([1, 3, 4, 2])

If you call sort on the array, then the array will be sorted in-place and there’s no need to save it to another array. Both methods perform the same functionality and are used depending on the required use-case

>> x.sort()

>> x array([1, 2, 3, 4])

Concluding Notes

numPy has a lot more capabilities, especially with regards to higher-dimensional data. numPy can easily work with, and manipulate data of higher dimensions. I have not decided to go into such details as that would defeat the purpose of this post.

I have compiled a Jupyter Notebook with the entire contents of this blog here.

There are a lot of things I haven’t covered as its outside the scope of this blog, however, if you are interested in reading more about numpy, I would highly recommend the following resources: