Get Started with NumPy now!
By: Nazmul Ahsan
Tags: Python, Numpy, Data Science

Photo by Jan Laugesen on Unsplash

NumPy is a very powerful tool for numerical calculation in large data set. It’s a python library which has many built-in methods for performing fast vectorized array operations. NumPy can be used to perform complex computations on entire array without the need for Python for loops and it is significantly more efficient in memory use than their pure python counterparts. In this post I will talk about some basic ways of creating numpy arrays and then some common operations and how we can use them for efficient performance. You can find all the codes in my github repository here where I have almost the same blog in a jupyter notebook. You can download it, run the codes, experiment with them to get a better understanding.

Importing NumPy

If you have numpy installed in your computer, then you can just type the following line of code bellow to import the library in your code file. Here we gave it a nickname ‘np’ so that we can refer to it by this short name instead of typing the full long name each time we use it.

import numpy as np

I install python using Anaconda distribution which includes the numpy library. If you don’t have it installed, just install it by running the following command in the command prompt.

pip install numpy

Photo by Anders Jildén on Unsplash

How fast is it?

Before jumping into details, let’s first convince ourselves that numpy is actually a very effective library for numerical computation in large data by a little test. We will be performing the same task in two way, one by using numpy array and the other by the pure python counter part with help of loop. We will be comparing the computation time that the both operation will take to complete.

I can use a jupyter notebook’s magic function %time to measure the time each operation takes, which you will find in my notebook in github. But in other IDEs, magic functions won’t work. So, here is an alternative way to do it.

import time

Now we will create an array and a python equivalent list.

array = np.arange(10000000)

list_1 = [x for x in range(10000000)]

Now we will do a simple operation, squaring all the elements, from 1 to 10000000, first in a vectorized way and then using loop. We will measure the time for both operation.

tick = time.time()

array = array ** 2

tock = time.time()

print(tock - tick)

Output:

0.04497408866882324

In my case, it took 0.0449s (it will vary according to hardware). Now lets check the loop operation.

tick = time.time()

list_2 = [x ** 2 for x in list_1]

tock = time.time()

print(tock - tick)

Output:

6.054413557052612

In my case, it took 6.0544s! Almost 130 times slow! So it’s clear how fast vectorized operations are compared to loop operations. Now that you are convinced that numpy is actually very efficient, lets get ourselves familiar with it.

Creating NumPy arrays

NumPy arrays can be one dimensional or multidimensional. Let’s first see how we can create a numpy array from a python list. We can easily do that with numpy’s built in ‘array()’ function.

# Creating one dimensional NP array

list_1 = [1,2,3]

array_1 = np.array(list_1)

print(array_1) # Creating multi dimensional NP array

list_2 = [[1,2,3],[4,5,6],[7,8,9],[10,11,12]]

array_2 = np.array(list_2)

print(array_2)

output: array([1, 2, 3]) array([[ 1, 2, 3],

[ 4, 5, 6],

[ 7, 8, 9],

[10, 11, 12]])

This array() function accepts any sequence-like object (including other arrays). In case of creating multi demensional array from list which contains other lists, the length of those lists needs to be equal to each other. Otherwise only one dimensional array will be created with data type as “object”. Try that yourself for better understanding.

If we want to create an array by autogenarating data in a sequence, we can do that by NP’s “arange()” function.

# arange([start,optional] stop[, step, optional], dtype=None) array_3 = np.arange(5) # One dimensional

print("array 3:

{0}".format(array_3)) # Printing the array array_4 = np.arange(2,5) # One dimensional with specified start

print("

array 4:

{0}".format(array_4)) # Printing the array array_5 = np.arange(2,10,2) # One dimensional with specified start and steps, starts at 2, ends at 10, steps 2

print("

array 5:

{0}".format(array_5)) # Printing the array

output: array 3:

[0 1 2 3 4] array 4:

[2 3 4] array 5:

[2 4 6 8]

There are two built-in functions in numpy for creating arrays with only 1s or 0s. They might come useful in some situation. So it’s good to have a built-in function for it.

some_array_6 = np.zeros((2,3)) # Dimension is set to be 2x3

some_array_7 = np.ones((2,3))

By default, the values will be float values. If you need integer values, you can also specify that.

array_6 = np.zeros((2,3),’int’)

array_7 = np.ones((2,3),'int')

We also have a function for creating identity matrix (matrix having only 1s diagonally and 0s in other positions).

identity = np.identity(3)

identity

output:

array([[1., 0., 0.],

[0., 1., 0.],

[0., 0., 1.]])

We can create array with randomly generated data using numpy’s “random.randn()” (for getting float numbers) or “random.randint()” (for getting integer numbers) functions.

# Float data type, 2x3 dimension

array_8 = np.random.randn(2,3) # Int data type, 3x3

array_9 = np.random.randint(low = 1, high = 10, size = (3,3))

Keep in mind, the datatype of an array is not only limited to float or integer. They can be complex number, boolean object, strings and other python objects. We can easily convert the data types if they are convertible. See the following example.

# Creating an array with strings as data type

string_array = np.array(['1','2','2.5','3.5']) # Now converting its data type to float numbers

numeric_array = string_array.astype('float')

Attributes

Now that we know how to create NP arrays, lets take a look at how we can find the different attributes of an array. “Shape” is the size of the dimensions i.e. number of rows, number of colums. Dimension is the number of axis the array has. And “Size” indicates the number of elements the array contains. To get the shape of an array, we can use the “shape” attribute. For finding the dimension, we can use “ndim”. For finding the data types, we use “dtype” attribute. And for finding the total number of elements in the array i.e. its size, we use “size” attribute.

print(array_2.shape)

print(array_2.ndim)

print(array_2.dtype)

print(array_2.size)

Output:

(4, 3)

2

int32

12

You can reshape an array. For that, you can use the built-in resize funtion.

array_2.resize(2,6)

array_2.shape

Output:

(2, 6)

Indexing and Slicing

1D array

Indexing and slicing of one dimensional arrays are simple. It’s exactly like how you would index elements in a python List.

# Creating a new array first for demonstration

base_array = np.arange(10)

base_array

Output:

array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

base_array[5]

# Indexing in python starts at 0, so index 5 is the 6th element

Output:

5

Now, if you need to select a section of an array, you can use slicing just like a python list.

sliced_array = base_array[2:6] # The upper bound is excluded

sliced_array

Output:

array([2, 3, 4, 5])

This sliced array is just a view, not a new copy. So if you change anything, it will be reflected on the main array.

sliced_array[0] = 20

sliced_array[1] = 30

sliced_array

Output:

array([20, 30, 4, 5])

base_array

Output:

array([ 0, 1, 20, 30, 4, 5, 6, 7, 8, 9])

To create a copy, so that changes in the sliced array doesn’t change the main array, use the copy() function.

base_array = np.arange(5)

copied_array = base_array.copy()

Multi-dimensional array

Indexing of multidimensional array has options. You can get an individual element by a recursive call, like you would do in a list of lists, or you can just pass a comma separated list of indices.

# Creating a 2D array first

base_array = np.array([[1,2,3],[4,5,6],[7,8,9]])

base_array

Output:

array([[1, 2, 3],

[4, 5, 6],

[7, 8, 9]])

# Getting the first row

base_array[0]

Output:

array([1, 2, 3])

# Getting the element of the first column of the first row

base_array[0][0]

Output:

1

# or we can pass the indices separated by comma

base_array[0,0]

Output:

1

You might have guessed already how to slice a multi-dimensional array.

base_array[:2] # selecting every row upto the 2nd row

Output:

array([[1, 2, 3],

[4, 5, 6]])

base_array[1:] #selecting every row after the first one

Output:

array([[4, 5, 6],

[7, 8, 9]])

base_array[:,:2]

#selecting every column upto the 2nd one, had to put an empty ':' to indicate 'take all rows'

Output:

array([[1, 2],

[4, 5],

[7, 8]])

You will find a few more example in the jupyter notebook I provided in my github repo.

Boolean Indexing

There is also another very interesting type of indexing called boolean indexing. We can pass an array of boolean values as the index, it will give us only the rows for which the boolean value is ‘True’. Let’s take a look at the following example.

# Creating a boolean array first

boolean_array = np.array([True, False, True]) # Using boolean indexing

base_array[boolean_array]

Output:

array([[1, 2, 3],

[7, 8, 9]])

Boolean indexing is very useful and can be used in a very creating way. Lets see that in a simple example.

Suppose we have four different things to mix (sugar, salt, spice, water or whatever). We tried 10 different combinations. Some proved to be a good mix, some proved to be bad mix. We stored the combinations in an array. Each column represents the ingredients and each row is the different mixes.

combinations = np.random.randint(low = 1, high = 10, size = (10,4)) # Randomly creating an array for demonstration combinations

#output will change every time the code is run, as they are random data

Output:

array([[6, 6, 8, 3],

[9, 8, 2, 7],

[3, 5, 4, 2],

[6, 7, 6, 2],

[9, 2, 6, 4],

[7, 7, 7, 8],

[7, 9, 3, 7],

[4, 7, 5, 6],

[8, 6, 6, 1],

[6, 8, 5, 3]])

results = np.array(['g','g','vg','g','b','b','b','vg','b','b'])

# Randomly created the result, assume g = good, vg = very good, b = bad

So we want to find mixes that are very good or just good (designated by ‘g’ in the result) combinations or anything but bad combinations. Here is how we can use boolean indexing to do this.

very_good = results == 'vg'

combinations[very_good]

Output:

array([[3, 5, 4, 2],

[4, 7, 5, 6]])

condition = results=='b'

combinations[~condition]

# using ~ before a condination will yield every other results except the given condition

Output:

array([[6, 6, 8, 3],

[9, 8, 2, 7],

[3, 5, 4, 2],

[6, 7, 6, 2],

[4, 7, 5, 6]])

In Boolean indexing, we have to make sure that the length of the Boolean index matches the length of the axis its indexing.

Operations

Simple arithmetic operations

Simple addition or subtraction of two arrays will perform an element-wise addition or subtraction. You can also multiply or divide each element by a scalar value in standard way.

print(base_array)

Output:

[[1 2 3]

[4 5 6]

[7 8 9]]

base_array + base_array

Output:

array([[ 2, 4, 6],

[ 8, 10, 12],

[14, 16, 18]])

base_array * 10

Output:

array([[10, 20, 30],

[40, 50, 60],

[70, 80, 90]])

base_array + 10

Output:

array([[11, 12, 13],

[14, 15, 16],

[17, 18, 19]])

Multiplication of Arrays

There are two ways of multiplying arrays. One is the simple element-wise multiplication between two arrays. Simply putting the two arrays with the multiplication sign, like in normal multiplication operation, will perform this.

base_array * base_array

Output:

array([[ 1, 4, 9],

[16, 25, 36],

[49, 64, 81]])

But if we want to perform matrix multiplication, we can use numpy’s “dot()” function. (For those who are not familiar with matrix multiplication, it is a little bit complex. Check this link to know how matrix multiplication is done mathematically.)

base_array.dot(base_array)

# np.dot(array, array), alternate syntax

Output:

array([[ 30, 36, 42],

[ 66, 81, 96],

[102, 126, 150]])

Metric Transpose

We can transpose an array (exchanging its rows and columns) by using “.T” after the array.

base_array.T

Output:

array([[1, 4, 7],

[2, 5, 8],

[3, 6, 9]])

For more complex multidimensional array, we can use the “transpose()” function and pass a tuple of the axes in the order we want as arguments. For example array.transpose((1,2,0)) will make the second axis first, the third axis second and the first axis last. (Remember python indexing, first axis is indexed 0, second is 1, third is 2.)

array = np.arange(16).reshape((2, 2, 4))

array

Output:

array([[[ 0, 1, 2, 3],

[ 4, 5, 6, 7]], [[ 8, 9, 10, 11],

[12, 13, 14, 15]]])

array.transpose((1, 2, 0))

Output:

array([[[ 0, 8],

[ 1, 9],

[ 2, 10],

[ 3, 11]], [[ 4, 12],

[ 5, 13],

[ 6, 14],

[ 7, 15]]])

Applying Functions

There are some universal functions which helps us perform fast element wise operation. For example, np.sqrt() performs square root on each element of the array, np.exp() function returns an array with the exponential value of each element.

np.sqrt(base_array)

Output:

array([[1. , 1.41421356, 1.73205081],

[2. , 2.23606798, 2.44948974],

[2.64575131, 2.82842712, 3. ]])

np.exp(base_array)

Output:

array([[2.71828183e+00, 7.38905610e+00, 2.00855369e+01],

[5.45981500e+01, 1.48413159e+02, 4.03428793e+02],

[1.09663316e+03, 2.98095799e+03, 8.10308393e+03]])

There are many functions like this in numpy that helps to perform different mathematical calculations like finding log, square, sine, cosine, tangent etc. in a fast way. In my notebook, I have mentioned a few more functions that you might often find useful.

There are some mathematical functions to evaluate statistics of an array. For example, np.mean() returns the mean value of the given array, max() gives the maximum value, std() gives standard deviation of the data.

array.mean()

# np.mean(array), alternate syntax

Output:

5.0

Conclusion

I hope this post will give you a basic idea about numpy and prepares you to get started. I would recommend you to start using numpy with the basics that I have provided in this notebook. At the initial stage, you might always first think of applying a for loop in you arrays whenever you will need to do something, at least that’s what I did. But remember, if you do so, you are missing the advantage of using a vectorized operation. But no worries. You will build up the habbit of using vectorized operations if you keep practicing. When you get stuck with a problem or when you need to perform an operation that is not listed in this notebook, just simply do a google search or search it in stackoverflow. There is a good chance that numpy has a vectorized way of executing your operation without the need for a for loop and you know, that will be a better alternative performance-wise.