Top Python Libraries: Numpy & Pandas
By: Md Arman Hossen
Tags: Numpy, Data, Programming, Pandas, Python

In this tutorial, I’ll try to make a brief description about two of the most important libraries in Python Numpy and Pandas. Without further delay lets go through Numpy first.

Numpy

numpy is the core library for scientific computing in Python. It provides a high-performance multidimensional array object and tools for working with these arrays. Numpy is a powerful N-dimensional array object which is Linear algebra for Python. Numpy arrays essentially come in two flavors: Vectors and Matrics. Vectors are strictly 1-d array whereas Matrices are 2-d but matrices can have only one row/column.

To install numpy library in your system and to know further about python basics you can follow the below link:

Now to use numpy in the program we need to import the module. Generally, numpy package is defined as np of abbreviation for convenience. But you can import it using anything you want.

import numpy as np

np.array([1, 2, 3]) # Create a rank 1 array

np.arange(15) # generate an 1-d array from 0 to 14

np.arange(15).reshape(3, 5) # generate array and change dimensions

Now to learn more about numpy pakages and its function you can follow directly https://numpy.org/ , the official website. Here we’ll discuss some important commands and functions of numpy library.

In the example figure above we can observe numpy is imported first and then a 1-d numpy array a is defined. Then we can examine the type, dimension, shape, and length of the array using mentioned commands. Here are some important commands for creating arrays:

np.linespace(0,3,4) #create 4 equally spaced points 0-3 range inclusively

np.linespace(0,3,4, endpoint=False) # remove endpoint and other equally spaced values.

np.random.randint(1,100,6) #create array of 6 random values in 0-100 range.

np.random.randint(1,100,6).reshape(3,2) #reshape the array according to row and column vectors.

np.random.rand(4) #create an array of uniform distribution (0,1)

np.eye(3) #create a 3*3 identity matrix

np.zeros(3) #create array([0,0,0])

np.zeros((5,5)) #create a 5*5 2-d array of zeros

np.random.randn(2,2) #return standard normal distribution vcenter around zreo.

np.empty((2,3)) # uninitialized

np.arange(0, 2, 0.3) # from 0 to a number less than 2 with 0.3 intervals

np.ones((2,3,4), dtype=np.int16) # all element are 1

np.array([[1,2],[3,4]], dtype=complex) # complex array

np.array([(1.5,2,3),(4,5,6)]) # two-dimensional array

np.array([2,3,4]) # one-dimensional array

Important attributes of the ndarray object

ndarray.shape : The dimensions of the array. This is a tuple of integers indicating the size of the array in each dimension. For a matrix with n rows and m columns, shape will be (n,m) .

ndarray.ndim : The number of axes (dimensions) of the array.

ndarray.dtype : If you want to know the data type of an array, you can query the attributes of dtype. An object describing the type of the elements in the array. One can create or specify dtype using standard Python types.

Additionally, numpy provides types of its own. numpy.int32, numpy.int16, and numpy.float64 are some examples.

Attributes of ndarray object

ndarray.itemsize : The size in bytes of each element of the array. For example, an array of elements of type float64 has itemsize 8 (=64/8), while one of type complex32 has itemsize 4 (=32/8). It is equivalent to ndarray.dtype.itemsize.

ndarray.size : The total number of elements of the array. This is equal to the product of the elements of shape.

Printing an array: When you print an array, numpy displays it in a similar way to nested lists, but with the following layout: the last axis is printed from left to right, the second-to-last is printed from top to bottom, the rest are also printed from top to bottom, with each slice separated from the next by an empty line.

Basic Operations of ndarray:

A = np.array([[1,1],[0,1]])

B = np.array([[2,0],[3,4]])

A+B #addition of two array

np.add(A,B) #addition of two array

A * B # elementwise product

A @ B # matrix product

A.dot(B) # another matrix product

B.T #Transpose of B array

A.flatten() #form 1-d array

B < 3 #Boolean of Matrix B. True for elements less than 3

A.sum() # sum of all elements of A

A.sum(axis=0) # sum of each column

A.sum(axis=1) # sum of each row

A.cumsum(axis=1) # cumulative sum along each row

A.min() # min value of all elements

A.max() # max value of all elements

np.exp(B) # exponential

np.sqrt(B) # squre root

A.argmin() #position of min value of elements

A.argmax() #position of max value of elements

A[1,1] #member of a array in (1,1) position

Indexing, Slicing and Iterating in numpy:

a = np.arange(4)**3 # create array a

a[2] # member of a array in 2nd position

a[::-1] # reversed a

a[0:4,1] # each row in the second column of b

a[1,...] # same as a[1,:,:] or a[1]

a[a>5] # a with values greater than 5

x = a[0:4] # assign x with 4 values of a

x[:]=99 # change the values of x to 99 which will change the 4 values of a also.

If any position of an array is allocated to another array and its being broadcasted to a new value, then the original array also changed. This is because numpy doesn’t want to use more memory for the same array. As shown in the following figure, the value of a is changed with changing the values of x which is a part of the array a.

When we use a comparison operator in an array it returns a boolean array. Then using the boolean array we could select elements conditionally from the original array.

Pandas

pandas is an open-source library built on top of numpy providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. It allows for fast analysis and data cleaning and preparation. It excels in performance and productivity. It can work with data from a wide variety of sources. pandas is suited for many different kinds of data: tabular data, time-series data, arbitrary matrix data with row and column labels, and Any other form of observational/statistical data sets. To install pandas in your system you can use this command pip install pandas or conda install pandas .

import numpy as np #importing numpy

import pandas as pd #importing pandas

arr=np.array([1,3,5,7,9]) #create arr array

s2=pd.Series(arr) #create pandas series s2

print(s2) #print s2

print(type(s2)) #print type of s2

Output:

0 1

1 3

2 5

3 7

4 9

dtype: int64

<class 'pandas.core.series.Series'>

To make series in pandas we need to use pd.Series(data, index) format where data are input data and index are selected index for data. To understand it fully we can follow the below example.

Pandas series works the same way both in list and numpy array as well as dictionary also. Panel, a three-dimensional data structure, has three axes,

Axis 0 (items), Axis 1 (major_axis), and Axis 2 (minor_axis). Axis 0 corresponds to a two-dimensional DataFrame. Corresponds to the row of, and Axis 2 corresponds to the column of the DataFrame. The example below uses numpy to generate a three-dimensional random number and then applies it to

pandas.Panel() . The output shows that a Panel object of size 2 (items) x 3 (major_axis) x 4 (minor_axis) was created. If you look up p[0] from Panel object p, you can see that DataFrame, the first element of Axis 0, is displayed.

Pandas DataFrames create a tabular data structure with labeled axes(rows and columns). The default format of a DataFrame would be pd.Dataframe(data, index, column) . You need to mention the data, index and columns value to generate a DataFrame. Data should be at least two-dimensional, index will be the row name and columns values for the columns.

Example of pandas Series & DataFrame

Below I’ve mentioned some basic commands used in pandas libraries along with their usage:

s4=pd.DataFrame(np.random.randn(20,7), columns=['A','B','C','D','E','F','G'])

s4[(5<s4.index) & (s4.index<10)] # s4 with values that satisfy both conditions

s4.head() # First five rows of s4

s4.tail() # Last five rows of s4

s4.describe() # statistical information of data

s4['B'] # data of the 'B' column

s4[['B','E']] # data of the 'B' and 'E' column

s4[0:3] # data of the 1~3 rows

s4.iloc[0:3] # data of the 1~3 rows

s4.loc[[2,3],['A','B']] # value of row 2,3 column 'A' ,'B'

s4[2 < s4] # s4 with values matching conditions

s4.mean() # means of each column

s4.mean(1) # mean of each row

s4.drop('A') # delete row 'A'

s4.drop('D',axis=1) # delete 'D' column

s4.drop('D',axis=1, inplace=True) # delete 'D' column permanently

s4['H']=np.random.rand(20,1) # add a new column of same length.

Dictionary can be used in creating both pandas Series and DataFrames. Dictionary can be used as data to create tabular data but values should be more than one for each key and all values should be of same length whereas in pandas Series different value length is okay.

Pandas Series and Dataframe through Dictionary

To reset the index of the frame and add the previous index in a column we need to follow the below command. Reset index will be numerical value.

df.reset_index(inplace=True)

df.set_index('Name') #index will be 'Name' column but not permanent.

df.set_index('Name', inplace=True)#permanent new index 'Name' column

Most of the DataFrames has several Nan values in different columns. Sometimes we need to remove the Nan items or replace them with something else. We can remove or replace the Nan value in pandas DataFrame in following ways:

df.dropna() # remove the rows that have Nan value

df.dropna(inplace=True) # remove the Nan value rows parmenently df.dropna(axis=1) # remove columns that has Nan value

df.dropna(axis=1, inplace=True) # remove the Nan valued columns parmanently.

df.fillna(value='Arman') # fill Nan values with 'Arman'.

df.fillna(value='Arman', inplace=True) #fill values parmanently.

df.fillna(value=df.mean(), inplace=True) #fill Nan value with each column mean value.

df['A'].fillna(value=df['A'].mean()) #fill Nan value of column 'A' with its mean.

In my next tutorial, I’ll try to summarize about Matplotlib and seaborn, two mostly used visualization libraries. I’ll try to add a repo for this tutorial ASAP. Thanks for your time and any kind of suggestions or criticism is highly appreciable. You can follow my profile to have a look at several tutorials.