Top Python Libraries: Numpy & Pandas | by Md Arman Hossen | Towards Data Science

In this tutorial, I’ll try to make a brief description about two of the most important libraries in Python Numpy and Pandas. Without further delay lets go through Numpy first.
numpy is the core library for scientific computing in Python. It provides a high-performance multidimensional array object and tools for working with these arrays. Numpy is a powerful N-dimensional array object which is Linear algebra for Python. Numpy arrays essentially come in two flavors: Vectors and Matrics. Vectors are strictly 1-d array whereas Matrices are 2-d but matrices can have only one row/column.
To install numpy library in your system and to know further about python basics you can follow the below link:
towardsdatascience.com
Now to use numpy in the program we need to import the module. Generally, numpy package is defined as np of abbreviation for convenience. But you can import it using anything you want.
import numpy as npnp.array([1, 2, 3])         # Create a rank 1 arraynp.arange(15)               # generate an 1-d array from 0 to 14np.arange(15).reshape(3, 5) # generate array and change dimensions
Now to learn more about numpy pakages and its function you can follow directly https://numpy.org/ , the official website. Here we’ll discuss some important commands and functions of numpy library.
In the example figure above we can observe numpy is imported first and then a 1-d numpy array a is defined. Then we can examine the type, dimension, shape, and length of the array using mentioned commands. Here are some important commands for creating arrays:
np.linespace(0,3,4)  #create 4 equally spaced points 0-3 range inclusivelynp.linespace(0,3,4, endpoint=False) # remove endpoint and other equally spaced values.np.random.randint(1,100,6)  #create array of 6 random values in 0-100 range.np.random.randint(1,100,6).reshape(3,2) #reshape the array according to row and column vectors.np.random.rand(4)    #create an array of uniform distribution (0,1)np.eye(3)            #create a 3*3 identity matrixnp.zeros(3)          #create array([0,0,0])np.zeros((5,5))      #create a 5*5 2-d array of zerosnp.random.randn(2,2) #return standard normal distribution vcenter around zreo.np.empty((2,3))      # uninitializednp.arange(0, 2, 0.3) # from 0 to a number less than 2 with 0.3 intervalsnp.ones((2,3,4), dtype=np.int16)       # all element are 1np.array([[1,2],[3,4]], dtype=complex) # complex arraynp.array([(1.5,2,3),(4,5,6)])          # two-dimensional arraynp.array([2,3,4])                      # one-dimensional array
Important attributes of the ndarray object
ndarray.shape : The dimensions of the array. This is a tuple of integers indicating the size of the array in each dimension. For a matrix with n rows and m columns, shape will be(n,m) .
ndarray.ndim : The number of axes (dimensions) of the array.
ndarray.dtype : If you want to know the data type of an array, you can query the attributes of dtype. An object describing the type of the elements in the array. One can create or specify dtype using standard Python types.Additionally, numpy provides types of its own. numpy.int32, numpy.int16, and numpy.float64 are some examples.
ndarray.itemsize : The size in bytes of each element of the array. For example, an array of elements of type float64 has itemsize 8 (=64/8), while one of type complex32 has itemsize 4 (=32/8). It is equivalent to ndarray.dtype.itemsize.
ndarray.size: The total number of elements of the array. This is equal to the product of the elements of shape.
Printing an array: When you print an array, numpy displays it in a similar way to nested lists, but with the following layout: the last axis is printed from left to right, the second-to-last is printed from top to bottom, the rest are also printed from top to bottom, with each slice separated from the next by an empty line.
Basic Operations of ndarray:
A = np.array([[1,1],[0,1]])B = np.array([[2,0],[3,4]])A+B              #addition of two arraynp.add(A,B)      #addition of two arrayA * B            # elementwise productA @ B            # matrix productA.dot(B)         # another matrix productB.T              #Transpose of B arrayA.flatten()      #form 1-d arrayB < 3            #Boolean of Matrix B. True for elements less than 3A.sum()          # sum of all elements of AA.sum(axis=0)    # sum of each columnA.sum(axis=1)    # sum of each rowA.cumsum(axis=1) # cumulative sum along each rowA.min()          # min value of all elementsA.max()          # max value of all elementsnp.exp(B)        # exponentialnp.sqrt(B)       # squre rootA.argmin()       #position of min value of elements A.argmax()       #position of max value of elementsA[1,1]           #member of a array in (1,1) position
Indexing, Slicing and Iterating in numpy:
a = np.arange(4)**3  # create array aa[2]                 # member of a array in 2nd positiona[::-1]              # reversed aa[0:4,1]             # each row in the second column of ba[1,...]             # same as a[1,:,:] or a[1]a[a>5]               # a with values greater than 5x = a[0:4]           # assign x with 4 values of ax[:]=99              # change the values of x to 99 which will change the 4 values of a also.
If any position of an array is allocated to another array and its being broadcasted to a new value, then the original array also changed. This is because numpy doesn’t want to use more memory for the same array. As shown in the following figure, the value of a is changed with changing the values of x which is a part of the array a.
When we use a comparison operator in an array it returns a boolean array. Then using the boolean array we could select elements conditionally from the original array.
pandas is an open-source library built on top of numpy providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. It allows for fast analysis and data cleaning and preparation. It excels in performance and productivity. It can work with data from a wide variety of sources. pandasis suited for many different kinds of data: tabular data, time-series data, arbitrary matrix data with row and column labels, and Any other form of observational/statistical data sets. To install pandas in your system you can use this command pip install pandas or conda install pandas .
import numpy as np          #importing numpyimport pandas as pd         #importing pandasarr=np.array([1,3,5,7,9])   #create arr arrays2=pd.Series(arr)           #create pandas series s2print(s2)                   #print s2print(type(s2))             #print type of s2
Output:
0    11    32    53    74    9dtype: int64<class 'pandas.core.series.Series'>
To make series in pandaswe need to use pd.Series(data, index)format where data are input data and index are selected index for data. To understand it fully we can follow the below example.
Pandas series works the same way both in list and numpy array as well as dictionary also. Panel, a three-dimensional data structure, has three axes,Axis 0 (items), Axis 1 (major_axis), and Axis 2 (minor_axis). Axis 0 corresponds to a two-dimensional DataFrame. Corresponds to the row of, and Axis 2 corresponds to the column of the DataFrame. The example below uses numpy to generate a three-dimensional random number and then applies it topandas.Panel(). The output shows that a Panel object of size 2 (items) x 3 (major_axis) x 4 (minor_axis) was created. If you look up p[0]from Panel object p, you can see that DataFrame, the first element of Axis 0, is displayed.
Pandas DataFrames create a tabular data structure with labeled axes(rows and columns). The default format of a DataFrame would be pd.Dataframe(data, index, column) . You need to mention the data, index and columns value to generate a DataFrame. Data should be at least two-dimensional, index will be the row name and columns values for the columns.
Below I’ve mentioned some basic commands used in pandas libraries along with their usage:
s4=pd.DataFrame(np.random.randn(20,7), columns=['A','B','C','D','E','F','G'])s4[(5<s4.index) & (s4.index<10)]  # s4 with values that satisfy both conditions s4.head()                 # First five rows of s4s4.tail()                 # Last five rows of s4s4.describe()             # statistical information of datas4['B']                   # data of the 'B' columns4[['B','E']]             # data of the 'B' and 'E' columns4[0:3]                   # data of the 1~3 rowss4.iloc[0:3]              # data of the 1~3 rowss4.loc[[2,3],['A','B']]   # value of row 2,3 column 'A' ,'B's4[2 < s4]                # s4 with values matching conditionss4.mean()                 # means of each columns4.mean(1)                # mean of each rows4.drop('A')              # delete row 'A's4.drop('D',axis=1)       # delete 'D' columns4.drop('D',axis=1, inplace=True) # delete 'D' column permanentlys4['H']=np.random.rand(20,1)      # add a new column of same length.
Dictionary can be used in creating both pandas Series and DataFrames. Dictionary can be used as data to create tabular data but values should be more than one for each key and all values should be of same length whereas in pandas Series different value length is okay.
To reset the index of the frame and add the previous index in a column we need to follow the below command. Reset index will be numerical value.
df.reset_index(inplace=True)df.set_index('Name') #index will be 'Name' column but not permanent. df.set_index('Name', inplace=True)#permanent new index 'Name' column
Most of the DataFrames has several Nan values in different columns. Sometimes we need to remove the Nan items or replace them with something else. We can remove or replace the Nan value in pandas DataFrame in following ways:
df.dropna()                  # remove the rows that have Nan valuedf.dropna(inplace=True)      # remove the Nan value rows parmenentlydf.dropna(axis=1)            # remove columns that has Nan valuedf.dropna(axis=1, inplace=True) # remove the Nan valued columns parmanently. df.fillna(value='Arman')     # fill Nan values with 'Arman'.df.fillna(value='Arman', inplace=True) #fill values parmanently.df.fillna(value=df.mean(), inplace=True) #fill Nan value with each column mean value.df['A'].fillna(value=df['A'].mean()) #fill Nan value of column 'A' with its mean.
In my next tutorial, I’ll try to summarize about Matplotlib and seaborn, two mostly used visualization libraries. I’ll try to add a repo for this tutorial ASAP. Thanks for your time and any kind of suggestions or criticism is highly appreciable. You can follow my profile to have a look at several tutorials.
medium.com
Numpy: https://numpy.org/Stanford cs231 : http://cs231n.github.io/python-numpy-tutorial/Numpy basic cheetsheet: https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Numpy_Python_Cheat_Sheet.pdfPandas : https://pandas.pydata.org/Pandas cheetsheet: https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf https://assets.datacamp.com/blog_assets/PandasPythonForDataScience.pdf
Numpy: https://numpy.org/
Stanford cs231 : http://cs231n.github.io/python-numpy-tutorial/
Numpy basic cheetsheet: https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Numpy_Python_Cheat_Sheet.pdf
Pandas : https://pandas.pydata.org/
Pandas cheetsheet: https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf https://assets.datacamp.com/blog_assets/PandasPythonForDataScience.pdf