Learning the Pandas Library from Scratch
By: Charanraj Shetty
Tags: Pandas Dataframe, Pandas Series, Data Science, Programming, Pandas

Learning the Pandas Library from Scratch

Basic Guide Tutorial

Photo by Cassandra Norman on Unsplash

Pandas Introduction

Pandas is the library set or modules of Python language extensively used for Exploratory Data Analysis. It consists of various functions or methods which are useful to reduce the effort and time while performing analysis on any given dataset. Pandas can be extremely helpful when we want to play around with the given feature values.

Where can we apply Pandas?

Fetching the dataset. Cleaning, reshaping, and getting information about the dataset. Indexing and selection of data. Adding, Removal, and Updation of the data. Sorting of data. Analysis of time-series data.

Dataframe

A Dataframe is a table created from the dataset given in an excel file. Dataframe consists of rows and columns. Rows consist of the total number of data available to perform the Analysis, and Columns represent the total features available based on which the predictions are made related to the given task.

Series

Series in pandas is a one-dimensional array that can hold data of any type(string, int, float, python objects, etc.). Pandas Series can be equivalent to a single column of an excel sheet that represents a particular feature. In the above figure, each column of the Dataframe represents the Series.

Here we are using the dataset of Survey details from a customer consisting of user details and feedbacks.

1. Playing with the Dataframe

Read the excel

We can pull the dataset from the excel sheet using the read_csv method of pandas. We can apply the below function read_csv to extract the data and convert it into excel. Here we take an example of weather data for a given city. When we create a Dataframe, it doesn't display all the features. To display all the rows and columns, we need to use the set_option function from the pandas' library.

Image by Author

When there are 1000s of records present in a Dataframe, then we can filter out the first n number of records or last n number of records. For this purpose, e can use the head() and tail() method of the Dataframe.

Image by Author

Getting the information on data[ info(),shape(),describe(),pandas inbuilt math functions ]

info()

We can get information about column names, data types of columns, non-null objects using the info() function.

shape()

We can find the total rows and columns of a given Dataframe using shape().

Math functions

We can find the maximum, minimum, mean, and standard deviation values of each feature represented as columns.

describe()

Also, we can use the describe() to calculate the 25%, 50%, 75%, max, min, std for the features.

df.describe()

Image by Author

2. Ways to create Dataframe

We can pull the data from excel, CSV file, and create a Dataframe. Apart from this, we can create Dataframe using the Dictionaries or tuples list.

Excel to Dataframe

read_excel() function is used to fetch data from an excel sheet and represent it as a Dataframe. We specify the workbook and sheet name as parameters to the read_excel() function.

Image by Author

CSV to Dataframe

read_csv() function is used to fetch data from the CSV file and convert it to a Dataframe. We specify the CSV file name and the indexing if required as parameters in the read_csv() function.

Image by Author

Dictionary to Dataframe

We can create a dictionary of key-value pairs with key as columns and value as record values for that particular column. The dictionary is then passed as a parameter to the dataframe() function.

Image by Author

Tuple list to Dataframe

We can pass the values in a tuple and create a list of such tuples containing values of records. The list, along with the columns to be represented, is passed as parameters to the Dataframe() function.

Image by Author

3. Locating records based on requirements

loc() function

It is used to get the records based on the index value passed to the function. E.g., in the figure shown above, we can notice that the index is given by the integer starting from 0. For getting the value of any particular record, we utilize the index id and pass it to the loc() function.

Image by Author

We can also use the loc() function to obtain the value of any particular feature for a given record. Also, multiple records’ data can be pulled up from the Dataframe.

Image by Author

Also, we can specify the columns to be displayed in multiple records.

Image by Author

set_index()

Apart from this, we can even reset the index as per our preference. This helps us to separate the data for our analysis purpose. E.g., in the below image, we have replaced the index with the time(i.e., EST in our case below)

Image by Author

So while trying to filter out the records, we will use the location index parameter as a date instead of an integer value.

Image by Author

4. Filtering Data

Suppose we want to note down the wind speed and Sealevel Pressure for the records having a temperature more than 45, then we apply the concept of filtering in a Dataframe, which we used in fig1.

To figure out the same, we will apply the filtering concept. Initially, we will filter out the records having more than 45 as temperature and store it in a variable that acts as a Dataseries as it consists of data w.r.t only one column.

Image by Author

Now, if we pass the variable to the loc() function, then the index w.r.t to those records having temp>45 will be passed as parameter1. The column values we want to analyze for temperature more than 45 can be passed as the second argument.

Image by Author

5. Updation of Data

Image by Author

Updating column header names

We can update the column name as per our better understanding of data. There are various ways of updating the column names in Pandas. The column name changing is applied to the dataset given in fig. The below pic contains 3 methods to update the column names as per user requirement.

Image by Author

Updating of record data

We can update the record data w.r.t column values in a given Dataframe. There are various methods of updating the record value. We can apply the loc() function to apply the changes to a specific index. We can also use the filtering condition to update the rows based on the conditioning applied, which can be seen as the 4th line of code below Image.

Image by Author

replace() to update the column values

We can apply the replace() function to change the values of a particular column. Consider the case of column name as first, and we want to update the first name of record 1 and record 2. Below is an example of using the replace() function.

Image by Author

6. Adding or Removal of Data

Adding a new column

We can update a new column in the existing Dataframe by merging the details of already present columns. E.g., as per the fig, we have first and last name columns and using it, we want to create the full name column.

Image by Author

append()

We can use the append() function to input a new record in a given a Dataframe, or it can be used to merge two or more Dataframes. In the below images, we first append a record into the existing Dataframe using append(). After that, we create a new Dataframe and append it to the old Dataframe.

Image by Author

Image by Author

drop()

We can use the drop() method to remove a column from the Dataframe or eliminate an existing record in a Dataframe. In the below image, we drop the last and first name columns as we have merged them into the Full name column. In the second image, we drop an existing record using indexing and filtering.

Image by Author

Image by Author

7. Sorting the Data

Sort single column

We can arrange the record data either in ascending or descending order as per our convenience. It is also possible to sort data in multiple columns as per need. E.g., in the below-given image, we sort the data in ascending order and then later set the data in descending order. The parameter value ascending is by default set to True always, and if we want to arrange the data descending order, then we have to set it to False.

Image by Author

Sort multiple columns

We can also sort by Column A first and then column b next. For, E.g., we can sort the Dataframe first by using the first name and then sort the last name column as the second one. In the below image, we sort by the last name in descending order(by setting the ascending value to False) and then apply the ascending value to the first name column, i.e., by arranging the values of the first name in ascending order if they share the same common last name.

Image by Author

8. Handling the Missing Data

Image by Author

dropna()

While we have the dataset, we spend most of the time cleaning it and scaling it. One of the processes involved in cleaning data is the removal of null or incomplete values wherever it is necessary. We can propose to drop these incomplete values to reduce complexity or outliers in the Analysis. We can use the dropna() method to clear out the records containing null values. In the above image, we applied the dropna() function to the dataset and got the below result.

Image by Author

We can also drop any particular record as well if there is any particular field value of a record that is having a missing value by inputting the ‘any’ value into the parameter ‘how’. Or else we can drop a record if all the fields are missing in the given record by inputting the ‘all’ value into the parameter ‘how’.

Image by Author

fillna()

To understand this function, we take another dataset, as shown below image.

Image by Author

We can update the NaN values with proper integers or any other random string, which can help us in figuring out the analysis part of a given dataset in a better way.

Image by Author

We can also fill the above or below values in the column for a missing value. By using the ‘ffill’ value in the method parameter of the fillna function, we use the value of the above record in the column.

Image by Author

Also, we can use the ‘bfill’ value in the method parameter of the fillna function to update the missing values as per the below record value in the particular column.

Image by Author

9. Time Series Analysis of Data using Pandas

Image by Author

Pandas can be really helpful in observing the change of an entity over a given period of time. We have a very useful module to_datetime, which can help us in converting the given Date into a timestamp and then apply the various methods on this timestamp to conclude the given data. We are given here the share price of apple company from the year 2017 to 2020. We have various columns or features such as the closing value, high value, low value at each instance for all the given days from 2017–20.

to_datetime() and day_name()

Consider a case that we want to figure out the day for a given records’ date from the Dataframe. As the values in the Date column are string, we need to convert them into a date. So we apply the to_datetime() module to convert into Date. Once done, we utilize the day_name() method to get the day for the given date.

Image by Author

We can even add the ‘DayofWeek’ column into the Dataframe representing the day of the week.

Image by Author

min(), max() and total no of days

We can use the min() to determine the initial date in the given Dataframe. The max() method can be used to determine the last extracted date of the share price. The difference between both the dates will result in the total number of days being involved in the given Dataframe.

Image by Author

Filtering on the time series Data

We can even apply the filtering on the column if we want to obtain the price of share from 2019 to 2020.

Image by Author

Reset the index of the Dataframe

We can even set the index of the Dataframe as Date and then apply the filtering using the index directly to pull out the records.

Image by Author

We can even apply the describe functions on the particular column for a given date.

Image by Author

Resampling Data

We have the data of share price at every minute in a given dataset. If any curious user wants to determine the highest price of the share in an entire week. How will he find this out as we have data of every minute and every day? Suppose we want to find out the average price, highest price, closing price, and total volume of a share over the entire week. How will we get these data? Will we calculate manually every day and then take the day for the entire week and then apply the combined mathematics on the same? For this purpose Sampling( resample() ) was introduced.

Image by Author

Image by Author

This was all about the basics of Pandas. One can still go through the library set online and explore various functions to reduce their effort.

Thank you for reading the article!!

Please feel free to connect me on LinkedIn and Github.