Most Important Pandas Functions(Full Tutorial)| Towards Data Science | Towards Data Science

Pandas is one of the most popular python libraries used for data manipulation and analysis. It enables a variety of reading functions for a wide range of data formats, commands to best select the subset you want to analyze. It provides a variety of tools for data manipulation such as merging, joining and concatenation. In this post, we will focus on discovering pandas most important functions and commands which are most used to explore the dataset and unveil the underlying relationships among variables.
This post is an attempt to mark out the different functions and commands available by the library all in a single post .
Importing DataViewing & Selecting DataStatistics & Simple Data OperationsDealing with Missing valuesData Manipulation
Importing Data
Viewing & Selecting Data
Statistics & Simple Data Operations
Dealing with Missing values
Data Manipulation
I will use a customer churn dataset for the examples. The first step is to read the dataset into a pandas data frame.
Before starting going through functions and implementations, I would like to emphasize the importance of Understanding the Axis and the Inplace parameter.
A DataFrame object has two axes: “axis 0” and “axis 1”:
axis 0: Wherever you see this -> it represents rows
axis 1: Wherever you see this -> it represents columns
Understanding the “inplace” parameter can help us a lot of time and memory!
When inplace = False -> which is the default, then the operation is performed and it returns a copy of the object. You then need to save it to something.
temp=df.set_index(‘CustomerId’)# here by Default inplace = Falsetemp
While , When inplace = True -> the data is modified in place, which means it will return nothing and the dataframe is now updated.
df.set_index(‘CustomerId’,inplace=True)df
import pandas as pdimport numpy as nppd.set_option('display.max_columns', None)pd.set_option("display.precision", 2) df = pd.read_csv("Churn_Modelling.csv") # import from a CSV
Use these commands to take a look at specific sections of your pandas DataFrame or Series.
df.head(5) -> First 5 rows of the DataFrame
df.tail(5) -> Last 5 rows of the DataFrame
df.shape -> Return the number of rows and columns
df.info() ->Return Index, Datatype and Memory information
Now, Let‘s find out which functions and commands are best used to select a specific subset of your data.
df['Gender'] or df.Gender ->Returns column with label col as Series
df[['Gender', 'Age','Balance]] -> Returns columns as a new DataFrame
loc is label-based, which means that we have to specify the name of the rows and columns that we need to filter out.
df.loc[1:5] -> Select a range of rows using loc.
Note: if the indices are not numbers, then we cannot slice our data frame. In that case, we need to use the iloc function.
df.loc[df.Age>=45] -> Select only required columns with a condition
# Some more examples- df.loc[0]                            # forst row details - df.loc[:,"Gender"]                   # Selects all rows of the     Gender column as a series- df.loc[:,"Balance":"HasCrCard"]      # Selects all rows of the columns  going from Balance to HasCrCard
iloc is integer index-based , we have to specify rows and columns by their integer index.
df.iloc[[100,200]] -> Selecting Rows 100 & 200
Note that: Another thing to remember in iloc are the indexes:
for instance: df.iloc[1:5]
- 1 is inclusive
- 5 is exclusive, meaning it won’t print it the 5th row instead it will return the the previous row -> it will go from 1 to (5–1);check out the below example
df.iloc[15:20,3:7]-> Select a range of rows and columns using iloc
df.iloc[20][2]-> Accessing the [3rd ]element of the [20th] row
# Some more examples- df.iloc[0:3] # 0 is inclusive and 3 is exclusive   - df.iloc[15:20,'Gender':] -> the latter will give an error since we are passing a label rather than index
Use these commands to perform descriptive statistical analysis.
df.describe() -> Summary statistics for numerical columns
df.mean() -> Returns the mean of all columns
df.corr() -> Returns the correlation between columns in theDataFrame
df.count() -> Returns the number of non-null values in each DataFrame column
df['Geography'].unique() -> Return the list of unique values in the column
df['Geography'].nunique() ->Return how many unique values are present in such feature/column
df['Gender'].value_counts(dropna=False) | View unique values and counts
df.sample(n=10) -> Randomly select 10 rows
df.nlargest(5,'Balance') ->Select and order top n entries.
df.nsmallest(5,'EstimatedSalary')->Select and order bottom n entries
df.query("(Gender=='Male') and (Age<23)") ->Query the columns of a DataFrame with a boolean expression
Where function will replace values where your condition is False. It is useful when you have values that do not meet criteria, and they need replacing.
df['Age'].where(df['Age']>40,"too Young") -> if df[‘Age’]>40 is false it will substitute with ‘Less than fourty’
df['Age']=df['Age'].astype(float) -> Convert the datatype of the Age “series” to float64
df['HasCrCard'].replace(1,'yes') -> Replace all values equal to 1 with 'yes'
df.rename('columns={'Geography':'Nation'}) -> renaming the Geography column into Nation
df.reset_index() ->reset index of DataFrame to row numbers, moving index to columns.
Let‘s find out which functions and commands are best used to deal with missing values.
df.isnull()or df.isna() -> Checks for null Values, Returns Boolean Arrray
df.notnull() or df.notna()-> Opposite of df.isnull()
df.isnull().sum() -> returns the sum of null values of every single feature/column
The Dataset we are currently using as we see above has got no missing values.
Use these commands and functions to drop columns and raws with missing values, dropping duplicates and columns.
df.dropna()-> Drop all rows that contain null values
df.dropna(axis=1) -> Drop all columns that contain null values
df.dropna(axis=1,thresh=n) -> Drop all rows have have less than n non null values
df.drop_duplicates() -> Remove duplicate rows (only considers columns).
df.drop(['RowNumber', 'Surname'],axis=1, inplace=True).head(5) -> Return the dataframe after dropping the the two specified columns
df.fillna(x) -> Replace all null values with x
Use these commands to sort, filter and group your data.
df.sort_values('Balance') -> Sort values of the Balance column in ascending order
df.sort_values('EstimatedSalary',ascending=False) -> Sort values in descending order
pd.Dataframe(df.sort_values(['Balance','EstimatedSalary'],ascending=[True,False]))
-> Sort “Balance” column in ascending order & and Sort “EstimatedSalary” in descending order
df[(df['CreditScore] > 700) & (df['CreditScore] < 900)].sort_values(['CreditScore'],ascending=True) — > Filter out CreditScore raws where values are from 700 to 900 and present them in ascending order
Pandas Groupby function is a versatile and easy-to-use function that helps to get an overview of the data.
df[['Gender','NumOfProducts']].groupby('Gender).mean()
Data Aggregations refer to any data transformation that produces scalar values from arrays: (mean, count, min, and sum).
df[['Gender','EstimatedSalary','Balance]].groupby(['Geography]).agg(['mean','max'])
In order to group by multiple columns, we simply pass a list to our groupby function with the columns we want to groupby:
df[['Gender','EstimatedSalary','Balance]].groupby(['Gender','Geography']).agg(['mean','count'])
df.pivot_table(index='CreditScore',values=['Age','Balance']) ->
Create a pivot table that groups by CreditScore and calculates the mean of Age and Balance
pd.melt(df) -> Gather columns into rows
Discretize variable into equal-sized buckets based on rank or based on sample quantiles.
-Create a new Column[‘Salary_level”] using “pd.qcut” splitting EstimatedSalary into 5 different levels:[‘Bronze’, ‘Silver’, ‘Gold’, ‘Platinum’, ‘Diamond’]
bin_labels_5 = [‘Bronze’, ‘Silver’, ‘Gold’, ‘Platinum’, ‘Diamond’]df[‘Salary_level’] = pd.qcut(df[‘EstimatedSalary’],q=[0, .2, .4, .6, .8, 1], labels=bin_labels_5)
Use these commands to combine multiple data frames into a single one.
df1.append(df2) -> Add the rows in df1 to the end of df2 (columns should be identical)
pd.concat([df1, df2],axis=1) -> Add the columns in df1 to the end of df2 where rows are identical
df1.join(df2 ,on=col1, how='inner') -> SQL-style join the columns in df1 with the columns on df2 where the rows for col have identical values.
'how' can be : 'left', 'right', 'outer', 'inner'
You can do it with a simple code
tudf.head(5).style.set_properties(**{'background-color': 'lightgreen'}, subset=['Balance','EstimatedSalary'])
By adding the style function we can choose to set a colour to our columns and much more if you would like to know more about styling I would recommend the official pandas website: https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html
In this Beginner-friendly tutorial, I implemented some of the most important Pandas functions and command used for Data Analysis. I went over for every function different types of examples. I would recommend finding additional data sets and playing around with these functions and explore as much as you can, at the end of the day it is a matter of practice. Mastering Pandas is a great asset for whoever is on the path to becoming a Skillful Data Scientist/Data analyst/ Machine learning engineer ecc...you name it.
I hope this post gives you a nice foundation for whoever is trying to master it, and for whoever has well crafted this art may this post works as a re-freshening.
Thank you for reading!