7 Pandas Functions That I Use the Most | by Soner Yıldırım | Towards Data Science

Pandas is a prominent data analysis and manipulation library. It provides numerous functions and methods that expedite the data analysis and exploration process.
Within the rich selection of functions and methods, some of them are more commonly used. They provide a quick way to obtain a basic understanding of the data at hand.
In this post, I will go over 7 pandas functions that I most commonly use in my data analysis projects.
I will use a diabetes dataset available on Kaggle. Let’s first read the dataset into a pandas dataframe.
import pandas as pdimport numpy as npdiabetes = pd.read_csv("/content/diabetes.csv")
Once we read a dataset into a pandas dataframe, we want to take a look at it to get an overview. The simplest way is to display some rows. Head and tail allow us to display rows from the top of bottom of dataframe, respectively.
diabetes.head()
diabetes.tail()
5 rows are displayed by default but we can adjust it just by passing the number of rows we’d like to display. For instance, diabetes.head(10) will display the first 10 rows.
When working with categorical data or features that have discrete values, it is very important to know the number of unique values. It is an essential step towards data exploration.
One way is to use value_counts function which returns a pandas series with unique values in a column and the number of occurrences of each value. The length of this series is the number of unique values.
len(diabetes.Pregnancies.value_counts())17
I find nunique easier to use:
diabetes.Pregnancies.nunique()17
We can also apply it to the entire dataframe and see the number of unique values in each column which I think make nunique more functional:
diabetes.nunique()
Describe function gives a quick overview of numerical columns by providing basic statistics such as mean, median, and standard deviation.
diabetes.describe()
25%, 50%, and 75% are the percentiles. 50% is also known as the median which is the value in the middle when all values are sorted. 25% is the first quartile so 25% of the values are below this point. You can think of it as median of the lower half of a column. Similarly, 75% is the third quartile.
The basic statistics give us valuable insights. For instance, by comparing mean and median (50%), we have an understanding of the outliers. If mean is much higher than the median, there are outliers on the upper side. The basic statistics also give us an overview of the distribution of data.
You can also select different percentiles to display using percentiles parameter. The default values are [.25, .50, .75]. For instance percentiles=[.1, .25, .5, .75, .9] will display 10% and 90% percentiles in addition to default values.
Handling missing values is a critical step to build a robust data analysis process. The missing values should be a top priority since they have a significant effect on the accuracy of any analysis.
Isna function returns a dataframe filled with boolean values with true indicating missing values.
diabetes.describe()
Unless you want to analyze the dataframe cell by cell, isna should be combined with an aggregation. isna().any() indicates if there is any missing value in a column whereas isna().sum() returns the number of missing values in columns.
diabetes.isna().any()
diabetes.isna().sum()
The dataset we have does not have any missing values but this is a highly unlikely case in real-life.
Pandas groupby function is a great tool in exploring the data. It makes it easier to unveil the underlying relationships among variables. The figure below shows an overview of what groupby function does.
Assume we have two features. One is color which is a categorical feature and the other one is a numerical feature, values. We want to group values by color and calculate the mean (or any other aggregation) of values for different colors. Then finally sort the colors based on average values.
You can, of course, create more complex grouping operations but the concept will be the same.
Let’s create a simple grouping operation with our dataset. The following code will show us the average glucose values of diabetes positive and negative people.
diabetes[['Outcome','Glucose']].groupby('Outcome').mean()
As expected, glucose values are higher for diabetes positive (1).
We do not have many categorical variables in the dataset so what we can do with groupby function on this dataset is kind of limited. However, if you’d like to read more about groupby function and see more complex examples, I have a post dedicated to pandas groupby function.
towardsdatascience.com
We need to have the values stored in an appropriate data type. Otherwise, we may encounter errors. For large datasets, memory-usage is greatly affected by correct data type selection. For example, “categorical” data type is more appropriate than “object” data type for categorical data especially when the number of categories is much less than the number of rows.
Dtypes shows the data type of each column.
diabetes.dtypes
The data type of “BloodPressure” column is not appropriate. It should be int or float. Object data type can be to store strings or categorical data.
We can easily change the data type with astype function.
diabetes.BloodPressure = diabetes.BloodPressure.astype('int64')diabetes.dtypes
Shape can be used on numpy arrays, pandas series and dataframes. It shows the number of dimensions as well as the size in each dimension.
Since dataframes are two-dimensional, what shape returns is the number of rows and columns. It is a measure of how much data we have and a key input to the data analysis process.
Furthermore, the ratio of rows and columns is very important when designing and implementing a machine learning model. If we do not have enough observations (rows) with respect to features (columns), we may need to apply some pre-processing techniques such as dimensionality reduction or feature extraction.
diabetes.shape(768, 9)
Let’s apply it on a column which is actually a pandas series.
diabetes.Glucose.shape(768,)
Size, as the name suggests, returns the size of a dataframe which is the number of rows multiplied by the number of columns.
diabetes.size6912
Thank you for reading. Please let me know if you have any feedback.