3 Examples For Using Python Pandas More Efficiently | by Soner Yıldırım | Towards Data Science

Pandas is one of the most widely-used Python libraries in the data science ecosystem. It has numerous functions and methods to perform efficient and quick data analysis and manipulation.
In this article, we will go over 3 examples that show how to do some specific operations with Pandas.
The first and foremost requirement for practicing is data. I prepared a sales dataset that you can download from this repository on my GitHub page. Feel free to use it and follow along.
Let’s start with importing Pandas and reading the dataset into a data frame.
import pandas as pdsales = pd.read_csv("sales.csv")sales.head()
One of the most frequently used Pandas functions is the groupby. It basically allows for calculating aggregate values based on different categories in a column.
For instance, we can calculate the average product price for each product group as follows:
sales.groupby("product_group", as_index=False)["price"].mean()
There are several aggregate functions that we can use with the groupby such as min, max, mean, count, and so on. What I want to show here is that we can extend the capabilities of the groupby with a lambda function.
The data frame contains price and cost for each product. Consider we want to find the average profit of products in each product group. One method is to create a column that contain the profit information and then apply the mean function on it. If you do not prefer creating a new column, this operation can be done by applying a lambda function as follows.
sales.groupby("product_group").apply(   lambda x: (x["price"] - x["cost"]).mean())product_groupPG1    98.071795PG2    42.973467PG3    26.299231PG4    13.141605PG5    18.657765PG6    15.200864dtype: float64
Duplicate rows in a data frame might cause serious issues. The analysis we do will not be correct so we need to make sure there are no duplicate rows in a data frame.
How we define duplicate rows depends on the dataset. In our case, two rows that have the same product code are considered to be duplicate. In some cases, two rows need to have the same values in all columns in order to be duplicates.
The duplicated and drop_duplicates functions come in quite handy when dealing with duplicate rows.
We can find the number of duplicate entries in the product code column as follows.
sales.duplicated(subset='product_code').sum()47
The duplicated function returns true if a product code is duplicated. By applying the sum function, we get the total number of duplicated product codes.
If we consider two rows being duplicate if they have the same product code and product group, then we pass these columns to the subset parameter as a list.
The drop_duplicated function drops the duplicated rows from the data frame.
sales_new = sales.drop_duplicates(subset="product_code", keep="first")sales_new.duplicated(subset="product_code").sum()0
The keep parameter is used for determining which one of the duplicated rows to be kept in the data frame.
For both duplicated and drop_duplicates functions, if we want to evaluate rows based on all the columns, we should not use the subset parameter.
We do not always have numerical data. It comes in different data types such as integer, float, string, object, datetime, and so on. In order to perform efficient and accurate operations, it is important to use appropriate data types.
Some functions and methods are only available for certain data types. Thus, before performing any analysis, we should convert the data types of columns that are not stored correctly.
Consider we have the following data frame.
df.dtypesA      int64B    float64C    float64D     objectE     objectdtype: object
Column C seems to have integers but the data type is float because of the NaN value. Similarly, column E is stored with object data type but it should be boolean.
We can use the astype function to change the data types of each column separately. However, there is a more efficient option which is the convert_dtypes function. It converts columns to the best possible data types.
df_new = df.convert_dtypes()df_new.dtypesA      Int64B    Float64C      Int64D     stringE    booleandtype: object
The best way to learn Pandas, or any other software tool, is through practicing. The more you get your hands dirty, the more you will make use of Pandas.
When you start working with real life datasets that need more cleaning and preprocessing, you will appreciate the functionality and efficiency of Pandas even better.
Thank you for reading. Please let me know if you have any feedback.