Supermarket Data Analysis with Pandas | by Soner Yıldırım | Towards Data Science

Pandas is the most widely-used data analysis and manipulation library for Python. Its intuitive and versatile functions make the data analysis process efficient, simple, and easy to understand.
In this article, we will practice pandas on a supermarket sales dataset available on Kaggle. It contains sales data of different branches of a supermarket chain during a 3-month-period.
Let’s start by importing the libraries and reading the dataset.
import numpy as npimport pandas as pddf = pd.read_csv("/content/supermarket.xls", parse_dates=['Date'])
We use the parse_dates parameter to store the date column with datetime data type so that we do not have to convert the data type later on. The datetime data type allows for using the functions under the dt accessor that are specific to dates and times.
There are some columns which are either redundant and irrelevant to our analysis.
Invoice ID: A unique invoice identification number. Does not possess any information for analsis.
Cogs: Product of the unit price and quantity columns.
Gross margin percentage: Consists of a single value which is 4.761905.
Gross income: Can be obtained from multiplying the total column with 0.04761905 (i.e. gross margin percentage).
City: Perfectly correlated with the branch column. There is one branch in each city.
In order to simplify the dataset, we drop these columns.
to_drop = ['Invoice ID', 'City','cogs', 'gross margin percentage', 'gross income']df.drop(to_drop, axis=1, inplace=True)df.head()
We should always check if there are missing values in the dataset.
df.isna().sum().sum()0
The isna function returns the dataframe indicating missing values with “True”. By chaining a sum function, we get the number of missing values in each column. The second sum function gives the number of missing values in the entire dataframe.
We might be interested in a general overview of the sales at each branch. The total amount of sales and average sales amount per invoice can be calculated with the groupby function.
df[['Branch', 'Total']]\.groupby('Branch').agg(['mean','sum','count'])
The average amount per invoice is higher in the C branch. As a result, the total sales amount of this branch is higher than the other branches although the number of invoices is less.
We can also check the average unit price of products in each product line. Let’s also sort them in descending order to get a more structured overview.
df[['Product line', 'Unit price']].groupby('Product line').mean()\.sort_values(by='Unit price', ascending=False).round(2)
Fashion accessories lead the list but the average unit prices are quite close to each other.
We can also check the trend in weekly sales amounts of this supermarket chain. The first step is to group the total sales amount by date.
dates = df[['Date','Total']].groupby('Date').sum()dates.head()
The dates dataframe contains daily sales amounts. In order to check the weekly sales, we need to resample it before plotting. A simple way is to use the resample function.
dates.resample('7D').sum().plot(figsize=(9,4))
The period for resampling is specified by ‘7D’ indicating 7 days. We also need a function to tell pandas how to aggregate values in the resampled periods (e.g. sum or mean).
Let’s also check the product preferences of females and males. One way is to calculate the total number of items purchased in each product line. We first use the groupby function to obtain the data and then create a bar plot to make a comparison.
gender = df[['Gender','Product line','Quantity']]\.groupby(['Gender','Product line'], as_index=False).sum()gender.head()
We now have the quantities in each category for both females and males. The next step is to plot them.
import seaborn as snssns.set(style='darkgrid')sns.catplot(data=gender, y='Product line', x='Quantity',            kind='bar', hue='Gender', aspect=1.6, orient='h')
The catplot function is a figure-level interface to generate categorical plots. The type of plot is defined by the kind parameter.
Females purchase more products in the fashion accessories category which is expected. However, I’m a bit surprised to see males surpass females in the health and beauty category.
Filtering data based on a condition is a typical task in data analysis. Pandas is quite efficient in terms of how you can filter data. For instance, the query function accepts the condition as a string. There are, of course, some limitations but it comes in handy for many cases.
We can filter the purchases with total amount higher than 300 and quantity higher than 4 using the query function as below.
df.query('Total > 300 and Quantity > 4').shape(393, 12)
There are 393 purchases that fit these criteria.
The distribution of variables always provides valuable insight for data analysis. For instance, we can check the distribution of the total sales amount of purchases. One method is to plot a histogram which can easily be done with the plot function of pandas.
df.Total.plot(kind='hist', figsize=(8,5),               title='Histogram of Total Sale Amounts')
Histogram divides the value range of a continuous variable into discrete bins and counts the number of data points (i.e. rows) in each bin. As a result, we get an overview of the distribution of values.
We can also analyze the sales based on time period of the day. For instance, we might be interested in finding out if people tend to spend more in the mornings than evenings.
In order to efficiently use date and time functionalities under dt accessor of pandas, we should convert the data type of the time column to datetime.
df.Time = pd.to_datetime(df.Time)
The hour method returns the hour of the time as a string. We need to convert it to integer to use for filtering.
df[df.Time.dt.hour.astype('int') >= 20]['Total'].mean().round(2)306.26df[df.Time.dt.hour.astype('int') <= 10]['Total'].mean().round(2)311.10
Let’s elaborate on what we have done above. The hour method of dt accessor extracts the hour as string. We convert it to an integer using the astype function and use it to filter purchased made after 20. The final operation is taking the mean of the purchases that fit the specified condition. Then we do the same for purchases made before 10.
The main purpose of this article to demonstrate various pandas functions that help us analyze tabular data. The parameters of pandas functions are highly important as they make the functions more powerful and versatile. Thus, it is very important to know what parameters achieve.
One of the things I like about pandas is that there are almost always more than one way to accomplish a task. For instance, what we have done in the examples can also be done in a different way. In fact, you may challenge yourself to do the same using different functions.
Thank you for reading. Please let me know if you have any feedback.