Pivot Tables and Operations Using Pandas | by Ayush Ruel Das | Towards Data Science

We will be learning how to effectively create pivot tables and perform the required analysis. We will be doing this with a famous automobile dataset, taken from UC Irvine. You may find the dataset from the following link. It is suggested that you read a little bit about the dataset as well. I used this dataset as a part of an online course that I took on Data Analysis.
https://archive.ics.uci.edu/ml/machine-learning-databases/autos/imports-85.data
So let's get the libraries first, load our data, and have a peek of the data.
import pandas as pdimport numpy as nppath = "Downloads/imports-85.data"df = pd.read_csv(path, header=None)
If we look at the data right now, you won't be able to understand anything. But still, let’s have a peek.
df.head()
So, the data looks something like this. First things first, let us give the data some nice headers.
headers = ["symboling","normalized-losses","make","fuel-type","aspiration", "num-of-doors","body-style",         "drive-wheels","engine-location","wheel-base", "length","width","height","curb-weight","engine-type",         "num-of-cylinders", "engine-size","fuel-system","bore","stroke","compression-ratio","horsepower",         "peak-rpm","city-mpg","highway-mpg","price"]df.columns = headers
So now data looks something like this.
There are some data cleaning steps that you will need to perform. But, I won’t be going in detail about them. Since we are discussing pivot tables, I am assuming that you might be having some basic knowledge about data cleaning. For that, use the code below.
df.replace("?", np.nan, inplace = True)df["normalized-losses"].replace(np.nan, df["normalized-losses"].astype("float").mean(axis=0), inplace=True)df["bore"].replace(np.nan, df['bore'].astype('float').mean(axis=0), inplace=True)df["stroke"].replace(np.nan, df['stroke'].astype('float').mean(), inplace=True)df["horsepower"].replace(np.nan, df['horsepower'].astype('float').mean(), inplace=True)df["peak-rpm"].replace(np.nan, df['peak-rpm'].astype('float').mean(), inplace=True)df[["price"]] = df[["price"]].astype("float")
I’ll just give a brief of what I just did. If you check the data types of each column, using df.dtypes you will find that some of the columns are not of the correct data type and that, you don't want at all while working with pivot tables. For example, the ‘Price’ column is of data type ‘Object’ which makes no sense. And also, the data has null values, a question mark (?) to be more precise.
To fix this, we replaced it with NAN. Then, these NAN values for some numerical columns have been replaced by the mean of that column.
Now since we have clean data, let's create our first pivot table. Firstly, we have to select the data that we want from the dataframe.
df_selction = df[['fuel-type', 'drive-wheels', 'price']]
Now, we will group the data by the columns ‘fuel-type’ and ‘drive-wheels’. And we will be using the average values of the car prices.
df_selection_gb = df_selction.groupby(['drive-wheels', 'fuel-type'], as_index=False).mean()df_selection_gb
This may be a bit difficult to read or perform any analysis. Hence, we convert it to a pivot table, and let's find out what we get.
df_first_pivot = df_selection_gb.pivot(index='drive-wheels', columns='fuel-type')df_first_pivot
Now, you may notice some NAN values in the pivot table. But that's fine because we don't have any data on cars which are four-wheel drive and powered by diesel.
With the above pivot table, you may answer questions like —
What is the average price of diesel-powered cars having a forward wheel drive?
Which car type is the most expensive?
So you can get the answer as diesel-powered cars with rear-wheel drive is the most expensive. Let's say we want to answer questions like -
The total number of gas-powered cars with four-wheel drive?
Then you may use the count function instead of mean —
df_selection_gb_2 = df_selction.groupby(['drive-wheels', 'fuel-type'], as_index=False).count()df_count_pivot = df_selection_gb_2.pivot(index='drive-wheels', columns='fuel-type')df_count_pivot
Now, let's go ahead and create another group and try to analyze it.
df_selection_2 = df[['drive-wheels','body-style','price']]grouped_df = df_selection_2.groupby(['drive-wheels','body-style'],as_index=False).mean()grouped_df
As you can see very clearly, this grouped data is much more complex to read and understand than the first one. So let us convert it in a pivot table and try again.
grouped_pt = grouped_df.pivot(index='drive-wheels',columns='body-style')grouped_pt
Now, this is much easier to analyze. If you want to perform operations on these pivot tables, you may do so just as a normal dataframe but with a slight difference in column naming.
Let us look at the columns of the pivot table.
grouped_pt.columns
As you can see, it’s multi-indexed. Say we want to get the difference of mean prices of sedan and wagon body-styled cars.
grouped_pt[('price','sed-wag')] = (grouped_pt[('price', 'sedan')]-grouped_pt[('price', 'wagon')])grouped_pt
Similarly, if you want to compare a value over a period of time like the previous month and the current month. You can go ahead and do that the very same way. You may want to calculate the growth in sales or perhaps the rate at which it is growing. Since we don't have a date column in our dataset, I’ll show you some more possible insights that you may find out.
For example, we want the average price of a single sedan according to the drive-wheels column.
For that, you may want to delete the ‘sed-wag’ column that we just created.
del(grouped_pt[('price','sed-wag')])
Now let’s create another pivot with the count of cars.
df_selection_3 = df[['drive-wheels','body-style','price']]grouped_df_2 = df_selection_3.groupby(['drive-wheels','body-style'],as_index=False).count()grouped_count_pt = grouped_df_2.pivot(index='drive-wheels',columns='body-style')grouped_count_pt
Let us do the math and see our results in the form of a Dataframe.
rate_4wd_sedan = (grouped_pt[('price', 'sedan')]) / (grouped_count_pt[('price', 'sedan')])rate_4wd_sedan.to_frame()
So, you can see that we have got the average price of a sedan according to the drive-wheels column. You may play around a bit more and get lots of useful insights like these.
If you have any doubts, please feel free to reach out and suggestions are always welcomed. I hope the post was useful.