Mastering Summary Statistics with Pandas | by Sadrach Pierre, Ph.D. | Towards Data Science

Pandas is a python library used for data manipulation and statistical analysis. It is a fast and easy to use open-source library that enables several data manipulation tasks. These include merging, reshaping, wrangling, statistical analysis and much more. In this post, we will discuss how to calculate summary statistics using the Pandas library.
Let’s get started!
For our purposes we will be exploring the Movies on Netflix, Prime Video, Hulu and Disney Plus data set. The data can be found here.
Let’s get started!
First, let’s read the data into a Pandas data frame:
import pandas as pd pd.set_option('display.max_columns', None)pd.set_option('display.max_rows', None)df = pd.read_csv("MoviesOnStreamingPlatforms_updated.csv")
Next, we will print the columns available in this data:
print(df.columns)
We won’t need the ‘Unamed: 0’ column so we can delete it with the ‘del’ keyword:
del df['Unnamed: 0’]
Now, let’s print the first five rows of data using the ‘head()’ method:
print(df.head())
The first method we will look at is the Pandas’ ‘mean()’ method. Let’s display the mean IMDb rating:
print("Mean IMDb Rating: ", df['IMDb'].mean())
We can use the Numpy ‘round()’ method to round the mean to the nearest tenth:
import numpy as npprint("Mean IMDb: ", np.round(df['IMDb'].mean(), 1))
We can also display the standard deviation, median, max and min for the ‘IMDb’ column:
print("Standard Deviation in IMDb Rating: ", np.round(df['IMDb'].std()))print("Median IMDb Rating: ", np.round(df['IMDb'].median(), 1))print("Max IMDb Rating: ", df['IMDb'].max())print("Min IMDb Rating: ", df['IMDb'].min())
For convenience we can write a function that gives these summary statistics for any numerical column:
def get_statistics(column_name):    df_copy = df.copy()    print(f"Mean {column_name}: ", np.round(df_copy[column_name].mean(), 1))    print(f"Standard Deviation in {column_name}: ", np.round(df_copy[column_name].std()))    print(f"Median {column_name}: ", np.round(df_copy[column_name].median(), 1))    print(f"Max {column_name}: ", df_copy[column_name].max())    print(f"Min {column_name}: ", df_copy[column_name].min())
If we call our function with ‘IMDb’ we get the following:
get_statistics('IMDb')
Let’s call our function on the ‘Runtime’ column:
get_statistics('Runtime')
Finally, let’s try the ‘Rotten Tomatoes’ column:
get_statistics('Rotten Tomatoes')
We get the following error:
It’s clear that we need to treat this case separately. If we print the first five rows of ‘Rotten Tomatoes’ values we see that the values are strings:
print(df[‘Rotten Tomatoes’].head())
In our function, we can check if the ‘column_name’ input is ‘Rotten Tomatoes’. If it is we remove the ‘%’ symbol and convert the string to a float before calculating the summary statistics:
def get_statistics(column_name):    df_copy = df.copy()    if column_name == 'Rotten Tomatoes':        df_copy[column_name] = df[column_name].str.rstrip('%')        df_copy[column_name] = df_copy[column_name].astype(float)        ...
Now we can call our function with ‘Rotten Tomatoes’:
get_statistics('Rotten Tomatoes')
We can use the describe function to generate the statistics above and apply it to multiple columns simultaneously. It also provides the lower, median and upper percentiles. Let’s apply the ‘describe()’ method to ‘IMDb’ and ‘Runtime’:
print(df[['IMDb', 'Runtime']].describe())
If we wanted to include ‘Rotten Tomatoes’ in these statistics we’d need to convert it into a float like before:
df['Rotten Tomatoes'] = df['Rotten Tomatoes'].str.rstrip('%')df['Rotten Tomatoes'] = df['Rotten Tomatoes'].astype(float)print(df[['IMDb', 'Runtime', 'Rotten Tomatoes']].describe())
Suppose we wanted to know the average runtime for each genre. We can use the ‘groupby()’ method to calculate these statistics:
runtime_genre = df[["Genres", "Runtime"]].groupby("Genres").mean()print(runtime_genre.head())
We can also look at the average ‘Rotten Tomatoes’ by ‘Country’:
rottentomatoes_country = df[["Country", "Rotten Tomatoes"]].groupby("Country").mean().dropna()print(rottentomatoes_country.head())
Finally, we can write a function that allows us to reuse this logic for any categorical column and numerical column:
def get_group_statistics(categorical, numerical):    group_df = df[[categorical, numerical]].groupby(categorical).mean().dropna()    print(group_df.head())
Let’s call our function with ‘Genres’ and ‘Runtime’:
get_group_statistics('Genres', 'Runtime')
I’ll stop here but I encourage you to play around with the data and code yourself.
To summarize, in this post we discussed how to generate summary statistics using the Pandas library. First we discussed how to use pandas methods to generate mean, median, max, min and standard deviation. We also implemented a function that generates these statistics given a numerical column name. Next we discussed the ‘describe()’ method which allows us to generate percentiles, in addition to the mean, median, max, min and standard deviation, for any numerical column. Finally, we showed how to generate aggregate statistics for categorical columns. If you are interested in learning more about data manipulation with pandas, machine learning or even just some of the basics of python programming check out Python for Data Science and Machine Learning: Python Programming, Pandas and Scikit-learn Tutorials for Beginners. I hope you found this post useful/interesting. The code from this post is available on GitHub. Thank you for reading!