Reading and interpreting summary statistics | by Mahbubul Alam | Towards Data Science

A typical data science project starts with data wrangling. It is the process of cleaning messy data and transforming them into appropriate formats for further analysis and modeling.
The next step in the process is exploratory data analysis or EDA. This is where you spot hidden issues and anomalies in the data. Most importantly, this is where the discovery of information and insights begins.
In this article, I’ll talk about one of the most useful yet underutilized tools in EDA — descriptive statistics.
Descriptive statistics is packed with information and insights. A pair of experienced eyes will take a look at every single data point and extract valuable information from the summary table. However, I’ve seen many people in data science who face difficulty interpreting statistical summaries in a meaningful way. Some people would simply take a brief look and then move on to the next part as soon as possible. The purpose of this article is to walk you through how to read descriptive statistics and extract useful information.
Let’s first see what a table of summary statistics looks like for a given dataset. I’ll use a built-in dataset that comes with seaborn library in Python.
import seaborn as snsimport pandas as pddf = sns.load_dataset('tips')df.head()
Each observation (row) in this dataset represents dining in a restaurant. The columns names here are self-explanatory. Among the numeric columns, ‘total_bill’ refers to how much bills the diners paid and ‘tip’ represents the amount of tip they paid.
Just a simple method call df.describe() gives you the summary statistics for the numeric columns (I’ll touch upon categorical columns towards the end).
So how do you read this summary statistics? You can, in fact, extract 3 kinds of information from this table:
Statistical distribution of variables
Anomalies in the data
Other points of interest
The first thing you probably are looking at in the summary statistics is the mean — a key measure of central tendency.
With the mean value, you are trying to get a sense of what an average data point looks like. You also want to compare it with your expectation and experience. For example, the mean bill paid by a customer is $19.7. Does it sound normal to you? For an average meal for 1 person it’s a bit high, isn’t it? But if you knew that it’s a bill for 2 diners (look at the mean of “size” column) then the bill sounds reasonable.
The mean of ‘tip ’ is $2.9, which is about 15% of the ‘total bill’. It looks pretty normal to me.
The next number you should check out is the standard deviation (STD).
Standard deviation is a measure of variation/dispersion of data points with respect to the mean. Smaller STD indicates that the data are mostly centered around the mean, whereas a higher STD value indicates the data points are rather dispersed.
In the example above, the STD of total_bill is $8.9. That tells me that there’s some variation in how much people spend in restaurants. This variation is not totally unexpected and can be further corroborated by additional statistics later on.
Next comes the 50th percentile (the 50% column), also known as the median. Like mean, it’s another measure of central tendency. Median is a preferred metric over mean if there are outliers or high variability in the data.
If the difference between mean and median is small, you can infer that the data is symmetrically distributed. If the median is higher than the mean, data is likely left-skewed in distribution.
import matplotlib.pyplot as pltsns.displot(df['total_bill'])plt.axvline(x = df['total_bill'].mean(), linewidth = 3, color='r')plt.axvline(x = df['total_bill'].median(), linewidth = 3, color='y');
Min and max values represent the lower and upper limit, respectively, of a variable in the dataset.
The total bill in our dataset ranges between $3-$50. The upper-end of the bill in a restaurant seems reasonable, although in a large dataset I’d expect even a higher value. The minimum ($3) seems a bit low for a restaurant bill, especially when the average is $19. But that’s not implausible.
You can get a sense of outliers, anomalies and other points of interest in the dataset using descriptive statistics. How?
A large difference between the 75th percentile and the maximum value indicates the presence of potential outliers. The same goes for the difference between the minimum value and the 25th percentile.
To absolutely confirm outliers you can create a boxplot for visual inspection.
sns.boxplot(y=df['total_bill']);
Sometimes descriptive statistics can raise red flags. We do not have any red flag in this dataset, however, I’ve seen other places with unexpected minimum values (0 or negative) or absolutely insane maximum values (e.g. someone’s age 120 years!). These are obvious indications that there are issues in the data and need further investigation.
So far we have investigated descriptive statistics for numeric variables. Python pandas also offer a summary for categorical variables.
How to interpret this table:
‘count ’refers to the total number of observations in the dataset.
‘sex ’ column has two unique categories
‘top ’ refers to the dominant category (male in this case)
‘freq ’ refers to the proportion of the dominant category (there are 157 male customers who paid the bills)
Descriptive statistics presents a powerful synthesis of a dataset presented concisely and can be used to extract valuable information as part of the exploratory data analysis, including statistical distribution of variables (min, max, mean, median, percentiles), potential anomalies, and other points of interest. Alongside numerical data, descriptive statistics also synthesizes information on categorical variables, showing the unique number of categories in each column, the dominant category, and the proportion of that category.
Thanks for reading, if you have comments and thoughts feel free to write them down below or connect with me via Medium, Twitter, or LinkedIn.