Statistics for Data Science — A beginners guide to Descriptive Statistics in Python | by Angel Das | Towards Data Science

Statistics are much-needed components of Data Science. Why Statistics? In simpler terms, before any model can be developed, we need to be 100% sure of the data we are working with. The general idea being, “it is advisable to use clean data” for any machine learning algorithm, else the model ends up producing results contrary to what’s expected out of it. Think of it like a painter, who starts painting a portrait, but the palette used is not clean enough. Will that produce a painting that will go for an auction or will stay in one corner of the house?
But wait. Statistics is not that simple. Multiple underlying theories need to be validated before any assumption or methodology can be used. We all are familiar with bell curves or curves used to demonstrate the normal distribution of the data. If you are familiar with the fundamentals of statistics, you will realize that the normal distribution is a probability function that describes how the values of a variable are distributed. The X-axis represents the data points, whereas the Y-axis talks about the probability density estimate of the given point. Confused? I will make it clear later on. But the general idea being, as a beginner we need not bother ourselves with every component of statistics, rather focus on the more useful ones.
Statistics can be divided into Descriptive and Inferential. Descriptive statistics talk about data summarization, tables, graphs, and charts whereas Inferential statistics is a method that talks about a population parameter from a sample. All soccer fans, Lionel Messi scoring 48 goals in a season is descriptive statistics. What does that tell us about his chances of winning a Champions League final is inferential statistics. How can we go from a description of Lionel Messi’s performance to what his probability of winning a Champions League football reflects the journey of statistics? It is more about answering “What is?” vs. “What else it might be?”.
Population — The universe of all possible data for a given scenario. E.g. All customers with loan accounts in a bank
Sample — The set of observations from a given population. E.g. All customer with loan accounts in a bank who opted for customer care services in the last quarter is a sample. They are the same customers we spoke of initially, however, we are only filtering those out who opted for customer care services in the previous quarter, hence this is a sample
Parameter — A numerical summary or value associated with a population. E.g. Average debt of all customers with loan accounts in a bank
Statistics — A numerical summary or value associated with a sample. E.g. Average calls made by customers opting for a customer care facility at the start of the week
A population is usually not observed, the reason being when working with data, we don’t have access to every entity of the problem space. Also often there are data leaks or data lost during collation. Hence every data is considered as a subset of a population.
Data types are an important component of statistics as it determines the type of descriptive or inferential operation one can perform on the data. As illustrated below all variables (often referred to as attributes, data, columns) can be broken into two categories of Qualitative and Quantitative. Qualitative data allows for operations like count, graphs, plots, modes, whereas Quantitative data allows for operations like mean, median, mode, standard deviation, quartiles, and so on.
Raw data represents numbers and facts in the original format in which the data have been collected. The raw data needs to be converted into a form that can be utilized for decision making. Data extracted → is converted into information → which is then used to obtain knowledge about a scenario. Descriptive statistics is more about converting data to information. Now to start, let’s look at frequency distribution first. A frequency distribution is used to classify raw data into information. One common example of using frequency distribution is to analyze the count of data across different buckets using a histogram plot.
The range of data is divided into bins. These bins are then used to calculate the count of records under each group. E.g. Say we have access to two weeks of sales data from K Mart post lockdown. We want to analyze, how the sale value looks like across different buckets. In the example below, we can observe that in the past two weeks, there were three instances when the observed figure ranged from 44 to 46. Using this we can analyze, are sales figures different across days, or have they been similar post lockdown? What do you think?
From here on, we will take examples to understand the concepts of statistics. K Mart Supermarket recorded sales of their store in Melbourne post lockdown. Now using a central tendency they want to analyze how their business went in the last week of September. Here are the three questions the store manager wants to answer.1. What is the average sale volume?2. What is the median sales last week?3. Is “sale volume” similar for most of the days?
Using figures 4 and 5 we can answer the trend observed post lockdown. In general, the average sale was observed to be around 85 units whereas the median and mode values are 80 and 50 respective. The mode provides us a bigger picture of the trend observed post lockdown. Can we infer that post lockdown K Mart usually observes a sale volume of 50 units?
In simple terms, dispersion indicates how large the spread of the distribution is around the central tendency. The range is the simplest of all measures. It is calculated as the difference between the minimum and maximum value in a given data. If maximum data is equal to the minimum then it implies all observations are the same. However, when the difference is >0 we can infer that the data is dispersed. E.g. In the figure below, when we look into the sale volume by days, we can infer that the minimum and maximum sales figures are 30 and 65 respectively. Hence the range here is 65–30 which is 35. The variation of dispersion provides a picture of the range of sales that was observed post lockdown.
Note: Range is not used for scenarios where either the maximum or minimum value becomes an extreme, i.e. if the minimum is too small or the maximum is too large we should refrain from using the range.
What is the Median value in this example below?
Number of Observation = 14, hence median is the average of 7th and 8th value (n/2 + ((n/2) + 1))th observation divided by 2, hence 45. The 7th and 8th values are 45 and 45 respectively.
Let’s calculate p and q respective. For Q1, p=floor(1*(14+1)/4) which is 3 and q=(1*(14+1)) MOD 4 which is 3 as well. MOD is the modulus or remainder of a division operation and the floor takes the lowest integer of a value (e.g. floor of 3.7 is 3). So Q1 = X3 + 3/4(X4 — X3) where X3 and X4 are third and fourth value in an ordered set. Hence from the example below, X3 and X4 are 40 and 42 respectively, hence Q1 = 40 + (3/4)*(42–40)=41.5, similarly Q3 = 50 + (3/4)*(52–50) = 50.5. Interquartile Range is given by Q3-Q1 which is 50.5–41.5=9.
Dispersion provides a wide range of information about the data and is used as a key concept to detect and handle outliers. Dispersion provides information about the data distribution and answers key questions about the trend of a variable.
The boxplot provides a picture of how sales figures are distributed. Note, the Median is 45, Q1 and Q3 values are 41.5 and 50.5 respectively. IQR is represented as Q3-Q1=9. Any value beyond Q1–1.5xIQR or Q3+1.5xIQR, which are 28 and 64 are considered as an outlier. Since 65 (sales figure observed on 30th September) falls beyond the upper limit, it is considered as an outlier.
Standard deviation is a measure of the amount of variation or dispersion of a set of values or observations. Standard deviations are calculated as the square root of variance. Variance is a measure of how far the data points are compared to the mean. A low standard deviation indicates that the values are closer to the mean, whereas a high standard deviation is an indication of extreme values or skewness of the data. Standard deviation like dispersion is used as a metric to analyze the distribution of data.
Time to pick up a calculator and do the math. The formulas for variance and skewness are outlined on the right side of the image. Standard deviation can be used to answer a multitude of questions ranging from:
Are we observing variation in sales? — Yes, observe the minimum and maximum figures
How different are the sales figures? — The sales figures on an average lies between 38 and 54 which are +- 1 standard deviation of mean
How many days observed such a trend mentioned above? — 79% of the sales figure lies between 38 and 54 which means although we observe a variation of sales, the variations aren’t significant enough when compared to the distribution
Correlation is a measure of the degree of association between two variables. It provides some directionality into how one variable changes w.r.t the other. Correlation can range between -1 to +1, where -1 signifies two variables being negatively correlated, i.e. if one variable increases, the other decreases, and vice versa. A correlation of 0 signifies no relationship between two variables, i.e. increase or decrease of one variable, doesn’t impact the other. A correlation of +1 indicates that if one variable increases then the other variable displays a similar trend. Always remember that correlation doesn’t imply causation. It only talks about the degree of association.
Statistics helps us in understanding the following:
The average value of any KPI (Key Performance Indicators like Sales, Customer Count, NPS, Revenue, Market Basket, etc.)
Mean, median, and mode are one of the widely used imputations for missing value
These central tendencies help us validate the quality of the data and align them with domain knowledge. E.g. For a grocery shop average sale volume increases during festive seasons when compared to the rest of the year. When a data is procured and average figures are observed, if the central tendencies are significantly different from what is known, we need to check the data quality
Linear regression, logistic regression, neural networks, follow certain assumptions which are validated using correlation, outlier checks, and so on
This dataset is originally from the National Institute of Diabetes and Digestive and Kidney Diseases. The objective of the dataset is to diagnostically predict whether or not a patient has diabetes, based on certain diagnostic measurements included in the dataset. Several constraints were placed on the selection of these instances from a larger database. In particular, all patients here are females at least 21 years old of Pima Indian heritage. Dataset is available here.
Number of times pregnantPlasma glucose concentration a 2 hours in an oral glucose tolerance testDiastolic blood pressure (mm Hg) — Blood PressureTriceps skinfold thickness (mm)2-Hour serum insulin (mu U/ml)Body mass index (weight in kg/(height in m)2)Diabetes pedigree function — a function which scores the likelihood of diabetes based on family historyAge (years)Class variable (0 or 1) — 0 signifies non-diabetic and 1 signifies diabetic
Number of times pregnant
Plasma glucose concentration a 2 hours in an oral glucose tolerance test
Diastolic blood pressure (mm Hg) — Blood Pressure
Triceps skinfold thickness (mm)
2-Hour serum insulin (mu U/ml)
Body mass index (weight in kg/(height in m)2)
Diabetes pedigree function — a function which scores the likelihood of diabetes based on family history
Age (years)
Class variable (0 or 1) — 0 signifies non-diabetic and 1 signifies diabetic
import warningswarnings.filterwarnings(‘ignore’)import pandas as pdimport numpy as npimport matplotlib.pyplot as plt   import seaborn as sns#from sklearn import metrics#from sklearn.model_selection import train_test_split# To enable plotting graphs in Jupyter notebook%matplotlib inline sns.set(color_codes=True) #adds a good background to our plots
#Load the file from local directory using pd.read_csv which is a special form of read_tablepima_df = pd.read_csv(“pima-indians-diabetes.csv”)print(pima_df.shape)print(pima_df.size)pima_df.head(10)
#Central tendency calculationsprint(‘MEAN: \n’,pima_df.mean())print(‘\r \n’)print(‘MEDIAN: \n’,pima_df.median())print(‘\r \n’)print(‘MODE: \n’,pima_df.mode())print(‘\r \n’)print(‘Q1-Quartile 1: \n’,pima_df.quantile(q=(0.25,0.50,0.75)))
mean=pima_df[‘Plas’].mean();median=pima_df[‘Plas’].median();mode=pima_df[‘Plas’].mode();fig, ax = plt.subplots(figsize=(10,6)); # — — — — — — — — -Setting size of the canvassns.distplot(pima_df[‘Plas’]);plt.title(‘Distribution plot if Plas variable’);plt.axvline(mean,color=’green’,label=’Mean’); plt.axvline(median,color=’blue’,label=’Median’);# plt.axvline(mode[0],color=’red’,label=’Mode1')# plt.axvline(mode[1],color=’red’,label=’Mode2')plt.legend();
# # Pairplot using scorr = pima_df.corr();print(corr)# sns.heatmap(corr, annot = True);fig, ax = plt.subplots()fig.set_size_inches(10, 10)plt.title(‘Heat Map of Correlation \n’)sns.heatmap(corr,annot=True,cmap=’YlGnBu’,vmin=-1,vmax=1, linewidths=.5, center=0);
plt.figure(figsize=(15,10))pos = 1for i in pima_df.drop(columns = {‘class’}).columns: plt.subplot(3, 3, pos) sns.boxplot(pima_df[i]) pos += 1
About the Author: Advanced analytics professional and management consultant helping companies find solutions for diverse problems through a mix of business, technology, and math on organizational data. A Data Science enthusiast, here to share, learn and contribute; You can connect with me on LinkedIn and Twitter.