Understanding Probability And Statistics: Statistical Inference For Data Scientists | by Farhad Malik | Towards Data Science

Every data scientist must be familiar with the concepts of statistical inference. Therefore, this article aims to provide an overview of statistical inference. It will take you deep into the statistical world of inference in a manner that is easy to grasp and understand.
Some scientists regard statistical inference as one of the most difficult concepts in statistics and understanding it thoroughly can really help them add significant value to their projects and the team they are in.
I will aim to explain statistical inference in a simplified manner so that everyone can understand it.
This article will aim to explain:
What is the statistical inference?Understanding Statistical Inference ProcessTest Statistics — Bigger Picture With An ExampleHypothesis TestingTypes Of Error
What is the statistical inference?
Understanding Statistical Inference Process
Test Statistics — Bigger Picture With An Example
Hypothesis Testing
Types Of Error
Data scientists usually spend a large amount of time to gather and assess data. The data is then used to deduce conclusions using data analysis techniques.
Sometimes these conclusions are observed and the findings are easily described using charts and tables. This is known as descriptive statistics. Other times, we have to explore a measure that is unobserved. This is where the statistical inference comes in.
The descriptive statistical inference essentially describes the data to the users but it does not make any inferential from the data. Inferential statistics is the other branch of statistical inference. Inferential statistics help us draw conclusions from the sample data to estimate the parameters of the population. The sample is very unlikely to be an absolute true representation of the population and as a result, we always have a level of uncertainty when drawing conclusions about the population.
As an instance, the data scientists might aim to understand how a variable in their experiment behaves. Gathering all of the data (population) for that variable might be a humongous task. Data scientists, therefore, take a small sample of the population of their target variable to represent the population, and then they perform statistical inference on the small sample(s).
The samples are used to estimate the population
The aim of the data scientists is to generalise from a sample to a population knowing there is a degree of uncertainty. Hence the analyses help them make propositions about the entire population of the data. Sometimes data scientists simulate the samples to understand how the population behaves and for that they make assumptions about the underlying probability distributions of the variable. This is one of the core reasons why the concept of probability is heavily recommended to the data scientists.
Subsequently, a number of hypotheses and claims are made about the properties of the population. Next, the statistical models are used to infer conclusions from the sample to deduce the properties of the population.
The article below provides a thorough understanding of what probability distributions are and I highly recommend everyone to read the article
towardsdatascience.com
This section will help us understand the process of statistical inference. Let’s assume that the data scientists want to learn about the behaviour of their target variables. They might be interested in understanding how a parameter of a population behaves.
As an instance, they might want to assess whether all overnight batch jobs across all of the departments in a bank complete within a particular time frame.
Or, they might want to find the average height of a population in a country.
Or, maybe they want to understand whether a business made the same profit and the users behaved differently before or after a specific event, such as after a new product was launched.
Or they want to prove a particular claim about a population wrong.
Occasionally it is too difficult to gather all of the data of a population. Consequently, the data scientists prepare their sample set from the population.
For instance, the parameter the data scientists want to learn about could be the mean or variance of the population. They extract the sample from the population and they can then perform statistical analysis to estimate the population parameter. Sometimes, they check whether the parameter meets a specific value which is believed to be true.
The diagram below illustrates the process:
Notice how the statistical inference is as good as the sampling techniques
There is always some element of noise in the sample. The standard deviation of the sample over the square root of the sample size is the sample standard error. This is the noise/dispersion of the sample from the mean. This measure is based on the sample size. The formula indicates that the larger the sample size, the lower the impact of standard deviation, and the closer the sample value is to the population value. If we increase the sample size then eventually the sample will start resembling the population. I recommend reading this article that explains what convergence and CLT are. These are crucial topics for data scientists to understand:
towardsdatascience.com
As an instance, we can make a claim that our target variable follows a normal distribution and its mean is always zero, and variance is 1 and so on.
So, after gathering a small sample, we start plotting the values within 10 bins in a histogram. From the chart, we conclude that the data follows a normal distribution because of the bell-shaped curve that is produced. And from there, we can start estimating the mean and variance of the sample to draw inference about the population. We can also start producing more data from the believed probability distribution of the sample.
But how confident are we? How can we reject this claim? Or is there a way to calculate this claim so that we can be sure? Can we prove it via quantifiable measures? This is where statistical inference comes in!
It’s vital to understand statistical inference because it can help us understand our statistical choices better
A sample can be thought of as a random variable having its own probability distribution, patterns, and trends.
We can collect a number of samples and workout their means, standard deviation, and variances to gain better insight into the data.
The process of test statistics can be used to help us make calculated decisions. Before performing an experiment on the sample, scientists have an idea of what the expected results need to be. This is usually gathered by exploratory data analysis.
Now, from the theory, let’s review how statistical inference works.
Let’s imagine that we are working on a data science project and our client is a large financial institution. Let me refer to our client as Bank A. They want us to understand whether their overnight batch systems on average take the same amount of average time as other banks in the industry. The client wants to use our results to understand whether they need to invest differently in technology this year.
Let’s also assume that it is widely believed in the industry that the overnight batch job times in all of the banks on average take ≤ 6 hours to complete.
This is a big data problem. We can’t go around every bank and get hold of all of their batch job data. This will be an extremely time-consuming and costly job. We can imagine how much data our population holds. Therefore, the first step would be to gather a good sample set across different banks.
For our example, we can categorise the banks into groups based on the size of their transactions and take one-year worth of the batch job timings from a member of each group.
The sample is chosen such that it is the best representation of the population of data under test. Always spend time thinking through the sampling techniques
Success of hypothesis analysis is based on the quality of the chosen sample.
We then make a claim that on average the overnight batch jobs for Bank A take longer than 6 hours to complete. Now, this is our hypothesis under test. Let’s prove it right.
Let’s hold the thought while I cover the foundation and then I will get back to it.
This brings me to the second sub-topic of this article — Test Statistics
We perform the following 5 steps to prove a claim:
Start by stating the widely believed claim, which is known as the Null Hypothesis.Outline the minimum significance level/confidence level before we can reject the claim. This could be 5%. It means that we believe that around 5% of the time, our model will produce inaccurate results.Calculate our sample results mean and standard deviationCalculate the test statisticsFinally, based on the outcome, the chosen result is stated.
Start by stating the widely believed claim, which is known as the Null Hypothesis.
Outline the minimum significance level/confidence level before we can reject the claim. This could be 5%. It means that we believe that around 5% of the time, our model will produce inaccurate results.
Calculate our sample results mean and standard deviation
Calculate the test statistics
Finally, based on the outcome, the chosen result is stated.
The chosen test statistic is dependent on the distribution of the sample along with the sample size.
The next section will explain each of the steps in detail.
There are two hypotheses in any test:
Null Hypothesis — what is known as the accepted truth and what we want to test. This is what we want to prove wrong. For our case, it is that the average (mean) of batch jobs take less than or equal to 6 hours to complete.Alternate Hypothesis — what we need to accept if the Null Hypothesis is not true. This is what we believe is true. This is our hypothesis about the bank. Our hypothesis is that the mean of the jobs is greater than 6 hours.
Null Hypothesis — what is known as the accepted truth and what we want to test. This is what we want to prove wrong. For our case, it is that the average (mean) of batch jobs take less than or equal to 6 hours to complete.
Alternate Hypothesis — what we need to accept if the Null Hypothesis is not true. This is what we believe is true. This is our hypothesis about the bank. Our hypothesis is that the mean of the jobs is greater than 6 hours.
Note, the null and alternative hypothesis cannot be true at the same time.
Side note: This is a one-tail test. One Tail Alternative Hypothesis is a uni-directional test. The two tail alternative hypothesis tests are bidirectional tests whereby a statistician is interested in checking equality of data e.g. Whether a value is within a range of values.
Your significance level indicates how confident you are about the sample and your methodology to support your claim. The significance level is known as Alpha. The usual value for alpha is 1% or 5%. Lower alpha implies that you are very certain about the results. The chosen confidence level forms the foundation of risk management credit metrics.
Alpha is the level of significance in Hypothesis analysis.To elaborate, Alpha is the range of values that can be accepted before the Null Hypothesis is rejected.
It is the lower threshold.
We can choose T, Z, or F statistics amongst others. I will briefly explain them.
Once a sample is chosen to represent a population, its mean and standard deviation are calculated. And then we perform the test statistic.
There are a number of test statistics e.g. T, Z, F, etc.
I highly recommend reading this article as it explains the three most important statistics in-depth:
towardsdatascience.com
T statistics is used for testing the equality of means of two small populations. The sample follows Student T Distribution and the sample size is around 30 observations. The population standard deviation is unknown.
Example: You have a sample of 10 cars and you want to measure the average fuel consumption of all cars in the town. Your hypothesized claim is that on average, cars consume 10 liters of fuel per day. Let’s also consider that you are 99% confident in the methodology. You can then compare the hypothesized mean with the sample mean and work out if you need to reject the Null Hypothesis based on the T distribution table at 99%.
Z statistics is used for testing means of two large populations. The sample follows Normal Distribution and the sample size is usually greater than 30. The population standard deviation is known.
Example: Assume you have collected a sample of 500 individuals to estimate the average number of people wearing blue shirts on a daily basis. Let’s also consider that you are 95% confident in your model. You can then compare the hypothesized mean with the sample mean and work out if you need to reject the Null Hypothesis based on the Z distribution table at 95%.
F statistics is used for comparing the variances of two populations. Variation is the sum of the squared deviations of each observation from its group mean divided by the error degree of freedom.
Example: You can use an F test to compare the variability of software bugs in two IT systems in your company.
Each of these test-statistics has its own straightforward formulae. For the sake of simplicity, I am not covering these formulae in this article.
When the results are calculated, they are checked against the value from the distribution table. Note, the results are dependent on the sample size, standard deviation, and the mean of the sample.
We can perform test statistics using Python. As an instance, to perform T-Test, we can do:
from scipy import statsstats.ttest_ind(collection_one, collection_two)
From the results of the calculation, we can state the result.
Let’s assume we computed 0.50 from the test statistics calculation. We can now look up the value in the probability distribution table for our sample for our chosen alpha of 95%. The Z-distribution table gives us a value of 1.96.As 0.50 ≤ 1.96, we have to accept the Null Hypothesis.
This means that the batch jobs do complete within the claimed 6 hours time and our claim was wrong. That’s how statistical inference works.
Lastly, I wanted to cover the are two types of errors. It is important to understand how and where the experiments can go wrong.
There are two types of errors which can occur while we state the results:
Type 1 error: Null Hypothesis was correct but the analysis proved it wrong.Type 2 error: Null Hypothesis was wrong but the analysis couldn’t prove that it was wrong
Type 1 error: Null Hypothesis was correct but the analysis proved it wrong.
Type 2 error: Null Hypothesis was wrong but the analysis couldn’t prove that it was wrong
This article explained one of the most important subjects in the field of statistics. Statistical inference is a must-know topic for every data scientist.
The article then explained how we calculate the test-statistic and perform a hypothesis analysis.
The steps can be used to articulate whether a hypothesis is correct or not. It helps one make conscious risk-averse decisions and better understand our data with quantitative measures.