Statistics Revisited
By: Alvin Mak
Tags: Normal Distribution, P Value, Getting Started, Statistics, Confidence Interval

Inferential Statistics

Now that we have recapped to some extent descriptive statistics, let us revisit the question that was asked earlier on: To what confidence can I trust that the sample results speak the truth about the population?

Suppose we want to determine the average weight of a newborn baby. By collecting a sample of weights from 8 babies, the sample mean is found to be 3.2kg. Another sample consisting of 20 babies reveals a sample mean of 3.4kg. Intuitively, which mean should we trust more?

Equation [5]

This example illustrates the concept of errors in estimates — What should be our “best guess” when determining a population parameter. From equation[5], we can see that when the sample size N increases, the standard error decreases, where s is the standard deviation of the sample. This gives some indication of how much variation there could be in the sample mean.

Now knowing there is also variation across and within samples, a point estimate i.e. population parameter taking a specific value does not seem like a feasible option. Instead, a confidence interval is introduced, which is a range of numbers within which the population parameter is believed to fall into.

A normal distribution, shaped like a bell curve, is commonly used to determine the confidence interval. This is because of the unique observation brought about by the Central Limit Theorem (CLT). The CLT states that if the population has a finite mean and standard deviation, when sufficiently large random samples are taken from it with replacement, the distribution of the sample means will be approximately normally distributed.

It is important to note that we may not necessarily know what distribution the population data follows. Similarly, each sample taken will follow an unknown random distribution. However, when a sufficiently large number of samples are drawn, their averages will follow a normal distribution due to the Central Limit Theorem. The diagram below illustrates this point.

Image by author

From the normal distribution graph created, we can visualize where the estimated population mean lies. The standard deviation is calculated by considering each individual sample mean as a data point. However, if there is only one sample, and the distribution is assumed to be normal, the standard deviation can be estimated to be the standard error of the mean, per Equation[5].

To construct the confidence interval we need to first transform the normal distribution into a z-distribution, otherwise known as standardizing the normal distribution. In doing so, the distribution will eventually take a mean of 0 and a standard deviation of 1.

Equation[6]

Image by author

In a normal distribution, the mean, median and modes are equal with the curve symmetrical about the mean. Total area under the curve is 1, which signifies 100% probability. Z- scores are values along the z axis which correspond to the number of standard deviations from the mean. At varying z-scores, the z-table identifies the total area bounded by these scores. The area also represents the percentage of observations that will lie within this interval, which is sometimes called the p-value. Therefore, we can estimate a statistic with probabilistic confidence represented as confidence intervals.

Image by author

If the sample size n is small (< 30), and/or the population standard deviation is unknown, we use a t-distribution instead. Also having a bell shaped curve, the t-distribution has a lower peak and fatter tails compared to a z-distribution, resulting in a more conservative confidence interval given the same significance level. We can think of this as a way of penalizing the estimate since less reliable information is provided.

Image by author

Instead of referring to the z-table, the t-table is used instead. One point to note when using the t-table is that the degrees of freedom (D.O.F) is used instead, where D.O.F = n -1. The D.O.F refers to the numbers of values in a final calculation that are free to vary. In this case, the D.O.F = n -1 because the last term in n has to be a fixed value given a known sample mean and sample standard deviation.

The concept of confidence intervals and significance levels is used extensively in statistics and machine learning techniques. While it does provide a range of estimates for the population parameter at a relatively high confidence, there exists a chance of error where the mean actually lies outside of the stated confidence interval. More will be covered in my next post on Hypothesis Testing. Stay tuned!