On the Importance of the Central Limit Theorem
By: James Andrew
Tags: Machine Learning, Data Science, Statistics, Machine Learning Engineer

The Central Limit Theorem (CLT). Something that we likely learned in high school math (AP Stats for me). What I remember about it was that because of the CLT, the magic number for sampling was n = 30. Like many sleep-deprived teens, I nodded and jotted that down in my notebook as I sat in the back of the class, struggling to read the faded projector from the back of the class. As an aside I swear that this was among the last projectors in the entire school, with all my other classes having those fancy smart boards.

While I remembered the number, I never fully appreciated the importance of the CLT and just how fundamental it is to statistics, and for me, its application to machine learning models. The CLT is the phenomenon that allows for what we consider to be statistics: it enables sampling methods. Without it, we could not reliably compute confidence intervals, and most statistical methods and machine learning algorithms rely on the CLT, Hypothesis tests being just one example. I believe that this fundamental theorem deserves a better understanding by the ML community and I hope to explain its beautiful simplicity in a profound manner.

History

The Central Limit Theorem is not new. It has been around in many forms and was refined for over two centuries. The first published version was by deMoiver in 1738. He proved a special case for Bernoulli trials.

Laplace published generalizations of the CLT in 1776, 1785, and 1820. A rigorous proof of a version close to the modern form was published by Chebyshev in 1887. Feller and Lévy worked on generalizations and some special cases into the mid 1930s.

The CLT is a statistical theory stating that given a sufficiently large sample size from a population with finite level of variance, the mean of all the samples from the same population will be approximately equal to the mean of the population.

Sample a population many times, and the distribution of means of all samples are normally distributed, regardless of the population distribution. The highest frequency value is the population mean.

More formally, in a simple yet general form we can write the CLT as:

N.B. The standard deviation of the distribution decreases by (√n). For example, if we randomly selected n = 100 values from a distribution, the uncertainty changes by 1/10. As n gets larger there is a tighter range and lower variance.

Generate a crazy distribution

What follows is the Python code I used, in Jupyter notebook, to In this example, I will create and sample a crazy, definitely not normal distribution created as a mixture of normals. By the CLT the distribution of the mean of the sample should be normal.

The code in the cell below computes 2000 realizations of distribution from a mixture of arbitrarily defined normals.

import pandas as pd

import numpy as np

from matplotlib import pyplot as plt

import seaborn as sns

import statsmodels.api as sm

import scipy

%matplotlib inline x = np.concatenate([

np.random.normal(loc=0, scale=1.5, size=1000),

np.random.normal(loc=7, scale=3.3, size=1000)])

sns.kdeplot(x)

plt.show()

This arbitrary new distribution is far from normal. I affectionately refer to it as the “Igor” distribution due to the pronounced shoulder or hump at 6.

Sample the distribution and examine sample means

The code in the cell below computes a list of 500 sample means (N) from samples of size 50 (n) from the population distribution (Igor). A histogram and Q-Q plot are generated from the results. I execute this code:

x_means = np.array([

np.random.choice(x, size=50, replace=True).mean()

for i in range(500)]) break_format = np.linspace(x_means.min(), x_means.max(), num=50)

plt.hist(x_means, bins=break_format)

plt.show()

sm.qqplot(x_means, line=’s’)

plt.show()

The mean of this random sample is approximately that of the population (the Igor distribution). The mean is the most frequent value. Eyeballing it, the mean appears to be around 3.3. The summary stats will be computed shortly.

Notice how the mean values themselves will form a normal distribution, regardless of the shape of the parent distribution! As predicted, the distribution of the sample means is very close to normal, even though the parent population is not.

This Quantile-Quantile (QQ) plot confirms my suspicion: that this new distribution is normally distributed.

Compute summary statistics

Next, I run the code in the cell below to compute some summary statistics and examine the results:

pop_mean = x_means.mean()

pop_std = x_means.std()

print(‘Estimated mean (%.3f) vs actual mean (%.3f)’ % (pop_mean, x.mean()))

print(‘Estimated std (%.3f) vs actual std (%.3f)’ % (pop_std, x.std())) Estimated mean (3.378) vs actual mean (3.391)

Estimated std (0.607) vs actual std (4.354)

Confidence Intervals

CI’s are a type of interval estimate, computed from the statistics of the observed data, that MIGHT CONTAIN the true value of an unknown population parameter. The interval has associated confidence level that, loosely speaking, quantifies the level of confidence that the parameter lies in the interval.

To create confidence intervals for population means, we use the central limit theorem and create confidence intervals based on the normal distribution. For 95% certainty of the true population mean, look two standard deviations to the left and 2 standard deviations to the right of the mean. Within that region there is a 95% chance the ACTUAL MEAN is in there.

Repeatedly sample from the population.

Calculate the mean for each sample.

Use the average of the sample means as the population estimate and create a CI based on the std of the sample means.

Confidence intervals are a way to express uncertainty in population parameters, as estimated by the sample. However, I cannot say:

“There is a 95% chance that the mean of another sample will be in this range.”

“95% of the sample values are in this range.”

The above claims are incorrect and are erroneously inferred.

From the CLT and normal distribution of means I can create a CI. I will run the code in the cell below to compute the CI for this normal distribution:

alpha = 0.95

half_width = scipy.stats.norm.ppf((1 + alpha)/2, loc=pop_mean, scale=pop_std) — pop_mean

print(‘The half-width is %.2f’ % half_width) ci_low = pop_mean — half_width

ci_high = pop_mean + half_width print(‘The actual (population) mean is %.2f’ % x.mean())

print(‘The %.2f level CI is (%.2f , %.2f)’ % (alpha, ci_low, ci_high)) The half-width is 1.19

The actual mean is 3.39

The 0.95 level CI is (2.19 , 4.57)

What I can say, with 95% confidence (an arbitrary CI level), is that the population mean lies within the interval between (2.19–4.57).

Uniform Distribution

Now the cynical might say that the synthetic example above was the sum of two scaled normal distributions and that the CLT creating or “decomposing” a normal distribution is not a surprise. Fair enough, but what happens if the population sampled is a uniform distribution: where all values have equal frequencies/probabilities? The code in the cell below computes a population from a uniform distribution. I execute this code and examine the density plot:

x = np.random.uniform(size=1000000)

sns.kdeplot(x)

plt.show()

This is super duper not anything like a normal Gaussian distribution. And yet if we take sample means, and then plot the distribution of those means the CLT tells us that a Normal distribution will emerge!

x_means = np.array([

np.random.choice(x, size=50, replace=True).mean()

for i in range(500)]) break_format = np.linspace(x_means.min(), x_means.max(), num=50)

plt.hist(x_means, bins=break_format)

plt.show()

sm.qqplot(x_means, line=’s’)

plt.show() pop_mean = x_means.mean()

pop_std = x_means.std()

print(‘Estimated mean (%.3f) vs actual mean (%.3f)’ % (pop_mean, x.mean()))

print(‘Estimated std (%.3f) vs actual std (%.3f)’ % (pop_std, x.std()))

Estimated mean (0.497) vs actual mean (0.500)

Estimated std (0.040) vs actual std (0.289)

Even sampling a population created from a uniform distribution, the CLT tells us a Gaussian distribution will emerge, and indeed one does. The artifacts from binning distort the actual shape, which becomes clear with a KDE plot.

sns.kdeplot(x_means)

With KDE smoothing, the algorithm is looking at the immediate neighbors for each bin and calculating the average. I was a little unconvinced before but after KDE smoothing the distribution I can see that yes, this is definitely close to a normal distribution.

N.B. The varying distribution is due to np.random being “randomly” initiated. Remember this when generating train/tune/test splits! To counteract this and generate repeatable behavior of the random function, generalize with a seed.

The QQ plot confirms that this is a normal distribution. The fact the tails in the upper right and lower left deviate from the line is an expected artifact. This new distribution is Gaussian in shape, ergo I can calculate the CI’s from it:

alpha = 0.95

half_width = scipy.stats.norm.ppf((1 + alpha)/2, loc=pop_mean, scale=pop_std) — pop_mean

print(‘The half-width is %.2f’ % half_width) ci_low = pop_mean — half_width

ci_high = pop_mean + half_width print(‘The actual (population) mean is %.2f’ % x.mean())

print(‘The %.2f level CI is (%.2f , %.2f)’ % (alpha, ci_low, ci_high)) The half-width is 0.08

The actual (population) mean is 0.50

The 0.95 level CI is (0.42 , 0.58)

With 95% confidence, I can say that the true mean of the population is in the interval (0.42 — 0.58).

Conclusion

Something as simple as the CLT is so important to any ML model — without it we could not easily calculate the summary statistics needed to compare model performances, or get probabilities and certainties. I demonstrated how, regardless of the population — the CLT states that the distribution of the means sampled will form a Normal distribution. Relating back to n = 30 and AP Stats in high school, that is a good heuristic for the size of a sample. The sample needs to be “large enough” to overwhelm any oddities that can come from random chance in sampling, essentially the same logic as the Law of Large Numbers. Hopefully you found this a useful refresher on the CLT and why it is important for Data Scientists, Statisticians, and Machine Learning Engineers alike to understand this simple theorem’s profound and outsized importance.