Illustration with Python: Central Limit Theorem
By: Chaya Chaipitakporn
Tags: Central Limit Theorem, Probability, Data Science, Matplotlib, Python

This blog uses knowledge of Chebyshev’s inequality and weak law of large numbers, you can review those topics by visiting on the links.

The theorem states that the distribution of independent sample means is an approximately normal distribution, even if the population is not normally distributed. In other words, if we independently sample from population many times and plot a mean of each sampling the plot will be a normal distribution, regardless of the population distribution.

I will use python to demonstrate the theorem with the following steps

Step:

1.) Create a population of 1,000,000 values, I use a gamma distribution with shape = 2 and scale = 2 to show that theorem work with non-normal distribution

# build gamma distribution as population

shape, scale = 2., 2. # mean=4, std=2*sqrt(2)

s = np.random.gamma(shape, scale, 1000000)

2.) Sample from the gamma distribution with 500 sample size, calculate the mean and repeat the step 1,000 times (this is a number of sampling). I repeat this step but increase the number of sampling until the number is 50,000 times.

## sample from population with different number of sampling

# a list of sample mean

meansample = []

# number of sample

numofsample = [1000,2500,5000,10000,25000,50000]

# sample size

samplesize = 500

# for each number of sampling (1000 to 50000)

for i in numofsample:

# collect mean of each sample

eachmeansample = []

# for each sampling

for j in range(0,i):

# sampling 500 sample from population

rc = random.choices(s, k=samplesize)

# collect mean of each sample

eachmeansample.append(sum(rc)/len(rc))

# add mean of each sampling to the list

meansample.append(eachmeansample)

3.) Plot each sample mean.

# plot

cols = 2

rows = 3

fig, ax = plt.subplots(rows, cols, figsize=(20,15))

n = 0

for i in range(0, rows):

for j in range(0, cols):

ax[i, j].hist(meansample[n], 200, density=True)

ax[i, j].set_title(label="number of sampling :" + str(numofsample[n]))

n += 1

We can see from the plots that as the number of sampling increases, the distribution becomes smoother. This theorem is extremely powerful because we can apply to any population, so if we have tools to work with normal distribution, we can use that tool with the sample mean of any distribution such as calculate probability using an area under a normal curve.

Standardize the Sample Mean

We can change sample mean distribution into standard normal distribution by subtracting each sample mean with an expected value and dividing by a standard deviation.

Step:

1.) Using the distribution from the last sampling

# use last sampling

sm = meansample[len(meansample)-1]

2.) Calculate the mean and standard deviation of the sample mean.

# calculate start deviation

std = np.std(sm)

# set population mean

mean = np.mean(sm)

3.) Subtract each value by mean and divide it by standard deviation, so the mean and standard deviation of the sample mean is 0, 1 respectively.

# list of standarded sample

zn = []

# for each sample subtract with mean and devided by standard deviation

for i in sm:

zn.append((i-mean)/std)

4.) Plot the result.

# plot hist

plt.figure(figsize=(20,10))

plt.hist(zn, 200, density=True)

# compare with standard normal disrtibution line

mu = 0

sigma = 1

x = np.linspace(mu - 3*sigma, mu + 3*sigma, 100)

# draw standard normal disrtibution line

plt.plot(x, stats.norm.pdf(x, mu, sigma),linewidth = 5, color='red')

plt.show()

the red line is standard normal distribution line, the mean of the blue distribution is 0 with a unit standard deviation

One of the reasons we standardize the sample mean is the complexity of a normal distribution function. We have to integrate the complicated function which can take hours to do, so instead, we standardize the distribution and use the Z table to find an area under the function.

Sample size

The rule of thumb of sample size is that it should be larger than 30 to make the sample mean distributed normally. However, the theorem still works if the sample size is less than 30 but the population is normally distributed. I will illustrate what will happen if the sample size is less than 30, 30 and greater than 30.

Step:

1.) Sample from the same gamma distribution with 1 sample size, calculate the mean and repeat the step 25,000 times. I repeat this step but increase the sample size until it reaches 1,000 sample size.

## sample with different sample size

# list of sample mean

meansample = []

# number of sampling

numofsample = 25000

# sample size

samplesize = [1,5,10,30,100,1000]

# for each sample size (1 to 1000)

for i in samplesize:

# collect mean of each sample

eachmeansample = []

# for each sampling

for j in range(0,numofsample):

# sampling i sample from population

rc = random.choices(s, k=i)

# collect mean of each sample

eachmeansample.append(sum(rc)/len(rc))

# add mean of each sampling to the list

meansample.append(eachmeansample)

2.) Plot each sample mean.

# plot

cols = 2

rows = 3

fig, ax = plt.subplots(rows, cols, figsize=(20,15))

n = 0

for i in range(0, rows):

for j in range(0, cols):

ax[i, j].hist(meansample[n], 200, density=True)

ax[i, j].set_title(label="sample size :" + str(samplesize[n]))

n += 1

From the plot, the distribution of sample size that is less than 30 is not normally distributed.

I will combine this theorem with Chebyshev’s inequality and the weak law of large numbers, but before we go there, let’s look at the expected value and standard deviation of sample means.

The expected value and standard deviation of sample means

Suppose that X is a random variable that is independent and identical distributed with the expected value μ and standard deviation σ. If we sample the X n sample, the expectation and variance of X will be as follow.