The relationship between the Law of Large Numbers & Central Limit Theorem
By: Dina Jankovic
Tags: Probability, Data Science, Normal Distribution, Statistics

The relationship between the Law of Large Numbers & Central Limit Theorem

Saint Petersburg, Russia. Image source: Unsplash (photo by Elina Sitnikova — thanks for this lovely shot! ❤)

In this article, I’ll talk about two important mathematical laws that every Data Science professional should have under their belt. I like explaining abstract concepts in a simple way, so even if you don’t have a particularly strong Math background, this post will be a down-to-earth approach.

First, let’s start from the Law of Large Numbers (LLN), and then we’ll move on to the Central Limit Theorem (CLT). Once you fully grasp the intuition behind LLN, the CLT will be easier to understand.

The Law of Large Numbers — Simply Explained

Let’s start from two simple examples. This is the best way to understand abstract concepts.

Example #1: The Coin Toss 🪙

Imagine a situation where you toss a fair coin twice. What would you expect to happen? You’d probably expect to get 1 head & 1 tail. However, things may go a little differently and you may get 2 heads or 2 tails. The latter 2 outcomes are less likely, but you probably won’t be too surprised if that happens. And why would you? It’s just two coins :)

Now, if you toss that same coin 20 times, you’d expect around 10 tails and 10 heads. This is all under the assumption that the coin is fair. Again, the actual results may be different, so you may end up with, for instance, 8 tails and 12 heads. Alternatively, you may end up with 1 tail and 19 heads, although this is a highly unlikely scenario. And if this ever happens, you should suspect the fairness of the coin.

Let’s now imagine tossing the same coin 1000 times. How many tails would you expect to get? Likely around 500. Maybe 450, maybe 510 —but probably very close to 500. You may end up with 10 heads and 990 tails, but this is quite improbable. You’d be quite surprised if this happens.

So the more we toss the same fair coin, the more likely it is that the actual ratio of heads : tails will be close to the expected ratio of 50–50. This is perfectly intuitive. And again, this is all under the assumption that the coin is fair. Otherwise, if you had a biased coin with 90% chances of getting a tail, you’d expect to get a tail most of the times when tossing the coin.

So what does all of this mean to us?

The reality will meet the expectations more closely when we keep on increasing the number of times we run an experiment.

Let’s consider another example.

Example #2: The Dice Roll 🎲

Now let’s grab a fair dice instead of a coin, and roll it.

If we roll a dice many, many times, the expected average of these rolls is going to be 3.5, i.e. (1 + 2 + … + 6) / 6. The more we roll the dice, the more stable this expectation is, and the average won’t deviate much from 3.5. At the very beginning, the average outcome will be quite unstable (see the graph below), but later on it will stabilize around 3.5.

LLN in Action: The more we roll a dice, the closer the average outcome is to 3.5. Image author: Dina Jankovic (me)

Now we understand what happens when we repeat an experiment more and more times. The reality begins to meet the expectations. In other words, the actual average closely resembles the expected average.

To formalize this concept in a more mathematical way, let X₁, X₂, … be an finite or infinite sequence of random variables, each with the same expectation mu: E(X₁) = E(X₂) = … = 𝜇. In the Coin Toss example, this condition ensures that the coin tosses are fair, with the same expected values.

According to the LLN, the sample average X̄= (X₁ + X₂ + …) / n converges to 𝜇 as n converges to infinity (i.e. as the number of trials keeps increasing).

Now you have an intuition of what the Law of Large Numbers means to us. It has numerous real-life applications — e.g. a casino may lose money from a single player that got lucky, but in the long run, it will gain money from many other players who were unfortunate to lose. Ultimately, the statisticians working at this casino are actually fully aware of the LLN — they intentionally optimize the game rules so that over the long run, the wins and losses will cluster around a predictable ratio. This way the casino’s earnings are maximized.

Expanding into the Central Limit Theorem lane

OK, we clarified what LLN is. This concept can now be further extended to another law called the Central Limit Theorem (CLT).

Remember the outcome average X̄? What is X̄ essentially? It’s a number. But that number varies depending on the sample size and a bit of randomness. Thus, X̄ can be perceived as a random variable. And just like any random variable, X̄ has a special place in this world — it has a distribution, a mean, and a variance.

But it has a very special distribution. X̄ is normally distributed, with a mean of 𝜇 and a standard deviation of 𝜎. If we subtract 𝜇 from X̄ and divide this by sigma, then Z = (X̄ — 𝜇)/ 𝜎 has a standard normal distribution with a mean of 0 and standard deviation of 1.

If we apply the CLT to the Coin Toss example, we learn that the the average number of tails or heads would follow a normal distribution. To show you what this looks like in action, check out the simulations below. I simulated a Coin Toss with a small and large n, and got an almost perfect bell curve when increasing n.

A perfectly shaped Normal Distribution, with a centre at 3.5. All plots are my own.

As an exercise, I suggest you to write the code to generate the LLN & CLT plots I shared. Feel free to play around with small & large sample sizes to see what happens.

Fun fact: according to the 3-𝜎 rule, we can expect X̄ to be within 2 standard deviations of the expected average 𝜇, 95% of the time.