Why is the Normal Distribution so Normal?
By: Ravi Charan
Tags: Mathematics, Central Limit Theorem, Towards Data Science, Data Science, Statistics

Why is the Normal Distribution so Normal?

Beyond the Central Limit Theorem

The Normal Distribution (or a Gaussian) shows up widely in statistics as a result of the Central Limit Theorem. Specifically, the Central Limit Theorem says that (in most common scenarios besides the stock market) anytime “a bunch of things are added up,” a normal distribution is going to result.

But why? Why that distribution? Why is it special? Why not some other distribution? Are there other statistical distributions where this happens?

Teaser: the answer is yes, there are other distributions that are special in the same way as the Normal distribution. The Normal distribution is still the most special because:

It requires the least math

It is the most common in real-world situations with the notable exception of the stock market

If you’re intrigued, read on! I’ll give an intuitive sketch of the Central Limit Theorem and a quick proof-sketch before diving into the Normal distribution’s oft-forgotten cousins.

The Central Limit Theorem

Here is a quick official statement:

Suppose you have n random variables X₁, X₂, … etc. representing a sample of size n from some population with population mean μ and finite variance σ². The population could follow any distribution at all.

We are interested in their mean X, which itself a random variable. (It is random because each time we take a sample of size n, we get a different result).

We already know that the mean X will have mean μ and variance σ²/n (this is true by the independence assumption and is a general property of random variables).

The central limit theorem says that when n is large (usually 40+ is close enough in real life) the mean X follows a normal distribution, no matter what the distribution of underlying population is.

Formally,

Formal statement of the Central Limit Theorem

Where Φ represents the normal distribution with mean and variance as given. (You may be used to seeing the equivalent standard deviation σ/√n instead). The “in distribution” is a technical bit about how the convergence works. We’ll ignore such technicalities from here on out.

Why the Central Limit Theorem Shows Up

The Central Limit Theorem shows up in all sorts of places in real-world situations. For example, it’s a pretty reasonable assumption that your height can be expressed as the sum of a bunch of factors related to, among others:

How much milk you drank every day when you were 8 years old How many X and/or Y chromosomes you have Which variant of the GH1 gene you have A whole bunch of other genes Whether you slept in a Procrustean bed as a child

Take a whole bunch of factors, each of which makes a small difference in your final (adult) height, and, presto, you end up with a (roughly) normal distribution for human heights!

Note that I cheated slightly – the variables here aren’t i.i.d. But the independence assumption is a reasonable approximation, and there are stronger versions of the central limit theorem that relax the identical-distribution hypothesis. We did choose to leave out the cases of extreme genetic conditions that affect height however.

So in sum, any time something you measure is made up of a whole bunch of contributions from smaller parts being added up, you are likely to end up with a normal distribution.

A Quick Proof

This proof is necessarily a sketch because, well, if you want a full proof with all of the analysis and probability theory involved, go read a textbook. The main point I want to get across is that there is a reason Euler’s constant e shows up.

First of all, we will need one high-powered mathematical tool. To every reasonable random variable X there is a characteristic function φ which is, in essence, the Fourier Transform of the Probability-Density Function (PDF) of X.

Characteristic Function of a Random Variable X

Why did we take our perfectly good, real-valued, random variable and start throwing in complex numbers and just generally making it more complicated? Well, for some reason no one has ever satisfactorily explained to me, things in math tend to come with two ways of looking at them (“duality”). In physics too: space/momentum, time/frequency. And for whatever reason, it is sometimes much simpler to look at the other side.

The Characteristic Function has two nice properties we will need. Suppose A and B are independent random variables and c is a scalar constant. Then we want to know the characteristic function for the sum variable A +B and the product cA. You should be able to derive these with a pencil and paper pretty quickly from the definition:

Convenient properties of the Characteristic Function

The derivatives of φ at t=0 also encode valuable information. After we set t=0, the exponential parts go away and we get a simple enough expression to evaluate. (Again, I promise you you can do this yourself in 2 minutes with pencil and paper. Just use the definition and do what seems right).

Derivatives of the Characteristic Function at 0 assuming finite mean and variance

Now we are ready to get started. Recall our X₁, X₂, … etc., iid random variables each with mean μ and finite variance σ². Lets shift them so we get

and define

Please convince yourself that if we show that the sum Y is normally distributed with mean 0 and variance σ², then we have shown that the mean X is normally distributed with mean μ and variance σ²/n, which is what we want.

Now we just need to compute the characteristic function

Characteristic Function for the Sum Y

where we have used the characteristic-function properties for the sum and scalar multiple of a random variable from above. The Y’s are iid so they all have the same characteristic function: repeated multiplication turns into exponentiation.

Now we Taylor expand φ to second order

Taylor expansion of φ

Take the limit as n → ∞. You know this one! It’s the compound interest formula (Pert!) with the interest rate r = –σ²t²/2

The Compound Interest Formula

The result we have arrived at is in fact the characteristic function for a normal distribution with mean 0 and variance σ². By the Lévy Continuity Theorem, we are done. And we can see why that sneaky Euler’s constant e shows up!

Beyond the Central Limit Theorem

Okay, the whole point of this was to find out why the Normal distribution is so normal. All we’ve done is prove the Central Limit Theorem without really getting a deeper explanation for why the Normal distribution is the result. Time for the payoff.

We saw that the Central Limit Theorem is so common because it tells us what happens when we sum/average a bunch of stuff. So it is natural to ask, what distributions can arise as the sum/average of a bunch of stuff? Let’s think of these distributions as “cousins” of the Normal (Gaussian) distribution.

Infinitely Divisible Distributions

A distribution Y is infinitely divisible if it can be expressed as the sum of arbitrarily many iid distributions X. Formally, for every n, we can find a distribution X such that:

The Poisson distribution, Student’s t-distribution, and the Gamma distribution are infinitely divisible – as are Gaussians and the distributions we will see below.

Stable Distributions

We want to ask for a stronger condition however, before we deem a distribution a “cousin” of the Normal distribution. Suppose we have a distribution X and iid copies of it X₁ and X₂. We take a linear combination of those copies and see if we end up with the same thing we started with except possible scaled by a constant c and shifted by a constant d. If we can always do this, we say X is stable.

In the context of infinitely divisible distributions above, this means that we can write Y as the sum of n copies of itself (suitably scaled and/or shifted). Roughly speaking, with an unstable distribution like the Poisson distribution, adding more and more of them will get you closer and closer to a stable distribution. So it is the stable distributions that we will be interested in.

The Levy α-Stable Distributions

The α-stable distributions are the only stable distributions. They were studied by the Frenchman Paul Lévy as early as the 1920s. Their description is the heart of the Generalized Central Limit Theorem given by the Russians Gnedenko and Kolmogorov in the 1950s. In turn they provide the proper perspective on the nature of the Normal Distribution.

Each α-stable distribution is characterized by 4 parameters. This is a lot of parameters.¹

Location (in essence the mean) Scale c (related to the standard deviation) Tail Size, α. Between 0 and 2 including 2 but not 0. Left/right Tail Ratio, β. Between -1 and 1 inclusive.

The first two parameters are pretty reasonable: location and scale. In the context of the normal distribution, the location is the mean μ and the scale may be taken to be the standard deviation σ.

Comparison of different tail sizes (Wikipedia). α=2 is the Normal distribution.

The third parameter, α (hence the name of the family of distributions) controls the size of the tails. The smallest tails belong to the Normal distribution with kurtosis 3 (and excess kurtosis 0). For all the other stable distributions, the tails are fatter (“leptokurtic”) with kurtosis at least 9. You are familiar with the idea of “fat tails” from books like The Black Swan. In short, if you think things are normally distributed but they in fact have larger tails that you think, you will see a surprising number of “freak” occurrences like stock-market crashes (which, of course, are quite common). We will return to this parameter in a minute.

Comparison of Different “skewness” settings (Wikipedia)

The last parameter is a bit boring. It controls the ratio of the size of the right tail to the left tails (intuitively, the skew). In the diagram you can see that for β=1 the distribution is only supported for non-negative numbers.

In a sense I have given the parameters out of order. They control the 1st (location), 2nd (scale), 3rd (“skewness”), and 4th (“kurtosis”) moments. Hopefully this gives you an intuitive feel for each of them. To conclude, we are going to look at the all-important 4th parameter, α.

The Pareto Distribution

In order to understand tail size, we should look at the Pareto distributions. The Pareto distribution has a PDF that is non-zero only for x≥1 and comes with a parameter α > 0 which plays the same role as in the Lévy distributions.

PDF for the Pareto Distribution, with α > 0

The Pareto distribution was first studied by the Economist Italian Vilfredo Pareto in the context of the distribution of wealth in the late 1800s. It is the genesis of the 80/20 rule (20% of the population has 80% of the wealth²; 20% of the work accomplishes 80% of the result). This was also, apparently, an inspiration for the Fascism of Mussolini, who attended his lectures.

Some crucial points to note

For α = 0 (or less), the distribution wouldn’t be normalized: the total probability wouldn’t add up to 1 (it would add up to ∞). Hence the constraint on α.

For α ≤ 1, the distribution doesn’t even have a finite mean! Its expected value E(X) is ∞.

For α ≤ 2, the distribution doesn’t have finite variance! Its variance is ∞.

The Pareto distribution always has fatter tails than the normal distribution, as measured by the kurtosis. In the limit α→∞, the kurtosis is 6 more than the Normal distribution’s.

As α gets bigger, the tails get smaller (but stay larger than a Gaussian). If the tails are too large, the distribution can have infinite mean and variance!

Okay? If you understand that the Pareto distribution is the prototypical “fat tailed” distribution with a single parameter controlling the “largeness” of the tails, then you understand the role of α in the Lévy α-stable distributions.

Relationship between Pareto and Lévy distributions

To put everything together, you may have noticed that I sneakily haven’t yet explicitly said why the Central Limit Theorem doesn’t apply to the α-stable distributions.

In fact, the Central Limit Theorem requires our distribution have finite variance σ². But for distributions with infinite variance, the Central Limit Theorem won’t apply. Instead, if you average a bunch of samples, you will get an α-stable distribution with α < 2. We’ll give some example below.

This is also why I talked about “location” and “scale” parameters. Your intuition for what the mean and variance do apply perfectly well, but not all of our distributions actually have a well-defined mean or variance.

To recap/clarify, suppose we have an underlying distribution X and we sample with size n and take the mean. We are wondering what distribution the sample mean will follow in the limit where n is large.

If the distribution‘s tails looks like a Pareto distribution α < 2 after we go far enough, we will get an α-stable distribution with the same α. If the left tail and right tail have different α-parameters, the largest one wins and the smallest one dies: we end up with an α-stable distribution that only has a left or right tail (β = ±1). Otherwise β just sets the ratio between the two tails.

If the distribution’s tails look like a Pareto distribution with α > 2 or else they fall off faster than any polynomial (like, say exponentially fast as in a Gaussian), then you will end up with a Normal distribution.

The case α = 2 is special. Even though the underlying distribution has infinite variance, the resulting distribution will still be a Gaussian. For this reason, the Lévy distribution with α=2 is the Normal (Gaussian) distribution.

The case α=2 is also special because the Normal distribution can’t have any Skewness (so β=0) and the tail size is fixed (kurtosis = 3). For this reason, there are no α-stable distributions with α > 2. All such hypothetical distributions are instead “captured” by the Central Limit Theorem and turned into Gaussians when we sample and average.

The α-stable distributions are important because they provide the version of the Central Limit Theorem in the case of “fat” tails. As alluded to above, stock-market returns are believed to follow an α-stable distribution with α≠2, providing a real-life example of a situation where you can lose your shirt by making the assumption something is normally distributed.

Conclusion

Interestingly, a closed form probability-density function (PDF) isn’t known except for a few special cases. It is only possible to write down the characteristic function φ in general. It is quite ugly, so I’m not going to bother. However, it is possible to sample from and numerically compute the pdfs, and you’ll find an implementation in any good statistical package (like scipy).

Let’s close with an example. If you generate standard random normal variables A and B and consider their ratio A/B, you will get a Cauchy Distribution with α=1. As an exercise, you can generate n numbers with n large and fit both a normal distribution and a Lévy-stable distribution to the resulting distribution of sample means. You should see that the Normal distribution doesn’t fit (the tails are too fat) and the Lévy distribution fits with α close to 1. What happens if you change the means and standard deviations of A and B?