Seven Must-Know Statistical Distributions and Their Simulations for Data Science | by Zijing Zhu | Towards Data Science

A statistical distribution is a parameterized mathematical function that gives the probabilities of different outcomes for a random variable. There are discrete and continuous distributions depending on the random value it models. This article will introduce the seven most important statistical distributions, show their Python simulations with either the Numpy library embedded functions or with a random variable generator, discuss the relationships among different distributions and their applications in data science.
Bernoulli distribution is a discrete distribution. The assumptions of Bernoulli distribution include:
1, only two outcomes;
2, only one trial.
Bernoulli distribution describes a random variable that only contains two outcomes. For example, when tossing a coin one time, you can only get “Head” or “Tail.” We can also generalize it by defining the outcomes as “success” and “failure.” If I assume that when I toss a die, I only care if I get six, I can define the outcome of a die showing six as “success” and all other outcomes as “failure.” Even though tossing a die has six outcomes, in this experiment that I define, there are only two outcomes, and I can use Bernoulli distribution.
The probability mass function (PMF) of a random variable x that follows the Bernoulli distribution is:
p is the probability that this random variable x equals ‘success,’ which is defined based on different scenarios. Sometimes we have p = 1-p, like when tossing a fair coin.
From the PMF, we can calculate the expected value and variance of random variable x depending on the numerical value of x. If x=1 when “success” and x=0 when “failure,” E (x) and Var (x) are:
Simulating a Bernoulli trial is straightforward by defining a random variable that only generates two outcomes with a certain “success” probability p:
import numpy as np#success probability is the same as failure probabilitynp.random.choice([‘success’,’failure’], p=(0.5, 0.5))#probabilities are differentnp.random.choice(['success','failure'], p=(0.9, 0.1))
Binomial distribution is also a discrete distribution, and it describes the random variable x as the number of success in n Bernoulli trials. You can think of the binomial distribution as the outcome distribution of n identical Bernoulli distributed random variables. The assumptions of the Binomial distribution are:
1, each trial only has two outcomes (like tossing a coin);
2, there are n identical trials in total (tossing the same coin for n times);
3, each trial is independent of other trials (getting “Head” at the first trial wouldn’t affect the chance of getting “Head” at the second trial);
4, p, and 1-p are the same for all trials (the chance of getting “Head” is the same across all trials);
There are two parameters in the distribution, the success probability p and the number of trials n. The PMF is defined using the combination formula:
The probability that we have x number of success out of n trials is like choosing x out of n when order doesn’t matter.
Thinking about Binomial distribution as n identical Bernoulli distributions helps understand the calculation of its expected value and variance:
If you are interested in getting these two equations above, you can watch these wonderful videos from Khan Academy.
Python’s Numpy library has a built-in Binomial distribution function. To simulate it, define n and p, and set to simulate 1000 times:
n = 100p = 0.5size = 1000binomial = np.random.binomial(n,p,size)plt.hist(binomial)
We can get the histogram:
When setting n equals to 1, we can simulate the Bernoulli distribution:
n = 1p = 0.5size = 10000bernoulli = np.random.binomial(n,p,size)plt.hist(bernoulli)
Geometric distribution is a discrete distribution that models the number of failures (x failures) before the first success in repeated, independent Bernoulli trials. For example, the random variable can be how many “Tails” would you get before you get your first “Head.” It can also model the number of trials to get the first success (x-1 failures), like how many times you have to toss until you get the first “Head.” The only difference between these two random variables is the number of failures. The Geometric distribution assumptions are the same as the Binomial distribution because they both derive from some identical independent Bernoulli trails.
When the random variable x is the number of failures before the first success, the PMF is:
the expected value and variance are:
When the random variable x is the number of trials to get the first success, the PMF is:
the expected value and variance are:
We need to use the geometric series to derive the expected value and variance for Geometric distribution. There is a great tutorial at Khan Academy that explains the details.
To simulate Geometric distribution, we can use the Bernoulli trials and count for the number of failures before the first success, and then plot the number of failures (Many thanks to Tiffany Sung who caught the typo in this code snippet):
geometric = []failure = 0n=0p=0.5while n<10000:    result = np.random.choice(['success','failure'],p=(p,1-p))    if result == 'failure':        failure+=1    else:        geometric.append(failure)        failure = 0        n+=1plt.hist(geometric)plt.axvline(np.mean(geometric),color='red')
Uniform distribution models a random variable whose outcomes are equally likely to happen. The outcomes can be discrete, like the outcomes getting form tossing a die, or continuous, like the waiting time for a bus to arrive. Thus Uniform distribution can be a discrete or continuous distribution depending on the random variable. The assumptions are:
1, there are n outcomes (discrete), or a range for the outcomes to be at (continuous);
2, All values in the outcome set or the range are equally likely to occur.
The discrete uniform distribution is straightforward, and it is easy to calculate the expected values and variance. For a continuous Uniform distribution that is uniformly distributed at [a, b], the probability density function (PDF) is:
Using integrals, the expected value and variance are:
To simulate Uniform distribution, we can use the Numpy’s embedded function and specifying the range of the distribution. Based on a Uniform distribution at [0,1], we can generate the Uniform distribution at [a, b]:
#generate a random variable follows U(0,1)np.random.uniform(0,1,size=10000)#use U(0,1) to generate U(a,b)def uniform(a,b):    return a + (b-a) * np.random.uniform(0,1,size=10000)
Normal (Gaussian) distribution is the most widely used continuous distribution because it represents the most universe situations. Many random variables are normally distributed because of the Central Limit Theory, or they are assumed to be normally distributed before fitting them into a statistical model. Normal distribution has some unique characteristics:
1, mean=mode=median=μ ;
2, the PDF is bell-shaped and symmetric at x=μ;
3, the values between [μ-σ, μ+σ] takes roughly 68% of the data, where σ is the standard deviation, and:
the PDF for Normal distribution is:
When μ=0 and σ=1, we have a Standard Normal distribution, where the PDF is simplified as:
The expected value and variance are embedded in the PDF of Normal distribution. The expected value is the mean, which is μ, and the variance is the square of standard deviation, σ2.
To simulate the Normal distribution, we can use the Numpy function:
np.random.normal(mu, sigma, 1000)
Or we can also use Central Limit Theory to simulate Normal distribution:
def clm(N,n):#generate a sample from any random population N    lis = np.random.random(size=N)    means = []#take a random sub sample with size n from N    for i in range(n):        lis_index = np.random.randint(0,N,n)        means.append(lis[lis_index].mean())        i+=1#plot means    return plt.hist(means)clm(10000,1000)
Poisson distribution is a discrete distribution that models the probability of a number of events occurring in a fixed interval of time or space. Poisson distribution can be used to present the number of customers arriving in a store in an hour, or the number of phone calls a company receives one day, etc. Poisson distribution is closely related to binomial distribution if you measure the number of event occurrences as the number of success. For example, when measuring how many cars will pass a particular street in an hour, the number of cars passing is a random variable that follows Poisson distribution. To understand the car passing event, you can break down one hour into 60 minutes, and see how many cars will pass in a minute, then generalize it into an hour. For a minute, maybe more than one cars pass the street, thus it is not a binary random variable. However, if we break down one minute into 60 seconds, it is likely that only one car passes or no car pass in a single second. We can keep breaking down a second to make a more confident claim. We can then consider car passing as a successful event and no car passing as a failure event, and model how many success events (how many cars passing) in 3600 trials (3600 seconds in an hour), which is a Binomial distribution with success probability p, and the number of trials equals to 3600.
Connecting Poisson distribution with binomial distribution helps us understand the assumptions and PMF of Poisson distribution. The assumptions Poisson distribution are:
1, any successful event should not influence the outcome of other successful events (observing one car at the first second doesn’t affect the chance of observing another car the next second);
2, the probability of success p, is the same across all intervals (there is no difference between this hour with other hours to observe cars passing by);
3, the probability of success p in an interval goes to zero as the interval gets smaller (if we are discussing how many cars will pass in a millisecond, the probability is close to zero because the time is too short);
The PMF of Poisson distribution can be derived from the PMF of binomial distribution:
We know x is the number of success in n Bernoulli trials, and E(x)=np. If we define λ =E(x) =np as the average number of success in n Bernoulli trials, the success probability p can be estimated by λ/n. As n goes to infinity, the PMF becomes:
After some calculation, we can get a new PMF, which is the PMF for Poisson distribution:
The Poisson distribution gives the probability of observing a certain number of events in an interval, given the average number of events in the interval. The expected value and variance of a random variable that follows Poisson distribution are both λ:
To simulate Poisson distribution, we can use the Numpy function:
lam = 1poi = np.random.poisson(lam=lam,size=1000)plt.hist(poi)plt.axvline(x=lam,c='red')
The Exponential distribution is a continuous distribution that is closely related to the Poisson distribution. It is the probability distribution of the time intervals between Poisson events. If the number of calls a company receives in an hour follows Poisson distribution, then the time interval between calls follows Exponential distribution. Since exponential distribution is closely related to Poisson distribution, its assumptions follow the Poisson distribution’s assumptions.
We can derive the PDF of exponential distribution by first deriving its Cumulative Distribution Function (CDF) and then take the derivative of the CDF. We are using this trick because we can get the CDF using Poisson distribution. Suppose we are still observing how many cars are passing in the same street, and now we care about the random variable ω, which is defined as when we see one car passing, it takes at least ω minutes to observe another car passing. Let’s start from the beginning — how long it takes to observe the first car? Assume it takes ω minutes to observe the first car in ω minutes, there is zero car passing. We know that the number of cars passing in the street in a minute follows Poisson distribution:
For ω minutes, the number of cars passing in the street follows:
the probability that we observe zero cars in ω minutes is:
From exponential distribution’s perspective, we now already know the probability of taking at least ω minutes to observe the first car, then the probability of taking less than ω minutes is:
This is the CDF of the random variable ω, taking the derivative with respect to ω, we have the PDF:
If we know in a minute, on average, we are likely to observe three cars (λ=3) passing by the street, then it is expected that every 1/3 minutes, we will observe one car passing by. Thus, the expected value and variance of exponential distribution are:
To simulate the Exponential distribution, we can use the Numpy random function:
lam = 2exp = np.random.exponential(lam, 10000) plt.hist(exp)plt.axvline(x=lam, c=’red’)
Or we can use the CDF and Uniform distribution from (0,1) to simulate it. The following equations show the process:
Following the code:
rate = 1/20inverse_cdf=np.random.uniform(0,1,size=1000)interval = [-np.log(1-u)/rate for u in inverse_cdf]plt.hist(interval)
As discussed partially above, these distributions are closely related to each other in different ways:
1, Bernoulli distribution is a special case of Binomial distribution when n equals one;
2, As n goes to infinity, and p goes to zero, np = λ, a finite number, binomial distribution approaches to Poisson distribution;
3, As n goes to infinity, and p are not indefinitely small, λ goes to infinity as well, binomial distribution approaches normal distribution;
4, If the total number of events in a time interval of length t follows the Poisson distribution with parameter λ, the times between random events follow an Exponential distribution with rate λ/t.
5, Geometric distribution is the only discrete distribution that has the memoryless property, and exponential distribution is the only continuous distribution with the memoryless property. The mathematical definition of memoryless property is:
P( X > a + b | X> a ) = P ( X> b )
You can read more details about this property here. One of the implications you can think of is that if the lifetime of batteries has an Exponential distribution, then a used battery is as good as a new one, as long as it’s not dead.
Understanding the assumptions, properties of different statistical distributions definitely helps data scientists with their daily tasks:
1, Monte Carlo simulations start with generating random variables from a Uniform distribution;
2, Binomial distribution sets the foundation of any binary classification model;
3, All regression models that use the least-squares cost function assume the residuals to follow a Normal distribution with mean equals to zero;
4, The transformed distributions from the Normal distribution, like the Student t distribution, Standard Normal distribution are the distributions used for conducting hypothesis testing, computing p values, and getting confidence interval; Log-normal distribution describes all samples of data when the data is right-skewed;
5, Poisson and Exponential distributions are greater for modeling events with a fixed time rate (λ). For example, we can use Poisson distribution to model how many customers will show up in a shop in a day, and use Exponential distribution to model how many time it takes between two consecutive customers to enter the shop.
Statistical distributions are everywhere in daily life. Understanding statistical distributions play a very important role for data scientists to know the data more thoroughly, conduct better data analysis, choosing the more suitable model, etc. Help this article helps. Thank you for reading.
Here is the list of all my blog posts. Check them out if you are interested!