Seven Must-Know Statistical Distributions and Their Simulations for Data Science
By: Zijing Zhu
Tags: Getting Started, Probability Distributions, Simulation, Data Science, Statistical Analysis

Different Distributions and Simulations

1, Bernoulli Distribution

Bernoulli distribution is a discrete distribution. The assumptions of Bernoulli distribution include:

1, only two outcomes;

2, only one trial.

Bernoulli distribution describes a random variable that only contains two outcomes. For example, when tossing a coin one time, you can only get “Head” or “Tail.” We can also generalize it by defining the outcomes as “success” and “failure.” If I assume that when I toss a die, I only care if I get six, I can define the outcome of a die showing six as “success” and all other outcomes as “failure.” Even though tossing a die has six outcomes, in this experiment that I define, there are only two outcomes, and I can use Bernoulli distribution.

The probability mass function (PMF) of a random variable x that follows the Bernoulli distribution is:

p is the probability that this random variable x equals ‘success,’ which is defined based on different scenarios. Sometimes we have p = 1-p, like when tossing a fair coin.

From the PMF, we can calculate the expected value and variance of random variable x depending on the numerical value of x. If x=1 when “success” and x=0 when “failure,” E (x) and Var (x) are:

Simulating a Bernoulli trial is straightforward by defining a random variable that only generates two outcomes with a certain “success” probability p:

import numpy as np #success probability is the same as failure probability

np.random.choice([‘success’,’failure’], p=(0.5, 0.5)) #probabilities are different

np.random.choice(['success','failure'], p=(0.9, 0.1))

2, Binomial Distribution

Binomial distribution is also a discrete distribution, and it describes the random variable x as the number of success in n Bernoulli trials. You can think of the binomial distribution as the outcome distribution of n identical Bernoulli distributed random variables. The assumptions of the Binomial distribution are:

1, each trial only has two outcomes (like tossing a coin);

2, there are n identical trials in total (tossing the same coin for n times);

3, each trial is independent of other trials (getting “Head” at the first trial wouldn’t affect the chance of getting “Head” at the second trial);

4, p, and 1-p are the same for all trials (the chance of getting “Head” is the same across all trials);

There are two parameters in the distribution, the success probability p and the number of trials n. The PMF is defined using the combination formula:

The probability that we have x number of success out of n trials is like choosing x out of n when order doesn’t matter.

Thinking about Binomial distribution as n identical Bernoulli distributions helps understand the calculation of its expected value and variance:

If you are interested in getting these two equations above, you can watch these wonderful videos from Khan Academy.

Python’s Numpy library has a built-in Binomial distribution function. To simulate it, define n and p, and set to simulate 1000 times:

n = 100

p = 0.5

size = 1000 binomial = np.random.binomial(n,p,size)

plt.hist(binomial)

We can get the histogram:

binomial distribution, mean around n*p = 50

When setting n equals to 1, we can simulate the Bernoulli distribution:

n = 1

p = 0.5

size = 10000 bernoulli = np.random.binomial(n,p,size)

plt.hist(bernoulli)

Bernoulli Trials

3, Geometric Distribution

Geometric distribution is a discrete distribution that models the number of failures (x failures) before the first success in repeated, independent Bernoulli trials. For example, the random variable can be how many “Tails” would you get before you get your first “Head.” It can also model the number of trials to get the first success (x-1 failures), like how many times you have to toss until you get the first “Head.” The only difference between these two random variables is the number of failures. The Geometric distribution assumptions are the same as the Binomial distribution because they both derive from some identical independent Bernoulli trails.

When the random variable x is the number of failures before the first success, the PMF is:

the expected value and variance are:

When the random variable x is the number of trials to get the first success, the PMF is:

the expected value and variance are:

We need to use the geometric series to derive the expected value and variance for Geometric distribution. There is a great tutorial at Khan Academy that explains the details.

To simulate Geometric distribution, we can use the Bernoulli trials and count for the number of failures before the first success, and then plot the number of failures:

geometric = []

failure = 0

n=0

p=0.5 while n<10000:

result = np.random.choice(['success','failure'],p=(p,1-p))

if result == 'failure':

failure+=1

else:

geometric = []

failure = 0

n+=1 plt.hist(geometric)

plt.axvline(np.mean(geometric),color='red')

geometric distribution with mean around (1-p)/p

4, Uniform Distributions

Uniform distribution models a random variable whose outcomes are equally likely to happen. The outcomes can be discrete, like the outcomes getting form tossing a die, or continuous, like the waiting time for a bus to arrive. Thus Uniform distribution can be a discrete or continuous distribution depending on the random variable. The assumptions are:

1, there are n outcomes (discrete), or a range for the outcomes to be at (continuous);

2, All values in the outcome set or the range are equally likely to occur.

The discrete uniform distribution is straightforward, and it is easy to calculate the expected values and variance. For a continuous Uniform distribution that is uniformly distributed at [a, b], the probability density function (PDF) is:

Using integrals, the expected value and variance are:

To simulate Uniform distribution, we can use the Numpy’s embedded function and specifying the range of the distribution. Based on a Uniform distribution at [0,1], we can generate the Uniform distribution at [a, b]:

#generate a random variable follows U(0,1)

np.random.uniform(0,1,size=10000) #use U(0,1) to generate U(a,b)

def uniform(a,b):

return a + (b-a) * np.random.uniform(0,1,size=10000)

Uniform distribution at [2,3]

5, Normal Distribution

Normal (Gaussian) distribution is the most widely used continuous distribution because it represents the most universe situations. Many random variables are normally distributed because of the Central Limit Theory, or they are assumed to be normally distributed before fitting them into a statistical model. Normal distribution has some unique characteristics:

1, mean=mode=median=µ ;

2, the PDF is bell-shaped and symmetric at x=µ;

3, the values between [µ-σ, µ+σ] takes roughly 68% of the data, where σ is the standard deviation, and:

image by Wikipedia, µ=0 here

the PDF for Normal distribution is:

When µ=0 and σ=1, we have a Standard Normal distribution, where the PDF is simplified as:

The expected value and variance are embedded in the PDF of Normal distribution. The expected value is the mean, which is µ, and the variance is the square of standard deviation, σ².

To simulate the Normal distribution, we can use the Numpy function:

np.random.normal(mu, sigma, 1000)

Or we can also use Central Limit Theory to simulate Normal distribution:

def clm(N,n):

#generate a sample from any random population N

lis = np.random.random(size=N)

means = [] #take a random sub sample with size n from N

for i in range(n):

lis_index = np.random.randint(0,N,n)

means.append(lis[lis_index].mean())

i+=1 #plot means

return plt.hist(means) clm(10000,1000)

Normal Distribution by CLM

6, Poisson Distribution

Poisson distribution is a discrete distribution that models the probability of a number of events occurring in a fixed interval of time or space. Poisson distribution can be used to present the number of customers arriving in a store in an hour, or the number of phone calls a company receives one day, etc. Poisson distribution is closely related to binomial distribution if you measure the number of event occurrences as the number of success. For example, when measuring how many cars will pass a particular street in an hour, the number of cars passing is a random variable that follows Poisson distribution. To understand the car passing event, you can break down one hour into 60 minutes, and see how many cars will pass in a minute, then generalize it into an hour. For a minute, maybe more than one cars pass the street, thus it is not a binary random variable. However, if we break down one minute into 60 seconds, it is likely that only one car passes or no car pass in a single second. We can keep breaking down a second to make a more confident claim. We can then consider car passing as a successful event and no car passing as a failure event, and model how many success events (how many cars passing) in 3600 trials (3600 seconds in an hour), which is a Binomial distribution with success probability p, and the number of trials equals to 3600.

Connecting Poisson distribution with binomial distribution helps us understand the assumptions and PMF of Poisson distribution. The assumptions Poisson distribution are:

1, any successful event should not influence the outcome of other successful events (observing one car at the first second doesn’t affect the chance of observing another car the next second);

2, the probability of success p, is the same across all intervals (there is no difference between this hour with other hours to observe cars passing by);

3, the probability of success p in an interval goes to zero as the interval gets smaller (if we are discussing how many cars will pass in a millisecond, the probability is close to zero because the time is too short);

The PMF of Poisson distribution can be derived from the PMF of binomial distribution:

binomial distribution

We know x is the number of success in n Bernoulli trials, and E(x)=np. If we define λ =E(x) =np as the average number of success in n Bernoulli trials, the success probability p can be estimated by λ/n. As n goes to infinity, the PMF becomes:

After some calculation, we can get a new PMF, which is the PMF for Poisson distribution:

The Poisson distribution gives the probability of observing a certain number of events in an interval, given the average number of events in the interval. The expected value and variance of a random variable that follows Poisson distribution are both λ:

To simulate Poisson distribution, we can use the Numpy function:

lam = 1 poi = np.random.poisson(lam=lam,size=1000)

plt.hist(poi)

plt.axvline(x=lam,c='red')

Poisson distribution with lambda at one

7, Exponential Distribution

The Exponential distribution is a continuous distribution that is closely related to the Poisson distribution. It is the probability distribution of the time intervals between Poisson events. If the number of calls a company receives in an hour follows Poisson distribution, then the time interval between calls follows Exponential distribution. Since exponential distribution is closely related to Poisson distribution, its assumptions follow the Poisson distribution’s assumptions.

We can derive the PDF of exponential distribution by first deriving its Cumulative Distribution Function (CDF) and then take the derivative of the CDF. We are using this trick because we can get the CDF using Poisson distribution. Suppose we are still observing how many cars are passing in the same street, and now we care about the random variable ω, which is defined as when we see one car passing, it takes at least ω minutes to observe another car passing. Let’s start from the beginning — how long it takes to observe the first car? Assume it takes ω minutes to observe the first car in ω minutes, there is zero car passing. We know that the number of cars passing in the street in a minute follows Poisson distribution:

number of cars per minute

For ω minutes, the number of cars passing in the street follows:

number of cars in ω minutes

the probability that we observe zero cars in ω minutes is:

From exponential distribution’s perspective, we now already know the probability of taking at least ω minutes to observe the first car, then the probability of taking less than ω minutes is:

This is the CDF of the random variable ω, taking the derivative with respect to ω, we have the PDF:

If we know in a minute, on average, we are likely to observe three cars (λ=3) passing by the street, then it is expected that every 1/3 minutes, we will observe one car passing by. Thus, the expected value and variance of exponential distribution are:

To simulate the Exponential distribution, we can use the Numpy random function:

lam = 2

exp = np.random.exponential(lam, 10000)

plt.hist(exp)

plt.axvline(x=lam, c=’red’)

Exponential Distribution

Or we can use the CDF and Uniform distribution from (0,1) to simulate it. The following equations show the process:

Following the code:

rate = 1/20

inverse_cdf=np.random.uniform(0,1,size=1000) interval = [-np.log(1-u)/rate for u in inverse_cdf] plt.hist(interval)