Discrete vs Continuous Probability Distributions in context of Data Science
By: Rohan Paul
Tags: Machine Learning, Python, Probability, Python3, Data Science

First lets define some terms for clarity

The sample space Ω

The sample space is the set of all possible outcomes of the experiment,

usually denoted by Ω. For example, two successive coin tosses have

a sample space of {hh, tt, ht, th}, where “h” denotes “heads” and “t”

denotes “tails”.

The event space A

The event space is the space of potential results of the experiment. A

subset A of the sample space Ω is in the event space A if at the end

of the experiment we can observe whether a particular outcome ω ∈ Ω

is in A. The event space A is obtained by considering the collection of

subsets of Ω, and for discrete probability distributions (Section 6.2.1)

A is often the power set of Ω.

The probability P

With each event A ∈ A, we associate a number P (A) that measures the

probability or degree of belief that the event will occur. P (A) is called

the probability of A.

The probability of a single event must lie in the interval [0, 1], and the

total probability over all outcomes in the sample space Ω must be 1, i.e.,

P (Ω) = 1. Given a probability space (Ω, A, P ), we want to use it to model

some real-world phenomenon. In machine learning, we often avoid explic-

itly referring to the probability space, but instead refer to probabilities on

quantities of interest, which we denote by T as the target space and refer to elements of of T as states.

The term probability relates is to an event and probability distribution relates is to a random variable .

It is a convention that the term probability mass function refers to the probability distribution of a discrete random variable and the term probability density function refers to the probability function of a continuous random variable .

Understanding Probability Density

First a quick reference on PMF, PDF and CDF

In order to understand the heart of modern probability, we need to extend the concept of integration from basic calculus.

To begin, let us consider the following piecewise function

Applying the fundamental Riemann integration of Calculus we get

which has the usual interpretation as the area of the two rectangles that make up f (x).

The question is given f (x) = 1, what is the set of x values for which this is true? For our example, this is true whenever x ∈ (0, 1]. So now we have a correspondence between the values of the function (namely, 1 and 2) and the sets of x values for which this is true, namely, {(0, 1]} and {(1, 2]}, respectively. To compute the integral, we simply take the function values

(i.e., 1,2) and some way of measuring the size of the corresponding interval.

Since areas can be defined by definite integrals, we can also define the probability of an event occuring within an interval [a, b] by the definite integral

where f(x) is called the probability density function (pdf).

A function f(x) is called a probability density function if

f(x)≥0 for all x The area under the graph of f(x) over all the real line is exactly 1 The probability that x is in the interval [a, b] is

i.e. the area under the graph of f(x) from a to b.

In the problem above, the probability density function f(x) is called a uniform (flat) probability density function (pdf).

So fundamentally, what does a probability density at point 𝒙 mean? Probability density function’s value at some specific point does not give you probability; it is a measure of how dense the distribution is around that value. It means how much probability is concentrated per unit length (d𝒙) near 𝒙, or how dense the probability is near 𝒙.

For discrete random variables, we look up the value of a PMF at a single point to find its probability P(𝐗=𝒙)

For continuous random variables, we take an integral of a PDF over a certain interval to find its probability that X will fall in that interval.