PDF is not a probability.. The probability density at x can be... | by Aerin Kim | Towards Data Science

It’s a well-known fact that the largest value a probability can take is 1.
However, for some PDFs (e.g. the PDF of the exponential distribution, the graph below), when λ= 1.5 and x = 0, the probability density is 1.5, which is obviously greater than 1!
Even if the PDF f(x) takes on values greater than 1, if the domain that it integrates over is less than 1, it can add up to only 1. Let’s take an example of the easiest PDF — the uniform distribution defined on the domain [0, 0.5]. The PDF of the uniform distribution is 1/(b-a), which is constantly 2 throughout.
The total probability is the total area under the graph f(x), which is 2 * 0.5 = 1.
As you can see, even if a PDF is greater than 1, because it integrates over the domain that is less than 1, it can add up to 1.
Isn’t the PDF f(x) a probability?
No. Because f(x) can be greater than 1. (“PD” in PDF stands for “Probability Density,” not Probability.)
f(x) is just a height of the PDF graph at X = x. (Are you confused with X vs x notation? Check it out here.)
The whole “PDF = probability” misconception comes about because we are used to the notion of “PMF = probability”, which is, in fact, correct. However, a PDF is not the same thing as a PMF, and it shouldn’t be interpreted in the same way as a PMF, because discrete random variables and continuous random variables are not defined the same way.
For discrete random variables, we look up the value of a PMF at a single point to find its probability P(X=x) (e.g. Remember how we plugged x into the Poisson PMF?)For continuous random variables, we take an integral of a PDF over a certain interval to find its probability that X will fall in that interval.
f(x) ≠ P(X = x)* f(x): PDF for a continuous r.v.* P(X = x) : PMF for a discrete r.v.
Sure, all good. However, you might wonder...
Why do we have to *integrate* a PDF?
Can we just sum up PDF values, just like we do with PMF values?
No. Because, for continuous random variables, the probability that X takes on any particular value x is 0.
Let's look at the previous example — the Uniform Distribution in [0, 0.5].
The probability density at x=1 is two.But why is the probability at x=1 zero?
To answer the above question, we need to answer the following question first: How many total real numbers do we have in [0, 0.5]?
Infinite ∞. (To be mathematically thorough, uncountably infinite.)0.1, 0.01, 0.001, 0.0001,... You can keep inserting 0 in front of the smallest decimal.
Therefore, a continuous random variable has an infinite # of possible values that it can take, even if the domain is small and fixed. And let's say, the probability density for each value in [0, 0.5] takes an extremely small value, e.g. 000000001. Still, the sum of infinitely (uncountably) many values will reach infinity, no matter how small their values are.
Then, in order to make the sum of probabilities 1, the probability at a single point should be 1/∞, which is 0.
Well, this doesn’t really make sense, either. If you add the infinite number of zeros, you will still get zero. The total probability should add up to one, not to zero.
The thing is, we can’t use the notion of discrete PMF (one value has one matching probability) for continuous variables. We can’t define the probability of continuous variables as we did for discrete variables.
If the probability of X being exactly at point x is zero, how about an extremely small interval around the point x? Say, [x, x+dx]? Let’s say dx is 0.00000000001.
Then the probability that X will fall in [x, x+dx] is the area under the curve f(x) sandwiched by [x, x+dx].
If dx is infinitesimally small, this approximation will be good enough for P(X=x).
f(x)dx : The probability of X in [x, x+dx].f(x): Probability density.dx : Interval length.
A few things to note:
1. If you take a look at the definition of both a PDF and a PMF, it’s actually simply changing the summations in the discrete case (PMF) to integrals in the continuous case (PDF).
2. Why do we use the terms “Density” and “Mass”?
This is analogous to mass density in physics — integrating the density to get the mass. If you think of a mass as a probability, we are integrating a probability density to get a probability (mass).
3. What does a probability density at point x mean?
It means how much probability is concentrated per unit length (dx) near x, or how dense the probability is near x.
4. We need to fix the Wikipedia graph of the exponential distribution. The level of Y-axis P(X) sounds like a probability. We need to change it to f(x) or “Probability Density”.