Multivariate Probability Theory: All About Those Random Variables
By: Shuai Guo
Tags: Getting Started, Statistics, Editors Pick, Probability, Data Science

5. Conditional distribution

For two random variables X₁ and X₂, the marginal distribution of X₁ informs us about the behavior of X₁ regardless of how X₂ is distributed. However, on some occasions, we may have already observed X₂ value, and we want to find out whether our knowledge of X₂ will influence how we think X₁ is distributed.

In the language of probability, we are actually looking for the distribution of X₁, conditioned on X₂ takes a specific value. This type of distribution is known as the conditional distribution. Conditional distribution constitutes a key concept in statistical inference, where a major goal is to deduce the parameter distribution conditioned on the observed data.

5.1 Intuition

Given a scatter plot of X₁ and X₂, we can easily estimate the conditional distribution of X₂ given X₁ equals a, for example. Mathematically, we express this conditional distribution function as f(x₂|x₁=a), where the variable on the left side of “|” denotes the variable that is described by the distribution function, while the expression on the right side of “|” represents the applied condition.

We can estimate f(x₂|x₁=a) in the following way:

Step 1: we only focus on those data points conforming to the applied condition, i.e., x₁=a. Those eligible points now constitute the new sample space. In practice, we usually settle for a thin band [a, a+ε] for X₁ (as illustrated below), as finding a point with its X₁ value being exactly a is not always possible.

Step 2: we project those data points bounded by the thin band (marked in red below) onto the X₂ axis. By plotting the histogram and performing the normalization procedure on the histogram, we obtain f(x₂|x₁=a).

Fig. 7 Only plotting the histogram of the points within a think band of X₁ yields the conditional distribution of X₂. (Image by Author)

If we step back and review what we just did from a more theoretical point of view, we would notice that to derive the conditional distribution of X₂ given X₁ equals a, we are in fact looking for a probability density function f(x₂|x₁=a), which, according to the definition of a PDF, when multiplying dx₂, yields the following probability:

To get f(x₂|x₁=a), we need to know how to compute the probability on the right-hand side of the equation. To that end, we need to recall the concept of conditional probability.

5.2 Conditional probability

The probability of an event B occurring given some event A has occurred is known as a conditional probability, denoted by P(B|A). By applying this definition to the above equation, we would see that event A corresponds to X₁ falling within [a, a+ε], and event B corresponds to X₂ falling within [x₂, x₂+dx₂].

As illustrated in the figure below, we have a sample space S and two subspaces A and B, representing events A and B, respectively. Calculating P(B|A) amounts to calculating the number of elements contained in the intersection of subspace A and B, relative to the number of elements contained in the whole subspace A:

Fig. 8 Sample space. (Image by Author)

If we divide both the numerator and denominator by n(S), i.e., the number of elements in the entire sample space S, we obtain:

This gives us the formula to calculate the conditional probability.

5.3 Theoretical treatment

Now we are ready to calculate P(x₂ ≤ X₂ ≤ x₂+dx₂|a ≤ X₁ ≤ a+ε):

We already know how to calculate the numerator and denominator above:

[numerator] → P(x₂ ≤ X₂ ≤ x₂+dx₂, a ≤ X₁ ≤ a+ε) = f(x₁=a, x₂)εdx₂;

[denominator] → P(a ≤ X₁ ≤ a+ε) = f(x₁; x₁=a)ε.

Here, f(x₁=a, x₂) is the joint PDF f(x₁, x₂) evaluated with x₁=a, and f(x₁; x₁=a) is the marginal PDF f(x₁) evaluated at a.

Once we understand how to calculate the required conditional probability, we can derive the target conditional PDF f(x₂|x₁=a):

Eliminating dx₂ on both sides, we obtain the formula for calculating the conditional PDF:

By generalizing a to be any value that X₁ could take, we obtain the expression of the conditional PDF for X₂ given X₁ as:

Similarly, we could express the conditional PDF for X₁ given X₂ as:

5.4 Properties

First of all, the conditional PDF f(x₂|x₁) is a probability density function of x₂ (similarly, f(x₁|x₂) is a PDF of x₁). This means:

To see this more clearly, we have

Notice that the integral on the right-hand side of the equation is actually the definition of the marginal distribution of X₁:

Therefore, we can prove that the integral of f(x₂|x₁) with respect to x₂ equals one, i.e., f(x₂|x₁) is indeed a PDF.

Consequently, we can express conditional probabilities like P(a ≤ X₂ ≤ b|X₁=x₁) as:

Secondly, the conditional PDF f(x₂|x₁) is, in general, a function of x₁, meaning that the shape of f(x₂|x₁) will be different when X₁ takes different values. However, one notable exception is when X₁ and X₂ are statistically independent. We will dive deeper into this topic in the latter part of this post.

Finally, the conditional distribution gives us another way to express the marginal distribution. Notice that we can write the joint PDF as:

If we integrate both sides with respect to X₁, then

Since the left-hand side term gives us the marginal distribution of X₂, we have the following:

This formula is known as the law of total probability.

5.5 Multivariate conditional distributions

We can extend our discussion of bivariate conditional PDF to multivariate situations. For example, the conditional distribution of X₁ given that X₂=x₂, X₃=x₃,…, Xₙ=xₙ is written as:

where the numerator is the joint PDF of X₁,…, Xₙ, and the denominator is the joint marginal distribution of X₂,…, Xₙ.

Also, we can write the joint conditional distribution of X₁ and X₂ given that X₃=x₃,…, Xₙ=xₙ as:

where the numerator is the joint PDF of X₁,…, Xₙ, and the denominator is the joint marginal distribution of X₃,…, Xₙ.

A common application of conditional distribution is to factorize the joint PDF. For example, we can factorize f(x₁, x₂, x₃) in the following form:

This could be useful in practice as f(x₁, x₂, x₃) may be hard to compute directly, but its associated conditional distributions like f(x₁|x₂, x₃) and f(x₂|x₃) are relatively easy to calculate.