Solving Conditional Probability Problems with the Laws of Total Expectation, Variance, and Covariance
By: Saurabh Maheshwari
Tags: Probability, Mathematics, Data Science, Statistics

In this article, we’ll see how to use the Laws of Total Expectation, Variance, and Covariance, to solve conditional probability problems, such as those you might encounter in a job interview or while modeling business problems where random variables are conditional on other random variables.

I am going to start by asking a couple of real-world probability questions:

You work for a public transit company. You have the past data on the number of passengers that get on and off the bus at various stops in your city. Find the expected number of passengers that are on the bus when it arrives at any stop. Quantify the uncertainty in this estimate. You have a hotel booking website. From the past data, you find that for a given amount of traffic, there is a conversion rate of 0.1. Also, on weekends, people visiting the site follow a Poisson process (10 people/hr). Find the expected revenue on a Saturday. How does this number vary?

In these scenarios, you can observe that the variables we are interested in depend on other random variables. For example, in the first question, the number of passengers on the bus at ith stop is most likely dependent on the number of passengers on the bus at (i-1)th stop. These conditional probability questions can seem mysterious at first, but with a solid grip on the Laws of Total Expectation, Variance, and Covariance we can solve them easily and efficiently.

We’ll first see how one can apply these Laws to a problem (related to the bus question above) and later will verify the results by simulating the problem in R.

From here on I assume that you have a basic understanding of random variables, their expectation and variance, and conditional probability. Here is a good resource, in case you want a refresher:

And if you want a deeper and more thorough understanding of basic concepts of probability, this is the ultimate book:

Let’s get started. Here again, is a version of the bus problem [1]:

An autonomous bus (yes, we are in 2050) arrives at the 1st station (i = 1) with zero passengers on board. At every station, 0, 1, or 2 passengers could get on the bus with probability 0.3, 0.5, 0.2, respectively. At every station, a passenger could alight the bus with a probability of 0.1. Find the expected value and variance of the number of passengers on the bus

1) after it leaves the 1st station

2) that alight the bus at 2nd station

3) after it leaves the 2nd station

Assume that the number of passengers on boarding the bus at a station is independent of the other stations and the vehicle has an infinite capacity.

Let’s define the following variables:

If you think about it, the number of passengers getting off the bus at a station is a binomial distribution with parameters (n = number of passengers on the bus when it arrives at that station, p = 0.1). For example, A2 ~ Binom(L1, 0.1)

Alright, given all this information, how can we go about solving this? To begin, here are a few observations we can draw from the question that motivates the need for using conditional relationships between variables:

The number of passengers on the bus after the 2nd station (L2) is dependent on the number of passengers on the bus after the 1st station (L1). For example, if there are more people on the bus after it leaves the 1st station, then it’s highly likely that there will be more people on the bus after the bus leaves the 2nd station. The number of passengers alighting the bus at any station depends on the number of people on board when the bus arrives at that station, for example, A2 will be dependent on L1.

So, we will first calculate estimates for variables on which other variables are dependent, and then use these estimates to estimate our dependent variables. For example, we will calculate the estimates for L1, then we will use these to calculate estimates for A2 and L2.

First, we are going to calculate the expectations for variables in our problem.

Calculation of Expected Values

Just following the definition of expected value, the expectation of the number of passengers on the bus when it leaves station 1, E(L1), can be calculated as follows:

Now, let’s calculate E(A2), i.e., the expectation of the number of passengers that get off the bus when it leaves station 2. As mentioned above A2 depends on L1, thus the E(A2) can be calculated by conditioning on L1, which brings us to the Law of Total Expectation.

Law of Total Expectation

The idea here is to calculate the expected value of A2 for a given value of L1, then aggregate those expectations of A2 across the values of L1. To understand this better, here is the Law:

Given random variables X and Y, the expected value of X is equal to the expected value of the conditional distribution of X on Y.

When Y is a discrete random variable, the Law becomes:

The intuition behind this formula is that in order to calculate E(X), one can break the space of X with respect to Y, then take a weighted average of E(X|Y=y) with the probability of (Y = y) as the weights.

Given this information, E(A2) can be calculated as follows:

[*] (A2|L1 = m)~ Binomial(m, 0.1), thus E(A2| L1 = m) = 0.1*m.

Similarily, expectation of the number of passengers that are on the bus when it leaves station 2, E(L2), can be calculated as follows:

[*] One must understand that the expected value and variance of B2 are equal to that for L1. The reason being, the number of people boarding the bus at any station has a similar probability distribution as people boarding the bus at the 1st station (L1).

This concludes the expectation part of the question. Keeping the business problem in mind, we should also consider the uncertainty in these estimates, which is measured by variance. So let’s solve for variance now.

Calculation of Variances

Let’s start by calculating the variance of L1, denoted by Var(L1). Since L1 is not dependent on any other variable, we can solve for Var(L1) directly by using the basic formula.

Now let’s look into the variance for A2. Again, since A2 is dependent on L1, taking the conditional variance makes the calculation easier. This brings us to the Law of Total Variance.

Law of Total Variance

The idea is similar to the Law of Total Expectation. We are going to divide the values of A2 into groups w.r.t L1, take the variance in groups, and then aggregate over those groups to get the desired variance.

Since we are calculating the variance, there are 2 sources of variability: (expected within the group variability in A2) + (variability in the expected value of A2 across the groups). To understand this better, have a look at this formula:

This explains the intuition behind the Law of Total Variance very clearly, which is summarised here:

Similar to the Law of Total Expectation, we are breaking up the sample space of X with respect to Y. Note that both Var(X|Y) and E(X|Y) are random variables. The first component calculates the expected variance of X as we average over all the Y values. But, Var(X|Y) is based upon E(X|Y) which is also random. Thus, we include the second term to account for the variance in that expected value.

Now let’s calculate Var(A2).

[*] A2|L1 = m~ Binomial(m, 0.1), thus Var(A2| L1 = m) = m*0.1*(1–0.1).

Here, we have also used the basic properties of expectations and variances that

Alright, so far so good. Time to get the variance of L2.

[*] Since B2 is independent of L1 and A2, B2 does not share a covariance with L1 and A2. But, L1 and A2 are dependent, thus expanding the variance introduces a covariance between them.

So, to calculate Var(L2), we need to calculate cov(L1, A2). Again, since A2 is dependent on L1, we will be using their conditional relationship to calculate covariance, which brings us to the Law of Total Covariance.

Law of Total Covariance

The idea is similar to the Law of Total Variance, so I will jump straight to the Law: Given 3 random variables, X, Y, and Z, the Law of Total Covariance states that

This does a great job explaining the intuition behind the Law of Total Covariance which I have summarized below.

cov(X,Y|Z), E(X|Z) and E(Y|Z) are random variables. Similar to the Law of Total Variance, the first term accounts for the average covariance between X and Y, over different Z values. Note that cov(X,Y|Z) is based on E(X|Z) and E(Y|Z) which are random. That is, for a value of Z, there will be a value realised for E(X|Z) and E(Y|Z) simultaneously. Thus, the second term incorporates the covariance between the X and Y coordinates realised for various values of Z.

Another way to understand this is to break the Law into:

(expected covariance between X and Y within the groups) + (covariance in the expected values of X and Y across the groups)

Now, let’s calculate cov(L1, A2) using this Law.

[*] By the property of covariance: cov(a*X, b*X) = a*b*Var(X)

Now, we have all the pieces for calculating Var(L2).

And, that’s it. We neatly used all the 3 Laws to exploit the relationship between dependent variables and derive the expected values and variances of the number of passengers on the bus

1) after it leaves the 1st station (L1)

2) that alight the bus at 2nd station (A2)

3) after it leaves the 2nd station (L2)

Next, let’s simulate this in R and verify our answers.

Simulation in R

Below, I have created a function that simulates the bus trips in R. This function takes in the number of bus trips to aggregate over as input and returns the desired estimates.

Next, I am going to use this function to generate 10,000 estimates, with each estimate calculated using a sample of 100,000 bus trips. Finally, we take an average of our 10,000 estimates to get the final value. Doing this provides the best estimate of the true population parameters as per the Law of Large Numbers (LLN). As per LLN, the more the estimates you use, the closer the average of these estimates gets to the true parameter value.

Here are the results:

Comparing these to the results we got theoretically, restated below, we can see that we have verified our solutions!

E(L1): 0.9

Var(L1): 0.49

E(A2): 0.09

Var(A2): 0.0859

cov(L1, A2): 0.049

E(L2): 1.71

Var(L2): 0.9679

I hope this blog helped you understand the Laws of Total Expectation, Variance, and Covariance and that they make a valuable addition to your probability theory knowledge and problem-solving strategies.