Detect Change Points with Bayesian Inference and PyMC3 | by Khuyen Tran | Towards Data Science

Imagine you use Google Analytics to track the views of your website. After looking at the views, you suspect there might be a sudden change in the number of views after a certain date.
If there is indeed a date when the number of views changes suddenly, how do you find that date?
Wouldn’t it be nice if you can leverage observed data and some beliefs to guess the change point with high certainty?
That is when Bayesian inference comes in handy. In this article, we will learn what Bayesian inference is and how to use PyMC3 to perform Bayesian analysis.
Bayesian inference is a technique in which Bayes’ theorem is used to specify how one should update one’s beliefs upon observing data.
towardsdatascience.com
For example, our prior belief can be that a coin is biased and that it follows a Beta(1.36, 2.36) distribution. However, after flipping the coin over 100 times (observing real data), our belief shifts to a normal distribution with a mean of 0.5.
We can see that the posterior distribution can be very different from the prior distribution after gathering more data.
We will use the data showing the number of views collected from my website Data Science Simplified from January 2021 to July 2021.
Now, how can we find the date when the distribution of the number of views changes?
Let’s start with finding prior distributions of time τ and the number of views.
Since the number of views is a count data, it can be modeled using a discrete distribution, such as the Poisson distribution.
The Poisson distribution shows how many times an event is likely to occur in a fixed interval of time. The Poisson distribution uses the parameter λ to control its shape.
Let’s plot a Poisson distribution to see how it looks like.
Poisson distribution with λ = 1.5:
When increasing λ, we add more probability to the larger values. For example, increasing the value of λ from 1.5 to 3 shifts the mean of distribution to the right:
From the plots above, we can see that the mean number of views is close to λ.
Cool! Now we know how to model the prior distribution of the number of views, how do we find an appropriate λ?
We don’t know the specific value of λ, but we know that there is a time τ after which the distribution of the number of views changes significantly. Thus, we can say that λ =λ1 before the time τ and λ =λ2 after the time τ.
But how can we model the prior distribution of λ1 and λ2? Since λ1 and λ2 are continuous random variables, we can model their prior distributions using an exponential distribution.
The exponential distribution is often used to model the time elapsed between events. The exponential distribution also uses a parameter, say α, to control its shape.
Given a specific α, the expected value of λ is equal to the inverse of α.
Let’s see how the exponential distribution looks like.
Exponential distribution with α = 0.1:
Exponential distribution with α = 1:
Now, what is the value of α? We can combine the two equations we found previously to estimate the value of α.
From the equation above, it is reasonable to set α equal to the inverse mean of the number of views.
Because all dates are equally likely to be τ, we can use the uniform distribution to model the prior distribution of τ.
Since there are a total of 182 days, we can write:
This means that the probability of τ is equal to the date k is:
Let’s visualize how that uniform distribution looks like:
We can see from the plot above that all dates are equally likely to be τ.
Next, we will perform Bayesian analysis using PyMC3. PyMC3 is a Python package for Bayesian statistical modeling using intuitive syntax.
To install PyMC3, type:
pip install pymc3
Start with creating the PyMC3 variables for λ1, λ2, and τ.
Next, we use pm.math.switch to assign a value to λ. The date t represents the date λ is on. If t is smaller than τ, it assigns the value of λ1 to λ. It assigns the value of λ2 to λ otherwise.
Next, we model the number of views using the Poisson(λ) distribution:
Note that so far, we only model the prior distributions of λ1, λ2, and τ. To get some samples from the posterior distributions of λ1, λ2, and τ, we will use Markov Chain Monte Carlo (MCMC).
MCMC is a method for randomly sampling a probability distribution that honors the probabilistic dependence between samples.
Based on Cam Davidson-Pilon, in general, algorithms to perform MCMC are:
1. Start at the current position.
2. Propose moving to a new position.
3. Accept/Reject the new position based on the position’s adherence to the data and prior distributions.
4. If you accept: Move to the new position. Return to Step 1.
Else: Do not move to a new position. Return to Step 1.
5. After a large number of iterations, return all accepted positions.
The GIF below shows the search strategy of Metropolis-Hastings, an MCMC method.
Let’s use MCMC to get thousands of random variables from the posterior distributions of λ1, λ2, and τ.
Next, visualize how these random variables look like:
From the plot above, we can see that:
There is a significant difference between the posterior distribution of λ1 and the posterior distribution of λ2. This indicates that there is indeed a date τ after which the distribution of the number of views changes.
The probability of τ = 91 is 99.99%. This shows that there is a very high chance that the change point is 91.
Now we have 40k samples of λ1, λ2, and τ, let’s get the expected number of views per day.
Steps to calculate the expected number of views for the date t:
Pick a date t
For each sample of τ, see if the date t is before or after the date τ.
If t is before τ, the value of λ is λ1. If t is after τ, the value of λ is λ2.
Take the mean of all values of λ on date t
For example, if there are only 3 samples of λ1, λ2, and τ, our calculation will look similar to the below:
Use the calculation above to compute the expected number of views on each day:
Plot the expected number of views per day:
Pretty cool! We can see that there is a shift in the expected number of views after day 91st. This matches what we see from the real data.
Before closing this article, we should take a moment to appreciate how powerful Bayesian inference is. We get the change point with such high certainty using only observed data and some initial beliefs.
Plus, we get the distributions of the data before and after the change point. These distributions can tell us much more than single values can.
I hope this article will give you the knowledge and motivation to use Bayesian inference and PyMC3 to analyze other interesting data around you.
Feel free to play and fork the source code of this article here:
github.com
I like to write about basic data science concepts and play with different data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these:
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
Brownlee, J. (2019, September 24). A gentle introduction to Markov chain Monte Carlo for probability. Machine Learning Mastery. Retrieved December 10, 2021, from https://machinelearningmastery.com/markov-chain-monte-carlo-for-probability/.
Davidson-Pilon, C. (2016). Bayesian methods for hackers: Probabilistic Programming and bayesian inference. Addison-Wesley.