MCMC Intuition for Everyone. Easy? I tried. | by Rahul Agarwal | Towards Data Science

All of us have heard about the Monte Carlo Markov Chain sometime or other. Sometimes while reading about Bayesian statistics. Sometimes while working with tools like Prophet.
But MCMC is hard to understand. Whenever I read about it, I noticed that the crux is typically hidden in deep layers of Mathematical noise and not easy to decipher.
I had to spend many hours to get a working understanding of the concept.
This blog post is intended to explain MCMC methods simply and knowing what they are useful for. I will delve upon some more applications in my next post.
So let us get started.
MCMC is made up of two terms Monte Carlo and Markov Chains. Let us talk about the individual terms one by one.
In simple terms, we can think of Monte Carlo methods as simple simulation.
Monte Carlo methods derive their name from Monte Carlo Casino in Monaco. Many card games need the probability of winning against the dealer.
Sometimes calculating this probability can be mathematically complex or highly intractable. But we can always run a computer simulation to simulate the whole game many times and see the probability as the number of wins divided by the number of games played.
So that is all you need to know about Monte Carlo Methods.
Yes, it is just a simple simulation technique with a Fancy Name.
So as we have got the first part of MCMC, we also need to understand what are Markov Chains. But, before Jumping onto Markov Chains let us learn a little bit about Markov Property.
Suppose you have a system of M possible states, and you are hopping from one state to another.
Don’t get confused yet. A concrete example of a system is the weather which jumps from hot to cold to moderate states. Or another system could be the stock market which jumps from Bear to Bull to stagnant states.
Markov Property says that given a process which is at a state Xn at a particular point of time, the probability of Xn+1=k, where k is any of the M states the process can jump to, will only be dependent on which state it is at the given moment. And not on how it reached the current state.
Mathematically speaking:
Intuitively, you don’t care about the sequence of states the market took to reach the bull market. The probability that the next state is going to be “bear” state is determined just by the fact that the market is currently at “bull” state.
It makes sense too in the practical scheme of things.
If a process exhibits the Markov Property, then it is known as a Markov Process.
Now, Why is a Markov Chain important?
It is important because of its Stationary Distribution.
So what is a Stationary Distribution?
I will try to explain stationary distribution by actually calculating it for the below example. Assume you have a Markov process like below for a stock market.
You have a matrix of the transition probabilities
Which defines the probability of going from a state Xi to Xj. In the above Transition Matrix Q,
the probability that the next state will be “bull” given the current state is “bull”=0.9
the probability that the next state will be “bear” given the current state is “bull”=0.075
And so on.
Now, we start at a particular state. Let us begin at bear state. We can define our state as [bull, bear, stagnant] in a vector form. So our starting state is [0,1,0]
We can calculate the Probability distribution for the next state by multiplying the current state vector with the transition matrix.
See how the probabilities add up to 1. And the next state distribution could be found out by
and so on. Eventually, you will reach a stationary state s where we will converge and:
For the above transition matrix Q the Stationary distribution s is:
You can get the stationary distribution programmatically as:
Q = np.matrix([[0.9,0.075,0.025],[0.15,0.8,0.05],[0.25,0.25,0.5]])init_s = np.matrix([[0, 1 , 0]])epsilon =1while epsilon>10e-9:    next_s = np.dot(init_s,Q)    epsilon = np.sqrt(np.sum(np.square(next_s - init_s)))    init_s = next_sprint(init_s)------------------------------------------------------------------matrix([[0.62499998, 0.31250002, 0.0625    ]])
You can start with any other state too; you will reach the same stationary distribution. Change the initial state in the code if you want to see that.
Now we can answer the question- why is a stationary distribution important?
The stationary state distribution is important because it lets you define the probability for every state of a system at a random time.
For this particular example, you can say that 62.5% of the times market will be in a bull market state, 31.25% times it will be a bear market and 6.25% time it will be stagnant.
Intuitively you can think of it as a random walk on a chain. You are at a state, and you decide on the next state by seeing the probability distribution of next state given the current state. We might visit some nodes more often than others based on node probabilities.
This is how Google solved the search problem in the early internet era. The problem was to sort the pages based on page importance. Google solved it using Pagerank Algorithm.
In the Google Pagerank algorithm, you might think of a state as a page and the probability of a page in the stationary distribution as its relative importance.
Woah! That was a lot of information, and we have yet not started talking about the MCMC Methods yet. Well if you are with me till now, we can now get on to the real topic now.
Before answering this question about MCMC, let me ask one question. We all know about beta distribution. We know its pdf function. But can we draw a sample from this distribution? Can you think of a way?
MCMC provides us with ways to sample from any probability distribution. This is mostly needed when we want to sample from a posterior distribution.
The above is Bayes theorem. Sometimes we need to sample from the posterior. But is it easy to calculate the posterior along with the normalizing constant(also called evidence)? In most of the cases, we are able to find the functional form of likelihood x prior.
But we are not able to calculate the evidence(p(D)). Why?
Let us expand the evidence.
If H only took 3 values:
p(D) = p(H=H1).p(D|H=H1) + p(H=H2).p(D|H=H2) + p(H=H3).p(D|H=H3)
The P(D) was easy to calculate. What if the value of H is continuous? Would one be able to write it as simply as now H could take infinite values? It would be a difficult integral to solve.
We want to sample from the posterior but we want to treat p(D) as a constant.
According to Wikipedia:
Markov Chain Monte Carlo (MCMC) methods are a class of algorithms for sampling from a probability distribution based on constructing a Markov chain that has the desired distribution as its stationary distribution. The state of the chain after a number of steps is then used as a sample of the desired distribution. The quality of the sample improves as a function of the number of steps.
So let’s explain this with an example:
Assume that we want to sample from a Beta distribution. The PDF for the beta is:
where C is the normalizing constant. It is actually some function of α and β but I want to show how we don’t really need it to sample from a beta distribution so I am treating it as a constant.
This is a somewhat tricky problem with the Beta Distribution if not intractable.
In reality, you might need to work with a lot harder Distribution Functions, and sometimes you won’t know the normalizing constants.
MCMC methods make life easier for us by providing us with algorithms that could create a Markov Chain which has the Beta distribution as its stationary distribution given that we can sample from a uniform distribution(which is relatively easy).
If we start from a random state and traverse to the next state based on some algorithm repeatedly, we will end up creating a Markov Chain which has the Beta distribution as its stationary distribution and the states we are at after a long time could be used as a sample from the Beta Distribution.
One such MCMC Algorithm is the Metropolis-Hastings Algorithm
First, what’s the goal?
Intuitively, what we want to do is to walk around on some (lumpy) surface(our Markov chain) in such a way that the amount of time we spend in each location is proportional to the height of the surface at that location(our desired pdf from which we need to sample).
So, e.g., we’d like to spend twice as much time on a hilltop that’s at an altitude of 100m as we do on a nearby hill that’s at an altitude of 50m. The nice thing is that we can do this even if we don’t know the absolute heights of points on the surface: all we have to know are the relative heights. e.g., if one hilltop A is twice as high as hilltop B, then we’d like to spend twice as much time at A as we spend at B.
There are more complicated schemes for proposing new locations and the rules for accepting them, but the basic idea is still:
(1) pick a new “proposed” location;
(2) figure out how much higher or lower that location is compared to your current location;
(3) probabilistically stay put or move to that location in a way that respects the overall goal of spending time proportional to the height of the location.
The goal of MCMC is to draw samples from some probability distribution without having to know its exact height at any point(We don’t need to know C).
If the “wandering around” process is set up correctly, you can make sure that this proportionality (between time spent and the height of the distribution) is achieved.
Let us define the problem more formally now.
Let s=(s1,s2,....,sM) be the desired stationary distribution. We want to create a Markov Chain that has this stationary distribution. We start with an arbitrary Markov Chain with M states with transition matrix P, so that pij represents the probability of going from state i to j.
Intuitively we know how to wander around this Markov Chain, but this Markov Chain does not have the required Stationary Distribution.
This chain does have some stationary distribution(which is not of our use)
Our Goal is to change the way we wander on the this Markov Chain so that this chain has the desired Stationary distribution.
To do this, we:
Start at a random initial State i.Randomly pick a new Proposal State by looking at the transition probabilities in the ith row of the transition matrix P.Compute a measure called the Acceptance Probability which is defined as: aij=min(sj.pji/si.pij,1).Now Flip a coin that lands head with probability aij. If the coin comes up heads, accept the proposal, i.e. move to next state else reject the proposal, i.e. stay at the current state.Repeat for a long time
Start at a random initial State i.
Randomly pick a new Proposal State by looking at the transition probabilities in the ith row of the transition matrix P.
Compute a measure called the Acceptance Probability which is defined as: aij=min(sj.pji/si.pij,1).
Now Flip a coin that lands head with probability aij. If the coin comes up heads, accept the proposal, i.e. move to next state else reject the proposal, i.e. stay at the current state.
Repeat for a long time
After a long time, this chain will converge and will have a stationary distribution s. We can then use the states of the chain as the sample from any distribution.
While doing this to sample the Beta Distribution, the only time we are using the PDF is to find the acceptance probability, and in that, we divide sj by si, i.e., the normalizing constant C gets canceled.
Now let us move on to the problem of sampling from Beta Distribution.
Beta Distribution is a continuous Distribution on [0,1] and it can have infinite states on [0,1].
Let us assume an arbitrary Markov Chain P with infinite states on [0,1] having transition Matrix P such that pij=pji=All entries in Matrix.
We don’t need the Matrix P as we will see later, But I want to keep the problem description as close to the algorithm we suggested.
Start at a random initial State i given by Unif(0,1).
Randomly pick a new Proposal State by looking at the transition probabilities in the ith row of the transition matrix P. Let us say we pick up another Unif(0,1) state as a proposal state j.
Compute a measure called the Acceptance Probability :
Which simplifies to:
since pji=pij, and where
Now Flip a coin that lands head with probability aij. If the coin comes up heads, accept the proposal, i.e. move to next state else reject the proposal, i.e. stay at the current state.
Repeat for a long time
So enough with the theory, let us Move on to python to create our Beta Sampler.
Let us check our results of the MCMC Sampled Beta distribution against the actual beta distribution.
As we can see, our sampled beta values closely resemble the beta distribution. And thus our MCMC Chain has reached the stationary state.
We did create a beta sampler in the above code, but the same concept is universally applicable to any other distribution we want to sample from.
That was a big post. Congrats if you reached the end.
In Essence, MCMC Methods might be complicated, but they provide us with a lot of flexibility. You can sample any distribution function using MCMC Sampling. They usually are used to sample the posterior distributions at the inference time.
You can also use MCMC to Solve problems with a large state space. For Example, Knapsack Problem Or decryption. You can take a look at some fun examples in my next blog post. Keep tuned.
One of the newest and best resources that you can keep an eye on is the Bayesian Statistics Specialization from University of California.
I am going to be writing more of such posts in the future too. Let me know what you think about the series. Follow me up at Medium or Subscribe to my blog to be informed about them. As always, I welcome feedback and constructive criticism and can be reached on Twitter @mlwhiz.
Introduction to Probability Joseph K Blitzstein, Jessica HwangWikipediaStackExchange
Introduction to Probability Joseph K Blitzstein, Jessica Hwang