Introduction to MCMC and Metropolis | Towards Data Science

Probabilistic modelling is all the rage these days but there was always one thing that bugged me when I first learned about it. Many Bayesian modelling methods require the computation of integrals and any worked examples I saw seemed to use Gaussian or Bernoulli distributions for the simple reason that it becomes an analytical nightmare (or even intractable) if you try to use anything more complicated than that. Restricting Bayesian modelling to the small subset of “well behaved” distributions can dramatically hinder your ability to model a problem well so it’s important that we come up with methods of overcoming this limitation.
So what do I do if I don’t want to analytically compute some pesky integral? Enter Monte Carlo approximations.
We know that we can compute an expectation by using samples from the target distribution and calculating their sample mean. Why’s that important? Well, what is an expectation but an integral...
This method of estimating the integral has some nice guarantees courtesy of the central limit theorem. Firstly, this is an unbiased estimate of the expectation and secondly, we can calculate the variance of the estimate.
Calculating integrals using Monte Carlo samples is all very well but how do we draw samples from the target distribution in general? Drawing Gaussian or uniform samples is easy, but anything more difficult and np.random is going to fail you.
The simplest way of drawing samples is using the inverse CDF method but this relies on having access to the inverse CDF function which often doesn’t have a nice analytical form and only makes sense for 1-dimensional random variables.
The Metropolis algorithm is one of the building blocks of many Markov Chain Monte Carlo (MCMC) sampling methods. It allows us to draw samples when all you have access to is the pdf of the target distribution.
MCMC methods come with the caveat that we’re no longer taking independent samples so we can’t have the same guarantees for how the variance of our estimate reduces with the number of samples we take. If samples are drawn independently, the Central Limit Theorem tells us that the variance of our estimate will decrease inversely in proportion to the number of samples (N). For MCMC we can discount this by adjusting the number of samples from N to N_eff. N_eff is (almost) always smaller than N and is related to how correlated the samples in the chain are.
The steps of the Metropolis algorithm are as follows:
1. Sample a starting point uniformly from the domain of the target distribution or from the prior distribution.2. Calculate the pdf at that point.3. Make a proposal for the new sample by taking a step from the current position in accordance with some state transition function.4. Calculate the new pdf value.5. Calculate the value of new pdf / old pdf.6. If the ratio is greater than 1, accept the step.7. If the ratio is less than 1:    1. Sample a uniform random variable.    2. If the ratio greater than the random sample, accept the step.8. Else reject the proposal, add the current position as a sample and take a new step.
Note that the process described in 5–8 is equivalent to accepting a sample based on a Bernoulli probability with probability min(1, p(new)/p(old)), remember this for later...
For any MCMC method, we want to ensure a property known as detailed balance or reversibility.
If π satisfies this then π is a stationary distribution of the Markov chain (1). We can show this by summing over both sides of the equation. If we can guarantee detailed balance then we also know we’re sampling from the stationary distribution of the Markov chain which we will dictate to be our target distribution.
This intuition of detailed balance is that since the transfer of probability ‘mass’ at each transition is the same from state i to state j as it is from state j to state i, after each transition of the chain, we remain at the stationary distribution.
So now let’s show how the Metropolis algorithm satisfies this condition...
To find p(i,j) such that it satisfies detailed balance, we first propose arbitrary “jump” probability q(i,j), and then obtain p(i,j) by only accepting the “jump” with probability α(i,j). When a “jump” is rejected, the state remains j=i. This “acceptance” idea is not unique to the Metropolis algorithm and exists in most variants of MCMC.
This is subject to α being a valid probability distribution. So a valid form of α is:
If the jumpy probability is symmetric then this can be simplified to:
Otherwise, it can be left in its full form which is called Metropolis-Hasting MCMC.
Now we can guarantee detailed balance, we can allow the Markov chain machinery to take over. If the Markov chain is ergodic (all states are irreducible), then at some point the chain will reach the stationary distribution and we are able to take samples from the target distribution.
You might have also noticed that since alpha is a function of π(j)/π(i). The result of this means that the target distribution doesn’t need to be normalised. This is particularly helpful when using Metropolis for Bayesian posterior estimation where the evidence term is hard to compute.
A common version of the Metropolis algorithm is called “Random walk Metropolis” where the proposed state is the current state plus a multivariate Gaussian with zero mean and covariance matrix σ2I. σ should be chosen to be large enough such that sufficiently many samples are rejected. This is to ensure a good exploration of the target distribution.
The second thing to note is the concept of burn-in. Samples that are taken before the Markov chain reaches the stationary distribution should be removed as they’re not representative of the target distribution until the chain has converged. It’s somewhat difficult to identify how many samples should be removed but in general, 10–20% of samples are removed (or 10–100 effective samples).
Here we implement random walk Metropolis for the sake of simplicity.
def metropolis(pi, dims, n_samples, burn_in=0.1, var=1):    theta_ = np.random.randn(dims)*var    samples = []    while len(samples) < n_samples:        theta = theta_ + np.random.randn(dims)*varratio = pi(theta)/pi(theta_)        if np.random.rand(1) < ratio:            sample = theta            theta_ = theta            samples.append(sample)        else:            sample = theta_            samples.append(sample)    samples = np.array(samples)    return samples[int(samples*burn_in):,:]
We can see how this performs on the sum of two Gaussians (note this is an unnormalised distribution).
from scipy.stats import multivariate_normaldef make_pdf(mean1, mean2, cov1, cov2):    pdf1 = multivariate_normal(mean1, cov1)    pdf2 = multivariate_normal(mean2, cov2)    def pdf(x):        return pdf1.pdf(x) * pdf2.pdf(x)    return pdfpdf = make_pdf(    [3, 3],    [-1, -1],    np.array([[1,0.1],[0.1,1]], dtype=float),    np.array([[1,0.1],[0.1,1]], dtype=float),)samples = metropolis(pdf, 2, 1_000, 0.1)
The gif above shows how the algorithm traverses the distribution, occasionally hopping between the two different modes of the distribution. Note, this also highlighted one of the weaknesses of the Metropolis algorithm, it deals relatively poorly with multi-model distributions. This is due to the fact that to explore a new mode, the step must be sufficiently large to hope from one mode to another. This either requires a large step size or a distribution with modes that are close together. Modifications such as Hamiltonian MCMC can help with this but generally, it is a problem with most MCMC methods.
Video on detailed balance
The original paper on the Metropolis-Hastings algorithm
Paper on Hamiltonian dynamics (MCMC method used by stan and other probabilistic languages)