The ABCs of Approximate Bayesian Computation | by Tom Leyshon | Towards Data Science

Bayesian statistics are methods that allow for the systematic updating of beliefs in the evidence of new data [1]. The fundamental theorem that these methods are built upon is known as Bayes’ theorem. This says, given two events A and B , the conditional probability of A given that B is true is expressed as
. Where P(A|B) represents the probability of A given B has happened and P(B|A) represent the probability of B given A has happened, P(A) and P(B) are the probabilities of A and B happening regardless of anything else respectively.
A conclusion reached on the basis of evidence and reasoning [2]. If we infer the value of a given parameter we use information available to us to deduce what the most likely value of that parameter is.
Scientists quantify their uncertainty in their inferences using probabilities. For example, “I’m 99% sure that X has a value of Y”. So say we have a model that predicts a given event happening or not. The model requires a single input, the value of which we are not exactly sure. We can however assign a distribution of possible values it can take and the probability of it taking each value, this is known as a probability distribution. We run an experiment and observe some new data. Bayesian inference then uses Bayesian statistics (Equation 1) to update the probability distribution of our parameter given the new data. Bayesian inference is very powerful as it allows us to both quantify, and incorporate, a varying level of prior believes as to what distribution a given parameter should be.
If A represents the prior belief and B represents the new evidence then:
P(A) is known as the prior probability. A prior probability can be informative meaning we have a strong prior belief, or uninformative meaning we have a much more uncertain prior understanding of the parameter’s true value.
P(B|A) is known as the likelihood function. The probability of the recent data/evidence given that A is true. This allows us to quantify to what degree the evidence agrees with our prior beliefs.
P(A|B) is known as the posterior probability. The probability of A after taking into account the evidence.
Equation 1 tells us that the posterior probability is proportional to the product of our prior probability and the likelihood function,
. In simple cases the likelihood function is known and therefore an expression for the posterior distribution can be derived directly from the likelihood and prior [3]. When this is the case, the parameter’s probability distribution can be continuously inferred as new data come in no problem!
However, in most real world problems the likelihood function is usually horrifically complicated or impossible to derive! In these cases, an expression for the posterior distribution can not be easily determined. This is where Approximate Bayesian computation can be used to replace the calculation of the likelihood function.
An ABC algorithm estimates the posterior of a parameter by simulating the model to produce artificial data sets X using sample parameters taken from the prior distribution. A comparison of how similar the artificial data set produced by the sample parameter from the prior to the observed real data set Y is undertaken by computing a distance ρ(X,Y). Parameter samples that produce artificial data sets that are ‘very close’ to the real data set Y are collected as samples from the posterior distribution. How close is ‘very close’ is determined by a pre-defined hyper-parameter called the tolerance (ε), with parameter’s that produce artificial data closer than ε to the observed data being accepted as part of the posterior.
The general ABC approach:
The distance function ρ(X,Y)
The distance function is normally chosen to be the simplest summary statistic that can accurately compare the observed data (X) and the simulation data (Y). Popular choices:
The absolute difference
The sum of squared difference
Kullback–Leibler divergence (KL)
choice of tolerance (ε):
The tolerance (ε) defines the boundary threshold between prior and posterior samples. This value can be as strict or lenient as the task requires. An adaptive threshold can often also be used. This starts relatively large and slowly decreases as sampling progresses (this is similar to the method of simulated annealing). So like many hyper-parameters in statistical learning the optimum tolerance value is up for debate!
ABC rejective sampling method in python
#Distance function - Absolute difference 
def distance_function(X,Y):
    distance = (1/len(X))*abs(sum(X)-sum(Y))
    return distance

#The ABC method with uniform prior, returns samples from the posterior
def ABC_Method_Uniform_Prior(Observed_data,Number_of_Samples,threshold):
    #The observed data 
    #initialise Posterior array
    Posterior_distribution = []
    #trials
    n = len(Observed_data) 
    #loop through to get the samples. 
    for i in range(0,Number_of_Samples):
        distance = threshold+1
        #While the distance is greater than the threshold continue to sample theta from the beta distribution
        while distance > threshold:
            #sample theta from the prior
            theta = np.random.beta(1, 1, size=1)[0]
            # generate the sim data 
            X = np.random.binomial(1, theta, n)
            # calcalute the distance from Y 
            distance = distance_function(X,Observed_data)
        Posterior_distribution.append(theta)
    return Posterior_distribution

A dodgy friend has given us a coin and bets us that we won’t be able to tell him what the true probability (p) of flipping a heads is. The coin can either land of heads or tails so the problem is a binomial. Our friend is a lunatic and we don’t trust his coin one bit. We impart this distrust into our prior belief by using an uninformative prior, that is a uniform distribution for the probability of flipping a heads. This means we consider all possible values of p from 0 to 1 as equally probable. In Bayesian inference the Beta distribution is often used as a uninformative prior for a binomial likelihood [1]. The Beta distribution is given by
The hyper-parameters of the beta distribution can be thought of as the number of successes (α) and the number of failures (β) observed. The uninformative Beta distribution prior has hyper-parameters(α,β) =(1,1). This represents the observation of one of each outcome previously. Figure 2 shows how the Beta distribution varies for different values of (α,β).
The Beta distribution has the neat property that using it as a prior distribution for a binomial likelihood will result in a Beta posterior distribution with updated α and β. This property is known as conjugacy. The posterior for the model is then given by
This posterior distribution of p can now be used to evaluate hypotheses about the true probability of flipping a heads (p). If we consider trial observations of n = 10, n= 100 and n = 1000 coin flips the posterior can be analytically derived, figure 3 plots these posteriors.
So let’s pretend our binomial example has a horrific likelihood function and therefore we must estimate the posterior distribution using ABC methods.
We use the simplest ABC algorithm: ABC rejection sampling. Our choice of distance function is the absolute difference between the proportions of observed and simulated number of heads.
A tolerance of ε= 0 is used. This means we only accept samples that produce the exact proportions of heads and tails as our observed data. Figure 4 shows an animation of the ABC process for this example. Samples that are similar to the true parameter value are much more likely produce the same observed data and so are accepted more often into the posterior distribution. We require N=10,000 samples in the posterior before terminating the ABC algorithm. In figure 4 the observed data produced 7 heads and 3 tails out of n=10 trials. When the simulation data produces 7 heads and 3 tails the parameter sample is accepted into the posterior distribution.
Considering the same trial observations as above n = 10, n=100 and n=1,000 the ABC estimated posteriors with 10,000 samples are given in Figure 5 below.
It can be seen that the estimates of the posteriors found using ABC in figure 5 are highly accurate, almost exactly equal to the true posterior (red lines). The simplicity of this binomial example allows us to sample at negligible cost. However, In most cases, the posterior distribution has to be found numerically via MCMC techniques (using Stan, WinBUGS, OpenBUGS, JAGS, PyMC or some other program).
This simple tutorial demonstrates the power of ABC techniques to provide accurate inference of a parameter’s posterior distribution. This approach is particularly useful when the model in question has a horrific or intractable likelihood function, this is more often than not the case for complex models of the real world. As shown here, ABC approaches are very easy to use and with reliable data can produce fantastically accurate results. Although ABC methods allow us to evade dealing with complex likelihood function they are considered computationally expensive relative to analytical solutions of a given problem. However, ABC methods provide a means to performing parameter inference when confronted with an unreasonably complex model.
[1] Evans, M., Hastings, N. and Peacock, B., 2001. Statistical distributions.
[2] Edited by Stevenson, Angus. Oxford Dictionary of English, Oxford University Press, 2010.
[3] Turner, B.M. and Van Zandt, T., 2012. A tutorial on approximate Bayesian computation. Journal of Mathematical Psychology, 56(2), pp.69–85.