Multi-Armed Bandits: Thompson Sampling Algorithm with Python Code | Towards Data Science

In this series of posts, we experiment with different bandit algorithms to optimise our movie nights — more specifically, how we select movies and restaurants for food delivery!
For newcomers, the name bandit comes from slot machines (known as one-armed bandits). You can think of it as something that can reward you (or not) every time you interact with it (pull its arm). The objective is, given a bunch of bandits that give different rewards, to identify the one that gives the highest ones, as fast as possible. As we start playing and continuously collect data about each bandit, the bandit algorithm helps us choose between exploiting the one that gave us the highest rewards so far and exploring others.
medium.com
towardsdatascience.com
You and your friend have been using bandit algorithms to optimise which restaurants and movies to choose for your weekly movie night. So far, you have tried different bandits algorithms like Epsilon-Greedy, Optimistic Initial Values and Upper Confidence Bounds (UCB). You’ve found the UCB1-Tuned algorithm to work slightly better than the rest, for both Bernoulli and Normal rewards, and have ended up using it for the last few months.
Even though your movie nights have been going great with the choices made by UCB1-Tuned, you miss the thrill of trying a new algorithm out.
“Have you heard of Thompson Sampling?” your friend asks.
Excited you pick up your phone and start reading about it.
Thompson Sampling, otherwise known as Bayesian Bandits, is the Bayesian approach to the multi-armed bandits problem. The basic idea is to treat the average reward μ from each bandit as a random variable and use the data we have collected so far to calculate its distribution. Then, at each step, we will sample a point from each bandit’s average reward distribution and select the one whose sample had the highest value. We subsequently get the reward from the selected bandit and update its average reward distribution.
How do we calculate/update distributions I hear you ask? The name Bayesian Bandits might have already given this away, but we will use Bayes Theorem.
Bayes Theorem P(A|B) = P(B|A) P(A) / P(B)where A, B: events in sample space P(A|B): conditional probability of A given B P(B|A): conditional probability of B given A P(A): probability of A P(B): probability of B
The above definition is for discrete variables, for continuous variables it is defined for probability density functions.
In our case, for each bandit, we want to calculate their P( μ | data ), i.e., how likely different values of μ are, given the data we have collected so far.
P( μ | data ) = P(data | μ) P(μ) / P(data)              ∝ P(data | μ) P(μ)    posterior ∝ likelihood x prior
Bandits with many data points will have a narrow distribution hence the data point we sample will have a high chance of being close to the real mean. Bandits with few data points, however, will have a wide distribution giving high values a good chance of being selected. This mechanism is what balances exploration vs exploitation.
Now, calculating P( μ | data ) is tricky unless the likelihood and prior are conjugate priors. This means, that when we combine the likelihood and the prior we get a posterior distribution from the same family as the prior. This is convenient for two reasons:
We can avoid nasty calculations for finding out what the posterior is since there is a closed-form expression for it.
We can use the posterior from step i-1 as the prior for step i. This makes going from P( μ | data up to step i-1) to P( μ | data up to step i) easy and ensures we only work with conjugate priors.
For the rest of this post, we will assume that a bandit gives a reward of 1 with probability μ and 0 otherwise, i.e.
ith bandit rewards ~ Bernoulli(prob = μ_i)
Now we need to choose a prior distribution for μ. Since μ is the probability of success, it can only take values between 0 to 1. Hence we need a distribution with support [0, 1]. One such distribution is the Beta distribution with parameters a and b. Before we collect any data, all values of μ are equally likely, hence a uniform prior over [0, 1] would make sense. Choosing a = 1 and b = 1 gives us exactly that.
Next, we need to pick a likelihood and preferably one for which the Beta distribution is a conjugate prior. The Binomial distribution satisfies this and it also makes sense for our use case as it models the probability of getting k successes (sum of a bandit’s rewards) out of N trials (number of times we played the bandit) with a probability of success p (this is the μ we are trying to estimate).
If  Prior is μ ~ Beta(α, β)  Likelihood is k | μ ~ Binomial(N, μ)Then  Posterior is μ ~ Beta(α + successes, β + failures)Where  N = successes + failures
This is a simple update rule, if the reward from a bandit is 1 we increment its a if it is 0 we increment its b.
We have defined the base classes you will see here in the previous posts, but they are included again for completeness.
The code below defines the class BernoulliBandit that represents the bandits, the base class Agent to represent agents and the helper class BanditRewardsLog that will keep track of the rewards.
Now, the only thing left is to subclass the Agent class and implement Thompson Sampling.
... and we are done! Now, let’s see how it holds up against the algorithms from the previous posts, namely, Epsilon-Greedy, Optimistic Initial Values and Upper Confidence Bounds (UCB).
We will use the following code to compare the different algorithms.
First, let’s define our bandits.
probs = [0.6, 0.7, 0.8, 0.9]bernoulli_bandits = [BernoulliBandit(p) for p in probs]
After this, we can simply run
compare_agents(  get_agents(),   bernoulli_bandits,   iterations=500,   show_plot=True,)
which gives us the following.
Hmm ... it’s not very clear, but it seems like Epsilon-Greedy and Bayesian Bandits have similar performances. To find the best one, let us repeat the above experiment and count how many times each algorithm got the best result.
wins = run_comparison(bernoulli_bandits)for g, w in zip(get_agents(), wins):    print(g, w)
My results were 397 wins for Epsilon-Greedy, 0 for UCB1, 220 for UCB1-Tuned and 383 for Thompson Sampling. It looks like Thompson Sampling and Epsilon-Greedy are the winners for our specific setting. UCB1-Tuned coming in third place is a bit surprising given that in previous experiments it outperformed the Epsilon-Greedy algorithm. This could be due to the fact that Thompson Sampling is stealing some of its wins, but this needs more investigation.
In this post, we have looked into how the Thompson Sampling algorithm works and implemented it for Bernoulli bandits. We then compared it to other multi-armed bandits algorithms and saw that it performed about the same as Epsilon-Greedy.
Become a member of Medium and get full access to all stories. Your membership fee directly supports the writers you read.
From the same author.
towardsdatascience.com
medium.com
[1] A Tutorial on Thompson Sampling. (n.d.). Retrieved September 25, 2021, from https://web.stanford.edu/~bvr/pubs/TS_Tutorial.pdf.