Multi-Armed Bandits: Upper Confidence Bound Algorithms with Python Code | Towards Data Science

In this series of posts, we experiment with different bandit algorithms in order to optimise our movie nights — more specifically how we select movies and restaurants for food delivery!
For newcomers, the name bandit comes from slot-machines (known as one-armed bandits). You can think of it as something that can reward you (or not) every time you interact with it (pull its arm). The objective is, given a bunch of bandits that give different rewards, to identify the one that gives the highest ones, as fast as possible. As we start playing and continuously collect data about each bandit, the bandit algorithm helps us choose between exploiting the one that gave us the highest rewards so far and exploring others.
In our story, we use restaurants instead of bandits and our reward is food satisfaction! :)
medium.com
medium.com
It’s been a while since you and your friend started using bandit algorithms to optimise your movie nights. So far, you’ve been using Epsilon-Greedy (with ε=1/#actions) to help with finding the best restaurant and your friend has stubbornly been using the Optimistic Initial Values algorithm to pick movie directors (even if your simulations have shown that it’s likely not as good for your use case).
While killing time on the internet, being trapped inside because of the lockdown, you come across some bandit algorithms based on Upper Confidence Bounds (UCB). Excited about the possibility of improving your search for the best restaurant, and more efficiently solving the exploitation-exploration dilemma, you start reading in anticipation of the next movie night.
Will one of them be better than Epsilon-Greedy? Let’s find out!
The idea is to use probability theory to come up with an upper bound on the average reward of each bandit, using the data we have collected so far. We will call this upper bound the upper confidence index of the bandit, like in the original paper. As with confidence intervals, this bound around the sample mean will be wide if we don’t have a lot of data. At each step, we select the bandit with the highest upper confidence index, get the reward and subsequently update its index.
This sounds a bit similar to the Optimistic Initial Values Algorithm, right? Coming up with this index may seem more complicated than just choosing an optimistic initial value, but the implementation remains relatively simple (I swear). They, really, are simpler to use as they don’t have any parameters that need to be chosen by us!
In general, the upper confidence index for each bandit has the following form.
Bandit Upper Confidence Index = X_sample_mean + Cwhere X_sample_mean:  sample mean for rewards so far C > 0:          size of one sided confidence interval
C should decrease as we gather more data for the bandit, which makes sense since we become more certain on the possible values for the average reward. Bandits that have not been sufficiently explored will have a larger C that will cause us to select them for exploration.
In pseudo-code:
Choose n # number of iterationsInitialisation step. # for UCB1 & UCB1-Tuned we play each bandit oncefor i = 1 to n do:  current_bandit = pick bandit with greatest upper confidence index  reward = current_bandit.pull_arm()    Update upper confidence index for current_bandit
OK... We know how to calculate sample means, but how do we calculate C? Well, it depends!
There are a few UCB algorithms presented in the original paper, each with different definitions for C. UCB1 and UCB1-Tuned were designed for rewards that are either 0 or 1, i.e. rewards from a Bernoulli distribution (see this article). If your rewards come from a Normal distribution, there is another algorithm, called UCB-Normal, that was designed for that. You could use UCB1 for rewards from other distributions but the performance is said to not be as good (we will test this).
The upper confidence index for a bandit for UCB1 is equal to
Bandit Upper Confidence Index = Sample Mean + √(2logN / n)where   N = total rounds (e.g. number of movie nights so far)  n = number of times we've played the bandit (    e.g. number of times we've ordered from a restaurant  ).
It is derived using the Chernoff-Hoeffding bound, that states
P( Sample Mean + t ≤ True Mean) ≤ exp(-2nt2) for t > 0
where nis the number of data points we have collected so far from the specific bandit. This means, that the probability that the error of the sample mean is greater than t decreases as n increases.
To derive the upper bound, we start by setting the right-hand side equal to b and solving for t.
Let  b = exp(-2nt2).Then logb = -2nt2 <=> t = √(-logb / (2n))Choose (this is a heuristic) b = (1 / N4)Then t = √(2logN / n)
In the above, n is the number of times we’ve played the specific bandit and N is the number of times we’ve played in total (number of rounds or movie nights if you like). You can think of b as a hyperparameter.
We know  P( Sample Mean + t ≤ True Mean) ≤ exp(-2nt2) for t > 0Substitute for t P( Sample Mean + √(2logN / n) ≤ True Mean) ≤ 1 / N4Define  Upper Confidence Index = Sample Mean + √(2logN / n)Hence   C = √(2logN / n)
From the above, we can deduce that the probability that the true mean is greater than the upper confidence index is less than 1 / N4 — this converges to 0 as the number of rounds N increases. Therefore it makes sense to use it as an upper bound.
One thing to note is the ratio logN / n. As the gap between the number of rounds N and number of times we’ve played the bandit n increases so does the bound.
For UCB1-Tuned, we replace C with the following
C = √( (logN / n) x min(1/4, V(n)) )where V(n) is an upper confidence bound on the variance of the bandit, i.e.V(n) = Σ(x_i2 / n) - (Σ x_i / n)2 + √(2log(N) / n) and x_i are the rewards we got from the bandit so far.
In the original paper (see section 4), UCB1-Tuned outperformed UCB1 in every experiment.
For UCB1-Normal, C is based on the sample variance and, in contrast to the previous two algorithms, is designed for rewards from normal distributions.
C = √( 16 SV(n) log(N - 1) / n )where the sample variance is  SV(n) = ( Σ x_i2 - n (Σ x_i / n)2 ) / (n - 1)  x_i are the rewards we got from the bandit so far
UCB1-Normal does not normally have an initialisation step. Instead, at each round N , it checks if there’s a bandit that has been played less than the ceiling of 8logN and plays that one instead of looking at the upper confidence index. However, to calculate C we need n to be at least 2 so that we don’t divide by zero. Hence, we will play each bandit twice as an initialisation step.
Phew.... that was a lot of bounds. Some of this code has been presented in previous posts, but I will include it here as well for completeness. First, we need some classes to represent the bandits, one for Bernoulli rewards and one for Normal rewards.
Next, we will define some bases classes for the Agents (and a helper class to keep track of the rewards — if you went through the previous article BanditRewardsLog has been modified to keep track of the sum of the squared rewards).
We still need to define the classes for the Epsilon-Greedy Agent, as we will use it for comparison purposes, and the UCB Agents. They will inherit from the Agent class and need to implement take_action as it is an abstract method.
We now have all the necessary classes for the bandits and the agents. Time for some fun!
We will go straight into throwing all UCB algorithms and Epsilon-Greedy into the battleground. We will consider two cases, one with rewards from a Bernoulli distribution and one with rewards from a Normal distribution.
We will make use of the following utility functions.
def compare_agents(    agents: List[Agent],     bandits: List[Bandit],     iterations: int,    show_plot=True,):    for agent in agents:        logger.info("Running for agent = %s", agent)        agent.bandits = bandits        if isinstance(agent, ucb.UCBAgent):            agent.initialise()        N = iterations - agent.rewards_log.total_actions        agent.take_actions(N)        if show_plot:            cumulative_rewards = np.cumsum(                agent.rewards_log.all_rewards,            )            plt.plot(cumulative_rewards, label=str(agent))    if show_plot:        plt.xlabel("iteration")        plt.ylabel("total rewards")        plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))        plt.show()def get_agents():    agents = [        EpsilonGreedyAgent(),        ucb.UCB1Agent(),        ucb.UCB1TunedAgent(),        ucb.UCB1NormalAgent(),    ]    return agentsdef run_comparison_ucb(bandits):    win_count = [0, 0, 0, 0]        for _ in range(1000):        agents = get_agents()        iterations = 1000        compare_agents(agents, bandits, iterations, show_plot=False)            rewards = [            agent.rewards_log.total_rewards             for agent in agents        ]        win_count[np.argmax(rewards)] += 1            return win_count
May the best algorithm win!
Let’s define our bandits and agents.
probs = [0.6, 0.7, 0.8, 0.9]bernoulli_bandits = [BernoulliBandit(p) for p in probs]agents = get_agents()
Then we can simply do
compare_agents(agents, bernoulli_bandits, 1000, show_plot=True)
It looks like Epsilon-Greedy and UCB1-Tuned are doing better than the other two, but it’s close. We will run_comparison_ucb to repeat this experiment 1000 times and keep count of which agent got the most rewards.
run_comparison_ucb(bernoulli_bandits)
My results were 497 wins for Epsilon-Greedy, 3 for UCB1, 500 for UCB1-Tuned and 0 of UCB1-Normal. Looks like choosing either Epsilon-Greedy or UCB1-Tuned is the way to go.
We repeat the same experiment but this time with bandits that give rewards according to a normal distribution.
means = [3, 5, 7, 9]normal_bandits = [Bandit(m=m, sigma=1) for m in means]
Surprisingly, the worst one seems to be UCB1-Normal... shame. It’s not clear which one is the best from the other three, so we will repeat the experiment a bunch of times and keep count of which agent got the most rewards.
I got 156 wins for Epsilon-Greedy, 364 for UCB1, 480 for UCB1-Tuned and 0 (boooo) for UCB1-Normal.
In this post, we’ve looked into how Upper Confidence Bound bandit algorithms work, coded them in Python and compared them against each other and Epsilon-Greedy (with ε=1/#actions).
We found that UCB1-Tuned performed the best for both Bernoulli and Normal rewards, even though it wasn’t designed for Normal rewards. The worst of the bunch was UCB1-Normal by far — which is probably the reason it is not as popular as the others.
[1] Auer, P., Cesa-Bianchi, N. & Fischer, P. Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning 47, 235–256 (2002). https://doi.org/10.1023/A:1013689704352
eminik355.medium.com
Become a member of Medium and get full access to all stories. Your membership fee directly supports the writers you read.
More from the same author.