Upper Confidence Bound for Multi-Armed Bandits Problem
By: Amit Ranjan
Tags: Upper Confidence Bound, Data Science, Multi Armed Bandit, Machine Learning, Reinforcement Learning

In this article we will discuss the Upper Confidence Bound and its steps of algorithm.

As we have seen in the Multi-Armed Bandits Problem article that we have 5 or more number of slot machines in which we bet our money in such a way that our profits are maximized.

Every machine has its own distribution according to which your profits is determined. We use the combination of exploration and exploitation techniques in a balanced manner in order to increase our profits. We cannot have only exploration or only exploitation as they might not give us the optimal solution for our problem.

Upper Confidence Bound

UCB is a deterministic algorithm for Reinforcement Learning that focuses on exploration and exploitation based on a confidence boundary that the algorithm assigns to each machine on each round of exploration. These boundary decreases when a machine is used more in comparison to other machines.

Steps followed in Upper Confidence Bound

At each round n, we consider two numbers for machine m.

-> Nₘ(n) = number of times the machine m was selected up to round n.

-> Rₘ(n) = number of rewards of the machine m up to round n. From these two numbers we have to calculate,

a. The average reward of machine m up to round n, rₘ(n) = Rₘ(n) / Nₘ(n).

b. The confidence interval [ rₘ(n) — Δₘ(n), rₘ(n)+Δₘ(n) ] at round n with, Δₘ(n)= sqrt( 1.5 * log(n) / Nₘ(n) ) We select the machine m that has the maximum UCB, ( rₘ(n)+Δₘ(n) )

I know its very mathematical but let’s understand this by example:

Suppose you are having 5 slot machines and each of the machines has certain amount of distributions associated with them. If any one knows prior of these distributions one can easily maximize the profits. Let’s assume we know the prior distributions of all these machines such that Machine M1 has distribution D1, M2 has distribution D2, M3 has distribution D3, M4 has distribution D4 and M5 has distribution D5.

As we can clearly see that the Machine 3 has the highest distribution. But as in real life you wouldn’t know the distributions prior so we will assume our distributions are same for all the machines.

The formula in the algorithm creates a confidence bound and it is assumed that the confidence bound contains the actual distribution.

These confidence bound is represented as rectangle in the above picture. Right now each of these has same confidence bound so we are taking machine 1 with distribution 1.Now in round 1 after applying UCB algorithm, its confidence bound goes slightly lower.

After applying UCB on D1

Now, again we will iterate through all the machines and find which one has higher confidence bound. Apart from D1 every other distribution has same confidence bound which are greater than D1. So, in this case we can take any of the machines other than M1. This is the beauty of this algorithm. Even if one of the machines is performing better than others the algorithm doesn’t starts exploiting that machine only. It gives chance to explore other machines also.

Let’s say on round 2 you get the confidence bound something like this,

And after round 5 let’s say confidence bound looks like this,

After each round the confidence bound will be shorten for the machine which is selected thereby avoiding biasness towards one machine thus giving chance to others. Lastly the machine which can give greater profit ( in this case its machine 3 ) will have always higher confidence bound than others thereby exploiting it for maximum reward.

This is how Upper Confidence Bound algorithm works. It started with exploring all the machines which has highest confidence bound thereby finding the best machine after some exploration and then started reaping the benefits to maximize the profits.

References: