johnmyleswhite/BanditsBook — pull request #10 by rmminusrslash (2014-08-05 08:27:00+00:00)
Title: for python: prevent deterministic start for each simulation (usually one...

Hey John,

really great book, thank you for that! Very great explanations and encouraged me a lot to play with the boundary conditions.

I spotted a little error when experimenting.

For 5 arms, the start percentage of chosing the right arm at the first draw should be 20% on average over the 1000 experiments that are the default, but it was around 2% for epsilon greedy with epsilon=.1. That is equivalent to always choosing an inferior arm at the first draw for the 90% share of epsilon greedy. If the expected values from the arms are similar, this behaviour also influences the following steps for quite a number of draws.

The reason was the function ind_max which chooses the first index if all values are equal.

In python/algorithms/epsilon_greedy/test_standard.py, you permute the means list. That means it can also happen that the algorithm choses deterministicly the best arm if the best arm happens to be the first.

After permuting the means, the arms' order is never touched again, so the behaviour of drawing the same arm (whichever one) persists over all similation, they all start with the same arm.

I could provide my plots if the explanation was poor,

Proposed solution: In case of same valued arms, choose randomly.

Lina