SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 21: Average regret at horizon 10 000 of the algorithms on a 6-arms Bernoulli bandit problem with means  $\{0.3, 0.4, 0.45, 0.5, 0.52, 0.55\}$
Finally, we check the effect of the learning rate on two Beta bandit problems where means are  $\{0.3, 0.4, 0.45, 0.5, 0.52, 0.55\}$  and one problem has Beta distributions with shape parameter 5 and the other with shape parameter 50. Those two Beta bandit problems are presented in Appendix F.3.3.
Figure 22: Average regret of the algorithms on two 6-arms Beta bandit problems with means  $\{0.3, 0.4, 0.45, 0.5, 0.52, 0.55\}$  and shape parameter of 5 (left) and 50 (right)
When the variance of the distributions is smaller (large shape parameter), we can see that regret curves are better separated and the the bonus of choosing a larger learning rate parameter  $r$  is amplified. When the variance is larger (small shape parameter), regret curves are closer to each other and more within each others quantile tubes. This last experiment confirms all our previous findings about the numerical stability of OIMED.
F.6 IMED with discretized rewards
In Section 1 of this paper we introduced a known trick to reduce the time and memory complexities of MED/IMED. It consists in using algorithms designed for multinomial rewards, by using a discretization procedure on the collected rewards. We furthermore detailed this technique in Appendix F.1, and proved its inevitable sub-optimality for some problems. We end this section by experimenting with multinomial IMED, comparing the empirical performance and computation time of several instances using different grids. In each case, we fix a number of ticks  $M$ , and use the grid  $\{0, \frac{1}{M - 1}, \dots, \frac{M - 2}{M - 1}, 1\} (\{0, 1\} \text{ if } M = 2)$ . Our objective is to identify the number of ticks needed
to get a regret close to IMED, and the evolution of the computation time with the number of ticks. We first run an experiment on the DSSAT bandit problem, and then on a Beta bandit problem where means are close to zero.
Figure 23: Average regret (left) and run time (right) of the algorithms on DSSAT
Alg. (ticks)	IMED (2)	IMED (3)	IMED (5)	IMED (7)	IMED (10)	IMED (20)	IMED
Regret	151.5	75.7	50.9	45.2	44.1	40.25	40.25
Run time	32.8	33.1	36.8	39.4	42.2	44.8	61.1
Table 13: Average regret and run time at horizon 10 000 of the algorithms on DSSAT
Unsurprisingly, the more ticks, the better the regret and the larger the running time, with IMED and k1-IMED providing the range for both metrics. For this problem,  $M = 20$  (intervals of length  $\approx 0.05$  between each tick) seems to be enough to make the algorithm using discretized rewards almost match the true IMED in terms of regret, while being  $25\%$  faster.
We remark that the evolution of the time complexity per time step is not linear in the number of ticks. This is likely because it is actually dependent of the "effective" number of ticks used in memory, i.e. those that contain at least 1 samples. Getting at least 1 sample for each tick/arm takes some time, that depends on the shape of the distributions on the  $[0,1]$  support.
We run a final experiments on a Beta bandit problem that confirms our findings. IMED and k1-IMED again provide the range for both computation time and regret, and for this problem  $M = 20$  is also the value from which IMED and discretized IMED start to match in terms of regret, while the latter is approximately  $20\%$  faster.
Figure 24: Average regret (left) and run time (right) of the algorithms on a 6-arms Beta bandit problem with means  $\{0.05, 0.1, 0.15, 0.2, 0.22, 0.25\}$  and shape parameter of 5
Alg. (ticks)	IMED (2)	IMED (3)	IMED (5)	IMED (7)	IMED (10)	IMED (20)	IMED
Regret	81.8	43.8	32.9	30.8	29.6	29.2	29.2
Run time	31.3	41.5	45.5	46.4	49.2	51.2	62.5
Table 14: Average regret and run time at horizon 10 000 of the algorithms on a 6-arms Beta bandit problem with means  $\{0.05, 0.1, 0.15, 0.2, 0.22, 0.25\}$  and shape parameter of 5