SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Both Vanilla UCB and Normal-TS depend on the variances which were assumed known in the previous simulation. We repeat the simulation with the variances incorrectly set to 2. The result is shown in Figure 2b. Evidently MARS, GIRO, and Bootstrapped UCB outperform both Vanilla UCB and Normal-TS when incorrect values for the variances are used. MARS demonstrates superior performance over Bootstrapped UCB and GIRO after an initial set of rounds. Moreover, unlike GIRO and Bootstrapped UCB, MARS does not require the full storage of reward history, resulting in lower computational complexity. As previously observed, PHE with a value of 2.1 demonstrated the poorest performance, whereas PHE with a value of 5.1 has the best performance.
(b)
Figure 2: Cumulative regret for Gaussian bandit. The variances of rewards are true in (a) and wrong in (b). The results are averaged over 2000 realizations.
The MARS and GIRO do not use the tail information of the rewards. However, Vanilla UCB, NormalTS, and Bootstrapped UCB use the distribution and the tail information of the rewards respectively and their performance can deteriorate when the prior knowledge is wrong or conservative. To illustrate this we repeated the simulation for the cases where the rewards admit uniform distribution over  $[-1, 1]$ . The results are shown in Figure 3. It shows that MARS which does not use the distribution of the rewards and the tail information, outperforms the other methods except PHE ( $a = 2.1$ ) in this case. An interesting observation is that PHE ( $a = 5.1$ ) exhibits outstanding performance in a Gaussian setup, yet it performs poorly in a Uniform setup, indicating a strong reliance on the tunable
2The package can be found at https://www.di.ens.fr/olivier.cappe/Code/PymaBandits.
parameter. This dependence on the parameter could pose challenges in real-world applications where the environment is unknown. In such cases methods like MARS and GIRO may be more practical alternatives.
Figure 3: Cumulative regret for uniform bandit. The results are averaged over 2000 realizations.
See Section 5 in the supplement for further simulations.
In this paper, a non-parametric and scale-free confidence bound was proposed based on random sub-sampling of rewards. The confidence bound was used to propose a data-driven UCB algorithm. A regret bound of the method for multi-armed bandit problems with symmetric distribution was presented, and it was shown that the regret bound is always  $O(\log (n))$ . Moreover, the regret bound is  $O(\sum_{i:\Delta_i > 0}\log (n) / \Delta_i)$  for a class of Bandit problems. Numerical experiments show that the proposed method performs well, and it outperforms baselines algorithms when the prior information is mis-specified.
Extending the method to other bandit and reinforcement learning problems are interesting topics for future works e.g. evaluating the method on non-stationary bandits and contextual bandits using the ideas in [13] and [9]. As MARS does not use any concentration inequalities in the construction of the confidence bound, the application of MARS to bandits with heavy tail is another interesting direction. Another interesting future direction is evaluation of the method on asymmetric rewards and robustification of the approach following ideas in [10].
[1] Yasin Abbasi-Yadkori, David Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems, pages 2312-2320, 2011.
[2] Shipra Agrawal and Navin Goyal. Further optimal regret bounds for thompson sampling. In Artificial intelligence and statistics, pages 99-107, 2013.
[3] Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.
[4] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002.
[5] Akram Baransi, Odalric-Embrym Maillard, and Shie Mannor. Sub-sampling for multi-armed bandits. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 115–131. Springer, 2014.
[6] Dorian Baudry, Emilie Kaufmann, and Odalric-Ambrym Maillard. Sub-sampling for efficient non-parametric bandit exploration. Advances in Neural Information Processing Systems, 33:5468-5478, 2020.
[7] Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning, 5(1):1-122, 2012.
[8] Marco C. Campi and Erik Weyer. Guaranteed non-asymptotic confidence regions in system identification. Automatica, 41(10):1751-1764, 2005.
[9] Marco C. Campi and Erik Weyer. Non-asymptotic confidence sets for the parameters of linear transfer functions. IEEE Transactions on Automatic Control, 55(12):2708-2720, 2010.
[10] Algo Care, Balázs Csanád Csáji, and Marco C. Campi. Sign-perturbed sums (SPS) with asymmetric noise: Robustness analysis and robustification techniques. In 2016 IEEE 55th Conference on Decision and Control (CDC). IEEE, 2016.
[11] Marco Dalai, Erik Weyer, and Marco C. Campi. Parameter identification for nonlinear systems: Guaranteed confidence regions through LSCR. Automatica, 43(8):1418-1425, 2007.
[12] Botao Hao, Yasin Abbasi Yadkori, Zheng Wen, and Guang Cheng. Bootstrapping upper confidence bound. In Advances in Neural Information Processing Systems, pages 12123-12133, 2019.
[13] JA Hartigan. Exact confidence intervals in regression problems with independent symmetric errors. The Annals of Mathematical Statistics, pages 1992-1998, 1970.
[14] John Hartigan. Using subsample values as typical values. Journal of the American Statistical Association, 64(328):1303-1317, 1969.
[15] Masoud Moravej Khorasani and Erik Weyer. Non-asymptotic confidence regions for the parameters of EIV systems. Automatica, 115:108873, 2020.
[16] Masoud Moravej Khorasani and Erik Weyer. Non-asymptotic confidence regions for the transfer functions of errors-in-variables systems. IEEE Transactions on Automatic Control, 67(5):2373–2388, 2021.
[17] Branislav Kveton, Csaba Szepesvari, Mohammad Ghavamzadeh, and Craig Boutilier. Perturbed-history exploration in stochastic multi-armed bandits. arXiv preprint arXiv:1902.10089, 2019.
[18] Branislav Kveton, Csaba Szepesvari, Sharan Vaswani, Zheng Wen, Tor Lattimore, and Mohammad Ghavamzadeh. Garbage in, reward out: Bootstrapping exploration in multi-armed bandits. In International Conference on Machine Learning, pages 3601–3610. PMLR, 2019.
[19] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. preprint, page 28, 2018.
[20] Masoud Moravej Khorasani and Erik Weyer. Non-asymptotic confidence regions for errors-in-variables systems. In 18th IFAC symposium on system identification, volume 51, pages 1020–1025, Stockholm, Sweden, July 2018. IFAC-PapersOnLine.
[21] Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527-535, 1952.
[22] Chi-Hua Wang, Yang Yu, Botao Hao, and Guang Cheng. Residual bootstrap exploration for bandit algorithms. arXiv preprint arXiv:2002.08436, 2020.