SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now introduce some notation and terminology to describe the duel between a challenger  $k$  and a leader  $\ell$ . In some cases (described below) we perform a greedy duel, for which the winner is  $\operatorname{argmax}\{\mu_k(t),\mu_\ell (t)\}$ . We introduce a variable  $Z_{k}(t)$ , that indicates if the duel played by arm  $k$  was greedy  $(Z_{k}(t) = 0)$  or not  $(Z_{k}(t) = 1)$ , and then  $\widetilde{N}_{k,\ell}(t) = \sum_{s = 1}^{t - 1}\mathbb{I}(k\in \mathcal{A}_{s + 1},\ell_{s} = \ell ,Z_{k}(s) = 1)$ , the number of observations of a challenger  $k$  collected against the leader  $\ell$  after a non-greedy duel.
Using this notation and considering a function  $f: \mathbb{N} \mapsto \mathbb{R}^+$  satisfying  $n = o(f(n))$ , the duels of 0MED/0IMED are implemented as follows: if  $N_{\ell}(t) \leqslant f(\widetilde{N}_{k,\ell}(t))$  or  $\mu_{\ell}(t) \geqslant \mu_{\max}$  the duel is greedy, otherwise MED and IMED are respectively adapted as follows.
Non-greedy 0IMED duel:  $k$  wins if  $L_{k,\ell}(t) + \log (\widetilde{N}_{k,\ell}(t))\leqslant \log (N_{\ell}(t))$  . (10)
Non-greedy 0MED dual:  $k$  wins if  $W_{k}(t) = 1$ , with  $W_{k}(t)\sim \mathrm{Ber}\left(e^{-L_{k,\ell}(t)}\right)$ . (11)
Then, we update  $L_{k,\ell}(t)$  using the portfolio selection algorithm only if the duel was non-greedy, i.e.  $Z_{k}(t) = 1$ , providing  $\mu_{\ell}(t)$  and the reward collected from  $k$ : this is the main ingredient to control the portfolio bias in our analysis. We provide a condensed implementation of 0IMED in Algorithm 1, and the detailed implementation of 0MED in Appendix A.2 (Algorithm 6).
3 Theoretical guarantees
We present our theoretical results on the MED algorithms introduced in Section 2, and some insights from their analysis. We start with the theoretical guarantees of FMED and FIMED.
Theorem 1 (Regret bound for fast MED). Consider  $(F_{1},\ldots ,F_{K})\in \mathcal{F}^{K}$  and  $\mu^{\star} = \max_{k\in [K]}\mu_{k}$ . Then, for any time horizon  $T\in \mathbb{N}$  and  $\varepsilon >0$  MED, IMED, FMED and FIMED all satisfy
$$
\mathbb {E} \left[ N _ {k} (T) \right] \leqslant \frac {\log (T)}{\mathcal {K} _ {i n f} ^ {\mathcal {F}} \left(F _ {k} , \mu^ {\star}\right) - \varepsilon} + o _ {\varepsilon} (\log (T)), \tag {12}
$$
where  $o_{\varepsilon}(\log (T))$  denotes a term that is asymptotically dominated by  $\log (T)$  for a fixed  $\varepsilon$ , but with a polynomial dependency in  $\varepsilon^{-1}$ . Furthermore, all the algorithms are asymptotically optimal.
We prove Theorem 1 in Appendix B. The main result is that FMED and FIMED both preserve the theoretical guarantees of their original algorithm. We apply our analysis to MED and IMED too, in
order to exhibit more precisely the new terms induced by the approximation, and their scaling in  $\varepsilon$ . Note that the components of the  $o_{\varepsilon}(\log (T))$  terms are explicit in the proof. When  $\varepsilon$  is small, we obtain more precisely a scaling in  $\varepsilon^{-6}$ , that allows to obtain a sub-linear problem-independent bound of order  $T^{5 / 6}$ , though not the optimal  $\sqrt{T}$  bound. Note however that the same scaling can be obtained for MED with the proof techniques presented in [Baudry et al., 2023], so the fast implementation do not deteriorate that bound. The strong empirical performance of MED/IMED hints that this result is likely to be not tight, but to the best of our knowledge it remains open to prove that an optimal (instance-dependent) algorithm achieves the  $\mathcal{O}(\sqrt{T})$  problem-independent bound for general bounded distributions. Finally, we emphasize that the analysis of IMD presented in Appendix B is drastically simplified compared to Honda and Takemura [2015], although their result is more general (they allow  $b = -\infty$ ).
Proof sketch. First, the inequality  $\mathcal{K}_k(t) \geqslant \mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(F_k(t), \mu^\star(t))$  (Eq. (5) and (6)) makes FMED and FIMED at least as exploratory as the vanilla algorithms. Their regret is thus smaller in a pre-convergence regime, defined by the time steps for which  $\mu^\star(t) \leqslant \mu^\star - \varepsilon$ . The main challenge is then to prove that using  $(\mathcal{K}_k(t))_{k \in [K]}$  do not lead to over-exploration of the sub-optimal arms in a post-convergence regime, where  $\mu^\star(t) \geqslant \mu^\star - \varepsilon$ . The main ingredient of the proof consists in showing that this kind of scenario can only be caused by events of the form
$$
\exists j \in [ K ]: \left\{A _ {t + 1} = j, N _ {j} (t) = n, \mu_ {j, n} \geqslant \mu^ {\star} + \varepsilon \right\} \text {o r} \left\{A _ {t + 1} = j, N _ {j} (t) = n, \mu_ {j, n} \leqslant \mu^ {\star} - \varepsilon \right\}.
$$
Each of them can cause at most 1 pull of a sub-optimal arm. With union bounds and Hoeffding's inequality we obtain an additional  $\mathcal{O}(\varepsilon^{-2})$  term in the regret bound compared to MED and IMED.
We now provide a very similar result on OMED and OIMED, under some additional assumptions: first, we need the largest mean to be well-separated from  $B$  to control the portfolio bias. This is a mild assumption, since the decision-maker can also choose to slightly increase the value of  $B$  used by the algorithm if it does not clearly hold. Then, we require strong guarantees on the portfolio regret, that we discuss at the end of this section.
Theorem 2 (Regret bound for online MED algorithms). Consider  $(F_{1},\ldots ,F_{K})\in \mathcal{F}^{K}$ , and  $\mu^{\star} = \max_{k\in [K]}\mu_k < \mu_{max}$ , for a known  $\mu_{max} < B$ . Assume that OMED and OIMED use a portfolio selection algorithm ALG that satisfies for any  $(k,\ell)\in [K]^2$  and  $n\in \mathbb{N}$  the deterministic guarantee
$$
\left| R _ {k, \ell , n} \right| = o (n) \quad (\text {s u b - l i n e a r a b s o l u t e p o r t f o l i o r e g r e t}). \tag {13}
$$
Then, both OMED and OIMED satisfy Equation (12) (with a different second-order term) and are asymptotically optimal: they enjoy the same asymptotic guarantees as MED and IMED.
Strikingly, the performance of OMED/OIMED only depend on the regret bounds of the portfolio regret. As detailed below, this is largely due to our modified algorithm's structure.
Proof sketch. The general proof scheme is inspired by [Chan, 2020], that proposed a similar. duel-based approach. The main difficulty consists in controlling the deviation of  $L_{k,\ell_t}(t)$  from  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(F_{k,\ell_t}(t),\mu_{\ell_t}(t))$  , denoting by  $F_{k,\ell_t}(t)$  the empirical distribution of the  $\widetilde{N}_{k,\ell_t}(t)$  observations used to update  $L_{k,\ell_t}(t)$  . Both over and under-estimation are bad: one may prevent sufficient exploration of the best arm, while the other may lead to over-exploration of sub-optimal arms. Building on the discussion of Section 2, our proof relies on the following crucial result: for fixed arms  $(k,\ell)$  , a sample size  $\widetilde{N}_{k,\ell}(t) = n$  , and any threshold  $\mu$  it holds that
$$
n \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k, \ell , n}, \mu\right) = L _ {k, \ell , n} + R _ {k, \ell , n} + B _ {k, \ell , n} (\mu), \tag {14}
$$
where  $B_{k,\ell,n}(\mu)$  is the portfolio bias with respect to a fixed threshold  $\mu$ , defined in (8), and where  $L_{k,\ell,n}$  and  $F_{k,\ell,n}$  are used to denote  $L_{k,\ell}(t)$  and  $F_{k,\ell}(t)$  when  $\widetilde{N}_{k,\ell}(t) = n$ , for simplicity. In the proofs, the portfolio regret is controlled by assumption, but the bias requires more careful examination. To analyze it, we prove the following result (details in Appendix E). For  $i \leqslant n$ , we denote by  $t_{k,\ell,i}$  the time when the  $i$ -th iteration of  $L_{k,\ell,n}$  occurred.
i.e. with bounds depending on  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  and not  $\Delta_k^2$  or k1
Lemma 1 (Deviations of  $B_{k,\ell,n}(\mu)$ ). Let  $\mu \in (0,1)$ , and set  $C_{k,\ell,i} = \frac{B - \mu_{\ell}(t_{k,\ell,i})}{B - \mu}$  it holds that
$$
- \sum_ {i = 1} ^ {n} \log \left(C _ {k, \ell , i} ^ {- 1}\right) \mathbb {I} \left(\mu_ {\ell} \left(t _ {k, \ell , i}\right) \geqslant \mu\right) \leqslant B _ {k, \ell , n} (\mu) \leqslant \sum_ {i = 1} ^ {n} \log \left(C _ {k, \ell , i}\right) \mathbb {I} \left(\mu \geqslant \mu_ {\ell} \left(t _ {k, \ell , i}\right)\right). \tag {15}
$$
We always use this result for  $\mu \in (\max_{k:\mu_k < \mu^*}\mu_k,\mu^\star)$ , and  $\mu_{\ell}(t_{k,\ell ,i}) < \mu_{\mathrm{max}}$ . Hence, the upper bound (resp. lower bound) of (15) is expected to be sub-linear when  $\ell = 1$  (resp.  $\ell \neq 1$ ), thanks to the use of different estimates according to the leader's identity. The rate of convergence of these sums is also important: our proof techniques work if  $n = o(N_{\ell}(t_{k,\ell ,n}))$ , which justifies the greedy duels. Indeed, we need  $e^{an}\mathbb{P}(B_{k,\ell ,n} < - nx)$  to be small enough (for some  $a,x > 0$  and large  $n$ ), and with this trick we obtain  $e^{an - f(n)}y_x\to 0$  (for some  $y_{x}$  depending on  $x$ ).
The modifications introduced in 0MED and 0IMED guarantee a small absolute bias with large probability. It is natural to ask whether they are necessary. We cannot formally prove it, but our experiments in Section 4 suggest that the regret of 0IMED/0MED may be linear without them.
Assumption on the portfolio regret As detailed in Section 2 there are several candidates for the implementation of 0IMED/IMED. The literature on regret lower bounds is however scarce. In [Gofer and Mansour, 2016], the authors characterize algorithms with non-negative regret, but for linear losses. Guzmán et al. [2021] showed that the regret of FTRL algorithms is bounded as  $\mathcal{O}(\sqrt{T})$  both from above and below, in a setting that encompasses portfolio selection. Unfortunately, FTRL is computationally inefficient, which makes it unsuitable for our application. We used Soft-Bayes for its simplicity and computational efficiency, but it might not have the desired lower bound. Our problem seems to be among the first where a large negative regret is detrimental, and obtaining a portfolio algorithm with both small computational complexity and a regret lower bound is an open question. On the other hand, it may be that this lower bound requirement could be relaxed in the analysis, and that for example a high-probability lower bound could be enough. Our experimental study in the next section supports the hypothesis that using Soft-Bayes in 0IMED gives good regret bounds for the bandit problem.
We numerically assess the regret and run time of the two novel approaches presented in this paper. We benchmark our approaches with the known asymptotically optimal algorithms: IMED [Honda and Takemura, 2015], KL-UCB [Cappé et al., 2013] and NPTS [Riou and Honda, 2020]; and the more efficient but sub-optimal UCB [Auer et al., 2002], k1-UCB [Cappé et al., 2013] and k1-IMED (that denotes the binarized version of IMED). We provide the pseudo-code of KL-UCB and NPTS in Appendix A.2 for completeness. Our code is available in the supplementary material of the paper.
We showcase our main findings by presenting a selection of experiments, focusing on FIMED and OIMED, while we present a broader set of experiments and more detailed results in Appendix F. In each case, we plot the average run time and regret of each algorithm, along with their quantiles  $10\% - 90\%$ . The run times are dependent of the Python implementation of our algorithms, and are thus only indicative. We refer to Appendix F.1 for discussions on their theoretical complexity.
Distributions from DSSAT We first consider the crop-management optimization problem introduced in Section 1 and detailed for instance in [Baudry et al., 2021a], as it illustrates the theoretical advantage of asymptotically optimal algorithms (Figure 1). A learner needs to select a planting date for maize grains among seven possible options. For each distribution, the rewards are drawn uniformly at random among  $10^{4}$  points sampled from the DSSAT simulator, in order to emulate the distributions at a reduced cost. Our results, displayed in Figure 2, show that on this problem optimal algorithms also achieve better empirical performance. IMED, NPTS and KL-UCB perform similarly, while k1-IMED and UCB achieve significantly larger regret. Furthermore, FIMED is 10 times faster than IMED for  $T = 10^{4}$  (and as fast as k1-UCB), with almost no difference in terms of regret. As expected, fIMED is as fast as UCB and k1-IMED. It achieves better regret than those algorithms, but its performance is deteriorated compare to fIMED. This can be seen as the cost of the ability to forget
The dataset is available in the supplementary material of the paper
past observations. In summary, this experiment illustrates the respective benefits of our two novel approaches: FIMED is the fastest algorithm among those with the smallest regret, while OIMED has the smallest regret among the fastest algorithm.
Figure 2: Average regret (left) and run time (right) of the algorithms on the DSSAT bandit problem
Bernoulli bandit For our second experiment we consider a Bernoulli bandit with several means close to 0.5, for which all optimal algorithms match their implementation with the Bernoulli KL-divergence. Intuitively, sequences of 0 and 1 with high variance may lead to the most potentially confusing inputs for portfolio algorithms, so our objective is to check the performance of OIMED in that case. Our results, summarized in Figure 3, are promising: the average regret of OIMED is on par with other algorithms, while it still is among the fastest. In that experiment only UCB is sub-optimal, and performs much worse than the other algorithms.
Figure 3: Average regret (left) and run time (right) of the algorithms on a 6-arms Bernoulli bandit problem with means  $\{0.3, 0.4, 0.45, 0.5, 0.52, 0.55\}$ .
Importance of the duel-based structure In Section 3 we detailed our theoretical motivations for introducing a duel-based approach when approximating  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  with a portfolio algorithm. We now provide empirical evidence that this change may prevent OIMED from suffering a linear regret. In Figure 4, we compare the regret of OIMED and a variant of OIMED that does not use the duel-based structure of Algorithm 1 for the two experiments introduced in previous paragraphs. In that algorithm, the online estimates of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  are updated at each pull of an arm with the current best empirical mean. We observe that the regret of the "no duel" variant of OIMED may be better in some cases: for the DSSAT experiment, its average regret matches the best algorithms of Figure 2. However, on the Bernoulli experiment its regret is linear. Overall, OIMED needs a modification like the duels to ensure that it has sub-linear regret.
Further experiments In Appendix F we show that FMED and Omed exhibit the same characteristics as FIMED and OIMED, as expected. We also consider other Bernoulli bandits, for instance with means close to the boundaries of the support. We then show that the conclusions of this section still hold on problems with distributions of various shapes, by testing our algorithms on several examples of Beta distributions. We also investigate the sensitivity of OIMED to the learning rate of Soft-Bayes, which is the only hyper-parameter of the algorithm. Finally, we perform experiments with the discretized IMED algorithms that is briefly presented in the introduction and discussed more thoroughly in Appendix D.
Figure 4: Average regrets of 0IMED and "OIMED no duels" on the previous 7-armed DSSAT bandit (left) and 6-armed Bernoulli bandit (right).
We introduced methods to compute efficiently approximations of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  and demonstrated their use in algorithms for regret minimization in stochastic bandits. The FMED and FIMED variants have the same asymptotic optimality properties as the base MED algorithms, but have a much reduced computational complexity. OMED and OIMED push the computational gains further and are also memory efficient. They can also be asymptotically optimal, under the hypothesis that the portfolio algorithm they use satisfies a deterministic regret lower bound.
While our experiments show the good practical performance of 0IMED and 0MED with the Soft-Bayes portfolio algorithm, this question is however still open: can we have a portfolio algorithm which is computationally efficient, does not store all past gains in memory and has sub-linear regret upper and lower bounds?
Finally, our work towards enabling the use of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  in a computationally efficient way has potential applications beyond regret minimization in stochastic bandits. First, similar quantities are used in RL algorithms like IMED-RL [Pesquerel and Maillard, 2022]. Second, other bandit tasks like best arm identification also have complexities that depend on a  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  quantity, and could benefit from faster variants of the algorithms that need to compute it [Jourdan et al., 2022].
R. Degenne acknowledges the funding of the French National Research Agency under the project FATE (ANR-22-CE23-0016-01). This work beneficiated from the support of the French Ministry of Higher Education and Research, of Inria and of the Hauts-de-France region.
T. L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4-22, 1985.
T. Lattimore and C. Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
L. Orseau, T. Lattimore, and S. Legg. Soft-bayes: Prod for mixtures of experts with log-loss. In International Conference on Algorithmic Learning Theory, pages 372-399. PMLR, 2017.
F. Pesquerel and O.-A. Maillard. Imed-rl: Regret optimal learning of ergodic markov decision processes. In NeurIPS 2022-Thirty-sixth Conference on Neural Information Processing Systems, 2022.
C. Riou and J. Honda. Bandit algorithms based on Thompson sampling for bounded reward distributions. In Algorithmic Learning Theory, pages 777-826. PMLR, 2020.
W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-294, 1933.
C.-E. Tsai, H.-C. Cheng, and Y.-H. Li. Online self-concordant and relatively smooth minimization, with applications to online portfolio selection and learning quantum states. In International Conference on Algorithmic Learning Theory, pages 1481-1483. PMLR, 2023.
J. Zimmert, N. Agarwal, and S. Kale. Pushing the efficiency-regret pareto frontier for online learning of portfolios and quantum states. In Conference on Learning Theory, pages 182-226. PMLR, 2022.
A Index, notation and algorithms
A.1 Index and notation
In this section we summarize the notation used in the proofs of Theorem 1 and 2, and recall some of the definitions presented in the paper.
-  $[K] = \{1, \ldots, K\}$ .
-  $\mathcal{F}$  the family of distributions supported on  $[b, B] \subset \mathbb{R}$ .
-  $(F_{1},\ldots ,F_{K})\in \mathcal{F}^{K}$  denote the distribution of the  $K$  arms of a bandit problem. Their expectations are denoted by  $\mu_1,\dots ,\mu_K$ . Without loss of generality we assume in the proofs that  $\mu_{1} = \max_{k\in [K]}\mu_{k}$ , and also define  $x_{k} = \frac{\mu_{1} + \mu_{k}}{2}$ .
- For any  $F \in \mathcal{F}$ ,  $\mu \in [b, B]$ ,  $\mathcal{K}_{\inf}^{\mathcal{F}}(F, \mu) = \max_{\lambda \in [0,1]} \mathbb{E}_F\left[\log \left(1 - \lambda \frac{X - \mu}{B - \mu}\right)\right]$ .
-  $A_{t + 1}$ : arm selected by the learner at the beginning of round  $t + 1$ .
-  $N_{k}(t)$ : number of pulls of arm  $k$  up to (including) time step  $t$ .
-  $(X_{k,n})_{k\in [K],n\in \mathbb{N}}$  : sequences of rewards:  $X_{k,n}$  is the  $n$ -th reward collected from arm  $k$ .
-  $F_{k}(t)$ : empirical distribution of arm  $k$  at time  $t$ . We also denote for convenience by  $F_{k,n}$  the empirical distribution of arm  $k$  after collecting  $n$  observations:  $F_{k,N_k(t)} = F_k(t)$ .
-  $\mu_{k}(t)$  and  $\mu_{k,n}$  denote the empirical mean of arm  $k$  respectively at time  $t$  and after collecting  $n$  observations.
-  $\mu^{\star}(t) = \max_{k\in [K]}\mu_k(t), k^{\star}(t) = \operatorname*{argmax}_{k\in [K]}\mu_k(t).$
- For a randomized algorithm, we use the shorthand notation  $p_k(t) \coloneqq \mathbb{P}(A_{t+1} = k|\mathcal{F}_t)$ , where  $\mathcal{F}_t$  denotes the  $\sigma$ -algebra generated by the rewards collected by the learner up to (and including) time  $t$ . We call  $p_k(t)$  the sampling probability of arm  $k$  at time  $t$ .
- We use "MED" a general terminology to call some existing bandit algorithms based on the computation of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(F_k(t), \mu^\star(t))$ . Some of these algorithms are DMED [Honda and Takemura, 2010], MED [Honda and Takemura, 2011] and IMED [Honda and Takemura, 2015]. Hence, MED denotes a specific algorithm and MED a general family of algorithms in this paper.
FMED and FIMED:
-  $s_k(t)$ : time of the last pull of arm  $k$  at time  $t$ , formally  $s_k(t) = \inf \{s \leqslant t : N_k(s) = N_k(t)\}$ .
#
$$
\mathcal {K} _ {k} (t) = \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k} \left(s _ {k} (t)\right), \mu^ {\star} \left(s _ {k} (t)\right)\right) + \lambda_ {k} \left(s _ {k} (t)\right) \left(\mu^ {\star} (t) - \mu^ {\star} \left(s _ {k} (t)\right)\right),
$$
is a first order approximation of  $\mathcal{K}_{\inf}^{\mathcal{F}}(F_k(t),\mu^\star (t))$  satisfying  $\mathcal{K}_k(t)\leqslant \mathcal{K}_{\inf}^{\mathcal{F}}(F_k(t),\mu^\star (t))$  where for any  $s\in \mathbb{N}$
$$
\lambda_ {k} (s) = \operatorname * {a r g m a x} _ {\lambda \in [ 0, \frac {1}{B - \mu^ {\star} (s)} ]} \mathbb {E} _ {F _ {k} (s)} \left[ \log (1 - \lambda (X - \mu^ {\star} (s))) \right].
$$
OMED and OIMED:
-  $\ell_t$ : leader at round  $t$ , defined as  $\ell_t \in \operatorname{argmax}_{k \in [K]} N_k(t)$ . Ties are broken in favor of the arm with the best empirical mean. If there are still several candidates choose one of them randomly.
- Challengers: all arms except the leader  $\ell_t$
-  $\mathcal{A}_{t + 1}$ : set of arms to pull at the beginning of round  $t + 1$
- Duel: comparison between a challenger arm  $k$  and the leader  ${\ell }_{t}$  at a time  $t$  .
-  $Z_{k}(t) = 0$  if the duel for challenger  $k$  is a greedy comparison,  $Z_{k}(t) = 1$  otherwise (e.g. if the index comparison of IMED is used). We use the expressions "greedy duel" and "non-greedy duel" to distinguish the two cases.
-  $\widetilde{N}_k(t) = \sum_{s=1}^{t-1} \mathbb{I}(k \in \mathcal{A}_{t+1}, \ell_s = \ell_t, Z_k(s) = 1)$ : number of observations collected by arm  $k$  as a challenger against the current leader  $\ell(t)$  and thanks to a non-greedy duel. We denote by  $\widetilde{F}_k(t)$  the empirical distribution associated with these observations, and use the shorthand notation  $\widetilde{F}_{k,n}$  for  $\widetilde{N}_k(t) = n$ .
- For any  $k \in [K], \ell \in [K], n \in \mathbb{N}, X_{k,\ell,n}$  is the  $n$ -th observation collected from arm  $k$  against the leader  $\ell$  after a non-greedy duel
- The online estimate used by arm  $k$  against leader  $\ell$  with  $\widetilde{N}_k(t) = n$  is
$$
L _ {k, \ell , n} := \sum_ {i = 1} ^ {n} \log \left(1 - \lambda_ {k, i} \left(\frac {X _ {k , \ell , i} - \mu^ {\star} (t _ {k , \ell , i})}{B - \mu^ {\star} (t _ {k , \ell , i})}\right)\right),
$$
where  $t_{k,\ell,i}$  is the time step satisfying  $\{\widetilde{N}_k(t + 1) = \widetilde{N}_k(t) + 1, \widetilde{N}_k(t) = n\}$ , and  $(\lambda_{k,i})_{i \in \mathbb{N}}$  is provided by a portfolio selection algorithm.
$$
R _ {k, \ell , n} := \max  _ {\lambda \in [ 0, 1 ]} \sum_ {i = 1} ^ {n} \log \left(1 - \lambda \left(\frac {X _ {k , \ell , i} - \mu^ {\star} (t _ {k , \ell , i})}{B - \mu^ {\star} (t _ {k , \ell , i})}\right)\right) - L _ {k, \ell , n}
$$
is the portfolio regret.
- For any  $\mu \in [b, B)$ ,