SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Jacob D Abernethy, Chansoo Lee, and Ambuj Tewari. Fighting bandits with a new kind of smoothness. In Advances in Neural Information Processing Systems, volume 28, pages 2197-2205, 2015.
[2] Chamy Allenberg, Peter Auer, László Györfi, and György Ottucsák. Hannan consistency in online learning in case of unbounded losses under partial monitoring. In International Conference on Algorithmic Learning Theory, pages 229-243, 2006.
[3] Noga Alon, Nicolò Cesa-Bianchi, Ofer Dekel, and Tomer Koren. Online learning with feedback graphs: Beyond bandits. In Proceedings of The 28th Conference on Learning Theory, volume 40, pages 23-35, 2015.
[4] Jean-Yves Audibert and Sébastien Bubeck. Minimax policies for adversarial and stochastic bandits. In Conference on Learning Theory, volume 7, pages 1-122, 2009.
[5] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing, 32(1):48-77, 2002.
[6] Gábor Bartók. A near-optimal algorithm for finite partial-monitoring games against adversarial opponents. In Proceedings of the 26th Annual Conference on Learning Theory, volume 30, pages 696-710, 2013.
[7] Gábor Bartók, Dávid Pál, and Csaba Szepesvári. Minimax regret of finite partial-monitoring games in stochastic environments. In Proceedings of the 24th Annual Conference on Learning Theory, volume 19, pages 133–154, 2011.
[8] Gábor Bartók, Navid Zolghadr, and Csaba Szepesvári. An adaptive algorithm for finite stochastic partial monitoring. In the 29th International Conference on Machine Learning, pages 1–20, 2012.
[9] Sébastien Bubeck and Aleksandrs Slivkins. The best of both worlds: Stochastic and adversarial bandits. In Proceedings of the 25th Annual Conference on Learning Theory, volume 23, pages 42.1-42.23, 2012.
[10] Sébastien Bubeck, Michael Cohen, and Yuanzhi Li. Sparsity, variance and curvature in multiarmed bandits. In Proceedings of Algorithmic Learning Theory, volume 83 of Proceedings of Machine Learning Research, pages 111-127, 2018.
[11] Sébastien Bubeck, Yuanzhi Li, Haipeng Luo, and Chen-Yu Wei. Improved path-length regret bounds for bandits. In Conference on Learning Theory, pages 508-528, 2019.
[12] Nicolò Cesa-Bianchi, Gábor Lugosi, and Gilles Stoltz. Regret minimization under partial monitoring. Mathematics of Operations Research, 31(3):562-580, 2006.
[13] Christoph Dann, Chen-Yu Wei, and Julian Zimmert. A blackbox approach to best of both worlds in bandits and beyond. arXiv preprint arXiv:2302.09739, 2023.
[14] Liad Erez and Tomer Koren. Towards best-of-all-worlds online learning with feedback graphs. In Advances in Neural Information Processing Systems, volume 34, pages 28511-28521, 2021.
[15] Dean Foster and Alexander Rakhlin. No internal regret via neighborhood watch. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22, pages 382-390, 2012.
[51] Kai Zheng, Haipeng Luo, Ilias Diakonikolas, and Liwei Wang. Equipping experts/bandits with long-term memory. In Advances in Neural Information Processing Systems, volume 32, pages 5929-5939, 2019.
[52] Alexander Zimin and Gergely Neu. Online learning in episodic Markovian decision processes by relative entropy policy search. In Advances in Neural Information Processing Systems, volume 26, pages 1583-1591, 2013.
[53] Julian Zimmert and Yevgeny Seldin. Tsallis-INF: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research, 22(28):1-49, 2021.
Table 3 summarizes the symbols used in this paper.
Table 3: Notation
Symbol	Meaning
Pk	(k-1)-dimensional probability simplex
T∈N	time horizon
k∈N	number of arms (or actions)
At∈[k]	arm (or action) chosen by learner at round t
s≤k	maxt||lt|0, sparsity level of losses
L2	∑Tt=1||lt|2
L∈[0,1]k×d	loss matrix
Σ	set of feedback symbols
Φ∈Σk×d	feedback matrix
d∈N	number of outcomes
m∈N	maximum number of distinct symbols in a single row of Φ
xt∈[d]	outcome chosen by opponent at round t
qt∈Pk	output of FTRL at round t
pt∈Pk	arm selection probability at round t
Φt:Pk→R	regularizer of FTRL at round t
ηt=1/βt>0	learning rate of FTRL at round t
ψnS:Rk+→R	∑i=1k xi log xi, negative Shannon entropy
ψLB:Rk+→R	∑i=1k log(1/xi), log-barrier
φnS:R+→R	x log x
φLB:R+→R	log(1/x)
ht	penalty component at round t
zt	stability component at round t
ωt	stability componentztintroduced in (6) (Section 5.1.1)
νt	stability componentztintroduced in (7) (Section 5.1.2)
V′t	stability componentztintroduced in (13) (Appendix B)
C≥0	corruption level
B Detailed background and algorithm description omitted from Section 6
This section supplements the material that was briefly explained in Section 6. We also consider full information (FI) and MAB as well as non-degenerate locally observable PM (PM-local), and let  $\mathcal{M}$  be a such underlying model.
B.1 Exploration by optimization and its extension
Exploration-by-optimization We first describe the approach of EbO by Lattimore and Szepesváři [32], which is a strong technique to bound the regret in PM with local observability. The key idea behind EbO is to minimize a part of a regret upper bound of the FTRL with the Shannon entropy. Recall that  $\mathcal{H}$  is the set of all functions from  $[k] \times \Sigma$  to  $\mathbb{R}^d$ . Then in EbO we consider the choice of  $(p_t, G_t) \in \mathcal{P}_k \times \mathcal{H}$  to minimize the sum of the stability and transformation terms for the worst-case outcome given as follows (also defined in (9)):
$$
\mathsf {S T} (p, G; q _ {t}, \eta_ {t}) = \max  _ {x \in [ d ]} \left[ \frac {(p - q _ {t}) ^ {\top} \mathcal {L} e _ {x}}{\eta_ {t}} + \frac {\operatorname {b i a s} _ {q _ {t}} (G ; x)}{\eta_ {t}} + \frac {1}{\eta_ {t} ^ {2}} \sum_ {a = 1} ^ {k} p _ {a} \Psi_ {q _ {t}} \left(\frac {\eta_ {t} G (a , \Phi_ {a x})}{p _ {a}}\right) \right]. \tag {11}
$$
Note that the first and third terms in (11) correspond to the stability and transformation terms (divided by the learning rate  $\eta_t$ ), respectively. We define the optimal value of the optimization problem by
Algorithm 1: BOBW algorithm with a game-dependent bound for locally observable games
1 input:  $B$    
2 for  $t = 1,2,\ldots$  do   
3 Compute  $q_{t}$  using (1).   
4 Solve (10) to determine  $V_{t}^{\prime} = \max \{0,\mathrm{opt}_{q_{t}}^{\prime}(\eta_{t})\}$  and the corresponding solution  $p_t$  and  $G_{t}$    
5 Sample  $A_{t}\sim p_{t}$  and observe feedback  $\sigma_t\in \Sigma$    
6 Compute  $\widehat{y}_t = G_t(A_t,\sigma_t) / p_{tA_t}$  and update  $\beta_{t}$  using (13).
$\mathrm{opt}_q(\eta) = \min_{(p,G)\in \mathcal{P}_k\times \mathcal{H}}\mathsf{ST}(p,G;q_t,\eta)$  and its truncation at round  $t$  by  $V_{t} = \max \{0,\mathrm{opt}_{q_{t}}(\eta_{t})\}$  (appeared in Table 2). Note that this optimization problem is convex and can be solved numerically by using standard solvers [32].
Extending exploration-by-optimization While the vanilla EbO is a strong tool to derive a regret bound in PM-local, it only has a guarantee for the adversarial regime. Recall that in the self-bounding technique, we require a lower bound of the regret expressed in terms of  $q_{t}$  (see Lemma 1). However, when we use the vanilla EbO, it may make a certain action selection probability  $p_{ta}$  for some action  $a$  become extremely small even when the output of FTRL  $q_{ta}$  is far from zero [32], which makes it impossible for us to use the self-bounding technique.
To solve this problem, the vanilla EbO was recently extended so that it is applicable to the stochastic regime (and the adversarial regime with a self-bounding constraint) for PM-local [46]. We define  $\mathcal{P}_k'(q,\mathcal{M})$  for a class of games  $\mathcal{M}$ , which is the extended version of  $\mathcal{P}_k'(q)$  in (10), by
$$
\mathcal {P} _ {k} ^ {\prime} (q, \mathcal {M}) = \{p \in \mathcal {P} _ {k}: \operatorname {c o n d} (q, \mathcal {M}) \} \quad \text {w i t h} \quad \operatorname {c o n d} (q, \mathcal {M}) = \left\{ \begin{array}{l l} p = q & \text {i f \mathcal {M} i s F I o r M A B}, \\ p \geq q / (2 k) & \text {i f \mathcal {M} i s P M - l o c a l}. \end{array} \right.
$$
We then consider the following optimization problem, which can be seen as a slight generalization of the approach developed in [46]:
$$
\left(p _ {t}, G _ {t}\right) = \underset {p \in \mathcal {P} _ {k} ^ {\prime} \left(q _ {t}, \mathcal {M}\right), G \in \mathcal {H}} {\arg \min } \mathrm {S T} \left(p, G; q _ {t}, \eta_ {t}\right), \tag {12}
$$
where the feasible region  $\mathcal{P}_k$  of  $p$  is replaced with  $\mathcal{P}_k'(q, \mathcal{M})$ . We define the optimal value of (12) by  $\mathrm{opt}_q'(\eta, \mathcal{M})$  and its truncation at round  $t$  by  $V_t'(\mathcal{M}) = \max \{0, \mathrm{opt}_{q_t}'(\eta_t, \mathcal{M})\}$ . We will abbreviate  $\mathcal{M}$  when it is clear from a context. The following proposition shows that the component of the regret in (9) can be made small enough even if the feasible region is restricted to  $\mathcal{P}_k'(q, \mathcal{M}) \subset \mathcal{P}_k$ .
Proposition 1. Let  $\mathcal{M}$  be an underlying model. If  $\mathcal{M}$  is FI, MAB, or PM-local with  $\eta \leq 1 / (2mk^2)$ ,
$$
\operatorname {o p t} _ {*} ^ {\prime} (\eta) := \sup  _ {q \in \mathcal {P} _ {k}} \operatorname {o p t} _ {q} ^ {\prime} (\eta) \leq \bar {V} (\mathcal {M}) := \left\{ \begin{array}{l l} 1 / 2 & i f \mathcal {M} i s F I \\ k / 2 & i f \mathcal {M} i s M A B \\ 3 m ^ {2} k ^ {3} & i f \mathcal {M} i s P M - l o c a l. \end{array} \right.
$$
One can immediately obtain this result by following the same lines as [32, Propositions 11 and 12] and [46, Lemma 5].
B.2 Algorithm
We use the negative Shannon entropy regularizer  $\Phi_t = \frac{1}{\eta_t} \psi^{\mathrm{nS}}$  for (1) with learning rate  $\eta_t = 1 / \beta_t$  with
$$
\beta_ {1} = B \sqrt {\frac {\log (1 + T)}{\log k}} \quad \text {a n d} \quad \beta_ {t + 1} = \beta_ {t} + \frac {c _ {1} V _ {t} ^ {\prime}}{\sqrt {\bar {V} h _ {1} + \sum_ {s = 1} ^ {t - 1} V _ {s} ^ {\prime} h _ {s + 1}}}, \tag {13}
$$
where  $B = 1/2$  for FI,  $B = k/2$  for MAB, and  $B = 2mk^2$  for PM-local, which corresponds to the learning rate in Definition 2 with  $h_t \gets H(q_t)$ ,  $z_t \gets V_t'$ ,  $\bar{z}_t \gets 0$ , and  $c_2 \gets 0$ . Algorithm 1 summarizes the proposed algorithm.
B.3 Main result
Let  $r_{\mathcal{M}}$  be 1 if  $\mathcal{M}$  is FI or MAB, and  $2k$  if  $\mathcal{M}$  is PM-local. Then we have the following bound.
Corollary 6 (Extended version of Corollary 5). Let  $\mathcal{M}$  be  $FI$ ,  $MAB$ , or  $PM$ -local. Then the above algorithm with  $c_{1} = \sqrt{\log(1 + T) / 2}$  (Algorithm 1) achieves
$$
\operatorname {R e g} _ {T} \leq \mathbb {E} \left[ \sqrt {2 \sum_ {t = 1} ^ {T} V _ {t} ^ {\prime} \log (k) \log (1 + T)} \right] + O (B \sqrt {\log (k) \log (T)})
$$
in the adversarial regime, and
$$
\operatorname {R e g} _ {T} = O \left(\frac {r _ {\mathcal {M}} \bar {V} \log (T) \log (k T)}{\Delta_ {\min }} + \sqrt {\frac {C r _ {\mathcal {M}} \bar {V} \log (T) \log (k T)}{\Delta_ {\min }}} + B \sqrt {\log (k) \log (T)}\right)
$$
in the adversarial regime with a  $(\Delta, C, T)$  self-bounding constraint.
The bound for the adversarial regime with a self-bounding constraint with  $C = 0$  yields the bound in the stochastic regime, which is optimal up to logarithmic factors in FI and MAB, and has the same order as the bounds in [46, Theorem 6].
The bound for the adversarial regime has a form similar to [32] and is game-dependent in the sense that it can be bounded by the empirical difficulty  $V_{t}^{\prime}$  of the current game. In addition, we can also obtain the worst-case bound by replacing  $V_{t}^{\prime}$  with its upper bound  $V$ . This bound is optimal up to  $\log (T)$  factor in FI and  $\log (k) \log (T)$  factor in MAB, and is a factor of  $\sqrt{\log T}$  worse than the best known bound in [32], which can be seen as the cost for the BOBW guarantee (see also Table 2).
C Additional related work
This appendix provides additional related work that could not be included in the main body due to the numerous studies related to this paper.
Multi-armed bandits In the stochastic regime, it is known that the optimal regret is approximately expressed as  $\mathrm{Reg}_T = O(k\log T / \Delta_{\mathrm{min}})$  [29]. In the adversarial regime (a.k.a. non-stochastic regime), it is known that the Online Mirror Descent (OMD) framework with the (negative) Tsallis entropy regularizer achieves  $O(\sqrt{kT})$  regret bounds [1, 4], which match the lower bound of  $\Omega (\sqrt{kT})$  [5].
Data-dependent bound In the adversarial MAB, algorithms with various data-dependent regret bounds have been developed. Typical examples of such bounds are first-order bounds dependent on the cumulative loss and second-order bounds depending on sample variances in losses. Allenberg et al. [2] provided an algorithm with a first-order regret bound of  $O(\sqrt{kL^*\log k})$  for  $L^{*} = \min_{a\in \mathcal{A}}\sum_{t = 1}^{T}\langle \ell_{t},a\rangle$ . Second-order regret bounds have been shown in some studies, e.g., [10, 19, 49]. In particular, Bubeck et al. [10] provided the regret bound of  $O(\sqrt{Q_2\log k})$  for  $Q_{2} = \sum_{t = 1}^{T}\| \ell_{t} - \bar{\ell}\|^{2}$ . Other examples of data-dependent bounds include path-length bounds in the form of  $O(\sqrt{kV_1\log T})$  for  $V_{1} = \sum_{t = 1}^{T - 1}\| \ell_{t} - \ell_{t + 1}\|_{1}$  as well as a sparsity-dependent bound, which have been investigated [10, 11, 27, 49, 51].
Sparsity-dependent bound The study on a sparsity-dependent bound was initiated by Kwon and Perchet [27], who showed that when  $\ell_t \in [0,1]^k$ , the OMD with Tsallis entropy can achieve the bound of  $\mathrm{Reg}_T \leq 2\sqrt{\mathrm{e}}\sqrt{sT\log(k / s)}$  and prove the matching (up to logarithmic factor) lower bound of  $\mathrm{Reg}_T = \Omega (\sqrt{sT})$  when  $T \geq k^3 /(4s^2)$ . Bubeck et al. [10] also showed that OMD with a hybrid-regularizer consisting of the Shannon entropy and a log-barrier can achieve  $\mathrm{Reg}_T \leq 10\sqrt{L_2\log k} + 20k\log T$  when  $\ell_t \in [-1,1]^k$ . Zheng et al. [51] investigated the sparse MAB problem in the context of the switching regret. Although their result is not directly related to our study, they show that sparsity is useful in some cases. Note that all of these algorithms assume the knowledge of the sparsity level and do not have a BOBW guarantee. The study to exploit the sparsity was investigated also in the stochastic regime by Kwon et al. [28]. However, they define the sparsity level  $s$  by the number of arms with rewards larger than or equal to 0 (i.e., losses smaller than or equal to 0), and hence the definition of sparsity is different from that in our paper.
Adaptive learning rate The stability-dependent learning rate is quite ubiquitous (see [38] and the references therein). To our knowledge, the literature on the penalty-dependent bound is quite scarce in bandits and considered in the context of BOBW algorithms [22, 46], both of which consider the Shannon entropy regularizer.
Best-of-both-worlds Since the seminal study by Bubeck and Slivkins [9], BOBW algorithms have been developed for many online decision-making problems. Although they have been investigated mainly in the context of an MAB [43, 53], other settings have also been investigated, [18, 24, 49], to name a few.
FTRL and OMD are now one of the most common approaches to achieving a BOBW guarantee owing to the usefulness of the self-bounding technique [18, 49, 53], while the first [9] and earlier work [43, 44] on BOBW do not rely on the technique. Most of the recent algorithms beyond the MAB are also based on FTRL (to name a few, [24, 42, 49]).
Our BOBW algorithm with the sparsity-dependent bound can be seen as one of the studies that aim to achieve BOBW and data-dependent bound simultaneously. There is not so much existing research, and we are only aware of [21, 23, 47, 49]. They consider first-, second-order, and path-length bound, and we are the first to investigate the sparsity-dependent bound in this line of work.
Log-barrier regularizer and hybrid regularizer The log-barrier regularizer has been used in various studies (to name a few, [14, 16, 35, 40, 49]). The time-invariant log-barrier (a.k.a. constant amount of log-barrier [33]), whose properties are extensively exploited in this paper, was invented by Bubeck et al. [10] and has been used in several subsequent studies [33, 51].
Partial monitoring Starting from the work by Rustichini [41], PM has been investigated in many works in the literature [7, 12, 39]. It is known that all PM games can be classified into four classes based on their minimax regrets [7, 30]. In particular, all PM games fall into trivial, easy, hard, and hopeless games, for which its minimax regrets is  $0$ ,  $\Theta(\sqrt{T})$ ,  $\Theta(T^{2/3})$ , and  $\Theta(T)$ , respectively. PM has also been investigated in both the adversarial and stochastic regimes as for MAB. In the stochastic regime, there are relatively small amounts of works [8, 26, 45, 48], some of which are proven to achieve an instance-dependent  $O(\log T)$  regrets for locally or globally observable games. In the adversarial regime, since the development of the FeedExp3 algorithm [12, 39], many algorithms achieving the minimax optimal regret have been developed [6, 7, 15, 31].
Proof. We first prove (5) in Part (I).
Penalty term First, we consider the penalty term  $\sum_{t=1}^{T}\left(\frac{1}{\eta_{t+1}} - \frac{1}{\eta_t}\right)h_{t+1}$ . By the definition of  $\beta_t$  in (4),
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) h _ {t + 1} = \sum_ {t = 1} ^ {T} \left(\beta_ {t + 1} - \beta_ {t}\right) h _ {t + 1} = \sum_ {t = 1} ^ {T} \frac {c _ {1} z _ {t} h _ {t + 1}}{\sqrt {c _ {2} + \bar {z} _ {t} h _ {1} + \sum_ {s = 1} ^ {t - 1} z _ {s} h _ {s + 1}}} \\ \leq c _ {1} \sum_ {t = 1} ^ {T} \frac {z _ {t} h _ {t + 1}}{\sqrt {\sum_ {s = 1} ^ {t} z _ {s} h _ {s + 1}}} \leq c _ {1} \int_ {0} ^ {\sum_ {t = 1} ^ {T} z _ {t} h _ {t + 1}} \frac {1}{\sqrt {x}} d x = 2 c _ {1} \sqrt {\sum_ {t = 1} ^ {T} z _ {t} h _ {t + 1}}, \tag {14} \\ \end{array}
$$
where the first inequality follows from  $\bar{z}_t h_1 \geq z_t h_{t+1}$  and the second inequality follows by Lemma 8 given in Appendix J.
Stability term Next, we consider the stability term  $\sum_{t=1}^{T} \eta_t z_t$ . Using the definition of  $\beta_t$  in (4) and defining  $U_t = \sqrt{c_2 + \bar{z}_t h_1 + \sum_{s=1}^{t-1} z_s h_{s+1}}$  for  $t \in \{0\} \cup [T]$ , we bound  $\beta_t$  from below as
$$
\beta_ {t} = \beta_ {1} + \sum_ {u = 1} ^ {t - 1} \frac {c _ {1} z _ {u}}{\sqrt {c _ {2} + \bar {z} _ {u} h _ {1} + \sum_ {s = 1} ^ {u - 1} z _ {s} h _ {s + 1}}} = \beta_ {1} + \sum_ {u = 1} ^ {t - 1} \frac {c _ {1} z _ {u}}{U _ {u}} \geq \beta_ {1} + \frac {c _ {1}}{U _ {T}} \sum_ {u = 1} ^ {t - 1} z _ {u}, \tag {15}
$$
where the inequality follows since  $(U_{t})$  is non-decreasing. Using the last inequality, we can bound  $\sum_{t = 1}^{T}\eta_{t}\mathcal{Z}_{t}$  as
$$
\sum_ {t = 1} ^ {T} \eta_ {t} z _ {t} = 2 \sum_ {t = 1} ^ {T} \frac {z _ {t}}{\beta_ {t} + \beta_ {t}} \leq 2 \sum_ {t = 1} ^ {T} \frac {z _ {t}}{\beta_ {t} + \beta_ {1} + \frac {c _ {1}}{U _ {T}} \sum_ {s = 1} ^ {t - 1} z _ {s}} = \frac {2 U _ {T}}{c _ {1}} \sum_ {t = 1} ^ {T} \frac {z _ {t}}{\frac {U _ {T}}{c _ {1}} \left(\beta_ {t} + \beta_ {1}\right) + \sum_ {s = 1} ^ {t - 1} z _ {s}}. \tag {16}
$$
Since we have  $\frac{U_T}{c_1} (\beta_1 + \beta_t) \geq \frac{\sqrt{c_2 + \bar{z}_t h_1}}{c_1} (\beta_1 + \beta_t) \geq \varepsilon + z_t$  by the assumption, a part of the last inequality is further bounded as
$$
\sum_ {t = 1} ^ {T} \frac {z _ {t}}{\frac {U _ {T}}{c _ {1}} \left(\beta_ {t} + \beta_ {1}\right) + \sum_ {s = 1} ^ {t - 1} z _ {s}} \leq \sum_ {t = 1} ^ {T} \frac {z _ {t}}{\varepsilon + \sum_ {s = 1} ^ {t} z _ {s}} \leq \int_ {\varepsilon} ^ {\varepsilon + \sum_ {t = 1} ^ {T} z _ {t}} \frac {1}{x} d x \leq \log \left(1 + \sum_ {t = 1} ^ {T} \frac {z _ {t}}{\varepsilon}\right), \tag {17}
$$
where the second inequality follows by Lemma 8. Combining (16) and (17) yields
$$
\sum_ {t = 1} ^ {T} \eta_ {t} z _ {t} \leq \frac {2 U _ {T}}{c _ {1}} \log \left(1 + \sum_ {t = 1} ^ {T} \frac {z _ {t}}{\varepsilon}\right) = \frac {2}{c _ {1}} \log \left(1 + \sum_ {t = 1} ^ {T} \frac {z _ {t}}{\varepsilon}\right) \sqrt {c _ {2} + \bar {z} _ {T} h _ {1} + \sum_ {t = 1} ^ {T} z _ {t} h _ {t + 1}}. \tag {18}
$$
Combining (14) and (18) completes the proof of (5) in Part (I).
We next prove Part (II). For the penalty term, setting  $h_t = h_1$  for all  $t \in [T]$  in (14) gives
$$
\sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) h _ {t + 1} \leq 2 c _ {1} \sqrt {h _ {1} \sum_ {t = 1} ^ {T} z _ {t}}.
$$
For the stability term, since there exists  $a > 0$  such that  $\beta_{t} \geq \frac{ac_{1}}{\sqrt{h_{1}}}\sqrt{\sum_{s = 1}^{t}z_{s}}$  for any  $t \in [T]$  by the assumption,
$$
\sum_ {t = 1} ^ {T} \eta_ {t} z _ {t} = \sum_ {t = 1} ^ {T} \frac {z _ {t}}{\beta_ {t}} \leq \frac {\sqrt {h _ {1}}}{a c _ {1}} \sum_ {t = 1} ^ {T} \frac {z _ {t}}{\sqrt {\sum_ {s = 1} ^ {t} z _ {s}}} \leq \frac {2}{a c _ {1}} \sqrt {h _ {1} \sum_ {t = 1} ^ {T} z _ {t}}.
$$
Summing up the above arguments completes the proof of Part (II).
E Basic facts to bound stability terms
Here, we introduce basic facts, which are useful to bound the stability term. We have
$$
\xi (x) := \exp (- x) + x - 1 \leq \left\{ \begin{array}{l l} \frac {1}{2} x ^ {2} & \text {f o r} x \geq 0 \\ x ^ {2} & \text {f o r} x \geq - 1, \end{array} \right. \tag {19}
$$
$$
\zeta (x) := x - \log (1 + x) \leq x ^ {2} \quad \text {f o r} x \in \left[ - \frac {1}{2}, \frac {1}{2} \right]. \tag {20}
$$
We also have the following inequalities for  $\phi^{\mathsf{nS}}(x) = x\log x$  and  $\phi^{\mathrm{LB}}(x) = \log (1 / x)$ , which are components of the negative Shannon entropy and log-barrier function:
$$
\max  _ {y \in \mathbb {R}} \left\{a (x - y) - D _ {\phi^ {\mathrm {n S}}} (y, x) \right\} = x \xi (a) \quad \text {f o r} a \in \mathbb {R}, \tag {21}
$$
$$
\max  _ {y \in \mathbb {R}} \left\{a (x - y) - D _ {\phi^ {\mathrm {L B}}} (y, x) \right\} = \zeta (a x) \quad \text {f o r} a \geq - \frac {1}{x}. \tag {22}
$$
It is easy to prove these facts by the standard calculus and you can find the proofs of (21) and (22) in Lemma 15 of Tsuchiya et al. [46] and Lemma 5 of Ito et al. [23], respectively.
F Proof of Corollary 2
Let  $\operatorname{Reg}_T(a) = \mathbb{E}\left[\sum_{t=1}^T\left(\ell_{tA_t} - \ell_{ta}\right)\right]$  for  $a \in [k]$ . Here we provide the complete proof of Corollary 2.
Proof of Corollary 2. Fix  $i^* \in [k]$ . Since  $p_t = (1 - \gamma) q_t + \frac{\gamma}{k} \mathbf{1}$ , it holds that
$$
\begin{array}{l} \operatorname {R e g} _ {T} \left(i ^ {*}\right) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell_ {t A _ {t}} - \sum_ {t = 1} ^ {T} \ell_ {t i ^ {*}} \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \ell_ {t}, p _ {t} - e _ {i ^ {*}} \right\rangle \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \ell_ {t}, q _ {t} - e _ {i ^ {*}} \rangle \right] + \mathbb {E} \left[ \gamma \sum_ {t = 1} ^ {T} \left\langle \ell_ {t}, \frac {1}{k} {\bf 1} - q _ {t} \right\rangle \right] \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \widehat {y} _ {t}, q _ {t} - e _ {i ^ {*}} \rangle \right] + \gamma T, \\ \end{array}
$$
where the last inequality follows by  $\mathbb{E}[\widehat{y}_t\mid q_t] = \ell_t$  and the Cauchy-Schwarz inequality. Then, using the standard analysis of the FTRL described in Section 3, the first term in the last inequality is bounded as
$$
\sum_ {t = 1} ^ {T} \left\langle \widehat {y} _ {t}, q _ {t} - e _ {i ^ {*}} \right\rangle \leq \sum_ {t = 1} ^ {T} \left(\frac {1}{\eta_ {t + 1}} - \frac {1}{\eta_ {t}}\right) H (q _ {t + 1}) + \frac {H (q _ {1})}{\eta_ {1}} + \sum_ {t = 1} ^ {T} \left(\left\langle q _ {t} - q _ {t + 1}, \widehat {y} _ {t} \right\rangle - D _ {\Phi_ {t}} \left(q _ {t + 1}, q _ {t}\right)\right).
$$
By (19) and (21) given in Appendix E, the stability term  $\langle q_t - q_{t + 1},\widehat{y}_t\rangle -D_{\Phi_t}(q_{t + 1},q_t)$  in the last inequality is bounded as