SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tiancheng Jin *  
University of Southern California  
tiancheng.jin@usc.edu
Junyan Liu *  
University of California, San Diego  
ju1037@ucsd.edu
Haipeng Luo  
University of Southern California  
haipengl@usc.edu
We study the problem of designing adaptive multi-armed bandit algorithms that perform optimally in both the stochastic setting and the adversarial setting simultaneously (often known as a best-of-both-world guarantee). A line of recent works shows that when configured and analyzed properly, the Follow-the-Regularized-Leader (FTRL) algorithm, originally designed for the adversarial setting, can in fact optimally adapt to the stochastic setting as well. Such results, however, critically rely on an assumption that there exists one unique optimal arm. Recently, Ito [2021] took the first step to remove such an undesirable uniqueness assumption for one particular FTRL algorithm with the  $1/2$ -Tsallis entropy regularizer. In this work, we significantly improve and generalize this result, showing that uniqueness is unnecessary for FTRL with a broad family of regularizers and a new learning rate schedule. For some regularizers, our regret bounds also improve upon prior results even when uniqueness holds. We further provide an application of our results to the decoupled exploration and exploitation problem, demonstrating that our techniques are broadly applicable.
We study the problem of multi-armed bandits (MAB) where a learner sequentially interacts with an environment for  $T$  rounds. In each round, the learner selects one of the  $K$  arms and observes its loss. The goal of the learner is to minimize her regret, which measures the difference between her total loss and that of the best fixed arm in hindsight. Depending on how the losses are generated, two settings have been heavily studied in the literature: the stochastic setting, where the loss of each arm at each round is an i.i.d. sample of a fixed and unknown distribution, and the adversarial setting, where the losses can be arbitrarily decided by an adversary. In the stochastic setting, the UCB algorithm [Lai and Robbins, 1985, Auer et al., 2002a] attains the instance-optimal regret  $\mathcal{O}(\sum_{i:\Delta_i > 0}\frac{\log T}{\Delta_i})$ , where the sub-optimality gap  $\Delta_{i}$  is the difference between the expected loss of arm  $i$  and that of the optimal arm. On the other hand, in the adversarial setting, the minimax-optimal regret is known to be of order  $\Theta (\sqrt{KT})$  [Auer et al., 2002b, Audibert and Bubeck, 2009], achieved via the well-known Follow-the-Regularized-Leader (FTRL) framework.
*Equal contribution, in alphabetical order.
Given the rather different algorithmic ideas in UCB and FTRL, it is natural to ask whether there exists an adaptive algorithm that achieves the best of both worlds (BOBW), simultaneously enjoying the instance-optimal  $(\log T)$ -regret in the stochastic setting and minimax-optimal  $\sqrt{T}$ -regret in the adversarial setting. This question was first answered affirmatively in [Bubeck and Slivkins, 2012], followed by a sequence of improvements and extensions in the past decade. Among these works, a somewhat surprising result by Wei and Luo [2018] shows that, when configured and analyzed properly, FTRL, an algorithm originally designed for the adversarial setting to achieve  $\sqrt{T}$ -type regret, in fact can also achieve  $(\log T)$ -type regret in the stochastic setting automatically. This result was latter significantly improved to optimal by Zimmert and Seldin [2019, 2021] using the  $1/2$ -Tsallis entropy regularizer and extended to many other problems. In fact, these algorithms not only achieve BOBW, but also automatically adapt to intermediate settings with a regret bound that interpolates smoothly between the two extremes.
A key drawback of such FTRL-based approaches, however, is that their analysis for the stochastic setting critically relies on a uniqueness assumption, that is, there exists one unique optimal arm (with  $\Delta_{i} = 0$ ). A recent work by Ito [2021] took the first step to address this issue and proposed a novel analysis showing that the exact same Tsallis-INF algorithm of [Zimmert and Seldin, 2019] in fact works even without this assumption. Unfortunately, his analysis is specific to the  $1/2$ -Tsallis entropy regularizer with an arm-independent learning rate, and it is highly unclear how to extend it to other regularizers which often require an arm-dependent learning rate. For example, extending it to the log-barrier regularizer was explicitly mentioned as an open problem in [Ito, 2021].
In this work, we significantly improve and generalize the analysis of [Ito, 2021], greatly deepening our understanding on using FTRL to achieve BOBW. Our improved analysis allows us to obtain a suite of new results, all achieved without the uniqueness assumption. Specifically, we consider a new and unified arm-dependent learning rate schedule and the following regularizers (plus a small amount of extra log-barrier); see also Table 1 for a summary.
- Log-barrier: with our new learning rate schedule, we show a regret bound of order  $\mathcal{O}\left(\mathbb{G}\log T\right)$  for the stochastic setting and  $\mathcal{O}\left(\sqrt{KT\log T}\right)$  for the adversarial setting. Here,  $\mathbb{G} = \frac{|U|}{\Delta_{\mathrm{MIN}}} + \sum_{i\in V}\frac{1}{\Delta_i}$  measures the difficulty of the instance, with  $U = \{i:\Delta_i = 0\}$  being the set of optimal-arms,  $V$  being the set of all remaining sub-optimal arms, and  $\Delta_{\mathrm{MIN}} = \min_{i\in V}\Delta_i$  being the minimum non-zero sub-optimality gap. Our bound for the stochastic setting improves those in [Wei and Luo, 2018, Ito, 2021], both of which require the uniqueness assumption.
- Shannon entropy: we show a regret bound of order  $\mathcal{O}\left(\mathbb{G}(\log T)^2\right)$  for the stochastic setting and  $\mathcal{O}(\sqrt{KT}\log T)$  for the adversarial setting. This improves [Ito et al., 2022] (when applying their more general results to MAB) in two ways: first, their result requires the uniqueness assumption while ours does not; second, their  $\mathbb{G}$  is defined as  $\frac{K}{\Delta_{\mathrm{MIN}}}$ , strictly larger than ours.
-  $\beta$ -Tsallis entropy: we also consider Tsallis entropy with a general parameter  $\beta \in (0,1)$ , and show a regret bound of order  $\mathcal{O}\left(\frac{\mathbb{G}\log T}{\beta(1 - \beta)}\right)$  for the stochastic setting and  $\mathcal{O}\left(\sqrt{\frac{KT(\log T)^{\mathbb{I}\{\beta\neq 1/2\}}}{\beta(1 - \beta)}}\right)$  for the adversarial setting. The only prior work that uses  $\beta$ -Tsallis entropy for BOBW in MAB is [Zimmert and Seldin, 2021], but their algorithm is infeasible (unless  $\beta = 1/2$ ) since the learning rate is tuned in terms of the unknown sub-optimality gaps. We not only address this issue with our new learning rate schedule, but also remove the uniqueness assumption. While  $\beta = 1/2$  leads to best bounds in MAB, following [Rouyer and Seldin, 2020], we showcase the importance of other values of  $\beta$  (specifically,  $\beta = 2/3$ ) in the so-called Decoupled Exploration and Exploitation (DEE-MAB) setting, and again significantly improve their results (see Table 2).
It is worth noting that, as it is common for FTRL-based approaches, our algorithms also automatically adapt to more general corrupted settings (or the so-called adversarial regime with a self-bounding constraint [Zimmert and Seldin, 2021]). The complete statements of our results can be found in Section 3.
Techniques Inspired by [Ito, 2021], we decompose the regret into three parts: the regret related to the sub-optimal arms, the regret related to the optimal arms, and the residual regret. Bounding each of them requires new ideas, as discussed below (see Section 4 for details).
Table 1: Overview of our BOBW results for MAB, all achieved via Algorithm 1 and a unified learning rate  $\gamma_i^t = \theta \sqrt{1 + \sum_{\tau < t} \left( \max \{ p_i^\tau, 1 / T \} \right)^{1 - 2\alpha}}$  for arm  $i$  in round  $t$ , with  $p_i^\tau$  being the probability of picking arm  $i$  in round  $\tau$  and the values of  $\theta$  and  $\alpha$  specified in the table. "Sto." and "Adv." denote respectively the stochastic and the adversarial setting.  $\mathbb{G}$  is defined as  $\frac{|U|}{\Delta_{\mathrm{MIN}}} + \sum_{i \in V} \frac{1}{\Delta_i}$ , where  $U$  is the set of optimal-arms,  $V$  is the set of sub-optimal arms, and  $\Delta_{\mathrm{MIN}} = \min_{i \in V} \Delta_i$ .
Regularizera	α,θ	Regret (w/o uniqueness)	Comments
Log-barrier	α=0	Sto. O (G log T)	First to remove uniqueness for log-barrier
-∑iγt i log pi	θ=√1/logT	Adv. O (√KT log T)
β-Tsallis entropy	α=β	Sto. O (G log T/β(1-β))	First BOBW result for β ≠ 1/2 (even with uniqueness)
-1/1-β ∑iγt i pβ	θ=√1-β	Adv. O (√KT(log T)1{β≠1/2}/β(1-β))
Shannon entropy	α=1	Sto. O (G (log T)2)	Improve [Ito et al., 2022] which defines G as K/ΔMIN and requires uniquenessb
∑iγt i p i log (pi/e)	θ=√1/logT	Adv. O (√KT log T)
${}^{a}$  To be more precise, a small amount of extra log-barrier has been omitted in this table for simplicity.
${}^{b}$  To be clear, the setting in [Ito et al., 2022] is more general than MAB. Here, the comparison is solely based on their results specified to MAB.
Table 2: Regret bounds of our algorithm using  $^{2/3}$ -Tsallis entropy for the Decoupled Exploration and Exploitation MAB problem.  $V$  is the set of sub-optimal arms and  $\Delta_{\mathrm{MIN}} = \min_{i \in V} \Delta_i$ .
Ours (w/o uniqueness)	[Rouyer and Seldin, 2020] (w/ uniqueness)
Sto. O(√Σi∈V K/Δi2)	Sto. O(√Σi∈V K/Δi·ΔMIN)
Adv. O(√KT)	Adv. O(√KT)
To bound the regret related to the sub-optimal arms by a so-called self-bounding quantity (the key to achieve BOBW using FTRL), we design a novel arm-dependent learning rate schedule (which is also our key algorithmic contribution). For example, when using  $\beta$ -Tsallis entropy, this schedule balances the corresponding stability term and penalty term of a sub-optimal arm  $i$  to  $\mathcal{O}\left(\sum_{t=1}^{T} \left(p_i^t\right)^{1 - \beta} / \gamma_i^{t+1}\right)$ , which is then bounded by a self-bounding quantity. Apart from removing the uniqueness assumption, as mentioned this learning rate schedule also enables us to achieve the first BOBW guarantees for  $\beta$ -Tsallis entropy with any value of  $\beta$ , and also to improve the bound of Ito et al. [2022] for Shannon entropy, which are notable results on their own.
Then, to bound the regret related to the optimal arms, we greatly extend the idea of [Ito, 2021] that is highly specific to the simple form of  $\frac{1}{2}$ -Tsallis entropy with an arm-independent learning rate. Specifically, we develop a new analysis based on a key observation of a certain monotonicity of Bregman divergences. Such monotonicity only requires two mild conditions on the regularizer that are usually satisfied, allowing us to apply it to a broad spectrum of regularizers.
Our arm-dependent learning rate does make the residual regret much more complicated compared to [Ito, 2021]. To handle it, we carefully consider two cases and show that in both cases it can be related to some self-bounding quantities.
Finally, we note that various places of our analysis require the learner's distribution over arms to be stable in a multiplicative sense between two consecutive rounds. We achieve this by adding an extra small amount of log-barrier, a technique first proposed in [Bubeck et al., 2018]. While we do not know how to prove the same results without this extra tweak, we conjecture that it is indeed unnecessary.
Related work For early results solely for the stochastic setting or solely for the adversarial setting, we refer the readers to the systematic survey in [Lattimore and Szepesvári, 2020]. The study of BOBW for MAB starts from the pioneering work of Bubeck and Slivkins [2012], followed by many improvements via different approaches [Seldin and Slivkins, 2014, Auer and Chiang, 2016, Seldin and Lugosi, 2017, Wei and Luo, 2018, Lykouris et al., 2018, Gupta et al., 2019, Zimmert and Seldin, 2019, 2021] and many extensions from MAB to other problems such as semi-bandits [Zimmert et al., 2019], linear bandits [Lee et al., 2021], MAB with feedback graphs [Ito et al., 2022, Erez and Koren, 2021, Rouyer et al., 2022], MAB with switching cost [Rouyer et al., 2021, Amir et al., 2022], model-selection [Pacchiano et al., 2022], partial monitoring [Tsuchiya et al., 2023], and Markov Decision Process (MDP) [Lykouris et al., 2019, Jin and Luo, 2020, Jin et al., 2021, Chen et al., 2021]. Among these works, the FTRL-based approach is particularly appealing since it is simple in both the algorithm design and the analysis, and also extends seamlessly to other more general settings (such as the corrupted setting). The uniqueness assumption used to be critical for the analysis of this approach, but plays no role in other methods such as [Auer and Chiang, 2016, Seldin and Lugosi, 2017]. Following the first step by [Ito, 2021], our work further demonstrates that this was merely due to the lack of a better analysis (and sometimes a better learning rate schedule). We believe that our techniques shed light on removing the uniqueness assumption for using FTRL in more complicated problems such as semi-bandits and MDPs.
In multi-armed bandits (MAB), a learner is given a fixed set of arms  $[K] = \{1,2,\dots ,K\}$  and has to interact with an environment for  $T\geq K$  rounds. In each round  $t$ , the learner chooses an arm  $i^t\in [K]$  while simultaneously the environment decides a loss vector  $\ell^t\in [0,1]^K$ . The learner then suffers and observes the loss  $\ell_{it}^{t}$  of the selected arm for this round. The goal of the learner is to minimize her (pseudo) regret, which measures the difference between her expected cumulative loss and that of the best arm in hindsight. Formally, the regret is defined as  $\mathrm{Reg}^T = \mathbb{E}\big[\sum_{t = 1}^T\ell_{it}^t -\sum_{t = 1}^T\ell_{i^\star}^t\big]$ , where  $i^{\star}\in \operatorname{argmin}_{i\in [K]}\mathbb{E}\big[\sum_{t = 1}^{T}\ell_{i}^{t}\big]$  is one of the best arms in hindsight, and  $\mathbb{E}[\cdot ]$  denotes the expectation with respect to the internal randomness of both the algorithm and the environment.
Adversarial setting versus stochastic setting We consider two different settings according to how the loss vectors are decided by the environment. In the adversarial setting, the environment decides the loss vectors in an arbitrary way with the knowledge of the learner's algorithm. In this case, the minimax optimal regret is known to be  $\Theta (\sqrt{KT})$  [Audibert and Bubeck, 2009].
In the stochastic setting, following prior work such as [Zimmert and Seldin, 2021], we consider a situation much more general than the vanilla i.i.d. case (sometimes called the adversarial regime with a self-bounding constraint). Formally, we assume that the loss vectors satisfy the following condition: there exists a gap vector  $\Delta \in [0,1]^K$  and a constant  $C\geq 0$  such that
$$
\operatorname {R e g} ^ {T} \geq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sum_ {i \in [ K ]} \Pr \left[ i ^ {t} = i \right] \Delta_ {i} \right] - C, \tag {1}
$$
where  $\operatorname*{Pr}[i^t = i]$  denotes the learner's probability of taking arm  $i$  in round  $t$ . This condition subsumes the well-studied i.i.d. setting (discussed in Section 1) where the loss vectors are independently sampled from a fixed but unknown distribution and thus Condition (1) holds with equality,  $C = 0$ , and  $\Delta_i = \mathbb{E}[\ell_i^t - \ell_{i^*}^t]$  being the sub-optimality gap of arm  $i$  (independent of  $t$ ). In this case, the instance-optimal regret is  $\mathcal{O}\left(\sum_{i:\Delta_i > 0} \frac{\log T}{\Delta_i}\right)$ , achieved by the UCB algorithm [Auer et al., 2002a]. More generally, Condition (1) covers the corrupted i.i.d. setting where the loss vectors are first sampled from a fixed distribution and then corrupted by an adversary in an arbitrary way as long as the expected cumulative  $\ell_{\infty}$  distance between the corrupted loss vector and the original one is bounded by  $C \in [0,T]$ . While these two examples both involve iidness, note that Condition (1) itself is much more general and does not necessarily require that.
Uniqueness assumption Prior work using FTRL to achieve BOBW crucially relies on a uniqueness assumption when analyzing the regret under Condition (1). Specifically, it is assumed that there exists one and only one arm  $\mathring{i}$  with  $\Delta_{\mathring{i}} = 0$ . In the special i.i.d. case, this simply means that there
Algorithm 1 FTRL for BOBW without Uniqueness
Input: coefficient  $\theta$ , learning rate  $\alpha$ , Tsallis entropy parameter  $\beta$ , log-barrier coefficient  $C_{\mathrm{LOG}}$ , number of arms  $K$ , number of rounds  $T$  (not necessary if a doubling trick is applied; see [Ito, 2021, Section 5.3])
$$
\text {D e f i n e r e g u l a r i z e r} \phi^ {t} (p) = \underbrace {- C _ {\text {L O G}} \sum_ {i \in [ K ]} \log p _ {i}} _ {\text {e x t r a l o g - b a r r i e r}} + \left\{ \begin{array}{l l} - \sum_ {i \in [ K ]} \gamma_ {i} ^ {t} \log p _ {i}, & (\text {l o g - b a r r i e r}) \\ - \frac {1}{1 - \beta} \sum_ {i \in [ K ]} \gamma_ {i} ^ {t} p _ {i} ^ {\beta}, & (\beta \text {- T s a l l i s e n t r o p y}) \\ \sum_ {i \in [ K ]} \gamma_ {i} ^ {t} p _ {i} \log (p _ {i} / e), & (\text {S h a n n o n e n t r o p y}) \end{array} \right.
$$
$$
\mathrm {w i t h l e a n s e r r o r} \gamma_ {i} ^ {t} = \theta \sqrt {1 + \sum_ {\tau <   t} \left(\max \left\{p _ {i} ^ {\tau} , ^ {1} / T \right\}\right) ^ {1 - 2 \alpha}}.
$$
$$
\text {C o m p u t e} p ^ {t} = \operatorname * {a r g m i n} _ {p \in \Omega_ {K}} \left\{\left\langle p, \sum_ {\tau <   t} \widehat {\ell} ^ {\tau} \right\rangle + \phi^ {t} (p) \right\} \text {w h e r e} \Omega_ {K} \text {i s t h e s i m p l e x}.
$$
$$
\text {D r a w} i ^ {t} \sim p ^ {t}, \text {s u f f e r} \ell_ {i t} ^ {t}.
$$
$$
\text {C o n s t r u c t} \widehat {\ell} ^ {t} \text {a s a n u n b i a s e d e s t i m a t o r o f} \ell_ {t} \text {w i t h} \widehat {\ell} _ {i} ^ {t} = \frac {\mathbb {I} \left\{i ^ {t} = i \right\} \ell_ {i} ^ {t}}{p _ {i} ^ {t}}, \forall i \in [ K ].
$$
exists a unique optimal arm  $(\stackrel{\circ}{i} = i^{\star})$ . Under this uniqueness assumption, the Tsallis-INF algorithm of [Zimmert and Seldin, 2021] achieves  $\mathrm{Reg}^T = \mathcal{O}\left(\sum_{i\neq i}\frac{\log T}{\Delta_i} +\sqrt{C\sum_{i\neq i}\frac{\log T}{\Delta_i}}\right)$ . The recent work [Ito, 2021] takes the first step to remove such an assumption for the Tsallis-INF algorithm and develops a refined analysis with regret bound  $\mathcal{O}\left(\sum_{i:\Delta_i > 0}\frac{\log T}{\Delta_i} +\sqrt{C\sum_{i:\Delta_i > 0}\frac{\log T}{\Delta_i}} +D + K\right)$ , where  $D$  is such that  $\mathbb{E}\left[\sum_{t = 1}^{T}\max_{i:\Delta_i = 0}\mathbb{E}^t\left[\ell_i^t -\ell_{i^*}^t\right]\right]\leq D$ , and  $\mathbb{E}^t [\cdot ]$  is the conditional expectation with respect to the history before round  $t$ . Note that, in the i.i.d. setting,  $D$  is simply 0, while in the corrupted setting,  $D$  is at most  $C$ . We significantly generalize and improve this result to a borad family of algorithms, and all our results hold without the uniqueness assumption.
We denote by  $U = \{i \in [K] : \Delta_i = 0\}$  the set of arms with a zero gap, and  $V = [K] \backslash U$  the set of arms with a positive gap. In the i.i.d. setting,  $U$  is simply the set of optimal arms and  $V$  is the set of sub-optimal arms. We also define  $\Delta_{\mathrm{MIN}} = \min_{i \in V} \Delta_i$  to be the minimum nonzero gap.
3 Algorithms and Results
The pseudocode of our algorithm is presented in Algorithm 1. It is based on the general FTRL framework, which finds  $p^t$ , the distribution of selecting arms in around  $t$ , via solving the optimization problem  $p^t = \operatorname*{argmin}_{p\in \Omega_K}\left\langle p,\sum_{\tau < t}\widehat{\ell}^\tau \right\rangle + \phi^t (p)$ . Here,  $\Omega_K$  is the set of all possible distributions over  $K$  arms,  $\widehat{\ell}^{\tau}$  is an loss estimator for  $\ell^{\tau}$ , and  $\phi^t$  is a regularizer. The learner then samples arm  $i^t$  from the distribution  $p^t$  and observes the suffered loss  $\ell_{i^t}^t$ . With this feedback, the algorithm constructs the standard unbiased importance-weighted loss estimator:  $\widehat{\ell}_i^t = \frac{\mathbb{I}\{i^t = i\} \ell_i^t}{p_i^t}$ ,  $\forall i\in [K]$ , where  $\mathbb{I}\{\cdot\}$  denotes the indicator function.
We consider three different reuglarizers  $\phi^t$ : log-barrier,  $\beta$ -Tsallis entropy, and Shannon entropy; see Algorithm 1 for definitions. By now, they are standard reuglarizers used extensively in the MAB literature, each with different useful properties, but there are some small tweaks in our definitions: 1) a linear term (from  $p_i\log (p_i / e) = p_i\log p_i - p_i$ ) is added to the canonical form of Shannon entropy, which is critical to ensure a certain type of monotonicity of Bregman divergences (see Section 4); 2) for technical reasons, we also incorporate a small amount of extra log-barrier (with coefficient  $C_{\mathrm{LOG}}$ ), which ensures multiplicative stability of the algorithm.
More importantly, we propose the following unified arm-dependent learning rate:
$$
\gamma_ {i} ^ {t} = \theta \sqrt {1 + \sum_ {\tau = 1} ^ {t - 1} \left(\max  \left\{p _ {i} ^ {\tau} , ^ {1} / T \right\}\right) ^ {1 - 2 \alpha}}, \quad \forall i \in [ K ], t \in [ T ], \tag {2}
$$
where  $\alpha \in [0,1]$  and  $\theta \in \mathbb{R}_{>0}$  are parameters (set differently for different regularizers). The clipping of  $p_i^\tau$  to  $1 / T$  is because  $(p_i^t)^{1 - 2\alpha}$  itself could be unbounded for  $\alpha \in (1 / 2,1]$  when  $p_i^t$  is too small. This learning rate is not only conceptually simpler than those in [Ito, 2021, Ito et al., 2022] and important for removing the uniqueness assumption, but also leads to better bounds in some cases as we discuss below.
Main results We now present our main results. Our regret bounds in the stochastic setting are expressed in terms of an instance complexity measure  $\mathbb{G} = \frac{|U|}{\Delta_{\mathrm{MIN}}} + \sum_{i \in V} \frac{1}{\Delta_i}$ , which is of the same order as the standard complexity measure  $\sum_{i \in V} \frac{1}{\Delta_i}$  when  $|U| = O(1)$  (in particular, this is the case when the uniqueness assumption holds). Similar to [Ito, 2021], our bounds are also in terms of the constant  $D$  defined in Section 2. We start with the following result for the log-barrier regularizer.
Theorem 3.1. When using the log-barrier regularizer with  $C_{\mathrm{LOG}} = 162$ ,  $\alpha = 0$ , and  $\theta = \sqrt{1 / \log T}$ , Algorithm 1 ensures  $\operatorname{Reg}^T = \mathcal{O}\left(\sqrt{KT\log T}\right)$  always, and simultaneously the following regret bound when Condition (1) holds:  $\operatorname{Reg}^T = \mathcal{O}\left(\mathbb{G}\log T + \sqrt{C\mathbb{G}\log T} +\frac{K^2}{\sqrt{\log T}} +K\log T + D\right)$ .
Log-barrier was first used to achieve BOBW in [Wei and Luo, 2018] and later improved in [Ito, 2021], both of which require the uniqueness assumption. The  $\mathcal{O}\left(\sqrt{KT\log T}\right)$  bound for the adversarial setting is almost minimax optimal except for the extra  $\sqrt{\log T}$  factor (a common caveat for log-barrier). On the other hand, the bound under Condition (1) matches that of [Ito, 2021] when uniqueness holds and generalizes it otherwise. It is worth noting that this bound (and the same for our other results) suffers an  $\mathcal{O}(D)$  term, which unfortunately can be as large as  $C$ , making the bound weaker than those always with only  $\sqrt{C}$  dependence under the uniqueness assumption. It is unclear to us whether such  $\mathcal{O}(D)$  dependence is necessary when we do not make the uniqueness assumption.
Next, we present our results for Shannon entropy.