SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A. Pacchiano, C. Dann, and C. Gentile. Best of both worlds model selection. In Advances in Neural Information Processing Systems, 2022.
C. Rouyer and Y. Seldin. Tsallis-inf for decoupled exploration and exploitation in multi-armed bandits. In Proceedings of Thirty Third Conference on Learning Theory, 2020.
C. Rouyer, Y. Seldin, and N. Cesa-Bianchi. An algorithm for stochastic and adversarial bandits with switching costs. In International Conference on Machine Learning, pages 9127-9135. PMLR, 2021.
C. Rouyer, D. van der Hoeven, N. Cesa-Bianchi, and Y. Seldin. A near-optimal best-of-both-worlds algorithm for online learning with feedback graphs. In Advances in Neural Information Processing Systems, 2022.
Y. Seldin and G. Lugosi. An improved parametrization and analysis of the  $\exp 3++$  algorithm for stochastic and adversarial bandits. In Proceedings of the Annual Conference on Learning Theory, 2017.
Y. Seldin and A. Slivkins. One practical algorithm for both stochastic and adversarial bandits. In Proceedings of the International Conference on Machine Learning, 2014.
T. Tsuchiya, S. Ito, and J. Honda. Best-of-both-worlds algorithms for partial monitoring. Algorithmic Learning Theory, 2023.
C.-Y. Wei and H. Luo. More adaptive algorithms for adversarial bandits. In Proceedings of the Annual Conference On Learning Theory, 2018.
J. Zimmert and Y. Seldin. An optimal algorithm for stochastic and adversarial bandits. In The International Conference on Artificial Intelligence and Statistics, 2019.
J. Zimmert and Y. Seldin. Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research, 22(28):1-49, 2021.
J. Zimmert, H. Luo, and C.-Y. Wei. Beating stochastic and adversarial semi-bandits optimally and simultaneously. In Proceedings of the International Conference on Machine Learning, 2019.
A Proofs for MAB Results in Table 1 14
A.1 Preprocessing Cost 17
A.2 Regret on Sub-Optimal Arms 23
A.3 Regret on Optimal Arms 28
A.4 Residual Regret 29
B Proof of Theorem 3.4: Decoupled-Tsallis-INF 38
B.1 Regret Decomposition for Decouple-Tsallis-INF 41
B.2 Regret on Sub-Optimal Arms 43
B.3 Regret on Optimal Arms 43
B.4 Residual Regret 43
B.5 Auxiliary Lemmas for Analysis of Decoupled-Tsallis-INF 48
C Supplementary Lemmas 52
C.1 Proof of Lemma 4.1 52
C.2 Proof of Lemma 4.2 52
C.3 Multiplicative Relation 53
C.4 Generalization of [Ito, 2021, Lemma 21] 57
C.5 Analysis of FTRL Framework and Bregman Divergence 60
A Proofs for MAB Results in Table 1
In this section, we provide details of our novel analysis framework and take a closer look at our learning rate schedule. Importantly, our analysis framework is able to be applied to all configurations of Algorithm 1 for different regularizers in a unified way. As mentioned, the key term in the regret is  $\sum_{t=1}^{T} D^{t,t+1}(p^t, p^{t+1}) - D_U^{t,t+1}(q^t, q^{t+1})$  with  $q_t$  defined in Eq. (5). We thus define the preprocessing cost (PRECOST) as
$$
\operatorname {P R E C O S T} = \mathbb {E} \left[ \operatorname {R e g} ^ {T} - \sum_ {t = 1} ^ {T} \left(D ^ {t, t + 1} \left(p ^ {t}, p ^ {t + 1}\right) - D _ {U} ^ {t, t + 1} \left(q ^ {t}, q ^ {t + 1}\right)\right) \right], \tag {13}
$$
which we later show is small. Then, we use Lemma 4.1 to decompose the key term into the regret related to the sub-optimal arms (REGSUB), the regret related to the optimal arms (REGOPT) and the residual regret (RESREG):
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} D ^ {t, t + 1} \left(p ^ {t}, p ^ {t + 1}\right) - D _ {U} ^ {t, t + 1} \left(q ^ {t}, q ^ {t + 1}\right) \right] = \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} D _ {V} ^ {t , t + 1} \left(p ^ {t} , \bar {p} ^ {t + 1}\right) \right]} _ {\text {R E G S U B}} \\ + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} D _ {U} ^ {t , t + 1} \left(p ^ {t} , \bar {p} ^ {t + 1}\right) - D _ {U} ^ {t , t + 1} \left(q ^ {t} , q ^ {t + 1}\right) \right]} _ {\text {R E G O P T}} + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} D ^ {t + 1} \left(\bar {p} ^ {t + 1} , p ^ {t + 1}\right) \right]} _ {\text {R E S R E G}}. \\ \end{array}
$$
Our goal is to bound all these four terms in terms of two self-bounding quantities for some  $x > 0$  (plus other minor terms):
$$
\mathbb {S} _ {1} (x) = \mathbb {E} \left[ \sqrt {x \cdot \sum_ {t = 1} ^ {T} \sum_ {i \in V} p _ {i} ^ {t}} \right], \quad \mathbb {S} _ {2} (x) = \mathbb {E} \left[ \sum_ {i \in V} \sqrt {x \cdot \sum_ {t = 1} ^ {T} p _ {i} ^ {t}} \right]. \tag {14}
$$
These two quantities enjoy a certain self-bounding property which is known to be critical to achieve the gap-dependent bound in the stochastic setting. Specifically, as we illustrate at the beginning of Section 4, under Condition (1), these self-bounding quantities can be related back to the regret itself when the set  $V$  coincides with that of Condition (1).
Lemma A.1 (Self-Bounding Quantities). Under Condition (1), we have for any  $z > 0$ :
$$
\mathbb {S} _ {1} (x) \leq z \left(\operatorname {R e g} ^ {T} + C\right) + \frac {1}{z} \left(\frac {x}{4 \Delta_ {\mathrm {M I N}}}\right),
$$
$$
\mathbb {S} _ {2} (x) \leq z \left(\operatorname {R e g} ^ {T} + C\right) + \frac {1}{z} \left(\sum_ {i \in V} \frac {x}{4 \Delta_ {i}}\right).
$$
Proof. For any  $z > 0$ , we have
$$
\begin{array}{l} \mathbb {S} _ {1} (x) = \mathbb {E} \left[ \sqrt {\frac {x}{2 z \Delta_ {\mathrm {M I N}}} \cdot \left(2 z \Delta_ {\mathrm {M I N}} \sum_ {t = 1} ^ {T} \sum_ {i \in V} p _ {i} ^ {t}\right)} \right] \\ \leq \mathbb {E} \left[ z \Delta_ {\mathrm {M I N}} \sum_ {t = 1} ^ {T} \sum_ {i \in V} p _ {i} ^ {t} \right] + \frac {x}{4 z \Delta_ {\mathrm {M I N}}} \\ \leq z \left(\operatorname {R e g} ^ {T} + C\right) + \frac {x}{4 z \Delta_ {\mathrm {M I N}}}, \\ \end{array}
$$
where the second step follows from the AM-GM inequality:  $2\sqrt{xy} \leq x + y$  for any  $x, y > 0$ , and the last step follows from Condition (1).
By similar arguments, we have  $\mathbb{S}_2(x)$  bounded for any  $z > 0$  as:
$$
\mathbb {E} \left[ \sum_ {i \in V} \sqrt {\frac {x}{2 z \Delta_ {i}} \cdot 2 z \Delta_ {i} \sum_ {t = 1} ^ {T} p _ {i} ^ {t}} \right] \leq \sum_ {i \in V} \frac {x}{4 z \Delta_ {i}} + z \mathbb {E} \left[ \sum_ {i \in V} \sum_ {t = 1} ^ {T} p _ {i} ^ {t} \Delta_ {i} \right] = z (\operatorname {R e g} ^ {T} + C) + \sum_ {i \in V} \frac {x}{4 z \Delta_ {i}}.
$$
Table 3: Bounds of PRECOST, REGSUB, REGOPT, and RESREG for Algorithm 1 with different regularizers. For simplicity, We omit the  $\mathcal{O}(\cdot)$  notation in these bounds.
Terms	Shannon entropy (CLOG=162 log K)	β-Tsallis entropy (CLOG=162β/1-β)	log-barrier (CLOG=162)
PRECOST (Section A.1)	K(log T)(log K)+D (Lemma A.1.4)	K/√β(1-β)+βK log T/1-β+D (Lemma A.1.6)	S2(log T)+K log T+D (Lemma A.1.5)
REGSUB (Section A.2)	S2(log2T) (Lemma A.2.4)	S2(log T/β(1-β)) (Lemma A.2.3)	S2(log T) (Lemma A.2.2)
REGOPT (Section A.3)	0 (Lemma A.3.1)	0 (Lemma A.3.1)	0 (Lemma A.3.1)
RESREG	S1(|U| log2T)+S2(log2T)+K2 log3/2T (Lemma A.4.5)	S1(|U| log T/β(1-β))+S2(log T/β(1-β))+K2√β/(1-β)3/2 (Lemma A.4.3)	S1(|U| log T)+S2(log T)+K2√logT/√logT (Lemma A.4.4)
RegT	S1(|U| log2T)+S2(log2T)+K2 log3/2T+D (Lemma A.3)	S1(|U| log T/β(1-β))+S2(log T/β(1-β))+K2√β/(1-β)3/2+βK log T/1-β+D (Lemma A.2)	S1(|U| log T)+S2(log T)+K2√logT+K log T+D (Lemma A.4)
In Table 3, we summarize the bounds of PRECOST, REGSUB, REGOPT and RESREG when using Algorithm 1 with Shannon entropy,  $\beta$ -Tsallis entropy, and log-barrier, respectively, together with the corresponding lemmas. Importantly, though the bounds are stated in separated lemmas for different regularizers, the proof ideas and techniques are almost the same with only slight modifications to adapt to the specific regularizer and the choice of parameters.
Using the bound on each of there four terms, we now show how to prove our main theorems, starting with the case for  $\beta$ -Tsallis entropy. First, we simply sum up all the bounds to arrive at the following adaptive regret bound.
Lemma A.2 (Regret Bound for  $\beta$ -Tsallis Entropy). For any  $\beta \in (0,1)$ , when using  $\beta$ -Tsallis entropy with  $C_{\mathrm{LOG}} = \frac{162\beta}{1 - \beta}$ ,  $\alpha = \beta$ , and  $\theta = \sqrt{\frac{1 - \beta}{\beta}}$ , Algorithm 1 guarantees:
$$
\operatorname {R e g} ^ {T} = \mathcal {O} \left(D + \frac {K ^ {2} \sqrt {\beta}}{(1 - \beta) ^ {3 / 2}} + \frac {\beta K \log T}{1 - \beta} + \mathbb {S} _ {1} \left(\frac {| U | \log T}{\beta (1 - \beta)}\right) + \mathbb {S} _ {2} \left(\frac {\log T}{\beta (1 - \beta)}\right)\right),
$$
for any subset  $U \subseteq [K]$ ,  $V = [K] \backslash U$ , and  $D = \mathbb{E}\left[\sum_{t=1}^{T} \max_{i \in U} \mathbb{E}^t\left[\ell_i^t - \ell_{i^\star}^t\right]\right]$ .
We emphasize that this bound holds for any subset  $U \subseteq [K]$ , not just the  $U$  defined with respect to Condition (1) (so a slight abuse of notations here). This is important for proving the regret bound in the adversarial case, as shown in the following proof for Theorem 3.3.
Proof of Theorem 3.3. First, for the adversarial setting, we set  $U = \{i^{\star}\}$  so that  $D = 0$  by its definition. Therefore, we have the self-bounding quantity  $\mathbb{S}_1\left(\frac{|U| \log T}{\beta(1 - \beta)}\right)$  bounded as:
$$
\mathbb {S} _ {1} \left(\frac {| U | \log T}{\beta (1 - \beta)}\right) = \mathbb {E} \left[ \sqrt {\frac {| U | \log T}{\beta (1 - \beta)} \cdot \sum_ {t = 1} ^ {T} \sum_ {i \in V} p _ {i} ^ {t}} \right] = \mathcal {O} \left(\sqrt {\frac {T \log T}{\beta (1 - \beta)}}\right),
$$
where the second step follows from the fact  $|U| = 1$  and  $\sum_{t=1}^{T} \sum_{i \in [K]} p_i^t = T$ . Similarly, we bound the other self-bounding quantity  $\mathbb{S}_2\left(\frac{\log T}{\beta(1 - \beta)}\right)$  as:
$$
\mathbb {S} _ {2} \left(\frac {\log T}{\beta (1 - \beta)}\right) = \mathbb {E} \left[ \sum_ {i \in V} \sqrt {\frac {\log T}{\beta (1 - \beta)} \sum_ {t = 1} ^ {T} p _ {i} ^ {t}} \right] \leq \mathbb {E} \left[ \sqrt {\frac {| V | \log T}{\beta (1 - \beta)} \sum_ {i \in V} \sum_ {t = 1} ^ {T} p _ {i} ^ {t}} \right] = \mathcal {O} \left(\sqrt {\frac {K T \log T}{\beta (1 - \beta)}}\right),
$$
where the second step uses the Cauchy-Schwarz inequality. Combining the bounds together concludes that  $\mathrm{Reg}^T = \mathcal{O}\left(\sqrt{\frac{KT\log T}{\beta(1 - \beta)}}\right)$  in the adversarial setting.
Next, suppose that Condition (1) holds. We now set  $U = \{i \in [K] : \Delta_i = 0\}$  as in Section 2, which makes  $D$  the same as that defined in Section 2 as well. We further denote  $W = D + \frac{K^2 \sqrt{\beta}}{(1 - \beta)^{3/2}} + \frac{\beta K \log T}{1 - \beta}$  to simplify notations. Then, we have  $\operatorname{Reg}^T$  bounded for any  $z > 0$  by
$$
\begin{array}{l} \operatorname {R e g} ^ {T} \leq \kappa \cdot W + \kappa \cdot \mathbb {S} _ {1} \left(\frac {| U | \log T}{\beta (1 - \beta)}\right) + \kappa \cdot \mathbb {S} _ {2} \left(\frac {\log T}{\beta (1 - \beta)}\right) \\ \leq z \left(\operatorname {R e g} ^ {T} + C\right) + \frac {\kappa^ {2}}{z} \cdot \frac {\mathbb {G} \log T}{\beta (1 - \beta)} + \kappa \cdot W, \\ \end{array}
$$
where  $\kappa > 0$  is an absolute constant (based on Lemma A.2), and the second step applies Lemma A.1 (with the arbitrary  $z$  there set to  $\frac{z}{2\kappa}$ ) together with our complexity measure  $\mathbb{G} = \frac{|U|}{\Delta_{\mathrm{MIN}}} + \sum_{i \in V} \frac{1}{\Delta_i}$ .
For any  $z\in (0,1)$  , we can further rearrange and arrive at
$$
\begin{array}{l} \operatorname {R e g} ^ {T} \leq \frac {z C}{1 - z} + \frac {1}{z (1 - z)} \cdot \frac {\kappa^ {2} \mathbb {G} \log T}{\beta (1 - \beta)} + \frac {1}{1 - z} \cdot \kappa W \\ = \frac {C}{x} + \frac {(x + 1) ^ {2}}{x} \cdot \frac {\kappa^ {2} \mathbb {G} \log T}{\beta (1 - \beta)} + \frac {x + 1}{x} \cdot \kappa W \\ = \frac {1}{x} \cdot \left(C + \frac {\kappa^ {2} \mathbb {G} \log T}{\beta (1 - \beta)} + \kappa W\right) + x \cdot \frac {\kappa^ {2} \mathbb {G} \log T}{\beta (1 - \beta)} + \left(\frac {2 \kappa^ {2} \mathbb {G} \log T}{\beta (1 - \beta)} + \kappa W\right), \\ \end{array}
$$
where we define  $x = \frac{1 - z}{z} > 0$  in the second step. Picking up the optimal  $x$  to minimize the right hand side gives
$$
\begin{array}{l} \operatorname {R e g} ^ {T} \leq 2 \sqrt {\left(C + \frac {\kappa^ {2} \mathbb {G} \log T}{\beta (1 - \beta)} + \kappa W\right) \cdot \frac {\kappa^ {2} \mathbb {G} \log T}{\beta (1 - \beta)}} + \frac {2 \kappa^ {2} \mathbb {G} \log T}{\beta (1 - \beta)} + \kappa W \\ \leq 6 \sqrt {\kappa^ {2} \cdot \frac {C \mathbb {G} \log T}{\beta (1 - \beta)}} + \frac {8 \kappa^ {2} \mathbb {G} \log T}{\beta (1 - \beta)} + \kappa W + 6 \sqrt {\kappa W \cdot \frac {\kappa^ {2} \mathbb {G} \log T}{\beta (1 - \beta)}} \\ \leq 6 \sqrt {\kappa^ {2} \cdot \frac {C \mathbb {G} \log T}{\beta (1 - \beta)}} + \frac {1 1 \kappa^ {2} \mathbb {G} \log T}{\beta (1 - \beta)} + 4 \kappa W \\ = \mathcal {O} \left(\sqrt {\frac {C \mathbb {G} \log T}{\beta (1 - \beta)}} + \frac {\mathbb {G} \log T}{\beta (1 - \beta)} + W\right), \\ \end{array}
$$
where the second step follows from the fact that  $\sqrt{x + y + z} \leq 3\left(\sqrt{x} + \sqrt{y} + \sqrt{z}\right)$  for any  $x, y, z \geq 0$ , and the third step uses the AM-GM inequality  $2\sqrt{xy} \leq x + y$  for any  $x, y \geq 0$ .
For the Shannon entropy regularizer and the log-barrier regularizer, we again summarize the adaptive regret bound in the following two lemmas. Theorem 3.2 and Theorem 3.1 then follow immediately. Since the arguments are exactly the same as above, we omit them for simplicity.
Lemma A.3 (Regret Bound for Shannon Entropy). When using the Shannon entropy regularizer with  $C_{\mathrm{LOG}} = 162 \log K$ ,  $\alpha = 1$ , and  $\theta = \sqrt{1 / \log T}$ , Algorithm 1 ensures
$$
\operatorname {R e g} ^ {T} = \mathcal {O} \left(D + K ^ {2} \log^ {3 / 2} T + \mathbb {S} _ {1} \left(| U | \log^ {2} T\right) + \mathbb {S} _ {2} \left(\log^ {2} T\right)\right),
$$
Lemma A.4 (Regret Bound for Log-barrier). When using the log-barrier regularizer with  $C_{\mathrm{LOG}} = 162$ ,  $\alpha = 0$ , and  $\theta = \sqrt{\frac{1}{\log T}}$ , Algorithm 1 ensures
$$
\operatorname {R e g} ^ {T} = \mathcal {O} \left(D + \frac {K ^ {2}}{\sqrt {\log T}} + K \log T + \frac {\beta K \log T}{1 - \beta} + \mathbb {S} _ {1} (| U | \log T) + \mathbb {S} _ {2} (\log T)\right),
$$
In the following four subsections, we discuss how to bound PRECOST, REGSUB, REGOPT, and RESREG respectively. We recall the following definitions discussed in Section 4: the skewed Bregman divergence for two time steps  $s, t \in \mathbb{N}$  is
$$
D ^ {s, t} (x, y) = \phi^ {s} (x) - \phi^ {t} (y) - \left\langle \nabla \phi^ {t} (y), x - y \right\rangle ,
$$
and its variant restricted to any subset  $\mathcal{I} \subseteq [K]$  is
$$
D _ {\mathcal {I}} ^ {s, t} (x, y) = \phi_ {\mathcal {I}} ^ {s} (x) - \phi_ {\mathcal {I}} ^ {t} (y) - \left\langle \nabla \phi_ {\mathcal {I}} ^ {t} (y), x - y \right\rangle ,
$$
where  $\phi_{\mathcal{I}}^t (x) = -C_{\mathrm{LOG}}\sum_{i\in \mathcal{I}}\log x_i - \frac{1}{1 - \beta}\sum_{i\in \mathcal{I}}\gamma_i^t x_i^\beta$  (that is,  $\phi^t$  restricted to  $\mathcal{I}$ ). The standard Bregman divergence associated with  $\phi^t$  is denoted by  $D^{t}(x,y)$ , a shorthand for  $D^{t,t}(x,y)$ . For any subset  $\mathcal{I}\subseteq [K]$  and any vector  $x\in \mathbb{R}^K$ , we define  $x_{\mathcal{I}}\in \mathbb{R}^{K}$  such that its entries in  $\mathcal{I}$  are the same as  $x$  while those in  $[K]\backslash \mathcal{I}$  are 0. Finally, we use  $\mathbf{1}\in \mathbb{R}^{K}$  to denote the all-one vector.
A.1 Preprocessing Cost
Due to the extra log-barrier added to stabilize the algorithm, we consider distribution  $\widetilde{q}^t$  defined as
$$
\widetilde {q} ^ {t} = (1 - \epsilon) \cdot q ^ {t} + \frac {\epsilon}{| V |} \cdot \mathbf {1} _ {V}, \tag {15}
$$
where  $\mathbf{1}_V\in \mathbb{R}^K$ , according to our earlier notations, is the vector with all entries in  $V$  being 1 and the remaining being 0. This specific distribution moves a small amount of weights of  $q^{t}$  from  $U$  to  $V$ , which guarantees that our framework can also work for the log-barrier and  $\log (1 / \tilde{q}_i^t)$  is bounded for  $\forall i\in V$  with a proper  $\epsilon$ .
Lemma A.1.1. For any  $\epsilon \in (0,1)$ , any subset  $U \subseteq [K]$  and  $V = [K] \backslash U$ ,  $q^t$  defined in Eq. (5), and  $\widetilde{q}^t$  defined in Eq. (15), Algorithm 1 ensures
$$
\operatorname {R e g} ^ {T} \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \widehat {\ell} ^ {t}, p ^ {t} - \widetilde {q} ^ {t} \right\rangle \right] + D + \epsilon T,
$$
where  $D = \mathbb{E}\left[\sum_{t=1}^{T} \max_{i \in U} \mathbb{E}^t\left[\ell_i^t - \ell_{i^\star}^t\right]\right]$ .
Proof. By the definition of  $\widetilde{q}^t$  in Eq. (15), one can show
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \left\langle \ell^ {t}, \widetilde {q} ^ {t} \right\rangle - \ell_ {i \star} ^ {t} \right] = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \left\langle \ell^ {t}, \widetilde {q} _ {U} ^ {t} \right\rangle - \ell_ {i \star} ^ {t} \right] + \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \left\langle \ell^ {t}, \widetilde {q} _ {V} ^ {t} \right\rangle \right] \\ = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {E} ^ {t} \left[ \left\langle \ell^ {t}, \widetilde {q} _ {U} ^ {t} \right\rangle - \ell_ {i ^ {\star}} ^ {t} \right] \right] + \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \frac {\epsilon \sum_ {i \in V} \ell_ {i} ^ {t}}{| V |} \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ (1 - \epsilon) \max  _ {i \in U} \mathbb {E} ^ {t} \left[ \ell_ {i} ^ {t} \right] - \mathbb {E} ^ {t} \left[ \ell_ {i ^ {\star}} ^ {t} \right] \right] + \epsilon T \leq D + \epsilon T. \tag {16} \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {R e g} ^ {T} = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \ell_ {i ^ {t}} ^ {t} - \left\langle \ell^ {t}, \widetilde {q} ^ {t} \right\rangle + \left\langle \ell^ {t}, \widetilde {q} ^ {t} \right\rangle - \ell_ {i ^ {\star}} ^ {t} \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \ell_ {i ^ {t}} ^ {t} - \left\langle \ell^ {t}, \widetilde {q} ^ {t} \right\rangle \right] + D + \epsilon T = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \widehat {\ell} ^ {t}, p ^ {t} - \widetilde {q} ^ {t} \right\rangle \right] + D + \epsilon T, \\ \end{array}
$$
where the second step uses Eq. (16), and the last step follows from the law of total expectation.
According to Lemma A.1.1, our main goal here is to bound  $\left\langle \widehat{\ell}^t, p^t - \widetilde{q}^t \right\rangle$ . To this end, we present the following decomposition lemma which is helpful for our further analysis.
Lemma A.1.2 (Lemma 16, Ito [2021]). For  $p^t$  in Algorithm 1 and  $\widetilde{q}^t$  in Eq. (15), we have
$$
\left\langle \widehat {\ell} ^ {t}, p ^ {t} - \widetilde {q} ^ {t} \right\rangle = D ^ {t, t + 1} (p ^ {t}, p ^ {t + 1}) + D ^ {t} (\widetilde {q} ^ {t}, p ^ {t}) - D ^ {t, t + 1} (\widetilde {q} ^ {t}, p ^ {t + 1}).
$$
Armed with Lemma A.1.2, we are now ready to introduce our main result of the regret preprocessing. It is worth noting that the following lemma does not rely on the specific form of regularizers, and thus, we can use it for all the regularizers we consider.
Lemma A.1.3. Suppose that there exist  $B_{1}, B_{2} \in \mathbb{R}$  such that
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \phi_ {U} ^ {t + 1} \left(\widetilde {q} ^ {t + 1}\right) - \phi_ {U} ^ {t} \left(\widetilde {q} ^ {t}\right) \right] \leq B _ {1} + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \phi_ {U} ^ {t + 1} \left(q ^ {t + 1}\right) - \phi_ {U} ^ {t} \left(q ^ {t}\right) \right], \tag {17}
$$
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \phi_ {V} ^ {t + 1} \left(\widetilde {q} ^ {t + 1}\right) - \phi_ {V} ^ {t} \left(\widetilde {q} ^ {t}\right) \right] \leq B _ {2}. \tag {18}
$$
Let  $B = B_{1} + B_{2}$ , and we have
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \widehat {\ell} ^ {t}, p ^ {t} - \widetilde {q} ^ {t} \right\rangle \right] \leq \mathbb {E} \left[ D ^ {1} (\widetilde {q} ^ {1}, p ^ {1}) + \sum_ {t = 1} ^ {T} D ^ {t, t + 1} (p ^ {t}, p ^ {t + 1}) - D _ {U} ^ {t, t + 1} (q ^ {t}, q ^ {t + 1}) \right] + \mathcal {O} \left(\epsilon K T\right) + B.
$$
In other words, we have PRECOST bounded by  $\mathbb{E}\left[D^{1}(\tilde{q}^{1},p^{1}) + \mathcal{O}(\epsilon KT) + B + D\right]$ .
Proof. Adding and subtracting  $D^{t + 1}(\bar{q}^{t + 1},p^{t + 1})$  from the bound in Lemma A.1.2, we have
$$
\begin{array}{l} \left\langle \widehat {\ell} ^ {t}, p ^ {t} - \widetilde {q} ^ {t} \right\rangle = D ^ {t, t + 1} \left(p ^ {t}, p ^ {t + 1}\right) + D ^ {t + 1} \left(\widetilde {q} ^ {t + 1}, p ^ {t + 1}\right) - D ^ {t, t + 1} \left(\widetilde {q} ^ {t}, p ^ {t + 1}\right) \\ + D ^ {t} (\widetilde {q} ^ {t}, p ^ {t}) - D ^ {t + 1} (\widetilde {q} ^ {t + 1}, p ^ {t + 1}). \\ \end{array}
$$
First, we consider the term  $D^{t + 1}(\widetilde{q}^{t + 1},p^{t + 1}) - D^{t,t + 1}(\widetilde{q}^t,p^{t + 1})$  as follows:
$$
\begin{array}{l} D ^ {t + 1} \left(\widetilde {q} ^ {t + 1}, p ^ {t + 1}\right) - D ^ {t, t + 1} \left(\widetilde {q} ^ {t}, p ^ {t + 1}\right) \\ = \phi^ {t + 1} (\widetilde {q} ^ {t + 1}) - \phi^ {t} (\widetilde {q} ^ {t}) - \left\langle \nabla \phi^ {t + 1} (p ^ {t + 1}), \widetilde {q} ^ {t + 1} - \widetilde {q} ^ {t} \right\rangle \\ = \phi^ {t + 1} (\widetilde {q} ^ {t + 1}) - \phi^ {t} (\widetilde {q} ^ {t}) - (1 - \epsilon) \cdot \left\langle \nabla \phi^ {t + 1} \left(p ^ {t + 1}\right), q ^ {t + 1} - q ^ {t} \right\rangle \\ = \phi^ {t + 1} (\widetilde {q} ^ {t + 1}) - \phi^ {t} (\widetilde {q} ^ {t}) - (1 - \epsilon) \cdot \left\langle \nabla \phi_ {U} ^ {t + 1} (p ^ {t + 1}), q ^ {t + 1} - q ^ {t} \right\rangle \\ = \phi^ {t + 1} (\widetilde {q} ^ {t + 1}) - \phi^ {t} (\widetilde {q} ^ {t}) - \left\langle \nabla \phi_ {U} ^ {t + 1} (p ^ {t + 1}), q ^ {t + 1} - q ^ {t} \right\rangle + \epsilon \left\langle \nabla \phi_ {U} ^ {t + 1} (p ^ {t + 1}), q ^ {t + 1} - q ^ {t} \right\rangle \\ = \phi^ {t + 1} (\widetilde {q} ^ {t + 1}) - \phi^ {t} (\widetilde {q} ^ {t}) - \left\langle \nabla \phi_ {U} ^ {t + 1} (q ^ {t + 1}), q ^ {t + 1} - q ^ {t} \right\rangle + \epsilon \left\langle \nabla \phi_ {U} ^ {t + 1} (p ^ {t + 1}), q ^ {t + 1} - q ^ {t} \right\rangle \\ = \phi^ {t + 1} (\widetilde {q} ^ {t + 1}) - \phi^ {t} (\widetilde {q} ^ {t}) - \left\langle \nabla \phi_ {U} ^ {t + 1} (q ^ {t + 1}), q ^ {t + 1} - q ^ {t} \right\rangle + \epsilon \left\langle \widehat {L} _ {U} ^ {t + 1}, q ^ {t + 1} - q ^ {t} \right\rangle , \\ \end{array}
$$
where the second step follows from the fact that  $\widetilde{q}^{t + 1} - \widetilde{q}^t = (1 - \epsilon)\left(q^{t + 1} - q^t\right)$  according to Eq. (15); the fifth step follows from facts that  $\nabla \phi_U^{t + 1}(q^{t + 1}) = \nabla \phi_U^{t + 1}(p^{t + 1}) + c\cdot \mathbf{1}_U$  for a Lagrange multiplier  $c\in \mathbb{R}$  and  $\langle c\cdot \mathbf{1}_K,q^{t + 1} - q^t\rangle = 0$ ; the last step uses the fact that  $\nabla \phi^{t + 1}(p^{t + 1}) = \widehat{L}^{t + 1} + c'\cdot \mathbf{1}_K$  where  $\widehat{L}^t\triangleq \sum_{\tau < t}\widehat{\ell}^\tau$  is the cumulative loss vector prior to round  $t$  and  $c^\prime \in \mathbb{R}$  is another Lagrange multiplier.
By Eq. (17), Eq. (18), and the definition  $B = B_{1} + B_{2}$ , we can further show
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} D ^ {t + 1} \left(\widetilde {q} ^ {t + 1}, p ^ {t + 1}\right) - D ^ {t, t + 1} \left(\widetilde {q} ^ {t}, p ^ {t + 1}\right) \right] \\ \leq B + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \phi_ {U} ^ {t + 1} (q ^ {t + 1}) - \phi_ {U} ^ {t} (q ^ {t}) - \left\langle \nabla \phi_ {U} ^ {t + 1} (q ^ {t + 1}), q ^ {t + 1} - q ^ {t} \right\rangle + \epsilon \left\langle \widehat {L} _ {U} ^ {t + 1}, q ^ {t + 1} - q ^ {t} \right\rangle \right] \\ \end{array}
$$
$$
= B - \mathbb {E} \left[ \sum_ {t = 1} ^ {T} D _ {U} ^ {t, t + 1} \left(q ^ {t}, q ^ {t + 1}\right) \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \epsilon \left\langle \widehat {L} _ {U} ^ {t + 1}, q ^ {t + 1} - q ^ {t} \right\rangle \right].
$$
Then, we can show the following by reorganizing the summation: