SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {R e g} (u) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(f _ {t} \left(a _ {t}\right) - f _ {t} \left(u ^ {\mathcal {A} _ {t}}\right)\right) \right] \\ \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle a _ {t} - u ^ {\mathcal {A} _ {t}}, y _ {t} \right\rangle \right] + \sum_ {t = 1} ^ {T} \varepsilon_ {t} \\ \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {t}} - \boldsymbol {U} ^ {\mathcal {A} _ {t}}, \gamma_ {t} \right\rangle \right] + \varepsilon T = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}}, \gamma_ {t} \right\rangle \right] + \varepsilon T \\ \leq \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}} , \gamma_ {t} - \hat {\gamma} _ {t} \right\rangle \right]} _ {\text {B i a s}} + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}} , \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle \right]} _ {\text {B o u n s}} \\ + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}} , \hat {\gamma} _ {t} - \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle \right]} _ {\text {F T R L - R e g}} + \varepsilon T. \\ \end{array}
$$
Now  $\hat{y}_t = \hat{\Sigma}_t^{-1}(a_t - \hat{x}_t)\ell_t$  with  $\mathbb{E}[\ell_t] = a_t^\top y_t + c_t(a_t)$ .
For the Bias term, the proof is almost the same as Lemma 20. The only difference is that from the fourth line, we have
$$
\mathbb {E} _ {t} \left[ (x _ {t} - u) ^ {\top} \left(y _ {t} - \hat {\Sigma} _ {t} ^ {- 1} (a _ {t} - \hat {x} _ {t}) \left(a _ {t} ^ {\top} y _ {t} + c _ {t} (a _ {t})\right)\right) \right]
$$
for some  $c_{t}(a_{t})$  such that  $|c_{t}(a_{t})|\leq \varepsilon_{t}$  . This leads to an additional term of
$$
\begin{array}{l} \mathbb {E} _ {t} ^ {\mathcal {E}} \left[ - (x _ {t} - u) ^ {\top} \hat {\Sigma} _ {t} ^ {- 1} (a _ {t} - \hat {x} _ {t}) c _ {t} (a _ {t}) \right] \\ \leq \mathbb {E} _ {t} ^ {\mathcal {E}} \left[ \sqrt {(x _ {t} - u) ^ {\top} \hat {\Sigma} _ {t} ^ {- 1} c _ {t} (a _ {t}) ^ {2} (a _ {t} - \hat {x} _ {t}) (a _ {t} - \hat {x} _ {t}) ^ {\top} \hat {\Sigma} _ {t} ^ {- 1} (x _ {t} - u)} \right] \\ \leq \mathbb {E} _ {t} ^ {\mathcal {E}} \left[ \sqrt {(x _ {t} - u) ^ {\top} \hat {\Sigma} _ {t} ^ {- 1} \mathbb {E} _ {\mathcal {A} _ {t} , a _ {t}} [ c _ {t} (a _ {t}) ^ {2} (a _ {t} - \hat {x} _ {t}) (a _ {t} - \hat {x} _ {t}) ^ {\top} ] \hat {\Sigma} _ {t} ^ {- 1} (x _ {t} - u)} \right] \\ \leq \mathbb {E} _ {t} ^ {\mathcal {E}} \left[ \varepsilon_ {t} \sqrt {(x _ {t} - u) ^ {\top} \hat {\Sigma} _ {t} ^ {- 1} (\mathbb {E} _ {\mathcal {A} _ {t} , a _ {t}} [ (a _ {t} - \hat {x} _ {t}) (a _ {t} - \hat {x} _ {t}) ^ {\top} ]) \hat {\Sigma} _ {t} ^ {- 1} (x _ {t} - u)} \right] \\ \leq \mathbb {E} _ {t} ^ {\mathcal {E}} \left[ \varepsilon_ {t} \sqrt {(x _ {t} - u) ^ {\top} \hat {\Sigma} _ {t} ^ {- 1} H _ {t} \hat {\Sigma} _ {t} ^ {- 1} (x _ {t} - u)} \right] \\ \leq \varepsilon_ {t} \| x _ {t} - u \| _ {\hat {\Sigma} _ {t} ^ {- 1}} \\ \end{array}
$$
Plugging it into the proof of Lemma 20, we have
$$
\begin{array}{l} \mathbb {E} _ {t} ^ {\mathcal {E}} \left[ \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}}, \gamma_ {t} - \hat {\gamma} _ {t} \right\rangle \right] \leq \mathcal {O} \left(\sqrt {\frac {d ^ {3} \log (T / \delta)}{t}} + \varepsilon_ {t}\right) \| x _ {t} - u \| _ {\hat {\Sigma} _ {t} ^ {- 1}} \\ \leq \frac {\alpha_ {t}}{4} \| x _ {t} - u \| _ {\hat {\Sigma} _ {t} ^ {- 1}} ^ {2} + \mathcal {O} \left(\frac {d ^ {3} \log (T / \delta)}{\alpha_ {t} t} + \frac {\varepsilon_ {t} ^ {2}}{\alpha_ {t}}\right) \\ \end{array}
$$
Other parts of the proof follow those in Lemma 20. Finally, we get
$$
\begin{array}{l} \mathbf {B i a s} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}}, \gamma_ {t} - \hat {\gamma} _ {t} \right\rangle \right] \\ \leq \frac {1}{4} \sum_ {t = 1} ^ {T} \alpha_ {t} \| x _ {t} - u \| _ {\hat {\Sigma} _ {t} ^ {- 1}} ^ {2} + \mathcal {O} \left(\sum_ {t = 1} ^ {T} \frac {d ^ {3} \log (T / \delta)}{\alpha_ {t} t} + \sum_ {t = 1} ^ {T} \frac {\varepsilon_ {t} ^ {2}}{\alpha_ {t}} + \delta T ^ {2}\right) \\ \end{array}
$$
The Bonus term will not be affected, according to Lemma 23, we have
$$
\mathbf {B o u n s} \leq 2 (d + 2) \sum_ {t = 1} ^ {T} \alpha_ {t} - \frac {1}{4} \sum_ {t = 1} ^ {T} \alpha_ {t} \| u - x _ {t} \| _ {\hat {\Sigma} _ {t} ^ {- 1}} ^ {2} + \mathcal {O} \left(\sum_ {t = 1} ^ {T} \frac {d ^ {3} \alpha_ {t} \log (T / \delta)}{t} + \delta T ^ {2}\right)
$$
The Penalty term will not be affected, according to Lemma 24, we have
$$
\frac {F (\overline {{\boldsymbol {U}}} ^ {\mathcal {A} _ {0}}) - \min  _ {\boldsymbol {H} \in \mathcal {H} ^ {\mathcal {A} _ {0}}} F (\boldsymbol {H})}{\eta_ {T}} \leq \frac {2 d \log (T)}{\eta_ {T}}
$$
Stability-1 term is also unchanged, as we assume that  $\ell_t$  still lies in  $[-1, 1]$  even under misspecification. We still have
$$
\mathbf {S t a b i l i t y - 1} \leq \mathcal {O} \left(d \sum_ {t = 1} ^ {T} \eta_ {t} + \delta T ^ {2}\right)
$$
The Stability-2 term will not be affected as long as  $\eta_t\alpha_t \leq \frac{1}{64t}$ . According to Lemma 27, we have
$$
\mathbf {S t a b i l i t y - 2} \leq \mathcal {O} \left(d \sum_ {t = 1} ^ {T} \alpha_ {t} + \delta T ^ {2}\right)
$$
The Error term is also unaffected. We still have  $\mathbf{Error} = \mathcal{O}(1)$
Adding these terms together, the regret caused by bias and the negative term induced by bonus cancel out. We have
$$
\operatorname {R e g} = \mathcal {O} \left(\frac {d \log (T)}{\eta_ {T}} + d \sum_ {t = 1} ^ {T} \left(\eta_ {t} + \alpha_ {t}\right) + \sum_ {t = 1} ^ {T} \frac {d ^ {3} \log (T / \delta)}{\alpha_ {t} t} + \sum_ {t = 1} ^ {T} \frac {d ^ {3} \alpha_ {t} \log (T / \delta)}{t} + \sum_ {t = 1} ^ {T} \frac {\varepsilon_ {t} ^ {2}}{\alpha_ {t}} + \delta T ^ {2}\right)
$$
Recall that we pick  $\alpha_{t} = \frac{d}{\sqrt{t}} +\frac{\varepsilon}{\sqrt{d}}$ .  $\eta_{t} = \frac{1}{64d\sqrt{t} + 64\frac{\varepsilon}{\sqrt{d}}t}$  and  $\delta = \frac{1}{T^2}$ . This gives
$$
\operatorname {R e g} = \mathcal {O} (d ^ {2} \sqrt {T} \log (T) + d ^ {4} \log (T) + \sqrt {d} \varepsilon T) = \mathcal {O} (d ^ {2} \sqrt {T} \log (T) + \sqrt {d} \varepsilon T)
$$
where we assume  $d^2 \leq \sqrt{T}$  without loss of generality.
E.2 Unknown misspecification
In this subsection, we use a model selection technique to convert the algorithm in Appendix E.1 which requires knowledge on  $\varepsilon$  into an algorithm that achieves a similar regret bound without knowing  $\varepsilon$ . Such a procedure to handle unknown misspecification/corruption has appeared in several previous works [Foster et al., 2020, Wei et al., 2022], though we adopt the technique in Jin et al. [2023] to handle the adversarial case.
The idea here is a black-box reduction which turns an algorithm that only deals with known  $\varepsilon$  to one that handles unknown  $\varepsilon$ . More specifically, the reduction has two layers. The bottom layer takes as input an arbitrary misspecification-robust algorithm that operates under known  $\varepsilon$  (e.g., Algorithm 1), and outputs a stable misspecification-robust algorithm (formally defined later) that still operates under known  $\varepsilon$ . The top layer follows the standard Corral idea and takes as input a stable algorithm that operates under known  $\varepsilon$ , and outputs an algorithm that operates under unknown  $\varepsilon$ . Below, we explain these two layers of reduction in details.
Bottom Layer (from an Arbitrary Algorithm to a Stable Algorithm) The input of the bottom layer is an arbitrary misspecification-robust algorithm, formally defined as:
Definition 29. An algorithm is misspecification-robust if it takes  $\theta$  as input, and achieves the following regret for any random stopping time  $t' \leq T$  and any policy  $u$ :
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {t ^ {\prime}} \left(f _ {t} \left(a _ {t}\right) - f _ {t} \left(u ^ {\mathcal {A} _ {t}}\right)\right) \right] \leq \mathbb {E} \left[ c _ {1} \sqrt {t ^ {\prime}} + c _ {2} \theta \right] + \Pr \left[ \varepsilon_ {1; t ^ {\prime}} > \theta \right] T
$$
for problem-dependent and  $\log (T)$  factors  $c_{1},c_{2}\geq 1$  and  $\varepsilon_{1:t^{\prime}}\triangleq \sqrt{t^{\prime}\sum_{\tau = 1}^{t^{\prime}}\varepsilon_{\tau}^{2}}.$
Input:  $\varepsilon$  and a base algorithm satisfying Definition 29.
Initialize:  $\lceil \log_2T\rceil$  instances of the base algorithm  $\mathsf{ALG}_1,\ldots ,\mathsf{ALG}_{\lceil \log_2T\rceil}$ , where  $\mathsf{ALG}_j$  is configured with the parameter
$$
\theta = \theta_ {j} \triangleq 2 ^ {- j} \varepsilon T + 4 \sqrt {2 ^ {- j} T \log T} + 8 \log (T).
$$
Algorithm 3 STable Algorithm By Independent Learners and Instance SElection (STABILISE)
for  $t = 1,2,\ldots$  do
Receive  $w_{t}$    
if  $w_{t}\leq \frac{1}{T}$  then play an arbitrary policy  $\pi_t$  continue (without updating any instances)   
Let  $j_{t}$  be such that  $w_{t}\in (2^{-j_{t} - 1},2^{-j_{t}}]$    
Let  $\pi_t$  be the policy suggested by  $\mathsf{ALG}_{j_t}$    
Output  $\pi_t$    
If feedback is received, send it to  $\mathsf{ALG}_{j_t}$  with probability  $\frac{2^{-j_t - 1}}{w_t}$  , and discard it otherwise.
In our case,  $c_{1} = \Theta(d^{2}\log T)$  and  $c_{2} = \Theta(\sqrt{d})$ . While the regret bound in Definition 29 might look cumbersome, it is in fact fairly reasonable: if the guess  $\theta$  is not smaller than the true amount of  $\varepsilon_{1:t'}$ , the regret should be of order  $d^{2}\sqrt{t'} + \sqrt{d}\theta$ ; otherwise, the regret bound is vacuous since  $T$  is its largest possible value. The only extra requirement is that the algorithm needs to be anytime (i.e., the regret bound holds for any stopping time  $t'$ ), but even this is known to be easily achievable by using a doubling trick over a fixed-time algorithm. It is then clear that Algorithm 1 (together with a doubling trick) indeed satisfies Definition 29.
As mentioned, the output of the bottom layer is a stable robust algorithm. To characterize stability, we follow Agarwal et al. [2017] and define a new learning protocol that abstracts the interaction between the output algorithm of the bottom layer and the master algorithm from the top layer:
Protocol 1. In every round  $t$ , before the learner makes a decision, a probability  $w_{t} \in [0,1]$  is revealed to the learner. After making a decision, the learner sees the desired feedback from the environment with probability  $w_{t}$ , and sees nothing with probability  $1 - w_{t}$ .
One can convert any misspecification-robust algorithm (defined in Definition 29) into a stable misspecification-robust algorithm (characterized in Theorem 30).
This conversion is achieved by a procedure that called STABILISE (see Algorithm 3 for details). The high-level idea of STABILISE is as follows. Noticing that the challenge when learning in Protocol 1 is that  $w_{t}$  varies over time, we discretize the value of  $w_{t}$  and instantiate one instance of the input algorithm to deal with one possible discretized value, so that it is learning in Protocol 1 but with a fixed  $w_{t}$ , making it straightforward to bound its regret based on what it promises in Definition 29.
More concretely, STABILISE instantiates  $\mathcal{O}(\log_2T)$  instances  $\{\mathsf{ALG}_j\}_{j = 0}^{\lceil \log_2T\rceil}$  of the input algorithm that satisfies Definition 29, each with a different parameter  $\theta_{j}$ . Upon receiving  $w_{t}$  from the environment, it dispatches round  $t$  to the  $j$ -th instance where  $j$  is such that  $w_{t}\in (2^{-j - 1},2^{-j}]$ , and uses the policy generated by  $\mathsf{ALG}_j$  to interact with the environment (if  $w_{t}\leq \frac{1}{T}$ , simply ignore this round). Based on Protocol 1, the feedback for this round is received with probability  $w_{t}$ . To equalize the probability of  $\mathsf{ALG}_j$  receiving feedback as mentioned in the high-level idea, when the feedback is actually obtained, STABILISE sends it to  $\mathsf{ALG}_j$  only with probability  $\frac{2^{-j - 1}}{w_t}$  (and discards it otherwise). This way, every time  $\mathsf{ALG}_j$  is assigned to a round, it always receives the desired feedback with probability  $w_{t}\cdot \frac{2^{-j - 1}}{w_{t}} = 2^{-j - 1}$ . This equalization step allows us to use the original guarantee of the base algorithm (Definition 29) and run it as it is, without requiring it to perform extra importance weighting steps as in Agarwal et al. [2017].
The choice of  $\theta_{j}$  is crucial in making sure that STABILISE only has  $\varepsilon T$  regret overhead instead of  $\frac{\varepsilon T}{\min_{t\in[T]}w_t}$ . Since  $\mathsf{ALG}_j$  only receives feedback with probability  $2^{-j - 1}$ , the expected total misspecification it experiences is on the order of  $2^{-j - 1}\varepsilon T$ . Therefore, its input parameter  $\theta_{j}$  only needs to be of this order instead of the total amount of misspecification  $\varepsilon T$ .
The formal guarantee of the conversion is stated in the following Theorem 30.
Theorem 30. If an algorithm is misspecification robust according to Definition 29 for some constants  $(c_{1}, c_{2})$ , then Algorithm 3 ensures
$$
\operatorname {R e g} \leq \mathcal {O} \left(\mathbb {E} \left[ c _ {1} ^ {\prime} \sqrt {T \rho_ {T}} \right] + c _ {2} ^ {\prime} \varepsilon T\right)
$$
under Protocol 1, where  $\rho_T = \frac{1}{\min_{t \in [T]} w_t}$ , with  $c_1' = \Theta((c_1 + c_2) \sqrt{\log T})$ .
Proof of Theorem 30. Define indicators
$$
g _ {t, j} = \mathbb {I} \left\{w _ {t} \in \left(2 ^ {- j - 1}, 2 ^ {- j} \right] \right\}
$$
$h_{t,j} = \mathbb{I}\{\mathsf{ALG}_j$  receives the feedback for episode  $t\}$
Now we consider the regret of  $\mathsf{ALG}_j$ . Notice that  $\mathsf{ALG}_j$  makes an update only when  $g_{t,j}h_{t,j} = 1$ . By the guarantee of the base algorithm (Definition 29), we have
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(f _ {t} \left(a _ {t}\right) - f _ {t} \left(u ^ {\mathcal {A} _ {t}}\right)\right) g _ {t, j} h _ {t, j} \right] \\ \leq \mathbb {E} \left[ c _ {1} \sqrt {\sum_ {t = 1} ^ {T} g _ {t , j} h _ {t , j}} + c _ {2} \theta_ {j} \max  _ {t \leq T} g _ {t, j} \right] + \Pr \left[ \sqrt {\left(\sum_ {t = 1} ^ {T} g _ {t , j} h _ {t , j}\right) \left(\sum_ {t = 1} ^ {T} \varepsilon_ {t} ^ {2} g _ {t , j} h _ {t , j}\right)} > \theta_ {j} \right] T. \tag {40} \\ \end{array}
$$
We first bound the last term: Notice that  $\mathbb{E}[h_{t,j}|g_{t,j}] = 2^{-j - 1}g_{t,j}$  by Algorithm 3. Therefore,
$$
\sum_ {t = 1} ^ {T} \varepsilon_ {t} ^ {2} g _ {t, j} \mathbb {E} [ h _ {t, j} | g _ {t, j} ] = 2 ^ {- j - 1} \sum_ {t = 1} ^ {T} \varepsilon_ {t} ^ {2} g _ {t, j} \leq 2 ^ {- j - 1} \varepsilon^ {2} T \tag {41}
$$
$$
\sum_ {t = 1} ^ {T} g _ {t, j} \mathbb {E} \left[ h _ {t, j} \mid g _ {t, j} \right] = 2 ^ {- j - 1} \sum_ {t = 1} ^ {T} g _ {t, j} \leq 2 ^ {- j - 1} T \tag {42}
$$
By Freedman's inequality, with probability at least  $1 - \frac{1}{T^2}$ ,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \varepsilon_ {t} ^ {2} g _ {t, j} h _ {t, j} - \sum_ {t = 1} ^ {T} \varepsilon_ {t} ^ {2} g _ {t, j} \mathbb {E} [ h _ {t, j} | g _ {t, j} ] \\ \leq 2 \sqrt {\sum_ {t = 1} ^ {T} \left(\varepsilon_ {t}\right) ^ {4} g _ {t , j} \mathbb {E} \left[ h _ {t , j} \mid g _ {t , j} \right] \log (T)} + 4 \log (T) \\ \leq 4 \sqrt {\sum_ {t = 1} ^ {T} \varepsilon_ {t} ^ {2} g _ {t , j} \mathbb {E} \left[ h _ {t , j} \mid g _ {t , j} \right] \log (T)} + 4 \log (T) \\ \leq \sum_ {t = 1} ^ {T} \varepsilon_ {t} ^ {2} g _ {t, j} \mathbb {E} \left[ h _ {t, j} \mid g _ {t, j} \right] + 8 \log (T) \quad (\text {A M - G M i n e q u a l i t y}) \\ \end{array}
$$
$$
\sum_ {t = 1} ^ {T} \varepsilon_ {t} ^ {2} g _ {t, j} h _ {t, j} \leq 2 \sum_ {t = 1} ^ {T} \varepsilon_ {t} ^ {2} g _ {t, j} \mathbb {E} [ h _ {t, j} | g _ {t, j} ] + 8 \log (T) \leq 2 ^ {- j} \varepsilon^ {2} T + 8 \log (T)
$$
with probability at least  $1 - \frac{1}{T^2}$  using Eq. (41). Similarly,
$$
\sum_ {t = 1} ^ {T} g _ {t, j} h _ {t, j} \leq 2 \sum_ {t = 1} ^ {T} g _ {t, j} \mathbb {E} [ h _ {t, j} | g _ {t, j} ] + 8 \log (T) \leq 2 ^ {- j} T + 8 \log (T)
$$
with probability at least  $1 - \frac{1}{T^2}$ . Therefore, with probability at least  $1 - \frac{2}{T^2}$ ,
$$
\begin{array}{l} \sqrt {\left(\sum_ {t = 1} ^ {T} g _ {t , j} h _ {t , j}\right) \left(\sum_ {t = 1} ^ {T} \varepsilon_ {t} ^ {2} g _ {t , j} h _ {t , j}\right)} \leq \sqrt {2 ^ {- 2 j} \varepsilon^ {2} T ^ {2} + 1 6 \cdot 2 ^ {- j} T \log T + 6 4 \log^ {2} T} \\ \leq 2 ^ {- j} \varepsilon T + 4 \sqrt {2 ^ {- j} T \log T} + 8 \log (T) \\ \leq \theta_ {j} \\ \end{array}
$$
Therefore, the last term in Eq. (40) is bounded by  $\frac{2}{T^2} T \leq \frac{2}{T}$ .
Next, we deal with other terms in Eq. (40). Again, by  $\mathbb{E}[h_{t,j}|g_{t,j}] = 2^{-j - 1}g_{t,j}$ , Eq. (40) implies
$$
2 ^ {- j - 1} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(f _ {t} \left(a _ {t}\right) - f _ {t} \left(u ^ {\mathcal {A} _ {t}}\right)\right) g _ {t, j} \right] \leq \mathbb {E} \left[ c _ {1} \sqrt {2 ^ {- j - 1} \sum_ {t = 1} ^ {T} g _ {t , j}} + c _ {2} \theta_ {j} \max  _ {t \leq T} g _ {t, j} \right] + \frac {2}{T}.
$$
which implies after rearranging:
(using that when  $g_{t,j} = 1$ ,  $\frac{1}{2^{-j - 1}} \leq \frac{2}{w_t}$ , and the definition of  $\theta_j$ )
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(f _ {t} \left(a _ {t}\right) - f _ {t} \left(u ^ {\mathcal {A} _ {t}}\right)\right) g _ {t, j} \right] \\ \leq \mathbb {E} \left[ c _ {1} \sqrt {\frac {1}{2 ^ {- j - 1}} \sum_ {t = 1} ^ {T} g _ {t , j}} + \left(\frac {c _ {2} \theta_ {j}}{2 ^ {- j - 1}}\right) \max  _ {t \leq T} g _ {t, j} \right] + \frac {2}{T 2 ^ {- j - 1}} \\ \leq \mathbb {E} \left[ c _ {1} \sqrt {\sum_ {t = 1} ^ {T} \frac {2 g _ {t , j}}{w _ {t}}} + 4 c _ {2} \left(\varepsilon T + \sqrt {\frac {T \log T}{2 ^ {- j}}} + \log T\right) \max  _ {t \leq T} g _ {t, j} \right] + \frac {2}{T 2 ^ {- j - 1}}. \\ \end{array}
$$
Now, summing this inequality over all  $j \in \{0, 1, \dots, \lceil \log_2 T \rceil \}$ , we get
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(f _ {t} \left(a _ {t}\right) - f _ {t} \left(u ^ {\mathcal {A} _ {t}}\right)\right) \mathbb {I} \left\{w _ {t} > \frac {1}{T} \right\} \right] \\ \leq \mathcal {O} \left(\mathbb {E} \left[ c _ {1} \sqrt {N \sum_ {t = 1} ^ {T} \frac {1}{w _ {t}}} + N c _ {2} \varepsilon T + c _ {2} \sqrt {\frac {T \log T}{\min  _ {t \leq T} w _ {t}}} + c _ {2} N \log T \right] + 1\right) \\ \leq \mathcal {O} \left(\mathbb {E} \left[ (c _ {1} + c _ {2}) \sqrt {T \log (T) \rho_ {T}} \right] + c _ {2} \varepsilon T \log T\right) \\ \end{array}
$$
where  $N \leq \mathcal{O}(\log T)$  is the number of  $\mathrm{ALG}_j$ 's that has been executed at least once.
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(f _ {t} \left(a _ {t}\right) - f _ {t} \left(u ^ {\mathcal {A} _ {t}}\right)\right) \mathbb {I} \left\{w _ {t} \leq \frac {1}{T} \right\} \right] <   T \mathbb {E} \left[ \mathbb {I} \left\{\rho_ {T} \geq T \right\} \right] \leq \mathbb {E} \left[ \rho_ {T} \right].
$$
Combining the two parts and using the assumption  $c_{2} \geq 1$  finishes the proof.
Top Layer (from Known  $\varepsilon$  to Unknown  $\varepsilon$ ) In this subsection, we use the algorithm that we construct in Theorem 30 as a base algorithm, and further construct an algorithm with  $\sqrt{T} + \varepsilon$  regret under unknown  $\varepsilon$ . The idea is to run multiple base algorithms, each with a different hypothesis on  $\varepsilon$ ; on top of them, run another multi-armed bandit algorithm to adaptively choose among them. The goal is to let the top-level bandit algorithm perform almost as well as the best base algorithm. This is the Corral idea outlined in Agarwal et al. [2017], Foster et al. [2020], Luo et al. [2022], and the algorithm is presented in Algorithm 4.
Theorem 31. Using an algorithm constructed in Theorem 30 as a base algorithm, Algorithm 4 ensures  $\operatorname{Reg} = \mathcal{O}\left(c_1' \sqrt{T \log^3 T} + c_2' \varepsilon T\right)$  without knowing  $\varepsilon$ .
Algorithm 4 (A Variant of) Corral
Initialize: a log-barrier algorithm with each arm being an instance of an algorithm satisfying the guarantee in Theorem 30. The hypothesis on  $\varepsilon T$  is set to  $2^{i}$  for arm  $i$  ( $i = 1, 2, \ldots, M \triangleq \lceil \log_2 T \rceil$ ). Initialize:  $\rho_{0,i} = M$ ,  $\forall i$ .
for  $t = 1,2,\ldots ,T$  do
$$
w _ {t} = \operatorname * {a r g m i n} _ {w \in \Delta (M), w _ {i} \geq \frac {1}{T}, \forall i} \left\{\left\langle w, \sum_ {\tau = 1} ^ {t - 1} \left(\hat {z} _ {\tau} - r _ {\tau}\right) \right\rangle + \frac {1}{\eta} \sum_ {i = 1} ^ {M} \log \frac {1}{w _ {i}} \right\}
$$
where  $\eta = \frac{1}{4c_1'\sqrt{T}}$ .
For all  $i$ , send  $w_{t,i}$  to instance  $i$ .
Draw  $i_t \sim w_t$ .
Execute the  $a_{t}$  output by instance  $i_{t}$
Receive the loss  $z_{t,i_t}$  for action  $a_t$  (whose expectation is  $f_t(a_t)$ ) and send it to instance  $i_t$ . Define for all  $i$ :
$$
\hat {z} _ {t, i} = \frac {z _ {t , i} \mathbb {I} [ i _ {t} = i ]}{w _ {t , i}},
$$
$$
\rho_ {t, i} = \min  _ {\tau \leq t} \frac {1}{w _ {\tau , i}},
$$
$$
r _ {t, i} = c _ {1} ^ {\prime} \left(\sqrt {\rho_ {t , i} T} - \sqrt {\rho_ {t - 1 , i} T}\right).
$$
The top-level bandit algorithm is an FTRL with log-barrier regularizer. We first state the standard regret bound of FTRL under log-barrier regularizer, whose proof can be found in, e.g., Theorem 7 of Wei and Luo [2018].
Lemma 32. The FTRL algorithm over a convex subset  $\Omega$  of the  $(M - 1)$ -dimensional simplex  $\Delta(M)$ :
$$
w _ {t} = \underset {w \in \Omega} {\operatorname {a r g m i n}} \left\{\left\langle w, \sum_ {\tau = 1} ^ {t - 1} \ell_ {\tau} \right\rangle + \frac {1}{\eta} \sum_ {i = 1} ^ {M} \log \frac {1}{w _ {i}} \right\}
$$
ensures for all  $u\in \Omega$
$$
\sum_ {t = 1} ^ {T} \langle w - u, \ell_ {t} \rangle \leq \frac {M \log T}{\eta} + \eta \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} w _ {t, i} ^ {2} \ell_ {t, i} ^ {2}
$$
as long as  $\eta w_{t,i}|\ell_{t,i}|\leq \frac{1}{2}$  for all  $t,i$
Proof of Theorem 31. The Corral algorithm is essential an FTRL with log-barrier regularizer. To apply Lemma 32, we first verify the condition  $\eta w_{t,i}|\ell_{t,i}| \leq \frac{1}{2}$  where  $\ell_{t,i} = \hat{z}_{t,i} - r_{t,i}$ . By our choice of  $\eta$ ,
$$
\eta w _ {t, i} | \hat {z} _ {t, i} | \leq \eta z _ {t, i} \leq \frac {1}{4}, \quad (\text {b e c a u s e} c _ {1} ^ {\prime} \geq 1)
$$
$$
\eta w _ {t, i} r _ {t, i} = \eta c _ {1} ^ {\prime} \sqrt {T} w _ {t, i} (\sqrt {\rho_ {t , i}} - \sqrt {\rho_ {t - 1 , i}}).
$$
The right-hand side of the last equality is non-zero only when  $\rho_{t,i} > \rho_{t - 1,i}$ , implying that  $\rho_{t,i} = \frac{1}{w_{t,i}}$ . Therefore, we further bound it by
$$
\begin{array}{l} \eta w _ {t, i} r _ {t, i} \leq \eta c _ {1} ^ {\prime} \sqrt {T} \frac {1}{\rho_ {t , i}} (\sqrt {\rho_ {t , i}} - \sqrt {\rho_ {t - 1 , i}}) \\ = \eta c _ {1} ^ {\prime} \sqrt {T} \left(\frac {1}{\sqrt {\rho_ {t , i}}} - \frac {\sqrt {\rho_ {t - 1 , i}}}{\rho_ {t , i}}\right) \\ \leq \eta c _ {1} ^ {\prime} \sqrt {T} \left(\frac {1}{\sqrt {\rho_ {t - 1 , i}}} - \frac {1}{\sqrt {\rho_ {t , i}}}\right) \quad \left(\frac {1}{\sqrt {a}} - \frac {\sqrt {b}}{a} \leq \frac {1}{\sqrt {b}} - \frac {1}{\sqrt {a}} \text {f o r} a, b > 0\right) \tag {43} \\ \leq \eta c _ {1} ^ {\prime} \sqrt {T} \quad (\rho_ {t, i} \geq 1) \\ = \frac {1}{4} \quad (\text {d e f i n i t i o n o f} \eta) \\ \end{array}
$$
which can be combined to get the desired property  $\eta w_{t,i}|\hat{z}_{t,i} - r_{t,i}|\leq \frac{1}{2}$
Hence, by the regret guarantee of log-barrier FTRL (Lemma 32), we have