SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(z _ {t, i _ {t}} - z _ {t, i ^ {*}}\right) \right] \\ \leq \mathcal {O} \left(\frac {M \log T}{\eta} + \eta \mathbb {E} \left[ \underbrace {\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} w _ {t , i} ^ {2} (\hat {z} _ {t , i} - r _ {t , i}) ^ {2}} _ {\text {t e r m} _ {1}} \right]\right) + \mathbb {E} \left[ \underbrace {\sum_ {t = 1} ^ {T} \left(\sum_ {i = 1} ^ {M} w _ {t , i} r _ {t , i} - r _ {t , i ^ {\star}}\right)} _ {\text {t e r m} _ {2}} \right] \\ \end{array}
$$
where  $i^*$  is the smallest  $i$  such that  $2^i$  upper bounds the true total misspecification amount  $\varepsilon T$ .
Bounding term:
$$
\mathbf {t e r m} _ {1} \leq 2 \eta \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} w _ {t, i} ^ {2} \left(\hat {z} _ {t, i} ^ {2} + r _ {t, i} ^ {2}\right)
$$
$$
2 \eta \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} w _ {t, i} ^ {2} \hat {z} _ {t, i} ^ {2} = 2 \eta \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} z _ {t, i} ^ {2} \mathbb {I} \{i _ {t} = i \} \leq \mathcal {O} (\eta T)
$$
$$
\begin{array}{l} 2 \eta \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} w _ {t, i} ^ {2} r _ {t, i} ^ {2} \leq 4 \eta \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} \left(c _ {1} ^ {\prime} \sqrt {T}\right) ^ {2} \left(\frac {1}{\sqrt {\rho_ {t - 1 , i}}} - \frac {1}{\sqrt {\rho_ {t , i}}}\right) ^ {2} \quad (\text {c o n t i n u e f r o m E q . (4 3)}) \\ \leq 4 \eta c _ {1} ^ {\prime 2} T \times \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} \left(\frac {1}{\sqrt {\rho_ {t - 1 , i}}} - \frac {1}{\sqrt {\rho_ {t , i}}}\right) \\ \left(\frac {1}{\sqrt {\rho_ {t - 1 , i}}} - \frac {1}{\sqrt {\rho_ {t , i}}} \leq 1 \text {a n d} 1 - a \leq - \ln a\right) \\ \leq 4 \eta c _ {1} ^ {\prime 2} T M ^ {\frac {3}{2}}. \quad (\text {t e l e s c o p i n g a n d u s i n g} \rho_ {0, i} = M \text {a n d} \rho_ {T, i} \leq T) \\ \end{array}
$$
Bounding term2:
$$
\begin{array}{l} \mathbf {t e r m} _ {2} = \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} w _ {t, i} r _ {t, i} - \sum_ {t = 1} ^ {T} r _ {t, i ^ {\star}} \\ \leq c _ {1} ^ {\prime} \sqrt {T} \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {M} \left(\frac {1}{\sqrt {\rho_ {t - 1 , i}}} - \frac {1}{\sqrt {\rho_ {t , i}}}\right) - \left(c _ {1} ^ {\prime} \sqrt {\rho_ {T , i ^ {\star}} T} - c _ {1} ^ {\prime} \sqrt {\rho_ {0 , i ^ {\star}} T}\right) \\ \end{array}
$$
(continue from Eq. (43) and using  $1 - a \leq -\ln a$ )
$$
\leq \mathcal {O} \left(c _ {1} ^ {\prime} \sqrt {T} M ^ {\frac {3}{2}}\right) - c _ {1} ^ {\prime} \sqrt {\rho_ {T , i ^ {\star}} T}.
$$
Combining the two terms and using  $\eta = \Theta \left(\frac{1}{c_1' \sqrt{T} + c_2'}\right)$ ,  $M = \Theta (\log T)$ , we get
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(f _ {t} \left(a _ {t}\right) - z _ {t, i ^ {\star}}\right) \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(z _ {t, i _ {t}} - z _ {t, i ^ {\star}}\right) \right] \\ = \mathcal {O} \left(c _ {1} ^ {\prime} \sqrt {T \log^ {3} T}\right) - \mathbb {E} \left[ c _ {1} ^ {\prime} \sqrt {\rho_ {T , i ^ {*}} T} \right] \tag {44} \\ \end{array}
$$
On the other hand, by the guarantee of the base algorithm (Theorem 30) and that  $\varepsilon T\in [2^{i^{\star} - 1},2^{i^{\star}}]$ , we have
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(z _ {t, i ^ {*}} - f _ {t} \left(u ^ {\mathcal {A} _ {t}}\right) \right. \right] \leq \mathbb {E} \left[ c _ {1} ^ {\prime} \sqrt {\rho_ {T , i ^ {*} T}} \right] + c _ {2} ^ {\prime} \varepsilon T. \tag {45}
$$
Combining Eq. (44) and Eq. (45), we get
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(f _ {t} \left(a _ {t}\right) - f _ {t} \left(u ^ {\mathcal {A} _ {t}}\right)\right) \right] \leq \mathcal {O} \left(c _ {1} ^ {\prime} \sqrt {T \log^ {3} T}\right) + c _ {2} ^ {\prime} \varepsilon T,
$$
Proof of Theorem 3. As shown in Appendix E.1, our Algorithm 1 can be adapted to satisfy Definition 29 with  $c_{1} = \Theta (d^{2}\log T)$  and  $c_{2} = \Theta (\sqrt{d})$ . By a concatenation of Theorem 30 and Theorem 31, we conclude that there is an algorithm that achieves
$$
\mathcal {O} \left((c _ {1} + c _ {2}) \sqrt {T} \log^ {2} T + c _ {2} \varepsilon T \log T\right) = \mathcal {O} \left(d ^ {2} \sqrt {T} \log^ {2} T + \sqrt {d} \varepsilon T \log T\right).
$$
regret under unknown  $\varepsilon$
F Analysis for Linear EXP4
Proof of Theorem 4. We first show that
$$
\forall \pi \in \Pi : \operatorname {R e g} (\pi) \triangleq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} a _ {t} ^ {\top} y _ {t} - \sum_ {t = 1} ^ {T} \pi \left(\mathcal {A} _ {t}\right) ^ {\top} y _ {t} \right] \leq \mathcal {O} \left(\gamma T + \frac {\ln | \Pi |}{\eta} + \eta d T\right). \tag {46}
$$
The magnitude of the loss is bounded by
$$
\begin{array}{l} | \hat {\ell} _ {t, \pi} | = \left| \left\langle \pi (\mathcal {A} _ {t}), \tilde {H} _ {t} ^ {- 1} a _ {t} \ell_ {t} \right\rangle \right| \\ \leq \left\| \pi (\mathcal {A} _ {t}) \right\| _ {\tilde {H} _ {t} ^ {- 1}} \left\| a _ {t} \right\| _ {\tilde {H} _ {t} ^ {- 1}} \\ \leq \frac {1}{\gamma} \| \pi (\mathcal {A} _ {t}) \| _ {G _ {t} ^ {- 1}} \| a _ {t} \| _ {G _ {t} ^ {- 1}} \leq \frac {d}{\gamma}. \\ \end{array}
$$
If  $\gamma \geq 2d\eta$ , then we have  $|\hat{\ell}_{t,\pi}| \leq \frac{1}{2}$  and we can use the standard regret bound of exponential weights:
$$
\forall \pi \in \Pi : \quad \operatorname {R e g} (\pi) \leq \gamma T + \frac {\ln | \Pi |}{\eta} + \eta \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {E} _ {a _ {t} \sim p _ {t}} \left[ \sum_ {\pi \in \Pi} P _ {t, \pi} \hat {\ell} _ {t, \pi} ^ {2} \right] \right].
$$
Let  $H_{t} = \mathbb{E}_{a\sim p_{t}}[aa^{\top}]$ . Then we have  $\tilde{H}_t^{-1}\preceq \frac{1}{1 - \gamma} H_t^{-1}$ , and thus
$$
\begin{array}{l} \mathbb {E} _ {a _ {t} \sim p _ {t}} \left[ \sum_ {\pi \in \Pi} P _ {t, \pi} \hat {\ell} _ {t, \pi} ^ {2} \right] \leq \mathbb {E} _ {a _ {t} \sim p _ {t}} \left[ \sum_ {\pi \in \Pi} P _ {t, \pi} \cdot \langle \pi (\mathcal {A} _ {t}), \tilde {H} _ {t} ^ {- 1} a _ {t} \rangle^ {2} \right] \\ = \mathbb {E} _ {a _ {t} \sim p _ {t}} \mathbb {E} _ {a \sim p _ {t}} \left[ \langle a, \tilde {H} _ {t} ^ {- 1} a _ {t} \rangle^ {2} \right] \quad (\text {b y}) \\ \leq \frac {1}{(1 - \gamma) ^ {2}} \operatorname {T r} \left(H _ {t} H _ {t} ^ {- 1} H _ {t} H _ {t} ^ {- 1}\right) = \mathcal {O} (d). \\ \end{array}
$$
Combining all proves Eq. (46).
Next, we show that there exists  $\theta \in \Theta$  such that
$$
\mathbb {E} _ {\mathcal {A} \sim D} \left[ \sum_ {t = 1} ^ {T} \left(\pi_ {\theta} (\mathcal {A}) - \pi^ {\star} (\mathcal {A})\right) ^ {\top} y _ {t} \right] \leq \mathcal {O} (1). \tag {47}
$$
Let  $\hat{\theta}$  be the closest element in  $\Theta$  to  $\sum_{t=1}^{T} y_t$ . By the definition of  $\Theta$  and the assumption that  $\|y_t\| \leq 1$ , we have  $\left\|\hat{\theta} - \sum_{t=1}^{T} y_t\right\| \leq \epsilon$ . Thus, for any  $\mathcal{A}$ ,
$$
\sum_ {t = 1} ^ {T} (\pi_ {\hat {\theta}} (\mathcal {A}) - \pi^ {\star} (\mathcal {A})) ^ {\top} y _ {t} \leq \sum_ {a \in \mathcal {A}} (\pi_ {\hat {\theta}} (\mathcal {A}) - \pi^ {\star} (\mathcal {A})) ^ {\top} \hat {\theta} + \epsilon \leq \epsilon
$$
where the last inequality is by the fact that  $\pi_{\hat{\theta}}(\mathcal{A}) = \operatorname*{argmin}_{a\in \mathcal{A}}a^{\top}\hat{\theta}$ . Taking expectation over  $\mathcal{A}$  gives Eq. (47).
Finally, combining Eq. (46) and Eq. (47), choosing  $\epsilon = 1$  and  $\gamma = 2d\eta = 2d\sqrt{\frac{\log T}{T}}$ , we get
$$
\begin{array}{l} \operatorname {R e g} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} a _ {t} ^ {\top} y _ {t} - \sum_ {t = 1} ^ {T} \pi^ {\star} (\mathcal {A} _ {t}) ^ {\top} y _ {t} \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} a _ {t} ^ {\top} y _ {t} - \sum_ {t = 1} ^ {T} \pi_ {\hat {\theta}} (\mathcal {A} _ {t}) ^ {\top} y _ {t} \right] + \mathbb {E} _ {\mathcal {A} \sim D} \left[ \sum_ {t = 1} ^ {T} \left(\pi_ {\hat {\theta}} (\mathcal {A}) - \pi^ {\star} (\mathcal {A})\right) ^ {\top} y _ {t} \right] \\ = \mathcal {O} \left(\gamma T + \frac {\ln ((2 T) ^ {d})}{\eta} + \eta d T + 1\right) \\ = \mathcal {O} \left(d \sqrt {T \log T}\right), \\ \end{array}
$$
G Comparison with Dai et al. [2023] and Sherman et al. [2023]
We state the exponential weight algorithm adopted by Luo et al. [2021], Dai et al. [2023], Sherman et al. [2023] in Algorithm 5, which is an algorithm that we know to achieve the prior-art regret bound in our setting (though they studied a more general MDP setting).
Their algorithm proceeds in epochs (indexed by  $k$ ), where every epoch consists of  $W$  rounds. The policy on action set  $\mathcal{A}$  in the  $k$ -th epoch is defined as
$$
p _ {k} ^ {\mathcal {A}} (a) \propto \exp \left(- \eta \sum_ {s = 1} ^ {k - 1} \left(a ^ {\top} \hat {y} _ {s} - b _ {s} (a)\right)\right)
$$
where  $\hat{y}_k$  is the loss estimator for epoch  $k$ , and  $b_k(a)$  is a (non-linear) bonus. In all  $W$  rounds in epoch  $k$ , the same policy is executed. The samples obtained in these  $W$  rounds are randomly divided into two halves. One half is used to estimate the covariance matrix  $\hat{\Sigma}_k$ , and the other half is used to construct the loss estimator  $\hat{y}_k$  (see Line 5 of Algorithm 5).
Algorithm 5 Exponential weights with magnitude-reduced loss estimators
1 for  $k = 1,2,\ldots ,\frac{T}{W}$  do
2 For all  $\mathcal{A}$ , define
$$
p _ {k} ^ {\mathcal {A}} (a) = \frac {\exp \left(- \eta \sum_ {s = 1} ^ {k - 1} \left(a ^ {\top} \hat {y} _ {s} - b _ {s} (a)\right)\right)}{\sum_ {a ^ {\prime} \in \mathcal {A}} \exp \left(- \eta \sum_ {s = 1} ^ {k - 1} \left(a ^ {\prime \top} \hat {y} _ {s} - b _ {s} (a ^ {\prime})\right)\right)} \quad \text {f o r a l l} a \in \mathcal {A}.
$$
3 | Randomly partition  $\{(k - 1)W + 1,\dots ,kW\}$  into two equal parts  $\mathcal{T}_k,\mathcal{T}_k^{\prime}$
4 for  $t = (k - 1)W + 1,\dots ,kW$  do receive  $\mathcal{A}_t$  , sample  $a_{t}\sim p_{k}^{\mathcal{A}_{t}}$  , and receive  $\ell_t$
5 Define
$$
\begin{array}{l} \hat {\Sigma} _ {k} = \beta I + \frac {1}{| \mathcal {T} _ {k} |} \sum_ {t \in \mathcal {T} _ {k}} a _ {t} a _ {t} ^ {\top} \\ \hat {y} _ {k} = \hat {\Sigma} _ {k} ^ {- 1} \left(\frac {1}{| \mathcal {T} _ {k} ^ {\prime} |} \sum_ {t \in \mathcal {T} _ {k} ^ {\prime}} a _ {t} \ell_ {t}\right) \\ b _ {k} (a) = \alpha \| a \| _ {\hat {\Sigma} _ {k} ^ {- 1}}. \\ \end{array}
$$
G.1 Regret Analysis Sketch
The regret analysis starts with a standard decomposition that is similar to ours. We abuse the notation by defining  $y_{k} = \frac{1}{W}\sum_{t=(k-1)W}^{kW}y_{t}$ . Then
$$
\begin{array}{l} \operatorname {R e g} = W \mathbb {E} \left[ \sum_ {k = 1} ^ {T / W} p _ {k} ^ {\mathcal {A} _ {0}} (a) \langle a - u ^ {\mathcal {A} _ {0}}, y _ {k} \rangle \right] \\ = \underbrace {W \mathbb {E} \left[ \sum_ {k = 1} ^ {T / W} p _ {k} ^ {\mathcal {A} _ {0}} (a) \left(\langle a , \hat {y} _ {k} \rangle - b _ {k} (a)\right) - \left(u ^ {\mathcal {A} _ {0}} - b _ {k} \left(u ^ {\mathcal {A} _ {0}}\right)\right) \right]} _ {\text {E W - R e g}} + \underbrace {W \mathbb {E} \left[ \sum_ {k = 1} ^ {T / W} p _ {k} ^ {\mathcal {A} _ {0}} (a) b _ {k} (a) - b _ {k} \left(u ^ {\mathcal {A} _ {0}}\right) \right]} _ {\text {B o u n s}} \\ + \underbrace {W \mathbb {E} \left[ \sum_ {k = 1} ^ {T / W} p _ {k} ^ {\mathcal {A} _ {0}} (a) \langle a - u ^ {\mathcal {A} _ {0}}, y _ {k} - \hat {y} _ {k} \rangle \right]} _ {\textbf {B i a s}}. \\ \end{array}
$$
Bounding the regret term follows the standard analysis of exponential weight:
$$
\begin{array}{l} \mathbf {E W} - \mathbf {R e g} \leq W \mathbb {E} \left[ \frac {\ln | \mathcal {A} _ {0} |}{\eta} + \eta \sum_ {k = 1} ^ {T / W} \sum_ {a \in \mathcal {A} _ {0}} p _ {k} ^ {\mathcal {A} _ {0}} (a) \langle a, \hat {y} _ {k} \rangle^ {2} + \eta \sum_ {k = 1} ^ {T / W} \sum_ {a \in \mathcal {A} _ {0}} p _ {k} ^ {\mathcal {A} _ {0}} (a) b _ {k} (a) ^ {2} \right] \\ \leq W \mathbb {E} \left[ \frac {\ln | \mathcal {A} _ {0} |}{\eta} + \eta \sum_ {k = 1} ^ {T / W} \sum_ {a \in \mathcal {A} _ {0}} p _ {k} ^ {\mathcal {A} _ {0}} (a) a ^ {\top} \hat {\Sigma} _ {k} ^ {- 1} H _ {k} \hat {\Sigma} _ {k} ^ {- 1} a + \eta \sum_ {k = 1} ^ {T / W} \frac {\alpha^ {2}}{\beta} \right] \\ \end{array}
$$
where  $H_{k} = \mathbb{E}_{\mathcal{A}\sim D}\mathbb{E}_{a\sim p_{k}^{\mathcal{A}}}[aa^{\top}]$ . Then they use the following fact to bound the stability term: as long as  $W\geq \frac{d}{\beta^{2}}$ , it holds with high probability that  $\hat{\Sigma}_k^{-1}H_k\hat{\Sigma}_k^{-1}\preceq 2\hat{\Sigma}_k^{-1}$ . Thus EW-Reg can be
further bounded by
$$
\begin{array}{l} \mathbf {E W} - \mathbf {R e g} \lesssim W \left(\frac {\ln | \mathcal {A} _ {0} |}{\eta} + \eta \mathbb {E} \left[ \sum_ {k = 1} ^ {T / W} \sum_ {a \in \mathcal {A} _ {0}} p _ {k} ^ {\mathcal {A} _ {0}} (a) \| a \| _ {\hat {\Sigma} _ {k} ^ {- 1}} ^ {2} \right] + \eta \frac {T}{W} \frac {\alpha^ {2}}{\beta}\right) \\ \leq \frac {W \ln | \mathcal {A} _ {0} |}{\eta} + \eta d T + \eta T \frac {\alpha^ {2}}{\beta}. \\ \end{array}
$$
By the definition of the bonus function  $b_{t}$ , it holds that
$$
\mathbf {B o o n u s} = W \mathbb {E} \left[ \alpha \sum_ {k = 1} ^ {T / W} \sum_ {a \in \mathcal {A} _ {0}} p _ {k} ^ {\mathcal {A} _ {0}} (a) \| a \| _ {\hat {\Sigma} _ {k} ^ {- 1}} \right] - W \mathbb {E} \left[ \alpha \sum_ {k = 1} ^ {T / W} \| u ^ {\mathcal {A} _ {0}} \| _ {\hat {\Sigma} _ {k} ^ {- 1}} \right].
$$
Finally, the bias term can be bounded as follows:
$$
\begin{array}{l} \mathbf {B i a s} = W \mathbb {E} \left[ \sum_ {k = 1} ^ {T / W} p _ {k} ^ {\mathcal {A} _ {0}} (a) \left(a - u ^ {\mathcal {A} _ {0}}\right) ^ {\top} \left(y _ {k} - \hat {\Sigma} _ {k} ^ {- 1} H _ {k} y _ {k}\right) \right] \\ = W \mathbb {E} \left[ \sum_ {k = 1} ^ {T / W} p _ {k} ^ {\mathcal {A} _ {0}} (a) (a - u ^ {\mathcal {A} _ {0}}) ^ {\top} \hat {\Sigma} _ {k} ^ {- 1} (\hat {\Sigma} _ {k} - H _ {k}) y _ {k} \right] \\ \leq W \mathbb {E} \left[ \sum_ {k = 1} ^ {T / W} p _ {k} ^ {\mathcal {A} _ {0}} (a) \| a - u ^ {\mathcal {A} _ {0}} \| _ {\hat {\Sigma} _ {k} ^ {- 1}} \| (\hat {\Sigma} _ {k} - H _ {k}) y _ {k} \| _ {\hat {\Sigma} _ {k} ^ {- 1}} \right]. \\ \end{array}
$$
The bias here has a similar form as in our case. They use the following fact to bound the bias: as long as  $W \geq \frac{d}{\beta^2}$ , it holds that  $\| (\hat{\Sigma}_k - H_k)y_k\|_{\hat{\Sigma}_k^{-1}} \leq \sqrt{\beta d}$ . Therefore, the bias can further be upper bounded by
$$
\mathbf {B i a s} \leq W \mathbb {E} \left[ \sqrt {\beta d} \sum_ {k = 1} ^ {T / W} \sum_ {a \in \mathcal {A} _ {0}} p _ {k} ^ {\mathcal {A} _ {0}} (a) \| a \| _ {\hat {\Sigma} _ {k} ^ {- 1}} + \sqrt {\beta d} \sum_ {k = 1} ^ {T / W} \| u ^ {\mathcal {A} _ {0}} \| _ {\hat {\Sigma} _ {k} ^ {- 1}} \right].
$$
Combining the three parts, we get that the overall regret is of order
$$
\mathbb {E} \left[ \frac {W \ln | \mathcal {A} _ {0} |}{\eta} + \eta d T + \eta T \frac {\alpha^ {2}}{\beta} + W (\alpha + \sqrt {\beta d}) \sum_ {k = 1} ^ {T / W} \sum_ {a \in \mathcal {A} _ {0}} p _ {k} ^ {\mathcal {A} _ {0}} (a) \| a \| _ {\hat {\Sigma} _ {k} ^ {- 1}} + W (\sqrt {\beta d} - \alpha) \sum_ {k = 1} ^ {T / W} \| u ^ {\mathcal {A} _ {0}} \| _ {\hat {\Sigma} _ {k} ^ {- 1}} \right].
$$
Choosing  $\alpha \approx \sqrt{\beta d}$ , we further bound it by
$$
\begin{array}{l} \mathbb {E} \left[ \frac {W \ln | \mathcal {A} _ {0} |}{\eta} + \eta d T + W \sqrt {\beta d} \sum_ {k = 1} ^ {T / W} \sum_ {a \in \mathcal {A} _ {0}} p _ {k} ^ {\mathcal {A} _ {0}} (a) \| a \| _ {\hat {\Sigma} _ {k} ^ {- 1}} \right] \\ \leq \mathbb {E} \left[ \frac {W \ln | \mathcal {A} _ {0} |}{\eta} + \eta d T + W \sqrt {\beta d} \sum_ {k = 1} ^ {T / W} \sqrt {\sum_ {a \in \mathcal {A} _ {0}} p _ {k} ^ {\mathcal {A} _ {0}} (a) \| a \| _ {\hat {\Sigma} _ {k} ^ {- 1}} ^ {2}} \right] \\ \leq \frac {W \ln | \mathcal {A} _ {0} |}{\eta} + \eta d T + \sqrt {\beta} d T. \\ \end{array}
$$
Recall the constraint  $W \geq \frac{d}{\beta^2}$ . Choosing  $W = \frac{d}{\beta^2}$  gives
$$
\frac {d \ln | \mathcal {A} _ {0} |}{\eta \beta^ {2}} + \eta d T + \sqrt {\beta} d T \tag {48}
$$
which gives  $d\big(\ln |\mathcal{A}_0|\big)^{\frac{1}{6}}T^{\frac{5}{6}}$  with the optimally chosen  $\eta$  and  $\beta$ .
Remark Due to the restrictions on the magnitude of the loss estimator required by the exponential weight algorithm, there is actually another constraint  $\frac{\eta}{\beta} \leq 1$ , which makes Eq. (48) be  $d(\ln |\mathcal{A}_0|)^{\frac{1}{7}}T^{\frac{6}{7}}$  at best. This is exactly the bound obtained by Sherman et al. [2023]. A more sophisticated way to construct  $\hat{y}_k$  developed by Dai et al. [2023] removes this additional requirement and allows a bound of  $d(\ln |\mathcal{A}_0|)^{\frac{1}{6}}T^{\frac{5}{6}}$ . The sub-optimal bound  $T^{\frac{8}{9}}$  reported in Dai et al. [2023] is due to issues related to MDPs, which are not presented in the contextual bandit case here.