SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The lemma presented below introduces a property that will be instrumental in analyzing the variance of loss estimates, presented as Lemma 6 in Ito et al. (2020):
Lemma B.4. If  $y$  follows a log-concave distribution over  $\mathbb{R}$ , and if  $\mathbb{E}[y^2] \leq 1/100$ , we have
$$
\mathbb {E} \left[ \psi (y) \right] \leq \mathbb {E} \left[ y ^ {2} \right] + 3 0 \exp \left(- \frac {1}{\sqrt {\mathbb {E} \left[ y ^ {2} \right]}}\right) \leq 2 \mathbb {E} \left[ y ^ {2} \right], w h e r e \psi (x) = \exp (x) - x - 1.
$$
C Proof of Theorem 3.1
The proof of Theorem 3.1 proceeds in a sequence of lemmas. First, we need to show that the distribution of  $Z_{t}(X_{t})$  is log-concave for all  $t\in [T]$ , and after we follow the analysis of Algorithm 1 of Ito et al. (2020), bounding both components of (19) taking into account the required alterations to incorporate contextual structure
Lemma C.1. Suppose  $z(q,x) = \sum_{a}q_{a}\varphi (x,a)$  for  $\varphi (x,a) = (\bar{0}^{\intercal},\dots ,x^{\intercal},\dots)$  such that  $x$  is on the da'th co-ordinate and  $Q(x)\sim p(\cdot |x)$  for  $p(\cdot |x)$  log-concave. If  $X\sim p_X(\cdot)$  and  $p_X(\cdot)$  is log-concave and  $Z(x) = z(Q(x),x)$ , then  $Z(X)$  also follows a log-concave distribution.
Proof. Assume that  $|x^i| > 0$  for all  $i \in [d]$ . Set  $(z_1, \ldots, z_{K-1}, x) = h(q_1\bar{1}, \ldots, q_{K-1}\bar{1}, x)$ , where  $h: \mathbb{R}^{dK} \to \mathbb{R}^{dK}$  and  $z_i = (\ldots, z_i^j, \ldots)^\top$  for each  $i \in \{1, \ldots, K-1\}$ . Thus  $z_i^j = h_i(q_i, x^j) = q_i(x^j)$  and  $h_K(x) = x$ . The Jacobian of  $h^{-1}(z_1, \ldots, z_{K-1}, x)$  can be expressed as the block matrix
$$
J (h ^ {- 1} (z _ {1}, \ldots , z _ {K - 1}, x)) = \left[ \begin{array}{c c} \Lambda_ {x} & \Gamma_ {z, x} \\ 0 _ {d \times (K - 1)} & \mathrm {I d} _ {d \times d} \end{array} \right],
$$
where  $\Lambda_{x}\in \mathbb{R}^{(K - 1)\times (K - 1)}$  is diagonal with  $(\Lambda_{x})_{ii} = \frac{1}{x^{i}}$  and  $\Gamma_{z,x}\in \mathbb{R}^{(K - 1)\times d}$  with  $(\Gamma_{z,x})_{ij} = -\frac{z_i^j}{(x^j)^2}$ . Since  $J(h^{-1}(z_1,\ldots ,z_{K - 1},x))$  is upper-triangular,  $\operatorname*{det}(J(h^{-1}(z_1,\dots,z_{K - 1},x))) = \left(\prod_{i = 1}^{d}\frac{1}{x^i}\right)^{K - 1}$ . The joint distribution of  $Z_{i}$  and  $X$  can thus be written
$$
p _ {Z _ {1}, \dots , Z _ {K - 1}, X} (z _ {1}, \dots , z _ {K - 1}, x) = p _ {Q, X} \left(h ^ {- 1} (z _ {1}, \dots , z _ {K - 1}, x)\right) \left(\prod_ {i = 1} ^ {d} \frac {1}{x ^ {i}}\right) ^ {K - 1}
$$
with the joint distribution between  $Q$  and  $X$  of the form
$$
p _ {Q, X} \left(h ^ {- 1} \left(z _ {1}, \dots , z _ {K - 1}, x\right)\right) = \frac {e ^ {- \eta \left\langle \psi (z , x , \varphi) , \widehat {\Theta} _ {t - 1} \right\rangle}}{\int_ {q ^ {\prime} \in C} e ^ {- \eta \left\langle \sum_ {a = 1} ^ {K - 1} q _ {a} ^ {\prime} \varphi (x , a) , \widehat {\Theta} _ {t - 1} \right\rangle} d q ^ {\prime}} p _ {X} (x)
$$
where  $(\psi (z,x,\varphi))_i = \sum_{a = 1}^{K - 1}\frac{z_a^i}{x^i}\varphi (x,a)^i + \left(1 - \sum_{a = 1}^{K - 1}\frac{z_a^i}{x^i}\right)\varphi (x,K)^i$  has been defined for readability. We can reabsorb the factor  $\left(\prod_{i = 1}^{d}\frac{1}{x^i}\right)^{K - 1}$  in the denominator to rewrite the normalization constant as a in terms of the random variable  $Z(x)$ , and so
$$
p _ {Z _ {1}, \dots , Z _ {K - 1}, X} (z _ {1}, \dots , z _ {K - 1}, x) = \frac {e ^ {- \eta \left\langle \psi (z , x , \varphi) , \widehat {\Theta} _ {t - 1} \right\rangle}}{\int_ {z ^ {\prime} \in Z (x)} e ^ {- \eta \left\langle \psi (z ^ {\prime} , x , \varphi) , \widehat {\Theta} _ {t - 1} \right\rangle} d z ^ {\prime}} p _ {X} (x).
$$
Define a new function  $g: \mathbb{R}^{d \times K} \to \mathbb{R}^{d \times K}$  such that  $y = g(z_1, \ldots, z_{K-1}, x) = (\ldots, g_i(z_1, \ldots, z_{K-1}, x), \ldots)^{\top}$ , where for  $i \in [1, K-1]$ ,  $g_i(z_1, \ldots, z_{K-1}, x) = z_i$  and
$g_{K}(z_{1},\ldots ,z_{K - 1},x) = (\ldots ,\left(1 - \frac{1}{x^{i}}\sum_{a = 1}^{K - 1}z_{a}^{i}\right)x^{i},\ldots)$ . Then for  $i\in \{1,\dots ,K - 1\}$ ,  $g_{i}^{-1}(y) = y_{i}$  and  $g_{K}^{-1}(y) = \sum_{a = 1}^{K}y_{a}$ . The determinant  $\operatorname*{det}(J(g^{-1}(y))) = 1$ , so
$$
\begin{array}{l} p _ {Y} (y) = p _ {Z _ {1},.., Z _ {K - 1}, X} \left(g ^ {- 1} (y)\right) \\ = \frac {e ^ {- \eta \left\langle y , \widehat {\Theta} _ {t - 1} \right\rangle}}{\int_ {y ^ {\prime} \in Y (y)} e ^ {- \eta \left\langle y ^ {\prime} , \widehat {\Theta} _ {t - 1} \right\rangle} d y ^ {\prime}} p _ {X} \left(\sum_ {a = 1} ^ {K} y _ {a}\right). \\ \end{array}
$$
Since both  $p_X$  and  $\frac{e^{-\eta\langle y, \widehat{\Theta}_{t-1} \rangle}}{\int_{y' \in Y(y)} e^{-\eta\langle y', \widehat{\Theta}_{t-1} \rangle dy'}}$  are both log-concave, the lemma follows.
Having shown the log-concavity of  $Z(X_{t})$ , we may safely proceed.
We state the analog of Lemma 4 in Ito et al. (2020) adapted to our setting, leading to a bound on the first term of (19) as well as providing a useful relation between  $\Sigma_{t}$  and  $\widetilde{\Sigma}_{t}$ .
Lemma C.2.
$$
\left| \mathbb {E} _ {t} \left[ \left\langle Z _ {t} \left(X _ {t}\right) - \widetilde {Z} _ {t} \left(X _ {t}\right), \theta_ {t} \right\rangle \right] \right| \leq \frac {1}{2 T ^ {2}}, \tag {28}
$$
$$
\frac {3}{4} \Sigma_ {t} \preceq \widetilde {\Sigma} _ {t} \preceq \frac {4}{3} \Sigma_ {t}. \tag {29}
$$
Proof. From definition of  $\tilde{p}_t$ , for any  $x \in \mathcal{X}, \theta \in \Theta$ , we have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left\langle \widetilde {Z} _ {t} (X _ {t}), \theta \right\rangle \right] \\ = \frac {1}{\mathbb {P} _ {t} \left[ \| Z _ {t} (X _ {t}) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} \leq d K \gamma^ {2} \right]} \int_ {\Delta^ {K}} \int_ {\mathcal {X}} \langle z (q, x), \theta \rangle \mathbb {1} \left\{\| z (q, x) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} \leq d K \gamma^ {2} \right\} p _ {t} (q | x) p (x) d x d q \\ = \frac {1}{1 - \delta} \int_ {\Delta^ {K}} \int_ {\mathcal {X}} \langle z (q, x), \theta \rangle \mathbb {1} \left\{\| z (q, x) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} \leq d K \gamma^ {2} \right\} p _ {t} (q | x) p (x) d x d q \\ = \frac {1}{1 - \delta} \left(\mathbb {E} _ {t} \left[ \langle Z _ {t} (X _ {t}), \theta \rangle \right] - \int_ {\Delta^ {K}} \int_ {\mathcal {X}} \langle z (q, x), \theta \rangle \mathbb {1} \left\{\| z (q, x) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} > d K \gamma^ {2} \right\} p _ {t} (q | x) p (x) d x d q\right), \\ \end{array}
$$
where  $\delta = \mathbb{P}_t\left[\| Z_t(X_t)\|_{\Sigma_t^{-1}}^2 >dK\gamma^2\right]$ . Plugging this into the l.h.s. of (28) yields
$$
\begin{array}{l} \left| \mathbb {E} _ {t} \left[ \left\langle Z _ {t} \left(X _ {t}\right) - \widetilde {Z} _ {t} \left(X _ {t}\right), \theta_ {t} \right\rangle \right] \right| \\ = \frac {1}{1 - \delta} \left| \delta \mathbb {E} _ {t} \left[ \left\langle Z _ {t} \left(X _ {t}\right), \theta_ {t} \right\rangle \right] + \int_ {\Delta^ {K}} \int_ {\mathcal {X}} \left\langle z (q, x), \theta \right\rangle 1 1 \left\{\left\| z (q, x) \right\| _ {\Sigma_ {t} ^ {- 1}} ^ {2} > d K \gamma^ {2} \right\} p _ {t} (q | x) p (x) d x d q \right| \\ \leq \frac {1}{1 - \delta} \left(\delta + \int_ {\Delta^ {K}} \int_ {\mathcal {X}} \mathbb {1} \left\{\| z (q, x) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} > d K \gamma^ {2} \right\} p _ {t} (q | x) p (x) d x d q\right) = \frac {2 \delta}{1 - \delta}. \\ \end{array}
$$
Since the distribution of  $Z_{t}(X_{t})$  is log-concave (Lemma C.1), we can apply Lemma 1 of Ito et al. (2020) to  $x = \Sigma_{t}^{-1 / 2}Z_{t}(X_{t})$ . The assumptions of Lemma 1 of Ito et al. (2020) hold since we have  $\mathbb{E}[xx^{\intercal}] = I$  and since log-concavity is preserved under linear maps. Using Lemma 1 of Ito et al. (2020), we have
$$
\delta = \mathbb {P} _ {t} \left[ \| Z _ {t} (X _ {t}) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} > d K \gamma^ {2} \right] \leq d K \exp (1 - \gamma) \leq 3 d K \exp (- \gamma) \leq \frac {1}{6 T ^ {2}},
$$
where the last inequality follows from  $\gamma \geq 4\log (10dKT)$ , which obtains (28). We proceed to showing (29). For any  $y \in \mathbb{R}^{dK}$ , we have
$$
\begin{array}{l} y ^ {\top} \widetilde {\Sigma} _ {t} y = \mathbb {E} \left[ \left(y ^ {\top} \widetilde {Z} _ {t} (X _ {t})\right) ^ {2} \right] = \frac {1}{1 - \delta} \mathbb {E} _ {t} \left[ \left(y ^ {\top} Z _ {t} (X _ {t})\right) ^ {2} \mathbb {1} \left\{\| Z _ {t} (X _ {t}) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} \leq d K \gamma^ {2} \right\} \right] \\ \leq \frac {1}{1 - \delta} \mathbb {E} _ {t} \left[ (y ^ {\intercal} Z _ {t} (X _ {t})) ^ {2} \right] = \frac {1}{1 - \delta} y ^ {\intercal} \Sigma_ {t} y. \\ \end{array}
$$
Since this holds for all  $y \in \mathbb{R}^{dK}$  and  $\frac{1}{1 - \delta} \leq \frac{4}{3}$ , the second inequality in (29) holds. Furthermore, we have
$$
\begin{array}{l} y ^ {\intercal} \Sigma_ {t} y - y ^ {\intercal} \widetilde {\Sigma} _ {t} y = \mathbb {E} _ {t} \left[ (y ^ {\intercal} Z _ {t} (X _ {t})) ^ {2} \right] - \frac {1}{1 - \delta} \mathbb {E} _ {t} \left[ (y ^ {\intercal} Z _ {t} (X _ {t})) ^ {2} \mathbb {1} \left\{\| Z _ {t} (X _ {t}) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} \leq d K \gamma^ {2} \right\} \right] \\ \leq \mathbb {E} _ {t} \left[ \left(y ^ {\top} Z _ {t} (X _ {t})\right) ^ {2} \mathbb {1} \left\{\| Z _ {t} (X _ {t}) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} > d K \gamma^ {2} \right\} \right] \\ \leq y ^ {\intercal} \Sigma_ {t} y \mathbb {E} _ {t} \left[ \| Z _ {t} (X _ {t}) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} \mathbb {1} \left\{\| Z _ {t} (X _ {t}) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} > d K \gamma^ {2} \right\} \right], \tag {30} \\ \end{array}
$$
where the last inequality follows from Cauchy-Schwarz:
$$
\left(y ^ {\intercal} Z _ {t} \left(X _ {t}\right)\right) ^ {2} = \left(\left\langle \Sigma_ {t} ^ {1 / 2} y, \Sigma_ {t} ^ {- 1 / 2} x \right\rangle\right) ^ {2} \leq \left\| \Sigma_ {t} ^ {1 / 2} y \right\| _ {2} ^ {2} \cdot \left\| \Sigma_ {t} ^ {- 1 / 2} x \right\| _ {2} ^ {2} = y ^ {\intercal} \Sigma_ {t} y \| x \| _ {\Sigma_ {t} ^ {- 1}} ^ {2}.
$$
The right-hand side of (30) can be bounded using Lemma B.3 as follows:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \| Z _ {t} (X _ {t}) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} \mathbb {1} \left\{\| Z _ {t} (X _ {t}) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} > d K \gamma^ {2} \right\} \right] \\ \leq \sum_ {n = 1} ^ {\infty} (n + 1) ^ {2} d K \gamma^ {2} \mathbb {P} _ {t} \left[ n ^ {2} d K \gamma^ {2} \leq \| Z _ {t} (X _ {t}) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} \leq (n + 1) ^ {2} d K \gamma^ {2} \right] \\ \leq \sum_ {n = 1} ^ {\infty} (n + 1) ^ {2} (d K) ^ {2} \gamma^ {2} \exp (1 - n \gamma) \\ \leq (d K) ^ {2} \gamma^ {2} \sum_ {n = 1} ^ {\infty} \exp (2 + n - n \gamma) = (d K) ^ {2} \gamma^ {2} \frac {\exp (3 - \gamma)}{1 - \exp (1 - \gamma)} \leq \frac {1}{4}. \tag {31} \\ \end{array}
$$
Combining (31) and (30) we get the first inequality of (29).
Lemma C.3. Let  $\pi^{*}$  be any fixed stochastic policy and let  $X_0\sim \mathcal{D}$  be a sample from the context distribution independent from  $\mathcal{F}_T$ . Suppose that  $p_t\in \mathcal{F}_{t - 1}$ , such that  $p_t(\cdot |x)$  is a probability density with respect to Lebesgue measure with support  $\Delta^K$  and let  $Q_{t}(x)\sim p_{t}(\cdot |x)$ . Then,
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle z \left(Q _ {t} \left(X _ {t}\right), X _ {t}\right) - z \left(\pi^ {*} \left(X _ {t}\right), X _ {t}\right), \theta_ {t} \right\rangle \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle z \left(Q _ {t} \left(X _ {0}\right), X _ {0}\right) - z \left(\pi^ {*} \left(X _ {0}\right)\right), X _ {0}, \widehat {\theta} _ {t} \right\rangle \right].
$$
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left\langle Z _ {t} \left(X _ {0}\right) - Z ^ {*} \left(X _ {0}\right), \widehat {\theta} _ {t} \right\rangle \right] = \mathbb {E} _ {t} \left[ \mathbb {E} _ {t} \left[ \left\langle Z _ {t} \left(X _ {0}\right) - Z ^ {*} \left(X _ {0}\right), \widehat {\theta} _ {t} \right\rangle \mid X _ {0} \right] \right] \\ = \mathbb {E} _ {t} \left[ \mathbb {E} _ {t} \left[ \langle Z _ {t} (X _ {0}) - Z ^ {*} (X _ {0}), \theta_ {t} \rangle | X _ {0} \right] \right] = \mathbb {E} _ {t} \left[ \langle Z _ {t} (X _ {t}) - Z ^ {*} (X _ {t}), \theta_ {t} \rangle \right]. \\ \end{array}
$$
Then, we prove the almost sure regret bound for any  $x$  and then take an expectation over  $X_0$ . We further proceed with an adaptation of the analysis of the continuous exponential weights algorithm, which was stated in Ito et al. (2020) as Lemma 16, but we include it here for the clarity. Let  $\psi(y) = \exp(y) - y - 1$ . For any  $x \in \mathcal{X}$ , we show the following:
Lemma C.4. Assume that  $\eta_{t + 1} \leq \eta_t$  for all  $t$ , let  $q_0$  be a uniform distribution over  $[K]$  and  $\psi(y) = \exp(y) - y - 1$ . Then, the regret  $\widehat{R}_T(x)$  for any  $x \in \mathcal{X}$  of CONTEXTEW almost surely satisfies
$$
\widehat {R} _ {T} (x) \leq \frac {1}{T} \sum_ {t = 1} ^ {T} \left\langle z (q _ {0} - \pi^ {*} (x), x), \widehat {\theta} _ {t} \right\rangle + \frac {K \log T}{\eta_ {T}} + \sum_ {t = 1} ^ {T} \frac {1}{\eta_ {t}} \mathbb {E} _ {Q _ {t} (x) \sim p _ {t} (\cdot | x)} \left[ \psi \left(- \eta_ {t} \left\langle z (Q _ {t} (x), x), \widehat {\theta} _ {t} - m _ {t} \right\rangle\right) \right].
$$
Proof. Note that we can write  $\widehat{R}_T(x)$  as
$$
\widehat {R} _ {T} (x) = \sum_ {t = 1} ^ {T} \left(\int_ {\Delta^ {\kappa}} p _ {t} (q | x) \left\langle z (q, x), \widehat {\theta} _ {t} \right\rangle d q - \left\langle z \left(\pi^ {*} (x), x\right), \sum_ {t = 1} ^ {T} \widehat {\theta} _ {t} \right\rangle\right).
$$
Define  $W_{t}(x) = \int_{\Delta^{K}}w_{t}(q|x)dq, u_{t}(q|x) = \exp \left(-\eta_{t}\sum_{a}q_{a}\left\langle x,\sum_{s = 1}^{t}\widehat{\theta}_{s,a}\right\rangle\right), U_{t}(x) =$
$\int_{\Delta^K} u_t(q|x) dq$  and  $\left. v_t(q|x) = \exp \left(-\eta_{t+1} \sum_a q_a \left\langle x, \sum_{s=1}^t \widehat{\theta}_{s,a} \right\rangle \right)\right.$ ,  $V_t(x) = \int_{\Delta^K} v_t(q|x) dq$ . We have
$$
\begin{array}{l} U _ {t} (x) = \int_ {\Delta^ {\kappa}} w _ {t} (q | x) \exp \left(- \eta_ {t} \left\langle z (q, x), \widehat {\theta} _ {t} - m _ {t} \right\rangle\right) d q = W _ {t} (x) \int_ {\Delta^ {\kappa}} p _ {t} (q | x) \exp \left(- \eta_ {t} \left\langle z (q, x), \widehat {\theta} _ {t} - m _ {t} \right\rangle\right) d q \\ = W _ {t} (x) \int_ {\Delta^ {K}} p _ {t} (q | x) \left(1 - \eta_ {t} \left\langle z (q, x), \widehat {\theta} _ {t} - m _ {t} \right\rangle + \psi (- \eta_ {t} \left\langle z (q, x), \widehat {\theta} _ {t} - m _ {t} \right\rangle)\right) d q. \\ \end{array}
$$
Taking the logarithm of both sides, we get
$$
\begin{array}{l} \log (U _ {t} (x)) = \log (W _ {t} (x)) + \log \left(\int_ {\Delta^ {K}} p _ {t} (q | x) \left(1 - \eta_ {t} \left\langle z (q, x), \widehat {\theta} _ {t} - m _ {t} \right\rangle + \psi (- \eta_ {t} \left\langle z (q, x), \widehat {\theta} _ {t} - m _ {t} \right\rangle)\right) d q\right) \\ \leq \log \left(W _ {t} (x)\right) + \int_ {\Delta^ {K}} p _ {t} (q | x) \left(- \eta_ {t} \left\langle z (q, x), \widehat {\theta} _ {t} - m _ {t} \right\rangle + \psi \left(- \eta_ {t} \left\langle z (q, x), \widehat {\theta} _ {t} - m _ {t} \right. \right. \right. \tag {32} \\ \end{array}
$$
where we used the inequality  $\log (1 + x)\leq x$  for  $x > - 1$
$$
\begin{array}{l} V _ {t - 1} (x) = \int_ {\Delta^ {K}} w _ {t} (q | x) \exp \left(\eta_ {t} \sum_ {a} q _ {a} \langle x, m _ {t, a} \rangle\right) d q = W _ {t} (x) \int_ {\Delta^ {K}} p _ {t} (q | x) \exp \left(\eta_ {t} \sum_ {a} q _ {a} \langle x, m _ {t, a} \rangle\right) d q \\ \geq W _ {t} (x) \exp \left(\eta_ {t} \int_ {\Delta^ {\kappa}} p _ {t} (q | x) \sum_ {a} q _ {a} \langle x, m _ {t, a} \rangle d q\right), \tag {33} \\ \end{array}
$$
using Jensen's inequality. It holds that
$$
\int_ {\Delta^ {K}} p _ {t} (q | x) \sum_ {a} q _ {a} \left\langle x, m _ {t, a} \right\rangle d q \leq \frac {1}{\eta_ {t}} \log \frac {V _ {t - 1} (x)}{W _ {t} (x)}.
$$
$$
\sum_ {t = 1} ^ {T} \int_ {\Delta^ {\kappa}} p _ {t} (q | x) \left\langle z (q, x), \widehat {\theta} _ {t} \right\rangle d q \leq \sum_ {t = 1} ^ {T} \frac {1}{\eta_ {t}} \left(\log \frac {V _ {t - 1} (x)}{U _ {t} (x)} + \int_ {\Delta^ {\kappa}} p _ {t} (q | x) \psi (- \eta_ {t} \left\langle z (q, x), \widehat {\theta} _ {t} - m _ {t} \right\rangle) d q\right).
$$
Noting that  $V_{0} = U_{0}$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \frac {1}{\eta_ {t}} \log \frac {V _ {t - 1} (x)}{U _ {t} (x)} = \sum_ {t = 1} ^ {T} \frac {1}{\eta_ {t}} \left(\log \frac {V _ {t - 1} (x)}{V _ {0}} - \log \frac {U _ {t} (x)}{U _ {0}}\right) \\ = \sum_ {t = 1} ^ {T - 1} \left(\frac {1}{\eta_ {t + 1}} \log \frac {V _ {t} (x)}{V _ {0}} - \frac {1}{\eta_ {t}} \log \frac {U _ {t} (x)}{U _ {0}}\right) - \frac {1}{\eta_ {T}} \log \frac {U _ {T} (x)}{U _ {0}} \\ \end{array}
$$
To bound the first term, we use that  $\eta_{t + 1} \leq \eta_t$  and an additional application of Jensen's inequality:
$$
\begin{array}{l} \frac {1}{\eta_ {t + 1}} \log \frac {V _ {t} (x)}{V _ {0}} = \frac {1}{\eta_ {t + 1}} \log \mathbb {E} \left[ \exp \left(- \eta_ {t + 1} \langle \sum_ {s = 1} ^ {t} \widehat {\theta} _ {s}, z (Q _ {t}, x) \rangle\right) \right] \\ = \frac {1}{\eta_ {t + 1}} \log \mathbb {E} \left[ \exp \left(- \eta_ {t} \langle \sum_ {s = 1} ^ {t} \widehat {\theta} _ {s}, z (Q _ {t}, x) \rangle\right) ^ {\frac {\eta_ {t + 1}}{\eta_ {t}}} \right] \\ \leq \frac {1}{\eta_ {t}} \log \mathbb {E} \left[ \exp \left(- \eta_ {t} \langle \sum_ {s = 1} ^ {t} \widehat {\theta} _ {s}, z (Q _ {t}, x) \rangle\right) \right] = \frac {1}{\eta_ {t}} \log \frac {U _ {t} (x)}{U _ {0}}, \\ \end{array}
$$
Set  $Q_{\pi^{*}(x)} \coloneqq \{(1 - \frac{1}{T})\pi^{*}(x) + \frac{1}{T} q|q \in \Delta^{K}\}$ , and denote  $q_0$  as the uniform distribution over  $K$  arms. We then have
$$
U _ {T} (x) \geq \int_ {Q _ {\pi^ {*} (x)}} \exp \left(- \eta_ {T} \left\langle z (q, x), \sum_ {t = 1} ^ {T} \widehat {\theta} _ {t} \right\rangle\right) d q
$$
$$
\begin{array}{l} = T ^ {- K} \int_ {\Delta^ {K}} \exp \left(- \eta_ {T} \left\langle z ((1 - \frac {1}{T}) \pi^ {*} (x) + \frac {1}{T} q, x), \sum_ {t = 1} ^ {T} \widehat {\theta} _ {t} \right\rangle\right) d q \\ \geq T ^ {- K} U _ {0} (x) \exp \left(- \eta_ {T} \left\langle z ((1 - \frac {1}{T}) \pi^ {*} (x) + \frac {1}{T} q _ {0}, x), \sum_ {t = 1} ^ {T} \widehat {\theta} _ {t} \right\rangle\right), \\ \end{array}
$$
where the first inequality constitutes a change of variables and the second follows from Jensen's bound. After rearranging and taking the logarithm, we get
$$
\begin{array}{l} - \frac {1}{\eta_ {T}} \log \frac {U _ {T} (x)}{U _ {0} (x)} \leq \sum_ {t = 1} ^ {T} \left\langle z ((1 - \frac {1}{T}) \pi^ {*} (x) + \frac {1}{T} q _ {0}, x), \widehat {\theta} _ {t} \right\rangle + \frac {K \log T}{\eta_ {T}} \\ = \sum_ {t = 1} ^ {T} \left\langle z \left(\pi^ {*} (x), x\right), \sum_ {t = 1} ^ {T} \widehat {\theta} _ {t} \right\rangle + \frac {1}{T} \sum_ {t = 1} ^ {T} \left\langle z \left(q _ {0} - \pi^ {*} (x), x\right), \widehat {\theta} _ {t} \right\rangle + \frac {K \log T}{\eta_ {T}}. \\ \end{array}
$$
Combining everything together, we get
$$
\begin{array}{l} \sum_{t = 1}^{T}\left(\int_{\Delta^{\kappa}}p_{t}(q|x)\left\langle z(q,x),\widehat{\theta}_{t}\right\rangle dq - \left\langle z(\pi^{*}(x),x),\sum_{t = 1}^{T}\widehat{\theta}_{t}\right\rangle\right)\\ \leq \sum_{t = 1}^{T}\frac{1}{\eta_{t}}\int_{\Delta^{\kappa}}p_{t}(q|x)\psi (-\eta_{t}\left\langle z(q,x),\widehat{\theta}_{t} - m_{t}\right\rangle)dq \\ + \frac {1}{T} \sum_ {t = 1} ^ {T} \left\langle z (q _ {0} - \pi^ {*} (x), x), \widehat {\theta} _ {t} \right\rangle + \frac {K \log T}{\eta_ {T}}. \\ \end{array}
$$
From Lemma 4.3 and Lemma 4.4, we get a bound on the second term of (19):
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle Z _ {t} (X _ {t}) - Z ^ {*} (X _ {t}), \theta_ {t} \rangle \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle Z _ {t} (X _ {0}) - Z ^ {*} (X _ {0}), \widehat {\theta} _ {t} \right\rangle \right] \\ \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \frac {1}{\eta_ {t}} \psi \left(- \eta_ {t} \left\langle Z _ {t} \left(X _ {0}\right), \widehat {\theta} _ {t} - m _ {t} \right\rangle\right) + \frac {1}{T} \sum_ {t = 1} ^ {T} \left\langle z \left(q _ {0} - \pi^ {*} \left(X _ {0}\right), X _ {0}\right), \widehat {\theta} _ {t} \right\rangle + \frac {K \log T}{\eta_ {T}} \right] \tag {34} \\ \end{array}
$$
We first find a bound on the first term using Lemma B.4. To satisfy the assumptions of Lemma B.4,
we need to show that  $\mathbb{E}_t\left[\left(-\eta_t\left\langle Z_t(X_0),\widehat{\theta}_t - m_t\right\rangle\right)^2\right]\leq \frac{1}{100}$
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \left(- \eta_ {t} \left\langle Z _ {t} (X _ {0}), \widehat {\theta} _ {t} - m _ {t} \right\rangle\right) ^ {2} \right] = \mathbb {E} _ {t} \left[ \eta_ {t} ^ {2} \left(\ell_ {t} (A _ {t}, X _ {t}) - X _ {t} ^ {\intercal} m _ {t, A _ {t}}\right) ^ {2} \operatorname {t r} \left(\widetilde {\Sigma} _ {t} ^ {- 1} Z _ {t} (X _ {0}) Z _ {t} (X _ {0}) ^ {\intercal} \widetilde {\Sigma} _ {t} ^ {- 1} Z _ {t} (X _ {t}) Z _ {t} (X _ {t}) ^ {\intercal}\right) \right] \\ = \eta_ {t} ^ {2} \mathbb {E} _ {t} \left[ \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\top} m _ {t, A _ {t}}\right) ^ {2} \operatorname {t r} \left(\widetilde {\Sigma} _ {t} ^ {- 1} \Sigma_ {t} \widetilde {\Sigma} _ {t} ^ {- 1} Z _ {t} \left(X _ {t}\right) Z _ {t} \left(X _ {t}\right) ^ {\top}\right) \right] \\ \leq \eta_ {t} ^ {2} \frac {4}{3} \mathbb {E} _ {t} \left[ \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\top} m _ {t, A _ {t}}\right) ^ {2} \operatorname {t r} \left(\widetilde {\Sigma} _ {t} ^ {- 1} \widetilde {\Sigma} _ {t} \widetilde {\Sigma} _ {t} ^ {- 1} Z _ {t} \left(X _ {t}\right) Z _ {t} \left(X _ {t}\right) ^ {\top}\right) \right] \\ = \eta_ {t} ^ {2} \frac {4}{3} \mathbb {E} _ {t} \left[ \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\top} m _ {t, A _ {t}}\right) ^ {2} Z _ {t} \left(X _ {t}\right) ^ {\top} \widetilde {\Sigma} _ {t} ^ {- 1} Z _ {t} \left(X _ {t}\right) \right] \\ \leq \eta_ {t} ^ {2} \mathbb {E} _ {t} \left[ \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\intercal} m _ {t, A _ {t}}\right) ^ {2} Z _ {t} \left(X _ {t}\right) ^ {\intercal} \Sigma_ {t} ^ {- 1} Z _ {t} \left(X _ {t}\right) \right] \\ \leq d K \eta_ {t} ^ {2} \gamma^ {2} \mathbb {E} _ {t} \left[ \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\top} m _ {t, A _ {t}}\right) ^ {2} \right] (35) \\ \leq \frac {1}{1 0 0}, (36) \\ \end{array}
$$
where the first inequality follows from  $\ell_t \leq 1$  and (29), the second is immediate from (29) and the fact that for symmetric positive definite matrices  $A \succeq B$  follows from  $B^{-1} \succeq A^{-1}$ . The third
inequality follows from the truncation in the algorithm and the last is immediate from plugging in the definition of  $\eta_t$ . So, by applying Lemma B.4 and (35), we get:
$$
\frac {1}{\eta} \mathbb {E} \left[ \psi \left(- \eta \left\langle Z _ {t} \left(X _ {0}\right), \widehat {\theta} _ {t} - m _ {t} \right\rangle\right) \right] \leq \frac {2}{\eta} \mathbb {E} \left[ \left(- \eta \left\langle Z _ {t} \left(X _ {0}\right), \widehat {\theta} _ {t} - m _ {t} \right\rangle\right) ^ {2} \right] \leq 2 d K \eta \gamma^ {2} \mathbb {E} _ {t} \left[ \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\top} m _ {t, A _ {t}}\right) ^ {2} \right]. \tag {37}
$$
For the second term of (34), we simply get from  $-1\leq \ell_t\leq 1$  and  $\widehat{\theta}_t$  is unbiased:
$$
\mathbb {E} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} \left\langle z \left(q _ {0} - \pi^ {*} \left(X _ {0}\right), X _ {0}\right), \widehat {\theta} _ {t} \right\rangle \right] = \mathbb {E} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} \left\langle z \left(q _ {0} - \pi^ {*} \left(X _ {0}\right), X _ {0}\right), \theta_ {t} \right\rangle \right] \leq 2. \tag {38}
$$
The expression that we use for the learning rate is the following:
$$
\eta_ {t} = \left(1 0 0 d K \gamma^ {2} + d \left(\widehat {V} _ {t - 1} + 1 + G _ {t}\right)\right) ^ {- 1 / 2},
$$