SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since  $\Phi(y)$  is the CDF of the standard normal distribution,  $y_1 \in (0, 2\mathfrak{R} / \sigma)$  and  $y_2 \in -2\mathfrak{R} / \sigma, 0)$ , we have that  $L_1$  is a constant well defined by the above expression. Also observe that
$$
\begin{array}{l} \mathbb {E} \left[ S _ {j} - S _ {j - 1} \mid S _ {j - 1}, \dots , S _ {1} \right] = \mathbb {E} \left[ u _ {t} (\alpha) \mid S _ {j - 1}, \dots , S _ {1} \right] \\ = \mathbb {P} \{R _ {t} = \mathfrak {R} \} \cdot \frac {- \phi (\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma})}{\left[ 1 - \Phi (\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma}) \right]} + \mathbb {P} \{R _ {t} = - \mathfrak {R} \} \cdot \frac {\phi (\frac {- \mathfrak {R} - \alpha R _ {t - 1}}{\sigma})}{\left[ \Phi (\frac {- \mathfrak {R} - \alpha R _ {t - 1}}{\sigma}) \right]} \\ + \mathbb {P} \{- \mathfrak {R} <   R _ {t} <   \mathfrak {R} \} \cdot \frac {\phi \left(\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) - \phi \left(\frac {- \mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right)}{\left[ \Phi \left(\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) - \Phi \left(\frac {- \mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) \right]} \\ = - \phi (\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma}) + \phi (\frac {- \mathfrak {R} - \alpha R _ {t - 1}}{\sigma} + \phi (\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma}) - \phi (\frac {- \mathfrak {R} - \alpha R _ {t - 1}}{\sigma}) \\ = 0. \\ \end{array}
$$
where  $\Phi(y)$  and  $\phi(y)$  are the CDF and PDF of the standard normal distribution. The third equality above follows from  $\epsilon_t$  being independent from  $S_1, \ldots, S_{j-1}$ . Hence,  $S_j$  is a martingale with bounded difference.
By Azuma-Hoeffding Inequality (see Lemma J.5), we thus have that for  $\gamma > 0$ , the following concentration result holds:
$$
\mathbb {P} \left[ \left| S _ {T _ {\text {e s t}}} \right| > L _ {1} \sqrt {2 \beta T _ {\text {e s t}} \log T _ {\text {e s t}}} \right] \leq \frac {2}{T _ {\text {e s t}} ^ {\gamma}}. \tag {63}
$$
Let us denote the high probability event, i.e.,  $|S_{T_{\mathrm{est}}}|\leq L_1\sqrt{2\gamma T_{\mathrm{est}}\log T_{\mathrm{est}}}$ , as  $\mathcal{E}_1$ . Note that conditioning on  $\mathcal{E}_1$ , we would have
$$
\mathcal {L} ^ {\prime} (\alpha) = \frac {1}{T _ {\text {e s t}}} \sum_ {t = 1} ^ {T _ {\text {e s t}}} u _ {t} (\alpha) \cdot \frac {R _ {t - 1}}{\sigma} \leq \left(\frac {L _ {1} \mathfrak {X}}{\sigma}\right) \cdot \sqrt {\frac {2 \gamma \log T _ {\text {e s t}}}{T _ {\text {e s t}}}}. \tag {64}
$$
We also bound the second derivative using a constant  $L_{2} \geq 0$  as follows:
$$
\begin{array}{l} \mathcal {L} ^ {\prime \prime} (\alpha) = \frac {1}{T _ {\mathrm {e s t}}} \sum_ {t = 1} ^ {T _ {\mathrm {e s t}}} \mathbb {1} \{R _ {t} = \mathfrak {R} \} \cdot \log^ {\prime \prime} \left[ 1 - \Phi \left(\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma}\right) \right] \cdot \left(\frac {R _ {t - 1} ^ {2}}{\sigma^ {2}}\right) \\ + \mathbb {1} \{R _ {t} = - \mathfrak {R} \} \cdot \log^ {\prime \prime} \left[ \Phi (\frac {- \mathfrak {R} - \alpha R _ {t - 1}}{\sigma}) \right] \cdot \left(\frac {R _ {t - 1} ^ {2}}{\sigma^ {2}}\right) \\ + \mathbb {1} \left\{- \mathfrak {R} <   R _ {t} <   \mathfrak {R} \right\} \cdot \log^ {\prime \prime} \left[ \Phi (\frac {\mathfrak {R} - \alpha R _ {t - 1}}{\sigma}) - \Phi (\frac {- \mathfrak {R} - \alpha R _ {t - 1}}{\sigma}) \right] \cdot \left(\frac {R _ {t - 1} ^ {2}}{\sigma^ {2}}\right) \\ \geq \frac{1}{T_{\mathrm{est}}}\sum_{t = 1}^{T_{\mathrm{est}}}\Big(\frac{R_{t - 1}^{2}}{\sigma^{2}}\Big)\cdot \inf_{\substack{y_{1}\in (0,2}\mathfrak{R}_{/ \sigma}),\\ y_{2}\in -2\mathfrak{R}_{/ \sigma ,0)}}\Big\{\min \Big\{-\log^{\prime \prime}[1 - \Phi (y_{1})], - \log^{\prime \prime}\Phi (y_{2}), - \log^{\prime \prime}(\Phi (y_{1}) - \Phi (y_{2}))\Big\} \Big\} \\ \triangleq \frac {1}{T _ {\text {e s t}}} \sum_ {t = 1} ^ {T _ {\text {e s t}}} \left(\frac {R _ {t - 1} ^ {2}}{\sigma^ {2}}\right) \cdot L _ {2} \tag {65} \\ \end{array}
$$
Here, we define the constant  $L_{2}$  as
$$
L _ {2} = \inf  _ {\substack {y _ {1} \in (0, 2) \mathfrak {R} / \sigma), \\ y _ {2} \in - 2) \mathfrak {R} / \sigma , 0)}} \left\{\min  \left\{- \log^ {\prime \prime} [ 1 - \Phi (y _ {1}) ], - \log^ {\prime \prime} \Phi (y _ {2}), - \log^ {\prime \prime} (\Phi (y _ {1}) - \Phi (y _ {2})) \right\} \right\}. \tag{66}
$$
Similar to our arguments above, since  $\Phi(y)$  is the CDF of the standard normal distribution,  $y_1 \in (0, 2\Re/\sigma)$  and  $y_2 \in -2\Re/\sigma, 0$ , we have that  $L_2 \geq 0$  is a constant well defined by the last equality. Let  $V = \mathbb{E}[R_t^2] > 0$  denote the variance of reward under the steady state distribution. Since  $R_t^2$  is bounded, we can apply Hoeffding's Inequality (see Lemma J.4) that states
$$
\mathbb {P} \left[ \left| \sum_ {t = 1} ^ {T _ {\mathrm {e s t}}} R _ {t - 1} ^ {2} - T _ {\mathrm {e s t}} V \right| \geq \frac {T _ {\mathrm {e s t}} V}{2} \right] \leq 2 \exp \left(- \frac {T _ {\mathrm {e s t}} V ^ {2}}{2 \mathfrak {R} ^ {2}}\right).
$$
Let  $\mathcal{E}_2$  denote the high probability event that  $|\sum_{t=1}^{T_{\mathrm{est}}} R_{t-1}^2 - T_{\mathrm{est}}V| \leq \frac{T_{\mathrm{est}}V}{2}$ . Then, under this event, we have that  $\sum_{t=1}^{T_{\mathrm{est}}} R_{t-1}^2 \geq \frac{T_{\mathrm{est}}V}{2}$ . Hence, conditioning on the event  $\mathcal{E}_2$ , we have
$$
\mathcal {L} ^ {\prime \prime} (\alpha) \geq \frac {V L _ {2}}{2 \sigma^ {2}}. \tag {67}
$$
After bounding the first and second derivatives of  $\mathcal{L}(\alpha)$  respectively, we are now ready to show the proximity between our estimated parameter  $\widehat{\alpha}$  and the true parameter  $\alpha$ , conditioning on the high-probability events  $\mathcal{E}_1 \cap \mathcal{E}_2$ . By optimality of  $\widehat{\alpha}$ , we have that  $\mathcal{L}(\widehat{\alpha}) \leq \mathcal{L}(\alpha)$ . This, together with (61), gives
$$
\frac {1}{2} \mathcal {L} ^ {\prime \prime} (\tilde {\alpha}) (\alpha - \widehat {\alpha}) ^ {2} \leq - \mathcal {L} ^ {\prime} (\alpha) (\widehat {\alpha} - \alpha) \leq | \mathcal {L} ^ {\prime} (\alpha) | \cdot | \widehat {\alpha} - \alpha |. \tag {68}
$$
Using (67), we can bound the left-hand side as
$$
\frac {1}{2} \mathcal {L} ^ {\prime \prime} (\tilde {\alpha}) (\alpha - \widehat {\alpha}) ^ {2} \geq \frac {1}{2} \cdot \left(\frac {V L _ {2}}{2 \sigma^ {2}}\right) \cdot (\alpha - \widehat {\alpha}) ^ {2}. \tag {69}
$$
Conditioning on  $\mathcal{E}_1$ , we also know from (64) have that
$$
\left| \mathcal {L} ^ {\prime} (\alpha) \right| \cdot \left| \widehat {\alpha} - \alpha \right| \leq \left(\frac {L _ {1} \mathfrak {R}}{\sigma}\right) \cdot \sqrt {\frac {2 \gamma \log T _ {\mathrm {e s t}}}{T _ {\mathrm {e s t}}}} \cdot \left| \widehat {\alpha} - \alpha \right| \tag {70}
$$
Finally, combining (68), (69) and (70), we have that with probability at least  $1 - 2 / T_{\mathrm{est}}^{\gamma} - 2 \exp(-T_{\mathrm{est}} V^{2} / 2 \mathfrak{R}^{2})$ , we have
$$
\frac {1}{2} \cdot \left(\frac {V L _ {2}}{2 \sigma^ {2}}\right) \cdot (\alpha - \widehat {\alpha}) ^ {2} \leq \left(\frac {L _ {1} \mathcal {Q}}{\sigma}\right) \cdot \sqrt {\frac {2 \gamma \log T _ {\mathrm {e s t}}}{T _ {\mathrm {e s t}}}} \cdot | \widehat {\alpha} - \alpha |
$$
$$
\left| \widehat {\alpha} - \alpha \right| \leq \frac {4 \sigma \mathcal {R} L _ {1}}{V L _ {2}} \sqrt {\frac {2 \gamma \log T _ {\mathrm {e s t}}}{T _ {\mathrm {e s t}}}}.
$$
J Concentration Inequalities
In this section, we state some useful concentration inequalities for subgaussian variables (see [29, 8, 45] for references).
Definition J.1 (Subgaussian variables). We say that a zero-mean random variable  $X$  is  $\sigma$ -subgaussian (or,  $X \sim \text{sub} G(\sigma)$ ), if for all  $\lambda \in \mathbb{R}$ ,  $\mathbb{E}[e^{\lambda X}] \leq e^{\frac{\lambda^2\sigma^2}{2}}$ .
In particular, note that if  $X \sim N(0, \sigma)$ , then  $X \sim \mathrm{subG}(\sigma)$ . Subgaussian variables satisfy the following concentration inequalities:
Lemma J.2 (Concentration of subgaussian variables). A zero-mean random variable  $X$  is  $\sigma$ -subgaussian if and only if for any  $s > 0$ ,  $\mathbb{P}[|X| \geq s] \leq 2\exp \left(-\frac{s^2}{2\sigma^2}\right)$ .
Lemma J.3 (Hoeffding's Inequality). Let  $X_{1},\ldots ,X_{n}$  be independent zero-mean random variables such that  $X_{j}\sim subG(\sigma_{j})$ , then for any  $s > 0$ ,  $\mathbb{P}\left[\left|\frac{1}{n}\sum_{j = 1}^{n}X_{j}\right|\geq s\right]\leq 2\exp (-\frac{n^2s^2}{2\sum_{j = 1}^{n}\sigma_j^2})$
In particular, bounded variables are known to be subgaussian, and we thus have a special case of the Hoeffding's Inequality that applies to bounded variables:
Lemma J.4 (Hoeffding's Inequality for Bounded Variables). Let  $X_{1}, \ldots, X_{n}$  be independent random variables such that  $a_{i} \leq X_{i} \leq b_{i}$ . Consider the sum of these random variables  $S_{n} = \sum_{i=1}^{n} X_{i}$ . We have that for  $s > 0$ ,  $\mathbb{P}\left[|S_{n} - \mathbb{E}[S_{n}]| \geq s\right] \leq 2\exp\left(-\frac{2s^{2}}{\sum_{i=1}^{n}(b_{i} - a_{i})^{2}}\right)$ .
Another related concentration result is the Azuma-Hoeffding's Inequality stated below, which applies to martingales with bounded differences.
Lemma J.5 (Azuma-Hoeffding's Inequality). Suppose that  $\{S_{j}: j = 0,1,2,\ldots\}$  is a martingale and has bounded difference  $|S_{k} - S_{k-1}| \leq b_{k}$  almost surely. Then, the following concentration bound holds for any  $N \in \mathbb{Z}^{+}$  and  $s > 0$ :  $\mathrm{P}\left(|S_N - S_0| \geq s\right) \leq 2 \exp \left(\frac{-s^2}{2\sum_{k=1}^{N} b_k^2}\right)$ .