SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i} \leq \sqrt {\frac {2 \sigma^ {2} \cdot \log (1 / \delta)}{n}} + \frac {2 \xi}{3 n} \log (1 / \delta).
$$
Lemma E.4 (Empirical Bernstein's Inequality [Maurer and Pontil, 2009]). Let  $x_{1}, \ldots, x_{n}$  be i.i.d random variables such that  $|x_{i}| \leq \xi$  with probability 1. Let  $\bar{x} = \frac{1}{n}\sum_{i=1}^{n}x_{i}$  and  $\widehat{V}_{n} = \frac{1}{n}\sum_{i=1}^{n}(x_{i} - \bar{x})^{2}$ , then with probability  $1 - \delta$  we have
$$
\left| \frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i} - \mathbb {E} [ x ] \right| \leq \sqrt {\frac {2 \widehat {V} _ {n} \cdot \log (2 / \delta)}{n}} + \frac {7 \xi}{3 n} \log (2 / \delta).
$$
Lemma E.5 (Lemma I.8 in [Yin and Wang, 2021b]). Let  $n \geq 2$  and  $V \in \mathbb{R}^S$  be any function with  $\| V \|_{\infty} \leq H$ ,  $P$  be any  $S$ -dimensional distribution and  $\widehat{P}$  be its empirical version using  $n$  samples. Then with probability  $1 - \delta$ ,
$$
\left| \sqrt {\operatorname {V a r} _ {\widehat {P}} (V)} - \sqrt {\frac {n - 1}{n} \operatorname {V a r} _ {P} (V)} \right| \leq 2 H \sqrt {\frac {\log (2 / \delta)}{n - 1}}.
$$
Lemma E.6 (Claim 2 in [Vietri et al., 2020]). Let  $y \in \mathbb{R}$  be any positive real number. Then for all  $x \in \mathbb{R}$  with  $x \geq 2y$ , it holds that  $\frac{1}{x - y} \leq \frac{1}{x} + \frac{2y}{x^2}$ .
E.1 Extended Value Difference
Lemma E.7 (Extended Value Difference (Section B.1 in [Cai et al., 2020])). Let  $\pi = \{\pi_h\}_{h=1}^H$  and  $\pi' = \{\pi_h'\}_{h=1}^H$  be two arbitrary policies and let  $\{\widehat{Q}_h\}_{h=1}^H$  be any given  $Q$ -functions. Then define  $\widehat{V}_h(s) := \langle \widehat{Q}_h(s, \cdot), \pi_h(\cdot \mid s) \rangle$  for all  $s \in S$ . Then for all  $s \in S$ ,
$$
\begin{array}{l} \widehat {V} _ {1} (s) - V _ {1} ^ {\pi^ {\prime}} (s) = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {\prime}} \left[ \langle \widehat {Q} _ {h} (s _ {h}, \cdot), \pi_ {h} (\cdot | s _ {h}) - \pi_ {h} ^ {\prime} (\cdot | s _ {h}) \rangle \mid s _ {1} = s \right] \tag {60} \\ + \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {\prime}} \left[ \widehat {Q} _ {h} \left(s _ {h}, a _ {h}\right) - \left(\mathcal {T} _ {h} \widehat {V} _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right] \\ \end{array}
$$
where  $(\mathcal{T}_hV)(\cdot ,\cdot)\coloneqq r_h(\cdot ,\cdot) + (P_hV)(\cdot ,\cdot)$  for any  $V\in \mathbb{R}^S$
Lemma E.8 (Lemma I.10 in [Yin and Wang, 2021b]). Let  $\widehat{\pi} = \{\widehat{\pi}_h\}_{h=1}^H$  and  $\widehat{Q}_h(\cdot, \cdot)$  be the arbitrary policy and  $Q$ -function and also  $\widehat{V}_h(s) = \langle \widehat{Q}_h(s, \cdot), \widehat{\pi}_h(\cdot|s) \rangle \forall s \in S$ , and  $\xi_h(s, a) = (\mathcal{T}_h \widehat{V}_{h+1})(s, a) - \widehat{Q}_h(s, a)$  element-wise. Then for any arbitrary  $\pi$ , we have
$$
\begin{array}{l} V _ {1} ^ {\pi} (s) - V _ {1} ^ {\widehat {\pi}} (s) = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \xi_ {h} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right] - \sum_ {h = 1} ^ {H} \mathbb {E} _ {\widehat {\pi}} \left[ \xi_ {h} \left(s _ {h}, a _ {h}\right) \mid s _ {1} = s \right] \\ + \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \langle \widehat {Q} _ {h} (s _ {h}, \cdot), \pi_ {h} (\cdot | s _ {h}) - \widehat {\pi} _ {h} (\cdot | s _ {h}) \rangle \mid s _ {1} = s \right] \\ \end{array}
$$
where the expectation are taken over  $s_h, a_h$ .
E.2 Assisting lemmas for linear MDP setting
Lemma E.9 (Hoeffding inequality for self-normalized martingales [Abbasi-Yadkori et al., 2011]). Let  $\{\eta_t\}_{t=1}^{\infty}$  be a real-valued stochastic process. Let  $\{\mathcal{F}_t\}_{t=0}^{\infty}$  be a filtration, such that  $\eta_t$  is  $\mathcal{F}_t$ -measurable. Assume  $\eta_t$  also satisfies  $\eta_t$  given  $\mathcal{F}_{t-1}$  is zero-mean and  $R$ -subgaussian, i.e.
$$
\forall \lambda \in \mathbb {R}, \quad \mathbb {E} \left[ e ^ {\lambda \eta_ {t}} \mid \mathcal {F} _ {t - 1} \right] \leq e ^ {\lambda^ {2} R ^ {2} / 2}.
$$
Let  $\{x_{t}\}_{t = 1}^{\infty}$  be an  $\mathbb{R}^d$ -valued stochastic process where  $x_{t}$  is  $\mathcal{F}_{t - 1}$  measurable and  $\| x_t\| \leq L$ . Let  $\Lambda_{t} = \lambda I_{d} + \sum_{s = 1}^{t}x_{s}x_{s}^{\top}$ . Then for any  $\delta >0$ , with probability  $1 - \delta$ , for all  $t > 0$
$$
\left\| \sum_ {s = 1} ^ {t} x _ {s} \eta_ {s} \right\| _ {\Lambda_ {t} ^ {- 1}} ^ {2} \leq 8 R ^ {2} \cdot \frac {d}{2} \log \left(\frac {\lambda + t L}{\lambda \delta}\right).
$$
Lemma E.10 (Bernstein inequality for self-normalized martingales [Zhou et al., 2021]). Let  $\{\eta_t\}_{t=1}^{\infty}$  be a real-valued stochastic process. Let  $\{\mathcal{F}_t\}_{t=0}^{\infty}$  be a filtration, such that  $\eta_t$  is  $\mathcal{F}_t$ -measurable. Assume  $\eta_t$  also satisfies
$$
\left| \eta_ {t} \right| \leq R, \mathbb {E} \left[ \eta_ {t} \mid \mathcal {F} _ {t - 1} \right] = 0, \mathbb {E} \left[ \eta_ {t} ^ {2} \mid \mathcal {F} _ {t - 1} \right] \leq \sigma^ {2}.
$$
$$
\left\| \sum_ {s = 1} ^ {t} \mathbf {x} _ {s} \eta_ {s} \right\| _ {\boldsymbol {\Lambda} _ {t} ^ {- 1}} \leq 8 \sigma \sqrt {d \log \left(1 + \frac {t L ^ {2}}{\lambda d}\right) \cdot \log \left(\frac {4 t ^ {2}}{\delta}\right)} + 4 R \log \left(\frac {4 t ^ {2}}{\delta}\right)
$$
Lemma E.11 (Lemma H.4 in [Yin et al., 2022]). Let  $\Lambda_{1}$  and  $\Lambda_{2} \in \mathbb{R}^{d \times d}$  be two positive semi-definite matrices. Then:
$$
\left\| \Lambda_ {1} ^ {- 1} \right\| \leq \left\| \Lambda_ {2} ^ {- 1} \right\| + \left\| \Lambda_ {1} ^ {- 1} \right\| \cdot \left\| \Lambda_ {2} ^ {- 1} \right\| \cdot \left\| \Lambda_ {1} - \Lambda_ {2} \right\|
$$
$$
\| \phi \| _ {\Lambda_ {1} ^ {- 1}} \leq \left[ 1 + \sqrt {\| \Lambda_ {2} ^ {- 1} \| \cdot \| \Lambda_ {2} \| \cdot \| \Lambda_ {1} ^ {- 1} \| \cdot \| \Lambda_ {1} - \Lambda_ {2} \|} \right] \cdot \| \phi \| _ {\Lambda_ {2} ^ {- 1}}.
$$
for all  $\phi \in \mathbb{R}^d$
Lemma E.12 (Lemma H.4 in [Min et al., 2021]). Let  $\phi : \mathcal{S} \times \mathcal{A} \to \mathbb{R}^d$  satisfies  $\| \phi(s, a) \| \leq C$  for all  $s, a \in \mathcal{S} \times \mathcal{A}$ . For any  $K > 0, \lambda > 0$ , define  $\bar{G}_K = \sum_{k=1}^{K} \phi(s_k, a_k) \phi(s_k, a_k)^\top + \lambda I_d$  where  $(s_k, a_k)$ 's are i.i.d samples from some distribution  $\nu$ . Then with probability  $1 - \delta$ ,
$$
\left\| \frac {\bar {G} _ {K}}{K} - \mathbb {E} _ {\nu} \left[ \frac {\bar {G} _ {K}}{K} \right] \right\| \leq \frac {4 \sqrt {2} C ^ {2}}{\sqrt {K}} \left(\log \frac {2 d}{\delta}\right) ^ {1 / 2}.
$$
Lemma E.13 (Lemma H.5 in [Min et al., 2021]). Let  $\phi : \mathcal{S} \times \mathcal{A} \to \mathbb{R}^d$  be a bounded function s.t.  $\| \phi \|_2 \leq C$ . Define  $\bar{G}_K = \sum_{k=1}^{K} \phi(s_k, a_k) \phi(s_k, a_k)^\top + \lambda I_d$  where  $(s_k, a_k)$ 's are i.i.d samples from some distribution  $\nu$ . Let  $G = \mathbb{E}_{\nu}[\phi(s, a) \phi(s, a)^\top]$ . Then for any  $\delta \in (0, 1)$ , if  $K$  satisfies
$$
K \geq \max  \left\{5 1 2 C ^ {4} \left\| \mathbf {G} ^ {- 1} \right\| ^ {2} \log \left(\frac {2 d}{\delta}\right), 4 \lambda \left\| \mathbf {G} ^ {- 1} \right\| \right\}.
$$
Then with probability at least  $1 - \delta$ , it holds simultaneously for all  $u \in \mathbb{R}^d$  that
$$
\left\| u \right\| _ {\bar {G} _ {K} ^ {- 1}} \leq \frac {2}{\sqrt {K}} \left\| u \right\| _ {G ^ {- 1}}.
$$
Lemma E.14 (Lemma H.9 in [Yin et al., 2022]). For a linear MDP, for any  $0 \leq V(\cdot) \leq H$ , there exists a  $w_h \in \mathbb{R}^d$  s.t.  $\mathcal{T}_hV = \langle \phi, w_h \rangle$  and  $\| w_h \|_2 \leq 2H\sqrt{d}$  for all  $h \in [H]$ . Here  $\mathcal{T}_h(V)(s, a) = r_h(x, a) + (P_hV)(s, a)$ . Similarly, for any  $\pi$ , there exists  $w_h^\pi \in \mathbb{R}^d$ , such that  $Q_h^\pi = \langle \phi, w_h^\pi \rangle$  with  $\| w_h^\pi \|_2 \leq 2(H - h + 1)\sqrt{d}$ .
E.3 Assisting lemmas for differential privacy
Lemma E.15 (Converting zCDP to DP [Bun and Steinke, 2016]). If  $M$  satisfies  $\rho$ -zCDP then  $M$  satisfies  $(\rho + 2\sqrt{\rho\log(1 / \delta)}, \delta)$ -DP.
Lemma E.16 (zCDP Composition [Bun and Steinke, 2016]). Let  $M: \mathcal{U}^n \to \mathcal{Y}$  and  $M': \mathcal{U}^n \to \mathcal{Z}$  be randomized mechanisms. Suppose that  $M$  satisfies  $\rho$ -zCDP and  $M'$  satisfies  $\rho'$ -zCDP. Define  $M'': \mathcal{U}^n \to \mathcal{Y} \times \mathcal{Z}$  by  $M''(U) = (M(U), M'(U))$ . Then  $M''$  satisfies  $(\rho + \rho')$ -zCDP.
Lemma E.17 (Adaptive composition and Post processing of zCDP [Bun and Steinke, 2016]). Let  $M: \mathcal{X}^n \to \mathcal{Y}$  and  $M': \mathcal{X}^n \times \mathcal{Y} \to \mathcal{Z}$ . Suppose  $M$  satisfies  $\rho$ -zCDP and  $M'$  satisfies  $\rho'$ -zCDP (as a function of its first argument). Define  $M'': \mathcal{X}^n \to \mathcal{Z}$  by  $M''(x) = M'(x, M(x))$ . Then  $M''$  satisfies  $(\rho + \rho')$ -zCDP.
Definition E.18 ( $\ell_1$  sensitivity). Define the  $\ell_1$  sensitivity of a function  $f:\mathbb{N}^{\mathcal{X}}\mapsto \mathbb{R}^{d}$  as
$$
\Delta_{1}(f) = \sup_{\substack{\text{neighboring} U,U^{\prime}}}\| f(U) - f(U^{\prime})\|_{1}.
$$
Definition E.19 (Laplace Mechanism [Dwork et al., 2014]). Given any function  $f: \mathbb{N}^{\mathcal{X}} \mapsto \mathbb{R}^{d}$ , the Laplace mechanism is defined as:
$$
\mathcal {M} _ {L} (x, f, \epsilon) = f (x) + \left(Y _ {1}, \dots , Y _ {d}\right),
$$
where  $Y_{i}$  are i.i.d. random variables drawn from  $\mathrm{Lap}(\Delta_1(f) / \epsilon)$ .
Lemma E.20 (Privacy guarantee of Laplace Mechanism [Dwork et al., 2014]). The Laplace mechanism preserves  $(\epsilon, 0)$ -differential privacy. For simplicity, we say  $\epsilon$ -DP.
F Details for the Evaluation part
In the Evaluation part, we apply a synthetic linear MDP case that is similar to [Min et al., 2021, Yin et al., 2022] but with some modifications for our evaluation task. The linear MDP example we use consists of  $|S| = 2$  states and  $|\mathcal{A}| = 100$  actions, while the feature dimension  $d = 10$ . We denote  $S = \{0,1\}$  and  $\mathcal{A} = \{0,1,\dots ,99\}$  respectively. For each action  $a\in \{0,1,\dots ,99\}$ , we obtain a vector  $\mathbf{a}\in \mathbb{R}^8$  via binary encoding. More specifically, each coordinate of  $\mathbf{a}$  is either 0 or 1.
First, we define the following indicator function  $\delta(s,a) = \begin{cases} 1 & \text{if } \mathbb{1}\{s=0\} = \mathbb{1}\{a=0\} \\ 0 & \text{otherwise} \end{cases}$ , then our non-stationary linear MDP example can be characterized by the following parameters.
The feature map  $\phi$  is:
$$
\boldsymbol {\phi} (s, a) = \left(\mathbf {a} ^ {\top}, \delta (s, a), 1 - \delta (s, a)\right) ^ {\top} \in \mathbb {R} ^ {1 0}.
$$
The unknown measure  $\nu_{h}$  is:
$$
\boldsymbol {\nu} _ {h} (0) = \left(0, \dots , 0, \alpha_ {h, 1}, \alpha_ {h, 2}\right),
$$
$$
\boldsymbol {\nu} _ {h} (1) = \left(0, \dots , 0, 1 - \alpha_ {h, 1}, 1 - \alpha_ {h, 2}\right),
$$
where  $\{\alpha_{h,1},\alpha_{h,2}\}_{h\in [H]}$  is a sequence of random values sampled uniformly from  $[0,1]$ . The unknown vector  $\theta_h$  is:
$$
\theta_ {h} = \left(r _ {h} / 8, 0, r _ {h} / 8, 1 / 2 - r _ {h} / 2, r _ {h} / 8, 0, r _ {h} / 8, 0, r _ {h} / 2, 1 / 2 - r _ {h} / 2\right) \in \mathbb {R} ^ {1 0},
$$
where  $r_h$  is also sampled uniformly from  $[0,1]$ . Therefore, the transition kernel follows  $P_h(s'|s,a) = \langle \phi(s,a), \pmb{\nu}_h(s') \rangle$  and the expected reward function  $r_h(s,a) = \langle \phi(s,a), \theta_h \rangle$ .
Finally, the behavior policy is to always choose action  $a = 0$  with probability  $p$ , and other actions uniformly with probability  $(1 - p) / 99$ . Here we choose  $p = 0.6$ . The initial distribution is a uniform distribution over  $S = \{0,1\}$ .