SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.4. DP-APVI (Algorithm 1) satisfies  $\rho$ -zCDP. Furthermore, under Assumption 2.1, denote  $\bar{d}_m := \min_{h \in [H]} \{d_h^\mu(s_h, a_h) : d_h^\mu(s_h, a_h) > 0\}$ . For any  $0 < \delta < 1$ , there exists constant  $c_1 > 0$  such that when  $n > c_1 \cdot \max \{H^2, E_\rho\} / \bar{d}_m \cdot \iota$  ( $\iota = \log(HSA / \delta)$ ), with probability  $1 - \delta$ , the output policy  $\widehat{\pi}$  of DP-APVI satisfies
$$
0 \leq v ^ {\star} - v ^ {\widehat {\pi}} \leq 4 \sqrt {2} \sum_ {h = 1} ^ {H} \sum_ {(s _ {h}, a _ {h}) \in \mathcal {C} _ {h}} d _ {h} ^ {\pi^ {\star}} (s _ {h}, a _ {h}) \sqrt {\frac {\operatorname {V a r} _ {P _ {h} (\cdot \mid s _ {h} , a _ {h})} (V _ {h + 1} ^ {\star} (\cdot)) \cdot \iota}{n d _ {h} ^ {\mu} (s _ {h} , a _ {h})}} + \widetilde {O} \left(\frac {H ^ {3} + S H ^ {2} E _ {\rho}}{n \cdot \bar {d} _ {m}}\right), \tag {6}
$$
where  $\widetilde{O}$  hides constants and Polylog terms,  $E_{\rho} = 4\sqrt{\frac{H\log\frac{4HS^2A}{\delta}}{\rho}}$
Comparison to non-private counterpart APVI [Yin and Wang, 2021b]. According to Theorem 4.1 in [Yin and Wang, 2021b], the sub-optimality bound of APVI is for large enough  $n$ , with high probability, the output  $\widehat{\pi}$  satisfies:
$$
0 \leq v ^ {\star} - v ^ {\widehat {\pi}} \leq \widetilde {O} \left(\sum_ {h = 1} ^ {H} \sum_ {(s _ {h}, a _ {h}) \in \mathcal {C} _ {h}} d _ {h} ^ {\pi^ {\star}} (s _ {h}, a _ {h}) \sqrt {\frac {\operatorname {V a r} _ {P _ {h} (\cdot | s _ {h} , a _ {h})} \left(V _ {h + 1} ^ {\star} (\cdot)\right)}{n d _ {h} ^ {\mu} (s _ {h} , a _ {h})}}\right) + \widetilde {O} \left(\frac {H ^ {3}}{n \cdot \bar {d} _ {m}}\right). \tag {7}
$$
Compared to our Theorem 3.4, the additional sub-optimality bound due to differential privacy is  $\widetilde{O}\left(\frac{SH^2E_{\rho}}{n\cdot d_m}\right) = \widetilde{O}\left(\frac{SH^{\frac{5}{2}}}{n\cdot d_m\sqrt{\rho}}\right) = \widetilde{O}\left(\frac{SH^{\frac{5}{2}}}{n\cdot d_m\epsilon}\right)$ . In the most popular regime where the privacy budget  $\rho$  or  $\epsilon$  is a constant, the additional term due to differential privacy appears as a lower order term, hence becomes negligible as the sample complexity  $n$  becomes large.
Comparison to Hoeffding type pessimism. We can simply revise our algorithm by using Hoeffding type pessimism, which replaces the pessimism in line 5 with  $C_1H \cdot \sqrt{\frac{\iota}{\tilde{n}_{s_h,a_h} - E_\rho}} + \frac{C_2SHE_\rho \cdot \iota}{\tilde{n}_{s_h,a_h}}$ . Then
<sup>6</sup>The non-private empirical estimate, defined as (15) in Appendix C.
Here we apply the second part of Lemma 2.7 to achieve  $(\epsilon, \delta)$ -DP, the notation  $\widetilde{O}$  also absorbs  $\log \frac{1}{\delta}$  (only here  $\delta$  denotes the privacy budget instead of failure probability).
with a similar proof schedule, we can arrive at a sub-optimality bound that with high probability,
$$
0 \leq v ^ {\star} - v ^ {\widehat {\pi}} \leq \widetilde {O} \left(H \cdot \sum_ {h = 1} ^ {H} \sum_ {(s _ {h}, a _ {h}) \in \mathcal {C} _ {h}} d _ {h} ^ {\pi^ {\star}} (s _ {h}, a _ {h}) \sqrt {\frac {1}{n d _ {h} ^ {\mu} (s _ {h} , a _ {h})}}\right) + \widetilde {O} \left(\frac {S H ^ {2} E _ {\rho}}{n \cdot \bar {d} _ {m}}\right). \tag {8}
$$
Compared to our Theorem 3.4, our bound is tighter because we express the dominate term by the system quantities instead of explicit dependence on  $H$  (and  $\mathrm{Var}_{P_h(\cdot |s_h,a_h)}(V_{h + 1}^\star (\cdot))\leq H^2$ ). In addition, we highlight that according to Theorem G.1 in [Yin and Wang, 2021b], our main term nearly matches the non-private minimax lower bound. For more detailed discussions about our main term and how it subsumes other optimal learning bounds, we refer readers to [Yin and Wang, 2021b].
Apply Laplace Mechanism to achieve pure DP. To achieve Pure DP instead of  $\rho$ -zCDP, we can simply replace Gaussian Mechanism with Laplace Mechanism (defined as Definition E.19). Given privacy budget for Pure DP  $\epsilon$ , since the  $\ell_1$  sensitivity of  $\{n_{s_h,a_h}\} \cup \{n_{s_h,a_h,s_{h+1}}\}$  is  $\Delta_1 = 4H$ , we can add independent Laplace noises  $\mathrm{Lap}\left(\frac{4H}{\epsilon}\right)$  to each count to achieve  $\epsilon$ -DP due to Lemma E.20. Then by using  $E_{\epsilon} = \widetilde{O}\left(\frac{H}{\epsilon}\right)$  instead of  $E_{\rho}$  and keeping everything else ((3), (5) and Algorithm 1) the same, we can reach a similar result to Theorem 3.4 with the same proof schedule. The only difference is that here the additional learning bound is  $\widetilde{O}\left(\frac{SH^3}{n\cdot d_m\epsilon}\right)$ , which still appears as a lower order term.
4 Results under linear MDP: DP-VAPVI(Algorithm 2)
In large MDPs, to address the computational issues, the technique of function approximation is widely applied, and linear MDP is a concrete model to study linear function approximations. Our second result applies to the linear MDP setting. Generally speaking, function approximation reduces the dimensionality of private releases comparing to the tabular MDPs. We begin with private counts.
Private Model-based Components. Given the two datasets  $\mathcal{D}$  and  $\mathcal{D}'$  (both from  $\mu$ ) as in Algorithm 2, we can apply variance-aware pessimistic value iteration to learn a near optimal policy as in VAPVI [Yin et al., 2022]. To ensure differential privacy, we add independent Gaussian noises to the  $5H$  statistics as in DP-VAPVI (Algorithm 2) below. Since there are  $5H$  statistics, by the adaptive composition of zCDP (Lemma E.17), it suffices to keep each count  $\rho_0$ -zCDP, where  $\rho_0 = \frac{\rho}{5H}$ . In DP-VAPVI, we use  $\phi_1, \phi_2, \phi_3, K_1, K_2^8$  to denote the noises we add. For all  $\phi_i$ , we directly apply Gaussian Mechanism. For  $K_i$ , in addition to the noise matrix  $\frac{1}{\sqrt{2}}(Z + Z^\top)$ , we also add  $\frac{E}{2}I_d$  to ensure that all  $K_i$  are positive definite with high probability (The detailed definition of  $E, L$  can be found in Appendix A).
Below we will show the algorithmic design of DP-VAPVI (Algorithm 2). For the offline dataset, we divide it into two independent parts with equal length:  $\mathcal{D} = \{(s_h^\tau, a_h^\tau, r_h^\tau, s_{h+1}^\tau)\}_{\tau \in [K]}^h$  and  $\mathcal{D}' = \{(\bar{s}_h^\tau, \bar{a}_h^\tau, \bar{r}_h^\tau, \bar{s}_{h+1}^\tau)\}_{\tau \in [K]}$ . One for estimating variance and the other for calculating  $Q$ -values.
Estimating conditional variance. The first part (line 4 to line 8) aims to estimate the conditional variance of  $\widetilde{V}_{h+1}$  via the definition of variance:  $[\mathrm{Var}_h\widetilde{V}_{h+1}](s,a) = [P_h(\widetilde{V}_{h+1})^2](s,a) - ([P_h\widetilde{V}_{h+1}](s,a))^2$ . For the first term, by the definition of linear MDP, it holds that  $\left[P_h\widetilde{V}_{h+1}^2\right](s,a) = \phi(s,a)^\top \int_S \widetilde{V}_{h+1}^2(s') \, \mathrm{d}\nu_h(s') = \langle \phi, \int_S \widetilde{V}_{h+1}^2(s') \, \mathrm{d}\nu_h(s') \rangle$ . We can estimate  $\beta_h = \int_S \widetilde{V}_{h+1}^2(s') \, \mathrm{d}\nu_h(s')$  by applying ridge regression. Below is the output of ridge regression with raw statistics without noise:
$$
\underset {\beta \in \mathbb {R} ^ {d}} {\operatorname {a r g m i n}} \sum_ {k = 1} ^ {K} \left[ \left\langle \phi (\bar {s} _ {h} ^ {k}, \bar {a} _ {h} ^ {k}), \beta \right\rangle - \widetilde {V} _ {h + 1} ^ {2} \left(\bar {s} _ {h + 1} ^ {k}\right) \right] ^ {2} + \lambda \| \beta \| _ {2} ^ {2} = \bar {\Sigma} _ {h} ^ {- 1} \sum_ {k = 1} ^ {K} \phi (\bar {s} _ {h} ^ {k}, \bar {a} _ {h} ^ {k}) \widetilde {V} _ {h + 1} ^ {2} \left(\bar {s} _ {h + 1} ^ {k}\right),
$$
<sup>8</sup>We need to add noise to each of the  $5H$  counts, therefore for  $\phi_{1}$ , we actually sample  $H$  i.i.d samples  $\phi_{1,h}, h = 1,\dots ,H$  from the distribution of  $\phi_{1}$ . Then we add  $\phi_{1,h}$  to  $\sum_{\tau = 1}^{K}\phi (\bar{s}_h^\tau ,\bar{a}_h^\tau)\cdot \tilde{V}_{h + 1}(\bar{s}_{h + 1}^\tau)^2,\forall h\in [H]$ . For simplicity, we use  $\phi_{1}$  to represent all the  $\phi_{1,h}$ . The procedure applied to the other  $4H$  statistics are similar.
Algorithm 2 Differentially Private Variance-Aware Pessimistic Value Iteration (DP-VAPVI)
1: Input: Dataset  $\mathcal{D} = \{(s_h^\tau, a_h^\tau, r_h^\tau, s_{h+1}^\tau)\}_{\tau,h=1}^{K,H}$ $\mathcal{D}' = \{(\overline{s}_h^\tau, \overline{a}_h^\tau, \overline{r}_h^\tau, \overline{s}_{h+1}^\tau)\}_{\tau,h=1}^{K,H}$ . Budget for zCDP  $\rho$ . Failure probability  $\delta$ . Universal constant  $C$ .  
2: Initialization: Set  $\rho_0 \leftarrow \frac{\rho}{5H}$ ,  $\widetilde{V}_{H+1}(\cdot) \leftarrow 0$ . Sample  $\phi_1 \sim \mathcal{N}\left(0, \frac{2H^4}{\rho_0} I_d\right)$ ,  $\phi_2, \phi_3 \sim \mathcal{N}\left(0, \frac{2H^2}{\rho_0} I_d\right)$ ,  $K_1, K_2 \leftarrow \frac{E}{2} I_d + \frac{1}{\sqrt{2}} (Z + Z^\top)$ , where  $Z_{i,j} \sim \mathcal{N}\left(0, \frac{1}{4\rho_0}\right)$  (i.i.d.),  $E = \widetilde{O}\left(\sqrt{\frac{Hd}{\rho}}\right)$ . Set  $D \leftarrow \widetilde{O}\left(\frac{H^2L}{\kappa} + \frac{H^4E\sqrt{d}}{\kappa^{3/2}} + H^3\sqrt{d}\right)$ .  
3: for  $h = H, H-1, \ldots, 1$  do  
4: Set  $\widetilde{\Sigma}_h \leftarrow \sum_{\tau=1}^{K} \phi(\overline{s}_h^\tau, \overline{a}_h^\tau) \phi(\overline{s}_h^\tau, \overline{a}_h^\tau)^\top + \lambda I + K_1$   
5: Set  $\widetilde{\beta}_h \leftarrow \widetilde{\Sigma}_h^{-1}[\sum_{\tau=1}^{K} \phi(\overline{s}_h^\tau, \overline{a}_h^\tau) \cdot \widetilde{V}_{h+1}(\overline{s}_{h+1}^\tau)^2 + \phi_1]$   
6: Set  $\widetilde{\theta}_h \leftarrow \widetilde{\Sigma}_h^{-1}[\sum_{\tau=1}^{K} \phi(\overline{s}_h^\tau, \overline{a}_h^\tau) \cdot \widetilde{V}_{h+1}(\overline{s}_{h+1}^\tau) + \phi_2]$   
7: Set  $[\widetilde{\mathrm{Var}}_h\widetilde{V}_{h+1}](\cdot,\cdot) \leftarrow \langle \phi(\cdot,\cdot), \widetilde{\beta}_h\rangle_{[0,(H-h+1)^2]} - [\langle \phi(\cdot,\cdot), \widetilde{\theta}_h\rangle_{[0,H-h+1]}\]^2$   
8: Set  $\widetilde{\sigma}_h(\cdot,\cdot)^2 \leftarrow \max\{1, \widetilde{\mathrm{Var}}_h\widetilde{V}_{h+1}(\cdot,\cdot)\}$   
9: Set  $\widetilde{\Lambda}_h \leftarrow \sum_{\tau=1}^{K} \phi(s_h^\tau, a_h^\tau) \phi(s_h^\tau, a_h^\tau)^\top / \widetilde{\sigma}_h^2(s_h^\tau, a_h^\tau) + \lambda I + K_2$   
10: Set  $\widetilde{w}_h \leftarrow \widetilde{\Lambda}_h^{-1}\left(\sum_{\tau=1}^{K} \phi(s_h^\tau, a_h^\tau) \cdot (r_h^\tau + \widetilde{V}_{h+1}(s_{h+1}^\tau)) / \widetilde{\sigma}_h^2(s_h^\tau, a_h^\tau) + \phi_3\right)$   
11: Set  $\Gamma_h(\cdot,\cdot) \leftarrow C\sqrt{d} \cdot (\phi(\cdot,\cdot)^\top \widetilde{\Lambda}_h^{-1} \phi(\cdot,\cdot))^1/2 + D/K$   
12: Set  $\bar{Q}_h(\cdot,\cdot) \leftarrow \phi(\cdot,\cdot)^\top \widetilde{w}_h - \Gamma_h(\cdot,\cdot)$   
13: Set  $\widehat{Q}_h(\cdot,\cdot) \leftarrow \min\{\bar{Q}_h(\cdot,\cdot), H - h + 1\}^+$   
14: Set  $\widehat{\pi}_h(\cdot||\cdot) \leftarrow \operatorname{argmax}_{\pi_h}\bigl\langle\widehat{Q}_h(\cdot,\cdot),\pi_h(\cdot|\cdot)\bigr\rangle_A, \widetilde{V}_h(\cdot) \leftarrow \max_{\pi_h}\bigl\langle\widehat{Q}_h(\cdot,\cdot),\pi_h(\cdot|\cdot)\bigr\rangle_A$   
15: end for  
16: Output:  $\{\widehat{\pi}_h\}_{h=1}^H$ .
where definition of  $\bar{\Sigma}_h$  can be found in Appendix A. Instead of using the raw statistics, we replace them with private ones with Gaussian noises as in line 5. The second term is estimated similarly in line 6. The final estimator is defined as in line 8:  $\widetilde{\sigma}_h(\cdot,\cdot)^2 = \max\{1, \widetilde{\mathrm{Var}}_h \widetilde{V}_{h+1}(\cdot,\cdot)\}$ .<sup>9</sup>
Variance-weighted LSVI. Instead of directly applying LSVI [Jin et al., 2021], we can solve the variance-weighted LSVI (line 10). The result of variance-weighted LSVI with non-private statistics is shown below:
$$
\underset {w \in \mathbb {R} ^ {d}} {\operatorname {a r g m i n}} \lambda \| w \| _ {2} ^ {2} + \sum_ {k = 1} ^ {K} \frac {\left[ \langle \phi (s _ {h} ^ {k} , a _ {h} ^ {k}) , w \rangle - r _ {h} ^ {k} - \widetilde {V} _ {h + 1} (s _ {h + 1} ^ {k}) \right] ^ {2}}{\widetilde {\sigma} _ {h} ^ {2} (s _ {h} ^ {k} , a _ {h} ^ {k})} = \widehat {\Lambda} _ {h} ^ {- 1} \sum_ {k = 1} ^ {K} \frac {\phi (s _ {h} ^ {k} , a _ {h} ^ {k}) \cdot [ r _ {h} ^ {k} + \widetilde {V} _ {h + 1} (s _ {h + 1} ^ {k}) ]}{\widetilde {\sigma} _ {h} ^ {2} (s _ {h} ^ {k} , a _ {h} ^ {k})},
$$
where definition of  $\widehat{\Lambda}_h$  can be found in Appendix A. For the sake of differential privacy, we use private statistics instead and derive the  $\widetilde{w}_h$  as in line 10.
Our private pessimism. Notice that if we remove all the Gaussian noises we add, our DP-VAPVI (Algorithm 2) will degenerate to VAPVI [Yin et al., 2022]. We design a similar pessimistic penalty using private statistics (line 11), with additional  $\frac{D}{K}$  accounting for the extra pessimism due to DP.
Main theorem. We state our main theorem about DP-VAPVI below, the proof sketch is deferred to Appendix D.1 and detailed proof is deferred to Appendix D due to space limit. Note that quantities  $\mathcal{M}_i, L, E$  can be found in Appendix A and briefly,  $L = \widetilde{O}(\sqrt{H^3d / \rho})$ ,  $E = \widetilde{O}(\sqrt{Hd / \rho})$ . For the sample complexity lower bound, within the practical regime where the privacy budget is not very small,  $\max\{\mathcal{M}_i\}$  is dominated by  $\max\{\widetilde{O}(H^{12}d^3/\kappa^5), \widetilde{O}(H^{14}d/\kappa^5)\}$ , which also appears in the sample complexity lower bound of VAPVI [Yin et al., 2022]. The  $\sigma_V^2(s,a)$  in Theorem 4.1 is defined as  $\max\{1, \operatorname{Var}_{P_h}(V)(s,a)\}$  for any  $V$ .
Theorem 4.1. DP-VAPVI (Algorithm 2) satisfies  $\rho$ -zCDP. Furthermore, let  $K$  be the number of episodes. Under the condition that  $K > \max \{\mathcal{M}_1, \mathcal{M}_2, \mathcal{M}_3, \mathcal{M}_4\}$  and  $\sqrt{d} > \xi$ , where  $\xi := \sup_{V \in [0, H], s' \sim P_h(s, a), h \in [H]} \left| \frac{r_h + V(s') - (\mathcal{T}_h V)(s, a)}{\sigma_V(s, a)} \right|$ , for any  $0 < \lambda < \kappa$ , with probability  $1 - \delta$ , for
The max{1,} operator here is for technical reason only: we want a lower bound for each variance estimate.
all policy  $\pi$  simultaneously, the output  $\widehat{\pi}$  of DP-VAPVI satisfies
$$
v ^ {\pi} - v ^ {\widehat {\pi}} \leq \widetilde {O} \left(\sqrt {d} \cdot \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \sqrt {\phi (\cdot , \cdot) ^ {\top} \Lambda_ {h} ^ {- 1} \phi (\cdot , \cdot)} \right]\right) + \frac {D H}{K}, \tag {9}
$$
where  $\Lambda_{h} = \sum_{k = 1}^{K}\frac{\phi(s_{h}^{k},a_{h}^{k})\cdot\phi(s_{h}^{k},a_{h}^{k})^{\top}}{\sigma_{\tilde{V}_{h + 1}(s_{h}^{k},a_{h}^{k})}^{2}} +\lambda I_{d}$ ,  $D = \widetilde{O}\left(\frac{H^2L}{\kappa} +\frac{H^4E\sqrt{d}}{\kappa^{3 / 2}} +H^3\sqrt{d}\right)$  and  $\widetilde{O}$  hides constants and Polylog terms.
In particular, define  $\Lambda_h^\star = \sum_{k=1}^K \frac{\phi(s_h^k, a_h^k) \cdot \phi(s_h^k, a_h^k)^\top}{\sigma_{V_{h+1}^* (s_h^k, a_h^k)}^2} + \lambda I_d$ , we have with probability  $1 - \delta$ ,
$$
v ^ {\star} - v ^ {\widehat {\pi}} \leq \widetilde {O} \left(\sqrt {d} \cdot \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {\star}} \left[ \sqrt {\phi (\cdot , \cdot) ^ {\top} \Lambda_ {h} ^ {\star - 1} \phi (\cdot , \cdot)} \right]\right) + \frac {D H}{K}. \tag {10}
$$
Comparison to non-private counterpart VAPVI [Yin et al., 2022]. Plugging in the definition of  $L, E$  (Appendix A), under the meaningful case that the privacy budget is not very large,  $DH$  is dominated by  $\widetilde{O}\left(\frac{H^{\frac{11}{2}}d / \kappa^{\frac{3}{2}}}{\sqrt{\rho}}\right)$ . According to Theorem 3.2 in [Yin et al., 2022], the sub-optimality bound of VAPVI is for sufficiently large  $K$ , with high probability, the output  $\widehat{\pi}$  satisfies:
$$
v ^ {\star} - v ^ {\widehat {\pi}} \leq \widetilde {O} \left(\sqrt {d} \cdot \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {\star}} \left[ \sqrt {\phi (\cdot , \cdot) ^ {\top} \Lambda_ {h} ^ {\star - 1} \phi (\cdot , \cdot)} \right]\right) + \frac {2 H ^ {4} \sqrt {d}}{K}. \tag {11}
$$
Compared to our Theorem 4.1, the additional sub-optimality bound due to differential privacy is  $\widetilde{O}\left(\frac{H^{\frac{11}{2}}d / \kappa^{\frac{3}{2}}}{\sqrt{\rho\cdot K}}\right) = \widetilde{O}\left(\frac{H^{\frac{11}{2}}d / \kappa^{\frac{3}{2}}}{\epsilon\cdot K}\right)$ . In the most popular regime where the privacy budget  $\rho$  or  $\epsilon$  is a constant, the additional term due to differential privacy also appears as a lower order term.
Instance-dependent sub-optimality bound. Similar to DP-APVI (Algorithm 1), our DP-VAPVI (Algorithm 2) also enjoys instance-dependent sub-optimality bound. First, the main term in (10) improves PEVI [Jin et al., 2021] over  $O(\sqrt{d})$  on feature dependence. Also, our main term admits no explicit dependence on  $H$ , thus improves the sub-optimality bound of PEVI on horizon dependence. For more detailed discussions about our main term, we refer readers to [Yin et al., 2022].
Private and safe policy improvement. In addition to the sub-optimality bound (10), we have the so-called oracle inequality (9). Therefore, the performance  $v^{\widehat{\pi}}$  can be lower bounded by  $\sup_{\pi} \left[ v^{\pi} - \widetilde{O}\left(\sqrt{d} \cdot \sum_{h=1}^{H} \mathbb{E}_{\pi}\left[\sqrt{\phi(\cdot,\cdot)^{\top}\Lambda_{h}^{-1}\phi(\cdot,\cdot)}\right]\right) - \frac{DH}{K} \right]$ . When choosing  $\pi$  to be the optimal policy in the neighborhood of the behavior policy  $\mu$ , our DP-VAPVI (Algorithm 2) sheds light on safe policy improvement with differential privacy guarantee.
5 Tightness of our results
We believe our bounds for offline RL with DP is tight. To the best of our knowledge, APVI and VAPVI provide the tightest bound under tabular MDP and linear MDP, respectively. The suboptimality bounds of our algorithms match these two in the main term, with some lower order additional terms. The leading terms are known to match multiple information-theoretical lower bounds for offline RL simultaneously (this was illustrated in Yin and Wang [2021b], Yin et al. [2022]), for this reason our bound cannot be improved in general. For the lower order terms, the dependence on sample complexity  $n$  and privacy budget  $\epsilon$ :  $\widetilde{O}\left(\frac{1}{n\epsilon}\right)$  is optimal since policy learning is a special case of ERM problems and such dependence is optimal in DP-ERM. In addition, we believe the dependence on other parameters  $(H,S,A,d)$  in the lower order term is tight due to our special tricks as (3) and Lemma D.6.
In this section, we carry out simulations to evaluate the performance of our DP-VAPVI (Algorithm 2), and compare it with its non-private counterpart VAPVI [Yin et al., 2022] and another pessimism-based algorithm PEVI [Jin et al., 2021] which does not have privacy guarantee.
Experimental setting. We evaluate DP-VAPVI (Algorithm 2) on a synthetic linear MDP example that originates from the linear MDP in [Min et al., 2021, Yin et al., 2022] but with some modifications. $^{11}$  For details of the linear MDP setting, please refer to Appendix F. The two MDP instances we use both have horizon  $H = 20$ . We compare different algorithms in figure 1(a), while in figure 1(b), we compare our DP-VAPVI with different privacy budgets. When doing empirical evaluation, we do not split the data for DP-VAPVI or VAPVI and for DP-VAPVI, we run the simulation for 5 times and take the average performance.
(a) Compare different algorithms,  $H = 20$
(b) Different privacy budgets,  $H = 20$
Figure 1: Comparison between performance of PEVI, VAPVI and DP-VAPVI (with different privacy budgets) under the linear MDP example described above. In each figure, y-axis represents suboptimality gap  $v^{\star} - v^{\widehat{\pi}}$  while x-axis denotes the number of episodes  $K$ . The horizons are fixed to be  $H = 20$ . The number of episodes takes value from 5 to 1000.
Results and discussions. From Figure 1, we can observe that DP-VAPVI (Algorithm 2) performs slightly worse than its non-private version VAPVI [Yin et al., 2022]. This is due to the fact that we add Gaussian noise to each count. However, as the size of dataset goes larger, the performance of DP-VAPVI will converge to that of VAPVI, which supports our theoretical conclusion that the cost of privacy only appears as lower order terms. For DP-VAPVI with larger privacy budget, the scale of noise will be smaller, thus the performance will be closer to VAPVI, as shown in figure 1(b). Furthermore, in most cases, DP-VAPVI still outperforms PEVI, which does not have privacy guarantee. This arises from our privatization of variance-aware LSVI instead of LSVI.
In this work, we take the first steps towards the well-motivated task of designing private offline RL algorithms. We propose algorithms for both tabular MDPs and linear MDPs, and show that they enjoy instance-dependent sub-optimality bounds while guaranteeing differential privacy (either zCDP or pure DP). Our results highlight that the cost of privacy only appears as lower order terms, thus become negligible as the number of samples goes large.
Future extensions are numerous. We believe the technique in our algorithms (privitization of Bernstein-type pessimism and variance-aware LSVI) and the corresponding analysis can be used in online settings too to obtain tighter regret bounds for private algorithms. For the offline RL problems, we plan to consider more general function approximations and differentially private (deep) offline RL which will bridge the gap between theory and practice in offline RL applications. Many techniques we developed could be adapted to these more general settings.
<sup>11</sup>We keep the state space  $S = \{1,2\}$ , action space  $\mathcal{A} = \{1,\dots ,100\}$  and feature map of state-action pairs while we choose stochastic transition (instead of the original deterministic transition) and more complex reward.
The research is partially supported by NSF Awards #2007117 and #2048091. The authors would like to thank Ming Yin for helpful discussions.