SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dan Qiao
Santa Barbara, CA 93106
danqiao@ucsb.edu
The offline reinforcement learning (RL) problem is often motivated by the need to learn data-driven decision policies in financial, legal and healthcare applications. However, the learned policy could retain sensitive information of individuals in the training data (e.g., treatment and outcome of patients), thus susceptible to various privacy risks. We design offline RL algorithms with differential privacy guarantees which provably prevent such risks. These algorithms also enjoy strong instance-dependent learning bounds under both tabular and linear Markov Decision Process (MDP) settings. Our theory and simulation suggest that the privacy guarantee comes at (almost) no drop in utility comparing to the non-private counterpart for a medium-size dataset.
Offline Reinforcement Learning (or batch RL) aims to learn a near-optimal policy in an unknown environment<sup>1</sup> through a static dataset gathered from some behavior policy  $\mu$ . Since offline RL does not require access to the environment, it can be applied to problems where interaction with environment is infeasible, e.g., when collecting new data is costly (trade or finance [Zhang et al., 2020]), risky (autonomous driving [Sallab et al., 2017]) or illegal / unethical (healthcare [Raghu et al., 2017]). In such practical applications, the data used by an RL agent usually contains sensitive information. Take medical history for instance, for each patient, at each time step, the patient reports her health condition (age, disease, etc.), then the doctor decides the treatment (which medicine to use, the dosage of medicine, etc.), finally there is treatment outcome (whether the patient feels good, etc.) and the patient transitions to another health condition. Here, (health condition, treatment, treatment outcome) corresponds to (state, action, reward) and the dataset can be considered as  $n$  (number of patients) trajectories sampled from a MDP with horizon  $H$  (number of treatment steps). However, learning agents are known to implicitly memorize details of individual training data points verbatim [Carlini et al., 2019], even if they are irrelevant for learning [Brown et al., 2021], which makes offline RL models vulnerable to various privacy attacks.
Differential privacy (DP) [Dwork et al., 2006] is a well-established definition of privacy with many desirable properties. A differentially private offline RL algorithm will return a decision policy that is indistinguishable from a policy trained in an alternative universe any individual user is replaced, thereby preventing the aforementioned privacy risks. There is a surge of recent interest in developing RL algorithms with DP guarantees, but they focus mostly on the online setting [Vietri et al., 2020, Garcelon et al., 2021, Liao et al., 2021, Chowdhury and Zhou, 2021, Luyo et al., 2021].
<sup>1</sup>The environment is usually characterized by a Markov Decision Process (MDP) in this paper.
Offline RL is arguably more practically relevant than online RL in the applications with sensitive data. For example, in the healthcare domain, online RL requires actively running new exploratory policies (clinical trials) with every new patient, which often involves complex ethical / legal clearances, whereas offline RL uses only historical patient records that are often accessible for research purposes. Clear communication of the adopted privacy enhancing techniques (e.g., DP) to patients was reported to further improve data access [Kim et al., 2017].
Our contributions. In this paper, we present the first provably efficient algorithms for offline RL with differential privacy. Our contributions are twofold.
- We design two new pessimism-based algorithms DP-APVI (Algorithm 1) and DP-VAPVI (Algorithm 2), one for the tabular setting (finite states and actions), the other for the case with linear function approximation (under linear MDP assumption). Both algorithms enjoy DP guarantees (pure DP or zCDP) and instance-dependent learning bounds where the cost of privacy appears as lower order terms.
- We perform numerical simulations to evaluate and compare the performance of our algorithm DP-VAPVI (Algorithm 2) with its non-private counterpart VAPVI [Yin et al., 2022] as well as a popular baseline PEVI [Jin et al., 2021]. The results complement the theoretical findings by demonstrating the practicality of DP-VAPVI under strong privacy parameters.
Related work. To our knowledge, differential privacy in offline RL tasks has not been studied before, except for much simpler cases where the agent only evaluates a single policy [Balle et al., 2016, Xie et al., 2019]. Balle et al. [2016] privatized first-visit Monte Carlo-Ridge Regression estimator by an output perturbation mechanism and Xie et al. [2019] used DP-SGD. Neither paper considered offline learning (or policy optimization), which is our focus.
There is a larger body of work on private RL in the online setting, where the goal is to minimize regret while satisfying either joint differential privacy [Vietri et al., 2020, Chowdhury and Zhou, 2021, Ngo et al., 2022, Luyo et al., 2021] or local differential privacy [Garcelon et al., 2021, Liao et al., 2021, Luyo et al., 2021, Chowdhury and Zhou, 2021]. The offline setting introduces new challenges in DP as we cannot algorithmically enforce good "exploration", but have to work with a static dataset and privately estimate the uncertainty in addition to the value functions. A private online RL algorithm can sometimes be adapted for private offline RL too, but those from existing work yield suboptimal and non-adaptive bounds. We give a more detailed technical comparison in Appendix B.
Among non-private offline RL works, we build directly upon non-private offline RL methods known as Adaptive Pessimistic Value Iteration (APVI, for tabular MDPs) [Yin and Wang, 2021b] and Variance-Aware Pessimistic Value Iteration (VAPVI, for linear MDPs) [Yin et al., 2022], as they give the strongest theoretical guarantees to date. We refer readers to Appendix B for a more extensive review of the offline RL literature. Introducing DP to APVI and VAPVI while retaining the same sample complexity (modulo lower order terms) require nontrivial modifications to the algorithms.
A remark on technical novelty. Our algorithms involve substantial technical innovation over previous works on online DP-RL with joint DP guarantee<sup>2</sup>. Different from previous works, our DP-APVI (Algorithm 1) operates on Bernstein type pessimism, which requires our algorithm to deal with conditional variance using private statistics. Besides, our DP-VAPVI (Algorithm 2) replaces the LSVI technique with variance-aware LSVI (also known as weighted ridge regression, first appears in [Zhou et al., 2021]). Our DP-VAPVI releases conditional variance privately, and further applies weighted ridge regression privately. Both approaches ensure tighter instance-dependent bounds on the suboptimality of the learned policy.
Markov Decision Process. A finite-horizon Markov Decision Process (MDP) is denoted by a tuple  $M = (\mathcal{S}, \mathcal{A}, P, r, H, d_1)$  [Sutton and Barto, 2018], where  $\mathcal{S}$  is state space and  $\mathcal{A}$  is action space. A non-stationary transition kernel  $P_h: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \mapsto [0,1]$  maps each state action  $(s_h, a_h)$  to a probability distribution  $P_h(\cdot | s_h, a_h)$  and  $P_h$  can be different across time. Besides,  $r_h: \mathcal{S} \times A \mapsto \mathbb{R}$  is the expected immediate reward satisfying  $0 \leq r_h \leq 1$ ,  $d_1$  is the initial state distribution and  $H$  is the horizon.
Here we only compare our techniques (for offline RL) with the works for online RL under joint DP guarantee, as both settings allow access to the raw data.
A policy  $\pi = (\pi_1, \dots, \pi_H)$  assigns each state  $s_h \in S$  a probability distribution over actions according to the map  $s_h \mapsto \pi_h(\cdot | s_h), \forall h \in [H]$ . A random trajectory  $s_1, a_1, r_1, \dots, s_H, a_H, r_H, s_{H+1}$  is generated according to  $s_1 \sim d_1, a_h \sim \pi_h(\cdot | s_h), r_h \sim r_h(s_h, a_h), s_{h+1} \sim P_h(\cdot | s_h, a_h), \forall h \in [H]$ .
For tabular MDP, we have  $S \times \mathcal{A}$  is the discrete state-action space and  $S \coloneqq |\mathcal{S}|, A \coloneqq |\mathcal{A}|$  are finite. In this work, we assume that  $r$  is known<sup>3</sup>. In addition, we denote the per-step marginal state-action occupancy  $d_h^\pi(s, a)$  as:  $d_h^\pi(s, a) \coloneqq \mathbb{P}[s_h = s | s_1 \sim d_1, \pi] \cdot \pi_h(a|s)$ , which is the marginal state-action probability at time  $h$ .
Value function, Bellman (optimality) equations. The value function  $V_h^\pi(\cdot)$  and Q-value function  $Q_h^\pi(\cdot, \cdot)$  for any policy  $\pi$  is defined as:  $V_h^\pi(s) = \mathbb{E}_\pi[\sum_{t=h}^H r_t | s_h = s]$ ,  $Q_h^\pi(s, a) = \mathbb{E}_\pi[\sum_{t=h}^H r_t | s_h, a_h = s, a]$ ,  $\forall h, s, a \in [H] \times S \times A$ . The performance is defined as  $v^\pi := \mathbb{E}_{d_1}[V_1^\pi] = \mathbb{E}_{\pi, d_1}\left[\sum_{t=1}^H r_t\right]$ . The Bellman (optimality) equations follow  $\forall h \in [H]$ :  $Q_h^\pi = r_h + P_h V_{h+1}^\pi$ ,  $V_h^\pi = \mathbb{E}_{a \sim \pi_h}[Q_h^\pi]$ ,  $Q_h^\star = r_h + P_h V_{h+1}^\star$ ,  $V_h^\star = \max_a Q_h^\star(\cdot, a)$ .
Linear MDP [Jin et al., 2020b]. An episodic MDP  $(S, \mathcal{A}, H, P, r)$  is called a linear MDP with known feature map  $\phi: S \times \mathcal{A} \to \mathbb{R}^d$  if there exist  $H$  unknown signed measures  $\nu_h \in \mathbb{R}^d$  over  $S$  and  $H$  unknown reward vectors  $\theta_h \in \mathbb{R}^d$  such that
$$
P _ {h} \left(s ^ {\prime} \mid s, a\right) = \left\langle \phi (s, a), \nu_ {h} \left(s ^ {\prime}\right) \right\rangle , \quad r _ {h} (s, a) = \left\langle \phi (s, a), \theta_ {h} \right\rangle , \quad \forall (h, s, a, s ^ {\prime}) \in [ H ] \times \mathcal {S} \times \mathcal {A} \times \mathcal {S}.
$$
Without loss of generality, we assume  $\| \phi (s,a)\| _2\leq 1$  and  $\max (\| \nu_h(S)\| _2,\| \theta_h\| _2)\leq \sqrt{d}$  for all  $h,s,a\in [H]\times S\times \mathcal{A}$ . An important property of linear MDP is that the value functions are linear in the feature map, which is summarized in Lemma E.14.
Offline setting and the goal. The offline RL requires the agent to find a policy  $\pi$  in order to maximize the performance  $v^{\pi}$ , given only the episodic data  $\mathcal{D} = \{(s_h^\tau, a_h^\tau, r_h^\tau, s_{h+1}^\tau)\}_{\tau \in [n]}^h$  rolled out from some fixed and possibly unknown behavior policy  $\mu$ , which means we cannot change  $\mu$  and in particular we do not assume the functional knowledge of  $\mu$ . In conclusion, based on the batch data  $\mathcal{D}$  and a targeted accuracy  $\epsilon > 0$ , the agent seeks to find a policy  $\pi_{\mathrm{alg}}$  such that  $v^{\star} - v^{\pi_{\mathrm{alg}}} \leq \epsilon$ .
2.1 Assumptions in offline RL
In order to show that our privacy-preserving algorithms can generate near optimal policy, certain coverage assumptions are needed. In this section, we will list the assumptions we use in this paper.
Assumptions for tabular setting.
Assumption 2.1 ([Liu et al., 2019]). There exists one optimal policy  $\pi^{\star}$ , such that  $\pi^{\star}$  is fully covered by  $\mu$ , i.e.  $\forall s_h, a_h \in S \times \mathcal{A}$ ,  $d_h^{\pi^{\star}}(s_h, a_h) > 0$  only if  $d_h^\mu(s_h, a_h) > 0$ . Furthermore, we denote the trackable set as  $\mathcal{C}_h := \{(s_h, a_h) : d_h^\mu(s_h, a_h) > 0\}$ .
Assumption 2.1 is the weakest assumption needed for accurately learning the optimal value  $v^{\star}$  by requiring  $\mu$  to trace the state-action space of one optimal policy ( $\mu$  can be agnostic at other locations). Similar to [Yin and Wang, 2021b], we will use Assumption 2.1 for the tabular part of this paper, which enables comparison between our sample complexity to the conclusion in [Yin and Wang, 2021b], whose algorithm serves as a non-private baseline.
Assumptions for linear setting. First, we define the expectation of covariance matrix under the behavior policy  $\mu$  for all time step  $h\in [H]$  as below:
$$
\Sigma_ {h} ^ {p} := \mathbb {E} _ {\mu} \left[ \phi \left(s _ {h}, a _ {h}\right) \phi \left(s _ {h}, a _ {h}\right) ^ {\top} \right]. \tag {1}
$$
As have been shown in [Wang et al., 2021a, Yin et al., 2022], learning a near-optimal policy from offline data requires coverage assumptions. Here in linear setting, such coverage is characterized by the minimum eigenvalue of  $\Sigma_h^p$ . Similar to [Yin et al., 2022], we apply the following assumption for the sake of comparison.
Assumption 2.2 (Feature Coverage, Assumption 2 in [Wang et al., 2021a]). The data distributions  $\mu$  satisfy the minimum eigenvalue condition:  $\forall h\in [H]$ ,  $\kappa_{h}\coloneqq \lambda_{\mathrm{min}}(\Sigma_{h}^{p}) > 0$ . Furthermore, we denote  $\kappa = \min_h\kappa_h$ .
This is due to the fact that the uncertainty of reward function is dominated by that of transition kernel in RL.
4For clarity we use  $n$  for tabular MDP and  $K$  for linear MDP when referring to the sample complexity.
2.2 Differential Privacy in offline RL
In this work, we aim to design privacy-preserving algorithms for offline RL. We apply differential privacy as the formal notion of privacy. Below we revisit the definition of differential privacy.
Definition 2.3 (Differential Privacy [Dwork et al., 2006]). A randomized mechanism  $M$  satisfies  $(\epsilon, \delta)$ -differential privacy  $((\epsilon, \delta)-DP)$  if for all neighboring datasets  $U, U'$  that differ by one data point and for all possible event  $E$  in the output range, it holds that
$$
\mathbb {P} [ M (U) \in E ] \leq e ^ {\epsilon} \cdot \mathbb {P} [ M (U ^ {\prime}) \in E ] + \delta .
$$
When  $\delta = 0$ , we say pure  $DP$ , while for  $\delta > 0$ , we say approximate  $DP$ .
In the problem of offline RL, the dataset consists of several trajectories, therefore one data point in Definition 2.3 refers to one single trajectory. Hence the definition of Differential Privacy means that the difference in the distribution of the output policy resulting from replacing one trajectory in the dataset will be small. In other words, an adversary can not infer much information about any single trajectory in the dataset from the output policy of the algorithm.
Remark 2.4. For a concrete motivating example, please refer to the first paragraph of Introduction. We remark that our definition of DP is consistent with Joint DP and Local DP defined under the online RL setting where JDP/LDP also cast each user as one trajectory and provide user-wise privacy protection. For detailed definitions and more discussions about JDP/LDP, please refer to Qiao and Wang [2023a].
During the whole paper, we will use zCDP (defined below) as a surrogate for DP, since it enables cleaner analysis for privacy composition and Gaussian mechanism. The properties of zCDP (e.g., composition, conversion formula to DP) are deferred to Appendix E.3.
Definition 2.5 (zCDP [Dwork and Rothblum, 2016, Bun and Steinke, 2016]). A randomized mechanism  $M$  satisfies  $\rho$ -Zero-Concentrated Differential Privacy ( $\rho$ -zCDP), if for all neighboring datasets  $U, U'$  and all  $\alpha \in (1, \infty)$ ,
$$
D _ {\alpha} (M (U) \| M (U ^ {\prime})) \leq \rho \alpha ,
$$
where  $D_{\alpha}$  is the Renyi-divergence [Van Erven and Harremos, 2014].
Finally, we go over the definition and privacy guarantee of Gaussian mechanism.
Definition 2.6 (Gaussian Mechanism [Dwork et al., 2014]). Define the  $\ell_2$  sensitivity of a function  $f:\mathbb{N}^{\mathcal{X}}\mapsto \mathbb{R}^{d}$  as
$$
\Delta_{2}(f) = \sup_{\substack{\text{neighboring} U,U^{\prime}}}\| f(U) - f(U^{\prime})\|_{2}.
$$
The Gaussian mechanism  $\mathcal{M}$  with noise level  $\sigma$  is then given by
$$
\mathcal {M} (U) = f (U) + \mathcal {N} (0, \sigma^ {2} I _ {d}).
$$
Lemma 2.7 (Privacy guarantee of Gaussian mechanism [Dwork et al., 2014, Bun and Steinke, 2016]). Let  $f: \mathbb{N}^{\mathcal{X}} \mapsto \mathbb{R}^{d}$  be an arbitrary  $d$ -dimensional function with  $\ell_{2}$  sensitivity  $\Delta_{2}$ . Then for any  $\rho > 0$ , Gaussian Mechanism with parameter  $\sigma^{2} = \frac{\Delta_{2}^{2}}{2\rho}$  satisfies  $\rho$ -zCDP. In addition, for all  $0 < \delta, \epsilon < 1$ , Gaussian Mechanism with parameter  $\sigma = \frac{\Delta_{2}}{\epsilon}\sqrt{2\log\frac{1.25}{\delta}}$  satisfies  $(\epsilon, \delta)$ -DP.
We emphasize that the privacy guarantee covers any input data. It does not require any distributional assumptions on the data. The RL-specific assumptions (e.g., linear MDP and coverage assumptions) are only used for establishing provable utility guarantees.
3 Results under tabular MDP: DP-APVI (Algorithm 1)
For reinforcement learning, the tabular MDP setting is the most well-studied setting and our first result applies to this regime. We begin with the construction of private counts.
Private Model-based Components. Given data  $\mathcal{D} = \{(s_h^\tau, a_h^\tau, r_h^\tau, s_{h+1}^\tau)\}_{\tau \in [n]}^h$ , we denote  $n_{sh, a_h} := \sum_{\tau=1}^{n} \mathbb{1}[s_h^\tau, a_h^\tau = s_h, a_h]$  be the total counts that visit  $(s_h, a_h)$  pair at time  $h$  and  $n_{sh, a_h, s_{h+1}} :=$
$\sum_{\tau = 1}^{n}\mathbb{1}[s_h^\tau ,a_h^\tau ,s_{h + 1}^\tau = s_h,a_h,s_{h + 1}]$  be the total counts that visit  $(s_h,a_h,s_{h + 1})$  pair at time  $h$  , then given the budget  $\rho$  for zCDP, we add independent Gaussian noises to all the counts:
$$
n _ {s _ {h}, a _ {h}} ^ {\prime} = \left\{n _ {s _ {h}, a _ {h}} + \mathcal {N} (0, \sigma^ {2}) \right\} ^ {+}, n _ {s _ {h}, a _ {h}, s _ {h + 1}} ^ {\prime} = \left\{n _ {s _ {h}, a _ {h}, s _ {h + 1}} + \mathcal {N} (0, \sigma^ {2}) \right\} ^ {+}, \sigma^ {2} = \frac {2 H}{\rho}. \tag {2}
$$
However, after adding noise, the noisy counts  $n'$  may not satisfy  $n_{s_h,a_h}' = \sum_{s_{h+1} \in S} n_{s_h,a_h,s_{h+1}}'$ . To address this problem, we choose the private counts of visiting numbers as the solution to the following optimization problem (here  $E_{\rho} = 4\sqrt{\frac{H \log \frac{4HS^2A}{\delta}}{\rho}}$  is chosen as a high probability uniform bound of the noises we add):
$$
\{\widetilde {n} _ {s _ {h}, a _ {h}, s ^ {\prime}} \} _ {s ^ {\prime} \in \mathcal {S}} = \operatorname * {a r g m i n} _ {\{x _ {s ^ {\prime}} \} _ {s ^ {\prime} \in \mathcal {S}}} \max  _ {s ^ {\prime} \in \mathcal {S}} \left| x _ {s ^ {\prime}} - n _ {s _ {h}, a _ {h}, s ^ {\prime}} ^ {\prime} \right|
$$
$$
\text {s u c h} \left| \sum_ {s ^ {\prime} \in \mathcal {S}} x _ {s ^ {\prime}} - n _ {s _ {h}, a _ {h}} ^ {\prime} \right| \leq \frac {E _ {\rho}}{2} \text {a n d} x _ {s ^ {\prime}} \geq 0, \forall s ^ {\prime} \in \mathcal {S}. \tag {3}
$$
$$
\widetilde {n} _ {s _ {h}, a _ {h}} = \sum_ {s ^ {\prime} \in \mathcal {S}} \widetilde {n} _ {s _ {h}, a _ {h}, s ^ {\prime}}.
$$
Remark 3.1 (Some explanations). The optimization problem above serves as a post-processing step which will not affect the DP guarantee of our algorithm. Briefly speaking, (3) finds a set of noisy counts such that  $\widetilde{n}_{s_h,a_h} = \sum_{s' \in S} \widetilde{n}_{s_h,a_h,s'}$  and the estimation error for each  $\widetilde{n}_{s_h,a_h}$  and  $\widetilde{n}_{s_h,a_h,s'}$  is roughly  $E_{\rho}$ . In contrast, if we directly take the crude approach that  $\widetilde{n}_{s_h,a_h,s_{h+1}} = n_{s_h,a_h,s_{h+1}}'$  and  $\widetilde{n}_{s_h,a_h} = \sum_{s_{h+1} \in S} \widetilde{n}_{s_h,a_h,s_{h+1}}$ , we can only derive  $|\widetilde{n}_{s_h,a_h} - n_{s_h,a_h}| \leq \widetilde{O}(\sqrt{SE}_{\rho})$  through concentration on summation of  $S$  i.i.d. Gaussian noises. In conclusion, solving the optimization problem (3) enables tight analysis for the lower order term (the additional cost of privacy).
Remark 3.2 (Computational efficiency). The optimization problem (3) can be reformulated as:
$$
\min  t, \text {s . t .} | x _ {s ^ {\prime}} - n _ {s _ {h}, a _ {h}, s ^ {\prime}} ^ {\prime} | \leq t \text {a n d} x _ {s ^ {\prime}} \geq 0 \forall s ^ {\prime} \in \mathcal {S}, \left| \sum_ {s ^ {\prime} \in \mathcal {S}} x _ {s ^ {\prime}} - n _ {s _ {h}, a _ {h}} ^ {\prime} \right| \leq \frac {E _ {\rho}}{2}. \tag {4}
$$
Note that (4) is a Linear Programming problem with  $S + 1$  variables and  $2S + 2$  linear constraints (one constraint on absolute value is equivalent to two linear constraints), which can be solved efficiently by the simplex method [Ficken, 2015] or other provably efficient algorithms [Nemhauser and Wolsey, 1988]. Therefore, our Algorithm 1 is computationally friendly.
The private estimation of the transition kernel is defined as:
$$
\widetilde {P} _ {h} \left(s ^ {\prime} \mid s _ {h}, a _ {h}\right) = \frac {\widetilde {n} _ {s _ {h} , a _ {h} , s ^ {\prime}}}{\widetilde {n} _ {s _ {h} , a _ {h}}}, \tag {5}
$$
if  $\widetilde{n}_{s_h,a_h} > E_\rho$  and  $\widetilde{P}_h(s'|s_h,a_h) = \frac{1}{S}$  otherwise.
Remark 3.3. Different from the transition kernel estimate in previous works [Vietri et al., 2020, Chowdhury and Zhou, 2021] that may not be a distribution, we have to ensure that ours is a probability distribution, because our Bernstein type pessimism (line 5 in Algorithm 1) needs to take variance over this transition kernel estimate. The intuition behind the construction of our private transition kernel is that, for those state-action pairs with  $\widetilde{n}_{s_h,a_h} \leq E_{\rho}$ , we can not distinguish whether the non-zero private count comes from noise or actual visitation. Therefore we only take the empirical estimate of the state-action pairs with sufficiently large  $\widetilde{n}_{s_h,a_h}$ .
This conclusion is summarized in Lemma C.3.
Algorithm 1 Differentially Private Adaptive Pessimistic Value Iteration (DP-APVI)
1: Input: Offline dataset  $\mathcal{D} = \{(s_h^\tau, a_h^\tau, r_h^\tau, s_{h+1}^\tau)\}_{\tau,h=1}^{n,H}$ . Reward function  $r$ . Constants  $C_1 = \sqrt{2}$ ,  $C_2 = 16$ ,  $C > 1$ , failure probability  $\delta$ , budget for zCDP  $\rho$ .
2: Initialization: Calculate  $\widetilde{n}_{s_h,a_h},\widetilde{n}_{s_h,a_h,s_{h + 1}}$  as (3),  $\widetilde{P}_h(s_{h + 1}|s_h,a_h)$  as (5).  $\widetilde{V}_{H + 1}(\cdot)\gets 0.$ $E_{\rho}\gets$ $4\sqrt{\frac{H\log\frac{4HS^2A}{\delta}}{\rho}}.\iota \gets \log (HSA / \delta).$
3: for  $h = H, H - 1, \ldots, 1$  do
4:  $\widetilde{Q}_h(\cdot ,\cdot)\gets r_h(\cdot ,\cdot) + (\widetilde{P}_h\cdot \widetilde{V}_{h + 1})(\cdot ,\cdot)$
5:  $\forall s_h, a_h$ , let  $\Gamma_h(s_h, a_h) \gets C_1 \sqrt{\frac{\operatorname{Var}_{\widetilde{P}_{s_h}, a_h} (\widetilde{V}_{h+1})^{\cdot\iota}}{\widetilde{n}_{s_h, a_h} - E_{\rho}}} + \frac{C_2 S H E_{\rho}^{\cdot\iota}}{\widetilde{n}_{s_h, a_h}}$  if  $\widetilde{n}_{s_h, a_h} > E_{\rho}$ , otherwise  $CH$ .
6:  $\widehat{Q}_h^p (\cdot ,\cdot)\gets \widetilde{Q}_h(\cdot ,\cdot) - \Gamma_h(\cdot ,\cdot).$
7:  $\overline{Q}_h(\cdot ,\cdot)\gets \min \{\widehat{Q}_h^p (\cdot ,\cdot),H - h + 1\}^+$
8:  $\forall s_h,$  let  $\widehat{\pi}_h(\cdot |s_h)\gets \mathrm{argmax}_{\pi_h}\langle \overline{Q}_h(s_h,\cdot),\pi_h(\cdot |s_h)\rangle$  and  $\widetilde{V}_h(s_h)\gets \langle \overline{Q}_h(s_h,\cdot),\widehat{\pi}_h(\cdot |s_h)\rangle .$
9: end for
10: Output:  $\{\widehat{\pi}_h\}$ .
Algorithmic design. Our algorithmic design originates from the idea of pessimism, which holds conservative view towards the locations with high uncertainty and prefers the locations we have more confidence about. Based on the Bernstein type pessimism in APVI [Yin and Wang, 2021b], we design a similar pessimistic algorithm with private counts to ensure differential privacy. If we replace  $\widetilde{n}$  and  $\widetilde{P}$  with  $n$  and  $\widehat{P}^6$ , then our DP-APVI (Algorithm 1) will degenerate to APVI. Compared to the pessimism defined in APVI, our pessimistic penalty has an additional term  $\widetilde{O}\left(\frac{SHE_{\rho}}{\widetilde{n}_{s_h,a_h}}\right)$ , which accounts for the additional pessimism due to our application of private statistics.
We state our main theorem about DP-APVI below, the proof sketch is deferred to Appendix C.1 and detailed proof is deferred to Appendix C due to space limit.