Title: Offline Reinforcement Learning with Differential Privacy
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential privacy

offline reinforcement learning

reinforcement learning theory
Abstract: The offline reinforcement learning (RL) problem is often motivated by the need to learn data-driven decision policies in financial, legal and healthcare applications.  However, the learned policy could retain sensitive information of individuals in the training data (e.g., treatment and outcome of patients), thus susceptible to various privacy risks. We design offline RL algorithms with differential privacy guarantees which provably prevent such risks. These algorithms also enjoy strong instance-dependent learning bounds under both tabular and linear Markov Decision Process (MDP) settings. Our theory and simulation suggest that the privacy guarantee comes at (almost) no drop in utility comparing to the non-private counterpart for a medium-size dataset.

Offline Reinforcement Learning with Differential Privacy

Abstract

The offline reinforcement learning (RL) problem is often motivated by the need to learn data-driven decision policies in financial, legal and healthcare applications. However, the learned policy could retain sensitive information of individuals in the training data (e.g., treatment and outcome of patients), thus susceptible to various privacy risks. We design offline RL algorithms with differential privacy guarantees which provably prevent such risks. These algorithms also enjoy strong instancedependent learning bounds under both tabular and linear Markov Decision Process (MDP) settings. Our theory and simulation suggest that the privacy guarantee comes at (almost) no drop in utility comparing to the non-private counterpart for a medium-size dataset.

1 Introduction

Offline Reinforcement Learning (or batch RL) aims to learn a near-optimal policy in an unknown environment 1 through a static dataset gathered from some behavior policy µ . Since offline RL does not require access to the environment, it can be applied to problems where interaction with environment is infeasible, e.g. , when collecting new data is costly (trade or finance [Zhang et al., 2020]), risky (autonomous driving [Sallab et al., 2017]) or illegal / unethical (healthcare [Raghu et al., 2017]). In such practical applications, the data used by an RL agent usually contains sensitive information. Take medical history for instance, for each patient, at each time step, the patient reports her health condition (age, disease, etc.), then the doctor decides the treatment (which medicine to use, the dosage of medicine, etc.), finally there is treatment outcome (whether the patient feels good, etc.) and the patient transitions to another health condition. Here, (health condition, treatment, treatment outcome) corresponds to (state, action, reward) and the dataset can be considered as n (number of patients) trajectories sampled from a MDP with horizon H (number of treatment steps). However, learning agents are known to implicitly memorize details of individual training data points verbatim [Carlini et al., 2019], even if they are irrelevant for learning [Brown et al., 2021], which makes offline RL models vulnerable to various privacy attacks.

Differential privacy (DP) [Dwork et al., 2006] is a well-established definition of privacy with many desirable properties. A differentially private offline RL algorithm will return a decision policy that is indistinguishable from a policy trained in an alternative universe any individual user is replaced, thereby preventing the aforementioned privacy risks. There is a surge of recent interest in developing RL algorithms with DP guarantees, but they focus mostly on the online setting [Vietri et al., 2020, Garcelon et al., 2021, Liao et al., 2021, Chowdhury and Zhou, 2021, Luyo et al., 2021].

Offline RL is arguably more practically relevant than online RL in the applications with sensitive data. For example, in the healthcare domain, online RL requires actively running new exploratory policies (clinical trials) with every new patient, which often involves complex ethical / legal clearances, whereas offline RL uses only historical patient records that are often accessible for research purposes. Clear communication of the adopted privacy enhancing techniques (e.g., DP) to patients was reported to further improve data access [Kim et al., 2017].

Our contributions. In this paper, we present the first provably efficient algorithms for offline RL with differential privacy. Our contributions are twofold.

We design two new pessimism-based algorithms DP-APVI (Algorithm 1) and DP-VAPVI (Algorithm 2), one for the tabular setting (finite states and actions), the other for the case with linear function approximation (under linear MDP assumption). Both algorithms enjoy DP guarantees (pure DP or zCDP) and instance-dependent learning bounds where the cost of privacy appears as lower order terms.

Weperform numerical simulations to evaluate and compare the performance of our algorithm DP-VAPVI (Algorithm 2) with its non-private counterpart VAPVI [Yin et al., 2022] as well as a popular baseline PEVI [Jin et al., 2021]. The results complement the theoretical findings by demonstrating the practicality of DP-VAPVI under strong privacy parameters.

Related work. To our knowledge, differential privacy in offline RL tasks has not been studied before, except for much simpler cases where the agent only evaluates a single policy [Balle et al., 2016, Xie et al., 2019]. Balle et al. [2016] privatized first-visit Monte Carlo-Ridge Regression estimator by an output perturbation mechanism and Xie et al. [2019] used DP-SGD. Neither paper considered offline learning (or policy optimization), which is our focus.

There is a larger body of work on private RL in the online setting, where the goal is to minimize regret while satisfying either joint differential privacy [Vietri et al., 2020, Chowdhury and Zhou, 2021, Ngo et al., 2022, Luyo et al., 2021] or local differential privacy [Garcelon et al., 2021, Liao et al., 2021, Luyo et al., 2021, Chowdhury and Zhou, 2021]. The offline setting introduces new challenges in DP as we cannot algorithmically enforce good 'exploration', but have to work with a static dataset and privately estimate the uncertainty in addition to the value functions. A private online RL algorithm can sometimes be adapted for private offline RL too, but those from existing work yield suboptimal and non-adaptive bounds. We give a more detailed technical comparison in Appendix B.

Among non-private offline RL works, we build directly upon non-private offline RL methods known as Adaptive Pessimistic Value Iteration (APVI, for tabular MDPs) [Yin and Wang, 2021b] and Variance-Aware Pessimistic Value Iteration (V APVI, for linear MDPs) [Yin et al., 2022], as they give the strongest theoretical guarantees to date. We refer readers to Appendix B for a more extensive review of the offline RL literature. Introducing DP to APVI and VAPVI while retaining the same sample complexity (modulo lower order terms) require nontrivial modifications to the algorithms.

A remark on technical novelty. Our algorithms involve substantial technical innovation over previous works on online DP-RL with joint DP guarantee 2 . Different from previous works, our DP-APVI (Algorithm 1) operates on Bernstein type pessimism, which requires our algorithm to deal with conditional variance using private statistics. Besides, our DP-V APVI (Algorithm 2) replaces the LSVI technique with variance-aware LSVI (also known as weighted ridge regression, first appears in [Zhou et al., 2021]). Our DP-VAPVI releases conditional variance privately, and further applies weighted ridge regression privately. Both approaches ensure tighter instance-dependent bounds on the suboptimality of the learned policy.

2 Problem Setup

Markov Decision Process. A finite-horizon Markov Decision Process (MDP) is denoted by a tuple M = ( S , A , P, r, H, d 1 ) [Sutton and Barto, 2018], where S is state space and A is action space. A non-stationary transition kernel P h : S×A×S ↦→ [0 , 1] maps each state action ( s h , a h ) to a probability distribution P h ( ·| s h , a h ) and P h can be different across time. Besides, r h : S × A ↦→ R is the expected immediate reward satisfying 0 ≤ r h ≤ 1 , d 1 is the initial state distribution and H is the horizon.

A policy π = ( π 1 , · · · , π H ) assigns each state s h ∈ S a probability distribution over actions according to the map s h ↦→ π h ( ·| s h ) , ∀ h ∈ [ H ] . A random trajectory s 1 , a 1 , r 1 , · · · , s H , a H , r H , s H +1 is generated according to s 1 ∼ d 1 , a h ∼ π h ( ·| s h ) , r h ∼ r h ( s h , a h ) , s h +1 ∼ P h ( ·| s h , a h ) , ∀ h ∈ [ H ] .

For tabular MDP, we have S × A is the discrete state-action space and S := |S| , A := |A| are finite. In this work, we assume that r is known 3 . In addition, we denote the per-step marginal state-action occupancy d π h ( s, a ) as: d π h ( s, a ) := P [ s h = s | s 1 ∼ d 1 , π ] · π h ( a | s ) , which is the marginal state-action probability at time h .

Value function, Bellman (optimality) equations. The value function V π h ( · ) and Q-value function Q π h ( · , · ) for any policy π is defined as: V π h ( s ) = E π [ ∑ H t = h r t | s h = s ] , Q π h ( s, a ) = E π [ ∑ H t = h r t | s h , a h = s, a ] , ∀ h, s, a ∈ [ H ] × S × A . The performance is defined as v π := E d 1 [ V π 1 ] = E π,d 1 [ ∑ H t =1 r t ] . The Bellman (optimality) equations follow ∀ h ∈ [ H ] : Q π h = r h + P h V π h +1 , V π h = E a ∼ π h [ Q π h ] , Q ⋆ h = r h + P h V ⋆ h +1 , V ⋆ h = max a Q ⋆ h ( · , a ) .

Linear MDP [Jin et al., 2020b]. An episodic MDP ( S , A , H, P, r ) is called a linear MDP with known feature map ϕ : S × A → R d if there exist H unknown signed measures ν h ∈ R d over S and H unknown reward vectors θ h ∈ R d such that

√

Without loss of generality, we assume ∥ ϕ ( s, a ) ∥ 2 ≤ 1 and max( ∥ ν h ( S ) ∥ 2 , ∥ θ h ∥ 2 ) ≤ d for all h, s, a ∈ [ H ] ×S × A . An important property of linear MDP is that the value functions are linear in the feature map, which is summarized in Lemma E.14.

Offline setting and the goal. The offline RL requires the agent to find a policy π in order to maximize the performance v π , given only the episodic data D = { ( s τ h , a τ h , r τ h , s τ h +1 ) } h ∈ [ H ] τ ∈ [ n ] 4 rolled out from some fixed and possibly unknown behavior policy µ , which means we cannot change µ and in particular we do not assume the functional knowledge of µ . In conclusion, based on the batch data D and a targeted accuracy ϵ > 0 , the agent seeks to find a policy π alg such that v ⋆ -v π alg ≤ ϵ .

2.1 Assumptions in offline RL

In order to show that our privacy-preserving algorithms can generate near optimal policy, certain coverage assumptions are needed. In this section, we will list the assumptions we use in this paper.

Assumptions for tabular setting.

Assumption 2.1 ([Liu et al., 2019]) . There exists one optimal policy π ⋆ , such that π ⋆ is fully covered by µ , i.e. ∀ s h , a h ∈ S × A , d π ⋆ h ( s h , a h ) > 0 only if d µ h ( s h , a h ) > 0 . Furthermore, we denote the trackable set as C h := { ( s h , a h ) : d µ h ( s h , a h ) > 0 } .

Assumption 2.1 is the weakest assumption needed for accurately learning the optimal value v ⋆ by requiring µ to trace the state-action space of one optimal policy ( µ can be agnostic at other locations). Similar to [Yin and Wang, 2021b], we will use Assumption 2.1 for the tabular part of this paper, which enables comparison between our sample complexity to the conclusion in [Yin and Wang, 2021b], whose algorithm serves as a non-private baseline.

Assumptions for linear setting. First, we define the expectation of covariance matrix under the behavior policy µ for all time step h ∈ [ H ] as below:

As have been shown in [Wang et al., 2021a, Yin et al., 2022], learning a near-optimal policy from offline data requires coverage assumptions. Here in linear setting, such coverage is characterized by the minimum eigenvalue of Σ p h . Similar to [Yin et al., 2022], we apply the following assumption for the sake of comparison.

Assumption 2.2 (Feature Coverage, Assumption 2 in [Wang et al., 2021a]) . The data distributions µ satisfy the minimum eigenvalue condition: ∀ h ∈ [ H ] , κ h := λ min (Σ p h ) > 0 . Furthermore, we denote κ = min h κ h .

2.2 Differential Privacy in offline RL

In this work, we aim to design privacy-preserving algorithms for offline RL. We apply differential privacy as the formal notion of privacy. Below we revisit the definition of differential privacy.

Definition 2.3 (Differential Privacy [Dwork et al., 2006]) . A randomized mechanism M satisfies ( ϵ, δ ) -differential privacy ( ( ϵ, δ ) -DP) if for all neighboring datasets U, U ′ that differ by one data point and for all possible event E in the output range, it holds that

When δ = 0 , we say pure DP, while for δ > 0 , we say approximate DP.

In the problem of offline RL, the dataset consists of several trajectories, therefore one data point in Definition 2.3 refers to one single trajectory. Hence the definition of Differential Privacy means that the difference in the distribution of the output policy resulting from replacing one trajectory in the dataset will be small. In other words, an adversary can not infer much information about any single trajectory in the dataset from the output policy of the algorithm.

Remark 2.4. For a concrete motivating example, please refer to the first paragraph of Introduction. We remark that our definition of DP is consistent with Joint DP and Local DP defined under the online RL setting where JDP/LDP also cast each user as one trajectory and provide user-wise privacy protection. For detailed definitions and more discussions about JDP/LDP, please refer to Qiao and Wang [2023a].

During the whole paper, we will use zCDP (defined below) as a surrogate for DP, since it enables cleaner analysis for privacy composition and Gaussian mechanism. The properties of zCDP (e.g., composition, conversion formula to DP) are deferred to Appendix E.3.

Definition 2.5 (zCDP [Dwork and Rothblum, 2016, Bun and Steinke, 2016]) . A randomized mechanism M satisfies ρ -Zero-Concentrated Differential Privacy ( ρ -zCDP), if for all neighboring datasets U, U ′ and all α ∈ (1 , ∞ ) ,

where D α is the Renyi-divergence [Van Erven and Harremos, 2014].

Finally, we go over the definition and privacy guarantee of Gaussian mechanism.

Definition 2.6 (Gaussian Mechanism [Dwork et al., 2014]) . Define the ℓ 2 sensitivity of a function f : N X ↦→ R d as

The Gaussian mechanism M with noise level σ is then given by

Lemma 2.7 (Privacy guarantee of Gaussian mechanism [Dwork et al., 2014, Bun and Steinke, 2016]) . Let f : N X ↦→ R d be an arbitrary d-dimensional function with ℓ 2 sensitivity ∆ 2 . Then for any ρ > 0 , Gaussian Mechanism with parameter σ 2 = ∆ 2 2 2 ρ satisfies ρ -zCDP. In addition, for all 0 < δ, ϵ < 1 , √ -DP.

Gaussian Mechanism with parameter σ = ∆ 2 ϵ 2 log 1 . 25 δ satisfies ( ϵ, δ )

We emphasize that the privacy guarantee covers any input data. It does not require any distributional assumptions on the data. The RL-specific assumptions (e.g., linear MDP and coverage assumptions) are only used for establishing provable utility guarantees.

3 Results under tabular MDP: DP-APVI (Algorithm 1)

For reinforcement learning, the tabular MDP setting is the most well-studied setting and our first result applies to this regime. We begin with the construction of private counts.

Private Model-based Components. Given data D = { ( s τ h , a τ h , r τ h , s τ h +1 ) } h ∈ [ H ] τ ∈ [ n ] , we denote n s h ,a h := ∑ n τ =1 1 [ s τ h , a τ h = s h , a h ] be the total counts that visit ( s h , a h ) pair at time h and n s h ,a h ,s h +1 :=

∑ n τ =1 1 [ s τ h , a τ h , s τ h +1 = s h , a h , s h +1 ] be the total counts that visit ( s h , a h , s h +1 ) pair at time h , then given the budget ρ for zCDP, we add independent Gaussian noises to all the counts:

However, after adding noise, the noisy counts n ′ may not satisfy n ′ s h ,a h = ∑ s h +1 ∈S n ′ s h ,a h ,s h +1 . To address this problem, we choose the private counts of visiting numbers as the solution to the following optimization problem (here E ρ = 4 √ H log 4 HS 2 A δ ρ is chosen as a high probability uniform bound of the noises we add):

Remark 3.1 (Some explanations) . The optimization problem above serves as a post-processing step which will not affect the DP guarantee of our algorithm. Briefly speaking, (3) finds a set of noisy counts such that ˜ n s h ,a h = ∑ s ′ ∈S ˜ n s h ,a h ,s ′ and the estimation error for each ˜ n s h ,a h and ˜ n s h ,a h ,s ′ is roughly E ρ . 5 In contrast, if we directly take the crude approach that ˜ n s h ,a h ,s h +1 = n ′ s h ,a h ,s h +1 and ˜ n s h ,a h = ∑ s h +1 ∈S ˜ n s h ,a h ,s h +1 , we can only derive | ˜ n s h ,a h -n s h ,a h | ≤ ˜ O ( √ SE ρ ) through concentration on summation of S i.i.d. Gaussian noises. In conclusion, solving the optimization problem (3) enables tight analysis for the lower order term (the additional cost of privacy).

Remark 3.2 (Computational efficiency) . The optimization problem (3) can be reformulated as:

Note that (4) is a Linear Programming problem with S +1 variables and 2 S +2 linear constraints (one constraint on absolute value is equivalent to two linear constraints), which can be solved efficiently by the simplex method [Ficken, 2015] or other provably efficient algorithms [Nemhauser and Wolsey, 1988]. Therefore, our Algorithm 1 is computationally friendly.

The private estimation of the transition kernel is defined as:

if ˜ n s h ,a h > E ρ and ˜ P h ( s ′ | s h , a h ) = 1 S otherwise.

Remark 3.3. Different from the transition kernel estimate in previous works [Vietri et al., 2020, Chowdhury and Zhou, 2021] that may not be a distribution, we have to ensure that ours is a probability distribution, because our Bernstein type pessimism (line 5 in Algorithm 1) needs to take variance over this transition kernel estimate. The intuition behind the construction of our private transition kernel is that, for those state-action pairs with ˜ n s h ,a h ≤ E ρ , we can not distinguish whether the non-zero private count comes from noise or actual visitation. Therefore we only take the empirical estimate of the state-action pairs with sufficiently large ˜ n s h ,a h .

Algorithm 1 Differentially Private Adaptive Pessimistic Value Iteration (DP-APVI)

1: Input: Offline dataset D = { ( s τ h , a τ h , r τ h , s τ h +1 ) } n,H τ,h =1 . Reward function r . Constants C 1 = √ 2 , C 2 = 16 , C > 1 , failure probability δ , budget for zCDP ρ .

2: Initialization: Calculate ˜ n s h ,a h , ˜ n s h ,a h ,s h +1 as (3), ˜ P h ( s h +1 | s h , a h ) as (5). ˜ V H +1 ( · ) ← 0 . E ρ ← 4 √ H log 4 HS 2 A δ ρ . ι ← log( HSA/δ ) .

3: for h = H,H -1 , . . . , 1 do

4: ˜ Q h ( · , · ) ← r h ( · , · ) + ( ˜ P h · ˜ V h +1 )( · , · )

5: ∀ s h , a h , let Γ h ( s h , a h ) ← C 1 √ Var ˜ Ps h ,a h ( ˜ V h +1 ) · ι ˜ n s h ,a h -E ρ + C 2 SHE ρ · ι ˜ n s h ,a h if ˜ n s h ,a h > E ρ , otherwise CH .

6: ̂ Q p h ( · , · ) ← ˜ Q h ( · , · ) -Γ h ( · , · ) .

7: Q h ( · , · ) ← min { ̂ Q p h ( · , · ) , H -h +1 } + .

8: ∀ s h , let π h ( ·| s h ) ← argmax π ⟨ Q h ( s h , · ) , π h ( ·| s h ) ⟩ and ˜ V h ( s h ) ←⟨ Q h ( s h , · ) , π h ( ·| s h ) ⟩ .

9: end for

̂ h ̂

10: Output: { ̂ π h } .

Algorithmic design. Our algorithmic design originates from the idea of pessimism, which holds conservative view towards the locations with high uncertainty and prefers the locations we have more confidence about. Based on the Bernstein type pessimism in APVI [Yin and Wang, 2021b], we design a similar pessimistic algorithm with private counts to ensure differential privacy. If we replace ˜ n and ˜ P with n and ̂ P 6 , then our DP-APVI (Algorithm 1) will degenerate to APVI. Compared to the pessimism defined in APVI, our pessimistic penalty has an additional term ˜ O ( SHE ρ ˜ n s h ,a h ) , which accounts for the additional pessimism due to our application of private statistics.

We state our main theorem about DP-APVI below, the proof sketch is deferred to Appendix C.1 and detailed proof is deferred to Appendix C due to space limit.

Theorem 3.4. DP-APVI (Algorithm 1) satisfies ρ -zCDP. Furthermore, under Assumption 2.1, denote ¯ d m := min h ∈ [ H ] { d µ h ( s h , a h ) : d µ h ( s h , a h ) > 0 } . For any 0 < δ < 1 , there exists constant c 1 > 0 , such that when n > c 1 · max { H 2 , E ρ } / ¯ d m · ι ( ι = log( HSA/δ ) ), with probability 1 -δ , the output policy ̂ π of DP-APVI satisfies

where ˜ O hides constants and Polylog terms, E ρ = 4 √ H log 4 HS 2 A δ ρ .

Comparison to non-private counterpart APVI [Yin and Wang, 2021b]. According to Theorem 4.1 in [Yin and Wang, 2021b], the sub-optimality bound of APVI is for large enough n , with high probability, the output ̂ π satisfies:

Compared to our Theorem 3.4, the additional sub-optimality bound due to differential privacy is ˜ O ( SH 2 E ρ n · ¯ d m ) = ˜ O ( SH 5 2 n · ¯ d m √ ρ ) = ˜ O ( SH 5 2 n · ¯ d m ϵ ) . 7 In the most popular regime where the privacy budget ρ or ϵ is a constant, the additional term due to differential privacy appears as a lower order term, hence becomes negligible as the sample complexity n becomes large.

Comparison to Hoeffding type pessimism. We can simply revise our algorithm by using Hoeffding type pessimism, which replaces the pessimism in line 5 with C 1 H · √ ι ˜ n s h ,a h -E ρ + C 2 SHE ρ · ι ˜ n s h ,a h . Then

with a similar proof schedule, we can arrive at a sub-optimality bound that with high probability,

Compared to our Theorem 3.4, our bound is tighter because we express the dominate term by the system quantities instead of explicit dependence on H (and Var P h ( ·| s h ,a h ) ( V ⋆ h +1 ( · )) ≤ H 2 ). In addition, we highlight that according to Theorem G.1 in [Yin and Wang, 2021b], our main term nearly matches the non-private minimax lower bound. For more detailed discussions about our main term and how it subsumes other optimal learning bounds, we refer readers to [Yin and Wang, 2021b].

Apply Laplace Mechanism to achieve pure DP. To achieve Pure DP instead of ρ -zCDP, we can simply replace Gaussian Mechanism with Laplace Mechanism (defined as Definition E.19). Given privacy budget for Pure DP ϵ , since the ℓ 1 sensitivity of { n s h ,a h } ∪ { n s h ,a h ,s h +1 } is ∆ 1 = 4 H , we can add independent Laplace noises Lap( 4 H ϵ ) to each count to achieve ϵ -DP due to Lemma E.20. Then by using E ϵ = ˜ O ( H ϵ ) instead of E ρ and keeping everything else ((3), (5) and Algorithm 1) the same, we can reach a similar result to Theorem 3.4 with the same proof schedule. The only difference is that here the additional learning bound is ˜ O ( SH 3 n · ¯ d m ϵ ) , which still appears as a lower order term.

4 Results under linear MDP: DP-VAPVI(Algorithm 2)

In large MDPs, to address the computational issues, the technique of function approximation is widely applied, and linear MDP is a concrete model to study linear function approximations. Our second result applies to the linear MDP setting. Generally speaking, function approximation reduces the dimensionality of private releases comparing to the tabular MDPs. We begin with private counts.

Private Model-based Components. Given the two datasets D and D ′ (both from µ ) as in Algorithm 2, we can apply variance-aware pessimistic value iteration to learn a near optimal policy as in VAPVI [Yin et al., 2022]. To ensure differential privacy, we add independent Gaussian noises to the 5 H statistics as in DP-V APVI (Algorithm 2) below. Since there are 5 H statistics, by the adaptive composition of zCDP (Lemma E.17), it suffices to keep each count ρ 0 -zCDP, where ρ 0 = ρ 5 H . In DP-VAPVI, we use ϕ 1 , ϕ 2 , ϕ 3 , K 1 , K 2 8 to denote the noises we add. For all ϕ i , we directly apply Gaussian Mechanism. For K i , in addition to the noise matrix 1 √ 2 ( Z + Z ⊤ ) , we also add E 2 I d to ensure that all K i are positive definite with high probability (The detailed definition of E,L can be found in Appendix A).

Below we will show the algorithmic design of DP-VAPVI (Algorithm 2). For the offline dataset, we divide it into two independent parts with equal length: D = { ( s τ h , a τ h , r τ h , s τ h +1 ) } h ∈ [ H ] τ ∈ [ K ] and D ′ = { (¯ s τ h , ¯ a τ h , ¯ r τ h , ¯ s τ h +1 ) } h ∈ [ H ] τ ∈ [ K ] . One for estimating variance and the other for calculating Q -values.

Estimating conditional variance. The first part (line 4 to line 8) aims to estimate the conditional variance of ˜ V h +1 via the definition of variance: [Var h ˜ V h +1 ]( s, a ) = [ P h ( ˜ V h +1 ) 2 ]( s, a ) -([ P h ˜ V h +1 ]( s, a )) 2 . For the first term, by the definition of linear MDP, it holds that [ P h ˜ V 2 h +1 ] ( s, a ) = ϕ ( s, a ) ⊤ ∫ S ˜ V 2 h +1 ( s ′ ) d ν h ( s ′ ) = ⟨ ϕ, ∫ S ˜ V 2 h +1 ( s ′ ) d ν h ( s ′ ) ⟩ . We can estimate β h = ∫ S ˜ V 2 h +1 ( s ′ ) d ν h ( s ′ ) by applying ridge regression. Below is the output of ridge regression with raw statistics without noise:

Algorithm 2 Differentially Private Variance-Aware Pessimistic Value Iteration (DP-V APVI)

1: Input: Dataset D = { ( s τ h , a τ h , r τ h , s τ h +1 ) } K,H τ,h =1 D ′ = { (¯ s τ h , ¯ a τ h , ¯ r τ h , ¯ s τ h +1 ) } K,H τ,h =1 . Budget for zCDP ρ . Failure probability δ . Universal constant C .

,

2: Initialization: Set ρ 0 ← ρ 5 H , ˜ V H +1 ( · ) ← 0 . Sample ϕ 1 ∼ N ( 0 , 2 H 4 ρ 0 I d ) , ϕ 2 , ϕ 3 ∼ N ( 0 , 2 H 2 ρ 0 I d ) K 1 , K 2 ← E 2 I d + 1 √ 2 ( Z + Z ⊤ ) , where Z i,j ∼ N ( 0 , 1 4 ρ 0 ) ( i.i.d. ) , E = ˜ O (√ Hd ρ ) . Set D ← ˜ O ( H 2 L κ + H 4 E √ d κ 3 / 2 + H 3 √ d ) .

3: for h = H,H -1 , . . . , 1 do

4: Set ˜ Σ h ← ∑ K τ =1 ϕ (¯ s τ h , ¯ a τ h ) ϕ (¯ s τ h , ¯ a τ h ) ⊤ + λI + K 1

5: Set ˜ β h ← ˜ Σ -1 h [ ∑ K τ =1 ϕ (¯ s τ h , ¯ a τ h ) · ˜ V h +1 (¯ s τ h +1 ) 2 + ϕ 1 ]

6: Set ˜ θ h ← ˜ Σ -1 h [ ∑ K τ =1 ϕ (¯ s τ h , ¯ a τ h ) · ˜ V h +1 (¯ s τ h +1 ) + ϕ 2 ]

7: Set [ ˜ Var h ˜ V h +1 ] ( · , · ) ← 〈 ϕ ( · , · ) , ˜ β h 〉 [ 0 , ( H -h +1) 2 ] -[〈 ϕ ( · , · ) , ˜ θ h 〉 [0 ,H -h +1] ] 2

8: Set ˜ σ h ( · , · ) 2 ← max { 1 , ˜ Var h ˜ V h +1 ( · , · ) }

10: Set ˜ w h ← ˜ Λ -1 h ( ∑ K τ =1 ϕ ( s τ h , a τ h ) · ( r τ h + ˜ V h +1 ( s τ h +1 ) ) / ˜ σ 2 h ( s τ h , a τ h ) + ϕ 3 )

9: Set ˜ Λ h ← ∑ K τ =1 ϕ ( s τ h , a τ h ) ϕ ( s τ h , a τ h ) ⊤ / ˜ σ 2 h ( s τ h , a τ h ) + λI + K 2

11: Set Γ h ( · , · ) ← C √ d · ( ϕ ( · , · ) ⊤ ˜ Λ -1 h ϕ ( · , · ) ) 1 / 2 + D K

12: Set ¯ Q h ( · , · ) ← ϕ ( · , · ) ⊤ ˜ w h -Γ h ( · , · )

13: Set ̂ Q h ( · , · ) ← min { ¯ Q h ( · , · ) , H -h +1 } +

14: Set ̂ π h ( · | · ) ← argmax π h 〈 ̂ Q h ( · , · ) , π h ( · | · ) 〉 A , ˜ V h ( · ) ← max π h 〈 ̂ Q h ( · , · ) , π h ( · | · ) 〉 A

16: Output: { ̂ π h } H h =1 .

15: end for

where definition of ¯ Σ h can be found in Appendix A. Instead of using the raw statistics, we replace them with private ones with Gaussian noises as in line 5. The second term is estimated similarly in line 6. The final estimator is defined as in line 8: ˜ σ h ( · , · ) 2 = max { 1 , ˜ Var h ˜ V h +1 ( · , · ) } . 9

Variance-weighted LSVI. Instead of directly applying LSVI [Jin et al., 2021], we can solve the variance-weighted LSVI (line 10). The result of variance-weighted LSVI with non-private statistics is shown below:

where definition of ̂ Λ h can be found in Appendix A. For the sake of differential privacy, we use private statistics instead and derive the ˜ w h as in line 10.

Our private pessimism. Notice that if we remove all the Gaussian noises we add, our DP-VAPVI (Algorithm 2) will degenerate to VAPVI [Yin et al., 2022]. We design a similar pessimistic penalty using private statistics (line 11), with additional D K accounting for the extra pessimism due to DP.

Main theorem. We state our main theorem about DP-VAPVI below, the proof sketch is deferred to Appendix D.1 and detailed proof is deferred to Appendix D due to space limit. Note that quantities M i , L, E can be found in Appendix A and briefly, L = ˜ O ( √ H 3 d/ρ ) , E = ˜ O ( √ Hd/ρ ) . For the sample complexity lower bound, within the practical regime where the privacy budget is not very small, max {M i } is dominated by max { ˜ O ( H 12 d 3 /κ 5 ) , ˜ O ( H 14 d/κ 5 ) } , which also appears in the sample complexity lower bound of VAPVI [Yin et al., 2022]. The σ 2 V ( s, a ) in Theorem 4.1 is defined as max { 1 , Var P h ( V )( s, a ) } for any V .

Theorem 4.1. DP-VAPVI (Algorithm 2) satisfies ρ -zCDP. Furthermore, let K be the number of episodes. Under the condition that K > max {M 1 , M 2 , M 3 , M 4 } and √ d > ξ , where ξ :=

all policy π simultaneously, the output ̂ π of DP-VAPVI satisfies

where Λ h = ∑ K k =1 ϕ ( s k h ,a k h ) · ϕ ( s k h ,a k h ) ⊤ σ 2 ˜ V h +1 ( s k h ,a k h ) + λI d , D = ˜ O ( H 2 L κ + H 4 E √ d κ 3 / 2 + H 3 √ d ) and ˜ O hides constants and Polylog terms.

In particular, define Λ ⋆ h = ∑ K k =1 ϕ ( s k h ,a k h ) · ϕ ( s k h ,a k h ) ⊤ σ 2 V ⋆ h +1 ( s k h ,a k h ) + λI d , we have with probability 1 -δ ,

Comparison to non-private counterpart VAPVI [Yin et al., 2022]. Plugging in the definition of L, E (Appendix A), under the meaningful case that the privacy budget is not very large, DH is dominated by ˜ O ( H 11 2 d/κ 3 2 √ ρ ) . According to Theorem 3.2 in [Yin et al., 2022], the sub-optimality bound of VAPVI is for sufficiently large K , with high probability, the output ̂ π satisfies:

Compared to our Theorem 4.1, the additional sub-optimality bound due to differential privacy is ˜ O ( H 11 2 d/κ 3 2 √ ρ · K ) = ˜ O ( H 11 2 d/κ 3 2 ϵ · K ) . 10 In the most popular regime where the privacy budget ρ or ϵ is a constant, the additional term due to differential privacy also appears as a lower order term.

Instance-dependent sub-optimality bound. Similar to DP-APVI (Algorithm 1), our DP-VAPVI (Algorithm 2) also enjoys instance-dependent sub-optimality bound. First, the main term in (10) improves PEVI [Jin et al., 2021] over O ( √ d ) on feature dependence. Also, our main term admits no explicit dependence on H , thus improves the sub-optimality bound of PEVI on horizon dependence. For more detailed discussions about our main term, we refer readers to [Yin et al., 2022].

Private and safe policy improvement. In addition to the sub-optimality bound (10), we have the so called oracle inequality (9). Therefore, the performance v ̂ π can be lower bounded by sup π [ v π -˜ O ( √ d · ∑ H h =1 E π [ √ ϕ ( · , · ) ⊤ Λ -1 h ϕ ( · , · ) ]) -DH K ] . When choosing π to be the optimal policy in the neighborhood of the behavior policy µ , our DP-VAPVI (Algorithm 2) sheds light on safe policy improvement with differential privacy guarantee.

5 Tightness of our results

We believe our bounds for offline RL with DP is tight. To the best of our knowledge, APVI and VAPVI provide the tightest bound under tabular MDP and linear MDP, respectively. The suboptimality bounds of our algorithms match these two in the main term, with some lower order additional terms. The leading terms are known to match multiple information-theoretical lower bounds for offline RL simultaneously (this was illustrated in Yin and Wang [2021b], Yin et al. [2022]), for this reason our bound cannot be improved in general. For the lower order terms, the dependence on sample complexity n and privacy budget ϵ : ˜ O ( 1 nϵ ) is optimal since policy learning is a special case of ERM problems and such dependence is optimal in DP-ERM. In addition, we believe the dependence on other parameters ( H,S,A,d ) in the lower order term is tight due to our special tricks as (3) and Lemma D.6.

6 Simulations

In this section, we carry out simulations to evaluate the performance of our DP-V APVI (Algorithm 2), and compare it with its non-private counterpart V APVI [Yin et al., 2022] and another pessimism-based algorithm PEVI [Jin et al., 2021] which does not have privacy guarantee.

Experimental setting. Weevaluate DP-VAPVI (Algorithm 2) on a synthetic linear MDP example that originates from the linear MDP in [Min et al., 2021, Yin et al., 2022] but with some modifications. 11 For details of the linear MDP setting, please refer to Appendix F. The two MDP instances we use both have horizon H = 20 . We compare different algorithms in figure 1(a), while in figure 1(b), we compare our DP-VAPVI with different privacy budgets. When doing empirical evaluation, we do not split the data for DP-V APVI or VAPVI and for DP-VAPVI, we run the simulation for 5 times and take the average performance.

0

200

400

600

800

1000

Number of episodes

1

2

3

4

5

6

Suboptimality gap

PEVI

VAPVI

DP-VAPVI, =1

DP-VAPVI, =25

(a) Compare different algorithms,

H

= 20

0

200

400

600

800

1000

Number of episodes

1

2

3

4

5

6

7

Suboptimality gap

VAPVI

DP-VAPVI,

=0.1

DP-VAPVI, =1

DP-VAPVI, =5

DP-VAPVI, =25

(b) Different privacy budgets,

H

= 20

Results and discussions. From Figure 1, we can observe that DP-VAPVI (Algorithm 2) performs slightly worse than its non-private version V APVI [Yin et al., 2022]. This is due to the fact that we add Gaussian noise to each count. However, as the size of dataset goes larger, the performance of DP-VAPVI will converge to that of VAPVI, which supports our theoretical conclusion that the cost of privacy only appears as lower order terms. For DP-VAPVI with larger privacy budget, the scale of noise will be smaller, thus the performance will be closer to VAPVI, as shown in figure 1(b). Furthermore, in most cases, DP-VAPVI still outperforms PEVI, which does not have privacy guarantee. This arises from our privitization of variance-aware LSVI instead of LSVI.

7 Conclusion and future works

In this work, we take the first steps towards the well-motivated task of designing private offline RL algorithms. We propose algorithms for both tabular MDPs and linear MDPs, and show that they enjoy instance-dependent sub-optimality bounds while guaranteeing differential privacy (either zCDP or pure DP). Our results highlight that the cost of privacy only appears as lower order terms, thus become negligible as the number of samples goes large.

Future extensions are numerous. We believe the technique in our algorithms (privitization of Bernstein-type pessimism and variance-aware LSVI) and the corresponding analysis can be used in online settings too to obtain tighter regret bounds for private algorithms. For the offline RL problems, we plan to consider more general function approximations and differentially private (deep) offline RL which will bridge the gap between theory and practice in offline RL applications. Many techniques we developed could be adapted to these more general settings.

Acknowledgments

The research is partially supported by NSF Awards # 2007117 and # 2048091. The authors would like to thank Ming Yin for helpful discussions.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems , pages 2312-2320, 2011.

Naman Agarwal and Karan Singh. The price of differential privacy for online learning. In International Conference on Machine Learning , pages 32-40. PMLR, 2017.

Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning , pages 463-474. PMLR, 2020.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 263-272. JMLR. org, 2017.

Borja Balle, Maziar Gomrokchi, and Doina Precup. Differentially private policy evaluation. In International Conference on Machine Learning , pages 2130-2138. PMLR, 2016.

Debabrota Basu, Christos Dimitrakakis, and Aristide Tossou. Differential privacy for multi-armed bandits: What is it and what is its cost? arXiv preprint arXiv:1905.12298 , 2019.

Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In ACM SIGACT Symposium on Theory of Computing , pages 123-132, 2021.

Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography Conference , pages 635-658. Springer, 2016.

Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning , pages 1283-1294. PMLR, 2020.

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium (USENIX Security 19) , pages 267-284, 2019.

Xiaoyu Chen, Kai Zheng, Zixin Zhou, Yunchang Yang, Wei Chen, and Liwei Wang. (locally) differentially private combinatorial semi-bandits. In International Conference on Machine Learning , pages 1757-1767. PMLR, 2020.

Herman Chernoff et al. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. The Annals of Mathematical Statistics , 23(4):493-507, 1952.

Sayak Ray Chowdhury and Xingyu Zhou. Differentially private regret minimization in episodic markov decision processes. arXiv preprint arXiv:2112.10599 , 2021.

Sayak Ray Chowdhury, Xingyu Zhou, and Ness Shroff. Adaptive control of differentially private linear quadratic systems. In 2021 IEEE International Symposium on Information Theory (ISIT) , pages 485-490. IEEE, 2021.

Chris Cundy and Stefano Ermon. Privacy-constrained policies via mutual information regularized policy gradients. arXiv preprint arXiv:2012.15019 , 2020.

Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning. In Advances in Neural Information Processing Systems , pages 5713-5723, 2017.

Cynthia Dwork and Guy N Rothblum. Concentrated differential privacy. arXiv preprint arXiv:1603.01887 , 2016.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pages 265-284. Springer, 2006.

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci. , 9(3-4):211-407, 2014.

Frederick Arthur Ficken. The simplex method of linear programming . Courier Dover Publications, 2015.

Pratik Gajane, Tanguy Urvoy, and Emilie Kaufmann. Corrupt bandits for preserving local privacy. In Algorithmic Learning Theory , pages 387-412. PMLR, 2018.

Minbo Gao, Tianle Xie, Simon S Du, and Lin F Yang. A provably efficient algorithm for linear markov decision process with low switching cost. arXiv preprint arXiv:2101.00494 , 2021.

Evrard Garcelon, Vianney Perchet, Ciara Pike-Burke, and Matteo Pirotta. Local differential privacy for regret minimization in reinforcement learning. Advances in Neural Information Processing Systems , 34, 2021.

Abhradeep Guha Thakurta and Adam Smith. (nearly) optimal algorithms for private online learning in full-information and bandit settings. Advances in Neural Information Processing Systems , 26, 2013.

Bingshan Hu, Zhiming Huang, and Nishant A Mehta. Optimal algorithms for private online learning in a stochastic environment. arXiv preprint arXiv:2102.07929 , 2021.

Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning , pages 4870-4879. PMLR, 2020a.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 2137-2143. PMLR, 2020b.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Hyeoneui Kim, Elizabeth Bell, Jihoon Kim, Amy Sitapati, Joe Ramsdell, Claudiu Farcas, Dexter Friedman, Stephanie Feudjio Feupe, and Lucila Ohno-Machado. iconcur: informed consent for clinical data and bio-sample use for research. Journal of the American Medical Informatics Association , 24(2):380-387, 2017.

Jonathan Lebensold, William Hamilton, Borja Balle, and Doina Precup. Actor critic with differentially private critic. arXiv preprint arXiv:1910.05876 , 2019.

Chonghua Liao, Jiafan He, and Quanquan Gu. Locally differentially private reinforcement learning for linear mixture markov decision processes. arXiv preprint arXiv:2110.10133 , 2021.

Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. In Uncertainty in Artificial Intelligence , 2019.

Paul Luyo, Evrard Garcelon, Alessandro Lazaric, and Matteo Pirotta. Differentially private exploration in reinforcement learning with linear representation. arXiv preprint arXiv:2112.01585 , 2021.

Sunil Madhow, Dan Qiao, Ming Yin, and Yu-Xiang Wang. Offline policy evaluation for reinforcement learning with adaptively collected data. arXiv preprint arXiv:2306.14063 , 2023.

Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penalization. Conference on Learning Theory , 2009.

Yifei Min, Tianhao Wang, Dongruo Zhou, and Quanquan Gu. Variance-aware off-policy evaluation with linear function approximation. Advances in neural information processing systems , 34, 2021.

George Nemhauser and Laurence Wolsey. Polynomial-time algorithms for linear programming. Integer and Combinatorial Optimization , pages 146-181, 1988.

Dung Daniel Ngo, Giuseppe Vietri, and Zhiwei Steven Wu. Improved regret for differentially private exploration in linear mdp. arXiv preprint arXiv:2202.01292 , 2022.

Hajime Ono and Tsubasa Takahashi. Locally private distributed reinforcement learning. arXiv preprint arXiv:2001.11718 , 2020.

Dan Qiao and Yu-Xiang Wang. Near-optimal differentially private reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 9914-9940. PMLR, 2023a.

Dan Qiao and Yu-Xiang Wang. Near-optimal deployment efficiency in reward-free reinforcement learning with linear function approximation. In The Eleventh International Conference on Learning Representations , 2023b.

Dan Qiao, Ming Yin, Ming Min, and Yu-Xiang Wang. Sample-efficient reinforcement learning with loglog(T) switching cost. In Proceedings of the 39th International Conference on Machine Learning , pages 18031-18061. PMLR, 2022.

Dan Qiao, Ming Yin, and Yu-Xiang Wang. Logarithmic switching cost in reinforcement learning beyond linear mdps. arXiv preprint arXiv:2302.12456 , 2023.

Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference , pages 147-163, 2017.

Rachel Redberg and Yu-Xiang Wang. Privately publishable per-instance privacy. Advances in Neural Information Processing Systems , 34, 2021.

Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. Deep reinforcement learning framework for autonomous driving. Electronic Imaging , 2017(19):70-76, 2017.

Roshan Shariff and Or Sheffet. Differentially private contextual linear bandits. Advances in Neural Information Processing Systems , 31, 2018.

Karthik Sridharan. A gentle introduction to concentration inequalities. Dept. Comput. Sci., Cornell Univ., Tech. Rep , 2002.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Aristide Charles Yedia Tossou and Christos Dimitrakakis. Achieving privacy in the adversarial multi-armed bandit. In Thirty-First AAAI Conference on Artificial Intelligence , 2017.

Tim Van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory , 60(7):3797-3820, 2014.

Giuseppe Vietri, Borja Balle, Akshay Krishnamurthy, and Steven Wu. Private reinforcement learning with pac and regret guarantees. In International Conference on Machine Learning , pages 97549764. PMLR, 2020.

Baoxiang Wang and Nidhi Hegde. Privacy-preserving q-learning with functional noise in continuous spaces. Advances in Neural Information Processing Systems , 32, 2019.

Ruosong Wang, Dean P Foster, and Sham M Kakade. What are the statistical limits of offline rl with linear function approximation? International Conference on Learning Representations , 2021a.

Tianhao Wang, Dongruo Zhou, and Quanquan Gu. Provably efficient reinforcement learning with linear function approximation under adaptivity constraints. Advances in Neural Information Processing Systems , 34:13524-13536, 2021b.

Tengyang Xie, Philip S Thomas, and Gerome Miklau. Privacy preserving off-policy evaluation. arXiv preprint arXiv:1902.00174 , 2019.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 2021a.

Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems , 2021b.

Jianyu Xu, Dan Qiao, and Yu-Xiang Wang. Doubly fair dynamic pricing. In International Conference on Artificial Intelligence and Statistics , pages 9941-9975. PMLR, 2023.

Ming Yin and Yu-Xiang Wang. Asymptotically efficient off-policy evaluation for tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 3948-3958. PMLR, 2020.

Ming Yin and Yu-Xiang Wang. Optimal uniform ope and model-based offline reinforcement learning in time-homogeneous, reward-free and task-agnostic settings. Advances in neural information processing systems , 2021a.

Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems , 34, 2021b.

Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal provable uniform convergence in offline policy evaluation for reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 1567-1575. PMLR, 2021.

Ming Yin, Yaqi Duan, Mengdi Wang, and Yu-Xiang Wang. Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. arXiv preprint arXiv:2203.05804 , 2022.

Andrea Zanette. Exponential lower bounds for batch reinforcement learning: Batch rl can be exponentially harder than online rl. In International Conference on Machine Learning , pages 12287-12297. PMLR, 2021.

Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems , 2021.

Zihao Zhang, Stefan Zohren, and Stephen Roberts. Deep reinforcement learning for trading. The Journal of Financial Data Science , 2(2):25-40, 2020.

Fuheng Zhao, Dan Qiao, Rachel Redberg, Divyakant Agrawal, Amr El Abbadi, and Yu-Xiang Wang. Differentially private linear sketches: Efficient implementations and applications. Advances in Neural Information Processing Systems , 35:12691-12704, 2022.

Kai Zheng, Tianle Cai, Weiran Huang, Zhenguo Li, and Liwei Wang. Locally differentially private (contextual) bandits learning. Advances in Neural Information Processing Systems , 33:1230012310, 2020.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR, 2021.

Xingyu Zhou. Differentially private reinforcement learning with linear function approximation. arXiv preprint arXiv:2201.07052 , 2022.

A Notation List

A.1 Notations for tabular MDP

A.2 Notations for linear MDP

B Extended related work

Online reinforcement learning under JDP or LDP. For online RL, some recent works analyze this setting under Joint Differential Privacy (JDP) , which requires the RL agent to minimize regret while handling user's raw data privately. Under tabular MDP, Vietri et al. [2020] design PUCB by revising UBEV [Dann et al., 2017]. Private-UCB-VI [Chowdhury and Zhou, 2021] results from UCBVI (with bonus-1) [Azar et al., 2017]. However, both works privatize Hoeffding type bonus, which lead to sub-optimal regret bound. Under linear MDP, Private LSVI-UCB [Ngo et al., 2022] and Privacy-Preserving LSVI-UCB [Luyo et al., 2021] are private versions of LSVI-UCB [Jin et al., 2020b], while LinOpt-VI-Reg [Zhou, 2022] and Privacy-Preserving UCRL-VTR [Luyo et al., 2021] generalize UCRL-VTR [Ayoub et al., 2020]. However, these works are usually based on the LSVI technique [Jin et al., 2020b] (unweighted ridge regression), which does not ensure optimal regret bound.

In addition to JDP, another common privacy guarantee for online RL is Local Differential Privacy (LDP) , LDP is a stronger definition of DP since it requires that the user's data is protected before the RL agent has access to it. Under LDP, Garcelon et al. [2021] reach a regret lower bound and design LDP-OBI which has matching regret upper bound. The result is generalized by Liao et al. [2021] to linear mixture setting. Later, Luyo et al. [2021] provide an unified framework for analyzing JDP and LDP under linear setting.

Some other differentially private learning algorithms. There are some other works about differentially private online learning [Guha Thakurta and Smith, 2013, Agarwal and Singh, 2017, Hu et al., 2021] and various settings of bandit [Shariff and Sheffet, 2018, Gajane et al., 2018, Basu et al., 2019, Zheng et al., 2020, Chen et al., 2020, Tossou and Dimitrakakis, 2017]. For the reinforcement learning setting, Wang and Hegde [2019] propose privacy-preserving Q-learning to protect the reward information. Ono and Takahashi [2020] study the problem of distributed reinforcement learning under LDP. Lebensold et al. [2019] present an actor critic algorithm with differentially private critic. Cundy and Ermon [2020] tackle DP-RL under the policy gradient framework. Chowdhury et al. [2021] consider the adaptive control of differentially private linear quadratic (LQ) systems. Zhao et al. [2022] designed differentially private linear sketch algorirthms.

Offline reinforcement learning under tabular MDP. Under tabular MDP, there are several works achieving optimal sub-optimality/sample complexity bounds under different coverage assumptions. For the problem of off-policy evaluation (OPE), Yin and Wang [2020] uses Tabular-MIS estimator to achieve asymptotic efficiency. In addition, the idea of uniform OPE is used to achieve the optimal sample complexity O ( H 3 /d m ϵ 2 ) [Yin et al., 2021] for non-stationary MDP and the optimal sample complexity O ( H 2 /d m ϵ 2 ) [Yin and Wang, 2021a] for stationary MDP, where d m is the lower bound for state-action occupancy. Such uniform convergence idea also supports some works regarding online exploration [Jin et al., 2020a, Qiao et al., 2022, Xu et al., 2023]. For offline RL with single concentrability assumption, Xie et al. [2021b] arrive at the optimal sample complexity O ( H 3 SC ⋆ /ϵ 2 ) . Recently, Yin and Wang [2021b] propose APVI which can lead to instance-dependent sub-optimality bound, which subsumes previous optimal results under several assumptions. Madhow et al. [2023] consider offline policy evaluation for adaptively collected datasets.

Offline reinforcement learning under linear MDP. Recently, many works focus on offline RL under linear representation. Jin et al. [2021] present PEVI which applies the idea of pessimistic value iteration (the idea originates from [Jin et al., 2020b]), and PEVI is provably efficient for offline RL under linear MDP. Yin et al. [2022] improve the sub-optimality bound in [Jin et al., 2021] by replacing LSVI by variance-weighted LSVI. Xie et al. [2021a] consider Bellman consistent pessimism for general function approximation, and their result improves the sample complexity in [Jin et al., 2021] by order O ( d ) (shown in Theorem 3.2). However, there is no improvement on horizon dependence. Zanette et al. [2021] propose a new offline actor-critic algorithm that naturally incorporates the pessimism principle. Besides, Wang et al. [2021a], Zanette [2021] study the statistical hardness of offline RL with linear representations by presenting exponential lower bounds. When relaxing the offline setting to low adaptive RL, Gao et al. [2021], Wang et al. [2021b], Qiao and Wang [2023b], Qiao et al. [2023] designed algorithms with low adaptivity.

C Proof of Theorem 3.4

C.1 Proof sketch

Since the whole proof for privacy guarantee is not very complex, we present it in Section C.2 below and only sketch the proof for suboptimality bound.

First of all, we bound the scale of noises we add to show that the ˜ n derived from (3) are close to real visitation numbers. Therefore, denoting the non-private empirical transition kernel by ̂ P (detailed definition in (15)), we can show that ∥ ˜ P -̂ P ∥ 1 and | √ Var ˜ P ( V ) -√ Var ̂ P ( V ) | are small.

Next, resulting from the conditional independence of ˜ V h +1 and ˜ P h , we apply Empirical Bernstein's inequality to get | ( ˜ P h -P h ) ˜ V h +1 | ≲ √ Var ˜ P ( ˜ V h +1 ) / ˜ n s h ,a h + SHE ρ / ˜ n s h ,a h . Together with our definition of private pessimism and the key lemma: extended value difference (Lemma E.7 and E.8), we can bound the suboptimality of our output policy ̂ π by:

Finally, we further bound the above suboptimality via replacing private statistics by non-private ones. Specifically, we replace ˜ n by n , ˜ P by P and ˜ V by V ⋆ . Due to (12), we have ∥ ˜ V -V ⋆ ∥ ∞ ≲ √ 1 n ¯ d m .

Together with the upper bounds of

The final bound using non-private statistics results from (12) and (13).

C.2 Proof of the privacy guarantee

The privacy guarantee of DP-APVI (Algorithm 1) is summarized by Lemma C.1 below.

Lemma C.1 (Privacy analysis of DP-APVI (Algorithm 1)) . DP-APVI (Algorithm 1) satisfies ρ -zCDP.

Proof of Lemma C.1. The ℓ 2 sensitivity of { n s h ,a h } is √ 2 H . According to Lemma 2.7, the Gaussian Mechanism used on { n s h ,a h } with σ 2 = 2 H ρ satisfies ρ 2 -zCDP. Similarly, the Gaussian Mechanism used on { n s h ,a h ,s h +1 } with σ 2 = 2 H ρ also satisfies ρ 2 -zCDP. Combining these two results, due to the composition of zCDP (Lemma E.16), the construction of { n ′ } satisfies ρ -zCDP. Finally, DP-APVI satisfies ρ -zCDP because the output ̂ π is post processing of { n ′ } .

C.3 Proof of the sub-optimality bound

C.3.1 Utility analysis

First of all, the following Lemma C.2 gives a high probability bound for | n ′ -n | .

Lemma C.2. Let E ρ = 2 √ 2 σ √ log 4 HS 2 A δ = 4 √ H log 4 HS 2 A δ ρ , then with probability 1 -δ , for all s h , a h , s h +1 , it holds that

Proof of Lemma C.2. The inequalities directly result from the concentration inequality of Gaussian distribution and a union bound.

According to the utility analysis above, we have the following Lemma C.3 giving a high probability bound for | ˜ n -n | .

Lemma C.3. Under the high probability event in Lemma C.2, for all s h , a h , s h +1 , it holds that

Proof of Lemma C.3. When the event in Lemma C.2 holds, the original counts { n s h ,a h ,s ′ } s ′ ∈S is a feasible solution to the optimization problem, which means that

Due to the second part of (14), it holds that for any s h , a h , s h +1 ,

|

˜

n

s

h

,a

h

,s

h

+1

-

n

s

h

,a

h

,s

h

+1

˜

n

s

h

,a

h

,s

h

+1

-

n

′

s

h

,a

h

,s

h

+1

|

+

|

n

′

s

h

,a

h

,s

h

+1

-

n

s

h

,a

h

,s

h

+1

For the second part, because of the constraints in the optimization problem, it holds that

Due to the first part of (14), it holds that for any s h , a h ,

Let the non-private empirical estimate be:

if n s h ,a h > 0 and ̂ P h ( s ′ | s h , a h ) = 1 S otherwise. We will show that the private transition kernel ˜ P is close to ̂ P by the Lemma C.4 and Lemma C.5 below.

Lemma C.4. Under the high probability event of Lemma C.3, for s h , a h , if ˜ n s h ,a h ≥ 3 E ρ , it holds that

Proof of Lemma C.4. If ˜ n s h ,a h ≥ 3 E ρ and the conclusion in Lemma C.3 hold, we have

The second inequality is because ˜ n s h ,a h ,s ′ -E ρ ˜ n s h ,a h + E ρ ≤ n s h ,a h ,s ′ n s h ,a h ≤ ˜ n s h ,a h ,s ′ + E ρ ˜ n s h ,a h -E ρ and ˜ n s h ,a h ,s ′ + E ρ ˜ n s h ,a h -E ρ -˜ n s h ,a h ,s ′ ˜ n s h ,a h ≥ ˜ n s h ,a h ,s ′ ˜ n s h ,a h -˜ n s h ,a h ,s ′ -E ρ ˜ n s h ,a h + E ρ . The third inequality is because of Lemma E.6. The last inequality is because ˜ n s h ,a h ≥ 3 E ρ .

E

ρ

.

Lemma C.5. Let V ∈ R S be any function with ∥ V ∥ ∞ ≤ H , under the high probability event of Lemma C.3, for s h , a h , if ˜ n s h ,a h ≥ 3 E ρ , it holds that

Proof of Lemma C.5. For s h , a h such that ˜ n s h ,a h ≥ 3 E ρ , we use ˜ P ( · ) and ̂ P ( · ) instead of ˜ P h ( ·| s h , a h ) and ̂ P h ( ·| s h , a h ) for simplicity. Because of Lemma C.4, we have

Therefore, it holds that

(19)

The second inequality is due to the definition of variance.

C.3.2 Validity of our pessimistic penalty

Now we are ready to present the key lemma (Lemma C.6) below to justify our use of Γ as the pessimistic penalty.

Lemma C.6. Under the high probability event of Lemma C.3, with probability 1 -δ , for any s h , a h , if ˜ n s h ,a h ≥ 3 E ρ (which implies n s h ,a h > 0 ), it holds that

where ˜ V is the private version of estimated V function, which appears in Algorithm 1 and ι = log( HSA/δ ) .

Proof of Lemma C.6.

where the third inequality is due to Lemma C.4.

Next, recall ̂ π h +1 in Algorithm 1 is computed backwardly therefore only depends on sample tuple from time h +1 to H . As a result, ˜ V h +1 = ⟨ Q h +1 , ̂ π h +1 ⟩ also only depends on the sample tuple from time h +1 to H and some Gaussian noise that is independent to the offline dataset. On the other side, by the definition, ̂ P h only depends on the sample tuples from time h to h +1 . Therefore ˜ V h +1 and ̂ P h are Conditionally independent (This trick is also used in [Yin et al., 2021] and [Yin and Wang,

2021b]), by Empirical Bernstein's inequality (Lemma E.4) and a union bound, with probability 1 -δ , for all s h , a h such that ˜ n s h ,a h ≥ 3 E ρ ,

Therefore, we have

The second and forth inequality is because when ˜ n s h ,a h ≥ 3 E ρ , n s h ,a h ≥ 2 ˜ n s h ,a h 3 . Specifically, these two inequalities are also because usually we only care about the case when SE ρ ≥ 1 , which is equivalent to ρ being not very large. The third inequality is due to Lemma C.5. The last inequality is due to Lemma C.3.

Note that the previous Lemmas rely on the condition that ˜ n is not very small ( ˜ n s h ,a h ≥ 3 E ρ ). Below we state the Multiplicative Chernoff bound (Lemma C.7 and Remark C.8) to show that under our condition in Theorem 3.4, for ( s h , a h ) ∈ C h , ˜ n s h ,a h will be larger than 3 E ρ with high probability.

Lemma C.7 (Lemma B.1 in [Yin and Wang, 2021b]) . For any 0 < δ < 1 , there exists an absolute constant c 1 such that when total episode n > c 1 · 1 / ¯ d m · log( HSA/δ ) , then with probability 1 -δ , ∀ h ∈ [ H ]

Furthermore, we denote

then equivalently P ( E ) > 1 -δ .

In addition, we denote

then similarly P ( E ′ ) > 1 -δ .

Remark C.8. According to Lemma C.7, for any failure probability δ , there exists some constant c 1 > 0 such that when n ≥ c 1 E ρ · ι ¯ d m , with probability 1 -δ , for all ( s h , a h ) ∈ C h , n s h ,a h ≥ 4 E ρ . Therefore, under the condition of Theorem 3.4 and the high probability events in Lemma C.3 and Lemma C.7, it holds that for all ( s h , a h ) ∈ C h , ˜ n s h ,a h ≥ 3 E ρ while for all ( s h , a h ) / ∈ C h , ˜ n s h ,a h ≤ E ρ .

Lemma C.9. Define ( T h V )( · , · ) := r h ( · , · ) + ( P h V )( · , · ) for any V ∈ R S . Note ̂ π , Q h , ˜ V h are defined in Algorithm 1 and denote ξ h ( s, a ) = ( T h ˜ V h +1 )( s, a ) -Q h ( s, a ) . Then it holds that

Furthermore, (26) holds for all V h ( s ) -V ̂ h ( s ) .

Proof of Lemma C.9. Lemma C.9 is a direct corollary of Lemma E.8 with π = π ⋆ , ̂ Q h = Q h , ̂ V h = ˜ V h and ̂ π = ̂ π in Algorithm 1, we can obtain this result since by the definition of ̂ π in Algorithm 1, ⟨ Q h ( s h , · ) , π h ( ·| s h ) -̂ π h ( ·| s h ) ⟩ ≤ 0 . The proof for V π ⋆ h ( s ) -V ̂ π h ( s ) is identical.

Next we prove the asymmetric bound for ξ h , which is the key to the proof.

Lemma C.10 (Private version of Lemma D.6 in [Yin and Wang, 2021b]) . Denote ξ h ( s, a ) = ( T h ˜ V h +1 )( s, a ) -Q h ( s, a ) , where ˜ V h +1 and Q h are the quantities in Algorithm 1 and T h ( V ) := r h + P h · V for any V ∈ R S . Then under the high probability events in Lemma C.3 and Lemma C.6, for any h, s h , a h such that ˜ n s h ,a h > 3 E ρ , we have

where ι = log( HSA/δ ) .

Proof of Lemma C.10. The first inequality: We first prove ξ h ( s h , a h ) ≥ 0 for all ( s h , a h ) , such that ˜ n s h ,a h ≥ 3 E ρ .

Indeed, if ̂ Q p h ( s h , a h ) < 0 , then Q h ( s h , a h ) = 0 . In this case, ξ h ( s h , a h ) = ( T h ˜ V h +1 )( s h , a h ) ≥ 0 (note ˜ V h ≥ 0 by the definition). If ̂ Q p h ( s h , a h ) ≥ 0 , then by definition Q h ( s h , a h ) = min { ̂ Q p h ( s h , a h ) , H -h +1 } + ≤ ̂ Q p h ( s h , a h ) and this implies

where the second inequality uses Lemma C.6, and the last equation uses Line 5 of Algorithm 1.

The second inequality: Then we prove ξ h ( s h , a h ) ≤ 2 √ 2Var ˜ P h ( ·| s h ,a h ) ( ˜ V h +1 ( · )) · ι ˜ n s h ,a h -E ρ + 32 SHE ρ · ι ˜ n s h ,a h for all ( s h , a h ) such that ˜ n s h ,a h ≥ 3 E ρ .

First, since by construction ˜ V h ≤ H -h +1 for all h ∈ [ H ] , this implies

which is because r h ≤ 1 and ˜ P h is a probability distribution. Therefore, we have the equivalent definition

Then it holds that

The proof is complete by combining the two parts.

C.3.3 Reduction to augmented absorbing MDP

Before we prove the theorem, we need to construct an augmented absorbing MDP to bridge ˜ V and V ⋆ . According to Line 5 in Algorithm 1, the locations with ˜ n s h ,a h ≤ E ρ is heavily penalized with penalty of order ˜ O ( H ) . Therefore we can prove that under the high probability event in Remark C.8, d ̂ π h ( s h , a h ) > 0 only if d µ h ( s h , a h ) > 0 by induction, where ̂ π is the output of Algorithm 1. The conclusion holds for h = 1 . Assume it holds for some h > 1 that d ̂ π h ( s h , a h ) > 0 only if d µ h ( s h , a h ) > 0 , then for any s h +1 ∈ S such that d ̂ π h +1 ( s h +1 ) > 0 , it holds that d µ h +1 ( s h +1 ) > 0 , which leads to the conclusion that d ̂ π h +1 ( s h +1 , a h +1 ) > 0 only if d µ h +1 ( s h +1 , a h +1 ) > 0 . To summarize, we have

Let us define M † by adding one absorbing state s † h for all h ∈ { 2 , . . . , H } , therefore the augmented state space S † = S ∪ { s † h } and the transition and reward is defined as follows: (recall C h := { ( s h , a h ) : d µ h ( s h , a h ) > 0 } )

and we further define for any π ,

where E † means taking expectation under the absorbing MDP M † . Note that because π ⋆ and π are fully covered by µ (27), it holds that

Define ( T † h V )( · , · ) := r † h ( · , · ) + ( P † h V )( · , · ) for any V ∈ R S +1 . Note ̂ π , Q h , ˜ V h are defined in Algorithm 1 (we extend the definition by letting ˜ V h ( s † h ) = 0 and Q h ( s † h , · ) = 0 ) and denote ξ † h ( s, a ) = ( T † h ˜ V h +1 )( s, a ) -Q h ( s, a ) . Using identical proof to Lemma C.9, we have

where V † π 1 is defined in (28). Furthermore, (30) holds for all V † π ⋆ h ( s ) -V † ̂ π h ( s ) .

C.3.4 Finalize our result with non-private statistics

For those ( s h , a h ) ∈ C h , ξ † h ( s h , a h ) = r h ( s h , a h ) + P h ˜ V h +1 ( s h , a h ) -Q h ( s h , a h ) = ξ h ( s h , a h ) . For those ( s h , a h ) / ∈ C h or s h = s † h , we have ξ † h ( s h , a h ) = 0 .

̂

Therefore, by (30) and Lemma C.10, under the high probability events in Lemma C.3, Lemma C.6 and Lemma C.7, we have for all t ∈ [ H ] , s ∈ S ( S does not include the absorbing state s † t ),

The second and third inequality are because of Lemma C.10, Remark C.8 and the the fact that either ξ † = 0 or ξ † = ξ while ( s h , a h ) ∈ C h . The forth inequality is due to Lemma C.3. The fifth inequality is because of Remark C.8. The last inequality is by Lemma C.7.

Below we present a crude bound of ∣ ∣ ∣ V † π ⋆ t ( s ) -˜ V t ( s ) ∣ ∣ ∣ , which can be further used to bound the main term in the main result.

Lemma C.11 (Self-bounding, private version of Lemma D.7 in [Yin and Wang, 2021b]) . Under the high probability events in Lemma C.3, Lemma C.6 and Lemma C.7, it holds that for all t ∈ [ H ] and s ∈ S , √

where ¯ d m is defined in Theorem 3.4.

Proof of Lemma C.11. According to (31), since Var ˜ P h ( ·| s h ,a h ) ( ˜ V h +1 ( · )) ≤ H 2 , we have for all t ∈ [ H ] , √

Next, apply Lemma E.7 by setting π = ̂ π , π ′ = π ⋆ , ̂ Q = Q , ̂ V = ˜ V under M † , then we have

Also, apply Lemma E.7 by setting π = π ′ = ̂ π , ̂ Q = Q , ̂ V = ˜ V under M † , then we have

The proof is complete by combing (32), (33) and (34).

Now we are ready to bound √ Var ˜ P h ( ·| s h ,a h ) ( ˜ V h +1 ( · )) by √ Var P h ( ·| s h ,a h ) ( V † ⋆ h +1 ( · )) . Under the high probability events in Lemma C.3, Lemma C.6 and Lemma C.7, with probability 1 -δ , it holds that for all ( s h , a h ) ∈ C h ,

The second inequality is because of Lemma C.11. The third inequality is due to Lemma C.5. The forth inequality comes from Lemma C.3 and Remark C.8. The fifth inequality holds with probability 1 -δ because of Lemma E.5 and a union bound.

Finally, by plugging (35) into (31) and averaging over s 1 , we finally have with probability 1 -4 δ ,

where ˜ O absorbs constants and Polylog terms. The first equation is due to (29). The first inequality is because of (31). The second inequality comes from (35) and our assumption that n · ¯ d m ≥ c 1 H 2 . The second equation uses the fact that d π ⋆ h ( s h , a h ) = d † π ⋆ h ( s h , a h ) , for all ( s h , a h ) . The last equation is because for any ( s h , a h , s h +1 ) such that d π ⋆ h ( s h , a h ) > 0 and P h ( s h +1 | s h , a h ) > 0 , V † ⋆ h +1 ( s h +1 ) = V ⋆ h +1 ( s h +1 ) .

C.4 Put everything together

Combining Lemma C.1 and (36), the proof of Theorem 3.4 is complete.

D Proof of Theorem 4.1

D.1 Proof sketch

Since the whole proof for privacy guarantee is not very complex, we present it in Section D.2 below and only sketch the proof for suboptimality bound.

First of all, by extended value difference (Lemma E.7 and E.8), we can convert bounding the suboptimality gap of v ⋆ -v ̂ π to bounding ∑ H h =1 E π [Γ h ( s h , a h )] , given that | ( T h ˜ V h +1 -˜ T h ˜ V h +1 )( s, a ) | ≤ Γ h ( s, a ) for all s, a, h . To bound ( T h ˜ V h +1 -˜ T h ˜ V h +1 )( s, a ) , according to our analysis about the upper bound of the noises we add, we can decompose ( T h ˜ V h +1 -˜ T h ˜ V h +1 )( s, a ) to lower order terms ( ˜ O ( 1 K ) ) and the following key quantity:

For the term above, we prove an upper bound of ∥ ∥ ∥ σ 2 ˜ V h +1 -˜ σ 2 h ∥ ∥ ∥ ∞ , so we can convert ˜ σ 2 h to σ 2 ˜ V h +1 . Next, since Var [ r τ h + ˜ V h +1 ( s τ h +1 ) -( T h ˜ V h +1 ) ( s τ h , a τ h ) | s τ h , a τ h ] ≈ σ 2 ˜ V h +1 , we can apply Bernstein's inequality for self-normalized martingale (Lemma E.10) as in Yin et al. [2022] for deriving tighter bound.

Finally, we replace the private statistics by non-private ones. More specifically, we convert σ 2 ˜ V h +1 to σ ⋆ 2 h ( Λ -1 h to Λ ⋆ -1 h ) by combining the crude upper bound of ∥ ∥ ∥ ˜ V -V ⋆ ∥ ∥ ∥ ∞ and matrix concentrations.

D.2 Proof of the privacy guarantee

The privacy guarantee of DP-VAPVI (Algorithm 2) is summarized by Lemma D.1 below.

Lemma D.1 (Privacy analysis of DP-VAPVI (Algorithm 2)) . DP-VAPVI (Algorithm 2) satisfies ρ -zCDP.

Proof of Lemma D.1. For ∑ K τ =1 ϕ (¯ s τ h , ¯ a τ h ) · ˜ V h +1 (¯ s τ h +1 ) 2 , the ℓ 2 sensitivity is 2 H 2 . For ∑ K τ =1 ϕ (¯ s τ h , ¯ a τ h ) · ˜ V h +1 (¯ s τ h +1 ) and ∑ K τ =1 ϕ ( s τ h , a τ h ) · ( r τ h + ˜ V h +1 ( s τ h +1 ) ) / ˜ σ 2 h ( s τ h , a τ h ) , the ℓ 2 sensitivity is 2 H . Therefore according to Lemma 2.7, the use of Gaussian Mechanism (the additional noises ϕ 1 , ϕ 2 , ϕ 3 ) ensures ρ 0 -zCDP for each counter. For ∑ K τ =1 ϕ (¯ s τ h , ¯ a τ h ) ϕ (¯ s τ h , ¯ a τ h ) ⊤ + λI and ∑ K τ =1 ϕ ( s τ h , a τ h ) ϕ ( s τ h , a τ h ) ⊤ / ˜ σ 2 h ( s τ h , a τ h ) + λI , according to Appendix D in [Redberg and Wang, 2021], the per-instance ℓ 2 sensitivity is

Therefore the use of Gaussian Mechanism (the additional noises K 1 , K 2 ) also ensures ρ 0 -zCDP for each counter. 12 Combining these results, according to Lemma E.17, the whole algorithm satisfies 5 Hρ 0 = ρ -zCDP.

D.3 Proof of the sub-optimality bound

D.3.1 Utility analysis and some preparation

We begin with the following high probability bound of the noises we add.

Lemma D.2 (Utility analysis) . Let L = 2 H √ d ρ 0 log( 10 Hd δ ) = 2 H √ 5 Hd log( 10 Hd δ ) ρ and E = √ 2 d ρ 0 ( 2 + ( log(5 c 1 H/δ ) c 2 d ) 2 3 ) = √ 10 Hd ρ ( 2 + ( log(5 c 1 H/δ ) c 2 d ) 2 3 ) for some universal constants c 1 , c 2 . Then with probability 1 -δ , the following inequalities hold simultaneously:

For all h ∈ [ H ] , ∥ ϕ 1 ∥ 2 ≤ HL, ∥ ϕ 2 ∥ 2 ≤ L, ∥ ϕ 3 ∥ 2 ≤ L. For all h ∈ [ H ] , K 1 , K 2 are symmetric and positive definite and ∥ K i ∥ 2 ≤ E, i ∈ { 1 , 2 } . (38)

Proof of Lemma D.2. The second line of (38) results from Lemma 19 in [Redberg and Wang, 2021] and Weyl's Inequality. The first line of (38) directly results from the concentration inequality for Guassian distribution and a union bound.

Define the Bellman update error ζ h ( s, a ) := ( T h ˜ V h +1 )( s, a ) -̂ Q h ( s, a ) and recall ̂ π h ( s ) = argmax π h ⟨ ̂ Q h ( s, · ) , π h ( · | s ) ⟩ A , then because of Lemma E.8,

Define ˜ T h ˜ V h +1 ( · , · ) = ϕ ( · , · ) ⊤ ˜ w h . Then similar to Lemma C.10, we have the following lemma showing that in order to bound the sub-optimality, it is sufficient to bound the pessimistic penalty.

Lemma D.3 (Lemma C.1 in [Yin et al., 2022]) . Suppose with probability 1 -δ , it holds for all s, a, h ∈ S × A × [ H ] that | ( T h ˜ V h +1 -˜ T h ˜ V h +1 )( s, a ) | ≤ Γ h ( s, a ) , then it implies ∀ s, a, h ∈ S × A × [ H ] , 0 ≤ ζ h ( s, a ) ≤ 2Γ h ( s, a ) . Furthermore, with probability 1 -δ , it holds for any policy π simultaneously,

Proof of Lemma D.3. We first show given | ( T h ˜ V h +1 -˜ T h ˜ V h +1 )( s, a ) | ≤ Γ h ( s, a ) , then 0 ≤ ζ h ( s, a ) ≤ 2Γ h ( s, a ) , ∀ s, a, h ∈ S × A × [ H ] .

Step 1: The first step is to show 0 ≤ ζ h ( s, a ) , ∀ s, a, h ∈ S × A × [ H ] .

Indeed, if ¯ Q h ( s, a ) ≤ 0 , then by definition ̂ Q h ( s, a ) = 0 and therefore ζ h ( s, a ) := ( T h ˜ V h +1 )( s, a ) -̂ Q h ( s, a ) = ( T h ˜ V h +1 )( s, a ) ≥ 0 . If ¯ Q h ( s, a ) > 0 , then ̂ Q h ( s, a ) ≤ ¯ Q h ( s, a ) and

Step 2: The second step is to show ζ h ( s, a ) ≤ 2Γ h ( s, a ) , ∀ s, a, h ∈ S × A × [ H ] .

Under the assumption that | ( T h ˜ V h +1 -˜ T h ˜ V h +1 )( s, a ) | ≤ Γ h ( s, a ) , we have

which implies that ̂ Q h ( s, a ) = max( ¯ Q h ( s, a ) , 0) . Therefore, it holds that

For the last statement, denote F := { 0 ≤ ζ h ( s, a ) ≤ 2Γ h ( s, a ) , ∀ s, a, h ∈ S × A × [ H ] } . Note conditional on F , then by (39), V π 1 ( s ) -V ̂ π 1 ( s ) ≤ ∑ H h =1 2 · E π [Γ h ( s h , a h ) | s 1 = s ] holds for any

policy π almost surely. Therefore,

which finishes the proof.

D.3.2 Bound the pessimistic penalty

By Lemma D.3, it remains to bound | ( T h ˜ V h +1 )( s, a ) -( ˜ T h ˜ V h +1 )( s, a ) | . Suppose w h is the coefficient corresponding to the T h ˜ V h +1 (such w h exists by Lemma E.14), i.e. T h ˜ V h +1 = ϕ ⊤ w h , and recall ( ˜ T h ˜ V h +1 )( s, a ) = ϕ ( s, a ) ⊤ ˜ w h , then:

where ̂ Λ h = ˜ Λ h -K 2 = ∑ K τ =1 ϕ ( s τ h , a τ h ) ϕ ( s τ h , a τ h ) ⊤ / ˜ σ 2 h ( s τ h , a τ h ) + λI .

Term (ii) can be handled by the following Lemma D.4

Lemma D.4. Recall κ in Assumption 2.2. Under the high probability event in Lemma D.2, suppose K ≥ max { 512 H 4 · log ( 2 Hd δ ) κ 2 , 4 λH 2 κ } , then with probability 1 -δ , for all s, a, h ∈ S × A × [ H ] , it holds that

Proof of Lemma D.4. Define ˜ Λ p h = E µ,h [ ˜ σ -2 h ( s, a ) ϕ ( s, a ) ϕ ( s, a ) ⊤ ] . Then because of Assumption 2.2 and ˜ σ h ≤ H , it holds that λ min ( ˜ Λ p h ) ≥ κ H 2 . Therefore, due to Lemma E.13, we have with

probability 1 -δ ,

The first inequality is because of Cauchy-Schwarz inequality. The second inequality holds with probability 1 -δ due to Lemma E.13 and a union bound. The third inequality holds because √ a ⊤ · A · a ≤ √ ∥ a ∥ 2 ∥ A ∥ 2 ∥ a ∥ 2 = ∥ a ∥ 2 √ ∥ A ∥ 2 . The last inequality arises from ∥ ( ˜ Λ p h ) -1 ∥ = λ max (( ˜ Λ p h ) -1 ) = λ -1 min ( ˜ Λ p h ) ≤ H 2 κ .

The difference between ˜ Λ -1 h and ̂ Λ -1 h can be bounded by the following Lemma D.5

Lemma D.5. Under the high probability event in Lemma D.2, suppose K ≥ 128 H 4 log 2 dH δ κ 2 , then with probability 1 -δ , for all h ∈ [ H ] , it holds that ∥ ̂ Λ -1 h -˜ Λ -1 h ∥ ≤ 4 H 4 E/κ 2 K 2 .

Proof of Lemma D.5. First of all, we have

The first inequality is because ∥ A · B ∥ ≤ ∥ A ∥ · ∥ B ∥ . The second inequality is due to Lemma D.2. Let ̂ Λ ′ h = 1 K ̂ Λ h , then because of Lemma E.12, with probability 1 -δ , it holds that for all h ∈ [ H ] ,

which implies that when K ≥ 128 H 4 log 2 dH δ κ 2 , it holds that (according to Weyl's Inequality)

Under this high probability event, we have λ min ( ̂ Λ h ) ≥ Kκ 2 H 2 and therefore λ min ( ˜ Λ h ) ≥ λ min ( ̂ Λ h ) ≥ Kκ 2 H 2 . Plugging these two results into (41), we have

Then we can bound term (iii) by the following Lemma D.6

Lemma D.6. Suppose K ≥ max { 128 H 4 log 2 dH δ κ 2 , √ 2 L √ dκ } , under the high probability events in Lemma D.2 and Lemma D.5, it holds that for all s, a, h ∈ S × A × [ H ] ,

Proof of Lemma D.6. First of all, the left hand side is bounded by

due to Lemma D.5. Then the left hand side can be further bounded by

The first inequality is because ∥ a ∥ 2 = √ a ⊤ a = √ Tr ( aa ⊤ ) . The second inequality is due to Cauchy-Schwarz inequality. The third inequality is because for positive definite matrix A , it holds that Tr ( A ) = ∑ d i =1 λ i ( A ) ≤ dλ max ( A ) . The equation is because for symmetric, positive definite matrix A , ∥ A ∥ 2 = λ max ( A ) . The forth inequality is due to ∥ A · B ∥ ≤ ∥ A ∥ · ∥ B ∥ . The fifth inequality is because of Lemma D.2, Lemma D.5 and the statement in the proof of Lemma D.5 that λ min ( ˜ Λ h ) ≥ Kκ 2 H 2 . The last inequality uses the assumption that K ≥ √ 2 L √ dκ .

Now the remaining part is term (i), we have

We are able to bound term (iv) by the following Lemma D.7.

Lemma D.7. Recall κ in Assumption 2.2. Under the high probability event in Lemma D.2, suppose K ≥ max { 512 H 4 · log ( 2 Hd δ ) κ 2 , 4 λH 2 κ } , then with probability 1 -δ , for all s, a, h ∈ S × A × [ H ] ,

Proof of Lemma D.7. Recall T h ˜ V h +1 = ϕ ⊤ w h and apply Lemma E.13, we obtain with probability 1 -δ , for all s, a, h ∈ S × A × [ H ] ,

where ˜ Λ p h := E µ,h [ ˜ σ h ( s, a ) -2 ϕ ( s, a ) ϕ ( s, a ) ⊤ ] . The first inequality applies Cauchy-Schwarz inequality. The second inequality holds with probability 1 -δ due to Lemma E.13 and a union bound. The third inequality uses √ a ⊤ · A · a ≤ √ ∥ a ∥ 2 ∥ A ∥ 2 ∥ a ∥ 2 = ∥ a ∥ 2 √ ∥ A ∥ 2 and ∥ w h ∥ ≤ 2 H √ d . Finally, as λ min ( ˜ Λ p h ) ≥ κ max h,s,a ˜ σ h ( s,a ) 2 ≥ κ H 2 implies ∥ ∥ ∥ ( ˜ Λ p h ) -1 ∥ ∥ ∥ ≤ H 2 κ , the last inequality holds.

For term (v), denote: x τ = ϕ ( s τ h ,a τ h ) ˜ σ h ( s τ h ,a τ h ) , η τ = ( r τ h + ˜ V h +1 ( s τ h +1 ) -( T h ˜ V h +1 ) ( s τ h , a τ h ) ) / ˜ σ h ( s τ h , a τ h ) , then by Cauchy-Schwarz inequality, it holds that for all h, s, a ∈ [ H ] ×S × A ,

We bound √ ϕ ( s, a ) ⊤ ̂ Λ -1 h ϕ ( s, a ) by √ ϕ ( s, a ) ⊤ ˜ Λ -1 h ϕ ( s, a ) using the following Lemma D.8. √

Lemma D.8. Suppose K ≥ max { 128 H 4 log 2 dH δ κ 2 , 2 L √ dκ } , under the high probability events in Lemma D.2 and Lemma D.5, it holds that for all s, a, h ∈ S × A × [ H ] ,

The first inequality uses | a ⊤ Aa | ≤ ∥ a ∥ 2 2 · ∥ A ∥ . The second inequality is because for a, b ≥ 0 , √ a + √ b ≥ √ a + b . The last inequality uses Lemma D.5.

Remark D.9. Similarly, under the same assumption in Lemma D.8, we also have for all s, a, h ∈ S × A × [ H ] ,

D.3.3 An intermediate result: bounding the variance

Before we handle ∥ ∥ ∥ ∑ K τ =1 x τ η τ ∥ ∥ ∥ ̂ Λ -1 h , we first bound sup h ∥ ∥ ∥˜ σ 2 h -σ 2 ˜ V h +1 ∥ ∥ ∥ ∞ by the following Lemma D.10.

Lemma D.10 (Private version of Lemma C.7 in [Yin et al., 2022]) . Recall the definition of ˜ σ h ( · , · ) 2 = max { 1 , ˜ Var h ˜ V h +1 ( · , · ) } in Algorithm 2 where [ ˜ Var h ˜ V h +1 ] ( · , · ) = 〈 ϕ ( · , · ) , ˜ β h 〉 [ 0 , ( H -h +1) 2 ] -[〈 ϕ ( · , · ) , ˜ θ h 〉 [0 ,H -h +1] ] 2 ( ˜ β h and ˜ θ h are defined in Algorithm 2) and σ ˜ V h +1 ( · , · ) 2 := max { 1 , Var P h ˜ V h +1 ( · , · ) } . Suppose K ≥ max { 512 log ( 2 Hd δ ) κ 2 , 4 λ κ , 128 log 2 dH δ κ 2 , √ 2 L H √ dκ } and K ≥ max { 4 L 2 H 2 d 3 κ , 32 E 2 d 2 κ 2 , 16 λ 2 d 2 κ } , under the high probability event in Lemma D.2, it holds that with probability 1 -6 δ ,

Proof of Lemma D.10. Step 1: The first step is to show for all h, s, a ∈ [ H ] ×S×A , with probability 1 -3 δ ,

Proof of Step 1. We can bound the left hand side by the following decomposition:

where ¯ Σ h = ˜ Σ h -K 1 = ∑ K τ =1 ϕ (¯ s τ h , ¯ a τ h ) ϕ (¯ s τ h , ¯ a τ h ) ⊤ + λI .

Similar to the proof in Lemma D.5, when K ≥ max { 128 log 2 dH δ κ 2 , √ 2 L H √ dκ } , it holds that with probability 1 -δ , for all h ∈ [ H ] ,

(The only difference to Lemma D.5 is here E µ,h [ ϕ ( s, a ) ϕ ( s, a ) ⊤ ] ≥ κ .)

Under this high probability event, for term (2), it holds that for all h, s, a ∈ [ H ] ×S × A ,

For term (3) , similar to Lemma D.6, we have for all h, s, a ∈ [ H ] ×S × A ,

(The only difference to Lemma D.6 is that here ˜ V h +1 ( s ) 2 ≤ H 2 , ∥ ϕ 1 ∥ 2 ≤ HL , ∥ ∥ ∥ ˜ Σ -1 h ∥ ∥ ∥ 2 ≤ 2 Kκ and ∥ ∥ ∥ ˜ Σ -1 h -¯ Σ -1 h ∥ ∥ ∥ 2 ≤ 4 E/κ 2 K 2 .)

We further decompose term (1) as below.

For term (5), because K ≥ max { 512 log ( 2 Hd δ ) κ 2 , 4 λ κ } , by Lemma E.13 and a union bound, with probability 1 -δ , for all h, s, a ∈ [ H ] ×S × A ,

where Σ p h = E µ,h [ ϕ ( s, a ) ϕ ( s, a ) ⊤ ] and λ min (Σ p h ) ≥ κ .

For term (4), it can be bounded by the following inequality (because of Cauchy-Schwarz inequality).

Bounding using covering. Note for any fix V h +1 , we can define x τ = ϕ (¯ s τ h , ¯ a τ h ) ( ∥ ϕ ∥ 2 ≤ 1 ) and η τ = V h +1 (¯ s τ h +1 ) 2 -P h ( V h +1 ) 2 (¯ s τ h , ¯ a τ h ) is H 2 -subgaussian, by Lemma E.9 (where t = K and L = 1 ), it holds that with probability 1 -δ ,

Let N h ( ϵ ) be the minimal ϵ -cover (with respect to the supremum norm) of V h := { V h : V h ( · ) = max a ∈A { min { ϕ ( s, a ) ⊤ θ -C 1 √ d · ϕ ( · , · ) ⊤ ˜ Λ -1 h ϕ ( · , · ) -C 2 , H -h +1 } + } } . That is, for any V ∈ V h , there exists a value function V ′ ∈ N h ( ϵ ) such that sup s ∈S | V ( s ) -V ′ ( s ) | < ϵ . Now by a union bound, we obtain with probability 1 -δ ,

which implies

choosing ϵ = d √ λ/K , applying Lemma B.3 of [Jin et al., 2021] 13 to the covering number N h +1 ( ϵ ) w.r.t. V h +1 , we can further bound above by

Apply a union bound for h ∈ [ H ] , we have with probability 1 -δ , for all h ∈ [ H ] ,

and similar to term (2) , it holds that for all h, s, a ∈ [ H ] ×S × A ,

Combining (45), (46), (47), (48), (49), (50), (51) and the assumption that K ≥ max { 4 L 2 H 2 d 3 κ , 32 E 2 d 2 κ 2 , 16 λ 2 d 2 κ } , we obtain with probability 1 -3 δ for all h, s, a ∈ [ H ] ×S × A ,

Step 2: The second step is to show for all h, s, a ∈ [ H ] ×S × A , with probability 1 -3 δ ,

The proof of Step 2 is nearly identical to Step 1 except ˜ V 2 h is replaced by ˜ V h .

Step 3: The last step is to prove sup h || ˜ σ 2 h -σ 2 ˜ V h +1 || ∞ ≤ 36 √ H 4 d 3 κK log ( ( λ + K )2 KdH 2 λδ ) with high probability.

Proof of Step 3. By (52),

Combining this with Step 1, we have with probability 1 -6 δ , ∀ h, s, a ∈ [ H ] ×S × A ,

Finally, by the non-expansiveness of operator max { 1 , ·} , the proof is complete.

D.3.4 Validity of our pessimism

Recall the definition ̂ Λ h = ∑ K τ =1 ϕ ( s τ h , a τ h ) ϕ ( s τ h , a τ h ) ⊤ / ˜ σ 2 h ( s τ h , a τ h ) + λ · I and Λ h = ∑ K τ =1 ϕ ( s τ h , a τ h ) ϕ ( s τ h , a τ h ) ⊤ /σ 2 ˜ V h +1 ( s τ h , a τ h )+ λI . Then we have the following lemma to bound the term √ ϕ ( s, a ) ⊤ ̂ Λ -1 h ϕ ( s, a ) by √ ϕ ( s, a ) ⊤ Λ -1 h ϕ ( s, a ) .

Lemma D.11 (Private version of lemma C.3 in [Yin et al., 2022]) . Denote the quantities C 1 = max { 2 λ, 128 log(2 dH/δ ) , 128 H 4 log(2 dH/δ ) κ 2 } and C 2 = ˜ O ( H 12 d 3 /κ 5 ) . Suppose the number of episode K satisfies K > max { C 1 , C 2 } and the condition in Lemma D.10, under the high probability events in Lemma D.2 and Lemma D.10, it holds that with probability 1 -2 δ , for all h, s, a ∈ [ H ] ×S × A ,

Proof of Lemma D.11. By definition √ ϕ ( s, a ) ⊤ ̂ Λ -1 h ϕ ( s, a ) = ∥ ϕ ( s, a ) ∥ ̂ Λ -1 h . Then denote

,

where Λ h = ∑ K τ =1 ϕ ( s τ h , a τ h ) ϕ ( s τ h , a τ h ) ⊤ /σ 2 ˜ V h +1 ( s τ h , a τ h ) + λI . Under the assumption of K , by the conclusion in Lemma D.10, we have

Next by Lemma E.12 (with ϕ to be ϕ/σ ˜ V h +1 and therefore C = 1 ) and a union bound, it holds with probability 1 -δ , for all h ∈ [ H ] ,

Therefore by Weyl's inequality and the assumption that K satisfies that

K > max { 2 λ, 128 log(2 dH/δ ) , 128 H 4 log(2 dH/δ ) κ 2 } , the above inequality leads to

Hence with probability 1 -δ , ∥ Λ ′ h ∥ ≤ 2 and ∥ ∥ Λ ′-1 h ∥ ∥ = λ -1 min (Λ ′ h ) ≤ 2 H 2 κ . Similarly, one can show ∥ ∥ ∥ ̂ Λ ′-1 h ∥ ∥ ∥ ≤ 2 H 2 κ with probability 1 -δ using identical proof.

Now apply Lemma E.11 and a union bound to ̂ Λ ′ h and Λ ′ h , we obtain with probability 1 -δ , for all h, s, a ∈ [ H ] ×S × A ,

where the third inequality uses (53) and the last inequality uses K > ˜ O ( H 12 d 3 /κ 5 ) . Note the conclusion can be derived directly by the above inequality multiplying 1 / √ K on both sides.

In order to bound ∥ ∥ ∥ ∑ K τ =1 x τ η τ ∥ ∥ ∥ ̂ Λ -1 h , we apply the following Lemma D.12.

Suppose K ≥ ˜ O ( H 12 d 3 /κ 5 ) 14 , then with probability 1 -δ ,

where ˜ O absorbs constants and Polylog terms.

Now we are ready to prove the following key lemma, which gives a high probability bound for ∣ ∣ ∣ ( T h ˜ V h +1 -˜ T h ˜ V h +1 )( s, a ) ∣ ∣ ∣ .

Lemma D.13. Assume K > max {M 1 , M 2 , M 3 , M 4 } , for any 0 < λ < κ , suppose √ d > ξ , where ξ := sup V ∈ [0 ,H ] , s ′ ∼ P h ( s,a ) , h ∈ [ H ] ∣ ∣ ∣ ∣ r h + V ( s ′ ) -( T h V )( s,a ) σ V ( s,a ) ∣ ∣ ∣ ∣ . Then with probability 1 -δ , for all h, s, a ∈ [ H ] ×S × A ,

where ˜ Λ h = ∑ K τ =1 ϕ ( s τ h , a τ h ) ϕ ( s τ h , a τ h ) ⊤ / ˜ σ 2 h ( s τ h , a τ h ) + λI + K 2 ,

and ˜ O absorbs constants and Polylog terms.

Proof of Lemma D.13. The proof is by combining (40), (42), Lemma D.4, Lemma D.6, Lemma D.7, Lemma D.8, Lemma D.12 and a union bound.

Remark D.14. Under the same assumption of Lemma D.13, because of Remark D.9 and Lemma D.11, we have with probability 1 -δ , for all h, s, a ∈ [ H ] ×S × A ,

Because D = ˜ O ( H 2 L κ + H 4 E √ d κ 3 / 2 + H 3 √ d ) and ˜ O absorbs constant, we will write as below for simplicity:

D.3.5 Finalize the proof of the first part

We are ready to prove the first part of Theorem 4.1.

Theorem D.15 (First part of Theorem 4.1) . Let K be the number of episodes. Suppose √ d > ξ , where ξ := sup V ∈ [0 ,H ] , s ′ ∼ P h ( s,a ) , h ∈ [ H ] ∣ ∣ ∣ ∣ r h + V ( s ′ ) -( T h V )( s,a ) σ V ( s,a ) ∣ ∣ ∣ ∣ and K > max {M 1 , M 2 , M 3 , M 4 } .

Then for any 0 < λ < κ , with probability 1 -δ , for all policy π simultaneously, the output ̂ π of Algorithm 2 satisfies

where Λ h = ∑ K τ =1 ϕ ( s τ h ,a τ h ) · ϕ ( s τ h ,a τ h ) ⊤ σ 2 ˜ V h +1 ( s τ h ,a τ h ) + λI d , D = ˜ O ( H 2 L κ + H 4 E √ d κ 3 / 2 + H 3 √ d ) and ˜ O absorbs constants and Polylog terms.

Proof of Theorem D.15. Combining Lemma D.3 and Remark D.14, we have with probability 1 -δ , for all policy π simultaneously,

now the proof is complete by taking the initial distribution d

D.3.6 Finalize the proof of the second part

To prove the second part of Theorem 4.1, we begin with a crude bound on sup h ∥ ∥ ∥ V ⋆ h -˜ V h ∥ ∥ ∥ ∞ .

Lemma D.16 (Private version of Lemma C.8 in [Yin et al., 2022]) . Suppose K ≥ max {M 1 , M 2 , M 3 , M 4 } , under the high probability event in Lemma D.13, with probability at least 1 -δ ,

Proof of Lemma D.16. Step 1: The first step is to show with probability at least 1 -δ , sup h ∥ ∥ V ⋆ h -V ̂ π h ∥ ∥ ∞ ≤ ˜ O ( H 2 √ d √ κK ) .

1 on both sides.

Indeed, combine Lemma D.3 and Lemma D.13, similar to the proof of Theorem D.15, we directly have with probability 1 -δ , for all policy π simultaneously, and for all s ∈ S , h ∈ [ H ] ,

Next, since K ≥ max { 512 log ( 2 Hd δ ) κ 2 , 4 λ κ } , by Lemma E.13 and a union bound over h ∈ [ H ] , with probability 1 -δ ,

where Λ p h = E µ,h [ σ -2 ˜ V h +1 ( s, a ) ϕ ( s, a ) ϕ ( s, a ) ⊤ ] and λ min (Λ p h ) ≥ κ H 2 .

Lastly, taking π = π ⋆ in (57) to obtain

This implies by using the condition K > max { H 2 L 2 dκ , H 6 E 2 κ 2 , H 4 κ } , we finish the proof of Step 1.

∥

h

h

π

h

∥

∥

∞

˜

Indeed, applying Lemma E.7 with π = π ′ = ̂ π , then with probability 1 -δ , for all s, h

where the second inequality uses Lemma D.13, Remark D.14 and the last inequality holds due to the same reason as Step 1.

Step 3: The proof of the lemma is complete by combining Step 1, Step 2, triangular inequality and a union bound.

Then we can give a high probability bound of sup h || σ 2 ˜ V h +1 -σ ⋆ 2 h || ∞ .

Lemma D.17 (Private version of Lemma C.10 in [Yin et al., 2022]) . Recall σ 2 ˜ V h +1 = max { 1 , Var P h ˜ V h +1 } and σ ⋆ 2 h = max { 1 , Var P h V ⋆ h +1 } . Suppose K ≥ max {M 1 , M 2 , M 3 , M 4 } , then with probability 1 -δ ,

(

H

2

√

√

d

κK

)

Step 2:

The second step is to show with probability

1

-

δ

,

sup

∥

V

-

V

̂

∥

O

.

Proof of Lemma D.17. By definition and the non-expansiveness of max { 1 , ·} , we have

The second inequality is because of the definition of variance. The last inequality comes from Lemma D.16.

We transfer √ ϕ ( s, a ) ⊤ Λ -1 h ϕ ( s, a ) to √ ϕ ( s, a ) ⊤ Λ ⋆ -1 h ϕ ( s, a ) by the following Lemma D.18.

Lemma D.18 (Private version of Lemma C.11 in [Yin et al., 2022]) . Suppose K ≥ max {M 1 , M 2 , M 3 , M 4 } , then with probability 1 -δ ,

Proof of Lemma D.18. By definition √ ϕ ( s, a ) ⊤ Λ -1 h ϕ ( s, a ) = ∥ ϕ ( s, a ) ∥ Λ -1 h . Then denote

where Λ ⋆ h = ∑ K τ =1 ϕ ( s τ h , a τ h ) ϕ ( s τ h , a τ h ) ⊤ /σ 2 V ⋆ h +1 ( s τ h , a τ h ) + λI . Under the condition of K , by Lemma D.17, with probability 1 -δ , for all h ∈ [ H ] ,

Next by Lemma E.12 (with ϕ to be ϕ/σ V ⋆ h +1 and C = 1 ), it holds with probability 1 -δ ,

Therefore by Weyl's inequality and the condition K > max { 2 λ, 128 log ( 2 dH δ ) , 128 H 4 log(2 dH/δ ) κ 2 } , the above inequality implies

Hence with probability 1 -δ , ∥ ∥ ∥ Λ ⋆ ′ h ∥ ∥ ∥ ≤ 2 and ∥ ∥ ∥ Λ ⋆ ′ -1 h ∥ ∥ ∥ = λ -1 min (Λ ⋆ ′ h ) ≤ 2 H 2 κ . Similarly, we can show that ∥ ∥ ∥ Λ ′ -1 h ∥ ∥ ∥ ≤ 2 H 2 κ holds with probability 1 -δ by using identical proof.

Now apply Lemma E.11 and a union bound to Λ ⋆ ′ h and Λ ′ h , we obtain with probability 1 -δ , for all h, s, a ∈ [ H ] ×S × A ,

where the third inequality uses (59) and the last inequality uses K ≥ ˜ O ( H 14 d/κ 5 ) . The conclusion can be derived directly by the above inequality multiplying 1 / √ K on both sides.

Finally, the second part of Theorem 4.1 can be proven by combining Theorem D.15 (with π = π ⋆ ) and Lemma D.18.

D.4 Put everything toghther

Combining Lemma D.1, Theorem D.15, and the discussion above, the proof of Theorem 4.1 is complete.

E Assisting technical lemmas

Lemma E.1 (Multiplicative Chernoff bound [Chernoff et al., 1952]) . Let X be a Binomial random variable with parameter p, n . For any 1 ≥ θ > 0 , we have that

Lemma E.2 (Hoeffding's Inequality [Sridharan, 2002]) . Let x 1 , ..., x n be independent bounded random variables such that E [ x i ] = 0 and | x i | ≤ ξ i with probability 1 . Then for any ϵ > 0 we have

Lemma E.3 (Bernstein's Inequality) . Let x 1 , ..., x n be independent bounded random variables such that E [ x i ] = 0 and | x i | ≤ ξ with probability 1 . Let σ 2 = 1 n ∑ n i =1 Var[ x i ] , then with probability 1 -δ we have

Lemma E.4 (Empirical Bernstein's Inequality [Maurer and Pontil, 2009]) . Let x 1 , ..., x n be i.i.d random variables such that | x i | ≤ ξ with probability 1 . Let ¯ x = 1 n ∑ n i =1 x i and ̂ V n = 1 n ∑ n i =1 ( x i -¯ x ) 2 , then with probability 1 -δ we have

Lemma E.5 (Lemma I.8 in [Yin and Wang, 2021b]) . Let n ≥ 2 and V ∈ R S be any function with || V || ∞ ≤ H , P be any S -dimensional distribution and ̂ P be its empirical version using n samples. Then with probability 1 -δ ,

Lemma E.6 (Claim 2 in [Vietri et al., 2020]) . Let y ∈ R be any positive real number. Then for all x ∈ R with x ≥ 2 y , it holds that 1 x -y ≤ 1 x + 2 y x 2 .

E.1 Extended Value Difference

Lemma E.7 (Extended Value Difference (Section B.1 in [Cai et al., 2020])) . Let π = { π h } H h =1 and π ′ = { π ′ h } H h =1 be two arbitrary policies and let { ̂ Q h } H h =1 be any given Q-functions. Then define ̂ V h ( s ) := ⟨ ̂ Q h ( s, · ) , π h ( · | s ) ⟩ for all s ∈ S . Then for all s ∈ S ,

where ( T h V )( · , · ) := r h ( · , · ) + ( P h V )( · , · ) for any V ∈ R S .

Lemma E.8 (Lemma I.10 in [Yin and Wang, 2021b]) . Let ̂ π = { ̂ π h } H h =1 and ̂ Q h ( · , · ) be the arbitrary policy and Q-function and also ̂ V h ( s ) = ⟨ ̂ Q h ( s, · ) , ̂ π h ( ·| s ) ⟩ ∀ s ∈ S , and ξ h ( s, a ) = ( T h ̂ V h +1 )( s, a ) -̂ Q h ( s, a ) element-wisely. Then for any arbitrary π , we have

where the expectation are taken over s h , a h .

E.2 Assisting lemmas for linear MDP setting

Lemma E.9 (Hoeffding inequality for self-normalized martingales [Abbasi-Yadkori et al., 2011]) . Let { η t } ∞ t =1 be a real-valued stochastic process. Let {F t } ∞ t =0 be a filtration, such that η t is F t -measurable. Assume η t also satisfies η t given F t -1 is zero-mean and R -subgaussian, i.e.

Let { x t } ∞ t =1 be an R d -valued stochastic process where x t is F t -1 measurable and ∥ x t ∥ ≤ L . Let Λ t = λI d + ∑ t s =1 x s x ⊤ s . Then for any δ > 0 , with probability 1 -δ , for all t > 0 ,

Lemma E.10 (Bernstein inequality for self-normalized martingales [Zhou et al., 2021]) . Let { η t } ∞ t =1 be a real-valued stochastic process. Let {F t } ∞ t =0 be a filtration, such that η t is F t -measurable. Assume η t also satisfies

Let { x t } ∞ t =1 be an R d -valued stochastic process where x t is F t -1 measurable and ∥ x t ∥ ≤ L . Let Λ t = λI d + ∑ t s =1 x s x ⊤ s . Then for any δ > 0 , with probability 1 -δ , for all t > 0 ,

Lemma E.11 (Lemma H.4 in [Yin et al., 2022]) . Let Λ 1 and Λ 2 ∈ R d × d be two positive semi-definite matrices. Then:

and for all ϕ ∈ R d .

Lemma E.12 (Lemma H.4 in [Min et al., 2021]) . Let ϕ : S × A → R d satisfies ∥ ϕ ( s, a ) ∥ ≤ C for all s, a ∈ S × A . For any K > 0 , λ > 0 , define ¯ G K = ∑ K k =1 ϕ ( s k , a k ) ϕ ( s k , a k ) ⊤ + λI d where ( s k , a k ) 's are i.i.d samples from some distribution ν . Then with probability 1 -δ ,

Lemma E.13 (Lemma H.5 in [Min et al., 2021]) . Let ϕ : S × A → R d be a bounded function s.t. ∥ ϕ ∥ 2 ≤ C . Define ¯ G K = ∑ K k =1 ϕ ( s k , a k ) ϕ ( s k , a k ) ⊤ + λI d where ( s k , a k ) 's are i.i.d samples from some distribution ν . Let G = E ν [ ϕ ( s, a ) ϕ ( s, a ) ⊤ ] . Then for any δ ∈ (0 , 1) , if K satisfies

Then with probability at least 1 -δ , it holds simultaneously for all u ∈ R d that

Lemma E.14 (Lemma H.9 in [Yin et al., 2022]) . For a linear MDP, for any 0 ≤ V ( · ) ≤ H , there exists a w h ∈ R d s.t. T h V = ⟨ ϕ, w h ⟩ and ∥ w h ∥ 2 ≤ 2 H √ d for all h ∈ [ H ] . Here T h ( V )( s, a ) = r h ( x, a ) + ( P h V )( s, a ) . Similarly, for any π , there exists w π h ∈ R d , such that Q π h = ⟨ ϕ, w π h ⟩ with ∥ w π h ∥ 2 ≤ 2( H -h +1) √ d .

E.3 Assisting lemmas for differential privacy

Lemma E.15 (Converting zCDP to DP [Bun and Steinke, 2016]) . If M satisfies ρ -zCDP then M satisfies ( ρ +2 √ ρ log(1 /δ ) , δ ) -DP.

Lemma E.16 (zCDP Composition [Bun and Steinke, 2016]) . Let M : U n →Y and M ′ : U n →Z be randomized mechanisms. Suppose that M satisfies ρ -zCDP and M ′ satisfies ρ ′ -zCDP. Define M ′′ : U n →Y×Z by M ′′ ( U ) = ( M ( U ) , M ′ ( U )) . Then M ′′ satisfies ( ρ + ρ ′ ) -zCDP.

Lemma E.17 (Adaptive composition and Post processing of zCDP [Bun and Steinke, 2016]) . Let M : X n → Y and M ′ : X n ×Y → Z . Suppose M satisfies ρ -zCDP and M ′ satisfies ρ ′ -zCDP (as a function of its first argument). Define M ′′ : X n →Z by M ′′ ( x ) = M ′ ( x, M ( x )) . Then M ′′ satisfies ( ρ + ρ ′ ) -zCDP.

Definition E.18 ( ℓ 1 sensitivity) . Define the ℓ 1 sensitivity of a function f : N X ↦→ R d as

Definition E.19 (Laplace Mechanism [Dwork et al., 2014]) . Given any function f : N X ↦→ R d , the Laplace mechanism is defined as:

where Y i are i.i.d. random variables drawn from Lap(∆ 1 ( f ) /ϵ ) .

Lemma E.20 (Privacy guarantee of Laplace Mechanism [Dwork et al., 2014]) . The Laplace mechanism preserves ( ϵ, 0) -differential privacy. For simplicity, we say ϵ -DP.

F Details for the Evaluation part

In the Evaluation part, we apply a synthetic linear MDP case that is similar to [Min et al., 2021, Yin et al., 2022] but with some modifications for our evaluation task. The linear MDP example we use consists of |S| = 2 states and |A| = 100 actions, while the feature dimension d = 10 . We denote S = { 0 , 1 } and A = { 0 , 1 , . . . , 99 } respectively. For each action a ∈ { 0 , 1 , . . . , 99 } , we obtain a vector a ∈ R 8 via binary encoding. More specifically, each coordinate of a is either 0 or 1 .

First, we define the following indicator function δ ( s, a ) = { 1 if 1 { s = 0 } = 1 { a = 0 } 0 otherwise , then our non-stationary linear MDP example can be characterized by the following parameters.

The feature map ϕ is:

The unknown measure ν h is:

where { α h, 1 , α h, 2 } h ∈ [ H ] is a sequence of random values sampled uniformly from [0 , 1] . The unknown vector θ h is:

where r h is also sampled uniformly from [0 , 1] . Therefore, the transition kernel follows P h ( s ′ | s, a ) = ⟨ ϕ ( s, a ) , ν h ( s ′ ) ⟩ and the expected reward function r h ( s, a ) = ⟨ ϕ ( s, a ) , θ h ⟩ .

Finally, the behavior policy is to always choose action a = 0 with probability p , and other actions uniformly with probability (1 -p ) / 99 . Here we choose p = 0 . 6 . The initial distribution is a uniform distribution over S = { 0 , 1 } .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two offline RL algorithms with differential privacy guarantees. The two pessimism-based algorithms apply to both tabular and linear MDP settings. In theory, the authors prove that the proposed algorithms achieve instance-dependent sub-optimality bounds while guaranteeing differential privacy. A nice property is that the cost of privacy only appears as lower order terms, thus become negligible as the number of samples goes large.\n\nSTRENGTHS:\n(1) While DP algorithm has been studied in online RL settings, the study of DP in offline RL is limited. This paper provides the first provable study along this line. \n\n(2) In spite of a theoretical paper, it is well written and easy to follow.\n\nWEAKNESSES:\n(1) The motivation of the considered problem needs more real justifications. In the introduction, the authors used a medical example to motivate the need of considering privacy in offline RL. Why could not the data owner (hospital or doctors) do offline policy evaluation or policy optimization directly on the raw data? Why do we need to generate a private policy? On the other hand, if the offline policy optimization is requested from a third-part not the data owner, it is justified that the patient's data needs to be protected. However, in this case, the third-part would not have access to the raw data. This contradicts with the setting considered in this paper as in the proposed Algorithm 1 and Algorithm 2, the input is the raw data. Therefore, it is important and helpful to provide a convincing real example to justify the problem setting and algorithm designs. \n\n(2) In Assumption 2.2, the data distribution needs to satisfy the minimum eigenvalue condition. This assumption might be violated when the feature space is large or some features are highly correlated. It is helpful to provide some discussion on this assumption and how to remedy it when this assumption is violated. \n\n(3) In the tabular MDP (DP-APVI) algorithm, the Gaussian noise is added to the integer counts $n_{s_h, a_h}$ and $n_{s_h, a_h, s_{h+1}}$ to obtain a private estimation of the transition kernel. Because of the design, the private counts might be negative or very small, which makes the uncertainty estimation (line 5 of Algorithm 1) to be unstable. To handle it, the authors used some truncation approach with some theoretical truncation rate $E_{\\rho}$. It is unclear if it is a good choice to add the Gaussian noise to the count statistics. Can we add the Gaussian noise directly to the non-private estimation of transition kernel? Some justifications on the proposed private estimation would be helpful. It is also important to discuss the advantages and limitations of the proposed private estimation while comparing with other choices in existing RL literature. \n\n(4) In Algorithm 2 and line 235 of Page 7, the authors require two independent offline dataset with equal length. Can authors clarify what the rigorous condition of the \"two independent offline dataset\"? Do you need each sample (across $K$) at each time horizon (across $H$) to be independent in each offline data? If yes, this is an unrealistic condition in offline RL problem. Moreover, I did not find such condition in the statement of Theorem 4.1. Where do you use this \"independence\" assumption? \n\n(5) In the experiments (Figure 1), the simulations are for 5 replicates. Can authors include the uncertainty in Figure 1? If the uncertainty is large, it is helpful to increase the replication times.  \n\n\n~~~~~~~~~~~\nAfter rebuttal: my major comments have been nicely addressed. I have increased the score to 7.\n\nQUESTIONS:\nSee Weakness section\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the offline RL with Differential Privacy constraints problem. Tabular and linear MDP are considered, while both forms of DP, traditional DP and zCDP are studied. The authors cast the DP definition into the offline RL problem as a constraint for protecting trajectories. Two algorithms, DP-APVI (resp. DP-VAPVI), are introduced, treating the tabular (resp. linear) case. Those algorithms rely on the pessimism principle, and DP is obtained by adding a Gaussian mechanism, either on the counts or during variance estimation. Authors prove zCDP compliance for both algorithms, and a comparison with non-private algorithms is provided. Finally, authors empirically evaluate the performance of DP-VAPVI under different privacy budgets.\n\nSTRENGTHS:\nThe main strength of this paper is that it is, according to my knowledge, the first to tackle the offline RL with DP constraints, which is an important problem. This paper introduce two sound and practical algorithms, respecting Differential Privacy by design. Furthermore, we  have a discussion comparing the algorithms with their non-private counterpart and experimental validation.\n\nWEAKNESSES:\n- Globally, the paper is well-written, however, I found the section on DP-VAPVI very hard to follow, especially the algorithm\n- Experiments study the performance of the algorithms under differential privacy budgets, but it feels very hard to have an interpretation of those budgets for zCDP. In the traditional DP case, one often sees very high values of \\epsilon, leading to a discussable Differentially Private algorithm in practice. Would it be possible to have a discussion about \\alpha?  What would be reasonable values?\n- Although there may not be other paper directly studying directly DP offline RL, there is work in literature on privacy attacks in RL, such as membership inference attack: R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models) or Maziar Gomrokchi, Susan Amin, Hossein Aboutalebi, Alexander Wong, and Doina Precup. Membership Inference Attacks Against Temporally Correlated Data in Deep Reinforcement Learning. It would have been nice to have a discussion on those attacks to have a better estimation of the practical impact of DP casted like this.\n\nQUESTIONS:\n- See section weakness, could the authors comment on the \\rho factor? And would it allow to robustness to membership inference attacks?\n- You only consider an MDP with 2 states in the experiments, would the algorithm scale to bigger MDPs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on reinforcement learning (RL) in an offline setting under differential privacy considerations. It studies the proposal and analysis of a new approach to learning policies in this specific setting, leveraging the Bernstein concentration inequality.\n\nReinforcement learning explores an agent's interaction with its environment over a series of episodes. The environment is defined as a Markov decision process (MDP) comprising states, actions available to the agent, a reward function, and transition dynamics. Through this interaction, the agent gathers feedback to learn a policy that optimizes the cumulative reward. In this paper, the agent operates under the constraints of differential privacy, implying that it interacts in an environment with sensitive data. Here, observations from each episode are deemed private, being tied to individual users.\n\nTraditionally, two settings have been studied: the online and offline settings. In the online setting, an agent learns a policy through active data collection by exploring the environment. Conversely, the offline setting involves the agent receiving all data upfront, barring any access to the environment for training. Prior studies have explored these settings without privacy constraints, but only the online setting has been studied under differential privacy.\n\nThis paper breaks new ground by studying reinforcement learning in the offline setting under differential privacy. Like its online counterpart, the offline setting necessitates the construction of tight confidence bounds around sufficient statistics for policy approximation. However, due to privacy-induced noise, the algorithm must inflate the confidence bounds to compensate.\nThe primary innovation of this paper is the proposal of new confidence bounds, utilizing the Bernstein concentration inequality, a departure from the traditional Hoeffding concentration. This novel method offers improved error bounds under specific instance-dependent conditions, although the bounds are equivalent to Hoeffding's in the worst-case scenario.\n\nThe authors cleverly adapt techniques from previous research while also introducing new methodologies that could potentially extend beyond this work. A significant contribution lies in the estimation of variance from noisy statistics, thereby enabling the utilization of the Bernstein concentration.\n\nIn sum, this paper illuminates a hitherto unexplored area in the RL community, offering innovative solutions and techniques for reinforcement learning in the offline setting under differential privacy.\n\nSTRENGTHS:\nThe primary strength of this paper lies in its pioneering examination of differential privacy reinforcement learning (DP-RL) in the offline setting, effectively bridging a notable gap in existing academic literature. By venturing into this uncharted territory, the paper opens up possibilities for practical applications in real-world scenarios where sensitive data may be involved.\n\nThe paper's successful application of differential privacy to offline RL deserves commendation, given the unique challenges associated with this endeavor. While one might assume that the principles governing DP-RL in the online setting would seamlessly translate to the offline setting, this is not the case. The paper adeptly navigates these challenges, setting a benchmark for future investigations into this area.\nAnother significant contribution of this work is the innovative use of Bernstein concentration in the estimation of error bounds. Prior studies primarily employed Hoeffding concentration, which, although effective, offered limited utility under specific conditions. In contrast, Bernstein concentration proves to be more flexible, offering improved error bounds, particularly under specific instance-dependent conditions. This innovative application, therefore, has the potential to enhance results not only in the offline setting but also in the online RL environment.\n\nFurthermore, the paper presents an empirical evaluation that provides rich insights into the practical implications of their proposed techniques. This robust empirical analysis not only validates the theoretical contributions but also offers tangible results that underscore the effectiveness of the proposed methods.\n\nIn conclusion, the paper's merits extend from filling a knowledge void in DP-RL literature, effectively tackling challenges in translating DP-RL principles from online to offline settings, to innovatively employing Bernstein concentration in error bound estimation. The comprehensive empirical evaluation serves as the icing on the cake, demonstrating the practicality and effectiveness of the proposed methods. The paper's contributions are both theoretical and practical, promising to advance understanding and application of DP-RL in offline settings.\n\nWEAKNESSES:\nMany of the techniques presented in this paper are not new. The concentration bounds utilized, for instance, have been previously developed and employed in other works. However, the paper's merit lies in its successful adaptation of these existing techniques for a specific context. Therefore, this paper effectively demonstrates how to repurpose these pre-existing tools for its unique setting, thereby contributing to the literature in a meaningful way.\n\nQUESTIONS:\nNo questions.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an algorithm for offline reinforcement learning with differential privacy (DP), which protects the privacy of the original information using a Gaussian mechanism based on pessimism. The motivation and ideas behind the paper are clear and meaningful. However, there are some issues with the methods and experiments.\n\nOverall, the paper presents some interesting ideas, but additional experiments and comparisons are needed to fully evaluate the proposed method and its practical value.\n\nSTRENGTHS:\n1.The authors propose a method for protecting the privacy of original information in offline reinforcement learning. They achieve this goal by implementing differential privacy (DP) in their proposed method, which is a meaningful contribution to the field of privacy-preserving machine learning. \n2. The authors implement their ideas in APVI and VAPVI models and provide a thorough theoretical analysis of the proposed method. The writing is clear and the theoretical analysis is extensive, providing a strong foundation for the authors' claims. \n3. The authors conduct experiments on simulated datasets, which provide preliminary evidence of the method's performance.\n\nWEAKNESSES:\nThe paper proposes two models, DP-APVI and DP-VAPVI, for solving the offline reinforcement learning problem with privacy guarantees. While the paper presents some interesting ideas, there are several issues:\n1. The paper only includes results for DP-VAPVI, and does not provide any experimental results for DP-APVI. It would be helpful to see how DP-APVI performs in comparison to DP-VAPVI and other baseline methods.\n2. The results for DP-VAPVI consistently show a performance gap compared to VAPVI, which raises questions about the competitiveness of DP-VAPVI in practice. Without additional experiments or comparisons with other methods, it is difficult to assess the practical value of DP-VAPVI.\n3. The paper claims that DP-VAPVI will converge to VAPVI as the dataset size increases, but there is no experimental evidence to support this claim.\n4. The paper does not discuss the impact of the privacy budget (ρ) on the privacy protection and performance of the algorithms. It would be helpful to see how different values of ρ affect the results.\n5. The paper lacks ablation experiments to investigate the extent to which DP itself as an optimization plug-in for APVI and VAPVI models maintains privacy and affects performance. It would be helpful to see how different components of the DP-APVI and DP-VAPVI models contribute to the overall performance.\n6. The paper only compares DP-VAPVI and VAPVI with PEVI, which is not a privacy-preserving method. It would be helpful to see how DP-VAPVI and DP-APVI compare to other privacy-preserving methods.\n7. The experiments are conducted on synthetic datasets, which may not fully reflect the complexity and diversity of real-world problems. It would be helpful to see how the proposed methods perform on real-world datasets.\nMethodologically, the paper proposes to add additional Gaussian mechanisms to APVI and VAPVI models to ensure privacy, but the main methods are still based on APVI and VAPVI. While the paper's definition of neighboring datasets and the use of Gaussian mechanism for differential privacy are contributions, the experimental results are not convincing enough. The authors should demonstrate the advantages of their proposed methods on a wider range of datasets and models.\n\nI'm not an expert on differential privacy. My evaluation is based on how well I was able to comprehend the information in the paper. I don't fully comprehend how this work contributes to the overall growth of the field.\n\nQUESTIONS:\nPlease refer to the weakness\n\nLIMITATIONS:\nPlease refer to the weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}

comments: SUMMARY:
This paper focuses on the offline RL problem with differential privacy. The authors propose algorithms for offline tabular MDP and offline linear MDP with $\rho$-DP. For the first problem, the sub-optimality bound almost matches the best-existing non-private counter-part in spite of an additional term $O(\sqrt{\frac{1}{\rho}})$. For the second problem, the gap between the proposed sub-optimality bound to the best existing non-private counter part is $\mathrm{poly}(d,H,\kappa^{-1})/\sqrt{\rho}$, where $\kappa$ is the minimal coverage parameter.

STRENGTHS:
1. This paper firstly provides analysis for offline RL, and proposes error bounds which  matches the non-privacy counterparts up to some lower order terms.

2. In technique, the authors  make efforts to operate on Bernstein type pessimism while keeping privacy to achieve the tighter sub-optimality bound.

WEAKNESSES:
1. The technique novelty is  somewhat limited given literature in online RL with DP.

2. The discussion about related work is insufficient. In particular, I wonder what are the best existing regret bounds for online RL with DP (either tabular or linear MDPs)? If the regret bounds are not tight, could we use the Bernstein-style bonus with DP to improve the regret bounds?

QUESTIONS:
1. For the non-privacy problem, does the error bound have a polynomial dependence on $\kappa$ for the linear MDP problem?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-14