SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yasin Abbasi-Yadkori, David Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems, pages 2312-2320, 2011.
Naman Agarwal and Karan Singh. The price of differential privacy for online learning. In International Conference on Machine Learning, pages 32-40. PMLR, 2017.
Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning, pages 463-474. PMLR, 2020.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 263-272. JMLR.org, 2017.
Borja Balle, Maziar Gomrokchi, and Doina Precup. Differentially private policy evaluation. In International Conference on Machine Learning, pages 2130-2138. PMLR, 2016.
Debabrota Basu, Christos Dimitrakakis, and Aristide Tossou. Differential privacy for multi-armed bandits: What is it and what is its cost? arXiv preprint arXiv:1905.12298, 2019.
Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In ACM SIGACT Symposium on Theory of Computing, pages 123–132, 2021.
Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography Conference, pages 635-658. Springer, 2016.
Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning, pages 1283-1294. PMLR, 2020.
Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium (USENIX Security 19), pages 267-284, 2019.
Xiaoyu Chen, Kai Zheng, Zixin Zhou, Yunchang Yang, Wei Chen, and Liwei Wang. (locally) differentially private combinatorial semi-bandits. In International Conference on Machine Learning, pages 1757-1767. PMLR, 2020.
Herman Chernoff et al. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. The Annals of Mathematical Statistics, 23(4):493-507, 1952.
Sayak Ray Chowdhury and Xingyu Zhou. Differentially private regret minimization in episodic markov decision processes. arXiv preprint arXiv:2112.10599, 2021.
Sayak Ray Chowdhury, Xingyu Zhou, and Ness Shroff. Adaptive control of differentially private linear quadratic systems. In 2021 IEEE International Symposium on Information Theory (ISIT), pages 485-490. IEEE, 2021.
Chris Cundy and Stefano Ermon. Privacy-constrained policies via mutual information regularized policy gradients. arXiv preprint arXiv:2012.15019, 2020.
Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning. In Advances in Neural Information Processing Systems, pages 5713-5723, 2017.
Cynthia Dwork and Guy N Rothblum. Concentrated differential privacy. arXiv preprint arXiv:1603.01887, 2016.
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 2021a.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 2021b.
Jianyu Xu, Dan Qiao, and Yu-Xiang Wang. Doubly fair dynamic pricing. In International Conference on Artificial Intelligence and Statistics, pages 9941-9975. PMLR, 2023.
Ming Yin and Yu-Xiang Wang. Asymptotically efficient off-policy evaluation for tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 3948-3958. PMLR, 2020.
Ming Yin and Yu-Xiang Wang. Optimal uniform ope and model-based offline reinforcement learning in time-homogeneous, reward-free and task-agnostic settings. Advances in neural information processing systems, 2021a.
Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems, 34, 2021b.
Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal provable uniform convergence in offline policy evaluation for reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 1567-1575. PMLR, 2021.
Ming Yin, Yaqi Duan, Mengdi Wang, and Yu-Xiang Wang. Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. arXiv preprint arXiv:2203.05804, 2022.
Andrea Zanette. Exponential lower bounds for batch reinforcement learning: Batch rl can be exponentially harder than online rl. In International Conference on Machine Learning, pages 12287-12297. PMLR, 2021.
Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 2021.
Zihao Zhang, Stefan Zohren, and Stephen Roberts. Deep reinforcement learning for trading. The Journal of Financial Data Science, 2(2):25-40, 2020.
Fuheng Zhao, Dan Qiao, Rachel Redberg, Divyakant Agrawal, Amr El Abbadi, and Yu-Xiang Wang. Differentially private linear sketches: Efficient implementations and applications. Advances in Neural Information Processing Systems, 35:12691-12704, 2022.
Kai Zheng, Tianle Cai, Weiran Huang, Zhenguo Li, and Liwei Wang. Locally differentially private (contextual) bandits learning. Advances in Neural Information Processing Systems, 33:12300-12310, 2020.
Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory, pages 4532-4576. PMLR, 2021.
Xingyu Zhou. Differentially private reinforcement learning with linear function approximation. arXiv preprint arXiv:2201.07052, 2022.
A Notation List
A.1 Notations for tabular MDP
Eρ	4√H log 4HS2A/ρ
n	The original counts of visitation
n'	The noisy counts, as defined in (2)
∛n	Final choice of private counts, as defined in (3)
∛P	Private estimate of transition kernel, as defined in (5)
∛P	Non-private estimate of transition kernel, as defined in (15)
ι	log HSA/δ
ρ	Budget for zCDP
δ	Failure probability
A.2 Notations for linear MDP
L	2H√5Hd log(10Hd/δ)/ρ
E	√10Hd/ρ(2+(log(5c1H/δ)c2d)2/3)
D	O(H2L/K+H4E√d/K3/2+H3√d)
Δh	∑K=1φ(shk,ahk)φ(shk,ahk)T/σ2h(shk,ahk)+λId
Δh	∑K=1φ(shk,ahk)φ(shk,ahk)T/σ2h(shk,ahk)+λId+K2
Δhp	Eμ,h[σh-2(s,a)φ(s,a)φ(s,a)T]
Δh	∑K=1φ(shτ,ahτ)φ(shτ,ahτ)T/σ2Vh+1(shτ,ahτ)+λI
Δhp	Eμ,h[σ-2Vh+1(s,a)φ(s,a)φ(s,a)T]
Δh*	∑K=1φ(shτ,ahτ)φ(shτ,ahτ)T/σ2Vh* (shτ,ahτ)+λI
Σh	∑K=1φ(¯ŝh,¯âh)φ(¯ŝh,¯âh)T+λId
Σh	∑K=1φ(¯ŝh,¯âh)φ(¯ŝh,¯âh)T+λId+K1
Σhp	Eμ,h[φ(s,a)φ(s,a)T]
κ	minh λmin(Σhp)
σ2V(s,a)	max{1,VarPh(V)(s,a)} for any V
σ2h(s,a)	max{1,VarPhVh+1(s,a)}
σ2(h,s,a)	max{1,VarhVh+1(s,a)}
M1	max{2λ,128 log(2dH/δ),128H4 log(2dH/δ)/K2,√2L/√dκ}
M2	max{O(H12d3/κ5),O(H14d/κ5)}
M3	max{512H4 log(2dH/δ)/κ2,4λH2/κ}
M4	max{H2L2/dk,H6E2/κ2,H4κ}
ρ	Budget for zCDP
δ	Failure probability (not the δ of (ε,δ)-DP)
ξ	supV∈[0,H], s′~Ph(s,a), h∈[H] |rh+V(s′)-(TrV)(s,a)/σV(s,a)
Online reinforcement learning under JDP or LDP. For online RL, some recent works analyze this setting under Joint Differential Privacy (JDP), which requires the RL agent to minimize regret while handling user's raw data privately. Under tabular MDP, Vietri et al. [2020] design PUCB by revising UBEV [Dann et al., 2017]. Private-UCB-VI [Chowdhury and Zhou, 2021] results from UCBVI (with bonus-1) [Azar et al., 2017]. However, both works privatize Hoeffding type bonus, which lead to sub-optimal regret bound. Under linear MDP, Private LSVI-UCB [Ngo et al., 2022] and Privacy-Preserving LSVI-UCB [Luyo et al., 2021] are private versions of LSVI-UCB [Jin et al., 2020b], while LinOpt-VI-Reg [Zhou, 2022] and Privacy-Preserving UCRL-VTR [Luyo et al., 2021] generalize UCRL-VTR [Ayoub et al., 2020]. However, these works are usually based on the LSVI technique [Jin et al., 2020b] (unweighted ridge regression), which does not ensure optimal regret bound.
In addition to JDP, another common privacy guarantee for online RL is Local Differential Privacy (LDP), LDP is a stronger definition of DP since it requires that the user's data is protected before the RL agent has access to it. Under LDP, Garcelon et al. [2021] reach a regret lower bound and design LDP-OBI which has matching regret upper bound. The result is generalized by Liao et al. [2021] to linear mixture setting. Later, Luyo et al. [2021] provide an unified framework for analyzing JDP and LDP under linear setting.
Some other differentially private learning algorithms. There are some other works about differentially private online learning [Guha Thakurta and Smith, 2013, Agarwal and Singh, 2017, Hu et al., 2021] and various settings of bandit [Shariff and Sheffet, 2018, Gajane et al., 2018, Basu et al., 2019, Zheng et al., 2020, Chen et al., 2020, Tossou and Dimitrakakis, 2017]. For the reinforcement learning setting, Wang and Hegde [2019] propose privacy-preserving Q-learning to protect the reward information. Ono and Takahashi [2020] study the problem of distributed reinforcement learning under LDP. Lebensold et al. [2019] present an actor critic algorithm with differentially private critic. Cundy and Ermon [2020] tackle DP-RL under the policy gradient framework. Chowdhury et al. [2021] consider the adaptive control of differentially private linear quadratic (LQ) systems. Zhao et al. [2022] designed differentially private linear sketch algorithms.
Offline reinforcement learning under tabular MDP. Under tabular MDP, there are several works achieving optimal sub-optimality/sample complexity bounds under different coverage assumptions. For the problem of off-policy evaluation (OPE), Yin and Wang [2020] uses Tabular-MIS estimator to achieve asymptotic efficiency. In addition, the idea of uniform OPE is used to achieve the optimal sample complexity  $O(H^{3} / d_{m}\epsilon^{2})$  [Yin et al., 2021] for non-stationary MDP and the optimal sample complexity  $O(H^{2} / d_{m}\epsilon^{2})$  [Yin and Wang, 2021a] for stationary MDP, where  $d_{m}$  is the lower bound for state-action occupancy. Such uniform convergence idea also supports some works regarding online exploration [Jin et al., 2020a, Qiao et al., 2022, Xu et al., 2023]. For offline RL with single concentrability assumption, Xie et al. [2021b] arrive at the optimal sample complexity  $O(H^{3}SC^{\star} / \epsilon^{2})$ . Recently, Yin and Wang [2021b] propose APVI which can lead to instance-dependent sub-optimality bound, which subsumes previous optimal results under several assumptions. Madhow et al. [2023] consider offline policy evaluation for adaptively collected datasets.
Offline reinforcement learning under linear MDP. Recently, many works focus on offline RL under linear representation. Jin et al. [2021] present PEVI which applies the idea of pessimistic value iteration (the idea originates from [Jin et al., 2020b]), and PEVI is provably efficient for offline RL under linear MDP. Yin et al. [2022] improve the sub-optimality bound in [Jin et al., 2021] by replacing LSVI by variance-weighted LSVI. Xie et al. [2021a] consider Bellman consistent pessimism for general function approximation, and their result improves the sample complexity in [Jin et al., 2021] by order  $O(d)$  (shown in Theorem 3.2). However, there is no improvement on horizon dependence. Zanette et al. [2021] propose a new offline actor-critic algorithm that naturally incorporates the pessimism principle. Besides, Wang et al. [2021a], Zanette [2021] study the statistical hardness of offline RL with linear representations by presenting exponential lower bounds. When relaxing the offline setting to low adaptive RL, Gao et al. [2021], Wang et al. [2021b], Qiao and Wang [2023b], Qiao et al. [2023] designed algorithms with low adaptivity.
C Proof of Theorem 3.4
C.1 Proof sketch
Since the whole proof for privacy guarantee is not very complex, we present it in Section C.2 below and only sketch the proof for suboptimality bound.
First of all, we bound the scale of noises we add to show that the  $\widetilde{n}$  derived from (3) are close to real visitation numbers. Therefore, denoting the non-private empirical transition kernel by  $\widehat{P}$  (detailed definition in (15)), we can show that  $\| \widetilde{P} -\widehat{P}\| _1$  and  $|\sqrt{\mathrm{Var}_{\widetilde{P}}(V)} -\sqrt{\mathrm{Var}_{\widehat{P}}(V)} |$  are small.
Next, resulting from the conditional independence of  $\widetilde{V}_{h+1}$  and  $\widetilde{P}_h$ , we apply Empirical Bernstein's inequality to get  $|(P_h - P_h)\widetilde{V}_{h+1}| \lesssim \sqrt{\operatorname{Var}_{\widetilde{P}}(\widetilde{V}_{h+1}) / \widetilde{n}_{s_h,a_h}} + SHE_{\rho} / \widetilde{n}_{s_h,a_h}$ . Together with our definition of private pessimism and the key lemma: extended value difference (Lemma E.7 and E.8), we can bound the suboptimality of our output policy  $\widehat{\pi}$  by:
$$
v ^ {\star} - v ^ {\widehat {\pi}} \lesssim \sum_ {h = 1} ^ {H} \sum_ {(s _ {h}, a _ {h}) \in \mathcal {C} _ {h}} d _ {h} ^ {\pi^ {\star}} (s _ {h}, a _ {h}) \sqrt {\frac {\operatorname {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot))}{\widetilde {n} _ {s _ {h} , a _ {h}}}} + S H E _ {\rho} / \widetilde {n} _ {s _ {h}, a _ {h}}. \tag {12}
$$
Finally, we further bound the above suboptimality via replacing private statistics by non-private ones. Specifically, we replace  $\widetilde{n}$  by  $n$ ,  $\widetilde{P}$  by  $P$  and  $\widetilde{V}$  by  $V^{\star}$ . Due to (12), we have  $\| \widetilde{V} - V^{\star} \|_{\infty} \lesssim \sqrt{\frac{1}{nd_m}}$ . Together with the upper bounds of  $\| \widetilde{P} - \widehat{P} \|_1$  and  $|\sqrt{\operatorname{Var}_{\widehat{P}}(V)} - \sqrt{\operatorname{Var}_{\widehat{P}}(V)}|$ , we have
$$
\begin{array}{l} \sqrt {\frac {\mathrm {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (\widetilde {V} _ {h + 1} (\cdot))}{\widetilde {n} _ {s _ {h} , a _ {h}}}} \lesssim \sqrt {\frac {\mathrm {V a r} _ {\widetilde {P} _ {h} (\cdot | s _ {h} , a _ {h})} (V _ {h + 1} ^ {\star} (\cdot))}{\widetilde {n} _ {s _ {h} , a _ {h}}}} + \frac {1}{n \bar {d} _ {m}} \\ \lesssim \sqrt {\frac {\operatorname {V a r} _ {\widehat {P} _ {h} (\cdot | s _ {h} , a _ {h})} (V _ {h + 1} ^ {\star} (\cdot))}{\widetilde {n} _ {s _ {h} , a _ {h}}}} + \frac {1}{n \bar {d} _ {m}} \lesssim \sqrt {\frac {\operatorname {V a r} _ {P _ {h} (\cdot | s _ {h} , a _ {h})} (V _ {h + 1} ^ {\star} (\cdot))}{\widetilde {n} _ {s _ {h} , a _ {h}}}} + \frac {1}{n \bar {d} _ {m}} \tag {13} \\ \lesssim \sqrt {\frac {\mathrm {V a r} _ {P _ {h} (\cdot | s _ {h} , a _ {h})} (V _ {h + 1} ^ {\star} (\cdot))}{n d _ {h} ^ {\mu} (s _ {h} , a _ {h})}} + \frac {1}{n \bar {d} _ {m}}. \\ \end{array}
$$
The final bound using non-private statistics results from (12) and (13).
C.2 Proof of the privacy guarantee
The privacy guarantee of DP-APVI (Algorithm 1) is summarized by Lemma C.1 below.
Lemma C.1 (Privacy analysis of DP-APVI (Algorithm 1)). DP-APVI (Algorithm 1) satisfies  $\rho$ -zCDP.
Proof of Lemma C.1. The  $\ell_2$  sensitivity of  $\{n_{s_h,a_h}\}$  is  $\sqrt{2H}$ . According to Lemma 2.7, the Gaussian Mechanism used on  $\{n_{s_h,a_h}\}$  with  $\sigma^2 = \frac{2H}{\rho}$  satisfies  $\frac{\rho}{2}$ -zCDP. Similarly, the Gaussian Mechanism used on  $\{n_{s_h,a_h,s_{h+1}}\}$  with  $\sigma^2 = \frac{2H}{\rho}$  also satisfies  $\frac{\rho}{2}$ -zCDP. Combining these two results, due to the composition of zCDP (Lemma E.16), the construction of  $\{n'\}$  satisfies  $\rho$ -zCDP. Finally, DP-APVI satisfies  $\rho$ -zCDP because the output  $\widehat{\pi}$  is post processing of  $\{n'\}$ .
C.3 Proof of the sub-optimality bound
C.3.1 Utility analysis
First of all, the following Lemma C.2 gives a high probability bound for  $|n' - n|$ .
Lemma C.2. Let  $E_{\rho} = 2\sqrt{2}\sigma \sqrt{\log\frac{4HS^{2}A}{\delta}} = 4\sqrt{\frac{H\log\frac{4HS^{2}A}{\delta}}{\rho}}$ , then with probability  $1 - \delta$ , for all  $s_h, a_h, s_{h+1}$ , it holds that
$$
\left| n _ {s _ {h}, a _ {h}} ^ {\prime} - n _ {s _ {h}, a _ {h}} \right| \leq \frac {E _ {\rho}}{2}, \left| n _ {s _ {h}, a _ {h}, s _ {h + 1}} ^ {\prime} - n _ {s _ {h}, a _ {h}, s _ {h + 1}} \right| \leq \frac {E _ {\rho}}{2}. \tag {14}
$$
Proof of Lemma C.2. The inequalities directly result from the concentration inequality of Gaussian distribution and a union bound.  $\square$
According to the utility analysis above, we have the following Lemma C.3 giving a high probability bound for  $|\widetilde{n} - n|$ .
Lemma C.3. Under the high probability event in Lemma C.2, for all  $s_h, a_h, s_{h+1}$ , it holds that
$$
\left| \widetilde {n} _ {s _ {h}, a _ {h}} - n _ {s _ {h}, a _ {h}} \right| \leq E _ {\rho}, \left| \widetilde {n} _ {s _ {h}, a _ {h}, s _ {h + 1}} - n _ {s _ {h}, a _ {h}, s _ {h + 1}} \right| \leq E _ {\rho}.
$$
Proof of Lemma C.3. When the event in Lemma C.2 holds, the original counts  $\{n_{s_h,a_h,s'}\}_{s' \in S}$  is a feasible solution to the optimization problem, which means that
$$
\max  _ {s ^ {\prime}} \left| \widetilde {n} _ {s _ {h}, a _ {h}, s ^ {\prime}} - n _ {s _ {h}, a _ {h}, s ^ {\prime}} ^ {\prime} \right| \leq \max  _ {s ^ {\prime}} \left| n _ {s _ {h}, a _ {h}, s ^ {\prime}} - n _ {s _ {h}, a _ {h}, s ^ {\prime}} ^ {\prime} \right| \leq \frac {E _ {\rho}}{2}.
$$
Due to the second part of (14), it holds that for any  $s_h, a_h, s_{h+1}$
$$
\left| \widetilde {n} _ {s _ {h}, a _ {h}, s _ {h + 1}} - n _ {s _ {h}, a _ {h}, s _ {h + 1}} \right| \leq \left| \widetilde {n} _ {s _ {h}, a _ {h}, s _ {h + 1}} - n _ {s _ {h}, a _ {h}, s _ {h + 1}} ^ {\prime} \right| + \left| n _ {s _ {h}, a _ {h}, s _ {h + 1}} ^ {\prime} - n _ {s _ {h}, a _ {h}, s _ {h + 1}} \right| \leq E _ {\rho}.
$$
For the second part, because of the constraints in the optimization problem, it holds that
$$
\left| \widetilde {n} _ {s _ {h}, a _ {h}} - n _ {s _ {h}, a _ {h}} ^ {\prime} \right| \leq \frac {E _ {\rho}}{2}.
$$
Due to the first part of (14), it holds that for any  $s_h, a_h$
$$
\left| \widetilde {n} _ {s _ {h}, a _ {h}} - n _ {s _ {h}, a _ {h}} \right| \leq \left| \widetilde {n} _ {s _ {h}, a _ {h}} - n _ {s _ {h}, a _ {h}} ^ {\prime} \right| + \left| n _ {s _ {h}, a _ {h}} ^ {\prime} - n _ {s _ {h}, a _ {h}} \right| \leq E _ {\rho}.
$$
Let the non-private empirical estimate be:
$$
\widehat {P} _ {h} \left(s ^ {\prime} \mid s _ {h}, a _ {h}\right) = \frac {n _ {s _ {h} , a _ {h} , s ^ {\prime}}}{n _ {s _ {h} , a _ {h}}}, \tag {15}
$$
if  $n_{s_h, a_h} > 0$  and  $\widehat{P}_h(s'|s_h, a_h) = \frac{1}{S}$  otherwise. We will show that the private transition kernel  $\widetilde{P}$  is close to  $\widehat{P}$  by the Lemma C.4 and Lemma C.5 below.
Lemma C.4. Under the high probability event of Lemma C.3, for  $s_h, a_h$ , if  $\widetilde{n}_{s_h, a_h} \geq 3E_{\rho}$ , it holds that
$$
\left\| \widetilde {P} _ {h} (\cdot | s _ {h}, a _ {h}) - \widehat {P} _ {h} (\cdot | s _ {h}, a _ {h}) \right\| _ {1} \leq \frac {5 S E _ {\rho}}{\widetilde {n} _ {s _ {h} , a _ {h}}}. \tag {16}
$$
Proof of Lemma C.4. If  $\widetilde{n}_{s_h,a_h} \geq 3E_{\rho}$  and the conclusion in Lemma C.3 hold, we have
$$
\begin{array}{l} \left\| \widetilde {P} _ {h} (\cdot | s _ {h}, a _ {h}) - \widehat {P} _ {h} (\cdot | s _ {h}, a _ {h}) \right\| _ {1} \leq \sum_ {s ^ {\prime} \in \mathcal {S}} \left| \widetilde {P} _ {h} (s ^ {\prime} | s _ {h}, a _ {h}) - \widehat {P} _ {h} (s ^ {\prime} | s _ {h}, a _ {h}) \right| \\ \leq \sum_ {s ^ {\prime} \in \mathcal {S}} \left(\frac {\widetilde {n} _ {s _ {h} , a _ {h} , s ^ {\prime}} + E _ {\rho}}{\widetilde {n} _ {s _ {h} , a _ {h}} - E _ {\rho}} - \frac {\widetilde {n} _ {s _ {h} , a _ {h} , s ^ {\prime}}}{\widetilde {n} _ {s _ {h} , a _ {h}}}\right) \\ \leq \sum_ {s ^ {\prime} \in \mathcal {S}} \left[ \left(\frac {1}{\widetilde {n} _ {s _ {h} , a _ {h}}} + \frac {2 E _ {\rho}}{\widetilde {n} _ {s _ {h} , a _ {h}} ^ {2}}\right) \left(\widetilde {n} _ {s _ {h}, a _ {h}, s ^ {\prime}} + E _ {\rho}\right) - \frac {\widetilde {n} _ {s _ {h} , a _ {h} , s ^ {\prime}}}{\widetilde {n} _ {s _ {h} , a _ {h}}} \right] \tag {17} \\ \leq \frac {S E _ {\rho}}{\widetilde {n} _ {s _ {h} , a _ {h}}} + \frac {2 E _ {\rho}}{\widetilde {n} _ {s _ {h} , a _ {h}}} + \frac {2 S E _ {\rho} ^ {2}}{\widetilde {n} _ {s _ {h} , a _ {h}} ^ {2}} \\ \leq \frac {5 S E _ {\rho}}{\widetilde {n} _ {s _ {h} , a _ {h}}}. \\ \end{array}
$$
The second inequality is because  $\frac{\widetilde{n}_{s_h,a_h,s'} - E_\rho}{\widetilde{n}_{s_h,a_h} + E_\rho} \leq \frac{n_{s_h,a_h,s'}}{n_{s_h,a_h}} \leq \frac{\widetilde{n}_{s_h,a_h,s'} + E_\rho}{\widetilde{n}_{s_h,a_h} - E_\rho}$  and  $\frac{\widetilde{n}_{s_h,a_h,s'} + E_\rho}{\widetilde{n}_{s_h,a_h} - E_\rho} - \frac{\widetilde{n}_{s_h,a_h,s'}}{\widetilde{n}_{s_h,a_h} - E_\rho} \geq \frac{\widetilde{n}_{s_h,a_h,s'} - E_\rho}{\widetilde{n}_{s_h,a_h} + E_\rho}$ . The third inequality is because of Lemma E.6. The last inequality is because  $\widetilde{n}_{s_h,a_h} \geq 3E_\rho$ .
Lemma C.5. Let  $V \in \mathbb{R}^S$  be any function with  $\| V\|_{\infty} \leq H$ , under the high probability event of Lemma C.3, for  $s_h, a_h$ , if  $\widetilde{n}_{s_h,a_h} \geq 3E_{\rho}$ , it holds that
$$
\left| \sqrt {\operatorname {V a r} _ {\widehat {P} _ {h} \left(\cdot \mid s _ {h} , a _ {h}\right)} (V)} - \sqrt {\operatorname {V a r} _ {\widetilde {P} _ {h} \left(\cdot \mid s _ {h} , a _ {h}\right)} (V)}} \right| \leq 4 H \sqrt {\frac {S E _ {\rho}}{\widetilde {n} _ {s _ {h} , a _ {h}}}}. \tag {18}
$$
Proof of Lemma C.5. For  $s_h, a_h$  such that  $\widetilde{n}_{s_h, a_h} \geq 3E_{\rho}$ , we use  $\widetilde{P}(\cdot)$  and  $\widehat{P}(\cdot)$  instead of  $\widetilde{P}_h(\cdot|s_h, a_h)$  and  $\widehat{P}_h(\cdot|s_h, a_h)$  for simplicity. Because of Lemma C.4, we have
$$
\left\| \widetilde {P} (\cdot) - \widehat {P} (\cdot) \right\| _ {1} \leq \frac {5 S E _ {\rho}}{\widetilde {n} _ {s _ {h} , a _ {h}}}.
$$
$$
\begin{array}{l} \left| \sqrt {\operatorname {V a r} _ {\hat {P} (\cdot)} (V)} - \sqrt {\operatorname {V a r} _ {\tilde {P} (\cdot)} (V)} \right| \leq \sqrt {| \operatorname {V a r} _ {\hat {P} (\cdot)} (V) - \operatorname {V a r} _ {\tilde {P} (\cdot)} (V) |} \\ \leq \sqrt {\sum_ {s ^ {\prime} \in \mathcal {S}} \left| \widehat {P} (s ^ {\prime}) - \widetilde {P} (s ^ {\prime}) \right| V (s ^ {\prime}) ^ {2} + \left| \sum_ {s ^ {\prime} \in \mathcal {S}} \left[ \widehat {P} (s ^ {\prime}) + \widetilde {P} (s ^ {\prime}) \right] V (s ^ {\prime}) \right| \cdot \sum_ {s ^ {\prime} \in \mathcal {S}} \left| \widehat {P} (s ^ {\prime}) - \widetilde {P} (s ^ {\prime}) \right| V (s ^ {\prime})} \\ \leq \sqrt {H ^ {2} \left\| \widetilde {P} (\cdot) - \widehat {P} (\cdot) \right\| _ {1} + 2 H ^ {2} \left\| \widetilde {P} (\cdot) - \widehat {P} (\cdot) \right\| _ {1}} \\ \leq 4 H \sqrt {\frac {S E _ {\rho}}{\widetilde {n} _ {s _ {h} , a _ {h}}}}. \tag {19} \\ \end{array}
$$
The second inequality is due to the definition of variance.
C.3.2 Validity of our pessimistic penalty
Now we are ready to present the key lemma (Lemma C.6) below to justify our use of  $\Gamma$  as the pessimistic penalty.
Lemma C.6. Under the high probability event of Lemma C.3, with probability  $1 - \delta$ , for any  $s_h, a_h$ , if  $\widetilde{n}_{s_h, a_h} \geq 3E_\rho$  (which implies  $n_{s_h, a_h} > 0$ ), it holds that
$$
\left| \left(\widetilde {P} _ {h} - P _ {h}\right) \cdot \widetilde {V} _ {h + 1} \left(s _ {h}, a _ {h}\right) \right| \leq \sqrt {\frac {2 \operatorname {V a r} _ {\widetilde {P} _ {h} \left(\cdot \mid s _ {h} , a _ {h}\right)} \left(\widetilde {V} _ {h + 1} (\cdot)\right) \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}} - E _ {\rho}}} + \frac {1 6 S H E _ {\rho} \cdot \iota}{\widetilde {n} _ {s _ {h} , a _ {h}}}, \tag {20}
$$
where  $\widetilde{V}$  is the private version of estimated  $V$  function, which appears in Algorithm 1 and  $\iota = \log (HSA / \delta)$ .