SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiayi Guan
Tongji University
guanziayi@tongji.edu.cn
Guang Chen*
guangchen@tongji.edu.cn
Jiaming Ji
jiamg.ji@gmail.com
Long Yang
yanglong001@pku.edu.cn
Ao Zhou
221107@tongji.edu.cn
Zhijun Li
zjli@ieee.org
Changjun Jiang
cjjiang@tongji.edu.cn
Offline safe reinforcement learning (RL) algorithms promise to learn policies that satisfy safety constraints directly in offline datasets without interacting with the environment. This arrangement is particularly important in scenarios with high sampling costs and potential dangers, such as autonomous driving and robotics. However, the influence of safety constraints and out-of-distribution (OOD) actions have made it challenging for previous methods to achieve high reward returns while ensuring safety. In this work, we propose a Variational Optimization with Conservative Estimation algorithm (VOCE) to solve the problem of optimizing safety policies in the offline dataset. Concretely, we reframe the problem of offline safe RL using probabilistic inference, which introduces variational distributions to make the optimization of policies more flexible. Subsequently, we utilize pessimistic estimation methods to estimate the Q-value of cost and reward, which mitigates the extrapolation errors induced by OOD actions. Finally, extensive experiments demonstrate that the VOCE algorithm achieves competitive performance across multiple experimental tasks, particularly outperforming state-of-the-art algorithms in terms of safety. Our code is available at github.VOCE.
Reinforcement learning (RL) algorithms have made remarkable achievements in various fields such as robot control [1-3] and strategy games [4-6]. However, limited by the online and trial-and-error nature, standard RL is challenging to apply to dangerous and expensive scenarios [7-9]. Offline safe RL is a promising and potential approach to tackle the above problem, which learns policy satisfying safety constraints from pre-collected offline datasets without interacting with the environment [10, 11].
Since offline safe RL methods aim to learn policies that satisfy safety constraints from offline datasets, which requires the agent to not only comply with safety constraints but also consider the influence of out-of-distribution (OOD) actions [12-15]. This makes it difficult for the algorithm to learn policies that achieve high reward returns while satisfying safety constraints [16]. There are two main
approaches to deal with the aforementioned challenges: one is based on linear programming [17, 18], and the other is based on exploration and evaluation [19-21]. Linear programming provides a way to research offline safe RL problems, but it heavily depends on the F-divergence or KL-divergence to constrain the distance between the optimal distribution and the sample distribution [17]. Therefore, it is difficult for linear programming to eliminate the extrapolation errors caused by OOD actions [10, 11]. The exploration evaluation approaches are to introduce conservative Q-value estimation under the actor-critic framework. Although exploration evaluation approaches avoid the overestimation issue through lower bound Q-values, they prematurely terminate trajectories that exceed the constraints during policy updates [16]. The exploration evaluation method with strict constraints on the sample space makes it challenging to search for reward-maximizing policies. In conclusion, existing methods face challenges in learning policies from offline datasets that maximize reward while satisfying safety constraints, particularly when considering the impact of extrapolation errors.
In this work, to solve the constrained policy optimization problem and eliminate the extrapolation errors caused by OOD actions, we propose a variational optimization with conservative estimates for the offline safe RL algorithm. Concretely, we reframe the objective of offline safe RL using probabilistic inference, enhancing the flexibility of policy optimization by replacing parameterized policies with variational distributions. Furthermore, to avoid extrapolation errors caused by OOD actions, we derive upper and lower bounds for Q-values and estimate the Q-values of costs and rewards based on these bounds. Finally, we adopt a supervised learning approach to train a parameterized policy network according to the variational distribution of policies. The main contributions of this work are listed as follows:
- We utilize probabilistic inference to address the problem of offline safe RL, which introduces non-parametric variational distributions to replace parameterized policies, providing increased flexibility for optimizing safe policies in offline datasets.
- We derived upper and lower bounds for Q-value estimation using the pessimistic estimation approach. Subsequently, we utilized these bounds to estimate the Q-value of costs and rewards, respectively, to avoid extrapolation errors caused by OOD actions.
- We carry out extensive comparative experiments, and the results indicate that the VOCE algorithm outperforms state-of-the-art algorithms, especially in terms of safety.
Constrained Markov decision processes (CMDP) provide a theoretical framework to solve safe RL problems [22], where the agent is cost-bounded by safety constraints. A CMDP is defined as a tuple  $(S, \mathcal{A}, C, P, r, \rho_0, \gamma)$ , where  $S \in \mathbb{R}^n$  is the state space,  $\mathcal{A} \in \mathbb{R}^m$  is the action space,  $P: S \times \mathcal{A} \times S \to [0,1]$  is the transition kernel, which specifies the transition probability  $p(s_{t+1} | s_t, a_t)$  from state  $s_t$  to state  $s_{t+1}$  under the action  $a_t$ ,  $r: S \times \mathcal{A} \to \mathbb{R}$  represents the reward function,  $C$  is the set of costs  $\{c_i: S \times \mathcal{A} \to \mathbb{R}_+, i = 1, 2, \dots, m\}$  for violating  $m$  constraints,  $\gamma \in (0,1]$  is the discount factor, and  $\rho_0: S \to [0,1]$  is the distribution of initial states. A policy  $\pi$  maps a probability distribution from state  $s_t$  to action  $a_t$ . We utilize  $\pi_\theta$  to denote the parameterization of the policy with parameter  $\theta$ . In safe RL, the goal is to maximize the cumulative reward while satisfying safe constraints:
$$
\pi^ {*} = \arg \max  _ {\pi} \mathbb {E} _ {\tau \sim \pi} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, a _ {t}\right) \right], \quad \text {s . t .} \quad \mathbb {E} _ {\tau \sim \pi} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} c _ {i} \left(s _ {t}, a _ {t}\right) \right] \leq \bar {c} _ {i}, \tag {1}
$$
where the  $\tau = \{s_0,a_0,\dots \} \sim \pi$  denotes the trajectory.  $\bar{c}_i$  is the  $i$ -th safe constraint limit.
In the offline RL setting, we learn a policy according to the offline dataset  $D$  collected by one or more data-collection policies, and without online interaction with the environment  $(D = \{(s_t, a_t, r_t, c_t)_i\}_{i=1}^n)$ . Although this way of not interacting with the environment brings a lot of advantages, the offline dataset can not cover all action-state transitions, and the policy evaluation step actually utilizes the Bellman equation of a single sample [23]. This makes the estimated Q-value vulnerable to OOD actions, which in turn severely affects the performance of the algorithm [24, 25]. As shown in Fig. 1, under the offline setting, off-policy safe RL methods neither learn a high-reward policy nor guarantee safety. Additionally, although offline RL methods obtain high rewards under safe expert data, it is difficult to guarantee safety because it directly ignores the cost. We can conclude that both offline RL and safe RL face challenges in learning policies that satisfy safety constraints
from offline datasets. Therefore, it is important to design an algorithm that learns high-reward and satisfies safety constraints, which is the focus of this paper.
Figure 1: Reward and cost curves of safe RL and off-policy RL under high-reward and safe datasets in Point-Button task [26]. The solid line is the mean of the performance, and the shaded line represents the variance of the performance. All performance curves employ more than 3 random seeds.
In this section, we present the details of Variationally Optimized with Conservatively Estimated for offline-safe RL algorithm (VOCE), which is the main contribution of this paper. We first reformulate the objective of offline-safe RL with probabilistic inference. Based on this, we derived upper and lower bounds for Q-value estimation, and adopted a pessimistic and conservative estimation approach to estimate the Q-value of costs and rewards respectively to eliminate the extrapolation error of OOD actions. Finally, we employ supervised learning to transform the variational distribution into a parameterized policy.
3.1 Offline-safe RL as Inference
From the probabilistic inference perspective, the offline safe RL can be viewed as the problem of observing safe actions in states with high rewards. As the probabilistic inference model and notation in [26, 27], we introduce an optimal variable  $\mathcal{O}$  to represent the event of maximizing rewards. Assuming the likelihood of a given trajectory being optimal is proportional to the cumulative discounted reward, the infinite discounted reward formula is expressed as  $\mathcal{P}(\mathcal{O} = 1|\tau)\propto \sum_t\gamma^t r(s_t,a_t)$ . Since  $\mathcal{P}(\mathcal{O} = 1|\tau)\geq 0$ , we further confirm that the likelihood of a given trajectory being optimal is proportional to the exponential of the accumulative discounted reward, then we rewrite the infinite discounted reward formula as  $\mathcal{P}(\mathcal{O} = 1|\tau)\propto \exp (\sum_t\gamma^t r_t / \alpha)$ . The  $\alpha$  is a temperature parameter, and  $r_t$  is short for  $r(s_{t},a_{t})$ . Let  $P_{\pi}(\tau)$  be the probability of the trajectory under the policy  $\pi$ , then the log-likelihood of optimality under policy  $\pi$  can be written as  $\log \mathcal{P}_{\pi}(\mathcal{O} = 1) = \log \sum_{\tau}\mathcal{P}(\mathcal{O} = 1|\tau)P_{\pi}(\tau)$ . Furthermore, according to importance sampling and Jensen's inequality, we obtain a lower bound of the log-likelihood of optimality under policy  $\pi$ .
$$
\begin{array}{l} \log \mathcal {P} _ {\pi} (\mathcal {O} = 1) = \log \mathbb {E} _ {\tau \sim \mathcal {Q}} \left[ \frac {\mathcal {P} (\mathcal {O} = 1 | \tau) P _ {\pi} (\tau)}{\mathcal {Q} (\tau)} \right] \geq \mathbb {E} _ {\tau \sim \mathcal {Q}} \log \frac {\mathcal {P} (\mathcal {O} = 1 | \tau) P _ {\pi} (\tau)}{\mathcal {Q} (\tau)} \tag {2} \\ \propto \mathbb {E} _ {\tau \sim \mathcal {Q}} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r _ {t} \right] - \alpha D _ {K L} (\mathcal {Q} (\tau) | | P _ {\pi} (\tau)), \\ \end{array}
$$
where the  $\mathcal{Q}(\tau)$  is the auxiliary trajectory distribution. Since offline safe RL not only needs to consider maximizing cumulative rewards but also guaranteeing safety, we restrict  $\mathcal{Q}(\tau)$  to the constrained feasible distribution space. According to the constraint threshold  $\bar{c}_i$ , we define the feasible distribution space as:
$$
\Omega^ {\bar {c} _ {i}} \triangleq \left\{q \left(a _ {t} \mid s _ {t}\right): \mathbb {E} _ {\tau \sim q} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} c _ {i} \left(s _ {t}, a _ {t}\right) \right] \leq \bar {c} _ {i} \right\}, \tag {3}
$$
where each  $q(a_{t}|s_{t}) \in \Omega^{\bar{c}_{i}}$  indicates that the action distribution of the safe-condition satisfies the safety constraint. By factorizing the trajectory distribution [27], we express the trajectory distribution as:
$$
\mathcal {Q} (\tau) = \rho (s _ {0}) \prod_ {t \geq 0} p \left(s _ {t + 1} \mid s _ {t}, a _ {t}\right) q \left(a _ {t} \mid s _ {t}\right), q \left(a _ {t} \mid s _ {t}\right) \in \Omega^ {\bar {c} _ {i}}, \tag {4}
$$
$$
P _ {\pi_ {\theta}} (\tau) = \rho (s _ {0}) \prod_ {t \geq 0} p \left(s _ {t + 1} \mid s _ {t}, a _ {t}\right) \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right), \tag {5}
$$
where  $\rho(s_0)$  is the distribution of the initial state  $s_0$ . Substituting Eq. (4) and (5) into Eq. (2) to eliminate the transitions  $p(s_{t+1}|s_t, a_t)$ , and combining the feasible distribution space of Eq. (3), we obtain the object of offline safe RL problem as shown in Proposition 3.1.
Proposition 3.1. The objective of the offline safe RL problem can be defined through probabilistic inference as follows:
$$
\mathcal {L} (q, \pi) = \mathbb {E} _ {\tau \sim q} \left[ \sum_ {t = 0} ^ {\infty} \left(\gamma^ {t} r _ {t}\right) - \alpha D _ {K L} \left(q (\cdot | s _ {t}) \mid \mid \pi_ {\theta} (\cdot | s _ {t})\right) \right], q \left(a _ {t} | s _ {t}\right) \in \Omega^ {\bar {c} _ {i}}. \tag {6}
$$
Proposition 3.1 provides a new objective for the problem of offline safe RL. The aforementioned probabilistic inference method has many optimizations over the previous dual methods in solving the problem of offline safe RL. This approach provides flexibility for optimizing the optimal policy by introducing a variational distribution  $q(a_{t}|s_{t})$  to connect the strong correlation between dual variable optimization and policy improvement. Furthermore, the approach decouples the optimal policy optimization problem into independent processes of optimal variational distribution optimization and parameterized policy update. We will introduce these two update processes in detail later.
3.2 Variational Optimization with Conservative Estimates for Offline-safe RL
The previous standard RL and safe RL also employed the above-mentioned idea of probabilistic inference to solve the policy optimization problem [26, 28]. However, due to the influence of OOD actions in the offline setting, previous RL algorithms based on probabilistic inference are difficult to address the tasks of offline-safe RL. Therefore, we adopt a pessimistic conservative estimation method to improve the offline-safe RL algorithm based on probabilistic inference and obtain a Variational Optimization with Conservative Estimation algorithm (VOCE). We divide the VOCE into two steps: conservatively estimated variational optimization and parameterized policy update.
3.2.1 Conservatively Estimated Variational Optimization
The goal of the conservatively estimated variational optimization is to optimize the optimal variational distribution  $q(a_{t}|s_{t})$  with a high reward while satisfying the safety constraints. In this step, we perform the maximization of the variational distribution  $q(a_{t}|s_{t})$  by fixing the policy  $\pi_{\theta}(a_{t}|s_{t})$ . According to proposition 3.1, we write the objective of solving the optimal variational distribution as:
$$
\mathcal {L} (q) = \mathbb {E} _ {\rho (s _ {0})} \left[ \mathbb {E} _ {q \left(a _ {t} \mid s _ {t}\right)} Q ^ {r} \left(s _ {t}, a _ {t}\right) - \alpha D _ {K L} \left(q (\cdot \mid s _ {t}) \mid \mid \pi_ {\theta} (\cdot \mid s _ {t})\right) \right], \quad \text {s . t .} \mathbb {E} _ {\rho (s _ {0})} \left[ \mathbb {E} _ {q \left(a _ {t} \mid s _ {t}\right)} Q ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) \right] \leq \bar {c} _ {i}. \tag {7}
$$
The optimization objective of the above variational distribution can be regarded as a KL-regularized constrained optimization problem. However, since the expected reward term  $\mathbb{E}_{q(a_t|s_t)}Q^r (s_t,a_t)$  may be on different scales depending on the environment, it is difficult to set an appropriate penalty coefficient  $\alpha$  for different environments. Therefore, we introduce hard constraints to replace the regularization term of the KL-divergence and rewrite the optimization objective of Eq. (7) as:
$$
\begin{array}{l} \max  _ {q} \mathbb {E} _ {\rho (s _ {0})} \left[ \sum_ {a _ {t}} q (a _ {t} | s _ {t}) Q ^ {r} (s _ {t}, a _ {t}) \right], \quad \text {s . t .} \mathbb {E} _ {\rho (s _ {0})} \left[ \sum_ {a _ {t}} q (a _ {t} | s _ {t}) Q ^ {c _ {i}} (s _ {t}, a _ {t}) \right] \leq \bar {c} _ {i}, \\ \mathbb {E} _ {\rho (s _ {0})} \left[ D _ {K L} \left(q \left(a _ {t} \mid s _ {t}\right) \mid \mid \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right)\right) \right] \leq \epsilon , \quad \sum_ {a _ {t}} q \left(a _ {t} \mid s _ {t}\right) = 1, \forall s _ {t} \in \mathcal {D}, \tag {8} \\ \end{array}
$$
where  $\epsilon$  is the KL-divergence threshold between the variational distribution  $q(a_{t}|s_{t})$  and the parameterized policy  $\pi_{\theta}(a_t|s_t)$ . To solve the above-constrained problem (8), we replace the parametric  $q(a_{t}|s_{t})$  with the non-parametric to avoid performance degradation caused by approximation errors [29]. Furthermore, we apply the Lagrange multiplier method to transform the above-constrained problem into an unconstrained problem. In the non-parametric form  $q(a_{t}|s_{t})$ , since the objective function is linear and the constraints are convex, the constrained optimization problem shown in Eq. (8) is a convex optimization problem. Furthermore, through mild assumptions, we can obtain the strong dual form of Eq. (9).
Assumption 3.2. (Slater's condition). There exists a variational distribution that satisfies the safety constraints  $q(a_{t}|s_{t}) \in \Omega^{\bar{c}_{i}}$  within the current policy trust region  $D_{KL}(q(a_{t}|s_{t})||\pi_{\theta}(a_{t}|s_{t})) < \epsilon$ .
Lemma 3.3. Under Assumption 3.2, there exists a strong duality between the primal problem in Eq. (8) and the unconstrained problem in Eq. (9).
$$
\begin{array}{l} \mathcal {L} (q, \lambda , \eta , \delta) = \min  _ {\lambda \geq 0, \eta \geq 0, \delta} \max  _ {q \geq 0} \mathbb {E} _ {\rho (s _ {0})} \left[ \sum_ {a _ {t}} q \left(a _ {t} \mid s _ {t}\right) Q ^ {r} \left(s _ {t}, a _ {t}\right) - \lambda^ {T} \left[ \sum_ {a _ {t}} q \left(a _ {t} \mid s _ {t}\right) Q ^ {c _ {i}} \left(s _ {t}, a _ {t}\right) - \bar {c} _ {i} \right] \right. \tag {9} \\ \left. - \delta \left[ \sum_ {a _ {t}} q \left(a _ {t} \mid s _ {t}\right) - 1 \right] - \eta \left[ D _ {K L} \left(q \left(a _ {t} \mid s _ {t}\right) \mid \mid \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right)\right) - \epsilon \right] \right], \\ \end{array}
$$
where  $\lambda, \eta, \delta$  are the Lagrange multipliers of the above multiple constraints. Based on the strong duality of the unconstrained problem in Eq. (9), we further derive a closed-form solution  $q^{*}(a_{t}|s_{t})$  for its internal maximization, as shown in Proposition 3.4. Proofs and discussions are in Appendix B.1.
Proposition 3.4. The closed-form solution of the optimal variational distribution that satisfies the safety constraint in Eq. (9) is given as:
$$
q ^ {*} \left(a _ {t} \mid s _ {t}\right) = \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) \exp \left[ \frac {Q ^ {r c} \left(s _ {t} , a _ {t}\right)}{\eta} \right] \exp \left(- \frac {\eta + \delta}{\eta}\right), \tag {10}
$$
where  $Q^{rc}(s_t,a_t)\triangleq Q^r (s_t,a_t) - \lambda^T Q^{c_i}(s_t,a_t)$
By substituting the closed-form solution (10) into Eq. (9) and eliminating similar terms, we obtain the convex minimization problem shown in Proposition 3.5. Subsequently, we solve for the dual variables  $\eta$  and  $\lambda$  using Eq. (11). Proofs and discussions are in Appendix B.2.
Proposition 3.5. The  $\lambda, \eta$  are dual variables and are solved via the convex optimization problem of the following formula.
$$
\mathcal {L} (\lambda , \eta) = \min  _ {\lambda \geq 0, \eta \geq 0} \mathbb {E} _ {\rho (s _ {0})} \left[ \eta \log \sum_ {a _ {t}} \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) \exp \left(\frac {Q ^ {r c} \left(s _ {t} , a _ {t}\right)}{\eta}\right) + \lambda^ {T} \bar {c} _ {i} + \eta \epsilon \right]. \tag {11}
$$
The Proposition 3.4 provides a proposal to solve a non-parametric variational distribution  $q(a_{t}|s_{t})$  for the given the Q-value  $Q^{r}(s_{t},a_{t})$  and  $Q^{c_i}(s_t,a_t)$ . In addition, we provide the optimality and uniqueness of the above closed-form  $q^{*}(a_{t}|s_{t})$  solution under the premise of strong convexity in the Appendix B.2. Note that it can be seen from Eq. (10) that providing accurate  $Q^r (s_t,a_t)$  and  $Q^{c_i}(s_t,a_t)$  are the premise and guarantee for accurately computing the non-parametric variational distribution  $q(a_{t}|s_{t})$ . We can find similar conclusions in the online safe RL algorithm CVPO [26], as described in Proposition 3.4 and 3.5. In the online setting, the empirical Bellman equation is directly applied to iteratively update the Q-value. However, in the offline setting, this approach would lead to clear extrapolation errors in  $Q^{c_i}(s_t,a_t)$  and  $Q^{r}(s_{t},a_{t})$  due to the OOD actions. Furthermore, it is difficult to accurately compute the variational distribution  $q(a_{t}|s_{t})$  according to Eq. (10).
To eliminate the extrapolation error caused by the OOD actions during the evaluation of the Q-value, we utilize the pessimistic conservative estimation approach to estimate  $Q^r$  and  $Q^{c_i}$ , respectively. Specifically, to eliminate the impact of extrapolation error on  $Q^r$ , we need to avoid overestimation of  $Q^r$  [24]. Therefore, similar to CQL [30] to learn a conservative lower bound of the Q-function by additionally minimizing the Q-values alongside a standard Bellman error objective, we choose the penalty that minimizes the expected Q-value under special action-state transitions for unknown actions produced by  $\pi_{\mathcal{M}}(a_t|s_t)$ . Then, we define the Q-value of the iterative reward function as:
$$
\hat {Q} _ {k + 1} ^ {r} \leftarrow \arg \min  \frac {1}{2} \mathbb {E} _ {s _ {t}, a _ {t} \sim D} \left[ \left[ \left(Q ^ {r} (s _ {t}, a _ {t})\right) - \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} ^ {r} (s _ {t}, a _ {t}) \right] ^ {2} \right] + \kappa \mathbb {E} _ {s _ {t} \sim D, a _ {t} \sim \pi_ {\mathcal {M}} (\cdot | s _ {t})} \left[ Q ^ {r} (s _ {t}, a _ {t}) \right], \tag {12}
$$
where the  $\hat{B}^{\pi}\hat{Q}^{r}(s_{t},a_{t}) = r(s_{t},a_{t}) + \gamma \hat{Q}^{r}(s_{t + 1},a_{t + 1})$ . In Proposition 3.6, we show that  $\hat{Q}^{r}(s_{t},a_{t})$  converges to a lower bound on  $\hat{Q}^{r}(s_{t},a_{t})$ . However, we can tighten this bound if we are interested in  $V^{r}(s_{t})$ . We improve the bounds by introducing an additional maximization term under the sample distribution  $\hat{\pi}_{\beta}$ . Then the iterative update Eq. (12) of the reward Q-value can be rewritten as:
$$
\begin{array}{l} \hat {Q} _ {k + 1} ^ {r} \leftarrow \arg \min  \frac {1}{2} \mathbb {E} _ {s _ {t}, a _ {t} \sim D} \left[ \left[ \left(Q ^ {r} \left(s _ {t}, a _ {t}\right)\right) - \hat {\mathcal {B}} ^ {\pi} \hat {Q} _ {k} ^ {r} \left(s _ {t}, a _ {t}\right) \right] ^ {2} \right] + \tag {13} \\ \kappa \left[ \mathbb {E} _ {\substack {s _ {t} \sim D \\ a _ {t} \sim \pi_ {\mathcal{M}} (\cdot | s _ {t})}} Q ^ {r} (s _ {t}, a _ {t}) - \mathbb {E} _ {\substack {s _ {t} \sim D \\ a _ {t} \sim \hat {\pi} _ {\beta} (\cdot | s _ {t})}} Q ^ {r} (s _ {t}, a _ {t}) \right], \\ \end{array}
$$
where  $\kappa$  is a tradeoff factor. Note that the Eq. (12) and (13) utilize the empirical Bellman operator  $\hat{B}^{\pi}$  instead of the actual Bellman Operator  $B^{\pi}$ . Following the related work [23, 31], we employ the
concentration properties to compute the error. For any  $\forall s_t, a_t \in D$ , with the probability  $\geq 1 - \delta$ , the sampling error can be written as:
$$
\left| \mathcal {B} ^ {\pi} \hat {Q} \left(s _ {t}, a _ {t}\right) - \hat {\mathcal {B}} ^ {\pi} \hat {Q} \left(s _ {t}, a _ {t}\right) \right| \leq \frac {C _ {r , p , \delta} R _ {\max }}{\left(1 - \gamma\right) \sqrt {\left| \mathcal {D} \left(s _ {t} , a _ {t}\right) \right|}}, \tag {14}
$$
where  $C_{r,p,\delta}$  is a constant depending on the concentration properties of  $r(s_t, a_t)$ ,  $p(s_{t+1}|s_t, a_t)$  and the  $\delta, \delta \in (0,1)$ . The  $\frac{1}{\sqrt{|\mathcal{D}(s_t, a_t)|}}$  represents a vector of size  $|\mathcal{S}||\mathcal{A}|$  containing the square root inverse count of each state action transition. Considering the sampling error as shown in Eq. (14), we can derive the condition for  $\hat{Q}^r$  converging to the lower-bound  $Q^r$  at all  $(s_t, a_t)$  through Eq. (12). The Proposition 3.6 provides the condition for  $\hat{Q}^r$  to converge to the lower-bound of the  $Q^r$ . Proofs and discussions are in Appendix B.3.