SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\hat{A}^{k + 1}(s,a) = \hat{Q}^{k + 1}(s,a) - \hat{V}^{k + 1}(s)$ . Eq.9 updates the policy  $\pi$  by applying a weighted maximum likelihood method. This is computed by re-weighting state-action samples in  $D$  using the estimated advantage  $\hat{A}^{k + 1}$ . It avoids explicit estimation of the behavior policy, and its resulting sampling errors, which is an important issue in offline RL [12, 4].
Implicit policy constraints. We adopt the advantage-weighted policy update which imposes an implicit KL divergence constraint between  $\pi$  and  $\pi_{\beta}$ . This policy constraint is necessary to guarantee that the next state  $s^{\prime}$  in Eq. 5 can be safely generated through policy  $\pi$ . As derived in [12] (Appendix A), Eq. 9 is a parametric solution of the following problem (where  $\epsilon$  depends on  $\beta$ ):
$$
\max _ {\pi^ {\prime}} \mathbb {E} _ {a \sim \pi^ {\prime} (\cdot | s)} [ \hat {A} ^ {k + 1} (s, a) ]
$$
$$
s. t. \mathrm {D} _ {\mathrm {K L}} (\pi^ {\prime} (\cdot | s) | | \pi_ {\beta} (\cdot | s)) \leq \epsilon , \int_ {a} \pi^ {\prime} (a | s) d a = 1.
$$
Note that  $\mathrm{D}_{\mathrm{KL}}(\pi' || \pi_{\beta})$  is a reserved KL divergence with respect to  $\pi'$ , which is mode-seeking [19]. When treated as Lagrangian it forces  $\pi'$  to allocate its probability mass to the maximum likelihood supports of  $\pi_{\beta}$ , re-weighted by the estimated advantage. In other words, for the space of  $A$  where  $\pi_{\beta}(|s)$  has no samples,  $\pi'(|s)$  has almost zero probability mass too.
Model-based Exploration on Near States. As suggested by remarks in Section 3.1, in practice, allowing the policy to explore the predicted next states transition  $(s\sim D)$  following  $a\sim \pi^{\prime}(\cdot |s))$  leads to better test performance. With this kind of exploration, the policy is updated as follows:
$$
\pi \leftarrow \underset {\pi^ {\prime}} {\arg \min } L _ {\pi} \left(\pi^ {\prime}\right) - \lambda \underset {s ^ {\prime} \sim \hat {P} (s, a)} {\mathbb {E}} \left[ r (s, a) + \gamma \hat {V} ^ {k + 1} \left(s ^ {\prime}\right) \right]. \tag {10}
$$
The second term is an approximation to  $E_{s \sim d_{\pi}(s)}[V^{\pi}(s)]$ . The optimization of this term involves calculating the gradient through the learned dynamics model. This is achieved by employing analytic gradients through the learned dynamics to maximize the value estimates. It is important to note that the value estimates rely on the reward and value predictions, which are dependent on the imagined states and actions. As all these steps are implemented using neural networks, the gradient is analytically computed using stochastic back-propagation, a concept inspired by Dreamer[20]. We adjust the value of  $\lambda$ , a hyper-parameter, to balance between optimistic policy optimization (in maximizing V) and the constrained policy update (as indicated by the first term).
Table 1: Performance comparison on Gym control tasks v2. The results of CSVE are over ten seeds and we reimplement AWAC using d3rlpy. Results of IQL, TD3-BC, and PBRL are from their original papers ( Table 1 in [21], Table C.3 in [22], and Table 1 in [10] respectively). Results of COMBO and CQL are from the reproduction results in [23] (Table 1) and [10] respectively, since their original results were reported on v0 datasets.
AWAC	CQL	CQL-AWR	COMBO	IQL	TD3-BC	PBRL	CSVE
Random	HalfCheetah	13.7	17.5 ± 1.5	16.9 ± 1.5	38.8	18.2	11.0 ± 1.1	13.1 ± 1.2	26.8 ± 1.5
Hopper	8.7	7.9 ± 0.4	8.7 ± 0.5	17.9	16.3	8.5 ± 0.6	31.6 ± 0.3	26.1 ± 7.6
Walker2D	2.2	5.1 ± 1.3	0.0 ± 1.6	7.0	5.5	1.6 ± 1.7	8.8 ± 6.3	6.2 ± 0.8
Medium	HalfCheetah	50.0	47.0 ± 0.5	50.9 ± 0.6	54.2	47.4	48.3 ± 0.3	58.2 ± 1.5	48.4 ± 0.3
Hopper	97.5	53.0 ± 28.5	25.7 ± 37.4	94.9	66.3	59.3 ± 4.2	81.6 ± 14.5	96.7 ± 5.7
Walker2D	89.1	73.3 ± 17.7	62.4 ± 24.4	75.5	78.3	83.7 ± 2.1	90.3 ± 1.2	83.2 ± 1.0
Medium Replay	HalfCheetah	44.9	45.5 ± 0.7	40.0 ± 0.4	55.1	44.2	44.6 ± 0.5	49.5 ± 0.8	54.5 ± 0.6
Hopper	99.4	88.7 ± 12.9	91.0 ± 13.0	73.1	94.7	60.9 ± 18.8	100.7 ± 0.4	91.7 ± 0.2
Walker2D	80.0	83.3 ± 2.7	66.7 ± 12.1	56.0	73.9	81.8 ± 5.5	86.2 ± 3.4	78.0 ± 1.5
Medium Expert	HalfCheetah	62.8	75.6 ± 25.7	73.4 ± 2.0	90.0	86.7	90.7 ± 4.3	93.1 ± 0.2	93.1 ± 0.3
Hopper	87.2	105.6 ± 12.9	102.2 ± 7.7	111.1	91.5	98.0 ± 9.4	111.2 ± 0.7	94.1 ± 3.0
Walker2D	109.8	107.9 ± 1.6	98.0 ± 21.7	96.1	109.6	110.1 ± 0.5	109.8 ± 0.2	109.0 ± 0.1
Expert	HalfCheetah	20.0	96.3 ± 1.3	87.3 ± 8.1	-	94.6	96.7 ± 1.1	96.2 ± 2.3	93.8 ± 0.1
Hopper	111.6	96.5 ± 28.0	110.0 ± 2.5	-	109.0	107.8 ± 7	110.4 ± 0.3	111.3 ± 0.6
Walker2D	110.6	108.5 ± 0.5	75.1 ± 60.7	-	109.4	110.2 ± 0.3	108.8 ± 0.2	108.5 ± 0.1
Average	65.8	67.4	60.6	64.1	69.7	67.5	76.7	74.8
4.3 Discussion on implementation choices
Now we examine the technical considerations for implementing CSVE in a practical algorithm.
Constraints on Policy Extraction. It is important to note that the state value function alone does not suffice to directly derive a policy. There are two methods for extracting a policy from CSVE. The first method is model-based planning, i.e.,  $\pi \gets \arg \max_{\pi} \mathbb{E}_{s \sim d, a \sim \pi(\cdot | s)} [\hat{r}(s, a) + \gamma \mathbb{E}_{s' \sim \hat{P}(s, a)} [V(s')]$ , which involves finding the policy that maximizes the expected future return. However, this method heavily depends on the accuracy of a model and is difficult to implement in practice. As an alternative, we suggest the second method, which learns a Q value or advantage function from the V value function and experience data, and then extracts the policy. Note that CSVE provides no guarantees for conservative estimation on OOD actions, which can cause normal policy extraction methods such as SAC to fail. To address this issue, we adopt policy constraint techniques. On the one hand, during the value estimation in Eq.5, all current states are sampled from the dataset, while the policy is constrained to be close to the behavior policy (ensured via Eq.9). On the other hand, during the policy learning in Eq.10, we use AWR [18] as the primary policy extraction method (first term of Eq.10), which implicitly imposes policy constraints and the additional action exploration (second term of Eq.10) is strictly applied to states in the dataset. This exploration provides a bonus to actions that: (1) themselves and their model-predictive next-states are both close to the dataset (ensured by the dynamics model), and (2) their values are favorable even with conservatism.
Taking Advantage of CSVE. As outlined in Section 3.1, CSVE allows for a more relaxed lower bound on conservative value estimation compared to conservative Q values, providing greater potential for improving the policy. To take advantage of this, the algorithm should enable exploration of out-of-sample but in-distribution states, as described in Section 3. In this paper, we use a deep ensemble dynamics model to support this speculative state exploration, as shown in Eq. 10. The reasoning behind this is as follows: for an in-data state  $s$  and any action  $a \sim \pi(\cdot | s)$ , if the next state  $s'$  is in-data or close to the data support, its value is reasonably estimated, and if not, its value has been penalized according to Eq.5. Additionally, the deep ensemble dynamics model captures epistemic uncertainty well, which can effectively cancel out the impact of rare samples of  $s'$ . By utilizing CSVE, our algorithm can employ the speculative interpolation to further improve the policy. In contrast, CQL and AWAC do not have this capability for such enhanced policy optimization.
Table 2: Performance comparison on Adroit tasks. The results of CSVE are over ten seeds. Results of IQL are from Table 3 in [21] and results of other algorithms are from Table 4 in [10].
AWAC	BC	BEAR	UWAC	CQL	CQL-AWR	IQL	PBRL	CSVE
Human	Pen	18.7	34.4	-1.0	10.1 ± 3.2	37.5	8.4 ± 7.1	71.5	35.4 ± 3.3	106.2 ± 5.0
Hammer	-1.8	1.5	0.3	1.2 ± 0.7	4.4	0.3 ± 0.0	1.4	0.4 ± 0.3	3.5 ± 2.6
Door	-1.8	0.5	-0.3	0.4 ± 0.2	9.9	3.5 ± 1.8	4.3	0.1 ± 0.0	2.8 ± 2.4
Relocate	-0.1	0.0	-0.3	0.0 ± 0.0	0.2	0.1 ± 0.0	0.1	0.0 ± 0.0	0.1 ± 0.0
Cloned	Pen	27.2	56.9	26.5	23.0 ± 6.9	39.2	29.3 ± 7.1	37.3	74.9 ± 9.8	54.5 ± 5.4
Hammer	-1.8	0.8	0.3	0.4 ± 0.0	2.1	0.31 ± 0.06	2.1	0.8 ± 0.5	0.5 ± 0.2
Door	-2.1	-0.1	-0.1	0.0 ± 0.0	0.4	-0.2 ± 0.1	1.6	4.6 ± 4.8	1.2 ± 1.0
Relocate	-0.4	-0.1	-0.3	-0.3 ± 0.2	-0.1	-0.3 ± 0.0	0.0	-0.1 ± 0.0	-0.3 ± 0.0
Expert	Pen	60.9	85.1	105.9	98.2 ± 9.1	107.0	47.1 ± 6.8	117.2	135.7 ± 3.4	144.0 ± 9.4
Hammer	31.0	125.6	127.3	107.7 ± 21.7	86.7	0.2 ± 0.0	124.1	127.5 ± 0.2	126.5 ± 0.3
Door	98.1	34.9	103.4	104.7 ± 0.4	101.5	85.0 ± 15.9	105.2	95.7 ± 12.2	104.2 ± 0.8
Relocate	49.0	101.3	98.6	105.5 ± 3.2	95.0	7.2 ± 12.5	105.9	84.5 ± 12.2	102.9 ± 0.9
Average	23.1	36.7	38.4	37.6	40.3	15.1	47.6	46.6	53.8
This section evaluates the effectiveness of our proposed CSVE algorithm for conservative value estimation in offline RL. In addition, we aim to compare the performance of CSVE with state-of-the-art (SOTA) algorithms. To achieve this, we conduct experimental evaluations on a variety of classic continuous control tasks of Gym[7] and Adroit[8] in the D4RL[9] benchmark.
Our compared baselines include: (1) CQL[4] and its variants, CQL-AWR (Appendix D.2) which uses AWR with extra in-sample exploration as policy extractor, COMBO[6] which extends CQL with model-based rollouts; (2) AWR variants, including AWAC[12] which is a special case of our algorithm with no value penalization (i.e.,  $d = d_u$  in Eq. 2) and exploration on OOD states, IQL[21] which adopts expectile-based conservative value estimation; (3) PBRL[10], a strong algorithm in offline RL, but is quite costly on computation since it uses the ensemble of hundreds of sub-models; (4) other SOTA algorithms with public performance results or high-quality open source implementations, including TD3-BC[22], UWAC[24] and BEAR[25]). Comparing with CQL variants allows us to investigate the advantages of conservative estimation on state values over Q values. By comparing with AWR variants, we distinguish the performance contribution of CSVE from the AWR policy extraction used in our implementation.
5.1 Overall Performance
Evaluation on the Gym Control Tasks. Our method, CSVE, was trained for 1 million steps and evaluated. The results are shown in Table 1. Compared to CQL, CSVE outperforms it in 11 out of 15 tasks, with similar performance on the remaining tasks. Additionally, CSVE shows a consistent advantage on datasets that were generated by following random or sub-optimal policies (random and medium). The CQL-AWR method showed slight improvement in some cases, but still underperforms compared to CSVE. When compared to COMBO, CSVE performs better in 7 out of 12 tasks and similarly or slightly worse on the remaining tasks, which highlights the effectiveness of our method's better bounds on V. Our method has a clear advantage in extracting the best policy on medium and medium-expert tasks. Overall, our results provide empirical evidence that using conservative value estimation on states, rather than Q, leads to improved performance in offline RL. CSVE outperforms AWAC in 9 out of 15 tasks, demonstrating the effectiveness of our approach in exploring beyond the behavior policy. Additionally, our method excels in extracting the optimal policy on data with mixed policies (medium-expert) where AWAC falls short. In comparison to IQL, our method achieves higher scores in 7 out of 9 tasks and maintains comparable performance in the remaining tasks. Furthermore, despite having a significantly lower model capacity and computation cost, CSVE outperforms TD3-BC and is on par with PBRL. These results highlight the effectiveness of our conservative value estimation approach.
Evaluation on the Adroit Tasks. In Table 2, we report the final evaluation results after training 0.1 million steps. As shown, our method outperforms IQL in 8 out of 12 tasks, and is competitive with other algorithms on expert datasets. Additionally, we note that CSVE is the only method that
can learn an effective policy on the human dataset for the Pen task, while maintaining medium performance on the cloned dataset. Overall, our results empirically support the effectiveness of our proposed tighter conservative value estimation in improving offline RL performance.
Effect of Exploration on Near States. We analyze the impact of varying the factor  $\lambda$  in Eq. 10, which controls the intensity on such exploration. We investigated  $\lambda$  values of  $\{0.0, 0.1, 0.5, 1.0\}$  in the medium tasks, fixing  $\beta = 0.1$ . The results are plotted in Fig. 1. As shown in the upper figures,  $\lambda$  has an obvious effect on policy performance and variances during training. With increasing  $\lambda$  from 0, the converged performance gets better in general. However, when the value of  $\lambda$  becomes too large (e.g.,  $\lambda = 3$  for hopper and walker2d), the performance may degrade or even collapse. We further investigated the  $L_{\pi}$  loss as depicted in the bottom figures of Eq. 9, finding that larger  $\lambda$  values negatively impact  $L_{\pi}$ ; however, once  $L_{\pi}$  converges to a reasonable low value, larger  $\lambda$  values lead to performance improvement.
Figure 1: Effect of  $\lambda$  to performance scores (upper figures) and  $L_{\pi}$  losses (bottom figures) in Eq. 9 on medium tasks.
Effect of In-sample Policy Optimization. We examined the impact of varying the factor  $\beta$  in Eq. 9 on the balance between behavior cloning and in-sample policy optimization. We tested different  $\beta$  values on Mujoco medium datasets, as shown in Fig.2. The results indicate that  $\beta$  has a significant effect on the policy performance during training. Based on our findings, a value of  $\beta = 3.0$  was found to be suitable for medium datasets. Additionally, in our implementation, we use  $\beta = 3.0$  for random and medium tasks, and  $\beta = 0.1$  for medium-replay, medium-expert, and expert datasets. More details can be found in the ablation study in the appendix.
The main idea behind offline RL algorithms is to incorporate conservatism or regularization into the online RL algorithms. Here, we briefly review prior work and compare it to our approach.
Conservative Value Estimation: Prior offline RL algorithms regularize the learning policy to be close to the data or to an explicitly estimated behavior policy. and penalize the exploration
Figure 2: Effect of  $\beta$  to performance scores on medium tasks.
of the OOD region, via distribution correction estimation [26, 27], policy constraints with support matching [11] and distributional matching [1, 25], applying policy divergence based penalty on Q-functions [28, 29] or uncertainty-based penalty [30] on Q-functions and conservative Q-function estimation [4]. Besides, model-based algorithms [15] directly estimate dynamics uncertainty and translate it into reward penalty. Different from this prior work that imposes conservatism on state-action pairs or actions, ours directly does such conservative estimation on states and requires no explicit uncertainty quantification.
In-Sample Algorithms: AWR [18] updates policy constrained on strictly in-sample states and actions, to avoid extrapolation on out-of-support points. IQL[21] uses expectile-based regression to do value estimation and AWR for its policy updates. AWAC[12], whose actor is AWR, is an actor-critic algorithm to accelerate online RL with offline data. The major drawback of AWR method when used for offline RL is that the in-sample policy learning limits the final performance.
Model-Based Algorithms: Model-based offline RL learns the dynamics model from the static dataset and uses it to quantify uncertainty [15], data augmentation [6] with roll-outs, or planning [16, 31]. Such methods typically rely on wide data coverage when planning and data augmentation with roll-outs, and low model estimation error when estimating uncertainty, which is difficult to satisfy in reality and leads to policy instability. Instead, we use the model to sample the next-step states only reachable from data, which has no such strict requirements on data coverage or model bias.
Theoretical Results: Our theoretical results are derived from conservative Q-value estimation (CQL) and safe policy improvement [32]. Compared to offline policy evaluation[33], which aims to provide a better estimation of the value function, we focus on providing a better lower bound. Additionally, hen the dataset is augmented with model-based roll-outs, COMBO [6] provides a more conservative yet tighter value estimation than CQL. CSVE achieves the same lower bounds as COMBO but under more general state distributions.
In this paper, we propose CSVE, a new approach for offline RL based on conservative value estimation on states. We demonstrated how its theoretical results can lead to more effective algorithms. In particular, we develop a practical actor-critic algorithm, in which the critic achieves conservative state value estimation by incorporating the penalty of the model predictive next-states into Bellman iterations, and the actor does the advantage-weighted policy updates enhanced via model-based state exploration. Experimental evaluation shows that our method performs better than alternative methods based on conservative Q-function estimation and is competitive among the SOTA methods, thereby validating our theoretical analysis. Moving forward, we aim to delve deeper into designing more powerful algorithms grounded in conservative state value estimation.
We first redefine notation for clarity and then provide the proofs of the results from the main paper.
Notation. Let  $k \in N$  denotes an iteration of policy evaluation (in Section 3.2).  $V^k$  denotes the true, tabular (or functional) V-function iterate in the MDP.  $\hat{V}^k$  denotes the approximate tabular (or functional) V-function iterate.
The empirical Bellman operator is defined as:
$$
\left(\hat {B} ^ {\pi} \hat {V} ^ {k}\right) (s) = E _ {a \sim \pi (a | s)} \hat {r} (s, a) + \gamma \sum_ {s ^ {\prime}} E _ {a \sim \pi (a | s)} \hat {P} \left(s ^ {\prime} \mid s, a\right) \left[ \hat {V} ^ {k} \left(s ^ {\prime}\right) \right] \tag {11}
$$
where  $\hat{r}(s, a)$  is the empirical average reward derived from the dataset when performing action  $a$  at state  $s$ . The true Bellman operator is given by:
$$
\left(\mathcal {B} ^ {\pi} V ^ {k}\right) (s) = E _ {a \sim \pi (a | s)} r (s, a) + \gamma \sum_ {s ^ {\prime}} E _ {a \sim \pi (a | s)} P \left(s ^ {\prime} \mid s, a\right) \left[ V ^ {k} \left(s ^ {\prime}\right) \right] \tag {12}
$$
Now we first prove that the iteration in Eq.2 has a fixed point. Assume the state value function is lower bounded, i.e.,  $V(s) \geq C \forall s \in S$ , then Eq.2 can always be solved with Eq.3. Thus, we only need to investigate the iteration in Eq.3.
Defining this iteration as a function operator  $\mathcal{T}^{\pi}$  on  $V$  and supposing that  $\mathrm{supp}d\subseteq \mathrm{supp}d_u$ , it's evident that the operator  $\mathcal{T}^{\pi}$  displays a  $\gamma$ -contraction within  $L_{\infty}$  norm where  $\gamma$  is the discounting factor.
Proof of Lemma 3.1: Let  $V$  and  $V'$  be any two state value functions with the same support, i.e.,  $\operatorname{supp} V = \operatorname{supp} V'$ .
$$
\begin{array}{l} \left| \left(\mathcal {T} ^ {\pi} V - \mathcal {T} ^ {\pi} V ^ {\prime}\right) (s) \right| = \left| \left(\hat {\mathcal {B}} ^ {\pi} V (s) - \alpha \left[ \frac {d _ {(s)}}{d _ {u} (s)} - 1 \right]\right) - \left(\hat {\mathcal {B}} ^ {\pi} V ^ {\prime} (s) - \alpha \left[ \frac {d _ {(s)}}{d _ {u} (s)} - 1 \right]\right) \right| \\ = \left| \hat {\mathcal {B}} ^ {\pi} V (s) - \hat {\mathcal {B}} ^ {\pi} V ^ {\prime} (s) \right| \\ = | (E _ {a \sim \pi (a | s)} \hat {r} (s, a) + \gamma E _ {a \sim \pi (a | s)} \sum_ {s ^ {\prime}} \hat {P} (s ^ {\prime} | s, a) V (s ^ {\prime})) \\ - \left(E _ {a \sim \pi (a | s)} \hat {r} (s, a) + \gamma E _ {a \sim \pi (a | s)} \sum_ {s ^ {\prime}} \hat {P} (s ^ {\prime} | s, a) V ^ {\prime} (s ^ {\prime})\right) | \\ = \gamma \left| E _ {a \sim \pi (a | s)} \sum_ {s ^ {\prime}} \hat {P} (s ^ {\prime} | s, a) [ V (s ^ {\prime}) - V ^ {\prime} (s ^ {\prime}) ] \right| \\ \end{array}
$$
$$
\begin{array}{l} \left| \left| \mathcal {T} ^ {\pi} V - \mathcal {T} ^ {\pi} V ^ {\prime} \right| \right| _ {\infty} = \max  _ {s} \left| \left(\mathcal {T} ^ {\pi} V - \mathcal {T} ^ {\pi} V ^ {\prime}\right) (s) \right| \\ = \max _ {s} \gamma \left| E _ {a \sim \pi (a | s)} \sum_ {s ^ {\prime}} \hat {P} (s ^ {\prime} | s, a) [ V (s ^ {\prime}) - V ^ {\prime} (s ^ {\prime}) ] \right| \\ \leq \gamma E _ {a \sim \pi (a | s)} \sum_ {s ^ {\prime}} \hat {P} (s ^ {\prime} | s, a) \max _ {s ^ {\prime \prime}} | V (s ^ {\prime \prime}) - V ^ {\prime} (s ^ {\prime \prime}) | \\ = \gamma \max  _ {s ^ {\prime \prime}} | V (s ^ {\prime \prime}) - V ^ {\prime} (s ^ {\prime \prime}) | \\ = \gamma | | (V - V ^ {\prime}) | | _ {\infty} \\ \end{array}
$$
We provide a bound on the difference between the empirical Bellman operator and the true Bellman operator. Following previous work [4], we make the following assumptions. Let  $P^{\pi}$  be the transition matrix associated with the policy, specifically,  $P^{\pi}V(s) = E_{a' \sim \pi(a'|s'), s' \sim P(s'|s, a')} [V(s')]$
Assumption A.1.  $\forall s, a \in \mathcal{M}$ , the relationship below holds with at least a  $(1 - \delta)$ $(\delta \in (0,1))$  probability,
$$
\left| r - r (s, a) \right| \leq \frac {C _ {r , \delta}}{\sqrt {| D (s , a) |}}, \left\| \hat {P} \left(s ^ {\prime} \mid s, a\right) - P \left(s ^ {\prime} \mid s, a\right) \right\| _ {1} \leq \frac {C _ {P , \delta}}{\sqrt {| D (s , a) |}} \tag {13}
$$
Given this assumption, the absolute difference between the empirical Bellman operator and the true one can be deduced as follows:
$$
\begin{array}{l} \left| \left(\hat {\mathcal {B}} ^ {\pi}\right) \hat {V} ^ {k} - \left(\mathcal {B} ^ {\pi}\right) \hat {V} ^ {k}\right) | = E _ {a \sim \pi (a | s)} | r - r (s, a) + \gamma \sum_ {s ^ {\prime}} E _ {a ^ {\prime} \sim \pi \left(a ^ {\prime} \mid s ^ {\prime}\right)} (\hat {P} \left(s ^ {\prime} \mid s, a\right) - P \left(s ^ {\prime} \mid s, a\right)) [ \hat {V} ^ {k} \left(s ^ {\prime} \mid s, a\right) ] \\ \leq E _ {a \sim \pi (a | s)} | r - r (s, a) | + \gamma | \sum_ {s ^ {\prime}} E _ {a ^ {\prime} \sim \pi \left(a ^ {\prime} \mid s ^ {\prime}\right)} \left(\hat {P} \left(s ^ {\prime} \mid s, a ^ {\prime}\right) - P \left(s ^ {\prime} \mid s, a ^ {\prime}\right)\right) [ \hat {V} ^ {k} \left(s ^ {\prime}\right) ] (15) \\ \leq E _ {a \sim \pi (a | s)} \frac {C _ {r , \delta} + \gamma C _ {P , \delta} 2 R _ {\max } / (1 - \gamma)}{\sqrt {| D (s , a) |}} (16) \\ \end{array}
$$
The error in estimation due to sampling can therefore be bounded by a constant, dependent on  $C_{r,\delta}$  and  $C_{t,\delta}$ . We define this constant as  $C_{r,T,\delta}$ .
Thus we obtain:
$$
\forall V, s \in D, | \hat {\mathcal {B}} ^ {\pi} V (s) - \mathcal {B} ^ {\pi} V (s) | \leq E _ {a \sim \pi (a | s)} \frac {C _ {r , t , \delta}}{(1 - \gamma) \sqrt {| D (s , a) |}} \tag {17}
$$
Next we provide an important lemma.
Lemma A.2. (Interpolation Lemma) For any  $f \in [0,1]$ , and any given distribution  $\rho(s)$ , let  $d_f$  be an  $f$ -interpolation of  $\rho$  and  $D$ , i.e.,  $d_f(s) := fd(s) + (1 - f)\rho(s)$ , let  $v(\rho, f) := E_{s \sim \rho(s)}[\frac{\rho(s) - d(s)}{d_f(s)}]$ , then  $v(\rho, f)$  satisfies  $v(\rho, f) \geq 0$ .
The proof can be found in [6]. By setting  $f$  as 1, we have  $E_{s\sim \rho (s)}[\frac{\rho(s) - d(s)}{d(s)} ] > 0$ .
Proof of Theorem 3.2: The V function of approximate dynamic programming in iteration  $k$  can be obtained as:
$$
\hat {V} ^ {k + 1} (s) = \hat {\mathcal {B}} ^ {\pi} \hat {V} ^ {k} (s) - \alpha \left[ \frac {d (s)}{d _ {u} (s)} - 1 \right] \forall s, k \tag {18}
$$
The fixed point:
$$
\hat {V} ^ {\pi} (s) = \hat {\mathcal {B}} ^ {\pi} \hat {V} ^ {\pi} (s) - \alpha \left[ \frac {d (s)}{d _ {u} (s)} - 1 \right] \leq \mathcal {B} ^ {\pi} \hat {V} ^ {\pi} (s) + E _ {a \sim \pi (a | s)} \frac {C _ {r , t , \delta} R _ {\max }}{(1 - \gamma) \sqrt {| D (s , a) |}} - \alpha \left[ \frac {d (s)}{d _ {u} (s)} - 1 \right] \tag {19}
$$