SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Liting Chen*
98chenliting@gmail.com
Jie Yan
dasistyanjie@gmail.com
Zhengdao Shao*
University of Sci. and Tech. of China Hefei, China
zhengdaoshao@mail.ustc.edu.cn
Lu Wang
wlu@microsoft.com
Qingwei Lin
qlin@microsoft.com
Saravan Rajmohan
Seattle, USA
saravar@microsoft.com
Thomas Moscibroda
Redmond, USA
moscitho@microsoft.com
Dongmei Zhang
dongmeiz@microsoft.com
Offline reinforcement learning faces a significant challenge of value over-estimation due to the distributional drift between the dataset and the current learned policy, leading to learning failure in practice. The common approach is to incorporate a penalty term to reward or value estimation in the Bellman iterations. Meanwhile, to avoid extrapolation on out-of-distribution (OOD) states and actions, existing methods focus on conservative Q-function estimation. In this paper, we propose Conservative State Value Estimation (CSVE), a new approach that learns conservative V-function via directly imposing penalty on OOD states. Compared to prior work, CSVE allows more effective state value estimation with conservative guarantees and further better policy optimization. Further, we apply CSVE and develop a practical actor-critic algorithm in which the critic does the conservative value estimation by additionally sampling and penalizing the states around the dataset, and the actor applies advantage weighted updates extended with state exploration to improve the policy. We evaluate in classic continual control tasks of D4RL, showing that our method performs better than the conservative Q-function learning methods and is strongly competitive among recent SOTA methods.
Reinforcement Learning (RL) learns to act by interacting with the environment and has shown great success in various tasks. However, in many real-world situations, it is impossible to learn from scratch online as exploration is often risky and unsafe. Instead, offline RL([1, 2]) avoids this problem by learning the policy solely from historical data. Yet, simply applying standard online RL techniques to static datasets can lead to overestimated values and incorrect policy decisions when faced with unfamiliar or out-of-distribution (OOD) scenarios.
*Work done during the internship at Microsoft. † Work done during full-time employment at Microsoft.
Recently, the principle of conservative value estimation has been introduced to tackle challenges in offline RL[3, 4, 5]. Prior methods, e.g., CQL(Conservative Q-Learning [4]), avoid the value over-estimation problem by systematically underestimating the Q values of OOD actions on the states in the dataset. In practice, it is often too pessimistic and thus leads to overly conservative algorithms. COMBO [6] leverages a learned dynamic model to augment data in an interpolation way. This process helps derive a Q function that's less conservative than CQL, potentially leading to more optimal policies.
In this paper, we propose CSVE (Conservative State Value Estimation), a novel offline RL approach. Unlike the above methods that estimate conservative values by penalizing the Q-function for OOD actions, CSVE directly penalizes the V-function for OOD states. We theoretically demonstrate that CSVE provides tighter bounds on in-distribution state values in expectation than CQL, and same bounds as COMBO but under more general discounted state distributions, which potentially enhances policy optimization in the data support. Our main contributions include:
- The conservative state value estimation with related theoretical analysis. We prove that it is lower bounds the real state values in expectation over any state distribution that is used to sample OOD states and is up-bounded by the real state values in expectation over the marginal state distribution of the dataset plus a constant term depending on sampling errors. Compared to prior work, it enhances policy optimization with conservative value guarantees.
- A practical actor-critic algorithm implemented CSVE. The critic undertakes conservative state value estimation, while the actor uses advantage-weighted regression(AWR) and explores states with conservative value guarantee to improve policy. In particular, we use a dynamics model to sample OOD states that are directly reachable from the dataset, for efficient value penalizing and policy exploring.
- Experimental evaluation on continuous control tasks of Gym [7] and Adroit [8] in D4RL [9] benchmarks, showing that CSVE performs better than prior methods based on conservative Q-value estimation, and is strongly competitive among main SOTA algorithms.
Offline Reinforcement Learning. Consider the Markov Decision Process  $M \coloneqq (S, \mathcal{A}, P, r, \rho, \gamma)$ , which comprises the state space  $S$ , the action space  $\mathcal{A}$ , the transition model  $P: S \times \mathcal{A} \to \Delta(S)$ , the reward function  $r: S \times \mathcal{A} \to \mathbb{R}$ , the initial state distribution  $\rho$  and the discount factor  $\gamma \in (0, 1]$ . A stochastic policy  $\pi: S \to \mathcal{A}$  selects an action probabilistically based on the current state. A transition is the tuple  $(s_t, a_t, r_t, s_{t+1})$  where  $a_t \sim \pi(\cdot | s_t)$ ,  $s_{t+1} \sim P(\cdot | s_t, a_t)$ , and  $r_t = r(s_t, a_t)$ . It's assumed that the reward values adhere to  $|r(s, a)| \leq R_{max}$ ,  $\forall s, a$ . A trajectory under  $\pi$  is the random sequence  $\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_T)$  which consists of continuous transitions starting from  $s_0 \sim \rho$ .
Standard RL is to learn a policy  $\pi \in \Pi$  that maximize the expected cumulative future rewards, represented as  $J_{\pi}(M) = \mathbb{E}_{M,\pi}[\sum_{t=0}^{\infty}\gamma^{t}r_{t}]$ , through active interaction with the environment  $M$ . At any time  $t$ , for the policy  $\pi$ , the value function of state is defined as  $V^{\pi}(s) \coloneqq \mathbb{E}_{M,\pi}[\sum_{k=0}^{\infty}\gamma^{t+k}r_{t+k}|s_{t}=s]$ , and the Q value function is  $Q^{\pi}(s,a) \coloneqq \mathbb{E}_{M,\pi}[\sum_{k=0}^{\infty}\gamma^{t+k}r_{t+k}|s_{t}=s,a_{t}=a]$ . The Bellman operator is a function projection:  $\mathcal{B}^{\pi}Q(s,a) \coloneqq r(s,a) + \gamma \mathbb{E}_{s' \sim P(\cdot|s,a),a' \sim \pi(\cdot|s')} [Q(s',a')]$ , or  $\mathcal{B}^{\pi}V(s) \coloneqq \mathbb{E}_{a \sim \pi(\cdot|s)}[r(s,a) + \gamma \mathbb{E}_{s' \sim P(\cdot|s,a)}[V(s')]]$ , resulting iterative value updates. Bellman consistency implies that  $V^{\pi}(s) = \mathcal{B}^{\pi}V^{\pi}(s), \forall s$  and  $Q^{\pi}(s) = \mathcal{B}^{\pi}Q^{\pi}(s,a), \forall s,a$ . When employing function approximation in practice, the empirical Bellman operator  $\hat{B}^{\pi}$  is used, wherein the aforementioned expectations are estimated with data. Offline RL aims to learn the policy  $\pi$  from a static dataset  $D = \{(s,a,r,s')\}$  made up of transitions collected by any behavior policy, with the objective of performing well in the online setting. Note that, unlike standard online RL, offline RL does not interact with the environment during the learning process.
Conservative Value Estimation. One main challenge in offline RL arises from the over-estimation of values due to extrapolation in unseen states and actions. Such overestimation can lead to the deterioration of the learned policy. To address this issue, conservatism or pessimism is employed in value estimation. For instance, CQL learns a conservative Q-value function by penalizing the value
of unseen actions:
$$
\hat {Q} ^ {k + 1} \leftarrow \arg \min  _ {Q} \frac {1}{2} \mathbb {E} _ {s, a, s ^ {\prime} \sim D} [ (Q (s, a) - \hat {\beta} _ {\pi} \hat {Q} ^ {k} (s, a)) ^ {2} ] + \alpha \left(\mathbb {E} _ {a \sim \mu (\cdot | s)} [ Q (s, a) ] - \mathbb {E} _ {a \sim \hat {\pi} _ {\beta} (\cdot | s)} [ Q (s, a) ]\right) \tag {1}
$$
where  $\hat{\pi}_{\beta}$  and  $\pi$  are the behaviour policy and learnt policy separately,  $\mu$  is an arbitrary policy different from  $\hat{\pi}_{\beta}$ , and  $\alpha$  represents the factor for balancing conservatism.
Constrained Policy Optimization. To address the issues of distribution shift between the learning policy and the behavior policy, one approach is to constrain the learning policy close to the behavior policy [10, 11, 12, 13, 1]. As an example, AdvantageWeighted Regression(AWR)[14, 12] employs an implicit KL divergence to regulate the distance between policies:
$$
\pi^ {k + 1} \leftarrow \operatorname * {a r g   m a x} _ {\pi} \mathbb {E} _ {s, a \sim D} \left[ \frac {\log \pi (a | s)}{Z (s)} \exp \left(\frac {1}{\lambda} A ^ {\pi^ {k}} (s, a)\right) \right]
$$
Here,  $A^{\pi^k}$  is the advantage of policy  $\pi^k$ , and  $Z$  serves as the normalization constant for  $s$ .
Model-based Offline RL. In RL, the model is an approximation of the MDP  $M$ . Such a model is denoted as  $\hat{M} \coloneqq (S, \mathcal{A}, \hat{P}, \hat{r}, \rho, \gamma)$ , with  $\hat{P}$  and  $\hat{r}$  being approximation of  $P$  and  $r$  respectively. Within offline RL, the model is commonly used to augment data [15, 6] or act as a surrogate of the real environment during interaction [16]. However, such practices can inadvertently introduce bootstrapped errors over extended horizons[17]. In this paper, we restrict the use of the model to one-step sampling on the next states that are approximately reachable from the dataset.
3 Conservative State Value Estimation
In the offline setting, the value overestimation is a major problem resulting in failure of learning a reasonable policy [13, 1]. In contrast to prior works[4, 6] that get conservative value estimation via penalizing Q function for OOD state-action pairs, we directly penalize V function for OOD states. Our approach provides several novel theoretic results that allow better trade-off of conservative value estimation and policy improvement. All proofs of our theorems can be found in Appendix A.
3.1 Conservative Off-policy Evaluation
We aim to conservatively estimate the value of a target policy using a dataset to avoid overestimation of OOD states. To achieve this, we penalize V-values evaluated on states that are more likely to be OOD and increase the V-values on states that are in the distribution of the dataset. This adjustment is made iteratively:
$$
\hat {V} ^ {k + 1} \leftarrow \arg \min  _ {V} \frac {1}{2} \mathbb {E} _ {s \sim d _ {u} (s)} \left[ \left(\hat {B} ^ {\pi} \hat {V} ^ {k} (s) - V (s)\right) ^ {2} \right] + \alpha \left(\mathbb {E} _ {s ^ {\prime} \sim d (s)} V \left(s ^ {\prime}\right) - \mathbb {E} _ {s \sim d _ {u} (s)} V (s)\right) \tag {2}
$$
where  $d_{u}(s)$  is the discounted state distribution of D,  $d(s)$  is any state distribution, and  $\hat{B}^{\pi}$  is the empirical Bellman operator (see appendix for more details). Considering the setting without function approximation, by setting the derivative of Eq. 2 as zero, we can derive the V function using approximate dynamic programming at iteration  $k$ :
$$
\hat {V} ^ {k + 1} (s) = \hat {\mathcal {B}} ^ {\pi} \hat {V} ^ {k} (s) - \alpha \left[ \frac {d _ {(s)}}{d _ {u} (s)} - 1 \right], \forall s, k. \tag {3}
$$
Denote the function projection on  $\hat{V}^k$  in Eq. 3 as  $\mathcal{T}^{\pi}$ . We have Lemma 3.1, which ensures that  $\hat{V}^k$  converges to a unique fixed point.
Lemma 3.1. For any  $d$  with  $\operatorname{supp} d \subseteq \operatorname{supp} d_u$ ,  $\mathcal{T}^{\pi}$  is a  $\gamma$ -contraction in  $L_{\infty}$  norm.
Theorem 3.2. For any  $d$  with  $\operatorname{supp} d \subseteq \operatorname{supp} d_u$  ( $d \neq d_u$ ), with a sufficiently large  $\alpha$  (i.e.,  $\alpha \geq \mathbb{E}_{s \sim d(s)} \mathbb{E}_{a \sim \pi(a|s)} \frac{C_{r,t,s} R_{max}}{(1 - \gamma) \sqrt{|D(s,a)|}} / \mathbb{E}_{s \sim d(s)}[\frac{d(s)}{d_u(s)} - 1]$ ), the expected value of the estimation  $\hat{V}^{\pi}(s)$  under  $d(s)$  is the lower bound of the true value, that is:  $\mathbb{E}_{s \sim d(s)}[\hat{V}^{\pi}(s)] \leq \mathbb{E}_{s \sim d(s)}[V^{\pi}(s)]$ .
$\hat{V}^{\pi}(s) = \lim_{k\to \infty}\hat{V}^{k}(s)$  is the converged value estimation with the dataset  $D$ , and  $\frac{C_{r,t,\delta}R_{max}}{(1 - \gamma)\sqrt{|D(s,a)|}}$  is related to sampling error that arises when using the empirical operator instead of the Bellman operator. If the counts of each state-action pair is greater than zero,  $|\bar{D} (s,a)|$  denotes a vector of size  $|\mathcal{S}||\mathcal{A}|$  containing counts for each state-action pair. If the counts of this state action pair is zero, the corresponding  $\frac{1}{\sqrt{|D(s,a)|}}$  is a large yet finite value. We assume that with probability  $\geq 1 - \delta$ , the sampling error is less than  $\frac{C_{r,t,\delta}R_{max}}{(1 - \gamma)\sqrt{|D(s,a)|}}$ , while  $C_{r,t,\delta}$  is a constant (See appendix for more details.) Note that if the sampling error can be disregarded,  $\alpha >0$  can ensure the lower bound results.
Theorem 3.3. The expected value of the estimation,  $\hat{V}^{\pi}(s)$ , under the state distribution of the original dataset is the lower bound of the true value plus the term of irreducible sampling error. Formally:  $\mathbb{E}_{s\sim d_u(s)}[\hat{V}^\pi (s)]\leq \mathbb{E}_{s\sim d_u(s)}[V^\pi (s)] + \mathbb{E}_{s\sim d_u(s)}(I - \gamma P^\pi)^{-1}\mathbb{E}_{a\sim \pi (a|s)}\frac{C_{r,t,\delta}R_{max}}{(1 - \gamma)\sqrt{|D(s,a)|}}.$
where  $P^{\pi}$  refers to the transition matrix coupled with policy  $\pi$  (see Appendix for details).
Now we show that, during iterations, the gap between the estimated V-function values of in-distribution states and OOD states is higher compared to the true V-functions.
Theorem 3.4. For any iteration  $k$ , given a sufficiently large  $\alpha$ , our method amplifies the difference in expected  $V$ -values between the selected state distribution and the dataset state distribution. This can be represented as:  $\mathbb{E}_{s\sim d_u(s)}[\hat{V}^k (s)] - \mathbb{E}_{s\sim d(s)}[\hat{V}^k (s)] > \mathbb{E}_{s\sim d_u(s)}[V^k (s)] - \mathbb{E}_{s\sim d(s)}[V^k (s)]$ .
Our approach, which penalizes the V-function for OOD states, promotes a more conservative estimate of a target policy's value in offline reinforcement learning. Consequently, our policy extraction ensures actions align with the dataset's distribution.
To apply our approach effectively in offline RL algorithms, the preceding theorems serve as guiding principles. Here are four key insights for practical use of Eq. 2:
Remark 1. According to Eq. 2, if  $d = d_u$ , the penalty for OOD states diminishes. This means that the policy will likely avoid states with limited data support, preventing it from exploring unseen actions in such states. While AWAC [12] employs this configuration, our findings indicate that by selecting a  $d$ , our method surpasses AWAC's performance.
Remark 2. Theorem 3.3 suggests that under  $d_{u}$ , the marginal state distribution of data, the expectation estimated value of  $V^{\pi}$  is either lower than its true value or exceed it, but within a certain limit. This understanding drives our adoption of the advantage-weighted policy update, as illustrated in Eq. 9.
Remark 3. As per Theorem 3.2, the expected estimated value of a policy under  $d$ , which represents the discounted state distribution of any policy, must be a lower bound of its true value. Grounded in this theorem, our policy enhancement strategy merges an advantage-weighted update with an additional exploration bonus, showcased in Eq. 10.
Remark 4. Theorem 3.4 states  $\mathbb{E}_{s\sim d(s)}[V^k (s)] - \mathbb{E}_{s\sim d(s)}[\hat{V}^k (s)] > \mathbb{E}_{s\sim d_u(s)}[V^k (s)] -$ $\mathbb{E}_{s\sim d_u(s)}[\hat{V}^k (s)]$  . In simpler terms, the underestimation of value is more pronounced under  $d$  With the proper choice of  $d$  , we can confidently formulate a newer and potentially superior policy using  $\hat{V}^k$  . Our algorithm chooses the distribution of model predictive next-states as  $d$  , i.e.,  $s^\prime \sim d$  is implemented by  $s\sim D,a\sim \pi (\cdot |s),s'\sim \hat{P} (\cdot |s,a)$  , which effectively builds a soft 'river' with low values encircling the dataset.
Comparison with prior work: CQL (Eq.1), which penalizes Q-function of OOD actions, guarantees the lower bounds on state-wise value estimation:  $\hat{V}^{\pi}(s) = E_{\pi(a|s)}(\hat{Q}^{\pi}(s,a)) \leq E_{\pi(a|s)}(Q^{\pi}(s,a)) = V^{\pi}(s)$  for all  $s \in D$ . COMBO, which penalizes the Q-function for OOD states and actions of interpolation of history data and model-based roll-outs, guarantees the lower bound of state value expectation:  $\mathbb{E}_{s \sim \mu_0}[\hat{V}^{\pi}(s)] \leq \mathbb{E}_{s \sim \mu_0}[V^{\pi}(s)]$  where  $\mu_0$  is the initial state distribution (Remark 1, section A.2 of COMBO [6]); which is a special case of our result in Theorem 3.2 when  $d = \mu_0$ . Both CSVE and COMBO intend to enhance performance by transitioning from individual state values to expected state values. However, CSVE offers the same lower bounds but under a more general state distribution. Note that  $\mu_0$  depends on the environment or the dynamic model during offline training. CSVE's flexibility, represented by  $d$ , ensures conservative guarantees across any discounted state distribution of the learned policy, emphasizing a preference for penalizing  $V$  over the Q-function.
3.2 Safe Policy Improvement Guarantees
Now we show that our method has the safe policy improvement guarantees against the data-implied behaviour policy. We first show that our method optimizes a penalized RL empirical objective:
Theorem 3.5. Let  $\hat{V}^{\pi}$  be the fixed point of Eq. 3, then  $\pi^{*}(a|s) = \arg \max_{\pi} \hat{V}^{\pi}(s)$  is equivalently obtained by solving:
$$
\pi^ {*} \leftarrow \arg \max  _ {\pi} J (\pi , \hat {M}) - \frac {\alpha}{1 - \gamma} _ {s \sim d _ {\hat {M}} ^ {\pi}} [ \frac {d (s)}{d _ {u} (s)} - 1 ]. \tag {4}
$$
Building upon Theorem 3.5, we show that our method provides a  $\zeta$ -safe policy improvement over  $\pi_{\beta}$ .
Theorem 3.6. Let  $\pi^{*}(a|s)$  be the policy obtained in Eq. 4. Then, it is a  $\zeta$ -safe policy improvement over  $\hat{\pi}^{\beta}$  in the actual MDP  $M$ , i.e.,  $J(\pi^{*},M)\geq J(\hat{\pi}^{\beta},M) - \zeta$  with high probability 1-  $\delta$ , where  $\zeta$  is given by:
$$
\begin{array}{l} \zeta = 2 \left(\frac {C _ {r , \delta}}{1 - \gamma} + \frac {\gamma R _ {m a x} C _ {T , \delta}}{(1 - \gamma) ^ {2}}\right) \underset {s \sim d _ {\tilde {M}} ^ {\pi} (s)} {\mathbb {E}} \left[ c \sqrt {\underset {a \sim \pi (a | s)} {\mathbb {E}} [ \frac {\pi (a | s)}{\pi_ {\beta} (a | s)} ]} \right] \\ - \underbrace {(J (\pi^ {*}, \hat {M}) - J (\hat {\pi} _ {\beta} , \hat {M}))} _ {\geq \alpha \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\hat {M}} ^ {\pi} (s)} [ \frac {d (s)}{d _ {u} (s)} - 1 ]} w h e r e c = \sqrt {| \mathcal {A} |} / \sqrt {| \mathcal {D} (s) |}. \\ \end{array}
$$
In this section, we propose a practical actor-critic algorithm that employs CSVE for value estimation and extends Advantage Weighted Regression[18] with out-of-sample state exploration for policy improvement. In particular, we adopt a dynamics model to sample OOD states during conservative value estimation and exploration during policy improvement. The implementation details are in Appendix B. Besides, we discuss the general technical choices of applying CSVE into algorithms.
4.1 Conservative Value Estimation
Given a dataset  $D$  acquired by the behavior policy  $\pi_{\beta}$ , our objective is to estimate the value function  $V^{\pi}$  for a target policy  $\pi$ . As stated in section 3, to prevent the value overestimation, we learn a conservative value function  $\hat{V}^{\pi}$  that lower bounds the real values of  $\pi$  by adding a penalty for OOD states within the Bellman projection sequence. Our method involves iterative updates of Equations 5 - 7, where  $\hat{Q}^k$  is the target network of  $\hat{Q}^k$ .
$$
\begin{array}{l} \hat {V} ^ {k + 1} \leftarrow \underset {V} {\arg \min } L _ {V} ^ {\pi} (V; \overline {{\hat {Q} ^ {k}}}) \tag {5} \\ = \mathbb{E}_{s\sim D}\left[(\mathbb{E}_{a\sim \pi (\cdot |s)}[\overline{\hat{Q}^{k}} (s,a)] - V(s))^{2}\right] + \alpha \left(\mathbb{E}_{\substack{s\sim D,a\sim \pi (\cdot |s)\\ s^{\prime}\sim \hat{P} (s,a)}}[V(s^{\prime})] - \mathbb{E}_{s\sim D}[V(s)]\right) \\ \end{array}
$$
$$
\begin{array}{l} \hat {Q} ^ {k + 1} \leftarrow \underset {Q} {\arg \min } L _ {Q} ^ {\pi} (Q; \hat {V} ^ {k + 1}) = \underset {s, a, s ^ {\prime} \sim D} {\mathbb {E}} \left[ \left(r (s, a) + \gamma \hat {V} ^ {k + 1} \left(s ^ {\prime}\right) - Q (s, a)\right) ^ {2} \right] (6) \\ \overline {{\hat {Q} ^ {k + 1}}} \leftarrow (1 - \omega) \overline {{\hat {Q} ^ {k}}} + \omega \hat {Q} ^ {k + 1} (7) \\ \end{array}
$$
The RHS of Eq. 5 is an approximation of Eq. 2, with the first term representing the standard TD error. In this term, the target state value is estimated by taking the expectation of  $\widehat{Q}^k$  over  $a\sim \pi$ , and the second term penalizes the value of OOD states. In Eq. 6, the RHS is TD errors estimated on transitions in the dataset  $D$ . Note that the target term is the sum of the reward  $r(s,a)$  and the next step state's value  $\hat{V}^{k + 1}(s')$ . In Eq. 7, the target Q values are updated with a soft interpolation factor  $\omega \in (0,1)$ .  $\widehat{Q}^k$  changes slower than  $\hat{Q}^k$ , which makes the TD error estimation in Eq. 5 more stable.
Constrained Policy. Note that in RHS of Eq. 5, we use  $a \sim \pi(\cdot|s)$  in expectation. To safely estimate the target value of  $V(s)$  by  $\mathbb{E}_{a \sim \pi(\cdot|s)}[\hat{Q}(s,a)]$ , we almost always requires  $\mathrm{supp}(\pi(\cdot|s)) \subset$
$\mathrm{supp}(\pi_{\beta}(\cdot |s))$ . We achieve this by the advantage weighted policy update, which forces  $\pi (\cdot |s)$  to have significant probability mass on actions taken by  $\pi_{\beta}$  in data, as detailed in section 3.2.
Model-based OOD State Sampling. In Eq. 5, we implement the state sampling process  $s' \sim d$  in Eq. 2 as a flow of  $\{s \sim D; a \sim \pi(a|s), s' \sim \hat{P}(s'|s, a)\}$ , that is the distribution of the predictive next-states from  $D$  by following  $\pi$ . This approach proves beneficial in practice. On the one hand, this method is more efficient as it samples only the states that are approximately reachable from  $D$  by one step, rather than sampling the entire state space. On the other hand, we only need the model to do one-step prediction such that it introduces no bootstrapped errors from long horizons. Following previous work [17, 15, 6], We use an ensemble of deep neural networks, represented as  $p\theta^1, \ldots, p\theta^B$ , to implement the probabilistic dynamics model. Each neural network produces a Gaussian distribution over the next state and reward:  $P_{\theta}^{i}(s_{t+1}, r|s_t, a_t) = \mathcal{N}(u_{\theta}^{i}(s_t, a_t), \sigma_{\theta}^{i}(s_t, a_t))$ .
Adaptive Penalty Factor  $\alpha$ . The pessimism level is controlled by the parameter  $\alpha \geq 0$ . In practice, we set  $\alpha$  adaptive during training as follows, which is similar to that in CQL([4])
$$
\max  _ {\alpha \geq 0} \left[ \alpha \left(\mathbb {E} _ {s ^ {\prime} \sim d} \left[ V _ {\psi} \left(s ^ {\prime}\right) \right] - \mathbb {E} _ {s \sim D} \left[ V _ {\psi} (s) \right] - \tau\right) \right], \tag {8}
$$
where  $\tau$  is a budget parameter. If the expected difference in V-values is less than  $\tau$ ,  $\alpha$  will decrease. Otherwise,  $\alpha$  will increase, penalizing the OOD state values more aggressively.
4.2 Advantage Weighted Policy Update
After learning the conservative  $\hat{V}^{k + 1}$  and  $\hat{Q}^{k + 1}$  (or  $\hat{V}^{\pi}$  and  $\hat{Q}^{\pi}$  when the values have converged), we improve the policy by the following advantage weighted update [12].
$$
\pi \leftarrow \underset {\pi^ {\prime}} {\arg \min } L _ {\pi} \left(\pi^ {\prime}\right) = - \underset {s, a \sim D} {\mathbb {E}} \left[ \log \pi^ {\prime} (a | s) \exp \left(\beta \hat {A} ^ {k + 1} (s, a)\right) \right] \tag {9}
$$