SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yixiu Mao $^{1}$ , Hongchang Zhang $^{1}$ , Chen Chen $^{1}$ , Yi Xu $^{2}$ , Xiangyang Ji $^{1}$ \
 $^{1}$ Department of Automation, Tsinghua University\
 $^{2}$ School of Artificial Intelligence, Dalian University of Technology\
myx21@mails.tsinghua.edu, xyji@tsinghua.edu
Offline reinforcement learning suffers from the extrapolation error and value overestimation caused by out-of-distribution (OOD) actions. To mitigate this issue, value regularization approaches aim to penalize the learned value functions to assign lower values to OOD actions. However, existing value regularization methods lack a proper distinction between the regularization effects on in-distribution (ID) and OOD actions, and fail to guarantee optimal convergence results of the policy. To this end, we propose Supported Value Regularization (SVR), which penalizes the  $Q$ -values for all OOD actions while maintaining standard Bellman updates for ID ones. Specifically, we utilize the bias of importance sampling to compute the summation of  $Q$ -values over the entire OOD region, which serves as the penalty for policy evaluation. This design automatically separates the regularization for ID and OOD actions without manually distinguishing between them. In tabular MDP, we show that the policy evaluation operator of SVR is a contraction, whose fixed point outputs unbiased  $Q$ -values for ID actions and underestimated  $Q$ -values for OOD actions. Furthermore, the policy iteration with SVR guarantees strict policy improvement until convergence to the optimal support-constrained policy in the dataset. Empirically, we validate the theoretical properties of SVR in a tabular maze environment and demonstrate its state-of-the-art performance on a range of continuous control tasks in the D4RL benchmark.
Offline Reinforcement Learning (RL) aims to learn a policy from a fixed dataset collected by some behavior policy [24, 26]. It can tap into existing large-scale datasets [15, 30] for safe and efficient learning. However, it suffers from the extrapolation error [10] caused by out-of-distribution (OOD) actions, which can further result in severe overestimation of value functions [26].
To mitigate this issue, value regularization approaches attempt to penalize the  $Q$ -values of OOD actions to introduce conservatism in value estimation [23, 20, 46, 14, 3, 6]. However, we observe that existing value regularization methods not only fall short in penalizing all OOD  $Q$ -values but also may introduce detrimental changes to in-distribution (ID) ones. Specifically, most of them involve adding penalties to the critic learning objective [23, 50, 27, 6, 48]. However, due to the difficulty of distinguishing between ID and OOD actions, they typically adopt the idea of being pessimistic about the actions under the current policy and optionally optimistic about the ones within the dataset [23, 50, 46, 6]. Rather than relying on ID or OOD, this regularization is essentially based on the policy density, which we show is problematic when the dataset is heavily corrupted by suboptimal actions. Other works involve reducing the Bellman target with uncertainty quantifiers [14, 3] or ensembles [2, 34]. However, if not incorporating additional penalties, they do not provide learning signals on OOD  $Q$ -values, logically only suppressing the overestimation in the ID region [14, 2].
In this work, we revisit the original objective of value regularization and ask a question - "Can we devise a value regularization method that penalizes all OOD  $Q$ -values without affecting ID ones"? We point out that the inability of prior methods to achieve this leads to a lack of strong theoretical guarantees for policy performance. In offline RL, the best policy that can be guaranteed should lie within the support of the behavior policy, known as the optimal support-constrained policy [22]. However, existing value regularization methods do not provide reliable guarantees for converging to it. On the other hand, several endeavors aim to learn it through specific policy constraints [10, 22, 11], but their empirical performance leaves considerable room for improvement.
To this end, we propose Supported Value Regularization (SVR), a new value regularization method that penalizes the  $Q$ -values for all OOD actions while maintaining standard Bellman updates for ID ones. Specifically, we leverage the bias of importance sampling to calculate the summation of  $Q$ -values over the entire OOD region, which serves as the penalty term for policy evaluation. This design circumvents the dilemma of distinguishing between ID and OOD actions and separates the regularization effects on them automatically. Theoretically, SVR offers stronger guarantees than previous methods. Under tabular MDP, we show that the SVR policy evaluation operator is a contraction in the whole state-action space and its fixed point outputs unbiased  $Q$ -values for ID actions and underestimated  $Q$ -values for OOD ones, while the fixed point of  $Q$  in CQL [23] may underestimate or overestimate  $Q$ -values. More importantly, SVR guarantees strict policy improvement until convergence to the optimal support-constrained policy, while prior value regularization methods lack such each-step improvement and global optimal convergence guarantees.
In practice, SVR is easy to implement by adding a penalty term to the ordinary policy evaluation loss. Empirically, we validate the support-constrained optimality of SVR in a tabular maze environment, where baselines fail to converge. Moreover, SVR achieves state-of-the-art performance on a range of continuous control tasks in the D4RL benchmark [7] and shows strong advantages on noisy datasets.
Offline RL. The environment in RL is typically modeled as a Markov Decision Process (MDP)  $\mathcal{M} = (\mathcal{S},\mathcal{A},P,R,\gamma ,d_0)$ , with state space  $\mathcal{S}$ , action space  $\mathcal{A}$ , transition dynamics  $P:S\times \mathcal{A}\to \Delta (\mathcal{S})$ , reward function  $R:S\times \mathcal{A}\rightarrow [R_{\mathrm{min}},R_{\mathrm{max}}]$ , discount factor  $\gamma \in [0,1)$ , and initial state distribution  $d_0$  [41]. The goal of RL is to find a policy  $\pi :S\to \Delta (\mathcal{A})$  that maximizes the expected discounted return:  $\mathbb{E}_{s_0\sim d_0,a_t\sim \pi (\cdot |s_t),s_{t + 1}\sim P(\cdot |s_t,a_t)}[\sum_{t = 0}^{\infty}\gamma^{t}R(s_{t},a_{t})]$ . For any policy  $\pi$ , we define the value function as  $V^{\pi}(s) = \mathbb{E}_{\pi}[\sum_{t = 0}^{\infty}\gamma^{t}R(s_{t},a_{t})|s_{0} = s]$  and the state-action value function ( $Q$ -value function) as  $Q^{\pi}(s,a) = \mathbb{E}_{\pi}[\sum_{t = 0}^{\infty}\gamma^{t}R(s_{t},a_{t})|s_{0} = s,a_{0} = a]$ . By the boundedness of rewards, we have  $Q^{\pi}\in [Q_{\mathrm{min}},Q_{\mathrm{max}}]$ , where  $Q_{\mathrm{min}}\coloneqq R_{\mathrm{min}} / (1 - \gamma)$  and  $Q_{\mathrm{max}}\coloneqq R_{\mathrm{max}} / (1 - \gamma)$ . In addition, we use  $\rho_{sa}^{\pi}$  to denote the normalized and discounted state-action occupancy of policy  $\pi$  with the initial state-action pair  $(s,a)$ :  $\rho_{sa}^{\pi}(s',a') = (1 - \gamma)\sum_{t = 0}^{\infty}\gamma^{t}\mathbb{E}_{\pi}[\mathbb{I}[s_{t} = s',a_{t} = a']|s_{0} = s,a_{0} = a]$ .
Actor-critic methods [19] alternate between evaluating the policy by iterating the Bellman operator  $(\mathcal{T}^{\pi}Q)(s,a)\coloneqq R(s,a) + \gamma \mathbb{E}_{s'}\mathbb{E}_{a'\sim \pi (\cdot |s')}[Q(s',a')]$ , and improving the policy by maximizing the  $Q$ -value. In offline RL, the agent is provided with a fixed dataset  $\mathcal{D}$  collected by some behavior policy  $\beta$ . Ordinary actor-critic algorithms [9, 12, 39] minimize the following losses alternately:
$$
L _ {Q} (\theta) = \mathbb {E} _ {(s, a, s ^ {\prime}) \sim \mathcal {D}} \left[ \left(Q _ {\theta} (s, a) - R (s, a) - \gamma \mathbb {E} _ {a ^ {\prime} \sim \pi_ {\phi} (· \mid s ^ {\prime})} Q _ {\theta^ {\prime}} (s ^ {\prime}, a ^ {\prime})\right) ^ {2} \right] \text {(p o l i c y e l a t u a t i o n)} \tag {1}
$$
$$
L _ {\pi} (\phi) = - \mathbb {E} _ {s \sim \mathcal {D}, a \sim \pi_ {\phi}} \left[ Q _ {\theta} (s, a) \right] \quad (\text {p o l i c y i m p r o v e m e n t}) \tag {2}
$$
where  $\pi_{\phi}$  is a policy parameterized by  $\phi$ ,  $Q_{\theta}(s,a)$  is a  $Q$  function parameterized by  $\theta$ , and  $Q_{\theta'}(s,a)$  is a target  $Q$  function whose parameters are updated via Polyak averaging [32].
Value regularization. In offline RL, OOD actions  $a'$  can produce erroneous Bellman targets and lead to an inaccurate estimation of  $Q$ -values. Then in policy improvement, the policy tends to prioritize OOD actions whose values have been overestimated, resulting in poor performance.
To address this issue, value regularization methods regularize the  $Q$  function to introduce conservatism in value estimation [23, 20]. As the most representative one, CQL [23] minimizes the following policy evaluation loss, which guarantees to obtain underestimated  $V$  functions:
$$
\min  _ {Q} \mathbb {E} _ {s \sim \mathcal {D}, a \sim \beta} \left[ (Q (s, a) - \mathcal {T} ^ {\pi} Q ^ {\prime} (s, a)) ^ {2} \right] + \alpha (\mathbb {E} _ {s \sim \mathcal {D}, a \sim \pi} [ Q (s, a) ] - \mathbb {E} _ {s \sim \mathcal {D}, a \sim \beta} [ Q (s, a) ]) \tag {3}
$$
where  $Q^{\prime}$  is the target  $Q$  function and  $\alpha$  is a hyperparameter.
3 Supported Value Regularization
In this section, we first briefly analyze the existing density-based value regularization and identify some important issues. Then we propose Supported Value Regularization (SVR) to penalize all OOD  $Q$ -values while maintaining standard updates for ID ones. Next, we conduct a thorough analysis of SVR and demonstrate its theoretical superiority. Last, we present the implementation details of SVR.
3.1 Density-based value regularization
A large portion of value regularization methods are built on the idea of being pessimistic about the actions under current policy and optimistic about the actions within the dataset [23, 50, 27, 6]. Instead of relying on ID or OOD, we show that this regularization is essentially based on the policy density, which we refer to as density-based value regularization. Specifically, we take CQL (Eq. (3)) [23], the foundational work on value regularization, as an example for mathematical analysis. Here we remove the strong assumption  $\mathrm{supp}(\pi) \subseteq \mathrm{supp}(\beta)$  in their paper and analyze its regularization effects on the  $Q$  functions. From Eq. (3), we can obtain the analytical solution of  $Q$ , which corresponds to the following policy evaluation operator  $\mathcal{T}_{\mathrm{CQL}}^{\pi}$ :
$$
\mathcal {T} _ {\mathrm {C Q L}} ^ {\pi} Q (s, a) = \left\{ \begin{array}{l l} \mathcal {T} ^ {\pi} Q (s, a) - \alpha \left(\frac {\pi (a | s)}{\beta (a | s)} - 1\right), & \beta (a | s) > 0, \\ - \infty , & \beta (a | s) = 0, \pi (a | s) > 0, \\ Q (s, a), & \beta (a | s) = 0, \pi (a | s) = 0. \end{array} \right. \tag {4}
$$
Considering the iteration of policy evaluation:  $Q^{k + 1}(s,a) = \mathcal{T}_{\mathrm{CQL}}^{\pi}Q^{k}(s,a)$ , we have Observation 1:
Observation 1. In each iteration  $k$ , compared to  $\mathcal{T}^{\pi}$ ,  $\mathcal{T}_{\mathrm{CQL}}^{\pi}$  (1) lowers  $Q^{k + 1}(s,a)$  when  $\pi (a|s) > \beta (a|s)$ ; (2) raises  $Q^{k + 1}(s,a)$  when  $\pi (a|s) < \beta (a|s)$ ; (3) obtains the same  $Q^{k + 1}(s,a)$  when  $\pi (a|s) = \beta (a|s) > 0$ ; (4) does not update  $Q^{k + 1}(s,a)$  when  $\pi (a|s) = \beta (a|s) = 0$ .
For (1)(2)(3), lowering or raising  $Q$ -values based on the relative density between  $\pi$  and  $\beta$  could be problematic, especially when the dataset contains a large portion of bad actions. In such cases, assuming the policy  $\pi$  has found the optimal action  $a^*$ , it holds that  $\pi(a^*|s) > \beta(a^*|s)$ , and thus CQL will lower  $Q(s, a^*)$ ; for some bad action  $\hat{a} \in \mathcal{D}$ , it holds that  $\pi(\hat{a}|s) < \beta(\hat{a}|s)$ , and thus CQL will raise  $Q(s, \hat{a})$ . That is, CQL tends to raise the  $Q$ -values of numerous bad actions and lower the  $Q$ -values of scare good actions, forcing the policy to choose bad actions in policy improvement stage. Regarding (4), in the entire OOD region  $(\beta(a|s) = 0)$ , the penalization is only performed where  $\pi(a|s) > 0$ . Since the distribution of  $\pi$  is narrow compared to the whole action space, the regularization region of CQL and most existing methods [50, 6, 27, 3, 48] tends to be too narrow.
3.2 Supported value regularization
In this paper, we aim to penalize  $Q$ -values for all OOD actions ( $a \notin \operatorname{supp}(\beta)$ ) and maintain standard Bellman updates for ID ones ( $a \in \operatorname{supp}(\beta)$ ). That is, the regularization is solely determined by the support of  $\beta$ , which we refer to as supported value regularization.
However, it is a well-known challenge to distinguish between ID and OOD actions. Directly determining if the behavior density exceeds a threshold [43] would necessitate an extremely precise density estimator, and it is particularly difficult to estimate the density of OOD actions. In this work, we draw inspiration from the link between support and Importance Sampling (IS) to circumvent this dilemma and achieve the same goal.
We begin by explaining its principle with simplified notations. IS computes  $l_{1} = \mathbb{E}_{q}[p(x)f(x) / q(x)]$  to estimate  $l_{2} = \mathbb{E}_{p}f(x)$ . When assuming  $\mathrm{supp}(p)\subseteq \mathrm{supp}(q)$ , IS is unbiased:  $l_{1} = l_{2}$ . However, when removing the support assumption, IS actually computes  $l_{1} = \sum_{x\in \mathrm{supp}(q)}p(x)f(x)$ . We observe that  $l\coloneqq l_1 - l_2 = \sum_{x\notin \mathrm{supp}(q)}p(x)f(x)$  gives the summation over the out-of-support region. In the offline RL setting, let  $f$  be the  $Q$  function. If we choose the behavior policy  $\beta$  as  $q$  and any sampleable distribution that covers the whole action space as  $p$ , then minimizing  $l$  would lower all OOD  $Q$ -values without affecting the ID ones. Besides, since both distribution  $q$  and  $p$  are sampleable in this case, we can obtain  $l_{1}$  and  $l_{2}$  based on sampling, which provides an unbiased  $l$  to optimize.
Therefore, we minimize the following loss for policy evaluation in SVR:
$$
\begin{array}{l} \min _ {Q} \mathbb {E} _ {s \sim \mathcal {D}, a \sim \beta} \left[ (Q (s, a) - \mathcal {T} ^ {\pi} Q ^ {\prime} (s, a)) ^ {2} \right] \\ + \alpha \left(\mathbb {E} _ {s \sim \mathcal {D}, a \sim u} \left[ (Q (s, a) - Q _ {\min }) ^ {2} \right] - \mathbb {E} _ {s \sim \mathcal {D}, a \sim \beta} \left[ \frac {u (a | s)}{\beta (a | s)} (Q (s, a) - Q _ {\min }) ^ {2} \right]\right) \tag {5} \\ \end{array}
$$
where  $u(\cdot |s)$  is any sampleable distribution (e.g., Gaussian, uniform) whose support covers the whole action space, and  $Q_{\mathrm{min}} \coloneqq R_{\mathrm{min}} / (1 - \gamma)$  is the minimal possible  $Q$  of the MDP. According to the analysis above, Eq. (5) is equivalent to the following minimization problem:
$$
\min  _ {Q} \mathbb {E} _ {s \sim \mathcal {D}, a \sim \beta} \left[ \left(Q (s, a) - \mathcal {T} ^ {\pi} Q ^ {\prime} (s, a)\right) ^ {2} \right] + \alpha \mathbb {E} _ {s \sim \mathcal {D}} \left[ \sum_ {a \notin \operatorname {s u p p} (\beta (\cdot | s))} u (a | s) \left(Q (s, a) - Q _ {\min }\right) ^ {2} \right] \tag {6}
$$
It is clearer from Eq. (6) that the loss in Eq. (5) penalizes all OOD  $Q$ -values without affecting the ID ones. Note that optimizing Eq. (5) requires pre-training a behavior model  $\beta$  to output the behavior density  $\beta(a|s)$  in the IS ratio. Compared with other methods that also require the behavior model [43, 20, 11, 27], SVR is less susceptible to model errors. This is because SVR only needs to query the behavior density of in-dataset  $(s, a)$  pairs, thus not requiring much generalization ability of the model, making it relatively easier to estimate accurately.
From an optimization perspective, minimizing Eq. (6) reduces all OOD  $Q$ -values with a strength proportional to  $u(a_{\mathrm{odd}}|s)$ . Therefore, we can even choose various  $u$  to flexibly penalize the OOD region. For example, let  $u$  assign higher weight to areas with a high probability of overestimation. Note that  $u$  can have any positive density in the ID region without affecting the ID  $Q$ -values, thereby eliminating the necessity to manually distinguish between ID and OOD regions. From an optimal solution perspective, Eq. (5,6) lead to the following SVR policy evaluation operator:
$$
\mathcal {T} _ {\mathrm {S V R}} ^ {\pi} Q (s, a) = \left\{ \begin{array}{l l} \mathcal {T} ^ {\pi} Q (s, a), & \beta (a | s) > 0, \\ Q _ {\min }, & \text {e l s e .} \end{array} \right. \tag {7}
$$
In contrast to Observation 1, we make the following claim for SVR.
Claim 1. In each iteration  $k$ , compared to  $\mathcal{T}^{\pi}$ ,  $\mathcal{T}_{\mathrm{SVR}}^{\pi}$  obtains the same  $Q^{k+1}(s, a)$  when  $\beta(a|s) > 0$  (ID region); and lowers  $Q^{k+1}(s, a)$  when  $\beta(a|s) = 0$  (OOD region).
In this section, we refer to the policy iteration, whose evaluation part is repeatedly applying  $\mathcal{T}_{\mathrm{SVR}}^{\pi}$  and whose improvement part is vanilla maximization of  $Q$ , as SVR. We will give a comprehensive analysis of SVR, including the fixed point in policy evaluation, the monotonic improving performance in policy improvement, and the support-constrained optimal convergence for the whole policy iteration.
We first define the support-constrained policy set [22], which plays an important role in our analysis. However, it is worth noting that SVR does not impose any constraint or regularization on the policy.
Definition 1 (Support-constrained policy). The support-constrained policy class  $\Pi$  is defined as
$$
\Pi = \{\pi \mid \pi (a | s) = 0 \text {w h e n e v e r} \beta (a | s) = 0 \} \tag {8}
$$
Following prior works [22], we also define the optimal support-constrained policy  $\pi_{\Pi}^{*}$ .
Definition 2 (Optimal support-constrained policy). The optimal support-constrained policy  $\pi_{\Pi}^{*}$  is:
$$
\pi_ {\Pi} ^ {*} (a | s) := \mathbb {I} \left[ a = \underset {a ^ {\prime} \in \operatorname {s u p p} (\beta (| s))} {\operatorname {a r g m a x}} Q _ {\Pi} ^ {*} (s, a ^ {\prime}) \right] \tag {9}
$$
where  $Q_{\Pi}^{*}$  satisfies the support-constrained Bellman optimality equation:
$$
Q _ {\Pi} ^ {*} (s, a) = R (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \max  _ {a ^ {\prime} \in \operatorname {s u p p} \left(\beta (\cdot | s ^ {\prime})\right)} Q _ {\Pi} ^ {*} \left(s ^ {\prime}, a ^ {\prime}\right) \right]. \tag {10}
$$
Proposition 1 (Contraction). In the whole  $S \times \mathcal{A}$  space and for any  $\pi$ ,  $\mathcal{T}_{\mathrm{SVR}}^{\pi}$  is a  $\gamma$ -contraction operator in the  $\mathcal{L}_{\infty}$  norm.
Therefore, in the policy evaluation stage of SVR, any initial  $Q$  function can converge to a unique fixed point by repeatedly applying  $\mathcal{T}_{\mathrm{SVR}}^{\pi}$ . We give this fixed point in the following theorem.
Theorem 1 (Fixed point). SVR yields support-constrained  $\pi_i : \pi_i \in \Pi$ . The fixed point of  $\mathcal{T}_{\mathrm{SVR}}^{\pi_i}$  is
$$
f ^ {\pi_ {i}} (s, a) = \left\{ \begin{array}{l l} Q ^ {\pi_ {i}} (s, a), & \beta (a | s) > 0, \\ Q _ {\min }, & \beta (a | s) = 0. \end{array} \right. \tag {11}
$$
Therefore, for the policies  $\pi_{i}$  in SVR, this fixed point provides unbiased  $Q$ -values for all ID actions and underestimated  $Q$ -values for all OOD actions.
Though Theorem 1 guarantees that SVR only obtains support-constrained policies during learning, we show that, even for any  $\pi$  ( $\pi \notin \Pi$  due to various errors in practice), the fixed point of  $\mathcal{T}_{\mathrm{SVR}}^{\pi}$  still ensures that  $Q$  will not be overestimated over the entire action space.
Proposition 2. For any  $\pi$ , the fixed point of  $\mathcal{T}_{\mathrm{SVR}}^{\pi}$  satisfies
$$
\left\{ \begin{array}{l l} Q _ {\min } \leq f ^ {\pi} (s, a) \leq Q ^ {\pi} (s, a), & \beta (a | s) > 0, \\ f ^ {\pi} (s, a) = Q _ {\min }, & \beta (a | s) = 0. \end{array} \right. \tag {12}
$$
For comparison, the following proposition characterizes the contraction property of CQL.
Proposition 3. Only in the ID region  $(\beta (a|s) > 0)$  and when the evaluated policy is support-constrained  $(\pi \in \Pi)$ ,  $\mathcal{T}_{\mathrm{CQL}}^{\pi}$  is a contraction operator. Its fixed point satisfies
$$
f ^ {\pi} (s, a) = Q ^ {\pi} (s, a) - \frac {\alpha}{1 - \gamma} \left(\rho_ {s a} ^ {\pi}\right) ^ {T} \left(\frac {\pi}{\beta} - 1\right), \beta (a | s) > 0. \tag {13}
$$
Therefore,  $\mathcal{T}_{\mathrm{CQL}}^{\pi}$  has no fixed point in the OOD region. Even in the ID region, the condition  $\pi \in \Pi$  is required and its fixed point may underestimate or overestimate  $Q$ -values in a complicated way.
Finally, we show that SVR guarantees strict policy improvement for each iteration until convergence to the optimal support-constrained policy  $\pi_{\Pi}^{*}$ . Note that existing value regularization methods fail to guarantee such each-step policy improvement and optimal convergence results.
Theorem 2 (Strict policy improvement to support-constrained optimal). SVR yields support-constrained  $\pi_{i}$  and guarantees monotonic performance improvement:
$$
V ^ {\pi_ {i + 1}} (s) \geq V ^ {\pi_ {i}} (s) \quad \forall s, \tag {14}
$$
where the improvement is strict in at least one state until  $\pi_{\Pi}^{*}$  is found.
3.4 Practical implementation of SVR
SVR is easy to implement and we design the practical algorithm to be as simple as possible to avoid some complex modules confusing our algorithm's impact on the final performance<sup>1</sup>.
Behavior model. Following previous works [22, 44], we learn a Gaussian model  $\beta_{\omega}$  for the behavior policy by maximizing
$$
J _ {\beta} (\omega) = \mathbb {E} _ {s, a \sim \mathcal {D}} \log \beta_ {\omega} (a | s) \tag {15}
$$
Sampling distribution  $u$ . In our method,  $u$  in Eq. (5) can be any samplable distribution that covers the entire action space. In our implementation, we set  $u$  to the Gaussian with the same mean as  $\pi$  and a fixed variance that is much larger than that of  $\pi$ . This choice is made because during learning,  $\pi$  is typically
located where  $Q$  is maximized and where overestimation is most likely to occur. So if  $\pi$  is OOD, this choice will assign higher weight to the area around  $\pi$  to reduce the  $Q$ -value. Note that if  $\pi$  is ID, it will not penalize the  $Q$ -values in the ID region even though  $u$  may have a higher weight there.
Algorithm 1 Supported Value Regularization (SVR)
1: Initialize behavior policy  $\beta_{\omega}$ , policy network  $\pi_{\phi}$ ,  $Q$ -network  $Q_{\theta}$ , and target  $Q$ -network  $Q_{\theta'}$
2: // Behavior Policy Pre-training
3: for each gradient step do
4: Sample minibatch  $(s,a)\sim \mathcal{D}$
5: Update  $\omega$  by maximizing  $J_{\beta}(\omega)$  in Eq. (15)
6: end for
7: // Policy Training
8: for each gradient step do
9: Sample minibatch  $(s, a, r, s') \sim \mathcal{D}$
10: Update  $\theta$  by minimizing  $L_{Q}(\theta)$  in Eq. (16)
11: Update  $\phi$  by minimizing  $L_{\pi}(\phi)$  in Eq. (2)
12: Update target network:  $\theta^{\prime}\gets (1 - \tau)\theta^{\prime} + \tau \theta$
13: end for
1Our code is available at https://github.com/MAOYIXIU/SVR.
(a) tabular maze
(b) support optimal
(c) vanilla
Figure 1: (a) The maze environment with OOD actions. (b) The optimal support-constrained policy and its value function. (c) (d) (e) The value functions and policies learned by vanilla policy iteration, CQL and SVR, respectively. The resulting trajectories are illustrated by white dashed lines. SVR is capable of obtaining the optimal support-constrained policy and value function.
(d) CQL
(e) SVR
Policy evaluation. With policy  $\pi_{\phi}$ ,  $Q$  function  $Q_{\theta}$ , and target  $Q$  function  $Q_{\theta'}$ , we minimize the following loss for policy evaluation: