Title: Supported Value Regularization for Offline Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: offline reinforcement learning
Abstract: Offline reinforcement learning suffers from the extrapolation error and value overestimation caused by out-of-distribution (OOD) actions. To mitigate this issue, value regularization approaches aim to penalize the learned value functions to assign lower values to OOD actions. However, existing value regularization methods lack a proper distinction between the regularization effects on in-distribution (ID) and OOD actions, and fail to guarantee optimal convergence results of the policy. To this end, we propose Supported Value Regularization (SVR), which penalizes the Q-values for all OOD actions while maintaining standard Bellman updates for ID ones. Specifically, we utilize the bias of importance sampling to compute the summation of Q-values over the entire OOD region, which serves as the penalty for policy evaluation. This design automatically separates the regularization for ID and OOD actions without manually distinguishing between them. In tabular MDP, we show that the policy evaluation operator of SVR is a contraction, whose fixed point outputs unbiased Q-values for ID actions and underestimated Q-values for OOD actions. Furthermore, the policy iteration with SVR guarantees strict policy improvement until convergence to the optimal support-constrained policy in the dataset. Empirically, we validate the theoretical properties of SVR in a tabular maze environment and demonstrate its state-of-the-art performance on a range of continuous control tasks in the D4RL benchmark.

Supported Value Regularization for Offline Reinforcement Learning

Abstract

Offline reinforcement learning suffers from the extrapolation error and value overestimation caused by out-of-distribution (OOD) actions. To mitigate this issue, value regularization approaches aim to penalize the learned value functions to assign lower values to OOD actions. However, existing value regularization methods lack a proper distinction between the regularization effects on in-distribution (ID) and OOD actions, and fail to guarantee optimal convergence results of the policy. To this end, we propose Supported Value Regularization (SVR), which penalizes the Q -values for all OOD actions while maintaining standard Bellman updates for ID ones. Specifically, we utilize the bias of importance sampling to compute the summation of Q -values over the entire OOD region, which serves as the penalty for policy evaluation. This design automatically separates the regularization for ID and OOD actions without manually distinguishing between them. In tabular MDP, we show that the policy evaluation operator of SVR is a contraction, whose fixed point outputs unbiased Q -values for ID actions and underestimated Q -values for OOD actions. Furthermore, the policy iteration with SVR guarantees strict policy improvement until convergence to the optimal support-constrained policy in the dataset. Empirically, we validate the theoretical properties of SVR in a tabular maze environment and demonstrate its state-of-the-art performance on a range of continuous control tasks in the D4RL benchmark.

1 Introduction

Offline Reinforcement Learning (RL) aims to learn a policy from a fixed dataset collected by some behavior policy [24, 26]. It can tap into existing large-scale datasets [15, 30] for safe and efficient learning. However, it suffers from the extrapolation error [10] caused by out-of-distribution (OOD) actions, which can further result in severe overestimation of value functions [26].

To mitigate this issue, value regularization approaches attempt to penalize the Q -values of OOD actions to introduce conservatism in value estimation [23, 20, 46, 14, 3, 6]. However, we observe that existing value regularization methods not only fall short in penalizing all OOD Q -values but also may introduce detrimental changes to in-distribution (ID) ones. Specifically, most of them involve adding penalties to the critic learning objective [23, 50, 27, 6, 48]. However, due to the difficulty of distinguishing between ID and OOD actions, they typically adopt the idea of being pessimistic about the actions under the current policy and optionally optimistic about the ones within the dataset [23, 50, 46, 6]. Rather than relying on ID or OOD, this regularization is essentially based on the policy density, which we show is problematic when the dataset is heavily corrupted by suboptimal actions. Other works involve reducing the Bellman target with uncertainty quantifiers [14, 3] or ensembles [2, 34]. However, if not incorporating additional penalties, they do not provide learning signals on OOD Q -values, logically only suppressing the overestimation in the ID region [14, 2].

In this work, we revisit the original objective of value regularization and ask a question - 'Can we devise a value regularization method that penalizes all OOD Q -values without affecting ID ones'? We point out that the inability of prior methods to achieve this leads to a lack of strong theoretical guarantees for policy performance. In offline RL, the best policy that can be guaranteed should lie within the support of the behavior policy, known as the optimal support-constrained policy [22]. However, existing value regularization methods do not provide reliable guarantees for converging to it. On the other hand, several endeavors aim to learn it through specific policy constraints [10, 22, 11], but their empirical performance leaves considerable room for improvement.

To this end, we propose Supported Value Regularization (SVR), a new value regularization method that penalizes the Q -values for all OOD actions while maintaining standard Bellman updates for ID ones. Specifically, we leverage the bias of importance sampling to calculate the summation of Q -values over the entire OOD region, which serves as the penalty term for policy evaluation. This design circumvents the dilemma of distinguishing between ID and OOD actions and separates the regularization effects on them automatically. Theoretically, SVR offers stronger guarantees than previous methods. Under tabular MDP, we show that the SVR policy evaluation operator is a contraction in the whole state-action space and its fixed point outputs unbiased Q -values for ID actions and underestimated Q -values for OOD ones, while the fixed point of Q in CQL [23] may underestimate or overestimate Q -values. More importantly, SVR guarantees strict policy improvement until convergence to the optimal support-constrained policy, while prior value regularization methods lack such each-step improvement and global optimal convergence guarantees.

In practice, SVR is easy to implement by adding a penalty term to the ordinary policy evaluation loss. Empirically, we validate the support-constrained optimality of SVR in a tabular maze environment, where baselines fail to converge. Moreover, SVR achieves state-of-the-art performance on a range of continuous control tasks in the D4RL benchmark [7] and shows strong advantages on noisy datasets.

2 Preliminaries

Offline RL. The environment in RL is typically modeled as a Markov Decision Process (MDP) M = ( S , A , P, R, γ, d 0 ) , with state space S , action space A , transition dynamics P : S×A → ∆( S ) , reward function R : S × A → [ R min , R max ] , discount factor γ ∈ [0 , 1) , and initial state distribution d 0 [41]. The goal of RL is to find a policy π : S → ∆( A ) that maximizes the expected discounted return: E s 0 ∼ d 0 ,a t ∼ π ( ·| s t ) ,s t +1 ∼ P ( ·| s t ,a t ) [ ∑ ∞ t =0 γ t R ( s t , a t )] . For any policy π , we define the value function as V π ( s ) = E π [ ∑ ∞ t =0 γ t R ( s t , a t ) | s 0 = s ] and the state-action value function ( Q -value function) as Q π ( s, a ) = E π [ ∑ ∞ t =0 γ t R ( s t , a t ) | s 0 = s, a 0 = a ] . By the boundedness of rewards, we have Q π ∈ [ Q min , Q max ] , where Q min := R min / (1 -γ ) and Q max := R max / (1 -γ ) . In addition, we use ρ π sa to denote the normalized and discounted state-action occupancy of policy π with the initial state-action pair ( s, a ) : ρ π sa ( s ′ , a ′ ) = (1 -γ ) ∑ ∞ t =0 γ t E π [ I [ s t = s ′ , a t = a ′ ] | s 0 = s, a 0 = a ] .

Actor-critic methods [19] alternate between evaluating the policy by iterating the Bellman operator ( T π Q ) ( s, a ) := R ( s, a ) + γ E s ′ E a ′ ∼ π ( ·| s ′ ) [ Q ( s ′ , a ′ )] , and improving the policy by maximizing the Q -value. In offline RL, the agent is provided with a fixed dataset D collected by some behavior policy β . Ordinary actor-critic algorithms [9, 12, 39] minimize the following losses alternately:

where π ϕ is a policy parameterized by ϕ , Q θ ( s, a ) is a Q function parameterized by θ , and Q θ ′ ( s, a ) is a target Q function whose parameters are updated via Polyak averaging [32].

Value regularization. In offline RL, OOD actions a ′ can produce erroneous Bellman targets and lead to an inaccurate estimation of Q -values. Then in policy improvement, the policy tends to prioritize OOD actions whose values have been overestimated, resulting in poor performance.

To address this issue, value regularization methods regularize the Q function to introduce conservatism in value estimation [23, 20]. As the most representative one, CQL [23] minimizes the following policy evaluation loss, which guarantees to obtain underestimated V functions:

where Q ′ is the target Q function and α is a hyperparameter.

3 Supported Value Regularization

In this section, we first briefly analyze the existing density-based value regularization and identify some important issues. Then we propose Supported Value Regularization (SVR) to penalize all OOD Q -values while maintaining standard updates for ID ones. Next, we conduct a thorough analysis of SVR and demonstrate its theoretical superiority. Last, we present the implementation details of SVR.

3.1 Density-based value regularization

A large portion of value regularization methods are built on the idea of being pessimistic about the actions under current policy and optimistic about the actions within the dataset [23, 50, 27, 6]. Instead of relying on ID or OOD, we show that this regularization is essentially based on the policy density, which we refer to as density-based value regularization. Specifically, we take CQL (Eq. (3)) [23], the foundational work on value regularization, as an example for mathematical analysis. Here we remove the strong assumption supp( π ) ⊆ supp( β ) in their paper and analyze its regularization effects on the Q functions. From Eq. (3), we can obtain the analytical solution of Q , which corresponds to the following policy evaluation operator T π CQL :

Considering the iteration of policy evaluation: Q k +1 ( s, a ) = T π CQL Q k ( s, a ) , we have Observation 1: Observation 1. In each iteration k , compared to T π , T π CQL (1) lowers Q k +1 ( s, a ) when π ( a | s ) > β ( a | s ) ; (2) raises Q k +1 ( s, a ) when π ( a | s ) < β ( a | s ) ; (3) obtains the same Q k +1 ( s, a ) when π ( a | s ) = β ( a | s ) > 0 ; (4) does not update Q k +1 ( s, a ) when π ( a | s ) = β ( a | s ) = 0 .

For (1)(2)(3), lowering or raising Q -values based on the relative density between π and β could be problematic, especially when the dataset contains a large portion of bad actions. In such cases, assuming the policy π has found the optimal action a ∗ , it holds that π ( a ∗ | s ) > β ( a ∗ | s ) , and thus CQL will lower Q ( s, a ∗ ) ; for some bad action ˆ a ∈ D , it holds that π (ˆ a | s ) < β (ˆ a | s ) , and thus CQL will raise Q ( s, ˆ a ) . That is, CQL tends to raise the Q -values of numerous bad actions and lower the Q -values of scare good actions, forcing the policy to choose bad actions in policy improvement stage. Regarding (4), in the entire OOD region ( β ( a | s ) = 0 ), the penalization is only performed where π ( a | s ) > 0 . Since the distribution of π is narrow compared to the whole action space, the regularization region of CQL and most existing methods [50, 6, 27, 3, 48] tends to be too narrow.

3.2 Supported value regularization

In this paper, we aim to penalize Q -values for all OOD actions ( a / ∈ supp( β ) ) and maintain standard Bellman updates for ID ones ( a ∈ supp( β ) ). That is, the regularization is solely determined by the support of β , which we refer to as supported value regularization.

However, it is a well-known challenge to distinguish between ID and OOD actions. Directly determining if the behavior density exceeds a threshold [43] would necessitate an extremely precise density estimator, and it is particularly difficult to estimate the density of OOD actions. In this work, we draw inspiration from the link between support and Importance Sampling (IS) to circumvent this dilemma and achieve the same goal.

We begin by explaining its principle with simplified notations. IS computes l 1 = E q [ p ( x ) f ( x ) /q ( x )] to estimate l 2 = E p f ( x ) . When assuming supp( p ) ⊆ supp( q ) , IS is unbiased: l 1 = l 2 . However, when removing the support assumption, IS actually computes l 1 = ∑ x ∈ supp( q ) p ( x ) f ( x ) . We observe that l := l 1 -l 2 = ∑ x/ ∈ supp( q ) p ( x ) f ( x ) gives the summation over the out-of-support region. In the offline RL setting, let f be the Q function. If we choose the behavior policy β as q and any samplable distribution that covers the whole action space as p , then minimizing l would lower all OOD Q -values without affecting the ID ones. Besides, since both distribution q and p are samplable in this case, we can obtain l 1 and l 2 based on sampling, which provides an unbiased l to optimize.

Therefore, we minimize the following loss for policy evaluation in SVR:

where u ( ·| s ) is any samplable distribution (e.g., Gaussian, uniform) whose support covers the whole action space, and Q min := R min / (1 -γ ) is the minimal possible Q of the MDP. According to the analysis above, Eq. (5) is equivalent to the following minimization problem:

It is clearer from Eq. (6) that the loss in Eq. (5) penalizes all OOD Q -values without affecting the ID ones. Note that optimizing Eq. (5) requires pre-training a behavior model β to output the behavior density β ( a | s ) in the IS ratio. Compared with other methods that also require the behavior model [43, 20, 11, 27], SVR is less susceptible to model errors. This is because SVR only needs to query the behavior density of in-dataset ( s, a ) pairs, thus not requiring much generalization ability of the model, making it relatively easier to estimate accurately.

From an optimization perspective, minimizing Eq. (6) reduces all OOD Q -values with a strength proportional to u ( a ood | s ) . Therefore, we can even choose various u to flexibly penalize the OOD region. For example, let u assign higher weight to areas with a high probability of overestimation. Note that u can have any positive density in the ID region without affecting the ID Q -values, thereby eliminating the necessity to manually distinguish between ID and OOD regions. From an optimal solution perspective, Eq. (5,6) lead to the following SVR policy evaluation operator:

In contrast to Observation 1, we make the following claim for SVR.

Claim 1. In each iteration k, compared to T π , T π SVR obtains the same Q k +1 ( s, a ) when β ( a | s ) > 0 (ID region); and lowers Q k +1 ( s, a ) when β ( a | s ) = 0 (OOD region).

3.3 Analysis

In this section, we refer to the policy iteration, whose evaluation part is repeatedly applying T π SVR and whose improvement part is vanilla maximization of Q , as SVR. We will give a comprehensive analysis of SVR, including the fixed point in policy evaluation, the monotonic improving performance in policy improvement, and the support-constrained optimal convergence for the whole policy iteration.

We first define the support-constrained policy set [22], which plays an important role in our analysis. However, it is worth noting that SVR does not impose any constraint or regularization on the policy.

Definition 1 (Support-constrained policy) . The support-constrained policy class Π is defined as

Following prior works [22], we also define the optimal support-constrained policy π ∗ Π .

Definition 2 (Optimal support-constrained policy) . The optimal support-constrained policy π ∗ Π is:

where Q ∗ Π satisfies the support-constrained Bellman optimality equation:

Proposition 1 (Contraction) . In the whole S × A space and for any π , T π SVR is a γ -contraction operator in the L ∞ norm.

Therefore, in the policy evaluation stage of SVR, any initial Q function can converge to a unique fixed point by repeatedly applying T π SVR . We give this fixed point in the following theorem.

Theorem 1 (Fixed point) . SVR yields support-constrained π i : π i ∈ Π . The fixed point of T π i SVR is

Therefore, for the policies π i in SVR, this fixed point provides unbiased Q -values for all ID actions and underestimated Q -values for all OOD actions.

Though Theorem 1 guarantees that SVR only obtains support-constrained policies during learning, we show that, even for any π ( π / ∈ Π due to various errors in practice), the fixed point of T π SVR still ensures that Q will not be overestimated over the entire action space.

Proposition 2. For any π , the fixed point of T π SVR satisfies

For comparison, the following proposition characterizes the contraction property of CQL.

Proposition 3. Only in the ID region ( β ( a | s ) > 0 ) and when the evaluated policy is supportconstrained ( π ∈ Π ), T π CQL is a contraction operator. Its fixed point satisfies

Therefore, T π CQL has no fixed point in the OOD region. Even in the ID region, the condition π ∈ Π is required and its fixed point may underestimate or overestimate Q -values in a complicated way.

Finally, we show that SVR guarantees strict policy improvement for each iteration until convergence to the optimal support-constrained policy π ∗ Π . Note that existing value regularization methods fail to guarantee such each-step policy improvement and optimal convergence results.

Theorem 2 (Strict policy improvement to support-constrained optimal) . SVR yields supportconstrained π i and guarantees monotonic performance improvement:

where the improvement is strict in at least one state until π ∗ Π is found.

3.4 Practical implementation of SVR

SVR is easy to implement and we design the practical algorithm to be as simple as possible to avoid some complex modules confusing our algorithm's impact on the final performance 1 .

Behavior model. Following previous works [22, 44], we learn a Gaussian model β ω for the behavior policy by maximizing

Sampling distribution u . In our method, u in Eq. (5) can be any samplable distribution that covers the entire action space. In our implementation, we set u to the Gaussian with the same mean as π and a fixed variance that is much larger than that of π . This choice is made because during learning, π is typically

Algorithm 1 Supported Value Regularization (SVR)

1: Initialize behavior policy β ω , policy network π ϕ , Q -network Q θ , and target Q -network Q θ ′

2: // Behavior Policy Pre-training

3: for each gradient step do

4: Sample minibatch ( s, a ) ∼ D

5: Update ω by maximizing J β ( ω ) in Eq. (15)

6: end for

7: // Policy Training

8: for each gradient step do

9: Sample minibatch ( s, a, r, s ′ ) ∼ D

10: Update θ by minimizing L Q ( θ ) in Eq. (16)

11: Update ϕ by minimizing L π ( ϕ ) in Eq. (2)

12: Update target network: θ ′ ← (1 -τ ) θ ′ + τθ

13: end for

located where Q is maximized and where overestimation is most likely to occur. So if π is OOD, this choice will assign higher weight to the area around π to reduce the Q -value. Note that if π is ID, it will not penalize the Q -values in the ID region even though u may have a higher weight there.

(a) tabular maze

(b) support optimal

(c) vanilla

(d) CQL

10

8

6

4

2

0

(e) SVR

?

Policy evaluation. With policy π ϕ , Q function Q θ , and target Q function Q θ ′ , we minimize the following loss for policy evaluation:

where α is a hyperparameter and Q min := R min / (1 -γ ) . For the environment where R min is unknown, we choose the smallest reward in the dataset as R min .

Overall Algorithm. Putting everything together, we summarize our final algorithm in Algorithm 1. Our algorithm first trains the estimated behavior policy to obtain the behavior density. Then it turns to the actor-critic framework for policy training.

4 Experiments

In this section, we conduct several experiments to justify the validity of our method. We aim to answer five questions: (1) Does SVR actually converge to the optimal support-constrained policy? (2) Does SVR perform better than previous methods on standard offline RL benchmarks? (3) When does SVR empirically benefit the most compared to the density-based regularization? (4) How should we select the sampling distribution of SVR in practice? (5) How does the implementation of each component affect SVR? More experimental details and results are provided in Appendix B and C.

4.1 Support-constrained optimality in the tabular setting

We use a simple maze environment to verify the supportconstrained optimality of SVR. As depicted in Fig. 1(a), the task is to navigate from bottom-left to top-right, with a wall in the middle. The agent receives a reward of 0 for reaching the goal and -1 for all other transitions. Episodes are terminated after 100 steps and γ is set to 0 . 9 . We first collect 10 , 000 transitions using a random policy. Then we remove all the transitions containing rightward actions in the bottom-left 4 × 4 region to introduce OOD actions. It makes the optimal support-constrained policy (see Fig. 1(b)) differ from the actual optimal policy in the environment.

We test three algorithms: vanilla policy iteration [41], CQL [23], and SVR. Fig. 1(c, d, e) depict their learned value functions and policies respectively. To show convergence results, we also present

0

10

20

30

40

50

Iteration

20

60

100

Toy Maze

SVR

CQL

Vanilla PI

Supp. Optimal

Behavior

the learning curves in Fig. 2, along with the performance of the behavior policy and the optimal support-constrained policy. The results indicate that vanilla policy iteration has a severe overestimation of value functions, leading to poor performance. On the other hand, although CQL does not overestimate V functions, it fails to converge and performs poorly when the dataset is highly suboptimal. In contrast, SVR converges to the optimal support-constrained policy and the learned value function closely matches the true support-constrained optimal value function, verifying Theorem 2.

4.2 Comparisons on D4RL benchmarks

Then we evaluate our approach on the D4RL benchmarks [7]. We compare SVR with prior state-ofthe-art offline RL methods, including BC [37], BCQ [10], BEAR [22], OneStep RL [4], TD3+BC [8], AWAC [33], UWAC [45], CQL [23], and IQL [21].

The results are reported in Table 1. In both Gym-MuJoCo and Adroit domains, SVR achieves state-of-the-art performance and outperforms prior value regularization methods (CQL, UWAC) by a large margin. We also observe that SVR has strong advantages on the sub-optimal datasets (medium, medium-replay, random). This is because SVR exploits the optimal support-constrained policy in the dataset in a theoretically sound way and is less affected by the average quality of the dataset. In addition, while BCQ and BEAR are the policy constraint methods that are also designed to search for the optimal policy within the behavior support, they have much poorer performance. As stated in previous works [11, 44], they are limited respectively by the errors of the generative model and the unsuitability of using Maximum Mean Discrepancy (MMD) to characterize the support constraint. In contrast, without querying out-of-dataset behavior density, SVR takes the form of value regularization to search for the optimal support-constrained policy and achieves superior performance. For learning curves and more experimental details, please see Section B in the supplementary material.

4.3 Comparisons on noisy datasets.

In this section, we aim to validate that, compared with the existing density-based value regularization, the supported value regularization of SVR will benefit when the dataset contains a large portion of bad actions. To this end, we construct a 'noisy' dataset by combining the random dataset and the expert dataset, and then evaluate SVR and CQL under different expert ratios.

The results are shown in Fig. 3. In all environments, SVR outperforms CQL over nearly all expert ratios. Moreover, as the expert ratio gets lower, the improvement is more significant. The performance of CQL is susceptible to the expert ratio and exhibits a sharp decrease at the expert ratio 20% , while SVR can still retain good or even expert performance (most spark in Hopper and Walker2d).

4.4 Empirical study on the sampling distribution of SVR

In this section, we investigate how to practically select the sampling distribution u in SVR, which controls the relative strength of penalizing different OOD Q -values. In our implementation, u is the

1.0

0.8

0.6

0.4

0.2

0.0

Expert Ratio

0

20

40

60

80

100

120

Normalized Return

Hopper

1.0

0.8

0.6

0.4

0.2

0.0

Expert Ratio

0

20

40

60

80

100

120

Walker2d

1.0

0.8

0.6

0.4

0.2

0.0

Expert Ratio

0

30

60

90

120

150

Ant

1.0

0.8

0.6

0.4

0.2

0.0

Expert Ratio

0

20

40

60

80

100

120

Halfcheetah

SVR

CQL

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

Normalized Return

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

hopper-medium

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

hopper-medium-replay

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

walker2d-medium-replay

=0.1

=0.2

=0.3

=0.5

=0.7

Uniform

Gaussian with the same mean as the current policy. Here we vary its standard deviation σ and present the corresponding results in Fig. 4. We do not show the results for σ > 0 . 7 because such a large σ would cause most sampled actions to lie at the boundary of the action space. Instead, we use Uniform distribution to represent σ = ∞ .

As shown in Fig. 4, SVR is able to converge to good performance over a very wide range of σ . However, if σ is too small, SVR may not be able to adequately penalize all OOD Q -values and the IS ratio will have a large variance, thereby disrupting the learning process (see σ = 0 . 1 in hopper-m-r). Conversely, if σ is too large or if the uniform distribution is used, sampling from u may fail to emphasize the key areas where overestimation is most likely to occur (as indicated by the current policy), leading to insufficient mitigation of overestimation and inferior performance (see Uniform in hopper-m, hopper-m-r, walker2d-m-r and σ = 0 . 7 in hopper-m).

4.5 Effects of components in SVR

Density estimator. We speculate that a more precise behavior density estimator may further improve SVR. So following previous works [10, 43], we consider to replace the Gaussian density estimator β ω with the conditional variational autoencoder [18, 40]. We refer to this variant as SVR-VAE. The results of SVRVAE are shown in Fig. 5 (left). Overall, SVR-VAE has only marginal performance gains over SVR.

IS techniques. Self-Normalized Importance Sampling (SNIS) [35] is sometimes adopted to reduce the high variance

HC-m

H-m

W-m

HC-mr

H-mr

W-mr

HC-me

H-me

W-me

HC-r

H-r

50

25

0

25

50

Percent Difference

SVR-VAE

HC-m

H-m

W-m

HC-mr

H-mr

W-mr

HC-me

H-me

W-me

HC-r

H-r

50

25

0

25

50

SVR-SNIS

of IS [38]. The SNIS estimator has lower variance but is biased [35]. Here we test an SVR variant SVR-SNIS, which normalizes the IS ratio across the batch. As shown in Fig. 5 (right), the perfor-

mance of SVR-SNIS is worse in hopper-m and hopper-m-e, and slightly better in other tasks. This discrepancy may be due to the varying requirements for variance and bias across different tasks.

5 Related Work

Offline RL. In offline RL, extrapolation error and overestimation caused by OOD actions pose significant challenges [10]. Among various solutions, value regularization methods aim to introduce conservatism in value estimation [23, 20, 29, 6, 3, 27], while policy constraint approaches enforce proximity between the trained policy and the behavior policy, either explicitly via divergence penalties [44, 22, 13, 8], implicitly by weighted behavior cloning [5, 36, 33, 42, 31], or directly through specific parameterization of the policy [10, 11, 53]. Another line of the methods, instead, opt for in-sample learning, which involves formulating the Bellman target with only the actions in the dataset [4, 28, 21, 51, 47]. However, the performance of most existing methods is largely confined by the average quality of the dataset [25]. In contrast, SVR guarantees the convergence to the optimal support-constrained policy and achieve superior performance in sub-optimal datasets.

Value regularization. We divide the value regularization methods in offline RL into two categories: direct Q penalization and Bellman target reduction. The former involves adding penalties to the critic learning loss, trying to penalize OOD Q -values [23, 50, 46, 6, 27]. Due to the difficulty of determining the OOD region, they typically penalize Q -values under the current policy distribution [23, 3, 27, 48] and optionally incorporate a maximization term under the data distribution for milder pessimism [23, 50, 6]. Thus, existing works of this category belong to policy density based value regularization, rather than ID/OOD based. On the other hand, the Bellman target reduction methods either subtract an uncertainty quantifier from the Bellman target [14, 3] or directly use the minimum of Q ensembles [34] to compute the target [2, 9]. However, if not incorporating additional penalties, they do not provide learning signals on OOD Q -values [2]. It empirically works because after the minimization over ensemble (or subtracting the uncertainty quantifier), the random generalized OOD Q -values will be smaller than ID ones with high probability, reducing the impact of OOD Q -values on learning. However, it requires a large number of Q networks and is much more computationally expensive [2].

Support-constrained optimality. In absence of additional information beyond the dataset, the optimal support-constrained policy represents the best policy that we can hope to obtain. Recent works have considered various policy constraints to learn it, but their performance still leaves considerable room for improvement. For example, BEAR [22] attempts to keep the learned policy within the support of the behavior policy by minimizing MMD between them. However, this choice lacks a theoretical guarantee, and Wu et al. [44] empirically found that MMD has no performance gain over KL. Other works adopt a specific parameterization of the policy to constrain it to the behavior support [10, 11]. They first pre-train a generative model (e.g., V AE) for the behavior policy. Then during learning, they sample several actions at each state and choose the one with the highest Q -value as the output of the policy, aiming to obtain the maximum within the behavior support. However, these methods are vulnerable to model errors [11] and have a high computational cost for generating sufficient actions for each state. Instead of working on policy constraints, SVR is the first value regularization algorithm to achieve support-constrained optimality. Moreover, SVR only needs to query the behavior density of in-dataset ( s, a ) pairs, making it less susceptible to model errors.

6 Conclusions and Limitations

In this work, we propose a novel value regularization method, SVR, which penalizes the Q -values for all OOD actions while maintaining standard Bellman updates for ID ones. We show that the policy evaluation operator of SVR is a contraction, whose fixed point outputs unbiased Q -values in ID region and underestimated Q -values in OOD region. Furthermore, SVR guarantees strict policy improvement until convergence to the optimal support-constrained policy. Empirical results validate the theoretical properties of SVR and demonstrate its SoTA performance on the D4RL benchmarks.

One limitation of SVR lies in the need of pre-training the behavior model. An exciting direction for future work would be to achieve supported value regularization without explicit behavior policy estimation.

Acknowledgment

This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801, National Natural Science Foundation of China under Grant 61827804.

References

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , pages 10-4, 2019.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems , 34:7436-7447, 2021.

Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=Y4cs1Z3HnqL .

David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in Neural Information Processing Systems , 34:4933-4946, 2021.

Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Bestaction imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems , 33:18353-18363, 2020.

Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning , pages 3852-3878. PMLR, 2022.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning , pages 3682-3691. PMLR, 2021.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456 , 2019.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016.

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning . Citeseer, 2002.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems , 12, 1999.

Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning , pages 5774-5783. PMLR, 2021.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations , 2022. URL https: //openreview.net/forum?id=68n2s9ZJWF8 .

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing offpolicy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32, 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33: 1179-1191, 2020.

Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Reinforcement learning: State-of-the-art , pages 45-73, 2012.

Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning , pages 1702-1712. PMLR, 2022.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https:// openreview.net/forum?id=VYYf6S67pQc .

Xiaoteng Ma, Yiqin Yang, Hao Hu, Qihan Liu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with value-based episodic memory. arXiv preprint arXiv:2110.09796 , 2021.

Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems , 34:19235-19247, 2021.

Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman. 1 year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research , 36(1):3-15, 2017.

Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine Learning , pages 23829-23851. PMLR, 2023.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016.

Art B. Owen. Monte Carlo theory, methods and examples . https://artowen.su.domains/ mc/ , 2013.

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems , 1, 1988.

Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-policy temporal-difference learning with function approximation. In ICML , pages 417-424, 2001.

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning , pages 387-395. Pmlr, 2014.

Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems , 28, 2015.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems , 33:7768-7778, 2020.

Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=KCXQ5HoM-fy .

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140 , 2021.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellmanconsistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:6683-6694, 2021.

Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=ueYYgo2pSSU .

Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust offline reinforcement learning via conservative smoothing. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=_QzJJGH_KE .

Shentao Yang, Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou. A behavior regularized implicit policy for offline reinforcement learning. arXiv preprint arXiv:2202.09673 , 2022.

Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems , 34:28954-28967, 2021.

Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, and Xiangyang Ji. In-sample actor critic for offline reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id= dfDv0WU853R .

Jing Zhang, Chi Zhang, Wenjia Wang, and Bing-Yi Jing. Apac: Authorized probabilitycontrolled actor-critic for offline reinforcement learning. arXiv preprint arXiv:2301.12130 , 2023.

Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning , pages 1719-1735. PMLR, 2021.

A Proofs

In this section, we present the proofs for all the theories in the paper.

We first restate the two definitions in the paper.

Definition 3 (Support-constrained policy, Definition 1) . The support-constrained policy class Π is defined as

Definition 4 (Optimal support-constrained policy, Definition 2) . The optimal support-constrained policy π ∗ Π is:

where Q ∗ Π satisfies the support-constrained Bellman optimality equation:

Proposition 4 (Contraction, Proposition 1) . In the whole S × A space and for any π , T π SVR is a γ -contraction operator in the L ∞ norm.

Proof.

Let f 1 and f 2 be two arbitrary functions.

For all ( s, a ) s.t. β ( a | s ) = 0 , we have

For all ( s, a ) s.t. β ( a | s ) > 0 , we have

Therefore, we have ∥T f 1 -T f 2 ∥ ∞ ≤ γ ∥ f 1 -f 2 ∥ ∞ .

Proposition 5 (Proposition 2) . For any π , the fixed point of T π SVR satisfies

Proof. By Proposition 4, T π SVR is a contraction and has a unique fixed point. Assume the fixed point is f π . Thus,

Define (ˆ s, ˆ a ) := argmin s ∈S ,a ∈ supp( β ( ·| s )) f π ( s, a ) , and p π ( s, a ) := E s ′ ∼ P ( ·| s,a ) ,a ′ ∼ π ( ·| s ′ ) π ( a ′ | s ′ ) .

As

The last inequality holds because f π (ˆ s, ˆ a ) := min s ∈S ,a ∈ supp( β ( ·| s )) f π ( s, a ) . Now we define a shorthand

Because λ ∈ [0 , 1] , we have

(

Therefore, for all ( s, a ) s.t. β ( a | s ) > 0 , it holds that f π ( s, a ) ≥ f π (ˆ s, ˆ a ) ≥ Q min . Besides, For s, a ) s.t. β ( a | s ) = 0 , it holds that f π ( s, a ) = Q min . Thus f π ( s, a ) ≥ Q min , ∀ s, a . Before we prove f π ( s, a ) ≤ Q π ( s, a ) when β ( a | s ) > 0 , we first prove T π SVR f π ( s, a ) ≤ T π f π ( s, a ) , ∀ s, a

For any ( s, a ) , We have

Therefore,

Thus, it holds that T π SVR f π ( s, a ) ≤ T π f π ( s, a ) , ∀ s, a .

It follows that

Now we prove π π

In conclusion, for any π , the fixed point of T π SVR satisfies

Theorem 3 (Fixed point, Theorem 1) . SVR yields support-constrained π i : π i ∈ Π . The fixed point of T π i SVR is

Therefore, for the policies π i in SVR, this fixed point provides unbiased Q -values for all ID actions and underestimated Q -values for all OOD actions.

Proof. By Proposition 4, T π SVR is a contraction and has a unique fixed point. For any π i , the fixed point of policy evaluation satisfies f π i = T π SVR f π i .

According to Proposition 5, the policy improvement at this iteration satisfies

Therefore, for arbitrary initial policy π 0 , π i ( i > 0) in SVR is support-constrained: π i ∈ Π , i > 0 . Now we show the fixed point of T π i SVR ( i > 0) . Assume the fixed point is f π i :

The fourth equality holds because π i is support-constrained and thus the expectation E a ′ ∼ π i ( ·| s ′ ) has non-zero mass only on support-constrained a ′ , i.e., β ( a ′ | s ′ ) > 0 .

Therefore, for the policies π i ( i > 0) in SVR, the fixed point of T π i SVR is

This fixed point provides unbiased Q -values for all ID actions and underestimated Q -values for all OOD actions.

Proposition 6 (Proposition 3) . Only in the ID region ( β ( a | s ) > 0 ) and when the evaluated policy is support-constrained ( π ∈ Π ), T π CQL is a contraction operator. Its fixed point satisfies

Proof.

Let f 1 and f 2 be two arbitrary functions.

We first consider the OOD region ( β ( a | s ) = 0 ). We separate it into π ( a | s ) = 0 and π ( a | s ) > 0 .

For all ( s, a ) s.t. β ( a | s ) = 0 and π ( a | s ) = 0 ,

For all ( s, a ) s.t. β ( a | s ) = 0 and π ( a | s ) > 0 ,

In both cases, T π CQL is not a contraction. Therefore, the contraction region of T π CQL (if exists) cannot contain the OOD region ( β ( a | s ) = 0 ).

Now we consider the ID region ( β ( a | s ) > 0 ).

If π is not support-constrained, for all ( s, a ) s.t. β ( a | s ) > 0 and ∀ π / ∈ Π , we have

Since π / ∈ Π , the expectation E a ′ ∼ π ( ·| s ′ ) can have non-zero mass on some ( s ′ , a ′ ) in the OOD region. Thus T π CQL cannot be a contraction.

If π is support-constrained, for all ( s, a ) s.t. β ( a | s ) > 0 and ∀ π ∈ Π , we have

The last inequality holds because π ∈ Π and thus the expectation E a ′ ∼ π ( ·| s ′ ) has non-zero mass only on support-constrained a ′ , i.e., β ( a ′ | s ′ ) > 0 .

Therefore, in the ID region ( β ( a | s ) > 0 ) and when ∀ π ∈ Π , T π CQL is a γ -contraction operator. Assume the fixed point is f π . We have

We define R as the vector of reward function, and P π as the transition matrix on state-action pairs induced by policy π : P π ( s,a ) , ( s ′ ,a ′ ) := P ( s ′ | s, a ) π ( a ′ | s ′ ) . Now Eq. (38) can be written in vector form:

The row ( s, a ) of the matrix ( I -γP π ) -1 is ρ π sa / (1 -γ ) , i.e., the unnormalized discounted stateaction occupancy induced by policy π with initial state-action pair ( s, a ) (Lemma 1.6 in [1]). Therefore, in the ID region ( β ( a | s ) > 0 ) and when π ∈ Π , the fixed point is:

Theorem 4 (Strict policy improvement to support-constrained optimal, Theorem 2) . SVR yields support-constrained π i and guarantees monotonic performance improvement:

where the improvement is strict in at least one state until π ∗ Π is found.

Proof. In Theorem 3, we have proved that SVR yields support-constrained π i and the fixed point of policy evaluation is:

Now we prove the monotonic improvement results. We start out with the performance difference lemma [16]. Given two policies π ′ , π ,

where d π ′ ,s is the normalized discounted state occupancy induced by policy π ′ from starting state s . Thus for i > 0 ,

The third equality holds because π i ( i > 0) is support-constrained and f π i ( s, a ) = Q π i ( s, a ) for β ( a | s ) > 0 by Eq. (40). The last inequality holds because π i +1 is the greedy policy with respect to f π i ( s, a ) and it holds that E a ∼ π i +1 f π i ( s ′ , a ) ≥ E a ∼ π i f π i ( s ′ , a ) at every s ′ .

When the improvement V π i +1 ( s ) -V π i ( s ) is 0 at every state s , it implies E a ∼ π i +1 f π i ( s, a ) = E a ∼ π i f π i ( s, a ) at every s , because the point mass on s is 'contained' in d π i +1 ,s : d π i +1 ,s ( s ) > 0 . In this case,

For β ( a | s ) > 0 ,

Consequently,

The last equality holds as π i +1 is support-constrained and is greedy with respect to f π i . It can be seen that Eq. (49) is the support-constrained Bellman optimality equation in Eq. (19): Therefore,

where the last equality is the definition of the support-constrained optimal policy π ∗ Π .

To conclude, the policy iteration with T π SVR guarantees monotonic performance improvement, and the improvement is strict until the optimal support-constrained policy π ∗ Π is found.

B Experimental Details

B.1 Experimental details on D4RL experiments

All hyperparameters of SVR are included in Table 2. Note that the only hyperparameter we tuned is the penalty coefficient α . We use α = 10 for Adroit tasks and α = { 0 . 001 , 0 . 02 } for Gym-MuJoCo tasks ( λ = 0 . 02 for expert and medium-expert datasets, λ = 0 . 001 for medium, medium-replay, random datasets). The discrepancy of α in this two domains is due to their characteristics. Adroit

10

5

10

4

10

3

10

2

10

1

Penalty Coefficient

0

50

100

hopper-medium-replay-v2

10

5

10

4

10

3

10

2

10

1

Penalty Coefficient

0

50

100

walker2d-medium-replay-v2

10

5

10

4

10

3

10

2

10

1

Penalty Coefficient

0

50

100

walker2d-medium-v2

10

5

10

4

10

3

10

2

10

1

Penalty Coefficient

0

50

100

walker2d-medium-expert-v2

datasets are mostly collected by human behavior and the data coverage is very narrow, so strong value regularization is needed to keep value functions from overestimation. We set the standard deviation of the sampling distribution in SVR as 0 . 2 , which our experiments have demonstrated to be an insensitive parameter. And following TD3+BC [8], we normalize the states in all datasets.

For evaluation, we average returns over 10 evaluation trajectories and 5 random seeds on all tasks. The reported results are the normalized scores, which are offered by the D4RL benchmark [7] to measure how the learned policy compared with random and expert policy:

B.2 Experimental details on tabular maze experiments

As depicted in Fig. 1(a) in the paper, the task is to navigate from bottom-left to top-right in as few steps as possible, with a wall in the middle. The agent receives a reward of 0 for reaching the goal and -1 for all other transitions. Episodes are terminated after 100 steps and γ is set to 0 . 9 . We first collect 10 , 000 transitions using a random policy. Then we remove all the transitions containing rightward actions in the bottom-left 4 × 4 region to introduce OOD actions. It makes the optimal support-constrained policy (see Fig. 1(b) in the paper) differ from the actual optimal policy in the environment. To magnify the impact of bootstrapping from OOD actions, we used optimistic initialization for each algorithm (i.e., initialized all Q -values with Q max ). This ensures overestimation occurs in some states and we can observe how well the algorithms mitigate it.

Here we are verifying the support-constrained optimality of the SVR operator in the tabular MDP, and the actual updates of CQL and SVR are Eq. (4) and Eq. (7) in the paper respectively. We run experiments over 5 random seeds that affect dataset collection and policy/Q initialization.

B.3 Experimental details on noisy dataset experiments

In noisy dataset experiments, we construct a 'noisy' dataset by combining the random and expert datasets in D4RL with different expert ratios. The size of the combined dataset is set to 1 × 10 6 . In some environments, the size of the random or expert dataset in D4RL is slightly smaller than 1 × 10 6 , so we directly utilize the corresponding D4RL dataset when expert ratio is 0 or 1 .

The hyperparameters of SVR follow Table 2: α = 0 . 001 for expert ratio of 0 and α = 0 . 02 for all other expert ratios. For CQL, we tune its regularization coefficient in { 5 , 10 , 20 , 30 } (performs relatively well in this range) and present the best results obtained for each dataset.

C More Experimental Results

C.1 Ablation on the penalty coefficient

We also conduct an ablation study on the penalty coefficient α of SVR in Fig. 6. Experimental results indicate that different datasets have different requirements for α , but choosing α ∈ [1 e -3 , 1 e -2] generally induces relatively satisfying performance.

0

10 1

10 2

10 3

10 4

10 5

Trained steps of

0

50

100

Normalized Return

hopper-medium-v2

0

10 1

10 2

10 3

10 4

10 5

Trained steps of

0

50

100

Normalized Return

walker2d-medium-v2

0

10 1

10 2

10 3

10 4

10 5

Trained steps of

0

50

100

Normalized Return

hopper-medium-exp-v2

0

10 1

10 2

10 3

10 4

10 5

Trained steps of

0.0

0.5

Training Loss of

hopper-med

walker2d-med

hopper-med-exp

C.2 Comparisons with additional baselines

To make a more comprehensive comparison, we also compare SVR with APAC [52], GAN-Joint [49], SPOT [43], MCQ [27], and IAC [51] on the D4RL benchmark. The results are shown in Table 3. The results indicate that our methods have better performance than these baselines.

C.3 Empirical study on the behavior model error of SVR

To empirically investigate SVR under different behavior model errors, we run SVR using different checkpoints of the behavior model, which are obtained at different steps in the behavior model training process (Eq. (15) in the paper). The model error is controlled by the number of steps taken to train the behavioral model. The results are shown in Fig. 7. We observe that the performance of SVR increases with the number of training steps of the behavioral model. Notably, the performance of SVR stabilizes at a high level after only 10 2 steps of behavioral model training, where the model has not been adequately trained. Theoretically, in Eq. (16) in the paper, the error of β ω only affects the weights of rewarding ID actions. Thus, an imperfect model can make the maximization and minimization of ID Q -values not cancel out well, but have little effect on OOD ones (still penalizing all OOD Q-values). We hypothesize that this is the reason why SVR can still achieve good performance with an imperfect behavior model.

C.4 Computational cost

We test the runtime of SVR on halfcheetah-medium-replay-v2 on a GeForce RTX 3090. The results of SVR and other baselines are shown in Table 4. It takes 2h40min for SVR to finish the task, which is comparable to other baselines. Note that it only takes two minutes for the pre-training part. SVR is computationally more efficient than CQL, because the practical CQL algorithm needs to sample 10

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

halfcheetah-expert-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

15

30

halfcheetah-random-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

hopper-expert-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

hopper-medium-expert-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

hopper-medium-replay-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

hopper-medium-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

15

30

hopper-random-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

walker2d-expert-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

walker2d-medium-expert-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

walker2d-medium-replay-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

walker2d-medium-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

15

30

walker2d-random-v2

actions at each state to compute log-sum-exp [23], while SVR only needs to sample a single action at each state. SVR can achieve better performance with lower computational costs.

C.5 Learning curves of SVR

Learning curves on Gym-MuJoCo tasks and Adroit tasks are presented in Fig. 8 and Fig. 9, respectively. The curves are averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.

D Broader Impact

Societal. Offline reinforcement learning (RL) holds significant promise in facilitating and expanding practical applications of RL, including domains such as robotics, recommendation systems, healthcare,

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

75

150

pen-expert-v0

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

pen-human-v0

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

pen-cloned-v0

and education, where the data collection processes are often costly or risky. However, it is important to acknowledge the potential negative societal impacts associated with any offline RL algorithm. One such concern is that the offline data used for training may contain inherent biases, and these biases can potentially transfer to the learned policy. Additionally, it is worth considering the potential impact of offline RL on employment, as it contributes to the automation of tasks traditionally performed by human experts, such as factory automation or autonomous driving. Addressing these challenges will contribute to the responsible development and deployment of offline RL algorithms, maximizing their positive impact while minimizing negative societal consequences.

Academic. We reexamine the fundamental objective of value regularization in offline RL and propose supported value regularization, which not only offers enhanced theoretical guarantees but also demonstrates remarkable improvements over existing methods on widely recognized offline RL benchmarks. This research potentially offers researchers a novel perspective and a promising avenue for exploring value regularization and achieving support-constrained optimality in offline RL.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the offline RL problem, and the authors proposes adding the Support Value  Regularization (SVR) in learning Q functions, motivated from the way how CQL add value regularizations. The authors add SVR for all OOD while maintain Bellman update for ID samples. Experiments shows that the SVR-regularized method could outperforms other competitors in D4RL tasks.\n\nSTRENGTHS:\n1. This paper proposes Support Value Regularization (SVR) on solving offline RL problems. The authors suggest adding extra regularization on OOD points, and utilize importance sampling techniques to calculate the Q-values in OOD region.\n2. The superiority of SVR method is demonstrated from experiment studies. \n3. Theoretical proof shows the policy improvement could lead to a better policy in each iteration.\n\nWEAKNESSES:\nSince the method add regularization to OOD regions, it should belongs to Conservative offline RL methods. It is suggested to compare this methods with other less/mild conservative method. In addition, as the SVR is density-based regularization, the authors is suggested to compare with other density-based Offline RL methods. Some less/mild conservative or density-based offline RL methods are listed below, the authors cites some of them, and it seems better to compared some of them.  \n\n[1] Supported Policy Optimization for Offline Reinforcement Learning.  https://arxiv.org/abs/2202.06239\n[2] Mildly Conservative Q-Learning for Ofﬂine Reinforcement Learning.  https://arxiv.org/abs/2206.04745\n[3] Provably Good Batch Reinforcement Learning Without Great Exploration. https://arxiv.org/abs/2007.08202 \n[4] A Behavior Regularized Implicit Policy for Offline Reinforcement Learning. https://arxiv.org/pdf/2202.09673.pdf\n[5] APAC: Authorized Probability-controlled Actor-Critic For Offline Reinforcement Learning. https://arxiv.org/abs/2301.12130\n\nQUESTIONS:\n1. Some notations are not clear. In addition, what is $\\beta$ in Equation 3? It also does not appear before, though we know it is behavior policy. It is suggested to give an explanation. \n\n2. A key component in this paper is to estimate the support values. The state-action pairs are regarded as ID when $\\beta(a|s) >0$, however, when using the Gaussian model to estimate behavior policy, all actions has a density larger than 0 (since gaussian density are larger than 0 everywhere), but clearly not everywhere are in distribution. \n\n3. In addition, when estimating behavior policy $\\beta_w$, the simple Gaussian model is suggested. However, as the dimension of state could be large, and the entire Support of the behavior policy relies on the behavior density estimation. A simple gaussian model may not be able to deliver a strong estimator.  In section 4.5, the authors give a simple comparison between SVR and SVR-VAE, we suggest to give more detailed analysis. \n\n4. The sampling distribution $u(a,s)$ is selected as gaussian in Section 3.4 with mean $\\pi$. Could the authors give extra abalation study on how $u(a,s)$ affects the results? Selecting $u(a,s)$ as gaussian is generally optimal/close-optimal?\n\nLIMITATIONS:\nSee weakness and questions", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the use of Importance sampling to distinguish between ID and OOD actions, and to operate the corresponding actions accordingly. In addition, as most of the current work is similar, it also uses model to fit behavior policy, but does not need to overly consider the accuracy of the model. It underestimates OOD actions and unbiased estimates ID actions, achieving good performance on D4RL and Adroit tasks.\n\nSTRENGTHS:\n1. The paper conducts relatively sufficient experiments to demonstrate its viewpoints.\n2. The proofs of theorems and propositions are relatively sufficient.\n3. The hyperparameters and parameters settings in the paper are clearly defined for easy reproduction.\n\nWEAKNESSES:\n1. Importance sampling is widely used in reinforcement learning because of its unbiased nature. In this paper, the main contribution is to use importance sampling to punish the deviation of OOD action and ID action respectively. The differences or advantages between the proposed SVR method and existing methods should be described clearly.\n2. The description of certain experiments should be clearer. For example, what do the colors of the grid represents in Figure 1? In Figures 1 and 2, it is hard to see how the value function described in the paper is represented.\n3. The algorithms in the comparative experiment are only some classic algorithms, lacking some algorithms of the same type, such as using importance resampling (Zhang et al. 2023), and there is no mention of the ablation experiments about the penalty coefficient α.\n(Zhang et al. 2023) Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, Xiangyang Ji. In-sample Actor Critic for Offline Reinforcement Learning. International Conference on Learning Representations, 2023.\nhttps://openreview.net/forum?id=dfDv0WU853R\n\nQUESTIONS:\n1. Recently, there has been a way to use Importance sampling or Importance resampling to determine whether it belongs to OOD actions. Please explain the relationship between your work and the current works, or explain the differences and advantages of your method.\n2. In Figure 2, I cannot see that there has a serious overestimation problem with the value function caused by the iteration of the vanilla policy.\n3. Please provide a more detailed explanation of Figure 1, such as what the colors of the grid represent.\n4. Why does the Walker2d-v2-random in Figure 1 in appendix only run a small portion and not continue to run?\n\nLIMITATIONS:\nThere is no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new offline RL method in which the popular  assumption that the new policy should be close to the behavior poplicy is abandoned and just penalizing the OOD action would be OK. Analysis show that the method has the policy improvement property. Experimental results partially verify the effectiveness of this 'more simple' method (compared to CQL).\n\nSTRENGTHS:\nThe presentatation is clear and the proposed method is easy to understand. \n\nThe final results look good.\n\nWEAKNESSES:\n1. By relaxing the behavior clone requirement (first eq of eq(4)) and only penalizing the OOD action, the new policy would have better chance to find a better one. This is reasonable. However, since the search space becomes larger, the learning process should become more difficult. Unfornuately, the experimental results (Table 1) are eager to show this method is good, without illustrating how the reward is accumulated  as usually done.\n\n2. According to eq.(7), this method would be even more sensitive to the quality of the learnt behavior policy itself than CQL, while the behavior policy  is generlly quite difficult to learn. This may generate extrapolation error. For example, if we utilize a Gaussian approximal behavior policy and the a1, a2 are both the in-sample actions at state s. Then the intropolated OOD a3 between a1 and a2 would likely has a large probability density of the behavior policy, which would introduce extropolation error. Hence the paper should provide more evidence to  support the claim that 'SVR is less susceptible to model errors' .\n\n3. Generalization problem. The proposed SVR could be considered as a clipped Conservative Q-Learning, where the part with ID actions is dropped. Then the Q value generated by this method would have the same generalization problem as CQL. For example, if we have learnt an approximal behavior policy perfectly, then the Q value of any perturbed state-action pair (unseen but closed to the dataset) would drop greatly, although we may need a smoother Q function to enhance the robustness [RORL: Robust Offline Reinforcement Learning via Conservative Smoothing].\n\n4. Lack of the experiments that exploring the behavior of the SVR agent. That is, the paper is supposed to show how the SVR behaves at the ID states and OOD states in practice.\n\n5.  Although the theoretical analysis demonstrates the performance improvement guarantee of SVR, it deviates from the core problem that this paper attempts to solve, where SVR introduces smaller detrimental changes to in-distribution Q-values.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to enforce a new squared penalization term for computing  the target Q-values in offline RL. In particular, their penalty tries to only apply to target out-of-distribution (OOD) actions by taking the difference between the importance-sampled and the true estimates of a uniform distribution over actions (or any other distribution will full support on the action space), where the importance-ratio is estimated with a proxy model for the behavior policy. Under idealized theoretical settings, the paper shows that off-policy optimization using this penalty has a fixed point which penalizes only OOD actions and guarantees strict policy improvements. The authors empirically validate their algorithm showing effective performance on a subset of D4RL and a toy maze setting.\n\nSTRENGTHS:\n1) The paper provides a concise and comprehensive introduction to the off-policy literature and the utilized notation in Section 2.\n2) The paper correctly recognizes an important issue of existing popular algorithms in the offline literature: while their stated aim is to penalize behavior only outside the support set of the offline datasets, in practice they resort to density-based heuristic penalties.\n3) Overall, the paper is clear and easy to read.\n4) The authors provide some analysis and ablations beyond just reporting their method's performance.\n\nWEAKNESSES:\n1) Generalizing some of the mentioned regularization issues to the whole literature would require very comprehensive evidence which the paper does not provide,  as it mostly focuses on CQL when analyzing prior work. Hence, I would tone down some of the statements, e.g. lines 26-28\"existing value regularization methods not only fall short in penalizing all OOD Q-values but also may introduce detrimental changes to in-distribution (ID) ones\" -> \" some of the most popular existing value...\". Additionally, claims like \"SVR guarantees strict policy improvement until convergence to the optimal support-constrained policy\" (333-334) should specify that this assumes an idealized setting, and does not apply in the stochastic optimization regime of deep learning.\n2) Related to the point above, I believe it would be very appropriate for the paper to mention the assumptions/conditions used to derive the theoretical results in the introduction and abstract to avoid having unbacked claims.\n3) A limitation of the method is that it requires access to the behavior's policy density.  It would be useful to also analyze the properties of the proposed penalization (theoretically and/or empirically) based on the modeling errors for the trained behavior policy proxy.\n4) In practice, CQL's policy is most often also a Gaussian distribution, and, thus, it covers the full support of the action space. Yet, the authors do not seem to consider or mention this property in their analysis. Moreover, since the authors also resolve to using the agent's Gaussian policy with an increased variance for the sampling (action-space covering) penalization distribution, it appears to me that the actual implementation of their algorithm is very similar to existing off-policy methods (e.g., even to CQL).\n5) The toy example in Section 4.1 seems quite misleading. From a practical perspective, it seems to me like CQL and SVR are algorithms with similar properties whose performance depends on hyperparameters.  Hence, I believe that simply showing one works and the other fails with no context (swept hyperparameters/different collected datasets) is not very informative.\n6) While the authors show some extensions of SVR replacing the Gaussian density estimator with a conditional VAE and using self-normalized importance sampling, results appear inconclusive and there is very little analysis done to motivate these findings.\n8) Selection of the distribution to cover the whole action space, clearly influences the penalization magnitude of different actions (Equations 5-6), and, therefore, performance.  There does not appear to be be any clear principle for choosing this important hyper-parameter for a given problem - apart from looking at online performance, which would break the assumptions of a fully-offline setting.\n\nMinor:\nTypos - Line 108: 'in policy improvement stage' -> 'in the policy improvement stage', Line 108: scare -> scarce.\n\nQUESTIONS:\n- Since also CQL uses a Gaussian policy with full support over the action space, do some of the theoretical claims of SVD generalize beyond this specific algorithm?\n- What is the intuition and what are the theoretical consequences of applying SNIS over importance sampling (Section 4.5)?\n- Even if considered a standard benchmark in the offline literature, I generally do not find evaluating on a subset of D4RL very informative in terms of empirically properties. Have the authors considered evaluating on additional/more comprehensive offline benchmarks? (e.g. [1])\n\n[1] Gulcehre, Caglar, et al. \"Rl unplugged: A suite of benchmarks for offline reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020).\n\nLIMITATIONS:\nThe authors quickly dismiss the constraint of having to model the behavior policy by stating that \"we empirically find that the Gaussian model can usually induce excellent performance, which only takes two minutes for pre-training.\" (Lines 337-338) This claim should be contextualized, as a shallow Gaussian policy is most definitely not sufficient beyond the considered toy settings and the relatively simple D4RL benchmark.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}