Supported Value Regularization for Offline Reinforcement Learning
Authors: Yixiu Mao
Hongchang Zhang
Chen Chen
Yi Xu
Xiangyang Ji
Venue: NeurIPS 2023 poster (2023)
Keywords: offline reinforcement learning
TL;DR: Supported value regularization for offline RL with support-constrained optimal convergence.

Abstract: Offline reinforcement learning suffers from the extrapolation error and value overestimation caused by out-of-distribution (OOD) actions. To mitigate this issue, value regularization approaches aim to penalize the learned value functions to assign lower values to OOD actions. However, existing value regularization methods lack a proper distinction between the regularization effects on in-distribution (ID) and OOD actions, and fail to guarantee optimal convergence results of the policy. To this end, we propose Supported Value Regularization (SVR), which penalizes the Q-values for all OOD actions while maintaining standard Bellman updates for ID ones. Specifically, we utilize the bias of importance sampling to compute the summation of Q-values over the entire OOD region, which serves as the penalty for policy evaluation. This design automatically separates the regularization for ID and OOD actions without manually distinguishing between them. In tabular MDP, we show that the policy evaluation operator of SVR is a contraction, whose fixed point outputs unbiased Q-values for ID actions and underestimated Q-values for OOD actions. Furthermore, the policy iteration with SVR guarantees strict policy improvement until convergence to the optimal support-constrained policy in the dataset. Empirically, we validate the theoretical properties of SVR in a tabular maze environment and demonstrate its state-of-the-art performance on a range of continuous control tasks in the D4RL benchmark.

Supported Value Regularization for Offline Reinforcement Learning
Yixiu Mao', Hongchang Zhang', Chen Chen', Yi Xu², Xiangyang
DDepartment of Automation, Tsinghua University
Schooo of Artificial Intelligence, Dalian University of Technology myx21 @eailmmllmlaaammaaaammmaam 1s tsinghua. edu, xyji@tsinghua edu
Abstract
Offline reinforcement learning suffers from the extrapolation error and value overes- timation caused by out-of-distribution (OOD) actions. To mitigate this issue, value regularization approaches aim to penalize the learned value functions to assign lower values to OOD actions, However, existing value regularization methods lack a proper distinction between the regularization effects on in-distribution (ID) OOD actions, and fail to guarantee optimal convergence results of the policy. To this end, we propose Supported Value Regularization (SVR), which penalizes the Q-values for all OOD actions while maintaining standard Bellman updates for ID ones. Specifically, we utilize the bias of importance sampling to compute the summation of Q-values over the entire OOD region, which serves the penalty for policy evaluation. This design automatically separates the regularization for ID and OOD actions without manually distinguishing between them. In tabular MDP, we show that the policy evaluation operator of SVR is contraction, whose fixed point outputs unbiased Q-values for ID actions and underestimated Q-values for OOD actions. Furthermore, the policy iteration with SVR guarantees strict policy improvement until convergence the optimal support-constrained policy in the dataset. Empirically, we validate the theoretical properties of SVR in a tabular maze environment and demonstrate its state-of-the-art performance range of continuous control tasks in the D4RL benchmark.
Introduction
Offline Reinforcement Learning (RL) aims to learn a policy from fixed dataset collected by some behavior policy [24, 26). It can tap into existing large-scale datasets [15, 30] for safe and efficient learning. However, it suffers from the extrapolation error [10] caused by out-of-distributin (OOD) actions, which can further result in severe overestimation of value functions [26].
To mitigate this issue, value regularization approaches attempt to penalize the Q-values of OOD actions to introduce conservatism in value estimation [23, 20, 46, 14, 3, 6]. However, we observe that existing value regularization methods not only fall short penalizing all OOD Q-values but also may introduce detrimental changes to in-distribution (ID) ones. Specifically, most of them involve adding penalties to the critic learning objective [23, 50, 27, 6. 48]. However, due to the difficulty of distinguishing between ID and OOD actions, they typically adopt the idea of being pessimistic about the actions under the current policy and optionally optimistic about the ones within the dataset [23, 50, 46, 6]. Rather than relying ID OOD, this regularization is essentially based on the policy density, which we show is problematic when the dataset is heavily corrupted by sub- optimal actions. Other works involve reducing the Bellman target with uncertainty quantifiers [14, 3] or ensembles [2, 34]. However, if not incorporating additional penalties, they do not provide learning signals OOD Q-values, logically only suppressing the overestimation in the ID region [14, 2].
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this work, we revisit the original objective of value regularization and ask a question "Can we devise a value regularization method that penalizes all OOD Q-values without affecting ID ones"? We point out that the inability of prior methods to achieve this leads to a lack of strong theoretical guarantees for policy performance. offline RI., the best policy that can be guaranteed should lie within the support of the behavior policy, known as the optimal support-constraine policy [22]. However, existing value regularization methods do not provide reliable guarantees for converging to it. On the other hand, several endeavors aim to learn it through specific policy constraints [10, 22, 1 I1], but their empirical performance leaves considerable room for improvement.
To this end, we propose Supported Value Regularization (SVR), new value regularization method that penalizes the Q-values for all OOD actions while maintaining standard Bellman updates for ID ones. Specifically, we leverage the bias of importance sampling to calculate the summation of Q-values over the entire OOD region, which serves as the penalty term for policy evaluation. This design circumvents the dilemma of distinguishing between ID) and OOD actions and separates the regularization effects on them automatically. Theoretically, SVR offers stronger guarantees than previous methods. Under tabular MDP, we show that the SVR policy evaluation operator is a contraction in the whole state-action space and its fixed point outputs unbiased Q-values for ID actions and underestimated Q-values for OOD ones, while the fixed point of Q in CQL [23] may underestinate or overestimate Q-values. More importantly. SVR guarantees strict policy improvement until convergence to the optimal support-constrained policy. while prior value regularization methods lack such each-step improvement and global optimal convergence guarantees.
In practice, SVR is easy to implement by adding penalty term to the ordinary policy evaluation loss. Empirically, we validate the support-constrained optimality of SVR in a tabular maze environment, where baselines fail to converge. Moreover, SVR achieves state-of-the -hrtthrrt performance on range of continuous control tasks in the D4RL benchmark [7] and shows strong advantages on noisy datasets.
2 Preliminaries
Offline RI. The environment in RI. is typically modeled as a Markov Decision Process (MDP) (M, AS P, R. y, do). with state space S, action space A, transition dynamics P A ASS). reward function R : SS x AA- > RRmin Rmax] discount factor 7 E [0. 1), and initial state distribution do [41]. The goal of RL to find policy π. S + A(A) that maximizes the expected discounted return: Be (-1,))))))))))) P 0 R'{ss, (t)]. For any policy π, we define the value function as (s) = Ex IE 0o
M =
tt= 7 as) s] and the state-action value function (Q-value function) as (s, = D2 z=0 R((,, 05) 8, ag a). By the boundedness of rewards, we have Q" E [Qmin: Qmax], where Qmin == Rmin/ (1 - 7) and Qmax == Rmax/ (11 - -7). In addition, we use P. to denote the normalized and discounted state-action occupancy of policy with the initial state-action pair (s, a): PTa (s',a') (1 7) Ei "I, [1] 8', az a'] |so 8, ag = a]
Actor-critic methods [19] alternate between evaluating the policy by iterating the Bellman operator (T Q) (s, a) := R(s a) + (() IQ (s', aa)], and improving the policy by maximizing the Q- valuee In offiine RL, the agent is provided with a fixed dataset D collected by some behavior policy
B. Ordinary actor-critic algorithms [9, 12, 39] minimize the following losses alternately:
(o) = ((..a.))~DD((0
a) R(s,a) a) --
(8', ad))²) (policy evaluation)
(1)
(Ø) = Eu-D.a-n , (Qo (s) (policy improvement)
(2)
where ne. is a policy parameterized by Φ Qo a) a Q function parameterized by 0, and Q (s,a) is target Q function whose parameters updated via Polyak averaging [32].
Value regularization. In offline RL, OOD actions a' can produce erroneous Bellman targets and lead to an inaccurate estimation of Q-values. Then in policy improvement, the policy tends to prioritize OOD actions whose values have been overestimated, resulting in poor performance.
To address this issue, value regularization methods regularize the Q function to introduce conservatism in value estimation [23, 20]. As the most representative one, CQL [23] minimizes the following policy evaluation loss, which guarantees to obtain underestimated V functions:
min E.-D,a Q
[Q((s.)) a) - 1000--"-'' Q' (s, a))]] + (BuwD, (Q(s, Ex-D,a ((s, (3)
where Q' is the target Q function and is hyperparameter.

---

3 Supported Value Regularization
In this section, we first briefly analyze the existing density-based value regularization and identify some important issues. Then we propose Supported Value Regularization (SVR) to penalize all OOD Q-values while maintaining standard updates for ID ones. Next, we conduct thorough analysis of SVR and demonstrate its theoretical superiority. Last, we present the implementation details of SVR.
3.1
Density-based value regularization
large portion of value regularization methods are built on the idea of being pessimistic about the actions under current policy and optimistic about the actions within the dataset [23, 50, 27, 6J. Instead of relying on ID or OOD, we show that this regularization is essentially based on the policy density, which we refer as density-based value regularization. Specifically, we take CQL (Eq. (3)) [23], the foundational work on value regularization, as an example for mathematical analysis. Here we remove the strong assumption supp(z) S sup)(3) in their paper and analyze its regularization effects on the Q functions. From Eq. (3), we can obtain the analytical solution of Q, which corresponds to the following policy evaluation operator TeQL:
(*((,)) -a [(ale]
[[ass] -1),
B(als) > 0,
Teou a)
20,
B(als) (( 0, B(a(s) - 0, (a(s)= = 0.
(4)
((s,a),
Considering the iteration of policy evaluation: Q*+ (s, = Tequ Q² (s, a), we have Observation 1:
Observation 1. In each iteration k, compared to T TeQr. (1) lowers Qu+ a) when (aass) > ((als); (2) raises Qi+1 (s, a) when =(aas) < B(als); (3) obtains the same Qh+1 (8,a) when
(als)) = B(als) > 0; (4) does not update Q a) when t(als) = B(aas)=
For (1)(2)(3), lowering or raising Q-values based on the relative density between π and B could be problematic, especially when the dataset contains a large portion of bad actions. In such cases, assuming the policy has found the optimal action a", it holds that ा (a" (8) > =((a]s), and thus CQL will lower (s, a²) )); for some bad action à E D, it holds that (ā(s)) < B(āss), and thus CQL will raise Q(s, â). That is, CQL tends to raise the Q-values of numerous bad actions and lower the Q-values of scare good actions, forcing the policy to choose bad actions in policy improvement stage. Regarding (4), in the entire OOD region (8(als) = 0), the penalization is only performed where (a]s) > 0. Since the distribution of æ is narrow compared to the whole action space, the regularization region of CQL and most existing methods [50, 6, 27, 3, 48] tends to be too narrow.
3.2 Supported value regularization
In this paper, we aim to penalize Q-values for all OOD actions (a supp(3)) and maintain standard Bellman updates for ID ones (a E supp(3)). That is, the regularization is solely determined by the support of B, which we refer as supported value regularization.
However, it is a well-known challenge to distinguish between ID and OOD actions. Directly determining if the behavior density exceeds a threshold [43] would necessitate extremely precise density estimator, and it is particularly difficult to estimate the density of OOD actions. In this work, we draw inspiration from the link between support and Importance Sampling (IS) to circumvent this dilemma and achieve the same goal.
We begin by explaining its principle with simplified notations. IS computes 4 = [E))) (x))
to estimate 12 = fxr).. When assuming supp(p) S supp(g), IS is unbiased: / - 12. However, when removing the support ussumption, IS actually computes 1 Exeempp(g) p(x) We observe that / := 1 - 122 = Engsupp(g) f() gives the summation over the out-of-support region. In the offline RL. setting, let / be the Q function. If we choose the behavior policy / as g and any samplable distribution that covers the whole action space as p. then minimizing would lower all OOD Q-values without affecting the ID ones. Besides, since both distribution q and p are samplable in this case, we can obtain / and la based on sampling, which provides unbiased to optimize.
3

---

Therefore, we minimize the following loss for policy evaluation in SVR:
min Ea [(((s,u) (
Q
(5)
u(als) (Q(s, Qmin) )²
+ BawDDan [(Q(s, Qmin) Bgg Dan
B(als)
where ((-ls) is any samplable distribution (e.g., Gaussian, uniform) whose support covers the whole action space, and Quin = Rminn/ (1 -Y) -) is the minimal possible of the MDP. According the analysis above, Eq. (5) is equivalent to the following minimization problem:
min D - Q
aBOND
Σ
u(als)(Q(s,a) - -umin)
(6)
agsmpp(#((14))
It is clearer from Eq. (6) that the loss in Eq. (5) penalizes all OOD Q-values without affecting the ID ones. Note that optimizing Eq. (5) requires pre-training behavior model to output the behavior density ((als) 3) in the IS ratio. Compared with other methods that also require the behavior model [43, 20, 11, 27], SVR is less susceptible to model errors. This is because SVR only needs to query the behavior density of in-dataset (s, a) pairs, thus not requiring much generalization ability of the model, making it relatively easier to estimate accurately.
From an optimization perspective, minimizing Eq. (6) reduces all OOD Q-values with a strength proportional to u (d)oddss) Therefore, we can even choose various flexibly penalize the OOD region. For example, let assign higher weight to areas with a high probability of overestimation. Note that u can have any positive density in the ID region without affecting the ID Q-values, thereby eliminating the necessity to manually distinguish between ID) and OOD regions. From an optimal solution perspective, Eq. (5,6) lead the following SVR policy evaluation operator:
TV = (=Q(s,)),
((a > 0,
Qmin.
else.
In contrast to Observation 1, make the following claim for SVR. Claim 1. In each iteration k, compared to Tx, TSvR obtains the same Qu+ a) when B(als) > 0 (ID region); and lowers 1 a) when ((a)s) 3) = 0 (OOD region).
3.3 Analysis
In this section, we refer to the policy iteration, whose evaluation part is repeatedly applying Tgvn and whose improvement part is vanilla maximization of Q, as SVR. We will give a comprchensive analysis of SVR, including the fixed point in policy evaluation, the monotonic improving performance in policy improvement, and the support-constraine constraiend optimal convergence for the whole policy iteration.
We first define the support-constrainn policy set [22]. which plays an important role in our analysis. However, is worth noting that SVR does not impose any constraint or regularization on the policy.
Definition (Support-constraing) policy). The support-constrained policy class 11 is defined as
II
[
(als) whenever 3(als) = =0}
Following prior works [22], we also define the optimal support--onstrained policy को. Definition 2 (Optimal support-constrained policy). The optimal support support-constrrinee policy कn is:
का. (als) =
argmax
Qin(s,a))
(9)
a' d'Esupp(B((le)
where Q satisfies the support-constrriined Bellman optimality equation:
(s) = R(s,a) 7By P(.baa)
max a' ('Esupp(B((10)
Quis' (a)s",d))
(10)
Proposition (Contraction). In the whole S space and for any n, Tsvn is "-contraction operator in the Lox norm.

---

Therefore, in the policy evaluation stage of SVR, any initial Q function can converge to a unique fixed point by repeatedly applying TSVR. We give this fixed point in the following theorem. Theorem (Fixed point), SVR yields support- constrrinnn n; : ni E II. The fixed point of TsVR is
(Th 3(als) 0, Quins
(T)
(11)
3{a = 0.
Therefore, for the policies n; in SVR, this fixed point provides unbiased Q-values for all ID actions and underestimated Q-values for all OOD actions.
Though Theorem guarantees that SVR only obtains support-constrainet policies during learning, we show that, even for any x (IT & IT due various errors practice), the fixed point of TsvR still ensures that Q will not be overestimated over the entire action space.
Proposition For any n, the fixed point of Tsvv satisfies
{
Quii S (8, S Q" (s, (s. Qiin:
B(als) > 0, B(ass) O.
(12)
For comparison, the following proposition characterizes the contraction property of CQL. Proposition 3. Only in the ID) region (3(als) > 0) and when the evaluated policy is support- constrained (π E LI), TĒQL is a contraction operator. Its fixed point satisfies
(= ( -1)) B(als) 0.
= ("(ssa)=Q((s.a)) T- (1%a)
(13)
Therefore, TeQu has no fixed point in the OOD region. Even in the ID) region, the condition æ E IT is required and its fixed point may underestimate or overestimate Q-values a complicated way.
Finally, we show that SVR guarantees strict policy improvement for each iteration until convergence the optimal support-constrinnet policy को. Note that existing value regularization methods fail to guarantee such each-step policy improvement and optimal convergence results.
Theorem (Strict policy improvement to support-constrainet optimal). SVR yields support- constrained Ti and guarantees monotonic performance improvement:
VK+1 (s) 2 V**(s) Vss
(14)
where the improvement is strict in at least one state until "i is found.
3.4 Practical implementation of SVR
SVR is easy to implement and we design the practical algorithm to be as simple as possible
Algorithm Supported Value Regularization (SVR)
to avoid some complex modules confusing our
1: Initialize behavior policy Bu, policy network TØ.
algorithm's 's impact on the final performance I.
Q-network Qo. and target Qentwwork Qo
2: // Behavior Policy Pre-training
Behavior model. Following previous
for each gradient step do
works [22, 44], we learn Gaussian model
4: Sample minibatch (s, a) ~ D Update by maximizing Jo(W) in Eq. (15)
Bw for the behavior policy by maximizing
6: end for
Jm(w) = Ee,a-D log (a)s)
(15)
// Policy Training
8: for each gradient step do
Sampling distribution u. In our method, u in Eq. (5) can be any samplable distribution that covers the entire action space. In our implementation.
Sample minibatch (s, a, r, D
10: Update 0 by minimizing Lo in Eq. (16)
11:
Update & by minimizing [.)(0) Eq. (2)
set to the Gaussian with the same mean as π and a fixed variance that is much larger than that of क. This choice is made because during learning, π is typically located where Q is maximized and where overestimation is most likely occur. So if is OOD, this choice will assign higher weight to the area around π to reduce the Q-value. Note that if x is ID, will not penalize the Q-values in the ID region even though may have a higher weight there.
12:
Update target network: 0' <- 7)0' + T
13:
end for
Our code is available at https//gihab.ccm//AAoYXXUUU
5

---

(a) tabular maze (h) support optimal
(c) vanilla
(d) CQL
(c) SVR
Figure I: (a) The maze environment with OOD actions. (b) The optimal support-constrained policy and its value function. (c) (d) (e) The value functions and policies learned by vanilla policy iteration, CQL and SVR, respectively. The resulting trajectories are illustrated by white dashed lines. SVR is capable of obtaining the optimal support-constrained policy and value function.
Policy evaluation. With policy TΦ. Q function Qo, and target Q function Qo we minimize the following loss for policy evaluation:
(0) ((s.as))= a) R(s, (s))) (')
u(aas) (Qo(s, m2in)²
(16)
+ EawDa [[Q a) Qmin) EE D.DD
B_ (a|s)
where is hyperparameter and Qmin := unknown, we choose the smallest reward in the dataset as Rmin-
Rmin/(C (1 7)). For the environment where Rmin is
Overall Algorithm. Putting everything together, we summarize our final algorithm Algorithm 1. Our algorithm first trains the estimated behavior policy to obtain the behavior density. Then it turns to the actor-critic framework for policy training.
Experiments
In this section, we conduct several experiments to justify the validity of our method. We aim to answer five questions: (1) Does SVR actually converge to the optimal support-constrained policy? (2) Does SVR perform better than previous methods standard offline RL benchmarks? (3) When does SVR empirically benefit the most compared to the density-base
regularization? (4) How should
we select the sampling distribution of SVR in practice? (5) How does the implementation of each component affect SVR? More experimental details and results are provided Appendix B and C.
4.1 Support--onstrained optimality in the tabular setting
Toy Maze
We use simple maze environment to verify the support- constrained optimality of SVR. As depicted in Fig. 1(a), the task is to navigate from bottom-left to top-right, with wall in the middle. The agent receives a reward of 0 for reaching the goal and for all other transitions. Episodes are terminated after 100 steps and y is set to 0.9. We first collect 10, 000 transitions using random policy. Then we remove all the transitions containing rightward actions the bottom-left
-20
Episode Return
SVR
CQL
Vanilla
60
Supp. Optimal
100
4 region to introduce OOD actions. It makes the optimal support-constrained policy (see Fig. 1(b)) differ from the actual optimal policy in the environment.
10
20
30
40
50
Iteration
Figure Learning curves of SVR, CQL and vanilla policy it- eration on the toy maze.
We test three algorithms: vanilla policy iteration [41], CQL [23]. and SVR. Fig. ((c, d, e) depict their learned value functions and policies respectively. To show convergence results, we also present the learning curves in Fig. along with the performance of the behavior policy and the optimal cupport-constraine policy. The results indicate that vanilla policy iteration has severe over- estimation of value functions, leading to poor performance. On the other hand, although CQL does not overestimate V functions. it fails to converge and performs poorly when the dataset is highly sub- optimal. In contrast, SVR converges to the optimal support-constrained policy and the learned value function closcly matches the true support- conptratco optimal value function, verifying Theorem 2.

---

Table 1: Averaged normalized scores on the D4RL benchmarks over five random seeds. Note that m
medium, m-r medium-replay, m-e medium- expper, expert, random.
Dataset
BC
OneStep
TD3BC
AWAC
BCQ
BEAR
UWAC
CQL
IQL
SVR (Ours)
halfcheetah-m
42.0
50.4
48.3
47.9
46.6
43.0
42.2
47.0
47.4
1.2
hopper-m
56.2
87.5
59.3
59.8
59.4
51.8
50.9
53.0
66.2
103.5+00.4
walker2d-m
71.0
84.8
83.1
71.8
75.4
73.3
78.3
92.4++1.2
halfchectah-mm-r
36.4
42.7
44.6
44.8
42.2
36.3
35.9
45.5
44.2
52,5++3.0
hopper-m-r
21.8
98.5
60.9
69.8
60.9
52.2
25.3
88.7
94.7
103.7::1.3
walker2d-m-r
24.9
61.7
81.8
78.1
57.0
7.0
23.6
81.8
73.8
95.6.1.2.5
halecheetah-m-e
75.1
64.9
95.4
46.0
42.7
75.6
94.2.1..22
hopper-m-c
51.7
108.6
98.0
100.1
106.9
50.6
44.9
105.6
91.5
111.2:+00.9
walker2d-m-c
101.2
111.3
110.1
110.0
107.7
22.1
96.5
107.9
109.6
109.3++0.2
halfchcetah-c
92.9
88.2
96.7
81.7
89.9
92.7
92.9
96.3
95.0
96.1=0.7
hopper-e
110.9
106.9
107.8
109.5
109.0
54.6
110.5
96.5
109.4
111.1-10.4
walker2d-e
107.7
110.7
110.2
106.3
106.6
108.4
108.5
109.9
110.0
halfcheetah-r
2.6
2.3
11.0
6.1
2.2
2.3
2.3
17.5
13.1
1.2
hopper-r
4.1
5.6
8.5
9.2
7.8
3.9
2.7
7.9
7.9
31,0
walker2d-r
1.2
6.9
1.6
0.2
4.9
12.8
2.0
5.1
5.4
1.5
gym-v2 total
784.2
1041.2
1013.2
975.6
968.0
581.7
756.2
1010.2
1033.1
1200.5
pen-expert
85.1
61.6
111.0
115.2
114.9
105,9
111.9
107.0
110.2
138.9119.2
pen-human
34.4
73.7
54.9
25.5
68.9
-1.0
21.7
37.5
71.5
73.1
pen-cloned
56.9
31.8
10.4
44.0
26.5
33.1
39.2
37.3
70.2.11..4
adroit-vO total
176.4
167.1
229.7
151.1
227.8
131.4
166.7
183.7
219.0
282.2
4.2
Comparisons on D4RL benchmarks
Then we evaluate our approach on the D4RL benchmarks [7]. We compare SVR with prior state-of-
the-art offline RL methods, including BC [37], BCQ [10], BEAR [22]. OneStep RL [4], TD3+BC [8]. AWAC [33], UWAC [45], CQL [23], and IQL [21].
The results are reported in Table 1. In both Gym-MuJoCo and Adroit domains, SVR achieves state-of-the-art performance and outperforms prior value regularization methods (CQL., UWAC)
large margin. We also observe that SVR has strong advantages on the sub-optimal datasets (medium, medium-replay, random). This is because SVR exploits the optimal support-constraine the dataset in a theoretically sound way and is less affected by the average quality of the dataset. In addition, while BCQ and BEAR are the policy constraint methods that are also designed to search for the optimal policy within the behavior support, they have much poorer performance. As stated in previous works [11, 44], they are limited respectively by the errors of the generative model and the unsuitability of using Maximum Mean Discrepancy (MMD) to characterize the support constraint, In contrast, without querying oft-oe-dataset behavior density, SVR takes the form value regularization to search for the optimal support-constrained policy and achieves superior performance. For learning curves and more experimental details, please see Section the supplementary material.
policy in
4.3 Comparisons noisy datasets.
In this section, we aim to validate that, compared with the existing density- based value regularization, the supported value regularization of SVR will benefit when the dataset contains large portion of bad actions. To this end, we construct a "noisy" dataset by combining the random dataset and the expert dataset, and then evaluate SVR and CQL under different expert ratios.
The results are shown in Fig. 3. In all environments, SVR outperforms CQL over nearly all expert ratios. Moreover, the expert ratio gets lower, the improvement is more significant. The performance of CQL is susceptible to the expert ratio and exhibits sharp decrease at the expert ratio 20%, while SVR can still retain good or expert performance (most spark in Hopper and Walker2d).
4.4 Empirical study the sampling distribution of SVR
In this section, we investigate how to practically select the sampling distribution u in SVR, which controls the relative strength of penalizing different OOD Q-values. In our implementation, is the
7

---

Hopper
Walker2d
Ant
Haffcheetah
120
150
120
Nomalized Return
SVR
100
100
120
100
80-
80
80
60
40
40
40
20
20
20
1.0
0.8
0.6
0.4 0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8 0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
Expert Ratio
Expert Ratio
Expert Ratio
Expert Ratio
Figure 3: Evaluations of SVR (supported value regularization) CQL (density value regularization)
on noisy datasets, which are made by mixing random and expert datasets with varying expert ratios.
0=0.1
0.2
0.3
-= 0.5
a = 0.7
Uniform
halfchoctah-mmedium
hopper-medium
hopper-medium-reelees
100
Normalized Return
100
100
100
50
S0
0.25 0.5
0.75
0.25
0.25 0.5
0.75
0.25
0.75
Gradient Steps (x109)
Gradient Steps (x10%)
Gmddient Steps (x10%)
Gradient Steps (x10%)
Figure 4:
Learning curves SVR with different sampling distribution u. a is standard deviation of Gaussian. SVR is able to converge to good performance under various sampling distributions.
Gaussian with the same as the current policy. Here vary its standard deviation and present the corresponding results in Fig. 4. We do not show the results for a > 0.7 because such a large o would cause most sampled actions to lie at the boundary of the action space. Instead, we use Uniform distribution to represent a = 0O.
As shown in Fig. 4, SVR is able to converge to good performance over very wide range of However, if a is too small, SVR may not be able to adequately penalize all OOD Q-values and the IS ratio will have large variance, thereby disrupting the learning process (see 0.1 in hopper-m-r). Conversely, if a is too large or if the uniform distribution is used, sampling from u may fail to emphasize the key areas where overestimation is most likely to occur (as indicated by the current policy), leading to insufficient mitigation overestimation and inferior performance (see Uniform in hopper-m, hopper-m-r, walker2d-m-4 and 0.7 in hopper-m).
4.5 Effects of components in SVR
Density estimator. We speculate that a more precise behavior density estimator may further improve SVR. So following previous works [10, 43], consider to replace the Gaussian density estimator Bu with the conditional variational auto- encoder [18, 40]. We refer to this vari- ant SVR-VAE VAE. The results of SVR- VAE are shown in Fig. (left). Over- all, SVR-VAE has only marginal perfor- mance gains over SVR.
SVR-VAE
SVR-SNIS
50
50
Percent Diffeence
25
25
01
-50
NCC 20
Wam
H1C
71-000
-50
BCC-0
17.00
W-m
HC
NC
Figure 5: Percentage performance difference of the SVR variant compared to the original algorithm. (Left) SVR with VAE density estimator. (Right) SVR with self- normalized importance sampling. HC HalfCheetah,
IS techniques. Nelf-Normalized Im- portance Sampling (SNIS) [35] is some- times adopted to reduce the high variance of [38]. The SNIS estimator has lower variance but is biased [35]. Here we test an SVR variant EVR-SNIS which normalizes the IS ratio across the batch. As shown in Fig. 5 (right), the perfor-
H= Hopper, Walker2d.
8

---

mance of SVR-SNIS is worse in hopper-m and hopper-m-e, and slightly better in other tasks. This discrepancy may be due the varying requirements for variance and bias across different tasks.
5 Related Work
Offline RL. In offline RL, extrapolation error and overestimation caused by OOD actions pose significant challenges [10]. Among various solutions, value regularization methods aim to introduce conservatism in value estimation [23, 20, 29, 6, 3, 27], while policy constraint approaches enforce proximity between the trained policy and the behavior policy, either explicitly via divergence penal- ties [44, 22, 13, 8]. implicitly by weighted behavior cloning [5, 36, 33, 42, 31J, or directly through specific parameterization of the policy [10, 11, 53J. Another line of the methods, instead, opt for in-sample learning, which involves formulating the Bellman target with only the actions in the dataset [4, 28, 21, 51, 47]. However, the performance of most existing methods is largely confined by the average quality of the dataset [25]. In contrast, SVR guarantees the convergence to the optimal support-constrained policy and achieve superior performance in sub-optimal datasets.
Value regularization. We divide the value regularization methods in offline RL into two categories: direct Q penalization and Bellman target reduction. The former involves adding penalties to the critic learning loss, trying to penalize OOD Q-values [23, (2, 46, 6, 27]. Due to the difficulty of determining the OOD region, they typically penalize Q-values under the current policy distribution [23, 3, 27, 48] and optionally incorporate a maximization term under the data distribution for milder pessimism [23, 50, 6]. Thus, existing works of this category belong policy density based value regularization, rather than ID/OOD based. On the other hand, the Bellman target reduction methods either subtract an uncertainty quantifier from the Bellman target [14,3) 3] directly use the minimum of Q ensembles [34] to compute the target [2, 9]. However, if not incorporating additional penalties, they do not provide learning signals on OOD Q-values [2]. It empirically works because after the minimization over ensemble (or subtracting the uncertainty quantifier), the random generalized OOD Q-values will be smaller than ID ones with high probability, reducing the impact of OOD Q-values on learning. However, it requires a large number of Q networks and is much more computationally expensive [2].
Support-constrained optimality. In absence of additional information beyond the dataset, the optimal support-constrained policy represents the best policy that we can hope to obtain. Recent works have considered various policy constraints to learn it, but their performance still leaves considerable room for improvement. For example, BEAR [22] attempts to keep the learned policy within the support of the behavior policy by minimizing MMD between them. However, this choice lacks a theoretical guarantee, and Wu et al. [44] empirically found that MMD has no performance gain over KL. Other works adopt specific parameterization of the policy to constrain to the behavior support [10. 11]. They first pre-train generative model (e.g., VAE) for the behavior policy. Then during learning, they sample several actions at each state and choose the one with the highest Q-value as the output of the policy, aiming to obtain the maximum within the behavior support. However, these methods are vulnerable to model errors [11] and have high computational cost for generating sufficient actions for each state. Instead of working on policy constraints, SVR is the first value regularization algorithm to achieve support-constrained optimality. Moreover, SVR only needs to query the behavior density of in-dataset (s, a) pairs, making it less susceptible to model errors.
6 Conclusions and Limitations
In this work, we propose novel value regularization method, SVR, which penalizes the Q-values for all OOD actions while maintaining standard Bellman updates for ID ones. We show that the policy evaluation operator of SVR is contraction, whose fixed point outputs unbiased Q-values in ID region and underestimated Q-values in OOD region. Furthermore, SVR guarantees strict policy improvement until convergence to the optimal support-constraiie policy. Empirical results validate the theoretical properties of SVR and demonstrate its SoTA performance on the D4RL benchmarks.
One limitation of SVR lies in the need of pre-training the behavior model. An exciting direction for future work would be to achieve supported value regularization without explicit behavior policy estimation.

---

Acknowledgment
This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801, National Natural Science Foundation of China under Grant 61827804.
References
[1] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pages 10-4, 2019.
[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- peastadnty-busedd based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
[3] Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations, 2022. URL htt.httpss//oppneeviee pet//orrumid=YA4ss11
[4] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in Neural Information Processing Systems, 34:4933-4944 2021.
[5] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33:1 111181833136333 2020.
[6] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning. pages 3852-3878. PMLR, 2022.
[7] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
[8] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 11-20132-20144 2021.
[9] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018
[10] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[11] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. International Conference Machine Learning, pages 3682-3691. PMLR, 2021.
[12] Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018
[13] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences dialog. arXiv preprint arXiv:1907.0044 0000000000046.. 2019.
[14] Ying Jin, Zhuoran Yang, and Zhaoran Wang. pessimism provably efficient for offline rI? In International Conference on Machine Learning, pages 5084-5096 PMLR, 2021.
[15] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
10

---

[16] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
[18] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
[19] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
[20] Ily Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference Machine Learning, pages 5774-5783. PMLR, 2021.
[21] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference Learning Representations, 2022. URL https: //openreview net/ dd-.mmm2.dd....22......
[22] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[23] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33: 1179-1191, 2020.
[24] Sascha Lange Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning, Reinforce- ment learning: State-of-the-ar, pages 43-73, 2012.
[25] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference Robot Learning, pages 1702-1712. PMLR, 2022.
[26] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 20ti:2000500644: 2020.
[27] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https:// openreview www.eeeweeeetfooeeeeeeeeeee wet./eoruu/idd-eVEESSSSS.
[28] Xiaoteng Ma, Yiqin Yang. Hao Hu, Qihan Liu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with value-based episodic memory. arXiv preprint 1xXi::2100009996 2021.
[29] Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems, 34:19235- 19247, 2021.
[30] Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman, 1 year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research, 36(1):3-15,200 2017.
[31] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine Learning, pages 13829-23855 PMLR, 2023.
[32] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[33] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09353 2020.
11

---

[34] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
[35] Art B. Owen. Monte Carlo theory, methods and examples. https://aattn su domains/ 2013
[36] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighhtd regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2019.
[37] Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
[38] Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-policy lemporal-difference learning with function approximation. In ICML, pages 417-424, 2001.
[39] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wienstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning. pages 387-395. Pmir, 2014.
[40] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
[41] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[42] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778. 2020.
[43] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning- In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: // /oppnrveeeeeeeeee seet/forumnid-xCXSSta
[44] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, arXiv preprint arXi::1911.1136/ 2019.
[45] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint 21X.::2005001000 2021.
[46] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683-6694, 2021.
[47] Haoran Xu, Li Jiang. Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL, with no OOD actions: In-sample learning via implicit value regularization. In The Fleventh International Conference on Learning Representations, 2023.
URL https::/ net/ wwwpeeeeouuueeeeeeeeeeeeeeeeeeee forum? http//formmmmudddo idmmiiiyoousssss
[48] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust offline reinforcement learning via conservative smoothing. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL htttp:://pperretteeeee wtwp:rewwwwweeeeeeeeeeeeeeeeeeee net feet/forrmiid= _trpiddd________H___ KE.
[49] Shentao Yang, Zhendong Wang. Huangjie Zheng, Yihao Feng, and Mingyuan Zhou. A behavior regularized implicit policy for offline reinforcement learning. arXiv preprint 00676,, 2022.
[50] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances neural information processing systems, 34:28954-28966 2021.
12

---

[51] Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, and Xiangyang Ji. In-sample actor critic for offline reinforcement learning. In The Eleventh International Con- ference on Learning Representations, 2023. URL. htt.hhtst//opeeeeiieieeeee wwt/ffrumm? id=
dfDvOWU853R.
[52] Jing Zhang, Chi Zhang, Wenjia Wang, and Bing- Yi Jing. Apac: Authorized probability- controlled actor-critic for offline reinforcement learning. arXiv preprint arXiv:2301.1211 2023.
[53] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference Robot Learning, pages 1719-1735. PMLR, 2021.
13

---

A Proofs
In this section, we present the proofs for all the theories in the paper.
We first restate the two definitions in the paper.
Definition 3 (Support-constrained policy, Definition 1), The support-constrained policy class IT is defined as
II π
([ls) O whenever 3(als) = 0}
(17)
Definition 4 (Optimal support-constrainet policy, Definition 2). The optimal support-constrainee
policy mf is:
Tn (a(s) =
argmax Pils,
(18)
a' d'Esupp(B((Ia)
where Qi satisfies the support-constrained Bellman optimality equation:
Qirls' .d)
@n(s = R(s,a) +7BBs
max ('Esuup(((110))
(19)
Proposition 4 (Contraction, Proposition In the whole S A space and for any n. TSVR y-contraction operator in the Coo norm.
Proof.
TSRRQ(s, {
T=Q(s,a), B(als) >
(20)
Qmin: else.
Let Ji and f2 be two arbitrary functions.
For all (s s.t. B(als) O, we have
(s, Tivnf((s,a) Qmin Qmio S 711f1 f2lloo
(21)
For all (s) s.t. B(als) 0, we have
(T,nfi(s,a) - Tsvnf2(s,a) (B
f2('',d')]
S ( f2))'
(22)
$ rllsi - f21lo
Therefore, we have I[TSR - -TVvRf2lle S 711fi - f2lloc.
Proposition 5 (Proposition 2). For any the fixed point of Tsvn satisfies
Qmin < (s, s (s,
B(als) > 0, B(a 0.
(23)
f = Qmin:
Proof. By Proposition 4, Tsvn is contraction and has unique fixed point. Assume the fixed point fa. Thus,
(²) = (s,) B(als) 0. Quin
(24)
else.
Define (s, argminness.ccmmp((ttt))) (s, a), and p" (s) = Es Be--
14

---

As B(ā[s) > 0,
(=, " (s, (8.0)
=r(3, à) + Ba Bo
wl (8
r((,a) +ylBg tl (8) T
=)(š,a) + +yBBg
Σ
((),))
Σ
<T r(a'ls')Qmin Qm
('Esup((((119)
('ssup((3(118))
=r(3,ā) +yBBa
Σ
(d''s'))T((''''
('Euup(3()119)
@'guup((((181)
Tmin +BBg
Σ
Σ
T(a' Q
d'Esupp(8(18))
Esupp((((18))
The last inequality holds because (*, a) = min, (e (s)
Now we define a shorthand
= Ey
Σ
(())
('Esup(((()88))
It follows that
[= a) 2 Tmin + +*A/"(8,0) + 7(1 -))
Because À E [0, 1], we have
(" Tmin + 7 - A)Q
1-72
(1 - 7) + 7(1 A)Qmin
1 1-7d
Q min
Therefore, for all (s, a) s.t. ((a]s) > O, it holds that (* a) 2 (8) 2 Qmin- Besides, For (8, a) s.t. B(als) = 0, it holds that f"((s a) = Qmin- Thus f (8, a) 2 Qmin Vs,a.
Before we prove (s) s (s, when B(als) > 0, we first prove TEvRJ" (s, (s, S Tx (*, Vs,a
For any (8, a), We have
TTfe r((s,)) + (o
2 Tmin + 7B, B, (-10 (-)8) [Qmin]
Qmin
(25)
Therefore,
(TFPM(8))) B(ass) > 0.
T (s,a)
(26)
Qmm S
else.
Thus. it holds that Tsvn fdt (s, TT) Vs.a.
15

---

Now we prove (s) & (s))) when B(als) > O. For B(als) > 0,
as = =
= =r(s,a) + P ( (s',
= -r(s,a)+ + t.
[vR (((''''))
<r(s,a) + t, E.
[T "F'(6'
- -r(s,a) + +TE, E.
(r(6 ,a)) + 7 E (-lor) (100)) (f" (s"))""")]]
- r((s,) + +E,, E.
(r(6 ,d) + Ba (11o)) [ ad",
I <((s, + +,E,
E. (66 + Bu (o0r)) (T"
("
$ E
(r($)+);a+)))l 8, a, =
= Q" (s,a)
In conclusion, for any ı, the fixed point of SVR satisfies
{
Qmin I (" a) s (s. 3(als) > 0, B(a 0.
(27)
(s, = Qmin
Theorem 3 (Fixed point, Theorem 1). SVR yields support-comstrriteddd n; E I1. The fixed point 7f is
(T a) = 9Th (3.
3(als) >
Quin
(28)
B(als)=
Therefore, for the policies Ti in SVR, this fixed point provides unbiased Q-values for all ID actions and underestimated Q-values for all OOD actions.
Proof. By Proposition 4, TVR is contraction and has a unique fixed point. For any Th, the fixed point of policy evaluation satisfies f f
According to Proposition 5, the policy improvement at this iteration satisfies
(als)) = argmax fT (8, a') = T
a argmax a' e'Esup((((11))
(s,
(29)
Therefore, for arbitrary initial policy no. mi (i> 0) in SVR support-constrained: my E II, > 0. Now we show the fixed point of TSVR (i > 0). Assume the fixed point is fm::
= (s, {
fT B(als) 0, Qmin:
(30)
else.
For B(als) > 0,
(8. [ (s,
r((s a) + EBg E,g (ft (s',a') (s', a')]
r(s, E E.g m
-r(s,a) a) E. E,, (1 (r(5 + 7E, Egg n (189)) ( (s"""" "Miss"aa"] a")]]
r(s, a) + yE, E, (r + EE, Ea (187) [TVR (s" aa)]
8
ET,
((()))), 8,
1=
TT (s,a)
16

---

The fourth equality holds because is non-zero mass only on support-constrained a', i... 3(a'ls') > 0.
and thus the expectation (1) has
Therefore, for the policies n. (i > 0) in SVR, the fixed point of T5V is
B(als) 0, f (s, a) 9th ( Qua:
(8,a), a),
(31)
B(a|s)= = 0.
This fixed point provides unbiased Q-values for all ID actions and underestimated Q-values for all OOD actions.
Proposition 6 (Proposition 3), Only in the ID region (3(als) > 0) and when the evaluated policy is support-constrriine (IT E 11), Téo,, a contraction operator. Its fixed point satisfies
a*) = (s,sa)) 1-7 (p -1). B(als) > 0.
(32)
Proof.
T=)(s,a)
s(aas) B[a|s] 1)
B(als) 0,
((q =
-00,
B(als) = 0, T(assss)) > 0,
(33)
a(s,a),
B(als) 0, (a
Let fi and f2 be two arbitrary functions.
We first consider the OOD region (8(a 0). We separate it into x<als) and m(ass) >
For all (s, a) s.t. B(als) = and ([ass) = Té, Tcq,ff(s,,) (f, - fa(s,a)
(34)
For all as s.t. B(als) = and ((als) > 0,
(T, Tequf2(s, a) = =|00- xol
(35)
In both cases, Tequ is not a contraction. Therefore, the contraction region of Tequ (if exists) cannot contain the OOD region (3(als) = 0).
Now we consider the ID region (B(als) 0).
If is not support-constrained, for all (s, a) s.t. B(als) and Vx II, we have
(i - T2 a) = fi a) (2) (tt [fi (s', a') fa(s',
(36)
Since & II, the expectation (+)=) can have non-zero on some (s', a') in the OOD region.
Thus Teq cannot be contraction.
is support-constrained, for (s, a) 8.l. B(als) and VT II, we have
(T, -T,,
iT (s, - (((s
(y (s f2(5' a')l
< P (l f2(s', a')l]
(37)
S? (s.)):((a(=)20
max
11 (s f2(8,a)
The last inequality holds because II and thus the expectation (ar=a(()")) has non-zero mass only on support--onstrraned a', i..., ((a'ls') > 0.
Therefore, the ID region (B(als) Assume the fixed point is fr. We have
and when Vo Too, is 7-contraction operator.
(s, T -(..)))7T((( - [<aas)) ((ass) -1 1) B((as) > 0.
(38)
17

---

We define R as the vector of reward function, and Pa as the transition matrix on state-action pairs induced by policy x: P(r (x.a).('' := P(s'ls, a))(aa)))(a))))))) s'). Now Eq. (38) can be written in vector form:
[(aas)) 1)
(( = [R - (5, -
B(als) >
3(a|s)
π
+ f = -I-
R - a
-
(s, B(als) >
8
a= = (s) - a (1-
π
(s, a), B(d|s) > 1 3
The row (s, a) of the matrix - y is P% (1 - 7), i.e., the unnormalized discounted state- y action occupancy induced by policy ι with initial state-action pair (s,0) (Lemma 1.6 in [T]). Therefore, in the ID region (3(als) 0) and when II, the fixed point is:
= - a 1-7 (Paa)² (Ø (1-1) 3(als) > 0.
Theorem 4 (Strict policy improvement to support-constrainet optimal, Theorem 2). SVR yields support-constrained n, and guarantees monotonic performance improvement:
V (s) Z VT((s) Vs,
(39)
where the improvement is strict in at least one state until की is found.
Proof. In Theorem 3, we have proved that SVR yields support-constrained n, and the fixed point of policy evaluation is:
(s, a) = {{ TT (s.a), 3(a)s) > 0, Qmin B(a 0.
(40)
Now we prove the monotonic improvement results, We start out with the performance difference lemma [16]. Given two policies n², .,
V (s) (T(()
1 1-7 E (A x'(()'x'(('))))
(41)
where d 80 is the normalized discounted state occupancy induced by policy x' from starting state
Thus for O,
1
(8)+ V^h (s)
E o' Nn
11 (1
(+))(
(42)
1 γ
1
E Eg
[ BaNnis ( Earn, (5t(saa)))
(43)
1 7
1 En Bawwey (s' BEa 8 (8', ,a)
(44)
2
(45)
The third equality holds because n; > 0) is support-eonstrained and f (s. = t a) for 3(als) > 0 by Eq. (40). The last inequality holds because 7441 is the greedy policy with respect to fms (x, a) and holds that En-tun (8', (6',0) a) 2 E (s''',
34, a) at every s'.
When the improvement V*++1 (s) - 4Th (**(s) is O at every state 8, it implies Eanten f (8, = f" a) every s, because the point mass is "contained" in M 1 ( (s) 0.
In this case,
(" a) = f {
r(s, + T.y B.g was (f Qmin:
B(ass) > 0.
B(als) 0. (46)
18

---

For B(als) > 0,
(* = r(s, a) + B. (10) (f (s))
(47)
= = a) + y Ba (5'
(48)
= = + t.
max a' d'Esupp((((le"))
f (')
(49)
The last equality holds 1 is support-constraneet and greedy with respect to f It can be seen that Eq. (49) is the support-constrained Bellman optimality equation in Eq. (19):
Therefore.
(" =
Qi B(als) > 0, Qmin. B(a)s) O.
(50)
Consequently,
(a|s)
=
argmax fT (s,
(51)
argmax (s, ((supp(((()e)
(52)
argmax d'Esupp(((1a))
@n(s,aa)
(53)
(54)
where the last equality is the definition the support-constrained optimal policy को.
To conclude, the policy iteration with TšvR guarantees monotonic performance improvement, and the improvement is strict until the optimal support-constrained policy क is found.
L
B Experimental Details
B.1 Experimental details on D4RL experiments
Table 2: Hyperparameters SVR.
Hyperparameter
Value
Optimizer
Adam [17]
Critic learning rate
3 x 10
Actor learning rate
3 10-4 with cosine schedule
Batch size
256
Discount factor
0.99
Number of iterations
106
SVR
Target update rate
0.005
Policy update frequency
2
Number of Crities Penalty coefficient a
4
(0.001, 0.02) for Gym-MuloCo
(10) for Adroit
Standard deviation
0.2
Actor Critic
imput2256-26-ooupuu
Architecture
input-256-256-1
hyperparameters of SVR are included Table 2. Note that the only hyperparameter we tuned the penalty coefficient a. We use 10 for Adroit tasks and {0.001, 0.02) for Gym-MuJoCo tasks (A 0.02 for expert and medium-expert datasets, = 0.001 for medium, medium-replay, random datasets). The discrepancy of in this two domains is due to their characteristics. Adroit
19

---

walker26-medium-22
wallkerzddmmeddu---opp-e
100
Epiiodee Return
100-
Episods Return
Episode Return
Episnde Return
100-
50-
50-
50
0
10
10
10
10
10 Penalty Coefficient
10"
10
10
10
10
10
10
10
10
10
10
Penalty Coe foeffieeett
Penalit Coefficient
Penalty Coefficient
Figure 6:
Parameter study on the penalty coefficient of SVR. The curves are averaged over 4
random seeds, with the shaded area representing the standard deviation across seeds.
datasets are mostly collected by human behavior and the data coverage is very narrow, so strong value regularization is needed to keep value functions from overestimation. We the standard deviation of the sampling distribution in SVR as 0.2, which our experiments have demonstrated to be an insensitive parameter. And following TD3+BC [8], we normalize the states in all datasets.
For evaluation, we average returns over 10 evaluation trajectories and 5 random seeds on all tasks. The reported results the normalized scores, which are offered by the D4RL benchmark [7] to measure how the learned policy compared with random and expert policy:
learned policy return random policy return expert policy return random policy return
D4RL score 100
B.2 Experimental details on tabular maze experiments
As depicted in Fig. ((a) in the paper, the task is to navigate from bottom-left to top-right in as few steps possible, with wall in the middle. The agent receives reward of 0 for reaching the goal and -1 for all other transitions. Episodes are terminated after 100 steps and y is set to 0.9. We first collect 10, 000 transitions using random policy. Then remove all the transitions containing rightward actions in the bottom-left 4 x 4 region to introduce OOD actions. It makes the optimal support-constrained policy (see Fig. ((b) in the paper) differ from the actual optimal policy in the environment. To magnify the impact of bootstrapping from OOD actions, we used optimistic initialization for each algorithm (i.e., initialized all Q-values with Qmax). This ensures overestimation occurs in some states and we can observe how well the algorithms mitigate it.
Here verifying the support-constrained optimality of the SVR operator in the tabular MDP, and the actual updates of CQL and SVR are Eq. (4) and Eq. (7) in the paper respectively. We run experiments over 5 random seeds that affect dataset collection and policy/Q initialization.
B.3 Experimental details on noisy dataset experiments
In noisy dataset experiments, construct "noisy" dataset by combining the random and expert datasets in D4RL with different expert ratios. The size of the combined dataset is set to x 10 In some environments, the size of the random expert dataset in D4RL slightly smaller than 1 10 so we directly utilize the corresponding D4RL dataset when expert ratio is 0 or 1.
The hyperparameters of SVR follow Table 2: 0.001 for expert ratio of 0 and all other expert ratios. For CQL, we tune its regularization coefficient in (5, 10, 20, 30) (performs relatively well in this range) and present the best results obtained for each dataset.
0.02 for
C More Experimental Results
C.1 Ablation on the penalty coefficient
We also conduct ablation study on the penalty coefficient of SVR in Fig. 6. Experimental results indicate that different datasets have different requirements for o, but choosing a E [le - 3, - -2] generally induces relatively satisfying performance.
20

---

Table 3: Comparisons with additional baselines on the D4RL benchmark.
Dataset
APAC
GAN-Joint
SPOT
MCQ
IAC
SVR (Ours)
halfcheetah-med
58.4+22.6
44.0+H.2
58.4=1.0
64.3+-0.2
51.6+0.3
60.5+11.2
hopper-med
93.8+
86.4+10.9
86.0=8.7
78.4+-4.3
74.6+11.5
103.5++0.4
walker2d-med
58.5+7.4
69.9+6.4
86.4-2.7
91.0++0.4
85.2+-0.4
92,4+1.2
serthhetth--ee--ree
57.3112.8
33.4:12.4
52.2=1.2
56.8110.6
47.2-10.3
52.513.0
hopper-med
51.91:5.6
30,913.2
100.2:1.99
101.6:10.8
103.2:11.0
103.7
walker2d-med
16.31.5.
6.71222
91.6-28
91.315.77
93.2-1..8
95.6.12.55
ma
halfchectah
72.6+11.1
86.9=4.3
87,5+1.3
92.9=0.7
94,2++2,2
hopper-med-eee
71.1+10.7
99.3=7.1
111.240.1
109.3=4.0
111.2=00.9
walker2d ex
79.6
112.0#0.5
114.2++0.7
110.1
109.3+0.2
gym total
495.2
796.3
773.0
767.3
822.9
pen-expert
13..5+10.8
116.3=22.3
103.3
138.9=9.2
pen-human
71.02332
91.4=13.4
68.5
45.9+23.0
12.1
pen-cloned
27.647.1
47.94-20.5
49.44.4.3
36.41.9.2
70.2+17.4
adroit total
233.1
255.6
185.6
282.2
Nomalize Retura
100
boppar-medium--22
Normalizud Return
100
walker24-medum--v2
Nomalized lndd Reltura
hopper-modium--en
0.5
Training Loss of
hoppermme
wilkerd-mml
med
50-
50
50
0 10
10ah
10a
10
10a
0
10
10a
10
10"
10a
10
10a
10
10" 10a
10 10 10 10 10
Trained steps
Trained steps B.
Trained steps of A..
Trained steps P..
Figure 7: (Left) Performance of SVR under different behavioral model checkpoints, which are obtained at different steps in the behavioral model training process. (Right) Training loss of the behavioral model 3g at different training steps. The curves are averaged over 4 random seeds, with the shaded area representing the standard deviation across seeds.
C.2 Comparisons with additional baselines
To make more comprehensive comparison, we also compare SVR with APAC [52], GAN-Joint [49], SPOT [43], MCQ [27]. and IAC [51] on the D4RL benchmark. The results are shown in Table 3. The results indicate that our methods have better performance than these baselines.
C.3 Empirical study on the behavior model error of SVR
To empirically investigate SVR under different behavior model errors, we run SVR using different checkpoints of the behavior model, which are obtained different steps in the behavior model training process (Eq. (15) in the paper). The model error is controlled by the number of steps taken to train the behavioral model. The results are shown in Fig. 7. We observe that the performance of SVR increases with the number of training steps of the behavioral model. Notably, the performance of SVR stabilizes a high level after only 102 steps of behavioral model training, where the model has not been adequately trained. Theoretically. in Eq. (16) in the paper, the error of B.,. only affects the weights of rewarding ID actions. Thus, an imperfect model can make the maximization and minimization of ID Q-values not cancel out well, but have little effect OOD ones (still penalizing all OOD Q-values). We hypothesize that this is the reason why SVR can still achieve good performance with an imperfect behavior model.
C.4 Computational cost
test the runtime SVR on wuafihhetah-mmdduu--eelllee of SVR and other baselines are shown in Table 4. It takes 2h40min for SVR to finish the task, which comparable to other baselines. Note that it only takes two minutes for the pre-training part. SVR computationally more efficient than CQL, because the practical CQL algorithm needs to sample 10
on a GeForce RTX 3090. The results
21

---

Table 4: Runtime of TD3BC, IQL, CQL, SVR for halfcheetd-mmcciuu--rp----222 on a GeForce RTX 3090.
Algorithm
TD3BC
IQL
CQL
SVR
pre-training SVR
Runtime
1h
1h50min
4h10min
2h40min
2min
Sallchettab-opppp-22
Lod wwwellhhettab-menee hem.eetttmmmmmmmepepeet
100.
bulfeiccta-emmmmmm
Uptedd Rourr
Eptood Hoaer
Epiook harer
Erieck Iturss
20
50
50)
0.00
0:25
0.50 076 1.00
Yoo
025
0.50 0.75 1.00
0.25
0.50 0.75 1.00
8.00
0.25
0.50 1:25 1.00
0.35
0,50 0.75 1.00
Orvdierz Stppe Ox 10f)
Clvdient Stepe (x10f)
Gracienr Stege (x10
Cradient Stepe (x 101)
Qustier Stepe Ix10
bappermeduro
medunc-eppbyy.dd
mediumn
bepper 42
t00
10
Ipoodds Rtors
Eppodds Nitees
Ipical Ittues
Fpiiade
50
So
15
1000 025 0.5 1.00
0.0
0150
1.30
80u 0.5 0.25 1.00
8000 0:50 1.00
8:00 025 0.75
Grader Shps OK 10f)
Gindeet
(K1)
Gracier (x (x))
Gradient Oo 10%)
Gestion 101)
welker22 experty22
walker2 expertvv2
100
100
Eppaoek itmmm
Epponed iteam
Epiaak Rteum
Eppoodk Iteim
Epoodd
10
50
50
So
025
Steps 109)
(x10*)
Gradieor Stego (x100))
Stepe (x4)
Stepe
Figure 8: Learning Curves of SVR Gym-MuJoCo Tasks.
actions at each state to compute log-sum-exp [23], while SVR only needs to sample single action at each state. SVR can achieve better performance with lower computational costs.
C.5 Learning curves of SVR
Learning curves on Gym-MuJoCo tasks and Adroit tasks are presented Fig. and Fig. respec- tively. The curves are averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.
D Broader Impact
Societal. Offline reinforcement learning (RL) holds significant promise in facilitating and expanding practical applications of RL, including domains such as robotics, recommendation systems, healtheare.
pen expert vD
pen human
pen cloced
100
100
Eppicat Roturn
Eppisdde Behum
Epioado Rehern
75
30
8700 0.25
0.50 0.75
1.00
0.00
0.23
0.30
0.75
1.00
8.00
0.25
0.50
0.75
1.00
Gradient (x10f)
Gindient
1000.)
Gradient (x119)
Figure 9: Learning Curves of SVR on Adroit Tasks.
22

---

and education, where the data collection processes are often costly or risky, However, it is important acknowledge the potential negative societal impacts associated with any offline RL algorithm. One such concern is that the offline data used for training may contain inherent biases, and these biases can potentially transfer to the learned policy. Additionally, is worth considering the potential impact of offline RL on employment, as it contributes to the automation of tasks traditionally performed by human experts, such as factory automation or autonomous driving. Addressing these challenges will contribute to the responsible development and deployment of offline RL algorithms, maximizing their positive impact while minimizing negative societal consequences.
Academic. We reexamine the fundamental objective of value regularization in offline RL and propose supported value regularization, which not only offers enhanced theoretical guarantees but also demonstrates remarkable improvements over existing methods widely recognized offline RL benchmarks. This research potentially offers researchers a novel perspective and a promising avenue for exploring value regularization and achieving support-constrained optimality in offline RL.
23


Reviews:
[{"review_id": "WZB2U0Z5IN", "reviewer": "Reviewer_CX3L", "summary": "The authors propose to enforce a new squared penalization term for computing  the target Q-values in offline RL. In particular, their penalty tries to only apply to target out-of-distribution (OOD) actions by taking the difference between the importance-sampled and the true estimates of a uniform distribution over actions (or any other distribution will full support on the action space), where the importance-ratio is estimated with a proxy model for the behavior policy. Under idealized theoretical settings, the paper shows that off-policy optimization using this penalty has a fixed point which penalizes only OOD actions and guarantees strict policy improvements. The authors empirically validate their algorithm showing effective performance on a subset of D4RL and a toy maze setting.", "questions": "- Since also CQL uses a Gaussian policy with full support over the action space, do some of the theoretical claims of SVD generalize beyond this specific algorithm?\n- What is the intuition and what are the theoretical consequences of applying SNIS over importance sampling (Section 4.5)?\n- Even if considered a standard benchmark in the offline literature, I generally do not find evaluating on a subset of D4RL very informative in terms of empirically properties. Have the authors considered evaluating on additional/more comprehensive offline benchmarks? (e.g. [1])\n\n[1] Gulcehre, Caglar, et al. \"Rl unplugged: A suite of benchmarks for offline reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020).", "limitations": "The authors quickly dismiss the constraint of having to model the behavior policy by stating that \"we empirically find that the Gaussian model can usually induce excellent performance, which only takes two minutes for pre-training.\" (Lines 337-338) This claim should be contextualized, as a shallow Gaussian policy is most definitely not sufficient beyond the considered toy settings and the relatively simple D4RL benchmark.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The paper provides a concise and comprehensive introduction to the off-policy literature and the utilized notation in Section 2.\n2) The paper correctly recognizes an important issue of existing popular algorithms in the offline literature: while their stated aim is to penalize behavior only outside the support set of the offline datasets, in practice they resort to density-based heuristic penalties.\n3) Overall, the paper is clear and easy to read.\n4) The authors provide some analysis and ablations beyond just reporting their method's performance.", "weaknesses": "1) Generalizing some of the mentioned regularization issues to the whole literature would require very comprehensive evidence which the paper does not provide,  as it mostly focuses on CQL when analyzing prior work. Hence, I would tone down some of the statements, e.g. lines 26-28\"existing value regularization methods not only fall short in penalizing all OOD Q-values but also may introduce detrimental changes to in-distribution (ID) ones\" -> \" some of the most popular existing value...\". Additionally, claims like \"SVR guarantees strict policy improvement until convergence to the optimal support-constrained policy\" (333-334) should specify that this assumes an idealized setting, and does not apply in the stochastic optimization regime of deep learning.\n2) Related to the point above, I believe it would be very appropriate for the paper to mention the assumptions/conditions used to derive the theoretical results in the introduction and abstract to avoid having unbacked claims.\n3) A limitation of the method is that it requires access to the behavior's policy density.  It would be useful to also analyze the properties of the proposed penalization (theoretically and/or empirically) based on the modeling errors for the trained behavior policy proxy.\n4) In practice, CQL's policy is most often also a Gaussian distribution, and, thus, it covers the full support of the action space. Yet, the authors do not seem to consider or mention this property in their analysis. Moreover, since the authors also resolve to using the agent's Gaussian policy with an increased variance for the sampling (action-space covering) penalization distribution, it appears to me that the actual implementation of their algorithm is very similar to existing off-policy methods (e.g., even to CQL).\n5) The toy example in Section 4.1 seems quite misleading. From a practical perspective, it seems to me like CQL and SVR are algorithms with similar properties whose performance depends on hyperparameters.  Hence, I believe that simply showing one works and the other fails with no context (swept hyperparameters/different collected datasets) is not very informative.\n6) While the authors show some extensions of SVR replacing the Gaussian density estimator with a conditional VAE and using self-normalized importance sampling, results appear inconclusive and there is very little analysis done to motivate these findings.\n8) Selection of the distribution to cover the whole action space, clearly influences the penalization magnitude of different actions (Equations 5-6), and, therefore, performance.  There does not appear to be be any clear principle for choosing this important hyper-parameter for a given problem - apart from looking at online performance, which would break the assumptions of a fully-offline setting.\n\nMinor:\nTypos - Line 108: 'in policy improvement stage' -> 'in the policy improvement stage', Line 108: scare -> scarce.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort that you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**[W1] I would tone down some of the statements, e.g. lines 26-28\"existing value regularization methods\" -> \" some of the most popular existing value...\".**\n\nThanks for your constructive comment. We used \"existing value regularization methods\" because we tried our best to summarize the existing value regularization algorithms in the related work section, and to the best of our knowledge, few works achieve the original purpose of value regularization - to penalize all OOD Q-values without affecting ID ones.\n\n**[W1, W2] Mention the assumptions/conditions used to derive the theoretical results in the introduction and abstract.**\n\nThanks a lot for pointing it out. We analyze the policy iteration with the SVR operator in the tabular MDP setting, which is common in the analyses of offline RL algorithms [1,2]. We will make sure to add it in the latter revision.\n\n[1] Seyed Kamyar Seyed Ghasemipour, et al. EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Ofﬂine and Online RL. ICML, 2021.\n\n[2] Jiafei Lyu, et al. Mildly Conservative Q-Learning for Ofﬂine Reinforcement Learning. NeurIPS, 2022.\n\n**[W3] It would be useful to also analyze the properties of the proposed penalization (theoretically and/or empirically) based on the modeling errors for the trained behavior policy proxy.**\n\nThank you for your valuable suggestion. To empirically investigate SVR under different behavioral model errors, we run SVR using different checkpoints of the behavioral model, which are obtained at different steps in the behavioral model training process. The model error is controlled by the number of steps taken to train the behavioral model. The results are shown in Figure 2 in the one-page PDF (attached to the global response). We observe that the performance of SVR increases with the number of training steps of the behavioral model. Notably, the performance of SVR stabilizes at a high level after only 1e2 steps of behavioral model training, where the model has not been adequately trained.\n\nTheoretically, in SVR, the error of the behavioral model $\\beta_\\omega$ only affects the weights of rewarding ID actions (see Eq. 16). Thus, an imperfect model can make the maximization and minimization of ID Q-values not cancel out well, but have little effect on OOD ones (still penalizing all OOD Q-values). We hypothesize that this is the reason why SVR can achieve good performance with an imperfect behavioral model.\n\n**[W4, Q1] Since CQL also uses a Gaussian policy with full support over the action space, do some of the theoretical claims of SVR generalize beyond this specific algorithm?**\n\nIn our analyses of CQL, the policy $\\pi$ can be any distribution, including Gaussian covering the full action space. According to Proposition 3, $\\mathcal{T}^\\pi_{CQL}$ is a contraction operator only if $supp(\\pi) \\subseteq supp(\\beta)$. This is consistent with the analysis in CQL's paper, which also assumes this strong assumption. On the one hand, Gaussian policy covering the full action space does not satisfy this assumption of CQL. On the other hand, even for $\\pi$ satisfying the assumption, the fixed point may underestimate or overestimate Q-values in a complicated way (Proposition 3). Therefore, the theoretical claims of SVR cannot be extended to CQL. In addition, from a practical perspective, the Guassian policy in CQL is usually very narrow and hardly covers the action space.\n\n**[W4] It appears to me that the actual implementation of their algorithm is very similar to existing off-policy methods (e.g., even to CQL).**\n\nYes, our implementation is similar to CQL but with two main differences: much wider penalization for all Q-values and weighted maximization for ID Q-values.\n\n**[W5] In the toy example, I believe that simply showing one works and the other fails with no context (swept hyperparameters/different collected datasets) is not very informative.**\n\nSorry that we miss some experimental details. In the toy example, we were verifying the support-constrained optimality of the SVR operator in the tabular MDP, and the actual updates of CQL and SVR are like Eq.4 and Eq.7 respectively, which are quite different. We ran experiments over 5 random seeds that affect dataset collection and policy/Q initialization, and the learning curves are shown in Figure 2. We also swept the hyperparameter, but the results were similar (SVR converged, CQL did not). We will add the details in the latter revision.\n\n**[W6, Q2] While the authors show some extensions of SVR replacing the Gaussian density estimator with a conditional VAE and using self-normalized importance sampling, results appear inconclusive and there is very little analysis done to motivate these findings. What is the intuition and what are the theoretical consequences of applying SNIS over IS?**\n\nSVR-VAE: We find that the advantages of VAE are not well demonstrated under the common D4RL datasets. Thus, we conduct additional experiments on bimodal datasets, which are constructed by mixing hopper-expert dataset and another dataset collected by a narrow and highly suboptimal Gaussian policy $N(0,0.04)$. In this case, Gaussian can not model the behavior policy well. The results are shown in Figure 3 in the one-page PDF. Benefit from the flexibility of VAE estimator, SVR-VAE obtains better results. \n\nSVR-SNIS: Theoretically, SNIS is biased, but the bias is small, and the improvement in variance makes it a preferred alternative to IS sometimes [3]. The results in Section 4.5 show that SVR-SNIS performs comparably to SVR in most tasks but worse than SVR on hopper-med, probably due to the bias.\n\n[3] Art B. Owen. Monte Carlo theory, methods and examples. 2013.\n\n**Due to the page limit, please refer to the global response block on the top of this page for the remaining responses. Thanks!**"}, {"review_id": "ZD41lFxIwy", "reviewer": "Reviewer_DikP", "summary": "This paper proposes a new offline RL method in which the popular  assumption that the new policy should be close to the behavior poplicy is abandoned and just penalizing the OOD action would be OK. Analysis show that the method has the policy improvement property. Experimental results partially verify the effectiveness of this 'more simple' method (compared to CQL).", "questions": "see above", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The presentatation is clear and the proposed method is easy to understand. \n\nThe final results look good.", "weaknesses": "1. By relaxing the behavior clone requirement (first eq of eq(4)) and only penalizing the OOD action, the new policy would have better chance to find a better one. This is reasonable. However, since the search space becomes larger, the learning process should become more difficult. Unfornuately, the experimental results (Table 1) are eager to show this method is good, without illustrating how the reward is accumulated  as usually done.\n\n2. According to eq.(7), this method would be even more sensitive to the quality of the learnt behavior policy itself than CQL, while the behavior policy  is generlly quite difficult to learn. This may generate extrapolation error. For example, if we utilize a Gaussian approximal behavior policy and the a1, a2 are both the in-sample actions at state s. Then the intropolated OOD a3 between a1 and a2 would likely has a large probability density of the behavior policy, which would introduce extropolation error. Hence the paper should provide more evidence to  support the claim that 'SVR is less susceptible to model errors' .\n\n3. Generalization problem. The proposed SVR could be considered as a clipped Conservative Q-Learning, where the part with ID actions is dropped. Then the Q value generated by this method would have the same generalization problem as CQL. For example, if we have learnt an approximal behavior policy perfectly, then the Q value of any perturbed state-action pair (unseen but closed to the dataset) would drop greatly, although we may need a smoother Q function to enhance the robustness [RORL: Robust Offline Reinforcement Learning via Conservative Smoothing].\n\n4. Lack of the experiments that exploring the behavior of the SVR agent. That is, the paper is supposed to show how the SVR behaves at the ID states and OOD states in practice.\n\n5.  Although the theoretical analysis demonstrates the performance improvement guarantee of SVR, it deviates from the core problem that this paper attempts to solve, where SVR introduces smaller detrimental changes to in-distribution Q-values.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort that you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**[W1] The experimental results (Table 1) are eager to show this method is good, without illustrating how the reward is accumulated as usually done.**\n\nWe included all the learning curves of SVR in Section B of the appendix. Overall, SVR has a stable learning process and good asymptotic performance.\n\n**[W2] According to eq.(7), this method would be even more sensitive to the quality of the learnt behavior policy. It may generate extrapolation error. The paper should provide more evidence to support the claim that 'SVR is less susceptible to model errors'.**\n\nThanks for your suggestion. With an estimated behavior policy $\\hat{\\beta}$, SVR does not distinguish between ID and OOD actions by whether $\\hat{\\beta}>0$ like Eq.7, as the density of OOD actions is particularly difficult to estimate and the behavioral model error can generate extrapolation error in this way as you mentioned. Instead, like Eq.5, SVR penalizes all the actions in action space and compensates for ID actions. When $\\hat{\\beta}$ is estimated accurately, the regularization effects on ID Q-values cancel out, achieving penalization for all OOD Q-values only (Eq.6), leading to the optimal solution Eq.7. However, **with model error of $\\hat{\\beta}$ incorporated, Eq.7 does not become that where $\\beta$ is substituted by $\\hat{\\beta}$**, because in Eq.5, the model error does not affect $\\mathbb{E}_{a \\sim \\beta}$ and only affects the weight of rewarding ID actions (the IS ratio in the last term). As a result, an imperfect model can make the minimization and maximization for ID Q-values not cancel out well, but have little effect on OOD ones (still penalizing all OOD Q-values). **In conclusion, the behavioral model is not used to distinguish between ID and OOD actions in SVR, and the model error can only lead to some unnecessary changes to ID Q-values and will not generate extrapolation error or OOD overestimation.** In addition, compared with other methods that require the behavior model, SVR is less susceptible to model errors because SVR only needs to query the behavior density of in-dataset (s, a) pairs, thus not requiring much generalization ability of the model, making it relatively easier to estimate accurately.\n\nTo empirically investigate SVR under different behavioral model errors, we run SVR using different checkpoints of the behavioral model, which are obtained at different steps in the behavioral model training process. The model error is controlled by the number of steps taken to train the behavioral model. The results are shown in Figure 2 in the one-page PDF (attached to the global response). We observe that the performance of SVR increases with the number of training steps of the behavioral model. Notably, the performance of SVR stabilizes at a high level after only 1e2 steps of behavioral model training, where the model has not been adequately trained. It indicates that SVR can achieve good performance with an imperfect behavioral model.\n\n**[W3] The Q value generated by this method would have the same generalization problem as CQL.**\n\nThanks for your meaningful comment. Recently, some works consider the generalization problem of Q functions, aiming to learn smoother or less conservative Q functions [1,2]. In contrast, SVR aims to solve a different, more fundamental problem. SVR focuses on which action will be penalized, instead of the strength of penalization that these methods consider. In addition, these methods that improve the generalization of Q-networks can also be combined into SVR in practice. For example, substitute $Q_{\\mathrm{min}}$ in Eq.16 with a value that is slightly smaller than in-distribution maximum, as MCQ did [2].\n\n[1] Rui Yang, et al. RORL: Robust Offline Reinforcement Learning via Conservative Smoothing. Advances in Neural Information Processing Systems, 2022.\n\n[2] Jiafei Lyu, et al. Mildly Conservative Q-Learning for Ofﬂine Reinforcement Learning. Advances in Neural Information Processing Systems, 2022.\n\n**[W4] The paper is supposed to show how the SVR behaves at the ID states and OOD states in practice.**\n\nThanks for your meaningful comment. The main purpose of this work is to punish OOD actions, which causes value over-estimation. Compared to OOD actions, OOD states cause less severe problems (mainly the state deviation issue during test time [3]), and much fewer works focus on it. Since the contribution of SVR does not lie in handling OOD states, we only investigate the behavior of SVR with respect to ID/OOD actions. Figure 1 in the paper shows that SVR not only avoids taking OOD actions, but also chooses the optimal ID actions.\n\n[3] Hongchang Zhang, et al. State Deviation Correction for Offline Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2022.\n\n**[W5] Although the theoretical analysis demonstrates the performance improvement guarantee of SVR, it deviates from the core problem, where SVR introduces smaller detrimental changes to in-distribution Q-values.**\n\nWe would like to make an emphasis on Theorem 1. According to it, the policy iteration with SVR operator outputs unbiased Q-values for all ID actions and underestimated Q-values for all OOD actions. Benefit from this property, SVR can achieve performance improvement and optimal convergence guarantee."}, {"review_id": "T3B47YhwX9", "reviewer": "Reviewer_c3nj", "summary": "This paper proposes the use of Importance sampling to distinguish between ID and OOD actions, and to operate the corresponding actions accordingly. In addition, as most of the current work is similar, it also uses model to fit behavior policy, but does not need to overly consider the accuracy of the model. It underestimates OOD actions and unbiased estimates ID actions, achieving good performance on D4RL and Adroit tasks.", "questions": "1. Recently, there has been a way to use Importance sampling or Importance resampling to determine whether it belongs to OOD actions. Please explain the relationship between your work and the current works, or explain the differences and advantages of your method.\n2. In Figure 2, I cannot see that there has a serious overestimation problem with the value function caused by the iteration of the vanilla policy.\n3. Please provide a more detailed explanation of Figure 1, such as what the colors of the grid represent.\n4. Why does the Walker2d-v2-random in Figure 1 in appendix only run a small portion and not continue to run?\n", "limitations": "There is no negative societal impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper conducts relatively sufficient experiments to demonstrate its viewpoints.\n2. The proofs of theorems and propositions are relatively sufficient.\n3. The hyperparameters and parameters settings in the paper are clearly defined for easy reproduction.\n", "weaknesses": "1. Importance sampling is widely used in reinforcement learning because of its unbiased nature. In this paper, the main contribution is to use importance sampling to punish the deviation of OOD action and ID action respectively. The differences or advantages between the proposed SVR method and existing methods should be described clearly.\n2. The description of certain experiments should be clearer. For example, what do the colors of the grid represents in Figure 1? In Figures 1 and 2, it is hard to see how the value function described in the paper is represented.\n3. The algorithms in the comparative experiment are only some classic algorithms, lacking some algorithms of the same type, such as using importance resampling (Zhang et al. 2023), and there is no mention of the ablation experiments about the penalty coefficient α.\n(Zhang et al. 2023) Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, Xiangyang Ji. In-sample Actor Critic for Offline Reinforcement Learning. International Conference on Learning Representations, 2023.\nhttps://openreview.net/forum?id=dfDv0WU853R\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort that you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**[W1, Q1] Importance sampling is widely used in reinforcement learning because of its unbiased nature. The differences or advantages between the proposed SVR method and existing methods should be described clearly. Recently, there has been a way to use Importance sampling or Importance resampling to determine whether it belongs to OOD actions.**\n\nThanks for your suggestion. Importance sampling's application in RL has a long history. Most of them utilize some form of importance sampling to either evaluate the return of a given policy, or to estimate the corresponding policy gradient [1,2]. However, they can have very high variance due to the product of importance weights. In contrast, SVR focuses on value regularization in offline RL and involves only one importance weight. Recently in offline RL, some works use importance sampling (or importance resampling) to realize in-sample learning [3,4]. They formulate the Bellman target with the actions in the dataset (SARSA update), and weigh the update by an importance ratio. Thus, they do not differentiate between ID and OOD actions or perform regularization on Q functions. One limitation of these methods is that, they only deal with in-sample actions (actions in the dataset) and can not take advantage of the generalization ability of neural networks. The goal of SVR is quite different from these works - to penalize all OOD Q-values without affecting ID ones. To the best of our knowledge, our work is the first to leverage importance sampling to achieve proper value regularization in offline RL.\n\n**[W2, Q3] What do the colors of the grid represent in Figure 1? In Figures 1 and 2, it is hard to see how the value function described in the paper is represented.**\n\nSorry for the unclear description. The colors of the grid in Figure 1 represent the value functions of the corresponding states, and their specific values are indicated by the color bar on the right side of Figure 1. In addition, the gray/white rectangle area in the middle of the map represents a wall.\n\n**[W3] Lack some algorithms of the same type in the comparative experiment, such as using importance resampling [4], and there is no mention of the ablation experiments about the penalty coefficient $\\alpha$.**\n\nThanks lot for your suggestions. We supplement the results of IAC [4] in Table 1 in the one-page PDF (attached to the global response) for comparison. It shows that SVR has better performance than IAC on both Gym-MuJoCo and Adroit tasks. In addition, we also conduct an ablation study on the penalty coefficient $\\alpha$ in Figure 1 in the PDF. Experimental results indicate that choosing  $\\alpha \\in [1e-3, 1e-2]$ generally induces good performance.\n\n**[Q2] In Figure 2, I cannot see that there is a serious overestimation problem with the value function caused by the iteration of the vanilla policy.**\n\nSorry for the unclear description. Figure 2 presents the Episode Return obtained by evaluating the learned policy at each iteration, which represents the actual performance of learned policy, rather than learned value function. As mentioned in [W2, Q3], the colors of the grid in Figure 1 represent the learned value functions. It shows that vanilla policy iteration has a severe overestimation of value functions.\n\n**[Q4] Why does the Walker2d-v2-random in Figure 1 in appendix only run a small portion and not continue to run?**\n\nThis is because a severe overestimation occurred and we stopped running automatically. Walker2d-random has extremely narrow data coverage and most of the existing algorithms suffer from severe over-estimation, resulting in poor performance (see Table 1 in the paper). While choosing a large penalty coefficient $\\alpha$ in SVR can mitigate the over-estimation in Walker2d-random, we don't set a particular hyperparameter for it, as hyperparameter tuning in offline RL should be avoided as much as possible.\n\n[1] Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, 2000.\n\n[2] Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. International Conference on Machine Learning,  2016.\n\n[3] Yiqin Yang, et al. Believe what you see: Implicit constraint approach for ofﬂine multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 2021.\n\n[4] Hongchang Zhang, et al. In-sample Actor Critic for Offline Reinforcement Learning. International Conference on Learning Representations, 2022."}, {"review_id": "KlWCbL2dQ3", "reviewer": "Reviewer_2cRJ", "summary": "This paper studies the offline RL problem, and the authors proposes adding the Support Value  Regularization (SVR) in learning Q functions, motivated from the way how CQL add value regularizations. The authors add SVR for all OOD while maintain Bellman update for ID samples. Experiments shows that the SVR-regularized method could outperforms other competitors in D4RL tasks. ", "questions": "1. Some notations are not clear. In addition, what is $\\beta$ in Equation 3? It also does not appear before, though we know it is behavior policy. It is suggested to give an explanation. \n\n2. A key component in this paper is to estimate the support values. The state-action pairs are regarded as ID when $\\beta(a|s) >0$, however, when using the Gaussian model to estimate behavior policy, all actions has a density larger than 0 (since gaussian density are larger than 0 everywhere), but clearly not everywhere are in distribution. \n\n3. In addition, when estimating behavior policy $\\beta_w$, the simple Gaussian model is suggested. However, as the dimension of state could be large, and the entire Support of the behavior policy relies on the behavior density estimation. A simple gaussian model may not be able to deliver a strong estimator.  In section 4.5, the authors give a simple comparison between SVR and SVR-VAE, we suggest to give more detailed analysis. \n\n4. The sampling distribution $u(a,s)$ is selected as gaussian in Section 3.4 with mean $\\pi$. Could the authors give extra abalation study on how $u(a,s)$ affects the results? Selecting $u(a,s)$ as gaussian is generally optimal/close-optimal?", "limitations": "See weakness and questions", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposes Support Value Regularization (SVR) on solving offline RL problems. The authors suggest adding extra regularization on OOD points, and utilize importance sampling techniques to calculate the Q-values in OOD region.\n2. The superiority of SVR method is demonstrated from experiment studies. \n3. Theoretical proof shows the policy improvement could lead to a better policy in each iteration. ", "weaknesses": "Since the method add regularization to OOD regions, it should belongs to Conservative offline RL methods. It is suggested to compare this methods with other less/mild conservative method. In addition, as the SVR is density-based regularization, the authors is suggested to compare with other density-based Offline RL methods. Some less/mild conservative or density-based offline RL methods are listed below, the authors cites some of them, and it seems better to compared some of them.  \n\n[1] Supported Policy Optimization for Offline Reinforcement Learning.  https://arxiv.org/abs/2202.06239\n[2] Mildly Conservative Q-Learning for Ofﬂine Reinforcement Learning.  https://arxiv.org/abs/2206.04745\n[3] Provably Good Batch Reinforcement Learning Without Great Exploration. https://arxiv.org/abs/2007.08202 \n[4] A Behavior Regularized Implicit Policy for Offline Reinforcement Learning. https://arxiv.org/pdf/2202.09673.pdf\n[5] APAC: Authorized Probability-controlled Actor-Critic For Offline Reinforcement Learning. https://arxiv.org/abs/2301.12130", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort that you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**[W] It is suggested to compare with other less/mild conservative method. In addition, it is suggested to compare with other density-based offline RL methods.** \n\nThanks a lot for the references. We supplement the results of these methods in Table 1 in the one-page PDF (attached to the global response). As the results show, SVR has better overall performance than these methods on both Gym-MuJoCo and Adroit tasks. Compared with less/mild conservative methods, SVR aims to solve a different, more fundamental problem. SVR focuses on which action will be penalized, instead of the strength of penalization that less/mild conservative methods considered.\n\n**[Q1] Some notations are not clear, e.g.,  $\\beta$ in Equation 3.**\n\nSorry for that. Before Equation 3, $\\beta$ appears in Line 75 \"a fixed dataset $\\mathcal{D}$ collected by some behavior policy $\\beta$\". We will make it clearer in the latter revision.\n\n**[Q2,Q3] The state-action pairs are regarded as ID when $\\beta(a|s)>0$. However, when using the Gaussian model to estimate behavior policy, all actions have a density larger than 0, but clearly not everywhere is in distribution. As the dimension of state could be large, and the entire support of the behavior policy relies on the behavior density estimation. A simple Gaussian model may not be able to deliver a strong estimator.**\n\nThanks for your comment. The definition of ID state-action pairs is $\\beta(a|s)>0$ where $\\beta$ is the true behavior policy. However, with an estimated behavior policy $\\hat{\\beta}$, SVR does not distinguish between ID and OOD actions by whether $\\hat{\\beta}>0$, as the density of OOD actions is particularly difficult to estimate and the behavioral model error can cause extrapolation error and overestimation in this way. Instead, like Eq.5, SVR penalizes all the actions in action space and compensates for ID actions. When $\\hat{\\beta}$ is estimated accurately, the regularization effects on ID Q-values cancel out, achieving penalization for all OOD Q-values only (Eq.6). **With model error of $\\hat{\\beta}$ incorporated, $\\mathbb{E}_{a \\sim \\beta}$ in Eq.5 is not affected and the model error only affects the weight of rewarding true ID actions (the IS ratio in the last term)**. As a result, an imperfect model can make the minimization and maximization for ID Q-values not cancel out well, but have little effect on OOD ones (still penalizing all OOD Q-values). **In conclusion, the behavioral model is not used to distinguish between ID and OOD actions in SVR, and the model error can only lead to some unnecessary changes to ID Q-values and will not generate extrapolation error or OOD overestimation.** In addition, compared with other methods that require the behavior model, SVR is less susceptible to model errors because SVR only needs to query the behavior density of in-dataset (s, a) pairs, thus not requiring much generalization ability of the model, making it relatively easier to estimate accurately.\n\nTo empirically investigate SVR under different behavioral model errors, we run SVR using different checkpoints of the behavioral model, which are obtained at different steps in the behavioral model training process. The model error is controlled by the number of steps taken to train the behavioral model. The results are shown in Figure 2 in the one-page PDF. We observe that the performance of SVR increases with the number of training steps of the behavioral model. Notably, the performance of SVR stabilizes at a high level after only 1e2 steps of behavioral model training, where the model has not been adequately trained. It indicates that SVR can achieve good performance with an imperfect behavioral model.\n\n\n**[Q3] In section 4.5, the authors give a simple comparison between SVR and SVR-VAE, we suggest to give more detailed analysis.**\n\nThanks for your suggestion. In Section 4.5, we found that the advantages of the VAE estimator are not well demonstrated under the common D4RL datasets. Thus, we conduct additional experiments on bimodal datasets, which are constructed by mixing hopper-expert dataset and another dataset collected by a narrow and highly suboptimal Gaussian policy $N(0,0.04)$. In this case, Gaussian can not model the behavior policy well. The results are shown in Figure 3 in the one-page PDF. Benefit from the flexibility of VAE estimator, SVR-VAE obtains better results.\n\n\n**[Q4] The sampling distribution $u(a|s)$ is selected as gaussian in Section 3.4 with mean $\\pi$. Could the authors give extra abalation study on how $u(a|s)$ affects the results? Selecting $u(a|s)$ as gaussian is generally optimal/close-optimal?**\n\nThanks for your comment. We conducted this ablation in Section 4.4 of the paper, including Guassian with various variance and the uniform distribution. It is shown that SVR is insensitive to $u$ in a wide range. Theoretically, a choice of $u$ is optimal as long as it covers the entire action space. Empirically, Guassian with moderate variance satisfies this requirement and can emphasize the areas where overestimation is most likely to occur (as indicated by the current policy). We only considered Gaussian and uniform distributions as they are the most common samplable continuous distributions. While it is possible to design more complex $u$ (like based on uncertainty estimation), it can be prohibitively difficult or expensive to sample from such complex distributions."}]

Author rebuttal:
### **Global Response**

We thank all the reviewers for the insightful comments and suggestions. We are greatly encouraged by the positive comments of reviewers, e.g., 

* The paper correctly recognizes an important issue of existing popular algorithms in the offline literature. (CX3L)
* The presentation is clear and the proposed method is easy to understand. (DikP)
* The proofs of theorems and propositions are relatively sufficient. (c3nj)
* The superiority of SVR method is demonstrated from experiment studies. (2cRj)

Meanwhile, we have made every effort to address all the reviewers' concerns and responded to the individual reviews below. We have also uploaded a **one-page PDF** (attached to this response) that contains the additional experiment results. Summary of the PDF:

* Comparisons with additional baselines on the D4RL benchmark in Table 1.
* Experimental results on a subset of the RL unplugged benchmark in Table 2.
* Ablation results on the penalty coefficient $\alpha$ of SVR in Figure 1.
* Experimental results of SVR under different behavioral model errors in Figure 2.
* Comparisons between SVR-VAE and SVR-Gaussian on the bimodal datasets in Figure 3.

We hope our response could address the reviewers' concerns. We would be more than happy to resolve any remaining questions in the time we have, and are looking forward to engaging in a discussion.

---

### **Additional Response to Reviewer CX3L (Part 2 of 2)**

**[W7] There does not appear to be any clear principle for choosing this important hyper-parameter (the sampling distribution $u$) for a given problem - apart from looking at online performance.**

We conducted a parameter study on the sampling distribution $u$ in Section 4.4. As shown in Figure 4, SVR is insensitive to $u$ in a wide range. We also propose an intuitive approach for choosing $u$ - Gaussian with the same mean as the current policy to emphasize the areas where overestimation is most likely to occur and with a moderate variance to cover the entire action space.

**[Q3] Have the authors considered evaluating on additional/more comprehensive offline benchmarks? (e.g. Rl unplugged)**

Thank you for your suggestion. The experimental results on a subset of the RL unplugged benchmark are shown in Table 2 in the one-page PDF. We observe that SVR performs better than baseline methods on three tasks and slightly worse on one task.

**[L1] (Lines 337-338) This claim should be contextualized, as a shallow Gaussian policy is most definitely not sufficient beyond the considered toy settings and the relatively simple D4RL benchmark.**

We apologize for the unclear statement. While the Gaussian behavioral model performs well empirically in D4RL, it can be intuitively problematic when dealing with datasets with complex distributions. As mentioned in [W6, Q2], we conduct additional experiments on the bimodal datasets, where SVR-VAE outperforms SVR due to the flexibility of the VAE estimator.

Meta-review: I went through the reviews and discussion carefully, as well as through the paper. Although this work builds on CQL, it is a fundamentally distinct and reasonably novel approach. I thought the authors' analysis in sections 3.1 and 3.2 motivated things very well, and to me it seems like an interesting enough idea that others can build on.

The fact that this is complemented with theoretical proofs of convergence and improvement, as well as strong empirical performance, leads me to recommend acceptance of this paper.
Decision: Accept (poster)