Conservative State Value Estimation for Offline Reinforcement Learning
Authors: Liting Chen
Jie Yan
Zhengdao Shao
Lu Wang
Qingwei Lin
Saravan Rajmohan
Thomas Moscibroda
Dongmei Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Offline Reinforcement Learning

Abstract: Offline reinforcement learning faces a significant challenge of value over-estimation due to the distributional drift between the dataset and the current learned policy, leading to learning failure in practice. The common approach is to incorporate a penalty term to reward or value estimation in the Bellman iterations. Meanwhile, to avoid extrapolation on out-of-distribution (OOD) states and actions, existing methods focus on conservative Q-function estimation. In this paper, we propose Conservative State Value Estimation (CSVE), a new approach that learns conservative V-function via directly imposing penalty on OOD states. Compared to prior work, CSVE allows more effective state value estimation with conservative guarantees and further better policy optimization. Further, we apply CSVE and develop a practical actor-critic algorithm in which the critic does the conservative value estimation by additionally sampling and penalizing the states around the dataset, and the actor applies advantage weighted updates extended with state exploration to improve the policy. We evaluate in classic continual control tasks of D4RL, showing that our method performs better than the conservative Q-function learning methods and is strongly competitive among recent SOTA methods.

Conservative State Value Estimation for Offline Reinforcement Learning
Liting Chen" McGill University Montreal, Canada
Jie Yan f Microsoft Beijing, China
98chenl ting@gmail.com
dasisttyajjiedgmaal.co
Zhengdao Shao"
Lu Wang
Qingwei Lin
University Sci. and Tech. of China Hefei, China
Microsoft
Microsoft
Beijing, China vlu@microsoff com
Beijing, China q wwiiiiiicosoot.commmmm
zhengdaoshao@maill eestc.edu... en
Saravan Rajmohan
Thomas Moscibroda
Dongmei Zhang Microsoft
Microsoft 365 Seattle, USA
Microsoft Redmond, USA
Beijing, China dongmeiz@miccosoofe com
aaravar@microsoff com
mosci moscitho@@miccoooft.com
Abstract
Offline reinforcement learning faces a significantecaaleengee of value over-estimation due to the distributional drift between the dataset and the current learned policy, leading to learning failure in practice. The common approach is to incorporate a reward value estimation in the Bellman iterations. Meanwhile, to avoid extrapolation on methods focus conservative Q-function estimation. In this paper, we propose Conservative State Value Estimation (CSVE), a new approach that learns con- servative V-function via directly imposing penalty on OOD states. Compared prior work, CSVE allows more effective state value estimation with conservative guarantees and further better policy optimization. Further, we apply CSVE and develop practical actor-critic algorithm in which the critic does the conservative value estimation by additionally sampling and penalizing the states around the dataset, and the actor applies advantage weighted updates extended with state exploration to improve the policy. We evaluate in classic continual control tasks D4RL, showing that our method performs better than the conservative Q-function learning methods and strongly competitive among recent SOTA methods.
penalty
(OOD) states and actions, existing
Introduction
Reinforcement Learning (RL) learns act by interacting with the environment and has shown great success in various tasks. However, in many real-world situations, is impossible to learn from scratch online exploration is often risky and unsafe. Instead, offline (L((2) avoids this problem by learning the policy solely from historical data. Yet, simply applying standard online RL techniques static datasets can lead to overestimated values and incorrect policy decisions when faced with unfamiliar or out-of- -it-o---iitriuttion (OOD) scenarios.
* Work done during the internship at Microsoft. Work done during full-time employment at Microsoft.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Recently, the principle of conservative value estimation has been introduced to tackle challenges in offline R [. Prior methods, c.g., CQL(Conservative Q-Learnng 4), avoid the value over-estimation problem by systematically underestimating the Q values of OOD actions on the states in the dataset. In practice, it is often too pessimistic and thus leads to overly conservative algorithms. COMBO [6] leverages a learned dynamic model to augment data in an interpolation way. This process helps derive a Q function that's ' less conservative than CQL, potentially leading to more optimal policies.
In this paper, we propose CSVE (Conservative State Value Estimation), a novel offline RL approach. Unlike the above methods that estimate conservative values by penalizing the Q-function for OOD actions, CSVE directly penalizes the V-function for OOD states. We theoretically demonstrate that CSVE provides tighter bounds on in-distribution state values in expectation than CQL, and same bounds as COMBO but under more general discounted state distributions, which potentially enhances policy optimization in the data support. Our main contributions include:
The conservative state value estimation with related theoretical analysis. We prove that it lower bounds the real state values in expectation over any state distribution that is used to sample OOD states and is up-bounded by the real state values in expectation over the marginal state distribution of the dataset plus a constant term depending on sampling errors. Compared to prior work, it enhances policy optimization with conservative value guarantees.
A practical actor-critic algorithm implemented CSVE. The critic undertakes conservative state value estimation, while the actor uses advantage- advantage-weighted weighted regression(AWR) and explores states with conservative value guarantee to improve policy. In particular, use a dynamics model to sample OOD states that are directly reachable from the dataset, for efficient value penalizing and policy exploring.
Experimental evaluation on continuous control tasks of Gym 7 and Adroit [ in D4RL 9 benchmarks, showing that CSVE performs better than prior methods based on conservative Q-value estimation, and is strongly competitive among main SOTA algorithms.
2 Preliminaries
Offline Reinforcement Learning, Consider the Markov Decision Process M == (S, A, p. 7). which comprises the state space S, the action space A, the transition model P x A (S)). the reward function r S x A + R, the initial state distribution & and the discount factor ? E (0. 1). A stochastic policy π S A selects an action probabilistically based the current state. A transition is the tuple (st, at, Tt, 1t+1) where ag ~ T (.1(st), 8441 ~ (( at), and r. = r(5,, az). It's assumed that the reward values adhere to |r(s, a)) Rmary Vs, a. A trajectory under is the random sequence T (80, Op, ro, 81, (1, r1,,,,,...... 8T) which consists of continuous transitions starting from 80 p.
Standard RL is to learn policy E IT that maximize the expected cumulative future rewards, repre- sented as (M) = EM 1Eioo through active interaction with the environment M. At any time I, for the policy t. the value function state is defined as V (=) = EM EΣ 7 = s). and the Q value function is QT a) = EM, [E= + / = s, a = a). The Bellman operator is function projection: B (s, a) := r( + [gg_ ((('' or (s) Ee-x((ie([r(s, a) + E.' (s']]], resulting initerative value updates. Bellman consistency implies that V =
Vs and (s) = E "" a), Vs. a. When employing function approximation in practice, the empirical Bellman operator 1TTh is used, wherein the afore- mentioned expectations are estimated with data. Offline RL aims to learn the policy и from a static dataset D= = {(s, a, r, 8')} made up of transitions collected by any behavior policy, with the objective of performing well in the online setting. Note that, unlike standard online RL, offline RL does not interact with the environment during the learning process.
Conservative Value Estimation. One main challenge in offline RL arises from the over-estimation of values due to extrapolation in unseen states and actions. Such overestimation can lead to the deterioration of the learned policy. To address this issue, conservatism or pessimism employed in value estimation. For instance, CQL learns conservative Q-value function by penalizing the value

---

of unseen actions:
@h+l arg min E. ( (s) anD [Q(s.a)]-B SnD
[Q(s, a)l)
Q
amu(-1s))
a>
where सt and т are the behaviour policy and learnt policy separately, l is an arbitrary policy different from Rg. and a represents the factor for balancing conservatism.
Constrained Policy Optimization. To address the issues of distribution shift between the learning policy and the behavior policy, one approach is to constrain the learning policy close to the behavior policy [10 11 12 13 I As an example, Advantage Weighted Regression(AWRR) 14 12 employs
an implicit KL divergence to regulate the distance between policies:
llog (als) exp (s.(a) ((s)
π h+1 + arg max
E 8,00
Here, A= is the advantage of policy n², and Z serves the normalization constant for
Model-based Offline RL. In RL, the model an approximation of the MDP M. Such model denoted as M (S, A, P, f. p. 7), with P and being approximation of and r respectively. Within offline RL, the model is commomly used to augment data [[5] 6 or act as a surrogate of the real environment during interaction [66 However, such practices can inadvertently introduce bootstrapped errors over extended horizons 17.. In this paper, we restrict the use of the model to one-step sampling on the next states that are approximately reachable from the dataset.
3 Conservative State Value Estimation
In the offline setting, the value overestimation is major problem resulting in failure of learning reasonable policy 13 [. In contrast to prior works[4] 6 that get conservative value estimation via penalizing Q function for OOD state-action pairs, directly penalize V function for OOD states. Our approach provides several novel theoretic results that allow better trade-off of conservative value estimation and policy improvement. All proofs of our theorems can be found in Appendix A
3.1 Conservative Off-policy Evaluation
We aim to conservatively estimate the value of a target policy using dataset to avoid overestimation of OOD states. To achieve this, we penalize V-values evaluated on states that are more likely to be OOD and increase the V-values on states that in the distribution of the dataset. This adjustment made iteratively::
prk+1 + aig E. (o) V(s)) a ((B V(8) EE-d V(s))
where (s) is the discounted state distribution of D. d(s) is any state distribution, and B is the empirical Bellman operator (see appendix for more details). Considering the setting without function approximation. by setting the derivative of Eq, as zero, we can derive the V function using approximate dynamic programming at iteration k::
al d(o) (s)
(x+ = B (*)
1], Vs
Denote the function projection on V/ in Eq. 3 as T.. We have Lemma 3.1 which ensures that V converges to a unique fixed point.
Lemma 3.1. For any d with supp C supp du. T is a c-contraction in Loo norm.
Theorem 3.2. For any d with suppd C supp d,, (d # d.), with a sufficiently large a (i.e., o 2
d(s) - 1]) J, the expected value of the estimation v*((s) Eg-d(s) Eara(al)) e) (U-7)//D(<saa) I d_(e)
Cra,sRmass
under d(s) is the lower bound of the true value, that is: Eend(s) [V= (((((())) (s)] S Ee-d()) (s)).
3

---

(=())= = lim (s) is the converged value estimation with the dataset D, and Cr. Rmur /I-2)/D((s is related to sampling error that arises when using the empirical operator instead of the Bellman operator: If the counts of each state-action pair is greater than zero, [8 a)l denotes a vector of size ||| containing counts for each state-action pair. If the counts of this state action pair is zero, the corresponding
is a large yet finite value. We assume that with probability 2 1 6, the
VID(s,a)]
sampling error is less than &mar while Cr,t,5 is a constant (See appendix for more details.)
(1-7)//P(e.a))
Note that if the sampling error be disregarded, o: 0 can ensure the lower bound results.
Theorem 3.3. The expected value of the estimation, (*),,, under the state distribution of the original dataset is the lower bound of the true value plus the term of irreducible sampling error. Formally:
Cr.. & &aaar
Esndu(s) [*())] S (gndd (=) (8)] + Egndu(s) -p=) (awx(als))
(1-1)
where P refers to the transition matrix coupled with policy æ (see Appendix for details).
Now we show that, during iterations, the gap between the estimated V-function values of in- distribution states and OOD states is higher compared to the true V-functions.
Theorem 3.4. For any iteration k, given a sufficiently large a, our method amplifies the difference in expected V-values between the selected state distribution and the dataset state distribution. This can be represented as: Ex-d,(s)] (V>
[s)] Ex-d(s) (V* [s)] > (s)d_(s) [V* (s)] Es-d(s) (s)).
Our approach, which penalizes the V-function for OOD states, promotes a more conservative estimate of a target policy's ss value in offline reinforcement learning. Consequently, our policy extraction ensures actions align with the dataset's distribution.
To apply our approach effectively in offline RL algorithms, the preceding theorems serve as guiding principles. Here are four key insights for practical use of Eq. 2
Remark 1. According to Eq. 2, d = du. the penalty for OOD states diminishes. This means that the policy will likely avoid states with limited data support, preventing it from exploring unseen actions in such states. While AWAC [[2]employs this configuration, our findings indicate that by selecting d, our method surpasses AWAC's performance.
Remark 2. Theorem 3.3 suggests that under d,,, the marginal state distribution of data, the expectation estimated value of V is either lower than its true value or exceed it, but within a certain limit. This understanding drives our adoption the iavantage-weighter policy update, as illustrated Eq. 9
Remark 3. As per Theorem 3.2 the expected estimated value of a policy under d, which represents the discounted state distribution of any policy, must be a lower bound of its true value. Grounded in this theorem, our policy enhancement strategy merges an advantage -weigtee update with an additional exploration bonus, showcased in Eq. 10
Remark 4. Theorem 3.4 states [ (V*()) - Egndie)(V
Rend (v)
Ex=d.[s)] [s)). In simpler terms, the underestimation of value is more pronounced under d. With the proper choice of d, we can confidently formulate a newer and potentially superior policy using f' Our algorithm chooses the distribution of model predictive next-states as d, i.e., s' ~ d is implemented by D.a T(-1s), P(-ls, which effectively builds a soft 'river' with low values encireling the dataset.
Comparison with prior work: CQL [Eq which penalizes Q-function of OOD actions, guarantees the lower bounds on state-wise value estimation: V**(s) = Ex(o)s) (Q= a)) < (s,
Priclei(Q** (s, a)) = (*()) for all E D. COMBO, which penalizes the Q-function for OOD states and actions of interpolation of history data and model-based roll-outs, guarantees the lower bound of state value expectation: Eg [ [s) < Eg [s](s) where Po is the initial state dis- tribution (Remark 1, section A.2 of COMBO [D:; which is a special case of our result in Theorem 3.2 when d = Mg- Both CSVE and COMBO intend enhance performance by transitioning from individual state values to expected state values. However, CSVE offers the same lower bounds but under a more general state distribution. Note that μ40 depends on the environment or the dynamic model during offline training. CSVE's flexibility, represented by d, ensures conservative guarantees across any discounted state distribution of the learned policy, emphasizing a preference for penalizing V over the f-fuctiion.

---

3.2 Safe Policy Improvement Guarantees
Now we show that our method has the safe policy improvement guarantees against the data-implied behaviour policy. We first show that our method optimizes a penalized RL empirical objective:
Theorem 3.5. Let v be the fixed point of Eq. 3, then π (als) arg m V*(s) is equivalently obtained by solving:
d(s)
π f- arg max [ M)
E a (s)
17.
Building upon Theorem 3.5 we show that our method provides a Ç-safe policy improvement over z-- Theorem 3.6. Let x² (als) be the policy obtained in Eq. Then, it is Ç-safe policy improvement over 8 in the actual MDP M, ie., [(T", M) 2 J ( M) S with high probability 1- 6, where is given by:
( 2 1-7 + RRmaoOra
Crd
T(als) T3 (a(s))
(
(1-7)2
and (4)
ami((as))
(J(T", ,M) - [ M)) where c= v [
2 B, Brndyy (o)) (at -1]
Methodology
In this section, propose a practical actor-critic algorithm that employs CSVE for value estimation and extends Advantage Weighted Regression [1]] with out-of-sample state exploration for policy improvement. In particular, adopt dynamics model to sample OOD states during conservative value estimation and exploration during policy improvement. The implementation details are in Appendix B Besides, we discuss the general technical choices of applying CSVE into algorithms.
[18]
4.1 Conservative Value Estimation
Given a dataset D acquired by the behavior policy TH our objective is to estimate the value function V for target policy т. As stated in section 3 to prevent the value overestimation, we learn conservative value function v/ that lower bounds the real values of x by adding penalty for OOD states within the Bellman projection sequence. Our method involves itcrative updates of Equations 5
where Q* is the target network of Q*
p++t1 arg argmin min (V; Q*)
(5)
v
v()))p] (Banni--pp (-++) (s) -V())) + (RonD - BBaD(V(s))
= BDND
d'aP(a,a)
@u+l +- arg min L3 (Q; (Q:
E (r(s,e) + + ((s)))
(6)
Q
8,0:8'DD
gu+I (1 wt + wQuh1
(7)
The RHS of ijs an approximation of Eq. with the first term representing the standard TD error. In this term, the target state value is estimated by taking the expectation of Q* over π and the second term penalizes the value of OOD states. Eq. 6 the RHS is TD errors estimated on transitions in the dataset D. Note that the target term is the sum of the reward r(s, a) and the next step state's 's value (s'). In Eq. 7 the target Q values are updated with soft interpolation factor (0,1). 1). @ changes slower than Q which makes the TD emror estimation in Eq. more stable. Constrained Policy. Note that in RHS of Eq. 5 we
T(-ss) in expectation. To safely estimate the target value of (s) by E Ea (s, a)}, we almost always requires supp(mT-Is)) c
5

---

supp(TTa(1ss) We achieve this by the advantage weighted policy update, which forces T-|ss) to have significant probability mass on actions taken by ng in data, as detailed in section 3.2
Model-based OOD State Sampling. In Eq. 5 we implement the state sampling process s' ~ d in Eq. 2 as a flow of {s ~ D:a ~ T(aas), s' ~ P(s'Is,a) that is the distribution of the predictive next-states from D by following n. This approach proves beneficial in practice. On the one hand, this method is more efficient as it samples only the states that are approximately reachable from D by one step, rather than sampling the entire state space. On the other hand, we only need the model to do one-step prediction such that it introduces no bootstrapped errors from long horizons. Following previous work [7 15] o We use an ensemble of deep neural networks, represented as p0! ..... p0 to implement the probabilistic dynamics model. Each neural network produces Gaussian distribution over the next state and reward: (s 1" (t; at) (u,((s, ae), (8(,, at)).
Adaptive Penalty Factor a. The pessimism level is controlled by the parameter a 2 0. In practice, we set adaptive during training as follows, which is similar to that in (
max ~ -alaaaalV Vo((s)) (V 7)],
(8)
a2u
where 7 is budget parameter. If the expected difference in V-values is less than T, will decrease. Otherwise, a will increase, penalizing the OOD state values more aggressively.
4.2 Advantage Weighted Policy Update
After learning the conservative ÿa++1 and Q++1 (or oi and Q when the values have converged), we improve the policy by the following advantage weighted update [12].
[og n' (a)s))) exp (s, m)]]
< arg min (=(=)======)))) -- E
(9)
x/
8,0~D
where A a) Q as.a) (s). Eq 9 updates the policy π by applying weighted maximum likelihood method. This is computed by re-weighting state-action samples in D using the estimated advantage A avoids explicit estimation of the behavior policy, and its resulting sampling errors, which is an important issue in offline RL 112
Implicit policy constraints. We adopt the advantage-weightee policy update which imposes an implicit KL divergence constraint between π and TT. This policy constraint is necessary to guarantee that the next state s' in Eq. can be safely generated through policy x. As derived in [12] (Appendix A), Eq. gis parametric solution of the following problem (where depends on B):
max E (1,)) ( (8, a)]
(s(+s)) se, J. (a(ass)da= = 1.
s.t. DKL (-1s) (T's5))))))
Note that DKL (Tr' || TH) is a reserved KL divergence with respect to n', which is mode-seeking 19 When treated as Lagrangian it forces n' to allocate its probability mass the maximum likelihood supports of Ta. re-weighted by the estimated advantage. In other words, for the space of A where (-(- has no samples, n' (-Is) has almost zero probability mass too.
Model-based Exploration on Near States. As suggested by remarks in Section 3.I in practice, allowing the policy to explore the predicted next states transition (s ~ D) following leads to better test performance. With this kind of exploration, the policy is updated as follows:
(+1+ss)
<- arg min (T') BB (.aT s'mP(s))))
(s(s.a) + (Dk+ (v))
(10)
x²
The second term is an approximation to Exsd, (s) (s)). The optimization of this term involves calculating the gradient through the learned dynamics model. This is achieved by employing analytic gradients through the learned dynamics maximize the value estimates. It is important to note that the value estimates rely on the reward and value predictions, which are dependent on the imagined states and actions. As all these steps are implemented using neural networks, the gradient is analytically computed using stochastic propppropag,tion, a concept inspired by Dreamer (0) We adjust the value of X, a syper-parameter, to balance between optimistic policy optimization (in maximizing V) and the constrained policy update (as indicated by the first term).

---

Table 1: Performance comparison on Gym control tasks v2. The results of CSVE are over ten seeds and we reimplement AWAC using d3rlpy. Results of IQL, TD3-BC, and PBRL are from their original papers ( (TTale I in 221, Table C.3 in [22), and Table 1 in [00 respectively). Results of COMBO and CQL are from the reproduction results in 23
(Table 1) and [10 respectively, since their original
results were reported on vO datasets.
AWAC
CQL
CQL-AWR
COMBO
IQL
TD3-BC
PBRL
CSVE
Random
HalfCheetah
13.7
17.5 1,5
16.9 1.5
38.8
18.2
11.0 1.1
13.1
26.8 1.5
Hopper
8.7
7.9
8.7
17.9
16.3
8.5 + 0.6
31.6
26.1 7.6
Walker2D
2.2
5.1 1.3
0.0
7.0
5.5
1.6 1.7
8.8
6.2 0.8
Medium
HalfCheetah
50.0
47.0 0.5
60.9
54.2
47.4
48.3 0.3
58.2 + 1.5
48.4 0.3
Hopper
97.5
53.0
2
94.9
66.3
59.3 4.2
81.6 = 14.5
96.7 2 5.7
Walker2D
89.1
73.3 17.7
244
75.5
78.3
2.1
90.3 1.2
83.2
Medium Replay
HalfCheetah
44,9
45.5 0.7
40.0
55.1
44.2
44.6 - 0.5
49.5 + 0.8
54..5 0.6
Hopper
99.4
88.7 = 12.9
91.0 13.0
73.1
94.7
60.9 + 18.8
100.7 = 0.4
91.7 1 0.2
Walker2D
80.0
83.3 2.7
66.7 12.1
56.0
73.9
81.8 5.5
86.2 3.1
78.0 1.5
Mediom Expent
HalfCheetah
62.8
75.6 25.7
73.4
90.0
86.7
90.7 = 4.3
93.1
93.1 = 0.3
Hopper
87.2
105.6 + 12.9
102.2 = 7.7
111.1
91.5
98.0 9.1
111.2 0.7
9-11 = 3.0
Walker2D
109.8
107.9 1.6
98.0 21.7
96.1
109.6
110.1 0.5
109.8 0.2
109.0 + 0.1
Expert
HaltCheetah
20.0
96.3 + 1.3
87.3 8.1
94.6
96.7 = 1.1
96.2 2.3
93.8 0.1
Hopper
111.6
96.5 = 28.0
110.0 = 2.5
109.0
107
110.4 = 0.3
111.3 0.6
Walker2D
110.6
108.5 + 0.5
75.1 60.7
109.4
110.2 0 0.3
108 0.2
108.5 + 0.1
Averuge
63.8
67.4
60.6
64.1
69.7
67.3
76.7
74.8
4.3 Discussion on implementation choices
Now we examine the technical considerations for implementing CSVE in a practical algorithm.
Constraints on Policy Extraction. It is important to note that the state value function alone does not suffice to directly derive a policy. There are two methods for extracting a policy from CSVE. The first method is model-based planning, i.c., π arg max, Bydd, (fs, a) + (s, [V (s')), which involves finding the policy that maximizes the expected future return. However, this method heavily depends on the accuracy of a model and is difficult to implement in practice. As an alternative, we suggest the second method, which learns a Q value or advantage function from the V value function and experience data, and then extracts the policy. Note that CSVE provides no guarantees for conservative estimation on OOD actions, which can cause normal policy extraction methods such as SAC to fail. To address this issue, we adopt policy constraint techniques. On the one hand, during the value estimation E all current states are sampled from the dataset, while the policy is constrained to be close to the behavior policy (ensured via Eq.9). On the other hand, during the policy learning in T0 we use AWR [ as the primary policy extraction method (first term of [q( which implicitly imposes policy constraints and the additional action exploration (second term of Eq TO) is strictly applied to states in the dataset. This exploration provides bonus to actions that: (1) themselves and their model-predictive next-states are both close to the dataset (ensured by the dynamics model), and (2) their values are favorable even with conservatism.
Taking Advantage of CSVE. As outlined in Section 3.1 CSVE allows for more relaxed lower bound on conservative value estimation compared to conservative Q values, providing greater potential for improving the policy. To take advantage of this, the algorithm should enable exploration of out- of-sample but in-distribution states, as described in Section 3 In this paper, we use a deep ensemble dynamics model to support this speculative state exploration, as shown in Eq. TO The reasoning behind this is as follows: for an in-data state s and any action a ~ (-Iss) if the next state s' is in-data or close to the data support, its value is reasonably estimated, and if not, its value has been penalized according to Eq 5 Additionally, the deep ensemble dynamics model captures epistemic uncertainty well, which can effectively cancel out the impact of rare samples of s'. By utilizing CSVE, our algorithm can employ the speculative interpolation to further improve the policy. In contrast, CQL. and AWAC do not have this capability for such enhanced policy optimization.
7

---

Table 2: Performance comparison on Adroit tasks. The results of CSVE are over ten seeds. Results of IQL are from Table in 21 and results of other algorithms are from Table in (10]
AWAC
BC
BEAR
UWAC
CQL
CQL-AWR
IQL.
PBRL
CSVE
Human
Pen
18.7
34.4
-1.0
10.1 + 3.2
37.5
8.4 14+77
71.5
35.4 + 3.3
106.2 +-5.0
Hammer
1.2 0.7
0.3 + 0.0
U.4 + 0.3
Door
-1.8
0.5
-0.3
0.4 0.2
9.9
3.5 + 1.8
4.3
0.1 + 0.0
2.8
Relocate
-0.1
0.0
-0.3
0.0 + 0.0
0.2
0.1 + 0.0
0.1
0.0 + 0.0
0.1
Cloned
Pen
27.2
26.5
23.0
39.2
29.3
37.3
74.9
54.5 5.4
Hammer
-1.8
0.8
0.3
0.4
2.1
0.31 + 0.06
2.1
0.8 + 0.5
0.5
Door
-2.1
-0.1
-0.1
0.0
0.4
-02
1.6
1.6 4.8
1,2
Relocate
-0.4
-0.1
-0.3
-0.3 -0.3 + 0.2
-0.1
0.3 + 0.0
0.0
00.1 + 0.0
-0.3 0.0
Expert
Pon
60.9
85.1
105.9
96.2
107.0
47.1 = 6.8
117.2
135.7 + 3.4
144.0 9.4
Hammer
31.0
125.6
127.3
107.7 21.7
86.7
0.2
124.1
127.5 + 0.2
126.5
Door
98.1
34,9
103.4
104.7 0.4
101.5
85.0 4: 15.9
105.2
95.7 + 12.2
104.2
Relocate
49.0
101.3
98.6
105.5 +: 3.2
95.0
7.2 2 12.5
105.9
84.5 + 12.2
102.9 0.9
Average
23.1
36.7
38.4
37.6
40.3
15.1
47.6
46.6
53.8
5
Experiments
This section evaluates the effectiveness of proposed CSVE algorithm for conservative value estimation in offline RL. In addition, we aim to compare the performance of CSVE with state-of-the. art (SOTA) algorithms. To achieve this, we conduct experimental evaluations on a variety of classic continuous control tasks of Gym 7 and Adroit ( in the D4RL benchmark.
Our compared baselines include: (1) CQL and its variants, CQL-AWR (Appendix D.2) which uses AWR with extra sn-sample exploration as policy extractor, COMBO[6] which extends CQL with model-based rollouts; (2) AWR variants, including AWAC which is a special case of our algorithm with no value penalization (i.c., d = d, in Eq. 2 and exploration on OOD states, [QL which adopts expectile-basee conservative value estimation; (3) [TRL strong algorithm in offline RL, but is quite costly on computation since it uses the ensemble of hundreds sub-models; (4) other SOTA algorithms with public performance results or high-quality open source implementations, including TD3-BC(22), UWAC[24] and BEAR(25). Comparing with CQL variants allows us to investigate the advantages of conservative estimation state values over Q values. By comparing with AWR variants, we distinguish the performance contribution of CSVE from the AWR policy extraction used in our implementation.
5.1 Overall Performance
Evaluation on the Gym Control Tasks. Our method, CSVE, was trained for million steps and evaluated. The results are shown in Cooppe[e to CQL, CSVE outperforms it in 11 out of 15 tasks, with similar performance on the remaining tasks. Additionally, CSVE shows a consistent advantage on datasets that were generated by following random or sub-optimal policies (random and medium). The CQL-AWR method showed slight improvement in some cases, but still underperforms compared to CSVE. When compared to COMBO, CSVE performs better in 7 out of 12 tasks and similarly or slightly worse on the remaining tasks, which highlights the effectiveness of our method's better bounds V. Our method has a clear adcantage in extracting the best policy on medium and medium-expert tasks. Overall, our results provide empirical evidence that using conservative value estimation on states, rather than Q leads to improved performance in offline RI. CSVE outperforms AWAC in 9 out of 15 tasks, demonstrating the effectiveness of our approach in exploring beyond the behavior policy. Additionally, mixed policies (medium-expertt) where AWAC falls short. In comparison to IQL, our method achieves higher scores in 7 out of 9 tasks and maintains comparable performance in the remaining tasks. Furthermore, despite having a significantly lower model capacity and computation cost, CSVE outperforms TD3-BC and is par with PBRL. These results highlight the effectiveness of our conservative value estimation approach.
method excels in extracting the optimal policy on data with
Evaluation on the Adroit Tasks. In Table 2 we report the final evaluation results after training 0.1 million steps. As shown, our method outperforms IQL. in 8 out of 12 tasks, and is competitive with other algorithms on expert datasets. Additionally, note that CSVE is the only method that
8

---

can learn an effective policy on the human dataset for the Pen task, while maintaining medium performance on the cloned dataset. Overall, our results empirically support the effectiveness of our proposed tighter conservative value estimation in improving offline RI. performance.
5.2 Ablation Study
Effect of Exploration Near States. We analyze the impact of varying the factor À in Eq. 10 which controls the intensity on such exploration. We investigated À values of (0.0, 0.5, 1.0) in the medium tasks, fixing 3 0.1. The results are plotted in Fig. [ As shown in the upper figures, has an obvious effect on policy performance and variances during training. With increasing À from O, the converged performance gets better in general. However, when the value of À becomes too large (e.g.. = 3 for hopper and walker2d), the performance may degrade or even collapse. We further investigated the Lg loss as depicted in the bottom figures of Eq. [ finding that larger À values negatively impact L however, once Lg converges to reasonable low value, larger À values lead to performance improvement.
halfcheetah-medlum
hopper-medium
walker2d- d-meeuum
50
80
30
60-
40-
0.0
0.0
10
20
3.5
400
1000
200
800
1000
200
1000
stapsitho.aanda
grandient
grandient stepsthoussrds)
(*1+3)
hopper-medium
walker2d-medium
(+1e3)
haifcheetah-meeduum
350
0.1
300
0.5
250
-20
100
0.0
0.1
200
0.0
-40
0.5
3.0
-50
1000000000
01000000000000 000000
200
600
1000
200
grandlem step:(lhoussaa))
grandiant www.sthhussaneeeeoo ossandds
grandient stopenhh
Figure 1: Effect of performance scores (upper figures) and L losses (bottom figures) in Eq. on medium tasks.
Effect of In-sample Policy Optimization. We examined the impact of varying the factor 3 in Eq. on the balance between behavior cloning and in-samlle policy optimization. We tested different The results indicate that has significant effect on the policy performance during training. Based on our findings, a value of [ = 3.0 was found to be suitable for medium datasets. Additionally, in our implementation, random and medium tasks, and B 0.1 for medium- medium-expers and expert datasets. More details can be found in the ablation study in the appendix.
values on mujoco medium datasets, as shown in Fig 2
3.0 for
6 Related work
The main idea behind offline RL algorithms to incorporate conservatism or regularization into the online RL algorithms. Here, briefly review prior work and compare approach.
Conservative Value Estimation: Prior offiine RL algorithms regularize the learning policy to be close to the data to an explicitly estimated behavior policy. and penalize the exploration

---

Hallcheeita-meddumz22
Hopper-medium-v?
Walker Walkerr2dmediuum-2
80
90
30
50-
40
0.1
30
10.0
20
10.0
20-
10
0
200
600
200
200
grandlent sepsithoussds)
grandilent (laps((hossseeess)
grandient stepstthousands
Figure 2: Effect of performance scores on medium tasks.
ofthe OOD region, via distribution correction estimation
policy constraints with support 25 applying policy divergence based penalty Q-functions 28 or uncertainty-baase penalty 30 on Q-functions and conservative Q-function
26
matching [11 and distributional matching [
estimation 4. Besides, model-based algorithms [5 directly estimate dynamics uncertainty and translate it into reward penalty. Different from this prior work that imposes conservatism on state- action pairs actions, ours directly does such conservative estimation on states and requires explicit uncertainty quantification.
In-Sample Algorithms: AWR [8] updates policy constrained strictly in-sample states and actions, to avoid extrapolation on out-of-support points. [QL uses expectile-based regression to do value estimation and AWR for its policy updates. AWAC[[2], whose actor is AWR, is actor-critic algorithm to accelerate online RL with offline data. The major drawback of AWR method when used for offline RL is that the in-sample policy learning limits the final performance.
Model-based offline RL learns the dynamics model from the static dataset and uses it to quantify uncertainty [13. data augmentation [6] with roll-outs, planning [16 31). Such methods typically rely on wide data coverage when planning and data augmentation with roll-outs, and low model estimation error when estimating uncertainty, which is difficult to satisfy in reality and leads to policy instability. Instead, we use the model to sample the next-step states only reachable from data, which has no such strict requirements data coverage model bias.
Model-Basee Algorithms:
Our theoretical results are derived from conservative Q-value estimation (CQL) and safe policy improvement [32. Compared to offline policy evaluation [3], which aims to provide a better estimation of the value function, we focus on providing a better lower bound. Additionally, hen the dataset is augmented with model-based roll-outs, COMBO [6] provides more conservative yet tighter value estimation than CQL. CSVE achives the same lower bounds as COMBO but under more general state distributions.
Theoretical Results:
7 Conclusions
In this paper, we propose CSVE, new approach for offline RL based on conservative value estimation states. We demonstrated how its theoretical results can lead to effective algorithms. In particular, we develop a practical actor-critic algorithm, in which the critic achieves conservative state value estimation by incorporating the penalty of the model predictive next-states into Bellman iterations, and the actor does the advantage-weighted policy updates enhanced via model-based state exploration. Experimental evaluation shows that our method performs better than alternative methods based on conservative Q-function estimation and is competitive among the SOTA methods, thereby validating theoretical analysis. Moving forward, aim delve deeper into designing more powerful algorithms grounded in conservative state value estimation.
10

---

References
[1] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 2052-2062. PMLR, 09-15 Jun 2019.
[2] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[3] Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 19967-20025. PMLR, 17-23 Jul 2022.
[4] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179- 1191, 2020.
[5] Jacob Buckman, Carles Gelada, and Mare Bellemare. The importance of pessimism in fixed-dataset policy optimization. In International Conference on Learning Representations, 2020.
[6] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954 29-2777--28977, 2021.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: 1606001540 2016.
[8] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint 1eXXix:709..10000 2017.
[9] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[10] Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations, 2021.
[11] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 19Xiv:911..11300 2019.
[12] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets, 2020.
[13] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.
[14] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighter regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXhv:1910.00177,, 2019.
[15] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 1114419914111 2020.
[16) Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810-21888 2020.
[17] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances Neural Information Processing Systems, 32, 2019.
11

---

[18] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. eerantage-weightee regression: Simple and scalable off-po reinforeement learning. CoRR, abs/1910.00177,, 2019.
[19] Jonathon Shlens. Notes on kullback-leibler divergence and likelihood. CoRR, 2bs/1144,,000,, 2014.
[20] Danijar Hafner, Timothy Lillierap, Jimmy Ba, and Mohammad Norouzi. Dream control:
Learning behaviors by latent imagination. arXiv preprint 19Xiv:1922.01600 2019.
[21] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offiin reinforcement learning with implicit q-learning In International Conference on Learning Representations, 2021.
[22] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-2011 2021.
[23] Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in Neural Information Processing Systems, 2022.
[24] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv: 40, 2021.
[25] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[26] Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off-policy confidence interval estimation. Advances in neural information processing systems, 33:9398-944 2020.
[27] Mengjiao Yang, Ofir Nachum, Bo Dai. Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33:6551-
6561, 2020.
(28) Iya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783 PMLR, 2021.
[29] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778. 2020.
[30] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pages 104-114 114. PMLR, 2020.
[31] Xiong-Hul Chen, Yang Yu, Qingyang Li, Fan-Ming Luo, Zhiwei Qin, Wenjie Shang, and Jieping Ye. Offline model-based adaptable policy learning, Advances in Neural Information Processing Systems, 34:8432-8443, 2021.
[32] Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International Conference on Machine Learning, pages 3652-3661. PMLR, 2019.
[33] Carles Gelada and Mare G Bellemare. Off-policy deep reinforcement learning by bootstrapping the covariate shift. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3647-3655, 2019.
[34] Takuma Seno and Michita Imai d3rlpy: An offline deep reinforcement learning library. arXiv preprint wxXv:2111.0070 2021.
[35] Yuwei Fu, Di Wu, and Benoit Boulet. A closer look at offline rl agents. In Advances in Neural Information Processing Systems, 2022.
12

---

A Proofs
We first redefine notation for clarity and then provide the proofs of the results from the main paper.
Notation. Let N denotes iteration of policy evaluation(in Section 3.2). V denotes the true, tabular (or functional) V-function iterate in the MDP. y denotes the approximate tabular (or functional) V-function iterate.
The empirical Bellman operator is defined as:
(Ba [ (awa(()s)((ss))) + ((s
(11)
where i(8, a) the empirical average reward derived from the dataset when performing action at state The true Bellman operator is given by:
(B (*) = ((s - Σ P(
(12)
Now we first prove that the iteration in Eq has fixed point. Assume the state value function is lower bounded, i.e., (s)) 2 Vs E S. then Eq can always be solved with Eq Thus, we only need to investigate the iteration in 3
Defining this iteration as function operator T on V and supposing that supp d C supp du it's evident that the operator T displays "coottraction within L. norm where is the discounting
factor.
Proof of Lemma 3.1 Let and V be any two state value functions with the same support, i.e., suppV suppV'
(T - (sV = (B*V(s) al d() - 11) (B)V - a. d(s) (s(s)
- 1])
ds(s)
Vs) - 13 V (s)
=I =((Ban((els)((s a) + 7 Eant(a))) Σ P(5((s,))((5)))
(Ea) + 2 Eana(als)
=7 Fawa(als)
TTV-TVV'l max -TV--VV')())
maxx Farx(ols) Σ P(s)ss. v'(')
Sa Σ P(s'|s,a) v'(5))
=y max (ss") v'(s")
-
=>|((V - -V)l--
We provide bound on the difference between the empirical Bellman operator and the true Bellman operator. Following previous work 4 we make the following assumptions. Let pa be the transition matrix associated with the policy, specifically, P V(s) Easaatelle)) P(e' (V (V
13

---

Assumption A.1. Vs,a E M, the relationship below hold with at least (1 6) (8 E (0, probability, r r(-r(s,))
G _ -(6))))))) S VID(s,a) a)
Cps
(13)
VID(s,a)
Given this assumption, the absolute difference between the empirical Bellman operator and the true one can be deduced as follows:
(BTT)V - (B3)) = Fo-s(al))-- r(8,a) +7 E Fe (P P(s' ((s))s.))
(14)
$ ( +r E (P - P(s'ls p
(15)
Cr + t2 ?)
S Eawz(als)
(16)
VID(s,
The error in estimation due to sampling therefore be bounded by constant, dependent C₁,5 and Ct,5- We define this constant as C2,
Thus we obtain:
Cr.4.6
D. (*)(s) BV - S (ara)()))) (1 DDsssa)
(17)
Next we provide an important lemma.
Lemma A.2. (Interpolation Lemma) For any f E [0, 1), and any given distribution p(s), ler d, be an f-interppolation of and D, i.e.dy(s) == fd(s) + (1 f)p(s), let v(. == Esmp(s) [ then v(p, f) satisfies u(p f) 2
The proof can be found in (6). By setting f as 1, we have E E
((s) p(s)-d(s) d(s)
>
d(x)
Proof of Theorem 3.2 The function approximate dynamic programming in iteration can be obtained as:
d(s) 1] Vs, k
= BB) - al al2,(1) (s)
(18)
The fixed point:
d(s)
Cr,,,6 max
d(s)
(**(s) = B (+)) (s)
(B) + Eana(al))
(1- ((s.a)
(s(s)
(19)
Thus obtain:
Eana(als) (1 7) (V-))) a(l PP") d(s) 1] (20)
Br
du (8)
where pr is the transition matrix coupled with the policy and (*) P(
Then the expectation of v (x) under distribution d(s) satisfies:
Exwd(s) (s) < {E (V* + Esnd(ss - -P=)- Ea-z(als) Cr, Rmax
(1-7)v/D() ((s
d(s)
(21)
-a Eandles - p PF
1])
(s(s)
14

---

When Esndlta Fandtt) Fowartll)
6mmux
R--yybia Bxnd(s) (s) Esnd(s) (VT(s)). (V)))))
Exratta) (a 11)
Proof of Theorem
The expectation of (s) under distribution d(s) satisfies:
Esndu(s) (8) S (s) + (s) - -PP") - Cra.s
1177)11 ((s,,))
(22)
d(s)
aaEodd -1 -P=)- - ds((s)
11)
Noticed that the last term:
d1(8) du(s)
ds(s)
Σ
1) 1) Σ du(s)
-1) -1)= -di(s)->
Edd(s) ds))s =0
(23)
ds(s)
(o)
We obtain that:
&
(s) Esndd +
- (1 ( (1 7) () (ss
Eanx(als)
(24)
Proof of Theorem 3.4: Recall that the expression of the V-function iterate given by:
po+1 (s) = 1=* f (s) - a d(s)
1| k
(25)
(s)
Now the expectation of (* (8) under distribution dy (s) is given by:
Essdd (a) V k+1 (s) (x)du(s)
1 1 v*()) a d(s)
11 E Esnd (*)
(26)
(s)
The expectation of (*)) under distribution d(s) is given by:
d(s) ds(s)
d(s) 1]
Fand(e) (x) F B=* fk (*) -a(
1] F Fend( B (8)
du(s)
(27)
Thus we can show that:
Egdd, (s) (s) - Exnd(a) ( = Exdd Ba (*((s) - Esna(s) (*)(s) + aB c(s))
1]
du(s)
Exndy (s) V.tll (s) - Esnd((s) (let((s)
Esnd(s) 18 (irl
-V²)]
d(s) 1]
+ E Esnd (B8 (Dx - Vk]] + (Esud((s du(s)
(28)
By choosing a:
a > Eand(e) (vn (VK-V*) Eandi B (D4-V*) v4y
(29)
Exnd(s) d(e)
(a)
We have Exsd (x)
Vktl(g) (s) Esnd(s) (++1 Eands(s) V ( Esnd(e) V (s)
hold.
Proof of Theorem 3.5 V is obtained by solving the recursive Bellman fixed point equation in the
empirical MDP, with an altered reward, r(s, a)
af ald,.(s)
dia)
1). hence the optimal policy n (a]s)
obtained by optimizing the value under Eq. 3.5
L
Proof of Theorem 3.6; The proof of this statement is divided into two parts. We first relates the return of 7" in the empirical MDP M with the return of Te. we can get:
d(s)
( M) a
1 2 J ( M) 0 = ((Tg,
(30)
d
15

---

Algorithm CSVE based Offline RL Algorithm
Input: data D
(())
Parametered Models: Qo, V4, To. Qg. M,
Hyperparameters: a, X, learning rates ne. ng. nl. w
> Train the transition model with the static dataset D
M &- Irain( D)-
Train the conservative value estimation and policy functions Initialize function parameters Oo, 20, Φ0. Ug = 00
for step to N do
Lh - wk- Th V LV (Voi Qea.)
0g - 176 L (Qo: Ves)
@k - k T6 V V (T²)
0g f w + (1 w)
end for
The next step is to bound the difference between J(Tg, M)) and ( M) and the difference between J(x", .M) and J(x", M). We quote useful lemma from 4 (Lemma D.4.1):
Lemma A.3. For any MDP M, an empirical MDP M generated by sampling actions according to the behavior policy Tg and given policy m,
Cr.8 Rmaa CT,6 B sond (e) (1-?)²
A
<[aas)) xg(aas)
[[T,, J(T S
E Ban
) (31)
1-1
(D(s)
Setting и in the above lemma as TB. we get:
Cr.8
Bmaar CT
|A
T (a|ss)
[J(TT, M) [TT
)
Eaws (als)
)
I-7 (1 117)²
[D(s]]
TT
(32)
(x" (ala)) given that v Eana (ala) (pp(ls) is pointwise upper bound of v Enwag(als) get,
(g(a xp(a)s)
Thus we
Crs
+ B.mma CT.5
VIA
x" (als)
J( (M JTT M) - 2(
) Band (s)
EaNa (als)
) T3(aas)
7
(1 7)2
(D(s)
1 + a
d(s)
Ennay
3z (1)
(s(s)
1]
(33)
which completes the proof.
Here, the second term represents the sampling error, which arises due the discrepancy between M and M. The third term signifies the enhancement in policy performance attributed to our algorithm in M. It's worth noting that when the first term is minimized, smaller values of a can achieve improvements over the behavior policy.
B CSVE Algorithm and Implementation Details
In Section we deteiled the complete formula descriptions of the CSVE practical offline RL algorithm. Here consolidate those details and present the full deep offline reinforcement learning algorithm as illustrated Alg. [ In particular, the dynamic model, value functions, and policy are parameterized with deep neural networks and optimized via stochastic gradient descent methods.
We implement our method based on an offline deep reinforcement learning library d3rlpy 34 The
code is available at:
16

---

Table 3: Hyper-parameters of CSVE evaluation
Hapee-paramelers
Value and description
5, number of ensembles in dynamics model
10, to control the penalty of OOD states
10, budget parameter in Eq. 8
In Gym domain, 3 for random and medium tasks, 0.1 for the other lasks;
In Adroit domain, for human and cloned tasks, 0.01 for expert tasks 0.99, discount factor.
million Mujoco while 0.1 million for Adroit tasks. 0.005, target network smoothing cnefficient.
Ir of actor
3e-4, policy learning rate
Ir critic
le-4, critic learning rate
C Additional Ablation Study
Effect of model errors. Compared traditional model-based offline RL algorithms, CSVE exhibits greater resilience to model biases. To access this resilience quantitatively. we measured the perfor- mance impact of model biases using the average 1.2 error in transition prediction
indicator. As shown in Fig. 3 the influence of model bias on RL performance is CSVE is marginal. Specifically, in the halfcheetah task, there is observable impact model errors scores, model errors show no discernible impact on scores, For the hopper and walker2d tasks, only minor decline in scores is observed as the errors escalate.
halfcheetah-modiummz2
120
hopper-medium--2
120
walker2d-medium-42
100
100
100
80
60
60
20
original values
20
fitted
0.12
0.14
0.16
errors
errors
error
Figure 3: Effect of the model biases performance scores. The correlation coefficient is -0.32, -0.34, and -0.29 respectively.
D Experimental Details and Complementary Results
D.1 Hyper-parameters of CSVE evaluation in experiments
Table provides a detailed breakdown of the hyper-parameters used for evaluating CSVE in our experiments.
D.2 Details of Baseline CQL-AWR
To facilitate a direct comparison between the effects conservative state value estimation and Q-value estimation, formulated baseline method named CQL-AWR detailed below:
@k+1 +- arg min a (Es-D, (Q(s, FonDDanas(al)))0 (Q(s, 1 E. - - (s)
Q
π arg min La ( -E, 1og n' (a(s) exp (A (BARt) \EmDDaNT(() (Q(t)(s,a))
where ( = (s,)(s, Earx ( (s,
17

---

halfcheciah medium--ea
hopper mediummrooaa roplay
roplay
80-
80
-0
20-
10
20-
grancient StQpa (oasaa))
grandiens seaastilesaaaaaa
grand (wssalha
ropllay
hopger-medlum replay
replay
3-
70-
0-
1000
200
450
693
a2o
1000
200
400
6:00
an0
1000
grand slasslhaassnnaa)
grandient Sapostoossinsss
grandient Sispathsssss)))
Figure 4: Effect of À to Score (upper figures) and Lu loss in Eq. (bottom figures)
In CQL-AWR, the critic adopts standard CQL equation, while the policy improvement part uses AWR extension combined with novel action exploration as denoted by the conservative Q function. When juxtaposed with our CSVE implementation, the policy segment of CQL-AWR mirrors ours, with the primary distinction being that its exploration is rooted in a Q-based and model-free approach.
D.3 Reproduction COMBO
In Table our main paper, we used the results of COMBO as presented in the literature 23 Here we detail additional attempts to reproduce the results and compare the performance of CSVE with COMBO.
Official Code. We initially aimed to rerun the experiment using the official COMBO code provided by the authors. The code is implemented in Tensoflow 1.x and relies software versions from 2018. Despite our best efforts to recreate the computational environment, we encountered challenges in reproducing the reported results. For instance, Fig. 5 illustrates the asymptotic performance on medium datasets up to 1000 epochs, where the scores have been normalized based on SAC performance metries. Notably, for both the hopper and walker2d tasks. the performance scores exhibited significant variability. The average scores over the last 10 epochs for halfcheetah, hopper, and walker2d were 71.7, 65.3, and -0.26, respectively. Furthermore, we observed that even when using the D4RL vO dataset, COMBO demonstrated similar performance patterns when recommended hyper-parameters were applied.
Figure Return of official COMBO implementation on D4RL mujoco v2 tasks, fixing seed=0.
18

---

JAX-based optimized implementation Code [35]. We also tested one recent implementation available RIQL, This version regarded as the most highly-tuned implementation date. The results of our tests can be found in 6 For the random and expert datasets, we applied the same nyper-parameters as those used for the medium and medium-expert datasets, respectively. For all other datasets, we adhered to the default hyper-paramette these efforts, when we compared our outcomes with the original authors' results (as shown in Table 10 and F 7 of [35]), our reproduced results consistently exhibited both lower performance scores and greater variability.
provided by the authors [35]. Despite
hafeweawwordmmmd
(171
No wp
1300
Uree 1161
850
400
Figure 6: Return of optimized COMBO implementation (35 D4RL mujoco v2 tasks. The data are obtained by running with 5 seeds for each task, and the dynamics model has 7 ensembles.
D.4 Effect of Exploration on Near Dataset Distributions
As discussed in Section 3.1 and 4.2 the appropriate selection of exploration on the distribution (d) beyond data (d,,) should help policy improvement, The factor Eq. 10 controls the trade-off on such 'bonus' exploration and complying with the data-implied behavior policy.
19

---

we examined the effect of À on the medium datasets of mujoco tasks. Now let us further take the medium-replay type of datasets for more analysis of its effect. In the experiments, with fixed / = 0.1, we investigate À values of (0.0, 0.5, 1.0, 3.0). As shown in the upper figures in Fig. 4 shows an obvious effect on policy performance and variances during training. In general, there is value under which increasing À leads to performance improvement, while above which further increasing hurts performance. For example, with = 3.0 in hopper-medium-replyy task task, the performance gets worse than with smaller À values. The value of À is lask-specific, and we find that its effect is highly related to the loss in Eq. 9 which can be observed by comparing the bottom and upper figures in Fig. 4 Thus, in practice, we choose proper according to the above loss without online interaction.
o==0.15 In section 5.2
and walker22dmedium-ree
D.5 Conservative State Value Estimation by Perturbing Data State with Noise
In this section, we investigate model-free method for sampling OOD states and compare its results with the model-based method adopted section
The model-free method samples OOD states by randomly adding Gaussian noise to the sampled states from the data. Specifically, we replace the Bq/5with the following Eq. 34 and other parts consistent with the previous technology.
G + arg min (VV
-Bo----))) ESnD(V(s)))
(34)
+ EaND [ (1) V
The experimental results on the Mujoco control tasks are summarized in Table 4 different noise levels (a²), the model-free free CSVE perform worse than our original basel- CSVE implementation; and for some problems, the model-free method shows very large variances across seeds. Intuitively, if the noise level covers the reasonable state distribution around data, its performance is good; otherwise, it mishehaves. Unfortunately, it is hard to find noise level that is consistent for different tasks or even the same tasks with different seeds.
As shown, with
Table 4: Performance comparison on Gym control tasks. The results of different noise levels are over three seeds.
CQL
CSVE
o²=0.05
7²=0.1
Medium Random
HalfCheetah
17.5 + 1.5
26.7 2.0
2.4
20.4 1.3
18.6 + 1.1
Hopper
7.9
8.5
4.1
14,2 15.3
6.7 5.4
Walker2I)
5.1 1.3
6.1 +11+00
3.9
7.5
1.7 1.7+30 3.5
HalfCheetah
47.0
4.06+
0.2
47.5
46.0
Hopper
53.0
5.3
36.9 32.6
2.11
18.4 30.6
Walker2D
73.3
1.5
8
75.5
78.6
Expert Medium Medium Expert
HalfCheetah
4.7
5..8
4..8
4.5
43.8 0.4
Replay
Hopper
+8.7+12.9
91.7+0.3
8..
78.3 4.3
70.2 + 12.0
Walker2D
2.7
78.5
3.3
1.3
66.8 - 4.0
HallCheetah
75.6 25.7
0.3
6.00
89.7
93.8 + 1.6
Hopper
105.6 4 12.9
9.8
63.2
99.0
37.6 63.9 110.4 1.3
Walker2D
107.9 1.6
1.9.0
108.4 1.9
HalfCheetah
93.8
59.0 28.6
67.5
75.3 27.3
Hopper
28.0
1.6
109.2 67.3
109.4 2.1
Walker2D
0.5
1.1.7 0.0
1.6.9
108.6 + 0.3
20


Reviews:
[{"review_id": "4aIfxHhiCY", "reviewer": "Reviewer_EMsr", "summary": "In this paper, the authors propose an offline RL algorithm for conservative state value estimation (CSVE). This work is different than prior works that learn conservative state-action values like CQL or COMBO, in that it penalizes OOD states rather than OOD state-actions. \n\nThe authors show that CSVE gets similar theoretical lower bounds to COMBO, but under more general state distributions. The practical version of the algorithm is similar to COMBO with the following key differences:\n\n*  CSVE learns a NN value function and penalizes OOD states (rather than state-actions for the Q function)\n*  the policy is updated using AWR rather than SAC\n*  the learned dynamics models are used for 1-step rollouts to generate fictitious state transitions for the OOD state penalty\n*  the learned dynamics model is additionally used to allow the policy to explore local 1-step transitions around the data\n\nThe authors evaluate the CSVE method on the Mujoco and Adroit tasks from the D4RL benchmark. Generally, they find that their method outperforms or matches the relevant baselines (CQL, COMBO, AWAC) on these tasks.", "questions": "Are there any environments or data regimes where you expect to see a bigger improvement for CSVE relative to prior methods? The current results are a bit underwhelming considering that many of the D4RL tasks are pretty saturated. Perhaps evaluating on the half-cheetah jump or ant angle tasks that have been tackled in other offline MB RL approaches like COMBO could lead to a stronger result. Considering that you only require 1-step predictions from your dynamics model, you could also potentially test on environments with more complicated observations like images.", "limitations": "No obvious limitation.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "To the best of my knowledge, the proposed CSVE algorithm is novel and is an interesting alternative to COMBO. The authors present good theoretical and empirical results illustrating the potential benefits of CSVE over COMBO and CQL. Thus, I believe that further exploration of whether incorporating the conservative penalty into a state-action value Q function or the state value function in a wider range of settings is an interesting direction for future research. Thus, I believe this work provides a worthwhile contribution to the research field.", "weaknesses": "There are a lot of small syntactical and word choice errors that do distract the reader and should be addressed in future versions, but generally the ideas presented in the paper are still comprehensible and well organized.\n\nIt is unclear to me exactly how the \"Model-based Exploration on Near States\" is performed and how Equation 9 is being optimized. Specifically, it is unclear to me whether the 2nd term in Equation 9 is being optimized with some variant of DPG (like TD3 or SAC) with the gradient being taken through the learned dynamics model, or by some variant of AWR like with the rest of the algorithm. In my opinion, this design choice is quite significant and should be fully explained in the main body of the paper. Additionally, the tradeoff between the 2 terms in Equation 9 seems important and I would appreciate the ablation study over the different values for $\\lambda$ to be in the main body of the paper.\n\nThe Experiments section is a bit hard to follow, mostly because Tables 1 and 2 are hard to interpret without assistance. I think the clarity of the results would greatly be improved by bolding the top scores, and including aggregate scores like is done in many prior offline RL works. This would make it much easier to interpret how CSVE compares to priors algorithms.\n\nFinally, the current ablation study in section 5.2 needs more analysis otherwise it doesn't seem to add much to the main body of the paper. I think an ablation with accompanying analysis on varying $\\lambda$ and $\\tau$ would be more interesting to include in the main body of the paper as those parameters seem in my opinion to be more related to the novel components presented in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and detailed comments. We are glad to hear that Reviewer EMsr believes that our work presents good theoretical and empirical results illustrating the potential benefits of CSVE.  We respond to specific questions and comments below.\n\n## 1. Main concerns\n\nConcern 1: A lot of small syntactical error and word choice errors\n\nThank you for pointing that out. We appreciate your feedback on the readability of our paper and recognize that there are some syntactical and word choice errors that might be distracting. In future revisions, we will give more attention to these issues, ensuring that the text is polished and easier to read.\n\nConcern 2: how the \"Model-based Exploration on Near States\" is performed\n\nThe optimization of the second term in Equation 9 involves calculating the gradient through the learned dynamics model. This is achieved by employing analytic gradients through the learned dynamics to maximize the value estimates. It is important to note that the value estimates rely on the reward and value predictions, which are dependent on the imagined states and actions. As all these steps are implemented using neural networks, the gradient is analytically computed using stochastic back-propagation, a concept inspired by Dreamer[1]. Additionally, the detailed implementation can be found in the accompanying code provided with our paper.\nWe agreed that this discussion is important and will include this in our updated version.\n\nConcern 3: The Experiments section is a bit hard to follow\n\n\nIn response to your suggestion, we have revised the paper and included two updated tables in the attached PDF. In these tables, we have highlighted the scores that exceed 90% of the highest score. The average score is also provided for a more comprehensive comparison.\n\nConcern 4: More ablation should be added into the main body\n\nWe've included an ablation study about different $\\lambda$ in Appendix B. And according to your follow-up suggestion, we will place the ablation of $\\lambda$ in our main text and move the current ablation study of $\\beta$ to the appendix.\n \n\n## 2. Questions\n\n> Q1: Any environments or data regimes that we expect to see a bigger improvement?\n\nWe fully agree that it is valuable for further exploration of more applications. We check the halfcheetah-jump and ant-angle used in MOPO and COMBO. However, we run into some reproduction difficulty using the source code of COMBO（Appendix C.4). In future work, we would indeed love to test CSVE on tasks like Half-Cheetah Jump, Ant Angle, and environments with more complex observations such as images. This will allow us to better assess the method's potential and capabilities across a broader range of applications and compare it with existing approaches.\n\nIn general, CSVE have advantage in the broader set of scenarios where the environment is well-defined real-world physical system and the behaviour policy is not optimal. Compared to IQL and CQL, CSVE has less value underestimation and better exploration on state-action pairs near the dataset. \n\nReferences:\n[1]Dream to control: Learning behaviors by latent imagination, ICLR 2020."}, {"review_id": "7qoCoT2pvV", "reviewer": "Reviewer_QSVW", "summary": "<< I have read the authors' rebuttal and had raised my score based on the discussion >>\n\nThe paper discusses challenges in Reinforcement Learning (RL), particularly in real applications where online learning from scratch is often risky and unfeasible. To address this, the authors introduce Conservative State Value Estimation (CSVE), a novel offline RL approach that deviates from traditional methods that estimate conservative values by penalizing the Q-function on Out-Of-Distribution (OOD) states or actions. Instead, CSVE penalizes the V-function directly on OOD states. The authors present theoretical evidence that CSVE provides tighter bounds on true state values than Conservative Q-Learning (CQL), with similar bounds as COMBO but under more general discounted state distributions. This allows for potentially more effective policy optimization within the data support. The primary contributions of the paper include the proposal and theoretical analysis of conservative state value estimation, the introduction of a practical actor-critic algorithm applying CSVE, and an experimental evaluation demonstrating superior performance of CSVE over prior methods based on conservative Q-value estimation in tasks of Gym and Adroit in D4RL benchmarks. The simplicity of the proposed changes augments their potential for practical adoption in the field.", "questions": "Q1: Assume we express Q(s,a) as R(s,a) + V(s') and learn a world model for one-step transitions rather than predicting Q(s,a) directly. In this case, minimizing the CQL objective in equation 1 would implicitly penalize the state value of OOD states. Could the authors explain how this approach differs from the one proposed in this paper?\n\nQ2: What implications would an inaccurate world model have, specifically when the predicted next state $\\hat{s'}$ doesn't match the actual next state ${s'}$?\n\nQ3. Could the authors provide experiment results with more seeds?", "limitations": "The authors did not discuss the limitations of their work. No discussion needed regarding potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clarity: The manuscript is effectively composed, offering straightforward comprehension. However, there's room for improvement in the explanation of certain equations. By simplifying these complex components, the reader's cognitive load could be significantly reduced.\n\nTechnical Soundness: The theoretical foundations of the paper are solid. The authors offer convincing theoretical derivations to support the proposed approach, contributing to the technical soundness of the paper.\n\nOriginality: Overestimation of action values represents a recurring challenge in the offline reinforcement learning landscape. The authors innovatively address this issue by learning a conservative estimate of state values and penalizing OOD states, offering a potentially tighter bound to the actual state value function. In my view, this represents a novel contribution to the field.\n\nSignificance: Attaining a tighter bound on the value estimate can considerably enhance performance in offline RL problems. Additionally, the simplicity of the proposed approach augments its potential for widespread adoption and application.", "weaknesses": "W1: The experimental results primarily focus on a range of continuous control tasks, neglecting discrete action space problems. Furthermore, the decision to use only three seeds for performance comparison appears limited, especially considering the growing trend to use ten seeds to apply metrics such as the Interquartile Mean (IQM) [1]. This raises concerns about the statistical validity of the proposed approach.\n\nW2: The observed performance gains from implementing CSVE do not appear consistently significant across different domains or discernible patterns. The sporadic nature of these gains may question the efficacy of the proposed approach in diverse applications.\n\n[1] https://arxiv.org/abs/2108.13264", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: neglecting discrete action space problems and only three seeds for performance appears limited\n\nRecently, we have conducted additional experiments with seven more seeds on HalfCheetah-medium, HalfCheetah-medium-replay, and HalfCheetah-medium-expert tasks. The updated scores can be found in the attached PDF. The results are consistent with the results reported in the paper. Given the limit on time and available computing resources, we can not reproduce all experiments in this rebuttal window. In the future revision, we will re-run all experiments over 10 seeds and report the results. \n\nThough discrete action is excluded in our analysis, In future work, we aim to include a wider range of tasks to better demonstrate the effectiveness and versatility of the proposed approach.\n\n\n\nW2: The observed performance gains from implementing CSVE do not appear consistently significant .\n\nWe have updated Table 1 and Table 2 in the attached PDF for a better illustration of the performance gains of our method. Considering the average score, our method is on par with PBRL in the gym domain and better than PBRL in the adroit domain. We agree with the reviewer that our method is not consistently significantly better than other methods, but overall it outperforms them.\n\n\n\n\n\n\nQ1: Minimizing the CQL objective in equation 1 would implicitly penalize the state value of OOD states\n\nWhen we express Q(s,a) as R(s,a) + V(s') and  minimize the CQL objective in equation 1 would indeed penalize the state value of OOD states. However, CQL underestimates state values point-wisely, while  CSVE underestimate state values on expectation. \n- CQL:\n$$ \\mathbb E_{\\pi(a|s)}[\\hat{Q}^{\\pi}(s, a)] \\leq \\mathbb E_{\\pi(a|s)}[Q^{\\pi}(s, a)], \\forall s \\in D $$\n$$ \\hat{V}^{\\pi}(s) \\leq V^\\pi(s) , \\forall s \\in D $$\n\n\n- CSVE:\n$$ \\mathbb E_{s \\sim d}[\\hat{V}^{\\pi}(s)] \\leq \\mathbb E_{s \\sim d}[V^{\\pi}(s)]$$\n\nCQL learns the conservative Q-values of $Q(s, a)$ on transitions $\\{(s, a, r, s')\\}^N$ of dataset $D$ where the value overestimation can only be introduced by OOD actions on $s'$. We aruge that **this is also a limitation of CQL that hinders further performance improvement**, since it does not learn Q-values on any out-of-sample state $s \\notin D$ even the state is in-distribution with respect to the behaviour policy.\n\nThe intuition is that proper exploration on out-of-sample but in-distribution states (i.e., the states near data as in our paper) has benefits on improving the learnt policy compared to the behaviour policy that collected the data. As a comparision, penalizing Q for OOD actions has no guarranttees on OOD states (in CQL). However, when directly penalizing V function, CSVE get the same lower bounds as COMBO but under more general state distribution (Section 3.1). This means for those OOD actions, we can still get lower bound its value by bounding the state value, thus decreasing the extrapolaation error. We will refine our discussion of this comparison in the updated version. \n\n \nQ2: What implications would an inaccurate world model have\n\nWith an inaccurate world model, the theoretical part of our method will not be affected, given that the whole deduction does not require a model. However, the optimization part of Equation 9 will be affected. An inaccurate dynamics model may guide the policy to produce erroneous actions that result in high imaginary states.\n\nIn the experiments, we find that the effect of model bias on RL performance is subtle in the medium tasks. We include experiments regarding different model errors in Appendix B. We use the average L2 error on transition prediction as a surrogate for model biases. Specifically, for the HalfCheetah task, there is no observable impact of model errors on scores, while in the Hopper and Walker2D tasks, there is only a slight decrease in scores as the errors increase.\n\n\nQ3. Could the authors provide experiment results with more seeds?\n\nWe provide results over 10 seeds of three tasks in the pdf attached.  The results are consistent with the results reported in the paper. We will re-run all experiments over 10 seeds and report the results."}, {"review_id": "HRpwnOS06e", "reviewer": "Reviewer_ze4E", "summary": "The paper proposes a method to tackle the overestimation of values in offline RL by focussing on state-values instead of state-action values and using in-data policy optimization techniques based on model-based RL. They propose an actor-critic variation for their approach and apply the method on various offline RL tasks.", "questions": "Questions\n1. I don’t fully understand why model-based helps here. Aren’t the issues of OOD actions still relevant given that the transition model is a function of the action and the action is sampled from the evaluation policy?\n2. Is this method scalable to when multiple policies generate the fixed dataset? And can we have a behavior policy-agnostic version where we don’t know what policies generated the data?\n3. While this paper is different, there does seem to be some relation to [1]. In that paper, they learn the state density ratio and use that for better control. Can the authors comment on the difference between how this paper uses the state-density ratio vs. how [1] uses the state-density ratio (not the ratio learning part)?\n\n[1] Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift. Gelada et al.\n", "limitations": "See above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is derivative of existing ideas such as conservative Q learning, which I see as a pro because it does not drastically depart from an already well-established algorithm.\n2. I think there is appeal in the method in that it gives some sense of how results can be different when we start to incorporate state-based quantities instead of state-action based quantities, and when we include model-based approaches. The paper may be able to spark some interesting ideas for other papers.\n", "weaknesses": "1. The paper should bold the results in their experiments sections. Its incredibly tedious to read and discern where CSVE performs well and where it doesnt.\n2. There is little intuition for why state-based methods in this case can work better than state-action. I think given that the tweak is somewhat minimal, the paper should stress why this tweak can actually make things better than is typically done.\n3. In terms of bounds, I don’t see a comparison to CQL? That is, there are V-function bounds computed by CSVE and true, but there aren’t V-function bounds computed implicitly by CQL (when CQL computes its Q-functions, and then using that to compute the V-function) and the true V-function. I think that is related to point 2 and it would be illuminating.\n4. I find it a bit unsettling that the results of prior work in Table 1 were just copy-pasted here. It's not clear to me if it is an absolutely fair comparison since setups between various papers can be different (random seeds etc). It would make more sense to me to re-run the algorithms in the setup used in the paper. Moreover, the algorithms have been run only for 3 seeds, which is far too little since these are not image-based environments.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and detailed comments. We respond to specific questions and comments as bellow.\n## 1. Concerns\nConcern 1: Table is hard to read\n\nIn response to your suggestion, we have revised the paper and included two modified tables in the pdf attached. In these tables, we have highlighted the scores that are larger than 90% of the largest score. \n\nConcern 2 : Why state-based method helps in our method？\n\nTo clarify the contribution of our state-based method, we compare it with CQL and COMBO.\n\nAs shown below, CQL underestimates state values point-wisely, while COMBO and CSVE underestimate state values on expectation. \n- CQL:\n$$ \\mathbb E_{\\pi(a|s)}[\\hat{Q}^{\\pi}(s, a)] \\leq \\mathbb E_{\\pi(a|s)}[Q^{\\pi}(s, a)], \\forall s \\in D $$\n$$ \\hat{V}^{\\pi}(s) \\leq V^\\pi(s) , \\forall s \\in D $$\n\n- COMBO:\n$$\\mathbb E_{s \\sim \\mu_0, a \\sim \\pi(a|s)}[\\hat{Q}^{\\pi}(s, a)] \\leq \\mathbb E_{s \\sim \\mu_0, a \\sim \\pi(a|s)}[Q^{\\pi}(s, a)]$$\n$$\\mathbb E_{s \\sim \\mu_0}[\\hat{V}^\\pi(s)] \\leq \\mathbb E_{s \\sim \\mu_0}[V^\\pi(s)] $$\n\n- CSVE:\n$$ \\mathbb E_{s \\sim d}[\\hat{V}^{\\pi}(s)] \\leq \\mathbb E_{s \\sim d}[V^{\\pi}(s)]$$\n\nwhere $\\mu_0$ in COMBO represents behavior policy, $\\hat{Q}^{\\pi}$ is the estimation of $Q^{\\pi}$,  and $d(s)$ in CSVE represents any state distribution.\nThe motivation for using the state-based method can be summarized as follows:\n- Compared to CQL, both CSVE and COMBO aim to achieve better performance by relaxing the conservative estimation guarantee from point-wise state values to the expectation of state values. However, their conservative approaches differ: CSVE directly penalizes out-of-distribution (OOD) states, while CQL and COMBO penalize OOD state-action pairs.\n- In comparison with COMBO, by directly penalizing the OOD states, CSVE obtains the same lower bounds but under a more general state distribution. This offers a more flexible space for algorithm design, which is one of the main reasons for penalizing $V$ rather than $Q$.\n- By controlling the distance of $d$ to the behavior policy's discounted state distribution $d_u$, CSVE has the potential for further performance improvement.\n\nThe bound is provided in the comparison with prior work in Section 3.1. We will refine this comparison in the future version.\n\nConcern 3: Questions about evaluation\n\nWe understand your concerns regarding the comparison of results in Table 1. It is worth mentioning that copying results from prior work is a common practice in the offline literature; for instance, PBRL copy-pasting results from TD3-BC. However, we recognize the importance of ensuring a fair comparison across different setups. We have attempted to reproduce some of the results from previous work, but most of our reproduced results were inferior to those reported in the original studies. For example, we used the source code of COMBO to reproduce their results (Appendix C.4), but we could not achieve the same level of stability as presented in their paper. Therefore, we opted to use the results reported in the original papers for a fair comparison.\n\nRecently, we have conducted additional experiments with seven more seeds on HalfCheetah-medium, HalfCheetah-medium-replay, and HalfCheetah-medium-expert tasks. The updated scores can be found in the attached PDF. The results are consistent with the results reported in the paper. Given the limit on time and available computing resources, we can not reproduce all experiments in this rebuttal window. In the future revision, we will re-run all experiments over 10 seeds and report the results. \n\n## 2. Questions\n\nQ1: issues of OOD actions\n\nCQL learns the conservative Q-values of $Q(s, a)$ on transitions $\\{(s, a, r, s')\\}^N$ of dataset $D$ where the value overestimation can only be introduced by OOD actions on $s'$. We aruge that **this is also a limitation of CQL that hinders further performance improvement**, since it does not learn Q-values on any out-of-sample state $s \\notin D$ even the state is in-distribution with respect to the behaviour policy.\n\n In contrast to adding conservatisim on Q-values as CQL, **CSVE proposes the idea of imposing conservatism on state values and proves that this approach is better in theory.** \n\nThe intuition is that proper exploration on out-of-sample but in-distribution states (i.e., the states near data as in our paper) has benefits on improving the learnt policy compared to the behaviour policy that collected the data. As a comparision, penalizing Q for OOD actions has no guarranttees on OOD states (in CQL), or require assumptions on a penalty coefficient(in COMBO). However, when directly penalizing V function, CSVE get the same lower bounds as COMBO but under more general state distribution (Section 3.1). This means for those OOD actions, we can still get lower bound its value by bounding the state value, thus decreasing the extrapolaation error. We will add this discussion in the updated version. \n\nQ2:  scalability to dataset generated by multiple policies\n\nOur method is indeed scalable to situations where the fixed dataset is generated by multiple policies. Throughout the development of our algorithm, we did not assume that we have access to the policy that generated the dataset. Moreover, in our experimental evaluation, the medium-expert dataset serves as an example of a mixed dataset, as it is generated by both medium-played policy and expert policy.\n\nQ3: Relation to [1]\n\n[1] aims to learn an accurate value function $V{\\pi}$, whereas in our work, we focus on obtaining a lower bound for $E_{s \\sim d}[V{\\pi}(s)]$. In [1], it is necessary to learn the  state-density ratio to ensure a more precise estimation of $V{\\pi}$. In contrast, our approach does not require learning this ratio. Lower bounding $E_{s \\sim d}[V{\\pi}(s)]$ allows us to avoid visiting out-of-distribution states when executing actions.\n\nWe acknowledge that this paper is related to off-policy evaluation, and we will include it in the related work section of our updated version.\n"}, {"review_id": "0e5NKanjzx", "reviewer": "Reviewer_Qqxx", "summary": "This paper proposes Conservative State Value Estimation (CSVE) for offline reinforcement learning, which directly penalizes the V-function on out-of-distribution (OOD) states and guarantees conservative value estimation under specific state distributions. The authors develop a practical actor-critic algorithm based on CSVE and evaluate its performance on classic continual control tasks.", "questions": "I do not have additional questions.", "limitations": "The authors do not discuss about limitations.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "The paper has both theoretical and empirical results.", "weaknesses": "The significance of the proposed method is not clear. To be more specific, compared with the penalization of Q-function for OOD actions of CQL, the advantage of penalizing the V-function for OOD states is not clear. This penalization alone cannot address the core issue of offline RL - overestimation and extrapolation error. Additional policy constraint needs to be incorporated in the proposed method. Direct penalization of the V-function in offline RL does not affect the Q-value of OOD actions (see Eq. 5 and 6). It is Q used in action selection rather than V, so the agent will still choose over-estimated OOD actions.\n \nThe significance of the theories is not clear. The assumption $\\text{supp}~ d \\subseteq \\text{supp}~ d_u$ is very strong and hard to satisfy. Besides, with this assumption satisfied, the algorithm can never penalize the value for OOD states (since $d$ is in-distribution). Theorem 3.2. and Theorem 3.3., that the expected value of the estimated V under $d$ or $d_u$ is a lower bound of the true values, is of little significance. It is still likely that V is severely over-estimated at the states out of $\\text{supp}~d_u$.\n\nThe paper is not well organized.\n\nThe empirical evaluation is not sufficient to support the effectiveness of CSVE's core component - Eq. 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the comments. Nevertheless, we believe that there should be some misunderstanding points that require to clarify or discuss further. As commented by other reviewers, the paper does have clear technical contributions and evaluation. We explain on them as bellow.\n\n## Q1: The significance of CSVE. \n\n> the advantage of penalizing the V-function for OOD states is not clear... This penalization alone cannot address the core issue of offline RL - overestimation and extrapolation error...\n\nResponse: It is true that CSVE alone cannot address the OOD actions and derive a conservative policy. However, CSVE does have advantages by indirectly affect the Q-function and policy. By penalizing values for OOD states,  CSVE reduces  excessive exploration in OOD states while allowing reasonable exploration in those states close to the samples. This effect is generated in the second term of Eq. 5, and transferred to the Q-function via Eq.6.\n\n> Direct penalization of the V-function in offline RL does not affect the Q-value of OOD actions (see Eq. 5 and 6). It is Q used in action selection rather than V, so the agent will still choose over-estimated OOD actions.\n\nResponse: Our methodology in Section 4.2 should address this concern while preserving the advantage of conservative state value estimation. As demonstrated in Eq. 9, our algorithm selects from in-sample actions and near-sample actions that have higher values even under conservative estimation.\n\n## Q2: The significance of theory.\n\n> The issue of assumption $supp \\ d \\subseteq  supp \\ d_u$\n\nResponse: Let me clarify. In the theory section, the $d_u$ refers to the underlying state distribution with the behaviour policy, and thus its support is not only the samples in the dataset but the whole reachable space of states with the policy $u$. With this setting, it should be reasonable to assume $supp \\ d \\subseteq  supp \\ d_u$. Indeed, prior works including CQL, COMBO and most theoretic papers (see Table 1 in [3]) have the same assumption. This assumption is mainly for convenience in theoretic analysis. In practice, it could be satisfied by constraining the exploration in and near the dataset generated by $d_u$. \n\n\n## Q3: The empirical evaluation is not sufficient to support the effectiveness of CSVE's core component - Eq. 5.\n\nResponse: Directly measuring the effect of the components in Eq.5 is hard. Instead, we evaluated its effectiveness comprehensively through controlled experiments of baselines and alternative components of CSVE. The design logics is as bellow.\n- By comparing with CQL-AWR / IQL / AWAC, we assess the value estimation component of CSVE (Eq.5-6) versus CQL / IQL(expectile) / AWAC (normal TD-based value estimation), under the same policy extraction method AWR.\n- By comparing CSVE and the model-free CSVE in Appendix C.6, we verify the benefits of model-based next state sampling over model-free state perturbation used in Eq.5.\n- Ablation study. In Appendix B.1 and 5.2, we evaluate with varying hyper-parameters $\\lambda$ and $\\beta$ to assess how the policy extraction is affected by the conservative state value estimation.\n\n\n**Table R1**: Implementation comparision among different offline RL algorithms.\nAlgorithm | Evalue estimation | Policy improvement |\n--- |:---:|:---:|\nCQL|Q-values with penality on OOD actions|SAC-style Actor|\nCQL-AWR|Q-values with penality on OOD actions|AWR + State exploration|\nIQL|Q-values with expectile regression|AWR|\nAWAC|Normal Q-values|AWR|\nCOMBO|Q-values with penalty on OOD states and actions (via multi-step model rollouts)|SAC-style Actor|\nCSVE (default)|V-values with penalty on OOD states (via 1-step model rollouts) |AWR + State exploration|\nCSVE (Appendix C.6)|V-values with penalty on OOD states (via perturbing in-sample states) |AWR + State exploration|\n\nReferences:\n[1]CQL; [2]COMBO; [3] Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage, 2022."}]

Author rebuttal:
We have included two updated tables in the attached PDF. In these tables, we have highlighted the scores that exceed 90% of the highest score. The average score is also provided for a more comprehensive comparison.

We have conducted additional experiments with seven more seeds on HalfCheetah-medium, HalfCheetah-medium-replay, and HalfCheetah-medium-expert tasks. The updated scores can be found in the attached PDF. The results are consistent with the results reported in the paper. Given the limit on time and available computing resources, we can not reproduce all experiments in this rebuttal window. In the future revision, we will re-run all experiments over 10 seeds and report the results. 

Meta-review: After discussion with reviewers, I am recommending acceptance of this work.

I would suggest the authors incorporate the feedback and suggestions provided by the reviewers in preparation of their camera-ready version.
Decision: Accept (poster)