Effective Offline Reinforcement Learning via Conservative State Value Estimation
Authors: Liting Chen
Jie Yan
Lu Wang
Qingwei Lin
Dongmei Zhang
Saravan Rajmohan
Thomas Moscibroda
Venue: Submitted to ICLR 2023 (2023)
Keywords: Offline Reinforcement Learning

Abstract: Offline RL promises to learn effective policies from static experience datasets without further interaction, which expect to perform well in the online environment. However, it faces up to a major challenge of value over-estimation introduced by the distributional drift between the dataset and the current learned policy, which leads to learning failure in practice. The common approach is to add a penalty term to reward or value estimation in the Bellman iterations, which has given rise to a number of successful algorithms such as CQL. Meanwhile, to avoid extrapolation on unseen states, existing methods focus on conservative Q-function estimation. In this paper, we propose CSVE, a new approach that directly imposes penalty on out-of-distribution states. We prove that for the evaluated policy, our conservative state value estimation satisfies: (1) over the state distribution that samples penalizing states, it lower bounds the true values in expectation, and (2) over the marginal state distribution of data, it is no more than the true values in expectation plus a constant decided by sampling error. Further, we develop a practical actor-critic algorithm in which the critic does the conservative value estimation by additionally sampling and penalizing the states 'around' the dataset, while the actor applies advantage weighted updates to improve the policy. We evaluate in classic continual control tasks of D4RL, showing that our method performs better than the conservative Q-function learning methods (e.g., CQL) and is strongly competitive among recent SOTA methods.

Under review as a conference paper at ICLR 2023
EFFECTIVE OFFLINE REINFORCEMENT LEARNING VIA CONSERVATIVE STATE VALUE ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Offline RL seeks to learn effective policies solely from historical data, which ex- pects to perform well in the online environment. However, it faces major chal- lenge of value over-estimation introduced by the distributional drift between the dataset and the current learned policy, leading learning failure in practice. The common approach is adding a penalty term to reward or value estimation in the Bellman iterations, which has given rise to number of successful algorithms such as CQL. Meanwhile, to avoid extrapolation on unseen states and actions, existing methods focus on conservative Q-function estimation. In this paper, we propose CSVE, new approach that learns conservative V-function via directly imposing penalty on onto---distrbuttinn states. We prove that for the evaluated policy, our conservative state value estimation satisfies: (I) over the state distribution that samples penalizing states, lower bounds the true values in expectation, and (2) over the marginal state distribution of data, it is more than the true values in expectation plus a constant decided by sampling error. Further, we develop a practical actor-critic algorithm in which the critic does the conservative value esti- mation by additionally sampling and penalizing the states around the dataset, and the actor applies advantage weighted updates to improve the policy. We evaluate in classic continual control tasks of D4RL, showing that our method performs bet- ter than the conservative Q-function learning methods (e.g.. CQL) and strongly competitive among recent SOTA methods.
INTRODUCTION
Reinforcement Learning (RL), which learns to act by interacting with the environment, has achieved remarkable success in various tasks. However, in most real applications, it is impossible to learn on- line from seratch as exploration is often risky and unsafe. Instead, offline (Lujimoooto et al. 2019] Lange al. 2012)) avoids this problem by learning the policy solely from historical data. However, the naive approach, which directly uses online RL algorithms to learn from a static dataset, suffers from the problems of value over-estimation and policy extrapolation on OOD (out-of-distribution)) states or actions.
Recently, conservative value estimation, being conservative on states and actions where there are no
enough samples, has been put forward as a principle to effectively solve offline RL (iShi et al. 2022 Kumar et al. 2020 Buckman et al. 2020). Prior methods, e.g., Conservative Q-Learning (CQL Ku-
mar et al. (220)), avoid the value over-estimation problem by systematically underestimating the Q values of OOD actions on the states the dataset. In practice,
often too pessimistic and thus
leads to overly conservative algorithms. COMBO (Yu et al. (022) leverages a learnt dynamic model
to augment data an interpolation way, and then learn Q function that is less conservative than
CQL and derives a better policy in potential.
In this paper, we propose CSVe/Conservative State Value Estimation), a new offline RL approach. Unlike the above traditional methods that estimate conservative values by penalizing Q-function on OOD states or actions, CSVE directly penalizing the V-function on OOD states. We prove in theory that CSVE has tighter bounds true state values than CQL, and same bounds as COMBO but under more general discounted state distributions which leads to more space for algorithm design. Our main contributions are follows.

---

Under review as a conference paper at ICLR 2023
The conservative state value estimation with related theoretical analysis. We prove that it lower bounds the real state values in expectation over any state distribution that is used to sample OOD states, and is up-bounded by the real values in expectation over the marginal state distribution of the dataset plus a constant term depending on only sampling errors. Compared prior work, it has several advantages derive better policy in potential.
A practical Actor-Critic implementation. It approximately estimates the conservative state values in the offline context and improves the policy via advantage weighting updates. In particular, we use a dynamics model to generalize over in-distribution space and sample OOD states that are directly reachable from the dataset.
Experimental evaluation on continuous control tasks of Gym (Brockman et al. 2016) and Adroit (Rajeswaran et al. 2017) in D4RL (Fu et al. 2020) benchmarks, showing that CSVE performs better than prior methods based on conservative Q-value estimation, and is strongly competitive among main SOTA offline RI. algorithms.
PRELIMINARIES
Offline Reinforcement Learning. Consider the Markov Decision Process M := which consists of the state space S, the action space A, the transition model P 8 AS)). the reward function r: S x A + R. the initial state distribution p and the discount factor 7 E (0, S + A(A) takes
action in probability given the current state. A transition is the tuple (8t, Ge. Tt, 8t++1) where ae ~ π x-((s)), 88+1 ~ P(-lse, at) and r2 = r(tr, az). We assume that the reward values satisfy [r(s; a)] Rmarr Vs, a. A trajectory under is the random sequence (So, do, $1, 41, . ST) which consists of continuous transitions starting from so p.
stochastic policy
The standard RL Jx (M) = EM o 7're] via active interaction with the environment M. At any time t, for the policy т, the value function of state is defined as V*(s) = EM, [ ? p and the Q value function is Q" (s, a) == EM, [E Tetklss = s, = a). The Bellman operator function projection: 8" (s,,(s a) == r(s,a) a) +
learn a policy π II that maximize the future cumulative rewards
s),
( a')), or 8 (s) = Eawr(-ls) (r(ss a) + 2'E3' [Vs')]], which leads to iterative value updates. Bell- man consistency implies that (*(s) BV (V"(s,, (s), Vs and Q*(ss) B QT a), Ys, a. In practice with function approximation, we use the empirical Bellman operator 33Th where the former expecta- tions are estimated with data.
The offline RL to learn the policy π from static dataset D as,,,,,,')) consisting of transi- tions collected by any behaviour policy, aiming to behave well in the online environment. Note that, unlike the standard online RI,, offline RI. cannot interact with the environment during learning.
a,
Conservative Value Estimation. One main challenge in offline RL is the over-estimation of val- ues introduced by extrapolation on unseen states and actions, which may make the learned policy collapse. To address this issue, conservatism or pessimism are used in value estimation, e.g. CQL learns a conservative Q-value function by penalizing the value of unseen actions on states:
Qu+1 arg min (Bo-D,awp(a [[((s, a)] B tal (Q(s.
Q
1
2 E. B.... (s.
where åeS and π are the behaviour policy and learnt policy separately, i is any arbitrary policy different from ft., and a the factor for trade-off of conservatism.
Constrained Policy Optimization. To address the issues of distribution drift between learning policy and behaviour policy, one approach is to constrain the learning policy close to the behaviour
policy (Bai al., 2021 | Wu al. 2209: Nair et al. 22200 Levine et 2020 Fujimoto al., 2019).
Here we take Advantage Weighted Regression(Peng et al. (2019b); Nair et al. (2020)) which adopis an implicit KL divergence to constrain the distance of policies as example:
k+1 <- arg max E, Eg log (ags(a)s)
1
exp ( ((r((s)))
(2)
Z(s)
2

---

Under review as a conference paper at ICLR 2023
where A= is the advantage of policy x², and Z the normalization constant for
Model-based Offline RL. In RL, the model is an approximation of the MDP M. We denote model as M = (S, A, P, p. 7), where P and f are approximations of and respectively. In the setting of offline RL, the model is used roll out and augment data (Yu et al. [020 (02T] or act as
a surrogate of real environment to interact with agent (Kidambi et al., (020). In this paper, we use model sample the next states that are approximately reachable from the dataset.
CONSERVATIVE STATE VALUE ESTIMATION
In the offline setting, the value overestimation is a major problem resulting in failure of learning reasonable policy (Levine al. 2020 Fujimoto et al. 2019). In contrast to prior (umas et al. 2020: Yu et al., 2 that get conservative value estimation via penalizing Q function for OOD state-action pairs, we directly penalize V function for OOD states. Our approach provides several novel theoretic results that allow better trade-off of conservative value estimation and policy improvement. All proofs of our theorems can be found in Appendix A
3.1 CONSERVATIVE OFF-POLICY EVALUATION
Our approach an alternative approach to servative Q function, aim to conservatively estimate the value (*(s) of target policy given a dataset /) to avoid overestimation of out-o--distributtin states. To achieve this, we penalize the V-values evaluated on states that is more likely to be out-of-distribttion and pushing up the V-values on states that is in the distribution of the dataset, which is achieved through the following iteration:
et al., 2020 Instead of learning a con-
p+++1 < arg min 1 Eand, (3) (s) (B3 v
((s) + -
V(5') Esdd, (s) V())
(3)
where (s) is the discounted state distribution of D. d(s) is any state distribution, and 8 is the empirical Bellman operator (see appendix for more details). Considering the setting without function approximation, by setting the derivative of Eq. B as zero, the V function found by approximate dynamic programming in iteration & can be obtained:
al d(s) ds(s)
prk+1 B*
1]. Vs
Denote the function projection 1/r in Eq. 4 as T We have Lemma and thus ÿ converges to a unique fixed point.
Lemma 1. For any with supp S supp T is y-contraction in Loc norm.
Theorem 1. For any d with supp d S suppd, (d * du), with sufficiently large (i.e., 2 Co ) Enwn(als)) (2-1)//D(s.aa)) /Es-d(s) ds(s) 11)) the expected value of the estimation 1/a (s)
d(s)
under d(s) the lower bound of the true value, that is: Rawd() (V t (s)] S [V" (s)].
(T*(s) = 1im (s) the converged value estimation with the dataset D, and C
(1-1)//D(eas D(e.a))
related to sampling error introduced by the use empirical rather than Bellman operator. If the counts of each state-action pair is greater than zero, (D(s,, a)] denotes a vector of size |8]]|A] containing counts for each state-action pair. If the counts of this state action pair is zero, the corresponding is large but finite value. We assume that with probability 2 8, the sampling error
V[D(s.a]
less than C while Cr.,5 5 is a constant (See appendix for more details.) Note that if the (1-7) (I-7)/(D(s.a)
sampling error is ignorable, a > 0 can guarantee the lower bound results.
Theorem 2. The expected value of the estimation 8T under the state distribution of the original dataset is the lower bound of the true value plus the term of irreducible sampling error, that is: C.,,, B
Egdd (s) [s)) < (V) ( Exdd. P -
- Bawn(als) (1-7) (D((sa)
3

---

Under review as a conference paper at ICLR 2023
where p refers to the transition matrix coupled with policy (see Appendix for details).
Now we show that, during iterations, the gap between the value in-distribution state and out-of- distribution state in the estimated V-function is higher than in the true V-functions.
Theorem 3. At any iteration k. with a large enough a. our method expands the difference in expected V-values under the chosen state distribution and the dataset state distribution, that is: Exdd,(s) (s-d,,s). [s)]
(V -B-dt > (snd, (V)lV [V
In the policy extraction part, this property enables our policy to take actions a in state s(s ~ D) that remains in distribution instead of out of distribution, given that our estimated V-function does not overestimate the erroneous out-of-distribution states compared to the in-distribution states.
Now we present four remarks explain how the above theorems guide applications of Eq. in offline RL algorithms.
Remark 1. In Eq. 3 if d = du, the penalty out-of-distribution states degenerates, which means that the policy should not reach states with low support in data, and consequently never explore the unseen actions at the state. Indeed, AWAC Nair et al. (2020) adopts this setting. We show that with proper choice of d different from du. our method performs better than AWAC in practice.
Remark Theorem@limpian iimpless that under du, the marginal state distribution of data, the expectation estimated value of should either be lower than the true value, or higher than the true value but within a threshold. This fact motivates our advantage weighted policy update method in Eq. []]
Remark 3. Theorem Implies that under d, say the discounted state distribution of any policy, the expectation estimated value of should lower bounds the true value. This fact motivates our policy improvement method of unifying advantage weighted update with a bonus exploration in Eq. 12
Remark 4.
Theorem states
[V* (s)] (V> >
(x (v))
By (s) [s)]. That is to say, under the distribution d, the amount of value under-estimation in
expectation is larger than that of behaviour policy du. With proper choice of d, it is safe and effee- tive to derive a new and potentially better policy with 1i Our algorithm choose the distribution of
model predictive next-states as d, i.e., s' ~ is implemented by ~ D.a T(-1s), s P(- a),
which effectively builds a soft 'river' with low values around the dataset.
Comparison with prior work: CQL [E which penalizes Q-function of OOD actions on
states in history data, guarantees the lower bounds on state-wise value estimation: (**(s)
V=(s) for all E D. COMBO, which penalizes Q- function of OOD states and actions of an interpolation of history data and model-based roll-outs,
Ex(al-j (s,(ss))))))))))))))) < (QT
guarantees the lower bound of state value expectation: Egppo [VT (s)] < E [ [V*(s)] where μo is the initial state distribution (Remark I, section A.2 of COMBO Yu al. (222I); which is spe-
cial case of our result in Theorem T when = Although both CSVE and COMBO intend to
get better performance by releasing conservative estimation guarantee from the state-wise values to expectation of state values, CSVE get the same lower bounds but under more general state distri- bution. This provide more flexible space for algorithm design, and is also one main reason of penalizing on V rather than Q. By controlling distance of d to the behaviour policy's 'sicys discounted state distribution d3, CSVE has the potential of more performance improvement.
Note that bounding [s]] rather than state-wise (s).. would introduce more adventurous pol- icy, which would achieves better performance in in-iistribution states and have more risk behaviors in OOD states. To deal with that limitation, we introduce deep ensemble dynamic model to sample the OOD states for better estimation.
3.2 SAFE POLICY IMPROVEMENT GUARANTEES
Following prior works Laroche et al. 2019 Kumar al. 2020); Yu et al. (2021)), show that our method has the safe policy improvement guarantees against the data-implied behaviour policy. We first show that method optimizes a penalized RI. empirical objective:
4

---

Under review as a conference paper at ICLR 2023
Theorem 4. Let v be the fixed point of Equation 3, then x² r*(ass) arg max V*(s) is equiva- lently obtained by solving:
d(s) 1]
(als) t arg max (T, 01 E Bond (x) (s((s)
(5)
Building upon Theorem we show that our method provides a Ç-safe policy improvement over m
Theorem S. Let कt (a|s) be the policy obtained in Equation 5 Then, it is (-safe policy improve-
ment over H" in the actual MDP M, i.e., [TT", where Ç is given by:
, M) 2 J( M) - C with high probability 1- 6,
Ç < (
Crd
Rmar Cr (1-1)²
Bondy (e)
VIAL viD(s)] v Euunr(alo) m(ale)
(
(J(x" M) J J(
(6)
2u Bur-yB Bonay (0) dosla
d(e)
4 METHODOLOGY
In this section, we propose a practical Actor-Critic method for computing conservative value estima- tion function by approximately solving Equation Jand and taking advantage weighted policy updates. is mainly motivated by the theoretic results, as explained by the four remarks in section 3.1 Besides, the full algorithm of deep learning implementation presented in Appendix B
4.1 CONSERVATIVE VALUE ESTIMATION
Given the access to dataset D collected by some behaviour policy wg, our aim is to estimate the value function V for a target policy æ. As stated in section 3 to prevent the value overestimation, we instead learn a conservative value function a/ that lower bounds the real values of x by adding penalty
states into the flow of Bellman projections. Our method consists of
the following iterative updates of Equations 7 where Q² is the target network of Q*.
p+++1 + arg min L (V:
= a (
[V(')) Eg D D(((s))
(7)
+ EAND
v(s))
@u+1 + mig LQ vk+l)= = EEDDSND ( Q(s.a))
(8)
Q
gu+z t with + (1 @)
The RHS of Eq. is an approximation Eq. where the first term gives out-of- distribbbbtoonn states a penalty, and the second term follows the defimition of V values and Q values. In Eq. 8 the RHS is TD errors estimated on transitions in the dataset D. Note that the target term here uses the sum of the immediate reward rss, a) and the next step state's value (s'+ Eq. the target Q values are updated with a soft interpolation factor w (0, Q+ changes slower than Q*, which makes the TD error estimation in Eq. 7 more stable.
Constrained policy. Note that in RHS of Eq. we use a T(-ss) in expectation. To safely estimate the target value of V(s) by En-x(-Ia) ass, almost always requires supp(s(-Is)) c supp(TT (+))). We achieves this by the advantage weighted policy update, which forces л. (1- have significant probability mass on actions taken by xg in data, as detailed in section 3.2
Model-based OOD state sampling. In Eq. 7 we implement the state sampling process s' d in Eq. flow of {s D; T(a s' (ss'|ss, a)), that is the distribution of the predictive next-states from D by following л. is beneficial in practice. On one hand, this method is efficient sample only the states that are approximately reachable from D by one step, rather than sample
5

---

Under review as a conference paper at ICLR 2023
the whole state space. On the other hand, only need the model to do one-step prediction such that no bootstrapped errors due to long horizon are introduced. Following previous work (Janner et al, 2019 Yu et al., 2020; 2021), implement the probabilistic dynamics model using ensemble of deep neural networks {po' ......pb) b. Each neural network produces a Gaussian distribution over at)).
the next state and reward: P)((s+ 8/8 a+) (1
Adaptive penalty factor a. The pessimism level is controlled by the parameter & 2 0. In practice, set adaptive during training as follows, which is similar as that in CQL al. (2020))
max max(aa(Ea (s)) E V -)) a20
(10)
where is budget parameter. If the expected difference in V-values is less than т, a will decrease. Otherwise, will increase and penalize the out of distribution state values more aggressively.
Discussion: As stated in former sections, our method focuses on estimating conservative state value for learning policy. The effectiveness of adding conservatism on V function are two folds. First, penalizing V values is with a smaller hypothesis space than penalizing Q, which would reduce the computation complexity. Second, penalizing V values can achieve more relaxed lower bound than penalizing Q with ignoring the explicitly marginalization on Q values. A more relaxed lower bound guarantees more opportunities achieving better policy.
4.2 ADVANTAGE WEIGHTED POLICY UPDATES
After learning the conservative p*++1 and Qu+1 (or Vi and Q when converged), we improve the policy by the following advantage weighted policy update (Nair et al. 2020).
π + arg min (TT') = - -EEsanD [og (als)) exp (BAat) (s.a))
(1I)
where (s, = (s, (V)t(ss)
Eq. 11 updates the policy π to amounts of weighted maximum likelihood which are computed by re-weighting state-action samples in D with estimated advantage A As discussed in the AWAC (Nair et al., (020). this method avoids explicitly estimating the behaviour policy and its resulted sampling errors which is an import issue in the offline RL setting (Kumar et al. 2020).
Implicit policy constraints. We adopt the advantage weighted policy updates which imposes an implicit KL divergence constraints between π and TT. This policy constraint is necessary to guaran- tee that the next state s' in Equation 7 can be safely generated through policy æ. As derived Nair et al. (2020) (Appendix A), the Eq. 11 is an parametric solution of the following problem:
s.t. DKL(T'((ss) (p({ss) S,, J. <'((as)dd==1 =
max MaxeEaxre (-1s) (1)) [A ss, a)]
Note that DKL (TT' TT)) an reserve KL divergence with respect to <', which is mode-secking (Shlens 2014)). When treated as Lagrangian it forces n' allocate its probability mass to the max- imum likelihood supports of TT. re-weighted by the estimated advantage. In other words, for the space of A where (=(+ss) has no samples, x x'(+ss has almost zero probability mass too.
Bonus of Exploration on Near States. suggested by remarks Section 3.11 in practice allowing the policy explore the predicated next states transition (s ~ D) following a ~ x (-1s)) leads to better test performance. With this kind of exploration, the policy is updated as follows.
π arg arg min (*'') L
L=(x') AE D.
(r(s.o) + (
(12)
The second term is an approximation Egnd (s)], while the first term is the approximation of Ex-dd (s) (s)]. While the choice of À is ultimately just hyper- param-errreeeer balance between optimistic policy optimization (in maximizing V) and constrained policy update (the first term) by adjusting X.

---

Under review as a conference paper at ICLR 2023
AWAC
CQL
CОMBО
TD3-BC
PHRL
CSVE (ours)
Random
HalfCheetah
12.1
17.5 1.5
38.8
11.0111
1.21
26.7 2.0
Hopper
8.7
7.9 0.4
17.9
0.6
31.6 0.3
2..0
Walker2D
2.2
5.1 1.3
7.0
1.6
8.8 - 6.3
6.1 + 0.8
Medium
HallCheetah
50.0
0.5
54.2
47.4
48.3 1: 0.3
58.2 1: 1.5
48.6 0.0
Hopper
96.0
53.0 = 28.5
94.9
66.3
59.3 + 4.2
81.6 + 14.5
99.4 5.3
Walker2D
89.1
73.3 17.7
75.5
78.3
83.7 + 2.1
90.3 + 1.2
82.5 1.5
Medium Replay
HalfCheetah
43.4
45.5 0.7
55.1
44.2
44.6 + 0.5
49.5 + 0.8
54.8 + 0.8
Hopper
97.8
88.7 12.9
73.1
94.7
60.9 + 18.8
100.7 + 0.4
91.7 + 0.3
Walker21)
80.0
81.8 +27
56.0
73.9
81.8 + 5.5
86.2 + +.4
78.5 1.8
Medium Expert
HalfChectah
62.8
75.6 + 25.7
90.0
86.7
90.7 + 4.3
93.6 + 2.3
93.1 + 0.3
Hopper
87.2
105.6 12.9
111.1
91.5
98.0 + 9.4
111.2 + 0.7
95.2 3.8
Walker2D
73.7
107.9 + 1.6
109.6
110.1 0.5
109.8 + 0.2
109.0 0.1
Expert
HallCheetah
10.3
96.3 1.3
96.7 + 1.1
96.2 + 2.3
93.8 = 0.1
Hopper
111.6
28.0
107.8 +
110.4 1 0.3
111.2 0.6
Walker2D
110.6
108.5 + 0.5
110.2 0.3
108.8 + 0.2
108.5 + 0.0
Table 1: Performance comparison on Gym control tasks. The results of CSVE over three seeds and we reimplement AWAC using d3rlpy. Results of IQL, TD3-BC, and PBRL are from their original paper Table in Kostrikov et al. (2021b), Table C.3 [Fujimoto Gu (202T), and Table in Bai et al. (2021) respectively. Results of COMBO are from the reproduction result in Rigter ct al. (2022) (Table I), given that the original paper report their results on vO dataset. For the same reason, results of CQL are from Bai et al. (2021).)
5 EXPERIMENTS
The primary goal of this section is investigate whether the proposed tighter conservative value estimation leads to performance improvement. Besides, we would like to ascertain when further exploration has benefits and how well CSVE performs compared with SOTA algorithms.
We evaluate our method on classical continuous control tasks of Gym Brockman et al., 2016 and Adroit(Rajeswaran et al.) 2017) in the standard D4RL. (Fu et al. (0220) benchmark. The Gym control tasks include HalíCHeetah, Hopper and Walker2D, each with 5 datasets collected by follow- ing different types of policies (random, medium, medium-replay, medium-expert, and expert). The Adroid tasks include Pen, Hammer, Door and Relocate, each with 3 dataset collected by different policies (human, cloned, and expert).
Our method, namely CSVE, and the compared basclines CQL Kumar al. 2020), COMBO Yu ct al. 202I), AWr et al. (2020), PBRL (Bai et al. 202I) and other SOTA algorithms TD3- BC & Gu 2021), UWAC(Wu et al. 2021). IQL Kostrikko et al., 2021b), (Emaa et al. 2019)) whose performance results are public or have high-quality open implementations. CQL which estimates the conservative Q values on state-action pairs rather than states, the direct comparing method to ours. COMBO also lower bounds the estimated V function. AWAC is one special case of our Eq. when dy. PBRL a very strong performant in offline RL, but is quite costly computation since it uses the ensemble of hundreds of sub- uub-models.
5.1 OVERALL PERFORMANCE
We first test on the Gym control tasks. We train our methods for 1 million steps and report the final evaluation performance. The overall results are shown in Table Compared to CQL, our method has better performance on 11 of 15 tasks and similar performance our method shows consistent advantage on the datasets that generated by following random or sub- optimal policies (random and medium). Compared AWAC, method has better performance on tasks and comparable performance on others, which demonstrates the effect of our fur- ther exploration beyond cloning the hehaviour policy. In particular, method shows obvious
others. In particular,

---

Under review as a conference paper at ICLR 2023
BC
BEAR
UWAC
CQL.
PERI.
CSVE (ours)
Human
Pen
34.4
-1.0
10.1 3.2
37.5
71.5
3..4
105.0 16.1
Hammer
1.5
0.3
0.7
4.4
1.4
0.4
3.2
Door
0.5
0.3
0.2
9.9
4.3
0.1 0.0
3.1 2.1
Relocate
0.0
-0.3
0.0 1.0+0.0
0.2
0.1
0.0 + 0.0
0.1 0.0
Cloned
Pen
56.9
26.5
6.9
39.2
37.3
74.9 9.8
55.2 - 6.1
Hammer
0.8
0.3
0.0
2.1
2.1
0.8 = 0.5
0.5 0.2
Door
-0.1
-0.1
0.0 0.0
0.4
1.6
4.6 - 4.8
1.3 1.1
Relocate
-0.3
-0.
0.0
0.3 1 0.0
Expert
Pen
85.1
105.9
98.2 + 9.1
107.0
135.7 3.4
142.9
10.6
Hammer
125.6
127.3
107.7 = 21.7
86.7
127.5 0.2
126.5 = 0.4
Door
34.9
103.4
104.7 0.4
101.5
95.7 = 12.2
104.3 + 0.9
Relocate
101.3
98.6
105.5 + 3.2
95.0
84.5 + 12.2
103.0 +
Table 2: Performance comparison on Adroit tasks. The results of CSVE are over three seeds.
Results of IQL from Table in Kostrikov et al. (2021b) and results of other algorithm are from Table 4 in Bai et al. (202I).
advantage in extrating the best policy on data of mixed policy (Medium Expert) while AWAC can not. Compared to COMBO, our method has better performance on out 12 tasks and comparable performance or slightly worse others, which demonstrates the effect of our better bounds on V. In particular, our method shows an obvious advantage in extrating the best policy on medium and medium-expert tasks. In tasks evaluated, our method gets higher score than IQL in of them, and has similar performance in the other tasks. Finally, our method performs close to PBRL, even PBRL has almost orders of more model capacity and computation cost.
We now evaluate our method on the Adroit tasks. For CSVE, we report the final evaluation results after training in 0.1 million steps. The full results reported in Table Copared IQI., method performs better in 8 out of 12 tasks, and performs similarly in the other 4 tasks. For the expert datasets, all methods including simple BC (behaviour cloning) can perform well, among which ours is the most competitive on all four tasks. For human and cloned datasets, almost all methods fail learn effective policies on three tasks except the Pen task. For the Pen task, CSVE is the only one that succeeds to learn a good policy on the human dataset, while it can learn medium policy on the cloned dataset as BC and PBRL.
5.2 SENSITIVENESS OF HYPER-PARE PARAMETERS
We anaylyze apper-parameter 3, which trades off between behaviour cloning and policy optimiza- tion. For smaller values, the objective behaves similarly to behavior cloning (weights are close for all actions), while for larger values, attempts to recover the maximum of the Q-function. To quantita- tively analyze its effect, datasets, whose results are shown in Fig. I
different from (0.1, 10} in mujoco tasks with the medium-type We can see that has effect on the policy performance during training. Empirically, we found out that in general.
3.0 is suitable for such medium type datasets. Besides, in practice, by default we use & = 3.0 for random and medium task while 0.1 for medium-replay, medium-expert and expert datasets.
RELATED WORK
Offline RL (Fujimoto et al. 2019 Levine et al. 2020) aims to learn reasonable policy from static dataset collected by arbitrary policies, without further interactions with the environment. Compared to interactive RL, offline RL suffers two critical inherent issues, i.e., the distribution drift introduced by off- policy learning and the out-of-distribution extrapolation value estimation (Ostrovski et al.) 2021 Levine et al.) 2020). The common mind of offline RL algorithms to incorporate conser- vatism or regularization into the online RL algorithms. Here we briefly review the prior work with comparison to ours.
8

---

Under review as a conference paper at ICLR 2023
0300
4000
3000
Robum
2500
3000
2000
2030
1800
530
500
42.0
500
420
(a) halfcheetah-meedinn-y
(b) hooper-meddumn-22
(c) calker2ddmediim-
Figure The effect of 3 in medium tasks.
Conservative value estimation: Prior offline RL algorithms regularize the learning policy close the data or explicitly estimated behaviour policy) and penalize the exploration the out-of- distribution region, via distribution correction estimation (Dai et al. [0200] Yang et al. [2020), policy constraints with support matching (Wu al.] 2019) and distributional matching Fujimoto et al.
(2019; Kumar et al. (2019). applying policy divergence based penalty on Q-functions (Kostrikov al. 2021a Wang al. 2020) or uncertainty- based penalty (Agarwal et al. 2020) on Q-functions and conservative Q-function estimation (Kumar et al. 2020). Besides, model-based algorithms (Yu jet al., 2020) directly estimate dynamics uncertainty and translated it into reward penalty. Different from these prior work that imposes conservatism state-action pairs actions, directly does such conservative estimation on states and requires no explicit uncertainty quantification.
With learned conservative value estimation, offline policy can be learned via implicit derivation from a state-action joint distribution or Q-Learning and actor-critic framework. In this paper, our implementation adopts the method proposed in AWAC (Nair al. 2020 Peng al. 2019a).
Model-based algorithms: Model-based offline RL learns the dynamics model from the static dataset and uses to quantify uncertainty (Yu et al. 2020), data augmentention (Yu et al., 2021) with roll-outs, or planning (Kidambi et al. 2020 Chen et al. 2021). Such methods typically rely on wide data coverage when planning and data augmentation with roll-outs, and low model estimation егror when estimating uncertainty. which is often difficult to satisfy in reality and leads to policy instability. Instead, we use the model to sample the next-step states only reachable from data, which has no such strict requirements on data coverage or model bias.
Theoretical results: Our theoretical results are derived from conservative Q-value estimation (CQL) and safe policy improvement (Laroche et al. (201)). Besides, COMBO (Yu et al. (021) gives a result conservative but tighter value estimation than CQL, when dataset is augmented with model-based roll-outs. Compared to our result, COMBO's lower bounds additionally assume same initial state distribution which may not always satisfy continuous control.
7 DISCUSSION
In this paper, we propose new approach for offline RL based on conservative value estimation on states and discussed how the theoretical results could lead to the new RL algorithms. In particular, we developed practical actor-critic algorithm, in which the critic does conservative state value estimation by incorporating the penalty of the model predictive next-states into Bellman iterations, and the actor does the advantage weighted policy updates with a bonus of exploring states with conservative values. Experimental evaluation shows that method performs better than alternative methods based on conservative Q-function estimation and is competitive among the SOTA methods, confirming our theoretical analysis well. Moving forward, we hope to explore the design of more powerful algorithms based on conservative state value estimation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
Chenjia Bai, Lingxiao Wang. Zhuoran Yang. Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- ran Wang. Pessimistic bootstrapping for uncertainty-drriven offline reinforcement learning. In International Conference Learning Representations, 2021.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint wroXiv:1006.0155 2016.
Jacob Buckman, Carles Gelada, and Marc G Bellemare. The importance of pessimism in fixed- dataset policy optimization. In International Conference on Learning Representations, 2020.
Xiong-Hui Chen, Yang Yu, Qingyang Li, Fan-Ming Luo, Zhiwei Qin, Wenjie Shang, and Jieping Ye. Offline model-based adaptable policy learning. Advances in Neural Information Processing Systems, 34:8432-8443, 2021.
Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off-policy confidence interval estimation. Advances in neural information processing systems, 33: 9398-9411 9411, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning- Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 2062-2062. PMLR, 09-15 Jun 2019. URI. https: http::e//proceeedii ngs mmlr.
press/v97 http://99//jjiimooooooo html
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810 21803, 2020.
Iya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021a.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference Learning Representations, 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy q- learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 1191199-1 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International Conference on Machine Learning, pp. 3652-3661. PMLR, 2019.
10

---

Under review as a conference paper at ICLR 2023
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tu- torial, review, and perspectives on open problems, 2020. URL httpp://aaxii..ortt./t/. abs/ 2005.01644
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online re- inforcement learning with offline datasets, 2020. URL httpp::/arr//..or//abb////
09359
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances in Neural Information Processing Systems, 34:23283-23295, 2021.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. wevintage-weighted regression: Simple and scalable off-policy reinforcement learning. CoRR, abs/1910.00177,, 2019a. URL
00177
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintage-weighted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint 1rXiv:1910.00117,, 2019b.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint 1rXiv: 10087, 2017.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in Neural Information Processing Systems, 2022.
Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library. arXiv preprint arXiv:2111.03786 2021.
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 19967-20025. PMLR, 17-23 Jul 2022. URL /ttpps://proeeeedingg.tt mlrr.
press v1 62 sh 22c. html
Jonathon Shlens. Notes on kullback-leibler divergence and likelihood. CoRR, abs//440.22000, 2014.
URL http:ttp:///xxxw.org//a/aaa/
22000
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulechre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7777, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning-
arXiv preprint arXiv: /911.1 1361, 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.081440 2021.
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33:6551-6561 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 11144129-44444 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-225 28967, 2021.

---

Under review as a conference paper at ICLR 2023
A PROOFS
We first redefine notation for clarity and then provide the proofs of the results in the main paper.
Notation. Let l E N denote an iteration of policy evaluation(in Section 3.2). Vk denotes the true, tabular (or functional) V-function iterate in the MDP, without any correction. /* denotes the approximate tabular (or functional) V-function iterate.
The empirical Bellman operator can be expressed follows:
(Ba V)){s (a-x(a)) - Ee [)(
(13)
where i(s,, a) is the empirical average reward obtained in the dataset when performing action a at state 8. The true Bellman operator can be expressed as follows:
(B" (*)))(s) (es - P(s (V
(14)
Now we first prove that the iteration in Eq 3 has a fixed point. Assume state value function is lower
bounded, i.e. (s)) C Vs S, then Eq can always be solved with Eq Thus, we only need to
investigate the iteration in Eg
Denote the iteration as a function operator T V. Suppose supp d S sup then the operator T is a y-contraction in Lxs norm where y is the discounting factor.
Proof of Lemma 1 Let V and V are any two state value functions with the same support, i.c.,
suppV = suppl'
1T -TV (s) al del 1)) (B= al d(s) (s(
11) d,))s
VS) 13 V (s)
[((aan((e)))((s + F Fa Σ ((5))s,)VV(6')
(() + ) Eawa(als)
Ea-z(als)
TT - -T= = max -VVV-TV''(a
max Eanz(als) Σ
P(s)). - -
SyEawn(als)
P(s'ls.a) 8"
- ((s"'-'''s))
=y -Vs"")-((s"") V(("))
->|(V V
We present the bound on using empirical Bellman operator compared to the true Bellman operator. Following previous work Kumar et al. (2020), make the following assumptions that: par is the transition matrix coupled with policy, specifically, P" (*)(s)) = Fas P( (V('))
Assumption 1. Vs,a a M, the following relationships hold with at least (1 6) (0, probability,
Cr. a)
Cp.
r |r-r(s,a))
||P(s'|s, P(s'ss
(15)
VID(s,a)
12

---

Under review as a conference paper at ICLR 2023
Under this assumption, the absolute difference between the empirical Bellman operator and the actual one can be calculated as follows:
((B))D -080-(B")/)) - - r(s,a) + E (aenttalls P(s (,a)))(((())
(16)
s F Σ (a - p(s')s,
())
(17)
Or + + 7)
S Eaws(ols)
(18)
())
Thus, the estimation error due to sampling error can be bounded by a constant as a function of Cr.6 and Ct,b. We define this constant as Cr.T.A.
Thus we obtain:
Cras
vV, E D, (*) - - V(s) & Eanz(als) (1 (1-7)v/D(ssp)
(19)
Next we provide an important lemma.
Lemma 2. (Interpolation Lemma) For any f E [0. 1). and any given distribution p(s), let dy be an f-interpolation of and D, i.e.dy(s) fd(s) + (1 f)p(s), let v(p,)) Eampls) [(a))_ dy(s)
then v(p, f) satisfies f) O.
The proof can be found in Yu al. (2021). By setting 1, we have E
pps)) ((s))d(s)) 0.
d(s)
Proof of Theorem 1 The V function of approximate dynamic programming in iteration l can be obtained as:
(* = B>>>(ss - al d(s) - 1] Vss k
(20)
(s)
The fixed point:
d(s) du(s) 1] (*) + E Fant(als)
Cr.t,6 R
d(s)
(*((s) B B*
(1 7) D(s,a)
du(s)
(21)
Thus obtain:
Cr Rmaa a(I (PTI-7pa--1 d(s) 1] (22)
Bann(ale)
(1 (D(s
ds(())
where pD is the transition matrix coupled with the policy and P=V(s) = (e
Then the expectation of V (s) under distribution d(s) satisfies:
Cra.sRmex
Eandlee VT((s) S (Eendis)(V + Esnd( 7P=7 Eawn(ale) (1 (1-7) (1- a()
p -1 d(s)
(23)
-a Esnd(a) p
1))
ds((s)
20
Cre.8Rmaz
Bandie, (1 >)/lll E
When a 2
Exdd(a) (7( < Ex-d(y) (V)
E Ex du(a) -1]) -1))
d(o)
13

---

Under review as a conference paper at ICLR 2023
Proof of Theorem 2
The expectation of (+(s) under distribution d(s) satisfies:
Cr.t,6 Rmax
(xnde()) ( (s) SEanda (x)
(s)) (s) + Es-d_ (a) - -
Eama(ass 8) (1 - 71-7) (s),
(24)
1 d(s)
(Fsndd(s) -
-1])
ds(s)
Noticed that the last term:
dr(s)
11 = Σ ds(s)) dy(s)
Σ
-1) d ddu(s)= (25)
andd du(s)
ds(s)
We obtain that:
Cra
Es-de(t)V S Eand (e + - - Bawa(ale)
(26)
M
Proof of Theorem
3 Recall that the expression of the V-function iterate is given by:
p (s) Bx* (s) a d(s)
1| K
(27)
(s)
Now the expectation of V (s) under distribution (s) is given by:
(s) B V*()) - cx d(s)
Eendu ( (8) = (s)
1| Eand B (1k(s)
(28)
(8)
The expectation of (s) under distribution d(s) is given by:
Egnd(ss (s) = Esnd(s) B=* (*))s) -a( d(s) 1] = E Exdd(s) B (al))-aBBndll du(s)
d(s)
ds((s)
(29)
Thus we can show that:
Egwdy (s) (s)+ - Eand(a) (x) = E Eawdy (s) Ba (*) (*))s) - Esnd(a) ( + a aBend(s) d(s))
du(s)
= Exndy (s) (s) - EsSd((s) ()+t(()) Esnd(s) [B" (5 Vk]
d(s)
+ E Eand (B) (B* - Vk]] + aBoud(a) ds(()
1]
(30)
By choosing a:
E Fend (B5* (v+ -bb)] - Eand, (8) B55* (f [by
(31)
E Eand(s))
d(s)
[.(
We have Fandi (8) pk+ Fendlss) +++ (8) Bond (5) Vk+l(s) (8) Eend(t) (x) hold.
Proof of Theorem 4 V is obtained by solving the recursive Bellman fixed point equation in the
empirical MDP, with an altered reward, r(s, a) a(a.() 1). hence the optimal policy x² (als)
d(e)
obtained by optimizing the value under Eq.
Proof of Theorem 5} The proof of this statement is divided into two parts. We first relates the return of n7" in the empirical MDP M with the return of m can get:
d(s)) ds) 1] JTT M) 0= J[ M)
[a",, M)
E
(32)
The next step is to bound the difference between J[ M) and J( M) and the difference be- tween [n=,, M) and J(x", M). We quote useful lemma from Kumar et al. 2020) (Lemma D.4.1):
14

---

Under review as a conference paper at ICLR 2023
Lemma 3. For any MDP M, empirical MDP M generated by sampling actions according the behavior policy Tg and a given policy n,
Cra 1- + ABmoa Cr.8 <
VIA
π <aass) TT(aas))
}E Bo (s)l
Banalalo)
)]
(33)
[ID(s]
Setting in the above lemma Tg, we get:
Cr,8
Rmmar CT,6 }
[A /D(s)) V
(als))
[J M) J(T,, - 1-7 (1 (1-
snud ( (o)l
E Bans (als)
)
(a(a)s)
(34)
.given that V Baws (a(s)
(s)ss) sa(a/s)
a pointwise upper bound of Farns (a)a) Ta(als)
(a(a)))
Kumar et al.
2020 Thus we get,
Cr, --9 + y/mma Cr.
A [D(s)]
J( M) 2 J(TT,
)E
(als)
)
(1 7)2
To(aas)
1
d(s)
+ E Esndy (8)
11
st
dx((s)
(35)
which completes the proof.
Here, the second term is sampling error which occurs due to mismatch of M and M; the third term denotes the increase in policy performance due CSVE in M. Note that when the first term is small, the smaller value of a are able to provide an improvement compared to the behavior policy.
B CSVE ALGORITHM
Now put all in section together and describe the practical deep offline reinforcement learning algorithm. In particular, the dynamic model model, value functions and policy are all parameterized with deep neural networks and trained via stochastic gradient decent methods. The pseudo code is given in Alg. 1
Algorithm 1: CSVE Algorithm
Input : Data D = [88 8')
Parameters: Qo. Vu. To. Qg. Mr
Hyperparameters: X, learning rates r7e, Tle Nlw begin
// Train transition model with the static dataset D M, (D);
// Train the conservative value and policy functions
Initialize function parameters 0o to, do. 0o 0o:
foreach step k = + N do
dx f - - The LT (Vig Q.,); Op - 0z- No V 1 (Qo: Von)):
Φ Φk-1 no V L (T));
Os f w x- + w 0k:
C IMPLEMENTATION DETAIL.
We implement method based offline deep reinforcement learning library d3rlpy (Seno &
Imai 2021). The code is available at
The detailed hyper-parameters provided in Table
15

---

Under review as a conference paper at ICLR 2023
Hyper-parameters
Value and description
5, number of ensembles in dynamics model
10, to control the penalty of out-of out-of-disttibution states
10, budget parameter in Eq. 10
In Gym domain, 3 for random and medium tasks, 0.1 for the other tasks; In Adroit domain, 30 for human and cloned tasks, 0.01 for expert tasks 0.99, discount factor.
million for Mujoco while 0.1 million for Adroit tasks. 0.005, target network smoothing coefficient.
actor Ir critic
3e-4, policy learning rale
le-4, critic learning rate
Table 3: Hyer-parameters
D EXTENDED EXPERIMENTAL RESULTS
D.I MORE EXPERIMENTS ON HYPER--PA SPPER-PARAMETERS EFFECT
We also investigated A values of {0.0, 0.1, 0.5, 1.0] the medium tasks. The results are shown in Fig.
4000
3000
5000
2590
3000
4000
2002
2000
2000
1500
1000
590
500
620
203
630
2000
202
6:0
grandient stepsiuussaadds
grandiere stopott
grondlent stapott tessanss)
(a) halfcheetah- mediium
(b) hopper-medium
(c) walker2d-medium
40
50
-25
-20
5,0
6,0
-50
5.3
-40
-00
130
200
200
402
230
400
3000
grandantt swssithooaandaa
goandiant shassthousanda)
grancient shassthhusadd
(d) halfcheetah-mmedim
(e) hopper-medium
(f) walker22dmeddium
Figure 2: Return (upper sub-figures) and loss Eq. 11 (bottom sub-figures) during training with
different À values
D.2 COMPARISON WITH PESSIMISM ON Q
We implement an ablation version our method ceciood--entttyttee Q, which directlly penalize the value of state action pairs. Specifically, change the critic loss function into
16

---

Under review as a conference paper at ICLR 2023
@u+1 + arg min LQ(Q: Q²) = a -(,
(36)
E + ( Q
We use the same policy extraction method and test this method on the medium-tasie task, in which the data is collected using peddium-performet policy. In all the three tasks, the performance of the penalty-Q are worse than the the original implementation, the penalty-V counterpart. When penalty is on the state-action pair, illustrated by our theoretical discussion, the value of the evaluated Q value tends to pointwise lower bounds the true Q value, which results more conservative and thus worse policy. While when penalize V, the estimated value function only bounds the expectation the true function, which results more flexible and well-performed policy.
halfcheetah-mediumnz2
w.lker22-medium-22
hopp erpper--melum--
50
100
80
scoreinmmalized
40-
score/normallizee)
80
score(nemmliieed
30-
60
40
20
penally-V
40
penalty-V penalty-Q
penally-
Qenalty-Q
20
penally-O
10
250
500
1000
2.50
1000
250
5000
1000
grandlent steps(thousanss)
grandient stepstthousands)
grandlent staphhhuuss
Figure 3: Permance comparison between our original implementation and the penalty-Q version.
D.3 RELATIONSHIP BETWEEN MODEL BIAS AND FINAL PERFORMANCE
As stated in the main paper, compared normal model-based offline RL algorithms, CSVE is insensitive to model biases. To understand this quantitatively, now we investigate the effect of model biases the performance. We use the the dynamic model's 's average L2 error on transition prediction as the surrogate of model biases. As shown in Fig. in CSVE, the model bias has very little effect to RL performance. Particularly, for halfcheetah there observed effect of model errors to scores, while in hopper and walker2d with increasing errors, the scores have slight downward trend where the decrease is relatively very small.
halfcheetah-mediunnz2
hopper-medium--2
walker2d-medium-v22
120
100
100
sconinamssizzdd
score/noraalzzed
Scorefnomalized)
80
80
60
60
60
40
20
original values
20
original values
original values
fit values
values
0.10
0.12
0.14
0.16
10
12
errors
Figure 4: The relationship between score and the model biases. The correlation coefficient is re- spectively -0.32, -0.34, and -0.29.
17

---

Under review as a conference paper at ICLR 2023
D.4 REPRODUCTION OF COMBO
In the main body of this paper, our results of COMBO adopt the results presented in literature (Rigter al. 2022). Our goal here is to look into more details of COMBO's asymptotic performance evaluated during training. For comparison fairness, we adopt the official COMBO code provided by author, and rerun evaluate with the medium dataset of D4RL mujoco v2.
Fig. 5S] shows the asymptotic performance until 1000 epochs, in which the scores have been nor- malized with corresponding SAC performance. We found that both hopper and walker2d, the scores show dramatic fluctuations. The average scores of last 10 epochs for halfcheetah, hopper and walker2d are 71.7, 65.3 and -0.26 in respect. Besides, we found even D4RL vO dataset, COMBO's behaviours are similar.
Sinaraa, Scare fetu Average
FREBROO SCOE NERT Awraag
Figure 5: Return of COMBO on D4RL mujoco v2 tasks
D.5 EFFECT OF EXPLORATION NEAR DATASET DISTRIBUTIONS
As discussed in Section 3.1 and 4.2 proper choice of exploration on the distribution (d) beyond data
(dw) should help policy improvement. The factor À in Eq. T2 controls the trade-off on such 'bonus' exploration and complying the data-implied behaviour policy.
Let us take the medium-replay type of datasets to analyze its effect. In the experiments, with fixed
= 0.1, we investigate À values of {0.0, 0.5, 1.0, 3.0}. As shown in the upper figures Fig. 6 shows obvious effect to policy performance and variances during training. In general, there is a value under which increasing À leads to performance improvement, while above which further increasing À hurts performance. For example, with X 3.0 in hopper-medium-rees midiumrd-mmdiuu--pllalyy task, the performance get worse than with smaller values. The value of À is task-specific, and find that its effect is highly related to the loss in Eq. 11 which can be observed by comparing bottom and upper figures in Fig. [6 Thus, practice, we can choose proper according to the above loss without online interaction.
task and
18

---

Under review as a conference paper at ICLR 2023
3500
5000
4002
2000
2100
3003
1500
2000
1000
300
500
410
605
1050
220
500
800
1000
220
500
800
1003
grancient stopothassands)
glandient depethocrransss
grandient stpeethaurannss
L. Loss
40-
0.0
0.0
0.5
0.5
20
1.0
3.0
200
420
6CU
B00
2000
203
403
6:00
800
1000
205
420
6:00
1000
0'3ndlent wtw.stttro.ooooneooo
grandlunt Stepoit sossands)
granclort slopost Tossanns)
(d) Ialfchectab-meddin--eee
(e) mepper-meddu
(f) malkerrd-mediiu--eella
Figure 6: Effect of to Return (upper figures) and L loss in Eq. 11 (bottom figures)
19


Reviews:
[{"review_id": "qtKc0URwv4r", "reviewer": "Reviewer_KLUN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new algorithm--CSVE--that learns conservative estimates of state value functions by penalizing values of OOD states. The authors prove that the estimated state value functions are lower-bounds of the true value in expectation over any state distribution. Finally, the authors evaluate their algorithm against state-of-the-art offline RL baselines. ", "summary_of_the_review": "To my understanding, the key idea of the proposed CSVE algorithm is to consider a penalty over states rather than state-action pairs. However, CSVE also consider some differences in algorithm design (see (2) in weaknesses) to similar baselines, so it is difficult to judge how important this key idea is to the improved performance. I believe the paper could be drastically improved by removing theses extensions as ablations, as well as considering COMBO as an additional baseline. I feel that CSVE and COMBO are much more closely related than CSVE and CQL, as the former are both model-based algorithms. ", "strength_and_weaknesses": "Strengths:\n\n(1) The paper has a clear organization and is well-written.\n\n(2) CSVE achieves impressive empirical performance against strong offline RL baselines.\n\nWeaknesses:\n\n(1) The algorithm itself is very similar to existing conservative offline RL algorithms, particularly CQL and COMBO. To my understanding, the authors do demonstrate theoretical and empirical advantages over CQL but not COMBO. In my opinion, COMBO is more similar as both CSVE and COMBO are model-based in the sense that they require learning a dynamics model.\n\n(2) Though the backbone of the CSVE algorithm is clear, the authors add several extensions such as learning an ensemble of transition models (which COMBO does not do), and using a separate policy extraction step rather than alternating value- and policy-improvement as CQL, COMBO do. It is not clear from the empirical analysis whether those changes are important to the empirical performance, or the fact that the penalty is over states rather than state-action pairs (which is the central novelty of the proposed algorithm). The authors should consider such ablations, as well as a comparison to COMBO, in the experiments to make their results more compelling.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X1WZvHCVuj1", "reviewer": "Reviewer_pQB8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A major challenge in offline RL is the distribution mismatch between the behavior policy generating the offline data and the target policy we want to learn. Such a mismatch can result in an overestimation error in the Bellman update, which can further result in divergence. The paper proposes to learn a conservative state value estimate (in contrast to the existing CQL, which learns conservative action-value estimates). The conservative state-value estimate is learned by subtracting a term from the regular Bellman backup. The term is proportional to the density ratio of the target and behavior distribution. The main theory shows the bound of the expected state value (i.e., expectation over states) under both the offline data distribution and the target policy distribution. In algorithmic design, the conservative state value estimate is used to learn a critic in an actor-critic algorithm. Experiments on Mujoco domains are conducted to show its effectiveness. \n", "summary_of_the_review": "I have limited time to review the paper. But to the best knowledge, the current weaknesses of the paper outweigh the strength. I will refer to the author's response and other reviews and adjust my score accordingly. ", "strength_and_weaknesses": "Strength: \n\n1. The paper studies an interesting topic which could be practically useful to a broad range of RL applications. \n\n2. The paper has a reasonably clear presentation. \n\nWeaknesses: \n\n---------\n1. The basic idea is incremental to CQL. The actions can be marginalized from the CQL’s bound to get the state value bound. But the paper does not discuss any connections. \n\nOn close examination, both the theoretical result and algorithmic design are highly similar to CQL.\n\n---------\n2. The theorems are not informative. First, the theoretical results are bounding E[V(s)], but CQL provides a bound for any state-action pair. Bounding E[V(s)] does not give a guarantee for any state value estimate’s conservativeness.  \n\nSecond, before showing the bound, shouldn’t the convergence be shown first? To me, it is even unclear if the proposed conservative state value Bellman update would converge or not. \n\n---------\n4. Experiments. I expect to see clear evidence to show at least three things: \n1). The necessity of learning a conservative state value, rather than action-value; \n2). The the effectiveness of using the proposed method to learn conservative state value, rather than other methods (for example, using CQL and use the learned policy to get a state value estimate); \n3). The comparison to IQL (Offline RL with implicit q learning by Ilya et al.), which has a similar high-level idea. IQL attempts to avoid overestimation by avoiding learning action values. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QBVcjZl8LmM", "reviewer": "Reviewer_zsLW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an algorithm called CSVE, which imposes penalization on states rather than state actions used in CQL. By doing such, we may benefit from learning a better conservative value since learning a conservative Q-value over joint state-action space is more challenging. The paper provides theoretical results that are improved from that of CQL. Although it requires learning a model to learn, the experiment results suggest that CCSVE overall performs better than CQL.", "summary_of_the_review": "Overall, I enjoyed reading this paper and I think there will be a number of readers interested in this paper. There are quite a few typos in the papers, and I would like to see more empirical results (e.g. when the model is incorrect), but overall I recommend acceptance of this paper.", "strength_and_weaknesses": "Strength\n- Theoretically well supported\n- Quite an original idea and algorithm design\n- Descent performance\n\nWeakness\n- Requires model:\n\nWhen compared to other model-free methods, CSVE requires estimating dynamics as well, so it may not be very fair to compare to other model-free algorithms. Since the authors argue that the proposed method is not very sensitive to model bias, it would be also interesting to see how CSVE is affected by varying model bias.\n\n- Not enough interpretation on why the suggested algorithm is good:\n\nIt is kind of vague in the paper why suggested algorithm works well when compared to CQL. Is it really easy to learn pessimistic V value instead of Q? Can it be shown as an example? Or is V-learning algorithm is just better than Q-learning in these domains, even in the online setting? It is hard to see the clear reason from the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EQJ13A_0rBL", "reviewer": "Reviewer_uGHe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a CQL-like method for penalizing out-of-distribution states in offline RL. The method first describes an approach for off-policy evaluation with OOD state penalization, and then incorporates into the training pipeline for Q-functions, and then uses it for offline RL.", "summary_of_the_review": "In all, I am not convinced if this paper is doing anything more than COMBO. Moreover, the paper doesn't compare to COMBO, and doesn't justify design decisions. In all, this makes me think that the paper is not yet significant for the community, and must address these issues. Thus, I opt to give a reject score.", "strength_and_weaknesses": "Strength: The method is clearly intuitive and a good idea to do, so that's a plus.\n\nWeaknesses:\n\n- Many design decisions, no clear explanation: There are many design decisions in the paper that have not been ablated or explained in more detail -- for example, why AWR, and not just standard SAC/CQL-like policy extraction? why learn separate V(s) and Q(s, a), and not just directly penalize Q(s, a) on OOD states? \n\n- Missing comparisons: the method uses a dynamics model to obtain new states, yet the method doesn't compare to COMBO? Seeing the resutls, seems like COMBO would perform better than this, which means that the results are not significant...\n\n- Novelty: It seems like if the Q(s, a) were directly penalized on OOD states, and policy actions, the method in this paper would just be COMBO identically, and indeed COMBO is just a more generalization of this approach. So, unless the choice of V(s) can be justified rigorously, I don't think this paper provides a novel method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject