Mildly Conservative Q-Learning for Offline Reinforcement Learning
Authors: Jiafei Lyu
Xiaoteng Ma
Xiu Li
Zongqing Lu
Venue: NeurIPS 2022 Accept (2022)

Abstract: Offline reinforcement learning (RL) defines the task of learning from a static logged dataset without continually interacting with the environment. The distribution shift between the learned policy and the behavior policy makes it necessary for the value function to stay conservative such that out-of-distribution (OOD) actions will not be severely overestimated. However, existing approaches, penalizing the unseen actions or regularizing with the behavior policy, are too pessimistic, which suppresses the generalization of the value function and hinders the performance improvement. This paper explores mild but enough conservatism for offline learning while not harming generalization. We propose Mildly Conservative Q-learning (MCQ), where OOD actions are actively trained by assigning them proper pseudo Q values. We theoretically show that MCQ induces a policy that behaves at least as well as the behavior policy and no erroneous overestimation will occur for OOD actions. Experimental results on the D4RL benchmarks demonstrate that MCQ achieves remarkable performance compared with prior work. Furthermore, MCQ shows superior generalization ability when transferring from offline to online, and significantly outperforms baselines. Our code is publicly available at https://github.com/dmksjfl/MCQ.

Mildly Conservative Q-Learning for Offline Reinforcement Learning
Jiafei Lyu', Xiaoteng Ma Xiu Li Zongqing Lu
Tsinghua Shenzhen International Graduate School, Tsinghua University
Department of Automation, Tsinghua Unversity School of Computer Science, Peking University [1vjf20, mma.xt17) xtt1]]maiis tsinghua. eeighhaaaduu.... edu. cn,
xiuūsz, tsinghua, edu. en, zongqing lu@pku. teulpku.edu.com edu. en
Abstract
Offline reinforcement learning (RL) defines the task of learning from static logged dataset without continually interacting with the enviromment. The distribution shift between the learned policy and the behavior policy makes it necessary for the value function to stay conservative such that out-of-distribution (OOD) actions will not be severely overestimated. However, existing approaches, penalizing the unseen actions regularizing with the behavior policy, suppresses the generalization of the value function and hinders the performance improvement. This paper explores mild but enough conservatism for offline learn- ing while not harming generalization. We propose Mildly Conservative Q-learning (MCQ), where OOD actions are actively trained by assigning them proper pseudo Q values. We theoretically show that MCQ induces policy that behaves at least well as the behavior policy and no erroneous overestimation will occur for OOD actions. Experimental results on the D4RL benchmarks demonstrate that MCQ achieves remarkable performance compared with prior work. Furthermore, MCQ shows superior generalization ability when transferring from offline to on- line, and significantly outperforms baselines. Our code is publicly available https:/giitub.oom/mnkks///////
pessimistic, which
1 Introduction
Continually interacting with the environment of online reinforcement learning (RI.) is often infeasible and unrealistic, since the data collection process of the agent may expensive, difficult, or even dangerous, especially in real-world applications. Offline RI, instead, aims at learning from a static dataset that was previously collected by some unknown process [36], hence climinating the need for environmental interactions during training.
The main challenge of offline RL the distribution shift of state-action visitation frequency between the learned policy and the behavior policy. The evaluation of out-of- oftoo--distribttion (OOD) actions causes extrapolation error [14], which can be exacerbated through bootstrapping [34] and result in severe overestimation errors. Thus, keeping conservatism in value estimation is necessary in offline RI. [24, 50, 60]. Previous methods achieve the conservatism by compelling the learned policy to be close to the behavior policy [14, 58, 34, 13, 57]. by penalizing the learned value functions from being over-optimistic upon duttr--distritttionn (OOD) actions [35, 33, 59], or by learning without querying OOD samples [56, 8, 62, 32, 40].
Equaal Contribution
Corresponding Authors
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

OOD Penalization
Policy Regularization
Mild Conservatism
Value, / Policy PDF
Value Policy PDF
Value Policy PDF
Qmin
amax
4min
4max
Qmin
4max
ction
A ction
Action
Optimal Value
Estimated Value
Behavior Policy
Leamed Policy
Figure 1: Comparison of prior methods against mild conservatism. The red spots represent the dataset samples. The left figure shows that penalizing OOD actions makes the value function drop sharply at the boundary the dataset's support, which barriers policy learning. The central figure depicts that policy regularization keeps the policy near behavior policy, leading to undesired performance if the behavior policy unsatisfying. On the right side, we illustrate the basic idea of mild conservatism. The estimated values for OOD actions allowed to be high long it does not affect the learning for the optimal policy supported by the dataset, i.e., ((s, a (ond) < mapxeEuupprt(ju)) Q(s, a).
In practice, we rely on neural networks to extract knowledge from the dataset and generalize to the nearby unseen states and actions when facing continuous state and action spaces. In other words, we need the networks to "stitch" the suboptimal trajectories to generate the best possible trajectory supported by the dataset. Unfortunately, there is no free lunch. Conservatism, which offline RL celebrates, often limits the generalization and impedes the performance of the agent. Existing approaches are still inadequate in balancing conservatism and generalization. As illustrated Figure! I, policy regularization unreliable for offline RL when the data-collecting policy poor, and value penalization methods often induce unnecessary pessimism in both the in-dataset region and OOD region. We argue that the proper conservatism should be as mild as possible. As depicted in Figure 1, we aim at well estimating the value function in the support of the dataset, and allowing value estimates upon OOD actions to be high (even higher than their optimal values) as long as (od) < SuppoeSupo (s a) is satisfied. The mild conservatism benefits generalization since value estimates upon OOD actions are slightly optimistic instead of being overly conservative.
To fulfill that, we propose a novel Mildly Conservative Bellman (MCB) operator for offline RL, where we actively train OOD actions and query their Q values. We theoretically analyze the convergence property of the MCB operator under the tabular MDP setting. We show that the policy induced by the MCB operator is guaranteed to behave better than the behavior policy, and can consistently improve the policy with a tighter lower bound compared with policy constraint methods value penalization methods like CQL [35]. For practical usage, we propose the practical MCB operator and illustrate its advantages by theoretically showing that erroneous overestimation error will not occur with it. We then estimate the hehavior policy with a conditional variational autoencoder (CVAE) [29, 51], and integrate the practical MCB operator with the Soft Actor-Critte (SAC) [20] algorithm. To this end, we propose our novel offline RL algorithm, Mildly Conservative Q-learning (MCQ).
Experimental results on the D4RL MuJoCo locomotion tasks demonstrate that MCQ surpasses recent strong baseline methods on most of the tasks, especially on non-expert datasets. Meanwhile, MCQ shows superior generalization capability when transferring from offline to online, validating our claims that mild pessimism of importance to offline learning.
Preliminaries
We consider Markov Decision Process (MDP) specified by tuple (S, A. Po, Po P, 7), where S is the state space, A is the action space, r(s, a) A the reward function, Po(s) is the initial state distribution, p(s'ls, a) : A S + [0, 1] is the transition probability, 7 E [0, 1) is the discount

---

factor. Reinforcement learning (RL) aims at finding a policy x(-[s) such that the expected cumulative
long-term rewards J(T) -E, (xxrroo()) (a -111 (p.))))..a,)) 12 7'r(s,, az)] are maximized. The state-action function Q(s, a) measures the discounted return starting from state and action a, and following the policy π. We assume that the reward function r(s, a) is bounded, i.e., |r(s, a)] & mmax
Given a policy x x--ss), the Bellman backup for obtaining the corresponding Q function gives:
(*)(s,a) == r(s, a) + B. Bo .a)]-
The Q function of the optimal policy satisfies the following Bellman optimal operator:
TQ(s, a) == r(x, a) + t. max Q(s', a'EAA
In offline RL setting, the online interaction is infeasible, and we can only have access to previously collected datasets D = {(Ss, 0s, Ti, 5'+1, d4) J/z where d is the done flag. We denote the behavior policy as (((-(s). The Bellman backup relies on actions sampled from the learned policy, a' ~ (-1ss). However, a' can lie outside of the support of μ due to the distribution shift between π and μ. The value estimates upon can then be arbitrarily wrong, resulting in bad policy training. Unlike prior work, actively train OOD actions by constructing them pseudo target values. In this way, retain pessimism while enjoying better generalization.
3 Mildly Conservative Q-Learning
In this section, we first formally define the MCB operator and characterize its dynamic programming properties in the tabular MDP setting. We further give a practical version of the MCB operator. We show that no erroneous overestimation will occur with the MCB operator. Finally, we incorporate the MCB operator with SAC [20] and present our novel offline RL algorithm.
3.1 Mildly Conservative Bellman (MCB) Operator
Definition 1. The Mildly Conservative Bellman (MCB) operator is defined
a) = (7.
(3)
where
TiQ(s,a) = {
u(als) > else. [ Q(s, Q(s,,)))))) -,
a(s,a),
(4)
a2Q r(s, a) + y., (max Q(s' p(als) > O. Q(s,a),
(5)
else.
The basic idea behind this novel operator is that if the learned policy outputs actions that lie in the support region of p. then we go for backup; while if OOD actions are generated. we deliberately replace their value estimates with m " (Spppor((((((1s)) Q(s, a') 8, where 8 > 0 can be arbitrarily small. That is, different from standard Bellman backup, we set up checking procedure (i.e., TT) of whether the previous backup (i.e., T2) involves OOD actions for the update. Intrinsically. we construct pseudo target values for OOD actions. We subtract a small positive such that OOD actions will not be chosen when executing policy via arg max.e Q(s,a),
For better understanding of the MCB operator, we theoretically analyze its dynamic programming properties in the tabular MDP setting. All proofs are deferred to Appendix A.
Proposition 1. In the support region of the behavior policy, iee. Support(μi). the MCB operator is y-contraction operator in the Lox norm, and any initial Q function can converge to unique fixed point by repeatedly applying TMCB-
Proposition 2 (Behave at least as well as behavior policy). Denote QMCB as the unique fixed point acquired by the MCB operator, then in Support(p) we have: Q₁ S QMCB I Qpe, where Qu is the Q function of the behavior policy and Q₂- is the Q function of the optimal policy in the batch.
Proposition 2 indicates that the policy induced by the MCB operator can behave at least as well as the behavior policy, and can approximate the optimal batch-constraint policy. Apart from this advantage, further show that the MCB operator results in milder conservatism. We start by observing that
3

---

value penalization method, like CQL [35], guarantees that the learned value function (s. a) is lower bound of its true value (" a). It also ensured that following such conservative update leads to a safe policy improvement, i.e., (TCQLL) 2 J(u) O ক-- (Theorem 3.6 in [35]). For explicit policy constraint methods, e.g., TD3+BC [13], the learned policy яい mimics the behavior policy μA, and can hardly behave significantly better than μ.. We show Proposition 3 that explicit policy constraint methods also exhibit safe policy improvement, [(pp) 2 J(H) - - (1-1))))) while the MCB operator can consistently improve the policy with tighter lower bound.
Proposition 3 (Milder Pessimism). Suppose there exists explicit policy constraint offline rein- forcement learning algorithu such that the Ki-viveeeeece of the learned policy Tp(-|s) and the behavior policy (u(-ss) is optimized guarantee max (KL(U, Tp), KT <T 1c) S c, vs. Denote
Je e max, BE A a)), where A" A" a.) is the advantage function. Then
V2ye Sp J(mp) 2 J(n)
(1 (1-7)) 7) VE,
(6)
while for the policy TMCB learned by applying the MCB operator, we have
J (TMCB) 2 J(H).
(7)
In summary, the MCB operator benefits the offline learning in aspects: (1) the operator is contraction, and any initial Q functions are guaranteed to converge to unique fixed point; (2) the learned policy of the MCB operator is ensured be better than the behavior policy, and reserve milder pessimism compared with policy constraint methods or CQL.
3.2 Practical MCB Operator
In practice, it is intractable acquire max,' SSupooniii (s,, a') in Ti of Eq. (4) in continuous control domains, and the behavior policy is often unknown. Thus, we fit an empirical behavior policy with supervised learning based on the static dataset. The pseudo target values for the OOD actions are then computed by sampling N actions from fa, and taking maximum over their value evaluation. Formally, we define the practical MCB operator below, accompanied by the theoretical analysis. Definition 2. The practical Mildly Conservative Bellman (MCB) operator is defined as
((s,a)
(8)
where
a(),a), TiQ(s,a) = B(03)N=((111) maxx ((x,
u(aas) > 0. else.
Compared with Eq. (4), make small modification of Tı, and keep Tz unchanged. There is need to subtract 5 here as generally ((at)~~A(())) [aaxa'(a'])* Q(s, a') < maxp'=Support(e)) @(s, d').
The practical MCB operator is much easier to implement practice. We show that the practical MCB operator is still y-contraction in the support region of the behavior policy μ.
Proposition 4. Proposition / still holds for the practical MCB operator.
Since fit the empirical distribution ja of the behavior policy μa, there may exist shift between and μr, especially when we represent the policy via neural networks. That suggests that OOD actions a' can still be sampled from e such that a' & Support(u(-Is)) Our last main result reveals that erroneous overestimation issue will not occur with the aid of the practical MCB operator.
Proposition 5 (No erroneous overestimation will occur). Assuming that sup, Drv((((1s) II u(-ls)) < we have
E [[c o' d'E(o
max
0(s,a')
v max a' ('ESappor((((()))
Q(s,a') +1 (1- N) Fmax
Remark: This proposition generally requires comparatively well-fitted empirical behavior policy j.. practice, we model f with CVAE. In most cases, CVAE can already the dataset well and guarantee good performance. Whereas there may exist some situations, e.g., the dataset is highly

---

multi-modal, then one can replace the CVAE as the conditional GAN (CGAN) to better capture the different modes in the dataset as depicted in [61]. We believe generative models like CGAN will be good choice by then.
Intuitively, the above conclusion says that if the empirical behavior policy j well fits μt, i.e., is small enough, then regardless of how {a{}-V are sampled, the pseudo target value will approximate the maximum Q-value within the dataset's support with high probability. The extrapolation error is under the scale of (1 - (1 2c)*) ) Timg We expect a good empirical behavior policy such that most of the actions sampled from it will be in-distribution. However, if e is large, NN can act as a trade-off parameter. The smaller N we use, the more conservative we are. Fortunately, we find empirically that our method performs well in large interval of N over different tasks (see Section 4.2). Hence, it is safe to fix a N in practice.
3.3 Algorithm
As aforementioned, we often cannot get prior information about the behavior policy Thus, we need empirically fit behavior policy j with supervised learning for applying the practical MCB operator. Our algorithm, Mildly Conservative Q-learning (MCQ), trains an additional generative model, which is also adopted by many prior work [14, 17, 33, 67]. We build our novel offline algorithm upon an off-the-shelf off-policy online RL algorithm, Soft Actor-Critic (SAC) [20].
Modelling the behavior policy with the CVAE. We utilize conditional variational autoencoder (CVAE) 51, 14] to model the behavior policy μ. Given a fixed logged dataset, the goal of the CVAE is to reconstruct actions conditioned on the states such that the reconstructed actions come from the same distribution as the actions in the dataset, i.e., (((11s). That generally satisfies the assumption we make in Proposition 5. As concerned by [32], training generative model like CVAE still may produce out-of-dataset actions, which leads to extrapolation error since undefined Q values can be possibly queried. Prior methods, like BCQ [14], do not well address such issue. While for our algorithm, such concern is mitigated because overestimation error is actually under control as is guaranteed by Proposition 5.
The CVAE (s) parameterized by w is made up of an encoder Eş(s, a) and a decoder Do parameterized by &, 1 respectively. w ={S. (). The CVAE is optimized by maximizing its variational lower bound, which is equivalent to minimizing the following objective function.
LCVAE = ((..,))D,
[(a (s,(s, =))² + KT. (T.(FFF (Fe(s, (), 1))]
(10)
where KL(p, 9) denotes the KL-divergence between probability distribution p(-) and q(.), and I is the identity matrix. When sampling actions from the CVAE, we first sample a latent variable a from the prior distribution, which is set to be multivariate normal distribution (0, I), and then pass it in conjunction with the state 8 into the decoder De (8, =) to get the desired decoded action.
It is also worth noting that we do not choose GAN [19] the generative model because it is known to suffer from training instability and mode collapse [52, 6, 5]. Also, GAN consumes much more time and memories to train compared with the CVAE.
Loss functions. In deep RL, the Q function is represented with neural network parameterized by and is updated via minimizing the temporal difference (TD) loss E,,a.,, [QQo(s a) (TQ(s, a] We actually are performing the regression task (s, a) + (Q(s, a) to train the Q function. The target value TQ(s, a) is usually computed by utilizing lagging target network parameterized by O' without gradient backpropagation. As a typical actor-critic [30, 31, 53] algorithm, SAC uses its critic networks to perform value estimation and uses separate actor network for policy improvement. In order to incorporate the MCB operator with the off- thelth SAC algorithm, we need to check whether the sampled action a' ~ T-|ss) lies outside of the behavior policy's 's support, i.c., whether ([aals)) > O. However, such a criterion is not reliable, because the true behavior policy μ is unknown and it is difficult to examine whether uua'(s (s) > 0 in practice. It is also problematic if we rely on the empirical behavior policy j to check whether a' is OOD as ja itself can produce OOD actions.
We then resort to constructing an auxiliary loss for OOD actions and integrating it with the standard Bellman error. Specifically, we sample ood from the learned policy m(-[s")) based on the sampled state gln ~ 'D from the dataset and assign them pseudo target values based on the practical MCB operator. Note that the superscript ood used to distinguish from the in-dataset real actions, and god is not necessarily OOD action. We remark that if a ood Support((u(( (s(()s)), the pseudo Q
5

---

Algorithm Mildly Conservative Q-learning (MCQ)
1: Initialize CVAE G.. critic networks Q0 Qo, and actor network T with random parameters
2: Initialize target networks 8y < 01,03 02 t- B2 and offline replay buffer D.
3: for tort= 1 to T do
4: Sample a mini-batch B = {(s, a, r, d)) from D, where d is the done flag
5: Train CVAE via minimizing Eq. (10)
6:
Get target value: = r(s, a) + [min Qe (8', aa) - a log (a [6')], , ~ Ta(-ss))
7:
Sample N actions from based on each and s', set gt" {8,
8:
Compute the target value for the OOD actions via Eq. (13)
9:
Update critic 0, with gradient descent via minimizing Eq. (11)
10:
Update actor ó with gradient ascent via Eq. (14)
11: Update target networks: 8 - T0, + (1 T)6 i 1,2
12:
end for
value will not negatively affect the evaluation upon it, because in-distribution actions are still trained to approximate the optimal batch-constraint Q value. In this way, we actively train both possible OOD actions and in-distribution actions simultaneously via OOD sampling. The resulting objective function for the critic networks is presented in Eq. (11).
Ceritic )'
aood [Qo, (s, a) - +1-A)()))BB)))) Donddwow [[Qu, (s ) - - (11)
where the target value for the in-distribution actions gives
= r() +2 min Qu (s', a') - a log (o((((s') a ER.+,
(12)
which follows the standard target value of vanilla SAC. The hyperparameter À balances the in- distribution data training and OOD action training. Following the formulas of the practical MCB operator in Eq. (9), the pseudo target value for the OOD action is computed by:
V/ min M 1=1
max
Q8, (8" a')
(13)
[a'>(a))
Note that we experimentally find that replacing the min operator with mean operator does not raise much difference performance. We hence take advantage of the min operator to fulfill the pseudo clipped double Q-learning for OOD actions.
The policy is then optimized by solving the following optimization problem:
To = max Eg min Qa, (8, a) a log n (-1s) 1=122
(14)
We detail the learning procedure of our MCQ in Algorithm 1. Different from [13], our method does not require normalization over states or value functions. The only change we make to the vanilla SAC algorithm is extra auxiliary loss term (blue term in Eq. (11)) such that OOD actions
actively
and properly trained. The additional critic loss term can also be plugged into other off- off-policy online RL algorithms directly. As an evidence, combine the MCB operator with TD3 [15]. yielding deterministic version of MCQ. Please refer to Appendix for more details.
Experiments
In this section, first empirically demonstrate the effecctiveness and advantages of our proposed MCQ algorithm on D4RL benchmarks [12]. We then conduct a detailed parameter study to show the hyperparameter sensitivity of MCQ. We also experimentally illustrate that the value estimation of MCQ will not incur severe overestimation and pessimistic value estimates are witnessed in practice. Finally, we show the superior oflinne-to-nnine fine-tuning benefits of MCQ on some MuJoCo datasets.
4.1 Results on MuJoCo Datasets
We experimentally compare our MCQ against behavior cloning (BC), SAC, and several recent strong baseline methods, CQL [35], UWAC [59], TD3+BC [13], and IQL [32], on D4RL [12] benchmarks.

---

We choose these methods as they typically represent different categories of model-free offline RL, i.e. CQL is value penalization method, TD3+BC involves explicit policy constraint (BC loss). UWAC relies on uncertainty estimation for training, and IQL. learns without querying OOD samples.
We conduct experiments MuJoCo locomotion tasks, which made up of five types of datasets (random, medium, medium-replay, medium-expert, and expert), yielding total of 15 datasets. We use the most recently released "-v2" datasets for performance evaluation. The results of BC and SAC are acquired by using our implemented code. The results of CQL and UWAC are obtained by running their official codes, because the reported scores in their papers are not obtained on MuJoCo "-v2" datasets. We take the results of TD3+BC from its original paper (Table 7 in [13]). Since the IQL paper does not report its performance on MuJoCo random and expert datasets, we run IQL using the official codebase on them and take the results on medium, medium-replay, medium-exxieeeeeeeee expuurteeeeettttt datasets from its original paper directly. All methods are run for 1M gradient steps.
Table Normalized average score comparison of MCQ against baseline methods on D4RL bench- marks over the final 10 evaluations. 0 corresponds to a random policy and 100 corresponds to an expert policy. The experiments are run on MuloCo "-v2" datasets over random seeds. r= random, m medium, m-r medium-replay. m-e medium-expert, e expert. We bold the highest mean.
Task Name
BC
SAC
CQL
UWAC
TD3+BC
IQL
MCQ (ours)
halfcheetah-r
2.2.10.0
29.7
17.5
2.3.1.00
11.01111
13.1
28.5110.6
hopper-r
3.740.6
9.9
7,94044
2.7=0.3
8.5
7.9+0.2
01,8+0.5
walker2d-r
1.3:0.1
0.940.8
5.1113
2.0=0.4
1.61.7
5.4+1.2
17.0#3.0
halfcheetah-n
43.2:10.6
55.2+27.8
47.0+:..5
42.2+0.4
48.3+0.3
47.410.2
64.3
hopper-m
54.14:3.8
0.8+0..0
53.0++28.5
50.9+4.44
59.3+44.2
66.2+:5.7
78.4+t4.3
walker2d-m
70.9+11.0
-0.3++0.2
73.3+17.7
75.4+3.0
83.7+2.1
78.3+88.7
91.0++0.4
halfcheetah-m-r
37.6+-2.1
0.8+1.0
45.5+0077
35.9+3377
44.6+0..5
44.2+-1.2
56.8+00.6
hopper-m-r
16.6:14.8
7.410.5
88.7+12.9
25.31117
60.9118.8
94.7-18.6
101.61088
walker2d-m-r
20.3119.
-0.410.3
81.812.77
23.616.9
81.815.5
73.817.1
91.315.7
halfcheetah-m--e
44.0.11.6
28.411944
75.61125.7
42.710.3
90.71443
86.715.3
87.5111.
hopper-m-c
53.9++4.7
0.7
105.6=12.9
44.9+8.1
98.0+9.4
111.2+0.11 91.5++1.3
walker2d-m-e
90.1+13.2
1.943.9
107.9=1.6
96.5+99.1
110.1
109.6+1.0
114.2+0.77
Average Above
36.5
11.3
59.1
37.0
58.2
59.9
72.8
halfcheetah-c
91,8+1.5
10,8
1.3
92.9=0.6
96.7#1
95,0+0.5
96.2+0.44
hopper-c
107.7
0.7+0.0
96.5+28.0
110.5
107.8+7
111440.4 10.
walker2d-e
106.7:0.2
0.7
108.5=0.5
108.4
110.2+00.3
109.9+1.2
107.2
Total Average
49.6
9.0
67.3
50.4
67.6
68.9
79.2
In our experiments, we set the number of sampled actions N = 10 by default and tune the weighting coefficient A. We report the À used for all tasks in Appendix C, along with details on the experiments and implementation. We summarize the normalized average score comparison of MCQ against recent baselines Table Unsurprisingly, we observe that MCQ behaves better than BC on all of the tasks, which is consistent with our theoretical analysis in Proposition 2 and 3. MCQ also significantly outperforms the base SAC algorithm. Prior offline RI. methods struggle for good performance on non-expert datasets like random and medium-replas while MCQ surpasses them with remarkable margin on many non-expert datasets. We attribute the less satisfying performance of prior offline RL methods to their strict conservatism, which restricts their generalization beyond the support of the dataset and leads to limited performance. The results, therefore, validate our claim that milder pessimism is more we need for offline learning. Furthermore, MCQ is also competitive baselines on expert datasets. MCQ achieves the best performance on 11 out of 15 datasets, yielding a total average score of 72.8 non-expert datasets, and average score of 79.2 all 15 datasets. Whereas the second best method, IQI, has an average score of 59.9 on non-expert datasets and a total average score of 68.9 across all tasks.
4.2 Parameter Study
In this subsection, we conduct a detailed parameter study MCQ. MCQ generally contains two hyperparameters, weighting coefficient À and number of sampled actions N. To demonstrate the parameter sensitivity of MCQ, choose two datasets from MuJoCo locomotion tasks and conduct
7

---

hallcheetah-medium-22
hopper-medium-rellay-2
hosper-medium-replayyy
300
Normalized score
60
Normalized score
120
100
A=0.3
A=0.3
100
A=0
A-0.5
A-0.5
A-0.5
20-
A-0.9
A-0.7
A-D.7
-100
A-0.7
A=9.55
A=
-200
A-
-300
0.2
Millian Steps
Millian Steps
Million Steps
(a) Effects of A
(b) Effects of A
(c) Q value w.r.t.
halichhetah-medluu-222
hopper-meduum-rreelayyvv
hopper-medlum-reelayy-22
Normal aellzzdd score
Normalized lied Score
125
250
200
value
40
M-5
150
N=10
50-
M=110
100
N=10
N-20
N-20
N-50
N-50
N-50
0.4
0.6
0.2
0.4
0.G
0.8
1.0
Million Steps
Million Steps
Steps
(d) Effects of
Effects of. NV
(1) Q value w.r.t. N
Figure 2: Parameter study and Q function estimation on halfchettah-mediun--22 and hopper-medium- replay-v2. The shaded region captures the standard deviation.
experiments on them, halfcheetah-medinee mediumm-2, and hoppe-meddiun-replayy... The experiments are run for IM gradient steps over 4 different random seeds.
Weighting coefficient X. The weighting coefficient À is a critical hyperparameter for MCQ, which directly controls the balance between in-distribution actions training and OOD actions training. If we set À = 1, then MCQ degenerates into the base SAC algorithm. If À leans towards O, the critics will be overwhelmed by OOD actions. Intuitively, one ought not to use small because more weights are desired for standard Bellman error such that in-distribution state-action pairs can be well-trained. We observe significant performance drop with smaller in Figure 2(a) and 2(b). Also, we find that choosing 0.7 < X generally induces good performance.
Number of sampled actions N. N works regularizer control the potential extrapolation error. In case the behavior policy is known, we require N to be as large as possible to better estimate the maximum Q value. While in practice, then plays a role to balance pessimism and generalization. To see the influence of N, fix À 0.95 for the two datasets. Experimental results in Figure 2(d) and 2(e) indicate that MCQ is insensitive to N for a wide range of N. We therefore set N 10 by default.
leverage the CVAE approximate Jr, from which OOD
actions can be sampled.
Value estimation. We present the Q value estimates with respect to (w.r.t)) À and N in Figure 2(c) and 2(f). The Q estimation calculated via E;=1,2B( ( (s, a)]. The results illustrate that (1) smaller À will incur severe underestimation issue (as depicted by Figure 2(c), Q values collapse with 0.5 or À= = 0.3); (2) no overestimation is observed, even with a large À = 0.95, which validates the theoretical result in Proposition 5; (3) the Q estimates resemble each other under different N. We conclude that MCQ ensures a stable and good value estimation with proper X.
4.3 Offline-to-nnlnee Fine-tuning
We examine the offlineto-onnlne fine-tuning capability of MCQ against some prior strong offline RL baselines, CQL [35], TD3+BC [13], IQL [32]. We additionally compare against AWAC [45], which is designed intrinsically for affline-to-onling adaptation. We conduct experiments on MuJoCo random and medium-replar datasets. It is challenging to train these datasets for both offline and offline-to-online fine-tuning as they are non-expert, or even contain many bad transitions. We first train baselines and MCQ for 1M gradient steps offline and then perform online fine-tuning for another 100K gradient steps. Note that IQL paper [32] adopts 1M steps for online fine-tuning. However, we argue that IM steps of online interactions are even enough to train off off-policy online RL algorithms

---

AWAC
COL
TD3+BC
hallcheessahranddomv2
hopper-random-v2
walker2d-random-2
2230
Retur
Return
600
Return
2006
200
-200
600
64
10
0.4
64
36
Environmental Steps (*leS)
Environmental Steps (*le5)
Environmental Steps (*1e5)
halfcheetah..o medium-replay-v2
hopper medium replay-v2
3006
3500
Return
700
Return
1893
60.0
1000
1930
300
Environmental Steps (*leS)
Environmental Steps (*1e5)
Environmental Steps (*1e5)
Figure 3: Offline-to-online fine-tuning results on 6 D4RL MuJoCo locomotion tasks.
from scratch to perform very well. We thus believe 100K steps is more reasonable for the online interaction. All methods are run over 4 random seeds. The results shown in Figure 3, where the shaded region denotes the standard deviation. As expected, we observe that MCQ consistently outperforms prior offline RL methods as well as AWAC on all of the datasets, often surpassing all of them with large margin. The mild pessimism MCQ makes it adapt faster, or keep the offline good performance during online interactions. Other prior offline RL methods, unfortunately, fail in achieving satisfying performance during online interaction due to strict conservatism and lack of generalization ability.
5
Related Work
Model-free offline RL. Prior model-free offline RL methods are typically designed restrict the learned policy from producing OOD actions. They usually achieve this by leveraging importance
sampling [47, 54, 39, 44, 16], incorporating explicit policy constraints (8,,58, 58, 17, 13, 11], learning
latent actions [67, 3]. penalizing learned value functions such that low values are assigned to unseen
actions [35, 33, 41], using adaptive methods [18], and uncertainty quantification [59, 65, 4]. Another
line of the methods, instead, resorts to learning without querying OOD actions [56, 32]. By doing so, they constrain the learning process within the support of the dataset. Nevertheless, existing methods may induce unnecessarily wver-pessimistic value functions, and their performance is largely confined by how well the behavior policy [45, 38, 4]. That partly explains why these methods not satisfiable when trained non-expert datasets (e.g., random datasets). MCQ keeps milder conservatism and better generalization ability as OOD actions are actively trained with proper targets.
Model-based offline RL. Model-based offline RL methods, in contrast, learn the dynamics model supervised manner, and leverage the learned dynamics for policy optimization. Advances in this field include uncertainty quantification [46, 64, 27, 10], learning conservative value functions [63], representation learning [37, 48], constraining the learned policy with a behavior cloning loss [42], and sequential modelling [7, 23, 43]. However, there is guarantee that the trained dynamics models are reliable, e.g., poor transitions can be generated, especially in complex nigh-dimensional environments [22]. Meanwhile, training dynamics models raises extra computation costs.
Offinne-t-online RL. There are some efforts on accelerating online interactions with the aid of offline logged data, which is also referred to as learning from demonstration [21, 26, 49]. Offline-to-one RL, instead, aims at enhancing the well-trained offline policy via online interactions. To ensure fast adaptation and stable policy improvement, many techniques are adopted, such as model ensemble [38], explicit policy constraints [45, 66). Offline-to-online fine-tuning will difficult if the trained value function or policy is overly pessimistic, which may lead to a suboptimal policy.

---

6 Conclusion
In this paper, we propose Mildly Conservative Q-learning (MCQ) to alleviate the over pessimism in existing offline RL methods. MCQ actively train OOD actions by constructing them proper pseudo target values following the guidance of the practical Mildly Conservative Bellman (MCB) operator. We theoretically illustrate that the policy induced by the MCB operator behaves at least as well as the behavior policy, and no erroneous overestimation will occur for the practical MCB operator. Furthermore, we extensively compare MCQ against recent strong baselines on MuJoCo locomotion tasks. Experimental results show that MCQ surpasses these baselines with a large margin on many non-expert datasets, and is also competitive with baselines on expert datasets. Moreover, we demonstrate the superior generalization capability of MCQ when transferring from offline to online. These altogether reveal that mild conservatism is critical for offline learning. We hope this work can promote the offline RL towards mild pessimism, and bring new insights into the community.
One drawback of our current algorithm lies in the need of tuning the weighting coefficient A. However, we empirically find that 0.7 S < can usually induce satisfying performance. We leave the automatic tuning of À as future work.
Acknowledgments and Disclosure of Funding
This work was supported in part by the Science and Technology Innovation 2030-Key Project under Grant 2021ZD0201 4044444444, in part by the NSF China under Grant 61872009. The authors would like to thank the anonymous reviewers for their valuable comments and advice.
References
[1] Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained Policy Optimization. In International Conference on Machine Learning, 2017.
[2] R. Agarwal, M. Schwarzer, Learning at the Edge of the Statistical Precipice, In Advances in Neural Information Procexsing Systems, 2021.
Castro, C. Courville, and M. G. Bellemare. Deep Reinforcement
[3] A. Ajay, Kumar, P. Agrawal, S. Levine, and Nachum. OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning. In International Conference on Learning Representations, 2021.
[4] C. Bai, Wang, Yang, Z.-H. Deng, Garg, Liu, and Wang. Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning. In International Conference on Leurning Representa- tions, 2022.
[5] D. Bang and H. Shim. MGGAN: Solving Mode Collapse Using Manifold-Guided Training. JEER/CVF International Conference on Computer Vision Workshops, pages 2347-2356, 2021.
[6] D. Bau, J.-Y. Zhu, J. Wulff, W. S. Peebles, H. Strobelt, Zhou, and A. Torralba. Seeing What GAN Cannot Generate. In IEEE/CVF International Conference on Computer Vision, pages 4501 -5100, 2019.
L. Chen, K. Lu, A. Rajeswaran, Lee, A. Grover, Laskin, Abbeel, A. Srinivas, and I. Mordatch. Decision Transformer: Reinforvement Learning via Sequence Modeling, ArXiv, abs/2106.01 2021.
[8] X. Chen, Z. Zhou, Z. Wang, C. Wang. Y. Wu, Q. Deng, and K. W. Ross. BAIL: Best-Action Imitation Learning Batch Deep Reinforcement Learnin, Advances in Neural Information Processing Systems. 2020.
[9] I. Csiszár and J. Körmr. Information Theory- Coding Theorems for Discrete Memoryless Systems, Second Edition. Cambridge University Press, 2011.
[10] C. P. Diehl, T. Sieveriic, M. Krüger, F. Hoffmann, and T. Bertram. UMBRELLA: Uncertainty- Model-Based Offline Reinforcement Learning Leveraging Planning, ArXiv, abs/2111.11ll 1097, 2021.
[11] R. Fakoor, J. Mueller, P. Chaudhari, and A. Smola. Continuous Doubly Constrained Batch Reinforcement Learning. In Advances in Neural Information Processing Systems, 2021.
[12] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4RL: Datasets for Deep Data-Driven Reinforce- ment Learning. ArXiv, 0t2/2004..7219,22222 2020.
10

---

[13] S. Fujimoto and S. S. Gu. A Minimalist Approach to Offline Reinforcement Learning, In Advances in Neural Information Processing Systems, 2021.
[14] S. Fujimoto. D. Meger, and D. Precup. Off-Policy Deep Reinforcement Learning without Exploration. In International Conference on Machine Learning. 2019.
[15] S. Fujimoto, H. van Hoof, and D. Meger. Addressing Function Approximation Error in Actor-Critic Methods, In International Conference on Machine Learning, 2018.
[16] C. Gelada and M. G. Bellemare. Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift. In AAAI Conference on Artificial Intelligence, volume 33, 2019.
[17] S. K. S. Ghasemipour, D. Schuurmans, and S. S. Gu. EMaQ: Expected-Mai Q-Learning Operator for
Simple Yet Effective Offiine and Online RL. In International Conference on Machine Learning, 2021.
[18] D. Ghosh, A. Ajay. P. Agrawal, and S. Levine. Offlin RL Policies Should be Trained to be Adaptive. ArXiv, 0202207700200, 2022.
[19] I. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde- S. Ozair, C. Courville, and Y. Bengio. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, 2014.
[20] T. Haarnoja, A. Zhou, Abbeel, and Levine. Soft CritieCritii: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In International Conference on Machine Learning, 2018.
[21] T. Hester, M. Vecerík, O. Pictquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Os- band, G. Dulac-Amol,, J. P. Agapiou, J. Z. Leibo, and A. Gruslys. Deep Q-learning From Demonstrations. In AAAI Conference Artificial Intelligence, 2018.
[22] M. Janner, J. Fu, M. Zhang, and S. Levine. When Trust Your Model: Model-Based Policy Optimization. In Advances in Neural Information Processing Systems, 2019.
[23] Janner, Q. Li, and S. Levine. Offline Reinforcement Learning as One Big Sequence Modeling Problem. In Advances in Neural Information Processing Systems, 2021.
[24] Y. Jin, Z. Yang, and Z. Wang. Is Pessimism Provably Efficient for Offline RL? In International Conference on Machine Learning, pages
PMLR, 2021.
[25] S. M. Kakade and J. Langford. Approximately Optimal Approximate Reinforcement Learning. In International Conference on Machine Learning, 2002.
[26] B. Kang, Z. Jic, and J. Feng. Policy Optimization with Demonstrations. In International Conference on Machine Learning, 2018.
[27] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. MORel.: Model- Based Offline Reinforcement Learning. In Advances in Neural Information Processing Systems, 2020.
[28] D. Kingma
Ba. Adam: A Method Stochastic Optimization. In International Conference on Learning Representation, 2015.
[29] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. ArXiv, abs/ 1bs/133..6114, 2013.
[30] V. R. Konda and V. S. Borkar. Actor-Critic - Type Learning Algorithms for Markov Decision Processes.
SIAM J. Control. Optim., 38:94-123, 1999.
[31] V. R. Konda and J. N. Tsitsiklis. Actor-Critic Algorithms. In Advances in Neural Information Processing Systems, 1999.
[32] I. Kostrikov, A. Nair, and S. Levine. Offline Reinforcement Learning with Implicit Q-Learning. In International Conference on Learning Representations, 2022.
[33] Kostrikov, J. Tompson, R. Fergus, and Nachum. Offline Reinforcement Learning with Fisher
Divergence Critic Regularization. In International Conference on Machine Learning, 2021.
[34] A. Kumar, J. Fu, G. Tucker, and S. Levine. Stabilizing Off-Policy Q-Learning via Bootstrapping Emor Reduction. In Advances in Neural Information Processing Systems, 2019.
[35] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems, 2020.
11

---

[36] S. Lange, T. Gabel. and M. A. Riedmiller. Batch Reinforcement Learning. In Reinforcement Leurning. 2012.
[37] B.-J. Lee, J. Lee, and K.-F. Kim. Representation Balancing Offline Model-based Reinforcement Learning. In International Conference on Learning Representations, 2021.
[38] S. Lec, Y. Sco, K. Lee, P. Abbeel, and J. Shin. Offline-to-Onling Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble. In Conference on Robot Learning, 2021.
[39] Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill. Off-Policy Policy Gradient with State Distribution Correction, ArXiv, tbss/1994.0847,, 2019.
[40] X. Ma, Y. Yang. H. Hu, J. Yang, C. Zhang, Q. Zhao, B. Liang, and Q. Liu. Offliee Reinforcement Learning with Value-based Episodic Memory. International Conference on Learning Representations, 2022.
[41] Y. J. Ma. D. Jayaraman, and O. Bastani. Conservative Offline Distributional Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 34, 2021.
[42] T. Matsushima, H. Furuta, Y. Matsuo, O. Nachum, and S. Gu. Deployment-Efficien Reinforcement Learning via Model-Based Offline Optimization. In International Conference on Learning Representations, 2021.
[43] L. Meng. M. Wen, Y. Yang. C. Lc, X. Li, W. Zhang, Y. Wen, H. Zhang, J. Wang, and B. Xu. Offfine Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks. ArXiv, abs/21 12.02845, 2021.
[44] O. Nachum, Y. Chow, B. Dai, and I.. Li. DualDICE: Behavior- Estimation of Discounted Stationary Distribution Corrections. In Advances in Neural Information Processing Systems, 2019.
[45] A. Nair, M. Dalal, A. Gupta, and S. Levine. Accelerating Online Reinforcement Learning with Offline Datasets. ArXiv, abs/2006.09359, 2020.
[46] Y. Ovadia, E. Fertig, J. Ren. Z. Nado. D. Sculley, S. Nowozin, J. V. Dillon. B. Lakshminarayanan, and
J. Snock. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. Advances Neural Information Processing Systems, 2019.
[47] D. Precup. R. S. Sutton, and S. Dasgupta. Off-Policy Temporal Difference Learning with Function Approximation. International Conference on Machine Learning. 2001.
[48] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Offline Reinforcement Learning from Images with Latent Space Models. In Learning for Dynamics and Control (L4DC), 2021.
[49] A. Rajeswaran, V. Kumar, A. Gupta, Schulman, F. Todorov, and Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. ArXiv, abs/ 10087, 2018.
[50] P. Rashidinejad, B. Zhu, C. Ma, J. Jiao, and Russell, Bridging Offline reinforcement Learning and Imitation Learning; Tale of Pessimism. Advances in Neural Information Processing Syxtems, 34, 2021.
[51] K. Sohn, H. Lee, and X. Yan. Leaming Structured Output Representation using Deep Conditional Generative Models. In Advances in Neural Information Processing Systems, 2015.
[52] A. Srivastava, L. Valkov, C. Russell, M. Gutmann, and C. Sutton. VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning. In Advances in Neural Information Processing Systems, 2017.
[53] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018
[54] R. S. Sutton, A. R. Mahmood, and M. White. An Emphatic Approach to the Problem of Off-policy Temporal-Diffeee Difference Learning. Journal of Machine Learning Research, 17:2603-2631, 2016.
[55] J. Wang, W. Li, H. Jiang, G. Zhu, S. Li, and C. Zhang. Offline Reinforcement Learning with Reverse Model-based Imagination. In Advances in Neural Information Processing Systems, 2021.
[56] Q Wang, J. Xiong, L. Han, P. Sun, H. Liu, and T. Zhang. Exponentially Weighted Imitation Learning for Batched Historical Data. In Advances in Neural Information Processing Systems, 2018.
[57] Z. Wang, A. Novikov, K. Zolna, J. T. Springenberg, S. E. Reed, B. Shahriari, N. Siegel, J. Merel, C. Gulcehre, N. M. O. Heess, and N. de Freitas. Critic Regularized Regression. In Advances in Neural Information Processing Systems, 2020.
12

---

[58] Y. Wu, G. Tucker, and O. Nachum. Behavior Regularized Offline Reinforcement Learning. ArXiv, 2019.
[59] Y. Wu, S. Zhai, N. Srivastava, J. M. Susskind, J. Zhang. R. Salakhutdinov, and H. Goh. Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning. In International Conference Machine Learning, 2021.
[60] T. Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal, Bellman-Consistent Pessimism for Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 34, 2021.
[61] S. Yang, Z. Wang. H. Zheng. Y. Feng, and M. Zhou. A Regularized Implicit Policy for Offline Reinforce- ment Learning, ArXiv, abs/2202.09673, 2022.
[62] Y. Yang, X. Ma, I.. Chenghao, 7. Zheng, Q. Zhang, G. Huang, J. Yang, and Q Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforeement learning. Advances in Neural Information Processing Systems, 34, 2021.
[63] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. COMBO: Conservative Offline Model-Based Policy Optimization. In Advances Neural Information Processing Systems, 2021.
[64] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. MOPO: Model-based Offline Policy Optimization. In Advances in Neural Information Processing Systems, 2020.
[65] A. Zanctte, M. J. Wainwright, and F. Brunskill. Provable Benefits of Actor-Critie Methods for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems, 2021.
[66] Y. Zhao, R. Boney, A. Ilin, J. Kannala, and J. Pajarinen. Adaptive Behavior Cloning Regularization for Stable Offline-to fo-Online Reinforcement Learning, 2022.
[67] Zhou, Bajracharya, and D. Held. PLAS: Latent Action Space for Offfine Reinforcement Learning. In Conference on Robot Learning. 2020.
Checklist
1. For all authors...
(a) Do main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethies review guidelines and ensured that your paper conforms to them? [Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosea)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
If you are using existing assets (e.g., code, data, models) or curatingfreleasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're us-
ing/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
13

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent
participant compensation? [N/A]
14


Reviews:
[{"review_id": "NeJy062t6gK", "reviewer": "Reviewer_fVHB", "summary": "This paper proposes using mild conservatism in offline RL to benefit generalization and to avoid overly conservative on OOD actions.\nSpecifically, this paper develops a Mildly Conservative Bellman (MCB) operator for offline RL, where OOD actions are actively trained and their Q values are actively queried.\nTheoretical results under the tabular setting and a practical MCB operator are provided.\nEmpirically, combining the practical MCB operator with SAC performs well on the D4RL MuJoCo locomotion tasks and when transferring from offline learning to online.", "questions": "Below are my questions and concerns on this paper.\n\n1. From Proportion 5, even under the rather strong assumption on the discrepancy between $\\hat \\mu$ and $\\mu$, is it still possible that the OOD actions will Q-values higher than the supremum of the in-distribution Q-values? If so, how the proposed Q-learning-based method avoids the detrimental impact of those \"overestimated\" OOD actions?   \n2. Why in the proof of Proportion 4 (Below L72 of Appendix), we have $\\\\{a_i'\\\\}^N \\sim \\mathrm{support}(\\mu)$? And the $\\hat T_1$ here seems inconsistent with Eq. (9).\n3. Why in L183-184 \"in-distribution actions are still trained to approximate the optimal batch-constraint Q value\"? Is it possible that the target value of such in-distribution actions are inflated by the psedo-label? \\\nSince Eq. (13) is independent of $a^{ood}$ and $\\pi$, will the psedo-target value Eq. (13) collapese $Q(s^{in}, a^{ood})$ at different $a^{ood}$ into a same value, even those in-distribution? If so, how could the proposed the method select out good in-distribution actions?", "limitations": "The authors briefly addressed the limitations.\nNo potential negative societal impacts is discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The proposed method is well-motivated by theory.\n2. The idea of actively train the Q-values of OOD actions is interesting, though in some sense similar to the CQL paper [1].\n3. Experiments are extensive and the proposed method generally performs well.\n\n### Weaknesses\n1. Since the proposed method requires behavior cloning (BC), it is doubtful if the proposed method can work well on higher-dimensional and/or non-Markovian datasets where BC can be difficult. Theoretical results such as Proportion 5 requires sufficiently accurate BC, which may not be possible on harder datasets. \\\nIn fact, from Table 8, the proposed method performs less favorable on maze2d datasets compared with other stronger baseline such as OptiDICE [2], and is slightly worse than one-step RL [3] on Adroit dataset.\n2. There seems to be discrepancies between the impractical theoretical algorithm and the theoretically less-supported practical algorithm, in particular the practical Loss functions (L167-193). \n3. From Table 4 & 9, the proposed method requires per-dataset tuning of the weighting coefficient $\\lambda$ on a relatively fine grid, which muds the empirical significance since many of the compared baselines actually *unify* hyperparameters across the MuJoCo datasets, *e.g.*, CQL, IQL and TD3+BC.\n\n\n\\\n[1] Kumar, Aviral, et al. \"Conservative q-learning for offline reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020): 1179-1191.\n\\\n[2] Lee, Jongmin, et al. \"Optidice: Offline policy optimization via stationary distribution correction estimation.\" International Conference on Machine Learning. PMLR, 2021.\\\n[3] Brandfonbrener, David, et al. \"Offline rl without off-policy evaluation.\" Advances in Neural Information Processing Systems 34 (2021): 4933-4946.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Esk2paR47UX", "reviewer": "Reviewer_EAmG", "summary": "The paper presents a method for offline reinforcement learning based that involves assigning pseudo-Q-values to OOD values called MCQ. The method's main idea is to modify the Q-targets by detecting the out-of-distribution actions via a density model and assign the Q values to these actions similarly to BCQ by taking a maximum of a Q function over N samples from a density model. The main difference to BCQ is that the authors propose to use this backup operator only for OOD actions instead of all actions and use an actor to recover optimal actions from the modified Q-function. The method is evaluated on severals datasets from D4RL where it outperforms the baselines.", "questions": "* Did you try implementing the version of the method that regularizes only OOD actions?\n* How well does the method perform on other D4RL datasets (for example, antmaze, adroit and kitchen tasks)?\n* What implementations of baselines did you use for offline to online experiments?", "limitations": "Right now, the main limitation of this work is limited experimental evaluation. In particular, the method is evaluated only on locomotion tasks using an insufficient number of runs. The paper considers a set of tasks different from the standard tasks used in prior work for offline to online experiments. I will raise my score if these concerns are addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* The method is theoretically sound and practical. Even though all method components have been explored in prior work, the idea of not penalizing the values of OOD actions but using a BCQ-style value estimate to impute the values for these actions is novel.\n* The practical version of the algorithm (Eqn. 11) is easy to implement. \n* MCB demonstrates good empirical results on a subset of D4RL tasks.\n* The paper is well written and easy to follow.\n\n# Weaknesses\n* The method can be seen as an extension of BCQ. However, the comparison to BCQ is missing.\n* The practical implementation of the method diverges from the theory. In particular, the practical implementation of the method omits OOD evaluation and instead regularizes all actions sampled from the training policy, which are not necessary OOD and, throughout training, will certainly become less OOD which can result in over-penalizing the optimal actions due to value underestimation caused by the BCQ-style operator.\n* The method is evaluated only on the locomotion tasks from D4RL, which do not require stitching [1] (dynamical programming).\n* Also, the method is evaluated using only 4 seeds which might be insufficient.\n* The offline to online experiments miss essential details. In particular, it is unclear how the authors obtained results for other methods. Also, the authors pick a different subset of tasks for offline to online finetuning than considered in the original papers (AWAC, IQL).\n\n[1] RvS: What is Essential for Offline RL via Supervised Learning?\nS Emmons, B Eysenbach, I Kostrikov, S Levine", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6gjmp8wu473", "reviewer": "Reviewer_5dg8", "summary": "This paper points out the previous offline reinforcement learning methods are too conservative about the out-of-distribution (OOD) actions and instead propose a mild conservative algorithm. It introduces an auxiliary loss term to properly train the value function for OOD actions. The proposed method, MCQ, is shown to outperform previous methods empirically and theoretically proved to behave at least as well as the behavior policy, and has no erroneous overestimation.", "questions": "Please see the main review.", "limitations": "The authors have addresses the limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nThis paper is clearly written, and the whole structure is organized and easy to follow. The method is well-motivated and the claims in the paper are all supported by either theoretical analysis or experimental results. Although over-pessimism is not a new problem in offline RL, this method is very innovative and elegant in its processing of OOD actions. In comparison with the baselines, MCQ archives a remarkable improvement on random or medium datasets. The authors also make a careful analysis of the sensitivity of hyperparameters.\n\nCons:\nI want to draw the authors' attention to a recent paper that also addresses the over-pessimism of offline RL algorithms (https://arxiv.org/pdf/2207.02200.pdf), which uses an adaptive method. Besides, I am also confused about the results in Figure 2. It seems that decreasing $\\lambda$ is always beneficial to the final performance, and there is no trend that introducing the auxiliary loss can help the final performance. I think the authors should use a small scale of $\\lambda$ (0-0.3) to explicitly show the benefits of the auxiliary loss.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3ExOO6GRtn_", "reviewer": "Reviewer_7rnm", "summary": "Offline RL is a topic of significant interest. One common class of approaches is to learn an action-value function but to enforce that the function is `conservative' so that it does not result in a policy which takes actions that were not in the training data (and therefore of unknown value).\n\nThis work introduces a ``mildly'' conservative Bellman operator. In particular, for actions in the support of the behavior policy the operator behaves like standard Bellman operator, but for actions outside the behavior policy support it assumes that value is delta less than an action in the support.\n\nThey show that this operator will always result in a conservative Q estimate (that is, it will not over-estimate the value of any action). They then introduce a practical approximation (where the behavior policy is estimated by a CVAE) and test on a set of offline RL control tasks. It performs notably better than prior work on poor demonstrates, but does not consistently outperform TD3+BC when expert demonstrations are available.\n", "questions": "Why do you think TD3+BC seems to be better for expert-level demonstrations (for most tasks)?\n\nThe checklist for this paper indicates the code is available for reproducing the experiments but I didn't see a link anywhere to the code?\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Well communicated.\n\n- Principled explanation of an algorithm that empirically performs well on a set of benchmark tasks.\n\n- Offline RL is a topic of significant interest to the community and active research.\n\nWeaknesses:\n\n- Ideally, the algorithm would be tested on a different style of tasks as well (e.g. perhaps Atari), rather than only MuJoCo control tasks.\n\n- This approach does not seem to perform as well when expert level demonstrations are available.\n\nMinor:\n\nI found definition 2 (line 123) confusing since it refers to $\\mu(a|s)$ which is stated above you are trying to avoid. It is explained further what the actual \"practical\" solution is when $\\mu$ is not known, but I found this a bit confusing on first read.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept