ACQL: An Adaptive Conservative Q-Learning Framework for Offline Reinforcement Learning
Authors: Kun Wu
Yinuo Zhao
Zhiyuan Xu
Zhengping Che
Chengxiang Yin
Chi Harold Liu
Qinru Qiu
Feifei Feng
Jian Tang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Reinforcement Learning
Offline Deep Reinforcement Learning

Abstract: Offline Reinforcement Learning (RL), which relies only on static datasets without additional interactions with the environment, provides an appealing alternative to learning a safe and promising control policy. Most existing offline RL methods did not consider relative data quality and only crudely constrained the distribution gap between the learned policy and the behavior policy in general. Moreover, these algorithms cannot adaptively control the conservative level in more fine-grained ways, like for each state-action pair, leading to a performance drop, especially over highly diversified datasets. In this paper, we propose an Adaptive Conservative Q-Learning (ACQL) framework that enables more flexible control over the conservative level of Q-function for offline RL. Specifically, we present two adaptive weight functions to shape the Q-values for collected and out-of-distribution data. Then we discuss different conditions under which the conservative level of the learned Q-function changes and define the monotonicity with respect to data quality and similarity. Motivated by the theoretical analysis, we propose a novel algorithm with the ACQL framework, using neural networks as the adaptive weight functions. To learn proper adaptive weight functions, we design surrogate losses incorporating the conditions for adjusting conservative levels and a contrastive loss to maintain the monotonicity of adaptive weight functions. We evaluate ACQL on the commonly-used D4RL benchmark and conduct extensive ablation studies to illustrate the effectiveness and state-of-the-art performance compared to existing offline DRL baselines.

Under review as a conference paper at ICLR 2023
ACQL: AN ADAPTIVE CONSERVATIVE Q-LEARNING FRAMEWORK FOR OFFLINE REINFORCEMENT LEARN- ING
Anonymous authors Paper under double-blind review
ABSTRACT
Offline Reinforcement Learning (RL), which relies only on static datasets without additional interactions with the environment, provides an appealing alternative to learning safe and promising control policy. Most existing offline RL methods did not consider relative data quality and only crudely constrained the distribu- tion gap between the learned policy and the behavior policy in general. More- over, these algorithms cannot adaptively control the conservative level in more fine-grained ways, like for each state-action pair, leading to a performance drop especially over highly diversified datasets. In this paper, propose Adaptive Conservative Q-Learning (ACQL) framework that enables more flexible control over the conservative level of Q-function for offline RL. Specifically, two adaptive weight functions to shape the Q-values for collected and out-of- distribution data. Then discuss different conditions under which the conser- vative level of the learned Q-function changes and define the monotonicity with respect data quality and similarity. Motivated by the theoretical analysis, we propose a novel algorithm with the ACQL framework, using neural networks as the adaptive weight functions. To learn proper adaptive weight functions, we de- sign surrogate losses incorporating the conditions for adjusting conservative levels and a contrastive loss to maintain the monotonicity adaptive weight functions. We evaluate ACQL on the commonly-used D4RL benchmark and conduct exten- sive ablation studies to illustrate the effectiveness and satee-oftheewee performance compared to existing offline DRL baselines.
present
INTRODUCTION
With the help of deep learning, Reinforcement Learning (RL) has achieved remarkable results on variety of previously intractable problems, such as playing video games (Silver et al.) 2016). con- trolling robot (Kalashnikov et al. 2018 Akkaya et al. 2019) and driving autonomous cars (Yu al.) 2020a Zhao et al.] (002)) However, the prerequisite that the agent has to interact with the envi- ronments makes the learning process costly and unsafe for many real-world scenarios. Recently, offline RI (Lange et al. 2012, Prudencio et al. 2022) has been proposed as promising alternative to relax this requirement In offline RI., the agent directly learns control policy from a given static dataset, which is mreviously-collette by an unknown behavior policy. Offline RL enables the agent to achieve comparable or even better performance without additional interactions with environment.
Unfortunately, stripping the interactions from the online RL, offline RL is very challeng- ing due to the distribution shift between the behavior policy and the learned policy. It of- ten leads to the overestimation of values of out-of-distribution (OOD) actions (Kumar al.) 2019 Levine et al.) 2020) and thus misleads the policy into choosing these erroncously esti- mated actions. To alleviate the distribution shift problem, recent methods (Kumar al.) 2019 Jaques et al.) [009 [Wu et af.) [219 Siegel et al.] 2020) proposed to constrain the learned policy to the behavior policy in different ways, such as limiting the action space (Fujimoto et al.) 2019), using KL divergence (Wu et al. 2019) and using Maximum Mean Discrepancy (MMD) (Kumar et al. 2019). Besides directly constraining the policy, other methods (Ku- mar et al.) 2020; Yu et al.) 2021abb) Ma et al., (021) choose to learn a conservative Q-function to constrain the policy implicitly and thus alleviate the overestimation problem of Q-function.

---

Under review as a conference paper at ICLR 2023
However, most previous methods optimize all transition samples equally rather than selec- tively adapting, which may be overconserva- tive especially for those non-expert datasets with high data diversity. As shown in Fig- ure [ more conservative (with a larger al- pha) CQL (Kumar al. 2020) agent achieves higher returns on expert dataset while suffering from performance degradation on the random dataset, indicating for high-quality data, higher conservative level works better and vice versa. It clearly shows the importance of the conser- vative level on the final results. Therefore, it is more proper use adaptive weights for differ- ent transition samples to control the conserva- tive level, such raising the Q-values more for good actions and less for bad actions.
aggha1
alca-55
neturn
20-
Pxpert
Medium
Pandom
Dataset Types.
Figure 1: Performance gaps of CQL with different conservative levels on HalfCheetah tasks.
In this paper, focus how constrain the Q-function in a more flexible way and propose general Adaptive Conservative Q-Learning (ACQL) framework, which sheds the light on how to design proper conservative Q-function. To achieve more fine-graines control over the conservative level, we use two adaptive weight functions to estimate the conservative weights for each transition sample. In the proposed framework, the form of the adaptive weight functions is not fixed, and we are able define particular forms according to practical needs. We theoretically discuss in detail that the correlation between the different conservative levels and their corresponding conditions that the weight functions need to satisfy. We also formally define monotonicity of the weight functions to depict the property that weight functions should raise the Q-values more for good actions and less for bad actions.
With the guidance of theoretical conditions, propose practical algorithm with learnable neu- ral networks as adaptive weight functions. Overall, ACQL consists three components. Firstly, preprocess the fixed dataset calculate the transition quality measurements, showing the data quality and similarity as pseudo labels. Then, with the help of the measurements, we construct sur- rogate losses keep the conservative level of ACQL between the true Q-function and CQL. We also add contrastive loss to maintain the monotonicity of adaptive weight functions with respect to data quality and similarity. Lastly, we train the adaptive weight functions, actor network and critic network alternatively.
We summarize our contributions as follows: We propose more flexible framework ACQL that supports the fine- tne-grained control of the conservative level in offline DRL. 2) We theoretically analyze how the conservative level changes conditioned on different forms of adaptive weight functions. 3) With the guidance of the proposed framework, we present novel practical algorithm with carefully designed surrogate and contrastive losses to control the conservative levels and monotonicity. 4) We conduct extensive experiments on the D4RL. benchmark and the state-of-the-art results well demonstrate the effectiveness our framework.
RELATED WORK
Imitation Learning. To learn from given static dataset, Imitation Learning (IL) is the most straightforward strategy. The core spirit of is mimic the behavior policy. As the simplest form, behavior cloning still hold place for offline reinforcement learning, especially for expert dataset. However, having expert datasets only a minority cases. Recently some methods (Chen) al. 2020 Siegel et al.) 2020: Wang et al. 2020 Liu al. 2021) aim to filter sub-optimal data and then apply the supervised learning paradigm afterward. Specifically. BAIL (Chen et al. 2020) performed imitation learning only
high-quality subset of dataset purified by a learned value function. However, these methods often neglect the information contained in the bad actions with lower returns and thus often fail in tasks with non-optimal datasets. We believe these methods
2

---

Under review as a conference paper at ICLR 2023
beneficial to our framework ACQL, since they split different data regions where different conserva- tive levels can be set.
Model-free Offline RL. A large number of model-free offline RL methods aim to maximize the returns while constraining the learned policy and the behavior policy to be close enough. There are various ways for the direct constraint
policy including minimizing the KL--ivergence (Jaques) et al. 2019; Wu et al. 2019 Zhou et al., 2020). MMD (Kumar et al., 22011) or Wasserstein dis- tance [Wu et al., 2019) and adding behavior cloning regularization (Fujimoto & Gu 2021). The policy can be also constrained implicitly by actions space reduction (Fujimoto et al. 2019). im-
portance sampling based algorithms (Sutton al., 2016 Nachum et al. | 2019), the implicit form quantifica-
of KL-divergence (Nair et al., 2020; PPng et al. 2019; Simāo et al.]
tion (Agarwal et al. 2020; Kumar al.) 2019) or conservative Q-function (Kumar et al. 2020
Ma et | et al. 2022). More recently, Onestep RI. (Brandfonbrener et al.) [22]] and
IQL (Kostrikko al. 202Tb) proposed to improve the policy after the convergence of Q functions.
Trajectory Transformer (TT) (Janner al. 202I) and Decision Transformer (DT) (Chen et al.
202T] leverages the advantage of Transformer (Vaswani et al. 2017) to optimize on trajectories. In this work, we propose flexible framework ACQL for the model-free methods that constrain on Q-functions. ACQL supports defining different conservative levels for Q-function over each state- action pair, where CQL (Kumar al. 2020) one special case when all levels equal.
Model-based Offline RL. Recently, model-based methods (Janner et al.] 2019 Kidambi et al. 2020 Yu et al. 2020b 2021b Matsushima et al.) 2020) attract much attention for offline RL. They first learn the transition dynamics and reward function as a proxy environment which can be subsequently
used for policy search. Given the proxy environment, offline methods (Ross & Bagnell 2012
Kidambi et al. 2020), or run planning and trajectory optimization like LQR (Tassa et al. [2 22) and MCTS (Browne et al. (2012) can be directly used for controlling, Although model-based offline RL can be highly sample efficient, direct use of it can be challenging due to distribution shift issue. In this paper, we only focus on model-free offline RL.
PROBLEM STATEMENT
We consider the environment as fully-observed Markov Decision Process (MDP), which is rep- resented by a tuple (S, A, P. p.r.Pu.7)) 7o. The MDP consists of the state space S, the actions space A. the transition probability distribution function P S x A S - [0. 1]. the reward fune- tion r : S x A x S + IR, the initial state distribution po(s) and the discount factor y E (0, 1)- The goal is to leam control policy (als) that maximizes the cumulative discounted return G = > 7'r(st, at, st-1 ~ Po, ag ~ (-|sst,, St+1 P(-|st, az)). In the Actor-Critic frame- work, the learning process repeatedly alternates between the policy evaluation that computes the value function for policy and the policy improvement that obtains better policy from the value function. Given a current replay buffer (dataset) D {ssaarr,s)) consisting of finite transition
samples, the policy evaluation is defined as follows:
Qu+1 arg min E. Ea Q
where le is the iteration number and the Bellman operator is defined as B B= (s, = r(s, + (a'
(Q"(s (Q""ss''')) Note that the empirical Bellman operator B= is used in practical, which backs up only one transition, because it is difficult to contain all possible transitions (s, a, s') in D, especially for continuous action space. After approximating the Q-function, the policy improvement is performed as the following:
k+ +- arg max Eg- (als)
[Qtt
(2)
Compared to online RL, offline RL only allows learning from a fixed dataset D collected by an unknown behavior policy Tg, while prohibiting additional interactions with the environment. One of core issues in offline RL is the existence of the action distribution shift during training (Kumar Levine et al. 2020).
et al. 2019 Wu et al. 2019 Jaques al. 2019
3

---

Under review as a conference paper at ICLR 2023
4 ADAPTIVE CONSERVATIVE Q-LEARNING FRAMEWORK (ACQL)
In this section, we propose a general framework, Adaptive Conservative Q-Learning (ACQL), which enables Q-function constrained algorithms (Kumar et al., 2020 Ma et al. 2021] Yu et al. 2021a b). With- loss of generality, we can consider the dataset collected by the behavior policy usually contains the data with both high and low returns, even though the behavior policy is random policy. At the same time, among the actions sampled from the particular distribution μ4, there are also (relatively) good and bad actions instead of all actions having the same returns. In that case, we need a more flexible and fine-grained control method to constrain the Q-function for each state-action pair. To- wards our goal, we propose to use two adaptive weight functions (s, a) and d (8, a) to control the conservative level over the distribution μ and empirical behavior policy 7g. respectively. Now the family of optimization problems of our framework ACQL is presented as below:
flexible control over the conservative level of O-function, compared to other
min max (Eg~ (aand(a)))) [du(s, a) Q(s, a)] - Q
(d a) Q(s,a)))
21 + R(p).
+ E S
(3)
Note that the form of the adaptive weight functions d=,(s, a) and dr (s, a) is not fixed and can be customized according to different situations. the Q-function more finely. In the following, we discuss the conditions about how the conservative level of ACQI. compared to the true Q-function and CQL (Kumar et al.) 2020). and the properties that the adaptive weight functions (y a) and dug (s, a) should have. First, we list different conditions on which ACQL is more conservative than the true Q-function in different levels in Proposition
their arbitrary form that supports us in shaping
adjust
(The conservative level of ACQL). For any p. with supp μs C. supp R.. without considering the sampling error between the empirical B7Th Q and true Bellman buckups B Q, the conservative level of ACQL can be controlled over the Q-values. The learned Q-function Q is more conservative than the true Q-function Q" point-wise, if:
Proposition 4.1
Vs E D.a, 0, dy dxp g T 2 0.
(4)
Moreover, as shown in Appendix A we can also control the conservative level over other regions like the V-values or empirical MDP by changing the state, action space in Equation 4 As a special instance of our framework ACQL, CQL (Kumar et al., 2020) proposes to constraint the conservative level over the excepted V-values. And CQL performs the optimization for all aaateacction pairs with the same weight a, which may be too rigid and over conservative for some scenarios. Now, if we replace the "2" to "<" in Equation (4). we can get the conditions on which ACQL is less conser- vative than the true Q-function. Following (Auer al. 2008 Osband et al.) 2016 Kumar al.) 2020), we also show that ACQL bounds the gap between the learned Q-values and true Q-values with the consideration of the sampling error between the empirical 8 Q and actual Bellman opera- tor B² 3.Q. Due to the space limitation, we present the proposition and proof in Appendix A Besides the comparison to the true Q-function, we also give a theoretical discussion about the comparison to CQL (Kumar et al. 2020) in the following.
Proposition 4.2 (The conservative level compared to CQL). For any with supp C. supp ft. given the Q-function learned from CQL is Teq = Q" - a "=ag, similiar as in 4.1 the conservative level of ACQL compared to CQL can be controlled Q-fanction Q is less conservative than the CQL Q-function Qoqr point-wise, if:
the Q-values. The Tearned
Vs E D.a, (a dy) (a dxg )
2 0.
(5)
T
Besides the discussion about the conservative level of ACQL, to depict the property that good action should have a higher Q-value than a bad action, we formally define the monotonicity of the adaptive weight functions following:
4

---

Under review as a conference paper at ICLR 2023
Definition 4.1 (The monotonicity of the adaptive weight functions). For any state D. the monotonicity of the adaptive weight functions is defined as that a good action a E A with a higher true Q value has lower d, value and higher dıg value:
0z, Og E p(aass), (s 0;) du(sj, 0z) x -((,,,))-Q((s,4)) Vsi, $j E D. aj E # d=p (8,, a.) - d7p ($j. az) x (si a;) (s), (7)
Vsss 83 E D,,
(6)
where Q" is the optimal Q-function. In Definition 4.1 we use the optimal Q-function which is a natural ideal metric to measure the action quality define the monotonicity. However, it is ill- posed problem since if we know the optimal Q-function, we can directly train an optimal policy and solve the problem. And we also do not know what the optimal proportional relationship should be. In the next section 5 we construct a contrastive loss using transition quality measurements to approximate this property for ACQL. All the proofs are provided in Appendix
ACQL WITH LEARNABLE WEIGHT FUNCTIONS
In this section, derived from the theoretical discussion, we propose one practical ACQL algorithm in the guidance of theoretical conditions with learnable neural network as adaptive weight functions. To adaptively control the conservative level, there are three steps for ACQL. Firstly, we preprocess the fixed dataset to calculate the transition quality measurements. Then, with the help of the transition quality measurements, we construct surrogate losses to control the conservative level of ACQL. We also add contrastive losses maintain the monotonicity. Lastly, we train the adaptive weight functions, actor network and critic network alternatively.
5.1 TRANSITION QUALITY MEASUREMENTS
To seek replacement for the optimal Q-function Q' Definition 4.1 we firstly preprocess the fixed dataset and calculate the transition quality measurements for each state-action pair. Note that different tasks may have different magnitudes of Q-values, thus it is better to seek a normalized measurement ranging in 1). For the action in the fixed dataset, define Relative Transition Quality mss, a) by combining both the single step reward and the whole discounted trajectory return measure the data quality:
Vis a) E D, a) = (Fnorm a) + Inorm (8, a));
(8)
Imas Amin The T'in, Tmar Jmin: Imaz are the min- imum and maximum values of the single step reward and the Monte Carlo return of the whole trajectory in the dataset respectively. Note that the range of vn(s, a) is (0, 1) and the higher mn(s, a) indicates that a is a better action. For the OOD actions, since they do not appear in the dataset and are not able to use the single rewards and Monte Carlo returns, choose to use both the quality of its corresponding in-dataset action m(s ain) and the Euclidean distance between the OOD action and action in the dataset with the same state.
where Tnorm Tour Teur-Fmas Tman and Inorm 9our gmiin Tmaz -aain
V(s, a E D, E p(als), m(s, = (11mmmmm,, a.)) - a:nll2 + +1).
m(ss
Consistent with m(s, din), we shift and scale in Equation to make the range of m(ss az) is also (0, 1) and the higher m(s, a,)) indicates that a₂ is a better action. Note that m(s, ap) is calculated over every training batch. We argue that the above Equations X and [ is only a simple and effec-
tive way to be the replacement of the optimal Q-function and it can be served a baseline for
future algorithms. Many methods including the "upper envelope" in BAIL (Chen et al. 2020 and uncertainty estimation in YYu et al. 2020b) have the potential be incorporated ACQL.
5.2 OPTIMIZATION TO CONTROL THE CONSERVATIVE LEVEL
Suppose present the functions d dr as deep neural networks, the key point is how to design the loss functions. Recapping the Equations 4 and (5). we have known the conditions for different conservative level and thus adapt them to the loss functions. More specifically, incorporate
5

---

Under review as a conference paper at ICLR 2023
the conditions, which aims learn Q-funtion conservative than true Q-function but less conservative than CQL, into the hinge losses as shown in the following:
Let true trueldyy dry! - m0a((0, dag TT dy 14 + C1 TT))
(10)
L Lot_egt(d dr,)) maz(0, (d,, - a) μ (dm, a) T C2 -TB);
(11)
where C₁ and C2 are used to control the soft margin of the conservative level compared to the true Q-function and CQL respectively. higher C1 means a more conservative Q-function than true Q- function, while higher C2 represents a less conservative Q-function than CQL. However, C₁ and C2 are difficult to tune and not adaptive enough as fixed hyperparameters. We leverage the relative transition quality to calculate C1 and C2 automatically based on the property in Definition 4. that a good action should have less conservative Q-value and thus a lower C₁ and a higher C2.
Ci a) = m(s,a)) Tmmas
(12)
m(s, a) Tmma-
(13)
The ranges of C1 and C2 are both (m,z)) Note that C1 and C2 are the soft margin over the Q-values, thus we use the maximum single reward I' 'maa as the maximum margin to avoid excessive fluctuations in policy evaluation. Nevertheless, during the optimization for Lol_true and Ldl_ogs we find it is prone to cause arithmetic underflow since the log,,, logn3 is usually very small like -1000, then the resulting j(a(s) and (als)) become O after exponentiation operation. To avoid the arithmetic underflow problem, we use a necessity of Equations (4) and (5) to form surrogate losses.
Lemma 5.1 For x > 0, I S x
1. In = x - 1 if and only if = 1.
The proof is provided in Appendix A Then the resulting surrogate losses are the following:
Let tiltrueldp dxa) = mar (0, dxa (In + 1) dn (In 1) + C1 (In + 1)), (14) eeg (dus dap) = max (dn a)I e + +) - (d=, - (In + 1) + C2(In Te + 1)). (15)
5.3 OPTIMIZATION TO MAINTAIN THE MONOTONICITY
Besides the surrogate losses control the conservative levels of ACQI., also construct trastive loss to main the monotonicity as stated in Definition 4.1 We use Mean Squared Error (MSE) for simplicity. The contrastive loss defined as the following:
V(si, ai), (sj, az) & D, (Sk, ax), (s1, D.
(16)
Lmono (dus daz) (d |1(o(d1 ax)) (d (3), a1))) (mmm(sk, ax)) (m((s, a ,))) (18)
((d a,))) (a(m(s), a,)) amm(s a:)))
(17)
where we use the softmax operation (-)) over the current batch of the training data to unify the orders of magnitude between the adaptive weights and transition quality measurements.
5.4
FINAL OBJECT
To learn conservative Q-function, the adaptive weight should be positive, and thus we add a regu- larizer term as the following:
Lpoe (dus dmş) = (a ) + max -dxp);
(19)
The final object for training the adaptive weight functions dyy drgg is:
[(dys dxæ) trtetreel (du. (dy, LLd_eql drz)) + Lmonold dma) ) + Loos (duy dag).
(20)
ACQL is built on the top of the CQL (Kumar et al. 2020,, which sets P and R(H))
- (p,L(p, Uni)(a)) in Equation in Section 5 And the optimization problem becomes:
min max min (B (,a-Daaw)(aas) (s,(s, Q(s,a)] -E EE. (awao(als) [da (s, a) Q(s, a)])
du.dey
2 B (Q(s,0)-
DKL(T, (nif(())) I(d,, (21)
During the training process, add weight neural networks dy drş and train the weight networks, Q networks and policy networks alternatively. Due to the space limitation, the implementation details provided in Appendix B.

---

Under review as a conference paper at ICLR 2023
Table 1: Normalized results on D4RL Gym-MuloCo environments.
Task Name
BCQ
BEAR
BRAC-v
BRAC-p
AWR
TD3+BC
F-BRC
CQL
ACQL
balfcheetah-e-v0
107.0
108.2
3.8
105.7
108.4
103.5
97.7
halfcheetall-m-e-v0
35.8
64.7
53.4
41.9
44.2
52.7
97.9
93.3
62.4
93.2
halfehectah-m----U
38.4
38.2
38.6
47.7
45.4
40.3
43.3
43.2
46.2
46.4
halfchectah--m-vO
36.1
40.7
41.7
46.3
43.8
37.4
42.8
41.3
44.4
42.8
halfcheetah-r-vO)
2.1
2.2
25.1
31.2
24.1
2.5
10.2
33.3
35.4
31.3
balfcheetah-sum
219.4
267.0
166.0
161.3
299.9
319.5
291.9
311.4
hupper-e-v0
109.0
110.3
3.7
6.6
112.2
112.3
112.2
116.3
hupper-m-e-v0
111.9
110.9
1.9
112.2
112.4
98.7
111.8
hopper-m-r-v0
11.8
33.1
33.7
0.6
0.6
28.4
31.4
35.6
48.6
77.3
hopper-m-vD
29.0
54.5
52.1
31.1
32.7
35.9
99,5
99.4
58.0
85.8
hopper-r-v0
9.8
10.6
11.4
12.2
11.0
10.2
11.0
11.3
10.8
12.7
hupper-sum
271.5
303.8
48.4
52.8
366.3
371.0
328.3
403.9
walker-e-vO
125.7
106.1
0.0
-0.2
105.7
103.0
107.2
116.1
walker-m-e-v0
6.4
57.5
40.1
81.6
76.9
53.8
101.1
105.2
111.0
94.5
walker-m-r-vo
11.3
15.0
19.2
0.9
-0.3
25.2
41.8
26.7
45.2
walker-m-vO
53.1
81.1
77.5
17.4
78.8
79.2
81.8
walker-r-vO
1.6
4.9
7.3
1.9
-0.2
1.5
1.4
1.5
7.0
11.8
walker-sum
151.6
231.8
165.5
153.7
313.1
330.3
331.1
349.4
EXPERIMENTS
6.1 COMPARISONS TO OFFLINE RL BASELINES
We conducted the experiments on the commonly-used offline RL benchmark D4RL (Fu et al. 2020 which includes many task domains (Todorov et al. 2012 Brockman et al. 2016 Rajeswaran et al.
2017) and a variety of dataset types. Aiming to provide a comprehensive comparison, we compared ACQL many state-of- ooffthe-aartt model-free algorithms including behavioral cloning (BC), SAC-off (Haarnoja et al. 2018). BEAR (Kumar et al. 2019). BRAC [Wu et al. 2019), AWR (Peng al. 2019), BCQ (Fujimoto al. 2019). aDICE (Nachum et al. 2019) TD3+BC (Fujimoto & Gu 2021 Fisher-BRC (Kostrikov et al. 2021a). and CQL (Kumar et al. (020). For the sake of fair comparisons, we directly reported the results of all baselines from the D4RL whitepaper (Fu al.
2020) and their original papers. To be consistent with previous works, we trained ACQL, for 1.0 M gradient steps and evaluated for 10 episodes every 1000 training iterations. The results are the average values of 10 episodes over 3 random seeds and are obtained from the workflow proposed by (Kumar et al. 2021).
Gym-MuJoCo Tasks. The Gym-MuJoCo tasks include "halfcheetah", "hopper" and "walker" with kinds of dataset types each, ranging from expert data to random data. For brevity, we marked "-expert", --medium-experss
"-medium-replay", "-medium" and "random" "-e", "-m-e". "-m-r", "-m" and "-r" respectively. Table [ shows the normalized returns all 15 Gym-MuJoCo version-O tasks. We can observe that ACQL outperforms other baselines Hopper and Walker environments by a large margin. From the perspective of the dataset types, ACQL is very a balanced algorithm that achieves excelling results on all kinds of datasets with expert, medium and random data. We also built additaional comparisons to more state-of-the-ard model-free algorithms including Decision Transformer (DT) (Chen et al. (22T)) AWAC (Nair et al. (020). Onestep RL (Brandronbrener al. 2021). TD3+BC (Fujimoto & Gu 2021), IQL (Kostrikov et al.
2021a) and CQL (Kumar et al. 2020
on version-2 datasets Appendix C.I
Adroit Tasks. The adroit tasks (Rajeswaran et al. 2017) are high- nigh-dimensional robotic manipulation tasks with sparse reward and include expert data and human demonstrations from narrow distribu- tions. Table 2 shows the normalized returns of all 12 Kitchen tasks. ACQL delivers higher perfor- mance than other baselines on "cloned" datasets, i.e., with mixed expert data and human demonstra- tions.
Franka Kitchen Tasks. The Franka Kitchen tasks (uppta et al. 2019 include complex trajectories the offline datasets and aim to evaluate the "stitching" ability of the agent realistic kitchen environment. Table 3 shows the normalized returns of 3 Kitchen tasks. We can see that ACQL consistently exceeds CQL, BC and other baselines all kinds of datasets, since ACQL can control the conservative level adaptively.
AntMaze Tasks. The AntMaze tasks mimic real-world robotic navigation tasks that aim to control an "Ant" quadruped robot to reach the goal location from a start location with only 0-1 sparse rewards given. Due the space limitation, reported the experimental results in Appendix

---

Under review as a conference paper at ICLR 2023
Table 2: Normalized results on D4RL Adroit environments.
Task Name
BC
SAC-Off
BEAR
BRAC-p
BRAC-y
AWR
BCQ
aDICE
CQL
ACQL
door-cloned-v0
-0.1
0.0
-0.1
-0.1
-0.1
0.0
0.0
0.0
0.4
6.6
door-cxpert-v0
34.9
7.5
103.4
-0.3
102.9
99.0
0.0
101.5
101.5
door-human-v0
0.5
3.9
-0.3
-0.3
-0.3
0.0
9.9
0.2
hammer-cloned--0
0.8
0.2
0.3
0.3
2.1
hammer-cxpert-v0
125.6
25.2
127.3
0.3
39.0
107.2
0.3
86.7
77.3
hanmer-human-v0
1.5
0.5
0.3
0.3
0.2
1.2
0.5
0.3
4.4
10.6
pen-cloned-v0
56.9
23.5
26.5
1.6
-2.5
28.0
44.0
-2.9
39.2
47.6
pen-expert-vO
85.1
6.1
105.9
-3.5
-3.0
111.0
114.9
-3.5
107.0
143.1
pen-human-vO
34.4
6.3
-1.0
8.1
0.6
12.3
68.9
-3.3
37.5
54.7
relocate-clonddv00
-0.1
-0.2
-0.3
-0.3
-0.3
-0.2
-0.3
-0.3
-0.1
0.1
relocate-cxpert-v0
101.3
-0.3
98.6
-0.3
-0.4
91.5
41.6
-0.1
95.0
60.4
relocate-human-v0
0.0
0.0
-0.3
-0.3
-0.3
-0.1
-0.1
0.2
Table 3: Normalized results D4RL Franka Kitchen environments.
Task Name
BC
SAC-Off
BEAR
BRAC-p
BRAC-v
AWR
BCQ
aDICE
COL
ACQL
kitchen-commileee-v0
33.8
0,0
0.0
43.8
mitchen-mixed-v0
47.5
2.5
47.2
0.0
10.6
2,5
51.0
55.9
kitchen-partial-O0
33.8
0.0
13.1
0.0
0.0
15.4
18.9
0.0
49.8
51.7
6.2 COMPARISONS AMONG DIFFERENT CONSERVATIVE LEVELS
To demonstrate more intuitively that ACQL. can adaptively control the conservative level, we com- pared ACQL to CQL with different values ranging from 1, 2, 5, 10, 20. The different values represent different conservative levels and a higher a means more conservative. learning curves of ACQL and CQL with 5 kinds of on the Hopper-v0 environments. We also pro- vided a more detailed Figure and a Table the quantitative results in Appendix C22 From the
2lotted the
Figure one trend we can clearly observe that CQL with higher conservative level can usually achieve higher performance on expert datasets as opposed to that lower conservative level is more effective for random datasets. Moreover, it is difficult for CQL achieve satisfactory performance on all kinds of dataset with fixed a. It is exactly the issue that the ACQL focuses on. Since ACQL generates adaptive weight for each state-action pair and control the conservative level in a more fine-grained way, ACQL can achieve balanced and state-of-tie -ote--f-----tttttt results for different dataset types.
Table 4 shows the comparisons of average Q-values over the datasets on HalfCheetah- vO environ- ments. We can observe that as the of CQL increase, the learned Q-values decreases, since a higher represents a higher conservative level. Compared to CQL, the Q-values learned by ACQL higher than CQL showing ACQL is less conservative than CQL while the Q-values do not explode.
6.3 ABLATION STUDY
Effeet of the Proposed Losses. Due to the space limitation, we reported the quantitative results on the Hopper environment including kinds of datasets Table More results about other environ- ments are provided in Appendix C.4 As shown in Table 5 when only control the conservative level using true (second row) or Leteeg (third row), ACQL dropped its performance seriously, with only around 21.0 of normalized returns to limit the range of the adaptive weights, is prone to learn erroneously adaptive weights and the errors increase like snowball the policy evaluation repeats and lead to failure.
expert dataset. Without using Lmooo and Los
Figure 2: Learning curves comparing to CQL with different conservative levels on Hopper-v0 envi- ronments.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Comparisons of average Q-values over the datasets on HalfCheetah- environments.
Task Name
CQL-1
CQL-2
CQL-5
CQL-10
CQL-20
ACQL
halfcchettah-expert-ee
1276.3
1252.9
1221.9
1202.7
1178.3
1271.1
maalfcheetah-mediinee medium--eeeeee experrtt----e
901.9
803.6
627.5
545.7
439.1
755.1
halfcheetah- medium--rplayy
283.5
237.6
72.3
-178.1
-248.3
452.4
halfcheetthh-medium-
103.8
-9.1
-49.7
-241.1
-313.6
385.9
halfcheetahh saaliheettah-andom--y
154.6
126.7
76.9
40.5
29.4
304.1
Table 5: Ablation Study on Gym-MuJoCo Hopper-vO environments in terms of normalized results.
Lol
Lmmoo
pos
hopper-e
hopper-m-e
hopper-m-r
hopper-m
hopper-r
21.7
8.3
33.1
52.8
10.6
/
21.5
31.5
38.2
58.4
24.3
d
/
34.8
85.4
66.9
41.7
12.5
/
/
V
114.4
112.4
71.5
56.3
10.7
116.3
111.8
77.3
85.8
12.7
Visualization of the Adaptive Weights. To further check the monotonicity of the adaptive weight functions, we visualized the adaptive weights and the corresponding relative transition quality mea- surements for HalsCheetab--raddom-wW dataset in Figure 3 Note that values are normalized to uniform the magnitudes and sorted according to the quality measurements. In the left of Fig- ure danr are the d (s, a) where s are in dataset and a are sampled randomly, while d, policy the (s, a) where are predicted from the training policy. We ity measurements (green and red points) increase, the adaptive weights d, (blue and yellow points) decrease, indicating the Q-values of good OOD state-action pairs should be suppressed less. In the right of Figure 3 dx dataset are the dxę (s, a) where a are sampled from the dataset. As the the quality measurements (green points) increase, the adaptive weights d7s (blue points) also increase, indicating the Q-values of good in in-dataset aateonction pairs should be higher.
the trend that as the qual-
10
0.8
quality datenst
08
0.8
02
0.0
d rand
guadty_rand
pollcy
0.5
quality polcy
25
50
100
125
25c
175
200
20
60
100
Figure 3: Visualization of the adaptive weights and relative transition quality m a) on HallChectab-random dataset.
CONCLUSION
In this paper, we proposed flexible framework named Adaptive Conservative Q-Learning (ACQL), which sheds the light on how control the conservative level of the Q-function in a fine-grained way. In ACQL, two weight functions, corresponding to the out-of-distribution (OOD) actions and actions in the dataset, are introduced to adaptively shape the Q-function. More importantly, the form of these two adaptive weight functions is not fixed and it is possible to define particular forms for different scenarios, e.g., elaborately hand-designed rules or learnable deep neural networks. We provide detailed theory analysis about how the conservative level of the learned Q-function changes under different conditions and define the monotonicity of the adaptive weight functions. To illustrate the feasibility of our framework, we propose novel practical algorithm using neural networks as the weight functions. With the guidance the theoretical analysis, we construct two surrogate and contrastive losses to control the conservative level and maintain the monotonicity. We build extensive experiments on commonly-un offline RL benchmarks and the ofate-of-the-art results well demonstrate the effectiveness of our method.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abien Fred Agarap. Deep learning using rectified
linear
units
(relu).
arXiv preprint
(00.0088330875 2018.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning. PMLR, 2020.
lige Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, al. Solving rubik's cube with robot hand. arXiv preprint arXiv: 13, 2019.
Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning, Advances neural information processing systems, 2008
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off- policy evaluation. Advances in Neural Information Processing Systems, 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01 540, 2016.
Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in games, 2012.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 2021.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprini arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning. PMLR, 2019.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11556 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. PMLR, 2018.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine, When to trust your model: Model- based policy optimization. Advances in Neural Information Processing Systems, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. Advances in neural information processing systems, 2021.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint
2019.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforce- ment learning for vision-based robotic manipulation. In Conference on Robot Learning. PMLR, 2018.
10

---

Under review as a conference paper at ICLR 2023
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 2020.
Thomas N Kipf and Max Welling. Sepi-super classification with graph convolutional net-
works. arXiv preprint arXiv: 160900290 2016.
Hya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning. PMLR. 2021a.
lyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit qr learning. arXiv preprint 10X:v22110.06169, 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy q- learning via bootstrapping eroor reduction. Advances Neural Information Processing Systems, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 2020.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. arXiv preprint arXiv:2 1081,, 2021.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning- Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Minghuan Liu, Hanye Zhao, Zhengyu Yang. Jian Shen, Weinan Zhang, Li Zhao, and Tie- Yan Liu. Curriculum offline imitating learning. Advances in Neural Information Processing Systems, 2021.
Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative onfline distributional reinforce- ment learning. Advances in Neural Information Processing Systems, 2021.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. arXiv preprint 20XY::2006.03647, 2020.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint
2019.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint wrxiv22006.093355 2020.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn, Advances in neural information processing systems, 2016.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightee regression:
Simple and scalable off- off-policy reinforcement learning. arXiv preprint 19X10:1910.001177, 2019.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. arXiv preprint arXiv:2203.01387, 2022.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10083 2017.
Stéphane Ross and J Andrew Bagnell. Agnostic system identification for model-based reinforcement learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pp. 1905-1912, 2012.
11

---

Under review as a conference paper at ICLR 2023
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08390 2020.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 2016.
Thiago D Simão, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with an estimated baseline policy. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, 2020.
Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics. In Conference on Robot Learning. PMLR, 2022.
Richard S Sutton, A Rupam Mahmood, and Martha White. An emphatic approach to the problem of off-policy temporal-difference learning. The Journal of Machine Learning Research, 2016.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems. IEEE, 2012.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEF/RSJ international conference on intelligent robots and systems. TEEE, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- lion processing systems, 2017.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprini arXiv: 19X1:1111..1300 2019.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020a.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 2020b.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances Neural In- formation Processing Systems, 2021a.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 2021b.
Yinuo Zhao, Kun Wu, Zhiyuan Xu, Zhengping Che, Qi Lu, Jian Tang, and Chi Harold Liu. Cadre: A cascade deep reinforcement learning framework for vision-based autonomous urban driving. arXiv preprint arXiv:2202.08557, 2022.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. arXiv preprint 2XXv:20/1.0022133 2020.
12

---

Under review as a conference paper at ICLR 2023
A PROOFS
Proposition A.1 (The conservative level of ACQL). For any i with supp μ C. supp क g. without considering the sampling error between the empirical B Q and true Bellman backups B=Q, the conservative level of ACQL can controlled at three levels according to different conditions: 1) Control over the Q-values. The learned Q-fanction Q'CQL more conservative than the true Q-function Q" point-wise, if:
Ys D.O, 1 dxg 3
2
(22)
2) Control over the V-values. The excepted values of learned Q-function QACQr. more conserva- five than excepted values of the true Q-function Q², if:
vs E E&DD
D
dn 1 dap
n 2
(23)
T
3) Control over the empirical MDP. The learned Q-function VACQr. is more conservative over the empirical MDP, if:
ΣΣ SED
du p1
d
W8
(24)
33
Proof of Proposition A.1
Without considering the sampling error between the empirical 8 Q and Q, we first show the optimization problem of Q-function in ACQL as the
true Bellman backups B
following:
ACQL min (asaas)) (s,(s, a) Q(s,a)] E ((ls)) (s, Q(s,
E, B.a [Q(6..) -B"d 2
(25)
By setting the derivative of Equation 25 to 0, we can obtain the form of the resulting Q-function Q++1 in ACQL:
aQ
Vs D,a, a,
(26)
8Q
= dp 1t d T3 + mB (Q B Q+)
(27)
@k+1 CACQL = BT Q² d, 11 dn T
(28)
Note that the true O-function is only derived from the true Bellman operator Q++1 Q=QU. Based
on the Equation 28 we have exactly the condition control the conservative level over the Q-
function as shown in Equation|22 22] If we want to relax the conservative level like requiring to control over the V-values or the empirical MDP,
can easily relax the Equation 28 to the integration over
each state or the whole empirical MDP as shown in Equations
(23 and (24 respectively. For the
conditions where we want to make the learned Q-function Queqi is less conservative than the true Q-function Q", we can easily replace the ">" "<" Proposition A.I]
Next we show that ACQL bounds the gap between the learned Q-values and true Q-values with the consideration of the sampling етrоr between the empirical 8 Q and actual Bellman operator B Q. Following (Auer et al. 2008 Osband et al., 2016 Kumar et al., 2020), the error can be bounded by leveraging the concentration properties of BT. We introduce the the bound in brief here: with high probability 2 - 6, - Q BT Q),, a) <
C...ss Vs, E D, where Cr,,,8 is a constant VID(s,)))
relating the reward function r(s, a), environment dynamic P(-|s, a), and 5 (0.1).
13

---

Under review as a conference paper at ICLR 2023
Proposition A.2 (ACQL bounds the gap between the learned Q-values and true Q-values). Consid- ering the sampling error between the empirical Q=Q and true Bellman backups B Q with a high probability 2 1 6, the gap between the learned Q-function V'CCQr and the true Q-function Q" satisfies the following inequality:
vs D,a, g(s,a) err(s,a) s -)cqu(s,a)- - (s, s {s(sa) + ter((s,)),
(29)
where
(I (1->pa--1 du 1 dag AB
((s.a) =
(8, (8,a),
(30)
+B
(I yp" C.
err(s,a) =
(s,a) 20
(31)
(1 TV D
1) Thus, if (y a) (s. as D.a, it the case of CQL, where the V lower-bounds the v with large a instead of point-wise lower-bound for Q-function.
2) If g(g,a) a) 2 err(s, (s,(a,))333)) optimistic than the true Q-fanction Q in these regions.
a) 3s E D, a, with the left inequality, the learned Q-function Q'CQLL is more
3) If g(s, a) S -err(s, a), 3s D, a, with the right inequality, the learned Q-function QacQL is more conservative than the true O-function Q" in these regions.
Note that the term err(s, a) is positive value for any state-action pair. Given the bounds in Proposi- tion A.2 instead of only knowing the size relationship (i.e., more or less conservative), we can con- trol the fine-grained range of the gap more precisely by carefully designing d, a) and d.,, (s,a). a).
Proof of Propusition A.2 In Proposition A.I, we calculate the gap between the learned Q-function LaoQr and the true Q-function Q*, representing the conservative level, without the sampling error between the empirical B² and true Bellman backups B Now we can obtain a more precise bound for the conservative level with the consideration of the sampling error. Following (Auer et al.) 2008 Osband al. 2016 Kumar al. 2020),
relate the empirical Bellman backups 8 Q and true Bellman backups B Q the following: with high probability - 6, (0, 1).
C.pa.RRmaz
vQ,8,0 D,
18 B²
(32)
((s,)
where Rmar is the upper bound for the reward function (i.e., |r(s, a)l Rmax), Cr.P. s is constant relating to the reward function r(s, q). environment dynamic P( a). More detailed proofs of the relationship provided in (Kumar al. 2020).
14

---

Under review as a conference paper at ICLR 2023
For the right inequality in Equation 29 reason the fixed point of the Q-function in ACQL as the following:
Cr.P &o
[ BT AcC - (1 (1-7)v (s)
(33)
Cr.p,RRmox
+ QACQL & B QACQL +
(34)
(1- (1-7)v/D(ss..))
= B QsCQL- dp l p dzp" a33
du u dxy nB
Cr.p,& Rimax
1 Bd QAQQL
1B
(1-7)V
D a
(35)
dp. p drg ng
Cr.pishaar
+ VAcQL S B QAQQQ
(36)
2B
(1 (1- (1-7)//D ()ssaa
dn 1 - dag àB
&r.p,
L'cQL S (r + ypm (
Q (AcQu)=
+ (1 (1-1) (1-s))DD(s,a))
(37)
- VicQu. S (I (1-ypp (1- r d. 1
dxp A3
&
(38)
hp
+ Q'CQL SQ- -(I - 11--pP) ppa)- - 1 dro #
&
+ (I (1-PP-)-1 7) (D
(39)
- V'CQL- - -Q" S g(s a) + er)(s,a).
(40)
For the left inequality in Equation 29
we have the similar process the following:
Cr.p. Rmax
18T QACQL - B" B'Q
(41)
(1-7)/ a))
Cr.p,s
= =B 2 BTt
(42)
(1 (1- v) ass
- d. dxe= R/ 2 BT Q=QQCQQ
- - 10-H--ap dre= R (1 7) D
Cr.p.s
= B QuCQL-
tB
4B
D
(s,
(43)
du - 11 drg ng
Cr.p,&Rman
= Ticqr 2 Bd B=QAOQL
(44)
àB
(1- (1-))VD (1-7VVD(s,a)) ((s
dn 11
dag -Tg
Cr. Cr.p,s
- Vicqu 2 (r + 7 7pm Q QhoQu)
(45)
Rg
(1 // D(s,a)
= VicQu 2 ( r dz M
drp n3
Cr.p,&Ruaaz
(46)
*
(1 (1-7) D
VicQr 2 Q²= - (I 11---0 -pm) dn . 1 dng 7B
C.p.ssmaa
(I - -P")-
(47)
Ag
(1 /1-7)//(ss,
= Vicqr. - -Q" 2 2g(s, crr(s,a).
(48)
Proposition A.3 (The conservative level compared to CQL.. For any It with supp ps C supp
given the Q-function learned from CQL is Qōq (s, a) = Q" a -==T, similiar as in A.1 the conservative level of ACQL compared to CQL can be controlled at least at three different levels according to different conditions:
1) Control over the Q-values. The learned Q-function Q is less conservative than the CQL
Q-
function Too piint-wise, if:
- (d-d)))-- (a d T 0.
Vs E D,a,
(49)
T
15

---

Under review as a conference paper at ICLR 2023
2) Control over the V-values. The excepted values learned Q-function Q" is less conservative than excepted values of the true Q-function Q", if:
(a dn)p (a dx )
Vs E D. Σ
2 0.
(50)
T
3) Control over the empirical MDP. The learned Q-fumction Q is less conservative over the empir- ical MDP, if:
ΣΣ
(a dp)p (ar
drg )
JT 2 0.
(51)
K3
RED a
Proof of Proposition A.3 As shown in CQL (Kuma et al. 2020), we first recap the optimization
problem CQL. as the following:
dOOL + min a D. [Q(s, - (anaaa)))(als) 1Q a)))
+ 2 E.
(o(ss,a)BB**tt))
(52)
We can observa that CQL the special case of ACQL where the adaptive weight functions d,, a) and dr (8, a) are both constant By setting the derivative of Equation 52 to O, obtain the form of the resulting Q-function Q++1 in CQL:
vs E D
00
(53)
8Q
3 a a T3 +
IQ
B Q =
(54)
QCQL B Q+
TB
(55)
T
Similar to the proof of Proposition A. we calculate the difference between the Q-values of ACQL
and CQL in Equations (28 and [55 as the following:
As E D,a,
Q dctt
(56)
B²
du
daş
TT TTB
-B=
a 1
T
(57)
π
(a dn) 1 (a du )TB
(58)
If want learn less conservative Q-function than CQL ACQL, we need make the differ- ence in Equation 58 greater than O, as shown in Equation 49 If we want to relax the conservative the V-values or the empirical MDP, we can easily relax the Equa-
level like requiring control
tion 58 to the integration over each state or the whole empirical MDP as shown in Equations (50)
and 5 respectively. For the conditions where want to make the learned Q-function QAcQL
more conservative than the CQL Q-function LEqL we can easily replace the ">" to "<" in Proposition A.3]
Lemma A.1 For In Sx 1. In and only if x
Proof of Lemma A.1
Suppase f(x) = In x + 1, then f'(x) 1-2 It is easy to know that f(z) 1 over (0, 1) and f(x) over (1, +00). Then f(x) & f(1) 0.
16

---

Under review as a conference paper at ICLR 2023
B EXPERIMENTAL DETAILS
Software. We run our experiments with the following packages and softwares:
d4rl 1.1
Python 3.8.13
Pytorch 1.10.0ccul11
Gym 0.23.1
MuJoCo 2.1.5
mujoco-p py 2.1.2.14
dm-control 1.0.2
numpy 1.22.3
h5py 3.6.0
Table 6: Implementation Details of ACQL.
Hyperparameter
Value
Critic learning rate
3e-4
Actor learning rate
le-5
Target update rate Optimizer
5e-3
Adam
Kiif & Welling 2016
Batch size
256
Discount factor
0.99
ACQL Hyperparameter
Lagrange for
False
20.0
Training steps
1.5
Evaluation frequency
1000
Evaluation episodes
10
Weight function learning rate
3e-4
Behavior policy training steps
1.0
Actor type
Tanh Gaussian
Actor hidden layers
3
Actor hidden dim
[256. 256. 256
Actor activation function
ReLU Agarap
2018
ACQL Network
Critic hidden layers
3
Critic hidden dim
[256. 256, 256] ReLU
Architectures
Critic activation function
Weight function hidden layers
Weight function hidden dim
[256, 256, 256]
Weight function ouput dim Weight function activation function
ReLU
Implementation Details of ACQL. In ACQL, we represent two adaptive weight functions dy a) and d= (s. a) by one neural network, which has the same network architecture as the Q-function but the output dimension is 2. During each gradient descent step, we train the weight network, the Q networks and the policy networks in turn. We list all the hyperparameters and network architectures of ACQL Table
17

---

Under review as a conference paper at ICLR 2023
C ADDITIONAL EXPERIMENTS
C.I COMPARISONS TO OFFLINE RL BASELINES ON GYM-MUJOCO-V2 DATASETS
Aiming to provide more comprehensive comparison, we compared ACQL many state-of- the-art model-free algorithms including behavioral cloning (BC), 10%BC, Decision Transformer (DT) (Chen et al., 202T). AWAC (Nair et al. 2020p, Onestep RL (Brandionbrener et al. 2021),
TD3+BC (Fujimoto & Gu 2021). IQL (Kostrikov et al., 2021a) and CQL (Kumar al., 2020) on the version-2 dataset. For the sake of fair comparisons, we directly reported the results of all baselines from the D4RL whitepaper (Fu et al., 2020) and their original papers.
Table ssoows the normalized returns of all 15 Gym-MuloCo tasks on the version 2 datasets. Due to most baselines did not report their results on "-expert" datasets, conducted experiments other 4 types of datasets. We can observe that ACQL consistently outperforms other baselines on all three environments, specifically on the Hopper environment, i.e., achieving normalized returns 302.9 for hopper-sum compared to 268.2 from CQL which is the second rank. From the perspective of the dataset types, ACQL is very balanced algorithm that achieves excelling results on all kinds of dataset with expert, medium and random data. Furthermore, we find that ACQL can achieve higher performance than other baselines when there are more medium and random data in datasets, we argue it is because ACQL. can adaptively calculate the weights for data with different qualities and thus learn better policy. For instance, while the best result on hopper-randoms is 9.6 from AWAC, the result of ACQL is 3 31.4, which is more than 3 times better.
of other baselines
Table 7: Normalized results on D4RL Gym- Gym-MuJoCo environments.
Task Name
BC
10% BC
D)T
AWAC
OnestepRI.
TD3+BC
IQL.
CQL
ACQL
malfcheetah-m-ee
$5.2
92.9
86.8
36.8
93.4
90.7
86.7
91.6
87.9
halfcheetah-m--et
36.6
40.6
36.6
40.5
38.1
44.6
44.2
45.5
46.3
halfcheetah-m--2
42.6
42.5
42.6
37.4
48.4
48.3
47.4
44.0
48,4
halfcheetah-r-v2
2.3
2.0
2.2
2.2
6.9
11.0
18.6
25.9
balfcheetah-sum
136.7
178.0
168.2
116.9
186.8
194.6
178.3
199.7
208.5
hopper-m-e-v2
52.5
110.9
107.6
80.9
103.3
98.0
91.5
105.4
82.2
hopper-m-r-v2
18.1
75.9
82.7
37.2
97.5
60.9
94.7
95.0
hopper-m-v2
52.9
56.9
67.6
72.0
59.6
59.3
66.3
58.5
91.4
hopper-r-v2
4.8
4.1
7.5
7.8
8.5
9.3
31.4
hopper-sum
128.3
247.8
265.4
199.7
268.2
226.7
252.5
268.2
302.9
walker-m-e-v2
107.5
109.0
108.1
42.7
113.0
110.1
109.6
108.8 .8
108.3
walker-m-r-v2
26.0
62.5
66.6
27.0
49.5
81.8
73.9
77.2
85.5
walker-m-v2
75.3
75.0
74.0
30.1
81.8
83.7
78.3
72.5
84.9
walker-r-v2
1.7
1.7
2.0
5.1
6.1
1.6
2.5
5.2
walker-sum
210.5
248.2
250.7
250.4
277.2
261.8
261.0
283.9
C.2 COMPARISONS AMONG DIFFERENT CONSERVATIVE LEVELS
To demonstrate more intuitively that ACQL can adaptively control the conservative level, we com- pared ACQL to CQL with different values ranging from 1, 2, 5, 10, 20. The different a values represent different conservative levels and a higher means more conservative.
Figure plotted the learning curves of ACQL and CQL with 5 kinds of a. From the Figure 4 one trend we can clearly observe is that CQL with higher conservative level can usually achieve higher performance on expert datasets opposed to that lower conservative level is more effective for random datasets. For instance, the CQL agent with a = (green line) delivered rapidly increasing dataset (top left comner,, while
results as the training epochs increased on the halfcheetah-expe dataset (top right corner). Moreover,
hovered at the bottom the plot on the halfcchettah-rndde it is difficult for CQL achieve satisfactory performance all kinds of dataset with fixed a. is exactly the issue that the ACQL focuses on. Since ACQL generates adaptive weight for each state-action pair and control the conservative level in a more fine-go -inennnieedd way, ACQL achieve balanced and state-of-the- results for different dataset types. addition, the plot of hopper- medium-replay task (2th row, 2th column), ACQL outperformed CQL agents by large margin,
18

---

Under review as a conference paper at ICLR 2023
Figure 4: Learning curves comparing to CQL with different conservative levels.
Table 8: Normalized results on D4RL Gym-MuJoCo environments.
Task Name
CQL-1
CQL-2
CQL-5
CQL-10
CQL-20
ACQL
haffchectah-c-v0
9.8
18.7
67.1
84.5
96.3
97.7
halfcheetah-m--e-00
9.7
12.0
17.4
40.0
53.9
93.2
m
48.9
47.5
45.7
44.4
43.9
46,4
halfcheetah-m--00
43.9
43.3
41.7
39.9
39.2
42.8
halfcheetah-r-v0
36.8
35.3
33.6
19.7
13.7
31.3
halfchectah-sum
149.1
156.8
205.5
228.5
247.0
311.4
hopper-e-v0
28.2
48.5
75.6
112.6
112.9
116.3
hopper-m-c-v0
116.1
114.1
112.9
113.5
113.4
111.8
hopper-m-r-vO
33.3
32.1
29.5
29.1
27.1
77.3
hopper-m-vO
99.8
93.3
51.7
56.5
46.4
85.8
hopper-r-v0
11.5
11.2
10.9
10.7
10.4
12.7
hopper-sum
288.9
299.2
280.6
322.4
310.2
403.9
walker-e-vO
44.5
103.0
84.4
108.9
109.7
116.1
walker-m-e-v0
21.7
37.1
86.9
103.5
94.5
walker-m-r-vD
56.1
43.8
30.0
19.9
11.4
45.2
walker-m-vO
86.9
85.3
82.9
82.2
81.1
81.8
walker-r-vO
0.5
-0.2
8.3
3.5
4.9
11.8
walker-sum
209.7
269.0
292.5
318.0
308.7
349.4
all-sum
647.7
725.0
778.6
868.9
865.9
1064.7
showing the significance of adaptive weight for each data sample instead of using the same weight a.
Table showeed the quantitative results of ACQL and CQL with kinds of a. From the Table one trend we can clearly observe is that CQL with higher conservative level can usually achieve higher performance on expert datasets as opposed to that lower conservative level is more effective for random datasets. For instance, the CQL agent with 96.3 on the halfcheetah-exxertt dataset, while dropped its performance dramatically to 13.7 on halfcheetah-randon dataset. Moreover, it is difficult for CQL to achieve satisfactory performance on all kinds dataset with fixed a. It is exactly the issue that the ACQL focuses on. Since ACQL generates adaptive weight for each state-action pair and control the conservative level in more fine-grained way, ACQL can achieve balanced and state-of sice--ff-th-tttt art results for different dataset types. In addition, the hopper-mediuu--epla task, ACQL outperformed CQL agents by a large margin,
20 delivered high performance value of
19

---

Under review as a conference paper at ICLR 2023
Table 9; Normalized results on D4RL AntMaze environments.
Task Name
BC
10%BC
DT
AWAC
OnestepRL
TD3+BC
IQL
CQL
ACQL
aonmaazelarge-divereeee
0.0
6.0
0.0
1.0
0.0
0.0
47.5
14.9
0.0
antmaze-large notmaze-large-play-o00
0.0
0.0
0.0
0.2
39.6
15.8
simmazz-moddum-diverrss..
0.0
0.0
3.0
53.7
utmmaeemedduum-llyy-v
0.0
10.6
71.2
61.2
sumuuzo-umazeediveee---
45.6
50.2
53.0
49.3
60.7
71.4
62.2
84.0
76.8
antmazz-umaze-vo
54.6
62.8
59.2
56.7
64.3
78.6
87.5
74.0
94.9
showing the significance of adaptive weight for each data sample instead of using the same weight O.
C.3 COMPARISONS TO OFFLINE RI. BASELINES ON ANTMAZE TASKS
The AntMaze tasks mimic real-world robotic navigation tasks that aim control an "Ant" quadruped robot to reach the goal location from a start location with only 0-1 sparse rewards given. Aiming to provide more comprehensive comparison, we compared ACQL to many state- of-the-art model-free algorithms including behavioral cloning (BC), 10%BC, Decision Transformer (DT) (Chen et al. 202 I). AWAC (Nair et al. 2020p, Onestep RL (Brandfonbrener et al. 202
TD3+BC (Fujimoto & Gu 2021). IQL (Kostrikov al. 202 and CQL (Kumar al. 2020 For the sake of fair comparisons, we directly reported the results all baselines from the D4RL whitepaper (Fu et al. 2020) and their original papers.
Table shows the normalized returns of all AntMaze tasks. We can observe that ACQL delivers state-of-the-ar performance umaze-diverse-ve v0 and umaze-vO datasets, demonstrating its effec- tiveness on small mazes. However, while IQL can achieve better performance on larger mazes like "-large" and "-medium" tasks, ACQL fails in these scenarios. The experimental results show that it is more difficult for ACQL to tell whether the state-action pairs are good or bad when only 0-1 sparse rewards are given.
C.4 ADDITIONAL ABLATION STUDY
Besides the main results of comparison to other baselines and detailed comparison to different conservative levels, we also conducted extensive ablation studies for ACQL to study the effect of the proposed losses. We reported the quantitative results the all Gym-MuJoCo environments including 15 kinds of datasets in Tables 10] and 12
Table 10: Ablation Study on D4RL Gym-MuloCo HalfCheetah-v0 environments in terms of nor- malized results.
La troe ogs
Limno
Epas
halfcheelah-e
halfcheetah-mm-e
halfcheetal-mm-r
halfcheetah-m
balfcheetah-r
9.9
15.2
28.5
16.4
5.4
2
57.8
23.9
39,9
33,5
24.1
3
8.8
22.5
47.6
47.0
26.9
4
6.7
19.9
49.6
46.4
9.8
34.7
38.1
6
/
42,3
75.3
42.5
43.1
23,5
27.1
8
/
97.7
93.2
46.4
42.8
31.3
20

---

Under review as a conference paper at ICLR 2023
Table 11: Ablation Study on D4RL Gym-MuJoCo Hopper-vO environments in terms of normalized results.
Let toue
Let eqt.
Emann
Epna
hopper-e
hopper-m-e
hopper-m-r
hopper-m
hopper-r
21.7
8.3
33.1
52.8
10.6
31.8
31.3
36.6
51.9
31.7
38.2
58.4
36.5
70.1
93.5
12.9
34.8
85.4
66.9
41.7
12.5
/
61.9
74.5
95.4
70.9
11.7
/
/
/
114.4
112.4
71.5
56.3
10.7
/
/
/
116.3
111.8
77.3
85.8
12.7
Table 12: Ablation Study on D4RL Gym-MuJoCo Walker environments in terms of normalized results.
La true
Leteegl
Emono
Cpos
walker-e
walker-m-e
walker-m-r
walker-m
walker-r
23.3
32.4
22.5
9.3
18,2
93.7
38.5
30.4
27.3
11.9
11.6
28.3
64.8
20.2
17.5
55.1
22.4
29.7
82.8
3,5
6.3
113.5
63.4
13.3
110.2
55.5
21.4
62.2
10.7
/
102.9
81.1
28.5
81.1
11.2
116.1
94.5
45.2
81.8
11.8
21


Reviews:
[{"review_id": "FGXkDq0l2m", "reviewer": "Reviewer_AX8y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the problem of offline reinforcement learning, where the authors consider the limitation of the conservative Q learning (CQL) method and propose a new way of applying a fine-grained control of the conservatism level. The authors argue that setting the level of conservatism to be a global constant as done in CQL is suboptimal, because the dataset often contains both good and poor actions and hence it is intuitive to put conservatism on good actions compared to the poor actions. Instead, the paper introduces adaptive coefficients to weight the conservatism terms.\n\n\nBuilding on top of the formulation of CQL, the authors introduce two state-action dependent coefficients to weight the push up and push down loss instead of using a single global conservatism coefficient. The authors show theoretically that a properly chosen set of coefficients can still produce value estimates that lower bounds the true Q function, but with less conservatism than having a globally constant coefficient. The authors then propose a concrete formulation of the coefficient d based on the quality of the transition, which is measured by its relative reward and cumulative return magnitudes.\n\nThe authors evaluate the proposed method in the D4RL suite of tasks, and show that the proposed method improves upon the baseline CQL method in some environments.\n", "summary_of_the_review": "Overall, while the general direction of research is interesting, due to the concerns about the empirical evaluations and justifications for the proposed method, I still cannot recommend acceptance of the paper in its current state.", "strength_and_weaknesses": "I reviewed this paper for NeurIPS 2022 before it was withdrawn by the authors. Overall I think this paper presents an interesting direction towards improving value conservatism methods, where they often suffer from the problem of being overly conservative. However I do still have some concerns about the particular approach and empirical results.\n\n \n\n### Strength\n\nThis paper presents a very interesting and potentially very effective direction for improving the use of value conservatism in Q-learning style offline RL algorithms. Conservative algorithms in offline RL are known to be difficult to tune, and often suffer the problem of either being overly conservative where the policy does not improve beyond the behavior policy, or being insufficiently conservative where the Q functions suffer from overestimation. Having adaptive conservatism coefficients that depend on the transition could be a promising way of solving this problem.\n\n\n\n### Weaknesses\n\nFirst of all, I want to emphasize (again) that certain baseline evaluations for CQL in this paper seem flawed, where a properly tuned CQL can achieve much better performance than reported in this paper. For example, in one open source CQL implementation I used, CQL can achieve 95 in hopper-medium-replay-v2 and the reported performance is 48, and in walker-medium-replay-v2, CQL can achieve 82 while the reported performance is only 26. Hence I believe that some of the CQL results are not valid, and this presents a major problem for this paper because the proposed algorithm directly improves upon CQL. I’ve brought up this point in my NeurIPS 2022 review of this paper and pointed out the open source CQL implementation suggested by the original CQL authors to reproduce these numbers. Unfortunately, the authors have not addressed this problem in this submission.\n\n\nBesides the flaw in certain baseline evaluations, the empirical performance of the proposed method is also not very strong. Following the numbers on the D4RL suite, it seems that ACQL does not really outperform significantly, despite introducing many more degrees of freedom and complexity. It makes me question whether the marginal improvement is really due to the proposed method, or because of the extra tunable components introduced.\n\nThe specific choices of the loss functions to learn the conservative coefficients are not well justified. It is intuitive to see why the agent should be less conservative on good actions and more conservative on bad actions, but I’m not convinced that the proposed transition quality function is a good way of measuring this. Moreover, the overall loss formulation is quite complicated with many moving parts, so it is difficult to understand the necessity and choices of each part without good justifications.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Aj9b9cUJxL", "reviewer": "Reviewer_3P64", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces ACQL, adaptive conservative Q-learning for offline reinforcement learning. Different from the standard Q learning framework, ACQL learns a less conservative policy by controlling the weight of each sample with a learnable parameter. The weight is directly associated with the statistics of the dataset and the quality of the transitions. Theoretically, ACQL gives monotonic improvement on the conservative level of CQL constraints. Empirically, the authors have presented improved results on the D4RL benchmark across multiple tasks.", "summary_of_the_review": "I think the paper is interesting and novel. However, as discussed above, the empirical study needs significant improvements. Thus, I think it is not ready for publication yet.\n\n", "strength_and_weaknesses": "Strengths:\n\nOverall, the paper is well-written, well-motivated, and easy to follow. The paper has studied an important and interesting problem of the famous conservative learning methods for offline reinforcement learning, i.e., the over-conservative value estimation. Theoretically, the proposed ACQL addresses the issue of over-conservative value estimation.\n\nWeaknesses:\n\nThe main weakness of this paper is the insufficient empirical results.\n\n1/ The results presented in Table 1 are susceptible. According to the authors, the results are directly taken from the original paper of baselines or the D4RL whitepaper [1]. However, the reported results of CQL are significantly worse than the ones discussed in IQL [2]. For example, in the submitted manuscript, CQL has a normalized return of 26.7 on walker2d-medium-replay, while in IQL paper, CQL has a normalized return of 77.2, which is already higher than the reported ACQL result (45.2). There are 4 such environments where the reported CQL performs worse than the CQL results in the IQL paper, including halfcheetah-m-e, hopper-m-e, hopper-m-r, and walker2d-m-r. The performance gap is caused by the misuse of environments. In gym-locomotion-v0 environments, the performance is generally worse than in the v2 environments. It seems that the reported results of CQL are directly taken from the D4RL whitepaper, where v0 environments are used, while the reported IQL results are run on v2 environments. As a result, the results reported in Table 1 are not comparable. In addition, the submitted manuscript mentions no information about the version of its environment, which further makes it hard to justify the soundness of its empirical results.\n\n2/ No results on antmaze are reported. Antmaze is considered one of the most difficult environments due to its sparse and delayed reward. In addition, low-quality transitions often exist in such environments, which I believe should be an ideal environment to test ACQL against CQL. It would be good to add such results.\n\n3/ In addition to the ablation studies, some intuition should be provided to better understand the influences of these two weights. Although theoretically, ACQL guarantees a monotonic improvement, it remains unclear how such behavior reflects on the learned weights. It would be great to visualize the relationship between the learned weight and the quality of transitions, i.e., m(s, a).\n\n4/ According to the ablation studies, it seems that ACQL is fairly unstable. Without L_{mono}, ACQL would fail completely and even perform worse than the reported CQL result. How about training only with L_{mono} without other losses?\n\nBesides the above-mentioned issues, another issue is that ACQL has multiple losses to ensure the learned weight is reasonable and stable. How does this compare with directly weighting the samples with a fixed weight? For example, m(s, a)?\n\nReferences:\n\n[1] Fu, J., Kumar, A., Nachum, O., Tucker, G., & Levine, S. (2020). D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219.\n[2] Kostrikov, I., Nair, A., & Levine, S. (2021). Offline reinforcement learning with implicit q-learning. International Conference on Learning Representations (ICLR)\n[3] Kumar, A., Zhou, A., Tucker, G., & Levine, S. (2020). Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6TMnGQPU_rP", "reviewer": "Reviewer_KtHP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes ACQL, which weights the regularization term used in CQL to improve performance. By using a weight function that is based on heuristics, ACQL regularizes each state action differently according to the weight. In the experiment, the paper shows the results on various benchmarks that ACQL can improve on CQL.", "summary_of_the_review": "The paper is overall clearly written, and there may be some readers interested in reading the paper. However, in my opinion, the theoretical results are weak, and the algorithm is not well supported by the theory. Despite the extensive evaluation of the algorithm, considering the low originality of ACQL, I am more leaning toward rejection. ", "strength_and_weaknesses": "Strength\n- performance improvement over CQL\n- paper is clearly written and easy to follow\n- extensive experiments on many different benchmarks including ablation study\n\nWeakness\n- weak support on \"monotonicity\"\nWhile it is interesting to have proposition 4.1~4.2 and propose weight functions having such characteristics, the definition 4.1 is not well supported. Even in the case where we have $Q^*$, using the weight functions following eq 6, 7 will not recover an optimal policy. If we have some other choice, e.g. weight that is proportional to exponential of $Q$s, we may get some other intuitions on the algorithm by relating it to AWAC or etc.\n\n- not well-motivated transition quality measurements\nWhile the authors suggested using $Q^*$ as a weights, the suggested transition quality measurements are too irrelevant to $Q^*$, even considering that $Q^*$ cannot be computed. There are number of choices that are not explained, e.g.\n1. averaging normalized reward and return: it is not a normal choice to average between reward and the return; if we wanted to make a bit more myopic measure than the return, I may just increase the discount factor for this. This choice is too ad-hoc and will be very suboptimal in some specific cases.\n2. using the behavior $Q$ instead of $Q^*$: using a simple return $g$ corresponds to using a behavior $Q$ function. It does not matches the above motivation of using $Q^*$ as a weight function.\n\n- about the positivity\nI think using the positivity objective (19) won't ensure the model has positive output. What is the reason of not using a positive transformation at the output of the neural network? e.g. softplus", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject