Exclusively Penalized Q-learning for Offline Reinforcement Learning
Authors: Junghyuk Yeom
Yonghyeon Jo
Jeongmo Kim
Sanghyeon Lee
Seungyul Han
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Deep reinforcement learning
offline RL
Q-learning
overestimation reduction
Primary area: reinforcement_learning
TL;DR: We propose a novel penalizing method, which efficiently reduces overestimation for offline reinforcement learning

Abstract: Constraint-based offline reinforcement learning (RL) involves policy constraints or imposing penalties on the value function to mitigate overestimation errors caused by distributional shift. This paper focuses on a limitation in existing offline RL methods with penalized value function, indicating the potential for underestimation bias due to unnecessary bias introduced in the value function. To address this concern, we propose Exclusively Penalized Q-learning (EPQ), which reduces estimation bias in the value function by selectively penalizing states that are prone to inducing estimation errors. Numerical results show that our method significantly reduces underestimation bias and improves performance in various offline control tasks compared to other offline RL methods.

Exclusively Penalized Q-learning for Offline Reinforcement Learning
Junghyuk Yeom*
Yonghyeon Jo
Jungmo Kim Sanghyeon Lee Seungyul Han
Graduate School of Artificial Intelligence
UNIST
Ulsan, South Korea 44919 jmkimm2,
Abstract
Constraint-based offline reinforcement learning (RL) involves policy constraints imposing penalties on the value function to mitigate overestimation errors caused by distributional shift. This paper focuses on limitation in existing offline RL methods with penalized value function, indicating the potential for underestimation bias due unnecessary bias introduced the value function. To address this concern, we propose Exclusively Penalized Q-learning (EPQ), which reduces estimation bias in the value function by selectively penalizing states that are prone to inducing estimation errors. Numerical results show that our method significantly reduces underestimation bias and improves performance various offline control tasks compared to other offline RL methods.
Introduction
Reinforcement learning (RL) is gaining significant attention for solving complex Markov decision process (MDP) tasks. Traditionally, online RI. develops advanced decision-making strategies through continuous interaction with environments [1, 2, 3, 4, 5, 6]. However, in real-world scenarios, high-risk environments like disaster situations, where obtaining sufficient data for learning is challenging [7, 8). In such setups, the need for exploration [9, 10, 11, 12] to discover optimal strategies often incurs additional costs, as agents must try various actions, some of which may be inefficient or risky [13, 14]. This highlights the significance of research on offline setups, where policies are learned using pre-collected data without any direct interaction with the environment [15, 16]. In offline setups, policy actions not present in the data may introduce extrapolation errors, disrupting accurate value estimation by causing large overestimation error in the value function, known the distributional shift problem [17].
interacting with the environment can be costly, particularly
To address the distributional shift problem, Fujimoto et al. [17] proposes batch-constrained Q-learning (BCQ), assuming that policy actions are selected from the dataset only. Ensuring optimal convergence of both the policy and value function under batch-constrained RL setups [17], BCQ demonstrates stable learning in offline setups and outperforms behavior cloning (BC) techniques [18], which simply mimic actions from the dataset. However, the policy constraint of BCQ strongly limits the policy space, prompting further research to find improved policies by relaxing constraints based on the support of the policy using metrics like maximum mean discrepancy (MMD) [19] or Kullback-Leinn (KL) divergence [20]. While these methods moderately relax policy restrictions, the issue of limited policies persists. Thus, instead of constraining the policy space directly, alternative offline RL methods have been proposed reduce overestimation bias based on penalized Q-functions [21, 22]. Conservative Q-learning (CQL) [21], a representative offline RL algorithm using Q Q-penalty, penalizes
indicates equal contribution and indicates the corresponding author: Seungyul Han. Special thanks to Whiyoung Jung from LG Research providing experimental data in this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the Q-function for policy actions and provides bonus to the Q-function for actions in the dataset. Consequently, CQL selects more actions from the dataset, effectively reducing overestimation errors without policy constraints.
While CQL has demonstrated outstanding performance across various offline tasks, we observed that it introduces unnecessary estimation bias in the value function for states that do not contribute to overestimation. This issue becomes more pronounced as the level of penalty increases, resulting in performance degradation. To address this issue, this paper introduces a novel Exclusively Penalized Q-learning (EPQ) method for efficient offline RL. EPQ imposes threshold penalty on the value function exclusively for states causing estimation errors to mitigate overestimation bias without introducing unnecessary bias in offline learning. Experimental results demonstrate that our proposed method effectively reduces both overestimation bias due to distributional shift and underestimation bias due to the penalty, allowing more accurate evaluation of the current policy compared to the existing methods. Numerical results reveal that EPQ significantly outperforms other Late-of-the-arr offline RL. algorithms on various D4RL, tasks [23].
2 Preliminaries
2.1 Markoy Decision Process and Offline RL
We consider Markov Decision Process (MDP) environment denoted as M := (S. A, R, 7),
where the state space, A is the action space, P represents the transition probability, y the discount factor, and R is the bounded reward function. In offline RI., transition samples d, = (st, az, Tt, $1+1) are generated by a behavior policy 3 and stored the dataset D. We can empirically estimate 3 as ((a|s) = Nis.a)
N(e) where N represents the number of data points in D. We assume that Ex~D,a~a (f a]] ss - a)] E..eee (s, a]] for arbitrary function f. Utilizing only the provided dataset without interacting with the environment, our objective find a target policy that maximizes the expected discounted return, denoted as (xx := Eso.ao [Go]. where
G. = -Ee R[si, at) represents the discounted return.
2.2 Distributional Shift Problem in Offline RI.
In online RL, the optimal policy that maximizes ((r) is found through iterative policy evalu- ation and policy improvement [2, 3]. For policy evaluation, the action value function is de- fined QT ae) == st.
[s)s,, ar). Q" can be estimated by it- eratively applying the Bellman operator T to an arbitrary Q-function, where (B*Q) a) == R(s,a) a) + -EEr-P()ss)
7
( a')]. The Q-function is updated to minimize the Bellman
error using the dataset D. given by Es,a- (Q(s, - Q=Q(=, a)) In offline RL, samples are generated by the behavior policy only, resulting in estimation errors in the Q-function for policy ac- tions not present in the dataset D. The policy æ is updated to maximize the Q-function, incorporating the estimation error in the policy improvement step. This process accumulates positive bias in the Q-function as iterations progress [17].
2.3 Conservative Q-learning
To mitigate overestimation in offline RL, conservative Q-learning (CQL) [21] penalizes the Q- function for the policy actions a ~ π and increases the Q-function for the data actions a ~3 while minimizing the Bellman error, where the Q-loss function of CQL is given by
E, EnaD [(((s,a) ( + aEE=D|EaNNT00 [Q(s, a)] E IQ
(1)
2
where > 0 penalizing constant. From the value update in (1), the average Q-value of data actions E E ( a)] becomes larger than the average Q-value of target policy actions Ea-r a)] as a increases. As a result, the policy will tend choose the data actions more from the policy improvement step, effectively reducing overestimation error in the Qufunction [21].

---

3 Methodology
3.1 Motivation: Necessity of Mitigating Unnecessary Estimation Bias
In this section, we focus on the penalization behavior of CQL., one of the most representative penalty-based offline RL methods, and present illustrative example show that unnecessary estimation bias can occur in the Q-function due to the penalization. As explained Section 2.3, CQL penalizes the Q-function for policy actions and increases the Q-function for data actions in (1). When examining the Q-function for each state- sate-action pair (s, a), the Q-value increases if (aas) >> (aass) otherwise, the Q-value decreases as the penalizing constant a becomes sufficiently large [21].
To visually demonstrate this, Fig. 1 depicts histograms of the fixed policy π and the es- timated behavior policy 3 for various т and at the initial state 8o the Pendulum task with a single-dimensional action space in (a), (b), and (c), along with the estimation bias in the Q-function for CQL with various pe- nalizing factors a. In this example, for all states except the initial state, consider Unif(- 2, 2). In each case, CQL only updates the Q-function with its penalty to evaluate ι in an offline setup, as shown equation (1), and we plot the estimation bias of CQL, which represents the average difference between the learned Q-function and the expected return Go-
2.0
(a-0-))
Blas
Density
(a-20.0)
0.5
OpenAI Gym tasks [24],
Action
Actinn
Action
(a)
(b)
(c)
Figure 1: Histograms of π and 3 (left axis), and the
estimation bias of CQL with various a (right axis) at so for three cases: (a) 8 = Unif(- and π = N 0.2) (b)
(13) and π 0.2) (c) 3 = (--10,03)) N (1, and π (N, 0.2), where Unif(- 2, 2, represents uniform distribution and N (H, or) denotes a Gaussian distribution with mean and standard deviation oT
N - 0.3)
From the results in Fig. 1, we observe that CQL suffers from unnecessary estimation bias in the Q-function for cases (a) and (b). In both cases, the histograms illustrate that policy actions are fully contained in the dataset B. suggesting that the estimation error in the Bellman update is unlikely to occur even without any penalty. However, CQL introduces a substantial negative bias for actions near 0 where x (0)ss) > {(0]so) and positive bias for other actions. Furthermore, the bias intensifies as the penalty level increases. order mitigate this bias, reducing the penalty level zero may seem intuitive in cases like Fig. (a) and Fig. 1(b). However, such an approach would be inadequate in cases like Fig. 1(c). In this case, because policy actions close to are rare the dataset, penalization is necessary to address overestimation caused by estimation errors in offline learning Furthermore, this problem may become more severe actual offline learning situations, as the policy continues to change as learning progresses, compared to situations where fixed policy is assumed.
3.2 Exclusively Penalized Q-learning
To address the issue outlined in Section 3.1, our goal is to selectively give penalty the Q- function in cases like Fig. I(c), where policy actions are insufficient in the dataset while minimizing unnecessary bias due the penalty in scenarios like Fig. ((a) and Fig. 1(b), where policy actions sufficient in the dataset. To achieve this goal, we introduce novel exclusive penalty P, defined by
t(als) (aass)
-
P
penalty adaptation factor
penalty term
where (.) = Ea-a( a) [x is penalty adaptation factor for given 3 and policy Here, - min(1.0, exp(- (log (als) T))) represents the amount of adaptive penalty that is reduced as log 3 exceeds the threshold r. Thus, the adaptation factor f... indicates the average penalty that policy actions should receive. If the probability estimated behavior policy for policy actions exceeds the threshold т, i.c., policy actions are sufficiently present in the datasct, then z. will be
3

---

m1
R2
ST3
(N1 & N₂)
log B
1.0
Enough data
x2
1Engigh data
A
F7
Ex
[
-
f
1
<t <1
(a)
(b)
Figure 2: An illustration of our exclusive penalty: (a) The log-probability of 3 and the thresholds
T1 and т2 according to the number of data samples N1 and N2, where Nı << N2. (b) The penalty adaptation factor fr. which represents the amount of adaptive penalty, indicating how much log B exceeds the threshold r. Three different policies n,, i 1,2,3, 3, considered.
smaller than and reduce the amount of penalty as much as the amount by which 3 exceeds the threshold T to avoid unnecessary bias introduced in Section 3.1. Otherwise, it will be 1 due to min(1.0,.) -) to maintain the penalty since policy actions are insufficient in the dataset. The latter penalty term s(al=) - 1, positive if r(ass) > B(als) and otherwise negative, imposes a positive 8(a[s)
penalty on the Q-function when x[a|s) B(a|s), and otherwise, it increases the Q-function since the penalty is negative, as the Q-penalization method considered in CQL [21].
To elaborate further on our proposed penalty, Fig. 2(a) depicts the log- log-probability of and the thresholds 7 used for penalty adaptation, with N representing the number of data points. In Fig. 2(a), the log-probability log 3 of an action E exceeds the threshold т, this indicates that the action is sufficiently represented in the dataset. Thus, we reduce the penalty for such actions. Furthermore, as shown Fig. 2(a), when the number of actions increase from N1 to N2, the threshold for determining "enough data" decreases from T1 to 72, even if the data distribution remains unchanged.
Furthermore, to explain the role of the threshold in the proposed penalty P,, consider two thresholds, Tı and 72. In Fig. 2(b), which illustrates the proposed penalty adaptation factor fT. and fT. for thresholds T1 and T2. of is larger than 2. because 71 -22. As result, in the case of TI, P., only reduces the penalty for T3. In other words, fT1. fT2. B 1, and f3a.. 1. On the other hand, as the number of data samples increases from N1 to Ni, more actions generated by the behavior policy 3 will be stored in the dataset, policy actions more likely to be in the dataset. In this case, the threshold should be lowered from Ti 72. As result, 3 exceeds the threshold ro in the support of all policies Tú, and Pr, reduces the penalty in the support of all policies t. i.e., f < < f. < f < 1. Thus, even without knowing the exact number of data samples, the proposed penalty P, allows adjusting the penalty level appropriately according to the given number of data samples based on the threshold T.
Now, we propose exclusively penalized Q-learning (EPQ), a novel offline RL method that minimizes the Bellman error while imposing the proposed exclusive penalty P. the Q-function as follows:
min E.n.,' Q
[Q(s, a) (B=Q(s, - -PP,))
Then, we can prove that the final Q-function of EPQ underestimates the true value function Q in offline RI if is sufficiently large, as stated in the following theorem. This indicates that the proposed EPQ can successfully reduce overestimation bias in offline RI, while simultaneously alleviating unnecessary bias based on the proposed penalty P,.
Theorem 3.1. We denote the Q-function converged from the Q-update of EPQ using the proposed penalty P, in (3) by Q",. Then, the expected value of Q" underestimates the expected true policy value,
i.e. E. (Q= a)] S Ea-zIQ= (s, a)), Vs E D, with high probability 1 - 8 for some ō E (0, 1), if the penalizing factor a is sufficiently large. Furthermore, the proposed penalty reduces the average penalty for policy actions compared to the average penalty of CQL.

---

2.0
Esttmatton Blas
0.8
Denstty
0.0
COL
0.5
0.2
Policy
Palicy
Action
Action
Action
(a)
(b)
(a)
(b)
(c)
Figure 3: Histogram (left axis), and Figure 4: Histograms и and (left axis), and the estima-
the corresponding f7 with various tion bias of CQL and EPQ with various (right axis) for
T (right axis) for two cases: (a) 8 three cases: (a) 8 Unif(-2,22 2) and
Unif(-22,) (b) B (.330333 - ( 0.3) + (1, 0.3) and N (0,0.2) (b) [1, 0.3)
(1, (c)
2 N 1, + +2 (1, and = N 0.2).
Proof) Proof of Theorem 3.1 is provided in Appendix A. In order to demonstrate the Q-function convergence behavior of the proposed EPQ in detail, revisit the previous Pendulum task in Fig. Fig. 3 shows the histogram of and the penalty adaptation factor f,
fr (s) for Gaussian policy π = N (ju, 0.2), where μ varies from -2 to 2, with varying B. In Fig. 3(a), (F should be less than 1 for any policy mean μ since all policy actions are sufficient the dataset. In 3(b), 17 (s) is less than only the 3 probability near policy mean μ is high, and otherwise, (s) is 1, which indicates the lack of policy action in the dataset. Thus, the result shows that 1 /x (s) reflects our motivation in Section
well. Moreover, Fig. compares the estimation bias curves of CQL. and EPQ with 10 in the scenarios presented in Fig. I. CQL exhibits unnecessary bias for situations Fig. 4(a) and Fig. 4(b) where no penalty is needed, discussed in Section 3.1. Conversely, proposed method effectively reduces estimation bias in these cases while appropriately maintaining the penalty for the scenario in Fig. 4(c) where penalization is required. This experiment demonstrates the effectiveness of proposed approach, and the subsequent numerical results in Section will numerically show that our method significantly reduces estimation bias in offline learning, resulting in improved performance.
3.3 Prioritized Dataset
π
Be x Bexp(()
B
B
1
A
A
Maximum Q
(a)
(b)
Figure 5: An illustration of the prioritized dataset. As the policy focuses on actions with maximum Q-values, the difference between 3 and becomes substantial, inducing large penalty: (a) The change of data distribution from 3 (w/o PD) to 88 (with PD) (b) The corresponding penalty graphs for 3 (w/o PD) and 3 (with PD).
In Section 3.2, EPQ effectively controls the penalty the scenarios depicted in Fig. However, in cases where the policy is highly concentrated not be completely eliminated due to the latter penalty term in Pr, as я significantly exceeds
side, as shown in Fig, 4, the estimation bias may
3. This situation, detailed in Fig. 5, arises when there is substantial difference in the Q-function values among data actions. As the policy is updated maximize the Q-function, the policy shifts towards the data action with larger Q, resulting in more significant penalty for COL. To further
5

---

alleviate the penalty to reduce unnecessary bias in this situation, instead of applying penalty based on B, we introduce a penalty based on the prioritized dataset (PD) 3 x 3 eex((Q, As shown in Fig. 5(a), which illustrates the difference between the original data distribution 3 and the modified data distribution 3 after applying PD, 39 prioritizes data actions with higher Q-values within the support of B. According to Fig. 5(a), when the policy focuses on specific actions, the penalty /
increases significantly, as depicted in Fig. 5(b). In contrast, by applying PD, 3 is adjusted to approach 32 x exp(()), aligning the data distribution more closely with the policy я. Consequently, we anticipate that the penalty will be significantly mitigated, as the difference between and 39 is much smaller than the difference between and B. Following this intuition, modify our penalty using 1). It is important to note that the penalty adaptation factor
PD as Pr, PD == 17.0 (s) (B](a]))
[[aas)]
(s) remains unchanged since we use all data samples in the dataset for Q updates. Additionally, we consider the prioritized dataset for the Bellman update focus more data actions with higher Q-function values for better performance as considered in [25]. Then, we can derive the final Q-loss function of EPQ with PD as
M(Q)
1
E 5, D
[00 (Q=0 aP,, aP,,)
(4)
2
{1 { (Q(5,a) B Q(x, +a (1Q(s)))) Q(s, C,
E 8,6' D.amBaa'
where w.. (a(s))
3(a]s) B [xxp((((aa] is the importance sampling (IS) weight, C is the remaining constant term, and the detailed derivation of (4) is provided in Appendix B.1. The ablation study in Section 4.3 will show that EPQ performs better when prioritized dataset 39 is considered.
exp(Q(e.a))
3.4 Practical Implementation and Algorithm
Now, we propose the implementation of EPQ based on the value loss function (4). Basically, our implementation follows the setup of CQL [21]. For policy, we utilize the Gaussian policy with a Tanh(-) layer proposed by Haarnoja et al. [4] and update the policy to maximize the Q-function with its entropy. Then, the policy loss function is given by
[(r) EgwD, a~* [- Q(+ a) + log (a]s)).
(5)
Based on the Q-update in (4) and the policy loss function (5), we summarize the algorithm of EPQ as Algorithm More detailed implementation, including the calculation method of the IS weight w and redefined loss functions for the parameterized Q and æ, is provided in Appendix B.2.
Algorithm Exclusively Penalized Q-learning
Require: Offline dataset D
Train the behavior policy / based on behavior cloning (BC)
2: Initialize Q and æ
3: for gradient step 1: = 0, 2, do
4: Sample batch transitions {(,,a, 8')} from D.
5: Calculate the penalty adaptation factor f. f5.3 (s) and IS weight 2
6: Compute losses L(Q) in Equation (4) and L(x) in Equation (5)
7: Update the policy to minimize L(T))
8: Update the Q-function Q to minimize T.())
9: end for
4 Experiments
In this section, we evaluate our proposed EPQ against other state-of-the-an offline RL algorithms using the D4RL benchmark [23], commonly used in the offline RL domain. Among various D4RL tasks, we mainly consider Mujoco locomotion tasks, Adroit manipulation tasks, and AntMaze navigation tasks, with scores normalized from to 100, where represents random performance and 100 represents expert performance.

---

Table I: Performance comparison: Normalized average return results
Tisk name
BC
10% BC
TD3+BC
CQL (paper)
CQL (reprod.)
Onestep
IQL
MCQ
MISA
EPQ
balfaheetuh-randow
20.8
33.0-2.4
hogger-randons
4.7
22.5
10.8
9.7
9.6
31.8
9.9
32.1=0.3
walkeräd-random
1.7
2.3
7.2
7.0
7.1
6.2
6.9
17.0
9.0
23.0 11.7
halfeheetuh-melle
42.6
42.5
44.0
47.4
67.3 0.5
hogpcr-mecciu walker&d-madium
$2.9
56.9
59.3
86.6
58.5
59.6
66.3
78.4
67.1
101.3:0.2
75.3
75.0
83.7
74.5
72.5
81.8
78.3
91.0
87.8-211 84.1
halfcheetah- medium expert
56.2
92.9
90.7
62.4
91.6
93.4
$6.7
87.5
96.7=0.3
hogger medium- expert
105.4
502.815.2
walker2d exxdiut--cctt
107.5
109.0
110.1
98.7
108.8
113.0
109.6
114.2
109.1
112.0=00.6
salfcheetah-expert hogger expert
92.9
91.9
98.5
104.
96.3
92.3
95.44
96.2
95.9
107.2=0.2
111.2
109.6
111.7
109.9
110.8
112.3
112.4
111.4
111.9
0.2.4=0.5
walker2d expert
108.5
110.0
109.811.0
halfcheetah-medlie medium-rpply
56.6
40.6
44.6
46.2
45.5
38.1
44.2
56.8
45.6
62.0
hogper-medium- medium- replay
18.1
75.9
60.9
48.6
95.0
97.5
94.7
101.6
98.6
97.8=11 1.0
walker&d-noe immppaalu
halfehectah--ie full-replay
62.4
68.7
75.9
76.9
73.3
X2.3
74,8
85.3-0.7
hogper- full-replay
34.3
92.8
81.5
101.0 0
107.8
107.2
108.5
103.5
108.5-40.
waller2dlffll--oe
45.0
89.4
95.2
93.4
102.0
98.1
95.7
94,8
107.4::.6
Mujoco Tasks Total
929.1
1236.9
1293.0
1325.8
1311.0
1318.5
1474.9
1354,5
1536.7
pen-human
63.9
-2.0
64.88
55.8
57.5
71.8
71.5
68.55
88.1
83.9
doce-human
2.0
0.0
0.0
9.1
9.9
5.4
4.3
2.3
5.2
13.2-2.4
hammer human
0.0
2.1
4.4
3.9.5.00
relocare-hurra
0.0
0.2
0.3+0.2
pen-eloned
37.0
0.0
49
40.3
39.2
60.0
57.3
49.4
58.6
91.8 4.7
doce-cloned
0.0
0.0
0.0
3.5
0.4
0.4
1.6
1.3
0.5
5.8-2.8
lammer-clone
22.8-15.3
relocate-cloned
0.0
-0.1
Adroit Tasks
93.6
54.6
62.8
73.5
74.0
80.4
72.5
87.5
98.3
92.3
99.4= 1.0
umaze
medium-plan playimm-payy
0.0
5.4
10.5
61.2
67.5
5.0
71.2
52.5
63.0
85.0=11.2
medium- divense
0.0
9.8
3.0
53.7
62.5
5.0
70.0
37.5
62.8
86.7=186
large
0.0
0.2
15.8
35.0
2.5
39.6
2.5
17.5
40,0.882
large- diveers
6.0
13.3
36.7=4.7
AntMaze Tasks Total
134,2
315.0
162.5
426.1
The D4RL dataset comprises offline datasets obtained from Mujoco tasks [26] like HalfCheetah, Hopper, and Walker2d. Each task has 'random', 'medium', and 'expert' datasets, obtained by a random policy, the medium policy with performance 50 to 100 points, and the expert policy with performance of 100 points, respectively. Additionally, there are 'medium-expert" dataset that contains both 'medium' and 'expert' data, 'medium-replay" and 'full-replay' datasets that contain the buffers generated while the medium and expert policies are trained, respectively.
Mujoco Locomotion Tasks:
Adroit Manipulation Tasks: Adroit provides four complex manipulation tasks: Pen, Hammer, Door, and Relocate, utilizing motion-captured human data with associated rewards. Each task has two datasets: 'human' dataset derived from human motion-
data, and 'cloned' dataset comprising samples from both the cloned behavior policy using BC and the original motion-capture data.
AntMaze Navigation Tasks: AntMaze is composed of six navigation tasks including 'umaze', "umaze-diverse" 'medium-pla -aauuumpplyy...a agent is trained to reach goal within the maze. While 'play' dataset is acquired under fixed set of goal locations and fixed set of starting locations, the 'diverse' dataset is acquired under a random goal locations and random starting locations setting.
ciddiummdivveres 'large- pange-play.....m and 'large-diverse" where robot ant
4.1 Performance Comparisons
We compare our algorithm with various constraint-basee offline RL methods, including CQL baselines [21] on which our algorithm is based on. For other baseline methods, we consider behavior cloning (BC) and 10% BC, where the latter only utilizes only the top 10% of demonstrations with high returns, TD3+BC [27] that simply combines BC with TD3 [3], Onestep RL [28] that performs single policy iteration based on the dataset, implicit Q-learning (IQL) [29] that seeks the optimal value function for dataset through expectile regression, mildly conservative Q-learning (MCQ) [30] that reduces overestimation by using pseudo Q values for out-of-distributiin actions, and MISA [31] that considers the policy constraint based on mutual information. To assess baseline algorithm performance, utilize results directly from the original papers for CQL (paper) [21] and MCQ
7

---

Hopper-random
Hopper-mediim medium
Halfcheetah-medlum
Estimation Blas
CCL (--0.0)
Estimation Bias
1e13
Estimation Bias
300
100
250
CCL (2m3.0)
400
CCK (a-10 10.00
80
200
001 (n=20.0)
150
40
100
0.6
Gradient Step
le6
Gradient Step
led
Gradient Step
le6
(a) Squared value of estimation bias
Normalized Returms
Hopper-random
Normalized Returss
Happer-medium
Normalized Returns
Halfcheetah-medium
30
80
80-
20
60
80
10:
20
8.0
0.2
0.4
0.6
0.0
1.0
8.0
0.2
0.4
0.6
0.8
1.0
8.0
0.2
0.1
0.8 1.
Gradient Step
le6
Gradient Step
lec
Gradient Step
160
(b) Normalized average return
Figure 6: Analysis of proposed method
[30], as well reported results from other baseline algorithms according to Ma et al. [31]. For CQL, reproducing its performance is challenging, so we also include reproduced CQL performance labeled as CQL (reprod.) from Ma al. [31]. Any missing experimental results have been filled in by re-implementing each baseline algorithm. For our algorithm, we explored various penalty control thresholds 7 E (c- p. c [0, 10]]. where p represents the log-density of Un) For Mujoco tasks, the EPQ penalizing constant is fixed at a 20.0, and for Adroit tasks, we consider either a = 5.0 or Q 20.0. To ensure robustness,
run our algorithm with four different seeds for each task. Table 1 displays the average normalized returns and corresponding standard deviations for compared algorithms. The performance of EPQ is based on the best hyperparameter setup, with additional results presented in the ablation study in Section 4.3. Further details on the hyperparameter setup are provided in Appendix C.
The results in Table shows that our algorithm significantly outperforms the other constraint-based offline RL algorithms in all considered tasks. In partieular, in challenging tasks such Adroit tasks and AntMaze tasks, where rewards are sparse or intermittent, EPQ demonstrates remarkable performance improvements compared recent offline RL methods. This because EPQ can impose appropriate penalty on each state, even if the policy and behavior policy varies depending on the timestep as demonstrated in Section 3.2. Also, we observe that our proposed algorithm shows large increase in performance in the 'ooper-randomi environments compared to CQL, so we will further analyze the causes of the performance increase in these tasks in the following section. For adroit tasks, the performance of CQL (reprod.) is too low compared to CQL (paper), so we provide the enhanced version CQL in Appendix E, but the result in Appendix E shows that EPQ still performs better than the enhanced version of CQL.
HHopper-medium medium' and
sefffcheeta--mediin
4.2 The Analysis of Estimation Bias
In Section 4. EPQ outperforms CQL baselines significantly across various D4RL tasks based on our proposed penalty in Section 3. To analyze the impact of our overestimation reduction method on performance enhancement, we compare the estimation bias for EPQ and CQL baselines with various penalizing constants a E {0, 1, 5, 10, 20} on 'opper-random',, 'Hopper-medium" and "Halfcheetah- medium" tasks. In Fig. 6(a), we depict the squared value of estimation bias, obtained from the difference between the Q-value and the empirical average return for sample trajectories generated by the policy, to show both overestimation bias and underestimation bias. In the experiment shown in Fig. 6(a), the estimation bias in CQL with 0 became excessively large, causing the gradients to explode and resulting in forced termination of the training. Fig. 6(b) illustrates the corresponding normalized average returns, emphasizing learning progress after 200k gradient steps.
8

---

Hopper-random
Hopper-medium
Halfcheetah-medium
Normalized Returss
Normalized Returss
Normalized Returns
100
60
20
1.5
90
40
20
23
Bng wuth FU)
85
km
Gradient Step
146
Gradient Step
Gradlent Step
(a) Component evaluation
Hopper-random
Hopper-medium
Halfchectah-meddum
Normalizen lizdllized Returns
Normal lized Returss
Norma lized Returns
100
60
23
20
gPo (e-di2 sest
(t-d de sedt
can seel
bank beci
30
TRO tr-0
15
Snd (140-hake2))
Bng 2102
2PQ (ce) but
10
S.l Eei
(owd En, 277h
8
#00 (1w) Art
(PQ E-kg)
ER0 (n-3
BPQ [tw] tud
(1-10.50
the (1-10
280 (e-5
80
6.5 Gradient Step
0.4
85
0.2
8.6 Gradient Step
54
1.0
0.0
0:2
145
Gradient Step
(b) Penalty control thresholds E [0.2p, 0.5p, 1.0p. 1.0p 2.0p. 5.0p, 10.0pl
Figure 7: Additional ablation studies on the Hopper-random, Hopper-medium, and Halfcheetah- medium tasks are presented. The best hyperparameter in the paper is denoted by the orange curve.
In Fig. 6(a), we observe increase in estimation bias for CQL as the penalizing constant a rises, attributed to unnecessary bias highlighted in Fig. 1. Reducing a to nearly 0 in CQL. however, fails to effectively mitigate overestimation error, leading to a divergence of the Q-function in tasks such as 'Hopper-random' and 'Hopper-medium', as shown in Fig. 1. Conversely, EPQ demonstrates superior reduction of estimation bias in the Q-function compared CQL baselines for all tasks in Fig. 6(a). indicating its capability to mitigate both overestimation and underestimation bias based on the proposed penalty, As a result, Fig. 6(b) shows that EPQ significantly outperforms all CQL tasks.
variants on 'Hopper-raddom' 'Hopper-medium', and
4.3 Ablation Study
To understand the impact of EPQ's components and hyperparameters, we conduct ablation studies to evaluate each component and the penalty control threshold r on the 'Hopper-random', 'Hopper- medium', and SiliChheeah-meedun tasks where our proposed method showed significant perfor- mance improvement compared to the baseline CQL.
Component Evaluation: In Section we introduced two variants of the EPQ algorithm: EPQ (w/o PD), which does not incorporate a prioritized dataset as in equation (3), and EPQ (with PD), which leverages a prioritized dataset based on 3 as in equation (4). In Fig. 7(a), we compare the performance of EPQ (w/o PD), EPQ (with PD), and the CQL baseline to analyze the impact of each component. EPQ (w/o PD) still outperforms CQL, demonstrating that the proposed penalty Prr in Section 3.2 enhances performance by efficiently reducing overestimation without introducing unnecessary estimation bias, as discussed in Section 3.2. Additionally, Fig. 7(a) shows that EPQ (with PD) outperforms EPQ (w/o PD) significantly in the HalfCheetah-meddin task, indicating that the proposed prioritized dataset contributes to improved performance, as anticipated in Section 3.3.
Penalty Control Threshold :: As discussed in Section 3.2, EPQ can dynamically control the penalty amount based on the penalty control threshold t, as illustrated in Fig. 2, even in the absence of knowledge about the exact number of data samples. Fig. 7(b) demonstrates the performance of EPQ with various penalty control thresholds 7 E [0.2p, 0.5p, 1.0p. 2.0p, 3.0p. 10.0p), where represents the log-density of Un). Note that p is negative, so T = 10.0p the lowest threshold while T = 0.2p is the highest. The results indicate that in tasks like Hopper-medium, where a variety of actions are not sufficiently sampled, a higher threshold performs better. Conversely, in tasks like Hopper-random, where broad range of actions is sampled, a lower threshold is more effective, An exception is the HalfCheetah-mediim task, which, despite having fewer action variations, visits diverse range of states. This may result in lower overestimation errors for OOD

---

actions, benefiting from lower threshold. Furthermore, the performance on the considered tasks appears be surprisingyy less sensitive to changes т. We initially expect that performance might be sensitive to since it reflects the fixed number of data samples, but the results indicate that the performance not significantly affected by variations in т. Moreover, EPQ algorithms with different consistently outperform the CQL baseline, highlighting the superiority of the proposed method.
5
Related Works
5.1
Constraint-basee Offline RL
In order to reduce the overestimation in offline learning, several constraint-based offline RL methods have been studied. Fujimoto et al. [17] propose a batch-constrained policy to minimize the extrapola- tion error, and Kumar et al. [19]. Wu et al. [20] limits the distribution based on the distance of the distribution, rather than directly constraining the policy. Fujimoto and Gu [27] restricts the policy actions to batch data based the online algorithm TD3 [3]. Furthermore, Kumar al. [21], Yu et al. [32] aims to minimize the probability of -f-tiiibiiitttiionnn actions using the lower bound of the true value. By predicting more optimistic cost for tuples within the batch data, Xu et al. [22] provides stable training for offline-based safe RL tasks. On the other hand, Ma et al. [31] utilizes mutual information constrain the policy.
5.2 Offline Learning based on Data Optimality
In offline learning setup, the optimality of the dataset greatly impacts the performance [25]. Simply using n-% BC, applying weighted experiences, [33, 34] which utilize only a portion of the data based on the evaluation results of the given data, fails to exploit the distribution of the data. Based on Haarnoja al. [35], Reddy al. [36], Garg et al. [37] uses the Boltzmann distribution for offline learning, training the policy to follow actions with higher value in the imitation learning domain [38, 39]. Kostrikov al. [29] and Xiao al. [40] argue that the optimality of data can be improved by using expectile regression and in-sample SoftMax, respectively. Additionally, methods that learn the value function from the return of the data in a supervised manner have been proposed [41, 28, 42, 43].
5.3 Value Function Shaping
In offline RL, imposing constraints on the policy can decrease the performance, thus Kumar et al. [21], Lyu al. [44] impose penalties out-of-distribution actions by structuring the learned value function as a lower bound to the actual values. Additionally, Fakoor et al. [45) addresses the issue by imposing a policy constraint based on divergence and suppressing overly optimistic estimations the value function, thereby preventing excessive expansion of the value function. Moreover, Wu et al. [46] predicts the instability of actions through the variance of the value function, imposing penalties on the out-of-distribune
actions, while Lyu et al. [30] replaces the Q values for out-of-distribution actions with pseudo Q-values and Agarwal et al. [47], An al. [48], Bai et al. [49], Lee et al. [50] mitigates the instability of learning the value function by applying ensemble techniques. In addition, Ghosh al. [51] interprets the changes in MDP from Bayesian perspective through the value function, thereby conducting adaptive policy learning.
6 Conclusion
To mitigate overestimation error in offline RL, this paper focuses on exclusive penalty control, which selectivelys gives the penalty for states where policy actions are insufficient in the dataset. Furthermore, we propose a prioritized dataset to enhance the efficiency of reducing unnecessary bias due to the penalty. As result, proposed method, EPQ, successfully reduces the overestimation error arising from distributional shift, while avoiding underestimation error due to the penalty. This significantly reduces estimation bias in offline learning, resulting in substantial performance improvements across various D4RL tasks.
10

---

Acknowledgements
This work was supported in part by Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No.2022-0-00469, Develop- ment of Core Technologies for Task-oriented Reinforcement Learning for Commercialization of Autonomous Drones) and in part by IITP grant funded by the Korea government (MSIT) (No. RS- 2022-0015632 Innovative Human Resource Development for Local AnellectullizaioonNNNSST and in part by Artificial Intelligence Graduate School support (UNIST), IITP grant funded by the Korea government (MSIT) (N0.2020-)))1336..
References
[1] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
[2] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint (tti:::199.029111.. 2015.
[3] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596.
PMLR, 2018.
[4] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[5] Seungyul Han and Youngchul Sung. Dimension-wise importance sampling weight clipping for sample-efficient reinforcement learning. In International Conference on Machine Learning, pages 2586-2595. PMLR, 2019.
[6] Seungyul Han and Youngchul Sung. Diversity actor-critic: Sample-aware entropy regularization for sample-efficient exploration. In International Conference on Machine Learning, pages 4018 4029. PMLR, 2021.
[7] Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near real-world benchmark for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:24753-24765, 2022.
[8] Gaoyue Zhou, Liyiming Ke, Siddhartha Srinivasa, Abhinav Gupta, Aravind Rajeswaran, and Vikash Kumar. Real world offline reinforcement learning with realistic data source. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7176-7183. IEEE, 2023.
[9] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurek, and Picter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems. 30, 2017.
[10] Zhang-Wei Hong, Tzu-Yun Shann, Shih-Yang Lee. Diversity-driven exploration strategy for deep reinforcement learning. Advances in neural information processing systems, 31, 2018.
Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Vy Yi
[11] Scungyul Han and Youngchul Sung. A max-min entropy framework for reinforcement learning. Advances in Neural Information Processing Systems, 34:25732-25745, 2021.
[12] Yonghyeon Jo, Sunwoo Lee, Junghyuk Yeom, and Seungyul Han. Fox: Formation-aware exploration in multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12985-12994, 2024.
[13] Martin Pecka and Tomas Svoboda. Safe exploration techniques for reinforcement learning- an overview. In Modelling and Simulation for Autonomous Systems: First International Workshop, MESAS 2014, Rome, Italy, May 5-6, 2014, Revised Selected Papers I, pages 357-375. Springer, 2014.

---

[14] Jongseong Chae, Seungyul Han, Whiyoung Jung, Myungsik Cho, Sungho Choi, and Youngchul Sung. Robust imitation learning against variations in environment dynamics. In International Conference on Machine Learning, pages 2828-2852, PMLR, 2022.
[15] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 2XXi:2000500144) 2020.
[16] Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.056118 2022.
[17] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference machine learning, pages 2052-2062. PMLR, 2019.
[18] Michael Bain and Claude Sammut. framework for behavioural cloning. Machine Intelligence 15, pages 103-129, 1995.
[19) Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[20] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911:1911..1331,, 2019.
[21] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
[22] Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8753-8760, 2022.
[23] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[24] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 100.04400 2016
[25] Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201.13422 2022.
[26] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 6026-5033. IEEE, 2012.
[27] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145,, 2021.
[28] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems, 144933-4946, 2021.
[29] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learming. arXiv preprint arXiv:2110.06166 2021.
[30] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongging Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1722 2022.
[31] Xiao Ma, Bingyi Kang, Zhongwen Xu, Min Lin, and Shuicheng Yan. Mutual information reg- ularized offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[32] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28966 2021.
[33] Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv: 15..0952 2015.
[34] Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[35] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pages 1352-1361. PMLR, 2017.
[36] Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv:1905 11 108, 2019.
[37] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song. and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34: 4028-4039 2021.
[38] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[39] Sungho Choi, Seungyul Han, Woojun Kim, Jongseong Chae, Whiyoung Jung, and Youngchul Sung. Domain adaptive imitation learning with visual observation. Advances in Neural Information Processing Systems, 36, 2024.
[40] Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sample softmax for offline reinforcement learning. arXiv preprint arXiv: 143772 2023.
[41] Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420. IEEE, 2020.
[42] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:21 10.51 2021.
[43] Zifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization. arXiv preprint arXiv:2302.11312. 2023.
[44] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongging Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1770 2022.
[45] Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander J Smola. Continuous doubly constrained batch reinforcement learning. Advances in Neural Information Processing Systems, 34:1 10111126-11200 1273, 2021.
[46] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind. Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.00 (0000088140. 18000,1120000 2021.
[47] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.
[48] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-base offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-744) 7447, 2021.
13

---

[49] Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint arXiv:2202. 1566, 2022.
[50]
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 77121712. PMLR, 2022.
[51] Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning, pages 7513-7530. PMLR, 2022.
[52] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
arXiv preprint [53] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.611 14, 2013.
[54] Nitin Bhatia et al. Survey of nearest neighbor techniques. arXiv preprint Xiv: 100.1007..008, 2010.
[55] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing. Advances in neural information processing systems, 35:23851-23866, 2022.
[56] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 8 2,05505, 2018.
14

---

A Proof
Theorem 3.1 We denote the Q-function converged from the Q-update of EPQ using the proposed penalty P- in (3) by QT. Then, the expected value of Q" underestimates the expected true policy value, a)), Vs E D. with high probability the penalizing factor a is sufficiently large. Furthermore, the proposed penalty reduces the average penalty for policy actions compared to the average penalty of CQL..
i.e., Ea~r, (s. a)] I Ean
ô for some 8 E (0, 1). if
A.1 Proof of Theorem 3.1
Proof of Theorem 3.1 basically follows the proof of Theorem 3.2 in Kumar et al. [21] since Pr multiplies the penalty control factor f7 (s) to the penalty of CQL. At each k-th iteration, Q-function is updated by equation (4), then
Q=-1 t (* - a.P. Vs,a,
(A.1)
where 8 is the estimation of the true Bellman operator B based on data samples. is known that the error between the estimated Bellman operator _ and the true Bellman operator is bounded with high probability of 1 - 5 for some 5 E (0, 1) as 1(8* ((B*Q)(s, a) - (8 (s, a)l I (sss, a), Vs, a, where 5 is a positive constant related to the given dataset D, the discount factor 7, and the transition probability P [21]. Then, with high probability 1 1-6. 8,
Qu+1(s, a) + 5 Qx, a) - aP- + 5 (s, Vs,a, Now, with the state value function V(s) =-
(A.2)
(s)
(s+ = 1a (atea)iQuuss,, a)] = V% - aBlaw [Pr] + (s)
(s) T(cls)
= (s) - a'Bans
1
+ Eawa 15 (s,a)]
B(als)
(s) - a (PQ + Bana (s,aa)
(A.3)
Upon repeated iteration, Va+1 converges to (s) = (I- ( +
Ea- })]} based the fixed point theorem, where A (s) == Ean [P-] is the average penalty for policy æ, I is the identity matrix, and pa is the state transition matrix where the policy π given. Here, we can show that the average penalty (s) is positive follows:
(T(als) ( -1)]
]
(s)
A (p) Earz
B(ass)
i(als)
<[aas)
(s)
Σ (alss)
-1)
Σ B(als)
B(als)
B(ass)
aEA
=0
( - B(als))
= -f (s)
2 0,
(A.4)
3(als)
aEA
where the equality in (A.4) satisfies when ग = or f 3 0. Given that Vh+ converges to Voo (s) + - T. + En (s, a)]} choosing the penalizing constant a that satisfies a 2 max, max, (5 (s, a)l max.en (s)) will satisfy,
()) + ( (s, a)]
< max 1s (s, a)] max A (PQ -1 A (s) +Ea (s,
s.ED
DED
21
< max m EaNn (s, (s, S 5O, Vs. maa
5-max
(A.5)
15

---

Since I -pP is non-singular M --matrix and the inverse of non-singular M M-matrix is non- negative, i.e., all elements of (I ( are non-negative, (s) V*(s) (I -
- VT (s), Vs. Therefore, Voc underestimates the true value func- tion V the penalizing constant a satisfies a 2 aa a)] mmxx (A addition, according [21], the average penalty of CQL for policy actions can be represented as AoQu(() Ba-al / 11. Thus, (s) - 17 (z) and f7 (s) S from the definition
Er(()) Es-
In
in (2), so v A ErrQ (s) I AZQu (s). In addition, ग = 3, then 0 A (s) = A cor (s) from the equality condition in (A.4), which indicates that the average penalty for data actions is for both EPQ and CQL.
8
16

---

B Implementation Details
In this section, we provide the implementation details of the proposed EPQ. First of all, we provide a detailed derivation of the final Q-loss function(4) of EPQ in Section B.1. Next, we introduce practical implementation of EPQ to compute the loss functions for the parameterized policy and Q-function in Section B.2. In addition, to calculate loss functions in Section B.2, we provide the additional implementation details Appendices B.3, B.4, and B.5. We conduct our experiments on a single server equipped with Intel Xeon Gold 6336Y CPU and one NVIDIA RTX A5000 GPU, and we compare the running time of EPQ with other baseline algorithms in Section B.6. For additional hyperparameters in the practical implementation of EPQ. provide detailed hyperparameter setup and additional ablation studies in Appendix and Appendix D, respectively.
B.1 Detailed Derivation of Q-Loss Function
In Section 3.3, the final Q-loss function with the proposed penalty PT,rD = - 1) is given by
L(Q) = _ D.a-d (Q - Q - aP, pD)) In this section, we provide a more detailed
calculation of L(Q) to obtain (4) as follows:
L(Q)= E
S. (0 - ( aP,, pD))
2
1
[Q Bd Q²()²
E
+ +al s.' [Pr,
PD PD-
Q] CC
2
4,5
1
2
a,
Q Q²)
+ alE 5' D.a
3Q
-
Q
+ +C
E
[Q Rd Q)
+ aE aL S'
1 aEA
-1)
1
Qda
+
2
1
E
1Q B=Q)²
+ aE DEE
/ aEA
87.8
( (3) Qda +C
2
5,5'
1
[Q Q'
+a
TZ Qua'
BR 17 +
2
W'EA
JUEd
1
E.
(Q RD Q)
+ aE taE,,
EN'NT
[0 - B BanBo [A+ +o
2
A. D.a
s.s'~ 09
(Q B" Q)²
+ aE S.
EUINT ( Q] - f +C +
2
80
1 (Q 2B² 2 + ..... (Bo'nT 101 +
Fg
())
DDam
B
2
{ - ta - 2(3.0)) + +c,
= =E a, D.a
where C' the remaining constant term that can ignored for the Q-update since B Q the fixed target value. For (+). apply the IS technique, which states that [x Ezng for any probability distributions and g, and arbitrary function f. and w (a/a)) is the importance sampling (IS) ratio between 30 and 3. ( [xxa(Q(s.a)))
p(z)
S (x)
g(z)
B(a)s)
exp(Q(a.a)
17

---

B.2 Practical Implementation for EPQ
Our implementation basically follows the setup of CQL [21]. We use the Gaussian policy with a Tanh(-) layer proposed by Haarnoja et al. [4], and parameterize the policy and Q-function using neural network parameters @ and 6, respectively. Then, we update the policy to maximize Qo with its entropy H((T)) = Tx,, | log J following the maximum entropy principle [4] as explained in Section 3.3, to account for stochastic policies. Then, we can redefine the policy loss function ((x) defined in (5) as the policy loss function L (0) for policy parameter Φ given by
L (6) DD aroill
Qo(s, a) + log π (a]s))
(B.1)
For the Q-loss function in (4), we use the IS ratio 2 in (4) to account for prioritized sampling based on 89, However, 8 discards samples with low IS weights, which can reduce sample efficiency. To address this, we utilize the clipped IS weight max(Cmin: w where Cmin E (0, 1] is the IS clipping constant. This clipped IS weight is multiplied only to the term (Q(s, a) 8 ((,, a))² in (4) to ensure that we can exploit all data samples for Q-learning while preserving the proposed penalty. The detailed analysis for Cmin is provided in Appendix D. In addition, the optimal policy that maximizes (B.1) follows the Boltzmann distribution, proportional to exp(Qo(e
)). It has been proven in Kumar et al. [21] that the optimal policy satisfies Bawz Qo a) + (()) log Ep-i exp Qo a.,. so we can replace the En [Qe(s, ssa') term in (4) with log _ A exp Qo(s, a²), given that H(T) does not depend on the Q-function. The Bellman operator 5 can be estimated by samples in the dataset as QT x r(s, a) + (Qp's' d'), where 0 is the parameter of the target Q-function. The target network updated using exponential moving average (EMA) with temperature np 0.005, as proposed in the deep Q-network (DQN) [52]. Finally, by applying IS clipping and log E. exp Q to the Q-loss function (4) and redefining it as the value loss function for the value parameter we obtain the following refined value loss function (o) as follows:
LQ(0)=
- E Ex,a,s<DD max(Cmii, w.aa) (r(s.a) + Bo Qa(s,a))
1
(B.2)
2
[ w (
log Σ -o(s,a')-o(ss.))) a'EA
+ olE, alE
where 3 is pre-trained by behavior cloning (BC) [18, 53] to compute -. The parameters & and 0 are updated minimize their loss functions T., (1) and (0) with learning rate 1/ and 1jo, respectively. Detailed implementations for estimating the behavior policy 3, the IS weight wf.. and log E. exp Q are provided in Appendices B.3, B.4, and B.5, respectively.
B.3 Behavior Policy Estimation Based on Variational Auto-Encoder
In Section B.2, we estimate the behavior policy 33 that generates the data samples in D necessary for calculating the penalty adaptation factor f in equation (2). To estimate the behavior policy B, employ the variational auto-encoder (VAH), one of the most representative variational inference methods, to approximate the underlying distribution of a large dataset based on the variational lower bound [53]. In the context of VAF, we define encoder model P (z]s, a) and decoder model qu (a]z, s) parameterized by v. where z is the latent variable whose prior distribution p(2) follows the multivariate normal distribution, i.e., p(z) ~ N (0, f). Assuming independence among all data samples, we can derive the variational lower bound for the likelihood of B as proposed by Kingma and Welling [53]: log B(als) 2 Exrpo (log
s)] DK (Dg
Vs D
(B.3)
the variational lower bound
where (pl(p)lg) Epplogp - log q] is the Kullback-Leibler (KL) divergence between two distributions p and q. In this paper, since we consider the deterministic decoder qu s2) the formal term (elepo(leea)) [log go (a|z, x)] can be replaced with the mean square error (MSE) as Eg-po [log qe (a/ (al2,s) ~ Egspo(-laoo (a
- a) At each k-th iteration, we update the parameter 1/ of VAE to maximize the lower bound in equation (B.3). The log / can be estimated using the variational lower-bound in (B.3) to obtain f f The hyperparameter setup for the VAE provided in Table 2.
18

---

control the radius parameter Table 2: Hyperparameter setup for VAE
VAE Hyperparameters
dimension
2- state dimension
Hidden activation function
ReLU Layer
(512. 2 dim.) (512,512)
Encoder network Pt
(state dim. + action dim., 512) (512, action dim.)
Decoder network de
(512,512) (= dim. + state dim., 512)
B.4 Implementation of IS Weight w
In order to consider the prioritized data distribution 3o proposed in Section 3.3, we use the importance sampling (IS) weight defined by
(a(sss) B(als)
exp(Q(s.a)) (exp((((s,,')]] (
as D.
(B.4)
E B(-1)
Since the computation of (.r makes it difficult to know exact possible action set for state 8,
we approximately estimate the IS weight based on clustering as follows:
1 exp(Q(s, E ( (exp(Q(ss a())
exp(Q(s,a) (s exp(Q(s(, Vs D.
(B.5)
C.aal
Here, Cs,a is the cluster that contains data samples adjacent to (8, a), defined by
Co.a ((s'''') E s'1l2 detoseat;;
(B.6)
where the cluster Cs.a can be directly obtained using the nearest neighbor (NN) algorithm [54] provided in the Python library. e detosest is the radius the cluster, and detosest is the average distance between the closest states from each task. In implementation,
e > 0 to adjust the number of adjacent samples for the estimation of IS Weight w? In addition, using the Q-function in the IS weight term makes the learning unstable since the Q-function continuously changes as the learning progresses. Thus, instead of the Q-function, we use the regularized empirical return G./C for each state-action pair obtained by the trajectories stored in D, where 0 is the regularizing temperature. Upon the increase of Ç, the returned difference between adjacent samples in the cluster decreases, so the effect of prioritization can be reduced. The detailed analysis for and < is provided in Appendix D.
19

---

B.5 Implementation of Q-loss Function
In equation (B.2), the final Q-loss function of proposed EPQ given by
(0) =
2 B D
max(Cmin w (r(s, a) + En'na - Q)(s,a))
+ aBeeeD
w
0.alt (5)))
log Σ
exp exp,a(s,a") -Q0(3,)
a'EA
Here, we can estimate log En exp Q(, a) based the method proposed in CQL [21] as follows:
log Σ exp Q(s,Q(sss)) = log 2
Σ
(exss) a) logs(a|s))) + Σ pa pu(exp(Q(ss a) log pu))
log
Σ
(exp(Q(s, a.) log ((nls))) + 2N. an)-Uiil(
1
Σ (exp(Q(s,a.) - log po)p))))))
2N
(B.7)
where Na is the number of action sampling, Unif(A) is a Uniform distribution on A. and Pa is the density of uniform distribution.
B.6 Time comparison with other offline RL methods
In this sectrion, we compare the runtime of EPQ with other baseline algorithms: CQL, Onestep. IQL, MCQ, and MISA in Table below. For fair comparison across all algorithms, conducted experiments on the Hopper-mediun task, which is a popular dataset for comparing computational costs [48, 55], on single server equipped with an Intel Xeon Gold 6336Y CPU and one NVIDIA RTX A5000 GPU. We measured both epoch runtime during 1,000 gradient steps and score runtime that each algorithm takes to achieve certain normalized scores.
From the epoch runtime results in Table 3, we can observe that EPQ takes approximately 2-30% more runtime per gradient step compared to the CQL baseline. Note that Onestep RL may seem to have very short execution time compared to other algorithms, but one must consider the significantly longer pretraining time required learn the Q-function of behavior policy accurately. Additionally, compared to faster offline RL algorithms such as IQL and MISA, EPQ requires more runtime per step and exhibits a similar runtime MCQ, another conservative Q-learning algorithm. However, according to the score runtime results in Table 3, we can observe that only proposed EPQ achieves score of 100 points, while all other algorithms fail to reach this score. Particularly, compared to MCQ, which also considers CQL baseline, EPQ achieves the same score with significantly less runtime. Therefore, while EPQ may consume slightly more runtime per gradient step compared to other algorithms, we can conclude that proposed EPQ offers substantial advantages in terms of convergence performance over other algorithms.
Table 3: Runtime comparison: Epoch runtime and Score runtime
epoch runtime(s)
CQL
Onestep
IQL
MCQ
MISA
EPQ
1,000 gradient steps
43.1
12.6
13.8
58.1
23.5
54.8
score runtime(s)
CQL
Onestep
IQL
MCQ
MISA
EPQ
Normalized average return
60
3540.0
252.5 568.4
1600.2
31,143.4 49,359.7
4,632.7
3,232,2
80
21,920.0
100
30,633.2
20

---

C Hyperparameter Setup
The implementation of proposed EPQ basically follows the implementation of the CQL algorithm [21]. First, we provide the details of the shared algorithm hyperparameters in Table 4. In Table 4, we compare the shared algorithm hyperparameters of CQL, the revised version of CQL (revised), and proposed EPQ. CQL (revised) considers the same hyperparameter setup with our algorithm for Adroit tasks since the reproduced performance of CQL (reprod.) using the author-provided hyperparameter setup significantly underperforms compared to the result of CQL (paper) in Table 1.
For the coefficient of entropy term in the policy update (B. I), CQL. automatically controls the entropy coefficient so that the entropy of т goes to the target entropy, as proposed in Haarnoja et al. [56]. We observe that while the automatic control of policy entropy proves effective for Mujoco tasks, it adversely affects the performance in Adroit tasks since a policy with low entropy can lead to significant overestimation errors in Adroit tasks. Thus, we considered fixed entropy coefficient for Adroit tasks as in Table 4. In addition, CQL controls the penalizing constant a based on Lagrangian method [21] for Adroit tasks, but we also observe that the automatic control of a destabilizes training. leading to poor performance. Therefore, we considered fixed penalizing constant for Adroit tasks in Table 4 for stable learning.
In addition, in Table 5, we provide the details of the task hyperparameters regarding our contributions in the proposed EPQ: the penalty control threshold 7 and the IS clipping factor Cmin in the Q-loss implementation in (B.2), and the cluster radius e and regularizing temperature for the practical implementation of IS clipping factor w in Section B.4. Note that p in Table 5 represents the log- density of uniform distribution. For the task hyperparameters, we consider various hyperparameter setups and provide the best hyperparameter setup for all considered tasks in Table 5. The results are based on the ablations studies provided in Section 4.3 and Appendix D.
Table 4: Algorithm hyperparameter setup of CQL, CQL (revised), and EPQ (ours) algorithms
CQL (revised) (for Adroit)
Hyperparameters
CQL.
EPQ
Policy leamning Vi
le-4
le-4
le-4
Value function learning rate Ye
3c-4
3e-4
3e-4
Soft target update coefficient ng
0.005
0.005
0.005
Batch size
256
256
256
The number sampling N.
10
Initial behavior cloning steps
10000
10000
10000
Gradient steps for training
3m (0.3m for Adroit)
0.3m
3m (0.3m for Adroit)
Entropy coefficient
Auto
0.5
Auto (0.5 for Adroit)
20 for MuloCo
Penalizing constant o
Auto (10 for MuloCo)
5 or 20
5 or 20 Adroit
Auto for AntMaze
Discount factor y
0.99
0.9 or 0.95
0.99 (0.9 or 0.95 for Adroit)
21

---

Table 5: Tass hyperparameter setup for Mujoco tasks and Adroit tasks
Mujoco Tasks
T/p
Cmin
S
halfcheetah-random
10
0.2
2
2
hopper-random
0.1
0.5
walker2d-random
0.2
2
0.5
halfcheetah-meddumm
10
0.2
0.5
2
hopper-medium
0.2
0.5
2
5
walker2d-medium
0.5
2
halffcheetab-medium expert
1.0
0.2
0.5
hopper-medium-expee expert
0.2
0.5
2
walker2d-mediim-expeee
1.0
0.2
0.5
halfcheetah-expert
0.2
0.5
2
hopper-expert
0.2
0.5
2
walker2d-expert
0.5
0.2
2.0
2
malffcheetah-mediume
0.2
0.5
2
hopper-medium- replay
0.2
0.5
2
walker2d-medium-
0.2
0.5
1.0
2
halffcheetah-full-rephal
1.5
0.2
0.5
2
hopper-full-replay
2.0
0.2
1.0
2
walker2d-full full-replay
1.0
0.2
0.5
Adroit Tasks
T/P
Cmin
e
<
pen-human
0.05
0.5
1.0
200
door-human
0.05
0.5
0.5
200
hammer-human
0.1
0.2
100
relocate-human
0.2
0.2
2
10
pen-cloned
0.2
0.2
5
50
door-cloned
0.2
0.5
1
10
hammer-cloned
0.2
0.2
5
100
relocate-cloned
0.2
0.2
5
10
AntMaze Tasks
T/P
Cmin
6
C
umaze
10
0.2
2
2
umaze-diverse
10
0.2
2
2
medium-play
0.1
0.2
1
2
medium-diverse
0.1
0.2
1
2
large-play
0.1
0.2
1
2
large-diverse
0.1
0.2
1
2
22

---

D Additional Ablation Studies Related to Estimation
5,a
In this section, we provide additional ablation studies related to IS weight w estimation in Appendix B. For analysis, Fig. 8 shows the performance plot when the IS clipping factor Cmin. the cluster radius e, and the temperature Ç change.
Normalized Returms
Normallzzs leld Returns
Normalized Returns
102
-00
80
20Q (oa 21
(0u-
DC (-11 RI
40
(0-1.0
20
20-
COL
COL
85
07
Gradient Step
Gradient Step
Gradient Step
10
(a) IS clipping factor Canin
(b) Cluster radius
(c) Temperature Ç
Figure 8: Additional ablation studies on Hopper-medium task
IS Clipping Factor Cmin the EPQ implementation, the clipping factor Gmin employed to clip
the IS weight w to prevent the exclusion of data samples with relatively low w When Cmin 0.
low-quality samples with low wº are not utilized all based on the prioritization in Section 3.3. However, as Cmin increases, these low-quality samples are increasingly exploited. Fig. 7(c) illustrates the performance of EPQ with varying Cmin- and EPQ achieves the best performance when Cwin 0.5. This result suggests that it is more beneficial to use low-quality samples with proper priority rather than discarding them entirely.
Cluster Radius <: As explained in Appendix B.4, we can control the number of adjacent samples in the cluster based the radius From the results illustrated in Fig. 8(a), can observe that EPQ with = 2.0 performs best, and a decrease or an increase in can significantly affect the performance indicating that must be chosen properly for each task to find the cluster that contains adjacent samples appropriately. If is too small, the cluster will hardly contain adjacent samples, and if too large, samples that should be distinguished will aggregate in the same cluster, adversely affecting the performance.
Temperature Ç: As proposed in Section 3.3, samples in the dataset are prioritized according to the definition of w? Since the samples with higher Q values more likely to be selected for the update of the Q-function, temperature Ç controls the amount prioritization, as explained in Appendix B.4. Increasing reduces the difference in the Q-function between the samples, putting less emphasis on prioritization. Fig. 8(b) shows the performance change according to the change in Ç. where the results state that the performance does heavily depend Ç. From the ablation study, we can conclude that the radius has a greater influence on the performance of Hopper-mediun task compared to the temperature
23

---

E Additional Performance Comparison on Adroit Tasks
For adroit tasks, the performance of CQL (reprod.) is too low compared CQL (paper) in Table so we additionally provide the performance result of the revised version of CQL provided in Section C. We also compare the performance of EPQ with the performance of CQL (revised) on various adroit tasks, and Table 6 shows the corresponding comparison results. From the result, we can see that CQL (revised) greatly enhances the performance of CQL on adroit tasks, but EPQ still outperforms CQL (revised), which demonstrates the intact advantage of the proposed exclusive penalty and prioritized dataset well on the adroit tasks.
Table 6: Performance comparison of CQL (paper), CQL (revised), and EPQ (ours) Adroit tasks.
Task
CQL (paper)
CQL (revised)
EPQ
pen-human
55.8
82.0++6.2
83.9-16.8
door-human
9.1
7.8=0.5
13.2 2.4
hammer-human
2.1
6.4=5.4
3,945,0
relocate-human
0.4
0.1=0.2
0.3
pen-cloned
40.3
90.7:14.8
91.8:14.7
door-cloned
3.5
1.3=2.2
5.8
hammer-cloned
5.7
2.0
22.8+=15.3
relocate-cloned
-0.1
0.0=0.0
0.1
Adroit Tasks Total
116.8
190.3
221.8
F
Limitations
The proposed EPQ significantly improves performance over the existing CQL baseline on various D4RL tasks, but there many hyperparamaters that need to be optimized. We newly consider the penalty control threshold т, clipping factor Cmins the cluster radius c, and the regularizing temperature Ç. Therefore, in order for the proposed EPQ to perform well, it is necessary to find the optimal performance by considering various hyperparameter setup, which may require some interaction with the environment.
G Broader Impact
Nevertheless, in real-world situations, engaging with the environment can be costly. Particularly in high-risk contexts such as disaster scenarios, acquiring adequate data for learning can be quite challenging. Our research is primarily focused on offline settings and we present novel method, real-life situations where the interaction
EPQ, holds the potential for practical applications not available, and exhibits promise in addressing the challenges posed by offline RI. algorithms. Consequently, our work carries several potential societal implications, although we believe that none require specific emphasis in this context.
24

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims made in the abstract and introductions are well reflected in Section 3 Methodology and Section 4 Experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
25

---

Justification:
The limitations are addressed in the Appendix Limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sele--ppeiiiainnn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: For each theoretical result, the detailed proofs and assumptions are provided in Appendix Proof and Appendix Implementation Details.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The specific environment descriptions and experimental setups including the Justification: hyperparameters can be found in Section Experiments and Appendix Hyperparameter
Setup.
26

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The data and code for reproducing the main experimental results are included in supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nipp.cco wwwppppplcgggpiiiecceeeeeeooeeee www.////e/e//eee///e (oodeSummiisssssee wwtpoiimssssooooosooioooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
27

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The specific experimental setups including the hyperparameters can be found in Section Experiments and Appendix C Hyperparameter Setup.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The graphs included in the paper such as Figure 6 and Figure 7 in Section 4 Experiments well demonstrate the statistical significance of the experiment.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The information on computation resources are provided in Appendix Imple- mentation Details.
28

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compule required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httppp/////uuip....// wttpiiss...pppppeee wttppc/cc//tttpeiieceee.c wtw.hhhiiccceeeeeeeeee ines?
Answer: [Yes]
Justification: The research conducted in the paper conforms the NeurlIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal impacts of the proposed paper is included in appendix G Broader Impacts section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
29

---

Answer: [NA]
Justification:
The proposed paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The baseline code and experimental data are cited both in-text and in the References section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. conddttaetss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
The proposed paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
30

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The proposed paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "0LikOXXCW1", "reviewer": "Reviewer_36me", "summary": "This paper introduces a novel approach to handling distribution shift in\noff-policy RL by the means of Q-function regularization. This is\naccomplished by modulating a penalty term that is overly conservative in\nCQL. The authors argue that CQL overcompensates for the distribution\nshift in cases where state-action pairs exist with non-negligible\ndensity in the dataset, but are still less likely under the reference\npolicy than the learned policy. Based on this insight, the authors\npropose to modulate the CQL value function penalty by a decreasing term\nin the density of state-action pairs under the reference policy, which\nactivates as soon as this density crosses a threshold. Moreover, the\nauthors propose a prioritized sampling method to further reduce the\nunderestimation bias. The resulting algorithm, called EPQ, achieves\nsuperior performance relative to its competitors across many familiar\noffline RL benchmarks.", "questions": "In Figure 1, what is the relationship between $\\tau_i$ and $N_i$\n($i=1,2$)? Such a relationship has not been discussed up to this point.\n\nIn the proof of Theorem 3.1, I believe some assumptions are missing. If\n$\\pi(\\cdot\\mid\ns)$ is ever supported on an action $a'$ such that\n$\\hat{\\beta}(a'\\mid s) = 0$, then $\\Delta^\\pi_{EPQ}\\to\\infty$.\nTherefore, there would be no $\\alpha$ large enough to underestimate the\nvalue function for your argument on line 465 as long as there exists a\nsingle $(s, a)$ in the dataset for which $\\xi^\\delta(s, a) > 0$. Since\n$\\hat{\\beta}$ was defined to be the empirical conditional distribution\nover actions from the dataset, this result actually suggests that\n$\\alpha$ must be infinite whenever your dataset does not fully cover the\naction space (which is always the case in the experiments, where the\naction space is continuous).\n\nIn table 1, why aren't confidence intervals given for the baseline\nmethods?\n\nWhy are there no confidence regions shown in Figure 6? Particularly, it\nwould have been helpful to see these in Figure 6a. As it stands, the\neffect of the prioritized dataset depicted in this figure is a little\nunderwhelming.\n\nWith regard to the analysis of the penalty threshold, why should we\nscale $\\tau$ linearly with the density of the uniform distribution over\n$\\mathcal{A}$ (that is, inversely proportional to the volume of the\naction space)? Figure 6b does not indicate whether the choice of\n$\\tau = 0.2\\rho$ is actually a good choice across environments, so\nindeed it could have just been that $\\tau=0.2\\rho$ happens to work well\nin `hopper-medium` by chance. You have have investigated (or at least\npresented the results) that show how this form of scaling with the\naction space performs.", "limitations": "Limitations are mostly discussed, except for potentially missing assumptions in\nTheorem 3.1.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The motivation for this work was fairly strong, and the authors identified an\ninteresting shortcoming with the overestimation correction in CQL. Figure 1\nfollowed by Figure 4 do a nice job of depicting the influence of this\nshortcoming, and how the proposed method corrects it.\n\nMoreover, the EPQ algorithm is deployed on a large suite of benchmarks, and\noutperforms all competitors with remarkable consistency. To complement these\nresults, the authors conducted experiments to verify that EPQ does in fact\nreduce value estimation bias, shown by comparing EPQ value estimates with\nMonte-Carlo estimates, as well as predictions from CQL. The experiments are\nconducted over four random seeds. While this is relatively few seeds, I think\nthis is acceptable given the range of tasks that were tested.", "weaknesses": "The presentation of the paper (e.g., writing, figures) can be improved.\nIn particular, many of the figures were difficult to read and/or\ninterpret. Both facets of Figure 2 took substantial effort to understand\nfor me (in fact, I think I would have more easily understood the paper\nwithout having seen these figures; see Questions below).\n\nThe statement of Theorem 3.1 is not precise enough, particularly for the\nlatter claim. Moreover, I suspect there are some technical assumptions\nmissing; see Questions below.\n\nFurthermore, I do not entirely understand the motivation for the\nprioritized dataset. Particularly, it is not clear to me that Theorem\n3.1 (the theoretical justification for EPQ) actually applies with the\nprioritized dataset, since the prioritzation depends on the estimated\nQ-function being updated. Beyond that, the ablation of this feature is\nnot very convincing.\n\nWhile there is a wealth of empirical results, confidence intervals from\nthe baselines is largely lacking—this is especially relevant in Table 1.\nSee for example the `door-cloned` row: EPQ is identified as the best,\nbut its confidence region definitely overlaps that of CQL, and probably\nthose of many of the baselines as well. The same goes for\n`relocate-cloned`. as well as (I'd suspect) many of the AntMaze tasks.\nThat said, the results do suggest that EPQ frequently outperforms its\ncompetitors, and rarely does substantially worse.\n\nFinally, it would have been nice to see stronger heuristics for choosing\nthe threshold parameter $\\tau$. Figure 2a suggests that $\\tau$ should be\na function of the amount of data in the dataset, but this is not\nactually discussed anywhere. Rather, the authors claim to have found a\nchoice for $\\tau$ that is inversely proportional to the volume of the\naction space, but results for this choice are only given on one\nenvironment, precluding any conclusion that this choice/trend is good in\ngeneral. The proof of Theorem 3.1 also gives a condition for determining\nwhen $\\alpha$ is large enough, which is roughly inversely proportional\nto the lowest density under the reference policy over all states and\nactions – therefore, choosing $\\alpha$ to be inversely proportional to\nthe volume of the action space is only theoretically justified when the\nreference policy is uniform.\n\n## Minor Issues\n\nThe notation/definition of the Bellman operator $\\mathcal{B}^\\pi$ s not\nexactly correct. In its definition on line 61, $\\mathcal{B}^\\pi$\naverages over all next states $s'$. Then, in the expression on line 63,\nyou are taking an expectation over all state transitions $(s, a, s')$ in\nthe dataset with $\\mathcal{B}^\\pi$ evaluated in this expectation. Since\n$s'$ isn't used anywhere in this expression explicitly, my assumption is\nthat you're using this state as the state to bootstrap from in the\napplication of $\\mathcal{B}^\\pi$; but then $\\mathcal{B}^\\pi$ on line 63\nis not the same as its definition on line 61.\n\nOn line 83, you refer to the \"actual average return $G_0$\", but $G_0$\nwas defined as the random return (not averaged) on line 56.\n\nFormatting of equation (2) is not nice – it almost looks like it's\ndepicting two separate formulas. It may read easier if you instead\ncolored the two factors and described their influence in the text below.\n\nIn Figure 1 and Figure 4, it would be very helpful to see where $0$ lies\non the y-axis on the estimation bias side.\n\nFigure 2a is very busy and difficult to interpret. Firstly, I think it\nwould be better to highligh the magnitude of the penalty itself, as\nopposed ot the penalty reduction (which implicitly depends on some\ninitial penalty, I'm guessing from CQL). Moreover, the relationship\nbetween the amount of data and $\\tau$ should be discussed before this\nfigure, even if superficially (e.g., with a sentence that says that\n$\\tau$ decreases as you collect more data). Then, the figure would have\na much more clear interpretation: increase the amount of data, and the\npenalty will be relaxed more aggressively.\n\nThere appears to be a formatting error on line 140, \"Proof) Proof…\".\n\nThere is a formatting error in equation (5), $Q_(s, a)$ -\\> $Q(s, a)$.\n\nOn line 243, $200k$ should be $200\\mathrm{k}$.\n\nIn Table 1, the \"total\" rows are not good indications of performance,\nfirstly because the returns for the different environments are not\nnormalized. That said, the results for EPQ still look good if you\nneglect the \"total\" rows.\n\nIn Figure 5, it would help to use a different line style to emphasize\nwhich curve corresponds to EPQ. I found it difficult to distinguish EPQ\nfrom CQL ($\\alpha=0.0$) – fortunately these curves generally occupied\ndisjoint regions in the graphs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have addressed the feedback regarding Fig. 2 and conducted additional ablation studies to answer the reviewer's concerns related to the proposed prioritized dataset (PD) and the penalty control threshold $\\tau$, as detailed in our global response. In addition, we have included confidence intervals for all results in the author-provided PDF.\n\n**Scaling and selection of the threshold $\\tau$**\n\nAs mentioned in the global response, the penalty control threshold $\\tau$ is proportional to the log-density of $\\mathrm{Unif}(\\mathcal{A})$. However, we did not intend for the threshold $\\tau$ to scale proportionally to the action volume. Instead, since $\\tau$ is compared to the log-density of $\\hat{\\beta}$, a distribution over the action space $\\mathcal{A}$, we designed the setup so that similar scale factors $\\tau/\\rho \\in [0.2,...,10]$ yield consistent thresholding effects across different action dimensions, without being overly sensitive to changes in the action space. \n\nBased on the experimental results provided for the global response, the overestimation error arises from samples that have not been visited, so the optimal $\\tau$ tends to be determined by how diversely actions are sampled in the dataset for each environment. Fig. 2 and R.1 suggest that, \"with a fixed data distribution\", as the number of datasets $N$ increases, actions are sampled more diversely, which implies that $\\tau$ should be lower. In actual experimental environments, however, since the data distribution varies across environments and states, these factors need to be additionally considered for choosing the optimal $\\tau$. According to our analysis related to $\\tau$, a low threshold is effective when the dataset contains sufficiently diverse state-action pairs, whereas a higher threshold is preferable when the dataset is less comprehensive.\n\n**Confidence interval of the experimental results**\n\nIn offline setups, where performance measurement methods are generally similar, it is common to cite reported results for certain algorithms or experiments [1, 2, 3]. As outlined in Section 4.1, we utilized reported results from the recent MISA paper [1], which did not include standard deviations. For the missing results, we conducted our own experiments to provide the necessary performance data. However, for the experiments we reproduced, such as the modified CQL comparisons in Table 6 for Adroit tasks, we have included confidence intervals as requested by the reviewer. Even when considering these confidence intervals, our proposed EPQ algorithm consistently demonstrates superior performance compared to the CQL baseline. We believe that the proposed EPQ's superiority is well-supported, given its significant enhancement in average performance relative to other baselines.\n\n**Theoretical soundness of Theorem 3.1**\n\n**Penalizing constant $\\alpha$:** Theorem 3.1 states that when the penalizing constant $\\alpha$ is sufficiently large, the $Q$-values $\\hat{Q}^\\pi$ learned based on the EPQ penalty $\\mathcal{P}\\_\\tau$ will underestimate the true $Q$-values. The reviewer inquired whether, according to Theorem 3.1, $\\alpha$ needs to be infinite for $\\hat{\\beta}$ approaching zero to allow for underestimation. We observed that this confusion arises from a typo in the proof of Theorem 3.1 in Appendix A.1. Currently, the proof incorrectly states that $\\alpha$ must satisfy $\\alpha \\geq \\max_{s,a \\in D}[\\xi^\\delta(s,a)] \\cdot \\max_{s \\in D} \\Delta_{EPQ}^\\pi(s)$. However, the correct condition should be $\\alpha \\geq \\max_{s,a \\in D}[\\xi^\\delta(s,a)] \\cdot \\max_{s \\in D} (\\Delta_{EPQ}^\\pi(s))^{-1}$. As $\\hat{\\beta}$ approaches zero, $\\Delta_{EPQ}^\\pi(s)$increases towards infinity, allowing the condition to still be satisfied with a relatively small $\\alpha$, which is an intuitively natural result. This behavior is empirically supported by our experiments, as shown in Fig. 4(c), where we observe significant underestimation of the $Q$-values when $\\hat{\\beta}$ is very small within the support of $\\pi$. Thank you for pointing out this issue, and we will correct the typo in the proof to clarify this matter.\n\n**Theorem 3.1 with PD:** As the reviewer noted, Theorem 3.1 assumes a scenario without the PD and provides a proof under that assumption. When considering PD, $\\hat{\\beta}^Q$ continuously changes with $Q$, which makes it challenging for Theorem 3.1 to hold directly. However, if we assume a fixed $Q$ for PD, Theorem 3.1 can be applied to a fixed $\\hat{\\beta}^Q$ in a similar manner. To address the reviewer's concern in the presence of PD, we use the actual returns from the dataset to obtain $\\hat{\\beta}^Q$ rather than the learned $Q$ during training, as discussed in Appendix B.4. We will add a more detailed explanation of this part in the main paper.\n\n**Minor issues**\n\nWe agree that our notation of the Bellman operator $\\mathcal{B}^\\pi$ and the return $G_0$ should be changed as the reviewer suggests. For $\\mathcal{B}^\\pi$, since we are not using the next state $s'$ the expectation should be over $(s,a)$ rather than over all state transitions $(s,a,s')$. For $G_0$ we will address the issue of overlapping notation between the \"random return\" defined in line 56 and the \"actual average return\" referenced in line 83, as well as correct any typos. Additionally, we will incorporate the reviewer's suggestions on formatting to improve the clarity of the paper. We appreciate your valuable feedback and will work to enhance the paper based on the reviewer's suggestions.\n\n[1] Ma, Xiao, et al. \"Mutual information regularized offline reinforcement learning.\" Advances in Neural Information Processing Systems 36, 2024.\n\n[2] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline reinforcement learning with implicit q-learning.\" arXiv preprint arXiv:2110.06169, 2021.\n\n[3] An, Gaon, et al. \"Uncertainty-based offline reinforcement learning with diversified q-ensemble.\" Advances in neural information processing systems 34, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AX8zzOHJML", "reviewer": "Reviewer_RxmU", "summary": "The paper studies the problem of value estimation bias mitigation in offline reinforcement learning. Specifically, the paper takes the well-known Conservative Q-Learning algorithm as a starting point and improves its penalization scheme with an alternative one that applies provably less value underestimation bias. The paper reports results on the well-known D4RL benchmark and compares against alternative approaches.", "questions": "* Is there a particular reason why the EDAC algorithm of An et al. [40] is not in the comparison list? It is also model free, reports results on the same benchmarks and it performs better than MISA on Mujoco Tasks, i.e. 85.2*18 = 1533.6.\n\n * Section 4.2 does not specify whether the estimation bias is compared between the predicted Q-value and the \"discounted\" observed return. Can the authors confirm that discounting has been applied in the results shown in Fig 6? By bare eye the bias looked to me too high.", "limitations": "The paper does not address its own limitations and does not specify the potential negative societal impact of the presented work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The paper points to a key problem in model-free offline reinforcement learning, i.e. model underfit caused by overconservatism due to excessive penalty application.\n\n * The paper follows a solid methodology to the problem that starts from illustrative data-driven toy results, continues with theoretical analysis, and ends up with a well-understood and well-justified improvement to an established algorithm\n\n* The shown results are particularly comprehensive and strong.\n\n* The presentation of the paper at stellar level. It is extremely easy to follow the story line, even though it is technically quite dense.\n\n* The paper addresses the related literature well.", "weaknesses": "The paper builds its whole problem statement and solution on a big assumption: The environment dynamics will not be learned. In other terms, the problems it highlights and addresses are specific to the model-free offline RL approaches, although learning an environment model is not such a major limitation in the offline setting, where the assumption is that data and compute sources are generous, training time is also not an issue, unlike the online MBRL setup. The paper seems to miss this key positioning element in problem formulation. Its presentation will improve if this choice is made more explicit and the impact of the proposed solution is presented accordingly.\n\nMinor: I found Figure 2 extremely difficult to grasp. The authors may consider simplifying it a little and extending the caption with an explanation of it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have addressed the feedback regarding Fig. 2 and conducted additional ablation studies related to the proposed prioritized dataset (PD) and the penalty control threshold $\\tau$, as detailed in our global response. In response to other concerns raised by the reviewers, we provide the following author response.\n\n**Problem formulation in model-free offline RL**\n\nModel-free offline RL focuses on measuring and addressing overestimation bias in policies $\\pi$ and solving distributional shift issues based on this measure, whereas model-based offline RL concentrates on how to mitigate distributional shift when learning from samples generated using a dynamics model. As a result, these two domains have different focuses and are both areas of active research. In our work, we considered the model-free setting to focus on how CQL, a model-free offline RL method, induces underestimation bias and how to address this issue. However, as noted by the reviewer, since model-based RL can lead to more efficient learning, extending our method to the model-based setting could be a valuable area for future work. We appreciate the reviewer’s suggestion and will actively consider it in our future research.\n\n**Comparison with EDAC**\n\nThere are various approaches addressing the estimation bias of policy actions that may not be present in the dataset in offline RL setups. Our work specifically focuses on directly penalizing $Q$-functions to reduce overestimation, based on the analysis of estimation bias in $Q$-functions. Consequently, we prioritized comparisons with methods like CQL [1] and IQL [2], which share a similar focus on mitigating overestimation. On the other hand, EDAC [3] employs a clipped $Q$-learning method based on the confidence of $Q$-value predictions, presenting a different approach to addressing overestimation. Given that EDAC's methodology differs from ours, we opted to compare offline RL methods that reduce overestimation through penalty or constraint mechanisms, by examining the distributions of $\\pi$ and $\\beta$ from a methodological perspective rather than solely focusing on performance.\n\nHowever, we acknowledge the significance of EDAC in the offline RL domain, as highlighted in our Related Works section. In response to the reviewer’s comments, we will directly compare the performance of our proposed EPQ method with EDAC based on reported results for commonly considered environments in both EDAC and our study. For the Mujoco tasks, the average scores across all tasks were similar, with EPQ scoring 85.4 and EDAC scoring 85.2. In the Adroit tasks, however, EPQ demonstrated superior performance with an average score of 27.7 compared to EDAC’s average score of 17.7, showing nearly a twofold improvement. The Adroit task is particularly challenging due to its sparse rewards and limited variety of state-action pairs in the dataset, which makes learning difficult. Thus, EPQ effectively demonstrates its superiority over EDAC in such challenging scenarios.\n\n**Clarity issues for figures in the paper**\n\n**Fig. 2:** We have regarded the reviewer's concerns on the clarity of Fig. 2 in our global response. Please refer to it for our detailed response.\n\n**Large estimation bias in Fig. 5(a):** In Section 4.2, we have conducted experiments to analyze the impact of our overestimation reduction method along various penalizing constants and reported the results in Fig. 5(a). As the reviewer points out, the estimation bias may look quite large to the bare eye since we plotted the squared value of estimation bias to accurately represent the effects of overestimation and underestimation as mentioned in line 242. Regarding to reviewer's concerns for the results shown in Fig. 6, we can confirm that that discounting has been applied. We will provide additional explanation for this in the paper. \n\n[1] Kumar, Aviral, et al. \"Conservative q-learning for offline reinforcement learning.\" Advances in Neural Information Processing Systems 33, 2020.\n\n[2] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline reinforcement learning with implicit q-learning.\" arXiv:2110.06169 2021.\n\n[3] An, Gaon, et al. \"Uncertainty-based offline reinforcement learning with diversified q-ensemble.\" Advances in neural information processing systems 34, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MVUMN3iIOR", "reviewer": "Reviewer_eWrQ", "summary": "This paper showed that the popular conservative Q-learning introduces bias into the Q function by its restrictive penalty. It proposed to enforce an adaptive penalty term based on the dataset to avoid bias when the dataset support disagrees with the learned policy. Experimental results supported the authors' claims that the proposed method can effectively reduce estimation bias and outperform many baselines.", "questions": "1. In Figure 2 the authors suggested the relationship \n> let $\\tau_2 < \\tau_1$ when $N_1 < N_2$\\\n\nHowever, it seems there is chance that if $\\tau_2$ is only slightly smaller than $\\tau_1$, then $\\tau_2$ would only be penalizing $\\pi_3$ and not covering $\\pi_1, \\pi_2$. More generally, this relates to choosing an appropriate threshold. In the experiments the authors swept $(c\\cdot \\rho, c\\in[0, 10])$, but what was the observation or rule of thumb here to make sure the threshold is meaningful?\n\n2. Another question concerns the implementation. It seems the empirical behavior policy $\\hat{\\beta}(a|s)$ plays a crucial role in the adaptive penalty. In continuous domains the offline datasets often comprise only a single copy for any action, making the estimation of $\\hat{\\beta}(a|s) = \\frac{N(s,a)}{N(s)}$ rather hard and inaccurate. Therefore, existing work like [1, 2] often use sophisticated models to estimate $\\hat{\\beta}(a|s)$ with the hope that it would generalize better. By contrast, it is surprising to see that Algorithm 1 used only simple BC and that was sufficient to fuel the superior performance of EPQ. What could be the reason?\n\nReferences:\\\n[1] Mildly Conservative Q-Learning for Offline Reinforcement Learning\n[2] Supported Policy Optimization for Offline Reinforcement Learning", "limitations": "Potential negative societal impact does not apply and the authors have discussed some of the limitations in the appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Conservatism is common in offline RL methods. CQL is one of the most popular algorithms that enforces the conservatism by penalizing its Q value, and hence it runs the risk of introducing bias. This paper nicely illustrated a possible source of bias originating from the difference between dataset actions and policy actions, and proposed to correct the bias by the penalty adaptation factor. The authors justified the proposed penalty with both pedagogical examples and more challenging problems.", "weaknesses": "While the paper is well-written in general, I believe the presentation can be further improved by adding making the exlanation more accesible, especially Figure 1 and 2. The two figures served a pivotal role in illustrating the downside of CQL and the core idea of the proposed EPQ. Figure 1 would benefit from separating the information into two parts: (1) overlaps of the dataset/policy actions and (2) estimation bias of CQL under different $\\alpha$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have addressed the feedback regarding Fig. 2 and conducted additional ablation studies to answer the reviewer's concerns related to the proposed prioritized dataset (PD) and the penalty control threshold $\\tau$, as detailed in our global response. In response to other concerns raised by the reviewers, we provide the following author response.\n\n**Clarity issues for figures in the paper**\n\n**Fig. 1:** Although we could not include the revision of Fig. 1 due to limited space, we will improve the clarity of the paper by separating the illustration of CQL's estimation bias according to various penalizing constant $\\alpha$ and the distributions of $\\pi$ and $\\hat{\\beta}$, as suggested by the reviewer.\n\n**Fig. 2:** We have regarded the reviewer's concerns on the clarity of Fig. 2 in our global response. Please refer to it for our detailed response.\n\n\n**Choice of the penalty control threshold $\\tau$** \n\nAs noted by the reviewer, it is crucial to select an appropriate $\\tau$ to avoid the situation described by the reviewer. Fig. 2 and R.1 suggest that, \"with a fixed data distribution\", as the number of datasets $N$ increases, actions are sampled more diversely, which implies that $\\tau$ should be lower. However, in actual experimental environments, since the data distribution varies across environments and states, these factors need to be additionally considered for choosing the optimal $\\tau$. \n\nTo address this issue, we provide additional ablation studies regarding $\\tau$ and the rule for selecting the threshold $\\tau$ are detailed in the global response. Based on the experimental results provided for the global response, the overestimation error arises from samples that have not been visited, so the optimal $\\tau$ tends to be determined by how diversely actions are sampled in the dataset for each environment. According to our analysis related to $\\tau$, choosing a low threshold $\\tau$ is effective when the dataset contains sufficiently diverse state-action pairs, whereas a higher threshold is preferable when the dataset is less comprehensive.\n\n**Behavior cloning**\n\nAs the reviewer pointed out, predicting the empirical behavior policy $\\hat{\\beta}$ significantly impacts the performance of EPQ, especially since our penalty adaptation factor $f^{\\pi,\\hat{\\beta}}_\\tau$ is calculated using samples $D$ generated from the predicted behavior policy. Predicting the behavior policy in continuous space is challenging, and various models have been employed for this task, as the reviewer mentions. Instead of using simple behavior cloning (BC), we utilize a prediction model based on the variational lower bound [1], one of the most representative variational inference methods. As described in Appendix B.3, assuming independence among all data samples, the variational lower bound for the likelihood of $\\beta$ can be derived as $\\log\\beta(a|s) \\geq \\mathbb{E}\\_{z\\sim p\\_\\psi(\\cdot|s,a)}[\\log q\\_\\psi(a|z,s)]- D\\_{KL}(p\\_\\psi(z|s,a)||p(z)),~\\forall s,a \\in D$, where $p\\_\\psi (z|s,a)$ is an encoder model and $q\\_\\psi(a|z,s)$ a decoder model parameterized by $\\psi$, and $z$ is the latent variable whose prior distribution $p(z)$ follows the multivariate normal distribution, i.e., $p(z)\\sim N(0,I)$. Please refer to Appendix B.3 for a detailed explanation. By employing this variational approach, we obtained a more precise estimate of the behavior policy $\\hat{\\beta}$ compared to using simple BC.\n\n[1] Kingma, Diederik P., and Max Welling. \"Auto-encoding variational bayes.\" arXiv:1312.6114, 2013.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aFgGnOSDRW", "reviewer": "Reviewer_Hh4A", "summary": "This paper investigates an important problem in offline reinforcement learning (RL), say mitigating unnecessary conservatism in value function. The authors achieve this by selectively penalizing states that are prone to inducing estimation errors, i.e., $f$. The core idea is to train an exclusive penalty $P_\\tau=f_\\tau^{\\pi,\\hat{\\beta}}(s) \\left(\\dfrac{\\pi}{\\hat{\\beta}}-1\\right)$, where $f_\\tau^{\\pi,\\hat{\\beta}}(s)$ is the penalty adaptation factor. $f_\\tau^{\\pi,\\hat{\\beta}}(s)$ assigns smaller weights on in-distribution transitions and a larger weight on out-of-distribution (OOD) transitions. Furthermore, the authors propose the *Prioritized Dataset* (PD) trick to reduce unnecessary bias. Based on PD, the authors derive the final optimization objective of their EPQ approach. The authors conduct experiments on numerous D4RL datasets and show that EPQ can outperform previous methods on many datasets.", "questions": "I have the following questions\n\n- Is there a reason that you do not report the standard deviation of EPQ on halfcheetah-expert dataset in Table 1?\n\n- it seems CQL($\\alpha=0.0$) in Figure 5(a) middle early stops, it there a reason for this?\n\n- how do you expect EPQ to be applied in real-world tasks? Any suggestions or instructions on how to tune the introduced hyperparameters?\n\nSome minor points:\n\n- I noticed that one cannot be directed to the corresponding images/tables by clicking, e.g. Fig 2. This problem hinders the smooth reading and understanding of the content and should be fixed in the revision\n\n- The referenced figures/tables should be distinguished, e.g., Line 90 *Fig. 1(a) and (b)* should be *Fig. 1(a) and 1(b)*. Please check the manuscript and fix all such types of issues.\n\n- Equation 5, $Q\\_( s,a)$ ==> $Q(s,a)$\n\n- sometimes the authors just write *adroit* instead of *Adroit* (e.g., Line 232)\n\n- the captions in Figure 5 and Figure 6 are too simple", "limitations": "The authors include an honest discussion part on the limitations of their work in the appendix. I personally agree with that.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I appreciate that the authors set their focus on value-based offline RL methods and propose an interesting weighting method for mitigating the over-conservatism phenomenon in CQL, akin to mildly conservative Q-learning (MCQ) algorithm. Many recent offline RL algorithms study policy regularization approaches and somewhat neglect the advances of learning offline policies purely from a value function optimization perspective. The overall method is, as far as the reviewer can tell, novel. Though the proposed EPQ method can be seen as one variant of CQL, the proposed method is interesting and can address the unnecessary conservatism on in-distribution data.\n\nThe presentation of this paper is very nice and I personally quite like it. The authors include many toy examples (e.g., Figure 3, Figure 4) and illustrations of their method (Figure 2). This is of great help in aiding the readers to quickly capture the key points that the authors would like to convey. The authors also compare their method against numerous strong offline RL value-based and policy regularization algorithms and demonstrate that EPQ exhibits quite strong performance on many datasets.", "weaknesses": "This paper has the following potential drawbacks\n\n- EPQ introduces many hyperparameters, and one needs to manually find the optimal ones on a new dataset. This can impede the practical application of EPQ in real-world problems\n\n- The ablation study part is insufficient. The authors only conduct experiments on one single environment in the main text and the appendix (see Figure 6 and Figure 7). This ought to be evaluated on wider datasets to thoroughly examine the hyperparameter sensitivity and how different components/hyperparameters affect the performance of EPQ. The influence of the PD trick also should be investigated on wider datasets. Based on Figure 6, it turns out that EPQ with PD and EPQ without PD exhibit similar performance.\n\n- Equation 3 depicts that the target value is *corrected* by the introduced exclusive penalty term $P_\\tau$. The final objective (Equation 4) is also derived based on this. Equation 3 reminds me of the RND-series in offline RL, e.g., SAC-RND [1], and SAC-DRND [2]. Typically, they also subtract a penalty term from the target value to pursue conservatism and mitigate overestimation. I think $P_\\tau$ can also be viewed as a similar role. Any comments here? Are there any advantages of the introduced penalty term $P_\\tau$ over the penalty term given by RND? How these methods can be connected? What if we directly optimize Equation 3 and tune $\\alpha$?\n\n[1] Anti-Exploration by Random Network Distillation. ICML 2023\n\n[2] Exploration and Anti-Exploration with Distributional Random Network Distillation. ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have addressed the feedback regarding Fig. 2 and conducted additional ablation studies to answer the reviewer's concerns related to the proposed prioritized dataset (PD) and the penalty control threshold $\\tau$, as detailed in our global response. In response to other concerns raised by the reviewers, we provide the following author response.\n\n**Practical application of EPQ**\n\nWe understand the reviewer's concern about the practical application of EPQ, given that tuning many hyperparameters for each environment can be costly. However, as the experimental results in Appendix D suggest, EPQ demonstrates plausible performance even with non-optimal parameters. Additionally, as the reviewer suggested, we conducted additional experiments to test the sensitivity EPQ on our main parameter $\\tau$ in Halfcheetah-medium and Hopper-random environments, and reported these findings in Fig. R.3. As explained in the global response, while the optimal $\\tau$ varies depending on the variety of possible actions in each task, EPQ showed consistent performance within a certain range of $\\tau$ values. Notably, in the Hopper-random environment, EPQ achieved superior performance across a range of $\\tau/\\rho \\in [0.2, 0.5, 1.0, 2.0]$. This suggests that precise tuning of all parameters may not be necessary, as EPQ performs well with a reasonable range of parameter settings.\n\n\n**Distinction to intrinsic reward based offline RL**\n\nAs the reviewer pointed out, many penalty-based offline RL methods apply penalties to the target values to reduce the estimation error of policy actions not present in the dataset [1, 2, 3]. Unlike SAC-RND [1], which uses random network distillation as an uncertainty estimator for penalizing uncertain actions, our approach adjusts the penalty based on whether the actions being evaluated are sufficiently represented in the dataset. This allows for a theoretical analysis of the causes of estimation bias and the appropriate degree of penalty required to mitigate it. Additionally, we observed that directly optimizing Eq. (3) in the paper, which imposes the penalty as an intrinsic reward, can lead to unstable learning due to the large penalty continuously fluctuating the target values. In contrast, our approach ensures stable learning by directly applying the penalties to the $Q$-values, thereby enabling the superior performance of EPQ.\n\n**Clarity issues for figures in the paper**\n\n**Fig. 2:** We have regarded the reviewer's concerns on the clarity of Fig. 2 in our global response. Please refer to it for our detailed response.\n\n**Table 1:** The exclusion of the standard deviation of EPQ on Halfcheetah-expert dataset is a typo. The experimental results showed  $107.2 \\pm 0.2$. Thank you for pointing it out.\n\n**Early stopped result in Fig. 5:** In the experiment shown in Fig. 5(a), the estimation bias in CQL with $\\alpha=0$ became excessively large, causing the gradients to explode and resulting in forced termination of the training. In fact, Fig. 5(a) shows that the estimation bias reached up to $1e^{13}$. Therefore, we reported the results in Fig. 5 up to the point where the experiment was terminated.\n\n**Minor issues**\n\nWe will address the reviewer's feedback by correcting typos, distinguishing figure references, and enhancing captions to improve the overall presentation of the paper.\n\n\n[1] Nikulin, Alexander, et al. \"Anti-exploration by random network distillation.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Kumar, Aviral, et al. \"Conservative q-learning for offline reinforcement learning.\" Advances in Neural Information Processing Systems 33, 2020.\n\n[3] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline reinforcement learning with implicit q-learning.\" arXiv:2110.06169, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for your valuable feedback. Based on the reviewers' comments, many expressed difficulties in understanding the figures presented in the paper and suggested that additional ablation studies would be beneficial. Therefore, we provide the following detailed responses to address each of these points:


**Clarity issues for figures in the paper**


In particular, there seems to be difficulty in understanding Fig. 2, which illustrates the motivation behind EPQ, as it lacks sufficient detail. To provide a clearer illustration of our methods, we have revised Fig. 2(a) and Fig. 2(b) into Fig. R.1 and Fig. R.2, respectively, which are included in the author-provided one-page PDF.

**Fig. R.1:** As detailed in Section 3.2, our exclusive penalty is designed to minimize unnecessary bias in the $Q$-function by imposing penalties only when the policy actions are insufficiently represented in the dataset. To illustrate the rationale behind our exclusive penalty, Fig. R.1(a) depicts the log-probability of $\hat{\beta}$ and the thresholds $\tau$ used for penalty adaptation, with $N$ representing the number of data points. In Fig. R.1(a), if the log-probability $\log\hat{\beta}$ of an action $a \in \mathcal{A}$ exceeds the threshold $\tau$, this indicates that the action $a$ is sufficiently represented in the dataset, thus, we reduce the penalty for such actions. Furthermore, as shown in Fig. R.1, when the number of actions increases from $N_1$ to $N_2$, the threshold for determining "enough data" decreases from $\tau_1$ to $\tau_2$, even if the data distribution remains unchanged.

Fig. R.1(b) illustrates the proposed penalty adaptation factor $f\_\tau^{\pi,\hat{\beta}} = \mathbb{E}\_{\pi}[x\_\tau^{\hat{\beta}}]$ for a given $\hat{\beta}$ and policy $\pi$. Here, $x_\tau^{\hat{\beta}} = \min(1.0, \exp(-(\log \hat{\beta} - \tau)))$ represents the amount of adaptive penalty that is reduced as $\log \hat{\beta}$ exceeds the threshold $\tau$. In Fig. R.1(b), $x_{\tau_1}^{\hat{\beta}}$ is larger than $x_{\tau_2}^{\hat{\beta}}$ because $\tau_1 > \tau_2$. Thus, the adaptation factor $f_\tau^{\pi,\hat{\beta}}$ indicates the average penalty that policy actions should receive. As illustrated in Fig. R.1(b), the adaptation factors for different policies vary with their position. Specifically, for threshold $\tau_1$, we have $f_{\tau_1}^{\pi_1,\hat{\beta}} = f_{\tau_1}^{\pi_2,\hat{\beta}} = 1$ and $f_{\tau_1}^{\pi_3,\hat{\beta}} < 1$. For threshold $\tau_2$, $f_{\tau_2}^{\pi_3,\hat{\beta}} < f_{\tau_2}^{\pi_1,\hat{\beta}} < f_{\tau_2}^{\pi_2,\hat{\beta}} < 1$, as depicted in Fig. R.1(b).

**Fig. R.2:** As explained in Section 3.3, we introduce the prioritized dataset (PD) to further reduce the penalty when the policy is highly concentrated on actions that maximize the $Q$-function. To illustrate this, Fig. R.2(a) shows the difference between the original data distribution $\hat{\beta}$ and the modified data distribution $\hat{\beta}^Q$ after applying PD, and Fig. R.2(b) depicts the corresponding penalty graphs. As shown in Fig. R.2(a), when the policy $\pi$ focuses on specific actions, the penalty $\frac{\pi}{\hat{\beta}} - 1$ increases significantly in Fig. R.2(b). In contrast, by applying PD, $\hat{\beta}$ is adjusted to approach $\hat{\beta}^Q \propto \beta \exp(Q)$, aligning the data distribution more closely with the policy $\pi$. Consequently, the penalty is substantially reduced, as depicted in Fig. R.2(b). We believe that Fig. R.1 and Fig. R.2 will provide a clearer understanding of our proposed methods.

**Additional ablation studies**

In response to the feedback from reviewers 36me and Hh4A, we have conducted additional ablation studies for the Hopper-random, Hopper-medium, and HalfCheetah-medium tasks. These tasks demonstrate a significant performance improvement of our method compared to the baseline CQL, as discussed in Section 4.2. Figure R.3(a) provides a component evaluation to illustrate the impact of the proposed PD, while Figure R.3(b) examines performance across various penalty control thresholds $\tau \in [0.2\rho, 0.5\rho, 1.0\rho, 2.0\rho, 5.0\rho, 10.0\rho]$, where $\rho$ represents the log-density of $\textrm{Unif}(\mathcal{A})$. (There is a typo in the paper. We will fix it.) Note that $\rho$ is negative, so $\tau = 10\rho$ is the lowest threshold and $\tau = 0.2\rho$ is the highest.

**The effectiveness of PD:** For the component evaluation, reviewers expressed concerns about the effectiveness of the proposed PD, particularly noting its minimal impact on performance in the Hopper-medium task. However, as shown in Fig. R.3(a), while PD has a limited effect on the Hopper-random and Hopper-medium tasks, it significantly improves performance in the HalfCheetah-medium task, thereby validating the effectiveness of PD.

**The analysis of threshold $\tau$:** Additionally, reviewers inquired about the selection and impact of the proposed hyperparameter $\tau$. Fig. R.3(b) provides insights into the optimal $\tau$ values for each task. The results indicate that in tasks like Hopper-medium, where a variety of actions are not sufficiently sampled, a higher threshold performs better. Conversely, in tasks like Hopper-random, where a broad range of actions is sampled, a lower threshold is more effective. An exception is the HalfCheetah-medium task, which, despite having fewer action variations, visits a diverse range of states. This results in lower overestimation errors for out-of-distribution actions, benefiting from a lower threshold. Additionally, the Adroit task performs well with an extremely high threshold due to minimal noise in the dataset, leading to a limited variety of state-action pairs.

We believe that the additional experiments and analyses, as suggested by the reviewers, robustly validate the effectiveness of the proposed components and significantly enhance the quality of the paper. We again appreciate the reviewers' valuable feedback.

Meta-review: All reviewers (and myself) are in agreement that this work is well-motivated and provides enough empirical evidence for the introduced method. As such, I am recommending acceptance.
Decision: Accept (spotlight)