Peng's Q($\lambda$) for Conservative Value Estimation in Offline Reinforcement Learning
Authors: Byeongchan Kim
Min-hwan Oh
Venue: ICLR 2026 Poster (2026)
Keywords: Offline reinforcement learning
Offline-to-online settings
Multi-step operator
Primary area: reinforcement learning
TL;DR: This paper introduces CPQL: Conservative Peng's Q($\lambda$), mitigates overly-pessimistic value estimation, achieves the performance greater than (or equal to) that of the behavior policy, and provides near-optimal performance guarantees.

Abstract: We propose a model-free offline multi-step reinforcement learning (RL) algorithm, Conservative Peng's Q($\lambda$) (CPQL). Our algorithm adapts the Peng's Q($\lambda$) (PQL) operator for conservative value estimation as an alternative to the Bellman operator. To the best of our knowledge, this is the first work in offline RL to theoretically and empirically demonstrate the effectiveness of conservative value estimation with the \textit{multi-step} operator by fully leveraging offline trajectories. The fixed point of the PQL operator in offline RL lies closer to the value function of the behavior policy, thereby naturally inducing implicit behavior regularization. CPQL simultaneously mitigates over-pessimistic value estimation, achieves performance greater than (or equal to) that of the behavior policy, and provides near-optimal performance guarantees --- a milestone that previous conservative approaches could not achieve. Extensive numerical experiments on the D4RL benchmark demonstrate that CPQL consistently and significantly outperforms existing offline single-step baselines. In addition to the contributions of CPQL in offline RL, our proposed method also contributes to the offline-to-online learning framework. Using the Q-function pre-trained by CPQL in offline settings enables the online PQL agent to avoid the performance drop typically observed at the start of fine-tuning and to attain robust performance improvements. Our code is available at https://github.com/oh-lab/CPQL.

Published as a conference paper at ICLR 2026
PENG'S Q(1) FOR CONSERVATIVE VALUE ESTIMATION IN OFFLINE REINFORCEMENT LEARNING
Byeongchan Kim Seoul National University bckim97e @snuu ac.k.
Min-hwan Oh" Seoul National University minoh@ anu.ac....
ABSTRACT
We propose a model-free offline multi-step reinforcement learning (RL) algorithm. Conservative Peng's 's Q(A) (CPQL). Our algorithm adapts the Peng's Q()) (PQL) operator for conservative value estimation as an alternative to the Bellman operator. To the best of our knowledge, this is the first work in offline RL theoretically and empirically demonstrate the effectiveness of conservative value estimation with the multi-step operator by fully leveraging offline trajectories. The fixed point of the PQL operator in offline RL lies closer to the value function of the behavior policy, thereby naturally inducing implicit behavior regularization. CPQL simultaneously mitigates over-pessimistic value estimation, achieves performance greater than (or equal to) that of the behavior policy, and provides near-optimal milestone that previous conservative approaches could not achieve. Extensive numerical experiments on the D4RL benchmark demonstrate that CPQL consistently and significantly outperforms existing offline single-step baselines. In addition to the contributions of CPQL in offline RL, our proposed method also contributes the offline-to-online learning framework. Using the Q-function pre-trained by CPQL in offline settings enables the online PQL agent avoid the performance drop typically observed at the start of fine- tuning and to attain robust performance improvements. Our code is available at hwwp://thh...oooooaoaeaboaao
performance guarantees-
INTRODUCTION
Offline RL aims to learn policies from a static dataset collected under unknown behavior policies without further interactions with the actual environment. However, offline RL faces major challenge known as distributional shift (Levine et al. 2020). A distributional shift arises when the state-action distribution under the learned policy diverges significantly from that under the behavior policy. This issue is exacerbated when the application of the Bellman updates to value functions requires querying values of outtof-disttibution (OOD) state-action pairs, which can lead to an accumulation of extrapolation errors and ultimately result in poor performance of learned policies.
To tackle OOD actions in policy evaluation, conservative Q-learning (CQL) (Kumar et al. 2020)
penalizes the learned Q-function for OOD actions induced by the learning policy. Building on CQL,
subsequent algorithms (Ma et al. 2021 Lyu et al., 2022 Chen et al., [023 Nakamoto et al. 2023 Shao et al. 2023 Yeom et al.) (024) address the potential over-pessimism
both in-distribution and OOD actions, which stems from excessively conservative value estimates. These approaches rely on additional components, such estimating the unknown behavior policy to handle OOD actions (Lyu eet al. 2022 Yeom et al.) 2024) or introducing extra networks for a quantile (Ma et al. (02T) or a state value function (Chen al. 2023 Nakamoto al. 2023). which may lead to increased complexity and further drawbacks despite their intentions to address the over-pessimism - such drawbacks include distribution mismatches between the estimated behavior policy and the dataset (Zhuang et al.
2023 Kun et al. 2024). the need for extensive parameter tuning, and prolonged training times.
Intuitively, leveraging offline trajectories that span multiple timesteps, rather than individual single- timestep transitions, provides more information about the behavior policy and can potentially prevent
"ooresponding Author

---

Published as a conference paper at ICLR 2026
the selection of OOD actions for offline datasets. Although trajectories readily available in offline datasets, most previous model-free offline RL methods for policy evaluation have utilized these trajectories only in the form of fragmented single-step transitions (Fujimoto et al. 2019 Kumar et al., 2019; Wu et al. 2019 arises:
Fujimoto & Gu 2021 Kostrikov et al. 2021). Hence, the following question
Can we design value estimation method for offline RL that utilizes the multi-step learning?
In online RL counterparts of offline RL, there is line of work that extends single-step temporal-
difference (TD) learning (e.g., Q-learning (WATKINS) 1989) to multi-step generalizations (Peng &
Williams| 1994 Precup 2000 Munos et al., 2016 Harutyunyan et al. 2016; Rowland al. [020
Kozuno et al. 2021). introducing multi-step operators that leverage temporally extended trajectories to update Q-values. These operators improve learning efficiency and provide a more forward-looking perspective, leading to enhanced performance in determining optimal actions compared to the single- step Bellman operator across various benchmarks (Harb & Precup MousaMo
Mousavi al. [001 Hessel et al. 2018 Barth-Maron et al. 20188) Kapturowski et al. 2018 Daley & Amato 2019). However, whether such an extension to multi-step TD learning is possible in offline RL, is still unclear. Hence, the follow-up questions arise:
What is suitable multi-step operator for offline RL?
possible to demonstrate that the multi-step operator enhances performance?
In this paper, we propose an effective conservative multi-step Q-learning algorithm for model-free offline RI,, Conservative Peng's 's Q A) (CPQL). Our algorithm builds on conservative value estimation by incorporating the Peng's Q(A) (PQL) operator (Peng & Williams 1994| Kozuno et al. 2 instead of the standard Bellman operator. Unlike other multi-step operators (Precup 2000; Munos cet al, 2016; Harutyunyan et al. 2016 Rowland et al. 2020) that truncate trajectories, the PQL operator fully leverages whole trajectories improve policy evaluation. Since the PQL operator does not use importance sampling, which requires estimating the behavior policy from offline datasets, avoids the mismatch issues arising from inaccurate behavior policy estimation (Zhuang al. [023 Kun et al.] 2024). Because the fixed point of the PQL operator in offline RI. converges to the Q- function of a mixture policy that interpolates between the behavior policy and target policy, coupling it with a conservative value estimation method ensures that even mild conservatism is sufficient to mitigate Q-value overestimation caused by distribution shift. In contrast to other conservative
methods (Kumar et al. 2020 Ma et al. 2021 Lyu et al.) 2022 Chen et al.) 2023 Nakamoto et al. 2023, Shao et al.,|202: Yeom et al. 2024). CPQL mitigates over-pessimistie value estimation in the Q-function (Theorem without requiring additional estimation procedures auxiliary networks.
Our main contributions are summarized as follows:
We propose CPQL,, the first multi-step Q-learning algorithm for model-free offline RL. CPQL adapts the PQI. operator to conservative value estimation and fully leverages offline trajectories without estimating additional models. To the best of our knowledge, our work is the first to demonstrate both theoretically and empirically the effectiveness of conservative multi-step value estimation.
We provide rigorous theoretical analyses for CPQL. The policy learned by CPQL is guaran- teed to achieve performance greater than (or equal to) that of the behavior policy (Theorem and further reduces the sub-optimalit gap compared to CQL (Theorem 3). Our theoretical analyses effectively address the key limitations of over-pessimistic value estimation in offline RI., ensuring balanced conservatism and improved policy exploration.
Extensive numerical experiments on the D4RL benchmark demonstrate that CPQL consis- tently and significantly outperforms existing offline single-step RL algorithms. In contrast to CQL, whose performance is highly sensitive the choice of the conservatism parameter (An) et al. 2021 Ghasemipour et al. 2022 Tarasov et al. 2024b), CPQL remains robust across a wide range of conservatism parameter values.
Beyond the contribution of CPQL to mitigating over-pessimistic value estimation in offline RL, CPQL also contributes to the offline-to-onlin learning framework. Using the Q-funtionn pre-trained by CPQL enables the online PQL agent to avoid the performance drop observed at the start of the online phase and attain robust performance improvements.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Model-free Offline RL. To overcome the distributional shift and extrapolation error, model-free
offline RL methods focus on learning policies using techniques such as penalizing learned value
functions to assign low values to unseen actions (Kumar et al. 2020 Ma et al., 2021] Lyu et al.
2022
Chen et al., 2023 Nakamoto et al.
2023 Shao et al.) 2023 Ma et al. 2023] Yeom et al.
2024). constraining the learned policy to remain similar to the behavior policy (Fujimoto et al.
2019
Kumar et al., 2019 Wu et al. [019 Fujimoto & Gu 2021 Fakoor et al.) 2021 Ghasemipour
ct al.
2021: ct al. 2022 Tarasov et al. (022a), quantifying the uncertainty (Wu et al. I
Zanette et al.] 2021) by adding ensemble techniques to obtain a robust value function (Bai et al.
[022]] Yang et al. 2022 Nikulin et al., 2023). and learning 2022). However, most model-free offline RL algorithms use the single-step TD learning in off-policy methods based on TD3 (Fujimoto et al. 2018). SAC (Haarnoja et al. 2018), and AWR (Peng et al., 2019). Thus, our work addresses over over- pessimiiiii value estimates by leveraging multi-step TD learning based on offline trajectories.
2021]AA al. 2021 Ghasemipour et al.
without querying OOD actions (Chen et al. [020 Kostrikov et al.
Multi-step Operators. Among off-policy multi-step operators (WATKINS) 1989; Peng & Williiam
1994. Cichosz 1994. Sutton & Barto, 1998, Precup 20000; Munos et al. [016;] 2016; Harutyunyan et al.
2016: Rowland et al.
Kozuno et al. 20211 Daley et al.) 2023) in online RL, the PQL operator consistently outperforms the Bellman operator and other multi-step operators across several complex online tasks (Harb & Precup, 2011 Mousvvi et al. 2017 Hessel et al. 2018 BBrth-Maron et al. [0018) Kapturowski et al. 2018: Daley Amato 2019). The fixed point of the PQL operator in online RL has been criticized for its inability to converge to the optimal Q-function without additional technical conditions, such as the updated behavior policy being close to the target policy (Harutyunyan et al., 2016 Kozuno et al. (021)- However, under the fixed behavior policy (offline settings), exploit the property (Kozuno et al.] 2021)) that its fixed point is closer to the Q-function of the behavior policy. By integrating conservative value estimation into this property, CPQL tackles two central offline RL. challenges: distributional shift and overly pessimistic value estimates.
Oftline-to-Onliee RL. To prevent the forgetting of offline pre-training benefits and enable efficient online exploration, filliee-to-onlng RL methods have explored diverse techniques such as leveraging an offline dataset to cumple-efficient online fine-tuning (Nair et al., 2020 Lee et al. [0222] Song et al.
2022), avoiding the need to retain offline data (Uchendu et al. 22233, Zhou et al. (2224) maintaining
a balanced replay buffer (Ball et al. 2023, Ji et al. 2024; I Luo et al. (024)) calibrating the value function (Nakamoto et al., (023), adopting Bayesian approaches (Hu et al.] (024).. bridging the value gap between offline and online RL (Yu Zhang 2023 Wagenmaker & Pacchiano 2003), and
proposing policy expansion schemes (Zhang al. 2023). However, since CPQL mitigates over- pessimistic value estimation in the offline phase, it eliminates the need for additional mechanisms such critic-actor calibration or alignment when transitioning to vanilla PQL in the online learning. This allows the online agent to directly leverage the pre-trained Q-function without further adjustment, ensuring a smoother transition to online fine-tuning. As result, CPQL avoids the performance drop typically observed at the start of fine-tuning and attains robust performance improvements.
3 PRELIMINARIES
3.1 MARKOV DECISION PROCESS
We consider a Markov Decision Process (MDP) defined by a tuple M == (S.A,P, 7). where S is the state space, A is the action space, P S x A + As represents the state tran- sition probability kernel, R is the reward distribution, do E As is the initial state distribution, and 7 E [0, 1) is the discount factor. We let the reward function r E R be defined as r(s, a) := f r'R (dr'ls,a) a). and assume that |r(s,a)) < Rmax. V (s, a) E SSSx A. Let policy т : + AA be a mapping from states to actions (deterministic) or a probability distribution over actions (stochastic). Given any policy n, an agent starts from initial state so and interacts with M, repeatedly taking actions, receiving rewards, and observing subsequent states. This process generates a trajectory 7 {(s,, a,, (s,, a)))) where a, TT((s,) and 8 P(: s,, at). The value function and action-vall value function (Q-function) for the policy are defined as (=((s := E, 1>=n" "rr(s,, a;) so = s] and Q" (s, a) := E,, E 7 a.) | = s, ag = a]. respectively. We define the discounted state visitation distribution of policy under the environment
3

---

Published as a conference paper at ICLR 2026
Su do. P), where Pr" (sy = so do, P) de- note the probability of reaching state s at time-step under π and P. starting from initial state 5o distributed according to the initial state distribution do. Similarly, we define the discounted state-action visitation distribution as d'M (8,a)= == d'M (8) (als).
M dī = (1-
[s (s
3.2 OFF-POLICY OPERATORS
Off-policy RL consists of two main tasks: evaluation and improvement. The evaluation process is to learn the Q-function of a fixed policy. In the improvement setting, the goal is to obtain an optimal policy x² that maximizes the expected discounted return under do, represented as max,, (TM := --=,Bs,ady (r (s, a)]. Operators are a fundamental concept in RI. because all value-based RL algorithms update the Q-function using recursion Q == O&Qu. where O₂ RSxA + S is an operator that specifies the update rule of the algorithm. We define p as the transition matrix coupled with the policy, given by Pa ((,,Q((ss a) Ex
By-do [V* (s)
20) 1Q(s', a')].
Bellman Operator. The Bellman operator T RSS R is defined as T We denote the set of all greedy policies with respect to Q as G(Q). The Bellman optimality operator T is defined by T = T Q. where नq
yP"
GQQ..
Peng's Q(1) (PQL). For À E 00 PQL updates the Q-function using the recursion QK+1
To R. Qk Peng & Williams, 1994 Kozuno et al. 2021 where F5 G(Qx). The PQL operator
RSxX RSXA is defined Tx T
1) X"-1Tpp Q. where T, Q
T is the uncorrected n-step return operator (WATKINS 1989 Cichosz 1994
Sutton
Barto 1998
Hessel et al. 2018).
3.3
OFFLINE RL
In offline RI., the learned policy is constrained to a static dataset without additional interactions with the environment during the control process. The offline dataset consists of either trajectories {T>)= gathered by unknown behavior policies gg. On all states D, we denote the empirical
Eoeep [|m=s,0=a]
behavior policy as a (a]s) ==
Eiep [|==]] We define the state space induced by D as S(D),
consisting of all states in Since D typically covers a subset of the tuple space, offline RL algorithms based on the Bellman operator suffer from action distribution shift. Because the learning policy is updated to maximize Q-values, cumulative extrapolation errors in unseen actions can drive it toward OOD actions with erroneously high Q-values (Levine et al.. 2020).
To address the overestimated Q-value problem, CQL (Kumar al.) 2020) penalizes the learned Q-function for OOD actions induced by the learning policy. The objective function of CQL with non-negative conservatism parameter ar is defined as follows:
1
E EsD [1Q Q (s, { + (Ea-Daazt 3) 1Q (s. a)] Eg 1 (s) (I)
2
4 CONSERVATIVE PQL
In this section, we develop the CPQL algorithm, where the learned Q-function mitigates overestima- tion bias in value estimation. We provide several novel theoretical results that include guarantees for the sub-optimality gap between the optimal policy and the policy learned via CPQL. It is important note that PQL has not been studied under offiie RL settings. Hence, we first present how previ- ous findings on online PQL can be adapted to offline PQL, addressing fundamental challenges of Q-learning offline RL.
4.1 TOWARDS OFFLINE PQL
Prior works (Peng& & Williams 1994: Sutton & Barto| 1998: Kozuno et al. 2021) have investigated the PQL operator only in online RL. In this work, we focus on constructing the PQL operator in offline RL for the first time. Adapting PQL to offline RL not only facilitates faster convergence the fixed point but also mitigates the effects of extrapolation errors and over- pessimmiiiiiti value estimation, which are key issues in offline RI. We begin by recalling the fixed-point characterization of the PQL.
4

---

Published as a conference paper at ICLR 2026
operator and reinterpreting it from offtine RL perspective. We consider the exact case where update errors exist in the value functions.
Proposition 1 (Harutyunyan et al. (016)) The fixed point of the PQL operator, QTA.", satisfies:
Q=a (AT + (1 T) Q
Proposition states that a fixed point of the PQL operator coincides with the fixed point of XT (1 TM for the target policy π. Since X7 (1 A) T is a contraction with modulus under -nom,, the existence and uniqueness of this fixed point are guaranteed. However, this fixed point does not ensure the convergence of the optimal O-function in online RL unless Th is sufficiently close to π (Harutyunyan et al. 2016 Kozuno et al. 202T). In contrast, when we use the fixed empirical behavior policy R from D, the Q-function updated by the PQL operator converges to QARa+
Proposition 2 (ozuno et al. (2021) Let π be a policy such that (*R 2 Q1 holds pointwise for any policy n. Then, Qs for the k-th iteration, updated by the PQL operator, uniformly converges to Qxxx+(11>)))) with a contraction rate of Bh, where == 2(1-A) 1-7A
Proposition states trade-off between bias and contraction rate, that is, PQL with the fixed behavior policy converges to a biased fixed point that differs from Q", "with a contraction rate 3.
Interpretation offline RL. Prior work (Kozuno et al. 2021) focused on online RL, particularly on how updating the behavior policy is necessary for this fixed point to converge to Q". However, the fixed point Q* with X = O, corresponding to the value derived from the Bellman operator, can still deviate from Q due to distribution shift in offline RL (Fujimoto et al. 2019: Kumar et al. 2019 Levine et al.. 2020). Thus, one our main points is that we should focus on how an appropriately chosen À mittgates Q-value overestimation for the learned policy by shifting the fixed point closer to ºa2, rrtheer than focusing on increasing the bias introduced by a large X. Because the fixed point lies closer to the behavioral value, naturally induces implicit behavior regularization. A carefully chosen can effectively address the pesr-pessmmsmm problem in conservative value estimation methods and yield more robust learned Q-function, mitigates the influence of the learned policy.
4.2 THEORETICAL ANALYSIS
We aim to mitigate the over-pessimistic estimation of Q-values for the learned policy induced by conservatism. We integrate the PQL operator into the CQL loss, as it provides a simple and effective way to alleviate the wer-pessimism of Q-values. We replace the standard Bellman operator Equation with the PQL. operator. This leads to the following iterative Q-value update in CPQL:
( E Eg
[Q((..) TA Qu (s.aa)
Qu+1 E argmin Q
+ a (BawD,awTs (-1s) 1Q (s, a)] E.aND 1Q (s, (s))))) }. (2)
The followwng theorem shows that the expectation of the learned Q-function obtained by iterat- ing Equation] lower-bounds the expectation of the true Q-function. This result an adaptation of Theorem 3.2 in Kumar et al. 2020). The proofs with sampling error are deferred to Appendix B.I
Theorem (Lower Bound on the State Value Function of CPQL) Ler (1-A)( denote the O-function derived CPQL as defined in Equation 2 Then, the state value of Xn + (1 X) m (1 En--(tt
(Q A) (s, a)). lower-bounds the true state value
of the policy obtained via exact policy evaluation, (1-0+(11))(s))) ()).s) Formally, with probability at least 1 - 8, for all S(D) and some 0,
(1 < ( (18x-
The next two theorems show that the policy learned by CPQL is guaranteed to achieve the performance greater than (or equal to) that of ftg (Theorem 2 and reduces the sub sub-optimality gap (Theorem B)) which previous conservative value estimation methods had not achieved. The proof with sampling emror is deferred to Appendix B.2 and B.3
5

---

Published as a conference paper at ICLR 2026
Theorem (Comparison to the Behavior Policy) Let î := argmax, Beddo [
With probability least 6, Ang (1 - X) achieves a policy improvement over iig in the actual MDP M as follows:
a(1 1) and 2 (s) BenACle))
# (als) nB (als)
[]
JM (Añt + #) 2JM (#t)
1-9
Theorem (Sub-Opting (Sub-Optimality Gap) With probability at least 1-8, 6, the gap tfe expected discounted return between the optimal policy πt" and the mixture policy Xi + (1 A) @t under the actual MDP M satisfies
JM (T¹) JM (it, Ant + (1 - R)
2ARmax E
(1 - SnndM 2 drv (at, (t))
20 2)
7 E. ( drv (TT", #)) (())(()) soude Ma Al0" (s)
(((aas) As(aas)
-1 -]
1-7
1-7
where :=)(ss)==eea)) ((aaa) and drv T2) is the total variation distance of π and x2- x)
Discussion of Theorems 2 and 3 In Theorem 2 Arg (1 - A) achieves at least the performance of itS under the actual MDP MM. When accounting for sampling error, a is chosen such that the conservative term exceeds the of the sampling error terms. However, excessively large does not guarantee that the sub-optimality gap decreases. In Theorem 3 increasing a significantly can lead larger influence of {(+) and (-)+) ( (als) / (a)s) s)] on the RHS. To reduce these gaps, is crucial to control the two unbounded terms, since their reduction has a greater effect than reducing the total variation distance. Thus, Ω approaches ihg when a takes on large value by Theorem 2 However, CPQL can reduce the sub-optimality gap more effectively than CQL. If deviates from tæ. leading to {(#) and Fa-+(-le)]f a (a|s) f (a[s) growing to infinity. While CQL. lacks a mechanism to directly mitigate this divergence, CPQL addresses it through X. which balances between the first and second terms on the RHS. For example, since ' and â. are fixed policies, if â.g is similar to x², choosing large value of X further reduces the sub-optimality gap. Conversely, if f.g differs from x², adjusting À appropriately is more effective than CQL at reducing the sub-optimality gap.
4.3 PROPOSED ALGORITIM
Algorithm 1 Conservative Peng's Q()) (CPQL)
Require:
Critic networks Qo, Q0,, Actor network Tg. Dataset D. Conservatism factor a, and Initialize target networks 0_ + 01, 85 + 02
2: for gradient step / = 2, do
3: Sample batch partial trajectories each of length n, {(00: do, ro, $1,01, "1, ",, {n)], from D
4: for = - to 0 do
5: Compute Øb = r, (si+1, (s(+1))) (ot) Qo, (8441, ((())))))))))).11 1.2
6: end for
7: Construct target value y = min, Uú - ana log T (-13,)
8: Update critic 0 for j = 1, with gradient descent via minimizing
aELnD log Σ exp (Qe, (s, a)) - Ea-rs(1s) [Qo, (s, ai]
Be
+
1
[Q0, (s, - -y)
9: Update actor for learned policy ne with gradient ascent via maximizing
EanDannao
min Qo, (s, a) appll log n (-|s) ==1
10: Update target networks: 07 + T0, + (1 0) .j = 1,2
11: end for

---

Published as a conference paper at ICLR 2026
Table 1: Results for MuJoCo locomotion, Adroit manipulation, and AntMaze navigation tasks in offline D4RL. indicates reproduced results: (algorithm*) for all datasets, (score*) for specific dataset. Bold numbers are the scores within 2% of the highest in each environment.
Task
BC
TD3+BC
CQL
IQL
MCQ
MISA
CSVE
EPQ"
CPQL (ours)
2.2
11.0
17.5"
13.1"
28.5
2.5"
26.8
31.9
38.8 + 1.0
hopper-random
3.7
8,5
7.9
7,9"
31.8
9.9*
26.1
30,3
31.5 : 0.5
walker2d-random
1.3
1.6
5.1"
5.4"
17.0
9.0"
6.2
11.2
21.2 : 0.7
malfcheelah-mmdiinn
43.2
48.3
47.0"
47.4
64.3
47.4
48.
67.1
66.6 + 0.9
hopper-medium
54.1
59.3
53.0"
66.2
78.4
96.7
100.4
: 2.0
walker2d-mediut
70.9
83.7
73.3"
78.3
91.0
84.1
83.2
86.4
90.0 = 1.5
halfchectah-
37.6
44.6
45.5"
44.2
56.8
45.6
54.5
51.4
60.3 + 0.8
hopper-ne seeeer--mdddddllllleeee
16.6
60.9
88.7"
94.7
101.6
98.66
91.7
97.3
103.0 + 0.6
replaay
20.3
81.8
81.8"
73.8
91.3
86.2
78.0
86.0
97.4 + 4.0
halfcheetah- medium-cxpee
44.0
90.7
75.6'
86.7
87.5
94.7
93.1
86.6
95.3 + 0.6
hopper-medium-expee expert
53.9
98.0
105.6'
91.5
111.2
109.8
94.1
110.4
111.3 + 1.2
waller22d-meiime exppett
90.1
110.1
107.9*
109.6
114.2
109.4
109.0
110.9
halfcheetah-cxpert
91.8
96.7
96.3"
95.0"
96.2
95.9"
93.8
102.9
98.0 :
hopper-expert
107.7
107.8
96.5"
109.4"
111.4
111.9"
111.3
111.1
112.0
walker2d-expert
106.7
110.2
108.5"
109.9*
107.2
109.3*
108.5
109.8
114.1 + 0.5
MuJeCo Total
744.1
1013.2
1010.2
1033.1
1188.4
1081.4
1121.4
1193.7
1252.1
pen-human
34.4
64.8"
37.5
71.5
68.5
88.1
106.2
65.7
72.1 1: 4.6
door-human
0.5
9.9
4.3
2.3
5.2
2.8
5.1
1: 2.2
hammer-human
1.5
1.8"
4.4
1.3
8.1
3.5
0.3
1.4
relocate-human
0.0
0.1"
0.2
0.1
0.1
0.1
0.1
0.1
0.1 + 0.0
pen-cloned
56.9
49.0"
39,2
37.3
49.4
58.6
54.5
55.8
70.9 + 6.9
door-cloned
-0.1
0.0"
0.4
1.6
1,3
0.5
1.2
0.5
6,4 1 5.0
hammer-cloned
0.8
2.1
2.1
1.4
2.2
0.5
1.2
1.6 + 1.1
relocate-cloned
-0.2"
-0.1
-0.2
Adroit Total
93.9
115.7
93.6
118.1
124.3
162.7
168.5
128.7
166.7
antmazz-umaze
65.0
78.6
74.0
87.5
98.3"
92.3
96.2
96.7 + 1.9
salmaze-umuze-dliigg
55.0
71.4
84.0
62.2
80.0'
89.1
72.3
68.6 + 0.5
nntmaze-mediume play
0.0
10.6
61.2
71.2
52.5"
63.0
59.0
72.4 + 1.2
antmaze- wem--mmmddiddeeeeeee
0.0
3.0
53.7
70.0
37.5'
57.5
71.7 : 0.8
antmazz-large-play antmaze-largee lainazz--arge-diverse
0.0
15.8
39.6
2.5"
17.5
23.8
41.6 5.2
0.0
0.0
14.9
47.5
7,5'
23.4
17.4
46.6 1 4.9
Antmaze Total
120.0
163.8
303.6
378.0
278.3*
348.1
326.2
397
Algorithm| D presents a general version our proposed method. In Line 5, given a partial trajectory of length 71, we recursively compute the target Q-function using the trace parameter A. While updates are based on SAC (Haarnoja et al. 2018)) we set Ord = O at all steps except the last (Line 7), ensuring stability during Q-function updates. This because the entropy bonus term is added to the target O-function at each step, amplifying its numerical scale and complicating value estimations (Kozuno al. 202T). In Line 8, adopt the log-sum-exp method from CQL (Kumar al.) (019) to incorporate conservative value estimation. Compared to CQL, CPQL reduces the influence of the learned policy on Q-value estimates, enabling stable learning even with a small conservatism factor a (see Question (ii) in Section
EXPERIMENTS
In this section, describe detailed experimental procedures and report the corresponding results to address the following pertinent questions:
(i) How does the performance CPQL compare to prior single-step offline baselines, some of which incorporate conservative value estimation methods, across various tasks and datasets?
(ii) What advantage does CPQL provide over CQL in terms of sensitivity to the conservatism parameter a, and does mitigate over-conservatism while achieving strong performance?
iiii How does CPQL compare with other multi-step operators (c.g., Uncorrected N-step, Retrace, and Tree-backup) when combined with conservative value estimation? (Note that there

---

Published as a conference paper at ICLR 2026
existing offline RL methods with multi-step operator. Here, are asking question on ablation.)
(iv) Can the online PQL agent, using the Q-function pre-trained by CPQL in offline settings, mitigate performance drop at the start of the online phase and enable faster adaptation and improvement in online learning compared to offline-to-online baselines?
For fair comparison, we evaluate all algorithms using results after 1M gradient steps in offline D4RL (Fu et al., 2020). In the offlinc-to-online setting, we first pre-train algorithms for 0.25M offline steps and then fine-tune them for 0.3M online steps. Our score is computed from the policy during the last 10 iterations, averaged over 5 seeds, with denoting the standard deviation across seeds. For CPQL evaluation, we details).
5 to cap the length of the partial trajectories. (See Appendix C for
Tasks. MuJoCo (Todorov et al. 2012) consists of datasets from three environments (HalfCheetthh Hopper, and Walker2d), each with five dataset types (Random, Medium, Medium-Replay, Medium- Expert, and Expert). Adroit (Rajeswaran et 2017 involves two dataset types (human and cloned) and four Shadow Hand robot tasks (hammer, door, pen, and relocate). AntMaze provides three maze layouts (umaze, medium, and large) and three dataset types (umaze, play, and diverse).
Baselines. In the offline setting, we compare CPQL to prior model-free single-step offline RL algorithms: (i) TD3+BC (Fujimoto Gu 202T) that incorporates an explicit policy constraint through the behavior cloning (BC), (ii) CQL (Kumar et al., 2020) that penalizes the Q-function for OOD actions, (iii) IQL (Kostrikov et al., 2022) that learns the Q-function without querying OOD actions, (iv) MCQ (Lyu al.] (2022) that uses the mildly conservative Bellman operator, (v)
MISA Ma et al. 2023) that constrains the policy based on mutual information, (vi) CSVE (Chen ct al. 2023) that learns conservative state- value function, and (vii) EPQ (Yeom et al. 2024) that learns the Q-function by selectively penalizing states with insufficient action coverage. In offline- to-online RL, evaluate the performance of CPQL (offline pretraining) followed by PQL (online fine-tuning), and compare it against several algorithms: (i) AWAC (Nair et al. (020) that utilizes the advantage weighted actor-critic with weighted maximum likelihood, (ii) Cal-QL (Nakamoto al. 2023) that calibrates the value-function. (iii) IQL (Kostrikov et al., 2022). (iv) SPOT (Wu et al. 2022) that uses density-basse based regularization, and (v) CQL (Kumar al.
2020 (offline) to SAC (online).
5.1 RESULTS ON OFFLINE AND TOOIIIE-O-ONLINE D4RL
Question (i): Our experimental results, summarized in Table are based on evaluations carried out across diverse tasks. CPQL achieves high performance the vast majority the tasks with 22 out of 29 tasks. In MuJoCo locomotion tasks, CPQL consistently achieves remarkable performance improvements across all tasks, regardless of data distribution-wheeee whether diverse (Random, Medium- Replay) narrow (Medium, Medium Mediiu-Experrt). In Adroit manipulation tasks, CPQL surpasses all other algorithms for door on human and cloned datasets. Excluding only CSVE in the pen-human dataset, achieve high performance in pen tasks. In Antmaze navigation tasks, CPQL demonstrates outstanding performance despite sparse rewards and diverse datasets (undirected and multi-task). These results diverse tasks demonstrate that CPQL effectively mitigates the problem of over- pessimistic value estimation by leveraging actual trajectories and the PQL operator.
COL
CRQL (00.)
+ CPOL 03
CPCL 0355)
CPQL 0.7)
CPQL 0.5)
Mediu
Medium Replay
Meclum-Expert
Expert
1artiette Srow SrOW
10 60 Consenatiom Parameter
Consenation Parameter
to
Consenvtis Parameter
Consonvation Parameter
Figure I: Normalized scores of different conservatism parameters in Walker2d tasks.
Question (ii): CPQL maintains high performance even at small a, unlike CQL. The smaller a helps
CPQL address the issue of overly penalizing the Q-luuues of certain states in CQL, particularly less observed or unobserved states in D. Prior works (An et al.] 202T Ghasemipour et al.) 2022
Tarasov al. 2024b) have pointed out that CQL is extremely sensitive the choice of a,
8

---

Published as a conference paper at ICLR 2026
CPQL (ours)
COL w/ N-step
CQL w/ Retrace
CQL w/ Tree-backup
Hailchectah-random
Hoppen-edlum-x2
Walker22 modiumm-sllls Haifchhetahh medumrreplyy-u
140
Timemase
Timeetope
Tinedape
Timecips
Figure Comparisons of CPQL (ours) with CQL using alternative multi-step operators on MuJoCo tasks.
CPQL PQL (eurs)
AMACC
Cal-QL
SPOT
CQL SAC
Hb iantchecah-anndmmi
Hoppen r2ndom
Mallie2d-radomme
Hopper medium-v22
llqpeerredmmepplayy D
2000
Halicheeah--mmduume sapern--pppri22
Wallber
N
Millen Timedappe
Millin Timechegs
Millan Timedigg
la Timeerege
Figure 3: Comparing CPQL-+PLL (ours) with several baselines for offline-to-online RL.
small changes can lead to significant performance differences. In Figure I the red line representing CQL. clearly illustrates this sensitivity issue. In contrast, CPQL outperforms CQL. and exhibits less sensitivity to across diverse datasets. By mitigating over-conservatism, CPQL enables the learned policy to better explore promising actions. As shown in Theorem selecting an appropriate reduces the sub-optimality gap and yields remarkable scores across diverse datasets.
Question (iii): In Figure 2 the Uncorrected ri-tep return, Retrace, and Tree-backup operators indeed learn faster during the first 0.2M steps, but their performance drops after reaching an early peak. Retrace (Munos et al.) 2016) suffers from performance degradation because it relics on accurate behavior policy estimation, which is difficult estimate (Zhuang et al. 2023 Kun al., 2024). Tree-backup (Precup) 2000) developed for discrete action spaces, and in continuous spaces, it leads unstable updates due the numerical scale of The Uncorrected n-step return overly restricts exploration of OOD actions, which can lead to unstable performance in the later stages of training. However, CPQL achieves both stable and competitive performance without additional requirements.
Question (iv): In Figure 3 we show that initial- izing PQL with the Q-function pre-trained by CPQL helps the online agent avoid or quickly recover from the performance drop at the start of online fine-tuning and achieve robust improve- ment. First, CPQL outperforms other offline- to-online baselines with only 0.25M gradient steps, so the online agent is initialized with the well-trained Q-functon, reducing exploration trials. Second, in Figure the Q-values learned by POL do not degrade at the start of the online
Halfcheelaait Sittteeea---edimm22222
Walleer Wallker2d-mesium-v2
(ochee
Average Valuus
3.0
154
(90) ai Ion't
62- Timestops
Millonn Timestops
Figure 4: Average Q-values by conservatism parameter for CPQL-PPOL (ours), CAC, and Cal-QI..

---

Published as a conference paper at ICLR 2026
phase. Since CPQL reduces the influence of the learned policy Q-value estimation (Proposition and Theorem I. the average Q-value gradually increases after pretraining across different values of a. In contrast, when transitioning from CQL SAC, larger shows severe performance drop. While Cal-QI. avoids the performance drop, its performance improvement is significantly slower.
CONCLUSION
CPQL proposes the first approach a model-free offline multi-step RL algorithm by incorporating the PQL operator for conservative value estimation, mitigating over-pessimism in the Q-function, and reducing the sub-optimality gap. A key insight of CPQL that the fixed point of the PQL. operator lies closer to the value function of the behavior policy, thereby inducing implicit behavior regularization. CPQL outperforms existing offline RL algorithms, and its pre-trained Q-function enables PQL to avoid the performance drop at the start of fine-tuning and achieve robust performance improvements in the online phase. There are two limitations of CPQL. First, CPQL incurs additional computational cost due multi-step backups, but in practice, the overhead and the increase in running time small. Second, on low-quality datasets, performance may degrade, and single-step updates can be preferable multi-step operators. However, CPQL reproduce the single-step TD learning.
ACKNOWLEDGEMENTS
This work was supported by the National Research Foundation of Korea (NRF) grant and the Institute of Information communications Technology Planning Evaluation (ITTP) grant both funded by the Korea government (MSIT) (No. RS-2022-NR07IIE 1853, RS-2023-00222663, RS-2025-25463386 by the Basic Science Research Program through the NRF funded by the Ministry of Education (No. RS- RS-2023-00275091) and by AI-Bio Research Grant through Scoul National University.
USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) were used solely as an assistive tool for writing. Specifically, we employed an LLM improve clarity, grammar, and style of exposition. No part of the research ideation, algorithm design, theoretical analysis, or experimental results involved the use of LLMs. The authors take full responsibility for the content of the paper.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pp. 22-31. PMLR, 2017.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified g-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations, 2021.
PhilipJ Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference Machine Learning, pp. 1577-1594. PMLR, 2023.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, TB Dhruva, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In International Conference Learning Representations, 2018
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in newral information processing systems, 34:15084-15099 2021.
10

---

Published as a conference paper at ICLR 2026
Liting Chen, Jie Yan, Zhengdao Shao, Lu Wang. Qingwei Lin, Saravanakumar Rajmohan, Thomas Moscibroda, and Dongmei Zhang. Conservative state value estimation for offline reinforcement learning. Advances in Neural Information Processing Systems, 37, 2023.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33:18353-18366 2020.
Pawel Cichosz. Truncating temporal differences: On the efficient implementation of td (lambda) for reinforcement learning. Journal of Artificial Intelligence Research, 2:287-318, 1994.
Brett Daley and Christopher Amato. Reconciling X-returns with experience replay. Advances in Neural Information Processing Systems, 32, 2019.
Brett Daley, Martha White, Christopher Amato, and Marlos C Machado. Trajectory-aware eligibility off-polico policy reinforcement learning. In International Conference on Machine Learning. pp. 18318-6835.. PMLR, 2023.
traces for off-
Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander J Smola. Continu- ous doubly constrained batch reinforcement learning, Advances in Neural Information Processing Systems, 34:1 1260-116111 1273, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 35:20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2062-2062. PMLR, 2019.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent without entropy. arXiv preprint arXiv:2301.023:: 2023.
Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. Advances in Neural Information P Processing Systems, 35: 18267-188881, 2022.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq; Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp. 3682-3691. PMLR, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Jean Harb and Doina Precup. Investigating recurrence and eligibility traces in deep q-networks. arXiv preprint arXiv:1704.000 05495, 2017.
Anna Harutyunyan, Mare G Bellemare, Tom Stepleton, and Rémi Munos. Q O with off-policy corrections. In International Conference on Algorithmic Learning Theory, pp. 305-320. Springer, 2016.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski. Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Hao Hu, Yiqin Yang. Jianing Ye, Chengjie Wu, Ziqing Mai, Yujing Hu, Tangjie Lv, Changjie Fan, Qianchuan Zhao, and Chongjie Zhang. Bayesian design principles for offline-to-online reinforcement learning. In International Conference on Machine Learning, pp. 19491-19515. PMLR, 2024.
11

---

Published as a conference paper at ICLR 2026
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. Advances in neural information processing systems, 34:1273-122 1286, 2021.
Tianying Ji, Yu Luo, Fuchun Sun, Xianyuan Zhan, Jianwei Zhang, and Huazhe Xu. Seizing serendipity: Exploiting the value of past success off- actor-critic. International Conference Machine Learning, pp. 11672-21718. PMLR, 2024.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Byeongchan Kim and Min-hwan Oh. Model-based offline reinforcement learning with count-based conservatism. In International Conference Machine Learning, pp. 16728-16746. PMIR, 2023.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783, PMLR, 2021.
Ily Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.
Tadashi Kozuno, Yunhao Tang, Mark Rowland. Rémi Munos, Steven Kapturowski, Will Dabney, Michal Valko, and David Abel. Revisiting peng's s q(A) for modern reinforcement learning. In International Conference on Machine Learning, pp. 7794-5800 PMLR, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy qelarrning via bootstrapping error reduction. Advances neural information processing systems, 33, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 34:1179-1199 2020.
LEI Kun, Zhengmao He, Chenhao Lu, Kaizhe Hu, Yang Gao, and Huazhe Xu. Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization. In The Twelfth International Conference on Learning Representations, 2024.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643 2020.
Jinyi Liu, Ma, Jianye Hao, Yujing Hu, Yan Zheng, Tangjie Lv, and Changjie Fan. A trajectory perspective the role of data sampling techniques in offline reinforcement learning. In Proceed- ings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 1229 1237,1277,000 2024.
Cong Lu, Philip Ball, Jack Parker-Holder Michael Osborne, and Stephen Roberts. Revisiting design choices in offline model based reinforcement learning. In International Conference Learning Representations, 2021.
Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, and Xianyuan Zhan. Offline-boosted actor-critic: Adaptively blending optimal historical behaviors in deep off off-policy rl. International Conference Machine Learning, pp. 3341 1-33331. PMLR, 2024.
12

---

Published as a conference paper at ICLR 2026
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 36:1711-1724, 2022.
Xiao Ma, Bingyi Kang, Zhongwen Xu, Min Lin, and Shuicheng Yan. Mutual information regularized offline reinforcement learning. Advances in Neural Information Processing Systems, 37, 2023.
Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. Advances in neural information processing systems, 14:19235-19247, 2021.
Seyed Sajad Mousavi, Michael Schukat, Enda Howley, and Patrick Mannion. Applying (A)-earning in deep reinforcement learning to play atari games. In AAMAS Adaptive Learning Agents (ALA) Workshop, pp. 1-6, 2017.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09353 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre- pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems. 37, 2023.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In International Conference Machine Learning, pp. (6228-26644- PMLR, 2023.
Kwanyoung Park and Youngwoon Lee. Model-based offline reinforcement learning with lower expectile q-learning. In International Conference Learning Representations, 2025.
Jing Peng and Ronald J Williams. Incremental multi-step q-learning. In Machine Learning Proceed- ings 1994, pp. 226-232. Elsevier, 1994.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintagg-weighhted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 2019.
Doina Precup. Eligibility traces for off off-policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for dynamics and control, pp. 1154-1168.
PMLR, 2021.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087 2017.
Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in neural information processing systems. 35:16082-16097, 2022.
Mark Rowland, Will Dabney, and Rémi Munos. Adaptive trade-off's off-policy learning. In International Conference Artificial Intelligence and Statistics, pp. 34 44. PMLR, 2020.
Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji. Counterfactual conserva- tive q learning for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 37, 2023.
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid Using both offline and online data can make rl efficient. arXiv preprint arX1v:2210.06718, 2022.
Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman inconsistency for model-based offline reinforcement learning. International Conference on Machine Learning, pp. 33177-33194, PMLR, 2023.
13

---

Published as a conference paper at ICLR 2026
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024a.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: mesearch--rieente deep offline reinforcement learning library. Advances in Neural Information Processing Systems, 37, 2024b.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In International Conference Machine Learning, pp. 14556-34583. PMLR, 2023.
Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data in online reinforcement learning. In International Conference on Machine Learning, pp.
PMLR, 2023.
CICH WATKINS. Learning from delayed rewards. PhD thesis, Cambridge University, 1989.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimiza- tion for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 31278-318 31291, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint /911.1 1361, 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdi- nov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In International Conference Machine Learning, pp. 1 11319-13 1328. PMLR, 2021.
Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sample softmax for offline reinforcement learning. arXiv preprint ar Ar3iv:2302. 143.,2230.,, 2023.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang. Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In In-sample learning via implicit value regularization. arXiv preprint MxXXi:2303./8100 2023.
Tengyu Xu, Yue Wang, Shaofeng Zou, and Yingbin Liang. Provably efficient offline reinforcement learning with trajectory reward. IEEE Transactions Information Theory, 2024.
Rui Yang, Chenjia Bai, Xiaoteng Ma. Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. Advances neural information processing systems, 35:23851-23888 2022.
Junghyuk Yeom, Yonghyeon Jo, Jungmo Kim, Sanghyeon Lee, and Seungyul Han. Exclusively penalized g-laarnnng for offline reinforcement learning. Advances Neural Information Processing Systems, 38, 2024.
Tianhe Yu, Garrett Thomas, Lantao Yu. Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:141 11144199-4444 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28967 2021.
Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-online reinforcement learning. In PMLR, 2023.
International Conference on Machine Learning, pp.
Yang Yue, Bingyi Kang, Xiao Ma, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Boosting offline reinforcement learning via data rebalancing. arXiv preprint w:Xiv:2210.09241 2022.
14

---

Published as a conference paper at ICLR 2026
Andrea Zanette, Martin Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances neural information processing systems, 34: 14:16626 13640, 2021.
Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online rennrremeen learning. arXiv preprint urXiv:2302.00935, 2023.
Jing Zhang, Linjiajie Fang. Kexin Shi, Wenjia Wang, and Bingyi Jing. Q-distribution guided learning for offline reinforcement learning: Uncertainty penalized q- q-value via consistency model. Advances in Neural Information Processing Systems, 37:54421-54462 2024.
Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online rein- forcement learning fine-tuning need not retain offline data. arXiv preprint arXiv:2412.07762, 2024.
Zifeng Zhuang, LEI Kun, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization. In The Eleventh International Conference on Learning Representations, 2023,
15

---

Published as a conference paper at ICLR 2026
A PROOF OF TECHNICAL LEMMAS FOR THEOREMS
First, we provide Lemma and proof for the sampling error bound of the PQL operator. We assume the concentration properties of the reward function and the transition dynamics:
Assumption Given a state-action pair (s,a) a) E "D the following relationships hold with probability at least 1- 6,
C. (N
CP
(s) - ((s,a))
- (P(
-
(N
where C and C. constants that depend on 8 E (0, N (s,a) a) is the number of samples for (s, a). and the concentration properties of r and T, respectively.
Under Assumption D and Proposition T the sampling error between the empirical PQL operator and the actual PQL operator can be bounded, as shown in the following proof:
Lemma 1 (Sampling Error Bound of the PQL operator) Given a state-action pair (s. E DD with probability at least 1 1,8, 5, the sampling error between the empirical PQL operator and the actual PQL operator for (8, a) satisfies the following inequality;
C + nG R Rmax/ (1 77)
T Q(s,a) - TR Q(v,a)|
S
(1 (1-,))NN(s,,)
where Ch,p is a constant dependent on the concentration properties r and P, with ō E (0, 1).
Proof For (s,
x a ((s,a) - T/ {(s.a)
(I X)D*0)
(r (1==((-))''Q(sss)) - )
$ (I TAP#))
+ 2 (1=1))(())) (I (APRO) -1
II >AP=.) (1
(I - ))P*)) (1(
f(s a)) + 711 21 /T 8 (T ((s,a))
+ I - >x (I=)) -1
A + 7(1 Q(s
$ (I *DD*> -1 (l((s,)) - f(s,a)) + + - 1) |||pT p1 (-18, - -T ((s,a))
+ +\7 (I >DD#N) - | (.18, (-s - *5 w|)l (ユ->ングm) ) -1 (1 Pmmax
7
c + 71 (1 A)C Amax//aax/
7XCip Rmax/ (1 (1-7)
(1-7A) (s,
(1 (1- / (s,
O + -.0.Bmmae (1 77)
(1 (1-7A)VN(s,a))
This completes the proof of Lemma T
Based on the interpretation of the sampling error of the PQL operator, if À is zero, the sampling error of the PQL operator is equivalent to that of the Bellman operator. For example, when = 0, the sampling error between the empirical PQL operator and the actual PQL operator for (s,a) a) is bounded by 05 C5
This result aligns with the sampling error between the empirical Bellman
(N
operator and the actual Bellman operator (Section D.3 in Kumar et al. (2020))
16

---

Published as a conference paper at ICLR 2026
Now, provide proofs for several technical lemmas that utilize our theorems, such the construction of the conservative value estimation and the sub-optimality gap between the optimal and learned has non-negative values for all states in S(D). In -1 1
((als) Ra(als)
policies. In Lemma
E (a
other words, E. (-1s)
s(als) #a(a]s)
is greater than or equal for any л.
Lemma 2 For any state and any two policies T1 and n2, the following inequality holds:
Ti (als)
Ea (+ss
2 0.
[2(a(()
with equality if and only if m T2-
Proof For any state 8,
( (a|s)
(d|ss) Ti (als))
(2(a)s) -1)
Ba (11a)
1
22 [
Ti (aass) (aass)
(mi(a)s) (2(ss)) + T2(as))
Σ (Ti(a)s)- m2(sss)
(a(ss)
1 + Σ (als)
(a(ss)
1)
(2(a)))
(2(a)s)
(mi(als) T2(als)) (a(as)) -
(T (als) =2(ass)
T2(aas)
(ml()))) =2(as)))
(a)s)
2 0,
where the last equality follows from the fact that Ii (a|s) and 72 (als) are positive values for all
actions and >n Ri (a]s) Ea (2(a This coneludes the proof.
Next, two lemmas are adaptations of Lemma 3 from AAchiam et al. (2017)
Lemma For any two polices क and u2, the vector difference of the discounted state visitation distributions on two different policies holds:
d
2 7 -I- (p - - d
Proof Recall that the discounted state visitation distribution of a policy d", which is defined as
dt (s)= PPr s P.
For finite state spaces, d= can be expressed in vector form as follows:
d= = (1 Σ (TPT) do (1 - -1 -1 do.
where do is the initial state distribution. Then, obtain
d n2 = (1 7) | (1- -p=1)~~> (1 - -pptt) do
= (1 - (I - -/p=1--1 (1- - -pp=)) (I - -/pm))) [[] (I-7p=2)
do
7 (1-7) (1 7pm1)- (px) - -2aa) (1 - yppsa- do
=7 (1 - >p=))~ -1
> (pTi - p d=
This concludes the proof.
17

---

Published as a conference paper at ICLR 2026
Lemma 4 The divergence between discounted state visitation distributions, ||d= d is bounded by an average divergence of the policies Tı and Tz!
||d 2
? 11 S
7
Σ T (als)) =2 (als))
2 7 Bend" (drv (TT) (s)]
where drv (TI, T2) (s) (1/2) E. (a((s) =x((ass)
Proof First, from Lemma
we obtain
|| d 11 = = (T - -/pm) 1 (p - p te ||1
S <> (I - (I m/p=) -1 ||1|| (p pmu d=a 111
( 11 is bounded by:
(I - >P=)) 112 5
(1|Pal = (1 11
t=0
To conclude the lemma, we bound (p=) - pm pm d=
|| (pTL - p=2) d = Σ Σ (prL pmz drd
Σ
pp p 1
8,6
Σ - d=a (s)
Σ ((s'ss.a\\TI(( (2(as)) T (s)
5,0,5'
9,4
= Exnd Σ (i(a|s) n2(a|s)
Therefore, we obtain that:
| d=a 111 S
7
E
Σ (als) T2(als)
If express this inequality in terms of the total variation distance, it becomes the following inequality:
2y Exndez (drv (2) (s))
|d - d= S
1 1-7
This concludes the proof.
18

---

Published as a conference paper at ICLR 2026
We prove the following lemma, which bounds the difference between the expected discounted return under M and M.
Lemma Given any policy n. for any MDP M and the empirical MDP M, the following holds with probability at least 1 6:
JM + (1 - 1) (At + (1 A) m)
Of + RBmax Cp/(1-11
A
π (als)
Afg1(1 1a A)x
(1
1iV Ba s)
1 1-7
snud
(6)
(s)
t (a|s)
Proof To prove this inequality, we use the triangle inequality to separate the gap in the expected discounted return into differences in rewards and transition dynamics, follows:
m) - JM (Art + (1 - >) m) JM (Aii + (1 - 2)
Σ ( (b) (ATT((als)) + (1 - () (8,
8,0
M (s) (AT + (1 A)T) (als)r (s,a)
AM a
d -1- A)a (5) (ATg(a)s) + + -))(a(s)) (1 (3, (s,
M
(s)
+
Σ 5,a
(dtt+t(1 (1- (8) (
+1 A)= (s) ((als))m((s,
1
=:)a(s)
We first bound the term that includes the difference between the actual rewards and the estimated rewards by applying concentration inequalities to derive an upper bound for A, (s). Note that under concentration assumptions, and using the fact that A (s)] in the limit of infinite data, we obtain:
A (s)l < Σ (Ams(aass) (1 A)r(als)) mM (s,a) rM (s,
C
+ (1- (1 ())(((ss / N (s) (a(a)s)
C
> [V +1- A) =(als)
Σ
(3)
(ap((
C (s)
(VIAA+
T(als)
(t((ass)
Next, we bound the term that involves the difference between the actual and estimated transition dynamics by applying concentration inequalities to derive an upper bound for Aa (s). By Lemma 3 we obtain the following equation:
A A== = --(I - (I-yphttt(-Abbr)
-1 (pitatt (DAta+(1-N)) + +
(prt
M
M
=:AP
19

---

Published as a conference paper at ICLR 2026
We know that y is positive and (I (I->p=) 111 $ (1 --)) for any policy n, only need to bound the remaining terms.
A d 1
M
Σ Ap (5 d d (s)
Σ Ap (8 [s)|
d (
M
Σ
Σ
PM (o'ls,a)) (Aāg(als) + (1- A) m(als))) d - (1-0((11111 (s)
Σ
PM PM (1,, - -PM (1b,aa
(Atp(als) (1 ma((())
d - (8)
M
A
(s)
Cp
{ss(aas) + (1 2) t(a|s)
M
VN
Vag(als)
π {(ass)
(6)
A [ + (11
M
(N(s)
/ (
Σ
d A
C
AVIA|+(1-
π [a
(s)
(a|s)
where the last inequality is derived from the Cauchy-Schwar inequality. Hence, we can bound Ad (s) as follows:
Aa(s) 5 20g
Σ
+
(s)
Σ t(als)
(4)
M
(+(()
V V
To derive the final upper bound of the objective function, it is necessary to bound follows:
Σ.
x(als)
as
VFp(ala)
Taass)
(T(als))
π (aass)
Bawa(-ll)
(a(s)
(als)
Talls)
π (als)
LA E
Ba
(a(s))
AT((ass)
Then obtain
(aass)
(als)
(als)) [
E Ba-n((
BaNT((1))
(5)
(
(ass
By Equation 3 Equation 4 and Equation 5 we have that:
JM( An + (1 A) π) J Jz (T, A + (1 - 2) x)
C + y Runae -0-7 7) E
VIA
X+(1- 1)
π (a|s)
1-7
snud st (0
(s)
Banr 8)
के (als)
Equation 6 reflects the trade-off between and
Baar s)
sials) (a(a
1), by weighting them
with À and respectively. This equation can more effectively reduce the difference than the single-step method. This completes the proof of Lemma
20

---

Published as a conference paper at ICLR 2026
B PROOF OF THEOREMS
In this appendix, we provide all proofs of our main theorems with sampling error.
B.1 THEOREM
Theorem (Lower Bound on the State Value Function of CPQL) Let (h denote the Q-function derived from CPQL. as defined in Equation Then, the state value of Xăx + (1 X)
( (8) (oo-(xas+11ANm))i (QM A)m (s, at) lower bounds the true state value of the policy obtained via exact policy evaluation, V fs). sufficiently large Formally, with probability at least 6, for all & E S(D),
D (1-))())) - ( (s),
of 2 Ch+ Chiii//mr////11111 Cb/(1-7) max -1111-1((1->) eD0ED (s,aa) sES(D)
a(aas) - 1))
max
(e
Tz(a
Proof By setting the derivative of Equation 2 to zero, obtain the following recursive update expression for Qk+1 in terms of Qk, incorporating the sampling error under Lemma state-action pair (s, a), with high probability 2 11--00 6:
[ Given
Tx (a(s))
@k+1 (s,a) = po Qu (s,a)
TT (als))
C
STX Fo. Qu (s,
rp
a
+
(a(a)s)
(1- (1-2) /
In Proposition 2 we known that lim Dx -) when the function approximation error zero for every (s,a) E A. Thus, the state value function of Xn + (1 A)TK. on the other hand, VE+1 is underestimated, since:
Th (als))
Ve+1(s) =T,""
Trai Rg. Vi(s) (1
At(aas))
C
+ Boo-(Asst(L-N)aaaG
(1- (1-
N (s,
Now, we can compute the fixed point of the recursion in the above equation. Because the fixed point of the PQL operator coincides with the unique fixed point of (1-xx()))) this gives us the following estimated policy value:
A
(s) -a [[T ()Dh)++(1>))) -1 -V)0)+(1-))(()
(alss)
(s)
(t((ass)
+ ([II-ppAto-a-N Bo-(Xto+(1-A)m))i
C
(s). (7)
(1 (1) 7)
N (s, a)
In this case, the choice of a, that prevents overestimation is given by:
C + y Ruas (
x (als)
- 1 1))
/N(s,a)eESS) (Rawa(-1o)
max max e,EED
(1 7)) (1 2) 7)
āg (a)s)
This completes the proof of Theorem T
21

---

Published as a conference paper at ICLR 2026
B.2 THEOREM 2
We prove that A + (1 - λ) achieves at least the performance of के in the actual MDP M.
Theorem (Comparison to the Behavior Policy) Let π := argmax, Bendo ( (s)
With probability least 6, Xng (1 X) achieves a policy improvement over üa in the actual MDP M as follows:
JM (Ar + +1 a) JJM (+t) + a(1 - -) A)
A (als)
E
Bawa((as)
-
andd
M
Agg (als)
r.P B
VIA
# (als)
and
and
(a)
N (s)
1+
Bana((1s)
Agg (a|s)
where C2 is a constant dependent on the concentration properties r and P.
Proof The proof of this statement divided into three parts:
JM (A#g (1 A) ft JM (it) JM (Aig + (1 A) n) - JM (Ai + (1-
=
J (AT + (1- 1) - J/ (+z) + J JM (TB) - JM (ft))
=:
=:As
By Lemma 5 we obtain the upper bound of A1 and Ds, as follows:
C Rmmax
Cp/(1 Pamaxx//(1
?)
VIA
क (als)
A+ 1) V (
1-7
and 39
145t0-MM -15
(x)
V
Ap [a]s)
A S C 7/maxx C2/(1 1 7)
VIAA
Agg 12 (1- 215
1
and
N (s)
Next, we obtain the lower bound of A2 by the definition of at and Equation 7
a -1 10
*a(s))
JM (f, Ano + (1- x)
Bawa((ls)
2 J (fts 0.
1-7
(5)
A
Thus, have that:
A2 a (1 (1-1)
((ass)
1 1-7
E
and snd
Aig10 are (s)
Baw
E
Aa((as))
1
Therefore, by integrating the bound of A1, A2, and As. we obtain that:
JM (Añg + +11 -)
2 JM (t8) + a
a1
a(1 A)
#(ass)
Bawa(le)
and and ( (s)
1-7
(t(aass)
Ch E (
TA
# (a s)) #e [a|s)
1+ 2)
1 1-9 smid A
(s)
where C, C + R Cp/(1
(1-
This completes the proof of Theorem
2
When = O, we obtain the following equality:
JM (+) JM (ft)
2
E Band (+) Baw+((ls)
#(als)
1
2 T
E
A
Baw+(()))
(a A (als)
(als)
s
(N()))
This result coincides with Theorem 3.6 from CQL (Kumar et al. under the same conditions, thereby ensuring consistency with the COL framework.
2020). Our theorem converges
22

---

Published as a conference paper at ICLR 2026
B.3 THEOREM 3
We first present, to the best of our knowledge, theoretical guarantees concerning the sub-optimality gap between the optimal policy and the mixture policy.
Theorem (Sub-Optimality Gap) With probability at least 1- the gap ofe expected discounted return between the optimal policy n' and the mixture policy AĀ + (1 A) under the actual MDP M satisfies
JM (T*) JM (Artg + (1 2)
2X 2XRmaxx E
snd Melll -0-- (dTv (x", (8)
(1 7)
M
20(1 - 2)
drv (TT #s) E(+)(s) + 7
#(aas)
1
+
E
and 1
1- (-
1-7
(a)
As(aas)
C rPP
VIA
n² (a|s)
1 ?
E
gnud
Ji
(1-)()-
(a)
N (8)
A A
E Bawr-(1))) (-1s)
AB (a ss)
+ 1-7
P.P
E
"ondi (8)
VIAI
N
Eunar(-1))) (-1s)
ftg (a|s)
(als)
where ((#)(s): :- E
x ((als) and dry (TI, is the total variation distance of and (s(a]e)
Proof The proof of this statement divided into four parts:
JM (T-) - JM (A (1 A) R)
JM (x²) JM (T + (T") J JM + (A X) x*)
=:A2
+ JM (Aãt + (1 A) x))²) J/ (Ni + +1-
=:A3
+ J M (Att + (1-))
JM (A#t + (1 A)
=:A+
By Lemma 5 we obtain the upper bound of A1 and 4., as follows:
C y Rmax Cp/(11 7)
VIA
(V ( (11s)
x² (a|s)
A <
E
Ea (x)
1-7
(s)
Te (als)
A C + Rmax
Cp/(1 -))
VIAI (1 / E Bawn (11)
n" (als)
1
andd
A3 + Na*
(s)
N (s)
àg (als)
Next, we derive the upper bound of Az as follows:
|
Σ
(a|s) d (1-)a"'
(s) (A# (1- 1)T (1-1)=') (als)) T ()
7
< 1 1-
Rmax
Σ
ce M
(s)
A (s)
+ 1 1-9
X Xmmxx
d 2 (s) - Az) (s,
3,a
RARmax E
anue A Aln Σ (als) *
XRmax E
Σ r #s(als)]
and Aigll A)
(1 71
1 7
2XRmax
Aas (dry (Tr", #t) (s)],
sodd M
where the second inequality follows from Lemma and the last equality holds with the definition of total variance distance, dry (I,, 72) (s) = Σ. | (a]s) T2(a[s)] /2.
23

---

Published as a conference paper at ICLR 2026
By the definition of f and Equation 7 we derive the upper bound of Ag, as follows:
a
π (a|ss
A S
1 ?
E smd
A -3n
(a
(4)
*(als)
E
and snd - (0)
E (1-
At((ass)
a(1 - X)
x (a|s)
*(a|s)
1 y
E
snud A8 + +01-))=* Ala (s)
E (1a))
Ats(als)
E (1
At(aas))
a(1 X)
(1
(8)
A
(s) Bawa(-10)
*(als)
1
M
M
*t(aas))
a(1 2)
(T* + f (a|s)) (T
A (a|s)]
Σ
1 1-7 7
E
and Ma a(= (6)
tz(a|s)
+ a(1 - >)
*(als)
Σ
- A)a (s)
d'
A 1)m (s)
E (a
1 Y
At(aass)
a(1 ))
Σ n (als) + # (als)
|xt (alas))) # (a|s)]
1 smal ta 1000000 No
(
{())(())
ar (1 (1-
E
Buwa(-1e)
A(als)
1 Σ A(als) π (abs))
(1 71=
and MM
(x)
*t((ass)
2a(1 - A) 1 1-7 ari
(o) drv (TT" #t) (8) ( (8) + 7 BaNa(-18)
T(als) (a((a())
1 1)]
E
(1-+
where the last inequality follows from the definition of & (ft) and Lemma
Therefore, by integrating the bound of A1, A2, As and As, we have that:
JM (T*) JM (Ar + (1 - 2) a)
2X RXRaxx E
snddi Ai -0-- [dTv (m" (s)
(1 7)
+
20(1 - 2)
E
and -00+0--)00+ (o)
drv (TT", ()) ((+)(s) + 7 (.
#(ass)
1 1)]
1-7
--
[a(a)s)
C66 PPP E
VIA
n²" (a|s)
(1-)((-)))+
x -1 A)
E Baw
1 ?
anud Ja
(s)
N (8)
AB (a ss)
PPP
VIA N
n (a|s)
E Eunar(He)) (-le)
1-7
Sond (8)
t/ (als)
This completes the proof of Theorem 3
24

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL DETAILS AND PARAMETER SETUP
In this appendix, we first briefly introduce how normalized scores are calculated the D4RL benchmark. We then describe implementation and experimental details.
C.1 D4RL BENCHMARKS
D4RL provides a metric, the normalized score, which represents a normalized undiscounted average return, to evaluate the performance of offline RL algorithms. It is calculated as follows:
average return return of the random policy
Normalized score return of the expert policy return the random policy 100.
Note that represents the performance an expert policy. In D4RL, if the task is in the same environment, different types of datasets share the same reference minimum and maximum scores. We summarize the reference score for each environment in Table For AntMaze, of times the goal is reached. the ant successfully reaches the goal location, it is rewarded with 1.0, indicating successful episode. Conversely, if the ant fails to reach the goal, it receives reward of 0.0, reflecting an unsuccessful attempt.
random policy, and 100 represents the performance of
set the number of episodes to 100 and evaluate the number
Table The reference minimum and maximum scores for MuloCo, Adroit, and AntMaze datasets.
Domain
Task
Reference Min Score
Reference Max Score
MuloCo
Halfcheetah
-280.18 18
12135.0
MuloCo
Hopper
-20.27
3234.3
MuJoCo
Walker2d
1.63
4592.3
Adroit
Pen
96.26
3076.83
Adroit
Door
-56.51
2880.57
Adroit
Hammer
-274.86
12794.13
Adroit
Relocate
-6.43
4233.88
AntMaze
Umaze Medium Large
0.0
1.0
C.2 BASELINES
C.2.1 OFFLINE BASELINES
To generate the results reported Table I we conduct experiments on MuJoCo "-v2", Adroit "-v0", and Antmaze "-v0" datasets. We adopt behavior cloning (BC), several canonical offline RL algorithms (TD3+BC (Fujimoto & Gu 2021). CQL (Kumar et al. 2020). and IQL (Kostrikov et al., (022)). ) and recent extensions of CQL (MCQ (Lyu al. 2022), MISA (Ma et al. 2023), CSVE (Chen et al. 2023). and EPQ (Yomm et al. 2024)). For a fair comparison, we evaluate all algorithms using results after 1M gradient steps. Thus, certain algorithms must be reproduced for all datasets, while for some datasets, several algorithms with missing values must also be reproduced.
MuJoCo Locomotion Tasks. We take the results for TD3+BC (Table Fujimoto Gu 202 MCQ (Table in Lyu et al. (2022)), and CSVE (Table 1 in Chen et al. (2023) as reported in their original papers. Since the reported scores in the CQL paper are based on "=v datasets, and the scores for BC are needed, we take the scores for BC and CQL from Table 1 in Lyu al. (2022). Since IQL. and MISA papers do not report performance on the Random and Expert datasets, we take the results for IQL from Table in Lyu al. (2022) and for MISA from Table 1 in Yeom al. (2024). For the Medium, Medium- Repllay, and Medium-Esse Expert datasets, we directly take the results of IQL (Table 1 in Kostrikov et al. (2022) and MISA (Table 2 in Ma et al. (2023)) from their original papers. Since the EPQ paper reports scores after 3M gradient steps, we run the official implementation of EPQ on all datasets for IM gradient steps, available at https://gituub.com/yyeeeo
1996/EPQ
Adroit Manipulation Tasks, We take the results for CQL (Table 2 in Kumar et al. (020 IQL (Table 1 in Kostrikov et al. 2022)), MCQ (Table 9 in Lyu et al. (022)). MISA (Table 2 in Ma et al.
25

---

Published as a conference paper at ICLR 2026
(2023) and CSVE (Table 2 in Chen al. (2023) reported in their original papers. Since the scores for BC are needed, we take the scores for BC from Table in Kumar et al. (2020). Since Adroit tasks, take the results for TD3+BC from Table in Yeom et al. (2024). Since the EPQ paper reports scores after 0.3M gradient steps, the official implementation of EPQ on all Adroit datasets for 1M gradient steps, available htwpm//hcoo99///PO I
the TD3+BC papers do not report performance
at
AntMaze Navigation Tasks. We take the results for TD3+BC (Table in Fujimoto & Gu 202
CQL (Table in Kumar et al. (2020)), IQL (Table in Kostrikov et al. (2022), ), and MISA (Table 2 in et al. (2033) as reported in their original papers. Since the scores for BC needed, we take the scores for BC from Table 2 in Kumar et al. (2020). Since the MCQ papers do not report performance on AntMaze tasks, we take the results for MCQ from Table in Yeom et al. (0224). Although repository httpss//github.comm22 wnn.0/2nnnnnmooooooo couooyyoouuuuuoo/cseeee appears to be the code for the paper, it does not provide parameters for the AntMaze dataset, preventing us from conducting experiments. Since the EPQ paper reports scores after 3M gradient steps, the official implementation of EPQ on all AntMaze datasets for 1M gradient steps, available at httpp://gthuub.commhoooooll 1996//
C.2.2 OFFLIN--TO-NNINNE BASELINES
To generate the performance curve reported in Figure 3 we conduct experiments on MuJoCo "-y2" datasets. We adopt canonical offline-to-onling RL algorithms (AWAC (Nair et al. 2020) and Cal-
QL (Nakamoto et al., 2023)), offline RL algorithms that achieve high performance in online RL (IQL. (Kostrikov et al. 2022) and SPOT (Wu et al., 2022 and CQL. (Kumar et al. 2020) (offline) to SAC (online). For fair comparison, we evaluate all algorithms using results after 0.25M gradient
steps for offline settings and 0.3M gradient steps for online settings. run the implementations
of the five algorithms based on the CORL (Tarasov al., 2024b) GitHub repository, available
at https:/gghuub..oottutoofaaaaill aCCCo
C.3 CPQL IMPLEMENTATION DETAILS
Table 3: Hyperparameters setup for CPQL
Hyperparameter
Value
Optimizer
Adam (Kingma 2014
Critic learning rate
3e-4
Actor lcarning rate
lc-4
Batch size
256
SAC hyperparameters
Discount factor
0.99 / MuJoCo and AntMaze
0.90 Adroit
Target update rate
5c-3
Target entropy Entropy in Q-target
-1 Action Dimension
False
Critic hidden dim Critic hidden layers
256
3 MuloCo and Adroit
5 AntMaze
Architecture
Critic activation function
ReLU
Actor hidden dim
256
Actor hidden layers
3
Actor hidden layers
ReLU
Lagrange
True / AntMaze
False / MuloCo and Adroit
conservatism parameter
{0.1, 0.5, 1.0, 5.0, 7.0, 10.0)
Lagrange gap
0.8 / AntMaze
CPQL hyperparameters
Pre-training steps
0
Num sampled actions (during eval)
10
Num sampled actions (logsumexp)
10
Trajectory Length
5
(0.0.0.1,0.33 0.5, 0.7, 0.9, 0.95, 0.99)
26

---

Published as a conference paper at ICLR 2026
We set the trajectory length 5 for CPQL cap the length of the partial trajectories. Across all of our experiments, we tune the conservatism parameter a and from the following potential values with grid search: E {0.1, 0,,, 3, 7, 10} and À E {0, 0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 0.99). In offline-to-onlin RI., we set the conservatism parameter a to either or 5. We extend our experiments include values lower than the previously typical choices of and 10 used in CQL.. We optimize the learned policy following the standard SAC (Haarnoja et al. (2018) approach. We run the CPQL implementation based on the CORL (Tarasov et al. (024b) GitHub repository, available at http://githuu..oo/iinee waakkoof-aa/CORR SAC configuration, detailed Table We summarize the hyperparameters used for running the MuloCo, Adroit, and AntMaze tasks in Table We plot the performance of CPQL in Figure using the best parameters from Tables
ai/CORL The hyperparameter setup for CPQL, including the default
Table Detailed hyperparameters of CPQL, where we conduct experiments on MuJoCo-Gym (*v2") and Adroit and AntMaze ("v0") datasets.
Task
conservatism parameter
PQL parameter À
halfcheetah-randomm
0.1
0.3
halfcheetah-mediumm
0.1
0.0
pircheetab-meddiu--rpp
0.1
0.3
srrcchetta--meiiu--eeeee
10.0
0.1
halfcheetah-expert
3.0
0.0
hopper-random
0.1
0.0
hopper-medium
0.1
0.7
popper-medium-relaa
0.5
0.1
hopper-medium-expeet
5.0
0.1
hopper-expert
10.0
0.9
walker2d-random
0.5
0.9
walker2d-medium
1.0
0.5
walker2dd-medduu--repay
1.0
0.7
wallker2d-mediuu-expeer
1.0
0.95
walker2d-expert
1.0
0.99
pen-human
10.0
0.5
door-human
5.0
0.7
hammer-human
7.0
0.9
relocate-human
1.0
0.9
pen-cloned
1.0
0.5
door-cloned
3.0
0.1
hammer-cloned
5.0
0.7
relocate-cloned
10.0
0.1
antmaze-umaze
7.0
antmaze-diverse
5.0
0.9
untmaze-medium-pple
10.0
0.3
untmaze-meddum-diverss
5.0
0.1
antmaze-large-play
10.0
0.1
antmaze-large-diverse
5.0
0.0
Analysis of the Halfcheettah-experr-22 dataset suggests that the underlying behavior policy is not truly near-optimal (the normalized score the trajectories approximately 85, compared approximately 100 for Hopper and Walker2d). Thus, large À may degrade performance.
27

---

Published as a conference paper at ICLR 2026
COI
CPQL (X-0.3)
CPOL (--0.3)
CPOL (A=0.5)
CPQI (A-0.7)
CPOL (Am0.9)
Hallcheetaahrrnddmm22
Hopperraanom--22
Walke22-rraddm-222
Noraaizee
400
14
Haltchortah-meduu-222
Hogper-medium-v2
Wallker2d mer--meedimmmm.
Noma
316
1s
10
Maiichhettah-mediii---eeaaaayy
Hoppermeeduumreplaay-2
Walker baiikker--mmiiuum-eeeaaa---
Nomaltzed
Halfchoetah-mediie www.mmdddddmmooommmm expert-v2
Hopper mwdium..mmmmmm www.mmime-xppttttt2222
Walkerr 22d medium exp.mrt-t22222222
Nomalized
Halttheetah-exxeett222
Hopper-expert-42
Walker2d-expert--2
Norrelized Score
Timasteps
Timesteps
Timestops
Figure 5: Performance of CPQL in MuJoCo locomotion tasks.
28

---

Published as a conference paper at ICLR 2026
C.4 RUNNING TIME
We compare the computational cost of CQL and CPQL, which use a single-step operator and multi-step operator, respectively. We run such comparison based the Hopper-Medium--22 dataset with single GeForce RTX 3090 GPU. We measure the average runtime per epoch (1K training steps) except for the evaluation step. The results are reported Table We observe that CQL and CPQL have average runtimes of 40.6 and 42.4 seconds, respectively. The runtime difference between the two algorithms minimal, but as shown in Table
we observe significant performance differences.
Table 5: Computational costs of CQL. and CPQL.
Epoch runtime (s) CQL
CPQL
1,000 gradient steps
40.6
42.4
Compared to two recent conservative value estimation algorithms, MCQ (Lyu et al.) 2022) and EPQ (Yeom al. 2024), CPQL only outperforms diverse tasks but also has lower runtime. According to Table 3 in Yeom et al. (2024), the reported runtimes using single NVIDIA RTX A5000 GPU are follows: CQL (43.1 seconds), MCQ (58.1 seconds), and EPQ (54.8 seconds). For fair comparison, we compute the ratio that indicates how much the training time increases in Table We confirm that CPQL is the most efficient compared other algorithms.
Table 6:
Epoch runtime-increase relative to CQL.
Ratio of epoch runtime (%)
CPQL
MCQ
EPQ
Epoch time growth
4.4
34.8
27.1
Additionally, MCQ and EPQ require more training time because they rely on autoencoder- based OOD
action estimation (Lyu et al. 2022) and additional penalty adaptation factors YYeom al.) (022)).
respectively. Therefore, CPQL achieves superior performance with significantly lower computational
cost, outperforming MCQ and EPQ while requiring less training time by avoiding autoencoder-based OOD action estimation and additional penalty adaptation factors.
29

---

Published as a conference paper at ICLR 2026
D COMPARISON WITH OTHER MULTI-STEP OPERATORS
CPQL lourst
COL ml N-stap
LOL Natrace
COL ml Tree-backup
Hallcheetah--anddm--2
Hopper-random-22
Walker26-random-z2
W
Normalzedd Scone
3
Hallcheetah-meddiumz2
Hopper-medium-42
Wallee 2d-m 2d-eedium-22
Normalbed Score
Halfcheetah medium- repluuuy-rrp--
Hepper. replay.v2
Walkerr2d medium replay
Nomallzed Scome
Halfcheetah medium expert-v2
Hopper-medium. medium expertiv2
Walker2d medium export-v2
Nommellpa modddalaadddd Score
falfcheetah--exeett-v
Hopper-xxert--22
Walker 2d-exeer--xx
Nomalized
Timesteps
Timesteps
Timestops
Figure 6: Comparison of CPQL (our) with CQL using alternative multi-step operators (Uncorrected N-step, Retrace, and Tree-backup) on MuJoCo locomotion tasks from D4RL.
30

---

Published as a conference paper at ICLR 2026
Table 7: Computational costs of CQL (baseline), CPQL (our), Uncorrected N-step, Retrace, and Tree-backup on the Hopper-medium--2. We report the runtime per 1,000 gradient steps (in seconds). For Retrace, the additional cost + a accounts for the extra time required to estimate the behavior policy, typically using behavior cloning, CPQL's computational cost is comparable to the single-step operator, with only marginal increase in runtime.
Epoch runtime (s)
CQL (baselinc)
CPQL (ours)
CQL w/ N-step
CQL w/ Retrace
CQL w/ Tree-backkp
1,000 gradient steps
40.6
42.4
41.3
43.0
43.0
E COMPARISON WITH PQL
In this section, we address the following question:
How does CPQI. compare method that purely uses PQI, without the conservatism penalty?
In the main text, focused primarily on evaluating performance across the D4RL benchmarks (MuloCo, AntMaze, and Adroit) in both offline and offline-to-onling settings. We interpret the results to clarify why the PQL operator useful offline RL and why CPQL needed. To this end, we evaluate CPQL and PQL on MuJoCo locomotion tasks, using the normalized return and the critic model's s average Q-values over batch size of samples as evaluation metrics.
Table 8: Normalized Return (Real Performance) of CPQL and PQL.
Task
Algorithm
À = 0.3
À= = 0.7
CPQL (a=
102.6 + 0.8
102.5 + 0.7
hopper-meedium-replly
PQL
24.9 + 10.8
45.3 + 27.9
CPQL (a= PQI.
85.1 + 5.5
79.4 + 18.6
walker2d-medium
-0.2 + 0.0
-0.2 + 0.0
Table 9: Average Q-values (Estimated Values) of CPQL and PQL.
Task
Algorithm
0.3
0.7
CPQL =a
235.8 + 5.6
222.2 1: 4.9
hopper-medium-repphee
PQL
314.7 3.2
271.5 1 4.0
CPQL (a PQL.
335.4
332.6 7.7
walker2d-mediue
4x10 + 1010
475.8 + 6.1
We observe several notable findings in Table 8 and 9 Simply applying PQL to the offline dataset substantially mitigates one of the most important challenges in offline RL, the overestimation of Q-values caused by distribution shift. For instance, in the hopper-medium-repla dataset, SAC reports the normalized score of only around 3.5 (from Table in CQL paper), indicating failure to learn the optimal policy, whereas PQL achieves significantly higher performance. Nevertheless, the distribution shift induced by the learned policy persisted, underscoring the necessity of CPQL to address this limitation more effectively. In the welker2d-medium dataset, PQL with = 0.7 reduced average Q-value overestimation compared to À = 0.3, yet this reduction did not translate into improved normalized return.
In contrast, CPQL combines conservative value estimation with the PQL operator, suppressing overestimation while incorporating long-horizon information. As a result, in both hopper-mediune replay and walker2d-medium datasets, CPQL achieves much more stable and higher returns than PQL, demonstrating that the synergistic integration of conservatism and the multi-step operator plays critical role in improving offline RL performance.
31

---

Published as a conference paper at ICLR 2026
F CUSTOMIZED OFFLINE DATASETS
From the D4RL datasets, it is difficult to determine the exact behavior policy, which makes challenging to precisely measure the role of X. To address this issues, we constructed customized offline datasets in the Halfcheetah and Walker2d environment. Using SAC, we collected 200K samples with the policy obtained the point where the normalized score reached 20. We continued training until the normalized score reached 100, designating this policy as the optimal policy. Based on these setups, we conducted several ablation studies to better understand the effects of CPQL. and X.
F.1 COMPARISON OF CQL, PQL. AND CPQL
Table 10:
Normalized Return (Real Performance) and Average Q-values (Estimated Values) for the customized dataset of Walker2d.
Walker2d
Behavior Policy (T²)
Optimal Policy (л²)
CQL
Normalized Return
20
100 z 267.76
45.9 + 9.5
Average Q-values
2o 192.74
75.9 + 5.4
Walker2d
PQI. (A 0.3)
PQL. 0.7)
CPQI.
0.3)
CPQL (A 0.7)
Normalized Return
-0.5 + 0.0
0.1 1 0.1
63.5 + 8.9
81.3 1 4.5
Average Q-values
4x10 + 2.3x 10"
438.7 + 20.1
129.6 + 5.9
174.3 + 8.2
In Table 10
we set the conservatism parameter a to 5.0 for both CQL and CPQL. Comparing CQL and PQL. CQL produces relatively low average Q-values due the conservatism term, achieving performance of around 45.9. In contrast, PQI. with À = 0.3 suffers from the typical overestimation problem in offline RI, but as À increased to 0.7, its average Q-value decreased to around 438.7. 7. It shows the migration of the over-conservatism effect with the PQL operator. However, PQL failed learn the optimal policy, because the learned policy still suffers large distribution shift, leading to high Q-values.
By adding the conservatism term PQL,, CPQL alleviates this issue and outperforms CQL in terms of performance. This improvement occurs because, under the same conservatism parameter, CPQL has mildly conservative Q-values. This aligns with the theoretical insights in Theorems [ Furthermore, we observe that as À increased, PQL's average Q-values approached those of the behavior policy, whereas CPQL's average Q-values approached those of the optimal policy.
F.2 COMPARISON OF CPQL AND OTHER MULTI-STEP OPERATORS
Multi-step operators without conservatism the optimal policy. Thus, we add the conservatism term with a = 1.0 for all algorithms.
expected to fail to learn policy that approaches
Table 11: Normalized Return (Real Performance) and Average Q-values (Estimated Values) for the customized dataset of Halfcheetah.
Halfcheetah
CPQL.
CQL w/ Nstep
CQL w/ Retrace
CQL w/ Tree-backup
Normalized Return
39.6 + 2.6
31.0 + 2.8
39.5 + 2.8
34.4 + 3.1 130.5 + 7.6
Average Q-values
213.7 + 10.1
127.6 5.2
212.4 10.5
In Table TT CPQL and CQL with Retrace achieved the highest performance (39.6 and 39.5), maintaining relatively high average Q-values, 213, which indicates milder conservatism. In contrast, CQL with n-step returns and Tree-backup showed lower returns (31.0 and 34.4) and substantially lower average Q-values, suggesting stronger conservatism. In this case, the n-step return prevents the agent from exploring OOD actions. Tree-backer backup, the other hand, was developed for discrete action spaces, and in continuous spaces it leads to very unstable updates due the numerical scale of In I.
32

---

Published as a conference paper at ICLR 2026
In the above case of offline dataset collected from single policy, in the previous experiments, estimating the behavior policy is relatively straightforward. This explains why Retrace achieved performance comparable to CPQL. However, open question is whether Retrace would still perform well when the offline dataset is generated by multiple behavior policies. To investigate this, we collected four datasets in Walker2d with normalized scores of 20, 60, and 100, containing 200K, 120K, and 80K samples (ratio 5 3 2), resulting in a total of 400K samples for training. We add the conservatism term with 5.0 for all algorithms.
Table 12: Normalized Return (Real Performance) for toy example of the mixture dataset of Walker2d.
Halfcheetah
CPQL
CQL w/ Retrace
Normalized Return
98.6 + 3.5
87.8 22.8
CPQL outperforms CQL with Retrace, indicating that CPQL has more robust performance for datasets collected from multiple behavior policies. This trend is consistent with the results observed on the D4RL random and medium-replay datasets
G ADDITIONAL BASELINES
G.1 OFFLINE RL
We evaluate our method on MuloCo locomotion tasks in offline settings, comparing it against Q- value uncertainty approaches for conservative estimation (ensemble-based, EDAC (An et al. 2021]
PBRL (Bai al. 22021;; non-ensemble: UWAC (Wu et al.) (021). QDQ (Zhang et al. 2024)) as
well as trajectory-based methods (DT (Chen et al. 202 l). TT (Janner et al. 2021)). Comparison
to additional single-step baselines in effectively regulating OOD actions (X'QLL (Garg et al. 2023).
SQL, EQL (Xu et al.) 2023), and InAC (Xiao al. )023))
Table 13: Results for MuJoCo locomotion tasks. indicates methods trained with 3M gradient steps as reported in original papers. All other methods are trained with 1M gradient steps. Bold numbers are the scores within 2% of the highest in each environment.
Task
EDAC
PBRL
UWAC
QDQ
DT
TT
CPQL (ours)
nalfcheetah-randdon
28.4
11.0
2.3
38,8 + 1.0
hopper-random
25.3
26.8
2.7
31.5 + 0.5
walker2d-random
16.6
8.1
2.0
21.2 + 0.7
halfcheetah-mmediun--2
65.9
58.2
42.2
74.1
42.6
46.9
66.6 + 0.9
hopper-medium-y-
101.6
81.6
50.9
99.0
67.6
61.1
99.7 + 2.0
walker2d medium-v2
92.5
90.3
75.4
86.9
74.0
79.0
90.0 - 1.5
aalicheetah-mediun--epllll
35,9
63.7
36.6
41.9
1 0.8
hopper-medium-rcplaa walker2d-mediune replay-v2
101.0
100.7
25.3
102.4
82.7
91.5
103.0 - 0.6
87.1
86.2
23.6
93.2
66.6
82.6
97.4 = 4.0
halichectal-mmddim--epppn--
106.3
93.6
42,7
99.3
86.8
95.0
95.3 + 0.6
hopper-medium -etp:mmiiiuuumee--t2222222222222
110.7
111.2
44,9
113.5
107.6
110.0
111.3 = 1.2
114.7
109.8
96.5
115.9
108.1
101.9
112.5 = 0.5
aeiicheetaa--xpeer-tee
106.8
96.2
92.9
98.0 + 1.6
hopper-expert-v2 walker2d-expert-v2
110.1
110.4
110.5
112.0 + 0.6
115.1
108.8
108.4
112.5 1 0.4
Across MuJoCo locomotion tasks, in Table [3| CPQL consistently achieves competitive or superior performance compared to both Q-value uncertainty methods (with and without ensembles) and trajectory-based approaches. In particular, it matches or exceeds the strongest baselines medium, medium-replay, and expert datasets, demonstrating robustness across varying data qualities. Fur- thermore, in Table 14] when compared to recent single-step baselines designed to regulate OOD actions, CPQL achieves the highest near-hight near-highest scores across all benchmark settings. These results confirm that CPQL is not only effective in addressing conservatism but also reliable balancing exploration and value estimation, leading to strong and stable returns across diverse offline RL tasks.
33

---

Published as a conference paper at ICLR 2026
Table 14: Results for MuloCo locomotion tasks. Bold numbers are the scores within 2% of the highest each environment.
Task
X-QL
SQL
EQL
InAC
CPQL (ours)
halfcheetah- medium
48.3
48.3
47.2
66.6 0.9
hopper- meeper-
74.2
75.5
74.6
60.3
99.7 2.0
walker2d -mddium
84.2
84.2
83.2
82.7
90.0 = 1.5
45.2
44.8
44.5
44.3
60.3 0.8
Sepper--ediumm-eee
100.7
99.7
98.1
92.1
103.0 + 0.6
malker2d-medium--ee
82.2
81.2
76.6
69.8
97.4 4.0
malfihheetah-meiiu---ee
94.2
94.0
90.6
83.5
95.3 1: 0.6
hopper-meeiiumeeperr
111.2
111.8
105.5
93.8
111.3
1.2
walker2d-medium-eee medium-expert
112.7
110.0
110.2
109.0
112.5
0.5
halfcheetah-expert
93.6
98.0
1.6
hopper-expert
103.4
112.0
walker2d -xppeett
110.6
112.5
0.4
G.2 OFFLINE- TO-OLINE RL
We evaluate our method on the MuJoCo locomotion tasks and AntMaze navigation tasks after fine- tuning with 300k online samples. We report the final normalized score average over five random seeds, with + indicating the 95%%-confiennc interval.
MuJoCo locomotion tasks:
We compare CPQL (offline) to PQL (online) against several algorithms: (i) CQL (offline) to SAC (online), (ii) PEX (Zhang et al. (023) that expands
the policy set during online fine-tuning using optimistic exploration (ii)) RLPD (Ball et al.. 2023) that regularizes online updates using value and policy constraints from offline data, and (iv) Cal-QL. that calibrates the value-function. (see Table 15
These environments known be extremely challenging for standard off- oof-colicy RL algorithms like SAC, due to their sparse rewards and complex exploration requirements. As vanilla online algorithm fails to learn successful policies in these tasks, we compare CPQL against several algorithms, including CQL. (see Table [6]
AntMaze navigation tasks:
Table 15:
Results for MuloCo locomotion tasks in offline to online settings. Bold numbers the scores within 2% of the highest in each environment.
MuJoCo
CQL->SAC
PEX
RLPD
Cal-QL
CPQL->PQL
halfcheetah-random
90.3 4:
60.9
91.5 3.1
32.9 10.1
+ 6.3
hopper-random
33.7 + 34.9
48.5 48.3
90.2 + 23.7
17.7
1.74:32.3
102.0 + 1.7
walker2d-random
3.8 +: 7.9
9.8 + 2.0
87.7 + 17.5
9,4 9.4 7.0
88.6 + 20.1
halfcheetah-medium
96.3 +: 1.6
70.4 + 2.9
95.5 + 1.9
77.0 + 2.7
96.5 + 1.7
hopper-medium
109.3 + 1.1
86,2 + 32.7
91.4 34.5
100.7 + 1.0
111.5 + 0.7
walker2d-medium
114.4 3.2
91.4 17.8
121.6 + 2.9
97.0 10.2
127.8 + 3.4
halfchhctah--mcdiumee
94.8 + 1.9
55.4 6.3
90.1 + 1.6
62.1 + 1.4
95.8 + 2,2
hopper-medium-rrehee replay
108.4 3.4
8.9
78.9 + 30.4
101.4 + 2.6
112.1 + 2.5
114.7 11.1
87.2 16.9
119.0 2.6
98.4 1 4.1
128.6 4.8
From the results presented the table above, CPQL to PQL method achieves significantly better performance compared to other baselines. Several factors contribute to this advantage. First, the Q-function learned by PQL does not degrade at the beginning of the online phase. This is because CPQL reduces the influence of the learned policy on Q-value estimation, resulting in more stable value learning. Second, compared to PEX, RLPD, and Cal-QL, PQL benefits from a stronger exploration capability, it is guided by well-trained Q-function obtained from CPQL..
34

---

Published as a conference paper at ICLR 2026
Table 16: Results for AntMaze tasks in offline-to-onllee settings. Bold numbers are the scores within 2% of the highest in each environment.
AntMaze
CQL
PEX
RLPD
Cal-QL
CPQL
99.0 0.7
95.2 2.0
99.4 + 1.0
90.1 = 13.4
98.2 1.0
catmaze-umaze-ee diverse
76.9 + 49.3
37.4
99.2 - 1.2
90.4
medmaae--mddiu-Ppee
04,4
83,4
97.4 77441.1
95.1 7.8
93.4
1.9
antraaze-mcdummddieens
98.8 3.1
86.6 6.2
98.6 18.6-11 1.7
96.3 6.0
98.2 + 1.8
antmaze-large-play antmazz-large-divense
87.3
56.0 4.8
93.0
75.0
18.2
85.4
6.3
65.3 + 35.1
60.4
90.4 4.8
74.4 =
14.6
82.0 5.6
In the AntMaze tasks, CPQL outperforms (or equal to) other baselines except for RLPD, with only slight performance gap compared to RLPD. The advantage of CPQL becomes even more pronounced when compared to CQL. Taken together, the results from both the MuloCo and AntMaze tasks demonstrate that our algorithm is more robust and delivers superior overall performance.
H ADDITIONAL RELATED WORKS
Model-based Offline RL. Model-based offline RL methods build dynamics and reward models from the offline dataset, leveraging state transitions and rewards of estimated model outputs for planning and policy improvements. They typically achieve this by penalizing the reward function with the
error between the ground truth and estimated models (Yu et al.) 2020 Kidambi et al. 2020 Rafailov
iet al., [021]] Lu et al. 202T; Kim & Oh| 2023 Sun et al., 2023), learning conservative Q function within the model-based regime (Yu al. 2021), D) and traming the policy and the dynamics model adversarially (Rigter et al., 2022). Algorithms that learn by planning synthetic trajectories under esti- mated dynamics typically perform policy evaluation using a single-step approach. However, applying CPQL in model-based offline RL settings can be particularly beneficial, similar to COMBO (Yu et al., 2021). It enables more conservative learning of the Q-function, mitigating overestimation issues. This suggests that CPQL has broad applicability and can enhance various aspects of model-based reinforcement learning.
Recently, Park & Lee (2025) considered model-based offline RL approach, computing the target Q-function by applying lower expectile regression to X-returns on synthetic trajectories planned from the estimated dynamics. This method differs from ours: take model-free offline RL approach, leveraging offline trajectories collected from the actual environment. Our method effectively enhances performance by utilizing real trajectories rather than relying on synthetic trajectories, which are subject to model estimation uncertainty. Furthermore, while they additionally employ lower expectile regression to obtain a conservative return estimate, CPQL derives a conservative value estimate solely by integrating the multi-step operator with conservative estimation mechanism.
Kun et (2024) proposed Uni-O4, an offline policy evaluation method for safe multi-step policy improvement based on approximate model and fitted Q evaluation. However, CPQL. differs conceptually from Uni-O4. First, CPQL applies the PQL operator directly to offline trajectories from the real environment, whereas Uni-O4 computes its multi-step objective by rolling out an approximate transition model π' trained on the offline dataset and aggregating fitted-Q estimates along simulated trajectories (AM-Q). Second, CPQL uses these multi-step returns in the critic update to obtain conservative value estimates for control, while Uni-O4 uses AM-Q only as offline policy evaluation oracle that gates PPO-style policy updates. Finally, Uni-O4 trains an ensemble behavior-cloning policy to stay close the data-support region and stabilize AM-Q, whereas CPQL does not require learning the behavior policy or a transition model.
Offtine Trajectory. Several works (Yue et al. 2022 Liu et al. 2024 Xu al. 2024) have attempted to handle offline trajectories in different ways to adaptively utilize information from past observations, where rewards have already been realized. They propose several methods, such as return-base data rebalancing in Yue et al. (2022). priority assignment based on trajectory quality using average, well as a least-squares-batt based
minimum, maximum, and quantile rewards in Liu et al. (2024), reward redistribution method for reward estimation in Xu et al. (2024). However, these methods are not applicable in sparse reward settings, such as AntMaze tasks, and were not empirically tested in such environments. In contrast, we show that CPQL achieves superior performance in sparse reward settings.
35

---

Published as a conference paper at ICLR 2026
I
ABLATION OF TD LOSS AND Q-VALUES
COL
CPQL (0.1)
CPQL (0.3)
CPQL (0.5)
CPQL (0.7)
CPQL (0.9)
walker2d-medium-reee
waker2ddmatuummaapp expert
240
430
172
320
73
100
10
010
0.11
D.5
000
02
D00
2.0
Figure 7: TD loss of different À
COL
CPOL (0.1)
CPOL (0.3)
CPQL (0.5)
CPOL (0.71
CPOL (0.9)
walker2ddmeedium
wollkerzd-medium---eepyyy
walker2e-medium-xppet
405
300
300
200
Stpps (1e))
360
Stags 2odb
Trmee 11xt)
Figure Q-values of different
CPQL (0.1)
CHQL (o.7)
(0.9)
Bell Markkeer-mmdddun
Wal 2d mecium--ellay
Walloer Sellieer2d-mmee Selllee-d-medduu--exeec
:00
Normaltd Scorra
50
Normalzeed Sccre
Nermabeed Seree
60
20
Tim Stage (1ed)
11me Smapa (1n0)
1ime Staps (ueto
Figure 9: Normalized scores of different
36

---

Published as a conference paper at ICLR 2026
J ABLATION OF TRAJECTORY LENGTH
Table 17: Comparison of CPQL with different trajectory lengths MuJoCo locomotion tasks.
Same (a, A) means that CPQL with πt 10 uses the same hyperparameters (a, À) as = 5, while Best uses the best (a, X) tuned for 10.
CPQL.
Task
5
10
Remarks
Same (a. A)
Best (a, X)
halfchectah-random
38.8 + 1.0
37.4 + 0.9
37.4 + 0.9 +.5
hopper-random
31.5 0.5
31.5 + 0.5
walker2d-random
21.2 0.7
21.3 0.5
2.5
(0.5, 0.9) (0.5, 0.95)
malfchectah-medinge
66.6
66.6
66.6 1.66000 0.9
99.7
100.0 1.5
100.0 1.5
(0.1,
malker2d-medium
90.0
89.4 1.3
89.4 1.3
halfcheetah medium- replay
60.3
60.5
60.5
hopper-medium medium medium-rrpaay
103.0 0.6
102.1
103.2 0.8
(0.5, 0.35,
walker2d--ddium-rella
9.0
95.7 + 4.4
95.7 + 4.4
malfcheetah-medium-exea expert
95.3
94.2 0.7
95.4 0.6
(10.0, 0.1) + (10.0, 0.3)
hopper-medium-expert
+11.3
106.7 + 6.0
110.8 + 2.4
(5.0, 0.1) 7 0.70
malker2d-medium-eee
112.9 0.5
112.8 + 0.4
112.8 + 0.4
(1.0, 0.95) - (1.0, 0.99)
halfcheetah-expertt
98.0 1.6
98.0 + 1.6
98.7 1: 1.0
(3. - (3, 0.3)
hopper-expert
112.0 0.6
111.9 0.6
111.9 +.6
walker2d-expers
114.1 0.4
114.3 0.4
114.3 - 0.4
K
ABLATION OF CONSERVATISM PARAMETER
CPQL (-10.)
CPOL
CPOL 0-55)
Q-
CPQL
Medium
Medkum-Replay
Medium-Espert
Expert
to Parameaa
64
Conderatism Paromezar
Porametor
Consination
Figure 10: Normalized scores of different conservatism parameters in Halfcheetah tasks.
COL
(20.(1-0.11)
CPOL
CPCL (A (.5)
CPQL 0.))
(PQL
Medium
Medium-Replay
Mecium-Expper
Expert
loned Scotee
16
30
10
52
35
Figure 11: Normalized scores of different conservatism parameters in Hopper tasks.
37


Reviews:
[{"review_id": "90rbZYw3Kh", "reviewer": "Reviewer_jSat", "summary": "The paper proposes Conservative Peng’s Q(λ) (CPQL), an offline RL algorithm that replaces the single-step Bellman operator in CQL with the multi-step Peng’s Q(λ) operator. The authors prove conservative lower bounds on value estimates, establish policy-improvement guarantees w.r.t. the behavior policy, and derive a λ–α sub-optimality trade-off. This creates implicit behavior regularization without explicit policy constraints or extra networks, reducing both overestimation (from OOD actions) and over-pessimism.", "questions": "1. Algorithm 1 line 7 subtracts γⁿ α_td log π, but α_td = 0 except at the final step. What role does this term play? Would omitting it entirely affect stability or theoretical guarantees? \n\n2. What is the runtime cost of CPQL relative to CQL and IQL? Please report wall-clock time (to 1 M steps) and sample-efficiency comparisons.\n\n3. Are there tasks where single-step conservative Bellman updates (λ = 0) outperform CPQL? If so, how can practitioners detect when multi-step λ > 0 is harmful?\n\n4.Could you compare to recent multi-step offline RL algorithms such as Uni-O4 (2024) or clarify conceptual differences?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. First (to my knowledge) to pair PQL with conservative value estimation and analyze it theoretically for offline RL. Theorems 1–3 extend CQL-style results: (i) conservative lower bounds; (ii) policy improvement over behavior; (iii) an explicit λ–α trade-off with detailed proofs.\n\n2. Algorithm 1 is straightforward; no behavior-policy estimation (unlike MCQ/CSVE/EPQ).\n\n3. On MuJoCo, Adroit, AntMaze, CPQL is best or second-best on 22/29 tasks; sensitivity to α is substantially lower than CQL (clear practical win). Alternative multi-step operators and offline-to-online results reinforce the story.", "weaknesses": "1. Theory mostly treats the exact evaluation case; neural critics + replay introduce approximation error and potential instability (classic issue with λ-returns). Provide a finite-sample / approximation note (even informal), or an experiment tracking Bellman error vs. λ and critic drift under function approximation. A short stability plot (loss, TD error, divergence events) would help.\n\n2. Recent multi-step offline methods (e.g., Uni-O4) are conceptually close. Either add Uni-O4 (or the closest available multi-step competitor) to Table/Fig. 2 or clearly justify incompatibility. A paragraph contrasting CPQL’s no-IS λ-returns vs. methods relying on IS/retrace would clarify the novelty.\n\n3. Multi-step targets add compute; the paper omits time-to-X (e.g., to 1M steps) vs. CQL/IQL. Add a runtime table and learning-curve area-under-curve (AUC). If CPQL converges faster for similar wall-clock, that strengthens the practical claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c26UtYDVYP", "reviewer": "Reviewer_W4Kk", "summary": "The paper proposes Conservative Peng’s $Q(\\lambda)$ (CPQL) for offline RL. The main idea is to replace the standard Bellman operator in the conservative Q-learning (CQL) objective with the Peng’s $Q(\\lambda)$ operator. The authors present theoretical results showing that CPQL can achieve a tighter suboptimality gap than standard CQL, as the introduction of $\\lambda$ could be used to mitigate the CQL 's excessive pessimism. They also provide extensive numerical experiments and ablation studies demonstrating the effectiveness of CPQL.", "questions": "Could you discuss more on the trade-offs introduced by incorporating Peng’s $Q(\\lambda)$ into offline RL? In addition, please clarify the main limitations of the proposed framework in the \"Conclusion\" section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Interesting idea: Bringing Peng’s $Q(\\lambda)$ from online RL to the offline setting (where the behavior policy is fixed offline) is interesting, and can potentially motivate future research. \n\n2. Mitigates over-pessimism and improves robustness: Introducing $\\lambda$ alongside $\\alpha$ adds a practical control knob that balances conservatism and extrapolation risk, helping to mitigate excessive pessimism and showing reduced hyperparameter sensitivity in experiments.\n\n3. Theory is solid and sound.\n\n4. Comprehensive experiments and thorough ablations", "weaknesses": "1. Although the authors note in the last paragraph of Section 4.1 that we should focus on how an appropriately chosen $\\lambda$ mitigates Q-value overestimation rather than on the additional bias it introduces, I would still suggest adding a discussion of the impact of this introduced bias. One potential direction in my mind is to analyze how the bias affects the suboptimality gap of the learned policy $\\hat{\\pi}$ (not just the mixture policy), but other forms of analysis are also welcome. In short, it would help to quantify how much we gain vs. how much we sacrifice, by introducing Peng's $Q(\\lambda)$ into offline RL.\n\n2. In Algorithm 1, it is not fully clear which policy is ultimately evaluated/deployed: the mixture $\\lambda \\pi_\\beta +(1-\\lambda)\\pi_\\phi$ or the learned actor $\\pi_\\phi$ itself. I suggest to clarify this explicitly in Algorithm 1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AXNXxsAc5r", "reviewer": "Reviewer_MQiE", "summary": "CPQL (Conservative Peng's $\\mathrm{Q}(\\lambda)$ ) integrates the conservative Q penalty structure of Conservative Q-Learning (CQL) with the multi-step backup of Peng's $\\mathrm{Q}(\\lambda)$. To address the limitation of standard CQL, which suffers from restricted reward propagation leading to over-pessimism, CPQL introduces $\\lambda$ to leverage trajectory-level information for long-term reward propagation. This allows the algorithm to stably utilize multi-step information without importance sampling while implicitly capturing the characteristics of the behavior policy in its Q-values. While empirical results demonstrates that CPQL outperforms existing baselines on the D4RL benchmark, the authors further provide theoretical evidence to prove that CPQL guarantees that the learned policy always performs at least as well as the behavior policy.", "questions": "$\\textbf{(1)}$\n\nGiven that CPQL is a multi-step method for offline reinforcement learning, it seems reasonable to consider the trajectory length $n$ as an important hyperparameter in the implementation.\nIn the current paper, $n$ is fixed to 5 for all experiments, but I believe that both the performance and the behavior of the hyperparameters $\\alpha$ and $\\lambda$ could vary significantly depending on the choice of $n$.\nCould the authors clarify the motivation for fixing $n=5$ in all experiments? Additionally, have the authors conducted (or considered) any experiments analyzing how performance or the interaction between $\\alpha$ and $\\lambda$ changes as $n$ varies?\n\n\n$\\textbf{(2)}$\n\nIn Figure 1, the experimental results on the sensitivity of the conservatism parameter $\\alpha$ suggest that, for the same $\\alpha$ values, CPQL tends to outperform CQL.\nMoreover, it appears that as $\\lambda$ increases, the sensitivity of CPQL to $\\alpha$ becomes smaller, indicating that the method may be more robust to the choice of $\\alpha$ under higher $\\lambda$ values.\nI am curious whether similar patterns were observed consistently across other tasks or environments.\n\nCould the authors provide additional results or discussions that show whether this trend holds beyond the task presented in Figure 1?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "$\\textbf{(1)}$\n\nAccording to the authors, Conservative Peng's $Q(\\lambda)$ (CPQL) represents the first attempt to incorporate multi-step temporal difference learning into the offline reinforcement learning setting. The method combines Peng's $\\mathrm{Q}(\\lambda)$ operator with a conservative Q-penalty, aiming to utilize multi-step information through the hyperparameter $\\lambda$, while maintaining the stability of conservative value estimation through the hyperparameter $\\alpha$. Unlike many prior approaches that rely on additional auxiliary networks-such as behavior policy estimators, importance-sampling correction models, or ensemble-based uncertainty estimators, CPQL performs this integration without additional components. This design allows the algorithm to propagate reward signals more effectively across trajectories, potentially capturing richer temporal dependencies without introducing additional instability.\n\n$\\textbf{(2)}$\n\nCPQL extends the mixture policy convergence property of Peng's $Q(\\lambda)$ to the offline reinforcement learning setting, providing a theoretical analysis of how multi-step learning interacts with conservative value estimation. The study examines both the role of the hyperparameter $\\alpha$, which controls the degree of conservatism, and the parameter $\\lambda$, which determines how strongly the mixture policy reflects the characteristics of the behavior policy. Specifically, while Theorem 2 proves that the policy learned by CPQL guarantees performance that is at least equal to or better than that of the behavior policy, Theorem 3 shows that, with an appropriately chosen $\\lambda$, the sub-optimality gap can be reduced by effectively balancing conservatism and multi-step backup. Overall, these theoretical results strengthen the justification for CPQL's design choices and suggest that it provides an interpretable and mathematically grounded framework for policy improvement.\n\n\n$\\textbf{(3)}$\n\nWhile the conservative Q-penalty in CQL plays an important role in suppressing overestimation of out-of-distribution actions, it often leads to an overly pessimistic value function, which can limit policy improvement. To mitigate this issue, CPQL introduces a multi-step temporal difference update, allowing reward signals to be propagated over longer horizons within offline trajectories. By leveraging trajectory-level information, the algorithm captures a more accurate relationship between actions and long-term returns, alleviating the problem of over-pessimism without compromising the stability provided by conservative regularization. Such balance between conservatism and optimism contributes to stable performance not only during purely offline training, but also in off-to-on fine-tuning, where the learned policy can adapt to online interaction without suffering from the sharp performance drop often observed in overly conservative methods.", "weaknesses": "$\\textbf{(1)}$ \n\nThe authors suggest that this work may represent an early attempt to extend Conservative Q-Learning into an N-step formulation, where multi-step temporal difference updates are incorporated within a conservative value estimation framework. However, the proposed method primarily integrates an existing algorithm, Peng's $Q(\\lambda)$, into the established CQL framework, and its core idea appears to be more of an adaptation or refinement of prior methodologies rather than a fundamentally new algorithmic contribution. Moreover, much of the theoretical basis related to PQL has already been extensively discussed in previous studies, and the theoretical analyses of CPQL closely follow the analytical structure developed for CQL. Consequently, while the paper provides an interesting perspective on reinterpreting conservative value learning through a multi-step lens, its methodological originality appears to be relatively limited compared to earlier works.\n\n$\\textbf{(2)}$ \n\nAccording to the authors, the interaction between $\\lambda$, which controls the degree to which the learned mixture policy follows the behavior policy, and $\\alpha$, which determines the level of conservatism, is designed to regulate the balance between stability and policy improvement. Theoretically, a higher-quality behavior policy is expected to benefit from a larger $\\lambda$, encouraging the learned policy to stay closer to the behavior policy, while a lower quality or inconsistent policy would require a smaller $\\lambda$ to prevent the accumulation of erroneous transitions. However, the experimental results in domains such as HalfCheetah do not appear to consistently reflect this theoretically suggested relationship. Moreover, although Figure 1 claims that the presence of $\\lambda$ makes $\\alpha$ less sensitive in CPQL compared to CQL, it is not entirely convincing from the presented results that such reduced sensitivity actually holds in practice.\n\n$\\textbf{(3)}$ \n\nWhile CPQL conceptually allows for richer reward propagation across trajectories, its empirical validation has so far been limited to relatively simple or short-horizon tasks within the D4RL benchmark. As a result, it remains uncertain whether the proposed method can generalize its effectiveness to more complex offline RL environments that involve longer temporal dependencies or higher behavioral variability. For instance, evaluations on more diverse and realistic benchmarks-such as non-goal conditioned settings in frameworks like OGBENCH: BENCHMARKING OFFLINE GOAL-CONDITIONED RL[1]-could provide further insight into whether CPQL consistently maintains a stable balance between conservatism and policy performance across a wider range of task dynamics. \n\n(reference)\n[1] Park, Seohong, et al. \"Ogbench: Benchmarking offline goal-conditioned rl.\" International Conference on Learning Representations (2025)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a6dIJ8YPU1", "reviewer": "Reviewer_qHcg", "summary": "The paper presents Conservative Peng’s $Q(\\lambda)$ (CPQL), a model-free offline multi-step reinforcement learning algorithm that utilizes the Peng’s $Q(\\lambda)$ operator for conservative value estimation. CPQL achieves superior performance on standard benchmarks.", "questions": "Please see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experimental evaluation is comprehensive.\n2. The algorithm demonstrates superior performance across the majority of tasks.", "weaknesses": "1. A primary concern is that the claim of \"multi-step\" does not intuitively address the over-optimistic value estimation challenge in offline reinforcement learning. If the multi-step approach were fundamentally effective, it raises the question of why conservative value estimation, which is widely used in offline RL, is still necessary.\n\n2. The authors assert that their method mitigates over-pessimistic Q-values, a challenge already addressed in other approaches such as MCQ. A comparison of the Q-value curves between CPQL and MCQ would help clarify the extent of this improvement.\n\n3. Can CPQL be viewed as a result of simply modifying the target value $y$ in CQL? Would it be possible to perform a similar substitution in MCQ to further demonstrate the effectiveness of the \"multi-step\" approach? What impact would this have on the results?\n\n4. While the authors conduct offline-to-online experiments, they do not compare their method against recent state-of-the-art algorithms, such as WSRL, which has demonstrated superior performance in the AntMaze domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)