Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning
Authors: Kwanyoung Park
Youngwoon Lee
Venue: ICLR 2025 Poster (2025)
Keywords: model-based offline reinforcement learning
offline reinforcement learning
lower expectile return
Primary area: reinforcement learning
TL;DR: Lower Expectile Q-learning (LEQ) improves model-based value estimation and policy optimization via lower expectile regression of $\lambda$-returns.

Abstract: Model-based offline reinforcement learning (RL) is a compelling approach that addresses the challenge of learning from limited, static data by generating imaginary trajectories using learned models. However, these approaches often struggle with inaccurate value estimation from model rollouts. In this paper, we introduce a novel model-based offline RL method, Lower Expectile Q-learning (LEQ), which provides a low-bias model-based value estimation via lower expectile regression of $\lambda$-returns. Our empirical results show that LEQ significantly outperforms previous model-based offline RL methods on long-horizon tasks, such as the D4RL AntMaze tasks, matching or surpassing the performance of model-free approaches and sequence modeling approaches. Furthermore, LEQ matches the performance of state-of-the-art model-based and model-free methods in dense-reward environments across both state-based tasks (NeoRL and D4RL) and pixel-based tasks (V-D4RL), showing that LEQ works robustly across diverse domains. Our ablation studies demonstrate that lower expectile regression, $\lambda$-returns, and critic training on offline data are all crucial for LEQ.

Published as a conference paper at ICLR 2025
MODEL-BASED OFFLINE REINFORCEMENT LEARNING WITH LOWER EXPECTILE Q-LEARNING
K Kwanyoung Park
Youngwoon Lee
Yonsei University https:/ httpss///wwanooosprr....gg Lo/E///
ABSTRACT
Model-based offline reinforcement learning (RL) is a compelling approach that addresses the challenge of learning from limited, static data by generating imaginary trajectories using learned models. However, these approaches often struggle with inaccurate value estimation from model rollouts. In this paper, we introduce novel model based offline RL method, Lower Expectile Q-learning (LEQ), which provides low-bias model-basee based value estimation via lower expectile regression of X-returns. Our empirical results show that LEQ significantly outperforms previous model-based offline RL methods long-horizon tasks, such the D4RL AntMaze tasks, matching or surpassing the performance of model-free approaches and sequence modeling approaches. Furthermore, LEQ matches the performance of state-of-
state-of-the-art art model-basee and model-free methods in reward environments across both state- based tasks (NeoRL and D4RL) and pixel-based tasks (V-D4RL), showing that LEQ works robustly across diverse domains. Our ablation studies demonstrate that lower expectile regression, X-returns, and critic training on offline data are all crucial for LEQ.
INTRODUCTION
One of the major challenges in offline reinforcement learning (RL) the overestimation of values for out-of-distribution actions due to the lack of environment interactions (Levine et al.) 2020 Kumar jet al., 2020). Model-based offline RL. addresses this issue by generating additional (imaginary) training data using learned model, thereby augmenting the given offline data with synthetic experiences that cover of-distribution states and actions (Yu et al. 2020 Kidambi et al.) 2020 eu 2021 Argenson and Dulac-Arnodd 2021 Sun al. (0033). While these approaches have demonstrated strong performance in simple, short-horizon tasks, they struggle with noisy model predictions and value estimates, particularly long-horizon tasks (Park et al. 2024). This challenge is evident in their poor performances (i.e. near zero) on the D4RL AntMaze tasks (Fu et al. 2020).
Typical model-based offline RL methods alleviate the inaccurate value estimation problem (mostly overestimation) by penalizing Q-values estimated from model rollouts with uncertainties in model predictions Yuu et al.) 20200 Kidambi et al. 2020) or value predictions (Sun et al., [023] Jeong et al.) 2023). While these penalization terms prevent policy from exploiting erroneous value estimates, the policy now does not maximize the true value, but maximizes the value penalized by heuristically estimated uncertainties, which can lead to sub-optimal behaviors. This is especially problematic in long-horizon, sparse-reward tasks, where Q-values are similar across nearby states (Park et al. 2024).
Another way to reduce bias in value estimates is using multi-step returns (Sutton 1988 Hessel et al. 2018). CBOP (Jeong et al., 2023) constructs an explicit distribution of multi-step Q-values from thousands of model rollouts and uses this value target for training the Q-function. However, CBOP is computationally expensive for estimating a target value and uses multi-step returns solely for Q-learning, not for policy optimization.
To tackle these issues in model- based offline RL, we introduce a simple yet effective model-based offline RL algorithm, Lower Expectile Q-learning (LEQ). As illustrated in Figure [ LEQ uses expectile regression with a small 7 for both policy and Q-function training, providing an efficient and elegant way achieve conservative Q-value estimates. Moreover, we propose optimize both

---

Published as a conference paper at ICLR 2025
Return distribution
Jj
R(F))
Jj
R R
Jk
R(TK)
R
Imagined trajectory
Lower expectile of returns TQ a) = [T[[(T)]
Possible trajectoriessTT,..
Figure Lower Expectile Q-learning (LEQ). (left) In model-based offline RL, an agent can generate imaginary trajectories using world model. (right) For conservative Q-evaluation of the policy, LEQ learns the lower expectile of the target Q-distribution from few sampled rollouts T, without estimating the entire Q-distribution with exhaustive rollouts.
policy and Q-function using A-returns (i.e. TD(A) targets) of long (10-step) model rollouts, allowing the policy to directly learn from low-bias multi-step returns (Schulman al. 2016).
The experiments the D4RL AntMaze, MuJoCo Gym (Fu et al. 2020), NeoRL (Qin et al. 2022). and V-D4RL (Lu et al. 2023) benchmarks show that LEQ improves model-basce offline RL across diverse domains. To the best of our knowledge, LEQ is the first model-based based offline RL algorithm capable outperforming model-free offline RL algorithms on the long-horize horizon AntMaze tasks (Fu et al. 2020; Jiang al., 2023). Moreover, LEQ matches the top scores across various benchmarks, while prior methods demonstrate superior performances only for a specific domain.
RELATED WORK
Offline RL (Levine et al. 2020) aims solve an RL problem only with pre-collected datasets,
outperforming behavioral cloning policies [Pomerleau] 1989). While it is possible to apply off-policy RI algorithms on fixed datasets, these algorithms suffer from the overestimation of Q-values for
actions unseen in the offline dataset (Fujimoto et al. 2019 Kumar et al. 2019 2020 since the overestimated values cannot be corrected through interactions with environments as in online RL.
Model-free offline RL algorithms have addressed this value overestimation problem on out-of- distribution actions by (1) regularizing policy to only output actions the offline data (Peng et al.) 2019 Kostrikov et al. 2022: Fujimoto and Gu 2021) (2) adopting conservative value estimation for executing actions different from the dataset (Kumar et al.] 2020 An et al., 2021). Despite their strong performances the standard offline RI. benchmarks, model-free offline RL policies tend to be constrained to the support of the data (i.c. state-action pairs in the offline dataset), which may lead limited generalization capability.
Model-based offfine RL approaches have tried to overcome this limitation by suggesting a better use of the limited offline data - learning world model and generating imaginary data with the learned actions. Similar to Dyna-style online model- based RL (Sutton 1991. Hafner et 2019 2021 2023). an offline model-based RL policy can be trained on both offline data and model rollouts. But, again, learned models may be inaccurate on states and actions outside the data support, making a policy easily exploit the learned models.
model that covers cot-of-distribuute
Recent model-based offline RL algorithms have adopted the conservatism idea from model-free offline RL, penalizing policies incurring (I) uncertain transition dynamics (Yu et al. 2020 Kidambi et al. 2020 Yu et al. 2 ) or (2) uncertain value estimation (Sun et al. 2023 Jeong et al. 2023).
This conservative use of model-generated data enables model-based offline RL to outperform model- free offline RL in widely used offline RL benchmarks (Sun al.] 2023). However, uncertainty
estimation is difficult and often inaccurate Yu et al. 2021). Instead of relying on such heuristic (Yu
et al. 2020 Kidambi et al. 2020 Sun al. 2023 expensive (Jeong et al. 2023) uncertainty
2

---

Published as a conference paper at ICLR 2025
Q-learning
Lower Expectile Q-learning
[o(0) (Qu(s - (T))²
T- 1--1(Q)T)= > Qa(s; a))l Lo(()
VoL, (s)) V. (T(
| 1(Q2(T) Qp(s,, ai)) (e).
Figure 2: Comparison of standard Q-learning and Lower Expectile Q-learning (LEQ). LEQ generalizes standard Q-learning (with X-returns Q; by multiplying a simple asymmetric weight "|7 (T) ( 4))) to the Q-learning objectives. T = (So, ao To, 81, A1. "i, ,8T) is a model -eoeerlgenerated trajectory and f I 0.5 is the expectile hyperparameter controlling the degree of conservatism. When 0.5, LEQ reduces to standard Q-learning.
estimation, we propose to learn conservative value function via expectile regression with small which is simple, efficient, yet effective, as illustrated in Figure 2
Expectile regression for offline RL has been first introduced by IQL (Kostrikov et al. 2022 which has been extended to model-based offline RL, such as IQL- IQL-TD-MPC (Chitnis et al. (2244). IQL uses "upper expectile" to approximate the "max operation" in V(8) = max, Q a) without querying out-of-distribution actions. On the other hand, our work fundamentally differs from IQL-like approaches in that our method uses "lower expectile" to get "conservative return estimates" " from trajectories generated by potentially inaccurate model rollouts.
3 PRELIMINARIES
Problem setup. We formulate our problem as a Markov Decision Process (MDP) defined as a tuple, .M (S, A. p. p.
(Sutton and Barto 2018). S and A denote the state and action spaces, respectively. r S x A + denotes the reward function. transition dynamics. p(so) E A(S) denotes the initial state distribution and y is discounting factor. The goal of RL is to find a policy, π S + A(A), that maximizes the expected return,
S x A + A(S denotes the
E 7r((st, ax) where T is a sequence of transitions with a finite horizon T, T (Sg. ag, To. S a ri..... ST), following π Ta, ss)) and p(s,+1 S,, a,) starting from So ~ p(-)-
In this paper, we consider the offline RL setup (Levine et al., 2020), where policy is trained with a fixed given offline dataset, Dem (Ti, T2, ..., JNJ. without any additional online interactions.
Model-based offline RL. As offline RL policy is trained from a fixed dataset, one of the major challenges in offline RL is the limited data support; thus, lack of generalization to -f-iitribitttinnnnnnn states and actions. Model-based based offline RL (Kidambi et al. 2020 Yu et al. 2020 2021 Rigter et al., 2022: Sun et al. [023] Jeong et al. 2023) tackles this problem by augmenting the training data with imaginary training data (i.e. model rollouts) generated from the learned transition dynamics and reward model, (S St, a.).
The typical process of model-based offline RL is follows: (1) pretrain model (or ensemble of models) and an initial policy from the offline data, (2) generate short imaginary rollouts (T) using the pretrained model and add them the training dataset Dmodel f- D model U {T), (3) perform an offline RL algorithm on the augmented dataset D Dmodel U Denv. and repeat (2) and (3).
Expectile regression. Expectile is generalization of the expectation of a distribution X. While the expectation of X, E[X] can be viewed as a minimizer of the least- objective, E [L2(y- a:] E [2 [y- z)²} r-expectile of X, E [x)), can be defined as a minimizer of the asymmetric least-square objective Ex-x ( x)], where (-) is defined as:
L5(u) T 1 (u > -u,
where | 1I > 0)] is an asymmetric weighting of least-squared objective with 0 1
We refer to r-expectile with 0.5 lower expectile of X. When < 0.5, the objective assigns a high weight 1 for smaller æ and a low weight for bigger Thus, minimizing the objective with r 0.5 leads conservative statistical estimate compared to the expectation.
A ( denotes the set of prohahility distributions over X.
3

---

Published as a conference paper at ICLR 2025
Algorithm LEQ: Lower Expectile Q-learning with X-returns
Input: Offline dataset Denv, expectile S 0.5, imagination length H, dataset expansion length R.
1: Initialize world models { :PoM ). policy no, and Q-function Qo
2: Pretrain (Pr)." "....... >P on Denv
b (mm(t)) = - (a. (ta.a.na))e C. log Pe (s', s.a)
3: Pretrain My and Qs on Denv
b using BC for mg and FQE Le et al. 2019 for Q
4: Dmodel + 0
5: while not converged do
6: // Expand dataset using model rollouts
7: So Denv
start dataset expansion from any state Dow
8:
for 1- O,..., R do
9:
Dmodel +- mmodel U {8t)
10:
a2 (S:)
11:
St+1: ~ P.(. St, ar), where Ps ~ {Po.. " Pont
D sample pv every step
12:
1/ Generate imaginary data, T = {SS0 ao, ro, "* -1,-1, 11,-1, 1M 811 8
13:
So Dmodal
start imaginary rollout from any state in Doodel
14:
for t = O,..., H do
a2 (S:))
16:
St+1, P (- ar), where Pe ~ {Po..'.. -Pon
D sample pe every step
17:
// Update critic using both model-generated data and offline data
18:
Update critic Qs to minimize C3 (Ø) in Eq. (7) using T and {s, a, 7, s'} ~ Denv
19:
// Update actor using only model-generated data
20:
Update actor Te to minimize L (0) in Eq. 11 using T
4
APPROACH
The primary challenge of model-based offline RL is inherent errors in a world model and critic outside the support of offline data. Conservative value estimation can effectively handle such (falsely optimistic) errors. In this paper, we introduce Lower Expectile Q-learming (LEQ), an efficient model- based offline RL method that achieves conservative value estimation via expectile regression of Q-values with lower expectiles when learning from model-genee -enerrtedde data (Section 4.1 Additionally,
we address the noisy value estimation problem (Park et al., 2024) using A-returns on 10-step imaginary rollouts (Section 4.2). Finally, we train deterministic policy conservatively by maximizing the lower expectile of A-retums (Section 4.3). The overview of LEQ is described in Algorithm
4.1 LOWER EXPECTILE Q-LEARNING
Most offline RL algorithms primarily focus on learning a conservative value function for out-of- distribution actions. In this paper, we propose Lower Expectile Q-learning (LEQ), which learns a conservative Q-function via expectile regression with small avoiding unreliable uncertainty estimation and exhaustive Q-value estimationn.
As illustrated in Figure I the target value for (s, a), where a t- (s)), can be estimated by rolling an ensemble of world models and averaging (s a') over all possible s':
Vmodel Besn (r [r Qu (s', mo mo(s)))
This target value has three error sources: the predicted future state and reward s', p s, and future Q-value @o (s)))) Thus, the target value from model-generated data, Dmnudel is more prone to overestimation than the original target Q-value, Denv. computed from (s. a, r,,'s ~ Dem:
New (
To mitigate the overestimation of Fmodel from inaccurate H-step model rollouts, we propose to use lower expectile regression on target Q-value estimates with small т. As illustrated in Figure 2 expectile regression with small learns Q-function predicting Q-values lower than the expectation, ie., conservative estimate of a target Q-value. Another ad advantage of using lower expectile regression that we do not have to exhaustively evaluate Q-values to get 7-expectiles as Jeong et al. 20233:
4

---

Published as a conference paper at ICLR 2025
instead, we can learn conservative Q-function with sampling:
H
LQ model (Ø) = LBso EDunta, Po
(T) - Dmodst)
(4)
H
1=
Additionally, we train the Q-function with the transitions in the dataset Deny. which do not have the risk of overestimation caused by the inaccurate model, using the standard Bellman update:
cav cav (6) ())
(Qo(s. a) Denv
(5)
To stabilize training of the Q-function, we adopt EMA regularization (Hafner et al. 2023 which prevents drastic change of Q-values by regularizing the difference between the predictions from Q and ones from its exponential moving average Q3:
LQEMA (Ø) = (a.a)ED... [(Q (s)
(6)
Finally, by combining the three aforementioned losses, define the critic loss follows: Lo(Ø) = _
model (0) + - Lm + LEMA
4.2 LOWER EXPECTILE Q-LEARNING WITH A-RETURN
To further improve LEQ, a Q-function and policy to learn from low-bias multi-step returns (Schulman et (016). Using
X-return instead of 1-step return for Q-learning. A-return allows
Ntstep returns GE++ (T) i= Tett +y (s( NttN), we define A-return of an H-step
trajectory T in timestep t, (T(T)
as
H-t (T)+:(T).
@ = 1
(8)
Then, can rewrite the Q-learning loss in Equation
with A-return targets:
H-1
ma (0) = Egg Dranae Trupus
To(s))
(9)
t=0
4.3 LOWER EXPECTILE POLICY LEARNING WITH X-RETURN
For policy optimization, we use a deterministic policy a = nal and update the policy using the deterministic policy gradients similar to DDPG (Lillierap al. 2016 To provide more accurate learning targets for the policy, instead of maximizing the immedhate Q-value, Qs, a), we maximize the lower expectile of X-return, analogous
conservative critic learning in Section
H
[o/(T)]
C (0)=
- Exg Dasdet
E
(10)
2
However, because of the expectile term, ET [QAL. we cannot directly compute the gradient of (+).
the relationship E7 [T B[i--10 [|1==1(=TQ\1>00
1Q11=0)111
To change the expectile expectation,
and optimize the unnormalized version, E [|7 (E [Q:] > QA) @A]. By approximating E
QQ,1
with the learned Q-estimator (St, az), we derive differentiable surrogate loss of Equation
10
H
(0) = -Es, D T T-poo
Σ T (QQ a.) (((T)) (((I)--Q((T)) 2N(T)
(11)
1=
Intuitively, this surrogate loss sets higher weight (1-T) on a conservative X-return estimates (i.e. Qo(s,, > Q7 (T),, encouraging policy to optimize for this conservative X-retur. On the other hand, an optimistic A-return estimates (i.e. (s, ) < (T) has a less impact to the policy with smaller weight (7). We provide proof in Appendix showing that the proposed surrogate loss provides a better approximation of Equation (10) than the immediate Q-value, Q ( at).
*Ou X-return slightly differs from (Sutton 1988 that puts high weight to the last N-step return, G.a(())
"IFO also works with stochastic policy; but, deterministic policy is sufficient for experiments.
5

---

Published as a conference paper at ICLR 2025
4.4 EXPANDING DATASET WITH MODEL ROLLOUTS
One of the problems of offline RL is that data distribution is limited to the offline dataset Denv- To tackle this problem, we expand the dataset using simulated trajectories, which we refer as Dmodel (u et al. 2021). To diversify the simulated trajectories in Dmodel. we use noisy exploration policy, which adds noise (0, esp)) . the current policy and generate a trajectory of length R.
EXPERIMENTS
In this paper, we propose a novel model-based offline RL method with simple, efficient, yet accurate conservative value estimation. Through our experiments, aim to answer the following questions: (1) Can LEQ solve diverse domains of problems, including both dense-reward tasks and long- long-horizon sparse-reward tasks? (2) Can LEQ be applied to pixel-based environments? (3) How individual components of LEQ affect the performance?
5.1 TASKS
To verify the strength of our low-bias model-based conservative value estimation in diverse domains, test LEQ four benchmarks: D4RL AntMaze, D4RL MuJoCo Gym (Fu et al., 2020), Ne- ORL. (Qin et al. 2022). and V-D4RI. (Lu et al. 2023). We first test on long-horizon AntMaze tasks: uma ze, medi um, l rge from D4RI,, and trm from Jiang al. al. (2023), shown in Figure
We also evaluate LEQ on locomotion tasks (Figure 4) state-based tasks from D4RL,, NeoRL and pixel-based tasks from V-D4RL. Please refer Appendix A for more experimental details.
5.2 COMPARED OFFLINE RL, ALGORITHMS
Model-free offline RL. We consider behavioral cloning (BC) (Pomerleau 1989
TD3+BC (Fuji-
moto and Gu 2021), which combines BC loss to TD3; COL (Kumar et al. 2020 which penalizes out-of-distribution actions; and IQL (Kostrikov al. 2022). which uses upper-expectile regression to estimate the value function. For locomotion tasks, we also compare with EDAC (An et al. 2021), which penalizes Q-values based on its uncertainty. For pixel-based tasks, we compare with DrQ+BC (Lu et al. 2023), which combines BC loss to DrQ-v2 (Yarats et al. [002); ACRO, which learns representations with multi-step inverse dynamics model.
Model-based offline RL. We consider MOPO Yu al. 2020) and MOBILE (unn et al. 2023). which penalize Q-values according to the transition uncertainty and the Bellman uncertainty of a world model, respectively; COMBO YYu et al. 202T), which combines CQL with MBPO; RAMBO (Rigter et al. 2022), which trains an adversarial world model against the policy; and CBOP (Jeong et al. 2023). which utilizes multi-step returns for critic updates; IQL-TD-MPC (Chitmis al. 2024), which extends TD-MPC (Hansen et al., (022) to offline setting with IQL. For pixel-based environments, we consider OfflineDV2 (Lu al. 2023). which penalizes Q-values according to the dynamics errors, and ROSMO (Liu et al. 2023). which uses one-step model rollouts for policy improvement. LEQ follows MOBILE for most implementation details but implemented in JAX (Bradbury et al., 2018). which makes it 6 times faster than the Py Torch versions of MOBILE and CBOP. Please refer to Table 4 for detailed comparison.
Sequence modeling for offfine RL. We consider TT (Janner et al. 2021), which trains Trans- former model [Vaswani 2017) to predict offline trajectories and applies beam search to find the best
(a) Umaze
(b) Medium
(c) Large
(d) Ultra
(a) Hopper
(b) Walker2d (c) HalfCheetah
Figure 3: AntMaze tasks.
Figure 4: Locomotion tasks.

---

Published as a conference paper at ICLR 2025
Average
umare
medium-play
large-play
ultraeplay
100
100
100
100
106
75
73
25
75
WORLE
CB02
RaMbo
10L
C.OP
COL
MOALE
COOP
50-
umaze-diversa
medium-diversa
lange-diverse
uttraddivrsa
100
100
100
100
75
73
75
25
CQL
MOBILE
CBOP
RAMBO
LEQ
50
MOBLE
CS09
RaMbo
g
Look
MOBLL
CoOP
rakroo
8
COL
MORE
CROP NUHDO
Model-free
Model-based
LEQ (Ours)
Figure 5: AntMaze results. Each graph displays the average success rate over 100 trials. LEQ
significantly outperforms prior model-based approaches, which achieve near-zero scores, and also
surpasses model-free baselines.
Average
hooper-L
hopper-H
walker2dd-
halfcheetahn
halfthectah-ii
120
120
120
120
120
120
90
10L
BI-C
MCULL
KOOD
KOBEL
L70
30
BOME
NORO MODEL
L20
0
60
hopper-M
halfcheetah
120
30
MOBILE
IQL
EDAC
MOPO
J0
HUnO
HORILE
J0
100
HOBILE
60
J0
8
30
Model-free
Model- based
LEQ (Ours)
Figure 6: NeoRL results. Each graph displays the average normalized score over 100 trials. LEQ
achieves better performance than prior methods, particularly in Hopper and Walker2d domains.
trajectory; and TAP (Jiang et al. 2023). which improves TT by quantizing the action space with VQ-VAE (Van Den Oord al. (017).
5.3 RESULTS ON LONG-HORIZON ANTMAZE TASKS
As shown in Figure 5 LEQ significantly outperforms the prior model-based approaches. For example, LEQ achieves 60.2 and 55.8 for large- diverse and ultra- www.raadiiveeee..e while the second best method, IQL-TD-MPC (Chitnis et al. 2024), scores only 4.0 and 3.6, respectively. We believe these performance gains come from our conservative value estimation, which works more stable than the uncertainty- based penalization of prior works. Moreover, LEQ even significantly outperforms the model-free approaches in umaze, large, and ultra mazes, and outperforms sequence modeling methods, TT and TAP, which serve as strong baselines for AntMaze tasks, in the most challenging ultra mazes, showing the advantage of utilizing low-bias multi-step return ang-horizon tasks.
Despite its superior performance, LEQ shows high variance in the performance on antmaze -medium um. We found that the medium mazes include states separated by walls that are very close to each other (denoted as red circles in Figure ?7). such that all of the learned world models falsely believe the
&
agent can pass through the walls. This incorrect prediction makes the agent to plan faster, but impossible trajectories as shown in Figure ? We believe that this could be addressed employing improved world models or increasing the number of ensembles for the world models.
Figure Failure in medium mazes. The agent plans impossible trajecto- ries on certain states (red circles).
7

---

Published as a conference paper at ICLR 2025
Average
hoppere
hopper-m
walker2ddr
wnlker2d-mr
hallcheelober
halicheetah-mr
120
120
120
90
90
90-
so.
60
30
MODELE
30
60
MODELL
Go
1000
POOBLE
60
hopger-m
hopper-me
walker2d-m
walker26-me
halfcheetah-m
halfchoetah-me
120
120
120
120
120
126
90
90
90
g 8
8
MOBILE
COL
101
EDAC
CBOP
LEQ
60
30
MOBLE
120
30
NOBILE
g
MOBILE
I.0
g
101
NOBILE
COOP
0
Model-Tree
Model-based
LEQ (Ours)
Figure 8: D4RL results. Each graph displays the average normalized score over 100 trials. LEQ
achieves comparable results with state-of-the-ar model-based and model-free offiin RL methods.
Average
walker walker
wakker walker_walk-ng walk-mr
cheetah rutht
cheetain_run-m mun-mrr
75
25
50-
MCRO
Iand) sc
cominnuapee ROSHO
25
50
50-
O O2
KOSHO
50
walker walier__alli-m walk-m
walker walk-me
cheetah Putwi
cheetah, runeme
100
100
100
OfflineDV2
25
DrQ+8C
ROSMO
CQL
ACRO
LFQ
25
50-
BIOTBC
AERO
3
M010 00
Model-free
Model-based
LEQ (Ours)
Figure 9: V-D4RL results. Each graph displays the average normalized score over 100 trials. LEQ
shows comparable results with state-of-the-art offline visual control methods.
5.4 RESULTS MuJoCo GYM LOCOMOTION TASKS
For the NeoRL benchmark in Figure 6 LEQ outperforms most of the prior works, especially in the Hopper and Walker2d domains. Furthermore, for D4RL MuJoCo Gym tasks in Figure g LEQ achieves comparable results with the best score of prior works in 7 out of 12 tasks, These results show that LEQ serves as a general offline RL algorithm, widely applicable to various domains.
Similar to ze-me um, LEQ ex- periences high variance in MuloCo tasks. During training, LEQ often achieves high performance, but then, suddenly falls back to 0, as shown in Figure TO This is mainly because the learned models sometimes fail capture failures (e.g. hopper and walker falling off) and predict optimistic future (e.g. hopper and walker walking forward).
Hopper-H (NeoRL)
walker2d (D4RL)
Resur
90
Return
60
30
%
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8 1.0
Iterations (x109
Iterations (x10*)
Figure 10: High variance during training. Our algo- rithm experiences oscillation due optimistic imagina- tions near the initial states.
5.5 RESULTS WITH VISUAL INPUTS
As shown in Figure LEQ combined with DreamerV3 (Hafner et al. 2023) performs on par with the state-of-the-art methods on V-D4RL datasets, demonstrating its scalability to visual observations. Notably, LEQ achieves the highest score on the walker malker_walkknee mosel--based methods, where Off and MOPO struggles. LEQ also outperforms model-free approaches on random datasets and walker mell--meeii um replay dataset, highlighting the strength of model-basee methods in datasets with diverse state-action distributions.
walk-medi um expert _m_expert dataset among
8

---

Published as a conference paper at ICLR 2025
Table 1: Impact of lower expectile Q-learning and X-returns on AntMaze. We ablate the effects of lower expectile and X-returns on the critic and policy updates in LEQ. The design choices from LEQ are colored in blue and other options are colored in red. The results are averaged over 5 seeds.
Design choicees
update
policy updane
play
modiiu
divesue
play
lczgc
play
Total
expectile
94.4
06.2 :00.3
espectile
H-stop
M-stop
98.0
01.7 +38.
40:3 4:2
57.0 4.3.
23.3 4.20
29033
Lower expectile
I-step
Qtw.
24.0
g390
55.8 gm6
20.8 gm4
31.2 g11.
49.3.g
g222
35.5
373.5
Lower expectile
A-returns
@tw.n) a)
81.0 2395
61.8 2194
00.
30.2 211.5
40.5 2117
42.8
47.5 450
410.3
Lower expectlle
returns
AWR
092 uns
44.4 4384
0.6 164
0.00 1s3
0.0 60.0
0.0 c02
0.0 103
0.01 100
114.2
MOBILE MOBILE
Aereturas
Avreourns Q1a.n)
84.3 425
40.8 4384
51.3 vic
29.7 412
28.3 ks1.
18.7 4.00
88.0 -20-
23.3 c4a
188.9
I-stop
28.5
67.0
54.0 gm
23.5 g31
38.0
g015
g0.5
35000
5.6 ABLATION STUDIES
To deeply understand how LEQ (LEQ-X) work, we conduct ablation studies in AntMaze environ- ments and answer the following five questions: (1) Is lower expectile Q-learning better than prior umcertainty-basee penalization methods? (2) Does lower expectile policy learning better than existing policy learning methods? (3) Does X-return help? (4) Which factor enables LEQ to work in AntMaze? and (5) How do imagination length H and data expansion affect the performance?
(1) Lower expectile Q-learning. We compare our lower expectile Q-learning with the conservative value estimator in MOBILE (Sun et al. 2023), which penalizes Q-values based on the standard deviation of Q-ensemble networks. In T replacing lower expectile Q-learning with MOBILE decreases the success rate, both with X-retums (461.8 vs 338.9) and without them (373.5 vs 339.0).
(2) Lower expectile policy learning. We also compare our lower expectile policy learning with AWR (Peng et al. (019)) and directly maximizing Q(8, a). As shown in Table [ LEQ shows better performance compared to AWR (461.8 vs 114.2) and maximizing the Q-values (461.8 vs 410.3).
(3) X-returns. The first three rows of Table I show the effect of X-returss in LEQ. Substituting A- returns with H-step returns (461.8 vs 290.3) 1-step returns (461.8 vs 373.5) significantly decreases the performance. Moreover, while LEQ shows better performance than MOBILE without X-returns (373.5 vs 339.0), the performance of LEQ gets significantly better with X-returns, compared to MOBILE with X-returns (461.8 vs 338.9).
(4) What makes model-based offline RL work in AntMaze? LEQ shows outstanding perfor- mance compared to previous offline model-based RL methods, especially in large and ultra mazes. To understand which aspects of LEQ enabled this success, applied its changes MOBILE and analyzed the impact. The results are detailed in Table
We first re-implement MOBILE with some technical tricks used in LEQ (denoted as MOBILE") LayerNorm (Ba et al., 2016), SymLog (Hafner et al. 2023). single Q-network, and no target Q-value clipping. However, MOBILE" still achieves barely non-zero score, 14.0.
Table 2: Impact of hyperparameters in MOBILE on AntMaze. MOBILE" uses the hyperpa- rameters from MOBILE: 3 = 0.95. y 0.99, and R = 5, whereas LEQ uses B = 0.25, 0.997, and It = 10. The results show that 3 is the most critical hyperparameter that makes MOBILE work in AntMaze.
Hyperparams. 7
medi diverse
large
ultra diverse
Total
uma Z.
diverse
play
play
diverse
play
0.997
120.a
22.5 323.a
54.0 asa
49.5 ne.
28.3 ag.0
28.0 211.4
25.5 aG.0
23.8 1:1.a
0.25
0.907
5
74.0
3.7 194
64.7 207:
28.0 1eo
18.7 1:50
8.0 20.3
9.7 14.5
9.0 2a
205.7
0.25
0.90
39.7 425.
30.3 amaa
38.0 215.0
0.0 200
3.7 2b.2
25.0
5.0 20.0
125.7
0.25
0.99
5
77.0 26.1
20.4 g15.7
64.6 2m13
31.6 216.
2.6 2aa
7.2 28.
4.6 13.0
213.0
0.95
0.997
0.0 20.0
20.0
1.8 =se
0.5 20.9
0.2 204
2.2 22.3
#1.7
0.0 20.0
0.95
0.997
0.0 so.0
0.0
7.2
1.6 #2.1
9.6 #T.1
5.1 =4.9
0.0 +c.0
1.8
25.6
0.95
0.09
0.0 so.0
0.0 30.0
5.0 us.1
0.6 s:.2
7.4 8148
1.6 na.2
0.0 s5.0
0.0 a6.0
14.6
0.95
0.99
1.0 =2.0
0.0 20.0
6.1
5.0 +5.0
0.8 #14
0.8 =1.2
0.0 +.0
0.0
11.0
9

---

Published as a conference paper at ICLR 2025
Table LEQ with different imagination length and data expansion. A longer can mitigate critic biases, while increasing model errors, which leads to poor performance. Each number is averaged over 5 random seeds.
Dataset
H 10 (ours)
H = 5
H = 15
w/o datasct expansion
antmaze-umazz
94.4
95.2
98.6
97.4 11.4
verse
71.0 1123
67.2 1m.1
70.7 115.
63.0 123.2
antmaze-mediume
58.8 +33.0
46.4 +31.9
76.3 +17.2
58.2 +25.0
aatmazee-eddium-diieeee
46 +23.2
18.6 #28.7
30.3 #40.1
28.6 #33.7
aztmaze-lazge-ppay
58.6 =0.1
48.6 +15.4
62.0 +9.9
56.0 #9.5
entmare-1arre-ddieerr
60.2 115.3
35.2 1..7
33.0 23.2
57.0 14.5
antmaze catmaaze-ultzapplay
25.8 1:::2
54.2 1:0.8
0.0 10.0
39.2 1:3.1
antmaze-ultra-ddvvers.
55.8
39.4 16.11
0.0 10.0
36.0
Total
470.4
404.8
371.0
435.4
We found that reducing the ratio 3 between the losses from imaginary rollouts and dataset transitions is key to make MOBILE work (i.e. achieving meaningful performances in umaze and medi um mazes, with a total score of 213.0). This adjustment also allows for higher discount rate and longer imagination horizon, yielding the best results for MOBILE" We suggest that utilizing the true transition from the dataset important in long-horizon tasks, which has been undervalued in prior works. We provide additional extensive ablation results on LEQ in Appendix E
(5) Imagination length H and dataset expansion. As shown in Table 3 the performance increases when it goes to H 10 from H 5, but drops when H = 15. This result shows the trade-off of using the world model - the further the agent imagines, the more the agent becomes robust to the error of the critic, but the more it becomes prone to the error from the model prediction.
We also evaluate LEQ without the dataset expansion. In AntMaze, the results with and without the dataset expansion are similar, as shown in Table 3 On the other hand, the dataset expansion makes the policy more stable and better in the D4RL MuJoCo tasks (in Appendix, Table 20
6 LIMITATIONS
Following prior work on model-based offline RL (Sun et al. 2023 Jeong et al. 2023), we assume access to the ground-truth termination function of task, different from online model-based RL approaches, which learn termination function from interactions. As shown in Table [2] Using learned termination function instead of the ground-truth termination function results in signif- icant performance drop (461.8 8 >> 232.6), particularly in diverse datasets (233.2 -> 66.8), where termination signals are limited because the dataset is collected by navigating to randomly selected goals. While relying on the ground-truth termination function simplifies the problem, it limits the applicability of the method to scenarios where this information is readily available. Extending the proposed approach to learn terminal signals from the dataset would be an immediate next step.
CONCLUSION
In this paper, we propose novel offline model-based reinforcement learning method, LEQ, which uses expectile regression to get a conservative evaluation of a policy from model-generated trajectories. Expectile regression cases the pain of constructing the whole distribution of Q-targets and allows for learning a conservative Q-function via sampling. Combined with X-returns in both critic and policy updates for the imaginary rollouts, the policy can receive learning signals that are more robust to both model errors and critic errors. We empirically show that LEQ robustly improves the performance of model-based approaches in various domains, including state-based locomotion, long-horizon navigation, and visual control tasks.
ACKNOWLEDGMENTS
We would like to thank Junik Bae for helpful discussion. This work was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant
10

---

Published as a conference paper at ICLR 2025
PRS-2020-1120136.1 Artificial Intelligence Graduate School Program (Yonsei University)) and the National Research Foundation of Korea (NRF) grant (RS- 005-2222-00333633 funded by the Korean Govemment (MSIT). Kwanyoung Park was supported by the Electronics and Telecommunications Research Institute (ETRI) grant (24ZR1 100) and IITP grant (RS- R0S-2024-00599279 Global Al Frontier Lab) funded by the Korea government (MSIT). This research supported by Korea Basic Science (Nationte research Facilities and Equipment Center) grant funded by the ministry of Science and ICT RS-2024-00403860)
ETHICS STATEMENT
Our method aims to increase the ability of autonomous agents, such robots and self-driving cars, to learn from static, offline data without interacting with the world. This enables autonomous agents to utilize data with diverse qualities (not necessarily from experts). We believe that this paper does not have any immediate negative ethical concerns.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide the full code of LEQ in the supplementary materials, along with instructions to replicate the experiments presented in the paper. We provide the experimental details in Appendix and the proof on the derivation of our surrogate policy objective in Appendix B
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Arthur Argenson and Gabriel Dulac-Arold. Model-based offline planning. In International Confer- on Learning Representattinss 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1601vv6607.0444 2016.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1577-1594. PMLR, 2023.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http:: /gthhbb.ttttbtgggg comm/jax-mmmp11
Rohan Chitnis, Yingchen Xu, Bobak Hashemi, Lucas Lehnert, Urun Dogan, Zheqing Zhu, and Olivier
Delalleau. Iql-td-mpe: Implicit q-learning for hierarchical model predictive control. In IEEE International Conference on Robotics and Automation, pages 9154-9160. IEEE, 2024.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.000 000000 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/9 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. In Neural Information Processing Systems, volume 34, pages 20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pages 1861-1870. PMLR, 2018.
11

---

Published as a conference paper at ICLR 2025
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04 100444 2023.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In International Conference on Machine Learning, 2022.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Association for the Advancement of Artificial Intelligence, 2018.
Riashat Islam, Manan Tomar, Alex Lamb, Yonathan Efroni, Hongyu Zang, Aniket Didolkar, Dipendra Misra, Xin Li, Harm Van Seijen, Remi Tachet Des Combes, al. Principled offline in the presence of rich exogenous information. In International Conference on Machine Learning, 2023.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Neural Information Processing Systems, volume 32, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Neural Information Processing Systems, volume 34, pages 1273-1286, 2021.
Adam Jelley, Trevor Mclnroe, Sam Devlin, and Amos Storkey. Efficient offline reinforcement learning: The critic is critical. arXiv preprint arXiv:2406.13376, 2024.
Jihwan Jeong, Xiaoyu Wang, Michael Gimelfarb, Hyunwoo Kim, Baher Abdulhai, and Scott San- ner. Conservative bayesian model-based value expansion for offline policy optimization. In International Conference Learning Representations, 2023.
Zhengyao Jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefenstette, and Yuandong Tian. Efficient planning in compact latent action space. In International Conference on Learning Representations, 2023.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In Neural Information Processing Systems, volume 33, pages 21810-21823, 2020.
llya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning In International Conference on Learning Representations, 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Neural Information Processing Systems, volume 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative g-larnnin for offline reinforcement learning. In Neural Informution Processing Systems, volume 33, pages 1179-1191, 2020.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Interna- tional Conference on Machine Learning, pages 3703-3712. PMLR, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint (XXiv:2005.076433 2020.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference Learning Representations, 2016.
12

---

Published as a conference paper at ICLR 2025
Zichen Liu, Siyi Li, Wee Sun Lee, YAN Shuicheng, and Zhongwen Xu. Efficient offline policy optimization with a learned model. In International Conference on Learning Representations. 2023.
Cong Lu, Philip Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. Transactions on Machine Learning Research, 2023.
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: rl with latent states as actions. In Neural Information Processing Systems, volume 36, 2024.
Offline goal-conditioned
Xue Bin Peng, Aviral Kumar, Grace Zhang. and Sergey Levine. Advantage-weiggtte regressi regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arxiv: 2019.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Advances in Neural Information Processing Systems, pages 305-313, 1989.
Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. NeoRL: A near real-world benchmark for offline reinforcement learning. In Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL
net nett/oormmicc NNIiddijNLLLaa jNdLszxdt ra
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. In Neural Information Processing Systems, volume 35, pages 16082-16097, 2022.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016.
Yihao Sun. Offlinerl-kit: An elegant pytorch offline reinforcement learning library. httpthubb.cmmmyhhhaausaaaa/////a
https
2023.
Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman inconsistency for model-based offline reinforcement learning, In International Conference on Machine Learning, pages 33177-33194, PMLR, 2023.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 44, 1988.
Richard S Sutton. Dyna, integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 164):160 1991.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
Vaswani. Attention all you need. Neural Information Processing Systems, 2017.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control:
Improved data-augmented reinforcement learning. In International Conference Learning Representations, 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Neural Information Processing Systems, volume 33, pages 14142, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model- policy optimization. Neurul Information Processing Systems, volume 34, pages 09954728967, 2021.
13

---

Published as a conference paper at ICLR 2025
A TRAINING DETAILS
Computing resources. All experiments are done on a single RTX 4090 GPU and 8 AMD EPYC 9354 CPU cores, supported by Advanced Database System For state-based environments, we use 5 different random seeds for each experiment and report the mean and standard deviation, while each offline RL experiment takes hours for ours, 12 hours for MOBILE, and 24 hours for CBOP, For pixel-based environments, we use 3 different random seeds and each experiment takes 2 hours for ours and 8 hours for ROSMO.
Environment details. For state-based locomotion tasks, we use the datasets provided by D4RL Fu et al., 2020) and NeoRL (Qin et al. 2022). Following IQL (Kostrikov et al., 2022), we normalize rewards using the maximum and minimum returns of all trajectories. We use the true termination func- tions of the environments, implemented in MOBILE (Sun et al. 2023). For pixel-based environments, we do not normalize the rewards.
For AntMaze tasks, we use the datasets provided by D4RL (Fu et al. 2020). Following IQL. (Kostrikoy et al. 2022), we subtract 1 from the rewards in the datasets that the agent receives -1 for each step and 0 on termination. We use the true termination functions of the environments. The termination functions of the AntMaze tasks are not deterministic because a goal of maze is randomized every time the environment is reset. Nevertheless, we follow the implementation of CBOP (Jeong et al., 2023), where the termination region set to a circle around the mean of the goal distribution with the radius 0.5.
Implementation details of compared methods. For all compared methods, we use the results from their corresponding papers when available. For IQL (Kostrikov al. 2022), we run the official implementation with 5 seeds to reproduce the results for the random datasets in D4RL and NeoRL. For the AntMaze tasks, we the official implementation of MOBILE and CBOP with 5 random seeds. Please note that the original MOBILE implementation does not use the true termination function, so we replace with our termination function. For MOPO, COMBO, and RAMBO, we use the results reported in RAMBO (Rigter et al. (002).. For DMControl tasks, we replace the categorical distribution of the policy with gaussian distribution of the official ROSMO codebase, and run the experiments with sampling hyperparameter N =
World models. For based environments, we use the architecture and training script from OfflineRL-Kit (Sun) 2023), matching the implementation of MOBILE (Sun et al. (202)). Each world model is implemented as a 4-layer MLPs with the hidden layer size of 200. We construct an ensemble of world models by selecting 5 out of 7 models with the best validation scores. We pretrain the ensemble of world models for each of 5 random seeds (i.e. training in total 35 world models and using 25 models), which takes approximately hours in average. For pixel-based environments, we use the 12M model of Dreamer (Hafner et al. 2023) and pretrain the world model using its loss function. We follow the implementation of Of2 (Lu et al., 2023), training ensemble of world models for stochastic latent prediction, which takes 4 hours in average. We select one model for each step, following the design choice of MOBILE, as we found that randomly choosing a model every step can make imaginary rollouts more robust to model biases, leading to better performance.
Policy and critic networks. For state-based environments, we use 3-layer MLPs with size of 256 both for the policy network and the critic network. We use layer normalization (Ba et al. 2016) to prevent catastrophic over/underestimaatio (Ball et al. (023), and squash the state inputs using symlog to keep training stable from outliers in long-horizon model rollouts (Hafner al. 2023). For pixel-based environments, we use the architecture of the 12M model in DreamerV3 models.
Pretraining policy and critic networks. For some environments, we found that a randomly initialized policy can lead to abnormal rewards or transition prediction from the world models in the early stage, leading to unstable training (Jelley et al., 2024).. Following CBOP (Jenng et al.) 2023), we pretrain a policy no and a critic Qo using behavioral cloning and FQE (Le al. (201)). respectively for state-based experiments. We use a slightly different implementation of FQE from the original implementation, where the arg min operation is approximated with mini-batch gradient descent, similar to standard Q-learning as shown in Algorithm 2
14

---

Published as a conference paper at ICLR 2025
Algorithm 2 FQE: Fitted Q Evaluation (Le et al. 2019)
Input: Offline dataset Den., policy To
1: Randomly initialize Q-function Qe
2: while not converged do
3: (84, a, res s s1);==
Dan
4: = sg( + Q (s())))
sg(() is stop stop- gradient operator
5: (FQ N Σ (Qp a;) -/)²²
6: Update Qø using gradient descent to minimize (p)
Comparisons with prior methods. We provide a comparison of LEQ with the prior model-based approaches and the baseline methods used in our ablation studies in Table
Table 4: Comparisons with the prior model-based methods and the baseline method. The hyperpa- rameters same with LEQ are colored in blue; others are colored red.
Components
CBOP
MOBILE
MOBILE
LEQ (ours)
Training scheme
MVE
Feinberg
MBPO Janner
MBPO
Janner
Dreamer (Hafer
et al. 2018)
atal.
2019
2019
al. 2023
Conservatism
Lower-
Lower-
Lower-
Lower expectile
confidence bound
confidence bound Stochastic
confidence bound Stochastic
Policy
Stochastic
Deterministic
Policy objective
(s.
(s.,a)
(s,, a)
A-returns
Policy pretraining
BC
BC
# of critics
20-50
2
1
1
Critic objective
Multi-step
One-step
One-step
X-returns
(adaptive
weighting)
Critic pretraining
FQE
Le al.
FQE
Le
2019
2019
Horizon length (H)
10
1
T0
Expansion length (ft)
or
10
5
Discount rate (7)
0.99
0.99
0.997
0.997
/ in Equation
1.0
0.95
0.25
0.25
Impl. tricks
Clip Q-values
LayerNorm Symlog
LayerNorm
with 0
Symlog
Running time
24h
12h
40m
2h
Hyperparameters of LEQ. of our method in [5 We note that we use the same hyperparameters across all state-based tasks, except We search the value of in 10.1, 0.3, 0.4, 0.5) and report the best value for the main experimental results. In addition, we report the exhaustive results Tables 15 and T6 and summarize used in the main results in Table
For state-based experiments, we report task agnostic hyperparameters
For pixel-based experiments, we decrease the horizon length of DreamerV3 from 15 since we do not observe performance improvement with the longer imagination horizon (Appendix E.. consistent with the finding from Lu et al. (2023). Moreover, we remove the entropy bonus, as exploration not required in offline RL. We search the value of in 1. 0.3, 0.4, 0.5) and report the best value in the main experimental results. All other hyperparameters follow the default settings of Dreamer V3. We also report the exhaustive results in Table 17 and summarize in Table
15

---

Published as a conference paper at ICLR 2025
Table 5: Shared hyperparameters of LEQ in state-based experiments.
Hyperparameters
Value
Description
Vactor
3c-5
Learning rate of actor
Traitic
le-4
Learning rate of critic
Optimizer
Adam
Optimizer
Texpand
5000
Interval of expanding dataset
N Nexpand
50000
Number of data for each expansion of dataset
R
5
Rollout length for datasct expansion
Oesp
1.0
Exploration noise dataset expansion
Niser
1M
Total number of gradient steps.
Ban
256
Batch size from original dataset
Bmode
256
Batch size from expanded dataset
0.997
Discount factor
0.95
value X-return
H
10
Imagination length
1
Weight for critic EMA regularization
CEMA
0.995
Critic EMA decay
Table 6: Task-specific hyperparameter r of LEQ in state-based experiments.
Domain
Task
AntMaze
umaze
0.1
umaze-diverse
0.1
medium-play
0.3
medium-diverse
0.1
large-play
0.3
large-diverse
0.3
uitra-play
0.1
ultra-diverse
0.1
MuJaCo
hopper-r
0.1
hopper-m
0.1
hopper-mr
0.3
hopper-me
walker2d-r
0.1
walker2d-m
0.3
walker2d-mr
0.5
walker2d-me
0.1
halfchectah-r
0.3
halfcheetah-m
0.3
halfcheetah-m-
0.4
haheeatta
0.1
NeoRL
Hopper-L
0.1
Hopper-M
0.1
Hopper-H
0.1
Walker2d-l.
0.3
Walker2d-M
0.1
Walker2d-H
0.1
L
HalfChoctah-M
0.3
HalfChectah-##
0.3
Task-specific hyperparameters of the compared methods. We report the best hyperparameters of MOBILE' for the AntMaze tasks in Tables 7 and For MOBILE and MOBILE* we search
the value of within {0.1, 0.5, 1.0, 1.5), as suggested MOBILE (Sun et al. 2023). where c the coefficient of the penalized bellman operator:
TÔ(s. a) r(s. +nQ(s' - Std(Q(s',a))
(12)
16

---

Published as a conference paper at ICLR 2025
For CBOP, conduct hyperparameter search for in {0.5, 2.0, 3.0, suggested in the original paper, where Y is an LCB coefficient of CBOP. We do not report the best hyperparameter for MOBILE and CBOP because both methods score zero points for all hyperparameters in AntMaze.
For MOPO V-D4RL experiments, we search for in {3, 10), as suggested in Lu et al 2023), where À is the penalization coefficient in MOPO. Then, report the best value in 'lable 10 For ROSMO, we use the hyperparameter specified in the official code.
Table 7: Task Task-specific hyperparameters in MOBILE*.
Table 8: Task-specifie hyperparameters in MOBILE" with A-returns.
Domain
Task
Domain
Task
AntMaze
umaze
1.0
AntMaze
umaze
1.0
umaze-diverse
1.0
umaze-diverse
0.5
medium-play
1.0
medium-play
0.1
medium-diverse
0.1
medium-diverse
0.1
large-play
0.1
large-play
0.1
large-diverse
0.1
0.1
ultra-play
1.0
ultra -atry
1.0
ultra-diverse
1.0
uitra-diverse
0.5
Table 9: Task-specific hyperparameter of LEQ in V-D4RI experiment.
Domain
Task
T
walker_wal walk
random
0.5
medium
0.3
mediun_roplay
0.3
mediun_cxpert
0.1
cheetah_run
random
0.3
nedium
0.1
meditum_repla
0.1
medi un_experettt
0.5
Table 10: Task-specific hyperparameter of MOPO V-D4RL experiments
Domain
Task
À
Lilker_wall
rangom
3.0
medium
3.0
medium_replas
replay
3.0
medi reepert
3.0
cheetah_run
random medium um
3.0
3.0
medi @edium_rree medium_replay
10.0
medium medium_expers expert
10.0
17

---

Published as a conference paper at ICLR 2025
B PROOF OF THE POLICY OBJECTIVE
We show that the surrogate loss in Equation leads to a better approximation for the expectile of X-returns in Equation (10) than maximizing (s a). other words, we show that optimizing the following policy objective:
(x) TTp (W'' (Qp a:) > (T)))
(13)
leads to optimizing lower-bias estimator of By -e 19 (7) than Qs(s;, ac).
To show this, we first prove that Vnew 3
closer 10
(Q)
than (s, a.) for most of the situations. For deriving the proof, we generalize the problem by considering an arbitrary distribution X and its estimate Y, which correspond to X (T)
Qo a). Then, we show Yoow E[W' (F>X))X] is closer Y than Y. We split the case to two E|WT(>>>]]
cases where Y 2 Y (Lemma and Y < Y (Lemma 2 and show the effectiveness of Ynew instead of Y for each case.
Lemma 1. Let X be distribution and Y = [X be a lower expectile of X (i.e. 0 T S 0.5). Let Y be an arbitrary optimistic estimate of Y (i.e., Y 2 Y), and define W (-) = |T 1 (.)) If we
let Ynow B[W' (V>> X)-X] be a new estimate of Y. then (Yew Y S I - -Y
Proof.
-Yl
E(WT(Ý > X
E[ > X) .X]
C: Def. of Y and Y)
F.W" (V > X)]
EY > X)]
E[WT(Y > X) - + E[(1 1 -
FW X X)
Def of (T
T(W X)] + E[(1 27) x1)
EW > X)]
E[ > x))E[(1 -
X) 2,
22 S X Xx555 81)
B[ > - 1(y < x X)))
BW XY
B[W > -
(1 - (T)
2r) 10 X x
EJI(Y < X < -X
(1 - 2r)
B[TT(Y X)) + -(x
p(Y s X S () -Y)
= =(1 -11-21 2r)
BW X)) + B11 1(V x
p(v xV)) -Y) s (1- (7) 5 x1 -2)).
= (1 (T)
sxsi
MV X x (F X)-
= -1-2rr)
-1 2T) 2 2X)
MY X (V5x) ( X] -Y)
(1-
T+ (1-27) p(x <
PV X X
(F -Y-
$ 8Bysxsp\A|
-
18

---

Published as a conference paper at ICLR 2025
Lemma 1 shows that when the estimated value is optimistic (Y 2 Y), the bias of the new estimate is always smaller than that of the original estimate. In the context of LEQ algorithm, the lemma tells that if the critic network ( overestimates the lower-expectile of the target returns (Y = E (X), the surrogate loss (Ynw)) compensates the overestimation of the critic values.
Unfortunately, when the estimated value is pessimistic (Y S Y), there are some exceptional cases that the surrogate loss overeompensates the underestimation, resulting in an even larger error. The boundary for these cases is characterized in Lemma 2
Lemma Let X be a distribution and Y = F² [X] be a lower expectile of X (i.e. < < S 0.5).
Let Y be an arbitrary conservative estimate Yf (i.e. Y I Y), and define WT(-) = |T - 1()]-
If we let Ynew x be new estimate of Y, then 1 Y S IY -Y,, when
p(X S M 2 10(x SY) 1-2
Proof.
YI
E(W"(P > X)
(Y > > .
(: Def. of Peow and Y)
F/Wa X)]
(Y
EI > x)- -[(1 27) -(x
-X]
(: Def. of WT()
BIW' > -))
EM > X)]
BY > -
X) - X 27) 1(8 sX
S X Y)))
> ",W00--AAUUUXXXXXXXX0000-03--0"" s <X xY)
(1 27)
108 -
=11- 2r)
BI > >X -00 YY]
p(6
-1 2r)
BI > ([(1- 27) 108 S < {Y)
x(² sX Ba
(1 2r)
(1-r) (1- p(Yss) X) (1 - 22)) p(Y s Y
x - Eygxgr( (X)
(1 27)
-1-7) - S X)
M( S X Eygxgr (X)
-1 (r)
++ +1-1))) 2 8
m S x Y - Rxsx
1-2 x1
p(X x(X s - B Bygxgrix)
1 + 5
SY Eygxer|X] [X]
SY v
By combining Lemma
I Lemma 2
we get Theorem as below:
19

---

Published as a conference paper at ICLR 2025
0.4
0.4
041
02
0.3
0.3
0.0
0.2
02-
-00
0.1
0.1
-04
0.0
0.0
tsu
(a) Normal distribution
(b) Uniform distribution
Figure 11: Region of exceptions for the surrogate loss in realistic distributions. We visualize cases where the surrogate loss worse than the direct policy optimization, for two common distributions for X: (a) a normal distribution (00 1) and (b) uniform distribution ((0, 1). We sample and 7 from arange (0, 0.5, 0.0025) and compute Y E [X]. Ý E [x[] The plots show the difference of errors Ynew Y| L Y|: positive values (red yellow) indicate that the surrogate loss performs worse than the estimate, while negative values (green to blue) indicate that the surrogate loss outperforms the estimate. There is no exception for normal distribution, and only 39 out of 40000 cases exceptional for uniform distribution.
Theorem 1. Let X be a distribution and Y = [X] be a lower expectile of X (i.e. 0 < 0.5). Let Y be an arbitrary estimate of Y, and define (- = - 1(-)l. If we let Ynew BW (
be new estimate YfY, Y, then YnEw 5 - -Y| if the following condition holds:
p(X $ r (p(x SV) 7
(14)
2T
Proof. We combine the two cases dealt in Lemmal
Lemma
Here, Y 2 Y from Lemma can be omitted, since the condition Equation
includes the case 4 2 Y.
if Y 2 Y, then p( Y) p(X Y) 1 (x(X Y) 1-22))
The condition of Equation 14 is conservative bound applicable to any distribution, and 1Y Y
I - Y| holds across much broader regions in general. For example, Figure 11 shows that the inequality holds for 100%, 99.9% of the cases for normal, uniform distributions, respectively.
Here, illustrate how optimize E[WT (Qo au) Qi(T) Q7(T) instead of TWWTQu((s..a)))0000000007 The normalizing factor EIWT(Q, a.) > Q((T)) is non- (UN
differentiable with T and the gradient is 0 everywhere (except (s = (T))). Thus, if we calculate the gradient of y new. the gradient for the normalizing factor disappears. Therefore, omit the normalizing factor and get equivalent formula E[W" (Q, ai) > Q2()) (((T) for gradient- based optimization.
20

---

Published as a conference paper at ICLR 2025
C COMPLETE NUMERICAL RESULTS
For completeness, we provide the full tabular results corresponding to the graphical summaries presented in Figures 5 a Some low-performing baselines were omitted from the graphical summaries for clarity but are included in the table below for reference.
Table 11: AntMaze results. Each number represents the average success rate on 100 trials over different seeds. The results for LEQ, MOBILE, and CBOP are averaged over seeds. The results for other methods are reported following their respective papers.
Model-free
Seq. modellng
Model-hased
Dataset
CQL
IQL
TAP
COMBO
RAMBO
MOBILE'
CBOP
IQL-ID-MPC
LEQ (ours)
74.0
87.5
100.0
81.5
80.3
25.0
52.0
94.4
umaze umaze-dil divieeee
84.0
62:2
21.5
68.5
37.3
0.0 s6.0
30.0
72.6
71.0 #12.3
medi um play
61.2
71.2
93.3
78.0
0.0
16.1
0.0
0.0
88.5
58.8
26.0
large-play
15.8
39.6
66.7
71.0
01.0
0.0
0.0 s6.0
010 30.0
86.6
38.6 :9.1
lacge-de sesee-dvven
60
ultra-play
20.0
22.0
0.0 g6.0
0.0 200
20.6
25.8 116
ulize ulize-diveese divesse
15.6
33.3
26.0
0.0
0.0.20
3.6
55.8
Total w/o ultra Total
303.6
354.1
441.5
469. 0
137.6
67.0
0.0
0.0
324.3
388.8
378.0
494.8
517.0
0.0
348.3
470.4
We the official implementation of MOBILE and CBOP.
Table 12: NeoRL results. LEQ and IQL results are averaged over 5 seeds. The results for prior works are reported following Sun al. (2023) and Qin al. (2022). MOPO an improved version of MOPO presented in Sun et al. (2023). We highlight the results better than 95% of the best score.
Model-free
Model-based
Dataset
BC
TD3+BC
CQL
EDAC
IQL
MOPO
MOBILE
LEQ (ours)
Hopper-I
15.1
15.8
16.0
18.3
16.7
6.2
17.4
24.2 #2.a
Hopper-M
51.3
70.3
64.5
44.9
28.4
1.0
51.1
104.3 25.2
Hopper-H
43.1
75.3
76.6
52.5
22.3
11.5
87.8
95.5 213.8
Ralker2d-L
43.0
44.7
40.2
30.7
37.6
#2.3
Walker2d-M
48.7
58.5
57.3
57.6
51.8
39.9
62.2
45.2 110.4
Walker2d-H
72.6
69.6
75.3
75.5
76.3
18.0
74.9
73.7 #1.1
Hal FChettttt
29.1
30.0
38.2
31.3
30.7
40.1
54.7
33.4 21.6
HalfCheetah-M
49.0
54.6
51.8
77.8
59.2 #3.9
Hal fchhCheetah-H
71.4
75.3
77.4
81.4
76.3
65.9
83.0
71.8 28.0
Total
408.8
490.1
504.6
456.6
385.0
256.5
546.5
572.4
Table 13: D4RL MuJoCo Gym results. Each number is normalized score averaged over 100 trials (Fu et al. 2020). Our results are averaged over seeds. The results for prior works are reported following their respective papers. MOPO improved version of MOPO, introduced in /unn et al. (203)). We highlight the results that are better than 95% of the best score.
Model-free
Seq, modeling
Model-bused
Dataset
CQL
EDAC
IQL
TT
TAP
MОPO
COMBO
RAMBO
MOBILE
CBOP
LEQ (ours)
hopper-r
5.3
25.3
7.6
6.9
31.7
17.9
25.4
31.9
32.8
32.4 20a
hopper-m
61.9
101.6
66.3
67.4
63.4
62.8
97.2
87.0
106.6
102.6
103.4 4
hopper-mr
$6.3
101.0
94.7
99.4
87.3
99.5
104.3
103.9 21
hopper-ma
96.9
110.7
91.5
110.0
105.5
81.6
111.1
88.2
112.6
111.6
109.4 61.6
walker2d-r
16.6
17.9
walker2d-m
79.5
92.5
78.3
81.9
61.9
61.3
81.1
81.9
81.9
87.7
74.9 125.
walker2d-nc
76.8
73.9
89.2
66.8
85.6
walker2d-ne 109.1
114.7
1099.6
101.9
107.4
112.9
103.3
56.7
115.2
117.2
108.2 113
hal fcheetah-rr
31.3
28.4
11.8
6.1
38.5
39.3
32.8
30.8
MelffeheeLei elt-m
46.9
65.9
47.4
46.9
15.0
73.0
54.2
77.9
74.6
74.3
71.7 24A
hal Tcheet
45.3
61.3
44.2
44.1
40.8
72.1
55.
68.7
71.7
66.4
65.5 21
holffheetah--m
86.7
95.0
20.4
Total
709.7
911.4
717.2
747.5
844.0
802.0
812.4
959 3.9.5
953.4
923.2
21

---

Published as a conference paper at ICLR 2025
Table 14: V-D4RL results. We report the mean and standard deviation of returns over seeds. For ROSMO, we replace the categorical distribution in their official code with Gaussian distribution to support continuous action spaces. For MOPO, we implement MOPO on top of DreamerV3, The results for other prior works are reported following Lu et al. (2023) ; Islam et al. (2023).
Model-free
Model-based
Method
BC
CQL
DrQ+BC
ACRO
OfflineDV2
MOPO
ROSMO
LEQ
salker_wakk-ron
2.0
14.4
5.5
0.0
28.7
3.2 10.4
2.9 10.4
22.4 11.1
walker_walk-nn
37.1 13.
16.5
11.4
28.7
27.8
56.5
11.4
28.1
43.0 17.3
47.7
56.4
86.4
91.4
43.9
46.6 ja.
82.9 10.7
87.2 12.4
cheetatt_cun-radde
0.0
5.9
0.0
3.2
2.5
14.8
cheetah_run seeetahhrun-mediim
51.6
40.9
53.0
32.8
17.2
34.7 26.8
50.8
37.9 dc.0
cheetaht_un me met--m_eddddeeeesessse
25.0
10.7
44.5
41.7
61.5
30.3 =a.1
18.6 #1.5
31.3 #t.1
cheetah_run- wewahhuruu-meddd_eeeeeeeeee
57.5
20.9
50.6
46.6
36.6
45.4
25.1
Total
241.2
175.4
321.6
309.0
28100
203.3
310.9
307.6
22

---

Published as a conference paper at ICLR 2025
D RESULTS FOR ALL EXPECTILES
To give insights how the expectile parameter т affects the performance LEQ, we report the performance of LEQ with all expectile values {0.1, 0.3, 0.4, 0.5}. The expectile parameter T has trade-off- - high expectile makes the model's predictions less conservative while making a policy easily exploit the model. We recommend first trying T 0.1, which works well for most of the tasks, and increase 7 until the performance starts to drop.
Table 15:
Antmaze results of LEQ with different expectiles. We report the results in Antmaze task with expectiles value of 0.1, 0.3, 0.4, 0.5. The best value is highlighted.
Expectile
0.1
0.5
antmaze-umaze
94.4 4 #6.3
39.0 +28.1
0.2 #0.4
3,0 =5.3
entmaze--mm87--iveees
71.0
23.6
4.0
0.0 20.0
antmate-meddium-playy
50.2
58.8
36.0
0.6
antiaae--ndiummdddveree
46.2
13.2
11.6
10.6 13..
antmaaz--argo-play
42.0 $30.6
58.0 29.1
52.2 #15.
42.2 17.3
antmaze-large-divers
60.6 132.1
60.2 18.3
48.8 #5.8
36.8 #9.7
antmaze-ultrappleg
25.8 #18.2
10.8
11,6 +12.5
9.2 =11.5
antmaze-ultra-civeet
55.8 +18.3
4.6+a.4
7.6 +7.3
0,6 =1.2
Table 16: D4RL mujoco results of LEQ with different expectiles. We report the results in D4RL mujoco task with expectiles value of 0.1, 0.3, 0.4, 0.5. The best value is highlighted.
Expectile
0.1
0.3
0.4
0.5
hopper-r
32.4 #o.a
13.7 +0.1
+9.3
12.5 #10.1
hopper-m
103.4 203
102.7 11.7
81.4 221.
38.6 2:9.2
hopper-mr
103.2 21.0
103.9 =1.3
71.5 2:4.7
103.8 21.9
hopper-me
109.4 +
108.0 =8.7
64.2 #:3..8
33.7 +0.5
walker2d-r
21.5 #0.1
21.5 :0.55
14.0 #:8.8
8.7 16.7
walker2d-m
26.3 137.4
74.9 196.9
60.3 1:409
34.8 134.3
walker2d-mr
48.6 +19.5
60.5 +27.4
88,5 +3.66
98.7 #6.00
walker2d-me
108.2 +1.a
98.8 +28.8
105.8 +25.9
33.7 +31.
halfchestah-2
23.8 #1.8
30.8 +3.3
29.0 +2.9
30.2 42.5
Lahh
65.3 23.0
71.7
58.5
55.5 116.7
balfcheetah-mm
60.6
55.4
65.5
52.4 426.7
balfcheetahh-m
102.8
81.5
58.1
46.3
Table 17: V-D4RL results of LEQ with different expectiles. We report the results in Antmaze task with expectiles value 0.1, 0.3, 0.4, 0.5. The best value is highlighted.
Expectile
0.1
0.3
0.4 0.5
walker.walk-randee
14.5 +1.1
20.2 #3.
22.4 4 +1.1
21.8 +0.8
walker_walk-nes
43. 1 #3.2
37.2 +5.3
35.0 +a.7
26.6 +1.0
welker_wullk-mellunnreeee ay
40.6 16.6
40.8
41.3
43.0 273
welker_walk-mmedlnnn.xxxx
87.2 22.4
82.7
77.0
85.0
cheetah__cuozzandmm
12.1
14.8
14.2
14.6
cheetsh_rrunmmedl.um
25.0
37.9 18.0
23.6
32.2 16.2
cheetsh_rrunmmeiiummnns
34.3 21.1
32.3
31.2 13.5
31.2 10.4
c um_epperrt
23.9 14.7
18.9 13.9
25. 16.8
24.3 17.8
Total
280.7
284.8
269.9
278.6
23

---

Published as a conference paper at ICLR 2025
E MORE ABLATION RESULTS
Ablation of each component in LEQ. Table 18 we ablate LEQ's design choices on AntMaze tasks to investigate their importance. Notably, the performance of LEQ drops significantly if we remove the Q-learning loss (461.8 to 312.8) or apply expectile regression for real transitions (461.8 to 249.2), showing the importance of utilizing the real transitions long-horizon expectile loss from the policy learning is crucial (461.8 to 357.0), .0,, since simply maximizing À X-returns, even Qs, a) is conservative, can lead to a suboptimal policy due to the lack of conservatism in the rewards from model rollouts. Minor design choices, such as EMA regularization and dataset expansion, have a small impact on the performance in AntMaze.
tasks. Moreover, the
Table 18: Ablation studies about various components in LEQ on AntMaze. We conduct ablation
studies on (1) EMA regularization, (2) dataset expansion, (3) utilizing data transitions, (4) lower expectile policy optimization, (5) lower expectile regression for data transitions
Method
umaze
medium um
larga
uitra
Total
uma
diverse
play
diverse
play
diverse
play
di verae
1.BQ
94.4 gt
71.0
46.2 gar>
58.6
60.2 21
25.8 2m6
EMA regularization
96.0 .4.
65.6 us.
33.8 33.4
21.2 125.1
58.8 ss.2
62.2 115.0
5.0 :33.9
33.5 115.
379.1
espansion
63.0
=33.7
56.0
39.2 235-1
36.0
data transitions
63.0 apa
75.3 142
31.8 -34.0
28.0 #27.
30.5 120
57.5 us.
0.8 11.3
6.0 45.1
312.8
expectile policy update
93.4 243
43.4 4001
45.8 2314
11.4 6++4
53.8 2/
54.0 es.
33.8
21.4 a::
357.0
expectile data transitions
a7.
19.2
69.0 217.
75.4
Ablation pretraining. Table T9 shows the effect of BC and FQE pretraining for LEQ and MOBILE in AntMaze tasks. Without pretraining, the performance of LEQ decreases (461.8 -> 322.2), especially in medium mazes (96.4 4.6) and ultra mazes (81.6 12.4) due to early instability of training, matching the observation in CBOP (Jcong et al. 2023). However, for MOBILE', pretraining worsens the performance (285.3 + 232.2).
Table 19: Ablation results for pretraining on AntMaze. Results are averaged over random seeds.
Method
uttlicc cre
medi
large
uil Lie
Total
conservarism
pretrain
umamee
diveree
play
61 civeree
play
diverre
play
diverse
LEQ
91.4 20.3
71.0 #12.
50.2 =m1.
16.2 2212
58.6=ax =aa1
60.2 116.3
25.8 a1a.
55.8 316.3
161.8 322.2
LEQ
X
94.0
65.6 26.
0,8
3.8 27M
2184
60.8 2128
11.6 2124
26.2
MOBILE" MOBILE
73.4 #12.
46.0 894
24.6 ran.
11.6 n0.
31.0 ma44
33.2 a10.6
12.4 n6.
0.0 sou
232.2
X
53.8 125.5
22.5
51.
19.5
28.3
28.0 #11.4
25.5 463
23.8
285.3
Ablation study dataset expansion. Table shows the ablation results the dataset expan- sion in D4RL MuloCo tasks. The results show that the dataset expansion generally improves the performance, especially in Hopper environments.
Table 20: D4RL MuJoCo ablation results for dataset expansion. Results averaged over random seeds. The dataset expansion generally improves the performance of LEQ.
Dataset
LEQ (ours)
LEQ w/o Dataset Expansion
hopper-r
32.4 10.3
17.6 25.6
hopper-m
103.4 103
52.7 245.5
hopper-mr
103.9
103.7
hopper-me
109.4 #1.8
79.7 =42.4
walker2d-r
21.5
20.5 1:222
walker2d-m
74.9 1o6.
87.2
walker2d-mz
98.7 +s.0
78.7 =35.5
walker2d-me
108.2 #1.3
110.4 tu.a
hal fcheettttb
30.8
27.7
hal fchhetahhhmm
71.7 14.4
71.6 13.8
hal falffheetaaa---
65.5 #1.1
54.4 =26.5
ha 1. fcheeeeeehhhmm
102.8 1o.4
83.9 128.0
Total
923.2
788.2
24

---

Published as a conference paper at ICLR 2025
Ablation using deterministic policy. We parameterize the stochastic policy TT-ss) T(als)
tanh(N(p(s)) (s))) as Haarnoja et al. (2018) and run LEQ with this configuration. However, we found that the policy quickly becomes deterministic, because LEQ inadvertently penalizes the stochasticity of the policy while penalizing the uncertainty of the model rollouts. Specifically, when a stochastic policy, the stochasticity of the policy contributes increasing the variance of A-returns, which are therefore heavily penalized by LEQ.
To compensate for this effect, use an entropy bonus coefficient 0.0003. As demonstrated in Table 21 stochastic policy shows slightly worse performance compared to deterministic policy (461.8 380.4). However, believe that LEQ can be extended to stochastic policies with further hyperparameter tuning on the stochasticity of the policy.
Table 21: Ablation results on using deterministic policy. Results are averaged over 5 random seeds.
Policy
modi um
large
ultra
Total
Lmaze
diverse
play
diverse
play
diverse
play
diverse
Deterministic
94.4 16.3
71.0 11aa
50.2 138.9
46.2 aear
58.6 1
60.2 1iaa
25.8
118.2
55.8 116a
461.8
Stochastic
91.0 27.9
70.4
35.8 $20.7
0.0 #aa
#30.5
50.2 a7a
43.4 +25.0
47.8 =23.3
380.4
Ablation on using learned terminal function. We conduct an ablation study to evaluate the impact of using learned terminal functions instead of the ground-truth terminal function. For the terminal prediction network, we use 3-layer MI.P with a hidden size of 256, consistent with the architecture of the policy and critic networks. As shown in Table 21 replacing the true terminal function with learned terminal function leads to a significant drop in performance.
Table 22: Ablation results on using learned terminal function. Results are averaged over 5 random seeds.
Terminal
medium
large
ultra
Total
umaze
diverse
play
diverse
play
diverse
play
diverse
Groundrruhh
94.4 16.
71.0 2183
50.2 2an9
46.2 223.
58.6 aa:
60.2 21sa
25.8 alsa
55.8 a/8.
461.8 232.6
Learned
66.8 15.
44.0 222.3
54.0 111.4
2.0 14.0
29.4 181
0.0 100
15.6 25.9
20.8
15.2
Ablation study on horizon length in V-D4RL. Table 23 shows that when we use the default hyperparameter of DreamerV3, H 15, the performance drops in V-D4RL. The result suggests that we need to use a shorter imagination horizon for offline model- RL.
Table 23: V-D4RL ablation results for horizon length. Results are averaged over 3 random seeds. H generally improves the performance of L.EQ.
Dataset
H=5
H=15
walker www.wwlkkaaaded
23.2
15.4 +1.5
50.0 +3.6
40.0 +3.6
selker__walk-mecimmmeeee
44.0 228.5
45.4 27.5
walker_walk-medde eeppreeriiieeeeeteeee
90.3 +1.8
79.9 +1.4
cheetah_run-raane
15.3 +1.7
15.5 +3.0
cheetah_rrunmediin
40.1 11.6
32.2 24.3
replay
39.9 42.0
36.4 42.1
cheetahuruu--eediun_cem
27.7 212.0
29.3 24.1
Total
330.5
294.1
25

---

Published as a conference paper at ICLR 2025
F VIDEO PREDICTIONS ON V-D4RL
Figure 12 presents the imagined trajectory generated by the action sequence during evaluation in the V-D4RL datasets: walker_wald walk__wakk waallk--mdium_eeee medeuu_repla.,
Balkeerwaak-mer um @mmexeet, cheetah_run www.run--ediing mediun_replay, cheetah_run medi un_epper, from the top to the bot- tom. Overall, world models trained medi um replre datasets show better prediction compared to medium mediun_expert. dataset, likely due to their broader state distribution of the dataset. Neverthe- less, LEQ achieves high performance on the wa 1 ker_wall walk-medi um ammexper" rt dataset, despite inaccurate predictions, highlighting robustness of LEQ in handling imperfect world models.
True
Model
Diff.
(a) wal weilker_walkkmmdiiuneepplll.
True
Model
Diff.
(b) wa walkerr_eae Baleeeerrraaaaee wawk--mdiuuu_xxtee _xpeet
True
Model
Diff.
(c) checttchhuuuo-mmdiiuurrrooaay
True
Model
Diff.
(d) cheetah_sst run medd- um expert
Figure 12: Video prediction results in V-D4RL. The world model receives initial 5 frames and simulates 64 additional frames based on the action sequence during evaluation. For each trajectory, each row displays the true image, model prediction, and difference between these two, respectively.
26


Reviews:
[{"review_id": "XX9jrvP21G", "reviewer": "Reviewer_FFdb", "summary": "This paper proposes a lower expectile Q-learning (LEQ) approach to address the overestimation issue in model-based offline RL. The LEQ loss assigns higher weight to lower Q-target values and lower weight to higher Q-target values. Through this training mechanism, LEQ achieves conservative value function estimation without the need for uncertainty estimation. Additionally, the authors introduce the $\\lambda$-return as the Q-target and the policy update objective. Extensive experiments on offline RL tasks demonstrate that LEQ consistently outperforms previous baselines.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. LEQ is a simple yet highly effective algorithm, easy to implement, and does not introduce significant computational complexity. It exhibits substantial performance advantages across a range of offline RL tasks.\n2. Beyond its impressive performance improvements, the authors provide in-depth analysis of LEQ’s instability on several tasks, giving readers a comprehensive understanding of the algorithm and hinting at multiple future research directions.\n3. The paper includes extensive performance comparisons and ablation studies, offering thorough insights into the contributions of each component of the algorithm.", "weaknesses": "I would greatly appreciate clarification on the following points:\n\n1. Why did the authors choose to update the policy based on deterministic policy gradients? What considerations influenced this choice, and are there experimental differences between using a stochastic versus deterministic policy?\n2. In Algorithm 1, the authors pretrain the policy with BC using the offline data, which is not typically done in algorithms like MOPO or MOBILE. How does BC pretraining impact the results, particularly in tasks like AntMaze?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jtVcPPRYlt", "reviewer": "Reviewer_xSxW", "summary": "This paper introduces Lower Expectile Q-learning (LEQ), a novel model-based offline reinforcement learning method that utilizes lower expectile of Q values to avoid overestimation.  The success of LEQ is attributed to lower expectile regression, λ-returns, the lower expectile target for the policy training. LEQ significantly outperforms previous MBRL methods on long-horizon tasks and matches state-of-the-art model-free methods in various environments.", "questions": "- I do not find any reference of Figure 2 in the paper. Could you please insert the figure into the proper place to better explain your method?\n- How do you use the ensemble of world models for sampling? For example, do you randomly select one model for each step or for each rollout?\n- For the experiments, though you include TT and TAC as baselines, I do not find any analysis of the related results. Could you please explain why to include TT and TAC, and provide what we can learn from comparing LEQ against them?\n- In the proof of Theorem 1, it seems that the last inequality holds if $\\mathbb{E}[W^\\tau(Y>X)] \\ge 1-\\tau$. However, when $0<\\tau \\le 0.5$, we have $\\tau \\le W^\\tau(\\cdot) = |\\tau - 1(\\cdot)| \\le 1-\\tau$, so $\\tau \\le \\mathbb{E}[W^\\tau(\\cdot)] \\le 1-\\tau$. As a result, the last inequality might be in the wrong direction.\n- What is the difference between MBPO and Dyna training schemes, as you classify in Table 8? Previous works usually regard MBPO as a Dyna-style algorithm. For example, though you classify MOBILE into the MBPO style, MOBILE itself claims that \"MOBILE is under the Dyna-style framework\" and cite MBPO as a \"Dyna-style model-based RL\" method in Section 5 of its original paper[1].\n\n[1] Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning. Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, Yang Yu. Proceedings of the 40th International Conference on Machine Learning, PMLR 202:33177-33194, 2023.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow. The description to the thought of design and the algorithm is clear.\n- The paper provides both experimental and theoretical analysis, which enhances the soundness of the lower expectile policy learning.\n- The experiments are conducted on various types of tasks, including dense- and sparse-reward environments, along with state and image input formats, showing the effectiveness of LEQ.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wNy8Aidy9j", "reviewer": "Reviewer_2jfa", "summary": "To to address the uncertainty of model data $Q$ value estimation, the authors  introduce low expectile regression into the algorithm, and propose a novel model-based offline RL method. This method can achieve conservative estimation for model data and demonstrates superior performance across multiple datasets.", "questions": "1) Does the addition of the third loss term in Eq. (7) affect final performance? The authors could provide simple experiments for analysis.\n\n2) Why does the policy update use only the generated model data and not the precise offline data? Most model-based offline RL algorithms currently use accurate offline data to update policy.\n\n3) Is there an error about the definition of $L_2^{\\tau}$ in Eq. (1)? According to my understanding, in Eq. (4), $L_2^{\\tau}(u)=\\|\\tau-\\mathbb{I}(u>0)\\|u^2$.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The article is well-written overall, with clear logic and a detailed introduction on how to incorporate low expectile regression into RL algorithm. The experiments are thorough, validating the algorithm's superiority across multiple datasets.\n\n2) The methodology presented in the article is very intriguing. Unlike traditional value penalization methods, this paper innovatively employs low expectile regression to alleviate the issue of inaccurate model data estimation.", "weaknesses": "1) Theoretical: The article does not provide the theoretical analysis of how the use of low expectile regression impacts the final Q-values of the model data. Additionally, it does not offer theoretical guarantees for policy improvement , as seen in algorithms like COMBO and MOBILE.\n\n2) Algorithm: The method presented in this paper directly scales the loss values of the overestimated model data by a factor of $\\tau$, without considering the differences among the model data. For example, the scaling coefficient $\\tau$ should vary depending on the degree of error in the model data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sXW82pJetw", "reviewer": "Reviewer_LGd6", "summary": "This paper introduces a novel model-based offline reinforcement algorithm called Lower Expectile Q-Learning (LEQ). LEQ is similar to prior model-based offline RL algorithms with three main additions. 1) LEQ uses expectile regression for both critic and policy training, to obtain conservative Q-value estimates and actions respectively. 2) LEQ uses $\\lambda$-returns for both critic and policy training. 3) The critic is trained with both model-generated and the provided offline data, while the actor is trained with only model-generated data. There are a number of other smaller additions, such as the use of EMA regularization when training the critic and some hyperparameter changes to prior work.\n\nIn summary, the approach is clearly explained, comprehensively evaluated and the components are individually studied. Therefore while none of the contributions of the paper are particularly novel or likely to have a major impact on the direction of the field, I believe the paper should be accepted due to the thoroughness of the work. LEQ will likely provide a strong general-purpose baseline for model-based offline RL in future work.", "questions": "- On line 256, the authors state \"because of the expectile term ..., we cannot directly compute the gradient... thus we optimize the unnormalized version of the expectile\". Please could the authors expand on this line? What is meant by the 'unnormalized' version of the expectile? Also, on line 258 the authors say they approximate the expectation with the learned Q-estimator in order to derive the surrogate loss. But then on line 266 they say the surrogate loss provides a better approximation than the Q-estimator, which naively seems contradictory (without going through Appendix B). Could the authors clarify this please?\n- In Equation 7, can you confirm that $\\beta$ and $\\beta_{EMA}$ are independent hyperparameters? Assuming so, it would be less confusing to use a different symbol for $\\beta_{EMA}$.\n- Could you explain the main difference between LEQ and IQL-TD-MPC and include this in the paper?\n- In Section 5, LEQ is compared with the baselines in terms of performance. It would also be helpful to have some comparison in terms of computational cost/run-time, as this is also an important aspect of these algorithms (and LEQ appears quite strong in this regard from the appendix). Could this be included?\n- In Section 5.3, the authors discuss failed trajectories that attempt to pass through walls, and that 'this could be addressed by... improving the number of ensembles for the world models.' This is an interesting failure case, but I found this confusing as my understanding is that LEQ uses a single world model - is this the case? Do the authors have any hypothesis or evidence that using an ensemble would address this issue? \n- Similarly in Section 5.4, the authors discuss variance in performance during training due to optimistic prediction. Does a reduced value of $\\tau$ in the expectile regression (more conservative estimates) address this issue?\n- Section 6.1 on Limitations seems out of place at the end of the work. This seems more appropriate before the conclusion (ideally included as an additional ablation in 5.6 with a minimal termination predictor on one or two environments). Could this paragraph be moved before the conclusion?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Clarity**\n  - The method is clearly motivated and explained.\n  - The results are clearly analyzed and summarized.\n  - The paper is generally well-written and structured.\n- **Quality and Comprehensiveness**\n  - LEQ is comprehensively evaluated across a wide-range of benchmarks. The results are realistically strong all-around, and the authors do a good job of analyzing the strengths of the method. In particular, LEQ appears to perform better than the baselines (impressively both model-based and model-free) at long horizon tasks, such as the larger AntMaze environments.\n  - The authors appear to do a good job of ensuring fair comparisons with a wide range of baselines.\n  - There are extensive ablation studies to justify the method's contributions and design decisions.\n- **Significance**\n  - The quality of the work suggests that LEQ will provide a strong and computationally fast baseline for future work on offline RL.", "weaknesses": "- **Significance**\n  - None of the components of the model are particularly novel, and as a result, I wouldn't expect LEQ to be able to solve any problem that is not already solvable by existing offline RL algorithms. While the all-around improvement is of benefit to the community, this reduces the significance of the work.\n  - Other insights such as the benefit of using $\\lambda$-returns are generally well-established, and the improvements in hyperparameters are likely to be environment specific and therefore not very generalizable. \n  - While standard practice in offline RL, the use of differing hyperparameters per environment (specifically $\\tau$, as provided in Table 10), also limits the significance and generality of the work. However, the results for all expectiles provided in the Appendix is helpful and appreciated.\n- **Originality**\n  - In addition to the novelty of the components, the use of expectile regression as used in IQL for model-free offline RL appears to have already been applied ot model-based offline RL in IQL-TD-MPC [1]. This method appears as a baseline for the AntMaze results, but does not appear in the other results (Tables 2,3,4) or in the Related Work. Given the apparent similarity in the approach, this method should be explained and distinguished in the paper's motivation/related work and ideally used as a baseline for all results. The citation for this method is also wrong in Section 5.3.\n\n**References:**\n\n[1] *IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Control*, Chitnis et al. 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)